ML040710185

Wat - Init Exam - 8/2003 - Draft Written Exam Comments ES-401-9

ML040710185
Person / Time
Site:	Waterford
Issue date:	02/14/2005
From:	- No Known Affiliation
To:	Office of Nuclear Reactor Regulation
References
50-382/03-301
Download: ML040710185 (5)

Text

ES-401 Written Examination Form ES-401-9 (R8, S1)

Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 X S 2 H 2 X S 3 H 3 S 4 H 2 X S 5 F 2 X S 6 F 3 S 7 H 2 X S 8 K 2 S 9 H 3 X S 10 K 2 X S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ More than one distractor is not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 (R8, S1)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 3 X S 12 F 3 X S 13 F 2 E This is a fundamental question, not comp/analysis as idd (Chgd to memory) 14 H 2 X S 15 F 3 U S Reverse logic, acceptable in this case because of clear stem notice.

16 F 2 S 17 F 3 X E This is a fundamental question, not comp/analysis as idd (Sat as -is) 18 H 3 S 19 F 2 U X X U TS bases question, SRO level (low discrimination) (Chgd distractor B) 20 H 3 E Discussion of correct answer says incorrect in comments. (Revised comments) 21 H 1 U U Although this is bank, it is essentially match the answer, weak discrim.

(SAT as-is) 22 H 3 X S 23 F 2 X S 24 F 2 X S 25 F 2 U E Distractor A may also be correct, although C is the correct answer.

(SAT as-is) 26 F 2 U X S Reverse logic, acceptable in this case because of clear stem notice.

(SAT as-is) 27 H 3 X S 28 F 2 S 29 H 2 S 30 F 2 E This is a fundamental question, not comp/analysis as idd (SAT as-is)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 31 F 2 U S Reverse logic, acceptable in this case because of clear stem notice.

32 H 3 S 33 H 2 X S 34 F 1 U X U GFES type question. Suggest more operationally oriented question.

(Rewritten to use stm tbls) 35 F 2 S 36 H 3 S 37 F 1 U U U Reverse logic + T/F+ low disc. Value (Rewritten to quest abt lights) 38 F 3 U X S Reverse logic, acceptable in this case because of clear stem notice.

39 F 2 X S 40 F 2 X S 41 H 3 X S 3rd question (37,41) requiring knowledge of setpoints related to cont.

press. (SAT as-is) 42 H 3 S 43 H 2 S 44 H 3 S 45 H 3 X S 46 F 2 X S 47 H 3 X S 48 H 3 X S 49 H 2 S 50 H 3 S 51 H 2 S 52 F 2 S 53 F 2 X S This is a fundamental question, not comp/analysis as idd 54 F 2 X S 55 H 3 X S

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 56 F 2 S 57 F 2 S 58 F 2 S 59 F 2 X S 60 F 3 X S Comments incicate that D is incorrect, but idd as correct. (Rewrite comments) 61 F 2 X S Comments incicate that B is incorrect, but idd as correct. (Rewrite comments) 62 F 2 X S 63 H 3 S 64 H 3 S 65 H 2 X S 66 F 2 X S Comments incicate that C is incorrect, but idd as correct. (Rewrite comments) 67 H 3 X S 68 F 2 S 69 F 2 X S 70 F 3 S 71 F 2 X S 72 H 1 U U/E Distractors are very obviously wrong. Too easy. Bank question.

(Rewrite stem) 73 H 2 X S 74 H 3 X S U 75 F 2 X S 76 H 3 X X S 77 H 2 X X S 78 H 3 X X S 79 H 2 U X X E Although argument that A is wrong, it may still be acceptable.

(Rewritten)

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 80 F 2 X X S 81 H 2 X X S 82 H 1 U X X U The answer will always be the shortest time. No discrimination.

(Rewritten) 83 H 2 U X X U/E C and D are easily rejected. Offsite dose is not relevant to stem.

(ReWritten) 84 H 2 X X S 85 H 2 X X S 86 H 3 X X S 87 H 3 X X S 88 H 2 X X S 89 H 2 X X S Reverse logic, acceptable in this case because of clear stem notice.

(SAT as-is) 90 H 3 X X S 91 H 2 X X S 92 H 3 X X S 93 F 2 X X S 94 F 2 X X S 95 H 3 X X S 96 H 2 X X S 97 F 2 X X S 98 F 2 X X S 99 H 3 X X S 100 H 2 X X S NOTE: Resolution to comments are provided in BOLD

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only