ML050490371
| ML050490371 | |
| Person / Time | |
|---|---|
| Site: | Waterford  |
| Issue date: | 12/20/2004 |
| From: | Gody A Operations Branch IV |
| To: | Venable J Entergy Operations |
| References | |
| 50-382/04-301, ES-401, ES-401-9, NUREG-1021, Draft Rev 9 50-382/04-301 | |
| Download: ML050490371 (6) | |
Text
ES-401 W3 12/2004 Written Examination Form ES-401-9 Review Worksheet
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 - S 2 H 1 X - U ; including the HPSI Throttle Criteria as a handout clearly cues the correct answer, and also makes distractor D not credible; reference was not provided when previously administered on NRC 2003 exam.
3 F 3 - S 4 H 3 - S 5 F 3 X U doesnt test knowledge of REASONS for ensuring proper S/G levels for nat. circ. for RCP malfunctions per K/A.
6 H 2 X - U including the handout graph immediately cues that distractors C and D are not credible since they are not related to minimum operating level as indicated on the graph.
7
- H 2 X U question was claimed as modified, but was not sufficiently changed from parent to be called modified - must call it bank; doesnt match the K/A at all.
8 H 3 X U add CCW to the 5th and 6th bullets in the stem for clarity; the proposed student handout seems irrelevant to the question; doesnt test ability to diagnose and recognize trends using control room reference material per K/A 9 H 3 - S 10 H 3 - E change distractor D to DRTS and EFAS-2 should have...
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
@ The answer choices are a collection of unrelated true/false statements.
@ One or more distractors is (are) not credible.
@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
@ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
33 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 2 X - X U this is not RO license level (it is 55.43.b.2); it is not clear whether any answer is correct (TS3.4.5.2 only requires shutdown if leakage cannot be reduced within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />);
12 F 2 - S 13 H 2 - S 14 F 3 - S 15 H 3 - S 16
- F 3 - S ; question is insufficiently changed from original to be called modified as submitted by the author - must be called bank.
17 H 3 - S 18 H 3 - S 19 H - S 20 F 3 - S 21 H 3 X U doesnt test knowledge of REASONS for NORMAL OPERATING PRECAUTIONS to preclude/minimize SGTR per K/A.
22 F 3 - E change stem units to inches Hg/minute 23 F 2 - S 24 F 3 - S 25 F 3 - S 26 H 3 - E comments should say Provide...Appendix 2 Attachment 2-A...
27 F 3 - S 28 F 3 - S 29 H 3 - S 30 H 3 - S 31 H 3 - S 32 F 3 X - U distractor D is not credible.
34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 33 H 3 - X U not RO license level (more like 55.43.b.2) 34 F 3 - S 35 H 3 - S 36 H 2 X - U distractor A is not credible.
37 H 3 - S 38
- H 3 - E question not sufficiently changed to be called modified - must be bank 39 H 3 - S 40 F 3 - S 41 H 3 - S 42 H 3 - E replace indicates closed with is closed in B and D.
43 H 3 - E should bus B1" actually read bus 1B in the stem?
44 F 2 - S 45 H 3 - S 46 H 3 X U doesnt test knowledge of EFAS interlocs with blowdown/sampling isolations per K/A.
47 F 3 X U change stem to ...procedure, and has secured all...; doesnt test knowledge of symptom-based EOP mitigation strategies regarding EFW per K/A.
48 F 2 X U doesnt test ability to predict the impact of and follow procedures in response to paralleling out of phase per K/A.
49 F 3 - E change D to ...charger startup, a phase fail light is expected and must be manually reset.
50 F 2 - S 51 F 2 X U doesnt test knowledge of operational implications of radiation theory on PRM system per K/A 52 F 3 - S 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 53 H 2 - S 54 H 1 - U answer is exceedingly obvious 55
- F 3 - E ; not sufficiently changed to be called modified - must be bank 56 F* 3 - S I disagree with the authors LOK; 57 F 3 - S 58 H 2 X - U set stem at 35% power vice 55%, and then choose distractors based on Th, and Tc, as well as ~560 deg F.
59
- H 3 X - U ; question not sufficiently changed to be called modified - must be bank; having appeared on the previous 2 NRC exams, this question should be replaced.
60 H 3 - S 61 H 3 - S 62 F 3 - S 63 H 3 X - U distractors C and D are not credible.
64 F 1 X U answer is obvious; doesnt test knowledge of ARM features related to fuel building isolation per K/A.
65 F 3 - S 66
- F 2 X - U ; question not sufficiently changed to be called modified -
must be bank; distractor A is not credible.
67 F 2 - S 68 H 2 - S 69 F 3 - S 70 H 2 - S 71 F 2 - S 72 H* 3 - S I disagree with authors LOK rating.
73 H* 3 - S I disagree with authors LOK rating; 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 74 F 2 - S 75 H* 3 - S 76 H 4 - - S 77 H 3 X - - S 78 H 3 - - S 79 H 3 - - E eliminate comma in first sentence of stem; use semi-colons to separate the two halves of the answer in each choice.
80 F 3 - - S 81 H 3 - - S 82 H 3 X - - U meaning of 2nd sentence in stem is unclear; should capitalize entire title of Att. 4 in stem;; answer is cued by handout Att. 4; Change with in each distractor to within; 83 F 3 - - S 84 H 3 - - S 85 H 3 - - S 86 H 3 - - E this is not, as indicated in the submittal, related to 55.43.b.3 - maybe it is 55.43.b.5; 87
- H 2 - - S question not sufficiently changed to be called modified - must be bank 88
- H 3 - - S question not sufficiently changed to be called modified - must be bank; 89 H 2 - - S 90 H 3 - - S 91 H 2 - - S 92 H 2 - X U not SRO-only (this is 55.41.b.13) 34 of 34 NUREG-1021, Draft Revision 9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 93 F 1 - - U answer is too obvious.
94 F 2 - - S 95 F 2 - X U not SRO-only (this is 55.43.2) 96 H 3 - - S 97 H 2 - - S 98 F 3 - - S 99 H 3 - - S 100 H 3 - - S 34 of 34 NUREG-1021, Draft Revision 9