ML050490371

11/2004 - Draft Written ES-401-9 Review

ML050490371
Person / Time
Site:	Waterford
Issue date:	12/20/2004
From:	Gody A Operations Branch IV
To:	Venable J Entergy Operations
References
50-382/04-301, ES-401, ES-401-9, NUREG-1021, Draft Rev 9 50-382/04-301
Download: ML050490371 (6)

Text

ES-401 W3 12/2004 Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 - S 2 H 1 X - U  ; including the HPSI Throttle Criteria as a handout clearly cues the correct answer, and also makes distractor D not credible; reference was not provided when previously administered on NRC 2003 exam.

3 F 3 - S 4 H 3 - S 5 F 3 X U doesnt test knowledge of REASONS for ensuring proper S/G levels for nat. circ. for RCP malfunctions per K/A.

6 H 2 X - U including the handout graph immediately cues that distractors C and D are not credible since they are not related to minimum operating level as indicated on the graph.

7

• H 2 X U question was claimed as modified, but was not sufficiently changed from parent to be called modified - must call it bank; doesnt match the K/A at all.

8 H 3 X U add CCW to the 5th and 6th bullets in the stem for clarity; the proposed student handout seems irrelevant to the question; doesnt test ability to diagnose and recognize trends using control room reference material per K/A 9 H 3 - S 10 H 3 - E change distractor D to DRTS and EFAS-2 should have...

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ One or more distractors is (are) not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

33 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11 H 2 X - X U this is not RO license level (it is 55.43.b.2); it is not clear whether any answer is correct (TS3.4.5.2 only requires shutdown if leakage cannot be reduced within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />);

12 F 2 - S 13 H 2 - S 14 F 3 - S 15 H 3 - S 16

• F 3 - S  ; question is insufficiently changed from original to be called modified as submitted by the author - must be called bank.

17 H 3 - S 18 H 3 - S 19 H - S 20 F 3 - S 21 H 3 X U doesnt test knowledge of REASONS for NORMAL OPERATING PRECAUTIONS to preclude/minimize SGTR per K/A.

22 F 3 - E change stem units to inches Hg/minute 23 F 2 - S 24 F 3 - S 25 F 3 - S 26 H 3 - E comments should say Provide...Appendix 2 Attachment 2-A...

27 F 3 - S 28 F 3 - S 29 H 3 - S 30 H 3 - S 31 H 3 - S 32 F 3 X - U distractor D is not credible.

34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 33 H 3 - X U not RO license level (more like 55.43.b.2) 34 F 3 - S 35 H 3 - S 36 H 2 X - U distractor A is not credible.

37 H 3 - S 38

• H 3 - E question not sufficiently changed to be called modified - must be bank 39 H 3 - S 40 F 3 - S 41 H 3 - S 42 H 3 - E replace indicates closed with is closed in B and D.

43 H 3 - E should bus B1" actually read bus 1B in the stem?

44 F 2 - S 45 H 3 - S 46 H 3 X U doesnt test knowledge of EFAS interlocs with blowdown/sampling isolations per K/A.

47 F 3 X U change stem to ...procedure, and has secured all...; doesnt test knowledge of symptom-based EOP mitigation strategies regarding EFW per K/A.

48 F 2 X U doesnt test ability to predict the impact of and follow procedures in response to paralleling out of phase per K/A.

49 F 3 - E change D to ...charger startup, a phase fail light is expected and must be manually reset.

50 F 2 - S 51 F 2 X U doesnt test knowledge of operational implications of radiation theory on PRM system per K/A 52 F 3 - S 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 53 H 2 - S 54 H 1 - U answer is exceedingly obvious 55

• F 3 - E  ; not sufficiently changed to be called modified - must be bank 56 F* 3 - S I disagree with the authors LOK; 57 F 3 - S 58 H 2 X - U set stem at 35% power vice 55%, and then choose distractors based on Th, and Tc, as well as ~560 deg F.

59

• H 3 X - U  ; question not sufficiently changed to be called modified - must be bank; having appeared on the previous 2 NRC exams, this question should be replaced.

60 H 3 - S 61 H 3 - S 62 F 3 - S 63 H 3 X - U distractors C and D are not credible.

64 F 1 X U answer is obvious; doesnt test knowledge of ARM features related to fuel building isolation per K/A.

65 F 3 - S 66

• F 2 X - U  ; question not sufficiently changed to be called modified -

must be bank; distractor A is not credible.

67 F 2 - S 68 H 2 - S 69 F 3 - S 70 H 2 - S 71 F 2 - S 72 H* 3 - S I disagree with authors LOK rating.

73 H* 3 - S I disagree with authors LOK rating; 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 74 F 2 - S 75 H* 3 - S 76 H 4 - - S 77 H 3 X - - S 78 H 3 - - S 79 H 3 - - E eliminate comma in first sentence of stem; use semi-colons to separate the two halves of the answer in each choice.

80 F 3 - - S 81 H 3 - - S 82 H 3 X - - U meaning of 2nd sentence in stem is unclear; should capitalize entire title of Att. 4 in stem;; answer is cued by handout Att. 4; Change with in each distractor to within; 83 F 3 - - S 84 H 3 - - S 85 H 3 - - S 86 H 3 - - E this is not, as indicated in the submittal, related to 55.43.b.3 - maybe it is 55.43.b.5; 87

• H 2 - - S question not sufficiently changed to be called modified - must be bank 88

• H 3 - - S question not sufficiently changed to be called modified - must be bank; 89 H 2 - - S 90 H 3 - - S 91 H 2 - - S 92 H 2 - X U not SRO-only (this is 55.41.b.13) 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 93 F 1 - - U answer is too obvious.

94 F 2 - - S 95 F 2 - X U not SRO-only (this is 55.43.2) 96 H 3 - - S 97 H 2 - - S 98 F 3 - - S 99 H 3 - - S 100 H 3 - - S 34 of 34 NUREG-1021, Draft Revision 9