ML053330596

08-2005 - Initial Draft Written Examination Comments

ML053330596
Person / Time
Site:	Palo Verde
Issue date:	08/31/2005
From:	NRC Region 4
To:	Arizona Public Service Co
References
50-528/05-301, 50-529/05-301, 50-530/05-301
Download: ML053330596 (6)

Text

ES-401 PV-08-2005 Written Examination Form ES-401-9 Review Worksheet

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 3 - S 2B* H 2 - S NRC2003 RO#2; doesnt qualify as new - must count as bank; 3B* H 3 - E doesnt qualify as modified - must count as bank; wording in C and D is confusing; 4 H 3 X - U C not credible and has typo; 5B H 3 - S NRC2003 RO#5 (same KA also) 6 H 2 - S 7B H 3 - S 8 F 2 X - U B and D are both correct; 9B* F 3 - E NRC2003 SRO#4; doesnt qualify as modified - must count as bank; 10 H 3 - S Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

@ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

@ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

@ The answer choices are a collection of unrelated true/false statements.

@ One or more distractors is (are) not credible.

@ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

@ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

@ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

@ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

@ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

7. At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).

33 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 11B H 3 - S 12B H 3 - S 13B H 3 X X - U does the stem have enough information to exclude C? Typo in C (too many onlys);

14B H 1 X - U too easy - direct lookup; B not credible; 15B H 3 - S 16B F 3 - S 17B H 3 - S NRC2001 RO#36 (same KA also);

18B H 3 X - U Loss of MFW KA topic has already been sampled by Q12, so either Q12 or Q18 must be replaced with an unsampled topic (15/17 or 57);

add oF to temperature in stem; C and D are not credible; 19 H* 3 X - U LOK is H vice F; this is not RO knowledge (LO doesnt match Q either);

20 F 2 S 21 H 3 - S 22B F 2 - S 23 F 3 - S 24M H 3 X X U doesnt test knowledge of REASONS... per KA; D is correct also; 25B H 3 - S 26B H 3 X - U D is also correct; 27B H* 3 - E LOK is H vice F; explanation in comments doesnt match Q; 28B F 3 - S 29B H 3 - S 30 H 3 - S 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 31 H 2 - E is D the only potential concern?;

32 H 3 X - U eliminate pressurizer level... from all 4 distractors; 33B F 2 - S 34B H 2 - E change starting pressure to 1385 psia and the answer would be ~2650F; KA in comments doesnt match outline; 35B* F 3 X - U doesnt qualify as modified - must count as bank; B is not credible; 36B F 2 - S 37B H 3 - S NRC2003 SRO#3; 38B H 3 - S 39B H 3 - S 40B* F 3 X U doesnt qualify as modified - must count as bank; doesnt test knowledge related to ESFAS logic per KA; 41B H 3 - S 42B F 3 - S 43B H 3 - E stem cues answer to Q1 - replace valves 7 & 8" with atmospheric; 44B H 3 - E NRC2003 SRO#17; replace second semicolon on C and D with to; 45 H 3 X X - U A isnt credible; add only to B and C; 46B H 3 - S 47B H 3 X U doesnt test knowledge of effect of loss of ac will have on MAJOR SYSTEM LOADS per KA; 48M F 2 X - U D is not credible; 49 F 3 - S 50 H 3 X U doesnt test knowledge of effect of loss/malfunction of EDG on ESFAS per KA; 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 51 F 4 X X - U do ROs have to know these LCOs by memory?

52 H* 3 - E LOK is H vice F; 53B F 3 - S 54 H 3 - S 55B H 3 - S 56B F 3 - S 57 F* 1 X - U LOK is F vice H; reword without negative stem; answer is obvious; 58B F 2 - S 59B* H 3 - E doesnt qualify as modified - must count as bank; 60B F 2 - S 61 H 3 - S 62B F 4 X - U is this an RO learning objective?

63 H 3 - S 64M F 3 - S 65 F 3 - S 66 F 2 X X U doesnt seem operationally important for an RO exam; doesnt test ability to operate plant communication systems per KA; answer is not supported by reference provided; 67B* F 3 - E doesnt qualify as modified - must count as bank; 68B F 2 - E change downpowered to powered in D; 69B F 3 - S 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 70B F 2 - S 71B F 2 X - U D is not credible; 72 H 3 X - U add only to A and B; 73B F 2 - S 74B F 2 - S NRC2001 RO#99 (same KA also);

75B F 3 X - U is this RO knowledge? it looks like an SRO LO and Q; S1B H 3 X - X U NRC2003 RO#63;C and D not credible; this is RO knowledge (55.41.b.10);

S2 H 3 X - - E is the D also correct?

S3M H 3 - - S S4 H 3 - - S S5 H 3 - - E replace its with its; S6 H 3 X - X U A is a subset of D; this is RO knowledge (55.41.b.10);

S7 H 3 - X U this is RO knowledge (55.41);

S8B H 3 X X U doesnt test ability to determine/interpret CHARGING SUBSYSTEM FLOW INDICATOR AND CONTROLLER for PZR level control malfunctions per KA; this is RO knowledge (55.41);

S9B H 3 - X U this is RO knowledge (55.41.b.10);

S10B H 3 - X U this is RO knowledge (55.41.b.10);

S11B H 3 - - - S S12 H 3 X X - - U shouldnt amps be more than slightly above zero for this condition? A is not credible; S13 H 3 X - - U C is a subset of D; 34 of 34 NUREG-1021, Draft Revision 9

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only S14 H 3 X X - - U change rapid to rapidly in stem; A is also correct; S15 H 3 - - S S16 H 2 X X - - U as worded, there are no correct answers; S17 H* 2 X - - E establish symmetry in distractors between A to C and B to D; S18 H 3 X - - E how much of the EAL tables are being provided? Appears to be a direct lookup; S19B H 3 X - - U NRC2003 SRO#18 (same KA also);currently, there are no correct answers; eliminate must from the end of the stem, put must in A, may in C and D, and replace wait... with but only with TSC approval in C; S20B F 3 X X U should the graph be provided?; this is knowledge of a plant procedure, not 1-hour TS per KA; this is RO knowledge (55.41.b.10);

S21B F 3 - - S S22B H 2 - - S S23B F 2 X - - U B and D not credible; S24B F 2 X X U KA on Q doesnt match outline; this is RO knowledge (55.41.b.10);

S25 H 2 X - - E how much of the reference will be provided (and will this be a direct lookup)?

General Comments:

1. How was outline developed? It looks like KA statements were selected to match bank Qs.

2. There were no question histories provided (no indication of when any of the Bank Qs might have appeared on previous NRC exams).

3. 7 repeats from last two NRC exams on RO exam (2 on SRO) - non of which identified by licensee.

NOTE: Bank questions are indicated by B; Modified are indicated by M; comment resolution is indicated in blue; 34 of 34 NUREG-1021, Draft Revision 9