ML103090619

Nine Mile Point, Unit 2 - NRC Comments on Draft Licensee-Developed Exam (Written & Operating Test) (Folder 2)

ML103090619
Person / Time
Site:	Nine Mile Point
Issue date:	10/08/2010
From:	Operations Branch I
To:
Hansell S
Shared Package
ML101900599	List: IR 05000410/2010302 ML102140374 ML103090619 ML103090636 ML103090647 ML103120049 ML103640250 ML103640338 ML103640340 ML103640349 ML110310468
References
TAC U01790
Download: ML103090619 (8)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (FtH) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #/ Back-Q= SRO B/M/N UtEtS Explanation Focus Dis!. Link units ward KIA Only 1 F 3 Y N S 2 H 3 Y N S 3 F 2 Y N S 4 F 3 Y N S 5 F 4 Y N S 6 H 3 Y N S 7 H 2 Y N S 8 H 2 Y N S Instructions

[Refer to Section 0 of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 -5 (easy -difficult) rating scale (questions in the 2 -4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

• The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• The stem or distractors contain cues (I.e., clues, specific determiners, phrasing, length, etc).

• The answer choices are a collection of unrelated true/false statements.

• The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

• One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

• The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

• The question requires the recall of knowledge that is too specific for the closed reference test mode (I.e., it is not required to be known from memory).

• The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer's judgment, is the question as written (U}nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 ---.-........ 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (FtH) (1-5) Stem Cues TtF Credo Partial Job-Minutia #t Back-Q= SRO BtMtN UtEtS Explanation Focus Dis!. Link units ward KIA Only 9 H 3 Y B S 10 F 3 Y N S 11 H 3 Y B S 12 H 4 Y N S 13 F 2 Y N S 14 H 2 Y N S 15 H 3 Y B S 16 H 3 Y N S 17 H 4 Y N S 18 H 3 Y N S 19 H 4 Y N S 20 H 4 Y N S 21 F 4 Y N S 22 F 3 Y N S 23 H 4 Y N S 24 H 3 Y N S 25 H 3 Y N S 26 F 2 Y N S 27 F 3 Y N S 28 H 3 Y N S ES-401 3 Form ES-401-9 -........... 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LOD (F/H) (1-5) 8tem Cues T/F Credo Partial Job-Minutia #1 Back-Q= 8RO B/M/N U1E/8 Explanation Focus Dis!. Link units ward KIA Only 29 H 2 Y N 8 30 H 2 Y N 8 31 F 3 Y N 8 32 H 3 Y N 8 33 H 2 Y N 8 34 H 2 Y N 8 35 F 2 Y N 8 36 F 3 Y N 8 37 F 4 Y N 8 38 F 4 Y N 8 39 H 3 Y N 8 40 H 2 X X Y N U Minutia; not operationally valid. Better for GFE. Replaced KIA and Q. 41 H 2 Y B 8 42 F 2 Y N 8 43 F 3 Y N 8 44 H 3 Y N 8 45 F 2 Y B 8 46 F 2 Y B 8 I 47 F 3 Y B 8 I 48 H 3 Y N 8 ES-401 4 Form ES-401-9 Q# 1. LOK (F/H) 2. LOD (1-5) -3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. B/M/N 7. UlE/S 8. Explanation Stem Focus -Cues T/F Credo Dis!. Partial Job-Link Minutia #1 units ward Q= KIA SRO Only 49 H 3 Y B S 50 H 2 Y B S 51 H 2 Y B S 52 H 2 Y B S . 53 H 3 Y N S 54 F 3 Y B S 55 F 2 Y N S 56 H 2 Y N S 57 H 2 Y N S 58 H 3 Y B S 59 H 3 Y B S 60 F 3 Y N S 61 F 2 Y N S 62 H 2 Y N S 63 H 2 X Y B U A&B implausible, replaced KIA and Q. 64 F 2 Y M S 65 F 3 Y B S LOCA will put water into OW, which dumps to SP via downcomer 66 F 2 Y N S 67 F 2 Y N S 68 H 3 Y B S ES-401 5 Form ES-401-9 Q# 1 LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. B/M/N 7. UtE/S 8. Explanation Stem Focus Cues T/F Cred. Dis!. Partial Job-Link Minutia #1 units ward Q= KIA SRO Only 69 F 2 Y N S 70 H 3 Y B S 71 F 2 Y M S 72 F 3 Y B S 73 F 3 Y N S 74 F 2 Y N S 75 F 2 Y N S 76 H 3 Y Y N S 77 H 3 Y Y N S 78 H 2 Y Y B S 79 H 2 Y Y N S 80 H 2 Y Y N S 81 H 3 Y Y B S 82 H 3 Y Y M S 83 H 2 Y Y B S 84 H 2 Y Y N S 85 F 2 Y Y B S 86 H 2 Y Y N S 87 H 2 Y Y N S 88 H 2 Y Y N S ES-401 6 Form ES-401-9 --..... ....-1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8. Q# LOK LCD (F/H) (1-5) Stem Cues T/F Cred. Partial Job-Minutia #1 Back-Q= SRO B/M/N UtE/S Explanation Focus Dist. Link units ward KIA Only 89 H 2 Y Y N S 90 H 3 Y Y B S 91 H 3 Y Y N S 92 H 3 Y Y N S 93 H 2 Y Y N S 94 F 2 Y Y N S 95 F 2 Y Y N S 96 F 2 Y Y N S 97 H 2 Y Y B S 98 H 2 Y Y N S 99 H 2 Y Y B S 100 F 3 Y Y B S Post-submittal changes to Nine Mile Point Unit 2 NRC exam In-Plant JPMs P-l; changed step 4.0 from critical to 5at/Unsat P-2; minor formatting change P-3; added two cues for evaluator

{information provided in initial conditions}

and minor rewording of two other cues. Simulator JPMs 5-1; Added cue to time compress after step 17; step 11 changed to 5at/Unsat from NA, step 17 changed 3 valves from Pass/Fail to 5at/Unsat 5-2; Added step to verify AOVI0l open and one train GT5 operable, changed step 8 to 5at/Unsat from Pass/Fail (verify), added two cues to indicate two dampers were closed, revised turnover sheet 5-3; Added stopwatch as Tools/Equipment, added "50C has been in service for >1 hour" in initial conditions and turnover sheet, added cue that another operator is monitoring RPV coolant temperatures in step 4,revised step 11 to indicate dual indication vs. green closed indication S-4; minor editorial changes, added cue and role play to step 3, Revised note in step 4, added annunciator to step 5, fixed typo in turnover sheet S-5; minor editorial and formatting changes, removed cue in step 2 (not required), revised turnover sheet to include "Fast loading not required" 5-6; minor editorial and formatting changes, added cue to step 3 and revised cue in step 17, added Sat/Unsat to step 4, revised step 8 Pass/Fail and Sat/Unsat criteria (verify valve position) S-7; minor editorial and formatting changes, added two expected annunciators to instructor note after step 5, added Sat/Unsat to bullet in step 6 S-8; minor editorial and formatting changes, added cue in step 13 (CCS pump not required, flow is normal) Simulator Scenarios #1; minor editorial change, minor change to role play in event #5, added "place sync switch to on" in event #7 #2; minor editorial and formatting changes #3; minor editorial and formatting changes, added examiner note regarding placing SPC in service on page 24

4. RO Admin JPMs COOl; corrected typos in standard, step #3, and terminating cue, added enhancement to initial conditions and initiating cue, corrected standard in step #5 C002; no changes EP; minor formatting changes EC; minor editorial and formatting changes, changed standard from shaft seal to pump packing, changed two Sat!Unsat criteria in step #4 to Pass/Fail (open and uncapped)

• SRO Admin JPMs COOl; added enhancement to initiating cue, revised step #3 TS references and included TS actions C002; added enhancement to initiating cue EC; added procedure step in step #3 EP; added enhancements to initial conditions, corrected typo in step #8 EC; minor formatting change