ML21193A312

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML21193A312
Person / Time
Site:	Calvert Cliffs
Issue date:	06/16/2021
From:	NRC Region 1
To:	Exelon Generation Co
Shared Package
ML20261H327	List: ML21187A307 ML21187A308 ML21187A309 ML21187A310 ML21189A288 ML21193A298 ML21193A307 ML21193A312 ML21193A316 ML21193A321 ML21193A369 ML21202A164 ML22073A117
References
EPID L-2021-OLL-0028
Download: ML21193A312 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q LOK LOD Source Status 8. Explanation (F/H) (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward Focus Dist Link K/A Only 1 H 2 B E Added approximately to distractors 2 H 2 N E Added full 3 H 2 B E Added noun name to EOP 5 Removed lowest temperature in stem and removed caps on Bleedoff. added correct AOP implemented 4 F 2 B E added words to D explanation 5 H 3 B S 6 H 3 B E Overlaps with question 29. Rewrote question.

7 H 3 B S Question is not significantly modified. Will be called a bank 8 H 3 B S question Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3. Check the appropriate box if a psychometric flaw is identified:

• Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

• T/F: The answer choices are a collection of unrelated true/false statements.

• Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

• Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4. Check the appropriate box if a job content flaw is identified:

• Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

• #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• Backward: The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)

6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9 1.

LO 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q K LOD Source Status 8. Explanation (F/ (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward H) Focus Dist Link K/A Only Changed distractor D to different EOP 9 H 3 N E Changed EOP-8 to EOP updated plausibilities 10 F 2 N E Added safety related to description of busses 11 H 3 N S 12 F 3 N S Question contains a none of the above distractor in D.

13 H 2 B E Rewrote distractors.

D stands out being the only Unit 2 component. Changed 14 F 3 N E Distractor D to 11.

15 F 2 B S 16 H 3 B S 17 F 3 B S 18 F 2 B S 19 H 3 B E Added noun name to EOPs 20 H 2 N S 21 H 3 B S 22 F 3 M S 23 H 2 N E Reworded stem to say termination criteria 24 F 2 N S 25 F 3 B S

ES-401 3 Form ES-401-9 1.

LO 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q K LOD Source Status 8. Explanation (F/ (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward H) Focus Dist Link K/A Only A2 and C2 implausible. Made distractors more plausible:

changed alpha to neutron 26 F 3 X N U removed RCS leak and added in CRUD burst 27 H 3 B S 28 H 3 N S 29 H 2 B S Overlaps with question 6. Question 6 was rewritten 30 H 3 B E Edited stem to cooldown to Mode 5 31 H 3 N E Removed stops opening from all 4 distractors 32 F 2 N S 33 H 3 B S 34 F 3 N S 35 H 2 B S 36 H 3 B E Subscripted Pvar 37 H 3 B E Added RPS to pretrips to all distractors 38 H 2 B S 39 H 3 B S 40 H 3 B S 41 H 2 N S

ES-401 4 Form ES-401-9 1.

LO 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q K LOD Source Status 8. Explanation (F/ (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward H) Focus Dist Link K/A Only 42 H 3 B S 43 H 2 B E Added space in 950 psia 44 F 3 B S 45 H 3 N S 46 H 2 N S 47 F 2 N S 48 F 3 B S Rewrote in a format more like 2x2 49 H 2 B E Capitalize Mode 50 F 3 B S 51 H 3 M E Change distractor C to shifts to condenser:

52 F 2 B S Small editorial change to spacing in stem - removed a space 53 F 3 B E in D Change B2 and D2 to Fire Main with/without hoses 54 F 2 B E changed distractor to DI water vice condensate 55 F 3 N S 56 H 3 N S 57 F 3 B S 58 F 3 B E Made all Thot and Tcolds subscripts

ES-401 5 Form ES-401-9 1.

LO 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q K LOD Source Status 8. Explanation (F/ (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward H) Focus Dist Link K/A Only 59 H 3 B S 60 F 3 N E Added per AOP 6F to stem 61 H 2 B S 62 H 3 N S 63 F 3 N E changed 20% to 8%

64 F 3 B S 65 H 3 B E Added 10 inches below setpoint and continuing to lower 66 F 3 B S 67 H 3 N S 68 F 3 B S 69 F 2 B S 70 F 3 B S 71 F 3 B S 72 F 1 B U LOD1. Question was rewritten Too easy LOD1. Rewrote question to be more operationally 73 F 1 N U valid and challenging to test continuous step concept.

74 F 2 B S 75 F 3 N S

ES-401 6 Form ES-401-9 1.

LO 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q K LOD Source Status 8. Explanation (F/ (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward H) Focus Dist Link K/A Only 76 H 3 B S 77 F 3 B S 78 H 3 N S added will be directed by the Unit Supervisor, SRO 79 H 2 B E Justification 80 H 3 N S 81 H 3 N S 82 H 3 N S Changed stem wording, added SRO justification to 83 F 2 N E strengthen question to be SRO level 84 F 3 B S 85 H 3 N U B2 and C2 implausible. Picked a new distractor from EOP-8 86 H 3 B S K/A mismatch, this tests reportability as written. Question 87 H 2 X B U rewritten - Made question a 2x2 with actions per procedure to correct issue 88 H 3 N S Not SRO level as written. added SRW/SW pressures to the 89 H 3 X N U stem changed distractor to selecting the action in the appropriate AOP 90 F 2 B E Edited stem to remove whether or not statements are true 91 F 3 B E changed to will the Unit Supervisor direct

ES-401 7 Form ES-401-9 1.

LO 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.

Q K LOD Source Status 8. Explanation (F/ (1-5) Stem Cred. Job- Q- SRO (B/M/N) (U/E/S)

Cues T/F Partial Minutia #/Units Backward H) Focus Dist Link K/A Only 92 H 3 N S 93 H 3 B S 94 F 2 B S 95 F 2 N S 96 F 3 B S 97 H 3 B S 98 F 2 N E Edited stem to remove following groups Edited stem to ensure that this is a transition to an EOP (SRO 99 F 3 B E knowledge) 100 F 3 X B U K/A mismatch. Edited question and selected new K/A.