ML22355A650

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML22355A650
Person / Time
Site:	FitzPatrick
Issue date:	08/30/2021
From:	Brian Fuller Operations Branch I
To:	Kelly D Exelon FitzPatrick
References
EPID: L-2021-OLL-0000
Download: ML22355A650 (1)

Text

ES-401 FitzPatrick 2021 Written Examination Review Worksheet Form ES-401-9

2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

1. LOK 6. Source 7. Status Q LOD 8. Explanation (F/H) Stem Cred. Job- SRO (B/M/N) (U/E/S)

(1-5) Cues T/F Partial Minutia #/Units Backward Q - K/A Focus Dist Link Only 1 H 2 N S 2 F 3 N S 3 H 4 B S Rework the 1st part of options C & D (use info 4 F 2 N E from ST eliminate potential for 2 correct) 5 H 3 X B E Remove ref pic (give valve pos instead) 6 H 4 B S 7 F 2 B E Change level indication options to NPSH 8 H 4 M S Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3. Check the appropriate box if a psychometric flaw is identified:

• Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

• Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

• T/F: The answer choices are a collection of unrelated true/false statements.

• Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

• Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4. Check the appropriate box if a job content flaw is identified:

• Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

• Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

• #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

• Backward: The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)

6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. Job- SRO (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Minutia # / Units Backward Q - K/A Focus Dist Link Only 9 H 2 N S 10 H 3 N E Enhance verbiage to improve clarity 11 F 2 B S 12 F 2 B S 13 H 3 N S 14 F 2 N S 15 H 2 B S 16 H 2 N S 17 F 3 N S 18 H 4 B S 19 H 3 N S 20 H 4 B S 21 H 3 B S 22 F 2 N S 23 F 2 B S 24 F 2 B S Revise plausibility statement and change 25 H 3 N E option A verbiage to match B

ES-401 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. Job- SRO (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Minutia # / Units Backward Q - K/A Focus Dist Link Only 26 F 2 B S 27 F 3 N S 28 H 3 N S 29 F 2 N S 30 H 3 B S 31 H 3 B S 32 F 2 N S 33 H 4 B S 34 F 2 N S 35 H 2 M S 36 H 3 B E Change to 2 double & 2 single options 37 H 2 M E Make column headers look the same 38 H 3 B S 39 F 2 N S 40 F 2 B S 41 H 4 B S 42 H 3 B S

ES-401 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. Job- SRO (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Minutia # / Units Backward Q - K/A Focus Dist Link Only 43 H 3 B E Change to 100% power 44 H 2 N S 45 F 3 B E Change option B to cooling for Recirc Pumps 46 H 2 N S 47 H 3 B S 48 H 2 N S 49 F 2 N S 50 H 3 N S 51 H 2 N S 52 H 2 B S 53 H 3 N S 54 F 3 M S 55 F 3 N E Fix typo 56 F 3 B S 57 F 2 B S 58 H 3 B S 59 H 2 N S

ES-401 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. Job- SRO (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Minutia # / Units Backward Q - K/A Focus Dist Link Only 60 H 2 B S 61 F 2 B S 62 H 3 M S 63 H 3 M S 64 H 4 M E Eliminate unnecessary words 65 F 3 B S 66 F 3 B S 67 F 2 B S 68 F 2 N S 69 H 2 B S 70 F 2 M S 71 H 2 B S 72 F 2 B S 73 H 2 B S 74 F 2 B S 75 F 2 B S 76 H 3 M S

ES-401 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. Job- SRO (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Minutia # / Units Backward Q - K/A Focus Dist Link Only 77 H 2 B S 78 H 2 B E Add 100% power, refueling preparations 79 H 3 B S 80 H 2 N S 81 H 2 N S 82 F 3 N S 83 H 2 N E Combine 2nd & 3rd columns into one 84 H 2 N S 85 H 2 B S 86 H 2 N E Add SRO level justification 87 F 2 N S 88 H 2 M S 89 H 2 B S 90 H 2 X N U Changed to give voltage values vs. unsat 91 F 2 N S 92 H 4 M S 93 H 3 N S

ES-401 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other

6. Source 7. Status Q LOK LOD 8. Explanation Stem Cred. Job- SRO (B/M/N) (U/E/S)

(F/H) (1-5) Cues T/F Partial Minutia # / Units Backward Q - K/A Focus Dist Link Only 94 H 2 N S 95 F 2 N S 96 H 3 X X N U Rewrite to match K/A. Question replaced 97 F 2 N S 98 H 2 B S 99 H 3 B S 100 F 2 N S