ML19044A606
| ML19044A606 | |
| Person / Time | |
|---|---|
| Site: | Millstone  |
| Issue date: | 02/13/2019 |
| From: | Brian Fuller Operations Branch I |
| To: | Dominion Energy Nuclear Connecticut |
| Shared Package | |
| ML 17234A732 | List: |
| References | |
| CAC 000500 | |
| Download: ML19044A606 (7) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 Q
- 1. LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q - K/A SRO Only 1
H 2
X Y
N M U
Not operationally oriented, replaced with new Q - SAT 2
H 3
Y M
E Editorial changes to clean up the Q 3
H 2
Y M
E Editorial changes to clean up the Q 4
F 3
Y N
E Editorial fixes, added IAW EOP 2541, App2 Figure 5 to stem 5
H 3
Y B
S 6
H 4
N M B U
K/A MM, Q concerns loss of L/D not charging and is confusing, replaced with a Bank Q on loss of charging - SAT 7
H 3
Y N
H 2
Y B
S Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3.
Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).
T/F: The answer choices are a collection of unrelated true/false statements.
Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.
Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).
- 4.
Check the appropriate box if a job content flaw is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
- /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
Backward: The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)
- 6.
Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.
- 7.
Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9 Q
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- / Units Backward Q - K/A SRO Only 9
H 3
Y B
E Editorial change 10 F
2 Y
N S
11 H
3 Y
B E
Editorial change 12 F
3 Y
N E
Editorial change 13 F
2 Y
B E
Removed window dressing 14 F
3 N
N B U
K/A MM, replaced Q - SAT 15 F
3 Y
B E
Editorial change 16 H
4 Y
N E
Editorial change 17 H
3 Y
B E
Confirmed with the utility that this is an RO level Q.
Editorial change 18 F
3 Y
B E
Editorial change 19 H
3 Y
N S
20 H
3 Y
B E
Editorial change 21 F
2 Y
B S
22 H
2 X
Y N
U Multiple implausible distractors, changed conditions in stem and wrote new distractors, changed correct answer due different conditions of stem. - SAT 23 F
3 Y
B S
24 H
4 Y
B E
Editorial change
ES-401 3
Form ES-401-9 Q
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- / Units Backward Q - K/A SRO Only 25 F
3 Y
B E
Editorial change 26 H
2 X
Y B
E B Distractor implausible, reworded A and D. - SAT 27 H
3 Y
B S
28 F
3 Y
B S
29 F
2 Y
N S
30 H
3 Y
B E
Editorial change 31 F
2 Y
B S
32 H
2 Y
N S
33 H
2 Y
B E
Editorial change 34 H
3 Y
B S
35 H
3 Y
B E
Editorial change 36 H
2 Y
B S
37 H
3 Y
B E
Editorial change 38 H
2 Y
M E
Editorial change 39 H
3 Y
N S
40 H
2 Y
M E
Editorial change 41 F
2 Y
B S
Confirmed with utility that this Q is RO level.
Added K/A justification
ES-401 4
Form ES-401-9 Q
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- / Units Backward Q - K/A SRO Only 42 F
2 Y
B S
43 F
2 Y
B E
Editorial change 44 H
3 Y
B E
Editorial change 45 H
3 Y
B E
Editorial change 46 H
2 Y
B E
Editorial change 47 H
2 Y
N E
Editorial change 48 F
3 Y
B S
49 F
2 Y
B E
Editorial change 50 H
3 Y
M S
51 F
2 Y
B E
Editorial change 52 F
2 Y
B S
53 H
3 Y
N E
Editorial change 54 F
2 Y
B S
55 F
3 Y
M S
56 H
3 Y
N S
57 H
3 Y
M S
58 F
2 Y
B S
ES-401 5
Form ES-401-9 Q
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- / Units Backward Q - K/A SRO Only 59 F
2 Y
N E
Editorial change 60 F
3 Y
N E
Editorial change 61 F
2 Y
N S
62 F
3 Y
M E
Editorial change 63 F
2 Y
M S
64 H
2 Y
B E
Editorial change 65 F
2 Y
B E
Editorial change 66 F
2 Y
M E
Editorial change 67 F
3 N
N B U
K/A MM, drew new K/A, new Q selected - SAT 68 H
2 Y
B E
Editorial change 69 F
2 Y
M S
70 F
2 Y
B E
Editorial change 71 F
3 Y
B S
72 H
3 Y
M S
73 F
2 Y
B S
74 H
3 Y
N E
Editorial change 75 F
2 Y
B S
ES-401 6
Form ES-401-9 Q
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- / Units Backward Q - K/A SRO Only 76 H
3 Y
Y N
E Editorial change, Added LTOP TS to justification 77 H
3 Y
Y B
S Added SRO Only justification statement 78 F
3 Y
N Y N
U Q not SRO Only, rewrote Q to include procedure selection 79 F
3 Y
Y N
E Editorial change 80 H
3 Y
Y B
E Editorial change 81 H
3 Y
Y B
E Editorial change 82 H
2 Y
Y M
S 83 H
2 Y
N Y N
U Not SRO only, modified stem to require selection of procedure steps. - SAT 84 H
3 Y
Y B
E Editorial change 85 H
2 Y
Y N
E Editorial change, Changed answer D to increase plausibility 86 H
3 Y
Y B
E Editorial change, changed answer A to increase plausibility 87 H
3 Y
Y N
E Editorial change 88 H
2 Y
Y M
E Editorial change.
89 H
3 Y
Y N
E Editorial change 90 H
3 Y
Y N
E Editorial change 91 F
2 Y
Y B
E Editorial change 92 H
3 Y
Y M
S
ES-401 7
Form ES-401-9 Q
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6. Source (B/M/N)
- 7. Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- / Units Backward Q - K/A SRO Only 93 H
3 Y
Y N
S 94 F
3 Y
Y N
S 95 F
2 Y
Y N
E Editorial change 96 F
2 N Y Y
N M U
K/A MM. Modified a bank Q to replace. -SAT 97 F
2 Y
Y M
E Editorial change 98 H
3 Y
Y B
E Editorial change, Rewrote A and D to improve plausibility 99 F
2 Y
N Y B
U Not SRO only, unable to write SRO level Q to this K/A.
Selected new K/A wrote new Q.
SAT 100 F
3 Y
Y M
S