ML20303A144
| ML20303A144 | |
| Person / Time | |
|---|---|
| Site: | Millstone  |
| Issue date: | 09/18/2020 |
| From: | Dominion Nuclear Connecticut |
| To: | NRC Region 1 |
| JAckson, D. | |
| Shared Package | |
| ML19105A205 | List: |
| References | |
| CAC 000500 | |
| Download: ML20303A144 (7) | |
Text
ES-401 Millstone Unit 2 Written Examination Review Worksheet Form ES-401-9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
F?
2 X
N E
Remove most and replace with an objective term.
2 H
3 X
M E
Comma needed for clarity in first sentence after lift.
3 F
3 X
B N
E Add alarm in stem to meet KA. B not plausible - why add water.
4 F
2 X
X N
U Fix stem to address T/F situation. Replace will with is required in A&D and cannot be in B&C. Q was rewritten.
5 F
2 X
N E
Reword part 1 Q. Fix column heading with 1 & 2..
6 H
3 N
S 7
H 3
B S
8 F
2 C
N E
Reword Q for clarity and add that a trip signal was generated..
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 9
H 2
N S
10 H
2 X
N E
Specify SG1/SG2 in choices.
11 H
3 B
S 12 F
1 N
U Low LOD. Reselect KA. New Q was provided.
13 H
3 N
S 14 H
3 X
N E
Clarify that VA-20 lost power and shorten choice D.
15 H
3 X
B S
Remove () from choices.
16 H
2 X
N E
Insert that in part 1.
17 F
2 N
S 18 H?
2 X
M E
Delete 2nd column and provide KA justification.
19 F?
2 X
N E
Remove best and reword to make objective.
20 H
1 X
X X
N U
Low LOD. Backward logic. Distractors are not plausible. Rewrite Q.
21 F?
2 X
N E
Add info that CNMT purge is progress.
22 F
2 C
N E
Change C to gpd vs gph.
23 F
2 X
X N
E Eliminate first two lines. Reword Q and switch columns.
24 F?
2 X
X N
E Reword Q for clarity. State plant condition first.
25 H
3 M
S 26 F
2 X
M E
Choice C - start all remaining charging pumps.
27 F?
2 N
S 28 F
1 X
A,D N
U Low LOD. B is leading. A&D implausible. Reselected KA.
ES-401 3
Form ES-401-9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 29 H
3 X
X N
E Reword Q to present malfunction and then ask for impact.
30 H
2 X
M U
KAMM No actions is not using procedures to mitigate. Q rewritten.
31 H
2 X
X X
B E
Add if any to avoid cue.
32 H
3 M
S Explain justifications for C & D.
33 H?
3 N
S 34 F
2 N
S 35 H
2 N
S 36 F
3 N
S 37 H
2 N
S 38 H
2 X
M E
Eliminate spray column.
39 F
4 N
S 40 H
3 B
S 41 H
3 X
X N
E Reword Q for clarity and ease of reading.
42 H
2 X
N E
Typo - future vs futures.
43 H
2 M
S 44 H
3 N
S 45 H
4 X
X M
U KAMM. Subsets A&B and C&D. Reselect KA.
46 F
3 X
M E
Ensure C is wrong. D doesnt seem plausible. Modify Q for clarity.
47 H
3 X
M E
Reword Q to elicit proper response.
48 F
3 X
N E
Add regarding DC buses and loads to Q.
444
ES-401 4
Form ES-401-9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 49 H?
2 B
S 50 H
2 X
B M
E Provide power levels to make B plausible.
51 F?
1 M
U LOD 1. GET type Q. Reselected KA.
52 H
2 N
S 53 F
2 N
S 54 F
2 X
N E
Do not provide info about SOVs. Reword Q to assess loss of DC.
55 F
3 X
M E
Make a statement about TS being followed.
56 H?
2 X
M E
Add procedure to stem.
57 H
2 M
S 58 H
2 X
A N
E Best should be avoided. Change choice A.
59 F
2 X
B M
E Replace A or B with borated water source. Use preference in stem.
60 F
2 X
N E
Insert a lead-in line for Q.
61 H
3 B
S 62 H
3 X
N E
Remove could. Replace with will to make Q more definitive.
63 F
2 B
S 64 F?
3 X
N E
Add question marks.
65 H
3 A/C B
E Put Unit 3 in one other choice.(C?)
66 H?
2 X
X N
E Modify Q to be applicable for RO quals.
67 F
4 X
B E
Delete first line and ask about TS staffing requirements.
68 F?
3 A
B E
Reword A2 to avoid subset issue with B.
696CC
ES-401 5
Form ES-401-9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 69 H
3 X
B?
B E
Ask per procedure. Verify that B is incorrect.
70 H?
3 N
S 71 F
2 X
B E
Reword to include operator activity in CNMT.
72 F
2 B
S 73 F?
2 S
B S
74 F
2 N
S 75 H
2 X
M E
Make Q singular vs plural. Provide alarm in stem to meet KA.
N = 43; M = 18; B = 14 NRC :F = 34 (45.3%), H = 41 (54.7%); Licensee F = 31 (41.3%),
H = 44 (58.7%). F?/H? indicates disagreement w/ licensee.
76 H
2 X
X N
U Answered by systems knowledge. KA reselected. Q replaced.
77 H
3 X
N E
Add per procedure or per TS to end of choices.
78 H
2 X
A N
E Reword Q. Make A & C symmetrical.
79 H
2 X
X N
U Answered by system knowledge. Reword to select and take action.
80 H
3 A,C X
N U
KAMM - Q does not test instrument bus. Q was rewritten.
81 H
3 B
S Previous NRC exam Q.
82 H
2 B
S 83 H
2 N
S 84 H
3 X
N E
Reword stem.
85 H
2 X
M E
What are vs Describe. Add procedures to A&C.
86 H
2 X
N E
Add ONLY to B and D.
ES-401 6
Form ES-401-9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 87 H
3 N
S 88 H
2 X
B E
Add per TS to stem.
89 H
3 X
B E
Specify procedural location. Faulty 2x2. Just ask when.
90 H
3 X
A N
E Reword Q wrt RCS temperature limits. A not plausible. Clarify B.
91 H
2 X
X N
U Systems knowledge and AOP entry conditions. Reword to select procedure and take actions.
92 H?
2 N
S 93 H
3 X
N E
Typo - delete first in.
94 H
2 X
B E
Faulty 2x2. Ask for only the first part.
95 F
3 X
N E
Remove phrase about TS bases from stem.
96 F
2 B
E Same knowledge was tested on Q90. Q as written was acceptable.
Replace Q to avoid overlap with Q 90.
97 F?
3 X
N U
Q not SRO level.
98 F
2 D
N E
Make D be DSEO.
99 H
2 X
N E
Add worsened. Use a valve for calculated leak rate. SG not faulted.
100 F?
2 X
N E
Rework Q to be more discriminating.
+ - From previous two exams
- met the criteria for a modified question
- - did not meet the criteria for a modified question RO - 6/75 UNSAT = 8.0%
SRO - 5/25 UNSAT = 20.0%