ML16340B082
| ML16340B082 | |
| Person / Time | |
|---|---|
| Site: | Cook  |
| Issue date: | 12/02/2016 |
| From: | NRC/RGN-III |
| To: | Indiana Michigan Power Co, American Electric Power |
| Shared Package | |
| ML15274A380 | List: |
| References | |
| Download: ML16340B082 (9) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 1 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1$
H 3
N S
LIC: Reworded Sample Question stem and distractors to result in only one correct answer.
NRC: Question is SAT.
2+
H 3
B S
2014 NRC Exam 3
F 2
B E
S 2006 NRC Exam LIC: Added Initial Condition to clarify stem.
NRC: Question is SAT.
4 H
2 B
S 5+
F 2
B S
2007 NRC Exam 6$
H 3
N S
LIC: Adjusted Sample Question SI and RHR pump flowrates to match conditions.
NRC: Question is SAT.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1(easy) to 5(difficult); (questions with a difficulty between 2 and 4 are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).
T/F: The answer choices are a collection of unrelated true/false statements.
Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
Partial: One or more distractor(s) is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- / Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
Backward: The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Enter question source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7.
Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any U Status ratings (e.g., how the Appendix B psychometric attributes are not being met).
A + in the Q# column indicates that question was reviewed as part of the representative sample of 30 questions (K/A match & references I.V.).
A $ in the Q# column indicates that the question was a Sample Test Item; not counted in the 20% threshold for acceptable submittal quality.
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 2 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 7
H 2
B S
Previous NRC Exam? - NRC: Why is distractor C plausible?
LIC: Units may procedurally cross-tie ECCS for SI and Charging.
NRC: Question is SAT.
8+
H 2
N B
U S
2002 NRC Exam - NRC: Doesnt match the K/A - reason for the action taken, not what is the action.
LIC: Reworded reason on correct answer to match K/A.
NRC: Question is SAT.
9 F
3 M
S Modified? 2002 NRC Exam (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
10 H
3 B
S 11+
F 3
B S
2010 NRC Exam - NRC: What is the SI pump shutoff head?
LIC: 1560 psig.
12 H
3 M
S 2006 NRC Exam (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
13 F
2 N
B U
S 1 of 2 twice 2008 NRC Exam - NRC: 1) Assume no operator action? Under what conditions would distractor C also be a correct answer? 2) The K/A states that the applicant must determine a loss of DC has occurred, and then verify proper actions; stem states that a loss has occurred?
LIC: Replaced with a Bank question to match K/A.
NRC: Question is Modified: Question is SAT.
14+
H 2
M E
S Previous NRC Exam? (Modified) - NRC: 1) Why is distractor D plausible? (Dont understand the explanation for being incorrect.)
- 2) Need to review original question to verify M vs B source.
LIC: Reworded distractor D and provided original question.
NRC: Question is Modified: Question is SAT.
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 3 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 15 F
2 B
S 16 F
2 M
S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
17+
F 2
N N
U S
NRC: 1) How does this question tie to the EOPs in order to match the K/A?
LIC: Reworded question to tie to EOPs and match K/A.
NRC: Question is SAT.
18 F
3 B
E S
2008 NRC Exam - NRC: Why is distractor A plausible?
LIC: Reworded distractor A.
NRC: Question is SAT.
19 F
3 B
S 2008 NRC Exam 20+
F 3
B S
1 of 2 twice 21 F
3 B
S 2006 NRC Exam 22 F
2 B
S 2002 NRC Exam 23+
F 3
M S
(Modified) - NRC: 1) Doesnt appear to match the K/A for knowledge of the reasons for the steps called for in the fire plan. 2) Need to review original question to verify M vs B source.
LIC: Tied question to Generic K/A for Unit differences.
NRC: Question is Modified: Question is SAT.
24 H
3 B
S 2004 NRC Exam 25 H
3 B
S 2006 & 2010 NRC Exams 26+
H 3
N S
1 of 2 twice - NRC: Why only ask for indication on #11 SG?
LIC: To ensure correct answer is selected for the right reason.
NRC: Question is SAT.
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 4 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 27$
F 3
B S
2004 & 2012 NRC Exams - NRC: Is the applicant required to have this information memorized?
LIC: Provided learning objective.
NRC: Question is SAT.
28 H
3 B
S 29$
H 3
N S
NRC: Is the applicant required to have this information memorized?
LIC: Provided learning objective.
NRC: Question is SAT.
30+
F 2
B S
31 H
3 X
B U
S 2002 NRC Exam - NRC: The distractors, as written may be a collection of T/F conditions, which lead to the possibly correct answer without reading the stem.
LIC: Reworded distractors C & D to correct T/F collections.
NRC: Changed to 1 of 2 twice format: Question is SAT.
32 F
3 B
S 2004 NRC Exam 33+
H 3
B S
34 H
3 B
S 1 of 2 twice - 2007 NRC Exam 35 F
2 B
S 1 of 2 twice 36+
H 2
B S
2008 NRC Exam 37 H
3 N
S 38 H
2 B
S 2010 NRC Exam 39+
F H
2 B
E S
LIC: Agreed; this is higher cognitive question.
NRC: Question is SAT.
40 H
3 B
S 1 of 2 twice 41 H
3 B
S 2006 NRC Exam
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 5 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 42+
H 3
B S
2007 NRC Exam 43 H
2 M
B S
Modified? 2002 NRC Exam (Modified) - NRC: Need to review original question to verify M vs B source, LIC: Provided original question. Not significantly modified to change answer.
NRC: Question is Bank: Question is SAT.
44 H
3 N
S 45+
H 3
B S
1 of 2 twice 2010 NRC Exam 46$
H 3
N S
LIC: Reworded Sample Question distractors to match the K/A.
NRC: Question is SAT.
47$
H 3
N S
LIC: Reworded Sample Question stem and distractors to match the initial conditions.
NRC: Question is SAT.
48 H
3 N
B U
S 2006 NRC Exam - NRC: Doesnt appear to match the K/A to use procedures to correct, control, and mitigate the predicted impact.
LIC: Added a required action to match K/A.
NRC: Question is SAT.
49 H
3 N
S 50+
H 3
B S
51 H
3 M
S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
52 F
3 M
S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 6 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 53+
H 3
M S
(Modified) - NRC: 1) Need to review original question to verify M vs B source. 2) Is this RO knowledge for Tech Specs?
LIC: Provided learning objective. Provided original question.
NRC: Question is Modified: Question is SAT.
54 H
3 M
S (Modified) - NRC: 1) Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
55 H
3 B
S 56+
F 3
B S
57 F
3 B
S 58 F
2 B
S 1 of 2 twice 59+
H 2
B S
60 F
3 B
S 61 H
3 B
S 62+
H 3
B S
63 H
3 M
S Handout provided (Steam Tables)
(Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
64 F
3 B
S 65+
H 3
M S
(Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Original question mistakenly submitted; modified question provided for review and approval.
NRC: Used Modified Question: Question is SAT.
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 7 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 66 H
3 B
S 67 F
3 B
S 2007 NRC Exam 68+
F 3
B S
69 F
2 B
S 70 F
2 B
S 71 H
3 B
S 2012 NRC Exam 72$
H 3
B S
LIC: Reworded Sample Question stem and distractors to match K/A.
NRC: Question is SAT.
73+
F 3
B S
2008 NRC Exam 74 H
3 B
S 1 of 2 twice 75 F
3 B
S NRC: Swapped Distractors C and D.
LIC: Agreed.
NRC: Question is SAT.
For RO portion of Exam: New = 10Q, Modified = 11Q, Bank = 54Q (72%): LOK: 30/75 = 40%, H= 60%
76 H
3 X
B E
S NRC: Reword Stem to direct procedure selection.
LIC: Reworded stem.
NRC: Question is SAT.
77$
H 3
X N
S NRC: Not good K/A match to SI pump indication.
LIC: Reworded required actions.
NRC: Question is SAT.
78 H
3 X
M S
(Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
79 H
3 X
B S
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 8 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 80+
H 3
X B
S 2007 NRC Exam 81 F
3 X
B S
1 of 2 twice 82 H
3 X
B S
83+
H 3
X N
S 84 H
3 X
B S
Handout provided 2010 NRC Exam (Modified) - NRC: Question is a Bank - SAT.
85 H
3 X
B S
Handout provided 86+
H 3
X B
S 2004 & 2010 NRC Exams 87$
H 3
X N
S LIC: Reworded Sample Question stem and distractors to match K/A.
NRC: Question is SAT.
88 H
3 X
N S
89 H
3 X
M S
(Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
90+
F 3
X B
S 91 H
3 X
M S
Modified 2014-99 NRC Exam (Modified) - NRC: 1) Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
92$
H 3
X B
S 2006 NRC Exam - NRC: Question is SAT as submitted.
93+
H 3
X B
S 94 H
3 X
B S
2012 NRC Exam 95 F
2 X
N S
96+
F 3
X B
S 1 of 2 twice
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam 9 of 9 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source B/M/N
- 7.
Status U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 97 H
3 X
B S
Handout provided 98 H
3 X
B S
2004 NRC Exam (Modified)
LIC: Clarified wording in the stem.
NRC: Question is SAT.
99+
H 3
X N
S Handout provided 100 H
3 X
B S
For SRO portion of Exam: New = 6Q, Modified = 3Q, Bank = 16Q (71%): LOK: 4/25 = 16%, H= 84%