ML16340B082
| ML16340B082 | |
| Person / Time | |
|---|---|
| Site: | Cook  |
| Issue date: | 12/02/2016 |
| From: | NRC/RGN-III |
| To: | Indiana Michigan Power Co, American Electric Power |
| Shared Package | |
| ML15274A380 | List: |
| References | |
| Download: ML16340B082 (9) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status Explanation (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Focus Dist. Link units ward K/A Only 1$ H 3 N S LIC: Reworded Sample Question stem and distractors to result in only one correct answer.
NRC: Question is SAT.
2+ H 3 B S 2014 NRC Exam 3 F 2 B E 2006 NRC Exam LIC: Added Initial Condition to clarify stem.
S NRC: Question is SAT.
4 H 2 B S 5+ F 2 B S 2007 NRC Exam LIC: Adjusted Sample Question SI and RHR pump flowrates to 6$ H 3 N S match conditions.
NRC: Question is SAT.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1(easy) to 5(difficult); (questions with a difficulty between 2 and 4 are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).
- T/F: The answer choices are a collection of unrelated true/false statements.
- Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- Partial: One or more distractor(s) is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- # / Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U Status ratings (e.g., how the Appendix B psychometric attributes are not being met).
A + in the Q# column indicates that question was reviewed as part of the representative sample of 30 questions (K/A match & references I.V.).
A $ in the Q# column indicates that the question was a Sample Test Item; not counted in the 20% threshold for acceptable submittal quality.
1 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Previous NRC Exam? - NRC: Why is distractor C plausible?
7 H 2 B S LIC: Units may procedurally cross-tie ECCS for SI and Charging.
NRC: Question is SAT.
2002 NRC Exam - NRC: Doesnt match the K/A - reason for the U action taken, not what is the action.
8+ H 2 N B S LIC: Reworded reason on correct answer to match K/A.
NRC: Question is SAT.
Modified? 2002 NRC Exam (Modified) - NRC: Need to review original question to verify M vs B source.
9 F 3 M S LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
10 H 3 B S 11+ F 3 B S 2010 NRC Exam - NRC: What is the SI pump shutoff head?
LIC: 1560 psig.
12 H 3 M S 2006 NRC Exam (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
13 F 2 N B U 1 of 2 twice S 2008 NRC Exam - NRC: 1) Assume no operator action? Under what conditions would distractor C also be a correct answer? 2) The K/A states that the applicant must determine a loss of DC has occurred, and then verify proper actions; stem states that a loss has occurred?
LIC: Replaced with a Bank question to match K/A.
NRC: Question is Modified: Question is SAT.
14+ H 2 M E Previous NRC Exam? (Modified) - NRC: 1) Why is distractor D plausible? (Dont understand the explanation for being incorrect.)
S 2) Need to review original question to verify M vs B source.
LIC: Reworded distractor D and provided original question.
NRC: Question is Modified: Question is SAT.
2 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 15 F 2 B S 16 F 2 M S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
17+ F 2 N N U NRC: 1) How does this question tie to the EOPs in order to match the K/A?
S LIC: Reworded question to tie to EOPs and match K/A.
NRC: Question is SAT.
18 F 3 B E 2008 NRC Exam - NRC: Why is distractor A plausible?
S LIC: Reworded distractor A.
NRC: Question is SAT.
19 F 3 B S 2008 NRC Exam 20+ F 3 B S 1 of 2 twice 21 F 3 B S 2006 NRC Exam 22 F 2 B S 2002 NRC Exam 23+ F 3 M S (Modified) - NRC: 1) Doesnt appear to match the K/A for knowledge of the reasons for the steps called for in the fire plan. 2) Need to review original question to verify M vs B source.
LIC: Tied question to Generic K/A for Unit differences.
NRC: Question is Modified: Question is SAT.
24 H 3 B S 2004 NRC Exam 25 H 3 B S 2006 & 2010 NRC Exams 26+ H 3 N S 1 of 2 twice - NRC: Why only ask for indication on #11 SG?
LIC: To ensure correct answer is selected for the right reason.
NRC: Question is SAT.
3 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 27$ 3 B S 2004 & 2012 NRC Exams - NRC: Is the applicant required to have this information memorized?
F LIC: Provided learning objective.
NRC: Question is SAT.
28 H 3 B S 29$ H 3 N S NRC: Is the applicant required to have this information memorized?
LIC: Provided learning objective.
NRC: Question is SAT.
30+ F 2 B S 31 H 3 X B U 2002 NRC Exam - NRC: The distractors, as written may be a collection of T/F conditions, which lead to the possibly correct answer S without reading the stem.
LIC: Reworded distractors C & D to correct T/F collections.
NRC: Changed to 1 of 2 twice format: Question is SAT.
32 F 3 B S 2004 NRC Exam 33+ H 3 B S 34 H 3 B S 1 of 2 twice - 2007 NRC Exam 35 F 2 B S 1 of 2 twice 36+ H 2 B S 2008 NRC Exam 37 H 3 N S 38 H 2 B S 2010 NRC Exam 39+ F 2 B E LIC: Agreed; this is higher cognitive question.
H S NRC: Question is SAT.
40 H 3 B S 1 of 2 twice 41 H 3 B S 2006 NRC Exam 4 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 42+ H 3 B S 2007 NRC Exam 43 H 2 M S Modified? 2002 NRC Exam (Modified) - NRC: Need to review original question to verify M vs B source, B
LIC: Provided original question. Not significantly modified to change answer.
NRC: Question is Bank: Question is SAT.
44 H 3 N S 45+ H 3 B S 1 of 2 twice 2010 NRC Exam 46$ H 3 N S LIC: Reworded Sample Question distractors to match the K/A.
NRC: Question is SAT.
47$ H 3 N S LIC: Reworded Sample Question stem and distractors to match the initial conditions.
NRC: Question is SAT.
48 H 3 N B U 2006 NRC Exam - NRC: Doesnt appear to match the K/A to use procedures to correct, control, and mitigate the predicted impact.
S LIC: Added a required action to match K/A.
NRC: Question is SAT.
49 H 3 N S 50+ H 3 B S 51 H 3 M S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
52 F 3 M S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
5 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 53+ H 3 M S (Modified) - NRC: 1) Need to review original question to verify M vs B source. 2) Is this RO knowledge for Tech Specs?
LIC: Provided learning objective. Provided original question.
NRC: Question is Modified: Question is SAT.
54 H 3 M S (Modified) - NRC: 1) Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
55 H 3 B S 56+ F 3 B S 57 F 3 B S 58 F 2 B S 1 of 2 twice 59+ H 2 B S 60 F 3 B S 61 H 3 B S 62+ H 3 B S 63 H 3 M S Handout provided (Steam Tables)
(Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
64 F 3 B S 65+ H 3 M S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Original question mistakenly submitted; modified question provided for review and approval.
NRC: Used Modified Question: Question is SAT.
6 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 66 H 3 B S 67 F 3 B S 2007 NRC Exam 68+ F 3 B S 69 F 2 B S 70 F 2 B S 71 H 3 B S 2012 NRC Exam 72$ H 3 B S LIC: Reworded Sample Question stem and distractors to match K/A.
NRC: Question is SAT.
73+ F 3 B S 2008 NRC Exam 74 H 3 B S 1 of 2 twice 75 F 3 B S NRC: Swapped Distractors C and D.
LIC: Agreed.
NRC: Question is SAT.
For RO portion of Exam: New = 10Q, Modified = 11Q, Bank = 54Q (72%): LOK: 30/75 = 40%, H= 60%
76 H 3 X B E NRC: Reword Stem to direct procedure selection.
S LIC: Reworded stem.
NRC: Question is SAT.
77$ H 3 X N S NRC: Not good K/A match to SI pump indication.
LIC: Reworded required actions.
NRC: Question is SAT.
78 H 3 X M S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
79 H 3 X B S 7 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 80+ H 3 X B S 2007 NRC Exam 81 F 3 X B S 1 of 2 twice 82 H 3 X B S 83+ H 3 X N S 84 H 3 X B S Handout provided 2010 NRC Exam (Modified) - NRC: Question is a Bank - SAT.
85 H 3 X B S Handout provided 86+ H 3 X B S 2004 & 2010 NRC Exams 87$ H 3 X N S LIC: Reworded Sample Question stem and distractors to match K/A.
NRC: Question is SAT.
88 H 3 X N S 89 H 3 X M S (Modified) - NRC: Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
90+ F 3 X B S 91 H 3 X M S Modified 2014-99 NRC Exam (Modified) - NRC: 1) Need to review original question to verify M vs B source.
LIC: Provided original question.
NRC: Question is Modified: Question is SAT.
92$ H 3 X B S 2006 NRC Exam - NRC: Question is SAT as submitted.
93+ H 3 X B S 94 H 3 X B S 2012 NRC Exam 95 F 2 X N S 96+ F 3 X B S 1 of 2 twice 8 of 9
ES-401 Written Examination Review Worksheet Form ES-401-9 DC Cook July 2016 Exam
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD Source Status (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 97 H 3 X B S Handout provided 98 H 3 X B S 2004 NRC Exam (Modified)
LIC: Clarified wording in the stem.
NRC: Question is SAT.
99+ H 3 X N S Handout provided 100 H 3 X B S For SRO portion of Exam: New = 6Q, Modified = 3Q, Bank = 16Q (71%): LOK: 4/25 = 16%, H= 84%
9 of 9