ML17103A263
| ML17103A263 | |
| Person / Time | |
|---|---|
| Site: | Clinton  |
| Issue date: | 02/09/2017 |
| From: | Chuck Zoia Operations Branch III |
| To: | Exelon Generation Co, Exelon Nuclear |
| Shared Package | |
| ML16308A462 | List: |
| References | |
| 50-461/17-301 | |
| Download: ML17103A263 (10) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 1 F 2 Y N S 2X2 2+ H 2 Y M S 2X2 3 H 2 Y B S 4 F 2 Y B S 2X2 (Effective ly) 5+ H 1 2 Y N U S 2X2 NRC: LOD=1. It's clear that the reducing loads on 1B is required
. Due to the 2X2 format, this results in eliminating 2 distractors.
Licensee comments (below) were agreed to at OV.
LIC: The facility contends that the part 2 of the question is not LOD 1 for the following reasons:
ators answered correctly).
1B3, and 1B4), any one of which could logically provide power to DC MCC 1B Battery Charger.
Instructions
[Refer to Section D of ES- Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content). The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory). The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO
-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
NOTE: .
ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 6 F 3 Y N S 2X2 7 H 3 X Y N U S NRC: The stem says that automatic actions are the reason air is maintained to the selected components, but the se were simply unaffected by the automatic closure of the Ring Header isolation valves. The eliminated system was negatively affected by these valves closing
. Licensee comments were agreed to at OV. LIC: Question meets the KA (knowledge of the reasons for the following responses as they apply to partial or complete loss of IA: Service Air isolations) as written. SA header isolations are designed to automatically isolate to preserve air to critical components in the plant.
8+ F 2 Y N S 9 H 3 Y N S Reference provided: H 3 Y B S 2X2 11 H 2 Y N S 2X2 12+ F 2 Y N S 13+ H 2 Y N S 14 F 2 Y N S 15 H 2 Y B S 16 F 3 Y B E S NRC: Specify why known from memory.
Explanation provided at OV.
LIC: The natural circulation level value is contained in the precautions sections -Shutdown Cooling (SDC) and Fuel Pool Cooling and Assist (FPC and A) and is tied to
-CL-OPS-and N-CL-OPS- 17 F 2 X Y N E S NRC: The "NEITHER" distractor
("D") is implausible.
Explanation provided at OV. LIC: The facility contends that distracter 'D' is plausible because both loops of RHR share a common SDC suction line. If the examinee correctly recalls that LPCI is impacted by the SDC alignment, it is plausible to believe that neither loop will realign into LPCI mode.
18+ H 3 Y B S 19 F 2 Y N S H 3 Y N S ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 21+ H 3 Y N S 2X2 22 H 2 Y B S 23 H 2 Y N S 2X2 24+ F 2 Y N S 2X2 25 F 3 Y N S 2X2 , Reference provided: CPS 31, Figure 1 26+ F 2 Y B S 2X2 27 H 2 Y B S CPS NRC Exam ILT
-1 28 H 3 X Y N E NRC: Using distractor "C" at psig would be more discriminating , because 1165 psig is not associated with the three SRVs
. LIC: -ordered answers/distracters from low pressure to high pressure. Changed answer explanation.
29 H 3 Y N S 2X2 H 3 Y B S CPS NRC Exam ILT
-1 31+ H 3 Y N S 32 H 2 Y B S 33 H 3 Y B S 2X2, CPS NRC Exam ILT
-1 34+ H 3 Y B S 35 H 2 Y N S 2X2 36 H 2 Y B E 2X2 (Effectively)
NRC: "SLIM" acronym needs to be defined.
LIC: Added full noun name for the SLIM controller (Small Loop Interface Module) to the answer and distracter.
37 H 3 Y N S 38+ H 3 Y M S 2X2 (Effectively) 39+ F 3 Y B S CPS NRC Exam ILT 9-1 ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only F 2 Y B S CPS NRC Exam ILT
-1 41 H 2 Y N S 42 H 2 Y N E S NRC: It appears that the procedural aspect of the KA could be addressed; what actions would be required in this situation? Question accepted as
-is after discussion at OV
. LIC: Asking what procedural actions are required to mitigate the loss of DC would invalidate part 1 of the question. Left question as
-is. 43+ F 3 Y N S 44 F 2 Y N S 45 H 3 Y N E S NRC: Does "Fail Point" appear on actual plant indication
? Yes LIC: The meter is identical to the plant.
46+ F 1 2 Y N U S 2X2 NRC: LOD=1. It is obvious lights are dark at 12 o'clock. Due to the 2X2 format, this results in eliminating 2 distractors.
Discussed and agreed to at OV as indicated below
. LIC: The facility contends that the part 1 of the question does not contain implausible distracters. Novice candidates with limited experience paralleling a DG may select that the DG output breaker is closed when the synchronizing lights are bright (lit). In addition, 4 exam validators rated the question as LOD 2.8.
47 F 2 Y M S 2X2 48 F 3 Y N S 2X2. 49 H 3 Y B S 2X2 H 3 Y N S 2X2 51 F 2 Y N S 52 F 2 Y N S 2X2 53+ H 3 Y B S Reference provided: CPS 54 H 2 Y N S 55 H 3 Y N S ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 56 H 2 Y M S 2X2 57+ F 1 Y B U NRC: LOD=1. It is obvious that the velocity limiter is the choice among those given
. Accepted replacement
. LIC: Replaced the question with a different T2G2 RO question deemed suitable by the CE.
58 H 2 Y N S 59 H 3 Y B S F 3 Y N S 61 H 3 Y N S 62 F 1 2 Y B U S 2X2 NRC: LOD=1. It's very clear that rod motion is inhibited.
Due to the 2X2 format, this results in eliminating 2 distractors.
Discussed at OV and considered acceptable.
LIC: The facility contends that the question is not LOD 1. Each control rod contains 2 position channels, only one of which is required to be operable. The examinee may plausibly believe that a single channel failure would not result in a rod block, making A1 and validators).
63+ F 2 Y B S 64 H 2 Y B S -1 65 H 3 Y B S 66 F H X Y B E 2X2 NRC: This would be an "H" question, because the applicant would have to calculate that > 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in a 7 day period would be exceeded. Also, the question would be better if asked what would be permitted, versus what's NOT allowed.
Agreed to cog change.
LIC: Changed the question to high cog.
The facility also contends that the question meets the KA as written and would not be enhanced by asking what is permitted. Covered employees are required to ensure that any overtime worked does not our Limits when accepting overtime to cover shift vacancies.
ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 67+ F 1 2 X Y N U S NRC: Distractor "A" is implausible and the entire question is LOD=1 for an ILT student who has performed this task many times
. Discussed at OV and considered acceptable.
LIC: The facility contends that the question is not LOD1 and that average LOD of 2.5. Additionally, the facility contends that a less than competent examinee may incorrectly believe that the SM can authorize entry into the MCR with no further authorizations required.
68 H 3 Y N S 2X2 69 F 2 Y B S F 2 Y B S 71 F 3 Y B S -1 NRC: Please explain why this is an RO
-level question
. Discussed at OV and considered acceptable.
LIC: principles and procedures. CFR data is already documented in the General Data Table.
72 H 2 X Y B E NRC: Distractor "B" would be more plausible if the work could be . Change incorporated.
LIC: 73 F 3 Y B E -1 NRC: Please provide additional information to support LOD and plausibility of the distractors
. LIC: Included an elevation map of the containment to the answer explanation.
74+ H 2 Y N S 75 H 2 Y N S 76 1 H 3 Y Y N S NRC: -only requires procedure selection versus simply actions. SRO-only task referenced.
LIC: N/A ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 77 2 F 3 X Y Y N E NRC: -only requires procedure selection versus simply actions.
Explanation for distractor 'A' LIC: Modified distracter A to enhance the plausibility. Modified the "SRO only justification" to explain how the question constitutes a "selection of procedures".
78+ 3 H 2 Y Y N S NRC: -only requires procedure selection versus simply actions. SRO
-only task referenced.
LIC: N/A 79 4 H 2 Y Y B S 80 5 H 3 Y Y B S 2X2, CPS ILT 14
-1 NRC Exam 81 6 H 3 Y Y N S 2X2 NRC: (5) for SRO
-only requires procedure selection versus simply actions. SRO
-only task referenced for FH component.
LIC: N/A 82+ 7 F 2 Y Y N S 2X2 83 8 F 2 Y Y B E NRC: -only requires procedure selection versus simply actions.
LIC: Modified the "SRO only justification" to explain how the question constitutes a "selection of procedures".
84 9 F H 3 Y Y B E NRC: Higher Cog question because it requires figuring out where you are and where you would be after a LOCA, etc. before applying the TS Bases information.
LIC: Incorporated change to cognitive level as suggested.
85+ F 2 Y Y N E 2X2 NRC: -only requires procedure selection versus simply actions.
Also, the KA requires the ability to perform procedures versus recalling the actions performed by them.
LIC: 86 11 H 2 Y Y M S ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 87 12 H 1 2 Y Y N U S 2X2 NRC: The SRO-only part of the question is LOD=1, since the choice is limited to whether redundant required features are operable or not. This, in turn, eliminates 2 distractors.
LIC: The facility contends that the question is not LOD1, because the examinee has to know that the ITS bases considers Div 3 DG redundant to Div 1 and 2 for the purposes of applying ITS 3.8.1 B.2.
This is not normally the case. The validators rated this question as LOD 3. 88 13 H 2 Y Y N E U 2X2 NRC: Replacing one of the "OPEN direction ONLY" with a new choice, "CLOSED direction ONLY," would enhance the discriminatory value of the question.
LIC: Resolved by changing A.1 and B.1 to "closed direction only". Revised the answer explanation accordingly.
Additional Issue
- Replaced "bypassed" with "inserted" in the stem (there were NO correct answers before this correction was made and communicated during administration of the written exam to ALL SRO applicants
). 89+ 14 H 3 Y Y N E 2X2 NRC: for SRO-only requires procedure selection versus simply actions.
LIC: Modified the "SRO only justification" to explain how the question constitutes a "selection of procedures".
90 15 F 2 Y Y B S 2X2, CPS ILT 1 2-1 NRC Exam 91 16 H 3 Y Y M E NRC: If distractor "D" was changed to "Scenarios 1 & 2," the choices would be more balanced
- the correct scenario would only appear in 2 of the choices.
LIC: Changed Distracter D to scenarios 1 & 2 as requested. Adjusted answer explanation accordingly.
ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017)
Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 92 17 H 1 2 Y Y N U S 2X2 NRC: LOD=1. interlocks are INOP, it is more obvious that the One
-Rod-Out interlock would not be affected by any of the stem conditions.
LIC: This question validated 2.8. The validator that responded incorrectly chose D (refueling interlocks and refuel position one
-rod-out interlock) were both inoperable. The validators, FR and FA believe this is a fair question for a novice candidate with no refueling experience.
93+ 18 H 2 Y Y N S 2X2 94 19 H 3 Y Y B S 95 H 3 Y Y B S 2X2 96+ 21+ F 2 Y Y B S 97 22 F 2 Y Y N S 98 23 F 2 Y Y N S 2X2, SRO-only task referenced.
99 24 F 3 Y Y N S Reference provided:
EP-AA-112-F01 References are contained in the development notebook for questions 24 and 25.
References were reviewed by the CE during prep week activities.
100+ 25 H 2 Y Y B S Reference provided:
EP-AA-1003 Barrier Matrix References are contained in the development notebook for questions 24 and 25. References were reviewed by the CE during prep week activities.
SRO-only task referenced
. Questions from Previous NRC Exam: New Q's: 55 RO LOK: F - ; H - 45 ES-401 Written Examination Review Worksheet Form ES-401-9 Clinton Initial License Exam (2017) 2x2 questions:
39 Modified Q's:
6 SRO LOK: F - 9; H - 16 Unsat questions:
39