ML20070H270
| ML20070H270 | |
| Person / Time | |
|---|---|
| Site: | Fermi  |
| Issue date: | 02/27/2020 |
| From: | Chuck Zoia NRC/RGN-III/DRS/OLB |
| To: | DTE Energy, Detroit Edison, Co |
| Zoia C | |
| Shared Package | |
| ML19303A625 | List: |
| References | |
| Download: ML20070H270 (21) | |
Text
ES-401 Facility: Fermi (Retake)/Year: 2020 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 H 2 N S 2X2 2015 NRC Exam NRC: In the pedigree, explain in greater detail the indications of lockouts of buses 64B and 64C based on the plant information provided in the stem of the question.
Response
2 H 3 B E Provided greater detail on bus lockout indication in pedigree. Also, after discussion with NRC, removed 2nd part of question as the second half was not needed to match the K/A or to make the question valid.
NRC: Question SAT with enhancement made.
2012 NRC Exam; 2009 & 2015 LOR Exams U NRC: Question does not match the K/A. Specifically, the K/A is associated with the 3 H 2 X B E ability to determine or interpret the cause of the partial loss of DC. The question requires the applicant to assess the nature of the partial loss and its implications.
Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3. Check the appropriate box if a psychometric flaw is identified:
$ Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length).
$ T/F: The answer choices are a collection of unrelated true/false statements.
$ Cred. Dist>: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
$ #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only. (K/A and license-level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 21
ES-401 Facility: /Year: 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Also, is this a closed reference question? This is not specified.
Response
K/A Match justification added to the question pedigree.
Closed reference, pedigree updated.
Recommend NRC change from U to E.
NRC: Question SAT with enhancement made.
NRC: The question stem should indicate that annunciator 4D13 is alarming. The alarm response procedure indicates that a combination of annunciators 4D9 and 4D13 for bearings 9 and 8 or 10 will result in an automatic turbine trip. In addition to the vibration values provided, the stem should indicate that the logic to initiate annunciator 4D13 has been met.
4 H 3 X B E Also, is this a closed reference question? This is not specified.
Response
Added 4D13 to stem as being in alarm.
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
River Bend 2014 NRC Exam; 2X2 NRC: Is this a closed reference question? This is not specified.
5 F 3 B E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
2012 NRC Exam; 2017 NRC Exam NRC: Is this a closed reference question? This is not specified. Also, distractor A should state RCIC Pump Flow indication ONLY.
Response
In this instance, use of the word ONLY might make the candidate not select this 6 F 2 X B E response (the correct response) because RCIC flow indication is NOT the ONLY indication provided at the Remote Shutdown Panel (valve positions, etc., are also provided). Use of the word ONLY is not necessary for this question because RCIC Pump Flow indication is not included in any of the other distractors.
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
7 H 3 X N E NRC: Distractor D is not credible - there is no Technical Specification action that would direct immediate scramming of the reactor. There are parameters or Page 2 of 21
ES-401 Facility: /Year: 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only conditions which must be addressed immediately but none that direct immediate scram from Mode 1.
Response
New distractor written, and pedigree updated.
NRC: Question SAT with enhancement made.
8 H 3 B S 2X2 9 H 2 B S 2012 NRC Exam 10 H 2 M S 2015 NRC Exam; 2X2 NRC: Explain why 2.25 psig is plausible in the pedigree - are there TS pressure limits/setpoints that it could be confused with?
Response
11 F 2 X B E Could not locate 2.25 psig in TS or TS bases to enhance plausibility.
Changed distractor to 1.50 psig (high drywell pressure alarm setpoint) and updated pedigree information accordingly.
NRC: Question SAT with enhancement made.
12 H 3 B S NRC: Is this a closed reference question? This is not specified.
Response
13 H 2 B E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
NRC: Are applicants expected to know the vortex and NPSH graphs from memory based on containment conditions?
14 H 3 X N E Response:
Revised to identify EOP Sheet 6 provided as reference.
NRC: Question SAT with enhancement made.
2018 NRC Exam; 2X2 NRC: Specify in the pedigree what, if any, references are provided during the exam.
15 H 2 B E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
Page 3 of 21
ES-401 Facility: /Year: 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
16 H 2 N E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
2X2 NRC: Distractors B and D are not plausible - they clearly allow the cooldown to be continued when stem conditions indicate rising reactor power with a cooldown U occurring.
17 H 3 X N S Response:
Question modified. Rather than continue CD at 50°F/hr, the distractor was changed to reduce CD rate to 25°F/hr.
NRC: Question SAT with modifications made.
18 F 2 B S 2009 NRC Exam Preliminary Review NRC: Not all question answer/distractors clearly address the BASES as required in the stem. The proposed correct answer specifically identifies the next actions to be taken, and not entirely on the bases. Therefore, the question has no correct answer.
19 F 2 X N E Eliminating the need to identify what BASES need to be considered first and re-focusing the question to what needs to be done and why would address this flaw.
Licensee:
New version written to address NRC comments.
NRC: New question SAT.
2019 LOR Exam NRC: Is this a closed reference question? This is not specified in the pedigree.
20 H 2 B E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
Embedded reference NRC: Is this a closed reference question? This is not specified in the pedigree.
21 H 3 N E Response:
Closed reference, pedigree updated. - Image in question.
NRC: Question SAT with enhancement made.
Page 4 of 21
ES-401 Facility: /Year: 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 2X2 NRC: This question is like example C.2 in NUREG 1021, App B, Attachment for low discriminatory validity. The question also has confusing wording. Also, is this a closed reference question? This is not specified in the pedigree.
U Response:
22 F 3 X N S Closed reference, pedigree updated.
New version of question written to address NRC concerns.
Note: 569 was chosen by two validators, on the original question version, supporting the plausibility of the use of this number as a distractor.
NRC: New question SAT.
2X2 NRC: The pedigree references information from the TS Bases, which is generally considered to be SRO-Only knowledge. Are there RO knowledge objectives in the training notes or information provided in the UFSAR for the RBM system? Also, is this a closed reference question? This is not specified in the pedigree.
Response
The information necessary to answer this question is located in the Purpose section of the Student Text for the Power Range Monitoring System, ST-OP-315-0024-001, which states:
23 F 2 X N E The purpose of Power Range Neutron Monitoring (PRNM) is to monitor local neutron flux at various locations within the core and provide accurate measurement of average reactor power. PRNM also prevents local fuel damage that may result from a single control rod withdrawal error (Rod Block Monitor).
Objective A001, State the purpose of the Power Range Neutron Monitoring System, is required of ALL Licensed Operators.
Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
24 F 2 N E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
NRC: Is this a closed reference question? This is not specified in the pedigree.
25 H 3 N E Response:
Closed reference, pedigree updated.
Page 5 of 21
ES-401 Facility: /Year: 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: Question SAT with enhancement made.
26 F 2 N S 27 F 2 B S 2017 NRC Exam; 2009, 2015, & 2019 LOR Exams ; 2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
28 H 4 B E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
Preliminary Review NRC: Distractors B and D can be eliminated immediately and have little very plausibility with basic knowledge of TS LCO 3.5.1. Also, this question should be enhanced by raising its discriminatory value. Of the choices given, losing a 2nd RH pump with an RH pump OOS stands out as the correct answer and it appears that the other information was included to enhance the plausibility of the other distractors.
Also, is this a closed reference question? This is not specified in the pedigree.
Licensee:
Alternate question created. Two RHR pumps inoperable in different divisions is not a LCO 3.0.3 entry. Instead it is an entry into action statement LCO 3.5.1.B. This is the starting point of the alternate question. At this point, 4 other systems (ADS, RCIC, SBFW and CS) are made inoperable as the 4 responses, thus avoiding the 2nd RHR pump standout referenced by the chief examiner.
This is judged to still be a K/A tie as entry into LCO 3.5.1.K is not due solely to the CS 29 F 3 X N E pump inoperability, but due to the simultaneous inoperability of CS Pump A along with RHR/LPCI pumps A and B.
RCIC (distractor B originally, distractor C as modified) inoperability is addressed in TS 3.5.1, as it is required by LCO 3.5.1.E, when HPCI is inoperable, to immediately verify RCIC operability. If that verification fails, (due to RCIC inoperability) then entry into LCO 3.5.1.I would be immediately required. This would establish a 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> timeclock to be in mode 3, which is an action very similar to LCO 3.0.3 requirements SBFW (distractor D) is a TRM system and is an important defense in depth system for the station blackout procedure (20.300.SBO) and in the dedicated shutdown procedure (20.000.18.) It thus requires licensed operator level knowledge to properly reject that response, in keeping with the KA of recognize system parameters that are entry-level conditions for Technical Specifications.
This question is not LOD 1. TS 3.5.1 has 11 different LCO action statements and is not provided as a reference on the exam. There are 14 day, 7 day, 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />, 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> and LCO 3.0.3 action statements. To ask, from memory, a detailed analysis of TS Page 6 of 21
ES-401 Facility: /Year: 7 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 3.5.1 actions at the RO level was judged by the validators to be LOD 4, with some discussion about whether it was instead really LOD 5.
NRC: Question SAT with enhancements made.
NRC: 1E1150-F017A is not described in the stem and therefore serves as a cue making distractors C and D not plausible - there is no reason provided why it would be either. Also, is this a closed reference question? This is not specified in the pedigree.
Response
E1150-F017A is not described in the stem because it is closed during SDC. To give this position would cue the applicant. The normal flowpath (through fully open U E1150-F611A and an orifice) is also not given. To give either valve position in the 30 H 3 X X N stem would cue the applicant regarding the correct flowpath for SDC flow.
E E1150-F017A is a well-known valve and is used to throttle LPCI flow during emergency operations. A less than competent applicant might select this flowpath as the normal throttle valve for SDC flow.
Pedigree updated to provide plausibility for choosing the E1150-F017A flowpath, and to indicate closed reference question.
Recommend NRC change from U to E.
NRC: Question SAT with enhancements made.
NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
31 H 3 B E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
Embedded reference NRC: E2100-F007A is clearly labeled as a manual valve downstream of the containment inboard isolation valve. Therefore, it is not plausible that an applicant would assess that a manual valve inside the primary containment would be manipulated per procedure at power. This is an example where including a diagram lowers the LOD of the question and adversely affects the plausibility of the U distractors. Not including the diagram and providing the noun names of the valves 32 H 2 X N E operated would be a more effective way to ask the question. Also, is this a closed reference question? This is not specified in the pedigree. Finally, answer A and distractor B need to add the letter designation A to E2150-F015 when the valve is closed.
Response
- Fermi 2 disagrees that the question is LOD 1, due to the picture being provided, and instead contends that providing this picture raises the Page 7 of 21
ES-401 Facility: /Year: 8 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only operational validity of the question by providing the candidates with the exact section of the panel mimic that they would use to perform the procedure section from which the question was written.
- It appears that the NRC evaluator made the determination that this is LOD 1 because the E2100-F007A is located downstream of E2150-F005A (Div 1 CS inbd Iso Valve) and the NRC evaluator has assumed that this valve is inside containment. In fact, E2150-F005A is located outside containment, therefore providing no clue as to the physical location of the E2100-F007A.
- The only information that the candidate can glean from the picture is (1) the noun name of the valve (Div 1 CS Manual Iso Valve) and the type of valve (Manual because of name and the fact that the indications are bordered by white lines, indicating manual operation). The candidate must use his/her knowledge of the system to determine that the valve is inside containment.
Nothing on the picture provides this information.
- NOTE: During validation, 2 (out of 4) ROs chose distractors that required manually closing the E2100-F007A. During review, the operators confessed that they failed to recall this valve was located inside containment and thought it was accessible, even with the picture provided, which adds to the plausibility of these distractors.
- Finally, answer A and distractor B were revised to add the letter designation A to E2150-F015 when the valve is closed.
Recommend NRC change from U to E NRC: Question SAT with enhancements made.
33 H 3 M S NRC: The K/A addresses what a failure of the nuclear boiler instrumentation will have on the RPS. This question simply states that a pressure controller failure occurred and an unrelated set of RPS trip channel states happen to exist. Question should relate the state of RPS trip channels because of a failure associated with nuclear boiler instrumentation.
Response
U Removed pressure regulator failure from stem of question to make a clearer K/A 34 F 3 X M E match. Provided clarification in question pedigree that shows a clearer link between the malfunction of two Nuclear Boiler instruments (failing to trip when exceeding their trip setpoints) and the resulting impact on the RPS system (i.e., the candidate must determine that the failures will prevent RPS A from tripping, thus only a half scram will be received on RPS B because of the failure of the Nuclear Boiler instruments).
Note: This question is a modifi ed version of Question 53 on the 2018 Fermi 2 ILO NRC Exam for the SAME K/A. The only changes made were to change the malfunction from one instrument to two such that only a half scram will be received Page 8 of 21
ES-401 Facility: /Year: 9 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only vice a full scram.
NRC: Question SAT with enhancements made.
NRC: Provide additional justification for retract permit light. There was no information provided with regards to the Mode switch in Run. There was no information provided with regards to whether the IRMs retract whether or not the permit light is lit. IRMs are withdrawn to protect the detectors during high power operations. Distractor D is not plausible as those are the detectors (A&F) reading the lowest power. Also, is this a closed reference question? This is not specified in the pedigree.
35 F 4 X B E
Response
Explanation Updated to add more justification. Detectors B and E changed to range 1, and distractor D changed to state C, D, G and H (all on range 2 or bypassed) can be withdrawn without a rod block.
Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
36 H 2 N S 37 H 2 N S 38 H 3 B S 2015 NRC Exam NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
39 F 4 N E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
40 H 2 N E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
NRC: The answer and distractor choices need to specify the exact keylock switches, valves, and pushbuttons that are being referred to. Because of the current lack of specificity in the distractor choices, an applicant could select option B or D and U then argue that if it is assumed that the valves and/or pushbuttons operated in 41 H 3 X N conjunction with the key switches were ones that were unrelated to Drywell S
Pneumatics, either of these answers could also be considered as correct. Also, is this a closed reference question? This is not specified in the pedigree.
Closed reference, pedigree updated.
Page 9 of 21
ES-401 Facility: /Year: 10 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Question revised to specify exact components to be operated.
NRC: Question SAT with changes and enhancement made.
42 F 2 N S 43 H 3 B S 2009 NRC Exam; 2X2 NRC: The distractor analyses for B and D appear to be reversed in comparison with the same choices in the question.
44 F 3 N E Licensee:
Revised distractor analyses for B & D.
NRC: Question SAT with enhancements made.
Preliminary Review NRC: What is the cutoff flow rate for choices B and C, where the flow rate becomes insufficient to maintain dP at -0.25? Is there a more plausible choice for the stem value than 1000 cfm? This value seems too low - does the procedure (23.404) contain flow rates that may be more plausible (2500 cfm, 3000 cfm, for exam)?
Licensee:
The only CFM values listed in 23.404 for SGTS are as listed in the answer explanation after fan start (3879 and 4180 scfm) and the minimum design value 45 H 2 N E (3800 scfm.)
UFSAR assumptions for SGTS are a 3800 scfm minimum value and the 1000 scfm capacity of each of four parallel HEPA filters.
As the 3800 minimum would be sufficient as it is the design value, a value less than that value was used to meet the K/A of predicting and monitoring secondary containment differential pressure and to get the correct answer of insufficient. The only available scfm value listed in reference material for the system was the design value of one of four parallel HEPA filters, each of which is rated for 1000 scfm.
NRC: Question SAT with enhancements made.
Preliminary Review NRC: Explain how this question was significantly modified from the original bank question in the pedigree.
Licensee:
46 H 3 M E Source was NMP 2010 Q#15. Modified was chosen because of the difference in design between the two plants. At FERMI, NMPs distractor (restart manually) is the correct answer.
NRC: Question SAT with enhancement made.
Page 10 of 21
ES-401 Facility: /Year: 11 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 47 H 3 B S 2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
48 F 2 N E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
49 F 2 N E Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
2008 NRC Exam, 2017 Audit Exam; 2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
50 H 3 B E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
Embedded reference and Reference Provided NRC: Placing the words starting and running in the stem unnecessarily reduces the level of difficulty of this question. In an operational setting, an operator would need to be able to determine which component (electrical bus or diesel) was being represented by each voltmeter based upon the plant equipment alignment.
51 H 2 M E Response:
Question revised to remove starting and running in the stem. Also, based on discussion with NRC, a rate of 10 RPM was given to the synchroscope instead of a description of fast. This exceeds the procedural requirement and thus requires the speed reduction in the correct answer.
NRC: Question SAT with enhancements made.
2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
52 H 2 N E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
53 F 2 M S 2019 Fermi Exam (Modified) 54 F 2 N S 2X2 Page 11 of 21
ES-401 Facility: /Year: 12 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 2011 VY NRC Exam NRC: This question is written at a fundamental level of knowledge, and not at the listed higher cognitive level. This question can be answered based upon recall of system setpoints. Also, is this a closed reference question? This is not specified in H the pedigree.
55 2 B E F Licensee:
Closed reference, pedigree updated.
Changed LOK to F.
NRC: Question SAT with enhancements made.
Preliminary Review NRC: Question is at the HIGH cognitive level.
F 56 3 N E Licensee:
H Changed LOK to H.
NRC: Question SAT with enhancement made.
NRC 2015 Exam NRC: Is this a closed reference question? This is not specified in the pedigree.
57 H 2 B E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
58 F 2 N S 2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
59 F 3 N E Response:
Closed reference, pedigree updated.
NRC: Question SAT with enhancement made.
60 H 4 B S NRC: Is this a closed reference question? This is not specified in the pedigree. This was also not a Preliminary Review question.
Response
61 F 3 N E Closed reference, pedigree updated.
Preliminary review tag removed.
NRC: Question SAT with enhancement made.
Page 12 of 21
ES-401 Facility: /Year: 13 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 62 F 3 N S 63 F 2 B S NRC 2009 Exam 64 F 3 N S NRC: Distractor A is not plausible and can be easily eliminated. Also, is this a closed reference question? This is not specified in the pedigree.
Licensee:
65 F 2 X B E Closed reference, pedigree updated.
Revised distractor A per NRC comment.
NRC: Question SAT with enhancements made.
NRC 2015 Exam NRC: The A and B distractors can be easily eliminated, since Medical and an immediate supervisor would minimally be notified. Medical and immediate supervisor should be brought up into the stem and the distractors changed to other plausible choices, including no additional notification required. Also, is this a closed reference question? This is not specified in the pedigree.
66 F 2 X B E Licensee:
Revised question based on feedback. Distractors are now more homogenous in nature.
Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
NRC 2001 Retake Exam NRC: The pedigree needs to be edited to reflect the correct NRC Exam Usage and whether the question is closed reference or not.
67 F 2 B E Licensee:
Edited pedigree to identify previous exam use on 2001 retake.
Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
2X2; 2017 NRC Exam NRC: The question requires RPS system knowledge to answer, as opposed to purely generic concepts as required for Tier 3. Suggest selecting another K/A. Also, is this a 68 F 3 X N E closed reference question? This is not specified in the pedigree.
Licensee:
NRC examiner systematically reselected K/A, resulting in K/A 2.1.41. New question Page 13 of 21
ES-401 Facility: /Year: 14 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only provided for the selected K/A.
NRC: Question SAT with enhancements made.
NRC 2017 Exam NRC: Is this a closed reference question? This is not specified in the pedigree.
69 F 2 B E Response:
Closed reference, pedigree updated NRC: Question SAT with enhancement made.
70 F 2 M S NRC 2012 Exam/2017 Audit Exam NRC: Is this a closed reference question? This is not specified in the pedigree.
71 F 2 B E Response:
Closed reference, pedigree updated NRC: Question SAT with enhancement made.
Preliminary Review NRC: The question is not properly focused. Phrasing the question in terms of performing specific operations/tasks/surveillances on various pieces of equipment in the distractors would make the question clearer. Also, specifically state that the survey maps are provided references for the exam.
Licensee:
72 H 2 X X N E Focus added. The operation is listed as clearance application and one tag is to be hung in each area.
The survey maps are not provided to the applicants during the exam. They were included in the exam package to provide justification for correct / incorrect responses.
Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
NRC 2017 Exam; 2X2 NRC: LOD=1; the correct answer can be easily recognized. Only A and B are plausible distractors. Also, is this a closed reference question? This is not specified in the pedigree.
73 F 2 X B E Licensee:
LOD=1: Disagree.
Question is relevant to the K/A and B is the Answer. A, C and D are combinations of CRS/SRO. Per regulatory requirements any SRO can fill this role, however every Page 14 of 21
ES-401 Facility: /Year: 15 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only site has different conduct manual requirements on who should fill the role at that site.
Because both the CRS and SM have an SRO license it is plausible that a site could designate either for each role.
Additional factors History: Question is BANK and was accepted on the 2017 NRC exam for LOD and plausibility.
Validation data: 4 out of 8 validators got this wrong, indicating this question and its distractors have discriminatory validity as referenced Section C.2 Examples App B in NUREG 1021. Answer and distractors are homogeneous and highly plausible (both have SRO license) as required by NUREG 1021 C.2.h Closed reference, pedigree updated NRC: Question SAT with enhancement made.
DAEC NRC 2017 Exam NRC: Is this a closed reference question? This is not specified in the pedigree.
74 F 2 B S Response:
Closed reference, pedigree updated NRC: Question SAT with enhancement made.
NRC: The questions pedigree contains typographical errors which should be fixed.
Also, is this a closed reference question? This is not specified.
Licensee:
75 F 2 N E Corrected typos in pedigree Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
76 H 3 B S NRC 2017 Exam; Embedded reference Preliminary Review; 2X2 NRC: Stem phrasing is inconsistent with answer choices - stem asks 1) what the CRS must enter and 2) directs action to shut which valve(s). The answer choices are backwards from the stem logic. Rephrase either answer choices or the stem. Also, H is this a closed reference question? This is not specified. Finally, as written, the 77 X N E question does not require higher cognitive analysis to answer. It requires only F 3 recognizing which valve to close and the basis for doing so.
Licensee:
Revised question to address NRC comment.
Corrected typos in pedigree Page 15 of 21
ES-401 Facility: /Year: 16 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Closed reference, pedigree updated.
LOK changed to F.
NRC: Question SAT with enhancements made.
NRC 2015 Exam NRC: It appears that procedure 20.000.18 is entered from 20.000.19 based on the If/Then statement; in which case, the procedures are performed concurrently. Is this correct? If so, is there anything that describes whats preferred between opening the generator output breakers in 20.000.18 (step B.5/6) and taking APRMs out of operate in 20.000.19 (step B.2)? Is preferred method the best choice in the stem to ensure distractor A is not also a correct answer? Please explain!
78 H 3 X B E Licensee:
The reviewers comments are correct in that 20.000.18 has an action to open the generator output breaker and this step would be performed if it was necessary to enter 20.000.18. However, conditions in the stem of the question do not indicate a reason to enter 20.000.18. Distractor A justification was modified to provide clarification to this end.
NRC: Question SAT with enhancement made.
79 H 3 N S NRC: Question states it is closed reference - are the applicants expected to know NPSH limit & vortex limit curves from memory?
80 H 3 B E Licensee:
Open reference, pedigree updated - Sheet 6 NRC: Question SAT with enhancement made.
81 F 3 N S NRC: Plural LCO(s) is not necessary because part 2 of the question only has one potential answer (either 3.8.1 or 3.8.7), and fix typo (peform). Also, specify what, if any, references are available to answer this question.
82 H 2 N E Licensee:
Edited stem as noted.
Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
83 H 3 N S 84 H 2 N E 2X2 Page 16 of 21
ES-401 Facility: /Year: 17 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only F NRC: Question is low-cog, not high-cog.
Licensee:
Updated to fundamental LOK.
NRC: Question SAT with enhancement made.
2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
85 F 3 N E Response:
Closed reference, pedigree updated NRC: Question SAT with enhancement made.
NRC 2015 Exam NRC: With RPV level given in the normal operating band and stable, how is distractor B plausible, as it can be eliminated based off the provided initial conditions. Also, specify what, if any, references are available to answer this question.
86 H 3 X B E Licensee:
New distractor written for B.
Closed reference, pedigree updated.
NRC: Question SAT with enhancements made.
87 H 3 N S 2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
Response
88 H 3 N E Closed reference, pedigree updated NRC: Question SAT with enhancement made.
NRC 2008 Exam; NRC 2017 Audit Exam NRC: Are references provided? Under question use, closed reference is stated, however, under reference information, for closed book exam provide references is stated (?). Without providing MSIV stroke times, the stroke times could be increased, and the valves would remain operable (from a >3 second closure time to U <5 second closure), thus changing the answer such that the valve remained operable 89 H 2 X B S but degraded.
Licensee:
Revised the stem to indicate that the maximum stroke time has been exceeded to meet NRC comments.
Removed for closed book exam provide references statement in explanation.
Page 17 of 21
ES-401 Facility: /Year: 18 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Closed reference, pedigree updated.
NRC: Question SAT with changes made.
90 H 3 N S Reference provided NRC: With LCO 3.1.3 provided, the question becomes direct look-up.
U Licensee:
91 H 2 X N S Removed TS 3.1.3 from provided references.
Reclassified question as closed reference. Pedigree updated.
NRC: Question SAT with changes made.
Reference provided NRC: Clearly specify what references are available to answer this question in the pedigree.
92 H 3 N E Licensee:
Open reference, pedigree updated - Sheet 6.
NRC: Question SAT with enhancement made.
93 H 3 N S Preliminary Review; 2X2 NRC: Why is the (FB Only) designation necessary for the Other individual?
Licensee:
(FB only) is listed with a Fire Protection Inspector to indicate that they have assignments outside of operations and will only respond to Fire Brigade events and 94 F 2 M E are not part of the pool that OPS manpower provides to operations, and therefore not available for other assignments. The listing in the stem of the question is formatted to appear as a normal watch bill, so the Fire Protection Inspector is listed as they would be on the normal watch bill.
NRC: Question SAT with enhancement made.
NRC: Question does not match the K/A (refueling admin requirements) as the question is addressing refueling vessel water TS limits and the bases. Also, specify what, if any, references are available to answer this question.
U Licensee:
95 F 3 X M E As required by NUREG 1021 ES-201 C.1.h This MODIFIED question is listed as being used on the St. Lucie HLC exam for 2015. On the 2015 Exam, the original question was accepted for this K/A. The change made to the question replaced the potentially two correct irradiated and recently irradiated with shielding and iodine Page 18 of 21
ES-401 Facility: /Year: 19 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only retention. This does not take it away from the K/A as listed.
Closed reference, pedigree updated Recommend NRC change from U to E.
NRC: Question SAT with enhancement made.
2017 Audit Exam NRC: Reference MMA26 was provided in the Exam References folder, but the question does not state whether the reference is provided. Does not appear that the applicant would be required to memorize the procedural guidance for troubleshooting 96 F 3 B E plan approvals based on the risk of the evolution. Specify what, if any, references are available to answer this question.
Licensee:
MMA26 Added as reference, Open reference pedigree updated NRC: Question SAT with enhancement made.
2X2 NRC: Is this a closed reference question? This is not specified in the pedigree.
97 F 3 M E Response:
Closed reference, pedigree updated NRC: Question SAT with enhancement made.
NRC 2018 Exam, NRC 2012 Exam Reference provided NRC: Need to clarify in the pedigree which exam question was modified to create this question. It is not clear.
98 F 2 M E
Response
Pedigree updated. Question was modified from NRC exam 2018.
Open reference, pedigree updated - MCE06.
NRC: Question SAT with enhancements made.
Preliminary Review NRC: Restate the question stem to say, As the CRS, which of the following actions should be directed first and what is the 99 F 2 X B E basis for this prioritization?
The stem, as written, is vague and could lead to misunderstanding of what the question is asking. Also, explain in the pedigree why B and C are incorrect.
Response
Page 19 of 21
ES-401 Facility: /Year: 20 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Question rephrased as recommended. Of the listed actions, the correct answer is still the first that should be performed (each of the others need other actions to be completed before they can be performed). Plausibility updated to better explain why B and C are incorrect.
NRC: Question SAT with enhancements made.
100 F 2 N S 2X2 Page 20 of 21
ES-401 Facility: /Year: 21 Form ES-401-9 B= 32 F= 37 E= 48 RO TOTALS: M= 7 H= 38 U= 3 Additional Notes:
N= 36 B= 7 F= 11 E= 15 SRO TOTALS: M= 4 H= 14 U= 2 Additional Notes:
N= 14
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 2. Chief Examiner comments are indicated in blue.
Page 21 of 21