ML21020A120
| ML21020A120 | |
| Person / Time | |
|---|---|
| Site: | Monticello  |
| Issue date: | 11/03/2020 |
| From: | Nuclear Management Co |
| To: | NRC/RGN-III/DRS/OLB |
| Zoia C | |
| Shared Package | |
| ML19127A312 | List: |
| References | |
| Download: ML21020A120 (18) | |
Text
ES-401 Facility: Monticello /Year: 2020 Written Examination Review Worksheet Form ES-401-9 Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3. Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length).
T/F: The answer choices are a collection of unrelated true/false statements.
Cred. Dist>: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
- /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only. (K/A and license-level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
F 2
M S
2016 NRC Exam; 2X2 NRC: As a Significantly Modified NRC exam question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: 2016 NRC Exam #5 included NRC: 2016 NRC Exam #5 was included as a separate file - SAT 2
H 2
B S
2015 Audit Exam; 2X2 NRC: The discriminatory value of the question would be raised if fails to start were included in the possibilities.
Response: Including fails to start would introduce a psychometric flaw and allow the candidate to choose either the most prevalent options or least prevalent options. No changes required.
NRC: SAT as submitted 3
H 3
?
M E
2010 NRC Exam NRC:
ES-401 Facility: Monticello /Year: 2020 2
Form ES-401-9 Page 2 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 1.
As a Significantly Modified NRC exam question, please explain in the pedigree how it was modified and/or include the original question as a reference.
- 2.
Underlining/emphasizing certain conditions should be avoided
- 3.
Explain how the K/A is met in the pedigree.
- 4.
Is this level of detail required to be known from memory (Objective)?
Response
- 1.
2010 NRC Exam #90 included
- 2.
IAW with NUREG-1021 Appendix B, Attachment 1, #10; it is recommended that key points are underlined or highlighted.
- 3.
K/A - The examinee must interpret and determine that a battery voltage of 90 VDC (Partial Loss) will result in a loss of DC supply to the inverter. This in turn will result in a lockup of both MFRVs. This has been included in pedigree of question.
- 4.
Yes - MT-ILT-AOP-020L Objective 4: Describe the applicable operational implications as they relate to a loss of a specific UPS. Put in pedigree.
No changes required.
NRC: 2010 NRC Exam #90 was included as a separate file and how the K/A is met is explained in the pedigree - Question is now SAT 4
H 2
N S
NRC: Reword distractor D: Start P-61 from C-07 after resetting it by momentarily taking its control switch to off.
Response: This is required for options B, C and D. Starting from C-07 implies that they need to flag the switch to off before starting. No changes required.
NRC: SAT as submitted 5
F 2
B S
2015 Audit Exam 6
H 3
B S
2015 NRC Exam; Embedded Reference 7
F 2
B S
2013 NRC Exam; 2X2 8
F 3
B S
2018 Audit Exam 9
F 2
B S
2015 NRC Exam 10 F
2 N
S NRC: For distractors B and C, need to specify under what condition(s) these values would apply and/or how they are used.
Response: OPEN - Do not understand comment.
NRC: Explained at OV, SAT as submitted 11 H
3 B
E 2018 Audit Exam
ES-401 Facility: Monticello /Year: 2020 3
Form ES-401-9 Page 3 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only NRC: Where the stem says, Complete the following statement relating to the initiation of drywell sprays at this time? is not a question but a direction. Suggest revising that sentence to, Which of the following statements describes why this action is needed?
Response: Reworded stem as suggested.
NRC: Question is now SAT 12 H
2 N
E Embedded Reference NRC:
- 1.
Shouldnt the applicant recognize WR versus NR instruments without being told in the stem? This information should be removed from the Embedded Reference, since it is important to selecting the proper response.
- 2.
Will B cause the entire GOOD DATA column to change to NO?
- 3.
Elaborate in the pedigree specifically why each distractor is either correct or not and include the expected responses for each.
Response
- 1.
The K/A requires the use of SPDS. This is the display that the operator actually uses in the control room (operational validity). Additionally, operators are not required to memorize instrument ID numbers; they would always be given the instrument nomenclature.
- 2.
No, the wide range instruments would not change to NO until pressure rises above 1200 psig.
- 3.
Pedigree has been enhanced. Additionally, distracter C pressure lowered to 850 psig to raise its plausibility.
Question is Satisfactory with enhancements.
NRC: Question is now SAT 13 H
3 N
E NRC: The pedigree needs to include why each distractor is correct or not, and include plausibility, which A and B do not.
Response: Provided detailed explanations of plausibility for A and B. No further action required.
NRC: Question is now SAT 14 H
3 B
E 2016 NRC Exam NRC: Reword distractor B to: DW Sprays can ONLY be initiated with 10A-S18A in MANUAL OVERRIDE. This makes it consistent with A and avoids negative logic.
Response: Distracter B changed as suggested above. No further action required.
NRC: Question is now SAT
ES-401 Facility: Monticello /Year: 2020 4
Form ES-401-9 Page 4 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 15 H
2 X
B E
Early Review Question NRC: The stem directly states RPV Flooding is in progress, which is cueing, since the correct answer is The RPV is showing indications of being flooded. Reword either the correct response and/or the stem condition to avoid this issue.
Response: Reworded the correct response to state The main steam lines are showing indications of being full. No further action required.
NRC: Question is now SAT 16 H
3 N
S Embedded Reference 17 F
2 B
S 2009 NRC Exam NRC: Are ROs responsible for knowing the bases of this procedure? If so, consider revising the stem to read: What is the basis, per C5-2007, for performing this step?
Response: Yes, ROs are required to have basis knowledge of the EOPs. Stem revised as suggested. No further action required.
NRC: SAT as submitted 18 H
2 X
M S
2015 NRC Exam; Embedded References; 2X2 NRC:
- 1.
As a Significantly Modified NRC exam question, please explain in the pedigree how it was modified and/or include the original question as a reference.
- 2.
The RED outline around the stack radiation release value is a cue that C.5-1400 entry is needed. Consider providing a black and white picture here.
Response
- 1.
2015 NRC Exam question #18 provided.
- 2.
Even though the indication is outlined in red it is plausible because an examinee may believe that both are required to be outlined red for the C.5-1400 entry. Changing the color to black and white removes the operational validity of the picture. Additionally, color is needed because another indication of NOT VALIDATED information is the outline and values being magenta color.
No changes required.
NRC: 2015 NRC Exam question #18 was included as a separate file - SAT as submitted 19 F
2 B
E 2016 NRC Exam NRC:
ES-401 Facility: Monticello /Year: 2020 5
Form ES-401-9 Page 5 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 1.
Not sure that Reactor Mode Switch placed in SHUTDOWN plausible - consider pulling Reactor Mode Switch placed in RUN into the stem and modifying each of the distractors accordingly.
- 2.
Elaborate in the pedigree specifically why each distractor is either correct or not, and address plausibility where appropriate.
Response
- 1.
Plausible as the normal immediate action after pressing the scram push buttons is to place the Mode Switch in shutdown.
- 2.
Plausibility explanations have been elaborated on in the pedigree.
No further action required.
NRC: Question is now SAT 20 H
2 B
E 2010 NRC Exam NRC:
- 1.
Consider changing the stem question to; Which actions from C.4-A (Reactor Scram) should be performed by the OATC?
- 2.
Elaborate in the pedigree specifically why each distractor is plausible.
Response
- 1.
Using the word should would make all answer options correct. These are all actions that normally would be taken but two of them cannot because of the loss of power.
- 2.
Pedigree has been elaborated on to provide more detail.
No further action required.
NRC: Question is now SAT 21 H
3 B
E NRC: Elaborate in the pedigree specifically why each distractor is plausible.
Response: Pedigree has been elaborated on to provide more detail.
No further action required.
NRC: Question is now SAT 22 H
2 M
E 2010 NRC Exam; Embedded Reference NRC:
- 1.
As a Significantly Modified NRC exam question, please explain in the pedigree how it was modified and/or include the original question as a reference.
- 2.
Elaborate in the pedigree specifically why each distractor is plausible.
Response
- 1.
2010 NRC Exam question #16 included.
- 2.
Pedigree has been elaborated on.
ES-401 Facility: Monticello /Year: 2020 6
Form ES-401-9 Page 6 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only No further actions required.
NRC: 2010 NRC Exam #16 was included as a separate file and plausibility is better explained in the pedigree - Question is now SAT 23 H
2 B
E 2015 NRC Exam NRC:
- 1.
With power well below the point where MFRVs are in-service, two distractors can be easily excluded. Raising power to 15% would make the question more discriminating.
- 2.
Bring Loss of up into the question stem and reduce the amount of bolding and underlining in the stem and distractors.
Response
- 1.
Raised power to 12%. 15% is too close to (approximately 17%). Two distracters being excluded is consistent with a 2x2 question.
- 2.
Brought Loss of to the stem of the question and removed underlining.
No further action required.
NRC: Question is now SAT 24 F
2 B
S 2015 NRC Exam 25 H
2 B
S 26 F
2 B
S 2013 NRC Exam 27 H
2 B
S 28 F
3 B
E 2015 NRC Exam NRC: Elaborate in the pedigree specifically why each distractor is both plausible and not correct.
Response: Elaborated on pedigree. No further actions required.
NRC: Question is now SAT 29 F
2 M
S 2016 NRC Exam NRC: As a Significantly Modified NRC exam question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: 2016 NRC Exam Question 50 included. No further action required.
NRC: 2016 NRC Exam #50 was included as a separate file - SAT as submitted 30 H
2 M
S 2018 NRC Exam
ES-401 Facility: Monticello /Year: 2020 7
Form ES-401-9 Page 7 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only NRC: As a Significantly Modified NRC exam question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: 2018 NRC Exam Question 29 included. No further action required.
NRC: 2018 NRC Exam Question 29 was included as a separate file - SAT as submitted 31 H
2 N
E 2018 Audit Exam NRC: While the correct answer may be associated with the K/A (Ability to predict and/or monitor changes in parameters associated with HPCI controls including:
Suppression pool level), the distractors are essentially a set of false statements for given conditions unrelated to the K/A, and two of these are implausible.
Response: NUREG 1021 doesnt specifically state that distracters must meet the K/A; however, as an enhancement the question has been re-written (2x2) to satisfy the comment above. No further action required.
Edits to first bullet and added semicolon in stem question.
NRC: Question is now SAT 32 F
3 N
S NRC: A condition should be added to support the status of motor cooling.
Response: Without stating the condition of 11 Core Spray Pump motor cooling, it would be assumed that it is operable. No action required.
NRC: SAT as submitted 33 F
3 N
S 34 H
3 N
S NRC: Are all applicable equations required to be known from memory? If so, provide a summary of the applicable learning objective in the pedigree.
Response: Yes, these equations are required to be known from memory. Applicable learning objective provided in pedigree. No further action required.
NRC: SAT as submitted 35 H
2 X
N E
Early Review Question; Embedded Reference NRC: If Bypass IRM 11 on C-05 is included with other non-essential but not incorrect actions, that answer can be considered partially correct. This is a psychometric flaw, even when those actions are not limited to the minimum required action(s). Limiting questions to only minimum required actions raises operational validity concerns as well, especially when extra actions dont invalidate a longer answers correctness. Replace distractor D with plausible but incorrect actions.
Also, the plausibility of every distractor needs to be explained in the pedigree.
Response: Distracter D has been rewritten and moved to distracter B. Distracter pedigree has been updated.
ES-401 Facility: Monticello /Year: 2020 8
Form ES-401-9 Page 8 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only NRC: Question is now SAT 36 H
3 B
S 2013 NRC Exam; Embedded Reference 37 F
2 N
E Early Review Question; 2X2 NRC: Add additional set(s) which may be operable to the distractors. This requires the applicant to identify one (or more) operable set(s), versus limiting the number of possibilities, and increases the questions discriminatory value. Also, the plausibility of every distractor needs to be explained in the pedigree.
Response: Added 3rd set for answer options and pedigree has been updated. No further actions required.
NRC: Question is now SAT 38 H
3 B
E 2013 NRC Exam; Embedded Reference NRC:
- 1.
Is PRNM a common acronym? If not, it should be spelled out in the stem.
- 2.
Remove ALL from distractor D.
Response: PRNM is a common acronym. Removed ALL from distracter D. No further action required.
NRC: Question is now SAT 39 F
2 N
E NRC: This question is like the Low Level of Knowledge example on page B-19 of NUREG-1021, except that ALL major RCIC valves receive main/control power from Division I 250 VDC, so this question seems less discriminating than the example.
Response: The example on page B-19 states that these can be used judiciously. In this case the question relates to an important TS related system making it operationally valid and the question exactly meets the K/A that has an IR>2.5. The stem of the question and one distracter have been enhanced to raise the discriminatory value. Question is satisfactory with enhancement. No further action required.
Simplified stem.
NRC: Question is now SAT 40 H
2 N
E Reference Provided NRC: Dont point to a specific provided reference in the stem. If a reference is needed, it is expected that the applicant would look for it.
Response: NUREG-1021 ES401 and App. B do not inhibit the calling out of a reference in the stem for an Initial Written Exam open reference question. An open reference question for ILT is not meant to be an exercise in locating procedures. No changes required.
ES-401 Facility: Monticello /Year: 2020 9
Form ES-401-9 Page 9 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Removed reference.
NRC: Question is now SAT 41 F
3 B
S 42 F
2 B
E NRC: Add reference B.03.03-01 and a brief explanation of what the amber light signifies to the pedigree.
Response: Reference added, and pedigree updated. No further action required.
NRC: Question is now SAT 43 H
3 X
B E
2010 NRC Exam NRC:
- 1.
The stem provides conditions that are not needed to answer the question. For example, why the ED was needed. These should be removed.
- 2.
Given that there are non-functional ADS valves, explain more fully in the pedigree why it is incorrect to use NON-ADS valves.
Response: Stem has been simplified and pedigree updated. No further action required.
NRC: Question is now SAT 44 F
2 B
S 2015 NRC Exam NRC: Underlining/emphasizing certain conditions should be avoided.
Response: IAW with NUREG-1021 Appendix B, Attachment 1, #10; it is recommended that key points are underlined or highlighted.
NRC: SAT as submitted 45 H
4 B
S 46 H
2 X
B S
2009 NRC Exam; Embedded Reference NRC: Distractor A is not credible, given the number of times each applicant has performed EDG manipulations where the output breaker is closed when the synchroscope reaches 1200 after rotating slowly in the FAST direction.
Response: Distracter A is credible. The examinee must interpret and understand counterclockwise in the stem, apply that knowledge to the picture and determine that the rotation is incorrect. It is plausible that an examinee could misread or misinterpret the rotation and choose this answer. No changes required.
NRC: SAT as submitted 47 H
3 X
N E
Early Review Question; Embedded Reference
ES-401 Facility: Monticello /Year: 2020 10 Form ES-401-9 Page 10 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only NRC: Expand and correct the stem and/or pedigree to include the following information:
- 1.
- 11 Core Spray (CS) System status (this was not provided on the following page, as stated in the stem).
- 2.
The correct and incorrect answers must be properly identified in the pedigree.
- 3.
Tie the distractors to the E.2-05 steps and bases, including plausibility reasoning for all distractors.
Response
- 1.
Picture is provided.
- 2.
Corrected the pedigree for incorrect options.
- 3.
E.2-05 has been linked to the distracters in the pedigree.
NRC: Question is now SAT 48 H
3 M
S 2013 NRC Exam; Embedded Reference NRC: As a Significantly Modified NRC exam question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: 2013 NRC Exam Question 45 included.
NRC: 2013 NRC Exam Question 45 was included as a separate file - SAT as submitted 49 H
2 N
E Early Review Question NRC: The plausibility of every distractor needs to be explained in the pedigree.
Response: Plausibility of distracters updated.
NRC: Question is now SAT 50 H
2 B
E 2016 NRC Exam; Embedded Reference NRC: The plausibility of every distractor needs to be explained in the pedigree.
Response: Pedigree updated.
NRC: Question is now SAT 51 F
2 N
S 2X2 52 H
3 B
S 2016 NRC Exam; 2X2 53 H
3
?
B E
2010 NRC Exam NRC: Explain in the pedigree how a K/A for CCW pump power supplies is met for the EDG-ESW system.
Response: MNGP only has 2 RBCCW pumps with simple power supplies. EDG-ESW is a critical and operationally important Component Cooling Water system. Chief
ES-401 Facility: Monticello /Year: 2020 11 Form ES-401-9 Page 11 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Examiners in the past have allowed the use of this system for K/A 400000. Pedigree has been updated.
NRC: Question is now SAT 54 H
3
?
B E
2009 NRC Exam NRC: Explain in the pedigree how the K/A for a LOSS or MALFUNCTION of RMCS is met - is the RWM condition a normal response?
Response: Selecting the wrong control rod will result in a RWM and RMCS Rod Block. This will result in the loss of ability to withdraw and in this case, also insert control rods because of the select block. Pedigree updated.
NRC: Question is now SAT 55 H
3 B
E 2009 NRC Exam NRC: If the feed pump were to trip on motor overload, complete the following statement concerning the Recirc pumps? is not a question, and can be shortened to read If the feed pump trips on motor overload, complete the following statement:
Response: Stem updated as stated above.
NRC: Question is now SAT 56 H
2 B
E NRC: The plausibility and correctness of every distractor needs to be explained in the pedigree, not just stated. Examples - Why would strainer DP rise but filter DP lower?
The pedigree states that the DP of the questionable filter may drop. Why is it plausible for strainer DP to lower?
Response: Pedigree updated.
Removed Reactor Power option and changed B Filter D/P to lower.
NRC: Question is now SAT 57 F
2 X
B E
2015 NRC Exam NRC:
- 1.
Why would INDEXING MECHANISM (distractor C) be plausible?
- 2.
Why would SHEAR VALVE (distractor B) be plausible?
- 3.
Provide Reference B.05.03 it was not provided on the DVD.
Response
- 1.
Pedigree updated.
- 2.
Pedigree updated.
- 3.
Reference will be provided during review.
NRC: Question is now SAT 58 F
2 X
B E
Early Review Question
ES-401 Facility: Monticello /Year: 2020 12 Form ES-401-9 Page 12 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only NRC: The 2/3 core height description within the Containment Spray permissive name provides a clue that A is the correct answer. Also, the analysis of every distractor in the pedigree should include which level instrument(s) feed(s) into its logic and indications to support both plausibility and that it is not a correct answer.
Response: Reworded stem to remove cue and updated pedigree.
NRC: Question is now SAT 59 H
3 M
B E
Reference Provided NRC: As a Significantly Modified bank question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: Question incorrectly coded as SM. Question is Bank.
NRC: Question is now SAT 60 H
3 B
S 2015 NRC Exam 61 F
2 B
S 62 H
2 B
S 2015 NRC Exam; Embedded Reference NRC: Dont point to a specific provided reference in the stem. If a reference is needed, it is expected that the applicant would look for it - separate it from the body of the exam and provide it as a separate reference.
Response: NUREG-1021 ES401 and App. B do not inhibit the calling out of a reference in the stem for an Initial Written Exam open reference question. An open reference question for ILT is not meant to be an exercise in locating procedures. No changes required.
Embedded reference is an operator aide on the panel.
NRC: Question is SAT as submitted.
63 F
2 N
E NRC: Need references and more detail in the pedigree to support this question. For example, neither B.08.05-05 or any lesson plans were provided on the DVD.
Response: Reference will be provided during review.
More detail added to pedigree.
NRC: Question is now SAT 64 F
2 N
E NRC: The questions discriminatory value would be enhanced by not mentioning SBGT competing with RB exhaust fans.
Response: Without stating that, it would be difficult to determine what the question is asking. No changes required.
Updated pedigree.
NRC: Question is now SAT
ES-401 Facility: Monticello /Year: 2020 13 Form ES-401-9 Page 13 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 65 H
2 B
S 2009 NRC Exam 66 F
2 M
S 2018 NRC Exam NRC: As a Significantly Modified bank question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: 2018 NRC Exam question #66 provided.
NRC: 2018 NRC Exam question #66 was included as a separate file - SAT 67 F
2 B
E NRC: The plausibility and correctness of every distractor needs to be explained in the pedigree, not just stated. For example, why are they incorrect, and in what circumstances would they be correct?
Response: Distracter pedigree updated.
NRC: Question is now SAT 68 F
2 B
S 2015 NRC Exam 69 F
2 X
B U
S 2018 Audit Exam (Replacement question)
NRC: The distractors are implausible and answer a different question than asked in the stem, addressing changes to the TS and USAR!
Response: Unsure of the different question the comment is referring to?
Chief Examiner concurred that K/A was not suitable for an operationally valid RO question. Randomly reselected K/A (2.2.39) and provided new question.
NRC: Replacement question is SAT 70 H
3 B
S 2015 NRC Exam; Embedded Reference NRC: Is all information in the embedded reference needed to answer the question?
Response: Yes, question can be answered with reference provided.
NRC: Question is SAT as submitted.
71 H
3 B
S 2015 NRC Exam 72 H
2 B
S 2010 NRC Exam 73 F
2 B
S 74 F
2 X
N E
Early Review Question NRC: Distractors A and B are not plausible. Consider replacing them or incorporating proper notification techniques into the question using the plausible distractors (2X2).
Response: Distracters A and B replaced, all options enhanced, B is now the correct answer and question is New.
ES-401 Facility: Monticello /Year: 2020 14 Form ES-401-9 Page 14 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only NRC: Question is now SAT 75 H
3 N
E 2X2 NRC: Document in the pedigree the Lesson Plan objective (wording) that this knowledge is required to be known from memory Response: LP LO 8: Predict the impact/consequences on the SBGT/SCT Systems given the following: Reactor building ventilation exhaust high radiation.
NRC: Question is now SAT 76 H
2 B
E 2018 Audit Exam; References Provided NRC: Given the distractor choices available, pointing to the C.5-2007 Pressure Leg actions in the stem is unnecessary and is effectively cueing.
Response: Since all the answer options come from the pressure leg, stating it in the stem in not queuing. Its the only place they could go to find the right answer.
However, Pressure Leg has been removed from the stem as it is not needed.
Question is SAT with Enhancement.
NRC: Question is now SAT 77 H
3 B
S 2018 Audit Exam; References Provided; 2X2 78 H
3 X
N E
References Provided NRC: General Emergency is not plausible for the given conditions.
Response: GE removed from options. Changed answer options to specify which EAL Rad Effluent shall be declared. New correct answer is C.
NRC: Question is now SAT 79 H
3 X
B S
2018 Audit Exam; References Provided NRC: Distractor A is implausible. Multiple misconceptions are needed to get here.
Response: One plotting error and one knowledge error can result in the examinee choosing this option. No changes required.
NRC: Question is SAT as submitted.
80 H
2 B
S 2013 NRC Exam; References Provided; 2X2 81 H
2 X
B S
Early Review Question; 2013 NRC Exam; References Provided NRC:
- 1.
Distractor C is implausible.
- 2.
Underlining/emphasizing certain conditions should be avoided.
ES-401 Facility: Monticello /Year: 2020 15 Form ES-401-9 Page 15 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only
- 3.
Per the pedigree, this is a C.4-A entry condition and therefore RO knowledge.
Consider reselecting the K/A for this question, since entry condition knowledge is needed to meet it.
Response: 2013 NRC Exam
- 1.
Distracter C is plausible as the examinee must determine that power is <
4% based on the given conditions. Otherwise this would be a correct answer.
- 2.
IAW with NUREG-1021 Appendix B, Attachment 1, #10; it is recommended that key points are underlined or highlighted.
- 3.
This is not a C.4-A Entry Condition. This is an EOP redirect action determined and directed by the CRS. An RO would not make this decision to use C.4-A to insert the remaining control rods.
NRC: Question is SAT as submitted.
82 H
2 B
S 2013 NRC Exam; References Provided 83 H
3 B
S 2018 Audit Exam; References Provided 84 H
2 M
S 2018 NRC Exam; References Provided; 2X2 NRC: As a Significantly Modified NRC question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: 2018 NRC Exam question #82 provided.
NRC: 2018 NRC Exam question #82 was included as a separate file - SAT 85 H
3 B
S NRC: Define WCT and why this would be plausible, Response: WCT = Waste Collection Tank. This is plausible as Leakoff from the post filters drains to this tank.
NRC: Question is SAT as submitted.
86 H
3 N
E Embedded Reference NRC: Very close to meeting RO question criteria (Can the question be answered solely by knowing the purpose, overall sequence of events, or overall mitigative strategy of a procedure?}. Also, the question could be improved and would better match the K/A if the procedure was selected and used to correct the low CST level.
Response: This question meets 10CFR55.43(b)(5) Assessment of facility conditions and selection of appropriate procedures during abnormal conditions. Additionally, changing the question to correct the CST level wouldnt match the RCIC portion of the K/A and wouldnt allow for plausible answer options. No changes required.
Reworded question portion of stem.
NRC: Question is now SAT
ES-401 Facility: Monticello /Year: 2020 16 Form ES-401-9 Page 16 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 87 H
3 N
S 88 F
3 B
S 2013 NRC Exam 89 F
2 B
S 90 H
3 N
E Early Review Question NRC: Explain the significance of the Continuous Awareness Step in the pedigree.
Are they required to be known from memory and are they required RO knowledge?
Response: The CRS would brief the procedures continuous awareness step in the PJB and direct the actions if the event occurred. Not RO responsibility. Updated pedigree.
91 H
3 M
S 2018 NRC Exam; Embedded Reference & References Provided NRC: As a Significantly Modified NRC question, please explain in the pedigree how it was modified and/or include the original question as a reference.
Response: 2018 NRC Exam Question #77 included.
NRC: 2018 NRC Exam Question #77 was included as a separate file - SAT 92 H
2 B
S 2015 NRC Exam; References Provided 93 H
3 B
S 2018 Audit Exam; Embedded Reference 94 F
2 N
S 95 H
3 B
S 2018 NRC Exam; Embedded References 96 H
1 2
X B
U S
2018 Audit Exam; References Provided NRC:
- 1.
Direct Lookup - The stem says where to look and exactly what the problem is.
2.10.8.A.1 & A.2 says what to do and by when. Add the times to 0800, and the correct answer is identified. No other choice is plausible.
- 2.
The plausibility and correctness of every distractor needs to be explained in the pedigree, not just stated. Specifically, why are they incorrect, and in what circumstances would they be correct?
Response: MNGP agrees that question was UNSAT as written.
- 1.
Simplified the stem as to not queue the examinee as to what TS section applies. Additionally, provided more TS references to add to the plausibility.
- 2.
Distractor pedigree updated.
NRC: Question is now SAT
ES-401 Facility: Monticello /Year: 2020 17 Form ES-401-9 Page 17 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 97 H
3 N
E Early Review Question; Reference Provided NRC:
- 1.
Underlining/emphasizing certain conditions should be avoided.
- 2.
Dont point to a specific provided reference in the stem. If a reference is needed, it is expected that the applicant would look for it.
- 3.
The plausibility of every distractor needs to be explained in the pedigree.
Include why the misconception might reasonably occur.
- 4.
Electrical control is plausible RED - Not a given choice.
Response
- 1.
IAW with NUREG-1021 Appendix B, Attachment 1, #10; it is recommended that key points are underlined or highlighted.
- 2.
NUREG-1021 ES401 and App. B do not inhibit the calling out of a reference in the stem for an Initial Written Exam open reference question.
An open reference question for ILT is not meant to be an exercise in locating procedures. No changes required.
- 3.
Pedigree updated.
- 4.
Electrical control plausible RED - Modified distracter D.
NRC: Question is now SAT 98 H
2 B
S 2013 NRC Exam; Embedded Reference 99 H
2 B
S 2015 NRC Exam 100 F
3 B
E 2018 NRC Exam NRC:
- 1.
Put a summary of the objective in the pedigree where this is knowledge required to be known from memory.
- 2.
Correct spelling - down wind to be downwind per procedure.
Response
- 1. Learning Objective added to pedigree.
- 2. Spelling of downwind corrected.
NRC: Question is now SAT
ES-401 Facility: Monticello /Year: 2020 18 Form ES-401-9 Page 18 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
B= 47 F= 31 E= 35 Additional Notes: Replaced the unsatisfactory question.
M= 8 H= 44 U= 1 N= 20 SRO TOTALS:
B= 17 F= 4 E= 6 Additional Notes: Corrected the unsatisfactory question.
M= 2 H= 21 U= 1 N= 6
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.