ML19275H185
| ML19275H185 | |
| Person / Time | |
|---|---|
| Site: | Braidwood  |
| Issue date: | 06/19/2019 |
| From: | Jesse Seymour Exelon Generation Co LLC |
| To: | NRC/RGN-III/DRS/OLB |
| Seymour J | |
| Shared Package | |
| ML17214A825 | List: |
| References | |
| Download: ML19275H185 (18) | |
Text
ES-401 Facility: Braidwood (2019)
Written Examination Review Worksheet Form ES-401-9 Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3. Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length).
T/F: The answer choices are a collection of unrelated true/false statements.
Cred. Dist>: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
- /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only. (K/A and license-level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
H 2
M S
2 H
3 B
S 3
F 2
B S
Free Look Question.
4 H
3 B
S 5
H 2
N S
6 F
2 X
N U
S NRC: The first half of distractors A and B are not plausible. Given the stem conditions, the plant is not in a condition where ECCS suction is being taken from the CV sump. Additionally, the stem conditions do not establish conditions under which the EOP network would be expected to be in effect. Furthermore, a potential overlap concern is created by testing knowledge related to both RH pump amperage and BwCA-1.3 on questions #6 and #32.
ES-401 Facility: Braidwood (2019) 2 Form ES-401-9 Page 2 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Facility: Revised stem and distractors A and B to continue in 2BwGP 100-1, plant startup. This addresses the overlap concern and the distractor plausibility concern.
NRC: The revised question is satisfactory.
7 H
2 N
E S
Reference provided.
NRC: This question does not appear to match the listed K/A statement, specifically in that the question does not address knowledge related to the Loss of Nuclear Service Water. Additionally, for questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided. Furthermore, as an enhancement related to the overall number of questions on the RO section of the exam that have provided references, this question should either be modified to remove embedded reference material from its stem or be replaced with a question having no provided reference material associated with it.
Facility: The KA for this question is loss of component cooling water not loss of nuclear service water (which is the KA for question #14) and should therefore not be marked as unsat due to KA match, the incorrect KA was inadvertently input due to an inappropriate software issue in the exam writing software. The picture reference in the stem has been replaced with a statement of current CC surge tank levels and all answers have been revised to state levels vice interpretation of pictures.
NRC: Based upon the actual K/A statement, it is agreed that a K/A mismatch does not exist for this question. Thus, this question should be treated as an enhancement versus being unsatisfactory. The revised question is satisfactory.
8 H
2 B
S Unit Differences Question (Unit 2 7300 controls).
9 H
3 B
S 10 H
2 N
E S
Reference provided.
NRC: The distractor analysis information for distractors A and B appears to be reversed. Additionally, for questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided.
Facility: Corrected distractor analysis, added information in ES-401-5 to show the reference provided in the stem.
NRC: The revised question is satisfactory.
11 F
2 B
E S
NRC: The distractor analysis for distractor D contains a typographical error.
Specifically, the word may is omitted in the third sentence (after the word examinee).
ES-401 Facility: Braidwood (2019) 3 Form ES-401-9 Page 3 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Facility: Corrected typo as requested.
NRC: The revised question is satisfactory.
12 H
3 B
S 13 H
2 N
E S
Reference provided.
NRC: For questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided.
Facility: Corrected ES-401-5 as requested.
NRC: The revised question is satisfactory.
14 H
F 3
B E
S NRC: The LOK of this question is at the fundamental level versus the listed higher cognitive level; the applicant can arrive at the correct answer solely by recalling information from BD-CA-0.0.
Facility: Corrected LOK as requested.
NRC: The revised question is satisfactory.
15 H
2 X
B E
S NRC: Distractor D is not plausible; 1CV8105 is a MOV, thus making it unreasonable to assume that an applicant would have a misconception regarding the response of this valve in a loss of instrument air scenario.
Facility: Changed distractor D to 1CV8142, NO.1 SEAL BYP ISOL VLV, which is an AOV inside containment.
NRC: The revised question is satisfactory.
16 H
3 N
S 17 H
3 N
S Unit Differences Question.
18 H
2 B
S 19 H
2 N
E S
NRC: The current wording of the question may need to be clarified. The question currently asks what switch position shows the maximum concentration boric acid flow being delivered, via 1CV8104, EMER BORATION VALVE, after the completion of step 4? Is it assumed that the applicant will know that step 4 (not an immediate action step) directs them to start the Boric Acid Transfer Pump?
Facility: The high-level action of Step 4 is listed as the last condition in the stem prior to the interrogative. Added switch title to interrogative to be consistent with other questions.
NRC: The revised question is satisfactory.
20 H
2 X
N E
NRC: Distractor A is not plausible; since the emergency boration pathways are
ES-401 Facility: Braidwood (2019) 4 Form ES-401-9 Page 4 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only F
S supplied directly to the suction of the CV pumps, and not to the VCT, it is unreasonable to assume that an applicant would have such a misconception regarding the basis for the action associated with this question.
Facility: Revised question to create a third plausible distractor as required. These changes resulted in the question being a memory LOK.
NRC: The revised question is satisfactory.
21 F
3 B
S 22 F
2 B
E S
NRC: The last portion of distractor D (and initiates a Containment Vent Isolation) appears to be unnecessary, as well as inconsistent with the structure of the other answer/distractor options. Additionally, it would be desirable to adjust the wording on answer B so as to remove the comma located within the statement, as this is the only answer/distractor option that is grammatically structured in this manner. Furthermore, the distractor analysis for distractor A is inconsistent with the half-correct nature of this distractor as the analysis begins with plausible: an ion chamber is incorrect.
Facility: Removed comma from answer B. Edited last portion of distractor D for uniformity between answers, the last portion is needed as each answer has a detector type and purpose (the to section of the answer). Corrected distractor analysis for distractor A.
NRC: The revised question is satisfactory.
23 H
1 3
M U
S Reference provided.
NRC: This question is a direct lookup, and thus lacks adequate discriminatory value.
The applicant may answer this question solely by applying the directions provided in step 17 of the provided reference to the associated table. No specific knowledge is required on the part of the applicant in order to identify the correct answer.
Facility: Revised question to have a second part that will determine knowledge of BwAP 1100-16 requirements.
NRC: The revised question is satisfactory.
24 H
2 N
E S
Reference provided.
NRC: For questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided.
Facility: Included reference in ES-401-5 as required.
NRC: The revised question is satisfactory.
25 F
2 B
S
ES-401 Facility: Braidwood (2019) 5 Form ES-401-9 Page 5 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 26 F
1 3
N U
S NRC: This question lacks adequate discriminatory value. Specifically, the selection of distractor C or D would require an applicant to not recognize that the contents of the RWST are expected to enter the containment under LOCA conditions; this would be a misconception involving a very basic concept. This low degree of discriminatory validity associated with two out of the four answer/distractor options results in the overall level of difficulty for the question being unacceptably low.
Facility: Revised question as required to test the source of water at the system level per the basis of Step 1 of 2BwFR-Z.2. Revised distractors A and C to remove potential subset issue, heat removal capabilities could be considered as a reason for isolating the flooding. Changed to prevent radioactive materials from being released.
NRC: The revised question is satisfactory.
27 H
2 B
S 28 F
2 M
S Previous 2 NRC Exams (2018) Question (Modified).
29 H
2 X
M E
S NRC: Distractor C is not plausible; it appears unlikely that an applicant would associate the failure of a primary water valve to open (itself a misconception within the context of this question) with a resultant higher-than-expected RCS temperature.
Additionally, the wording in the second half of the two-part question is awkward, specifically in that the last portion of it states than expected, without a failure. It would be clearer to, for example, word this portion as something similar to than would otherwise be expected without such a failure. Finally, the distractor analyses for distractors C and D appear to incorrectly describe the expected responses of 1CV111A and 1CV111B (respectively) to the discussed mode selector switch positions.
Facility: Revised distractor C, stem, and corrected distractor analysis as stated above.
NRC: The revised question is satisfactory.
30 H
1 2
B N
U S
NRC: This question lacks adequate discriminatory value. Identifying answer A only requires the applicant to identify from the stem that the boron concentration in the RH train being placed in service is less than the existing RCS boron concentration and to recognize that a lesser overall boron concentration in the RCS would correspond to a reduced shutdown margin (itself a very basic concept). This low degree of discriminatory validity results in the overall level of difficulty for the question being unacceptably low.
Facility: KA has been rejected and K5.05 has been selected as its replacement. New question has been written for the KA reselect.
NRC: Replacement question is satisfactory.
ES-401 Facility: Braidwood (2019) 6 Form ES-401-9 Page 6 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 31 F
2 N
S Free Look Question.
32 H
2 N
E S
NRC: Distractor B appears to potentially be a second correct answer due to the applicant being able to make unstated assumptions that are not contradicted by information in the stem. Specifically, step 1 of 2BwCA-1.3 directs the operator to stop any CV, SI, or CS pumps that are taking suction from the affected RH train. After this has been completed (if necessary), the operator is then directed to check to see if RH pump amperage has stabilized. It is only if fluctuating amperage still exists at this point (step 1.d RNO) that RH pump flow would be reduced. Since the stem does not provide any information on the running status of the CV, SI, and CS pumps, it appears that an applicant could select either answer A or distractor B depending on what assumptions are made about the status of ECCS pumps. Additionally, see comments for question #6: a potential overlap concern exists due to testing knowledge related to both RH pump amperage and BwCA-1.3 on questions #6 and
- 32.
Facility: The Stem asks if ECCS flow will be reduced not RH pump flow. Stopping any running CV or SI pump will reduce ECCS flow prior to further reducing RH pump flow if oscillations continue. This is consistent with the overall mitigation strategy of BwCA-1.3. Therefore, distractor B is incorrect by the information given in the stem, as the students are expected to know to that oscillating current will require securing the CV and SI pumps. Question #6 was revised and should not longer present an overlap concern.
NRC: Based upon clarification provided by the facility, the concern regarding a potential second correct answer no longer exists. Thus, this question should be treated as an enhancement versus being unsatisfactory. The revised question is satisfactory.
33 H
2 M
S 34 H
3 B
S 35 H
3 M
S Unit Differences Question (Unit 2 7300 controls).
36 H
2 B
S Unit Differences Question (Unit 2 7300 controls).
37 H
2 N
S 38 H
3 B
S 39 F
2 B
S 40 H
2 B
S Previous 2 NRC Exams (2016) Question.
41 F
3 B
S
ES-401 Facility: Braidwood (2019) 7 Form ES-401-9 Page 7 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 42 H
2 X
B U
S NRC: Distractor D appears to potentially be a second correct answer due to the applicant being able to make unstated assumptions that are not contradicted by the stem. Specifically, in the second half of the question, the applicant is asked what action(s) will the NSO perform? Distractor D lists 1CS019A/B, which appears to be located in the same flow path as 1CS046A/B from answer A. While 1CS019A/B is not included in the list of valves that 1BwEP-0, step 14.f RNO, directs to verify, the question being asked of the applicant is not worded in such a manner so as to restrict their potential options to this list. This factor, in conjunction with the first half of all answer/distractor options appear to be correct as well, results in distractor D becoming another potentially correct answer option.
Facility: Changed 1CS019A/B to 1CS007A/B to resolve this issue.
NRC: The revised question is satisfactory.
43 F
H 2
B E
S Reference provided.
NRC: The LOK of this question is at the higher cognitive level versus the listed fundamental level; determining the correct answer requires the applicant to analyze and trend provided data, in addition to applying the supplied graph.
Facility: Revised LOK as directed.
NRC: The revised question is satisfactory.
44 H
2 N
E S
Unit Differences Question (Unit 2 7300 controls).
NRC: The stem conditions should be clarified so as to describe that the 1B and 1C Main Feed Pumps were initially running, however, the discussed trip annunciator has just been received. Furthermore, additional technical reference material will be required to support the different unit responses associated with the answer/distractor choices corresponding to the first half of the two-part question.
Facility: Revised stem as noted above additional technical information will be provided.
NRC: The revised question is satisfactory.
45 F
2 M
S 46 F
2 X
B E
S Previous 2 NRC Exams (2016) Question.
NRC: This question contains a cue resulting from a difference in the wording used in the answer when compared to that of the distractors. Specifically, all three distractors begin with main feed line while, conversely, answer C begins with tempering feed line, thus creating a lack of balance in the answer/distractor choices. Due to this question having previously appeared on an NRC exam at the facility, this flaw is being categorized as an enhancement (versus being unsatisfactory).
Facility: Revised distractor A to tempering feed line vice main feed line to resolve the stem cueing issue.
ES-401 Facility: Braidwood (2019) 8 Form ES-401-9 Page 8 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only NRC: The revised question is satisfactory.
47 H
3 B
S 48 F
H 3
B E
S NRC: The LOK of this question is at the higher cognitive level versus the listed fundamental level; determining the correct answer requires the applicant to analyze the effect that a failure will have on the operation of a plant component, necessitating multiple mental steps on the part of the applicant (e.g. understand the general effects of blown control power fuses on switchgear, understand what the expected response of a load breaker on the affected bus would have otherwise been under the conditions presented, apply the aforementioned knowledge to analyze the sequence of events presented in the stem, etc.).
Facility: Changed LOK as required.
NRC: The revised question is satisfactory.
49 H
2 B
S 50 H
3 N
E S
NRC: This question does not appear to match the listed K/A statement, specifically in that the question does not address knowledge associated with the loss/malfunction of a DG as it relates to the automatic loader/sequencer and systems powered by the DG. Instead, this question solely tests knowledge related to ECCS systems and to the content of 1BwEP ES-1.4. Additionally, the information provided in the stem does not appear to support the loss of power scenario described in the distractor analysis (e.g. the necessary combinations of loss of offsite and DG sources to the emergency buses).
Facility: Edited distractor analysis as requested. KA match determined to be sat after discussion.
NRC: Based upon clarification provided by the facility, the concern regarding a potential K/A mismatch no longer exists. Thus, this question should be treated as an enhancement versus being unsatisfactory. The revised question is satisfactory.
51 F
2 N
S 52 F
2 M
S 53 F
3 N
S 54 F
2 M
E S
Free Look Question.
NRC: This question is written in a low level of knowledge manner (refer to NUREG-1021 Appendix B, Attachment 2, section A.4). While the K/A statement is met by this question in its current form, this question should be enhanced by making it more operationally valid (refer to Appendix A, section C.2).
ES-401 Facility: Braidwood (2019) 9 Form ES-401-9 Page 9 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Facility: Revised question to be more operationally valid by asking where to reset the lockout device on a loss of offsite power for the U0 SAC.
NRC: The original level of knowledge concern that was previously discussed has been resolved by improving the operational validity of this question. However, the stem conditions in the revised version of this question should be clarified so as to describe that both units were operating at 100%, however, a loss of offsite power has just occurred on both units. Additionally, the order in which the answer/distractor options are presented should be adjusted such that the answer (143) becomes option A (and, correspondingly, the 144 distractor becomes option B) in order to avoid creating a potential specific determiner. The discussed ordering is the same as that provided in the distractor analysis and it is assumed that the distractor analysis ordering of answer/distractor options was the intended order.
Facility: Revised as recommended. Changed wording in stem to make LOOP after the initial unit status.
NRC: The revised question is satisfactory.
55 F
2 B
E S
NRC: This question appears to require reverse logic/application compared to the job requirements of an RO. Refer to NUREG-1021 Appendix B, section C.1.h and, section G. Specifically, this question that asks the applicant for information that is normally received (in this instance, a status light) while providing the examinee with information that they normally have to supply (specifically, the diagnosis that a containment isolation has occurred). Additionally, this bank question does not match question #55 on the 2006 Byron exam; it does, however, correspond to question #55 on the 2006 Braidwood exam. The listed bank information for this question should be updated accordingly. Due to this question having previously appeared on an NRC exam, this flaw is being categorized as an enhancement (versus being unsatisfactory).
Facility: Updated question pedigree to reflect NRC exam use. The status lights are driven by valve positions, essentially the question asks which of these valves close on a phase A. The KA for this question is the ability to monitor automatic operation of the containment system including: containment isolation. The status lights are checked in BwEP-0 to verify containment isolation phase A has actuated.
NRC: Based upon clarification provided by the facility, the concern regarding potential backward logic no longer exists. Thus, this question should be treated as an enhancement with regard to the question pedigree information only. The revised question is satisfactory.
56 F
3 M
S Unit Differences Question (Unit 2 7300 controls).
NRC: The LOK of this question appears to be at the higher cognitive level versus the listed fundamental level; determining the correct answer requires the applicant to analyze multiple indications and diagnose the type of failure that would be consist with each indication (a process necessitating multiple mental steps beyond the simple
ES-401 Facility: Braidwood (2019) 10 Form ES-401-9 Page 10 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only recollection of information).
Facility: The examinee could recall the plant response for the impulse pressure channel failure, from training material and apply that to the conditions in the stem from memory. This would be similar to the logic applied for the comment on question
- 86.
NRC: Based upon clarification provided by the facility, it is agreed that that the LOK of this question is at the fundamental level.
57 H
2 N
E S
Reference provided.
NRC: For questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided.
Furthermore, as an enhancement related to the overall number of questions on the RO section of the exam that have provided references, this question should either be modified to remove embedded reference material from its stem or be replaced with a question having no provided reference material associated with it.
Facility: Revised question to remove picture of indications as required. A list of pressurizer level is now given therefore no reference material exists for the question.
NRC: The revised question is satisfactory.
58 F
2 N
S 59 H
3 M
S Unit Differences Question (Unit 2 7300 controls).
60 H
2 N
E S
Reference provided.
NRC: For questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided.
Facility: Added note to reference on ES-401-5 as required.
NRC: The revised question is satisfactory.
61 H
2 B
S 62 F
3 X
B E
S NRC: This question is flawed due to the stem lacking sufficient focus to elicit the correct answer. Specifically, the stem states that the plant is at 100% power.
1BwGP 100-3 provides direction that level control should be from the 80 series (West) controllers at lower powers... At higher powers with all recircs closed transfer to the lowest indicated level. Based up this procedural direction (in conjunction the plant conditions provided in the stem) it appears that the first half of the two part question could be either East or West based upon whichever level happened to be indicating lower. Due to this question having previously appeared on
ES-401 Facility: Braidwood (2019) 11 Form ES-401-9 Page 11 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only an NRC exam, this flaw is being categorized as an enhancement (versus being unsatisfactory). Additionally, although this question is listed as having previously appeared as 2013 Braidwood NRC Exam #8, it was on the aforementioned exam as question #7 instead. Furthermore, the original bank version of this question also included the wording at this power level in the stem; this additional wording is missing from the current version of the question.
Facility: The at this power level statement was removed from the stem since the power level is stated as being 100%. Revised power level to be 20% in the stem to address the comment about the note from 1BwGP 100-3 being vague for high power levels. This changed the correct answer from C to A. Re-worded plausibility statements to account for this change.
NRC: The revised question is satisfactory.
63 F
2 B
E S
NRC: The explanations contained in the distractor analyses for distractors A, B, and C do not appear to be aligned with the distractors themselves (e.g. the explanations appear to be out of order). Additionally, this question overlaps excessively with Cert Exam question #9 (which also uses the same K/A statement). Specifically, both questions test limits associated with Gas Decay Tank hydrogen and oxygen concentrations. One of these two questions will require replacement to eliminate this overlap concern.
Facility: Cert exam question #9 has been removed and a new KA was selected. The KA is 071 K3.05 Waste gas disposal ARM and PRM systems. No overlap will exist with this newly selected KA. Revised distractor analysis as required.
NRC: The revised question is satisfactory.
64 F
2 B
S 65 H
2 X
B E
S NRC: Distractor A is not plausible; since the purpose of jockey pumps is to maintain fire header pressure, it is unlikely that an applicant would believe that a fire pump would be sequenced to start prior to a jockey pump under conditions of falling fire header pressure.
Facility: Changed distractor A to only the 0A jockey pump to address this issue.
NRC: The revised question is satisfactory.
66 F
3 B
S 67 H
3 B
S 68 F
2 B
S 69 H
3 N
S Reference provided.
ES-401 Facility: Braidwood (2019) 12 Form ES-401-9 Page 12 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 70 F
5 3
X N
E S
Free Look Question. (counted as an enhancement versus unsatisfactory)
Reference provided.
NRC: The first half of the question requires the recall of knowledge (e.g. 4000 versus 4030 series) that is too specific for a question without a provided reference. This also has the effect of making the overall difficulty of this question too high.
Facility: Revised to include a sample print from each series as requested.
NRC: The revised question is satisfactory.
71 H
2 B
E S
Previous 2 NRC Exams (2018) Question NRC: The wording in the stem should be clarified to indicate that only the evolution with the lowest radiological potential is being sought by asking which ONE of the following evolutions has (or similar wording to that effect).
Facility: Corrected stem wording as required.
NRC: The revised question is satisfactory.
72 F
2 B
S 73 H
F 2
X B
U S
NRC: This question does not match the listed K/A statement, specifically in that the question does not address knowledge related to radiological safety procedures pertaining to licensed operator duties, such as response to radiation monitor alarms, containment entry requirements, fuel handling responsibilities, access to locked high-radiation areas, aligning filters, etc. (emphasis added). Instead, this question tests knowledge of radiation and contamination area posting requirements, knowledge of which is not specific to licensed operator duties (rather, it is knowledge required of all facility employees). Additionally, although this question is listed as being a bank question that previously appeared as 2006 Braidwood NRC Exam #72, it does not correspond to the actual question #72 that appeared on the exam in question. Based upon this consideration, this flaw is being categorized as unsatisfactory (versus an enhancement).
Facility: Replaced question as required.
NRC: Replacement question is satisfactory.
74 F
3 B
E S
(comments redacted due to question containing security-related content)
ES-401 Facility: Braidwood (2019) 13 Form ES-401-9 Page 13 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 75 F
2 X
B E
S Free Look Question. (counted as an enhancement versus unsatisfactory)
NRC: The stem lacks sufficient focus to elicit the correct answer (e.g. more information is needed). In the event that a Critical Safety Function Status Tree yellow path condition existed, it would then be possible to have the associated BwFR procedure be assigned a lesser priority than the other listed procedure types (since entry into the yellow path functional restoration procedure would likely be discretionary on the part of the Unit Supervisor). Such an assumption, in combination with the remainder of the stem statement, would yield a situation in which it may be concluded that no correct answer exist for this question. Note that the facility that this bank question was modified from was a BWR and thus had considerable procedural differences in comparison. The stem needs to be revised in order to avoid the potential for such an assumption being reasonably made.
Facility: modified stem to include a red path condition existing for core cooling to address comment.
NRC: The revised question is satisfactory.
76 H
2 Y
N S
Reference provided.
77 H
2 Y
N S
Reference provided.
78 H
3 Y
B E
S NRC: The ordering of the information in the distractor analysis does not correlate to answer/distractor options A, B, and C as they are presented in the question. For example, while the question lists A as being the answer, the distractor analysis lists option C as being correct.
Facility: Corrected distractor analysis as required.
NRC: The revised question is satisfactory.
79 H
3 Y
N E
S Free Look Question.
NRC: The bulleted statement Rod control is in MANUAL in the stem appears to be unnecessary. Unless there is an underlying technical reason why this information was included, then it should be removed from the question.
Facility: removed rod control status from stem as recommended.
NRC: The revised question is satisfactory.
80 H
3 Y
B E
S NRC: The technical references listed for this question should also include TS 3.5.2 and it associated basis. Additionally, there is a typographical error in the distractor analyses for distractors A and B. Specifically, the second sentence of each contains the wording are met. Is incorrect.
Facility: Added reference and corrected distractor analysis as required.
NRC: The revised question is satisfactory.
ES-401 Facility: Braidwood (2019) 14 Form ES-401-9 Page 14 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 81 H
2 B
E S
Reference provided.
NRC: The distractor analysis contains a typographical error. Specifically, the distractor analysis for distractor D is duplicated. Additionally, for questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided. Furthermore, as an enhancement related to the overall number of questions on the exam that have provided references, this question should be modified to remove embedded reference material from its stem.
Facility: Removed duplicate distractor analysis and reference as required.
NRC: The revised question is satisfactory.
82 H
3 Y
B E
S Reference provided.
Unit Differences Question (Unit 2 7300 controls).
NRC: The distractor analysis should include discussion of relevant differences in plant response between the two units as it relates to the scenario in question and distractor plausibility. Additionally, for questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided. Furthermore, as an enhancement related to the overall number of questions on the exam that have provided references, this question should be modified to remove embedded reference material from its stem.
Facility: Added discussion for distractor C as it relates to ovation. This question should retain the graphic provided in the stem as it is more operationally valid for the operators to diagnose. Question allowed to retain stem reference picture based upon further discussion with chief examiner.
NRC: The revised question is satisfactory.
83 H
2 X
Y N
E S
Free Look Question.
NRC: Distractor D is not plausible; it is unrealistic to assume that an applicant would believe that essentially not taking any action in light of degrading condenser vacuum would be the correct answer.
Facility: Modified distractor D (and changed it to distractor C for psychometrics) to check DEHC - MW OUT SELECTED per step 2 MONITOR CONDENSER PRESSURE.
NRC: The revised question is satisfactory.
84 H
3 X
Y B
U S
NRC: Distractor C appears to potentially be a second correct answer due to the applicant being able to make unstated assumptions that are not contradicted by the stem. Specifically, since the conditions in the stem indicate that a Critical Safety Function Status Tree yellow path condition exists (specifically due to high pressurizer
ES-401 Facility: Braidwood (2019) 15 Form ES-401-9 Page 15 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only water level), a condition is thereby established in which 1BwFR-I.1 could be entered on a discretionary basis by the Unit Supervisor.
Facility: Revised stem condition to lower Lpzr, this ensures the entry criteria to 1BwFR-I.1 are not met but a higher Lpzr than normal still exists.
NRC: The revised question is satisfactory.
85 H
F 2
X Y
N U
S NRC: This question contains a cue resulting from a difference in the wording used in the answer when compared to that of the distractors. Specifically, all three distractors begin with continue in 1BwEP ES-0.2 and direct the while, conversely, answer D begins with transition to 1BwEP-0, REACTOR TRIP OR SAFETY INJECTION, thus creating a lack of balance in the answer/distractor choices. Additionally, this question also contains a cue resulting from the provided indication of the occurrence of a safety injection in the stem (1-BP-4.1, SI ACTUATED) in conjunction with the unique option of transition to 1BwEP-0, REACTOR TRIP OR SAFETY INJECTION that is provided by answer D. Furthermore, the LOK of this question is at the fundamental level versus the listed higher cognitive level; the applicant can arrive at the correct answer solely by recalling the caution statement regarding SI actuation that precedes 1BwEP ES-0.2 step 1.
Facility: Changed distractor C to address these issues.
NRC: The revised question is satisfactory.
86 H
3 Y
B E
S Previous 2 NRC Exams (2016) Question.
NRC: Distractor D currently uses a VCT level value of 37% (which appears to correspond to the VCT automatic makeup setpoint). As an enhancement, it appears that a better VCT level value to use might be 20% instead (which appears to correspond to the VCT low level alarm).
Facility: Changing to 20% could introduce an unstated assumption that would cause this to be a second correct answer. At 37% VCT makeup should start and VCT level should begin to rise. By changing the value to 20% that could show that VCT makeup was not successful. This could make distractor D a second correct answer using previous logic related to unstated assumptions in the stem. Updated distractor analysis.
NRC: Based upon clarification provided by the facility, the existing VCT level value will be retained. Thus, this question should be treated as an enhancement with regard to the distractor analysis information only. The revised question is satisfactory.
87 H
2 Y
B S
88 H
3 Y
N S
89 H
3 Y
B S
Reference provided.
ES-401 Facility: Braidwood (2019) 16 Form ES-401-9 Page 16 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 90 H
3 Y
B S
Unit Differences Question (Unit 2 7300 controls).
91 H
2 Y
B S
92 F
3 Y
B S
Free Look Question.
NRC: The version of question #92 submitted with the proposed exam does not match the previously reviewed free look version of the question. The attributes reflected on this form still reflect the version of the question submitted as part of the free look.
The facility will need to clarify which version of the question is being proposed for the exam.
Facility: Question submitted for early review will NOT be utilized. The question included in the submittal was revised after the free look was submitted by a subsequent validation, due to wording that may cause the first part of distractors A and B to be interpreted as correct. The submitted question removed this psychometric flaw and raises operational validity.
NRC: The replacement question is satisfactory and the associated 401-9 attributes have been updated.
93 F
2 Y
N S
94 F
2 Y
N S
95 F
2 Y
B S
Free Look Question.
96 H
2 X
Y N
E S
Free Look Question.
NRC: Distractor C is not plausible; it is unrealistic to assume that an applicant would deem a wind speed of 20 MPH as warranting severe weather actions. Additionally, the distractor analysis for distractor C does not match the distractor itself; the facility will need to clarify whether or not the distractor discussed in the analysis was intended to be the distractor in lieu of that currently shown in the question.
Facility: modified stem for the National Weather Service issuing a severe thunderstorm watch.
NRC: The revised question is satisfactory.
97 H
1 2
Y N
U S
Reference provided.
NRC: This question is a direct lookup, and thus lacks adequate discriminatory value.
The applicant may answer this question solely by applying the information provided the stem to MA-AA-716-012, Attachment 1 (page 30 of 35). No specific knowledge is required on the part of the applicant in order to identify the correct answer.
Additionally, the wording of the question can be improved by either adding an and to option D or by placing the word and after motor current in the question instead of in the answer options.
ES-401 Facility: Braidwood (2019) 17 Form ES-401-9 Page 17 of 18 Q#
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
B/M/N
- 7.
U/E/S
- 8.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only Facility: Replaced question with a new question.
NRC: The replacement question is satisfactory.
98 F
2 Y
M S
99 F
3 X
Y B
N U
S Previous 2 NRC Exams (2016) Question.
NRC: This question does not match the listed K/A statement. Although this question is a bank question that previously appeared as 2016 Braidwood NRC Exam #85, it is now associated with a different K/A statement on the current exam. Specifically, the previous usage of this question on an NRC exam was associated with K/A 000037 (SG tube leak)/Generic 4.46 (ability to verify that the alarms are consistent with the plant conditions), while the current usage of this question is now associated with only Generic 4.46 (e.g. Tier 3). NUREG-1021 ES-401, section D.2.a, states to ensure that the questions selected for Tier 3 maintain their focus on plantwide generic K/As and do not become an extension of Tier 2. As specific systems knowledge is required to eliminate distractors and select the correct answer for this question (i.e. radiation monitor and steam generator systems knowledge), the plantwide generic focus required for the current K/A statement is not being met.
Additionally, based upon this change in the associated K/A statements between exams, this flaw is being categorized as unsatisfactory (versus an enhancement).
Facility: Replaced question with a new question.
NRC: The replacement question is satisfactory.
100 F
3 Y
B S
ES-401 Facility: Braidwood (2019) 18 Form ES-401-9 Page 18 of 18 RO TOTALS:
B= 41 40 (53.3%)
F= 32 E= 28 Additional Notes: 4 bank questions on RO exam from previous 2 NRC exams.
M= 11 H= 46 (61.3 %)
U= 6 (8%)
N= 23 24 (32%)
43 (57.3%)
SRO TOTALS:
B= 14 13 (52%)
F= 8 E= 8 Additional Notes: 2 1 bank question on SRO exam from previous 2 NRC exams.
M= 1 H= 17 (68%)
U= 4 (16%)
N= 10 11 (44%)
GENERAL COMMENT
S:
- 1.
Average difficulty is __2.3 2.4__ on the RO exam and 2.4 2.5 on the SRO exam.
- 3. For all questions with provided references (including those with reference information such as graphics embedded in the stem), the ES-401-5 information must be updated to list a description of the reference information that is provided. (Note: updates to 401-5 information made by facility and documented in the comments for individual questions on this form).
- 4. The number of questions on the RO portion of the exam that are written at the comprehension/analysis level exceeded the ES-401-6 criteria of 38 to 45; there were initially 46 questions written at the higher cognitive level on this portion of the exam (subsequent revisions have lowered this number to 43, which is acceptable).
- 5. The number of questions on the RO section (initially 10) containing provided reference materials (defined by NUREG-1021 ES-401, section D.2.g, as diagrams, sketches, and portions of facility procedures) exceeds the guideline of FAQ 401.53, which provides a RO section guideline of 4 questions of this type. Based upon the overall level of difficulty of the RO portion of the examination, as an enhancement, it is desired to reduce the number of questions with provided references on the RO section of the exam to a maximum of 8. Consistent with this, notes discussing modifications to remove such provided references have been included for questions #7 and #57. Additionally, to further address the relatively high number of questions having provided references on the overall exam, and in consideration of the fact that all applicants will be taking the senior reactor operator version of the exam (e.g. there are no RO applicants), it is also desirable to, as an enhancement, reduce the number of questions with provided references on the SRO section of the exam to a maximum of 4 (thereby reducing the number of questions with references provided on the overall exam from a total of 16 to 12). Consistent with this, notes discussing modifications to remove such provided references have been included for questions #81 and #82 as well.
(Note: based upon further discussion with facility, the reference associated with question #82 was retained and the reference associated with question #97 was removed instead).
- 6. Unit differences are tested by a total of 9 questions across the overall written examination.