ML19122A182
| ML19122A182 | |
| Person / Time | |
|---|---|
| Site: | Ginna  |
| Issue date: | 11/09/2018 |
| From: | NRC Region 1 |
| To: | Exelon Generation Co |
| Shared Package | |
| ML17234A729 | List: |
| References | |
| CAC U01977 | |
| Download: ML19122A182 (7) | |
Text
ES-401 Ginna 2018 Written Examination Review Worksheet Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOO (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- I Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only Minor changes to formatting. Changed to fill-in-the-blank style Q. Added time frame 1
H 2
B E
to stem for clarity. Reworded choices from "requiring a" to "resulting in" for technical accuracy.
2 H
3 B
E Emphasized "basis" with bolding in stem and deleted the" and "procedure" from the Q for clarity.
Attached handout allows elimination of two distractors. Replaced KA and eliminated 3
H 3
X X
B u
the unnecessary handout. Some parameters provided in stem were modified for technical accuracy. Minor stem reformatting.
4 H
3 B
E Modified Part 2 of Q from flow to pressure. Changed components in the choices to ensure only one correct answer.
5 H
3 B
E Minor stem wording change for readability.
6 H
3 B
E Multiple Qs on exam related to this topic. At CE discretion, reselected KIA and replaced with a different bank Q.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOO) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved KIA and those that are designated SRO-only (KIA and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 7
ES-401 Ginna 2018 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOO (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- I Back-Q=
SRO 8/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only 7
H 3
N E
Modified stem and choice wording to improve clarity and ensure technical accuracy.
8 F
2 B
E Added additional plant status item and Oprocedure reference to stem to ensure answer validity.
9 F
2 N
E Removed 2nd part of each choice as unnecessary since correct answer selected based on validity of 1st part.
10 H
4 B
E Minor stem rewording for clarity. Added assumption information to stem to preclude a 2nd potentially correct answer.
Added additional information to stem about indicating lights. As written, the 11 H
3 B
E information was incomplete. Modified two distractor choices for clarity and to balance the choices.
12 F
2 B
E Minor edits to stem and choices for clarity. Changed Q format to fill-in-the-blank.
13 H
3 B
E Corrected locations in the choices.
14 F
2 N
E Removed unnecessary information from the stem.
15 F
2 B
E Changed one stem bullet from a specific value to a trend. The specific value served no purpose.
16 F
2 X
X N
u Implausible boration distractors and multiple correct answers based on vagueness of Q. Replaced with another new Q.
17 H
3 N
E Modified one stem bullet value to improve operational validity of Q. Modified wording of two choices for better balance with other two choices.
Poor stem focus/cue: Only the key answer is a direct "result" of the failure. All three 18 H
2 X
M u
distractors are additional actions that would potentially need to be taken to properly align with the question asked. RCP and SG distractors are not very plausible.
Reworded Q and choices.
19 H
3 N
E The cause of the malfunction in stem was not operational validity. Changed the cause. Replaced one implausible distractor.
3 B
u Q unreasonable and could not write another Q to the KA. Reselected the KA and 20 F
replaced Q with bank Q.
21 F
3 N
E Minor stem and choice wording changes for clarity.
22 H
3 N
E One distractor not plausible given that the described function does not exist at Ginna.
Rewrote Q to address the concern.
Page 2 of 7
ES-401 Ginna 2018 3
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- I Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only 23 F
2 N
E Reworded for clarity. Changed choice wording to eliminate a word that has a specific meaning which conflicts with the key answer.
24 F
3 N
E Changed nomenclature to align with components at station. Minor rewording of stem and choices.
25 F
3 N
E Clarified stem. Modified wording of key answer to make more technically accurate.
Reworded question in two-part format. Modified two distractors slightly to align with 26 F
2 N
E the 2nd part of the question. Changed a stem condition to be consistent with expected conditions during the event.
27 F
2 N
E Minor stem wording change and reformatted for clarity.
28 F
2 N
E Modified Choice D to maintain the 2x2 structure.
29 F
2 N
E Modified stem to tie action to procedure. Modified choices to improve discriminatory value.
30 F
2 N
E Reformatted stem and choices for clarity.
Choice D was implausible with given information in stem. Modified stem and choices 31 H
3 B
E slightly to ensure Choice D is plausible. Added additional information about the leak diagnosis for clarity and to improve operational validity.
32 H
3 N
E Minor wording changes for clarity.
33 F
3 B
E Minor reformatting for clarity.
34 H
2 B
E Minor reformatting. Minor change to the question. Changed two distractors to ensure they couldn't be argued as partially correct.
35 H
2 B
E Minor reformatting of stem. Changed the wording of Part 1 choices.
36 F
3 X
N u
Implausible distractors. All three distractors can be eliminated if the applicant knows that TS consequence of low tank level. Replaced the Q.
37 F
2 B
E Reformatted stem for clarity. Added the word 'requirement' to the stem.
38 H
3 B
E Reworded Q and choices to improve distractor plausibility.
It is not credible to think that a manual SI can be blocked, therefore answers A and C are not plausible. It is not credible to think that a failure to the "block" position results 39 H
2 X
N u
in no change to the functionality of any of the provided SI signals. This leaves answer choice D as the only credible answer. Wrote a new Q to the same KA to address the concern.
Page 3 of 7
ES-401 Ginna 2018 4
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- I Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only 40 F
3 B
E 2nd part of Q appears to potentially cue the first part. Rewrote the Q to ensure no cueing.
There is nothing in the stem to indicate an issue requiring additional service water, so 41 H
2 X
M u
answer choice C is not credible. Also, it is not credible that the current equipment Is adequate based on rising containment pressure. Rewrote the Q.
Choice C is a subset of choice A. If the pumps may be stopped below 28 psig, then 42 F
2 N
E they can certainly be stopped below 4 psig. Therefore, 28 psig must not be the correct answer. Reworded the Q to fix.
Cuing. Question asks how to identify that the TDAWF turbine had tripped and the valve that closes is the one titled "TDAFW Pump Trip Throttle Valve." This makes 43 F
2 X
X B
u distractors A and B not plausible and cues that the correct answer is either C or D.
Also, no technical reference provided for the basis of the 1500 psig in the answer choices. Replaced the Q.
44 H
2 B
E Capitalized "MODE". Bolded and capitalized "AND" in the stem.
45 F
3 N
E Modified stem by removing an alarm and modified Q to enhance plausibility of distractors.
46 H
3 N
s 47 F
3 N
E Modified Q wording to improve distractor plausibility and minor modification to key answer to ensure correct.
48 H
2 N
s 49 F
2 N
E Modified two distractors to improve plausibility and ensure cannot argue a 2nd correc1 answer.
50 F
3 B
E Reworded Q and choices to better align with the KA and to make Q more operationally relevant.
Formatting and minor stem wording changes for clarity. Deleted the second bullet as unnecessary since, per Appendix E, the applicants should assume everything works as designed unless stated otherwise. The answer choices are not homogeneous as 51 F
3 N
E required by NUREG-1021, Appendix B, C.2.h, because two of the choices are in % of rated load and 2 are in KW. If the KW are converted to % of rated load, there is then a reverse cuing issue with answer choice B as it is the only one that is a different rated load. Modified choices to fix homogeneity issue..
52 H
4 N
s Page 4 of 7
ES-401 Ginna 2018 5
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOO (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- I Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only Minor changes to choices to add specificity. Added prints as handout references 53 H
3 N
E because unreasonable to expect applicants to be able to determine Att-2.5 system effects from memory.
54 H
4 N
E Raised initial power level to enhance discriminatory value of a distractor choice.
55 H
3 X
N u
Question KIA was on the containment system (103) but the question tests knowledge of the containment spray system (026). Replaced the Q.
56 H
3 N
E Pressurizer level going higher is not very plausible. Changed Distractor B to improve plausibility. Minor stem rewording for accuracy and clarity.
57 H
3 M
E Adjusted parameter value in stem to ensure to remove any confusion as to what caused P-6 to energize.
58 F
3 N
E Modified stem and choices to ensure cannot argue for a 2nd correct answer.
59 H
3 N
E Minor stem modifications for clarity. Changed Choice D based on required I Prerequisite condition of procedure. Changed Choice A to balance choices.
60 H
2 X
B u
Question is SRO level with fuel handling requirements and operability determination.
Also very similar to Question #63. Reselected KA and wrote new Q.
Added initial information to stem to remove variability associated with applicant 61 H
3 N
E assumptions. Modified values slightly to ensure change sufficient to cause the key answer response. Modified choice wording consistent with normal station practice.
62 F
3 B
E Reformatted Q for clarity.
63 F
3 B
E Minor edits. Reworded one distractor to eliminate potential for argument as 2nd correct answer.
64 H
3 N
E Modified to more straightforward 2x2 structure.
65 F
3 B
E Minor reformatting.
66 F
3 X
M u
Not a Tier 3 question. Question is too specific to the design of the GREATS system.
Reselected the KA and replaced with a modified bank Q.
67 F
2 X
B u
Not a Tier 3 question. Question is too specific to the design of the GREATS system.
Replaced with a bank Q.
68 F
2 N
E Modified choices to ensure distractors are not subsets.
69 F
2 M
s 70 H
3 N
E Minor wording changes for clarity, consistency with station terminology. Changed choice values for accuracy.
Page 5 of 7
ES-401 Ginna 2018 6
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOO (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only 71 F
2 B
E Changed Distractor C to enhance plausibility.
72 H
3 X
B u
Not a Tier 3 Q. Reselected KA and replaced Q.
73 F
2 X
B u
Not a Tier 3 Q. Replaced Q.
74 F
4 B
E Fixed valve identifiers in Choice B.
75 H
3 X
B u
Not Tier 3. This question is an extension of a Tier 2 topic as it requires detailed system knowledge of containment isolation status lights. Replaced Q.
76 H
2 N
E Reformatted stem and minor wording changes for clarity. Rewrote distractors to improve plausibility.
77 F
2 N
E Minor format changes.
78 H
4 X
N u
Not SRO-only. Modified to test SRO-only knowledge. Also added stem information to ensure eliminate variability of answers based on otherwise valid assumptions.
79 H
3 N
E Modified stem information and a distractor to enhance distractor plausibility.
80 H
3 N
E Removed unnecessary stem information.
81 H
4 N
E Modified stem conditions to eliminate potential for arguing 2 correct answers based on assumptions made by aoplicant.
82 H
3 N
s Revised stem to provide a specific tank discharge in progress and to imbed a picture 83 H
2 N
E of the monitor such that applicant can evaluate meaning of indication rather than being told.
84 H
4 X
N u
Two correct choices, both A and C. Wrote a new Q.
Not SRO level because it tests knowledge of</= 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> TS (or TRM) action statements as discussed in the optional SRO guidance attached in ES-401. Also, the 85 H
3 X
N u
answer could be argued as incorrect because it doesn't specify performance of the fire watch inspection within the hour, only establishing an hourly fire watch. Wrote a newQ.
86 H
4 B
E Reworded distractors to improve plausibility.
87 H
3 X
B u
Q can be answered with RO knowledge. Not SRO-only. Replaced Q.
88 H
3 X
N s
89 H
3 N
E Modified choices to improve balance and distractor plausibility.
Page 6 of 7
ES-401 Ginna 2018 7
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
- 8.
Q#
LOK LOO (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- I Back-Q=
SRO B/M/N U/E/S Explanation Focus Dist.
Link units ward KIA Only KIA mismatch - question does not test limitations of facility license, question just asks 90 H
3 X
X N
u which procedure to use to restore the diesel. Question tests minutia of details contained deep in an ARP rather than conditions/limitations on the plant license.
Replaced with a new Q.
91 F
2 B
E Changed one stem parameter to be consistent with expected plant response to the postulated event.
92 H
2 N
s 93 H
3 B
E Added clarifying information to stem. Modified key answer to make correct under all conditions. Changed Distractor D to balance with Choice A.
Cue: Question asks about reason for min boron concentration, and the answer is 94 F
3 X
X N
u boron dilution Distractors are not plausible - no reactivity impacts associated with these events. Replaced the Q.
95 F
2 B
E Minor changes to choices for technical accuracy.
96 H
2 B
s 97 F
2 B
E CE replaced Q to better balance the exam.
98 F
3 B
E Minor stem clarification.
99 F
3 X
N u
Not a Tier 3 Q. Replaced Q.
100 H
4 N
E Minor wording and formatting changes for clarity.
Page 7 of 7