ML22179A249
| ML22179A249 | |
| Person / Time | |
|---|---|
| Site: | Prairie Island  |
| Issue date: | 10/16/2018 |
| From: | Randy Baker NRC/RGN-III/DRS/OLB |
| To: | Nuclear Management Co |
| Baker R | |
| Shared Package | |
| ML17165A317 | List: |
| References | |
| Download: ML22179A249 (9) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only General note: Majority of the questions to not provide justification for why the correct answer is correct. This is required in support of the distractor analysis.
NRC: Why would Distractor B be considered plausible?
1 H 3 N S Response: Plausible if applicant thinks Train A SI only removes power to the UV coil.
Question is SAT.
2 F 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.
NRC: 1 out-of 2 twice. [Reference Provided-1E-1 pg. 3]
3 H 3 N S Response: Providing the reference requires the applicant to work through the procedure. Question is SAT as written.
4 H 3 B S NRC: Question is SAT as written.
NRC: {Correct K/A IR=4.0}
5 H 2 N S Response: Agreed - Corrected K/A IR. Question is SAT as written.
Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3. Check the appropriate box if a psychometric flaw is identified:
- Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length).
- T/F: The answer choices are a collection of unrelated true/false statements.
- Cred. Dist>: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
- #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only. (K/A and license-level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 9
ES-401 2 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: [2014 ILT Audit Exam] How does this question relate to a PZR PCS U malfunction? (K/A match)?
6 F 2 N B S Response: Agreed -Modified Distractor A to read . . . if PRZR press is less than 2235. Question is now SAT.
NRC: [2014 ILT Exam] Removing Boron will add negative reactivity?!
7 H 2 B S Response: Modified question to remove first part and reorder the distractors.
Question is SAT.
E NRC: {Correct K/A IR=3.1} [Reference Provided-Steam Tables]
8 H 3 N S Response: Agreed - Question is now SAT.
9 H 2 N S NRC: Question is SAT as written.
10 H 3 B S NRC: Question is SAT as written.
NRC: {Original Bank question provided} [2016 ILT Exam Significantly Modified]
11 H 2 M S Question is SAT as written.
E NRC: {Correct K/A IR=3.7}
12 F 2 N S Response: Agreed - Question is now SAT.
NRC: [K/A IR=2.2*; Justification: Important at facility because of loss of remote 13 F 2 N S operating capability (from Control Room) for Safeguards equipment.]
Question is SAT as written.
14 F 3 N S NRC: Question is SAT as written.
NRC: {Need to see the original Bank question to ensure modified.}
15 H 2 M S Response: Agreed - Question is Modified and SAT.
16 H 2 B S NRC: [2017 LOR Exam] Question is SAT as written.
17 F 2 N S NRC: Question is SAT as written.
18 F 2 B S NRC: [2016 ILT Audit Exam] Question is SAT as written.
NRC: [2014 ILT Audit Exam] {Add K/A IR=3.6}
19 F 3 B S Response: Agreed - Question is SAT as written.
20 F 3 N S NRC: Question is SAT as written.
E NRC: [Reference Provided- Fig. C1-20]
21 H 2 N S Response: Removed the reference. Question is now SAT.
22 F 2 N S NRC: Question is SAT as written.
Page 2 of 9
ES-401 3 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: [2017 LOR Exam] {Correct K/A IR=3.1} [Reference Provided-pg. 16 of 2C1.3 23 H 3 B S AOP1 and Fig. C1-10B]
Response: References provided to permit calculation. Agreed - Question is SAT.
NRC: [Reference Provided-pg. 10 of 1FR-C-1]
24 H 2 N S Response: Providing the reference requires the applicant to work through the procedure. Agreed - Question is SAT.
25 F 2 N S NRC: Question is SAT as written.
NRC: [2014 ILT Exam (Similar)] [Reference Provided-1E-0 pg. 5]
26 H 3 B S Response: The reference is provided to preclude reference to minimum AWF flow requirements. Agreed - Question is SAT.
27 F 3 N S NRC: 1 out-of 2 twice. Question is SAT as written.
E NRC: {Correct K/A IR=3.9}
28 F 2 N S Response: Agreed - Question is now SAT.
29 H 3 N S NRC: Question is SAT as written.
30 H 3 B S NRC: 1 out-of 2 twice. [2013 LOR; 2017 LOR Re-Exam] Question is SAT as written.
31 H 3 N S NRC: 1 out-of 2 twice. Question is SAT as written.
32 F 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.
E NRC: 1 out-of 2 twice. {Correct K/A IR=2.9}
33 H 2 N S Response: Agreed - Question is now SAT.
E NRC: [Reference Provided-1C-14 AOP 1 pg. 5]
34 H 2 N S Response: Removed The reference (Immediate action.) Question is now SAT.
35 H 2 N S NRC: Question is SAT as written.
E NRC: 1 out-of 2 twice. [2014 ILT Audit Exam] {Correct K/A IR=3.6}
36 H 3 B S Response: Agreed - Question is now SAT.
E NRC: 1 out-of 2 twice. {Correct K/A IR=3.9*}
37 F 2 N S Response: Agreed - Question is now SAT.
38 H 2 N S NRC: Question is SAT as written.
NRC: 1 out-of 2 twice {Original Bank question provided} [2016 ILT Exam Significantly 39 H 3 M S Modified] [Reference Provided-NF-86186-3,-4]
Page 3 of 9
ES-401 4 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Response: Applicants required to work through the logic diagrams. Agreed -
Question is SAT.
NRC: {Original Bank question provided.} {Correct K/A IR=3.9*} [2016 ILT Exam E
40 H 3 M Significantly Modified]
S Response: Agreed - Question is now SAT.
NRC: {Original Bank question provided.} [2016 ILT Exam Significantly Modified]
E 41 H 3 M [Reference Provided-1E-0 Attachment L pg. 4]
S Response: Remove the reference. Agreed - Question is now SAT.
E NRC: {Correct K/A IR=3.2*}
42 H 2 N S Response: Agreed - Question is now SAT.
43 F 2 N S NRC: Question is SAT as written.
NRC: Which comes first the reactor or the turbine trip? Is there only 1 correct U answer? Is it actually B? (Does the turbine trip cause the Reactor trip at power?)
44 F 2 X X N S Response: Agreed - The stem was reworded to ask what the result will be of a loss of Both MFWP, therefore only one answer is correct. Question is now SAT.
45 H 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.
NRC: [Reference Provided-B20.2 pg. 18]
46 H 2 N S Response: Providing the reference requires the applicant to determine the normal line-up. Agreed - Question is SAT.
E NRC: 1 out-of 2 twice. {Correct K/A IR=2.8*}
47 H 3 N S Response: Agreed - Question is now SAT.
48 F 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.
E NRC: 1 out-of 2 twice. {Correct K/A IR=2.7*}
49 F 2 N S Response: Agreed - Question is now SAT.
E NRC: 1 out-of 2 twice. [Reference Provided-1C20.7 pg. 12-15]
50 H 2 N S Response: Remove reference. (Validate Stem ICs.) Agreed - Question is now SAT.
M E NRC: This question does not appear to be modified.
51 H 2 B S Response: Agreed - This is a B question. Question is now SAT.
52 H 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.
NRC: {Original Bank question provided.} [2014 ILT Exam Significantly Modified]
53 H 3 M S Question is SAT as written.
Page 4 of 9
ES-401 5 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only E NRC: 1 out-of 2 twice. {Correct K/A IR=2.8*}
54 H 2 N S Response: Agreed - Question is now SAT.
NRC: [2015 LOR Exam] [Reference Provided-XH-1-242]
55 H 3 B S Response: Providing the reference requires the applicant to work through the logic diagram. Agreed - Question is SAT.
NRC: [Reference Provided-C47012-0507]
56 H 3 N S Response: Reference provided to support plausibility of distractors. Agreed -
Question is SAT.
57 F 2 N S NRC: Question is SAT as written.
58 H 3 N S NRC: Question is SAT as written.
E NRC: {Correct K/A IR=2.9}
59 F 3 N S Response: Agreed - Question is now SAT.
60 F 2 N S NRC: Question is SAT as written.
NRC: This appears to be a simple limit value memory question not Higher Cog.
H E 61 2 N Is Distractor D (30 Hg) plausible for the facilitys physical location?
F S Response: Agreed -changed Distractor D to 25. Question is now SAT.
NRC: {Original Bank question provided.} [2016 ILT Exam Significantly Modified]
62 H 2 M S Question is SAT as written.
NRC: [Reference Provided-SP1783.1A pg. 6]
63 F 3 N S Response: Providing the reference requires the applicant to determine which annunciators are expected. Agreed - Question is SAT.
NRC: 1 out-of 2 twice. {Original Bank question provided.} [2017 LOR Exam]
64 H 4 M S [Reference Provided-PINGP 0045, C25 Fig. 1]
Response: Matches K/A! Agreed - Question is SAT.
65 F 2 N S NRC: Question is SAT as written.
NRC: 1 out-of 2 twice. {Original Bank question provided.} [2014 ILT Exam Significantly Modified] [Reference Provided-Fig. C1-8, TS 3.1.4 pg. 1]
66 H 3 M S Response: Reference provided to allow calculation for K/A. Agreed - Question is SAT.
B E NRC: 1 out-of 2 twice. {Original Bank question provided.} [2016 ILT Exam 67 H 3 Significantly Modified] This question is modified; why is it listed simply as a B M S question?
Page 5 of 9
ES-401 6 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Response: Agreed - Question is now SAT.
NRC: Asking the question in the per TS 3.6.6 Bases. . . puts this into the SRO-Only arena. Is there another RO knowledge document that also discusses why this is U
68 F 2 X N done?
S Response: Agreed - the purpose is referenced to the procedural requirements.
Question is SAT.
NRC: 1 out-of 2 twice. Again, is there somewhere else that defines what qualified backup equipment is, other than the TS bases? This could also be SRO-Only. {Fix U
69 H 3 X N the grammar in the stem.}
S Response: Agreed - The stem was modified to add load test of portable charger and provided the installation procedure. Question is SAT.
70 H 2 N S NRC: Question is SAT as written.
NRC: How does knowing the general location of an annunciator, demonstrate U knowledge of the process used to track an inoperable annunciator? (K/A match)?
71 F 2 N N S Response: Stem was modified to instruct applicant to follow required procedure.
Question is SAT.
NRC: 1 out-of 2 twice. [Reference Provided-RWP 180051 Tasks 101/203]
72 H 3 N S Response: Applicant must decide which RWP is appropriate and then implement requirements to answer the question. Agreed - Question is SAT.
73 F 3 B S NRC: [2016 ILT Exam] Question is SAT as written.
74 F 2 N S NRC: 1 out-of 2 twice. Question is SAT as written.
75 F 3 N S NRC: {Fix the grammar in the stem. Remove the first in.} Question is SAT as written.
NRC: How does performing the next Step in 1ES-1.1 (which would be performed regardless) relate to verifying the existence of Natural Circulation for this question?
(K/A Match)? [Reference Provided-1ES-1.1] Why is the reference required for an U
76 H 3 N Y N SRO to answer this question?
S Response: Changed distractor to include action if natural circulation is not verified, requiring the use of the reference to support the correct answer. Now matches the K/A. Question is SAT.
NRC: 1 out-of 2 twice. [Reference Provided-TSs-3.5.1 and 3.5.4] Why are the TSs E
77 H 3 Y N provided as references for an SRO to answer this question?
S Response: Agreed - Removed the SRs from the references. Question is now SAT.
78 H 2 Y N S NRC: Question is SAT as written.
79 H 2 Y B E NRC: {Correct K/A Statement for 2.2.40 vs 2.2.22}
Page 6 of 9
ES-401 7 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only S Response: Agreed - Corrected. Question is now SAT.
80 H 2 Y N S NRC: 1 out of 2 twice. Question is SAT as written.
NRC: How does selecting which contingency procedure to enter relate to the ability to identify post-accident instrumentation for this question? (K/A Match)? [Reference U Provided-1E-1 pg. 9-12]
81 H 3 N Y M S Response: The applicant must diagnose the ICs, using the EOP reference to select the proper transition point. Agreed. Changed K/A to random selection - 2.4.4. Re-worked the question to match the K/A. Agreed - Question is SAT.
82 H 2 Y N S NRC: Question is SAT as written.
NRC: [Reference Provided-PINGP 1576]
83 H 2 Y N S Response: Reference does not answer any additional questions and requires the applicant to review all possibilities. Agreed - Question is SAT.
NRC: [Reference Provided-F5 Appendix A Zone 50]
84 H 3 Y N S Response: Reference provided to better match K/A. Agreed - Question is SAT.
85 H 3 Y N S NRC: 1 out of 2 twice. Question is SAT as written.
NRC: Perhaps it would be better to reword the stem to state that The Unit Supervisor E will direct transition into . . . [Reference Provided-1ES-1.2]
86 H 2 N S Response: Removed the bases section of the reference procedure. Agreed -
Question is now SAT.
E NRC: 1 out-of 2 twice. [Reference Provided-PINGP 577 and F3-2.1 pg. 6-F 17]
87 H 3 Y N S Response: Remove the F3 reference. Agreed - Question is now SAT.
NRC: [Reference Provided-1ES-1.2]
88 H 2 Y N S Response: See Question 86. (Already Provided) Question is SAT as written.
NRC: [Reference Provided-1FR-H.1 pg. 3-10]
89 H 3 Y N S Response: Providing the reference requires the applicant to work through the procedure. Agreed - Question is SAT.
NRC: {Correct K/A IR=4.3} [Reference Provided-1ECA-0.0 pg. 6-7]
E 90 H 3 Y N Response: Providing the reference requires the applicant to demonstrate how to S
implement the NOTE. Agreed - Question is now SAT.
91 H 2 Y N S NRC: Question is SAT as written.
E NRC: [Reference Provided-C20.3 AOP1 Fig. 5 and TS 3.8.1]
92 H 2 Y N S Response: Remove the SRs. Fig. OK. Agreed - Question is now SAT.
Page 7 of 9
ES-401 8 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: 1 out-of 2 twice. [2016 ILT Exam] [Reference Provided-H4 Table 2.2]
93 H 3 Y B S Response: Reference requires applicant to diagnose and implement corrective actions. Agreed - Question is SAT.
E NRC: {Correct K/A IR=4.4} [Reference Provided-C35 AOP6]
94 H 2 Y M S Response: Removed the system diagram. Agreed - Question is now SAT.
NRC: [Reference Provided-PINGP 1125 pg. 20-21]
95 H 2 Y N S Response: Reference provided to match the K/A. Question is SAT as written.
NRC: [Reference Provided-FP-OP-RSK-01 pg. 18]
96 H 3 Y N S Response: Table provided as reference required for calculating answer. Window dressing (ICs) provided for Operational validity. Agreed - Question is SAT.
97 H 2 Y N S NRC: 1 out-of 2 twice. Question is SAT as written.
98 H 2 Y B S NRC: [2016 ILT Audit Exam] Question is SAT as written.
NRC: 1 out-of 2 twice. [Reference Provided-C34 AOP1]
E 99 H 3 Y N Response: Limit the procedure provided to the first 20 pages. Agreed - Question is S
now SAT.
100 H 2 Y B S NRC: [2014 ILT Audit Exam] Question is SAT as written.
B=14 F=30 E=21 RO TOTALS: M=10 H=45 U=5 Additional Notes: H = 38-45; N 8; B 56 N=51 B=4 F=0 E=8 SRO TOTALS: M=2 H=25 U=2 Additional Notes: H 13; N 3; B 18 N=19
GENERAL COMMENT
S:
- 1. Question numbers highlighted as xx are sample test items the facility submitted for pre-review.
- 2. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 3. Questions from the previous 2 NRC Exams: _3 (8)_ (RO)/ _1 (2)_ (SRO) [Max (without OLB concurrence) is 4 RO and 2 SRO.]
Page 8 of 9
ES-401 9 Form ES-401-9 Facility: PRAIRIE ISLAND NUCLEAR GENERATING PLANT UNITS 1 and 2 Exam Date: September 10 - 21, 2018
41.2 = 41.9 = 43.2 =
41.3 = 41.10 = 43.3 =
41.4 = 41.11 = 43.4 =
41.5 = 41.12 = 43.5 =
41.6 = 41.13 = 43.6 =
41.7 = 41.14 = 43.7 =
- 6. The answer distribution is:
RO / SRO A = 0 (17%) / 0 (24%)
B = 0 (22%) / 0 (28%)
C = 0 (28%) / 0 (20%)
D = 0 (33%) / 0 (28%)
Page 9 of 9