ML17214A411
| ML17214A411 | |
| Person / Time | |
|---|---|
| Site: | Duane Arnold  |
| Issue date: | 06/14/2017 |
| From: | Bielby M Operations Branch III |
| To: | NextEra Energy Duane Arnold |
| Bielby M | |
| Shared Package | |
| ML16308A452 | List: |
| References | |
| Download: ML17214A411 (11) | |
Text
ES-401 Written Examination Review Worksheet (DAEC 2017 ILE) Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only NRC: 2X2 question - Source: 2013 NRC Exam question 1 H 3 M S NRC:
- 1. Although D appears to be the correct answer, the explanation is confusing since 1C08A(C-6) is in.
- 2. How does this match the KA for Cause of loss?
2 H 2 N S Response:
- 1. D explanation-typo, should be 1C08B(C-6).
- 2. Applicant must diagnose that fault is on transformer not bus, therefore may re-energize.
Question is S.
NRC:
- 1. Does the failure of both Feed & Steam flow and the resulting rise in RPV level to level 8 cause the turbine to trip first, or does the EOC-RPT cause an RPS trip on no flow?
3 H 3 N S
- 2. Were the distractors re-ordered maybe? Change distractor explanations.
Response
There is not an RPS trip for no flow.
Question is S.
Instructions (Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question a 1(easy) to 5 ( difficult); (questions with a difficulty between 2 and 4 are acceptable)
- 3. Check the appropriate box if a psychometric flaw is identified:
- Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- Cues: The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc.).
- T/F: The answer choices are a collection of unrelated true/false statements.
- Cred. Dist: The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- Partial: One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- Minutia: The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- # / Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only (K/A and license level mismatches are unacceptable)
- 6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any "U" Status ratings (e.g., how the Appendix B psychometric attributes are not being met).
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only NRC: 2X2 question - Used: 2011 NRC Exam question Is the last bullet required?
4 M 2 B ES Response:
Reworded/removed last bullet in stem.
Question is S.
NRC:
5 M 2 N S No comment.
NRC: Used: 2013-1 EOP Comprehensive???
- 1. What position was the MODE switch in . . .
- 2. KA Match? Maybe if you ask why?
6 M 2 B ES
Response
Question & distractors reworded to closely match KA.
Question is S.
NRC:
Why is distractor A plausible? Only assume whats in stem.
7 H 2 N ES Response:
Changed distractor to be plausible.
Question is S.
Part of 10 question pre-review NRC:
Question, as written, paraphrased: Where is the Backup Instrument Air Compressor located? (Reword to include 8 M 2 N S operation of a local control, and match KA better?)
Response
Rewrote question as two part to raise level of difficulty.
Question is S.
NRC: 2X2 question - Reference Provided Reword stem question to ask Which of the alternate decay heat removal methods is available with the given conditions? (Is there only ONE available alt decay heat 9 H 3 N ES removal method available?)
Response
Corrected stem to ensure only one correct answer.
Question is S.
NRC:
- 1. What is the definition of a refueling accident?
- 2. Why would distractor D (and possibly A) be plausible 10 MH 2 N ES for refueling?
Response
Stem left unchanged. Modify distractors A & D NRC: Reference Provided 11 H 2 N S No comment.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only NRC:
- 1. Distractor D is more plausible if it states, . . .
remaining Main Steam lines.
- 2. This is a higher cognitive level question.
12 MH 3 N ES
Response
- 1. Changed distractor D.
- 2. Changed to higher cognitive question.
Question is S.
NRC:
13 H 3 N S No comment.
NRC: Used: 2017-1 EOP Comprehensive???
The distractors as written cue the applicant that DW pressure should be higher in this condition (3 higher than selections) and this leads to correct answer for a saturated 14 H 2 X B US system.
Response
Changed distractor C to lower - lower to remove cueing.
Question is S.
NRC:
15 M 2 N S No comment.
NRC:
This is simply what occurs at LO, LO-LO, LO-LO-LO- level?
16 HM 3 N ES Response:
Changed to fundamental question.
Question is S.
NRC:
17 M 2 N S No comment.
NRC:
18 H 3 N S No comment.
NRC: 2X2 question - Used: 2015 Audit Exam question 19 M 2 B S No comment.
NRC: 2X2 question Why is the hydrogen pressure given is the curve is not?
20 H 3 N ES Response:
Removed hydrogen pressure bullet from stem.
Question is S.
NRC:
21 H 3 N S No comment.
NRC:
22 H 2 N S No comment.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only NRC:
23 M 2 B S No comment.
NRC:
This is simply recognizing Immediate Actions?
Response
24 HM 2 N ES
- 1. Changed to fundamental question.
- 2. Replaced distractor C.
Question is S.
NRC: 2X2 question - Used: 2013 NRC Exam question 25 H 2 B S No comment.
NRC: Used: 2017-1 Workup Comp question 26 H 2 B S No comment.
Part of 10 question pre-review NRC: Reference Provided Weak KA match to locate and operate when the correct answer is to not operate.
27 M 2 N S
Response
Rewrote the question in 2 part format to better match KA.
Question is S.
NRC:
28 H 2 M S No comment.
NRC:
29 H 2 N S No comment.
NRC:
Delete the . . . , and if not . . . portion of the stem, and change distractor B to read . . . start signal if 1C05B . . .
30 H 2 N ES Response:
Deleted words from stem question and changed the wording in distractor B.
Question is S.
NRC:
31 M 2 N S No comment.
NRC:
Distractors should state that HSBW will be reached within the required time limits. (vice normal time) 32 H 3 N S
Response
Changed distractors.
Question is S.
NRC: 2x2 question Used: 2017-1 EOP Comprehensive 33 M 2 B S No comment.
NRC:
34 H 3 N S No comment.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only NRC: 2X2 question
- 1. How is inserted and on Range 10 different from fully inserted?
- 2. The stem asks the question based upon the given 35 H 3 X B US conditions - cant take more actions after that as part of distractors. Appears A and C are not plausible.
Response
Reworked the question out of the 2X2 format.
Question is S.
NRC:
36 H 2 N S No comment.
NRC: Reference Provided In general, ROs are not required to memorize (know) the TS bases, which is where the 3 cps for one of the TS SR.
37 MH 3 B ES
Response
Provided TS as a handout. Changed to higher cognitive.
Question is S.
NRC: 2X2 question Question does not address the KA - knowledge of how APRMs/LPRMs respond to a flow unit failure.
38 H 2 X N US
Response
Reworded the question to match the APRM element of KA.
Question is S.
NRC:
39 H 3 N S No comment.
NRC:
40 H 3 N S No comment.
NRC:
41 M 2 B S No comment.
NRC: Used: 2013 NRC Exam question 42 M 3 B S No comment.
NRC:
Why, given stem ICs, is distractor D plausible?
43 H 2 N ES Response:
Changed distractor D to . . .B Feed REG Valve. . .
Question is S.
NRC:
44 M 2 N S No comment.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only NRC:
45 H 2 N S No comment.
NRC:
46 M 2 N S No comment.
NRC: 2X2 question - Used: 2009 NRC Exam question. No 47 H 2 B S comment.
NRC: 2x2 question 48 M 2 N S No comment.
NRC: 2X2 question - Used: 2009 NRC Exam question 49 M 2 B S No comment.
NRC: 2X2 question - Used: 2013 NRC Exam question 50 M 2 B S No comment.
NRC:
51 H 3 N S No comment.
Part of 10 question pre-review.
52 M 2 N S No comment.
NRC:
53 H 3 N S No comment.
NRC:
54 H 2 N S No comment.
NRC: 2X2 question 55 H 3 N S No comment.
NRC: Used: 2015 Audit Exam question 56 H 3 B S No comment.
NRC: Original Bank used on 2015 NRC exam.
57 H 3 M S No comment.
NRC:
58 M 2 N S No comment.
NRC: 2X2 question 59 H 3 N S No comment.
NRC:
Stem question was changed to IAW AOP-644, the OATC 60 H 2 N S will perform which of the following actions first. For clarity.
Question is S.
NRC:
61 M 2 N S No comment.
NRC:
62 M 2 N S No comment.
63 M 2 B S No comment.
NRC:
64 H 3 N S No comment.
NRC: Reference Provided 65 H 3 N S No comment.
NRC: Used: 2017-1 Normal Ops RO 66 H 2 B S No comment.
NRC:
67 M 2 N S No comment.
Part of 10 question pre-review. NRC:
C is only a partially correct answer; should add the clarifier
. . . for this activity.
68 M 2 N S
Response
Added clarifying wording to C so provide complete answer.
Question is S.
NRC:
69 M 3 N S No comment.
NRC:
Pull to Lock means will not start, therefore, not available.
(Might be better to tie to ESW or ECCS pump operability.)
70 M 2 N ES
Response
Changed picture to both EDG control switches in Auto.
Question is S.
Part of 10 question pre-review. NRC:
71 M 2 N S No comment.
NRC:
Does not match the KA for knowledge of RM systems, since the applicant is told that a valid RM alarm has occurred and 72 H 3 X N US simply asked where the leak could be coming from.
Response
Replaced question.
Question is S.
NRC: Used: 2015 DEAC NRC Exam Need to select a more discriminating EOP symbol.
73 M 2 N ES Response:
Selected different EOP symbol.
Question is S.
NRC: Used: 2013 Monticello NRC Exam 74 M 2 B S No comment.
NRC:
Change stem condition to a licensed operator not already in the control room. Makes other distractors plausible.
75** M 2 N ES Response:
Changed 2 Initial Conditions and 2 distractors to be plausible.
Question is S.
For RO portion of Exam: New = 53Q, Modified = 3Q, Bank =19Q (25%): LOK: M: 33/75 = 44%, H: 42/75 = 56% S: 56/75 = 75%, E: 15/75 = 20%, U: 4/75 = 5%
ES-401 8 Form ES-401-9 SRO Question Comments
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only NRC:
- 1. The questions pedigree fails to explain why the distractors are plausible OR why they are not correct. The correct answer states that the indications are consistent with a loss of jet pump, however, no other information is provided except the incorrect TS that would apply with the wrong conclusion about the event.
76/1 H 2 Y N E
- 2. The pedigree states that L3.4.1 is to be provided, however, it was not included in the TS references. Please provide for review with this question.
Response
- 1. Distractor explanations were clarified to address both the pedigree and rationale.
- 2. Provided. Question is NOW S.
77/2 M 2 N S Part of the 10 question pre-review NRC: The pedigree for distractor a) is incomplete. It consists of an incomplete sentence.
78/3 H 2 Y N E
Response
Distractor a) explanation completed. Question is NOW S.
NRC: 2X2 question - References Provided - Source: 2011 NRC Exam question
- 2. EAL of ALERT is implausible given supplied references.
Given the 2X2 format, causes 2 implausible distractors.
The same flaw did not exist on the 2011 NRC question.
79/4 H 2 X Y M E
Response
The plausibility statements were lacking but addressed.
ALERT EAL was plausible for this condition because the student must determine that since rad levels have not risen to the alarm setpoint for the given conditions, an ALERT EAL is not required.
The question was revised to address noted deficiencies.
Question is NOW S.
NRC:
80/5 H 2 Y N S No comment.
NRC: References Provided
- 1. The pedigree states that EOP-7, DWSIL is to be provided, however, it was not included in the EOP references.
81/6 H 3 Y N S Please provide for review with this question.
Response
References were provided as handout. The question is S.
82/7 M 2 N S Part of the 10 question pre-review
ES-401 9 Form ES-401-9 NRC: 2X2 question - References Provided
- 1. The use of AOP 262, Loss of Pressure Control is an obvious choice when the stem tells you that Attempts to lower Reactor Pressure were unsuccessful. Thus, with the 2X2 format, 2 distractors are eliminated and then the TS only needs to be reviewed to answer the question.
- 2. The pedigree states that L3.4.10 and L3.2.2 are to be 83/8 H 2 Y N E provided, however, they were not included in the TS references. Please provide them for review with this question.
Response
- 1. Plausible to select the AOP for EHC control of pressure; due to mistakes made in the past. EHC is given in stem.
Distractor clarified.
- 2. References provided as handout. Question is NOW S.
NRC:
84/9 H 3 Y N S No comment.
85/ Part of the 10 question pre-review H 2 N S 10 86/ NRC:
H 2 Y N S 11 No comment.
87/ NRC:
H 3 Y N S 12 No comment.
88/ Part of the 10 question pre-review NRC: Used: 2015-1 Work-H 2 B S up Comp 13 NRC:
- 1. The questions explanations need to be enhanced to justify why the correct answer is right, why the distractors are not, and given the complexity of all the conditions & references, why this knowledge should be known from memory.
- 2. What evidence is provided in the stem to assure that alternate injection sources will be available first?
Response
- 1. Enhanced question explanations. The operators are taught breakpoints in EOPs and are required to know these from 89/ memory. This question is not specific to what alternate injection H 3 Y N E 14** system is required to be aligned so the question is more strategy of implementation than specific technical guidance.
- 2. In accordance with EOP-1 Basis page 54, If the actions being taken in EOP 2 to preserve torus heat capacity are inadequate or not effective, RPV pressure may be reduced to provide additional margin to the Heat Capacity Limit. As in the 3rd Continuous recheck statement discussed above, however, RPV depressurization to mitigate primary containment challenges must be coordinated with core cooling strategies. Full depressurization is appropriate only if adequate core cooling will not be sacrificed as a result. The question is NOW S.
90/ NRC: 2X2 question M 3 Y B S 15 No comment
The question provides TS 3.1.3 and 3.1.6 as references, which yield the 2 plausible distractors. The choices of 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> and 12 91/ hours are not plausible.
H 2 X Y N E 16 Response:
Provided a rewritten question to eliminate the 2 and 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> distractors, however, their plausibility was addressed sufficiently in further discussions. The question is NOW S.
92/ NRC: 2X2 question - Source: 2009 NRC Exam question H 3 Y B S 17 No comment.
NRC:
The question is considered to be a modified bank question, 93/ however, the parent question was not attached in order to verify H 3 Y M E 18 that significant changes were made.
Response
Provided the parent bank question. The question is NOW S.
NRC:
- 1. The question only asks if leaving is permitted, which makes the no answer implausible, since this kind of thing occurs often. The stem should be revised to ask what 94/ needs to be done to allow leaving for emergencies.
M 2 Y B E 19 2. This is a memory-level question.
Response
- 1. Modified the stem to ask if leaving is permitted.
- 2. Changed to fundamental question.
Question is NOW S.
NRC:
This question is flawed as currently written. First, the Site Director and RPM can be easily eliminated from the choices given. The CRS is involved due to confirming TS requirements are met, so it can be argued that CRS approval is also required before core alterations proceed. Finally, it is inconceivable that core alterations would proceed without OSM approval, so this is 95/ the obvious answer.
M 2 X Y N U 20 Response:
Removed the RPM and CRS as choices. Changed to Operations Director and Startup Reactor Engineer. Left Site Director as a choice since this is plausible based upon the fact that the site director and PGM position being synonymous. The Site Director provides permission for various plant activities including permission to startup the reactor. Question is NOW S.
96/
M 2 N S Part of the 10 question pre-review 21
This question is LOD=1 as currently written. The given conditions can be easily determined to exceed the above the line TS LCO 3.6.2.2. Since L3.6.2.2 is provided as a reference, the required action only needs to be read and applied. Since there are no conditions given that support the plausibility of either a SCRAM or ED, 2 of the distractors are easily eliminated.
Response
97/
H 1 Y N U Changed answer choice D to Declare the Primary Containment 22 Inoperable. The applicant may determine that water level this low in the containment makes the containment inoperable and requires actions from tech spec 3.6.1.1. If the applicant is unaware of the Torus water level specification then they may erroneously choose to declare the primary containment inoperable. LCO 3.6.1.1 would therefore be provided as a given reference so a differentiation could be made.
The question is NOW S.
NRC:
98/ S Changes were made to the distractors upon further review by H 3 Y B 23 E the licensee to enhance the previously SAT question. The question is S NRC; Since neither the TSC nor the EOF have assumed command and control (C&C), it is clear that the SM maintains it. Since we are told that the SM declared the ALERT, it is yet another clue that he has C&C. If the 2nd bullet read simply An ALERT was declared due to, it would remove the un-needed cue. Also, 99/
M 2 X Y N E the Site Director distractor is implausible and should be 24 replaced.
Response
Changed one of the distractors to TSC Operations Supervisor from Site Director. Removed the bullet stating the OSM has declared an ALERT based on NRC comment.
Question is NOW S.
NRC:
Given the table and the possible EALs, the applicant needs only to compare the given conditions to the possible EALs. Choosing FU1 or FG1 requires implausible misinterpretation of the given conditions - either ignoring the RCS failure entirely or adding a 100/
H 2 X Y N U failure of the fuel cladding with no symptoms provided.
25 Licensee Agreed - a replacement question will be provided where FU1 or FG1 are not choices (likely a 2x2 where FA1 and FS1 provide half of the answer choices).
Question is NOW S.
For SRO portion of Exam: New = 18Q, Modified = 2Q, Bank =5Q (20%): LOK: M: 6/25 = 24%, H: 19/25 = 76% S: 12/25 = 48%, E: 10/25 = 40%, U: 3/25 = 12%