ML17270A170
| ML17270A170 | |
| Person / Time | |
|---|---|
| Site: | Callaway  |
| Issue date: | 09/20/2017 |
| From: | Vincent Gaddy Operations Branch IV |
| To: | Union Electric Co |
| References | |
| Download: ML17270A170 (11) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 to 5 (easy to difficult) rating scale (questions in the 2 to 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- a. The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- b. The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
- c. The answer choices are a collection of unrelated true/false statements.
- d. The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
- e. One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
- a. The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- b. The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
- c. The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- d. The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any Unsat ratings (e.g., how the Appendix B psychometric attributes are not being met).
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only Generic Comments It is unlikely that 15 of 17 Tier 3 questions are LOD 3. Most of them should be LOD 2.
Disporoportionate number of questions in the RO section have a correct answer of C. If possible, reorder some of the distractors so to get a closer balance to answer distribution.
1 F 2 N S 2 H 2 N S Is it necessary to say "If the break is isolated" because you are cueing that there has been success? Can you instead state that pressurizer E
3 F 3 M level and RCS pressure are rising?
S Changed to "If during the performance of ECA-1.2, RCS pressure and PZR level begin to rise." Rev. 1 question is now SAT.
4 F 2 B S By acknowledging that there will be some flow through train A in distractors C and D, you are cueing that if all works right, then flow will be 3500 U gpm. This cues that the correct answer is either B or 5 H 2 X N S D, if all works right. Could distractor C be 2000 gpm/2000 gpm?
Changed distractor C to "'A' Train is 2000 gpm, 'B' Train is 2000 gpm." Rev. 1 question is now SAT.
6 F 3 B S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only It is LOD knowledge to start a standby pump when the operating pump fails. This is a good question however that I think is easily fixable. Make the A train unavailable, which would make distractor C U correct. Replace distractor B with "INITIATE Actions 7 H 3 N S To Evacuate CTMT And Initiate CTMT Closure" which is the first step of Attachment C.
Changed stem so A train out of service. Reordered distractors. Replaced distractor B, modified distractor D. Rev. 1 question is now SAT.
The distractor analysis states, "Based on the data given in the stem, natural circulation cannot exist."
However, the analysis for distractor B states, "while E natural circulation exists." Am I correct that natural 8 H 3 N S circulation does not exist because of inadequate subcooling and cold leg temperatures too low?
Modified distractor analysis for distractors C and D.
Rev. 1 question now SAT.
Why is it plausible that loss of NPSH would be E imminent when there is still 55% in the RWST?
9 H 3 B S Changed RWST level to 38% in stem. Rev. 1 question is now SAT.
10 H 2 N S 11 H 2 N S 12 F 2 B S 13 H 3 N S Rather than giving a subcooling value, should give a temperature so they have to determine subcooling.
E 14 H 3 N Changed subcooling value to "Highest Core Exit S
Thermocouple is 570°F and stable." Rev. 1 question is now SAT.
15 H 3 B S 16 F 3 N S 17 H 3 N S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only Stem should say "Response to Nuclear Power E Generation / ATWS."
18 H 3 B S Changed stem to "Response to Nuclear Power Generation / ATWS." Rev. 1 question is now SAT.
Opening the reactor trip breakers are not plausible distractors. The part 2 distractors tell you that you don't have to trip the reactor until you exceed, at a U minimum, P-6.
19 F 3 X N S Changed part 2 of stem to "Why is this IMMEDIATE action applicable?" and developed new part 2 distractors. Rev. 1 question is now SAT.
2016 Exam - Correct answer is the only plausible choice. The stem ask for DIRECT conformation, and the three incorrect distractors are all area monitors. The CVCS letdown monitor is the only U process monitor.
20 F 2 X B S Changed question to With the unit at power, what radiation monitor in alarm is an entry condition for OTO-BB-00005, RCS High Activity?" removing reference to direct confirmation. Rev. 1 question is now SAT.
The distractor analysis discusses suspending operations involving positive reactivity, whereas the E
21 H 3 N distractor is discussing tripping the reactor.
S Modified distractor analysis. Rev. 1 question is now SAT.
22 H 3 B S E Distractor B should be P-9, not B-9.
23 H 3 M S Changed B-9 to P-9. Rev. 1 question is now SAT.
24 F 3 N S 25 H 3 N S 26 F 3 N S 27 F 3 N S 28 H 3 B S 29 H 3 N S 30 H 3 N S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only Giving this an E vice a U because it is previously approved by NRC. K/A requires knowledge of parameters to prevent exceeding design limits.
Question as written only requires knowledge of first action (the probable assumption being the applicant knows the parameters of these safety actions.)
Might be better to make this a 2x2 with parameters E
31 H 3 M and only two safety functions such as SI and PORV.
S Applicant still needs to determine whether pressure is actually increasing or decreasing in this scenario.
Removed low and high pressure reactor trips and made the question a 2x2. New 2nd part of question.
Changed question from bank to modified. Swapped the order of part 1 and part 2 for question flow. Rev.
1 question is now SAT.
32 H 3 B S 33 F 3 N S Change the part 2 distractor to eliminate reference to operability. Comment only: I am concerned that the part 2 distractors are SRO-only knowledge.
E However, in the comments section of the question, 34 F 3 N S there is a description of why the licensee believes the question to be RO-level of knowledge.
Changed part 2 of stem to remove reference to operability, and changed question to Rev. 1.
35 H 3 N S The stem should read, for brevity, "To LOWER the RCS Heatup Rate, the RO should either _(1)_ the E output of EJHIC606, or _(2)_ the output of 36 H 3 B S EJFK618."
Modified stem for brevity. Rev. 1 question now SAT.
37 F 3 N S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only Does not meet the K/A. The K/A asks for system logic and reliability of ESF, but the question does not ask this. There is not direct logic that SG level instruments are powered from AC vs DC for reliability. Also, the System in ESF Actuation.
U 38 F 3 X N Knowledge of the number of and power supply of SG S
instruments is not actuation. This question could be repaired if the part 1 distractors required knowing logic instead of numbers.
Modified part 1 distractors to ask AFAS actuation logic. Rev. 1 question is now SAT.
39 H 3 N S 40 F 3 B S 41 F 2 N S Distractor analysis states that no information is available for subcooling, but with CET temps and E
42 H 3 N pressure, one can obtain that info.
S Udated distractor analysis. Rev. 1 quesiton is now SAT.
I find it difficult to believe that anyone at the end of license class would believe that operating spray valves on a solid pressurizer would be correct.
U 43 H 3 X N Changed part 2 distractors to "Lower charging flow S
and raise PZR letdown flow," and "Ensure PZR Htrs are off." Reordered distractors. Rev. 1 question is now SAT.
44 F 3 N S 45 H 3 B S 2014 Exam 46 H 3 M S Reference Provided 47 F 3 B S 48 H 3 N S 49 H 3 B S RO level knowledge of operability is not required.
This is a SRO level question.
U 50 H 3 X N Changed part 2 question to ask whether the LCO S
was met, rather than if the pumps were operable.
Rev. 1 question is now SAT.
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only Should try and balance the pumps in the distractors.
As it is now, three distractors have TDAFP, three distractors have B MDAFP, but only one distractor E has A MDAFP.
51 H 4 N S Added 'A' MDAFP to the distractor of 'B' MDAFP ONLY and then reorganized distractors on length from shortest to longest. Rev. 1 question is now SAT.
52 H 3 N S Should the formatting marks be in the boxes representing the annunciators, or the bolded lines on E
53 H 3 B the border?
S Corrected formatting error. Rev. 1 question now SAT.
Does not meet the K/A. The K/A asks for effect on containment intergrity on the loss / malfunction of containment. The stem gives you a loss of integrity.
U Also, distractor B should be something other than 15 54 F 3 X N S minutes. There are no 15 minte shutdown requiremenets, that I am aware of.
Replaced with a new K/A and a new question. Rev.
1 question is now SAT.
55 H 3 N S The stem does not give the pertinent information to in order to get to the RNO column of step 7. Step 7 asks to check SFP temperature stable or rising, and the stem does not give a trend once it gets to 140°.
I'm not fond of the initiate ESW makeup distractor, E
56 F 3 N because it is obviously for level control.
S Revised 1st part of question making a new correct answer. Removed the distractor of "initiate ESW emergency makeup" Revised second part of question - (stem only) to say "is now 140°F rising."
Rev. 1 question is now SAT.
57 F 3 N S 58 F 3 N S 59 H 3 N S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only By stating that void fraction rises in the stem, you are cueing that NI response will increase. Can the answer be assured to only be "rise" if the stem just U states, "At the onset of core void formation, SR NI 60 H 3 X B S response will initially _(1)_?"
Changed part 1 of stem to state, "At the onset of core void formation, SR NI response will _(1)_?"
Rev. 1 question is now SAT.
61 H 3 N S Would prefer this question be such that T30 = about 650°F, and T29 and T30 be slightly above normal Thot, and the correct answer be subcooled. With T30 currently so much greater than normal Tave and E the other thermocouples, I think one could easily 62 H 3 N S guess that superheated is correct.
Change thermocouple readings to be 610°F, 670°F, 660°F, 610°F, 620°F, and 0°F. Rev. 1 question is now SAT.
63 F 2 N S 64 F 3 N S 65 F 3 N S 66 F 3 N S 67 F 3 N S 68 F 3 B S 69 F 3 N S 70 F 3 N S 71 F 3 B S 2014 Exam Reference Provided. Question is direct lookup, with the reference provided.
U 72 H 2 N Replaced part 2 of stem with stay time question, S
which eliminates the need for the reference. Rev. 1 question is now SAT.
73 F 3 B S 74 F 3 B S 2016 Exam 75 F 3 N S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only Is it possible to put bearing cooler flow out of spec high? Since bearing cooler flow is so much different than the others, I think optically it stands out so E
76 H 4 N much that one would immediately eliminate S
distractors A and D.
Replaced with a new K/A and a new question. Rev.
1 question is now SAT.
Distractor D easily eliminated, as it is LOD 1 that the NRC would have to be informed if invoking 50.54x.
E Consider replacing this K/A, as it is a duplicate K/A 77 H 2 N S to question 85.
Replaced with a new K/A and a new question. Rev.
1 question is now SAT.
Distractor D is cued by the statement "due to loss of subcooling." The distractor analysis talks about changing RCS temperature from 300°F (in the Diablo Bank) to 384°F, yet the stem says 390°F E
78 H 3 B Changed the stem to ask, "Based on the above S
conditions, what is the NEXT sequence of procedures to be utilized?" Updated distractor analysis regarding temperature for superheating.
Rev. 1 question is now SAT.
79 H 3 N S Reference Provided Distractor D should give the title of the procedure, and not the title of the step, since E-1 does not dictate the step. Additionally, I don't think step 5.5 would be utitilized. E-1 directs using the procedure to align makeup to the RWST. E-1 itself gives the E steps on aligning charging pump suction to the VCT.
80 H 3 N S Added title of OTN-BG-00002 to distractor D and eliminated Section name. Revised RWST level in stem to greater than 6%, changing the flowpath in the ECA - 1.1 procedure, step #9 contains the correct answer. Rev. 1 of question is now SAT.
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only Plausible to have distracters A and B with Mode 1 only? Would be better to have Mode 1 and 2 to increase discriminatory value. Additionally, the part 2 distractor for B & D are so much more detailed than A & C that one would likely choose those.
E Change part to distractors for B & D to simply say, 81 H 2 M S "To ensure protection against steam line break accidents."
Changed mode 1 only to modes 1and 2 only.
Removed detailed information regarding SLB indistractors B and D. Rev. 1 of question is now SAT.
Does not meet K/A as there is no interpretation of reference materials occurring. This is an extremely U difficult K/A to write an SRO question on without an 82 H 3 X N S "add-on". Consider replacing K/A.
Replaced with a new K/A and a new question. Rev.
1 question is now SAT.
83 H 3 B S 84 H 2 N S 85 F 3 N S Distractor C not plausible. While the distractor analysis talks about only being 3.2% tank level between high and low level alarm, the distractor is below the tech spec limit, which is 4.9% tank level difference. And is is not reasonable to believe that E
86 H 3 N the tank level would be the initial effect, which is S
what the question is asking.
Revised distractor C, and changed distractor A to restore within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> to eliminate concern over arguments that A could also be argued to be correct.
Rev. 1 of question is now SAT.
87 F 3 N S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only I don't see how distractor D is plausible. I know that the distractor is checking to see if reactor power is
>48%. But the stem states that the engineer wants to secure the pump "as soon as plant conditions E support." If a reactor trip was required, you wouldn't 88 H 3 N S be waiting for plant conditions to change.
Eliminated "Engineering is recommending that RCP "D" be secured as soon as plant conditions support securing a RCP" from the stem, and reordered the distractors. Rev. 1 of question is now SAT.
89 F 3 N S Part 2 distractors on A & C are not credible. The purpose of an RPS function is to mitigate an accident. While protecting the RCP motor is a good thing, it's not reasonable to assume one would believe RPS is for protecting equipment. Shouldn't the distinction be between Operable and Inoperable U
90 H 2 X N per TS? Is Non-functional really plausible for a TS S
diagnosis?
Changed part 1 distractors to operable and inoperable. Changed part 2 distractors A and C from "Provide RCP motor protection" to "Provide protection against exceeding 1% fuel cladding strain." Rev. 1 of question is now SAT.
91 F 2 N S 92 H 3 N S Reference Provided. K/A mismatch. K/A requires knowledge of the impact of a malfunction on the ITM system. Question as written requires knowledge of U CSFST and EAL. How would core damage effect 93 H 3 X N S the ITM system? What readings would a operator observe?
Replaced with a new K/A and a modified bank question. Rev. 1 of question now SAT.
Overlap with Op Test U
94 F 3 N Relaced with a new question. Rev. 1 of question is S
now SAT.
95 F 2 M S 96 F 3 N S
- 1. LOK 2. LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6 7 8 stem cred job #/ back- SRO-Q# (F/H) (1-5) cueing T/F partial minutia K/A B/M/N U/E/S Explanation focus dist link units ward only 97 F 3 N S 98 F 2 N S 99 F 3 N S 2014 Exam - Is it possible to change Director, Maintenance to Control Room Supervisor, without a potential for two correct answers? It seems unlikely to me that anyone would think someone outside of E
100 F 3 B operations would approve a plan that presents a risk S
of a safety system actuation.
Changed distractors from Director Maintenance to Troubleshooting team leader, and reordered distractors. Rev. 1 of question is now SAT.