ML20106E467
| ML20106E467 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 03/06/2020 |
| From: | Greg Werner Operations Branch IV |
| To: | Pacific Gas & Electric Co |
| References | |
| Download: ML20106E467 (11) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only REFERENCE PROVIDED. Better if part 1 for A and B show steam flow for 1-1 steam generator approximately the same as pre-trip. It doesn't E work well with part 2, particularly distractor A. Not 1 H 3 N S reasonable to think flow will go up in the affected steam generator and down in the others.
Changed steam generator levels for distractors A and B. Question now SAT.
With 0-5 in three of the distractors, it cues that 0-7 is not the correct answer. Change the distractors such that one compressor is not in three E
2 F 2 N distractors. (Could go 0-5 and 0-6, 0-5 and 0-7, S
06 only, 07 only.) Changed distractor A, then swapped distractors A and B, making A the correct answer. Question now SAT.
3 F 3 N S Question SAT, however, distractor A analysis should say, " trip the reactor if (not is) the ramp
" Additionally, might want to include in the 4 H 2 N S distractor analysis that manually initiating the ramp is an action for loss of heater drain pump.
Corrected distractor analysis.
Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3. Check the appropriate box if a psychometric flaw is identified:
- Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).
- T/F: The answer choices are a collection of unrelated true/false statements.
- Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.
- Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).
- 4. Check the appropriate box if a job content flaw is identified:
- Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
- Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
- #/Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
- Backward: The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)
- 6. Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.
- 7. Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).
Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only Question SAT, however, the question stem should 5 H 3 M S include a unit number. Added Unit 1 to the stem.
6 H 2 B S Question SAT, however, question information 7 F 2 M S header shows an importance rating of 3.0. It should be 3.2. Changed importance rating.
8 H 2 N S REFERENCE PROVIDED.
9 H 3 N S 10 F 3 N S Would like this question better if it were for unit 1.
E Spray pump 2-1 / EDG 2-1 and pump 2-2 / EDG 11 H 2 M S 2-2 seems too obvious. Changed to Unit 1.
Question now SAT.
12 H 3 N S Distractors C and D are not plausible. It is not possible to have item 3 be correct without item 2 also being correct. (i.e., subset issue makes two U
13 F 3 X N distractors implausible instead of two correct S
answers.) Changed items 2 and 3 in the stem from pressures to Phase A and B. Question now SAT.
14 F 2 N S Question overlaps an event in scenario 1.
U Resample to K/A 004, 6.26. Replaced question 15 F 3 X B S with a bank question to new K/A. Question now SAT.
Modify the stem such that there is no possibility that distractor B is correct. According to OP B-2:V, flow rate is set to between 3000 and 5000 E
16 H 3 B gpm. One could argue that if the flow rate was S
just above 3000 gpm, B could also be a correct answer. Added flow rate of 3000 gpm to the stem. Question now SAT.
Question SAT, however, reorder distractors D-C-17 H 2 N S B-A such that distractors generally read short to Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only long. This makes B the correct answer.
Additionally, the question stem should include a unit number. Reordered distractors.
Note 2 of Appendix B states not to isolate SFP HX or CFCUs UNLESS, so the note does allow them. As the such, the stem of the question should ask what heat loads SHOULD be Also, E reorder distractors D-C-B-A such that distractors 18 H 3 B S generally read short to long. This makes B the correct answer. Additionally, the question stem should include a unit number. Changed stem to ask what loads should, and reordered distractors.
Question now SAT.
Try to reword the stem some to prevent someone from claiming there is no correct answer. Since the tech spec LCO states only one is required, the E
19 F 2 N one could be RM-12, thus, technically, RM-11 is S
not required. Changed stem to ask which monitor can be used instead of is used. Question now SAT.
REFERENCE PROVIDED. Question SAT, however, change distractor C to say, "Instrument 20 H 2 B S air to containment valve may begin to close." This is what the note in the procedure says, and it is in line with distractor D. Changed distractor C.
21 F 2 B S 22 F 2 N S Question SAT, however, I do not think the question has been modified enough from the parent to call it a modified bank. The answer did 23 H 3 B S not change, nor did the containment parameters.
This question should be marked "Bank." Marked question as "Bank."
2018 NRC EXAM. Question SAT, however, question information header shows an importance 24 F 2 B S rating of 3.3. It should be 3.1. Changed importance rating to 3.1.
Question SAT, however, the question stem should 25 H 2 B S include a unit number. Added Unit 1 to the stem.
Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only Question SAT, however, reorder distractors C-D-A-B such that distractors generally read short to 26 H 2 N S long. This makes A the correct answer.
Reordered distractors.
Change part 1 distractors for A and B to 0.65 MW.
E Better distractor as STP-9A1 discusses operating 27 F 3 N S limits below 0.65 MW. Changed part 1 distractors to 0.5 MW and 0.65 MW. Question now SAT.
28 F 3 B S This question is a system overlap for system 029 with question 81. Write a new question to E resampled K/A 015 K4.03. Since this was the 29 F 2 B S NRC error at Outline creation, question counted as an EDIT. Wrote new question to 015 K4.03.
Question now SAT.
30 H 3 N S Question does not meet the system K/A of spent fuel pool cooling system. No knowledge of the system required to answer the question. (It is a U good question for fuel handling or radiation 31 H 3 X N S monitoring K/As.) There are no radiation monitors associated with spent fuel pool cooling. Replaced with bank question to K/A 033 A1.01. Question now SAT.
Could D be considered a correct answer? There is no indication in the stem regarding the status of the RCPs. If no RCPs are running, isn't D a E
32 F 3 B correct answer? The question stem should S
include a unit number. Added all RCPs are running and added Unit 1 to the stem. Question now SAT.
33 F 2 B S For item 1) in the stem, add ' are normally in service' to prevent the asking of a question during E
34 F 2 N the exam about whether a containment entry is in S
progress. Added are normally in service to the stem. Question now SAT.
Replace distractor C. If B is correct then C is also E
35 F 3 B correct since "both Hydrogen Recombiners" S
would be correct. Changed distractor C to one Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only train of containment spray only. Question now SAT.
Instead of asking when a main turbine trip causes a reactor trip, ask when the P-9 Power Range Permissive is enabled. I think the way it is E
36 H 3 N worded makes the question too easy, but the K/A S
is still met. Ensure you use the word MAXIMUM to eliminate subset issue. Changed to ask when P-9 is enabled. Question now SAT.
Question SAT, however, there is no 55.41 reference in the question pedigree. I think 37 F 2 B S 55.41.2 is the most appropriate. Added 55.41.2.
Question now SAT.
Distractors A and B are not plausible. If CETs only failing one way will have an effect on U
38 H 2 X B subcooling, high has to be the one. The orginal S
bank question is appropriate. Changed back to original bank question. Question now SAT.
39 F 2 B S 40 F 2 N S Distractor B obviously not correct since the stem E states pressure is still decreasing in S/Gs 1-2 and 41 H 3 M S 1-3. Changed pressures in S/Gs 1-2 and 1-3 to completely depressurized. Question now SAT.
42 H 2 N S 43 H 2 B S REFERENCE PROVIDED.
44 H 2 B S Distractor A not plausible, since tripping all RCPs would result in a reactor trip. Also, is there ever E an instance where you would trip RCPs prior to 45 H 2 N S tripping the reactor? Changed part 2 of distractors A and B to 5 and 15 minutes, respoectively. Question now SAT.
Question SAT, however, I do not think the 46 F 2 B S question has been modified enough from the parent to call it a modified bank. The answer did Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only not change, just one incorrect distractor. This question should be marked "Bank." Marked question as "Bank."
Question SAT, however, the question stem should 47 H 3 B S include a unit number. Added Unit 1 to the stem.
48 F 3 B S 49 H 2 B S 2018 NRC Exam.
Wouldn't CCW be a better distractor than E
50 H 2 B Charging? Changed part 1 of distractors C and D S
51 H 3 N S 52 H 2 N S Change distractor D to say Only S/G 1-1 and 1-4 MSIVs are closed to eliminate two correct E
53 H 3 N answers. (If all are closed, both 1-1 and 1-4 are S
closed.) Changed distractor D to state only 1-1 and 1-4 MSIVs are closed. Question now SAT.
Could you get a better alarm than 16-20? I think E
54 H 3 N 16-01 would be more realistic. Rewrote question S
as a 2x2. Question now SAT.
55 F 2 B S 56 F 2 B S 2018 NRC Exam.
Question SAT, however, swap distractors A and D such that distractors read short to long. Does not 57 H 2 B S affect correct answer. Swapped distractors A and D.
58 F 2 N S Use a different word than "unaffected" for distractors C and D since the startup would not be E unaffected but would stop with N36 simply 59 H 2 M S inoperable. Additionally, there is no 55.41 reference in the question pedigree. I think 55.41.2 is the most appropriate. Changed Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only "unaffected" to "terminated" and added 55.41.2 to the pedigree. Question now SAT.
This question is a system overlap for system 024 with question 93. Write a new question to E resampled K/A 061 AA2.01. Since this was the 60 F 2 B S NRC error at Outline creation, question counted as an EDIT. Rewrote question to K/A 061 AA2.01. Question now SAT.
Part 1 distractors for A nd B should be red instead E of magenta. More credible than a distractor that 61 H 3 M S does not exist. Changed "Magenta" distractors to "Red." Question now SAT.
Changed distractor B to state "Trip the reactor due to dropped control rods" since one dropped E rod would not result in tripping the reactor.
62 H 2 B S Changed distractor B to eliminate "rod" and swapped distractors A and B. Question now SAT.
I like the question, but not the group of incorrect distractors. It is pretty obvious that a rupture EOP would cover the actions of a leak AOP. I think the way to fix it is to have the question ask, "the procedure must be completed unless superseded E by EOP E-3, Steam Generator Tube Rupture:"
63 F 2 N S and have the four choices be ONLY; and the reactor trips; or safety injection actuates; and superseded by OP-AP-1. Changed to a three item selection question. Add the word "only" to the end of distractors C and D. Added only.
Question now SAT.
Question SAT, however, reorder distractors D-C-B-A such that distractors generally read short to 64 F 2 N S long. This makes C the correct answer.
Reordered distractors.
If the first two paragraphs of the stem were taken away, the question could still be answered, thus it is a system level question and not a Tier 1 U question. I think it could be fixed by eliminating 65 F 2 X N S the third paragraph, and asking in accordance with AP-14, what combination of fans will be running? Since step 2 of AP-14 is to put the system if Safeguards only with an S signal, A is Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only the only correct answer. Changed question per recommendation. Question now SAT.
66 H 3 B S 67 F 3 N S Question SAT, however, swap distractors B and C such that distractors read short to long. Also 68 H 2 B S Permissives are in numerical order for A-B-C.
Does not affect correct answer. Swapped distractors B and C.
Change part 1 of the stem to state, "a block aound the step number identifies a " and have the two choices be immediate action and continuous action. This better meets the K/A. Would prefer E
69 F 2 N part 2 be something different. Seems too obvious S
to me. A suggestion is "Verify has the same meaning as ..." and the two choices be Check and Ensure. Changed question per recommendation.
Question now SAT.
70 F 2 B S Question SAT, however, reorder distractors D-C-B-A such that distractors generally read short to 71 F 2 B S long. This makes D the correct answer.
Reordered distractors.
Question SAT, however, reorder distractors B-A-D-C such that distractors generally read short to 72 H 2 N S long. This makes B the correct answer.
Reordered distractors.
73 F 2 B S Question SAT, however, swap distractors A and C such that distractors read short to long. This 74 F 2 M S makes A the correct answer. Swapped distractors A and C.
75 F 2 B S Question SAT, however, change SFM in the stem 76 H 3 N S to Shift Foreman. Remove the word 'both' from current distractors B and D. Swap distractors C-Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only D-A-B such that distractors generally read short to long. This makes C the correct answer. Changed SFM to Shift Foreman, removed "both" from distractors B and D, and reordered distractors.
Question SAT, however, swap distractors C and D such that distractors read short to long. Does 77 H 3 B S not affect correct answer. Swapped distractors C and D.
78 H 3 B S 79 F 3 N S REFERENCE PROVIDED.
80 H 2 B S 81 F 2 B S 2018 NRC EXAM.
REFERENCE PROVIDED. Change distractors A and B to state to be in mode 3 in a MAXIMUM of hours to eliminate subset problems. Change distractor A to 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />. That is the time to be in mode 3 according to Action C2 and is more E
82 H 3 N plausible I think that 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />. Reorder distactors S
C-B-D-A such that distractors are short to long.
(Also reverse order of most limiting to least limiting.) This makes D the correct answer.
Changed question per recommendation.
Question now SAT.
83 H 3 B S 84 H 3 B S Counted as EDIT instead of UNSAT since this question has been used on a previous NRC exam. I have an issue with item 3, which makes distractors B and D improbable. While it will be a E
85 F 2 B consideration on how to to downpower per unit, it S
is obvious that that information would not come from the grid control center. Changed item 3 to adjustments of MVARS required. Question now SAT.
Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only REFERENCE PROVIDED. Question SAT, however, swap distractors C-D-A-B such that 86 H 4 N S distractors generally read short to long. This makes D the correct answer. Reordered distractors.
REFERENCE PROVIDED. With the EAL table provided, distractors A and C are easily eliminated giving the part 2 answer, and part 1 of the question is RO level of knowledge. This is because the heading above SA.6 specifically says 1 U if the "actions taken at the reactor control console 87 H X N 3 S ..." and the stem tells you that. Makes this question LOD 1. Additionally, this question does not meet the K/A of mitigation strategies in low power / shutdown conditions. Removed reference and changed part 2 distractor to ask if classification is required. Question now SAT.
REFERENCE PROVIDED. Bank question not used on previous NRC exam. If E-1.2 step 16 is given as a reference, this becomes LOD 1. Easy 1 U elimination skills eliminate A and D since Thot and 88 H M 3 S Pzr level are given. B is easily eliminated since it is given that one SI and one ECCS pumps are running. Rewrote question as a modified bank question from South Texas. Question now SAT.
89 F 3 N S REFERENCE PROVIDED. Question is is considered SAT, however, would like a 90 H 3 B S walkthrough on the curves which could change status. Changed CST levels to 65% and 50%.
91 F 3 N S 92 H 3 B S Question SAT, however, change distractor B to state "and direct the operator " instead of "and have another watchstander ..." to be consistent 93 H 3 B S with other distractors. Question pedigree missing the 55.43 reference. Should be 55.43.5.
Changed distractor B to "implement" and added 55.43.5 in the pedigree.
Rev. 11
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other
(F/H) (1-5) Cues T/F Partial Minutia Focus Dist Link Units ward K/A Only 94 F 3 N S 95 F 2 M S Stem should start with, "If the EP organization is not activated," or something similar. Consider E converting to a 2x2 question since it essentially is.
96 H 3 B S It might look cleaner. Changed to a 2x2 and eliminated reference to the Shift Manager.
Question now SAT.
97 F 2 N S 98 H 2 B S Balance the part 2 distractors such that two E choices are RP and two choices are SM.
99 F 2 M S Changed distractor A part 2 to Shift Manager.
Question now SAT.
I am concerned that B and C are easily eliminated because of the word AND (can't imagine anyone thinking two persons would be required to approve the dose). You can change one of those E to Site Emergency Coordinator. Is it possible to 100 F 2 B S find another plausible distractor that is one person? (Is RP Manager plausible?) Changed distractors B and C to RP Manager and Site Emergency Coordinator, and reordered distractors. Question now SAT.
Rev. 11