ML20365A089
| ML20365A089 | |
| Person / Time | |
|---|---|
| Site: | Cook  |
| Issue date: | 08/13/2018 |
| From: | Chuck Zoia Indiana Michigan Power Co |
| To: | NRC/RGN-III/DRS/OLB |
| Zoia C | |
| Shared Package | |
| ML17164A381 | List: |
| References | |
| Download: ML20365A089 (11) | |
Text
ES-401 DC COOK 2018 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 2007 NRC Exam Question 1 FH 2 B E NRC: This is an H question - the correct answer relies on multiple conditions.
Response: Change accepted 2 F 3 B S 2007 NRC Exam Question 3 H 2 B S 4 H 2 B S 2X2; 2012 NRC Exam Question 5 H 3 X B S 2X2 2004 NRC Exam Question NRC: This is an H question - the correct answer relies on multiple conditions. Also, 6 F 3 B E the referenced OHP note is prior to step 6, not step 4 - revise pedigree accordingly.
Response: Typo change accepted -F question agreed upon in further review Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 11
ES-401 DC COOK 2018 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: NPP-151 at 2310 psig and rising is a clear indication of failing high. The smaller deviation from 2235 psig (2225 psig) is not sufficient to cause concern and would not 7 H 2 X N E cause the deviation HIGH alarm. LOD=1?
Response: Upon further discussion, the question and stem were modified to eliminate concerns, and the LOD was agreed to be 2 upon further review.
NRC: Distractors A and B provide more detail than is required by the stem.
Response: Upon further discussion, the question was a K/A mismatch too. The K/A 8 H 2 X N U was changed and is documented on form 401-4, and the excess detail removed.
NRC: Changes accepted - Question now SAT NRC: Distractors B and D are not plausible. Maximizing AFW flow would not be required or even desirable in this situation.
9 H 3 X N U Response: The question was changed to incorporate better distractors for B & D.
NRC: Changes accepted - Question now SAT 10 F 2 N S 11 F 3 B S 12 H 3 M S 13 H 3 N S 14 H 3 B S 2007 NRC Exam Question 15 H 2 B S 2002 NRC Exam Question 16 H 2 B S 2008 NRC Exam Question 2X2; 2012 NRC Exam Question NRC: This is a very easy question. It is very basic knowledge to know that Phase A isolation occurs automatically upon SI actuation. The question would be more 17 H 1 B E discriminating with a less obvious first part of this 2X2 question.
Response: Upon further discussion, the issue with Phase A will be replaced with phase B and the answer changed.
NRC: 10 CFR 41.8 is a better fit than those referenced 18 F 2 B S Response: Those are K/A catalog values NRC: 10 CFR 41.5 is a better fit than those referenced 19 H 3 N S Response: Those are K/A catalog values Page 2 of 11
ES-401 DC COOK 2018 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 20 H 3 N S NRC: It is not clear why a large percentage of the fuel would be activated if half the core is reloaded. Explain and describe its impact. Also, distractor A should refer to 21 F 3 X N S core uncovery and distractor D is implausible.
Response: Upon further discussion, all issues were resolved.
22 F 3 B S 2012 NRC Exam Question NRC: This question needs to be clarified/simplified. The question would be better if it 23 H 2 X B E focused on any differences versus completing a long statement about them.
Response: Normal/Abnormal is removed and how is deluge performed 2012 NRC Exam Question NRC: Distractor C is not plausible and should be replaced. Also, plausibility is not supported for any distractors.
24 H 2 X B E Response: Upon further discussion, the distractor C was replaced to open the PORV to get pressure down for ECCS flow. D was also modified to preserve inventory.
25 F 2 B S NRC: The distractors improperly use the word farther - it should be further in this 26 H 3 B E context.
Response: Accepted NRC: The stem should better state the information being sought, rather than what is possible. Also, distractors D assume that the S/G is isolated - this is stated in distractors A and B.
27 F 3 X X B U Response: Agreed - Fixed by bringing S/G isolation up into the stem and clarifying that the cause of PTS is being asked for without any other information provided to enhance the incorrect distractors plausibility.
NRC: Changes accepted - Question now SAT 28 F 2 B S 29 H 2 M S 30 H 3 B S 2012 NRC Exam Question 2006 NRC Exam Question 31 H 3 B E NRC: Change Hx in stem to HX for consistency with distractors.
Response: Accepted Page 3 of 11
ES-401 DC COOK 2018 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: Plausibility is not supported for distractors C & D - why would energizing control power only be needed for valve operation in one direction?
32 F 2 X N U Response: Dropped the first statement, then created plausible distractors by essentially using a 2X2 methodology for the answer choices.
NRC: Changes accepted - Question now SAT 33 F 2 B S 2004 NRC Exam Question NRC: Distractors C and D are too high to be plausible - post-TMI training has made it very clear that PRT tailpipe temperatures would be lower than RCS temperature. If steam tables can be used when answering this question, I 34 H 2 X B E recommend the answer should be selected from four specific values in a tight range.
Response: Specific plausible temperatures are now used and 4 stem bullet were eliminated to clarify the information sought. The steam tables, in conjunction with system configuration knowledge, may be used to answer the question correctly.
2006 NRC Exam Question NRC: The loss of all CCW pumps is clearly the most limiting choice - LOD=1.
35 F 1 X B U Response: The question was replaced with an appropriate question from the 2006 NRC examination to resolve all concerns.
NRC: Changes accepted - Question now SAT 2012 NRC Exam Question NRC: Distractor C should include Manually trip the reactor, (move it into the stem) 36 H 3 B E since E-0 is entered. Also, tripping ALL RCPs would be more plausible than reducing power to 8%.
Response: The distractors now consist of various RCP tripping options only.
2X2 NRC: Pressure returning to a value is a cue that it returns to normal, and knowing that 2235 is NOP is LOD=1 (How long ago was 2085 normal? Its implausible now).
37 H 1 X X B U Response: 2085 was the NOP several cycles ago and so knowing the current value is too easy. Exactly how the question is being revised is TBD.
NRC: Changes accepted - Question now SAT 38 F 2 B S 2X3 39 H 3 B S 2008 NRC Exam Question 40 H 3 N S 41 F 3 B E 2006 NRC Exam Question Page 4 of 11
ES-401 DC COOK 2018 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: Spell out NESW in the question.
Response: Agreed REFERENCE PROVIDED N/A NRC: 1) This is an H question 2) Distractor A is not plausible and contradicts the information in the stem. 3) Why is there a pogo stick in the area? 4) Though the first part of the correct answer is a 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> TS (RO knowledge), the second part is 42 F 2 X M U SRO knowledge.
Response: Upon further review, it was determined to be an F question with NO Reference provided. In addition, distractor C will be replaced because it was not initially credible. The pogo stick is procedurally based.
NRC: Changes accepted - Question now SAT 2X3 NRC: Plausibility is not supported for distractors A & C - the actual time should be 43 H 1 B U well known (LOD=1), since it is considered a time-critical action.
Response: The 17 min distractors became 15 and plausibility is explained.
NRC: Changes accepted - Question now SAT 44 H 2 B S 2006 NRC Exam Question; 2X3 45 H 2 B S 2X2 46 H 3 B S 2X2 NRC: Plausibility is not supported for distractors B & D - closing the East pump 47 H 2 X B U discharge valve would directly increase its discharge pressure.
Response: Created a S/G flow-relationship question with pressure and flow.
NRC: Changes accepted - Question now SAT 48 H 3 B S 2010 NRC Exam Question 49 F 3 M S 50 H 3 N S 2X2 NRC: Plausibility is not supported for distractors A & B - having the synchroscope 51 H 1 X N U moving slowly in the fast direction is an often performed task when aligning diesels, etc. The other choices, with moving slowly in the slow direction, can be very easily rejected, thus eliminating 2 distractors Page 5 of 11
ES-401 DC COOK 2018 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Response: The concept behind how the synchroscope is moving is now described.
Thus, the cause of what is normally seen on the synchroscope must now be recognized, rather than a familiar phrase.
NRC: Changes accepted - Question now SAT 2006 NRC Exam Question NRC: Plausibility is not supported for distractor A - Are there other effluent rad 52 F 3 X B E monitors that only alarm on high radiation?
Response: Added explanations and made distractor A a plausible choice.
53 H 2 N S NRC: This question involves assessment of plant conditions and then selection of a procedure or section of a procedure to mitigate or recover, or with which to proceed -
SRO-only.
54 F 3 X N U Response Changed the question focus on RO level knowledge and eliminated procedure selections.
NRC: Changes accepted - Question now SAT 55 F 3 B S 2012 NRC Exam Question NRC: Step Counter information is irrelevant (and LOD=1) after a reactor trip - step counters / rod demand indications are not needed in these situations.
56 F 1 X N U Response: Remove Following a reactor trip from the stem.
NRC: Changes accepted - Question now SAT 57 H 2 B S 2008 NRC Exam Question NRC: All distractors should reference 1-OHP-4022-IFR-001 for consistency.
58 H 3 B E Response: Reference 1-OHP-4022-IFR-001 in the stem and removed other references, focusing on the actions required in the distractors.
NRC: Distractor D is implausible, given that 1-VCR-103 is between its MIN & MAX times.
59 H 3 X N E Response: D now relates to REF value and other distractors will be slightly modified for clarity.
60 H 3 B S 61 H 3 B S 2006 NRC Exam Question Page 6 of 11
ES-401 DC COOK 2018 7 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only NRC: Distractor B is implausible. Distractor A provides a reminder of how the 40%
loss of load is handled - the second sentence doesnt provide a reason for the trip 62 F 3 X N E but a reason the trip isnt needed.
Response: Change to focus on when the trip is active and why in a 2X2 format.
NRC: Distractors B & D dont fully answer the question - they need to specify what, if any, actions are required. If none, it should say so.
63 H 3 N E Response: Changed the question to a 2X2 format, limited the scope for clarity and the available distractor choices.
2007 NRC Exam Question NRC: Distractor D should provide a unit designation on the referenced valve for 64 F 3 B E consistency.
Response: Agreed.
65 F 3 M S 66 F 2 B S NRC: This is an H question - the correct answer relies on multiple conditions 67 FH 3 B E Response: Agreed Reference Provided 68 H 3 M E NRC: Distractors contain extra info Response: Remove data beyond minimum flowrate.
2X2 NRC: Answer explanations are confusing. A supports temperature rising and 69 H 3 B E lowering, and C talks about how LD inlet temperature changes while all the answers focus on exit temperatures. Clarify each choices explanation to the conditions given.
Response: Explanations enhanced.
NRC: Plausibility is not supported for any distractors, and A & B are not plausible, both citing choices 1 and 5 - Auct. Low Temp. & S/D margin, respectively.
70 F 2 X B U Response: Choice1 will now be Tcold and explanations will be added.
NRC: Changes accepted - Question now SAT NRC: Distractor A is too low, suggest 10000 mrem to fit in with D & B 71 F 2 X B E Response: A = 9800, B = 10000 2007 NRC Exam Question 72 F 3 B E NRC: Reconcile A & D explanations - air flows into/out of containment. Suggest Page 7 of 11
ES-401 DC COOK 2018 8 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only that distractor A cite lowering containment pressure versus minimizing radiation levels and C not refer to AB pressure but to maintain constant negative pressure.
Response: Explanations enhanced.
2010 NRC Exam Question 73 F 3 X B E NRC: The pedigree must clearly justify why this is considered an RO-level question.
Response: Agreed and explained in the correct answer pedigree.
74 F 3 B S 2004 NRC Exam Question NRC: Plausibility is not supported for tripping the reactor in distractors A & D, which is implausible. Also, a rapid power reduction would also be extreme and 75 F 3 X N U implausible.
Response: Tripping and rapid power reduction are no longer offered as choices.
NRC: Changes accepted - Question now SAT 2007 NRC Exam Question NRC: Distractor D not plausible as YELLOW functional recovery procedures are not 76 H 2 X B E required.
Response: DistractorD now references FR-P.1 and explains plausibility.
2x2; 2012 NRC Exam Question NRC: Distractor C includes the Unit number 2 in front of its procedure name while the 77 H 3 B E other answers do not include a Unit designation.
Response: Agreed & changed.
2010 NRC Exam Question NRC: NUREG 1021, Rev 11 page ES-401, 23 specifically states SRO knowledge should not be claimed for questions that can be answered solely using fundamental knowledge of plant parameters that require direct entry into Major EOPs (e.g. E0 and 78 H 2 X B E Red functional restoration). In this case the examinee had to assess that plant conditions placed them in E0 and FR-S.1 (red path).
Response: New K/A with new Generic K/A required because an SRO question was not possible. Replaced with a non-NRC bank question using new K/A.
79 H 3 M S 2004 NRC Exam Question 80 H 3 N S 81 H 2 N S 82 H 3 X B U NRC: This question doesnt meet SRO only justification, as it can be answered purely Page 8 of 11
ES-401 DC COOK 2018 9 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only N with systems level knowledge. In addition, the term operable applies to safety-related components controlled by the Technical Specifications. The Boric Acid Transfer pump is not covered by the Technical Specifications.
Response: Replaced K/A and wrote a NEW 2X2 question.
NRC: Changes accepted - Question now SAT 83 H 3 N S 84 H 3 M S 2007 NRC Exam Question 85 H 3 N S 2006 NRC Exam Question; Reference Provided NRC: This question doesnt meet SRO only justification, as it requires only above the line Technical Specification knowledge of TS 3.4.12 to arrive at the correct answer.
86 H 2 X B E See NUREG 1021, Rev. 11 page 20 of ES-401.
Response: Revised the question to include requiring SRO-only knowledge in the answer (LCO Actions).
87 H 3 B S 2014 NRC Exam Question 2007 NRC Exam Question Reference Provided NRC: This question doesnt meet SRO only justification, as it can be answered with systems level of knowledge. Specifically, that a general warning on both RPS trains 88 H 2 X B E results in a Reactor Trip. Which is an entry condition for E-0 which is also RO level of knowledge.
Response: The K/A was changed and the question was replaced with a non-NRC bank question which has references provided.
Reference Provided; 2007 NRC Exam Question NRC: This question is a direct look up (DLU) with TS 3.8.1 provided as a reference (LOD =1). The note Reference Provided should also be removed from the stem of the question. Examinees should not be given a cue to refer to any provided references in 89 H 1 B E order to determine what tools/information they need to answer the question.
Response: Provided references are NOT being referred to any longer. Upon further review, this question required putting together given information to arrive at the correct answer - NOT DLU.
Reference Provided; 2008 NRC Exam Question 90 H 2 B E NRC: Reference provided should be removed from the stem of the question.
Examinees should not be given a cue to refer to any provided references in order to determine what tools/information they need to answer the question. Page 9 of 11
ES-401 DC COOK 2018 10 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only Response: Provided references are NOT being referred to anywhere Reference Provided; 2010 NRC Exam Question NRC: Reference provided should be removed from the stem of the question.
Examinees should not be given a cue to refer to any provided references in order to determine what tools/information they need to answer the question. In addition, none of the distractors include the partially correct answer of 3.1.4.D which is the statement for multiple control rods outside of alignment. It should be relatively easy for 91 H 2 B E examinees to calculate that two rods are outside of alignment which would then make answer D the only possible solution, removing the knowledge point that the rods remain moveable and therefore are still OPERABLE. Perhaps a distractor that combines 3.1.4A and 3.1.4D would truly make this a difficulty 3 question.
Response: Provided references are NOT being referred to any longer. LOD = 2 is satisfactory, however, distractor A was enhanced for increased discriminatory value.
Reference Provided NRC: Reference provided should be removed from the stem of the question.
Examinees should not be given a cue to refer to any provided references in order to determine what tools/information they need to answer the question. Distractor A not 92 H 2 X N E credible as Table 3.3.3-1 breaks out CET by quadrant and in note defines channels with regards to CETs.
Response: Provided references are NOT being referred to any longer and A refers to quadrants.
2004 NRC Exam Question NRC: A recent change to the Technical Specification Bases indicates that at least one of the Gamma-Metrics neutron detectors can be configured for an audible count 93 H 3 B E rate. If an examinee contends the answer is C, how would this be addressed?
Response: Add a statement that No temp mod for Gamma Metrics has been made to stem 94 H 2 B S 2004 NRC Exam Question 95 F 2 B S NRC: One outcome of a 50.59 Screening is that a License Amendment is required (yes to change in Tech Specs). Be prepared to answer the contention that the answer 96 F 2 B E could be B or C.
Response: Deleted first paragraph and added procedure title - distractors were not changed, but the explanation is clear that B is incorrect.
97 H 3 B S 2008 NRC Exam Question Page 10 of 11
ES-401 DC COOK 2018 11 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 98 H 3 B S 2012 NRC Exam Question NRC: Distractor A is not plausible, regarding pressurizing RCS to fill ruptured S/G in order to cool it down. EOP bases is not SRO-only knowledge.
B 99 F 3 X X U Response: A modified bank question was selected to replace original question M utilizing the original K/A and meets SRO-only criteria.
NRC: Changes accepted - Question now SAT 100 F 3 B S 2008 NRC Exam Question B= 51 F= 31 E= 21 RO TOTALS: M= 5 H= 44 U= 13 Additional Notes: 13/75 = 17.3%
N= 19 B= 18 F= 4 E= 11 SRO TOTALS: M= 2 H= 21 U= 2 Additional Notes: 2/25 = 8%
N= 5 Page 11 of 11