ML13178A362
| ML13178A362 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 03/14/2013 |
| From: | NRC/RGN-III |
| To: | FirstEnergy Nuclear Operating Co |
| Bielby, M. | |
| Shared Package | |
| ML11354A269 | List: |
| References | |
| Download: ML13178A362 (12) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 1 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 1 F 2 X B E NRC: D. not plausible. None of the procedure table modes or staffing types contain "3" ROs, only "1 or 2". Revise the number of ROs to make D. plausible.
Licensee: Changed D to 1 RO since this is requirement for Modes 4 and 5. 2 F 2 B S 2009 PER RO Exam 3 F 2 B S 4 H 3 B M S 2009 PER RO Exam; NRC: Question was modified.
Licensee: Question was modified.
5 F 3 N E NRC: Replace "should-" in last line of stem with something like "are procedurally required-"
Licensee: Changed "you should" to "Plant procedures require you to". 6 H 3 N S 7 F 2 N E NRC: The referenced procedure, NOP
-OP-4101, Step 4.7.7, states that the "RPM and Shift Supervisor's approval may only be granted-". Is the term "Shift Supervisor" equivalent to "Unit Supervisor" or "Shift Manager"? If "Unit Supervisor" is equivalent, there may be no correct answer.
Licensee: The terms Shift Supervisor and Shift Manager are equivalent. Shift Supervisor is the older title and not changed in all Fleet procedures yet. Neither title is equivalent to Unit Supervisor.
8 F 2 N S 9 F 2 M S 10 H F 2 N E 1 of 2 twice; NRC: LOK is F vice H. Memorization of two separate facts (declaration time and location).
Licensee: Changed LOK to Fundamental.
11 H 2 B S 2005 PER RO Exam 12 H 2 B S 2007 PER RO Exam
ES-401 Written Examination Review Worksheet Form ES-401-9 2 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 13 H 2 N S 14 H 2 X B E 1 of 2 twice; NRC: Distract ers A. and D. are not independent of one another, ie, if A. were correct, then D. would also be correct, and vice versa. Replace one of distracters A. or D. with plausible distract e r. Licensee: D is plausible per the following additional explanation: The Applicability for TSV & TCV trips is Note: this was different than the original justification.
15 H 2 B S 1 of 2 twice 16 H 2 X M B E 1 of 2 twice; NRC: 1) KA doesn't match. KA 295016, AK2.02 is "Knowledge of the interrelations between control room abandonment and Local Control Stations", not AA2.02, "Ability to determine and/or interpret Local Control Stations as they applied to control room abandonment".
- 2) Stem, last sentence, spelling of "-Emergency-"
- 3) B not M, same correct answer, no distracters significantly modified. Licensee: 1) Originall wrote question to correct KA number, but had incorrect statement identified. Corrected KA statement to match KA number. 2) Corrected spelling of EMERGENCY.
- 3) Changed from M to B. 17 F 2 B S 18 H 2 M S 1 of 2 twice; 2009 PER RO Exam; NRC: Need original question.
Licensee: Submitted original question.
19 H 3 B S 1 of 2 twice 20 F 2 N S 21 F 2 M B S 1 of 2 twice; NRC: B not M. Did not change stem condition and one distracter, or correct answer. Licensee: Changed from M to B.
22 F H 2 B E 2003 PER RO Exam; NRC: LOK is H vice F. Need to recall and analyze non
-linear curve containing multiple parameters (SP temperature and level, RPV pressure).
Licensee: Revised LOK to H.
ES-401 Written Examination Review Worksheet Form ES-401-9 3 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 23 H 2 N S 24 F 2 X M E 2009 PER RO Exam; NRC:
- 2) Need original question.
Licensee: 1) 295028 EA1.01 WAS ON es 401-4, Rejected KA list. Left wrong KA number on Exam. Corrected Exam to show 295028 EA1.02. 2) Submitted original question.
25 H F 2 B E 2007 PER RO Exam; NRC: LOK is F vice H. Recall of entry condition for ED.
Licensee: Changed LOK to F. 26 H 1 X N U S 1 of 2 twice; NRC: Low LOD makes two distracters implausible. Second part of question lowers overall difficulty. Systems with auto
-
initiation generally have a manual initiation that bypasses the auto
-initiation interlocks, so it becomes easy to eliminate two of the distracters (B. and C. implausible). Rewrite second second part of choices. Licensee: Choices B and D are plausible because the ADS and SPMU systems do not always initiate using the A&D p/bs. If RHR LOCA signal not present, back
-panel switch manipulations must b e made to allow the SPMU A&D p/bs to work. If LP pump not running ADS A&D p/bs won't work. Added to question justification.
27 H 3 M S 28 H 2 M B S 1 of 2 twice; 2009 PER RO Exam; NRC:
B not M. Did not change stem condition and one distracter.
Licensee: Changed from M to B.
29 F 2 B S 1 of 2 twice
ES-401 Written Examination Review Worksheet Form ES-401-9 4 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 30 F 2 X N U NRC: 1) C. and D. are not plausible (doesn't make sense), simple math indicates that raising voltage in a voltage to hertz ratio will increase the ratio (ie, make it worse) without even knowing anything about the expected system response or operator action (ie, common sense: why would system be designed that way or procedure written as such?).
- 2) D. conflicts with last stem sentence, because one says "-if this alarm does not clear within a minute" and other says "...to clear the alarm." Licensee: Added info to stem: gen. freq. at 58 hz, terminal voltage at 24000 volts, Div 1 DG running for monthly surveillance, and Buses EH12 & EH13 lined up to preferred source.
Then changed C to "breakers feeding buses EH12&13 will trip on underfreq", and changed D to "Div 1 DG output breaker will trip on underfreq".
31 H 3 B E 1 of 2 twice; NRC: 1) Stem, first sentence need to be rephrased.
- 2) Re-format stem to separate and reflect the different time elements in which the events happen, ie, currently all are grouped together in bullets under sentence that says "-plant is operating-following conditions:".
Licensee: 1) and 2) changed stem to reveal timeline of events.
32 H 3 N S 1 of 2 twice 33 H 2 B S 34 F 2 B E NRC: Distracters B.,C. and D. all give actions and parameter results; however, correct answer A. only gives action with no parameter result. Need to make answer and distracters the same format. Licensee: Changed A format to action and result to resemble B,C and D. 35 H 3 X X B U S 1 of 2 twice; 2007 PER SRO Exam; NRC:
- 1) Second part of question is SRO only (LCO).
- 2) Doesn't meet KA. Supposed to be for Inadvertent Containment Isolation.
Licensee: 1) Per Clarification Guidance for SRO
-Only Questions, LCO's, Applicability's and Actions loss of RPS bus causes an inadvertent containment isolation which causes a loss of SDC.
ES-401 Written Examination Review Worksheet Form ES-401-9 5 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 36 F 2 X N E U NRC: Make order of "-equipment qualification and personnel access" the same in Distracters C. and D.
Licensee: This was KA miss, could not rewrite acceptable question and choices, replaced with new question.
37 H 3 M S 38 H 3 X M U S 1 of 2 twice; 2003 PER RO Exam; NRC:
This is a K5 KA, applied to ability A2.12. A2 is Ability to (a) predict the impacts-.of Pump Runout on RHR/LPCI: INJECTION MODE; and b) based on those predictions, use procedures--
Licensee: Wrote question to correct KA number, but had incorrect KA statement on exam. Corrected KA statement to match KA number, changed "guidance" to "procedural guidance" and added additional info to justification.
39 F 2 M S 40 H 2 M U NRC: Either A or B may be correct. Applicant could argue that there was no indication of why LPCS inadvertently initiated. As a result, how could the applicant be sure LPCS wouldn't auto start after restoration of the power supply even if LPCS was procedurally disabled from auto start?
Licensee: Added to stem "due to failure of DW pressure trip units" and added "and were successful" to immediate action statement in the stem. NOTE: Post exam comments changed answer from A to B.
41 F 2 B S 42 F 2 B S 43 F 2 B S 2010 PER RO Exam 44 F H 3 N E 1 of 2 twice; NRC: LOK is H vice F. Requires comprehension and analysis to determine correct answer.
Licensee: Changed LOK to H.
45 H 3 B S 46 F 3 B S ES-401 Written Examination Review Worksheet Form ES-401-9 6 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 47 F 2 B S 1 of 2 twice 48 H 2 N S Ref. Provided:
LPRM to APRM Channel Assig nment Attachment 2 of SOI-C51 (APRM) 49 H 3 B S 50 H 2 M S 2010 PER RO Exam; NRC: Need original question.
Licensee: Submitted original question.
51 H 2 M E 1 of 2 twice; NRC: Make Part 2 format of distracters C and D same as A and answer B, ie, either the action only, or both the action and reason for the action.
Licensee: Change C and D to make Part 2 action only.
52 H F 2 B E 2001 PER RO Exam; NRC:
LOK is F vice H.
Memorization of separate valve trips (one doesn't depend on the other).
Either know, or don't know the trips, cannot use comprehension or analysis to determine answer.
Licensee: Made LOK F.
53 H 2 M S 2007 PER RO Exam
- Need original question.
Licensee: Submitted original question.
54 H 3 M B S 2010 PER RO Exam; NRC:
B not M. Did not significantly change one stem condition or distracter. Same correct answer.
Licensee: Change to B.
55 F 3 N S 1 of 2 twice 56 H F 2 N E 1 of 2 twice; NRC: LOK is F vice H.
Memorization of monitor trip and location (one doesn't depend on the other). Either know, or don't know, cannot use comprehension or analysis to determine answer. Licensee: Made LOK F.
57 F H 1 X N U NRC: 1) Low LOD. Applicant only has to know that Hanna not connected and can eliminate three distracters.
- 2) Hanna distracter is not believable (plausible) because it would never provide power in/out of the Perry Transmission Yard (all other lines are connected).
Licensee: Replaced question with LOK=H.
ES-401 Written Examination Review Worksheet Form ES-401-9 7 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 58 F 3 B S 59 F 2 N S 1 of 2 twice 60 F 2 M B E NRC: 1) In order to balance choices with correct answers, suggest:
a) "RHR C= immediately" in distracter A.; b) "ESW B=18 seconds" in distracter B.
- 2) B not M, no significant change to condition in stem.
Licensee: 1) Changed A to "RHR
-immediately" and changed B to "ESW-18 seconds". 2) Changed M to B.
61 F 2 N S 1 of 2 twice 62 H 3 N S 1 of 2 twice 63 F 2 B S 64 H 3 B S 65 H 3 M E NRC: Change the tense of the choices to reflect that you are asking the applicant to predict the outcome:
a) insert "will" after "-Pump" and replace "tripped" with "trip".
b) insert "will" after "-Pump"
. c) insert "will" after "-ISOL" and replace "isolated" with "isolate".
d) insert "will" after "-ISOL" and replace "isolated" with "isolate".
Licensee: Changed tense of choices to predict outcome.
ES-401 Written Examination Review Worksheet Form ES-401-9 8 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 66 H 2 X X N U Ref. Provided: ABB Steam Tables 1 of 2 twice
- NRC
- 1) Does not meet KA. How does determining bulk RPV temperature and comparing to BHD or Vessel Head Flange temperature apply to "operational implications"?
- 2) Explain why A.,B., and D. are plausible (believable).
Licensee: 1) Without RRPs running, monitoring of bulk coolant temperature by instrumentation is not accurate. The operational implications are determining bulk coolant temperature in order to avoid exceeding cooldown rate. With the scram now reset, cold CRD water is no longer entering the RPV. 2) Revised format into 1 of 2 twice by adding determination of cooldown rate exceeding 100 deg F, and replaced the implausible determination of bulk RPV water temperature with plausible number.
NOTE: Post exam comment changed answer from C to A.
67 H 1 2 M U S Ref. Provided:
(Modified) EOP-SPI Supplement Figure #7 HDOL
- 2009 PER RO Exam; NRC
- Low LOD. Applicant only has to plot the given Containment parameters for H 2 and pressure to determine that Containment HDOL has not been exceeded.
Licensee: Provided reference drawing to be used (deleted notes) and required plotting [H 2]. 68 F 1 X M U NRC: Low LOD and C. and D. not plausible: a) C. and D. are same; b) why would moving empty hoist be prevented? Also, if no automatic interlocks, why are B., C. and D.
distracters plausible?
Licensee: Replaced question.
69 H 2 N S 70 H 3 N S 71 F 2 M B E NRC: 1) Paralleling in this fashion always requires SS to rotate in clockwise direction which eliminates need to know other distracters
. Eliminate part (3) to increase LOD and plausibility. 2) B or N, but not M. Questions are not related.
Licensee: 1) Eliminated part 3. 2) Changed to B.
72 F 3 N S 1 of 2 twice 73 F 3 N S ES-401 Written Examination Review Worksheet Form ES-401-9 9 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 74 H 2 M E NRC: 1) Knowing damper status or fan status eliminates need to know status of the other.
Eliminate either the damper positions or the fans status to increase LOD
. 2) B not M, no significant condition change in stem.
Licensee: 1) Eliminated damper positions from answer/distracters. 2) By eliminating damper positions, made question M.
75 H 2 X M U 1 of 2 twice
- NRC
- A.(1) and B.(1) not plausible.
It doesn't make sense that operating pressure within design of the SRV spring set pressure would allow the pressure safety limit to be exceeded. In order to exceed the pressure safety limit, nothing would work.
Licensee: Revised question choices A and B.
LOK (F/H) 3 5/40=4 73 (H=38-45) B/M/N 3 3/1 7/2 5=4 42333 (BM, N@H=10 , N@H8) UNSAT for RO Only 7/75=9 1 of 2 twice RO=2 5 2009 2010 B 2 2 M 4 1 1 H 3 N S 1 of 2 twice 2 H 3 N S 3 H 3 N S Ref. Provided: TS 3.8.3 (Partial) 4 F 3 N S 5 H 3 N E 1 of 2 twice; NRC: 1) Errata: Last bullet of stem, "T" and "N" in VENT have a space.
- 2) Revise choices to a two column format, one column for procedure and other for direction.
Licensee: 1) Corrected "VEN TPIPE". 2) Revised choices and put into two column format.
6 H 3 B E NRC: Revise last sentence of stem to eliminate the term "should" (ie, replace "should" with "would").
Licensee: Replaced "should" with "would".
ES-401 Written Examination Review Worksheet Form ES-401-9 10 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 7 H 3 N S Ref. Provided:
EPI-A1 8 H 2 X M U Ref. Provided:
PDB-A06, Power Flow Map (modified)
- NRC
- Determining that loss of 1 RRP would cause flow to lower knowledge.
Licensee: Directing the procedure is SRO duty. This is beyond Immediate Action of procedure.
Note: Post exam comments deleted this question because no correct answer, and on RRP trip plant ends up to right of the Controlled Entry/Immediate Exit region, not as indicated in the question justification.
9 H 2 N S 10 H 2 X X X N U NRC: 1) Stem contains cues and not SRO only. Knowledge of procedure contents is RO knowledge. The term "Nuclear Closed Cooling" is used in both the stem and choices, and taking into account the event, that knowledge gives away choices 1 and 4.
The only decision is whether Choice 2 or 3 is correct, and since 3 is used in three distracters, it obviously makes them wrong, and an astute test taker would choose the correct answer B.
- 2) Does not meet KA. How does question test knowledge based on system pressure?
- 3) Replace Item 3, or rewrite, one of the distracters. As written, applicant only would have to know what procedure did not contain direction to address mitigation of the event (ie, 3), not select the procedures that mitigate the event.
Licensee: Replaced question.
11 H 2 N S Ref. Provided:
PDB-C0011, PSA Pre
-Solved Configurations for On
-Line Risk(partial)
- 1 of 2 twice
- NRC
- Determining risk is RO job?
Notification of duty team can be done by RO?
Licensee: Determining Risk is SRO job. NOP
-OP-1007 specifically states Shift Manager responsibilities.
12 H 2 N S Ref. Provided:
TS 3.6.2.1 (partial) 13 H 2 B S Ref. Provided:
PAP-1604, Reports Management
- EPI-A1 , Emergency Action Levels
ES-401 Written Examination Review Worksheet Form ES-401-9 11 Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia #/ units Back-ward Q= K/A SRO Only 14 H 3 N S 15 H 2 M S 1 of 2 twice 16 F 2 B S 17 F 2 M S 1 of 2 twice 18 H 2 M S 19 H 3 M S Ref. Provided:
TS 3.5.1 (partial)
- 1 of 2 twice 20 H 2 B S Ref. Provided
- 1 of 2 twice 21 H 3 M S Ref. Provided
TS 3.3.6.1 (partial)
- PDB-I-5 (partial)
- 1 of 2 twice
- 2010 PER SRO Exam 22 H 3 N E NRC
- 1) What is OAI
-1701? 2) No, references, SRO's expected to memorize LCOs?
Licensee: 1) OAI-1701 is procedure for tracking LCOs.
It discusses Support-Supported Systems and describes the LCO 3.0.6 exception to LCO 3.0.2. 2) Since the temperature limit for ECCW is not covered in TS LCO 3.7.10, it is important for the SRO to know other factors that might cause a system to become inoperable. 23 H 2 B S Ref. Provided:
TS 3.1.3 (partial)
- TS 3.1.4 24 H 3 N S 1 of 2 twice 25 H 2 M S LOK (F/H) 3/22=1288 (H=38-45) B/M/N 5/7/1 3=20/2852 (B@H, M4) UNSAT for SRO Only 2/25=8 1 of 2 twice SRO=9 UNSAT Overall 1 2/100=1 2 2009 2010 B 0 2 M 0 0
ES-401 Written Examination Review Worksheet Form ES-401-9 12 Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F) fundamental or (H) higher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distracters contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
The distracters are not credible; single implausible distracte rs should be repaired, more than one is unacceptable.
One or more distract ers is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content). The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory). The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons). The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO
-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B) bank, (M) modified, or (N) new. Check that (M) modified questions meet criteria of ES
-401 Section D.2.f.
- 7. Based on the reviewer's judgment, is the question as written (U) unsatisfactory (requiring repair or replacement), in need of (E) editorial enhancement, or (S) satisfactory?
- 8. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).