ML103430555
| ML103430555 | |
| Person / Time | |
|---|---|
| Site: | Duane Arnold  |
| Issue date: | 09/17/2010 |
| From: | Brian Larson Operations Branch IV |
| To: | Duane Arnold |
| References | |
| 50-285/10-301 | |
| Download: ML103430555 (10) | |
Text
ES-401 FC-2010-09 Written Examination Review Worksheet Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only General Comments:
- 1. Explanations, while optional (ES-401 D.2.g), are generally weak. Explanations should contain a brief explanation of why each of the distracters is plausible but incorrect. Consider reference materials for more complicated distracters to assist the reviewer in determining credibility.
- 2. Technical references should include the references revision number (if applicable) (ES-401 D.2.g).
- 3. RO Exam - more Given the following type questions than normally seen. Some questions could be written simply as a statement/question without providing plant conditions.
- 4. Much higher number of questions taken from previous NRC exam than normally seen. (RO-20, SRO-3).
- 5. Considerable number of editorial/grammatical errors.
B/5 Open Reference - Steam Tables. Explanation for B doesnt address pressurizer 1 H 3 M S level.
Added explanation.
2 H 3 N S D/5 3 F 2 N E C/3 Bullets needed? No, removed bullets and revised stem.
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
$ The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.
- 7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 2 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 4 F 3 N E D/10 A/7 Does the position of the failure mode override switch have an effect on the failure mode of HCV-347/HCV-348 upon loss of power to PC-118? Also, provide more details on why distractors are wrong. What is P-118A?
5 F 3 B E Added more to explanation. Failure mode override switch can be used to reopen valves, but is normal position when pressurizer manway is installed. Logic diagram to be submitted with FINAL Exam.
B/11 1997 NRC exam. Remove VCT level is lowering from stem since not required 6 H 3 B E to answer the question based on explanation and ref material.
OK A/7 Should there be a procedure reference (ARP, AOP) since the answer is an action? Could B also be correct?
7 H 4 M E No. You need raise pressurizer pressure. Revised explanation. Added proc ref to stem - ARP CB-1,2,3/A4 A/7 Question is a basic system description type question and does not address an 8 H 2 X B U ATWS. Reworded Chief examiner agreed as High in earlier discussion.
C/11 NRC 1999 exam. Add and why to stem since answers are two-part. Also, 9 F 3 B E explanation dont address second part of answers.
Added "and why" to stem and explanation A/5. Would be helpful if one-line diagrams of MFW and AFW were included as ref 10 H 3 N E material. Remove A break in the from all answers.
Ok C/10 Overlap with Op Test - both test EOP/AOP Attachment 6 11 H 2 N U Replaced question.
C/5 Why is the atmospheric relief valve isolated? Is this credible? Revise stem for clarity. Revise last bullet to The Operators have not taken any actions to preclude 12 H 4 N E cueing.
Yes, it has often been isolated for an entire operating cycle due to leakage. Stem revised.
13 H 3 N E A/7 D/10 No reference provided that says to use Batt Chgr 3 to power DC bus 1. Revise 14 H 3 M E first two bullets for clarity.
EOP-AOP Attachment 5 (added as ref). Bullets revised. Page 2 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 3 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only B/4 Could answer A also cause low flow even though high raw water strainer differential pressure is present? Yes, but not the pressure indicating lights as stated 15 H 3 N E in the stem.
Is the screen or grid A and B differential hi necessary to answer the question?
No. Removed.
16 F 2 B S A/7 B/10 NRC 2002 exam 17 H 4 B E Believe cognitive level should be high.
B/7 Distractors A and D are not credible since break outside of containment as 18 H 3 X M U stated in stem (Room 81).
Fixed per discussion at FCS 19 H 4 N E D/6 A/5 Need to find another reference for this knowledge.
20 F 3 N E Added reference to procedure.
B/5 Find another reference and revise question. Not operationally valid to ask a 21 H 2 X N U question where a TS LCO was missed.
Revised question and correct answer. Added reference to procedure.
22 F 3 N E D/11 B/11 CFR reference 11 vice 7. Is explanation for A correct? Explanation says min of 23 F 3 N S range is 0.1, not 0. Will change CFR to 11. Changed explanation.
B/7 Change A to read UNABLE like B and C. All caps for UNABLE and ABLE.
Reference says AOP-06 but STM 02 was provided.
24 H 2 B E OK will provide AOP-06 in addition Believe cognitive level is high C/5 Question doesnt match KA.
25 H 3 X N U Does address operational implications and remedial actions Will add CPHS, SGIS to address indicating signals A/5 2004 NRC exam. What is ref for 30F delta T on natural circ?
26 H 4 B E EOP floating step says less than 50°F, plant experience shows about 30°F 27 F 3 N E D/10 Page 3 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 4 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only C/3 2004 NRC exam. No explanations provided. STM as ref?
28 H 3 B E Added explanation.
A/7 Answer B may also restore cooling water to RCPs since time delay is 30 sec.
Should there be a procedure ref since the answer is an action?
Choice B states that the Operator should deliberately wait 30 seconds, before 29 H 4 N E opening valves. There is no reason to wait. To enhance changed to wait 30 seconds to satisfy the interlock to B and D.
Procedure reference: AOP-23, section 4 page 28 open, att C page 66.
C/5 Overlap with Op Test 30 F 3 N U Question was replaced.
31 F 3 N E B/7 32 F 3 N E C/5 33 H 4 B E D/10 2005 NRC exam Add reference to stem. OK 34 F 3 B S A/10 2005 NRC exam D/10 For A and C, clarify inadvertent boron dilution. What happens when resin gets hot? Does it release boron or capture more boron? Add and reason why to the 35 H 3 N E stem since two-part answer.
Will add explanation and reason why Believe cognitive level is high.
C/7 2001 NRC exam. Add trend to stem for LI-106 (lowering). Add ref to procedure.
36 H 3 B E OK 37 F 3 N E B/6 B/6 What is significance of trip unit numbers? Should they have noun names included for comprehension? Answer C just says wide range input to the RPS fails 38 H 3 X M E high. Clarify wide range (power?) and modify explanation to address distractor.
Added trip unit noun names and power 39 H 3 B S B/7 2004 NRC exam.
40 F 2 B E D/7 2002 NRC exam.
41 H 4 N E B/7 Explanation doesnt match answer. Revised explanation.
42 H 3 M E B/7 Modify Distractor D to include SI-3A is running since correct answer is only one Page 4 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 5 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only with SI-3A running.
OK C/10 Distractor B says Condenser Bypass Valve - should be Turbine Bypass 43 H 3 N E Valve?
Yes C/7 Distractor D has the operator blocking SGIS with a switch that does not exist 44 F 3 N E according to the explanation. Modify D to reflect actual plant controls.
OK Will change to reset CIAS lockout relays, D explanation changed.
D/5 Change cognitive level to fundamental 45 F 2 N E OK 46 H 4 M E A/7 47 H 3 B E D/7 From 2001 NRC exam 48 H 2 B S D/7 From 1995 NRC exam D/8 2 part K/A not addressed. Also, question does not address stopping diesel 49 F 3 X N U under load on isolated bus.
K/A systematically changed 064000 A2.03 B/11 A is also correct since the continuous release limits have been reached and 50 H 3 X B U exceeded. Explanation doesnt match checked answer (B).
Resampled K/A and replaced question D/7 Delete voltages on answers since bus designators dictate the voltage level.
51 F 2 M E Table of info missing (KA, Ref, Source, etc). Is this a Modified question?
OK on voltages. Modified, stem changed to make another choice correct.
D/10 Distractors A and B do not seem credible. Rewrite or replace KA.
52 H 3 X B U Resampled K/A and replaced question B/7 Procedure does not specify how to isolate IA to containment so D is a possible 53 F 3 N E answer. Delete reference to procedure from stem. Verify D could not be correct.
Changed explanation, added normally to stem.
D/7 1995 NRC exam. No Explanations, no FCS Objective.
54 H 3 B E Will Add 55 F 2 B S C/7 2005 NRC exam. Page 5 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 6 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only D/6 1995 NRC exam. Is C credible? Does any signal insert CEAs to PDIL?
56 H 3 B E There is a rod block circuit that will stop CEA movement at PDIL.
D/7 1995 NRC exam. No Explanation, no FCS Objective, Ref is STM 36, not LP 57 F 2 B E 0718.14.
Will add explanation and objective B/7 Distractor D needs modifying to make credible. The controller in automatic and 58 H 3 N E operating correctly should control level at some setpoint.
Will add 60% to 30 % to stem and provide pzr level curve. Changed distractor "D."
C/5 Replace distractor B. Ref not adequate to justify correct answer.
59 H 2 N E Ok will replace distractor and explain why reference does justify the correct answer.
A/10 This question borders on minutia since actions are in Attachment 17 of an OI procedure.
60 F 4 B E Attachments at FCS are like procedures at some other plants. OI-VA-1 contains all procedures for containment ventilation including purge, hydrogen purge, detector well cooling, CEDM cooling. Question addresses major operation of Containment Purge system, not minutia.
61 F 3 N S B/7 62 H 3 B S B/10 2001 NRC exam 63 H 3 B S B/6 2001 NRC exam C/10 Not higher cognitive. Can all bullets be removed without impact to question?
64 F 3 N E OK, debulleted.
B/10 Reference doesnt address second part of answer.
65 F 2 N E Added ref 66 F 3 N S A/10 A/10 Although not in SO-G-7, Procedure Use and Adherence, distractors B and 67 F 2 X N U D would likely result in verifying revision. Modify stem to include per procedure.
OK changed choice C too 68 H 3 B E D/10 1997 NRC exam 69 H 4 N E B/10 Page 6 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 7 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only A/10 2002 NRC exam. The procedure does not mention annotating green dots with TM number; only affixing a green dot. Is B credible?
70 F 3 B E B is credible, Post-It notes are used in the Control Room to indicate things such as dummy signal installed on pressurizer level channel.. Will change to Red because red Post-it notes are used for annunciators. Added ref for green dots.
C/12 Clarify stem to include per SO-G-101, Radiation Worker Practices.
71 F 2 N E Otherwise, A could be correct.
OK 72 F 2 N E B/11 73 H 3 N E C/10 Revise stem for clarification. OK 74 F 2 N E B/10 A/10 Reference defines Command and Control but doesnt identify who is ED prior to 75 F 2 N E activation of TSC and EOF.
see Page 5 EPIP-OSC-2 Attachment 6.8 C/5 Pedigree missing from question to determine bank or new 76 H 4 N E Changed choices wording to better match statement in EOP-01.
B/5 Open Reference. Distractor D says go to attachment Q, however there is no Q on the handout. Also - why are distractor flow rates plausible?
77 H 3 N E Changed choices to 575 gpm and 55 gpm, Changed Attachment "Q" to Attachment "G" in choice "D." Provided basis for 55 gpm. Changed temperature in stem to 110°F.
Removed source of flow with simply "flow" in all the choices. Expanded explanation.
D/5 78 H 3 N E Capitalization change and additional reference material.
D/5 Change B to and borate from the BASTs to make more plausible.
79 H 3 N E Changed SIRWT to BASTs in choice "B," Minor Capitalization changes.
D/5 80 H 3 N E Added MAIN to procedure title in choice "B," Minor capitalization changes.
81 H 3 N E B/5 C/5 Where in EOP-3 is the required action stated (include as ref matl)? What is 82 H 3 N E ERFCS stand for? Question pedigree missing.
Changed reference to EOP/AOP Attachment 9, added EOP-03 page to included Page 7 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 8 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only references. Emergency Response Facility Computer System C/5 AOP-22, Step 11.c states to commence a plant shutdown to Mode 4, not 3 -
correct C to reflect procedure. For distractor B - why is 3% per hour chosen?
83 H 4 N E Changed Mode 3 to Mode 4 in choice "C," Changed letdown flow to 32 gpm, Capitalization changes. Included reference for 3% per hour (lower leakrate requirement).
84 H 3 B E D/5 2004 NRC exam D/5 Stem doesnt include LPSI and Containment Spray pumps in PTL as required by the procedure.
85 H 4 N E Changed bullet in stem to "All LPSI and Containment Spray Pumps are in Pull-To-Lock. Added normally to explanation.
A/5 Since to recover from the inadvertent PPLS actuation is in all answers, the first of the KA is not satisfied. It might as well be in the stem of the question since it 86 H 3 X N U provides no discriminatory value. The question in only testing procedure hierarchy.
Significantly changed wording, Question now addresses inadvertent CPHS A/5 How is the first part of the KA satisfied? Implied, action must be taken to avoid core damage. Candidate must determine core will be damaged instead of containment.
87 H 3 N E Are all bullets necessary? No Question pedigree missing Editorial changes, expanded explanation.
88 H 3 N E B/5 A/5 Delete bullets 5 and 6. We dont tell applicants what alarms they dont get (list would be really long). Eliminate alarm in stem so question is What action should be taken in response to these conditions?
89 H 3 N E Deleted Bullets associated with annunciators not in alarm from stem, Editorial changes.
B/2 Since the first part of all answers are different, adding a time reference is not required, not discriminatory.
90 H 3 X B U What ref says B/PT-102 provides an input to both 86A/B? What ref says Trip Units need to be bypassed?
Question reworded to a 2x2 question. expanded explanation, provided additional Page 8 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 9 Form ES-401-9
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only references.
A/5 Most of the stem is irrelevant to answer question. Also, change 10CFR 91 F 2 N E reference to 5 vice 4 Revised stem. Changed cognitive level to low, minor punctuation change.
92 F 3 N E C/7 A/5 AOP-10 step 5f says to implement AOP-01 Low River Water Level if river level is less than 983 feet. The stem of question gives intake cell level and not actual river 93 H 3 N E level. Clarify stem of question.
Added "to maintain Condenser Vacuum" to choice "A." Changed to River Level A/2 Question is not a Tier 3 - Generic 94 F 4 X N U Replaced question with a more generic one C/6 2002 NRC exam. Question is not a Tier 3 - Generic 95 H 3 X B U Replaced question with a more generic one 96 F 3 N S B/3 Change cognitive level to fundamental OK B/4 Distractors C and D could also be correct since they are greater than 5. Modify 97 F 2 X N U stem to reflect minimum dose requiring approval. OK 98 H 3 N S D/4 99 F 2 B E D/5 2004 NRC exam C/5 Question appears to be SUNSI due to third bullet - apply applicable markings and ensure this question is submitted IAW Corporate Notification Letter request (
any personal, proprietary, sensitive unclassified, or safeguards information in your 100 F 3 N E response be contained in a separate enclosure and appropriately marked.) Ref material not adequate to confirm correct answer.
OK, SUNSI listed on cover letter Added reference material, punctuation changes. Page 9 of 10 OBDI 202 - IOLE Process
ES-401 FC-2010-09 10 Form ES-401-9 B= 27 F= 33 E = 53 RO TOTALS: M= 8 H= 42 U = 10 Additional Notes:
N= 40 S = 12 B= 4 F= 7 E= 18 SRO TOTALS: M= 0 H= 18 U= 5 Additional Notes:
N= 21 S= 2
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N
- 3. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 = 0 41.2 = 43.2 = 2 41.3 = 2 43.3 = 1 41.4 = 1 43.4 = 2 41.5 = 12 43.5 = 18 41.6 = 5 43.6 = 1 41.7 = 25 43.7 = 1 41.8 = 1 41.9 =
41.10 = 22 41.11 = 6 41.12 = 1 41.13 =
41.14 =