ML103430555
| ML103430555 | |
| Person / Time | |
|---|---|
| Site: | Duane Arnold  |
| Issue date: | 09/17/2010 |
| From: | Larson B T Operations Branch IV |
| To: | Duane Arnold |
| References | |
| 50-285/10-301 | |
| Download: ML103430555 (10) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 FC-2010-09 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:
$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
$ The answer choices are a collection of unrelated true/false statements.
$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.
$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:
$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).
$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer
=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).
Attachment 8 Page 1 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnlyGeneral Comments
- 1. Explanations, while optional (ES-401 D.2.g), are generally weak. Explanations should contain a brief explanation of why each of the distracters is plausible but incorrect. Consider reference materials for more complicated distracters to assist the reviewer in determining credibility.
- 2. Technical references should include the reference's revision number (if applicable) (ES-401 D.2.g).
- 3. RO Exam - more "Given the following -" type questions than normally seen. Some questions could be written simply as a statement/question without prov iding plant conditions.
- 4. Much higher number of questions taken from previous NRC exam than normally seen. (RO-20, SRO-3).
- 5. Considerable number of editorial/grammatical errors.
1 H 3 M S B/5 Open Reference - Steam Tables. Explanation for "B" doesn't address pressurizer level. Added explanation.
2 H 3 N S D/5 3 F 2 N E C/3 Bullets needed? No, removed bullets and revised stem.
ES-401 2 Form ES-401-9 FC-2010-09 Attachment 8 Page 2 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly 4 F 3 N E D/10 5 F 3 B E A/7 Does the position of the "failure mode override switch" have an effect on the failure mode of HCV-347/HCV-348 upon loss of power to PC-118? Also, provide more details on why distractors are wrong. What is P-118A? Added more to explanation. Failure mode override switch can be used to reopen valves, but is normal position when pressurizer manway is installed. Logic diagram to be submitted with FINAL Exam.
6 H 3 B E B/11 1997 NRC exam. Remove "VCT level is lowering" from stem since not required to answer the question based on explanation and ref material. OK 7 H 4 M E A/7 Should there be a procedure reference (ARP, AOP) since the answer is an action? Could "B" also be correct? No. You need raise pressurizer pressure. Revised explanation. Added proc ref to stem - ARP CB-1,2,3/A4 8 H 2 X B U A/7 Question is a basic system description type question and does not address an ATWS. Reworded Chief examiner agreed as High in earlier discussion.
9 F 3 B E C/11 NRC 1999 exam. Add "and why" to stem since answers are two-part. Also, explanation don't address second part of answers. Added "and why" to stem and explanation 10 H 3 N E A/5. Would be helpful if one-line diagrams of MFW and AFW were included as ref material. Remove "A break in the" from all answers. Ok 11 H 2 N U C/10 Overlap with Op Test - both test EOP/AOP Attachment 6
Replaced question.
12 H 4 N E C/5 Why is the atmospheric relief valve isol ated? Is this credible? Revise stem for clarity. Revise last bullet to "The Operators have not taken any actions" to preclude cueing. Yes, it has often been isolated for an entire operating cycle due to leakage. Stem revised. 13 H 3 N E A/7 14 H 3 M E D/10 No reference provided that says to use Batt Chgr 3 to power DC bus 1. Revise first two bullets for clarity. EOP-AOP Attachment 5 (added as ref). Bullets revised.
ES-401 3 Form ES-401-9 FC-2010-09 Attachment 8 Page 3 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly 15 H 3 N E B/4 Could answer "A" also cause low flow even though high raw water strainer differential pressure is present? Yes, but not the pressure indicating lights as stated in the stem. Is the "screen or grid A and B differential hi" necessary to answer the question?
No. Removed.
16 F 2 B S A/7 17 H 4 B E B/10 NRC 2002 exam Believe cognitive level should be high.
18 H 3 X M U B/7 Distractors A and D are not credible since break outside of containment as stated in stem (Room 81). Fixed per discussion at FCS 19 H 4 N E D/6 20 F 3 N E A/5 Need to find another reference for this knowledge.
Added reference to procedure.
21 H 2 X N U B/5 Find another reference and revise question. Not operationally valid to ask a question where a TS LCO was missed. Revised question and correct answer. Added reference to procedure.
22 F 3 N E D/11 23 F 3 N S B/11 CFR reference 11 vice 7. Is explanation for "A" correct? Explanation says min of range is 0.1, not 0. Will change CFR to 11. Changed explanation.
24 H 2 B E B/7 Change "A" to read UNABLE like "B" and "C". All caps for UNABLE and ABLE. Reference says AOP-06 but STM 02 was provided. OK will provide AOP-06 in addition Believe cognitive level is high 25 H 3 X N U C/5 Question doesn't match KA.
Does address operational implications and remedial actions Will add CPHS, SGIS to address indicating signals 26 H 4 B E A/5 2004 NRC exam. What is ref for 30F delta T on natural circ? EOP floating step says less than 50°F, plant experience shows about 30°F 27 F 3 N E D/10 ES-401 4 Form ES-401-9 FC-2010-09 Attachment 8 Page 4 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly 28 H 3 B E C/3 2004 NRC exam. No explanations provided. STM as ref?
Added explanation.
29 H 4 N E A/7 Answer B may also restore cooling water to RCP's since time delay is 30 sec. Should there be a procedure ref since the answer is an action? Choice "B" states that the Operator should deliberately wait 30 seconds, before opening valves. There is no reason to wait. To enhance changed to wait 30 seconds to satisfy the interlock to B and D. Procedure reference: AOP-23, section 4 page 28 open, att C page 66.
30 F 3 N U C/5 Overlap with Op Test Question was replaced.
31 F 3 N E B/7 32 F 3 N E C/5 33 H 4 B E D/10 2005 NRC exam Add reference to stem. OK 34 F 3 B S A/10 2005 NRC exam 35 H 3 N E D/10 For A and C, clarify "inadvertent boron dilution." What happens when resin gets hot? Does it release boron or capture more boron? Add "and reason why" to the stem since two-part answer. Will add explanation and reason why Believe cognitive level is high.
36 H 3 B E C/7 2001 NRC exam. Add trend to stem for LI-106 (lowering). Add ref to procedure. OK 37 F 3 N E B/6 38 H 3 X M E B/6 What is significance of trip unit numbers? Should they have noun names included for comprehension? Answer C just says "wide range input to the RPS fails high." Clarify wide range (power?) and modify explanation to address distractor. Added trip unit noun names and power 39 H 3 B S B/7 2004 NRC exam.
40 F 2 B E D/7 2002 NRC exam.
41 H 4 N E B/7 Explanation doesn't match answer. Revised explanation.
42 H 3 M E B/7 Modify Distractor "D" to include SI-3A is running since correct answer is only one ES-401 5 Form ES-401-9 FC-2010-09 Attachment 8 Page 5 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnlywith SI-3A running. OK 43 H 3 N E C/10 Distractor "B" says Condenser Bypass Valve - should be Turbine Bypass Valve? Yes 44 F 3 N E C/7 Distractor D has the operator blocking SGIS with a switch that does not exist according to the explanation. Modify D to reflect actual plant controls. OK Will change to reset CIAS lockout relays, D explanation changed.
45 F 2 N E D/5 Change cognitive level to fundamental OK 46 H 4 M E A/7 47 H 3 B E D/7 From 2001 NRC exam 48 H 2 B S D/7 From 1995 NRC exam 49 F 3 X N U D/8 2 part K/A not addressed. Also, question does not address stopping diesel under load on isolated bus. K/A systematically changed 064000 A2.03 50 H 3 X B U B/11 "A" is also correct since the continuous release limits have been reached and exceeded. Explanation doesn't match checked answer (B). Resampled K/A and replaced question 51 F 2 M E D/7 Delete voltages on answers since bus designators dictate the voltage level. Table of info missing (KA, Ref, Source, etc). Is this a Modified question? OK on voltages. Modified, stem changed to make another choice correct.
52 H 3 X B U D/10 Distractors A and B do not seem credible. Rewrite or replace KA. Resampled K/A and replaced question 53 F 3 N E B/7 Procedure does not specify how to isolat e IA to containment so "D" is a possible answer. Delete reference to procedure from stem. Verify "D" could not be correct.
Changed explanation, added "normally" to stem.
54 H 3 B E D/7 1995 NRC exam. No Explanations, no FCS Objective.
Will Add 55 F 2 B S C/7 2005 NRC exam.
ES-401 6 Form ES-401-9 FC-2010-09 Attachment 8 Page 6 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly 56 H 3 B E D/6 1995 NRC exam. Is "C" credible? Does any signal insert CEAs to PDIL? There is a rod block circuit that will stop CEA movement at PDIL.
57 F 2 B E D/7 1995 NRC exam. No Explanation, no FCS Objective, Ref is STM 36, not LP 0718.14. Will add explanation and objective 58 H 3 N E B/7 Distractor "D" needs modifying to make credible. The controller in automatic and operating correctly should control level at some setpoint. Will add 60% to 30 % to stem and provide pzr level curve. Changed distractor "D." 59 H 2 N E C/5 Replace distractor "B." Ref not adequate to justify correct answer. Ok will replace distractor and explain why reference does justify the correct answer. 60 F 4 B E A/10 This question borders on minutia since actions are in Attachment 17 of an OI procedure. Attachments at FCS are like procedures at some other plants. OI-VA-1 contains all procedures for containment ventilation including purge, hydrogen purge, detector well cooling, CEDM cooling. Question addresses major operation of Containment Purge system, not minutia.
61 F 3 N S B/7 62 H 3 B S B/10 2001 NRC exam 63 H 3 B S B/6 2001 NRC exam 64 F 3 N E C/10 Not higher cognitive. Can all bullets be removed without impact to question? OK, debulleted.
65 F 2 N E B/10 Reference doesn't address second part of answer.
Added ref 66 F 3 N S A/10 67 F 2 X N U A/10 Although not in SO-G-7, Procedure Use and Adherence, distractors "B" and "D" would likely result in verifying revision. Modify stem to include "per procedure." OK changed choice "C" too 68 H 3 B E D/10 1997 NRC exam 69 H 4 N E B/10 ES-401 7 Form ES-401-9 FC-2010-09 Attachment 8 Page 7 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly 70 F 3 B E A/10 2002 NRC exam. The procedure does not mention annotating green dots with TM number; only affixing a green dot. Is "B" credible?
"B" is credible, "Post-It" notes are used in the Control Room to indicate things such as "dummy signal" installed on pressurizer level channel.. Will change to Red because red Post-it notes are used for annunciators. Added ref for green dots.
71 F 2 N E C/12 Clarify stem to include "per SO-G-101, Radiation Worker Practices." Otherwise, "A" could be correct. OK 72 F 2 N E B/11 73 H 3 N E C/10 Revise stem for clarification.
OK 74 F 2 N E B/10 75 F 2 N E A/10 Reference defines Command and Control but doesn't identify who is ED prior to activation of TSC and EOF.
see Page 5 EPIP-OSC-2 Attachment 6.8 76 H 4 N E C/5 Pedigree missing from question to determine bank or new Changed choices wording to better match statement in EOP-01.
77 H 3 N E B/5 Open Reference. Distractor "D" says go to attachment Q, however there is no "Q" on the handout. Also - why are distractor flow rates plausible? Changed choices to 575 gpm and 55 gpm, Changed Attachment "Q" to Attachment "G" in choice "D." Provided basis for 55 gpm. Changed temperature in stem to 110°F. Removed source of flow with simply "flow" in all the choices. Expanded explanation. 78 H 3 N E D/5 Capitalization change and additional reference material.
79 H 3 N E D/5 Change "B" to "- and borate from the BASTs" to make more plausible. Changed SIRWT to BASTs in choice "B," Minor Capitalization changes.
80 H 3 N E D/5 Added MAIN to procedure title in choice "B," Minor capitalization changes.
81 H 3 N E B/5 82 H 3 N E C/5 Where in EOP-3 is the required action stated (include as ref matl)? What is ERFCS stand for? Question pedigree missing. Changed reference to EOP/AOP Attachment 9, added EOP-03 page to included ES-401 8 Form ES-401-9 FC-2010-09 Attachment 8 Page 8 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnlyreferences. Emergency Response Facility Computer System 83 H 4 N E C/5 AOP-22, Step 11.c states to commence a plant shutdown to Mode 4, not 3 - correct "C" to reflect procedure. For distractor "B" - why is 3% per hour chosen? Changed Mode 3 to Mode 4 in choice "C," Changed letdown flow to 32 gpm, Capitalization changes. Included reference for 3% per hour (lower leakrate requirement).
84 H 3 B E D/5 2004 NRC exam 85 H 4 N E D/5 Stem doesn't include LPSI and Containment Spray pumps in PTL as required by the procedure. Changed bullet in stem to "All LPSI and Containment Spray Pumps are in Pull-To-Lock. Added "normally" to explanation.
86 H 3 X N U A/5 Since "to recover from the inadvertent PPLS actuation" is in all answers, the first of the KA is not satisfied. It might as well be in the stem of the question since it provides no discriminatory value. The question in only testing procedure hierarchy. Significantly changed wording, Question now addresses inadvertent CPHS 87 H 3 N E A/5 How is the first part of the KA satisfied? Implied, action must be taken to avoid core damage. Candidate must determine core will be damaged instead of containment.
Are all bullets necessary?
No Question pedigree missing Editorial changes, expanded explanation.
88 H 3 N E B/5 89 H 3 N E A/5 Delete bullets 5 and 6. We don't tell applicants what alarms they don't get (list would be really long). Eliminate "alarm" in stem so question is "What action should be taken in response to these conditions?" Deleted Bullets associated with annunciators not in alarm from stem, Editorial changes.
90 H 3 X B U B/2 Since the first part of all answers are different, adding a time reference is not required, not discriminatory.
What ref says B/PT-102 provides an input to both 86A/B? What ref says Trip Units need to be bypassed? Question reworded to a 2x2 question. expanded explanation, provided additional ES-401 9 Form ES-401-9 FC-2010-09 Attachment 8 Page 9 of 10 OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6.
B/M/N 7. U/E/S 8. Explanation Stem Focus Cues T/F Cred.
Dist. Partial Job-LinkMinutia#/
units Back-wardQ= K/A SROOnly references.
91 F 2 N E A/5 Most of the stem is irrelevant to answer question. Also, change 10CFR reference to 5 vice 4 Revised stem. Changed cognitive level to low, minor punctuation change. 92 F 3 N E C/7 93 H 3 N E A/5 AOP-10 step 5f says to implement AOP-01 Low River Water Level if river level is less than 983 feet. The stem of question gives intake cell level and not actual river level. Clarify stem of question. Added "to maintain Condenser Vacuum" to choice "A." Changed to River Level 94 F 4 X N U A/2 Question is not a Tier 3 - Generic Replaced question with a more generic one 95 H 3 X B U C/6 2002 NRC exam. Question is not a Tier 3 - Generic Replaced question with a more generic one 96 F 3 N S B/3 Change cognitive level to fundamental OK 97 F 2 X N U B/4 Distractors C and D could also be correct since they are greater than 5. Modify stem to reflect minimum dose requiring approval.
OK 98 H 3 N S D/4 99 F 2 B E D/5 2004 NRC exam 100 F 3 N E C/5 Question appears to be SUNSI due to third bullet - apply applicable markings and ensure this question is submitted IAW Corporate Notification Letter request (- any personal, proprietary, sensitive unclassified, or safeguards information in your response be contained in a separate enclosure and appropriately marked.) Ref material not adequate to confirm correct answer.
OK, SUNSI listed on cover letter Added reference material, punctuation changes.
ES-401 10 Form ES-401-9 FC-2010-09 Attachment 8 Page 10 of 10 OBDI 202 - IOLE Process RO TOTALS:
B= 27 F=33 E =53 Additional Notes:
M= 8 H=42 U =10 N= 40 S = 12 SRO TOTALS:
B= 4 F=7 E=18 Additional Notes:
M= 0 H=18 U=5 N= 21 S=2
GENERAL COMMENT
S:
- 1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N