ML022890116
| ML022890116 | |
| Person / Time | |
|---|---|
| Site: | Braidwood  |
| Issue date: | 07/25/2002 |
| From: | Caniano R, Grobe J Division of Reactor Safety III |
| To: | Skolds J Exelon Generation Co, Exelon Nuclear |
| References | |
| 50-456/02301, 50-457/02301, ES-401, ES-401-9 | |
| Download: ML022890116 (14) | |
Text
NRC COMMENTS TO THE INITIAL SUBMITTAL OF THE WRITTEN EXAM ON ES-401-9 AND THE OPERATING TEST FOR THE BRAIDWOOD INITIAL EXAMINATION - JULY 2002
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD 1 j j j -I (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A __Only__
6S H S New 8R HF S New NRC: Memory. LICENSEE: Agree.
91 H S New 12S H S New 176 H S New Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2. Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3. Check the appropriate box if a psychometric flaw is identified:
- The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
- The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
I
- 4. Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6. Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7. At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
I Page 1 of 10
1.12. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) StemCues T/F ICred. Partial Job- IMinutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only I 23R H S New NRC: Not appropriate RO tasks: procedure transition, assessment of conditions, and selection of appropriate procedures is SRO function.
LICENSEE: Question rewritten to RO level.
26R H E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT." NRC:
Question clarity better using the "NOT" format. Will accept original question.
27S F S New 33S H S New 36B H S New 44R H S New 46B H S New 52B H S New 55S F E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT" format.
57R H S New 58S H S New 60B H S New 63B H S New 73B H S New 78R H S New Page 2 of 10
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws
_1
- 5. Other 6. I 7.
Q#
83R LOK (F/H)
H LOD (1-5) Stem Cues T/F Focus 1 Cred. Partial Job- Minutia #/ Back-Dist. Link _ units ward Q= SRO U/E/S K/A Only I X U I
Explanation New NRC: KA is ability to perform procedural actions, but question stem asks for prediction of system response. LICENSEE: Rewrote question to meet KA.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- QA SRO U/E/S Explanation Focus ___ Dist. Link u___nitsward KAO~___ _______________________________
84B H S New 87S H E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT." NRC:
Question clarity better using the "NOT" format. Will accept original question.
90R H S New LICENSEE: Made correction, replaced "EP.0" with "ES-0.1" in b.
and d..
92S H S New 98B H S New 104S H S New 112R H S New 114R F S New 119S H S New IB F S New 2B H S New 3B H S Bank 4B F S Bank 5S H S Bank 7B F S Bank Page3of 10
Q# 1.
LOK 2.
- 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
(F/H) (1-5)Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 10B H U Bank NRC: Two correct answers as written. Insert "Only..." at beginning of distractors b. and c. to correct. LICENSEE: Agree.
11R H S Bank 13B F S New
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q#LOK LODI (F/H) (1-5) Stem CuesI T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation F 9 Focus Dist. Linkj units ward K/A Only 14B H S New 15S H X S New: NRC: Stem does not indicate that temperature exceeded design temperature. Alarms at 118 degrees F. Not SRO Only type question (RO knowledge) LICENSEE: Stem contains sufficient information to determine that design limit was exceeded. Not taking credit for an SRO Only question; however, it is only on the SRO exam.
16S H S New 18S F S New 19R FH S New NRC: ROs required level of knowledge for refuel equipment?
LICENSEE: Reviewed tasks and lesson plans. ROs are not trained and tested on this information, although some ROs run refueling equipment.
As a result, KA was rejected, new KA randomly selected (00WE13K102) and new question written.
20B F S Bank 21B H E Modified NRC: Rewrite stem in past tense to clarify past from present, and what is presently happening. LICENSEE: Rewrote question in past tense.
22S H S New 24B F S Bank LICENSEE: Removed "critical" from stem.
Page 4 of 10
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus JDist. LinkJ units ward K/A Only I 25S H S New 28B F S Modified Corrected mistake, replaced "ELEC-2" with "ELEC-1."
29B F S New 30B H X X U New NRC: TS basis not required RO knowledge; also, KA requires reasons for responses as applied to accidental gaseous radwaste release, but the question stem asking for which phase of LBLOCA provides basis for shifting AB ventilation. LICENSEE: Revised question to make RO level of knowledge and to meet KA.
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LO K (F/H) LO D (1-5) Stem jCues T/F Cred. Partial Job- - Minutia #/ Back- Q= SRO 7Q S R U/E/S
/ /E Explanation xl n t o Focus_ J Dst.J Link units ward K/AOnly 31B F E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT" format.
32B H S Modified 34B F S Bank 35B H S Bank 37B H S Modified 38B F S New 39B H S Bank LICENSEE: Replaced "early stages" with "first two minutes" in stem to clarify time.
40B F S New 41B H S New 42B F S New 43B H S Bank 45B H S Bank 47B F S New Page 5 of 10
Q#
S
- 1. 2.
LOK LOD
- 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
(F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus_ j Dist. Linki units ward K/A On y 48R F S Bank 49R H S Modified 50B H S Bank 51S H S New 53S H S New 54B F S Bank 56B F S Bank
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD (F/H) (1-5) Stem ICues T/F Cred. IPartial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focusj Dist. Link units ward K/A On _
59B F S Bank 61B H S Bank Corrected stem mistake, replaced "red" with "orange" path.
62R H S Modified 64R F E Bank NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) to eliminate "NOT" format.
says to avoid. LICENSEE: Rewrote question 65R F S New 66B H S Bank 67B F S New 68S H S Bank 69R H S Bank 70B F S New 71B H S Bank 72B H S Bank 74B F S New Page 6 of 10
Q# 1.
LOK 2.
LOD 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
(F/H) (1-5)FStem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation 1........
t Focus __j Dist. Link ___units Iward K/A Only ____________________________
75R F S New 76R H S Modified 77B H S Modified 79B F S Bank 80B H S New 81B F S New 82B H S New
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD A (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO U/E/S Explanation Focus Dist. Link units ward K/A Only 65B F S Modified NRC: ROs required level of knowledge for refuel equipment?
35S LICENSEE: Reviewed tasks and lesson plans. ROs are not trained and tested on this information, although some ROs run refueling equipment. Question was kept as an SRO Only question; however, the RO exam KA was rejected, new KA randomly selected (027000K201) and new question written (for the RO exam).
85R F S New 86B H S New 88B H S Bank 89S F S New 91R F S Bank 93R F S New 94R H S Bank 95R F S New LICENSEE: Add to stem, "If all of the following loads are started, which ONE will draw the largest run current?"
96B H S Bank Page7of 10
1 Q# LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws I
- 4. Job Content Flaws I 1 Stem ICues T/F Cred. Partial Job- Minutia #/ Back-
- 5. Other Q= SRO 6.
U/E/S j 7.
Explanation Focusj Dist. J Link units ward K/AI Only 97B H S Bank 99B F S Bank 10OR H S Bank 101B H S Bank 102B F S Bank 103B F S Modified 105B F S New Page 8 of 10
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD I I I 1 I (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ ,Back- Q= SRO U/E/S Explanation Focus I Dist. Link _ unitsI ward K/A On y 106B F E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT" format.
107R F E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT" format.
108S F S Modified 109B F S Bank 10B H S Modified 11S F S Modified 13B F E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT" format.
15S F S Bank 16S H S Modified 17S F S New 18B H S Modified 120S F S Bank 121B F S Modified 122B F S Bank.
123R H S Bank 124R H E Bank NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT" format.
125R H S Bank 126S H S Modified Page 9 of 10
- 1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7.
Q# LOK LOD I - I (F/H) (1-5) Stem Cues T/F Cred. Partial Job-IMinutia #/ Back- Q= SRO U/E/S Explanation Focus _ j _ Dist. Link units ward K/A OnlyI 127S F E New NRC: Negative stem, NUREG 1021, Appendix B (p. 10 of 26) says to avoid. LICENSEE: Rewrote question to eliminate "NOT" format.
128B F S New 129S F I I I S New NOTE:
- 1. First 30 questions used for sample review.
- 2. First 30 questions received independent verification of references.
- 3. 100% of questions received KA verification.
- 4. Total questions is 130 based on resolution to question 85.
- 5. SRO/RO: New 54/50; Bank 32/38; Modified 14/12; Fundamental 44/44; Higher 56/56 H F(1)
Not appropriate RO (1)
Negatively worded stem (Appendix D, p. 10 of 26) (10; 7%RO;6%SRO)
KA (2)
Two correct answers (1)
Page l0of 10
Braidwood Operating Exam Comments Administrative JPMs
- 1. Generic comment: arrange JPM so that back pages are in the order that examiner will give to applicant. From back to front: a) back page is Initial Conditions and Initiating Cue; b) next page is the first handout...etc.
A.1 .a (SRO) S-42, Review Calorimetric (2.1.7)
NRC Comments:
- 1) Step *3., should be "N-44" not "N-42."
- 2) Step *3., rather than hand the applicant a corrected D-2 sheet after he/she identifies the problem, have the applicant correct the problem on the original sheet (line through and initial, or go to the SM(?) to correct the problem. Otherwise, suppose the applicant incorrectly identifies another problem, the examiner won't be able to hand him/her a corrected copy.
Licensee: 1) and 2), agreed.
A. 1.b (SRO) S-02QPTR, Review QPTR Calculation (2.1.33)
NRC Comments: None.
A.2 (SRO) S-02BDPS, Review BDPS Out of Service (2.2.13)
NRC Comments:
- 2) If examiner does not hand the applicant the last sheet of the tag out, the applicant could be cued by the lack of"**** END OF ISOLATION POINTS ****" statement at the bottom of the sheet and consequently identify there was a missing sheet without really identifying the missing valves.
Licensee: 1) and 2), agreed.
A.3 (SRO) S-41, Review a Release Package (2.3.6)
NRC Comments: None A.4 (SRO) S-05, Classify and Screen Event for Reportability (2.4.30)
NRC Comments: None A.1.a (RO) N-02SHDLY, Perform Unit Common Shiftly Daily Rounds (2.1.18)
NRC Comments:
Licensee: 1) Agreed.
A.1.b (RO) N-18, Review QPTR Calculation (2.1.19)
NRC Comments: None.
Page 1 of 3
Braidwood Operating Exam Comments (continued)
A.2 (RO) N-02CS007. Perform 1CS007A Valve Stroke Surveillance (2.2.12)
NRC Comments: None.
A.3 (RO) N-32, Perform RM-1 1 Setpoint Change for Rad Release (2.3.11)
NRC Comments: None A.4 (RO) N-160, Activate Emergency Response Data System (ERDS) (2.4.29)
NRC Comments:
- 1) Step *3., add note for examiner, "Attachment 1 procedure starts here."
Licensee: Agreed.
System JPMs B.l.a N-03, ECCS / Increase SI Accumulator Pressure NRC Comments:
Licensee: Will replace.
B.1.b N-84. EDG / Sychronize SAT to a bus being fed by a EDG NRC Comments: None.
B.l.c N-27C, Perform Emergency Boration NRC Comments: None.
B. I.d N- 1 a, RCS / Excess Letdown Operations NRC Comments: None.
B. 1.e N-1 18, CCW / Respond to a RCP Thermal Barrier Leak NRC Comments: None.
B.1 .f N-1 19, PRT / Drain the Prezzurizer Relief Tank NRC Comments: None.
B.1 .g N-57a, SG / AFW Check Valve Leakage NRC Comments: None.
B.2.a N-138, ESW /Align Fire Protection Cooling to CV Pump after loss of SX NRC Comments: None.
B.2.b N-34. APE / Locally Align the Fire Hazzards Panel NRC Comments: None.
Page 2 of 3
Braidwood Operating Exam Comments (continued)
B.2.c N-91, ESF / Locally Reset Feedwater Isolation NRC Comments: None.
Scenarios (1,2,3,4)
Scenario 1 NRC Comments:
- 1) Errata, p. 4, middle of page, should be "Event 4"
- 2) Errata, p. 11, down 1/3rd of page, should be "...WITH..."
Licensee: Corrected.
Scenario 2 NRC Comments: None.
Scenario 3 NRC Comments:
- 1) Event 6, All MSIVs fail closed at power. Label as a Major Transient. Nothing operator can do about it.
Licensee: Agreed.
Scenario 4 NRC Comments: None.
Page 3 of 3