ML030970303
| ML030970303 | |
| Person / Time | |
|---|---|
| Site: | Fermi  |
| Issue date: | 03/10/2003 |
| From: | Lanksbury R NRC/RGN-III/DRS/OLB |
| To: | |
| References | |
| 50-341/03-301, ES-401-9 50-341/03-301 | |
| Download: ML030970303 (15) | |
Text
NRC COMMENTS TO THE INITIAL SUBMITTAL OF THE OPERATING TEST AND THE WRITTEN EXAMINATION INCLUDING NRC FORMS ES-401-9 AND ES-401 -10 FOR THE FERMI INITIAL EXAMINATION - MARCH 2003
is s ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
7.
Q#
LOK LOD (F/H)(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation Focus Dist.
Link
] units ward K/A Only 1
H 2
5 2
F 2
IS 3
F 2
S 4
H 2
S 5
H 2
S 6
H 2
U Does not qualify as a "Modified' question since distractor not changed l___
l__l(ref 401, D.2.f). Changed to 'Direct' from question bank.
7 F
2 S
8 H
2 S
9 H
2 X
U HPCI and RCIC may be inoperable simultaneously. Requires 3.0.3 S/D!
Changed distractors.
10 F
2 S
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.!
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy -difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
I
- More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if ajob content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that Is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-onlv (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
7.
Q#
LOK LOD TT-t (F/H)
(1 -5)
Stem Cues T/F Cred. lPartial Job-Minutial #/ lBack-Q= lSRO U/E/S Explanation Focus l___
l Dist.
Link units ward K/A l Only 11 H
3 S
12 H
2 S
13 H
3 X
U Required knowledge for RO applicants? Replaced question.
14 F
2 S
LOK - Relevance of first sentence to the condition? If not, it's a memory question. Comment not incorporated.
15 F
2 S
16 F
2 S
17 H
3 U
LOK - Higher due to knowledge of battery bus and affect to EDGs Comment incorporated.
18 H
2 S
19 H
2 X
E Delete 'containment parameters' in stem and pull 'drywell temperature' into stem - delete same from all distractors. Commentrincorporated.
20 F
2 S
Why is a Basis information from TS an RO question? Comment NOT incorporated.
21 F
2 S
22 F
2 S
23 F
2 S
24 F
2 S
Why is a Basis information from TS an RO question? Comment NOT incorporated.
25 F
2 X
U Why is a Basis information from TS an RO question? Question now references and AOP.
26 H
3 S
27 H
3 S
28 F
2 X
U Why is a Basis information from TS an RO question? Also, not a higher I
.level of cognizant question. Question replaced.
29 F
2 LOK - not higher since told high rad levels in SC, operator does not have to deduce conditions. Also, why must RO applicants know basis?
Comment NOT incorporated.
ES-401 3
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
7.
Q#
LOK LOD 1T (F/H)
(1-5)
Stem cues T/F Cred. Partial Job-Minutia
- /
Back-Q= lSRO UIEIS Explanation Focus l
Dist.
Link l lunits ward K/A lOnly 30 F
2 X
U LOK - not higher since told fuel failure occurred; can remove entire condition paragraph and still be able to answer question! Comment incorporated.
31 H
2 S
32 H
2 S
Can SBFW be operated from remote S/D panel? Yes 33 H
2 S
34 F
2 S
35 H
2 S
36 F
2 S
37 H
2 S
38 F
2 S
39 H
3 S
40 F
2 S
41 F
2 S
LOK - memory level question. Comment NOT incorporated.
42 H
2 S
43 F
2 S
44 F
2 S
45 F
2 U
LOK - memory level question. Comment inroroorated.
46 H
2 X
X U
Cue given in stem that SDV high level alarm in. Changed stem and distractors.
47 H
2 5
48 F
2 S
49 H
3 5
50 F 2
1 j
I
__j
_I U
LOK - Memory level question. Comment incorporated.
ES-401 4
Form ES-401-9 1
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
7.
Q#
LOK LOD T
(F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO UIEIS Explanation Focus Dist.
Link l units ward K/A Only 51 H
3 S
52 H
2 X
E Noun name for E41 -F011 ? Comment incorporated.
53 H
2 X
X U
LOK - Higher level of knowledge! Need to ask how this was taught in classroom.
Does not meet K/A. Multiple correct answers? Comment incorporated - changed stem.
54 H
2 X
E Remove from each distractor and add to stem, "ADS will begin depressurizing the plant." Comment Incorporated.
55 F
2 S
56 H
2 S
LOK - higher level of knowledge! Need to ask how this was taught in classroom. C not a creditable distractor. Comment NOT incorporated.
57 F
2 U
LOK - memory level question. Need to ask how this was taught in classroom. Comment incorporated.
58 F
2 S
59 H
3 S
60 F
2 S
LOK - memory level question. Comment NOT incorporated.
61 H
2 62 H
2 S
63 F
2 5
64 F
2 5
65 H
2
=
66 F
2 U
LOK - Memory level question. Comment incorporated.
67 H
2 S
68 F
2 S
Does not meet KIA, not a double valve isolation question. May have to make this a generic question. Comment NOT incorporated.
69 F
2 S
.70 F
2 5
=
-_S
ES-401 5
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
7.
Q# LOK LOD I
[
(FIH)
(1-5)
Stem JCues T/F Cred. Partial Job-lMinutia l #/
Back-Q=
SRO U/EIS Explanation Focus
[
Dist. l Link l lunits ward K/A Only 71 H
3 S
72 H
3 S
73 H
3 S
74 H
2 S
75 F
2 S
76 F
2 S
77 H
2 S
78 F
2 79 F
2 S
80 H
2 S
81 F
2 S
82 F
2 S
83 F
3 5
LOK - as constructed, not a higher level cognitive question, but a higher LOD question! Comment NOT incorporated, 84 F
2 S
85 F
2 86 F
2 87 F
2 88 F
2 S
89 F
2 90 F
2 S
ES-401 5
Form ES-401-9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
7 Q#
LOK LOD I -
(F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- / l Back-0=
SRO U/EIS Explanation Focus j
Dist.
Link units ward K/A Only 91 F
2 S
92 F
2 S
93 F
2 S
94 F
2 S
95 F
2 S
96 F
2 S
97 H
2 S
98 F
1 U
Too easy! Also, this action will be duplicated during scenarios! Need to beef this up. Question replaced.
99 F
2 S
Why must this be known by the RO? Comment NOT incorporated.
100 H
2 S
S2 H
3 5
S7 H
2 S
58 H
2 S
S9 H
2 S
S11 H
2 X
U Why is distractor B not correct also? In-plant busses dead. Comment incorporated - changed stem.
S14 H
2 S
Why is distractor A not also correct? How much smoke? Comment NOT incorporated.
Si1 F
2 S
LOK - memory level question! Comment not incorporated.
S17 H
2 S
S33 H
2 U
LOK - Higher level cognitive question!? Need to ask how this was taught in classroom. Comment incorporated.
S41 H
2 S
l
ES-401 6
Form ES-401-9 1
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
(F/H)
(1-5)
Stem cues T/F Cred. lPartial Job-Minutia
- /1 Back-Q=
SRO U/E/S Explanation Focus l l
Dist.
Link units ward K/A lOnly S48 H
2 S
LOK - high level question. Comment NOT incorporated.
S67 F
2 S
S85 F
2 S
S86 F
2 S
S87 F
3 S
S89 F
2 S
S91 F
2 X
X U
Question is directed to Refuel Floor Supervisor responsibilities not SRO Fuel handling supervisor as req'd by K/A. Also, clues to answer in stem.
Comment incorporated - changed stem, added better distractors.
S93 H
2 S
S94 H
2 S
S95 F
2 S
LOK - not a higher level question. Comment NOT incorporated.
S96 F
2 U
LOK - not a higher level question. Comment incorporated.
S97 H
2 S
Distractor C not plausible. Question asks about LPCI response, Distractor C references recirc pump response. Need a better distractor referencing LPCI. Comment not incorporated.
S98 H
2 S
LOK - higher level question. (Too many conditions in distractors?)
Comment NOT incorporated.
S99 H
3 U
LOK - higher level question. Comment incorporated.
100 H
2 E
Typo in 2rd sentence in stem. Typo fixed.
ES-401 Record of Rejected K/As Form ES-401-10 (R8, SI)
Tier / Group Randomly Selected K/A Reason for Rejection ES-401 295030 Low This K/A was selected as a common question, and Reactor Operators are not tested on Technical Tier 1/Grp 1 Suppression Pool Specification Bases.
Water Level / 5 ES401 Tier 1/Grp 2 Generic 2.2.25 Knowledge of bases in technical specifications for limiting conditions for operations and safety limits I
t t
I I
i I~
i i
I
-I I
i r
i.
._.I I
I
2003 NRC Exam Validation Visit JPM Comments No.
Document Comments Resolution I
JPM A1.a Initiating cue states that the computer Changed initial conditions to state the in the simulator may be used to task can be performed at any perform the task, but the first step terminal states that it can be performed at any terminal 2
Step 3 refers to Step 7, but should Added the missing step and refer to a step that is missing.
renumbered the entire JPM.
NA JPM A1.b No Comments None 3
JPM A2 Initiating cue should state " verify all Changed the initiating cue on both items meet acceptance criteria.."
the cover sheet and the cue sheet.
4 JPM RO A3 Initiating cue should state Changed the initiating cue on both
"..Determine if you meet the the cover sheet and the cue sheet.
requirements to perform this task.."
5 Terminating cue refers to "NSS."
Changed reference to Shift Manager (SM) 6 JPM RO A4 Initial conditions protective action Revised recommendation should not include
"...and evacuate to 5 miles the downwind sectors (Area 1) at this time... "
7 Cue before step 1 should not prompt Removed reference to TDF the examiner to give the candidate the Technical Data Form (TDF) 8 Step 1 should not include the TDF Removed reference to TDF 9
Cue before Step 2 should include Added to Cue that meteorological data will be given to the candidate when he either asks for it from the STA, calls the National Weather Service, or obtains by computer from ERIS or the meteorological computer 10 There should be a cue before Step 3 Added Cue to give the candidate the TDF 11 JPM SRO A3 Change cue for Step 4 to "Hans Blix" Changed Cue 12 Change Step 7 standard to Changed Step 7 "Examiner signs for Hans Blix 13 Last four steps of JPM are numbered Numbered steps correctly incorrectly 14 Change cue before Step 9 to ask the Added statement to Cue candidate who has to sign the form for approval 15 Add note for Step 9 that if applicant Added note obtains Site VP signature, make exam comment, rather than fail the
2003 NRC Exam Validation Visit JPM Comments No.
Document Comments Resolution critical step.
16 JPM SRO A4 Rewrite entire JPM to allow every Rewritten SRO to complete the JPM on the first day of JPMs - must include sufficient initial conditions to allow them to classify. Don't classify the scenario that is performed on the first day.
17 JPM B1.a Add steps to allow that the candidate Added note with reference to may perform the steps from memory Operations Department Expectation as it is a management expectation (ODE) 10, Emergency Operating that it should be done as so.
Procedure Expectations.
18 JPM B1.b Modify cue to state that all Modified Cue.
prerequisites are completed.
19 Have the Condenser Low Vacuum Added to Simulator Setup, Initiating Trip Bypass Keylock Switches in Cue, and Cue Sheet.
BYPASS at the beginning of the JPM and start the JPM at Step 7.1.2.3 20 Close N3013-F601, Gland Steam Added to Simulator Setup.
Inlet Hdr Iso Vlv as part of the simulator setup for the JPM.
21 JPM B1.c Add note that will allow the examiner Note added to stop the JPM after Al and A2 channels have been completed.
22 Add note that phone talker will be Note added needed at the SOER printer/terminal 23 Add that there is a person in the Note added Relay Room to the Initial Conditions 24 JPM B13.d Add "less" to the initiating cue for Added "less."
"less than 1.68 psig" 25 Add cue for Step 1.6.2 that if the Added Cue.
candidate reports to the CRS that T4800-F455 won't open and asks for direction, the CRS will tell the candidate to follow the procedure.
26 JPM B1.e Change initiating cue to state that Changed Cue.
prerequisites are complete 27 Add note to step 2 to indicate the location of lights to be verified.
28 JPM B1I.f Add standard to Step 6.1.2.1 for Added standard making plant announcement on Hi-Coin 29 JPM B1.g Correct typos in initiating cues Corrected
2003 NRC Exam Validation Visit JPM Comments No.
Document Comments Resolution 30 JPM B2.a Change all of JPM to place the "B" Changed all cues and notes to Flow Control Valve in service and indicate "A" CRD Flow control Valve remove "A" Flow Control Valve from is in service and "B" is being removed service.
from service.
NA JPM B2.b No Comments None NA JPM B2.c No Comments None
Scenario modifications as a result of the NRC Validation Scenario/Event Comment Changes ES-D-1 Scenario 1 Did not want to use MF Event 5 became MF 0016 BOP(C) 3656(3657) Pressure Main Steam SRV Failure Regulator Failure N001A(B)
B21-F013A (100%)
Event # 5 since BOP had few actions.
ES-D-2 Scenario 1 Event 1 None No Changes Power Change using rods ES-D-2 Scenario 1 Event 2 Applicant's Actions or Corrected the typographical APRM Channel 1 Upscale Behaviors referred to error to correctly refer to APRM # 3 APRM #1 ES-D-2 Scenario 1 Event 3 No reference to T.S. 3.1.3 Added reference to T.S.
Uncoupled Control Rod 3.1.3 ES-D-2 Scenario 1 Event 4 None No Changes 24.206.01 Section 5.1 ES-D-2 Scenario 1 Event 5 Wanted to replace Pressure Replaced event 5 with SRV Pressure Regulator Failure Regulator Failure as event A Fails Open. This is a SRV A Fails Open
- 5.
completely different event.
ES-D-2 Scenario 1 Event 6 Directed Torus Cooling and Directed Torus Cooling and sprays and drywell sprays sprays and drywell sprays to include pump and valve including pump and valve numbers.
numbers.
Scenario/Event Comment Changes ES-D-1 Scenario 2 Wanted to include in Initial Revised Initial conditions to conditions that B Steam include B Steam Tunnel Tunnel Cooler is in service, Cooler in service, and that and that RWM group 41 RWM group 41 rods were rods were to be withdrawn to be withdrawn prior to prior to transferring transferring feedwater from feedwater from SULC to SULC to Single element Single element control.
control.
Wanted to reverse the Event 1 is the Reactivity sequence of Event land Control and event 2 is the Event 2 and to include Normal Evolution.
RCIC Failure to manually RCIC Failure to manually trip as part of event 4.
trip now included as part of event 4.
ES-D-2 Scenario 2 Event 1 Originally Event 2 and did Now Event 1 and includes not include Booth Operator Booth Operator Guidance Guidance on providing Full on providing Full Out Out Coupling Verification Coupling Verification sheets sheets and starting rod and starting rod withdrawal withdrawal with RWM group with RWM group 41.
41.
ES-D-2 Scenario 2 Event 2 Reverse events 1 & 2.
Now Event 2 Transfer from SULC to Originally Event 1 single element control.
ES-D-2 Scenario 2 Event 3 No Comments None ES-D-2 Scenario 2 Event 4 Wanted a failure of RCIC to Event 4 did now includes a Respond to RCIC spurious manually trip from the failure of RCIC to manually initiation.
Control Room.
trip from the Control Room.
Added expectations on various ways of terminating unneeded RCIC flow to the RPV.
ES-D-2 Scenario 2 Event 5 No Comments None ES-D-2 Scenario 2 Event 6 Originally Event 6 did not Event 6 now includes SRO Respond to seismic event include information on and BOP actions on with small break LOCA with Emergency Depressurizing Emergency Depressurizing a failure to automatically when unable to Restore when unable to Restore SCRAM.
and Maintain RPV level and Maintain RPV level above -28".
above -28".
ES-D-1 Scenario 3 (Spare)
Desired whichever SLC Event # 5 MF 3603 SLC pump was started in Event Pump A Trip or 3604 SLC
- 5 to trip after a 5 minutes Pump B Trip based on of operation.
which pump is started.
ES-D-2 Scenario 3 Event 1 No Comments None ES-D-2 Scenario 3 Event 2 No Comments None ES-D-2 Scenario 3 Event 3 No Comments None ES-D-2 Scenario 3 Event 4 No Comments None ES-D-2 Scenario 3 Event 5 Originally Event 5 was a Event 5 now trips, after 5 A SLC Pump Failure failure of SLC Pump A to minutes of operation, start.
whichever (A or B) SLC pump was started, and includes expected RO actions.