ML021070778
| ML021070778 | |
| Person / Time | |
|---|---|
| Site: | Davis Besse  |
| Issue date: | 03/04/2002 |
| From: | NRC/RGN-III/DRS/OLB |
| To: | |
| References | |
| 50-346/02301 | |
| Download: ML021070778 (13) | |
Text
NRC COMMENTS TO INITIAL SUBMITTAL OF WRITTEN EXAM ON ES-401-9 AND INITIAL SUBMITTAL OF OPERATING T EST FOR THE DAVIS-BESSE INITIAL EXAMINATION - MARCH 2002
ES-401 Written Examination Form ES-401-9 Review Worksheet
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD Q# H (1-5 j
Stem Cuesf T/
CredPartial Jo-Minutia
- /1 Back-Q=
SRO U/EIS Explanation Focus D
I D st.
Link units ward K/A Only 2B H
S M
4B H
S M
13B HF S
B NRC: Memory level of knowledge.
Licensee: Agree.
19B F
S M
22B H
S B
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
- 3.
Check the a proprate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
- 4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
- 6.
Based on the reviewer's judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
- 7.
At a minimum, explain any "U" ratings (e.g., how the Appendix B psychometric attributes are not being met).
Page 1 of 9 I
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation
-Focusi Dist.j lLink units ward K/A Only I 24B H
X E
B (1999 NRC Exam)NRC: Reference is an Abnormal Procedure, but KA requires knowledge of EOP event based mitigation strategies related to a turbine trip.
Licensee: Revised parameter to give trip and place applicant in EOP.
25B H
S B
26B F
S B
27B H
S N
36B H
S N
37B H
S B
41B H
S N
46B F
S B
50B H
S N
55B F
X U
B NRC: KA is two part, predict consequences and use procedure to mitigate consequences. Question does not address use of procedure.
Licensee: Rewrote question to incorporate procedure useage.
64B F
S B
66B H
S B
72B H
S N
77R F
X U
M NRC: Reference is an Abnormal Procedure, but KA requires knowledge of reasons for EOP guidance.
Licensee: Rewrote question to require identification of EOP bases.
78R H
S N
82R H
S B
91R H
I S
B Page 2 of 9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cuesl T/F JCred. IPartial Job-iMinutiaLu #/
Back-0=
SRO U/E/S Explanation I
Focus J
Dist.j Linkj B__jnis ard-Q=/ j ny_
92R H
S N
IOOR F
S B
76S H
X U
N NRC: Not SRO only knowledge.
Licensee: Wrote new question for SRO only.
80S H
S B
83S H
S B
91S H
S M
94S H
S B
98S H
S B
1B F
S N
3B F
S N
5B F
S N
6B F
S N
7B F
S N
8B F
S N
9B F
S N
lOB F
S N
11B F
S N
12B F
S N
14B F
S N
15B F
S N
Page 3 of 9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-Q=
SRO U/E/S Explanation Focusj j
L Dist.
Link units ward K/A Only 16B F
S N
17B H
S N
18B F
S N
20B F
S N
21B F
S N
23B H
S N
28B H
S N NRC: ROs required to know TS entry condition?
Licensee: ROs are expected to know TS entry conditions. Also have LO.
29B H
S N
30B H
S N
31B F
S N
32B F
S N
33B F
S N
34B HF S
N NRC: Fundamental knowledge of rod motion vs temperature reactivity effect.
Licensee: Agreed.
35B F
S N
38B F
S N
39B H
U N NRC: KA is two part, predict consequences and use procedure to mitigate consequences. Question does not address use of procedure.
L I
I I
I I
I Licensee: Rewrote question to address procedure useage.
Page 4 of 9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD II I
(F/H)
(1-5)
Stem Cues I/F ICred. iPartial Job-1 Minutia
- /1 Back-Q=
SRO U/E/S Explanation Focus J J lDist.
Link units ward KA Only 40B F
X U
N NRC: KA is two part, predict consequences and use procedure to mitigate consequences. Question does not address predict consequences.
Licensee: Rewrote question to predict consequences.
42B F
S N
43B F
S N
44B F
X U
N NRC: KA is two part, predict consequences and use procedure to mitigate consequences. Question does not address predict consequences.
Licensee: Rewrote question to predict consequences.
45B F
S N
47B F
S N
48B F
S N
49B H
S N
51B H
S N
52B H
S B
53B H
S N
54B H
S N
56B H
S B
57B H
S N
58B H
S N
59B H
S N
Page 5 of 9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q# LOK LOD (F/H)
(1-5)
Stem Cues T/F Cred. Partial Job-Minutia
- /
Back-RO U/E/S Explanation FocusI I
Dist.
Link units ward KAO 60B H
S N
61B H
S N
62B H
X U
N NRC: KA is two part, predict consequences and use procedure to mitigate consequences. Question does not address use of procedure.
Licensee: Wrote new question to address both parts of KA.
63B H
S N
65B H
X U
B NRC: KA is two part, predict consequences and use procedure to mitigate consequences. Question does not address use of procedure.
Licensee: Wrote new question to address both parts of KA.
67B H
S N
68B F
S N
69B F
S N
70B F
S N
71B H
S N
73B F
S N
74B H
S N
75B H
S N
76R H
S N
79R H
S N
80R H
S M
81R H
S B
83R H
3 Page 6 of 9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD
-l t
l IHt (F/H)
(1-5)
Stem Cues TIF Cred.
Partial Job-j Minutia j #1 Back-Q=
SRO U/ES Explanation Focus i-I Dist.
Link units ward K/A Onoly 84R H
S B
85R F
S N
86R H
S N
87R H
S N
88R H
S B
89R H
S N
90R H
S N
93R F
X U
N NRC: If applicant knows item 4 is incorrect, he can eliminate all incorrect distractors. Need to replace item 4 in at least one of the distractors.
Licensee: Rewrote question stem and answers to prevent applicant from determining correct answer by evaluating the distractors only..
94R F
S N
95R F
S N
96R H
S M
97R F
S B
98R F
S N
99R F
S N
77S F
S N
78S H
X U
N NRC: Question does not address ability to monitor/operate the plant fire zone panel.
Licensee: Rewrote question stem and answers to address KA.
79S F
X U
N NRC: Question does not address effect of pressure on leak rate.
__Licensee:
Rewrote question stem and answers to address KA.
Page 7 of 9
- 1.
- 2.
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
- 7.
Q#
LOK LOD II I
I I
(F/H)
(1-5)
Stem Cues T/F Cred.
Partial Job-Minutia
- /
Back-0=
SRO U/E/S Explanation Focus D
Joist.
Link units ward K/A Onl /
81S H
X U
B NRC: Question does not address reason for EOP guidance for loss of IR Nis.
Licensee: Replaced KA because Davis-Besse EOP does not contain guidance on loss of IR NIs.
82S F
X U
N NRC: Not SRO only knowledge.
Licensee: Rewrote question to SRO only level.
84S F
S B
85S H
S B
86S F
S B
87S H
X U
N NRC: Question does not address knowledge of PZR PCS design features that provide overpressure protection.
Licensee: Rewrote question stem and answers to address KA.
88S H
S B
89S F
S N
90S F
S N
92S H
S B
93S F
S N
95S H
S B
96S H
S B
97S F
S N.
99S H
S B
O0S F
S N
Page 8 of 9
NOTE:
- 1. First 30 questions used for sample review.
- 2. First 30 questions received independent verification of references.
- 3. 100% of questions received KA verification.
Page 9 of 9
Davis-Besse Operating Exam Comments Administrative JPMs A.1.a (RO) Stroke Timing of a Locked Valve (2.1.1)
NRC Comments:
- 1)
Filled-in time/date..etc on Locked Valve Log Sheet.
- 2)
IC: EO has a short term OP-16 to close HP-31 breaker.
- 3)
Step 2, Cue: Test requirements initialized in the procedure.
Licensee: Agreed.
A.1.b (RO/SRO) Determine API in Spec with Alarms Inoperable (2.1.11)
NRC Comments:
- 1)
P.4 and 5, steps 9 and 10 critical for SRO.
- 2)
Provide Answer Key (p.18 of DB-NE-03220).
- 3)
IC: Add "plant conditions are as you see them," delete "plant power at 96%."
- 4)
Spell out acronym "CTRM" as "control room" first time used in JPM.
Licensee: Agreed.
A.2 (RO/SRO) Perform Surveillance Test for Post Accident Panel, Determine Instrument Out of Tolerance. SRO Determine Tech Spec Requirements (2.2.12)
NRC Comments:
- 1)
P.1, the initiating cue has the applicant complete "Section 4.4," but the JPM has the applicant complete 4.4.1 and 4.4.2, it should include Sections 4.4.3-.6.
Licensee: Agreed.
A.4 (RO/SRO) Activate the Computer Automated Notification System on an ALERT (2.4.43)
NRC Comments:
- 1)
During validation, RO noted that activating CANS is an SRO, not an RO, function.
Training department verified issue.
- 2)
IC: Delete "You are the spare RO."
- 3)
Step 1 Cue: If asked, plant announcements have been made per Step 6.1.
- 4)
Step 3 Comment: Add "(2222)".
- 5)
Step 8 Cue: Add "1-800-405-4758 and PIN #2533024"
- 6)
Step 9 Cue: Add "The control room has received a fax; all appropriate ERO personnel have called in."
Licensee: Agree with comments. Rewrote new A.4 JPM for RO, "Notify State and Counties on an ALERT."
Page 1 of 3
System JPMs B.1.b Align ECCS Suction to Emergency Sump MSL3 NRC Comments:
- 1), Section 2. Suction Transfer, states "the SRO shall direct this part of the procedure." If SRO directs each step, procedure is not discriminating.
Licensee: Agree with comment. Procedure recently changed from an "in-hand procedure performance" to having the SRO direct each step based on INPO evaluation/input. Replaced with "Operate the LPI System During a LOCA."
B.1.d Borate the Makeup Tank Using Attachment 13 of DB-OP-02000 NS1 NRC Comments:
- 1)
Steps 2 and 3 are not critical, only "verifying valve position."
Licensee: Agree with comment.
B.1.e Trip an RPS Channel DAS7 NRC Comments:
- 1)
Step 2, add "Key 33."
- 2)
Step 3 Note: Delete "Provide cues as required."
Licensee: Agree with comments.
B.1.f Start CTMT Purge On the Penetration Rooms NS8 NRC Comments:
- 1)
Step 1, provide procedure with prerequisites signed off.
- 2)
P.4, Step 4, only critical if applicant has to change valve position.
Licensee: Agree with comments.
B.1.g Manually Trip SFAS DAS2 NRC Comments:
- 1)
IC: Add: "(SFAS Levels 1-4 including containment spray)"
- 2)
Step 2 not critical.
- 3)
Step 8 Cue: "Another operator will verify SFAS actuation."
Licensee: Agree with comments.
B.2.b Line Up BUSW Pump to Service Water D4 NRC Comments:
- 1)
P.1, plant is using Rev.2 Licensee: Agree with comments.
B.2.c Loss of a TPCW Pump D8 NRC Comments:
- 1)
P.1, front initiating cue says 4.1.2, but the P.2 initiating cue says 4.1.3.
Licensee: Agree with comments.
Page 2 of 3
Scenarios (1,2,spare)
Scenario 1 NRC Comments: None.
Scenario 2 NRC Comments:
- 1)
ROP gets both N and R events. Stay consistent with rotation, give the N event to the ROS (Upgrades are only in 1 scenario). Want to ensure there are enough C and Is based on applicants being in two scenarios.
Licensee: Agree with comments. Replaced N event and gave it to the ROS.
Scenario 3 NRC Comments: None.
Page 3 of 3