ML050260625
| ML050260625 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 09/24/2004 |
| From: | Caruso J NRC/RGN-I/DRS/OSB |
| To: | Conte R NRC/RGN-I/DRS/OSB |
| Cont R | |
| Shared Package | |
| ML041450453 | List: |
| References | |
| Download: ML050260625 (32) | |
Text
J. Caruso 9/24/04 (reviewed comments with R.
Conte-prior to calling licensee)
Comments on NMP-1 Exam Scenarios Note: Bold reflects updates to original comments.
General (Apply to all Scenarios)
In general the operator actions appeared to be in good detail and the format was good.
Mark the steps in the text to clearly indicate those that are related to successful completion of the critical tasks - this is stated in the standards in Appendix D. Done.
For the malfunctions in the scripts, you do not specify which steps would normally be performed by the RO or BOP (e.g. SOP-1, SCRAM procedure). Please do so in your final revise version.
Done.
In the table for quantitative attributes for each scenario list the EOPs and EOP contingencies that you are taking credit for in your count. Done.
Guidance (per procedure) for shift briefs and communications is not included. Done.
Scenario #1 The scenario could be very long (2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />). Prep week made some minor revisions to scenario e.g. generator already parallel to grid to shorten slightly.
You indicate 6 malfunctions. I only counted 5 - list your six malfunctions. Are you counting the MT as a malfunction? Revised D-1 malfunction listing.
CT 01 - Dont believe this is a CT since the EOPs provide guidance if RPV level cant be maintained above TAF. Revised and better defined CT.
Scenario #2 You indicate 6 malfunctions. I only counted 3 - list your six malfunctions. Are you counting the MT as a malfunction? Revised D-1 malfunction listing.
Event #5 -the turbine vibrations and the degraded vacuum lead to the manual scram. This looks to be the start of the MT. Revised D-1 malfunction listing.
Event #6 is also part of the MT but is another Component failure that can be dealt with separately.
Event #7 may really be part of the MT. It really depends on the timing of the degraded vacuum.
For example, if vacuum degrades to the point where the MSlVs would shut or approaching that point at the time of the trip then it is clearly part of the MT. However, if it subsequently worsens then it could be counted separately as you suggested. Time delay is adequate to consider separate component failure.
CT 02 and 03 should be better defined if possible to establish clear failure criteria based on meaningful plant requirements and/or FSAR or PRA insights. Done.
Scenarios #3 Event #6 Looks like the initiator for the MT. Revised D-1.
Event #7 - could count EDG 102 failure and loss of PB 102 if you specify discrete operator recovery actions required, if not part of the MT. Combined 7 & 8 events for component failure. The timing was such for the Loss of offsite power that it occurred after the grid instability and the procedure guidance is different then for event 6, therefore evaluated to be separate component failure along with the loss of EDG 102.
Event # 8 - Looks like a second or subsequent MT. Revised D-1 to combine events 7 818 no real recovery actions for failure of EDG so combined LOOP with EDG failure.
CT 02 should be better defined if possible to establish clear failure criteria based on meaningful plant requirements and/or FSAR or PRA insights. Revised CT-2 With the turbine tripped how is 13 FW pump injecting (it is shaft driven by turbine)? Yes Backup Scenario You indicate 8 malfunctions. I only counted 3 - list your eight malfunctions.
Event #5 looks the start of the major transient (MT). Revised D-1 Event #6 ATWS - component failure Revised D-1 Event #7 looks like just a continuation or worsening of conditions - part of the same MT.
Revised D-1.
CT 01 not a valid CT to classify GE event. Deleted General For the SRO test we could revise some of the Initiating Cues so that the SRO candidate would have to determine the procedure to use. Done 4 JPMs revised see below.
The JPMs appear to be well written.
Please consider color coding all JPM handout paper (Le., procedures and tear off briefing sheets) for the applicants to help avoid exam security issues. Will do for exam.
Please have a briefcase or another container that can be staged in the plant for in-plant JPMs
in the holding area with the sequestering person. Ill provide, if you dont have one available.
Will provide.
For all JPMs and scenarios, please inform us when last used in the training program or for other exams (Le., overlap with last 2 NRC exams and audit exam). No repeats RO JPM outline - JPM F should be safety function #4, Heat Removal. Revised outline.
SRO JPM outline - JPM F safety function instruments - how did you arrive at that ? Revised outline.
JPM a- Withdrawal of Control Rods that Double Notch. 1) Spelling- Control Rod vs.
Control Rods 34-39. 2) Spell out CRDDL. 3) Modify initiating cue for SRO to have them pick correct procedure. Changes made.
JPM b Line-up and lniect CS - 1) Why are steps #6 & 8 not critical? 2) Modify initiating cue for SRO to have them pick correct procedure. Changes incorporated.
JPM c The RO at the controls respond appropriately modify initiating cue to make possibly more than just F panel. Change incorporated.
JPM d- Vent PC via D/W thru the RBEVS- 1) Page 5, (Alt Path) when valve 201-32 will not open could the candidate make the decision on securing drywell venting and establishing torus venting? The way it is written currently may not be Alt. Path.
- 2) Modify initiating cue for SRO to have them pick correct procedure. Licensee agreed not alt. path also revised initiating cue.
JPM e- Perform N1-ST-M4A for EDG 102 (DG Operabilitv). Have the candidate determine alt. path if possible Do we really need this prompt? The operators should all know how to properly secure the EDG. Change incorporated.
JPM f- Control Room Actions Prior to Control Room Evacuation per N1-SOP-9.1. In the initiating cue do we need to specify the applicants specific control room position? Change incorporated.
JPM g- ResDond to a Low SW Header Pressure/Loss of SW (PRA). 1) The JPM is ah.
path but not so designated on the outline. 2) Consider modifying initiating cue you are the BOP watch respond appropriately. Changes incorporated.
JPM h start-up CR vent. - Step 9 critical? Change incorporated.
JPM j- RPV lniection for S/D Outside the Control Room per N1-SOP-9.1. On page 5 it says start the electric fire pump... , But the procedure indicates (step 6.0) that the diesel fire pump is the preferred one to start. The licensee decided to write a different new than the diesel JPM that was already in the bank. The turnover was revised to indicate the diesel was inop.
RO Admin.
Actions for Defeated Annuciators - 1) revise initial conditions to agree with handout. 2) Does not appear to adequately discriminate. The applicant has to determine how many inputs are affected, if not all inputs affected so that need to review alarm contacts on prints then determine a yellow sticker.
Verifv Electronic Clearance - step #4 should also be critical to identify what point needs to be added to complete clearance. Follow-up questionkue, if not offered as part of the answer. Revised to make missing points critical.
Radioloqical Requirements Hiqh RAD & Contaminated Area - 1) spell out WCMOSSE 2) Order steps to match questionnaire format WCMOSSE is the program used by work control. Format of JPM revised.
Perform Actions For External Security Threats. This should be clearly and boldly marked as potential safeguards information and to be withheld from public disclosure due to the sensitivity of some of the information presented. Clearly and boldly marked withhold from public disclosure.
SRO Admin.
Evaluate plant Chemistrv Report - JPM seems overly simplistic does not discriminate adequately?? Rewrote to make more challenging.
Determine PMT for Refuel Bridqe Limit Switch - 1) Typos in initial conditions 2)
What if the applicant wants to do all procedure steps how could we fail?
Rewrote to make more challenging.
Direct An Exclusion Area Evacuation. 1) How is specific route determined -
show me? 2) Convince me this is a good discriminating JPM and not a 50/50 thing about determining the correct evacuation route. Rewrote to make more challenging.
Classifv Event/Complete Notification - How are planning to do this classify event after each scenario and then this JPM is for a notificationthat will be done later?
We will do later in the classroom. Yes, that is the intention.
From: John Munro
$hi-- 4 To: John Caruso Date: 1 1/17/04 4:19PM
Subject:
Re: NMP 1 Written Exam John - I talked with Dave and you have his concurrence Ito proceed as you recommend with a few caveats.
First, you need Rich's approval as well. In other words, Dave's concurrence is NOT direction.
Notwithstanding Dave's concurrence, Rich can still override your recommendation and state in the report that the overall examination submittal was outside the acceptable quality range as discussed in ES-501.
Second, please ensure that the licensee is "counseled at the exit meeting that this is [LIKELY] a one time exception" as discussed in your message below.
Finally, ensure that the wording of the report does not convey the wrong message. In other words, while Dave has concurred in your recommendation to not provide a negative observation regarding the adequacy of the facility's proposed examination, be careful that you do not indicate the converse - i.e., that the examination was adequate for administration as proposed - without some qualification.
>>>John Caruso 11/17/04 03:17PM >>>
Dave and John, Region I would like your approval of the following recommendation.
The regional review of the NMP-1 written exam submittal concluded a total of 26 unsat. questions in the following areas: 4-LOD=1(RO-46, 51, 69 & SRO-IO); 4-Direct Look-up (RO-27, 56, & SRO-6, 12); 6-more than one distractor not plausible (RO-5, 11, 29, 60 and SRO-8, 25); 9-WA mismatches (RO-15, 22, 23, 53, 61, 73, & SRO-l6,17, 20); 3-misc. (RO-16, 44, & SRO-7). The region would believes it necessary to invoke the new guidance in final revision 9, ES-501, E.3.a, page 10, footnote, "...no comment may be warranted if the same error was made in a number of questions..." In the case of the NMP-1, we would like to suggest with the program office's approval the four LOD=1 questions and the four direct look-up are each evidence of similar problems of misinterpretation of guidance. It is the chief examiner's opinion that this exam submittal (written and operating exams) was above average in quality when compared to other recent exam submittals and viewed overall. In addition, this exam appears to be a significant improvement over the last 2 submittals ( U-1 and 2) about two years ago. Although, this exam exceeded the 20% threshold for unsat questions, the contractor author went out of his way to be responsive in resolving the exam team's comments. Furthermore, one could argue that LOD=l is not well defined in the examiner standards and therefore is a somewhat subjective area. If the program office agrees and approves this region I recommendation then the licensee will be counseled at the exit meeting that this is a one time exception.
cc: David Trimble; Richard Conte
NINE MILE POINT 1 2004 Unit 1 NRC SRO EXAM EXAM COMMENTS QUESTION # COMMENT RESOLUTION Added discussion on how battery voltage is addressed to justification for correct /"
-Y' 1 answer.
Revised question stem to remove procedure stem focus. Changed distracter "b" to a new distracter and added new discussion for distracter "b". Changed CFR I/'
2 reference to 55.43(b)(5).
BOLD and CAPS words and phrases in the question and question stem. Revised 3 auestion stem to reference the "above event".
4 Changed RPV level to -90 inches (Fuel Zone) form -97 inches.
5 No change. SRO reviewed and determined okay as is.
6 New Question. Same WA.
7 Double jeopardy with Q M . New question. Same WA.
Added new distracter "c" and new distracter "d" and added correct discussion for 8 each. Changed distracter "b" to EPIP-EPP-18 (EPIP-EPP-08 was a typo).
9 Okay.
10 New question. New WA randomly selected.
Okay.
J 11 12 New question. New WA randomly selected.
13 Okay.
14 Changed to NO references provided.
Revised question stem to remove specific stem focus. Revised references provided 15 to CORE MAP; Nl-FHP-27C will not be a referenced authorized for use.
16 New question. New WA randomly selected.
17 New question. New W A randomly selected.
18 Corrected justification for the correct answer.
19 Okay.
20 New Question. Same WA. ~~
L 21 nn LL IOkay.
I h l uKay. ~ i 23 Replaced question with RO Q#73 which was determined to be SRO only.
-74
. Okav 1
I -
25 lNew Question. Same WA.
~
NINE MILE POINT 1 2004 Unit 1 NRC RO EXAM EXAM COMMENTS QUESTION # COMMENT RESOLUTION 1 Corrected editorial comments in justification.
2 Okav.
3 Okay.
4 Okay.
5 Added new distracters "a" and "b" and "c" and justifications for these distracters.
Added a new bullet to the question to identify FW flow magnitude. Corrected 6 justification for answer to "c".
Added "following a reactor scram" to the following conditions exist. Added a new bullet to the question to identify CR evacuation in progress. Changed SOP number to SOP-21.2 which is the new number - the correct answer is not affected by the 7 SOP chanae. UDdated Drocedure reference to SOP-21.2.
8 Corrected SDellina for the word stabilized ~~
9 Okay.
10 ,Correct the WA reference which was incorrectly listed.
11 New question. Same WA.
12 Okay. Changed to only EOP-4 provided.
No change. Question evaluates the response to of the pressure regulating system t a high reactor pressure and although the reactor pressure sensed is an instrument failure the question still evaluates the system response to high reactor pressure regardless of whether or not actual reactor pressure was changing rather than the 13 instrument failure indicated.
14 Okay. Changed to only EOP-2 and EOP-4 provided.
Replaced with question #23. New question developed for question #23. Changed ct 15 only EOP-2 and EOP-4 provided. . A i 16 A d d .- i C
- s. f-yxh 17 Okay. Changed to only EOP-2 provided.
18 Okay. Changed to only EOP-3 provided.
19 Okav J .
20 Okay.
Added sequence of events occur to the question introduction statement. Removed 21 unexpectedly frpm J@ last bullet. + ,
22 23 New quksti6n. Ne& I$&randomly selected.
added a new bullet (0755) indicating a point in time (500 psig) during the reactor startup. Added a discussion on this information with regards to the point in time for 25 the reactor startup and the action in response to the related reactor pressure.
26 Added correct EOP bases to question stem.
27 New question. Replaced with SRO#12. Changed to No reference.
28 Okay.
29 New question. Same WA.
30 Okay.
Revised question to remove "with a valid initiation signal present" and added "for 31 continued operation of ECI 1 following this isolation, the operator".
32 Added "each quarter" to question.
33 Revised question stem wording to "which valve(s) will".
Changed "CRD Pump 12" is third bullet to "RMCS". Capitalized "QUICKEST METHOD" in the question stem. Corrected discussion for distracters "c" and "d" to 34 SYS 11.
NINE MILE POINT 1 2004 Unit 1 NRC RO EXAM EXAM COMMENTS 37 IOkay.
38 IChanaed to coanitive level 1.
.- 39 Changed to cognitive level 1.
40 Changed to cognitive level I.
Added bullet to identify drywell pressure. Modified distracter "a" so that it is incorrect 41 even if low reactor pressure (50 psig) is assumed.
Essentially a new question now. Changed to cognitive level 3 . Rather than provided a inerting lineup, the applicable prints will be provided and the examinee must determine the valves that are aligned for inerting and those affected by the changing 42 plant conditions.
43 Added bullet to identify the state of the blue lights. /
44 New auestion. New KIA randomlv selected.
45 Changed to cognitive level 2.
46 New question. New K/A randomly selected.
There are no additional actions to take other than manually closing the valves.
Added "by the control room operator" to answers "a" and "c" to be more evident that 47 these are actions and fulfill the action part of the A.2 WA.
48 Added KIA wording.
Added a specific voltage to the question and revised distracter "d". Updated the 49 justification to discuss the significance of the voltage that was added to the question.
50 Okay.
Essentially a new question now. Rewrote the question to remove the LOD=I. Same 51 KIA.
52 Okay.
53 New question. New WA randomly selected.
54 Added "may result in" to "b" and "d" distracters.
Corrected identifiers (I.e., a, d) for justifications for correct answer and for 55 distracters.
Rewrote the question to remove the LOD=I. Same WA. Changed to only EOP-3 Y
56 provided.
57 Removed the statement identifying the failure from the question stem.
58 Okay.
Added discussion to the justification for the correct answer on why this is integrated 59 plant.
60 New question. Same WA. Rewrote the question to remove the LOD=I. /'
61 New question. New WA randomly selected.
62 Okay.
63 Okav.
64 Okay.
Changed question stem to "which one of the following is correct regarding the above ,,/
65 operator actions.
Added discussion why working hour limitations are considered part of the facility 66 license.
67 Okav.
68 Okay.
69 New auestion. Same WA.
70 Changed toladded cognitive level = 1. I/'
71 Okay.
72 Okay.
73 New auestion. Same WA. v
NINE MILE POINT 1 2004 Unit 1 NRC RO EXAM EXAM COMMENTS Changed "many control rods" to "twenty (20) control rods" in the question and added 74 that that "they are at position 48".
--- 75 Added the correct WA reference. The wrona WA was referenced.
ES-401 NineMile Point Unit 1 Written Examination 11/04 Form ,ES-401-9 Review Worksheet Licensee verified minimum overlap of written exam questions for last 2 NRC administered exams at this site. G. Johnson, H. Williams, and J Caruso reviewed this exam completed review and consolidating comments on Oct 2, 04. Provided to licensee on Oct 6,04.
NOTE 1: Reviewed all questions against WA statements specified in the question.
NOTE 2: All answers were verified to be correct in accordance with supplied references. Additional references were requestedused for additional technical verification.
NOTE 3: Initial evaluation of exam quality: 17 UNSAT for RO; 9 UNSAT for SRO overall = 26% unsat.. overall NOTE 4: Resolution of comments in bold and italics print.
- 1. 3. Psychometric Flaws I 4. Job Content Flaws Kp-
- 5. Other 6. 7.
LOK
[F/H) Stem Cues Q= SRO U/VS ExDlanation
-0cus
- units Only F Y N S H 2 Y N S H 3 YINlsI Completed further technical review using drawings C-19957 and C-19409 requested from licensee. I F 1 Y N S F 2 Y N U Answers A and C not credible. Replaced distractors and modified justifications.
H 2 Y N E Include feedwater flow in stem to confirm failure of setdown rather than some other cause for level > 45". Also, the explaination indicates " A is correct answer. Comments incorporated- added new bullet to identify FW flow magnitude and correctedjustifcation for "C.
-- ~~
H 3 YINl I E Modified stem and changed procedure number to reflect new
,procedure number. I F 3 Y N E Typo: Stabilized. Fixed F 2 Y N S Inadequate technical references to confirm none of the remaining distractors are correct Set point question okay in limited number.
Completed further technical review using additional excerpts from N1-OP-20 requested from licensee.
w :
% cn z z z z I I X X cu l-U U L b
=-l c
r-:
(I) 3 z
x X
I Z 7
u a,
0 0
P C
0 c
m S
m Qx W
W Z z I
> Z X
c3 m m N I I 0
rJ
u) W w z z z t t x
LL I CD N
+
z X
I I
IN N LL I"
I I co N
m N "
0
E P) v)
11 Q
s W W w (I)
X I
m I m cu m
-0 I I 0
z z z z I I (3 N N I
(D I
b ILa2 IL (3 (3 m
r-:
W w W z z z
> > > I X
cu I
K c
0 m
K
-m W
10 f 3 Iw X
d (D d d
- 5. Other 7.
-3. Psychometric Flaws - 4. Job Content Flaws Stem artial jack- Explanation ocus Completed further technical review to confirm C is correct answer.
Used Drawing C-19409 requested from licensee. Added WA wording.
Ddistractors is not credible and can easily be eliminated since manual actions should not be required to transfer power for an uninterruptable power supply. Added specific voltage value to stem to make more discriminating so that applicants have to evaluate and determine fault also revised D.
- 1) Distractor B implausible with 111 locked out (in stem) 2) LOD=1 simple power supply question too simplistic does not discriminate between safe and unsafe. Rewrote question using same KIA.
w W z z z z > >
N cu N r r I 0 d In In In In
t a
'0
.-a,
.c t
W w z z z X
(u I LL b
In
z z z Z X
N IN IN I
1%
cn
( d @
3 W
-/Q N
IN IN 4-X II 8 rD (D Is I S $1 8
p3 IN I
I I LL I
K c
0 m
C
-m X
Q w
w W w Z Z Z X
X X X N 0 r I I I v) r-
- 3. Psychometric Flaws 4. Job Content Flaws 5. Other Explanation S3 F 2 Minor revision to stem also bold and caps key words.
S4 H 3 revised values in stem for Fuel Zone level.
Used EOP-3 to confirm correct answer.
S5 H 3 Used EOP-4 and transient mitigation strategy to confirm correct answer S6 F 1
- 2) Distractor B implausible since no Blowdown requirement for ANY hydrogen condition Replaced question using same WA.
c c
0 m
r-: mS W
53 X
m IN I_
I " I" I" Gj
W I -h W v
> Z >-
Z z >
X X X I I In F
cn
?
m
z
E S23 1.
LOK FIH)
F 2.
LOD (1-5) 1-2
- 3. Psychometric Flaws ked. artial Dist.
- 4. Job Content Flaws Ainutia
- 5. Other 7.
Explanation LOD4. This is a GET question. Really equivalent to a set point type question - low discriminatory value.
. Licensee voluterarily agreed to replace this question with RO-73 which was determined to be SRO level.
S24 F 2 S 1
S25 F 2 Will the EAL matrix provided for Question #3 aid the applicant in answering the question? If so, this may be a L O D 4 question.
It should also be common knowledge (without reference to a procedure that a SAE does NOT require a PAR which eliminates distractors A and B as not plausible. . Replaced question using same WA.
The regional review of the NMP-I written exam submittal concluded a total of 26 unsat. questions in the following areas: 4-LOD=1 (RO-46, 51, 69
& SRO-10); 4-Direct Look-up (RO-27, 56, & SRO-6, 12); 6-more than one distractor not plausible (RO-5, 11, 29, 60 and 340-8, 25); 9-WA mismatches (RO-15, 22, 23, 53, 61, 73, & SRO-16, 17); 3-misc. (RO-16, 44, & SRO-7). The region would believes it necessary to invoke the nev guidance in final revision 9, ES-501, E.3.a, page 10, footnote, ...no comment may be warranted if the same error was made in a number of questions... In the case of the NMP-1, we would like to suggest with the program offices approval the four LOD=I questions and the four direc ook-up are each evidence of similar problems of misinterpretation of guidance. It is the chief examiners opinion that this exam submittal (writte and operating exams) was above average in quality when compared to other recent exam submittals and viewed overall. In addition, this exam appears to be a significant improvement over the last 2 submittals ( U-1 and 2) about two years ago. Although, this exam exceeded the 20%
threshold for unsat questions, the contractor author went out of his way to be responsive in resolving the exam teams comments. Furthermore, one could argue that LOD=1 is not well defined in the examiner standards and therefore is a somewhat subjective area. If the program office agrees and approves this region I recommendation then the licensee will be counseled at the exit meeting that this is a one time exception.