ML20237D007
| ML20237D007 | |
| Person / Time | |
|---|---|
| Site: | Cook, LaSalle, 05000000 |
| Issue date: | 01/10/1985 |
| From: | Persinko D, Rameysmith A Office of Nuclear Reactor Regulation |
| To: | Black K, Booher H, Lyons J, Rowsome F NRC OFFICE FOR ANALYSIS & EVALUATION OF OPERATIONAL DATA (AEOD), Office of Nuclear Reactor Regulation |
| Shared Package | |
| ML20237C946 | List: |
| References | |
| FOIA-87-652 NUDOCS 8712220311 | |
| Download: ML20237D007 (40) | |
Text
- ______ __ _ ___-__ _ __-______-_ ____-
o
'g UNITED STATES
(
g g
NUCLEAR REGULATORY COMMISSION
'j WASHINGT ON. D. C. 20555 o,
a
%,..... /
JAN 1019B5 MEMORANDUM FOR: Harold R. Booher, Chief Maintenance and Training Branch Division of Human Factors Technology, NRR Frank H. Rowsome, Chief
. Human Factors Issues Branch Division of Human Factors Technology, NRR Kathleen M. Black, Chief Nonreactor Asses _sment Staff Office for Analysis and Evaluation of Operational Data James E. Lyons, Chief i
Technical and Operations Support Branch Planning and Program Analysis Staff, NRR FROM:
Ann Ramey-Smith, Engineering Psychologist Human Factors issues Branch Division of Human Factors Technology, NRR Drew Persinko, Maintenance and Surveillance Engineer Maintenance and Training Branch Division of Human Factors Technology, NRR
SUBJECT:
TRIP REPORT FOR LASALLE AND D. C. COOK SITE VISITS REGARDING WRONG UNIT / WRONG TRAIN EVENTS This memorandum documents the activities and findings of an NRC staff visit to the LaSalle and D. C. Cook sites on November 12-15, 1985. Members of the NRC team for this visit included A. Ramey-Smith (DHFT) and D. Persinko (DHFT). The site visit was conducted as part of the short-tenn effort to determine whether simple, low cost improvements can be identified and implemented to reduce the frequency of wrong unit / wrong train events I
occurring at nuclear power reactor facilities.
Upon completion of all site visits, the factors contributing to the events will be evaluated and a report issued which discusses causes and recommendations.
Long term assessment will be addressed as part of the Maintenance and Surveillance Program Plan being conducted by DHFT.
General Information The LaSalle site is located 11 miles southeast of Ottawa, Illinoisr There are two reactors, LaSalle 1 and.LaSalle 2, located at the site, each of which has a maximum dependable capacity (net) of 1078 MWe.
LaSalle I was placed l
50 5 b
J [/) 3 b D h
?7//
I 8712220311 871218
[
. into commercial operation on January 1,1984, and LaSalle 2 on October 19, 1984.
Both units are General Electric BWRs and the architect /
engineer for both units was Sargent and Lundy. The licensee is Comenwealth Edison.
The D. C. Cook site is located 11 miles south of Benton Harbor, Michigar, and has two reactors, Cook I and 2.
Both units are Westinghouse PWRs and the architect / engineer for both units was the American Electric Power Service Corporation.
Cook I has a maximum dependable capacity (net) of 1020 We and was placed into commercial operation on August 27, 1975. Cook'2 has a maximum dependable capacity (net) of 1060 MWe and went into comercial operation on July 1, 1978. The licensee is Indiana and Michigan Electric.
Site Visit Agenda The discussions and in-plant observations centered around six wrong unit / wrong train events that occurred at LaSalle between 1983 and.1985, and four that occurred at Cook between 1981 and 1985.
The LER numbers for these events at LaSalle are 373-83-140 and 84-017,84-072, 85-012 and 85-020 and at Cook are 315-81-005,83-009, 83-048, and 84-014. Also discussed at Cook was the occurrence discussed in the Daily Report of 4/25/85. During both site visits, the NRC team inspected the locations of the reported wrong unit / wrong train events to the extent possible, and discussed the events with plant management as well as many of the individuals directly involved with the s
event.
Enclosures 1 and 2 provide the sequence of events resulting in t.he LERs at LaSalle and Cook, respectively, the licensee's conclusions regarding the event, and NRC staff observations.
Observations at LaSalle Color Coding.
LaSalle utilizes a color coding scheme of light tan for Unit 2 and white for Unit I which are painted on the floors of the respective units. The team notes that because of the similarities in the two colors, one cannot easily distinguish between the colors to determine unit except at the dividing line between the units where the two colors meet and provide a discernible contrast.
Plant Labeling.
Component labels do not correspond to the unit color coding scheme but rather, denote systems (e.g., safety, nonsafety, water, air).
Metal embossed tags attached to valves are used for valve identification (Figure 1) with plastic engraved tags and black stenciled labels used to identify electrical cabinets and other components (e.g., chain operators)
(Figures 2 and 3). Stenciled black labels are also used on the outside of doors throughout the plant to aid in the identification and location of equipment located beyond the door (Figure 4). Additionally, because of the similarity of equipment in the electrical equipment room, a map showing t.he location of the equipment within the room is presented as one enters the room (Figure 5).
Below the map is a listing of the panel numbers and a description of the panels.
i a
l
. k. L ?i.-r,
. F i.c f.: 4.c.,
c 5 dfM *M.,%f.
j 7
-;..- Q.
r j
e'
[ '
- Yi -
1;W f
m
.; *D., g.. phr
-t j
l
..g k
T'.t.[
~ 7N[.
l MMA
-_4, b [ '
l
=
s
"(. -
\\w?O C'~
y.=
'e M.
. db*#M
@,s,
4 a
't;QQQggQ.}l Figure 1.
LaSalle: Valve Identification Tag
-y
- e _
h
.T.
p_[
\\ i
= w!M
-%I w y
g v
v A
a
'f.g%; ' d..
R, i.,,.
E C' L f'*,' s '_
. v.'
d
,#, f ~
Ij n2 r,f C-
=
~ % ~ 'gf,}kQfyw
~'
~ ' -
- . y.c,k $, h ; q w
m h
h Figure 2.
LaSalle:
Cabinet Labeling
=, :::.;;.
E STOEMf...A gwa-rew=
4 e m m m a; ma D,Ws&C
- i -~
g erw,v ci.
t i
l msnm.y; y I
c, iWl24N;'1: _
y Y
I L
r& FR.T VL_e'[ j 4
- S d
M_
43p. l,4 4
it, %g T 03R: R#deiE RE00 m b.
Isrswta p.
ARE PROHlalTEDT*
1 f
AMbME M92'M El 5E N' 9 J*,hM,4%*m TECH. SPEC.
o;.261 Rwa 8jg flRE DDDR
~
3
- .M LNWR KEEP CLOSED g
l FA?;;
AT ALL TIMES s~
8we s
+
.a
. @ !!f 4
?
l I
S-) N. E
$$dM@Njhi Figure 3.
LaSalle: C 0 erator Figure 4.
LaSalle: Labeling of Rooms l
q e
i':-
4
.i rj s d
R-
)3myi <c E
s.n m
..'j'l
- i. }
t e
e n.
1,*
)1)
.$'5(' #
I'.
t I
?.
,t.
rE a',j.!
h ~'!)1}?$.'
.?. gg, -
U, &:.
~
.w, fc.
-4.. ;.
7 m
'l&
N_.,. n.. g.. +.~
,<v.
,=_...
~
.l. a,y@. -
M.
yf ma
.. sty:,
s.
J;t*
g:QJg'T
- t z.
f'N;,Yf g
2- ;._. %
ECM
*M
&yysft k"2:.; R5*
&ws 55r.
ty;<
= =.
y u
7dp m
. e c.-
=
up g;-[wN.@.?.:M ij 2 :::C::T bte
'3 s
I r.__
n W $ 7.
I Cr- --
?l '.;
di D* Q. ME"- -
- 'gc i
y v; rr T
"L*:
N c.
U. w
.tI Yg
%t~.z ;,
h
. +
y
=m; 4-y='t.
l.Q E
}g
. gf
-F Figure 5.
LaSalle: Map of Electrical Panel Location in Electrical Equipment Room.
l l
l 1
1
)
l Except for a few operating procedures, most operating procedures apply to both units with Unit 2 component designations listed after Unit 1.
When asked their opinion regarding contributing factors to WU/WT events, a Commonwealth Edison staff member believed that communications breakdowns and a lack of attention were the dominant contributors.
Because Commonwealth Edison is the licensee for both Dresden and LaSalle, matters directed from the corporate headquarters such as the investigation of l
events involving human error and training are handled identically for both l
LaSalle and Dresden and have been previously described in the staff's WU/WT trip report for Dresden (memo dated June 11, 1985, from Virgilio to Edison, Regan,Bocher, Black).
Observations at D. C. Cook philosophy. As stated by D. C. Cook personnel, a goal for the plant is to reduce the probability of human error and minimize their effects when they do occur. The programs discussed below provide examples of how D. C. Cook is trying to implement this philosophy.
Color Coding. A color code is being implemented to differentiate between the two units, with Unit 1 being orange and Unit 2 being blue.
The cabinets and panels are being painted, doorways are color coded and signs are provided, and tank bands have been painted (see Figures 6, 7 and 8).
Labels are color coded by train and channel for safety grade equipment; channel 1 is orange, channel 2 is blue, channel 3 is white, and channel 4 is yellow.
I&C procedures are color coded by protection channel.
Plant Labeling. The D. C. Cook staff indicated that everything on flow prints and everything that has a human interface will be labeled.
Cook has selected aluminum tags for labeling components as follows:
orange - Unit 1 mechanical components outside of containment; blue - Unit 2 mechanical
{
components outside containment; and, green - mechanical components on shared j
systems outside of containments.
Stainless steel tags will be used for labeling mechanical components inside both containments or in adverse
{
environments outside containments.
The readability of the aluminum tags has been enhanced by increasing the contrast between the tag lettering and the tag background. This was done by filing the raised letters to remove the color and contrast with the colored tag backgrounds (see Figure 9).
The same technique would not work with the stainless steel tags, resulting in what the NRC staff considered tags with very poor readability.
Each component is assigned a unique mark number. The first digit is the unit designation; in the second series of digits, the first letter is the sy' stem, the second is the type of device (e.g., if an instrument), and the third letter would indicate whether, for example, the component is a switch.
The l
facility database contains the official noun name of devices which will be used on all procedures and drawings. A list of standard abbreviations was
EWilTcMRT*V" au ali. il.!..
's.
.. - u.:!, :
f Y
esMi I
l (g$'(
.)+ ;.Ej +t, s
)
? ='-#Att k.
1
^ uk ) ;y.
. f '?Z?
5 h-i
'v Figure 6.
Color-coded cabinets at DC Cook. Unit I cabinet is i
orange. Unit 2 is blue.
I I
.,l f
=)),
-l. s r.
b 1
ma.m
{
..i, j
?
.. ::.2 7:
.y.1 I
a l
$;_N Figure 7.
Color-coded doorway and entrance sign at DC Cook.
Entryway is blue.
l i
'I
,... mi p
- ' Ab
- Q_R,
? %' ' '_
~
~
ubM5 c_- -
,u E t *:F&h?h1A Figure 8.
Color-coded tank band at DC Cook. Iand is orange.
9
l m
- _, 7 a y ---
~.th
[
1, 4
k l
v< s t !
I l
f
/.x 0
J m
{i' z.
}
.x i
)
1 N.q.
a
,i,
(
.qh m f l
^
-r hf filW> ww v._nwylf ry d Figure 9.
Color-coded component label with paint filed off lettering to improve readability. Also shown is a color-coded valve seal.
9 6
mRf a
4 developed during conduct of the control room design review. The Cook personnel sew the plant labeling process as an on-going, iterative process.
l A procedure exists for maintaining labels, and plant maintenance procedures include a check-off for the adequacy of labels on the equipment being maintainec'. An operator aids procedure has also been developed to control the quality of operator aids.
D. C. Cook personnel also described an effort underway to provide pipe labeling on all plant systems.
These labels (see Figure 10) are color coded according to unit and train and include flow arrows.
The Cook staff indicated that this pipe labeling has been approved for use on stainless steel piping, but had not yet been approved for use in containment.
Electrical cabinets will be labeled inside as well as outside per a commitment to INPO. The Cook staff indicated that a circuit directory will also be prominently placed inside the cabinet door or nearby if no door is present.
During the tour of the facility, the NRC staff was stunned by the gross lack of permanent labels in some areas of the plant and yet impressed at the stark difference between those areas that had been upgraded and those yet to be done. Components in areas where permanent labels had not yet been installed were identified with duct tape and magic marker lettering.
print Program.
In mid-1982, D. C. Cook initiated its Regulatory Performance Improvement Program (RPIP). As part of that effort, a crew of plant personnel, including those with drafting experience, started " walking down" prints. The following were the major aspects of that program: 1) to identify sequence of connection errors in which notes and references guide the user to the wrong next drawing; 2) identify components on the flow prints without unique component numbers and input the unique number to the facility database; and 3) clarification of drawings to identify those that are not easy to understand. To assist instrumentation people and aid in labeling, Engineering Control Procedures were developed to augment information on flow prints. A unique numbering convention was developed.
Previous to establishment of the numbering scheme, construction numbers were used, a practice that did not address the users' needs. The Cook staff indicated that the present numbering scheme is much more user friendly.
Sealed Valve Program. Cook has instituted a sealed valve program to protect and give administrative control over manual valves.
If a valve can inhibit a safety injection flow path, it gets a color coded (by unit) safety seal that 1s uniquely numbered.
(Fire protection seals are red.) Periodically, the l
positions of these valves are verified.
A procedure or clearance is re, quired to change the position of these valves, Computerized Clearance Permits.
D. C. Cook personnel indicated that a computerized clearance permit system was being established. At the present time, repetitive clearances have been computerized.
Eventually, all i
(pf7
. i.; ( (K"3 ut tsw b.;Q,,,g z;w c.-.w
,% 3
' f4;/>{
tN,T ' "
- 4hk,Q *'
('
{ i s, h w' l-- Q R 'l~"3A ggy
-+
- g.j Figu: a 10. Color-coded piping labels at DC Cook. The borders of these labels are orange.
l I
l I clearances will be incorporated into a computer system which will have the capability to check the clearance against the facility data base and will print out tags.
Miscellaneous. The D. C. Cook staff provided a hand-out that summarized the activities completed or underway to reduce the frequency of events in which human error is a major contributor.
That list is provided as Enclosure 3 to this report.
i Exit Meetings At the exit meetings at both sites, the NRC team expressed its appreciation to the LaSalle and Cook staffs for their cooperation in planning the visit, coordinating the tour and discussions and providing pertinent information.
These actions made the site visits informative and productive.
The teae would like to note that the staff at Cook was extremely well prepared for our visit. The handouts, containing labeling samples supplemented by the well-prepared presentations, assisted the staff greatly in understanding the WU/WT events at Cook and Cock's initiatives to reduce human errors.
The team also expressed its appreciation to the Resident Inspectors at LaSalle and Cook for their assistance. Mike Jordon, the Senior Resident Inspector at LaSalle, was extremely helpful in arrar.ging discussions with the licensee.
J?
Ann Ramey-Smith, Engineering Psychologist Human Factors Issues Branch Division of Human Factors Technology, NRR O
7lAwT9.Lwv9%
Drew Persinko, Maintenance and Surveillance Engineer Maintenance and Training Branch Division of Human Factors Technology, NRR
Enclosures:
As stated cc:
See next page 4
.mm.__ _ _ - -.
I i i
cc:
J. Funches F. Hebdon G. Cwalina 1
E. Trager i
V. Benaroya l
A. Bournia l
D. Wigginton i
l WU/WT EVENTS AT LASALLE (DOCKET NO. 50-373 Unit 1)
(DOCKET NO. 50-374 Unit 2) l I.
LER 373-83-140 Incorrect Jumper Installation (Wrong Unit)
A.
The following event information was provided by the licensee in the LER:
"On November 16, 1983 during removal of unit separation jumpers prior to performing LES-PC-02 and LES-PC-10, an incorrect jumper installation was identified. The jumper was installed on September 15, 1983 in accordance with LAP 240-3, Attachmert B, Jumper and Block installation form.
"The jumper should have been installed between terminals CC-79 and CC-8C in Panel IPA 14J to prevent a_ Unit 2 isolation signal from automatically starting the Standby Gas Treatment System.
The jumper was actually installed between terminals CC-79 and CC-80 in Panel 2PA14J which prevented Unit I and Unit 2 Standby Gas Treatment Systems from automatically starting on a Unit 1 Division 1 isolation signal. The-jumper was immediately removed and a Deviation Report was initiated.
"LaSalle Unit I was in Cold Shutdown when the incorrectly placed jumper was identified. During the time that the jumper was in place, LaSalle Unit I was in start-up testing at power levels between Cold Shutdown and 100%.
" Electrician installing the jumper failed to note that the jumper form, LAP 240-3, Attachment B, specified that the jumper was to be placed in Panel IPA 14J instead of Panel 2PA14J.
He was in the process of placing numerous jumpers in the Unit 2 panels to prevent spurious starts of shared safety related equipment caused by invalid signals from Unit 2 (Unit 2 is in a pre-fuel load status).
I
~2-
"The operator that verified the jumper installatice, did not understand the significance of independent verification with respect to this job and failed to notice the "1" on the panel EPN.
In addition, the operator that verified the jumper installation was with the electrician when the jumper was being installed which defeats the purpose of independent verification.
"A Daily Order book entry was made by the Unit Operating Engineer on November 28, 1983 outlining the problem that had occurred and directing the Shift Engineers to begin implementing a time separation for independent verification of jumpers immediately.
"A letter is being issued by the Station Superinterdert to all department heads re-emphasizing the requirement for independent verification of items as specified in LAP 240-6, Terpcrary System Changes and LAP 900-4, Equipment Out-of-Service Procedure.
"A discussion was held with the individuals involved with respect to ensuring that jumpers are installed in the correct location and the necessity for independent verification.
I "A discussion was conducted by the Master Electrician on November 29, 1983 with the Electrical Maintenance Department with -espect to this l
event. The importance of ensuring that jumpers are properly installed i
i and for independent verifications was stressed to all individuals."
i B.
NRC Discussions with Plant Staff During the site visit, the NRC team spoke with a member of the LaSalle staff about the event.
From that discussion, it was learned that the incident occurred at a time when Unit I was in operation and Unit 2 was under construction. The valves in question were part of a common isolation group whereby the valves from both units close upon a signal from either unit, The purpose of the jumpers was to prevent a Unit 2
_ _ - _ - _ _ - _ \\
signal from affecting Unit 1.
The logic is such that in order to typass Unit 1 valves from being operated by a Unit 2 signal, the jumper stould have been installed in a Unit 1 electrical panel. The panels, located in separate rooms, were labeled on the front but it was unclear whether they were also labeled in the back where the jumpers were installet.
\\
The electrician involved was considered to be experienced.
The independent verification performed was actually dual verification due to a lack of understanding about " independent" verification by the independent verifier.
\\
Currently, a separate procedure is used for each unit and the procedu-es now provide unit number cautions.
Additionally, the panels are identified on the backside as well as the front.
Panel identification also exists on the inside of the rear door so that one will know the panel number even with the door open.
The NRC team was able to view the panels where the jumper was inadvertently installed and the panel where it should have been installed.
II. LER 373-84-071 Inadvertent Closure of Reactor Water Cleanup Outboard Isolation Valve (Wrong Component)
A.
The following event information was provided by the licensee in the LER:
"On 10/24/84 at 1330, the Unit 1 Reactor Water Cleanup system (RWCU, CE)
Outboard Isolation Valve,1G33-F004, inadvertently closed, due to a Group 5 Isolation Signal (JM) generated during the logic testing of the Primary Containment Isolation System (PCIS).
At the time of the, occurrence Unit I was in Cold Shutdown.
LES-PC-10, Primary Containnen:
Isolation Manual Initiation Logic Test, required that specified control power breakers to valves be opened prior to testing the PCIS Manual
- L l
Initiation logic, to prevent spurious isolation of the systems during testing.
These breakers were specified by the Shift Foreman on Attachrent A of LES-PC-10.
During the testing of the Division 1 PCIS I
Manual Initiation logic, the RWCU Outboard Isolation Valve, IG33-F004, unexpectedly isolated upon PCIS initiation.
0 "The RW:V Outboard Isolation Valve closed on the manual initiation signal because the wrong. control power breaker had been opened.
Attachment A of LES-PC-10 had specified that breaker A4 at MCC 134X-1 be opened for IG33-F004, instead of the correct breaker E5 at MCC 135X-1.
The reason for the wrong breaker being identified on the Attachment is due to the closeness in the equipment number of 1G33-F004, instead of the Attechrent is due to the closeness in the equipment number of 1G33-201-4 which is fed from MCC 134X-1, A4, and an error made when the control power checklist was initially developed.
" Operating Equipment Attendants (Non-Licensed Operators) have been
~
instructed to look at the equipment name labels identified on the MCC breakers, as well as the MCC number and breaker identified by a che'cklist when removing and returning equipment from service.
Additional emphasis has been placed on the importance of independence of the second verification."
8.
NRC Discussions with Plant Staff From discussions with the plant staff, it was learned that the shift foreman who generated the list of valve breakers to be operated during this surveillance mistakenly listed the power supply for valve number IG33-2004 instead of 1G33-F004.
The error was attributed to the closeness of the two numbers.
The list had been generated from.
electrical drawings each time the surveillance was to be performed; however, subsequent to the event, a standardized list has bien developed.
The plant staff member indicated that developing standardized outage lists is still in its infancy at LaSalle.
_ ____ _ __ - _ _ _ _ _ _ _ _ _ _ _ __ _ _ _ The NRC team did not view these breakers because physical locati:m was-not a contributor to this incident.
III._ LER 374-84-017 Reactor Scram on Loss of Feedwater (Wrong Comp:nemt) l l
A.
The following information was provided by the licensee in the 19:
"The Unit 2 Reactor was scrammed manually on 5-3-84 at 2340 when it j
became apparent that the Motor Driven Reactor Feed Pump (SJ) c:ui:: not be restarted and reactor vessel level could not be maintained cocre 12.5 inches.
At the time of the event, the unit was proceeding witt a normal shutdown to obtain data for the Start-Up Test Program.
"The NSO (Licensed Operator) on the unit decided that at that ici.t in the shutdown, it was convenient to close all the Reactor Feed hma Warming Line Valves.
These valves provide a flow path to the Faactor Vessel even though the discharge valve on the Reactor Feed Pum; is closed.
With any Warming Line Valve open on any of the Reactor Feed Pumps, it provides enough make-up to the vessel that make-up wo21c exceed blowdown during low power operation and vessel level wou'd rise at a rapid rate, which is not desirable.
Therefore, the NSO on the unit instructed an Equipment Attendant (EA) (Non-Licensed Operator) o close the warming line valves on all three feedwater pumps. The wanting line valves on the 2A and 2B TDRFP were correctly valved out.
Inunedta tely after, the Equipment Attendant valved out the Balancing Line Va*ve 2CB037 on the Motor Driven Reactor Feed Pump instead of the Waming Line Valve 2FWO37.
The Motor Driven Reactor Feed Pump tripped and cruid not be restarted.
Both valves, the Balancing Line Valve 2CB037 and the l
Warming Line Valve 2FWO37, are in the same room and about five 15) feet apart.
"The Equipment Attendant who closed the Balancing Line Valve" 2C1037 instead of the Warming Line Valve 2FWO37 indicated that he thoutnt he was on the correct valve because the number "37" caught his eye en the
_m____
l valve Equipment Part Number. He did not read the entire noun name of the valve that was indicated on the associated valve tag.
The Equipment Attendant also thought he was on the correct valve when he read the number "37" on the valve tag because he had just closed the Warming Line Valves on the A and B Turbine Driven Reactor Feed Pumps and there was.
4 similarity associated with the piping and valves between the Turbine Driven Reactor Feed Pumps and the Motor Driven Reactor Feed Pump. The Equipment Attendant was not given any irdication that a Balancing Line Valve 2CB037 existed or the consequences if it were closed."
The following corrective actions were taken:
"1.
A Caution Card was placed immediately on the Balancing Line Valve 2CB037 with instructions indicating that it should not be closed unless the Motor Driven Reactor Feed Pemp is to be taken out of service (for both Units 1 and 2).
2.
Work Request L36421 was initiated so that repairs could be made to the Motor Driven Reactor Feed Pump.
(Repairs have been completed and the Motor Driven Reactor Feed Pump is back in operation.)
3.
Signs were placed on both Unit 1 and 2 Balancing Line Valves ICB037 and 2CB037 indicating that the valves should not be closed unless the pump is out of service. This replaced the Caution Cards.
4.
AIR 01-84-67076 was submitted to ensure personnel (Equipment Attendants and License Personnel and all new Equipment Attendants) were trained on the purpose of the Motor Driven Reactor Feed Pump Balancing Line.
5.
The following procedures were reviewed to ensure that the Reactor feed Pump Warming Lines were in the proper position for the plant condition:
a, LOP-FW-01M U-1 FW Mechanical Checklist b.
LOP-FW-02M U-2 FW Mechanical Checklist c.
LOP-FW-01 Feedwater System Filling and Venting j
d.
LOP-FW-02 Feedwater System Draining e.
LOP-FW-03 Start-Up of Motor Driven Reactor Feed Pump -
Change submitted f.
LOP-FW-06 Shutdown of Motor Driven Reactor Feed Pump to Hot Standby - Change submitted
-g.
LOP-FW-04 Start-Up of a Unit 1 TDRFP - Charge submitted h.
LOP-FW-05 Shutdown of a TDRFP - Change subnitted 1.
LOP-FW-07 Preparation for TDRFP S/U with Meir. Steam System Pressurized I
j.
LOP-FW-08 Condensate Recirc to Condenser via Heater Drain System k.
LOP-FW-09 TDRFP Exhaust Duct Isolation During Unit Operation 1.
LOP-FW-11 Start-Up of Unit TDRFP - Change subaritted m.
LGP-1-1 Normal Unit Start-Up - Change subnitted n.
LGP-1-2 Unit Start-Up to Hot Standby - Change submitted o.
LGP-1-3 Unit Start-Up from Hot Standby to Power Operation - Change submitted p.
LGP-2-1 Nonnal Unit Shutdown - Change subnitted q.
LGP-2-2 Shutdown to Hot Standby - Change submitted l
l r.
LGP-3-2 Reactor Scram - Change submitted i
i The above procedure changes will be tracked by AIR 01-8L-67077.
In addition, the procedures were reviewed to ensure correct reference l
and use of the Balancing Line Valves."
I
.g.
j 1
B.
NRC Discussions'with Plant Staff Plant staff told the NRC team that this is not the first time this event has happened. The warming lines which were to be closed warm the reactor feedwater pump casing to reduce thermal stresses in the casing which are developed from the high temperature (approx. 350 F) developed in the reactor feed pump. The balancing lines, one of which was incorrectly closed, keep the motor driven reactor feedwater pump thrust bearing in position.
At the time of the event, the valves were labeled with metal embossed identification tags; it is unclear whether these tags were the new or old tags.
The equipment attendant (EA) indicated that this was the first time he performed this action and the training he received did not cover this plant action and specifically, did not cover the differences between the warming and balancing lines. The operator was considered a good operator.
As a result of this event, this plant action is covered in classroom and on-the-job training. Caution cards have been installed on pump control switches which indicate when the warming valves are closed and signs warning not to close the' balancing valves with the pump running have been installed on the valves.
The procedures in use did not say specifically which valve to close and required operators to look on drawings for the valves whereas now, procedures call out the valve to be closed. Additionally, instead of specifying FWO37 to be closed, another valve (FW115) in series with FWO37 has been specified.
The FW115 designation does not resemble FWO37 and this valve is physically located outside of the room containing the warming line valves.
The NRC team was able to view valve FW115; however, due to radiological consideration, was not able to view the warming line valves'or balancing line valves involved in this incident.
. _ _ _ _ - _ - _ _ - - - The NRC team observed that metal embossed tags are difficult to read
]
unless lighting conditions are good due to very little contrast between the raised identification numbers and the background.
IV.
LER 374-84-072 VR Isolation Damper Closure on Wrong Unit A.
The following information was provided by the licensee in the LER:
"On 10/29/84 at 1952 hours0.0226 days <br />0.542 hours <br />0.00323 weeks <br />7.42736e-4 months <br /> an Equipment Attendant (nonlicensed operator) closed the Unit 2 Reactor Building Ventilation (VA, VR) isolation dampers 2VR04YB and 2VR05YA. This caused the Unit 2 Reactor Building Ventilation systet to shutdown.
The Equipment Attendant had been instructed to open the Unit 1 Reactor Building Ventilation dampers IVR04YA, IVR04YB, IVR05YA, and IVR05YB but inadvertently closed dampers on the other unit.
The Unit 2 dampers were reopened and the Unit 2 Reactor Building Ventilation system was restarted at 1953 hours0.0226 days <br />0.543 hours <br />0.00323 weeks <br />7.431165e-4 months <br />.
l "At the time of the incident Unit I was in cold shutdown and Unit 2 was in the "Run" mode at 73% power.
The Reactor Building differential pressure with respect to outdoors was being maintained by Unit 2 VR.
Unit 1 VR had been shutdown so that LaSalle Operating Procedure LOP-RP-01 could be performed.
This was used so electrical power could be transferred from Reactor Protection System (EE, RPS) "B" Motor l
Generator Set to Alternate Power. As a precaution against momentarily losing power to Unit 1 RPS the Unit 1 VR system was shutdown and secondary containment isolation dampers IVR04YA, IVR04YB, IVR05YA, and IVR05YB closed.
If power were to be momentarily lost to the damper actuators during the transfer the isolation dampers would fail closed and shut.
I "After the power has been successfully transferred the Unit 1 Control Room Operator (NS0) instructed an Equipment Attendant (EA) t'o open the Unit 1 VR isolation dampers so the system could be restarted. The EA inadvertently went to the Unit 2 isolation damper control panel 2PL27JB l
l
_ _ _ _ _ _ _ _ - _ _ _ _ located in the Auxiliary Building (NF) and closed dampers 2VR04YA and 2VR05YA.
The dampers promptly closed shut causing Unit 2 VR to shutdown.
The EA still in the Ausiliary Building called the NSO who informed him that he had closed the dampers-on Unit 2 instead of opening the dampers on Unit 1.
The NSO told the EA to open the Unit 2 dampers and then go over to the Unit 1 isolation damper control panel and open the Unit I dampers. The EA opened the Unit 2 dampers and Unit 2 VR was restarted at 1953 hours0.0226 days <br />0.543 hours <br />0.00323 weeks <br />7.431165e-4 months <br />.
The EA then went over to the Unit 1 VR damper control panels IPL27JA and IPL27JS and opened dampers IVR04YA, IVR04YE, IVR05YA, and IVR05YB as originally instructed.
Unit 1 VR was then started up at approximately 2000 hours0.0231 days <br />0.556 hours <br />0.00331 weeks <br />7.61e-4 months <br />.
"The station initiated an investigation and determined the incident was the result of an error made by the Equipment Attendant when he closed the dampers on the wrong unit. The damper control panels were labeled correctly and did not contribute to the error made."
B.
NRC Discussions with Plant Staff I
As stated in Section A, the EA was supposed to open Unit 1 VR isolation dampers, per verbal instructions from the control room operators, and instead, went to Unit 2 and closed the Unit 2 VR isolation dampers. The dampers are operated locally from the auxiliary building, which is common to both units. The controls for both units are on the same elevation with Unit 2 controls located on the north end of the auxiliary building and Unit I controls on the south end. The outside of the damper control box is shown in Figure 2 and upon opening the door, the controls which the operator inadvertently closed are shown in Figure 11.
The operator manipulated two Unit 2 control room damper boxes which were approximately ten feet tpart. At the time of the incident, the boxes were labelled with the plastic identification tags shown in Figure 2.
No independent verification was performed as the control rocim operator caught the error from a warning light in the control room shortly after the incident occurred. No explanation into the cause was put forth 1
\\
K73isy=E r s t' s a s wa
_ _.:" E'" %' M t t h=' m
'M' #
h T % ',"j. M i. E d b,
.,....?
~gl5E.E??.?
~
~M
.:' ~ ; '" Q.7? ~
.1 surs sNet l s.'
} };
l
. '1 i
' p,>
a
]'.
s Ns t
.r_
MEMmMTMwasq 27
- -h*,
fy l $
F_ L
- b
? - += m 9 {7s _1 K Figure 11. LaSalle: LER 374-84-072 Inside of Dam:er Control Boy..
y
_ _ _ _ _. 1 other than the operator trust have had in his mind that he was. required to change position of the dagers, whereupon after entering the wrong unit, the change in position closed the dampers. The NRC team observed the color coding described above which was painted onto the floor of the f
auxiliary building at this level.
i 9-V.-
LER 374-85-012 Inadvertent RhK Shutdown Cooling Isolation (Wrong Component)
A.
"At 1103 hours0.0128 days <br />0.306 hours <br />0.00182 weeks <br />4.196915e-4 months <br /> on March 31, 1985, with LaSalle Unit 2 in Cold Shutdown, a Residual Heat Removal (RHR, 80) Shutdown Cooling isolation occurred during the. performance of surveillance procedure, LIS-RH-10 "LPCS/RHR Injection Line Integrity Mcnitor Calibration and Functional Test.
"A Group 6, Division 2, Prima y Containment Isolation (PCIS, JM) occurred on an apparent RHP suction high flow signal.
The RHR Suction Inboard Isolation Valve, 2E12-F009, automatically closed. All other Group 6, Division 2, isolation valves had been previously closed.
The operating RHR pump, 2E12-C002A, tripped as required on the closure of valve 2E12-F009.
"The Instrument Maintenance Technician performing surveillance LIS-RH-10 promptly notified the Unit Nuclear Station Operator (NS0) of an instrument valving error. Th s was identified as the cause of the j
isolation; RHR Shutdown Cooling piping integrity was verified, and the
{
RHR loop was returned to operation at 1107 hours0.0128 days <br />0.308 hours <br />0.00183 weeks <br />4.212135e-4 months <br /> on March 31, 1985.
J "The 2E12-F009 isolation occurred due to the valving out of instrum'ent 2E31-N0128.
Under LIS-RH-10 instrument 2E12-N029B was to be valved out but the Instrument Maintenance Technician performing the surveillance incorrectly traced the piping from instrument 2E12-N1298 and without looking at the tag on the instrument isolation valve, mistak'enly isolated a line to instrument 2E31-N0128.
"2E31-N0128 is a Barton 288A Differential Fressure Indicating Switch (DPIS) and is physically located next to 2E12-N0298, another Barton 288A DPIS.
2E31-N0128 senses differential pressure caused by flow through the RHR Shutdown Cooling suction piping.
When the low pressure sensing line to 2E31-N012B was mistakenly isolated, a lower than actual pressure was sensed at the low side of 2E31-N012B. With the high pressure side unchanged, a high differential pressure (high flow) condition was simulated and 2E31-N0128 actuated.
"The Instrument Maintenance Technician's incorrect tracing of the piping from instrument 2E12-N029B and failure to read the valve tag before moving the valve caused the March 31, 1985 occurrence.
"An investigation followed this occurrence and a meeting was set up between the Instrument Maintenance Technician involved, the Foreman in charge of the work, and the Assistant Superintendent of Maintenance.
The events of this occurrence were discussed and the need for attention to detail was stressed.
"The Instrument Maintenance Department will be trained on this occurrence with a focus on positive identification of instruments to be valved out. The completion of this training is being tracked by AIR 01-85-67055."
B.
NRC Discussions with Plant Staff The NRC team was unable to view the instrument lines involved in this event due to radiological considerations; however, the team was able to view a typical instrument rack located on the CRD system. Through discussions with the plant staff, the team learned that the instruments
)
on the rack sit close together; however, the area is not very crowded.
Although the instruments and the valves are labeled, the valve labeling is not used because they are not considered to be trustworthy. As a result, the instrument maintenance technicians are instructed to hand trace the lines, a process which is done frequently. The instrument
_ _ _ - _ _ _ _ _ \\
l maintenance technician, in this case, was relatively new with less than 1 year experience, and may have been nervous due to the high consequence I
of error (reactor trip).
As a result of the event, training now covers hand tracing.
l The team notes that although the LER attributes not looking at the instrument valve tag as a contributor, such practice was conrnon because the tags were considered unreliable.
Had the tags been mairtained and considered reliable by the technicians, they may have been used whereupon, a discrepancy between hand tracing and the valve tag most likely would cause a technician to check out the discrepancy before proceeding with his intended functions.
The plant staff indicated an intent to eventually upgrade instrument valve tags so that they are considered reliable, i
VI.
LER 374-85-20 Missed Service Water Sample (Wrong Component)
A.
The following infonnation was provided by the licensee in the LER:
" Unit 2 Service Water (XG) PRM (IL) was declared inoperable due to low sample flow at approximately 1830 hours0.0212 days <br />0.508 hours <br />0.00303 weeks <br />6.96315e-4 months <br /> on April 23, 1985. A sample was not obtained until 0620 hours0.00718 days <br />0.172 hours <br />0.00103 weeks <br />2.3591e-4 months <br />, April 24, 1985.
This was contrary to Technical Specification 3.3.7.10 which requires samples to be t.aken at least once every 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />.
At the time of this event, Unit 2 was in Mode 4 at 0% power.
CAUSE I
An investigation was conducted as a result of the missed sample from the
{
Unit 2 Service Water PRM.
The Unit 2 "A" RHR (BI) Service Water.PRM was mistakenly sampled instead of the Service Water PRM.
\\
"According to a Unit 2 log entry made at 1830 hours0.0212 days <br />0.508 hours <br />0.00303 weeks <br />6.96315e-4 months <br /> on April 23, 1985, the Unit 2 Service Water raonitor was declared inoperable due to a low 1
) 4 1
flow alarm with no adjustment-for higher flow. The Center Desk Log entry made at the same time-essentially said the same thing.
)
"According to a log entry made at 1830 hours0.0212 days <br />0.508 hours <br />0.00303 weeks <br />6.96315e-4 months <br /> on April 23, 1985, by a Rad Chem Technician (RCT), the Unit 2 Service Water PRM was declared INOP as of 1830 hours0.0212 days <br />0.508 hours <br />0.00303 weeks <br />6.96315e-4 months <br />.
i "The responsible Rad Chem foreman was coming on shift and was taking a turnover when the call came in.
The RCT who took the call showed the Foreman the log entry and was acknowledged. About 5 minutes later, the Foreman called the hot lab and spoke to the RCT on duty. Direction was given to the RCT to sample and analyze the Unit 2 "A" RHR Service Water PRM instead of the Unit 2 Service Water PRM. This was done, and the Unit 2 Service Water PRM sample was not taken.
"During subsequent shift turnover around 0605 hours0.007 days <br />0.168 hours <br />0.001 weeks <br />2.302025e-4 months <br /> on Apri1 ~ 24,1985, the mistake was discovered by the oncoming Rad Chem Foreman.
An RCT was imediately dispatched to take a Unit 2 service water sample.
This was unpleted at 0620 hours0.00718 days <br />0.172 hours <br />0.00103 weeks <br />2.3591e-4 months <br />.
"The cause of this missed sample could be attributed to failure to pay attention to detail.
Ineffective communication was also a contributing factor.
"The responsible Rad Chem Foreman was counseled by the Rad Chem Supervisor as to the Foreman's responsibilities when communicating a verbal message.
In addition, the Rad Chem Supervisor will issue a department memo providing additional guidance for effective verbal cocrnunication. This will be tracked by AIR 374-200-85-00066.
B.
NRC Discussions with Plant Staff Through discussions with the plant staff, it was learned that RHR service water sampling was routinely performed whereas service water
_ _ _ _ _ _ _. radiation monitoring was not. As a result, the rad chen foreman developed a mindset and transmitted instructions to mor.itor the RHR service water system.
ENCLOSURE 2 P
WRONG UNIT / WRONG TRAIN EVENTS AT 0. C. COOK 1.
LER-315-81-005 - Safety injection Valve Breaker (Wrong Unit) l The following event information was provided by the licensee during the site visit:
"An auxiliary eouipment operator with approximately 1 1/2 years of experience was dispatched by the Unit Two supervisor to hang a clearance on a series of motor operated valves for the Unit Two i
Safety Injection pump. The operator then mistakenly went to Unit One and opened the breakers for the specified valves. This error was innediately recognized by the Unit One control room supervisor and power was restored in approximately two minutes.
3 "This event occurred on the af ternoon shif t at about 10:30. The operator involved had not worked any overtime during the time surrounding the event. The area this event occurred in was the Auxiliary Building Basement. This area is a well lit, controlled (radiologically) and spacious area in which to work. At this time the Motor Control Centers involved were the same color on both units. The only labelling that existed at this time was black component tags for each breaker and these were not con-sistently available for all breakers.
I "As a result of this event an operations meno was sent to,ll a
shif ts concerning operator attentiveness. The intended function of the ECCS system was not inhibited by this event."
l
.m_u__._
i In discussions during the site visit, the licensee's staff indicated
.that the auxiliary equipment operator ( AEO)' had been interrupted during the task of hanging the clearance tags. The AE0 had gone to the Waste Disposal System (WDS) because an alarm had sounded there. He spent five to ten minutes l
handling the situation.there before returning to the tag hanging task. The breaker panel that he mistakenly went to (Unit 1) is. located near the WSD.
The AE0 realized his mistake as he walked to the Unit 2 area to perform his next task and realized that he had manipulated the breakers on Unit 1 rather than Unit 2.
It was the opinion of the licensee staff involved or familiar. with the event that the interruption of the flow of work was the major contributor to the error. Another possible contributor discussed include the reliance on magic mari labels at that time, many of which did not provide com;1ete unit or system designations.
2.
LER-315-83-009 - Spray Additive Tank Outlet Valve (Wrong Unit)
The following event information was provided by the licertsee during the site visit:
"During a routine tour an auxiliary equipment operator discovered th6t the Unit 1 spray additive tank outlet valve was sealed closed. Further investigation of this event disclosed that this' valve was inadvertently left closed following the performance of 1-OHP 4030.STP1007 (Unit One Containment Spray System Surveillance.) The AE0 who performed the independent verifica-tion initially looked at the wrong unit's valve. Realizing his mistake he checked the correct unit's seal number but failed to verify the Salve position.
1 This in turn caused the loss of both trains of Containment Spray to utilize the Spray Additive tank.
l
i "Several reasons can be traced to explain this event. The Spray additional tank room contains both unit one's and unit two's spray additive tank. The AE0 involved in the initial lineup had been working on other major surveillance procedures during the shift. The Reactor Operator involved was assigned to the other i
Unit and crossed units to work on the lineup and the general wort load of the shift was heavier than normal.
"As a result of this event, procedures were split up and re-scheduled to aid reducing work load, an operations memo on shift manpower was distributed, and labelling enhancements of the area were made."
In discussions during the site visit, the licensee's staff indicate-d that the procedures being run at the time of the event were time consu'irp and involved as many as five people to run one pump.
As indicated, that procedure was subsequently broken into smaller procedures. Other potential contributors to this event exist as well. The second verifier was to verify valve position and seal number. The independent verifier originally went to the Unit 2 additive tank rather than the U, nit 1 tank.
He became concerned that a mistake had been made when hanging the other tags. Then realizing his error, he then went to the Unit 1 tank and verified that the valves had the correct seal number. What he did not check was whether the valve was in the correct position (which it was not). The course of events that led to the valve not being in the correct position was described as follows. The valves in question are difficult to operate in that it is difficult to " break the seal", but when the seal i's broken, i
the valve stem suddenly spins free.
When the AE0 involved in the original
valve lineup manipulated the valve, the sudden free spin of the valve caused the AE0 to loose his balance somewhat and the seals in his pocket to spill on the floor.
He picked up the seals and placed the seal on the valve with-out completing the task of opening the valve.
Another potential contributor i
to the errors involved in this event is the independent verification process in place at the time. The licensee's staff indicated that at the time of the event it seemed that there was as much emphasis put on recording the seal number as checking the pnsition during verification tasks. Now the independent verification practice has been changed such that seal numbers are recorded on a weekly seal tour, rather than when the independent verification is beina perf ormed. This has shifted the emphasis back to verifying positifon.
i 3.
LER-315-83-048 - Containment Spray Heat Exchanger (Wrong Train)
The following event information was provided by the licensee during the site visit:
" Surveillance test procedure 2-OHP 4030.STP.007 requires that the i
inlet and outlet valves be tagged closed on the Containment Spray Heat Exchanger under test.
"Two operators (one AE0 and one RO) utilizing the independent verification system closed the inlet valve on the heat exchanger not under test. Since the outlet valve on the heat exchanger under test was closed, both trains were simultaneously inoperable.
"The Containment Spray Heat Exchangers are located on the 609' elevation via the Aux Building. The heat exchanger rooms are mirror inages; that is, both rooms have exactly the same components l
.______-.m_
____.___a
but are positioned opposite of each other.
"The valves are labelled and sealed in their respective pos'.tions.
"Several factors contributed to this event taking place:
- 1) The Auxiliary Equipment Operator involved was inexperienced j
(first time on the job) on the surveillance test.
- 2) The work load on the shift was heavy and the second verifica-tion was performed by an operator assigned to another job. He aided the AE0 in the valve manipulation inadvertently adding te the error.
- 3) A third verification was performed by the Unit Superviscr who discovered the error and corrected the problem.
The time that both heat exchangers were out of service was a:proxi-mately 15 minutes."
Discussions with the licensee's staff indicated that the AE0 who manipulated the wrong valve was tired, was on the first night of the 12:00 midnight to 8:00 a.m. shif t, and had not had much sleep the night before. The error occured at approximately 4:00 a.m.
In addition, the individuals involved were in a hurry and "way behind sch edul e."
The valve in question is inside containment and requires the use of a reach rod to manipulate it. The operator involved had traced out t!he reach rod but to the wrong valve. The operator manipulated the valve and then went to check whether the valve had moved. The position of the valve was hard to see from the platform and there was no rising stem on the valve. It did not seem to the operator that the valve had moved so he checked the sane valve on
____________m_
1 I
the other unit. When the valve from the other unit was conpared to the valve in question, the two valve positions were different. The operator assumed that the valve he had manipulated was open.
It did not occur to l
i the operator that he was on the wrong train.
Labeling available to the operator could have been improved. The reach rod had a tag on it, but a more easily readable large, magic marker label l
was printed on the wall. This label did not contain the train designation.
- l In talking to the operator who was doing the dual verification, he indicated
^
that he does not believe that he looked at the label because he had done the procedure before. He also was not there when the other operator, who had hurried on ahead, manipulated the valve.
Consequently, he had not provided a dual verification. The independent verification performed later did find the error.
4.
LER-315-84-014 - Safety injection Pumps (Wrong Train)
The following event infonnation was provided by the licensee during the l
site visit:
"An Auxiliary Equipment Operator while in the process of performing a surveillance test (1-OHP 4030.STP.005) on the South safety injec-tion pump inadvertently isolated the North safety injection pump.
Since the Reactor Operator had isolated the South pump in the control room, this rendered both safety injection pumps inoperable.
"The safety injection pumps are located in two adjoining rooms and are designated North and South.
At the time this surveillance was conducted, this area was contaminated and required anti-contami-nation clothing.
l
"The event was caused by personnel error primarily due t the breakdown in communication.
- 1) The Reactor Operator involved in performing this test was doing so for the first time.
Evidence of ineffective communication.
between the R0 and AE0 which included uncertainty as te w$igh pump was to be isolated led to the occurrence of this event.
- 2) Also the procedure utilized did not differentiate between trains except for brackets which were placed around South / West train com-ponents.
- 3) As a result this procedure has been divided into independent train procedures. This will aid in minimizing the chan:es for crossing trains during surveillance testing."
The event was discussed with plant personnel familiar with or involved I
in it. Potential contributors to the wrong-train error that were discussed are as follows.
It was a very hot night with inside temperatures estimated at about 100 degrees, and the AE0 had to wear anticontamination clothing.
It was 4:30 or 5:00 a.m., and the AE0 had not had a break. Because cf the temperature, valves were turning hard.
The control room operator had never run this procedure before, a very long and complex procedure. The AE0 had also never run this procedure. General practice is for the AE0 to complete his/her tour prior to running any procedures.
Before the AE0 had completed tour, the control room operator called the AE0 from the control room to begin manipulating valves as called for in the procedure. The procedure involved is referred to as a
" double star" procedure which means that the AE0 must carry a copy of,the procedure when working on that procedure. The AE0 did not have a copy of the procedure so returned to the control room to get a copy. The control roon coerator did not mark up the procedure, which entails customizing it for the train being tested, by crossing out the component identifiers for the other train.
i
1
- re control room operator had apparently been anxious about running this pro:edure.
He was trying to hurry the AE0 along, but the valves were turning hard which slowed the work. The AE0 oot to a point in the procedure that had not been marked-up to indicate the proper train.
Rather than call the control room operator, the AE0 assumed which valve was supposed to be manipulated and turned the wrong one (i.e. the North train). The AE0 had worked on the North train the night before, but the procedure being run the night of the event was for the South train. The error was discovered.
when the AE0 called the control room operator and told which valve had been operated.
The NRC staff agrees with the licensee that the major contributor to this event was poor communication. That poor communication was evidenced by the lack of discussion of the procedure before the procedure was initiated, the lack of properly marked-up procedures for the use of the AEO, the frustrated and harried atmosphere developed as a result of verbal communications between the AE0 and control room operator, and the AE0's decision not to call the control room operator to get clarification of the valve to be manipulated.
EL Event Report C/R 02-04-85-836 - Containment Spray (Wrong Unit)
The following event information was provided by the licensee during the site visit:
e f
I
"An Auxiliary Equipment Operator was assigned to place a clearance on the Unit 1 West Containment Spray pump.
The operator correctly placed the tags on the Control Room switches and the pump breaker in the switchgear room.
The operator then proceeded to the auxiliary building to hang the remainder of the tags.
Once in the hallway which separates the Unit I and Unit 2 containment spray pumps the operator went to the Unit 2 side via the Unit I side.
The operator then closed the manual valves via the remote operators which protrude into the hallway.
Then the operator went into the pump room to hang a tag on a motor operated valve handwheel.
Af ter the operator entered the room he noticed that the motor operated valve was open when it should have been closed since the valve was closed prior to hanging the tag on the control switch in the Control Room.
The operator not realizing his error, went to the Control Room informed the Unit Supervisor of a possible limit switch problem.
The Unit Supervisor asked the operator if he went to the right Unit.
The operator was sure he had and so it was decided to write a Job Order on the limit switch.
When the operator went to hang the Job Order, he discovered his error.
At this point, 15 minutes after the error, the Control Room was informed and theapproximately Unit 2 valves were opened.
"The hallway where the error occurred was well lighted and was not in a contamination area.
The entrance ways to the pump rooms are painted blue and orange, and the valves were clearly labelled.
" A contributing factor to this error is that the operator was working non-routine hours in that he was coming in at 0330 and working until 1130 as an extra person during what normally would have been a training week. "
,3 3
%s T
5
" 3.
g
[j
^
i T.
ENHANCEMENTS TO PRECLC E HUMAN ERROR TYPF. EVTyTS
~
,a l
l 16.
Creation of master clearance on' which all wodk needing to btZ
'done is collected, rather than issuing /hangirig individual
' i.
clearance / tags for each work item.
(Equal protection, saves time, reduces chance for arror.)
n 17.-
Increased mobility of persor.nel among departments $
particularly licensed o.perators transferred to Planning, QA, etc..for benefit of that. training / experience to others offorts.
l
.i.
1\\
18.
Use of Department Outage Coordinators and much more attehtien put on planning / coordinating tjobs to reduce confusion /e: rer.
i 19.
Use of experienced revision SROs or Operations Outage a
Coordinators.
Reduces outage load on Shif t Supervwars and improves coordination of outage activities with operating unit, Tech. Specs., etc.x 20.
(Future) Definition within Plant systems of "b intenance areas" to which job orders vic,11 be keyuS by computerized job order system and for which standard claarance doundaries will be computerized.
r 21.
Direction given "If it isn't lam 1ed, confirmaLTe on ar.; approved drawing:
'l l
31
't a) cannot operate it, put at leasd a temporary tag e.t (
b) cannot use for isolati n poirt" i
s I
i
'(
k I..
( qo lh
's
{'
I i
6
)-
i s
f*
i
+
1 0
t
/*
1 g
. i.
I l
k 1
8:
f k
t
'A.
1
"Tu N4 t.
- ":vided by D. C. Cook ENHANCEMENTS TO PRECLUDE HUMAN ERRCR TYPE EVENTS e
t
(
d i
4'.?
1.
Colet coding the Plant units.
(Unit 1 orange, Unit 2 blue.)
' Carried out through tags, labels, job crders, considering for 4
procedure.
2.
Independent Verification Program.
3.
Electrical / wiring drawing enhancement.
4.,
Revision to piping flow diagrams being redrawn for clarity.
Creating many accuratescontrolled drawings for where none-existed.
(Fire protection, Plant air, turbine fluid control, etc.)
+2/p Frocedures belnd identified by color (Unit 2 cover sheet yellow).
Considering changing cover sheet color to correspond.to unit color.
6.
Job Order revised wi th a
'2' superimposed on the Job Order for Unit 2 jobs.
\\
7.
Labeling and walkdown program, Operator aids program.
8.
Full page change sheets instead of pen and ink.
9.
Reducing complex procedures into several more distinct and applicable ones.
10.
Personnel Error Forms for Condition Reperts.
Opportunity to discover contributing factors such as human factors contribution.
11.
Opening of Condition Report process to identification of N
drawing discrepancies.
Statistics available.
I 12 4 Prohibition.of shift people assigned to one unit from working 4 on other unit except under prescribed conditions (certain i tm$rs and independent verification are cross unit).
4 3 13.
Issue of Unit One and Unit Two procedures versus 12.
14,c Facility data bnse ef fort.
15.
Computerized Clearance Permit Program and creation of.
r, computerized
- standard clearance boundaries for. standard
' repeat type jobs.
(Saves time and decreases opportunity for error. )
r
- h _ _ _ _ _ _ __
m it a-
-t-a-
D;I5:E'i i
\\,,
DISTRIBUTION:
June 11,1985 TOSB R/F MVirgilio GTrager, AEOD 6
DPersinko, DHFS FIM0RA'tDU'i FOR:
G. E. Edison, Chief Technical and Operations Support Branch Planning and Program Analysis Staff, NRR W. Regan, Acting Chief Human Factors Engineering Branch Division of Human Factors Safety, NRR H. R. Booher, Chief Licensee Qualifications Branch Division of Human factors Safety, NRR i
K. Black, Chief Nonreactor Assessment Staff i
Office for Analysis and Evaluation of Operational Data FRDP1:
M. J. Virgilio, Technical Assistant Technical and Operations Support Branch Planning and Program Ar.alysis Staff, NRR 1
SUBJECT:
TRIP REPORT FOR DRESDEN SITE VISIT
REFERENCE:
Memorandum from M. Virgilio to G. E. Edison dated April 17, 1985 Purpose The purpose of this memorandum is to document the activities and findings of an NRC visit to the Dresden site on May 8 and 9,1985.
Members of the NRC group for this visit included G. Trager (AE0D), A. Ramey-Smith (DHFS),
D. Persinko (DHFS), and M. Virgilio (PPAS). This site visit was conducted as a part of the short-term effort to determine whether simple, low cost fr.provements can be identified and it:plemented to reduce the frequency of wrong unit / wrong train events occurring at nuclear power reactor facilities.
Additional background information on this subject and a detailed plan of action for the short-term effort are presented in the referenced memorandu:2.
General Information The Dresden site is located nine miles east of Morris, Illinois on the Kankakee River.
On the site are three rseactors operated by Commonwealth Edison Company, Dresden Units 1, 2 and 3.
Dresden Unit I has been permanently shut down.
Dresden Units 2 and 3 are both General Electric NSSS BWR-3 with Mark I containments, each having a maximum dependable capacity (Net) of 772 MW'e.
The architect / engineer for both units was Sargent and Lundy. The construction for both units was United Engineers.
Dresden Unit 2 was first niaced intn e nmo recal nnoratinn in.lun n 0, lo70 nroeden fini t "I wac fi re t omcq..........pl a ce d into comaerc' al operation
)n November 16, 1971.
,...,.m.... y.p..............
g.LV.l.p.(.y......
x
.m xs.
m.m a
--_m
n, 1,.
F
'c.
m I Site Visit Agenda The site visit discussions and tours were centered emand three reported wrong unit / wrong train events that occurred at Dresden in 334 and 1985.
The LER numbers for these events are 237-84-013, 249-85-005 end 237-84-012.
During the site visit the NRC team inspected the location o' the reported wrong unit /
wrong train events, superficially reviewed the procedares involved and discussed the events with the individuals involved, ineir supervisors and the Dresden Station Production Superintendent. provides a sequence of events for each of the LERs, a sumary the licensee's 'ollow-up actions for each event, observations from the NRC site visit tour and a sumary of the discussion of each event with the individuals involve:.
General Observations Dresden plant personnel stated that after several yea-s of multi-unit operation, the station management perceived a need tc irorove the labeling and identification systems.
Accordingly, the floors surr:unding components for Unit 2 were painted yellow and the floors surrounding Unit 3 components were painted blue. Sound powered communication systems fc-the Units were separated.
In addition, large plastic labels were installed on most components.
l At Dresden two administrative programs govern the evaluation of hunan errors; the PRO (professionalism) Investigation Program and tte Potential Significant Event Investigation Program.
Implementation of each ;rcgram is governed by procedures that include provisions for determining the r>eed for corrective actions such as hardware, procedural, and training im;rovements.
Two tag-out prograns are utilized at Dresden to contr:1 breaker, valve and switch positions.
In addition, monthly position / stat:;s checks are performed on certain essential support system components. Some automated system and component status acnitoring indication is provided in the control room, however, the plant design pre-dates the NRC guidelines on status monitoring contained in Reg. Guide 1.47, " Bypassed and Inoperable Status Indication for Nuclear Power Plant Safety Systems".
Training for commonwealth Edison employees located at the site varies L
depending on their assigned responsibilities. All nor-licensed operators and other station personnel responsible for manipulating ralves, switches and breakers in performance of their assigned duties receive six months of training which includes system and component location, identification and labeling systems, proper methods of implementing tag-out procedures and on-the-job training with other Dresden personnel.
Isolation of systems and components for maintenance is performed by these Operations Department l
l personnel and confirmed by the maintenance personnel (commonwealth Edison Employees) before work commences.
This is the case even when work is performed by contractor employees.
"c ' >
om>
nnamo ma a n.ar.-_m-._
^ - ^ - - ^
4
- v., V 3
l Closecut Meeting The site visit was concluded by a meeting with the Production Superintendent and a Training Supervisor.
Both individuals were asked what, in their opinion, were the overall causes leading to the wrong unit / wrong train events.
In the Training Supervisor's opinion labeling was most important and he felt that additional and improved labeling and identification systems would be helpful.
In the Production Superintendent's opinion additional communication between shift personnel prior to performing plant evolutions (e.g., rode change, test, tag-out) was the most important factor and that improved consnunications would reduce the number of wrong unit / wrong train events.
At the closeout meeting the NRC team expressed its appreciation to the Dresden site staff, in particular to the Production Superintendent whose cooperation in coordinating the tour and discussions made the site visit efficient, informative and productive.
Oridna(Sign,,$d,byj H. J. Virgilio, Technical Assistant Technical and Operations Support Branch Planning and Program Analysis Staff, MR
Enclosure:
As stated cc:
J. Funches T. Ippolito D. Beckham R. Gilbert f*'%
emer M. T.O..S.B..:. P.P..A. S.....
...A. E.0. 0. 6.......
.....H..F.E..D. M.....
....S E.P....
~=4 s.v.tr.g.i.1.ff.
GT
...h, r,a g,,,,,,,
, A,R a,m e,S,m,t t,h,,,, p,P,e r,s,1,n k,9,,,,,,,,,,,,,,,,,,,,,,
our) b
..........Y..../.8. 5...
. 6../.. 7......./. 8 5.....
... 8../.. 5......./. 8 5
- g. /..,3..../.8. 5....
Review of Dresden Units 2 and 3 Wrong Unit / Wrong Train Events The Dresden site visit discussions and tour were centered around three reported wrong unit / wrong train events that occurred'in 1984 and 1985. The purpose of this paper is to provide a brief discussion of each event, a summary of the licensee's follow-up actions for each event and observations from the NRC site visit tour and discussions of each event with the individuals involved.
l LER 237-84-013 Description of the Event The following event description was provided by the licensee to the NRC group during the site visit.
"On the af ternoon shift of July 22,1984, Unit 2 was steady state at 747 We and Ur.it 3 was being prepared for startup following a scram earlier that day.
A newly promoted Equipment Attendant (EA) was called in as extra to help with the Unit 3 startup.
The EA had completed preparing a reactor feed pump (RFP) for operation and called the unit Nuclear Station Operator (N50) to report the RFP ready for service. The NSO told him to go to the condensate pump room and prepare a condensate / condensate booster pump for service.
At this time, the NSO also told the EA that on his way to the condensate pump room he wanted him to open the turbine electrohydraulic control (EHC) system valve FV-1.
The EA went to the Unit 2 instead of the Unit 3 EHC system and opened the FV-1 valve.
This resulted in a low EHC system pressure, causing a turbine trip and reactor scram on Unit 2 from 747 We."
Licensee's Investigation E
At Dresden most events that involve human error are evaluated to determine the cause and appropriate corrective action.
At Dresden these evaluations are called PRO (professionalism) investigations.
From the evaluation of the event tha licensee determined that there were several contributors to the error.
The findings, conclusions and recommendations of the PRO investigation are as follows:
g j 06/03/85 1
TRIP REPORT FOR ORESDEN
)
"The EA began employment on January 5,1984.
He had not participated previously in any of the evolutions he was asked to perform.
His training included five weeks at the Production Training Center and about seven weeks of OJT.
The balance of his time was spent in on-site classroom training and associated plant During his entire OJT training period, Unit 2 was operating tours.
and Unit 3 was in a maintenance outage.
Shift supervision had told the newly promoted EA at the beginning of the shif t that if he encountered any problems out in the plant while performing his duties, he should call and ask for help.
The EA had asked for and received help by phone from the NSO in order to prepare a RFP for service.
He said he felt hurried in receiving orders to prepare the B
condensate / condensate booster pump and open the FV-1 valve.
During his OJT he had been shown the FV-1 valve on Unit 2 and had a mental image of that valve. When he erroneously went to the Unit 2 EHC unit, which is adjacent to tne Unit 3 EHC unit, he verified the FV-1 tag notation, which was in large letters and overlooked the smaller
" Unit 2".
This was his first shift working as an EA assigned to a unit.
He was called in as an extra EA for the start up of Unit 3 and did not know the people en the shift and they did not know him.
He clearly understood that he was assigned to Unit 3 and all his work that shif t would normally be associated with Unit 3.
PERSONNEL INVOLVED Equipment Attendant -
Nuclear Station Operator -
CONCLUSIONS 1.
There was no evidence of careless disregard for rules or requirements.
2.
The newly promoted EA received minimal shift management involvement or supervision.
3.
The root cause was assigning a person to do a task who was inexperienced.
4.
Human factor conditions regarding color coding, valve tagging and lighting were satisfactory.
RECOMMENDATIONS b
1.
Newly promoted EAs should receive close shif t supervision, especially when they are assigned a shift activity for the first time..
2.
Detailed briefings should be given before placing new EAs into stressful situations to ensure both proper understanding of assignment and method of accomplishment.
E 06/03/85 2
TRIP REPORT FOR DRESDEN
j 1
3.
0JT should be expanded for an interim period of about 2 weeks in which the newly promoted EA works with and is backed up by an experienced EA prior to assuming the job.
4.
OJT should be conducted to ensure that personnel receite adequate training on both units (most training is now done on Unit 2) to avoid a " mind set" or preference regarding only one unit."
l NRC Tour Observations a!
As a part of the site visit tour the NRC team looked at the EHC System Valves FV-1 for Unit 2 and Unit 3.
The majority of the electrical and mechanical com-porents for the Unit 2 EHC System are part of an assembly measuring approximately 4 ft (w) x 6 ft (1) x 6 ft (h).
FV-1 is located in the front center of the assembly approximately 5 ft from the floor.
The EHC system for Units 2 and 3 are located on either side of a passageway, separated by approximately 12 ft.
The base and floor surrounding the EHC assembly for Unit 2 is painted yellow, the base and floor surrounding the EHC assembly for Unit 3 is painted blue.
FV-1 is marked with a plastic tag measuring approximately 3 in x 5 in, with FV-1 in large letters and the unit number (Unit 2 or Unit 3) in smaller letters.
~
NRC Discussions With Plant Staff Ouring the site visit the NRC team met with the NSO and EA involved in the EHC system event discussed above. When asked his opinion on the cause of the event the EA attributed his mistake to the training program's focus on Unit 2, and the fact that this was his first day on the job independently performing assigned tasks.
Other Observations E
Based on a review of the event, discussions with the individuals involved and inspection of the location where the event occurred the NRC team made several additional observations.
Although these observations may not be directly related to this event they appear to be germane to the wrong unit / wrong train l
issue based on reviews of other wrong unit / wrong train events that have been reported to the NRC.
L !
06/03/85 3
TRIP REPORT FOR DRESDEN
.______-______-a
-1.
EA's as well as other shift personnel rotate through differert assignments as well as different shifts.
At Dresden there are six crews rctating through three working shif t (graveyard, day and evening), training and time off.
Within this rotation EA's serve in seven different assignments rotating at 3 day intervals (Unit 2 EA, Unit 3 EA, radwaste EA, etc.).
During any three day rotational assignment an EA may be askee to perform work outside his assigned responsibilities for that three day' interval.
'2.
Although the floor surrounding a plant-unique component is cc'or coded, there is no color coding scheme for labels, systems, or compo9ents.
Some components share identification numbers, FV-1 for example.
A'though the label for FV-1 includes the Unit 2 or Unit 3 identification or, the label this is not the case for all labeled components in the plant.
Tags for many Unit ~2 and Unit 3 components are identical (i.e. have the same identification number, same color).
Even though the unit des'gmation was on the EHC valve tags, the EA stated that he saw FV-3, and opened the i
valve.
3.
There is no formal label maintenance program.
When missing or coscured labels are reported, repair orders are initiated by station personnel.
4.
Unless a caution or out-of-service tagout is involved, an EA's instructions are provided verbally. An EA may be directed to perform tasks by different shif t personnel (e.g. shif t foreman, NSO, shif t engineer), and priorities assigned to the different tasks are subject to change.
5.
There did not appear to be a formal program for determining schedules and assigning responsibility for implementation, although all of the PRO in-1 vestigation's recommendations had been implemented.
Ej l
6.
Certain labels and tags include a functional description of the component (e.g. bypass valve, discharge throttle valve).
Others provide only a component identification number.
There did not toppear to be a logical scheme to explain the difference.
K 06/03/85 4
TRIP REPORT FCS DRESDEN 1
7.
Curbs surrounding components on both Units 2 and 3 were painted yellow to stand out as personnel hazards.
Yellow is also the color painted on the floor to designate Unit 2 conponents.
LER 249-85-005 l
l Description of the Event l
The following event description was developed from the IE follow-up investiga-tion report and discussions with the licensee during the site visit.
Unit 2 was in the refuel mode with fuel in the vessel and licensee personnel were preparing to conduct DOS 6600-5 " Bus Undervoltage and ECCS Integrated Functional Test for 2(3) Diesel Generator" on the Unit 2 emergency diesel generator.
Unit 3 was at or near full power and the Unit 2/3 (swing) emergency diesel generator was out of service for routine maintenance.
Several shift foremen, special shift foremen, nuclear system operators, equipment attendants, and equipment operators had been involved in preparation for the DOS 6600-5 test which had involved making up lengthy outage lists and Caution tags to be placed prior to the start of the test.
On February 16, 1985, the outage lists had been made up from the lists in the procedure.
An equipment attendant (B-man) was assigned to make out the Caution tags per the outage lists (two outages of about 85 tags each).
Since he was accustomed to seeing a valve or switch position listed on the tags, he questioned the Unit 2 nuclear systems operator (NS0) and then proceeded to a different assignment.
The NSO contacted a shift foreman and inquired about the test E
switch positions.
The reply was that the switches should be opened and the NSO reflected this on the Caution tags.
This is one of several errors that led to the event because the Caution tags were to be only placed on the switches to enhance later identification.
Placing of the Caution tags started at about 8:00 am on February 16, 1985, by an equipment operator (EO) (non-licensed operator) who noted that the locations 06/03/85 5
TRIP REPORT FOR ORESDEN
I l
of the test switches were r.et listed on the Caution tags.
He :0ntacted a special shif t foreman for the test, who provided a copy of the procedure i!
DOS 6600-5.
When the E0 opened the procedure, which contained portions common to both Units 2 and 3 as well as specific to Unit 2 or 3, and natched the switch numbers on the Caution tags with those in the procedure, he wrcte the location of the switches on the tags and failed to notice that the locations were from l
the Unit 3 list rather than the Unit 2 list.
Almost all comporents have identi-cal identification numbers, rather than identification numbers ;nique to Dresden Units 2 and 3.
The identification of wrong unit locati:n was another si nificant contributor to the events and resulted in the Cauti:n tags being 0
placed on Unit 3 rather than Unit 2.
Wh:n the E0 started placing the Caution tags and opening the sVtches he was met in the plant by the shif t overview superintendent (505).
Tre 505 noted
~
that the tags were being placed on Unit 3 rather than Unit 2 an: questioned the E0.
They did not have a copy of the procedure with them an: through their discussion agreed that hanging tags on Unit 3 was probably due t: the electrical cross-tie between the Unit 2 (Bus 24-1) and Unit 3 (Bus 34-1) diesel generator buses and concluded it was to prevent a perturbation on the Unit 3 side.
They agr:ed to continue hanging the tags and followup on the questior later.
The 50S's use of improperly completed tags to verify correct placeme,t was the third contributor to the event.
Th3 hanging of Caution tags and opening test switches continued antil the sixth and seventh tags were placed and the switches opened.
This caused control room alarm 903-8, D-4, "4kV Bus 34-1 Voltage degraded" te annunciate.
The special shift foreman (SSF), shif t engineer (SE), and shift control room engineer / shift technical advisor (SCRE/STA) recognized that by procedure, the loads from Bus 34-1 would shed in five minutes if voltage was not restored.
E The SSF, suspecting there may have been a problem with the outage placement, ran from the control room to the second floor of the reactor building where he contacted the E0 and the SOS who immediately closed the last two switches.
This cleared the alarm condition before the five minute time deley timed out.
A prompt review resulted in the removal of the Caution tags that nad been placed and returning the test switches to the closed position.
g 06/03/85 6
TRIP REPDiT FOR DRESDEN
i The errors were identified, the Caution tags corrected and properly placed and the test was conducted as planned on the Unit 2 diesel generator after the Unit 2/3 diesel generator was returned to service.
Licensee's Investigation At the time of the site visit the licensee had not yet completed its own eva:ua-tion of the event.
A preliminary report on the potential significant event was prepared and issued on February 20, 1985.
This report included a descriptio(
a of the event, a discussion of its safety significance and recommended actions.
The recommendations of this preliminary report are as follows:
l
" Corrective Actions i
The preliminary Station review of the event recommends the following actions:
1.
Separate DOS 6600-5 and 6600-6 into separate procedures for Unit 3 and Unit 2.
2.
Place in the prerequisite of DOS 6600-5 and DOS 6600-6 a sign-off that the test cannot be performed until all three i
diesel generators are operable.
i 3.
Schedule manpower such that at most only two specific personnel will be in charge of the test to ensure better continuity and communication.
4.
Pre printed caution card checklists will be included in the ECCS undervoltage test procedure to ensure that an accurate description of the undervoltage knife switches is included.
5.
An immediate review of this event will be held with all Operating personnel.
i 6.
An Operating Order will be issued to require that all personnel l
involved with a test or complex plan evaluation will discuss lE; the activity in detail " face to face" before proceeding with the activity, a
7.
A Pro investigation will be initiated on this event.
8.
Actions of the personnel involved will be reviewed directly by the Station Superintendent."
E 06/03/85 7
TRIP REPORT FOR DRESDEh
NRC Tour Observations During the site visit the NRC team members were shown procedures, DOS 6600-5,
{
" Bus Undervoltage and ECCS Integrated Functional Test for 2(3) Diesel Generator" and DAP 3-8, " Caution Cards".
During the tour the NRC review team looked at the electrical busses and switches caution-tagged during the event.
(
l From a superficial review of DOS 6600-5 we observed that the pages for the as j Unit 2 and Unit 3 system alignments were uniquely identified by small letters at the top of each page.
However, this uniqueness appeared to be too subtle considering most of the component numbers for the Diesel generators and associated equipment were the same.
NRC Discussions With Plant Staff j
During the site visit the NRC met with the 505 and SSF involved in the diesel generator tag-out event discussed above.
When asked their opinion on the cause of the event the individuals attributed their mistakes to miscommunication,.
lack of attention to details and procedural inadequacies.
1 Other Observations Based on a review of the event, discussions with the individuals involved and inspection of the procedures and the location where the event occurred the NRC team made several additional observations.
Although some of these observations m:y not be directly related to this event they appear to be germane to the wrong unit / wrong train issue based on reviews of other wrong unit / wrong train events that have been reported to the NRC.
EI 1.
Although one of the proposed corrective actions is to separate this test procedure involved into plant unique procedures for Unit 2 and Unit 3, other plant procedures remain combined two unit procedures.
2.
At Dresden administrative procedures are used to isolate and disable safety-related components.
1 06/03/85 8
TRIP REPORT FOR ORESDEN I
l l
Caution cards are utilized for a number of purposes including estatiishing plant conditions for tests.
Out-of-service cards are used control plant conditions for maintenance. Although the out-of-service card procecure (DAP 3-5) specifically requires independent physical verification by a competent, qualified individual when removing or restoring equipment to service, the caution card procedure DAP 3-8 does not.
1 3.
A limited number of station procedures and logs are unit specifc and color coded.
Unit specific colcr coding of additional procedures is being considered by Dresden Station management.
4.
As in LER 237-84-013, components in Units 2 and 3 contain the same coe-ponent designation.
In this case, the buses were specifically identified by Unit (i.e., 2253 for Unit 3 and 2252 for Unit 2), however, switen numbers on the buses were identical on both Units.
LER 237-84-012 Description of the Event The following event description was developed from discussions with the licensee during the site visit.
On July 9,1984, with Unit 2 operatir>g at 98 percent power, calibration of the main steam line log radiation monitoring system was in progress.
The test equipment was connected to conduct a calibration of the C main steam line radia-tion monitor associated with RPS Channel A.
However, the Instrument Mechanic mistakenly turned the RPS Channel B low setpoint bypass switch to the off posi-tion.
Since hydrogen additon was in progress, a low setpoint trip occurred in E
RPS Channel B.
The Instrument Mechanic was informed of the RPS Channel B trip by his assistant.
The Instrument Mechanic then attempted to lower the test i
i signal on C main steam line radiation monitor and remove his test equipment to
}
avoid a Channel A trip and reactor scram.
Before the test signal could be lowered and test equipment removed, a voltage spike tripped the C main stees line which in turn tripped RPS Channel A and initiated a reactor trip.
l 06/03/85 9
TRIP REPORT FOR DRESDEN
l k
Licensee's Investigation The licensee's investigation concluded that the Instrument Mechanic was at I
I fault by f ailing to follow the test procedure.
The Instrument Mechanic was
. disciplined by receiving one day off presumably without pay.
NRC Tour Observations During the site visit the NRC team members were shown the surveillance test a
procedure.
During the tour the NRC review team visited the control room and examined the panel containing the test jacks and switches manipulated by the Instrument Mechanic during the surveillance test of the main steam line radiation monitors.
From a superficial review of the surveillance procedure, we observed that one procedure was utilized for three test conditions, Unit 2 with hydrogen addition on, Unit 2 with hydrogen addition off and Unit 3 (no hydrogen addition).
From our examination of the control room panel we noted an apparent lack of human factors planning for the surveillance testing and calibration of the main steam line radiation monitors.
As depicted in Figure 1, the instrument channel drawers were located above and to the side of bypass switches.
The drawers for instrument channels A and C (feeding into the A RPS logic) were located above the B and D instrument channel drawers (feeding the B RPS logic).
4 K
L 06/03/85 10 TRIP REPORT FOR ORESDEN
RAD Monitor A RAD M:nitor C RAD Monitor B RAD Mo titor D OFF 0FF ON OFF 0FF ON C
Q Channel A Channel 8 Bypass Switch Bypass Switch FIGURE 1 NRC Discussions With Plant Staff During the site visit the NRC net with the Instrument Mechanic responsible for the plant trip discussed above.
When asked his opinion on the cause of the event the mechanic attributed the mistake to confusion, a defective procedure (that requires the user to flip back and forth between various secticms and data sheets), and his lack of familiarity with the procedure (had only been performed by this individual once before).
In addition to procedural changes, the mechanic recommended that covers be placed over the switches to prevent inadvertent operation.
g Other Observations Based on a review of the event, discussions with the individuals involved and inspection of the procedures and the location where the event occurred the NRC team made several additional observations.
Although some of the obser vations L
cay not be directly related to this event they appear to be germane to the 06/03/85 11 TRIP REPORT FOR DRESDEN
wrong unit / wrong train issue based on re.'iews of other wrong unit / wrong train events that have been reported to the NRC.
1.
There was no PRO investigation for this event.
Because of previous humar.
error events involving this individual and the more obvious circumstances of this event it was concluded that the Instrument Mechanic failed to follow procedures.
Objectivity of the review appeared limited.
2.
Commonwealth Edison is beginning to become involved in the INP0 accredita-tion program.
It is anticipated that participation in the program will result in definitive qualifications and improved training and procedures for Instrument Mechanics.
3.
Although the instrument mechanic statec that the procedure was confusing,-
he apparently was following the procedure correctly because he realized that an error occurred when he was infctmed of the Channel B trip instead of Channel A.
l 4.
This calibration had only been perfonned a total of six times since revi-sion of the procedure to account for hydrogen addition.
When asked about training given the recent revision of the procedure, we were informed that someone who has performed this calibration before walks through the proce-dure with someone who is doing it for the first time.
However, the expe-rienced person is not always present when this inexperienced person is doing the calibration.
l 06/03/85 12 TRIP REPORT FOR DRESDEN l
-2 on March 4, 19E5.
While assigned to unit 1, they had performed the delithification procedure using the unit I deborating demineraliter.
On March 11, they cade the attempt, for the first time since being assigned to uni: 1, to perform the similar procedure cc. unit 3.
Operations manacement had been consulted about whether the opposite valve numbering should be changed.
They decided to make no physical changes to the valve numbering because the status of the deborating demineralizers would change in the future, and that tre procedures were adequate.
The OSRG concluded this event was not a recurring problen and was not reportable.
This event was rect discussed during the site visit.
i 1
I
[
,'o.,,
UNITED sTA TEs y
3 g.
NUCLEAR REGULATORY COMMISSION y
wAssiwcios. o. c. rosss I.,a...+/
~
SEP 2 31985 MEMORA'4DUM FOR: Harold R. Bocher, Chief Licen.see Qualifications Branch Division of Human Factors Safety, NRR William H. Regan, Jr., Acting Chief Human factors Engineering Branch Division of Human Factors Safety, NRR Kathleen M. Black, Chief Nonreactor Assessment Staff Office for Analysis and Evaluation of Operational Data James E. Lyons. Acting Chief Technical and Operations Support Branch Planning and Program Analysis Staff, NRR b,( Maintenance and Surveillance Section THRU:
Gregory C. Cwalina, Section Leader Licensee Qualifications Branch Division of Human Factors Safety, NRR FROM:
Drew Persinko, Maintenance and Surveillance Engineer Maintenance and Surveillance Section Licensee Qualifications Branch
~
Division of Human Factors Safety, NRR
SUBJECT:
TRIP REPORT FOR SURRY AND NORTH ANNA SITE VISIT REC # DING
^
WRONG UNIT / WRONG TRAIN.
This memorandum'docum6nts the' activities and findings of an NRC staff visit.
to the Surry and North Anna sites on June 17-20, 1985. Members of the NRC team for this visit included A. Ramey-Smith (DHFS). E. Trager (AE00) and D. Persinko (DHFS).. The site visit was conducted as part of the short-tem effort to detemine whether simple, low cost improvements can be identified and implemented to reduce the frequency of wrong unit / wrong train events -
occurring at nuclear power reactor facilities. Upon completion of all site visits, a compilation of factors contributing to the events will be performed and a report issued which discusses causes and recommendations.,Long tern resolution will be accomplished as part of the Maintenance and Surveillar,ce l
Program Plan being conducted by the Maintenance and Surveillance Section of LQB/DHFS.
General Information
~
The Surry site is located 17 miles northwest of Newport News, Virginia, cei the James River.
There are two reactors, Surry 1 and Surry 2, located at the site.
Surry I has a maximum dependable capacity (net) of 781 MWe and was ji.
S g# N6WJ f(j'
,vy i
.. placed into comercial operation on December 22, 1972. Surry 2 has a maximum dependable capacity (net) of 775 MWe and was placed into comercial operation on May 1, 1973. Both units are Westinghouse PWRs and the architect / engineer for both units was Stone and Webster.
The North Anna site is located 40 miles northwest of Richmond, Virginia, on l
which are two reactors, North Anna 1 and North Anna 2.
Both units are 1
Westinghouse PWRs and the architect / engineer for both units was Stone and Webster. North Anna 1 has a maximum dependable capacity (net) of 890 MWe and was placed into commercial operation on June 6, 1978. North Anna 2 has a maximum dependable capacity (net) of 893 MWe and went into commercial operation on, December 14, 1980.
The licensee for both sites is Virginia Electric Power Company (VEPCo).
1 Site Visit Agenda
~-
The discussions and in-plant observations were centered around four wrong unit / wrong train events that occurred at Surry between 1981 and 1983, and two that occurred at North Anna between 1982 and 1985. The LER numbers for these events at Surry are 81-001,82-072, 83-033, and 83-051 and at North Anna are 82-022 and 85-006. During both site visits, the NRC team inspected the locations of the reported wrong unit / wrong train events to the extent possible, and discussed the events with the Huran Performance Evaluation System (HPES) coordinators at each site as well as many of the individuals directly involved with the event. At Surry, the licensee's corporate HPES coordinator also participated in the discussions while at North Anna, a nuclear so?cialist participated. At both sites, we brirefly spoke with the assi:, tant station manager.
Enclosures 1 and 2 provide a sequence of events for each of the LERs, a sumary of the licensee's conclusions regarding the event, NRC staff observations and a sunmary of the discussions concerning each event.
General Observations A.
Human Performance Evaluation System (HPES)
Both Surry and North Anna are participating in the HPES initiated by INP0. A large part of the discussions at Surry and North Anna focused on the HPES implementation, which has two main thrusts:
- 1) to investigate incorrect human actions, and 2) to prevent further incorrect human actions from occurring. Upon completion of an investigation, recommended corrective actions are presented to management for review and approval. To the extent possible', the system is non-punitive and relies on confidentiality. The HPES is-a d*
forward-looking program and only a few retroactive investigations have been conducted; no HPES investigations have been conducted on the LERs reviewed during this site visit. All completed reports are,provided to INPO.
The system has been in place at VEPCo for approximately one year but has been functional only since October 1984.
Prior to the HPES, no formal investigation of inappropriate human actions was conducted outside of normal LER investigations.
Although Surry and North Anna are both owned and operated by VEPCo each I
site implements the HPES differently. At Surry, one person is assigned as HPES coordinator and devotes up to 75% of his time to the HPES. This person conducts the investigations, writes the reports, and promotes the HPES to plant personnel. He is highly visible as the focal point for the HPES. At North Anr.a. one person is also assigned as HPES coordinator; however, five STAS in addition to the coordir.ater conduct investigations and write reports..As a result of the differing approaches to implementation, it appears that the HPES at North Anna may i
be more ingrained into the usual course of business; however, near
~
misses may not be as readily reported due to a perceived decrease in confidentiality because of the additional people involved in an investigation. Feedback sessions are held to infonn plant personnel about information learned from the HPES investigations. The feedback is supplied mainly to operators and instrument technicians, and in some instances, to mechanical, electrical and HP technicians.
As a result of the HPES. both sites have installed distinctive, identically designed signs which are color coded by i: nit. As shown in Figure 1, the signs are approximately I foot x 1 foot and warr4 plant personnel that equipment in a given area is associated with a specific ~
~
unit.
Y y[n..
[w.q['%"%
^
... d 6).. u e.
5..
y ;.
d-l, r
-! f
- """""*'*j
' ~'
e UNIT 2
.. ~.- vlt
)'k,.u.Nj k 5 i(
es. -
m9
,,f' +i,
,,j 77l"f,9 y g a gnh
- g r m
g
.j
'?
IEEEE"d"""
'l' ',A
,/
d
'1
." T 5W.Q.f..
i
'*2.9
. L.e o.,., l : _.
f g,., )
.:+.
Figure 1
B.
Observations at Scrry Surry utilizes a color-coding scheme of green for Unit I and yellow for Unit 2.
This color-coding scheme is evident on the unit designation signs described above and on other plexiglas signs located throughout both units which state the building, the unit number, the elevation, and etc.)ype of safety protection required (e.g., earplugs, safety glasses, the t The signs are approximately 1xli feet and are shown in Figure 2.
'Y
. '... 1
^
w',.m
~...
t
,,.N.
8
'O-f> ~;. ~ E ~ 5 u.
... s. n,s.,?
g@ ", h i
~
,y 34i
' F.'
- 1. H :
Figure 2 Station operating procedures are also color coded green for Unit 1, yellow for Unit 2 and blue if the procedure applies to both units.
W/WT events havs occurr5d at'Surrf which were non-reportable and were investigated by the HPES. The HPES coordinator at Surry agreed to provide the staff an edited description of these events.
C.
Observations at North Anna Although the unit identification signs described in Section A are in place, no other unit designation signs such as the plexiglas signs at Surry are utilized.
The unit designation signs that are utilized are color code) blue for Unit 1 and yellow for Unit 2
! 4 The operating procedures are color coded blue for Unit 1, ellow for Unit 2 and pink if the procedure applies to both units.
In addition to the events discussed in the enclosure, approximately 5 additional wrong unit / wrong train or wrong component events have i
occurred at NortE Anna which were non-reportable. At the site visit, the HPES coordinator agreed to provide an edited sumery of these events for staff use; however, after further review subsequere to the site visit, the HPES coordinator believed that these reports would not be very useful to the staff after editing out plant-specific infonnation.
D.
Observations Comon to Surry and North Anna Brass component identification tags on valves are currently being replaced with aluminum embossed tags for better visibility. The tags contain a description of the component and the component identification as shown in Figure 3.
W D'$ A:
,:E? A.~. w A.. h. G
' Figure 3 Emergency buses, switchgear, and cable trays are identically color coded in both Units 1 and 2.
Train H is orange, Train J is purple and any comon components (e.g. the line feeding a safety injection pump before splitting up into H and J Trains) are green (Trains H and J designations are equivalent to redundant Trains A and B). To distinguish between units on large electrical panels containing individual motor control switches, one must look for either the larger unit. designation signs described above or the motor control switch tags which will contain unit designation (e.g.,101A for Unit 1, 201A for Unit 2). Addit 3onally, larger component labels which designate unit number are loc,ated on electrical panels as shown in Figure 4.
i I
0
6-3
-! *Q: % IFd k 9 k~:.
w s.
YO $ W
~
r l
- v. egm c.v..
-xma
.U.
Y 3
g
,R i
g 1
4 1
Figure 4 A five shift rotation scheme is used with shifts rotating between Unit 1 l
control board, Unit ~ 2 control board, liquid waste operations, auxiliary l
building, service building (switch-gear, diesel generators), Unit 1 turbine building), Unit 2 turbine building, and outside (switchyard, intake structure. There are usually 10-11 people per shift who rotate every 6-7 days. Assignments within a shift crew rptate at the discretion of the supervisor.
Within the last few years, training programs have been revised to upgrade them and new ones implemented. Prior to these revisions, all general employee training was obtained on the job (0JT). Currently, after receiving 4 days of required radiological training for all employees, 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> of general training is offered on a voluntary basis at the discretion of an employee's supervisor to any interested
~
employees (e.g., mechanics, corporate,etc.)toprovidethe" big picture" regarding plant operations.
Beyond the training already mentioned, operators and engineers are required to take an 80 hour9.259259e-4 days <br />0.0222 hours <br />1.322751e-4 weeks <br />3.044e-5 months <br /> general employee training course and a 40 hour4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> course covering watch-standing principles. To become a Licensed Reactor Operator, 58 weeks of training both at the plant and classroom is required.
VEPCo utilizes a step program for operators, electricians, mechanics, engineers, and ofher plant personnel. The step program is similar to an apprentice /journeyr.an program used by unionized labor whereby one increases in steps with increased experience and training.
4
- _ _ - _ - _ _. _ _ _ I J
Since February 1985, the licensee has instituted corporate-wide training
]
on watch-standing techniques on independent verification. The training i
emphasizes the need, methodology, and circumstances regarding independent verification.
Independent verificati:n is required on tagging of safety,sor.tems and on non-safety systens at the discretion of sys the shift supervi It is used mainly on electrical components and on valve line-ups.
Exit Meetings i
At the exit meetings at both sites, the NRC team expressed its appreciation to the Surry and North Anna staffs, in particular to t6e site HPES coordinators at each site and the corporate HPES coordinator who assisted with the discussions at Surry and the nuclear specialist who assisted with ~
the discussions at North Anna. Their cooperation in plantning the visit, coordinating the tour and discussions, and providing irfomation made the site visits informative and productive.
/L4w-Drew Persinko, Maintenance and Surveillance En;ineer Licensee Qualifications Branch i
Division of Human Factors Safety, NRR
Enclosure:
As Stated cc:
J. Funches T. Ippolito L. Engle D. Neighbors s.,
e es
~
~
WU/WT EVENTS AT SURRY DOCKET NO. 50-280 UNIT 1 DOCKET NO. 50-281 UNIT 2 Boric Acid Improper Valve Line-Up (Wrong' Unit)
I.
LER 281-81-001 1
A.
The following event information was provided by the licensee in the LER:
"With Unit No. 2 at 100% power, valve 2-CH-226 was inadvertently c15 sed. This made one of the two boric acid flow paths to the core inoperable. The inoperable flow path was discovered soon 4
thereafter when operators attempted to use the boric acid
'~
blender to replenish the volume control tank."
~
[
"The cause of the event was personnel error. Unit No. I's valve, 1-CH-266, was to be tagged closed.
However, an operator inadvertently closed Unit No. 2's valve, 2-CH-226."
. i "The seriousness of the event was stressed to all personnel involved and the individual was appropriate 1y' disciplined."
B.
NRC Discussions with Plant Staff.
E During the site visit, the NRC team spoke with the HPES coordinator and the individual who discovered the event. From these discussions, it was learned that the valves are located in the same space below the boric acid tank and are not far* apart. There was j
general agreement that the individual who performed the event was
)
relatively inexperienced as he had been in operations only a few days before tagging the valve. Additionally,4.he label was crusted over with boric acid residue, the individual.was using a respirator due to high radiation -end no independent verification,was required.
Although not positively stated, the NRC team was told that possibly the supervisor gave incorrect instructions and that the lighting may have been poor.
9 i
m
]
2 c.
The HPES coordinator described steps which have been aten subsequent to this event which would cause.such an event to be less likely to recur.
These steps are:
1.
The area where the event occurred has since been cleaned up.
L 2.
Lighting in the area is adequate.
3.
Since the brass labels in use then became difficult to read over a period of time all valve labels are being replaced with' aluminum embossed 1abels.
- 4.. Plant procedures are now stamped with a section which is to be checked if.the label needs to be replaced.
5.
At the time of the event, only OJT was in effect. Now, improved employee training is in effect, as described earlier.
C.
NRC Observations
~~
The NRC team was unable to view the area where the incident occurred because of radiological considerations.
=
~,
1.
The, aluminum emb'ossed tags being installed on all valves are difficult to read, except in good lighting, due to a low contrast between the lettering and the background.
2.
Although all employees are required to receive 4 days of' radiological training, the 40 hour4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> course.regarding plant operations is voluntary at the discretien of an_ employee's supervisor.
l
! II. LER 280-82-072 "A" 5. J. Accumulator Level Below T. S. Ursit with "B" 5. I. Accumulator Valve Closed. (Wrong Train)
A.
The following event information was provided by the lice $see in the LER:
l 1
"With Unit I at 200t power, during the perfonnance of P. T.18.5 (flushing of sensitized stainless steel piping), "A" Safety I
Injection Accumulator was inadvertently drained to a level below the Tech. Spec. minimum.
/1so, at this time, "B" 5. I. Accumulator discharge valve (MOV-18058) was under administrative cortrol, in the closed position, to facilitate performance of P. T.18.5."
"The cause of the event was due to an operator opening the wrong
~
test valve during the performance of the P. T.
This action may have been enhanced by the arrangement of the accumulator test valve switches on the contr ol board."
i
~
"The arrangement of the accumulator test valve switches will be incorporated into the NUREG-0700 review process."
(i.e., detailed -
control room design review)
E "The operator involved was disciplined and reinstructed on the
(
importance of folicEwing ' procedures.,"
B.
NRC Discussions with Plant Staff.
The NRC term spoke with the HPES coordinator and the individual who turned the incorrect switch in this event.
The individual had just become a licensed reactor operator a few weeks prior to this event,
~'
was working on the swing shift, and indicated that this particular 4
7
}
(
,j
)
/
-s
.)
P. T. was never perfonned or thi simulator. The individual had the procedure but still turned the wrong sw'[sc'[ Both units contain the switch configuration shown above.
j l
C.
NRC Observations I
s l
1.
The NRC team viewed the switcher in the control rcva where the error occurred. The switches are rotating knobs such that the associated valve is diosed if the switch is peinting to the left and open if it is pointir:g to the right. The switches are d
associated with three accumulator trains A', 5, and C.
The
~-
configuration of the switches and the associated. train is as-follows:
\\
l>',
a
.i 5; j
r r
k - <
3 h,
A s,
~
i a
0 i
d i
O I
l
?
L *-
'9 e
o e
A B-C
~
ACCUM N2 & VNT ISOL YLYS a
.,*,)
y o
(h A
B C
e.
ACCUM DRN ISOL VLYS y
- s. :
3
.,?~..
e e
o A
B C
r, ACCUM FILL JSOL YLYS j
e o
o l
A B
C
- ACCUM DISCH ISO VLVS wr f,)
l o
e o
A A
B ACCUM-TEST LIN( ISO yLYS
-Swltches where error occurred B
C C
ACCUM TEST LINE 150 VLVS l
w
,1 k
't o
o o
?
~
B C
A
{
s ACCUM DISCH 150 VLVS e
l j 1
l i
2.
The arrangement of these valve switches is a clear violation of human factors engineering principles. The NRC~ team cannot presently state how this human engineering discrepancy'is being addressed as part of the detailed control room design (DCRDR)'
review because the licensee's DCRDR sunrnary report is not expected to be received by the staff until early 1986.
III.
LER 280-83-033 - Torque Switch Removed from MOV-12898 (Wrong Unit)
A.
The following event information was provided by the Licensee in the-
~ --
LER:
"With Unit 1 at 100% power, an electrician performing maintenanc.e removed a torque switch from MOV-12898 (nonnal charging isolation valve). With the torque switch removed, the MOV would not close on a Safety Injection Signal."
.L "At the. time of the event, Unit 2 was at cold shutdown.and a
~
maintenance request had been initiated to adjust limits on MOV-22898(Unit 2 charging 1.colationvalve). However, the electrician assigned to the task went to the operating unit (Unit 1) to perfonn the maintenance. As a result, the torque switch was removedfromMOV-1(898.",
~
" Subsequent Corrective Action: None" "The electrician involved received disciplinary action."
B.
NRC Discussions with Plant Staff.
i During the site visit..the NRC team spoke with the HPES coordinator and the electrician responsible for removing the torque switch. The l
electrician had started working at this site during an outage and
~
l.
4.
had been there for only 4-5 months.
He had worked previously at fossil plants and said that he didn't realize that two units would i
share the same basement because at fcssil plants, separn'te rooms are used for each unit.
He said that he was not faziliar with the plant and did not realize that the first number on the component identification tag designated the unit. He had to clean off the tag before he could read it and only checked the last four numbers be,cause he thought he was in the correct area.
The electrician sdted that the labels are clearer now than they were before.
' ~'
Subsequent to this event, the licensee instituted the general training program which covers unit designation, tags, procedures and dual systems which the licensee believes will reduce the likelihood of this type of event recurring.
C.
NRC Observations The NRC team was unable to view the area where,the event occurred due to radiological considerations. '
1.
Although plant personnel now receive general training, contractors do not.
s.,
' ~
2.
The auxiliary basement does not' contain unit identification signs, either at the time of the event or now.
0
-8 l
IV. LER 280-83-051 CV-P-1B Suction Line Blank Flanged (Wr ng Train)
A.
The following event information was provided by the 11Egrisee ir, the LER:
l l
"With Unit 1 at 100% and with 1-CV-P-1A ("A" Containment Vacuun Pump) tagged out for maintenance, the suction line for 1-CV-P-15
("B" Containment Vacuum Pump) was inadvertently blank flanged. As a result, both containment vacuum pumps were inoperable.
"A blank flange was to be placed on the suction line for 1-CV-P-1A to support maintenance activities on that pump.
However, an operator incorrectly identified the proper suction line for
[
1-CV-P-1A and the flange was installed on the suction line for.
1-CV-P-1B."
" Subsequent Corrective Action:
None" "The operator involved was disciplined.
The re'quirement for procedures and independent verification will be re-emphasized for r_.
the control of blank flanges."
S B.
NRC Discussion with Plant Staff.
The NRC team spoke with the operator who incorrectly identified the suction line to be blank flanged and the HPES coordinator regarding this event.
The reactor operator told the NRC team that hq was supposed to show a mechanic the line to be blank flanged inside conta_in_inent.
He had received verbal instrug,tions from his supervisor in th~e change roorm that the blank flange was to be installed after check valve 4 or 5; however, the supervisor never got back with the operator to confirn l
.g.
the valve number.
The operator was told to get the job going and thus was in a hurry. The reactor operator found the c_ heck valve with a 4 or 5 on it and identified this line to the mechanic as the line to be blank flanged.
Because the operator thought that only one line ended in this area, he did not look any further and believed he had identified the correct line. The reactor operator did not have a print with him, was using protective clothing and a respirator, and said that the area was confined. The reactor operator knew the pump number on which reintenance was to be perforned; however, the pumps are located on the opposite side of the containment from where the blank flange was to be installed and thus, he was not able to trace the line in the field. Since the operator knew the pump number, he would have been able to trac,e,the lir.e on the drawing and identify the correct check valve (4 or 5) associated with the pump requiring maintenance if he had had the drawing with him.
The operator said that the' system'had been 1
changed before he arrived at the plant but that the drawings had not been updated to reflect this change. The old drawing had been used during training.
The NRC team was told that reactor operators are not required to carry a print with them but that doing so is good '-
practice.
The licensee believ,es that the following changes which have been implemented subsequent to this event would make the recurrence of this type of event less likely:
1.
Training now reviews design change packages-(DCP) for training needs. Operators are now trained on design changes which have occurred.
2.
Since this is a safety-related system, independent,' verification is now required.
4
___._._m___._____._-__m._._
3.
Engineering and construction currently require th t safety related changes be reflected on drawings within 15 days and that other changes be reflected on the drawings within 6Q hys.
l 4.
More fonnalized mechanisms now exist to change drawings.
C.
NRC Observations I
The NRC team was unable to view the area where the event occurred because it was inside containment.
O O
at W
e wueuse i
3-g m
)
1 1
e d*
p l
WU/WT EVENTS AT NORTH ANNA DOCKET NO. 50-338 UNIT 1 DOCKET NO. 50-339 UNIT 2 1.
LER 339-82-022 Inoperability of Quench Spray Subsystem and the Recirculation Spray System (Wrong Train)
A.
The following event information was provided by the ifcensee in the LER:
"On May 28,1982, with Unit 2 in Mode 4 at approximately 345'F, both l
trains of the Quench Spray Subsystem and the Recirculation Spray.
l System were inoperable for 17 minutes. The Chemical Addition Syste:
' '~
was isolated for 43 minutes."
"This event occurred during the perforinance of the Emergency Bus Blackout and SI Functional Tests, 2-PT-83.5, when steps were being completed in preparation for the 23 Bus Blackout and SI.
In accordance with the procedure. Train B pumps 2-RS-P-1B, 2-RS-P-2B,
' ~
and 2-QS-P-1B were placed in " Pull-to-Lock".
Pump 2-RS-P-3B was running on recirculation. The Train A pumps, which had been previously tested, were in the " Auto" positidn with the exception of 2-RS-P-3A which was in "Nonnal."
An Engineer and an Electrician were_ instructed to go to Solid State Protection Train B Output Cabinet and install jumpers which insure that CDA loads which are shed from the emergency bus are reloaded onto the bus following restoration of bus voltage. At 2243, they installed the jumpers in Train A of Solid State Protection, instead of Train B, which began starting Train A Quench Spray and Recirculation Spray pumps which
~
were in the " Auto" or "Nonnal" modes of operation. Operations personnel began placing Train A pumps in " Pull-to-Lock" as their energization was annunciated in the Control Room. Casing Cooling Pump 2-RS-P-3A had to be held in the "Stop" position tince the control switch does not have a " Pull-to-Lock" function. The breaker for the Chemical Addition Tank Train A discharge valve MOV-QS-202A 0
1 2
was deenergized and the associated manual isolation valves were closed. At4his time, Train B Quench Spray and Recirculation Spray pumps were in the " Pull-to-Lock" or held in "Off" to terminate an inadvertent actuation of the Containment Spray Systens."
" Scheduled Corrective Action: The test procedare is being reviewed for modifications which will minimize the possibility of personnel error while it is being perfonned.
Color coding of the Solid State
~
Protection Cabinets is being considered as a prevention against personnel entering the wrong train."
"The incident was discussed with the engineers responsible for testing in order to minimize this type of error and to emphasize the proper cautions to be taken."
B.
NRC Discussions with Plant Staff During discussions with the HPES coordinator and the individual whE identified the incorrect cabinet, this individual said he had worked 13 hour1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br /> backshifts for 4 consecutive days and, for approximately one week. prior to the event, he had worked on the A (or H) Train prepar'ing to run the A bus procedure. When he arrived at work on the day of the test, he was infonned that the previous shift had completed testing Train A and that he would be testing Train B (or i
J). He believed that he incorrectly identified the Train A cabinet 4
instead of the Train B cabinet because of a mindset developed as a result of working on Train A the previous week. The individual had the correct procedure with him at the time of the event; however, the procedure had diagrams for both Trains on the s,am[page. The procedure required approximately 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br /> of set-up time for the first test and 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> for the second.
a e
i
! l l
Actions taken by the licensee subsequent to this event which the licensee believes reduce the likelihood of this event recurring are:
1.
The one long procedure has been broken down into 3 separate procedures, each requiring about 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> to set up and run.
This avoids any shift overlap.
2.
Each train has a separate procedure with the common part remaining as one test.
3.
Both trains are no longer diagramed on the same page.
C.
NRC Observations l
1.
The NRC team viewed the Solid State Protection Cabinets where the jumpers were incorrectly installed and,where they should have been installed. The room in which the cabinets were located consisted of a main aisle down the center with rows 0 6 cabinets on either side of the aisle. The rows of cabinets wem perpendicular to the main aisle and created aisles between the cabinets. To enter.a cabinet, one must walk down the main aisle l
and turn right or left into the correct aisle between cabinets, and then select the correct cabinet in that row.
In this case, the electrician and engineer entered the second aisle to the right of the main aisle where the Train A cabinets were located when they should have entered the third aisle where the Train 5 cabinets were located.
l 2.
Although these particular procedures have been, separated by train, this was not done for all procedures which contain
^
references to both trains.
l l
4
~ f 3.
There is no identification at the head of an aisle o, cabinets to aid in locating the correct cabinet. All aisles coming off of the main aisle look extremely similar from the main aisle in the room.
4.
A row of cabinets is approximately 6-7 feet high by 2 feet wide j
by 15 feet long. There are approximately 5 cabinets in a row.
The cabinets are gray in color and are identified by a gray label (approximately 6" x 6") with black lettering stating the function of cabinet contents and the associated train and a.
separate component numbering label. This latter label is
~
approximately 1" x 3" and contains white letters and number's on a black background. Both labels are shown in Figure 5.
=~
u,.;.
2
~
~..
'M W i
~~
i,
'sY. STEM
~ !
OTECTION"
~
i
- RAINEA, a.m y
musa
.a tSC' l
Figure 5
l l
1.
1 II. LER 339-85-006 De-Energization of 120V AC Vital Bus 2-1 i
A.
The following event information was provided by the licensee in the LER:
"At 0915 on April 26,1985, Unit 2 tripped from 100% power when the 120V AC Vital Bus 2-1 (EIIS System Identifier EF, EIIS Component Identifier BU) was inadvertently de-energized. The 120V AC Vital Bus 2-1 was de-energized when an unlicensed Control Room Operater
~-
(CRO) opened the power supply breaker to the inverter which feeds the 120V AC 2-I Vital Bus. The 120V AC Vital Bus 2-1 supplies power to the relay which senses the breaker position of Reactor Coolart Pump 'A'.
When the 120V AC 2-1 Vital Bus was de-energized, this relay was de-energized which caused the reactor protection system to sense, that the 'A' Reactor Coolant Pump breaker was open. A reactor trip signal was generated as a result of the reactor protection system sensing the 'A' Reactor Coolant Pump breaker open coincident with reactor power greater than 30%.
Reactor Coolant Pump 'A' did not actually trip during this event. The 120V AC Vital Bus 2-1 was re-energized seconds after the trip when the unlicensed CRO, who opened the supply h,reaker to the inverter, realized his mistake and closed the inverter' supply breakere All equipment powered from the 120V AC Vital Bus 2-1 responded as expected when the bus was de-energized. All eight circulating water waterbox vacuum breakers opened when the vital bus was de-energized, which caused all circulating water pumps to the main condenser to trip. As a result, the condenser was not available_td remove secondary side heat via the condenser steam dumps.
Ifstead, steam was released through the stear generator PORV's to the atmosphere.
In addition, two rupture discs (EIIS Component Identifier RPD) were
blown and four rupture discs were damaged o low pressure turbines because circulating water was not available to cool the condenser.
I These rupture discs were replaced.
All Auxiliary Feedwater pumps started as a result of low Steam Generator level.
TV-BD-2000, 'B' Steam Ger,trator blowdown inside containment isolation valve, indicated mid-;osition following a low Steam Generator level isolation signal.
The redundant blowdown trip valve, TV-BD-200C, closed during this event. The position indication problem with TV-BD-200C was corrected on April 26, 1985.
,[
The 2-1 120V AC Vital Bus was de-energized when an unlicensed CR0 opened a breaker in the wrong power supply cabinet. The Unit 2 Assistant Shift Supervisor (a SRO) had instmeted the unlicensed CR0 to open a breaker in a specific power suppls cabinet as part of preparation for a maintenance activity.
The SRO had obtained the 1
breaker number and power supply cabinet identification from name tags on the Main Control Board.
The power supply cabinet f-information on the Main Control Board was risleading which caused
- he SRO to associate the power supply cabinet infomation on the Main Control Board with,another power supply cabinet.
Corrective s
actions to prevent recurrence' are being evaluated.
l The Unit was returned to criticality on April 27, 1985 and reached 100% power on April 30, 1985."
i B.
NRC Discussions with Plant Staff i
I The licensee believed Jhat differences in c:mponent identification schemes contributed to de-energization of tte incorrect breaker.
The labelling used in the control room on tte control boards
f i 1 1
sometimes uses common names without unique mark numbers,(irdividual component numbers).
Individual component or mark numbers can take various forms, all of which are equivalent.
For example, each of the following mark numbers can designate the same piece of equipment:
9
_T_V CC-104A
~
TV-CC-104A 1-CC-TV-104A Component number i
Component System Designation (e.g.
component cooling)
Unit Number In this particular instance, the SRO gave an, unlicensed CRC a slip of paper containing the valves to be closed, the panel nunter in.]
which the circuit breaker is located and the circuit breaker number.
The information was taken from the main control board valve switch
=
label. The SRO turned t.o another CR0 to confirm the location of the
~.
panel and told the unlicensed CR0 that it was located in a level below the control room.
The unlicensed CR0 went to that location and opened the circuit corresponding to the circuit number on the piece of paper given to him by the SRO. Both panels were known as 2-1; however, the panel'where the unlicensed CR0 should have gone was Main Control Board DC SOV Panel 2-1 whereas he went downstairs to DC Distribution Panel 2-1 as directed by the SRD,,'The licensee said that there are several panels commonly referred to as 2-I.
Procedures reference unique mark numbers but were n'ot used in this M
4
-___m______.._
e The following depicts the labels involved in tkis error.
)
case.
Name tag of valve switch to be closed as shown on main control board:
1 l
LOOP B FILL HDR ISO VV HCV - 25568 i
~
l Information on the slip of paper given to the unlicensed CR0 by the SRO:
HCV-2556 A,B,C DC PNL 2 '
CKT 13 Name tag on the power supply panel that the unlicensed CR0 should have gone to behind the vertical boards in the control room:
MAIN CONTROL BOARD
=
DC SOV: PANEL 2-I
~.,
2-EP-CB-26A Name tag on DC Distribution Panel 2-I located below the control room that the unlicensed CR0 mistakenly went to as directed by the SRO:
2-EP-CB-12A Name tag next to breaker that incorrectly was opened in DC Distribution Panel 2-I located below the control room:
2-VB-I-01 2-VBBANK001
i J
< )
N 4
Although the SRO had not given the explicit mark numtdr, pf the j
cabinet and had given the incorrect location of panel, the i
unlicensed CR0 could have noticed the error by checking the valve q
numbers on the slip of paper given to him by the SRO against the tag next to the breaker in the panel.
These tags identify the valve numbers associated with the particular breaker adjacent to the tag.
The tag should have read 2-SOV-2556A, B, C instead of 2-VB-I-01..
The panels usually also have a list of the circuit breakers and their associated valves, known as the load list, located on the inside of the panel door.
The enlicensed CR0 believed that the load l
list may have been useful in stopping the error because it would have provided him with a list showing what valves are associated with each breaker; however, the load list was not in place at the _
time the error occurred.
Because of different amperage ratings, the-l 1-breakers in the correct panel are smaller than those in the incorrect panel which could have provided the unlicensed CR0 with a clue that something was in error. The unlicen' sed CRO, hcwever, said that he saw 2-EP-CB-12A on the panel and knew it was D:.
Because df this and because it was the location told to him by the SRO, he I
thought he was in the correct panel.
' ~
He did not check the number next to the breaker but rather just went
.to breaker 13 as written on the slip of paper and de-energized the circuit. He said that perhaps the missing load list way have thrown him off.
Subsequent to this event, the licensee has taken the following steps to reduce the likelihood of this event recur. ring:
~~
1.
The event has been discussed with other operators pointing out that one should stop and check if there are any discrepancies a
l *
~
between unique mark numbers. During operator qualif,1 cation and requalification, it will now be stressed that operations personnel use unique mark numbers.
2.
Control room labels are being revised to supplement comon names with mark numbers.
C.
NRC Observations
~.
1.
The power supply cabinet that was incorrectly entered is a dark gray cabinet approximately 6 feet high by 2 feet wide locat,ed at
[
a level which is below the control room.
There were two similar cabinets (2-1 and 2-II)-in the same general vicinity. The 2-11 cabinet in that vicinity in which no error had occurred had a large 2-II written in magic marker on the outside as well as a 1" x 3" black label with white lettering designating the component number glued to the front. The cabinet in which the error occurred had a large 2-1 written in magic marker on the C outside. The location for the black component label was clearly visible;however, the label was not in place. On the inside of
=
the cabinet door is,a load list giving the breaker number and -
its associated component nud ar. The component number is also
~
located next to the individual breaker. The NRC team also 1
1 viewed the panel and breaker that was supposed to be l
de-energized which was located in the control room behind the l
vertical boards. The interior of the cabinet also contained the
)
load list and breaker identifications described above.
I
~.
3 2.
Procedures at North Anna do not contain a stamp (as at Surry) to 1
check off if labels need replacement. Thus, it is less likely to correct labelling deficiencies such as that noted by the NRC team regarding the label denoting mark number 2-EP-CB-12A which had come off and had not been replaced.
l.
i III. Non-Reportable Event: Maintenance on Incorrect Lube Oil Pump (Wrong Train)
I I
A.
Discussion Due to a high differential pressure across an oil strainer, maintenance had been scheduled on a lube oil pump in Unit 2.
The pumps for each unit are the same and are located in separate pits, that are accessible by ladder. The Unit I pumps have mark numbers at the top of.the ladder whereas Unit 2 does not. However, when inside the pit, identifying mark numbers exist on the pump. The maintenance workers entered the wrong pit and began to perfonn, work on the pump associated with the wrong unit. The breaker to the pump that was requiring maintenance had been de-energized; however, the red and green lights indicating whether the' pump was running i
were both cff because of a problem with either the switch or the bulb. Knowing that the breaker had been de-energized, the workers assumed the pump they were about to work on was off.
The noise level in tk area is high with the main pump running; however, thef-mechanics were wearing ear protection and only the auxiliary oil I
i pump was running; the main pump was off. The mechanics proceeded to j
remove a bolt from,the pump and realized the pump was running when oil came shooting out of the bolt hole.
To reduce the likelihood of this event recurring, Unit I and Unit 2, signs with associated mark numbers have been placed on the wall in the cubicle.
B.
NRC Discussions With Plant Staff The NRC team did not speak with the workers who performed the incorrect action, f
e
i l
l I
22-i l
C.
NRC Observations The NRC team was unable to view this area due to radiological considerations.
e 4
m
{
\\
9
.=
egy s.
e e
e O
O 9
O m
1 4
f