ML20090D270
| ML20090D270 | |
| Person / Time | |
|---|---|
| Site: | Quad Cities  |
| Issue date: | 02/26/1992 |
| From: | Burgess S, Jablonski F NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML20090D276 | List: |
| References | |
| 50-254-92-07, 50-254-92-7, NUDOCS 9203060124 | |
| Download: ML20090D270 (18) | |
See also: IR 05000254/1992007
Text
>
s
,
d
AUGMENTED INSPECTION TEAM REPORT
U.-S. NUCLEAR REGULATORY COMMISSION
QUAD CITIES UNIT 1 REACTOR SCRAM AND ASSOCIATED EQUIPMENT FAILURES
FEBRUARY 27, 1992
INSPECTION REPORT N0, 50-254/92007(DRS)
!
0
50
54
--
G
. - . .
.
-
,
y
.
-TABLE OF CONTENTS
1.0
Introduction ................................................ 1
1.1
Event Summary ............................................... 1
1.2-
AIT Formation ............................................... 1
1.3
AIT Charter ................................................. 2
2.0
De scri pti on of the Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1
Sequence of Events .......................................... 2
2.2
Operator Response ........................................... 4
3.0
Inspection Results .......................................... 5
3.)
The Cause of.the Reactor Scram .............................. 5
3.2
Equipment Failures or Mal functions . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2.1
Fail ure o f the HPCI Stop Valve . . . . . . . . . . . . . . . . . . . . . . . . . , . . . . 7
3.2.2
Failure of "C" Electromatic Relief Valve . . . . . . . . . . . . . . . . . . . . 8
3.2.3
Failure of Reactor feed Pumps to Automatically Trip
........10
3.2.4'
Main Steam Line Flow Instrument Anomalies . . . . . . . . . . . . . . . . . .11
3.2.4.1
Off Normal Instruments ..................................... 12
3.2.5
Other Less fignificant Equipment failures .................. 12
4.0
Conclusions ................................................ 13
5.0
Exit Interview ............................................. 14
AIT Charter ...................................... Attachment 1
February 7, 1992 CAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 2
Personnel Contacted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attachment 3
,
a
s-
--
,es
,,
,-,
,
-
--
+
-
n-
,
U. S. NUCLEAR REGULATORY COMMISSION
REGION III
P2 port No. 50 254/92007(DRS)
Docket No. 50-254
License No. DPR 29
Licensee: Commonwealth Edison Company
1400 Opus Place
Downers Grove, IL 60515
. Facility Name:
Quad Cities Nuclear Power Plant, Unit 1
Inspection At:
Quad Cities Site, Cordova, IL
Inspection Conducted:
February 8 - 13, 1992
Inspectors: F. L. Brush, Resident Inspector - Clinton Po w Station
F. P. Paulitz, NRR, Instrument & Control Systems Branch
R. M. Pulsifer, NRR, Division of Reactor Projects, PD Ill-2
D. E. Roth, RIII-(observer)
H. A. Walker, Rill, Maintenance and Outages Section
.
Approved By:
DD h*b
2-2'-3I
ohia D. Bur'gess
Date
T am Leader
Approved By:
.h4
6(1
t.'
d-26' D
2
. Frank J. aablonski, Chief
Date
l
Maintenance and Outages Section
Inspection Summary
Inspection on February 8 - 13. 1992 (Recort No.- 50-254/92007(DRS))
Areas Inspected:
Special Augmented Inspection Team (AIT) inspection conducted
in response to the Quad Cities Unit I scram and equipment failures which
occurred on February 6 and 7, 1992. The review included validation of the
sequence of events, the cause of the reactor scram, the failure of the HPCI
stop valve, the failure of the "C" relief valve, the apparent failure of the
reactor _ feed pumps to automatically trip at the appropriate vessel level and
related operator actions, anomalies associated with the main steam line flow
!
instruments, and evaluation of the licensee's corrective actions.
~
Results:
No violations or. deviations were identified in any of the areas
inspected. No significant operational safety parameters were approached or
. exceeded. .The AIT concluded the following:
1.
The root cause could not be determined for the main steam high flow trip
signal, which caused the Group I isolation that ultimately led to the
l
reactor scram.
The licensee had performed reasonable analyses and
testing to determine root cause. Additional equipment to monitor for
!
--
. .
.
.
.
. - - _
_ _ _ _ _ _
'
,
abnormalities within the main steam high flow trip system was installed
prior to Unit I start-up.
2.
The failure of the HPCI stop valve was attributed to an inadequate
maintenance work package, which was performed in February 1991.
The
work package was considered inadequate because it did not include as-
found or as-left readings of the clearances between the poppet guide and
valve poppet.
After welding was performed on the poppet guide assembly,
incorrect tolerances caused the valve to eventually become stuck in the
open position during HPCI testing conducted on February 6,1992.
3.
The failure of the "C" Electromatic Relief Valve (ERV) was attributed to
brass dust on the shorting contact bar, which was caused by n.atn steam
system vibration.
The licensee determined that the dust originated from
brass components near the shorting contact bar,
in addition, the AIT
concluded that deficient preventive maintenance for the ERVs existed.
Although previous ERV failures indicated high resistance readings across
the contact shorting bar, the licensee did not evaluate the possibility
of adding preventive maintenance to periodically obtain resistance
readings and clean the shorting bar. Also, the licensee did not pursue
obtaining experience from other Ceco nuclear plants that have the same
type of relief valves.
Preventive maintenance practices may have
precluded the failure of the "C" relief valve.
4.
The apparent failure of the Reactor Feed Pumps to automatically trip at
the appropriate vessel level was attributed to instrument drift.
Because the trip setpoint for one of the two level indicating switches
had drifted from +48 inches to +53.5 inches, neither of-the RFPs would
have tripped automatically at the expected trip point.
Operator actions
manually tripped the RFPs before reaching the respective trip settings;
however, if manual actions had not been taken the pumps would have
automatically tripped at +53.5 inches.
5.
Anomalies associated with the "B", "C", and "0" main steam line flow
indicators, were attributed to a faulty power supply and square root
converters. These instruments are for indication only and are different
from the transmitters that provide the reactor protection shutdown.
The
control room flow indicators had "Off Normal Instruments" or ONI
stickers on the front, which meant that maintenance was needed to repair
previously identified problems of false flow indications.
These
erroneous flow indications occurred again after the MSIVs were closed
during this event and resulted in personnel being sont into the plant to
look for steam leaks that didn't exist.
The AIT was concerned the
operator's attention was diverted because of indicators that needed
repair.
.The team concluded that the operators performed well in mitigating the
,
consequences of the reactor scram and Group 1 isolation.
l
l
- -
,_
-.
.
- - . .
. - . -
..
1
i
,
1.0
Introduction
1.1
Event Summarv
On February 7, 1992, at 2:01 a.m., the Unit I reactor scrammed from 100
percent power after an erroneeus main steam line high flow signal apparently
caused the main steam isolatici valves (MSIV) to close automatically. During
the event, the high pressure ctolant injection (HPCI) system was out of
service and in day 1 of a 14 da.' Limiting Condition for Operation (LCO).
Immediately following the main steam isolation and reactor scram, the control
room operators manually tripped the "B" reactor feed pump (RFP) prior to the
expectes trip setpoint of +48 inches, the "A" RFP was also tripped by
operators after reactor water level exceeded the RFP trip setpoint of +48
inches, lhe "C" RFP automatically started and was also tripped by operators.
To lower reactor pressure, the operators opened the "B" electromatic relief
valve (ERV).
Once pressure dropped to an acceptable level, the "B" ERV was
closed. The reactor core isolation cooling-(RCIC) system was also manually
initiated to control reactor pressure, The operators attempted to open the
"C" ERV; however, the valve did not respond.
The "B" ERV was again opened to
control pressure.
During the event, the "B", "C", and "D" main steam line flow indicators
indicated a small amount of steam flow with the MSIVs closed. A team of
personnel was sent into the plant to inspect for steam leaks.
When no leaks
were identified, seven of the eight MSIVs were reopened. The "B"
inboard MSIV
was left closed due to a continued concern with the "B" main steam line flow
indication.
At 3:17 a.m., the reactor scram was reset and at 4:00 a.m. the RCIC pump was
taken off line.
1.2
Auamented Inspection Team Formation
Subsequent to the reporting of this event, Region III managers determined that
an Augmented Inspection Team (AIT) was warranted to gather information on the
causes, conditions, and circumstances relevant to several equipment failures
associated with the reactor scram. On Friday, February 7, 1992, an AIT was
formed consisting of the following personnel:
Team Leader:
S. D. Burgess, Team Leader, RIII - Maintenance and
Outages Section
Team Members:
F. L. Brush, RIII - Resident Inspector, Clinton Power
Station
F.- P. Paulitz, NRR - Instrument & Control Systems
Branch
R. M. Pulsifer, NRR - Division of Reactor Projects, PD
III-2
D. E. Roth, RIII - Observer
H. A. Walker, RIII - Maintenance and Outages Section
1
.
.
.
- -
_
__
.
.
.
.
.
.
The team arrived on site during the morning of February 8, 1992.
In parallel
with formation of the AIT, Rlli issued a Confirmatory Action Letter (CAL) on
February 7, 1992, which confirmed certain actions in support of the team and
established conditions required to be met prior to the restart of the plant
(Attachment 1).
1.3
AIT Charter
A charter was formulated for the AIT and transmitted from T. O. Martin to
S. D. Burgess on February 7, 1992, with copies to appropriate EDO, NRR, AE00,
and Rill personnel (Attachment 2).
The AIT was terminated on Thursday, February 13, 1992.
2.0
Description of the Event
2.1
Seouence of Events
At the AIT's request, a chronology of events related to the scram on February
7,1992, was assembled by the licensee.
The chronology, which included
"
operator actions, was verified to be accurate by AIT personnel by review of
operating logs and interviews with licensee. operating personnel.
The
chronology was as follows:
NOTE: All times are in Central Standard Time.
Jnitial Conditions
Unit 1
820 MWe
HPCI out of service (day 1 of 14)
Unit 2
Refueling outage - core unloaded
Reactor water level at flange
Time
Event Descripti_o_n
01:59
Center desk nuclear station operator (NS0) leaves control room.
02:01:06
Reactor scram, Group I isolation (MSIV closure)
02:01:09
Group 11 and 111 trip.
02:01:53
"B" RFP manually tripped by Unit 2 NSO.
NSO resumed duties on
Unit 2 by direction of Shift Control Room Engineer (SCRE).
02:02
Center desk NSO enters control room.
SCRE initiated entry into
abnormal procedure QGA 100.
02:02:53
Reactor water level indicates +48 inches on the "A" and "B" Yarway
instruments.
GEMAC level indicates +56 inches.
2
3
..
02:03
Shift engineer _(SE) enters control room.
Reactor water level:
GEMAC = +56 inches, Yarways >50 inches.
02:03:46
"A" RFP manually tripped by Unit 1 NSO.
02:03:46
"C" RFP automatically comes on.
RFP standby position was not
deselected.
02:03:48
"C" RFP turned off.
02:05:52
Reactor pressure at 1041 psig.
02:06
Extra NSO opens "B" ERV to lower reactor pressure.
02:07
Extra NSO places residual heat removal (RHR) in torus cooling
mode.
02:08:09
Reactor water cleanup blowdown flow established.
02:10
Communications center senior reactor operator (SRO) notifies the
Assistant Superintandent of Operations (AS0)_of event.
02:12
Shift foreman dispatched to inspect for steam break by SE.
sent equipment attendants (EAs) to RHR to inspect immediately,
nothing found.
SE sent instrument maintenance foreman to check
main steam line flow differential pressure (dP) instruments.
All
were consistent.
02:12:05
"A" RFP turned on.
02:14
Extra NS0 closes "B"
ERV.
02:19
Extra NSO placed RCIC-in service to control reactor pressure.
02:29
Extra NSO attempts to open "C" ERV to control pressure.
Valve did
not open-as indicated by position lights, acoustic monitors, steam
flow, alarms, and pressure indication.
. 02:29:38
Extra NS0 opens "B" ERV to control pressure.
02:30:36
"A" RFP turned off.
02:32
ASO contacts control room for briefing.
'
02:33:15
"A" RFP on.
SCRE monitoring level per procedures.
02:33:17
Group II and III isolation.
. 02:35
"B" ERV closed.
02:53
Group I isolation reset. Area inspections show no visible
indications of a steam leak.
I
!
3
l
4-,
. , . - - - , <
~
. < , ,
r,~
- - ,
-
. , , - , , . . , . . , ,
, - - -
. . , .
- - - , - . ,
_
_
.
02:58
.Seven MSIV's opened by extra NSO.
"B" inboard MSIV left closed
due-to concern with flow indication.
.
03:15
NRC resident notified of event.
03:17
Reactor scram reset.
03:30
AS0 notified nuclear d"ty officer of event.
04:00
RCIC turned off.
Condensate and feedwater system used for reactor
level control.
04:02
Exited abnormal procedure QGA 100,
04:12
Emergency notification completed for engineered safety feature
(ESF) actuations.
04:15
ASO notifies Technical Superintendent of event.
05:42
"A" RFP shutdown by Unit 1 NSO.
06:30
MSIVs closed to determine if MSIV flow obstruction existed.
All
MSIVs determined to be functional and isolatable.
2.2
Operator Response
The team reviewed plant logs, appropriate plant emergency and off-normal
procedures, and interviewed the operating crew to determine what actions were
taken in response to the event and the suitability of those actions.
On February 7, 1992, prior to the event, the operators were performing routine
functions with no major evolutions or plant transients in progress.
The HPCI
system was in day 1 of a 14 day LC0 for corrective maintenance as detailed in
section 3.2.1.
-The initial event information noted by the operators was as
follows:
" Channel B Main Steam Line High Flow"
e All MSIVs indicated shut
e Reactor scram
1
l
At 2:01:06 a.m. on February 7, 1992, an automatic reactor scram occurred when
l
the MSIVs closed on a main steam line high flow signal.
Almost immediately,
the Unit 2 NSO removed the "B" RFP from service after reactor vessel level,
which had reached a minimum of -20 inches below instrument zero, started to
l
increase. This was a normal action for the operators since the feedwater
regulating valves had a history of seat leakage when fully closed.
The SCRE
initiated entry into Quad Cities General Abnormal Procedure QGA 100, "RPV
Control," because reactor water level had decreased below +8 inches during the
event.
The following procedures were used as a result of the entry in QGA
l
100:
4
-
-. *
A
s-_=-
.a_
. .
L.+..Jm. _ . ,
,A.__
.-.
.
e Quad Cities Operating Procedure QCOP 3200 9, " Condensate /Feedwater"
e Quad Cities Operating Procedure QCOP 1300-2, "RCIC System Manual
Start-Up" (Injection / Pressure Control)
e Quad Cities General Procedure QGP 2-3, " Reactor Scram"
'After the shift engineer directed recovery operations, the Unit 1 NS0 removed
the "A" RFP from service after reactor water level exceeded the RFP trip
setpoint of +48 inches. The "C" RFP automatically started because the
operator failed to place the
"C" RFP switch in the "0FF" position before
securing the "A" RFP. Additional information on this action is contained in
section 3.2.3 of this report. The extra NS0 opened the "B"
ERV to control
reactor pressure and also placed RHR in torus cooling. At 2:12:05 the "A"
was placed in service. After pressure had decreased, the "B" ERV was closed
and the RCIC pump was placed into service to control reactor pressure;
however, the RCIC pump was not used to inject water into the reactor vessel.
As pressure again began to increase, the "C" ERV failed to respond when an
operator attempted to open it. A detailed discussion of this failure is
contained in section 3.2.2.
Immediately, the operator opened the "B"
ERV.
The "A" RFP was then secured for three minutes to control reactor water level.
Again, after pressure had decreased, the Unit 1 NSO closed the "B"
ERV.
Early in the event the SCRE requested that IM personnel check the MSIV high
flow sensors. After checks were made, the IMs reported that there were no
apparent problems. The operator also noticed anomalies with the "B", "C",
and
"D" main steam line flow indications.
Because these instruments indicated a
small amount of flow when the MSIVs were closed, a team of personnel was sent
into the plant to inspect for signs of a steam leak. No leaks were identified
and seven of the MSIVs were reopened.
The "B" inboard MSIV was left closed
due to a continued concern with the "B" main steam line flow indication.
Further discussion of these anomalies is contained in section 3.2.4.
At 3:17 a.m., the reactor scram was reset and at 4:00 a.m. the RCIC pump was
taken off line and procedure QGA 100 was exited.
The shift engineer also
reviewed Quad Cities Abnormal Procedure QGA 200, " Primary Containment
-Control," during the event and determined that plant conditions did not
require entry into that procedure.
Based on review of this event and operator interviews, the team determined
that the operators performed well in responding to the plant transient.
Their
actions were prudent and conservative, placing the plant in a stable condition
in a very short time.
3.0
Jm nection Results
3.1
The Cause of the Reactor Scram
The cause of the Group I isolation signal, which resulted in the closure of
the MSIVs, and ultimately in the reactor scram, could not be determined. When
the reactor scrammed, only the "B" Channel High Steam Flow annunciator was
5
_
_ _ .
_. __ _ _
-.
__
'
.
the reactor scrammed, only the "B" Channel High Steam flow annunciator was
received in the control room. No alarms were received for the "A" Channel
High Steam Flow, the A and B Group I isolation annunciators, High Steam Flow
or Group I Isolation computer point alarms nor any other Group I isolation
alarms.
The Group I isolation signal comes from Barton Model 27B differential pressure
indicating switches (DPIS), four of which are connected to each flow element
with a flow element on each of the four main steam lines.
Thcse 16 DPISs are
located in the RHR room in the southeast corner of the reactor building
basement.
Eight of the switches are on panel 2201-10 sections A and B and
eight are on section C.
In order to achieve a full Group I isolation, one of
the eight DPISs in each channel must actuate, which will cause the MSIVs to
close and scram the reactor.
The licensee speculated that the Group I isolation was caused by a spurious
actuation of the DPISs, possibly by personnel bumping the racks or instruments
since this had occurred-in the past.
The sei.uriiJ card readers and radiation
protection control records indicated only three per:onnel could have been in
the area of the racks when the plant scrammed; however, ir+arviews with these
personnel indicated that they were not in the immediate area of the
instruments when the event took place.
The licensee-performed a functional check and calibration of the trip
-
circuitry.
One DPIS was replaced and all indications and inputs to
annunciators and computer points worked correctly.
To determine whether bumping of these instruments or racks could have caused
the reactor scram, the licensee conducted a special test that pressurized the
DPISs to a pressure differential close to the trip setpoint.
The torus bulk
head door and rack enclosure door were slammed. The impulse sensing lines,
racks, and instruments were bumped for trip susceptibility.
No alarms were
received for high steam flow or Group I isolation as the result of the special
test.
The licensee also checked several of the flow check valves in the sensing
lines _and determined that_there was no blockage.
Walkdowns of the sensing
lines and the electrical wiring from the instrument racks to the circuit fuses
and trip relays (102 A through D) in the control room panels were performed
- but no abnormalities were identified.
L
The AIT team inspected the instrument racks to determine the degree of
protection of the instruments on the racks from accidental bumping.
The
l
probability of bumping these instruments or a rack when passing from the torus
!
area to the stairwell in that southeast RHR corner room is remote. The rack-
l
containing instruments A through H is located next to a wall with space
!
between the instruments and the stairway.
Thera is a wire mesh barrier over
i
the top of the rack and on the side of the stairway between the instruments
l-
with a door for access to the racks.
The other rack is 90 degrees around the
corner, also against the wall, and it has a wire mesh barrier above it.
Personnel must pass in front of rack s1ction A and B before getting to section
C.
6
m
,,r-
-=m--
w
--
-w
,-
w
ry
ag-
g.
-
w-
-
-
+,
w
..
..
- _ _ . . -
..
. - .
.
-
-.
.
.
'
,
During the exit, the licensee committed to monitor the eight main steam flow
sensing lines with pressure transducers whose output will be recorded to
determine if there are any_ process fluid disturbances.
The four DPIS' input
contacts to each of the four sub-channel trip logic relays will also be
monitored and recorded to pinpoint which set is actuating.
This monitoring is
intended to help identify the source of the signal that caused the MSIV
isolation on February 7,1992.
The monitoring equipment was in place prior to
start up from this event.
The licensee also proposed a long term modification to the main steam line
high flow detection system.
The digital Barton DPIS instruments will be
replaced with an analog system. Unlike the present DPISs, the functional
check of this new equipment would be made from an area not subject to
radiation.
This modification is projected to be more reliable as well as
reduce exposure to personnel.
3.2
Equipment Failures or Malfunctions
3.2.1
Failure of HPCI StoD Valve
During quarterly HPCI pump tests the HPCI steam stop valve successfully
operated a number of times from April 1991, until February 6, 1992, when the
valve failed to close during post modification testing of the remote HPCI
turbine trip pushbutton.
Unsuccessful attempts were made to close the valve;
however, the valve was eventually closed by applying an external force.
The HPCI turbine steam stop valve is a poppet type, hydraulically positioned
shut-off valve designed to close quickly on the following trip signals: HPCI
-turbine overspeed, high reactor water level, low HPCI booster pump suction
pressure, high HPCI turbine exhaust pressure, remote HPCI turbine trip
pushbutton, and local manual trip lever.
Following the failure and to allow for further troubleshooting, maintenance
personnel attempted to disconnect the actuator from the poppet stem.
The
valve bonnet was then removed to facilitate inspection of the poppet. The
outside of the poppet and the inside of the poppet guide, which was welded to
the bonnet, were severely galled.
This interference prevented the valve from-
-
stroking.
The AIT determined that the root cause of the valve failure was an inadequate
maintenance work package that was completed in February 1991. The work
package was inadequate, because it did not include as-found or as-left
readings of the clearances between the poppet guide and valve poppet after
welding was performed on the poppet guide assembly.
The welding caused the
guide to become oval shaped and to lose perpendicular 1ty with the bonnet,
-
which allowed the poppet to come into contact with the guide after the valve
was reassembled and resulted in fretting of both metal surfaces when the valve
was operated. The valve stuck open during the HPCI test that was conducted on
,
l
February 6, 1992.
The licensee repaired the Unit 1 poppet assembly in accordance with vendor
recommendations using the poppet guide and valve bonnet from the Unit 2 HPCI
7
L
_
-
.
. .. _
'
.
valve.
During inspection of the Unit 2 stop valve, indications of contact
between poppet and poppet guide were noted.
Measurements indicated that.the
poppet guide was perpendicular to the bonnet but the guide was slightly out of
round.
In 1988, cracks were also repaired on the poppet guide to bonnet weld
similar to the weld repair on the Unit 1 valve.
The work package had instruc-
ted maintenance personnel to take as-found measurements on the poppet guide's
position in_ relation to the bonnet.
However, there were no instructions to
measure the concentricity of the guide.
The licensee planned to replace or
repair the Unit 1 poppet guide and install it in the Unit 2 HPCI stop valva.
3.2.2
Failure of "C" Electromatic Relief Valve
During cool down from the Unit I reactor scram on February 7, 1992, pressure
control was accomplished by manually initiating the RCIC system and opening
the ERVs, as needed, to avoid reaching the automatic pressure relief set
points.
As pressure rose initially, ERV "B" was opened and, after pressure
dropped to an acceptable level, the valve was closed. After a short period,
as pressure began to rise again, the control switch for ERV "C" was placed in
,
the open position; however, the valve did not respond.
The "B" valve was then
opened a second time to control the pressure.
The ERV is a six inch solenoid actuated pressure relief valve, which may be
operated, when desired, by closing a switch that allows 125 volts DC to
actuate the operating solenoid. Automatic pressure relief is accomplished by
using a pressure sensing element to control the solenoid.
A cut out switch
(contact shorting bar) is built into the solenoid assembly to increase coil
resistance in order to reduce the solenoid current when the solenoid is
actuated and in the holding position.
The Unit 1 plant has fivc pressure
relief valves that are used to prevent over pressurization of the main steam
system.
The "A" relief valve is a dual purpose valve manufactured by Target
Rock and the other four valves, "B", "C",
"D" and "E", are ERVs manufactured
by Dresser Industries.
During troubleshooting,- two unsuccessful attempts were made to actuate the
valve from the control room. -After-the cover for the solenoid was removed,
.the valve opened on the first attempt.
On subsequent attempts the valve
appeared to open and close normally. The cover and-internal components were
carefully examined along with the plunger and other moving parts of the
solenoid; the results were inconclusive.
Electrical connections were tight
and electrical measurements indicated that the valve solenoid coil was normal.
The resistance reading across the cut out switch terminals was 182 ohms, which
was very high compared to the- normal resistance-of less than 1 ohm.
The
licensee concluded that the high resistance across the cut out switch contacts
caused the failure of the solenoid valve on ERV "C" to operate on demand.
An inspection of the "C" cut out switch, indicated a tarnish build up on the
contact points.
An analysis of the material indicated that the tarnish most
likely was caused by the loose dust found in tha enclosure which vaporized
during contact arcing.
The origin of the dust appeared to be wear from moving
brass parts located inside the actuator.
After cleaning, the moving parts of
the four Unit 1 ERVs were lubricated with a graphite based lubricant to reduce
friction and wear.
8
- -- _ .-.
.
.
. -
. . - . -
.-
--
- . . .
- - - -
__ ..
_
__
_ _
__
_
_
_ _ .
_ _ _ _ . __
_
i
..
b
The' AIT agreed with the-licensee that the failure of the "C" ERV was
attributed to brass dust on the contact shorting bar; however,- the team also
-
concluded that deficient preventive maintenance (PM) for the ERVs existed.
Maintenance history indicated that Unit 1 ERVs had failed to open 12 times
since 1975.: Four of these failures, including the most recent, were
attributed-to problems with the_ cut out switch; however, _it was difficult to
determine if the failure mechanism was the same as-the February 7, 1992
4
failure. One of the failures _ indicated a high electrical resistance across
the cut out switch contacts but none of the previous failures indicated
,
tarnish or other buil_d up on the switch contact surfaces.
Although electrical
i
PM-tasks were in place as. required by the EQ requirements and vendor
4
recommendations, the licensee did not evaluate the possibility of adding PM to
. periodically obtain resistance readings and clean the cet out switch. Also,
Ethe same type of relief valves are used at the licensee's Dresden Station, but
the-licensee had not evaluated Dresden's experience with the valves for
applicability to Quad Cities. Dresden's PM program included obtaining
resistance readings.and lubricating certain brass components in the valve
actuator. -Dresden had not noticed any dust in their valve actuators, nor
failures of the ERVs due to high resistance across the cut out switch.
The ."C" ERV cut out' switch was replaced and the solenoid was cleaned,
reassambled'and tested. _ The valve stroked properly after work was completed.
-The licensee checked resistance readings on the remaining Unit 1 ERVs and all
- were_below eight ohms.
The ERVs were-then cleaned and tested.
As-left-
-
resistance _ measurements across the-cut out switches for all four ERVs was less
.than one ohm. All four ERVs stroked properly after the cleaning and tests.
,
The team reviewed maintenance history dating back to 1986 for the "A" Target
Rock relief valve.
No significant failure trends or excessive repetitive
failures were noted.
Licensee _ personnel stated that there were no_ existing-mechanical PM tasks in
place for the ERVs;-however, some types of mechanical PM were being performed.
For example, the' ERV pilot valve internals were changed each time a unit
- outage occurred in order to prevent inadvertent opening of the ERVs due to
excessive leakage through the pilot valves. A-review of maintenance history
verified the pilot valve replacement.
The environmental qualifications (EQ) for the ERVs' qualified life of 40 years
-
-was based-on drywell_ temperatures of 150 degrees.
Information supplied by the
- licensee indicated that temperatures as high as 176 degrees had been recorded
in 1991 in the vicinity of_ the ERVs._ Licensee personnel stated that they
. thought that the temperature issue had been previously noted and evaluated,
-
-
however, the previous evaluation could not be located.
Engineering did a
brief analysis of the temperature concern and determined that the equipment
g
would.still be qualified for 40 years .at temperatures as high as 187 degrees.
- A copy of this. preliminary analysis was provided to' the NRC inspector.
The
'
licensee committed to providing the NRC the final analysis, complete with
appropriate' calculations, within ten days of the exit.
!~
9
-
,
.
._
-
._
.
_
,
3.2.3
Failure of the Reactor Feed Pumos to Automatically Trio _at the
Aonropriate Vessel level
'
The Unit 1 NSO tripped the "A" RFP after the "A" Yarway reactor water level
indication exceeded +50 inches since the feed pumps were expected to trip at
+48 inches. Once the "A" RFP was tripped, the "C" reactor feed pump
automatically started because it was in the STANDBY position. The Unit 1 NSO
then immediately secured the "C" RFP.
The Unit 1 NSO stated that he
remembered seeing the Unit 2 NSO place the "C" standby feed pump selector
switch in the OFF position but did not check the switch before securing the
"A"
RFP.
i
The RFPs are motor driven centrifugal pumps that supply water to the reactor
vessel. The "A" and "B" RFPs were in service with the "C" RFP in STANDBY when
the Unit I reactor tripped.
Forty-three seconds after the scram, the Unit 2
NS0 re.noved the "B" RFP from service per procedure Q0P 3200-5, " Reactor Feed
Pump Shutdown. " .The procedure required the "C" standby reactor feed pump
selector: switch be placed in the 0FF position when removing a feed pump from
service to-prevent an inadvertent start of the standby pump.
The switch was
then placed back in STANDBY after the "B" pump was stopped since the "A" feed
pump was still in service.
The "B" Yarway reactor water level instrumentation
indicated +48 inches, which was the setpoint for the main turbine and RFP
automatic trip.
As level continued to increase, the setpoint (+48 inches) for the "A" RFP trip
was passed, but the "A" pump did not trip.
This did not result in a safety
concern because the operator tripped the pump at >50 inches.
It was
subsequently determined that the setpoint for the "A" RFP had drifted and the
.
"A" RFP would have tripped at +53.5 inches.
The main turbine vendor
recommended that the trip level be set at +60 inches to prevent turbine damage
due to carryover of water associated with high reactor vessel water level in a
There is no credit taken for safety in the FSAR nor any setpoint
or surveillance in the Technical Specification for these Yarway level
indication transmitter switches. However, the drifting indicates that more
frequent calibrations should be performed to- ensure that the RFPs trip at the
expected -l evel . To assure the accuracy of the RFP trip setpoint, the licensee
committed to calibrating these instruments on a quarterly basis instead of-the
current refueling cycle basis.
-The licensee performed an as-found calibration check of the "A" and "B" Yarway
level indication transmitter switches (LITS 59 A&B) that should have initiated
the +48 inch trip signal.
Both LITS signals are ' required to trip the RFPs.
The "A" LITS trip setpoint was +53.5 inches. The "B" LITS setpoint was +48
inches. The operators stated that the reactor water level observed on the
Yarway indicators never exceeded +52 inches.
If the operator had not stopped
the "A" RFP at >50 inches, the.RFPs would have automatically tripped when the
reactor vessel water level reached +53.5 inches, which is 5.5 inches above the-
design trip point. -The "A" LITS was recalibrated to +48 inches and a
functional test was satisfactorily performed on the trip circuit.
The Yarway LITS instruments have had a history of drifting as evidenced by the
calibration and maintenance records for these instruments from October 9,
-
10
_ . -
-
__
_ _ _
-
_-
.
_
- .
'
,
1989, through February _8, 1992. Calibration history of the "A"
Yarway LITS
showed that the instrument had drifted both low and high three out of four of
the last calibrations (including the post-scram calibration).
The "B" Yarway
LITS had drifted low twice before being replaced in November 1989.
3.2.4
Anomalies Associated with the Main Steam Line Flow instruments
During this event, when the MSIVs were closed, the operator reported some
indication of main steam line flow on the control room board indicators 1-640-
23A, B, C, D.
(These instruments are for indication only and are different
from the transmitters that provide the reactor protaction shutdown.)
Flow of
0.5 million pounds per hour (MPH) was observed on the "A" flow indicator (FI)
and the "B" FI indication was observed to be bouncing up and down.
These
erratic flow changes were also seen on the total steam flow recorder after the
trip. .The recorder displayed spikes ranging from 0.0 MPH to 2.5 MPH.
The "B",
"C" and "D" main steam line ficcr indkators in the control room had
'
"Off Normal Instrument" or 0NI stickers on the-front of the indicator meaning
that maintenance was needed to repair previous problems noted with flow
indications. Although there was no indication of high temperature or high
-radiation in the main steam line area, these control room indications provided
questionable data to the operator during this event.
These erroneous flow
readings resulted in personnel being sent into the plant to investigate
potential steam leaks that didn't exist.
The AIT reviewed the open maintenance work requests corresponding to the ONI
stickers for these flow instruments.
The "B" flow indicator had two open work
requests and the "C" and "D" each had one. These work requests were in
various stages of completion and dealt with measured flow when no flow existed
on "B", "C" and "D"_, and erratic flow indication on the "B" indicator. One of
the work requests required work to be done on the Foxboro square root
converters that supplied input to the four main steam high flow indicators and
total steam flow for the recorder used during this event.
These same problems
were observed by operators during the event on February 7, 1992.
Unit 2 also
had- an ONI on the "C" main steam-flow indicator- with-similar problems 'as found
on the Unit 1 indicators. See section 3.2.4.1 for further discussion of the
licensee's ONI program.
The licensee performed troubleshooting on the instruments and determined that
the Foxboro square root converters needed to be replaced on all instruments,
and the- flow transmitter and power supply needed to be replaced on the "B"
-instrument. -These square root converters provided a signal for indication
only and not for the MSIV Group 1 isolation circuitry.
Because these square
root converters were no longer available from the manufacturer, converters
from Unit 2 (in a refueling outage) were used.
The licensee stated that an
alternate converter or a new design would be examined for the Unit 2
instruments.
-The team reviewed the maintenance and calibration history for the flow
indicato~rs and determined that outstanding work requests existed for these
problems and were the reason for the ONI tags.
11
.-
,
.
.
-.
.
-
-.
-.- -
- -
.
.
-
-~_
.
- .-
-.
-.
.-.
.-
.-
,
3.2.4.1
Off Normal Instruments
The A'I determined that the licensee needed to ensure timely corrective action
Lof ON1: to ensure that instruments needing repair do not divert the operator's
attention or impac.* the operator's response to plant transients.
The instruments and equipment in the control room that are operating off-
normal are tracked on the ONI list and governed by procedure QAP 300 34, "Off-
Normal Instruments and Equipment," Revision 4.
The licensee currently logs ON!s manually and on a computer list.
When all of
the data is on the computer the official log for ONis will be a Total Job
Management (TJM) report to be kept in the control room. QAP 300-34 required
that operations categorize these items into Category I or 11 depending upon
importance and route for assignment of work request and resolution.
QAP 300-34 stated that on a monthly basis the SCRE shall notify an Operating
Engineer on ONI items that have been active longer than two weeks in order to
ensure timely repairs on instruments that can be repaired. A copy of an audit
report-dated December 21, 1991, indicated that there were 110 ONis meeting
-this criteria.
At the time of the reactor scram, there were 84 ONI items. Of
the 84 open ONIs, none were Regulatory Guide 1.97 instrumentation, three were
safety-related, and_five could be used as Emergency Operating Procedure (E0P)
equipment. The team determined that,'in all cases, adequate and redundant
means were available to provide the necessary input.
The AIT reviewed all of the control room ONIs and concluded that no safety
significant problems existed. However, several 0NIs have been outstanding for
a long time.
One-safety related ONI corresponded to a work request written in
April 1988. Another safety-related ONI, initiated in October 1991, for a
closed indication light for Recirculation Pump A suction valve, was repaired
in February 1992, by tightening a loose wire for the bulb.
Overall, the AIT considered the ONI process to be weak. The resident
inspectors have been following this issue for several months and the licensee-
was slowly working off the backlog of open items; however, greater emphasis is
needed to ensure that ONIs are addressed and resolved in a timely manner.
The
licensee stated at the exit meeting that work had been completed on an
additional 25 to 30 ONIs on Unit 1 since February 7,1992.
3.2.5
Qt.her Less S'anificant Eouipment Failurgt
The team interviewed operators and reviewed the active out-of-service list and
_the ONI list and determined that there was no indication of other significant
equipment problems with safety related or balance of plant equipment that
could have interfered with the ability of the operators to safely operate the
pl ant. A review of the sequence of events computer data showed that the "B"
' ERV was-cycling open and closed; however, other data in the control room
adequately showed that the "B" ERV did not cycle but operated correctly.
This
problem was traced to a loosely mounted acoustic monitor that resulted in
false ERV position input for the sequence of events data recorder. All other
-
. acoustic monitors were found securely attached.
12
._.
.
__
___ _ __. _ _
_ __ _
._ _ .._ _ .._.. _ ___ _ _
,
!-
4.0
Conclusions
After completing the AIT Charter, the team was able to make the following
conclusions:
1.
The root cause could not t e determined for the main steam high flow trip
signal, which caused the Group I isolation that ultimately led to the
reactor scram.
The licensee had performed reasonable analyses and
testing to determine r'ot cause.
Additional monitoring equipment to
monitor for-abnormalit..s within the main steam high flow trip system
was installed prior to Unit I start-up.
2.
The failure of the HPCI stop valve was attributed to an inadequate
maintenance work package, which was performed in february 1991. The
work package was considered inadequate because it did not include as-
found or as-left readings of the clearances between the poppet guide and
valve poppet. After welding was performed on the poppet guide assembly,
incorrect tolerances caused the valve to eventually become stuck in the
open position during HPCI testing conducted on February 6, 1992.
3.
The failure of the "C" Electromatic Relief Valve (ERV) was attributed to
brass dust on the shorting contact bar, which was caused by main steam
-system vibration.
The licensee determined that the dust originated from
brass components near the contact shorting bar.
In addition, the AIT
concluded that deficient preventive maintenance for the ERVs existed.
Although previous ERV failures indicated high resistance readings across
the contact-shorting bar, the licensee did not evaluate the possibility
of adding preventive maintenance to periodically obtain resistance
readings and clean the shorting bar.
Also, the licensee did not pursue
obtaining experience from other CECO nuclear plants that have the same
type of relief valves.
Preventive maintenance practices may have
precluded the failure of the
"C" relief valve.
4.
The apparent failure-of the Reactor feed Pumps to automatically trip at
the appropriate vessel level was attributed to instrument drift.
Because the trip setpoint for one of tne two level indicating switches
had drifted from +48 inches to +53.5 inches, neither of the RFPs would
have tripped automatically at the expected trip point. Operator actions
manually tripped the RFPs before reaching the respective trip settings;
'however, if manual actions had not been taken the pumps would have
automatically tripped at +53.5 inches.
5.
Anomalies associated with the "B", "C", and "0" main steam line flow
indicators, were attributed to a faulty power supply and square root
converters.
The control room flow indicators had "Off Normal
Instruments"_ or ONI stickers on-the front, which meant that maintenance
was needed to repair previously identified problems of false flow
indications. These erroneous flow indications occurred again after the
MSIVs:were closed during this event and resulted in personnel being sent
into the plant to look for steam leaks that didn't exist. The AIT was
concerned the operator's attention was diverted besause of indicators
that needed repair.
13
.
-
.-
.
6.
Operators performed well in mitigating the consequences of the reactor
scram and Group I isolation.
7.
The licensee's recovery from this event was thorough.
Corrective
actions to address each of the equipment failures or anomalics were
considered adequate and corrective actions to prevent recurrence were
reasonable and complete.
5.0
Exit Interview
The team met with licensee representatives (denoted in attachment 3) in a
-
public exit meeting on February 13, 1992, and summarized the purpose, AIT
charter items, and findings of the inspection.
The team discussed the likely
informational content of the inspection report with regard to documents or
processes reviewed by the team during the inspection.
The licensee did not
-
identify any such documents or processes as proprietary.
_
14
II
- __ _ __-_-
-__ _ _ _ _--
_-