IR 05000361/1990037
| ML20024F807 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 12/03/1990 |
| From: | Johnson P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML20024F806 | List: |
| References | |
| 50-361-90-37, 50-362-90-37, NUDOCS 9012210129 | |
| Download: ML20024F807 (20) | |
Text
--_
_ _ _ - _ _ _ _ _ _ _ _.
_
-
,
,
..
.
,,
,, "
t e
U.S. NUCLEAR REGULATORY COMMISSION
REGION V
Report Nos.
50-301/90-37, 50-362/90-37 Docket Nos.
50-361, 50-362 License Nos.
Southern California Edison Company Irvine Operations Center 23 Parker Street Irvine, California 92718 Facility Name:
San Onofre Units 2 and 3 Inspection at:
San Onofre, San Clemente, California Inspection conducted:
October 1, 1990 through November 15, 1990 Inspectors:
A. L. Hon, Resident Inspector C. W. Caldwell, Senior Resident Inspector
,
k7c Approved By:
P. ty./ Johnson, Chief Date Signed Reactor Projects Section 3 I_nspection Summary Inspection on October 1 through November 15, 1990 (Report Nos. 50-361/90-37,
.
50-362/90-37)
Areas Inspected: This is a special resident inspection of Units 2 and 3 operations program elements involving two licensee-identified Technical Specifications violations that involved misalignment of valves and resulted in the inoperability of important safety equipment. One violation affected the emergency core cooling (ECCS) and containment spray (CS) sub-systems and the containment integrity in Unit 3.
The second violation affected the auxiliary feedwater system in Unit 2.
Inspection procedures 30703, 61726, 71707, 71710, 90712 and 92700 were utilized.
9012210129 901205 PDR ADOCK 05000361
_
-
_
_
-
- _ _ - _ _ _ _ - _ _ _ _ _ _ - _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ - _ - _ _ - _ _ - - _ _
_ _ _ _ _
_ _ _ _
-
-
.
,
1'
'
'
..
.
.
i l-2-Results:
General Conclusions and Specific Findings:
1.
Based on independent inspection of two incidents involving impor-tant safety systems and on review of the li ensee's evaluations, the c
inspectors concluded that these two incidents involved violations of the plant's Technical Specifications (T.S.).
These incidents revealed concerns in the areas of work control, design, procedures, operator awareness of plant conditions without complete reliance on cudible annunciators, response to the discovery of misaligned components, communications, and control room operator workload.
These concerns collect"!ely contributed to the reduced attentiveness which permitted the violations to occur and/or to continue undetec-ted for the periods discussed later. The violations, in turn, resulted in significant reduction of the overall margin needed for the safe operation of the plant.
2.
The licensee's efforts after the identification of these events were found to be aggressive.
For example, the licensee's Nuclear Oversight Division was promptly and actively involved in assessing the root cause by conducting a Human Performance Evaluation. The Operations Division also showed an objective and self-critical attitude in investigating these incidents.
,
Significant Safety Matters:
Because of the misaligned valves, plant safety equipment was rendered inoperable beyond the periods allowed by T.S.
One incident involved an opened containment emergency sump outlet isolation valve which rendered a Unit 3 ECCS and CS train inoperable under certain postulated accident conditions. This open valve also established a flow path between the containment and the outside environment. The other incident involved
.
starting up Unit 2 with a steam trap isolated, resulting in an accumula-tion of condensation in the: steam line to an auxiliary feedwater (AFW)
pump. This condition rendered the AFW pump inoperable, as indicated by repeated tripping on overspeed during subsequent surveillance tests.
Based on best estimate modeling and on consideration of other plant conditions, the licensee concluded that the occurrence of a design basis accident during these periods, assuming no other equipment fail-ures, would not have resulted in consequences more severe than those determined for previously analyzed accidents. Thus, there was no actual impact on public health and safety.
Summary of Violations:
A.
Due to the containment emergency sump outlet isolation valve 3HV-9302 being open between 4:52 p.m. on September 24, 1990 and l
l
- - - _ _. _ - _ _ - _ - _ - _ _ - _ _ _ _ _ _ _ - _ _ _ _ _ _ - - _ _ _ _ _ - _ _ _ _ _
_ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ -
_ _ _ - _ _ _ _ _ _ _ - _ _
-
.
,
. ',
-
..
,
e-3-10:52 a.m. on September 28, 1990 while Unit 3 was in Mode 1 Action requirements of T.S. 3.5.2, "ECCS Subsystems," and 3.6.2, "Contain-ment Spray System," and the requirements of T.S. 3.6.1, "Contain-ment Integrity," appear to have been violated between 4:52 p.m. on September 24, 1990 and 9:00 a.m. on September 28, 1990.
B.
An isolated steam trap rendered the Steam Driven AFW Pump, 2P-140, inoperable between 1:40 p.m. on August 24 and 2:00 p.m. on October 21, 1990 while Unit 2 was in Modes 1, 2, and 3.
As a result, T.S. 3.7.1.2, " Auxiliary Feedwater System", Action a.
appears to have been violated, in addi tion, T.S. 3.8.1.1, "A.C.
Sources," Action c.2., also appears to have been violated during concurrent emergency diesel generator (EDG) outages between 1:20 p.m. on September 5 and 5:00 a.m. on September 7,1990 and between 12:05 p.m. on September 19 and 3:05 a.m. on September 22, 1990.
.
G
--
_ - _ - - - -
.
, -
.
._
.
_.
!
'
.
,
,-
.
.
.
,
,
P
'
DETAILS
1.
Persons Contacted Southern California Edison Company H. Ray, Senior Vice President, Nuclear
- H. Morgan, Vice President and Site Manager
- R. Krieger, Station Manager
- B. Katz, Nuclear Oversight Manager
- K. Slagle, Deputy Station Manager
- R. Waldo, Operations Manager
- L. Cash, Maintenance Manager
- M. Short, Technical Manager M. Wharton, Nuclear Design Engineering Manager
- P. Knapp, Health Physics Manager
- D. Herbst, Quality Assurance Manager C. Chiu, Quality Engineering Manager
- V. Fisher Operations Superintendent,- Units 2/3
-
- R. Rosenblum, Manager, Nuclear Regulatory Affairs
_
- D._Brevig, Supervisor, Onsite Nuclear Licensing
- R. P1appert Compliance Manager
- M.-McDevitt, Unit 2/3 Analysis Group Supervisor San'Diego Gas'and' Electric Company
- R. Erickson, Site Representative
-
- Denotes those attending the exit meeting on November 15, 1990.
The inspectors also contacted other licensee employees during the course of the inspection,. including operations shift superintendents, control room supervisors, control room operators Nuclear Oversight engineers',
i compliance engineers and. system engineers.
'
2.
Introduction-This report addresses two licensee-identified T.S. violations involving plant operations..They involved misalignment of; valves that caused-important' safety equipment to be or become inoperable. One violation affected the~ECCS and Core Spray-(CS) subbsystem and containment-integ-
-rityLin Unit 3; the second violation affected the auxiliary feedwater system in= Unit 2.
The licensee initiated safety assessments, root cause
- determinations, and corrective actions following the discovery of these apparent: violations.
The results of the licensee's efforts were reported in Licensee Event Reports =(LER) submitted pursuant to the LER 50-362/90-10 was. submitted during
-
requirements of 10 CFR 50.73. ~
50-361/90-12 was this inspection period for. the Unit '3 incident, and LER-submitted after this inspection period-for the Unit 2 incident.
l
.
-
- - -,,
.
..
..
.. -
.,
,
,
,.
During this inspection, the inspector reviewed the licensee's efforts in response to these incidents.
In addition, the inspector independently-assessed selected areas for other contributing factors. A review of recent enforcement history was also conducted to determine whether similar violations had occurred; none were identified.
-3.
Misalignment of = Containment Emergency Sump Isolation Valve 3HV-9302 (Unit 3)
a.
Summary On September 24, 1990, while the Unit was operating at full power, 3HV-9302 was opened due to an inadvertent partial actuation of the Recirculation Actuation System (RAS). This condition remained undetected for 95 hours0.0011 days <br />0.0264 hours <br />1.570767e-4 weeks <br />3.61475e-5 months <br /> (while the Unit was in Mode 1) until
'
September _ 28, 1990, when it was recognized by an operator.
This
-
valve's being open under certain accident conditions could render the Train B ECCS and containment spray (CS) system inoperable due to loss of suction from the refueling water storage tank (RWST).
It also established a flow path from the containment through the.
ECCS and CS mini-flow recirculation lines to the RWST, which is vented to the outside environment.- This condition. appears to have violated T.S. 3.5.2 for the ECCS and 3.6.2 for the CS, both of which have 72-hour action requirements, and T.S. 3.6.1.1,
,
'
" Containment Integrity", which has a 6-hour action requirement.
The licensee evaluated the safety. significance of this condition with respect to ECCS capability, containment pressure control and
,
-radiological-release. The licensee concluded that the safety consequence was. bounded by previous safety analyses. Nevertheless,-
the licensee recognized the-importance of this event and promptly initiatedia root cause investigation:and corrective-actions.
Some
.
of the causes identified by the licensee were:
inadequate work-control, weak communications, weakness in system design for sing 1_e_
y-failure challenges to containment. integrity and ECCS reliability. -
-
failure of annunciator: design to include an important valve such as
HV-9302,: and weakness in the routine: surveillance program to monitor the status of this valve.
b!.
Background (1)-SystemOperationCharacteristic Units 2 and 3 have-two ECCS and CS trains.
Each train-includes the foilowing:
One High Pressure Safety Injection (HPSI) pump, plus a
-
--third pump which can be aligned to either train (HPSI pumps start automatically on a-Safety Injection Actuation Signal (SIAS)).
- >
i'
One Low Pressure Safety Injection System (LPSI) pump
-
L-(also starts automatically on a SIAS).
L
.
l
,
,
--
.
-
-,
n
. - - - -
- - - -
.
-
.
.
..
..
- -. -
,:
'
'...
3'
<^
,
,
..
- .
One Containment Spray Pump which is started by a CS
>
-
Actuation Signal (CSAS).
During.the injection phase, these pumps draw water from the
.
refuelingwater.storagetank(RWST).
Upon a SIAS, HPSI and LPSI start but recirculate to the RWST through mini-flow recir-culation valves until Reactor Coolant System (RCS) pressure decreases to the discharge head of the pumps. When the RWST J
1evel reaches 18%, a Recirculation Actuation Signal (RAS).is
,
initiated to autunatically shift the ECCS and CS sub-systems
-
to the recirculation phase. The RAS:
(a)stopsbothLPSI ps, (b) closes-the suction isolation valves from the RWST,.
opens the containment emergency sump isolation valves, and closes the HPSI and CS mini-flow recirculation valves when the emergency sump level reaches six feet.
The valve status in the control room is indicated by back-lichted pushbutton switches -- green for closed and red for
open.
To aid operators'during routine status checks, red and'
green colored dots had been affixed near the control panel
.
switches to indicate their normal and expected positions-during power operation. HV-9302 and HV-9304 switches and indicators are-located on the horizontal board and are not part of the containment integrity status panel, which.is located on the vertical boards.
Before this event, the licensee had elected to maintain-the inboard isolation valve (HV-9304) for the sump outletinormally
-
open while maintaining the outboard isolation valve (HV-9302)
~
normally closed, a valve alignment which is similar to that nonnally used at some other PWR~ facilities.
The ECCS and CS pum) motors and' the containment emergency fan coolers are cooled )y the Component-Cooling Water (CCW)
system, which in turn is cooled by salt water passing through thecomponentcoolingwaterheatexchanger(CCWHX).
&
c.-
Chronology
>
The inspector's review determined the chronolegy of this incident
<to have been as follows:
_
Date/ Time Event 9/24 10:52 While an identification' label, was being changed on an ESF auxiliary relay cabinet, a-power supply _
circuit breaker for the Train B RAS' logic was bumped and then-reset by the worker. This caused'a partial (Train B)' RAS actuation and' opened the emergency, sump outlet isolation valve 3HV-9302. No audible.
alarm was received on the main annunciator panel in the control room.
.
.
. -. -
-
.
.1
.
..b
'
.
,,
s.
Date/ Time'
Event 9/24 16:52 T.S. 3.6.1.1 -- 6-hour Action for loss of. contain-ment integrity was apparently exceeded.
9/27-10:52 T.S. 3.5.3 and 3.6.2 -- 72-hour Actions
.e ECCS and CS subsystems were apparently exceeded.
9/28 09:00 3HV-9302 was discovered to be open and was promptly [
closed by an operator. The licensee determined that, during these 95 hours0.0011 days <br />0.0264 hours <br />1.570767e-4 weeks <br />3.61475e-5 months <br />, Train B ECCS and CS could have been inoperable in the event of certain accident-conditions.
In addition, salt water to the Train A CCW HX was isolated seven times during this period for up to 23 minutes each time.
This was-done to reverse the salt water flow to flush the CCW HX in_ order to improve its performance.
10/5 16:JO During evaluation of the valve misalignment, the licensee recognized that a potentially degraded containment integrity condition might have existed; i.e. a pathway from the containment to the outside environment through the opened 3HV-9302 and the ECCS and CS system recirculation mini-flow valves.
,
d.-
Licensee's Safety Assessment The licensee evalt.ated the safety significance of the misaligned
-
- valve 3HV-9302.
Two major concerns addressed were the potential consequence _of partial ECCS and CS unavailability and the degraded
'
containment integrity.
"
With respect to ECCS and CS availability, the licensee postulated that for a Loss of Coolant Accident (LOCA) or a Main Steam Line Break (MSLB)_ having-a break size greater than 0.06 square foot, containment pressure could exceed that of the RWST water static
' head and thus back seat the Train B RWST outlet check valve. The
,
Train B ECCS and CS would-then take suction from the emergency sump, which would not yet have sufficient water level to assure adequate Net Positive Suction--Head (NPSH) for these pumps. Thus, these pumps would become air bound and fail due to excessive vibration, leading to bearing seizure and seal failure. However, the Train A ECCS and CS sub-system would still have performed its
safety function, even.during the brief periods when the Train A CCW HX was not in service.
The licensee's purchase order.to the vendor
,
for the ECCS/CS pump motors specified that "If the wa'ter has not'
L been drained from the heat exchanger, the motor can operate 30 L
minutes starting cold...."
The longest duration of the CCW HX l
outage was 23 minutes, and the licensee concluded-that CCW could l
have been restored even socner by the operators, if needed.
Regarding containment pressure and cooling, the licensee evaluated a MSLB (the most limiting containment pressure event) and a LOCA.
L
-
,
,
-
_
.- ---
____
- __
_ _ _ _ - _ _ - _ _ _ _ _ _ _ _ _
.-
,
,
..
,
?
,
The licensee determined that although the Train B CS (due to 3HV-9302 being misaligned) and the Train A containment cooler (due to the aforementioned CCW outages) might not have been available,
the containment peak pressure would still have been limited to the 55.7 psig indicated in the S6fety ralysis, based upon operation of the Train A CS and the Train B containment cooler (which would have beenunaffected).
Furthermore, even with the most conservative assumption of a complete loss of both CS trains in the event of a MSLB, operation of the Train B containment cooler would limit the peak containment pressure to approximately 57.3 psig, which is less than the containment design limit of 60 psig.
With respect to the potential consequence of degraded containment integrity due to the path through 3HV-9302 and the mini-flow recirculation valves, the licensee estimated realistically an additional thyroid dose of 20 Rom at the Exclusion Area Boundary, which is bounded by the FSAR design basis assumption of 90.3 Rem.
Similarly, the control room and Low population Zone doses would also be bounded by the previous safety analysis.
Therefore, the licensee concluded that the consequences of this valve misalignment would not have ext.eeded those previously analyzed and found acceptable, e.
Licensee's Investigation and Corrective Action Following the discovery of the misalignment of 3HV-9302, the licensee innediately initiated 4 Division Incident Investigation oy
-
the Operations Division and a Human performance Evaluation by the Nuclear Oversight Safety Engineering Group. The licensee sent two letters to the NRC (dated October 9 and 29,1990) to acknowledge the importance of this incident and to outline corrective actions initiated.
The following were identified and reported 'n the associated LER:
Details, of the Event The following detail is an excerpt from LER 50-361/90-10:
"On September 24, 1990, proposed installation of new color-coded labels associated with a Unit 3 ESF auxiliary relay cabinet was briefly reviewed in a meeting (referred to as a
"tailboard")betweentheworker(utility,nonlicensed)and theShiftSuperintendent(SS)(utility, licensed). During the tailboard, the worker indicated to the SS that he would b6 installing new labels in the cabinets. Assuming that the Unit 3 Control Room Operator (CRO) (utility, licensed) would do a thorough review of the proposed work, the SS cautioned the worker that in case of inadvertent circuit breaker o>eration in the cabinet, he should not reset any circuit brea(ers and should notify the control room. Later that day while seeking approval to proceed with the installation, the worker indicated to the CR0 that the label installation had been
. _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
-._- _ _ _ _ __ _ _ _ - _ _ _ - _ _ _ - _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ - - _ _ _ _ _ _ - _ _ _ _.
'
- '
,,
..
..
..
.
discussed with the SS.
The CR0 then cautioned the worker to
be careful and approved the work.
"In neither the meeting with the $$ nor with the CR0 did the
,
j worker conraunicate the fact that he planned to remove the
>
existing labels prior to installing the new labels. In addition, neither the SS nor the CR0 questioned the worker
,
about removal of the existing labels.
(Label removal involves
'
'
L a significantly higher risk than simple label installation due I
to: 1) the proximity of the labels to the circuit breakers and
,
2) the use of tools and substantial force that is required to
"
remove the old labels). Since the SS and the CO understood
'
only that new labels were being installed, they approved the
-
relabeling without completely assessing the risks of the work inside a cabinet with known hazards (i.e., plant transient or trip). As a result of inadequate communications in the work a proval process, consideration was not given to either:
'
- 1 providing supervision appropriate for the work, or 2 deferring the work (consistent with management efforts) to
.
minimize the potential for a plant trip or transients.
<
- While removing a label, the worker's finger caught a circuit breaker lever moving it in the open direction.. The worker
L restored the circuit breaker to its fully closed position.
,,
i
"The movement of the circuit breaker was sufficient to I
deenergize portions of the RAS circuitry, thus causing
-
. 3HV-9302 to open. The worker then req'lesteJ that.the CR0 check the ESF relay cabinet to make.sure it4 status was correct since he may have bumped a circuit treaker. However, the worker did not advise anyone that he hg reclosed the
,
'
circuit breaker. Two licensed operators promtly checked the '
status indications in the ESF auxiliary relay cabinet and
.
correctly determined that all were hormal and proper. Unaware i
that the worker had partially opened and then reclosed the
.;
- circuit breaker and in the absence of.any audible alarm
,
indication, the control rnom operators did not further check-
for other evidence'of a change in plant status..A1+ hough no-audible alann indication was received, indication ot the
'
change.of valve position occurred at: 1) the control room panel valve position controller; 2) the valve position status indication on the Critical' Functions Monitorin System (CFMS),
l and, 3) the Plant Monitoring System (PMS) log.g'
Cause of Event The licensee's investigation identified the following causes associated with this event:
,
Inadequate work authorization, supervision and communication
-
allowed the work to proceed without proper recognition of-risks.
l (
.
I P
~-
=
w r-wr-a-,w-m--,,.m
-,.,%_,-..my.m_,,
,_,._,,,m
_ _,
,,, __ _
,
,
,
.
-..
.
.
, ',
7
.
..
.
During relabeling, when the worker accidentally bumped the
-
circuit breaker, he reclosed it (contrary to instruction).
Moreover, he did not adequately inform the control operators of his actions, including reciosing of the breaker.
The design of this system did not provide an audible alarm in
-
the control room alarm window panels to alert the operators to the partial RAS actuation.
The valve position status light was changed from green to red on the control board (which contains many other similar switches), and was registered by the Critical Function Monitoring System and the Plant Monitoring System computer display screen (which contains the status of many other valves and breakers).
Neither the significance nor the potential for misalignment of
-
a normally closed containment emergency sump isolation valve was recognized when the ESF component mo:.;toring program was developed.
For example, the shiftly checklist for critical valves, such as those identified in the T.S., did not include HV-9302.
While evaluating this incident, the licensee also noted that the design of the RAS control circuitry was susceptible to a single failure.
Because the control relays for both the inboard and outboard sump outlet valves were on the same power supply, a single spurious power supply failure could open both valves.
"
Corrective Actions As corrective action, the licenseet Revised the routine shiftly checklist to include verification
-
of the containment emergency sump outlet valve positions.
Disciplined the individuals involved.
-
>
.
Temporarily closed the inboard valve in addition to the out-
-
board valve.
A 10 CFR 50.59 evaluation was performed to assure the acceptability of this change.
.
Modified the RAS control circuitry so that the control relays
-
for the two valves are on separate power supplies, to prevent spurious opening of both sump suction valves as a result of a single failure.
(This-was completed on both Units 2 and 3.)
In addition, the licensee planned to:
Provide an audible alarm for opened sump isolation valves and
-
closed ECCS and CS pump mini-flow recirculation valves.
In addition, an annunciator window will be provided in the long term.
Evaluate the PMS for means to enhance the visibility of
-
critical valve status.
... __
.
-
--
_. _.
i
-
.
,
. ',
..
,
-
,.
,
Identify other critical ECCS and CS valves not having audible
-
.
annunci6 tion of misalignment which could affect system operability.
Propose a T.S. change to include these valvt.s so that they
-
will be covered by an appropriate surveillar.ca test.
Review lessons learned with the operations staff and other
-
cognizant people, with ertphasis on the importance of com-pletely understanding the scope of proposed work and its potential risk.
.
Add a requirement for an independent check for all work, while
-
the plant is in power operation, on systems and components that can cause plant transients.
Review and improve the procedures and policies used by control
-
,
room staff for on shift equipment stetus checks.
Detennine the optimal position of the inboard sump isolation
-
valves. This effort will consider the balance of RAS
-
reliability and containment integrity from a probabilistic risk analysis (PRA) perspective. -
,
~ f.
NRC Conclusions-The following conclusions resulted from the inspector's independent assessment and review of the licensee's investigation of this
event:
Emergency sump isolation valve 3HV-9302 was inadvertently
-
isolated because insufficient control was exercised by licensed personnel over activities by non-licensed personnel which affected important safety equipment.
'
After the isolation occurred, insufficient evaluation was
-
,
performed to ensure that RAS alignment was properly returned to normal.
Insufficient attention was given to-impor'. ant panel indica-
-
tions during the 95 hours0.0011 days <br />0.0264 hours <br />1.570767e-4 weeks <br />3.61475e-5 months <br /> 3HV-9302 was open. Twelve shift turnovers occurred during this period. Panel walkdowns during these turnovers, along with routine review of indications at other times, should have more promptly identified the open
'
' isolation valve. Other activities which appear to have
-
contributed to reduced operator.' attentiveness are discussed in paragraph 5.
-Train B of the ECCS and CS subsystems were inoperable during
-
-the 95 hours0.0011 days <br />0.0264 hours <br />1.570767e-4 weeks <br />3.61475e-5 months <br /> while'3HV-9302 was open, and Unit 3 containment integrity was compromised.
!
!
!
L l
-. _
-
._, _..
.
_
.
.
.
..
...
. -
..
...
..
...
.
.
.
- _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
__-___ _ _ _ _ _ _ _ _ _ _. - _ - _ _ _ _ _ _ _ _ _ _ _ _
_ _ _.__ _ _ _ _ ___.__._ -_.
-
.
'
..
.,
,
,
J s-
4.
Steam Driven AFW Pump Inoperable Due To Overspeed Trip (tinit 2)
7o i
'
a.
Summary Unit 2 was taken to cold shutdown (Mode 5) in July 1990 to inspect and repair-the steam generator feed rings. High pressure steam
,
drain traps associated with the steam supply lines to the steam-driven auxiliary feedwater pump (SDAFWP) were isolated durirg this
'
outage to prevent loss of nitrogen while conducting nitrogen sparging of the steam generators. The SDAFWP, 2P-140, was declared operable during the subsequent Unit 2 restart on August 27, 1990, even though the drain traps had inadvertently not been unisolated.
One isolated trap was found and unisolated on September 25, but the other one remained isolated until October 21. _This resulted in
,
condensate buildup in the steam supply line, which caused the SDAFWP to trip on overspeed during several surveillance tests conducted during this period. This condition rendered 2P-140
-
"
inoperable during the period between August 27 and October 21, 1990, when the licensee identified the isolated steam trap.
T.S.
7.3.1.2, Action a., appears to have been violated during this period while the Unit was in Mode 1.
Because the emergency diesel ccr.erators (EDGs) (which supply backup power to the motor driven AFW pumps) had been removed from service at certain times during
'
i this period, T.S. 3.8.1.1, Action c.2, also appears to have been vir.ated during these times.
'
'
The licensee apparently missed at least two opportunities to identify the isolated steam trap. One occurred on September 26.
-
1990 when the licensee discovered that a similar-steam trap was isolated on the other steam line to the Unit 2 SDAFWP but failed to follow up on the root cause and assess-the pump's operability.. The
>
other opportunity was between October 6 and 21,1990, when 2P-140 tripped three times on overspeed. Troubleshooting efforts during.
this period included an alignment check of steam traps, but this misaligned steam trap was missed.
,
!
' '
b.
Background (1) Auxiliary Feedwater (AFW) System Operational Characteristicsi
~
SDAFWP -- Powered by a Terry Turbine single stage,
-
non-condensing steam turbine. -It can operate with steam inlet pressure ranging from 1200 psia to 65 psia.
The SDAFWP is actuated automatically by the emergency
-
,
M feedwater actuation system (EFAS), on EFAS-1 (for SG
h E-089)or.EFAS-2(forSGE-088).
It provides a backup for both motor driven AFW pumps- (MDAFWPs) P-141 (dedicated to SG E089, with emergency power from EDG 2G002) and P 504 (dedicated to SG E088, with emergency powerfromEDG2G003).
,
-
%,~
%-
y
,..h,,,,
, -. -,
,.,..
v__,__,,
,,,-y,,,y.
.,,.. _..,
n
.y.
_
,,%_.,,_,,.v..
_.
e
. _ _ _ _
.m., _ _,
_. -, _,,.,...
,e.,
.
.
. - - _ -. - -.._ - - -
-_
-
.
'
'..
-
.,
,
,
..
.
,
,
,
I Steam supply (from SG E 088) isolation valve 2HV-8201 is
-
normally closed to limit wear on check valve 521301MU003, caused by pressure differential between the two steam
.
generators.
(RefertoFigure1,enclosedwiththis
report).
Continuous condensate removal lines are connected
-
upstream and downstream of steam stop valve HV-4716 (Figure 1)
j
,
!
\\
Overspeed trip was set at 110% of the 3570 RPM nominal l
"
-
speed.
Applicable T.S. requirements while in Modes 1, 2, and 3:
-
- 3.7.1.2 Actions a and b:
one pump inoperable - 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> shutdown
.
two pumps inoperable - 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> shutdown 3.1.1.1, Action c.2:
with one EDG and P-140 Inoperable - 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> shutdown (2) Normal Plant Startup from an Outage-Procedures used for normal plant startup contain several steps which address restoration of system'1tneup after nitrogen sparging activities, as follows:
.
Step 6.S.L.1 of procedure 5023-5-1.8(TCN3-19)," Shut-
-
down Operation " directed operators to place steam
,
generators (SGs)intherecommendedconditionsof
$023-2-18, one of which is Attachments'6 and 7. " Nitrogen Blanketing or Sparging."
' Step 2.1.2.3 of S023-2-18 (TCN 6-7), " Draining and Refill-
-
-
ing Steam Generators," Attachments 6 and 7. " Remove SG'
from service," specified the closure of steam trap isola-
,
tion valves MU1257 and MU1258. Attachment 8 " Restoration
from SG Nitrogen Blanketing or Sparging" was provided for
.
securing from this evolution.
Step 6.1.5of5023-5-1.3(TCN13-1),"PlantStartupfrom
-
Cold Shutdown to Hot Standby," stated, " Align the main steam leads per S022-2-9".
Step 6.2.2 of 5023-2-9, " Placing Main Steam-(MS) Leads in
-
Service," specified the alignment of MS leads upstream of
~
the MSIV per Attachment 1 to the' procedure.
Steps 2.1.25
.and 2.1.45 of Attachment l'specified that steam trap.
l isolation valves MU1258 and MU1257, respectively, be opened and verified independently.
'
\\
Step'6.14 of S023-5-1.3 specified to reverify 6.1.5.-
l.
-
l l
!
o
.-,,.,..,.....,, _ -... _ _. _ _ _ _,
.,,.. _
.
.
-.
.
.
- _ _ _ _ _ _.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
. _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _.
_ _ _ _ _ _ _ _ - _ _ _ _ _ _ _
.
-
,
!
,'l
.
,
,,
,
Step 6.14.6 specified the establishment of condenser
-
vacuum.
During the Unit restart on August 27, however, as discussed further below, certain procedure steps were omitted (afttr management approval to do so was obtained). These steps were omitted because condenser vacuum had been maintained during the outage, and did not have to be reestablished, c.
Qronology Inspector review determined the following chronology of events:
Date Time Event 07/28 Unit 2 was shut down to repair the SG feed rings. Condenser vacuum was maintained during this outage.
08/23 14:00 The Unit was restarted per S023-5-1.3.
Because the condenser vacuum was maintained during the outage, step 6.1.5 was deleted.
This was dor,e in accordance with 50123-0-20(TCN0-5),"Use of Procedures," Attachment 4. " Procedure Modification Permit (PMP)."
It was reviewed and approved by the SRO, the Assistant Unit Operation Superintendent, and the Manager of Operations.
-
A nitrogen (N2) sparge and blanket were applied to the SGs twice per S023-2-18, Attachment 6.
This involved closing steam trap isolation valves MU1257 and MU1258.
08/24 13:40 N2 sparge was secured per $023-2-18 Attach-ment 8, but the procedure did not direct the operator to reopen MU1257 and MU1258.
'
08/26 22:25 The Unit entered Mode 3.
08/27 17:05 Af ter completion of approximately two days of continuous steam line blowdown, P-140 was satisfactorily tested and declared operable.
08/29 01:20 The Unit entered Mode 1.
08/31 04:25 The time limit of T.S. 3.7.1.2, Action a. was apparently exceeded.
09/05 05:20 EDG 2G003 was taken out of service for maintenance.
13:20 The 8-hour time limit of T.S. 3.8.1.1 Action c.2. was apparently exceeded.
,
_ _ _ _ - _ _ _
-_-
_
_ _ _ _
__-____
_ _ _ _ _ _ _ - - _ _ - _ _ _ _ _ _ _
'
.
.
'
- :,
'
-
.
,
Date Time Event
'
09/07 05:00 EDG2G003wasreturnedtoservice(exited T.S.3.0.1.1,Actionc.2.).
09/19 04:05 EDG 2G002 was taken out of service for maintenance.
09/19 12:05 The 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> time limit of T.S. 3.8.1.1,
-
Action c.2. was apparently exceeded.
09/22 03:05 EDG2G002wasreturnedtoservice(exited T.S.3.8.1.1,Actionc.2.).
09/25 P-140 was taken out of service for maintenance.
<
During restoration, steam trap isolation valve-
"
itV1258 was found closed.
It was reopened by the operator, as discussed further in paragraph 4.e.
09/26 11:20 After continuous steam line blowdown, P-140 was started and run satisfactorily for an Inservice Test (IST).
10/06 08:38 P-140 was started for-stroke test of the steam stop valve 2HV-4716.
P-140 tripped on overspeed and was declared inoperable.
It was manually started six times later that day
.
without tripping. Subscquent troubleshooting included a check of AFW system valve alignment, but this did not identify that steam trap isolation valve MU1257 was closed.
10/07 13:25 F-140 was started for IST and was declared operable after a satisfactory run. A daily test schedule was initiated because the over.
speed trip could not be reproduced.
.
10/8-11 P-140 was started daily without.any overspeed
,
trips. Test frequency was reduced to weekly.
10/16 10:00 Preventative Maintenance was initiated on HV-4714 (valve to E-008 from MDAFWP P-504).
P-504 was: considered inoperable and T.S Action 3.7.1.2.a. (72-hour shutdown) was entered.
14:20 SDAFWP P-140 was started for the weekly test..
It tripped on overspeed, was declared inoper-able, and T.S. Action 3.7.1.2.b. (6-hour shut-down.:with P-504 already inoperable) was entered.
15:54 Preventive maintenance was completed on 2HV-4714; P-504 was returned to-service and the 6-hcur Action was exited. 'The 72-hour Action remained
'
in effect (P-140 still inoperable).
I.
-
.x__-
. _
,., -.
- _,, _. _,,
_ _. - - _
., - -.. -.,
.
~
_, _ _.
-
_
_. _ _ _. _. _ _ _. _,
'
,
'
f.,
. '.
,
.-
)
Date Time Event
,
10/17 P-140 was started four times and tripped once due to problems with attached instrumentation.
10/19 08:20 After troubleshooting attempts. P-140 was started and the IST was satisfactory completed.
It was declared operable, and a daily test I
schedule was initiated.
10/20 P-140 was started and run successfully three times.
'
10/21 09:35 P-140 tripped on overspeed during the daily test.
Excessive water was noted at the steam exhaust. Operations was requested by Station Technical to verify steam trap alignment.
.
14:00 Operations found MV1257 closed and reopened it.
10/23 03:15 P-140 was started and run satisfactorily and j
was declared operable,
'
,
i 11/06 After extensive evaluation, the licensee concluded that P-140 was inoperable between August 27 and October 21, 1990 while the steam-trap was isolated, since it-could not= be assured that P-140 would not trip on overspeed
..
-
following an emergency actuation, d.
Safety Assessment
50AFWP P-140-is a diverse backup.to the MDAFWPs, one of which is i
dedicated to each SG. During the EDG outages, the associated MDAFWP could not have operated in the event of a design basis accident (i.e., loss of offsite power with MSLB, main feedwater line j
break (Mr;.8),orSGtuberupture-(SGTR)). The worst case scenario
- for a complete loss of AFW is as follows:
l-l'
Affected SG i
E-088 E-089 EDG 2G002 outage X
EDG 2G003 outage X
!
'
x = loss of AFW to the intact SG (in the event of loss of offs 1te-power)
The licensee completed the safety assessment of this incident and i
!
documented the results in LER 50-361/90-12, dated November 20, 1990..
(aftertheendofthisinspectionperiod),
i
<
'
.
._ _ __
.
- _ _ _.
._
.
..
_
._ _.
1
.
,,
Based on best estimate analysis, the licensee determined that even in the above worst-case scenario, the expected reduction in RCS temperature would prevent RCS overpressurization.
This is because remaining inventory in the affected SG inventory would evaporate gradually, and sufficient water inventory would remain in the intact SG. This condition would continue for at least 30 minutes after which the operator was expected to be able to reset the over-
,
speed trip, restart P-140 and thereby restore AFW to the intact SG.
'
With respect to the SGTR accident, the licensee detemined that the start of the cooldown would be delayed until P-140 is reset and i
restarted. This would result in a slightly higher mass transfer from the primary to the secondary and ultimately to the atmosphere than previously assumed in the FSAR. However, the RCS specific activities during the period of SDAFWP inoperability were less than 1% of that assumed in the FSAR.
Thus, the potential radioactive releases were bounded by the FSAR and were below the regulatory
,
limit.
Therefore, the licensee concluded that there would have been no safety consequence due to the inoperability of P-140 during this period, e.
Cause Analysis-The licensee initiated a root cause detennination via an Operations Division Investigation (0DI).
This ODI identified the causal factors which follow. The inspector monitored licensee activities
-
and reached the same conclusions after cor. ducting en independent assessment of this incident.
At the conclusion of the feed ring repair outage, the Unit was-
-
started up per 5023-5-1,3(TCN13-1),"PlantStartupfromCold Shutdown to Hot Standby." As part of the startup process, to purge oxygen from the steam generators, a N2 sparge and blanket wereappliedtwiceperS023-2-18(TCN6-7),"Drainingand
,
Refilling of Steam Generators," Attachment 6.
This involved closing steam trap isolation valves MU1257 and MU1258.
Because condenser vacuum had been maintained during the outage, step 6.1.5 (alignment check'of main steam leads) of $023-5-1.3 was deleted. -This deletion was done in accordance with proce-dure S0123-0-20 (TCN 0-5), "Use of Procedures," Attachment-4,
" Procedure Modification Permit'(PMP)." It was reviewed and approved by the SRO, the Assistant Unit Operation Superinten-dent, and the Manager of Operations.
Completion of. step 6.1.5 would likely have identified the closed isolation valves MU1257 and 1258.
5021-2-18 - Attachment 8, " Restoration from SG Nitrogen Blanket-
-
ing or Sparging," was used to secure from the N2 sparging.
i evolution, but it did not specify that MU1257 and MU1258 be
,
reopened. Step 2.2.3.3, " Return SG to service," of Attachments i
.
-
g c-
+e mg-1 +
-
a-,
. +.
,
.'
.
.
,
9 and 10, "SG Removal / Return to Service," specified that MU1257 and MU1258 be opened and verified independently. Completion of this step would have reopened the steam drain isolation valves; however, the startup procedure did not direct that Attachments 9 and 10 be accomplished.
When the closed steam trap isolation valve, MU1258, was
-
discovered on 9/26/90, no formal root cause or operability evaluation was perfonned. This resulted in a missed oppor-tunity to detect the misaligned valve MU1257.
Upon discovery of MU1258 status, the SR0 checked Unit 3 for a similar problem, but he did not check the SG E-088 side in Unit 2 because of a slight difference between the two steam line configurations.
He phoned the Assistant Unit Superin-tendent with this and another more pressing plant problem (a main feedwater pump lube oil pump tripping) that night.
However, no log entry was made and no memo was written to the PlantSuperintendentforfollowup(asprescribedbyStep 6.6.2.5 of S0123-0-23 on discovery of system misalignments).
On the other hand, he did take the initiative to review recent work authorization requests (WARS) for work performed on related systems. But the review did not result in identifying the reason MU1258 was closed, and no othee followup actions were initiated.
Furthennore, 50123-0-23 dld not specify that an operability assessment be made upon discovery of misaligned valves. As in this case, it was not done &nd the information was not passed to the AfW system engineers.
-
Another missed opportunity occurred during the troubleshooting
-
between October 6 and 21, 1990 when P-140 tripped three times due to overspeed. Although the condensation in the steam line was identified as a potential candidate, the engineer requested that Operations check the alignment without a clearly defined scope of the boundary. As a result, the steam trap alignment check was inadequately performed. Operations
,
checked the alignment up to HV-8200 and HV-8201 -- an arti-ficial boundary which is downstream of MU1258 and MU1257.
Equipment operators failed to identify MU-1257 as being closed
-
I on their rounds, even though it is in a fairly prominent location.
The overspeed trip setpoint was set at 110% of the nominal
-
operating speed of the pump. This left the pump with a j
relatively narrow margin to respond to condensation in steam
,
supply lines. The vendor believed that 125% would assure
'
sufficient margin, while still providing proper overspeed protection.
l l
Additional details regarding this event are provided in LER 50-361/90-012.
l t
.
- --.-
- - -
_ _ _.. -.
.
-. -
-. -_
._
._
-.--
'
,
s
..
,-
'
.-
,
,
.
L f.
NRC Conclusions The following conclusions resulted from the inspector's independent
,
assessment and from redew of the licensee's investigation of this
"
"
event:
Steam drain traps associated with the steam lines to the
-
l Unit 2 SDAFWP were isolated to conduct nitrogen sparging activities during the feed ring outage in August 1990.
Because necessary procedure steps were omitted during. plant
-
restart, apparently without sufficient review,-the steam drains were not properly realigned during the subsequent Unit 2 restart on August 27, 1990.
The steam trap for one steam line was found isolated and was
-
.
returned to service on September 25, 1990.
,
Several overspeed trips of the SDAFWP were experienced before i
-
the steam trap for the other steam line was found isolated and properly aligned on October 21, 1990. The SDAFWP was inoper-able during the intervening period (August 27 through October
.
21,1990).
'
At least two opportunities existed for the licensee to find
-
,
and correct the misalignment of steam trap isolation valve MU1257 -.on September 25 when MU1258 was found closed, and on October 16, af ter an additional overspeed trip was
-
experienced.
5.
Other Factors Contributing to Operator Perfonnance
- j Based on routine observations of control room activities and informal discussion with the operators, the inspector perceived that an excessive workload has been placed on the operators, even while the Units were at full power. At the exit meeting, the inspector questioned.whether the
,~.
workload was appropriate for the. operators, and whether it still allowed them sufficient time to monitor equipment status without placing excessive reliance on audible alarms and checklists. :The licensee acknowledged the inspector's concern and stated that a task had already been initiated to look into this concern, _as a result of previous
'
feedback from the operators.
6.
Exit Meeting-On November 15, 1990 an exit meeting was conducted with.the licensee representatives identified in Paragraph 1.
The inspectors summarized the inspection scope and findings as described in the Results section of.
this report.-
The licensee acknowledged the inspection findings and noted that appropriate corrective actions would be implemented where warranted.
1; The licensee did not identify as proprietary any of the information
'
provided to or reviewed by the. inspectors during this inspection.
,_
L
.-
_
-. _ _., _ _ _ _~_ _ _ _. _..
-..
_. -
_,
. -.
-.
....__._..__....-....~~.-....~_.-~~_.u-_-~.-..--.-
.. -... -
-
-u-_
-
.~.....-,.me.
m
.
-
!
.
l
. -
.
-
..
,,
.
s NN l
-
=
-
A_ _ _ u_ _
.
.
g
,
.
,
I I
(
[
!
e :::b*t i
au ry y 3l dg
-
g lu
'3M i
_
_
$;g-g:
y
t
p
-
+.
,
l
,4
--
'
- +I
=1
'g g
- .+-
i y: - i t:t +e A5
!
.
- t
.
E
}8l x-xl g lia s
- +
l
e
-
-
A
Il K4:
--
~
g lxp:-:?
If
,
_.
h,
~
El K)
E
.
-
_
7i
$
-:g: y I-A!$
y:
-
'
'
-
,
ix
g
-
il E
z x ix
-
I
,I
"
tX
-
-
$
f t(,I' g_ l{
Y
'l
.
u
-
m-
-
w ixi u
s.
.
a
xt a
>
-
a A
-
]&
~
,11
,
_A3
,,
i: :
<
_
N{
MI
& { 's
- N
,
x-x t..
-
l
-
sr
I
._
L
!
t
- -l e
\\
_
V Y
'I
.
,
1.
w
..
--..'s-..-..E
. -, - - -. -, -.
--+-...--,.=~.....--.-,-.%-.,+,.-...-.--..-.-.,.,..-.---m---,--+~e.
e.
_
._,-wwre--,w.-.-4--ma
-