IR 05000346/1998019

Insp Rept 50-346/98-19 on 981014-23.No Violations Noted. Major Areas Inspected:Operations,Maint & Engineering

ML20196C672
Person / Time
Site:	Davis Besse
Issue date:	11/20/1998
From:	NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML20196C654	List: IR 05000346/1998019 ML20196C649
References
50-346-98-19, NUDOCS 9812020083
Download: ML20196C672 (23)

Text

>

u 1 i-

U.S. NUCLEAR REGULATORY COMMISSION )

!

REGION 111

! 1 l Docket No: 50-346 I License No: NPF-3  ;

Report No: 50-346/98019(DRP)

Licensee: Toledo Edison Company i

i Facility: Davis-Besse Nuclear Power Station L

l

'

Location: 5501 N. State Route 2 Oak Harbor, OH 43449 -l

l

.)

Dates: October 14 - 23,1998 l

!-

Inspectors: S. Campbell, Senior Resident inspector l

'

K. Zellers, Resident inspector P. Lougheed, Reactor Engineer l

! Approved by: Thomas J. Kozak, Chief l Reactor Projects Branch 4 i

l j i l

l l

l l

t.

l

9812O20083 981120 PDR ADOCK 05000346

,

.G PDR 7

- _ - - . . . . .. - - -

--_______ _ - _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ -

l

.

EXECUTIVE SUMMARY Davis-Besse Nuclear Power Station NRC Inspection Report 50-346/98019(DRP)

This report contains the results of a special inspection which concluded on October 23,199 The resident inspector staff was augmented by a reactor engineer to review the circumstances surrounding a manual reactor trip during this inspectio Operations

.

The inspectors concluded that, overall, control room operatws responded well to the lockout of electrical buses D1 and D2, the component cooling water (CCW) system rupture disk failure, and the main steam safety valve that did not immediately resea Emergency procedures were effectively implemented and station management provided good oversight (Section 01.2).

.

The inspectors concluded that the operators should have consulted with engineering and maintenance personnel before proceeding with the CCW pump 2 start. Starting the pump resulted in the complete failure of the CCW rupture disk which resulted in a manual reactor trip with complications (Section 01.2).

.

Control room operators were slow to re-enter the overcooling section of Procedure DB-OP-02000 when it was recognized that the cooldown rate was excessive which resulted in an automatic rather than a manual steam and feed water rupture control system isolation of the steam and condensate system (Section 01.2).

Maintenance

.

Troubleshooting and equipment repairs associated with the bus lockout and CCW system were performed professionally. Personnel did not appear pressured to complete activities to meet the restart schedule (Section O2.1).

.

The inspectors concluded that although the maintenance rule risk matrix and Technical Specifications were complied with, the authorization of work to occur on components associated with a protected train while the plant is being operated brings into question the effectiveness of work control processes and the risk matrix used when evaluating and approving online maintenance work activities (Section M1.1).

Enaineerina

.

The CCW system respondt.d in accordance with its documented design throughout the event. The train 2 nonessential valves that cycled open and close was caused by conflicting inputs to valve logic, due to the train 1 flow sensor being out-of-service and the train 2 pump breaker being open. Following the rupture disk failure, the automatic containment isolation valve closure stopped the surge tank level decrease at the approximate level of the divider plate in the tank (Section O2.1).

\ _ - _ - _ - - _ _ - - _

_ _ _ _ _ _ - - _ _ _ - _ - - - - - - - - _ - _ - _ - - - - -

.

The inspectors concluded that the licensee's approach to identifying and resolving equipment problems was methodical and comprehensive. All known equipment anomalies were documented and entered into the licensee's corrective action program and resolved before plant restart (Section O2.1).

.

.

Engineering personnel support for resolving equipment problems was thorough and effective. However, engineers did not question the operability of emergency diesel generator (EDG) 2 after the diesel had run without cooling water greater than the time allowed in the USAR. No EDG damage was found during a subsequent inspection (Section O2.1).

.

Although no design basis information existed for the CCW system, the licensee calculated that the surge tank level interlocks provided adequate protection against four rupture disks failing. Because of the lack of design basis information, the licensee planned to validate the design for the entire CCW system (Section O2.1).

_ _ __ __ _ - _ _ _ . _

_ __ . _ _

_

f Report Details Summary of Plant Status At 1355 on October 14,1998, with the plant being operated at 100 percent power, a lockout of 4160 Volt (V) buses D1 and D2 resulted in a temporary loss of cooling flow to the letdown coolers. This ultimately resulted in a component cooling water (CCW) system rupture disk failure on a letdown cooler and the isolation of nonessential CCW loads (including reactor coolant pump seal and motor bearing cooling) which led operators to manually trip the reactor at 1524. Complications that followed the plant trip included the failure of a main steam safety valve to reseat at the expected pressure, the failure of a makeup pump to start, and the an overcooling of the reactor which resulted in the automatic isolation of a steam generator from the steam and condensate system. On October 19, during reactor restart, an automatic reactor trip occurred due to an inadequate procedure and on October 21, while the reactor was at 100 percent power, a turbine runback caused power to decrease to approximately 60 percen The circumstances surroundino the October 19 and 21 events will be described in Inspection Report 50-346/98017(DRP).

l. Operations 01 Conduct of Operations O1.1 General Comments The following is a brief description of the CCW system design which is needed to better understand the sequence of events which are described in Section 01.2 of this report:

The CCW system at Davis-Besse provides cooling for both essential and nonessential equipment. The system has an expansion tank which supplies 2 trains, one for each the essential equipment cooling loops. Either train of CCW can feed the nonessential loads, which include 2 letdown coolers and 4 reactor coolant pump (RCPs) seals and -

motor bearings, through cross-connect valves 5095 and 5096. Although the valves are termed cross-connect valves, the two trains cannot actually be cross-connected due to check valves which prevent back flow from the opposite train into the essential header Only one train of CCW is required for normal operation. In this configuration, one CCW pump is running (usually CCW Pump 2) with its associated cross-ccr.i.ect valve ope The other CCW pump is aligned to supply water to its essential loads, with its cross-connect valve closed to isolate the nonessential load. A third CCW pump is normally designated as a spar Each pump has a pump discharge flow switch. The flow switch provides an open signal to the opposite train's cross-connect valve, and a start signal to the opposite train's pump when flow is indicated at less than 1,000 gallons per minute (gpm). The cross-connect valves receive closure signals if its associated pump breaker is open and on low level in the CCW expansion tank. The CCW expansion tank serves to accommodate CCW expansion and contraction due to heating and cooling loads. The

_

[ )

'

expansion tank has a 33-inch divider plate welded vertically inside the tank that j separates trains 1 and 2 to ensure at least 33 inches of water is available to each trai The tank has the following level interlocks:

L

. 49 inches - low CCW level alarm L

'

. 47 inches - closure of Nonessential Cooling to Auxiliary Building isolation Valve 1495

-

37 % inches - low CCW level alarm and closure of:

, CCW Containment Isointion Valves 1411 A and B CCW Pump 1 Cross Connect Valve 5095

. CCW Pump 2 Cross Connect Valve 5096 1

..

CCW Loop 1 Nonessential Return Valves 5097 and 2645 i CCW Loop 2 Nonessential Retum Valves 5098 and 2649

+ 35 inches - - closu're of Nonessential Cooling Header to Makeup Pump Valve CC1460 f 01.2 Inadvertent Lockout of Electrical Buses D1 and D2 Inspection Scooe (93702)

l l At 1355 on October 14, with the plant being operated at 100 percent power, operators

.

'

received alarms and indications that electrical buses D1 and D2 were de-energize The inspectors responded to the control room, observed the operators' response to this event, and reviewed the circumstances leading up to the lockout. The inspectors developed a detailed sequence of events which is included as Attachment 1 to this

,

report.

L

' Observations and Findinas

,

initial Conditions The plant was being operated with train 2 equipment designated as the protected trai Work is not normally performed on protected train equipment. However, l

r transformer AC, a 13.2 to 4.16 kilovolt step-down transformer, was tagged out-of-service primarily for personnel safety so that scaffolding could be built around the transformer to support adjustment of the deluge system spray nozzles. This transformer is the alternate electrical supply for vital bus D1, which provides power for train 2 equipment. The clearance for this work required, among other things, that circuit

'

breaker ACCD1, which is the supply breaker between transformer AC and vital bus D1, i- be tagged open and racked out. The licensee decided to remove this breaker from its cubicle to perform maintenance on it. This work was approved by the shift manage j Other maintenance work that was ongoing included: 1) CCW pump 1 discharge flow l switch 1422D was valved out-of-service for a calibration check (CCW pump 2 was running; CCW pump 1 was in standby; and CCW pump 3 was in spare status), and  ;

2) the auxiliary boiler w6s out-of-service for code safety valve testin j i.

L

!.

!

' '

,

'

., , -- - , -. -

. -. . - - - - , - - - - - - -

.

- - _ - _ _ _ _ - _ _ _ _ _ _ _ - - _ - _ _ _ _ - _ _ _ _ _ _ _ _ _ _ ._ _

,

Lockout of Buses D1 and D2 and Partial Rupture of CCW Letdown Cooler 1-1 Rupture Disk The maintenance work was completed on breaker AACD1 and an electrician contacted the control room to request an operator remove the tagout so the electrician could reinstall the breaker into the cubicle. An operator removed the tagout and remained in the room to observe the electrician reinstall the breaker. Reinstallation required that the electrician align the breaker guide rails with the cubicle floor rails. While pushing the breaker into the cubicle, the right side breaker guide rail became stuck on the floor rail which required the electrician to apply additional force to the right side of the breake While applying this additional force, the left side of the metal shield barrier, located on the front of the breaker, contacted an exposed terminal screw on the back of the cubicle door. The operator observed a faint blue spark, the lights in the room went out, and the emergency room lighting illuminated. All breakers on buses D1 and D2 opened to lockout the buses. Both the electrician and the operator heard emergency diesel generator (EDG) 2, which was located next to the switchgebr room, automatically star The operator directed the electrician remove the breaker from the cubicle to prevent further damage to electrical circuits and notified the control roo The lockout of bus D1 caused a loss of power to CCW pump 2 and service water pump 2. The lockout of bus D2 caused a loss of power to nonessential cross-connect valve 5096 and return valves 5098 and 2649 (all of which were associated with CCW pump 2 and failed as-was, open) and the loss of power to one of the three condensate pumps. Operators reduced reactor power to 87 percent to maintain the plant within the capacity of the two remaining condensate pumps. Additionally, the control room operators directed an equipment operator to locally shutdown EDG 2 because of a loss of cooling to the EDG jacket water cooling system (CCW train 2 provided cooling for jacket water and service water train 2 provided cooling for CCW train 2).

The loss of CCW pump 2 caused flow to drop below 1000 gpm as sensed by CCW pJmp 2 discharge flow switch FIS1432C, which sent a signal to immediately start CCW pump 1 and to begin a 30 second delay in opening CCW pump 1 cross-connect valve 5095 and return valves 5097 and 2645. The drop in flow decreased cooling to the nonessential loads which provided cooling to, among other components, letdown cooler 1-1. During the 30 second delay, the loss of cooling flow from CCW pump 2 and RCS heat generation in letdown cooler 1-1 created saturated conditions in the CCW side of the cooler. After 30 seconds, the cross-connect valve 5095 fully opened and nonessential flow to letdown cooler 1-1 was initiated. A pressure spike from relatively cold CCW water entering the saturated CCW side of the letdown cooler caused CCW rupture disc PSE 3761 on letdown cooler 1-1 to partially fail. A 2 to 5 gpm CCW leak occurred as a result which operators correlated to an increased filling rate of the containment normal sump. The CCW expansion tank water level dropped to the low level alarm setpoint at 49 inches. The operators periodically filled the CCW expansion tank with demineralized water to maintain expansion tank leve The cause of the lockout was determined and the lockout was reset for buses D1 and D2 and the buses were energized. Power was restored to CCW pump 2 cross-connect

!

_ - - . - . . - ..-.--_.-- .- .. - .---.-. -.-.-_-...-.. - _ -. _

!

4  !

valve 5096 and retum valves 5098 and 2649, and the valves began to close. However, at less than 20 percent open, the circuit logic interpreted no flow from CCW pump 1 discharge flow switch FS 1422D, even though actual flow existed, because the flow switch was isolated for maintenance. In response, the logic circuit sent a signal to begin opening the CCW pump 2 cross-connect valve 5096 and retum valves 5098 and 264 However, when valve positions exceeded 20 percent open, the logic circuit sent a signal i to close the valves because CCW pump 2 breaker was open and train 2 was  !

'

unavailable to provide nonessential flow. Consequently, train 2 cross-connect and return valves cycled open and closed, which was noted by the control room operators 1 during the event. The inspectors confirmed that the cross-connect valves operated as !

designed when conflicting signals were input from the CCW pump 1 flow switch that was l out for maintenance and the CCW pump 2 breaker that was ope I Failure of CCW Rupture Disk and Manual Reactor Trip i There are two methods to provide cooling water to the RCP seals: 1) makeup pump l injection past the seals and,2) cooling to the seal packcge heat exchanger from the CCW pump aligned for nonessential cooling. Operators preferred electrical independence for providing seal cooling: a makeup pump powered from one essential bus (C1 or D1) and the running CCW pump powered from the other essential bus. Flow from the CCW system is the only way to cool the RCP motor bearing After the buses were restored and with makeup pump 1 running, the control room senior reactor operator (SRO) directed CCW pump 2 be started to restore electrically independent cooling to the seal package. Even though the CCW system was in a known degraded condition, he did not consult with engineering or maintenance personnel to determine the acceptability of starting the pump. Normally, a CCW pump i

'

is manually started with the associated cross-connect valve shut. However, since the

. valves were cycling open and were not closed, starting the pump caused a hydraulic ,

,

pressure surge which caused a complete failure of rupture disk PSE 3761. The licensee I l determined that the rupture disk would have failed when starting CCW pump 2 with or !

'

without the cross-connect valve closed because reopening the cross-connect valve after starting the pump would have caused a similar pressure respons The rupture disk frilure caused CCW expansion tank level to drop rapidly, actuating )

,

level alarms and a k' vet switch interlock at 37% inches that started closing, among other J l valves, CCW containmat isolation valves 1411 A and B. Ten seconds later, the CCW containment isolation valvm stroked fully closed, which isolated the source of the leak from the safety related porte ns of the CCW system and isolated nonessential CCW cooling water to letdown cociers 1-1 and 1-2 and all the RCP motor coolers. Without nonessential cooling to tne RCPs, the SRO directed a manual trip of the reactor, and it i

was manually tripped. Further, without CCW cooling to the letdown coolers, letdown L automatically isolated on high temperature. During the event, the shift manager noted ,

that CCW expansion tank level indicators LI 1403 and 1404 displayed 20 inches. A review of the indicators after the event revealed CCW expansion tank level stabilized at 32 inches,1 inch below the CCW expansion tank divider plate level.

!

1 i 7

.

l

, . _ _ _ . _ _ _ - _ _ _ __ ___ _ __

i e

Post Trip Recovery i

After the reactor was tripped, the main steam safety valves (MSSVs) lifted. The RCPs were tripped in accordance with procedures to prevent RCP damage on a loss of CCW cooling. When the RCPs were tripped, the Steam and Feedwater Rupture Control System (SFRCS) logic was satisfied and caused the automatic start of both auxiliary feedwater (AFW) pumps, as designed, to feed both steam generators. The open MSSVs relieved energy to atmosphere and the turbine bypass valves opened, as expected, to relieve energy to the condenser, which served as the heat sink for reactor 1 decay heat removal. Follouing the trip, an equipment operator reported that MSSV l

SP1787 had not reseated. The overcooling section of Procedure DB-OP-02000 was i entered which directed the operators to take manual control of the turbine bypass valves l to lower steam header pressure and reseat the MSSV. Steam generator pressure was lowered from 955 pounds per square inch gauge (psig) to approximately 920 psig and the MSSV reseate Operators responded to the manual reactor trip in accordance with Emergency i Procedure DB-OP-02000, "RPS, SFAS, SFRCS Trip, or SG Tube Rupture." Natural circulation cooling immediately occurred and was verified by operators observing a hot to cold leg temperature differential of 30*F. The operators attempted to start makeup pump 2 in accordance with Procedure DB-OP-02000, but the pump failed to start due to a faulty anti-pumping relay in the breaker closing circuitr I

Throughout the recovery process, the SRO followed the emergency procedure in an orderly fashion. Good three-way communication was evident throughout this portion of the incident response. Operations management maintained a support role during the post trip recovery actions, allowing the control room operators to perform their duties professionall Operator Response to Plant Overcooling Event Post trip response instructions in Procedure DB-OP-02000 required the starting of the auxiliary boiler to support the post trip steam loads and maintain steam generator pressures and RCS temperature stable. However, the auxiliary boiler was unavailable i because a relief valve was removed for code-required testing. Following this event, heat loads were limited because of: 1) the relatively low decay heat generated since the last trip three weeks earlier,2) the lack of heat input from the secured RCPs, and 3) the large steam consumption to operate main and auxiliary feed pumps and other secondary loads for maintaining condenser vacuum. Consequently, the RCS cooled which caused the steam generator pressure to decrease. The control room staff recognized this condition and tracked the cool-down rat When the cool-down rate exceeded the administrative limit of 50*F per hour, the overcooling section of Procedure DB-OP-02000 was re-entered, which directed that the steam / feed rupture control system (SFRCS) be actuated manual!y. However, about a minute before manual actuation, the low pressure setpoint on steam generator 2 was reached causing an automatic SFRCS actuation, which isolated the steam generator 2 from the steam and condensate system and aligned the AFW system to steam

?

,.

,

'

generator 1. Within 2 minutes, the low pressure SFRCS trip cleared on steam generator 2, and since a main feedwater differential pressure trip still existed (due to SFRCS isolation of steam to the main feedwater pumps), SFRCS automatically

_

realigned the AFW system to normal operation.' This action terminated the overcooling transient. The lowest parameters observed during the transient were:

. T - 518'F 1

.

Pressurizer level- 95 inches

. RCS pressure - 2180 psig

. Steam Generator Pressure - 600 psig

. Steam Generator level- 49 inches

. T, - 500 * F The maximum cool-down rate was 65"F/ hour. The inspectors reviewed the Technical Specifications (TSs) and determined that no violation of cool-down rate limits occurre The natural circulation flow of the RCS was maintained throughout the transien Restoration of Letdown As mentioned above, letdown was automatically isolated on a high letdown temperatur Consequently, RCP seal injection and makeup bypass flow to the RCS caused prestsurizer level to increase at an estimated 0.7 inch / minute. The licensee continuously monitored level and did not place letdown in service until confirming which of the two letdown coolers had the failed rupture disk. A plan was developed, a briefing was i conducted, and personnel entered containment and identified the failed disk. Letdown cooler 1-1 was isolated and letdown cooler 1-2 , which is in parallel flow path, was placed in service. The high letdown temperature condition cleared shortly thereafter and letdown was restored. Nonessential CCW was restored to the RCPs and RCP 1-2 and 2-2 were started to provide forced circulation of the RCS. Maximum pressurizer _

level was 139 inches, which was below the TS limit of 305 inche Conclusions The inspectors concluded that, overall, control room operators responded well to the !

lockout of buses D1 and D2, the CCW rupture disk failure, and the MSSV that did not immediately reseat. Emergency procedures were effectively implemented and station management provided good oversigh The inspectors concluded that the operators should have consulted with engineering and maintenance personnel before proceeding with the CCW pump 2 start because of the degraded condition of the CCW system. Starting the pump resulted in the complete failure of the CCW rupture disk which resulted in a manual reactor trip with complication The licensee identified that control room operators were slow to re-enter the overcooling section of Procedure DB-OP-02000, which resulted in an automatic rather that a manual SFRCS isolation of the steam and condensate system. This issue was appropriately documented in the corrective action progra __ . __ _

! V

'

4, ..

L l- 02 Operational Status of Facilities and Equipmen i O2.1 Bus Lockout. CCW System Resoonse. Eauioment Problems. and Recovery Actions

, Inspection Scope (93702)

The inspectors monitored the licensee's efforts to assess various equipment problems

- which occurred during the event. The licensee developed a list of issues and imposed a restart restraint until the issues were resolved. The inspectors independently reviewed the list and followed up on the resolution to each issue.

f

b. LObservations and Findinas

. Development of Licensee's Restart issues List j The licensee assessed equipment problems encountered during and subsequent to the event and captured the issues on a list entitled, " Problems Encountered During the l

'

10/14/98 Reactor Trip." The_ items were prioritized into the following categories:

resolution prior to plant restart, issues requiring an understanding, issues of minor

'

significance, and non-issues. The licensee determined that 12 items needed to be resolved prior to plant restart. . The inspectors performed an independent review of the

! list, determined that it was comprehensive, and monitored the licensee's efforts for issue j- resolution. The following equipment problems were considered to be the most significant encountered and resolved during the event and/or the recovery activities.

{ .

Lockout of Buses D1 and D2 While Installing Breaker AACD1 Following the loss of power to the buses, an investigation team consisting of plant l engineering, maintenance engineering, and operations personnel assembled in the j switchgear room to inspect breaker AAOD1, the switchgear for buses D1 and D2 and j l the associated relays. The team examined the breaker and identified a scratch on a j terminal screw. The team determined that contact with the breaker barrier shield

'

caused the scratch. This contact created a short circuit from the terminal screw through L ' the breaker chasis to the cubicle floor rails. The protective relay trip flags indicated the

!

lockout of buses D1 and D2 as designed. Trip flags and relays upstream in the logic

' circuit of this device did not actuate, which confirmed the short in the circuit was at the terminal screw. Consequently, undervoltage relays dropped out and caused EDG 2 to L automatically start when the loss of power to bus D1 occurred. The EDG supply '

L breaker did not close because of the bus D1 lockou The inspectors reviewed Electrical Wiring Diagram E-34B, Revision 12, '4.16kV FD l j; Breakers Bus C1(D1) Tripping & Lockout Relays & Synchro Check Relays," interviewed L electrical maintenance personnel and operations personnel, and confirmed the apparent cause of the event.

l l.

F

=

!

'

_ _ .. ._ _ _ _ _ _ _ _ -._ _ _ __

___

.

t CCW System Response Ruoture Disk History The inspectors reviewed the history of the letdown cooler rupture disks to determine if any previous breaks had occurred. Documents reviewed included PCAQR 93-0206, April 6,1993; PCAQR 93-0252, April 19,1993; Request for Modification 93-0046, May 18,1994; Temporary Modification 93-0025, April 19,1993; Safety Evaluation 93-0026, August 27,1993; PCAQR 97-0752, June 1,1997; ant'

MWO 7-97-0752-01, May 199 In 1993 the licensee found that, although shown on plant drawings, rupture disks were not installed in the letdown coolers. In place of the rupture disks, blank flanges were installed. The licensee could r,ot determine if the rupture disks were ever installed, or if the blank flanges were installed during initial plant construction. During the April 1993 outage, the licensee installed rupture disks rated at 110 psig in accordance with the

< original design specifications. However, at the end of that outage, while performing containment isolation valve testing, several of the rupture disks failed. Although the licensee speculated that the testing procedure methodology might have contributed to the disk rupture, blank flanges were reinstalled under a temporary modification for -

another operating cycl During that operating cycle, the licensee evaluated the purpose of the rupture disks and the operating constraints on the disks. The licensee determined that rupture disks with a break point of 250 psig would be sufficient to protect the coolers from a makeup system tube rupture while not interfering with normal operaton. Four rupture disks (two per letdown cooler) rated at 250 psig were installed during the October 1994 refueling outage. The inspectors reviewed the licensee's modification and associated safety evaluation and determined that the increase in break point pressure was ~

acceptable No problems were encountered with the rupture disks after their installation until j May 1997, when a 6-gallon per hour leak developed. The licensee evalucied the leakage and deemed that the most likely cause was a manufacturing defect in one rupture disk which resulted in a pinhole leak. The licensee operated the plant for approximately a year (until the April 1998 refueling outage) without any increase in l leakage. During the outage, all four rupture disks were replaced. Because of fixed  !

contamination on the disks, the licensee was unable to send the disks to the manufacturer for testin Saturated CCW Conditions in Letdown Cooler Causes Ruoture Disk Failure The licensee postulated that a 30-second disruption in CCW flow to Letdown Cooler 1-1 created conditions that contributed to the initial rupture disk leakage. The inspectors performed a review of this apparent cam The loss of bus D1 caused CCW pump 2 to trip and stop providing CCW flow to letdown cooler 1-1. Although the circuit logic immediately started the standby pump, CCW

- __ . -

.

y . ._ _ _ _ _ _ ._ _ . _ _ _ _ _ _ _ _.

i !

.

pump 1, a time-delay relay in the circuit logic delayed opening CCW pump 1 cross-connect valve 5095 and retum valves 5097 and 2645 for 30 seconds. which delayed supplying CCW flow to the letdown coolers. The reason for this time delay was not understood by the licensee since no design basis information existed for the CCW system. The licensee postulated, during this 30-second period, RCS temperature of  ;

approximately 580*F heated the stagnant CCW in the shell side of the cooler and i l created saturated conditions. After the relay timed out, valve 5095 and return valves 5097 and 2645 opened and delivered relatively cold (95'F) CCW to the letdown cooler. The introduction of the cold water caused a thermal shock and a pressure spike, 4 partially rupturing rupture disk PSE 371 l t Consequently, the licensee determined that this condition may occur whenever the  !

j CCW pumps are automatically swapped. The licensee reviewed operating logs since l l 1994, when previously installed blank flanges were replaced with rupture disks, and confirmed that the CCW pumps had never automatically swapped. Manual swapping of the pumps, which the operators had done before, does not cause a 30-second disruption in flow, and, therefore, saturated conditions in the cooler had most-likely never occurred.

l Licensee management recognized that they were vulnerable to future rupture disc failures any time the CCW pumps automatically swapped and implemented temporary modification 98-0037 to decrease the time delay for opening of the cross-connect valve l to 5 seconds before the plant restart. This shorter delay time limited the likelihood of l saturated CCW conditions in the cooler. The licensee also replaced the rupture disks

! on letdown coolers 1-1 and 1-2. Long-term corrective actions being evaluated included I

the possibility of installing a different relieving device (safety or relief valve) during the next refueling outag CCW Exoansion Tank Level Interlocks Since design basis information did not exist for the CCW system, the purpose of the divider plate in the CCW expansion tank was not well understood. As mentioned above,

the shift manager noted that CCW expansion tank level on both sides of the divider l plate fell below 33 inches to 20 inches. Component cooling water containment isolation valves 1411 A and 118 were confirmed to have a 10-second closure time. The

'

inspectors reviewed control room log entries between 1523, when the containment isolation valves closed, and 1535, when the final expansion tank level was recorded.

l The 1535 entry indicated expansion tank level at 32 inches. A review of the log and interviews with operators confirmed that no water had been added to the expansion tank between 1523 and 1535.

l The inspectors questioned whether a 10-second closing time of the containment isolation valves, with valve closure beginning at the 37%-inch tank level interlock, would have resulted in a final expansion tank level of 32 inches. The licensee performed Calculation C-NSA-016.04-002, and confirmed final level at 33 inches, which was a close correlation. Further, the calculation used Computer Code PROFLO, version 4.01, to verify sufficient containment isolation valve closure time to maintain a minimum tank i volume assuming a worst case condition (all rupture disks blown on both coolers with

one CCW pump running). The calculation indicated that the expansion tank volume and

l

. . .

, ___________ _ - - - - - - - - - _ - - - - - - - - - - - - - - - - - - - - - - - - - - - -

e levelinterlocks provided adequate protection under worst case conditions. The inspectors reviewed the calculation and found it acceptable. The inspectors interviewed the shift manager who read 20 inches in the expansion tank who reiterated that he read 20 inches in the tank. The inspectors determined that either the level indicator was misread or that the dynamic effects from the cycling valve, CCW leak, and/ or CCW pump starting caused an inaccurate indicatio Although the calculation determined that the level interlocks provided protection against worst case conditions (a four-disk rupture), the licensee recognized the need to validate the CCW desig Failure of MSSV SP1787 to Reseat After Lifting As mentioned above, MSSV SP17B7 did not reseat as expected following the plant trip and main steam pressure was reduced from 955 psig to 920 psig to reseat the valv The safety function of the valve is to lift at setpoint to provide overpressure protection of the main steam system. Reseating pressure, which is adjusted by the manufacturer and is independent of the lift pressure, was not required to be set per TSs. Further, this pressure is not required to be set per the ASME code. Nevertheless, failure to reseat the MSSV could cause an overcooling condition in the reactor coolant system. The required setpoint for the valve per TS 3.7.1.1 is 1050 psig, which had been checked before Refueling Outage 11 on April 7,1998, and during plant restart on July 17,1998,

,

following the tornado event. The valve lifted at 1127 psig (high by 7.33 percent) during the performance of the April 7 test and then was adjusted within tolerance. The setpoint was determined to be within tolerance during performance of the July 17 tes On October 17, following this event, the valve was tested at normal operating temperature and pressure and was determined to be within its setpoint. The inspectors determined that MSSV SP1787 lifted as designed during the event and was verified to be within the TS required lift setpoint before plant restart. Reseating of the valve at a certain pressure is not a regulatory requiremen EDG 2 Operation Without Cooling Water As mentioned above, EDG 2 automatically started when power was lost to bus D1. The EDG 2 supply breaker for bus D1 did not close, as designed, because of the lockou Without power, CCW pump 2 (powered from D1), could not start and was unable to provide cooling for the EDG Jacket water cooling system. Without jacket water cooling, the EDG engine could overheM and could be damaged. Therefore, the control room operators directed an operator to locally shutdown the EDG. Control room operators do not have the capability to shutdown the EDG from the control roo After the licensee completed their restart issues list, the inspectors noticed that running EDG 2 for a period without cooling water was not on the list. The inspectors contacted the system engineer and questioned the allowable EDG run time without cooling wate The engineer determined that Updated Safety Analysis Report (USAR),

Section 8.3.1.1.4.1, stated that each EDG was capable of running without cooling water for 3 minutes after startup without electrical loads. The engineer reviewed the sequence i 13

- _ _ _ _ _ _ _ - _ _ _ -

~

l l

l

! of events printout and confirmed that the EDG ran unloaded and without cooling water

! for 4 minutes and 54 seconds. The issue was then added to the restart issues lis The engineer contacted the vendor and the vendor stated that EDG damage was unlikely if an EDG jacket water high temperature alarm was not received. A review of the sequence of events printout confirmed that no Jacket water cooling high temperature alarms occurred. However, the licensee generated and performed MWO 1-98-01157-00 to conduct visualinspections of the EDG. On October 17, the engineer visually confirmed no jacket water leaks in the EDG cylinder heads and a crankcase oil sample confirmed no presence of jacket water. The cylinder test cocks were opened and the j EDG was manually barred which confirmed no presence of jacket water in the cylinders.

! Before the event, the licensee had planned to perform an extensive overhaul of the EDG l in February 1999. The licensee will perform further visual inspection of potentially l affected areas during this rescheduled overhaul.

!

The ingectors reviewed the sequence of events computer printout and confirmed no jacket water high temperature alarms occurred and the time length EDG 2 was l

operating without cooling. The inspectors determined that the inspections of the EDG l were reasonable to confirm that no damage occurred to engine areas cooled by jacket i wate Makeup Pump 2 Failure to Start on Demand l

Before the lockout of buses D1 and D2 and the subsequent manual reactor trip, makeup pump 1 was in operation and makeup pump 2 was in standby. After power was restored to the buses, the primary reactor operator attempted to start makeup pump 2 but the pump failed to start when he turned the handswitch to the "close" position. A

subsequent review by the licensee determined that makeup pump 2 breaker AD105 l failed to clos Maintenance Work Order (MWO) 1-98-1143-00 was generated and i Procedure DB-ME-09104, '13.8 kV and 4.16kV Westinghouse DHP Breaker," were used as guidelines to troubleshoot the breaker. Maintenance personnel found no failures of the permissive contacts for the control circuitry that were required to close.

l Further, electrical checks did not reveal deficiencies. The breaker was placed into the

!

test position and failed to close during post maintenance cycling of the breake Subsequent troubleshooting determined that an anti-pumping relay, which is used to prevent repetitive cycling of the breaker, had a contact that did not have electrical continuity. The anti-pumping relay was replaced with a new relay and the breaker was tested successfully both in the test position and in the racked in position.

i l While installing the coverplate over the new relay, an electrician noticed that the

, coverplate partially depressed the armature buttons on the relay which caused a contact l in the relay to partially open. Although the new relay operated properly with the armature buttons partially depressed, the licensee speculated that, over time, with the contacts partially open on the old relay, either arcing across the contact or dirt buildup on the surfaces fouled the contact surfaces. This long-term fouling created high resistance that allowed the anti-pumping relay to energize intermittently and prevented

l l

l

_ _ _

3-i Q

the spring release coil from actuating to close the breaker. Remedial actions taken to prevent this situation from recurring included installing washers between the coverplate and the coverplate support to increase the gap clearance and prevent partial depressing l of the armature buttons.

l The licensee performed an extent of condition review and by inspecting all 4160 V breakers prior to plant restart. Insufficient gap clearances were identified between the

, coverplate and armature buttons for anti-pumping relays in breakers AD111 (high i

pressure injection pump 2), AC108 (CCW pump 3), and AD 301 (supply breaker for the I

station blackout diesel). The licensee did not perform as found resistance readings across the contact nor did they replace the anti-pui.1 ping relays. Instead, these breakers were verified to actuate in this configuration and washers were installed to increase the gap to armature clearances.

l The inspectors observed a portion of the troubleshooting effort and reviewed the logic l scheme for closing the breaker and confirmed the licensee's assumption. The inspectors reviewed control room logs and verified that makeup pump 2 was started successfully on September 14,1998, to support an online maintenance outage for makeup pump 1. Makeup pump 2 passed its last surveillance. A review of breaker l maintenance procedures and the vendor technical manual determined that the coverplate/ armature button gap clearance were not required to be checke c. Conclusions The CCW system responded in accordance with its documented design drawing throughout the event. The train 2 nonessential valves that cycled open and close was caused by conflicting inputs to valve logic, due to the train 1 flow sensor being out-of-service and the train 2 pump breaker being open. Following the rupture disk failure, the automatic containment isolation valve closure stopped the surge tank level decreas Although no design basis information existed for the CCW system, the licensee l

calculated that the surge tank level interlocks provided adequate protection against four rupture disks failing. Because of the lack of design basis information, the licensee planned to validate the design for the entire CCW syste The inspectors concluded that the licensee's approach to identifying and resolving l equipment problems was methodical and comprehensive. All known equipment I anomalies were documented and entered into the licensee's corrective action program l and resolved before plant restar Engineering personnel support for resolving equipment problems was thorough and i

effective. However, engineers did not question the operability of EDG 2 after the diesel l had run without cooling water greater than the time allowed in the USAR. No EDG damage was found during a subsequent inspectio Troubleshooting and equipment repairs were performed professionally. Personnel did not appear pressured to complete activities to meet the restart schedul l

-Y l

11. Maintenance M1 Conduct of Maintenance

M1.1 Protected Train Concept for Maintenance Activities Not lmolemented Inspection Scoce (93702)

1 The inspectors reviewed the licensee's approach to scheduling maintenance on a train i of equipment that ultimately impacted the protected, redundant train.

! Observations and Findinas Through interviews, the inspectors determined that, in the past, transformer AC

' maintenance was normally performed during plant outages rather than during the

,

'

operating cycle. Plant personnelindicated that these types of outages were moved to the operating cycle due to an emphasis on shorter outages. This decision caused increased plant risk during the operating cycle and decreased plant risk during outages.

l.

L

'

' The decision to perform maintenance on breaker AACD1, in order.to take advantage of transformer AC being tagged out, increased the risk to the protected train. .The protected train concept emphasizes that work be done on one train with the other train of equipment maintained fully functional and operable so that the design function would be maintained. Work on this equipment was outside of the risk matrix used when evaluating online maintenance activities and it was not readily recognized that this work could have an effect on the protected train. As mentioned above, the protected train was effected when accidental grounding on AACD1 resulted in a lockout of bus D Maintenance on flow switch FIS1422D was performed even though the flow element affected the logic of train 2 components, which did not satisfy the protected train concept. A solution for performing maintenance on this component would have been to place CCW pump 3 (the spare) on as the train 1 CCW pump. This configuration would have overridden any control s;gnals from FIS1422D to the opposite side non-essential isolation valves and would have allowed the train cross-connect valves to close after the bus was re-energized. This may have reduced the impact of the pressure surge created when CCW pump 2 was placed back in servic The inspectors reviewed the station maintenance rule risk matrix and determined that the matrix allowed the above equipment to be taken out-of-service without any additional l, probabilistic risk assessment analysis to be performed. Additionally the inspectors

determined that the licensee was complying with all TS requirements for the removal of

- the above equipmen Conclusions The inspectors concluded that although the maintenance rule risk matrix and TSs were complied with, the authorization of work to occur on components associated with a

._

,

.2

,

i protected train while the plant is being operated brings into question the effectiveness of work control processes and the risk matrix used when evaluating and approving online maintenance work activitie V. Manaaement Meetinas  ;

l X1 Exit Meeting Summary

,

l l

\

) The inspectors presented the inspection results to members of licensee management at the !

I conclusion of the inspection on October 23,1998. The licensee acknowledged the findings i presented. The inspectors asked the licensee whether any materials examined during the l inspection should be considered proprietary. No proprietary information was identified.

i I.

l l

l l

l  !

!

!  !

'

i

l l

l

i

P t

!-

L' PARTIAL LIST OF PERSONS CONTACTED Licensee -

l' J. K. Wood, Vice President J. H. Lash, General Manager L

'

L._W. Woriey, Director, Nuclear Assurance

' R. E. Donnellon, Director, Engineering & Services J. L. Freels, Manager, Regulatory Affairs H. W. Stevens, Jr., Manager, Nuclear Safety & Inspections

' M. C. Beior, Manager, Quality Assessment D. L. Eshelman, Manager, Operations J. L Michaelis, Manager, Maintenance y L. M.- Dohrmann, Manager, Quality Services *

3 '

F. L. Swanger, Manager, Design Basis Engineering i I' P. R. Hess, Manager, Supply J. W. Rogers, Manager, Plant Engineering ,

W. J. Molpus, Manager, Nuclear Training .

C. A. Price, Manager, Business Services R. B. Coad, Jr., Superintendent, Radiation Protection 1'

- A. R. Schumaker, Supervisor, Nuclear Security Support D. H. Lockwood, Supervisor, Compliance

.

T. J. Chambers, Supervisor, Quality Assessment C. A. Kraemer, F.qineer, Regulatory Affairs G. M. Wolf, EngRer, Licensing -

'

J. M. Bonfiglio, Shift Supervisor  :

,

_ D. L. Miller, Senior Engineer, Licensing i L

=NRC Reaion ill

- J. Caldwell, Acting Regional Administrator, Rlll G. Grant, Director, Division of Reactor Projects J. Grobe, Director, Division of Reactor Safety

- J. Jacobson, Chief, Lead Engineers Branch, DRS T. Kozak, Chief, Reactor Projects Branch 4 S. Campbell, Senior Resident Inspector K. Zellers, Resident inspector HEB H J. Stolz, Brar.ch Chief, PECB i R. Dennig dection Chief, PECB l E. Goody. .n, PECB l E. Fields, PECB  !

T. Koshy, PECB -

T. Hiltz, EDO A.' Hansen, PD32 i. j

L

,

-a~ v , .. - v - ,-

a , , ,- ,n

• Y i-

.t: .l i

INSPECTION PROCEDURES USED l'P 93702: Prompt Onsite Response to Events at Operating Power Reactors l

<

l I

I

.

. l

j

'

i

.I l

! l

!-

i j '.

! 1 i

!

. . .

'

.

).

-

i l

< ,4 , , . . . . . _ _ , _ . _ . ._ . _ . .

L : iY

!

! _-

l LIST OF ACRONYMS AND INITIAlJSMS USED AFW Auxiliary Feedwater CCW Component Cooling Water  !

CFR Code of Federal Regulations EDG Emergency Diesel Generator

,

GPM Gallons Per Minute j l IFl Inspection Followup item 1 IR inspection Report MSSV. Main Steam Safety Valve MWe Megawatts Electric MWO- Maintenance Work Order

! NRC Nuclear Regtilatory Commission i

!

PCAQR Potential Condition Adverse to Quality Report t- PDR Public Document Room PSIG Pounds Per Square Inch Gauge RCS Reactor Coolant System RCP Reactor Coolant Pump SFRCS Steam & Feedwater Rupture Control System SRO Senior Reactor Operator TS Technical Specification USAR Updated Safety Analysis Report i V Volt  !

l I

l l

l i

!

l l'

<

l l

j-

,

.,(-

,,

_ _ . . _ _

,

1

Attachment 1 Time-lino of the Event On October 14,1998, a lockout of 4160 volt (V) buses D1 and D2 resulted in a temporary loss 1 of cooling flow to the letdown coolers. This ultimately resulted in a CCW rupture disk failure, the isolation of nonessential CCW flow and a manual reactor trip. Subsequent to the trip, a main steam safety valve (MSSV) failed to initially reseat and a makeup pump failed to star Also, operators failed to stop an overcooling event in a timely manner which resulted in an l automatic isolation of the steam generators from the steam and condensate system. The I following is a sequence of significant event I 1356 With the plant operating at approximately 100 percent power, a lockout of 4160 V l buses D1 and D2 occurred due to a lockout relay actuating during insertion of 4160 V breaker AACD1 into its cubicle. One of the condensate pumps, service water (SW)

pump 2 and CCW pump 2 tripped, CCW pump 1 automatically started, and emergency l diesel generator (EDG) 2 started but did not load onto bus D1. The train 2 CCW nonessential valves remained open due to the loss of power to their motor operator The train 1 CCW nonessential valves started to open after a 30-second time delay, and then fully opene EDG 2 was locally shutdown due to the lack of cooling water (CCW pump 2 and SW pump 2 provided cooling to Jacket water cooling and are powered from bus D1).

1409 Operators commenced a power reduction from about 930 MWe to 850 MWe due to the loss of a condensate pum A CCW expansion tank low level alarm was received in the control room. Control room operators remotely opened a demineralized water valve to refill the tan Operators noted that the containment normal sump level was rising at an increased rat A reactor operator noted no change in makeup tank levels, therefore, considering that I CCW expansion tank level alarms had been received, the operators postulated that a CCW rupture disk had been damage Re. actor power was decreased to about 820 MWe. Maintenance personnel reset the bus D2 lockou ,

1432 Breaker ABDD2 closed to re-energize bus D2,.

1437 Operators stopped filling the CCW expansion tan Control room operators entered the CCW abnormal procedur Operators started to fill the CCW expansion tan Operators stopped filling the CCW expansion tan ., P e

1459 Operators determined that the rate of CCW expansion tank decrease correlated to the rate of containment normal sump increase (2-5 gpm).

1505 CCW expansion tank low level alarm receive Operators commenced filling the CCW expansion tan The CCW expansion tank low level alarm cleare Operators stopped filling the CCW expansion tan Bus D1 is re-energized from bus D After restoring the required buses, operators noted that CCW nonessential valves were cyclin Operators started SW pump Operators started CCW pump 2 in order to restore an electrically diverse means of providing cooling to the reactor coolant pump (RCP) seals. This allowed the train 2 nonessential valves to stop cycling and begin openin Operators closed the CCW train 1 nonessential valves. The containment normal sump level high alarm and CCW expansion tank level low-low alarm were received in the control room. The CCW containment isolation valves automatically closed to isolate the source of the break which isolated CCW cooling to the RCPs. CCW train 2 nonessential isolation valves automatically close Operators initiated a manual reactor trip due to the CCW low-low level alarm and loss of CCW cooling to the RCP Operators tripped all RCPs due to the loss of CCW cooling. The steam and feedwater rupture control system (SFRCS) automatically started the auxiliary feedwater (AFW)

system when the RCPs were trippe Operators attempted to start makeup pump 2 in accordance with the emergency procedure, and it failed to star A high letdown temperature condition caused letdown to isolat A MSSV failed to reseat at the expected pressure. Operators entered the overcooling section of Procedure DB-OP-0200 Operators took manual control of turbine bypass valves to depressurize both steam generators to 920 psig in order to rescat the MSSV. The MSSV reseated at about 920 psig. Operators exited the overcooling section of Procedure DB-OP-02000 when they

determined that the plant was stabl The CCW expansion tank level was at 32 inche I ,Y

,

'

lF l

l

!

1536 The containment normal sump was pumped dow Operators commenced filling the CCW expansion tan Maintenance personnel reset the EDG 2 lockou Operators stopped filling the CCW expansion tank when the low level alarm cleare I 1619 Operators re-entered the overcooling section of Procedure DB-OP-02000 due to l lowering steam generator pressures. The overcooling was due to the unavailability of the auxiliary boiler, no RCPs running, low decay heat, and high steam load Operators manually tripped main feed pump I 1622 A steam generator 2 low steam pressure SFRCS trip caused the automatic isolation of ;

the steam generators from the steam and feedwater system. The AFW system was l isolated from steam generator 2 and aligned to steam generator 1. This essentially !

stopped the overcooling event and allowed the plant to stabiliz I l

1623 Operators manually initiated and isolateJ ine SFRCS. Some of the operators were not l aware that the SFRCS had just auten atically initiated and isolated on low steam l generator pressur I l

1624 Steam generator 2 low steam pressure trip cleared which allowed the SFRCS to align !

each AFW pump to its respective steam generato ;

1626 Operators opened the atmospheric vent valves to control the temperature of the plan Because letdown was isolated, operators reduced seal injection to the RCPs t3 a minimum of three gallons per minute to slow down the rate of pressurizer fillin Operators commenced filling the nonessential CCW header under controlled condition Low CCW expansion tank level alarm was received due to filling nonessential CCW heade Overcooling was terminated. Reactor temperature, pressure and steam generator pressure was stable. Maximum cooldown rate observed was about 65*F per hou Operators started RCP 2-2, restoring forced circulation of the RCS and pressurizer spra Operators started RCP i- After a containment entry, the CCW leak in containment was determined to be on letdown cooler 1. Operators then placed letdown cooler 2 in servic The letdown high temperature condition cleared and operators restored letdown. The maximum pressurizer level indication observed was 139 inche l l