05000323/LER-2003-005

LER-2003-005,

Event date:
Report date:
3232003005R01 - NRC Website

I. Plant Conditions

Unit 2 was in Mode 1 (Power Operation) at 54 percent power.

A. Background

The auxiliary feedwater system (AFWS) [BA] is a safety-related system that serves as a backup supply of feedwater to the secondary side of the steam generators (SG) [AB]. It maintains the heat sink function [BS] of the SGs whenever the Main Feedwater (MFW) system is unavailable.

The AFWS is Design Class I and includes the feedwater process and the steam supply portion of the system. The basis for the Class I designation is that the AFWS is considered an engineered safety feature system that is required for safe shutdown of the reactor. It is directly relied upon to prevent core damage and reactor coolant system (RCS) [AB] overpressurization in the event of transients, such as a loss of normal feedwater or a secondary system pipe rupture.

The design basis for the AFWS is to ensure that the minimum required flow (410 gpm) would be delivered to the minimum number of SGs (two SGs), within one minute, during any design bases event.

The normal operation of the AFWS, which is during startup and shutdown, is to supply the SGs with a secondary heat sink while main feedwater is unavailable. This is done with two AFW pumps providing the minimum required flow with suction taken from the condensate storage tank (CST) [KA]. If the CST becomes unavailable for any reason, several additional sources of water can be aligned to the AFWS.

The AFWS consists of three feedwater supply trains with diverse drive-power sources. One train employs a steam turbine-driven pump, AFW Pump 2-1, aligned to all four SGs. The other two trains consist of motor-driven pumps, AFW Pump 2-2 and AFW Pump 2-3, each supplying flow to two of the four SGs, with the capability to be aligned to any of the four generators. The steam-driven AFW pump is designed to deliver a minimum of 870 gpm and each motor-driven AFW pump is designed to deliver a minimum flow of 460 gpm.

This event involved the discharge line from AFW Pump 2-1 and 2-2 to the MFW line feeding SG 2-1. The first-off check valve, FW-2-377, from the MFW line is located on a common AFW line that can be fed by any of the LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) Diablo Canyon Unit 2 0 olo 0 1 3 2 1 3 2003 0 1 0 LER NUMBER (6) 5 IN 0 3 OF 9 three pumps. There are no valves between the first-off check valve and the MFW line. The second-off check valve, FW-2-370, is downstream of the level control valve (LCV) of AFW Pump 2-2 and upstream of the common AFW line. The third-off check valve, FW-2-362, is at the AFW Pump 2-2 discharge.

The check valves have the safety function of opening to permit the flow of water from the AFW pumps to the SG. During normal plant operation, when the AFW system is not in operation, the first-off and second-off check valves form two of three check valves in series that together serve to prevent flow of feedwater back through the AFW pumps to prevent possible thermal stresses, steam binding of the pumps and overpressurization of the AFW pump suction piping.

With respect to AFW performance, the first-off and second-off check valves are credited in the Final Safety Analysis Report events for Loss of Normal Feedwater (LONF), Feedline Break (FLB) and Main Steam Line Break (MSLB).

The LONF analysis assumes that with the limiting single failure of one Motor Driven Auxiliary Feedwater (MDAFW) pump, the second MDAFW pump provides the minimum required flow of 410 gpm to the minimum number of SGs. Since all four SGs are intact, any AFW flow diverted from the intended SG due a failed open check valve would still enter another intact SG and does not impact the event. For the LONF event, the first-off and second-off check valves only have a safety-related function to open.

The FLB analysis also assumes a minimum of 410 gpm to two intact SGs.

No AFW injection is credited until the faulted SG is isolated at ten minutes.

Since the FLB is a limiting loss of secondary heat transfer event, the analysis assumes that only the faulted SG blows down to minimize primary heat removal. Any check valve back flow from the three intact SGs prior to isolation of the faulted SG would actually provide additional primary heat removal and is not credited. For the FLB event, the first-off and second-off check valves only have a safety-related function to open.

The MSLB event establishes the maximum AFW flow imbalance assumed with respect to maximizing the mass and energy release from the faulted SG. Since by definition the maximum AFW flow assumes all AFW pumps are operating, all check valves experience positive AFW flow and backflow is not a credible failure. For the MSLB event, the first-off and second-off check valves only have a safety-related function to open.

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION 015 O FACILITY NAME (1) DOCKET NUMBER (2) 0 0 3 2 3 2003 LER NUMBER (6) 010 5 The MSLB core response event assumes the blowdown and heat transfer from a single SG maximizes the RCS cooldown, return to core power and the challenge to the fuel Departure from Nucleate Boiling Ratio (DNBR) limit. AFW injection does not impact this analysis, since the safety injection, boron addition, and peak DNBR occur in a very short time frame following accident initiation. The Diablo Canyon Power Plant design basis and the MSLB analysis bound the maximum RCS cooldown associated with the uncontrolled blowdown of one SG. Therefore, the second-off check valves must be credited to ensure that no back flow and no additional RCS heat removal occurs from the other three intact SGs. The backflow of the first-off check valve does not impact the event due to the second-off check valve credited with the safety-related function to prevent backflow.

The pump discharge check valves have the safety function of opening to permit flow of water from the AFW pump to the SGs. During normal plant operation, when the AFW system is not in operation, these check valves form the third in the series of check valves that together serve to prevent backflow of feedwater to the AFW pumps. This is to prevent overpressurization of the AFW pump suction piping. Back leakage prevention is desirable, but is not a credited safety function for the discharge check valve since the AFW suction piping is protected against overpressurization from back leakage by relief valves RV-535 and RV-537. These relief valves were added in response to NRC IE Bulletin 85-01-Steam Binding of AFW Pumps.

To detect leakage and gross backflow through the first-off check valves temperature elements (TE) are positioned on the discharge line between the first-off and second-off check valve. The TEs are not credited with a safety function, but will alarm PK09-16 and notify the operators when line temperatures exceed 200 degrees Fahrenheit. The annunciator response procedure AR PK09-16 has the auxiliary building watch measure temperatures downstream of the LCVs at least once every 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> while the AFWS leakage high temperature alarm is in. If the temperature measured is 180 degrees Fahrenheit or greater, the corresponding AFW pump is run until the alarm clears.

The original valve installed at FW-2-377 was a Western Hydraulics (Borg-Warner) 3-inch swing check valve with a bonnet hung disk assembly. The design of this check valve is such that the bonnet and disk assembly can rotate 360 degrees within the valve body before the bonnet retainer ring is installed, thus creating the potential for the incorrect orientation. This is a unique characteristic to this particular manufacturer.

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) TS 3.7.5, "Auxiliary Feedwater System," requires three AFW trains to be OPERABLE in Modes 1, 2, and 3. T.S.3.7.5 Condition B requires entry into a 72-hour Action Statement (AS) in Modes 1, 2 or 3, in which the one AFW train that is inoperable must be repaired. TS.3.7.5 Condition C is entered when two AFW trains become inoperable in Modes 1, 2, or 3, at which time the AS is entered to be in Mode 3 in six hours and Mode 4 in eighteen hours.

B. Event Description

On April 4, 2003, at 1800 PST, with Unit 2 at 54 percent power two AFWS trains were declared inoperable for SG 2-1 requiring entry into the action statement for TS 3.7.5.C.

On April 4, 2003, at 1824 PST, Operations commenced shutdown of Unit 2.

On April 4, 2003, at 2041 PST, Operators made an 8-hour nonemergency notification per 10 CFR 50.72(b)(2)(i) and (b)(3)(ii)(A), reference Event #39735.

On April 4, 2003, at 2309 PST, Unit 2 entered Mode 3 (Hot Standby) satisfying TS 3.7.5.C.1 fifty-one minutes ahead of the 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> AS time limit.

On April 5, 2003, at 1043 PST, Unit 2 entered Mode 4 (Hot Shutdown) satisfying TS 3.7.5.C.2 one hour and 17 minutes ahead of the eighteen hour AS time limit, thus completing TS 3.7.5.C.

The following time line spells out the details of the event and why Operations made a prudent decision to enter TS 3.7.5.C.

On March 26, 2003, initial startup following refueling outage 11 (2R11), auxiliary feedwater temperature sensor TS-117, AFW to SG 2-1 brought in annunciator alarm PK09-16. This alarm informs the operators that the AFW line to SG 2-1 has backflow through the first-off check valve, a second-off and possibly a pump discharge valve. The Operations staff then utilized annunciator response procedure AR PK09-16 to verify the alarm and maintain AFWS integrity. This procedure was utilized to protect the AFWS from thermal stresses and steam binding until the source of the leak was identified. Once the high temperature was confirmed AFW Pump 2-2, was run for 20 minutes to clear the alarm.

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (8) 015 2 0 5 Diablo Canyon Unit 2 01010 3 3 2003 l � 6 I OF I 9 The alarm repeated shortly after being cleared. Another walkdown of the piping revealed leakage past the second-off check valve, FW-2-370. Several attempts were made to seat FW-2-370 and the high temperature alarm came back in each time. In order to clear the alarm Operations proceeded to run AFW Pump 2-2 approximately four times per shift.

The system engineer conducted further evaluation and concluded that the leakage was flowing back towards AFW Pump 2-2, which indicated leakage past the pump discharge check valve, FW-2-362. With data indicating leakage to the pump, the decision was made to declare one AFW train inoperable and correct the backflow problem.

On April 2, 2003 at 0437 PST the unit entered TS 3.7.5.B.1 AS, "one AFW train inoperable in Mode 1, 2, or 3," to complete repairs to FW-2-362 and FW-2-370 with a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> completion time. Both of these valves can be isolated while at power for repair or replacement.

The valve internals for FW-2-362 were inspected and it was found that the disk had signs of being warped. To repair the valve, both the disk and valve seat were lapped. The subsequent blue check of the valve revealed them to be flat. The valve was then reassembled and the leak test at normal operating temperature and pressure was satisfactory.

PG&E decided to replace the original Borg-Warner FW-2-370 with a new Anchor Darling valve. Radiography was performed on FW-2-370 to analyze the repair welds to assure the new valve met the credited safety requirements. The removed Borg-Warner was hydro tested in the maintenance shop to investigate the leak source. The hydro test revealed a leak in the seal weld at the body seat.

On April 4, 2003, radiography was performed on FW-2-377 and PG&E Engineers reviewing the radiograph concluded that the valve internals were rotated 180 degrees.

C. Status of Inoperable Structures, Systems, or Components that Contributed to the Event None.

D. Other Systems or Secondary Functions Affected

None.

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKET NUMBER (2) 0 3 FACILITY NAME (1) LER NUMBER (6) PAGE (3) 2 3 2003 0 0 I 5 0 � 1 7 OF 9

E. Method of Discovery

PG&E engineers reviewing radiography images determined the internals were installed backwards with the disk full open.

F. Operator Actions

While the radiograph showed the valve disk open, Operations could not be sure the valve would stay open and allow AFW design flow to SG 2-1.

Therefore, operators declared two AFWS trains to SG 2-1 inoperable, and entered TS 3.7.5.0 AS.

G. Safety System Responses

None.

Ill. � Cause of the Problem

A. Cause

Personnel Error. FW-2-377 is in a vertical run of pipe with flow in the upward direction. To install the bonnet-hung internals without scratching the seats, the internals must be inserted 90 degrees from installed position. Once inserted, valve internals must be rotated 90 degrees to the correct position. The contract workers performing the valve work, during � 1.

2R11 in February 2003, installed the valve internals and rotated them 90 degrees in the wrong direction.

The bonnet flow direction marking is a small hand scribed arrow on the top of the bonnet, which is rusted and hard to see. The workers did not effectively match mark or scribe an easily seen flow arrow, which would have helped prevent the backwards installation. The workers did not self verify that the direction of the internals matched the direction of the valve body.

B. Contributory Causes 1. Oversight or verification of the reassembly was inadequate.

Management relied upon skill-of-the craft for reassembly.

2. Procedural deficiency. The vendor manual and check valve maintenance procedure did not warn of all the complexities of valve reassembly in a vertical run of pipe that are inherent to the valve design, and could result in an inoperable check valve.

ION

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6)

IV. Assessment of Safety Consequences

All three check valves in the AFW main flow path have a safety function of opening to permit AFW flow from AFW Pump 2-2. The safety function was never compromised. Engineers determined upon removal of the valve that the disk was stuck open due to interference with the valve body. Because the valve could not close in the stuck open position the valve did not limit flow below its design basis value and the AFW flow to SG 2-1 was not adversely impacted.

The forward flow capabilities for all three check valves along with backflow prevention for the second-off check valves were verified during STP P-AFVV-21 and P-AFW-22. These surveillance tests were successfully completed on March 26 and 27, 2003, which indicated that check valve FW-2-377 was still capable of delivering the required flow with the 180 degree rotation of the valve internals.

Each of the four lines that connect to the main feedwater leads contain a single, normally open stop valve upstream of the first-off check valve for each AFW discharge line. These stop valves are considered the primary containment isolation point of the AFWS discharge lines and fulfill the AFWS containment isolation function. FW-2-377 is within the containment isolation boundary and as such, the valve is required to maintain pressure boundary integrity, but not act as a containment isolation valve.

To summarize the assessment of safety consequences, the check valves provided the necessary forward flow for the safety analysis and FW-2-362 and FW-2-370 were repaired prior to entering the AS for TS 3.7.5.C. At the time of the event all credited safety functions were fulfilled. The as-found condition of check valve FW-2-377 did not have the potential to restrict AFW flow.

This event was not a safety system functional failure and did not adversely affect the health and safety of the public.

V. Corrective Actions

A. � Immediate Corrective Actions The operations staff brought the unit off line and moved to Mode 5 (Cold Shutdown) to allow replacement of FW-2-377 with a valve, which has internals that can only be installed in the proper alignment.

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) 0 15 Diablo Canyon Unit 2 0 I 0 I 0 I 3 2 3 2003 0 5 I " " I 0 0 1 9 OF 9 B. � Corrective Actions to Prevent Recurrence 1. PG&E will revise MP M-51.14, "Generic Check Valve Inspection" to warn that some valve internals can be installed backwards.

2. PG&E will revise Vendor Technical Manual VTM-0017 to warn of valve internals that may be installed backwards.

3. PG&E will require verification steps during re-assembly of bonnet-hung check valves, including valve internal installation in proper flow direction, height orientation of discs and axial orientation.

4. PG&E will replace the four remaining Borg Warner bonnet hung check valves with a design that is not susceptible to this type of failure.

VI. � Additional Information

A. Failed Components

FW-2-377, Borg-Warner 3-Inch Swing Check Valve, Model #74330.

B. Previous Similar Events

Palo Verde discovered Borg-Warner Corp model 77700 check valve internals installed backwards. Internals match marks made before disassembly were to studs, which were removed and installed at different locations when reassembled. Reference LER 528-92005.

NRC Information Notice 89-62, "Malfunction of Borg-Warner Pressure Seal Bonnet Check Valves Caused by Vertical Misalignment of Disk". The information notice was intended to alert addressees of the potential malfunctioning of Borg-Warner pressure seal bonnet check valves caused by the misalignment of the valve disk. The disk assemblies of the malfunctioning check valves were positioned too low in the valve bodies due to an essential step missing from the vendor's reassembly procedure.