IR 05000397/1998020

Insp Rept 50-397/98-20 on 980824-0917.Violations Noted.Major Areas Inspected:Operations,Maint,Engineering & Plant Support

ML17284A790
Person / Time
Site:	Columbia
Issue date:	10/16/1998
From:	NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML17284A788	List: IR 05000397/1998020 ML17284A787 ML17284A789
References
50-397-98-20, NUDOCS 9810220289
Download: ML17284A790 (80)

Text

Docket No.:

License No.:

Report No.:

licensee:

Facility:

Location:

Dates:

Inspector(s):

Approved By:

ENCLOSURE 2 U.S. NUCLEAR REGULATORYCOMMISSION

REGION IV

50-397 NPF-21 50-397/98-20 Washington Public Power Supply System Washington Nuclear Project-2 Richland, Washington August 24 through September 17, 1998 S. A. Boynton, Senior Resident Inspector J. E. Spets, Resident Inspector G. W. Johnston, Senior Project Engineer D. B. Pereira, Reactor Inspector, Engineering Branch C. J. Paulk, Senior Reactor Inspector, Maintenance Branch G. A. Pick, Acting Chief, Reactor Projects Branch E ATTACHMENTS:

Attachment 1: Supplemental Information Attachment 2: Flooding Event Root Cause Analysis 9810220289 98i016 PDR ADQCK 05000397

PDR

EXECUTIVE SUMMARY Washington Nuclear Project-2 NRC Inspection Report 50-397/98-20 This inspection was performed as a special followup inspection for the issues identified in NRC Inspection Report 50-397/98-16, which documented the Augmented Inspection Team evaluation of the June 17, 1998, flood event.

~Oerattons

~

As a result of human error, the watertight door between the reactor building northeast stairwell and residual heat removal pump Room C was left open prior to the flooding event. The open door resulted in substantial flooding of Room C, rendering Residual Heat Removal C inoperable and complicating operator recovery from the event.

Although no specific violation was identified for this deficiency, the inspectors confirmed that the licensee implemented human factors improvements to help personnel positively ensure door closure (Section III).

the flood Maintenance The actions of the operators to start the low pressure core spray pump during the flooding event, while in compliance with the wording of plant procedures, did not display conservative decision making. Although the actions were an attempt to maintain the maximum number of operable/available emergency core cooling system:pumps, the operators failed to recognize that other potential effects could have occurred because of ing (Section V.2).

~

The licensee failed to assign a level of importance to the emergency core cooling system pump room floor drain cross-connect valves that was commensurate with their design function. As a result, the maintenance and surveillance program for ensuring their reliability, when called upon to perform that function, was inadequate as evidenced by the failure of Valve FDR-V-609, residual heat removal pump Room C and low pressure core spray pump room floor drain cross-connect, during the flooding event.

The failure of Valve FDR-V-609 to perform its intended function resulted in the flooding of the low pressure core spray pump room and complicated, recovery from the plant transient.

The failure to monitor the performance of the valves against established goals or to demonstrate reliability of the valves through an effective preventive maintenance program was identified as a violation of 10 CFR 50.65 (Section II.1.b.2, EA 98-452).

~

The maintenance and surveillance program established for the watertight doors was found to be effective in ensuring that, when called upon, the doors would reliably perform their design function. During the flooding event, the doors successfully prevented common mode flooding into the high pressure core spray and reactor core isolation cooling system pump rooms (Section II.2.b.2).

~En ineerin

'I The root cause evaluation for the flooding event accurately concluded that the event resulted from design inadequacies of the fire protection water supply system.

Those

-2-inadequacies allowed for the generation of destructive forces within the system that ultimately failed Yalve FP-Y-29D, reactor building fire protection standpipe isolation.

The design inadequacies were attributed, in part, to noncompliances related to the installation of the fire pumps as specified in the National Fire Protection Association code.

The failure of the fire protection system pressure boundary upon a demand actuation would preclude the ability of the system to provide an adequate capacity of water to suppress a postulated fire and was identified as an apparent violation of 10 CFR Part 50, Appendix A, General Design Criterion 3, "Fire Protection" (Section l.1, EA 98-480).

The unexpected actuation of the Division II emergency diesel generator room preaction system (System 81) on June 17 was attributed to a material condition deficiency of the supply line check valve for the priming chamber of the System 81 preaction valve. The actuation of this second preaction system did not contribute to the severity of the hydraulic transient.

Subsequent modifications made to the priming chamber supply lines to all plant preaction valves were found to be appropriate.

Prior identification of the deficiency would not have been expected based upon the historical performance of the valves and their established maintenance and surveillance program (Section I.2).

The discrepancy between the actual performance of the reactor building watertight doors and their description in the Final Safety Analysis Report as being watertight was previously identified and analyzed by the licensee.

Although the analyses were found to be technically sound in concluding that the doors could continue to perform their function with the amount of leakage predicted, they did not result in appropriate changes to the Final Safety Analysis Report. The licensee identified this discrepancy during its followup to the flooding event and initiated appropriate action to address it. A noncited violation of 10 CFR 50.71(e) was identified for failure to update the Final Safety Arialysis Report in accordance with Section VII.B.1 of the Enforcement Policy (Section ll.2.b.1).

The fire protection corrective action program was ineffective in addressing water hammer in the fire protection water supply system.

The corrective actions taken in 1984 for known water hammer concerns were only partially effective in addressing the impact of hydraulic transients resulting from system initiation. Subsequent indications of severe hydraulic transients in the fire protection system were not evaluated and resultant component failures were treated as isolated maintenance items. These component failures and industry operating experience on water hammer both represented missed opportunities to ferret out continuing system design problems (Section IV.b..1).

The corrective actions from previous inadvertent actuations of the fire protection system were either ineffective in addressing personnel knowledge and procedure weaknesses in the ignition source permit process or not promptly implemented.

The inadvertent actuation of the diesel generator building corridor preaction system (System 66) on June 17, occurred over 4 months after an almost identical event in February 1998.

Although procedural enhancements were defined shortly after the event, the implementation of the enhancements was not scheduled until as late as August 199 A violation of the fire protection corrective action program was identified; however, because the corrective actions for the violation were appropriate, no response is required (Section IV.b.2).

The licensee responded well to the flooding event.

The shift manager made an appropriate decision to declare an Unusual Event and activate the onsite emergency response organization to quickly bring resources to bear on this complex event.

Declaration and notification of the emergency were both timely (Section V.1).

The Technical Support Center manager failed to confer with the emergency director prior to authorizing the discharge of the stairwell floodwater to the storm drains. The error was the result of the improper placement of an emergency response requirement into an operations procedure instead of the emergency plan implementing procedures.

The corrective actions taken to address this deficiency and evaluate the generic implications were appropriate.

A noncited violation of Technical Specification 5.4.1.a was identified for failure to followprocedure in accordance with Section VII.B.1 of the Enforcement Policy (Section V.3.b.1).

The licensee took appropriate measures to monitor the discharge of the stairwell floodwater to the storm drains to verify compliance with the National Pollutant Discharge Elimination System Wastewater Discharge Permit (Section V.3.b.2).

Because of competing priorities in responding to the fire protection system rupture and flooding event, required fire watches were not established within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> of the system impairment. The delay of approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> in implementing the compensatory measures was found to be reasonable based upon the nature of the event. A second example of a failure to implement compensatory measures for a fire protection system impairment was identified by the inspectors during planned corrective maintenance on June 26. A violation of Technical Specification 5.4.1.d was identified for failure to follow fire protection program implementing procedures; however, because the corrective actions were appropriate to address the root cause, no response to this violation is required (Section V.4).

Re ort Details

'ntroduction On June 17, 1998, a water hammer in the fire protection system caused the rupture of a fire main valve in the northeast stairwell of the reactor building. The resulting flood water entered into residual heat removal (RHR) pump Room C through a watertight door, which had been left in an unsecured condition. The water propagated to the adjacent low pressure core spray (LPCS) pump room via a sump isolation valve that failed to close, as designed.

The flood completely submerged the RHR pump and motor and the Division I keepfill pump (serves RHR B). The level in the LPCS pump room rose to just below the pump motor and completely submerged the Division II keepfill pump (serves RHR A).

The facts surrounding the flooding event were previously documented in NRC Augmented Inspection Team Report 50-397/98-16.

From that report, a number of questions were raised with regards to: (1) adequacy of the design of the fire protection system to preclude the generation of destructive forces during anticipated system transients, (2) adequacy of the design and maintenance of plant equipment credited for protection against internal flooding, (3) adequacy of corrective actions to address previous system actuations and water hammer events in the fire protection system, and (4) appropriateness of actions taken leading to the event initiation and in response to the event.

The purpose of this inspection was to answer those questions and to characterize the underlying issues within the regulatory framework of the license.

As described in NRC Inspection Report 50-397/98-16, the flooding event did not pose a risk to the public health and safety and the actual safety consequences were low. However, the potential safety'consequences were considered to be more significant. Had the fire protection system actuation been the result of an actual developed fire and had it occurred while the plant was operating at power, operators would have been faced with mitigating two separate events (fire and flooding) while also tasked with a plant shutdown in accordance with emergency operating procedures (EOP). The event could have been further complicated had it occurred outside of normal working hours when personnel resources to mitigate the event would have been minimal.

I. Fire Protection S stem I.1 Desi n of the Fire Protection Water Su I

S stem a.

Ins ection Sco e 92902 92903 Closed Ins ection Followu Item IFI 50-397/9816-14:

The inspectors reviewed the root cause analysis for the flooding event (refer to Attachment 2) with particular emphasis on the role design of the fire protection water supply system had in, contributing to the event. The review also included the fire protection evaluation provided in the WNP-2 Final Safety Analysis Report (FSAR), Appendix F, and compliance with National Fire Protection Association (NFPA) code requirement b.

Observations and Findin s-2-

The licensee concluded that the root cause was an "inadequate fire protection system design in that the system is configured such that destructive forces are generated during an anticipated challenge with only the jockey pump running." The June 17 event chronology illustrates the following:

~

Just prior to the event, fire protection system pressure is being maintained by a 220 gpm jockey pump at approximately 150 psig (typical standby lineup).

~

Smoke in the diesel generator building corridor actuates the diesel generator building corridor preaction system (System 66), placing a substantial demand on the system to fillthe preaction sprinkler piping.

~

Inabilityof the jockey pump to meet the demand leads to rapid depressurization of the fire protection system to a low of approximately 35 psig and the voiding of the reactor building standpipes, which are acting as surge volumes for the demand (column separation).

The depressurization leads to the simultaneous start of both motor-driven fire pumps (FP-P-2A, FP-P-2B) and a diesel fire pump (FP-P-110).

~

As the pumps start, rapid reflooding of the fire protection piping occurs, collapsing the voids in the reactor building standpipes.

~

Void collapse generates excessive forces in the reactor building northeast standpipe that are translated to a torsional stress on Valve FP-P-29D, reactor building fire protection standpipe isolation, which fails the cast iron valve.

Subsequent modeling and analyses of the fire protection system performed by Bechtel Corporation for the licensee showed close correlation with the event and accurately predicted the high stresses.

The functional requirements of the WNP-2 fire protection system are described in 10 CFR 50.48; 10 CFR Part 50, Appendix A, General Design Criterion 3, "Fire Protection"; Branch Technical Position APCSB 9.5-1, Appendix A, "Guidelines for Fire Protection for Nuclear Power Plants Docketed Prior to July 1, 1976"; and the applicable NFPA codes incorporated by reference in the Branch Technical Position.

Licensee implementation of these requirements is described in Appendix F of the FSAR.

The fire protection water supply system consists of two redundant water supplies that feed the plant main fire loop through a jockey pump and four main fire pumps.

Three of the pumps (FP-P-2A, FP-P-2B, and FP-P-1) take suction from the circulating water basin and are each rated at 2000 gpm. Pumps FP-P-2A and FP-P-2B are motor-driven and are the normal supply pumps.

Pump FP-P-1 is a diesel-driven pump to ensure availability of fire water on a loss of offsite power. A fourth pump (FP-P-110) takes suction from the backup water supply (an onsite tank), is diesel-driven, and is rated at 2500 gpm. Allof the pumps accommodate system demand through a starting sequence based upon system pressure.

As pressure decreases, the pumps willsequence start with Pump FP-P-2A starting at 120 psig, Pump FP-P-2B starting at 110 psig, and

~

FP-P-110 starting at 100 psig. After a 30 second time delay with system pressure less than 110 psig, Pump FP-P-1 willalso start.

In 1984, the licensee determined that the staggering of pump starts on pressure generated significant hydraulic forces when large demands were placed on the fire protection system.

Specifically, it was found that upon actuation of preaction or deluge sprinkler systems, the pressure drop in the fire main caused multiple fire pumps to start in rapid succession.

To alleviate the observed water hammer from these events, the licensee installed bypass lines on the discharge of the motor-driven fire pumps to return a portion of the flowfrom the pumps back to the basin during initial pump operation.

A normally open diaphragm valve installed in the bypass lines would slowly close upon pump start to ultimately direct all of the pump flow to the fire main (within 10 - 30 seconds).

No delay features were installed to address pressure surges created by the start of Pump FP-P-110.

As described in the FSAR, Section F.2.1, and Table F.2-4, the fire pumps were installed in accordance with NFPA 20-1974, "Standard for the Installation of Centrifugal Fire Pumps."

Section 7-5 of NFPA 20 states that the controller for each unit of multiple pump units shall incorporate a sequential timing device to prevent any one pump from starting simultaneously with any other pump. The NFPA Fire Pump Handbook elaborates on this requirement by noting that the sequence starting requirement prevents excessive loading of the electrical source (for motor-driven pumps) or excessive hydraulic stress to piping, valves, and other system components during pump acceleration.

With the exception of Pump FP-P-1, none of the main fire pumps utilize time delay sequencing.

Although the installation of the bypass lines on the discharge of the motor-driven fire pumps and pressure setpoint variations would provide a "staggering" effect on low system demand, the same would not be true when large demands are placed on the system.

As noted in the NFPA Fire Pump Handbook, sequence timers are required for fire protection systems with multiple pumps because staggering of the pressure switches willonly sequence the pumps when very low water flows exist. At nigher flows, multiple pumps would otherwise start at the same time. The lack of time delay sequencing on the fire pumps was not technically justified in the FSAR as a deviation from NFPA 20.

In the. root cause analysis, the licensee noted that the water hammer occurred within 6 seconds of event initiation. At that time, both motor-driven fire pumps were operating at close to runout conditions with their discharge bypass lines at least partially open and diverting a portion of the flowfrom the pumps.

Pump FP-P-110 was also operating at close to runout conditions.

Based upon the diversion of flowfrom the discharge of the motor-driven pumps, the licensee concluded that Pump FP-P-110 was the sole contributor to the void collapse and water hammer.

As such, the licensee concluded that pump sequencing did not appear to be an event contributor (i.e., a single pump starting could generate sufficient hydraulic forces to fail the system).

However, this conclusion was based upon qualitative information. The licensee did not have supporting data to show how much of the flowwas being diverted to demonstrate that the pumps did not contribute to the reflood of the standpipe.

Additionally, time delay sequencing was not analyzed using the Bechtel model to demonstrate that destructive

-4-forces would be generated even with sequencing.

The inspectors found that the conclusion on the impact of pump sequencing was not supported by an adequate technical basis.

With regards to the installation of the bypass lines on the discharge of the motor-driven fire pumps, the inspectors noted that NFPA 20, Section 2-10, requires a check valve to be installed in the pump discharge assembly.

For larger fire protection systems, the NFPA Fire Pump Handbook notes that the check valve may serve the purpose of protecting against water hammer generated by backflow when the pump is shut down.

The NFPA Fire Pump Handbook further states that "no device other than a listed antiwater hammer check valve is permitted to be installed to prevent-water hammer."

In 1983, a formal interpretation was made by the NFPA code committee on this issue (Formal Interpretation 83-6A). As an example, the committee noted that it was unacceptable to install a slow-opening type of pressure regulating valve in the fire pump discharge line to prevent water hammer when the pump starts.

This conclusion was based upon the potential for the failure of the valve to open when required. The same argument may also be applied to the installed bypass line isolation valves since failure of these valves to close could prevent adequate discharge flowto the fire main.

The inspectors found that the root cause analysis was sound in its conclusion that the design of the fire protection water supply system is inadequate.

However, because the licensee had not yet completed its evaluation of the specific design aspects that require correction, it could not be concluded that the design deficiencies were fullyattributable to NFPA code noncompliance.

10 CFR Part 50, Appendix A, General Design Criterion 3, states that fire-fighting systems of appropriate capacity and capability be provided and designed to minimize the adverse effects of fires on structures, systems, and components (SSC) important to safety.

The ability of the fire protection water supply system design to generate hydraulic forces during expected system transients (e.g. preaction/deluge system actuation in response to a real fire) sufficient to rupture system piping is in direct contrast to the requirements of General Design Criterion 3. Any failure of system piping in response to an actual fire would have precluded the ability of the system to provide appropriate capability and capacity to minimize the adverse effects of the fire. Quantitatively this can be described as an inability to provide the design basis flow rate and volume of water (2350 gpm for 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />) because of the loss through the break. The design inadequacies in the fire protection water supply system were identified as an apparent violation of 10 CFR Part 50, Appendix A, General Design Criterion 3 (50-397/9820-01).

c.

Conclusions The root cause evaluation for the flooding event accurately concluded that the event resulted from design inadequacies of the fire protection water supply system.

Those inadequacies allowed for the generation of destructive forces within the system that ultimately failed Valve FP-V-29D. The design inadequacies were attributed, in part, to noncompliances related to the installation of the fire pumps as specified in the NFPA code.

The failure of the fire protection system pressure boundary upon a demand actuation would preclude the ability of the system to provide an appropriate capability

-5-and capacity to suppress a postulated fire and was identified as an apparent violation of 10 CFR Part 50, Appendix A, General Design Criterion 3.

Fire Protection S stem Preaction Valve Performance Ins ection Sco e 92902 92904 Closed IFls 50-397/9816-12 and -13: The inspectors reviewed the evaluation of the unexpected actuation of preaction System 81 during the flooding event.

The review also assessed the role of System 81 in contributing to the hydraulic transient placed on the fire protection water supply piping.

Observations and Findin s Preaction System 66 is actuated from area smoke detectors, and the Division II emergency diesel generator room preaction system (System 81) is actuated by heat sensors located above the diesel generator sot. These systems and their detectors are normally physically separated by fire-rated barriers. The actuation of System 66 on June 17 resulted from smoke generated by grinding operations migrating into the diesel generator corridor. However, a valid actuation signal for System 81 should not have been present.

The licensee documented the actuation of System 81 as a'sympathetic" actuation because. of a combination of the hydraulic transient placed on the fire protection system when System 66 initiated and as a material condition deficiency of the System 81 preaction valve priming chamber that allowed the chamber to depressurize.

As noted in NRC Inspection Report 50-397/98-13, the inspectors found the evaluation of the "sympathetic" actuation of System 81 to be thorough. Troubleshooting efforts clearly identified a deficiency with the check valves in the supply line to the preaction valve priming chamber.

As such, upon system depressurization, the preaction valve priming chamber would also depressurize.

Pressurization of the priming chamber is the driving force for maintaining the preaction valve latched closed.

During the fire protection system transient, the actuation of System 66 rapidly depressurized the fire main and, consequently, the priming chamber on the System 81 preaction valve. The rapid repressurization of the fire main provided sufficient force on the underside of the System 81 preaction valve to overcome the latch mechanism being held in place by the priming chamber pressure.

Tr correct the identified material deficiency in the check valves, the licensee installed a modification to the priming chamber supply lines on all preaction valves. This modification removed the old style, swing check valve, and installed two spring-closed check valves in series.

The modification was consistent with valve vendor's current design of available preaction valves. The use of two spring-closed check valves was considered to be an enhancement by the valve vendor, and as such, no generic communications were issued to customers to note a potential design deficiency with the use of a single swing check valve. Additionally, a review of the routine testing performed on the preaction valves by the licensee indicated that the testing would not have readily identified the problem with the check valve A review of licensee work records and problem evaluation requests (PERs) did not identify any other recent sympathetic actuations of preaction systems.

Neither the licensee nor the inspectors, could substantiate the, assertion made on a control room deficiency tag that, previous, sympathetic actuations of System 81 had occurred with the actuation of System 66. However, two sympathetic actuations of preaction systems were identified in 1984 and documented in Licensee Event Reports (LERs) 50-397/84-026 and 50-397/84-096.

In both cases, a valid fire protection system actuation resulted in unexpected actuations of preaction systems in the diesel generator building, which resulted from hydraulic transients.

These were the Division II fuel transfer pump room (System 82) and the Division III fuel transfer pump room (System 84). No documentation could be found indicating that the material condition of the preaction valves had been evaluated.

Therefore, it is indeterminate whether or not the priming chamber supply line check valves played a part in those events.

In reviewing the timeline of the flooding event, it is most likelythat System 81 actuated during the rapid reflooding and repressurization of the fire protection piping by the fire protection pumps.

Thus, the inspectors concluded that the actuation of System 81 did not significantly contribute to the severity of the transient.

Also, fire protection system analyses and testing performed subsequent to the event demonstrated that simultaneous actuation of two preaction systems does not appreciably increase the transient hydraulic. loads.

c.

Conclusions The unexpected actuation of preaction System 81 on June 17 was attributed to a material condition deficiency of the supply line check valve for the priming chamber of the System 81 preaction valve. The actuation of this second preaction system did not effectively contribute to the severity of the hydraulic transient.

Subsequent modifications made to the priming chamber supply lines to all plant preaction valves were found to be appropriate.

Prior identification of the deficiency would not have been expected based upon the historical performance of the valves and their established maintenance and surveillance program.

II. Internal Flood Protection II.1 Desi n and Maintenance of the Reactor Buildin Floor Drains Radioactive FDR

~Sstem a.

Ins ection Sco e 92903 Closed IFIs 50-397/9816-20-21 and -22: The inspectors reviewed the design of the reactor building floor drains radioactive (FDR) system to evaluate conformance with the plant licensing basis and design requirements.

The maintenance program for the FDR system and system material condition were also reviewed to determine adequacy of maintenance and testing to ensure system availability and reliabilityto perform its design functio Observations and Findin s b.1 S stem Desi n

Protection against internal flooding is described in FSAR Section 3.4.1 and in responses to NRC Questions Q.010.028 and Q.211.078.

Flood protection features provided by the FDR system are described in FSAR, Section 9.3.3. Sections 3.4.1 and 9.3.3 of the Safety Evaluation Report document the staff's acceptance of the internal flood protection features at WNP-2.

FSAR, Section 3.4.1.4.1.2, states, in part, that common mode flooding is prevented by the feature that the floor drain sumps serving more than one pump room have two isolation valves in the drain headers from adjacent pump rooms, which automatically close on a high sump level. In response to NRC Question Q.211.078, with regards to the design of the drain system to prevent common mode flooding in emergency core cooling system (ECCS) pump rooms, the licensee indicated that each reactor building sump serves up to two rooms with an isolation valve in the interconnecting piping.

The description of the reactor building FDR system in Section 9.3.3 of the Safety Evaluation Report agrees with the response to NRC Question Q.211.078 in stating that each room is equipped with its own sump and sump pumps with a maximum of one other room draining into the sump through an air-operated, fail-closed valve. Therefore, it was concluded that the staff accepted the application of a single isolation valve in the drain lines connecting ECCS pump rooms to preclude flooding from affecting redundant ECCS equipment.

The description of the reactor building FDR system in FSAR, Section 3.4.1.4.1.2, was determined to be inaccurate.

However, this error was considered typographical in nature and not material in the licensing process.

The licensee has established an action to correct the description in FSAR, Section 3.4.1 4.1.2.

b.2 FDR Valve Maintenance The failure of Valve FDR-V-609, residual heat removal pump Room C and low pressure core spray pump room floor drain cross-connect, to close during the event from both automatic and manual action resulted from long-term aging of Valve FDR-SPV-609, the solenoid-operated pilot valve that provides the closure mechanism for Valve FDR-V-609.

The licensee attributed the cause of the solenoid pilot valve failure to aging of the Buna-N diaphragm material on the pressure supply side, which can stiffen or harden from heat-induced aging. The loss of resilience of the diapnragm material resulted in a slow bleed-off of air pressure to a four-way shuttle valve, which ports compressed air from the accumulator to the air operator for Valve FDR-V-609. The net effect was that insufficient air was ported to the operator of Valve FDR-V-609 to allow it to close as designed.

The vendor had specified an ethylene propylene rubber material for this application; hence, the material found in the pressure side of Valve FDR-SPV-609 did not conform to the manufacturer's specification.

The cause of this condition was indeterminate.

No licensee record could be located that suggested a field change of the material and the vendor stated that the valve was not supplied in this configuration. The licensee

-8-did not retain the other solenoid pilot valves from the other two floor drain applications (FDR-SPV-607 and -608) when they were replaced following the event, so it was not possible to determine if the configuration was similar. The replacement solenoid pilot valves have Viton diaphragms that is environmentally qualified for the application.

The inspectors noted that the solenoid pilot valves had not experienced a maintenance-related failure for the 14-year installed life span of the valves. These solenoid pilot valves and one other in a test feature of the high pressure core spray system were the only use of this type of solenoid pilot valve in a nonsafety-related application.

From the review of a January 1998 maintenance activity associated with Valve FDR-V-609, the inspectors determined that the action to place the repair of Valve FDR-V-609 as a low prioritywas reasonable.

At that time, the valve was not identified as a Maintenance Rute-affected component, and the valve had required little corrective maintenance.

Additionally, the elimination of all preventive maintenance and surveillances on the FDR valves in 1995 clearly established the expectation to plant personnel that it was acceptable for the valves to run to failure. Although the low priority placed upon the repair of Valve FDR-V-609 was considered reasonable in light of established expectations, the low priority and the expectations were not commensurate with the importance of the valve design function as described in the FSAR.

As discussed previously, Valve FDR-V-609 failed to close as designed during the flooding event. This resulted in flooding of the LPCS pump room and rendered the LPCS system and Division'I keepfill system inoperable.

In light of the design function of the drain line cross-connect valves, the inspectors evaluated the adequacy of the maintenance program established for the valves. The inspectors reviewed the scoping criteria for applicability to the sump cross-connect valves (FDR-V-607, -608; and -609),

as set forth in 10 CFR 50.65(b)(2) for nonsafety-related SSCs and discussed in NUMARC 93-01, "Industry Guideline for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants," Revision 2.

As described in FSAR, Section 3.4.1.4.1.2, the sun>p cross-connect valves I ~iiorm a mitigative function to limitthe extent of flooding in the reactor building. Proper operation of these valves is relied upon by Emergency Operating Procedure (EOP) 5.3.1,

"Secondary Containment Control," in that they are required to isolate the floor drain system if it is discharging into an affected pump room.

In addition, Abnormal Condition Procedure 4.602.A13, "602.A13 Annunciator Panel Alarms," Revision 9, directs operators to verify that Valve FDR-V-609 has closed in response to a high water level (high-high alarm) in the Reactor Building R4 sump.

10 CFR 50.65(b)(2)(l) states that the scope of the maintenance monitoring program shall include nonsafety-related SSCs that are relied upon to mitigate accidents or transients or are used in plant EOPs.

As described in Regulatory Guide 1.160,

"Monitoring the Effectiveness of Maintenance at Nuclear Power Plants," Revision 2, and NUMARC 93-01, 10 CFR 50.65(b)(2)(I), would apply to those SSCs utilized to mitigate accidents or transients describe in the FSAR or those explicitly utilized in the EOPs in accomplishing a significant fraction of the mitigating function. Regulatory Guide 1.160 further. states that SSCs not explicitly mentioned in the EOPs may also be within scope if they are needed to successfully mitigate an accident or transient, Although the floor

-9-drain cross-connect valves are not explicitly mentioned in EOP 5.3.1, the inspectors concluded that they do perform a significant fraction of the mitigating function for reactor building internal flooding, as described in FSAR Section 3.4.1.4.1.2.

Therefore," the sump cross-connect valves should have been in the scope ot the Maintenance Rule monitoring program. The failure to include the sump cross-connect valves in the scope of the Maintenance Rule program was identified as a violation of 10 CFR 50.65(b)(2)(l)

(50-397/9820-02).

Subsequent to the flooding event, the licensee closed the sump cross-connect valves and maintained them closed to ensure ECCS pump room separation for flooding. While in this condition, the Maintenance Rule expert panel determined that the valves are not required to be monitored under 10 CFR 50.65.

However, if the configuration was changed, the expert panel would reconsider the need for the sump cross-connect valves to be in the scope of the Maintenance Rute program.

Additionally, the licensee had initiated a review of the plant flooding analysis to determine ifa change to the functional requirements of the cross-connect valves is warranted.

Conclusions The licensee failed to assign a level of importance to the emergency core cooling system pump room floor drain cross-connect valves that was commensurate with their design function. As a result, the maintenance and surveillance program for ensuring tlieir reliability, when called upon to perform that function, was inadequate as evidenced by the failure of Valve FDR-V-609 during the flooding event. The failure of Valve FDR-V-609 to perform its intended function.resulted in the flooding of the LPCS pump room and complicated recovery from the plant transient.

The failure to monitor the performance of the valves against established goals or to demonstrate reliabilityof the valves through an effective preventive maintenance program was identified as a violation of 10 CFR 50.65.

Desi n and Maintenance of the Reactor Buildin Waterti ht Doors Ins ection Sco e 92902 92903

Closed IFls 50-397/9816-09-10 -15 -17 -18 and -19: The inspectors reviewed the design requirements of the reactor building watertight doors to protect the ECCS pump rooms from internal flooding events.

The maintenance and material condition of the doors were also reviewed to determine the adequacy of the maintenance program to ensure the doors can perform their design function.

Observations and Findin s Waterti ht Door Desi n

FSAR, Section 10.4.5.3, states, with respect to flooding in reactor building stairwells,

"... the access doors to the ECCS pump rooms at Elevation 422 ft are sealed watertight and designed to withstand a static head of 44 ft (measured from the centerline of the door) of water.

Allpenetrations into the reactor building below the 486.65-ft elevation are water resistant.

Water would not affect any safety-related

-10-equipment in the reactor building."

The staff's Safety Evaluation Report, Section 10.4.5, also indicated that ECCS pump rooms are designed to be watertight for stairwell flooding. FSAR, Table F.3-1, states, "Water flowing down stairwells or into elevator shafts will not degrade safety-related equipment."

In 1982, during facilityconstruction, the testing of these doors found that leakage would occur when the doors were subjected to pressure in the direction that tended to unseat the door gaskets.

The doors to the ECCS pump rooms had been specified to be watertight from both directions. To address the unanticipated leakage, the licensee purchased additional doors for installation between ECCS pump rooms. Therefore, each passage between pump rooms would have doors on both sides of the passage, providing a positive seal from leaks in each room. However, the passages between the stairwells and the individual pump rooms retained a single door for a flooding barrier.

The stairwell doors were determined to be acceptable as-is because of redu'ndancy in the ECCS. That is, the loss of pumps adjoining any single stairwell would not prevent the safe shutdown of the plant in response to a postulated flood. The FSAR was not updated to reflect actual performance of the doors.

LER 50-397/92-034-02 identified penetration seals in the ECCS rooms that were not barriers to flooding. As a result of this LER, the licensee also re-evaluated the design of the doors and developed Calculation ME 02-93-57 to determine expected leakage rates past the doors in the stairwells.

From the results of the calculation, the licensee concluded that leakage through the doors from the stairwells would be no more than 70 gpm and would not challenge the operability of the ECCS pumps. The calculation assumed that hourly flood watches would detect a flood with sufficient time to mitigate its effects through operator action. However, the FSAR again was not updated to reflect the flood protection strategy and the expected performance of the doors.

Notwithstanding the results of Calculation ME 02-93-57, in May 1994 the licensee identified that a common mode ECCS failure was possible when flooding in the reactor building northeast stairwell would leak past the watertight doors into RHR pump Room C and the LPCS pump rooms that could result in loss of the Division I and II ECCS water leg (keepfill) pumps located in those rooms. Without operator action, loss of the Division I and II keepfill pumps could result in depressurization of all of the low pressure ECCS and render them inoperable.

This condition was documented in Problem Evaluation Request (PER) 294-0463 and was resolved with Calculation Modification Record 94-0507 issued in June 1994. The calculation modification record determined the* operator action could mitigate a break in sufficient time prinr to affecting the operability of the ECCS water leg pumps. The licensee determined that leakage through the doors was acceptable and that existing plant design continued to provide sufficient protection against stairwell flooding.

The inspectors reviewed the existing flooding analysis, including Calculation ME 02-93-57 and Calculation Modification Record 94-0507, and found that the assumptions utilized and the conclusions reached were technically sound.

A review of the moderate energy line break analysis also found that design basis flooding rates into the northeast stairwell would be around 300 gpm (1/30th of the rate observed on June 17). Based upon'the design basis leakage into the stairwell coupled with the assumed leakage rate past the pump room doors, the inspectors agreed that ample time (in excess of 30 minutes) would

-11-be available to successfully mitigate such an event through operator action. Therefore, the inspectors concluded that the defined leakage past the watertight doors is acceptable in that the doors can perform their design function in response to a design basis flooding event and that sufficient equipment would be. available to bring the plant to a safe shutdown condition.

However, the inspectors also concluded that the predicted performance of the watertight doors and flooding mitigation strategies (e.g., starting of associated ECCS pumps on potential loss of keepfill pump) were not consistent with the description in the FSAR. The licensee had also identified these discrepancies and had initiated action to review the existing flooding analysis to determine if any corrective actions are needed.

This corrective action was considered appropriate to address the issue.

10 CFR 50.71(e)

requires licensees to periodically update the FSAR to assure that the information included in the FSAR contains the latest material developed.

The failure to update the FSAR to reflect the latest flooding analysis assumptions that were developed between 1982 and 1994 was identified as a violation of 10 CFR 50.71(e).

This nonrepetitive, licensee-identified and corrected violation is being treated as a noncited violation, consistent with Section VII.B.1 of the NRC Enforcement Policy (50-397/9820-03).

b.2 Waterti ht Door Material Condition and Maintenance Based upon the design requirements of the watertight doors, the actual performance of the doors during the flooding event raised questions as to the material condition of the doors and their seals and the adequacy of the maintenance and surveillance program established for them. As described in NRC Inspection Report 50-397/98-16, leakage

'as identified past the door seals on the watertight doors leading into the passages to the high pressure core spray and reactor core isolation cooling pump rooms. A small amount of leakage (less than 1 gpm) eventually migrated into the reactor core isolation cooling pump room.

It was also noted that the watertight door between the stairwell and RHR pump Room C had a relatively freewheeling closing mechanism.

Based upon the quantified leakage past the watertight doors that was observed during the flooding event, with the exception of the door between the northeast stairwell and the RHR pump Room C (Door R13), the inspectors concluded that each of the doors impacted by the flooding successfully performed its design function. That is, the minimal leakage into adjacent areas separated by the watertight doors did not represent a credible threat to the equipment in those areas.

However, the failure of Door R13 to perform its function substantially complicated the event and resulted in the inoperability of Rl-'R C.

The licensee monitors the performance of the watertight doors and the effectiveness of their preventive maintenance program in accordance with 10 CFR 50.65.

In testing the performance of the doors, the licensee performs a "paper test" of the sealing surfaces of the doors.

This test utilizes small widths of paper inserted between the knife edge of the door and its seal.

Upon closing and dogging the door, an attempt is made to remove the paper.

Ifthe paper is firmlyheld by the established door seal, the sealing mechanism is considered to be acceptable.

This procedure is performed around the entire perimeter of each of the doors.

Although the procedure does not verify zero leakage past the seals, the inspectors concluded that, if successfully completed, it provided for a reasonable

-12-C.

expectation that the doors can perform their design function of mitigating flooding events as defined by the current flooding analysis.

This conclusion is supported by the actual performance of the doors that remained closed during the flooding event.

In regards to Door R13 being open during the flooding event, the inspectors discussed the door mechanism characteristics with the system engineer, noting the light forces required to engage the door bolts in the door frame strikes. The engineer identified that the door was not much lighter in terms of closure forces than the other watertight doors.

The engineer stated that the door, if properly operated, would not open of its own accord and that the closure of the door has a positive feedback from the latch bolts engaging the flat portion in the door frame strikes.

From personal observation, the inspectors determined that other doors did, in fact, exhibit light.closure forces. Additionally, no evidence could be found to suggest that the light closure forces would result in any door opening without operator action. Therefore, the inspectors concluded that the freewheeling closure mechanism observed by plant personnel and the inspectors, by itself, did not represent a material condition deficiency. See Section III regarding the root cause of Door R13 being open.

Conclusions The discrepancy between the actual performance of the reactor building watertight doors and their description in the FSAR as being watertight was previously identified and analyzed by the licensee.

Although the analyses were found to be technically sound in concluding that the doors could continue to perform their function with the amount of leakage predicted, they did not result in appropriate changes to the FSAR. The licensee identified this discrepancy during its followup to the flooding event and initiated appropriate action to address it. A noncited violation of 10 CFR 50.71(e) was identified for failure to update the FSAR.

The maintenance and surveillance program established for the watertight doors was found to be effective in ensuring that, when called upon, the doors would reliably perform their design function. During the flooding event, the ooors successfully prevented common mode flooding into the high pressure core spray and reactor core isolation

~

cooling system pump rooms.

III. Human Performance III Failure to Close Waterti ht Door to RHR Pum Room C a.

Ins ection Sco e 92901 Closed IFI 50-397/9816-16:

During recovery actions from the flooding event, the licensee identified that the watertight door between the reactor building northeast stairwell and RHR pump Room C was open. The inspectors reviewed the information associated with the open watertight door and evaluated the corrective actions to address the root caus Observations and Findin s As described in the FSAR, the watertight doors to-the ECCS pump rooms are a structural design feature of the internal flood protection scheme.

FSAR, Section 3.4.1, states that,

"administrative controls assure that separation criteria is maintained and watertight doors and hatches are closed as appropriate."

The established administrative controls included:

(1) general employee training, which discusses the need to ensure fire doors are maintained closed (the watertight doors are also credited as fire doors); (2) weekly fire door inspections; and (3) equipment operator and fire tours that check the status of doors encountered but not necessarily all of the doors.

Following the flooding event, the licensee identified through the control room alarm printer that the door to the stairwell of RHR pump Room C had been opened approximately 30 minutes prior to the event.

Procedure 1.3.57, "Barrier Impairment,"

Revision 11, would normally require an approved impairment to leave this door opened and unattended.

A barrier impairment had not been issued; consequently, the licensee concluded that the door had been left open as a result of human error. Subsequently, the licensee implemented human factor improvements for the door closing mechanisms by applying labels to the handwheel operator that indicate when full engagement of the latch pins is achieved.

A training video was also developed to provide a visual training aid on operation of the watertight doors.

This video was viewed by all site personnel with plant access authorization.

Because the issuance of a barrier impairment would allow an open watertight door to be left unattended, the issuance of an impairment for the RHR pump Room C door would not have necessarily precluded the migration of flooding into the room. The inspectors agreed that the root cause of the door being open was human error and that a combination of insufficient training and poor visual indicators of proper door closure contributed to that error. The corrective actions implemented were found to be appropriate to address the underlying concerns.

The failure'to close the reactor building watertight doors had been previously identified as a concern, as documented in NRC Inspection Report 50-397/95-09, dated May 22, 1995, (PER 295-0333) and PER 297-0040, dated January 13, 1997.

However, neither of these events was identified as a significant condition adverse to quality and, as such, root cause analyses were not required to be performed. The inspectors reviewed those events and agreed with the conclusions that they were not significant and, therefore, did not require corrective actions to be developed to prevent recurrence.

However, it was recognized that the failure to adequately address underlying performance issues of training and operation of the watertight doors in response to the previous events was a potential contributor to the RHR pump Room C door being open prior to the event.

Conclusions As a result of human error, the watertight door between the reactor building northeast stairwell and RHR pump Room C was left open prior to the flooding event. The open door resulted in substantial flooding of RHR pump Room C, rendering RHR C inoperable and complicating event recover IV. Problem Identification and Resolution IV Fire Protection Corrective Action Pro ram.

a.

Ins ection Sco e 92904 Closed IFls 50-397/9816-05 and -11: The inspectors reviewed the corrective actions for the fire protection program issues as related to the fire protection system actuation and subsequent fire main rupture and flooding event. The review focused upon licensee performance in identifying, reporting, and correcting conditions adverse to quality related to the fire protection system design and the control of ignition sources.

b.

Observations and Findin s II The fire protection program is implemented in accordance with License Condition 2.C.(14). To address conditions identified for the fire protection program that are adverse to quality, the licensee has committed to implementing the WNP-2 Operational Quality Assurance Program.

In conformance with the program, conditions adverse to fire protection, such as failures, malfunctions, deficiencies, deviations, defective components, uncontrolled combustible material, and nonconformances, are to be promptly identified, reported, and corrected.

b.1 Fire Protection Water Su I

S stem Desi n lnade uacies In 1984, the licensee documented design problems with the fire protection system in LER 50-397/84-026.

In LER 50-397/84-026, the licensee noted that "the activation of preaction and deluge portions of the fire protection system have been causing pressure surges in the (system).

The surges have been of sufficient magnitude to unseat the standby gas treatment system deluge valves." To reduce the pressure surges and preclude damage to the fire protection system because of water hammer, the licensee implemented Project Engineering Directive S215-M-7376.

This rriodification installed bypass piping on the discharge of the two motor-driven fire pumps and utilized a diaphragm-operated valve in the line to regulate the discharge flowto the main loop.

After the pumps initiallystart with the discharge bypass valves fullyopen, the bypass valves slowly close to ultimately direct all pump flowto the main loop. Diesel-driven Pump FP-P-110 apparently did not contribute to the events associated with LER 50-397/84-026 since no modifications were made to its starting sequence.

Also, the documentation associated with the modification did not discuss the NFPA 20 requirement for time delay sequencing of the pumps.

Although detailed documentation of fire protection system component failures was found to be incomplete, additional system actuations had occurred that resulted in hydraulic forces sufficient to damage fire protection system components:

On March 9, 1990, the licensee incorrectly conducted a fire pump monthly operability test in accordance with Procedure 15.1.6, "Fire Pump 2B Monthly Operability," which caused the cable spreading room system (System 65) to actuate.

The licensee documented this event in PER 290-0141.

Post-event walkdowns of the fire protection piping found some damaged gage On May 5, 1992, the Division I emergency diesel generator room system (System 79) actuated because of manual pull actuation from an unknown cause, which started Pumps FP-P-110, FP-P-2A, and FP-P-2B. The licensee documented this event in PER 292-0417.

The licensee found that a seal supply strainer off of a blowdown line on Pump FP-P-2A was broken off.

~

On February 5, 1998, preaction System 66 actuated because of smoke generated during grinding operations.

The licensee documented this event in PER 298-0112.

The hydraulic transient from the event resulted in a valve packing leak in the cable spreading room and damage to the reactor building northeast riser pressure gage.

The frequent failure of pressure gages in the fire protection system was identified as a potential indicator of design deficiencies in the system exhibiting themselves through excessive forces on system components.

Since 1985, over 300 gages of the type utilized in the fire protection system had been issued from inventory. However, it was found through the limited documentation and discussions with fire protection personnel that gage replacement resulted from a variety of causes.

Only a few of the gage replacements could be directly linked to hydraulic transients in the fire protection system.

The lack of specific information precluded the ability to identify a clear adverse trend related to gage failures during system transients and resulted from the established corrective action program thresholds, which were not expected to track gage failures for trending purposes.

Another opportunity to collect and evaluate information on the performance of the fire protection system was available to the licensee through review of NRC Information Notice 91-50, Supplement 1, "Water Hammer Events Since 1991," dated July 17, 1997.

The notice described a number of industry events where water hammer had occurred in both high and low temperature systems and/or high and low pressure systems (as a result of a variety of causes) for safety-and nonsafety-related systems.

These events indicated potentially inadequate root cause evaluations and corrective actions applied to the design and/or operation of the systems.

The notice further stated that the more severe loading events resulted from column separation/void collapse (same phenomenon that occurred in the fire protection system).

The licensee did not evaluate the fire protection system as part of its review of the applicability of Information Notice 91-50.

b.2 Inadvertent Fire Protection S stem Actuations Durin Hotwork PER 295-0423, dated April29, 1995, was initiated for an inadvertent actuation of fire protection preaction System 66 because of smoke generated by welding in the general area.

As stated in Corrective Action Plan 1 of PER 295-0423, "the Fire Marshall reports that these incidents are not restricted to outages and have occurred at other times." The corrective actions for this event were narrowly focused on training Raytheon contractors on the requirements of Procedure 1.3.10A. The licensee did not propose any changes to Procedure 1.3.10A.

PER 298-0112, dated February 5, 1998, was initiated to document another inadvertent actuation of fire protection preaction System 66 because of grinding operations in the reactor building/radwaste building corridor on the 441-foot elevation.

From a review of

-16-the performance issues surrounding the inadvertent actuation, the licensee concluded that the previous corrective actions taken in response to PER 295-0423 were inadequate.

As such, the licensee planned revisions to Procedure 1.3.10A to focus personnel on the need to take appropriate precautions to preclude inadvertent system actuations.

These could include covering ionization detectors to prevent spurious alarms and actuations of preaction Systems 65 and 66. The vaiidator of PER 298-0112 also noted that training and procedure enhancements may be appropriate to ensure that control room and shift support supervisors understand differences between preaction systems (ionization vs.

heat detectors) and take precautionary measures to prevent inadvertent actuation.

Although the licensee concluded that there was a high probability of recurrence, no short-term training was provided to plant personnel, and the revisions to Procedure 1.3.10A were not scheduled until August 1.

On June 17, when grinding operations were being performed in the Division ll emergency diesel generator room, maintenance personnel noted that smoke from the grinding was being carried into the adjacent corridor through an open fire door (opened to support work). The control room supervisor was contacted about the smoke but allowed the work to continue without taking additional precautionary measures.

The smoke actuated preaction System 66, resulting in the fire protection system transient that failed the piping in the reactor building northeast stairwell. The inspectors concluded that, had planned enhancements to Procedure 1.3.10A been implemented with the appropriate training to responsible plant personnel, the inadvertent actuation of preaction System 66 could have been avoided.

'From a safety perspective, the inadvertent actuation of fire protection systems unnecessarily challenges those systems, increases the potential for accidental wetting of SSCs, and distracts operators from normal plant operating duties. The failure to implement adequate and timely corrective actions for personnel knowledge and procedural weaknesses in the ignition source permit process was identified as a violation of the fire protection corrective action program and License Condition 2.C.(14).

Subsequent to the flooding event, revisions to Procedure 1,3.10A were implemented.

Training, provided to plant personnel on the summary of changes to the procedure was completed on June 29. The inspectors found that the implemented actions were reasonable and appropriate to address the long-standing performance issues; consequently, no response to this violation is required (50-397/9820-04).

Conclusions The fire protection corrective action program was ineffective in addressing water hammer in the fire protection water supply system.

The corrective actions taken in 1984 for known water hammer concerns were only partially effective in addressing the impact of the hydraulic transients resulting from system initiation. Subsequent indications of severe hydraulic transients in the fire protection system were not evaluated and resultant component failures were treated as isolated maintenance items. These component failures and industry operating experience on water hammer both represented missed opportunities to ferret out continuing system design problems.

The corrective actions from previous inadvertent actuations of the fire protection system were either ineffective in addressing personnel knowledge and procedure weaknesses in

-17-the ignition source permit process or not promptly implemented.

The inadvertent actuation of System 66 on June 17, occurred over 4 months after an almost identical event in February 1998. Although procedural enhancements were defined shortly after the event, the implementation of the enhancements.was not scheduled until as late as August 1998.

A violation of the fire protection corrective action program was identified; however, because the corrective actions for the violation were appropriate, no response is required.

General Event Res onse V. Operational Performance lns ection Sco e 92901 4e Closed IFI 50-397/9816-02:, The inspectors evaluated the response to the flooding event, including the declaration of the unusual event and'activation of the. emergency response organization.

Observations and Findin s A review of the emergency response guidance revealed that there was no clear event classification that corresponded to this event.

Classification of the flooding as an Unusual Event was made under the "Other" or judgment category that reads, "In the judgment of the Emergency Director, events are in process or have occurred which indicate.a potential degradation of the level of safety of theplant," (Emergency Action Level 9.1.U.1).

It was determined that the notification of state and local agencies occurred 7 minutes after declaration of the event and that notification of the NRC occurred 20 minutes after the declaration was made.

The notifications were made within the required time frames.

The control room manager, acting in the capacity of emergency director, indicated that the RHR pump Room C high level alarm in itself was not a sufficient degradation in the level of safety of the plant to declare an Unusual Event because operation in Mode 4 without one RHR pump is allowed by the Technical Specifications.

Additionally, shutdown cooling had been verified to be adequate and was being provided by a nonaffected pump. The control room manager indicated that securing of the fire suppression system, in conjunction with the losses of both RHR C and LPCS, were the events that resulted in a review of the emergency plan and declaration of the event.

Based upon interviews and a review of operating togs, it was determined that an Unusual Event was declared in less than 15 minutes after stopping the fire pumps. The timeliness of the declaration was considered to be commensurate with the significance of the event.

The emergency director did not upgrade the event classification to an Alert because, for the operating mode that was in effect, sufficient Technical Specification equipment had been verified to be available.

It was verified that safe shutdown capabilities were being maintained and that the required shutdown system parameters did not indicate degraded performance.

Specifically, the train of the RHR system providing shutdown cooling was unaffected.

It was also known that the initiating event had been terminated when the fire protection system was secured and that a fire was not in progress.

Further, all work that

-18-had the potential to pose a fire hazard was terminated and offsite fire assistance was reporting to the site to provide standby fire protection coverage.

The licensee reported that these decisions were made with input from operations and emergency preparedness management.

The emergency director also reported that his initial decision was to not activate the site emergency response organization.

However, because of the magnitude of incoming calls and the resulting anticipated response efforts, his decision was quickly revised to activate the onsite emergency response facilities at the time of the declaration of the Unusual Event. Since the event occurred during normal working hours, activation of the facilities was expedient.

During the event, plant operators maintained communications with personnel outside the control room through the use of hand-held portable radios.

Several of the radios exhibited poor reception and had to be replaced during the event and in some cases alternate methods of communication were employed.

Additionally, the control room telephone lines were congested during the event by inquiries from personnel who were not directly involved in the event response.

Following the event, the licensee established an action item to review options for improving communications in certain areas of the plant. However, as of the end of the inspection, no specific improvements had been identified for implementation.

The inspectors noted that, although some communications were delayed, the delays did not significantly impair the ability of operators to respond to the event.

As such, the efforts to review enhancements to their emergency communications lines were considered to be prudent but not required.

c.

Conclusions The licensee responded well to the flooding event. The shift manager made an appropriate decision to declare an Unusual Event and activate the onsite emergency response organization to quickly bring resources to bear on an unusual and complex event.

Declaration and notification of the emergency were both timely.

V.2 Ade uac and Im lementation of Procedural Guidance For Internal Floodin a.

Ins ection Sco e 92901 Closed IFls 50-397/9816-01 and -03: The inspectors reviewed the operators'erformance in implementing procedural requirements addressing the flooding in the ECCS pump rooms and evaluated the actions to correct procedural weaknesses.

b.

Observations and Findin s During the event, operators started the LPCS pump following receipt of the room flooding alarm. The inspectors expressed concern that the action was not in accordance with Abnormal Condition Procedure 4.12.4.10, "Reactor Building 422 Area Flooding,"

Revision 5. The inspectors discussed the operators'ctions with the operations manager, who was present when the decision was made.

The operations manager noted that the operators were faced with conflicting requirements:

(1) to maintain operability of LPCS in accordance with Procedure 2.4.3, "Low Pressure Core Spray,"

and (2) to follow Procedure 4.12.4.10 to stop the affected pump. When addressing areas affected by the flooding, the operators interpreted "affected" to be RHR pump

-19-Room C, noting that the flooding rate into the LPCS room was lower than the rate into RHR pump Room C. The decision as to whether the LPCS pump was an affected pump was critical in making the decision.to start it.

Specifically, the wording of Step 3.2 in Procedure 4.12.4.10, "secure ECCS pump in affected room, if running,'as not clear in what "affected" meant.

Procedure 2.4.3 requires that the LPCS pump be started to maintain operability if the water leg pump is not running. The operators did consider that when starting the LPCS pump a criterion had to be established when to stop the pump, which was determined to be when the water leg pump was submerged.

This would be some time after the 6-inch high-level alarm was actuated in the LPCS pump room. The LPCS pump was subsequently stopped and the fuses pulled shortly after the operators determined that plant conditions did not require the LPCS pump to be operable.

The inspectors found that, although the decision to start the LPCS pump did not violate plant procedures, it was not considered prudent based upon the limited information available to operators to assess the condition of the LPCS pump motor. Had the motor been grounded because of the flooding, a motor start would have challenged the protective relaying of the Division I 4160 Vac vital bus.

The inspectors reviewed changes made to Procedure 4.12.4.10 as a result of the event. The changes made to Steps 4.1.8, 4.1.9, and 4.1.10 now require specific actions to start ECCS pumps ifthe train-related water leg pumps are affected by flooding.

Further, to preserve safe shutdown capability, the opposite train pump is started.

No change was made to the requirement in the procedure to stop the affected ECCS pump or pumps ifflooding is encountered.

The changes conformed with guidance in Procedure SWP.PRO.03, "WNP-2 Procedure Writers Guide, and were found to be appropriate.

C.

Conclusions The actions of the operators to start the LPCS pump during the flooding event, while in compliance with the wording of plant procedures, d o not display conservativ~

'ecision making. Although the actions were an attempt to maintain the maximum number of operable/available ECCS pumps, the operators failed to recognize that other potential effects could have occurred because of the flooding.

The implemented changes to the flooding response procedure conformed with accepted practice in procedure development and adequately addressed the need to start redundant divisional ECCS pumps to maintain operability.

V.3 Dischar e of Floodwater to the Storm Drains a.

Ins ection Sco e 92901 Closed IFls 50-397/9816-06-07 and -08: The inspectors reviewed the licensee actions associated with pumping of the flooded stairwell to the plant storm drains, a normally unmonitored release pathway to the environmen b.

Observations and Findin s-20-

b.1 Coordination of the Dischar e

As noted in NRC Inspection Report 50-397/98-16, the Technical Support Center manager authorized personnel to commence pumping of the reactor building stairwell to the storm drains without conferring with the shift manager who, at the time, was acting as the emergency director. The licensee documented this issue in PER 298-0804 to track resolution of underlying performance weaknesses.

~

Through the resolution of PER 298-0804, the licensee identified that Procedure 1.3.1,

"WNP-2 Operating Policies, Programs, and Practices," Revision 36, states that, "once emergency plan implementing procedures have been entered, recovery actions not specifically authorized by plant procedures, which have the potential for radioactive release to the environment, require emergency director. concurrence."

However, the requirement was located in an operations procedure and was not translated to the emergency plan implementing procedures.

As such, emergency response personnel were not formally being trained on the restriction and did not have specific guidance in their implementing procedures.

To correct the identified deficiencies, the licensee:

(1) revised applicable emergency plan implementing procedures to include the requirement to obtain emergency director concurrence prior to taking recovery actions not specifically covered by plant procedures, which have the potential for radioactive release to the environment; (2) issued a change management bulletin to emergency response personnel to reiterate the requirement; and (3) planned to revise emergency response training lesson plans to include lessons learned from the flooding event. The licensee also reviewed Procedure 1.3.1 in detail and found no other examples where emergency response requirements were not properly reflected in the emergency plan implementing procedures.

The inspectors concluded that the corrective actions were appropriate.

The failure of the Technical Support Center manager to obtain concurrence from the shift manager (emergency director) prior to commencing discharge of the stairwell floodwater to the storm drain was identified as a violation of. Procedure 1.3.1 and Technical Specification 5.4.1.a.

This nonrepetitive, licensee-identified, and corrected violation is being treated as a noncited violation consistent with Section VII.B.1 of the NRC Enforcement Policy (50-397/9820-05).

b.2 Monitorin of the Dischar e The inspectors also evaluated the adequacy of efforts to monitor the discharge for the potential release of radioactive material. Three 1-liter grab samples of the water discharged were analyzed.

The initial grab sample analysis results showed no detectable activity greater than the free release lower limitof detection established in the procedures.

Therefore, the decision to free release and discharge the fire protection system water to the storm drain pond, based on this analysis result, was in accordance with the National Pollutant Discharge Elimination System Wastewater Discharge Permit.

Additional monitoring of the discharge was accomplished through the use of a composite sampler installed on the discharge line to the storm drain pond. The collection rate of the

C.

-2'j-sampler was established based on collecting a representative sample of the estimated 100,000 gallons to be discharged.

Additional grab samples (not required by procedure)

were also collected to monitor the discharge.

The second grab sample, collected during the discharge to the storm. drain pond, showed trace amounts of Cobalt-60.

Upon detection of Cobalt-60 in the second grab sample, the licensee terminated the discharge.

This precluded the ability of the composite sampler to obtain a full liter of representative sample for analysis by the onsite lab. Although the size of the composite sample was insufficient for onsite analysis to verify the discharge levels were below the lower limitof detection for Cobalt-60, an offsite laboratory was able to demonstrate that the levels of Cobalt-60 were, in fact, below the limit. Additionally, a third grab sample collected after the discharge was terminated showed no detectable activity greater than the lower limit of detection established for free release of water from the plant.

Conclusions The Technical Support Center manager failed to confer with the emergency director prior to authorizing the discharge of the stairwell floodwater to the storm drains. The error was the result of the improper placement of an emergency response requirement into an operations procedure instead of the emergency plan implementing procedures.

The corrective actions taken to address this deficiency and evaluate the generic implications were appropriate.

A noncited violation of Technical Specification 5.4.1.a was identified for failure to follow.procedure, in accordance with Section VII.B.1 of the Enforcement Policy.

The licensee took appropriate measures to monitor the discharge of the stairwell floodwater to the storm drains to verify compliance with the National Pollutant Discharge Elimination System Wastewater Discharge Permit.

V.4 Ade uac of Fire Protection Com ensato Actions a.

Ins ection Sco e 92904 Closed IFI 50-397/9816-04:

The inspectors reviewed Procedure 1.3.10B, "Active Fire System Operability and Impairment Control," Revision 1; Procedure 1.3.57, "Barrier Impairment," Revision 11; WNP-2 Fire Protection Evaluation, Amendment No. 52, dated August 1997; PER 298-0800, which documented that compensatory measures were not in complete compliance with plant procedures following the flood event; PER 298-0816, which documented that compensatory measures were not in strict compliance with plant procedures for the planned impairment of Standpipe RB-1; walkeD down portions of the fire protection system and associated protected areas; and interviewed plant fire protection personnel.

b.

Observations and Findin s The inspectors found that on two separate occasions the licensee failed to comply with the requirements of Procedure 1.3.10B with the second occasion being identified by the NRC. As documented in PER 298-0816, fire protection compensatory actions {required as a result of Valve FP-V-29D rupture) were not in compliance with the requirements of Procedure 1.3.10B.

However, the circumstances immediately following the rupture of

-22-

~

response time for plant personnel to obtain complete compliance with Procedure 1.3.10B was not unreasonable but process improvements could be realized.

In contrast on June 26, the inspectors identified that the licensee failed to comply with the requirements of Procedure 1.3.10B during a planned, controlled evolution. Specifically, the licensee isolated fire protection water to reactor building Riser RB-1 to perform work on Valve FP-V-62 (supply isolation to standby gas treatment deluge valves).

Procedure 1.3.10B, Section 4.4.5.b.1) states:

"With one or more of the Essential hose stations

.

~. inoperable, issue a FPSIC permit and within one hour... provide gated wye(s) on the nearest operable hose station(s)... the wye shall be connected to sufficient hose to provide coverage for the area left unprotected by the inoperable hose station." These requirements are also described in'FSAR, Section F.5.5.2, "Essential Fire Hose Station Compensatory Measures."

Contrary to the requirements, the inspectors found that sufficient hose w'as not connected to the gated wye(s) on the nearest operable hose station to provide coverage for the area left unprotected and that greater than 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> had elapsed since the Essential hose station(s) were made inoperable.

The failure to implement compensatory measures in accordance with Procedure 1.3.10B was identified as a violation of Technical Specification 5.4.1.d (50-397/9820-06).

The licensee initiated PER 298-0816 to address the inspectors'indings.

The resolution to PER 298-0816 determined that the plant had initiated some actions to establish fire protection compensatory actions but because of schedule demands and personnel errors, all required compensatory actions were not taken.

Specifically, Procedure 1.3.10B, Section 4.1.3, states:

"The individual authorizing an impairment shall ensure appropriate compensatory actions are instituted for impaired fire systems

.

~.."

However, the fire protection engineer failed to ensure that all appropriate compensatory actions were instituted.

The inspectors found that the permanent corrective action plan (consisting of training involved personnel on procedural requirements and specifying management expectations) for PER 298-0816 was appropriate to address the performance concerns that resulted in the violation; therefore, no response to this violation is required.

Conclusions Because of competing priorities in responding to the fire protection system rupture and flooding event, required fire watches were not established within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> of the system impairment. The delay of approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> in implementing the compensatory measures was found to be reasonable based upon the nature of the event. A second example of a failure to implement compensatory measures for a fire protection system impairment was identified by the inspectors during planned corrective maintenance on June 26. A violation of Technical Specification 5.4.1.d was identified for failure to follow fire protection program implementing procedures; however, because the corrective actions were appropriate to address the root cause, no response to this violation is require Vl. Management Meetings Vl.1 Exit Meeting Summary

'he inspectors presented the inspection results to members of licensee management on September 17, 1998. The licensee acknowledged the findings presented.

The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary.

No proprietary information was identifie ATTACHMENT1 Supplemental Information PARTIALLIST OF PERSONS CONTACTED Licensee D. Coleman, Regulatory Affairs Manager F. Diya, System Engineering Manager D. Feldman, Assistant Operations Manager D. Kobus, Fire Protection Supervisor P. Inserra, Licensing Manager J. McDonald, General Engineering Manager R. McQuoid, Maintenance Rule Program Lead S. Oxenford, Operations Manager J. Peterson, Fire Protection System Engineer G. Smith, Plant General Manager S. Wood, System Engineering Supervisor D. Wyatt, Plant Fire Marshall NRC S. West, Fire Protection Specialist, Office of Nuclear Reactor Regulation P. Quails, Fire Protection Specialist, Office of Nuclear Reactor Regulation C. Poslusny, WNP-2 Project Manager, Office of Nuclear Reactor Regulation INSPECTION PROCEDURES USED IP 92901'P 92902:

IP 92903:

IP 92904:

Followup - Operations Followup - Maintenance Followup - Engineering Followup - Plant Support ITEMS OPENED, AND CLOSED

~Oened 50-397/9820-01 EEI Design inadequacies in the fire protection water supply system (Section l.1)

50-397/9820-02 VIO Failure to include the sump cross-connect valves in the, scope of the Maintenance Rule program (Section ll.1.b.2)

50-397/9820-03 NCV Failure to update the FSAR to reflect the latest flooding analysis assumptions (Section II.2.b.1)

50-397/9820-04 VIO Failure to implement adequate and timely corrective actions for personnel knowledge and procedural weaknesses in the ignition source permit process (Section IV.b.2)

-2-50/397/9820-05 NCV Failure of the Technical Support Center manager to obtain concurrence from the emergency director prior to commencing discharge of the stairwell. floodwater to the storm drain (Section V.3.b.1)

50-397/9820-06 VIO Failure to implement compensatory measures in accordance with Procedure 1.3.10B (Section V.4)

Closed 50-397/9816-01 50-397/9816-02 50-397/9816-03 50-397/9816-04 50-397/9816-05 50-397/9816-06 50-397/9816-07 50-397/9816-08 50-397/9816-09 50-397/9816-10 50-397/9016-11 50-397/9816-12 50-397/9816-13 50-397/9816-14 IFI IFI IFI IFI IFI.

IFI IFI IFI IFI IFI IFI IFI IFI IFI Adequacy of procedural guidance for reactor building flooding

'Section V.2)

Adequacy of radio communications (Section V.1)

Operating crew's decision to start the LPCS pump during the flooding event (Section V.2)

Implementation of compensatory measures for degraded fire protection system (Section V.4)

Adequacy of procedural guidance for control of ignition sources (Section IV.b.2)

Sampling methodology associated with the discharge to the storm drains system (Section V.3.b.2)

Coordination and control between the control room and the Technical Support Center (Section V.3.b.1)

Command and control associated with deeatering evolution (Section V.3.b.1)

Review of licensee flooding analyses (Section 11.2.b.1)

Assumptions and corrective actions for LER 50-397/92-034-02 (Section II.2.b.1)

Corrective actions for previous fire protection system water hammer events (Section IV.b.1)

Unexpected response of preaction System 81 (Section l.2)

Corrective actions associated with multiple preaction scenarios (Section 1.2)

Design adequacy of the fire protection system (Section 1.1),

-3-50-397/9816-15 IFI Design discrepancies associated with watertight doors (Section II.2.b.1 and II.2.b.2)

50-397/9816-16 IFI Issues related to RHR 2C door being left in an unsecured condition (Section III)

50-397/9816-17 50-397/9816-18 IFI Leakage of watertight doors (Section II.2.b.2)

IFI Corrective actions associated with watertight door maintenance deficiencies (Section ll.2.b.2)

50-397/9816-19 IFI Maintenance rule performance criteria for the watertight doors (Section Il.2.b.2)

50-397/9816-20 IFI Design issues associated with sump isolation valves (Section II.1.b.1)

50-397/9816-21 50-397/9816-22 50-397/9820-03 50-397/9820-04 IFI Corrective actions associated with sump isolation valve maintenance deficiencies (Section ll.1.b.2)

IFI Maintenance and testing program issues for the sump isolation valves (Section II.1.b.2)

NCV Failure to update the FSAR to reflect the latest flooding analysis assumptions (Section II.2.b.1)

VIO Failure to implement adequate and timely corrective actions'for personnel knowledge and procedural weaknesses in the ignition source permit process (Section IV.b.2)

50/397/9820-05 NCV Failure of the Technical Support Center manager to obtain concurrence from the emergency director prior to commencing discharge of the stairwell floodwater. to the storm drain (Section V.3.b.1)

50-397/9820-06 VIO Failure to implement compensatory measures in accordance with Procedure 1.3.10B (Section V.4)

LIST OF ACRONYMS USED CFR ECCS EEI EOP FDR FSAR gpm IFI LER Code of Federal Regulations emergency core cooling system escalated enforcement item (apparent violation)

.

emergency operating procedure floor drains radioactive Final Safety Analysis Report gallons per minute inspection foliowup item Licensee Event Report

-4-LPCS NCV NFPA NRC PDR PER pslg RHR SSC VIO WNP-2 low pressure core spray noncited violation National Fire Protection Association U.S. Nuclear Regulatory Commission Public Document Room problem evaluation request pounds per square inch gauge residual heat removal structures, systems, and components violation Washington Nuclear Project-2

ATTACHMENT 2 Root Cause Analysis Rcport PER 2984779 FIRE MAINFLOODING OF TWO ECCS ROOMS Mana ement Summa Description of Event Thc primaiy effect was thc flooding oftwo ECCS Pump Rooms, RHR 2C and LPCS. The event began with flooding ofthe Reactor building Northeast Stairwell with subsequent flooding ofthe two ECCS pump rooms.

On Wednesday, June 17, Maintenance personnel were removing a hanger in the DG-2 bay using a cutting torch. The acetylene tanks were placed in the corridor and the diesel bay door was open.

Smoke resulted from the effort and propagated into and down the corridor. As a result, two fire protection preaction systems were actuated; system 66 and 81. The fire protection system was in a normal lineup with only the jockey pump running. The rapid decrease in the fire protection header pressure that occurred as a result ofthe fillofthe preaction piping started the standby fire pumps. Allfour ofthe standby pumps auto started on decreasing system pressure with three stans occurring in a nearly simultaneous manner. A significant water hammer event occurred within the fire protection system that resulted in a catastrophic failure ofa valve (FP-V-29D), in the Reactor Building Northeast stairwell. The system was now being supplied by four fire pumps resulting in signiTicant flowthrough FP-V-29D breach and into thc stairwell. The flooding started in thc stairwell and entered thc RHR-P-2C pump room through door R013 which was found open. The reason the watertight door was open is believed to be a human performance

'rror.

Subsequently, some water cntcred the LPCS pump room through FDR-V%09 which failed to close as designed during the event. Operator mitigation resulted in securing thc fire main and reducing pump room water levels.

Root Cause Summary The root cause ofthe event is inadequate Fire Protection system design in that the system is configured such that destructive forces are generated during an anticipated challenge with only the jockey pump running.

Conditions Needing Correction to Prevent Rccurrcnce

~

Eliminate severe water hammer in the fire protection system.

Conditions Needing Correction To Address Contributing Factors

~

Eliminate unintentional challenges to preaction and high flowdeluge fire zones (refPER 298%112 corrective actions and PER 29&%786).

~

Train Station Personnel on thc proper operation ofwater tight doors emphasizing worker selfwheck techniques for watertight door use (refPER 2984788).

~

Repair FDR-V%09 and associated controls (rcfPER 298%780).

Descri tion of the Event (For a detailed description ofthc time line see Attachment 1)

On Wcdncsday, Junc 17, 1998, the plant was in cold shut down (mode 4) with shutdown cooling on RHR pump 2A. Thc LPCS keep fillpump ideas in service maintaining thc standby pressure ofthe LPCS system. RRC pump IB was on at 15 Hcnz. Thc dry well was being purged. The fire main was on the keep fill(jockey) pump at essentially zero demand. Preparations were being made to start up the plant later in thc day. Fire pump 2A and 2B monthly operability test had bccn complctcd pcr PPM 15.1.5 and PPM 15.1.6 earlier in tlic day at 0925 after wliich time both pumps were placed in standby.

Maintcnancc craft pcrsonncl werc cutting/grinding within thc Division II dicscl generator room to rcmove hangers and piping. This activity was part ofa design cliangc. The work was performed under Page I

Root Cause Analysis Report PER 2984779 an approved work order, ignition source permit, and transient combustible permit. The activity utilized a cutting torch with a cart located in thc corridor and hoses routed through the open door ofthe DG? room.

As a result ofthese activities, smoke propagated out ofthe room into the corridor and into another adjoining room. Personnel involved in the work activity noted the accumulation ofsmoke, stopped work and contacted thc control room to obtain guidance. The situation was reviewed by the on-shifl Control Room Supervisor who allowed thc work activity to continue.

At 13:43:49 (by TDAS), Fire Zone system 66 was actuated by a smoke detector in the corridor.

Zone 66 preaction system actuation consists ofan actuation ofthe sprinkler pieaction valve that, in turn, fillsthc piping to the sprinkler heads. The piping from the preaction valve to the sprinkler heads is normally pressurized with air at 2lh60 ounces gauge pressure. The preaction system provides additional assurance that a spurious actuation does not result in sprinkler flowonto sensitive station equipment withinthe protcctcd area. Four fire-protection detector~ms are located in the area ofthe DG2 room:

38, 39, 66, and 81. Zones 66 and 81 provide a prcaction control signal to thc respective preaction valves.

Zone 66 consists ofsmoke detectors (ionization), and Zone 81 consists ofheat detectors that are set to actuate at 200'

or a rapid change in ambient temperature. The Zone 66 detectors are located in thc corridor outside the DG2 room and in the adjoining old laundiy room. The Zone 81 detectors are located within the DG2 room but not within the electric room where the cutting was being performed. The Zone 39 smoke detectors are located in immediate proximityofthe Zone 81 detectors as well as in thc electric room.

Thc control room reported that Zone 66 and 81 preaction systems actuated concurrently. Itwas believed that these two systems actuated very close to each other. Subsequent analysis determined that Zone 81 actuated approximately 54 seconds after Zone 66. The control room reported that all fourfli pumps auto started. Actually, motor driven pumps 2A, 2B and diesel driven pump 110 started immediately, however, the motor driven pumps are equipped with a Clayton valve that controls thc discharge flowofthe pump such that there is a delay to achieve fullflowinto the piping system. Thc Clayton valves werc installed to prevent water pressure surges during startup ofthe motor driven pumps.

The diesel driven pump number 110, however, is not so equipped and would be discharging into the system as soon as the pump teaches speed about 2 seconds afler receiving a start signal. The fourth pump, diesel driven pump number 1, is equipped with a starting time delay that is set for 30 seconds and actually started about 34 seconds afler the other three pumps.

The actuation ofthc 66 system opened the corresponding fillvalve, which allows firewater to pressurize the normally empty preaction system piping. This requires approximately 400 gallons ofwater.

Not all ofthis volume is fiHed as the resident air is compressed unless a sprinkler head is open. Atthe time ofthe preaction actuation, the fire protection jockey pump FP-P-3 was running. This pump is a low volume high-pressure pump that maintains the system pressure during normal standby conditions. This pump is not designed to maintain system pressure with any appreciable load on the system.

The Reactor Building standpipes RB-1 (northeast corner) and RB-2 (Suuthwest corner) are the tallest standpipe risers at WNP-2 (approximately 180 feet). Thc large instantaneous drop in system pressure caused the water level to drop in these standpi pcs resulting on "voiding"ofthc upper sections of the riser piping. The fire protection system pressure dropped significantly and resulted in thc immediate start of 3 standby fire pumps. The associated rapid water surge collapsed the voids and caused severe water hammer. In riser RB-I, the pressure surge was ofsuQicient magnitude to cause rupture ofFP-V-29D isolation valve, located at the bottom ofthc riser piping in the stairwell at approximately 434'bove Mean Sea Level (AMSL)elevation or approximately 12 feet above thc floorofthe Reactor Building basement.

The valve is near thc pipe support transition from Seismic Class I to Seismic Class II. This valve catastrophically failed resulting in approximately 163,500 gallons ofwater entering the Reactor building stairwell and basement.

Page 2

Root Cause Analysis Rcport PER 2984779 At 13:45, alarms associated with the actuation ofthc prcaction systems and the auto start ofthe fire pumps alerted thc main control room. High water level alarms actuated by switches in the RHR 2C room also alerted the Main Control Room to water leakage in the reactor building. The RHR and LPCS pump rooms are adjacent to the vestibule and stairwell in the area ofthe riser piping. The flooding water entered the RHR 2C pump room through the door R013, which was found to be open during the follow-up inspection. Rising,water level in the RHR 2C pump room actuated a room high water level alarm that annunciated in the main control room. The operators entered the Emergency Operating Procedures (EOPs) based upon this alarm. The rising water in thc RHR 2C pump room reached the level ofthe kcep-fillpump (RHR-P-3) and resulted in a trip ofthis pump. RHR-P-2B located in a non<ffected watertight room was started in the suppression pool cooling mode to maintain this system operational. The RHR 2C pump room subsequently flooded to the -439 foot elevation or -17 feet above the floorsubmerging the RHR 2C pump motor.

The water hammer was loud enough to be heard by many station personnel. As a result, a Shift Support Supervisor in the area commenced a visual inspection and noted the flooding in the stairwell.

This individual promptly notiTied the Main Control Room ofthe flooding. The control room operators initiated action to secure the standpipe and fire pumps after verifying that there was no fite or threat of fire. Allfow pumps were sectiied to terminate thc source ofthe flooding.

A normally open floordrain isolation valve, FDR-V<09, was open at the time ofthe event and provided a flowpath from thc RA sump which is located in the RHR 2C pump room to the floor drains in the LPCS room. Water flowed through this pathway from the sump up through the drains and began to flood the LPCS pump room: Rising water level in the LPCS pump room actuated a high water level switch that also annunciated in thc main control room. The operators also noted this as an entry in'.o the EOPs.

LPCS-P-I was started to maintain this system operational. LPCS-P-2, the keep-fill pump for Division I, was secured. RHR-P-2A was running in shutdown cooling at the time.

At 14:14, the SNfl Manager declared an Unusual Event based upon thc non-functional fire protection system and the desire to obtain additional assistance to coordinate plant response and recovery activity. The TSC and the OSC were activated. During this time the water level withinthe LPCS pump room continued to rise. Due to the continued increase in water level, LPCS-P-I was secuiixL The unusual event cmcrgency classification was retained until the fire protection system was restored to scrvicc.

At 14:43 The Hanford Fire Department arrived on site. This fire protection capability was retained tluough the event and until the fire protection system was restored to an operablc status three days later.

'I Action was initiated to pump out the water from the ECCS pump rooms and the stairwell. This effort resulted in a removal ofthe water to a point below thc berm areas ofthe pump rooms by 2205.

Time Line Summary (refattachments I and 6)

The sequence ofevents is as follows. Alltimes and pressures are from the TDAS computer, flow rates are estimated from pump curves for indicated prcssurcs, pressures at the pumps may have bccn significantly different tlian those in thc Reactor Building or at the 110 pump discharge.

Initial conditions:

~

Fire main prcssure at 144 psig, 175 gpm jockey pump on line; system demand is near zero

~

Two motor driven fire pumps in standby

~

Two dicscl driven fire pumps in standby Transient conditions:

~

At time zero; Prcaction system 66 is actuated by smoke detector

~

System pressure starts to drop Page 3

Root Cause Analysis Report PER 2984779

~

At second 1; system pressure is at 72 psig, both motor driven pumps and one diesel driven pump have received start signal. Motor driven pumps start at fullrun out flowwith Clayton valves open back to the'circ water pit, 110 pump starts at fullengine throttle.

~

At second 2; system pressure is at 51 psig, motor driven pumps are starting to discharge to system at reduced flow, 110 pump commences discharge into the firemain, preaction 66 is beginning to fill,void is forming at Reactor building riser as water level drops from top of standpipe.

~ 't Second 3; system prcssure is at 32 psig; Reactor building riser void is at maximum size.

Preaction system 81 push rod chamber water pressure is being lost past the leaking check valve.

~

At Second 4; the 110 pump raises system pressure to 79 psig. The motor driven pumps are still under the Clayton valve influence and a large portion oftheir contribution is shunted back to the pit.

~

At Second 5; system pressure at 95 psig. Reactor building riser void is collapsing.

~

At-Second 5.5, the Reactor building riser void colhpses and hits the standpipe riser end caps, riser RB-1 isolation valve ruptures, spilling water to stairwell. The water hammer system pressure surge causes pressure at preaction 81 clapper valve to more than double the push rod chamber pressure and preaction 81 inadvertently actuates.

~

Second 6; system pressure at 90 psig

~

Second 7; system pressure at 82 psig

~

Second 8; system pressure at 85 psig

~

Second 9; system pressure at &0 psig, ~ pump 2A is discharging -2000 gpm

~

Second 10; system pressure'at 80 psig

~

Second 11; system pressure at 80 psig

~

Second 12; system pressure has risen to 98 psig

~

Second 13 to 34 system stabilizes at about 110 psig

~

Second 27; ~ pump 2B is discharging -2000 gpm

~

Second 34, Diesel pump ¹1 starts

~

Second 35 to 36 system pressure rises to 125 psig as diesel driven pump ¹1 comes on line, pressure stabilizes at 125 psig

~

At 17 minutes all fire pumps are oK Preliminary testing ofthe Clayton valves indicates that these valves function as anticipated eQ'ectively delaying fullflowoutput ofthe 2A &2B fire pumps. The valve for pump 2A closed in-9 sec and the valve for 2B closed in -27 sec.

Corrective actions with respect to the Clayton valves willbe determined at a later date.

Anal sis of Event

%ork Activities in the Diesel Generator Room The series ofevents leading up to the flooding event began when zone 66 preaction system was inadvertently actuated. Thc actuation was caused by smoke generated from a cutting torch in usc under work order KFC9. A separate PER (298%786) addresses thc inadvertent actuation offire protection zone 66. The cause was determined to be "job scoping did not identify special circumstances or conditions" (MM0108).

Fire Protection System Thc design ofthc fire protection system at WNP-2 contains a number ofpreaction systems that arc not norinally filledwith water. Thc actuation ofthese systems requires the fillingofspray piping with water. The normal system pressure, approximately 140 psig, is maintained by a low-flow, high-pressure'ockey pump (FP-P-3). Four main fire pumps arc installed in a standby configuration and sequentially start on decreasing system pressure as follows:

Page 4

Root Cause Analysis Report PER 2984779 FP-P-2A motor FP-P-2B motor FP-P-1 diesel FP-P-110 diesel Start Set int and Time Dela 120 PSIG no time dela 110 PSIG no time dela 110 PSIG 30 sec. time dela 100PSIG notimedela Ca ci 2000 GPM 2000 GPM 2000 GPM 2500 GPM The jockey pump is not designed to maintain system pressure upon the actuation ofany preaction system.

The system is therefore required to accept prcssure and flowtransients resulting from pump auto starts.

Post event research determined NFPA 20-1974 (WNP-2 code ofrecord) requires a fire pump sequential start delay of5 to 10 seconds.

The above table shows this requirement to not be satisfied (PER 2984813).

Preliminary testing ofthe Clayton valves indicates a closing time of-9 seconds for pump 2A and -27 sec for pump 2B. With thc water hammer occurrin near 5.5 seconds, itappears pump 110 was the sole contributor during thc void collapse and valve rupture. Thus, fire pump sequencing does not appear to be an event contributor.

Preaction System 81 Actuation Afault tree was generated from which a plan (Attachment 4) was developed that determined the scope and sequence ofthe testing and disassembly ofthe 81 prcaction valve assembly and trim. The following signiTicant issues surfaced:

~

The prcaction valve internal parts appeared in good condition with some minor corrosion noted in thc priming and actuation chambers ofthc valve.

~

Inspections under a troubleshooting plan determined that priming chamber pressure gages on surrounding systems were dropping offwith the header pressure.

This indicated thc check valves were leaking.

~

Discussions with the valve manufacturer disclosed that the trim on these valves has been modificd to include a much-improved method ofassuring that thc priming camber pressure is not subjected to prcssure cxcursions during normal system pressure transients.

WNP-2 implemented this upgrade modification to all prcaction and deluge valves per TER 9840644.

Failure Analysis of Fi". -29D The valve is a 12 inch gray cast iron gate valve manufactured by MLH.From the investigation it was concluded that the failure was caused by dynamic loading ofthe valve during the fire protection water hammer event (see attachment 9).

The cracking appears to have initiated at or near thc end ofone ofthe four integrally cast gate guides in thc valve body. The rapid propagation ofthe crack throughout the heavy flange section and into the bonnet ofthe valve is indicative ofa high strain rate and displacement caused by the water hammer.

From thc orientation ofthe fractures the initiation is believed to have had components oftorsional shear and tension. This conclusion is supported by the direction and orientation ofcrack propagation, metallurgical evaluation ofthe crack morphology, and scanning electron microscopy (SEM) ofthe, fracture surfaces. No precxisting flaws or inclusions were observed in the fracture surfaces. The valve body material chemistry and microstructure was characterized as a pearlitic gray cast iron which is consistent with the ASTM A126 specification. Corrosion ofthc continuously wetted gray cast iron was observed and through metallurgical examination thc wall loss was estimated to be 18 mils. This is considcrcd acceptable given the 80 mils design corrosion allowance for the system. In summary, the valve failure was caused by dynamic loading during the water hammer and no pre~xisting flaws were found.

RHR 2C Door R013 (PER 298-0788)

Each ofsix separate pump rooms are equipped with watertight doors at each floor level entrance.

The doors arc dcsigncd to prcvcnt flooding from onc ECCS pump room into another ECCS pump room.

Page 5

Root Cause Analysis Report PER 2984779 The doors were not intended to be leak tight although the manufacturers test requires that they be demonstrated as "watertight. The doors arc ofstccl construction with multiple latching dogs around the door circumference operated by a common "quick acting" mechanism. Dogging ofthe door is accomplished by a four-post handwheel that controls the position ofall ofthe door dogs. Each door is equipped with a rubber-like seal around thc circumference that mates with steel channel on the door frame in order to provide a continuous seal when the door is shut. Each set ofdoors is equipped with a magnetic switch tlat triggers a computer point, after a 45-second delay, whenever the door position changes from and to the full closed position. The security alarm typer indicated that R013 was open at 1307 and remained open up through the time ofthe event.

Bamer analysis and fault tice analyses were performed on the doors (Attachments 2 and 5). The conclusions ofthe analysis are as follows:

~

Based on the evidence and the fault tree analysis there arc many plausible ways for the R013 door to have come open during thc flooding event. Thc most probable ofthese mechanisms can bc suinmarized under the heading of"human error" and involve leaving thc door unlatched or only partially latched. R-13 door loose dogging mechanisms made self-checking more difficult.

~

There is a possibility that thc mixture ofwater and debris churning through the area outside door R013 after thc collapse ofthc stairwell flidoor could have moved the door handle to a position that allowed the door to open. This is not considered probable due to the limit switch indication that the door was open.

~

The principle cause offloodwaters from the reactor building stairwell reaching ECCS equipment in two rooms was an open door (R013) allowing water to enter the RHR 2C room.

The cause ofthe door being left opened and/or not properly scaled is attributed to personnel error. Note that the door to LPCS room remained closed during this event. Flooding in the LPCS room was mainly attributed to the failure ofFDR-VM9 to isolate the floordrains.

FDR-V%09 FDR-VM9 is a single isolation between the sumps in RHR C pump room and the LPCS pump room. This valve failed to close as designed with high water level in the RHR C rooin sump. This failure is addressed by a separate PER (298%780) and Root Cause Analysis and willnot be evaluated in detail here. Although not the root cause ofthis event, the valve failure contributed significantly by allowing the flooding to extend into the LPCS pump room which othenvise would have been minimally affected by the flooding.

Other Physical Barriers Walls and wall penctrations were inspected. This evaluation found walls and penetrations functional and intact. Emergency core cooling system equipment room walls arc steel-reinforced, concrete walls approximately thirty-six inches thick. Electrical cables and conduits, and pipes pcnctrate the walls.

Each penetration is sealed with a synthetic sealing material designed to prevent water and fire from migrating from one to another room. A post event inspection ofthese seals founc their integrity good but noted some minor deformation.

Event Prccursors Documented configuration or process issues tlat relate to the cause or the extent ofconsequences ofthe event under consideration.

PER 292%417, 5/5/92, P79 tripped due to manual pull actuation. P-2A, 2B, and 110 all started resulting in damage to seal supply strainer blow down line on P-2A.

Corrective Actions Implemented:

~

Replace glass on pull station

~

Repair broken strainer blow down linc

~

Discussed incident with crafts supervisor Page 6

Root Cause Analysis Rcport PER 2984779 PER 295%423, 4/29/95, P66 actuated due to hot work Note: FP-P-110 auto started per Ops Log. No other log entrees of note were included. Corrective actions dealt with training for involved personnel.

Corrective Actions Implemented:

~

Provide training to the Site Support Contractor Supervisors 4, Craft on cautions on existing PPM 1.3.10 to prevent inadvertent actuation ofpreaction and deluge systems.

PER 296%443, 6/1/96, P66 tripped duc to smoke from a transformer fire in Remote Shutdown Room. Fire pumps 2A and 110 started. Fire pump 2B was running at the time due to Jockey pump being inop. Also ref WOT BLPO. No water hammer or system damage was reported. Corrective actions dealt with transformer replacement and walkdown ofother sola transformers.

PER 298%112, 02/05/98, P66 tripped due to smoke generated during grinding operations.

The corrective actions below are complete.

~

CAP 2 - Revise 1.3.10A for control and dispersal ofsmoke (not complete at time ofevent)

~

CAP 2 - Removal from service ofclose proximity FP detectors

~

CAP 4 - Lessons learned for Shih Support Supervisors (SSS)

~

CAP 5 - Lessons learned for Raytheon personnel Note: During this event FP-P-2A was running, FP-P-110 auto started with the actuation ofP66.

Note: The operators logs indicated that a pressure gage on REACTOR 572'ad jammed at 300 psig (the upper limitsofthe gage scale). This indicates that the system cxpericnced a pressure transient during this event. A later operator log indicated that action was initiated to replace this gage.

PER 295%333: An NRC Inspector found RCIC room door (R005) open during plant tour.

Conectiye Actions ImplementaL

~

Issued Broadcast Message to all employees PER 297~40: Operations found RCIC room door - R005 open during plant tour.

Corrective Actions Implemented:

~

Issued Operations Night Order to reinforce that Operation persoiinel close doors

~

Training conducted by Security Force to reinforce that OGicers close doors e

Training conducted by Maintenance to reinforce that craft close doors

~

Training conducted by RMC to reinforce that laborers close doors WR 98000410 initiated 1/21/98 - FDR-V%09 failed to close during level switch calibration.

Corrective Actions Implemented:

FIN team inspection found valve to close 60 seconds aher initiation, suggested action was to wait longer for th. valve to stroke in the future. Evaluation was made that valve may need lub ication, which was scheduled for completion in August 199&.

I Discussions with plant personnel indicate that fire protection system pressure gages have been found failed upscale on several occasions in the past. These failures were considered "normal" for the fire protection system. Broken gages have typically bccn handled as "broke-fix" issues and seldom documcntcd since this is a class 2+ system and replacing a gage would be categorized as a "mundane task". Based on thc criteria in thc PPM at the time, these failures would not have met the threshold for initiating a PER. Since there is minimal documentation for these failures, it is diQicult to detcrminc the causes, frequency and therefore no opportunity existed to evaluate for an adverse trend prior to this event.

Security alarm typcr data indicates that RHR C door R013 had been lcR open 7 times for a significant period (>10 minutes) in the previous 6 months. There were no door impairments recorded for these periods, however, an impairment would not bc required ifthc door em continuously monitored. (ref Page 7

Root Cause Analysis Rcport PER 2984779 attachment 5).

Precursor Summary

~

Previous unintentional actuation's have occuned wluch resulted in multiple Qre pump starts.

There have been 8 recorded actuation's since 1990, 3 ofwhich vere associated with system 66 in the hst 3 years. Most unintentional actuation's occuited coincidentally with a main Qie pump abeady olietating Only one actuaiion (system 79) could be confirmed to have occuned with only the jockey pump operating, which resulted in damage to the seal supply stiainer blow down linc on FP-P-2A. These events occasionally resulted in minor damage such as packing leaks and broken gauges. No evidence ofprior signiQcant pipe movenicnt was rccoidcd. Tlic resultant conectivc actions fmm these events addiessed tlic aplxuent cause and repaired the associated minor damage.

Aflerthis event occurred, the fire water system was walked down several times to assess the extent of damage and to dctcinunc ifevidence existed that prior signiQcant pipe niovernent had occuned. The only evidence noted was slight movetncnt ofthe RB-I riser in the vicinityofthe ruptured valve and at thc location of thc hanger fora tap olfofthis riser (near FP-V%2) which was loose at the wall mounting.

~

There have been no dociunented occurrences ofsimultaneous preaction or deluge system actuations since 1990.

The deQciency tag in the contml room which stated theie were 3 previous occasions ofsimultaneous actuation of systems 66 and 81 was placed there after thc event and was not based on recorded 'history.

Interviews with thc personnel involved in placing thc dcQciency tag were urdiuitful in identifying the previous occurrences that were fnelnoiy based and thefefofe thc foot cause could llotvalidate the statement.

There are phcaids posted in the phnt at locations where scvcnd ~on systems tap oKa common header. It is not known when these were placed in thc plant. These systems share a common water motor ahrm and have thc potential to cause the actuation ofanother preection system due to a pressure build~ thmugh the check valve weep hole at the respective actuating piessuie switch location. However, this will not cause a

'imultaneous actuation, since it takes a Qnite time for the pressure build~ to acct the common pressure switch.

Conclusions The cause ofthc water hammer event is a direct result ofinadequate fire protection system design tliat allows severe water hammer to occur under designed system configuration and response to actuations.

Post event data is sufficiently strong to conclude that the actuation ofpreaction 66 was the sole contrib <c-to the initial pressure drop. Preliminary hydraulic analysis discloses that the actuation of preaction 66 with only the jockey pump on line at the start ofthe actuation produces high enough forces to cause the resultant damage. From this, it is concluded that the fire water system is not adequately designed to allow an anticipated transient without suffering destructive forces within the system.

Ifthe corrective actions from PER 298112 had bccn fullyimplemented in a morc timely manner the inadvertent actuation ofzone 66 due to smoke from cutting and grinding may have not occurred and this specific event could have been prevented. However, an actuation ofzone 66 due to actual fire or smoke condition would liave resulted in thc same water hammer and resulting flooding event duc to thc inadequate design ofthe fire protection system. Reviewing thc other previous PER's and Work Requests found it is not likely that thc water hammer, subsequent failure ofFP-V-29D, and resulting flooding would liave been predicted and prevented. Note tlat prior to NRC Information 98-31 on this event, there was no previously published EPM, INPO or NRC documents which are directly applicable to such nitncrabilitics in fire water systems.

Page 8

Root Cause Analysis Rcport PER 2984779 The most likelyevent scenario indicates tliat prcaction 81 actuated on rising system pressure at about time 5< seconds into the event. It is not likelythat the actuation ofpreaction 81 was causal in the water hammer event. Its participation in the severity of the water hammer is indeterminate. Initial hydraulic analysis discloses that the reactor building standpipe water hammer velocity impact for a preaction 66 only actuation would be more severe than a simultaneous actuation of66 and 81.

Previous events (see precursor section) indicate that actuation ofprcaction 79 can cause the auto start ofthree pumps when the system is challenged with only the jockey pump running. Preaction 79 has less volume than preaction 66. The RCA Team could find no other documented history ofany prc-actuation on only thc jockey pump.

The contribution ofthe fire protection pumps at the time ofthe event to supplying a preaction system demand is Thc lead pump was the diesel pump FP-P-110. The role that the motor driven pumps take in initialdemand supply is directly infiuenced by the interaction ofthc bypass Clayton Valve performance. The order ofsignificant contribution to fire loop Qow was diesel pump 110 followed by motor driven pump 2A then motor driven pump 2B, and finally, diesel driven pump 1. Based on preliminary Clayton valve testing, thc electric motor pumps and dicscl pump I do not appear to have contributed to the water hammer event. Followup Clayton valve testing willbe performed to validate this conclusion.

A test ofthe fire protection system configured as described in the FAO synopsis (attachment 7),

provides compelling evidence that interim operation without significant water hammers is assiircd.

Event and Causal Factors Analysis Sec Attachment 3 for details ofthe analysis. Significant terminal events are as follows:

~

The material condition ofthe prcaction valve trim allowed a sympathetic, unncccssary actuation of the preaction 81 system. The Attachment 4 analysis indicates that actuation ofpreaction 81 did not

'ontribute to the water hammer..

However, the priming chamber trim has been replaced to prevent recurrence.

~

Failure to implement the corrective actions outlined in PER 2984112 in a timely manner led to the lack ofknowledge needed by the operators to avert the inadvertent actuation ofzone 66.

~

The Control Room Supervisor that allowed work to continue atter smoke was reported to be present in the DG hallway either knew or should have known that the smoke could actuate the smoke detectors in the area and therefore missed an opportunity to avert this event.

~

The design ofthe firemain system is inadequate in that thc system is configured such that destructive forces are generated during an anticipated challenge with only the jockey pump running.

~

The rigid to inficxiblcpiping transition at FP-V-29D allows force concentration to occur. This was addressed by replacing the FP-V-29D and FP-V-394 with cast steel valves. Planned modirications to preclude water hammer willensure other cast iron valves are not subjected to large pressure transients.

~

The door leading into the RHR 2C pump room was not properly closed and dogged as the result ofa human error. This condition significantly exacerbated the consequences ofthe event.

~

Failure ofthe FDR-V%09 drain valve to close as designed contributed significantly to the consequences ofthe event.

Root Cause Thc root cause is inadequate system design (EQ0307) in that the system is configured such that destructive forces (water hammer) are generated during an actuation with only the jockey pump running.

Page 9

Root Cause Analysis Rcport PER 2984779 Contributin Causes g

~

Door R013 ms not properly closed and dogged by personnel (Self Checking PE0205). This allowed flooding from the Reactor 422'tairway and vestibule area to fiow into thc RHR C pump room. The level ofsensitivity for assuring that thc watertight doors are always closed and dogged is not adequate among the plant staff. Ref PER 2984788.

~

FDR-V409 failed to close on sump high level (Material Deficiency EQ0404). This failure allowed flooding into the LPCS pump room from the RHR C pump room. Ref PER 2984780.

~

Unintentional actuation ofprcaction fire zone 66 (MM0108). This was the initiating event for the water hammer and failure ofFP-Y-29D. Ref PER 2984112 and 2984786.

Recommendations Eliminate sevcrc water hammer in the fire protection system:

~

For thc near term, provide su6icient on-linc fire-pump volume capacity to maintain system prcssuie ifpreaction systems are triggered (refattachment 8).

~

Establish compensatory actions for immediate plant response ifthis on line capacity is lost.,

~

For thc long term, enhance the overall fire main system design to assure that anticipated transients do not create destructive pressure surges in thc system

~

Modifythe preaction valve priming trim to assure positive check valve closure and containmcnt ofactuating cyiinder fluidduring pressure transients.

~

Enhance preventive maintenance, and thc periodi testing program for preaction clapper valves, and associated trim, to detect and correct or mitigate aging and degradation induced spurious actuation.

Improve worker knowledge and sclfwhecking techniques for watertight door use.

Eliminate unintentional challenges to preaction and high flowdeluge fire zones. Reference PER 2984112 corrective actions and PER 2984786 resolution.

Perform follow-up testing ofthe Clayton valves to quantify the closuig response times and associated pressures and flows from the 2A and 2B pumps. Evaluate the eGcctivencss ofthese valves to minimize sharp pressure transients and their effect on water hammer in the system.

Heighten station personnel sensitivity to document equipment problems (such as fire protection gages failed high) via work request or PER. Eliminate the tendency to accept repetitive failures as "normal".

Page 10

Root Cause Analysis Report PER 298%779 Team Members Tcrly Meade; BSEE, SRO; Plant Managers Stair Jerral Rhoads; P.E., BSEE, MEM;WNP-2 Principal Engineer D.R Kidder, Director, Performance Improvement International David Krieg; Senior Engineer, WNP-2 Performance Engineering Group Calvin Robinson; Senior Quality Engineer, WNP-2 Quality Asseuments Yvonne Derrer, WNP-2 INFO Liaison Chadd Bliss, WNP-2 Root Cause Team Technical Contributors:

Kenneth A. Erdman, P.E.; Nuclear Design Engineer, Omaha Public Power District Dave Kobus, WNP-2 John Peterson; WNP-2 Wayne Harper, WNP-2 Tom Eldhardt; WNP-2 Steve Nichols; INFO Dave Thiederman; WNP-2 John Beld'; WNP-2 DaljitMand; WNP-2 A.R Cueto; WNP-2 Pat Cambell; WNP2 Attachments 1. Firemain Flooding Event Time Line 2. Barrier Analysis ofECCS Room Flooding 3. Event and Causal Factor Chart 4. Fault tree analysis for system 81 preaction valve 5. Fault tree analysis for watertight door R013 6. Firemain Pressure Chart 7. FAO Synopsis &Water kLunmer Mitigation EQ'ectiveness Test 8. Failure Analysis ofFP-V-29D Pace

WNP-2 Firemain Flooding of ECCS Rooms Root Cause Analysis September 1, 1998 (

ATTACHMENT1 Firemain Flooding Event Time Line 6/17/1998 Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September 1, 1998 NO.

TIME 0715 over several days 1100 1300 1324 1325 1330 DATE 6/17/98 6/17 6/17 6/17 6/17 6/17 6/17 SOURCE a. Control Room log a. Pre-Job Briefing Checklist a. Ignition Source Permit (98-203)

b. Work Order (KFC9)

c. Transient Combustible Permit (98-165)

d. interviews (workers)

a. CAS printer b. Card Reader printout c. HP fire door typer d. interview notes a. interview notes a. Maintenance Incident Review b. interview notes a. Work Order (KFC9)

DESCRIPTION Initialplant status:

mode 4; RHR pump IB providing coie cooling; dry well being purged using reactor building ventilation; plant startup planned for later in the shiA.

Prejob brieQng conducted for DG 2 room cutting activity Ignition Source Permit issued (expired at 16:00) and work order approved for activity.

Security officer (fire tour) and quality assurance (plant walkdown) individual transited through RHR pump and LPCS pump rooms.

Fire watch posted at door to DG-2. One mechanic assigned to perform watch.

Door impairment reviewed for DG 2 (no door impairment based on PPM 1.3.57, section 4.1.3.f.2 and 3)

Four WINmaintenance personnel began hot work in DG 2 per work order KFC 901.

1340 6/17 a. Maintenance Incident Review b. interview notes Mechanic contacted Control Room Shift Manager concerning smoke from work activity in DG 2 room that was Qowing into laundry room. Control Room advised the mechanic to continue work.

1343:49 6/17 a. TDAS record TDAS record ofdrop in Fire Main Pressure hem normal, 142 PSIG, to low pressure, 33 PSIG.

1345 6/17 a.. Control Room Log Control Room 6re alarms System 66 (Pre-action DG Bldg 441 corridor/Store Room) and System 81 (Pre-action DG2/Day Tank Room), AllFire Pumps started.

1345:25 6/17 1345:45 6/17 a. alarm typer a. Control Room log RHR C Pump Room Water Level High Alarm Control Room received RHR C Pump Room Water Level High Alarm, Entered EOP 5.3.1 on ECCS

WNP-2 Fire looding ofECCS Rooms Root Cause Analysis Se tember I, 1998 Pump Room high water level.

14

16

18

20

22

24

26

28

30

1348 1348 1351 1352 1352:04 1354 1358:05 1358:12 1358:28 1359 1401:57 1401:40 1401:59 1402 1402:39 1402:39 1402:51 1409 1412 6/17

'6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 a. Control Room log a. Ops Crew Debrief a. Control Room log a. Control Room log a. Control Room log b. alarm typer a. Control Room log a. alarm typer a. alarm typer a. TDAS trace a. Control Room log a. Control Room log b. alarm typer a. TDAS trace a. TDAS trace a. Control Room log a. alarm typer a. alarm typer a. alarm typer a. Control Room log a. Control Room log Loss ofRHR-P-3, started RHR-P-2B in Suppression Pool Cooling to ensure system prcssure.

Received a Reactor Bldg radiation high alarm and entered EOP 5.3.1. due to ARM-RIS-11 reading greater than 10000 Mr/Hr(downscale light also on). Alarm is believed to be caused by flooding.

Isolated Fire Protection Systems 66 and 81.

Control Room received report ofwater seepage around the door ofthe RCIC Pump Room from RHR-C Pump Room.

Received E773-B 1-2 battery ground circuit alarm Control Room received report that Reactor Bldg. NE stairway is flooded and fire main riser is ruptured. Directed all Fire Pumps be stopped.

Fire pump 2B power failure alarm Fire pump 2A power failure alarm TDAS record oflast Circ Water Pump House Fire Pump stopped.

Control Room received report that Reactor Bldg. NE stairway water level is approaching 441'levation.

Control Room received LPCS Pump Room Water Level High Alarm, entered EOP 5.3.1 on ECCS Pump Room high water level.

TDAS record oflast Fire Pump stopped.

TDAS record ofLPCS Pump Room Water Level High Alarm.

Control Room received report that water level in Reactor Bldg. NE stairwell has stopped rising.

Started LPCS-P-1 to maintain system operability, stopped LPCS-P-2.

LPCS pump discharge pressure ADS interlock alarm LPCS pump breaker position closed.

LPCS injection less than minimum flowalarm Control Room opened breaker to LPCS-P-2.

Control Room received report that RCIC Pump Room has very littlewater in leakage from RHR-C Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause 'Analysis Se tember 1, 1998 Pump Room and that RHR-A Pump Room has no water in leakage from RHR-C Pump Room.

33

35

37

39

41

43

45

47

1414 1414:08 1414:09 1415 1415 1418 1419 1421 1427 1428 1428 1433 1434 1443 1444 1446 1446:55 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 a. Control Room log a. alarm typer a. alarm typer a. Control Room log a. Chemistry log a. Notification Checklist b. Security log entry a. Control Room log a. Control Room log a. Security log entry a. Chemistry log a. Notification Checklist a. Control Room log a. Control Room log a. Control Room log a. Control Room log a. alarm typer a. alarm typer WNP-2 Declared an UNUSUALEVENTdue to flooding. OSC and TSC are being activated by Team

"C". (EAL9.1.U.I, judgment ofEmergency'Director)

LPCS injection flowlow LPCS pump breaker position open Control Room stopped LPCS-P-1, removed fuses from control power circuit for LPCS-P-1 due to rising water level in LPCS Pump Room.

Initial grab sample taken ofwater in Reactor Bldg NE stairwell (98-1972)

ANS Autodialer activated Control Room declared Secondary Containment INOP to open Doors to aid in dewatering the reactor building.

Required Offsite notifications completed as required by the UNUSUALEVENT.

Hanford Fire Department notified. Confirmed engines were dispatched.

Grab sample analysis count started for Reactor Bldg NE stairwell water. (98-1972)

Confirmation ofall off-site notifications complete.

Control Room received a report that 2'fwater is in the LPCS Room with level rising. Revived a report that the RHR-P-2C pump and motor are submerged.

Water is at 433'nd rising slowly.

One hour notification to the NRC on the UNUSUALEVENTand the 50.72 nonwmergency notification completed.

Hanforti Fire Department arrived on station at WNP-2 to support the loss ofthe Fire Protection System.

Control Room received a report that the water level in RHR-C Pump Room. is 8'reater than the maximum safe operating limit. The RHR-P-2C breaker has been racked out.

Reactor Bldg Secondary Containment pressure low dp alarm. (doors open for dewatering)

Alarm typer received repeated Reactor Bldg Secondaiy Containment alarms due to doors open during dewaterin WNP-2 Firema ing ofECCS Rooms Root Cause Analysis Se tember I, 1998

50

52

54

56

58 1448 1452 1456 1500 1504 1505 1508 15 I0 1512 1515 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 a. Control Room Log a. TSC log a. Control Room log a. Control Room log a. Control Room log a. OSC log a. Control Room log a. Control Room log a. Control Room log a. Control Room log OSC activated.

Carl Becker, OSC Manager TSC declared operational.

Terry Meade TSC Manager The LPCS Pump Room water level has been verified to be less than the maximum safe operating level:

Commenced pumping the Reactor Bldg. NE stairwell water to storm drains after a satisfactory preliminary chemistry free-release sample. (98-1972)

(see entry 74/77 for associated activities)

Transferred Emergency Director duties to the OSC. (error: OSC should be TSC)

Gary Weimer assumed OCS Manager duties.

Control Room received a report that the LPCS Pump Room. is 6" greater than the maximum safe operating level.

Entered EOP 5.3.1 due to Secondary Containment at 0 PSIG. This is expected with Secondary Containment open, Secondary Containment remains INOP.

TSC reported personnel accountability completed. (all personnel accounted for)

Control Room received a report that FP-V-29D, at the bottom ofthe Reactor Bldg. NE Stairwell, has a split in the body and is'the source ofthe flooding. Pumping out the LPCS Pump Room has lowered the water level by 3".

15I7 6/17 a. Crash Network System Log Crash call initiated to inform offsite agencies oftransfer ofEmergency Director duties.

62

64

66 1520 1520 1520:52 1521 1533 1535 1537 6/17 6/17 6/17 6/17 6/17 6/17 6/17 a. Control Room log a. Chemistry log a. alarm typer a. alarm typer a. Control Room log a. Chemistry log a. Control Room log Temporary pumps are pumping water from the LPCS Pump Room to the RHR-C Pump Room.

Grab sample taken at discharge to pump hose (98-1973)

Reactor Bldg Secondary Containment pressure dp high alarm (closed secondary containment doors)

Reactor Bldg Secondary Containment pressure dp normal alarm The TSC reports that the RHR-C water is being routed, via submersible pump, to sump T-4 (under the main condenser).

The water level in the Reactor Bldg. NE stairwell is decreasing.

(see entry 77 for start ofpumping)

Initialsample count completed ofwater from the Reactor Bldg NE stairwell indicated no detectable activity (98-1972).

The CRS has entered the followingPPMs: EOP 5.3.1 on high radiation (ARM-RIS-II) and high water level in RHR-C Room. 5.5.27, 4.11.2.1, 4. 12.4. 10, 4.8.7.1, 4.7.8.1, and 4.12.2.2.

Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis Se tember I, 1998

68

70

72

74

76

78

80

1538 1541 1541 1547 1559 1608 1620 1620 1620 1634 1647 1650 1653 1657 1702 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 6/17 a. Control Room log a. Control Room log a. Chemistry log a. Control Room log a. TSC log a. Control Room log a. Chemistry log a. Control Room log a. TSC log a. Chemistry log a. Control Room log a. Chemistry log a. Chemistry log a. Control Room log a. Control room log IAC has disconnect ARM-RIS-II due to false alarms because it is fiooded.

Control Room received a report that the LPCS water level is rising again.

Count started on grab sample (98-1973)

Control Room has placed DG-GEN-DGI and RHR-P-2A in protected status.

Control room nquested review by TSC to determine ifwe meet coriditions ofEAL8.4.A.5.

Started pump FP-P-3 to confirm the leak in the Fire Protection system has been isolated.

No increase in the drainage from FP-V-29D was observe A review ofin-progress count for 98-1973 indicated the possibility ofCo%0.

Stopped pumping water from the Reactor Bldg. NE stairwell to storm drains, aIIer increasing activity levels in the water. Aligned the discharge to T-SUMP-T4. (see entry 52/77 for related activities)

Decision made by TSC to remain at Unusual Event classification.

Athird grab sample, 98-1974 was taken aIIer the discharge was terminated. (see entry 74)

Recommenced pumping the Reactor Bldg. NE stairwell and RHR-C Pump Room. to the T-SUMP-T4.

(see entry 64/74 for related activities)

Count ofsample 98-1973 completed indicating GHi0 at 5.21E48 uCi/cc.

Grab sample 98-1974 count was started.

FP-V-19, FP-V-17P, and FP-V-17G have been Red Tagged closed to isolate FP-V-29D.

Completed isolating the LPCS Pump Room. from the RHR-C Pump Room. by closing FDR-V-609 (crosstie between the rooms). This had been attempted before unsuccessfully.

1702 6/17 a. Control Room log The system engineer, Dave Thiederman manually isolated air to FDR-V%09. (to close)

84

86

1710 1725 1800 1802 1811 6/17 6/17 6/17

'/17 6/17 a. Control Room log a. Chemistry log a. Chemistry log a. TSC log a. Chemistry log Completed discharge ofFDR-K-9 to river.

A composite sample Gum the discharge to the storm drain was taken.. (98-1975)

Third grab sample count completed (98-1974) and indicated no activity.

TSC determines time to boil estimate to be 2.78 hours9.027778e-4 days <br />0.0217 hours <br />1.289683e-4 weeks <br />2.9679e-5 months <br /> ifshutdown cooling capability is lost.

Composite sample analysis count started. (98-1975)

~

~

WNP-2 Firemain ooding ofECCS Rooms Root Cause Analysis Se tember I 1998

90

92

1832 1&58 1903 1911 1923 2015 6/17 6/17 6/17 6/17 6/17 6/17 a. Control Room log a. Control Room log a. Control Room log a. Chemistry log a. Control room log a. OSC log Groumi alarm on E-Bl-2 cleared. (reasons forground remains unknown at this time)

The TSC reported that fire protection system standpipe venting is in progress as part oftrouble shooting plan.

Reactor Buildingwater level reports in TSC:

RHR< room at 428'7" and dropping approximately 6" per hour RB NE stairwell at 428'nd dropping approximately 1'er hour LPCS room at 426'nd dropping approximately 1'er hour Composite sample analysis completed indicating no licensed radioactivity. (98-1975)

Control Room shUt change John Dabney assumed OSC Manager duties

2059

2100

2057 6/17 6/17 6/17 a. TSC log a Control room log a. Control room log Washirigton Emergency Management updated on event by TSC Manager Fire Protection Pre-action Systems 66 and 81 have been drained and returned to service.

Fire Protection Pre-action Systems 66 and 81 are dechred operable. System Engineer has walked down the system and found no discrepancies.

2144 6/17 a. Control room log RHR-C Pump Room Water Level High Alarm has cleared

2150

2151 100 2205 106 1139 101 2254 102 2331 103 0504 104 0730 105 1124 6/17 6/17 6/17 6/17 6/17 6/18 6/18 6/18 6/18 a. TSC log a. Control Room log a. Control Room log a. Control room log a. OSC log a. Control room log

.a. Control Room log a. Control Room log a. Control Room log TSC participates in NRC conference call.

LPCS Pump Room Water Level High Alarm has cleared.

RHR-P-2C and LPCS-P-1 rooms have been pumped down to only small areas ofwater within the bermed areas ofthe pump rooms. Exited PPM 5.3.1 and 5.5.27 due to threat from fire protection head rupture no longer exists.

Exited PPM 4.12.4.10 (Reactor Bldg Flooding)

Fire protection walkdown ofall ofisite warehouses complete. No leaks detected.

Opened FDR-VM9to continue pumping ofReactor Bldg sumps.

Fire protection system remains INOP (in reference to 21:00 log entry 6/17/98)

Commenced fillingand venting the fire protection header in Reactor Bldg NE stairwell, following replacement ofFP-V-29D. (WO MBK5)

Completed fillingand venting the fire protection header in the Reactor Bldg NE stairwelk Page 7

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis Se tember 1, 1998 107 108 109 110 1626 1727 1727 0539 0648 6/18 6/18 6/18 6/19 6/19 a. Control Room log a. Control room log a. Control Room log a. Control Room log a. Control Room log Declared FP-RB-1 operable and returned to a normal lineup. The Fire Protection System remains INOP.

Transferred Emergency Director duties to Shift Manager Deactivated TSC and OSC.

Performed a bump start ofLPCS-P-2 for a rotational, followingmotor replacement.

Received a Rx Building floor sump R4 level high high alarm. FDR-V%09 failed to close per ARP, placed the control switch in close and still did not indicate closed in Control Room. (known problem)

Verifiedlevel in sump was OK. Plant laborers or Operations willmonitor level as long as alarm is in.

112 1510 6/19

.

a. Control room log FDR-V409 closed.

113 114 115 116 117 118 119 1815 1815 1830 2029 2119 2207 2211 6/19 6/19 6/19 6/19 6/19 6/19 6/19 a. Classification Notification Form a. Classification Notification Form a. Control room log a. Control Room log a. Control Room log a. Control room log a. Control Room log Secured Gum Unusual Event NRC notified ofevent termination.

Offsite agencies notified ofclassification termination.

Started LPCS-P-2 forstarting current and 30 min test run for alignment check Secured LPCS-P-2 to check oil and coupling alignment Declared FP-P-1, 110, 2A, 2B and 3 operable..

Reactor Bldg Secondary Containment declared operable.

Dewatering from fire protection system flooding is complete and doors R108 and R109 are close WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September 1, 1998 ATTACHMENT2 Barrier Analysis of ECCS Room Flooding Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September 1, 1998 ECCS Room Flooding Barriers DOORS AND SEALS FDR-V%90 FROM STAIRWELL TO ROOMS BOTH LEAKED ALLOWEDWATER TO FLOW INTO LPCS ROOM SUMP ALARM SOUNDS IN RADWASTE CONTROL ROOM ROOM ALARM SOUNDS INMAIN CONTROL ROOM DOOR ALARMS RECORD INMAIN CONTROL ROOM NO YES FUNCTIONE D PER DESIGN DOORS ARE NOT DESIGNED TO STOP LEAKAGEFROM STAIRWELLINTO ROOMS SEE FAILURETREE FOR ECCS DOORS

&0PERCENT OF WATERIN ROOM FLOWED THROUGH THIS VALVE.IF VALVEPERFORMED TO DESIGN, INSIGNIFICANT FLOODING WOULDHAVE OCCURRED. CAUSALFACIQR FOR FLOODINGOF LPCS ROOM ALARMCANNOTPREVENT FLOODING, CANONLYADVISE THATIT IS PROGRESS CANNOTBE CAUSAL INFORMATIONWAS AVAILABLE TO INDICATEDOOR WAS OPEN.

CONTRIBUTORTO CONSEQUENCES OF THE EVENT BUTNOT CAUSALTO THE EVENT INSTALLEDPUMPS PIPING AND CONDUIT WALLSEALS PERSONNEL WOULDREMOVE UNKNOWN SOME WATER, BUT, NOT Cm:.MLE OF STOPPING FLOODING ARE DESIGNED TO NO PREVENT GROSS FLOODING, ROOM TO ROOM DOOR TO RHR ROOM YES NOT PROPERLY DOGGED PUMPS CANONLYPUMP SMALL AMOUNTOF WATER CANNOTBE CAUSAL PERFORMED PER DESIGN SEE SS2-PE-98457 OF 6/22/98 THIS CONDLHON CORIRIBUTED TO THE AMOUNTOF WATER THATENTERED ROOM BUTNOT TO THE CAUSE OF THE FLOODING. NOT CAUSALTO EVENT GUARD TOURS TOURS ARE DESIGNED TO DEIZCTEXCESSIVE LEAKAGEAND RESULTING FLOODING YES Page

DUE TO THE SPEED OF THE EVENTTOURS WOULDNOT HAVEBEEN EFFECTIVE IF THE GUARD WERE IN THE ROOM WHEN THE FLOODING STARTED.

NOT CAUSAL

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis Se tember 1, 1998 ECCS Room Flooding Barriers

. TOURS MGMTTOURS TIUQNING SELF CHECKING TOURS ARE DESIGNED TO DETECT EXCESSIVE LEAKAGEAND RESULTING FLOODING TOURS ARE DESIGNED TO LONG TERM INCIPIENT PROBLEMS TINNINGON PLANT POLICIES AND PROCEDURES WOULDASSURE THATPERSONNEL UNDERSTOOD THE NEED TO CLOSE AND LOCKDOORS.

SELF CHECKING WOULD INCREASE POSSIBILITYTHAT DOORS ARE PROPERLY CLOSED ANDDOGGED YES NO NO YES DUE TO THE SPEED OF THE EVENTTOURS WOULD NOT HAVEBEEN EFFECTIVE IF 'BiE.

E.O WERE INTHE ROOM WHEN THE FLOODING STARTED. NOT CAUSAL MGMTTOUR WOULDDETECT LONG TERM FAILURETO CLOSE ANDDOG DOORS. NOT CAUSAL THERE IS NO EVIDENCETO INDICATETHATTHERE WAS A FAILUREINTRAINING SOME EVIDENCEEXISTS THAT

~ DOOR TO THE RHR ROOM

.

MAYNOT HAVEBEEN PROPERLY DOGGED. EVEN THOUGH PROPERLY DOGGED, THE DOORS MAYNOT HAVE PREVENTED FLOODINGOF~

ROOM.

SIGNS, WARNINGS PROCEDURES PLACEMENTOF SIGNS COULD INCREASE POSSIBILITYTHAT DOORS ARE PROPERLY CLOSED ANDDOGGED EQUIPMENT CONFIGURATIONIS CONTROLLED BY PROCEDURE.

NOT IN PLACE SOME EVIDENCEEXISTS THAT THE DOOR TO THE RHR ROOM MAYNOT HAVEBEEN PROPERLY DOGGED.

SOME EVIDENCE EXISTS THAT THE DOOR TO THE RHR ROOM MAYNOT HAVEBEEN PROPERLY DOGGED. EVEN THOUGH PROPERLY DOGGED, THE DOORS MAYNOT HAVE PREVENTED FLOODING OF THE ROOMS.

Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September 1, 1QQ8 ATTACHMENT3 Event 0 Causal Factor Chart Page

WNP-2 Fi Flooding ofECCS Rooms Root Cause Analysis

September l, l998 Firemain Flooding Event and Causal Factor Chart WOT KFC9 prepared lo perform removal of thi DG 2 Peter Diesel Supports Workers reqliest approval of ignition source permit SSS approves Ignition source permit Botges are dltgcult lo bring Into DG-2 dw to curb obstrucUon Security approval requested to have DG-2 door open Fke door Impairment not required approved by Cralt Supervisor per PPM 1.3.57 A

WOT instructions do nol specify method of removal of supports.

Worker experience on DG-1 factored Into decision to change method from grinding to flame cutUng PER 29541 12 CIAs Not Implemented PSRO tt CRS knowledge LTA M!seed opportunity to stop work Work allowed to proceed Fire Alarms recieved In CR Security Guard 4L fire watch stationed, Gas lines routed though door. work begun Torch Cutting resuks ln smoke which exits through DQ-2 door Into corridor Workers stop and call CR to notifythat smoke Is drifting through door into corridor CR acknovdedges that additional alarms may be expefle riced Smoke In corridor causes Zone 88 smoke deteclor lo actwte Zone 88 Preactuation occurs 13:43:49 Fire system pressure drops rapidly CR notified that work was about to begin In DG-2 8 fire alarms may be experienced

FP4 2A &2B Clayton vatvu dump water Initial. Full liowto header In 5 seconds

Jockey pump can't keep up, pressure drops below that required to sustain Rx building standpipe full Rx Bldg standpipe partially voids supplying pieactuation volume FP-P-2A, FPJl-2B &FP-P-110 start e 13 4350 FPJr-110 supplies makeup first Rre 2'.one pressure drops funher to -32 PSIG Three fire pumps supplying e 8000 GPM to fire loop 13 4363 Jockey pump operating, all other fire pumps operable &

In standby System pressure two times chamber pressure

~Ilowlng Cteiiper to overcom"!etch Zone 81 preactuates Design of Fire water system Is LTAto prevent water hammer m

on Zone 81 deluge valve..

tuk aikiwtng chemo to Rdgld to tterdble piping trans!don at FP-Vee snows roroe conoentratlon to occur Preventive maintenance or testing Program fortrim check valves ls LTA Firemaln pressure Increases rapidly FP-P-110, 2A, tt 2B Fire

'unlps ep proachlilg mrcdmum fhw Rx Bldg standpipe void cot!apses result!ng In water miller at end tend pl Water hammer force travels down standpipe to FP-V-29D FP-V@0 ruptures I13:43;54 Fire water dumps rapidly Into NE stairwell Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September 1, 1998 Firemain Flooding Event and Causal Factor Chart (Conse(luential Damages)

Fire water pressure drops as 3 pumps seek to supply broach Stalnvod qutckty IINs WNlwater StatnvoN door buckhs from force ofclslng wahf Water Opens RHR<C PNllp foolldoor R 13 FP4r-1 shfts atter Nme delay (34 second)

D Person on tour of Rx Bldg enters &extts LPCS room vh R4t: dogs *

undogs RA twtce Person Undags RHR.2C door R 13 Person enters RHRW through R 13 Person dogs RHR-2C R 'f3door Person ls expedenced plant worker that has performed many tours In this area of RX Bldg HP Alarm Typer times out (45 seconds) and roconts door opening 131)7 Person nates RA fequlfos earns eldfs fafce to rotate opentflg handwh eel Person notes dear mechnshm h free wheeling and stops to Inspect Firewater pf<<sure stabdzes as 4 pumps supply tho breach Water In RHR-2C rises unrestrained Water level reaches high Iwet alarm switch 13.'45~

RHR-PW keep litt pump aumorgod RHR4r& Fags, breaker trips RHR-2B s'tatted by CR In Supra<<lon pool cooling Water enters LPCS Pump room vta FOR-V~

CR noNNed ofno i!re, but flooding In RX Bldg statlweN Of&54Ctlsch stapplflg aN Rfe pumps 3 Fire Pumps Stopped@13:55-13819, 4th pump stopped O 1401 40 LPCS room high water ahnns O 14N)107 CR receives report that water level In ataftwoN has stopped thing 14M)

LPCS pump started by control coom 140200 Keep NN F

pump stopped FDR-V-609 does not automaucatty close on high FDRA sump twel WOT to nx slow acting valve outshndtng Not aN departments In work phnnlng sonstNvo to Inter room Noodtng F

CR receives report that RCIC room has very INNe water &

RHR-Ahas no water CR stops LPCS pump O141 5 RHR-2C motor reported submerged O 1403. LPCS foam water twel 2 ft Purnpng ofRx Bktg NF Stakwed begins O 1 500 LPCS pump coom water level ls greater lhan max sate oPefadng Ot SCS CR rooslves report that water ls cecoedlng In NE statfwoN 1d:15 RCA Event Time Nne

~nds

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September I. 1998 ATTACHMENT4 Fault Tree Analysis for system 81 preaction valve Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September 1, 1998 Clapper Valve Actuator Manual Actuation Manual Electric Automatic Electrical Fault Mechanical Fault Human Error Drain actuator cylinder vh test valve Pull StaUon Leak develops In actuation chamber/piping Wrong Train U&sohUng test gauge Temperature Induced actuation of pressure switch Actuatkn dhphragm faUs Chpperh tripped Open by 2 to t system to cIMunber press.

Wrong Switch 8umplngl Vlbrathn Sneak circuit Tampering Heat Sensor VibraUon Wear on Actuator push rod Heat Sensor temierature change Wrong type inshlled Induced ElectromagneUc puhe Chpper 0htorted Problems Wl htchlng lever Priming chamber or push rod Forgeln Materhl In valve body Chpper is tripped Open by Water Hammer Page

WNF-2 Firemain Flooding of ECCS Rooms Root Cause Analysis September I. 1998 Clapper Valve Failure Tree Evaluation nual Actuation:

No evidence was found that would indicate that the Clapper Valve actuated because of manually draining off chamber pressure or that the test valve or gauge was leaking pressure.

Additionally, the actuation was in apparent sympathy to the Zone 66 actuation or the forces resulting from the hydraulic water hammer.

Neither of these failure mechanisms would be sympathetic to the actuation of Zone 66 or to the water hammer forces and remain undetectable post event..

Manual Electric:

No evidence was found that would indicate that the manual pull station was actuated.

Remote manual switch actuation is not possible in this circuit.

Automatic No evidence of any Heat detectors being activated or tripped was found during the post event walkdown.

Electric Fault -Pressure switch shorts.

Post event inspection of the pressure switch found it functioning correctly with no evidence of shorting.

Electric Fault-Temperature Induce Actuation of Pressure Switch, Heat Sensor Vibration, Heat Sensor Temperature change, Wrong Sensor installed, Defective Sensor Potential Evaluation.

These mechanisms were not aggressively pursued as likely in this event. There was no significant change in room temperature resulting from the weld-cutting operation or the open DG 2 Room door influence ofthe room

~

cooling. Spurious actuation of the heat detectors is not probable as a sympathetic mechanism to either Zone 66 actuation or forces induced into the building structure by the water hammer and transferred to the detectors.

The Zone 81 detectors were all the correct type and no defective Zone 81 sensors were found post event.

Previous incidents of spurious actuation of Zone 66 (preaction) due to maintenance activities in the area have been documented (PER 298-0112 and PER 295-0423).

However, no detectors in Zones 38 or 39 were Page

ctric Fault-Pressure Switch Vibrates The Pressure switch and the Manual Pull Station support systems were tested for harmonic vibration to see ifan induced vibration could have made contact chatter a capable mechanism.

The input force was sufficiently strong to rule out contact chatter as a credible mechanism for this event.

Electrical Fault - Sneak Circuit and induced Electromagnetic Pulse Potential Evaluation.

Backcaround A preaction system actuation consists of the actuation of the sprinkler preaction valve, which fillsthe piping to the sprinkler heads.

The actuation circuit design can include electrical initiation signals from fire detectors and manual release stations.

When the preaction valve opens, a downstream pressure switch actuates, which is used as a common alarm with the control elements.

This means that, from an operators standpoint, automatic.

manual, or spurious (electrical or mechanical) actuation of the system cannot be determined from inspection of the alarms.

Four fire alarm detection zones are located in the area of the DG2 room (reference Protection Systems Technology Drawing D00292, Sheet 31):

~

Zone 38 is initiated automatically by ionization detectors.

They serve an alarm only function and provide no system actuation.

Zone 38 smoke detectors are located in immediate proximity of the Zon'e 66 and 81 detectors.

~

Zone 39 is initiated automatically by smoke detectors.

They are alarm only and provide no system actuation.

Zone 39 smoke detectors are located in immediate proximity of the Zone 81 detectors.

~

Zone 66 is initiated automatically by smoke detectors (ionization). This zone provides a control signal to the respective preaction valve. The detectors are physically located in the corridor outside the DG2 room.

~

Zone 81 is initiated automatically by heat detectors.

This zone provides a control signal to the respective preaction valve. The detectors are located in the DG2 area, but outside the room where the ignition source work occurre n ~

WNP-2 Firemain Flooding of ECCS Rooms Root Cause Analysis September l. I 998 concurrently exposed to smoke.

None of these incidents resulted in spurious actuation of P81.

During this event, the Control Room reported that Zone 66 and 81 actuated concurrently. This occurrence is not considered credible from a valid electrical signal, since it is unlikely that the heat detectors were exposed to a rapid rate of rise or reached the actual temperature required for them to alarm (200 degrees F).

A~nal nln The actuation circuitry applies 24 VDC across the detectors connected in parallel, and monitors the current signal provided to generate trouble and alarm signals.

A current signal indicative of a break in the detector circuit generates a trouble alarm. A short in the detector circuit produced by actuating a detector produces a fire alarm and subsequent actuation of the associated preaction valve by energizing the solenoid to actuate the preaction valve.

An electrical interaction between the Zone 81 detectors can only exist with Zone 38 or 39, since the Zone 38 and 39 detector wiring is the only other wiring routed in conduit with the Zone 81 detector circuitry. It is likelythat actuation of the Zone 38 and 39 smoke detectors did occur which could possibly have actuated Zone 81.

However, this can not be confirmed since the alarms were immediately reset which automatically clears the individual detector alarm indication. The following analysis demonstrates that actuation of these zones resulting in a spurious alarm signal being generated in the Zone 81 detector circuitry is not credible.

As noted in the Pyrotronics CP-30 vendor literature, the magnitude of the current requirements forthe smoke detector devices is on the order of 100 mA/device, when in alarm. The alarm signal is produced simply from the bi-stable trip devices.

Inductive coupling between the detector circuitry could exist, but this coupling would have to be extremely close, since the output of the CP-30 alarm current is limited to 200 mA. Additionally, an EMI-induced signal on the Zone 81 circuitry due to inductive coupling could most probably only occur due to high frequency current present on the Zone 38/39 circuitry. Such a current signal could most probably only be produced by contact chatter.

The only control elements in the Zone 38/39 circuitry are photoelectric detectors, due to being alarm only, with no preaction valve solenoid actuation or flowswitch present in the circuit.

Therefore, the probability that this coupling exists is extremely remote.

Capacitive coupling between the Zone 38/39 circuitry and the Zone 81 circuitry is also possible.

However, this would necessitate a voltage transient be present on the Zone 38/39 circuitry. The nature of the circuit, consisting of bi-stable trip devices, small loads (milli-amps when the devices are in alarm), and an end-of-line capacitive load seems to preclude the potential for a fast-acting voltage transient to be generated on the Zone 38/39 circuitry.

A 'floating ground'eveloping on one of the detector circuits is also possible, such as a high resistance at the high or low side of the ZA-30 zone module detector loop connections.

This type of spurious signal has been investigated and discounted.

Both Class A (heat detection) and Class B (smoke detection) circuits are supervised, which means that a current signal indicative of an open fault or excessive line leakage willtrigger a trouble alarm. The absence of this alarm provides strong indication that a circuit grounding problem did not exist.

This concluc to~ is further supported by evidence that functional testing of the Zone 38/39 detectors has never resulted in spurious actuation of the Zone 81 preaction valve. Additionally, no evidence of maintenance effecting either the Zone 39 or 81 circuitry (wiring) has been found. Therefore, the possibility of an inadvertent electrical interaction between Zone 81 and any other Zone detector circuitry can be discounted unless, and until, other, more plausible explanations are disproved.

Conclusions It is concluded that the actuation of the Zone 66 or Zone 38/39 systems resulting in the actuation-of Zone 81 due to an electrical interface between the two systems is not credible.

Mechanical Fault-Leak Develops In Actuation Chamber/Piping & Clapper Is Tripped Open By 2/1 Sytem To Chamber Pressure.

A post event leakdown test of the header identified that chambper pressure followed system pressure for 4 of the 6 preactio>> sytems in the test. The pressure decay response was to approximately 30 psig within a few seconds Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September I. 1998

~

~

on zone 81 & 82's clapper valves.

The most likely cause of the loss of chamber pressure in response to system pressure decrease is that the trim check valve is not maintaining the chamber pressure.

The check valve is mounted horizontally and is not spring loaded.

Inspection of the check valves did not reveal significant aging or defects.

A vendor modification is available that would revise the trim design to include two spring loaded check valves mounted vertically to provide greater assurance against loss of chamber pressure in sympathy to a drop in system pressure.

Zone 81, 79 and 83 are on an eight inch header pipe and Zone 82, 80 and 84 tees offof the eight inch header through a reducer to a 4 inch header.

The piping configuration plays a role in the setting up the conditions where a 2/1 system to chamber pressure condition could be established to affect Zone 81 first. Once Zone 81 actuates, it would create a local pressure drop allowing the remaining zones additional time for chamber pressure to build to system pressure prior to reaching the 2/1 action point.

The histogram of the firemain loop pressure shows that at approximately 4 seconds after the start of the pressure drop, the conditions were established to place a 2/1 ratio of system to chamber pressure on Zone 81 and Zone 82. Zone 81 appears to be the most probable to actuate first as it is supplied by an eight inch line.

This mechanism is the most likelyof the failure tree.

Mechanical Fault-Actuation Diaphragm fails, Clapper Is Tripped Open B Vibration, War On Actuator Push Rod, Corrosion In Actuation Chamber Or On Actuator Rod, Clapper Distorted, Problems W/Latching Lever, Priming Chamber or Push Rod failure, 0 Rings, Foreign Material in Valve Body, Clapper Tripped Open by Water Hammer.

Internal inspection of the Zone 81 dapper valve did not reveal any significant degradation, wear or foreign or corrosion. Thus these mechanical fault mechanisms are not probable for this event.

man Error-Wrong Train, Wrong Switch, Bumping/vibration, Tampering, Aberrant behavior Post event interviews with workers in the area and the lack of physical evidence relating to human actuation make these Human Error mechanisms unlikely in this event.

Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis September I, 1,998 ATTACHMENT5 Fault Tree Analysis for watertight door R013 Page

WNP-2 Fire ooding ofECCS Rooms Root Cause Analysis September I, 1998 FAIf,URE TREE FOR OPENING OF WATERTIGHTDOOR R-13 (STAIRWELTO RHRW P UIIIPROOhj)

FLOOD WlNAllOR DOOR OFKRD DVltNO FLOOD CVBIT CTKKIIKAL fALIAI~ Of DOOI TKlCIIKAL IALINIOf004 NOONMNI CODON IN%INC WIIND040 NOCNNNN C1 0004 TNWONAN.T

~NNINIDIONtON CLIANNOCNI CCNKIIOICDON

02 IINWINCTILDWOf

'OATWIWOWNNO ITMOOOCLKNI ONIICllON ALLDWO 004 4 TO NNODIWI C2 NNONtANNO IINDIANIDOOII NINIf1000 OCCLNO 014 tIAANfOWVDN OANTNWCt NANOAWITOITDtt TAIWO 024

~ONON tOIWOTT TO DOO DNNI NOAIAIDIN I~IONNO C2

NIPPON IOIWCIOTO CLON DION NO IWOKNNITO NOID004INIIAL 004 ONINISDON

~4 COON

%OK@KILT IOO DCNNIOtNI

~OI OCNNCOW ONO N IOIADONO DTO CD DTD NltWCKLLAN10 0044 NOT TINDOOII 0251 ANAT IIAL004\\NI KIKM OHNC N IIACN~

~OKNNNNAN~

CTCOO OCHER tNOII TO tILLN D201 CONIC N004 IDIO CICOO 0044 IKON TO tILLN 01DT ~

~CIWON0 NOT TIAANITO KNOW NOWlllTNANINI TNO 0000 K0044NI COCCI 0 IIILT Page

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis Six Month Pre-Event History ofRHR-C Door R-13 Alarm Typer Door Open Period Event Date 6/17/98 RHR-C R43 Door Open 3/11/98

.540 t-35 Q

O30I E25 l-

3/19/98 F6/98 6/3/98

10 From Jan'o Sune 1?, 1998

WNP-2 Fire looding ofECCS Rooms Root Cause Analysis Six Month Pre-Event History ofRCIC Door R-05 Alarm Typer Door Open Period Event Date 6/17/98

~

~

~

~

~

g

~ ~ ~

RCIC.Room Stairwell Door RS Open 6 Month Preceding the Event.

5/23/98

~ 60 c

Q CL

El-

3/21/98 3/16/98 3/24/98 5/15/98 5/15/98 6/5/98 to 16/

5/6/98 5/1 5,16,20,2 6/12/98 Jan 98 to June 1T, 98

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis FLOOD MITIGATIONDOOR OPENS DURING FLOOD EVENT Based on the evidence at hand and the fault tree analysis below there are many plausible ways for the R13 door to have come open during the flooding event ofJune 17, 1998. The most probable ofthese mechanisms can be summarized under the heading of

"human error" and involve leaving the door unlatched or only partially latched. Though less probable, it remains plausible that the mixture ofwater and debris churning through the area outside door R13 after the collapse ofthe stairwell fire door could have moved the door handle to a position that allowed the door to open.

A Post flood inspection reveals no structural damage to door panel.

B Post flood inspection reveals no structural damage to latch mechanism.

C Post flood inspection does reveal that door R13 is in the open position. The fact that thc 6" level alarm in the RHR C pump room activates during the first minute ofthc event indicates that gross leakage across the doorway.

The HP alarm typer provides an indication that door R13 was opened at 13:07 on the day ofthe flood and remained that way until the door switch went to the faulted condition 7 minutes into the flood event, at 13:52. However, interview information with two individuals in the area provides detail for 12 times that doors leading to or exiting fmm the LPCS and RHR C pump room should have shown an open event and close event on the HP typer. Ofthese 24 events, only the R13 open event at 13:07 is recorded..

These door position switches are not used by any organization at this time and are not maintained. In addition, a close signal would provide no information concerning whether the doors are latched or not.

Cl No evidence or intcrvicwknowledge leads to this possibility. This isju(lged unlikelyforthis event since no known cleaning, maintenance, or construction was occurring in the RHR C pump mom at thc time ofthe event. The two individuals noted in thc interview data above state that no activity was going on in the room when they passed through shortly after 13:00.

C2 No evidence or interview knowledge leads to this possibility. Ifthis occurred and the person fled after hearing the valve break occur and the flood begin, they have not come forward. Itis unlikely that a fleeing individual would have carefiiilyclosed and latched the three doors encountered on the way out through the RCIC room. This failure mechanism is considered very unlikelyfor this event, but remains a plausible failure mode.

C3 Based on interviews and security door information interviews were conducted with two individuals that had gone through door R13 shortly before the event. One individual was a security guard on fire tour, and the other was an individual fiem the QA organization.

Based on the interviews it is unlikelythat either ofthese individuals forgot to close the door. Ofthese two, the QA person was last through the door and spccifically remembers that the hand wheel moved very &eely compared to door RS which he had just exited. He also remembers many details about the process ofclosing door R13, down to which hand he used and what he was doing with his other hand at the time. There is no conclusive evidence that this person was thc last one through the door.

Two PERs were written in thc thee years preceding this event on ECCS pump room doors that were found unlatched.

This fiulure mechanism remains plausible.

C4 No evidence or interview knowledge leads to this possibility. This failure mechanism is plausible, but judged unlikely in this event based on interviews with two individuals who were in the area shortly before the event.

CS No evidence or interview knowledge leads to this possibility. In addition it seems very unlikely'that the act ofleaving a door open would bc chosen to demonstrate defiance or to purposely disable the plant.

D Dl Dla No inslicction evidence or interview knowledge leads to this possibility. The dogs on R13 were able to fullyengage.

Dlb No inspection evidence or interview knowledge leads to this possibility. The dogs on R13 were able to fullyengage.

Dlc No inspection evidence or interview knowledge leads to this possibility. Though the hand wheel could spin very freely on door R13, itdid not spin by itselfwhile closed or open.

Dld No inspection evidence or interview knowledge leads to this possibility. Though the hand wheel could spin very freely on door R13, itdid not spin by itselfwhile closed or open.

D2 D2a Based on interviews and security door information interviews were conducted with two individuals that had gone through door R13 shortly before the event.

One individual was a security guard on fire tour, and the other was an individual from the QA organization.

Based on the interviews it is unlikely that either ofthese individuals forgot to latch the door. Ofthese two, the QA person was last through the door and specifically remembers that the hand wheel moved very Qeely compared to door R8 which he had just exited. He also remembers many details about the process ofclosing door R13, down to which hand he used and what he was doing with his other hand at the time. There is no conclusive evidence that this person was the last one through the door.

Two PERs were written in the three years preceding the event on ECCS pump room doors that were found open. This failure mechanism remains plausible.

D2b Scratch marks in the paint next to the dog holes provide evidence in support ofthis theory. Inspection ofdoor RS aficr the event demonstrated that a door only partially dogged can undog itselfwith no outside intervention.

D2bl D2b1 A Interviews indicate that workers know in general that fire doors should be closed and latched. The purpose ofthe dogs and

WNP-2 Firemain Flooding of ECCS Rooms Root Cause Analysis reason that they are important was less well known. In general, the method for determining whether the door latches are fully ged is that the handle stops rotating.

Some individuals do a second check ofpulling on the door. Very few individuals knew of certain method for determining whether the dogs were'fully engaged.

This failure mechanism remains plausible.

D2b 1B There is a circular piece ofmetal with a cut out where the locking mechanism could engage when the door latches are full in. Since there is only three quarters ofa revolution from fullout to fullin, it is possible to see for certain when the hand wheel has come to the fullin position. However, this circularplate has been covered with a very large circular yellow and black plate to keep people from injuring themselves with the hand wheel mechanism.

This makes itvery difficultto see the locking mechanism.

Certainly no person traversing these doors without specific training would know to look for the locking plate. Though it is possible to tell when the door is secured, this remains a plausible contributing factor.

D2b2 In support ofthis possibility, several individuals report that afier the flood it was possible to spin the hand wheel closed and have it rebound to the fully disengaged position. No evidence leads to the fact that this specific failure mode occurred prior to the flooding event, but the failure mechanism remains plausible.

D2b3 No inspection evidence or interview knowledge leads to this possibility.

D2b4 There is no evidence that this was true ofdoor R13. Interviews indicate that the hand wheel traveled freely over its entire range ofmovement.

However, NRC personnel found that door RS came to a point where the feeling ofbeing fullyengaged seemed possible.

This remains a plausible failure mode.

D2bs D2b5A No inspection evidence or interview knowledge leads to this possibility, but the method ofMure is plausible. Aperson in a hurty could have the hand wheel in motion before the door is fullyclosed. This would cause the dogs to stop against the jamb or dog blocks without engaging, but giving the impression that the hand wheel had come to the end ofits travel.

D2b5B No inspection evidence or interview knowledge leads to this possibility, but the method ofMure is plausible. Aperson who was not paying close attention, or did not check to see that fullclosure had occurred could close the door with debris trapped between the door seal and the knife edge. This would cause the dogs to stop against the jamb or dog blocks without engaging, but giving the impression that the hand wheel had come to the end ofits travel.

D2b6 No inspection evidence or interview knowledge leads to this possibility, but the method offailure is plausible.

Hard debris in one ofthe upward fitcing gear racks could bring the hand ~heel to a fullstop giving the impression ofthat the dogs were fullin.

h debris may have been washed away during the flooding event.

7 No inspection evidence or interview knowledge leads to this possibility, but the method ofMure is plausible. Hard debris in ne of.the dog.holes could bring the hand wheel to a fullstop giving the impression ofthat the dogs were fullin. Such debris may have been washed away during the flooding event.

D2c No inspection evidence or interview knowledge leads to this possibility. The door is inside an alcove out ofthe way so that a person would not brush it moving by. However, the ease ofmovement ofthe hand wheel on door R13 leaves the possibility open.

E Ifthis is the mechanism by which the door was opened, it would have to have happened very early in the event. The fact that the 6" level alarm in the RHR C pump room activates during the first minute ofthe event indicates gross leakage across the doorway.

El Requires that water enter the alcove to door R13 swirling at a sufficient rate and direction in a vertical plane to push the hand wheel to rotate open. In addition, in order for water to move the hand wheel it has to be at least as deep as the lower portion of the hand wheel. The pressure ofthe water at this depth should provide increased friction to oppose the movement ofthe dogs coming out oftheir holes. This possibility seems unlikely. However, the interview with the QA person and post flood inspection both indicate that the R13 hand wheel turned very freely.

E2 Requires that water enter the alcove to door R13 swirling at a sufficient rate and direction push debris at a fair velocity into the hand wheel to rotate it open. In addition, in order for a floating object to smack the hand wheel the water must be deep enough to cause the debris tv contact at least the lower portion ofthe hand wheel. The pressure ofthe water at this depth should provide increased friction to oppose the movement ofthe dogs coming out oftheir holes.

Based on post flood inspections, only an eighth ofone revolution ofthe hand wheel is required to back the dogs from fullin to the point where the untapered section ofthe dogs are not engaged. Ifthe dogs werc engaged to a position where the untapered portion is not engaged, pressure from rising water would open the door. However, in support ofthis theory, there was a large metal tool box resting in the alcove to door R13 afier the flood water was pumped out. The box is large enough to provide substantial momentum as it moved. The box is tall enough that the depth ofthe water when the banging occurred could have been fairly low, providing little pressure against the door.

In opposition, to this tool box theory, it is required that random water motion moved the tool box to exactly the right ation with one end poked inside the alcove all the way to the door just as the water level was rising to the right height. The fact t the tool box was in the alcove aher the event may not be good evidence ofculpability since water currents would have moved the box to that location afler the door R13 was open; or even during the pumping down ofthe rooms.

E3 Requires that a piece offloating debris have a mechanism to get caught on the hand wheel and be at precisely the right location and orientation to do so. In addition, in order for a floating object to move the hand wheel the water must be deep enough to

WNP-2 Firemain Flooding ofFCCS Rooms Root Cause Analysis cause the debris to contact the lower portion ofthe hand wheel. The pressure ofthe water at this depth should provide increased friction to oppose the movement ofthe dogs coming out oftheir holes. This possibility seems unlikely.

However, in support ofthis theory, there was a large metal tool box resting in the alcove to door R13 after the flood water was pumped out. Inspection in the days to followrevealed that the tool box has a handle on the side which is constructed in a manner to make it plausible that the door latch could have been caught as the tool box floated up. The box is large enough to provide substantial force as it floated up. The box is tall enough that the depth ofthe water when the handle caught could have been fairly low, providing little pressure against the door.

In opposition, to, this tool box theory, it is required that random water motion moved the tool box to exactly the right location with one end poked inside the alcove all the way to the door just as the water level was rising to the right height. The hct that the tool box was in the alcove after the event may not be good evidence ofculpability since water currents would have moved the box to that location afler the door R13 was open; or even during the pumping down ofthe rooms.

F F 1 The LPCS door, R8, which was subjected to the same water pressure did not come open.

F2 No evidence or interview knowledge leads to this possibility.

F3 The LPCS door, R8, which was subjected to the same water pressure did not come open.

However, ifthe door had been locked closed, many possible ways to get the door open would have been eliminated.

'I WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis

E ATTACHMENT6 Firemain Pressure Chart

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis FIRE MAINLINE PRESSURE Data plotted without time shift 150 140 130 120 110

~ 100 f

80 IL

60 Pn

40

20

0

>><<".g>gepyj:,!p,".".<<(gjp~<<.

IA CO Cl g~>>@QP+pC9@'@p~>>

~~sQQs-~~"

,Qt(~gP~RÃ<<<<<Y

<N(~PJ+-

!@~IF/Pi!q i5~i:~~!

<<

Rglg@gK '-

4j',"-p b~m~Q <R'<"",

WhlP-2 Firemain Flooding of ECCS Rooms Root Cause Analysis ATTACHMENT7 FAO Synopsis Water Hammer Mitigation Effectiveness Test July 2, 1998

WNP-2 Firemain Flooding of ECCS Rooms Root Cause Analysis C

Analysis from FAO to PER-298-0782 (synopsis)

The WNP-2 fire water supply system primarily distributes water to numerous suppression systems and standpipes in the Reactor, Radwaste, and Turbine Buildings. The highest standpi pcs are RB-1 and RB-2 risers that extend to the 612-foot elevation.

RB-1 is the larger ofthe two, consisting ofprimarily 12 and 8-inch diameter piping where RB-2 uses primarily 8 and 6-inch diameter pipe.

Other standpipes in the system rise only to 530 feet and are disregarded for purposes ofthis evaluation. The 66 system preaction valve is at the 445-foot elevation and uses a common supply to RB-2 but not RB-l.

When the preaction valve 66 opened the jockey pump could not maintain system pressure which dropped from a nominal 140 psig to 30 psig (TDAS). Therefore, the pressure at the top ofRB-1 and RB-2 dropped to vapor pressure (i.e. voided) as the standpipes initiallysupplied water to the preaction system piping. RB-1 standpipe is credited with providing most ofthe water supply in these initial moments since it is physically larger and less resistant to water flow.

As the main fire water supply pumps came on line due to low system pressure, the voided portion ofstandpipe RB-1 started to fill rapidly. The final rapid collapse ofthe vapor column resulted in a water hammer.

The 300-psig pressure gauge at the top ofRB-1 over ranged and became inoperable during the event. The surge pressure wave caused displacement ofthe water supply piping in RB-1. Post event inspection ofthe RB-1 pipe support system shows that piping above FP-V-29D experienced negligible movement but there appears to be movement below the valve. This suggests that pressure wave stresses concentrated at FP-V-29D leading to its failure.

~,i'-

Revision 0 to the FAO concluded that other plant piping integrity was undamanged and that maintaining two main fire prumps running continuously would preclude further water hammers.

Aflerperforming dynamic compluter modeling ofthe WNP-2 fire water system, it was concluded an air cushion on the top ofthe RB-1 and RB-2 risers was also needed.

Nitrogen bubbles were then installed, with the level monitoring equipment inspected hourly. The isolation valves at the bottom ofthe RB-1 and RB-2 risers were replaced with cast steel valves. Later FAO revisions addressed the potential ofa LOOP causing water hammer and the needed interim limitations ofn the use offire system water. By mid August,'Bechtel had refined their dynamic analysis to the point where there was adequate assurance that their HSTA program did accurately and conservatively model the WNP-2 fire water system. Th dynamic analysis concluded that running one fire pump adequately mitigated the magnitude ofpotential water hammers.

Thus, Revision 5 ofthe FAO now requires a single main fire pump remain operating at all times.

FAO Summary Thc FP water system is currently "operable but degraded".

This conclusion is based on piping inspections, dynamic modeling, system modifications, administrative controls and the system testing discussed below. The FAO states this configuration willbe maintained until the corrective action plan is implemented to eliminate the water hammer vulnerability.

System Functional Test Prior to R-13 Restart Prior to R-13 restart, PPM 8.3.403, "Fire Protection Water Hammer Test" was performed and consisted ofan actuation ofpreaction system 66 and a second test by simultaneously tripping preaction systems 66 and 81. The tests validated that the FAO configuration (two fire pumps running and nitrogien bubbles in RB-1 and RB-2) did preclude the potential for destructive water hammers.

The results are that with FP-P-1 &FP-P-2B in operation, the loop pressure decrease was less severe (improved from <32 psig to -90 psig). A sharp pressure drop occurred in the two reactor building standpipes (dropped from 70 psig to a negative value in -3.5 sec and recovered to 90-150 psig in 12-15 seconds) and in the readwaste building standpipe (dropped from 120 psig to negative and back to -35 psig in the first second and stepped up to -65 psig in 6 seconds and to 120 psig in 8-9 seconds) indicating that the standpipcs were participating in fillingthe preaction systems. This also indicates that the FAO modifications mitigated the resulting prcssure wave. Although thc loop pressure dropped suQiciently to cause the start ofthe two standby pumps (means pressure at the pumps sensors dropped somewhere below 100 psig), only a small pressure wave was created by the four participating pumps. The lowest level offire loop pressure recorded was during the dual system actuation which reached approximately 99psig (see attachement 10). Actual pressure was most likely 5-10 psig less as TDAS data was saved at 1 second intervals and most likely missed the lowest pressure level sensed by FP-PT-11. Observations along with accelerometer and strain gauge measurement showed that the pressure wave created was significantly mitigated and no damage occurred.

WNP-2 Fircrn i Flooding ofECCS Rooms Root Cause Analysis Post FAO Modification Water Hammer Mitigation Effectiveness Preaction Zone 66 Test 180 160 140

'?coax...::?

~x:i5:pC'i,,.

.':CA '?xN'.

NNA?x~?

",;..i!,.",::.?"'c,".'":3:42i

"Pjc~"-";",,'.

.'y~g4<g x

'c?,dc'.

?

?'Icm~+~??44

$ Vcc???

?

'+~AN

.a?'pWg A?I

~<<"'CvNcc?4

~++"

g~c~y+"?.

'7'w w.'j pj's 44x44?~?CZAR

~ gN?C,?j

'

"4??c":;~c.':, K--.":~?:

S=:-.".

P Loop e

.i'ressur

'c

~ 'rS 'j~ccII Cv Xc o

120 CL M

100

,?','yy;~4~+

"g8a,"g~g

?4~v ?vM:x

~pal".?pg gPj$

<:c g?44'?44vyN K~'x"?..cW?

, '.',

?Nc,.

ck'0

40

0

"='yacc,:? i".QN:

CCCCVXvXCVX xc.N4v x,P,;g,P'?

"""""'V'CC.

."<<Fgng

~:~?,".cxM+

Xc?v

',c~N~X"vy;:

N?N

'ccc )??ceo II~&;,">>'?3c,c?cpccrc'c

.yx?;~lcjV,,

g'xN@Ã~I'h 4 4'c?c>??'

4%4'.v vy4W>I

?&M@K P~?c<?Ia Mg$

c ?Vx~cS'CN jk~yy,,'I cc A~x~@N)?

4y,~CgF:4R

~MANMÃ::

VAMa~v'4?4 kx?:c??c:.'4 (~~~M'x c..N? ".

y~y"I'yc'Icu

~xj%@

. 4c VRF~PJ',

':.?'I,"IF'>)

y.

4V'?c"C~8$

rye~@':,":~'v4

,V4+

MQ?xy?4.

'x4yv.Nj?P.,

Nc'Y ~+j~j

,<Ccrc?g?3

@4~+4?c

4 v~'4?

~W44~m cc c?NVxym

'xN W

cvN'

C9

Q

~ lA lA If)

0 CO CO

IC)

0 Q

IA O

Q

0

v v-O Ih O

0)

O lA Pl C9 CO W

W Ql IA Time (seconds)

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis 180 160 140 120 100 OI-P

L

40

0 O

lA 0)

IA 1.441, Post FAO Modification Water Hammer Mitigation Effectiveness Preaction Zone 66+ Zone 81 Test

<<C"'%: <<>

':I':i:.:

g<<c> <<><>3:Y+

'>><<~< "c<<:c"'y>i

'+'1,1'.<<

++'+,

%Q 'CC jjg,(3<

p$"kgb;.

O'S%?:it:>>'r<

FP Loop Pressu CPr.C:>

K%":;:

,.'Y:

<<y

>45<N@S

'~+....w<<

'%"z".~d:

Ãc~>c+"!1y

'.>v<<c>a>'~c4Z$~ <<

~y... <33N@

~c,>y>,<<ccc>r'3

'P<.,~+@

.<g., r~ <',c:"3

8'

.c:.Xyy<<3<<'::<P~gg<<:0

. p<<<<3 y"v~m

%X!Ij":

"ygy~<~gP c~jgOyg<3) <<<

y ~<<sg$g

.;~~~,4y

<<N C.

L~j @<Cga

~;":,+Spm<

,."C <<'>r!y<<,',.q,..3 y,:, "3v;>P:)y$:

."c<<ccv.'~Wj<c<>

p'c

>3y'3.">&

'Mc<v<<>

>3 <H~CX

,~!a~Y>~k "k~<Q~ 4

+4@v<+v>

MX<Fk%<<.-%9'c<vc~

'j~c;.">YP~c,.

3y3<'c'K%A y<<<<y'<~<<Ã.3 N84~""+i+<<

"K>$'>j:~N

c'4'jX"a~>"'c pp<<p<:

.c3

~" p'~j'j IA O

O O

lA O

C9 O

O T

O O

O LA O

O O

O IA tA O

O O

O IA O

IA IA O

O O

'3A tA IA IA

'IA IA O

O Time (seconds)

"0 0,

WNP-2 Fire looding ofECCS Rooms Root Cause Analysis

'est-1 Actuation Zone 66 StandPiPe RB-1, RB-2, & RWB-q Response Data Estimated from Traces or Ndeo Accuracy+I-10% Full Scale 160 140 120 100 CLI Pn

O.

I.

40

~RB-1 (From Trace)

~ RB-2 (From Trace)

~RWB-1 (From Video Camera)

-20

Time (Seconds)

20

33

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis Test -2 Dual Actuation Zone 66+ 8't Standpipe RB-), RB-2 & RWB-1 Response Data Estimated from Traces or Wdeo.

Accuracy+/-10% Full Scale 180 160 140 120 100

a

8M

20-20

10

.Time (Seconds)

~RB-1 (From Trace)

~ RB-2 (From Trace)

~ RWB-1 (From Video Camera)

25

0 g WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis ATTAC NT 8 Failure Analysis of.

FP-V-29D

~

~

WNP-2. Firemain Flooding ofECCS Rooms Root Cause Analysis WASHINGTON I'UBLICPOWBR 44 SUPPLY SYSIXQH INTEROFFICE MEMOIVWDUM DATE:

September I, 1998 TO:.J. E. Peterson, Fire Protection Engineer FROM:

J.R. LaSalle, Materials and Welding SUMECT:

FAILUREA'NALYSISOF FP-V-29D REFERENCE:

1) Materials and Welding lab no 98-022 2) A Fractography Atlas ofCasting Alloys, 1992 Battelle Press, 1992.

3) ASM metals handbook, Vol.. 1, 1990.

SS2-PE-298460 Distribution:

JR Cole F Diya TMErwin WAEstes DR Kobus CM King JE Rhoads DS Mand LD Noble WJ Harper AALangdon RH Tones MP Reis JA McDonald SW Oxenford WNP-2 Qles JRLlLB PE24 988S PE27 988p PE27 PE27 PE20 PE24 PE24 PE27 988S PE26 PE26 927P 9270 964Y

~,

The examination ofFP-V-29D has been completed. The fire protection valve is a twelve inch gray cast iron gate valve manufactured by M&H. From the investigation it has been concluded that the valve failure was caused by dynamic loading ofthe valve during the fire protection water hammer event.

The cracking appears to have initiated at or near the end ofone ofthe four integrally cast gate guides in the valve body.

The rapid propagation ofthe crack through the heavy flange section and into the bonnet ofthe valve is indicative of a high strain rate and displacement caused by the water hammer.

From the orientation of the,,

&actures the initiation is believed to have had components of torsional shear and tension.

This conclusion is supported by the direction and orientation of crack propagation, metallurgical evaluation of the crack,,.

morphology'and scanning electron microscopy (SEM) ofthe &actrrre surfaces. No preexisting flaws or inclusions..,

were observed in the cast iron &acture surfaces.

The valve body material chemistry and microstructure was characterized as a pearlitic gray cast iron which is consistent with the ASTM A126 specification.

Corrosion of the continuously wetted gray cast iron was observed and through metallurgical examination the wall loss was estimated to be 18 Mills. This is considered acceptable given the 80 Millsdesign corrosion allowance for the system.

Procedure The fracture surfaces were examined visually. Sections containing &acture surfaces that were deemed important based upon location were cut from the valve for macro examination using a stereoscope at magnification up to 60X. A metallurgical examination of selected samples was conduct'ed to verify the microstructure at 25-1000x magnification and to determine the general cast iron corrosion rate.

Secondary cracks found in the body to bonnet flange area were also sampled to determine if they were pre-existing.

Two &acture surfaces wer examined using an SEM (Scanning Electron Microscope) to determine the loading direction. Carbon and sulfur analysis was performed on samples ofthe valve body material to help characterize the casting material.

WNP-2Firemain Flooding ofECCS Rooms Root Cause Analysis E. Peterson, Fire Protection Engineer

'lure analysis of FP-V-29D age 2 of3 Results The failed FP-V-29D is shown in Figure 1 in the as found condition in the stair well.

The crack, noted at the arrow, was observed to follow the edge of the integrally cast gate guide which helps explain the location and symmetry ofthe cracking. It was also noted that the crack opening was minimal which is an indication that the valve displacement was limited.

The visual examination ofthe valve yielded no evidence of inherent valve material casting defects. The &acture surfaces were observed to display uniform color and texture which indicates the casting was homogeneous.

No notable inclusions or flaws &om the casting process was observed.

Figure 2 shows the valve removed &om the system. It is believed that the cracking initiated at or near the edge ofone ofthe integrally cast gate guides noted at (A). One ofthe integrally cast gate guides is noted at the arrow (B). These guides run the length ofthe valve body casting and represent a change in casting section thickness.

Figure 3 shows the direction of crack propagation that occurred due to the event.

Initiation is believed to have occurred at (A) as a result of combined tensile and torsion loading.

The crack propagated &om (A) through the body of the valve and the body to bonnet flange connection terminating at (B).

It is likely that simultaneous crack propagation'also occurred &om (A) to (C). Only one area on the valve body was observed to display multiple crack initiations.

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

gure 4 shows this area.

The approximately 45 deg. angle shown is indicative of torsion shear for a material th low ductility. The short angled crack propagated throughout the thicker gate guide at this point.

The inside ofthe valve body is shown in Figure 5.

The end ofthe integrally cast gate guide and the 45 deg. shear is shown to go through the thicker section ofthe gate guide.

The cracking in the (B) direction clearly followed the edge ofthe gate guide which had a smaller net sectional area.

The microstructure displayed a relatively uniform structure and distribution of graphite flakes (Figure 6).

A

"""".equent etch verified p".".."litic gray cast iron (Figure 7).

The graphite morphology and distribution is characteristic of ASTMtype VIIgray cast iron size class BS. The secondary cracks in the body to bonnet flange area were determined to have initiated during the event which was evident by the &esh &acture surfaces.

From the carbon sulfur analysis it was determined that the casting carbon and sulfur content was 3.65/o and 0.134/o respectfully. The valve body material was specified as A126 class B.

The only chemical requirements for this specification is the limitation on the Phosphorus, 0.75 max. and the sulfur 0.15/0 max.. The SEM examination of two fractures displayed primarily tensile

&actures with no shear evident. Figure 8 and 9 shows the typical fracture appearance.

Discussion From the investigation it can be concluded that the valve failure was caused by dynamic loading of the valve during the water hammer.

No flaws in the casting were found that would have initiated or were a precursor to the valve failure. The rapid propagation ofthe crack through the heavy flange section and into the bonnet ofthe t

valve is indicative ofa high strain rate.

WNP-2 Firemain Flooding ofECCS Rooms Root Cause Analysis J. E. Peterson, Fire Protection Engineer Failure analysis ofFP-V-29D Page.3 of3 The gray cast iron material has very low toughness and fractures in a brittle manner with very little ductilitygiven the minimal yield strength and the low percent elongation (typically 0.6%).

The gray cast iron is weakest in the tensile direction and displays greater strength in compression and torsion. This may further explain the primarily tensile fractures observed.

When highly stressed, crack initiation and subsequent propagation is rapid. The graphite flakes in the material aid in crack propagation and provide little resistance for crack arrest. The general corrosion ofthe cast iron body was within the design allowance and did not contribute to the failure ofthe valve.

Ifthere are any comments or questions concerning this report please contact John LaSalle at extension 4613.

38