IR 05000286/2015010
| ML15204A499 | |
| Person / Time | |
|---|---|
| Site: | Indian Point  |
| Issue date: | 07/23/2015 |
| From: | Arthur Burritt Reactor Projects Branch 2 |
| To: | Coyle L Entergy Nuclear Operations |
| Burritt A | |
| References | |
| IR 2015010 | |
| Download: ML15204A499 (28) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION uly 23, 2015
SUBJECT:
INDIAN POINT NUCLEAR GENERATING - SPECIAL INSPECTION REPORT 05000286/2015010
Dear Mr. Coyle:
On June 24, 2015, the U.S. Nuclear Regulatory Commission (NRC) completed a Special Inspection Team (SIT) review of the circumstances surrounding the May 9, 2015, water accumulation in the Unit 3 safety-related switchgear room following the 31 main transformer failure and reactor trip event at your Indian Point Nuclear Generating (Indian Point) Unit 3. This event satisfied the criteria in NRC Inspection Manual Chapter 0309, Reactive Inspection Decision Basis for Reactors, for conducting a special inspection. As a result, the NRC sent a SIT to your site on May 19, 2015. The SIT Charter (Attachment 1 of the enclosed report)
provides the basis and additional details concerning the scope of the inspection. The enclosed report documents the inspection teams activities and observations conducted in accordance with the SIT Charter. On June 24, 2015, the SIT discussed the results of the inspection with you and other members of your staff.
The inspection examined activities conducted under your license as they relate to safety and compliance with Commission rules and regulations and with conditions of your license. The team reviewed selected procedures and records and interviewed personnel. In particular, the SIT reviewed technical analyses, hydraulic flow models, and causal investigations to assess the significance and potential consequences of the water accumulation in the Unit 3 safety-related switchgear room.
This report documents one NRC-identified finding of very low safety significance (Green). This finding involved a violation of NRC requirements. However, because of the very low safety significance, and because it has been entered into your corrective action program, the NRC is treating this finding as a non-cited violation (NCV), consistent with Section 2.3.2.a of the NRC Enforcement Policy. If you contest the NCV in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region I; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Senior Resident Inspector at Indian Point. In addition, if you disagree with the cross-cutting aspect assigned to the NCV in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region I, and the NRC Senior Resident Inspector at Indian Point.
In accordance with Title 10 of the Code of Federal Regulations (10 CFR) 2.390 of the NRC's
"Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of the NRCs Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Arthur L. Burritt, Chief Reactor Projects Branch 2 Division of Reactor Projects Docket No. 50-286 License No. DPR-64
Enclosure:
Inspection Report 05000286/2015010 w/Attachments 1, 2, and 3
REGION I==
Docket No. 50-286 License No. DPR-64 Report No. 05000286/2015010 Licensee: Entergy Nuclear Operations, Inc. (Entergy)
Facility: Indian Point Nuclear Generating Unit 3 Location: 450 Broadway, GSB Buchanan, NY 10511-0249 Dates: May 19, 2015 through June 24, 2015 Inspectors: T. Setzer, Senior Project Engineer, Division of Reactor Projects (DRP),
Team Leader R. Fuhrmeister, Senior Reactor Inspector, Division of Reactor Safety (DRS)
S. Rich, Resident Inspector, DRP W. Schmidt, Senior Reactor Analyst, DRS Approved by: Arthur L. Burritt, Chief Reactor Projects Branch 2 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
Inspection Report 05000286/2015010; 05/19/2015 - 06/24/2015; Indian Point Nuclear
Generating (Indian Point) Unit 3; Special Inspection Team (SIT) review of the May 9, 2015, water accumulation in the Unit 3 safety-related switchgear room following 31 main transformer failure and reactor trip event; Inspection Procedure 93812, Special Inspection.
A four-person U.S. Nuclear Regulatory Commission (NRC) team, comprised of regional inspectors, a Resident Inspector, and a regional Senior Reactor Analyst (SRA), conducted this Special Inspection. The team identified one non-cited violation (NCV) of very low safety significance (Green). The significance of most findings is indicated by their color (i.e., greater than Green, Green, White, Yellow, Red) and determined using Inspection Manual Chapter (IMC) 0609, Significance Determination Process, dated April 29, 2015. Cross-cutting aspects are determined using IMC 0310, Aspects Within the Cross-Cutting Areas, dated December 4, 2014. All violations of NRC requirements are dispositioned in accordance with the NRCs Enforcement Policy, dated February 4, 2015. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 5, dated February 2014.
Cornerstone: Mitigating Systems
- Green.
The inspectors identified a Green NCV of Condition 2.H of the Indian Point Unit 3 Facility Operating License DPR-64, Fire Protection Program, for failure to promptly identify, report, and correct a condition adverse to fire protection. Specifically, solenoid valve (SOV)-230-1, associated with the deluge valve for the 32 main transformer (MT), was documented to have opened during its 2-year deluge activation tests on April 7, 2011, April 2, 2013, and March 24, 2015, but did not close as designed after the deluge system actuated. This condition was not corrected, and recurred on May 9, 2015, when the deluge system actuated in response to a fire on the 31 MT.
Entergy entered this issue into the corrective action program (CAP) (condition report (CR)-IP3-2015-02921), and determined a clogged orifice in the SOV pressure switch prevented the SOV from de-energizing and going closed.
The performance deficiency was determined to be more than minor because it is associated with the Protection Against External Factors attribute of the Mitigating Systems cornerstone and adversely affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, water intrusion into the switchgear room can challenge the reliability of the safety-related electrical equipment required to respond to a reactor transient. The inspectors screened the finding for significance using the screening questions in IMC 0609, Appendix A, Exhibit 2, Mitigating Systems, and Exhibit 4, External Events, and determined that this performance deficiency required a Detailed Risk Evaluation because the potential existed for enough water leakage into the switchgear room to cause a loss of all safety-related power and station blackout (SBO) condition. The Detailed Risk Evaluation determined that this finding was of very low safety significance (Green) with an estimated increase in core damage frequency in the low E-7 per reactor year range (an increase of 1 in 10 million reactor years). The inspectors determined the finding had a cross-cutting aspect in the Human Performance cross-cutting area, Challenge the Unknown, because Entergy did not stop and fully explore an uncertain condition with SOV-230-1 when it failed to closed on three occasions since April 2011. Entergy replaced the SOV, but did not determine that the cause was a clogged pressure switch orifice until after the May 9, 2015, 31 MT fire event. [H.11] (Section 2.c)
REPORT DETAILS
1. Summary and Chronology of Event
In accordance with the SIT Charter (Attachment 1), the team conducted a detailed review of the events regarding water accumulation found in the Unit 3 safety-related 480V switchgear room following a 31 MT failure and reactor trip event on May 9, 2015.
The team gathered information from operator narrative logs, the plant process computer, sequence of events printouts, and interviews with plant operators and engineering staff to develop a detailed timeline of the event (Attachment 2). The following summary highlights the events and challenges encountered by the Indian Point staff.
On May 9, 2015, at 5:50 p.m., Indian Point Unit 3 experienced a main turbine-generator lockout, main turbine trip, and automatic reactor trip as a result of an explosion and fire on the 31 MT. The reactor trip was uncomplicated. All control rods inserted into the reactor core and all safety systems responded as designed.
The heat from the transformer fire caused activation of the 31 MT, 32 MT, unit auxiliary transformer (UAT), and curtain wall fire protection water deluge systems. At 5:52 p.m.,
the site fire brigade was activated and notified the Control Room that a fire was present on the 31 MT. At 5:53 p.m., Verplanck and Montrose Fire Departments were called for assistance, and arrived at the sites front gate approximately 9 minutes later. At 6:01 p.m., Entergy declared a Notification of Unusual Event for an explosion within the stations Protected Area.
At 6:12 p.m., with the majority of the fire out, the fire brigade leader ordered securing the deluge system in order to apply foam to the transformer in accordance with station pre-fire plans and oil fire-fighting strategies. Also at this time, water had been reported on the floor in the Unit 3 480V switchgear room. The Unit 3 480V switchgear room houses breakers that power safety-related equipment. Operators attempted to isolate the deluge system; however, the yoke on an isolation valve was broken during operation, which caused the system to leak-by. An alternate valve was used to isolate deluge water to all four deluge systems.
The fire was initially extinguished at 6:15 p.m.; however, it reflashed at 6:37 p.m., at which time Entergy brought Verplanck fire engines into the Protected Area. Additionally, a request for a tower ladder truck was made to Montrose Fire Department, which arrived at approximately 7:00 p.m. The fire brigade continued to fight the fire with foam and declared the fire extinguished at 8:05 p.m.
The water that accumulated on the floor of the Unit 3 480V switchgear room was initially reported to be approximately 1 to 2 inches in depth. The source of this water was determined to be deluge system water that had come from a deluge valve room, which is adjacent to the 480V switchgear room. The deluge valve room houses four deluge system valves; specifically, 31 MT, 32 MT, UAT, and station auxiliary transformer deluge valves. The deluge valve associated with the curtain wall system is located in the turbine building. In order for a deluge valve to open, an SOV associated with each deluge valve opens and bleeds water pressure off a diaphragm assembly. This water is bled to the floor towards a drain. Entergy determined that the water bled from the 31 MT, 32 MT, and UAT SOVs was unable to completely drain through a floor drain due to its limited capacity, and consequently made its way into the 480V switchgear room where it pooled to approximately 1-inch in depth. Operators reported that all the water drained completely within 30 minutes after the deluge system had been isolated.
2. Equipment Response to Fire Resulting in Water in Switchgear Room
a. Inspection Scope
During on-site inspections on May 19-22 and June 3-4, the inspectors evaluated the equipment response to the fire that resulted in water accumulation in the switchgear room. Specifically, the inspectors reviewed Entergy drawings, calculations, procedures, and analyses that describe the operation and design of the station deluge valves and their associated subcomponents. The inspectors reviewed Entergys hydraulic analyses of flows through the transformer yard storm drain system and Unit 3 switchgear room floor drain system, and performed independent confirmatory calculations. The inspectors conducted plant walkdowns of the systems and components to verify their condition and configuration in accordance with plant design basis documents. The inspectors reviewed preventive maintenance records, work orders, and CAP documents (CRs) for the deluge, floor drain, and storm drain systems to understand the history of the system and its proper operation. The inspectors viewed videos of boroscopic inspections of the floor and storm drain systems to verify that conditions aligned with those described in the analyses. The documents reviewed during the inspection are listed in Attachment 3.
b. Assessment and Observations
.1 Fire Protection Deluge Valves
On May 9, 2015, at 5:50 p.m., Indian Point Unit 3 control room received indications and alarms associated with an automatic turbine and reactor shutdown. Following the shutdown, the control room received alarms on the Fire Display Control Panel, alerting operators that the main transformer deluge system had activated due to a fire on the 31 MT. Based on the design of the deluge system for the transformer yard, only one deluge system actuation is anticipated. However, the heat generated from the fire also caused activation of the adjacent deluge systems on the 32 MT and UAT. Additionally, the curtain wall deluge system, which is a fire protection system designed to protect the turbine building from a fire in close proximity, activated and provided sprays along the turbine building outer wall. The deluge systems are activated by one or more heat detectors that are installed on each associated transformer, with a valid turbine trip signal. The inspectors reviewed the system response and determined that the 31 MT, 32 MT, UAT deluge valves, and curtain wall deluge system activated as designed to provide water to the sprays for each transformer and turbine building wall.
.2 Fire Protection Deluge Valve Solenoid Valve and Pressure Switch
The deluge valves and associated SOVs are located in a room adjacent to the Unit 3 switchgear room on 15-0 elevation. The switchgear room and deluge valve room are separated by a single door. When both a valid turbine trip signal is present and one or more 31 MT, 32 MT, or UAT heat detectors are activated, a SOV for the respective deluge valve is energized to open. By opening, the SOV ports water to the floor towards a drain, which causes water pressure within a diaphragm assembly to bleed off. This results in the retraction of a hold down pin, and system pressure is then able to force open a clapper and allow water to flow through piping and out the spray heads around each affected transformer. Once water pressure in the spray header reaches 6 pounds per square inch gauge (psig), a pressure switch de-energizes the SOV and closes it, which stops water from being continuously ported to the floor.
The SOVs associated with the 31 MT, 32 MT, and UAT deluge valves opened as designed to allow deluge initiation. However, they did not close once system pressure reached 6 psig, and continued to port water to the floor. Water from the SOVs flowed to the floor at approximately 50 gallons per minute (gpm). This water began to flood the deluge valve room to a height of 4 to 6 inches, and flowed underneath the door to the switchgear room because the floor drain in the room could drain only 6 gpm. At 6:12 p.m., operators reported observing water in switchgear room. The operators also reported observing water flowing out of hub drains near the instrument air compressors within the switchgear room. At this time, the fire brigade leader ordered the securing of the deluge system.
At 6:16 p.m., operators attempted to isolate the deluge valves by closing each of the isolation valves for the 31 MT, 32 MT, and UAT, but were unsuccessful when a yoke was broken on one of the valves (FP-2-2) during operation, causing the system to leak-by. The main system isolation, FP-75, was closed to aid in the isolation. At 6:20 p.m.,
water to the deluge system had been completely isolated. The estimated water level in the switchgear room was reported to be 1 to 2 inches, varying at different locations in the room. Operators reported that all the water drained completely within 30 minutes after the deluge system had been isolated.
The inspectors performed a review of historical CAP documents and found three instances where plant operators had identified a SOV failure to close during its 2-year deluge system surveillance tests. Specifically, in April 2011, operators identified the SOV associated with 32 MT (SOV-230-1) deluge valve was failing to close (CR-IP3-2011-02177). Additionally, the pressure switch was noted to have severely corroded internals. Entergy replaced SOV-230-1, but did not troubleshoot the pressure switch. In April 2013, SOV-230-1 failed to close during its 2-year surveillance test (CR-IP3-2013-02214), but corrective action was not taken to investigate the issue. In March 2015, SOV-230-1 failed to close for a third time (CR-IP3-2015-02133). Work orders were created to investigate the issue but were not completed before the event on May 9, when the SOVs for 31 MT, 32 MT, and UAT failed to close and caused water to accumulate in the switchgear room. The 31 MT and UAT deluge valve SOVs had no CRs which documented a failure to close during testing. However, the inspectors concluded that adverse conditions with these SOVs should have been discovered by Entergy had an extent of condition review been completed.
Following the May 9 event, Entergy reviewed the operation of the SOV and its failure to close at the appropriate system pressure and determined that orifices for the pressure switches were clogged with system debris, which prevented them from sensing system pressure and closing the SOVs. A Green NCV of Condition 2.H of the Indian Point Unit 3 Facility Operating License DPR-64, Fire Protection Program, is discussed below in Section 2.c because Entergy did not promptly identify, report, and correct a condition adverse to fire protection associated with SOV-230-1 failure to close.
.3 Deluge Valve Room and Switchgear Room Floor Drains
The deluge valve room and switchgear room have both hub and floor drains that are connected underground by 4-inch cast iron piping. During the event, operators observed water flowing out of hub drains in the switchgear room located near the instrument air compressors. Entergy performed a boroscopic inspection of the floor drains and found restrictions and standing water downstream of the instrument air compressor hub drains (CR-IP3-2015-03025). Entergy determined the water that flowed from the deluge valve room to the switchgear room was unable to completely drain due to these restrictions, and consequently flowed out through the path of least resistance, which was the hub drains nearest the instrument air compressors. Entergy performed flow tests and determined that the floor drains, as restricted, could pass a maximum of 25 gpm. After testing, the drains were cleaned (CR-IP3-2015-03121).
The inspectors reviewed the preventive maintenance program for the floor drains and determined that the testing methodology to determine the capability of the drains did not ensure they were clear of restrictions. Entergys test of the floor drains included pouring 10 gallons of water down the floor drain. If the water drained within one minute, the drain was considered satisfactory. Given the restrictions found during the boroscopic inspection, and the volume of piping in the drain system that could absorb 10 gallons of water while being restricted downstream, the inspectors concluded this approach may not consistently reveal whether the floor drains were clear of restrictions. Entergy entered an item into their CAP to enhance the preventive maintenance practices on the floor drain system for both Unit 2 and Unit 3 (Action Request 228018). Additionally, as an interim corrective action, Entergy installed a temporary modification and procedure change to direct operators to open doors between the switchgear room and the emergency diesel generator (EDG) room, so that water would be directed to the EDG rooms large capacity sump in the event of a water intrusion event.
.4 Transformer Yard Storm Drains
The storm drain piping system in the Unit 3 transformer yard is a series of eight manholes (MHs) that are connected by underground corrugated metal pipe. The MHs begin in the transformer yard at 18-0 elevation, and drain to a single outlet into the discharge canal. MH B3 receives normal water runoff from the transformer yard, and is also connected to the outlet of the Unit 3 switchgear room floor drains located at 15-0 elevation. The inspectors questioned whether water observed in the switchgear room during the event had originated from the storm drain system, since there is nothing to prevent a back-up of water from MH B3 to the switchgear room other than a difference in elevation (3-0).
Entergy performed a hydraulic analysis (PIPE-FLO) of the storm drain system to determine if water from the storm drain system could have been the source of water observed in the switchgear room. Additionally, Entergy retained an engineering firm to perform a separate analysis. The three deluge systems that activated during the event had a combined flow of 4300 gpm. The 4300 gpm deluge flow eventually overwhelmed the transformer yard moat, and resulted in water accumulation in the transformer yard.
Based on field observations from Entergy fire brigade members, this water accumulation may have been as high as 3 inches above the transformer yard ground level in some places.
In a bounding analysis, Entergy assumed that the entire volume of deluge water (4300 gpm) would flow into MH B3, which is the closest manhole connected to the switchgear room floor drain piping. This is considered a conservative assumption, since the maximum drainage capability through the MH B3 grate with 3 inches of available water height over the entire drain is approximately 1500 gpm. Additionally, during the event, a portion of the water flowed downhill and away from MH B3 and entered the storm drain system through manholes located at a lower hydraulic grade. Based on the maximum drainage capability exiting MH B3 and an assumed influx of 4300 gpm, water within MH B3 would rise to a maximum, equilibrium level of 10-4 elevation. Since this level is below the level of the switchgear room (15-0), Entergy concluded that the source of the water observed in the switchgear room could not have originated from transformer yard storm drain system.
To further support their conclusion, a model of the drainage system utilizing PIPE-FLO was created to determine what the maximum flow rate into MH B3 must be to cause an influx of water from the yard storm drains to the switchgear room. The result of the PIPE-FLO analysis indicated that a flow rate of approximately 6200 gpm would be needed to flow into MH B3 to cause water to backup into the switchgear room. This exceeds both the maximum drainage capability of the MH B3 grate and the maximum possible flowrate the fire protection system could deliver.
The inspectors reviewed the Entergy PIPE-FLO and supporting engineering firms analysis and determined it was reasonable with supported conclusions. However, the inspectors determined that the analysis for the maximum flow rate into MH B3 did not consider damage and a restriction in the storm drain system that was discovered by Entergy during a robotic inspection (CR-IP3-2015-03090). This restriction produced a 36 percent resistance to the flow out of the storm drain system, and would potentially result in lower flowrates needed to cause a backup of water into the switchgear room.
Entergy performed the analysis applying a 36 percent resistance and determined that the maximum flow rate into MH B3 decreased to 5600 gpm. This did not change the conclusion that water from the transformer yard storm drains did not flow into the switchgear room. Entergy further supported this conclusion based on the observation that the water flowing out of the hub drain next to the 31 instrument air compressor was clear with no oil sheen. Additionally, no water was seen flowing from the floor drains at even lower elevations than the switchgear room (5-0 elevation of the turbine building).
Considering the damage identified in the storm drain system, and the risk of a potential storm drain collapse, the inspectors questioned whether rain water could overcome the storm drain system and cause water to potentially flow to the switchgear room. Entergy entered the condition in the CAP (CR-IP3-2015-03328) to request vendor support in evaluating costs and repairs to the restricted or damaged portions of the storm drain piping. Additionally, Entergy instituted a Special Log to check MHs for unusually high water levels, and hourly firewatch tours in the switchgear room to determine if water was flowing out of the floor drains. Entergy concluded that the degraded conditions in the storm drain system did not significantly alter its capability. The inspectors determined that Entergy had an adequate basis for this conclusion.
c. Findings
Introduction.
The inspectors identified a Green NCV of Condition 2.H of the Indian Point Unit 3 Facility Operating License DPR-64, Fire Protection Program, for failure to promptly identify, report, and correct a condition adverse to fire protection. Specifically, SOV-230-1, associated with the deluge valve for the 32 MT, was documented to have opened during its 2-year deluge activation tests on April 7, 2011, April 2, 2013, and March 24, 2015, but did not close as designed after the deluge system actuated. This condition was not corrected, and recurred on May 9, 2015, when the deluge system actuated in response to a fire on the 31 MT. Entergy entered this issue into the CAP (CR-IP3-2015-02921), and determined a clogged orifice in the SOV pressure switch prevented the SOV from de-energizing and going closed.
Description.
On May 9, 2015, at 5:50 p.m., a fire on the 31 MT actuated the fire suppression deluge valves associated with the 31 MT, 32 MT, UAT, and curtain wall system. In order for a deluge valve to open, a SOV associated with each deluge valve opens and bleeds water pressure off a diaphragm assembly. This water is bled to the deluge valve room floor towards a drain. As designed, SOVs FP-SOV-230-1, FP-SOV-231-1, and FP-SOV-237-1 opened on a signal from the fire detection system. The SOVs ported water from each deluge valve diaphragm chamber onto the floor at approximately 50 gpm. The SOVs are designed to de-energize and close after a pressure switch senses 6 psig in the spray header. However, a buildup of corrosion products in the pressure switch orifices prevented the pressure switch from closing the SOVs. As a result, all three SOVs remained open and continued to discharge water to the floor at a rate of 50 gpm for approximately 30 minutes.
The inspectors performed a review of historical CAP documents and found three instances where plant operators had identified SOV-230-1 failure to close during its 2-year deluge system surveillance tests. Specifically, in April 2011, operators identified SOV-230-1 as failing to close (CR-IP3-2011-02177). Additionally, the pressure switch was noted to have severely corroded internals. Entergy replaced SOV-230-1, but did not troubleshoot the pressure switch. In April 2013, SOV-230-1 failed to close during its 2-year surveillance test (CR-IP3-2013-02214), but the issue was not investigated nor were corrective actions taken. In March 2015, SOV-230-1 failed to close for a third time (CR-IP3-2015-02133). Work orders were created to investigate the issue but were not completed before the event on May 9, when the SOV-230-1 failed to close and contributed to the water accumulation in the switchgear room. The 31 MT and UAT deluge valve SOVs had no CRs which documented a failure to close during testing.
However, the inspectors concluded that adverse conditions with these SOVs should have been discovered by Entergy had an extent of condition review been completed.
Analysis.
The inspectors determined that the failure to promptly identify, report, and correct a condition adverse to fire protection associated with SOV-230-1 failure to close on April 7, 2011, April 2, 2013, and March 24, 2015, was a performance deficiency that was reasonably within Entergys ability to foresee and correct, and should have been prevented. The performance deficiency was determined to be more than minor, because it is associated with the Protection Against External Factors attribute of the Mitigating Systems cornerstone and adversely affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, water intrusion into the switchgear room can challenge the reliability of the safety-related electrical equipment required to respond to a reactor transient.
The inspectors screened the finding for significance using the screening questions in IMC 0609, Appendix A, Exhibit 2, Mitigating Systems, and Exhibit 4, External Events, and determined that this performance deficiency required a Detailed Risk Evaluation because the potential existed for enough water leakage into the switchgear room to cause a loss of all safety-related power and SBO condition.
The actual water accumulation in the switchgear room was not measured. Entergy performed as-found testing and determined the combined leakage from the three deluge valve SOVs associated with 31 MT, 32 MT, and UAT was approximately 12 gpm. This flowrate would not have resulted in a sufficient volume of water in the room to challenge the critical height (flooding electrical switchgear cabinets) and therefore, did not result in any increase in core damage frequency. Absent a measured leakage rate, the Region I SRA conducted a bounding Detailed Risk Evaluation using conservative assumptions, to approximate the potential increase in risk associated with this performance deficiency.
The SRA developed a condition specific event tree and used the following assumptions in the Detailed Risk Evaluation:
If the deluge valve SOV leakage was not isolated, the switchgear room would fill to the critical height of 4.875 inches in six hours. This was based on the licensee determined maximum, as found, drainage system flowrate of 25 gpm and the approximate 30 minutes elapsed time for water to drain from the room after the deluge SOVs were completely isolated. The SRA used a 50 gpm maximum flowrate into the room for the bounding analysis, representing a flowrate approximately four times the 12 gpm licensee estimate (See Section 3.b below for the basis of the assumed 50 gpm flowrate).
An exposure period of one year.
A fire frequency of 6.6E-3 per reactor year, was taken from NUREG-2169, representing switchyard large transformer catastrophic failure.
Not all large transformer catastrophic failures result in three deluge valves actuating. A 50/50 split was assumed.
Successful deluge SOV isolated or other action taken to increase the drain rate from the room (i.e., open doors to EDG room or use portable pumps) within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, results in no increase in core damage probability.
Flooding of the switchgear room results in a SBO with subsequent emergency operating procedure entry and operators taking manual control of the turbine-driven auxiliary feedwater (TDAFW) pump and use the alternate safe shutdown methodology powered via MCC312A.
Maintaining Reactor Coolant System Inventory:
One charging pump has a capacity of 100 gpm. In the event of a reactor coolant pump (RCP) seal leakage exceeding 25 gpm per pump (>100 gpm total), one pump would not be sufficient.
Standard split fraction for Westinghouse high temperature RCP seal failures and leakage rates were used. Operators failing to trip RCPs on a complete loss of seal cooling following the SBO would result in a 480 gpm leak per pump. Assuming a complete loss of seal cooling and successful tripping of the RCPs, 0.8 chance of 21 gpm per pump and 0.2 chance of greater than 21 gpm per pump was used.
Maintaining Decay Heat Removal: manual local operation of the TDAFW pump is necessary to maintain steam generator inventory.
The SRA used Standardized Plant Analysis Risk (SPAR) H to develop the following conservative Operator Action human failure probabilities:
Isolate the deluge valve and the SOV was contained in the fire response procedure and was specific on operation of these two valves for each deluge system. Human Reliability Analysis (HRA)-OPS-ISO-SOV - 6E-3 was estimated assuming a procedurally driven action with high stress and low experience and training.
Secure the fire system, open the doors to the EDG room or use portable pumping equipment within the 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> recovery time. HRA-OPS-ISO-MIT-FLD - 1E-2 with expansive time, high stress, low experience and training, and poor procedures.
Manual operation of the TDAFW pump (HRA-OPS-TDAFW-MAN)assigned a 1E-2 failure probability (conservatively high given the lower decay heat rate after 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />).
Trip the RCPs, if a SBO occurs (loss of Component Cooling Water and charging). HRA-RCP-TRIP was given the baseline SPAR of 5E-4.
Align and operate alternate safe shutdown: 0.2, given the value from NUREG-1921 for outside the control room remote shutdown actions.
In core damage sequences where the combined HRA valve was less than 1E-5, the 1E-5 limit, in accordance with the Risk Assessment Standardization Project manual, was used.
The Detailed Risk Evaluation determined that this performance deficiency was of very low safety significance (Green) with an estimated increase in core damage frequency in the low E-7 per reactor year range (an increase of 1 in 10 million reactor years). The Detailed Risk Evaluation dominant core damage sequence for this condition involves:
operators failing to isolate the SOV leakage in accordance with the fire protection procedures, the loss of the 480V switchgear in approximately six hours and the resultant SBO, successful operator actions to control the TDAFW pump, tripping of RCPs on the loss of all seal cooling, and the subsequent RCP seal failures greater than 25 gpm each.
The inspectors determined the finding had a cross-cutting aspect in the Human Performance cross-cutting area, Challenge the Unknown, because Entergy did not stop and fully explore an uncertain condition with SOV-230-1 when it failed to close on three occasions since April 2011. Entergy replaced the SOV, but did not determine that the cause was a clogged pressure switch orifice until after the May 9, 2015, 31 MT fire event. [H.11]
Enforcement.
Condition 2.H of Facility Operating License DPR-64 requires Entergy to implement and maintain in effect all provisions of the approved Fire Protection Program as described in the Final Safety Analysis Report for Indian Point Nuclear Generating Unit No. 3, and as approved in the NRC fire protection safety evaluations (SEs) dated September 21, 1973; March 6, 1979; May 2, 1980; November 18, 1982; December 30, 1982; February 2, 1984; April 16, 1984; January 7, 1987; September 9, 1988; October 21, 1991; April 20, 1994; January 5, 1995; and supplements thereto.
The NRC SE dated May 2, 1980, Section 2.5.6, Quality Assurance, Subsection 6.1 states, The design, procurement, installation, testing, and administrative control activities for fire protection will be controlled in accordance with the quality assurance program criteria of Appendix A to BTP 9.5-1.
Appendix A to BTP 9.5-1, Section C.8, Corrective Action, requires measures to be established to assure that conditions adverse to fire protection, such as failures, malfunctions, deficiencies, deviations, defective components, uncontrolled combustible materials, and non-conformances are promptly identified, reported, and corrected.
Contrary to the above, on April 7, 2011, April 2, 2013, and March 24, 2015, Entergy did not implement all provisions of the approved Fire Protection Program; in that, Entergy did not assure that a condition adverse to fire protection was promptly identified, reported, and corrected. Specifically, SOV-230-1 associated with the deluge valve for the 32 MT did not close after energizing to actuate the deluge system. This condition was not corrected, and recurred on May 9, 2015, when the deluge system actuated in response to a fire on the 31 MT. Because the violation was of very low safety significance (Green) and it was entered into the CAP (CR-IP3-2015-02921), this violation is being treated as a NCV, consistent with Section 2.3.2.a of the NRC Enforcement Policy. (NCV 05000286/2015010-01, Failure to Correct a Degraded Condition of Fire Protection System Solenoid Valve SOV-230-1)
3. Operator and Fire Brigade Performance in Identifying and Isolating the Source of Water
into the Switchgear Room
a. Inspection Scope
During on-site inspections on May 19-22, the inspectors reviewed the operator and fire brigade performance in identifying and isolating the source of water into the switchgear room. Specifically, the inspectors reviewed operator logs, written event recollections, procedures, and CRs to assess the actions of licensed operators and fire brigade members taken in response to the identification of water on the floor of the switchgear room. The inspectors performed walkdowns of the switchgear room and deluge valve room both independently and with members of the fire brigade to verify observations and recollections from the event. The inspectors interviewed fire brigade members and the fire brigade leader to understand the actions they took while fighting the fire on 31 MT.
The inspectors interviewed licensed operators to understand their perspective on the event and their actions taken. The inspectors reviewed Entergy hydraulic analyses of water flow into and out of the switchgear room, and interviewed the engineers who performed the calculations to verify the calculations corresponded with the conditions observed in the room. The inspectors reviewed design documents of the safety-related switchgear to verify the height (4.875 inches) at which the water level would adversely affect the electrical components.
b. Assessment and Observations Fire brigade members who traversed the room during the event identified water approximately 1 to 2 inches in depth on the floor of the switchgear room. After the door between the deluge valve room and switchgear room was opened, operators reported that approximately 4 to 6 inches of water had flooded the deluge valve room and flowed out to the switchgear room. Additionally, water was observed to be flowing out of a hub drain near the instrument air compressor, and from the SOVs associated with the 31 MT, 32 MT, and UAT deluge valves. This was reported to the fire brigade leader, who at 6:12 p.m., ordered securing of the deluge system. Operators began isolating each deluge valve by closing the local upstream isolation valve, but were unsuccessful when a yoke on one of the isolation valves (FP-2-2) broke while being operated. Operators then closed the main stop valve to the deluge room (FP-75), which isolated deluge water to all four deluge systems. The water that accumulated in the switchgear room was reported to have drained completely within 30 minutes after the deluge system had been isolated. The inspectors determined that Entergy had appropriately identified the source of water, and performed the proper activities to ensure the water intrusion into the switchgear room was isolated.
The inspectors questioned the actual water level in the room, since reports were not all consistent (varying from 1 to 2 inches). As discussed in Section 2.b.3, Entergy performed flow tests and determined that the floor drains, as restricted, could pass a maximum of 25 gpm. Operators reported the water that accumulated in the switchgear room had completely drained within 30 minutes after the deluge system had been isolated. Assuming the water took 30 minutes to drain, this would have yielded approximately 750 gallons of water in the switchgear room, which based on the room volume would equate to 0.4 inches of water level. Therefore, the inspectors determined that the actual water level in the room could not have substantially exceeded 0.4 inches, and would have been less than the 1 to 2 inches reported. This is further supported by the observation that water did not flood into an area beneath the 32 instrument air compressor, as this area is effectively sealed around its perimeter by steel angle-iron that is approximately 1-inch in height.
Entergy performed a calculation (Engineering Calculation 58061) and determined the flowrate through one SOV, considering resistances and losses, is 25.3 gpm. Since three SOVs remained open during the event, this would produce approximately 75 gpm of water being ported to the floor of the deluge valve room, for 30 minutes. The inspectors questioned whether these flowrates could have been the actual flowrates during the event, since the 75 gpm SOV flow would produce a volume of water in the switchgear room that could not have drained completely in 30 minutes. Specifically, given an input of water of 75 gpm, and subtracting the known drain rate of 25 gpm, a net flowrate of 50 gpm would have flooded the room for 30 minutes and produced approximately 1500 gallons of water in the switchgear room. Given the known drain rate of 25 gpm, this volume could not have drained completely within the 30 minutes reported by the operators.
In response to the inspectors concerns regarding the SOV flowrates, Entergy performed as-found flowrate tests of each SOV, and determined SOV-231-1, SOV-230-1, and SOV-237-1 produced 5 gpm, 3.6 gpm, and 3.7 gpm, respectively. This would equate to a total of approximately 12 gpm of SOV flow during the event. The inspectors determined this could not have been the actual flowrates during the event, since this was within the drainage capability of the floor drain system (25 gpm), and therefore would not have produced significant water accumulation in the switchgear room.
After further evaluation, the inspectors concluded the total combined SOV flowrate during the event would have had to be approximately 50 gpm. Subtracting the drainage capability of 25 gpm, a net flow of 25 gpm would have entered the switchgear room for 30 minutes. This would have produced approximately 750 gallons of water (0.4 inches of level), and would have been able to drain completely within 30 minutes. The safety-related switchgear equipment was not adversely affected by 0.4 inches of water since the electrical components are 4.875 inches off the ground.
The inspectors determined that y-strainers in the piping leading to each SOV had never been cleaned or inspected, and concluded that clogging from debris known to be in the system may have produced a low as-found flowrate (12 gpm). The initial flowrate during the event reduced to 12 gpm as this debris clogged the y-strainer. Entergy entered this issue into the CAP to create a preventive maintenance item to open, inspect, and clean the y-strainers (CR-IP3-2015-02921).
c. Findings
No findings were identified.
4. Effectiveness of Entergy Response to Water Intrusion
a. Inspection Scope
The inspectors reviewed the effectiveness of Entergys response to the water accumulation in the switchgear room. Specifically, the inspectors reviewed corrective actions taken in response to the water intrusion both during and following the event to verify they would effectively address the issue. The inspectors reviewed the design of a temporary modification intended to divert water away from the safety-related switchgear room, and reviewed a standing order issued to site operators to ensure they would properly address the water intrusion event without introducing new safety concerns or challenges. The inspectors reviewed records associated with the repair and post-maintenance testing of the deluge valves. The inspectors reviewed documentation for cleaning of the transformer yard storm drains and switchgear room floor drains.
b. Assessment and Observations In response to the water intrusion, Entergy immediately secured the deluges and took action to remove the water from the room. Following the event, Entergy implemented a temporary modification to provide a pathway for the water to drain from the Unit 3 480V switchgear room. The room is connected to the 31 EDG room, which contains a large sump and drain line to mitigate potential service water pipe breaks. The temporary modification included a berm to direct water from the switchgear room to the sump, pre-staged chocks to hold two fire doors open, and steps added to the plant fire response and alarm response procedures to direct a plant operator to open the doors and institute a firewatch. To address extent of condition, a standing order was issued to Unit 2 that directed a plant operator to respond to the Unit 2 480V switchgear room in the event the transformer deluge systems actuate, who would then monitor for leakage or open the door between the deluge room and the turbine building to allow water to drain.
Entergy also took action to inspect the floor drain lines for blockages. Several blockages were identified downstream of the floor drains. Entergy performed troubleshooting activities and determined that the drain next to the deluge valves could drain a steady-state flow rate of only 6 gpm. As stated earlier, further testing showed that the entire floor drain system for the room could drain a maximum of 25 gpm. Entergy cleaned the floor drains and the post-maintenance testing verified the floor drains could drain approximately 100 gpm.
Through troubleshooting, Entergy identified that the pressure switches associated with each SOV were unable to sense downstream pressure accurately due to plugged orifices. Entergy cleaned the orifices, replaced one pressure switch, and performed post-maintenance testing to verify that the system would close the solenoid valves as designed. The inspectors concluded the actions taken by Entergy in response to the water intrusion were reasonable to prevent the event from recurring.
c. Findings
No findings were identified.
5. Evaluation of Pertinent Industry Operating Experience
a. Inspection Scope
The inspectors reviewed Entergys evaluation of pertinent operating experience.
Specifically, inspectors searched Entergys CAP database for previous similar events at Indian Point to determine whether those events should have driven Entergy to reduce the likelihood of the May 9 water accumulation event before it occurred. The inspectors also reviewed actions taken by the site in response to an industry operating experience event that involved a switchgear room flooding event.
b. Assessment and Observations In 1996, New York Power Authority, then-owner of Indian Point Unit 3, evaluated an industry operating experience event, Significant Event Report (SER) 95-8, where unexpected service water leakage flooded a switchgear room and required the plant to de-energize electrical busses in the room. As part of their evaluation, New York Power Authority identified that the sources of water in the Unit 3 480V switchgear room included the fire protection piping and two service water relief valves. They calculated that the expected flow rate from the service water relief valves was 58 gpm. They did not quantify the postulated leak rate from the fire protection piping in their evaluation, but concluded that it was bounded by the expected drain rate of the floor drains, which was calculated to be approximately 100 gpm. Entergys periodic maintenance program included testing the 480V switchgear room floor drains every 2 years to verify that they could drain 10 gallons of water in less than one minute, which was a substantially lower flowrate than that assumed in the operating experience analysis. The inspectors determined that had the floor drain system been rigorously tested and maintained clean, the May 9 event would not have resulted in as much water accumulation on the floor.
The inspectors also reviewed Entergys corrective actions taken in response to water accumulation in the Unit 2 switchgear room during a severe weather event in August 2011 (see NRC Inspection Report 05000247/2012002). As a corrective action, Entergy implemented a surveillance program for the Unit 2 switchgear room floor drains that mirrored the existing surveillance program on Unit 3, because at that time Unit 2 had not been performing preventive maintenance on the Unit 2 floor drain system.
c. Findings
No findings were identified.
6. Risk Significance of the Event
Initial Assessment The initial risk assessment for this event is documented in the enclosed SIT Inspection Team Charter (Attachment 1).
Final Assessment While there was no actual reactor core damage consequence of the May 9, 2015, 31 MT fire, the SRA used the model and assumptions developed to evaluate the risk significance of the performance deficiency discussed above in Section 2.c, to estimate the chance that such a fire could have progressed to core damage. This bounding conditional core damage probability estimate was in the range of low E-5 per event (1 in 100,000 such events).
7. Exit Meetings
On June 24, 2015, the team presented their overall assessment and observations to Mr. Larry Coyle, Site Vice President, and other members of his staff. The inspectors confirmed that any proprietary information reviewed during the inspection was returned to Entergy.
ATTACHMENT 1 - SPECIAL INSPECTION TEAM CHARTER ATTACHMENT 2 - DETAILED SEQUENCE OF EVENTS ATTACHMENT 3 -
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
Contacted
- S. Radomski, Fire Brigade Leader
- R. Burroni, Director, Engineering
- M. Kempski, Manager, Engineering
- J. Bridges, Fire Brigade Member
- G. Grimes, Fire Brigade Member
- A. Zastrow, Shift Manager
- J. Johnson, Fire Brigade Leader
- J. Pineda, Design Engineering Supervisor
- T. Chan, Design Engineering Supervisor
- K. Elliot, Fire Protection engineer
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened/Closed
- 05000286/2015010-01 NCV Failure to Correct a Degraded Condition of Fire Protection System Solenoid Valve SOV-230-1 (Section 2.c).