05000529/LER-2010-002
| Docket Numbersequential Revmonth Day Year Year Month Day Yearnumber No. | |
| Event date: | 09-15-2010 |
|---|---|
| Report date: | 01-17-2012 |
| 5292010002R01 - NRC Website | |
All times are Mountain Standard Time and approximate unless otherwise indicated.
1. REPORTING REQUIREMENT(S):
This LER is being submitted pursuant to 10 CFR 50.73 (a)(2)(i)(B) as a condition prohibited by Technical Specifications (TS) 3.8.1, "AC Sources — Operating," and 3.8.2, "AC Sources - Shutdown," for an inoperable Emergency Diesel Generator (EDG) (EllS Code: EK) due to a failure of the associated fuel oil transfer pump and 10 CFR 50.73 (a)(2)(v)(A), (B), (C) and (D) for a condition that could have prevented the fulfillment of a safety function of systems needed to shutdown the reactor and maintain it in a safe shutdown condition, remove residual heat, control the release of radioactive material, and mitigate the consequences of an accident.
2. DESCRIPTION OF STRUCTURE(S), SYSTEM(S) AND COMPONENT(S):
Diesel Fuel Oil Transfer System Description (EllS Code: DC) Each of the station's units is provided with two EDGs. Each EDG is served by a Diesel Generator Fuel Oil Transfer System, which consists of one diesel fuel oil (DFO) storage tank and one diesel fuel oil transfer pump (DFOTP), which provides fuel to the EDG fuel oil day tank. The diesel fuel day tank overflow returns to the storage tank.
The transfer pump is located in the diesel fuel oil storage tank and is accessible from a vault built above each diesel fuel oil storage tank. Failure of one pump does not affect the operability of any component in the other train.
The transfer pump can be operated from either the main control room or the local diesel control panel. Alarms and indications of day tank level and transfer pump status are displayed in the main control room and at the local diesel control panel. The DFOTP delivers a minimum of 15 gallons per minute to the EDG day tank. The DFOTP takes suction from the diesel generator fuel oil storage tank and discharges into the diesel generator fuel oil day tank which gravity feeds fuel oil to the EDG. The fuel oil day tank level is automatically controlled by a level control system that cycles the transfer pump as needed to maintain level.
Power for the transfer pumps is fed from Class 1 E 480 volt AC (VAC) motor control centers. In addition to the power cable, a 120 VAC control cable is used for the pump motor thermostat which trips the motor when winding temperature reaches 425°F. The cables providing power and control to the motors are manufactured with polyethylene insulation and a neoprene jacket. The cables are routed from the motor control center through a raceway and outside the diesel generator building through a sand encased duct bank. The cables are then routed through sealed conduits to the diesel fuel oil storage tank vault and terminate in a cast aluminum motor termination box located above the tank. This motor termination box is of a leak proof design to prevent diesel fuel oil from seeping into the conduit in the event of a diesel fuel transfer pump motor stator liner rupture.
3. INITIAL PLANT CONDITIONS:
On September 15, 2010, Palo Verde Unit 2 was in Mode 1 (Power Operation), at 100 percent power, at normal operating temperature (NOT) and normal operating pressure (NOP). There were no structures, systems, or components inoperable at the time of the event that contributed to the event.
4. EVENT DESCRIPTION:
On September 15, 2010, at 03:54, Unit 2 Operations personnel started EDG-2A to perform surveillance test 40ST-9DG01, "Diesel Generator A Test." This procedure ensures operability of the train A Emergency Diesel Generator. At 05:46, control room personnel received a diesel generator A fuel system trouble alarm. Investigation revealed the supply breaker for the DFOTP was in the trip free position. Operations personnel declared EDG 2A inoperable and entered TS Limiting Condition of Operation (LCO) 3.8.1, Condition B.
The equipment was quarantined and actions to troubleshoot the failure commenced.
The troubleshooting efforts determined a direct short to ground existed on a power connection in the motor termination box located in the fuel oil storage vault. Additionally, an inspection of the cable pull box above the motor termination box revealed a degraded condition on the outer jacket of the 120 VAC control cable with water migrating through a tear in the cable's outer jacket. The conduit from the pull box to the motor termination box was found to contain water. This conduit had been sealed at the pull box in 2004 and the conduit from the pull box to the diesel generator building had been sealed in 2005 to prevent water intrusion. The water that caused the short resulted from water accumulation in the underground trench which is part of the sand encased duct bank which then migrated into the cable conduit and into the control cable jacket which drained directly into the motor termination box.
The degraded control and power cables were replaced between the pull box and the motor termination box and the pump was retested satisfactorily. The control and power cables were spliced to interrupt the drain path and prevent possible future water drainage into the Operations personnel declared EDG-2A operable on September 18, 2010, at 15:32.
Following this event, corrective maintenance work orders were issued for the other five DFOTPs to inspect the associated pull boxes, motor termination boxes, and control and power cables for signs of moisture intrusion and possible damage. The work orders also replaced the NRC FORM 366A (10-20107) PRINTED ON RECYCLED PAPER control and power cables between the pull boxes and motor termination boxes of the five DFOTPs. Included in the replacement was installation of splices as described for DFOTP-2A above.
Prior to their replacement, the control cables and motor termination boxes of the five other station EDGs were examined. The four sets in Units 1 and 3 did not exhibit corrosion, visible water, or abnormal megger test results. The fifth control cable, DFOTP-2B, exhibited minor corrosion and a lower than normal megger value, 5.1 mega-ohms, but no visible water. Further electrical testing identified some moisture was present in the cable jacket. Nonetheless, these results were determined to be acceptable, and supported no interruption of functionality of DGFOTP-2B. All five cables were replaced and spliced as discussed above.
These work orders also ensured that the conduits between the pull boxes in the fuel oil storage tank vaults and the DFOTP motor termination boxes were inspected for water and dried out if water was found. Of the conduits for the six DFOTPs inspected, including DFOTP-2A, only the conduit for DFOTP-2A contained water, which was drained and dried.
5. ASSESSMENT OF SAFETY CONSEQUENCES:
On August 18, 2010, the DGFOTP-2A successfully demonstrated that it was capable of supporting EDG-2A operability when EDG-2A ran for 6.2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> to perform 40ST-9DG01, "Diesel Generator A Test." This run time and load caused the fuel oil transfer pump to cycle to maintain day tank level. There were no identified deficiencies and no performance issues identified during this surveillance run.
On September 15, 2010, at 05:46, during the next performance of 40ST-9DG01, DFOTP 2A failed. Following the failure of the transfer pump, the remaining fuel oil inventory in the diesel fuel day tank was sufficient to meet 176 minutes of run time based on actual load requirements as demonstrated from a previous loss of offsite power event.
The investigation team was not able to determine when EDG-2A could not meet its mission time of seven days. However, it is assumed EDG-2A was inoperable for a period greater than allowed by TS LCOs 3.8.1 and 3.8.2. During periods when EDG-2B was unavailable for maintenance and testing, the degradation in the EDG-2A motor termination box could have prevented fulfillment of the emergency AC power supply function. Therefore, this event represents a reportable condition that could have prevented the fulfillment of a safety function in accordance with 10 CFR 50.73 (a)(2)(v).
Prior to the event, on September 1, 2010, the 2B DFOTP demonstrated that it was capable of supporting EDG-2B operability when EDG-2B successfully ran during performance of 40ST-9DG02, "Diesel Generator B Test." EDG-2B was also successfully operated on September 16, following the discovery of the EDG 2A inoperability. The extent of condition inspections on September 20, for EDG 2B identified corrosion in the motor termination box but there was no water or moisture present. Electrical testing of the EDG 2B control and power cables identified normal megger readings for the control cable and lower than expected results for the power cable. The power cable megger reading was 5.1 mega ohm, which is less than the normal acceptance criteria of 50 mega-ohms, but greater than the emergency use criteria of 1 mega-ohm as defined in Palo Verde's cable specification 13-EN-306. Therefore, EDG 2B was capable of fulfilling its design function if required.
Emergency operating procedures at Palo Verde provide operator actions to support the restoration of power to the Class 1 E 4.16 kV Switchgear 2E-PBA-503 if normal power sources were lost. These procedures provide various methods for routing power to the effected switchgear, including a method to route power from the station black out generators during a loss of offsite power event. The risk associated with this event was determined using the current internal events and fire probabilistic risk assessment model.
The incremental conditional core damage probability associated with the event was determined to be 3.2E-7, which is characterized as a nominal risk impact as described in NRC Inspection Manual 0308.
6. CAUSE OF THE EVENT:
The direct cause for failure of the EDG-2A fuel oil transfer pump was determined to be a phase to ground fault at one of the motor termination box connections. The degraded condition on the outer jacket of the 120 VAC control cable allowed water to migrate through a tear in the cable's outer jacket. The water within the control cable's outer jacket drained directly into the motor termination box. The presence of water in the motor termination box for the EDG-2A fuel oil transfer pump caused a phase to ground fault at one of the motor termination box connections and resulted in the failure of the fuel oil transfer pump.
Although the investigation team was not able to determine the root cause for the failure of the neoprene cable jacket; it is likely that an application of an incompatible cleaning solvent to the surface of the cable jacket, or the presence of another contaminant caused a degradation to the physical properties of the neoprene jacket.
7. CORRECTIVE ACTIONS:
Based on the results of an investigation of this event, the following corrective actions have been taken to prevent recurrence:
- The two conductor control cables and three conductor power cables from the pull-box in the DFOTP vault to the motor termination box for all six DFOTPs have been replaced.
- The motor termination boxes for the Unit 2 A and B DFOTPs were replaced.
- The replacement control and power cables from the pull-box to the motor termination box were connected to the cables entering the pull-boxes from the diesel generator building using Raychem splices with no outer jacket heat shrink. This will serve as a break to possible future water draining inside the outer cable jacket and running down to the motor termination box.
- Control and power cables which were located underground for DFOTP-2A were replaced.
- Installation specification 13-EN-0300 "Installation Specification for Electrical Cable in Cable Trays" and 13-EN-0301 "Installation Specification for Electrical Cables in Conduit and Duct Banks" were revised to identify compatible chemicals for cleaning cables.
- Installation specification 13-EN-0306 "Installation Specification for Cable Splicing and Termination" was revised to add a requirement to install a cable splice or other means of blocking the flow of water internal to the outer jacket of cables where an electrical component is at a location such that a hydrostatic head may be developed with respect to the cables.
In addition to the actions described above, the following modification and action will be performed.
- Implement a Design Change Request to replace the underground duct banks and associated cabling from all three EDG buildings to the six site DFOST vaults with direct buried submersible cables.
- Evaluate the need for cable splices for critical components and components that support key safety or operational functions, that are located in areas where water may be transported through the outer jacket of the cables.
Additionally, Palo Verde will continue to monitor the EDG DFOTP cables as part of the Palo Verde preventive maintenance program to ensure the actions taken are effective to prevent water flow to the DFOTP motor termination box.
8. PREVIOUS SIMILAR EVENTS:
Previously, PVNGS reported a DFOTP-2B failure in April 2009 related to water intrusion via �