05000529/LER-2009-001
| Docket Numbersequential Revmonth Day Year Year Month Day Yearnumber No. | |
| Event date: | 04-22-2009 |
|---|---|
| Report date: | 07-21-2009 |
| 5292009001R00 - NRC Website | |
All times are Mountain Standard Time and approximate unless otherwise indicated.
1. REPORTING REQUIREMENT(S):
This LER is being submitted pursuant to 10 CFR 50.73 (a)(2)(i)(B) as a condition prohibited by Technical Specification (TS) 3.8.1, "AC Sources — Operating," and TS 3.8.2, "AC Sources — Shutdown," for an inoperable Emergency Diesel Generator (EDG) (EIIS Code: EK) due to a failure of the associated fuel oil transfer pump; and 10 CFR 50.73 (a)(2)(v)(A), (B), (C) and (D) for a condition that could have prevented the fulfillment of a safety function of systems needed to shutdown the reactor and maintain it in a safe shutdown condition, remove residual heat, control the release of radioactive material, and mitigate the consequences of an accident. On May 22, 2009, based on the results obtained from the fuel oil transfer pump failure investigation, this event was determined to be reportable since EDG-2B may not have functioned for the full duration of its mission time and for a period that exceeded the TS required action time. Additionally, at various times since 2004, the opposite train EDG was unavailable resulting in the loss of the safety function for emergency onsite AC power during these periods.
2. DESCRIPTION OF STRUCTURE(S), SYSTEM(S) AND COMPONENT(S):
Diesel Fuel Oil Transfer System Description (EIIS Code: DC) The Diesel Generator Fuel Oil Transfer System (DGFOS) consists of one diesel fuel oil (DF) storage tank, one diesel fuel oil transfer pump and one diesel fuel oil day tank per diesel generator along with the associated piping, valves, and instrumentation. The DF day tank overflow returns to the storage tank.
The transfer pump is located in the diesel fuel oil storage tank and is accessible from a vault built above each diesel fuel oil storage tank. Failure of one pump does not affect the operability of any component in the other train. The two DGFOSs, one per train for each generating unit, are physically separated from each other to prevent interaction from one generating unit to the other.
The transfer pump can be operated from either the main control room or the local diesel control panel. Alarms and indications of day tank level and transfer pump status are displayed in the main control room and at the local diesel control panel. The fuel oil transfer pump delivers a minimum of 15 gpm to the EDG day tank. The diesel fuel oil transfer pump takes suction from the diesel generator fuel oil storage tank and discharges into the diesel generator fuel oil day tank which gravity feeds fuel oil to the EDG. The fuel oil day tank level is automatically controlled by a level control system that cycles the transfer pump as needed to maintain level.
Power for the transfer pumps is fed from Class lE 480 VAC motor control centers. The cables providing power to the motors are manufactured with cross linked polyethylene insulation and a neoprene jacket. The cables are routed from the motor control center through a raceway and outside the diesel generator building through a sand encased duct bank. The cables are then routed through sealed conduits to the diesel fuel oil storage tank vault and terminate in a cast aluminum connection box located above the tank. This connection box is of a leak proof design to prevent diesel fuel oil from seeping into the conduit in the event of a diesel fuel transfer pump motor stator liner rupture.
3. INITIAL PLANT CONDITIONS:
On April 22, 2009, Palo Verde Unit 2 was in Mode 1 (Power Operation), at 100 percent power, at normal operating temperature (NOT) and normal operating pressure (NOP).
There were no structures, systems, or components inoperable at the time of the event that contributed to the event.
4. EVENT DESCRIPTION:
On April 22, 2009, at 10:05, Unit 2 Operations personnel were performing surveillance test 73ST-9DF01, "Diesel Fuel Oil Transfer Pump — Inservice Test," on the EDG 2B fuel oil transfer pump, 2MDFBP01. This procedure ensures that the fuel oil transfer pump will start and shut off at a specified level in the fuel oil day tank. The transfer pump started; A however, approximately 2 seconds later the pump breaker opened to the trip-free position.
A Safety Equipment Inoperable Status (SEIS) alarm was received and a diesel generator low priority alarm was received. The Operations Shift Manager (SM) declared EDG-2B inoperable and entered TS Limiting Condition of Operation (LCO) 3.8.1, Condition B. The equipment was quarantined and actions to troubleshoot the failure commenced.
The troubleshooting efforts determined a direct short to ground existed on a terminal lead in the cable connection box located in the fuel oil storage vault. The line side of the connection box had extensive corrosion and a broken insulator. The load side of the connection box was clean and free of corrosion. The cables were disconnected from the terminals in the connection box and tested for insulation failures. The test results indicated no insulation failures or degradation of the cables.
The connection box for the fuel oil transfer pump was replaced and the pump was retested satisfactorily. The Unit 2 SM declared EDG-2B Operable on April 23, 2009, at 08:58.
Prior to this event, moisture was discovered in the electrical conduit and the cable connection box in the Unit 3, Emergency Diesel Generator, A Train, (EDG-3A) fuel oil transfer pump in December, 2004. Work orders were generated to inspect and megger the other five site EDG fuel oil transfer pump cable connection boxes. Cable megger results for the five EDG fuel oil transfer pump cables were found acceptable. The inspection identified some moisture in the connection box for EDG-2B fuel oil transfer pump, but no corrosion was noted. Due to the small amount of moisture found in 2004, the EDG-2B conduit was not dried out. Corrective maintenance for EDG-2B consisted of wiping the moisture out of the connection box and sealing the conduits entering and leaving the connection box with a sealant. The failure of the EDG-2B connection box appears to have been caused from residual moisture that was left in the conduit from 2004.
During the 2009 extent of condition investigation, the other five site fuel oil transfer pump connection boxes were inspected, meggered and found to be acceptable. There was no evidence that any additional water had been introduced into these components since 2004.
�
5. ASSESSMENT OF SAFETY CONSEQUENCES:
On April 15, 2009, the fuel oil transfer pump successfully demonstrated that it was capable of supporting EDG-2B operability when EDG-2B ran for 6.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> at full load to perform 40ST-9DG02, "Diesel Generator 'B' Test." This run time and load caused the fuel oil transfer pump to cycle to maintain day tank level. There were no identified deficiencies and no performance issues identified during this surveillance run.
On April 22, 2009 at 10:05, during the performance of 73ST-9DF02, the Unit 2, B train, diesel fuel oil transfer pump failed. As a result of the failure, EDG-2B was declared inoperable per TS 3.8.1 and the 10 day TS LCO action statement was entered. Following the failure of the transfer pump, the remaining fuel oil inventory in the diesel fuel day tank was sufficient to meet 176 minutes of run time based on actual load requirements as demonstrated from a previous loss of offsite power event.
The investigation team was not able to determine when EDG-2B could not meet its mission time. However, based on the failure mechanism, it is assumed EDG-2B was inoperable for a period greater than allowed by TS LCO 3.8.1 and 3.8.2. Additionally, since 2004, EDG 2A was out of service at various times to support maintenance and testing activities.
If needed, the implementation of Standard Appendix 61 of the Emergency Operating Procedures (41EP-9E010), "Restore Offsite power to PBB-SO4 (energized from a Station Black Out Generator)," provides the steps needed to place either of the Station Black Out Generators on the Class 1E, 4.16 KV bus that would be supplied by EDG-2B if normal power sources were lost.
The risk for this event was calculated using the Probability Risk Assessment (PRA) model documented in calculation 13-NS-0O29, revision 16, with specific conditions, assumptions and system parameters. The results estimated a change in Conditional Core Damage Probability of 1.7E-7.
�
6. CAUSE OF THE EVENT:
The direct cause for failure of the EDG-2B fuel oil transfer pump was determined to be an electrical short to ground inside the connection box.
The root cause of the condition was that the diesel fuel oil storage tank vault electrical conduit and penetrations were not effectively maintained and monitored to prevent the intrusion of moisture into the connection box. The presence of moisture initiated and sustained a corrosion process within the motor connection box for the EDG-2B fuel oil transfer pump which eventually resulted in the failure of the fuel oil transfer pump.
7. CORRECTIVE ACTIONS:
Immediately after the failure, the terminal connection and associated connection box for EDG-2B fuel oil transfer pump were replaced. Retest results following the corrective maintenance were acceptable and the EDG-2B was declared Operable.
The following additional corrective actions were or will be implemented for all three units:
- The five other site EDG connection boxes were inspected and tested and all were found to be acceptable.
- A preventive maintenance task will be developed to inspect the connection box and test the power cables on a frequency of 18 months or less.
- The underground conduits between the Diesel Generator building and the diesel fuel vault will be inspected to verify the configuration is adequate to prevent water intrusion.
8. PREVIOUS SIMILAR EVENTS:
No similar conditions have been reported in the prior three years.