05000361/LER-2009-002
| San Onofre Nuclear Generating Station | |
| Event date: | 03-28-2009 |
|---|---|
| Report date: | 5-27-2009 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications |
| 3612009002R00 - NRC Website | |
Event Date: March 28, 2009 Reactor Vendor: Combustion Engineering Mode: Mode 1 Power: 98 percent
Background:
The Containment Cooling System [BK] consists of two trains of Emergency Cooling Units (ECU) each of sufficient capacity to supply 50 percent of the design cooling requirement. Heat is removed through the Component Cooling Water [CC] system. In a post accident operation, following a Containment Cooling Actuation Signal (CCAS), the ECUs are designed to start automatically by a demand to close signal to the associated 1 E breaker.
Description of Event:
On March 28, 2009 at 2228 PST, plant operators were preparing for a routine Technical Specification (TS) Surveillance Requirement (SR) 3.6.6.1.2 and attempted to close 1E 480V supply breaker, 2B0410, to the Emergency Cooling Unit, 2E399. (This was the first time an attempt was made to operate breaker 2B0410 since it was replaced and satisfactorily tested on March 11, 2009.) The breaker failed to close; the ECU was declared inoperable and TS 3.6.6.1 Action "C" was entered requiring restoration to operable status within 7 days.
During repair activities, Southern California Edison (SCE) maintenance personnel discovered a "hex" head bolt in the breaker cubicle. SCE electricians recognized the bolt as a likely cause common to a December 7, 2005 failure on a similar breaker (see Cause discussion below.) At that time, SCE determined the hex head bolt created a mechanical interference between the breaker and the cubicle, forcing the breaker to misalign when racked in. The hex head bolt was replaced with a round head bolt which restored with the required clearance. The ECU was returned to service on April 1, 2009, exiting the Action. Engineering analysis concluded the breaker likely was not functional since the return to service on March 11, 2009. Therefore, the ECU was inoperable for a period greater than the 7 days allowed by Technical Specification. This report is being submitted in accordance with 10 CFR 50.73(a)(2)(i)(B).
Cause of the Event:
SCE bench tested the breaker, ABB K-line (formerly ITC) model K600S, and the failure could not be repeated. Consequently, SCE has performed a failure mode analysis and concluded that the most likely cause was common to the 2005 event on a similar breaker.
The 2005 investigation concluded that interferences and misalignments between the breaker and breaker cubicle were causing the intermittent discontinuity in the secondary contacts necessary to control the opening and closing of the breaker remotely from the control room:
1. Interference between the breaker and the cubicle.
It determined the breaker support plates were interfering with the hex head bolts at the rear of the cubicle that hold the secondary contact carrier to the cubicle. The interference contributed to reducing contact pressure between the secondary contacts on the breaker and the stab connection on the cubicle. Round head bolts give the additional clearance to prevent the interference.
2. Dirt contaminated contacts causing high resistance.
3. Marginal alignment.
The cubicle was designed to accommodate either a 600 or a 1600 Amp breaker. The installation of the rails for a smaller breaker allows greater room for misalignment of the breaker cradle rails.
The current event was similar to the 2005 event with respect to the first cause. That is, there was evidence of metal to metal striking on the head of the bolt and the breaker support plates. Additionally, electricians noted a slight bobble as the breaker settled in place after racking in. The other two 2005 causes were not credible in the 2009 event. That is, the second 2005 cause of dirt contaminated contacts is not considered as a likely contributor to the failure. SCE regularly cleans the secondary contacts with Isopropyl Alcohol as part of the routine maintenance. The third cause was not applicable because the cubicle housing 2B0410 was the smaller 600 Amp cubicle. Misalignment due to this cause, as considered in the 2005 evaluation, is unlikely.
Contributing Causes:
The narrow scope of the 2005 cause evaluation may have contributed to the current event. The evaluation did not consider breakers in 600 Amp breaker cubicles. It was incorrectly believed that the hex head bolts were not installed in these cubicles. Corrective actions from the 2005 event required revising the preventive maintenance procedures for breakers and switchgear. Although the procedure included changing the hex head bolts for round head bolts, not all cubicles were modified as part of routine scheduled maintenance activity. The modification to change bolts was improperly classified as a betterment issue and not a corrective action.
Corrective Actions:
SCE has completed the following corrective/compensatory actions:
1) SCE replaced the hex head bolts with round head bolts on 2B0410.
2) SCE identified other breakers susceptible to this interference. The breakers were cycled in their as found condition to verify operability and any hex head bolts were replaced, if found. They include the breakers listed in Table 1.
3) The SONGS Corrective Action Program has been revised. The program now includes:
a. Rigorous challenge of the scope and Corrective Actions in cause evaluations by trained Corrective Action Review Boards.
b. Improved tracking of Corrective Actions for completeness and effectiveness.
Additional corrective actions may be identified to improve performance.
Safety Significance:
SCE has determined the ECU could not fulfill its safety function for a period beginning March 11, 2009 through April 1, 2009.�Due to the duration of unavailability, this event did not significantly add to baseline plant risk. SCE cycled all other identified (Table 1.) breakers successfully. Therefore, SCE has determined the other potentially affected breakers could have performed their safety function in all Design Basis Accidents, excluding seismic. SCE could not reproduce the failure. Therefore, without definitive evidence to the contrary, SCE conservatively considers all breakers susceptible to the hex head bolt interference not seismically qualified.
Additional Information:
On January 23, 2006, SCE reported the failure of the 1 E breaker powering the pressurizer heaters in Table 1.
Unit 2 480v Loadcenter 2B04 (Ref. 30118 Loc. 50 CB) 2B0402 200 KW PZR HTR (BACKUP GROUP) E615-E618 & ALT. PWR. TOQ. HATCH 2B0405 2P191 TRAIN 'A FEEDER BREAKER (SWING CHARGING PUMP) 2B0409 UPPER DOME AIR CIRCULATOR 2A071 FEEDER BREAKER 2B0410 CONTAINMENT EMERGENCY FAN 2E399 FEEDER
BREAKER
2B0411 CONTAINMENT EMERGENCY FAN 2E401 FEEDER
BREAKER
2B0413 CHARGING PUMP 2P190 FEEDER BREAKER 2B0418 CONTROL BLDG CONTROL ROOM EMERG AC UNIT 418
FDR
2B0419 STANDBY UPPER DOME AIR CIRCULATOR A074 FEEDER BKR Unit 2 480v Loadcenter 2B06 (Ref. 30120 Loc. 50 CB) 2B0602 PRESS. HTR BANK 2E129 & ALT. PWR. EQPMT. HATCH 2B0609 UPPER DOME AIR CIRCULATOR A072 FEEDER BREAKER 2B0610 CONTAINMENT EMERGENCY FAN 2E400 FEEDER
BREAKER
2B0611 CONTAINMENT EMERGENCY FAN 2E402 FEEDER
BREAKER
2B0613 CHARGING PUMP 2P192 FEEDER BREAKER 2B0615 CONTROL ROOM EMERGENCY AC UNIT A419 FEEDER BREAKER 2B0617 2P191 TRAIN 'B' FEEDER BREAKER (SWING CHARGING PUMP) 2B0619 STANDBY UPPER DOME AIR CIRCULATOR 2A073 FEEDER BKR 512C FORM 366A � U.S. NUCLEAR REGULATORY COMMISSION (7-2001) Table 1 cont.
Unit 3 480v Loadcenter 3B04 (Ref. 32118 Loc. 50 CB) 3B0402 PRESSURIZER HEATER BANK 3E128 (3E615-3E618)
FEEDER
3B0405 3P191 TRAIN 'A' FEEDER BREAKER (SWING CHARGING PUMP) 3B0409 UPPER DOME AIR CIRCULATOR 3A071 FEEDER BREAKER 3B0410 CONTAINMENT EMERGENCY FAN 3E399 FEEDER
BREAKER
3B0411 CONTAINMENT EMERGENCY FAN 3E401 FEEDER
BREAKER
3B0413 3P190 FEEDER BREAKER (TRN 'A' CHARGING PUMP) 3B0418 EMERGENCY AC UNIT E418 UNIT 3 FEEDER BREAKER 3B0419 STANDBY UPPER DOME AIR CIRCULATOR 3A074 FEEDER BREAKER Unit 3 480v Loadcenter 3B06 (Ref. 32120 Loc. 50 CB) 3B0602 PRESSURIZER HEATER BANK 3E129 (3E625 AND 3E627 THROUGH 3E629) 3B0609 UPPER DOME AIR CIRCULATOR 3A072 FEEDER BREAKER 3B0610 CONTAINMENT EMERGENCY FAN 3E400 FEEDER
BREAKER
3B0611 CONTAINMENT EMERGENCY FAN 3E402 FEEDER
BREAKER
3B0613 3P192 FEEDER BREAKER (TRN 'B' CHARGING PUMP) 3B0615 CONTROL ROOM EMERGENCY AC UNIT E419 UNIT 3 FEEDER 3B0617 3P191 TRAIN 'B' FEEDER BREAKER (SWING CHARGING PUMP) 3B0619 STANDBY UPPER DOME AIR CIRCULATOR 3A073 FEEDER BREAKER