05000445/LER-2012-002

LER-2012-002, COMANCHE PEAK 05000445 10OF06

Docket Number
Event date:	06-20-2012
Report date:	08-16-2012
Reporting criterion:	10 CFR 50.73(a)(2)(ix)(A) 10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident
4452012002R00 - NRC Website

A. REPORTABLE EVENT CLASSIFICATION

This condition is reportable per 10CFR50.73(a)(2)(1x)(A), "Any event or condition that as a result of a single cause could have prevented the fulfillment of a safety function for two or more trains or channels in different systems" and also as a defect per 10CFR21.

B.PLANT CONDITION PRIOR TO EVENT On June 20, 2012, Comanche Peak Nuclear Power Plant (CPNPP) Units 1 and 2 were in Mode 1 operating at 100% power.

OF THE EVENT AND THAT CONTRIBUTED TO THE EVENT

There were no structures, systems, or components that were inoperable at the start of the event that contributed to the event.

D. NARRATIVE SUMMARY OF THE EVENT, INCLUDING DATES AND APPROXIMATE TIMES

Multiple safety related breaker control devices manufactured by ABB Inc. have failed while in service at CPNPP, resulting in three cases in which Technical Specification equipment became inoperable. The control device is essentially a mechanical relay. It operates from levers internal to the breaker and functions similar to an electronic relay to provide contacts to charge the closing springs and also provides control interlocks and indication. A "blue light" provides indication that the closing springs are charged and the closing circuit is complete to allow automatic or remote (i.e. control board hand switch) closure. The control device is not repaired or refurbished during breaker maintenance at CPNPP, the control device is simply replaced as a subcomponent to the breaker. Investigation of the control device failures determined that a component internal to the control device (the contact carrier) had broken. The following is a summary of the three safety related control device failures:

On October 22, 2009, the control device failed on the Emergency Diesel Generator (EDG) 1-02 To 6.9 Kv Switchgear 1EA2 Emergency Feeder Breaker [EllS: (EK)(DG)(BKR)]. EDG 1-02 was declared inoperable, the breaker was replaced, and EDG 1-02 was declared operable.

On October 20, 2011, the control device failed on the Motor Driven Auxiliary Feedwater Pump (MDAFWP) 1­ 02 Motor Breaker [EIIS: (BA)(P)(M0)(BKR)]. The breaker was found closed with no blue light indication on the breaker door. MDAFWP 1-02, was declared inoperable, the breaker was replaced, and MDAFWP 1-02 was declared operable.

On March 18, 2012, the control device failed on the Centrifugal Charging Pump (CCP) 2-02 Motor Breaker [EIIS: (BQ)(P)(M0)(BKR)]. The breaker was found closed with no blue light indication on the breaker door.

CCP 2-02 was declared inoperable, the breaker was replaced, and CCP 2-02 was declared operable.

In addition to these failures, three other safety related control device failures occurred in 2010 and 2012 that did not affect operability (two on Unit 2 and one on Unit 1). Following the MDAFWP 1-02 Motor Breaker control device failure in October 2011, a failure analysis was initiated to investigate potential causes. The failed control devices from each lot in the warehouse (herein referred to as "New Replacement"), and control devices manufactured prior to 2000 (herein referred to as "Original Vintage") were sent for comparison.

Dynamic Mechanical Analysis (DMA) tests to measure the mechanical properties and Electron Spin Resonance (ESR) tests to assess the concentration of relevant chemical species were performed. The purpose of these tests was to determine whether or not the poor performance of the limit switch contact carrier frames from the failed control devices was attributed to changes in the overall process used to fabricate the cured phenolic material in the limit switch contact carrier frames. The failure analysis was completed on June 14, 2012 and concluded that:

1) The storage modulus curves generated by the DMA test on the set of broken limit switch contact carrier frames showed no evidence of a post cure in the phenolic material. This suggests that these control devices were fully cured. In contrast, the storage modulus curves for all of the contact carriers in the 2000 and prior control devices and new control devices from the warehouse showed evidence of a post cure during the DMA testing. It is believed that not completing the cure is beneficial to the material's toughness because the presence of some "rubbery" material reduces the susceptibility of the material to breaking during shock loading.

2) Lower Glass Transition Temperature values do not appear to be the dominant factor in producing a Limit Switch contact carrier frame that can tolerate service-induced stresses.

3) There were two examples where the Limit Switch contact carrier frame failed in service but the "tell-tale" tabs remained intact. This suggests that lateral movement of the crank handle did not contribute to these failures.

4) The shape, size and identifying marks on the "Original Vintage" and the "New Replacement" limit switch contact carrier frames appeared to be identical. There do not appear to be any changes in manufacturing dies.

5) It is likely that less than adequate control of the manufacturing process (e.g., temperature, time, feedstock purity, etc.) increased the susceptibility to shock loading failure of the cured phenolic material used to make the "New Replacement" Limit Switch contact carrier frames.

Based on the failure analyses and a history review, CPNPP concluded that all six of these failures were from lots RR34644, RR35670, and RR35610 that were purchased from ABB Inc. in July 2007 and July 2008 (RR numbers are assigned by CPNPP to each shipment of parts as received at CPNPP. This does not necessarily correspond to a single batch from the manufacturer). The failure analysis concluded that the control devices from these RRs have different post cure properties as compared to lots from earlier and later vintage control devices. Post cure is important to the control device because the control device experiences shock loading during breaker cycles. Phenolic material supplied within the above listed lots was more susceptible to failure during operation.

An extent of condition review was performed for this condition. This condition potentially applies to all 6.9Kv breakers that had control devices received in July 2007 and July 2008. A review of Work Order history for all 6.9 Kv breakers received since July 2007 was performed to identify the affected breakers. 38 control devices were received from the affected lots, and 37 control devices have been accounted for. All Safety related and non-safety breaker maintenance and corrective actions were reviewed for breakers received devices could be located, none are in the warehouse and none are installed in safety related applications in the plant. The extent of condition review also considered what other phenolic parts on 6.9 Kv breakers received in 2007 and 2008 might have been replaced during breaker maintenance or corrective maintenance. There are no other phenolic parts on the breakers that are replaced during breaker preventive maintenance. There are several other phenolic components used in the 6.9 Kv switchgears, however, the extent of condition review concluded that these components are not affected by this condition.

As noted above, control device failures caused EDG 1-02, MDAFWP 1-02, and CCP 2-02 to become inoperable. Therefore, this condition is reportable per 10CFR50.73(a)(2)(ix)(A), "Any event or condition that as a result of a single cause could have prevented the fulfillment of a safety function for two or more trains or channels in different systems." Since the cause of the these failures was due to a manufacturing problem, this condition is also reportable as a defect per 10CFR21.

E.THE METHOD OF DISCOVERY OF EACH COMPONENT OR SYSTEM FAILURE, OR PROCEDURAL

PERSONNEL ERROR

The control device failures were discovered by either Operations personnel (Utility, Licensed) performing equipment checks in the plant or by Maintenance personnel (Utility, Non-Licensed) performing breaker replacements/refurbishments.

A. CAUSE OF EACH COMPONENT OR SYSTEM FAILURE

The breaker control devices failed due to fractured limit switch contact carriers. The contact carriers fractured due to a manufacturing error. The material of the control devices was not considered a critical characteristic for verification by the vendor, and the vendor's control of the phenolic material curing was not consistent. The manufacturing process for the control devices from lots RR35610, RR35670 and RR34644 resulted in "cured" phenolic material with "saturated" cross links resulting in a more brittle material. These control device contact carriers later failed when subjected to shock loading during cycling of the breaker.

B.FAILURE MODE, MECHANISM, AND EFFECTS OF EACH FAILED COMPONENT The control devices failed due to a fractured limit switch contact carrier. The control device contact carriers failed when subjected to shock loading during cycling of the breaker. The control device failures caused EDG 1-02, MDAFWP 1-02, and CCP 2-02 to become inoperable.

WITH MULTIPLE FUNCTIONS

The control device contact carrier has three contacts that perform the following functions: The first function is provided by contact 8-9 of the contact carrier used in charging the closing springs of the breaker following closure. The second function is provided by contact 10-11 used in the closing circuit and provides the "blue light" indication that the closing springs are charged. The third function is controlled from contact 12-13 which provides contact in an anti-pump circuit.

D.FAILED COMPONENT INFORMATION Control Device part number: 191921T06 Description: Control Device 125VDC 5/7.5HK Manufacturer: ABB

• HI. ANALYSIS OF THE EVENT

A. SAFETY SYSTEM RESPONSES THAT OCCURRED

Not applicable - No safety system responses occurred as a result of this event.

B. DURATION OF SAFETY SYSTEM TRAIN INOPERABILITY

EDG 1-02 was Inoperable from 2247 hours0.026 days <br />0.624 hours <br />0.00372 weeks <br />8.549835e-4 months <br /> on October 21, 2009 to 2022 hours0.0234 days <br />0.562 hours <br />0.00334 weeks <br />7.69371e-4 months <br /> on October 22, 2009 (approximately 21 hours2.430556e-4 days <br />0.00583 hours <br />3.472222e-5 weeks <br />7.9905e-6 months <br /> and 35 minutes). MDAFWP 1-02 was inoperable from 1009 hours0.0117 days <br />0.28 hours <br />0.00167 weeks <br />3.839245e-4 months <br /> to 2351 hours0.0272 days <br />0.653 hours <br />0.00389 weeks <br />8.945555e-4 months <br /> on October 20, 2011 (approximately 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br /> and 42 minutes). CCP 2-02 was inoperable from 0830 hours0.00961 days <br />0.231 hours <br />0.00137 weeks <br />3.15815e-4 months <br /> to 1532 hours0.0177 days <br />0.426 hours <br />0.00253 weeks <br />5.82926e-4 months <br /> on March 3, 2012 (approximately 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> and 2 minutes).

C. SAFETY CONSEQUENCES AND IMPLICATIONS OF THE EVENT

The EDG 1-02 To 6.9 Kv Switchgear 1EA2 Emergency Feeder Breaker is required to close upon loss of all offsite AC power, so that the EDG can power the associated Class 1 E bus. With this control device failure, the breaker would not have been able to close following a loss of all offsite AC power. Since the redundant train EDG 1-01 breaker control device was also from lots RR34644, RR35670, and RR35610, it was removed and successfully cycled 40 times to envelope the cycles required during its mission time. Because this breaker completed the 40 cycles successfully, EDG 1-01 would have met its mission time and past operability was maintained. Therefore, there was a redundant operable train available to fulfill the applicable system safety function and there are no safety consequences associated with the failure of this control device.

The MDAFWP 1-02 Motor Breaker is required to close for a Safety Injection (SI) signal. In addition, if the SI is followed by a Blackout Signal (BOS) the breaker has a safety function to open after closing (re-close) on the SI in order to close again according to the timing set by the Blackout Sequencer. Based on the failure mode associated with the control device, for SI alone, the breaker would not be required to open and subsequently re-close and therefore the breaker would have met its safety function for SI. However, during an SI followed by BOS, the breaker has a safety function to re-close following a trip in order to sequence the loads onto the EDG. Therefore this breaker would not have performed this safety function during an SI followed by BOS. Since the redundant train MDAFWP 1-01 breaker control device was also from lots RR34644, RR35670, and RR35610, it was removed and successfully cycled 40 times to envelope the cycles required during its mission time. Since this breaker completed the 40 cycles successfully, MDAFWP 1-01 would have met its mission time and past operability was maintained. CPNPP also has a turbine driven AFWP, with approximately twice the capacity of each MDAFWP, which was unaffected by this event.

Therefore, there was a redundant operable train available to fulfill the applicable system safety function and there are no safety consequences associated with the failure of this control device.

The CCPs are required in SI mode for large and small break Loss Of Coolant Accident (LOCA) protection.

Thus, the CCP 2-02 Motor Breaker is required to close on a SI signal. In addition, if the SI is followed by a BOS, the breaker has a safety function to open after closing on the SI, in order to close (re-close) again according to the timing set by the Blackout Sequencer. Thus, for SI (alone) the breaker would not be required to open and subsequently close (re-close) and therefore, for the failure mode associated with the control device, the breaker would have met its safety function for SI. However, during an SI followed by BOS, the breaker has a safety function to re-close following a trip, in order to allow for sequencing of the loads onto the EDG. The breaker would not have performed this safety function during an SI followed by BOS. However, the breaker control device on the other train CCP (2-01) was not from lots RR34644, RR35670, and RR35610, and it was unaffected by this event. Therefore, there was a redundant operable train available to fulfill the applicable system safety function and there are no safety consequences associated with the failure of this control device.

Based on the above, there was a redundant operable train available to fulfill the applicable system safety functions. Therefore, this event is not reportable as a safety system functional failure per 10CFR50.73(a)(2)(v)(D) and the potential safety significance was very low. This event had minimal safety consequences and the health and safety of the public was not affected.

IV. CAUSE OF THE EVENT

The breaker control devices failed due to fractured limit switch contact carriers. The contact carriers fractured due to a manufacturing defect. The phenolic material of the control devices was not considered a critical characteristic for verification by the vendor, and the vendor's control of the phenolic material curing was not consistent. The manufacturing process for the control devices from lots RR35610, RR35670 and RR34644 resulted in "cured" phenolic material with "saturated" cross links resulting in a more brittle material. These control device contact carriers later failed in service when subjected to shock loading during cycling of the breaker.

As part of CPNPP's breaker refurbishment maintenance, the control device is a wholesale replacement item.

The control device is not repaired or refurbished at CPNPP, therefore, CPNPP work practices did not contribute to the premature failures associated with this manufacturing defect.

V. CORRECTIVE ACTIONS

All of the control devices from lots RR35610, RR35670 and RR34644 associated with safety related breakers installed in the plant were replaced. Receipt inspection testing criteria for the procurement of future breaker control devices was developed. An Operating Experience Report has been issued to the industry on this event, and the breaker vendor (ABB Inc.) was also notified.

VI. PREVIOUS SIMILAR EVENTS

There have been no previous similar reportable events at CPNPP in the last three years. Although there have been other safety related breaker control device failures from the affected lots, none of these resulted in a reportable condition.