05000446/LER-2010-002

LER-2010-002, UNIT 2 EDG 2-02 INOPERABLE DUE TO REMAINING IN DROOP MODE VERSUS ISOCHRONOUS MODE

Document Numbersequential Revmonth Day Year Year Month Day Year
Event date:	07-29-2010
Report date:	12-14-2011
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v), Loss of Safety Function
4462010002R01 - NRC Website

A. REPORTABLE EVENT CLASSIFICATION:

10CFR50.73(a)(2)(i)(B) "Any operation or condition which was prohibited by the plant's Technical Specifications" and 10CFR21 "Defect that could create a substantial safety hazard.

B.PLANT CONDITION PRIOR TO EVENT:

On 7/29/2010, Comanche Peak Nuclear Power Plant (CPNPP) Unit 2 was in Mode 1 operating at 100% power.

EVENT

There were no inoperable structures, systems, or components at the start of the event that contributed to the event.

D.NARRATIVE SUMMARY OF THE EVENT, INCLUDING DATES AND APPROXIMATE

TIMES:

CPNPP has two Emergency Diesel Generators (EDGs) [EllS: (EK) (DG)] per unit (one per safety related train) and they provide a backup power supply for their respective safety related buses.

Each EDG automatically starts in emergency Mode whenever an undervoltage condition is experienced on its respective emergency bus or a Safety Injection signal is received. The EDGs are 16 cylinder Delaval Enterprise Type DSRV-16-4 engines. The EDG Integrated Test Sequence (ITS) Surveillance, per procedure OPT-435B "Train B Integrated Test Sequence," is performed during refueling outages to meet the Surveillance Requirements (SR) identified in the Technical Specifications (TS) 3.8.1.

The speed control system on the EDGs is a control loop that utilizes a comparison of set point versus resultant value to maintain the frequency of the machine. When the EDGs are in standby I awaiting a start signal or running paralleled to the grid for testing, they are in Droop Mode. Droop Mode allows the EDG governor to maintain the EDG frequency based on Droop setting and percentage of EDG rated load. When operating in Isochronous Mode, the EDG is tied to the emergency bus, and isolated from the grid. In this Mode, the EDG governor overrides the Droop I Mode and maintains the EDG frequency to a set value.

On 10/21/2006 at 0455 hours0.00527 days <br />0.126 hours <br />7.523148e-4 weeks <br />1.731275e-4 months <br />, 4/3/2008 at 0524 hours0.00606 days <br />0.146 hours <br />8.664021e-4 weeks <br />1.99382e-4 months <br />, and on 10/12/2009 at 0635 hours0.00735 days <br />0.176 hours <br />0.00105 weeks <br />2.416175e-4 months <br />, Operators (Utility, Licensed) performed the ITS Surveillance per procedure OPT-435B. The Surveillance is performed in two parts. The first portion performed was Safety Injection (SI) with loss of offsite power (LOOP) and the second portion was LOOP. The test primarily verifies actuation of the various pumps and fans that occur on SI/LOOP and LOOP events. At the above times, it was believed that EDG 2-02 passed both portions of the test.

The acceptance criteria for frequency are:

• Highest steady state frequency in first 5 minutes after load sequencing (DAS Channel 4) 5. 61.2 Hz

• Lowest steady state frequency in first 5 minutes after load sequencing (DAS Channel 4) 58.8 Hz After the SI with LOOP portion of the above Surveillances, the data collected on EDG 2-02 was within TS limits for highest steady state frequency and within TS limits for steady state lowest frequency which satisfied the above test criteria. During the restoration of SI with LOOP Surveillance portion, OPT-435B requires a review of the EDG 2-02 voltage and frequency DAS traces against examples contained in OPT-435B to verify proper EDG operation. This check verifies the governor control system has shifted to Isochronous Mode. Next, the LOOP portion of the Surveillance was performed. After the LOOP portion of the past three Surveillances, the data collected on EDG 2-02 was also within TS limits. During the restoration with LOOP, OPT-435B does not require a review of EDG 2-02 voltage and frequency DAS traces to verify proper EDG operation. Therefore, Operations did not recognize that EDG 2-02 was operating in the Droop Mode during the LOOP portion of the Surveillances on 10/21/2006, 4/3/2008, and 10/12/2009.

Additionally, with the DAS trace scale selected (i.e, the 0-70 Hz scale versus the 55 to 65 Hz scale used in OPT-435B), the voltage and frequency tracked together during the test which made the lower frequency less noticeable.

On October 15, 2009, Meter and Relay technicians (Utility, Non-licensed) performed a scheduled preventive maintenance (PM) on EDG 2-02 to replace the electronic governor. On October 24, 2009, the Meter and Relay technicians performed a simulated ITS per procedure MSE-CO-0886 as part of post-work testing. The data collected on EDG 2-02 was: 59.9 Hz for highest steady state frequency and 59.69 Hz for steady state lowest frequency. The Meter and Relay test procedure does not require verification of the DAS trace during the recovery of the EDG.

However, Meter and Relay technicians reviewed the DAS trace as a good work practice and identified that the frequency was not returning to approximately 60 Hz. This indicated the EDG governor was in Droop Mode and not in lsochronous Mode, which is the proper operation of the I EDG governor when it is providing power to an isolated bus. The Meter and Relay technicians initiated corrective actions and began troubleshooting. The Meter and Relay technicians found Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 did not make-up preventing the GDR from energizing. If the EDG 2-02 relay [EllS: (EK) (DG) (RLY)] malfunctioned, it could have caused the EDG 2-02 to operate in Droop Mode when responding to loss of coolant accident (LOCA) with a LOOP accident. During troubleshooting, Meter and Relay technicians noticed the switch contacts 97-98 were slightly misaligned, but appeared to be making full contact. Voltage readings across the contacts indicated a high resistance contact. The Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 contact was burnished.

On October 25, 2009, Meter and Relay technicians performed multiple operations of the 2EA2-1 Breaker which verified proper operation of the Auxiliary Switch 2EA2-1 52b 97-98 contact. A retest of the simulated ITS successfully verified that the GDR energized which allows the EDG to operate in lsochronous Mode.

On April 5, 2010, in response to a question from the Nuclear Regulatory Commission (NRC) Resident Inspector, the EDG System Engineer (Utility, Non-licensed) was reviewing the Unit 2, Refueling Outage 11 (2RF11) ITS Surveillance data. The EDG System Engineer determined that during the LOOP portion of the ITS Surveillance, the EDG was operating in the Droop Mode.

Consequently, in the event of an EDG supplying power to an isolated bus, the EDG may not have operated in Isochronous Mode.

The EDG System Engineer then reviewed the ITS Surveillances for the last 5 years on all four EDGs, only the 2-02 EDG was found to have been operating in the droop mode. On 10/21/2006, 4/3/2008, and 10/12/2009, the first portion of the 2-02 EDG tests (SI with LOOP) passed and the second portion of the tests failed (LOOP). Based on the observation of the test patterns (e.g., pass/fail, pass/fail, and pass/fail), at some period of time it is likely that the 2-02 EDG could have gone to Isochronous Mode after failing the second portion of the ITS tests and would have performed its safety function. Initial evaluation of the past condition concluded that the 2-02 EDG frequency could have been above the TS limit of 58.8 Hz.

On 7/29/2010, an Engineer (Utility, non-licensed) completed the final evaluation that reviewed the test data for the 2-02 EDG during the past 3 years. The period identified was between 10/1/2006 to 11/1/2009 for previously failed Unit 2 Train B (2-02) ITS Surveillances where the EDG did not shift to Isochronous Mode during the LOOP portion of the test. The evaluation calculated the EDG 2-02 Droop Mode frequency considering motor demands at 58.8 Hz. The 2-02 EDG frequency during the periods 10/1/2006 to 9/8/2008 and 9/20/2009 to 11/1/2009 remained above the TS minimum of 58.8 Hz. Conversely, during the period 9/8/2008 to 9/20/2009 if the EDG remained in the Droop Mode, the 2-02 EDG frequency could have been slightly below the Technical Specifications minimum of 58.8 Hz (58.78 Hz) and the EDG could have been Inoperable. However, an evaluation completed by Westinghouse on 8/5/2010 concluded that had the 2-02 EDG operated at the reduced frequency of 58.78 Hz, the applicable ECCS systems would have performed their safety functions as described in the Safety Analysis. Therefore, this condition is reportable per 10CFR50.73(a)(2)(i)(B) as a condition that was prohibited by the plant's TS.

PROCEDURAL PERSONNEL ERROR

When the 10/12/2009 ITS Surveillance was reviewed on 4/5/2010, a lower voltage and frequency trend was identified. On 4/6/2010, a troubleshooting plan was completed and confirmed that the 2-02 EDG was Operable based on maintenance completed after the Surveillance in 10/2009. Initial evaluation of the past condition concluded that the 2-02 EDG frequency could have been above the TS limit of 58.8 Hz. However, on 7/29/2010 Engineering concluded that during an emergency start for the period 9/8/2008 to 9/20/2009 with EDG 2-02 remaining in Droop Mode, the EDG frequency could have been slightly below the TS minimum of 58.8 Hz (58.78 Hz) and the EDG could have been Inoperable per TS.

A. CAUSE OF EACH COMPONENT OR SYSTEM FAILURE

The Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 did not make-up preventing the GDR from energizing. With the GDR not energized, the EDG remained in Droop Mode.

The switch contacts 97-98 were slightly misaligned, but appeared to be making full contact.

Voltage readings across the contacts indicated a high resistance contact. Several factors could have contributed to the high resistance contact of the switch, but to further identify the cause of the mechanism and effects of the component failure, the switch would need to be removed and failure analysis performed.

B.FAILURE MODE, MECHANISM, AND EFFECTS OF EACH FAILED COMPONENT Several factors could have contributed to the failure of the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98, but to positively identify the failure Mode, mechanism, and effects of the component failure, the switch was removed on 04/11/2011 and failure analysis was performed on 04/16/2011. CPNPP contacted ABB to have a breaker/component specialist assist with the System Engineer in an on-site failure analysis.

The Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 was disassembled to allow for access to the moving contact (ABB part Number 700073A00), stationary contact (ABB part Number 700072A00) and contact molding. The failure analysis found the following:

The cause of this event was oxidation buildup on the rotating contact assembly of the 2EA2-1 52b auxiliary switch, resulting in a high resistance contact that inhibited proper function of the auxiliary switch.

The cause of this issue was a manufacturing issue during the fabrication of the 2EA2-1 52b auxiliary switch movable contact precluded proper operation of the switch.

The travel (wipe) of the rotating contact assembly was inhibited by insufficient travel of the movable contact. The insufficient travel of the movable contact reduced the amount of contact surface at the junction between the moving contact and stationary contact.

The insufficient wipe between the rotating and movable contact, coupled with the reduced contact surface of the moveable contact enabled an oxidation build up which developed into a high resistance contact over the in-service life (31 years installed, 21 and 18 commercial years for Units 1 and 2, respectively) of the 2EA2-1 52b auxiliary switch.

COMPONENTS WITH MULTIPLE FUNCTIONS

There were no secondary functions affected by this component failure.

Comanche Peak Nuclear Power Plant Unit 2 05000 - 446 YEAR I SEQUENTIAL I� REV NUMBER� NO.

D.FAILED COMPONENT INFORMATION � Nomenclature Manufacturer� Model Number Switch, Electrical,� � ABB Power� 700038K01 Auxiliary, 8 Contact Distribution-Nuclear

A. SAFETY SYSTEM RESPONSES THAT OCCURRED

Not applicable — No safety system responses occurred as a result of this event.

B.DURATION OF SAFETY SYSTEM TRAIN INOPERABILITY On 7/29/2010, an Engineer (Utility, Non-licensed) completed an evaluation reviewing the test data for the 2-02 EDG during the past 3 years. The period identified was between 10/1/2006 to 11/1/2009 for previously failed Unit 2 Train B (2-02) ITS Surveillances where the EDG did not shift to Isochronous Mode during the LOOP portion of the test. The evaluation calculated the EDG 2-02 Droop Mode frequency considering motor demands at 58.8 Hz. The 2-02 EDG frequency during the periods 10/1/2006 to 9/8/2008 and 9/20/2009 to 11/1/2009 remained above the TS minimum of 58.8 Hz. Conversely, during the period 9/8/2008 to 9/20/2009 if the EDG remained in Droop Mode, the 2-02 EDG frequency could have been slightly below the Technical Specifications minimum of 58.8 Hz (58.78 Hz) and could have been Inoperable.

C. SAFETY CONSEQUENCES AND IMPLICATIONS OF THE EVENT

The safety function of the CPNPP EDGs is to automatically start and provide power in emergency Mode whenever an undervoltage condition is experienced on their respective emergency bus or an SI signal is received. As noted above, the Unit 2 Train B EDG could have been operated at a frequency slightly below the TS minimum frequency limit and thus, could have been Inoperable from 9/8/2008 to 9/20/2009. However, the EDG would have started and run as designed (albeit at a lower than design frequency).

During this period of time, both of the required offsite sources were operable and available and no events occurred requiring the EDGs to start and perform their safety function of providing power to the safety related busses. In addition, an evaluation completed by Westinghouse on 8/5/2010 concluded that had the 2-02 EDG operated at the reduced frequency of 58.78 Hz, the applicable ECCS systems would have performed their safety functions as described in the Safety Analysis. Based on the above, it is concluded that the health and safety of the public were unaffected by this condition and this event has been evaluated to not meet the definition of a safety system functional failure per 10CFR50.73(a)(2)(v).

IV.CAUSE OF THE EVENT Troubleshooting determined the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 identified a high resistance contact preventing the GDR from energizing and placing the Governor Control in the Isochronous Mode.

The Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 was disassembled to allow for access to the moving contact (ABB part Number 700073A00), stationary contact (ABB part Number 700072A00) and contact molding. The failure analysis found the following:

The cause of this event was oxidation buildup on the rotating contact assembly of the Breaker 2EA2-1 52b Auxiliary Switch 2EA2-1 52b 97-98, resulting in a high resistance contact that inhibited proper function of the auxiliary switch.

The cause of this issue was a manufacturing issue during the fabrication of the Breaker 2EA2-1 52b Auxiliary Switch 2EA2-1 52b 97-98 movable contact precluded proper operation of the switch.

The travel (wipe) of the rotating contact assembly was inhibited by insufficient travel of the movable contact. The insufficient travel of the movable contact reduced the amount of contact surface at the junction between the moving contact and stationary contact.

The insufficient wipe between the rotating and movable contact, coupled with the reduced contact surface of the moveable contact enabled an oxidation build up which developed into a high resistance contact over the in-service life (31 years installed, 21/18 commercial years for Units 1 and 2, respectively) of the 2EA2-1 52b Auxiliary Switch.

V. CORRECTIVE ACTIONS

On 10/24/2009, Electrical Maintenance personnel burnished the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 to restore the 2-02 EDG Operability.

On October 25, 2009, Electrical Maintenance personnel performed multiple operations of the 2EA2- 1 Breaker which verified proper operation of the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97­ 98. A retest of the simulated ITS successfully verified the GDR relay energized which allowed the EDG to operate in isochronous mode.

The System Engineer reviewed the past 5 years of the ITS Surveillance data on all four EDGs.

Review of the data demonstrated that except for the 2-02 EDG, the other three EDGs responded as required to load and maintain generator frequency above TS requirements and therefore were not degraded and remained Operable.

Procedures OPT-430 and OPT-435 Train A and B LOOP restoration section were revised to add an action to require a review of voltage and frequency DAS traces against examples in Attachment 10.1.3 to verify proper EDG operation. In addition, a requirement was added so that the System Engineer reviews the DAS traces for trending and proper diesel generator operation.

0 NRC FORM 366A (10-2010) PRINTED ON RECYCLED PAPER On 04/11/2011, the remaining Auxiliary Switches associated with Breaker 2EA2 were visually inspected for proper alignment. No misalignments were noted during the inspection and none of the visible contacts were noted as unsatisfactory. Also on 04/11/2011, while Unit 2 was down for refueling, the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 was replaced by the Meter and Relay technicians (MR). To verify the new Auxiliary Switch alignment, MR performed the following:

• Visually inspected the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 and found all in proper alignment for open and closed positions,

• Checked the "AS LEFT" contact resistance of the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 against the "As Found' contact resistance

• Final alignment was verified with breaker 2EA2-1 installed and contact alignment verified following the closure and opening of the breaker.

While the 2EA2 switchgear was out of service the System Engineer inspected the remaining Auxiliary Switches in the cubicles within the 2EA2 switchgear and found them all properly aligned.

On 04/15/2011, the Breaker 2EA1-1 Auxiliary Switch 2EA1-1 52b 97-98 was replaced as a conservative measure since Unit 2 was down for refueling.

CPN PP contacted ABB to have a breaker/component specialist assist with the System Engineer in an on-site failure analysis. On 04/16/2011, a failure analysis of the Breaker 2EA2-1 Auxiliary Switch 2EA2-1 52b 97-98 was performed to determine any contributing factors that caused the high resistance contact.

On 07/20/2011, procedure MSE-P0-6000 associated with switchgear inspection was revised to incorporate visual inspection for proper Auxiliary Switch contact alignment and contact condition.

Additionally, Maintenance support procedure MSE-00-0866 was be revised to add work instructions to verify the EDG operating in the isochronous mode during simulated ITS.

On 08/17/2011, Work Orders were generated for the lE 6.9kV switchgear to replace the Auxiliary Switches 52b 97-98. This is to ensure continued reliability from potential age related issues. The replacements will coincide with the site bus outage schedule.

On 08/22/2011, cubicles associated with breakers 1EA1-1, 1EA1-2, 1 EA2-1, and 1EA2-2 were inspected for proper alignment and found satisfactory.

On 09/07/2011, procedure MSE-00-0866 was revised to add work instructions to verify a EDG would operate in isochronous mode during a simulated ITS.

On 09/21/2011, the freedom of movement on all the spare L2 switches in the warehouse was verified.

The following corrective action will be completed in accordance with the CPNPP Corrective Action Program:

• Develop receipt inspection verification plan acceptance criteria to verify the freedom of movement of L2 switches as they are received.

VI. PREVIOUS SIMILAR EVENTS/ADDITIONAL INFORMATION

On 4/9/2011, during Unit 2 twelfth refueling outage, a review of the Train B Diesel Generator (EDG 2-02) Integrated Test Sequence Surveillance identified a lowering frequency. Steady state frequency should have been approximately 60Hz. The documented lowest steady state frequency in OPT-435B Step 9.4.10 was 59.22 Hz. The frequency lowered as loads were sequenced on to the bus, which indicates that the EDG did not shift to Isochronous Mode and stayed in Droop Mode.

However, on 4/12/2011 an evaluation concluded that during an emergency start for the period of 10/25/09 to 04/9/11 with the generator remaining in droop mode, EDG 2-02 frequency would have been above the Technical Specification minimum of 58.8 Hz and the EDG 2-02 would have been operable. The corrective actions for this event address the conditions identified in the similar event discussed above.