05000316/LER-2005-001
| Docket Number | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(iv)(A), System Actuation 10 CFR 50.73(a)(2)(v)(A), Loss of Safety Function - Shutdown the Reactor |
| Initial Reporting | |
| 3162005001R00 - NRC Website | |
REPORT- REPORT-MANU- MANU-CAUSE SYSTEM COMPONENT ABLE TO CAUSE SYSTEM COMPONENT ABLE TOFACTURER FACTUREREPIX EPIX A EK BKR ITEI NO Conditions Prior to Event Unit 1 - MODE 1 at 100% power Unit 2 - MODE 1 at 100% power
Description of Event
On the morning of November 8, 2005, Donald C. Cook Nuclear Plant (CNP) Unit 2 was in steady-state operation at 100 percent power. All times included in this event report are approximate. At 0305 hours0.00353 days <br />0.0847 hours <br />5.042989e-4 weeks <br />1.160525e-4 months <br />, the 2CD Emergency Diesel Generator (EDG)[EK] was declared inoperable for scheduled maintenance. The 2CD EDG supports Train A equipment; CD refers to C bus and D bus. The 2AB EDG, which supports Train B, A and B 4 kilovolt (kV) safety buses, was confirmed to be operable at that time. [ED,EF] At 0355 hours0.00411 days <br />0.0986 hours <br />5.869709e-4 weeks <br />1.350775e-4 months <br />, perturbations were experienced in main generator and plant bus electrical parameters and operators observed reactor coolant pump (RCP) motor current oscillations and indications of low voltage on all four 600-volt AC safety buses [ED]. Control room operators contacted the transmission distribution center to determine if the perturbations were the result of a grid disturbance. No grid disturbances were reported.
At 0358 hours0.00414 days <br />0.0994 hours <br />5.919312e-4 weeks <br />1.36219e-4 months <br />, an RCP bus [EA] undervoltage reactor trip signal was received and the Unit 2 reactor and main turbine tripped. At the time of the trip, reactive loading spiked greater than 600 mega volt amps, indicating that the main generator was motoring at that time. Arcing was observed by a security officer in the vicinity of the Unit 2 Main Generator Exciter (2-0ME-81-EXC)[TL] at the approximate time the unit tripped.
Following the trip, the RCP buses automatically transferred to the reserve feed supply [EB], as designed. However, the low voltage conditions on the safety buses had resulted in the automatic EDG start and actuation of load shed for the A and B 4kV safety buses. Because the 2CD EDG output breakers were in Lockout status due to maintenance, the load shed function was blocked and did not occur for that train, and the C and D 4kV safety buses remained energized from the reserve feed supply via the RCP buses.
When the loads were shed from the A and B 4kV safety buses, the Unit 2 west (W) centrifugal charging pump (CCP) [CB], which had been in Service, lost power and because no charging pumps were running, letdown [CB] flow from the reactor coolant system (RCS) automatically isolated. The operating Unit 2 W essential service water (ESW) [131] pump lost power and the cross-tied Unit 1 East (E) ESW pump automatically started due to low pressure on the associated ESW header. The T21A, A, 4kV safety bus was re-energized automatically following the start of the 2AB EDG; however, the breaker from the diesel generator to the T21B, B, 4kV safety bus failed to close, and the B bus remained de-energized.
At approximately 0406 hours0.0047 days <br />0.113 hours <br />6.712963e-4 weeks <br />1.54483e-4 months <br />, operators started the Unit 2 E CCP and restored normal RCS makeup [CB] and letdown flow from the RCS.
At 0510 hours0.0059 days <br />0.142 hours <br />8.43254e-4 weeks <br />1.94055e-4 months <br />, the 2AB EDG output breaker providing power to the A 4kV bus (T21A11) unexpectedly opened. The breaker re-closed without manual action after 23 seconds. When the breaker opened, a load sequence signal on the T21A bus was initiated and all loads re-sequenced onto the bus.
The 2CD EDG was restored to operable status at 0606 hours0.00701 days <br />0.168 hours <br />0.001 weeks <br />2.30583e-4 months <br />. Power was restored to bus T21B from reserve feed at approximately 1100 hours0.0127 days <br />0.306 hours <br />0.00182 weeks <br />4.1855e-4 months <br />. The 2AB EDG was restored to operable status on November 10, 2005, at 0222 hours0.00257 days <br />0.0617 hours <br />3.670635e-4 weeks <br />8.4471e-5 months <br />.
Electrical perturbations seen three minutes prior to the trip are consistent with degradation of the exciter field brushes, culminating in a loss of excitation. As a result of the diminished exciter field, the unit output voltage and the 4kV voltage degraded and reached the UV protection relay [27] setpoint, actuating the reactor trip [JC,JD] and EDG [EK] automatic start and load shed signals.
Buses T21A and T21B comprise the B train, 4kV safety-related equipment circuits at CNP. They are supplied by the 2AB EDG via output breakers T21A11 and T21B4, respectively. Buses T21C and T21D comprise the A train 4kV safety-related equipment circuits. They are supplied by the 2CD EDG. Both A and B trains, and all 4 buses, would normally be energized from their associated EDG following a UV signal. The 2AB EDG started and energized the T21A bus as expected, supplying power to support the essential safety equipment associated with the B train. However, the T21B bus did not energize as expected. The T21B bus provides power to the associated 600V safety related bus 21B [ED]. Important loads include one train of non-essential service water [KG] and EDG room cooling [VJ]. The cause of T21B bus not re-energizing as expected was closing circuit failure of breaker T21B4, 2AB EDG output breaker. The A train safety buses transferred to reserve feed as the 2CD EDG was out of service for scheduled maintenance, and was under clearance at the time of the event.
Both motor driven auxiliary feedwater [BA] pumps and the turbine driven auxiliary feedwater pump started as a result of this event. The auxiliary feedwater system functioned as expected for the plant conditions.
This report is made in accordance with 10 CFR 50.73(a)(2)(iv)(A), "any event or condition that results in manual or automatic actuation of any systems listed in paragraph (a)(2)(iv)(B)," specifically reactor trip, auxiliary feedwater, and emergency power system actuations.
A report is also required due to failure to implement the requirements of Technical Specification (TS) 3.8.1, Electrical Power Systems, Alternating Current (AC) Sources —Operating, per 10 CFR 50.73(a)(2)(i)(B), "any operation or condition which was prohibited by the plant's technical specifications.
TS 3.8.1 requires two diesel generators to be operable. However, the 2AB EDG was inoperable from October 27, 2005, when a modification was installed which inadvertently affected the operation of the T21B output breaker, until repairs were completed on November 10, 2005. Because the impact to the operation of the T21B breaker was not recognized until the demand following the reactor trip, the TS actions for one EDG inoperable were not previously completed as required. Also, there were two instances during that same timeframe when the 2CD EDG had been intentionally removed from service. Because the impact to the operation of T21B had not been discovered, the TS actions for two EDGs inoperable were not completed as required. TS compliance was achieved when the 2CD EDG was returned to operable status on November 1, 2005, at 0606 hours0.00701 days <br />0.168 hours <br />0.001 weeks <br />2.30583e-4 months <br />.
This event is also being reported per 10 CFR 50.73(a)(2)(v)(A), "Any event or condition that could have prevented the fulfillment of the safety function of structures or systems that are needed ... to shut down the reactor and maintain it in a safe shutdown condition ..." The failure of the 2AB EDG output breaker T21B concurrent with the 2CD EDG being out of service represents a loss of the onsite emergency AC power safety function. This condition existed between the period of time the modification was installed on October 27, 2005, and November 8, 2005, when TS compliance was restored following the reactor trip and recovery of the 2CD EDG. The specific time that both EDGs were inoperable simultaneously was October 31, 2005, at 2258 hours0.0261 days <br />0.627 hours <br />0.00373 weeks <br />8.59169e-4 months <br /> through November 1, 2005, at 1125 hours0.013 days <br />0.313 hours <br />0.00186 weeks <br />4.280625e-4 months <br />. Both EDGs were again inoperable simultaneously on November 8, 2005, between 0305 hours0.00353 days <br />0.0847 hours <br />5.042989e-4 weeks <br />1.160525e-4 months <br /> and 0606 hours0.00701 days <br />0.168 hours <br />0.001 weeks <br />2.30583e-4 months <br />, and between 2052 hours0.0238 days <br />0.57 hours <br />0.00339 weeks <br />7.80786e-4 months <br /> and 2146 hours0.0248 days <br />0.596 hours <br />0.00355 weeks <br />8.16553e-4 months <br />. This is a total of 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> 22 minutes.
Root Cause of Event The root cause of the reactor trip was that the preventive maintenance program failed to provide adequate preventive maintenance on the main generator exciter brushes; specifically, periodic thermography, brush holder inspections, and brush holder replacements were not performed.
The main generator exciter field was lost when contact between the exciter negative slip ring and associated brushes failed. The failure appears to have been caused by an exciter brush not making contact with the slip ring due to binding in combination with a brush that was too short to maintain contact with the slip ring. The bound brush appears to have existed in that condition for a period of several months. The preventive maintenance program did not contain periodic tasks to detect bound brushes. When another exciter brush became worn to the point that it no longer made contact with the slip ring, the increased current carried by the other brushes contributed to subsequent failures of the remaining brushes. Methods to detect bound exciter brushes include thermography and physical removal of the brush holder from the exciter. These detection methods had not been incorporated in the periodic preventive maintenance tasks. In addition, the preventive maintenance tasks did not include periodic replacement of the brush holders to ensure that dimensional or condition changes in the holders did not contribute to brush binding.
Cause of Output Breaker Malfunctions T21A11 Breaker The output breaker anomalies which became evident during this event were evaluated to determine the cause of the failure. Breaker T21A11 cycled open unexpectedly. The cause of this unexpected occurrence was due to a failure of the 2-62-2X-DGAB, AB EDG Trip Control Auxiliary Time Delay Relay. Evaluation of the relay found that the cause of the failure was chlorine-induced corrosion of the relay coil as a result of contamination introduced during the manufacturing process.
T21B4 Breaker The T21B4 output breaker failed to close automatically. The root cause was determined to be an inadequate crimp of the old style lug on the wire terminated at Test Switch 2, Knife Switch 3, on panel SA. The inadequate crimp was a legacy human performance failure existing from initial installation.
The point when the crimp failed to provide adequate continuity to support its ftinction was apparently October 27, 2005, when a modification, 12-MOD-45617, "Addition of Supplemental Diesel Generators (SDGs) for EDG Allowed Outage Time Extension," installed two wires associated with T21B4. A tie-wrap was installed in the immediate vicinity of the improperly crimped lug to secure one of the new wires to the existing bundle. This activity apparently flexed the existing wire to the extent that the conductor at the improperly crimped lug no longer made electrical contact with the lug. This disabled a portion of the T21B4 closing circuitry that was designed to automatically close the breaker following a loss of power to the T21B bus.
Analysis of Event
With the exception of the T21B4 and T21A11 breaker (Safety-Related Train B distribution system) malfunctions, the reactor trip did not challenge overall plant safety functions. While the trip originated from degraded voltage on the RCP (non-safety related) buses, RCS forced circulation and operation of secondary systems continued without interruption following the trip. Auxiliary feedwater, decay heat removal and emergency AC power to the T21C and T21D buses (Safety Related Train A distribution system) were available at all times. The operating crew responded to the trip and subsequent complications using approved operating procedures.
The risk impact of the event was analyzed using Safety Monitor and the most current probabilistic risk assessment model considering the effect of both the T21B4 breaker and CD EDG unavailability. The analysis was calculated for the period October 27, 2005, through November 10, 2005. This period encompassed both the reactor trip and the entire time that the 2AB EDG output breaker was inoperable. Two cases were evaluated. In one case, breaker T21B4 was considered completely failed. The other case credited the ability of the operator to manually close T21B4. The total incremental core damage probability values for these two cases were estimated to be 5.52E-07 and 3.95E-07, respectively.
There was no actual industrial, radiological, or personal safety impact as a result of this event.
Corrective Actions
Immediate Corrective Actions to Address Exciter Brush Failure
- The exciter brushes and brush holders were replaced, the slip ring was repaired, and the exciter brush housing was cleaned (Job Order Activity (JOA) 05312011-01,-12).
RT #22164).
Corrective Action to Prevent Recurrence of the Reactor Trip Caused by Inadequate Preventive Maintenance
- Create new RTs for Unit 1 and Unit 2 Main Generator Exciter brush maintenance to incorporate thermography, brush holder inspections, and brush holder replacements into the preventive maintenance program (Condition Report (CR) 05312013-17, -18, due 01/10/2006).
Corrective Actions to Address Extent of Condition
- Components which could create a trip/transient condition upon failure will be identified and scheduled for detailed review of related preventive maintenance practices (CR 05312013-03, due 02/06/2006). Considerations during the review of the maintenance activity are adequacy of the basis, frequency, planning, and implementation. Additional actions will be created to correct deficiencies if identified from the review.
Corrective Actions to Address Diesel Generator Output Breaker T21A11 and Associated Agastat Relay
- The installed T21A11 breaker was replaced (JOA 05312006-04).
- Agastat relay 2-62-2X-DGAB was replaced. (JOA 05312006-11).
- A review will be performed to determine an alternative replacement relay and to determine a replacement schedule of the Agastat series 9412 relays (CR 05312013-11, due 03/30/2006).
Corrective Actions to Address Diesel Generator Output Breaker T21B4
- Replaced the lug on the wire terminated at Test Switch 2, Knife Switch 3, on panel SA (JOA 05312006-04).
- A visual verification of all lugs associated with knife switches for Unit 1 and Unit 2 that could have been disturbed during the installation of modification 12-MOD-45617 was completed (JOA 05312006-17).
The root cause investigation for this event is currently under station management review. Any substantive changes to the root cause or corrective actions to prevent recurrence will be reported in a supplemental LER.
Previous Similar Events
There have been no similar events at CNP.