05000261/LER-2010-002
| Docket Number | |
| Event date: | 03-28-2010 |
|---|---|
| Report date: | 05-27-2010 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| 2612010002R00 - NRC Website | |
I. DESCRIPTION OF EVENT
At 18:52 EDT on March 28, 2010, with the H. B. Robinson Steam Electric Plant (HBRSEP), Unit No. 2, operating in Mode 1 at approximately 99.5% power, a feeder cable [CBL] failure to 4kV non-vital Bus 5 [EA:BU] caused an arc flash and fire. Four kV Bus 5 failed to isolate from non-vital 4kV Bus 4 [EA:BU] due to a failure of circuit breaker 52/24 [EA:BKR], which resulted in reduced power to 'B' Reactor Coolant Pump (RCP) [AB:P] and a subsequent reactor trip on Reactor Coolant System (RCS) [AB] loop low flow. Subsequent to the reactor trip, an automatic safety injection occurred due to RCS cooldown. Plant response was complicated by equipment malfunctions and failure of the operating crew to diagnose plant conditions and properly control the plant. During restoration of plant systems, operators reset the main generator [TB] lockout relay [86] which re-initiated the electrical fault and caused a second fire event that resulted in an Alert declaration and activation of the emergency response facilities due to a fire that affected safety-related equipment.
II. CAUSE OF EVENT
The feeder cable failure from non-vital 4kV Bus 4 to non-vital 4kV Bus 5 initiated the sequence of events on March 28, 2010. The failure of circuit breaker 52/24 to open and isolate 4kV Bus 4 is the root cause of the fire event, which also resulted in a reactor trip with complications. The circuit breaker failed to open because of a mechanical flaw in the trip circuit fuse [EI:FU], which disabled the control power trip circuit. The fuse was found to be mechanically failed, which caused the red indication light on the front panel of the breaker cubicle to be extinguished. This condition was identified shortly after HBRSEP, Unit No. 2, exited from Refueling Outage 25 in November 2008. The Work Request (WR) written to repair this deficiency failed to receive the proper technical rigor in the work process evaluation. As such, personnel failed to recognize that not having the red indication light may result in removal of the circuit breaker 52/24 protective trip feature. The evaluation resulted in incorrect work prioritization and ineffective corrective action prior to the event.
During plant stabilization and restoration, the operating crew failed to recognize the condition of the electrical distribution system prior to attempting to reset the main generator lockout relay, which resulted in re-energizing the faulted 4kV system. The root cause of this aspect of the event is a lack of monitoring of key plant parameters by the operating crew.
III. ANALYSIS OF EVENT
The event described in this Licensee Event Report is reportable under 10 CFR 50.73(a)(2)(iv)(A), any event or condition that resulted in manual or automatic actuation of any of the systems listed in 10 CFR 50.73(a)(2)(iv)(B).
This event was investigated in accordance with the HBRSEP, Unit No. 2, Corrective Action Program (CAP) and documented in Significant Adverse Nuclear Condition Report 390095. This reportable event, and the associated significant adverse condition investigation, were reviewed by the Plant Nuclear Safety Committee on May 11, 2010.
The following figure illustrates the electrical distribution system:
4kV 480V 125VDC 120VAC Figure 1 - Simplified Electrical Distribution The following safety systems actuated and performed normally during the event: Reactor Protection, Safety Injection, Emergency Diesels, Auxiliary Feedwater, and Main Steam Isolation (on Instrument Bus 3 power loss).
1. Initial Electrical Fault At 18:52 EDT on March 28, 2010, a feeder cable to 4kV non-vital Bus 5 failed.
Specifically, cable insulation on the feeder cable failed at the 4kV Bus 5 cabinet entry point, resulting in an arc flash and cable fire in the conduit. When the fault occurred, circuit breaker 52/24 did not clear the fault as expected, and the breaker remained closed for the duration of the event. The circuit breaker failed to open because of a mechanical flaw in the trip circuit fuse, which disabled the control power to the trip circuit. Failure of the breaker to open led to internal damage to the Unit Auxiliary Transformer (UAT) [XFMR] and a lockout of the UAT on fault pressure that caused the main generator lockout relay (86P) to operate.
As a result of circuit breaker 52/24 failing to open, the fault persisted on 4kV Buses 4 and 5 as the time over-current protection for 4kV Bus 4 feeder circuit breaker 52/20 [BU:BKR] began timing. The voltage for 4kV Buses 4 and 5 became significantly depressed due to the fault, and the 'B' RCP motor supplied by 4kV Bus 4 slowed, actuating the low RCS flow reactor protection logic for the 'B' RCS loop, which tripped the reactor. The fault current was initially fed from the UAT. After three to four seconds, the internal failure of the UAT tripped the fault pressure protection, locking out the UAT and also the main generator. The fast transfer from the main generator lockout signal opened breaker 52/20 and closed breaker 52/19 [BU:BKR], transferring the fault from the UAT to the Start Up Transformer (SUT) [XFMR]. Following transfer of the fault to the SUT, voltage for 4kV Bus 3 [BU] became significantly depressed, resulting in actuation of the loss-of-voltage relays for the 480V E-2 safety bus [BU]. The 480V Bus E-2 then separated from 4 kV Bus 3, the 'B' Emergency Diesel Generator (EDG) automatically started and connected to the 480V E-2 Bus, and the load sequencer operated as designed.
After several seconds, the time over-current relays for circuit breaker 52/19 actuated and tripped the circuit breaker, clearing the fault and ending the first electrical fault event. Plant electrical conditions following the termination of the first event left 4kV Buses 1, 2, and 3 powered from the SUT, 480V Bus E-2 powered from the 'B' EDG, Station Service Transformer (SST) 2C [XFMR] de-energized, and 4kV Bus 4 and Bus 5 de- energized. The entire sequence of protective automatic actions occurred in approximately 20 seconds.
2. Operator Actions During Initial Electrical Fault The initial fault on 4kv Bus 5 also resulted in electrical fires at 4kV Bus 5 and at Breaker 52/24 on 4kV Bus 4. The fires were extinguished by fire brigade and security personnel using dry chemical fire extinguishers. Damage to 4kV Bus 4 was initially limited to the 52/24 cubicle. Damage to 4kV Bus 5 was limited to the incoming line compartment along with the feeder cables.
Following the reactor trip and the loss of power to electrical buses described above, Instrument Bus 4 [BU] also temporarily lost power. The loss of Instrument Bus 4 caused flow control valve, FCV-626, the component cooling water thermal barrier return, to close, which isolated cooling to the RCP thermal barrier. This condition went undetected by the operating crew for a period of approximately 39 minutes. The power loss in the plant also prevented remote closing of the moisture separator reheater (MSR) steam shut-off valves and opened the MSR drain tank alternate dumps to the condenser, resulting in an RCS cool down. The RCS cool down led to an automatic Safety Injection (SI) due to low pressurizer pressure. Station expectations are that operators should manually initiate SI if the automatic setpoint is being approached, but the operators were not closely monitoring pressurizer pressure or level.
Following the safety injection, the operating crew did not take appropriate actions to control RCS temperature, which resulted in an initial cooldown rate in excess of the 100 degrees Fahrenheit/hour Technical Specifications limit. Main steam isolation valves (MSIV) closed due to an Instrument Bus 3 power loss that occurred during restoration of the 'B' battery charger to the 'B' battery; the 'B' battery charger was automatically secured when 480V Bus E-2 de-energized. The instrument bus power loss actuated the high steam flow coincident with low Tave logic, which caused automatic closure of the MSIVs.
The closure of the MSIVs stopped the RCS cool down. RCS temperature had decreased to approximately 442 degrees Fahrenheit (105 degree Fahrenheit RCS cool down). Due to the cool down and subsequent depressurization of the RCS, the SI system injected to the RCS for approximately 12 minutes with a maximum flow of 260 gpm.
During the event, the expected automatic actions on low Chemical and Volume Control System Volume Control Tank (VCT) level to realign charging pump suction to the refueling water storage tank did not occur, due to an improper configuration of a level comparator. This condition went undetected by the operating crew for a period of approximately 49 minutes. Review of plant indications revealed that the charging system was no longer delivering flow to the RCS or reactor coolant pump seals after approximately 37 minutes. Seal cooling was maintained through manual operator action to re-open FCV-626.
3. � Second Electrical Fault The operating crew properly transitioned through the emergency operating procedures.
Following safety injection termination and plant stabilization, a transition was made to procedure GP-004, Post-Trip Stabilization.
At approximately 22:34 hours, during performance of GP-004, the operating crew attempted to reset the generator lockout relay (86P), based upon procedural guidance in GP-004.
The operating crew was not aware of the locked-in UAT Fault Pressure signal, even though a main control board annunciator was available to provide that indication. When the attempt was made to reset the 86P lockout relay, the UAT fault pressure trip signal would not allow the logic to be reset. The lockout relay would not latch into the reset position and was released back to its tripped position by the operator. Upon release of the 86P relay handle, the circuit breaker 52/19 anti-pump logic was reset, arming the breaker to reclose again. With the UAT fault pressure trip input to the generator lockout relay still locked in, the fast transfer contacts reclosed breaker 52/19. When circuit breaker 52/19 reclosed, it repowered 4kV Bus 4 and re-energized the faulted cable and breaker compartment from the SUT.
A fault occurred when circuit breaker 52/19 was closed. The fault lasted only a short amount of time, insufficient to trip the circuit breaker
- 52/19. Current was still flowing to the faulted components, but was not sufficient to trip the protective devices. Approximately 3 minutes later, a condition in the rear of the circuit breaker 52/24 cubicle caused a significant fault and resulted in an arc flash. The arc flash breached the rear of the circuit breaker 52/24 cubicle and caused damage to surrounding.
components. Circuit breaker 52/19 opened to clear the fault and isolated the faulted components.
A second fire was initiated from the arc flash. This fire was extinguished by the Fire Brigade using dry chemical fire extinguishers. Following the second event, the electrical conditions in the plant were: 4kV Buses [BU] 1, 2, and 3 powered from the SUT, 480V Bus E-2 powered from the 'B' EDG, Station Service Transformer [XFMR] (SST) 2C de-energized, and 4kV Bus 4 and 4kV Bus 5 de-energized.
Following the arc flash and fire due to the attempted reset of the generator lockout relay, both safety-related 125-VDC battery Buses [BU] developed electrical grounds.
At this point, the operating crew declared an Alert, based on a fire that was affecting safety-related equipment, and activated the emergency response organization.
4. O Operator Actions During Second Electrical Fault During plant stabilization and restoration, the operating crew failed to recognize the condition of the electrical distribution system prior to attempting to reset the main generator lockout relay, which resulted in re-energizing the faulted 4kV system. The crew did not identify a control room annunciator associated with the UAT that was locked in from the initial fault. The annunciator response procedure provides the inputs to the generator lockout relay and automatic actions associated with the UAT fault signal.
However, this annunciator response procedure was not referenced by the operating crew.
The root cause of this aspect of the event is a lack of monitoring of key plant parameters by the operating crew.
5. � Feeder Cable Failure In 1985, HBRSEP, Unit No. 2, issued design change modification (MOD) 851 to install a new 4kV switchgear lineup to expand the existing switchgear for the purposes of supplying power for the new Radwaste Facility and Crystallizer. The new switchgear would require the re-routing of the 'C' Circulating Water Pump (CWP) cable to the new 4kV switchgear, while using the existing feeder breaker for the 'C' CWP in 4kV Bus 4 to supply the new switchgear lineup. MOD 851 was installed in 1986.
The power feeder cable from 4kV Bus 4 to the new 4kV Bus 5 was specified in MOD 851 to be a parallel run of 500MCM cable rated for 5kV. The cables were installed in two 4-inch rigid steel conduit runs from 4kV Bus 4 to the new 4kV Bus 5. The Bill of Material (BOM) for this MOD required that this cable meet the specification requirements contained in specification L2-E-035. Notable portions of this specification are the requirement for 133% insulation level and the need for an overall shield to be supplied with the cable. In addition, the cable was required by the original specification to meet IEEE-383 flame test requirements to limit flame propagation in the plant. The specifications indicate that these requirements may be waived by the engineer in the purchase order. The original purchase order could not be retrieved; therefore, verification that these requirements had been waived could not be performed.
Review of the feeder cable and MOD 851 following the event identified the following deficiencies:
a. The original MOD called for 5kV shielded cable meeting Specification L2-E-035. Design Change Number 6, approved on February 14, 1986, changed the cable to an unshielded cable that did not meet the requirements of L2-E-035 for insulation rating or shielding. � Technical justification for this change was not located with the copy of MOD 851 retrieved from records.
b. The installation of the cable was contrary to the acceptable installations described in the manufacturer's datasheet for the cable. The manufacturer's datasheet stated that installation of three conductors of this cable in a conduit was permissible only in non-magnetic conduit. The installation at HBRSEP, Unit No. 2, was in rigid steel conduit, which is magnetic.
c. Forensic evaluation identified evidence of partial discharge on the section of cable removed from the north conduit at the point where it entered 4kV Bus 5. A contributing cause of the partial discharge was the lack of strain relief on the cable as it entered the switchgear. There was no support on the incoming cable before the termination points at the bottom of the switchgear. This hastened the insulation degradation of the cable. The insulation level of the cable (100% vs.
133%) and lack of an outer jacket may have contributed to the degradation of the cable insulation system. The same cables installed in other locations showed similar signs of degradation. Based upon the visual inspection of 4kV Bus 4 and a review of modifications and warehouse stock, this is not a systemic problem at HBRSEP, Unit No. 2.
6. Fuse Failure The failure of circuit breaker 52/24 to open and isolate 4kV Bus 4 is the cause of the event, which also resulted in a reactor trip with complications. The circuit breaker failed to open because of a mechanical flaw in the trip circuit fuse, which disabled the control power trip circuit. The fuse was found to be mechanically failed, which caused the red indication light on the front panel of the breaker cubicle to be extinguished. This condition was identified shortly after HBRSEP, Unit No. 2, exited from Refueling Outage 25 in November 2008. A WR written to repair this deficiency failed to receive the proper technical rigor in the work process evaluation. As such, personnel failed to recognize that not having the red indication light may result in removal of the circuit breaker 52/24 protective trip feature. The evaluation resulted in incorrect work prioritization and ineffective corrective action prior to the event.
7. Failure of Charging Pump Suction Automatic Realignment When the VCT reached a predefined level, the charging pump suction should have automatically realigned to the Refueling Water Storage Tank. Plant indications show that this automatic action did not occur. � Subsequent troubleshooting identified that the control module LC-112B was not properly configured and was not functioning properly.
The VCT level comparator was upgraded during Refueling Outage 25 (in Fall 2008), replacing a Hagan style comparator with an NUS type comparator. An error was made during the development of the comparator jumper configuration sheet that was used in the upgrade work order instructions that resulted in the control module improper configuration. In addition, the testing performed (including the post-maintenance testing) in the calibration procedure did not detect this condition or functionally test the entire control loop circuit.
8. RCS Cooldown The cool down following the trip was a decrease in the average RCS temperature from 547 degrees Fahrenheit to 443 degrees Fahrenheit in 32.5 minutes, rebounding to 468 degrees Fahrenheit 22.5 minutes later and continuing to rebound beyond that. The Technical Specification (TS) limitations for cool down rates are described in the TS Bases as, "This rate shall not exceed 100 degrees Fahrenheit/hr in any one hour period." One hour following the RCS temperature reaching 547 degrees Fahrenheit, the average RCS temperature was approximately 468 degrees Fahrenheit, a decrease of 79 degrees Fahrenheit, and within the 100 degrees Fahrenheit/hr in any one hour period requirement.
Also, the decrease of approximately 104 degrees Fahrenheit is considered a minor deviation that was corrected within the 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> period. The pressure and temperature surveillance requirements (SR) provided in the TS Bases paragraph SR 3.4.3.1 confirms the use of the one hour increment to judge the change in temperature. It states:
"Verification that operation is within the limits of Figures 3.4.3-1 and 3.4.3-2 is required every 30 minutes when RCS pressure and temperature conditions are undergoing planned changes. This Frequency is considered reasonable in view of the control room indication available to monitor RCS status. Also, since temperature rate of change limits are specified in hourly increments, 30 minutes permits assessment and correction for minor deviations within a reasonable time.
The one hour surveillance increment is predicated on lagging response time of the vessel to rapid changes in the RCS water temperatures and is consistent with the understanding within the industry of how the heat up and cool down rates are applied and described in the TS Bases. Consequently, the transient the reactor vessel experienced is bounded by the design basis transients.
The potential for exceeding the maximum cool down rate was not identified by the operating crew, and thus ITS LCO 3.4.3 was not properly evaluated and considered for applicability at the time of the transient conditions.
The shutdown reactivity at RCS cool down conditions is demonstrated to be in excess of the maximum required Mode 3 SDM (1770 pcm, per the Core Operating Limits Report FMP-001).
Therefore, sufficient shutdown margin was demonstrated throughout the RCS cool down.
IV. SAFETY SIGNIFICANCE
The initial cable fault had very low safety significance, as it affected only non-safety related equipment. The failure of circuit breaker 52/24 to clear the fault expanded the zone of influence to include additional electrical busses and the offsite power supply through the SUT. This resulted in a reduction of flow in one loop of the RCS, a rapid cooldown of the RCS and the loss of offsite power to one emergency bus. An automatic safety injection resulted from the cooldown. An additional equipment failure of the VCT level comparator, along with the closure of the RCP thermal barrier isolation valve due to the loss of power (FCV-626), resulted in a challenge to RCP seal cooling. Operator identification and response to these conditions was not timely and appropriate, and created a potential challenge to the RCS pressure boundary. RCP seal integrity was maintained throughout the event; however, extended operation with degraded cooling could result in seal failure. Additional operator actions during plant stabilization re-energized the faulted electrical system, which resulted in an additional fire that affected two trains of safety-related DC power, meeting the criterion for an Alert declaration.
1. Updated Final Safety Report Analysis The plant response to the transient conditions caused by the electrical fault on March 28, 2010, have been evaluated and concluded to be within the design and licensing basis as described in the Updated Final Safety Analysis Report (UFSAR). The reactor and RCS remained within analyzed limits. The required safety systems automatically responded as required and in accordance with the design and licensing basis. The fire events that occurred did not result in damage beyond that evaluated by the UFSAR.
2. Charging Pump Suction Automatic Transfer Failure The reactor coolant charging pumps normal suction supply is the VCT, and when the VCT is nearing depletion there is an automatic transfer to the refueling water storage tank.
One of the significant functions of the reactor coolant charging is to supply RCP seal injection flow. The interruption of the seal injection flow removes one of two systems that provides a cooling mechanism for the RCP seals, the other being component cooling water cooling the RCP thermal barriers. If both RCP seal cooling methods are lost, an induced RCP seal loss of coolant event could occur. � In this risk evaluation potential initiating events and mitigation systems that could interrupt RCP seal cooling are considered. The operator actions taken during the event did not mitigate the risk impact associated with this failed function. The fire external event risk contribution is not negligible and is added to the risk from internal events only. The evaluation of additional risk increase due to other postulated external events (seismic and high winds) does not add significant risk to the impact of the identified failures. The resulting increase in risk is judged to potentially result in a low to moderate increase to the Robinson Core Damage Frequency and is judged to have a very small increase in Large Early Release Frequency.
The RCP parameter limits for normal seal leakoff temperature and bearing cooling water temperature were exceeded for the 'B' RCP for approximately 11 minutes. The 'B' RCP had stopped due to the loss of power at the onset of the event. The parameter limit for the labyrinth seal differential pressure was exceeded for the 'A' RCP for 12 minutes prior to the restoration of thermal barrier cooling. Although these conditions represent a potential challenge to seal integrity, it should be noted that seal integrity was not lost, and seal leakoff flows remained acceptable throughout the event.
V. CORRECTIVE ACTIONS
Completed Corrective Actions:
(1) An engineering evaluation was completed for the RCS cooldown. This evaluation demonstrated that the transient the reactor vessel experienced during the event was bounded by the design basis transients.
(2) A leadership evaluation for shift managers and control room supervisors was completed to assess leadership attributes and to ensure strong leadership standards and behaviors.
(3)Operating crews were reconstituted to improve performance.
Planned Corrective Actions:
The following actions will be implemented prior to restart from Refueling Outage 26, currently anticipated to be late June 2010:
(1) Work processes and procedures will be reviewed and revised to provide a more rigorous approach in work process evaluation, with particular regard to critical equipment-related work requests to eliminate single point vulnerability and ensure proper prioritization.
(2) Cable, conduit, breakers, fuses, and other damaged equipment involved in the event will be repaired or replaced. RCP seals will be inspected and repaired as necessary. The UAT transformer will be replaced.
(3) A health assessment of the SUT will be completed.
(4) The cable that initiated the event will be replaced with a type appropriate for the application. In addition, site 4kV cabling will be inspected and replaced as necessary. Engineering controls have been implemented since 1986 to improve engineering product quality to prevent this occurrence for future cable replacements.
(5) A modification will be completed to prevent inadvertent closure of FCV-626 on loss of power.
(6) A modification will be completed to ensure battery chargers will automatically restart on restoration of Motor Control Center 5 or 6 following a loss of offsite power.
(7) Troubleshooting of loss of Instrument Bus 3 will be completed and appropriate corrective actions implemented.
(8) The incorrect configuration of the VCT level comparator will be corrected and tested. Past NUS comparator replacement post-maintenance tests will be reviewed and additional post-maintenance tests conducted as necessary.
(9) Operating crews will be provided additional simulator training on proper procedure use and adherence during Emergency Operating Procedure and Abnormal Operating Procedure execution.
(10) Operator individual and crew performance will be evaluated using simulator scenarios and other evaluation methods, with particular focus on plant monitoring and command and control practices. Improvements identified will be incorporated in operator training.
(11) Operator training will be conducted on the operation of the main generator lockout relay including inputs and expected responses to resetting the relay.
(12) Procedures involved in the event, including applicable Abnormal and Emergency Operating Procedures, and GP-004, will be reviewed to identify enhancements and revised as necessary.
(13) Simulator modeling regarding equipment response during bus failures will be evaluated.
(14) Progress Energy will conduct a challenge to ensure pre-startup actions have been adequately completed prior to restart of the plant.
The following actions will be completed by August 31, 2010:
(1) A standard for post-maintenance testing of protection and control circuits will be developed.
(2) A configuration document for installing new NUS modules will be developed.
VI. ADDITIONAL INFORMATION
Failed. Component Information:
The failed cable was manufactured by the Rome Cable Corp. A specific model has not been identified, but it is listed as 500MCM 5kv unshielded cable.
The failed fuse was manufactured by the Bussmann Division of Cooper Industries. The model number was LPN-RK-30SP.
Previous Similar Events:
Licensee Event Reports (LERs) for HBRSEP, Unit No. 2, were reviewed. No events were identified that were similar to the event described in this LER.