05000454/LER-2012-001
| Byron Station | |
| Event date: | |
|---|---|
| Report date: | |
| 4542012001R01 - NRC Website | |
The Unit 1 and Unit 2 AC electrical system consists of two 4.16 kilovolt (kV) Engineered Safety Features (ESF) buses, four nonsafety-related 6.9 kV buses and two nonsafety-related 4.16 kV buses. The two 4.16 kV ESF buses and two of the nonsafety-related 6.9 kV buses normally are supplied by one of the two System Auxiliary Transformers (SATs) [EB] connected through one 345 kV offsite circuit. The remaining two nonsafety-related 6.9 kV station buses and two nonsafety-related 4.16 kV station buses normally are supplied by one of two Unit Auxiliary Transformers (UATs) [EL] when the main generator is online. The four Reactor Coolant Pumps (RCPs) are powered by the four 6.9 kV buses. Each 4.16 kV ESF bus has a dedicated standby emergency Diesel Generator (DG) [EK].
The 4.16 kV ESF buses have two low voltage detection relays. One relay senses phase to phase voltage on A to B phases and the other relay senses B to C phases. When the undervoltage setpoint is sensed by both of these relays, offsite power will be switched to onsite power by opening the SAT feed breakers and starting the emergency DGs (i.e., two-out-of-two logic).
The 6.9 kV buses also have two low voltage detection relays which generate a reactor trip signal on low voltage on two of the four buses. These relays also sense phase to phase voltage on A to B phases and B to C phases. However, only one relay is needed on two of the four busses to generate the reactor trip signal (i.e., one-out-of-two logic). In the event of an undervoltage on either of the SAT feed buses or UAT fed buses, an automatic bus switchover to the energized transformer occurs.
Event 1 Plant Conditions Prior to the Event Unit 2 was in Mode 1, Power Operations at approximately 100% power. Reactor Coolant System (RC) [AB] was at normal operating temperature and pressure. The following Unit 2 safety related equipment was running to support Unit operations:
- 2A Essential Service Water (SX) [BS] Pump
- 2B Centrifugal Charging (CV) [CB] Pump
- 2B Component Cooling (CC) [CC] Pump The 2B DG was near the end of a planned maintenance work window and was inoperable but available to run. One of the two auxiliary building floor drain (WF) sump pumps was out of service. No other structures, systems, or components were inoperable at the start of this event that contributed to the initiation or mitigation of this event.
Description of the Event
On January 30, 2012, at 1001:54 hours, a mechanical failure of a 345 kV under hung porcelain insulator on the SAT 242-1/2 A-Frame structure in the switchyard resulted in an open phase non-faulted condition on the SAT 242-1/2 C phase S-Bus lead between the A-Frame structure support insulator and the 345 kV Bus 13 disconnect switch (See figure 2). When the insulator failed, the S-bus fell causing both a mechanical and electrical failure at the terminal pad of the Bus 13 disconnect switch. The nature of the failure resulted in a sustained open phase event to the SAT 242-1/2 and a low level ground fault occurred on the SAT 242- 1/2 side of the open phase.
The 4.16 kV ESF buses undervoltage protection two-out-two logic was not satisfied since the phase A to B relay was sensing normal voltage due to phase angle differences. SAT 242-1/2 continued to provide power to their respective 4.16 kV and 6.9 kV buses. However, the single phase open circuit created an unbalanced voltage condition to the SAT 242-1/2. The two 6.9 kV buses energized by SAT 242-1/2 sensed the undervoltage on Phases B to C. This undervoltage on two of four RCP buses resulted in an automatic Unit 2 reactor trip at 1001:55. At 1001:57 hours, the 2A motor driven Auxiliary Feedwater (AF)[BA] pump and the 2B Diesel Driven AF pump automatically started from an RCP bus undervoltage, as designed. The 2A AF pump, receiving power from 4.16 kV ESF bus 241, tripped seconds later at 1002:01 hours due to a phase A overcurrent. The 2B AF pump remained running.
Equipment powered by the SAT 242-1/2 energized buses began tripping due to a Phase A overcurrent.
This included the 2B CV pump, 2A SX pump, and the 2A CC pump. The 2B CC pump received an automatic start signal from low suction pressure and it also tripped from phase A overcurrent.
The Main Control Room (MCR) Operations crew began responding to the automatic reactor trip using the appropriate emergency response procedures.
After approximately 30 seconds from the reactor trip, a reverse power Main Generator (MG) [EL] trip occurred, as expected, and opened the circuit breakers to the switchyard and UAT feed breakers. As designed, buses fed by the UAT automatically transferred to the SATs 242-1/2. The resultant current flow on SAT 242-1/2 phases A and B increased and caused all four RCPs to trip on overcurrent within 40 seconds after the bus transfer. The Unit 2 loss of offsite power resulted in a RC system natural circulation cooldown condition. Numerous control power and input voltage alarms were received and several pieces of equipment powered by 480 V buses began to trip due to activation of their thermal overload relays.
The Operations crew completed the immediate actions portion of the reactor trip emergency procedure.
The crew recognized the automatic bus transfer and that the equipment powered by the 6.9 kV and 4.16 kV buses were tripped. Although the Emergency Operating procedures did not address the situation, the crew utilized integrated plant training and experience to diagnose and recover from this configuration.
An effort to restart the 2A SX pump was attempted but was unsuccessful. At approximately 1004 hours0.0116 days <br />0.279 hours <br />0.00166 weeks <br />3.82022e-4 months <br />, equipment.
All three phases of the 4.16 kV ESF buses were then verified. There were no abnormal alarms associated these buses and the bus alive lights were lit. The 4.16 kV ESF bus voltage indicated phases A to B indicated normal; however the voltages for phases C to A and B to C indicated approximately 2.5 kV. An operator was dispatched to visually inspect the SATs 242-1/2 and reported seeing smoke in the vicinity.
At 1009:48 hours, based on the abnormal voltage indication and the report of smoke, the crew manually opened the SAT feed breakers to the 4.16 kV buses. This intentionally caused an undervoltage condition on all three phases resulting in both 2A and 2B DGs to start and energize their respective 4.16 kV ESF buses. All safe shutdown loads properly sequenced onto the buses, as designed. Due to the report of smoke at the SAT, offsite fire protection assistance was requested.
Subsequently, the remaining SATs 242-1/2 feed breakers to the nonsafety-related busses were opened.
The 4.16 kV nonsafety-related buses were then re-energized via their crosstie breaker to their respective 4.16kV ESF bus. Bus 13 from the switchyard to SATs 242-1/2 was then opened. The Operations crew declared an Unusual Event (UE) at 1018 for a loss of offsite power to essential buses for greater than 15 minutes. The crew entered Unit 2 Technical Specification 3.8.1, Conditions A and E for the loss of Unit 2's normal offsite power in conjunction with the 2B DG being inoperable at the time. Condition A requires the restoration of the operability to the offsite circuitry with 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> and Condition E requires the restoration of offsite circuit or DG to operable status in 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />, otherwise to be in Mode 5 in 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />. Subsequent to the event, on February 2, 2012, Operations recognized that Unit 2 was outside any allowed conditions for TS 3.8.9, "Distribution Systems-Operating," and therefore was in TS 3.0.3 for the eight minutes the 4.16 kV ESF buses were not energized by an operable power source.
The Operations crew continued with the reactor trip recovery and the preparations for the natural circulation cool down to Mode 5. At 1100 hours0.0127 days <br />0.306 hours <br />0.00182 weeks <br />4.1855e-4 months <br />, the crew transitioned to the natural circulation cooldown procedure. RCS cooldown was commenced at 1158 hours0.0134 days <br />0.322 hours <br />0.00191 weeks <br />4.40619e-4 months <br />, using the Steam Generator Power Operated Relief Valves and the AF system. RCS depressurization was initiated at 1300 using CV Auxiliary Spray.
Unit 2 Cold Shutdown conditions were reached at 0228 hours0.00264 days <br />0.0633 hours <br />3.769841e-4 weeks <br />8.6754e-5 months <br />, on January 31, 2012.
The failed insulator was replaced and electrical lines reconnected. An assessment of the SAT condition concluded it was acceptable to re-energize. The report of smoke was later determined as water steaming off the neutral resistor box and was considered a normal condition. At 1955 hours0.0226 days <br />0.543 hours <br />0.00323 weeks <br />7.438775e-4 months <br />, on January 31, 2012 (approximately 34 hours3.935185e-4 days <br />0.00944 hours <br />5.621693e-5 weeks <br />1.2937e-5 months <br /> after the insulator failure), the 2A and 2B Diesel Generators were secured and offsite power was restored to the 4.16 kV ESF busses. The UE was terminated at 2000 hours0.0231 days <br />0.556 hours <br />0.00331 weeks <br />7.61e-4 months <br />, on January 31, 2012. Unit 2 remained in Mode 5 to conduct repairs and assessments of operability and functionality of plant equipment to support the Unit re-start.
Event Challenges 1. Due to the initial tripping of the 2A CV pump and the 2B CC pump, the RCP seals experienced a loss of seal cooling for the approximately eight minutes the 4.16 kV ESF buses were degraded. An extended loss of CV Seal Injection and CC flow to the RCP seal thermal barrier can lead to seal damage and an excessive loss of reactor coolant condition. An evaluation was performed and concluded that no operational limits were exceeded and the RCP seals remained within their design limits.
2. After restoration of the 4.16.kV ESF buses and the re-start of the 2A CV pump, a letdown system RC filter clogged. This resulted in the Letdown Orifice Outlet Header Relief Valve (i.e., 2CV8117) relieving into the Pressurizer Relief Tank (PRT) which raised pressure in the PRT to approximately 78 psi. In addition, the plugged filter caused the Letdown Heat Exchanger Outlet Header Relief Valve (i.e., 2CV8119) to lift and relieve to the CV Volume Control Tank. This caused the inlet, outlet, and the bypass valves to the RC filter to leak. The CV filter was bypassed and changed from a 0.1 micron to a 1 micron filter. The severity of the CV filter clogging during this event was not expected and the issue has been placed in the Corrective Action Program (CAP) and is being evaluated.
3. Due to the natural circulation cooldown, the CV Auxiliary Spray Header Isolation Valve (i.e., 2CV8145) was used to provide pressurizer spray flow to support plant shutdown to Mode 5. During its use, a surveillance procedure was required to monitor the pressurizer spray water temperature differential every 30 minutes to verify the temperature differential did not exceed 320°F.
Subsequent to the event, a monthly engineering review of transient events identified that during the opening of 2CV8145, the temperature difference exceeded the 320°F limit two times for approximately 1 to 2 minutes. For both excursions the temperature differential did not exceed 340°F. The Operations crew monitoring during the event did not recognize this because it occurred in between the 30 minute frequency of the surveillance. In accordance with the Transient Monitoring Program the RCS is allowed ten 2CV8145 cycles in which the spray water temperature differential exceeds 320°F before an evaluation is necessary. Prior to this event Unit 2 had zero events however, this issue has been placed in the CAP to improve temperature differential monitoring.
4. During the CV letdown pressure transient resulting from plugging the CV filter, the gross fuel failure radiation monitor skid experienced a pressure spike that caused an isolation valve near the skid to fail and initiated external leaking. The leakage resulted in water flowing into the containment penetration area below and then through the joints around a floor plug into the 2A Residual Heat Removal (RH) [BP] pump room. The leakage did not challenge the design basis flood assumptions; however, it did have the potential to affect electrical equipment in the 2A RH pump room. Though the 2A RH pump was subsequently found to be unaffected by the water, it was an operator distraction during the event.
This floor plug and others that could expose ECCS equipment to water have been sealed with caulk.
5. The nonsafety-related makeup isolation valve to the CC surge tank fails open upon a loss of offsite power. Makeup water continued to fill the tank until it overflowed into the tank's berm and then into the auxiliary building floor drain (WF) system. The second sump pump tripped on overcurrent. With no sump pump functional, the CC surge tank leak and the leak from the radiation monitor skid valve resulted in the flooding of the WF sump room. The water was subsequently removed by temporary pumps.
6. Due to the degraded voltage condition, there were a number of 480 V components found with their protective Thermal Overloads (TOLs) tripped, as designed, due to the overcurrent conditions. This condition complicated recovery efforts since the TOLs needed to be reset prior to component use.
7. The 2B and 2C turbine driven non-safety related Feed Water (FW) [SJ] pumps tripped, as expected.
The main lube oil pumps tripped on loss of AC power, and the DC emergency oil pumps started. The System Engineer identified during post-transient walkdown that no oil was being supplied to the 2B FW pump, and even though the emergency oil pump was energized it was not producing oil pressure/flow.
Bearing high temperatures caused bearing damage, as a result. This condition has been placed in CAP for resolution.
NRC Reporting Summary On January 30, 2012, at 1039, in accordance with 10 CFR 50.72 (a)(1)(i), (b)(2)(iv)(B) and (b)(3)(iv(A), an Emergency Notification System (ENS) notification to the NRC was made for the UE declaration, the reactor trip and the safety system actuations. This ENS notification was later updated at 1218, to include reporting criteria of 10 CFR 50.72 (b)(2)(i), (b)(2)(xi) and (b)(3)(v)(D) for TS required shutdown, offsite notification and a loss of safety function. A subsequent review of the reportability criteria selected concluded that the TS required shutdown should not have been checked on the ENS notification since the Unit was already in Mode 3, Hot Standby, when the TS condition requiring a shutdown to Mode 5 was entered. NUREG 1022 guidance indicates this reporting criterion does not apply after entering Mode 3. Periodic ENS update calls were made for the duration of the event. On January 31, 2012, at 2119, an ENS notification was made for the UE termination.
On February 3, 2012, at 2210, a voluntary ENS notification was made to communicate the design vulnerability in the 4.16 kV ESF bus undervoltage protection scheme and that it may have generic applicability to the industry. As a clarification to the voluntary ENS notification, the analysis mentioned in the report that was assessing the safety significance, was not a documented written analysis or assessment from the vendor supporting the plant modeling, but rather initial engineering judgment from the input. Further deterministic engineering judgment of the safety significance of this design vulnerability does not support the initial judgment and is being formally evaluated to make a final determination. In addition, this LER is submitted in accordance with 10 CFR 50.73 (a)(2)(iv)(A), (a)(2)(v)(D), (a)(2)(vii) and (a)(2)(ix)(A).
Event 2 Plant Condition Prior to the Event Unit 1 was in Mode 1, Power Operations at approximately 100% power. Reactor Coolant System (RC) [AB] was at normal operating temperature and pressure. No structures, systems, or components were inoperable at the start of this event that contributed to the initiation or mitigation of this event.
Description of the Event
On February 28, 2012, at 1730 hours0.02 days <br />0.481 hours <br />0.00286 weeks <br />6.58265e-4 months <br />, Unit 1 experienced a loss of its normal offsite power circuit due to the A phase line to the Unit 1 SAT 141-1/2 opening and causing a short (i.e., faulted condition) in the switch yard. The 4.16 kV ESF buses undervoltage protection functioned, as designed. The SAT 141-1/2 feed breakers opened and the DGs started as expected and restored power to the 4.16 kV ESF buses.
The 2A AF pump started, as designed and supplied FW to the steam generators. The 2B Diesel Driven AF pump does not receive an automatic start signal with a 4.16 kV ESF bus undervoltage.
The SAT 141-1/2 feeds to the 6.9kV buses automatically transferred to the UATs and the SAT feeds to the 4.16 kV non-ESF buses automatically transferred to the UATs. The 1B DG was running as part of a planned monthly surveillance run and responded as expected to energize 4.16 kV ESF bus 142. The Unit remained online. Operations crew followed appropriate Abnormal Operating procedures covering this condition. At 1736, on February 28, 2012, the Operations crew declared an UE for a loss of offsite power to essential buses for greater than 15 minutes.
TS 3.8.1 Condition A was entered due to the loss of the normal offsite power source to the 4.16 kV ESF buses. The Operators took action to reduce reactor power slightly to accommodate the cooler AF injection into the SGs.
There were no significant complications with plant responses. At 2154 hours0.0249 days <br />0.598 hours <br />0.00356 weeks <br />8.19597e-4 months <br />, on February 28, 2012, the Unit 1 4.16 kV ESF buses were crosstied to its reserve offsite power circuit via the Unit crosstie breakers and the 1A and 1B DGs were secured.
At 1817 hours0.021 days <br />0.505 hours <br />0.003 weeks <br />6.913685e-4 months <br />, on February 29, 2012 (approximately 25 hours2.893519e-4 days <br />0.00694 hours <br />4.133598e-5 weeks <br />9.5125e-6 months <br /> after the insulator failure), repairs were completed and normal offsite power to the Unit 1 4.16 kV ESF buses was restored. The UE was terminated at 2100 hours0.0243 days <br />0.583 hours <br />0.00347 weeks <br />7.9905e-4 months <br /> on February 29, 2012. Unit 1 ESF buses were energized by their respective DGs for approximately five hours and then cross-tied to Unit 2 offsite power for approximately 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br />.
Unlike the January 30, 2012 Unit 2 event, the insulator failure resulted in a ground fault rather than a non faulted open circuit.
NRC Reporting Summary At 1803 hours0.0209 days <br />0.501 hours <br />0.00298 weeks <br />6.860415e-4 months <br />, on February 28 2012, in accordance with 10 CFR 50.72 (a)(1)(i), (b)(2)(xi), (b)(3)(iv)(A) and (b)(3)(v)(D) an Emergency Notification System (ENS) notification to the NRC was made for the UE declaration, offsite notification, system actuation and loss of safety function entered. A subsequent review of the reportability criteria selected concluded that the loss of safety function criteria should not have been checked, since it did not involve a complete loss of offsite power to Unit 1. The reserve offsite power source, via the Unit 2 SAT, was always operable during this event. On February 29, 2012, at 2122 hours0.0246 days <br />0.589 hours <br />0.00351 weeks <br />8.07421e-4 months <br />, an ENS notification was made for the UE termination. In addition, this LER is submitted in accordance with 10 CFR 50.73 (a)(2)(iv)(A).
Cause of the Events Event 1 The Unit 2 SAT-1/2 insulator failure was caused by service propagation of a large manufacturing material defect that covered approximately 40% of the fracture cross-section in one section of the insulator stack.
The defect was characterized as poorly vitrified porcelain, which contained a high density of porosity and micro-cracks.
Additionally design vulnerabilities existed in the protective relaying schemes regarding the lack of single open phase detection that complicated plant and operator response by not automatically isolating all three phases on the affected line.
Event 2 The Unit 1 SAT insulator failure was caused by service propagation of a large manufacturing material defect that covered approximately 25% of the fracture cross-section in one section of the insulator stack.
The defect was characterized as poorly vitrified porcelain, which contained a high density of porosity and micro-cracks. Moreover, a second insulator section, which fractured as a result of the fall, exhibited the same poor vitrification as did the section that initially fractured.
Corrective Actions
Following the January 30, 2012 event, corona and ultrasonic inspections were completed on all switchyard inverted insulators. The corona inspections found no discernable discharges on any of the insulators and the ultrasonic inspections were normal.
All Unit 1 and Unit 2 Ohio Brass inverted insulators associated with the SATs and MPTs were replaced with Lapp high strength insulators.
A plan is being developed to replace other Ohio Brass inverted insulators associated with the remaining lines in the switchyard.
A preventive maintenance program is being developed for the replaced insulators.
The adequacy of the undervoltage protection design was assessed in an operability evaluation for the loss of offsite power DG start instrumentation. It concluded that the current protection design is operable and that detection down to the level of this type of failure was beyond regulatory design requirements.
However, the design vulnerability will be addressed with a means to eliminate this vulnerability in single open phase detection scheme.
In the interim, compensatory measures have been taken to enhance diagnosing and responding to a similar event. These measures include enhance procedural direction to diagnose and recover from this condition, a Control Room alarm function for detecting an phase imbalance, and a designated operator to monitor 4.16 kV ESF bus voltage and to open the SAT-1/2 feed breakers when an open single phase condition is detected.
Safety Significance
Event 1 There were minimal actual safety consequences to this event. The reactor tripped, as designed and Operators were able to diagnose and restore power to the 4.16 kV ESF buses in approximately eight minutes. These actions restored RCP seal cooling prior to RCP seal degradation and RCS coolant loss.
The protective overloads for components functioned as designed to protect plant equipment. Operators responded satisfactorily to the various event challenges. RC system cooldown to Mode 5 was safely conducted using the Steam Generator Power Operated Relief Valves and the AF system.
An engineering analysis was completed on the potential safety significance of a SAT single open phase condition occurring coincident with a design basis LOCA. The conclusion of this analysis is that the degraded voltage relays on the 4.16kV safety buses would have isolated from offsite power, loaded the DGs onto the 4.16kV buses, and started the ESF loads via the load sequencers in approximately 90 seconds. This is in excess of current safety analysis timing requirements for the 4.16kV buses to be re energized and ESF loads re-energized following loss of off-site power coincident with a design basis LOCA. Accordingly, prior to the design vulnerability's compensatory measures that were implemented, a design basis event concurrent with this open phase condition would likely to have resulted in exceeding 10 CFR 50.46, "Acceptance Criteria for emergency core cooling systems for light-water nuclear power reactors," limits. A risk assessment was also conducted and concluded this event was of low to moderate safety significance. This event is considered an event or condition that could have prevented fulfillment of I a safety function.
Event 2 There were no actual safety consequences to this event. The 1A and 1B DGs and the Motor Driven AF Pump started, as designed. Unit 1 remained stable at full power. The AC electrical power sources are designed with sufficient redundancy to ensure the availability of necessary power to ESF Systems. The Unit 1 ESF buses could have been powered by their respective DGs or the Unit 2's ESF buses, which were capable of being powered by either their offsite sources or their respective DGs. This event is not considered an event or condition that could have prevented fulfillment of a safety function.
Previous Events [sic] Auxiliary Transformer from Water Intrusion," dated, May 23, 1996.
Start Resulting from a Loss of Offsite Power Due to a Failed Insulator Causing a Differential Phase Overcurrent," dated March 25, 2008.
A review of these LERs concluded that the causes and corrective actions taken would not have been expected to prevent these events.
Component Failure Data
Part: Insulator Manufacturer: Ohio Brass Model Number: Post Stacking Assembly — Inverted Taper Part Number: 0470173071 345 KV Switchyard From rer7itor Generator
UAT SAT UAT
241-1 242-2 241-2 Note 3 Yv` 6,9KV�4KV 4 KV 6 9KV )rm ) D X-TiTi BUS 243�BUS 241 BUS 242�BUS 244BUS 257 BUS 259 BUS 255�BUS 2584 KV 4KV 4KV 4KV6.9 KV (39KV 6,9 KVNon-ESF�ESF ES!:�Non-ESFNon-ESF Non-ESF�Non-ESF Notes 1) 4 KV�buses 4re powered by SAT rfrd FOG ESF ,I.6f6 6 through � 111A ,-ross-tie breakers 3) No LSF buses rry.,,),�3/tT�with Fast Bus 71
- ,:1":f':, fer scheme Figure 1 345 kV Lines to Other ComEd ComEd Switchyard at Byron Station Blue Bus Red Bus Bus 4� Bus 6 Bus 13� Bus 11 MPT 1 SAT 142 SAT 242 MPT 2 Fram -Frame A-Frame A-Frame To Byron Station SATs and MPTs Figure 2