05000323/LER-2001-001
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(ix)(A) |
| Initial Reporting | |
| 3232001001R01 - NRC Website | |
I. Plant Conditions
Units 1 and 2 have operated in various modes and at various power levels with the degraded wire conditions described below. Unit 2 was in Mode 1 (Power Operation) at 100 percent power during the time the offsite power systems and Diesel Generator (DG) 2-2 were inoperable and PG&E failed to enter Technical Specification (TS) 3.0.3.
II. Description of Problem
A. B Background Vital 4.16-kV Switchgear The vital 4.16-kV metal-clad switchgear, manufactured by General Electric, contains buses [BU], breakers [BKR], relays [RLY], control switches [HS], and control and indication wiring [WEL] for proper distribution of 4.16-kV electrical power to all engineered safety feature (ESF) equipment, including the emergency core cooling system pumps and vital 4.16-kV electrical feeder circuits [EB]. The breakers are required to close upon receipt of the appropriate signal to start the associated ESF pump or energize the bus. These breakers are required to open upon overcurrent conditions. They are also required to open upon a loss of offsite power and reclose upon restoration of power to the vital buses from the DGs [EK].
Each breaker is housed in its own cubicle. Access to each cubicle is through a hinged door. Mounted on the doors of the various cubicles are relays (for example, undervoltage and overcurrent) and various control switches (for example, transfer, control, and feature cut-out). The wiring to the door-mounted devices is routed across the hinge area from the front corner of the cubicle walls onto a door-mounted wire loom. The wire loom is a metal clamping device that spreads and holds the wires when the doors are opened and closed to minimize wear. The wire addressed in this LER is factory-installed #10 and #12 AWG, with Type TA insulation, NEMA Class K (high strand) copper wire. The stranded wire provides flexibility and minimizes strain-hardening of the wire as doors are opened and closed. In some cases the factory-installed wire has been replaced with newer wire as part of equipment upgrades other design changes.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION Diablo Canyon Unit 2 0 Diesel Generators Each unit has three DGs that supply power to the 4.16-kV vital buses (EB) whenever offsite power is either unavailable, or bus voltage degrades below a point at which required loads would become inoperable.
Offsite Power Supplies and Double Sequencing Two sources of offsite power are available to Diablo Canyon Power Plant (DCPP): the 500-kV system and the 230-kV startup power system. The DCPP main generator [EL] feeds the 500-kV system during normal operation and supplies power to the house loads, including the 4.16-kV vital buses, through the auxiliary transformers (XFMR). This source of power to plant loads is known as auxiliary power.
The 230-kV startup power system is a dedicated standby/startup power source that can provide power to the plant through the startup transformers (SUTs). This system is available as a backup source of power to the vital buses if auxiliary power is lost. The 230- to 12-kV SUT has a load tap changer (LTC) on the 12-kV side of the transformer that is operated either in automatic or manual. The LTC allows the 12-kV voltage to be maintained if 230-kV voltage fluctuates. The LTC is normally in automatic, and the secondary voltage is maintained at 107.5 percent of normal (when not supplying loads). This ensures that adequate voltage is maintained to house loads following an automatic transfer to startup power.
Vital buses can be manually transferred from auxiliary to startup power for operational or maintenance purposes. However, compensatory measures must first be taken, in accordance with Operating Procedure (OP) J-2:VIII, "Guidelines for Reliable Transmission Service for DCPP," to ensure continued operability of the startup power source. Compensatory measures include defeating the auto-start of the standby condensate/ condensate booster pump set and cutting out the auto-transfer feature of one 12-kV non-vital bus (either D or E). Each non-vital 12-kV bus carries one 13,000 horsepower circulating water pump and two 6,000 horsepower reactor coolant pumps. The procedure also requires that when operating the LTC in manual, that the LTC be placed in the 7, 8, or 9 tap position.
These measures ensure that the SUT maintains adequate secondary voltage to accommodate the loading that occurs upon automatic transfer from auxiliary power to startup power.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) Diablo Canyon Unit 2 0 OP J-6A:II, "Transferring 4160 Volt Banks," provides the sequence of steps to perform the actual manual transfer. The procedure requires that operators take manual control of the LTC on the 230- to 12-kV SUT, reduce output voltage to match the bus voltage, and then make the transfer. Unless a 12-kV bus, carrying the large reactor coolant pumps and a circulating water pump loads, is being supplied by startup power, the LTC is left in manual.
With a vital bus being supplied by startup power and the LTC in manual at a reduced voltage level, the startup power source may be inoperable.
This inoperability is due to the potential for a double sequencing event to occur. The sequence of events characterized as "double sequencing" is summarized as follows:
- A loss-of-coolant accident (LOCA) causes a safety injection signal resulting in automatic transfer of vital buses to startup power and auto start of all DGs,
- ESF loads sequence onto the vital buses,
- The main generator trips after 30 seconds,
- The non-vital 12- and 4-kV buses transfer to startup power, causing a large voltage drop on the startup bus,
- The decreased startup bus voltage causes the second-level-under voltage relays to actuate after a time delay,
- Load shed relays strip all loads from the vital buses, and the buses are transferred to the DGs, and
- ESF loads resequence onto the vital buses.
Because the ESF loads do not remain on the startup power system and are sequenced onto the DGs, the startup power system is considered to be inoperable.
Technical Specifications The Limiting Condition for Operation for TS 3.8.1 states:
The following AC electrical sources shall be OPERABLE:
a. Two qualified circuits between the offsite transmission network and the onsite Class 1 E AC Electrical Power Distribution System; and b. Three DGs capable of supplying the onsite Class lE power distribution subsystem(s) LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) Diablo Canyon Unit 2 0 Condition J of TS 3.8.1 requires plant personnel to immediately enter TS 3.0.3 when one or more DGs are inoperable coincident with two required offsite circuits inoperable.
B. � Event Description On March 14, 2001, during routine functional testing of relays associated with Containment Spray Pump (CSP) 2-2, PG&E personnel determined the pump breaker in Cubicle 52-HH-9 would not trip on an overcurrent condition. Inspection and initial trouble shooting identified a broken overcurrent trip wire between the hinged cubicle door and the interior of the cubicle. The broken wire was subsequently replaced and the breaker returned to service.
On March 16, 2001, PG&E established a plan to inspect the Unit 1 and Unit 2 4.16-kV vital switchgear cubicles for evidence of additional wire degradation. The plan called for inspections during the associated bus/train scheduled maintenance windows.
On March 21, 2001, PG&E began inspections of the breaker cubicles on both Units. No additional degraded wires were identified during inspection activities on that day.
Cubicle 52-HG-7 for CSP 1-1. One wire was identified to have exposed and broken strands, and a second wire was identified with damaged insulation. The wires were subsequently replaced.
On March 25, 2001, preparations began for transferring 4.16-kV Vital Bus H to startup power to inspect the auxiliary power feeder breaker. The prerequisite compensatory measures were taken, the LTC on the SUT 2-1 was placed in manual, the voltages were matched, and at 0142 PST, Vital Bus H was transferred to startup power. The LTC was left in manual, in the tap-4 position, because operating personnel believed that leaving it in a higher setting would cause an unacceptable voltage condition on Bus H.
On March 25, 2001, at 0143 PST, 4-kV Breaker 52-HH-13 (and thus the 500-kV offsite power system to Bus H) was declared inoperable for the inspection. At approximately 0205 PST, degraded wires affecting the load-shed function of breakers for Residual Heat Removal Pump 2-2 and CSP 2-2 were identified in the cubicle. The cubicle door was closed while corrective maintenance work orders were prepared. As the affected LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) Diablo Canyon Unit 2 0 pumps might not shed during a load-shedding event, DG 2-2 was conservatively declared inoperable at 0205 PST and placed in manual.
(Subsequent evaluations concluded that sufficient strands of conductors were intact to enable DG 2-2 to perform its intended safety function.) The cubicle door was reopened between 0500 and 0530 PST to replace the degraded wires. At 0851 PST, the cubicle door was closed, the clearance was reported off, and the DG was declared operable. At 0858 PST, auxiliary power to Bus H was declared operable.
On the dayshift of March 25, 2001, inspections of the auxiliary feeder breakers continued, with Unit 2 Bus F, Breaker 52-HF-13. Compensatory measures were taken, the tap changer was placed in manual, and the bus was transferred to startup power. The LTC was left in tap-4 position to maintain normal voltage on 480-V Bus F. At that time, the Unit 1 Shift Foreman questioned if the startup power supply was still operable.
Maintenance was contacted and instructed not to open the auxiliary power breaker cubicle door until the question could be answered. Operators subsequently declared the startup power inoperable, performed the required surveillance, and transferred Bus F back to auxiliary power.
On March 25, 2001, PG&E engineering personnel determined that while the 230-kV to 12-kV SUT LTC was in tap-4 position, the 230-kV startup power system was inoperable. This is because under worst-case loading conditions, the low voltage on the secondary side of the transformer, concurrent with a LOCA, could result in double sequencing of the ESF loads.
Thus, with the 230-kV offsite power system inoperable concurrent with DG 2-2 and auxiliary power to Bus H inoperable, operating personnel should have entered TS 3.0.3, in accordance with condition J of TS 3.8.1.
The condition existed for approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> while the clearance was active on Bus H Breaker 52-HH-13 and work was in progress (from 0453 PST to 0851 PST). The condition is reportable to the NRC in accordance with 10 CFR 50.73(a)(2)(i)(B).
On March 28, 2001, with the exception of eight breakers determined to be operable, the wiring inspections were completed on the Units 1 and 2 vital � H 4.16-kV cubicles and breakers. A total of 15 wires were identified and replaced because they were either degraded or nonfunctional.
PG&E determined that none of the failed or degraded wires alone had made a train inoperable, nor could they have credibly prevented the fulfillment of an entire safety function. However, given the number of LICENSEE EVENT REPORT (LER) TEXT CONTINUATION Diablo Canyon Unit 2 0 degraded wires discovered, and assuming the condition had not been identified and corrected, additional failures could have occurred resulting in potentially overloading a DG. Although the degraded wires did not cause a plant problem, they caused a common condition that presented the potential to prevent the fulfillment of the safety functions of trains in different systems. Therefore, on March 30, 2001, PG&E determined that the condition was reportable to the NRC in accordance with 10 CFR 50.73(a)(2)(ix)(A).
C. Inoperable Structures, Components, or Systems that Contributed to the
Event
None.
D. Other Systems or Secondary Functions Affected
None.
E. Method of Discovery
Degraded Wire PG&E personnel discovered a broken wire in Unit 2 Cubicle 52-HH-9 during routine scheduled functional testing of an overcurrent relay for CSP 2-2. This discovery prompted further investigations for similar degraded or broken wires. The potential generic implications were identified upon discovery of a second degraded wire on March 22, 2001.
TS 3.0.3 Entry On March 25, 2001, maintenance and operating personnel were preparing for inspection of the auxiliary power feeder breaker on Unit 2 4.16-kV Vital Bus F. When the 230- to 12-kV SUT LTC was placed in manual and adjusted to the tap-4 position to compensate for high voltage on the 230-kV startup power system, the Unit 1 Shift Foreman questioned whether the startup power source was operable.
F. Operator Actions
None.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) Diablo Canyon Unit 2 0 G. � Safety System Responses None.
III. � Cause of the Problem
A. Immediate Cause
Degraded Wire Wire was abnormally degraded in service.
TS 3.0.3 Entry Operations personnel manually placed SUT LTC in the tap-4 position.
B. Root Cause
Degraded Wire The cause and failure analysis concluded the wiring degraded due to:
1. Aging of the wire insulation reduced the wire support. Over time, the plasticizer leached out of the PVC insulation, resulting in embrittlement. Flexing of the wire caused the insulation to break. The loss of mechanical support provided by the insulation focused the bending at the break, speeding the cold working of the wire strands and the eventual failure.
2. The manufacturer's wire loom design forced wires into the side of the breaker cubicle when the door was closed. The loom-side bending forced the wire(s) on the inside of the bend to exceed the minimum bend radius for a dynamic bend, resulting in cold working of the strands. Subsequent cycling of the door eventually resulted in wire failure.
TS 3.0.3 Entry There was incomplete and inconsistent guidance provided in the procedures. A table in OP J-2:VlIl only recommended the tap positions to which the LTC should be adjusted when transferring buses to the startup power system. Additionally, a caution in OP J-6A:II discussed overvoltage conditions that could occur on the 480-V and 4.16-kV buses when the SUT LTC is returned to automatic. Thus, operators manually adjusted the Cubicle/Breaker & Component Wire Condition Wire Function Effect 52-H H-9 CSP 2-2 Broken trip wire Trip circuit from overcurrent relay 51HH9.
Wouldn't protect bus given a fault in motor or cables.
Current circuit from CT to overcurrent relay 51HG7.
Spare wire — no function.
None.
First wire would have performed its safety function.
Second wire was a spare.
52-HG-7 CSP 1-1 2 wires degraded 52-HH-15 Safety Injection Pump 2-2 Partially broken current wire Current circuit from CT to overcurrent relay 51HH15.
None.
Relays on other two phases would trip given a two-phase fault.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) Diablo Canyon Unit 2 0 LTC to match voltages, transferred the buses to startup power, and left the LTC in the tap-4 position to maintain normal voltage conditions. The operators failed to recognize that in this condition, the startup power source was inoperable.
C. � Contributory Cause Degraded Wire None.
TS 3.0.3 Entry None.
IV. � Analysis of the Event Degraded Wire The following table identifies the degraded wires, their functions, and the effects of the degraded conditions.
Degraded Wire Table LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) Degraded Wire Table — Cont.
52-HF-7 DG 1-3 8 wires degraded 2 spare wires for alarm circuit.
1 wire for coil circuit for overcurrent auxiliary relay 51XHF7.
2 wires for C phase ammeter and overcurrent relay 51HF7 current circuit.
1 wire for A phase ammeter and overcurrent relay 51HF7 current circuit.
2 wires for ground sensor auxiliary relay 50NXHF7 alarm circuits.
None.
Wires were degraded but operable.
52-HH-13 Auxiliary power feeder breaker 2 wires degraded 2 wires from the load shed relay 27XHHT; 1 to RHR 2-2, and the other to CSP 2-2.
None.
Wires were degraded but operable. Loads would still shed.
52-HF-12 Component Cooling Water Pump 2-1 Partially broken 1 wire on trip circuit from overcurrent relay 51HF12.
None.
Wire was degraded but operable. Trip function would still occur.
As indicated in the table, thirteen of fifteen wires were identified as being degraded but judged to be operable. One wire was degraded in the cubicle for Safety Injection Pump 2-2. However, there was no effect, as the relays on the other two phases would have tripped the breaker given an overcurrent condition.
One wire was broken in the cubicle for CSP 2-2 and would not have performed its function. � In the latter case, another independent single failure would still be required to render a 4.16-kV vital bus inoperable (for example, a fault on the cable or motor for CSP 2-2). Single failures of this type are assumed in the accident analyses of the Final Safety Analysis Report. In this case, the two remaining 4.16-kV vital buses would have been available to perform their safety- related functions. Therefore, the condition was considered to be of very low risk significance and did not adversely affect the health and safety of the public.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION Diablo Canyon Unit 2 0 TS 3.0.3 Entry A PG&E analysis indicated that the adverse effect of double sequencing is a slight increase in peak containment pressure. This pressure increase would result from delays in effective operation of containment fan coolers and CSPs.
This analysis has shown that the increase is well within the design and licensing basis of DCPP. Therefore, the condition was considered to be of very low risk significance and did not adversely affect the health and safety of the public.
V � Corrective Actions
A. Immediate Corrective Actions
Degraded Wire The degraded wire that was initially discovered in Unit 2 Cubicle 52-HH-9 was replaced. Inspections were subsequently conducted on the 4.16-kV vital switchgear cubicles for both Units, and all degraded wires were replaced.
TS 3.0.3 Entry None required.
B. Corrective Actions to Prevent Recurrence
Degraded Wire 1. To address the age-related embrittlement cause, the 4.16-kV cubicle door hinge wiring will be replaced with non-PVC insulated high-strand wire.
2. To address the cause of the short-radius bending at the looms, the new wiring will be installed, routed, and supported using a new design that avoids cyclical short-radius bending of the wires.
TS 3.0.3 Entry 1. � OP J-2:Vlll will be revised to clearly indicate the acceptable positions that the LTC on the 230- to 12-kV SUT must be manually placed to maintain operable conditions when buses are transferred to startup power.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) Diablo Canyon Unit 2 0 2. OP J-6A:II has been revised to allow for preplanned evolutions that may cause overvoltage conditions on vital 4.16-kV and 480-V buses for up to 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />.
3. Operator training will be conducted on this event, including the effects of double sequencing on vital equipment response times and LOCA conditions.
VI. A Additional Information
A. Failed Components
Degraded Wire The degraded wire was factory-installed #10 and #12 AWG, Type TA insulation, NEMA Class K (high strand) copper wire. The cubicles with the wire looms were procured from General Electric as design class II, and qualified by the utility as design class I.
TS 3.0.3 Entry None.
B. Previous Similar Events
Degraded Wire None.
TS 3.0.3 Entry In LER 1-95-007-01 (PG&E Letter DCL-96-158, dated August 6, 1996), PG&E describes a condition in which the 230-kV offsite power source was unable to meet its design requirements for all conditions. The report addressed potential double sequencing resulting from degraded conditions on the 230-kV transmission system caused by continued load growth. A number of corrective actions were taken to address the conditions, including a modification to the component cooling water system. However, the corrective actions noted in this LER would not have prevented operators from failing to recognize the inoperable conditions caused by a low setting on the SUT tap changer.