05000528/LER-2004-006

LER-2004-006, Loss of Offsite Power - Three Unit trip

Palo Verde Nuclear Generating Station Unit 1
Event date:	06-14-2004
Report date:	10-13-2005
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v)(B), Loss of Safety Function - Remove Residual Heat 10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident 10 CFR 50.73(a)(2)(iv)(A), System Actuation
5282004006R01 - NRC Website

1. REPORTING REQUIREMENT(S):

Arizona Public Service (APS) is also reporting this condition pursuant to 10 CFR 50.73(a)(2)(v)(B) and 10 CFR 50.73(a)(2)(v)(D), as a result of a Loss of Offsite Power (LOOP). Due to the LOOP, APS is also reporting this condition pursuant to 10 CFR 50.73(a)(2)(iv)(A) from the subsequent automatic reactor trip, Main Steam Isolation Signal actuation in all three Palo Verde Plants, and the start of all six Emergency Diesel Generators.

Additionally, APS is reporting a condition in Unit 2 pursuant to 10 CFR 50.73(a)(2)(i)(B) for failing to change the Variable Over Power Trip setpoint after declaring the Main Steam Safety Valves (MSSV) Inoperable as required by Technical Specification 3.7.1 which states:

REQUIRED ACTION A.2� Reduce the variable overpower trip - high setpoint in accordance with Table 3.7.1-1. (Completion Time: 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />)

2. DESCRIPTION OF STRUCTURE(S), SYSTEM(S) AND COMPONENT(S):

Offsite Power Transmission and Distribution Systems The Palo Verde Nuclear Generating Station (PVNGS) is connected by its associated transmission system to the Arizona, New Mexico, California, and Southern Nevada high voltage grid, which is interconnected to other high voltage systems within the Western System Coordinating Council (WSCC).

Palo Verde Nuclear Generating Station (PVNGS) Switchyard (EIIS Code: FK) The PVNGS switchyard consists of two 525kV buses. The two buses are connected to the three PVNGS 525/24kV main step-up transformers, and seven transmission lines, using a breaker and a half scheme. A breaker and a half scheme uses two breakers to connect the source of power to the switchyard or transmission line. Both breakers are required to open to isolate a fault in the system. This scheme is used to increase reliability of power and allows flexibility for maintenance. The seven 525kV transmission lines comprising the PVNGS transmission system are situated in four corridors from the PVNGS switchyard as follows:

• One line to the Devers substation (240 mi.)

• Three lines to the Hassayampa substation (3 mi.)

• One line to the Rudd substation (25 mi.)

• Two lines to the Westwing 525kV substation (44 mi.) Westwino Substation The Westwing substation is comprised of a two-bus 230kV section and a two-bus 525kV section. The 525kV section is connected to the adjacent 230kV Westwing section through three 525/345/230kV load tap-changing transformers. The Westwing 230kV buses are connected to the transmission system using a breaker and a half scheme as follows:

• One line to the Surprise substation

• One line to the Pinnacle Peak substation

• One line to the Liberty substation

• One line to the Agua Fria substation

• One line to the Deer Valley substation

• One line to New Waldell substation

• Two 230/69kV transformers feeding the APS distribution system On-site Power Distribution System (EIIS Code: TB, EL, EA, EB, ED) Power is supplied to the PVNGS auxiliary buses from the offsite power supply through three startup transformers. In addition, during normal plant operation, power for the onsite non-Class lE alternating current (ac) system is supplied through the unit auxiliary transformer connected to the main generator isolated phase bus. The Class lE buses normally are supplied through the startup transformers. Each unit's non-Class lE power system is divided into two parts. Each of the two parts supplies a load group including approximately half of the unit auxiliaries. Three startup transformers connected to the 525kV switchyard are shared between Units 1, 2, and 3 and are connected to 13.8kV buses of the units. Each startup transformer is capable of supplying 100 percent of the startup or normally operating loads of one unit simultaneously with the engineered safety feature loads associated with two load groups of one other unit. The 4.16kV Class 1E buses are each normally supplied by an associated 13.8/4.16kV auxiliary transformer, and receive standby power from one of the six standby diesel generators. The Class 1E 4.16kV system supplies power to 480V and lower distribution voltages through eighteen 4.16kV to 480V load center transformers.

Palo Verde Nuclear Generating Station Generator Protective Relavinq The main generator protection schemes include relaying designed to protect the generators against internal as well as external faults. Protection against external faults includes backup distance relaying and negative sequence time over-current relaying. The backup distance relaying provides backup protection for 24kV and 525kV system faults close to the switchyard. The distance relay operates through an external timer. If the fault persists and the time delay step is completed, a lockout relay trips the unit auxiliary transformer 13.8kV breakers, generator excitation, 525kV generator unit breakers, main turbine, and the main transformer cooling pumps. The lockout relay also initiates transfer of station auxiliary loads.

The main generator negative sequence time over-current relay provides generator protection against possible damage from unbalanced currents resulting from prolonged faults or unbalanced load conditions. The relay operates through a lockout relay to trip the unit auxiliary transformer 13.8kV breakers, generator excitation, 525kV generator unit breakers, main transformer cooling pumps and the main turbine. The negative sequence relay also incorporates a sensitive alarm circuit that, in conjunction with a separately mounted ammeter, alerts operators on relatively low values of negative sequence current (just above normal system unbalance).

Emergency Diesel Generators (ENS Code: EK) The Class lE ac system distributes power at 4.16kV, 480V, and 120V to all Class lE loads to ensure safe shutdown of the facility during postulated events. Also, the Class lE ac system supplies power to certain selected loads that are not directly safety-related, but are important to the plant. The Class lE ac system contains standby power sources (i.e., EDGs) that automatically provide the power required for safe-shutdown in the event of loss of the Class lE bus voltage. In the event that preferred offsite power is lost, the Class lE system functions to shed Class lE loads and to connect the standby power source to the Class lE busses. The load sequencer then functions to start the required Class lE loads in programmed time increments.

Reactor Protection System/Plant Protection System (EllS Code: JC) The reactor protection system (RPS) portion of the plant protection system (PPS) provides a rapid and reliable shutdown of the reactor to protect the core and the reactor coolant system pressure boundary from potentially hazardous operating conditions. Shutdown is accomplished by the generation of reactor trip signals. The trip signals open the reactor trip switchgear (RTSG) breakers, de-energizing the control element drive mechanism (CEDM) coils, allowing all control element assemblies (CEAs) to drop into the core by the force of gravity.

Core Protection Calculator/Control Element Assembly Calculator The CPC/CEAC (core protection calculator / control element assembly calculator) system monitors pertinent reactor core conditions and provides an accurate, reliable means of initiating a reactor trip. The CPC system is an integral part of the plant protective system in that it provides two trips to the Reactor Protection System. Trip signals are provided to the reactor protection system whenever the minimum departure from nucleate boiling ratio (DNBR) or fuel design limit local power density (LPD) is approached during reactor operation.

3. INITIAL PLANT CONDITIONS:

On June 14, 2004, at approximately 07:41 Mountain Standard Time (MST), all three Palo Verde Units were in Mode 1, 100 percent power and at normal operating temperature (NOT) and normal operating pressure (NOP).

There were no other major structures, systems, or components that were inoperable at the start of the event that contributed to the event.

4. EVENT DESCRIPTION:

On June 14, 2004 at approximately 07:41 MST, the Western Area Power Authority 230kV Westwing to Liberty line phase C shorted to ground in the greater Phoenix northwest valley.

The event began with the failure of a ceramic insulator at Tower #73 on the Westwing - Liberty 230kV line immediately after a large bird was observed flying from the location. The most probable cause of this insulator failure was conductive material deposited by the departing bird. Due to a defective relay, the fault was not cleared from the Westwing 230kV West bus by Westwing breaker 1022, and it continued to be fed by three 525kV/230kV transformer banks in the Westwing 525kV switchyard, later becoming a three phase fault. This equipment failure permitted the fault to affect the transmission system for approximately 38 seconds, when normally, such a fault would be isolated in a few cycles.

The Westwing 525kV switchyard is connected to Palo Verde by transmission lines Westwing one and two. The Palo Verde Unit Control Rooms received main generator indications of 100 MW and 500 to 700 MVAR oscillations. Eventually, all transmission lines were disconnected from the Palo Verde Switchyard. Subsequently, all three Palo Verde Units were disconnected from the Palo Verde Switchyard by generator protection. A Loss­ of-Offsite Power (LOOP) condition existed when all transmission lines to the switchyard opened. All three Palo Verde Units' turbines tripped on over speed and subsequently all reactors tripped. All diesel generators started and connected to their respective Engineered Safety Features (ESF) busses. In Unit 2, the "A" EDG lost indication of output voltage and current after approximately 26 seconds. Unit 2 was reduced to a single power source, requiring an ALERT classification. Unit 1 & 3 also had E-plan requirements due to the loss of offsite power to their ESF busses for greater than 15 minutes, resulting in Notice of Unusual Event (NUE) classifications for those units. Offsite power was restored and the station downgraded the Unit 2 event from Alert to NUE at 09:51 MST. The event was terminated at 12:07 MST.

Additionally in Unit 2, Operations personnel declared all four low range MSSVs inoperable since one bank of MSSVs may have lifted. With the MSSVs inoperable, TS LCO 3.7.1, Action A.2 requires the VOPT setpoint be reduced within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. All four VOPT pretrips were reset as required by TS Table 3.7.1-1 but not within the 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> as required by TS.

5. ASSESSMENT OF SAFETY CONSEQUENCES:

The event experienced by Unit 1 and 2 on June 14, 2004, did not result in a transient more severe than those already analyzed in the Chapter 15 of PVNGS UFSAR. The reactors tripped from 100% power on a CPC generated reactor trip on low DNBR due to low RCP speed. The event initiator was a loss of load due to grid disturbance. The response time for the reactor trip on low DNBR due to low RCP shaft speed was well within the 0.3 second response time assumed in the UFSAR Table 7.2-4AA.

In Unit 1, the post trip NSSS response was normal with the exception of letdown flow.

Letdown flow normally isolates automatically when Nuclear Cooling Water (NCW) flow is lost. However, the loss of power caused the letdown controller to shift to manual which prevented the isolation of the system due to a high temperature condition. Letdown flow and charging pumps were secured manually which maintained adequate pressurizer level.

The Main Steam Isolation Signal (MSIS) was manually actuated as required by the Emergency Operating Procedure (EOP). No Primary Safety Valves (PSV) or Main Steam Safety Valves (MSSV) lifted as a result of this event.

Following the LOOP in Unit 2, and after shedding load on the switchgear PBAS03, the "A" Train EDG failed while sequencing equipment back into service. However, since the failure of a single Diesel is assumed in the Chapter 15 safety analyses, the limiting Loss of Reactor Coolant Flow as analyzed in Chapter 15 remained bounding for this event. Per EOP, MSIS was manually actuated.

One bank of MSSVs may have lifted and were, therefore, declared inoperable per TS LCO 3.7.1. Subsequent testing found only one MSSV out of tolerance (00T). This MSSV was adjusted to within the tolerance specified in the Surveillance Requirements and was returned to service. With one or more required MSSVs inoperable, Action A.2 of this TS requires the VOPT setpoint be reduced within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. All four VOPT trip setpoints were reset as required by TS Table 3.7.1-1 but not within the 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> as required by TS. The safety significance for failing to reset the VOPT setpoints was not a precursor to a significant event since no overpower event can be initiated from Mode 3 with the reactor trip switchgear breakers open. The failure to reset the VOPT setpoints would not become a more significant safety concern because a Mode 2 restraint was generated to ensure that the MSSVs would be retested and the valves declared Operable prior to proceeding into Mode 2. The VOPT is only required to be operable in Modes 1 and 2. Personnel involved with this TS non-compliance were advised of the need to meet all applicable TS actions.

The response to the LOOP was significantly different in Unit 3 than the response seen in Units 1 & 2. Voltage on the 13.8 kV buses (S01 and S02) remained somewhat higher in Unit 3 than in Units 1 & 2 as grid voltage dropped in response to the three phase fault on the 230 kV Liberty line. Analysis of the digital fault recorder (DFR) data identified a problem with the generator Maximum Excitation Limiter (MEL) circuit. The MEL did not clamp down excitation current as it did in Units 1 and 2. This caused SO1 and SO2 voltage to be higher and hence load shed came in later. When the Palo Verde switchyard was isolated from the frequency stabilization of the grid, the generators' frequency increased to experienced the increased RCS flow when the RCPs increased speed due to an increase in both generator frequency and generator voltage. The increased RCS flow resulted in a rapid reactor power increase due to cooler water in the upper core. The reactor tripped on initiation of the Reactor Protection System (RPS) Variable Over Power Trip (VOPT). A work mechanism (WM) has been generated to troubleshoot the MEL circuit in Unit 3.

During this event, Unit 3 did not experience a transient more severe than those already analyzed in the Chapter 15 of PVNGS UFSAR. The peak reactor power level recorded from CPC trip buffer report was approximately 109 percent power. The response time for the reactor trip on VOPT was well within the 0.45 second response time assumed in the UFSAR Table 7.2-4AA.

The Unit 3 post trip NSSS response was normal with the exception of the Steam Bypass Control System (SBCS) valve's modulating open approximately 1 minute after the reactor trip. The cause of the SBCS open signal is due to the manner and timing of the loss of power. With Unit 3's electrical buses NANSOI and NANS02 maintaining voltage for a longer time period, the SBCS received a shorter, incomplete interruption in power as compared to Units 1 and 2 until the class bus was energized from the diesel. Units 1 and 2 lost both Non-Vital Instrument Buses (D11 and D12) while Unit 3 only lost D11 for about one second. The short interruption in power to D11 in Unit 3 caused the SBCS to begin a 30 second test timer which blocks all output to the valves.

(NOTE: On 4/7/05 one of the two test timers that should have been powered from D12 was found to be miswired and connected to D11. This would have caused both timers to start on loss of power to D11 with the same effect. The two timers would have run in parallel and would have timed-out nearly at the same time. Either one of the timers will block all outputs. The only power affected was the logic power "Acopian" and not the Foxboro power supplies that supply the other components in this scenario.) This timer function is normally used for returning the system to service after on-line testing.

The loss of power was long enough to shift the modulation controller to manual but not long enough to run its output to zero. The modulation controller was still attempting to open the valves to 100% in manual but was blocked by the timer. The permissive controller saturated high since pressure was still above its setpoint. When the test timer timed-out, all valves received a 100% demand signal. With the modulate controller in manual and at 100% output, valve closure depended on losing the permissive signal. The permissive signal is designed to come in quickly on a pressure increase (setpoint is at 0.5%) but can have significant delays on a pressure decrease. The delay of the permissive controller to return from 100% saturation down to the 0.5% setpoint combined with a 15 second Off Delay timer, delayed valve closure for about 25 seconds after the steam pressure had dropped below setpoint. The resulting SBCS steam flow dropped steam generator pressure and resulted in a Low Pressure Main Steam Isolation Signal (MSIS) actuation about 25 seconds after the valves began to open. All eight valves started to quickly close with the loss-of-permissive signal about 5 seconds after the MSIS. The opening of all eight SBCS valves after a reactor trip is bounded by UFSAR 15.1.3 (LDCR 2001-F056) which assumes all SBCS valves open at 100% power. In this case, the CEAs were at the bottom of the core when the SBCS valves opened the second time and caused the MSIS. In Units 1 & 2 the longer power loss to D11 caused the master controller to go to manual and run its output to zero. With loss of both D11 and D12, the condenser interlock also sealed in.

There are two condenser interlock circuits. Both use a 2 out of 3 logic from a total of six vacuum switches. One circuit is powered from D11 and the other from D12. Either will block the condenser valves but both are required to seal in the interlock. The condenser interlock removes the open permissive from all SBCVs to the condenser, valves 1001 thru 1006. The atmospheric valves, 1007 & 1008, would still be available. All of the LOOP scenarios discussed in the UFSAR assume that condenser vacuum is lost, at the beginning of the event due to a loss of circulating water (CW). This event demonstrated that vacuum can be maintained below the SBCS interlock setpoint for several minutes after loss of CW.

Containment temperature exceeded 117 °F by a few degrees for a duration of 2-3 hrs post trip in all three units, due to loss of containment cooling. This did not have any negative impact on the Non-LOCA or LOCA Chapter 15 safety analysis. Equipment and systems in all three units assumed in UFSAR Chapter 15 were functional and performed as required except as noted above. The Auxiliary Feedwater (AFW) system was manually placed in service by operations prior to reaching an Auxiliary Feedwater Actuation Signal (AFAS) condition. Scenarios defined in UFSAR Chapter 15 concerning Turbine Trip (15.2.2), Loss of Condenser Vacuum (15.2.3), Loss of Reactor Coolant Flow (15.3.1), CEA Withdrawal at Power (15.4.2), and Increased Main Steam Flow (15.1.3 see LDCR 2001-F056) were reviewed and remained bounding for this event.

The safety function, to shut down the reactor and maintain it in a safe shutdown condition, remained fulfilled.

6. CAUSE OF THE EVENT:

The degraded insulator was caused by external contamination and did not, by itself, represent a concern relative to the reliability of the insulators on the 230kV transmission system. The direct cause of the PVNGS switchyard LOOP was identified as the failure of a relay in the 230kV protection for the Liberty line to open when the line was grounded at the contaminated insulator. Currently, the cause is expected to be a combination of non redundant design and the scope of relay testing at the fault location.

The direct cause of the failure of Unit 2's "A" EDG to sustain output and carry loads was identified as a failed power diode on the in-service rectifier bridge in the excitation control circuit. Laboratory analysis concluded that there was a possible minor defect on the device that made it slightly more susceptible than others, but any evidence of that was destroyed by the melting in the arc site. This failure was determined to be a random electronic failure and as such, corrective action cannot be reasonably established for the failure.

Transportability of this condition is limited to the six onsite EDGs which all have the same bridge power diodes installed. These specific diodes are not installed in any other systems at PVNGS. All six onsite EDGs have the same model (1N4056(R)) diodes installed in their excitation bridges that supply DC field volts/amps to the generators. Based on the fact that only one EDG bridge diode failure has occurred in almost 20 years of EDG service and that all the recorded EDG field parameters (volts/amps) taken during monthly full rated load surveillance test runs have remained within expected normal parameters, engineering has concluded that no common mode failure exists for the other onsite Class lE EDGs.

The secondary pressure indications suggested that the low set Main Steam Safety Valves (MSSVs) had lifted in Unit 2. PVNGS experience with MSSVs setpoints drifting following actuation had resulted in a policy change that presumes an inoperable safety valve if actuation occurs. With these four MSSVs inoperable until calibration could confirm or refute Operability, the required Technical Specification (TS) Limiting Condition for Operability (LCO) needed to be met. All four Variable Over Power Trip (VOPT) pretrips were set as required by TS Table 3.7.1-1, but not within the 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> as required by TS.

The direct cause of failing to properly implement the actions specified in Technical Specifications for inoperable equipment following the Unit 2 trip was a knowledge-based human performance error. The root cause was a failure to maintain independence when researching the appropriate action required for out of service equipment. An Operations News Flash was sent to emphasize lessons learned and transportability to other LCOs as an interim action to help prevent recurrence. The cause of failing to reset the VOPT setpoints within the 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> as required by TS Table 3.7.1-1 is human error. The VOPT is only required to be operable in Modes 1 and 2, but the MSSV required action to reset the VOPT setpoint included Mode 3. Personnel involved with this TS non-compliance were advised of the need to meet all applicable TS actions. (Note: TS 3.7.1 has recently been changed. The change is described below in the Corrective Actions section). No unusual characteristics of the work location (e.g., noise, heat, poor lighting) directly contributed to this event.

7. CORRECTIVE ACTIONS:

An independent investigation of this event was conducted in accordance with Palo Verde's corrective action program. Listed below are some of the corrective actions from the investigation that were taken.

• The failed relay was removed from service and visually inspected. The relay showed no apparent signs of contamination or deterioration.

• The Investigation Team found that only the Liberty and Deer Valley transmission lines at the Westwing substation featured a tripping scheme with only one Type AR relay. All of the newer lines featured two Type AR relays. APS modified the tripping schemes for the Liberty and Deer Valley lines to feature two AR relays energizing separate trip coils for each breaker. The Investigation Team is also evaluating the feasibility of installing two trip coils in all single trip-coil breakers.

• The Investigation Team is also evaluating the installation of dual trip coils and ground fault protection on lines that have transformers connecting 525kV and 230kV stations.

• Negative Sequencing Relays were reprogrammed to remove their function at the Hassayampa Switchyard.

• The failure to reset the VOPT setpoint and the need to meet all applicable TS actions was discussed with Operations Department personnel. Subsequently, on July 7, 2005, the NRC approved Palo Verde's amendment to TS 3.7.1 "Main Steam Safety Valves" to permit operations in Mode 3 with five to eight inoperable MSSV (two to five operable MSSVs) per steam generator (SG), increasing the Completion Time to reduce the VOPT setpoint when one to four MSSVs per SG are inoperable and no longer requiring the VOPT trip reset in Mode 3.

Any additional corrective actions taken as a result of the investigation of this event will be implemented in accordance with the APS corrective action program. If information is subsequently developed that would significantly affect a reader's understanding or perception of this event, a supplement to this LER will be submitted.

8. PREVIOUS SIMILAR EVENTS:

No similar condition has been reported in the past three years.

9. ADDITIONAL INFORMATION:

None.