05000498/LER-2004-001

LER-2004-001, Unit 1 Reactor Trip On High Steam Generator Levels Initiated by Failure of Inverter 1201

South Texas Unit 1
Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(iv)(A), System Actuation
4982004001R00 - NRC Website

A. REPORTABLE EVENT CLASSIFICATION

The failure of Class 1E 7.5KV Inverter 1201 resulted in a reactor trip. Therefore, this event is reportable pursuant to 10CFR50.73(a)(2)(iv)(A).

B. PLANT OPERATING CONDITIONS PRIOR TO EVENT

STP Unit 1 was in Mode 1 operating at 100% power.

AT THE START OF THE EVENT AND THAT CONTRIBUTED TO THE EVENT

Failure of Class 1E 7.5KV Inverter 1201 caused the Class 1 E 120V Vital Distribution Panel (DP) 1201 to be declared inoperable at 1616 hours0.0187 days <br />0.449 hours <br />0.00267 weeks <br />6.14888e-4 months <br /> on 01/23/04.

D. NARRATIVE SUMMARY OF THE EVENT, INCLUDING DATES AND APPROXIMATE TIMES

At approximately 1615 hours0.0187 days <br />0.449 hours <br />0.00267 weeks <br />6.145075e-4 months <br /> on 1/23/04, Control Room Operators noted numerous alarms and the Shift Supervisor announced to the Operators that DP-1201 had failed. The Secondary Reactor Operator identified main feedwater regulating valves for steam generators "A" and "B" had full open demand signals and attempted to take manual control of the manual/auto (WA) station for each valve. The other two steam generators were not affected by the DP-1201 failure. The Primary Reactor Operator responded to automatic rod insertion caused by a failed reference temperature signal. The Primary Reactor Operator verified the inward rod motion was due to the failed channel and then placed rod control in manual as directed by the Shift Supervisor. While the Shift Supervisor and Primary Reactor Operator were addressing the failure related rod motion, the actions taken by the Secondary Reactor Operator were inadequate to regain control of the "A" and "B" Steam Generator levels. At 1616, about 66 seconds after the inverter failure, high steam generator water levels initiated an automatic turbine trip/reactor trip signal. The plant was stabilized, and after troubleshooting and repair, the reactor was restarted at approximately 2237 hours0.0259 days <br />0.621 hours <br />0.0037 weeks <br />8.511785e-4 months <br /> on 01/25/04.

Failure of inverter 1201 caused the loss of all loads fed by distribution panel DP 1201, which includes the NSSS Process Cabinet Protection Set 1. The loss of Protection Set 1 caused both loop 1 and loop 2 steam generator level signals to fail low. These level signals are sent to NSSS control cabinets 01 and 02 as input signals to the steam generator level control loops. The steam generator level controls fully opened loop 1 and loop 2 feedwater regulating valves as a reaction to the false Steam Generator Low Level Signals. The operator attempted to take manual control at the manual/auto station, but turbine trip/reactor trip occurred on high water level in the loop 2 steam generator before full control could be achieved.

2004 � 01 � 00 Damage Assessment and Investigation Some board indications were noted to have erroneous indications such as very low, but not zero, indications. Subsequent investigation determined that the output of the 1201 inverter was at about 9 volts and had not failed to zero, causing the anomalous indications.

Troubleshooting determined that the "T1" Ferro-Resonant Transformer (FRT) had a turn-to- turn short in the secondary windings resulting in the 9-volt output, which is not enough to provide proper indication. The FRT was replaced and the failed FRT was sent to an offsite laboratory for failure analysis. The insulation varnish on the windings was found to be very brittle and fragile. The conclusion is that the failure was the result of thermal aging. The FRT that failed was a part of the original equipment installation.

Summary of Root Cause The Technical Root Cause is that the STP design of the Instrument Power Supply (VA) System is not single train fault-tolerant and failure of the inverter causes significant operational challenges. Failure of a Class 1 E instrument power supply bus inverter causes multiple instrument failures and creates rapid response events for operations. The STP design incorporates a manual transfer switch that is not effective at providing a fast transfer from the failed inverter to the bypass power source. Corrective actions in this area are centered on evaluating modifications to provide a more fault tolerant Instrument Power Supply (VA) System and on increasing the reliability of SG level controls and other instrumentation associated with bus 1 E vital instrument power supplies.

The Organizational Root Cause was that the Station has not aggressively pursued effective failure prevention, strategies even though previous events have shown that the inverters are susceptible to failure and the Instrument Power Supply (VA) System is not single train fault- tolerant.

The root cause analysis of a previous inverter failure identified several design changes that were implemented at other stations to address inverter failure. The focus during the investigation was centered on operational responses and addressing the direct cause of the failure. Modification to make the Instrument Power Supply (VA) System design more fault tolerant to inverter failures was not pursued. The root cause investigation was conducted by an SCAQ qualified investigator, reviewed by a division manager, and again reviewed by the Condition Review Group. The identification of a narrowly focused root cause and generic implications was either not questioned or proper follow-up was not performed to address these areas.

During a review by the Systems Engineering Equipment Reliability Team (EQRT), the team identified that failure of an inverter would cause a significant operational rapid response challenge. Credit for Operator action was used as justification for not initiating a Condition Report to address this issue.

Despite the fact that failures of FRTs have been responsible for the majority of inverter failures, no coping strategies such as initiating PMs for periodic replacement, obtaining a different FRT design, or developing a monitoring plan to be able to predict degraded FRTs 2. DOCKET 6. LER NUMBER 3. PAGE 2004 _ 01 _ 00 were established.

Corrective actions in this area include presentation of a case study for SCAQ Investigators, System Engineers, and key Station Managers. Additional corrective actions include Engineering review of all equipment failure related SCAQ investigations for effectiveness (2000, 2001 & 2002) and a review of potential Operations rapid response events initiated by equipment failures.

Additionally, inadequate operator actions to mitigate the consequences of the failed inverter was a contributing cause for this event. The failure of inverter 1201 initiated an operational rapid response challenge. Although the event diagnosis appeared to be quick, Operations was unable to regain steam generator level control following failure of inverter 1201. The unit tripped on high steam generator level. Communication failure is the underlying cause of the automatic reactor trip. Had proper communication techniques been employed, a manual reactor trip could have been initiated.

Failure to properly manipulate steam generator level control instruments is the direct cause for failure to control steam generator level.

Corrective actions in this area include identification of potential Operations rapid response events and incorporation of these events into the LOR training. _ Emphasis on communications will be included in this training.

PROCEDURAL ERROR

DP-1201 was discovered to have failed by the annunciation of the Yellow Alarm Indicator (125V AC CH 1 DIST PNL 1201 TRBL) on control panel 3 (CP-3).

A. SAFETY SYSTEMS THAT RESPONDED

The Reactor Protection System responded to this event by initiating an automatic turbine/reactor trip signal.

B. DURATION OF SAFETY SYSTEM INOPERABILITY

Class 1 E 120V Vital DP 1201 was inoperable from 01/23/04 at 1616 hours0.0187 days <br />0.449 hours <br />0.00267 weeks <br />6.14888e-4 months <br /> to 1/24/04 2003 hours0.0232 days <br />0.556 hours <br />0.00331 weeks <br />7.621415e-4 months <br />. Duration of inoperability was approximately 27 hours3.125e-4 days <br />0.0075 hours <br />4.464286e-5 weeks <br />1.02735e-5 months <br />.

C. SAFETY CONSEQUENCES AND IMPLICATIONS OF THE EVENT

This event resulted in no personnel injuries, no offsite radiological releases, and no damage to safety-related equipment. There were no challenges to plant safety.

III. CAUSE OF THE EVENT

A. Technical Root Cause: The design of the Class 1E Vital Instrument Power Supply (VA) System is not single train fault-tolerant and failure of the inverter causes significant operational challenges.

B. Organizational Root Cause: The Station has not aggressively pursued effective failure prevention strategies even though previous events have shown that the inverters are susceptible to failure and the Instrument Power Supply (VA) System is not fault tolerant.

C. Contributing Cause: Operations actions to mitigate the consequences of the failed inverter were inadequate.

IV. CORRECTIVE ACTIONS

A. Evaluate modifications needed to eliminate a significant plant transient or a plant trip as a result of inverter failures.

• Develop a list of all simulated malfunctions that necessitate prompt operator action (e.g., less than two minutes) to maintain plant control.

• Submit these scenarios to the LOR Curriculum Review Committee for incorporation into frequent LOR cycle simulator training.

B. Develop simulator training to include the following aspects:

• Flawed success path to raise awareness of diagnostics

• Monitoring of critical operating parameters and reinforcement of critical communications (i.e., use of PVT, margin to trip)

• Single operator to diagnose flaw while monitoring and communicating margin to actuation

• Discussion of Conduct of Operations expectations and background for avoiding an automatic reactor trip when possible C. Review Off-Normal procedures that include expected trip or actuation values for consistency with Operations Management expectations. Initiate procedure change requests as appropriate.

D. Using the list of malfunctions that necessitate prompt operator action, determine if fault tolerant designs should be developed for any other control systems.

E. Review all equipment failure related SCAQ root cause investigations completed in 2000, 2001 and 2002 to determine if appropriate root causes, corrective actions, and generic implications were evaluated.

F. Perform a reliability analysis of the Class 1E inverters to identify changes to preventive maintenance activities to increase the reliability of the inverters.

G. Present a case study of previous inverter failures to SCAQ investigators, Plant Engineering personnel and to key Station Management.

V. PREVIOUS SIMILAR EVENTS

A. On December 30, 2003 at 1813, STP Unit 2 inverter 1202 failed resulting in Steam Generator Power Operated Relief Valve (PORV) 2D partially opening. PORV 2D opened to approximately 12% and would not close from the Control Room hand switch. Reactor power increased from 3853 MWth to 3859 MWth and was promptly lowered by reducing turbine load. Steam Generator 2D PORV was manually isolated and distribution panel 1202 was transferred to regulating transformer 1202.

B. On July 7, 2002 Unit 2 was operating in Mode 1 at 100% power. The Unit 2 main turbine generator tripped automatically due to a High-High level in the 2B steam generator (SG).

The reactor tripped automatically as a result of the main turbine trip. The trips occurred shortly after the Channel II inverter and distribution panel de-energized. The loss of the distribution panel and inverter resulted in the loss of power to the instrumentation channels selected to control narrow range steam generator water level. This failure resulted in loss of SG level signal to all four SG Main Feedwater Regulating Valve (MFRV) control circuits because they were all selected to the same channel. This caused the MFRVs to go fully open. With the MFRVs fully open, water level increased in all four steam generators. Steam generator 2B reached its high-high level set point resulting in the main turbine trip and the feedwater isolation signal. The cause of the inverter failure and distribution panel loss of power was the blowing of the direct current (DC) input fuse which de-energized the inverter and power supply to the distribution panel. The second cause of the reactor trip was having all four steam generator level control switches aligned to a single control channel coupled with the loss of power to instruments on that channel.

VI. ADDITIONAL INFORMATION

None 6