05000483/LER-2004-003

LER-2004-003, Reactor trip due to faulty relay.

Callaway Plant Unit 1
Event date:	2-3-2004
Report date:	4-2-2004
Reporting criterion:	10 CFR 50.73(a)(2)(iv)(A), System Actuation
4832004003R00 - NRC Website

A. REPORTABLE EVENT CLASSIFICATION

This event is being reported per 10CFR50.73(a)(2)(iv)(A), system actuation. Both the Reactor Protection System (RPS) and PWR auxiliary feedwater systems were actuated during this event.

B. PLANT OPERATING CONDITIONS PRIOR TO THE EVENT

Callaway Plant was in Mode 1 at 100 percent power.

START OF THE EVENT AND THAT CONTRIBUTED TO THE EVENT

N/A

D. NARRATIVE SUMMARY OF THE EVENT, INCLUDING DATES AND APPROXIMATE TIMES

At 0439, 2/3/04, the Callaway Plant experienced a reactor trip while operating switchyard breaker MDV81 in the electrical distribution switchyard. When breaker MDV81 was opened, a trip of the Main Electrical Turbine Generator output breakers MDV53 and MDV55 occurred. With reactor power above 50 percent power, tripping of the main turbine generator causes a reactor trip to occur. As a result of the reactor trip and subsequent low Steam Generator (S/G) water levels, both motor driven auxiliary feedwater pumps (MDAFP) and the turbine driven auxiliary feedwater pump (TDAFP) actuated, and all other safety related systems operated properly. Plant operators used plant procedures to recover from the reactor trip and stabilize the plant in Mode 3 at normal operating temperature and pressure.

Two problems arose as a result of this event. These problems are:

1. Failed 362/DM1 relay which resulted in the turbine trip/reactor trip.

2. Unexpected trip of the Turbine Driven Auxiliary Feedwater Pump after greater than 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> of operation.

Each of these abnormalities will be addressed in order in the following paragraphs.

Item 1: Investigations following the turbine trip/reactor trip revealed that the failure was due to a faulted 362/DM1 timer relay (manufacturer ABB Power, Model RXKL1) in a circuit known as the "dead machine" trip. The function of this circuit is to protect the main generator from damage by inadvertent energization from a standstill condition. The condition can be detected by a combination of electrical and current circuits contained within the dead machine circuitry. The dead machine relay circuit contains two timer relay modules, 362/DM1 and 362/DM2. The 362/DM1 is energized during normal plant operation. The 362/DM2 is de-energized during normal plant operation. With.the failure of the 362/DM1 relay contacts, a combination of 362/DM2 seal-in contact and 362/DM1 contacts maintained the 362/DM2 energized during normal operations. The combination of these two conditions defeated the voltage supervision portion of the dead machine relay circuit. Under this condition, any phase current in excess of 29,600 amps would cause current relays to actuate and trip switchyard breakers MDV53 and MDV55 and result in a turbine trip.

Normal generator full-load current operation ranges between 28,000 — 29,000 Amps. When switchyard breaker MDV81 (MTGY-CAL-7 line from Callaway Bus A) was opened, it was determined that the dynamics of the generation system changed and caused the current to ramp up 4.5 percent just prior to the plant trip.

The current on the main generator increased by 4.5 percent over a 15 cycle period, and due to the change in system dynamics, the current trip setpoint in the dead machine relay circuit was exceeded, therefore initiating the dead machine relay trip.

The faulted 362/DM1 timer module on the generator dead machine relay was found to have severe heat related degradation which caused two contacts to stick closed. The failure of the timer module was postulated to be a circuit board power supply component failure but subsequent vendor testing has disproved this theory. Additional testing has been requested but it is unknown if a final failure mechanism can be identified. An extent of condition review was conducted and determined that there are no other solid state relays of this vintage and design that could exhibit this type of failure mode at Callaway.

Item 2: Upon the reactor trip, loss of both main feed pumps, and subsequent low-low levels in the S/G's, the TDAFP automatically started at 0439. The TDAFP started properly and exhibited normal operating performance. Due to the excess feedwater capacity and reduced feedwater demand to maintain steam generator (S/G) levels, flow from the TDAFP was throttled at 0448 hours0.00519 days <br />0.124 hours <br />7.407407e-4 weeks <br />1.70464e-4 months <br /> by closing flow control valves ALHV0006, 12, 10, and 8 (to the D, C, B, and A S/Gs respectively.

At 0756, the Control Room received indication that the TDAFP tripped. The TDAFP was declared inoperable in accordance with Technical Specification 3.7.5. Local inspection determined the TDAFP tripped on both electrical and mechanical overspeed which had settings of 4235 RPM and 4427 RPM respectively. A review of the local control panel FC-219 provided indication that TDAFP speed reached 4605 rpm.

An investigation of the overspeed trip was initiated. The investigation evaluated available data and concluded that the trips occurred in the expected sequence and within the expected speed range. All evidence supports that these overspeed trip devices functioned as designed.

The overspeed data was evaluated for its affect on system piping integrity. Based on pump affinity laws, the 4605 rpm overspeed condition would create an approximate 2411 psi differential pressure (2431 psia assuming 20 psig suction pressure). Calculations document the limiting hydro pressure as 3250 psig and the limiting maximum permissible pneumatic pressure as 2904 psig for the limiting case of an 8 inch diameter pipe. Based on this review, the auxiliary feedwater system (AFW) system piping integrity was not adversely affected. Components and fittings within the system are bounded by the piping maximum pressure ratings.

Following the preliminary investigation, a structured root cause and troubleshooting investigation was initiated to determine the cause of this event. In determining the root cause of the overspeed trip of the TDAFP, EPRI's Terry Turbine Maintenance Guide, AFW Application, Final Report 1007461, issued, November 2002, was used in addition to the Terry Turbine 505 Digital Control Instruction Manual. The following potential causes were reviewed. All were eliminated except for Controller Failure Events.

• Loss of Load Events

• Controller Failure Events

• Steam Pressure Increases

• Sabotage/Human Error

• Water Ingestion Events During the troubleshooting, the TDAFP was operated for approximately 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br /> and started approximately 25 times with no additional overspeed trips occurring. As a result of the additional testing conducted, approximately 15 months of additional wear occurred to the TDAFP governor valve. Degradation to the governor valve stem and carbon spacers was discovered, and the governor valve was overhauled to replace these components. During testing conducted after work was completed on the governor valve, trip throttle valve FCHVO312 exhibited faster than expected opening speeds which resulted in higher than normal startup speeds of approximately 3800 to 4000 RPM. Additional investigation identified a loose wiring connection which caused the faster stroke time of FCHVO312. Following repair of the loose connection, FCHVO312 was retested and both stroke times and TDAFP startup speeds returned to normal values.

Extensive troubleshooting of the governor control system narrowed the area of failure to three subcomponents: the PGPL (abbreviation for Proportional Governor Pipe Line), PGPL Driver, and Remote Servo (manufacturer Engine Systems Inc.). Due to component interactions, it was not possible to determine which of the three items were faulted. The three suspect components were replaced and the control system was then retested. Several pumps runs including one run lasting four hours were performed. Following satisfactory performance of surveillance testing, the TDAFP was declared Operable at 0823, 2/14/04.

Due to the complicated nature of this TDAFP failure, it was determined that troubleshooting and testing of the TDAFP at system pressures available only in Mode 3, would require more time than the 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> allowed per Technical Specification (T/S) 3.7.5. A one-time Emergency T/S 3.7.5 change was requested and granted by the NRC. This emergency T/S change granted an additional 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in Mode 3 for troubleshooting. It was stipulated that once a positive cause of failure was identified, then the plant had to proceed immediately to Mode 4 to facilitate repairs. The NRC acknowledged that it was possible for the problem to be corrected during troubleshooting, and that if that were to occur, it would not be necessary to enter Mode 4. Troubleshooting efforts extended beyond the additional 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> granted in Mode 3 and the plant entered Mode 4 to complete all necessary troubleshooting and testing possible at the reduced secondary system pressures. Once repairs were completed, all retests were performed per existing T/S guidance.

Engineering performed a past operability evaluation and concluded that there was no past operability concern. This is supported by other operations of the TDAFP which were satisfactory. One such run occurred on 9/29/03 and lasted for approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> 8 minutes. There were no prior indications of failure and based upon this lack of previous failures, time of discovery was established as the time of failure for T/S violation evaluations.

• Safety Analysis also performed evaluations to determine if the run time of approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> 5 minutes prior to the pump trip on 2/3/04 would have been sufficient to satisfy FSAR requirements. Their conclusion was that it would have been satisfactory The failure of the TDAFP was also evaluated as a potential event or condition that could have prevented fulfillment of a safety function. Per 10CFR50.72(b)(3)(vi), Individual component failures need not be reported pursuant to paragraph (b)(3)(v)... if redundant equipment in the same system was operable and available to perform the required safety function." In this instance both of the MDAFPs were operable, thus this event did not represent an event that could have prevented fulfillment of a safety function.

To summarize, time of failure was time of discovery and this did not represent a past operability issue. This event also did not represent a condition that could have prevented the fulfillment of a safety function, and this failure of the TDAFP was not reportable.

E. METHOD OF DISCOVERY OF EACH COMPONENT, SYSTEM FAILURE, OR PROCEDURAL ERROR

Failure of the 362/DM1 relay became evident as a result of the turbine trip/reactor trip that occurred while performing breaker operation in the transmission switchyard.

The fault in the TDAFP control system became evident when the TDAFP tripped during operation on 2/3/04. Troubleshooting subsequently determined the three possible faulted components.

II. O EVENT DRIVEN INFORMATION

A. SAFETY SYSTEMS THAT RESPONDED

As a result of the reactor trip, the Auxiliary Feedwater (AFW) system actuated.

B. DURATION OF SAFETY SYSTEM INOPERABILITY

The TDAFP train of AFW was inoperable from 0757, 2/3/04 until it was declared Operable at 0823, 2/14/04, for a total time span of 264 hours0.00306 days <br />0.0733 hours <br />4.365079e-4 weeks <br />1.00452e-4 months <br />, 26 minutes.

C. SAFETY CONSEQUENCES AND IMPLICATIONS OF THE EVENT.

A probabilistic risk assessment (PRA) determined that the reported event was of very low risk significance.

III. O CAUSE OF THE EVENT The cause of the reactor trip was a failed generator.

timer relay in the dead machine protection circuit for the main The cause of the TDAFP trip was determined to be one of three suspect components contained within the control system.

IV. O CORRECTIVE ACTIONS The faulty 362/DM1 timer relay was replaced and proper operation of the dead machine circuit was verified.

The PGPL, PGPL Driver, and Remote Servo were replaced and the control system was then retested.

Several pumps runs including one run lasting four hours were performed. Following satisfactory performance of surveillance testing, the TDAFP was declared Operable at 0823, 2/14/04.

V. O PREVIOUS SIMILAR EVENTS A review of LERs submitted since 2001 identified one past LER of a reactor trip caused by a relay failure in the main generator protection circuitry. This was submitted in LER 2004-002-00.

A review of the Callaway Action Request System (CARS) was conducted for similar occurrences of the two items, (1) main generator protective relay failures which resulted in a reactor trip, and (2) TDAFP inoperability due to governor control system failures.

(1) On 1/27/04, due to a faulty electrical relay, Callaway experienced a main electrical generator trip which resulted in a reactor trip also. This was described in CAR 200400629 and LER 2004-002-00. The causes of the two reactor trips were evaluated for common cause failure possibilities and it was determined that the two trips were unrelated.

No additional CARs were identified involving reactor trip from generator relay failures.

(2) A review of CARS identified CAR 200201211, which documented earlier problems experienced with TDAFP speed control. No CARs were discovered that addressed a failure similar to that experienced during the event documented in this LER.

VI. � ADDITIONAL INFORMATION The system and component codes listed below are from the IEEE Standard 805-1984 and IEEE Standard 803A-1984 respectively.

Electrical Relay failure System: � TB Component: � RLY TDAFP failure System: � BA Component: � SC