05000259/LER-2010-004

LER-2010-004, Residual Heat Removal System Pump Motor Failure

J1) Docket (2) Ler Number (6) Page (3)
Event date:	10-27-2010
Report date:	09-30-2013
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
2592010004R02 - NRC Website

I. PLANT CONDITION(S)

At the time of the event, Browns Ferry Nuclear Plant (BFN) Unit 1 was in Mode 5, the reactor vessel was flooded up, and the moderator temperature was less than 100 degrees Fahrenheit. Loop I Residual Heat Removal (RHR) [BO] pump 1C was in service in shutdown cooling (SDC).

II. DESCRIPTION OF EVENT

A. Event:

On October 23, 2010, at 0900 hours0.0104 days <br />0.25 hours <br />0.00149 weeks <br />3.4245e-4 months <br /> Central Daylight Time (CDT), Unit 1 entered refueling outage 8. At 1433 hours0.0166 days <br />0.398 hours <br />0.00237 weeks <br />5.452565e-4 months <br /> CDT, Operations personnel placed Loop I of RHR in SDC with RHR 1A and 1C pumps in service in accordance with Operating Instruction (01) 1-01-74, "Residual Heat Removal System.

2117 hours0.0245 days <br />0.588 hours <br />0.0035 weeks <br />8.055185e-4 months <br /> CDT Unit 1 entered Mode 5.

" On October 24, 2010, at On October 27, 2010, at 0131 hours0.00152 days <br />0.0364 hours <br />2.166005e-4 weeks <br />4.98455e-5 months <br /> CDT, Operations personnel secured the 1A RHR pump. The RHR 1C pump remained in service, providing SDC. At approximately 1240 hours0.0144 days <br />0.344 hours <br />0.00205 weeks <br />4.7182e-4 months <br /> CDT, the 1C RHR pump motor tripped. Operations personnel received reports of smoke coming from the 1C RHR pump room and responded in accordance with Emergency Plan Implementing Procedure (EPIP) - 17, "Fire Response Procedure," and Abnormal Operating Instruction (A01) 0-A01-26-1, "Fire Response." By approximately 1245 hours0.0144 days <br />0.346 hours <br />0.00206 weeks <br />4.737225e-4 months <br /> CDT, Operations personnel declared the 1C RHR pump inoperable and re-established SDC by placing the 1A RHR pump in SDC. Because there was no fire, Operations personnel exited 0-A01-26-1 and EPIP-17.

The review of the circumstances surrounding this event has found that the 1C RHR pump motor [MO] failed after approximately 94 hours0.00109 days <br />0.0261 hours <br />1.554233e-4 weeks <br />3.5767e-5 months <br /> of operation in SDC during the 2010 refueling outage, and after approximately 1400 hours0.0162 days <br />0.389 hours <br />0.00231 weeks <br />5.327e-4 months <br /> total operating time since being refurbished to support the restart of Unit 1 in May of 2007.

The Tennessee Valley Authority (NA) is submitting this report in accordance with 10 CFR 50.73(a)(2)(i)(B), as any operation or condition which was prohibited by the plant's Technical Specifications. The past inoperability is based on the inability for the 1C RHR pump to complete its 30 day mission time. The exact date at which the 1C RHR pump would have failed to meet its mission time is difficult to determine with certainty. However, violations of TS LCOs 3.6.2.3, RHR Suppression Pool Cooling, 3.6.2.4, RHR Suppression Pool Spray, and 3.6.2.5, RHR Drywall Spray, most likely occurred since November 2007 based on pump run time records. Additionally, since that time, because the degraded condition was not recognized, LCO 3.0.4 was not met due to mode change. Based on NUREG-1022 guidance for event date reporting and based on the knowledge that the event has been determined to be of very low safety significance, for reporting purposes, the discovery date will be retained as the event date.

1 B. Inoperable Structures, Components, or Systems that Contributed to the Event:

None

C. Dates and Approximate Times of Maior Occurrences:

October 23, 2010 0900 hours0.0104 days <br />0.25 hours <br />0.00149 weeks <br />3.4245e-4 months <br /> BFN Unit 1 entered Refueling Outage 8. CDT 1433 hours0.0166 days <br />0.398 hours <br />0.00237 weeks <br />5.452565e-4 months <br /> Operations personnel established SDC using Loop I 1A RHR pump. CDT 1517 hours0.0176 days <br />0.421 hours <br />0.00251 weeks <br />5.772185e-4 months <br /> Operations personnel placed 1C RHR pump in service to support Loop I SDC.

CDT

October 27, 2010 0131 hours0.00152 days <br />0.0364 hours <br />2.166005e-4 weeks <br />4.98455e-5 months <br /> Operations personnel secured 1A RHR pump from SDC. CDT 1240 hours0.0144 days <br />0.344 hours <br />0.00205 weeks <br />4.7182e-4 months <br /> 1C RHR pump motor tripped.

CDT

1244 hours0.0144 days <br />0.346 hours <br />0.00206 weeks <br />4.73342e-4 months <br /> Operations personnel declared RHR pump inoperable and placed 1A RHR pump in service in SDC.

CDT

November 2010 1C RHR pump motor replaced.

D. Other Systems or Secondary Functions Affected

None

E. Method of Discovery

Operations personnel received main control room indications that the 1C RHR pump motor tripped. They also received high motor winding temperature alarms on the 1C RHR pump motor.

F. Operator Actions

Operations personnel responded in accordance with applicable procedures and re-established SDC by placing the 1A RHR pump in SDC.

G. Safety System Responses

None

III. CAUSE OF THE EVENT

A. Immediate Cause

The immediate cause of this event was a dynamic physical rotor/shaft bow caused internal rubbing that led to the mechanical failure of the motor.

B. Root Cause

The root cause of the 1C RHR pump motor failure was concluded to be a physical bow in the rotor was misdiagnosed and treated as residual unbalance during the 2004-5 overhaul of the 1C RHR pump motor for BFN Unit 1 Recovery.

The investigation established that an improper diagnosis of the abnormal vibrations of the motor while on the test stand was the origin of the 1C RHR pump motor failure. The personnel involved in the troubleshooting activities did not fully understand the problem and made the conscious decision to field balance the motor to bring the vibrations to within acceptance criteria. The decision to treat the symptoms as residual unbalance resulted in eventual motor failure.

Based on the 1C RHR pump motor failure, an assessment was made of all similar safety-related BFN motors (i.e., RHR, Core Spray (CS) [BM], and RHR Service Water/Emergency Equipment Cooling Water [BI] pump motors). No anomalies in oil samples, temperature readings, and vibration analyses were found in any of these pump motors that can be directly linked to the 1C RHR pump motor failure precursors.

C. Contributing Factors

1. Lack of a formal process and documentation of the 1C RHR pump motor condition during the refurbishment activities for the Unit 1 Recovery.

2. BFN was not effective in evaluating and further investigating data documenting changes in 1C RHR pump/motor parameters associated with vibration and oil sample results. The ineffectiveness was determined to be a result of a lack of interface and ownership between the associated engineers.

3. Perceived time pressure to meet the Unit 1 Recovery schedule.

4. Inadequate Lubrication Program oversight at BFN.

IV. ANALYSIS OF THE EVENT

The 1C RHR pump motor was shipped to the TVA Power Service Shop (PSS) for refurbishment in 2004. The 1C RHR pump motor was reinstalled at BFN in 2005. As part of the refurbishment process, the rotor was balanced separately from the stator.

The balanced rotor was then installed in the vertically positioned stator for a no load test run. Although the rotor was balanced, the radial vibrations were found unacceptable.

The PSS disassembled the 1C RHR pump motor repeating the rebuild process. All fits and clearances were rechecked, new bearings installed, testing repeated, including a repeat of the rotor balance. The rotor balance was documented as being within an acceptable level. Again the balanced rotor was re-installed in the stator, the motor completely reassembled, and a no load test was performed. The vibrations were found unacceptable. The motor was then field balanced. Two weights totaling approximately 2.16 pounds were added to the upper bearing carrier. These weights were both located 180 degrees from the rotor/stator rub, which eventually caused the motor to fail.

The run speed radial vibrations during no load operation were reduced; however, the coast-down vibrations were elevated. A decision was made at that time to accept the 1C RHR pump motor even though it exhibited an odd characteristic, which was unlike any of the others and was not fully understood or analyzed. Upon return to service of the 1C RHR pump increasing trends began in both vibration and lower bearing oil iron content.

In 2006, following an initial uncoupled run of the 1C RHR pump motor, the lower bearing oil samples indicated high iron content and required flushing. In 2007, the 1C RHR pump motor had undergone multiple lower bearing oil flushes and was just starting to exhibit a low level trend of abnormal vibration. Beginning in 2008, low level trends in vibration began to increase and corrective action documents were initiated to document the condition. In 2009, vibration and lower bearing oil iron content exhibited step increases. In 2010, vibration points exceeded administrative vibration limits.

On October 27, 2010, the 1C RHR pump motor breaker tripped on a time over current condition. The pump motor could not be turned during troubleshooting; therefore, it was removed and sent to the TVA PSS for disassembly, inspection, and testing to determine the cause of the failure. When the rotor was removed from the stator during the disassembly process, it was immediately visually apparent the rotor had contacted the stator. The upper and lower ball bearings were found intact and did not fail during this event. The motor passed all post-failure electrical testing with satisfactory results and post-failure mechanical inspections concluded that the failure was not initiated by the failure of a mechanical component in the motor. Further inspection revealed that the rotor was rubbing the stator due to a physical bow in the shaft, which had become severe enough to induce a heavy rub between the rotating and stationary components causing the failure. The extent of damage was severe enough that the motor could not be salvaged.

V. ASSESSMENT OF SAFETY CONSEQUENCES

The ECCS is designed, in conjunction with the primary and secondary containment, to limit the release of radioactive materials to the environment following a Loss of Coolant Accident (LOCA). The ECCS uses two independent methods (flooding and spraying) to cool the core during a LOCA. The ECCS network consists of High Pressure Core Injection (HPCI), the CS System, the Low Pressure Coolant Injection (LPCI) mode of the Residual Heat Removal (RHR) System, and the Automatic Depressurization System (ADS).

Unit 1 was in Mode 5 and flooded up when the failure occurred. The applicable TS LCOs impacted by this inoperable RHR pump are discussed below.

To satisfy TS LCOs 3.6.2.3, 3.6.2.4, and 3.6.2.5, four RHR suppression pool cooling and spray and drywell spray subsystems are required to be operable during Modes 1, 2, and 3 to remove heat from these spaces, to absorb residual heat from the core, and to maintain containment pressures and temperatures within analyzed design limits.

For these LCOs, if one RHR subsystem is inoperable during Modes 1, 2, or 3, the inoperable subsystem must be restored to an operable status within 30 days. Any two of the four RHR subsystems are sufficient to provide required suppression pool cooling or condense the steam in the suppression pool or drywell airspace during the postulated design basis accident. With less than the required number of RHR subsystems operable, the potential exists that primary containment conditions could exceed design HPCI inoperable from September 1, 2009, at 1614 hours0.0187 days <br />0.448 hours <br />0.00267 weeks <br />6.14127e-4 months <br /> to September 3, 2009, at 0500 hours0.00579 days <br />0.139 hours <br />8.267196e-4 weeks <br />1.9025e-4 months <br /> Core Spray Loop II inoperable from September 1, 2009, at 1210 hours0.014 days <br />0.336 hours <br />0.002 weeks <br />4.60405e-4 months <br /> to September 2, 2009, at 1741 hours0.0202 days <br />0.484 hours <br />0.00288 weeks <br />6.624505e-4 months <br /> RCIC System Inoperable approximately 29 hours3.356481e-4 days <br />0.00806 hours <br />4.794974e-5 weeks <br />1.10345e-5 months <br /> from February 5, 2010, to February 7, 2010 RCIC System Inoperable approximately 35 hours4.050926e-4 days <br />0.00972 hours <br />5.787037e-5 weeks <br />1.33175e-5 months <br /> from August 30, 2010, to August 31, 2010 HPCI inoperable from July 24, 2009, at 1130 hours0.0131 days <br />0.314 hours <br />0.00187 weeks <br />4.29965e-4 months <br /> to July 25, 2009, at 0125 hours0.00145 days <br />0.0347 hours <br />2.066799e-4 weeks <br />4.75625e-5 months <br /> Risk significant systems (or portions of risk significant systems) unavailable for extended periods of time reduces the margin of safety in the plant. A Probabilistic Risk Assessment (PRA) was performed which calculated the Incremental Core Damage probability Deficit (ICDPD) and Incremental Large Early Release Probability Deficit (ILERPD). The ICDPD was determined to be 2.97E-7 and the ILERPD was determined to be 2.86E-8.

Based on the PRA, this event posed minimal reduction to public health and safety.

VI. CORRECTIVE ACTIONS

A. Immediate Corrective Actions

The 1C RHR pump motor was replaced and post maintenance testing was successfully completed. TVA evaluated the remaining RHR pumps and found no similar predictive maintenance trends.

B. Corrective Actions to Prevent Recurrence I 1. TVA revised maintenance agreements to ensure adequate Engineering oversight and acceptance for maintenance and repair of safety related electric I motors.

2. TVA revised NETP-107, Medium Voltage Motor Testing and Maintenance Program, to include operating experience lessons learned and no load acceptance criteria.

3. TVA identified the large motors that had been refurbished within the last five years and determined that no other motors were accepted with a misdiagnosis similar to the 1C RHR pump motor.

VII. ADDITIONAL INFORMATION

A. Failed Component A General Electric motor (model number 5K6348XC23A, serial number FEJ604001) refurbished by the TVA PSS failed.

B. Previous LERs or Similar Events None

C. Additional Information

Corrective action document for this report is Problem Evaluation Report 274840.

Safety System Functional Failure Consideration: D.

Because there is reasonable expectation that the safety functions of the RHR System could be fulfilled, this event is not a safety system functional failure.

E. Scram With Complications Consideration:

Because the unit was in Mode 5, Cold Shutdown, the event described was not a complicated scram according to NEI 99-02.

VIII. COMMITMENTS

None