05000296/LER-2013-001

LER-2013-001, Inoperable Emergency Diesel Generator due to Failed Electric Generator Casing Fan Bearing I

Browns Ferry Nuclear Plant, Unit 3
Event date:	01-09-2013
Report date:	05-10-2013
2962013001R01 - NRC Website

I. PLANT CONDITION(S)

At the time of discovery, Browns Ferry Nuclear Plant (BFN), Unit 3, was in Mode 1 at 100 percent rated thermal power.

II. DESCRIPTION OF EVENT

A. Event:

On January 9, 2013, at 0400 Central Standard Time (CST), an Auxiliary Unit Operator (AUO) performing rounds near the 3D Emergency Diesel Generator (EDG) [DG] noticed grease and metallic residue around the shaft leading from the EDG to the generator blower [BLO]. Subsequently, the 3D EDG was declared inoperable. Investigation of this condition led to disassembly of the generator blower assembly where it was discovered that one of the blower bearing [38] had failed. Further investigation revealed that the blower bearing failed 18 days earlier, on December 22, 2012, during the Post-Maintenance Test (PMT), following the 12 year Preventive Maintenance (PM) task for the 3D EDG. On January 10, 2013, at 1500 CST, the 3D EDG was declared operable following blower bearing replacement.

A Past Operability Evaluation (POE) concluded that the 3D EDG would have been unable to meet its 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> Probabilistic Risk Assessment (PRA) mission time beginning December 22, 2012, and its design basis mission time of 7 days beginning April 27, 2011.

B. Inoperable Structures, Components, or Systems that Contributed to the Event:

There were no inoperable structures, components, or systems that contributed to this event.

C. Dates and Approximate Times of Maior Occurrences:

January 9, 2013, at 0400 CST An AUO discovered metal residue around the 3D EDG blower shaft resulting from blower bearing failure.

January 10, 2013, at 1500 CST The 3D EDG was declared operable after the blower bearings were replaced.

D. Other Systems or Secondary Functions Affected

The 3D EDG supplies required loads for safe shutdown and cool down in the event of loss of offsite power and a design basis event in any one unit. Any system that relies on emergency power from the 3D EDG is potentially affected by this condition.

E. Method of Discovery

On January 9, 2013, at 0400 CST, an AUO performing rounds near the 3D EDG noticed grease and metallic residue around the shaft leading from the EDG to the generator blower. Further investigation concluded the blower bearing had failed.

F. Operator Actions

Operators declared 3D EDG inoperable, entered Technical Specifications (TS) 3.8.1 Actions for BFN, Unit 3, verified that there were no redundant inoperabilities, and walked down all other EDGs for immediate indications of common cause failures.

G. Safety System Responses

There were no safety system responses.

III. CAUSE OF THE EVENT

A. Immediate Cause

The immediate cause was the absence of lubrication to the internal parts of the 3D EDG blower bearing due to age related breakdown of the grease.

B. Root Causes This condition has two root causes. First, the shielded bearings in the EDG blowers were not adequately assessed on a component level to identify potential failure modes and impacts to EDG operability. Second, standard vibration data was ineffective in identifying the degradation of the lubrication in the shielded bearings of the BFN, Unit 3, EDG blowers due to the masking effect of generator vibrations.

C. Contributing Factors

Turnover by maintenance personnel was inadequately performed resulting in the night shift crew's failure to notify the day shift crew that two different vibration meters were used during the 3D EDG PMT. This event contributed to the bearing failure remaining unnoticed for 18 days.

IV. ANALYSIS OF THE EVENT

The Tennessee Valley Authority (TVA) is submitting this report in accordance with Title 10 of the Code of Federal Regulations (10 CFR) 50.73(a)(2)(i)(B), as any operation or condition which was prohibited by the plant's Technical Specifications.

The BFN, Unit 3, TS Limiting Condition for Operation (LCO) 3.8.1, AC Sources - Operating, requires EDGs to be Operable, during Modes 1, 2, and 3. With one BFN, Unit 3, EDG inoperable, TS 3.8.1 Required Action B.1 requires power availability from the offsite transmission network be verified in 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and once per 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> thereafter.

TS 3.8.1 Required Action B.2 requires the availability of both Temporary Diesel Generators (TDGs) be evaluated in 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and once per 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> thereafter. TS 3.8.1 Required Action B.3 requires the required feature(s), supported by the inoperable BFN, Unit 3, EDG, be declared inoperable when the redundant required feature(s) are inoperable within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> from discovery of one inoperable BFN, Unit 3, EDG concurrent with inoperability of redundant required feature(s). TS 3.8.1 Required Action B.4 requires Operations to determine if Operable BFN, Unit 3, EDG(s) are inoperable due to a common cause failure or perform Surveillance Requirement 3.8.1.1 for Operable BFN, Unit 3, EDG(s). TS 3.8.1 Required Action B.5 requires the inoperable BFN, Unit 3, EDG 1 to be restored to Operable status within 7 days if the TDGs are unavailable or 14 days if the TDGs are available. With multiple BFN, Unit 3, EDGs inoperable, TS 3.8.1 Required Action H.1 requires all but one BFN, Unit 3, EDGs be restored to Operable status in 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. If a single inoperable EDG, or multiple inoperable EDGs, cannot be restored to Operable status in the required time period, while in Mode 1, 2, or 3, TS 3.8.1 Required Actions 1.1 and 1.2 require the unit to be in Mode 3 in 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> and in Mode 4 in 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />.

The POE determined the 3D EDG was inoperable from April 27, 2011, until January 9, 2013. Based on this evaluation, BFN, Unit 3, operated with one inoperable EDG for longer than allowed by TS 3.8.1 Actions.

During the period of 3D EDG inoperability, other BFN, Unit 3, EDGs were inoperable in excess of 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> as a result of maintenance activities. Since it was not recognized that the 3D EDG was inoperable until the completion of the POE, BFN, Unit 3, operated with more than one inoperable EDG for longer than allowed by TS 3.8.1 Required Action H.1.

BFN, Unit 3, LCO 3.0.4 prohibits Mode changes when an LCO is not met except under certain conditions that were not applicable to this event. Since it was not recognized that one BFN, Unit 3, EDG was inoperable from April 27, 2011, until January 9, 2013, until the POE was completed, BFN changed Modes in violation of LCO 3.0.4 on multiple occasions.

The BFN, Unit 3, EDG blower bearings were installed when the EDGs were manufactured in 1973. The lack of adequate preventive maintenance on the EDG blowers resulted in the 3D EDG blower bearings remaining in service until failure. On multiple occasions BFN had the opportunity to identify the need for periodic replacement PMs on the EDG blower bearings, but it was not until 2011 that PM change requests were initiated to include the bearings into the 12 year diesel generator PMs. Since the new PMs will not be added to the 12 year PM until the next time the 12 year PMs are performed, the blower bearings were replaced by corrective maintenance work orders.

At the time of the 3D EDG blower bearing failure, blower bearings had been replaced on all BFN EDGs except A, 3A, 3B, and 3D EDGs since 2012.

Following the 3D EDG bearing failure, the two bearings from the 3D EDG blower were sent to SKF Group for failure analySis to identify the direct cause of the failure. The report from SKF identified the cause of failure as a lack of lubrication.

In 1986, the bearings in A, B, C, and D EDG blowers were replaced due to a 1974 bulletin that advised upgrading the original roller bearings with deep groove ball bearings. During this same timeframe, vendor recommendations were incorporated to add both a 6 and 12 year inspection to the EDGs. These inspections were defined as PMs at that time; however, personnel failed to incorporate any lifespan for the shielded bearings installed in the blower. The Maintenance Instruction included a recommendation in the six year inspection to remove the main generator bearing cover, inspect and re-grease if needed.

In 2009, self-assessment CRP-ENG-08-009 identified a learning opportunity regarding the lack of EDG blower PMs. Corrective Action (CA) 164475-003 addressed the need for PMs on the blowers and determined that the only PM needed was for replacement of the rubber inserts in the flex coupling. This conclusion was based on a lack of industry operating experience (OE), vendor guidance, Owners Group recommendations, and vendor manual recommendations. At the time, BFN was aligned with the industry for PMs on the EDG blowers.

On multiple occasions, BFN had the opportunity to identify the need for periodic replacement of the EDG blower bearings. The need for PMs was not identified until 2011 because the shielded bearings in BFN EDG blowers were not adequately assessed on a component level to identify potential failure modes and impacts to EDG operability. This is the root cause of the bearing failure of the 3D EDG.

The failure of the 3D EDG bearing was unexpected because degradation had not been seen in the vibration data collected monthly. A thorough review by independent vibration analysts determined that the historical vibration data did not show any signs of bearing degradation on any of the EDG blowers. The only time that the vibration data showed indications of bearing problems was when the 3D EDG blower bearing failed on December 22, 2012. Even then, signs of bearing failure did not trigger alarms on the vibration meter. The reason that the data did not show bearing issues is that the blowers are located on top of the generators and see considerable movement and induced vibrations. The vibration of the bearings was not large enough to show up over the generator vibrations, thus the bearing degradation went unnoticed.

Additionally, the industry standard for predictive vibration monitoring utilizes velocity data. No adverse trends have been noted based on the BFN Predictive Maintenance (PdM) personnel's review of the velocity data. The only signs of bearing failure were noted after the failure by discovering an increase in acceleration vibration data.

Standard vibration data was ineffective in identifying the degradation of the lubrication in the shielded bearings of the 3D EDG blower due to the masking effect of the generator.

This is the root cause of BFN's failure to identify the degradation of 3D EDG blower bearing before it failed.

After the failure of the 3D EDG blower bearing was discovered, PdM was contacted to determine if vibration data detected any anomalies during the December 2012, data review. No anomalies were detected, so PdM performed a point to point review of the data. It was discovered that several of the data points were missing.

Review into this missing data led to the discovery that two separate vibration meters were used to collect data during the 3D EDG PMT. Normally two vibration meters are taken to the jobsite in case problems arise with one, but Technical Instruction 0-TI-230V, Vibration Program, only requires one to be used in the collection of data. The nightshift crew used one meter to collect data on the 3D EDG blower and received a warning message. In order to confirm the data was valid, and not a malfunctioning meter, the second meter was used.

The night shift crew turned the vibration meter over to a supervisor on day shift because it was possible that a subsequent set of vibration readings would need to be collected.

Later into the shift, the supervisor gained confirmation that additional data collection was unneeded. Qualified personnel were tasked with the download of the data. The day shift crew did not receive an adequate turnover; therefore they were unaware that both vibration meters had been used. This resulted in only downloading data from one vibration meter.

Turnover by maintenance personnel was inadequately performed resulting in the failure to relay that two vibration meters were used during the 3D EDG PMT. This is the contributing cause to BFN's failure to identify that the 3D EDG blower bearing had failed on December 22, 2012. This event will be addressed by briefing the Electrical Maintenance Group on the use of complete and accurate communication during shift turnovers. In addition, focused observations documenting the quality of shift turnovers will be conducted.

The inadequate turnover was a cognitive human performance error because the procedure requires the data to be downloaded. Error precursors existed because a face to face turnover was not utilized and the event occurred immediately before the Christmas holiday.

Extent of Condition The extent of condition covers equipment with vibrations that are substantially greater than the vibrations of the smaller attached components leading to the vulnerability of the individual component's vibrations being masked resulting in unidentified failures. This population includes the EDGs, Aggreko diesel generator package units, the Security Diesel Generator, Diesel Fire Pumps, the EDG Room coolers, and the starting air compressors.

On January 29, 2013, the 3A EDG's blower bearings were inspected during its 12 year PM due to the failure on 3D EDG. The bearings were discovered to be in a degraded condition due to age related degradation of the bearing lubricant, and at the time it was unknown if 3A EDG could have met its mission time. The A EDG and 3B EDG were the only remaining EDGs with blower bearings that had not been recently replaced since 2012. Both EDGs were immediately declared inoperable until their blower bearings could be replaced. Inspection of the bearing during replacement determined that the A EDG blower bearings were in satisfactory condition; however, 3B EDG blower bearings were in a degraded condition due to age related degradation of the bearing lubricant.

On January 29, 2013, at 1837 CST, BFN, Unit 3, entered TS 3.8.1 Required Action H.1 due to the concurrent inoperability of 3A and 3B EDGs for blower bearing replacement.

BFN, Unit 3, then entered TS 3.8.1 Required Action 1.1 after 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. BFN did not begin power reduction to enter Mode 3 because it was known that the 3B EDG blower bearing replacement would not exceed the completion time of TS 3.8.1 Required Action 1.1. At 0030 CST on January 30, 2013, 3B EDG was declared operable and BFN exited TS 3.8.1 Required Action H.1 and 1.1.

The fuel oil transfer pumps and engine turbochargers located on the EDGs are also vulnerable to an unidentified failure due to the movement of the diesel engine and generator during the collection of vibration data. Based on multiple unpredicted EDG turbocharger failures, vibration data on the turbochargers has shown to be ineffective in identifying underlying degradation. The condition is being addressed by the installation of an additional oil system to provide pre-lubrication and post-lubrication to the turbochargers and by reevaluating the turbochargers maintenance strategy. The fuel oil transfer pump vibration data can also be masked by the diesel generator; therefore, PM change request will be submitted to require periodic replacement on all eight EDGs.

All EDG turbochargers have recently been replaced and the fuel oil transfer pumps were inspected for degradation or signs of failure. It was determined that there are no current operability concerns.

Extent of Cause The extent of cause for the failed blower bearing includes sealed and shielded bearings on the diesel generators and other safety or quality related rotating equipment where failure would result in BFN's inability to meet TS. An assessment will be performed to identify the population of sealed or shielded bearings associated with EDGs and other safety or quality related systems at BFN. These components will be analyzed to determine the potential failure modes and impact of each failure mode to the operability of the associated system. This analysis will be used to determine the appropriate PM strategy for maintaining this equipment.

The extent of cause for BFN's failure to identify the failed blower bearing extends to vibration monitoring on all eight EDG blowers. To address this condition, an additional vibration monitoring point will be added to the EDG blower fans and 2 existing vibration monitoring points will be moved to the drive end bearing housing. The new vibration monitoring point will monitor for velocity and acceleration and high value alarms will be tied to both parameters.

V. ASSESSMENT OF SAFETY CONSEQUENCES

There are a total of eight EDG units which serve BFN Units 1, 2, and 3. A set of four EDG units designated as A, B, C, D are located in the Unit 1/2 EDG building. Another set of four EDG units designated as 3A, 3B, 3C, and 3D are located in the Unit 3 EDG building.

During the first ten minutes of design basis events, only three of the BFN, Unit 1 and 2, EDGs, along with three of the BFN, Unit 3, EDGs are required to automatically start and come to a ready to load condition. The EDGs must be capable of accepting automatically sequenced plant safety loads. For the long term (greater than 10 minutes), three of the BFN, Unit 1 and 2, EDGs paralleled with three of the BFN, Unit 3, EDGs, are designed to supply all required loads for the safe shutdown and cool down of all three units in the event of loss of offsite power and a design basis event in any one unit.

The POE determined the 3D EDG was unable to meet the design basis 7 day mission time, and was therefore inoperable, from April 27, 2011, to January 9, 2013. These dates were determined by adding 7 days of EDG run time prior to blower bearing replacement and provide a conservative assumption for the extent of the past inoperability.

Vibration data indicates the blower bearing failure occurred on December 22, 2012.

Based on the amount of wear observed, there is no reasonable assurance that the 3D EDG remained capable of meeting the PRA mission time of 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, beginning with the failure on December 22, 2012, and ending when the 3D EDG blower bearings were replaced on January 9, 2013. During the 3D EDG unavailability, the 3B EDG was unavailable due to redundant start relay testing and subsequent relay replacement, beginning January 5, 2013, at 0903 CST, and ending January 8, 2013, at 0830 CST.

The 3A and 3B EDGs bearings were identified as having degraded lubrication as part of the Extent of Condition. An evaluation was performed, and documented in the POE, which conservatively determined that the 3A and 3B EDGs blower bearing were capable of operating for 7.8 days, therefore meeting their design basis 7 day mission time and PRA 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> mission time. Based on the Prompt Determination of Operability performed associated with Problem Evaluation Report (PER) 661328, adequate paralleling ability was determined to be available during the period of inoperability of the 3D EDG associated with the PRA 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> mission time.

A PRA evaluation of this event was performed. This evaluation determined that this condition did not significantly increase risk with respect to the incremental core damage probability deficit or the incremental large early release probability deficit. Therefore, this condition is considered to be of low safety significance, and there was no significant reduction to the health and safety of the public.

VI. CORRECTIVE ACTIONS - The corrective actions are being managed by TVA's corrective action program.

A. Immediate Corrective Actions 1. Reviewed the vibration data for the A, B, C, D, 3A, 3B, and 3C EDG to ensure that there no signs of supply air blower fan bearing failure following discovery of the 3D EDG bearing failure. No signs of failure were found.

2. Walked down the A, B, C, D, 3A, 3B, and 3C EDG to ensure that there were no signs of supply air blower fan bearing failure following discovery of the 3D EDG bearing failure. No signs of failure were found.

3. Replaced all blower bearings on remaining BFN EDGs.

B. Corrective Actions to Prevent Recurrence 1. The population of rotating equipment containing shielded or sealed bearings associated with the EDGs will be analyzed to determine the potential failure modes and impact of each failure mode to the operability of the EDGs. This analysis will be used to determine the appropriate PM strategy.

2. Identify the population of sealed or shielded bearings associated with safety and/or quality related rotating equipment at BFN. These components will be analyzed to determine the potential failure modes and impact of each failure mode to the operability of the associated system. This analysis will be used to determine the appropriate PM strategy.

3. Two existing vibration monitoring points will be relocated to the drive end bearing housing and an additional vibration monitoring point will be added to the EDG blower fans on the drive end bearing housing on all EDGs. This point shall monitor for velocity and acceleration and high value alarms shall be tied to both parameters.

A. Failed Components

The failed component is a Model 3312C deep groove, single row, double shielded ball bearing, manufactured by New Departure Hyatt (NDH), and installed while the EDG was being manufactured in 1973.

B. Previous Similar Events

A search of BFN LERs for Units 1, 2, and 3, for approximately the past five years did not identify any similar events.

A search was performed on the BFN corrective action program. Similar PERs 164475, 369956, 488208, 667866, 675339, and 675952 were identified.

Corrective actions from PER 369956 created the corrective maintenance work orders to replace the blower bearings on all EDGs. The corrective actions did not prevent this event because the 3D EDG blower bearing failed before the work orders could be completed.

C. Additional Information

The corrective action document for this report is PER 665217.

D. Safety System Functional Failure Consideration:

In accordance with NEI 99-02, this issue is considered a safety system functional failure.

E. Scram With Complications Consideration:

This condition did not include a reactor scram.

VIII. COMMITMENTS

There are no commitments.

1