05000341/LER-2001-001

LER-2001-001,

Docket Number
Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
Initial Reporting
3412001001R00 - NRC Website

Initial Plant Conditions:

Mode � 1 Reactor Power � 100 percent

Description of the Event

Emergency Diesel Generator (EDG) 14 [DG] was removed from service at 0940 hours0.0109 days <br />0.261 hours <br />0.00155 weeks <br />3.5767e-4 months <br /> on March 21, 2001, and was subsequently started and loaded for the purposes of performing the 24-hour endurance surveillance test run per Technical Specification Surveillance Requirement (SR) 3.8.1.13, on March 21, 2001 at 1011 hours0.0117 days <br />0.281 hours <br />0.00167 weeks <br />3.846855e-4 months <br />. At 2206 hours0.0255 days <br />0.613 hours <br />0.00365 weeks <br />8.39383e-4 months <br /> (EST), approximately 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> into the run, a generator trouble alarm was received. It was noted that the EDG 14 generator outboard bearing temperature was 228 degrees Fahrenheit and rising rapidly. The EDG 14 output breaker was opened, and EDG 14 was manually tripped at 2210 hours0.0256 days <br />0.614 hours <br />0.00365 weeks <br />8.40905e-4 months <br />. Approximately one minute later, the operator reported a fire on the EDG 14 generator outboard bearing housing. The fire was extinguished immediately by the operator using a hand held carbon dioxide fire extinguisher. Subsequently, an Alert was declared and terminated, and appropriate notifications (10CFR50.72(a)(1)(i)) were made (see Event Notification 37855, dated March 21, 2001).

Technical Specification Limiting Condition for Operation (LCO) 3.8.1, "AC Sources — Operating," requires two emergency diesel generators (EDGs) per division (two divisions) to be OPERABLE during MODES 1 (Power Operation), 2 (Startup), and 3 (Hot Shutdown). With one or both EDGs in one division inoperable, Condition A applies and Required Action A.6 requires restoring the inoperable EDG(s) to OPERABLE status within 7 days. Otherwise, Condition C is entered which requires that the plant be in MODE 3 within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> per Required Action C.1, and in MODE 4 within 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br /> per Required Action C.2.

LCO 3.8.1, Condition A for the inoperable EDG 14 was entered at 0940 hours0.0109 days <br />0.261 hours <br />0.00155 weeks <br />3.5767e-4 months <br /> on March 21, 2001, upon taking EDG 14 out of service to perform SR 3.8.1.13. Hence, the required 7 day Completion Time for Required Action A.6 was set to expire at 0940 hours0.0109 days <br />0.261 hours <br />0.00155 weeks <br />3.5767e-4 months <br /> on March 28, 2001. Detroit Edison determined that the extensive work required to replace the affected bearing and to perform the associated post-maintenance testing (which included a 24-hour endurance run), would extend beyond the LCO 3.8.1, Condition A required Completion Time, and therefore, requested that the NRC grant a request for enforcement discretion to extend the required Completion Time in LCO 3.8.1, Action A.6, from 7 days to up to 14 days on a one-time basis (Letter NRC-01- 0034 from Detroit Edison to NRC, dated March 26, 2001). The NRC verbally granted the enforcement discretion during a telephone conversation on March 27, 2001 at 1412 hours0.0163 days <br />0.392 hours <br />0.00233 weeks <br />5.37266e-4 months <br />, and followed this up with a written Notice of Enforcement Discretion (NOED No. 01-3-001) dated March 29, 2001.

The EDG 14 generator outboard bearing was subsequently replaced, SR 3.8.1.13 was successfully performed and EDG 14 was declared Operable at 0924 hours0.0107 days <br />0.257 hours <br />0.00153 weeks <br />3.51582e-4 months <br /> on March 31, 2001. This event is reportable as an operation prohibited by Technical Specifications (10CFR50.73(a)(2)(i)(B)).

Cause of the Event

This event resulted from a lower than adequate oil level in the bearing housing. The estimated actual oil level at the time of the event was 6-7/8 inches below the centerline of the generator shaft, which is 7/8 inch below the vendor recommended minimum level. Two causes led to this event. The first was the improper use of the design modification control process in 1984 that allowed the EDG oil sight glass [LG] piping configuration to be incorrectly modified and left uncorrected. The second cause was an inadequate process used to install oil level operating bands in 1999.

An opportunity to prevent the occurrence of this event was missed in 1999, when a new Condition Assessment Resolution Document (CARD) was not initiated following discovery that the EDG 14 oil sight glass and level mark were found to be positioned too low. Had a new CARD been initiated following this discovery, an operability assessment could have been requested and this likely would have led to some intermediate corrective action such as changing the oil sight glass level mark, or revising procedures to maintain oil level higher in the EDG 14 sight glass.

Historical Events that preceded the failure are as follows:

1984 A stiffener plate was added to the outboard end of the generator housing to reduce axial vibration on EDG 14 only. This led to a modification of the oil sight glass piping. It is believed that the oil level sight glass was incorrectly lowered at that time, thus indicating higher by approximately 0.9 inches than the actual oil level in the bearing.

1997 A nonconformance report (DER) was issued questioning the proper oil levels for the EDG generator bearings. The DER established the proper oil levels of 5.62 inches + 1/8 to —3/8 inches below the centerline of the shaft (5.5 to 6.0 inches). Prior to this time it was standard practice to keep oil above the sight glass mark. Operator rounds sheets and the maintenance procedure were revised with the new oil level tolerances.

1999 A site wide program for improving oil level indication installed oil level "green bands" on all EDG generator bearing sight glasses. The green band on EDG 14 was placed too low, using the oil sight glass level mark that had existed since 1984, resulting in a higher apparent indication (approximately 0.9 inches) than what actually existed. Maintaining oil in the installed green band could result in the oil level being too low.

1999 EDG 14 underwent a satisfactory 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> endurance run on October 22, 1999. Oil level remained in the green band, apparently high enough in the green band to preclude failure.

Cause of the Event (cont.) 2000 The EDG 14 cylinder liners were replaced and the EDG 14 bearing oil was changed in April. This was followed by several loaded runs of EDG 14 between May 1 and May 9, ranging from approximately 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> to 7 1/2 hours in duration, for a total run time of approximately 43 hours4.976852e-4 days <br />0.0119 hours <br />7.109788e-5 weeks <br />1.63615e-5 months <br />. No indication of bearing problems were noted.

2001 EDG 14 generator outboard bearing failed during the 24-hour endurance run. The oil level green band installed under the 1999 program was found to be positioned too low relative to the centerline of the shaft. The operator indicated that the oil level was at the bottom of the green band following the failure. Due to the error in the placement of the green band, the bearing oil level was determined to be approximately 7/8 of an inch too low.

The bearing is an SKF spherical roller bearing. A service representative from SKF inspected the bearing after the bearing cover was removed and determined that the failure had been caused by metal-to-metal contact, indicative of inadequate lubrication. Recent EDG 14 runs had not given any indication of bearing problems.

Generator bearing oil levels for EDGs 11, 12, and 13 were measured and determined to be adequate, eliminating a potential common mode problem. Additionally, all three of the remaining EDGs had recently completed their 24-hour endurance runs (within the previous month) without any bearing problems noted (i.e., vibration, temperature, and oil sample analyses were within normal ranges).

Following this event, the existing positions of the oil sight glass green bands for EDGs 11, 12, and 13 were checked using a laser level, and several of them were found not to be positioned correctly. The standstill green bands for EDGs 11 and 13 were found to be positioned from 13/64 to 26/64 of an inch too high. An engineering evaluation concluded that although foaming could occur at these levels, adequate lubrication existed for the bearings. Therefore, there were no Operability concerns associated with EDGs 11 and 13 as a result of the previously installed green bands. The EDG 12 outboard bearing standstill green band was found to be correctly positioned; however the inboard bearing standstill green band was found to be positioned 3/32 of an inch too low. It was determined that even if the oil level had been at the bottom of the installed green band (6-3/32 of an inch below the centerline of the shaft) the bearing would have remained sufficiently lubricated. Therefore, there were no Operability concerns associated with EDG 12 as a result of the previously installed green bands. The green bands for EDGs 11, 12, 13, and 14 were adjusted as part of the corrective actions, and they are in the correct positions relative to the centerlines of the shafts.

Analysis of the Event

The Fermi onsite AC power sources consist of four EDGs, two in Division 1, and two in Division 2, and one combustion turbine generator (CTG) 11-1, aligned to Division 1 for station blackout purposes. Technical Specification LCO 3.8.1 permits continued plant operation for 7 days with one or more EDGs in one division inoperable. With only EDG 14 inoperable, the remaining operable EDGs (i.e., two in Division 1 and one in Division 2) are sufficient for performing the safety functions assumed in the safety analyses. CTG 11-1 is also available for station blackout considerations.

Technical Specification Completion Times are established to be of limited duration because of the inability to meet single-failure criteria during the period of time that required equipment is not available. The 7-day Completion Time for restoring both EDGs in one division to Operable status is based upon a consideration of the capacity and capability of the remaining AC power sources, as well as the additional reliability afforded by the availability of CTG 11-1, and the low probability of a design basis accident (DBA) occurring during this period.

Upon determination of the extent of repairs necessary, a request for enforcement discretion to extend the Technical Specification Completion Time for restoring EDG 14 to Operable status from 7 days to up to 14 days was made. The basis for the extension cited no common cause failure potential for the remaining operable EDGs; a risk assessment against criteria for a permanent Technical Specification change in accordance with current approved regulatory guidance; additional equipment unavailability and testing restrictions as actions to minimize risk during the extended Completion Time; and a comparison to the potential risk incurred from a forced shutdown/startup evolution. The risk incurred from continued operation with EDG 14 unavailable from time of discovery to return to service was determined to be less than the conservative estimate of risk which would have been incurred from a forced shutdown evolution. Therefore, there was no net risk increase from the extended Completion Time. EDG 14 was inoperable for only approximately 10 of the 14 days for which this evaluation was performed, and thus, there was negligible risk to public health and safety.

This event was also evaluated assuming that EDG 14 was potentially degraded or unavailable for some period of time prior to the actual failure (the last successful 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> endurance run was in October 1999). Due to the robust electrical configuration at Fermi 2, adequate mitigation capability remains with one EDG unavailable in the event of a divisional or total loss of offsite AC power (the EDGs do not mitigate any other event). With EDG 14 in an assumed failed condition, and nominal unavailability and failure values assumed for other plant equipment, the resultant calculated increase in core damage frequency would be between 1E-6/yr and 1E-5/yr, which results in an overall plant risk consistent with accepted safety goals.

Corrective Actions

The immediate corrective actions taken for this event included removing the failed bearing, machining the damaged rotor, installing a shaft sleeve, and installing a new bearing. This was followed by several engine runs to verify EDG 14 operability, including a 24-hour endurance run followed by a hot fast restart of the engine. These actions were completed and EDG 14 was declared Operable at 0924 hours0.0107 days <br />0.257 hours <br />0.00153 weeks <br />3.51582e-4 months <br /> on March 31, 2001.

Additionally, the EDG 14 oil sight glass and green band were adjusted, such that they are now in the correct relative position to the centerline of the shaft.

Corrective actions taken to reduce the probability of similar events occurring in the future are related to the identified causes and are as follows:

1. The design modification control process was improperly used in 1984, allowing the EDG oil sight glass piping configuration to be incorrectly modified and left uncorrected. The Field Modification Request (FMR) used in 1984 during initial construction to add the stiffener plate, identified the resulting EDG oil sight glass piping configuration discrepancy, but was closed out without correcting the piping discrepancy or transferring the identified discrepancy to another closure mechanism. The FMR process used in 1984 is no longer in use. The design modification control process used today, with its inter-discipline reviews, detailed instructions, and followup documentation closure requirements (as-builting) would prevent the sort of error identified here from occurring. Therefore, no further corrective actions are needed to correct this cause.

2. The process used for green band installation did not provide adequate control. A Technical Services Request (TSR) Engineering Evaluation was used in 1999 to install the oil level operating bands on safety related plant equipment. This process did not provide for second checking or for verifying that the installed oil level operating bands were in the correct position. The TSR procedure has since been modified to completely eliminate the use of the TSR Engineering Evaluation. Consequently, changes of this nature now would be done using the appropriate design modification process for the activity (TSR Modification or Engineering Design Package (EDP)), and thus, would receive the proper review and design control measures. A check of the green band positioning on all safety related equipment using this process in 1999, which will be independently verified, is being conducted at this time. Additionally, an extent of condition review of work performed using the TSR Engineering Evaluation process in the past is being conducted to verify that there are no other installation discrepancies that could affect the safe operation of safety related plant equipment. Any further corrective actions that are identified as a result of this review will be controlled in accordance with the Fermi 2 corrective action program.

A. Failed Components

SKF Model 22240C3 spherical roller bearing B. Previous Similar Events (LERs) Fermi 2 LER 00-009 documents an event in which a review of past preventative maintenance activities identified that an EDG had been inoperable for a period of time that exceeded that the Technical Specification allowed outage time because the wrong lubricating oil had been used. No actual equipment failures occurred as a result of this previous event.