05000219/LER-2014-003
| Oyster Creek, Unit 1 ' | |
| Event date: | 09-12-2014 |
|---|---|
| Report date: | 12-01-2015 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications |
| 2192014003R02 - NRC Website | |
Internet e-maR to Infocceacts,ResourcaOnre.gov, and to the Desk Officer, Office of Information and Regulatory Maks, NEOB-10202, (3150-0104), OSce of Management and Budget, WasNngton, DC 20503.11a means used to impose an Information collection does not dsplay a currently valid OMB control number, the NRC may not conduct or sponsor, and a parson is not required to respond to. the informs/ion collodion.
S.
Description of Event
On July 28, 2014, Emergency Diesel Generator No. 2 (EDG-2) was being operated for its bi-weekly one-hour load test run, when alarms "EDG 2 ENGINE TEMP HIn and TOG 2 DISABLED° were received. Operations manually shut down EDG 2 due to an apparent cooling problem with the diesel engine. During initial troubleshooting, the fan duct was opened to access the upper fan shaft and it was found that the cooling fan shaft had failed. Without the fan in service, radiator heat transfer performance was degraded, leading to high jacket water (coolant) temperatures and associated alarms.
Equipment Description
Oyster Creek Nuclear Generating Station is equipped with two identical EDG units. The function of the EDGs is to provide AC power to the Class 1E busses upon a loss of the off-site power. The EDG must be able to provide this power rapidly, within 10 seconds, upon demand. This condition is referred to as a fast start signal.
If started with a fast start signal, a high jacket water temperature condition will not trip the EDG.
The EDG units are General Motors Corporation, Electromotive Division (EMD) Model EMD 20-645E4, 20- cylinder, 2-cycle, turbo-intercooled diesel engines, which drive their respective EMD A200 AC generators.
The EDGs are installed in enclosures inside the EDG vaults. Engine auxiliaries retain a locomotive-type layout with the radiators in duct compartments over the engines. Cooling air to each engine is drawn into this duct by a large fan at the south end of the enclosure. The fan is supported and rotated by a belt-driven shaft that is in turn rotated by a power-takeoff shaft connected to the engine. The failure location was at a groove in which a bearing retainer is mounted. The failure location could not be visually inspected without removing the bearing retainer and as such, would not be identified during operator rounds or normal maintenance.
Analysis of Event
Following the event, a complex troubleshooting team was formed to Identify and investigate possible failure modes. Bearing issues, bent fan shaft, fan imbalances, metallurgical defect, and incorrect belt tension were considered. Some of the follow-up actions to support or refute possible failure modes required laboratory failure analysis. Consequently, the shaft section and bearing parts were quarantined and sent to Exelon PowerLabs (PL) for examination.
Assessment of Safety Consequences
Two EDG units serve as the Standby Power Supply for the station by providing an emergency source of power to the 4.16 kV buses 1C and 1D in the event of a loss of normal power. The EDG units are designed to start and load automatically, if required. Nonessential loads are automatically shed by undervoitage sensing devices on loss of offsite power to ensure that the units are not overloaded. The capacity of the EDG units is sufficient to sequentially energize for starting all safety-related pumps and auxiliaries required for a safe shutdown of the reactor in the event of a Design Basis Accident. The EDG units are independent of each other, with the exception of a common bulk fuel storage supply, and are provided with auxiliary systems to ensure reliable starting and continuous operation with no operator attention. Power to start the EDG units is self-contained and is not dependent on the availability of any other source of normal plant power at the moment of initiation.
There are two types of automatic start signals for the EDG units. The first signal will cause the EDG units to start and idle. The second signal is called the Fast Start Signal. The EDG allowable time response to a Loss of Offsite Power (LOOP) event is 20 seconds as a basis for Core Spray System response to accident conditions, The time response period includes undervoltage sensor pick-up time, emergency bus logic to isolate and actuate the EDGs, and the period to bring the Emergency Buses to normal voltage level.
The failure of the upper fan shaft resulted in the "EDG 2 ENGINE TEMP HI" and "EDG 2 DISABLED" annunciators actuating in the Main Control Room. This condition would have precluded the EDG-2 from idling in the event a LOCA signal was received. In event of a Fast Start signal (LOOP), the EDG-2 would have started and ran until the failure. It was determined that the EDG-2 would not have achieved its required mission time due to the condition. The #1 EDG would have continued to operate and would have supplied the 1C 4160V bus as required to ensure power to the components required to achieve cold shutdown. The EDG-1 was evaluated to not be susceptible to a common cause failure and was maintained in an operable condition in accordance with station procedures.
PowerLabs Report Summary Laboratory evaluations indicated the shaft failure was caused by rotational bending fatigue. Investigations revealed that based on the smooth, flat, and planar fracture surface features and relatively small final overload area (approximately 10-20% of the total fracture surface), propagation occurred by a high cycle-low stress fatigue mechanism. The cracking initiated at the shaft groove diameter transition, which would act as a high stress concentrator. Multiple ratchet marks (which are indicative of multiple crack planes) were observed around the periphery and imply that propagation was due to rotating-bending fatigue. The PL report stated that no material defects were observed on the shaft outer surface during the failure analysis that would have contributed to the failure initiation.
PL also noted that the bearing contained minor damage that was considered collateral damage from the failed shaft. There was no misalignment of the roller paths on the inner or outer races. In addition, the pillow block base and bolt holes were not worn, which suggested the base was secured during operation.
Findings presented in the PL report lead to a review of fan belt tension, which was initially identified as a potential failure mode. Per Maintenance procedure MA-0C-86103-100, "Diesel Generator Fan Belt Replacement," belt tension is set at 60 +/- 2Hz (potential for a maximum hub load of -6350 lbs). This setting was discussed with Engine Systems Incorporated (ES!) and found to be 37% higher than necessary for the fan horsepower requirement. Technical Evaluation 01686101-06 concluded 47.4 ±.2 Hz (3700 lbs hub load equivalent) is an appropriate setting.
Structural Integrity Associates (SIA) was consulted to perform a stress analysis to determine potential causes that could further explain why the fan shaft failed. Additionally, they were asked to analyze fatigue crack Initiation and growth, and to determine at what point in the EDG-2 operating cycle the fan shaft had a remaining life of 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> (EDG mission time).
Structural Integrity Associates Report Summary SIA investigations revealed that crack growth to failure occurred during the last test run (less than 1.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />).
Beachmarks, features that typically define stops and starts of fatigue crack growth, combined with simulated crack growth data show that the shaft failure would have happened during the load test performed on July 28, 2014. The remaining shaft life would have met the 24-hour mission time for four test cycles or 43 days prior to shaft failure.
SIA investigations indicated that using a belt tension of 60 hertz corresponds to a hub load of 5955 lbs. Using nominal design values for shaft diameter and stress concentration factors (SCF) would produce stresses in the reduced (shaft groove transition) section of the shaft of 19.66 ksi. This stress is at or just below the endurance limit of the shaft material. Endurance limit is defined as the amplitude (or range) of cyclic stress that can be applied to the material without causing fatigue failure. A slight increase in stress for whatever reason would cause the predicted fatigue life to move from infinite hours of operation to approximately 250 hours0.00289 days <br />0.0694 hours <br />4.133598e-4 weeks <br />9.5125e-5 months <br /> (1x107 cycles) based on a change in stress amplitude from 19.66 ksi to 20.5 ksi. This number of cycles corresponds to about 174 tests. A smaller stress increment above the 19.66 ksi, or smaller, non-continuous time above a stress of 19.66 ksi (e.g., fan starts at full belt tension; over time belt loosens (decays) slightly to further reduce stress below 19.66 ksi) results in only a portion of the run time being accumulated towards the total number of cycles required to initiate a crack. Any fan run time with stress amplitudes less than the endurance limit does not accumulate time towards the number of hours to crack initiation. Only those times when stress amplitudes equal or exceed the endurance limit, 20.5 ksi, result in the accumulation of cycles that contribute to crack initiation time. It was concluded that if the full allowable tolerance for belt frequency of plus 2 Hz allowed by procedure was utilized during the 2-year period used for replacement or re-tensioning of the belt, the endurance limit could have been slightly exceeded (approximately 20.96 ksi) until the belt tension decays back below the endurance limit. The EDG was started over 350 times, from the time the maintenance procedure was implemented in 2005 with the revised belt tension, until the time of the shaft failure in 2014.
SIA concluded that at the loads the shaft might experience, fatigue life (both initiation and growth) would be very sensitive to small changes in the load or Stress Concentration Factor (SCF), a dimensionless number used to quantify how concentrated the stress is in a material. The stress amplitude at some point in time must have been higher than 19.66 ksi or the SCF is higher than three for the crack to initiate. SCF is affected by the dimensions of shaft transitions, such as the bearing groove; the smaller the radii or fillet, the higher the SCF.
Based on the number of hours of run time accumulated from the time the belt tensioning procedure was changed in 2004 to the time of actual failure, (1 .8x107 cycles) where the maximum endurance limit for this material is expected to be approximately lx107 cycles, it was concluded that the stress amplitude was only exceeding the endurance limit for limited periods of the run cycles.
The above stress values were computed based on a nominal shaft diameter of 3.0" and use of a conservative stress concentration factor (SCF) of 3 that accounts for any fabrication tolerances in the shaft or groove transitions. The SIA evaluation also provided a sensitivity study to compare changes in SCF values against variations in transition radius to demonstrate that the conclusions obtained using the nominal design values for diameter and SCF still bound the impacts from the as-measured conditions provided in the PL report with an average shaft diameter of 2.939" and average radius at the transition of 0.1807". This study demonstrated that the 6 percent increase to stress due to a smaller as-measured shaft diameter had substantially less impact than the stress improvement gained if the as-measured radius is used (>15 percent reduction in SCF).
Therefore, the maximum stresses calculated above using the nominal shaft diameter of 3" and conservative SCF value of 3 remains conservative.
SIA determined the potential causes of rotational bending fatigue as: 1) Hub loading exceeded the material properties, or 2) existence of a defect that produced a stress riser at the groove location (more notched or angular than designed, or presence of a very fine, indiscernible scratch). Either condition could result in stress conditions that accumulated cycles above the endurance limit.
Cause of Event
When reviewing the timeline for potential causes in support of SIA's conclusions, two instances were found worth considering, a fan shaft failure in 1993 and added belt tension in 2005. First, the shaft failure in 1993 demonstrates how susceptible the bearing groove locations are to imperfection. Questions regarding design details and manufacturing practices were discussed with ESI. Design information, such as dimensions, is limited to vendor manual references, and specifics of bearing retaining groove, as stated by ES!, was left at the discretion of the machinist.
Second, the 2005 change in tension frequency to 60 (+/- 2) Hz is important because it increases the likelihood of the shaft exceeding the endurance limit due to reducing stress margin. However, the conclusion from the SIA investigations is that the 24-month frequency for adjusting belt tension to 60 Hz, although high, should not have resulted in the shaft failure. It was noted that 60 Hz corresponds to approximately 19.66 ksi of stress, which is at or just below the endurance limit. if the shaft was consistent in diameter and stress relief contour profile throughout, it could effectively operate over an infinite number of cycles. Therefore, the plausible failure mode is attributed to an undetected imperfection or deficiency such as a scratch or nick that increased the stress above the endurance limit.
Furthermore, the EDG sets are adapted from transportation engines, The original Equipment Manufacturer Design (EMD) was qualified for safety-related service and subsequent parts procured through companies such as ESI. The shafts in service as well as spares were provided to the plant in the late 1960s. Discussions with ESI indicated limited information (dimensions and material) was on file for shaft part number 8441753.
The apparent cause of the failure was found to be higher than average stress concentration factor due to an undetected deficiency at the grooved location.
The high cycle fatigue mechanism as evaluated by PL and SIA concluded that either excessive, periodic hub loading (belt tension) or a notch type defect or very fine indiscernible scratch that went undetected were the most likely failure mechanisms. It is important to note that the SIA report data approximates material properties of the shaft. Design and fabrication variables such as material hardening, surface finish, EMD shaft design process, machinist proficiency, etc. are not readily available; however, the impacts from these variables were addressed in the SIA report using a sensitivity study.
ESI indicated that limited detail is available on the shaft design, Including control of parameters such as stress relief profiles, and finish. Further, control of the groove profile was left to the skill of the fabricating machinist.
Therefore, a deficiency in the groove profile or very small undetected surface defect is the most likely cause for the shaft failure.
Contributing to the failure was that belt tension, as outlined in station procedure, did not provide adequate margin necessary to address higher than expected stress risers et the notch.
In 2005, Maintenance procedure MA-00-86103-100 was issued to specify the 60 Hz belt tension setting, but no technical evaluation was performed or vendor document referenced to review this setting. The 60 Hz belt tension setting is excessive; which resulted in:
A. Reduced margin to the shaft stress design limits.
B. A possibility that the belt was tensioned to a point where hub loading combined with stress risers originating from control of the groove configuration exceeded the shaft material fatigue endurance limit.
The following immediate actions were taken:
- Replaced EDG-2 fan shaft.
- Performed Ultrasonic Testing of the EDG-1 fan shaft.
- Obtained failure Analyses for failed fan shaft (PL and SIA).
- Performed technical evaluation to specify correct fan belt tension.
- Performed Causal Analysis
- Performed a "Deep Dive" check-in assessment of station EDG maintenance and operating strategies.
Corrective Actions
In order to address the Apparent Cause the following actions were (or are being) taken:
- Repaired EDG-2 failed fan shaft by completing a work order to replace the shaft.
di The EOG-1 fan shaft will be replaced by May 15, 201g.
In order to address the Contributing Cause the following actions were (or are being) taken:
- Technical Evaluation was performed to determine the correct belt tension for both EDG-1 and EDG-2
- Re-tensioned EDG-1 and 2 fan belts based on Technical Evaluation.
- Revise station procedures to incorporate correct fan belt tension specified in Technical Evaluation.
- Technical Evaluation was performed by SIA for the failed cooling fan shaft.
Previous Occurrences
DR 93-387 EDG-2 Fan Shaft Failure - in August 1993, the EDG-2 fan shaft failed during routine testing. The cause of the failure was a combination of two factors: 1) the existence of a weld overlay in the vicinity of the bearing sleeve attachment that created an extremely hard subsurface layer, and 2) a machined groove immediately adjacent to the bearing sleeve attachment that extended to a depth corresponding to this extremely hard zone. The result was the initiation of a crack that propagated by torsional fatigue until failure.
Component Data Component IEEE 805 System ID IEEE 803A Function Emergency Diesel Generator EK DG