IR 05000321/2008008
ML082550033 | |
Person / Time | |
---|---|
Site: | Hatch |
Issue date: | 09/06/2008 |
From: | Wert L Division Reactor Projects II |
To: | Madison D Southern Nuclear Operating Co |
References | |
IR-08-008 | |
Download: ML082550033 (28) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION eptember 6, 2008
SUBJECT:
EDWIN I. HATCH NUCLEAR PLANT - NRC SPECIAL INSPECTION REPORT 05000321/2008008, 05000366/2008008
Dear Mr. Madison:
On July 24, 2008, the Nuclear Regulatory Commission (NRC) completed a special inspection at your Edwin I. Hatch Nuclear Plant, Units 1 and 2. The enclosed report documents the inspection findings which were discussed on July 24 with you and other members of your staff.
On July 18, 2008, NRC Region II management established a Special Inspection Team using the guidance contained in Management Directive 8.3, NRC Incident Investigation Program. The Special Inspection Team was chartered to identify the circumstances surrounding the degradation of an emergency diesel generator (EDG) coupling on July 12, 2008, resulting in the 1B EDG being declared inoperable.
This inspection was performed in accordance with Inspection Procedure 93812, Special Inspection, and focused on the areas discussed in the inspection charter described in the report. The inspection examined activities conducted under your licenses as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your licenses. The inspectors reviewed selected procedures and records, conducted field walkdowns, observed activities, and interviewed personnel. Based on the results of this inspection, no findings of significance were identified.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document
SNC, Inc. 2 Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (The Public Electronic Reading Room).
Sincerely,
/RA/
Leonard D. Wert, Jr., Director Division of Reactor Projects Docket Nos.: 50-321, 50-366 License Nos.: DPR-57, NPF-5
Enclosure:
Inspection Report 05000321/2008008, 05000366/2008008 w/Attachments 1: Supplemental Information 2: Event Timeline 3: EDG Inspection Results Summary 4: Coupling Photographs 5. Background Information
REGION II==
Docket Nos: 50-321, 50-366 License Nos: DPR-57 and NPF-5 Report No: 05000321/2008008, 05000366/2008008 Licensee: Southern Nuclear Operating Company, Inc.
Facility: Edwin I. Hatch Nuclear Plant Location: Baxley, Georgia 31513 Dates: July 22-24, 2008 Inspectors: J. Brady, McGuire Senior Resident Inspector, Team Leader P. Niebaum, Hatch Resident Inspector S. Crane, Materials Engineer, New Reactor Office Approved by: Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000321/2008-008, 05000366/2008-008; 07/22/2008-07/24/2008; Edwin I. Hatch Nuclear
Plant, Units 1 and 2; Special Inspection The special inspection team inspection was conducted by a senior resident inspector, a materials engineer and a resident inspector.
A. Inspection Results The licensees final operability determination (OD) and compensatory measures taken provided reasonable assurance of operability until the emergency diesel generator (EDG)couplings were replaced. The licensee revised the first two versions of the OD to incorporate newly identified information, to address inspector questions and to provide reasonable assurance of operability.
Based on interviews with the licensee, coupling gland cracks were observed by licensee maintenance technicians as early as 1994 but were not documented in licensee inspection records. As a result, these observations of EDG coupling cracking were not evaluated.
The inspectors found that the licensee did not have any documented prior indication of problems with the couplings; nothing had been communicated through operating experience about these couplings; and there were not any vendor bulletins that would have specifically alerted the licensee to the degradation of the 1B EDG coupling.
The potential for unexpected or catastrophic failure of the EDG couplings is a significant problem which could result in the EDG being incapable of performing its intended safety function. These couplings were used by at least four other utilities on their safety-related EDGs. This problem demonstrated the importance of proper periodic inspections to determine if the component will continue to perform the associated safety function. These inspections are of particular necessity where the shelf life and service life have not been specified. As seen in this application, storage of spare couplings in a warehouse actually reduced the shelf life. This reduced shelf life resulted in the spare couplings, purchased in 1988 and stored in the warehouse, being considered unusable by the licensee.
REPORT DETAILS
Event Description On July 12, 2008, the licensee was conducting a 24-hour surveillance run on the 1B EDG.
During this surveillance, the licensee observed high vibration on the EDG. The licensee stopped the EDG and subsequently determined that the coupling between the engine and the generator had significantly degraded. The licensee performed troubleshooting, which included additional vibration monitoring and received vibration acceptance criteria from the EDG vendor.
The licensee then ran the EDG as part of a trouble shooting plan. The vibration during this subsequent run exceeded the established acceptance criteria resulting in the 1B EDG being declared inoperable. The licensee isolated the problem to a degraded flex coupling which exhibited cracking on the 1B EDG. Evaluation of the other EDGs for common problems identified cracking of the 1C EDG flex coupling and it was declared inoperable. All diesels exhibited some coupling degradation. The 1B EDG coupling gland (see Photograph 1) had separate circumferential cracks that extended approximately 180 degrees on both sides of the coupling. This coupling had been furnished with the EDG when the EDG was installed in 1971.
The licensee replaced the coupling on July 16, 2008, and subsequently declared the 1B EDG operable. As of August 11, 2008, all five EDG couplings had been replaced.
Special Inspection Team (SIT) Charter Based on the criteria specified in Management Directive 8.3, NRC Incident Investigation Program, and Inspection Procedure 71153, Event Follow-up, a special inspection was initiated in accordance with Inspection Procedure 93812, Special Inspection. The objectives of the inspection listed below are addressed in the identified sections:
1. Assess the available common cause analysis for the 1B Emergency Diesel Generator (EDG)
coupling and operability evaluation for the remaining EDGs. (Section 4OA5.1)
2. Develop a sequence of events, including applicable management decision points from the
time of the coupling failure. (Section 4OA5.2)
3. Assess any reviews, preventative maintenance, or evaluations developed to support
continued operation of the coupling prior to failure. (Section 4OA5.3)
4. Review licensee documents to assess if the licensee knew or should have known that a
coupling failure was about to occur. Specifically, assess the following areas. (Section 4OA5.4)a. Operational decision making b. Operational experience (internal and external)c. Vendor information on expected service life, recommended preventative maintenance, and if any bulletins or addendums were issued d. Impact on Maintenance Rule implementation
5. Review and assess the previous 12 months of surveillance test data for all EDGs. (Section
4OA5.5)
6. Collect data necessary to support completion of the significance determination process, if
applicable. (Section 4OA5.6)
7. Identify any potential for generic safety implications. (Section 4OA5.7)
OTHER ACTIVITIES
4OA5 Other Activities -Special Inspection
.1 Review of the Licensees Common Cause Analysis for the 1B EDG Coupling and
Operability Determination for the Remaining EDGs
a. Inspection Scope
The inspectors reviewed the licensees three revisions of the operability determination (OD) and common cause analysis to determine if the licensee had adequately considered all available information concerning the degraded coupling and if their efforts to gather information pertaining to the couplings on the other EDGs were sufficiently comprehensive to support making an operability determination for the remaining diesel generators. The inspectors interviewed licensee staff and a representative of the coupling supplier; observed the degraded condition of the 1B and 1C EDG couplings which were removed from the EDG, inspected a replacement coupling, viewed an unused coupling that had been stored in the warehouse since 1988; and reviewed the inspection and testing/results of the installed couplings for the 1A, 1C, 2A, and 2C EDGs. Documents reviewed are listed in the Attachment.
b. Findings and Observations
At Hatch, there are five EDGs identified as 1A, 1B, 1C, 2A, and 2C. The engine of each EDG is a Fairbanks Morse turbo-charged two-cycle engine that develops 4500-horse power at 900 RPM. This engine has 12 cylinders using an opposed piston design. Each EDG had a Falk size 62A Airflex coupling (see Photograph 2) installed between the engine and the generator. The coupling on the generator side was exposed and allowed for limited visual inspection. The coupling was designed to cushion shock and limit torque fluctuations, accommodate radial, angular, and axial misalignment, dampen vibration, and operate without lubrication. This coupling consisted of steel inner and outer rims that were permanently bonded to a polyester corded natural rubber gland using a rubber-to-metal adhesive. The size 62A gland, which resembles an automobile tire, is comprised of eight plies of polyester fiber corded natural rubber which is layered on a 45 degree alternating bias around a membrane for sizing and shaping. This coupling is mounted to the flywheel on the engine side and to the generator shaft on the generator side. These EDGs and associated couplings were installed during original plant construction. All the installed EDG couplings were manufactured around 1970.
The licensee conducted detailed inspections of the 1A, 1C, 2A and 2C EDG coupling glands. The inspections required the licensee to observe the location, length, and depth of all cracks on the generator side of the coupling. If the depths of the cracks were less than or equal to 1/4 inch (), the coupling was considered acceptable. Following completion of the EDG operation, an additional inspection was performed on the previously identified cracks to observe any changes to the cracks length or depth. The results of these inspections are summarized in Attachment 3. The licensee observed cracking on the 1A, and 2A coupling glands which did not exceed the 1/4 crack depth acceptance criterion. The 1C EDG coupling gland had several cracks that exceeded the 1/4 crack depth acceptance criterion in multiple locations. The 1C EDG was declared inoperable until the coupling was replaced. An additional inspection criterion required a sample of the rubber to qualitatively assess the elasticity/pliability of the rubber when compared to a rubber sample from the degraded 1B EDG coupling. This criterion was later determined not to provide any meaningful information and was not used to support the OD. Vibration readings on the 1C EDG engine block and both generator bearings were also recorded during these operations. Vibration levels did not exceed the vendor limits for continued operation. Also, the licensee observed the cracks during EDG operation, both loaded and unloaded, to determine if the cracks experienced any growth or expansion under these conditions. The crack was observed to expand on the degraded 1B EDG coupling, which the licensee used as an indicator for potential failure.
Based on the common coupling design and similar component age, the inspectors concluded that the couplings represented a potential common mode failure mechanism.
Based on the manufacturer-supplied 1/4 crack depth limit, combined with other acceptance criteria (no crack growth, normal vibration, etc), the inspectors concluded that the final OD provided reasonable assurance of operability until the existing couplings on the 1A, 2A, and 2C EDGs were replaced. Additionally, the inspectors concluded that the intermittent service of the EDG couplings contributed to their extended service life (approximately 39 years). This is further discussed in Section 4OA5.7and Attachment 5.
Some cracking of the EDG couplings had been observed since 1994 (see Section 4OA5.4). These cracks did not result in significant coupling degradation until 2008.
This information, along with the OD and vendor input supported the licensees near term replacement schedule of the couplings. The licensees OD also included three compensatory measures and one corrective action. The compensatory measures were to perform crack depth measurements after each EDG run to determine if crack depth had exceeded the 1/4 crack depth acceptance criterion, to perform vibration monitoring during each EDG run, to evaluate the trend to determine if additional degradation was occurring and to inspect the engine side of the remaining EDG couplings expeditiously.
The corrective action was to expedite replacement of the couplings for the 2C, 1A, and 2A EDGs. The order of the replacement was based on crack depth measurement. The 1B and the 1C EDG couplings were replaced on July 16 and July 23 respectively.
Although the licensees final OD was adequate, the inspectors identified the following weaknesses which resulted in the licensee having to revise the OD. The previous versions did not fully support operability of the remaining EDGs.
- The initial coupling inspections were performed with the coupling guards installed which allowed for limited visual inspection of portions of the generator side of the coupling. These visual inspections lacked sufficient rigor for the licensee to assess the condition of the entire generator side of the coupling. A subsequent OD revision required inspections of the generator side of the coupling with the coupling guards removed.
- The licensee did not have plans to inspect the engine side of the couplings to support operability of the other EDGs. This was based on the licensees conclusion that there were no significant differences in the condition of the generator side and engine side of the degraded 1B EDG coupling. However, the engine side of the 1B EDG coupling gland did have more degradation. Therefore, the decision not to inspect the engine side of the remaining coupling glands lacked sufficient justification. Further, the spare couplings in the warehouse, purchased in 1988, had age related cracking on both the generator side and engine side. This condition was similar to that observed for the degraded 1B EDG coupling and reinforced that the licensees decision not to inspect the engine side of the couplings lacked sufficient justification. The licensee did include inspection of the engine side of the coupling in the final OD.
- As a result of the licensees inspection of the 2C diesel generator coupling, cracking was observed around a 60 degree radius which the licensee described as de-lamination cracking. An acceptability criterion was established by the licensee that, after each EDG run test, the crack would be measured, and if the crack had not grown to greater than 120 degrees circumferentially, the EDG would be considered operable. There was no basis to support that this crack growth criterion would ensure EDG operability. However, once the coupling supplier provided the 1/4 crack depth acceptance criterion, this criterion was replaced in the final OD.
- No analytical calculation for the 1/4 crack depth acceptance criterion was available.
However, based on discussions with the coupling supplier, the 1/4 crack depth would not have significantly affected the strength of the coupling gland. Although this criterion lacked an analytical basis, the inspectors concluded that, combined with the other acceptance criteria (no crack growth, normal vibration, etc), the final OD was acceptable. The licensee sent the degraded 1B and 1C EDGs couplings offsite for analysis and materials testing to obtain additional materials information on the rubber portion of the coupling. The results were not available prior to the end of the inspection.
.2 Sequence of Events, Including Applicable Management Decision Points from the Time of
the Coupling Failure The inspectors created a sequence of events starting on July 12, 2008. Included in this sequence of events are the applicable licensee management decisions as they pertain to troubleshooting the 1B EDG coupling failure and the OD for the remaining EDGs. A detailed timeline of the relevant sequence of events is provided in Attachment 2.
.3 Assessment of Reviews, Preventative Maintenance, and Evaluations Developed to
Support Continued Operation of the Coupling Prior to Failure
a. Inspection Scope
The inspectors reviewed maintenance work order history and the inspection records for the last two performances of licensee procedure, 52SV-R43-001-0, Diesel, Alternator, and Accessories Inspection. For the 1B EDG, the inspectors reviewed the inspection records for the previous three performances of 52SV-R43-001-0. Additionally, the inspectors interviewed licensee personnel that had participated in several previous EDG maintenance activities and were knowledgeable of the condition of the couplings. A summary of EDG coupling inspections that show the scope and results of the coupling inspections, performed before and after the 1B EDG coupling failure, is contained in
3.
b. Findings and Observations
Introduction:
An unresolved item was identified pertaining to the biennial preventative maintenance inspections performed on the EDG couplings. Cracks in the coupling gland that were identified during routine maintenance inspections were not documented in the associated plant records since 1994.
Description:
Based on information provided by the licensee and interviews with licensee personnel, it was concluded that preventive maintenance inspections had observed cracks in the couplings as early as 1994. The inspectors review of the coupling inspection records for 52SV-R43-001-0 indicated that no damage existed, as shown in
3. The procedure directed the coupling be visually examined for damage,
deterioration, and loose bolts. The procedure did not contain any specific acceptance criteria pertaining to coupling damage or deterioration. Because the earlier observation of coupling cracks were not documented, no actions were taken to determine if the observed cracks were acceptable or required further evaluation to determine operability.
The degradation of the EDG coupling subsequently resulted in the inoperability of the 1B and 1C EDGs. The licensee is performing a root cause investigation for the EDG coupling problems and the contributing causes surrounding this event. This issue is unresolved pending NRC review of the licensees completed root cause investigation, and is designated as unresolved item (URI) 05000321,366/2008008001, Review of EDG Coupling Root Cause Evaluation.
Additional information relevant to this charter item is discussed in other report sections.
.4 Review Licensee Documents to Assess If the Licensee Knew or Should Have Known
That a Coupling Failure Was About to Occur
a. Inspection Scope
The inspectors reviewed the documents in Attachment 1 and internal and external operational experience that was available to the licensee prior to the failure of the 1B EDG coupling to assess if the licensee knew or should have known that a coupling failure was about to occur. The inspectors reviewed the prior two-year preventative maintenance (PM) documentation for the each of the diesel generator couplings, reviewed the surveillance tests to determine if there were any observations relating to vibration or the performance of the EDG coupling, and discussed with the licensee the results of a licensee performed search for industry related experience with the same coupling.
b. Findings and Observations
The inspectors found that the licensee had no documented prior indication of problems with the couplings; nothing had been communicated through external operating experience (OE); and no vendor bulletins would have alerted the licensee to the degradation of the EDG couplings. However, as discussed in Section 4OA5.3, the lack of documentation and evaluation of previous EDG coupling gland cracks was identified as a URI.
Internal OE. Based on a review of licensee documents, the licensee was aware that the 1B EDGs bearings had shown a change in vibration levels since June of 2007 and were listed in the Alert range in the monthly maintenance engineering predictive maintenance status report. The licensee review of this condition did not consider the vibration change as a precursor to any operability problem. As such, the licensee continued with routine vibration monitoring. In addition, the licensee determined cracking of the coupling gland has been observed as early as 1994. These observations were not communicated or documented in the preventative maintenance procedure or in a condition report.
External OE. The inspectors discussed the results of a licensee performed search for industry related experience with this coupling. No relevant industry operating experience was identified. The licensee also contacted the Fairbanks Morse Diesel Generator Owners Group and found that the following licensees had similar experience with Falk Airflex couplings installed on their EDGs. As discussed below, each case of degradation was identified by visual observations or while diagnosing high EDG vibration and bearing wear. None of these licensees identified the degradation through vendor disseminated information.
- In 2002, Millstone identified minor cracks in a coupling. No established inspection criteria existed at that time. When Falk was contacted about the cracks, the cracks were deemed insignificant and Falk informed Millstone that the coupling could be used for another two years. The coupling was replaced in 2004. Originally, Millstone ordered two replacement couplings. However, the licensee determined that the second coupling was acceptable for use. Therefore, the other replacement coupling was kept as a spare until Hatch acquired it as a replacement for the degraded 1B EDG coupling. This coupling was manufactured in 2003. Hatch personnel performed a visual inspection prior to installation. The inspectors also did not observe any surface cracks and it appeared like relatively new rubber compared to the 1B EDG coupling.
- Arkansas Nuclear One (ANO) discovered a cracked coupling gland in 1991. Because no vendor data was available, ANO contacted Falk. Falk informed ANO that the cracking was from the rubber drying (an aging mechanism) and advised ANO that the coupling should be replaced. ANO now replaces their couplings when cracks are observed. ANO replaced couplings in 1991 and again recently in 2008 due to identified cracks.
- In May 2002, Fermi was troubleshooting failed generator bearings and high vibration issues on two EDGs. A contractor identified the coupling as a possible contributor to the high vibration. The contractor noted that, as the rubber in the coupling ages, its stiffness increases and this may have shifted the axial resonance of the rotor/coupling assembly closer to the running speed of the EDG. Falk informed Fermi that the in service life of a coupling gland was approximately 5 to 10 years based on their experience. Falk informed Fermi that the Shore A Hardness, a measure of relative resistance to indentation (or flexibility), of a new gland was 60+/-5. Since the flexibility of rubber changes over time, the indentation reading may also change over time. Fermi performed a Shore A hardness test of the couplings for the two EDGs which was measured at 74 and 75. The results indicated a decrease in the flexibility of the rubber. Also, because there was evidence of checking and surface distress, Fermi decided to replace all four couplings regardless of their contribution to the vibration levels. Fermi has since implemented a program to inspect, test, and replace the couplings periodically.
Coupling Supplier Information: The inspectors discussed with both the licensee and coupling supplier the availability of information on expected service life, recommended preventative maintenance, and vendor issued bulletins and addendums. The inspectors also reviewed the information in the licensees EDG owners manual, as well as information provided by the vendor after the 1B EDG coupling failure.
The inspectors reviewed the licensees EDG owners manual and found that the only information that was included for the EDG coupling was the original 1960 Falk Guide on the Installation of Airflex couplings and the 1960 Falk installation guide for Airflex Couplings with Floating Shaft. With respect to maintenance, the guidance instructs that the rubber glands be protected from oil and other substances which deteriorate rubber by cleaning the gland before installation. In addition, it recommends periodic rechecking of the alignment due to the settling of foundations. The inspectors observed the EDGs were mounted to seismically qualified concrete pad and concluded that foundation settling was not a factor. Furthermore, by procedure, the coupling was aligned following coupling removal.
The licensee did not receive any information on expected service life, recommended preventative maintenance, or vendor issued bulletins and addendums from the EDG vendor, Fairbanks Morse.
.5 Review and Assess the Previous 12 Months of Surveillance Test Data for All EDGs
a. Inspection Scope
The inspectors reviewed the surveillance tests listed in the attachment to determine if there were any observations relating to vibration or the performance of the EDG coupling. The inspectors reviewed this vibration data and interviewed the licensee vibration engineers to determine if there was any indication of an adverse trend.
b. Findings and Observations
There were no comments in the surveillance procedures that related to the performance of the EDG couplings. As discussed Section 4OA5.4, a change in vibration (decreasing)was noted; however, the licensee determined the change did not impact operability and continued with routine vibration monitoring. Based on the inspectors review of the available test data, the inspectors did not identify any adverse trend that would indicate a precursor to degradation of the 1B EDG coupling.
.6 Collect Data Necessary To Support Completion Of The Significance Determination
Process (SDP), If Applicable The inspectors collected relevant information and data on past operability of the EDGs.
The licensees root cause analysis will be reviewed for additional information to support the SDP analysis.
.7 Generic Safety Implications
a. Inspection Scope
The inspectors performed a literature search in order to fully assess the generic safety implications from the EDG coupling degradation. The results from the literature search are contained in Attachment 5 to this report, with some key points applicable to this application being summarized below.
b. Findings and Observations
Similar EDG couplings could represent a generic safety implication for other utilities if precursor cracking was not identified and evaluated. The potential for unexpected or catastrophic failure of the EDG couplings is a significant problem which would result in the EDG being incapable of performing its intended safety function. These couplings are used by at least four other utilities on their safety-related EDGs. A periodic inspection to determine if the component will continue to perform the associated safety function is of particular necessity where the shelf life and useful life have not been specified. In addition, the storage of the coupling as a spare part in the warehouse actually reduces the service life so shelf life and useful life requirements will likely be different.
The shelf life and useful life of natural rubber are dependant not only on mechanical loading, but also on temperature, thermal dissipation, and the environment. The shelf life can be greatly degraded by exposure to oxygen, ozone, light, heat, humidity or oils and solvents. According to Falk, the shelf life of the coupling was approximately five to seven years, but under ideal conditions the shelf life can be as long as ten years without an appreciable affect on the components properties. However, adverse conditions such as those previously mentioned, could shorten the couplings shelf life to less than the stated five to seven years. In addition, there are other applications, such as gaskets in safety-related systems, will be subject to the same aging and fatigue phenomenon that were observed with the couplings.
The Falk representative indicated that the expected service life for a coupling in continuous use is approximately 10-20 years. However, several factors may increase the useful life of the coupling. As mentioned in Attachment 5, periodic cycling of the coupling will release antiozonants to the surface of the rubber and increase the rubbers resistance to ozone attack. In addition, for natural rubber that exhibits strain crystallization, the unrelieved static strain on the assembled coupling has a beneficial effect on fatigue properties, and rest periods between loading dissipate stored energy in the form of heat that increases the fatigue life of the coupling.
4OA6 Meetings, Including Exit
On July 24, 2008, the inspectors presented the inspection results to Mr. Dennis Madison and other members of the plant staff. The inspectors reviewed information that could be considered proprietary information and was not included in this report.
ATTACHMENT 1:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee personnel
- S. Tipps, Principal Licensing Engineer
- D. Madison, Site Vice President
- W. Bargeron, Plant Manager
- G. Johnson, Engineering Director
- S. Crosby, Maintenance Supertinendent
- H. Mirzakhani, Engineering Support (Diesel Engineer)
- M. Torrance, Maintenance Engineer
- K. Underwood, Performance Improvement
- S. Soper, Engineering Support Manager
- G. Brinson, Operations Manager
- M. Crosby, Engineering Supervisor
- J. Lonting, Corporate Licensing
Other Personnel
NRC personnel
- J. Moorman, Deputy Director, DRP, RII
- S. Shaeffer, Branch Chief, DRP, RII
- K. Kennedy, Director, Division of Reactor Safety, RII
- R. Bernhard, Senior Reactor Analyst, RII
- J. Hickey, Senior Resident Inspector