IR 05000298/2009008

IR 05000298-09-008; on 02/23/09 - 05/12/09; Cooper Nuclear Station; Special Inspection Into Emergency Diesel Generator Lubricating Oil Pipe Cracking and Amphenol Connector Deficiencies

ML091610324
Person / Time
Site:	Cooper
Issue date:	06/09/2009
From:	Caniano R Division of Reactor Safety IV
To:	Minahan S Nebraska Public Power District (NPPD)
References
IR-09-008
Download: ML091610324 (59)

Text

UNITED STATES NUC LE AR RE G UL AT O RY C O M M I S S I O N R E GI ON I V 612 EAST LAMAR BLVD , SU I TE 400 AR LI N GTON , TEXAS 76011-4125 June 9, 2009 Stewart B. Minahan, Vice President-Nuclear and CNO Nebraska Public Power District 72676 648A Avenue Brownville, NE 68321 Subject: COOPER NUCLEAR STATION - NRC SPECIAL INSPECTION REPORT 05000298/2009008

Dear Mr. Minahan:

On May 12 2009, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Cooper Nuclear Station to evaluate the facts and circumstances surrounding emergency diesel generator lubricating oil discharge pipe cracking and Amphenol connector deficiencies. The enclosed report documents the inspection findings that were discussed on May 12, 2009, with Mr. B. OGrady, Site Vice President, and members of your staff.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed plant personnel.

Based upon the risk and deterministic criteria specified in NRC Management Directive 8.3,

"NRC Incident Investigation Program," including the effectiveness of corrective actions in response to previous similar events, NRC initiated a special inspection in accordance with Inspection Procedure 93812, "Special Inspection." The basis for initiating the special inspection and the focus areas for review are detailed in the Special Inspection Charter (Attachment 2).

The determination that the inspection would be conducted was made by the NRC on January 30, 2009, and the onsite inspection started on February 23, 2009.

On January 27, 2009, a crack developed in the Emergency Diesel Generator 1 lubricating oil discharge piping flange weld heat affected zone. A previous failure on Emergency Diesel Generator 2 occurred in the same location on February 13, 2008. Based on these failures, the NRC had concerns regarding the effectiveness the corrective actions implemented by your staff for the lubricating oil discharge pipe cracks.

In addition, your staff had several challenges to operating their emergency diesel generators because of the deficiencies related to Amphenol connectors that occurred on January 15, 2008; April 21, 2008, November 10, 2008; and January 31, 2009. The NRC also had concerns whether your staff had effectively considered the extent of condition, extent of cause, and common cause of the Amphenol connector deficiencies.

The team concluded, as did your staff, that ineffective root cause analyses contributed to recurrence of both the lubricating oil discharge pipe crack and to Amphenol connector

Nebraska Public Power District -2-deficiencies. NRC identified concerns related to the rigor of root cause analyses, technical training, craft skills, and procedure instructions. Previously, NRC identified similar concerns during the reviews of tape splices on environmentally qualified equipment, as documented in Inspection Report 05000298/2000007 (ADAMS ML003778318), and staking of motor-operated valve pinion gears, as documented in Inspection Report 05000298/2006005 (ADAMS ML070360639).

This report documents six NRC-identified violations of very low safety significance (Green). All six of these findings were determined to involve violations of NRC requirements. However, because of their very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as a noncited violations, consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the violations or the significance of the noncited violations, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 612 E. Lamar Blvd, Suite 400, Arlington, Texas, 76011-4125; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at the Cooper Nuclear Station facility. In addition, if you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at Cooper Nuclear Station. The information you provide will be considered in accordance with Inspection Manual Chapter 0305.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, and its enclosure, will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRCs document system (ADAMS).

ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Roy J. Caniano, Director Division of Reactor Safety Docket: 50-298 License: DPR-46

Enclosure:

NRC Inspection Report 05000298/2009008 w/Attachments:

Attachment 1: Supplemental Information Attachment 2: Special Inspection Charter Attachment 3: Lubricating Oil Piping Event Timeline with Vibration Issues Attachment 4: Amphenol Connector Event Timeline Attachment 5: Lubricating Oil S-Pipe Drawing and Fracture Pictures Attachment 6: Amphenol Connector Pictures and Diagrams

Nebraska Public Power District -3-

REGION IV==

Docket: 05000298 License: DPR-46 Report: 05000298/2009008 Licensee: Nebraska Public Power District Facility: Cooper Nuclear Station Location: 72676 648A Avenue Brownville, NE 68321 Dates: February 23 through May 12, 2009 Inspectors: G. Pick, Senior Reactor Inspector, Engineering Branch 2, Division of Reactor Safety S. Makor, Reactor Inspector, Engineering Branch 1, Division of Reactor Safety M. Baquera, Reactor Inspector, Plant Support Branch 2, Division of Reactor Safety Accompanying D. Rudland, Senior Materials Engineer, Office of Nuclear Reactor Research Personnel:

C. Alldredge, Reactor Engineer, Nuclear Safety Professional Development Program, Plant Support Branch 2, Division of Reactor Safety Approved By: Thomas R. Farnholtz, Chief Engineering Branch 1 Division of Reactor Safety-1-

SUMMARY OF FINDINGS

IR 05000298/2009008; 02/23/09 - 05/12/09; Cooper Nuclear Station; Special Inspection into emergency diesel generator lubricating oil pipe cracking and Amphenol connector deficiencies The report covered one week of onsite inspection and inoffice review through May 12, 2009.

Three regional inspectors and a materials engineer from NRC Headquarters performed the inspection. Six Green noncited violations were identified. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609,

"Significance Determination Process." Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

Green.

The team identified a noncited violation of 10 CFR Part 50, Appendix B,

Criterion XVI, "Corrective Action," for inadequate corrective actions to prevent recurrence of a significant condition adverse to quality. The licensee inappropriately changed the root cause for the Emergency Diesel Generator 2 lubricating oil discharge S-pipe failure on February 13, 2008, from high cycle fatigue to four piping overstress events. Consequently, the licensee implemented corrective actions that resulted in a high cycle fatigue failure of the Emergency Diesel Generator 1 lubricating oil discharge S-pipe on January 27, 2009. The licensee entered this deficiency in their corrective action program as Condition Report 2009-00098.

The performance deficiency involved the failure of the licensee to take adequate corrective actions to prevent recurrence of a significant condition adverse to quality. The finding was determined to be more than minor because it is associated with the mitigating systems cornerstone attribute of equipment performance and affected the associated cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 -

Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of problem identification and resolution associated with the corrective action program because the licensee did not thoroughly evaluate the Emergency Diesel Generator 2 failure such that the specified corrective actions addressed the causes of the failure P.1(c).

Green.

The team identified a noncited violation of 10 CFR Part 50, Appendix B,

Criterion V, "Instructions, Procedures, Drawings," regarding failure to follow the requirements of Procedure ENN-OP-104, "Operability Determinations," Revision 2.

Specifically, the team determined that operations personnel did not obtain necessary information to determine with reasonable assurance that Emergency Diesel Generator 2 remained operable and not subject to a common mode failure mechanism. The Emergency Diesel Generator 1 lubricating oil discharge S-pipe cracked as a result of vibrations in the X-direction and the common mode failure evaluation did not account for vibrations in the X-direction for Emergency Diesel Generator 2. Subsequent measurements confirmed that the vibrations remained within the normal operating range. The licensee documented this deficiency in Condition Report 2009-00655.

The performance deficiency associated with this finding involved the failure of operations personnel to perform an adequate operability assessment. The finding was determined to be more than minor because it would become a more significant event if left uncorrected in that the failure of Emergency Diesel Generator 2 by the same high cycle fatigue mechanism increased the likelihood that both emergency diesel generators could be inoperable concurrently. The finding affected the mitigating systems cornerstone. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because the licensee failed to use conservative assumptions when determining operability H.1(b).

Green.

The team identified a noncited violation of 10 CFR Part 50, Appendix B,

Criterion V, "Instructions, Procedures, Drawings," for the failure of maintenance personnel to initiate a condition report, as required by Procedure 0.5, "Conduct of The Condition Report Process," Revision 63, Section 7.1.3. Specifically, maintenance personnel failed to initiate a condition report for an adverse condition related to a significant change in vibration levels on Emergency Diesel Generator 1 between readings. The licensee documented this deficiency in Condition Report 2009-00694.

The performance deficiency associated with this finding involved the failure of maintenance personnel to initiate condition reports for adverse conditions as required by Procedure 0.5. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because the finding was associated with the human performance attribute of the mitigating systems cornerstone and affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. The finding affected the mitigating systems cornerstone. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with work practices because the licensee did not effectively communicate expectations regarding following procedures for initiating condition reports for adverse conditions H.4(b).

Green.

The team identified two examples of a noncited violation of Technical Specification 5.4.1.a for the failure of the licensee to provide work instructions appropriate to the circumstances. In the first example, the work orders generated to monitor vibrations on the emergency diesel generator lubricating oil discharge S-pipe did not have adequate quantitative acceptance criteria. The second example involved four instances of inadequate maintenance instructions, which resulted in deficiencies in emergency diesel generator Amphenol connectors. The licensee documented these deficiencies in Condition Reports 2009-01513 and 2009-02684.

The performance deficiency associated with this finding involved the failure to ensure maintenance personnel took the appropriate actions when performing maintenance. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because it is associated with the mitigating systems cornerstone attribute of equipment performance and it affects the cornerstone objective of ensuring availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because incorrect assumptions by the licensee regarding the skills and knowledge level of the craft resulted in maintenance procedures that had insufficient instructions H.1(b).

Green.

The team identified a Green noncited violation of 10 CFR Part 50,

Appendix B, Criterion XV, "Nonconforming Materials, Parts, or Components," for the failure of engineering personnel to review and accept a nonconforming pipe configuration. Specifically, when replacing the lubricating oil discharge S-pipe on Emergency Diesel Generator 1 for an extent of condition evaluation, engineers failed to evaluate critical characteristics and determine why the replacement characteristics did not match the installed configuration or why this configuration did not match the existing drawings. The licensee documented this deficiency in Condition Report 2009-00613.

The performance deficiency associated with this finding involved the failure to thoroughly evaluate and control configuration changes on the emergency diesel generator lubricating oil piping. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612,

Appendix B, because the finding was associated with the design control attribute of the mitigating systems cornerstone and affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04,

"Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because licensee personnel did not verify the critical characteristics of the replacement lubricating oil discharge S-pipe section, which had a different configuration than the pipe being removed for analysis H.1(a).

Green.

The team identified a noncited violation of 10 CFR Part 50, Appendix B,

Criterion XVI, "Corrective Action," for the failure to identify the root cause of a significant condition adverse to quality as required by Procedure 0.5.Root-Cause,

"Root Cause Analysis Procedure," Revision 8. Specifically, following an Emergency Diesel Generator 2 governor magnetic pickup unit Amphenol connector failure on April 21, 2008, the licensee incorrectly attributed the failure to a speed gear striking the magnetic probe without identifying that a faulty Amphenol connector caused voltage spikes. Consequently, the same defective Amphenol connector caused voltage fluctuations during testing on Emergency Diesel Generator 2 on November 10, 2008. In addition, the corresponding governor magnetic pickup unit Amphenol connector on Emergency Diesel Generator 1 caused a breaker trip on January 31, 2009, similar to the April 21, 2008, event.

The licensee documented this deficiency in their corrective action program as Condition Report 2009-00778.

The team determined that the failure to identify the correct root cause for the emergency diesel generator Amphenol connector failures was a performance deficiency. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because the finding was associated with the design control attribute of the mitigating systems cornerstone and affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because the licensee did not use conservative assumptions to ensure that they performed an effective root cause evaluation for failures of the emergency diesel generator Amphenol connectors H.1(b).

Licensee-Identified Violations

None

REPORT DETAILS

REACTOR SAFETY

Cornerstones: Initiating Events, Mitigating Systems, Barrier Integrity, and Emergency Preparedness 1.0 Special Inspection Scope On January 27, 2009, Emergency Diesel Generator 1 began leaking lubricating oil. The licensee determined the oil emanated from a crack in the Emergency Diesel Generator 1 lubricating oil discharge pipe at 125 milliliters per minute. The licensee had previously experienced an Emergency Diesel Generator 2 lubricating oil discharge pipe crack, which they replaced. Based on the most recent crack in the Emergency Diesel Generator 1 lubricating oil discharge pipe and the previous crack in the same location in Emergency Diesel Generator 2, questions arose regarding the effectiveness of the licensee=s corrective actions.

The inspection charter (refer to Attachment 2) required the team to:

(1) review the circumstances related to historical and present emergency diesel generator lubricating oil discharge piping failures,

(2) assess the effectiveness of actions taken to resolve and prevent recurrence of these problems, and

(3) assess the effectiveness of the immediate actions taken by the licensee in response to the Emergency Diesel Generator 1 lubricating oil discharge pipe failure that occurred on January 27, 2009.

In addition, the charter required that the team examine the effectiveness of the licensee's actions for emergency diesel generator Amphenol connectors. The team reviewed discrepancies that occurred in January 2008, April 2008, November 2008 and January 2009. The team evaluated the event conditions to determine if the licensee took appropriate actions to address these issues including extent of condition, extent of cause, and common cause questions.

The team conducted their reviews in accordance with NRC Inspection Procedure 93812, "Special Inspection Procedure." The special inspection team reviewed procedures, corrective action documents, as well as design and maintenance records for the equipment of concern. The team interviewed key station personnel regarding the events, reviewed the root cause analyses, and assessed the adequacy of corrective actions. A list of specific documents reviewed is provided in Attachment 1. The charter for the special inspection is provided as Attachment 2.

2.0 Review of Diesel Generator Lubricating Oil Pipe Failures January 2009 Emergency Diesel Generator 1 Lubricating Oil Discharge Pipe Crack On February 13, 2008, operators started Emergency Diesel Generator 2 for a monthly surveillance test in accordance with Procedure 6.2DG.101, "Diesel Generator 31 Day Operability Test (IST) (DIV2)," Revision 52. Approximately 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> after the start of the engine, an auxiliary operator discovered oil leaking from the emergency diesel generator lubricating oil pump discharge piping. The lubricating oil emanated from a four-inch radial crack at the 12 o'clock position in the toe of an elbow to flange weld in the heat-affected zone. Upon receiving the report of the oil leak, the operations shift manager directed that operators secure Emergency Diesel Generator 2.

The lubricating oil pump for the emergency diesel generator discharges horizontally to a six-inch diameter, approximately 3-foot long spool piece (S-pipe). This S-pipe consists of an upper elbow connected to a short vertical pipe connected to a lower elbow that connects to a horizontal flexible hose (refer to Figure 1 in Attachment 5). The licensee attributed the root cause to high cycle fatigue. However, the detailed vibration readings taken on the piping following repair did not generate sufficient stresses at the crack location to exceed the endurance limit and create the high cycle fatigue break. After obtaining this information the licensee revised the root cause. The revised root cause, through metallurgical and analytic analysis, attributed the failure to four piping overload events.

The licensee performed leak rate estimates based upon oil cleaned up from the emergency diesel generator pedestal with a shop vacuum. The licensee estimated the time of the oil leakage based upon operator entries into the room using key card records.

The licensee estimated the leakage flow rate to be approximately eight gallons per hour.

The licensee determined that lubricating oil leakage at this rate remained within the capacity of the lubricating oil sump volume and did not challenge the capability of Emergency Diesel Generator 2 to meet its 24-hour mission time.

Subsequently, as part of their extent of condition review, on October 30, 2008, the licensee removed the Emergency Diesel Generator 1 lubricating oil discharge S-pipe upper elbow to determine if that elbow had been similarly overstressed. The licensee replaced the elbow using the same material stock as that used for Emergency Diesel Generator 2 in February 2008. Since the revised root cause attributed the failure to four external overload events, the licensee used a smaller weld as specified on design drawings. The licensee performed a single pass 3/16" fillet weld with 3/16" weld legs at the flange and elbow on Emergency Diesel Generator 1 instead of a 7/16" weld, as the licensee had used for Emergency Diesel Generator 2. An increased weld size redistributes the stress at the weld location resulting in a weld less affected by fatigue.

Magnetic particle testing of the Emergency Diesel Generator 1 elbow showed no evidence of any type of cracking.

After reviewing the metallurgical evidence contained in two different vendor analysis reports, NRC materials engineers informed the licensee in December 2008 that the metallurgical evidence indicated the failure mechanism for the Emergency Diesel Generator 2 crack resulted from high cycle fatigue. Following additional discussions, the licensee initiated Condition Report 2009-00098, which documented the NRC position that the lubricating oil pipe fracture on Emergency Diesel Generator 2 resulted from high cycle fatigue.

On January 27, 2009, during performance of Procedure 6.1DG.101, "Diesel Generator 31 Day Operability Test (IST) (DIV1)," Revision 51, a station operator reported oil leaking from the Emergency Diesel Generator 1 lubricating oil discharge S-pipe. The leak developed 6 minutes after starting the engine at the toe of the flange fillet weld at the 12 oclock position. When identified, Emergency Diesel Generator 1 was running unloaded at 600 rpm. This elbow and flange assembly incurred 21.05 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> of operation and six starts prior to this failure. The licensee collected the oil for a period of 2 minutes on four occasions and determined the leakage remained steady at 125 milliliters per minute. The 3-inch long circumferential crack did not worsen during the 57 minutes operators had Emergency Diesel Generator 1 running unloaded. The fatigue stresses that caused the crack in the base metal resulted from vibration in the X-Direction (axial to the metal flexible hose and perpendicular to the crack face).

On January 28, 2009, operators started Emergency Diesel Generator 1 for diagnostic purposes. The licensee measured 6 - 8 inches per second vibrations on the bottom elbow in the X-Direction (represented the relevant oscillatory loading on the lubricating oil discharge S-pipe) with a normal vibration level of 1 - 2 inches per second. During the 53 minute run, the lubricating oil leak rate measured 125 milliliters per minute at the beginning of the period, increased while loading Emergency Diesel Generator 1 to 3800 kW to a rate of 250 milliliters per minute, then decreased back to 125 milliliters per minute with the emergency diesel generator unloaded. An impact test on the cracked lubricating oil discharge S-pipe found the piping system natural frequency resonated with the lubricating oil pump forcing function frequency of 110 Hz.

On January 30, 2009, the licensee installed a replacement lubricating oil discharge S-pipe using a full penetration weld with a two-to-one taper on both the upper and lower flange to elbow welded connections. The licensee performed these welds to improve the resistance of the lubricating oil discharge S-pipe to fatigue. Upon starting Emergency Diesel Generator 1, the licensee determined the piping system continued to have high vibrations in the 4 - 5 inches per second range. Consequently, the licensee installed a stiffener clamp to reduce the lubricating oil piping system vibration levels.

Vibration levels following stiffener clamp installation decreased to the 1 - 2 inches per second range.

Condition Report 2009-00613 documented this lubricating oil discharge S-pipe failure.

The licensee performed a root cause evaluation for this significant condition adverse to quality. The licensee concluded the direct cause of the crack resulted from high cycle fatigue exaggerated by a resonant vibration condition. The elbow installed on October 30, 2008, had less mass than that of the elbow it replaced. In addition, the licensee used a smaller fillet elbow/flange weld size than the weld on the elbow/flange it replaced. Although both changes met minimum design requirements, the mass change affected the natural frequency of the piping system. Because the crack propagated from the outside to the inside and had a flat and featureless surface with multiple crack initiation sites, the team concluded the failure resulted from high cycle fatigue (refer to Figure 2 in Attachment 5). High cycle fatigue initiated and propagated the crack because the vibration-induced stresses had exceeded the endurance limit of the material.

The licensee determined the piping system containing the crack for the Emergency Diesel Generator 1 had a natural frequency of 117 Hz by performing an impact test in standby mode and measuring the vibrations with an accelerometer. From vibration data acquired with Emergency Diesel Generator 1 in-service, the licensee determined the engine-mounted emergency diesel generator lubricating oil pump 11-vane impeller generated a forcing frequency for this piping system of 110 Hz. The licensee again performed an impact test and measured a natural frequency of 115.3 Hz immediately following shut down of the emergency diesel generator. The licensee concluded that, at normal operating temperatures, the natural frequency of the piping system tends to move closer toward the forcing frequency causing resonance. This natural frequency for the piping system did not fall outside the range of 99 - 121 Hz (good practice of a minimum separation of 10 percent between the natural frequency and any dominant forcing frequencies for the system).

After installing the stiffener clamp, the licensee operated Emergency Diesel Generator 1 at 3800 kW for 14.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> and at 1000 kW for 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br />. After operators secured Emergency Diesel Generator 1, maintenance performed a dye penetrant examination on the repaired lubricating oil discharge S-pipe in the area that had previously cracked. The licensee identified no evidence of crack initiation. After installing the clamp, operators operated Emergency Diesel Generator 1 for 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> at 3800 kW. Predictive maintenance personnel determined the vibrations with the clamp installed ranged from 1 - 2 inches per second.

Prior to replacing the Emergency Diesel Generator 1 lubricating oil discharge S-pipe on October 30, 2008, the original lubricating oil discharge S-pipe operated for more than 3000 hours0.0347 days <br />0.833 hours <br />0.00496 weeks <br />0.00114 months <br /> of engine operation with no failure. Piping vibration levels taken in February and August 2008 ranged from 1 - 2 inches per second. Based on the large number of hours and no crack development with vibration levels in this range, the licensee considered the measured vibration levels with the stiffener clamp installed acceptable.

Because of the uncertainty regarding the actual loading, the licensee initiated actions to monitor the vibration in the X-direction and to perform dye penetrant testing of the susceptible weld during each emergency diesel generator monthly operability tests. The licensee planned to continue these compensatory measures until they install a permanent design change. Based on the natural frequency being moved farther from the forcing frequency, the vibration levels being returned to values similar to those prior to the lubricating oil discharge S-pipe replacement in October 2008, no crack initiation identified following several hours of operation, and the continued monitoring until the licensee installed a redesigned lubricating oil discharge S-pipe, the team concluded that Emergency Diesel Generator 1 remained capable of performing its safety function.

a. Inspection Scope

The team evaluated the licensee response to the lubricating oil discharge S-pipe failures in January 2009 and February 2008. In order to review each area of the charter issued on February 17, 2009, the team reviewed calculations, design documents, licensing documents, work orders, modification packages, purchase order documents and corrective action documents. The team evaluated licensee compliance with the applicable regulatory requirements and codes and standards.

The team interviewed performance monitoring, predictive maintenance, design and system engineering, maintenance and corrective action program personnel.

The team assessed licensee implementation of their corrective action program, design controls, and procedure implementation.

.1 Inadequate corrective actions to repair lubricating oil pipe

Introduction.

The team identified a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action," for inadequate corrective actions to prevent recurrence of a significant condition adverse to quality. The licensee inappropriately changed the root cause for the Emergency Diesel Generator 2 lubricating oil discharge S-pipe failure on February 13, 2008, from high cycle fatigue to four piping overstress events. Consequently, the licensee implemented corrective actions that resulted in a high cycle fatigue failure of the Emergency Diesel Generator 1 lubricating oil discharge S-pipe on January 27, 2009. The licensee entered this deficiency in their corrective action program as Condition Report 2009-00098.

Description.

On February 13, 2008, during surveillance testing, Emergency Diesel Generator 2 developed a leak in the lubricating oil pump discharge piping. The circumferential through-wall crack extended for approximately four inches in the heat-affected zone of piping adjacent to the upper elbow to flange weld. Following this failure, the licensee did not measure any operational vibration or other relevant data to support the root cause assessment. The licensee submitted the failed upper elbow of the pipe for metallographic analysis to Southwest Research Institute and contracted with Structural Integrity Associates to perform a stress analysis. The licensee calculated the flow rate as 8 gallons per hour. After analysis the licensee determined that the leak rate remained within the capacity of the emergency diesel generator sump volume to meet the mission time of 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> without adding oil. Structural Integrity Associates took vibration measurements on the repaired piping so that they could estimate the stresses required to develop the crack in the lubricating oil pipe or the repaired lubricating oil pipe.

The vibrations measured from 1 - 2 inches per second in the X-Direction.

The initial root cause evaluation for Condition Report 2008-00968 attributed the misalignments in the flexible hose attached to the Emergency Diesel Generator 2 lubricating oil discharge S-pipe as the cause of the high cycle fatigue failure. The licensee restored the piping to a more rigid configuration by using a larger weld size of 7/16", as compared to 3/16", at the pipe flange to help redistribute stresses in the pipe.

After receiving the metallurgical analysis from Southwest Research Institute and the stress analysis results from Structural Integrity Associates, the licensee revised their root cause evaluation. Specifically, the Southwest Research Institute reports confirmed that the metallographic analysis identified high cycle fatigue as the crack failure mechanism in the Emergency Diesel Generator 2 lubricating oil discharge S-pipe. However, the Structural Integrity Associates stress calculations indicated that the limiting design transient produced cyclic loads that did not exceed the endurance limit of the material and would not cause a high cycle fatigue failure.

On November 17, 2008, the licensee revised the root cause evaluation for Condition Report 2008-00968 to specify that four distinct overload events, exemplified by ductile tearing caused the failure. The licensee concluded that the flat fracture surfaces resulted from the overload events combined with vibration induced wear, which occurred during replacement of the flexible hose attached to the lower elbow of lubricating oil discharge S-pipe. The licensee identified that the overloads resulted from the need to employ mechanical assistance to correct misalignments of the flanges and widen the gap between a check valve and the lubricating oil discharge S-pipe flange to install replacement flexible hoses.

During December 2008, NRC challenged the licensee on their root cause evaluation because of the discrepancies in their conclusions compared to the documented metallurgical results that indicated high cycle fatigue in the Southwest Research Institute report. The fracture surface appearance did not suggest ductile tearing. Ductile tearing is typically presented by dimple, or cup and cone fracture appearance both macro and microscopically. The Southwest Research Institute report clearly showed the surface was flat and smooth with clear evidence of beachmarks and striations indicative of a high cycle fatigue failure mechanism (refer to Figure 2). The licensee initiated Condition Report 2009-00098 to resolve the NRC concerns related to misdiagnosis of the failure mechanism for the Emergency Diesel Generator 2 lubricating oil discharge S-pipe crack.

Because the licensee concluded that mechanics overstressed the Emergency Diesel Generator 2 lubricating oil discharge S-pipe while replacing the flexible coupling, the licensee implemented inappropriate corrective actions. Specifically, as part of an extent of condition review related to an overstress condition, the licensee replaced the Emergency Diesel Generator 1 lubricating oil discharge S-pipe on October 30, 2008, to allow Southwest Research Institute to analyze the piping for evidence of ductile tearing.

Although the replacement piping for Emergency Diesel Generator 1 had the same material specification as the replacement piping for Emergency Diesel Generator 2, the licensee did not modify the welds because they believed the piping would not have to withstand high cycle fatigue.

Contrary to the revised root cause analysis, the team determined that the replaced Emergency Diesel Generator 1 lubricating oil discharge S-pipe failed from high cycle fatigue on January 27, 2009, and represented a failure of the licensee to preclude recurrence of a significant condition adverse to quality.

Analysis.

The performance deficiency involved the failure of the licensee to take adequate corrective actions to prevent recurrence of a significant condition adverse to quality. The finding was determined to be more than minor because it is associated with the mitigating systems cornerstone attribute of equipment performance and affected the associated cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings,"

the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of problem identification and resolution associated with the corrective action program because the licensee did not thoroughly evaluate the Emergency Diesel Generator 2 failure such that the specified corrective actions addressed the causes of the failure

P.1(c).

Enforcement.

Title 10 of the Code of Federal Regulations Part 50, Appendix B, Criterion XVI, "Corrective Action," specifies, in part, "In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition." Procedure 0.5, "Conduct of The Condition Report Process," Revision 63, Section 7.3.1, specifies for significant conditions that require a root cause evaluation corrective actions to prevent recurrence are assigned for each root cause. Contrary to the above on October 30, 2008, in the case of a significant condition adverse to quality, the licensee failed to take measures that assured that the cause of the condition was determined and corrective action taken to preclude repetition. Specifically, the licensee incorrectly revised their root cause for the Emergency Diesel Generator 2 lubricating oil discharge S-pipe failure from high cycle fatigue to four overstress events during maintenance.

The licensee revised their root cause despite evidence supplied by Southwest Research Institute; consequently, the licensee actions resulted in a similar high cycle fatigue failure of Emergency Diesel Generator 1 on January 27, 2009, as occurred in Emergency Diesel Generator 2 on February 13, 2008. The licensee entered this deficiency in their corrective action program as Condition Report 2009-00098. Because the finding is of very low safety significance and has been entered into the licensees corrective action program, this violation is being treated as a noncited violation, consistent with Section VI.A of the Enforcement Policy: Noncited Violation 05000298/2009008-01, Inadequate corrective actions to repair a lubricating oil pipe.

.2 Failure to perform effective common mode failure evaluation

Introduction.

The team identified a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, Drawings," regarding failure to follow the requirements of Procedure ENN-OP-104, "Operability Determinations," Revision 2.

Specifically, the team determined that operations personnel did not obtain necessary information to determine with reasonable assurance that Emergency Diesel Generator 2 remained operable and not subject to a common mode failure mechanism. The Emergency Diesel Generator 1 lubricating oil discharge S-pipe cracked as a result of vibrations in the X-direction and the common mode failure evaluation did not account for vibrations in the X-direction for Emergency Diesel Generator 2. Subsequent measurements confirmed that the vibrations remained within the normal operating range. The licensee documented this deficiency in Condition Report 2009-00655.

Description.

On January 27, 2009, during a monthly operability test, Emergency Diesel Generator 1 developed a leak in the lubricating oil discharge S-pipe. The circumferential through-wall crack extended for approximately three inches in the heat-affected zone of the piping adjacent to the upper elbow to flange weld. Operators shut down and declared Emergency Diesel Generator 1 inoperable. As directed by technical specifications, a common mode analysis was completed to assess the operability of Emergency Diesel Generator 2. In this analysis engineering personnel used monitored vibration data and a longer service time as justification for demonstrating with reasonable assurance that Emergency Diesel Generator 2 remained operable.

However, the vibration data for this analysis did not utilize the X-Direction. The oscillatory loading in this direction is the relevant driver of high cycle fatigue since an increase in vibration levels in the X-Direction exacerbated the forces that caused cracking and needed to be considered to provide a reasonable assurance of operability.

The team determined that the engineering evaluation without the X-Direction vibration information could not demonstrate with reasonable assurance that Emergency Diesel Generator 2 remained operable and would not fail because of high cycle fatigue. A subsequent test with measurements taken in the X-Direction confirmed that vibrations remained acceptable.

Analysis.

The performance deficiency associated with this finding involved the failure of operations personnel to perform an adequate operability assessment. The finding was determined to be more than minor because it would become a more significant event if left uncorrected in that the failure of Emergency Diesel Generator 2 by the same high cycle fatigue mechanism increased the likelihood that both emergency diesel generators could be inoperable concurrently. The finding affected the mitigating systems cornerstone. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because the licensee failed to use conservative assumptions when determining operability H.1(b).

Enforcement.

Title 10 of the Code of Federal Regulations Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," specifies, in part, that activities affecting quality shall be prescribed by documented instructions, procedures, or drawings, of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings. Procedure ENN-OP-104 requires that the shift manager ensure that reasonable assurance exists that equipment will perform its design function. Contrary to the above on January 28, 2009, operations personnel did not accomplish activities affecting quality in accordance with procedures.

Specifically, the shift manager did not have the necessary X-Direction vibration measurements to demonstrate with reasonable assurance that a common mode failure mechanism did not exist for the Emergency Diesel Generator 2 discharge lubricating oil piping and that the emergency diesel generator remained operable. The licensee documented this deficiency in Condition Report 2009-00655. Because the finding is of very low safety significance and has been entered into the corrective action program, this violation is being treated as a noncited violation, consistent with Section VI.A of the Enforcement Policy: Noncited Violation 05000298/2009008-02, Failure to perform effective common mode failure evaluation.

.3 Failure to identify a condition adverse to quality

Introduction.

The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, Drawings," for the failure of maintenance personnel to initiate a condition report, as required by Procedure 0.5, "Conduct of the Condition Report Process," Revision 63, Section 7.1.3. Specifically, maintenance personnel failed to initiate a condition report for an adverse condition related to a significant change in vibration levels on Emergency Diesel Generator 1 between readings. The licensee documented this deficiency in Condition Report 2009-00694.

Description.

On February 13, 2008, during surveillance testing, a leak developed in the Emergency Diesel Generator 2 lubricating oil discharge S-pipe. Because of the revised root cause for Condition Report 2008-00968 related to mechanical overstress of the lubricating oil pipe, the licensee replaced the Emergency Diesel Generator 1 lubricating oil discharge S-pipe as part of their extent of condition review and provided training to maintenance personnel. The licensee did not communicate that high cycle fatigue could have been the cause of the failure; hence, the licensee did not establish an appropriate sensitivity to vibratory issues associated with the emergency diesel generators.

In addition, the licensee had been experiencing lubricating oil discharge pipe flexible hose leaks on both emergency diesel generators for the past several years that had resulted from fretting. Consequently, the system engineer responsible for the diesel generators requested that vibration readings be taken quarterly in the Y- and Z-Directions (vertical and lateral (i.e., into the pipe), respectively, forces that create the most flexible hose degradation) to understand the vibration amplitudes that might result in failure of the flexible hose.

The team determined that the measured vibration levels on Emergency Diesel Generator 1 lubricating oil discharge S-pipe increased from 1 to 4 inches per second from August 27, to November 25, 2008, respectively. Maintenance personnel identified this change. However, the maintenance personnel obtained this data in response to a request from the system engineer. Because of the informational nature of the request, the maintenance personnel decided this condition did not warrant documentation in the corrective action program. Further, maintenance personnel decided that continued trending of the vibration levels would be the correct choice of action prior to notifying the system engineer. The team determined that Procedure 0.5, Section 7.1.3 required personnel to initiate condition reports for adverse conditions. Adverse conditions included conditions adverse to quality that reflected any condition that could affect the ability of safety-related components to function within design requirements or affect performance characteristics. The team determined the informality of the request and the failure to use existing processes contributed to this condition.

The team concluded that personnel had an incorrect understanding of the magnitude of the change in the vibratory issues associated with the emergency diesel generators and that the licensee had not communicated the potential for high cycle fatigue to maintenance personnel. The team considered the failure to document the change in vibration amplitude in the corrective action program a failure to identify oscillatory loading on the Emergency Diesel Generator 1 lubricating oil discharge S-pipe, which resulted in a failure because of high cycle fatigue.

Analysis.

The performance deficiency associated with this finding involved the failure of maintenance personnel to initiate a condition report for an adverse condition as required by Procedure 0.5. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because the finding was associated with the human performance attribute of the mitigating systems cornerstone and affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. The finding affected the mitigating systems cornerstone. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with work practices because the licensee did not effectively communicate expectations regarding following procedures for initiating condition reports for adverse conditions H.4(b).

Enforcement.

Title 10 of the Code of Federal Regulations Part 50, Appendix B, Criterion V, "Instructions, Procedures, Drawings," specifies, in part, "Activities affecting quality shall be prescribed by documented instructions, procedures, or drawings, of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings." Procedure 0.5, Section 7.1.3, requires that licensee personnel initiate condition reports for adverse conditions that could affect performance characteristics. Contrary to the above, on November 25, 2008, maintenance personnel failed to accomplish activities affecting quality in accordance with instructions, procedures, or drawings. Specifically, maintenance personnel failed to initiate a condition report for a step change in vibration levels that reflected the change in performance characteristics of Emergency Diesel Generator 1. The licensee documented this deficiency in Condition Report 2009-00694. Because the finding is of very low safety significance and has been entered into the corrective action program, this violation is being treated a noncited violation, consistent with Section VI.A of the Enforcement Policy: Noncited Violation 05000298/2009008-03, Failure to identify a condition adverse to quality.

.4 Failure to provide adequate work instructions to perform maintenance

Introduction.

The team identified the first of two examples of a Green noncited violation of Technical Specification 5.4.1.a for the failure of the licensee to provide work instructions appropriate to the circumstances. Specifically, the work orders generated as an interim corrective action to monitor vibrations on the emergency diesel generator lubricating oil discharge S-pipe did not have adequate quantitative acceptance criteria established commensurate with their training to prompt maintenance personnel to identify adverse conditions. The licensee documented this deficiency in Condition Reports 2009-01513.

Description.

On January 27, 2009, during monthly operability testing, a leak developed in the Emergency Diesel Generator 1 lubricating oil discharge S-pipe. The circumferential through-wall crack extended for approximately 3 inches in the piping adjacent to the upper elbow to flange weld. As part of troubleshooting to asses the cause of the cracked pipe, the licensee contracted Structural Integrity Associates to monitor the as-found and post repair conditions for vibrations to assess the magnitude of oscillatory loading. The licensee operated Emergency Diesel Generator 1 in a variety of loading conditions for approximately 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and measured vibrations in the X-Direction of approximately 7.5 inches per second.

The licensee attributed the failure mechanism in Condition Report 2009-00613 to high cycle fatigue and implemented corrective actions to prevent the recurrence of this significant condition adverse to quality. Specifically, in considering the impact of a high cycle fatigue mechanism, the licensee fabricated the Emergency Diesel Generator 1 replacement lubricating oil discharge S-pipe and revised the weld design to help relieve stress concentrations. The long-term corrective action involved developing and implementing a more fatigue resistant design for the lubricating oil discharge S-pipe.

As interim corrective actions, the licensee generated Work Orders 4683245 and 4681978 to monitor the lubricating oil discharge S-pipe for adverse vibrations and perform dye penetrant testing to inspect for cracks each time operators perform the monthly diesel generator operability surveillance. The team reviewed the work instructions and concluded that the licensee had not established quantitative acceptance criteria for adverse vibrations. The team had concerns because

(1) personnel taking the vibration readings previously did not consider an increase from 1 to 4 inches per second a significant change in vibration levels and

(2) informal communications among departments had failed to flag a step increase in vibration levels (as described in Section 2.b.3). The increase in X-Direction vibration levels for any system is significant since the impact increases the likelihood of high cycle fatigue failure at weak points.

Procedure 0.40.4, "Planning," Section 5.2.4.12.a, states, in part, that the planner shall ensure that any required procedure acceptance criteria are included in the work instructions. Consequently, the team determined that the lack of well-defined criteria in the work instructions could result in missed opportunities to identify potential degraded conditions.

Analysis.

The performance deficiency associated with this finding involved the failure to ensure maintenance personnel took the appropriate actions when performing maintenance. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because it is associated with the mitigating systems cornerstone attribute of equipment performance and it affects the cornerstone objective of ensuring availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings,"

the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because incorrect assumptions by the licensee regarding the skills and knowledge level of the craft resulted in a maintenance procedures that had insufficient instructions H.1(b).

Enforcement.

Technical Specification 5.4.1.a. requires, in part, that written procedures be established, implemented, and maintained covering the activities specified in Regulatory Guide 1.33, "Quality Assurance Program Requirements (Operations),"

dated February 1978. Regulatory Guide 1.33, Appendix A, "Typical Procedures for Pressurized Water Reactors and Boiling Water Reactors," Section 9.a, requires maintenance affecting the performance of safety-related equipment be properly performed in accordance with written procedures, documented instructions, or drawings appropriate to the circumstances. Contrary to the above, on February 23, 2009, for maintenance that can affect the performance of safety-related equipment, licensee personnel did not provide instructions appropriate to the circumstances. Specifically, Work Orders 4683245 and 4681978 generated to monitor vibrations on the emergency diesel generator lubricating oil discharge S-pipe, as interim corrective actions until the permanent modification was completed, did not specify quantitative acceptance criteria.

Since the licensee had established these work orders as part of the interim corrective actions for a significant condition adverse to quality, the team had concerns that maintenance personnel would not recognize the need to initiate a condition report for increased vibrations (an adverse condition). The licensee documented this deficiency in Condition Report 2009-01513. Because the finding is of very low safety significance and has been entered into the corrective action program, this violation is being treated as the first example (refer to Amphenol connectors Section 3.b.2 for the second example) of a noncited violation, consistent with Section VI.A of the Enforcement Policy: Noncited Violation 05000298/2009008-04, Failure to provide adequate work instructions to perform maintenance.

.5 Failure to identify nonconforming design characteristics

Introduction.

The team identified a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion XV, "Nonconforming Materials, Parts, or Components," for the failure of engineering personnel to review and accept a nonconforming pipe configuration. Specifically, when replacing the lubricating oil discharge S-pipe on Emergency Diesel Generator 1 for an extent of condition evaluation, engineers failed to evaluate critical characteristics and determine why the replacement characteristics did not match the installed configuration or why this configuration did not match existing drawings. The licensee documented this deficiency in Condition Report 2009-00613.

Description.

On February 13, 2008, during a monthly operability test, Emergency Diesel Generator 2 developed a leak in the lubricating oil discharge S-pipe. The circumferential through-wall crack extended for approximately 4 inches in the heat-affected zone of the upper elbow to flange weld. The licensee took vibrations of the Emergency Diesel Generator 2 lubricating oil discharge S-pipe after they had repaired the piping so that Structural Integrity Associates could perform the stress analysis. The licensee submitted the upper elbow of the failed lubricating oil discharge S-pipe for metallographic analysis to Southwest Research Institute and contracted with Structural Integrity to perform a stress analysis of the emergency diesel generator lubricating oil discharge S-pipe configuration.

The Southwest Research Institute concluded from the metallographic evidence that the Emergency Diesel Generator 2 lubricating oil discharge S-pipe upper flange crack resulted from high cycle fatigue. In contrast, calculations performed by Structural Integrity Associates stated that the limiting design transient for the metal in the heat affected zone experienced cyclic loads that did not exceed the endurance limit of the material. Structural Integrity Associates had based these calculations from vibration readings taken on the repaired rather than the failed piping. Subsequently, the licensee incorrectly concluded that fatigue could not have been the cause of the cracking. The licensee revised the root cause in Condition Report 2008-00968 to reflect the mechanistic cause of the failure as four distinct piping overload events.

Engineering failed to assess all critical characteristics when fabricating the replacement Emergency Diesel Generator 1 lubricating oil discharge S-pipe section. The licensee replaced this lubricating oil discharge S-pipe section to help address the extent of condition for Condition Report 2009-00968. The team identified that engineers did not evaluate the variations of the fabricated replacement lubricating oil discharge S-pipe from the specifications on design drawings and from the S-pipe section removed. These variations included:

(1) the return to the original 3/16" weld size in lieu of the larger 7/16" weld placed on Emergency Diesel Generator 2,

(2) a thicker flange, which resulted in personnel not fully inserting the pipe into the flange increasing the length and stresses, and

(3) the increased overall length of the lubricating oil discharge S-pipe, which generated a larger moment arm at the upper elbow to flange weld.

Procedure 3.21, "Fabrication of Replacement Parts," Revision 7, section 4.2, states that the engineer will define the part to be fabricated in concise terms such that the critical design characteristics are identified. In this case, licensee personnel failed to correctly specify the following critical design characteristics:

(1) weld size,

(2) pipe length, and

(3) amount of residual stresses allowed. The team determined that the failure to evaluate these critical characteristics exacerbated the resulting high cycle fatigue failure.

Analysis.

The performance deficiency associated with this finding involved the failure to thoroughly evaluate and control configuration changes on the emergency diesel generator lubricating oil piping. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because the finding was associated with the design control attribute of the mitigating systems cornerstone and affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because licensee personnel did not verify the critical characteristics of the replacement lubricating oil discharge S-pipe section, which had a different configuration than the pipe being removed for analysis H.1(a).

Enforcement.

Title 10 Code of Federal Regulations Part 50, Appendix B, Criterion XV, "Nonconforming Materials, Parts or Components," requires, in part, that "Nonconforming items shall be reviewed and accepted, rejected, repaired or reworked in accordance with documented procedures." Contrary to the above, the licensee failed to establish measures for the review and acceptance of nonconforming items. Specifically, the licensee had not adequately evaluated and accepted changes to the emergency diesel generator lubricating oil discharge S-pipe.

Procedure 3.21, Section 4.2, states that the engineer will define the part to be fabricated in concise terms such that the critical design characteristics are identified. Contrary to the above, on October 30, 2008, engineering personnel failed to review and accept a nonconforming item. Specifically, because engineers failed to identify those critical design characteristics for the replacement Emergency Diesel Generator 1 lubricating oil discharge S-pipe, the replacement configuration exacerbated the high cycle fatigue mechanism, which resulted in a failure after 21 hours2.430556e-4 days <br />0.00583 hours <br />3.472222e-5 weeks <br />7.9905e-6 months <br /> of service. The licensee documented this deficiency in Condition Report 2009-00613. Because the finding is of very low safety significance and has been entered into the corrective action program, this violation is an example of a noncited violation, consistent with Section VI.A of the Enforcement Policy: Noncited Violation 05000298/2009008-05, Failure to identify nonconforming design characteristics.

3.0 Review of Amphenol Connector Events Emergency Diesel Generator 2 Amphenol Connector Deficiency on January 14, 2008 On January 14, 2008, Emergency Diesel Generator 2 tripped 60 seconds after operators started the emergency diesel generator. Operators performed the post maintenance test to demonstrate the ability of Emergency Diesel Generator 2 to perform its safety function following repairs to the overspeed governor oil level sight glass. The licensee determined Emergency Diesel Generator 2 tripped because of a loose Amphenol connector on the relay tachometer magnetic pickup unit (DG-SE-3143) and performed the root cause for this deficiency in Condition Report 2008-00304. The licensee identified the failure as a significant condition adverse to quality since the failure of this Amphenol connector rendered Emergency Diesel Generator 2 inoperable.

The root cause evaluation determined that Emergency Diesel Generator 2 likely operated with a loose Amphenol connector since personnel completed maintenance on the emergency diesel generator in December 2000 to replace the relay tachometer magnetic pickup unit (DG-SE-3143). For the Amphenol connector to loosen, the licensee determined that maintenance personnel must not have assembled the Amphenol connector properly and that engine vibration had caused the final separation.

The licensee determined from third-party vibration testing that a properly assembled Amphenol connector would not loosen during emergency diesel generator operation.

The licensee determined that a similar Emergency Diesel Generator 2 trip occurred on December 10, 1995. Operations personnel initiated Problem Incident Report 1-19594 because the Amphenol connector for an Emergency Diesel Generator 2 mounted magnetic pickup unit vibrated loose during testing. As corrective actions, the licensee loosened and applied Loctite to all Amphenol connectors on both emergency diesel generators under Work Order 95-03959. Beyond the actions taken in the work order, the licensee did not develop work instructions or procedures for ensuring Amphenol connectors will not vibrate loose in the future.

Although the NRC previously reviewed this deficiency, the team evaluated this example since it reflected poor maintenance practices and a lack of instructions commensurate with the skill-of-the-craft. The NRC had taken regulatory action because the exposure time for this failure resulted in a finding of low to moderate safety significance. The NRC documented their activities related to review of this deficiency in Inspection Reports 05000298/2008002 and 2008009.

Emergency Diesel Generator 2 Amphenol Connector Deficiency on April 21, 2008 On April 20, 2008, Emergency Diesel Generator 2 experienced unexpected load perturbations 6.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> into the 24-hour test after operators increased the load to 110 percent. The licensee initiated Condition Report 2008-02692 to document the deficiency and perform a root cause analysis to identify why Breaker 1GB tripped.

During troubleshooting on April 21, 2008, 17 minutes into the instrumented test, personnel noticed that load swings began to occur and continued until operators secured Emergency Diesel Generator 2. Operators had increased and held the emergency diesel generator load at 2000 kW. The licensee identified that anomalies with the governor magnetic pickup unit output coincided with the demand to reduce or increase fuel to the emergency diesel generator.

The licensee replaced the digital reference unit and the governor magnetic pickup unit on April 23, 2008. With Emergency Diesel Generator 2 operating, instrumentation and control technicians performed wire termination checks that did not create load swings as a result of loose connections. The licensee did not identify any anomalies or load swings during the troubleshooting.

Based on the observed anomalies in the signal data acquired on April 21, 2008, the licensee hypothesized that the speed gear had intermittently contacted the governor magnetic pickup unit probe. The licensee inspected the magnetic pickup unit probe and identified very minute metallic debris and rub marks on the probe (refer to Figure 3 in 6). The licensee determined the speed sensing gear impacts on the magnetic pickup unit probe produced the load swings observed during the April 20 and 21, 2008, tests. Since the licensee replaced the digital reference unit and magnetic pickup unit, verified the gap settings on the new magnetic pickup unit probe, and successfully completed operability and post maintenance tests without anomalies, the licensee concluded Emergency Diesel Generator 2 would perform its required safety function (refer to Figure 4 in Attachment 6).

The licensee performed external inspections of the Emergency Diesel Generator 1 governor magnetic pickup unit and found no anomalies.

Emergency Diesel Generator 2 Amphenol Connector Deficiency on November 10, 2008 On November 10, 2008, during a monthly operability test, Emergency Diesel Generator 2 experienced intermittent load oscillations from 3600 to 4000 kW. Over the next hour, the oscillations became more frequent until they became continuous. The licensee documented this deficiency in Condition Report 2009-08248.

During troubleshooting, when instrumentation and control technicians wiggled the cable to the Amphenol connector, the meter deflected indicating a potential high resistance or open circuit. Additional testing of this cable revealed a potential problem with the shield wire at the approximate location of the Amphenol connector. The licensee inspected the speed gear and housing, performed a gap check of the governor magnetic pickup unit, and checked the total indicated runout for the speed gear. Based on these activities, the licensee determined no evidence existed of the speed gear coming in contact with the governor magnetic pickup unit. The licensee performed an instrumented run of Emergency Diesel Generator 2. Once operators loaded the Emergency Diesel Generator 2 to approximately 1000 kW, load oscillations recurred. The licensee correlated the governor magnetic pickup unit output signal and the magnetic pickup unit signal anomalies with the Emergency Diesel Generator 2 load oscillations.

The completed failure modes and effects analysis attributed the most likely cause to a problem associated with the governor magnetic pickup unit along with the high resistance or open circuit indication during governor magnetic pickup unit cable agitation. As immediate corrective action the licensee replaced the governor magnetic pickup unit, associated Amphenol connectors, and cable. Following replacement, the licensee determined no abnormalities existed during an additional instrumented run.

Southwest Research Institute performed a detailed failure analysis of the Amphenol connector and cable removed from Emergency Diesel Generator 2 (refer to Figures 5 and 6 in Attachment 6). The laboratory traced the intermittent open circuit to the interface between an Amphenol connector sensor pin and socket. The insertion of an oversized object, such as a voltmeter probe, deformed the connector socket jaws that allowed decreased electrical integrity of the contact. Further, the laboratory determined that poor assembly of the Amphenol connector cable clamp resulted in improper clamping of the cable jacket that allowed for easier transmission of engine vibrations.

The licensee reviewed emergency diesel generator signal traces related to the governor magnetic pickup unit. Although the licensee could not determine the exact onset of the governor magnetic pickup unit signal anomalies, the signal traces from engine data demonstrated the anomalies existed as far back as November 2006. The licensee determined from review of the signal traces that:

(1) the signal degraded from November 15, 2006, through November 10, 2008; and

(2) pre and post-maintenance signal traces indicated the Amphenol connector pins would reengage and "fix" the intermittent connection for some period of time.

The licensee considered the improper assembly of the Amphenol connector a significant condition adverse to quality. The licensee attributed this to improper use of measuring and test instruments by instrumentation and control technicians that splayed the Amphenol connector socket and to a failure to correctly assemble the clamp assembly during previous troubleshooting activities. The enlarged Amphenol connector socket and incorrectly assembled cable clamp resulted in the signal disruptions when exposed to emergency diesel generator vibrations.

The licensee performed external inspections of a sample of Amphenol connectors on safety-related systems to look for improperly assembled cable clamps and the presence of electrical tape.

The licensee considered this event repetitive to the April 20, 2008, event documented in Condition Report 2008-02692. The review of the signal traces from the April 20 and 21, 2008, surveillance tests identified anomalies that matched the intermittent open circuit indications identified from this event.

Emergency Diesel Generator 1 Amphenol Connector Deficiency on January 31, 2009 As documented in Condition Report 2009-00778, during post maintenance testing on January 31, 2009, with Emergency Diesel Generator 1 operating in parallel with offsite power at 3750 kW, Breaker 1FA tripped on a sensed overpower condition. Operators had started Emergency Diesel Generator 1 to demonstrate operability following repair of the lubricating oil discharge S-pipe. Breaker 1FA tripped to separate safety-related Bus F from nonsafety-related Bus A.

Erratic signals from both the governor magnetic pickup unit and the digital reference unit caused an erratic output from the electronic governor. The Emergency Diesel Generator 1 fuel racks opened immediately prior to the Breaker 1FA trip. With the Emergency Diesel Generator 1 operating parallel with the offsite grid, this fuel rack movement significantly increased Emergency Diesel Generator 1 power output and resulted in the overpower condition.

During Work Order 4681439 troubleshooting activities, while agitating the governor magnetic pickup unit cable leads, the cable became disconnected from the connector.

Based on visual inspection of the magnetic pickup unit, including an inspection of the speed gear and housing, gap check of the governor magnetic pickup unit, and check of the speed gear vertical total indicated runout, there was no evidence of the speed gear contacting the governor magnetic pickup unit.

After replacing the governor magnetic pickup unit, associated Amphenol connectors and cable, and digital reference unit, the licensee identified no anomalies during an instrumented run with Emergency Diesel Generator 1 loaded on February 2, 2009.

An investigation determined Emergency Diesel Generator 1 had power oscillations on January 27, 2009, very similar to the Emergency Diesel Generator 2 power oscillations and Breaker 1GB trip that occurred in April 2008. In response to the previous overpower trips and load swings (November 2008 event), the licensee had performed extent-of-condition inspections of the exterior of the Amphenol connectors.

The licensee attributed the root cause for this event in Condition Report 2009-00778 to a faulty governor magnetic pickup unit Amphenol connector (refer to Figures 7 and 8 in 6). The team verified that the anomalies in the governor control system opened the Emergency Diesel Generator 1 fuel rack. With Emergency Diesel Generator 1 connected to the grid through Breaker 1FA, the power generated by Emergency Diesel Generator 1 increased and an actual overpower condition resulted in actuation of the overpower relay and tripping of Breaker 1FA.

a. Inspection Scope

The team evaluated the licensee activities related to the four Amphenol connector deficiencies. Three deficiencies occurred on Emergency Diesel Generator 2 on January 14, April 21, and November 10, 2008; respectively. One deficiency occurred on Emergency Diesel Generator 1 on January 31, 2009. The team reviewed calculations, design documents, licensing documents, work orders, modification packages, purchase order documents and corrective action documents. The team evaluated licensee compliance with the applicable regulatory requirements and codes and standards.

The team interviewed predictive maintenance, design and system engineering, maintenance and corrective action program personnel. The team assessed licensee implementation of their corrective action program, design controls, and procedures. The team reviewed the training provided to maintenance personnel.

.1 Failure to identify root cause of a significant condition adverse to quality

Introduction.

The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Action," for the failure to identify the root cause of a significant condition adverse to quality as required by Procedure 0.5.Root-Cause, "Root Cause Analysis Procedure," Revision 8. Specifically, following an Emergency Diesel Generator 2 governor magnetic pickup unit Amphenol connector failure on April 21, 2008, the licensee incorrectly attributed the failure to a speed gear striking the magnetic probe without identifying that a faulty Amphenol connector caused voltage spikes. Consequently, the same defective Amphenol connector caused voltage fluctuations during testing on Emergency Diesel Generator 2 on November 10, 2008. In addition, the corresponding governor magnetic pickup unit Amphenol connector on Emergency Diesel Generator 1 caused a breaker trip on January 31, 2009, similar to the April 21, 2008, event. The licensee documented this deficiency in their corrective action program as Condition Report 2009-00778.

Description.

The team evaluated the Amphenol connector deficiencies to determine whether the licensee had identified the root cause and had taken all appropriate actions to preclude repetition of these significant conditions adverse to quality.

During surveillance on April 21, 2008, Emergency Diesel Generator 2 experienced unexpected load perturbations and Breaker 1GB tripped on overload to protect the emergency diesel generator by disconnecting Bus 1G from the grid. The licensee attributed the root cause to an improperly gapped governor magnetic pickup unit that became damaged when struck by the speed gear. As a result, the licensee replaced the governor magnetic pickup unit and revised Procedure 7.0.5, "Post-Maintenance Testing," to require that instrumentation and control technicians measure the air gap as a post maintenance work activity anytime the technicians disassemble the Amphenol connector. This deficiency did not render the emergency diesel generator inoperable since the mechanical overspeed governor would control the emergency diesel generator engine speed when operating in isochronous mode.

On November 10, 2008, Emergency Diesel Generator 2 experienced load swings during testing while operating paralleled to the grid. This deficiency did not render the emergency diesel generator inoperable since the mechanical overspeed governor would control the diesel engine speed when operating in isochronous mode. This event resulted because the governor magnetic pickup unit Amphenol connector that instrumentation and control technicians had not replaced in April 2008 exhibited high resistance readings, which corresponded to instability in the emergency diesel generator voltage output and fuel rack oscillations. Because of the repeat event, the licensee determined the Amphenol connector caused the perturbations and that they had not corrected the deficiency following the April 2008 event.

On January 31, 2009, as a result of poor assembly, Emergency Diesel Generator 1 experienced load swings while operating paralleled to the grid that resulted in trip of Breaker 1FA and disconnection from the grid. This deficiency did not render the emergency diesel generator inoperable since the mechanical overspeed governor would control the diesel engine speed when operating in isochronous mode. When the licensee disassembled the Amphenol connector, the licensee determined that personnel cross-threaded the back-shell during a prior assembly and that the loose connector broke off during a "wiggle test." The team determined that the licensee had taken too narrow of a focus when resolving the November 10, 2008, event since they concluded the deficiency did not exist on Emergency Diesel Generator 1 because of a lack of vibration-induced anomalies or deficiencies.

The licensee identified the root cause as improper assembly that resulted from disassembling the Amphenol connector without disconnecting the cable clamp. This deficiency provided a tensile stress and weakened the A and B connectors. Subsequent reassembly of the connector resulted in cross-threading the male and female sections.

The cross-threaded parts fretted, wore and loosened. Once they loosened this allowed the engine vibrations to fatigue and separate the Connector A wires because of the previous improper assembly.

Analysis.

The team determined that the failure to identify the correct root cause for the emergency diesel generator Amphenol connector failures was a performance deficiency.

The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because the finding was associated with the design control attribute of the mitigating systems cornerstone and affected the cornerstones objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because the licensee did not use conservative assumptions to ensure that they performed an effective root cause evaluation for failures of the emergency diesel generator Amphenol connectors H.1(b).

Enforcement.

Title 10 Code of Federal Regulations Part 50, Appendix B, Criterion XVI, "Corrective Action," requires that, "Measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition." Procedure 0.5.Root-Cause specified requirements for conducting root cause evaluations and determining the root cause. Contrary to the above, for the period April 21, 2008, through January 31, 2009, for a significant condition adverse to quality the licensee failed to identify the cause of the condition and take corrective action to preclude repetition. The licensee documented this deficiency in their corrective action program as Condition Report 2009-00778. Because the finding is of very low safety significance and has been entered into the corrective action program, this violation is being treated as the second example of a noncited violation, consistent with Section VI.A of the Enforcement Policy: Noncited Violation 05000298/2009008-06, Failure to identify root cause of a significant condition adverse to quality.

.2 Failure to provide adequate work instructions to perform maintenance

Introduction.

The team identified the second of two examples of a Green noncited violation of Technical Specification 5.4.1.a for the failure of the licensee to provide adequate work instructions that resulted in four instances of inadequate maintenance on emergency diesel generator Amphenol connectors that contributed to emergency diesel generator maloperation. The team determined that the licensee had not established appropriate maintenance instructions with sufficient detail to conduct maintenance on Amphenol connectors commensurate with the skill or training of personnel performing the maintenance. The licensee documented this deficiency in their corrective action program as Condition Report 2009-02684.

Description.

The team reviewed maintenance work orders, procedures, and interviewed personnel to determine the level of compliance with procedures and instructions and to evaluate the type of qualitative and quantitative guidance provided to maintenance personnel for the assembly of Amphenol connectors. The team determined that the licensee did not develop procedures with quantitative and qualitative acceptance criteria to ensure that personnel satisfactorily accomplished maintenance on safety-related equipment. The team determined that weaknesses related to skill-of-the-craft exacerbated the deficiency. Specifically, for the multiple Amphenol connector events the team identified four instances that resulted from either no instructions or procedure instructions with insufficient detail for performing maintenance on emergency diesel generator Amphenol connectors.

The first instance related to the event on January 14, 2008, when the licensee identified an improperly assembled Amphenol connector on Emergency Diesel Generator 2. At the time of this instance, the licensee relied on the skill-of-the-craft and no instructions related to Amphenol connectors existed. In September 2008, as corrective actions for this and other events, the licensee revised Procedure 7.3.28.5, "Amphenol-type Threaded Connector and Assembly Fixture Instructions," Revision 0, for the proper assembly of Amphenol connectors; determined that training should be provided; and checked for tightness and applied wax (this would show any loosening because of vibration if misaligned) to the Amphenol connectors on both emergency diesel generators.

The second instance related to the event on April 21, 2008, that resulted from an improperly gapped governor magnetic pickup unit (a different Amphenol connector). In response to this event, the licensee replaced the magnetic pickup unit and had a laboratory analyze the damaged magnetic pickup unit. In this example, the team concluded that Procedure 7.3.28.5 did not include precautions related to disturbing the magnetic pickup unit gap settings or for resetting the gap after replacing a magnetic pickup unit.

The third instance related to the event on November 10, 2008, that resulted from an improperly stripped wire jacket and improperly installed cable clamps (one of two screws installed incorrectly). This event involved the same Amphenol connector for the replaced governor magnetic pickup unit from the April 21, 2008, event. The licensee determined that a procedure needed to be created for Amphenol connector assembly and training provided on the procedure once created. Consequently, the licensee superseded Procedure 7.3.28.5, Revision 1 with Procedure 7.3.28.6, "Amphenol-type Threaded Connector and Assembly Fixture Instructions," Revision 0.

The fourth instance related to the event on January 31, 2009, that resulted from a loose Amphenol connector and a cross-threaded back-shell. The original assembly of the connector occurred when the licensee depended heavily on skill-of-the-craft prior to development of the procedure.

From review of Procedures 7.3.28.5 and 7.3.28.6, the team determined that the instructions specified that maintenance personnel record data on an attachment, clean sealant from the threads using a moist cloth if present, perform peer checks, and assure proper Amphenol connector alignment and assembly. The team determined that the instrumentation and control technicians assembled the majority of the Amphenol connectors prior to development of the procedures. The team concluded even with these instructions the licensee continued to rely on a high level of craft skills and did not develop detailed procedures. Personnel stated that the plant relied heavily upon skill-of-the-craft rather than the use of procedures. Further, the line organization expected instrumentation and control technicians to have acquired skills relating to Amphenol connectors before they began working at the site. During interviews of the instrumentation and control technicians, the team identified that only one out of eight personnel interviewed had any experience with Amphenol connectors prior to working at the facility.

Analysis.

The performance deficiency associated with this finding involved the failure to ensure maintenance personnel took the appropriate actions when performing maintenance. The team determined that the performance deficiency was more than minor in accordance with Manual Chapter 0612, Appendix B, because it is associated with the mitigating systems cornerstone attribute of equipment performance and it affects the cornerstone objective of ensuring availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings,"

the finding was found to have very low safety significance (Green) because it was not a qualification deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a cross cutting aspect in the area of human performance associated with decision making because incorrect assumptions by the licensee regarding the skills and knowledge level of the craft resulted in maintenance procedures that had insufficient instructions H.1(b).

Enforcement.

Technical Specification 5.4.1.a. requires, in part, that written procedures be established, implemented, and maintained covering the activities specified in Regulatory Guide 1.33, "Quality Assurance Program Requirements (Operations),"

dated February 1978. Regulatory Guide 1.33, Appendix A, "Typical Procedures for Pressurized Water Reactors and Boiling Water Reactors," Section 9.a, requires that maintenance affecting the performance of safety-related equipment be properly performed in accordance with written procedures, documented instructions, or drawings appropriate to the circumstances. Contrary to the above, between January 14, 2008, and January 31, 2009, the licensee failed to prescribe documented instructions and procedures, for activities affecting quality of a type appropriate to the circumstance.

Specifically, the licensee did not provide maintenance personnel with appropriate procedure guidance related to assembly of Amphenol connectors on the emergency diesel generators.

As a result, the team identified four instances related to the various events where inadequate instructions contributed to the events. The licensee documented this deficiency in their corrective action program as Condition Report 2009-02684. Because the finding is of very low safety significance and has been entered into the corrective action program, these issues are being treated as the second example with multiple instances of a noncited violation (refer to lubricating oil Section 2.b.4 for the first example), consistent with Section VI.A of the Enforcement Policy: Noncited Violation 05000298/2009008-04, Failure to provide adequate work instructions to perform maintenance.

4.0 Metallurgical Evaluation of Lubricating Oil Pipe Cracks

a. Inspection Scope

In response to the failure of the Emergency Diesel Generator 2 lubricating oil discharge S-pipe in February 2008, the revised root cause attributed the failure to four overstress events on the lubricating oil discharge S-pipe rather than high cycle fatigue. The resident inspectors requested assistance from materials experts in NRC headquarters in determining the appropriate failure mechanism. The NRC material engineers independently reviewed licensee and vendor calculations, drawings, and the original and revised root cause evaluations associated with Condition Report 2008-00998 for Emergency Diesel Generator 2.

One of the material engineers, who participated in the NRC independent evaluations, participated as part of the team during this special inspection. The team interviewed personnel from Structural Integrity Associates, Southwest Research Institute, and licensee experts in materials and mechanical engineering. The team independently evaluated the methods used to estimate the crack growth, and the stresses experienced by Emergency Diesel Generator 2 in February 2008, and Emergency Diesel Generator 1 on January 27, 2009.

The team interviewed personnel who took vibration measurements, personnel who performed the calculations, assessed the locations selected for taking vibration measurements The team independently assessed the licensee determinations of material stresses, Crack Opening Area, and crack growth rate.

b. Observations and Findings

Review of Lubricating Oil Discharge S-pipe Repair Actions The team reviewed the immediate actions taken by the licensee to restore operability of the Emergency Diesel Generator 1 lubricating oil discharge S-pipe. The licensee replaced the affected lubricating oil discharge S-pipe and took the following additional actions:

(1) used full penetration welds with a 2:1 weld profile and polished the weld-to-pipe interface to better distribute the stresses across the weldment and minimized stress risers;

(2) machined the pump end lubricating oil discharge S-pipe flange following welding to eliminate flange warping; and

(3) confirmed by dye penetrant examination no evidence of weld cracking.

Because the licensee measured high vibrations after initial installation, the licensee took the following additional actions:

(1) installed a clamp between the emergency diesel generator lubricating oil discharge S-pipe and suction pipe to stiffen the piping system and move the natural frequency farther from resonance and

(2) performed impact testing of the replacement lubricating oil discharge S-pipe with the clamp installed shortly after securing the diesel generator. The 173 Hz piping natural frequency demonstrated that the clamp had moved the piping system well away from the 110 Hz forcing frequency and demonstrated a significant stiffening of the installed configuration.

The team determined the changes in the weld style and actions taken to reduce stress risers aligned with recommendations in the Electric Power Research Institute Fatigue Management Handbook, Volume I. The licensee implemented appropriate immediate corrective actions to repair and ensure operability of the Emergency Diesel Generator 1 lubricating oil discharge S-pipe.

With the clamp installed, the licensee initiated actions to perform vibration monitoring and dye penetrant examinations on the welds during monthly operability tests. During design reviews the licensee initiated Condition Report 2009-02213 that described the potential for the stiffener brace to have created a new failure mechanism. Although the team expressed concern with the clamp installation, the team considered the configuration an appropriate compensatory measure until the licensee installed the permanent design change. The team determined that the licensee had contracted with Structural Integrity Associates to use the velocity measurements and associated displacement measurements in a detailed pipe finite element model to predict the actual stresses. After determining the stresses that created the Emergency Diesel Generator 1 lubricating oil discharge S-pipe failure, the licensee will have Structural Integrity Associates use the data to redesign the lubricating oil discharge S-pipe arrangement to be more high cycle fatigue resistant.

Safety Significance of Emergency Diesel Generator 2 Lubricating Oil Discharge S-pipe Cracking The team relied upon engineering approximation to estimate the significance of the Emergency Diesel Generator 2 lubricating oil discharge S-pipe crack leakage. For an elastic system, the team assumed that the following dependent factors were directly proportional: stress increased with the load, the crack driving force increased with the stress, and the Crack Opening Area increased with the stress intensity. Consequently, the team determined that a factor of two increase in the Crack Opening Area corresponded to a factor of two on the stress. The team considered this a conservative approximation since the stress intensity range and not just the maximum stress intensity was needed to accurately capture the crack driving force. In addition, the team ignored the effects of the mean stress in this estimate although the mean stress needed to be addressed for an accurate estimate.

However, the factor of 2 on stress, and factor of 2 on stress intensity, led to a factor of 8.4 (23.07) on crack growth rate1. This increased crack growth would decrease the time to failure. Using the average strength and minimum toughness values from literature, the team performed a fracture mechanics calculation to estimate the critical crack size.

For the maximum calculated stress amplitude of 7.6 ksi (determined from Structural Integrity Associates analysis), the team calculated failure would occur in about 170 days at a critical crack size of 9.5 inches. Using the increased crack growth rate and stresses from the engineering approximation, the team determined that for a 15 ksi maximum stress amplitude (factor of two on stress), the failure would occur in 12.5 days at a critical crack size of 5.7 inches. Within the 24-hour period after discovery, the flaw would have grown to about 4.08 inches giving a leak rate of approximately 8.3 gallons per hour. The analysis did not account for a decreased stiffness in the piping system once the crack grows and the corresponding decrease in the loads on the piping system. The decrease in loads demonstrates the calculated crack growth and thus leakage would be conservative since the applied loads because the change in system compliance would be less than used in the prediction.

The team determined that a loss of 344 gallons of lubricating oil within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> would render the emergency diesel generator inoperable. The team calculated a lubricating oil leak rate of 14.3 gallons per hour would create a loss of 344 gallons of lubricating oil in 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. Using the engineering approximation methodology describe earlier, the team estimated the load required to render the emergency diesel generator inoperable because of a crack in the lubricating oil discharge S-pipe. Using the orifice flow calculations from before, the Crack Opening Area calculated for the required (14.3 gallons per hour) leak rate is 0.00135 square inches, which corresponded to a factor of 2.3 on crack opening displacement for a 4-inch flaw. Again, by using a ratio to determine the stress intensities (hence loads), the ratio of 2.3 would require a stress equal 34.5 ksi to cause 14.3 gallons per hour leakage from a 4-inch crack. The team The team used the fatigue crack growth exponent of 3.07 listed in a Structural Integrity Analysis report.

agreed that the applied load would decrease by 25 percent because of a change in system compliance with the addition of the crack.

The team determined that, with no pre-existing crack, a bending stress of 46 ksi would be needed to create 14.3 gallon per hour leak. Since this bending stress exceeds the yield stress of the material, the conditions required to cause failure would be either limit failure or low cycle fatigue. The team determined that the metallographic evidence did not support either of these failure mechanisms. The team independently concluded the evidence from the Emergency Diesel Generator 2 failure suggested:

(1) a high cycle fatigue mechanism caused the failure created by a combination of cyclic loads and a mean stress;

(2) the mean stress and cyclic loads may not all be known, which prevents using their loads to accurately predict the leak rate or loading condition; and

(3) the loads required to produce a leak rate that would remove 344 gallons of oil within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> are unreasonably high and would promote a failure mode that did not occur in Emergency Diesel Generator 2. Therefore, the team determined the safety significance from a crack growth and leakage perspective was low.

Safety Significance of Emergency Diesel Generator 1 Lubricating Oil Discharge S-pipe Cracking At the end of the onsite portion of the Special Inspection, the licensee had not received the Emergency Diesel Generator 1 stress analysis from Structural Integrity Associates.

Consequently, the team could not independently evaluate a detailed calculation and assess the safety significance of the Emergency Diesel Generator 1 lubricating oil discharge S-pipe crack. The team interviewed design and analysis personnel and reviewed available documentation. In response to questions by the team, the licensee performed a qualitative analysis that provided information that the lubricating oil discharge S-pipe would not fail, Emergency Diesel Generator 1 would meet the 24-hour mission time, and this event had very low safety significance. The team had the following observations related to the qualitative analysis provided by the licensee.

Although the piping system had operated in resonance with the lubricating oil pump forcing frequency, Structural Integrity Associates estimated, using a basic calculation, that the stress amplitude remained below the endurance limit for the material. The team concluded the licensee had performed a reasonable estimate of the expected loads since the analyses showed that the system had a very low mass participation factor at the natural frequency corresponding to 110 Hz, which suggests the load amplification effect at this mode would be small.

Since the lubricating oil discharge S-pipe had cracked because of high cycle fatigue, the team determined the mean stress must have been high enough to cause the vibration-induced loads to exceed the endurance limit. The team identified the drivers for mean stress in this system could be deadweight, thermal expansion, welding residual stress and flange deflection, and pressure thrust. Several of these mean loads were displacement controlled and would be significantly reduced with a change in system compliance, i.e. the presence of a crack. Therefore as the crack grows, the loads would decrease and likely arrest before the crack reached a critical size.

The licensee obtained more accurate leak rate information after the Emergency Diesel Generator 1 lubricating oil discharge S-pipe cracked and found the leak rate to be much lower than those measured in Emergency Diesel Generator 2. With similar systems, similar through-wall crack sizes, and a lower leak rate, the team considered it appropriate to conclude that Emergency Diesel Generator 1 experienced lower operating loads. Further, with the licensee analyses results, the team concluded a high mean stress caused the cyclic loads to be above the endurance limit and lead to the high cycle fatigue failure. Once the crack propagated through wall and leaked, the resulting change in compliance relieved a portion of this mean stress, causing the lower leak rates. The team determined the above factors would result in a smaller amount of crack growth for Emergency Diesel Generator 1 lubricating oil discharge S-pipe.

The team qualitatively assessed the cracked Emergency Diesel Generator 1 lubricating oil discharge S-pipe by comparing the estimated vibratory stresses, mean stresses and leak rates between the Emergency Diesel Generator 1 and Emergency Diesel Generator 2 failures. The team concluded that the crack growth in Emergency Diesel Generator 1 would not have caused leakage that exceeded 14.3 gallons per hour, therefore, would not have threatened the ability to meet the 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> mission time.

Based on this qualitative evidence combined with the compensatory measures of monthly vibration readings and dye penentrant examination, the team concluded the Emergency Diesel Generator 1 crack growth and leakage have very low safety significance.

5.0 Review of Operating Experience

a. Scope

The team reviewed internal operating experience by obtaining a list of plant modifications related to the diesel generators and selecting those modifications that would have affected the emergency diesel generator lubricating oil subsystem or the electronic governor. The team further selected modifications that had addressed vibration-induced failure on the emergency diesel generators no matter the affected system. The team requested key word searches of the corrective action program related to electrical connectors. For the lubricating oil system, the team selected corrective maintenance activities that had affected diesel generator subsystems.

For external operating experience, the NRC Operating Experience Branch (IOEB)provided the results of keyword searches related to Amphenol connectors, Cooper Nuclear Station, and Cooper Bessemer diesel generators and vibrations. The NRC Operating Experience Branch provided a list of licensee event reports, NRC information notices and some other operating experience information. The team selected operating experience information that was applicable to this inspection and reviewed whether the licensee had addressed the items in their root cause analyses related to these events or had processed the information through their operating experience program.

b. Findings and Observations

The team determined that the licensee has had a long history of events that challenged operation of their emergency diesel generators from review of the licensee event reports.

Each of these failures resulted from vibration-induced high cycle fatigue. The team verified that the licensee had taken appropriate corrective actions to address each of the failures. The team described the impact of vibrations in Section 4OA2.b.1.

6.0 Review of Mitigating System Performance Indicator and Maintenance Rule

a. Inspection Scope

The team reviewed the licensee maintenance rule evaluations to determine whether the licensee appropriately accounted for unavailability and unreliability for each of the lubricating oil and Amphenol connector events. Similarly, the team reviewed the licensee assessments for the mitigating system performance indicator. The team evaluated whether the licensee considered the components in scope and, if in scope, whether the failure affected the mitigating system performance indicator.

b. Findings and Observations

The team determined that the licensee had appropriately assigned the failures as maintenance rule functional failures according to their impact on the ability of the diesel generator to perform the function related to starting and carrying safety-related loads.

The team determined that the licensee had placed the emergency diesel generators in 10 CFR 50.65(a)(1) and established appropriate performance monitoring goals.

7.0 Potential Generic Issues

a. Scope

The team evaluated both the Emergency Diesel Generator 1 and Emergency Diesel Generator 2 discharge lubricating oil S-pipe failures and the Amphenol connector deficiencies to determine whether any potential generic issues should be communicated to the industry (e.g., Information Notices, Generic Letters, and Bulletins).

b. Findings and Observations

The team determined that the number of different issues related to Amphenol connectors at this facility may warrant a generic communication informing other licensees of the types of problems encountered. The team determined that the problems with the Amphenol connectors resulted from poor work practices (i.e., poor wire stripping, poor solder joints and use of oversize voltmeter probes), improper Amphenol types (improper mating because of thread count), and less than optimum application (e.g., failure to use lock wires or not using a vibration resistant Amphenol connector design). The team will recommend issuance of an NRC information notice.

The team did not identify any potentially generic safety issues during the inspection.

4OA2 Identification and Resolution of Problems

a. Inspection Scope

The team reviewed maintenance, corrective action, and modification history related to the emergency diesel generators to evaluate whether any longstanding issues continued to impact current performance. The team also reviewed the operating experience database and previous inspection reports. The team interviewed the root cause team, the system engineer, and other personnel. The team determined that the licensee had experienced numerous failures related to high cycle fatigue and skill-of-the-craft had been an issue in the past.

Prior to the start of this special inspection, NRC had completed a 95002 inspection because three White findings had placed the licensee in the Degraded Cornerstone column. One of the findings related to the January 15, 2008, loose Amphenol connector.

Because the failures reviewed related to Amphenol connectors, the team reviewed the licensee common cause evaluation documented in Condition Report 2008-05767 that the licensee had performed as part of their evaluation for extent of condition. The team compared the conclusions related to the root cause and contributing causes in Condition Report 2009-05767.

.1 Vibration Impacts

The team determined that the licensee had experienced numerous vibration-induced failures of emergency diesel generator components, particularly related to control air tubing. In 1990, the licensee replaced copper control air tubing with stainless steel tubing and relocated tubing and gauges for the different subsystems (control air, lubricating oil, and jacket water) from the emergency diesel generators to local stand-alone pedestals. One of the relocated fittings did not match the design specified tubing length or fitting style, which resulted in four separate failures over a 5-year period.

The license determined maintenance personnel did not install the tubing in accordance with design drawings and specifications. As described in this inspection report, vibration contributed to the Amphenol connector failures. The team determined that vibration had introduced problems with the Amphenol connectors since initial installation in 1995 when Amphenol connectors loosened during acceptance testing. The licensee used Loctite to secure the Amphenol connectors as discussed in Problem Incident Report 1-19594.

Similarly, the vibration-induced high cycle fatigue resulted in cracking of the lubricating oil discharge S-pipe.

From interviews, the team determined that Condition Report 2008-09003 documented another instance where vibration contributed to the failure of a component located on the emergency diesel generators. Specifically, the nonsafety-related mechanical over speed trip cables grossly exceeded their pull test acceptance criteria of less than 35 lbs-force.

Based upon laboratory analysis, the licensee determined the cables bound as a result of vibration-induced False Brinelling (fretting that result from low loading high frequency vibration). The precision control, low sliding friction, ball-bearing type, mechanical cable bound when the bearings flattened because of vibration-induced fretting. The team determined that the licensee had identified these cables had history of failing to pass surveillance tests related to binding on numerous occasions since initial installation in 1995.

The responsible system engineer described that the apparent cause for Condition Report 2006-02054 identified binding of these over speed trip cables. The corrective actions increased the preventive maintenance frequency to annually and stopped lubricating the cables since the vendor described using a lubricant could worsen cable operation. The licensee had incorrectly decided to lubricate the cables to minimize binding as described in Condition Report 2001-01253. Condition Report 2003-00719 identified that the cables failed because of vibration-induced rubbing through the jacket.

.2 Procedure Guidance Not Commensurate with Skill-of-the-Craft

As discussed earlier in this inspection report, the team determined that the licensee had not developed procedure guidance sufficiently detailed to ensure instrumentation and control technicians correctly assembled Amphenol connectors. From review of plant procedures and discussions with instrumentation and control technicians, the team concluded that including more detail or quantitative criteria, as necessary, in work instructions could eliminate uncertainty and provide more consistency.

From review of previous inspection reports and discussions with personnel familiar with the facility, the team determined that the licensee has had continuing challenges developing procedures commensurate with the skill of the craft personnel performing the work activities. Over the last 9 years, the licensee had experienced two additional maintenance personnel capability issues. Specifically, in 2000 (refer to Inspection Report 05000298/2000-07), the licensee concluded that maintenance procedures that provided guidance for performing Environmental Qualification splices provided inadequate guidance for the skills possessed by the craft to perform the splices.

Another issue that occurred in 2006 (refer to Inspection Report 05000298/2006-05)related to assembly of gears in motor-operated valve operators. The licensee concluded the procedures provided insufficient guidance relative to the skill level of the maintenance personnel performing the maintenance.

The team determined that the licensee had recently identified that training considered several basic tasks as "no train tasks," that is, these tasks required no training and incoming craft personnel already possessed these skills. The licensee concluded that this may have been an erroneous conclusion. Specifically, the hiring pool used to include personnel from the armed forces that had some familiarity with the types of components used in nuclear plants. Personnel recently hired may not have had these same skills when hired from the local community with a technical degree or experience in the commercial electronics field.

.3 Relation to 95002 Inspection Results

The licensee determined that a programmatic deficiency among the three events that resulted in the licensee entering the regulatory response band and performance of a 95002 NRC evaluation related to poor rigor and guidance for performing failure mode effects analyses, as documented in Condition Report 2008-05767. The licensee determined that the poor performance of failure mode effects analysis in response to the April 2008 deficiency did not identify all potential failure modes and systematically support/refute each potential failure mode. The team did not identify any concerns with the licensee conclusion that the corrective actions taken to address the programmatic concerns addressed the poor failure mode effects analysis performed for Condition Report 2008-02692.

The licensee documented their Common Cause Evaluation in Condition Report 2008-05767. From review of the evaluation, the team determined that both Root Cause 1 that specified management had accepted marginal standards in risk-significant areas and Contributing Cause 3 that identified personnel incorrectly completed risk-significant tasks because of guidance that did not contain the required level of detail or contains ambiguous terms that unknowingly placed workers into an error-likely situation could apply to the issues identified by this special inspection.

Specifically, the team determined that the licensee documented in Condition Report 2008-08248 that a root cause of the Amphenol connector deficiencies resulted from the failure of management to establish fundamental standards. This failure to establish standards had resulted in poor work implementation by maintenance personnel on four separate occasions since 1999.

.4 Failure to identify conditions adverse to quality

The team identified two additional examples that related to failure of personnel to initiate condition reports, as required by Procedure 0.5, Section 7.1.3. The team determined that these examples were not more than minor in accordance with Manual Chapter 0612, Appendix B, because neither example of failure to write a condition report would have the potential to lead to a more significant safety concern. In the first instance, a technician identified the inappropriate use of Loctite on Amphenol connectors for their application on the emergency diesel generators since the licensee disconnects the connectors to perform testing. In the second instance, a technician found a small amount of foreign white substance from the threads of an Amphenol connector with a screwdriver that was preventing reassembly. In this case, the technician had not notified anyone nor documented the condition in a work order or condition report. The licensee initiated Condition Report 2009-03674 for these deficiencies.

The performance deficiency associated with this finding involved the failure of instrumentation and control technicians to initiate condition reports on two separate instances for an adverse condition as required by the corrective action program.

4OA6 Meetings, Including Exit

On February 27, 2009, the team presented the preliminary results of this inspection at the end of the onsite week to Mr. S. Minahan, Vice President Nuclear and Chief Nuclear Officer, and other members of his staff who acknowledged the findings. The team returned all proprietary information reviewed during the inspection prior to exiting the site.

On May 12 2009, the team leader presented the final results of the inspection to Mr. B. OGrady, Site Vice President, and other members of the licensee staff. The team obtained permission to use the diagrams and photographs in this report from the licensee.

ATTACHMENT 1:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

J. Austin, Manager, Emergency Preparedness

D. Buman, System Engineering Manager

R. Estrada, Corrective Action and Assessment Manager

D. Heath, Consulting Engineer, Structural Integrity Associates

T. Hottovy, Engineering Services Division Manager

K. Kreifels, Maintenance Engineer

M. Metzger, System Engineer - Diesel Generator

S. Minahan, Senior Vice President and Chief Nuclear Officer

B. OGrady, Site Vice President

J. Stough, Control Room Supervisor

S. Nelson, Risk Management Supervisor

G. Szasz, Consulting Engineer, Structural Integrity Associates

D. Vanderkamp, Licensing Manager

D. Willis, General Manager Plant Operations

A. Zaremba, Director, Nuclear Safety Assurance

NRC Personnel

M. Chambers, Resident Inspector

B. Rice, Reactor Engineer

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

050000298/2009008-01 NCV Inadequate corrective actions to repair a lubricating oil pipe (Section 2.b.1)

050000298/2009008-02 NCV Failure to perform effective common mode failure evaluation (Section 2.b.2)

050000298/2009008-03 NCV Failure to identify a condition adverse to quality (Sections 2.b.3)

050000298/2009008-04 NCV Failure to provide adequate work instructions to perform maintenance (Sections 2.b.4 and 3.b.2)

050000298/2009008-05 NCV Failures to identify nonconforming design characteristics (Sections 2.b.5)

050000298/2008005-06 NCV Failure to identify root cause of a significant condition adverse to quality (Section 3.b.1)

Attachment 1