ML12006A049

P-7C Coupling Failure Root Cause

ML12006A049
Person / Time
Site:	Palisades
Issue date:	09/08/2011
From:	Entergy Nuclear Operations
To:	NRC/RGN-III
References
EA-11-241, Meeting Notice - ML113530619 CR-PLP-2011-03902
Download: ML12006A049 (151)

Text

Entergy Operations, Palisades CR-PLP-2011-3902 Page 1 of 151 Root Cause Evaluation Report Service Water Pump 7-C Line Shaft Coupling Failure CR-PLP-2011-03902, EVENT DATE: 08-09-2011 REPORT DATE: 09-08-2011, Rev. 0 POSITION NAME/DEPT.

DATE RCE Evaluator Paul M. Deniston (PLP, Systems Engineering)

Jason Gosler (PLP, Mechanical Maintenance) 09-08-2011 Technical Reviewer George Licina (Structural Integrity) 09-08-2011 Independent Reviewer Kyle Langston (PLP, Systems Engineering) 09-08-2011 Responsible Manager Jody Haumersen/James Forehand (PLP, System Engineering) 09-08-2011 CARB Chairperson David Hamilton (PLP, GMPO) 09-08-2011

Problem Statement RCE Report CR-PLP-2011-03902 Page 2 of 136 Service Water Pump P-7C failed unexpectedly at ~1202 hrs on 8/9/2011 resulting in entry into Off Normal Procedure 6.1, "Loss of Service Water", and entry into LCO 3.7.8 (72 Hr Shutdown LCO).

Event Narrative RCE Report CR-PLP-2011-03902 Page 3 of 136

Background

PALISADES SERVICE WATER SYSTEM CONFIGURATION The Service Water System (SWS) at Palisades is comprised of three motor driven vertical multistage pumps supplying water from Lake Michigan to three service water headers. Two of the headers are termed critical headers A and B, which provide cooling to safety and non-safety related components. Each critical header supplies cooling water to one set of the redundant components including emergency diesel generator lube oil and jacket water coolers, a control room air-conditioning unit, an air compressor after-cooler and an engineered safeguards room cooler. In addition, critical header A supplies cooling water to the component cooling water heat exchangers while critical header B supplies cooling water to the containment air coolers. For accident conditions, either train fed by its associated diesel, is sufficient for accident mitigation. The third header is termed non-critical and provides cooling to non-safety related equipment.

Palisades Technical Specifications require that all three pumps be operable. The failure of a single pump requires entry into a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> shutdown LCO Action Statement.

A single header combining return streams from the three supply headers discharges into the cooling tower makeup basin. Leakage of radioactive contamination into the SWS is detected by a radiation monitor installed in the discharge line.

The three Service Water Pumps (SWPs), P-7A, P-7B, and P-7C, are modified Layne and Bowler pumps. They are comprised of a two stage pump end with stainless steel impellers connected to a discharge head by seven columns for a total height of over 40 feet from suction to discharge. The pump end is coupled to the motor through six line shafts, a packing shaft, and a motor shaft connected by eight couplings all of the same design. Figure 1, below, gives a visual representation of the SWPs. Figure 2 shows a shaft coupling.

Event Narrative RCE Report CR-PLP-2011-03902 Page 4 of 136 Figure 1: PLP SWP Rendering Showing 9/29/09 and 8/9/11 Coupling Failure Locations 2009 Failure 2011 Failure

Event Narrative RCE Report CR-PLP-2011-03902 Page 5 of 136 Figure 2: Coupling Drawing from VEN M-11 Sheet 55 The original SWS pumps were purchased to Specification M-11. The three 50% capacity service water pumps were rated at 8000 GPM and 140 ft Total Developed Head (TDH) each. At Palisades in May, 1999, FES-99-001 was approved to change SWS pump line shafts from carbon steel to 416 SS at the recommendation of the vendor at that time, Rotating Equipment Repair. The rationale for the change was improved corrosion resistance and improved material strength.

The specification for line shaft couplings for P-7A, P-7B, and P-7C was changed from carbon steel to 416 SS under EC 5000121762 in December 2007. EC-5000121762 was an update to the SWS configuration to allow the use of stainless steel couplings and shafts. According to the EC, 416 SS was chosen due to its strength, wear resistance and corrosion resistance. The couplings were also redesigned to incorporate an alignment hole that allows visual verification of proper shaft installation. The line shaft couplings for P-7A were replaced per EC-5000121762 under WO 51637416 on April 2009. The line shaft couplings on P-7B were replaced during a rebuild under Palisades PO 10246213 to HydroAire and P-7B was installed under WO 20082 in June 2010. The line shaft couplings for P-7C were changed from carbon steel to stainless steel through the PO 10237148 to HydroAire in June 2009.

Per ASTM Standard A582/A 582M - 95b Standard Specification for Free-Machining Stainless Steel Bars, the hardness of the material should be between 24 and 32 Rc (Rockwell C Hardness) (248 to 302 HB (Brinell Hardness)) for an intermediate temper condition. The material of the coupling is 416 SS per this ASTM standard, but the hardness has been specified to a range of 28-32 Rc per vendor specification. It should be noted that according to the metallurgists from Bodycote and Structural Integrity, Rc hardness greater than 35 dramatically increase the susceptibility of material to intergranular stress corrosion cracking due to excessive material hardness. A more ductile material (i.e., below 32 Rc) is not subject to this phenomenon to the extent that the overly hardened material is.

Event Narrative RCE Report CR-PLP-2011-03902 Page 6 of 136 The ASTM standard A582/A 582M-95B Standard Specification for Free-Machining Stainless Steel Bars contains the required material properties for 416 SS used as the material for the Service Water Pump couplings. The specific material properties provided for in the specification are chemistry and hardness requirements. There are no toughness or tensile requirements listed provided in the specification. The requirements of the specification were transmitted to HydroAire via Vendor Drawing M0011-Sh-00055 with the stipulation that the couplings provided meet the Tempered condition, with an even further restriction that the hardness meet a narrowed 28-32 RC range. Per the ASTM specification the hardness range for 416 stainless in the tempered condition is 248-302 Brinell which equates to a 24.2-32.1 Rc. Palisades chose to limit the hardness range to the upper half of this band in order to mitigate the effects of galling during pump assembly.

Heat treating of 416 stainless steel is covered under the American Society of Metallurgists (ASM) Heat Treaters Guide: Practice and Procedures for Irons and Steels.

For hardening, the guide indicates that the parts should be preheated at 1400-1455°F, austenitized at 1695-1850°F and oil quenched. A review of the provided BodyCote travelers provided for HydroAire Job #5912 (the couplings that were in P-7C when it failed in August 2011) indicate that they were hardened (per the guide).

For tempering 416 stainless steel, the guide provides the following guidance;

.Temper at 565 to 605°C (1050-1125°F) for hardness approximately 25 to 31 HRC. Tempering at 370-565°C (700-1050°F) not recommended for parts requiring high toughness and optimum corrosion resistance. Causes a marked dip in impact resistance and lowered stress corrosion cracking resistance. Double tempering beneficial. Cool to room temperature between tempers.

A review of the data provided in the BodyCote travelers provided for HydroAire Job

1. 5912 (the couplings that were in P-7C when it failed in August 2011), show that two tempers were performed. The first temper was performed at 1075-1100°F, which is in the range suggested by the guide. The second tempers, for all couplings, was performed at 1025°F, which is less than the range suggested by the guide and is in the range where impact resistance and stress corrosion cracking resistance are adversely impacted.

Some metrological terms will be used throughout this report. Provided here is a brief definition for clarity.

x Hardening: The heating of a material to a high temperature (1695-1850°F for 416 SS) in order to harden the material. This significantly reduces toughness and ductility x

Tempering: After hardening and quenching, the material is "tempered" at a temperature specified by the ASTM standard in order to improve ductility and toughness x

Tempering Embrittlement: Refers to the over-tempering of a material to the point where the material becomes brittle at the grain boundaries of the metal. This increases the statistical likelihood of Inter-granular Stress Corrosion Cracking (IGSCC), but does not guarantee that IGSCC will occur.

Event Narrative RCE Report CR-PLP-2011-03902 Page 7 of 136 x

Inter-granular Stress Corrosion Cracking (IGSCC): The condition caused by the presence of a susceptible material, a tensile stress and a corrosive environment.

Cracking occurs when all three conditions are met combined with a pit or flaw in the material. The materials between grain boundaries are eroded thus leading to material cracking. The definition of a corrosive environment is dependent on the specific material in question.

EVENT DESCRIPTION Palisades began receiving Operating Experience regarding 416 SS as early as 1993, NRC IN 93-68.

In May 2004, OE from Perry Station was issued regarding coupling failures due to Intergrannular Stress Corrosion Cracking (IGSCC) on 416 SS. Palisades evaluated this under LO-PLPLO-2007-00059. Under this same LO task, NRC IN 2007-05 was evaluated. Palisades concluded that the IN did not apply because the couplings were made of Carbon Steel rather than 416 SS. The responder to the OE was not the engineer who was pursuing the materials change in the SWS.

In 2007, an Engineering Change, EC 5000121762, was completed to allow the use of 416 SS (annealed) line shafts with 416 SS (Tempered) couplings. This EC referenced the above OE yet failed to acknowledge similarities between 410 and 416 SS.

Additionally, this EC incorrectly concluded that raw Lake Michigan water was not a corrosive environment for 400 series stainless steel. During this investigation, the System Engineer who completed the EC was contacted (he is currently retired). The Engineer indicated that the above referenced EC was used to document changes made to the system over time. He also indicated that he had no formal metallurgical background and did not feel that it was necessary to challenge a historical Specification Change that allowed the use of 416 SS couplings based on an equivalency evaluation suggested by the original pump OEM, Layne and Bowler.

During the spring 2009 refueling outage (1R20), P-7C failed to meet the performance criteria specified in RO-144 "Comprehensive Pump Test Procedure, Service Water Pumps". This prompted an emergent decision to replace P-7C online in June 2009.

The purchase order history and work orders associated with SWP 7-C are discussed below. The following table gives a condensed list of coupling purchase orders, their tempering times and the associated temperatures.

Event Narrative RCE Report CR-PLP-2011-03902 Page 8 of 136 Purchase Order Date parts Single or Double Material Min Max Tempering Temps Minutes Notes 19919 4/22/2008 8

S 416 30 32 1050 180 P-7A was rebuilt using couplings from these batches 10237148 5/21/2009 8

S 416 28 32 5/22/2009 3

D 28 31 1050 180 P-7C was rebuilt using couplings from this batch, one failed in 9.29.2009 and was found to be excessively hard 10253715 10/2/2009 10 D

416 28.5 29.5 1075, 1025 240 P-7C was rebuilt after the 8/9/2011 failure event with couplings from this batch 9/30/2009 2

D 416 29 30 1100, 1025 285 9/30/2009 "6+1 D

416 31 31 1070, 1025 340 P-7C was rebuilt after the 9/29/2009 failure using 8 couplings from these batches.

10246213 3/19/2010 8

D 416 28 32 1070, 1090 260 Installed in P-7B June 2010 Table 1: Summary of Coupling Heat Treats and Purchase Orders In June 2009, P-7C was replaced with new line shafts and couplings under WO

1. 190235. The pump tested satisfactorily during post maintenance testing and was returned to service.

On September 29, 2009, P-7C failed. It was determined that the failure had been caused by the failure of coupling #7. The failed coupling was determined to have been improperly heat treated based on high hardness (~37 Rc). The cause of the improper heat treatment was determined to be a quality program problem at HydroAire. The September 2009 coupling failure was investigated during a Root Cause Evaluation conducted under CR-PLP-2009-04519. The RCE conducted after the failure did not sufficiently investigate the base material properties of 416 SS. Specifically, corrosion in the Lake Michigan water environment and the toughness properties of the material were not investigated.

During the September 2009 event response for P-7C, an emergent Purchase Order #

10253715 was issued to HydroAire for replacement parts for P-7C due to the fact that Palisades had no stock of replacement couplings. As noted in the Higher Tier ACE conducted under CR-PLP-2009-04806, the lack of spare parts resulted in an expedited procurement, miscommunications and an elevated sense of urgency within the Engineering and Procurement activities. This evaluation team determined that the expedited procurement of replacement couplings in October 2009 was likely a

Event Narrative RCE Report CR-PLP-2011-03902 Page 9 of 136 contributing factor to the coupling failure in August 2011. If sufficient spare parts were in stock when the 2009 event occurred, the communication errors, verbal direction to HydroAire and incorrect independent testing would likely not have occurred. A directed corrective action has been created as part of this evaluation to develop the steps necessary to ensure sufficient SWP spare parts are available in-stock.

The 2009 event response Purchase Order # 10253715 had three revisions. The original had a quantity of zero shaft couplings, revision 1 had a quantity of 8 shaft couplings, revision 2 had a quantity of 16 shaft couplings, and revision 3 had a quantity of 18 shaft couplings. HydroAire began manufacturing the new shaft couplings, line 3 of PO 10253715, under their job #5912. During manufacturing, the components are sent to a separate heat treatment vendor for hardening and tempering. HydroAire uses a Chicago facility owned by BodyCote for heat treatment. BodyCote is a national heat treatment company and the Chicago facility has a 10 CFR 50, Appendix B, qualified QA Program.

HydroAire had three work order numbers of their own associated with the order of the shaft couplings. The shaft couplings were to be made from 416 SS. The three HydroAire WO #s were 27774, 27797, and 27799.

HydroAire WO # 27774 was to manufacture 8 shaft couplings. The shaft couplings in this work order were heat treated by Bodycote under two certification numbers.

Bodycotes certification number 92-42427 certified that 2 shaft couplings were heat treated per HydroAires PO 20163, and these shaft couplings had a minimum hardness of 29.0 Rc and a maximum hardness of 30.0 Rc. Bodycotes certification number 92-42429 certified that 6 shaft couplings were heat treated by Bodycote per HydroAires PO 20166, and these shaft couplings had a minimum hardness of 31.0 Rc and a maximum hardness of 31.0 Rc. Both Bodycote certifications also supplied the hardening, 1st temper, and 2nd temper furnace temperature graphs. The 8 couplings manufactured under HydroAires WO # 27774 were accepted by Palisades under receipt inspection numbers 4724, 4735, and 4737 on October, 1st 2009. These 8 couplings were installed under Palisades WO 208591 into P-7C. The shaft coupling position in P-7C was not known.

HydroAire WO # 27797 was to manufacture 8 shaft couplings, and HydroAire WO #

27799 was to manufacture 2 shaft couplings. The shaft couplings in these 2 work orders were heat treated by Bodycote under certification number 92-42442. Bodycote supplied the hardening, 1st temper, and 2nd temper furnace temperature graphs. Bodycotes certification number 92-42442 certified 10 shaft couplings were heat treated by Bodycote per HydroAires PO 20170, and these shaft couplings had a minimum hardness of 28.5 Rc and a maximum hardness of 29.5 Rc. These 10 shaft couplings were accepted by Palisades under receipt inspection number 4753 on October 5th, 2009 and October 13th, 2009. These couplings remained in storage until 2011 when they were used to restore P-7C.

According to the ACE completed in response to CR-PLP-2009-04608, Design Engineering gave verbal design direction to HydroAire allowing the use of "Double Tempering" of the line shaft couplings in order to assure that the proper material harnesses were achieved. Formal direction via the Purchase Order was not given and under current processes would not normally have been given to HydroAire. Typically when working with a qualified supplier, Palisades specifies the desired end result for a

Event Narrative RCE Report CR-PLP-2011-03902 Page 10 of 136 design characteristic (in this case hardness) and the supplier chooses the manufacturing method necessary to achieve the acceptance criteria. In this specific event, Palisades staff would not have specified heat treatment regimes because HydroAire is the pump OEM for the Service Water Pumps and should be responsible for delivering quality components.

HydroAire supplied Certificate of Conformance and CMTR for all 18 of the supplied shaft couplings. All certificates of conformance and manufacturing travelers indicated that the components purchased by Palisades met the specified material hardness values.

In 2010, Palisades learned that there was OE from Prairie Island Nuclear Generating Station (PINGS) regarding two coupling failures due to IGSCC in 400 Series SS. The failed couplings had been supplied by HydroAire. Information on the OE did not come through the normal OE channels but was provided verbally. Palisades then contracted Structural Integrity to examine the OE and provide an analysis of Palisades vulnerability.

This RCE team initiated CR-PLP-2011-04469 documenting that the OE was not available via the normal channels and requesting that the OE be processed according to Fleet procedures.

In March 2011, Palisades received a report from Structural Integrity (SI), a metallurgical firm under contract to Entergy, regarding the 2010 PINGS OE. SI concluded that the currently installed couplings were satisfactory, but more margin could potentially be gained by either (1) creating a better specification for the 416 SS couplings or (2) changing materials. One potential material that was identified was 17-4PH SS has a much higher toughness value than the 400 series stainless steels. Based on the initial metallurgist recommendation, Palisades began to work with SI and HydroAire to develop a better specification for 416 SS rather than change materials. Although SI indicated that the current parts were satisfactory, Palisades missed an opportunity to question whether or not the spare couplings that had been purchased in 2009 were still suitable for installation into the Service Water System.

At 0700 hrs on August 9, 2011, all three Service Water Pumps, P-7A/B/C, were in service with basket strainer differential pressures at 1/2/1 PSID respectively. Critical Service water Header Pressure was 73 psig and stable. Control Room alarms associated with Service Water System (SWS) were all clear.

At approximately 1202 hrs, with all three Service Water Pumps in service, the Control Room received alarms EK-1163, Critical Serv Water Header 'B' Lo Pressure, EK-1164, Critical Serv Water Header 'A' Lo Pressure, EK-1165, Noncritical Service Water Low Pressure, EK-0557, Diesel Gen No. 1-2 Trouble and EK-1132, Service Water Pump P7A Basket STR HI DP. The Control Room crew entered Off Normal Procedure (ONP) 6.1, Loss of Service Water. Control Room Operators observed SW Pump, P-7C, running at 31 amps. SW Pumps, P-7A and P-7B were in-service at 80 amps. Critical Service Water Header Pressure was 64 psig and stable. Tech Spec LCO 3.7.8 A1, a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action to restore P-7C to operable, was entered. A Nuclear Plant Operator (NPO) dispatched to the Screen house reported that P-7C had no discharge pressure and there was a loud banging noise from P-7C. Control Room Operators stopped P-7C. Critical Service Water Header Pressure was 64 psig and stable. All SWS and the DG 1-2 alarms were clear except EK-1132, Service Water Pump P7A Basket STR HI DP, with P-7A DP at 6 psid due to the rise in flow through the SW Pump P-7A basket strainer.

Event Narrative RCE Report CR-PLP-2011-03902 Page 11 of 136 Work to restore P-7C began immediately under WO #286627. During disassembly, it was determined that line shaft coupling #6 had failed. The failure of coupling #6 was similar in nature to the failure of coupling #7 in September, 2009. As can be seen in Figure #1 (page 4), couplings #6 and #7 are above the water line when the pump is not running. This means that when P-7C is in standby, coupling #6 and #7 have the chance to dry out. Coupling #5 may also be dry, depending on lake levels. As is discussed in the metallurgical analysis from LPI (Attachment V), surface deposits of contaminants were found in the area of the fracture surface. Energy Dispersive X-Ray Spectroscopy (EDS) was performed on the surface deposits which were determined to contain oxides, chlorides and sulfides.

During the restoration phase, spare couplings purchased in October, 2009 were to be used. These couplings were from the same group of couplings that were purchased in response to the September 2009 event. Since the couplings to be installed had a similar pedigree to the currently failed coupling, a conservative decision to have the couplings independently hardness checked prior to installation was made. The couplings were taken by Palisades personnel to Consumers Energy's Trail Street lab for hardness testing. Trail Street determined that all couplings destined for installation in P-7C were within the specified range of 28-32 Rc.

P-7C was restored to service at approximately 0309hrs on August 12th, 2011.

During the restoration of P-7C, the station decided to contract Lucius Pitkins Inc (LPI) to perform detailed metallurgical analysis on the couplings in P-7C. By the time P-7C was restored, preliminary data from LPI pointed to a potential materials issue with the 416 SS. Based on the fact that the replacement couplings installed into P-7C were from the same metal heat and purchase order as the recently failed coupling, a decision was made to do an emergent change to the coupling materials. The decision was made to suspend the development of an improved 416 SS coupling specification and complete EC # 31337. The EC changed the material of the line shaft couplings from 416 SS to 17-4PH SS. Furthermore, a conservative decision was made to change the couplings in all three pumps to the new material in an expedited fashion. As with the 2009 event response, this created a sense of urgency around the manufacturing and procurement of new couplings. Procurement of 17-4PH SS couplings supported the emergent station direction to replace the couplings in all three SWPs. During the procurement, several non-conforming parts were identified. The non-conformances will be addressed in a Higher Tier ACE in response to CR-PLP-2011-04317.

Shown below is a picture of the failed coupling and a close up of the fracture surfaces.

Coupling #6 is the second coupling below the stuffing box and is subject to wet/dry cycles.

Event Narrative RCE Report CR-PLP-2011-03902 Page 12 of 136 Figure 3: Failed Coupling #6 from P-7C Service Water Pump, Fracture Surfaces

Event Narrative RCE Report CR-PLP-2011-03902 Page 13 of 136 Event Investigation A cross-functional Root Cause Evaluation Team was created to conduct the investigation into the Service Water Pump failure. The team was consisted of representatives from System Engineering, Design Engineering, Operations, Training, Procurement Engineering, Mechanical Maintenance and consultants from the Metallurgic and Root Cause Evaluation fields.

The evaluation team looked at four main areas in order to determine the root cause of the Line Shaft Coupling #6 failure. Those areas were:

1. SWP Operation

2. SWP Maintenance and coupling replacement

3. Organizational and Programmatic Issues

4. Failure Mode and Metallurgical Analysis of the SWP Couplings Each area will be discussed in this report.

Service Water Pump Operation The team performed an examination of the Shift Narrative Logs from the period beginning on October 2nd, 2009 and ending on August 9th, 2011. The log entries did not identify any unusual operations of the Service Water Pumps during the time in question. Based on Palisades Standard Operating Procedures, there is neither a minimum run time nor limit on cycling of the Service Water Pumps.

In addition to examining the logs for abnormal usage, the team discussed the history of the Service Water Pumps and design of the pumps. These pumps have been in use since the plant was commissioned and there have been no coupling failures noted prior to the 2009 installation of the 416 SS couplings. Therefore, the team concluded that the SWP design was acceptable for the service environment.

Based on the above information, the evaluation team concluded that there were no indications that the usage pattern for the Service Water System contributed to the failure of the P-7C line shaft coupling.

Service Water Pump Maintenance SWP P-7C had previously experienced a coupling failure on September 29th, 2009 and was documented in the RCE conducted under CR-PLP-2009-04519. During the course of that investigation, the maintenance representative on the RCE team analyzed the work steps and recorded data for the restoration (WO 208591). The analysis concluded that the Mechanical Maintenance team had correctly restored the pump in accordance with the approved work instruction, WI-SWS-M-04.

Subsequent to the August 2011 coupling failure, the Maintenance representative on the current RCE team examined the data recorded during the restoration conducted under WO 286627. This analysis concluded that Mechanical Maintenance had restored P-7C correctly according to WI-SWS-M-04.

Event Narrative RCE Report CR-PLP-2011-03902 Page 14 of 136 In order to confirm that the work instruction was indeed robust and did not have any hidden error traps, the Palisades RCE team hosted a fleet call on 8/23/2011. During the fleet call, input was solicited from the participants regarding the robustness of the published work instruction. There was agreement amongst the non-PLP fleet participants that PLP's Service Water Pump installation instructions were complete, accurate and appropriate. Based on the FEA analysis, the in situ tensile stresses and a description of the actions taken to "snug" the shafts during installation, it was determined that Palisades practice of "bumping" the pump to tighten the shafts was not a likely contributor to the failures in September 2009 and August 2011.

Based on the above data, the RCE team concluded that the Mechanical Maintenance team restored P-7C correctly subsequent to both the September 2009 and August 2011 failures and that the work was completed using an accurate and appropriately rigorous work instruction.

Organizational and Programmatic Factors Investigation To ensure that organizational and programmatic causes were determined and described, an expert in Root Cause Evaluation, was contracted to assist in the review of the September 2009 event (CR-PLP-2009-04519, RCE), mis-communications with the NRC in 2009 (CR-PLP-2009-04806, HT ACE), and the August 2011 event, and then assist the RCE team in the O&P analysis.

The root cause team proceeded under the philosophy that (1) a failure mechanism (in this case stress corrosion cracking) caused the coupling failure and that (2) multiple flawed defenses allowed such a failure to occur. This is of special interest since the August 2011 failure occurred in the same service water pump as the September 2009 failure and was due to the same failure mechanism.

It was recognized that many individuals and groups were involved with service water pump design, material condition, and performance in the period 2009-2011 and earlier.

The assessment as conducted was not intended to blame any individual but to point out ways for Palisades and Entergy organizations to be more effective in the future.

The team began with a simple why staircase tree that laid out three necessary and sufficient requirements for pump shaft coupling failure, namely:

x The vendor HydroAire manufactured susceptible couplings.

x The customer Palisades accepted susceptible couplings.

x Couplings were exposed to a service environment that proved detrimental to them.

Technical details of coupling material deterioration and failure appear elsewhere in this report, and form the basis for the material related cause (RC1, CC1). The O&P evaluation will focus on why the failure was able to occur. An underlying premise is that while qualified vendors may be expected to ensure product quality and suitability for service, the burden is on the nuclear licensee to manage its relationships with such third parties. The team concluded that Palisades did not do so effectively.

Event Narrative RCE Report CR-PLP-2011-03902 Page 15 of 136

Event Narrative RCE Report CR-PLP-2011-03902 Page 16 of 136 Figure 4: Why Staircase Tree The analysis found one organizational Root Cause (RC2) and two Contributing Causes (CC2, CC3) existed they are:

O&P Root Cause (RC2):

Palisade's Engineering specified the wrong Stainless Steel alloy for use in Palisades Service Water operating environment. The choice of 416 SS was based on historical data by personnel who did not have sufficient metallurgic knowledge.

As a result, couplings susceptible to intergranular stress corrosion cracking were available for installation in June 2009, September 2009 and August 2011. Two subsequent coupling failures and one degraded coupling occurred in the intermittently-wetted shaft segment of service water pump P-7C, both originating from IGSCC.

Supporting Facts

1. Purchase specifications and Palisades-approved HydroAire drawings did not contain a toughness requirement for the 416 SS coupling material. This may have discouraged reliance on second or re-tempering to achieve acceptable Rockwell hardness values.

2. To offset concerns about galling during coupling assembly, Palisades narrowed the acceptable Rockwell hardness range from the ASTM value (24-32 Rc) to 28-32 Rc. This made the acceptable hardness band restrictive and moved it closer to 35 Rc, a hardness known for IGSCC susceptibility.

(CC-3)

(CC-2)

(CC-3)

(CC-2)

Event Narrative RCE Report CR-PLP-2011-03902 Page 17 of 136

3. Palisades did not dictate or approve hardness testing sample size, location, or timing (e.g., pre or post machining). This was left to HydroAire and its subcontractor Bodycote.

4. For hardness, Bodycote offered commercial (fractional testing) and MIL-SPEC (100% testing). HydroAire accepted fractional testing. There was no requirement to increase sample size when testing failures occurred.

5. HydroAire and Bodycote were free to retemper non-conforming couplings in an effort to reduce hardness below 32 Rc. If that succeeded, the product could be shipped.

6. The 2007 Engineering Change (5000121762) which produced the procurement specification did not consider fresh water to be corrosive.

7. In 2009, a receipt inspection staff member reported recent work practices that included verbal requests to Hydro-Aire on manufacturing and design issues. The practice led to occurrences that were outside the purchase order process as they frequently did not result in purchase order revisions.

8. As documented in CR-PLP-2009-04806, Palisades was not always well-informed regarding the testing plan details of HydroAire. This apparently resulted from verbal communication of the type described above.

9. Lab data (Trail Street and LPI) indicate that hardness can vary 2 - 4 units Rc from place to place on a finished coupling.

10. Trail Street data also demonstrate that some removed 2009 series couplings exceeded the hardness spec with one from an intact coupling (Top 4, Reading 3) reaching 36.1Rc. During the September 2009 event and this evaluation, the team sought expert advice to explain the discrepancies noted between the hardnesses noted on the Certificates of Compliance and the post failure data collection. Four separate metallurgists have agreed that there is no way that the couplings could have been work hardened within Palisades Service Water System as installed in the plant. Suggested reasons for the discrepancies include inherent piece part variation with in each coupling and M&TE differences between each lab.

11. Macroscopic photos of recently-removed, intact couplings (Pumps P-7B and P-7C) show orange discoloration at the midpoint and ends, similar to failed couplings. Couplings from pump P-7A show a greasy-gray interior appearance and no visual evidence of corrosion.

12. The 2011 procurement was the first time the Critical Procurement Plan (EN-MP-100) had been used for emergent purchase of SWS pump shaft couplings.

13. The system engineer worked closely with HydroAire, but he never chased down the heat treatment labs.

Event Narrative RCE Report CR-PLP-2011-03902 Page 18 of 136

14. Palisades receipt inspections were not written to require testing of any kind.

They included verification of shipment contents, including paperwork. Any other testing would be outside of the normal reciept inspection responsibilities.

O&P Contributing Cause 1:

In 2007, during the service water pump refurbishment Engineering Change, Palisades did not ensure technical oversight by Entergy personnel with expertise in metallurgy.

This resulted in failure to recognize the risks associated with material changes associated with Critical Safety Related equipment.

As a result, the station has accepted supplier product that was inadequate for predicted service conditions and has been unprepared to respond expeditiously to pump failures when they occurred.

Supporting Facts

1. Palisades did not have in 2007 (and does not now have) Engineering Staff with detailed metallurgical knowledge. The corporate culture in 2007 would have made it difficult to obtain outside resources unless a detailed case could be made supporting the need. Under current Entergy fleet procedures and expectations, this roadblock would not exist. As was demonstrated in 2009 and during the event being analyzed in this evaluation, the station has contracted outside expertise in a meaningful way.

2. Palisades considers HydroAire is one of a limited number of available firms capable of overhauling vertical service water pumps and producing replacement parts to Palisadess satisfaction.

3. Since 2000, Palisades has come to rely on HydroAire for reverse engineering and replacement parts related to maintaining the set of three service water pumps. Until 2007, the focus was on deteriorating carbon steel materials and impeller wear from sand entrained in lake water.

4. Alternatives to HydroAire are limited (1) by lack of other qualified nuclear suppliers and (2) PAL reluctance to use FlowServe. According to an interviewee, HydroAire was willing to stop and listen. There was less favorable responsiveness by other companies.

5. HydroAire typically measures old pump parts, redesigns/improves them, and submits drawings to PAL. Drawings include dimensional data and criteria like hardness (Rc). Materials chosen were those commonly used in fresh water service water pumps. According to the design engineer, there was an EC to document all reverse engineering activities.

6. PAL approves the drawings and orders parts from HydroAire. Much of this was handled by a System Engineer until his retirement (~2009). During this time, the Engineer was working under the Nuclear Management Company Contract process instead of using the Procurement activity. This would not be allowed under current Entergy Processes.The 2007 EC reviewed the Perry OE but failed to realize the susceptibility to IGSCC and that Perry had switched material to 17-

Event Narrative RCE Report CR-PLP-2011-03902 Page 19 of 136 4PH SS from 400 Series SS. No further action was taken on the OE because 1) the Engineer involved did not realize that Lake Michigan water was corrosive to 400 Series SS and 2) the equivalency EC process under the NMC did not require an extensive OE evaluation.

7. Recent conference calls have involved multiple metallurgists. The historical design engineer (retired ~ 2009) had no metallurgical background. He relied on input from reverse engineering by HydroAire. Palisades did not leverage group or corporate expertise in 2009. An interviewee stated I cannot phone someone in Jackson (no metallurgists in Echelon) for materials advice.

8. A root cause team member who had participated in 2009 visits to HydroAire and BodyCote characterized the HydroAire work areas as disorganized in appearance. He stated that the president of HydroAire told him that Palisades seemed more interested in speed. An interviewee stated that despite HydorAires nuclear credentials, HydroAire was surprised at the quantity of documentation required.

O&P Contributing Cause 2:

Since at least 2004, there has been substantial nuclear industry operating experience linking IGSCC susceptibility with high Rockwell hardnesses and/or low material toughness in type 416 SS exposed to fresh water. Palisades did not translate this OE into effective specification, contract, inspection, testing, or oversight actions.

Supporting Facts

1. When the first failure occurred (September 2009) there were no spares on hand.

HydroAire was engaged to provide them on relatively short notice (PO 10253715, Revs 0 through 3).

2. When the second failure occurred (August 2011), there were spares available to refurbish one pump, but they were from the 2009 rapidly-executed series.

3. In 2010 operating experience (Prairie Island) suggested that 416SS was potentially ill-suited for fresh water service and that 17-4 SS was preferable.

4. During this time Palisades continued to rely on its stock of 416 SS spares until the 2011 failure. At the time of the failure, Palisades Design Engineering was working with another Metallurgist, Sturctural Integrity, and HydroAire to develop a robust method of specifying a heat treatment of 416 SS that would meet the site needs.

5. In the mid 2000s, according to the retired design engineer, there was a lot of OE out there with hundreds of hits, all related to over-hardening the stainless steel material. Most involved failures of couplings and shafts in salt or brackish water.

6. The 2007 EC was developed using fleet procedures from The Nuclear Management Company (NMC), see discussion below. Current Palisades practice is to not reverify an existing approved Engineering Change when doing a related EC.

Event Narrative RCE Report CR-PLP-2011-03902 Page 20 of 136 NMC procedure FP-E-EQV-01, Equivalency Evaluations and Changes, was used in the preparation of EC5000121762. The procedure does not require the Responsible Engineer to do a review of industry operating experience related to the part being replaced; however, it does say to look at Attachment 6 for a description of error likely situations that may be faced when preparing an equivalency based on fleet operating experience. The attachment discusses error-likely situations for items like to poor original design and extreme service conditions. Attachment 4 of FP-E-EQV-01 actually discusses that the System Engineer be consulted for any site specific or industry operating experience related to the equivalent change.

During the timeframe that NMC operated Palisades there was no engineering pre-job procedure that would allow an engineering product to be considered for higher levels of review based upon a risk versus consequence approach. This type of consideration was typically left to the line supervision to determine. Since the implementation of the Entergy procedures at Palisades an improvement has been made in this regard. EN-HU-104 Engineering Task Risk and Rigor is the procedure that is used to perform pre-job briefs for engineering tasks. The process determines the risks and consequences of things that could go-wrong regarding the engineering product and contains a risk rank scoring system that drives the level of review required for the product. Typically the scoring results in Exisiting Engineering Process Review. However, increasing levels of review could also be specified for high risk rank scores (e.g. Independent Review by Station, Independent Review by Consultant/Specialists, or Independent Collegial Review or Challenge Board). When combined with the more stringent OE analysis and independent review criteria specified by EN-DC-115, Engineering Change Process, Palisades is less susceptible to human error in knowledge space analysis due to more robust awareness of potential error traps.

Entergy procedure EN-DC-115 governs all of the types of engineering changes performed by Entergy including equivalent changes. This process requires that a review of operating experience be performed by the responsible engineer. This operating experience should be both internal and external. While not explicitly stated in EN-DC-115, it would be good practice and thorough to discuss how the found operating experience is applicable to the equivalent change and disposition any of the operating experience that resulted in failures at other sites.

There is a significantly different cultural model in place within Palisades Engineering then in 2007. Site engineers now have a different mindset when it comes to technical rigor and product quality then in the past due to higher expectations from Entergy site and fleet management. When combined with the more rigorous Entergy procedures, the site is able to ensure that a final engineering product meets or exceeds standards.

Event Narrative RCE Report CR-PLP-2011-03902 Page 21 of 136 Failure Mode and Metallurgical Analysis On 8/15/11, the RCE team began development of a Failure Modes Analysis (FMA) based on the 8/9/11 failure. The development of the FMA included input from metallurgists, pump experts, system engineering, design engineering, and maintenance and encompassed over 70 different failure modes. The FMA is included in Attachment II to this report.

Over the next several weeks, this FMA was further refined with additional details and the appropriate action necessary to support or refute the individual failure modes. In addition, the FMA actions were further categorized under both aspects of failure, as well as plausibility.

As data was received from various parties, the information was added to the FMA spreadsheet, and resultant changes in the support, refute, and plausibility. The FMA was then sorted and the potential causal factors to be investigated are given below:

Causal Factors under Investigation x

Stress Corrosion Cracking o Components Replacement in last 5 years Improper Coupling Material Inadequate Hardness Specification for Coupling Inadequate Cooling Process x

Improper Hardness o Corrosion Caused by Environmental Conditions o Water Chemistry Lake Michigan Chlorination of Service Water Lucius Pitkin, Inc. (LPI) Testing NOTE:

Material Test data presented is based on a Draft Report from Lucius Pitkin, Inc. (LPI). The final copy was not completed at the time of this writing.

Couplings from P-7A and P-7B were sent to LPI for comparative analysis.

Based on current timing information from LPI, the final report will not be available until after the CARB approval date for this analysis. There will be a need for a substantive revision to this evaluation in order to include all final data from LPI. This revision will be brought back for approval at the level specified by CARB.

In order to properly characterize and diagnose the failure of the #6 coupling on the P-7C, Entergy contracted LPI to perform metallurgical examination of the Service Water components for both the 2011 and 2009 coupling failure. The metallurgical examination included several destructive and non-destructive tests. These tests and the currently available data are listed below, full details of the testing and results are in Attachment IV of this report.

Event Narrative RCE Report CR-PLP-2011-03902 Page 22 of 136 Test Parameters Visual and Photographic Examination Visual examination of the 2011 coupling failure indicates stress corrosion cracking (SCC), initiating from two locations from the interior of the coupling to the exterior perpendicular to the axis of the coupling.

The crack initiation sites were one thread apart vertically, and not associated with the vent hole. The SCC propagated in an elliptical fashion through the thickness of the coupling. Final fracture was due to overload of the remaining material.

Surface Hardness Testing Surface hardness of the failed coupling averaged 33.3 Rockwell C. This exceeds the maximum specified hardness of 32 Rc. Couplings #5 and #7 averaged 29.6 and 31.4 respectively.

Dimensional Examination Coupling OD and length are within specified dimensions and tolerance. Eccentricities of coupling were checked and coupling 4 had the highest at 7 mils.

Compositional Analysis of Surface Deposits EDS of surface deposits indicates presence of chlorides, sulfides and oxides Magnetic Particle Examination MT examination shows cracks through the failed coupling #6, and indications of a crack in coupling #7 at the thread root closest to the vent hole, but not through the vent hole.

Tensile Testing Ultimate Tensile Strength - 155 ksi Yield Strength - 140.6 ksi Charpy V-Notch Testing Charpy Impact testing indicates that the failed coupling had low impact toughness and leads the material to be vulnerable to SCC Through Thickness Hardness Testing Through thickness hardness test of failed coupling 6 ranged from 31.5 to 32.7. For 5 and 7 values were less than 32.

Compositional Analysis Compositional Analysis confirms the couplings are made of 416 Stainless Steel Scanning Electronic Microscopy SEM examination revealed the fracture surface morphology to exhibit a rock-candy appearance, characteristic of intergranular stress corrosion cracking (IGSCC).

Event Narrative RCE Report CR-PLP-2011-03902 Page 23 of 136 LPI Conclusion Summary Palisades SWS pump P-7C coupling #6 failed in August, 2011. The failure is determined, based on metallurgical evaluation, to be the result of intergranular stress corrosion cracking (IGSCC). The 2009 failure of the #7 coupling on the same pump (P-7C) was determined under CR-PLP-2009-04519 to also be a result of IGSCC. LPIs independent examination of the 2009 failed coupling 09-P7C-7F concurs with the failure mode as documented in the 2009 event RCE.

For IGSCC to occur three criteria to promote IGSCC must exist; 1) susceptible material,

2) tensile stress and 3) corrosive environment. The specified coupling material, ASTM A582 Type 416 stainless steel, is martensitic steel that is susceptible to IGSCC at low toughness. Charpy V-Notch (CVN) testing of the 2011 failed coupling resulted in toughness values in the range of 6 to 10 ft-lbs impact energy for test temperatures of 32°F and 70°F. CVN testing of the 2009 failed coupling resulted in impact toughness values in the range of 3 to 6 ft-lb for test temperatures of 32°F and 70°F, respectively.

These low impact toughness values make the couplings susceptible to IGSCC.

The couplings are subjected to tensile stresses during normal operation by the weight of the components below the coupling and hydrodynamic forces due to pump operation. In addition, the design of the couplings results in the shaft ends bearing against each other that likely led to sufficient tensile stresses (with a maximum value near the center where the two shafts bear against each other) in the coupling to initiate and propagate a crack.

The majority of the pump couplings below the packing (couplings #1 through #4) are submerged below the water level in the intake structure at normal basin levels.

Couplings #5 through #7, above normal basin water levels see intermittent cycles of wet and dry depending on whether the pump is operating. When the SW pumps are on, all couplings below the stuffing box are wet and when they are off, couplings #5, #6 and #7 begin to dry. Chemistry samples of the service water indicate that there are low levels of chlorine in the raw water of Lake Michigan on the order of 9 ppm. Chlorination of the service water increases the chlorine level slightly to approximately 10 ppm. Even these relatively low levels of chlorine combined with a high humidity oxygen rich environment (as is the case for the couplings #5, #6 and #7 when the pump is off) can lead to a local breakdown of the passivation layer. IGSCC can nucleate at these locally damaged sites, develop and propagate under sufficient tensile stress to form a highly branched network of fine cracks.

Root Cause Evaluation RCE Report CR-PLP-2011-03902 Page 24 of 136 The following Evaluation Methods were used to identify the Root and Contributing Causes:

x Event and Causal Factor Charting (Attachment I) x Failure Mode Analysis (Attachment II) x Metallurgical Testing by LPI (Attachment IV)

Root Causes RC1: The 2009 and 2011 Line Shaft Coupling failures were due to Intergranular Stress Corrosion Cracking (IGSCC). The coupling material is a quenched and tempered 416 martensitic SS with low toughness properties. This makes it particularly susceptible to IGSCC when subjected to the tensile stress and a corrosive environment (due to the presents of chlorides).

RC2: Palisade's Engineering specified the wrong Stainless Steel alloy for use in Palisades Service Water operating environment. The choice of 416 SS was based on historical data by personnel who did not have sufficient metallurgic knowledge.

Contributing Causes CC1: Increased Susceptibility to IGSCC caused by Tempering Embrittlement.

Temper embrittlement can result after tempering in the range of 700F to 1050F or slow cooling through this range. Tempering in the range of 700F to 1050F is not recommended because it results in low and erratic impact properties (i.e. toughness) and poor resistance to corrosion and stress corrosion, which increases the materials susceptibly to stress corrosion cracking.

CC2: Insufficient use of qualified metallurgical expertise. In 2007, during the service water pump refurbishment Engineering Change, Palisades did not ensure technical oversight by Entergy personnel with expertise in metallurgy. This resulted in failure to recognize the risks associated with material changes within Critical Safety Related equipment.

CC3: Ineffective use of Operating Experience.

Since at least 2004, there has been substantial nuclear industry operating experience linking IGSCC susceptibility with high Rockwell hardnesses and/or low material toughness in type 416 SS exposed to fresh water. Palisades did not translate this OE into effective specification, contract, inspection, testing, or oversight actions. (Addresses CR-PLP-2011-03975, closed to this evaluation)

Root Cause Evaluation RCE Report CR-PLP-2011-03902 Page 25 of 136 Organizational and Programmatic Weakness Evaluation Organization to Organization Interface Weaknesses OP1A: "Inadequate interface among organizations." (CC3)

Analysis of the 2009 coupling failure Root Cause (CR-PLP-2009-04519) and the 2009 HT ACE regarding miscommunications (CR-PLP-2009-04806) showed that Palisades Engineering bypassed MP&C by giving verbal directions to HydroAire.

Subsequently, these verbal directions were not captured in a PO.

Organizational to Program Interface Weaknesses There was no evidence of Organization to Program failure modes for the Problem Statement; therefore all failure modes in this category were eliminated.

Program to Program Interface Weaknesses There was no evidence of Program to Program failure modes for the Problem Statement; therefore all failure modes in this category were eliminated.

Programmatic Deficiencies OP4B: One or more necessary functions required by a process were missing in the implementing procedures. (RC2)

Palisades did not ensure (1) a toughness requirement in the purchase specification or (2) sufficient Rockwell hardness testing to guarantee the absence of coupling material with > 32 Rc.

OP4D: Inadequate verification process. (CC2)

Procedure quality (i.e., purchase specification) did not require HydroAire to test hardness in multiple locations nor did it equip Receipt Inspection to do it on behalf of Palisades.

OP4F: Response to a known or repetitive problem was untimely. (RC2)

Corrective actions that were feasible after the 2010 Prairie Island event were not taken. In 2011 Palisades was forced to rely on existing stocks of couplings that proved to have the same IGSCC susceptibility.

OP4K: Personnel assigned did not have adequate experience or training to perform the work. (CC2)

Before actual coupling failures began to occur, no involved Palisades person had metallurgical subject matter expertise. Even then, such expertise came from various external sources.

Root Cause Evaluation RCE Report CR-PLP-2011-03902 Page 26 of 136 OP4N: Personnel exhibited insufficient awareness of the impact of actions on safety/reliability. (RC2)

Despite the importance of service water to the PRA/PSA analysis (approximately 9th-ranking Palisades system) pump refurbishment efforts took place over a multi-year period, and efforts to ensure sufficient couplings on hand were not aggressive until the need for them arose.

OP4P: Previous industry or in-house operating experience was not effectively used to prevent problems (CC3)

Palisades did not translate a substantial body of OE into effective specification, contract, inspection, testing, or oversight actions.

Organizational Weaknesses OP5F: Corrective action for previously identified problem/event not adequate to prevent recurrence. (RC2)

Palisades coupling replacement in 2009 did not use a product sufficiently resistant to IGSCC in the application and environment. The 2011 failure was essentially a repeat, and the spare couplings on hand for use in the 2011 response were more with the same susceptibility.

OP5AD: Risk/consequences associated with change not adequately reviewed/assessed. (RC2)

Palisades over-relied on vendor expertise and did not provide sufficient technically competent oversight to ensure adequate testing of product quality.

In summary, Latent Organizational Weaknesses were identified as contributing to the causes of the problem. Corrective actions have been developed to correct the identified weaknesses

Root Cause Evaluation RCE Report CR-PLP-2011-03902 Page 27 of 136 Safety Culture Evaluation Attachment III - Safety Culture Evaluation, presents a screening of the Root and Contributing Causes of the RCE against the thirteen Safety Culture impact areas, and further evaluation of the causes at the specific aspect level for the applicable impact areas. The evaluation of the impact areas revealed a potential weakness in Resources, Operating Experience and Accountability. There was no evidence that identified potential weaknesses within the other impact areas.

A potential weakness was identified in the impact area Resources because there was evidence that Palisades specifications for the coupling required use of 416 SS and did not require toughness testing nor adequately test for hardness (insufficient sampling size). Furthermore, Palisades specifications for the coupling did not involve input/review by a qualified metallurgist. This weakness was further evaluated in Table 2 to determine the specific aspect(s) that were indicative of the potential weakness for this impact area with the following conclusions (see Attachment III for definitions of each code):

x The aspect H.2.b was evident in the one Cause. H.2.b is the training of personnel and sufficient qualified personnel to maintain work hours within working hour guidelines.

x The aspect H.2.c was evident in one Cause. H.2.c is having complete, accurate and up-to-date design documentation, procedures and work packages and correct labeling of components.

A potential weakness was identified in the impact area Operating Experience because there was evidence that Palisades did not take full advantage of operating experience suggesting that 416 SS was susceptible to IGSCC. This weakness was further evaluated in Table 2 to determine the specific aspect(s) that were indicative of the potential weakness for this impact area with the following conclusions:

x The aspect P.2.a was evident in one Cause. P.2.a is defined as the licensee systematically collects, evaluates and communicates to affected internal stakeholders in a timely manner relevant internal and external OE.

x The aspect P.2.b was evident in one Cause. P.2. is defined as the licensee implements and institutionalizes OE through changes to station processes, procedures, equipment and training programs.

The potential weakness was identified in the impact area Accountability because there was evidence that a Palisades system engineer functioned as a design engineer when dealing with HydroAire. This weakness was further evaluated in Table 2 to determine the specific aspect(s) that were indicative of the potential weakness for this impact area with the following conclusions:

x The aspect A.1.a was evident in one Cause. A.1.a is that accountability is maintained for important safety decisions in that the system of rewards and sanctions is aligned with nuclear safety policies and reinforces behaviors and outcomes which reflect safety as an overriding priority.

The Safety Culture Evaluation identified potential weakness; however there wasnt evidence that suggested there is an adverse trend within the impact areas and aspects with respect to this evaluation. The proposed corrective actions in this Root Cause Evaluation are sufficient to resolve the potential weaknesses identified in the Safety Culture Evaluation.

Generic Implications: Extent of Condition/Extent of Cause RCE Report CR-PLP-2011-03902 Page 28 of 136 Extent of Problem/Condition The intent for this review is to determine if the line shaft coupling failure of Service Water Pump P-7C may have identified a condition that could currently exist in other plant equipment.

A total of 17 (including P-7C) vertical turbine centrifugal pumps were identified to be in-service at Palisades and the safety classification for these pumps was determined from the Equipment Database in Asset Suite (Refer to Table 1). Of the pumps identified, three were found to be Safety Related, four were Augmented Quality, and two additional pumps that are also covered under the Maintenance Rule related. The line shaft couplings on the Condensate Pumps, P-2A and P-2B, are flexible gear design and out of the scope of this review because they are of a different design and material than those on P-7C. The line shaft coupling material was determined for the remaining pumps by looking into the pump drawings and model information in Asset Suite.

Any couplings with materials other than 416 Stainless Steel were determined to be out of the scope of this review due to the condition of the P-7C coupling failure being tempered 416 SS. Fire Pumps P-9A, P-9B and P-41 were found to have 416 SS line shafts with carbon steel couplings. Of the remaining pumps, Service Water Pump P-7A and Fire System Jockey Pump P-13 were discovered to also have 416 SS couplings; however, the line shaft couplings on P-13 are ASTM A582 416 Stainless Steel Condition A (Annealed) per Vendor Drawing M-33 Sh 22 and were supplied by the pump manufacturer (Johnston/Sulzer). SWPs P-7A/B/C use ASTM A582 416 SS Condition T (Tempered) based on the hardness range (28 to 32 Rc) given in HydroAire drawing 1047237 Rev 0. Per ASTM A582, Annealed 416 SS is softer (Hardness of 26 Rc max) than the Tempered 416 SS used for P-7A/B/C. Because of this, the couplings in P-13 are less susceptible to the same type of brittle fracture which occurred in P-7C. Due to the fact that P-13s couplings are constructed of 416 SS, which was annealed and not tempered, P-13 is not susceptible to the over-hardness due to heat treatment failure mode. For Service Water Pump, P-7A all the hardness data for the installed line shaft couplings provided by HydroAire was determined to be within the acceptable range for operation.

Potential Extent of Condition vulnerability for SWS Pumps P-7A/B/C will be eliminated by installing new couplings made of 17-4PH SS (addresses CR-PLP-2011-03961, closed to this evaluation). The new material was selected with the assistance of metallurgists from Structural Integrity. As of this writing, P-7A is complete, P-7B is in process and P-7C is scheduled for 9/29/2011. Analysis of the removed couplings is on-going at LPI and will be published in a revision to this analysis when the data is available.

Generic Implications: Extent of Condition/Extent of Cause RCE Report CR-PLP-2011-03902 Page 29 of 136 Table 1: Palisades Vertical Centrifugal Pumps ID #

Pump Title Safety Class Coupling/Line Shaft Material Coupling Vendor/PO #

Rebuild Date WO #

Class Seismic Q Group P-2A CONDENSATE PUMP NSR N

N FLEXIBLE GEAR COUPLINGS P-2B CONDENSATE PUMP NSR N

N FLEXIBLE GEAR COUPLINGS P-4 SCREEN WASH PUMP NSR N

N P-5 WARM WATER RECIRC PUMP NSR N

N P-7A SERVICE WATER PUMP SR 1

C 17-4PH, 416 SS HydroAire/10190242 4/5/2009 51637416 P-7B SERVICE WATER PUMP SR 1

C 416 Tempered SS, 416 SS HydroAire/10246213 6/2010 20082 P-7C SERVICE WATER PUMP SR 1

C 416 Tempered SS, 416 SS HydroAire/10253715 6/12/2009

/10/2/2009 190235 /

00208591 P-9A MOTOR DRIVEN FIRE PUMP QP N

N 1045 CS (Cat ID 0003716560),

1018 CS (5935-M32-DSL)

P-9B DIESEL DRIVEN FIRE PUMP QP 2

Y 1045 CS, 416 SS (EAR-2001-0457) 12/11/2001 P-10A HEATER DRAIN PUMP NSR N

N P-10B HEATER DRAIN PUMP NSR N

N P-13 FIRE SYSTEM JOCKEY PUMP QP 2

N 416 Annealed SS, 416 SS (VEN-M33-SH22)

Sulzer Pumps Inc.

2/4/2008 51633726 P-39A COOLING TOWER PUMP NSR N

N P-39B COOLING TOWER PUMP NSR N

N P-40A DILUTION WATER PUMP NSR N

N P-40B DILUTION WATER PUMP NSR N

N P-41 DIESEL DRIVEN COOLING TOWER FIRE WATER PP QP 2

Y 1045 CS (Cat ID 0003716560),

410 SS (5935-M-317-SK-1)

(- - - - -) : For items that are out of the scope of this review due to coupling material/ design or have no safety related function or have no maintenance rule function NSR: Not Safety Related QP: Augment Quality Program SR: Safety Related C: ASME Section III Class 3 This table is a snapshot of the pump status as of 8/9/2011

Generic Implications: Extent of Condition/Extent of Cause RCE Report CR-PLP-2011-03902 Page 30 of 136 Extent of Problem/Condition Summary The evaluation for this extent of condition concluded that the Service Water Pump P-7B and the Fire System Jockey Pump P-13 were discovered to also have 416 SS line shaft couplings and could be susceptible to the same condition as P-7C. The material used for the line shaft couplings on P-13 are ASTM A582 416 Stainless Steel Condition A (Annealed), which is softer than the Tempered 416 SS used in P-7C and makes it less susceptible to brittle fracture.

Extent of Cause The intent for this extent of cause review is to determine if the same causes that proved consequential in this instance currently exists in other plant equipment, processes or human performance. Additionally, the extent of cause determines if the identified causes also may have affected the performance of other individuals or work groups, the quality of other programs or processes, and/or the reliability of other types of equipment.

RC1 Extent (IGSCC induced failure):

IGSCC failures due to improper tempering have only been found in SWS P-7C in 2009 and 2011. In this case, the susceptibility to a common cause is limited to the Service Water Pumps. After the August 9th, 2011 failure, Palisades began pro-active replacement of all Service Water Pump couplings with couplings made from a different material. This was a conservative decision to eliminate/greatly reduce susceptibility to IGSCC.

RC2 Extent (Wrong Material Specified):

The entry point for the error which resulted in the failure of the P-7C Line Shaft couplings was introduced in 2007 during the completion of the Engineering Change that specified the 416 SS couplings. This occurred before the acquisition of Palisades by Entergy. As discussed in the O&P Investigation portion of the Event Narrative, the NMC processes in use when the EC was completed facilitated the entry of the error condition. The current implementation of EN-DC-115 is more prescriptive, thus making it less likely that the same type of error would pass through the EC process undetected. To verify this, corrective action 7 was written to review a sample size, per EN-QV-109, of modifications since 2007 where materials changes were introduced.

CC1 Extent (Increased IGSCC Susceptibility):

Tempering embrittlement can apply to any 416 SS hardened material if it is improperly heat treated. No evidence of other cases of incorrect heat treating was found outside of P-7C failures in 2009 and 2011. The change to 17-4PH SS will mitigate this risk.

CC2 Extent (Insufficient Use of Metallurgical Expertise):

As with the RC2 Extent of Cause, this condition would be mitigated under current Entergy Procedures through the effective EN-DC-115. Informal polling of the Design Engineering Staff showed a high degree of confidence of the process steps in EN-

Generic Implications: Extent of Condition/Extent of Cause RCE Report CR-PLP-2011-03902 Page 31 of 136 DC-115 would lead to identification of incorrect material selection prior to implementation. No previous occurrence of this cause has been noted at Palisades prior to the P-7C events.

CC3 Extent (Ineffective Use of OE)

This evaluation determined that the available OE was not accurately evaluated in 2007 and that opportunities to evaluate the OE again in 2009 were not effective. Ineffective use of OE represents a failure of one of the most cost effective barriers for preventing failures. By analyzing failures at other sites, Palisades gains the ability to take action before an actual failure here. In this case, OE from the 2003/2004 Perry Station repeat failure should have been a key area for investigation when specifying 416 SS for use in the line shaft couplings. CA #8 was written to take a sampling of RCE's and HT ACEs since 1/1/2008 and analyze the robustness of the OE Evaluation. CA #7 directs the evaluation of the OE in a sample of material change EC's since 2008.

Generic Implications Summary In Summary, the extents of causes for this evaluation were evaluated against other systems and programs. The identified Human Performance issues in this evaluation occurred in 2007, prior to the implementation of the Entergy Fleet Human Performance Procedures. Had these procedures (or equivalent) been in use at the time of the 2007 engineering change, there is high confidence that the error would not have occurred due to increased emphasis on Human Performance.

Previous Occurrence Evaluation RCE Report CR-PLP-2011-03902 Page 32 of 136 Summary of Operating Experience using Grade 416 Stainless Steel Operating experience that discussed temper embrittlement, material hardness, stress corrosion cracking and coupling design features were found to be applicable. The Palisades coupling material being 416 SS is prone to these attributes and requires consideration in light of this recent coupling failure. Operating Experience examined included the Perry Repeat failures in 2003 and 2004 as well as the recent 2010 Prairie Island Event. The Prairie Island OE was used by Palisades as part of the decision to change coupling materials from 416 SS to 17-4ph SS.

The OE highlights the need to ensure that proper material specifications and processes are applied for controlling hardness, toughness and other material properties that make 416 stainless steel less prone to temper embrittlement and corrosion cracking failures.

The OE also makes it clear that Licensees stipulate proper quality controls that assure coupons and testing results reflect actual material conditions.

Also, Design Engineering activities need to verify the coupling design to assure that stress risers are minimal and would not contribute to corrosion cracking.

Due to the change from 416 SS to 17-4PH SS, the above recommendations are historical in nature and are meant to highlight the actions that should be taken when specifying the material.

The detailed OE search can be found in Attachment IV Operating Experience Summary for using Grade 630, 17-4 PH Stainless Steel The OE identified the need to evaluate coupling environment and material conditioning to assure 17-4 stainless steel coupling are not prone to hydrogen embrittlement, thermal embrittlement, sulfide stress corrosion cracking, crevice corrosion and Microbiological influenced corrosion.

Also, there is a need to assure stringent process controls are applied to coupling manufacture to avoid contaminates during the heat treat process, such as nitrogen pick-up, that would affect the conditioning of 17-4 PH material. Additionally, the responsible design activity must ensure proper material specifications and processes are applied for controlling hardness, toughness, and other material properties that make 17-4 less prone to embrittlement and corrosion cracking failures. Equally important is the need to stipulate proper quality controls that assure coupons and testing results reflect actual material conditions.

Because 17-4PH SS arrives at the component machining site, the above recommendations are handled at by the facility supplying the bar stock to the machining shop. There are no specific steps required by the utility using components made from the 17-4PH bar stock. The machine shop purchasing the stock would be able to audit the Certified Material Test Report accompanying the stock to ensure adequate preparation and hardening.

The detailed OE search can be found in Attachment IV

Safety Significance Evaluation RCE Report CR-PLP-2011-03902 Page 33 of 136 This section of this report evaluates the impact on safety of the Service Water Pump P-7C failing to provide discharge pressure to the Service Water Header: Industrial, Environmental, Radiological, and Nuclear Safety were evaluated and summarized below.

General Safety of the Public (Environmental/Nuclear Safety):

From event initiation on 8/9/11,1202 hrs to the exiting of LCO 3.7.8 A1 on 8/12/09, 0309 hrs at no time during that time frame was the health and safety of the public jeopardized. On-shift licensed personnel verified that at least 100% of the post accident SWS cooling capacity available. To clarify, at least from the previous sentence, One hundred percent of the required SWS post accident cooling capability can be provided by any two SWS pumps if SWS flow either to the non-critical header (CV-1359) or to the critical loads inside the containment are capable of being isolated (CV-0824 or CV-0847). The SWS is divided in two separate and redundant trains, each associated with a Safeguards Electrical Train. The SWS train associated with the Left Safeguards Train consists of one Service Water Pump (P-7B), associated piping, valves, and controls for the equipment to perform their safety function. The SWS train associated with the Right Safeguards Train consists of two Service Water Pumps (P-7A and P-7C), associated piping, valves, and controls for the equipment to perform their safety function.

All Right Train components, except P-7C, remained operable. Compensatory actions to place protected equipment signs around the Non-Critical Service Water Isolation Valve, CV-1359, Service Water From Containment, CV-0824, and Service Water to Containment, CV-0847, were enacted to maintain isolation capabilities to ensure 100% of post accident SWS cooling capability with only two service water pumps being operable. The Risk Achievement Worth (RAW) score was 1.03 green prior to the loss of P-7C and the RAW score was determined and logged as 1.04 green after the P-7C coupling failure and subsequent removal from service. In comparison, if Service Water Pump, P-7B, is rendered inoperable, the RAW score would change to 3.77 yellow. Thus the necessary SWS equipment to safely shutdown the plant was intact and available to maintain Nuclear and Public safety.

Industrial Safety:

To repair P-7C, Service Water Pump within the allowed 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> LCO time frame a well thought out and planned evolution had to occur. The required the use of all the Human Performance Tools available to the site from Pre-job Briefings to the individual workers using STAR (Stop, Think, Act, & Review) and QV&V (Question, Verify, & Validate). A significant amount of Supervisory oversight was provided to Maintenance workers to provide the extra sets of eyes and ears on the scene for safety and repair accuracy. There are industrial safety challenges associated with removal of a Service Water Pump and OE was utilized. Operations Work Control Center personnel walked down the area with Mechanical Maintenance Supervision who had experience from the previous P-7C shaft coupling failure in October 2009.

The Screen House presents its own unique industrial safety challenges because of the noise level and water that is present on the floor along with FME concerns because of the hole that is created in the floor when the pump is removed. Adequate Industrial Safety actions were documented in the Work Order #286627, Mechanical Maintenance work instructions, WI-SWS-M-04, and Electrical Maintenance procedure SWS-E-5.

Safety Significance Evaluation RCE Report CR-PLP-2011-03902 Page 34 of 136 Radiological Safety:

There was no impact on radiological safety. P-7C, Service Water Pump, is located in the Screen House 590 elevation of the Turbine Building. The Service Water System is constantly monitored by RIA-0833, Service Water Discharge Monitor. RIA-0833 has a low flow alarm and radiation level readout in the Control Room for monitoring by the On-shift licensed operators.

Based on the above documented Safety Significance Evaluation, no further corrective actions are necessary.

Corrective Action Plan RCE Report CR-PLP-2011-03902 Page 35 of 136 Identified Cause Corrective Actions Responsible Dept.

Due Date Immediate Actions RC1, CC1 Processed EC31337 to change coupling material from 416SS to 17-4PH SS in order to minimize susceptibility to SCC Design Engineering (completed MM/DD/YY)

RC1CC1 Preemptively changed Line Shaft Couplings in P-7A/B/C to 17-4PH SS material.

P-7A and P-7B are done, P-7C is scheduled for 9/29/11.

Maintenance WW 1139 Interim Actions RC2, EOC,CC2, CC3, CR-PLP-2011-03902 CA-00014 Conduct an Information Sharing with Engineering Department covering the EN-DC-141 "Design Inputs" Attachment 9.3, focusing on identifying the need for robust reviews and technical analysis if component materials are being changed.

Ensure that there is awareness amongst the Engineering Staff that a new Engineering Standard will formalize material change requirements including the level of necessary reviews.

Engineering 12/01/2011 RC1, RC2, CC1, CC2, CC3 CR-PLP-2011-03902 CA-00015 Evaluate the need for additional training on the subject of Engineering Department and MP&C interfaces. Include the Critical Procurement Process (EN-MP-100) and Entergy requirements for the transmittal of Design Changes to suppliers. Create a TEAR to perform this evaluation. Circle back for CARB Chair approval if the evaluation determines that training is not required.

Training 12/01/2011 CC3 CR-PLP-2011-03902 CA-00016 Conduct an Information Sharing with Engineering covering the effective use of Operating Experience during the performance of Engineering Changes. Ensure that there is awareness amongst the Engineering Staff that a new Engineering Standard will formalize OE requirements including the depth of analysis.

Engineering 12/01/2011

Corrective Action Plan RCE Report CR-PLP-2011-03902 Page 36 of 136 EOC CR-PLP-2011-03902 CA-00017 Directed action from CARB and the Director of Engineering. Conduct an extent of condition analysis for all components supplied by HydroAire. Identify all tempered components and evaluate susceptibility to SCC in the components service environment (Dilution Water Pumps, Service Water Pumps, Heater Drain Pumps, Condensate Pumps, etc.)

Engineering 12/01/2011 Other CR-PLP-2011-03902 CA-00018 Update the Root Cause Evaluation after receipt of the LPI Metallurgical Analysis Report. Get CARB approval of final results regardless of the scope of changes.

Sys Engineering 10/28/2011 EOC CR-PLP-2011-03902 CA-00019 Review a sample size, per EN-QV-109, of modifications since 8/30/2007 where materials changes were introduced. Evaluate the material selected and the robustness of the OE analysis. Create a new action and return to CARB for review if the analysis shows a significant deficiency with regards to material changes.

Design Engineering 12/01/2011 EOC CR-PLP-2011-03902 CA-00020 Review of sample size, per EN-QV-109, of RCE and HT ACE OE evaluations at Palisades since 8/30/2007. Evaluate the robustness of the OE Analysis. Create a new action and return to CARB for review if the analysis shows a significant deficiency with regards to OE analysis.

Engineering 12/01/2011 Other CR-PLP-2011-03902 CA-00021 Examine previously determined stock levels of Service Water Pump Shafts, Couplings and Spiders. Create and gain approval of an action plan to ensure sufficient on-hand stock of materials, such that the station can recover from a component failure in a timely manner. Return to CARB if approval for the action plan is not obtained.

MP&C 12/01/2011

Corrective Action Plan RCE Report CR-PLP-2011-03902 Page 37 of 136 CC3 CR-PLP-2011-03902 CA-00022 Evaluate revising the ESP Training plan to include initial and continuing training on effective OE analysis and documentation. Create a TEAR as necessary. Circle back for CARB approval if the evaluation determines that training is not required.

Training 12/01/2011 Other CR-PLP-2011-03902 CA-00023 Petition the fleet owner of EN-DC-115 to include reference to the PLP Engineering Standard being developed as a CAPR. If the fleet owner will not reference EN-DC-115, document the basis for the decision in this CA.

Design Engineering 01/27/2012 Short & Long Term Actions RC1, RC2, CC1, CC2, CC3 CAPR #1 (Corrective Action to Preclude Repetition)

CR-PLP-2011-03902 CA-00024 Create a limited distribution Engineering Standard for PLP that clearly identifies station requirements and expectations for Material Changes affecting installed plant equipment. Consideration should be given to specifying required analysis, guidance on obtaining outside assistance, specification of deliverables, analysis of the service environment and documenting analysis of relevant OE.

Guidance on OE analysis should be designated as being applicable to all Engineering tasks/reports that require OE searches and evaluation.

Design Engineering 12/01/2011 RC1, CC1 CAPR #2 (Corrective Action to Preclude Repetition)

CR-PLP-2011-03902 CA-00025 Replace 416 SS Service Water Line Shaft Couplings with 17-4PH couplings per EC #31337. Disposition work done in this CA. This action can be closed after all Service Water Pump Line Shaft Couplings have been changed.

Maintenance 12/01/2011

Effectiveness Review Plan RCE Report CR-PLP-2011-03902 Page 38 of 136 LO-PLPLO-2011-00055 CAPR #1 Create a limited distribution Engineering Standard for PLP that clearly identifies station requirements and expectations for Material Changes affecting installed plant equipment. Consideration should be given to specifying required analysis, guidance on obtaining outside assistance, specification of deliverables, analysis of the service environment and documenting analysis of relevant OE. Guidance on OE analysis should be designated as being applicable to all Engineering tasks/reports that require OE searches and evaluation.

Action Resp. Dept Due Date Method:

Analysis of EC packages Design Eng Attributes:

416 SS (tempered)

Design Eng Success:

None Specified in Engineering Changes where the service environment is corrosive to 400 series stainless steels.

Design Eng Timeliness:

1 year Design Eng 09/09/2012 LO-PLPLO-2011-00055 CAPR #2 Replace 416 SS Service Water Line Shaft Couplings with 17-4PH couplings per EC #31337. Disposition work done in this CA. This action can be closed after all Service Water Pump Line Shaft Couplings have been changed.

Action Resp. Dept Due Date Method:

Analysis of WO Packages Maintenance Attributes:

Completion notes and Work Package sign-offs Maintenance Success:

Work Order Packages demonstrate the couplings were successfully changed.

Maintenance Timeliness:

4 months Maintenance 01/30/2012

References RCE Report CR-PLP-2011-03902 Page 39 of 136 Documents reviewed:

CR-PLP-2009-04519 CR-PLP-2009-04806 F11358-R-001 DRAFT G.docx "METALLURGICAL AND FAILURE ANALYSIS OF SWS PUMP P-7C COUPLING #6" Personnel contacted:

Ed Huss / Entergy MP&C John Kasishke / Entergy MP&C Tom Reddy / Entergy MP&C John Petro / Entergy Receipt inspection Jim Alderink / Retired Entergy Design Eng.

Team Members:

Team Leader / Evaluator Paul M. Deniston / System Eng.

Jason Gosler / Maintenance Team Members Aaron Verzwyvelt / System Eng.

Jim Forehand / System Eng Kevin Rose / Maintenance Rich Margol / Training Bill Townes / Operations Mike McCarthy / Design Eng.

Ben Gumieny / Procurement Eng.

Sontra Yim / LPI Ian Wilson / Manncini and Assoc George Licina / Structural Integrity Dana Cooley / SeaState Group Inc.

Analysis Methodologies Used:

1. Event & Causal Factor Charting

2. Failure Mode Analysis

3. Metallurgical Analysis Attachments:

1.

Event & Causal Factor Chart 2.

Failure Mode Analysis 3.

Safety Culture Evaluation 4.

Detailed Operating Experience 5.

LPI Report

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 40 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 41 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 42 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 43 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 44 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 45 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 46 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 47 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 48 of 136

Attachment I - Event & Casual Factor Chart RCE Report CR-PLP-2011-03902 Page 49 of 136

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 50 of 136 Note: This FMA is a living document. Some items are awaiting analysis that is incoming from LPI.

Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Coupling Design or Manufacture

1) Improper coupling hardness Hardness measurements of the failed coupling indicate an average hardness value of 33.3 Rockwell C, which exceeds the specified tolerance of 32 Rockwell C Yim (LPI)

Hardness on the ends and hardness on circumferential direction, through thickness on axial slice Hardness testing on seven other coupling for the same pump Compositional analysis of threading Retest failed coupling from 2009 failure event High Based on the surface hardness results in Table Report No.

F11358-R-001 Page 21 of 51 Revision DRAFT G

3-5, five couplings (11-P7C-4, 11-P7C-6F, 11-P7C-7, 11-P7C-8, and 09-P7C-7F) exhibit surface hardness above specification, and one coupling (11-P7C-3) exhibits surface hardness at the upper limit of the specification.

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 51 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results A. Verzwyvelt Examine Bodycote NDT results for couplings Low In October of 2009, Bodycote and Hydroaire began Double Tempering Process for 416 SS couplings that are currently installed in P-7C, and P-7B. Heat treatment details are not wholly consistent between certified batches, however hardness testing indicated all parts were sat.

Coupling Design or Manufacture 1a) Improper coupling hardness caused by deviation of heat treating from specified procedure If rework occurred during the manufacutring process, the expected life could have been reduced.

LPI Review Hydroaire/Bodycote procedures for deviations from procedure Medium rfi-62

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 52 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Coupling Design or Manufacture 1b) Improper coupling hardness caused by temperature gradient within furnace during manufacturing process Yim (LPI)

Look for inconsistent hardness, tensile, and charpy results 0

Coupling #7 had varying hardness across the part, where one end was out of spec, while the other was in spec.

Coupling #3 installed in 2011 had min max delta of 5.7. LPI report does not address the typical expected hardness ranges across the part.

Coupling Design or Manufacture 1c) Improper hardness caused by inadequate cooling process 0

See 1 High see Row 3 Coupling Design or Manufacture 1d) Inadequate hardness specification low impact toughness values (indicated by the CVN) are seen in the Yim (LPI)

Independent testing of fracture toughness charpy v notch, tensile testing on 2009, 2011 failed components, and other couplings High low impact toughness values (indicated by the CVN) are seen in the couplings that have failed with most CVN

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 53 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results couplings that have failed with most CVN values in the single digits at the temperature range of the service water (refer to Table 3-7).

values in the single digits at the temperature range of the service water (refer to Table 3-7).

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 54 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Coupling Design or Manufacture 1e) Crack propagation due to sulfide inclusion and secondary heat treament Based on tempering curves for the batch of couplings installed in P-7A, P-7B and P-7C (Figure 4-2), the tempering temperatures are in the range to be avoided between 400°C and 580°C (752°F to 1076°F) for 416SS. These tempering temperatures can lead to low toughness and susceptibility of the material to Yim (LPI)

Microspecimen examination of failed 2011 coupling, unfailed couplings, re-review of 2009 metalurgical report Low SEM verifies IGSCC of material

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 55 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results SCC.

Coupling Design or Manufacture 1f) Stress corrosion cracking initiated by neolube contamination Neolube is comprised of graphite in alcohol suspension, neither of which can cause SCC in 416 SS Yim (LPI)

Perform chemical analysis of failed surfaces looking for effects of neolube High Neolube is comprised of graphite in alcohol suspension, neither of which can cause SCC in 416 SS

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 56 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Coupling Design or Manufacture

2) Improper coupling material Commonly Used for Pump Shafts and Couplings Industry OE of IGSCC of 416 SS, Perry Root Cause from 2004, Perry changed to 17-4PH SS.

Are there better material options available?

IPEC when to Nitronics 50 Yim (LPI)

Based on Material analysis performed by LPI test procedure.

Compositional analysis, chemical analysis, and EDX.,

provide recommendation as to material for coupling Low Coupling material is quench hardened and tempered 416 SS.

Recommendation is for Nitronic 60.

Geerlings Interviews with Maintenance to verify that no evidence of damage to impellers exists Medium Boroscope inspection of P-9C Bowl Assembly Results: Nothing was observed indicating foreqin material.

Stress

3) Overtorque from foreign material Nothing was observed indicating foreign material causing overtorque Material that could have caused OT condition, may have fallen or otherwise missed during inspection Yim (LPI)

Evaluated failed coupling for single overload condition Low Tensile testing indicates failure mode not of "single overload" type

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 57 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Yim (LPI)

Visual Inspection of and TIR on three shafts removed Medium PENDING LPI REPORT Program Engineering Review Vibration Data, and effects of Seimic Supports Low Available vibration data at the motor does not provide indication of shaft imbalance, sensativity of data to shaft imbalance is poor based on location of data sample Stress

4) Misalignment of shaft Uneven wear is typically experienced in the Service Water Pumps, and may not be abnormal Evidence of uneven wear on shaft journals Shafts

1. 4, #5, #6.

System Engineering, DeBusscher review WI for packing adjustment Low P-7C has been repacked 3 times since 2009 failure.

No issues noted Stress 4a) Improper packing adjustment/installation result in poor shaft aligment causing failure of coupling Bronze bearing under packing box maintains shaft alignment through packing box Wear patterns observed on shafts #4, # 5,

1. 6 Yim (LPI)

Realign shafts and document wear patterns LCO PENDING LPI REPORT

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 58 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Electircal Maintenance Verify pump movement while out of service, when restraints are loosened Medium PENDING LPI REPORT Stress 4b) Improper alignment caused by upper or lower seismic restraint.

Lower seismic restraint is mounted near failed coupling location Maintenance Interview divers and maintenance High No relative movement observed during installation, no movement of unit required to install seismic restraints Environmental Effects

5) Stress Corrosion Cracking Initial reports from LPI indicate that SCC is present and likely cause Yim (LPI)

Perform Scanning Electron Microscopy (SEM) analysis N/A SEM verifies IGSCC of material Environmental Effects 5a) Changes in Water Chemistry Same chemical has been used for 10 years, if problematic would not be isolated to P-7C. Annual usage present for no more than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> per year Clamtrol does possess chlorides N/A No actions due input from chemistry indicating that this is not felt to be a contributor High N/A

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 59 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Environmental Effects 5b) Changes in Water Chemistry Sodium Hypochlorite possesses chlorine.

Present at Service Water Pumps 40 minutes/ day.

Chemistry Characterize chlorine concentration experienced by service water pumps, prior and post 1R21 High Chlorination increases chlorides by approximately 0.3 ppm, during the 40 minute period daily Chemistry Characterize chloride concentrations in Lake Michigan water, annually, seasonally, and over last few years.

Temperature, PH, High 9.72 ppm chlorine noted Environmental Effects 5b) Changes in Water Chemistry Lake Water possesses Chlorides, and is present at all times Chemistry Send sample to LPI of service water, with Temp, PH, and TDO of sample High Environmental Effects Microbiological induced corrosion No discussion of MIC in lab report Yim (LPI)

Perform Chemical analysis of coupling deposits Low EDS analysis of fracture surface revealed the presence of corrosive agents (chlorides, oxides and

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 60 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results sulfides),

consistent with stress corrosion cracking.

Coupling Design or Manufacture

6) Incorrect clearances in coupling to shaft (threading same?)

Hydroaire states that they have a gage and check these dimensions, P-7C,B,A all have same coupling design. Typical experience with couplings indicates no threading issues during installation Hydroaire Go/NoGo gage failed NoGo check Hydroaire Evalaute coupling and shaft thread clearances and verfiy that threads are in tolerance, for parts in question 0

PENDING LPI REPORT 6a) 0 Low Coupling Design or Manufacture 6b) Threads not concentric with coupling Yim (LPI)

Evaluate existing condition of threads within coupling High PENDING LPI REPORT 7a)

RCE Team Provide photos of shaft ends near failed coupling, or shafts if available, looking for evidence of Galling of 2009 failure High PENDING LPI REPORT

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 61 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Stress

7) Shaft faces not square (perfectly flat) and butted 1-4 shafts from failed pump are square Evidence of galling between shafts 5 and 6 Yim (LPI)

Derterimne cause of galling of shaft end, and squareness of shaft ends 5 and 6 LCO Galling on Shaft end determined to occur after failure of coupling, post failure damage Mechanical Maintenance Repack pump, and check position of stuffing box relative to shaft LCO Mechanical Maintenance Sweep stuffing box bore, TIR etc.

Medium As found shaft to stuffing box, 3 -4 places, 0.002" FIN WR - 246107, compare levelness of head mounting flange to sole plate 0

PENDING LPI REPORT 0

Check fit, for the following LCO Mechanical Maintenance

1. Radial clearance fit between column and head LCO Chamfered fit such that no clearance exists during installation.

Stress

8) Pump alignment Packing alignment issues may not be relevant to the presence of packing box bearing None Mechanical Maintenance

2. Concentricity of stuffing box bearing to upper head register Medium

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 62 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Engineering When was the head last machined (prior to 2009 failure???)

Medium Head not machined during 2009 rebuild of P-7C Engineering

1. How many times has this pump been repacked, because eccentric shaft position relative to the packing is indicative of head to column assembly Priority 3 Times since 2009 coupling Failure 0

0 0

0 0

0 Stress 8b) Nonconcentric Spider Mechanical Maintenance Disassemble pump, looking for non-concentric spiders LCO During disassembly, number columns and spiders/ After disassembly in the shop, measure radial fit between spider and associated column. P-7A had

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 63 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results slop ranging up to 0.004" unlikely to cause misalignment issue due to 0.004

- 0.009 tolerance between shaft journal and rubber sleeve bearing Stress 8c) Alignment of upper column to pump head and stuffing box Mechanical Maintenance Repack pump, and check position of stuffing box relative to shaft LCO See reference 40 0

0 System Engineering Evaluate packing for continued use in Service Water Pump Low A change in the type of packing used in P-7C pump, would not affect the failure mode of the coupling Stress

11) Shaft Wobble allowed by shaft wear at the packing Bronze bearing mounted directly below packing box should restrain shaft whip.

Initial failure modes analysis gives evidence against fatigue failure, typically expected to be caused by cyclic stresses from shaft alignment issues Evidence of uneven wear on shaft journals Shafts

1. 4 and #5 and

1. 6 System Engineering Evaluate need for hardened sleeve Low The addition of Hardened Sleeve to the P-7C Service Water Pump would not

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 64 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results change the failure mode of the coupling event Program Engineering/Maintenance Install Prox Probes to measure and record shaft movement/vibrations Medium PENDING LPI REPORT 0

0 0

0 0

0 Stress

12) Reverse rotation during pump start QO-14 documents check valve closed, by observing no rotation.

P-7A has similar distrubution but no failures Lack of Anti-Rotation Feature on Pump, 1 Rotation Opposite nominal direction noted during test bump (deadheaded)

WI-SWS-M-04 6.14 "Motor Bump and Pump Lift Reset" Cycle System Engineering Characterize reverse rotation behavior of P-7C when shut off HIgh QO-14 states to check for reverse pump rotation, prior to start, initate DRN to add similar statement to SOP-15, potential exists for rapid pump start after stop will cause excessive stresses to shafts and couplings due to reverse rotation driven by gravity, and starting during

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 65 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results data from 2009-2011 shows 27 start/stop intervals of less than 1.44 minutes in duration the reverse rotation Stress 12a) Reverse rotation from check valve leak by No walkdown evidence of rotation in recent memory System Engineering Characterize reverse rotation behavior of P-7C when shut off High No evidence of check valve leak by or persistant reverse rotation Coupling Design or Manufacture

13) Defect(s) in coupling from factory (stress risers - vent hole thru threads, machining of threads, edges, etc))

Cracks do not appear to go through the vent hole.

Cracks are located within 2-3 threads of the venthole vertical location, and appear to grow outward from thre thread roots Yim (LPI)

Perform ultrasonic testing and destructive testing of parts looking for defects High UT testing unable to be performed due to part design.

Inspection of coupling failure modes and FEA indicates highest stresses are within a band 1-2 threads of

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 66 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results venthole, but not through vent hole Coupling Design or Manufacture 13a) Defects caused by machining including the vent hole.

Cracks do not appear to go through the vent hole.

Yim (LPI)

Examine interior of drilled hole, looking for sources of FME Medium PENDING LPI REPORT 13b) 0 0

13c) 0 0

Stress

14) Contact of shaft with dry rubber on startup (coupler stressor, load applied to shaft by dry bearing)

Expected Wear of Rubber bearing in Spider, noted only one coupling deviating from nominal dimension and only by 0.002", and still in spec. In addtion, tenisile testing indicates that this failure was not a single overload type Observed coating of nitrile on shaft 4 journal N/A No Actions, due to lack of wear noted on rubber sleeve bearings 0

Tensile testing indicates failure mode not of "single overload" type

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 67 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Stress

15) Operation at critical speed Operating at Design Speed, longevity of of other pumps indicates not running at critical speed, Vibration testing indicates NO large 1X frequency response N/A No Actions, due to Vibration data indicating not operating at critical speed 0

Vibration data does not support this conclusion Engineering Compare reverse rotation time after pump shut off to typical shortest pump restart times Medium Stress

16) Restart of pump shortly after shutoff (e.g RT-8C, P-7C start 26 s after DG bkr closes = OK) applied extra load to pump shaft.

Timing for the event described would be Pump Running, Stopped, then Running over 26-32 seconds.

Pump reverse rotation observed for 2-5 seconds Reverse Rotation Observed 1 rotation over a couple a

seconds, 8/11/11 Yim (LPI)

Nondestructive -

Ultrasonic Test High UT testing unable to be performed due to part design.

Inspection of coupling failure modes and FEA indicates highest stresses are within a band 1-2 threads of venthole, but not through vent hole

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 68 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Yim (LPI)

Destructive - look for micro cracks High MT testing of Coupling #7 and

1. 5 indicates cracking present in Coupling 7 due to IGSCC Stress

17) Dead head start following rebuild and quarterly surveillance test caused coupling damage or bending of shaft Typical pump response during deadheaded start is to "jump". Shaft coupling failures have not occurred immediately after a start Yim (LPI)

Stress evaluation of coupling with consideration to shock load Medium Lab data indicates failure is not of "single overload event" type, does not support this failure cause Stress

18) Improper shaft coupling engagement A before and after measurement in the work order package refutes this Yim (LPI)

Document wear patterns on journals Medium Documentation does not support improper coupling engagment as a cause

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 69 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Stress

19) Improper pump lift setpoint Does not relate to coupling failure, unless Mech Mainteance is stretching the column significantly during lift setting, however, total float is verified and matches design, in addtion, adjustment of float is done by hand, making it very unlikely that significant addtional stresses are created during this process In work instruction there is a choice of two different settings, this differs from drawings?

System Engineering Determine basis for the 1/2" and 3/8" pump lift settings in Work Instruction, Possible Source Jim Alderink Medium CR-PLP-2011-03967 & RFI 18 associated with P-7C 8/2011 Coupling Failure

- Work methods do not support this as a cause for the observed failure Stress

20) Increase load by backpressure from strainer blockage P-7A/C has had large organic blockage occur with no recent coupling failure associated System Engineering, Debuscher Review ESOMS and characterize strainer DP over time Low History shows no association between Strainer backpressure and coupling failure

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 70 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Stress

21) Flow charactoristic changes due to sand build up in intake structure Over 18 months, the sand collection within the service water bay ranges from 0-4",

where the low point is near the service water pumps. Sand collection rate is linear over time, with the relative collection of sand in the service water bay significanly reduced compared to the intake structure System Engineering Review vibration history over time Document history of sand review from service water bay.

Listen for voiding using a stethoscope on discharge pipe Low Sand collection in service water bay is minimal, typically 0" depth in flow field of pumps, unlikely cause of coupling failure Stress

22) Pump assembly changes Similar procedure for P-7A, and P-7B which have not yet failed.

HydroAire Review design and dimensional changes due to 2009 rebuilt of pump, Review EC for 2009 rebuild High Stress

23) Component replacements within last 5 years P-7B, and P-7A have stainless steel couplings and no failures as of yet P-7C couplings were changed to stainless steel during 2009 rebuild, prior to that HydroAire Review design and dimensional changes due to 2009 rebuilt of pump High rfi-12

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 71 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results were Carbon Steel, no failures associated with couples observed Coupling Design or Manufacture

24) Fatigue failure Yim (LPI)

Evaluate Failed Coupling for Fatigue Failure Medium SEM examination revealed the fracture surface morphology to exhibit a rock-candy appearance, characteristic of intergranular stress corrosion cracking (IGSCC) as shown in Figure 3-10. This is typical for a quench and tempered steel, such as a 400 series martensitic steel.

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 72 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Coupling Design or Manufacture

25) Coupling design -

Entire coupling threaded vs. relief section in center Design Engineering Recommend and prepare Design Change package for Coupling Design Changes High rock-candy appearance, characteristic of intergranular stress corrosion Environmental Effects

26) Corrosion caused by environmental conditions

Perry, Palsades failure all above non-operating water levels within the pump Yim (LPI)

Perform chemical analysis of failed surfaces looking for environmental causes High SEM examination revealed the fracture surface morphology to exhibit a rock-candy appearance, characteristic of intergranular stress corrosion cracking (IGSCC) as shown in Figure 3-10. This is typical for a quench and tempered steel, such as a 400 series martensitic steel.

Attachment II - Failure Mode Analysis RCE Report CR-PLP-2011-03902 Page 73 of 136 Aspect Failure Mode (yellow lines cannot be completed within RCE team Refute Support Assigned to Action Items Priority Results Stress

27) Maintenance -

Pump Assembly Practices P-7A - WI-SWS-M-03, P-7B/C Wi-SWS-M -04

---

Coupling Assembly/Disassembly the same Some Fleet installation practices differ from Palisades Practices Maintenance Fleet Call with other plants that have raw water vertical turbine pumps with treaded couplings.

Send out work instruction for maintenance of pump, prior to High Palisades practices found meeting or exceeding fleet standards Environmental Effects

28) P-7C has different operation cycle from P-7B, P-7A, results Similar metals, chemical environment, pump design P-7C has failed 2 couplings, P-7B and P-7A have not System Engineering, Aaron Verzywvelt Compare run time, idle time, average operating interval periods for Service Water Pumps High P-7A and P-7C have similar operating cycle, P-7B has had different cycle due to saving the pump during early 2010 due to degraded performance

Attachment III - Safety Culture Evaluation RCE Report CR-PLP-2011-03902 Page 74 of 136 TA TABLE 1 - SAFETY CULTURE COMPARISON SAFETY CULTURE COMPONENT DESCRIPTION CR-PLP-2011-03902

1. Decision-Making Licensee decisions demonstrate that nuclear safety is an overriding priority:

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

2. Resources The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to assure nuclear safety.

RC1 - No indication RC2 - Yes CC1 - No indication CC2 - Yes, significant CC3 - No indication

3. Work Control The licensee plans and coordinates work activities, consistent with nuclear safety:

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

4. Work Practices Personnel work practices support human performance.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

5. Corrective Action Program The licensee ensures that issues potentially impacting nuclear safety are promptly identified, fully evaluated, and that actions are taken to address safety issues in a timely manner, commensurate with their significance.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - Yes, not significant

6. Operating experience The licensee uses operating experience (OE) information, including vendor recommendations and internally generated lessons learned, to support plant safety.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - Yes, significant

7. Self-and Independent Assessments The licensee conducts self-and independent assessments of their activities and practices, as appropriate, to assess performance and identify areas for improvement.

RC1 - No indication RC2 - No indication CC1 - No indication

Attachment III - Safety Culture Evaluation RCE Report CR-PLP-2011-03902 Page 75 of 136 SAFETY CULTURE COMPONENT DESCRIPTION CR-PLP-2011-03902 CC2 - No indication CC3 - No indication

8. Environment For Raising Concerns An environment exists in which employees feel free to raise concerns both to their management and/or the NRC without fear of retaliation and employees are encouraged to raise such concerns.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

9. Preventing, Detecting, and Mitigating Perceptions of Retaliation A policy for prohibiting harassment and retaliation for raising nuclear safety concerns exists and is consistently enforced.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

10. Accountability Management defines the line of authority and responsibility for nuclear safety.

RC1 - No indication RC2 - Yes CC1 - No indication CC2 - No indication CC3 - No indication

11. Continuous learning environment The licensee ensures that a learning environment exists.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

12. Organizational change management Management uses a systematic process for planning, coordinating, and evaluating the safety impacts of decisions related to major changes in organizational structures and functions, leadership, policies, programs, procedures, and resources. Management effectively communicates such changes to affected personnel.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

13. Safety policies Safety policies and related training establish and reinforce that nuclear safety is an overriding priority in that:

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - No indication

Attachment III - Safety Culture Evaluation RCE Report CR-PLP-2011-03902 Page 76 of 136 Notes 2

Palisades specifications for the coupling required use of 416 SS and did not require toughness testing nor adequately test for hardness (insufficient sample size). See Table 2. (RC2) 2 Palisades specifications for the coupling did not involve input/review by a qualified metallurgist. See Table 2. (CC2) 5 When the Prairie Island 2010 OE became available, Palisades did not initiate a condition report to question the suitability of couplings that were in stock. (CC3) 6 Palisades did not take full advantage of operating experience suggesting that 416 SS was susceptible to IGSCC. See Table 2 (CC3) 10 Palisades system engineer functioned as a design engineer when dealing with HydroAire. See Table 2. (RC2)

Attachment III - Safety Culture Evaluation RCE Report CR-PLP-2011-03902 Page 77 of 136 TABLE 2 - DETAILED SAFETY CULTURE COMPONENT REVIEW Description CR-PLP-2011-03902

1. Decision-Making Licensee decisions demonstrate that nuclear safety is an overriding priority. Specifically (as applicable):

2. Resources The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to assure nuclear safety. Specifically, those necessary for:

RES H.2(b)

Training of personnel and sufficient qualified personnel to maintain work hours within working hours guidelines.

Palisades specifications for the coupling did not involve input/review by a qualified metallurgist.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - Yes CC3 - No indication RES H.2(c)

Complete, accurate and up-to-date design documentation, procedures, and work packages, and correct labeling of components.

Palisades specifications for the coupling required use of 416 SS and did not require toughness testing nor adequately test for hardness (insufficient sample size).

RC1 - No indication RC2 - Yes CC1 - No indication CC2 - No indication CC3 - No indication

3. Work Control The licensee plans and coordinates work activities, consistent with nuclear safety. Specifically (as applicable):

4. Work Practices Personnel work practices support human performance. Specifically (as applicable):

5. Corrective Action Program The licensee ensures that issues potentially impacting nuclear safety are promptly identified, fully evaluated, and that actions are taken to address safety issues in a timely manner, commensurate with their significance. Specifically (as applicable):

6. Operating experience The licensee uses operating experience (OE) information, including vendor recommendations and internally generated lessons learned, to support plant safety. Specifically (as applicable):

OE P.2(a)

The licensee systematically collects, evaluates, and communicates to affected internal stakeholders in a timely manner relevant internal and external OE.

Palisades did not take full advantage of operating experience suggesting that 416 SS was susceptible to IGSCC.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - Yes OE P.2(b)

The licensee implements and institutionalizes OE through changes to station processes, procedures, equipment, and training programs.

Palisades did not take full advantage of operating experience suggesting that 416 SS was susceptible to IGSCC.

RC1 - No indication RC2 - No indication CC1 - No indication CC2 - No indication CC3 - Yes

Attachment III - Safety Culture Evaluation RCE Report CR-PLP-2011-03902 Page 78 of 136 Description CR-PLP-2011-03902

7. Self-and Independent Assessments The licensee conducts self-and independent assessments of their activities and practices, as appropriate, to assess performance and identify areas for improvement. Specifically (as applicable):

8. Environment For Raising Concerns An environment exists in which employees feel free to raise concerns both to their management and/or the NRC without fear of retaliation and employees are encouraged to raise such concerns. Specifically ( as applicable):

9. Preventing, Detecting, and Mitigating Perceptions of Retaliation A policy for prohibiting harassment and retaliation for raising nuclear safety concerns exists and is consistently enforced in that:

10. Accountability Management defines the line of authority and responsibility for nuclear safety. Specifically (as applicable):

ACC A.1(a)

(a) Accountability is maintained for important safety decisions in that the system of rewards and sanctions is aligned with nuclear safety policies and reinforces behaviors and outcomes which reflect safety as an overriding priority.

Palisades system engineer functioned as a design engineer when dealing with HydroAire. Management did not recognize the risk associated with this arrangement.

RC1 - No indication RC2 - Yes CC1 - No indication CC2 - No indication CC3 - No indication

11. Continuous learning environment The licensee ensures that a learning environment exists. Specifically (as applicable):

13. Safety policies Safety policies and related training establish and reinforce that nuclear safety is an overriding priority in that:

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 79 of 136 Stainless Steel - Grade 416 I

NTERNAL Operating Experience:

The Paperless Condition Reporting System (PCRS) was utilized to search for similar events involving coupling failures. The search was limited to Palisades events as similar searches were performed using Autonomy and documented in the External Operating Experience section, which included all of the Entergy stations.

The search criteria did not include any time constraints and specified keywords pump coupling. The search yielded twenty seven (27) Condition Reports (CRs). All 27 CRs were reviewed and concluded that none were similar to this coupling failure. An additional search, with no time constraints, was performed with the specified key word service water pump. The search yielded four hundred and ninety (490) Condition Reports. All 490 CRs were reviewed and concluded that none were similar to this coupling failure; however, a number of CRs were classified and evaluated under three broad areas:

x Foreign Material issues in service water bay (16 CRs) x Service Water Pump degraded monitoring parameters (19 CRs) x Disassembly/Assembly Service Water Pump parts degradation/discrepancies. (39 CRs)

Other Entergy sites operating experience was searched using the Autonomy system.

Various combinations of keywords involving service water pumps, couplings, embrittlement, and failures were used. No events were uniquely identified other than those Entergy sites identified in the External Operating Experience section.

EXTERNAL Operating Experience:

Note: Operating Experience denoted with

• contains information indicating an upper pump coupling was affected.

CR-PLP-2009-04519, Service Water Pump P-7C Failure to Provide Discharge Pressure identified operating experience.

The Institute of Nuclear Power Operations (INPO) Equipment Performance and Information Exchange System (EPIX) and Nuclear Plant Reliability Data System (NPRDS) searches were conducted using search terms shaft AND failure AND pump AND service water AND NOT bent AND coupling AND Layne Bowler" AND failure near coupling AND Service water. These searches yielded 11 OE articles that were analyzed for applicability as documented further in this attachment.

EPIX Failure #164 - River Bend Unit 1, March 23, 2001 (CR-RBS-2001-00403)

==

Description:==

Service Water Cooling pump failure occurred from fatigue failure of its bolting caused by a corrosion induced loss of pre-load. The bolting for two other Service Water Cooling pumps had not started cracking, but had similar but not as severe corrosion damage.

Stainless steel was recommended as a replacement for the carbon steel bolting. The relatively worse condition of the failed pumps bolting was contributed to manual addition of sulfuric acid in its bay during early years of Service Water Cooling system operation.

Applicability to Palisades:

Not relevant for this event. However, consideration to coupling material and service water chemistry needs to be considered.

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 80 of 136 NPRDS (Pumps, eductors) - Indian Point 2, September 22, 1993

==

Description:==

Pump failure was due to the sudden failure of one of the pump couplings. The pumps discharge check valve was subsequently discovered to be not seating properly and leaking by. This leak-by would cause the pump to rotate backwards, and this would produce an excessive start-upload. The coupling failure is attributed to non-ductile fracture because of temper embrittlement of the 410 SST. ASTM A276 Type 410SST minimum mechanical property requirements are adequate for normal pump loads per this standard but could be exceeded under impact or rapid loading conditions. The pump was replaced with a rebuilt pump using couplings with newly developed heat treatment specifications.

Applicability to Palisades:

Elements of this event are similar to Palisades. The coupling failure mechanism has been determined to be an initial Intergranular Stress Corrosion Crack (IGSCC) that developed and propagated to a point where a load-induced brittle failure occurred.

NPRDS (Pumps, eductors) - Beaver Valley 1, September 12, 1991

==

Description:==

Failure of a Byron-Jackson centrifugal pump resulted from a mechanical failure of the shaft coupling, Lehigh University laboratory test results attributed the coupling failure to embrittlement of the 410 SST material due to improper tempering temperatures and the potential impurities in the steel. The defective couplings were replaced with newly purchased safety-related couplings that were tested to ensure acceptability.

Applicability to Palisades:

Elements of this event are similar to Palisades. The coupling failure mechanism has been determined to be an initial Intergranular Stress Corrosion Crack (IGSCC) that developed and propagated to a point where a load-induced brittle failure occurred.

• EPIX Failure #167 - Perry Unit 1, September 1, 2003

==

Description:==

Emergency Service Water (ESW) A pump lost flow after 42 minutes of operation. Follow up investigation found no evidence of a pump or motor transient or any sign of foreign material obstruction in the pump impellers. Disassembly of other pump found the first line shaft coupling sleeve had failed and was found in two pieces inside the pump assembly. Visual inspection of wear marks on the broken coupling sleeve halves indicated the coupling was not centered between the two shafts. This left approximately one inch of the key extending above the coupling during operation.

Applicability to Palisades:

Palisades also had no evidence of a pump or motor transient or any sign of foreign material obstruction in the pump impellers. Nuclear Regulatory Commission Information Notice 2007-05 indicated the coupling failure was attributed to intergranular stress corrosion cracking. IGSCC was identified as a contributor to the Palisades coupling failures.

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 81 of 136 EPIX #659 - Catawba Unit 1, July 12, 2008 (C-08-04289)

==

Description:==

Root cause of the Catawba event was the use of the coupling sleeve manufactured from a deficiently formed alloy. Specifically, the pump failed due to the heterogeneity of the upper Johnston coupling material. Heterogeneity caused the material to be highly susceptible to intergranular corrosion and cracking (IGSCC) as shown by an accumulation of fine sulfide stringer inclusions along the boundaries of the failed coupling. Use of Martensitic stainless steel, A582 type 416 condition - T, the maximum hardness level of Rockwell C-25 should be specified to minimize the susceptibility of the material to stress corrosion cracking (SCC). Material susceptibility to SCC is border-line in the mid -20s range. The Johnston coupling had a hardness of Rockwell C-28; therefore, the specified hardness of the Johnston coupling is a contributing cause.

Applicability to Palisades:

Chemical analysis of the 2009 failed coupling at Palisades did not identify any issues with material alloy; therefore the heterogeneity of the Catawba coupling is not an area of concern for Palisades.

• INPO Operating Experience Digest 2006-02

==

Description:==

This document discusses Service Water Pump failures identifying specific plant sites that have experienced these failures. Service water pump shaft, coupling, and impeller failures have been identified as a continuing trend of service water mechanical problems. Twelve failures have been reported to the industry from 1998 through 2006, averaging more than one failure per year with several stations having multiple failures.

The most frequent cause has been corrosion of the shaft and bolting material. Corrosion has resulted in bolting failure; shaft shearing, impeller and coupling separation, and binding of the impeller to the bowl. Contributing causes include the following:

x Improper heat treatment during manufacture.

x Incorrect bolting and shaft material specification that are more susceptible to intergranular stress corrosion cracking.

x Use of dissimilar metals resulting in galvanic corrosion.

x Pump operation at low speed resulting in resonance vibration fatigue.

x Stray current flow from the cathodic protection system.

x Excessive bearing wear.

x Abnormal changes in lubricating water temperature (high or low) can result in differential rates of thermal expansion leading to binding of internal parts.

Applicability to Palisades:

This OE is applicable to Palisades because of material similarities and indications of intergranular stress corrosion cracking.

Electric Power Research Institute (EPRI)

EPRI web site was searched for documents using the keywords Service Water Pumps.

Two documents of interest are NP-7413, Deep Draft Vertical Centrifugal Pump Maintenance and Application Guide and Vertical Pump Maintenance Guide Supplement to NP-7413, Deep Draft Vertical Centrifugal Pump Maintenance and Application Guide. These documents provide an extensive discussion on vertical pump components, material specifications, maintenance, and troubleshooting. A review of

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 82 of 136 these documents did not reveal any specifics to this particular event, but does provide a validation of maintenance, operation, and engineering practices with accepted industry standards.

ADAMS Search / External OE document review The NRC Agency-wide Documents Access and Management System (ADAMS) was queried for events involving Service Water Pumps.

NRC INFORMATION NOTICE 2007-05: VERTICAL DEEP DRAFT PUMP SHAFT AND COUPLING FAILURES, February 9, 2007

==

Description:==

This Information Notice was issued to alert licensees to vertical deep draft pump shaft and coupling failures from intergranular stress corrosion cracking (IGSCC). Service Water Pump shaft failures were experienced at Columbia Generating Station. The metallurgical examination determined that the shaft material, TP410 martensitic stainless steel, was susceptible to tempering embrittlement. Tempering embrittlement reduced the corrosion resistance of the shaft material, thereby, increasing the materials susceptibility to IGSCC. NRC review of Operating Experience records identified at least 23 essential SW pump shaft and coupling failures since 1983 involving more than six different pump manufacturers. Many of these failures involved IGSCC as a primary cause. Other causes of shaft and coupling failures included: misalignment, imbalance, installation errors, and deferred maintenance. Two incidents since 2001, involving IGSCC are:

x IN 2007 Perry experienced SW pump shaft coupling failures due to IGSCC in September 2003 and May 2004.

x ML020920543 - VC Summer experienced SW pump shaft coupling failure during testing due to IGSCC in May 2001.

Applicability to Palisades:

This OE is applicable to Palisades because IGSCC was identified as a contributor to the coupling failures. This OE was reviewed under LO-PLPLO-2007-00059 and was concluded to be non applicable based on the material characteristics of 416 stainless steel. This OE was also used as input into EC5000121762, but the EC did not acknowledge that a fresh water environment should be considered as a potentially corrosive environment.

NRC INFORMATION NOTICE 93-68: FAILURE OF PUMP SHAFT COUPLING CAUSED BY TEMPER EMBRITTLEMENT DURING MANUFACTURE, September 1, 1993

==

Description:==

The U.S. Nuclear Regulatory Commission issued this information notice to alert addressees to problems caused by temper embrittlement of American Iron and Steel Institute Type 410 stainless steel couplings supplied by Byron Jackson. On June 20, 1991, a river water pump shaft coupling at Beaver Valley Nuclear Power Plant, Unit 1, failed during operation when a large section of one end of the coupling broke away from the rest of the coupling. This coupling, which was threaded internally, was used to join two shafts of a Byron Jackson vertical circulator river water pump. During its investigation of the failure, the licensee found that two more couplings from the same pump had cracks. All three of the Unit 1 pump shafts had at least one of the defective couplings. The licensee at Beaver Valley noted that increased vibration levels on pump

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 83 of 136 1A caused by a worn bearing, pump shaft misalignment or both contributed to the failure. The cause of the Beaver Valley failure was determined in independent laboratory testing as impact strength of the couplings due to temper embrittlement resulting from improper heat treatment.

Applicability to Palisades:

This is the same event described in the INPO EPIX search. Elements of this event are similar to Palisades. Palisades Service Water pump couplings are made from 416 SS.

410 SS and 416 SS have similar chemical compositions and properties.

ML020930345 Indian Point 2 Date: 06/30/94

==

Description:==

June 29, 1994, a service water pump failed to develop discharge pressure during a pump start. A contributing cause was the low impact resistance of the material used.

There have been three other service water pump failures at Indian Point 2. In August 1993, two pumps failed due to sudden impact loading caused by foreign object ingestion. In September 1993, a pump failure was attributed to impact loading caused by reverse flow through the check valve during startup. In all cases, a contributing cause was the low impact resistance of the material used for the pump shaft couplings.

Applicability to Palisades:

This event describes pump failures described in the INPO EPIX search as well as the 1994 Indian Point pump failure. Elements of this event are similar to Palisades. There is currently no evidence that impact or rapid loading pump conditions were experienced at Palisades but the coupling hardness was a contributor to the September 29, 2009 coupling failure.

Non-Nuclear An Internet search for similar non-nuclear pump failures was conducted. Similar pumps are used in water treatment plants and other facilities. Various key words were used such as layne bowler, vertical pump failures, and coupling failures. The internet hits did not produce any database that discussed similar coupling failures. Relevant internet hits identified NRC documents that already have been identified above. An additional search on 410 stainless failures did produce two articles of interest. The first article, published by Flowserve in a Materials Newsletter, dated September 2004, discusses temper embrittlement if materials like 410 stainless steel are cooled too slowly between 800 and 1000 °F. A second article, published by the US Army Corps of Engineers, September 2003, titled Results of Evaluation of Bolt Failures at the R.C. Byrd Locks and Dam, states:

Type 410 stainless steel is subject to temper embrittlement during the heat treatment process. If the material is held too long in the 700 to 1,000 °F, it allows the precipitation of carbides, which reduce the toughness and increase the tensile strength and hardness.

The tensile strength and hardness peak when the stainless steel is held at 885 °F for an extended period of time. If the material passes through this temperature range, then little precipitation occurs and; therefore, no embrittling effects affect the structure of the stainless steel. Even though the furnace temperature shows a fairly rapid movement through this range, the parts being heat treated can still be embrittled if they are clumped tightly together, thus affecting the overall mass that needs to be heated up or cooled down.

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 84 of 136 Applicability to Palisades The 2009 coupling failure indicated after a review of the heat treatment facilities operating processes, it was determined that this condition does not apply to heat treated materials purchased from HydroAire and their sub-supplier Bodycote. Bodycote uses nitrogen at 2 bar (approximately 29 psi) to quench after the high heat treating step. This eliminates the possibility of the concern in the OE article. Based on pending metallurgy on the current failure, this OE may have applicability.

Additional or new Operating Experience since CR-PLP-2009-04519, Service Water Pump P-7C Failure to Provide Discharge Pressure Additional INPO and Internet searches were conducted on the term pump coupling failure. Various alterations of these terms were used to narrow the return hits to those that might be more applicable to this event. The INPO Operating Experience web site was searched using pump failure and the date January 1, 2009 through August 17, 2011 to assure any new events were identified.

• One additional operating experience not previously identified was found on the Internet.

The Handbook of Case Histories in Failure Analysis identified a fracture of a coupling in a line-shaft vertical turbine pump. The pump was installed in a *dam foundation. The cause identified the fracture was brittle and initiated by an intergranular cracking mechanism. Improper heat treatment was attributed to the material being susceptible to corrosion being initiated by stress or hydrogen cracking.

Applicability to Palisades This event involved an upper coupling with failure initiated by intergranular stress corrosion cracking of similar material. This operating experience appears to have applicability to the Palisades failure.

• The Perry Nuclear Station experienced a second service water pump coupling failure May 21, 2004. This was one of the events identified in the INPO Operating Experience Digest 2006-02 (OE used in CR-PLP-2009-04519). A copy of Perrys root cause report was obtained and reviewed for additional details.

Applicability to Palisades This event involved an upper coupling with failure initiated by intergranular stress corrosion cracking of similar material. Design of the coupling was also investigated for stress points that may have exasperated the failure. This operating experience is applicable to the Palisades failure.

• EPIX Failure #339, Prairie Island - On 25 JUL 2010, at 10:21, the 121 Motor Driven Cooling Water Pump experienced a complete loss of pump discharge pressure due to the failure of two shaft couplings and the separation of their respective shaft segments.

The first and second couplings from the pump shaft motor end were found to be fractured 360 degrees. The failure of both couplings was identified as a faulty design specification. The specification did not limit the hardness. The high hardness caused the coupling to be less tough, which subsequently reduced coupling tolerance to the affects of MIC, and increased susceptibility to transgranular and intergranular stress corrsosion cracking. Combined with MIC pitting at the relief hole, stress cracks were exposed, which caused a rapid failure of the couplings by brittle intergranular fracture

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 85 of 136 Applicability to Palisades This event involved an upper coupling with failure initiated by intergranular stress corrosion cracking of similar material. Design of the coupling was in question. The coupling supplier is the same as Palisades. This operating experience is applicable to the Palisades failure.

Summary of Operating Experience using Grade 416 Stainless Steel Operating experience that discussed temper embrittlement, material hardness, stress corrosion cracking and coupling design features were found to be applicable. The Palisades coupling material being 416 stainless steel is prone to these attributes and requires consideration in light of this recent coupling failure. Operating Experience examined included the Perry Repeat failures in 2003 and 2004 as well as the recent 2010 Prairie Island Event. The Prairie Island OE was used by Palisades as part of the decision to change coupling materials from 416 SS to 17-4ph SS.

The OE highlights the need to ensure that proper material specifications and processes are applied for controlling hardness, toughness and other material properties that make 416 stainless steel less prone to temper embrittlement and corrosion cracking failures.

The OE also makes it clear that Licensees stipulate proper quality controls that assure coupons and testing results reflect actual material conditions.

Also, Design Engineering activities need to verify the coupling design to assure that stress risers are minimal and would not contribute to corrosion cracking Stainless Steel - Grade 630, 17-4 PH INTERNAL and External Operating Experience:

Operating experience search for 17-4 PH material was conducted. INPO and NRC websites searches were used. Limited events with 17-4 PH steel were identified and mainly pertained to valve stems and springs in primary coolant or engineered safeguards systems environments. Examples were NRC Information Notice 2007-02, Failure of Control Rod Drive Mechanism Lead Screw Male Coupling at a Babcock and Wilcox Designed Facility, and Information Notice 86-72, Failure 17-7 PH Stainless Steel Springs in Valcor Valves due to Hydrogen Embrittlement.

IN 2007-02 event resulted from thermal embrittlement due to the component being exposed to high temperature (550°F). IN 86-72 failure of disc guide assembly springs made out of 17-7 stainless steel resulted from hydrogen embrittlement which is a function of high temperature, water chemistry, water flow condition, and time of exposure to the service condition. Other operating experience identified appeared to be similar these and not particularly relevant to the application for service water pump couplings.

OE31481 - Salem Unit 2, Microbiologically Influenced Corrosion Causes Valve Shaft Failure, Event Date: 05/17/2010 While performing the periodic component cooling heat exchanger service water side high flow flush, the service water flow could not be adjusted to the required range. Initial troubleshooting determined that the inlet flow control valve was not controlling flow as expected. The plant entered an unplanned 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> Limiting Condition for Operation and

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 86 of 136 replaced the valve. The causal investigation determined that the A564 Grade 630 (17-4 PH) valve shaft failed from microbiologically influenced corrosion.

Lessons Learned for the Industry:

Microbiologically influenced corrosion may affect operation of A564 Grade 630 (17-4 PH) valve shafts in brackish raw water systems.

An internet search was conducted for 17-4 stainless steel with several documents providing discussions on applications of this material and in some cases vulnerabilities of the material.

Fastener Technology International, October 2003 article, Other Causes of Fastener Failures, presented a situation where after passivation of 17-4 stainless steel fasteners, the material appeared corroded or attacked with a dull gray surface finish. The fasteners had not been in service. A cross section etched metallographic examination revealed a distinct non-uniform white layer along the surface of the fastener that was found to be reverted austenite. Corrosion of this layer was visible. Reverted austenite is typically the result of nitrogen pick-up during heat treatment most likely from a contaminated furnace atmosphere.

NACE International, Document ID - 04126 An offshore production facility philosophy of material selection was to use corrosion resistant alloys wherever wet gas was being handled. A 17-4 stainless steel valve stem failed twelve days after startup. An operator walking by one of the main 32" ball valves heard a loud crack and saw the valve stem rise up through the top of the valve. The valve was sitting in the fully open position not being operated at the time.

The fracture was a classic sulfide stress cracking brittle failure. Insitu hardness testing of the broken stem, as well as, stems from four other similar valves on the platform found hardnesses in the range of 35 - 45 on the Rockwell C (HRC) hardness scale. Original specifications called for these stems to meet NACE MR0175, where the maximum allowed hardness is 33 HRC. Subsequent investigations found that the forging mill had taken raw 17-4 PH material, cut coupons, heat-treated the coupons and created mill specifications based on the coupons. The actual stem material was heat treated separately and incorrectly.

Stephen J. Morrow article, When High-Strength Means No-Strength In this article, the author points out that for High-strength materials metallurgical factors are important to understand, but of equal importance is environmental influences which can promote failures such as environmentally induced cracking. Two major types of environmental cracking are hydrogen embrittlement (HE) and stress corrosion cracking (SCC). Both of these phenomena often result in catastrophic, brittle fracture at stress levels significantly below the materials yield stress.

High-strength steels often utilized for pump shaft applications can be susceptible to hydrogen embrittlement (HE). Hydrogen embrittlement (HE) is the general term given for a loss of toughness resulting from hydrogen absorption. Embrittlement results from the interaction of hydrogen and tensile stresses in susceptible materials. This type of hydrogen damage occurs most often in alloys such as quenched and tempered martensitic steels, and the martensitic precipitation-hardened steels. Susceptibility to fracture generally increases with increased strength and hardness. Embrittlement can result from a very small amount of hydrogen, often as little as a few parts per million.

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 87 of 136 Hydrogen may enter susceptible alloys from various sources. Some include: solutions containing hydrogen sulfide (H2S); strong acids; galvanic coupling to more active (anodic) alloys in a corrosive environment such as seawater; cathodic protection; and even microbiological (e.g. SRB - sulfate reducing bacteria) corrosion. Other sources may include residual hydrogen pick up from electroplating or pickling operations, exposure to high-pressure hydrogen gas, or sodium sulfite decomposition in high pressure boiler feed-water, etc.

Sulfur/sulfide environments not considered sour by NACE MR-0175 definitions can result in failures by hydrogen embrittlement (HE) and sulfide stress cracking (SSC) of susceptible materials such as 17-4 PH precipitation hardened stainless.

Even though materials may be selected because of compliance to NACE MR-0175, it doesn't guarantee freedom from environmental cracking. It should be noted that materials included in this standard are resistant to, but not necessarily immune to SSC under many service environments. While the susceptibility to SSC can be strongly affected by heat treatment, 17-4 PH precipitation hardened steels that have been properly heat treated to the NACE MR0175 requirements still failed by cracking. Even with NACE requirements specified there is no guarantee that failures will be prevented in 17-4 PH stainless steel. The NACE MR-0175 heat treatment requirements for UNS S17400 precipitation-hardened stainless steels requires either a Double Aging treatment at 1150°F; or a three step process which is also a Double Aging treatment at 1400°F then at 1150°F for a maximum hardness of 33 HRC. The later three step process can be furnished by specifying steel to meet ASTM A564 UNS S17400 Type 630 in the H1150M condition, rather than the single aged H1150 condition, and adding the requirement for 33 HRC maximum hardness. The resistance of high-strength steels to environmental cracking improves with reduced strength (hardness), and alloying to improve toughness.

Specifying ASTM A564, Grade UNS S17400 in the H1150M condition results in significant reductions in strength; lowering the ultimate tensile strength from 135 Ksi to 115 Ksi min.; and the yield strength from 105 Ksi to 75 Ksi min.

Nickel Development Institute, Guidelines for selection of nickel stainless steels for marine environments, natural waters and brines, states 17-4 stainless steel is widely used in marine equipment wherever higher strength is required. However, it is somewhat prone to crevice attack. Crevice corrosion is the localized breakdown of the chromium oxide film, which is caused by micro and macro biofouling organisms attaching themselves to the stainless steel surface. Overaged condition, H1100 or H11150, is preferred for better resistance to stress corrosion attack.

NUREG/CR-6223, Review of the Proposed Materials of Construction fro the SBWR and AP600 Advanced Reactors, is mainly a review of materials being used in the primary systems of these reactor designs. It identifies uses of 17-4 PH materials in various components and the vulnerabilities mention in the previous operating experience documents. This document cautions that 17-4 PH precipitation-hardening stainless steel chosen for the control rod drive seal housing nuts in the SBWR is subject to severe SCC and hydrogen embrittlement if improperly heat treated, and stringent acceptance criteria are required for this component to avoid this potential problem.

Operating Experience Summary for using Grade 630, 17-4 PH Stainless Steel

Attachment IV - Operating Experience RCE Report CR-PLP-2011-03902 Page 88 of 136 The OE identified the need to evaluate coupling environment and material conditioning to assure 17-4 stainless steel coupling are not prone to hydrogen embrittlement, thermal embrittlement, sulfide stress corrosion cracking, crevice corrosion and Microbiological influenced corrosion.

Also, there is a need to assure stringent process controls are applied to coupling manufacture to avoid contaminates during the heat treat process, such as nitrogen pick-up, that would affect the conditioning of 17-4 PH material.

Additionally, the responsible design activity must ensure proper material specifications and processes are applied for controlling hardness, toughness, and other material properties that make 17-4 less prone to embrittlement and corrosion cracking failures.

Equally important is the need to stipulate proper quality controls that assure coupons and testing results reflect actual material conditions.

Attachment V: LPI Report Boston Area Office, 36 Main Street, Amesbury, MA 01913 Tel: 978-517-3100 Fax: 978-517-3110 www.luciuspitkin.com New York, NY Boston, MA Richland, WA Ensuring the integrity of todays structures for tomorrows world TM RCE Report CR-PLP-2011-03902 Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering METALLURGICAL AND FAILURE ANALYSIS OF SWS PUMP P-7C COUPLING #6 Report No. F11358-R-001 Revision DRAFT G September, 2011 Prepared For ENTERGY NUCLEAR OPERATIONS, INC.

PALISADES NUCLEAR PLANT

Attachment V: LPI Report Boston Area Office, 36 Main Street, Amesbury, MA 01913 Tel: 978-517-3100 Fax: 978-517-3110 www.luciuspitkin.com New York, NY Boston, MA Richland, WA Ensuring the integrity of todays structures for tomorrows world TM RCE Report CR-PLP-2011-03902 Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

Attachment V: LPI Report Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering DOCUMENT RECORD Document Type:

Calculation Report Procedure Document No:

F11358-R-001 Document

Title:

Metallurgical and Failure Analysis of SWS Pump P-7C Coupling #6 Client:

Entergy Nuclear Operations, Inc.

Client Facility:

Palisades Nuclear Plant Client PO No:

10325528 Quality Assurance:

Nuclear Safety Related?

No Yes Computer Software Used:

No1 Yes2

1. Check NO when EXCEL, MathCAD and/or similar programs are used since algorithms are explicitly displayed.

2. Include Software Record for each computer program utilized Instrument Used No Yes3

3. Include Document Instrument Record Revision Approval Date Preparer Checker Design Verification Approver4 DRAFT G 08/31/11 S. Yim John Mills, Ph.D Ryan Chen P. Bruck T. Esselman, Ph.D P. Bruck 4 The Approver of this document attests that all project examinations, inspections, tests and analysis (as applicable) have been conducted using approved LPI Procedures and are in conformance to the contract/purchase order.

Page 2

of xx Total Pages Include any Title Sheet and Attachments in page count This report is rendered upon the condition that it is not to be reproduced wholly or in part for advertising or other purposes over our signature or in connection with our name without special permission in writing. Be advised that all materials submitted for evaluation will be retained for six months.

After such time, all material will be discarded unless otherwise notified in writing to retain beyond six months.

Attachment V: LPI Report Report No. F11358-R-001 Page 92 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering RECORD OF REVISION Revision No.

Date Description of Change Reason DRAFT G See Document Record Issued for comment

Attachment V: LPI Report Report No. F11358-R-001 Page 93 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering DESIGN VERIFICATION CHECKLIST Document No(s)1: F11358-R-001 Rev.:

Review Method:

X Design Review Alternate Calculation Test Criteria DV2 1

Were the inputs correctly selected and incorporated into design?

2 Are assumptions necessary to perform the design activity adequately described and reasonable? Where necessary, are the assumptions identified for subsequent re-verifications when the detailed design activities are completed? If applicable, has an as built verification been performed and reconciled?

3 Are the appropriate quality and quality assurance requirements specified?

4 Are the applicable codes, standards and regulatory requirements including issue and addenda properly identified and are their requirements for design met?

5 Have applicable construction and operating experience been considered, including operation procedures?

6 Have the design interface requirements been satisfied?

7 Was an appropriate design method used?

8 Is the output reasonable compared to inputs?

9 Are the specified parts, equipment, and processes suitable for the required application?

10 Are the specified materials compatible with each other and the design environmental conditions to which the material will be exposed?

11 Have adequate maintenance features and requirements been specified?

12 Are accessibility and other design provisions adequate for performance of needed maintenance and repair?

13 Has adequate accessibility been provided to perform the in-service inspection expected to be required during the plant life?

14 Has the design properly considered radiation exposure to the public and plant personnel?

15 Are the acceptance criteria incorporated in the design documents sufficient to allow verification that design requirements have been satisfactorily accomplished?

16 Have adequate pre-operational and subsequent periodic test requirements been appropriately specified?

17 Are adequate handling, storage, cleaning and shipping requirements specified?

18 Are adequate identification requirements specified?

19 Are requirements for record preparation review, approval, retention, etc., adequately specified?

20 Has an internal design review been performed for applicable design projects? Have comments from the Internal Design Review been appropriately considered/addressed?

(1)

Include any drawings developed from reviewed documents, or include separate checklist sheet for drawings (2)

Design Verifier shall initial indicating review and mark N/A where not applicable DV Completed By:

Printed Name Signature Date Page of Total Pages Include DV Checklist and Comment Resolution sheets in page count

Attachment V: LPI Report Report No. F11358-R-001 Page 94 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering DOCUMENT SOFTWARE RECORD (Include Separate Sheet for Each Software Package Utilized) 1 Computer Software Used (Code/Version)

ANSYS Version 11.0 2

Software Supplier ANSYS, Inc.

3 Software Update Review Error notices; describe: Reviewed error reports for elements used Other; describe:

NO 4

Nuclear Safety Related Software YES1

1. If YES:

Hardware identification # used for execution:

Desktop Serial #: J2WTBM1 Basis for V & V: [16]

5 Input Listing(s)

Input listing(s) attached:

Not attached; identify File/Disc ID*:

Coupling Pump Bearing & Bending.txt Coupling Pump Bearing.txt Coupling Pump No Bearing.txt

• A CD with input listings and output data to be provided on project completion.

6 Output results attached:

Not attached; identify File/Disc ID*:

• A CD with input listings and output data to be provided on project completion.

Output Identifier(s)*

(see 6 above) 7

• e.g., run date/time; use for reference, as appropriate, within body of calculation 8

Comments Keywords**

SOLID45, Static 9

• • For use in describing software features used in this calculation; use common terms based on software user manual.

10 Project Manager Name:

S. Yim If computer software was used on project, complete form with required information.

Update the LPI Computer Software Use List per LPI Procedure 13.1 requirements.

Attachment V: LPI Report Report No. F11358-R-001 Page 95 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering DOCUMENT INSTRUMENT RECORD Instrument Used Instrument Description Serial No.

Calibration Due Date 1

Tensile Testing Machine (120 kips)

Baldwin 37205 4/7/12 2

Extensometer (1 in) 2620-824/1033 4/7/12 3

Charpy Impact Tester Satec Model SI-1K/1306 6/17/12 4

Hardness Tester Wilson 5YR/58 4/7/12 5

Thermocouple Omega 650 J/8320 7/12/12 6

Caliper Fowler 6/7082002 6/21/12 7

Magnetic Yoke Magnaflux Y-6/43530 Per use calibration 8

9 10 11 SEM/Oxford EDS 17218-118-01 Per use calibration 12 13 14 Project Manager Name:

S. Yim For instrument(s) used on the project, identify instrument and include the instrument calibration due date.

Update the LPI Instrument Use List per LPI Procedure 13.1 requirements.

Attachment V: LPI Report Report No. F11358-R-001 Page 96 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering TABLE OF CONTENTS Page No.

DOCUMENT TITLE PAGE......................................................................................................1 DOCUMENT RECORD SHEET...............................................................................................2 RECORD OF REVISION.......................................................................................................... 3 DESIGN VERIFICATION CHECKLIST.................................................................................. 4 DOCUMENT SOFTWARE RECORD.......................................................................................5 DOCUMENT INSTRUMENT RECORD...................................................................................6 TABLE OF CONTENTS............................................................................................................7 LIST OF FIGURES....................................................................................................................8

1.0 INTRODUCTION

/BACKGROUND...................................................................... 99 1.1 Scope and Purpose........................................................................................ 100 2.0 INPUTS/ASSUMPTIONS.................................................................................. 106 2.1 Inputs............................................................................................................. 106 2.2 Assumptions................................................................................................... 106 3.0 TESTING AND EXAMINATION......................................................................... 107 3.1 Visual and Stereomicroscopic Examination................................................... 107 3.2 Dimensional Examination............................................................................... 108 3.3 Magnetic Particle Examination (MT).............................................................. 109 3.4 Metallurgical and Scanning Electron Microscopy........................................... 109 3.5 Tensile Test....................................................................................................110 3.6 Composition of Base Metal............................................................................ 110 3.7 Composition of Surface Deposit..................................................................... 110 3.8 Hardness Surveys.......................................................................................... 110 3.9 Charpy V-Notch (CVN) Impact Testing.......................................................... 111 4.0 EVALUATION.................................................................................................... 126 4.1 Susceptible Material....................................................................................... 126 4.2 Corrosive Environment................................................................................... 127 4.3 Tensile Stress................................................................................................ 127 4.4 Crack Propagation......................................................................................... 131 5.0

SUMMARY

/RECOMMENDATION.................................................................... 141

6.0 REFERENCES

.................................................................................................. 142 LIST OF ATTACHMENTS Total Pages A: MISC INPUTS................................................................................................................XX B: TENSILE TEST DATA.....................................................................................................XX C: HARDNESS SURVEY DATA........................................................................................... XX

Attachment V: LPI Report Report No. F11358-R-001 Page 97 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering LIST OF FIGURES Figure 1-1: As-Received 09-P7C-7F........................................................................... 102 Figure 1-2: As-Received 11-P7C-6F........................................................................... 102 Figure 1-3: SWS Pumps Shaft Assembly [3b]............................................................. 103 Figure 1-4: PLP SWS Pump Rendering...................................................................... 104 Figure 1-5: Coupling Drawing [19]............................................................................... 105 Figure 3-1: As-Received 11-P7C-7............................................................................. 116 Figure 3-2: Coupling Dimensioning Scheme............................................................... 116 Figure 3-3: Visual of Fracture Surface on Coupling 11-P7C-6F.................................. 117 Figure 3-4: 11-P7C-6F showing Corrosion Deposit..................................................... 117 Figure 3-5: Ends of Shaft 5 and 6 Coupled by 11-P7C-6F.......................................... 119 Figure 3-6: MT Highlighting Un-Opened Fracture on 11-P7C-6F................................ 119 Figure 3-7: MT Highlighting Crack on Coupling 11-P7C-7.......................................... 120 Figure 3-8: As-Polished of 11-P7C-7 Specimen.......................................................... 121 Figure 3-9: Micro-Structure of 11-P7C-7 Specimen.................................................... 123 Figure 3-10: SEM of 11-P7C-6F Surface.................................................................... 123 Figure 3-11: EDS of 11-P7C-6F Surface Deposit - Spectrum 2................................. 124 Figure 3-12: EDS of 11-P7C-6F Surface Deposit - Spectrum 4................................. 124 Figure 3-13: Charpy Test Result................................................................................. 125 Figure 4-1: Hardening Heat Traces............................................................................. 132 Figure 4-2: Tempering Heat Traces............................................................................ 133 Figure 4-3: MTS1: Shaft Not Bearing.......................................................................... 133 Figure 4-4: MTS1: Shaft Bearing................................................................................. 134 Figure 4-5: Half FEA model of coupling....................................................................... 134 Figure 4-6: Cross-section of half FEA coupling model................................................ 135 Figure 4-7: Load application Sketch of loading condition in no bearing case.............. 136 Figure 4-8: Sketch of loading condition in shafts bearing case................................... 136 Figure 4-9: Sketch of axial force result from bending moment.................................... 137 Figure 4-10: Resultant stresses for LC1...................................................................... 139 Figure 4-11: Resultant stresses for LC2...................................................................... 140 Figure 4-12: Resultant stresses for LC3.......................... Error! Bookmark not defined.

LIST OF TABLES Table 1-1: Test Matrix................................................................................................. 100 Table 3-1: Measurement of Coupling Outside Dimensions.........................................112 Table 3-2: Measurement of Coupling Wall Thickness................................................. 112 Table 3-3: Tensile Test Results................................................................................... 112 Table 3-4: Metal Composition of Couplings (Wt. %).................................................... 113 Table 3-5: Surface Hardness Survey of Couplings..................................................... 114 Table 3-6: Through Thickness Hardness of Couplings................................................114 Table 3-7: CVN Impact Test Results........................................................................... 115

Attachment V: LPI Report Report No. F11358-R-001 Page 98 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

Attachment V: LPI Report Report No. F11358-R-001 Page 99 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering INTRODUCTION/BACKGROUND Two (2) failed coupling along with other intact components, as documented in the receipt inspection of 8/17/11 and 8/18/11 (provided in Attachment A), were submitted to LPI (Lucius Pitkin, Inc.) for material and failure assessment. The received components were extracted from Pump P-7C of the Service Water System (SWS) at Palisades Nuclear Plant (PLP). One of the two failed couplings was from a coupling failure event in September 2009 as documented in CR-PLP-2009-04519 [1]1 and the other failed coupling was from a failure event in August 2011 as documented in CR-PLP-2011-03902 [2]. The failed couplings from the 2009 and 2011 failure events are herein referred to as the 09-P7C-7F and 11-P7C-6F, respectively (refer to coupling identification convention in Section 2.1).

Photographs of couplings 09-P7C-7F and 11-P7C-6F are presented in Figure 0-1 and Figure 0-2, respectively.

The SWS comprise of three motor driven vertical multistage pumps, tagged P-7A, P-7B and P-7C, supplying water from Lake Michigan to three service water headers. All three SWS pumps are similar in design in that they are comprised of two stage stainless steel impellers coupled to the motor through six line shafts, a packing shaft and a motor shaft for a total height of over 40 feet from suction to discharge. Figure 0-3 shows the shaft and coupling arrangement for the SWS pumps and identifies couplings 09-P7C-7F and 11-P7C-6F. As can be seen in Figure 0-3, the 09-P7C-7F is coupling #7 and the 11-P7C-6F is coupling #6. A rendering that identifies the pump components (excluding the motor) is provided in Figure 0-4.

P-7A and P-7C are Layne and Bowler Model 25RKHC pumps while P-7B is a Johnston Model 25NMC pump. Each pump is driven by a 350 horsepower (HP) motor providing a rated 8000 GPM and 140 ft total developed Head (TDH) each at 50% service capacity [1].

The specified material of the 09-P7C-7F and 11-P7C-6F as well as all shaft couplings on the three SWS pumps are ASTM A582 Type 416 stainless steel (SS) [4]. The material specification for the shaft couplings on all three pumps was changed from carbon steel to 416 SS and specified with a Rockwell C hardness (HRC) value of 28 to 32 under EC-50000121762 [4] in December 2007. The couplings were also redesigned to incorporate an alignment hole that allows 1 Numbers in brackets (e.g. [5]), indicate references listed in Section 0.

Attachment V: LPI Report Report No. F11358-R-001 Page 100 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering verification of proper shaft installation. The shaft couplings for P-7A were replaced on April 4, 2009 per Work Order (WO) 51637416. The shaft couplings for P-7C were replaced on June 12, 2009. The shaft couplings on P-7B were replaced during rebuild of the pump and installed in June of 2010 [2]. A detail drawing of the line shaft coupling is provided in Figure 0-5.

Scope and Purpose The scope and purpose of this report is to provide results of the metallurgical examination and tests performed in accordance with LPI Procedure F11358-P-001 [5] and provide a probable root cause of the 2011 coupling failure. The scope of test and examinations performed are provided in the Test Matrix provided in Table 0-1.

Table 0-1: Test Matrix Test Components Test 2011 Couplings (Note 1) 2009 Failed Coupling Visual & Photographic 1 through 7 Surface Hardness 1 through 7 X

UT Exam See Note 1 Dimensional Exam 1 through 4 Comp Analysis of Surface Deposits 6 and 7 MT Exam 5 through 7 Tensile Test 5 through 7 CVN Test 5 through 7 X

Thru Thick Hardness 5 through 7 X

Comp Analysis 5 through 7 SEM 6

X Note(s):

1. UT examination of the couplings could not be performed due to the end geometry of the couplings. The lack of this examination in the overall Test Matrix does not diminish the capability of assessing the failure mechanism.

Attachment V: LPI Report Report No. F11358-R-001 Page 101 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Elevation View Top View

Attachment V: LPI Report Report No. F11358-R-001 Page 102 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-1: As-Received 09-P7C-7F Figure 0-2: As-Received 11-P7C-6F

Attachment V: LPI Report Report No. F11358-R-001 Page 103 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-3: SWS Pumps Shaft Assembly [3b]

11-P7C-6F 09-P7C-7F

Attachment V: LPI Report Report No. F11358-R-001 Page 104 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Note: Spider refers to intermediate shaft guidance bushing.

Figure 0-4: PLP SWS Pump Rendering

Attachment V: LPI Report Report No. F11358-R-001 Page 105 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-5: Coupling Drawing [19]

Attachment V: LPI Report Report No. F11358-R-001 Page 106 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering INPUTS/ASSUMPTIONS The following inputs and assumptions are utilized for this report. Inputs and assumptions requiring verification are identified as such.

Inputs

1. Specified hardness for the couplings is in the range of 28 to 32 HRC [4].

2. Hydraulic forces on the pump coupling are taken to be 8kips per [14].

3. Service water basin elevation ranges from 576 ft to 580 ft for the period from January, 2009 to August, 2011. For the same period, the water temperature ranged from a minimum of 32°F to a maximum of 76°F (see data in Attachment A)

4. The SWS pumps are vertical turbine pumps that take suction from the bottom and discharges through the column and out the discharge head.

Therefore line shafts, couplings and components below the stuffing box will be exposed to the service water when the pumps are on. Based on the service water basin elevation, couplings 1 through 4 are constantly submerged, whereas coupling 5 through 7 are subjected to cycles of wet and dry depending on whether the pumps are on or off. Also depending on water elevation, coupling 5 can be submerged when the pump is not on. Refer to Figure 0-4 for illustration of pump with water flow and relationship to basin water elevation variations.

5. The convention used in this report to reference couplings is as follows:

YY-Pump-CN with Optional F Where:

YY= two digit year in which the coupling was extracted from the pump.

Pump = P7A, P7B or P7C CN = Coupling Number Optional F = Identifies a coupling that has failed.

For example: 09-P7C-7F is the failed coupling extracted from Pump 7C in 2009.

Assumptions

1. There are no assumptions utilized in this report.

Attachment V: LPI Report Report No. F11358-R-001 Page 107 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering TESTING AND EXAMINATION The submitted components, including the 09-P7C-7F and 11-P7C-6F couplings, were examined and tested in accordance with LPI Procedure F11358-P-001 [5]

and the test matrix presented in Table 0-1. The result of the testing and examination is discussed in the following sections.

Visual and Stereomicroscopic Examination 11-P7C Intact Couplings In all seven (7) intact couplings (#1 through 5, 7 and 8), disassembled from pump P7C following failure of 11-P7C-6F, were shipped to LPI for examination. Each coupling was engraved on each end with the shaft numbers that connected to the coupling. A representative example of an intact coupling is shown in Figure 0-1. The outer diameter of each intact coupling exhibited wrenching marks, which likely occurred during removal of the couplings. Evidence of previous hardness tests performed was present on both the outer diameter and ends of many couplings.

Each coupling exhibited a single alignment hole at its center, 0.125 in. in diameter. The coupling ends were chamfered at 45° on both outer (0.187 in.) and inner diameters (0.125 in.). The inner diameter of each coupling was fully threaded.

Visual examination of all intact couplings did not reveal significant signs of corrosion or degradation on the exterior. The 1/8 diameter shaft alignment holes on the intact couplings did not show any signs of corrosion deposits.

Coupling dimensions taken at points shown in Figure 0-2, are provided in Table 0-1.

Coupling 11-P7C-6F The fracture surface of the impeller end of coupling 11-P7C-6F was sectioned for analysis, as shown in Figure 0-3. The fracture surface was located near the center of the coupling. Approximately half of the fracture surface was flat and aligned perpendicular to the coupling axis. The flat fracture surfaces occur in two regions that each display an elliptical shape emanating from the thread root at the inner diameter and extend to the outer diameter, as shown in Figure 0-3. The elliptical features reveal that cracks initiated at the thread root and propagated from the inner to the outer

Attachment V: LPI Report Report No. F11358-R-001 Page 108 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering diameter of the coupling. Visually, the two elliptical crack patterns are indicative of stress corrosion cracking (SCC). The remaining portion of the fracture surface consisted of slanted fracture, indicating an overload event.

The overload event occurred after the flat fractures propagated through the wall thickness. Thread roots exhibited red/brown corrosion products, as shown in Figure 0-4. Corrosion deposits are visually apparent on the facture surfaces, threads and at the 1/8 alignment hole of coupling 11-P7C-6F.

The corrosion deposit streak at the bottom of the 1/8 diameter shaft alignment hole is indicative of the coupling undergoing a wet and dry cycle with the pump on and off (Figure 0-4).

The pointed tips of the fracture surfaces of the motor end and impeller end of fractured coupling 11-P7C-6F each exhibited signs of mechanical damage.

Coupling 09-P7C-7F The fractured coupling 09-P7C-7F measured between 2.058 and 3.425 in.

in length. Visual examination of the 09-P7C-7F coupling failure surface revealed similar patterns as the 11-P7C-6F failure surface with a smooth surface initiated at the thread root of the coupling. Dark spots indicative of corrosion deposits were also evident on the failure surface.

2011 Shafts Two shafts (shaft #5 and 6) from pump P-7C were shipped to LPI for visual examination. The end of shaft 5 and 6 that touch each other inside of failed coupling 11-P7C-6F were visually examined to characterize the nature of the galling at the end of the shaft. The pointed tips of the fracture surfaces of the motor end and impeller end of fractured coupling 11-P7C-6F each exhibited signs of mechanical damage. The ends of shaft Nos. 5 and 6 also exhibited mechanical damage, as shown in Figure 0-5. This damage was most likely caused after the initial failure by repeated contact between the fractured couplings and the shaft ends.

Dimensional Examination Coupling 11-P7C-6F was received in two halves that each exhibited a circumferential fracture surface. The bottom half (impeller end) of 11-P7C-6F measured between 3.030 and 4.290 in. in length. The top half (motor end) of the same coupling measured between 1.200 and 3.148 in. in length.

The length and diameter of all intact coupling were measured and the results are summarized in Table 0-1. The outside dimensions are within the specified dimensions and tolerances of HydroAire drawing 1047237 [19].

Attachment V: LPI Report Report No. F11358-R-001 Page 109 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering To evaluate the concentric threading of each coupling, the wall thickness (from the outer diameter to the thread crest) was measured around the diameter on each coupling end, as shown in Figure 0-2. Table 0-2 gives the measured wall thicknesses. Coupling 11-P7C-4 exhibited the highest eccentricity (0.007 in.), occurring at its motor end. Couplings 11-P7C-5 and 11-P7C-6F were sectioned prior to dimensional analysis and subsequently some wall thickness measurements were not available Magnetic Particle Examination (MT)

Coupling 11-P7C-6F Visible cracks in coupling 11-P7C-6F readily highlighted the un-opened fracture surfaces upon MT examination as shown in Figure 0-6.

Intact Coupling Couplings 11-P7C-5 and 11-P7C-7 were split in half in the longitudinal direction to examine for cracks or other discontinuities by fluorescent magnetic particle testing (MT). MT did not reveal any indications on coupling 11-P7C-5. MT revealed an indication, observed as a well-defined bright fluorescent line, at the thread root near the shaft alignment hole of coupling 11-P7C-7 as presented in Figure 0-7. The MT indication in coupling 11-P7C-7 is approximately 0.86 in length around the inner circumference along the thread root.

Metallurgical and Scanning Electron Microscopy A longitudinal specimen cut through the 11-P7C-7 indication observed by MT was prepared by mounting in plastic, ground and polished for metallographic examination. Figure 0-8 shows, in the as-polished condition, a branching crack initiating from a pit in the thread root. The branching network of cracks is a typical feature of stress corrosion cracking (SCC).

Next, the specimen was suitably etched to reveal micro-structural details.

The microstructure in the vicinity of the branched crack, as shown in Figure 0-9, reveals the intergranular nature of the network of cracks, which is characteristic of intergranular SCC (IGSCC). The general microstructure of the 11-P7C-7, as shown in Figure 0-9, was observed to be tempered martensite. This material can be susceptible to SCC.

The fracture surface of the 11-P7C-6F and 09-P7C-7F were examined in a scanning electron microscope (SEM). The threaded side of the fracture was

Attachment V: LPI Report Report No. F11358-R-001 Page 110 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering cleaned in an ultrasonic bath containing an acetone/methanol mixture. The cleaned section was examined in a SEM at 20 kV accelerating potential.

SEM examination revealed the fracture surface morphology to exhibit a rock-candy appearance, characteristic of intergranular stress corrosion cracking (IGSCC) as shown in Figure 0-10. This is typical for a quench and tempered steel, such as a 400 series martensitic steel.

Tensile Test Tensile specimens were prepared for couplings 11-P7C-5, 11-P7C-6F and 11-P7C-7. The results of the tensile test on the specimens are documented in LPI Form LPI-13.1-Rev-3-Att-A-Tensile Test and included in Attachment B. The tensile test results are summarized in the Table 0-3. The yield and tensile stresses are consistent with ASTM A582 Type 416 stainless steel

[6].

Composition of Base Metal The composition of the base metal for the 09-P7C-7F, 11-P7C-5, 11-P7C-6F and 11-P7C-7 were evaluated by chemical analysis and the results are provided in Table 0-4. The composition of all tested couplings is consistent with the chemical requirements of ASTM A582 Type 416 stainless steel [6].

Composition of Surface Deposit Deposits on the 11-P7C-6F and 09-P7C-7F fracture surfaces were analyzed by energy dispersive x-ray spectroscopy (EDS) in the SEM prior to cleaning. As shown in Figure 0-11, the spectrum contained large peaks for iron, chromium, manganese, and silicon from the base material of the coupling. Note the high chromium level in the spectrum shown in Figure 0-11. The high level is attributed to a local concentration of chromium carbine in the EDS sampling volume. An additional EDS spectrum, taken over a larger area of the fracture surface, is provided in Figure 0-12. Also exhibited was a large peak for oxygen and smaller peak of chlorine and sulfur, indicating that corrosion products consisted of oxides, chlorides and sulfides. The presence of chlorine from the environment is known to be a primary cause of SCC in stainless steel.

Hardness Surveys Surface Hardness Surface hardness survey was performed in accordance with the requirements of ASTM E18-07 [7] on couplings 1 through 7 extracted from SWS P-7C following the 2011 event and coupling 09-P7C-7F. Results of

Attachment V: LPI Report Report No. F11358-R-001 Page 111 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering the hardness survey were documented in form LPI-13.1-Rev-3-Att-A-Hardness Survey. The completed survey forms are presented in Attachment C. Surface hardness was measured on the top and bottom ends of each coupling, unless noted otherwise. Results of Rockwell C hardness (HRC) measurements are summarized in Table 0-5.

The required hardness for an intermediate tempered 416 stainless steel meeting specification ASTM Standard A582 is between 24 HRC and 32 HRC (248 HB and 302 HB). Based on the surface hardness results in Table 0-5, five couplings (11-P7C-4, 11-P7C-6F, 11-P7C-7, 11-P7C-8, and 09-P7C-7F) exhibit surface hardness above specification, and one coupling (11-P7C-3) exhibits surface hardness at the upper limit of the specification.

Thru-wall Hardness Through thickness hardness was measured near the center of each coupling from the outer diameter (OD) to the inner diameter (ID) of couplings 11-P7C-5, 11-P7C-6F, and 11-P7C-7 at two diametrically opposite locations. Results of the through wall Rockwell C hardness measurements are presented in Table 0-6.

Charpy V-Notch (CVN) Impact Testing Charpy V-Notch (CVN) impact test specimens with radial notches facing the inner diameter were machined from couplings 11-P7C-5, 11-P7C-6F, 11-P7C-7 and 09-P7C-7F. Impact testing was performed on the coupling material in accordance with ASTM Standard E23 [9] over a temperature range of 32°F to 152°F. Results of impact testing are given in Table 0-7. A plot of the CVN data is provided in Figure 0-13.

No requirements for CVN impact test absorbed energy are specified in ASTM Standard A582. Nevertheless, the impact tests reveal low absorbed energy that indicates the coupling material to be notch sensitive under dynamic loading conditions. The low facture toughness of the couplings makes them susceptible to stress corrosion cracking (SCC) under the right environment and subject to tensile stress that would not otherwise fracture the material. For SCC to occur, three criteria to promote SCC must exist; 1) susceptible material, 2) corrosive environment and 3) tensile stress. The specified coupling material, ASTM 582 Type 416 stainless steel, is a martensitic steel that is susceptible to SCC at low toughness.

Attachment V: LPI Report Report No. F11358-R-001 Page 112 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Table 0-1: Measurement of Coupling Outside Dimensions Coupling Length (in.)

OD (in.)

11-P7C-1 6.000 3.186 11-P7C-2 6.002 3.190 11-P7C-3 5.998 3.187 11-P7C-4 5.995 3.189 11-P7C-5 6.000 3.187 11-P7C-6F (a) 3.186 11-P7C-7 5.997 3.187 11-P7C-8 5.998 3.187 09-P7C-7F (b) 3.187 Notes: (a) coupling No. 6 was fractured near the center of the length and measurements of fractured pieces are described in the text, (b) only one half of the fractured coupling No. 09-P7C-7F was sent to LPI and the measurements of this piece of coupling are described in the text.

Table 0-2: Measurement of Coupling Wall Thickness Wall thickness (Motor End)

Wall Thickness (Impeller End)

Coupling t1 t2 t3 t4 t1 t2 t3 t4 11-P7C-1 0.566 0.566 0.565 0.567 0.568 0.566 0.565 0.566 11-P7C-2 0.567 0.568 0.567 0.569 0.566 0.565 0.567 0.567 11-P7C-3 0.567 0.567 0.568 0.567 0.571 0.566 0.570 0.569 11-P7C-4 0.574 0.567 0.569 0.569 0.568 0.569 0.571 0.569 11-P7C-5 0.571 0.570 0.570 N/A 0.568 0.570 0.572 N/A 11-P7C-6F (a)

N/A N/A N/A N/A N/A 0.568 0.569 N/A 11-P7C-7 0.572 0.571 0.569 0.568 0.568 0.570 0.569 0.570 11-P7C-8 0.563 0.566 0.567 0.567 0.567 0.566 0.564 0.566 09-P7C-7F (b)

N/A N/A N/A N/A 0.569 0.569 0.568 0.567 Notes: (a) the top portion of fractured coupling No. 6 was not removed from shaft, (b) only the bottom half of fractured coupling 09-P7C-7F was available.

Table 0-3: Tensile Test Results Coupling Specimen Identification Yield Strength (ksi)

Tensile Strength (ksi)

Elongation

(%)

5-1 134 148 17.9 11-P7C-5 5-2 131 147 16.2 6-1 139 155 16.7 11-P7C-6F 6-2 142 155 15.7 7-1 138 151 13.3 11-P7C-7 7-2 137 152 15.5

Attachment V: LPI Report Report No. F11358-R-001 Page 113 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Table 0-4: Metal Composition of Couplings (Wt. %)

Coupling No.

Element 11-P7C-5 11-P7C-6F 11-P7C-7 09-P7C-7F ASTM A582 TP 416 [6]

C 0.10 0.12 0.11 0.12 0.15 max Cr 12.93 12.90 12.92 12.38 12.00 - 14.00 Cu 0.16 0.16 0.16 0.12 ns Mn 1.09 0.85 068 1.13 1.25 max Mo 0.03 0.03 0.03 0.05 0.60 max Ni 0.14 0.14 0.14 0.19 ns P

0.007 0.015 0.020 0.41 0.060 max S

0.51 0.36 0.34 0.32 0.15 min Si 0.23 0.23 0.25 0.46 1.00 max ns - not specified

Attachment V: LPI Report Report No. F11358-R-001 Page 114 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Table 0-5: Surface Hardness Survey of Couplings Coupling End Average (HRC)

Measurements (HRC) top 27.9 26.5, 28.0, 26.0, 28.6, 29.8, 28.2 11-P7C-3 bottom 31.3 32.1, 31.5, 32.0, 29.9, 31.0, 31.0 top 31.5 30.0, 33.6, 29.4, 31.5, 30.6, 33.8 11-P7C-4 bottom 30.8 29.7, 28.1, 30.8, 31.8, 32.1, 32.1 top 29.7 31.1, 29.6, 29.6, 30.0, 29.0, 29.0, 29.1, 30.0 11-P7C-5 bottom 29.6 28.9, 29.5, 29.4, 29.0, 29.9, 30.9, 30.5, 27.9, 30.2 top (a) 11-P7C-6F bottom 33.3 33.1, 33.0, 33.1, 33.0, 33.1, 33.5, 33.6, 33.6 top 32.2 31.5, 31.9, 32.0, 32.2, 32.6, 32.2, 32.2, 32.6 11-P7C-7 bottom 30.6 30.6, 31.1, 31.3, 28.7, 30.0, 31.4, 31.0, 31.0 top 32.2 32.0, 31.8, 31.4, 32.0, 33.0, 32.7 11-P7C-8 bottom (b) top 32.1 33.7, 33.1, 32.8, 32.0, 30.2, 31.8, 31.2 09-P7C-7F bottom (c)

Notes: (a) top side of coupling No. 6 was kept in its as-received position on shaft No. 6, (b) deposits on the bottom of coupling No. 8 were kept intact and prevented hardness testing of the underlying base metal, (c) no bottom section of coupling 09-P7C-7F was received.

Table 0-6: Through Thickness Hardness of Couplings Coupling Location Measurements from OD to ID (HRC) 1 27.2, 28.0, 28.0, 27.7, 28.0, 27.1 11-P7C-5 2

31.5, 30.9, 30.4, 30.2, 30.2, 30.7 1

31.5, 32.7, 32.0, 32.1, 32.2, 32.0 11-P7C-6F 2

31.5, 32.2, 31.9, 32.1, 31.5 1

31.2, 32.0, 31.7, 31.9, 31.3, 31.8 11-P7C-7 2

32.0, 32.0, 32.0, 31.9, 32.6, 32.0

Attachment V: LPI Report Report No. F11358-R-001 Page 115 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Sample through thickness hardness test specimen.

Table 0-7: CVN Impact Test Results Coupling Specimen Identification Test Temperature

(°F)

Absorbed Energy (ft-lb)

Lateral Expansion (in.)

Percent Shear

(%)

5-C2 32 9

0.005

<10 5-C4 32 9

0.006

<10 5-C1 70 10 0.007 10 5-C6 70 10 0.007 10 5-C8 70 10 0.007 10 5-C3 100 11 0.007 20 5-C5 100 10 0.006 20 11-P7C-5 5-C7 150 15 0.011 50 6-C2 32 6

0.003

<10 6-C4 32 8

0.006

<10 6-C1 70 9

0.005 10 6-C5 70 10 0.006 10 6-C3 100 11 0.007 10 11-P7C-6F 6-C6 150 14 0.008 50 7-C2 32 10 0.003

<10 7-C1 75 8

0.008 10 11-P7C-7 7-C3 100 11 0.008 10 709-C3 32 4

0.004

<10 709-C4 32 3

0.004

<10 709-C1 75 5

0.005

<10 709-C2 75 6

0.002

<10 709-C5 100 6

0.003

<10 09-P7C-7F 709-C6 152 6

0.006

<10

Attachment V: LPI Report Report No. F11358-R-001 Page 116 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Coupling No. 7 in as-received form. Shaft numbers engraved on the ends.

Coupling exhibited wrenching marks on outer diameter.

Figure 0-1: As-Received 11-P7C-7 Figure 0-2: Coupling Dimensioning Scheme Fracture surface was removed from the bulk of the coupling for analysis.

Elliptical pattern of crack on flat fracture surface, relative to coupling axis. Also, red/brown corrosion

Attachment V: LPI Report Report No. F11358-R-001 Page 117 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Fractured end exhibited mechanical damage (arrow).

product found on the insides of coupling threads (arrow)

Figure 0-3: Visual of Fracture Surface on Coupling 11-P7C-6F Figure 0-4: 11-P7C-6F showing Corrosion Deposit

Attachment V: LPI Report Report No. F11358-R-001 Page 118 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Half of coupling 11-P7C-6F remaining on shaft No. 6 exhibited damage on its fracture surface from contact with the mating coupling half and shaft 5, which occurred after fracture event.

End of shaft No. 5 exhibited gouging damage post 11-P7C-6F final failure.

Attachment V: LPI Report Report No. F11358-R-001 Page 119 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-5: Ends of Shaft 5 and 6 Coupled by 11-P7C-6F After fracture surface was sectioned, a remaining portion of coupling 11-P7C-6F was MT inspected. Arrows show location of indications.

Figure 0-6: MT Highlighting Un-Opened Fracture on Coupling 11-P7C-6F

Attachment V: LPI Report Report No. F11358-R-001 Page 120 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Top sketch show the cut line for coupling 11-P7C-7. Fluorescent MT examination of this coupling reveals an indication, shown by a well-defined fluorescent line, initiating from a thread root and propagating in the radial direction. The left and right images show the same indication on the two sections of this coupling after cutting in half longitudinally. Arrows show location of the indication.

Figure 0-7: MT Highlighting Crack on Coupling 11-P7C-7

Attachment V: LPI Report Report No. F11358-R-001 Page 121 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Branched cracking of the indication found in 11-P7C-7. Specimen in the as-polished condition and viewed at 50x.

Figure 0-8: As-Polished of Coupling 11-P7C-7 Specimen

Attachment V: LPI Report Report No. F11358-R-001 Page 122 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Etched specimen containing indication in coupling 11-P7C-7, showing microstructural features that reveal the intergranular nature of the branched cracking. Viewed at 200x.

General microstructure of coupling 11-P7C-7 is tempered martensite. Specimen etched and viewed at 400x.

Attachment V: LPI Report Report No. F11358-R-001 Page 123 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-9: Micro-Structure of Coupling 11-P7C-7 Specimen Fracture surface morphology exhibited a rock-candy appearance, characteristic of intergranular stress corrosion cracking. This is typical of a quench and tempered steel, such as a 400 series martensitic steel.

Figure 0-10: SEM of Coupling 11-P7C-6F Surface EDS analysis of fracture surface revealed the presence of corrosive agents (chlorides, oxides and sulfides), consistent with stress corrosion cracking. Note the high chromium level in the

Attachment V: LPI Report Report No. F11358-R-001 Page 124 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering spectrum. The high amount could be attributed to a local concentration of chromium carbine in the EDS sampling volume.

Figure 0-11: EDS of Coupling 11-P7C-6F Surface Deposit - Spectrum 2 Figure 0-12: EDS of Coupling 11-P7C-6F Surface Deposit - Spectrum 4

Attachment V: LPI Report Report No. F11358-R-001 Page 125 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-13: Charpy Test Result

Attachment V: LPI Report Report No. F11358-R-001 Page 126 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering EVALUATION Metallurgical analysis of the 11-P7C-6F coupling, documented in Section 3.0, identified the failure mechanism to be intergranular stress corrosion cracking (IGSCC). Stress corrosion cracking is defined as a failure of a material subjected to tensile stress in a corrosive environment in which the material is susceptible.

Each of the three criteria 1) susceptible material, 2) corrosive environment and 3) tensile stress; considered to be necessary for SCC to occur is discussed in the following subsection.

Susceptible Material The coupling material was specified to be ASTM A582 Type 416 SS with hardness in the Rockwell C range of 28 to 32 (HRC). ASTM A582 Type 416 SS is a martensitic stainless steel that has excellent machining characteristics and has generally low corrosion resistance due to its relatively high sulfur content.

Based on heat traces provided in Attachment A for the couplings currently installed in P-7A, P-7B and P-7C and the couplings extracted from P-7C post 2011 failure event, the couplings were hardened by quenching from approximately 1870°F using nitrogen and then air cooled. Tempering to achieve the desired hardness of 28 to 32HRC was performed at temperatures ranging from 1025°F to 1090°F. In some cases, a second temper was required to achieve the desired hardness. Plots of the SWS pump coupling heat treatment (hardening and tempering) are provided in Figure 0-1 and Figure 0-2.

The tempering temperature of the material can have an adverse effect on the toughness and corrosion resistance of the material. Based on tempering curves for the batch of couplings installed in P-7A, P-7B and P-7C (Figure 0-2), the tempering temperatures are in the range to be avoided between 400°C and 580°C (752°F to 1076°F) for 416SS. These tempering temperatures can lead to low toughness and susceptibility of the material to SCC. In fact, low impact toughness values (indicated by the CVN) are seen in the couplings that have failed with most CVN values in the single digits at the temperature range of the service water (refer to Table 0-7).

Attachment V: LPI Report Report No. F11358-R-001 Page 127 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Corrosive Environment The Service Water System (SWS) takes cooling water from Lake Michigan via pumps P-7A, P-7B and P-7C for the removal of waste and decay heat.

For the period between January 2009 to August of 2011, service water basin level ranged from elevation 576 to 580. For the same period, the water temperature ranged from a minimum of 32°F to a maximum of 76°F.

Chlorination occurs on a daily basis and consists of the addition of sodium hypochlorate (i.e. bleach) upstream of the traveling screens to control microbial species in the SWS. A water sample was taken by the Palisades chemistry department on 8/19/11 downstream of YS-0134 in the Chemistry cold lab prior and post chlorination of the Service Water System. The chemistry water sample data is presented below.

Pre Chlorination Post Chlorination Date/Time 8/19/2011 14:10 8/19/2011 18:48 Chlorination in Progress No Yes Temperature 22.8 C 22.5 C PH 8.30 8.21 Dissolved Oxygen 9 ppm 10 ppm Chloride Concentration 9.72 ppm 10.2 ppm This data indicates there is sufficient chlorine and dissolved oxygen in the service water for SCC of 416 SS to occur, even when chlorination is not in progress. Also due to the intermittent nature of the pump service the couplings above the normal water basin will experience wet /dry cycles that will leave a higher chlorine concentration on the couplings as the coupling dries out with pump stopped. Visual examination of the failed coupling shows corrosion products staining the area below the vent hole and are present on the fracture surface and on the internal threads.

Tensile Stress The fracture surface revealed that the 11-P7C-6F failed due to stress corrosion cracking from the inner diameter at the thread root to the outer diameter from two initiation sites traversing the thickness in an elliptical manner. To support this failure mechanism, an evaluation of the coupling stresses was performed to determine the tensile stress in the coupling.

Attachment V: LPI Report Report No. F11358-R-001 Page 128 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering The function of the couplings is to couple the various segments of shafts (i.e. line shafts, packing shaft and motor shaft) together in order to transmit the motor torque to the impeller approximately 40 feet below. The design of the couplings enables the shaft ends to bear against each other that could lead to both tensile stresses and shear stresses across the coupling. To determine the stresses across the couplings, a finite element analysis (FEA) model of the coupling was created in ANSYS [12]. ANSYS is a multipurpose finite element analysis software program and is verified and validated in accordance with LPI Procedure 4.1 [15], as documented within

[16].

FEA Model Description A half FEA model of an intact coupling was developed using ANSYS and consists of the steel body, alignment hole and threads. The model was constructed of the eight-node brick element, SOLID45 (see Figure 0-5). The symmetric boundary condition, Uz=0 and U=0, is applied on the inner surface as shown in Figure 0-6.

ASTM A582 Type 416 stainless steel material property for the coupling FEA model is as follows:

Youngs modulus: 29.2 x 106 psi Poissons ratio: 0.3 Coupling threads are 2-3/16, 8 TPI (see Figure 0-5) which is not a common thread form. Specific thread properties are not available in the Machinerys Handbook [13]. Therefore, internal thread properties of the coupling is taken to be the average internal diameter of 2-1/4, 8 TPI and 2-1/16, 8 TPI in the Machinerys Handbook [13].

Loading Condition Loading on the coupling model consists of the weight of components below the coupling, hydraulic thrust and motor torque. These loads are extracted from HydroAire calculation NQ5940 [14] as follows:

Two motor torque loading scenarios (MTS) are considered for transmittal of the motor torque across the coupling; 1) motor torque is transmitted across the coupling by shaft to coupling purely by thread friction (MTS1; see Figure 0-3) and 2) motor torque is transmitted across the coupling by bearing of

Attachment V: LPI Report Report No. F11358-R-001 Page 129 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering the shaft ends against each other within the coupling (MTS2; Figure 0-4).

To simulate uneven shaft alignment within the coupling, a bending moment is also considered as a load. These loads are combined as follows for evaluation of the couplings.

Load Combination 1 (LC1) = Weight + Thrust + MTS1 For this load combination, component weight and hydraulic thrust is combined with the motor torque loading scenario 1 (MTS1) in which motor torque is transmitted across the coupling purely by friction. Axial thrust, F=8780lb, is evenly distribute on the nodes on the inner surfaces of each thread (see Figure 0-7). Torque, T=18694 in-lb [14], is first converted into circumferential force, F by T= F*R where R is the coupling friction radius and then evenly applied on the same nodes that the axial thrust load is applied.

Load Combination 2 (LC2) = Weight + Thrust + MTS2 For this load combination, the weight and axial thrust is applied in the same manner as in LC1. Bearing of the shafts within the coupling will induce tensile stress across the coupling. The tensile force of 42 kips is evenly distributed to the first three threads from the contact plane of the two shafts (see Figure 0-8). Typically with threaded connections, the first few threads near the plane of induced load carry the majority of this load

[17]. For this assessment, the first three threads were considered to carry the load.

Load Combination 3 (LC3) = Weight + Thrust + MTS2 + Moment For this load combination, loads are applied in the same manner as LC2 with the addition of a moment on the coupling to account for misalignment or other postulated scenarios that can induce bending across the coupling. A bending moment equivalent to 20% of the stress induced by MTS2 of approximately 4,962 in-lb (see below) was also applied to the coupling. This moment was converted into axial force, Fz, and applied on the nodes on the end cross-section based on the nodes y direction distance from center (see Figure 0-9).

Attachment V: LPI Report Report No. F11358-R-001 Page 130 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering FEA Results

1) LC1: In this case, the circumferential stress, axial stress and first principal stress are relatively low due to the even distribution of loads on the coupling (see Figure 0-10). Axial tensile stresses at the thread root are on the order of 3.5 to 5 ksi. This result indicates that if the motor torque is transmitted across the coupling purely by thread frictional resistance, then the coupling tensile stresses are relatively low.

2) LC2: This load combination results in high stress concentrations at the thread root of the coupling at the contact plane of the two shafts. Axial tensile stresses at the thread root are over 10 times greater than LC1 with stresses on the order of 50 to 60 ksi.

3) LC3: This load combination does not significantly increase stresses at the thread root of the coupling from LC2. The additional bending moment on the coupling produces additional stresses on the outside diameter of the coupling however does not appreciably increase stresses at the thread root where SCC initiation and propagation is postulated.

Average tensile stresses at the first thread root for each load combination is summarized in the matrix below.

Attachment V: LPI Report Report No. F11358-R-001 Page 131 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Load Combination Average Tensile Stress (psi)

LC1: No Bearing 3790 LC2: Shaft Bearing 58250 LC3: Bearing Shaft and Bending 58220 The stress tabulation above indicates that the failure was not a single overload event since the average yield and tensile strength are approximately 136 ksi and 151 ksi (see Section 0), respectively. The typical stress intensity required to initiate a crack at a notch due to SCC is on the order of dependant upon material and environment. Clearly from the FEA, sufficient tensile stress is present to facilitate crack initiation for the load combination involving shaft end bearing.

Crack Propagation Given the tensile stresses for the three load combinations evaluated in Section 4.3 above, a crack propagation evaluation is performed in this section to estimate the amount of time to propagate a crack through coupling 11-P7C-7.

WORK IN PROGRESS

Attachment V: LPI Report Report No. F11358-R-001 Page 132 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-1: Hardening Heat Traces

Attachment V: LPI Report Report No. F11358-R-001 Page 133 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-2: Tempering Heat Traces Figure 0-3: MTS1: Shaft Not Bearing

Attachment V: LPI Report Report No. F11358-R-001 Page 134 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-4: MTS1: Shaft Bearing Figure 0-5: Half FEA model of coupling

Attachment V: LPI Report Report No. F11358-R-001 Page 135 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-6: Cross-section of half FEA coupling model Z

X Y

Local coordinate system numbered 11 is cylindrical coordinate system Symmetric boundary condition: Uz=0 U=0

Attachment V: LPI Report Report No. F11358-R-001 Page 136 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-7: Load application Sketch of loading condition in no bearing case Figure 0-8: Sketch of loading condition in shafts bearing case Inner side of thread face to hole F and Fp are applied on the nodes on the inner side of thress threads which are the most close to the hole

Attachment V: LPI Report Report No. F11358-R-001 Page 137 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-9: Sketch of axial force result from bending moment

Attachment V: LPI Report Report No. F11358-R-001 Page 138 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Circumferential Stress 1st Principal Stress Axial Stress

Attachment V: LPI Report Report No. F11358-R-001 Page 139 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Figure 0-10: Resultant stresses for LC1 Circumferential Stress 1st Principal Stress

Attachment V: LPI Report Report No. F11358-R-001 Page 140 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Axial Stress Figure 0-11: Resultant stresses for LC2 Circumferential Stress 1st Principal Stress Axial Stress Figure 0-12: Resultant stresses for LC3

Attachment V: LPI Report Report No. F11358-R-001 Page 141 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

SUMMARY

/RECOMMENDATION Palisades SWS pump P-7C coupling #6 (identified herein as 11-P7C-6F) failed in August, 2011. The failure is determined, based on metallurgical evaluation, to be the result of intergranular stress corrosion cracking (IGSCC). The 2009 failure of the #7 coupling (identified herein as 09-P7C-7F) on the same pump (P-7C) was determined in [18] to also be a result of IGSCC. LPIs independent examination of the 2009 failed coupling 09-P7C-7F concurs with the failure mode as documented in [18].

For SCC to occur three criteria to promote SCC must exist; 1) susceptible material, 2) corrosive environment and 3) tensile stress. The specified coupling material, ASTM A582 Type 416 stainless steel, is a martensitic steel that is susceptible to SCC at low toughness. Charpy V-Notch (CVN) testing of the 2011 failed coupling (11-P7C-6F) resulted in toughness values in the range of 6 to 10 ft-lbs impact energy for test temperatures of 32 and 70F. CVN testing of the 09-P7C-7F coupling resulted in impact toughness values in the range of 3 to 6 ft-lb for test temperatures of 32°F and 75°F, respectively. These low impact toughness values make the couplings susceptible to SCC in the presence of chlorides and sufficient tensile stress to initiate and propagate a crack.

The couplings are subjected to tensile stresses during normal operation by the weight of the components below the coupling and hydrodynamic forces due to pump operation. In addition, the design of the couplings results in the shaft ends bearing against each other that likely led to sufficient tensile stresses (with a maximum value near the center where the two shafts bear against each other) in the coupling to initiate and propagate a crack.

The majority of the pump couplings below the packing (couplings #1 through #4) are submerged below the water level in the intake structure at normal basin levels. Couplings #5 through #7, above normal basin water levels see intermittent cycles of wet and dry depending on whether the pump is operating. When the SW pumps are on, all couplings below the stuffing box are wet and when they are off, couplings #52, #6 and #7 begin to dry. Chemistry samples of the service water indicate that there are low levels of chlorine in the raw water of Lake Michigan on the order of 9 ppm. Chlorination of the service water increases the chlorine level slightly to approximately 10 ppm. Even these relatively low levels of chlorine combined with a high humidity oxygen rich environment (as is the case for the couplings #5, #6 and #7 when the pump is off) can lead to a local 2 Unless the service water basin level is above coupling #5. In which case, coupling #5 would be submerged in water.

Attachment V: LPI Report Report No. F11358-R-001 Page 142 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering breakdown of the passivation layer. SCC can nucleate at these locally damaged sites, develop and propagate under sufficient tensile stress to form highly branched network of fine cracks, as can be seen in Figure 0-8 and Figure 0-9.

REFERENCES

1. Entergy Palisades Root Cause Evaluation for CR-PLP-2009-04519, Service Water Pump P-7C Failure to Provide Discharge Pressure 3/4/2010, Rev. 1

2. Entergy Palisades Root Cause Evaluation Report for CR-PLP-2011-03902, Service Water Pump 7-C Line Shaft Coupling Failure

3. Entergy Palisades Drawing

a. Entergy Drawing No.M-213, Rev. 93 Piping and Instrumentation Diagram Service Water, Screen Structure and Chlorinator

b. Entergy Drawing No. VEN-M-11-Sht 45, Rev. 2 Service Water Pumps Modified Line Shaft Components For Pumps P-7A, P-7B and P-7C

4. Entergy Engineering Change (EC)-50000121762

5. LPI Procedure F11358-P-001, Rev. 0, Procedure for Metallurgical Examination of SW Pump P-7C Coupling Components

6. ASTM A582/A582M, Standard Specification for Free-Machining Stainless Steel Bars, 2005.

7. ASTM E 18-07, Standard Test Method for Rockwell Hardness of Metallic Materials

8. ASTM E8/E8M-09, Standard Test Methods for Tension Testing of Metallic Materials

9. ASTM E23-07ae1 Standard test Methods for Notched Bar Impact Testing of Metallic Materials

10. ASTM E3-11, Standard Guide for Preparation of Metallographic Specimens

11. ASTM E407-99, Standard Practice for Microetching Metals and Alloys

12. ANSYS Mechanical Software, ANSYS Inc., Southpoint, Canonsburg, PA 15317

13. OBERG E, et al. Machinerys Handbook 25th Ed. Industrial Press

14. HydroAire Calculation NQ5940, Maximum combined shear stress calculation for threaded coupling, Rev. 3

15. LPI Procedure 4.1, Rev. 2 Software Control

16. ANSYS Inc., LPI Report No. V&V-ANSYS-11, Rev. 3,Verification and Validation of ANSYS

17. Shigleys, Mchanical Engineering Design McGraw Hill

18. Structural Integrity Report 1100112.401, Additional Review of Palisades Service Water Pump Couplings, Rev. 0 March 2011.

Attachment V: LPI Report Report No. F11358-R-001 Page 143 of 136 Revision DRAFT G Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

19. HydroAire Inc. Drawing No. 1047237 Line Shaft Coupling Johnstone 25 NMC 2 STG Sht. 1/1, Rev. 0

20. Flowserve Materials Newsletter Volume 3, Number 4 Embrittlement of Martensitic Stainless Steels, September 2004.

Attachment V: LPI Report Report No. F11358-R-001 Attachment A Revision 0 Page A1 of Axx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Attachment A Misc. Inputs

Attachment V: LPI Report Report No. F11358-R-001 Attachment A Revision 0 Page A2 of Axx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

Attachment V: LPI Report Report No. F11358-R-001 Attachment A Revision 0 Page A3 of Axx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

Attachment V: LPI Report Report No. F11358-R-001 Attachment A Revision 0 Page A4 of Axx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

Attachment V: LPI Report Report No. F11358-R-001 Attachment A Revision 0 Page A5 of Axx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering

Attachment V: LPI Report Report No. F11358-R-001 Attachment B Revision 0 Page B1 of Bxx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Attachment B Tensile Test Data

Attachment V: LPI Report Report No. F11358-R-001 Attachment C Revision 0 Page C1 of Cxx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering Attachment C Hardness Survey Data

Attachment V: LPI Report Report No. F11358-R-001 Attachment C Revision 0 Page C1 of Cxx Lucius Pitkin, Inc.

Consulting Engineers Advanced Analysis Fitness-For-Service Failure & Materials Evaluation Nondestructive Engineering