ML120100494
| ML120100494 | |
| Person / Time | |
|---|---|
| Site: | Palisades  |
| Issue date: | 01/05/2012 |
| From: | Vitale A Entergy Nuclear Operations |
| To: | Office of Nuclear Reactor Regulation, Document Control Desk |
| Shared Package | |
| ML1200100495 | List: |
| References | |
| EA-11-241, EA-11-243, PNP 2012-06 CR-PLP-2011-03902 | |
| Download: ML120100494 (53) | |
Text
is
. Entergy Entergy Nuclear Operations, Inc.
Palisades Nuclear Plant 27780 Blue Star Memorial Highway Covert, MI 49043 Tel 269 764 2000 Anthony J . Vitale Site Vice President PNP 2012-06 EA-11-241 EA-1 1-243 January 05, 2012 U.S. Nuclear Regulatory Commission ATTN: Document Control Desk Washington , DC 20555
Subject:
Regulatory Conferences Supporting Documentation for Apparent Violations EA-11-241 and EA-11-243 Palisades Nuclear Plant Docket 50-255 License No. DPR-20
References:
- 1. Letter from Nuclear Regulatory Commission to Entergy Nuclear Operations, Inc. dated November 29, 2011, "Palisades Nuclear Plant, NRC Inspection Report 05000255/2011016 Preliminary White Finding"
- 2. Letter from Nuclear Regulatory Commission to Entergy Nuclear Operations, Inc. dated November 29, 2011, "Palisades Nuclear Plant - NRC Special Inspection Team (SIT) Report 05000255/2011014 Preliminary Yellow Finding"
- 3. Letter from Entergy Nuclear Operations, Inc. to Nuclear Regulatory Commission to dated December 8, 2011, "10-Day Response to IR 2011014 Preliminary Yellow Finding - DC Bus Failure and Plant Trip"
- 4. Letter from Entergy Nuclear Operations, Inc. to Nuclear Regulatory Commission to dated December 8, 2011, "10-Day Response to IR 2011016 Preliminary White Finding - Service Water Pump Coupling Failure"
Dear Sir or Madam:
Entergy Nuclear Operations, Inc, (ENO) received the referenced letters dated November 29, 2011, related to the failure of a DC Bus, causing a plant trip on September 25, 2011 and the failure of a service water pump coupling on August 9, 2011. By referenced letters dated December 8, 2011, ENO requested regulatory conferences to address the apparent violations.
The requested supporting documentation for the regulatory conferences is enclosed.
Document Control Desk PNP 2012-06 Page 2 This letter contains no new commitments and no revisions to existing commitments.
Sincerely, I /ItA AJV/tad k&
Attachments: 1. Root Cause Evaluation-Service Water Pump P-7C Coupling Failure, Rev 0
- 2. SDP Assessment of Service Water Pump P-7C Coupling Failures
- 3. Root Cause Evaluation-Plant Trip During Panel ED-11-2 Maintenance, Rev 2
- 4. SDP Assessment of DC Panel D11-2 Fault CC Administrator, Region III, USNRC Project Manager, Palisades, USNRC Resident Inspector, Palisades, USNRC
Attachment 1 CR-PLP-2011-3902 Root Cause Evaluation - Service Water Pump P-7C Coupling Failure, Rev 0 151 Pages Follow
Entergy Operations, Palisades Root Cause Evaluation Report Service Water Pump 7-C Line Shaft Coupling Failure CR-PLP-2011-03902, EVENT DATE: 08-09-2011 REPORT DATE: 09-08 -2011, Rev. 0 POSITION NAME/DEPT. DATE Paul M . Deniston (PLP, Systems Engineering)
RCE Evaluator 09-08-2011 Jason Gosler (PLP, Mechanical Maintenance)
Technical Reviewer George Licina (Structural Integrity) 09-08-2011 Independent Reviewer Kyle Langston (PLP, Systems Engineering) 09-08-2011 Responsible Manager Jody Haumersen/James Forehand (PLP, System Engineering) 09-08-2011 CARE Chairperson David Hamilton (PLP, GMPO) 09-08-2011 CR-P LP-2011-3902 Page 1 of 151
Problem Statement Service Water Pump P-7C failed unexpectedly at -1202 hrs on 8/9/2011 resulting in entry into Off Normal Procedure 6.1, "Loss of Service Water", and entry into LCO 3.7.8 (72 Hr Shutdown LCO).
RCE Report CR-PLP-2011-03902 Page 2 of 136
Event Narrative
Background
PALISADES SERVICE WATER SYSTEM CONFIGURATION The Service Water System (SWS) at Palisades is comprised of three motor driven vertical multistage pumps supplying water from Lake Michigan to three service water headers. Two of the headers are termed critical headers A and B, which provide cooling to safety and non-safety related components. Each critical header supplies cooling water to one set of the redundant components including emergency diesel generator lube oil and jacket water coolers, a control room air-conditioning unit, an air compressor after-cooler and an engineered safeguards room cooler. In addition, critical header A supplies cooling water to the component cooling water heat exchangers while critical header B supplies cooling water to the containment air coolers. For accident conditions, either train fed by its associated diesel, is sufficient for accident mitigation. The third header is termed non-critical and provides cooling to non-safety related equipment.
Palisades Technical Specifications require that all three pumps be operable. The failure of a single pump requires entry into a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> shutdown LCO Action Statement.
A single header combining return streams from the three supply headers discharges into the cooling tower makeup basin. Leakage of radioactive contamination into the SWS is detected by a radiation monitor installed in the discharge line.
The three Service Water Pumps (SWPs), P-7A, P-7B, and P-7C, are modified Layne and Bowler pumps. They are comprised of a two stage pump end with stainless steel impellers connected to a discharge head by seven columns for a total height of over 40 feet from suction to discharge. The pump end is coupled to the motor through six line shafts, a packing shaft, and a motor shaft connected by eight couplings all of the same design. Figure 1, below, gives a visual representation of the SWPs. Figure 2 shows a shaft coupling.
RCE Report CR-P LP-2011-03902 Page 3 of 136
Event Narrative Motor Support Motor Shaft Packing Shaft Discharge Head Gland Follower Fl ow Out Coupling #7 Stuffing Box Stuffing Box Bearing EL. 590'-7" Top Column\
Packing Shaft Coupling #7 Mounting Bracket Spider #6 Spider #6 Top Column Pipe Line Shaft Bearin Line Shaft #6 Line Shaft Bearin!
Coupling #6 Sleeve 2009 Spider #5 Failure Line Shaft #8 Line Shaft #5 EL. 580' 2011 Coupling #6 Middle Column Water Level 0 Failure Spider #4 Range Line Shaft #4 EL. 676' Coupling 46 Coupling #4 Spider #03 Spider #5 Middle Column Li ne Sh aft #3 Pipes Coupling #3 Spider #2 Line Shaft #5 Line Shaft #2 Middle Column Coupling #2 Spider #1 EL. 580' Line Shaft #1 Lower Column Top of Water Coupling #1 Pipe Level Range haft Nozzle Transition Coupling #5 Split P u mp tSR ag 2nd Stage Impeller Top Pump Bowl ----Spider #4 Split Thrust Ring Bottom Pump Middle Column 1st Stage Impeller - Bowl Suction Nozzle Line Shaft #4 Flow In Suction Nozzle Bearing Figure 1: PLP SWP Rendering Showing 9/29/09 and 8/9/11 Coupling Failure Locations RCE Report CR-P LP-2011-03902 Page 4 of 136
Event Narrative iTYP.} 45'CF1h1^FR
,, I,^I fli -_----- i; (400c) 0 12L^ iCIIAr1 E -^ I ^f ' ^^ f^' - HRLI 111 I) ^IiI { {5k NOTE "2) 45`[TYr_1 HOLE UENTt
}
LINE SHAFT C OUPLING
" i+T U.; fist?hi ^5 ? T P F)_ li5 STA NL `3 S STEEL HA DN .SSa F1^^x ^'^ ,'^1 I N: ? S F?L 18 FFJ a PF R FJr4P!
Figure 2 : Coupling Drawing from VEN M-1 1 Sheet 55 The original SWS pumps were purchased to Specification M-1 1. The three 50% capacity service water pumps were rated at 8000 GPM and 140 ft Total Developed Head (TDH) each. At Palisades in May, 1999, FES-99-001 was approved to change SWS pump line shafts from carbon steel to 416 SS at the recommendation of the vendor at that time, Rotating Equipment Repair. The rationale for the change was improved corrosion resistance and improved material strength.
The specification for line shaft couplings for P-7A, P-7B, and P-7C was changed from carbon steel to 416 SS under EC 5000121762 in December 2007. EC-5000121762 was an update to the SWS configuration to allow the use of stainless steel couplings and shafts. According to the EC, 416 SS was chosen due to its strength, wear resistance and corrosion resistance. The couplings were also redesigned to incorporate an alignment hole that allows visual verification of proper shaft installation. The line shaft couplings for P-7A were replaced per EC-5000121762 under WO 51637416 on April 2009. The line shaft couplings on P-7B were replaced during a rebuild under Palisades PO 10246213 to HydroAire and P-7B was installed under WO 20082 in June 2010. The line shaft couplings for P-7C were changed from carbon steel to stainless steel through the PO 10237148 to HydroAire in June 2009.
Per ASTM Standard A582/A 582M - 95b "Standard Specification for Free-Machining Stainless Steel Bars", the hardness of the material should be between 24 and 32 Rc (Rockwell C Hardness) (248 to 302 HB (Brinell Hardness)) for an intermediate temper condition. The material of the coupling is 416 SS per this ASTM standard, but the hardness has been specified to a range of 28-32 Rc per vendor specification. It should be noted that according to the metallurgists from Bodycote and Structural Integrity, Rc hardness greater than 35 dramatically increase the susceptibility of material to intergranular stress corrosion cracking due to excessive material hardness. A more ductile material (i.e., below 32 Rc) is not subject to this phenomenon to the extent that the overly hardened material is.
RCE Report CR-P LP-2011-03902 Page 5 of 136
Event Narrative The ASTM standard A582/A 582M-95B Standard Specification for Free-Machining Stainless Steel Bars contains the required material properties for 416 SS used as the material for the Service Water Pump couplings. The specific material properties provided for in the specification are chemistry and hardness requirements. There are no toughness or tensile requirements listed provided in the specification. The requirements of the specification were transmitted to HydroAire via Vendor Drawing M0011-Sh-00055 with the stipulation that the couplings provided meet the "Tempered" condition, with an even further restriction that the hardness meet a narrowed 28-32 RC range. Per the ASTM specification the hardness range for 416 stainless in the tempered condition is 248-302 Brinell which equates to a 24.2-32.1 Rc. Palisades chose to limit the hardness range to the upper half of this band in order to mitigate the effects of galling during pump assembly.
Heat treating of 416 stainless steel is covered under the American Society of Metallurgists (ASM) Heat Treater's Guide: Practice and Procedures for Irons and Steels.
For hardening, the guide indicates that the parts should be preheated at 1400-1455°F, austenitized at 1695-1850°F and oil quenched. A review of the provided BodyCote travelers provided for HydroAire Job #5912 (the couplings that were in P-7C when it failed in August 2011) indicate that they were hardened (per the guide).
For tempering 416 stainless steel, the guide provides the following guidance;
"....Temper at 565 to 605°C (1050-1125°F) for hardness approximately 25 to 31 HRC. Tempering at 370-565°C (700-1050°F) not recommended for parts requiring high toughness and optimum corrosion resistance. Causes a marked dip in impact resistance and lowered stress corrosion cracking resistance. Double tempering beneficial. Cool to room temperature between tempers."
A review of the data provided in the BodyCote travelers provided for HydroAire Job
- 5912 (the couplings that were in P-7C when it failed in August 2011), show that two tempers were performed. The first temper was performed at 1075-1100°F, which is in the range suggested by the guide. The second tempers, for all couplings, was performed at 1025°F, which is less than the range suggested by the guide and is in the range where impact resistance and stress corrosion cracking resistance are adversely impacted.
Some metrological terms will be used throughout this report. Provided here is a brief definition for clarity.
- Hardening: The heating of a material to a high temperature (1695-1850°F for 416 SS) in order to harden the material. This significantly reduces toughness and ductility
- Tempering: After hardening and quenching, the material is "tempered" at a temperature specified by the ASTM standard in order to improve ductility and toughness
- Tempering Embrittlement: Refers to the over-tempering of a material to the point where the material becomes brittle at the grain boundaries of the metal. This increases the statistical likelihood of Inter-granular Stress Corrosion Cracking (IGSCC), but d oes not guarantee that IGSCC will occur.
RCE Report CR-P L P-2011-03902 Page 6 of 136
Event Narrative Inter-granular Stress Corrosion Cracking (IGSCC): The condition caused by the presence of a susceptible material, a tensile stress and a corrosive environment.
Cracking occurs when all three conditions are met combined with a pit or flaw in the material. The materials between grain boundaries are eroded thus leading to material cracking. The definition of a corrosive environment is dependent on the specific material in question.
EVENT DESCRIPTION Palisades began receiving Operating Experience regarding 416 SS as early as 1993, NRC IN 93-68.
In May 2004, OE from Perry Station was issued regarding coupling failures due to Intergrannular Stress Corrosion Cracking (IGSCC) on 416 SS. Palisades evaluated this under LO-PLPLO-2007-00059. Under this same LO task, NRC IN 2007-05 was evaluated. Palisades concluded that the IN did not apply because the couplings were made of Carbon Steel rather than 416 SS. The responder to the OE was not the engineer who was pursuing the materials change in the SWS.
In 2007, an Engineering Change, EC 5000121762, was completed to allow the use of 416 SS (annealed) line shafts with 416 SS (Tempered) couplings. This EC referenced the above OE yet failed to acknowledge similarities between 410 and 416 SS.
Additionally, this EC incorrectly concluded that raw Lake Michigan water was not a corrosive environment for 400 series stainless steel. During this investigation, the System Engineer who completed the EC was contacted (he is currently retired). The Engineer indicated that the above referenced EC was used to document changes made to the system over time. He also indicated that he had no formal metallurgical background and did not feel that it was necessary to challenge a historical Specification Change that allowed the use of 416 SS couplings based on an equivalency evaluation suggested by the original pump OEM, Layne and Bowler.
During the spring 2009 refueling outage (1 R20), P-7C failed to meet the performance criteria specified in RO-144 "Comprehensive Pump Test Procedure, Service Water Pumps". This prompted an emergent decision to replace P-7C online in June 2009.
The purchase order history and work orders associated with SWP 7-C are discussed below. The following table gives a condensed list of coupling purchase orders, their tempering times and the associated temperatures.
RCE Report CR-PLP-2011-03902 Page 7 of 136
Event Narrative Single Purchase # or Tempering order Date arts Double Material Min Max Tem p s Minutes Notes 19919 4/22/2008 8 S 416 30 32 1050 180 P-7A was rebuilt using couplings from these batches P-7C was rebuilt using 10237148 5/21/2009 8 S 416 28 32 couplings from this batch, one failed in 9.29.2009 and was found to be 5/22/2009 3 D 28 31 1050 180 excessivel y hard P-7C was rebuilt after the 8/9/2011 failure event with couplings from this 10253715 10/2/2009 10 D 416 28.5 29.5 1075, 1025 240 batch 9/30/2009 2 D 416 29 30 1100, 1025 285 P-7C was rebuilt after the 9/29/2009 failure using 8 couplings from these 9/30/2009 "6+1 D 416 31 31 1070, 1025 340 batches.
Installed in P-7B June 10246213 3/19/2010 8 D 416 28 32 1070, 1090 260 2010 Table 1: Summary of Coupling Heat Treats and Purchase Orders In June 2009, P-7C was replaced with new line shafts and couplings under WO
- 190235. The pump tested satisfactorily during post maintenance testing and was returned to service.
On September 29, 2009, P-7C failed. It was determined that the failure had been caused by the failure of coupling #7. The failed coupling was determined to have been improperly heat treated based on high hardness (-37 Rc). The cause of the improper heat treatment was determined to be a quality program problem at HydroAire. The September 2009 coupling failure was investigated during a Root Cause Evaluation conducted under CR-PLP-2009-04519. The RCE conducted after the failure did not sufficiently investigate the base material properties of 416 SS. Specifically, corrosion in the Lake Michigan water environment and the toughness properties of the material were not investigated.
During the September 2009 event response for P-7C, an emergent Purchase Order #
10253715 was issued to HydroAire for replacement parts for P-7C due to the fact that Palisades had no stock of replacement couplings. As noted in the Higher Tier ACE conducted under CR-PLP-2009-04806, the lack of spare parts resulted in an expedited procurement, miscommunications and an elevated sense of urgency within the Engineering and Procurement activities. This evaluation team determined that the expedited procurement of replacement couplings in October 2009 was likely a RCE Report CR-PLP-2011-03902 Page 8 of 136
Event Narrative contributing factor to the coupling failure in August 2011. If sufficient spare parts were in stock when the 2009 event occurred, the communication errors, verbal direction to HydroAire and incorrect independent testing would likely not have occurred. A directed corrective action has been created as part of this evaluation to develop the steps necessary to ensure sufficient SWP spare parts are available in-stock.
The 2009 event response Purchase Order # 10253715 had three revisions. The original had a quantity of zero shaft couplings, revision 1 had a quantity of 8 shaft couplings, revision 2 had a quantity of 16 shaft couplings, and revision 3 had a quantity of 18 shaft couplings. HydroAire began manufacturing the new shaft couplings, line 3 of PO 10253715, under their job #5912. During manufacturing, the components are sent to a separate heat treatment vendor for hardening and tempering. HydroAire uses a Chicago facility owned by BodyCote for heat treatment. BodyCote is a national heat treatment company and the Chicago facility has a 10 CFR 50, Appendix B, qualified QA Program.
HydroAire had three work order numbers of their own associated with the order of the shaft couplings. The shaft couplings were to be made from 416 SS. The three HydroAire WO #s were 27774, 27797, and 27799.
HydroAire WO # 27774 was to manufacture 8 shaft couplings. The shaft couplings in this work order were heat treated by Bodycote under two certification numbers.
Bodycote's certification number 92-42427 certified that 2 shaft couplings were heat treated per HydroAire's PO 20163, and these shaft couplings had a minimum hardness of 29.0 Rc and a maximum hardness of 30.0 Rc. Bodycote's certification number 92-42429 certified that 6 shaft couplings were heat treated by Bodycote per HydroAire's PO 20166, and these shaft couplings had a minimum hardness of 31.0 Rc and a maximum hardness of 31.0 Rc. Both Bodycote certifications also supplied the hardening, 1st temper, and 2nd temper furnace temperature graphs. The 8 couplings manufactured under HydroAire's WO # 27774 were accepted by Palisades under receipt inspection numbers 4724, 4735, and 4737 on October, 1st 2009. These 8 couplings were installed under Palisades WO 208591 into P-7C. The shaft coupling position in P-7C was not known.
HydroAire WO # 27797 was to manufacture 8 shaft couplings, and HydroAire WO #
27799 was to manufacture 2 shaft couplings. The shaft couplings in these 2 work orders were heat treated by Bodycote under certification number 92-42442. Bodycote supplied the hardening, 1st temper, and 2"d temper furnace temperature graphs. Bodycote's certification number 92-42442 certified 10 shaft couplings were heat treated by Bodycote per HydroAire's PO 20170, and these shaft couplings had a minimum hardness of 28.5 Rc and a maximum hardness of 29.5 Rc. These 10 shaft couplings were accepted by Palisades under receipt inspection number 4753 on October 5th, 2009 and October 13tH 2009. These couplings remained in storage until 2011 when they were used to restore P-7C.
According to the ACE completed in response to CR-PLP-2009-04608, Design Engineering gave verbal design direction to HydroAire allowing the use of "Double Tempering" of the line shaft couplings in order to assure that the proper material harnesses were achieved. Formal direction via the Purchase Order was not given and under current processes would not normally have been given to HydroAire. Typically when working with a qualified supplier, Palisades specifies the desired end result for a RCE Report CR-PLP-2011-03902 Page 9 of 136
Event Narrative design characteristic (in this case hardness) and the supplier chooses the manufacturing method necessary to achieve the acceptance criteria. In this specific event, Palisades staff would not have specified heat treatment regimes because HydroAire is the pump OEM for the Service Water Pumps and should be responsible for delivering quality components.
HydroAire supplied Certificate of Conformance and CMTR for all 18 of the supplied shaft couplings. All certificates of conformance and manufacturing travelers indicated that the components purchased by Palisades met the specified material hardness values.
In 2010, Palisades learned that there was OE from Prairie Island Nuclear Generating Station (PINGS) regarding two coupling failures due to IGSCC in 400 Series SS. The failed couplings had been supplied by HydroAire. Information on the OE did not come through the normal OE channels but was provided verbally. Palisades then contracted Structural Integrity to examine the OE and provide an analysis of Palisades vulnerability.
This RCE team initiated CR-PLP-2011-04469 documenting that the OE was not available via the normal channels and requesting that the OE be processed according to Fleet procedures.
In March 2011, Palisades received a report from Structural Integrity (SI), a metallurgical firm under contract to Entergy, regarding the 2010 PINGS OE. SI concluded that the currently installed couplings were satisfactory, but more margin could potentially be gained by either (1) creating a better specification for the 416 SS couplings or (2) changing materials. One potential material that was identified was 17-4PH SS has a much higher toughness value than the 400 series stainless steels. Based on the initial metallurgist recommendation, Palisades began to work with SI and HydroAire to develop a better specification for 416 SS rather than change materials. Although SI indicated that the current parts were satisfactory, Palisades missed an opportunity to question whether or not the spare couplings that had been purchased in 2009 were still suitable for installation into the Service Water System.
At 0700 his on August 9, 2011, all three Service Water Pumps, P-7A/B/C, were in service with basket strainer differential pressures at 1/2/1 PSID respectively. Critical Service water Header Pressure was 73 psig and stable. Control Room alarms associated with Service Water System (SWS) were all clear.
At approximately 1202 hrs, with all three Service Water Pumps in service, the Control Room received alarms EK-1 163, Critical Serv Water Header 'B' Lo Pressure, EK-1 164, Critical Serv Water Header'A' Lo Pressure, EK-1 165, Noncritical Service Water Low Pressure, EK-0557, Diesel Gen No. 1-2 Trouble and EK-1 132, Service Water Pump P7A Basket STR HI DP. The Control Room crew entered Off Normal Procedure (ONP) 6.1, Loss of Service Water. Control Room Operators observed SW Pump, P-7C, running at 31 amps. SW Pumps, P-7A and P-7B were in-service at 80 amps. Critical Service Water Header Pressure was 64 psig and stable. Tech Spec LCO 3.7.8 Al, a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action to restore P-7C to operable, was entered. A Nuclear Plant Operator (NPO) dispatched to the Screen house reported that P-7C had no discharge pressure and there was a loud banging noise from P-7C. Control Room Operators stopped P-7C. Critical Service Water Header Pressure was 64 psig and stable. All SWS and the DG 1-2 alarms were clear except EK-1 132, Service Water Pump P7A Basket STR HI DP, with P-7A DP at 6 psid due to the rise in flow through the SW Pump P-7A basket strainer.
RCE Report CR-PLP -2011-03902 Page 10 of 136
Event Narrative Work to restore P-7C began immediately under WO #286627. During disassembly, it was determined that line shaft coupling #6 had failed. The failure of coupling #6 was similar in nature to the failure of coupling #7 in September, 2009. As can be seen in Figure #1 (page 4), couplings #6 and #7 are above the water line when the pump is not running. This means that when P-7C is in standby, coupling #6 and #7 have the chance to dry out. Coupling #5 may also be dry, depending on lake levels. As is discussed in the metallurgical analysis from LPI (Attachment V), surface deposits of contaminants were found in the area of the fracture surface. Energy Dispersive X-Ray Spectroscopy (EDS) was performed on the surface deposits which were determined to contain oxides, chlorides and sulfides.
During the restoration phase, spare couplings purchased in October, 2009 were to be used. These couplings were from the same group of couplings that were purchased in response to the September 2009 event. Since the couplings to be installed had a similar pedigree to the currently failed coupling, a conservative decision to have the couplings independently hardness checked prior to installation was made. The couplings were taken by Palisades personnel to Consumers Energy's Trail Street lab for hardness testing. Trail Street determined that all couplings destined for installation in P-7C were within the specified range of 28-32 Rc.
P-7C was restored to service at approximately 0309hrs on August 12th, 2011.
During the restoration of P-7C, the station decided to contract Lucius Pitkins Inc (LPI) to perform detailed metallurgical analysis on the couplings in P-7C. By the time P-7C was restored, preliminary data from LPI pointed to a potential materials issue with the 416 SS. Based on the fact that the replacement couplings installed into P-7C were from the same metal heat and purchase order as the recently failed coupling, a decision was made to do an emergent change to the coupling materials. The decision was made to suspend the development of an improved 416 SS coupling specification and complete EC # 31337. The EC changed the material of the line shaft couplings from 416 SS to 17-4PH SS. Furthermore, a conservative decision was made to change the couplings in all three pumps to the new material in an expedited fashion. As with the 2009 event response, this created a sense of urgency around the manufacturing and procurement of new couplings. Procurement of 17-4PH SS couplings supported the emergent station direction to replace the couplings in all three SWPs. During the procurement, several non-conforming parts were identified. The non-conformances will be addressed in a Higher Tier ACE in response to CR-PLP-2011-04317.
Shown below is a picture of the failed coupling and a close up of the fracture surfaces.
Coupling #6 is the second coupling below the stuffing box and is subject to wet/dry cycles.
RCE Report CR-P LP-2011-03902 Page 11 of 136
Event Narrative y.,
f^l- I Figure 3: Failed Coupling #6 from P-7C Service Water Pump, Fracture Surfaces RCE Report CR-PLP-2011-03902 Page 12 of 136
Event Narrative Event Investigation A cross-functional Root Cause Evaluation Team was created to conduct the investigation into the Service Water Pump failure. The team was consisted of representatives from System Engineering, Design Engineering, Operations, Training, Procurement Engineering, Mechanical Maintenance and consultants from the Metallurgic and Root Cause Evaluation fields.
The evaluation team looked at four main areas in order to determine the root cause of the Line Shaft Coupling #6 failure. Those areas were:
- 1. SWP Operation
- 2. SWP Maintenance and coupling replacement
- 3. Organizational and Programmatic Issues
- 4. Failure Mode and Metallurgical Analysis of the SWP Couplings Each area will be discussed in this report.
Service Water Pump Operation The team performed an examination of the Shift Narrative Logs from the period beginning on October 2nd, 2009 and ending on August 9th, 2011. The log entries did not identify any unusual operations of the Service Water Pumps during the time in question. Based on Palisades Standard Operating Procedures, there is neither a minimum run time nor limit on cycling of the Service Water Pumps.
In addition to examining the logs for abnormal usage, the team discussed the history of the Service Water Pumps and design of the pumps. These pumps have been in use since the plant was commissioned and there have been no coupling failures noted prior to the 2009 installation of the 416 SS couplings. Therefore, the team concluded that the SWP design was acceptable for the service environment.
Based on the above information, the evaluation team concluded that there were no indications that the usage pattern for the Service Water System contributed to the failure of the P-7C line shaft coupling.
Service Water Pump Maintenance SWP P-7C had previously experienced a coupling failure on September 29th, 2009 and was documented in the RCE conducted under CR-PLP-2009-04519. During the course of that investigation, the maintenance representative on the RCE team analyzed the work steps and recorded data for the restoration (WO 208591). The analysis concluded that the Mechanical Maintenance team had correctly restored the pump in accordance with the approved work instruction, WI-SWS-M-04.
Subsequent to the August 2011 coupling failure, the Maintenance representative on the current RCE team examined the data recorded during the restoration conducted under WO 286627. This analysis concluded that Mechanical Maintenance had restored P-7C co rrectly according to WI-SWS-M-04.
RCE Report CR-PLP- 2011-03902 Page 13 of 136
Event Narrative In order to confirm that the work instruction was indeed robust and did not have any hidden error traps, the Palisades RCE team hosted a fleet call on 8/23/2011. During the fleet call, input was solicited from the participants regarding the robustness of the published work instruction. There was agreement amongst the non-PLP fleet participants that PLP's Service Water Pump installation instructions were complete, accurate and appropriate. Based on the FEA analysis, the in situ tensile stresses and a description of the actions taken to "snug" the shafts during installation, it was determined that Palisades practice of "bumping" the pump to tighten the shafts was not a likely contributor to the failures in September 2009 and August 2011.
Based on the above data, the RCE team concluded that the Mechanical Maintenance team restored P-7C correctly subsequent to both the September 2009 and August 2011 failures and that the work was completed using an accurate and appropriately rigorous work instruction.
O rganizational and Programmatic Factors Investiaation To ensure that organizational and programmatic causes were determined and described, an expert in Root Cause Evaluation, was contracted to assist in the review of the September 2009 event (CR-PLP-2009-04519, RCE), mis-communications with the NRC in 2009 (CR-PLP-2009-04806, HT ACE), and the August 2011 event, and then assist the RCE team in the O&P analysis.
The root cause team proceeded under the philosophy that (1) a failure mechanism (in this case stress corrosion cracking) caused the coupling failure and that (2) multiple flawed defenses allowed such a failure to occur. This is of special interest since the August 2011 failure occurred in the same service water pump as the September 2009 failure and was due to the same failure mechanism.
It was recognized that many individuals and groups were involved with service water pump design, material condition, and performance in the period 2009-2011 and earlier.
The assessment as conducted was not intended to "blame" any individual but to point out ways for Palisades and Entergy organizations to be more effective in the future.
The team began with a simple why staircase tree that laid out three necessary and sufficient requirements for pump shaft coupling failure, namely:
- The vendor HydroAire manufactured susceptible couplings.
- The customer Palisades accepted susceptible couplings.
- Couplings were exposed to a service environment that proved detrimental to them.
Technical details of coupling material deterioration and failure appear elsewhere in this report, and form the basis for the material related cause (RC1, CC1). The O&P evaluation will focus on "why" the failure was able to occur. An underlying premise is that while qualified vendors may be expected to ensure product quality and suitability for service, the burden is on the nuclear licensee to manage its relationships with such third parties. The team concluded that Palisades did not do so effectively.
RCE Report CR-PLP -2011-03902 Page 14 of 136
Event Narrative RCE Report CR-PLP-2011-03902 Page 15 of 136
Event Narrative ure 4: Whv Staircase Tree Susceptible couplings present in SWP shaft Hydr o-Aire manufactured PAL acce p ted su sceptible suscepti ble couplin gs couplings Insufficient Receipt/independent Over-reliance on Rockwell hardness PAL choice of material impact inspection did not vendor quality out ,32) vendor toughness detect processes (RC-2)
No PAL spec for Insufficient Did not exploit toughness ( RC-2) metallurgyailableOE(CC-expertise (CC-2)
Insufficient Did not exploit metallurgy available OE (CC-3) experience (CC-2)
The analysis found one organizational Root Cause (RC2) and two Contributing Causes (CC2, CC3) existed they are:
O&P Root Cause ( RC2)_
Palisade's Engineering specified the wrong Stainless Steel alloy for use in Palisades Service Water operating environment. The choice of 416 SS was based on historical data by personnel who did not have sufficient metallurgic knowledge.
As a result, couplings susceptible to intergranular stress corrosion cracking were available for installation in June 2009, September 2009 and August 2011. Two subsequent coupling failures and one degraded coupling occurred in the intermittently-wetted shaft segment of service water pump P-7C, both originating from IGSCC.
Supporting Facts
- 1. Purchase specifications and Palisades-approved HydroAire drawings did not contain a toughness requirement for the 416 SS coupling material. This may have discouraged reliance on "second or re-tempering" to achieve acceptable Rockwell hardness values.
- 2. To offset concerns about galling during coupling assembly, Palisades narrowed the acceptable Rockwell hardness range from the ASTM value (24-32 Rc) to 28-32 Rc. This made the acceptable hardness band restrictive and moved it closer to 35 Rc, a hardness known for IGSCC susceptibility.
RCE Report CR-PLP-2011-03902 Page 16 of 136
Event Narrative
- 3. Palisades did not dictate or approve hardness testing sample size, location, or timing (e.g., pre or post machining). This was left to HydroAire and its subcontractor Bodycote.
- 4. For hardness, Bodycote offered "commercial" (fractional testing) and "MIL-SPEC" (100% testing). HydroAire accepted fractional testing. There was no requirement to increase sample size when testing failures occurred.
- 5. HydroAire and Bodycote were free to retemper non-conforming couplings in an effort to reduce hardness below 32 Rc. If that succeeded, the product could be shipped.
- 6. The 2007 Engineering Change (5000121762) which produced the procurement specification did not consider fresh water to be "corrosive."
- 7. In 2009, a receipt inspection staff member reported "recent work practices" that included verbal requests to Hydro-Aire on manufacturing and design issues. The practice led to "occurrences that were outside the purchase order process as they frequently did not result in purchase order revisions."
- 8. As documented in CR-PLP-2009-04806, Palisades was not always well-informed regarding the testing plan details of HydroAire. This apparently resulted from verbal communication of the type described above.
- 9. Lab data (Trail Street and LPI) indicate that hardness can vary 2 - 4 units Rc from place to place on a finished coupling.
- 10. Trail Street data also demonstrate that some removed 2009 series couplings exceeded the hardness spec with one from an intact coupling (Top 4, Reading 3) reaching 36 . 1 Rc. During the September 2009 event and this evaluation, the team sought expert advice to explain the discrepancies noted between the hardnesses noted on the Certificates of Compliance and the post failure data collection. Four separate metallurgists have agreed that there is no way that the couplings could have been work hardened within Palisades Service Water System as installed in the plant . Suggested reasons for the discrepancies include inherent piece part variation with in each coupling and M&TE differences between each lab.
- 11. Macroscopic photos of recently-removed, intact couplings (Pumps P-7B and P-7C) show orange discoloration at the midpoint and ends, similar to failed couplings. Couplings from pump P-7A show a "greasy-gray" interior appearance and no visual evidence of corrosion.
- 12. The 2011 procurement was the first time the Critical Procurement Plan (EN-MP-100) had been used for emergent purchase of SWS pump shaft couplings.
- 13. The system engineer worked closely with HydroAire, but he never "chased down" the heat treatment labs.
RCE Report CR-PLP-2011-03902 Page 17 of 136
Event Narrative
- 14. Palisades receipt inspections were not written to require testing of any kind.
They included verification of shipment contents, including paperwork. Any other testing would be outside of the normal reciept inspection responsibilities.
O&P Contributing Cause 1:
In 2007, during the service water pump refurbishment Engineering Change, Palisades did not ensure technical oversight by Entergy personnel with expertise in metallurgy.
This resulted in failure to recognize the risks associated with material changes associated with Critical Safety Related equipment.
As a result, the station has accepted supplier product that was inadequate for predicted service conditions and has been unprepared to respond expeditiously to pump failures when they occurred.
Supporting Facts Palisades did not have in 2007 (and does not now have) Engineering Staff with detailed metallurgical knowledge. The corporate culture in 2007 would have made it difficult to obtain outside resources unless a detailed case could be made supporting the need. Under current Entergy fleet procedures and expectations, this roadblock would not exist. As was demonstrated in 2009 and during the event being analyzed in this evaluation, the station has contracted outside expertise in a meaningful way.
- 2. Palisades considers HydroAire is one of a limited number of available firms capable of overhauling vertical service water pumps and producing replacement parts to Palisades's satisfaction.
- 3. Since 2000, Palisades has come to rely on HydroAire for "reverse engineering" and replacement parts related to maintaining the set of three service water pumps. Until 2007, the focus was on deteriorating carbon steel materials and impeller wear from sand entrained in lake water.
- 4. Alternatives to HydroAire are limited (1) by lack of other qualified nuclear suppliers and (2) PAL reluctance to use FlowServe. According to an interviewee, HydroAire was willing to "stop and listen." There was less favorable responsiveness by other companies.
- 5. HydroAire typically measures old pump parts, redesigns/improves them, and submits "drawings" to PAL. Drawings include dimensional data and criteria like hardness (Rc). Materials chosen were those "commonly used" in fresh water service water pumps. According to the design engineer, there was an EC to document all "reverse engineering" activities.
- 6. PAL approves the drawings and orders parts from HydroAire. Much of this was handled by a System Engineer until his retirement (-2009). During this time, the Engineer was working under the Nuclear Management Company Contract process instead of using the Procurement activity. This would not be allowed under current Entergy Processes.The 2007 EC reviewed the Perry OE but failed to realize the susceptibility to IGSCC and that Perry had switched material to 17-RCE Report C R-P LP-2011-03902 Page 18 of 136
Event Narrative 4PH SS from 400 Series SS. No further action was taken on the OE because 1) the Engineer involved did not realize that Lake Michigan water was corrosive to 400 Series SS and 2) the equivalency EC process under the NMC did not require an extensive OE evaluation.
- 7. Recent conference calls have involved multiple metallurgists. The historical design engineer (retired - 2009) had no metallurgical background. He relied on input from reverse engineering by HydroAire. Palisades did not leverage group or corporate expertise in 2009. An interviewee stated "I cannot phone someone in Jackson (no metallurgists in Echelon) for materials advice."
- 8. A root cause team member who had participated in 2009 visits to HydroAire and BodyCote characterized the HydroAire work areas as disorganized in appearance. He stated that the president of HydroAire told him that Palisades seemed more interested in speed . An interviewee stated that despite HydorAire' s nuclear credentials , HydroAire was "surprised " at the quantity of documentation required.
O&P Contributing Cause 2:
Since at least 2004, there has been substantial nuclear industry operating experience linking IGSCC susceptibility with high Rockwell hardnesses and/or low material toughness in type 416 SS exposed to fresh water. Palisades did not translate this OE into effective specification, contract, inspection, testing, or oversight actions.
Supporting Facts
- 1. When the first failure occurred (September 2009) there were no spares on hand.
HydroAire was engaged to provide them on relatively short notice (PO 10253715, Revs 0 through 3).
- 2. When the second failure occurred (August 2011), there were spares available to refurbish one pump, but they were from the 2009 rapidly-executed series.
- 3. In 2010 operating experience (Prairie Island) suggested that 416SS was potentially ill-suited for fresh water service and that 17-4 SS was preferable.
- 4. During this time Palisades continued to rely on its stock of 416 SS spares until the 2011 failure. At the time of the failure, Palisades Design Engineering was working with another Metallurgist, Sturctural Integrity, and HydroAire to develop a robust method of specifying a heat treatment of 416 SS that would meet the site needs.
- 5. In the mid 2000s, according to the retired design engineer, there was a "lot of OE out there" with hundreds of hits, all related to over-hardening the stainless steel material. Most involved failures of couplings and shafts in salt or brackish water.
- 6. The 2007 EC was developed using fleet procedures from The Nuclear Management Company (NMC), see discussion below. Current Palisades practice is to not reverify an existing approved Engineering Change when doing a related EC.
RCE Report CR-PLP-2011-03902 Page 19 of 136
Event Narrative NMC procedure FP-E-EQV-01, Equivalency Evaluations and Changes, was used in the preparation of EC5000121762. The procedure does not require the Responsible Engineer to do a review of industry operating experience related to the part being replaced; however, it does say to look at Attachment 6 for a description of error likely situations that may be faced when preparing an equivalency based on fleet operating experience. The attachment discusses error-likely situations for items like to poor original design and extreme service conditions. Attachment 4 of FP-E-EQV-01 actually discusses that the System Engineer be consulted for any site specific or industry operating experience related to the equivalent change.
During the timeframe that NMC operated Palisades there was no engineering pre-job procedure that would allow an engineering product to be considered for higher levels of review based upon a risk versus consequence approach. This type of consideration was typically left to the line supervision to determine. Since the implementation of the Entergy procedures at Palisades an improvement has been made in this regard. EN-HU-1 04 "Engineering Task Risk and Rigor" is the procedure that is used to perform pre-job briefs for engineering tasks. The process determines the risks and consequences of things that could "go-wrong" regarding the engineering product and contains a risk rank scoring system that drives the level of review required for the product. Typically the scoring results in "Exisiting Engineering Process Review". However, increasing levels of review could also be specified for high risk rank scores (e.g. "Independent Review by Station", "Independent Review by Consultant/Specialists", or "Independent Collegial Review or Challenge Board"). When combined with the more stringent OE analysis and independent review criteria specified by EN-DC-115, Engineering Change Process, Palisades is less susceptible to human error in knowledge space analysis due to more robust awareness of potential error traps.
Entergy procedure EN-DC-115 governs all of the types of engineering changes performed by Entergy including equivalent changes. This process requires that a review of operating experience be performed by the responsible engineer. This operating experience should be both internal and external. While not explicitly stated in EN-DC-115, it would be good practice and thorough to discuss how the found operating experience is applicable to the equivalent change and disposition any of the operating experience that resulted in failures at other sites.
There is a significantly different cultural model in place within Palisades Engineering then in 2007. Site engineers now have a different mindset when it comes to technical rigor and product quality then in the past due to higher expectations from Entergy site and fleet management. When combined with the more rigorous Entergy procedures, the site is able to ensure that a final engineering product meets or exceeds standards.
RCE Report CR-PLP- 2011-03902 Page 20 of 136
Event Narrative Failure Mode and Metallurgical Analysis On 8/15/11, the RCE team began development of a Failure Modes Analysis (FMA) based on the 8/9/11 failure. The development of the FMA included input from metallurgists, pump experts, system engineering, design engineering, and maintenance and encompassed over 70 different failure modes. The FMA is included in Attachment II to this report.
Over the next several weeks, this FMA was further refined with additional details and the appropriate action necessary to "support" or "refute" the individual failure modes. In addition, the FMA actions were further categorized under both aspects of failure, as well as plausibility.
As data was received from various parties, the information was added to the FMA spreadsheet, and resultant changes in the "support", "refute", and "plausibility". The FMA was then sorted and the potential causal factors to be investigated are given below:
Causal Factors under Investigation
- Stress Corrosion Cracking o Components Replacement in last 5 years Improper Coupling Material Inadequate Hardness Specification for Coupling Inadequate Cooling Process
- Improper Hardness o Corrosion Caused by Environmental Conditions o Water Chemistry Lake Michigan Chlorination of Service Water Lucius Pitkin, Inc. (LPI) Testing Material Test data presented is based on a Draft Report from Lucius Pitkin, NOTE:
Inc. (LPI). The final copy was not completed at the time of this writing.
Couplings from P-7A and P-7B were sent to LPI for comparative analysis.
Based on current timing information from LPI, the final report will not be available until after the CARB approval date for this analysis. There will be a need for a substantive revision to this evaluation in order to include all final data from LPI. This revision will be brought back for approval at the level s p ecified b y CARB.
In order to properly characterize and diagnose the failure of the #6 coupling on the P-7C, Entergy contracted LPI to perform metallurgical examination of the Service Water components for both the 2011 and 2009 coupling failure. The metallurgical examination included several destructive and non-destructive tests. These tests and the currently available data are listed below, full details of the testing and results are in Attachment IV of this report.
RCE Report CR-PLP-2011-03902 Page 21 of 136
Event Narrative Test Parameters Visual and Photographic Visual examination of the 2011 coupling failure Examination indicates stress corrosion cracking (SCC), initiating from two locations from the interior of the coupling to the exterior perpendicular to the axis of the coupling.
The crack initiation sites were one thread apart vertically, and not associated with the vent hole. The SCC propagated in an elliptical fashion through the thickness of the coupling. Final fracture was due to overload of the remainin g material.
Surface Hardness Testing Surface hardness of the failed coupling averaged 33.3 Rockwell C. This exceeds the maximum specified hardness of 32 Rc. Couplings #5 and #7 averaged 29.6 and 31.4 respectivel y .
Dimensional Examination Coupling OD and length are within specified dimensions and tolerance. Eccentricities of coupling were checked and coupling 4 had the highest at 7 mils.
Compositional Analysis of EDS of surface deposits indicates presence of Surface Deposits chlorides, sulfides and oxides Magnetic Particle MT examination shows cracks through the failed Examination coupling #6, and indications of a crack in coupling #7 at the thread root closest to the vent hole, but not throu g h the vent hole.
Tensile Testing Ultimate Tensile Strength - 155 ksi Yield Stren gth - 140.6 ksi Charpy V-Notch Testing Charpy Impact testing indicates that the failed coupling had low impact toughness and leads the material to be vulnerable to SCC Through Thickness Hardness Through thickness hardness test of failed coupling 6 Testing ranged from 31.5 to 32.7. For 5 and 7 values were less than 32.
Compositional Analysis Compositional Analysis confirms the couplings are made of 416 Stainless Steel Scanning Electronic SEM examination revealed the fracture surface Microscopy morphology to exhibit a rock-candy appearance, characteristic of intergranular stress corrosion crackin g ( IGSCC ) .
RCE Report CR-PLP-2011-03902 Page 22 of 136
Event Narrative LPI Conclusion Summary Palisades SWS pump P-7C coupling #6 failed in August , 2011. The failure is determined , based on metallurgical evaluation, to be the result of intergranular stress corrosion cracking (IGSCC ). The 2009 failure of the #7 coupling on the same pump (P-7C) was determined under CR-PLP-2009-04519 to also be a result of IGSCC. LPI's independent examination of the 2009 failed coupling 09-P7C- 7F concurs with the failure mode as documented in the 2009 event RCE.
For IGSCC to occur three criteria to promote IGSCC must exist; 1) susceptible material,
- 2) tensile stress and 3) corrosive environment. The specified coupling material, ASTM A582 Type 416 stainless steel, is martensitic steel that is susceptible to IGSCC at low toughness. Charpy V-Notch (CVN) testing of the 2011 failed coupling resulted in toughness values in the range of 6 to 10 ft-lbs impact energy for test temperatures of 32°F and 70°F. CVN testing of the 2009 failed coupling resulted in impact toughness values in the range of 3 to 6 ft-lb for test temperatures of 32°F and 70°F, respectively.
These low impact toughness values make the couplings susceptible to IGSCC.
The couplings are subjected to tensile stresses during normal operation by the weight of the components below the coupling and hydrodynamic forces due to pump operation. In addition, the design of the couplings results in the shaft ends bearing against each other that likely led to sufficient tensile stresses (with a maximum value near the center where the two shafts bear against each other) in the coupling to initiate and propagate a crack.
The majority of the pump couplings below the packing (couplings #1 through #4) are submerged below the water level in the intake structure at normal basin levels.
Couplings #5 through #7, above normal basin water levels see intermittent cycles of wet and dry depending on whether the pump is operating. When the SW pumps are on, all couplings below the stuffing box are wet and when they are off, couplings #5, #6 and #7 begin to dry. Chemistry samples of the service water indicate that there are low levels of chlorine in the raw water of Lake Michigan on the order of 9 ppm. Chlorination of the service water increases the chlorine level slightly to approximately 10 ppm. Even these relatively low levels of chlorine combined with a high humidity oxygen rich environment (as is the case for the couplings #5, #6 and #7 when the pump is off) can lead to a local breakdown of the passivation layer. IGSCC can nucleate at these locally damaged sites, develop and propagate under sufficient tensile stress to form a highly branched network of fine cracks.
RCE Report CR-PLP-2011-03902 Page 23 of 136
Root Cause Evaluation The following Evaluation Methods were used to identify the Root and Contributing Causes:
- Event and Causal Factor Charting (Attachment I)
- Failure Mode Analysis (Attachment II)
- Metallurgical Testing by LPI (Attachment IV)
Root Causes RC,: The 2009 and 2011 Line Shaft Coupling failures were due to Intergranular Stress Corrosion Cracking (IGSCC). The coupling material is a quenched and tempered 416 martensitic SS with low toughness properties. This makes it particularly susceptible to IGSCC when subjected to the tensile stress and a corrosive environment (due to the presents of chlorides).
RC2: Palisade's Engineering specified the wrong Stainless Steel alloy for use in Palisades Service Water operating environment. The choice of 416 SS was based on historical data by personnel who did not have sufficient metallurgic knowledge.
Contributing Causes CC1: Increased Susceptibility to IGSCC caused by Tempering Embrittlement.
Temper embrittlement can result after tempering in the range of 700F to 1050F or slow cooling through this range. Tempering in the range of 700F to 1050F is not recommended because it results in low and erratic impact properties (i.e. toughness) and poor resistance to corrosion and stress corrosion, which increases the materials susceptibly to stress corrosion cracking.
CC2: Insufficient use of qualified metallurgical expertise . In 2007, during the service water pump refurbishment Engineering Change, Palisades did not ensure technical oversight by Entergy personnel with expertise in metallurgy. This resulted in failure to recognize the risks associated with material changes within Critical Safety Related equipment.
CC3: Ineffective use of Operating Experience . Since at least 2004, there has been substantial nuclear industry operating experience linking IGSCC susceptibility with high Rockwell hardnesses and/or low material toughness in type 416 SS exposed to fresh water. Palisades did not translate this OE into effective specification, contract, inspection, testing, or oversight actions. (Addresses CR-PLP-2011-03975, closed to this evaluation)
RCE Report CR-PLP-2011-03902 Page 24 of 136
Root Cause Evaluation Organizational and Programmatic Weakness Evaluation Organization to Organization Interface Weaknesses OP1A: "Inadequate interface among organizations." (CC3)
Analysis of the 2009 coupling failure Root Cause (CR-PLP-2009-04519) and the 2009 HT ACE regarding miscommunications (CR-PLP-2009-04806) showed that Palisades Engineering bypassed MP&C by giving verbal directions to HydroAire.
Subsequently, these verbal directions were not captured in a PO.
Organizational to Program Interface Weaknesses There was no evidence of Organization to Program failure modes for the Problem Statement; therefore all failure modes in this category were eliminated.
Program to Program Interface Weaknesses There was no evidence of Program to Program failure modes for the Problem Statement; therefore all failure modes in this category were eliminated.
Programmatic Deficiencies OP4B : " One or more necessary functions required by a process were missing in the implementing procedures ." (RC2)
Palisades did not ensure (1) a toughness requirement in the purchase specification or (2) sufficient Rockwell hardness testing to guarantee the absence of coupling material with > 32 Rc.
OP4D : " Inadequate verification process ." (CC2)
Procedure quality (i.e., purchase specification) did not require HydroAire to test hardness in multiple locations nor did it equip Receipt Inspection to do it on behalf of Palisades.
OP4F : " Response to a known or repetitive problem was untimely ." (RC2)
Corrective actions that were feasible after the 2010 Prairie Island event were not taken. In 2011 Palisades was forced to rely on existing stocks of couplings that proved to have the same IGSCC susceptibility.
OP4K : " Personnel assigned did not have adequate experience or training to perform the work ." (CC2)
Before actual coupling failures began to occur, no involved Palisades person had metallurgical subject matter expertise. Even then, such expertise came from various external sources.
RCE Report CR-PLP-2011-03902 Page 25 of 136
Root Cause Evaluation OP4N : "Personnel exhibited insufficient awareness of the impact of actions on safety/reliability." (RC2)
Despite the importance of service water to the PRA/PSA analysis (approximately 9tn-ranking Palisades system) pump refurbishment efforts took place over a multi-year period, and efforts to ensure sufficient couplings on hand were not aggressive until the need for them arose.
OP4P : " Previous industry or in-house operating experience was not effectively used to prevent problems ..." (CC3)
Palisades did not translate a substantial body of OE into effective specification, contract , inspection , testing , or oversight actions.
Organizational Weaknesses OP5F : "Corrective action for previously identified problem/event not adequate to prevent recurrence." (RC2)
Palisades coupling replacement in 2009 did not use a product sufficiently resistant to IGSCC in the application and environment. The 2011 failure was essentially a repeat, and the spare couplings on hand for use in the 2011 response were more with the same susceptibility.
OP5AD : "Risk/consequences associated with change not adequately reviewed/assessed." (RC2)
Palisades over-relied on vendor expertise and did not provide sufficient technically competent oversight to ensure adequate testing of product quality.
In summary, Latent Organizational Weaknesses were identified as contributing to the causes of the problem. Corrective actions have been developed to correct the identified weaknesses RCE Report CR-PLP-2011-03902 Page 26 of 136
Root Cause Evaluation Safety Culture Evaluation Attachment III - Safety Culture Evaluation, presents a screening of the Root and Contributing Causes of the RCE against the thirteen Safety Culture impact areas, and further evaluation of the causes at the specific aspect level for the applicable impact areas. The evaluation of the impact areas revealed a potential weakness in Resources, Operating Experience and Accountability. There was no evidence that identified potential weaknesses within the other impact areas.
A potential weakness was identified in the impact area Resources because there was evidence that Palisades' specifications for the coupling required use of 416 SS and did not require toughness testing nor adequately test for hardness (insufficient sampling size). Furthermore, Palisades' specifications for the coupling did not involve input/review by a qualified metallurgist. This weakness was further evaluated in Table 2 to determine the specific aspect(s) that were indicative of the potential weakness for this impact area with the following conclusions (see Attachment III for definitions of each code):
- The aspect H.2.b was evident in the one Cause. H.2.b is the training of personnel and sufficient qualified personnel to maintain work hours within working hour guidelines.
- The aspect H.2.c was evident in one Cause. H.2.c is having complete, accurate and up-to-date design documentation, procedures and work packages and correct labeling of components.
A potential weakness was identified in the impact area Operating Experience because there was evidence that Palisades' did not take full advantage of operating experience suggesting that 416 SS was susceptible to IGSCC. This weakness was further evaluated in Table 2 to determine the specific aspect(s) that were indicative of the potential weakness for this impact area with the following conclusions:
- The aspect P.2.a was evident in one Cause. P.2.a is defined as the licensee systematically collects, evaluates and communicates to affected internal stakeholders in a timely manner relevant internal and external OE.
- The aspect P.2.b was evident in one Cause. P.2. is defined as the licensee implements and institutionalizes OE through changes to station processes, procedures, equipment and training programs.
The potential weakness was identified in the impact area Accountability because there was evidence that a Palisades' system engineer functioned as a design engineer when dealing with HydroAire. This weakness was further evaluated in Table 2 to determine the specific aspect(s) that were indicative of the potential weakness for this impact area with the following conclusions:
- The aspect A.1.a was evident in one Cause. A.1.a is that accountability is maintained for important safety decisions in that the system of rewards and sanctions is aligned with nuclear safety policies and reinforces behaviors and outcomes which reflect safety as an overriding priority.
The Safety Culture Evaluation identified potential weakness; however there wasn't evidence that suggested there is an adverse trend within the impact areas and aspects with respect to this evaluation. The proposed corrective actions in this Root Cause Evaluation are sufficient to resolve the potential weaknesses identified in the Safety Culture Evaluation.
RCE Report CR-PLP -2011-03902 Page 27 of 136
Generic Implications: Extent of Condition/Extent of Cause Extent of Problem/Condition The intent for this review is to determine if the line shaft coupling failure of Service Water Pump P-7C may have identified a condition that could currently exist in other plant equipment.
A total of 17 (including P-7C) vertical turbine centrifugal pumps were identified to be in-service at Palisades and the safety classification for these pumps was determined from the Equipment Database in Asset Suite (Refer to Table 1). Of the pumps identified, three were found to be Safety Related, four were Augmented Quality, and two additional pumps that are also covered under the Maintenance Rule related. The line shaft couplings on the Condensate Pumps, P-2A and P-2B, are flexible gear design and out of the scope of this review because they are of a different design and material than those on P-7C. The line shaft coupling material was determined for the remaining pumps by looking into the pump drawings and model information in Asset Suite.
Any couplings with materials other than 416 Stainless Steel were determined to be out of the scope of this review due to the condition of the P-7C coupling failure being tempered 416 SS. Fire Pumps P-9A, P-9B and P-41 were found to have 416 SS line shafts with carbon steel couplings. Of the remaining pumps, Service Water Pump P-7A and Fire System Jockey Pump P-13 were discovered to also have 416 SS couplings; however, the line shaft couplings on P-13 are ASTM A582 416 Stainless Steel Condition A (Annealed) per Vendor Drawing M-33 Sh 22 and were supplied by the pump manufacturer (Johnston/Sulzer). SWPs P-7A/B/C use ASTM A582 416 SS Condition T (Tempered) based on the hardness range (28 to 32 Rc) given in HydroAire drawing 1047237 Rev 0. Per ASTM A582, Annealed 416 SS is softer (Hardness of 26 Rc max) than the Tempered 416 SS used for P-7A/B/C. Because of this, the couplings in P-13 are less susceptible to the same type of brittle fracture which occurred in P-7C. Due to the fact that P-13's couplings are constructed of 416 SS, which was annealed and not tempered, P-13 is not susceptible to the over-hardness due to heat treatment failure mode. For Service Water Pump, P-7A all the hardness data for the installed line shaft couplings provided by HydroAire was determined to be within the acceptable range for operation.
Potential Extent of Condition vulnerability for SWS Pumps P-7A/B/C will be eliminated by installing new couplings made of 17-4PH SS (addresses CR-PLP-2011-03961, closed to this evaluation). The new material was selected with the assistance of metallurgists from Structural Integrity. As of this writing, P-7A is complete, P-7B is in process and P-7C is scheduled for 9/29/2011. Analysis of the removed couplings is on-going at LPI and will be published in a revision to this analysis when the data is available.
RCE Report CR-PLP-2011-03902 Page 28 of 136
Generic Implications: Extent of Condition/Extent of Cause Table 1: Palisades Vertical Centrifugal PumDs Coupling/Line Coupling Rebuild ID # Pum p Title Safe ty Class Shaft Material Vendor/ PO # Date WO #
Class Seismic Q Grou p-FLEXIBLE CONDENSATE GEAR ----- ----- -----
P-2A PUMP NSR N N COUPLINGS FLEXIBLE CONDENSATE GEAR -----
P-2B PUMP NSR N N COUPLINGS SCREEN P-4 WASH PUMP NSR N N WARM WATER RECIRC -
P-5 PUMP NSR N N SERVICE 17-4PH, 416 P-7A WATER PUMP SR 1 C SS H droAire/10190242 4/5/2009 51637416 SERVICE 416 Tempered P-7B WATER PUMP SR I C SS, 416 SS H droAire/10246213 6/2010 20082 SERVICE 416 Tempered 6/12/2009 1190235/
P-7C WATER PUMP SR 1 C SS, 416 SS H droAire/10253715 /10/2/2009 00208591 1045 CS (Cat ID 0003716560),
MOTOR 1018 CS DRIVEN FIRE (5935-M32-P-9A PUMP QP N N DSL )
DIESEL 1045 CS, 416 DRIVEN FIRE SS (EAR- -----
P-9B PUMP QP 2 Y 2001-0457 ) 12/11/2001 HEATER P-10A DRAIN PUMP NSR N N HEATER P-10B DRAIN PUMP NSR N N 416 Annealed FIRE SYSTEM SS, 416 SS JOCKEY (VEN-M33-P-13 PUMP QP 2 N SH22 Sulzer Pum p s Inc. 2/4/2008 51633726 COOLING P-39A TOWER PUMP NSR N N COOLING P-39B TOWER PUMP NSR N N DILUTION P-40A WATER PUMP NSR N N DILUTION P-40B WATER PUMP NSR N N DIESEL 1045 CS (Cat DRIVEN ID COOLING 0003716560), -- -- - - - -- - -----
TOWER FIRE 410 SS (5935-P-41 WATER PP QP 2 Y M-317-SK-1 )
( ) : For items that are out of the scope of this review due to coupling material/ design or have no safety related function or have no maintenance rule function NSR: Not Safety Related QP: Augment Quality Program SR: Safety Related C: ASME Section III Class 3 This table is a snapshot of the pump status as of 8/9/2011 RCE Report CR-PLP-2011-03902 Page 29 of 136
Generic Implications: Extent of Condition/Extent of Cause Extent of Problem/Condition Summary The evaluation for this extent of condition concluded that the Service Water Pump P-7B and the Fire System Jockey Pump P-13 were discovered to also have 416 SS line shaft couplings and could be susceptible to the same condition as P-7C. The material used for the line shaft couplings on P-13 are ASTM A582 416 Stainless Steel Condition A (Annealed), which is softer than the Tempered 416 SS used in P-7C and makes it less susceptible to brittle fracture.
Extent of Cause The intent for this extent of cause review is to determine if the same causes that proved consequential in this instance currently exists in other plant equipment, processes or human performance. Additionally, the extent of cause determines if the identified causes also may have affected the performance of other individuals or work groups, the quality of other programs or processes, and/or the reliability of other types of equipment.
RC1 Extent (IGSCC induced failure):
IGSCC failures due to improper tempering have only been found in SWS P-7C in 2009 and 2011. In this case, the susceptibility to a common cause is limited to the Service Water Pumps. After the August 9th, 2011 failure, Palisades began pro-active replacement of all Service Water Pump couplings with couplings made from a different material. This was a conservative decision to eliminate/greatly reduce susceptibility to IGSCC.
RC2 Extent (Wrong Material Specified):
The entry point for the error which resulted in the failure of the P-7C Line Shaft couplings was introduced in 2007 during the completion of the Engineering Change that specified the 416 SS couplings. This occurred before the acquisition of Palisades by Entergy. As discussed in the O&P Investigation portion of the Event Narrative, the NMC processes in use when the EC was completed facilitated the entry of the error condition. The current implementation of EN-DC-115 is more prescriptive, thus making it less likely that the same type of error would pass through the EC process undetected. To verify this, corrective action 7 was written to review a sample size, per EN-QV-109, of modifications since 2007 where materials changes were introduced.
CC1 Extent (Increased IGSCC Susceptibility):
Tempering embrittlement can apply to any 416 SS hardened material if it is improperly heat treated. No evidence of other cases of incorrect heat treating was found outside of P-7C failures in 2009 and 2011. The change to 17-4PH SS will mitigate this risk.
CC2 Extent (Insufficient Use of Metallurgical Expertise):
As with the RC2 Extent of Cause, this condition would be mitigated under current Entergy Procedures through the effective EN-DC-115. Informal polling of the Design Engineering Staff showed a high degree of confidence of the process steps in EN-RCE Report CR-PLP-2011-03902 Page 30 of 136
Generic Implications: Extent of Condition/Extent of Cause DC-115 would lead to identification of incorrect material selection prior to implementation. No previous occurrence of this cause has been noted at Palisades prior to the P-7C events.
CC3 Extent (Ineffective Use of OE)
This evaluation determined that the available OE was not accurately evaluated in 2007 and that opportunities to evaluate the OE again in 2009 were not effective. Ineffective use of OE represents a failure of one of the most cost effective barriers for preventing failures. By analyzing failures at other sites, Palisades gains the ability to take action before an actual failure here. In this case, OE from the 2003/2004 Perry Station repeat failure should have been a key area for investigation when specifying 416 SS for use in the line shaft couplings. CA #8 was written to take a sampling of RCE's and HT ACEs since 1/1/2008 and analyze the robustness of the OE Evaluation. CA #7 directs the evaluation of the OE in a sample of material change EC's since 2008.
Generic Implications Summary In Summary, the extents of causes for this evaluation were evaluated against other systems and programs. The identified Human Performance issues in this evaluation occurred in 2007, prior to the implementation of the Entergy Fleet Human Performance Procedures. Had these procedures (or equivalent) been in use at the time of the 2007 engineering change, there is high confidence that the error would not have occurred due to increased emphasis on Human Performance.
RCE Report CR-PLP-2011-03902 Page 31 of 136
Previous Occurrence Evaluation Summary of Operating Experience using Grade 416 Stainless Steel Operating experience that discussed temper embrittlement, material hardness, stress corrosion cracking and coupling design features were found to be applicable. The Palisades coupling material being 416 SS is prone to these attributes and requires consideration in light of this recent coupling failure. Operating Experience examined included the Perry Repeat failures in 2003 and 2004 as well as the recent 2010 Prairie Island Event. The Prairie Island OE was used by Palisades as part of the decision to change coupling materials from 416 SS to 17-4ph SS.
The OE highlights the need to ensure that proper material specifications and processes are applied for controlling hardness, toughness and other material properties that make 416 stainless steel less prone to temper embrittlement and corrosion cracking failures.
The OE also makes it clear that Licensees stipulate proper quality controls that assure coupons and testing results reflect actual material conditions.
Also, Design Engineering activities need to verify the coupling design to assure that "stress risers" are minimal and would not contribute to corrosion cracking.
Due to the change from 416 SS to 17-4PH SS, the above recommendations are historical in nature and are meant to highlight the actions that should be taken when specifying the material.
The detailed OE search can be found in Attachment IV Operating Experience Summary for using Grade 630, 17-4 PH Stainless Steel The OE identified the need to evaluate coupling environment and material conditioning to assure 17-4 stainless steel coupling are not prone to hydrogen embrittlement, thermal embrittlement, sulfide stress corrosion cracking, crevice corrosion and Microbiological influenced corrosion.
Also, there is a need to assure stringent process controls are applied to coupling manufacture to avoid contaminates during the heat treat process, such as nitrogen pick-up, that would affect the conditioning of 17-4 PH material. Additionally, the responsible design activity must ensure proper material specifications and processes are applied for controlling hardness, toughness, and other material properties that make 17-4 less prone to embrittlement and corrosion cracking failures. Equally important is the need to stipulate proper quality controls that assure coupons and testing results reflect actual material conditions.
Because 17-4PH SS arrives at the component machining site, the above recommendations are handled at by the facility supplying the bar stock to the machining shop. There are no specific steps required by the utility using components made from the 17-4PH bar stock. The machine shop purchasing the stock would be able to audit the Certified Material Test Report accompanying the stock to ensure adequate preparation and hardening.
The detailed OE search can be found in Attachment IV RCE Report CR-P LP-2011-03902 Page 32 of 136
Safety Significa nce Evaluation This section of this report evaluates the impact on safety of the Service Water Pump P-7C failing to provide discharge pressure to the Service Water Header: Industrial, Environmental, Radiological, and Nuclear Safety were evaluated and summarized below.
General Safety of the Public (Environmental / Nuclear Safety):
From event initiation on 8/9/11,1202 hrs to the exiting of LCO 3.7.8 Al on 8/12/09, 0309 hrs at no time during that time frame was the health and safety of the public jeopardized. On-shift licensed personnel "verified that at least 100% of the post accident SWS cooling capacity available." To clarify, at least from the previous sentence, One hundred percent of the required SWS post accident cooling capability can be provided by any two SWS pumps if SWS flow either to the non-critical header (CV-1359) or to the critical loads inside the containment are capable of being isolated (CV-0824 or CV-0847). The SWS is divided in two separate and redundant trains, each associated with a Safeguards Electrical Train. The SWS train associated with the Left Safeguards Train consists of one Service Water Pump (P-7B), associated piping, valves, and controls for the equipment to perform their safety function. The SWS train associated with the Right Safeguards Train consists of two Service Water Pumps (P-7A and P-7C), associated piping, valves, and controls for the equipment to perform their safety function.
All Right Train components, except P-7C, remained operable. Compensatory actions to place protected equipment signs around the Non-Critical Service Water Isolation Valve, CV-1359, Service Water From Containment, CV-0824, and Service Water to Containment, CV-0847, were enacted to maintain isolation capabilities to ensure 100% of post accident SWS cooling capability with only two service water pumps being operable. The Risk Achievement Worth (RAW) score was 1.03 green prior to the loss of P-7C and the RAW score was determined and logged as 1.04 green after the P-7C coupling failure and subsequent removal from service. In comparison, if Service Water Pump, P-7B, is rendered inoperable, the RAW score would change to 3.77 yellow. Thus the necessary SWS equipment to safely shutdown the plant was intact and available to maintain "Nuclear" and "Public" safety.
Industrial Safety:
To repair P-7C, Service Water Pump within the allowed 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> LCO time frame a well thought out and planned evolution had to occur. The required the use of all the Human Performance Tools available to the site from Pre-job Briefings to the individual workers using STAR (Stop, Think, Act, & Review) and QV&V (Question, Verify, & Validate). A significant amount of Supervisory oversight was provided to Maintenance workers to provide the extra sets of eyes and ears on the scene for safety and repair accuracy. There are industrial safety challenges associated with removal of a Service Water Pump and OE was utilized. Operations Work Control Center personnel walked down the area with Mechanical Maintenance Supervision who had experience from the previous P-7C shaft coupling failure in October 2009.
The Screen House presents its own unique industrial safety challenges because of the noise level and water that is present on the floor along with FME concerns because of the hole that is created in the floor when the pump is removed. Adequate Industrial Safety actions were documented in the Work Order #286627, Mechanical Maintenance work instructions, WI-SWS-M-04, and Electrical Maintenance procedure SWS-E-5.
RCE Report CR-PLP-2011-03902 Page 33 of 136
Safety Significance Evaluation Radiological Safety:
There was no impact on radiological safety. P-7C, Service Water Pump, is located in the Screen House 590' elevation of the Turbine Building. The Service Water System is constantly monitored by RIA-0833, Service Water Discharge Monitor. RIA-0833 has a low flow alarm and radiation level readout in the Control Room for monitoring by the On-shift licensed operators.
Based on the above documented Safety Significance Evaluation, no further corrective actions are necessary.
RCE Report CR-PLP -2011-03902 Page 34 of 136
Corrective Action Plan Identified Cause Corrective Actions Responsible Due Date Dept.
Immediate Actions Processed EC31337 to change coupling material from 416SS to 17-4PH SS in Design (completed RC1 , CC1 order to minimize susceptibility to SCC MM/DD/YY)
Engineering Preemptively changed Line Shaft Couplings in P-7A/B/C to 17-4PH SS material. Maintenance WW 1139 RC1CC1 P-7A and P-7B are done, P-7C is scheduled for 9/29/11.
Interim Actions CR-PLP-2011 -03902 CA-00014 Engineering 12/01/2011 Conduct an Information Sharing with Engineering Department covering the EN-DC-141 "Design Inputs" Attachment 9.3, focusing on identifying the need for RC2, EOC,CC2, CC3, robust reviews and technical analysis if component materials are being changed.
Ensure that there is awareness amongst the Engineering Staff that a new Engineering Standard will formalize material change requirements including the level of necessary reviews.
CR-PLP-2011 -03902 CA-00015 Training 12/01/2011 Evaluate the need for additional training on the subject of Engineering RC1, RCZ, CC1, CCZ, CC3 Department and MP&C interfaces. Include the Critical Procurement Process (EN-MP-100) and Entergy requirements for the transmittal of Design Changes to suppliers. Create a TEAR to perform this evaluation. Circle back for CARB Chair approval if the evaluation determines that training is not required.
CR-PLP-2011 -03902 CA-00016 Engineering 12/01/2011 Conduct an Information Sharing with Engineering covering the effective use of CC3 Operating Experience during the performance of Engineering Changes. Ensure that there is awareness amongst the Engineering Staff that a new Engineering Standard will formalize OE requirements including the depth of analysis.
RCE Report CR-PLP-2011-03902 Page 35 of 136
Corrective Action Plan CR-PLP -2011 -03902 CA-00017 Engineering 12/01/2011 Directed action from CARB and the Director of Engineering. Conduct an extent of EOC condition analysis for all components supplied by HydroAire. Identify all tempered components and evaluate susceptibility to SCC in the components service environment (Dilution Water Pumps, Service Water Pumps, Heater Drain Pumps, Condensate Pumps, etc.)
CR-PLP -2011 -03902 CA-00018 Sys 10/28/2011 Other Update the Root Cause Evaluation after receipt of the LPI Metallurgical Analysis Engineering Report. Get CARB approval of final results regardless of the scope of changes.
CR-PLP -2011-03902 CA-00019 Design 12/01/2011 Review a sample size, per EN-QV-109, of modifications since 8/30/2007 Engineering EOC where materials changes were introduced. Evaluate the material selected and the robustness of the OE analysis. Create a new action and return to CARIB for review if the analysis shows a significant deficiency with regards to material chan g es.
CR-PLP-2011 -03902 CA-00020 Engineering 12/01/2011 EOC Review of sample size, per EN-QV-1 09, of RCE and HT ACE OE evaluations at Palisades since 8/30/2007. Evaluate the robustness of the OE Analysis. Create a new action and return to CARB for review if the analysis shows a significant deficiency with regards to OE analysis.
CR-PLP -2011-03902 CA-00021 MP&C 12/01/2011 Examine previously determined stock levels of Service Water Pump Other Shafts, Couplings and Spiders. Create and gain approval of an action plan to ensure sufficient on-hand stock of materials, such that the station can recover from a component failure in a timely manner. Return to GARB if approval for the action plan is not obtained.
RCE Report CR-PLP-2011-03902 Page 36 of 136
Corrective Action Plan CR-PLP-2011-03902 CA-00022 Training 12/01/2011 Evaluate revising the ESP Training plan to include initial and continuing CC3 training on effective OE analysis and documentation. Create a TEAR as necessary. Circle back for CARB approval if the evaluation determines that training is not required.
CR-PLP-2011-03902 CA-00023 Design 01/27/2012 Other Petition the fleet owner of EN-DC-115 to include reference to the PLP Engineering Engineering Standard being developed as a CAPR. If the fleet owner will not reference EN-DC-115, document the basis for the decision in this CA.
Short & Long Term Actions CAPR #1 (Corrective Action to Preclude Repetition) Design 12/01/2011 CR-PLP-2011 -03902 CA-00024 Engineering Create a limited distribution Engineering Standard for PLP that clearly identifies station requirements and expectations for Material Changes affecting installed RC1, RC2, CC1, CC2, CC3 plant equipment. Consideration should be given to specifying required analysis, guidance on obtaining outside assistance, specification of deliverables, analysis of the service environment and documenting analysis of relevant OE.
Guidance on OE analysis should be designated as being applicable to all Engineering tasks/reports that require OE searches and evaluation.
CAPR #2 (Corrective Action to Preclude Repetition ) Maintenance 12/01/2011 CR-PLP-2011 -03902 CA-00025 RC1, CC1 Replace 416 SS Service Water Line Shaft Couplings with 17-4PH couplings per EC #31337. Disposition work done in this CA. This action can be closed after all Service Water Pump Line Shaft Couplings have been changed.
Mot Keport CR-PLP- 2011-03902 Page 37 of 136
Effectiveness Review Plan LO-PLPLO-2011-00055 CAPR #1 Create a limited distribution Engineering Standard for PLP that clearly identifies station requirements and expectations for Material Changes affecting installed plant equipment. Consideration should be given to specifying required analysis, guidance on obtaining outside assistance, specification of deliverables, analysis of the service environment and documenting analysis of relevant OE. Guidance on OE analysis should be designated as being applicable to all Engineering tasks/reports that require OE searches and evaluation.
Action Resp. Dept Due Date Method : Analysis of EC packages Desi g n Eng Attributes : 416 SS (tempered) Desi g n Eng Success : None Specified in Engineering Changes where the service environment is Design Eng corrosive to 400 series stainless steels.
Timeliness : 1 year Design Eng 09/09/2012 LO-PLPLO-2011-00055 CAPR#2 Replace 416 SS Service Water Line Shaft Couplings with 17-4PH couplings per EC #31337. Disposition work done in this CA. This action can be closed after all Service Water Pump Line Shaft Couplings have been changed.
Action Resp . Dept Due Date Method : Analysis of WO Packages Maintenance Attributes : Completion notes and Work Package sign-offs Maintenance Success : Work Order Packages demonstrate the couplings were successfully Maintenance chan g ed.
Timeliness : 4 months Maintenance 01/30/2012 RGE Report CR-PLP-2011-03902 Page 38 of 136
References Documents reviewed :
CR-PLP-2009-04519 CR-PLP-2009-04806 F11358-R-001 DRAFT G.docx "METALLURGICAL AND FAILURE ANALYSIS OF SWS PUMP P-7C COUPLING #6" Personnel contacted :
Ed Huss / Entergy MP&C John Kasishke / Entergy MP&C Tom Reddy / Entergy MP&C John Petro / Entergy Receipt inspection Jim Alderink / Retired Entergy Design Eng.
Team Members :
Team Leader / Evaluator Paul M. Deniston / System Eng.
Jason Gosler / Maintenance Team Members Aaron Verzwyvelt / System Eng. Ben Gumieny / Procurement Eng.
Jim Forehand / System Eng Sontra Yim / LPI Kevin Rose / Maintenance Ian Wilson / Manncini and Assoc Rich Margol / Training George Licina / Structural Integrity Bill Townes / Operations Dana Cooley / SeaState Group Inc.
Mike McCarthy / Design Eng.
Analysis Methodologies Used :
- 1. Event & Causal Factor Charting
- 2. Failure Mode Analysis
- 3. Metallurgical Analysis Attachments :
- 1. Event & Causal Factor Chart
- 2. Failure Mode Analysis
- 3. Safety Culture Evaluation
- 4. Detailed Operating Experience
Attachment I - Event & Casual Factor Chart Event & Causal Factor Chart CR-PLP-2009-04519 On Tuesday, September 29 2009 at 0908 hours0.0105 days <br />0.252 hours <br />0.0015 weeks <br />3.45494e-4 months <br />, Service Water Pump P-7C failed to provide discharge pressure to the Service Water Header, resulting in the station entering Off Norma/ Procedure 6.1 (©NP-6.1) loss of Service Water" and entry into LCO 3.7.8.1 Action A. 1 (72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />).
E&CF KEY Presumptive Primary Events Grey Shading /Primary Effect Secondary Event Terminal Event Ac,ions or happenings htdicates events occomng Undesirable event or Actions or happenings that directly leading up to the outside of Entergy`s wiring that awas critical for The 'Pr oblem" are impacted by the primary undesirable event the situation to occur. / being salved event but are not directly control involved.
Condition Presumptive Primary Effects Data supporting
' id o'or further t Causal Factor (Postulated condition that may have been detailin t t e event.
present during the event, but for which there is nsuffctantevidence to further confirm or ohminato its presrnce. )
Presumptive Causal Factors (rkistrtlated condition that may have been present ru s-tog the event, but for which there is "nsulficierrt awderee to further contimr or eliminate its presence.) NOTE:
Bold lines indicates primary events.
Failed Barrier Non-bold lines indicate secondary (Inetlectwe, Weati or events Missing Barriers)
Dashed outlines indicate presumptive conditions.
RCE Report CR-PLP- 2011-03902 Page 40 of 136
Attachment I - Event & Casual Factor Chart
" " 5117f4999 M2112004 Operating Experlance Operating Experience Operating Experience Initiated Fun Failure of pump shaft type Vertical deep d r aft pump 410 stainless steel non Equipment Substitution Coupling failures ailulu from shaft and cr p failures coupling caused by (F ES) Evaluation due to inter 9 r stress from intergranular i
T emper E brittle r in g' Vendor recommendation A corrosion cracking ccracking stress corrosion to change the line shaft inadequate Tempering (IGSCC) of 416 stainless (IGSCC) of 410 stai n less from Carbon Steel 01045 during manufacture, steel.
to 416 Stainless Steel. steel-INRC Information Notice NRC Infor eel Notice FE5 99 tKi1 Perry Root Gause fit? 04.02598 (IN) $3-fry f3Nr maUon05 rIr1 GC3 GC3 Event occurred at Beaver This r:hanne --s Event o curred at slot initiated as a result of Evaluated under L ©-PLPL©-
Valley on June 20. 1991 . I Columbia neratln 2007.40059 Station or. June 13, 2005.
Included in NRC lnfornnation Evaluated Notice 2W7-05 P alisadc-, under n hnld mi n Pr o dre .17. LO 00500 Technical Review completed ' ( concluded that IN 2007-05 did ^ of 511,1 apply.
Palisades review of IN 2007-05 failed to Properly difffrerrtiale
?tween Austenitic and Martensit stainless steel, r RCE Report CR-PLP- 2011-03902 Page 41 of 136
Attachment I - Event & Casual Factor Chart 121191:20117 P-7Ar P-7B & P-7C A Service Water Pump Issued 1108/2008 Refurbishment Design Purchase Order Palisades received Configuration Changes/ for P-7A. P-7A from HydroAire.
Documentation closed.
PO 101 9G242 EC 55009121762
-- The fullo.vi g barriers ware considered missed opportunities: the P-7A was shipped in pieces, as expected. due to its size.
potential exists that these missed opportunil es would have altered 1he Oitcoroe of this event.
PLPL0-2Q07-0OO54 (review of di did not acknow ledge that there are similarities between 410 and 416 stainless steels.
E0 50001 21 762 (Service Water"-..
Pump Change) did not document acknowledgement that a fresh water environment shoutd be considered as a.
potentially ccxrsive entuiroiment CC2 RCE Report CR-PLP-2011-03902 Page 42 of 136
Attachment I - Event & Casual Factor Chart Refueling Outage I R20 511 912009 062009 6/1412009 The old P-7C failed to 13 Issued Purchase Palisades replaced Palisades performed C meet its performance Order for P-7C.
P-7A. a Receipt Inspection criteria, during routine testing. PO a C123'1149 at HydroAire.
Con feted under Work Testing performed Order 51637416, in acoDrdunce with RO-144, Receipt Inspection Gore ehernsive Purrip Test completed satisfactory.
Procedure SerAce Water Pumps P-7A, P-7B, & P*7C. ,
P-7C failed to meet critera due to excessive wear and degredation.
Failure prompted decision to rep ace P-7C.
'-7A did not require testin, under RO-144 due to t he recent replacement.
P-7 tested satisfactory.
RCE Report CR-PLP-2011-03902 Page 43 of 136
Attachment I - Event & Casual Factor Chart 611 W2009 6110-6111,2009 611012009 611 112009 @ 0617 hrs G Engineering Change P-7C staged in Palisades received P-7C removed from approved for P-7G. Protected Area for P-7C from Hyd roAire. service.
EC 14577 installation, P-7C was shipped in pieces, as expected, due to its size.
6412/2003 @ 1634 hrs 712312009 917212003 06t12J2009 P-7C returned to Quarterly Inservice Quarterly Inservice P-7C replaced under service. Test Procedure - Test Procedure -
Work Order 190235. Service Water Pumps. Service Water Pumps.
QO-14C 0O-14C OO-14C Testad Satisfactory, Tested Satisfactory.
912912009 dal 0735 hrs Cycled Service Water P-7.A and 7C D in service-Routine evolution to clear tlw screens, RCE Report CR-PLP-2011-03902 Page 44 of 136
Attachment I - Event & Casual Factor Chart 09/2912009 a 0908 hrs 1010112009 @ 0822 hrs P-7C failed to provide P-7C returned to End of inilial E&CF discharge pressure. service.
'idicatien of failure via alarm P-70 replaced under Work an Control Roorn and Ioc Order 208591 response by AO.
Entered unplanned Limiting Exited Limiting Condition for Condition for Operations a' Operations at 0822 hrs.
0()08 ttr5 As of 9 2r9i21N ,
P-7C had 13 starts and 2,414.44 run hours.
I )
RCE Report CR-PLP-2011-03902 Page 45 of 136
Attachment I - Event & Casual Factor Chart CR-PLP-70t9-4806 Service Water Pump irgn Engineering E Start of 2011 E&CF System Engineering manufacturing and design Manager directed HA to contacted HydroAire {H.Ay cnanyes did riot result In e atme second heat perform for support Purchase Order {PO) treatrrrent and perform revisions- 100% independent verificatk)n .
ysterr^ £r^girtetrir There was no formal request mmitted verbal requests to for HA supl xt KA A agreed to independently CGS es4 sentatnre sampla.
IP&G was unaware that HA was being solicited for su pport, 1o P( Fe^ision per CR-PLP-2009-04806 Ctoipliings tempered 1n
-1050F for 4-5.7hrs CCI CR-PLP-2005-04659 Review of manufacturing F records could not validate that independen?
verification had ex.r rred RUE Report CR-PLP-2011-03902 Page 46 of 136
Attachment I - Event & Casual Factor Chart Decisw, to uti lize doub'e 2010 Praire Island CE tempered vlartensihc 416 stainless steel frorn HA. CC3
°dependenthardness testfor thi AND first two coupling not performed No CR generated to track ST`
recDOrrsTiendekiona for changes 4 based ors PI CE vendor documentation to Manufacturirg eptible material 416 testing requirements. descrepancies. stainless steel, r5 SS is volnerable to str'e defined or tested, inconsistent hardness results corrosion cracking (SCC)
CC2 - CGt.
Tempering process change.
gait Hardness Test Performed RCE Report CR-PLP-2011-03902 Page 47 of 136
Attachment I - Event & Casual Factor Chart KUL FCeport CR-PLP- 2011-03902 Page 48 of 136
Attachment I - Event & Casual Factor Chart Crack initiated on Crack grows Torsion failure of the interior of through the wall at remaining coupling coupling coupling. material.
Indications of F - 7C repaired sheared shaft on sunder v ork order Crud of 2011 E&CF P-7C #286627 Entered ONP*6.1 (lass of #G coutali ng Service Water and L 3 7,8 (72 hr).
Initiated work KU L Report CR-PLP-2011-03902 Page 49 of 136
Attachment II - Failure Mode Analysis Note: This FMA is a living document. Some items are awaiting analysis that is incoming from LPI.
Failure Mode (yellow lines cannot be Aspect Refute Support Assigned to Action Items Priority Results completed within RCE team Based on the surface hardness results in Table Report No.
Hardness on the F11358-R-001 Hardness Page 21 of 51 measurements ends and hardness Revision DRAFT of the failed on circumferential G coupling direction, through thickness on axial 3-5, five couplings indicate an (11-P7C-4, 11-Coupling average slice P7C-6F, 11-P7C-Design or 1) Improper coupling hardness Hardness testing on hardness Yim (LPI) High 7 , 11-P7C-8 , and Manufacture value of 33.3 seven other coupling 09-Rockwell C, for the same pump P7C-7F) exhibit which exceeds Compositional analysis of threading surface hardness the specified above tolerance of 32 Retest failed coupling from 2009 specification, and Rockwell C one coupling failure event (11-P7C-3) exhibits surface hardness at the upper limit of the s p ecification.
Kl:t Keport CR-PLP-2011-03902 Page 50 of 136