IR 05000255/2011012
| ML112780190 | |
| Person / Time | |
|---|---|
| Site: | Palisades  |
| Issue date: | 10/04/2011 |
| From: | West S K Division Reactor Projects III |
| To: | Vitale A Entergy Nuclear Operations |
| References | |
| IR-11-012 | |
| Download: ML112780190 (33) | |
Text
October 4, 2011
Mr. Anthony Vitale Vice-President, Operations Entergy Nuclear Operations, Inc. Palisades Nuclear Plant 27780 Blue Star Memorial Highway Covert, MI 49043-9530
SUBJECT: PALISADES NUCLEAR PLANT - NRC SPECIAL INSPECTION TEAM (SIT) REPORT 05000255/2011012
Dear Mr. Vitale:
On September 1, 2011, the United States Nuclear Regulatory Commission (NRC) completed a special inspection at your Palisades Nuclear Plant. The inspection was conducted in response to the circumstances surrounding the failure of the Service Water Pump P-7C, on August 9, 2011. Based on the risk and deterministic criteria specified in Management Directive 8.3, "NRC Incident Investigation Program," a special inspection was initiated in accordance with Inspection Procedure 93812, "Special Inspection." The special inspection charter (Attachment 2 of the enclosure) provides the basis and focus areas for the inspection. The enclosed inspection report documents the inspection results, which were discussed with you and other members of your staff at the exit meeting on September 1, 2011. The determination that the special inspection would be conducted was made on August 15, 2011, and the on-site inspection commenced the same day. The inspection examined activities conducted under your license as they relate to safety, compliance with the Commission's rules and regulations, and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, and interviewed personnel. Based on the results of this inspection, no findings of significance were identified. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room and from the Publicly Available Records (PARS) component of NRC's document system, Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.qov/readinq-rm/adams.html (the Public Electronic Reading Room).
Sincerely,/RA/
Steven West, Director Division of Reactor Projects Docket No. 50-255 License No. DPR-20
Enclosure:
Inspection Report 05000255/2011012;
w/Attachments:
1. Supplemental Information 2. Memo to Jandovitz 3. Palisades Timeline cc w/encl: Distribution via ListServ Enclosure U. S. NUCLEAR REGULATORY COMMISSION REGION III Docket No.: 50-255 License No.: DPR-20 Report No.: 05000255/2011012 Licensee: Entergy Nuclear Operations, Inc. Facility: Palisades Nuclear Plant Location: Covert, MI Dates: August 15, 2011 - September 1, 2011 Inspectors: J. Jandovitz, Project Engineer (Lead) A. Shaikh, Reactor Inspector P. Smagacz, Reactor Engineer Approved by: John B. Giessner, Chief Branch 4 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000255/2010012; 08/15/2011 09/01/2011; Palisades Nuclear Plant; Inspection Procedure 93812, Special Inspection. This report covers a 4-day period (August 15 -19, 2011) of on-site inspection and in-office review through September 1, 2011. A team, comprised of three regional inspectors, conducted this special inspection. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006. A. No findings were identified.
NRC-Identified and Self-Revealing Findings
B. No findings were identified.
Licensee-Identified Violations
REPORT DETAILS
At 12:02 p.m. EDT on August 9, 2011, the licensee entered an unplanned 72-hour shutdown Technical Specification limiting condition for operation (LCO) action statement (TSAC) due to failure of service water (SW) pump P-7C. The SW system includes three motor-driven, vertical multistage deep draft pumps that provide cooling water to safety related equipment (component cooling water, containment air coolers, diesel generators, control room coolers, etc.). Each pump has 50 percent capacity; and two pumps are required to be in service; however, the technical specification requires all three pumps to be operable. All three pumps were running prior to the failure. Summary of the Degraded Condition The licensee observed a low differential pressure condition and low pump amperage following the failure of the pump. The licensee disassembled pump P-7C and determined that the pump's first line shaft to second line shaft coupling, #6, had failed. The licensee replaced all of the pump P-7C couplings and two portions of the pump shaft. After completing the required maintenance and surveillance tests, the licensee returned pump P-7C to service on August 12, within the time allowed by the TSAC. Pump P-7C had previously experienced another coupling failure in September 2009. The SW pump couplings in both cases were made of 416 Stainless Steel. Subsequent metallurgical analysis determined the coupling's hardness was significantly higher than the design specification; and the failure was caused by Intergranular Stress Corrosion Cracking (IGSCC). The licensee's corrective actions in 2009 included replacing all the SW pump P-7C couplings with couplings of the same design and hardness criteria. However, hardness test results of several couplings removed from pump P-7C on August 10, 2011, indicated that these couplings were not within the hardness specification. Specifically, three of the couplings had at least one test location where the hardness measurement was higher than the licensee's design specification criteria. Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, "NRC Incident Investigation Program," a special inspection was initiated in accordance with NRC Inspection Procedure 93812, "Special Inspection Team." The special inspection charter, dated August 15, 2011, is included as Attachment 2. The team reviewed technical and design documents, procedures, maintenance records, and corrective action documents; interviewed station personnel and consultants; and performed plant walkdowns of plant equipment. A list of specific documents reviewed is provided in Attachment 1. Inspection Scope
4OA5 Other Activities - Special InspectionIn accordance with the Charter, the following items were reviewed.
contained in Attachment 3. b. No findings were identified. Findings and Observations The inspectors were concerned that the licensee had not adequately evaluated operating experience to recognize the vulnerabilities of 416SS material to IGSCC based on material properties, particularly heat treatment. This concern is discussed further in the unresolved item (URI) in Section
.07 of this report.
However, the inspectors did not identify any operational or surveillance results or observations that may have predicted the failure of SW pump P-7C on August 9, 2011.
.2 Review the licensee's basis for operability of the service water pumps P-7A, P-7B, and P-7C.
Determine whether the licensee had an adequately supported basis for operability of pumps P-7A and P-7B and for returning pump P-7C to service. Determine if additional corrective actions are requireda. . The inspectors reviewed the prompt operability basis for all three SW pumps contained in Engineering Change (EC) 31208 and EC 31218. The inspectors identified run times, service conditions and life, starts and stops, heats of materials, and differences in heat treatment for the in-service couplings. Inspection Scope b. No findings were identified. Findings and Observations Based on the information available, the inspectors did not disagree with the immediate or prompt operability evaluations. The prompt operability for SW pumps P-7B and P-7C was based on the higher confidence that the hardness of the couplings installed in the pumps were within the design specification range, and therefore less susceptible to IGSCC. Operability for pump P-7A was based on the fact that couplings installed in that pump had been in operation the longest with no failures. However, the inspectors noted that there was no discussion on temper embrittlement identified in the Structural Integrity (SI) report and in operating experience, as a factor that may affect the couplings' reliability. Therefore, it was recognized that the results from the metallurgy and root cause evaluation may impact the basis for current operability and the couplings' service life. After the on-site SIT inspection was complete, the licensee received initial metallurgical results on the failed coupling and initiated a near-term replacement plan of the couplings in all three SW pumps with couplings of a different material, more resistant to IGSCC. At the time of the SIT exit, the couplings in pump P-7A had been replaced with the new material. The NRC will continue to monitor results from the root cause investigation for impact on the operability basis for any pump that still has the 416SS couplings.
The residents will also evaluate the licensee's evaluation of past operability for all the SW pumps and the extent of condition evaluation for the 416SS components remaining in the pumps.
.3 Evaluate activities associated with the licensee's repair of service water pump P-7C for adequacy.
Confirm that the repair complies with licensee and NRC requirementsa. . The inspectors reviewed the licensee's repair and replacement activities associated with the rebuild of pump P-7C following the 2011 pump coupling failure. This review included the document review of post-maintenance testing of the pump as well as documentation of pre-installation testing of pump and motor. Also reviewed was the completed work order for this repair and replacement, and the material specifications (heat treat curves, hardness results) of couplings that were installed in this rebuilt pump P-7C. Inspection Scope b. No findings were identified Findings and Observations The inspectors did not identify any issues relating to the 2011 pump P-7C repair.
.4 Review information and processes relied upon by Palisades to determine whether or not critical attributes of service water pump line shaft couplings, such as hardness, are acceptable prior to coupling installationa. . The inspectors reviewed the information provided by the licensee regarding pump shaft couplings that included hardness measurements at time of procurement, independent lab results for coupling hardness measurements prior to coupling installation in the pumps, and the associated heat treatments for the couplings.
Also reviewed were Inspection Scope industry reports and operating experience regarding use of 416 Martensitic SS for these couplings and the 2009 SW pump P-7C coupling failure root cause evaluation. b. No findings were identified. Findings and Observations The inspectors identified a concern with the adequacy of specifying coupling hardness as the only critical material parameter. This was not a conservative approach given the collection of scientific literature and industry operating experience describing the vulnerability of 416/410 Martensitic SS to stress corrosion cracking; and suggested use of toughness measurements in addition to hardness values to determine acceptability of 416/410 Martensitic SS as coupling material. However, ensuring hardness that values are below a certain allowable maximum, reduces the vulnerability of the couplings to IGSCC; but hardness itself may not be sufficient to ensure critical parameters are met. This concern is captured as part of the URI included in Section
.07 of this report. In late 2007, an engineering change was initiated to replace the carbon steel components in the SW pumps with stainless steel components resistant to the erosion and corrosion issues the plant had experienced throughout its life.
In 2008, couplings for the SW pumps were procured and fabricated from 416SS with specified criteria for material chemistry and a Rockwell C (Rc) hardness range of 28-32 Rc. Information available from industry operating experience referenced that material, specifically water pump couplings, made from 410/416 SS were susceptible to IGSCC due to temper embrittlement and improper heat treatment. The 2009 Palisades failure of SW pump P-7C coupling was attributed to IGSCC due to significantly high hardness, ranging from 37-41 Rc; and all couplings were replaced with the same material and design specification for hardness. No additional criteria were specified for tempering or heat treatment. Results from a similar coupling failure at Prairie Island in 2010, showed that their coupling, supplied by same vendor as the Palisades pump couplings, failed due to IGSCC even though the hardness was near the specification criteria. Absent any design criteria from the licensee, the heat treatment performed on the couplings was focused on the final product having a hardness value in the specified range of 28-32 Rc. There was no consideration of the effect of heat treatment on other material properties, such as toughness, that may affect susceptibility to IGSCC. The inspectors noted the heat treatments were different for couplings for each of the three pumps. Couplings for pumps P-7B and P-7C were double-tempered. Also, the procedure used by the heat treatment vendor allowed multiple heat treatments (tempering) to achieve the desired hardness, and did not consider the affect of this process on IGSCC resistance. Because of the literature and industry operating experience of the vulnerability of 416/410 Martensitic SS to IGSCC, the plant requested Structural Integrity to evaluate their 416SS couplings. The report was received in March 2011, and suggested that the couplings were susceptible to temper embrittlement, but did not conclude that there was an immediate operability concern. The plant was evaluating this information when the coupling failed on August 9.
.5 Evaluate the licensee's procedures and processes for evaluating/assessing the performance of vendors.
Emphasis should be placed upon determining why couplings that were believed to be of an appropriate hardness were later found to be of an inappropriate hardness. Review any evaluations/assessments of HydroAire that were completed by the licensee following the September 2009 service water pump P-7C coupling failure. Also, review any actions taken or direction provided by the licensee to HydroAire in response to the recent hardness test resultsa. Inspection Scope . The inspectors reviewed procurement procedures for the licensee and vendor, the 2009 root cause evaluation, source surveillances completed since 2009, the industry audit of HydroAire, condition reports, corrective actions taken by the licensee and vendor after the 2009 coupling failure, purchase documents, and interviewed Quality Assurance personnel.
b. Findings and Observations
No findings were identified. The inspectors identified a concern that the licensee may not have provided adequate vendor procurement control after the 2009 coupling failure, which may have contributed to the 2011 coupling failure. The inspectors found that adequate procurement procedures and processes existed, but were not fully utilized after the 2009 coupling failure. This concern is included in the URI discussion in Section
.07 of this report. The inspectors reviewed a vendor surveillance performed in March 2010 on parts for a spare SW pump.
The surveillance noted that one pump part was heat-treated, and the hardness readings taken by the pump vendor and the heat treatment vendor were both out of the specified range. Therefore, the part was again heat-treated, resulting in acceptable hardness values obtained by both vendors. The part was then identified as acceptable. This demonstrated that as late as 2010, there were still issues with the controls on heat treatment for SW pump parts. Implementation of the corrective actions from the 2009 coupling failure resulted in some improvements to the vendor procurement processes and additional oversight. This was based on: frequent vendor surveillance activities were conducted on new parts that found vendor performance was satisfactory; procedure EN-MP-100, "Critical Procurements," was implemented for procurements after 2009; and a NUPIC [Nuclear Procurement Issues Committee] audit conducted in April 2011 found that HydroAire was implementing an effective quality assurance (QA) program.
This is not concluding the design criteria supplied to the vendor was correct, as discussed earlier, only that the parts supplied by the vendor appeared to meet licensee criteria. No additional actions were taken by or required of the vendor after the 2011 coupling failure, since the licensee considered this failure the result of activities that occurred in 2009, and that corrective actions implemented by the vendor then were effective.
Specific attention should be given to the licensee's evaluation of a 2010 service water pump coupling failure at Prairie Island and the evaluation of a March 2011 report from Structural Integrity regarding the potential for coupling failure due to temper embrittlement.
a. Inspection Scope
Operating experience regarding failures of couplings using 410/416 Martensitic SS was reviewed along with generic technical reports and metallurgical reports. The report of most interest were the Prairie Island coupling failure in 2010, the Structural Integrity report which was issued to address the 2009 coupling failure at Palisades, and the 2009 Palisades coupling failure root cause evaluation.
b. Findings and Observations
No findings were identified. The inspectors identified a concern that operating experience and industry information may not have been incorporated into the design of the pump nor addressed adequately in the 2009 coupling failure replacement and root cause evaluation. Further discussion of this concern is included in the URI discussion in section
.07. Operating experience that existed as early as 1991 discussed failures of pump couplings attributed to embrittlement of 410SS and 416SS due to improper tempering temperatures and heat treatment.
In addition, industry and NRC reports were available prior to 2009 and discussed the effect of temper embrittlement on 410/416SS material. The SI report was issued in March 2011, but the licensee did not accept the report until August. This report discussed temper embrittlement as an issue that could affect coupling reliability. The licensee had not yet evaluated nor initiated corrective actions based on the report. The inspectors found the licensee did identify pertinent operating experience in development of the 2011 coupling failure root cause evaluation, although the root cause report was not issued by the completion of this SIT report.
.7 Review the licensee's root cause evaluation plan and schedule.
Include the schedule for performing testing of any components removed from pump P-7C. Evaluate whether the root cause evaluation plan is of sufficient depth and breadth. Confirm that the time allowed to perform the root cause evaluation is commensurate with the safety significance of this issue. Communicate to the licensee that the NRC will inspect the completed root cause evaluation and the associated corrective actions as part of our normal inspection activities.
a. Inspection Scope
The inspectors reviewed the root cause charter, schedule, team make-up and action plan; and discussed actions with root cause team. b. Observations The inspectors found the root cause team was comprised of individuals dedicated solely to the root cause investigation; they had no additional responsibilities. Most members had not been involved in the root cause for the 2009 failure. Team members included consultants, including an independent pump manufacturer and the metallurgical lab contracted to perform the metallurgical analysis. The pump supplier, HydroAire, provided input to the root cause but was not a member. The root cause timeline started August 16, and was scheduled to be completed within 30 days in accordance with procedures. The root cause investigation was determined to be of adequate depth and breadth to be successful in determining the actual root cause. The action plan included the following aspects: improper hardness; improper material; over-torque from foreign material; shaft misalignment; Stress Corrosion Cracking; incorrect clearances in coupling to shaft; shaft faces not square; pump alignment; shaft wobble; reverse rotation; coupling defects; operation at critical speed; operation loads; strainer blockage; maintenance practices; and fatigue failure. The inspectors reviewed the draft metallurgical plan and considered the metallurgical analysis plan for selected coupling to be thorough, although the licensee had not yet finalized the plan or determined which couplings would be included in the plan. The NRC will continue to inspect the metallurgical results and root cause report under the normal baseline procedures.
Unresolved Item: Adequacy of SW Pump Couplings
Introduction:
A URI was identified by the inspectors during review of the SW pump P-7C coupling failure in 2011. Specifically, the inspectors were concerned with the adequacy of the procurement of the replacement couplings after the 2009 coupling failure on the same pump; whether operating experience was adequately incorporated into design specifications; and whether the corrective actions taken after the 2009 failure were adequate to prevent recurrence.
Description:
Palisades has three safety-related SW pumps that draw water from Lake Michigan to provide cooling to safety-related and non-safety-related components. Each pump has eight couplings to connect shaft sections. In 2009, the plant began a material change for the couplings and shafts from carbon steel to 416 Martensitic SS due to erosion issues of the carbon steel material. The material would be changed in series in of pumps P-7A, P-7C, and P-7B respectively. The timeline (Attachment 3) includes specific dates. In 2009, SW pump P-7C failed due to cracked coupling #7, resulting into entry a TSAC, requiring the plant to shut down if the pump was not returned to service within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. Metallurgical data on the failed coupling revealed the coupling hardness was significantly higher than the design specification. The specified hardness range was Rockwell C value of 28-32 Rc. Hardness readings on the failed coupling ranged from 37-41 Rc. The vendor, HydroAire, had supplied the couplings with certifications that the couplings were within the specified hardness. This failure was entered into the licensee's corrective action program and identified as a significant condition adverse to quality (SCAQ). At the time of the failure, no spare couplings were available. Immediate discussions with HydroAire ensued to obtain replacement couplings within the 72-hour timeframe. Based on the hardness data received on the failed coupling, which conflicted with the certification received from the vendor, the licensee verbally imposed additional requirements to obtain independent laboratory hardness checks on the replacement couplings. The requirement for independent hardness checks on the replacement couplings was not incorporated into the procurement process. Procurement documents were not updated with this additional requirement, and procurement personnel, such as the licensee procurement manager at the vendor site or the plant receipt inspection personnel, were not made aware of the additional requirement. Procedure EN-MP-100, "Critical Procurements," was not implemented even though criteria in the procedure, such as previous vendor issues, how critical the couplings were on plant operation, and the short procurement timeframe, appear to have been satisfied. Eight replacement couplings were received and immediately installed in the pump; the pump was returned to service within the 72-hour requirement. Soon after the pump was returned to service, the licensee discovered that independent hardness checks were not performed. The vendor did provide certifications indicating that the couplings had hardness values in the specified range. An additional ten couplings were ordered and received soon after the pump was returned to service. Independent hardness checks completed on these spare couplings agreed with vendor hardness results, thus providing some assurance to the licensee that the couplings currently installed in pump P-7C were of the correct hardness. On August 9, 2011, SW pump P-7C was declared inoperable due to the failure of coupling #6. This failure was entered into the licensee corrective action program and identified as a SCAQ. The cause of the failure had not been determined at the time this SIT report was completed. However, hardness results obtained from an independent metallurgical lab from four of the in-service couplings, including the coupling that failed, were found to have at least one spot of higher hardness than specified, greater than Rockwell Hardness value of 32 Rc. No hardness values approached the 37-41 Rc observed in the 2009 failure (the values were less than 34Rc). Therefore, the inspectors were concerned that the licensee did not ensure the vendor-supplied couplings with correct hardness values in 2009. Pending final results and conclusions from the metallurgical analysis and root cause analysis, the licensee may have failed to take measures in 2009 to assure that purchased material and equipment conform to the procurement documents, including provisions for, as appropriate, objective evidence of quality furnished by the contractor as required by Appendix B. Operating experience available to the licensee from 1991 through 2009, identified that Martensitic 416SS could be more susceptible to IGSCC, dependant on the heat treating process. Some of the previous operating experience is listed below: September 12, 1991 - Beaver Valley Unit 1 experienced a coupling failure of a Byron-Jackson centrifugal pump as a result of embrittlement of 410SS due to improper tempering temperatures and potential impurities. The failure mechanism was IGSCC; September 1, 1993 - NRC Information Notice 93-68, "Failure of Pump Shaft Coupling Caused by Temper Embrittlement During Manufacture," is issued informing licensees of problems stemming from temper embrittlement of 410SS supplied by Byron-Jackson (referring to the 1991 Beaver Valley event); the document noted in the material analysis that the toughness of the failed component was extremely low; September 22, 1993 - Indian Point Unit 2 experienced a pump failure due to a failed coupling attributed to a non-ductile fracture because of temper embrittlement of the 410SST used; the failure mechanism was IGSCC; September 1, 2003 - Perry Unit 1 experienced a coupling disassembly event on their emergency SW pump A. The failure mechanism was by IGSCC and no performance effects were noticed before the failure of the coupling; May 21, 2004 - Perry Unit 1 experienced another emergency SW pump coupling failure; the failure mechanism was by IGSCC, similar to the 2003 event; 2006 - INPO Operating Experience Digest 2006-02 - INPO released a document that discusses SW pump (shaft, coupling and impeller) failures occurring in the industry. The report noted that 12 failures occurred between 1998 and 2006, with the most frequent including corrosion causing coupling separation. One of the common causes identified was improper heat treatment of the material during manufacturing; and February 9, 2007 - NRC Information Notice 2007-05, "Vertical Deep Draft Pump Shaft and Coupling Failures," was issued and referenced the coupling failures from IGSCC that occurred at Columbia Generating Station. The couplings were 410 Martensitic SS and were susceptible to tempering embrittlement. The operating experience review identified at least 23 essential SW pump shaft and coupling failures since 1983 involving more than six different manufacturers.
The 2009 coupling failure root cause evaluation was focused on the material hardness. It did not fully evaluate the effect of other material properties, such as toughness which is discussed in the operating experience, nor the change of material properties from heat treatment on the susceptibility of the couplings to IGSCC. Corrective actions after the 2009 failure did not address the heat treatment process. The inspectors were concerned that the licensee may not have adequately addressed operating experience in the design of the original 416SS couplings or the 2009 replacement couplings as required by Appendix B, Criteria III. And the corrective actions taken after the 2009 failure may not have prevented recurrence of this SCAQ as required by Appendix B, Criteria XVI. These concerns will be further evaluated as part of the NRC baseline inspection program, after the final metallurgical results are issued and the licensee root cause evaluation is completed. (URI 05000255/2011012-01; Adequacy of SW Pump Couplings).
4OA6 Meetings, Including Exit On September 1, 2011, the special inspection team leader presented the preliminary inspection results to Mr. Anthony Vitale and members of his staff.
No proprietary information is included in this inspection report. ATTACHMENT:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- T. Vitale, Entergy/Site Vice President
- J. Haumersen, Entergy/System Engineering Manager
- B. Kemp, Entergy/Design Engineering Manager
- D. MacMaster, Entergy/Acting Design Engineering Manager
- D. Hamilton, Entergy/General Manager Plant Operations
- A. Blind, Entergy/Engineering Director D.Malone, Entergy/Acting Nuclear Safety Assurance Dir.
- J. C. Plachta, Entergy/Quality Assurance Manager
- P. Deniston, Entergy/Sr. Engineer
- J. Forehand, Entergy/Engineering Supervisor
NRC Personnel
J Ellegood, Senior Resident Inspector, Palisades
- J. Giessner, Branch Chief
- K. Stoedter, Acting Branch Chief
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened and Closed
None
Closed
- None
Opened
- 05000255/2011012-01 URI Adequacy of SW Pump Couplings
LIST OF DOCUMENTS REVIEWED
Condition Reports
- Title Date/Revision No.
- CR 04-02598 Root Cause Report for Perry Emergency Service Water Pump A Upper Shaft Coupling Sleeve Repeat Failure.
- CR 2011-03975 Relevant OE Not Considered in
- CR-PLP-2009-4519 (Root Cause Evaluation for the 2009 Service Water Pump P-7C coupling Failure) August 12, 2011
- CR-PLP-2009-04519 Root Cause Evaluation Report for Service Water Pump P-7C Failure to Provide Discharge Pressure March 4, 2009
- CR-PLP-2009-04542 Evaluate Early Station Response to Failure of P-7C September 30, 2009
- CR-PLP-2009-04547 Inadequate FME Postings and Boundaries for Initial Disassembly of P-7C September 30, 2009
- CR-PLP-2009-04561 Job Safety Hazard Analysis Not at Job Location for P-7C September 30, 2009
- CR-PLP-2009-04571 Station in Event Response for Unexpected Loss of Service Water Pump P-7C September 30, 2009
- CR-PLP-2009-04593 Steps Not Followed Within
- WI-SWS-M-04 for Reassembly of P-7C October 1, 2009
- CR-PLP-2009-04613 Administrative Issues Involving Work Scheduling in Response to Emergent Equipment Issues October 2, 2009
- CR-PLP-2009-04616 Procedure Not In-Hand for Packing Adjustments for P-7C Pump Shaft October 2, 2009
- CR-PLP-2009-04659 Initial CR for
- CR-PLP-2009-04806 October 6, 2009
- CR-PLP-2009-04806 Apparent Cause Evaluation to Evaluate Miscommunication of Independent Testing October 15, 2009
- CR-PLP-2009-04877 Service Water Pump P-7C Exceeds Maintenance Rule Unavailability October 21, 2009
- CR-PLP-2009-05647 Documentation of HydroAire Root Cause Evaluation of Failed Line Shaft Coupling on P-7C December 9, 2009
- CR-PLP-2009-05874 Evaluation of Past Operability Issues of P-7C Prior to the Event on 09/29/2009 December 22, 2009
- CR-PLP-2010-01015 Error Identified in Licensee Event Report 2010-001, Potential Loss of Safety Function Due to a Service Water Pump Shaft Coupling Failure March 10, 2010
- CR-PLP-2011-03902 Root Cause Report Service Water Pump 7-C Line Shaft Coupling Failure September 8, 2011
- CR-PLP-2011-03920 Non-covered Worker Has Exceeded Limits of
- EN-FAP-OM-006 August 10, 2011
- CR-PLP-2011-03921 A-Frame Designated for Lifting P-7C Service Water Pump Missing August 10, 2011
- CR-PLP-2011-03929 Non-Conforming Components Removed from P-7C August 10, 2011
- CR-PLP-2011-03938 Outside Diameter Value and/or Tolerance in Error August 10, 2011
- CR-PLP-2011-03939 Degradation of Coating with Blisters on the Discharge Head of P-7C August 10, 2011
- CR-PLP-2011-03942 Non-covered Worker Has Exceeded Limits of
- EN-FAP-OM-006 August 11, 2011
- CR-PLP-2011-03944 Personnel Pulled from Training for Plant Event Response August 11, 2011
- CR-PLP-2011-03946 P-7C Work Temporarily Stopped Due to Non-Tethered Tools in FME Zone 1 August 11, 2011
- CR-PLP-2011-03947 Hardhat Lost During P-7C Disassembly August 11, 2011
- CR-PLP-2011-03948 Issues with Licensed Operator Training Identified August 11, 2011
- CR-PLP-2011-03952 Chlorination Deferment without Potential Adverse Effect August 11, 2011
- CR-PLP-2011-03959 Violation of Protected Equipment Boundary August 11, 2011
- CR-PLP-2011-03961 Potential Extent of Condition on Service Water Pumps P-7A and P-7B August 11, 2011
- CR-PLP-2011-03963 Reverse Rotation Noticed of P-7C During Breaker Opening August 11, 2011
- CR-PLP-2011-03965 Engineer Exceeds Overtime Limit August 11, 2011
- CR-PLP-2011-03966 Removed Couplings from P-7C 2011 Failure Show Out-of-Spec Hardness August 12, 2011
- CR-PLP-2011-03966 Results of 4 Couplings Sent to Consumers Energy Laboratory Services for Hardness Testing August 12, 2011
- CR-PLP-2011-03967 Service Water Pump P-7C Lift Setting Not Validated Correctly August 12, 2011
- CR-PLP-2011-03967 Final Pump Lift Setting Not Validated Correctly August 11, 2011
- CR-PLP-2011-03968 Questioning of Pump Run in Dead Headed Condition August 11, 2011
- CR-PLP-2011-03975 Relevant OE not Considered in
- CR-PLP-2009-04519 August 12, 2011
- LO-CAR-2009-00103 Nonconformance of Parts from HydroAire October 5, 2009
- LO-PLPLO-2009-00065 Implementation of Effectiveness Review Plan for RCE
- CR-PLP-2009-04519 October 16, 2009
Documents
- Title Date/Revision No.
- LER 2004-001-01 Emergency Service Water Pump Failure September 18, 2004 Certificate of Conformance Certificate of Conformance for
- PO 10237148, Job# NQ5832 June 9, 2009
- CP-PLP-2009-002 P-7C Service Water Pump Refurbishment Procurement Plan May 18, 2009
- EC 31208 Engineering Basis for Service Water Pump P-7C to be Considered Fully Operable Post-Repair (August 2011 Coupling Failure) August 12, 2011
- EC 31218 Engineering Basis for Service Water Pump P-7A and P-7B to be Considered Operable Considering P-7C Coupling Failure (August 2011 Coupling Failure) August 10, 2011 Examination Report
- 00286627
- VT-2 (Pressure Test) of Service Water Pump P-7C Prior to Return to Service August 12, 2011 HydroAire NDR Nonconformance and Discrepancy Report 09-55 October 2, 2009
- LO-WTHQN-2011-00439 Source Activity for Entergy Palisades PO10262979* Rev 000 March 4, 2011
- LO-WTHQN-2011-00920 Source Activity for Entergy Palisades PO10262979, HydroAire Job No.
- NQ 5940 July 20, 2011 MAT Project
- 0900609 Consumers Energy Palisades Pump P7C Shaft Coupling Failure Analysis October 5, 2009 NDR No.
- QANC-09 97 HydroAire Nonconformance & Discrepancy Report and Part 21 Notification December 16, 2009 NUPIC Audit Report VA11-011 NUPIC Audit of HydroAire, Inc. June 1, 2011 Purchase Order
- 10190242 Procurement Documents for Pump P-7A Couplings for 2009 Pump Refurbishment Revision 003 Purchase Order
- 10237148 Procurement Documents for Service Water Pump Refurbishment June 9, 2009 Purchase Order
- 10246213 Procurement Documents for Pump P-7B Couplings for 2009 Pump Refurbishment Revision 004 Purchase Order
- 10253715 Procurement Documents for Pump P-7C Couplings Following 2009 Coupling Failure Revision 003 Report No. 1100112.401 Structural Integrity Additional Review of Palisades Service Water Pump Couplings Revision 0
- Report
- PO 10324017 Consumers Energy Rockwell Hardness Test Report for Couplings Removed from Failed Pump P-7C August 10, 2011 SQA2010-00131 HydroAire Source Surveillance Report SS010-016 March 16, 2010 SQA2010-00561 HydroAire Source Surveillance Report SS010-158 November 2, 2010
- WO 00190235 P-7C, Reduced Flow>RO-144; Replace Pump Assembly per
- EC-14577 June 10, 2009
- WO 00208591 P-7C: Motor Shaft Broke. Repair Pump October 28, 2009
- RO-216 Service Water Pump Flow Verification September 15, 2010
- WO 00286627 Test Spare P-7C Motor IAW
- EN-MA-134 Offline Motor Electrical Testing August 11, 2011
- WO 52211728 P-7C IST Service Water Pump October 1, 2009
Procedures
- Title Date/Revision No.
- ARP-7 Auxiliary Systems Scheme
- EK-11 Revision 80
- BMS-WI 09.15.03 Bodycote Heat Treatment of Martensitic Corrosion Resistant Steels Procedure July 10, 2006
- BMS-WI-09-00-02 Bodycote Determination of Heat Treat Process Time Procedure November 17, 2006
- EN-DC-115 Engineering Change Process Revision 11
- EN-DC-313 Procurement Engineering Process Revision 6
- EN-LI-102 Corrective Action Process Revision 16
- EN-LI-118 Root Cause Evaluation Process Revision 14
- EN-LI-118-01 Event & Causal Factor Charting Revision 0
- EN-LI-118-02 Change Analysis Revision 0
- EN-LI-118-03 Barrier Analysis Revision 0
- EN-LI-118-04 Task Analysis Revision 0
- EN-LI-118-05 Fault Tree Analysis Revision 0
- EN-LI-118-06 Common Cause Analysis Revision 1
- EN-LI-118-07 Behavioral Analysis Revision 0
- EN-LI-118-08 Failure Modes Analysis Revision 0
- EN-MA-118 Foreign Material Exclusion Revision 5
- EN-MP-100 Critical Procurements Revision 8
- EN-MP-101 Materials, Purchasing and Contracts Process Revision 5
- EN-MP-111 Inventory Control Revision 3
- EN-MP-117 Standardized Purchasing Process Revision 2
- EN-MP-120 Material Receipt Revision 4
- EN-MP-123 Term and Definitions Revision 4
- EN-MP-125 Control of Material Revision 7
- EN-QV-119 Corrective Action Requests, Supplier Stop Work Orders, and Recommendations Revision 7
- EN-QV-120 Planning, Performing, and Reporting Source Activities Revision 7
- EN-QV-121 Supplier Qualification/Maintenance of Qualifications Revision 5
- EN-QV-122 Qualified Suppliers List Revision 4
- EN-QV-123 Supplier Audits/Surveys Revision 4
- QO-14 Palisades Inservice Testing Procedure for Service Water Pumps May 25, 2011 Procedure No
- RO-144 Comprehensive Pump Test Procedure Service Water Pumps P-7A, P-7B and P-7C Revision 4, (4/10/09) Procedure No.F11358-P-001 LPI Procedure for Metallurgical Examination of SW Pump P-7C Coupling Components August, 2011
- SOP-15 Service Water System Revision 51
- SOP-5 Containment Air Cooling Revision 28
- WI-MSM-M-26 Palisades Compression Fitting Installation and Inspection Revision 1
- WI-SWS-M-04 Service Water Pump P-7B and P-7C Removal, Inspection, and Reinstallation Revision 7
Drawings
- Title Date/Revision No. M0011-SH-0055 Palisades Service Water Pump Coupling Drawing.
- M-398 Level Setting Diagram Service Water and Dilution Water Pump Revision 2
- VEN-M11 Service Water Pump Rotating Element Assembly for Pumps P-7A, 7B and 7C Revision 2
LIST OF ACRONYMS
- USED [[]]
- NRC [[]]
- 2 UNITED [[]]
- REGION [[]]
- MEMORA NDUM TO: John Jandovitz, Project Engineer Branch 5 Division of Reactor Projects FROM: Steven West, Director /RA/ Division of Reactor Projects SUBJECT: SPECIAL INSPECTION CHARTER
- FOR [[]]
- PLANT [[]]
- WATER [[]]
- ON [[]]
- AUGUST . A short discussion of the event follows. On August 9, 2011, Palisades Nuclear Plant experienced a sudden loss of the P-7C service water pump while it was in service. This failure resulted in the licensee entering a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> Limiting Condition for Operation due to the inoperability of one service water train. The licensee disassembled the P-7C pump and determined that the pump's first line shaft to second line shaft coupling had failed. The licensee replaced all of the P-7C pump's couplings and two portions of the pump shaft. After completing the required maintenance, the licensee returned pump P-7C to service on August 12, 2011 Based upon our initial review, we are concerned that the licensee may not have taken effective corrective actions following a P-7C service water pump coupling failure that occurred in September 2009. Following the September 2009 event, the licensee determined that the pump coupling failed due to improper heat treatment. The improper heat treatment resulted in the coupling's hardness being too high. The licensee's corrective actions included replacing the P-7C SW pump's couplings with couplings of appropriate hardness. Recent hardness test]]
- J. [[Jandovicz -2- results of several couplings removed from the P-7C pump on August 10, 2011, indicate that these couplings were of an inappropriate hardness. Specifically, each coupling had at least one test location where the hardness measurement exceeded the licensee's acceptance criteria. In addition, one coupling had multiple test locations that exceeded the acceptance criteria. Testing of the failed coupling produced hardness results that were within specifications. Based upon the hardness test results and a March 2011 report provided to the licensee by Structural Integrity, another metallurgical failure mechanism may be contributing to the coupling failures. Lastly, this event raises questions about the effectiveness of the licensee's program for assessing and applying operating experience and its oversight of vendor activities. The]]
NRC has also learned that the remaining service water pumps, P-7A and P-7B, have the same type of coupling installed. As a result, these pumps may also be susceptible to a coupling failure. The sequence of events and the root and contributing causes for this issue are being investigated by the licensee. Based on the deterministic and risk-based criteria in Management Directive 8.3, a Special Inspection at Palisades will commence on August 15, 2011. The Special Inspection Team, which is being led by you, will include Atif Shaikh. Other members may be assigned if specific
needs are identified. The special inspection will determine the sequence of events, and will evaluate the facts, circumstances, and the licensee's actions surrounding this issue. The Special Inspection Charter for you and your team is enclosed. Enclosure: As Stated cc w/encl:
- RI [[]]
- RI [[]]
- RI [[]]
- RI [[]]
- RII [[]]
- RII [[]]
- RII [[]]
- RII [[]]
- RIV [[]]
- RIV [[]]
- RIV [[]]
NRC has also learned that the remaining service water pumps, P-7A and P-7B, have the same type of coupling installed. As a result, these pumps may also be susceptible to a coupling failure. The sequence of events and the root and contributing causes for this issue are being investigated by the licensee. Based on the deterministic and risk-based criteria in Management Directive 8.3, a Special Inspection at Palisades will commence on August 15, 2011. The Special Inspection Team, which is being led by you, will include Atif Shaikh. Other members may be assigned if specific
needs are identified.
The special inspection will determine the sequence of events, and will evaluate the facts, circumstances, and the licensee's actions surrounding this issue. The Special Inspection Charter for you and your team is enclosed.
Enclosure: As Stated cc w/encl:
- RI [[]]
- RI [[]]
- RI [[]]
- RI [[]]
- RII [[]]
- RII [[]]
- RII [[]]
- RII [[]]
- RIV [[]]
- RIV [[]]
- RIV [[]]
- RIII [[]]
- DOCUME [[NT NAME: G:\DRPIII\PALI\Palisades Special Inspection Charter for P7C coupling failure.docx See Previous Concurrence Publicly Available Non-Publicly Available Sensitive Non-Sensitive To receive a copy of this document, indicate in the concurrence box "C" = Copy without attach/encl "E" = Copy with attach/encl "N" = No copy]]
- OFFICE [[]]
- NAME [[]]
- RECORD [[]]
COPY
PALISADES SPECIAL INSPECTION CHARTER This Special Inspection Team is chartered to assess the circumstances surrounding the failure of a coupling on the P-7C service water pump on August 9, 2011. The Special Inspection will be conducted in accordance with Inspection Procedure 93812, "Special Inspection." The
special inspection will include, but not be limited to, the items listed below. This charter may be revised based on the results and findings of the inspection. 1. Establish a historical sequence of events related to service water pump coupling failures up to and including the most recent failure on August 9, 2011. 2. In light of the failure of the P-7C service water pump, review the licensee's basis for operability of the P-7A, P-7B, and P-7C service water pumps. Determine whether the licensee has an adequately supported basis for operability of the P-7A and P-7B pumps and for returning the P-7C pump to service. Determine if additional corrective actions are required. 3. Evaluate activities associated with the licensee's repair of the P-7C pump for adequacy. Confirm that the repair complies with licensee and NRC requirements. 4. Review information and processes relied upon by Palisades to determine whether or not critical attributes of service water pump line shaft couplings, such as hardness, are
acceptable prior to coupling installation. 5. Evaluate the licensee's procedures and processes for evaluating/assessing the performance of vendors. Emphasis should be placed upon determining why couplings
that were believed to be of an appropriate hardness were later found to be of an inappropriate hardness. Review any evaluations/assessments of HydroAire that were completed by the licensee following the September 2009 P-7C coupling failure. Also, review any actions taken or direction provided by the licensee to HydroAire in response to the recent hardness test results. 6. Evaluate the licensee's operating experience program with an emphasis on operating experience related to pump components manufactured by Hyrdo-Aire. Specific attention should be given to the licensee's evaluation of a 2010 service water pump coupling failure at Prairie Island and the evaluation of a March 2011 report from Structural Integrity regarding the potential for coupling failure due to temper embrittlement. Review the licensee's root cause evaluation plan and schedule. Include the schedule for performing testing of any components removed from the P-7C pump. Evaluate whether the root cause evaluation plan is of sufficient depth and breadth. Confirm that the time allowed to perform the root cause evaluation is commensurate with the safety significance of this issue. Communicate to the licensee that the NRC will inspect the completed root cause evaluation and
the associated corrective actions as part of our normal inspection activities.
Additional Inspection Requirements 1. Determine if there are any lessons learned from this Special Inspection. Charter Approval /RA by Karla Stoedter for/ 8/15/11 John Giessner, Chief Branch 4 Division of Reactor Projects _________/RA/_ 8/15/11_______ Steven West, Director Division of Reactor Projects
- ATTACH MENT 3 Palisades P-7C Service Water Pump Timeline September 12, 1991 Beaver Valley Unit 1 experiences a coupling failure of a Byron-Jackson centrifugal pump as a result of embrittlement of
IGSCC. September 1, 1993 NRC Information Notice 93-68, "Failure of Pump Shaft Coupling Caused by Temper Embrittlement During Manufacture" is issued informing
licensees of problems stemming from temper embrittlement of
- 410 SS supplied by Byron-Jackson (referring to the 1991 Beaver Valley event). September 22, 1993 Indian Point Unit 2 experiences a pump failure via a failed coupling attributed to a non-ductile fracture because of temper embrittlement of the 410
- IGSCC. September 1, 2003 Perry Unit 1 experienced a coupling disassembly event on their Emergency Service Water 'A' pump. The failure mechanism was by
- 416SS. May 21, 2004 Perry experiences additional failures of Emergency Service Water pump shafts composed of 416
INPO Operating Experience Digest 2006-02 - INPO releases a document that discusses service water pump (shaft, coupling and impeller) failures
occurring in the industry. The report notes that 12 failures occurred between 1998 and 2006 with the most frequent including corrosion causing coupling separation. One of the common causes identified was
improper heat treatment of the material during manufacturing. February 9,
- 2007 NRC Information Notice 2007-05, "Vertical Deep Draft Pump Shaft and Coupling Failures" is issued that references the coupling failures from
- IGSCC that occurred at Columbia Generating Station. The couplings were TP410 martensitic stainless steel and were susceptible to tempering embrittlement. The operating experience review identified at least 23 essential
- SW pump shaft and coupling failures since 1983 involving more than six different manufacturers. December, 2007
WO 51637416 (Heat Number 49293)
April 10, 2009 P-7C fails Technical Specification Surveillance Test
- RO with degraded flow prior to startup from outage May 19, 2009 Purchase Order #10237148 issued to HydroAire to re-build Layne and Bowler pump to replace P-7C May 21-22, 2009 Pump P-7C couplings were heat treated once and then received a second heat treatment. June 12, 2009 P-7C couplings and shaft changed to]]
WO 190235 September 29, 2009 P-7C pump failed due to coupling #7 failure from IGSCC. The pump had a total run time of 2414.44 hours5.092593e-4 days <br />0.0122 hours <br />7.275132e-5 weeks <br />1.6742e-5 months <br /> with 13 starts/stops.
October 1, 2009 First two couplings from HydroAire arrive (Heat Order #20163) early. Remaining six couplings arrive later in day (Heat Order #20166) and installed.
October 2, 2009 P-7C restored to operation May 13, 2010 P-7B couplings and shaft changed to
- IGSCC. These couplings were also supplied by HydroAire. March, 2011 Structural Integrity (SI) report received at Palisades concerning failure mechanisms of
SI report accepted by Licensee.
August 9, 2011 0700 All 3 SW pumps in-service with differential pressures of 1/2/1 psid, respectively and header pressure is 73 psig and stable. P-7C operated for a total of 14,114.86 hours9.953704e-4 days <br />0.0239 hours <br />1.421958e-4 weeks <br />3.2723e-5 months <br /> with a total of 95 starts/stops and had passed all surveillances
202 Control room receives following alarms:
- STR [[]]
- HI [[]]
- TS [[]]
SW pump P-7C to operable in 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />) 1203 Nuclear Plant Operator reports that P-7C has no discharge pressure and a loud banging noise from the pump. Operators stop pump P-7C. Header pressure is stable at 64 psig. All alarms except EK-1132 clear. 1246 CR-PLP-2011-03902 created to document failure of P-7C and initiates root cause investigation and identified as significant condition adverse to quality 1303 Operators exit ONP 6.1
1530 BS-1320 completed identifying only zebra mussel shells in strainer August 10, 2011 0107 P-7C motor and pump uncoupled
215 P-7C motor removed
25 All P-7C shafts removed. P-7C pump failed due to coupling #6 failure. August 10, 2011 Jackson Labs independently performs hardness tests on 4 couplings and identifies that three couplings have at least one hardness value out-of-specification high. August 11, 2011 0135 P-7C shaft reassembly commenced with couplings from Traveler 27797 (Heat Order 20170).
1059 P-7C pump reassembly completed. 1215 P-7C motor reinstallation completed.
August 12, 2011 P-7C declared Operable after completion of Post Maintenance Testing (WO 286627) and return-to-service Surveillance Testing (WO 286774). August 16-17, 2011 All P-7C couplings sent to Lucius Pitkin Inc (LPI). Metallurgical plan is still in draft form.