IR 05000255/2011012: Difference between revisions
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
||
(4 intermediate revisions by the same user not shown) | |||
Line 3: | Line 3: | ||
| issue date = 10/04/2011 | | issue date = 10/04/2011 | ||
| title = IR 05000255-11-012, on 08/15/2011 - 09/01/2011, Palisades Nuclear Plant, Inspection Procedure 93812, Special Inspection | | title = IR 05000255-11-012, on 08/15/2011 - 09/01/2011, Palisades Nuclear Plant, Inspection Procedure 93812, Special Inspection | ||
| author name = West S | | author name = West S | ||
| author affiliation = NRC/RGN-III/DRP | | author affiliation = NRC/RGN-III/DRP | ||
| addressee name = Vitale A | | addressee name = Vitale A | ||
Line 18: | Line 18: | ||
=Text= | =Text= | ||
{{#Wiki_filter: | {{#Wiki_filter:ber 4, 2011 | ||
==SUBJECT:== | |||
PALISADES NUCLEAR PLANT - NRC SPECIAL INSPECTION TEAM (SIT) | |||
REPORT 05000255/2011012 | |||
==Dear Mr. Vitale:== | |||
On September 1, 2011, the United States Nuclear Regulatory Commission (NRC) completed a special inspection at your Palisades Nuclear Plant. The inspection was conducted in response to the circumstances surrounding the failure of the Service Water Pump P-7C, on August 9, 2011. Based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a special inspection was initiated in accordance with Inspection Procedure 93812, Special Inspection. | |||
The special inspection charter (Attachment 2 of the enclosure) provides the basis and focus areas for the inspection. | |||
The enclosed inspection report documents the inspection results, which were discussed with you and other members of your staff at the exit meeting on [[Exit meeting date::September 1, 2011]]. The determination that the special inspection would be conducted was made on August 15, 2011, and the on-site inspection commenced the same day. | |||
The inspection examined activities conducted under your license as they relate to safety, compliance with the Commissions rules and regulations, and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, and interviewed personnel. | |||
Based on the results of this inspection, no findings of significance were identified. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room and from the Publicly Available Records (PARS) component of NRC's document system, Agencywide Documents Access and Management System (ADAMS). | |||
ADAMS is accessible from the NRC Website at http://www.nrc.qov/readinq-rm/adams.html (the Public Electronic Reading Room). | |||
Sincerely,/RA/ | Sincerely, | ||
Steven West, Director Division of Reactor Projects Docket No. 50-255 License No. DPR-20 | /RA/ | ||
Steven West, Director Division of Reactor Projects Docket No. 50-255 License No. DPR-20 | |||
===Enclosure:=== | ===Enclosure:=== | ||
Inspection Report 05000255/2011012; | Inspection Report 05000255/2011012; w/Attachments: 1. Supplemental Information 2. Memo to Jandovitz 3. Palisades Timeline | ||
REGION III== | |||
Docket No.: 50-255 License No.: DPR-20 Report No.: 05000255/2011012 Licensee: Entergy Nuclear Operations, Inc. | |||
Facility: Palisades Nuclear Plant Location: Covert, MI Dates: August 15, 2011 - September 1, 2011 Inspectors: J. Jandovitz, Project Engineer (Lead) | |||
A. Shaikh, Reactor Inspector P. Smagacz, Reactor Engineer Approved by: John B. Giessner, Chief Branch 4 Division of Reactor Projects Enclosure | |||
=SUMMARY OF FINDINGS= | =SUMMARY OF FINDINGS= | ||
IR 05000255/2010012; 08/15/2011 | IR 05000255/2010012; 08/15/2011 09/01/2011; Palisades Nuclear Plant; Inspection | ||
Procedure 93812, Special Inspection. | |||
This report covers a 4-day period (August 15 -19, 2011) of on-site inspection and in-office review through September 1, 2011. A team, comprised of three regional inspectors, conducted this special inspection. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 4, dated December 2006. | |||
===NRC-Identified and Self-Revealing Findings=== | ===NRC-Identified and Self-Revealing Findings=== | ||
=== Licensee-Identified Violations=== | No findings were identified. | ||
===Licensee-Identified Violations=== | |||
No findings were identified. | |||
=REPORT DETAILS= | =REPORT DETAILS= | ||
At 12:02 p.m. EDT on August 9, 2011, the licensee entered an unplanned 72-hour shutdown Technical Specification limiting condition for operation (LCO) action statement (TSAC) due to failure of service water (SW) pump P-7C. The SW system includes three motor-driven, vertical multistage deep draft pumps that provide cooling water to safety related equipment (component cooling water, containment air coolers, diesel generators, control room coolers, etc.). Each pump has 50 percent capacity; and two pumps are required to be in service; however, the technical specification requires all three pumps to be operable. All three pumps were running prior to the failure. | |||
Summary of the Degraded Condition At 12:02 p.m. EDT on August 9, 2011, the licensee entered an unplanned 72-hour shutdown Technical Specification limiting condition for operation (LCO) action statement (TSAC) due to failure of service water (SW) pump P-7C. The SW system includes three motor-driven, vertical multistage deep draft pumps that provide cooling water to safety related equipment (component cooling water, containment air coolers, diesel generators, control room coolers, etc.). Each pump has 50 percent capacity; and two pumps are required to be in service; however, the technical specification requires all three pumps to be operable. All three pumps were running prior to the failure. | |||
The licensee observed a low differential pressure condition and low pump amperage following the failure of the pump. The licensee disassembled pump P-7C and determined that the pumps first line shaft to second line shaft coupling, #6, had failed. The licensee replaced all of the pump P-7C couplings and two portions of the pump shaft. After completing the required maintenance and surveillance tests, the licensee returned pump P-7C to service on August 12, within the time allowed by the TSAC. | |||
Pump P-7C had previously experienced another coupling failure in September 2009. The SW pump couplings in both cases were made of 416 Stainless Steel. Subsequent metallurgical analysis determined the couplings hardness was significantly higher than the design specification; and the failure was caused by Intergranular Stress Corrosion Cracking (IGSCC). | |||
The licensees corrective actions in 2009 included replacing all the SW pump P-7C couplings with couplings of the same design and hardness criteria. However, hardness test results of several couplings removed from pump P-7C on August 10, 2011, indicated that these couplings were not within the hardness specification. Specifically, three of the couplings had at least one test location where the hardness measurement was higher than the licensees design specification criteria. | |||
Inspection Scope Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a special inspection was initiated in accordance with NRC Inspection Procedure 93812, Special Inspection Team. The special inspection charter, dated August 15, 2011, is included as Attachment 2. The team reviewed technical and design documents, procedures, maintenance records, and corrective action documents; interviewed station personnel and consultants; and performed plant walkdowns of plant equipment. A list of specific documents reviewed is provided in Attachment 1. | |||
{{a|4OA5}} | {{a|4OA5}} | ||
==4OA5 Other Activities - Special | ==4OA5 Other Activities - Special Inspection== | ||
{{IP sample|IP=IP 93812}} | {{IP sample|IP=IP 93812}} | ||
===.1 Establish a historical sequence of events related to service water pump coupling failures up to and including the most recent failure on August 9, | In accordance with the Charter, the following items were reviewed. | ||
===.1 Establish a historical sequence of events related to service water pump coupling failures=== | |||
up to and including the most recent failure on August 9, 2011. | |||
====a. Inspection Scope==== | |||
The inspectors reviewed corrective action documents, operating experience reports, technical literature, operation history, surveillance procedures and results, vendor procurement and manufacturing records, and interviewed plant personnel for information related to the use of 416 Stainless Steel (SS) material, and similar material such as 410SS, in power plant applications, particularly, SW pumps. | |||
A detailed historical timeline of activities and information involving use and experience is contained in Attachment 3. | |||
====b. Findings and Observations==== | |||
No findings were identified. | |||
The inspectors were concerned that the licensee had not adequately evaluated operating experience to recognize the vulnerabilities of 416SS material to IGSCC based on material properties, particularly heat treatment. This concern is discussed further in the unresolved item (URI) in Section | |||
===.07 of this report. However, the inspectors did not=== | |||
identify any operational or surveillance results or observations that may have predicted the failure of SW pump P-7C on August 9, 2011. | |||
===.2 Review the licensees basis for operability of the service water pumps P-7A, P-7B, and=== | |||
P-7C. Determine whether the licensee had an adequately supported basis for operability of pumps P-7A and P-7B and for returning pump P-7C to service. Determine if additional corrective actions are required. | |||
====a. Inspection Scope==== | |||
The inspectors reviewed the prompt operability basis for all three SW pumps contained in Engineering Change (EC) 31208 and EC 31218. The inspectors identified run times, service conditions and life, starts and stops, heats of materials, and differences in heat treatment for the in-service couplings. | |||
====b. Findings and Observations==== | |||
No findings were identified. | |||
Based on the information available, the inspectors did not disagree with the immediate or prompt operability evaluations. The prompt operability for SW pumps P-7B and P-7C was based on the higher confidence that the hardness of the couplings installed in the pumps were within the design specification range, and therefore less susceptible to IGSCC. Operability for pump P-7A was based on the fact that couplings installed in that pump had been in operation the longest with no failures. | |||
However, the inspectors noted that there was no discussion on temper embrittlement identified in the Structural Integrity (SI) report and in operating experience, as a factor that may affect the couplings reliability. Therefore, it was recognized that the results from the metallurgy and root cause evaluation may impact the basis for current operability and the couplings service life. | |||
After the on-site SIT inspection was complete, the licensee received initial metallurgical results on the failed coupling and initiated a near-term replacement plan of the couplings in all three SW pumps with couplings of a different material, more resistant to IGSCC. At the time of the SIT exit, the couplings in pump P-7A had been replaced with the new material. | |||
The NRC will continue to monitor results from the root cause investigation for impact on the operability basis for any pump that still has the 416SS couplings. | |||
The residents will also evaluate the licensees evaluation of past operability for all the SW pumps and the extent of condition evaluation for the 416SS components remaining in the pumps. | |||
===.3 Evaluate activities associated with the licensees repair of service water pump P-7C for=== | |||
adequacy. Confirm that the repair complies with licensee and NRC requirements. | |||
====a. Inspection Scope==== | |||
The inspectors reviewed the licensees repair and replacement activities associated with the rebuild of pump P-7C following the 2011 pump coupling failure. This review included the document review of post-maintenance testing of the pump as well as documentation of pre-installation testing of pump and motor. Also reviewed was the completed work order for this repair and replacement, and the material specifications (heat treat curves, hardness results) of couplings that were installed in this rebuilt pump P-7C. | |||
====b. Findings and Observations==== | |||
No findings were identified The inspectors did not identify any issues relating to the 2011 pump P-7C repair. | |||
===.4 Review information and processes relied upon by Palisades to determine whether or not=== | |||
critical attributes of service water pump line shaft couplings, such as hardness, are acceptable prior to coupling installation. | |||
====a. Inspection Scope==== | |||
The inspectors reviewed the information provided by the licensee regarding pump shaft couplings that included hardness measurements at time of procurement, independent lab results for coupling hardness measurements prior to coupling installation in the pumps, and the associated heat treatments for the couplings. Also reviewed were industry reports and operating experience regarding use of 416 Martensitic SS for these couplings and the 2009 SW pump P-7C coupling failure root cause evaluation. | |||
====b. Findings and Observations==== | |||
No findings were identified. | |||
The inspectors identified a concern with the adequacy of specifying coupling hardness as the only critical material parameter. This was not a conservative approach given the collection of scientific literature and industry operating experience describing the vulnerability of 416/410 Martensitic SS to stress corrosion cracking; and suggested use of toughness measurements in addition to hardness values to determine acceptability of 416/410 Martensitic SS as coupling material. However, ensuring hardness that values are below a certain allowable maximum, reduces the vulnerability of the couplings to IGSCC; but hardness itself may not be sufficient to ensure critical parameters are met. | |||
This concern is captured as part of the URI included in Section | |||
===.07 of this report.=== | ===.07 of this report.=== | ||
In late 2007, an engineering change was initiated to replace the carbon steel components in the SW pumps with stainless steel components resistant to the erosion and corrosion issues the plant had experienced throughout its life. In 2008, couplings for the SW pumps were procured and fabricated from 416SS with specified criteria for material chemistry and a Rockwell C (Rc) hardness range of 28-32 Rc. | |||
Information available from industry operating experience referenced that material, specifically water pump couplings, made from 410/416 SS were susceptible to IGSCC due to temper embrittlement and improper heat treatment. | |||
The 2009 Palisades failure of SW pump P-7C coupling was attributed to IGSCC due to significantly high hardness, ranging from 37-41 Rc; and all couplings were replaced with the same material and design specification for hardness. No additional criteria were specified for tempering or heat treatment. | |||
Results from a similar coupling failure at Prairie Island in 2010, showed that their coupling, supplied by same vendor as the Palisades pump couplings, failed due to IGSCC even though the hardness was near the specification criteria. | |||
Absent any design criteria from the licensee, the heat treatment performed on the couplings was focused on the final product having a hardness value in the specified range of 28-32 Rc. There was no consideration of the effect of heat treatment on other material properties, such as toughness, that may affect susceptibility to IGSCC. The inspectors noted the heat treatments were different for couplings for each of the three pumps. Couplings for pumps P-7B and P-7C were double-tempered. Also, the procedure used by the heat treatment vendor allowed multiple heat treatments (tempering) to achieve the desired hardness, and did not consider the affect of this process on IGSCC resistance. | |||
===.5 Evaluate the | Because of the literature and industry operating experience of the vulnerability of 416/410 Martensitic SS to IGSCC, the plant requested Structural Integrity to evaluate their 416SS couplings. The report was received in March 2011, and suggested that the couplings were susceptible to temper embrittlement, but did not conclude that there was an immediate operability concern. The plant was evaluating this information when the coupling failed on August 9. | ||
Emphasis should be placed upon determining why couplings that were believed to be of an appropriate hardness were later found to be of an inappropriate hardness. Review any evaluations/assessments of HydroAire that were completed by the licensee following the September 2009 service water pump P-7C coupling failure. Also, review any actions taken or direction provided by the licensee to HydroAire in response to the recent hardness test | |||
===.5 Evaluate the licensees procedures and processes for evaluating/assessing the=== | |||
performance of vendors. Emphasis should be placed upon determining why couplings that were believed to be of an appropriate hardness were later found to be of an inappropriate hardness. Review any evaluations/assessments of HydroAire that were completed by the licensee following the September 2009 service water pump P-7C coupling failure. Also, review any actions taken or direction provided by the licensee to HydroAire in response to the recent hardness test results. | |||
====a. Inspection Scope==== | |||
The inspectors reviewed procurement procedures for the licensee and vendor, the 2009 root cause evaluation, source surveillances completed since 2009, the industry audit of HydroAire, condition reports, corrective actions taken by the licensee and vendor after the 2009 coupling failure, purchase documents, and interviewed Quality Assurance personnel. | |||
====b. Findings and Observations==== | ====b. Findings and Observations==== | ||
No findings were identified. The inspectors identified a concern that the licensee may not have provided adequate vendor procurement control after the 2009 coupling failure, which may have contributed to the 2011 coupling failure. The inspectors found that adequate procurement procedures and processes existed, but were not fully utilized after the 2009 coupling failure. This concern is included in the URI discussion in Section | No findings were identified. | ||
The inspectors identified a concern that the licensee may not have provided adequate vendor procurement control after the 2009 coupling failure, which may have contributed to the 2011 coupling failure. The inspectors found that adequate procurement procedures and processes existed, but were not fully utilized after the 2009 coupling failure. This concern is included in the URI discussion in Section | |||
===.07 of this report.=== | |||
The inspectors reviewed a vendor surveillance performed in March 2010 on parts for a spare SW pump. The surveillance noted that one pump part was heat-treated, and the hardness readings taken by the pump vendor and the heat treatment vendor were both out of the specified range. Therefore, the part was again heat-treated, resulting in acceptable hardness values obtained by both vendors. The part was then identified as acceptable. This demonstrated that as late as 2010, there were still issues with the controls on heat treatment for SW pump parts. | |||
Implementation of the corrective actions from the 2009 coupling failure resulted in some improvements to the vendor procurement processes and additional oversight. This was based on: | |||
* frequent vendor surveillance activities were conducted on new parts that found vendor performance was satisfactory; | |||
* procedure EN-MP-100, Critical Procurements, was implemented for procurements after 2009; and | |||
* a NUPIC [Nuclear Procurement Issues Committee] audit conducted in April 2011 found that HydroAire was implementing an effective quality assurance (QA)program. | |||
This is not concluding | This is not concluding the design criteria supplied to the vendor was correct, as discussed earlier, only that the parts supplied by the vendor appeared to meet licensee criteria. | ||
===.6 Evaluate the | No additional actions were taken by or required of the vendor after the 2011 coupling failure, since the licensee considered this failure the result of activities that occurred in 2009, and that corrective actions implemented by the vendor then were effective. | ||
Specific attention should be given to the | |||
===.6 Evaluate the licensees operating experience program with an emphasis on operating=== | |||
experience related to pump components manufactured by HydroAire. Specific attention should be given to the licensees evaluation of a 2010 service water pump coupling failure at Prairie Island and the evaluation of a March 2011 report from Structural Integrity regarding the potential for coupling failure due to temper embrittlement. | |||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
Line 89: | Line 190: | ||
====b. Findings and Observations==== | ====b. Findings and Observations==== | ||
No findings were identified. | No findings were identified. | ||
The inspectors identified a concern that operating experience and industry information may not have been incorporated into the design of the pump nor addressed adequately in the 2009 coupling failure replacement and root cause evaluation. Further discussion of this concern is included in the URI discussion in section | |||
===.7 Review the | ===.07. Operating experience that existed as early as 1991 discussed failures of pump couplings=== | ||
attributed to embrittlement of 410SS and 416SS due to improper tempering temperatures and heat treatment. In addition, industry and NRC reports were available prior to 2009 and discussed the effect of temper embrittlement on 410/416SS material. | |||
The SI report was issued in March 2011, but the licensee did not accept the report until August. This report discussed temper embrittlement as an issue that could affect coupling reliability. The licensee had not yet evaluated nor initiated corrective actions based on the report. | |||
The inspectors found the licensee did identify pertinent operating experience in development of the 2011 coupling failure root cause evaluation, although the root cause report was not issued by the completion of this SIT report. | |||
===.7 Review the licensees root cause evaluation plan and schedule. Include the schedule for=== | |||
performing testing of any components removed from pump P-7C. Evaluate whether the root cause evaluation plan is of sufficient depth and breadth. Confirm that the time allowed to perform the root cause evaluation is commensurate with the safety significance of this issue. Communicate to the licensee that the NRC will inspect the completed root cause evaluation and the associated corrective actions as part of our normal inspection activities. | |||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
The inspectors reviewed the root cause charter, schedule, team make-up and action plan; and discussed actions with root cause team | The inspectors reviewed the root cause charter, schedule, team make-up and action plan; and discussed actions with root cause team. | ||
Unresolved Item: | b. Observations The inspectors found the root cause team was comprised of individuals dedicated solely to the root cause investigation; they had no additional responsibilities. Most members had not been involved in the root cause for the 2009 failure. Team members included consultants, including an independent pump manufacturer and the metallurgical lab contracted to perform the metallurgical analysis. The pump supplier, HydroAire, provided input to the root cause but was not a member. | ||
The root cause timeline started August 16, and was scheduled to be completed within 30 days in accordance with procedures. | |||
The root cause investigation was determined to be of adequate depth and breadth to be successful in determining the actual root cause. The action plan included the following aspects: | |||
* improper hardness; | |||
* improper material; | |||
* over-torque from foreign material; | |||
* shaft misalignment; | |||
* Stress Corrosion Cracking; | |||
* incorrect clearances in coupling to shaft; | |||
* shaft faces not square; | |||
* pump alignment; | |||
* shaft wobble; | |||
* reverse rotation; | |||
* coupling defects; | |||
* operation at critical speed; | |||
* operation loads; | |||
* strainer blockage; | |||
* maintenance practices; and | |||
* fatigue failure. | |||
The inspectors reviewed the draft metallurgical plan and considered the metallurgical analysis plan for selected coupling to be thorough, although the licensee had not yet finalized the plan or determined which couplings would be included in the plan. The NRC will continue to inspect the metallurgical results and root cause report under the normal baseline procedures. | |||
Unresolved Item: Adequacy of SW Pump Couplings | |||
=====Introduction:===== | =====Introduction:===== | ||
Line 106: | Line 239: | ||
=====Description:===== | =====Description:===== | ||
Palisades has three safety-related SW pumps that draw water from Lake Michigan to provide cooling to safety-related and non-safety-related components. Each pump has eight couplings to connect shaft sections. In 2009, the plant began a material change for the couplings and shafts from carbon steel to 416 Martensitic SS due to erosion issues of the carbon steel material. The material would be changed in series in of pumps P-7A, P-7C, and P-7B respectively. The timeline (Attachment 3) includes specific dates. In 2009, SW pump P-7C failed due to cracked coupling #7, resulting into entry a TSAC, requiring the plant to shut down if the pump was not returned to service within 72 hours. Metallurgical data on the failed coupling revealed the coupling hardness was significantly higher than the design specification. The specified hardness range was Rockwell C value of 28-32 Rc. Hardness readings on the failed coupling ranged from 37-41 Rc. The vendor, HydroAire, had supplied the couplings with certifications that the couplings were within the specified hardness. This failure was entered into the | Palisades has three safety-related SW pumps that draw water from Lake Michigan to provide cooling to safety-related and non-safety-related components. Each pump has eight couplings to connect shaft sections. In 2009, the plant began a material change for the couplings and shafts from carbon steel to 416 Martensitic SS due to erosion issues of the carbon steel material. The material would be changed in series in of pumps P-7A, P-7C, and P-7B respectively. The timeline (Attachment 3) includes specific dates. | ||
In 2009, SW pump P-7C failed due to cracked coupling #7, resulting into entry a TSAC, requiring the plant to shut down if the pump was not returned to service within 72 hours. | |||
Metallurgical data on the failed coupling revealed the coupling hardness was significantly higher than the design specification. The specified hardness range was Rockwell C value of 28-32 Rc. Hardness readings on the failed coupling ranged from 37-41 Rc. | |||
The vendor, HydroAire, had supplied the couplings with certifications that the couplings were within the specified hardness. | |||
This failure was entered into the licensees corrective action program and identified as a significant condition adverse to quality (SCAQ). | |||
At the time of the failure, no spare couplings were available. Immediate discussions with HydroAire ensued to obtain replacement couplings within the 72-hour timeframe. | |||
Based on the hardness data received on the failed coupling, which conflicted with the certification received from the vendor, the licensee verbally imposed additional requirements to obtain independent laboratory hardness checks on the replacement couplings. The requirement for independent hardness checks on the replacement couplings was not incorporated into the procurement process. Procurement documents were not updated with this additional requirement, and procurement personnel, such as the licensee procurement manager at the vendor site or the plant receipt inspection personnel, were not made aware of the additional requirement. Procedure EN-MP-100, Critical Procurements, was not implemented even though criteria in the procedure, such as previous vendor issues, how critical the couplings were on plant operation, and the short procurement timeframe, appear to have been satisfied. | |||
Eight replacement couplings were received and immediately installed in the pump; the pump was returned to service within the 72-hour requirement. Soon after the pump was returned to service, the licensee discovered that independent hardness checks were not performed. The vendor did provide certifications indicating that the couplings had hardness values in the specified range. An additional ten couplings were ordered and received soon after the pump was returned to service. Independent hardness checks completed on these spare couplings agreed with vendor hardness results, thus providing some assurance to the licensee that the couplings currently installed in pump P-7C were of the correct hardness. | |||
On August 9, 2011, SW pump P-7C was declared inoperable due to the failure of coupling #6. This failure was entered into the licensee corrective action program and identified as a SCAQ. The cause of the failure had not been determined at the time this SIT report was completed. However, hardness results obtained from an independent metallurgical lab from four of the in-service couplings, including the coupling that failed, were found to have at least one spot of higher hardness than specified, greater than Rockwell Hardness value of 32 Rc. No hardness values approached the 37-41 Rc observed in the 2009 failure (the values were less than 34Rc). Therefore, the inspectors were concerned that the licensee did not ensure the vendor-supplied couplings with correct hardness values in 2009. Pending final results and conclusions from the metallurgical analysis and root cause analysis, the licensee may have failed to take measures in 2009 to assure that purchased material and equipment conform to the procurement documents, including provisions for, as appropriate, objective evidence of quality furnished by the contractor as required by Appendix B. | |||
Operating experience available to the licensee from 1991 through 2009, identified that Martensitic 416SS could be more susceptible to IGSCC, dependant on the heat treating process. Some of the previous operating experience is listed below: | |||
* September 12, 1991 - Beaver Valley Unit 1 experienced a coupling failure of a Byron-Jackson centrifugal pump as a result of embrittlement of 410SS due to improper tempering temperatures and potential impurities. The failure mechanism was IGSCC; | |||
* September 1, 1993 - NRC Information Notice 93-68, Failure of Pump Shaft Coupling Caused by Temper Embrittlement During Manufacture, is issued informing licensees of problems stemming from temper embrittlement of 410SS supplied by Byron-Jackson (referring to the 1991 Beaver Valley event); the document noted in the material analysis that the toughness of the failed component was extremely low; | |||
* September 22, 1993 - Indian Point Unit 2 experienced a pump failure due to a failed coupling attributed to a non-ductile fracture because of temper embrittlement of the 410SST used; the failure mechanism was IGSCC; | |||
* September 1, 2003 - Perry Unit 1 experienced a coupling disassembly event on their emergency SW pump A. The failure mechanism was by IGSCC and no performance effects were noticed before the failure of the coupling; | |||
* May 21, 2004 - Perry Unit 1 experienced another emergency SW pump coupling failure; the failure mechanism was by IGSCC, similar to the 2003 event; | |||
* 2006 - INPO Operating Experience Digest 2006-02 - INPO released a document that discusses SW pump (shaft, coupling and impeller) failures occurring in the industry. The report noted that 12 failures occurred between 1998 and 2006, with the most frequent including corrosion causing coupling separation. One of the common causes identified was improper heat treatment of the material during manufacturing; and | |||
* February 9, 2007 - NRC Information Notice 2007-05, Vertical Deep Draft Pump Shaft and Coupling Failures, was issued and referenced the coupling failures from IGSCC that occurred at Columbia Generating Station. The couplings were 410 Martensitic SS and were susceptible to tempering embrittlement. The operating experience review identified at least 23 essential SW pump shaft and coupling failures since 1983 involving more than six different manufacturers. | |||
The 2009 coupling failure root cause evaluation was focused on the material hardness. It did not fully evaluate the effect of other material properties, such as toughness which is discussed in the operating experience, nor the change of material properties from heat treatment on the susceptibility of the couplings to IGSCC. Corrective actions after the 2009 failure did not address the heat treatment process. The inspectors were concerned that the licensee may not have adequately addressed operating experience in the design of the original 416SS couplings or the 2009 replacement couplings as required by Appendix B, Criteria III. And the corrective actions taken after the 2009 failure may not have prevented recurrence of this SCAQ as required by Appendix B, Criteria XVI. These concerns will be further evaluated as part of the NRC baseline inspection program, after the final metallurgical results are issued and the licensee root cause evaluation is completed. (URI 05000255/2011012-01; Adequacy of SW Pump Couplings). | |||
{{a|4OA6}} | {{a|4OA6}} | ||
==4OA6 Meetings, Including Exit On September 1, 2011, the special inspection team leader presented the preliminary inspection results to Mr. Anthony Vitale and members of his staff. | ==4OA6 Meetings, Including Exit== | ||
No proprietary information is included in this inspection report. ATTACHMENT: | |||
On September 1, 2011, the special inspection team leader presented the preliminary inspection results to Mr. Anthony Vitale and members of his staff. No proprietary information is included in this inspection report. | |||
ATTACHMENT: | |||
=SUPPLEMENTAL INFORMATION= | =SUPPLEMENTAL INFORMATION= | ||
Line 118: | Line 280: | ||
===Licensee Personnel=== | ===Licensee Personnel=== | ||
: [[contact::T. Vitale]], Entergy/Site Vice President | : [[contact::T. Vitale]], Entergy/Site Vice President | ||
: [[contact::J. Haumersen]], Entergy/System Engineering Manager | : [[contact::J. Haumersen]], Entergy/System Engineering Manager | ||
: [[contact::B. Kemp]], Entergy/Design Engineering Manager | : [[contact::B. Kemp]], Entergy/Design Engineering Manager | ||
: [[contact::D. MacMaster]], Entergy/Acting Design Engineering Manager | : [[contact::D. MacMaster]], Entergy/Acting Design Engineering Manager | ||
: [[contact::D. Hamilton]], Entergy/General Manager Plant Operations | : [[contact::D. Hamilton]], Entergy/General Manager Plant Operations | ||
: [[contact::A. Blind]], Entergy/Engineering Director D.Malone, Entergy/Acting Nuclear Safety Assurance Dir. | : [[contact::A. Blind]], Entergy/Engineering Director | ||
: [[contact::J. C. Plachta]], Entergy/Quality Assurance Manager | D.Malone, Entergy/Acting Nuclear Safety Assurance Dir. | ||
: [[contact::P. Deniston]], Entergy/Sr. Engineer | : [[contact::J. C. Plachta]], Entergy/Quality Assurance Manager | ||
: [[contact::J. Forehand]], Entergy/Engineering Supervisor | : [[contact::P. Deniston]], Entergy/Sr. Engineer | ||
: [[contact::B. Dotson]], Entergy/Licensing J.Pennington, Entergy/Corporate MP&C | : [[contact::J. Forehand]], Entergy/Engineering Supervisor | ||
: [[contact::B. Dotson]], Entergy/Licensing | |||
J.Pennington, Entergy/Corporate MP&C | |||
===NRC Personnel=== | ===NRC Personnel=== | ||
J Ellegood, Senior Resident Inspector, Palisades | |||
: [[contact::J. Giessner]], Branch Chief | |||
: [[contact::K. Stoedter]], Acting Branch Chief | |||
==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED== | ==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED== | ||
===Opened and Closed=== | ===Opened and Closed=== | ||
None | |||
None | |||
===Closed=== | ===Closed=== | ||
None | |||
===Opened=== | ===Opened=== | ||
: 05000255/2011012-01 | : 05000255/2011012-01 URI Adequacy of SW Pump Couplings | ||
==LIST OF DOCUMENTS REVIEWED== | ==LIST OF DOCUMENTS REVIEWED== | ||
}} | }} |
Latest revision as of 03:54, 21 December 2019
ML112780190 | |
Person / Time | |
---|---|
Site: | Palisades ![]() |
Issue date: | 10/04/2011 |
From: | Stephanie West Division Reactor Projects III |
To: | Vitale A Entergy Nuclear Operations |
References | |
IR-11-012 | |
Download: ML112780190 (33) | |
Text
ber 4, 2011
SUBJECT:
PALISADES NUCLEAR PLANT - NRC SPECIAL INSPECTION TEAM (SIT)
REPORT 05000255/2011012
Dear Mr. Vitale:
On September 1, 2011, the United States Nuclear Regulatory Commission (NRC) completed a special inspection at your Palisades Nuclear Plant. The inspection was conducted in response to the circumstances surrounding the failure of the Service Water Pump P-7C, on August 9, 2011. Based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a special inspection was initiated in accordance with Inspection Procedure 93812, Special Inspection.
The special inspection charter (Attachment 2 of the enclosure) provides the basis and focus areas for the inspection.
The enclosed inspection report documents the inspection results, which were discussed with you and other members of your staff at the exit meeting on September 1, 2011. The determination that the special inspection would be conducted was made on August 15, 2011, and the on-site inspection commenced the same day.
The inspection examined activities conducted under your license as they relate to safety, compliance with the Commissions rules and regulations, and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, and interviewed personnel.
Based on the results of this inspection, no findings of significance were identified. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room and from the Publicly Available Records (PARS) component of NRC's document system, Agencywide Documents Access and Management System (ADAMS).
ADAMS is accessible from the NRC Website at http://www.nrc.qov/readinq-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Steven West, Director Division of Reactor Projects Docket No. 50-255 License No. DPR-20
Enclosure:
Inspection Report 05000255/2011012; w/Attachments: 1. Supplemental Information 2. Memo to Jandovitz 3. Palisades Timeline
REGION III==
Docket No.: 50-255 License No.: DPR-20 Report No.: 05000255/2011012 Licensee: Entergy Nuclear Operations, Inc.
Facility: Palisades Nuclear Plant Location: Covert, MI Dates: August 15, 2011 - September 1, 2011 Inspectors: J. Jandovitz, Project Engineer (Lead)
A. Shaikh, Reactor Inspector P. Smagacz, Reactor Engineer Approved by: John B. Giessner, Chief Branch 4 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000255/2010012; 08/15/2011 09/01/2011; Palisades Nuclear Plant; Inspection
Procedure 93812, Special Inspection.
This report covers a 4-day period (August 15 -19, 2011) of on-site inspection and in-office review through September 1, 2011. A team, comprised of three regional inspectors, conducted this special inspection. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 4, dated December 2006.
NRC-Identified and Self-Revealing Findings
No findings were identified.
Licensee-Identified Violations
No findings were identified.
REPORT DETAILS
Summary of the Degraded Condition At 12:02 p.m. EDT on August 9, 2011, the licensee entered an unplanned 72-hour shutdown Technical Specification limiting condition for operation (LCO) action statement (TSAC) due to failure of service water (SW) pump P-7C. The SW system includes three motor-driven, vertical multistage deep draft pumps that provide cooling water to safety related equipment (component cooling water, containment air coolers, diesel generators, control room coolers, etc.). Each pump has 50 percent capacity; and two pumps are required to be in service; however, the technical specification requires all three pumps to be operable. All three pumps were running prior to the failure.
The licensee observed a low differential pressure condition and low pump amperage following the failure of the pump. The licensee disassembled pump P-7C and determined that the pumps first line shaft to second line shaft coupling, #6, had failed. The licensee replaced all of the pump P-7C couplings and two portions of the pump shaft. After completing the required maintenance and surveillance tests, the licensee returned pump P-7C to service on August 12, within the time allowed by the TSAC.
Pump P-7C had previously experienced another coupling failure in September 2009. The SW pump couplings in both cases were made of 416 Stainless Steel. Subsequent metallurgical analysis determined the couplings hardness was significantly higher than the design specification; and the failure was caused by Intergranular Stress Corrosion Cracking (IGSCC).
The licensees corrective actions in 2009 included replacing all the SW pump P-7C couplings with couplings of the same design and hardness criteria. However, hardness test results of several couplings removed from pump P-7C on August 10, 2011, indicated that these couplings were not within the hardness specification. Specifically, three of the couplings had at least one test location where the hardness measurement was higher than the licensees design specification criteria.
Inspection Scope Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a special inspection was initiated in accordance with NRC Inspection Procedure 93812, Special Inspection Team. The special inspection charter, dated August 15, 2011, is included as Attachment 2. The team reviewed technical and design documents, procedures, maintenance records, and corrective action documents; interviewed station personnel and consultants; and performed plant walkdowns of plant equipment. A list of specific documents reviewed is provided in Attachment 1.
4OA5 Other Activities - Special Inspection
In accordance with the Charter, the following items were reviewed.
up to and including the most recent failure on August 9, 2011.
a. Inspection Scope
The inspectors reviewed corrective action documents, operating experience reports, technical literature, operation history, surveillance procedures and results, vendor procurement and manufacturing records, and interviewed plant personnel for information related to the use of 416 Stainless Steel (SS) material, and similar material such as 410SS, in power plant applications, particularly, SW pumps.
A detailed historical timeline of activities and information involving use and experience is contained in Attachment 3.
b. Findings and Observations
No findings were identified.
The inspectors were concerned that the licensee had not adequately evaluated operating experience to recognize the vulnerabilities of 416SS material to IGSCC based on material properties, particularly heat treatment. This concern is discussed further in the unresolved item (URI) in Section
.07 of this report. However, the inspectors did not
identify any operational or surveillance results or observations that may have predicted the failure of SW pump P-7C on August 9, 2011.
.2 Review the licensees basis for operability of the service water pumps P-7A, P-7B, and
P-7C. Determine whether the licensee had an adequately supported basis for operability of pumps P-7A and P-7B and for returning pump P-7C to service. Determine if additional corrective actions are required.
a. Inspection Scope
The inspectors reviewed the prompt operability basis for all three SW pumps contained in Engineering Change (EC) 31208 and EC 31218. The inspectors identified run times, service conditions and life, starts and stops, heats of materials, and differences in heat treatment for the in-service couplings.
b. Findings and Observations
No findings were identified.
Based on the information available, the inspectors did not disagree with the immediate or prompt operability evaluations. The prompt operability for SW pumps P-7B and P-7C was based on the higher confidence that the hardness of the couplings installed in the pumps were within the design specification range, and therefore less susceptible to IGSCC. Operability for pump P-7A was based on the fact that couplings installed in that pump had been in operation the longest with no failures.
However, the inspectors noted that there was no discussion on temper embrittlement identified in the Structural Integrity (SI) report and in operating experience, as a factor that may affect the couplings reliability. Therefore, it was recognized that the results from the metallurgy and root cause evaluation may impact the basis for current operability and the couplings service life.
After the on-site SIT inspection was complete, the licensee received initial metallurgical results on the failed coupling and initiated a near-term replacement plan of the couplings in all three SW pumps with couplings of a different material, more resistant to IGSCC. At the time of the SIT exit, the couplings in pump P-7A had been replaced with the new material.
The NRC will continue to monitor results from the root cause investigation for impact on the operability basis for any pump that still has the 416SS couplings.
The residents will also evaluate the licensees evaluation of past operability for all the SW pumps and the extent of condition evaluation for the 416SS components remaining in the pumps.
.3 Evaluate activities associated with the licensees repair of service water pump P-7C for
adequacy. Confirm that the repair complies with licensee and NRC requirements.
a. Inspection Scope
The inspectors reviewed the licensees repair and replacement activities associated with the rebuild of pump P-7C following the 2011 pump coupling failure. This review included the document review of post-maintenance testing of the pump as well as documentation of pre-installation testing of pump and motor. Also reviewed was the completed work order for this repair and replacement, and the material specifications (heat treat curves, hardness results) of couplings that were installed in this rebuilt pump P-7C.
b. Findings and Observations
No findings were identified The inspectors did not identify any issues relating to the 2011 pump P-7C repair.
.4 Review information and processes relied upon by Palisades to determine whether or not
critical attributes of service water pump line shaft couplings, such as hardness, are acceptable prior to coupling installation.
a. Inspection Scope
The inspectors reviewed the information provided by the licensee regarding pump shaft couplings that included hardness measurements at time of procurement, independent lab results for coupling hardness measurements prior to coupling installation in the pumps, and the associated heat treatments for the couplings. Also reviewed were industry reports and operating experience regarding use of 416 Martensitic SS for these couplings and the 2009 SW pump P-7C coupling failure root cause evaluation.
b. Findings and Observations
No findings were identified.
The inspectors identified a concern with the adequacy of specifying coupling hardness as the only critical material parameter. This was not a conservative approach given the collection of scientific literature and industry operating experience describing the vulnerability of 416/410 Martensitic SS to stress corrosion cracking; and suggested use of toughness measurements in addition to hardness values to determine acceptability of 416/410 Martensitic SS as coupling material. However, ensuring hardness that values are below a certain allowable maximum, reduces the vulnerability of the couplings to IGSCC; but hardness itself may not be sufficient to ensure critical parameters are met.
This concern is captured as part of the URI included in Section
.07 of this report.
In late 2007, an engineering change was initiated to replace the carbon steel components in the SW pumps with stainless steel components resistant to the erosion and corrosion issues the plant had experienced throughout its life. In 2008, couplings for the SW pumps were procured and fabricated from 416SS with specified criteria for material chemistry and a Rockwell C (Rc) hardness range of 28-32 Rc.
Information available from industry operating experience referenced that material, specifically water pump couplings, made from 410/416 SS were susceptible to IGSCC due to temper embrittlement and improper heat treatment.
The 2009 Palisades failure of SW pump P-7C coupling was attributed to IGSCC due to significantly high hardness, ranging from 37-41 Rc; and all couplings were replaced with the same material and design specification for hardness. No additional criteria were specified for tempering or heat treatment.
Results from a similar coupling failure at Prairie Island in 2010, showed that their coupling, supplied by same vendor as the Palisades pump couplings, failed due to IGSCC even though the hardness was near the specification criteria.
Absent any design criteria from the licensee, the heat treatment performed on the couplings was focused on the final product having a hardness value in the specified range of 28-32 Rc. There was no consideration of the effect of heat treatment on other material properties, such as toughness, that may affect susceptibility to IGSCC. The inspectors noted the heat treatments were different for couplings for each of the three pumps. Couplings for pumps P-7B and P-7C were double-tempered. Also, the procedure used by the heat treatment vendor allowed multiple heat treatments (tempering) to achieve the desired hardness, and did not consider the affect of this process on IGSCC resistance.
Because of the literature and industry operating experience of the vulnerability of 416/410 Martensitic SS to IGSCC, the plant requested Structural Integrity to evaluate their 416SS couplings. The report was received in March 2011, and suggested that the couplings were susceptible to temper embrittlement, but did not conclude that there was an immediate operability concern. The plant was evaluating this information when the coupling failed on August 9.
.5 Evaluate the licensees procedures and processes for evaluating/assessing the
performance of vendors. Emphasis should be placed upon determining why couplings that were believed to be of an appropriate hardness were later found to be of an inappropriate hardness. Review any evaluations/assessments of HydroAire that were completed by the licensee following the September 2009 service water pump P-7C coupling failure. Also, review any actions taken or direction provided by the licensee to HydroAire in response to the recent hardness test results.
a. Inspection Scope
The inspectors reviewed procurement procedures for the licensee and vendor, the 2009 root cause evaluation, source surveillances completed since 2009, the industry audit of HydroAire, condition reports, corrective actions taken by the licensee and vendor after the 2009 coupling failure, purchase documents, and interviewed Quality Assurance personnel.
b. Findings and Observations
No findings were identified.
The inspectors identified a concern that the licensee may not have provided adequate vendor procurement control after the 2009 coupling failure, which may have contributed to the 2011 coupling failure. The inspectors found that adequate procurement procedures and processes existed, but were not fully utilized after the 2009 coupling failure. This concern is included in the URI discussion in Section
.07 of this report.
The inspectors reviewed a vendor surveillance performed in March 2010 on parts for a spare SW pump. The surveillance noted that one pump part was heat-treated, and the hardness readings taken by the pump vendor and the heat treatment vendor were both out of the specified range. Therefore, the part was again heat-treated, resulting in acceptable hardness values obtained by both vendors. The part was then identified as acceptable. This demonstrated that as late as 2010, there were still issues with the controls on heat treatment for SW pump parts.
Implementation of the corrective actions from the 2009 coupling failure resulted in some improvements to the vendor procurement processes and additional oversight. This was based on:
- frequent vendor surveillance activities were conducted on new parts that found vendor performance was satisfactory;
- procedure EN-MP-100, Critical Procurements, was implemented for procurements after 2009; and
- a NUPIC [Nuclear Procurement Issues Committee] audit conducted in April 2011 found that HydroAire was implementing an effective quality assurance (QA)program.
This is not concluding the design criteria supplied to the vendor was correct, as discussed earlier, only that the parts supplied by the vendor appeared to meet licensee criteria.
No additional actions were taken by or required of the vendor after the 2011 coupling failure, since the licensee considered this failure the result of activities that occurred in 2009, and that corrective actions implemented by the vendor then were effective.
.6 Evaluate the licensees operating experience program with an emphasis on operating
experience related to pump components manufactured by HydroAire. Specific attention should be given to the licensees evaluation of a 2010 service water pump coupling failure at Prairie Island and the evaluation of a March 2011 report from Structural Integrity regarding the potential for coupling failure due to temper embrittlement.
a. Inspection Scope
Operating experience regarding failures of couplings using 410/416 Martensitic SS was reviewed along with generic technical reports and metallurgical reports. The report of most interest were the Prairie Island coupling failure in 2010, the Structural Integrity report which was issued to address the 2009 coupling failure at Palisades, and the 2009 Palisades coupling failure root cause evaluation.
b. Findings and Observations
No findings were identified.
The inspectors identified a concern that operating experience and industry information may not have been incorporated into the design of the pump nor addressed adequately in the 2009 coupling failure replacement and root cause evaluation. Further discussion of this concern is included in the URI discussion in section
.07. Operating experience that existed as early as 1991 discussed failures of pump couplings
attributed to embrittlement of 410SS and 416SS due to improper tempering temperatures and heat treatment. In addition, industry and NRC reports were available prior to 2009 and discussed the effect of temper embrittlement on 410/416SS material.
The SI report was issued in March 2011, but the licensee did not accept the report until August. This report discussed temper embrittlement as an issue that could affect coupling reliability. The licensee had not yet evaluated nor initiated corrective actions based on the report.
The inspectors found the licensee did identify pertinent operating experience in development of the 2011 coupling failure root cause evaluation, although the root cause report was not issued by the completion of this SIT report.
.7 Review the licensees root cause evaluation plan and schedule. Include the schedule for
performing testing of any components removed from pump P-7C. Evaluate whether the root cause evaluation plan is of sufficient depth and breadth. Confirm that the time allowed to perform the root cause evaluation is commensurate with the safety significance of this issue. Communicate to the licensee that the NRC will inspect the completed root cause evaluation and the associated corrective actions as part of our normal inspection activities.
a. Inspection Scope
The inspectors reviewed the root cause charter, schedule, team make-up and action plan; and discussed actions with root cause team.
b. Observations The inspectors found the root cause team was comprised of individuals dedicated solely to the root cause investigation; they had no additional responsibilities. Most members had not been involved in the root cause for the 2009 failure. Team members included consultants, including an independent pump manufacturer and the metallurgical lab contracted to perform the metallurgical analysis. The pump supplier, HydroAire, provided input to the root cause but was not a member.
The root cause timeline started August 16, and was scheduled to be completed within 30 days in accordance with procedures.
The root cause investigation was determined to be of adequate depth and breadth to be successful in determining the actual root cause. The action plan included the following aspects:
- improper hardness;
- improper material;
- over-torque from foreign material;
- shaft misalignment;
- incorrect clearances in coupling to shaft;
- shaft faces not square;
- pump alignment;
- shaft wobble;
- reverse rotation;
- coupling defects;
- operation at critical speed;
- operation loads;
- strainer blockage;
- maintenance practices; and
- fatigue failure.
The inspectors reviewed the draft metallurgical plan and considered the metallurgical analysis plan for selected coupling to be thorough, although the licensee had not yet finalized the plan or determined which couplings would be included in the plan. The NRC will continue to inspect the metallurgical results and root cause report under the normal baseline procedures.
Unresolved Item: Adequacy of SW Pump Couplings
Introduction:
A URI was identified by the inspectors during review of the SW pump P-7C coupling failure in 2011. Specifically, the inspectors were concerned with the adequacy of the procurement of the replacement couplings after the 2009 coupling failure on the same pump; whether operating experience was adequately incorporated into design specifications; and whether the corrective actions taken after the 2009 failure were adequate to prevent recurrence.
Description:
Palisades has three safety-related SW pumps that draw water from Lake Michigan to provide cooling to safety-related and non-safety-related components. Each pump has eight couplings to connect shaft sections. In 2009, the plant began a material change for the couplings and shafts from carbon steel to 416 Martensitic SS due to erosion issues of the carbon steel material. The material would be changed in series in of pumps P-7A, P-7C, and P-7B respectively. The timeline (Attachment 3) includes specific dates.
In 2009, SW pump P-7C failed due to cracked coupling #7, resulting into entry a TSAC, requiring the plant to shut down if the pump was not returned to service within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />.
Metallurgical data on the failed coupling revealed the coupling hardness was significantly higher than the design specification. The specified hardness range was Rockwell C value of 28-32 Rc. Hardness readings on the failed coupling ranged from 37-41 Rc.
The vendor, HydroAire, had supplied the couplings with certifications that the couplings were within the specified hardness.
This failure was entered into the licensees corrective action program and identified as a significant condition adverse to quality (SCAQ).
At the time of the failure, no spare couplings were available. Immediate discussions with HydroAire ensued to obtain replacement couplings within the 72-hour timeframe.
Based on the hardness data received on the failed coupling, which conflicted with the certification received from the vendor, the licensee verbally imposed additional requirements to obtain independent laboratory hardness checks on the replacement couplings. The requirement for independent hardness checks on the replacement couplings was not incorporated into the procurement process. Procurement documents were not updated with this additional requirement, and procurement personnel, such as the licensee procurement manager at the vendor site or the plant receipt inspection personnel, were not made aware of the additional requirement. Procedure EN-MP-100, Critical Procurements, was not implemented even though criteria in the procedure, such as previous vendor issues, how critical the couplings were on plant operation, and the short procurement timeframe, appear to have been satisfied.
Eight replacement couplings were received and immediately installed in the pump; the pump was returned to service within the 72-hour requirement. Soon after the pump was returned to service, the licensee discovered that independent hardness checks were not performed. The vendor did provide certifications indicating that the couplings had hardness values in the specified range. An additional ten couplings were ordered and received soon after the pump was returned to service. Independent hardness checks completed on these spare couplings agreed with vendor hardness results, thus providing some assurance to the licensee that the couplings currently installed in pump P-7C were of the correct hardness.
On August 9, 2011, SW pump P-7C was declared inoperable due to the failure of coupling #6. This failure was entered into the licensee corrective action program and identified as a SCAQ. The cause of the failure had not been determined at the time this SIT report was completed. However, hardness results obtained from an independent metallurgical lab from four of the in-service couplings, including the coupling that failed, were found to have at least one spot of higher hardness than specified, greater than Rockwell Hardness value of 32 Rc. No hardness values approached the 37-41 Rc observed in the 2009 failure (the values were less than 34Rc). Therefore, the inspectors were concerned that the licensee did not ensure the vendor-supplied couplings with correct hardness values in 2009. Pending final results and conclusions from the metallurgical analysis and root cause analysis, the licensee may have failed to take measures in 2009 to assure that purchased material and equipment conform to the procurement documents, including provisions for, as appropriate, objective evidence of quality furnished by the contractor as required by Appendix B.
Operating experience available to the licensee from 1991 through 2009, identified that Martensitic 416SS could be more susceptible to IGSCC, dependant on the heat treating process. Some of the previous operating experience is listed below:
- September 12, 1991 - Beaver Valley Unit 1 experienced a coupling failure of a Byron-Jackson centrifugal pump as a result of embrittlement of 410SS due to improper tempering temperatures and potential impurities. The failure mechanism was IGSCC;
- September 1, 1993 - NRC Information Notice 93-68, Failure of Pump Shaft Coupling Caused by Temper Embrittlement During Manufacture, is issued informing licensees of problems stemming from temper embrittlement of 410SS supplied by Byron-Jackson (referring to the 1991 Beaver Valley event); the document noted in the material analysis that the toughness of the failed component was extremely low;
- September 22, 1993 - Indian Point Unit 2 experienced a pump failure due to a failed coupling attributed to a non-ductile fracture because of temper embrittlement of the 410SST used; the failure mechanism was IGSCC;
- September 1, 2003 - Perry Unit 1 experienced a coupling disassembly event on their emergency SW pump A. The failure mechanism was by IGSCC and no performance effects were noticed before the failure of the coupling;
- May 21, 2004 - Perry Unit 1 experienced another emergency SW pump coupling failure; the failure mechanism was by IGSCC, similar to the 2003 event;
- 2006 - INPO Operating Experience Digest 2006-02 - INPO released a document that discusses SW pump (shaft, coupling and impeller) failures occurring in the industry. The report noted that 12 failures occurred between 1998 and 2006, with the most frequent including corrosion causing coupling separation. One of the common causes identified was improper heat treatment of the material during manufacturing; and
- February 9, 2007 - NRC Information Notice 2007-05, Vertical Deep Draft Pump Shaft and Coupling Failures, was issued and referenced the coupling failures from IGSCC that occurred at Columbia Generating Station. The couplings were 410 Martensitic SS and were susceptible to tempering embrittlement. The operating experience review identified at least 23 essential SW pump shaft and coupling failures since 1983 involving more than six different manufacturers.
The 2009 coupling failure root cause evaluation was focused on the material hardness. It did not fully evaluate the effect of other material properties, such as toughness which is discussed in the operating experience, nor the change of material properties from heat treatment on the susceptibility of the couplings to IGSCC. Corrective actions after the 2009 failure did not address the heat treatment process. The inspectors were concerned that the licensee may not have adequately addressed operating experience in the design of the original 416SS couplings or the 2009 replacement couplings as required by Appendix B, Criteria III. And the corrective actions taken after the 2009 failure may not have prevented recurrence of this SCAQ as required by Appendix B, Criteria XVI. These concerns will be further evaluated as part of the NRC baseline inspection program, after the final metallurgical results are issued and the licensee root cause evaluation is completed. (URI 05000255/2011012-01; Adequacy of SW Pump Couplings).
4OA6 Meetings, Including Exit
On September 1, 2011, the special inspection team leader presented the preliminary inspection results to Mr. Anthony Vitale and members of his staff. No proprietary information is included in this inspection report.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- T. Vitale, Entergy/Site Vice President
- J. Haumersen, Entergy/System Engineering Manager
- B. Kemp, Entergy/Design Engineering Manager
- D. MacMaster, Entergy/Acting Design Engineering Manager
- D. Hamilton, Entergy/General Manager Plant Operations
- A. Blind, Entergy/Engineering Director
D.Malone, Entergy/Acting Nuclear Safety Assurance Dir.
- J. C. Plachta, Entergy/Quality Assurance Manager
- P. Deniston, Entergy/Sr. Engineer
- J. Forehand, Entergy/Engineering Supervisor
- B. Dotson, Entergy/Licensing
J.Pennington, Entergy/Corporate MP&C
NRC Personnel
J Ellegood, Senior Resident Inspector, Palisades
- J. Giessner, Branch Chief
- K. Stoedter, Acting Branch Chief
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened and Closed
None
Closed
None
Opened
- 05000255/2011012-01 URI Adequacy of SW Pump Couplings