IR 05000255/2011012

IR 05000255-11-012, on 08/15/2011 - 09/01/2011, Palisades Nuclear Plant, Inspection Procedure 93812, Special Inspection

ML112780190
Person / Time
Site:	Palisades
Issue date:	10/04/2011
From:	West S K Division Reactor Projects III
To:	Vitale A Entergy Nuclear Operations
References
IR-11-012
Download: ML112780190 (33)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION REGION III 2443 WARRENVILLE ROAD, SUITE 210 LISLE, IL 60532-4352 October 4, 2011 Mr. Anthony Vitale Vice-President, Operations Entergy Nuclear Operations, Inc.

Palisades Nuclear Plant 27780 Blue Star Memorial Highway Covert, MI 49043

-9530

SUBJECT: PALISADES NUCLEAR PLANT

- NRC SPECIAL INSPECTION TEAM (SIT) REPORT 05000255/2011012

Dear Mr. Vitale:

On September 1, 2011, the United States Nuclear Regulatory Commission (NRC) completed a special inspection at your Palisades Nuclear Plant. The inspection was conducted in response to the circumstances surrounding the failure of the Service Water Pump P

-7C, on August 9, 2011. Based on the risk and deterministic criteria specified in Management Directive 8.3, "NRC Incident Investigation Program," a special inspection was initiated in accordance with Inspection Procedure 93812, "Special Inspection."

The special inspection charter (Attachment 2 of the enclosure) provides the basis and focus areas for the inspection.

The enclosed inspection report documents the inspection results, which were discussed with you and other members of your staff at the exit meeting on September 1, 2011.

The determination that the special inspection would be conducted was made on August 15, 2011, and the on

-site inspection commenced the same day.

The inspection examined activities conducted under your license as they relate to safety, compliance with the Commission's rules and regulations, and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, and interviewed personnel.

Based on the results of this inspection, no findings of significance were identified. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room and from the Publicly Available Records (PARS) component of NRC's document system, Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.qov/readinq

-rm/adams.html (the Public Electronic Reading Room).

Sincerely,/RA/

Steven West, Director Division of Reactor Projects Docket No. 50

-255 License No. DPR

-20

Enclosure:

Inspection Report 05000255/20110 12; w/Attachment s: 1. Supplemental Information 2. Memo to Jandovitz 3. Palisades Timeline cc w/encl:

Distribution via ListServ

Enclosure U. S. NUCLEAR REGULATORY COMMISSION REGION I II Docket No.:

50-255 License No.:

DPR-20 Report No.:

05000 255/201 1 0 12 Licensee:

Entergy Nuclear Operations, Inc.

Facility:

Palisades Nuclear Plant Location:

Covert, MI Dates: August 15, 201 1 - September 1, 201 1 Inspectors

J. Jandovitz , Project Engineer (Lead) A. Shaikh, Reactor Inspector P. Smagacz, Reactor Engineer Approved by:

John B. Giessner, Chief Branch 4 Division of Reactor Projects

Enclosure

SUMMARY OF FINDINGS

IR 05000 255/20 10 012; 08/15/20 1 1 09/01/20 1 1; Palisades Nuclear Plant;

Inspection Procedure 93812, Special Inspection

. This report covers a 4

-day period (August 15

-19 , 20 1 1) of on-site inspection and in-office review through September 1, 2011. A team, comprised of three regional inspectors, conducted this special inspection.

The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG

-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

A. No findings were identified.

NRC-Identified and Self-Revealing Findings

B. No findings were identified.

=

Licensee-Identified Violations===

REPORT DETAILS

At 12:02 p.m. EDT on August 9, 2011, the licensee entered an unplanned 72

-hour shutdown Technical Specification limiting condition for operation (LCO) action statement (TSAC)due to failure of service water (SW) pump P-7C. The SW system includes three mot or-driven , vertical multistage deep draft pumps that provide cooling water to safety related equipment (component cooling water, containment air coolers, diesel generators, control room coolers, etc.). Each pump has 50 percent capacity

and two pumps are required to be in service

however, the technical specification requires all three pumps to be operable.

All three pumps were running prior to the failure. Summary of the Degrade d Condition The licensee observed a low differential pressure condition and low pump amperage following the failure of the pump. The licensee disassembled pump P-7C and determined that the pump's first line shaft to second line shaft coupling, #6, had failed. The licensee replaced all of the pump P-7C couplings and two portions of the pump shaft. After completing the required maintenance and surveillance tests, the licensee returned pump P

-7C to service on August 12 , within the time allowed by the TSAC. Pump P-7C had previously experienced another coupling failure in September 2009. The SW pump couplings in both cases were made of 416 Stainless Steel. Subsequent metallurgical analysis determined the coupling's hardness was significantly higher than the design specification; and the failure was caused by Intergranular Stress Corrosion Cracking (IGSCC)

. The licensee's corrective actions in 2009 included replacing all the SW pump P-7C couplings with couplings of the same design and hardness criteria

. However, hardness test results of several couplings removed from pump P-7C on August 10, 2011, indicate d that these couplings were not within the hardness specification. Specifically, three of the couplings had at least one test location where the hardness measurement w as higher than the licensee's design specification criteria. Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, "NRC Incident Investigation Program," a special inspection was initiated in accordance with NRC Inspection Procedure 93812, "Special Inspection Team." The special inspection charter, dated August 15, 2011, is included as Attachment 2. The team reviewed technical and design documents, procedures, maintenance records, and corrective action documents

interviewed station personnel and consultants

and performed plant walkdowns of plant equipment. A list of specific documents reviewed is provided in Attachment 1.

Inspection Scope

4OA5 Other Activities

- Special InspectionIn accordance with the Charter, the following items were reviewed.

(93812)

.1 Establish a historical sequence of events related to service water pump coupling failures up to and including the most recent failure on August 9, 2011

a. . The inspectors reviewed corrective action documents, operating experience reports, technical literature, operation history, surveillance procedures and results, vendor procurement and manufacturing records, and interviewed plant personnel for information related to the use of 416 Stainless Steel (SS) material, and similar material such as 410SS, in power plant applications, particularly, SW pumps. Inspection Scope A detailed historical timeline of activities and information involving use and experience is contained in Attachment 3. b. No findings were identified.

Findings and Observations The inspectors were concerned that the licensee had not adequately evaluate d operating experience to recognize the vulnerabilities of 416SS material to IGSCC based on material properties, particularly heat treatment. This concern is di scussed further in the unresolved item (URI) in Section

.0 7 of this report.

However, the inspectors did not identify any operational or surveillance results or observations that may have predicted the failure of SW pump P-7 C on August 9, 2011.

.2 Review the licensee's basis for operability of the service water pumps

P-7A, P-7B, and P-7C. Determine whether the licensee ha d an adequately supported basis for operability of pumps P-7A and P-7B and for returning pump P-7C to service. Determine if additional corrective actions are required a. . The inspectors reviewed the prompt operability basis for all three SW pumps contained in Engineering Change (EC) 31208 and EC 31218. The inspectors identified run times, service conditions and life, starts and stops, heats of materials, and differences in heat treatment for the in-service couplings.

Inspection Scope b. No finding s were identified

. Findings and Observations Based on th e information available, the inspectors did not disagree with the immediate or prompt operability evaluation s. The prompt operability for SW pumps P-7B and P-7C w as based on the higher confidence that the hardness of the couplings installed in the pumps were within the design specification range, and therefore less susceptible to IGSCC. Operability for pump P-7A was based on the fact that couplings installed in th at pump had been in operation the longest with no failures.

However, the inspectors noted that there was no discussion on temper embrittlement identified in the Structural Integrity (SI)report and in operating experience, as a factor that may affect the coupling s' reliability. Therefore, it was recognized that the results from the metallurgy and root cause evaluation may impact the basis for current operability and the couplings

' service life.

After the on

-site SIT inspection was complete, the licensee received initial metallurgical results on the failed coupling and initiated a near-term replacement plan of the couplings in all three SW pumps with couplings of a different material, more resistant to IGSCC.

At the time of the SIT exit, the couplings in pump P

-7A had been replaced with the new material. The NRC will continue to monitor results from the root cause investigation for impact on the operability basis for any pump that still has the 416SS couplings.

The residents will also evaluate the licensee's evaluation of past operability for all the SW pumps and the extent of condition evaluation for the 416SS components remaining in the pumps

.

.3 Evaluate activities associated with the licensee's repair of service water pump P-7C for adequacy.

Confirm that the repair complies with licensee and NRC requirements a. . The inspectors reviewed the licensee's repair and replacement activities associated with the rebuild of pump P

-7C following the 2011 pump coupling failure. This review included the document review of post

-maintenance testing of the pump as well as documentation of pre-installation testing of pump and motor. Also reviewed was the completed work order for this repair and replacement

, and the material specifications (heat treat curves, hardness results) of couplings that were installed in this rebuilt pump P

-7C. Inspection Scope b. No finding s were identified Findings and Observations The inspectors did not identify any issues relating to the 2011 pump P-7C repair.

.4 Review information and processes relied upon by Palisades to determine whether or not critical attributes of service water pump line

shaft couplings, such as hardness, are acceptable prior to coupling installation a. . The inspectors reviewed the information provided by the licensee regarding pump shaft couplings that included hardness measurements at time of procurement, independent lab results for coupling hardness measurements prior to coupling installation in the pump s, and the associated heat treatments for the couplings.

Also reviewed were Inspection Scope industry reports and operating experience regarding use of 416 Martensitic SS for these couplings and the 2009 SW pump P-7C coupling failure root cause evaluation.

b. No finding s were identified

. Findings and Observations The inspectors identified a concern with the adequacy of specifying coupling hardness as the only critical material parameter. This wa s not a conservative approach given the collection of scientific literature and industry operating experience describing the vulnerability of 416/410 Martensitic SS to stress corrosion cracking

and suggest ed use of toughness measurements in addition to hardness values to determine acceptability of 416/410 Martensitic SS as coupling material. However, ensuring hardness that values are below a certain allowable maximum , reduces the vulnerability of the couplings to I GSCC; but ha r dness itself may not be sufficient to ensure critical parameters are met.

This concern is captured as part of the URI included in Section

.07 of this report.

In late 2007 , an engineering change was initiated to replace the carbon steel components in the SW pumps with stainless steel components resistant to the erosion and corrosion issues the plant had experienced throughout its life. I n 2008 , couplings for the SW pumps were procured and fabricated from 416SS with specified criteria for material chemistry and a Rockwell C (Rc) hardness range of 28

-32 Rc. Information available from industry operating experience reference d that material, specifically water pump couplings , made from 410/416 SS were susceptible to IGSCC due to temper embrittlement and improper heat treatment.

The 2009 Palisades failure of SW pump P-7 C coupling was attributed to IGSCC due to significantly high hardness, ranging from 37

-41 Rc; and all couplings were replaced with the same material and design specification for hardness.

No additional criteria were specified for tempering or heat treatment.

Results from a similar coupling failure at Prairie Island in 2010

, showed that their coupling, supplied by same vendor as the Palisades pump couplings, failed due to IGSCC even though the hardness was near the specification criteria.

Absent any design criteria from the licensee, the heat treatment performed on the couplings was focused on the final product having a hardness value in the specified range of 28

-32 Rc. There was no consideration of the effect of heat treatment o n other material properties, such as toughness, that may affect susceptibility to IGSCC. The inspectors noted the heat treatments were different for couplings for each of the three pump s. Couplings for pumps P-7B and P-7C were double

-tempered. Also, the procedure used by the heat treatment vendor allowed multiple heat treatments (tempering) to achieve the desired hardness , and did not consider the affect of this process on IGSCC resistance.

Because of the literature and industry operating experience of the vulnerability of 416/410 Martensitic SS to IGSCC, the plant requested Structural Integrity to evaluate the ir 416SS couplings. The report was received in March 2011

, and suggested that the couplings were susceptible to temper embrittlement

, but did not conclude that there was an immediate operability concern. The plant was evaluating this information when the coupling failed on August 9.

.5 Evaluate the licensee's procedures and processes for evaluating/assessing the performance of vendors.

Emphasis should be placed upon determining why couplings that were believed to be of an appropriate hardness were later found to be of an inappropriate hardness. Review any evaluations/assessments of HydroAire that were completed by the licensee following the September 2009 service water pump P-7C coupling failure. Also, review any actions taken or direction provided by the licensee to HydroAire in response to the recent hardness test results

a. Inspection Scope

. The inspectors reviewed procurement procedures for the licensee and vendor, the 2009 root cause evaluation, source surveillance s completed since 2009, the industry audit of HydroAire, condition reports, corrective actions taken by the licensee and vendor after the 2009 coupling failure, purchase documents, and interviewed Quality Assurance personnel.

b. Findings and Observations

No finding s were identified

. The inspectors identified a concern that the licensee may not have provide d adequate vendor procurement control after the 2009 coupling failure

, which may have contributed to the 2011 coupling failure. The inspectors found that adequate procurement procedures and processes existed , but were not fully utilized after the 2009 coupling failure. This concern is included in the URI discussion in Section

.07 of this report

. The inspectors reviewed a vendor surveillance performed in March 2010 on parts for a spare SW pump. The surveillance noted that one pump part was heat

-treated , and the hardness reading s taken by the pump vendor and the heat treatment vendor were both out of the specified range. Therefore, the part was again heat

-treated, resulting in acceptable hardness values obtained by both vendors. The part was then identified as acceptable. This demonstrate d that as late as 2010, the re were still issues with the controls on heat treatment for SW pump parts.

Implementation of the corrective actions from the 2009 coupling failure resulted in some improvements to the vendor procurement processes and additional oversight.

This was based on: frequent vendor surveillance activities were conducted on new parts that found vendor performance was satisfactory

procedure EN

-MP-100, "Critical Procurements

," was implemented for procurements after 2009; and a NUPIC [Nuclear Procurement Issues Committee] audit conducted in April 2011 found that HydroAire was implementing an effective quality assurance (QA) program.

This is not concluding the design criteria supplied to the vendor was correct

, as discussed earlier, only that the parts supplied by the vendor appeared to meet licensee criteria. No additional actions were taken by or required of the vendor after the 2011 coupling failure , since the licensee considered th is failure the result of activities that occurred in 2009 , and that corrective actions implemented by the vendor then were effective

.

.6 Evaluate the licensee's operating experience program with an emphasis on operating experience related to pump components manufactured by HydroAire.

Specific attention should be given to the licensee's evaluation of a 2010 service water pump coupling failure at Prairie Island and the evaluation of a March 2011 report from Structural Integrity regarding the potential for coupling failure due to temper embrittlement

.

a. Inspection Scope

Operating experience regarding failures of couplings using 41 0/41 6 Martensitic SS was reviewed along with generic technical reports and metallurgical reports. The report of most interest were the Prairie Island coupling failure in 2010, the Structural Integrity report which was issued to address the 2009 coupling failure at Palisades, and the 2009 Palisades coupling failure root cause evaluation

.

b. Findings and Observations

No finding s were identified

. The inspectors identified a concern that operating experience and industry information may not have been incorporated into the design of the pump nor addressed adequately in the 2009 coupling failure replacement and root cause evaluation. Further discussion of this concern is included in the URI discussion in section

.07. Operating experience that existed as early as 1991 discussed failures of pump couplings attributed to embrittlement of 410SS and 416SS due to improper tempering temperatures and heat treatment.

In addition, industry and NRC reports were available prior to 2009 and discussed the effect of temper embrittlement on 410/416SS material.

The SI report was issued in March 2011 , but the licensee did not accept the report until August. This report discussed temper embrittlement as an issue that could affect coupling reliability. The licensee had not yet evaluated nor initiated corrective actions base d on the report.

The inspectors found the licensee did identify pertinent operating experience in development of the 2011 coupling failure root cause evaluation, although the root cause report was not issued by the completion of th is SIT report.

.7 Review the licensee's root cause evaluation plan and schedule.

Include the schedule for performing testing of any components removed from pump P-7C. Evaluate whether the root cause evaluation plan is of sufficient depth and breadth. Confirm that the time allowed to perform the root cause evaluation is commensurate with the safety significance of this issue. Communicate to the licensee that the NRC will inspect the completed root cause evaluation and the associated corrective actions as part of our normal inspection activities

.

a. Inspection Scope

The inspectors reviewed the root cause charter, schedule, team make-up and action plan; and discussed actions with root cause team

. b. Observations The inspectors found the root cause team was comprised of individuals dedicated solely to the root caus e investigation

they had no additional responsibilities. Most members had not been involved in the root cause for the 2009 failure. Team members included consultants

, including an independent pump manufacturer and the metallurgical lab contracted to perform the metallurgical analysis. The pump supplier, HydroAire , provided input to the root cause but was not a member. The root cause timeline started August 16

, and was scheduled to be completed within 30 days in accordance with procedures

. The root cause investigation was determined to be of adequate depth and breadth to be successful in determining the actual root cause. The action plan included the following aspects: improper hardness

improper material

over-torque from foreign material

shaft m isalignment; Stress Corrosion Cracking

incorrect clearances in coupling to shaft

shaft faces not square

pump alignment

shaft wobble

reverse rotation

coupling defects

operation at critical speed

operation loads

strainer blockage

maintenance practices; and fatigue failure. The inspectors reviewed the draft metallurgical plan and considered the metallurgical analysis plan for selected coupling to be thorough, although the licensee had not yet finalized the plan or determined which couplings would be included in the plan.

The NRC will continue to inspect the metallurgical results and root cause report under the normal baseline procedures

.

Unresolved Item: Adequacy of SW Pump Couplings Introduction

A URI was identified by the inspectors during review of the SW pump P

-7C coupling failure in 2011. Specifically, the inspectors were concerned with the adequacy of the procurement of the replacement couplings after the 2009 coupling failure on the same pump; whether operating experience was adequately incorporated into design specifications

and whether the corrective actions taken after the 2009 failure were adequate to prevent recurrence.

Description

Palisades has three safety

-related SW pumps that draw water from Lake Michigan to provide cooling to safety

-related and no n-safety-related components. Each pump has eight couplings to connect shaft sections. In 2009, the plant began a material change for the couplings and shafts from carbon steel to 416 Martensitic SS due to erosion issues of the carbon steel material. The material would be changed in series in of pumps P-7 A , P-7C, and P-7B respectively. The timeline (Attachment 3) includes specific dates

. In 2009 , SW pump P-7C failed due to cracked coupling #

7, resulting into entr y a TSAC , requiring the plant to shut down if the pump was not returned to service within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. Metallurgical data on the failed coupling revealed the coupling hardness was significantly higher than the design specification. The specified hardness range was Rockwell C value of 28

-32 Rc. Hardness readings on the failed coupling ranged from 37

-41 Rc. The vendor, HydroAire, had supplied the couplings with certifications that the couplings were within the specified hardness

. This failure was entered into the licensee's corrective action program and identified as a significant condition adverse to quality (SCAQ).

At the time of the failure, no spare couplings were available. Immediate discussions with HydroAire ensued to obtain replacement couplings within the 72

-hour timeframe. Based on the hardness data received on the failed coupling, which conflicted with the certification received from the vendor, the licensee verbally imposed addition al requirements to obtain independent laboratory hardness checks on the replacement couplings.

The requirement for independent hardness checks on the replacement couplings was not incorporated into the procurement process. Procurement documents were not updated with this additional requirement , and procurement personnel, such as the licensee procurement manager at the vendor site or the plant receipt inspection personnel , were not made aware of the additional requirement. Procedure EN

-MP-100, "Critical Procurements," was not implemented even though criteria in the procedure, such as previous vendor issues, how critical the couplings were on plant operation, and the short procurement timeframe, appear to have been satisfied.

Eight replacement couplings were received and immediately installed in the pump

the pump was returned to service within the 72

-hour requirement. Soon after the pump was returned to service, the licensee discovered that independent hardness checks were not performed. The vendor did provide certifications indicating that the couplings had hardness values in the specified range. An additional ten couplings were ordered and received soon after the pump was returned to service. Independent hardness checks completed on these spare coupling s agreed with vendor hardness results , thus providing some assurance to the licensee that the couplings currently installed in pump P-7C were of the correct hardness

. On August 9, 2011, SW pump P-7C was declared inoperable due to the failure of coupling #6. This failure was entered into the licensee corrective action program and identified as a SCAQ. The cause of the failure ha d not been determined at the time this SIT report was completed. However , hardness results obtained from an independent metallurgical lab from four of the in-service couplings, including the coupling that failed, were found to have at least one spot of higher hardness than specified, greater than Rockwell Hard ness value of 32 Rc. No hardness values approached the 37-41 Rc observed in the 2009 failure (the values were less than 34Rc). Therefore, the inspectors were concerned that the licensee did not ensure the vendor

-supplied coupling s with correct hardness values in 2009. Pending final results and conclusions from the metallurgical analysis and root cause analysis, the licensee may have failed to take measures in 2009 to assure that purchased material and equipment conform to the procurement documents, including provisions for, as appropriate, objective evidence of quality furnished by the contractor as required by Appendix B

. Operatin g experience available to the licensee from 1991 through 2009

, identified that Martensitic 416SS could be more susceptible to IGSCC

, dependant on the heat treating process. Some of the previous operating experience is listed below:

September 12, 1991

- Beaver Valley Unit 1 experience d a coupling failure of a Byron-Jackson centrifugal pump as a result of embrittlement of 410SS due to improper tempering temperatures and potential impurities.

The failure mechanism was IGSCC

September 1, 1993

- NRC Information Notice 93

-68, "Failure of Pump Shaft Coupling Caused by Temper Embrittlement During Manufacture

," is issued informing licensees of problems stemming from temper embrittlement of 410SS supplied by Byron

-Jackson (referring to the 1991 Beaver Valley event)

the document noted in the material analysis that the toughness of the failed component was extremely low

September 22, 1993

- Indian Point Unit 2 experience d a pump failure due to a failed coupling attributed to a non-ductile fracture because of temper embrittlement of the 410SST used

the failure mechanism was IGSCC

September 1, 2003

- Perry Unit 1 experienced a coupling disassembly event on their emergency SW pump A. The failure mechanism was by IGSCC and no performance effects were noticed before the failure of the coupling

May 21, 2004

- Perry Unit 1 experience d another emergency SW pump coupling failure; the failure mechanism was by IGSCC, similar to the 2003 event; 2006 - INPO Operating Experience Digest 2006 INPO release d a document that discusses SW pump (shaft, coupling and impeller) failures occurring in the industry. The report note d that 12 failures occurred between 1998 and 2006

, with the most frequent including corrosion causing coupling separation. One of the common causes identified was improper heat treatment of the material during manufacturing; and February 9, 2007

- NRC Information Notice 2007

-05, "Vertical Deep Draft Pump Shaft and Coupling Failures

," was issued and reference d the coupling failures from IGSCC that occurred at Columbia Generating Station. The couplings were 410 Martensitic SS and were susceptible to tempering embrittlement. The operating experience review identified at least 23 essential SW pump shaft and coupling failures since 1983 involving more than six different manufacturers.

The 2009 coupling failure root cause evaluation was focused on the material hardness. It did not fully evaluate the effect of other material properties, such as toughness which is discussed in the operatin g experience

, nor the change of material properties from heat treatment on the susceptibility of the couplings to IGSCC. Corrective actions after the 2009 failure did not address the heat treatment process. The inspectors were concerned that the licensee may not have adequately address ed operatin g experience in the design of the original 416SS couplings or the 2009 replacement couplings as required by Appendix B, Criteria II I. A nd the corrective actions taken after the 2009 failure may not have prevent ed recurrence of this SCAQ as required by Appendix B, Criteria XVI. These concerns will be further evaluated as part of the NRC baseline inspection program

, after the final metallurgical results are issued and the licensee root cause evaluation is completed.

(URI 05000255/2011012

-01; Adequacy of SW Pump Couplings).

4OA6 Meetings, Including Exit

On September 1, 2011, the special inspection team leader presented the preliminary inspection results to Mr. Anthony Vitale and members of his staff. No proprietary information is included in this inspection report.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

T. Vitale, Entergy/Site Vice President

J. Haumersen, Entergy/System Engineering Manager

B. Kemp, Entergy/Design Engineering Manager

D. MacMaster, Entergy/Acting Design Engineering Manager

D. Hamilton, Entergy/General Manager Plant Operations

A. Blind, Entergy/Engineering Director

D.Malone , Entergy/Acting Nuclear Safety Assurance Dir.

J. C. Plachta, Entergy/Quality Assurance Manager

P. Deniston, Entergy/Sr. Engineer

J. Forehand, Entergy/Engineering Supervisor

B. Dotson, Entergy/Licensing

J.Pennington , Entergy/Corporate MP&C

NRC Personnel

J Ellegood , Senior Resident Inspector, Palisades

J. Giessner, Branch Chief

K. Stoedter, Acting Branch Chief

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

None

Closed

None

Opened

05000 255/201 1 0 12-01 URI Adequacy of SW Pump Couplings

LIST OF DOCUMENTS REVIEWED

Condition Reports

Title Date/Revision No.

CR 04-02598 Root Cause Report for Perry Emergency Service Water Pump A Upper Shaft Coupling Sleeve Repeat Failure.

CR 2011-03975 Relevant OE Not Considered in CR

-PLP-2009-4519 (Root Cause Evaluation for the 2009 Service Water Pump P-7C coupling Failure)

August 12, 2011

CR-PLP-2009-04519 Root Cause Evaluation Report for Service Water Pump P

-7C Failure to Provide Discharge Pressure March 4, 2009

CR-PLP-2009-04542 Evaluate Early Station Response to Failure of P

-7C September 30, 2009

CR-PLP-2009-04547 Inadequate FME Postings and Boundaries for Initial Disassembly of P

-7C September 30, 2009

CR-PLP-2009-04561 Job Safety Hazard Analysis Not at Job Location for P

-7C September 30, 2009

CR-PLP-2009-04571 Station in Event Response for Unexpected Loss of Service Water Pump P-7C September 30, 2009

CR-PLP-2009-04593 Steps Not Followed Within WI

-SWS-M-04 for Reassembly of P

-7C October 1, 2009

CR-PLP-2009-04613 Administrative Issues Involving Work Scheduling in Response to Emergent Equipment Issues October 2, 2009

CR-PLP-2009-04616 Procedure Not In

-Hand for Packing Adjustments for P

-7C Pump Shaft October 2, 2009

CR-PLP-2009-04659 Initial CR for CR

-PLP-2009-04806 October 6, 2009

CR-PLP-2009-04806 Apparent Cause Evaluation to Evaluate Miscommunication of Independent Testing October 15, 2009

CR-PLP-2009-04877 Service Water Pump P

-7C Exceeds Maintenance Rule Unavailability October 21, 2009

CR-PLP-2009-05647 Documentation of HydroAire Root Cause Evaluation of Failed Line Shaft Coupling on P

-7C December 9, 2009

CR-PLP-2009-05874 Evaluation of Past Operability Issues of

P-7C Prior to the Event on 09/29/2009

December 22, 2009

CR-PLP-2010-01015 Error Identified in Licensee Event Report 2010

-001, Potential Loss of Safety Function Due to a Service Water Pump Shaft Coupling Failure March 10, 2010

CR-PLP-2011-03902 Root Cause Report Service Water Pump 7-C Line Shaft Coupling Failure September 8, 2011

CR-PLP-2011-03920 Non-covered Worker Has Exceeded Limits of EN

-FAP-OM-006 August 10, 2011

CR-PLP-2011-03921 A-Frame Designated for Lifting P

-7C Service Water Pump Missing August 10, 2011

CR-PLP-2011-03929 Non-Conforming Components Removed from P

-7C August 10, 2011

CR-PLP-2011-03938 Outside Diameter Value and/or Tolerance in Error August 10, 2011

CR-PLP-2011-03939 Degradation of Coating with Blisters on the Discharge Head of P

-7C August 10, 2011

CR-PLP-2011-03942 Non-covered Worker Has Exceeded Limits of EN

-FAP-OM-006 August 11, 2011

CR-PLP-2011-03944 Personnel Pulled from Training for Plant Event Response August 11, 2011

CR-PLP-2011-03946 P-7C Work Temporarily Stopped Due to Non-Tethered Tools in FME Zone 1

August 11, 2011

CR-PLP-2011-03947 Hardhat Lost During P

-7C Disassembly August 11, 2011

CR-PLP-2011-03948 Issues with Licensed Operator Training Identified August 11, 2011

CR-PLP-2011-03952 Chlorination Deferment without Potential Adverse Effect August 11, 2011

CR-PLP-2011-03959 Violation of Protected Equipment Boundary August 11, 2011

CR-PLP-2011-03961 Potential Extent of Condition on Service Water Pumps P

-7A and P-7B August 11, 2011

CR-PLP-2011-03963 Reverse Rotation Noticed of P

-7C During Breaker Opening August 11, 2011

CR-PLP-2011-03965 Engineer Exceeds Overtime Limit August 11, 2011

CR-PLP-2011-03966 Removed Couplings from P

-7C 2011 Failure Show Out

-of-Spec Hardness August 12, 2011

CR-PLP-2011-03966 Results of 4 Couplings Sent to Consumers Energy Laboratory Services for Hardness Testing August 12, 2011

CR-PLP-2011-03967 Service Water Pump P

-7C Lift Setting Not Validated Correctly August 12, 2011

CR-PLP-2011-03967 Final Pump Lift Setting Not Validated Correctly August 11, 2011

CR-PLP-2011-03968 Questioning of Pump Run in Dead Headed Condition August 11, 2011

CR-PLP-2011-03975 Relevant OE not Considered in CR

-PLP-2009-04519 August 12, 2011

LO-CAR-2009-00103 Nonconformance of Parts from HydroAire October 5, 2009

LO-PLPLO-2009-00065 Implementation of Effectiveness Review Plan for RCE CR

-PLP-2009-04519 October 16, 2009

Documents

Title Date/Revision No.

LER 2004-001-01 Emergency Service Water Pump Failure September 18, 2004

Certificate of Conformance Certificate of Conformance for

PO 10237148, Job# NQ5832

June 9, 2009

CP-PLP-2009-002 P-7C Service Water Pump Refurbishment Procurement Plan May 18, 2009

EC 31208 Engineering Basis for Service Water Pump P-7C to be Considered Fully Operable Post

-Repair (August 2011 Coupling Failure)

August 12, 2011

EC 31218 Engineering Basis for Service Water Pump P-7A and P-7B to be Considered Operable Considering P

-7C Coupling Failure (August 2011 Coupling Failure)

August 10, 2011

Examination Report 00286627

VT-2 (Pressure Test) of Service Water Pump P-7C Prior to Return to Service August 12, 2011

HydroAire NDR

Nonconformance and Discrepancy Report 09-55 October 2, 2009

LO-WTHQN-2011-00439 Source Activity for Entergy Palisades PO10262979* Rev 000

March 4, 2011

LO-WTHQN-2011-00920 Source Activity for Entergy Palisades PO10262979, HydroAire Job No. NQ

5940 July 20, 2011

MAT Project 0900609

Consumers Energy Palisades Pump P7C Shaft Coupling Failure Analysis October 5, 2009

NDR No. QANC

-09 97 HydroAire Nonconformance & Discrepancy Report and Part 21 Notification December 16, 2009

NUPIC Audit Report VA11

-011 NUPIC Audit of HydroAire, Inc.

June 1, 2011

Purchase Order 10190242

Procurement Documents for Pump P

-7A Couplings for 2009 Pump Refurbishment Revision 003

Purchase Order 10237148

Procurement Documents for Service Water Pump Refurbishment June 9, 2009

Purchase Order 10246213

Procurement Documents for Pump P

-7B Couplings for 2009 Pump Refurbishment Revision 004

Purchase Order 10253715

Procurement Documents for Pump P

-7C Couplings Following 2009 Coupling Failure Revision 003

Report No. 1100112.401

Structural Integrity Additional Review of Palisades Service Water Pump Couplings Revision 0

Report PO 10324017

Consumers Energy Rockwell Hardness Test Report for Couplings Removed from Failed Pump P

-7C August 10, 2011

SQA2010-00131 HydroAire Source Surveillance Report SS010-016 March 16, 2010

SQA2010-00561 HydroAire Source Surveillance Report SS010-158 November 2, 2010

WO 00190235

P-7C, Reduced Flow>RO

-144; Replace Pump Assembly per EC -14577 June 10, 2009

WO 00208591

P-7C: Motor Shaft Broke. Repair Pump October 28, 2009

WO 00209551

RO-216 Service Water Pump Flow Verification September 15, 2010

WO 00286627

Test Spare P

-7C Motor IAW EN

-MA-134 Offline Motor Electrical Testing August 11, 2011

WO 52211728

P-7C IST Service Water Pump October 1, 2009

Procedures

Title Date/Revision No.

ARP-7 Auxiliary Systems Scheme EK

-11 Revision 80

BMS-WI 09.15.03

Bodycote Heat Treatment of Martensitic Corrosion Resistant Steels Procedure July 10, 2006

BMS-WI-09-00-02 Bodycote Determination of Heat Treat Process Time Procedure November 17, 2006

EN-DC-115 Engineering Change Process Revision 11

EN-DC-313 Procurement Engineering Process Revision 6

EN-LI-102 Corrective Action Process Revision 16

EN-LI-118 Root Cause Evaluation Process Revision 14

EN-LI-118-01 Event & Causal Factor Charting Revision 0

EN-LI-118-02 Change Analysis Revision 0

EN-LI-118-03 Barrier Analysis Revision 0

EN-LI-118-04 Task Analysis Revision 0

EN-LI-118-05 Fault Tree Analysis Revision 0

EN-LI-118-06 Common Cause Analysis Revision 1

EN-LI-118-07 Behavioral Analysis Revision 0

EN-LI-118-08 Failure Modes Analysis Revision 0

EN-MA-118 Foreign Material Exclusion Revision 5

EN-MP-100 Critical Procurements Revision 8

EN-MP-101 Materials, Purchasing and Contracts Process Revision 5

EN-MP-111 Inventory Control Revision 3

EN-MP-117 Standardized Purchasing Process Revision 2

EN-MP-120 Material Receipt Revision 4

EN-MP-123 Term and Definitions Revision 4

EN-MP-125 Control of Material Revision 7

EN-QV-119 Corrective Action Requests, Supplier Stop Work Orders, and Recommendations Revision 7

EN-QV-120 Planning, Performing, and Reporting Source Activities Revision 7

EN-QV-121 Supplier Qualification/Maintenance of Qualifications Revision 5

EN-QV-122 Qualified Suppliers List Revision 4

EN-QV-123 Supplier Audits/Surveys Revision 4

EN-WM-100 Work Request (WR) Generation, Screening and Classification Revision 6

Procedure

No QO

-14 Palisades Inservice Testing Procedure for Service Water Pumps May 25, 2011

Procedure

No RO

-144 Comprehensive Pump Test Procedure Service Water Pumps P

-7A, P-7B and P-7C Revision 4, (4/10/09)

Procedure

No.F11358

-P-001 LPI Procedure for Metallurgical Examination of SW Pump P

-7C Coupling Components August, 2011

SOP-15 Service Water System Revision 51

SOP-5 Containment Air Cooling Revision 28

WI-MSM-M-26 Palisades Compression Fitting Installation and Inspection Revision 1

WI-SWS-M-04 Service Water Pump P

-7B and P-7C Removal, Inspection, and Reinstallation Revision 7

Drawings

Title Date/Revision No.

M0011-SH-0055 Palisades Service Water Pump Coupling Drawing.

M-398 Level Setting Diagram Service Water and Dilution Water Pump Revision 2

VEN-M11 Service Water Pump Rotating Element Assembly for Pumps P

-7A, 7B and 7C

Revision 2

LIST OF ACRONYMS

USED [[]]

ADAMS Agencywide Document Access Management System

EC Engineering Change

IGSCC Intergranular Stress Corrosion Cracking

IMC Inspection Manual Chapter

IP Inspection Procedure

IR Inspection Report

LCO Limiting Condition of Operation

LPI Lucius Pitkin, Inc

NRC [[]]

U.S. Nuclear Regulatory Commission

NUPIC Nuclear Procurement Issues Committee

PARS Publicly Available Records System

QA Quality Assurance

Rc Rockwell C

ROP Reactor Oversight Process

RS Reactor Safety

SBO Station Blackout

SCAQ Significant Condition Adverse to Quality

SI Structural Integrity

SS Stainless Steel

SW Service Water

TS Technical Specification

TSAC Technical Specification Action Statement

URI Unresolved Item

WR Work Request

ATTACH MENT

2 UNITED [[]]

STATES NUCLEAR REGULATORY COMMISSION

REGION [[]]

III 2443 WARRENVILLE ROAD,

SUITE 210

LISLE, IL 60532

-4352 Attachment

August 15, 2011 MEMORANDUM TO:

John Jandovitz, Project Engineer

Branch 5 Division of Reactor Projects

FROM: Steven West, Director

/RA/ Division of Reactor Projects

SUBJEC T: SPECIAL INSPECTION CHARTER

FOR [[]]

PALISA DES NUCLEAR

PLANT [[]]

FAILURE OF P

-7C

SERVIC E

WATER [[]]

PUMP COUPLING

ON [[]]

AUGUST 9, 2011

A potential repetitive failure of a coupling on the Palisades P

-7C service water pump was recently self

-revealed. The coupling failure is the subject of a Special Inspection that you have been identified to lead. A short discussion of the event follows.

On August 9, 2011, Palisades Nuclear Plant experienced a sudden loss of the P

-7C service water pump while it was in service. This failure resulted in the licensee entering a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> Limiting Condition for Operation due to the inoperability of one service water train. The licensee disassembled the P

-7C pump and determined that the pump's first

line shaft to second line shaft coupling had failed. The licensee replaced all of the P

-7C pump's couplings and two portions of the pump shaft. After completing the required maintenance, the licensee returned pump P

-7C to service on August 12, 2011

Based upon our initial review, we are concerned that the licensee may not have taken effective corrective actions following a P

-7C service water pump coupling failure that occurred in September 2009. Following the September 2009 event, the licensee determined that the pump coupling failed due to improper heat treatment. The improper heat treatment resulted in the coupling's hardness being too high. The licensee's corrective actions included replacing the

P-7C SW pump's couplings with couplings of appropriate hardness. Recent hardness test

J. Jandovicz

-2- results of several couplings removed from the P

-7C pump on August 10, 2011, indicate that these couplings were of an inappropriate hardness. Specifically, each coupling had at least one test location where the hardness measurement exceeded the licensee's acceptance criteria. In addition, one coupling had multiple test locations that exceeded the acceptance criteria. Testing of the failed coupling produced hardness results that were within specifications. Based upon the hardness test results and a March 2011 report provided to the licensee by Structural Integrity, another metallurgical failure mechanism may be contributing to the coupling failures. Lastly, this event raises questions about the effectiveness of the licensee's program for assessing and applying operating experience and its oversight of vendor activities.

The NRC has also learned that the remaining service water pumps, P

-7A and P-7B, have the same type of coupling installed. As a result, these pumps may also be susceptible to a coupling failure. The sequence of events and the root and contributing causes for this issue are being investigated by the licensee.

Based on the deterministic and risk

-based criteria in Management Directive 8.3, a Special Inspection at Palisades will commence on August 15, 2011. The Special Inspection Team, which is being led by you, will include Atif Shaikh. Other members may be assigned if specific

needs are identified.

The special inspection will determine the sequence of events, and will evaluate the facts, circumstances, and the licensee's actions surrounding this issue. The Special Inspection Charter for you and your team is enclosed.

Enclosure: As Stated

cc w/encl:

M. Satorius

C. Pederson

G. Shear S. Reynolds

K. O'Brien

D. Roberts,

RI [[]]

J. Clifford,

RI [[]]

P. Wilson,

RI [[]]

S. Weerakkody,

RI [[]]

R. Croteau,

RII [[]]

W. Jones,

RII [[]]

J. Munday,

RII [[]]

H. Christensen,

RII [[]]

K. Kennedy,

RIV [[]]

T. Pruett,

RIV [[]]

A. Vegel,

RIV [[]]

J. Lara,

RIII

V. Mitlying

P. Chandrathil

N. Valos D. Merzke A. Barker A. Shaikh RidsNrr

PMPalisadesResource

NRR Reactive

Inspection@nrc.gov

J. Jandovitz

-2- The NRC has also learned that the remaining service water pumps, P

-7A and P-7B, have the same type of coupling installed. As a result, these pumps may also be susceptible to a coupling failure. The sequence of events and the root and contributing causes for this issue are being investigated by the licensee.

Based on the deterministic and risk

-based criteria in Management Directive 8.3, a Special Inspection at Palisades will commence on August 15, 2011. The Special Inspection Team, which is being led by you, will include Atif Shaikh. Other members may be assigned if specific

needs are identified.

The special inspection will determine the sequence of events, and will evaluate the facts, circumstances, and the licensee's actions surrounding this issue. The Special Inspection Charter for you and your team is enclosed.

Enclosure: As Stated

cc w/encl:

M. Satorius

C. Pederson

G. Shear S. Reynolds

K. O'Brien

D. Roberts,

RI [[]]

J. Clifford,

RI [[]]

P. Wilson,

RI [[]]

S. Weerakkody,

RI [[]]

R. Croteau,

RII [[]]

W. Jones,

RII [[]]

J. Munday,

RII [[]]

H. Christensen,

RII [[]]

K. Kennedy,

RIV [[]]

T. Pruett,

RIV [[]]

A. Vegel,

RIV [[]]

J. Lara,

RIII

V. Mitlying

P. Chandrathil

N. Valos D. Merzke A. Barker A. Shaikh RidsNrr

PMP alisadesResource

NRR Reactive Inspection@nrc.gov

DOCUMENT NAME:

G:\DRPIII\PALI\Palisades Special Inspection Charter for P7C coupling

failure.docx

See Previous Concurrence

Publicly Available

Non-Publicly Available

Sensitive Non-Sensitive To receive a copy of this document, indicate in the concurrence box "C" = Copy without attach/encl "E" = Copy with attach/encl "N" = No copy

OFFICE [[]]

RIII RIII:

NAME [[]]

KStoedter:dtp

SW est

DATE 08/15/11 08/15/11

OFFICI AL

RECORD [[]]

COPY

PALISADES SPECIAL INSPECTION CHARTER

This Special Inspection Team is chartered to assess the circumstances surrounding the failure of a coupling on the P

-7C service water pump on August 9, 2011. The Special Inspection will be conducted in accordance with Inspection Procedure 93812, "Special Inspection." The

special inspection will include, but not be limited to, the items listed below. This charter may be revised based on the results and findings of the inspection.

1. Establish a historical sequence of events related to service water pump coupling failures up to and including the most recent failure on August 9, 2011.

2. In light of the failure of the P

-7C service water pump, review the licensee's basis for operability of the P

-7A, P-7B, and P-7C service water pumps. Determine whether the licensee has an adequately supported basis for operability of the P

-7A and P-7B pumps and for returning the P-7C pump to service. Determine if additional corrective actions are required. 3. Evaluate activities associated with the licensee's repair of the P

-7C pump for adequacy. Confirm that the repair complies with licensee and NRC requirements.

4. Review information and processes relied upon by Palisades to determine whether or not critical attributes of service water pump line shaft couplings, such as hardness, are

acceptable prior to coupling installation.

5. Evaluate the licensee's procedures and processes for evaluating/assessing the performance of vendors. Emphasis should be placed upon determining why couplings

that were believed to be of an appropriate hardness were later found to be of an inappropriate hardness. Review any evaluations/assessments of HydroAire that were completed by the licensee following the September 2009 P

-7C coupling failure. Also, review any actions taken or direction provided by the licensee to HydroAire in response to the recent hardness test results.

6. Evaluate the licensee's operating experience program with

an emphasis on operating experience related to

pump components manufactured by Hyrdo

-Aire. Specific attention should be given to the licensee's evaluation of a 2010 service water pump coupling failure at Prairie Island and the evaluation of a March 2011 report from Structural Integrity regarding the potential for coupling failure due to temper embrittlement.

Review the licensee's root cause evaluation plan and schedule. Include the schedule for

performing testing of any components removed from the P

-7C pump. Evaluate whether the root cause evaluation plan is of sufficient depth and breadth. Confirm that the time allowed to perform the root cause evaluation is commensurate with the safety significance of this issue. Communicate to the licensee that the NRC will inspect the completed root cause evaluation and

the associated corrective actions as part of our normal inspection activities.

Additional Inspection Requirements

1. Determine if there are any lessons learned from this Special Inspection.

Charter Approval

/RA by Karla Stoedter for/

8/15/11 John Giessner, Chief

Branch 4 Division of Reactor Projects

_________/RA/_ 8/15/11_______

Steven West, Director

Division of Reactor Projects

ATTACHMENT 3

Palisades P

-7C Service Water Pump Timeline

September 12, 1991

Beaver Valley Unit 1 experiences a coupling failure of a Byron

-Jackson centrifugal pump as a result of embrittlement of

410 SST due to improper tempering temperatures and potential impurities. The failure mechanism was

IGSCC.

September 1, 1993

NRC Information Notice 93

-68, "Failure of Pump Shaft Coupling Caused by Temper Embrittlement During Manufacture" is issued informing

licensees of problems stemming from temper embrittlement of 410 SS supplied by Byron

-Jackson (referring to the 1991 Beaver Valley event).

September 22, 1993

Indian Point Unit 2 experiences a pump failure via a failed coupling attributed to a non

-ductile fracture because of temper embrittlement of the

410 SST used. The failure mechanism was

IGSCC.

September 1, 2003

Perry Unit 1 experienced a coupling disassembly event on their Emergency Service Water 'A' pump. The failure mechanism was by IGSCC and no performance effects were

noticed before the failure of the coupling. The shafts were 416SS.

May 21, 2004 Perry experiences additional failure s of Emergency Service Water pump

shafts composed of

416 SS. The failure mechanism was by

IGSCC

. 2006 INPO Operating Experience Digest 2006

-02 - INPO releases a document that discusses service water pump (shaft, coupling and impeller) failures

occurring in the industry. The report notes that 12 failures occurred between 1998 and 2006 with the most frequent including corrosion causing coupling separation. One of the common causes identified was

improper heat treatment of the material during manufacturing.

February 9, 2007

NRC Information Notice 2007

-05, "Vertical Deep Draft Pump Shaft and Coupling Failures" is issued that references the coupling failures from

IGSCC that occurred at Columbia Generating Station. The couplings were

TP410 martensitic stainless steel and were susceptible to tempering embrittlement. The operating experience review identified

at least 23 essential SW pump shaft and coupling failures since 1983 involving more than six different manufacturers.

December ,

2007 EC -5000121762 was initiated to change the

SW pump internal components, in including the shaft couplings, from carbon steel to 416

SS.

April 4, 2009

P-7A couplings and shaft

s were changed to

416 SS from carbon steel under

WO 51637416 (Heat Number 49293)

April 10, 2009

P-7C fails Technical Specification Surveillance Test RO

-144 , "Comprehensive Pump Test Procedure Service Water Pumps P

-7A, P-7B and P-7C ," via a low flow condition and the pump is declared inoperable

(test performed during refueling outage)

April 25, 2009

P-7C declared Operable with Compensatory Measures due to Op Eval showing that P

-7C would demonstrate its 30 day mission time with degraded flow prior to startup from outage

May 19, 2009

Purchase Order #10237148 issued to HydroAire to re

-build Layne and Bowler pump to replace P

-7C May 21-22, 2009 Pump P-7C couplings were

heat treated once and then received a second heat treatment.

June 12, 2009

P-7C couplings and shaft changed to

416 SS from carbon steel under

WO 190235 September 29, 2009

P-7C pump failed due to coupling #7 failure from IGSCC. The pump had a total run time of 2414.44 hours5.092593e-4 days <br />0.0122 hours <br />7.275132e-5 weeks <br />1.6742e-5 months <br /> with 13 starts/stops.

October 1, 2009

First two couplings from HydroAire arrive (Heat Order #20163) early. Remaining six couplings arrive later in day (Heat Order #20166) and installed.

October 2, 2009

P-7C restored to operation

May 13, 201

P-7B couplings and shaft changed to

416 SS from carbon steel under

WO 213444 (Heat

Number W2464-B50) July, 2010

Prairie Island experience

d a failure of

0 SS couplings from

IGSCC. These couplings were also supplied by HydroAire. March, 2011 Structural Integrity (SI) report received at Palisades concerning failure mechanisms of

416 SS . August, 2011

SI report accepted

by Licensee.

August 9, 2011

0700 All 3 SW pumps in

-service with differential pressures of 1/2/1 psid, respectively and header pressure is 73 psig and stable. P-7C operated for a total of

,114.86 hours9.953704e-4 days <br />0.0239 hours <br />1.421958e-4 weeks <br />3.2723e-5 months <br /> with a total of 95 starts/stops and had passed all surveillances

202 Control room receives following alarms:

EK -1163 "Critical Service Water Header 'B'

LO Pressure;"

EK -1164 "Critical Service Water Header 'A'

LO Pressure;"

EK -1165 "Non-Critical Service Water

LO Press;" EK

-0557 "Diesel Gen No. 1

-2 Trouble;" and

EK -1132 "Service Water Pump P7A Basket

STR [[]]

HI [[]]

DP. Operators enter Off Normal Procedure 6.1 for Loss of Service Water and enter

TS [[]]

LCO 3.7.8.A1 (restore SW pump P

-7C to operable in 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />)

203 Nuclear Plant Operator reports that P

-7C has no discharge pressure and a loud banging noise from the pump. Operators stop pump P

-7C. Header pressure is

stable at 64 psig. All alarms except EK

-1132 clear.

246 CR-PLP-2011-03902 created to document failure of P

-7C and initiates root cause investigation and identified as significant condition adverse to quality 1303 Operators exit ONP 6.1

1530 BS-1320 completed identifying only zebra mussel shells in strainer

August 10, 2011

0107 P-7C motor and pump uncoupled

215 P-7C motor removed

25 All P-7C shafts removed. P-7C pump failed due to coupling #6 failure.

August 10, 2011 Jackson Labs independently performs hardness tests on 4 couplings and identifies

that three couplings have at least one hardness value out

-of-specification

high. August 11, 2011

0135 P-7C shaft reassembly commence

d with couplings from Traveler 27797 (Heat Order 20170)

.

1059 P-7C pump reassembly complete

d. 1215 P-7C motor reinstallation completed

.

August 12, 2011

P-7C declared Operable after completion of Post Maintenance Testing

(WO 286627)

and return-to-service Surveillance Testing (WO 286774)

. August 16-17, 2011 All P-7C couplings sent to Lucius Pitkin Inc (LPI). Metallurgical plan is still in draft form.