IR 05000440/2004011
| ML041900080 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 07/02/2004 |
| From: | Reynolds S Division Reactor Projects III |
| To: | Kanda W FirstEnergy Nuclear Operating Co |
| References | |
| NPF-058 IR-04-011 | |
| Download: ML041900080 (21) | |
Text
July 2, 2004
SUBJECT:
PERRY NUCLEAR POWER PLANT NRC SPECIAL INSPECTION REPORT 05000440/2004011
Dear Mr. Kanda:
On June 4, 2004, the U.S. Nuclear Regulatory Commission (NRC) completed a special team inspection at your Perry Nuclear Power Plant. The enclosed report documents the inspection findings which were discussed with you and other members of your staff on June 16, 2004.
On May 21, 2004, the uppermost split ring coupling on the Division 1 emergency service water (ESW) pump failed. This event was similar to a failure that occurred on September 1, 2003.
Boroscope videography of the Division 1 ESW pump showed that the coupling had failed catastrophically. Laboratory analysis of the coupling concluded that the failure mechanism was intergranular stress-corrosion cracking (IGSCC), the same as that found in September 2003.
However, in the September 2003 failure, markings on the failed coupling components showed that it had been installed improperly such that the key did not fully engage the coupling sleeve.
In the May 2004 failure, the alignment between the two components was proper. This second failure was of increased interest because of its repetitive nature and the uncertainty which surrounded the failure. Based on the criteria specified in Management Directive 8.3 and Inspection Procedure 71153, a special inspection was initiated in accordance with Inspection Procedure 93812. This special inspection evaluated the facts, potential significance, and your resolution of this issue.
The root cause determination for the ESW coupling failure was not complete at the time of this special inspection. However, during this special inspection, enough information had been collected and evaluated to determine the technical issues surrounding the coupling failures.
Actions implemented by your staff during this special inspection to replace both Division 1 and 2 pumps with redesigned pump couplings restored the systems to operability. Further work was ongoing to fully understand the management and organizational issues that allowed the repeat failure to occur. This inspection report documents one finding that was determined to involve an apparent violation of NRC requirements; however, the significance has not been determined at the end of this inspection. Based on your replacement of Division 1 and 2 ESW pumps with couplings with significantly higher design margins, the inspector concluded a safety issue did not exist at the conclusion of the inspection. In accordance with 10 CFR 2.390 of the NRCs "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA by P. Hiland Acting for/
Steven A. Reynolds, Acting Director Division of Reactor Projects Docket No. 50-440 License No. NPF-58
Enclosure:
Inspection Report 05000440/2004011 w/Attachment: Supplemental Information Charter for Special Inspection
REGION III==
Docket No:
50-440 License No:
NPF-58 Report No:
05000440/2004011 Licensee:
FirstEnergy Nuclear Operating Company (FENOC)
Facility:
Perry Nuclear Power Plant, Unit 1 Location:
P.O. Box 97 A200 Perry, OH 44081 Dates:
May 24 through June 4, 2004 Inspector :
J. Ellegood, Resident Inspector Approved by:
M. Ring, Chief Branch 1 Division of Reactor Projects
Enclosure
SUMMARY OF FINDINGS
IR 05000440/2004011; 05/24/04 - 06/04/04; Perry Nuclear Power Plant. Special Inspection for repeat failure of the Division 1 emergency service water (ESW) pump on May 21, 2004.
This special inspection examined the facts and circumstances surrounding a repeat failure of the uppermost split ring coupling on the Division 1 ESW pump. The ESW pump uppermost split ring coupling shaft failed on September 1, 2003, and again on May 21, 2004. The inspection was conducted by the resident inspector from Perry in accordance with NRC Inspection Procedure 93812. This inspection identified one apparent violation whose safety significance has yet to be determined but may be greater than green. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
NRC-Identified
and Self-Revealed Finding
Cornerstone: Mitigating Systems
- To be determined: A self-revealed apparent violation of 10 CFR Part 50, Appendix B,
Criterion XVI, corrective action requirements, having a potential safety significance greater than very low occurred, on May 21, 2004, when the Division 1 emergency service water (ESW) pump failed when the uppermost split ring coupling broke in half.
The primary cause for this failure was related to the cross-cutting issue of problem identification and resolution in that the licensee neither understood nor corrected the design deficiencies associated with the coupling. After a loss of ESW occurred due to a coupling failure in September 2003, the licensee did not take adequate corrective actions to preclude repetition of a significant condition adverse to quality.
This finding is unresolved pending completion of a significance determination. This finding is more than minor because it directly affects the mitigating system cornerstone objective of system operability, availability, and reliability. Specifically, the finding is associated with loss of one division of ESW for 12 days. This finding was determined to have a potential safety significance greater than very low because of the loss of one division of ESW. The licensee has replaced the pump. (Section 4OA3.3).
REPORT DETAILS
Background and Overview While in Mode 1 at 100 percent power on May 21, 2004, the Division 1 ESW pump at Perry failed when the uppermost shaft coupling failed. On September 1, 2003, the same coupling on the same pump failed in a similar manner. When the coupling failed in 2003, the licensee requested and was granted a Notice of Enforcement Discretion (NOED) which allowed the licensee to repair the pump while remaining at power.
Following the failure on May 21, the licensee shut down the plant to replace the pump.
The failed coupling consists of a two-piece split ring that is held in place by a steel sleeve. A steel key is inserted into a keyway so that torque can be transmitted down the shaft to the pump. The keyway creates both a thin spot on the sleeve and a stress riser location on the sharp edged corner of the keyway. In both failures, a crack started in the keyway corner and propagated through the sleeve. In addition, a second crack developed 180 degrees from the initial crack due to the increased stress as the sleeve began to fail at the keyway. Eventually, the sleeve broke into two pieces, and the entire coupling assembly fell to the bottom of the pump.
In both cases, laboratory analysis determined that the couplings failed due to intergranular stress-corrosion cracking (IGSCC). This failure mechanism required three conditions:
- (1) a susceptible material,
- (2) tensile stress, and
- (3) a corrosive environment. The couplings were made of 416 stainless steel which was known to be susceptible to IGSCC. The lake water provided the corrosive environment. Since the keyed sleeve provides the means to transfer torque to the pump impeller, tensile stress was present in the sleeve. While the coupling design had allowed the pumps to operate over the life of the plant, in both failures workmanship issues increased the stresses within the sleeve. In the 2003 failure, the sleeve was misaligned on the shaft which resulted in only two-thirds of the key transmitting torque. This misalignment increased stresses by slightly more than 30 percent. In the 2004 failure, difficulties in assembling the couplings led to workers reducing the keyway corner radius while removing a high spot in the keyway, thereby increasing stress risers at the keyway corner.
Following the failure in 2004, the licensee rebuilt the Division 1 and 2 ESW pumps with 17-4 precipitation-hardened stainless steel shafts and couplings. In addition, the licensee increased the thickness of the sleeve to reduce the stresses. These changes significantly improved the design margin by both reducing the stresses in the coupling and by using a material significantly more resistant to IGSCC.
OTHER ACTIVITIES
4OA3 Event Follow Up
.1 Sequence of Events - Division 1 ESW Pump Failure
February 1985 Shaft couplings on Division 1 and 2 ESW pumps changed from threaded to keyed couplings April 10, 1989 Division 1 ESW pump rebuilt for first time September 26, 1990 Division 2 ESW pump rebuilt for first time July 24, 1997 Division 1 ESW pump rebuilt for the second time July 27, 1998 Division 2 ESW pump rebuilt for the second time April 24, 2003 Division 2 ESW pump rebuilt completely during refueling outage 9 September 1, 2003 Division 1 ESW pump upper coupling fails September 4, 2003 NOED granted to permit continued operation during pump repair September 5, 2003 Division 1 ESW pump declared operable May 21, 2004 1:48 a.m.
Division 1 ESW pump started 1:50 a.m.
Division 1 ESW pump upper coupling fails 3:26 a.m.
Division 1 ESW pump placed in secure status
~9:00 a.m.
Boroscope identifies failed upper coupling on Division 1 ESW pump 4:02 p.m.
Commenced power reduction to support shutdown in accordance with Technical Specifications (TSs) for shutdown May 22, 2004 8:42 a.m.
Separated from grid 2:49 p.m.
Entered Mode 2 9:39 p.m.
Entered Mode 3 May 23, 2004 6:19 a.m.
Entered Mode 4 May 24, 2004 3:00 p.m.
Division 2 ESW pump declared inoperable due to common mode failure potential May 25, 2004 11:30 a.m.
Division 1 ESW available following pump replacement May 29, 2004 5:09 a.m.
Use of residual heat removal (RHR) cross-connected with main condenser and feed designated as alternate decay heat removal method 5:13 a.m.
Division 1 ESW pump operable following completion of immediate investigation and post-maintenance testing 6:08 a.m.
Division 2 ESW pump no longer available due to start of replacement of the pumps shaft June 3, 2004 7:52 a.m.
Division 2 ESW pump available following shaft replacement 3:23 p.m.
Division 2 ESW pump declared operable The inspector reviewed the licensees event classification. At 4:11 p.m. on May 21, the licensee notified the NRC that the plant was commencing a shutdown as required by TSs. The inspector reviewed the licensees emergency action levels and concluded that the emergency action levels did not require any additional notifications.
b. Findings
No findings of significance were identified.
.2 Review of Licensee Activities to Replace the Pump and Determine Root Cause
a. Inspection Scope
The inspector observed licensee actions to disassemble and rebuild the Division 1 ESW pump. The inspector focused his observations on coupling disassembly and reassembly. In addition, the inspector monitored the licensees actions with regards to compliance with TSs and performance of root cause analysis. The inspector reviewed documentation generated as part of licensees investigation of the root cause.
b. Findings and Observations
The failure of the Division 1 ESW pump on May 21 bore striking similarities to the failure that occurred in September of 2003. In both cases, the uppermost shaft coupling failed with no warning to plant operators. In both cases IGSCC was the failure mechanism. In addition, in both cases, the next coupling in the shaft had an incipient crack. However, in the first failure, misalignment of the coupling sleeve such that the key imparted additional stress onto the sleeve was clearly evident. In the May 2004 failure, the sleeve was properly positioned on the shaft and a cause for the failure was not readily apparent.
During the cooldown, the licensee recognized that with Division 1 ESW unavailable, they could not comply with the requirements of TS 3.4.10. Specifically, in Mode 4, the licensee was required to identify an alternate means of decay heat removal when one ESW system was inoperable. With high decay heat, no alternate means existed. Later, the licensee declared Division 2 ESW inoperable and designated it as its own alternate means of decay heat removal. Compliance with TS 3.4.10 is considered to be an unresolved item pending the staffs determination if the licensees failure to have an alternate means of decay identified represents a performance deficiency and whether an inoperable but technically available system can be used as its own alternate system.
The licensees initial evaluation concluded that inadequate design resulted in a condition where normally used assembly techniques might cause premature failure of the pump.
Based on this recognition, the licensee concluded that the Division 2 ESW pump could not be considered operable and it was declared inoperable at 3:00 p.m. on May 24. At that time, the licensee considered Division 1 ESW inoperable and unavailable; Division 2 ESW inoperable but available. Division 2 ESW continued to perform acceptably and provided shutdown cooling. After completion of maintenance and testing, the licensee declared Division 1 ESW operable on May 29 at 5:13 a.m. In parallel, the licensee developed an alternate decay heat removal method involving RHR feed and bleed to the main condenser. The licensee proceduralized the method in Off-Normal Instruction (ONI) E12-2, Loss of Decay Heat Removal, Revision 11. Following operability of Division 1 ESW pump and establishment of the alternate decay heat removal, the licensee began work on the Division 2 ESW pump.
As part of the investigation, the licensee performed laboratory analysis on the couplings for both Division 1 and 2 ESW pumps. The analysis for the failed coupling determined that IGSCC was the failure mechanism for the coupling. Dimensional data taken on the failed coupling determined that the keyway radius was
.01 inches.
The number two coupling, which had an incipient crack that was 75 percent through wall, had a keyway radius of
.03 to.05 inches.
During reassembly of the Division 2 ESW pump, several difficulties occurred that hampered the restoration of the pump. During installation of the shaft bearings onto the shaft, two of the bearings galled, resulting in the licensee machining the sleeves off of the shafts. Since the galling created scratches on the shaft, the licensee repaired the shaft and performed nondestructive examination (NDE) to verify there were no significant flaws left. Later, the licensee galled a similar bearing on the head shaft.
Although engineering established similar corrective actions, workers neglected to perform NDE on the head shaft, and engineering subsequently determined it could be used without NDE. It was only after these events occurred that the licensee recognized that the shafts were slightly larger than specified. Subsequently, after assembly of the head shaft, the upper bearing slipped several inches. The licensee attributed this slip to inadequate engagement of the setscrews into the shaft. In response, the Site Vice President put a stop-work order on Division 2 ESW pump reassembly until the technical issues were understood. The Site Vice President lifted the stop-work order after workers were briefed on site expectations for identification of problems and a coordinated plan was developed to resolve the identified issues.
In order to ensure a failure of the coupling did not recur, the licensee implemented a design change that had been developed following the September failure. The design change included use of 17-4 precipitation-hardened stainless steel instead of 416 stainless steel, and an increase in the coupling sleeves wall thickness by one quarter of an inch. Finite element analysis performed for the modified coupling concluded that a dual benefit existed from these changes. First, the material provided a greater resistance to IGSCC due to material properties. Second, the increased thickness reduced the stresses in the coupling. The inspector concluded that the licensee bounded the technical causes and the redesigned coupling provided significantly more margin.
Modeling of the coupling design using finite element analysis following the coupling failure in 2003 resulted in calculated stresses in excess of yield stress for an off-centered coupling and near yield for a properly aligned coupling. While the model was conservative in that it assumed a 90 degree keyway corner, the licensee did not pursue the results aggressively to refine the calculation such that margins could be thoroughly understood and evaluated. This represents a missed opportunity to prevent the shaft failure in May of 2004.
.3 Evaluation of Root Cause and Extent of Condition
a. Inspection Scope
The inspector reviewed the licensees approved immediate investigations, referenced documents, and related condition reports to evaluate the licensees effectiveness at determining the root cause and extent of condition.
b. Findings and observations
The licensee adopted a phased approach to the root cause such that declaration of pump operability would not occur until both testing and an immediate investigation was complete on the subject pump. This approach would allow for declaration of pump operability with management and organizational issues surrounding the pump failure still in question. In order to compensate, the licensee provided additional management oversight on the work activities to ensure that workers complied with procedure requirements. While the use of additional supervision may help in certain instances, it cannot resolve all latent organizational and management issues. For example, during the assembly of the Division 2 ESW pump, the licensee discovered that the shafts provided were of too large a diameter. The additional oversight was unable to detect this condition prior to ruining two new bearing sleeves. In addition, even after the problems with installation of the sleeves were known, management did not prevent subsequent installation attempts which resulted in stuck bearing sleeves and additional rework. Therefore, the selected method to compensate for the as yet to be determined organization and management issues was unable to detect problems resulting from receipt of out-of-specification components. Further, once the problems were evident, the additional oversight did not prevent then from recurring.
For this failure of the ESW pump, the licensee implemented more thorough and expansive efforts to identify the underlying technical issues that led to failure of the coupling. As a result, the root cause team surfaced problems, both with the assembly and materials used for the assembly, of the Division 1 ESW pump in September 2003.
In particular, the team discovered that the mechanics assembling the pump used a broach to remove a high spot in the coupling keyway on three of four couplings. In addition, the mechanics used files to remove high spots. Finally, additional finite element analysis demonstrated how close the coupling was to failure and explained how skill of the craft techniques used for assembly could provide the additional stress needed to initiate IGSCC or establish stress risers.
In order to correct the problems noted, the licensee implemented a design change to the coupling that replaced the 416 stainless steel with 17-4 precipitation-hardened stainless steel and increased the coupling size. The material change results in a coupling that is much tougher and better able to resist IGSCC. The larger diameter decreases the stress in the coupling increasing its resistance to IGSCC.
For extent of condition, the licensee validated the extent of condition performed after the September failure. The validation concluded that similar couplings existed on the Division 2 and 3 ESW pumps, low pressure core spray, and RHR. The licensee replaced the Division 2 ESW pump couplings and concluded that the stresses in the other pumps were well below allowable or the pumps environment was not corrosive.
Introduction:
A self-revealed apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI, corrective action requirements was identified when the Division 1 ESW pump failed during routine pump operation. The licensee rebuilt the pump in September 2003 following a similar failure. During the reassembly in September 2003, the parts used had non-conformances and correction of these non-conformances introduced stress risers to the affected keyways. Combined with a marginal design for the coupling, the coupling failed on May 21, 2004. The inspector assessed this finding in accordance with IMC 0609 and identified the issue as having potential safety significance greater than minor. This is an apparent violation pending completion of the SDP.
Description:
On May 21, 2004, the licensee started the Division 1 ESW pump in accordance with standard operating instructions. At 1:52 a.m., the control room received several alarms associated with the Division 1 ESW pump. Subsequent investigation revealed that the pump shaft had failed. The licensee investigated the shaft failure and concluded that a coupling had failed due to low design margin and the addition of stress risers during reassembly of the pump in September of 2003. The pump shaft consisted of four sections with four split ring couplings to connect the sections and transfer power from the motor to the pump. Each of the couplings consisted of a two-piece split ring, a coupling sleeve, two keys and two setscrews. The licensee restored the pump to operability at 5:13 a.m. on May 29.
The licensees root cause team developed a series of failure scenarios and systematically reviewed each scenario to determine if available evidence supported the scenario. In addition, the team reviewed documentation, performed calculations, conducted interviews, and obtained laboratory data. During these investigations, the team discovered that during reassembly of the pumps couplings, maintenance mechanics had removed
.020 " high spots in the keyways of three of four couplings.
The tool used to perform this activity, called a broach, has large steel teeth to remove metal from the work piece. The broach used has square sides such that the keyway produced by the broach would have 90 degree corners. The sleeve design specified that the keyway would have a
.030 -.040" radius.
Without a radius, stresses concentrated at the sharp angle and allowed for a crack to develop. Laboratory analysis of the failed coupling identified that the failed coupling had a keyway radius of
.010 " which resulted in
a significant increase in stresses at the keyway corner. In addition to the broach, mechanics used files to fit pieces of the coupling together. During this work, the mechanics were not aware of the importance of the keyway radius.
Modeling of the coupling design using finite element analysis following the coupling failure in 2003 resulted in calculated stresses in excess of yield stress for an off-centered coupling and near yield for a properly aligned coupling. While the model was conservative in that it assumed a 90 degree keyway corner, the licensee did not pursue the results aggressively to refine the calculation such that margins could be thoroughly understood and evaluated. In their root cause report, the licensee concluded that the design was adequate when the coupling was properly assembled, i.e. aligned. The licensee relied on hand calculations to determine the acceptability of coupling designs and did not fully understand how easily the design could be challenged during installation. This represents a missed opportunity to prevent the shaft failure in May of 2004. Even though the licensee developed a new coupling design following the failure in September, the lack of understanding of the design contributed to the installation of the design not receiving a high enough priority to be implemented prior to the failure in May. Finite element analysis performed on the new coupling design demonstrated its higher margin to failure. While performing portions of the analysis, the licensee determined that maximum torque values provided by the pump vendor were incorrect.
The vendor-provided values were for maximum steady state torque when the maximum torque occurs during acceleration of the pump to normal operating speeds. Inclusion of the increased torque in the calculations resulted in reduced calculated margin.
Analysis:
The inspector evaluated this finding under the SDP. The inspector concluded that this finding directly effected the mitigating system cornerstone objective of safety system availability. The inspector evaluated the finding under Phase 1 of the SDP and determined a Phase 2 evaluation was needed. The inspector based this conclusion on the loss of the Division 1 ESW safety function. With the shaft broken, the Division 1 ESW system could not perform its safety function. In addition, the loss of ESW resulted in inoperability of numerous supported systems including the Division 1 emergency diesel generator, RHR A, and low pressure core spray systems. The Division 1 ESW pump was considered to be unavailable for a duration of 3 to 30 days. Based on these results, the inspector concluded a Phase 3 analysis was required.
Enforcement:
Appendix B of 10 CFR Part 50, Criterion XVI requires that measures be established to assure that conditions adverse to quality are promptly identified and corrected. It also requires that for significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action is taken to preclude repetition. Contrary to this requirement, the licensee did not take adequate corrective action to prevent the repetitive failure of the Division 1 ESW pump. Specifically, following the pumps failure in September 2003, the licensee determined that one of the causes of the failure was inadequate coupling design and developed an enhanced design that, if implemented, would have prevented recurrence.
However, as of May 21, 2004, the licensee had not yet installed the enhanced design and the pump failed again due to the inadequate coupling design. Pending determination of the safety significance of this finding it is considered an apparent violation (AV 05000440/2004011-02). The licensee has entered this issue into their corrective action program as CR 04-02598.
4OA6 Meetings
Exit Meeting The inspector presented the inspection results to Mr. W. Kanda and other members of licensee management at the conclusion of the inspection on June 16, 2004. The inspector asked the licensee whether any materials examined during the inspection should be considered proprietary.
KEY POINTS OF CONTACT Licensee W. Kanda, Vice President-Nuclear F. von Ahn, General Manager, Nuclear Power Plant Department N. Bonner, Director, FENOC Support K. Cimorelli, Nuclear Maintenance V. Higaki, Manager, Regulatory Affairs M. Humphrey, Root Cause Coordinator, Work Control Section J. Lausberg, Supervisor, Compliance L. McGuire, Performance Unit Supervisor D. Miller, Engineer, Compliance Nuclear Regulatory Commission S. Reynolds, Deputy Director, Division of Reactor Projects M. Ring, Chief, Reactor Projects Branch 1 LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED Discussed 05000440/2004011-01 URI Operation in Mode 4 with One Method of Decay Heat Removal (Section 4AO3.2)05000440/2004011-02 AV Repeat Failure of ESW Pump Upper Shaft Coupling Sleeve on May 21, 2004 (Section 4AO3.3)
LIST OF
DOCUMENTS REVIEWED
The following is a list of documents reviewed during the inspection. Inclusion on this list does
not imply that the NRC inspector reviewed the documents in their entirety, but rather that
selected sections or portion of the documents were evaluated as part of the overall inspection
effort. Inclusion of a document on this list does not imply NRC acceptance of the document or
any part of it, unless this is stated in the body of the inspection report.
CR 03-05065; ESW Pump A Failed
CR 03-05091; Vendor Manufactured Part Dimension Out of Tolerance; dated
September 3, 2003
CR 03-05097; Engineering Disposition for Galled Sleeve Coupling; dated September 4, 2003
CR 04-02862; ESW Pump B Stuffing Box Sleeve Displaced Down on Head Shaft Scoring Shaft
CR 04-02656; Emergency Service Water B Pump; dated May 23, 2004
CR 04-02715; ESW A Pump Replacement Order Revised without Field Copy Being Updated;
dated May 25, 2004
CR 04-02818; Deficiencies Noted in CR 03-05065 for September 2003 ESW A Failure;
May 28, 2004
CR 04-02820; Operability of ESW B not Requested in a Timely Manner; May 30, 2004
CR 04-02826; Emergency Service Water B Pump Lineshaft; dated May 30, 2004
CR 04-02829; RFA - Emergency Service Water B Pump Lineshaft; dated May 30, 2004
CR 04-02841; Emergency Service Water B Pump Head Shaft; dated May 31, 2004
CR 04-02849; ESW B Pump Shafts are Oversized (Outside Diameter is Out of Tolerance);
dated May 31, 2004
CR 04-02851; Vendor ESW B Pump Shaft Parts do Not Meet Dimensional Specifications;
dated May 31, 2004
CR 04-02853; Evaluate Vendors Performance in Supplying ESW Parts; dated May 31, 2004
CR 04-02854; Final ESW A Restart Readiness Meeting Conducted 5/31/2004; dated
May 31, 2004
CR 04-02855; GMI-0039 Not Followed as Written Installing New ESW B Pump Head Shaft
Sleeve; dated May 31, 2004
CR 04-02856; Damage on Head Shaft, Near Coupling End, was not PT Inspected; dated
May 31, 2004
CR 04-02857; NRC Event Notification made for ESW Pump B Inoperability; dated
May 31, 2004
CR 04-02858; Condition Report Disposition was not Implemented as Written; dated
May 31, 2004
CR 04-02861; PCR-GMI-0039 Step 5.2.4.4 Does not Work as Written; dated May 31, 2004
CR 04-02864; Assessment of ESW B Pump Health-Following Event Documentation in
CR 04-02864
CR 04-02865; RFA 1P45C0001B Pump to Motor Coupling Key Material; dated June 1, 2004
CR 04-02867; RFA 1P45-C0001B Pump to Motor coupling Key; dated June 1, 2004
CR 04-02873; ESW Pump 1P45C0001A/B Pump Hub Key Design Adequacy; dated
June 1, 2004
CR 04-02887; Vice President Directs Work to be Stopped on ESW Maintenance Activity; dated
June 1, 2004
CR 04-02876; Cumulative Significance Review of All Conditions Reports for the ESW A&B
Pumps; dated June 1, 2004
CR 04-02889; Incorrect Set Screws Used to Hold Sleeve to Head Shaft in ESW A; dated
June 1, 2004
CR 04-02891; ESW Pump B Stuffing Box Sleeve Displaced Down on Head Shaft Scoring the
Shaft; dated May 31, 2004
CR 04-02875; Evaluate Stuffing Box Sleeve in the Emergency Service Water A Pump; dated
June 1, 2004
CR 04-02865; RFA 1P45C0001B Pump to Motor Coupling Key Material; dated June 1, 2004
CR 04-02904; Stop Work Directive Requirements not Met Prior to Release; dated June 2, 2004
Calc EA-254; Emergency Service Water System Pump Coupling Failure Analysis; Rev. 0
Calc. SQ-0146; Seismic Qualification for 1P45C001A (ESW Pump); Rev 1
Work Tracking Number 13551; Metallurgical Laboratory Report on ESW A Pump Shaft
Couplings; dated May 26, 2004
GMI-0039; Disassembly of the Emergency Service Water Pumps; Rev. 8
WO 200003985; Replanned to Rebuild Pump; Rev. 01100209
WO 01100510; cc-Replace Pump and Motor-ECP 03-0083; Rev. 01100510
Selected log entries; May 21 2004 thru June 3, 2004
Selected log entries; September 1-September 5, 2003
ECP 04-0151; Emergency Service Water (ESW) B Pump Coupling Modification; dated
May 23, 2004
File No. 165; Gould Pump Manual; Rev. 11
LIST OF ACRONYMS
apparent violation
CFR
Code of Federal Regulations
emergency service water
FirstEnergy Nuclear Operating Company
Intergranular stress-corrosion cracking
IMC
Inspection Manual Chapter
Notice of Enforcement Discretion
NRC
Nuclear Regulatory Commission
ONI
Off-Normal Instruction
Significance Determination Process
TS
Technical Specification
unresolved item
May 24, 2004
MEMORANDUM TO:
John Ellegood, Resident Inspector, Perry
Division of Reactor Projects
FROM:
Mark Ring, Chief, Branch 1
Division of Reactor Projects
/RA/
SUBJECT:
SPECIAL INSPECTION FOR FAILURE OF A EMERGENCY
SERVICE WATER (ESW) PUMP ON MAY 21, 2004
On May 21, 2004, at about 0150 (EDT), the Perry Nuclear Power Plant was operating at full
power when the A ESW pump was declared inoperable due to the failure of the pump to
produce appropriate discharge pressure and flow after about 2 minutes of running. Initial
indications were that the pump had experienced a shaft failure. The remaining 2 ESW pumps
were considered operable. Failure of the A ESW pump resulted in all Division 1 emergency
core cooling systems (ECCS) being considered inoperable. The licensee entered the Limiting
Condition for Operation of Technical Specification 3.7.1 for the inoperable ESW pump.
Subsequently, the licensee proceeded to shut down the reactor and put the plant in Mode 4
(Cold Shutdown).
The failure of the A ESW pump was determined to be due to a failure of a pump shaft
coupling. The pump shaft consisted of four sections with five couplings to connect the sections
and transfer power from the motor to the pump. The failed coupling was the first coupling from
the motor connecting the motor and the shaft. This was the same coupling which failed on
September 1, 2003, and resulted in the need for a Notice of Enforcement Discretion and a
White finding. The May 21, 2004, failure of the A ESW pump represents a repetitive failure or
event involving safety-related equipment as defined in Management Directive 8.3.
Based on the criteria specified in Management Directive 8.3 and Inspection Procedure 71153,
a Special Inspection was initiated in accordance with Inspection Procedure 93812 and Regional
Procedure RP-1219. The Special Inspection will be performed by yourself (inspection lead).
You may draw upon the resources of the Perry Senior Resident Inspector, Ray Powell, and the
other members of the Perry IP 95002 Supplemental Inspection Team (Steve Campbell, Carey
Brown, and Ron Langstaff) to assist you, as needed. The Special Inspection will evaluate the
facts, circumstances, and licensee actions surrounding the A ESW pump failure. A Charter
was developed and is attached. The inspection began with your monitoring of the licensees
immediate actions on May 21, 2004, and will officially entrance with the licensee today, May 24,
2004.
Attachment:
As stated
cc w/att:
- R. Powell, Perry
- S. Campbell, Fermi
- C. Brown, Clinton
SPECIAL INSPECTION (SI) CHARTER
This Special Inspection is chartered to assess the circumstances surrounding the failure of the
A Emergency Service Water (ESW) pump on May 21, 2004. This failure represents repeat of
the September 1, 2003 A ESW pump failure. The Special Inspection should:
1.
Establish a sequence of events including changes made to the A ESW pump and
pump operation history since the September 1, 2003, failure, as well as event
notification and classification.
2.
Monitor the licensees determination of the root cause of the failure of the A ESW
pump. Include in your evaluation such things as: removal and disassembly of the pump
components, any replacements of parts and reassembly of the pump, licensees current
operability justification and potential vulnerability to a repeat occurrence, operability prior
to the May 21, 2004, failure, and material composition of components as appropriate.
3.
Evaluate the licensees efforts to determine the extent of condition for root causes
identified in item 2 above.
Charter Approval
/RA/ - Mark A. Ring
Chief, Reactor Project Branch 1
/RA/ - Patrick L. Hiland for Steven A. Reynolds
Director, Division of Reactor Projects