IR 05000440/2004008

IR 05000440-04-008; 05/17/04-05/21/04 and 6/7/04-6/10/04; Perry Nuclear Power Plant. Inspection Procedure (IP) 95002, Inspection for One Degraded Cornerstone or Any Three White Inputs in a Strategic Performance Area, Root Cause and Extent

ML042250254
Person / Time
Site:	Perry
Issue date:	08/05/2004
From:	Reynolds S Division Reactor Projects III
To:	Myers L FirstEnergy Nuclear Operating Co
References
IR-04-008
Download: ML042250254 (38)

Text

ust 5, 2004

SUBJECT:

PERRY NUCLEAR POWER PLANT NRC SUPPLEMENTAL INSPECTION REPORT 05000440/2004008

Dear Mr. Myers:

The U.S. Nuclear Regulatory Commission (NRC) conducted this supplemental inspection in accordance with Inspection Procedure (IP) 95002, Inspection For One Degraded Cornerstone or Any Three White Inputs In A Strategic Performance Area, at your Perry Nuclear Power Plant. The enclosed report documents the inspection findings which were discussed on June 10, 2004, with you and other members of licensee management; and on June 21, 2004, at a public exit meeting with Mr. Bill Kanda and other members of your staff.

The NRC performed this supplemental inspection as required by the NRC Action Matrix based on our assessment of plant performance. As stated in our letter dated March 4, 2004, plant performance at the Perry Nuclear Power Plant was within the Degraded Cornerstone column of the NRC Action Matrix based on two White findings in the Mitigating Systems Cornerstone. A third White finding was subsequently identified and documented in our letter dated March 12, 2004. The third White finding was included in the scope of this inspection.

The first finding involved the failure of the high pressure core spray (HPCS) pump to start during routine surveillance testing. An apparent violation (AV) of Technical Specification (TS) 5.4 for inadequate breaker maintenance procedure was identified in Inspection Report (IR) 05000440/2003008. This performance issue was previously characterized as having low to moderate risk significance (White) in the NRCs final significance determination letter dated March 4, 2003. A supplemental inspection was performed in accordance with IP 95001 for the White finding and significant deficiencies were identified with regard to your extent of condition evaluation. Inspection Procedure 95001 was re-performed and the results of that inspection were documented in IR 05000440/2003012 which determined the extent of condition reviews were adequate.

The second finding involved air binding of the residual heat removal 'A' and low pressure core spray waterleg pump in August 2003. A special inspection was performed for this issue and an Unresolved Item (URI) was opened because the root cause, extent of condition, and past operability evaluations were not completed. The results of the inspection were documented in IR 05000440/2003009. The URI was closed to an AV of TS 5.4 for an inadequate venting procedure in IR 05000440/2003010. This performance issue was previously characterized as having low to moderate risk significance (White) in the NRCs final significance determination letter dated March 12, 2004.

The third finding involved the failure of emergency service water (ESW) pump 'A,' caused by an inadequate maintenance procedure for assembling the pump coupling which contributed to the failure of the pump on September 1, 2003. An AV of TS 5.4 was documented in IR 050000440/2003006 and a final significance determination characterizing this as a White finding was issued on January 28, 2004.

This supplemental inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel. The purpose of this inspection was to: 1) provide assurance that the root and contributing causes for the three White findings and for the overall performance issues which resulted in the Degraded Cornerstone are understood; 2) independently assess the extent of condition and generic implications; and 3) provide assurance that the corrective actions are sufficient to prevent recurrence.

The NRC has concluded that the corrective actions to prevent recurrence of a significant condition adverse to quality were inadequate. Specifically, the same ESW pump coupling that failed on September 1, 2003, failed again on May 21, 2004. This is considered a significant issue with your evaluation and will result in the ESW pump White finding, which contributed to the Degraded Cornerstone, remaining open. The NRC considered this significant in that it demonstrated your staffs inability to develop adequate corrective actions to preclude a repeat occurrence that ultimately affected the availability and reliability of equipment in the Mitigating Systems Cornerstone. In addition, the identification of three findings during our independent root cause review confirmed the need for a thorough assessment of the common causes for two of the three White findings. The 2002 HPCS pump failure to start was previously reviewed twice in accordance with IP 95001 and was closed after 5 quarters in December 2003.

Therefore, two of the White findings, the ESW pump and the waterleg pump issues, will remain open pending the following actions: 1) development of corrective actions to address the common cause regarding procedure adequacy and usage, including training; 2) corrective actions to address the root cause of the second failure of ESW pump 'A;' and 3) NRC inspection of the results of your reviews.

During this inspection, three findings of very low safety significance (Green) were identified which involved violations of NRC requirements. However, because of their very low safety significance and because they have been entered into your corrective action program, the NRC is treating these issues as Non-Cited Violations, in accordance with Section VI.A.1 of the NRC Enforcement Policy. If you contest the Non-Cited Violations, you should provide a response with the basis for your denial, within 30 days of the date of this inspection report, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region III, 2443 Warrenville Road, Suite 210, Lisle, IL 60532-4352; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the Resident Inspector Office at the Perry Nuclear Power Plant.

In accordance with 10 CFR 2.390 of the NRCs "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Steven A Reynolds, Acting Director Division of Reactor Projects Docket No. 50-440 License No. NPF-58

Enclosure:

Inspection Report 05000440/2004008 w/Attachment: Supplemental Information

REGION III==

Docket No: 50-440 License No: NPF-58 Report No: 05000440/2004008 Licensee: FirstEnergy Nuclear Operating Company (FENOC)

Facility: Perry Nuclear Power Plant, Unit 1 Location: P.O. Box 97 A200 Perry, OH 44081 Dates: May 17 through 21, 2004 June 7 through 10, 2004 Inspector : S. Campbell, Senior Resident Inspector, Fermi R. Powell, Senior Resident Inspector, Perry J. Ellegood, Resident Inspector, Perry C. Brown, Resident Inspector, Clinton R. Langstaff, Senior Reactor Inspector Approved by: M. Ring, Chief Branch 1 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000440/2004008; 05/17/04 - 05/21/04 and 6/7/04 - 6/10/04; Perry Nuclear Power Plant.

Inspection Procedure (IP) 95002, Inspection For One Degraded Cornerstone or Any Three White Inputs In A Strategic Performance Area, Root Cause and Extent of Condition Evaluation.

This report documents a supplemental inspection performed by regional based and resident inspectors. The inspectors identified three Green findings with three Non-Cited Violations (NCVs). The significance of most findings is indicated by their color (Green, White, Yellow,

Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG 1649, Reactor Oversight Process,

Revision 3, dated July 2000.

Cornerstone: Mitigating Systems

The NRC performed this supplemental inspection to assess the licensees evaluation of three White findings in the Mitigating Systems Cornerstone. This inspection was conducted in accordance with IP 95002, Inspection For One Degraded Cornerstone Or Any Three White Inputs In A Strategic Performance Area, and evaluated the licensees actions to address these White findings. The first White finding involved the failure of the high pressure core spray (HPCS) pump breaker to close during testing. The second White finding concerned air binding of a waterleg pump used on the low pressure core spray system (LPCS)/residual heat removal (RHR) 'A' system. The third White finding involved the failure of one coupling sleeve that joined the emergency service water (ESW) pump 'A' shaft segments together. Because of the common theme of potentially inadequate training and that procedure usage issues may have existed among the three White findings, the inspectors also conducted a limited review of one Green finding associated with the Division 1 emergency diesel generator (EDG) failing a surveillance test on August 21, 2003.

Generally, root cause evaluations were developed for each of the three White findings and corrective actions were assigned. A common cause report was also developed to identify the organizational and programmatic weaknesses and deficiencies that contributed to these events.

Likewise, corrective actions were developed to address these problems. However, the corrective actions to address the September 1, 2003, ESW pump 'A' failure were ineffective in preventing a significant condition adverse to quality in that the same pump coupling sleeve failed again on May 21, 2004. The circumstances surrounding this event were documented in Inspection Report (IR) 05000440/20040011. The 2002 HPCS pump failure to start was previously reviewed twice in accordance with IP 95001 and was closed after 5 quarters in December 2003. Consequently, two of the White findings, the ESW pump and the LPCS/RHR

'A' waterleg pump issues, will remain open. Three additional findings of very low safety significance were identified during this supplemental inspection and are summarized below.

A. Inspector-Identified and Self-Revealed Findings

Green.

A finding of very low significance was identified regarding the licensees failure to establish quality control requirements described in American Nuclear Standards Institute (ANSI) N45.2.8 - 1975 for reassembling the ESW pump 'A' coupling in 1997.

The primary cause of this finding was a general lack of knowledge of the quality control requirements.

This issue was more than minor because, if left uncorrected, it could lead to a more significant event. This finding was of very low safety significance because omitting the need for such inspections was a barrier to preventing the failure of the ESW pump coupling and not a direct cause of the failure. This finding was determined to be an NCV of 10 CFR 50, Appendix B, Criterion X. To address this issue, the licensee entered it into the corrective action program because the failure was programmatic in nature and not in need of an immediate corrective action. (Section 02.02b.2)

Green.

A finding of very low significance was identified in the root cause evaluation for CR 03-04764, Post-Loss of Offsite Power (LOOP) LPCS/RHR 'A' Waterleg Pump Air Binding, regarding the licensees failure to identify several missed opportunities that included the venting procedure biennial reviews between 1985 and 1995, a 1996 design review of the RHR system, and venting issues that occurred during the 2003 refueling outage. The primary cause of this finding was an inability to conduct a thorough root cause evaluation.

The issue was more than minor because, if left uncorrected, it could be a precursor to a significant event. This finding was of very low safety significance because the failing to identify these missed opportunities would not have directly prevented air binding of the LPCS/RHR waterleg pump. This finding was determined to be an NCV of 10 CFR 50,

Appendix B, Criterion XVI. To address this issue, the licensee entered it into the corrective action program because the failure was programmatic in nature and not in need of an immediate corrective action. (Section 02.02c.1)

Green.

A finding of very low significance was identified regarding the licensees failure to recognize whether training was effective for the following root cause evaluations addressed in: 1) CR 03-04912 for operators not properly restoring the Division 1 EDG to standby following the loss of offsite power event that occurred on August 14, 2003; 2) CR 02-03972 for correcting maintenance crafts inability to adjust breaker linkage rods for the HPCS breaker; and 3) CR 03-05065 when the ESW pump 'A' coupling design changed from a screwed to a keyed configuration in 1985. The primary cause of this finding was the failure to recognize that effective training could have prevented these events, since these events typically involved skill-of-the-craft activities.

This issue was more than minor because if left uncorrected, it could lead to a more significant event. This finding was of very low significance because failure to evaluate training effectiveness was not a direct cause to these three events. This finding was determined to be an NCV of 10 CFR 50, Appendix B, Criterion XVI. To address this issue, the licensee entered it into the corrective action program because the failure was programmatic in nature and not in need of an immediate corrective action.

(Section 02.02d.3)

Licensee-Identified Violations

None.

REPORT DETAILS

INSPECTION SCOPE

This inspection was conducted in accordance with IP 95002, Inspection For One Degraded Cornerstone Or Any Three White Inputs In A Strategic Performance Area, to assess the licensees evaluation of three White findings in the Mitigating Systems Cornerstone. The inspection objectives were to provide assurance that the root and contributing causes for the individual White findings and for the collective performance issues which resulted in the degraded cornerstone were understood, to independently assess the extent of condition and extent of cause for the individual White findings and collective performance issues, and to provide assurance that the corrective actions were sufficient to address the root and contributing causes for the White findings and to prevent their recurrence.

Perry Nuclear Power Plant was determined to be in the Degraded Cornerstone column of the NRCs Action Matrix during the annual end-of-cycle assessment for 2003 as a result of two White findings in the Mitigating Systems Cornerstone. A third finding in the Mitigating Systems Cornerstone was subsequently determined to be White and added to the scope of this IP 95002 inspection.

The first White finding was initially identified in the fourth quarter of 2002 and was associated with the failure of the HPCS system pump to start on October 23, 2002. An attachment to a breaker maintenance procedure did not contain the information necessary to ensure proper adjustment of the cell switch. The cell switch provides a permissive for the breaker to close. An apparent violation (AV) of Technical Specification (TS) 5.4 for inadequate breaker maintenance procedure was identified in IR 05000440/2003008. This performance issue was previously characterized as a White finding in the NRCs final significance determination letter dated March 4, 2003.

A supplemental inspection was performed in accordance with IP 95001, Inspection For One Or Two White Inputs In A Strategic Performance Area, for the White finding and significant deficiencies were identified with regard to the extent of condition evaluation as documented in IR 05000440/2003007. Following additional licensee corrective actions, Inspection Procedure 95001 was re-performed and the results of that inspection were documented in IR 05000440/2003012 which determined the extent of condition reviews were adequate. The White finding was removed from the individual plant performance summary for the most significant inspection findings in December 2003.

The second White finding, involving air binding of the LPCS/RHR 'A' waterleg pump on August 14, 2003, was initially identified in the fourth quarter of 2003. Operation of the pump caused entrained air to come out of the water and collect in feedwater leakage control system (FWLCS) piping connected to the discharge of the waterleg pump that had not been vented for years. A special inspection was performed for this issue and an Unresolved Item (URI) was opened because the root cause, extent of condition, and past operability evaluations were not completed. The results of the inspection were documented in IR 05000440/2003009. The URI was closed to an AV of TS 5.4 for an inadequate venting procedure in IR 05000440/2003010. This performance issue was previously characterized as a White finding in the NRCs final significance determination letter dated March 12, 2004.

The third finding, regarding the failure of the Division 1 ESW pump 'A' during a routine run on September 1, 2003, was initially identified in the third quarter of 2003. Unclear guidance in maintenance procedures, inadequate procedure content, and couplings made from stress-corrosion cracking (SCC) susceptible material with high stress concentrations were root causes for this finding. The inadequate procedure caused a mechanic to improperly assemble the coupling sleeves that joined ESW pump 'A' shaft segments together in 1997. Operation of the pump over several years created stress concentrations in the SCC susceptible material that ultimately caused the failure of the upper coupling. An AV of TS 5.4 was documented in IR 050000440/2003006 for failing to establish adequate written maintenance procedures and a final significance determination characterizing this as a White finding was issued on January 28, 2004.

Root cause evaluations, and inspection results documented in the reports, indicated potential issues with inadequate procedures and inadequate procedure usage. Further, the inspectors noted that training effectiveness, a common element in the HPCS and ESW pump 'A' White findings, was not addressed in the root cause evaluations. As a result, the inspectors reviewed, to a limited extent, a Green finding and an NCV of TS 5.4 documented in IR 05000440/2003009, that potentially had an element of inadequate training. This involved the Division 1 EDG exceeding the high voltage criteria during a monthly surveillance test on August 21, 2003. This EDG had tripped off after being paralleled to offsite power on August 14, 2004.

The scope of this supplemental inspection included the root cause evaluation, extent of condition, and corrective actions for all three White findings and the Division 1 EDG Green finding. The licensees evaluation of the first White finding was documented in CR 02-03972, Failure of the HPCS Pump to Start on Demand Resulting in Unavailability of the Division 3 Emergency Core Cooling System. An evaluation for the second White finding was documented in CR 03-04764, Post-LOOP LPCS/RHR 'A' Waterleg Pump Air Binding. The root cause evaluation for the third White finding was documented in CR 03-05065, Emergency Service Water Pump 'A' Upper Shaft Coupling Sleeve Failure. The licensees evaluation of the Green finding was documented in CR 03-04912, Division 1 Diesel Generator Failed SVI-R43-T1317 Run.

The common cause evaluation of the one Green and three White findings were documented in CR 03-05995, Common Cause Analysis Report. In addition to the detailed review of the licensees evaluation, the inspectors performed an independent review of other significant root cause evaluations unrelated to the three White findings described above. Additionally, a sampling of the corrective actions from the common cause analysis was performed.

EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.

Determine that the evaluation identifies who (i.e., licensee, self-revealing, or NRC), and under what conditions the issue was identified.

All three of the White findings were identified through self-revealing events. The Green finding for the Division 1 EDG failing the surveillance test was licensee-identified. The CRs for the root causes adequately documented the sequence of events and the conditions of the plant both before the events and during recovery from the events.

However, the inspectors identified an additional issue of training effectiveness for both operators and maintenance craft personnel that was a factor in each of the events except the waterleg pump air-binding event.

b.

Determine that the evaluation documents how long the issue existed and prior opportunities for identification.

For the HPCS pump failure to start, the ESW pump 'A' shaft-coupling sleeve failure, and the Division 1 EDG surveillance test failure events, the CRs and root cause evaluations adequately documented the issue for duration and for prior opportunities to identify the condition(s). However, the inspectors found that the licensee (in CR 03-04764) had failed to identify several prior opportunities to identify that the FWLCS was not included in the LPCS/RHR fill and vent procedure. These opportunities are discussed in Section 2.0.2.4.c.1 of this inspection report. It took the licensee about a month to identify the non-vented FWLCS piping collecting pressurized air as the causal factor for the LPCS/RHR waterleg pump being air-bound and losing pressure during the August 2003 LOOP event. This was fully reported in IR 05000440/2003009.

c.

Determine that the evaluation documents the plant specific risk consequences, (as applicable) and compliance concerns associated with the issue(s) both individually and collectively.

The inspectors reviewed the root cause reports for the three White findings and found that the final revisions of each had adequately addressed the plant specific risk consequences and that the common cause evaluation had collectively evaluated the risk from the common causes. The inspectors considered the initial root cause evaluations weak in addressing risk, particularly the root cause for CR 02-03972, HPCS Pump Failure To Start, because the initial NRC 95001 inspection found that the licensee had not completed an adequate investigation. Therefore, the risk consequences could not have been adequately evaluated.

02.02 Root Cause and Extent of Condition Evaluation Team Comments on Root Cause Evaluations for Each Finding

October 2002 Failure of the HPCS Pump to Start on Demand (CR 02-03972):

The root cause evaluation for the failure of the HPCS pump to start was reviewed as part of the NRC inspections documented in IRs 05000440/2003007 and 05000440/2003012. Four revisions to the root cause report were performed by the licensee, primarily because the extent of condition reviews were inadequate, as documented in IR 05000440/2003007. Further, the inspectors learned from interviews that the reason for the revisions was because the root cause evaluations were of insufficient depth and no problem statements were given. An additional revision, mostly editorial comments, was completed following a review of the evaluations by a contractor.

The final revision was completed primarily to validate the licensees findings for the common cause assessment. During this inspection, the inspectors noted that the root cause evaluation did not address training effectiveness. See Section 02.02.d.3 of this report.

2. August 2003 Post-LOOP LPCS/RHR A Waterleg Pump Air Binding (CR 03-04764):

Revision 1 of the root cause evaluation identified the two root causes as being both design and procedural. The design issue involved the LPCS/RHR A waterleg pump system which allowed air to collect in the Division 1 FWLCS. The identified procedure issue was that site venting procedures failed to periodically remove entrapped air from the high point vent in the Division 1 FWLCS piping. The FWLCS interfaces with the LPCS immediately downstream of the waterleg pump. Revision 1 of the root cause also identified an additional design issue as a contributing cause in that air accumulation in the LPCS/RHR A crossover piping contributed to air collection in the FWLCS piping.

The inspectors noted that the original root cause evaluation identified the design issue as the root cause and the procedural issue as being a contributing cause. In discussions with the members of the licensees root cause team, the inspectors learned that there had been considerable debate about whether the root cause was procedural in nature versus a design issue. The root cause team chose to include both as root causes to ensure corrective actions would fully address the problem.

In addition, the inspectors considered the following to be contributing causes which had not been identified by the licensees root cause evaluation: 1) lack of design reviews for systems, 2) weakness in engineering knowledge, and 3) lack of adequate procedure reviews.

3. September 2003 ESW Pump 'A' Upper Shaft Coupling Sleeve Failure (CR 03-05065):

Revision 1, dated April 16, 2004, of the root cause evaluation identified four root causes as follows: 1) unclear guidance for use of maintenance procedures; 2) procedure content less than adequate; 3) couplings made from SCC susceptible material with high stress concentrations; and 4) inappropriate prioritization of inventory restocking.

The inspectors considered the first root cause, unclear guidance for use of maintenance procedures, to be a good finding by the licensee. The licensee identified that maintenance personnel did not have clear guidance and expectations regarding the level of use for procedures, i.e., whether a procedure was required to be in hand and followed step-by-step versus used as a reference guide. Consequently, many maintenance personnel were not following maintenance procedures step-by-step.

For the second root cause, procedure content less than adequate, the inspectors identified that the attachment showing the pump shaft coupling and key assembly was not explicitly referenced for the pump assembly steps in the general maintenance instruction (GMI)-0039, Disassembly of the Emergency Service Water Pumps.

Although the procedure could be improved, the inspectors noted that the attachment was properly referenced for the pump assembly steps and that the attachment contained sufficient information for correct assembly of the shaft coupling. Both the revised root cause evaluation, and the inspectors, questioned the degree to which the mechanics had used the procedure during the pump assembly in 1997.

The third root cause, couplings made from SCC susceptible material with high stress concentrations, identified that the couplings may have failed over time even if properly installed. The licensees conclusion was based on finite element analysis and stress and crack propagation calculations of the coupling design.

Finally, the fourth root cause, inappropriate prioritization of inventory restocking, identified that the licensee did not have appropriate spare parts in place to repair the pump after the first failure. The inspectors did not review the adequacy of this root cause because it was unrelated to the failure of the pump. This was considered a timeliness issue for repairing and restoring the pump.

The inspectors noted that the original root cause analysis identified the sole root cause as SCC with improper coupling installation, attributed to procedural content, as a contributing cause. As such, the inspectors considered the original root cause evaluation to be lacking in that it did not address organizational or programmatic aspects which resulted in the failure. It was only after the three White findings occurred that the licensee included these issues (See Section 02.03.a.4). Further, during this inspection, the inspectors noted that the root cause evaluation did not address training effectiveness as well. See Section 02.02.d.3 below.

4. August 2003 Division 1 Diesel Generator Failed Test Run (CR 03-04912):

Revision 1, dated April 22, 2004, of the root cause evaluation identified three root causes as follows: 1) the Precautions and Limitations section of Procedure SOI-R43 did not contain the diesel generator high-voltage limit as a value not to be exceeded; 2)expectations for the review of operating parameters in the System Operations section were not reinforced; and 3) the rigor of the Post-Scram Report was less than adequate in that the trip of the diesel on reverse power and subsequent N/A of the procedure steps to adjust the EDG speed and voltage were not addressed.

The inspectors considered the first root cause to be a good procedure enhancement; however, the operators had been trained on the Perry TSs which specify a voltage limit of 4400 volts for diesel generator operability. The inspectors verified with the training department that the operators had been trained on the TS limitations for the EDGs.

Despite their training, three operators failed to recognize that increasing the EDG voltage to match the lightly-loaded grid voltage would exceed the TS voltage limitation.

Additionally, the operators failed to adjust the EDG load before considering the paralleling operation complete, an action that lead directly to the reverse power trip.

The inspectors considered the second and third root causes to be accurate statements of the event. The shift management and two reactor operators conferred and came to the conclusion that the Division 1 EDG did not need to be restarted to have the voltage and speed adjusted for operability because the EDG had just been running and had only tripped on reverse power. However, the inspectors identified that this was another instance of the operators not effectively applying the training they had been given on the operation of EDGs in the isochronous and droop modes. The failure of the Post-Scram Report to identify this issue was another example of the ineffective results from training.

Overall, the inspectors considered the licensees root cause for this event to be adequate; however, the inspectors identified that this root cause did not address training effectiveness. See Section 02.02.d.3 below.

a. Determine that the problem was evaluated using a systematic method(s) to identify root cause(s) and contributing cause(s).

1. August 2003 Post-LOOP LPCS/RHR 'A' Waterleg Pump Air Binding (CR 03-04764):

Revision 1 of the root cause evaluation for CR 03-04764 used multiple systematic methods to identify root and contributing causes. The licensee performed a failure mode analysis and developed an event and causal factor chart. For the original root cause evaluation, the licensee had developed a detailed time line and performed an Alamo cause analysis. The inspectors considered the licensees performance of this root cause evaluation to be systematic.

2. September 2003 ESW Pump 'A' Upper Shaft Coupling Sleeve Failure (CR 03-05065):

For the revised root cause evaluation, the licensee developed an event and causal factor chart, employed TapRooT© Root Cause Tree methodology, and used the Performance Improvement International Organizational and Programmatic Deficiencies Chart. In addition, a barrier analysis was performed for the evaluation. However, as discussed below in Section 02.02.b.2, the barrier analysis was not performed in sufficient depth to identify the lack of quality control as being an issue. For the original root cause evaluation, the licensee performed a failure mode analysis and a cause analysis using the Alamo technique. The inspectors considered the licensees performance of these root cause evaluations to be systematic.

3. Additional Root Cause Evaluations Reviewed

The inspectors randomly selected four other significant root causes to independently assess the licensees ability to conduct root cause evaluations and identify contributing causes. These included the following:

• CR 03-05745, Safety Relief Valves Lifted During the Performance of SVI-B21-T0369B, Revision 1; C CR 03-06739, Hydramotor 1P42F0665A Failure During a Concurrent Control Complex Chilled Water 'B' Chiller Maintenance Activity Which Was Extended Due to Brazing Rework, Revision 0; C CR 04-00935, Measuring and Test Equipment Program Requirements for Ensuring Traceability Are Not Consistently Met, Revision 0; and

• CR 04-01404, Timeliness of Off-hours Emergency Response Organization Unannounced Drill Manning.

No additional findings were identified.

b. Determine that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

1. August 2003 Post-LOOP LPCS/RHR A Waterleg Pump Air Binding (CR 03-04764):

The root cause evaluations were not performed in sufficient depth to fully identify issues which were within the licensees control. Root cause evaluations correctly identified that the design and procedural issues originated from the time of original construction, i.e., the 1980's. The inspectors recognized that, due to the age of the issues, a historical review of the causes would have had minimal benefit. Additionally, Revision 1 of the root cause evaluation did evaluate the issue of failing to detect the condition over time.

Design Reviews In Revision 1 of the root cause evaluation, the licensee specifically reviewed the issue of design review activities for the LPCS system and did not identify any issues for further consideration. The inspectors disagreed with the licensees assessment in this area.

The inspectors considered the lack of a recent design review and a lack of engineering knowledge to warrant further consideration.

The inspectors noted that no thorough design reviews of the LPCS system had been recently performed. The licensees root cause evaluation identified a reason for the LPCS system not being selected for the detailed review, was the Green (i.e., excellent)performance rating in the licensees System Health Report. The inspectors interviewed the current and two previous system engineers for the LPCS system and confirmed that the LPCS system was believed to be a relatively trouble-free system. Consequently, system engineers spent less time on the LPCS system in relation to other engineering assignments and responsibilities. The inspectors noted that while system health monitoring systems are generally effective for assessing the material condition of a system, such forms of monitoring do not provide effective indication of the quality of system design. The inspectors considered the lack of an effective design review to be a contributing cause. This contributing cause was not captured in the root cause evaluation.

As part of the licensees root cause evaluation, the licensee determined, based on interviewing several engineers, that the failure mode resulting in air binding of the waterleg pump would not have been detected due to the engineers lack of awareness of the failure mode. The inspectors interviewed three of the system engineers for the LPCS system and confirmed that two of the engineers were not aware of the potential for gases coming out of solution and accumulating in high points before the event.

Additionally, the two engineers were not previously aware that the FWLCS tied into the LPCS system just downstream of the waterleg pump. The inspectors did not consider these responses surprising, since the level of ongoing education provided by the licensee was minimal and that the engineers preoccupation with other systems and assignments demanded more of their attention. The inspectors determined that improvement in the engineering staffs knowledge levels was needed. Based on discussions with engineering management, the inspectors learned that the licensee planned training on the phenomena of gases coming out of solution in piping systems and the potential for accumulation in high points of systems. The inspectors considered weaknesses in engineering knowledge to be a contributing cause that was not captured in the root cause evaluation.

Procedure Reviews Revision 1 of the root cause evaluation specifically evaluated the issue of procedure reviews for the LPCS and FWLC systems and did not identify any issues for further consideration. The inspectors disagreed with the licensees assessment in this area.

The inspectors considered the lack of procedure reviews to warrant further consideration.

The inspectors noted that, due to a December 1995 change in the licensees quality assurance program (approved by the NRC), biennial reviews of operating procedures were no longer required. The inspectors also noted that the biennial reviews, which had been performed until the change in program requirements, were ineffective in identifying the deficiency associated with the venting procedures. Biennial reviews of the procedure had been performed from 1985 through 1995. Specifically, biennial reviews were performed for the LPCS System Operating Instruction (SOI)-E21, Low Pressure Core Spray, on February 14, 1985; August 3, 1987; August 29, 1989; September 23, 1991; September 21, 1993; and September 29, 1995. In addition, biennial reviews were performed for Surveillance Instruction (SVI)-E21-T1181, LPCS Venting and Valve Line-up Verification, on September 1, 1988; July 13, 1990; August 7, 1992; and May 25, 1994. The reviews were done in accordance with Perry Administrative Procedure (PAP)-0502, Preparation, Review, and Approval of Procedures. As part of the root cause evaluation, the licensee determined, based on interviews, that procedure writers and reviewers typically only reviewed the changes and the impact of changes to the procedures. The portions of procedures not affected by the change were assumed to be good. The inspectors considered the lack of adequate procedure review to be a contributing cause.

Based on the interviews conducted, the licensee also determined that the procedure writers would not have included the FWLCS system and LPCS/RHR 'A' crossover piping without specific knowledge of the potential for air binding of the waterleg pump. The inspectors took exception to this conclusion that specific knowledge of the phenomena was required. Section 7.2 of SOI-E21, the SOI for the LPCS system, was for filling and venting the LPCS system. The inspectors expected a procedure writer developing a system filling and venting procedure, such as Section 7.2 of SOI-E21, would have identified all the high points of the system in order to allow complete filling of the system, regardless of the individuals knowledge with respect to the potential for air accumulation in a system. The inspectors further noted SOI-E21 did identify valve 1E21-F522, the LPCS waterleg pump vent, to be opened as part of the system fill and vent. However, the procedure did not consider the FWLCS system piping even though the piping for the FWLCS came off the LPCS system piping vertically upwards approximately a foot downstream of the waterleg pump and valve 1E21-F522, i.e., well within sight of a valve being manipulated as part of the procedure. The inspectors recognized that knowledge of the potential for air coming out of solution and accumulating in high points would have been beneficial. However, the inspectors considered it reasonable that a procedure review would identify all the high points in a system, or connected system, for venting purposes. Further discussion on the subject of missed prior opportunities to identify for not including the FWLCS high point vent valve in the venting procedure is in Section 02.02.c.1 below.

2. September 2003 ESW Pump 'A' Upper Shaft Coupling Sleeve Failure (CR 03-05065):

Introduction:

The inspectors identified an NCV of 10 CFR 50, Appendix B, Criterion X, having very low safety significance (Green) for failing to include quality control sign-offs or hold points in the maintenance instructions for verifying proper assembly of ESW pump 'A' coupling sleeve as required by ANSI N45.2.8 - 1975, Supplementary Quality Assurance Requirements for Installation, Inspection, and Testing of Mechanical Equipment and Systems for the Construction Phase of Nuclear Power Plants. This issue was considered to be NRC-identified because the licensee had failed to identify the need for the requirement during the root cause evaluation for CR 03-05065 regarding the failure of ESW pump 'A' coupling sleeve.

Description:

The inspectors considered the revised root cause evaluation to have been performed with sufficient depth to identify organizational and programmatic issues.

However, the evaluation failed to identify the lack of quality control as being an issue.

Although the TapRooT© Root Cause Tree charts that the licensee used listed quality control as a cause category, the inspectors did not identify any consideration of quality control in the root cause evaluation. The inspectors reviewed the work package for the July 1997 ESW pump assembly and determined that the only quality control hold point in the package was for witnessing the re-terminating of the pump motor and heater leads. Additionally, the hold point had been waived by quality control personnel. No quality control witness or hold points existed for assembly of the pump shaft and couplings. Also, no quality control hold points were implemented for the pump shaft and coupling assembly performed in September 2003 and June 2004 for the train 'A' ESW pump, nor in April 2003 for the train 'B' ESW pump.

While discussing this with quality control personnel, the inspectors learned that the quality control witness or hold points were not routinely specified for maintenance activities unless it was specifically required. For example, quality control witness or hold points were typically specified for special processes, e.g., welding, and activities specified by the American Society of Mechanical Engineers Boiler Code as requiring inspection. However, quality control witness or hold points were seldom specified for assembly of mechanical components unless required by the Boiler Code.

Analysis:

The inspectors determined that the practice of not implementing quality control hold points for assembly of ESW pump 'A' was a performance deficiency warranting a significance evaluation. The inspectors concluded that the finding was greater than minor in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Disposition Screening, issued June 30, 2003. The finding, if uncorrected, could become a more significant safety concern. Specifically, continued failure to implement quality control hold points could allow mis-assembly of a pump to go undetected and result in a subsequent failure of a pump. The safety significance of the September 1, 2003, failure of ESW pump 'A' is addressed in IR 05000440/2003006.

The safety significance of the May 21, 2004, second failure of ESW pump A is addressed in IR 05000440/2004011.

The inspectors completed a significance determination of this issue using IMC 0609, Significance Determination Process, dated March 18, 2002, Appendix A, Significance Determination of Reactor Inspection Findings for At-Power Situations. The inspectors answered no to all five screening questions in the Phase 1 Screening Worksheet under the Mitigating Systems column. Aside from the two examples above, the inspectors were not aware, at the time of this inspection, of any other failures of ESW pumps due to incorrect installation. As such, the inspectors determined that the failure to implement quality control hold points for ESW pump assembly was of very low safety significance (Green).

Enforcement:

10 CFR Part 50, Appendix B, Criterion X, Inspection, requires, in part, that a program for inspection of activities affecting quality be established and executed by or for the organization performing the activity to verify conformance with the documented instructions, procedures, and drawings for accomplishing the activity.

Appendix B of 10 CFR Part 50, Criterion II, Quality Assurance Program, requires, in part, that a quality assurance program be established which meets the requirements of 10 CFR Part 50, Appendix B. The FENOC Quality Assurance Program Manual described the quality assurance program to satisfy 10 CFR Part 50, Appendix B, Criterion II. Section A.7.a.1 of the FENOC Quality Assurance Program Manual specifies, in part, that for non-routine maintenance activities, guidance applicable to construction-like activities is applicable to comparable plant activities. Section B.12.a of the FENOC Quality Assurance Program Manual specifies, in part, that a program be established and implemented for inspections of activities in order to verify conformance to the documented instructions, procedures, and drawings for accomplishing the activity.

Section L of Table 1, Regulatory Commitments, of the FENOC Quality Assurance Program Manual committed the licensee to Regulatory Guide 1.116, Quality Assurance Requirements for Installation, Inspection, and Testing of Mechanical Equipment and Systems, which endorsed ANSI N45.2.8 - 1975. Section 4.4 of ANSI N45.2.8 - 1975 requires, in part, that inspections of the work areas and the work in progress shall be performed to verify that mechanical items are being located, installed, assembled, or connected in compliance with the latest approved-for-construction drawings, manufacturers instructions, codes, installation instructions, and procedures. Contrary to the above, reassembly of the emergency service water pumps, a non-routine maintenance activity, performed in July 1997, April 2003, September 2003, and June 2004, was performed with no inspections of work in progress to verify that mechanical items were installed and assembled in compliance with the latest manufacturers instructions, installation instructions, and procedures. Once identified, the licensee entered the issue into their corrective action program as CR 04-03026.

Because this violation was of very low safety significance and it was entered into the licensees corrective action program, this violation is being treated as an NCV, consistent with Section VI.A of the NRC Enforcement Policy.

(NCV 05000440/2004008-01).

The inspectors concluded that implementing such controls was an additional barrier for ensuring the coupling was installed properly. Therefore, failure to follow the quality control inspection requirements for reassembling the pump coupling sleeve in 1997 was not a direct cause of the coupling failure.

c. Determine that the root cause evaluation included a consideration of prior occurrences of the problem and knowledge of prior operating experience.

1. August 2003 Post-LOOP LPCS/RHR A Waterleg Pump Air Binding (CR 03-04764):

Introduction:

A Green finding was identified by the inspectors while reviewing CR 03-04764 in that the licensee did not include multiple missed opportunities to identify and to include the FWLCS high point vent valve 1N27-F0786 in the root cause evaluation for the CR. The finding was considered an NCV of 10 CFR 50, Appendix B Criterion XVI.

Description:

The licensees root cause evaluations concluded there were no prior opportunities to identify the conditions resulting in the air binding of the LPCS waterleg pump. However, the inspectors identified the licensee had several prior opportunities to identify that system venting procedures were inadequate and, to a lessor extent, that there was air in the FWLCS piping which had not been vented. As such, the inspectors determined the licensees root cause evaluation was inadequate based on the following:

• 1996 Review of RHR System Venting In 1996 the licensee performed a review of the RHR system to resolve venting concerns. The licensee review was documented on Problem Identification Form 96-212, which was the corrective action document at that time. From this RHR system review, the licensee revised the filling and venting sections of the SOI for the RHR system and performed modifications to the RHR system. As part of the extent of condition determination, the review erroneously concluded that the LPCS system can be adequately vented. No actions were taken with respect to the LPCS system or procedures as a result of this review. The inspectors considered this a missed opportunity.

• Opening of FWLCS Vent Valve to Drain System The work in progress log for Work Order 99-3053 identified that the FWLCS high point vent valve, 1N27-F0786, was opened on April 20, 2003, to allow draining of the system for work on an RHR system valve. The log noted that operators in the field reported that no additional water had been drained out of the system as a result of opening valve 1N27-F0786. These actions were not documented in a CR. The inspectors noted that in April 2003, valve 1N27-F0786 was not listed in any venting procedure. As such, this occurrence provided the licensee an opportunity to identify that the existing procedure for venting RHR was inadequate. In addition, the fact that no additional water drained out of the system provided an indication that there was essentially no water in the FWLCS line at that time.

• Water Reported Coming from FWLCS Vent Valve The operator log entries for May 7, 2003, identified that water was reported coming from a ventilation plenum in the auxiliary building. The water was identified as coming from valve 1N27-F0786. The valve had been inadvertently left open following draining activities on April 20, 2003. Operators subsequently closed the valve. Although this event resulted in an unintentional breach of a system and demonstrated that the tagging system did not work, no CRs were written from this event. Had a corrective action been initiated, the fact that valve 1N27-F0786 was not listed in any venting procedure could have been readily identified. This was another missed opportunity to identify that the high point vent valve was missing from the procedure. The licensee initiated CR 04-03105 to address the inspectors concerns.

Analysis:

The inspectors determined that failing to identify the multiple missed opportunities to include the FWLCS high point vent valve 1N27-F0786 during the root cause evaluation of CR 03-04764 was a performance deficiency. This performance deficiency warranted a significance evaluation in accordance with IMC 0612,Power Reactor Inspection Reports, Appendix B, Issue Disposition Screening, issued on June 30, 2003. The inspectors concluded that not conducting a thorough root cause evaluation could be a precursor to a more significant event. Failing to identify multiple prior opportunities in the root cause evaluation associated with air binding of the LPCS/RHR 'A' waterleg pump, a significant self-revealed event, indicated an example of an ineffective corrective action program. Using IMC 0609, Appendix A, Significance Determination of Reactor Inspection Findings for At-Power Situations, the inspectors answered no to all five screening questions in the Phase 1 Screening Worksheet under the Mitigating Systems column. The inspectors concluded the issue was of very low safety significance (Green).

Enforcement:

Appendix B of 10 CFR Part 50, Criterion V, Instructions, Procedures, and Drawings, requires, in part, that activities affecting quality shall be prescribed by documented instructions, procedures, or drawings, of a type appropriate to the circumstances.

Appendix B of 10 CFR Part 50, Criterion XVI, Corrective Action, requires, in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. The identification of the significant condition adverse to quality, the cause of the condition, and the corrective action taken shall be documented and reported to appropriate levels of management.

Filling and venting the LPCS system was an activity affecting quality and was prescribed by SOI-E21, Low Pressure Core Spray System. A significant condition adverse to quality existed in that SOI-E21 did not provide instructions of a type appropriate to the circumstances in that the procedure did not direct operators to open all attached high point vents in the LPCS system for system filling and venting. Contrary to the above, although multiple opportunities existed to identify this condition, the licensee failed to identify and correct this issue until September 2003 after the failure to properly vent resulted in air binding of the Division 1 LPCS waterleg pump and inoperability of the Division 1 LPCS and RHR systems. The multiple opportunities to identify this condition included:

• Biennial procedure reviews of SOI-E21 conducted 1985 through 1995.

• The 1996 review of the RHR system, including the associated extent of condition review which addressed the LPCS system venting concerns. This review failed to identify the inadequacy of existing procedures for filling and venting the LPCS system and connected FWLCS system.

• The April 20, 2003, use of the FWLCS vent valve to drain the system. The need to use the FWLCS vent valve illustrated the inadequacy of existing procedures for system venting.

• The May 7, 2003, event involving water coming from the FWLCS vent valve provided an opportunity to identify that the valve was not in existing procedures.

Because this violation was of very low safety significance and is entered into the licensees corrective action program, this violation is being treated as an NCV, consistent with Section VI A of the NRC enforcement policy.

(NCV 05000440/2004008-02).

2. September 2003 ESW Pump 'A' Upper Shaft Coupling Sleeve Failure (CR 03-05065):

There were no prior instances of coupling failure. The original root cause evaluation did evaluate industry operating experience. As a result of this review, several failures were identified due to stress corrosion cracking (SCC) involving 416 stainless steel similar to that used for the ESW pump couplings at Perry. However, given that 416 stainless steel was a commonly used material, the material was not specifically identified as an issue as a result of this licensee review.

d. Determine that the root cause evaluation addresses the extent of condition and the extent of cause of the problem.

1. August 2003 Post-LOOP LPCS/RHR 'A' Waterleg Pump Air Binding (CR 03-04764):

The licensees root cause evaluation reviewed the extent of condition for other systems and did not identify any problems. The inspectors review of other systems did not identify any significant problems.

2. September 2003 ESW Pump 'A' Upper Shaft Coupling Sleeve Failure (CR 03-05065):

As part of the original root cause evaluation, the licensee reviewed the design of couplings for other vertical pumps for extent of condition. The inspectors did not identify any concerns with respect to the extent of condition review. However, the inspectors identified that the this root cause did not address training effectiveness. See Section 02.02.d.3 below.

3. Training Effectiveness Related to the Root Cause Evaluation of Extent of Cause for

Failure of the HPCS Breaker (CR 02-03972); Failure of ESW Pump A (CR 03-05065); and Tripping of Division 1 EDG (CR 03-04912):

Introduction:

A Green finding was identified by the inspectors while reviewing the root cause evaluations for the HPCS breaker failure, the ESW A pump shaft failure, and the Division 1 EDG reverse power trip, in that the licensee had failed to note that, in each case, training had been ineffective in achieving the desired behaviors and results from licensee staff. The finding was considered to be an NCV of 10 CFR 50, Appendix B, Criterion XVI.

Description:

The inspectors review of the three root cause evaluations revealed that the licensee had considered and discounted training as a possible root or contributing cause for the events. The inspectors held interviews with training department personnel and reviewed the lesson plans for the staff. The inspectors found that licensee staff had been trained in the areas where their performance was lacking; however, the training had been ineffective in attaining the desired results.

In particular, the staff had received training as follows:

• For the HPCS breaker, the maintenance staff had been trained to not use up all available adjustment on the operating rod for the cell switches; yet, the final extent of condition inspection found at least seven breakers where all of the available adjustment had been used and the cell switches were not correctly adjusted. The workers should have noted the discrepancy and sought resolution to the problem.

• For the ESW pump 'A' shaft coupling, the inspectors found that mechanics had been trained on assembling the pump coupling sleeves. However, the inspectors determined that the mechanic who assembled ESW pump 'A' coupling sleeve had not used the procedure or manufacturer's technical manual, had not installed the key into the coupling keyway correctly, and apparently had not stopped to get assistance with installing the shaft coupling. The result was that the coupling was installed incorrectly, causing considerably more strain than was normal. The excess strain was a direct factor in the coupling failing due to SCC. The 'B' ESW pump had the same type of coupling in the same environment but it had been installed correctly and had not failed.

• For the Division 1 EDG tripping on reverse power, the operators had been trained on operating the EDG, yet failed to use the most basic of training for paralleling a generator to another electrical source. The operators and the shift supervisor failed to recognize that the grid voltage was above the TS allowed diesel generator voltage, failed to complete the generator paralleling operation (balance loads before a reverse power trip occurred), and failed to analyze what the actual condition of the diesel generator would be before marking the procedure steps N/A to restore the diesel generator to a ready-to-start condition. The result was that the EDG failed the next operability surveillance test due to the voltage being outside the allowable value.

In each of these cases, training had been ineffective in attaining the desired results.

However, licensee root cause evaluations had not identified training effectiveness as a factor in the root causes.

Analysis:

The inspectors determined that failing to identify that training effectiveness was a causal factor in the root cause evaluations for the failure of the HPCS breaker (CR 02-03972), the failure of ESW pump A (CR 03-05065), and the tripping of the Division 1 EDG (CR 03-04912), was a performance deficiency. This performance deficiency warranted a significance evaluation in accordance with IMC 0612,Power Reactor Inspection Reports, Appendix B, Issue Disposition Screening, issued on June 30, 2003. The inspectors concluded that not conducting a thorough root cause evaluation could be a precursor to a more significant event. Failing to identify that staff training had been ineffective in attaining the desired staff performance in the root cause evaluations associated with setting the cell switches in the HPCS circuit breaker (CR 02-03972), installing the ESW pump 'A' shaft coupling sleeve (CR 03-05065), and correctly operating the EDGs (CR 03-04912); all significant self-revealing or licensee-identified events, indicated an example of an ineffective corrective action program. Using IMC 0609, Appendix A, Significance Determination of Reactor Inspection Findings for At-Power Situations, the inspectors answered no to all five screening questions in the Phase 1 Screening Worksheet under the Mitigating Systems column. The inspectors concluded the issue was of very low safety significance (Green).

Enforcement:

10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, requires, in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. The identification of the significant condition adverse to quality, the cause of the condition, and the corrective action taken shall be documented and reported to appropriate levels of management. Root cause evaluations are required to identify the causes for conditions adverse to quality. Contrary to the above, although multiple opportunities existed to identify ineffective training as a causal factor, the licensee failed to identify this issue in the root cause investigations for CR 02-03972, Failure of HPCS Pump to Start on Demand Resulting in Unavailability of the Division 3 Emergency Core Cooling System; CR 03-05065, Emergency Service Water Pump A Upper Shaft Coupling Sleeve Failure; and CR 03-04912, Division 1 Diesel Generator Inoperability During Mode Change. The licensee entered this issue into their corrective action program as CR 04-03020. Because this violation was of very low safety significance and is entered into the licensees corrective action program, this violation is being treated as an NCV, consistent with Section VI.A of the NRC enforcement policy (NCV 05000440/2004008-003).

02.03 Corrective Actions a.

Determine that appropriate corrective action(s) are specified for each root cause and contributing cause or that there is an evaluation that no actions are necessary.

Corrective actions had not yet been developed for failure to include quality control requirements for ESW pump A coupling sleeve assembly (Section 02.02b.2), failure to identify prior opportunities with the LPCS/RHR A inadequate venting procedure (Section 02.02c.1), and ineffective training (Section 02.02d.3) since the licensee failed to identify these issues in the associated root cause evaluations.

Generally, for two of the three White findings, the inspectors determined that the corrective actions to address the identified root and contributing causes were appropriate. However, the cause identification and corrective actions for the ESW pump were inadequate as noted in 3. below. Specific observations or concerns are listed below:

October 2002 Failure of HPCS Pump to Start on Demand (CR 02-03972):

The licensee determined that the HPCS pump breaker had failed to close because the cell switch contacts that provide breaker closure permissive, had been improperly adjusted. Cell switch contact tabs were to be set within the upper and lower mold lines of the cell switch viewing window. If the contacts are set above or below these window mold lines, the contacts may not make up. The licensee incorporated this correction, focusing solely on the maintenance procedure that applied to that type of breaker.

Other breaker maintenance procedures were not evaluated. The licensee failed the initial 95001 supplemental inspection because these corrective actions were inadequate and the extent-of-condition determinations failed to address all susceptible breakers. As described in IR 05000440/2003012 for the second 95001 supplemental inspection, the licensee conducted a walkdown of other breakers, made adjustments to cell switches, and revised maintenance procedures. No additional concerns were identified.

August 2003 Post-LOOP LPCS/RHR 'A' Waterleg Pump Air Binding (CR 03-04764):

As discussed in IR 05000440/2003009 and in Section 02.01b., it took the licensee about a month to identify excessive air inside the FWLCS waterleg piping. Condition Report 03-05201 was written to evaluate the weakness in timely identification of the issue. The inspectors reviewed the associated corrective action of discussing with Perry managers the importance of maintaining effective oversight. The inspectors considered this corrective action weak in that it was limited in scope, only counseling the current management team with no regard to addressing the issue from a programmatic or cultural standpoint. The corrective action provided no mechanism to ensure that new managers (replacing a counseled manager) would receive the same training. The licensee stated that this issue would be addressed in the Improvement Initiative Plan discussed below in Section 02.03.a.4

3. September 2003 ESW Pump A Upper Shaft Coupling Sleeve Failure (CR 03-05065):

The corrective actions to prevent a significant condition adverse to quality for the September 1, 2003, ESW pump A failure were ineffective since the same pump coupling failed again on May 21, 2004. Perry Resident Inspectors conducted a special inspection and documented in IR 05000440/2004011 an AV of 10 CFR 50, Appendix B Criterion XVI, for inadequate corrective action. The color significance of the finding is to be determined at a later date. No further enforcement action regarding this issue was taken during this inspection.

4. Common Cause Analysis Report (CR 03-05995):

The licensee recently had a Perry Organizational Effectiveness Assessment conducted between January and February 2004, and evaluations of the corrective action and operating experience programs were completed. As a result of weaknesses in self-improvement culture and program monitoring, managers from Perry Nuclear Power Plant identified that their staff had been ineffective in ensuring adequate attention to emerging plant issues. The licensee identified that supervisory weaknesses contributed to issues regarding communication of expectations, self-checking, and misjudgement.

Further, although several corrective actions have been located in the corrective action program database for many site-wide specific issues, the licensee recognized that underlying organizational and programmatic common causes of events may not have been effectively addressed. Also, the licensee identified that issues with procedure content were a common cause of the White findings identified in the Mitigating System Cornerstone. The licensee has developed additional corrective actions to address self-improvement culture and program monitoring common causes and developed an Organizational Effectiveness Improvement Plan.

A root cause team member responsible for completing the collective significance review for the three Mitigating System failures had noticed similar cross-cutting issues identified in the organizational assessment. Further, the individual who conducted the organizational assessment had reviewed significant CRs of all equipment associated with the Mitigating System Cornerstone. To prevent organizational and mitigating equipment issues from being lost in the corrective action program, the Organizational Effectiveness Improvement Plan and issues with equipment reliability, barrier integrity and procedure content and quality were rolled into one document called Improvement Initiative Plan. This plan received senior management oversight for completing the associated corrective actions. At the time of the inspection, the inspectors were unable to review the Improvement Initiative Plan and associated corrective actions.

However, the inspectors did review some corrective actions to common cause analysis CR 03-05995, specifically, procedure content weaknesses. For procedure content weaknesses, the inspectors randomly selected maintenance procedures for the 18-month EDG engine overhaul and for the reactor core isolation cooling (RCIC)system, to evaluate how the licensee had progressed in implementing the corrective action in this area. No concerns were identified in the EDG procedures.

Procedure markups of GMI-125, Reactor Core Isolation Cooling Turbine Overhaul; and GMI-0182, RCIC Trip and Throttle Valve Maintenance and Trip Linkage Adjustment; were conducted on April 21, 2004, to satisfy the common cause corrective action of improving the maintenance procedures. The marked-up copies had not been incorporated as a revision at the time the inspectors reviewed these copies. During the review, the inspectors noted that the licensee was not including guidance from the Electric Power Research Institute (EPRI) Nuclear Maintenance Applications Center (NMAC) in the markups. The EPRI organized Terry Turbine Users Group meetings to discuss improvements on turbine maintenance based on lessons learned and industry experience. Notes compiled from these meetings are published as EPRI NMAC documents. Engineers used these publications to incorporate improved instructions into Terry Turbine procedures.

During the licensees April 21, 2004, markup of the RCIC procedure, guidance specified in the EPRI Terry Turbine Maintenance Guide found in EPRI TR-1058745, issued in November 2002, was not included. Specifically, inspections of the trip throttle valve yoke compression spring coils for evidence of damage, and measurements of the spring free length and spring constant, were not included. Further, engineers did not include the NMAC guidance for incorporating visual inspection of the trip hook reset spring coils for evidence of damage and permanent distortion. Also, the licensee eliminated caution statements and notes essentially undoing previous NMAC guidance (issued in December 1995 and October 1998).

Although the decision to incorporate the NMAC guidance was solely voluntary, the common cause corrective action was to improve maintenance procedures, which could have been accomplished by incorporating this guidance. More significantly, Step 4.3, Trip Reports, of Nuclear Operating Business Practice (NOBP)-SS-7000, EPRI Committee and User Group Member, required Perry engineers to write CRs based on trip reports after every EPRI Users' Group meeting to investigate and possibly incorporate these improvements. Unfortunately, the RCIC system engineer, who was involved in the maintenance instruction revision, did not know this requirement existed because he had not been trained. Polling of other mitigating systems engineers determined that the HPCS engineer did not know of this requirement either. Essentially, the inspectors determined that by not incorporating the NMAC guidance, maintenance procedures were not being improved, which was the intent of the common cause corrective action. This was not considered a finding since the procedure revision and implementation process had not been completed when this observation was made. The licensee initiated CRs 04-03039 and 04-03036 to address the inspectors concerns.

b. Determine that the corrective actions have been prioritized with consideration of the risk significance and regulatory compliance.

The inspectors reviewed the root cause evaluations and determined that the prioritization of the corrective actions was not directly based upon risk perspectives or analysis, but rather based upon a deterministic approach that considered the significance of each corrective action. The inspectors concluded that, in general, actions of a higher priority were scheduled for completion ahead of those having lesser urgency.

c. Determine that a schedule has been established for implementing and completing the corrective actions Most of the corrective actions for addressing the root causes have been completed for the three White findings. As mentioned in Section 02.03.a.3, the corrective actions to prevent recurrence of a significant condition adverse to quality following the September 1, 2003, ESW pump coupling sleeve was ineffective because the same coupling failed again on May 21, 2004. Initially, the corrective action to replace the coupling with an improved design was scheduled for March 2003. A variety of reasons, such as parts availability and concerns over staff conduct during outages, caused several changes to this date. Following the May 21, 2004, failure, the improved coupling design was installed on May 25, 2004, under Work Order 200003985.

The licensee determined that the effectiveness of the corrective actions to the common cause evaluation would be evident by improved mitigating system reliability and performance. The licensee had established performance indicators to measure the performance of these systems. Additionally, the corrective action program procedure provided a means of controlling due date changes for CRs associated with the White findings. Changes to the due dates for corrective actions in the improvement initiative plan would have to be approved by senior management. The inspectors concluded that these tools, combined with the initiated corrective actions for the identified issues provided acceptable barriers to promote success of the organizational effectiveness improvement plan corrective actions.

d. Determine that quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence.

The inspectors reviewed the corrective actions for the three White findings and associated proposed corrective actions. Typically, NOBP-LP-2007, Condition Report Process Effectiveness Review, required an effectiveness review be developed by an independent individual -- someone not on the root cause team -- within 6 months of implementing the corrective action. Effectiveness reviews were implemented through several corrective actions developed from the root cause report for the three White findings. Generally, the inspectors had no additional concerns with these corrective actions.

However, the inspectors challenged the methodology used in determining the effectiveness required in NOBP-LP-2007. This concern arose while reviewing Corrective Action 03-04764-023 regarding developing the effectiveness review for air binding of the LPCS/RHR waterleg pump. Specifically, the inspectors determined that no plan existed to determine the measured parameter (i.e. air accumulation) or the acceptance criteria following implementation of the FWLCS piping configuration changes to prevent air accumulation. Since the individual responsible for developing the effectiveness review plan was independent from the root cause team, no one was assigned to review the reviewers plan or to determine if the plan was effective. The inspectors felt that someone from the root cause team was qualified to develop the effectiveness review plan. The licensee initiated CR 04-02762 in response to the inspectors concerns.

02.04 Independent Review of Extent of Condition and Cause a.

Extent of Condition Evaluation for Improperly Aligned Cell Switch, Failed ESW Pump Coupling and Waterleg Pump Air Binding The inspectors reviewed design information, performed walkdowns, and reviewed licensee maintenance procedures to determine if the licensee had adequately identified the extent of condition for the subject findings. The inspectors concluded that the licensee adequately bounded the alignment of cell switches; however, the adequacy was based, in part, on expending the scope of cell switch alignments based on NRC observations documented in IR 05000440/2003007. The inspectors also concluded that the licensee had bounded the extent of condition for inadequate venting of keep fill pump systems and split ring coupling alignments susceptible to SCC. In both cases, the number of vulnerable systems was limited.

b.

Extent of Cause The inspectors reviewed maintenance procedures for adequacy, including clarity of instruction and adherence to vendor manual requirements. The inspectors noted that during the last maintenance activities conducted on ESW pumps, the licensees instructions had a single broad step to assemble the pump. The vendor manual provided significantly more instruction regarding pump assembly. In particular, the inspectors noted that even though the pump uses threaded couplings, the vendor manual stipulated that the coupling should be centered between the two shafts. The inspectors noted that other procedures did not fully conform to vendor manual guidance.

The licensees root cause evaluation had similarly concluded that procedure inadequacies existed and the licensee had an ongoing procedure revision effort to identify and correct procedure inadequacies.

During the inspection, the licensee experienced a repeat failure of the Division 1 ESW pump coupling. This subsequent failure indicated that the causes identified for the subject findings were not fully understood. Therefore, the inspectors concluded that the licensees corrective actions for extent of cause were not complete and continued evaluation of the extent of cause would be deferred to future inspections.

MANAGEMENT MEETINGS

Exit Meeting Summary

The inspectors debriefed the results to Mr. L. Meyers and other members of licensee management at the conclusion of the inspection on June 10, 2004. The inspection results were presented to Mr. W. Kanda at a public exit meeting on June 21, 2004. The licensee acknowledged the findings presented at both meetings. No proprietary information was identified.

ATTACHMENTS:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

W. Kanda, Vice President-Nuclear

K. Cimorelli, Acting Director, Nuclear Maintenance

V. Higaki, Manager, Regulatory Affairs

J. Lausberg, Supervisor, Compliance

F. von Ahn, General Manager, Nuclear Power Plant Department

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000440/2004008-01 NCV Failure to Follow Quality Control Requirements of ANSI N45.2.8 - 1975

05000440/2004008-02 NCV Missed Prior Opportunities to Identify Missing Vent Valve in the Feedwater Leakage Control System During Root Cause Evaluation for CR 03-04764

05000440/2004008-03 NCV Training Effectiveness Not Addressed in Root Cause Report for CRs 02-03972, 03-05065 and 03-04912

Closed

05000440/2004008-01 NCV Failure to Follow Quality Control Requirements of ANSI N45.2.8 - 1975

05000440/2004008-02 NCV Missed Prior Opportunities to Identify Missing Vent Valve in the Feedwater Leakage Control System During Root Cause Evaluation for CR 03-04764

05000440/2004008-03 NCV Training Effectiveness Not Addressed in Root Cause Report for CRs CR 02-03972, 03-05065 and CR 03-04912

Discussed

None