IR 05000440/2003012

IR 05000440-03-012, on 12/01/03 - 12/04/03; Perry Nuclear Power Plant; Supplemental Inspection IP 95001; Mitigating Systems

ML033580519
Person / Time
Site:	Perry
Issue date:	12/22/2003
From:	Reynolds S Division Reactor Projects III
To:	Kanda W FirstEnergy Nuclear Operating Co
References
EA-03-007, IP 95001 IR-03-012
Download: ML033580519 (13)

Text

ber 22, 2003

SUBJECT:

PERRY NUCLEAR POWER PLANT NRC SUPPLEMENTAL INSPECTION REPORT 05000440/2003012

Dear Mr. Kanda:

On December 4, 2003, the U.S. Nuclear Regulatory Commission (NRC) completed a follow-up supplemental inspection at your Perry Nuclear Power Plant. The enclosed report documents the inspection findings which were discussed on December 4, 2003, with you and other members of your staff.

The NRC previously performed this supplemental inspection to assess your evaluation of the October 23, 2002, failure of the high pressure core spray (HPCS) pump to start during routine surveillance testing. This failure occurred as a result of inadequate procedure implementation during installation and inspection of the HPCS pump breaker from 1994 through October 23, 2002. This performance issue was previously characterized as having low to moderate risk significance (White) in the NRCs final significance determination letter dated March 4, 2003. As stated in our inspection report dated August 21, 2003, we concluded that your review of the performance issue was incomplete because of significant deficiencies with regard to your extent of condition review. As a result, the White finding associated with the performance issue remained open.

This supplemental inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license. The inspector reviewed selected procedures and records and interviewed personnel. The purpose of this inspection was to (1) provide assurance that the root and contributing causes for the performance issue were understood; (2) provide assurance that the extent of condition and extent of cause of the performance issue were identified; and (3) provide assurance that the corrective actions to address the performance issue were sufficient to prevent recurrence.

Based upon the results of this follow-up inspection, the inspector determined that an adequate extent of condition review had been completed. As a result of your acceptable performance in addressing the incomplete extent of condition evaluation, the White finding will be closed.

Consequently, the White finding will only be considered in assessing plant performance using the NRC Action Matrix through the end of the fourth quarter 2003. In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA Patrick L. Hiland for/

Steven A. Reynolds, Acting Division Director Division of Reactor Projects Docket No. 50-440 License No. NPF-58

Enclosure:

Inspection Report 05000440/2003012 w/Attachment: Supplemental Information

REGION III==

Docket No: 50-440 License No: NPF-58 Report No: 05000440/2003012 Licensee: FirstEnergy Nuclear Operating Company (FENOC)

Facility: Perry Nuclear Power Plant, Unit 1 Location: P.O. Box 97 A200 Perry, OH 44081 Dates: December 1 through 4, 2003 Inspector : R. Powell, Senior Resident Inspector Approved by: Mark A. Ring, Chief Branch 1 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000440/2003012; Perry Nuclear Power Plant; 12/01/03 - 12/04/03; Supplemental

Inspection IP 95001. Mitigating Systems.

This report covers a supplemental inspection performed by the senior resident inspector. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG 1649, Reactor Oversight Process, Revision 3, dated July 2000.

Cornerstone: Mitigating Systems

The NRC performed a follow-up supplemental inspection to assess the licensees extent of condition evaluation associated with the October 23, 2002, failure of the high pressure core spray (HPCS) pump to start during routine surveillance testing. This failure occurred due to the licensees failure to adequately implement procedures during installation and inspection of the HPCS pump breaker from 1994 through October 23, 2002. This performance issue was previously characterized as having low to moderate risk significance (White) in the NRCs final significance determination letter dated March 4, 2003 (VIO 2002008-02). The failure to perform an adequate extent of condition evaluation was identified during the initial supplemental inspection and was considered a significant weakness in the licensees evaluation. This resulted in the White finding remaining open pending the licensees completion of the extent of condition evaluation and the NRCs inspection of the evaluation.

The inspector concluded during the follow-up supplemental inspection that the licensee had completed an adequate extent of condition evaluation. As a result, the White finding will be closed at the end of the fourth quarter 2003.

A. Inspector-Identified and Self-Revealed Findings No findings of significance were identified.

Licensee-Identified Violations

None.

Report Details 01

INSPECTION SCOPE

The NRC performed this follow-up supplemental inspection to assess the licensees extent of condition evaluation associated with the October 23, 2002, failure of the high pressure core spray (HPCS) pump to start during routine surveillance testing. This inspection focused on those elements of the first supplemental inspection that could not be closed. Specifically, the first supplemental inspection concluded that the extent of condition review was less than adequate. Therefore, this inspection evaluated the revised extent of condition review, the adequacy of additional corrective actions identified during the licensees revised review, and the licensees evaluation of the cause of the initial inadequate extent of condition review.

EVALUATION OF INSPECTION REQUIREMENTS Inspection requirements 02.01a - c and 02.02a - c of Inspection Procedure 95001 were completed and documented in the initial supplemental Inspection Report 50-440/2003007. Inspection requirements 02.02d and 02.03 were only partially completed at that time because the licensees initial extent of condition evaluation was incomplete. The results of the additional inspection for these requirements are documented below.

02.02 Root Cause and Extent of Condition Evaluation d.

Consideration of potential common cause(s) and extent of condition of the problem The inspector reviewed the licensees extent of condition review for maintenance procedures involving equipment or components that contain dual contact rotary switches with adjustable linkage. The licensees extent of condition review was conducted as part of Root Cause Analysis Report, Failure of the HPCS Pump to Start on Demand, Rev. 3.

The inspector noted that the licensee initially identified 52 procedures which required at least a screen with respect to adjustment criteria. Of the 52, seven were identified as needing further subject matter expert review. Upon completion of the necessary reviews, five procedures were changed.

The inspectors also reviewed the licensees evaluation of the organizational deficiencies associated with the inadequate extent of condition identified during the initial NRC supplemental inspection. The licensee determined that less than adequate organizational effectiveness in the timely and effective resolution of problems resulting in improper allocation of resources and less than adequate rigor applied to investigation and review to be the root cause of the deficiencies. Specifically, the individual assigned to perform the initial review was not experienced with the root cause process and was provided with little oversight or guidance. Additionally, as the subject matter expert, the individuals opinion as to root cause and extent of condition were not aggressively challenged nor independently reviewed. The licensee also identified corrective action program implementation weakness as a contributing cause. Specifically, the licensee identified that while the requirement to conduct a generic implications review was contained in the FENOC root cause process, little guidance was provided on conducting such reviews.

As an extent of condition review for the generic implications inadequacies, the licensee reviewed previously completed generic implication reviews associated with root and apparent cause investigations both at Perry and at Davis-Besse. The licensee properly concluded that the problems with performance of generic implications/extent of condition reviews went beyond the HPCS failure to start event.

02.03 Corrective Actions a.

Appropriateness of corrective action(s)

The licensee took immediate corrective actions to make the HPCS system operable.

After troubleshooting identified the cause of the failure, the switch was promptly adjusted and the pump successfully tested and returned to service.

The licensees initial corrective actions focused on 5kv cell switches which were erroneously considered to be the population of at-risk components. The licensee completed walkdowns of safety related and non-safety related 5kv switchgear to identify all cell switches that required adjustment and generated the appropriate work orders to accomplish the adjustments. Although not specifically identified in the formal corrective action statement, the licensee inspected all 5kv auxiliary switches while inspecting the cell switches. Several auxiliary switches were identified to be in need of adjustment.

Again, work documents were generated to perform the necessary adjustments.

In July 2003, during the initial NRC supplemental inspection, the licensee recognized the inadequacies in the initial extent of condition review. Corrective action was promptly initiated to walkdown 15kv switchgear. During these walkdowns, an additional ten auxiliary switches were identified as requiring adjustment. The licensee also initiated corrective action to re-accomplish the extent of condition review.

The licensee properly identified procedure adequacy as the root cause of the HPCS failure to start event. Corrective action was initiated to revise procedure GEI-0135, ABB Power Circuit Breakers 5kv Types 5HK250 and 5HK530 Maintenance, and train technicians on the revised procedure. As previously noted in the initial NRC supplemental inspection, training on the procedures was not expeditiously pursued. The inspector did, however, note the personal involvement of the subject matter expert in switch inspections and adjustments and determined that to be an effective interim compensatory action.

After expanding the initial extent of condition review, the licensee revised procedure GEI-0136, ABB Power Circuit Breakers 15kv Type 15HK1000 Maintenance, to correct procedure inadequacies identical to GEI-0135. Additionally, the licensee revised procedures GEI-0009, ABB Low Voltage Power Circuit Breaker Types K-600 & K-600S Through K-3000 & K-3000S Maintenance, GEI-0012, Inspection and Cleaning of Electrical Equipment, and SOI-R22, Metal Clad Switchgear 5 - 15kv, to provide enhanced switch inspection guidance.

The licensee also initiated corrective action to develop detailed guidance on the conduct of generic implications review. Training on the detailed guidance was scheduled for early 2004 at the time of this inspection.

b.

Prioritization of corrective actions The corrective actions taken by the licensee to specifically address switch adjustments were appropriately prioritized. The licensee prioritized safety related switchgear and scheduled the adjustments in a manner which minimized on-line risk. The actions were expanded to include non-safety 15kv switches after the initial NRC supplemental inspection identified they were not included in the initial extent of condition review. As previously noted, technician training on procedure revisions was not expeditiously pursued. Technician training was not completed until September 2003. The licensee used subject matter expert oversight as an interim compensatory measure for the training deficiency.

c. Establishment of schedule for implementing and completing the corrective actions At the time of this follow-up supplemental inspection, 50 corrective actions were identified and scheduled to address the HPCS pump failure to start event, with 43 completed. The large number of corrective actions was due, in part, to the licensees failure to produce an adequate initial root cause evaluation as identified by the licensee in July 2003 and the licensees failure to produce an adequate revision 1 root cause evaluation as identified by the NRC in July 2003. Additionally, the licensee chose to track adjustment of each safety-related cell or auxiliary switch with an individual corrective action. Finally, a 51st corrective action was added to address the inspectors concerns with the licensees corrective action effectiveness review, as discussed in Section 02.03d of this report.

Additionally, senior licensee management required a root cause evaluation of the organizations failure to perform an adequate extent of condition review prior to the NRCs initial supplemental inspection. The root cause evaluation and associated condition report (CR) identified 13 corrective actions, including a corrective action effectiveness review. The inspector reviewed the licensees schedule for action completion and concluded that it was appropriate. The inspector noted that interim guidance on extent of condition reviews was promptly generated after the initial NRC supplemental inspection.

d. Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensee conducted a corrective action effectiveness review in accordance with licensee procedure NOBP-LP-2007, CR Process Effectiveness Review, Rev. 1. The licensee concluded that the combination of procedure changes and technician training was effective in that 25 switches had been successfully adjusted since the HPCS failure to start event. The licensees review also noted that when technicians encountered problems with switch adjustments they requested assistance of the maintenance engineer.

The inspector determined licensee actions to properly adjust cell and auxiliary switches had improved equipment reliability. The inspector, however, had several concerns with the effectiveness review. Specifically:

• Technician training did not occur until September 2003. As a result, only a fraction of the 25 switch adjustments were performed by technicians trained to the current procedure. As such, the inspector questioned whether the corrective action had been adequately challenged to allow an effectiveness determination.

As recently as July 30, the licensees quality assurance organization had documented an example of technicians failing to follow the switch adjustment procedure even under the direct observation of the maintenance engineer.

• Licensee procedure NOBP-LP-2007 stated that while not mandatory, it is desirable that the effectiveness reviewer/performer be independent of the corrective action development activity. In this instance, the reviewer was the corrective action owner, the procedure writer, the individual who performed direct oversight of the switch adjustments as a compensatory measure for the delayed training, a participant in the technician training, and the author of the first two versions of the root cause evaluation.

• The fact that technicians stopped and requested assistance when problems were encountered was not a meaningful measure of corrective action effectiveness. It was a fundamental expectation of all nuclear workers. The fact that several assistance requests were required might be more indicative of the need for additional training or procedure guidance - it was not indicative of corrective action effectiveness.

In summary, the inspector concluded inadequate data existed to draw a meaningful conclusion as to the effectiveness of the corrective actions. Additionally, the inspector noted that the use of a more independent reviewer might detect problems or issues not evident to an individual so actively engaged in an activity. The licensee acknowledged the inspectors concerns and generated an additional corrective action to independently assess performance of subsequent switch adjustments to verify the effectiveness of the corrective actions.

03.

MANAGEMENT MEETINGS

Exit Meeting Summary

The inspector presented the inspection results to Mr. W. Kanda and other members of licensee management at the conclusion of the inspection on December 4, 2003. The licensee acknowledged the findings presented. No proprietary information was identified.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

W. Kanda, Vice President-Nuclear

P. Arthur, Manager, Work Control Section

M. Humphrey, Root Cause Coordinator, Work Control Section

D. Miller, Engineer, Compliance

V. Higaki, Manager, Regulatory Affairs

J. Lausberg, Supervisor, Compliance

T. Rausch, General Manager, Nuclear Power Plant Department

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

None.

Opened and Closed

None

Closed

50-440/2002008-02 VIO High Pressure Core Spray Pump Failure to Start 50-440/2003007-01 NCV Inadequate Identification of Extent of Condition Associated With High Pressure Core Spray Pump Failure to Start