IR 05000440/2007007

IR 05000440-07-007 on 1/3/2007 - 1/5/2007 for Perry Nuclear Power Plant; Supplemental Inspection; Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area.

ML070310550
Person / Time
Site:	Perry
Issue date:	01/25/2007
From:	Satorius M Division Reactor Projects III
To:	Pearce L FirstEnergy Nuclear Operating Co
References
IR-07-007
Download: ML070310550 (19)

Text

ary 25, 2007

SUBJECT:

PERRY NUCLEAR POWER PLANT NRC SUPPLEMENTAL INSPECTION REPORT 05000440/2007007

Dear Mr. Pearce:

On January 5, 2007, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection at your Perry Nuclear Power Plant. The enclosed report documents the inspection results, which were discussed on January 5, 2007, with you and other members of your staff.

The NRC performed this supplemental inspection to assess your evaluation of a White Mitigating Systems Performance Index (MSPI) in the Emergency Alternating Current Power System area of the Mitigating Systems cornerstone resulting from four Emergency Diesel Generator failures that occurred over a 3 year period. We conducted this inspection in accordance with Inspection Procedure 95001, Inspection For One Or Two White Inputs In A Strategic Performance Area, and examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

The objectives of this inspection were to: (1) provide assurance that the root causes and the contributing causes for the risk significant performance issues associated with the White MSPI are understood; (2) provide assurance that the extent of condition and extent of cause of the issues are identified; and (3) provide assurance that corrective actions are sufficient to address the root causes and contributing causes, and to prevent recurrence.

Based on the results of this inspection, no findings of significance were identified and we concluded that you understood the root causes and contributing causes of the issues, that you identified the extent of condition and extent of cause of the issues, and that your corrective actions were sufficient to address the causes and to prevent recurrence of the issues. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS), accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Mark A. Satorius, Director Division of Reactor Projects Docket No. 50-440 License No. NPF-58 Enclosure: Inspection Report 05000440/2007007 w/Attachment: Supplemental Information cc w/encl: G. Leidich, President and Chief Nuclear Officer - FENOC J. Hagan, Senior Vice President of Operations and Chief Operating Officer - FENOC D. Pace, Senior Vice President, Fleet Engineering - FENOC J. Rinckel, Vice President, Fleet Oversight R. Anderson, Vice President, Nuclear Support Director, Fleet Regulatory Affairs Manager, Fleet Licensing Manager, Site Regulatory Compliance D. Jenkins, Attorney, FirstEnergy Public Utilities Commission of Ohio Ohio State Liaison Officer R. Owen, Ohio Department of Health

SUMMARY OF FINDINGS

IR 05000440/2007007; 1/3/2007 - 1/5/2007; Perry Nuclear Power Plant; Supplemental

Inspection; Inspection Procedure 95001, Inspection For One Or Two White Inputs In A Strategic Performance Area.

The Davis-Besse Resident Inspector and a Senior Reactor Analyst from Region III performed this supplemental inspection. No findings of significance were identified. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Cornerstone: Mitigating Systems

The NRC performed this supplemental inspection to assess the licensees evaluation of a White Mitigating Systems Performance Index (MSPI) in the Emergency Alternating Current Power System area of the Mitigating Systems cornerstone resulting from four Emergency Diesel Generator start failures that occurred over a 3 year period. This supplemental inspection was performed in accordance with Inspection Procedure 95001, Inspection For One Or Two White Inputs In A Strategic Performance Area. The inspectors concluded that: (1) the licensee understood the root causes and contributing causes of the risk significant performance issues that resulted in the White MSPI; (2) the licensee identified the extent of condition and the extent of cause of the issues; and (3) the licensees corrective actions were sufficient to address the root causes and contributing causes, and to prevent recurrence.

A. Inspector-Identified and Self-Revealed Findings No findings of significance were identified.

Licensee-Identified Violations

None.

REPORT DETAILS

INSPECTION SCOPE

The NRC performed this supplemental inspection to assess the licensees evaluation of a White Mitigating Systems Performance Index (MSPI) in the Emergency AC [Alternating Current]

Power System area of the Mitigating Systems cornerstone resulting from four Emergency Diesel Generator (EDG) failures that occurred over a 3 year period. The licensee determined that the Emergency AC Power System MSPI crossed the Green-to-White threshold in the 2nd quarter of 2006 and returned to Green in the 3rd quarter of 2006.

EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.

Determine that the root cause evaluation identifies who (i.e., licensee, self-revealing, or NRC) and under what conditions the issue was identified.

The inspectors determined that the root cause evaluation adequately identified who and under what conditions the issue was identified.

During the performance of Temporary Instruction (TI) 2515/169, MSPI Verification, the Perry resident inspectors identified a number of errors in the performance data used to develop the MSPI basis document and calculate the MSPIs. Errors that were identified included the number of hours the reactor was critical, the number of hours of safety system unavailability, and the crediting of operator recovery actions. Subsequently, as part of an effort to understand if any errors existed beyond those identified by the resident inspectors, licensee personnel reviewed areas that were not within the scope of TI 2515/169, but could impact the MSPIs. During this effort, licensee personnel identified an error in the input calculations to the Probabilistic Risk Assessment (PRA)that were used to develop the MSPIs reported for the 2nd and 3rd quarters of 2006.

On November 29, 2006, the NRC staff received FENOC letter PY-CEI/OIE-0681L, Mid-Quarter 2006 Correction for MSPI NRC Performance Indicator (PI) Data Files for 2nd and 3rd Quarters, 2006, dated November 28, 2006, which notified the NRC that FirstEnergy Nuclear Operating Company (FENOC) submitted a mid-quarter correction to the Perry PI data files for the 2nd and 3rd quarters of 2006. This letter also notified the NRC that as a result of this correction, the 2nd quarter 2006 Emergency AC Power System MSPI that was originally categorized as Green was re-categorized as White, but returned to Green in the 3rd quarter of 2006.

The licensees evaluation of the White Emergency AC Power System MSPI included an evaluation of four EDG failures that occurred over a 3 year period and resulted in the MSPI crossing the Green-to-White threshold, as well as the circumstances that delayed the identification of this White MSPI until the 4th quarter of 2006.

b.

Determine that the evaluation documents how long the issue existed, and prior opportunities for identification.

The inspectors determined that the root cause evaluation adequately identified how long the issue existed and prior opportunities for identification.

With regard to how long the issue existed, the Emergency AC Power System MSPI was categorized as White for the 2nd quarter of 2006 as a result of four EDG failures that occurred between July 2003 and November 2005. Because the 2nd quarter 2006 MSPI report was the first time that PIs were reported using the MSPI system, although the EDG failures occurred between July 2003 and November 2005, the 2nd quarter 2006 report was the licensees first opportunity to identify that a White MSPI existed.

With regard to prior opportunities for identification, the White Emergency AC Power System MSPI that existed in the 2nd quarter of 2006 was not properly identified and reported as White until the 4th quarter of 2006, although the White MSPI should have been identified in the 2nd quarter 2006 MSPI report. This White MSPI was identified during a licensee extent of condition and extent of cause effort that was conducted after the resident inspectors identified errors in MSPI data during the performance of TI 2515/169.

c. Determine that the evaluation documents the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue.

The inspectors determined that the licensees evaluation adequately documented the plant-specific risk consequences and compliance concerns associated with the White Emergency AC Power System MSPI that was reported for the 2nd quarter of 2006.

The MSPI process was established to determine the cumulative significance of equipment failures and unavailability for selected safety-related and/or risk-significant systems during a monitored time period. In the case of the Emergency AC Power System MSPI, the unavailability and unreliability of the licensees EDGs was monitored over 12 quarters. Because the MSPI was a means to monitor the performance of selected systems based on their ability to perform risk-significant functions, the identification that the Emergency AC Power System MSPI had crossed the Green-to-White threshold was, by definition, an increase in plant risk.

As documented in condition report (CR) 06-10069, NRC PI For Emergency AC Crossed the Threshold from Green to White, the Emergency AC Power System MSPI initially calculated and submitted to the NRC for the 2nd quarter of 2006 was 1.0 x 10- 6, and was therefore characterized as Green The revised Emergency AC Power System MSPI that resulted from changes to the PRA calculations was 1.1 x 10-6. The difference between the original and revised MSPI values (1.0 x 10-7) quantitatively represented only a slight increase in plant risk, and was not considered to be significant. Therefore, although the revised Emergency AC Power System MSPI was characterized as White, the specific risk consequences of this increase were also not significant.

With regard to compliance concerns, as documented in NRC Inspection Report 05000440/2006005, materially inaccurate and incomplete information was identified in the MSPI data submitted to the NRC on July 21, 2006. On October 13, 2006, during a review of the MSPI program in response to NRC-identified errors in the MSPI basis document, licensee personnel identified that incorrect data was submitted to the NRC in the 2nd and 3rd quarter 2006 MSPI reports due to an error associated with the incorrect use of an outdated PRA calculation. On November 16, 2006, the licensee submitted corrected data that resulted in the 2nd quarter 2006 Emergency AC Power System MSPI crossing the Green-to-White threshold. The inaccurate information provided on July 21, 2006, affected the color of the Emergency AC Power System MSPI and therefore affected the timeliness of the NRCs response to the White MSPI. As a result, the NRC determined that a Severity Level IV violation of 10 CFR 50.9, Completeness and Accuracy of Information, occurred. This violation was evaluated in accordance with the Enforcement Policy and enforcement discretion was exercised pursuant to Section VII.B.6, Violations Involving Special Circumstances, of the Enforcement Policy to refrain from issuing a Notice of Violation (EA-06-326).

02.02 Root Cause and Extent of Condition a.

Determine that the problem was evaluated using a systematic method(s) to identify root cause(s) and contributing cause(s).

The inspectors determined that the licensees evaluation of the four EDG failures that resulted in the White Emergency AC Power System MSPI, as well as the circumstances that delayed the identification of this White MSPI until the 4th quarter of 2006, was adequately evaluated using systematic methods to identify root and contributing causes.

Licensee personnel completed a root cause evaluation that reviewed the circumstances surrounding the four EDG failures over the previous 12 quarters that resulted in the Emergency AC Power System MSPI crossing the Green-to-White threshold in the 2nd quarter of 2006. The EDG failures were evaluated through a variety of systematic methods as documented in the following condition reports:

• CR 05-07531, Division 1 DG [Diesel Generator] Output Breaker Failed to Cycle During PMT [Post-Maintenance Testing]

An apparent cause evaluation was conducted using problem solving flowchart and event and causal factors charting methodologies to investigate a November 9, 2005, failure of the Division 1 EDG output breaker to close during a PMT activity. Licensee personnel determined that an actuating arm spring retaining pin had been dislodged sometime between the removal of the breaker for breaker cubicle work and the subsequent re-installation, which caused the failure.

• CR 05-00936, Division 2 DG Failed to Quick Start for SVI-R43-T1348" A root cause evaluation was conducted using fault tree analysis and failure mode and effects analysis methodologies to investigate a February 7, 2005, failure of the Division 2 EDG to start within 5 minutes after completion of a 24-hour rated load run. The investigation determined that the de-energized position of the governor sub-assembly pilot valve plunger unexpectedly changed due to internal bias drift and thermal growth, which caused the failure. This was an issue that the industry and vendor had not previously recognized to exist. As a result, a 10 CFR Part 21 report was generated to identify this new failure mode.

• CR 04-04154, Division 2 DG Inoperable Due to Jacket Water Leak An apparent cause evaluation was conducted using failure mode analysis and causal analysis methodologies to investigate an August 11, 2004, Division 2 EDG jacket water coupling leak that rendered the EDG inoperable and unavailable. The jacket water coupling leakage was attributed to the installation of a damaged jacket water coupling. The coupling was removed for EDG maintenance, and when the coupling was re-installed, the damaged condition was not identified prior to installation.

• CR 03-04374, Division 2 DG Unplanned Inoperability and Unavailability A root cause evaluation was conducted using failure mode analysis and causal analysis methodologies to investigate a July 23, 2003, failure of the Division 2 EDG to start. Licensee personnel determined that a contact associated with the EDG starting circuitry remained closed when it should have opened after the EDG was shut down following testing. This resulted in the exciter field breaker remaining in the closed position resulting in EDG component failures, rendering the EDG inoperable. Laboratory testing of the electronic module that housed this contact identified that a faulty microchip resulted in the failure of the starting circuitry contact to open.

The inspectors reviewed the four evaluations described above and concluded that the systematic methodologies that were utilized were appropriate and that these evaluations adequately addressed the individual equipment and organizational issues that were identified to exist. There were no additional EDG failures following the implementation of corrective actions to address identified root and contributing causes. The inspectors also concluded that the licensee had properly determined that no common themes between the events existed.

To identify the root and contributing causes that resulted in the delayed identification of the White Emergency AC Power System MSPI, licensee personnel utilized event and causal factors charting, barrier analysis, and performance improvement international failure mode charting methodologies. The inspectors determined that these systematic methodologies were also appropriate.

b. Determine that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

The inspectors conducted reviews of licensee documents and interviews with station personnel and determined that the root cause evaluation was conducted to an adequate level of detail commensurate with the significance of the issue.

The inspectors determined that the licensee investigations of the four EDG failures that resulted in the White Emergency AC Power System MSPI were sufficiently detailed and adequately identified the root causes of the failures and prescribed appropriate corrective actions to prevent reccurrence.

The inspectors determined that the licensees root cause evaluation of the delayed identification of the White Emergency AC Power System MSPI was also sufficiently detailed, adequately identified root causes, and prescribed appropriate corrective actions.

c. Determine that the root cause evaluation included a consideration of prior occurrences of the problem and knowledge of prior operating experience.

The inspectors determined that the root cause evaluation included adequate consideration of prior occurrences of the problem and knowledge of prior operating experience.

The inspectors determined that licensee personnel conducted a detailed review of previous EDG generator reliability issues that extended back to 1998. Licensee personnel reviewed conditions reports to identify and categorize previously identified reliability issues. Through document reviews and interviews with licensee personnel, the inspectors determined that licensee personnel had properly reviewed the FENOC database for previous occurrences. Licensee personnel also adequately reviewed industry operating experience to identify similar events.

d. Determine that the root cause evaluation addresses the extent of condition and the extent of cause of the problem.

The inspectors determined that the root cause evaluation adequately addressed the extent of condition and the extent of cause of the issue.

The following specific extent of condition and extent of cause reviews were conducted for each of the EDG failures:

• CR 05-07531, Division 1 DG Output Breaker Failed to Cycle During PMT Following the identification that an actuating arm spring retaining pin had been dislodged sometime between the removal and re-installation of the Division 1 EDG output breaker, licensee personnel conducted a comprehensive extent of condition review. This review included breakers installed in the plant, breakers stored in the warehouse, and breakers undergoing refurbishment. Following this review, the licensee concluded that no additional problems existed.

• CR 05-00936, Division 2 DG Failed to Quick Start for SVI-R43-T1348" Following the identification that the de-energized position of the governor sub-assembly pilot valve plunger had unexpectedly changed due to internal bias drift and thermal growth, licensee personnel conducted an extent of condition and extent of cause review. The scope of this review included the HPCS EDG, the RCIC system, and the feedwater turbine control system. These systems were selected since they potentially utilized electro-hydraulic controls that were similar to the Division 2 EDG controls. Following that review, licensee personnel determined that although the HPCS EDG and feedwater turbine control system were not susceptible to the same failure mechanism, the RCIC system failed to meet surveillance testing acceptance criteria in May 2002 due to a turbine governor null voltage shift. Corrective actions at the time included a revision to surveillance testing procedures to record and adjust the null voltage, as necessary.

• CR 04-04154, Division 2 DG Inoperable Due to Jacket Water Leak The licensees extent of condition and extent of cause review concluded that this issue was an isolated event. A significant jacket water leak that was identified at another plant was attributed to improper installation of the coupling and coupling gasket and was therefore not related to the Perry failure.

• CR 03-04374, Division 2 DG Unplanned Inoperability and Unavailability Following the identification that a contact in an electronic module associated with the EDG starting circuitry remained closed when it should have opened, licensee personnel reviewed the extent of condition of the applications as they applied to these electronic modules. Following that review, licensee personnel determined that no other failures had occurred at Perry or at any other plant in the industry due to a similar cause and that the event was isolated.

The inspectors reviewed the evaluations associated with each of the four EDG failures that resulted in the White Emergency AC Power System MSPI and determined that the licensee had performed an adequate extent of condition and extent of cause evaluation for each of the failures.

In addition, following the identification of the errors in the performance data used to calculate the MSPIs by the resident inspectors, licensee personnel reviewed areas that were not within the scope of TI 2515/169, but could impact the MSPIs. During this review effort, licensee personnel identified an error in the input calculations to the PRA that were used to develop the MSPIs reported for the 2nd and 3rd quarters of 2006 and determined that the Emergency AC Power System MSPI for the 2nd quarter of 2006 that was initially reported as Green should have been reported as White. Based on the licensees identification of this error as well as the results of the remainder of the resident inspector reviews of performance data after the errors were identified and following the implementation of corrective actions, the inspectors concluded that the licensee had adequately addressed the extent of condition and extent of cause for the late identification of the White MSPI.

02.03 Corrective Actions a.

Determine that appropriate corrective action(s) are specified for each root cause, or that there is an evaluation that no actions are necessary.

The inspectors determined that appropriate corrective actions were specified for each root cause.

The following specific corrective actions were implemented for each of the EDG failures:

• CR 05-07531, Division 1 DG Output Breaker Failed to Cycle During PMT Following the identification that an actuating arm spring retaining pin had been dislodged sometime between the removal and re-installation of the breaker, licensee personnel completed an examination of similar breakers and verified that no additional problems existed. Licensee personnel also revised procedures associated with the removal, installation, and inspection of breakers to include a verification that the actuating arm spring retaining pin was in place.

• CR 05-00936, Division 2 DG Failed to Quick Start for SVI-R43-T1348" Following the identification that the de-energized position of the governor sub-assembly pilot valve plunger had unexpectedly changed due to internal bias drift and thermal growth, licensee personnel revised EDG preventive maintenance procedures to add null voltage bands, acceptable operation criteria, and criteria for the notification of the EDG system engineer. In addition, licensee personnel revised EDG surveillance testing procedures to record the null voltage, prescribe acceptance criteria, and review present and previously recorded readings to identify adverse trends. Licensee personnel also generated Engineering Change Request 04-0049 to upgrade the EDG governors with a new proportional governor system to be implemented within the next two refueling outages.

• CR 04-04154, Division 2 DG Inoperable Due to Jacket Water Leak Following the determination that a damaged jacket water coupling was not identified during the inspection of the coupling prior to installation, licensee personnel replaced the damaged coupling and trained maintenance personnel involved with EDG maintenance activities. This training included a review of the event as well as a review of proper handling, storage and visual inspection of components prior to removal and/or installation. In addition, the licensee subsequently replaced the Division 1 and Division 2 jacket water couplings with a more robust model.

• CR 03-04374, Division 2 DG Unplanned Inoperability and Unavailability Following the identification that a contact associated with the EDG starting circuitry remained closed when it should have opened, the electronic module which housed this contact was subsequently replaced with a spare. In addition, licensee personnel revised EDG operating procedures to monitor components to ensure the field breaker opens, and therefore the contact opens, when the EDG is shut down.

The inspectors determined that the corrective actions identified above were appropriate to address each of the individual EDG failures.

To address the root causes associated with the delayed identification of the White Emergency AC Power System MSPI, the following corrective actions were planned and/or completed:

(1) performance of a gap analysis to compare the current licensee PRA model with the American Society of Mechanical Engineers PRA Standard Capability Categories using applicable methods prescribed by Regulatory Guide 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessments for Risk-Informed Activities;

(2) a revision to the MSPI Basis document using the initiating event frequencies documented in calculation DB-004, Revision 3; and

(3) an improvement in the administrative controls to ensure the completeness and accuracy of MSPI data by revision of applicable procedures. The inspectors reviewed these corrective actions and determined that they were adequate to address the issue.

b. Determine that the corrective actions have been prioritized with consideration of the risk significance and regulatory compliance.

The inspectors determined that the corrective actions were adequately prioritized with consideration of risk significance and regulatory compliance.

All corrective actions associated with the previous EDG failures had been completed, with the exception of EDG governor replacement activities that were scheduled to be completed within the next two refueling outages. However, interim procedure steps had been established to ensure that until the EDG governor replacements were accomplished, appropriate actions were in place to prevent the previous problem from recurring. In addition, the corrective actions associated with the delayed identification of the White Emergency AC Power System MSPI had either been completed or were planned for completion by June 2007. The inspectors concluded that these corrective actions had been properly prioritized.

c. Determine that a schedule has been established for implementing and completing the corrective actions.

The inspectors determined that an acceptable schedule was established for the implementation and completion of corrective actions in accordance with the licensees corrective action program. With the exception of Division 1 and Division 2 EDG governor modifications and some MSPI implementation corrective actions, all other corrective actions had been accomplished. For the actions that remained to be completed, the inspectors determined that an appropriate schedule had been established.

d. Determine that quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence.

The inspectors determined that adequate qualitative and quantitative measures of success were developed for determining the effectiveness of corrective actions to prevent recurrence.

The licensee had conducted effectiveness reviews for the previously identified EDG failures to assure these performance issues had been adequately addressed to prevent recurrence. The inspectors reviewed the completed effectiveness reviews for the EDG failures and concluded that the corrective actions were effective.

03 MANAGEMENT MEETINGS

Exit Meeting Summary

The inspectors presented the inspection results to Mr. Pearce and other members of licensee management and staff at the conclusion of the inspection on January 5, 2007.

The licensee acknowledged the information presented. The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

G. Leidich, Chief Nuclear Officer, FENOC

D. Pace, Senior Vice President, Fleet Engineering and Services, FENOC

J. Hagan, Chief Operating Officer, FENOC

J. Rinckel, Vice President, Oversight, FENOC

L. Pearce, Vice President, Perry

B. Allen, Director, Operations, Perry

F. Cayia, Director, Performance Improvement, Perry

G. Halnon, Director, Performance Improvement Initiative, Perry

J. Shaw, Director, Engineering, Perry

M. Wayland, Director, Maintenance, Perry

M. Loberling, Manger, System Engineering, Perry

D. Evans, Manager, Operations, Perry

J. Lausberg, Manager, Regulatory Compliance, Perry

LIST OF ITEMS

Opened

None.

Opened and Closed

None.

Discussed

None.

Attachment