IR 05000440/2008008
| ML081850183 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 07/02/2008 |
| From: | Jamnes Cameron NRC/RGN-III/DRP/B6 |
| To: | Bezilla M FirstEnergy Nuclear Operating Co |
| References | |
| IR-08-008 | |
| Download: ML081850183 (19) | |
Text
July 2, 2008
SUBJECT:
PERRY NUCLEAR POWER PLANT NRC SUPPLEMENTAL INSPECTION REPORT NO. 05000440/2008008
Dear Mr. Bezilla:
On May 16, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection at your Perry Nuclear Power Plant. The enclosed report documents the inspection results, which were discussed on May 16, 2008, with members of your staff.
The NRC performed this supplemental inspection to assess your evaluation of a White Performance Indicator (PI) in the Unplanned Scrams area of the Initiating Events cornerstone resulting from five scrams that occurred over a one-year period. We conducted this inspection in accordance with Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, and examined activities conducted under your license as they relate to safety, and compliance with the Commissions rules and regulations, and with the conditions of your license.
The objectives of this inspection were to: (1) provide assurance that the root causes and the contributing causes for the risk-significant performance issues associated with the White PI are understood; (2) provide assurance that the extent of condition and extent of cause of the issues are identified; and (3) provide assurance that corrective actions are sufficient to address the root causes and contributing causes, and to prevent recurrence.
Based on the results of this inspection, no findings of significance were identified and we concluded that you understood the root causes and contributing causes of the issues, that you identified the extent of condition and extent of cause of the issues, and that your corrective actions were sufficient to address the causes and to prevent recurrence of the issues. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS), accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA by Michael A. Kunowski for/
Jamnes L. Cameron, Chief Branch 6 Division of Reactor Projects Docket No. 50-440 License No. NPF-58 Enclosure: Inspection Report 05000440/2008008 w/Attachment: Supplemental Information cc w/encl: J. Hagan, President and Chief Nuclear Officer - FENOC J. Lash, Senior Vice President of Operations and Chief Operating Officer - FENOC D. Pace, Senior Vice President, Fleet Engineering - FENOC J. Rinckel, Vice President, Fleet Oversight - FENOC P. Harden, Vice President, Nuclear Support Director, Fleet Regulatory Affairs - FENOC Manager, Fleet Licensing - FENOC Manager, Site Regulatory Compliance - FENOC D. Jenkins, Attorney, FirstEnergy Corp.
Public Utilities Commission of Ohio C. OClaire, State Liaison Officer, Ohio Emergency Management Agency R. Owen, Ohio Department of Health
SUMMARY OF FINDINGS
IR 05000440/2008008; 05/13/2008 - 05/16/2008; Perry Nuclear Power Plant; Supplemental
Inspection; Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area.
The Perry Resident Inspector performed this supplemental inspection. No findings of significance were identified. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process,
Revision 4, dated December 2006.
Cornerstone: Initiating Events
The NRC performed this supplemental inspection to assess the licensees evaluation of a White Performance Indicator (PI) in the Unplanned Scrams area of the Initiating Events cornerstone resulting from five scrams that occurred during a 1-year period. This supplemental inspection was performed in accordance with Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area. The inspector concluded that: (1) the licensee understood the root causes and contributing causes of the risk-significant performance issues that resulted in the White PI; (2) the licensee identified the extent of condition and the extent of cause of the risk-significant issues; and (3) the licensees corrective actions were sufficient to address the root causes and contributing causes, and to prevent recurrence.
NRC-Identified and Self-Revealing Findings
No findings of significance were identified.
Licensee-Identified Violations
None.
REPORT DETAILS
INSPECTION SCOPE
This inspection was conducted in accordance with Inspection Procedure (IP) 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensees evaluation of one White Performance Indicator (PI) in the Initiating Events Cornerstone. The inspection objectives were to:
- Provide assurance that the root causes and contributing causes of risk-significant performance issues are understood;
- Provide assurance that the extent of condition and extent of cause of risk-significant issues are identified; and
- Provide assurance that licensee corrective actions to risk-significant performance issues are sufficient to address the root causes and contributing causes, and to prevent recurrence.
Perry notified the NRC in the first quarter of 2008 that the Initiating Event PI for Unplanned Scrams became White. By May 2008, the licensee notified the NRC that it had completed its evaluation of the circumstances surrounding the degraded performance and was ready for the NRC to assess the licensee's evaluation and subsequent corrective actions.
02.
EVALUATION OF INSPECTION REQUIREMENTS
.01 Problem Identification
a.
Determine that the root cause evaluation(s) identifies who (i.e., licensee, self-revealing, or NRC) and under what conditions the issue was identified.
On December 13, 2006, the licensee generated Condition Report (CR) 06-11339 identifying that an unplanned manual scram was inserted by the licensee due to loss of instrument air caused by an instrument air joint failure. The root cause evaluation concluded that the joint failure was due to poor construction workmanship and the licensees rejection for the operating experience recommendation of performing ultrasonic testing (UT) of joints in 1991.
On May 15, 2007, the licensee generated CR 07-20576 identifying that an unplanned automatic scram occurred due to a reactor pressure vessel (RPV) low level reactor protection system (RPS) signal caused by inadequate testing controls during tuning/testing of the digital feedwater control system (DFWCS). The root cause evaluation concluded that the licensee's current processes did not incorporate standards of excellence needed in performing testing evolutions.
On June 22, 2007, the licensee generated CR 07-22396 identifying the reactor recirculation (RR) pump 'B relay failure due to improper maintenance. The inspector determined that the manual scram was unplanned because the scram did not occur in accordance with the licensees normal shutdown sequence. The root cause evaluation concluded that there were less than adequate programmatic controls for maintenance associated with Agastat relays.
On June 27, 2007, the licensee generated CR 07-22706 identifying the failure of the RR 'A' pump motor. The inspector determined that the manual scram was unplanned because the scram did not occur in accordance with the licensees normal shutdown sequence. The root cause evaluation concluded that work management preventative maintenance (PM) process activities were not implemented on a timely basis. The effectiveness of the PM process and identifying gaps in the implementation were less than adequate. One of the process gaps not identified was the PM implementation for large motors.
On November 28, 2007, the licensee generated CR 07-30642 identifying that an unplanned automatic scram occurred due to RPV low level RPS signal caused by the failure of the DFWCS Input/Output (I/O) module power supplies. The root cause evaluation concluded there was less than adequate reliability of old style 400-FPS-24 power supplies due to a manufacturing defect of a subcomponent.
The inspector determined that the root cause evaluations adequately identified who and under what conditions the issue was identified.
b. Determine that the evaluation documents how long the issue existed, and prior opportunities for identification.
The first scram, in December 2006, was associated with an instrument air joint shear that was determined to have been caused by an as-built condition and the licensee determined that their inspection process would not have detected the flaw.
The second scram, in May 2007, involved a design software flaw of the new DFWCS installed in April 2007. The software flaw was introduced during development between the Phase 1 and Phase 2 installations. The licensee determined that their acceptance process would not have detected the error, but an adequate testing process would have detected the flaw without inducing a plant scram.
The third scram, in June 2007, was in response to securing the RR 'B' pump which resulted in the plant operating with one RR loop. This configuration was unplanned for the plant shutdown and the licensee conservatively inserted a manual scram. The licensee determined that the Agastat relay associated with the RR 'B' pump was incorrectly installed during the April/May 2007 refueling outage and no prior opportunities for identification existed.
The fourth scram, in June 2007, was associated the failure of the RR 'A' pump motor.
The licensee determined that the motor failure was due to operating the motor past its recommended life and the failure mechanism was a fault with the motor windings. The licensees PM program did not detect any motor degradation. The RR 'A' pump motor operated for 22 years compared to FENOCs replacement criteria of 20 years and Electric Power Research Institute (EPRI) recommendation of 15 years.
The fifth scram, in November 2007, was due to failure of both power supplies for the DFWCS I/O installed in 2005. The licensee determined that both failures were due to a subcomponent failure of a transformer. The licensee was not informed by the vendor of issues with these power supply models prior to the failures.
The inspector determined that the root cause evaluation adequately identified how long the issue existed and prior opportunities for identification.
c. Determine that the evaluation documents the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue.
The licensee's process performs the risk analysis for scram events in the Licensee Event Reports (LERs). The inspector reviewed the following LERs for plant-specific risk consequence.
LER 2006-005, "Decreasing Instrument Air Pressure Results in Manual Reactor Protection System Actuation," concluded that all plant safety systems responded as designed.
LER 2007-001, "Automatic Reactor Protection System Actuation Due to Reactor Coolant Level Decrease," concluded there was no risk-significant equipment out of service during the event.
LER 2007-007, "Reactor Recirculation Pump Failure Results in Manual Reactor Protection System Actuation," concluded that all risk-significant equipment was available during the pump failure.
LER 2007-008, "Single Recirculation Loop Operation Results in Planned Reactor Shutdown," concluded minimal plant risk because all risk-significant equipment was available during single loop operations.
LER 2007-004, "Automatic Reactor Protection System Actuation Due to Feedwater Control Power Supply Failure," calculated plant risk with the total loss of feedwater and reactor core isolation cooling. It concluded the event did not meet the threshold for a WHITE event.
Based upon the above documented observations, the inspector concluded that the licensee appropriately documented the risk consequences and compliance concerns associated with these issues.
.02 Root Cause and Extent of Condition
a. Determine that the problem was evaluated using a systematic method(s) to identify root cause(s) and contributing cause(s).
Licensee personnel completed a root cause evaluation of the circumstances surrounding the five scrams over the previous five quarters that resulted in the Unplanned Scrams PI crossing the Green-to-White threshold in the first quarter of 2008. The scrams were evaluated through a variety of systematic methods as documented in the following CRs:
CR 06-11339, Scram - Instrument Air Joint Failure Leading to Loss of Instrument Air A root cause evaluation was conducted using an event and causal factors chart, a barrier analysis, cause and linkage charting, and selected portions of Performance Improvement International (PII) organizational and programmatic deficiencies chart to investigate the December 13, 2006, instrument air joint failure. The PII methodology was used to evaluate human behaviors identified on the event and causal factor chart.
CR 07-20576, Reactor Scram During Digital Feedwater Control System Testing Under TXI-0373 A root cause evaluation was conducted using an event and causal factors chart, a barrier analysis, cause and linkage charting, and selected portions of PII organizational and programmatic deficiencies chart to investigate the May 15, 2007, low reactor water level scram.
CR 07-22396, Recirc Pump B Failed to Transfer to Slow Speed and Attempted to Start in Fast" A root cause evaluation was conducted using a barrier analysis, failure mode analysis, event and causal factors chart, and causal analysis for inappropriate actions leading to causal factors to investigate the June 22, 2007, failure of the
'B' RR pump to transfer to slow speed. The barrier analysis was used to identify weaknesses in the programmatic barriers of the licensee's processes.
CR 07-22706, Recirculation Pump A Trips After Receiving Recirc A Motor Lockout Annuciator A root cause evaluation was conducted using a management oversight and risk tree (MORT) analysis, event and causal factors, and chart and Why-staircase verification to investigate the June 29, 2007, trip of the RR 'A' motor. The chart and Why-staircase methodologies were used to confirm the causal factors and causes.
CR 07-30642, "Plant Scram on Wednesday, November 28, 2007" An apparent cause evaluation was conducted using failure mode analysis and component failure analysis methodologies to investigate the November 28, 2007, low reactor water level scram. These methodologies were used to initially identify the component failures of the digital feedwater power supplies. A root cause evaluation was conducted using a causal linkage, events and causal factors chart, barrier analysis, and PII methodology.
Based upon this, the inspector determined that the licensee's methods used to evaluate the root and contributing causes were adequate.
b. Determine that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.
CR 06-11339, Scram - Instrument Air Joint Failure Leading to Loss of Instrument Air In the root cause analysis, the licensee identified as a contributing cause the lack of program/process checks and balances to assure approved modifications and PM tasks are implemented as intended in a timely manner.
CR 07-20576, Reactor Scram During Digital Feedwater Control System Testing Under TXI-0373 The contributing causes identified with the May 15, 2007, digital feedwater scram were determined to be less than adequate test controls, less than adequate design configuration of the product from the vendor, and the licensee's owners' acceptance process was inconsistent with expectations for receiving a quality product.
CR 07-22396, Recirc Pump B Failed to Transfer to Slow Speed and Attempted to Start in Fast" The licensee identified a contributing cause for the RR 'B' failure to be less than adequate implementation of testing controls to ensure proper installation of equipment.
CR 07-22706, Recirculation Pump A Trips After Receiving Recirc A Motor Lockout Annuciator The licensee identified a contributing cause in that the Plant Health Committee accepted less than adequate risk assessment of the large motor program when the refurbishment of the motor was delayed, long range equipment/component planning is not integrated into the budget process, and the motor failure was an age-related failure of the stator.
CR 07-30642, "Plant Scram on Wednesday, November 28, 2007" The contributing causes identified by the licensee for the November 28, 2007, digital feedwater scram were the depth of the site's resources, expertise, and knowledge of digital control systems required improvement, and less than adequate use of the corrective action program to identify lower level issues.
Based upon the work performed for these root causes, the inspector concluded that the root cause evaluations were conducted to a level of detail commensurate with the significance of the problems.
c. Determine that the root cause evaluation included a consideration of prior occurrences of the problem and knowledge of prior operating experience.
The inspector determined that the root cause evaluation adequately included consideration of prior occurrences of the problem and knowledge of prior operating experience.
In its root cause analysis, the licensee conducted a detailed review of previous scrams and operating experience. The licensee determined that instrument air leaks were an industry issue in the early 1990's and performed additional inspections for a period of time. The two reactor recirculation pump issues were new to Boiling Water Reactor (BWR)-6 plants. The digital feedwater modification was performed to increase reliability of the feedwater control system based on industry experience and the dual failure of power supplies for feedwater controls did not exist in any operating experience database.
d. Determine that the root cause evaluation addresses the extent of condition and the extent of cause of the problem.
The following specific extent-of-condition and extent-of-cause reviews were conducted for each of the scrams:
CR 06-11339, Scram - Instrument Air Joint Failure Leading to Loss of Instrument Air Following the identification of the pipe joint failure, the licensee performed UT inspections on 160 similar joints. The licensee continued to inspect 2 to 3-inch diameter joints for proper engagement since initial testing in December 2006.
CR 07-20576, Reactor Scram During Digital Feedwater Control System Testing Under TXI-0373 Following the identification of the software design issue, the licensee engaged with the vendor, Foxsboro, and performed additional design review. The licensee reviewed its testing process and instituted procedural enhancements to the process.
CR 07-22396, Recirc Pump B Failed to Transfer to Slow Speed and Attempted to Start in Fast" Following the pump failure, the licensee examined all Agastat relays installed during the April/May 2007 refueling outage using the acceptance criteria identified by the procedure, GEI-0014, "Timing Relays Agastat Type 7012/7022, 7014/7024, and 2412/2422 Series," Revision 3, used during the refueling outage.
After the re-inspection, the licensee determined the cause to be an improperly installed relay and the procedure provided inadequate acceptance criteria.
The inspector identified that the extent-of-condition review was not performed with the updated acceptance criteria identified by the root cause team in GEI-0014, Revision 4. The licensee initiated CR 08-40410, "NRC 95001 Observation on CR 07-22396," to address this issue and provided analysis for acceptable extent of condition of the 11 Agastat relays.
CR 07-22706, Recirculation Pump A Trips After Receiving Recirc A Motor Lockout Annuciator The licensee initiated PM templates for replacement and refurbishment of other large motors that exceeded the recommended replacement schedule. These motors include the RR 'A' pump, high pressure core spray pump, low pressure core spray pump, residual heat removal pump, and emergency service water pumps.
CR 07-30642, "Plant Scram on Wednesday, November 28, 2007" The extent-of-condition and extent-of-cause investigation revealed other feedwater control power supplies of the same model. These power supplies were for the reactor feedwater pump turbine controls and the digital feedwater support system power supplies.
The inspector determined that the root cause evaluations adequately addressed the extent of condition and the extent of cause for each of the problems.
e. Determine whether the licensees root cause evaluation, extent of condition, and extent of cause appropriately considered the safety culture components as described in Inspection Manual Chapter (IMC) 0305.
The inspector determined that the licensees process for performing root cause, extent of condition, and extent of cause evaluations did not include consideration of the safety culture components as described in IMC 0305. The licensee plans to implement this consideration into their causal analysis process.
.03 Corrective Actions
a. Determine that appropriate corrective actions are specified for each root cause, or that there is an evaluation that no actions are necessary.
The inspector determined that appropriate corrective actions were specified for each root cause.
The following specific corrective actions were implemented for each of the scrams:
CR 06-11339, Scram - Instrument Air Joint Failure Leading to Loss of Instrument Air The licensee performed UT inspections on a large sample of accessible joints after the event, and performed additional UT inspections during the April/May 2007 refueling outage. The licensee instituted an inspection plan to perform periodic UT examinations on the large instrument air line joints.
CR 07-20576, Reactor Scram During Digital Feedwater Control System Testing Under TXI-0373" The licensee revised the temporary instruction procedure to properly delineate entry, exit, and suspension criteria to permanent operator instructions. The licensee updated its fleet instruction, NOP-CC-2003, "Engineering Changes," to provide adequate acceptance criteria for outsourced engineering design changes. The root cause team requested the development of a Conduct of Test Procedure to provide overall guidance for the different types of testing protocols at the site.
CR 07-22396, Recirc Pump B Failed to Transfer to Slow Speed and Attempted to Start in Fast" The licensee revised the installation procedure GEI-0114 to include a visual representation of the proper installation of an Agastat relay. The licensee instituted the lessons learned into both the technicians' certification for Agastat relays and the electrical engineers' training curriculum.
CR 07-22706, Recirculation Pump A Trips After Receiving Recirc A Motor Lockout Annuciator The licensee instituted a review of the other large motors to ensure timely scheduling of motor replacement and refurbishment. One corrective action was to require all Plant Health Committee members to review and understand the FENOC Value Rating System to ensure proper prioritization of component replacement. The licensee plans to incorporate maintenance of the large motors into their business processes to ensure completion of all PM requirements.
CR 07-30642, "Plant Scram on Wednesday, November 28, 2007" The licensee replaced all feedwater control power supplies with an updated model. This updated model from the vendor, Foxsboro, had a higher reliability rate than the earlier model. An added design feature by the licensee was to install a third diverse power supply from a different vendor, Lambda, to increase reliability of the feedwater control systems. The licensee had onsite four additional Foxsboro and two Lambda power supplies standing by for immediate replacement.
The inspector concluded that the corrective actions specified were appropriate to prevent recurrence.
b. Determine that the corrective actions have been prioritized with consideration of the risk significance and regulatory compliance.
The inspector determined that the licensee adequately prioritized the corrective actions with consideration of the risk significance and regulatory compliance.
Corrective actions associated with the scrams have been completed, with the exception of replacement of the RR 'B' pump scheduled for the next scheduled refueling outage.
One corrective action associated with the May 2007 digital feedwater scram was to develop a Conduct of Test Procedure, which was deemed a high priority task to be completed by the licensee's root cause team. The inspector determined that the corrective action was not completed and not approved by the licensee's Corrective Action Review Board as an alternate closure. The licensee wrote CR 08-40395, "Root Cause Corrective Action 07-20576-8 Not Implemented as Written," to address this issue.
The inspector concluded that the corrective actions had been prioritized with consideration of the risk significance and regulatory compliance.
c. Determine that a schedule has been established for implementing and completing the corrective actions.
The inspector determined that the licensee adequately established a schedule for implementing and completing the corrective actions.
d. Determine that quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence.
The inspector determined that the licensee adequately developed quantitative or qualitative measures of success for determining effectiveness of the corrective actions to prevent recurrence.
The licensee, as part of the corrective actions, scheduled effectiveness reviews for the causes of the five scrams to assure these performance issues have been adequately addressed to prevent recurrence.
03. MANAGEMENT MEETINGS
Exit Meeting Summary
The inspector presented the inspection results to Mr. Grabnar and other members of licensee management on May 16, 2008. The licensee representatives acknowledged the issues presented. The licensee did not identify any materials examined during the inspection and proposed for inclusion in this inspection report as proprietary in nature.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee
- J. Rinckel, Vice President, Oversight, FENOC
- M. Bezilla, Vice President, Perry
- K. Krueger, Plant General Manger, Perry
- F. Cayia, Director, Performance Improvement, Perry
- J. Grabnar, Director, Engineering, Perry
- H. Harlan, Director, Work & Outage Management, Perry
- K. Cimorelli, Director, Maintenance, Perry
- M. Koberling, Manger, System Engineering, Perry
- D. Evans, Manager, Operations, Perry
- J. Lausberg, Manager, Regulatory Compliance, Perry
- T. Hilston, Manager, Design Engineering, Perry
LIST OF ITEMS
OPENED, CLOSED AND DISCUSSED
Opened
None.
Opened and Closed
None.
Discussed
None.
Attachment