IR 05000361/2004017
| ML050040262 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 01/03/2005 |
| From: | Kennedy K NRC/RGN-IV/DRP/RPB-C |
| To: | Ray H Southern California Edison Co |
| References | |
| IR-04-017 | |
| Download: ML050040262 (17) | |
Text
January 3, 2005
SUBJECT:
SAN ONOFRE NUCLEAR GENERATING STATION - NRC SUPPLEMENTAL INSPECTION REPORT 05000361/2004017 AND 05000362/2004017
Dear Mr. Ray:
On December 3, 2004, the NRC completed a supplemental inspection at your San Onofre Nuclear Generating Station, Units 2 and 3, facility. The enclosed report documents the inspection findings which were discussed on December 3, 2004, with Mr. J. Wambold and other members of your staff.
As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure 95001. The purpose of the inspection was to examine the causes for and actions taken related to the performance indicator for unplanned scrams with loss of normal heat removal crossing the threshold from Green (very low risk significance) to White (low to moderate risk significance) for Unit 2. This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the events resulting in the White performance indicator are understood, to independently assess the extent of condition, and to provide assurance that the corrective actions for risk significant performance issues are sufficient to address the root causes and contributing causes and to prevent recurrence. The inspection consisted of selected examination of representative records and interviews with personnel.
The NRC concluded that your staff performed thorough evaluations for each of the three Unit 2 reactor trips with loss of normal heat removal and performed a broad-based self-assessment to identify any performance and process issues that should be addressed as a result of the performance indicator crossing the threshold from Green to White.
In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Southern California Edison Co.
- 2 -
Should you have any questions concerning this inspection, we will be pleased to discuss them with you.
Sincerely,
/RA/ by WDJohnson for Kriss M. Kennedy, Chief Project Branch C Division of Reactor Projects Dockets: 50-361 50-362 Licenses: NPF-10 NPF-15 Enclosure:
NRC Inspection Report 05000361/2004017 and 05000362/2004017 cc w/enclosure:
Chairman, Board of Supervisors County of San Diego 1600 Pacific Highway, Room 335 San Diego, CA 92101 Gary L. Nolff Power Projects/Contracts Manager Riverside Public Utilities 2911 Adams Street Riverside, CA 92504 Eileen M. Teichert, Esq.
Supervising Deputy City Attorney City of Riverside 3900 Main Street Riverside, CA 92522 Joseph J. Wambold, Vice President Southern California Edison Company San Onofre Nuclear Generating Station P.O. Box 128 San Clemente, CA 92674-0128
Southern California Edison Co.
- 3 -
David Spath, Chief Division of Drinking Water and Environmental Management California Department of Health Services P.O. Box 942732 Sacramento, CA 94234-7320 Michael R. Olson San Onofre Liaison San Diego Gas & Electric Company P.O. Box 1831 San Diego, CA 92112-4150 Ed Bailey, Chief Radiologic Health Branch State Department of Health Services P.O. Box 997414 (MS 7610)
Sacramento, CA 95899-7414 Mayor City of San Clemente 100 Avenida Presidio San Clemente, CA 92672 James D. Boyd, Commissioner California Energy Commission 1516 Ninth Street (MS 34)
Sacramento, CA 95814 Douglas K. Porter, Esq.
Southern California Edison Company 2244 Walnut Grove Avenue Rosemead, CA 91770 Dwight E. Nunn, Vice President Southern California Edison Company San Onofre Nuclear Generating Station P.O. Box 128 San Clemente, CA 92674-0128 D
SUMMARY OF FINDINGS
IR 05000361/2004017; IR 05000362/2004017; 11/30/2004-12/3/2004; San Onofre Nuclear
Generating Station; Units 2 and 3. Supplemental inspection.
Cornerstone: Initiating Events
The U.S. Nuclear Regulatory Commission performed this supplemental inspection to assess the licensees evaluations associated with two unplanned reactor trips with loss of normal heat removal of Unit 2 during calender year 2002 and one unplanned reactor trip with loss of normal heat removal of Unit 2 during calendar year 2004. The cumulative effect of these trips was that the Performance Indicator for unplanned scrams with loss of normal heat removal crossed the threshold from Green (very low risk significance) to White (low to moderate risk significance) for the second quarter of calendar year 2004. The licensee performed individual root cause evaluations for the three trips. In addition to the individual trip evaluations, the licensee performed a self-assessment evaluation to identify any performance and process issues that led to the White performance indicator. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspector determined that the licensee performed a comprehensive and thorough evaluation in which specific problems were identified, an adequate root cause evaluation was performed, and corrective actions were taken or planned to prevent recurrence.
Report Details
INSPECTION SCOPE
The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess the licensees evaluation associated with a Performance Indicator (PI) that crossed the threshold from Green to White. The PI was for unplanned scrams with loss of normal heat removal for Unit 2 and was related to the initiating event cornerstone in the reactor safety strategic performance area. The PI has been White since the second quarter of 2004.
San Onofre Nuclear Generating Station (SONGS), Unit 2, experienced two unplanned reactor trips with loss of normal heat removal in 2002 and one unplanned reactor trip with loss of normal heat removal in 2004. The cumulative effect of these trips was to cause the PI to cross the threshold from Green to White. The inspector reviewed the licensees actions associated with these three events and conducted interviews with licensee personnel.
The licensee performed a self-assessment evaluation to identify performance and process issues that led to the White PI. The scope of the licensees examination was significantly broader than the scope of this supplemental inspection. The inspector reviewed this self-assessment.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.
Method of identification The PI crossed the threshold from Green to White during the second quarter of 2004 as a result of an unplanned trip with loss of normal heat removal on April 10, 2004. Prior plant trips with loss of normal heat removal had occurred on June 30 and November 2, 2002. A brief description of each trip from the associated Licensee Event Report (LER)and Action Request (AR) is given below. For each trip the event was self-revealing and there were no indications of the impending failure prior to the event.
On February 4, 2003, the licensee initiated AR 030200408 to perform a self-assessment evaluation in response to the negative trend in plant performance indicated in part by unplanned reactor trips that resulted in a White PI for unplanned scrams per 7000 critical hours. In addition, the licensee generated AR 040400600 to perform an assessment of the reactor trip with loss of normal heat removal which occurred on April 10, 2004. The root causes and corrective actions developed in these ARs are discussed in Sections 02.02 and 02.03, respectively.
.1 June 30, 2002:
Inadequate Procedure Results in Incorrect Adjustment of the Steam Bypass Control System Causing a Reactor Trip (LER 05000361/2002003-00, AR 020602197)
Description.
On June 30, 2002, plant operators were performing startup testing with the reactor at about 18 percent thermal power. In response to a small change in steam
flow, the steam bypass control system (SBCS) inappropriately quick-opened all four steam bypass valves. This caused steam pressure to decrease to the low steam generator pressure trip setpoint, resulting in a reactor trip.
Cause.
Incorrect calibration of the SBCS control circuitry resulting from an inadequate maintenance procedure. The licensee identified that a lack of guidance for the revision and review of maintenance calibration procedures was the root cause for the inadequate procedure.
Corrective Action. The licensee developed guidance for the future review of maintenance calibration procedures, provided training to maintenance and engineering personnel, and confirmed the adequacy of the existing calibration procedures. The licensee also revised the calibration procedure and properly calibrated the SBCS in both units.
.2 November 2, 2002:
Main Feedwater (FW) Controller Fault Causes Loss of Main Feedwater and Reactor Trip on Low Steam Generator Level (LER 05000361/2002006-00, AR 021100079)
Description.
On November 2, 2002, an FW controller in Unit 2 failed, causing the associated regulating valve (2FV1111) to close, resulting in a reactor trip from 100 percent power on low steam generator level.
Cause.
Age-related failure of an operational amplifier in the FW control system master controller proportional plus integral control card.
Corrective Action. The licensee replaced the control cards for both FW regulating valves in Unit 2. The cards on Unit 3 were not replaced since they were significantly newer than those on Unit 2. The licensee also developed and began implementing a plan to replace the existing FW control system in both units with a digital, single-failure tolerant system to prevent recurrence of this type of event. The conversion to digital FW control was scheduled for the next Unit 2 refueling outage in October 2005.
.3 April 10, 2004:
Main FW Controller Fault Results in Manual Reactor Trip and Initiation of Auxiliary Feedwater (LER 05000361/2004002-00)
Description.
On April 10, 2004, two electrical grounds in the main FW control system power supply caused an FW regulating valve (2FV1111) to inadvertently shut, causing the main FW pumps to trip on high discharge pressure, prompting Operations personnel to manually trip the reactor.
Cause.
The licensee identified that the first ground was caused by a pinched interconnect wire on one of the two FW regulating control systems. The licensee was unable to locate the second ground, but concluded it was an intermittent ground located in the dc power supply of the FW control system. The licensee verified that 2FV1111 would close when two grounds were present.
Corrective Action. The licensee replaced the damaged wire and inspected and tested accessible electrical wires and components in the FW control system. The licensee also developed and began implementing a plan to replace the existing FW control system in both units with a digital, single-failure tolerant system to prevent recurrence of this type of event. The conversion to digital FW control was scheduled for the next Unit 2 refueling outage in October 2005.
b.
Duration of issue and prior opportunities for identification If no further scrams with loss of normal heat removal occur, the PI should return to the Green band in the third quarter of 2005. In their self-assessment evaluation, the licensee identified several opportunities for prior identification of performance issues related to equipment reliability and human performance.
The licensees self-assessment evaluation considered trips and plant transients dating back to January 2000. The licensee identified a trip on Unit 3 in February 2001 as the first explicit indication of significant problems in equipment reliability. In January 2002, the licensee performed an analysis to compare the equipment monitoring and maintenance programs at SONGS with accepted industry practices, which also identified weaknesses in equipment oversight. In response to these weaknesses, the licensee formed a site team to improve equipment reliability. This team has since assumed responsibility for implementing the corrective actions relating to equipment reliability from the licensees self-assessment evaluation for the White PI.
A self-assessment in March 2001 identified weaknesses in the use of human performance error prevention tools by workers and supervisors. Subsequent assessments in January and April 2002 indicated continued weaknesses in these areas.
In November 2002, the licensee initiated AR 021100235 to develop an integrated site solution to improve human performance. Since this AR was still open when the PI became White, corrective actions from the AR were incorporated with the corrective action for the White PI self-assessment.
c.
Risk consequences and compliance issues The licensee performed a quantitative probabilistic risk assessment for two of the three plant trips to determine the risk associated with each of these trips. The trip on November 2, 2002, was determined to have the highest conditional core damage probability of 5 x 10 -6 and a maximum conditional large early release probability of 2.2 x10 -7. The licensee performed a qualitative risk analysis based on the Updated Final Safety Analysis Report for the June 30, 2002, trip and determined it was also of very low safety significance.
One compliance issue associated with the above trips was addressed in a previous NRC inspection report. A Green finding for an inadequate procedure relating to the trip on
June 30, 2002, was discussed in NRC Inspection Report 05000361, 362/2002005 dated October 17, 2002. The inspector identified no additional findings of significance during this inspection.
02.02 Root Cause and Extent of Condition Evaluation a.
Evaluation of method used to identify root causes and contributing causes The inspector reviewed the licensees root cause evaluation for the White PI and determined that the licensee staff appropriately followed their procedural guidance for performing root cause evaluations.
The inspector determined that the licensees process was adequate to provide accurate root causes and to evaluate the extent of condition for the White PI.
b.
Level of detail of the root cause evaluation The licensees root cause evaluation for the White PI was thorough and identified five repetitive inappropriate actions (RIA) and associated apparent causes and one root cause for the performance issues. The causes identified cover the areas of human performance; procedural adequacy; corrective action effectiveness; and equipment design, monitoring, and maintenance. The RIAs and their associated apparent causes are as follows:
RIA 1: Plant design did not include equipment failure tolerance to avoid power transients. This was caused by an inadequate engineering program procedure which did not completely ingrain the need for power transient avoidance.
RIA 2: Equipment monitoring did not identify or resolve equipment problems prior to plant impact. This was caused by an inadequate program procedure for online equipment monitoring for components important to reliability.
RIA 3: Preventive maintenance did not maintain equipment and components in a condition to prevent plant impact. This was caused by inadequate program management, resulting in a lack of commitment to ongoing maintenance.
RIA 4: Inadequate defense in depth in work plans and in job performance. This was caused by a lack of resource management and inadequate knowledge levels, such that neither supervisors nor workers understood defense in depth and error precursors.
RIA 5: Corrective actions associated with plant problems did not prevent repeat events.
This was caused by inadequate management and supervisory job direction leading to insufficient extent of condition and extent of cause evaluations.
The licensee identified an overarching root cause for the above RIAs as inadequate resource management in that managers and supervisors lacked the skills needed to critically evaluate organizational effectiveness.
Based upon a review of the root cause evaluation, ARs associated with the trips, and discussions with plant personnel, the inspector determined that the evaluation to establish the root and contributing causes was adequate and accurate.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensees evaluation included a review of causes and actions for approximately 20 human performance and 20 equipment reliability issues at other industry sites.
Based on the review of industry data, the licensee concluded that the scope of their root cause evaluation and planned corrective actions were adequate to address the performance issues associated with the White PI.
The inspector found that the licensee appropriately considered operational and industry experience in their root cause evaluation.
d.
Consideration of potential common causes and extent of condition of the problem The licensees evaluation considered the potential for common cause and the extent of condition associated with the performance issues. In addition to the previously discussed inappropriate actions and apparent/root causes, the root cause evaluation identified two generic site issues.
The first generic issue identified a need to perform self-critical evaluations in areas other than equipment oversight and human performance. The second generic issue was a failure to recognize important program procedures (e.g., equipment reliability) as Site Programs, which would give these procedures higher management attention and expectations.
The inspector found that the licensee adequately considered and evaluated potential common causes and extent of condition of the performance issues associated with the White PI.
02.03 Corrective Actions a.
Appropriateness of corrective actions To address the root cause of the performance issue, the licensee participated in industry evaluations and considered industry best practices in the areas of equipment reliability and human performance. Additionally, the licensee assigned the Equipment Reliability Improvement Project (ERIP) Team to review and improve reliability standards for equipment and the Human Performance Improvement Team (HPIT) to improve overall
human performance. Both the ERIP and HPIT procedures were established as site programs to elevate the standards and management expectations for the teams.
The RIAs identified in Section 02.02 are repeated below with the associated corrective actions initiated by the licensee for each.
RIA 1: Plant design did not include equipment failure tolerance to avoid power transients. The licensee implemented a Three Cycle Plan to upgrade and replace fault intolerant equipment. Also, the ERIP Team was assigned to identify equipment not previously designated as safety-related, the failure of which could cause a plant trip or transient. Once identified, this equipment was classified as important to reliability (ITR)and given increased maintenance focus per RIAs 2 and 3, below.
RIA 2: Equipment monitoring did not identify or resolve equipment problems prior to plant impact. The ERIP Team established equipment classifications in the site routine/corrective maintenance database program to identify equipment that was important to reliable operation of the plant. The ERIP Team was also assigned to develop a long-term strategy for maintenance and use of ITR classifications and to establish an equipment monitoring program for critical ITR equipment.
RIA 3: Preventive maintenance did not maintain equipment and components in a condition to prevent plant impact. The ERIP Team was assigned to reevaluate and update the routine/corrective maintenance program and to establish a preventive maintenance program for critical ITR equipment.
RIA 4: Inadequate defense in depth in work plans and in job performance. The HPIT initiated site-wide human performance training and human performance leadership training for managers and supervisors. The purpose of the training was to provide management and the work force with the knowledge and skills to plan and perform error-free work.
RIA 5: Corrective actions associated with plant problems did not prevent repeat events.
The licensee established a Corrective Action Review Board (CARB) to challenge the breadth, adequacy, and timeliness of all cause evaluations. Additionally, the licensee revised the corrective action program governing procedure to clarify requirements for assigning cause evaluators, and developed plans for a qualification standard to train and qualify cause evaluators.
As corrective action for the generic issues discussed in Section 02.02.d, the licensee implemented plans to expand the self-assessment program and to correlate their current list of site programs with those recommended by an accepted industry model.
Based upon a review of the root cause evaluation, discussions with plant personnel, review of plant procedures, direct observation of human performance training, and a meeting of the CARB, the inspector determined that the corrective actions were appropriate.
b.
Prioritization of corrective actions Immediate corrective actions by the licensee involved industry benchmarking to establish a baseline for areas requiring improvement in human performance and equipment reliability. Based on these results, the licensee initiated a phased approach to implementing the above listed long-term corrective actions to improve overall performance. The CARB was initiated to improve the corrective action process, and the ERIP Team and the HPIT established implementation schedules with appropriate consideration of risk significance.
The inspector reviewed the prioritization of corrective actions and determined that the licensee properly prioritized these actions.
c.
Establishment of schedule for implementing and completing corrective actions The inspector found that the licensee established appropriate schedules for implementing and completing corrective actions. Actions completed to date were completed as scheduled. The licensee appropriately considered risk in the scheduling of corrective actions. Several corrective actions involve permanent plant modifications and are scheduled for upcoming Units 2 and 3 refueling outages.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensees corrective actions are intended to prevent a reactor trip resulting from equipment failure or from a human performance error. Additional corrective actions are intended to improve existing maintenance procedures and practices and the corrective action process as a whole. The licensee plans to measure the effectiveness of their corrective actions by conducting self-assessments and performing additional industry benchmarking. The licensee also revised the Self-Assessment Steering Committee charter to be more aggressive in assigning and reviewing division and site self-assessments.
The inspector considered the measures to determine the effectiveness of corrective actions to be adequate.
MANAGEMENT MEETINGS
Exit Meeting Summary
The results of the supplemental inspection were presented to Mr. J. Wambold, Vice President, Nuclear Generation, and other members of licensee management and staff on December 3, 2004. The inspector confirmed that proprietary information was not provided or examined during the inspection.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- T. Adler, Work Control Manager
- R. Clark, Manager, Performance Assessment
- R. George, Consulting Engineer
- K. Johnson, Manager, Electrical and Instrumentation Control Systems
- C. McAndrews, Manager, Nuclear Oversight and Assessment
- M. McBrearty, Engineer, Nuclear Regulatory Affairs
- J. Osborne, Corrective Action Program Lead
- M. Short, Systems Engineering Manager
- T. Vogt, Operations Manager
- R. Waldo, Station Manager
- C. Williams, Supervisor, Regulatory Compliance
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
None
Closed
None
Discussed
None