IR 05000389/2004008
| ML040980013 | |
| Person / Time | |
|---|---|
| Site: | Saint Lucie  |
| Issue date: | 04/06/2004 |
| From: | Joel Munday NRC/RGN-II/DRP/RPB3 |
| To: | Stall J Florida Power & Light Co |
| References | |
| IR-04-008 | |
| Download: ML040980013 (15) | |
Text
ril 6, 2004
SUBJECT:
ST. LUCIE NUCLEAR PLANT - NRC SUPPLEMENTAL INSPECTION REPORT NO. 05000389/2004008
Dear Mr. Stall:
By letter dated January 8, 2004, you were informed that the Nuclear Regulatory Commission (NRC) would conduct a supplemental inspection at your Saint Lucie Nuclear Power Plant for a White performance indicator in the initiating events cornerstone. On March 9, 2004, the NRC completed this supplemental inspection. The enclosed report documents the inspection results that were discussed with you and other members of your staff on March 9, 2004.
The purpose of this supplemental inspection was to examine your problem identification, root cause and extent-of-condition evaluation, and corrective actions associated with a White performance indicator in the initiating events cornerstone. The White performance indicator involved crossing the threshold from Green to White for the Unplanned Scrams per 7,000 Critical Hours Performance Indicator in the fourth quarter of calendar year 2003. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.
Based on the results of this inspection, the NRC determined that the problem identification, root cause and corrective actions for the White performance indicator were generally thorough and comprehensive. The inspectors did not find common cause aspects linking the four reactor scrams from a risk perspective. No findings of significance were identified during this inspection.
FPL 2 In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Joel T. Munday, Chief Reactor Projects Branch 3 Division of Reactor Projects Docket No. 50-389 License No. NPF-16
Enclosure:
NRC Inspection Report 05000389/2004008 w/Attachment : Supplemental Information
REGION II==
Docket No.: 50-389 License No.: NPF-16 Report No.: 05000389/2004008 Licensee: Florida Power & Light Company (FPL)
Facility: St. Lucie Nuclear Plant, Unit 2 Location: 6351 South Ocean Drive Jensen Beach, FL 34957 Dates: February 4 - March 9, 2004 Inspectors: T. Ross, Senior Resident Inspector Approved by: Joel Munday, Chief Reactor Projects Branch 3 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000389/2004008; 02/04/2004 - 03/09/2004; St. Lucie Nuclear Plant, Unit 2; supplemental inspection IP 95001 for a White performance indicator in the initiating events cornerstone.
This inspection was conducted by the St. Lucie Nuclear Plant senior resident inspector. No findings of significance were identified. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
Cornerstone: Initiating Events
This supplemental inspection was conducted to assess the licensees individual and collective evaluations associated with a Unit 2 White performance indicator (PI) in the initiating events cornerstone of the reactor safety strategic performance area. The White PI performance indicator involved crossing the threshold from Green to White for the Unplanned Scrams per 7,000 Critical Hours Performance Indicator in the fourth quarter of calender year 2003. More specifically, Unit 2 experienced four reactor trips during the last three quarters of 2003. The first reactor trip, which occurred on April 1, 2003, was a manual trip from 100 percent power due to the loss of main condenser vacuum caused by a degraded air removal system. The second reactor trip, which occurred on June 11, 2003, was an automatic trip from approximately 22 percent power initiated by equipment failures of the 2A steam generator (SG) low power and full power main feedwater (MFW) bypass flow control valves. The third reactor trip, which occurred on December 4, 2003, was a manual trip from approximately 60 percent power initiated by a loss of the 2A condensate pump due to sudden, catastrophic failure of the lower motor bearing. And, the fourth reactor trip, which occurred on December 20, 2003, was an automatic reactor trip from 100 percent power caused by the loss of main generator excitation due to failure of a voltage regulator control module.
The licensees problem identification, root cause and extent-of-condition evaluations, and corrective actions for the four specific reactor trips were generally thorough and complete.
Although the collective evaluation did conclude that degraded material condition of critical secondary system equipment was an apparent common contributor, it did not identify any specific risk-significant common cause(s) linking all four reactor trips.
REPORT DETAILS
INSPECTION SCOPE
The purpose of this supplemental inspection was to assess the licensees individual and collective evaluations associated with a Unit 2 White performance indicator (PI) in the initiating events cornerstone of the reactor safety strategic performance area. The White PI involved crossing the threshold from Green to White for the Unplanned Scrams per 7,000 Critical Hours Performance Indicator in the fourth quarter of calender year 2003. More specifically, Unit 2 experienced four reactor trips during the last three quarters of 2003. The first reactor trip, which occurred on April 1, 2003, was a manual trip from 100 percent power due to the loss of main condenser vacuum caused by a degraded air removal system. The second reactor trip, which occurred on June 11, 2003, was an automatic trip from approximately 22 percent power initiated by equipment failures of the 2A steam generator (SG) low power and full power main feedwater (MFW) bypass flow control valves. The third reactor trip, which occurred on December 4, 2003, was a manual trip from approximately 60 percent power initiated by a loss of the 2A condensate pump due to sudden, catastrophic failure of the lower motor bearing.
The fourth reactor trip, which occurred on December 20, 2003, was an automatic reactor trip from 100 percent power caused by the loss of main generator excitation due to failure of a voltage regulator control module.
This supplemental inspection was conducted in accordance with the guidance of NRC Inspection Procedure (IP) 95001. As such the following report details are organized according to the outlined IP 95001 guidelines.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.
Determination of who identified the issue and under what conditions All four Unit 2 reactor trips were self-revealing events which occurred during the course of routine startup and normal plant operations, as described below:
The April 1 manual reactor trip occurred during normal full power operations. Earlier the same day, operators had identified that main condenser back-pressure was slowly increasing. They attempted to recover condenser vacuum by placing the 2A hogging ejector into service in accordance with the off-normal operating procedure (ONOP) 2-0610031, Loss of Condenser Vacuum. However, when turbine operators tried to align the 2A hogging ejector, main condenser back-pressure increased rapidly from 3.0" to about 5.6"; whereupon, control room operators promptly tripped the reactor per ONOP 2-0610031.
The June 11 automatic reactor trip occurred during unit startup following a scheduled refueling outage. Prior to the event, SG water level was being maintained by the 15%
MFW bypass valves, during low power operation, when the stem suddenly sheared on the 2A SG 15% MFW bypass valve (LCV-9005) resulting in a complete loss of MFW to the 2A SG. Operators attempted to control 2A SG water level in accordance with ONOP 2-0700030, Main Feedwater, by opening the 2A SG 100% MFW bypass valve (MV 09-03). However, not only did the design of this valve prove problematic for controlling MFW flow at low power levels, but after repeated cycling it subsequently failed in a mid-position resulting in a turbine trip due to high SG water level which also tripped the reactor.
The December 4 manual reactor trip occurred during a rapid down-power as operators attempted to remove the 2A Condensate pump from service. Unit 2 had been operating at full power when it was reported from the turbine building that the 2A Condensate pump lower motor bearing was overheating and smoking. A rapid down power was initiated, but excessive pump vibration due to the failing motor bearing made it necessary for operators to manually trip the reactor so they could promptly secure the 2A Condensate pump.
The December 20 automatic reactor trip occurred during normal full power operations.
It was caused by a sudden loss of the main generator excitation field current supplied by the excitation system. The spontaneous loss of main generator excitation resulted in a main generator lockout, which initiated a turbine trip, that in turn tripped the reactor on loss of load.
b. Determination of how long the issue existed, and prior opportunities for identification The licensee initiated a Significance Level 1 condition report (CR) for each of the reactor trip events described above. These CRs did determine to the extent possible how long the conditions existed that led to the reactor trip events and prior opportunities for identification. Based upon the licensees root cause evaluations, that was some prior opportunity to identify and correct the causal problems for three of the four reactor trips.
The cause of the loss of condenser vacuum manual reactor trip on April 1, was attributed to an out-of-calibration pressure gauge (PI-12-48A) which resulted in inadequate steam supply pressure to the 2A hogging ejector. This local pressure gauge is an essential tool for operators to establish an adequate steam supply for placing the hogging ejectors in service. A review of past work orders determined that PI-12-48A has been found out-of-calibration on several occasions, primarily due to its in-service application. The gauge was located immediately downstream of the throttled pressure control valve and as such was subjected to severe pressure pulsing. The licensee had failed to recognize this longstanding, repetitive condition due to infrequent hogging ejector operation.
The automatic reactor trip of June 11 was initiated by the sudden stem failure of the 2A MFW 15% bypass flow control valve (LCV-9005) that was caused by low stress, high cycle fatigue due to loose piston rings that allowed excessive vibration. Although the event was exacerbated by control room operator performance issues, and subsequent failure of the 2A 100% MFW bypass valve, MV 09-03 the principal initiator was the sheared stem of LCV-9005. As part of a secondary critical valve maintenance program developed in 1994, LCV-9005 was planned and scheduled to be disassembled and inspected during the 14th Unit 2 refueling outage (SL2-14). However, during SL2-14, the licensee decided to defer this work activity until a later refueling outage. It is possible the deferred inspection could have identified the degraded piston rings, or the stem crack itself, and prevented the failure of LCV-9005.
Inadequate lubrication of the 2A Condensate pump, lower motor bearing, was considered the most probable cause that led to the manual reactor trip on December 4.
The 2A Condensate pump lower motor bearing was last replaced and lubricated as part of a pump/motor overhaul in March 1994. The prior preventative maintenance (PM)program required 72 month pump/motor overhauls which included bearing replacement.
However, this frequency-based PM was eliminated in favor of a condition-based monitoring program in June 2001. Part of the rationale for eliminating this routine pump/motor overhaul was the false assumption that the lower motor bearing was a sealed bearing, not an open bearing that required regular lubrication. This error in specifying the type of condensate pump motor bearing was contained in GMP-22, Plant Lubrication Manual, and was apparently introduced sometime in 1994 when various maintenance procedures were combined into the one site-wide lubrication manual. The licensee was unable to determine exactly how this error made.
The most likely cause of the automatic reactor trip on December 20 was the intermittent, random failure of the Current Regulator Instant Limiter module that was part of the Automatic Voltage Regulator (AVR) of the excitation switchgear. The unexpected and intermittent failure of this module made it difficult to discern and would have been nearly impossible to predict.
c. Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues The licensees evaluation of the plant specific risk significance of the four reactor trips concluded the following:
1) The probability risk analysis assumptions (e.g., initiators, component failure probabilities) used at St. Lucie would not need to be adjusted based on these reactor trip events.
2) The four reactor trips were estimated to result in a risk increase in core damage frequency of 6.0E-7/year which is less than the GREEN threshold of 1E-6/year.
The inspector, and a regional senior reactor analyst, reviewed the licensees evaluation and concluded the results were reasonable. No compliance issues were identified by the inspector regarding the licensees collective evaluation. Any compliance issues specifically related to the individual reactor trip events have been or will be addressed during closure of the applicable licensee event reports.
02.02 Root Cause and Extent-of-Condition Evaluation a.
Evaluation of methods used to identify root causes and contributing causes The licensee used several different root cause methodologies, some in combination, to identify the root and contributing causes for each of the four reactor trips according to the guidance in ADM 08.04, Root Cause and Apparent Cause Evaluations. The methods used in the four Significance Level 1 CRs included - Interviews; Events and Causal Factors Analysis/Flowchart; Fault Tree Analysis; Change Analysis; and human performance error evaluation.
In addition to the individual root cause evaluations for each reactor trip, the licensee also conducted a collective evaluation. One of the objectives of the licensees Collective Evaluation for Unplanned Scram NRC Performance Indicator was to conduct a systematic review of each Level 1 CR against the criteria of IP 95001 and ADM 08.04.
This collective evaluation was in essence an expert panel review that, among other objectives, assessed the adequacy and completeness of the individual Level 1 CRs, particularly the root cause analysis, for each reactor trip. Based on this review, the collective evaluation team identified a number of corrections and recommendations to enhance the effectiveness of the Level 1 CRs. For the two reactor trips (i.e., April 1 and June 11) in which the Level 1 CRs were completed, supplemental CRs were issued. For the two reactor trips in December, the applicable Level 1 CRs were revised accordingly.
The inspector concluded that the methods used by the licensee to identify root and contributing causes for each of the four reactor trips were appropriate and consistent with their programmatic guidance. Furthermore, the collective evaluation was self-critical, provided additional value to the final root cause evaluations for each reactor trip, and proposed specific corrective actions to improve existing guidance for conducting root cause evaluations.
b.
Level of detail of the root cause evaluation In general, the root cause evaluations, as revised or supplemented, were of sufficient detail to support the identified root and contributing causes for the four reactor trips.
However, for two of the reactor trip events the level of detail as documented in the Level 1 CRs was incomplete.
For the loss of main condenser vacuum event of April 1, the level of detail of the final root cause evaluation did not adequately address the fact that operators had successfully placed the 2A hogging ejector system in service just three days earlier on March 29. The root cause was attributed to an out-of-calibration pressure gauge, that evidently worked properly only a few days before. The licensees root cause evaluation did not adequately document an explanation of this apparent inconsistency, nor did it adequately explore any potential human performance issues that may have occurred while placing the 2A hogging ejector in service on April 1. The licensee had not conducted a human performance error evaluation to determine if there was a possibility that the root cause was human error, rather than a damaged gauge. To address this omission, the licensee initiated a Plant Manager Action Item (PMAI) to evaluate any potential human performance errors.
For the loss of main generator excitation event of December 20, several important aspects of the final root cause evaluation were not documented to an adequate level of detail. In particular, the root cause evaluation report, including the associated fault matrices, were not updated to incorporate the detailed results from onsite relay testing, and vendor testing of the exciter switchgear modules. These results substantially changed the licensees conclusions regarding the root cause of the event, but this information was not adequately reflected back into the root cause evaluation report. To address the inspectors finding, the licensee subsequently issued another supplement to the original Level 1 CR and initiated a number of PMAIs to ensure critical information from further vendor testing would be evaluated and incorporated as appropriate.
c. Consideration of prior occurrences of the problem and knowledge of prior operating experience The root cause evaluations for the four reactor trips did specifically consider prior plant history of similar events or applicable equipment problems, and industry operating experiences. Furthermore, one of the principal objectives of the licensees collective evaluation was to review all trips, forced outage, and unplanned power reductions over the past two years.
d. Consideration of potential common causes and extent of condition of the problem The licensees collective evaluation specifically reviewed the four Unit 2 reactor trips of 2003, and reviewed all other trips, forced outage and unplanned power reductions over the past two years for both units, to determine if there were any common causal factors that contributed to the high number of unplanned scrams. Although each of the four Unit 2 reactor trips were unique, and did not evidence any specific common equipment failures or performance issues, the licensee was able to conclude that the increased number of reactor trips was directly attributable to an overall common cause of degraded material condition of critical secondary equipment. The inspector likewise concluded that all four reactor trips were caused by non-safety related equipment failures, none of which could be linked to equipment or system related common causal factors. Human performance was only an issue for the June 11 reactor trip, it was not a common cause or significant contributor to any of the other events. Furthermore, the licensees assessment correctly acknowledged an apparent degradation in the overall material condition of secondary systems at St. Lucie, with particular regard given to critical secondary systems that could represent single-point reactor trip vulnerabilities.
The inspector also concluded that the licensee had correctly assessed the extent of condition for the equipment failures which contributed to the four reactor trips.
As part of the collective evaluation mentioned above, the licensee also reviewed 20 other potential precursor events over the past two years in order to discern any additional insights regarding common causal factors that would have contributed to the increased number of reactor trips. Although some insights, not directly related to the four reactor trips (e.g., weaknesses in the PM program ), were gleaned from this review, the licensee determined that the apparent cause investigations for the vast majority of these events (i.e., 16 of the 20) were of insufficient rigor or depth to uncover organizational, programmatic, or human performance causes. In general, the cause(s)of these events were attributed to a component failure. However, the CR investigations did not effectively probe the causal or contributing factors that led to the equipment failure(s). The licensee concluded this represented a programmatic deficiency of the CR process and initiated several PMAIs to improve the Condition Report Program.
Furthermore, the licensee initiated a PMAI to reinvestigate the power transient events that occurred over the past two years. Any additional insights from this reinvestigation will be evaluated and incorporated into the collective evaluation results and established corrective action plans.
02.03 Corrective Actions a.
Appropriateness of corrective actions As part of each Level 1 CR, the licensee took prompt corrective actions to repair the specific equipment failures that caused the four Unit 2 reactor trips in 2003.
Comprehensive corrective actions for each reactor trip to address root and contributing causes, and prevent recurrence, were promptly implemented, or were formally scheduled and tracked by the PMAI process for the longer term corrective actions.
As for the collective evaluation, the licensee had already established or was in the process of developing numerous longterm corrective action plans to address the overall common causal factor regarding the degraded material condition of critical secondary system equipment and components. These action plans included, the FPL Fleet Equipment Reliability Improvement Program, PM Optimization Program, CR Program improvement, a comprehensive corrective action inventory backlog review, System Health Report process, and a thorough review of critical secondary system reliability and material condition. All of these action plans had already been envisioned by the licensee prior to the collective evaluation, except for the critical secondary systems reliability and material condition review. Furthermore, these plans were intended to augment, expand upon, and work in concert with the licensees Maintenance Rule program to improve reliability and availability of all important safety and nonsafety-related plant equipment.
b.
Prioritization of corrective actions Specific corrective actions for the four reactor trips, and the broader scope corrective actions identified by the collective evaluation appeared to be properly prioritized.
However, it was recognized that because of the comprehensive scope and longterm nature necessitated by many of the licensees corrective action plans, there will be a period of time that the facility will be vulnerable to additional equipment failures that could challenge plant reliability. As time progresses, and these multi-year equipment reliability improvement and optimization programs continue to mature this vulnerability should decline. At the time of the inspection, the licensees plan for conducting a thorough review of critical secondary systems reliability and material conditions was not well defined and subsumed as part of the PMAI for conducting a review of the corrective action backlog. After further discussions, the licensee made a decision to separate the two efforts and to assign a distinct PMAI for developing and scheduling a specific plan for accomplishing the critical secondary systems reliability and material conditions review. The due date for developing the methodology and schedule for conducting a critical secondary systems review was set for June 2004, with completion of the review itself scheduled for June 2005.
c. Establishment of a schedule for implementing and completing the corrective actions Numerous near term and long term corrective actions were identified by the Level 1 CRs and the collective evaluation. The inspector verified that an PMAI, or MAI as applicable, had been initiated for each corrective action which assigned specific individual responsibility and due date for completion.
d. Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The inspector determined that effectiveness reviews for all four trips by the responsible organizations and by QA were identified, scheduled, and being tracked by the PMAI and MAI process.
MANAGEMENT MEETINGS
Exit Meeting Summary
The inspector presented the inspection results to Mr. Bill Jefferson, Site Vice President of the St. Lucie Nuclear Plant, and other members of his management team at the conclusion of the inspection on March 9, 2004. The inspector also confirmed that proprietary information was not provided or examined during the inspection.
These issues were discussed further during a meeting on March 23, 2004 at the St.
Lucie site between Mr. Bill Jefferson, key licensee managers, the NRC inspector, and Mr. Joel Munday, Chief, Reactor Projects Branch 3 of the NRC Region II office. This constituted the Regulatory Performance Meeting required per the NRC Action Matrix (contained in NRC Manual Chapter 305, Operating Reactor Assessment Program) for a licensee in the Regulatory Response Column. As discussed in the NRC Annual Assessment Letter dated March 3, 2004, St. Lucie Unit 2 is in the Regulatory Response Column due to the White PI that was the subject of this inspection.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- M. Alfonso, Corrective Action Program Supervisor
- H. Casper, Electrical Systems Engineering Supervisor
- C. Costanzo, Operations Manager
- R. Hughes, Site Engineering Manager
- W. Jefferson, Site Vice President
- G. Johnston, Plant General Manager
- B. Merryman, Daily Planning Supervisor
- T. Patterson, Licensing Manager
- M. Pearson, Human Performance Coordinator
- J. Tucker, Work Control Manager
- M. Wolaver, Acting Systems & Component Engineering Manager
NRC Personnel
- J. Munday, Branch Chief, Division of Reactor Projects, Region II
- R. Bernhard, Senior Reactor Analyst
ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
None
Opened and Closed
None
Closed
None
Discussed
None
LIST OF PRINCIPAL