IR 05000458/2005012
| ML052980072 | |
| Person / Time | |
|---|---|
| Site: | River Bend  |
| Issue date: | 10/24/2005 |
| From: | Kennedy K NRC/RGN-IV/DRP/RPB-C |
| To: | Hinnenkamp P Entergy Operations |
| References | |
| IR-05-012 | |
| Download: ML052980072 (26) | |
Text
October 24, 2005
SUBJECT:
RIVER BEND STATION - NRC SUPPLEMENTAL INSPECTION REPORT 05000458/2005012
Dear Mr. Hinnenkamp:
On September 9, 2005, the NRC completed a supplemental inspection at your River Bend Station. The enclosed report documents the inspection findings, which were discussed with you and other members of your staff.
As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure 95001. The purpose of the inspection was to examine the causes for and actions taken related to the performance indicator for unplanned scrams per 7000 critical hours crossing the threshold from Green (very low risk significance) to White (low to moderate risk significance). This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the events resulting in the White performance indicator are understood, to independently assess the extent of condition, and to provide assurance that the corrective actions for risk significant performance issues are sufficient to address the root causes and contributing causes and to prevent recurrence. The inspection consisted of selected examination of representative records and interviews with personnel.
The inspection concluded that the root causes of the unplanned reactor scrams were adequately defined and understood and the corrective actions resulting from the evaluations appropriately addressed the identified causes.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Entergy Operations, Inc.
- 2 -
Should you have any questions concerning this inspection, we will be pleased to discuss them with you.
Sincerely,
/RA/ WCWalker acting Kriss M. Kennedy, Chief Project Branch C Division of Reactor Projects Docket: 50-458 License: DPF-47
Enclosure:
NRC Inspection Report 05000458/2005012
REGION IV==
Docket:
50-458 License:
DPF-47 Report:
05000458/2005012 Licensee:
Entergy Operations, Inc.
Facility:
River Bend Station Location:
5485 US Highway 61N St. Francisville, LA 70775 Dates:
September 6-9, 2005 Inspectors:
R. Azua, Senior Project Engineer Approved By:
K. Kennedy, Project Branch C Division of Reactor Projects
Enclosure
SUMMARY OF FINDINGS
IR 05000458/2005012; 09/06/2005 - 09/09/2005; Entergy Operations, Inc.; River Bend Station.
Supplemental Inspection for one White Performance Indicator, Unplanned Scrams per 7000 Critical Hours, in the Initiating Events cornerstone.
Cornerstone: Initiating Events
The U.S. Nuclear Regulatory Commission performed this supplemental inspection to assess the licensees evaluations associated with four unplanned reactor scrams that occurred between August 15, 2004, and January 15, 2005. The cumulative effect of these trips was that the Performance Indicator for unplanned scrams per 7000 critical hours crossed the threshold from Green (very low risk significance) to White (low to moderate risk significance) for the first quarter of calendar year 2005. The licensee performed individual root cause evaluations for all of the four reactor scrams. In addition to the individual trip evaluations, the licensee performed a common cause analysis to identify any performance and process issues that led to the White performance indicator. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspector determined that for each scram the licensee performed a comprehensive and thorough evaluation in which specific problems were identified, an adequate root cause evaluation was performed, and corrective actions were taken or planned to prevent recurrence.
REPORT DETAILS
INSPECTION SCOPE
The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection in accordance with Inspection Procedure 95001, Inspection for One or Two White inputs in a Strategic Performance Area. The purpose of this inspection was to assess the licensees evaluation associated with the performance indicator for Unplanned Scrams for 7000 Critical Hours located in the Initiating Events Cornerstone. This performance indicator crossed the threshold from Green to White following four unplanned reactor scrams that occurred between August 15, 2004, and January 15, 2005.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.
Determine who identified the issue and under what conditions The PI crossed the threshold from Green to White during the first quarter of 2005 as a result of an unplanned trip on January 15, 2005. Prior plant trips had occurred on August 15, October 1, and December 10, 2004. A brief description of each trip from the associated licensee event report (LER) and condition report (CR) is given below. For each trip the event was self-revealing.
On January 18, 2005, the licensee initiated CR RBS-2005-0167 to perform a common cause analysis in response to the negative trend in plant performance indicated in part by the unplanned reactor trips and the resulting White PI. The root causes and corrective actions developed in this CR are discussed in Sections 02.02 and 02.03, respectively.
.1 August 15, 2004, Reactor Scram Following a Loss of a 230kV Offsite Power Line
(LER 50-458/04-001-00, CR-RBS-2004-02332):
Description.
A guy wire failed, allowing a 230 kV transmission line structure between Port Hudson and Fancy Point (Line 353) to fall, creating a ground fault condition on the line. The Fancy Point Switchyard protective relaying and breakers actuated in response to the fault. However, due to the slow operation of four breakers (McGraw-Edison), this caused a failure to clear the fault within the design time frame, resulting in a trip of one of the two main generator output breakers. Due to the slow clearing time, the ground fault protection system for the main generator step-up transformers also responded to the fault signal and tripped the remaining main generator output breaker. This caused a turbine trip and a subsequent reactor scram.
Cause.
The root causes that led to the conditions whereby four of the 230 kV circuit breakers in the Fancy Point switchyard opened slowly in response to the offsite electrical faults were:
C The McGraw-Edison breaker trip mechanisms were improperly lubricated.
C The Fancy Point 230 kV circuit breakers were not properly inspected and maintained. Circuit breaker speed time testing was flawed due to a testing methodology that required the breakers to be cycled in order to install the test equipment. This cycling effectively preconditioned the breakers just prior to testing, thereby biasing the test results.
The following items were identified as root cause contributors:
C The interface between the River Bend Station (RBS) and the Transmission group personnel lacked formality. Specifically, the use of the Plant Condition Reporting System was limited to the nuclear organization. RBS personnel were not familiar with the Transmission group work tracking system and could not initiate actions in that system due to Federal Energy Regulatory Commission mandated requirements. These organizations operate as separate entities with no interlinking processes. There was, as a result, no formal means to establish and maintain accountability.
C The licensee was more reactive than proactive. This was notable in the licensees response to CR-RBS-2002-02094. This CR documented a lightning strike, which resulted in the loss of the south bus. McGraw-Edison Breaker 20660 was identified as tripping slow, which caused the breaker failure backup scheme to actuate. A request for the Transmission group to perform the preventive maintenance on the McGraw-Edison breakers more often than what was originally planned clearly indicated an understanding of the significance of slow breaker operations. However, no actions were taken to raise the significance level of the CR. This would have elevated the level of review and attention and potentially resulted in more conservative actions. The CR was subsequently closed based on a memo stating that the required breaker preventive maintenance actions were agreed to with the Transmission group and would be accomplished. This event occurred before the 2-year preventive maintenance was scheduled to be performed.
C A lack of a questioning attitude was evident during the disposition of previous events. A review of RBS history related to slow breaker operations at Fancy Point found that RBS and the Transmission group accepted less than thorough investigation of the cause of the slow breaker operation.
C Personnel re-deployment in the Transmission Organization contributed to the depletion of expertise on the repair and maintenance of the McGraw-Edison breakers.
C Procedure quality and usage shortfalls were discovered. Specifically, the preventive maintenance instructions for the Fancy Point breakers were general in nature and relied on vendor manuals to supply the necessary details.
However, the vendor manuals were not written to a level of detail that gave the
technician specific instructions on which parts to lubricate and how often lubrication was needed. The instructions were not sufficient to ensure correct and repeatable maintenance of the breakers.
C Performance indicators and strategies were ineffective in that not all of the Fancy Point Switchyard circuit breakers were within the scope of the maintenance rule.
Appropriate consideration of previous CRs may have placed the 230 kV system into (a)(1) maintenance rule status, alerting station management to the issue.
This was considered a missed opportunity.
C Predictive maintenance methodology was found to be inadequate.
C There was a relevant issue with high voltage circuit breakers identified in industry operational events that was not appropriately identified and considered.
.2 October 1, 2004, Reactor Scram Caused by Load Reject Resulting from Contaminated
Insulator (LER 50-458/2004-002-00, CR-RBS-2004-02841):
Description.
Due to atmospheric conditions and contaminants, arcing occurred across the porcelain insulators that support one phase of the 230 kV jumpers associated with the generator output line within the protected area transformer yard. This resulted in an electrical fault on the RBS generator output 230 kV line. This subsequently resulted in a main generator trip, which led to a turbine trip and reactor scram.
Cause.
The licensees root cause investigation identified the following root causes for this event:
C Deposited contaminants on the porcelain insulators.
C Circulating water system cooling tower drift eliminators were degraded, allowing contaminants to be deposited on porcelain insulators. The degraded condition of the drift eliminators allowed excessive moisture carryover from the cooling towers to the transformer yard, thereby causing the insulators in the transformer yard to become contaminated. This was confirmed by analysis of the insulator contaminant makeup as well as measurement of the equivalent salt deposit density.
C Adverse weather conditions, specifically, reduced rainfall during the summer months resulted in little or no natural cleaning of the insulators during this time.
This condition was further exacerbated by heavy fog the morning of the event, which led to the flashover.
C Station personnel did not recognize the impact of cooling tower drift on plant equipment. Prior to this event, drift from the cooling tower had never resulted in any observed degradation in the 230 kV line insulators or any other adverse
affect on plant equipment. As a result, station personnel did not recognize the added impact that adverse environmental conditions would have on the insulators.
The following item was identified as a root cause contributor:
C Delays in Station Managements operational decision making resulted in the untimely resolution of the transformer yard insulator degradation. Licensee action plans to address insulator arcing were not specific and the station was not prepared to implement them in a timely manner once the trigger points were reached. Additionally, these action plans did not fully document the potential consequences of the arcing.
.3 December 10, 2004, Reactor Scram Caused by Inverter Failure (LER 50-458/2004-
005-01, CR-RBS-2004-04289):
Description.
The automatic reactor scram was the indirect result of the failure of a 120 volt ac uninterruptible power supply (UPS) on a nonsafety-related instrument bus.
The UPS failed due to a shorted capacitor on an internal circuit board. The loss of the instrument bus resulted in the downshift of the reactor recirculation pumps to slow speed and the lockup of the main feedwater regulation valves. The decrease in coolant flow caused a flow-biased simulated thermal power signal to be sensed in the average power range monitoring system, which actuated the reactor protection system. The reactor scram occurred as designed.
Cause.
The licensees root cause investigation identified the following root cause for this event:
C Design deficiency in Inverter BYS-INV01B resulted in a single point failure vulnerability. Specifically, the licensees investigation concluded that, while the UPS had triple redundant input power sources supporting the uninterruptible design, the logic power for the static switch circuit was fed through a single filter.
Failure at this point would cause the power output of the UPS to be interrupted, contrary to design requirements. The catastrophic failure of the tantalum capacitor resulted in such a failure. This design deficiency had been in existence since the plant began operating on October 31, 1985.
The following item was identified as a root cause contributor:
C The cause of the inverter failure was the catastrophic failure of the tantalum capacitor (C1) on the static switch silicon control rectifier drive board.
.4 January 15, 2005, Manual Scram Following Receipt of a Ground Fault Annunciator on
the Main Generator (LER 50-458/2005-001-01, CR-RBS-2005-0140):
Description.
While the plant was operating at 100 percent power, an alarm was received which indicated a ground fault in the main generator. Control room operators tripped the turbine in accordance with the instructions in the alarm response procedure (ARP-680-09). Subsequently, the licensee determined that a small ground fault current created by copper accumulation in the rectifier tubes was created due to an unforeseen failure in the generator stator cooling water system. In addition, the licensee found that the annunciator relay setpoint had apparently drifted down such that the alarm had come in at a lower setpoint. Finally, the alarm response procedure requirement to trip the turbine was determined to be conservative.
Cause.
The licensees root cause investigation identified the following root causes to this event:
- Unforeseen failure in the generator stator cooling water system led to hydrogen leakage into the stator cooling water system. This displaced the dissolved oxygen and created a condition where copper oxidation would occur. Ultimately, this condition led to copper accumulation in the rectifier tubes. These corrosion products in turn provided a ground path which eventually worsened to the point that a generator field ground fault was annunciated in the control room.
- Unclear vendor guidance resulted in station personnel operating the stator water cooling system with dissolved oxygen levels below vendor recommendations.
The following items were identified as root cause contributors:
- The preventive maintenance on the ground fault relay was less than expected.
Specifically, the relay had not been calibrated since initial plant startup. The relay drifted in a more sensitive direction. This caused the relay to actuate prematurely, and bring in the ground fault alarm.
- Previous industry or in-house operating experience was not aggressively used to prevent the problem. Specifically, industry technical literature had identified that a monitoring system could be utilized to prevent cuprous oxide buildup in the stator cooling system. Engineering requests to install such a monitoring system were presented in the mid-1990's but not approved.
- The prescribed operator alarm response procedure actions for ground fault alarm resulted in the conservative requirement to immediately trip the turbine.
b. Determination of how long the issue existed and prior opportunities for identification
The PI crossed the threshold from Green to White during the first quarter of 2005 as a result of an unplanned trip on January 15, 2005. The PI remained in the White band through the second quarter of 2005. For the individual scrams, the prior opportunities for identification are discussed below.
C August 15, 2004, Reactor Scram Following a Loss of a 230kV Offsite Power Line:
The licensees investigation indicated that the failure of the McGraw-Edison breakers to operate appropriately was due to improper lubrication of the trip mechanism which led to lubricant binding. This was compounded by the fact that the breakers in question were also not properly speed time tested. The test methodology required the breakers to be cycled in order to install the test equipment. This cycling, in effect, preconditioned the breakers prior to testing.
The licensee had a number of prior opportunities to identify the problems with these breakers. In 2002, lightning struck Line 352 from Fancy Point Switchyard, causing a high current fault condition. During this event, McGraw-Edison Breaker 20660 was slow to operate, resulting in the south bus being de-energized. The failure mechanism was identified to be hardened grease in the breaker mechanism. The licensees action at the time was to shorten the testing and lubrication cycle of the breakers from 4 years to 2 years. In February 2004, the licensee identified Operational Event (OE)18079, which described an event at the Duane Arnold plant. This OE identified that the Duane Arnold plant took the switchyard system to (a)1 maintenance rule status. This was due to repeated failures of McGraw-Edison breakers with OA-4 operating mechanisms to CLOSE due to lubrication issues. The licensee failed to take appropriate actions at that time (subject of a violation, documented in NRC Inspection Reports 05000458/2004-004 and 2004-005).
C October 1, 2004, Reactor Scram Caused by Load Reject Resulting from Contaminated Insulatator:
Following this event, the licensee determined that cooling tower drift deposited contaminants on the insulators, which led to the arcing conditions that caused the reactor trip.
The licensee found a missed opportunity where they could have identified and corrected the conditions that led to this scram. On March 2, 2004, a section of the plants cooling tower drift eliminators was found to be missing. The licensee decided to trend the condition and review it. The first indication of arcing that the licensee witnessed occurred on August 19, 2004. The licensee developed an action plan and documented it in Corona/Scintillation Operation Decision Making Issues (ODMI). The plan however did not provide sufficient guidance and the
licensee failed to respond aggressively when further arcing was noted. Other contributing causes to this event were reduced rainfall, which reduced the removal of contaminants from the insulators, and heavy fog, which facilitated the arcing.
Although the licensee had identified a number of related industry OEs during routine reviews, these events did not match exactly the conditions experienced at RBS at the time. Specifically, prior to the loss of the drift eliminators, the plant had not experienced significant drift or contamination of the insulators. This was reinforced by the fact that the plant had not experienced arcing in 18 years.
C December 10, 2004, Reactor Scram Caused by Inverter Failure:
The licensee identified the root cause of the reactor scram to be a design deficiency in Inverter BYS-INV01B, which created a single point vulnerability.
Specifically, while the UPS had triple redundant input power sources supporting the uninterruptible design, the logic power for the static switch circuit was fed through a single filter. Failure at this point would cause the power output of the UPS to be interrupted contrary to current design requirements. The catastrophic failure of the tantalum capacitor resulted in such a failure. This design deficiency had been in existence since the plant began operating. The licensee did not identify any OEs, either internal or industry-wide, which described similar failures.
C January 15, 2005, Manual Scram Following Receipt of a Ground Fault Annunciator on the Main Generator:
The licensee determined that the ground fault was due to an unforseen failure in the generator stator cooling water system. Specifically, this failure led to hydrogen leakage into the stator cooling water system, displacing the dissolved oxygen and changing the character of copper oxidation. Ultimately, this condition led to copper accumulation in the rectifier tubes. These corrosion products in turn provided a ground path which eventually worsened to the point that a generator field ground fault occurred.
The licensee did not identify any industry OEs that appeared to address similar plant complications.
c.
Determination of the plant-specific risk consequences and compliance concerns associated with the issue C
August 15, 2004, Reactor Scram Following a Loss of a 230kV Offsite Power Line:
NRC Inspection Reports 05000458/2004004 and 2004005 documented an inspection finding based on the licensees failure to adequately identify the root
cause of the April 21, 2001, turbine trip and reactor scram so as to prevent recurrence. This failure resulted in a subsequent turbine trip and reactor scram on September 22, 2003.
The finding was of very low safety significance because it did not affect loss of coolant accident initiators, did not contribute to increasing the likelihood of both an initiating event and affect mitigating equipment, and did not increase the likelihood of a fire or flood.
C October 1, 2004, Reactor Scram Caused by Load Reject Resulting from Contaminated Insulatator:
The licensee determined that the event is bounded by the main generator load reject scenario, as documented in the RBS Updated Safety Analysis Report. All safety systems operated as designed. This event was of minimal safety significance.
C December 10, 2004, Reactor Scram Caused by Inverter Failure:
The licensee determined that this event is bounded by analyzed transient for a decrease in reactor recirculation flow in the RBS Updated Safety Analysis Report. This event was of minimal safety significance.
C January 15, 2005, Manual Scram Following Receipt of a Ground Fault Annunciator on the Main Generator:
The plant responded as designed to the manual scram, and no actuations of emergency core cooling systems, reactor safety relief valves, or standby diesel generators were required. This event was of minimal safety significance.
The inspector reviewed the licensees evaluations and assumptions and determined that the licensees results were valid.
02.02 Root Cause and Extent of Condition Evaluation a.
Evaluation of methods used to identify root causes and contributing causes For the four reactor scram events, the licensee utilized Fault Tree Analysis, Event and Causal Factor Analysis, Kepner -Tregoe Problem Analysis, Failure Modes Analysis, PIIs Organizational and Programmatic Diagnostics, and Systems Improvement TapRoot. In addition, the licensee performed field walkdowns, documented reviews, and conducted personnel interviews. The inspector concluded that the licensee effectively utilized accepted root cause determination methods and adequately identified the root and contributing causes for each of the four reactor scram events.
For the Common Cause Analysis, the licensee evaluated all power reductions and scrams for the period of June 2002 through March 2005. The 20 events that were identified formed the scope of the analysis. The licensee formed an RBS common cause team to evaluate and analyze these 20 events. In addition, two separate and independent evaluations were performed of the same events by two other groups contracted by the licensee. These reviews focused on the inappropriate actions in the conditions selected for review. Ultimately, the two contract groups and members of the RBS common cause team collaborated to cross-check the results. The inspector concluded that the licensee utilized accepted methods to develop common causes for the events evaluated.
b.
Level of detail of the root cause evaluation The licensee was thorough in their analysis of the equipment failures and human performance errors for each of the events and the overall common cause analysis.
Common Cause Analysis For the common cause analysis, the licensee decided to evaluate all power reductions and scrams for the period of June 2002 through March of 2005. The 20 events that were identified formed the scope of the analysis. The following are the common causes that were identified as a result of this analysis:
CAUSE 1: Organizational - Management Monitoring/Assessment/Misjudgement
- Station management was not always fully aware of available industry information and equipment condition monitoring unknowns when making decisions about equipment reliability. Unknowns and nonlinear variables, weather, unexpected accelerated degradation or unforeseen plant transients, were not being equally weighted in the decision making process with the conventional technical information.
CAUSE 2: Organization to Program Interface Weak - Management Monitoring/
Assessment (Maintnenance/Engineering/Management)
- Previous in-house/industry operating experience was not effectively used to prevent problems (failure to optimize learning from experience).
- Lack of attention to changing conditions relative to equipment reliability and performance.
- Component assembly errors and inappropriate actions S
Procedure use and adequacy (knowledge, compliance, awareness, and usage at job site).
- Emphasis on job completion exceeded emphasis on methods (doing a good job).
Methods accepted did not meet expectations (ineffective communication of standards and expectations).
CAUSE 3: Organization to Program Interface Weak - Design Configuration and Analysis (Engineering)
- Engineering products such as engineering requests, preventive maintenance input, design data, etc.
S Mindset (accepting at face value without adequate questioning or evaluation) contributed to inappropriate actions.
CAUSE 4: Program to Program Interface Weak (Maintenance/Engineering/
Management)
- RBSs implementation of Entergy Nuclear Licensing Procedure EN-LI-118, Root Cause Analysis Process, as applied to root cause analysis (RCA) and trending does not effectively identify and resolve organizational and programmatic issue.
- RCAs and the stations corrective actions are more focused on technical issue resolution and not balanced in their approach to organizational and programmatic issues.
- Problem statements used in RCA activities are not always accurately depicting the true issues. This can color the findings of the analysis and or make it difficult to determine appropriate corrective actions.
- Corrective actions are not always assigned to the appropriate level of individual to effect corrective actions.
- Common cause analysis methods are not well understood by station personnel and the predominate focus has been on classic root cause determinations.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience C
August 15, 2004, Reactor Scram Following a Loss of a 230 kV Offsite Power Line:
The licensee reviewed their corrective action program and identified five CRs which were similar to this event. One CR, CR-RBS-2002-02094, was found to be relevant to this event. The CR described an event whereby a lightning strike caused a high current fault condition on Line 352, which initiated the line relaying
to trip the associated breakers on each end of the line to clear the fault. In this instance, McGraw-Edison Breaker 20660 was slow to operate. The licensee failed to take appropriate follow-up action to this event.
The licensee also reviewed operating experiences from several different events throughout the industry in addition to reviewing 10 CFR Part 21 reports. The licensee identified 10 events which appeared to be similar; however, only two were relevant to this event. Specifically, in 2002, lightning struck Line 352 from Fancy Point Switchyard, causing a high current fault condition. During this event, McGraw-Edison Breaker 20660 was slow to operate, resulting in the south bus being de-energized. The failure mechanism was identified to be hardened grease in the breaker mechanism. The licensees action at the time was to shorten the testing and lubrication cycle of the breakers from 4 years to 2 years.
In February 2004, the licensee identified OE 18079, which described an event at the Duane Arnold plant. This OE identified that the Duane Arnold plant took the switchyard system to (a)1 maintenance rule status due to repeated failures of McGraw-Edison breakers with OA-4 operating mechanisms to CLOSE due to lubrication issues. The licensee failed to take appropriate actions at that time.
These failures to take adequate corrective actions were the subject of a violation, documented in NRC Inspection Reports 05000458/2004-004 and 2004-005.
C October 1, 2004, Reactor Scram Caused by Load Reject Resulting from Contaminated Insulatator:
The licensee reviewed their corrective action program and did not identify any prior occurrences identical to this event.
The licensee identified 10 industry OEs that were similar. However, the licensee noted that the contamination mechanism described in each of these events differed from this event. There were no events that described contamination of insulators as a result of cooling tower drift.
C December 10, 2004, Reactor Scram Caused by Inverter Failure:
The licensee reviewed plant CRs, operating experiences from several different events throughout the industry, and 10 CFR Part 21 reports. The licensee did not identify any OEs, either internal or industry-wide, which described a similar total loss of UPS output power due to a single failure.
C January 15, 2005, Manual Scram Following Receipt of a Ground Fault Annunciator on the Main Generator:
The licensee reviewed plant CRs, operating experiences from several different events throughout the industry and 10 CFR Part 21 reports. The licensee did not identify any OEs, either internal or industry-wide, that appeared to address similar plant operating conditions.
d.
Consideration of potential common causes, extent of condition, and extent of cause of the problem "Extent of condition" is defined as the extent to which the actual condition exists with other plant processes, equipment, or human performance.
C August 15, 2004, Reactor Scram Following a Loss of a 230 kV Offsite Power Line:
The inspector verified that the licensee conducted a systematic review to identify all McGraw-Edison breakers in the plant, at the Fancy Point switchyard, and at other Entergy Nuclear South switchyards. No McGraw-Edison breakers were found to be used in the plant. Breakers identified in the switchyards were inspected, and the licensee verified that preventive maintenance work for the McGraw-Edison breakers met vendor recommendations for preventive maintenance requirements (the vendor was contacted).
The licensee also issued an RBS Operational Experience report concerning this event.
C October 1, 2004, Reactor Scram Caused by Load Reject Resulting from Contaminated Insulator:
The licensee evaluated other electrical equipment that was potentially impacted by the cooling towers. A review of all electrical equipment utilizing porcelain insulators was performed for reserve station service (RSS) Transformers RSS1 and RSS2 and main generator line bays, along with the normal station service Transformers and the first line structures away from the station.
Due to the increased cooling tower drift, an evaluation of the impact of cooling tower drift on other RBS equipment beyond Transformers RSS1 and RSS2 and main generator line bays was performed. This evaluation included a walkdown to identify potentially susceptible components within the boundary of the cooling tower drift.
The licensee also performed an evaluation of equipment which was subjected to or was in the immediate vicinity of the 230 kV line faults. This included all four transformers located in the transformer yards where the faults occurred. All
were tested and found to be satisfactory. Also evaluated were the isophase bus, main generator, and generator grounding transformer.
C December 10, 2004, Reactor Scram Caused by (Elgar) Inverter Failure:
The licensee identified that the design of the uninterruptible power supplies at RBS introduced a single point vulnerability that was unacceptable per current unit reliability expectations. As such, a study was performed to identify and evaluate design options that would mitigate or eliminate these vulnerabilities.
Other single point vulnerabilities have been identified throughout the plant.
Nuclear Management Manual Procedure DC-300, Periodic Maintenance Program, established the process by which Entergy identifies and implements preventive maintenance strategies to reduce the likelihood that single point vulnerability failures will occur. Nuclear Management Manual Procedure DC-175, Single Point Vulnerability Review Process, provided guidance on developing strategies for single point vulnerability components.
In the case of Inverter BYS-INV01B, and other inverters, this vulnerability was missed based on past plant operating experience.
For Elgar inverters, the single point vulnerability that was identified in this case can be minimized by board replacement. Actions were taken to replace the static switch drive board in the other Elgar inverter applications in accordance with the Entergy Nuclear South preventive maintenance template recommendations.
C January 15, 2005, Manual Scram Following Receipt of a Ground Fault Annunciator on the Main Generator:
The licensee did not identify any other component or equipment that could experience a condition that was similar to that of the stator cooling system hydrogen leak and develop similar complications to the ones described in this event.
In addition, the licensee reviewed all other alarm response procedures that required a manual scram to ensure that such an action was appropriate. In addition, a review was performed of those alarms requiring a manual scram to verify that they were listed as critical components in the preventive maintenance program. No problems were identified.
"Extent of cause" is defined as the extent to which the root causes of an identified problem have impacted other plant processes, equipment, or human performance.
C August 15, 2004, Reactor Scram Following a Loss of a 230 kV Offsite Power Line:
The licensee evaluated the extent of cause to be latent organizational weaknesses which led to weak accountability for proper maintenance of switchyard equipment. The licensee determined that these issues extend beyond the McGraw-Edison breakers, to the other Fancy Point (and ENS)switchyard equipment, including but not limited to the 230 kV and 500 kV breakers, switches, and line structures. In addition, the licensee determined that, since the maintenance templates are used by all Entergy Transmission departments, the condition could extend to any Entergy switchyard and the interfacing lines. The licensee addressed these concerns through their corrective action plan.
C October 1, 2004, Reactor Scram Caused by Load Reject Resulting from Contaminated Insulatator:
The failure of the drift eliminators is generic to all four circulating water system cooling towers and all four were addressed by the corrective action plan.
C December 10, 2004, Reactor Scram Caused by Inverter Failure:
The licensee determined that, from a design standpoint, the extent of cause being a single point vulnerability was applicable to all components identified as single point vulnerabilities that could be addressed with a design change.
However, for this root cause, the scope was limited to UPS designs at RBS. The licensees single point vulnerability identification process and single point vulnerability hardening strategies were deemed adequate to address other single point vulnerabilities as they are identified. Though failure of the subcomponent in this case was determined to be a random failure and not age related, other subcomponents on the single failure vulnerable circuit board do have an age-related failure mode. As a result, the licensee evaluated the design strategies to minimize or eliminate the single point vulnerabilities.
C January 15, 2005, Manual Scram Following Receipt of a Ground Fault Annunciator on the Main Generator:
The licensee determined that plant programs and procedures that implement recommendations from General Electric technical information letters, system information letters, or OEs or vendor documentation stating that a parameter should be maintained in a certain band may be impacted by differences in recommendations within and between documents. An example of this is the original alarm response procedure for the generator ground fault response, which was based on the first revision of the vendor recommendations. In subsequent revisions to the vendor recommendations, additional relaxed options were given for addressing a ground fault alarm that were not incorporated.
There were also conflicting ranges for dissolved oxygen listed in General Electric technical information Letter TIL-1098. Prior to this event, the chemistry procedures were based on the least stringent of these technical information letter requirements.
02.03 Corrective Actions a.
Appropriateness of corrective actions The inspector reviewed the licensees immediate and long-term corrective actions for each of the four reactor trip events that caused the performance indicator for unplanned scrams per 7000 critical hours to cross the threshold from Green to White. The inspector determined that the licensees proposed corrective actions were appropriate to address the root causes identified, for each event, and to prevent recurrence. For corrective actions that had already been completed, the inspector performed a review of the licensees efforts. No problems were noted.
The inspector also reviewed the licensees immediate and long-term corrective actions developed as a result of their common cause analysis of the power reductions and scrams for the period of June 2002 through March of 2005. These corrective actions are listed below.
1. Site lead team to conduct a lessons learned review of recent plant challenges
and the decisions made along with the results of those decisions. Identify any lessons that need to be factored into the site lead teams decision making in the future. (Cause 1/CR-RBS-2005-167-CA14)
2. Establish periodic meetings with the site lead team to conduct follow-up critique
sessions to continually improve decision making results. (Cause 1/CR-RBS-2005-167-CA14)
3. Implement process for common cause analyses at the station and within Entergy
Nuclear South. (Cause 4/CR-RBS-2005-167-CA14)
4. Provide conservative decision making expectations to site personnel to reinforce
expectations for decision making. (Causes 1, 2, and 3/CR-RBS-2005-167-CA15)
5. Establish and implement improved training to be applied to human error analysis
to ensure that human issues are accurately categorized, captured, and addressed. (Cause 4/CR-RBS-2005-167)
6. Determine any common causes for human performance issues and implement
action plan to address. (Cause 4/RLO-2005-102-CA3)
7. Provide lessons learned relative to human performance errors from Refueling
Outage RF012. (Cause 4/CR-RBS-2005-167)
8. Train management team on human performance policies and procedures.
(Cause 4/CR-2005-167/RLO-2005-102-CA3)
9. Improve precursor trending of human performance issues in the corrective action
program. (Cause 4/CR-2005-167/RLO-2005-102-CA23)
10.
Establish process that ensures relevant OEs are included in preventive maintenance work packages. (Causes 1 and 2/CR-2005-167/RLO-2005-102-CA8)
11.
Implement an additional emphasis on performing OE reviews of selected lower tier CRs. Focus on early use of industry OEs to prevent events from occurring.
(Cause 2/CR-RBS-2005-167)
12.
Provide expectations and guidance to station personnel on the effective use of information from OEs. (Cause 2/CR-RBS-2005-167/CR-RBS-2005-1105)
13.
Establish review schedule with effectiveness review guidelines for significant operating event reports. (Cause 2/CR-RBS-2005-167)
14.
Establish postjob brief process to capture worker lessons learned to apply to future jobs. The effective use of OEs should be on this agenda. (Cause 2/CR-RBS-2005-167/CR-RBS-2005-1105)
15.
Develop tools to perform OE reviews when preparing RCAs and missed opportunities. (Cause 2/CR-RBS-2005-167/CR-RBS-2005-1105)
16.
Reduce the number of persons conducting root cause evaluations to improve consistency. (Cause 4/CR-RBS-2005-167/CR-RBS-2005-1104)
17.
Develop and implement improved training and processes for the identification and correction of organizational and programmatic issues. Ensure that the Corrective Action Review Board members are appropriately indoctrinated. Train personnel responsible for RCA/ACE performance. (Cause 4/CR-RBS-2005-167/CR-RBS-2005-1104)
18.
Improve trending process to consistently capture organizational, programmatic, equipment, and human performance issues to facilitate common cause determination. (Cause 4/CR-RBS-2005-167)
19.
Implement and communicate standard for acceptable problem statements to site personnel responsible for RCA and common cause analysis activities.
(Cause 4/CR-RBS-2005-167)
20.
Improve quality of maintenance work packages to support human performance error prevention efforts. In addition, improve the way applicable OE information is reviewed and applied when developing maintenance work packages.
(Causes 1 and 2/CR-RBS-2005-167/CR-RBS-2005-030-CA11)
21.
Establish work package model/checklist defining expectations to improve and standardize the quality of maintenance work packages. (Causes 1 and 2/CR-RBS-2005-167)
22.
Provide training on critical step and error prevention to all instrumentation and controls, electrical, and mechanical maintenance technicians. (Causes 1 and 2/CR-RBS-2005-167/CR-RBS-2004-3518-CA15)
23.
Provide training on critical steps and error prevention to planners. (Causes 1 and 2/CR-RBS-2005-167/CR-RBS-2004-3518-CA20)
24.
Evaluate performance monitoring data base, validate monitoring points to ensure adequacy and verify triggers and thresholds are properly established.
(Cause 3/CR-RBS-2005-167)
25.
Train personnel on expected quality of equipment failure analysis/evaluation.
(Cause 3/CR-RBS-2005-167/CR-RBS-2005-1098-CAs 5, 6, and 7)
26.
Improve troubleshooting process and provide training to engineering and maintenance personnel. (Cause 3/CR-RBS-2005-167/CR-RBS-2005-104-CA15)
27.
Review open work orders and CRs on safety and risk significant systems to ensure low level long-term equipment deficiencies are identified and appropriately prioritized. (Cause 3/CR-RBS-2005-167/CR-RBS-2005-101-CA21)b.
Prioritization of corrective actions The inspector concluded that the corrective actions were properly prioritized. Actions of an immediate nature were given the highest priority and accomplished on an acceptable schedule. Actions to resolve program, training, and procedure weaknesses were established. A completion date and a responsible manager were assigned for each corrective action, and these were tracked through the corrective action system.
c.
Establishment of schedule for implementing and completing the corrective actions The inspector determined that the licensee had completed most of the corrective actions for each of the CRs. The inspector reviewed a sample of the completed corrective actions and concluded that they had been implemented successfully.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensees RCA and recommended corrective actions were reviewed and approved by the Plant Review Committee. Each recommended corrective action was assigned a member of licensee management for responsibility and completion. These actions will be tracked and trended through the licensees corrective action program. Additionally, the corrective action program required that the licensee evaluate the effectiveness of the corrective actions described in the CRs.
OTHER ACTIVITIES
40A6 Management Meetings
Exit Meeting Summary
The inspectors presented the inspection results to Don Vinci, General Manager, Plant Operations, RBS, and other members of licensee management at the conclusion of the inspection on September 9, 2005. The licensee acknowledged the information presented.
The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
Attachment
SUPPLEMENTAL INFORMATION
PARTIAL LIST OF PERSONS CONTACTED
Licensee
- D. Vinci, General Manager, Plant Operations
- H. Goodman, Director, Engineering
- G. Huston, Assistant Manager, Operations
- P. Russell, Manager, System Engineering
- D. Lorfing, Manager, Licensing
- N. Johnson, Manager, Programs and Components
- L. Ballard, Manager, Quality Assurance
- T. Coleman, Manager, Planning and Scheduling Outages
- C. Forpahl, Manager, Corrective Actions and Assessments
- T. Gates, Manager, Nuclear Engineering
- B. Biggs, Coordinator, Safety and Regulatory Affairs
- R. Cole, Supervisor, Engineering
- R. Gauthreaux, Supervisor, Engineering
- P. Mitkus, Supervisor, Engineering
- T. Watkins, Supervisor, Engineering
- M. Davis, Supervisor, Radiation Protection
- K. Huffstatler, Technical Specialist IV, Licensing
NRC
- R. Azua, Senior Project Engineer
LIST OF DOCUMENTS REVIEWED
RIVER BEND STATION CONDITION REPORTS (CRs)
Number
Topic
Fancy Point Slow Breaker Operations Resulting in River
Bend Station Plant Scram
Electrical Fault on RSS1 and the Main Generator Lines
Resulting in Main Generator Trip and Plant Scram
CR-RBS-2004 04289
Plant Scram Due to Inverter BYS-INV01B Failure
Generator Field Ground Alarm Manual Trip
Attachment
Unplanned Scrams Exceed the Criteria for a White
Performance Indicator
230 kV Insulator Arching
MISCELLANEOUS
Document
Description
Revision
Root Cause Analysis Process
Apparent Cause Evaluation (ACE) Process
Switchyard, Transformer Yard, and Sensitive
Equipment Controls
Loss of Main Condenser Vacuum/Trip of
Circulating Water Pump
Operational Decision Making Plan - 230 kV
Insulator Arching
Root Cause Analysis Report
Fancy Point Slow Breaker Operations
Resulting in River Bend Station Plant Scram -
dated 09/02/2004
Root Cause Analysis Report
Electrical Fault on RSS1 and the Main Generator
Lines Resulting in Main Generator Trip and Plant
Scram - dated 11/12/2004
Root Cause Analysis Report
Plant Scram Due to BYS-INV01B Failure -
dated 02/24/2005
Root Cause Analysis Report
Generator Field Ground Alarm Manual Trip -
dated 03/16/2005
Common Cause Analysis
Report
Unplanned Scrams Exceed the Criteria for a White
Performance Indicator - dated 07/06/2005