05000382/LER-2011-003
| Waterford 3 Steam Electric Station | |
| Event date: | 04-30-2011 |
|---|---|
| Report date: | 06-29-2011 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident |
| 3822011003R00 - NRC Website | |
REPORTABLE OCCURRENCE
On April 30, 2011 at 23:32 hours, the Emergency Diesel Generator (EDG) A [EK] output breaker [BKR] did not automatically close during Technical Specification (TS) Surveillance testing. This failure was caused by a human error that occurred on November 2, 2010 when an electrical lead was re-landed on an incorrect terminal on the associated timing relay [RLY] during maintenance. Following the maintenance, EDG A was declared operable on November 6, 2010 at 12:22 hours.
Reporting criteria 10CFR50.73(a)(2)(i)(B), operation prohibited by Waterford 3's Technical Specification TS 3.8.1.1 requires, with one diesel generator inoperable, to demonstrate the OPERABILITY of the remaining A.C. circuits by performing Surveillance Requirements (separately for each offsite A.C.
circuit) within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and at least once per 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> thereafter. With one diesel generator inoperable, verify that: (1) All required systems, subsystems, trains, components, and devices that depend on the remaining OPERABLE diesel generator as a source of emergency power are also OPERABLE, and (2) When in MODE 1, 2, or 3, the steam-driven emergency feed pump is OPERABLE. If these conditions are not satisfied within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> be in at least HOT STANDBY within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and in COLD SHUTDOWN within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />.
The mis-landed wire existed in plant modes of operation in which EDG A was required to be operable and the TS allowed action times were exceeded. There were additional occurrences of operation prohibited by Technical Specifications during this time period. Each specific example is not listed due to this reporting criteria already being met by the listed TS 3.8.1.1 condition.
Reporting criteria 10CFR50.73(a)(2)(v)(D), a condition that alone could have prevented the fulfillment of a safety function needed to mitigate the consequences of an accident.
While EDG A was inoperable between the period November 6, 2010 and April 30, 2011, there were occurrences where EDG B was made inoperable. Additionally, Train B safety related components were made inoperable while their redundant components on Train A were inoperable due to an inoperable EDG A. During the above occurrences, where components in both Trains were inoperable, a condition existed that could have prevented the fulfillment of a safety function needed to mitigate the consequences of an accident with a loss of off-site power.
INITIAL CONDITIONS
Waterford 3 has two EDGs supplied by Cooper-Bessemer. Each diesel provides emergency AC power needed to supply its associated train's safety loads following an accident coincident with a loss of off- site power. This function is satisfied by the automatic starting of the EDGs and automatically re- energizing the safety busses through an output breaker. There is a relay contact in the automatic close circuit for EDG A output breaker that must be closed to provide a permissive for the output breaker to close. The relay must be dropped out to allow automatic closure of the “A” EDG output breaker. This relay drops out after a time delay when the safety bus tie breaker to non-safety bus opens, allowing the EDG A output breaker closure. This relay is model number E7022PB supplied by Amerace Corporation (now Tyco).
On April 30, 2011, the plant was in cold shutdown (Mode 5) conducting planned refueling outage 17.
With EDG B operable, compliance was being maintained with the TS which required one EDG to be operable in mode 5. Compliance was also being maintained with the TSs which required two shutdown cooling loops to be operable and at least one shutdown cooling (SDC) loop in operation in mode 5 with the reactor coolant loops not filled. The safety busses were energized from off-site power through the normal distribution system.
There were no plant structures, systems, or components inoperable at the start of the event that contributed to the event.
EVENT DESCRIPTION
During each refueling outage, operations personnel perform procedure OP-903-115, “Integrated Train ‘A' Integrated Emergency Diesel Generator/Engineering Safety Features Test” to satisfy Technical Specification 4.8.1.1.2.3 requirements. On April 30, 2011 during the performance of OP-903-115, EDG A auto-started but did not energize safety busses 3A and 31A because the EDG A output breaker did not close as expected.
Following the failure of the EDG A output breaker to automatically close once rated voltage and speed was achieved, an attempt was made to close the EDG A output breaker manually from the control room by taking the EDG A output breaker switch to the CLOSE position. The EDG A output breaker did not close. EDG A was manually secured. OP-901-310, Loss of A Train Safety Bus was entered, and off site power was restored to the 3A and 31A Safety Bus. The total time the safety busses were de- energized was approximately 15 minutes.
This condition was discovered while the EDG A and its supported systems were already declared inoperable in accordance with the test procedure. However, TS 3.4.1.5 requires two shutdown cooling (SDC) loops to be operable and at least one shutdown cooling loop shall be in operation in mode 5 with reactor coolant loops not filled. Since the safety bus was de-energized for 15 minutes and the Train A SDC loop was not operable, TS 3.4.1.5 was entered because there were less than two operable shutdown cooling loops. The TS goes on to require that with less than the above required loops operable, immediately initiate corrective action to return the required loops to operable status as soon as possible. As stated earlier, off site power was restored to the 3A and 31A Safety Bus. With the busses energized, SDC loop A was declared operable and TS 3.4.1.5 was exited.
When discovered, EDG B and its associated Train B safety equipment were operable and operating as required to support mode 5 operational requirements.
Trouble-shooting of this failure determined that an electrical lead had been connected to an incorrect terminal on a timing relay during a previous maintenance activity. This failure was caused by a human error that occurred on November 2, 2010 when an electrical lead was re-landed on an incorrect terminal on the associated timing relay. The wiring error was corrected on May 1, 2011, which restored the relay to its proper configuration. EDG A was tested and restored to service on May 2, 2011 at 20:28 hours.
An investigation of this event was conducted under condition report CR-WF3-2011-03190 and it was determined that the wiring error occurred during a previous maintenance activity.
On November 1, 2010 at 0007, the scheduled six year maintenance on EDG A commenced which included a calibration of electrical relay EG EREL2327-C using Work Order 52230980-01. Two electricians were assigned the task of performing the calibration. The relay calibration is performed using the guidance contained in ME-007-005, “Time Delay Relay Setting Check, Adjustment, and Functional Test”. Additional guidance for configuration control and completion of Lifted Lead Verification Forms was contained in Management Guideline MG-33, “Configuration Control Guidelines and Completing Lifted Lead & Switch Manipulation Forms.” The preventive maintenance involves disconnecting all electrical leads to the relay, removing the relay from the panel, performing a bench test in the shop and reinstalling the relay in the field. Prior to disconnecting the leads, the wire identification (ID) and its termination ID are recorded on the lifted lead sheet, ME-007-005 Attachment 12.2, “Lifted Leads Verification Form,” to ensure the lead is re-landed on the same terminal from which it was removed.
Contrary to procedural requirements, the electricians did not follow procedure ME-007-005 and guideline MG-33 during the lead lifting process for relay EG EREL2327-C. The approved design document was not used to record the wiring information on the Lifted Lead Verification Form in accordance with MG-33. In addition, hand tracing the lead to the terminals could have identified the that the Lifted Lead Verification Form had incorrectly recorded the wire location EDG A was declared operable on November 6, 2010 following completion of the scheduled six year maintenance.
CAUSAL FACTORS
There were two root causes and one notable contributing cause attributed to EDG A failing to meet surveillance requirements.
Root Cause 1: Inadequate Concurrent Verification execution during the Lifted Lead Process The electricians did not follow procedure ME-007-005 when performing required “Concurrent Verification” and MG-33's Lifted Lead Verification Form Guidelines during the lifting leads process for relay EG EREL2327-C. An approved design document was not used to record the wiring information on the Lifted Lead Verification Form in accordance with MG-33.
Root Cause 2: Inadequate As Left Configuration Verification of the Lifted Lead Process No procedural guidance exists that requires performing an As Left Verification (such as use of an approved design document, performance of a post maintenance test, or other augmented inspection).
Contributing Cause: Lack of Knowledge of Procedural Requirements Procedure EN-WM-105, “Planning” requires every Work Order to have a PMT. No specific PMT was identified for this Work Order. Planning personnel and Operations personnel stated that the bench test and landing the leads using HU tools was the PMT for the job task. Interviews with 3 planning personnel revealed that they were unaware of the procedure requirements on the development of PMTs and the requirements that are to be followed if a PMT is not specified.
CORRECTIVE ACTIONS
The wiring was placed in its proper configuration as indicated on the design drawing, correcting the wiring error on 5/1/2011.
Human Performance Error Review (HPER) was performed and individual performance issues are being addressed in accordance with company policy.
Procedure ME-007-005, “Time Delay Relay Setting Check, Adjustment, and Functional Test” will be revised to require personnel to use approved plant design documents to verify the as left condition is in accordance with the approved plant design.
Waterford Maintenance will develop and implement a focused “Out of the Box” (specialized training) evaluation for all maintenance personnel with an emphasis on verification practices and behaviors applicable to lifting and landing leads.
Identify all maintenance procedures that govern lifting and landing leads. Issue additional actions to revise these procedures, as necessary, to ensure they are aligned with the changes made to ME-007- 005 that will require verification of as-left-configuration using the approved design document.
Perform training needs analyses using the SAT process to determine training requirements related to this incident, such as EN-WM-105 and EN-WM-107 procedure requirements for post maintenance tests, including the guidance for the WO operations assessments performed by Operations personnel.
Identify safety systems with online maintenance which cannot be fully functionally tested using online procedures. Establish an augmented inspection for each component. If an augmented inspection cannot be developed determine if the maintenance should be moved to the outage scope.
Procedure MD-001-042, “Maintenance Component Status Control” (which has replaced MG-33, “Configuration Control Guidelines and Completing Lifted Lead & Switch Manipulation Forms”) will be revised to require Concurrent Verification of the documentation that reflects the initial as found configuration and to provide prescriptive guidance that temporary labels are only to be used when existing conductor identification is not legible or not installed.
Procedure EN-WM-107, “Post Maintenance Testing” will be revised to establish guidance on development of post maintenance tests for relays.
Maintenance Management will reinforce to all maintenance supervisors the requirement that supervisors will participate in pre-job briefs performed for all critical work activities.
SAFETY SIGNIFICANCE
The Waterford 3 onsite power system, including the onsite electric distribution system, is designed to:
- Provide a reliable source of auxiliary power for safe shutdown of the reactor, assuming loss of offsite power and a single active failure in the onsite power system;
- Be capable of withstanding the effects of a design basis wind, tornado, flood and earthquake event without loss of power to safety-related components essential to safe shutdown; and
- Minimize the probability that the loss of one onsite power supply or its distribution system will cause loss of the other train or of the offsite power system.
Failure of the EDG A output breaker to automatically close was found during surveillance testing on April 30, 2011. This failure was due to an electrical lead that was reconnected to the wrong terminal on a timing relay on November 2, 2010. During the this period that EDG A was erroneously declared operable, EDG A was capable of being started and was available to provide power to its associated safety busses because the EDG A output breaker could have been closed locally by depressing the manual close pushbutton. The EDG A output breaker had not been called upon to perform its specified function for any transient or accident condition, thus this failure did not adversely affect the health and safety of the public.
The initial affect of the output breaker not closing was that the 3A and 31A Safety Busses were de- energized for approximately 15 minutes. There was no direct impact on plant operational parameters from this condition. Plant operational needs were being provided by train B equipment and Shutdown Cooling (SDC) Train A was not required to be in operation at the time of this event. Control Room personnel properly implemented procedure OP-901-310, “Loss of ‘A' Train Safety Bus” and off site power was satisfactorily restored to safety busses 3A and 31A.
This was an equipment related condition and did not present any industrial safety or environmental concerns.
For the period of this event, Waterford 3 operated in Modes 1 through 4 from 12:22 hours on November 6, 2010 through April 6, 2011 at 11:27 hours; after which, Waterford 3 operated in Modes 5, 6 and defueled (No Mode) through April 30, 2011 at 23:32 hours.
An evaluation was performed to evaluate the core damage risk associated with degradation of EDG A due to an incorrectly wired relay. This evaluation quantified risk based on two different conditions, at- power and shutdown.
As previous analyses have demonstrated that the increase in risk is insignificant for Train A equipment out of service along with EDG A, the maintenance periods where Train A equipment was unavailable during this condition were not evaluated as high risk configuration periods.
During the period that the relay was incorrectly wired, Waterford 3 did not experience any weather events that impacted the transmission grid or switchyard at the plant. Therefore, no increase in Core Damage Frequency (CDF) due to loss of off-site power frequency is necessary and the Equipment Out of Service (EOOS) risk evaluation utilizing the normal weather recovery rules is assumed to be conservative.
The risk associated with this condition was estimated using the Waterford 3 PRA model. The PRA includes a human error evaluation for manually closing the EDG A output breaker. A base risk was obtained by using the current at-power average maintenance model with no equipment out of service.
The resulting base Core Damage Frequency (CDF) was compared to the calculated CDF with the relay removed from service. The risk evaluation with the at-power average maintenance model resulted in an incremental core damage probability (ICDP) of 7.007E-7 and an Incremental Large Early Release Probability (ILERP) of 7.007E-8.
In addition, an evaluation was also performed using the shutdown model. A base risk was obtained using the outage schedule. That risk was compared to the risk obtained when taking the relay out of service for the time period in question. There was no change in Conditional Core Damage Probability (CCDP) associated with the shutdown model because risk during shutdown was dominated by all three high pressure injection pumps being taken out of service for maintenance at the same time. As a result, the overall risk for the entire time period is all directly a result from the at-power conditions and the overall ICDP and ILERP remain the same as noted above.
SIMILAR EVENTS
A review identified 2 previous similar events that had been captured in the corrective action program, which are discussed below.
CR-WF3-2007-0123 identified that leads from HVRIT5250A and HVRIT5255A were discovered incorrectly terminated between two different cards in the same loop. The leads were a historical incorrect labeling of the leads contributed to the error, but there was a lack of rigor in the de-termination process which did not catch this historical error, leading to the incorrect termination of the leads.
The root cause of the event was that the procedure was not followed correctly. The procedure has separate sections for the calibration of each card. The leads for one card were supposed to have been lifted and then re-terminated prior to proceeding to the next card. The technicians interpreted the procedure to allow the lifting of the leads from both cards at the same time. The technicians did not verify the cable tag to the terminal before lifting the leads. A contributing cause was that planning personnel did not specify re-test requirements for the calibration activity. Operations did not specify any re-test requirements as part of the Equipment Out of Service entry. Corrective actions included:
standard procedural guidance was implemented for lifting and landing of leads/wires including labeling and verification of labeling. The model work orders were updated for specific VLL cards. In addition, an extent of condition review of representative sample of mandatory preventative maintenance work orders was performed to determine if appropriate retests were specified in the Model Work Orders.
CR-WF3-2008-4822 Troubleshooting an alarm on Startup Transformer A discovered that wiring was terminated incorrectly. The affected wiring was required to be terminated to terminal #3 when it was found to be terminated to terminal #1. This relay is equipped with multiple contacts the only contact affected was for annunciation, contact for operation of the equipment was unaffected. Two experienced electricians disconnected a relay and reconnected the relay after testing it. A Human Performance Error Review concluded that the performer did not use adequate self-checking and the verifier did not use adequate verification techniques. Corrective actions were completed to raise the awareness of verification techniques, critical steps and accountability for these errors. An exercise on disconnecting and reconnecting leads was conducted with each electrical technician. This error only affected annunciator function, not affecting operability. Therefore, this condition did not require a root cause evaluation.
ADDITIONAL INFORMATION
Energy industry identification system (EIIS) codes are identified in the text within brackets [ ].