05000269/LER-2005-001

LER-2005-001,

Docket Number
Event date:	05-24-2005
Report date:	10-06-2005
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
2692005001R01 - NRC Website

BACKGROUND

This event is reportable per 10CFR 50.73(a)(2)(i)(B) as operation in a condition prohibited by Technical Specifications (TS) and 50.73(a)(2)(ii)(B) as an unanalyzed condition. The event involves operation in a configuration where train independence was not maintained, creating a condition where a system failed to meet the single failure criterion. Per guidance in NUREG 1022 section 3.2.4 this constitutes an unanalyzed condition. This system is analyzed for this condition (to support the TS completion times for required actions) but the condition existed for a period substantially longer than the allowed completion time.

The Keowee Emergency Power System [EIIS:EK] consists of two Keowee Hydroelectric Units (KHU-1, KHU-2) that provide an emergency on- site power source for Oconee Nuclear Station (Oconee) via two separate and independent paths. One path is the Underground feeder through transformer CT-4 [EIIS:XFMR] and the Standby Buses [EIIS:EB] and the other is the overhead path through the Keowee Main Step-up Transformer to the 230 KV Switchyard [EIIS:FK] and the individual Oconee Unit's Start-up Transformers.

TS 3.8.1 requires both KHUs and both power paths from Keowee to be operable whenever any Oconee unit is in Modes 1 through 4. One KHU or path may be removed from service for 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. Both KHUs may be inoperable for up to 60 hours6.944444e-4 days <br />0.0167 hours <br />9.920635e-5 weeks <br />2.283e-5 months <br /> for planned reasons if the standby buses are first energized from transformer CT-5 from a Lee Combustion Turbine (LCT) via the Central Switchyard using a dedicated line (the Lee path). This last limiting condition for operation is reduced to 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> if both KHUs are inoperable for unplanned reasons and the LCT is aligned to the Standby Bus within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. In addition, TS 3.8.2 requires one KHU or LCT and its emergency power path to be operable for any Oconee unit in modes 5 and 6.

Per the Bases for TS 3.8.1, either KHU can be tied to the underground or overhead power path. The normal lineup is to dedicate one KHU to the underground emergency power path by closing Air Circuit Breaker (ACB) -3 or 4 and to align the other KHU to the overhead power path through ACB-1 or 2.

The Bases for TS 3.8.1 indicate that the Overhead path KHU receives auxiliary power via Keowee's Main Step-up Transformer through ACB- 5, auxiliary transformer 1X, and 600 VAC Load Centers 1X (KHU-1) or through ACB-6 to transformer 2X and Load Center 2X (KHU-2).

Similarly, the Underground Path receives auxiliary power from Oconee Unit 1 via transformer CX through ACB-7 to auxiliary transformer 1X (KHU-1) or through ACB-8 to 2X (KHU-2).

Each Keowee Unit can start and accelerate without AC power from either of its auxiliary sources. This condition is known as a "Black Start.

Prior to this event Oconee Units 2 and 3 were operating at 100% power in Mode 1 with no safety systems or components out of service that would have contributed to this event. Unit 1 was in Mode 5 for a refueling outage. Subsequently, Oconee Unit 1 made the following Mode changes:

5/10/2005 9:42:22 PM Entered Mode 4 5/11/2005 9:20:02 AM Entered Mode 3 5/14/2005 5:21:00 PM Entered Mode 2 5/15/2005 11:30:00 AM Entered Mode 1 Thus Unit 1 entered the mode of applicability for TS 3.8.1 on 5/10/2005. These Mode changes did not impact the event described below.

EVENT DESCRIPTION

On 5/3/2005 Keowee Operator "A" (KO A) discovered parts of an electrical contactor [EIIS:CNTR]in the bottom of the control cabinet for the Keowee main step-up transformer. KO A reported this to Keowee Supervisor "B" (KS B). Both of these individuals are unlicensed personnel. Together they returned to the cabinet to assess the condition.

Several electrical contactors are located within the cabinet, but are hidden from view behind a closed inner sub-compartment door.

One of several fan bank control contactors had a known problem with an open maintenance work order for its repair. KO A and KS B assumed the parts to be from that contactor, took no action to verify this initial assumption, and therefore took no actions at that time to initiate additional investigation or repairs.

On 5/19/2005 a Nuclear Maintenance Team was replacing a cooling fan on the Keowee main step-up transformer. KO A showed the contactor parts to the maintenance technicians, who determined that the parts came from the Keowee main step-up transformer auxiliary power "Normal source" transfer contactor, which supplies power to the transformer cooling system fans and pumps. At the time, power was being supplied from the "back-up" source via another contactor. A work order and Problem Investigation Process (PIP) 05-3599 were initiated to document the broken contactor.

As part of the PIP documentation, Oconee Operations requested an Operability assessment for the Keowee main step-up transformer on 5/19/2005. Electrical Component Engineer "C" (ECE C) performed the assessment but focused on the requirements of the transformer. ECE C concluded that the transformer was operable because auxiliary power remained available to the transformer cooling system via the back-up source. The Operations Shift Manager (a licensed SRO) concurred.

At 1855 hrs on 5/23/2005, Electrical Engineering evaluated a postulated failure of the back-up source concurrent with an assumed design basis event if Keowee was initially generating to the grid such that the transformer was at operating temperature. In such a condition the transformer could be operated for only one hour before reaching temperature limits. As a result, Oconee conservatively elected to restrict Keowee Grid operation temporarily.

Also on 5/24/2005, Maintenance Support personnel suggested that Keowee Operations tag the contactor out of service and physically disable it to prevent it possibly causing additional damage if it received a close signal in its condition.

On 5/24/2005, while preparing safety tags to remove the contactor from service, Keowee Operators questioned if the resulting configuration could violate train independence criteria for the Emergency Power Path alignment.

Electrical Engineering personnel were consulted and recognized that the existing alignment allowed a single failure of the 2XA motor control center for KHU-2, the KHU aligned to the Underground path, to also result in loss of Main Transformer cooling on the Overhead power path. As a result, at 1350 hours0.0156 days <br />0.375 hours <br />0.00223 weeks <br />5.13675e-4 months <br /> Operations declared entry for all three Oconee Units into TS 3.8.1 Condition C for the Overhead power path being inoperable (a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> allowed completion time).

The immediate corrective action was to change which Keowee unit was aligned to which path. Therefore, KHU-1 was aligned to the underground path and KHU-2 was aligned to the overhead path. At 1540 hours0.0178 days <br />0.428 hours <br />0.00255 weeks <br />5.8597e-4 months <br /> Operations declared both paths operable and exited the TS condition. Also, the KHU-1 normal transformer feeder breaker was opened.

On 5/25/2005 Operations and Engineering removed the restriction on normal operation to the grid.

The contactor is obsolete and an equivalent replacement was not readily available from vendors. A suitable replacement has been identified and ordered. It will be installed following receipt.

Subsequent to this event, an investigation team used computer records to conclude the normal power source transfer contactor closed properly during a routine realignment on 4/4/2005 at approximately 1542 hours0.0178 days <br />0.428 hours <br />0.00255 weeks <br />5.86731e-4 months <br />. At the end of the evolution the contactor opened and it is presumed that the parts broke off at that time. During subsequent challenges, power was restored via the back-up source contactor after intervals ranging from 5 to 40 seconds. The only indication was a loss of cooling flow alarm which cleared immediately upon restoration of power. However, KHU- 1 was aligned to the Underground path, which is the acceptable configuration, until 4/21/2005. At 0820 hours0.00949 days <br />0.228 hours <br />0.00136 weeks <br />3.1201e-4 months <br /> on that date, KHU-1 was realigned to the overhead path using OP/0/A/2000/044. Although not recognized at the time, this alignment subjected both power paths to a single-failure.

The appropriate action for this condition is TS 3.8.1 condition C, required action C.1, which requires an operability check for the newly aligned underground unit within one hour. However, OP/0/A/2000/044 does not require an operability check following a routine swap. Therefore TS non-compliance began at 0920 hours0.0106 days <br />0.256 hours <br />0.00152 weeks <br />3.5006e-4 months <br /> when the allowed completion time was exceeded for TS 3.8.1 required action C.1 (perform an Operability verification surveillance).

Required actions C.2.2.1 - C.2.2.4 were also not met due to failure to recognize entry into the TS condition.

aligned to the overhead path. The TS condition was exited at 1200 hours0.0139 days <br />0.333 hours <br />0.00198 weeks <br />4.566e-4 months <br /> when this realignment was complete.

An ENS notification for this additional event was made at 1556 hours0.018 days <br />0.432 hours <br />0.00257 weeks <br />5.92058e-4 months <br /> on 07/18/2005.

CAUSAL FACTORS

Root Cause #1 - The engineer performing the operability assessment did not consider the system alignment impact on transformer operability with only one power source available to the transformer.

No analysis had documented the requirement that the Keowee main transformer cooling power either must be able to transfer between power sources or must be aligned specifically to the KHU aligned to the overhed path.

The interaction of the transformer cooling power and the alignment of the Keowee Units to the emergency power paths had not been identified and documented prior to this occurrence.

The fact that this interaction had not been recognized impacted this event in several ways:

1. No alarm or procedure guidance exists to verify availability of both power sources. A test that would have detected this failure is conducted every eighteen months but was last performed in September 2004. Therefore the failure on 4/4/2005 was not detected in a timely manner.

2. No Design Basis Document, training material, or Engineering document properly describes the required function. Therefore, even after the failure was discovered, its significance was not properly recognized. ECE C and the Operations Shift Manager performing the Operability Assessment reached the wrong conclusion because they did not know that required redundancy had been lost.

Root Cause #2 - The Keowee Supervisor did not verify the origin of the parts found in the transformer control cabinet.

The KO A had opened the transformer control cabinet door to perform a routine alignment change. While performing this task, the operator observed parts from a contactor had fallen into the bottom of the control cabinet. He notified KS B. Both made the assumption that the part was from a near-by component with a known problem and which had an outstanding Work Order for its repair. KS B failed to assure that the assumption was properly verified and, as result appropriate subsequent actions were not initiated.

Therefore, even after discovery of parts indicating a failed component, the condition was allowed to exist for a period of time longer than permitted by TS.

Contributing Cause - Equipment Failure Due to Aging along with heat fluctuation The broken contactor parts were a plastic plunger within the contactor coil and an attached relatively large piece of metal which, when closed, becomes part of the magnetic core of the contactor relay. Apparently the plastic plunger weakened over time, with thermal expansion/contraction caused by heat fluctuation being a factor in the degradation. Once weakened, it is postulated that the weight and force of the core while opening when de- energized was sufficient to break the plastic plunger and allow the core portion of the assembly to fall. Further investigation will be performed after the contactor is removed.

The equipment failure of the contactor is not considered a root cause of the reportable event because component failures are anticipated. TS provide appropriate action times to allow for timely corrective action of such failures. If proper action had been taken. within the TS action time, this event would not be reportable.

Root Cause #3 - Original Design Oversight The single failure vulnerability associated with relay 89T1X1 is an original design issue. Therefore a formal root cause investigation was not performed; the apparent cause is design oversight. The original design did not include an alternate DC control power source for the overhead path relay logic. The control logic of other overhead path relays permit operation in the emergency start mode even if DC power is lost. Therefore, the apparent deficiency in the original design was the lack of similar fail safe logic for relay 89T1X1.

In early 2000 and 2002, two minor design changes were performed which represented opportunities to discover this single failure vulnerability. Since the vulnerability was not recognized, the scope of the modifications did not address correction of the condition.

The first design change recognized that there was only one source of DC control power to the overhead path. A second source was provided with the intent of maintaining the overhead path operable during future maintenance on DC control power breakers. The presence of this back-up source allowed rapid realignment to exit the condition once it was recognized.

The second design change was to increase the reliability of the overhead path for tornado scenarios by adding additional logic contacts in the circuitry for relay 89T1X1. This logic change would allow the unit connected to the over head path to supply in- house AC auxiliary power ("hotel load") in tornado scenarios if certain components were damaged.

In both of these design changes, various reviews that are part of the modification/design change process addressed the scope of the actual changes and did not reassess the initial design.

CORRECTIVE ACTIONS

Immediate:

1. The KHUs were realigned to the opposite power paths. This aligned the KHU supplying power to the Main Transformer cooling fans to the Overhead path. The KHU serving as the power source to the failed contactor was aligned to the Underground path.

2. When the second single failure issue was found, the control power for the overhead path relay logic was realigned to KHU 2, which was connected to the overhead path.

Subsequent:

2. DOCKET 6. LER NUMBER 3. PAGE 1. A replacement contactor was identified and ordered.

2. Oconee conservatively elected to restrict Keowee Grid operation temporarily.

3.

use DC control power from the Keowee Unit aligned to the overhead as the power source for the overhead path relay logic.

Planned:

1. A suitable replacement contactor will be installed following receipt.

2. Following removal, continue the investigation of the failed contactor and document findings.

3. Evaluate Keowee equipment with power sources from both Keowee Units and logic that is power path specific to determine if there are any Keowee Unit alignment vs. path issues.

1 update: This is complete. The overhead path control logic issue was discovered as a result of this corrective action.> 4. Update Keowee System Design Basis Document and other documents as necessary to reflect the required functions of these power contactors and the impact of failures on the emergency power paths.

5. Provide guidance to control and verify proper operation and alignment of the Keowee Main Transformer power supply contactors.

The corrective actions indicated above are NOT considered NRC Commitment items. There are no NRC Commitment items contained in this LER.

SAFETY ANALYSIS

One actual single failure occurred associated with this event.

However, due to the failure to recognize that failure and take Oconee Nuclear Station, Unit 1 05000269 2005 - 01 - 01 10 OF 13 1. A Loss of Off site Power (LOOP) event occurs, 2. Recovery of offsite power is unsuccessful, 3. A fault disables the 2XA motor control center, and 4a. The SSF is unavailable, which leads to a Reactor Coolant Pump seal LOCA.

OR 4b. The emergency feedwater system [EIIS:BA] fails, resulting in a loss of all secondary side heat removal.

A risk assessment of this event determined that the impacts on the estimated Core Damage Frequency (CDF) and Large Early Release Frequency (LERF) were not significant.

Revision 1 addition:

As a result of a corrective action from the initial event, an additional single failure vulnerability was discovered associated with DC control power to the overhead path relay logic. Whenever KHU 1 was aligned to the underground path, a failure of DC panel 1DA would prevent KHU 1 from starting, causing the loss of the underground power path and prevent KHU-2 AC output breaker ACB-2 from closing, preventing KHU-2 from powering the overhead power path. Thus both paths would be vulnerable to a single failure (this single failure has not actually occurred). Since Oconee routinely rotates which KHU is aligned to the underground path, KHU 1 has been aligned to the underground path approximately 50 per cent of the time.

The significance of this additional identified single failure issue has been evaluated using the current update of the Oconee PRA. The Oconee PRA model was modified to include explicit modeling of the interaction described. The estimated increase in CDF as a result of operation with the recently discovered single failure vulnerability is small, less than 1E-06/RY. The dominant sequences contributing to the increase are loss of offsite power and tornado initiated sequences.

The associated increase in large early release frequency is estimated to be very small, much less than 1E-07/RY.

Therefore, the actual and postulated events described in this LER were not significant with respect to the health and safety of the public.

ADDITIONAL INFORMATION

There have been no similar failures of contactors at Oconee Nuclear Station. A data base search was performed for similar events with similar root causes within the previous three years. None were found; therefore this is not a recurring event.

There were no releases of radioactive materials, radiation exposures or personnel injuries associated with this event.

This event is considered reportable under the Equipment Performance and Information Exchange (EPIX) program. The failed component is a contactor, Part Number CR109E000ALD, made by General Electric.

Energy Industry Identification System (EIIS) codes are identified in the Text within brackets [EIIS:xx].