05000302/LER-2005-001

LER-2005-001, Design Change Creates Engineered Safeguards Bus Protective Relay Scheme Single Failure Vulnerability

Crystal River
Event date:	01-27-2005
Report date:	03-23-2005
Reporting criterion:	10 CFR 50.73(a)(2)(ix)(A) 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition
3022005001R00 - NRC Website

EVENT DESCRIPTION

At 18:30 on January 27, 2005, Progress Energy Florida, Inc. (PEF), Crystal River Unit 3 (CR-3) was operating in MODE 1 (POWER OPERATION) at 100 percent RATED THERMAL POWER when a non-emergency eight-hour notification was made to the NRC Operations Center (Event Number 41362). A design configuration subject to a single failure that could prevent both Emergency Diesel Generators (EGDGs) [EK, DG] and both offsite power sources from supplying power to their respective 4160 volt (V) Engineered Safeguards (ES) buses [EB, BU] had been identified. This condition met the notification requirements of 10CFR50.72(b)(3)(ii)(B).

On January 27, 2005, during the final week of the NRC 2005 Triennial Fire Protection Inspection, a response was prepared to NRC questions related to the CR-3 10CFR50, Appendix R, Fire Study.

Section 3 of the Fire Study, "Appendix R Circuits Listing," contained notes describing a design problem that could result from an Appendix R fire in the protective circuitry for incoming breakers 3211 and 3212 [EB, BKR] from the Offsite Power Transformer (OPT) [EB, XFMR] to both 4160V ES buses and incoming breakers 3205 and 3206 [EB, BKR] from the Backup ES Transformer (BEST) [EB, XFMR] to both 4160V ES buses. During the review of PEF responses to questions, NRC Inspectors identified an electrical protection and metering circuit which, if damaged, could electrically lock out both 4160V ES buses and prevent their re-energization both from offsite power sources (OPT and BEST) and from the EGDGs (breakers 3209 and 3210) [EB, BKR]. The full extent of this design configuration had not been fully recognized nor had its consequences been previously evaluated for a potential single failure occurring during a Loss of Coolant Accident (LOCA) coincident with a Loss of Offsite Power (LOOP) event.

Normally, only one 4160V ES bus is aligned to OPT or BEST. However, the design basis and capability of OPT and BEST is for each transformer to be able to feed both 4160V ES buses. Each 4160V ES bus also has one EGDG as a standby power source. The design of phase overcurrent relaying (51B relay) [EB, 46] and residual neutral ground overcurrent relaying (51BN relay) [EB, 64] for incoming breakers 3211 and 3212 from OPT to the 4160V ES buses employ a common neutral return path through a watt-hour meter [EB, JI]. A similar configuration exists for the protective relaying associated with incoming breakers 3205 and 3206 from the BEST to the 4160V ES buses.

A simplified schematic diagram is shown in Figure 1 for the OPT and in Figure 2 for the BEST.

Each watt-hour meter needs current and voltage inputs to operate. As shown in Figure 1, three current transformers (CT) [EB, XCT] at breakers 3211 and 3212 feed three phase overcurrent relays (51B) and one residual neutral ground overcurrent relay (51BN) with the return path for both the breaker 3212 and 3211 CT current circuits through the watt-hour meter. Similarly, three CTs at breakers 3205 and 3206 feed three overcurrent relays (51B) and one residual neutral ground overcurrent relay (51BN) with the return path for both the breaker 3205 and 3206 CT current circuit through the watt-hour meter. The location of ground for above relaying is electrically different for the BEST and OPT circuits as shown in Figures 1 and 2. Also, the OPT watt-hour meter has two current elements while the BEST watt-hour meter has three current elements. The common neutral return path through the watt-hour meter, coupled with the location of the ground, causes a potential for a single failure (fire induced cable fault or watt-hour meter failure) to affect both the 4160V ES buses simultaneously.

Simultaneous actuation of the 51BN relays would not only trip the associated offsite power source feeds, but also actuate lockout relays for both 4160V ES buses, stripping all bus feeds and locking out closure of the EGDG output breakers.

At 18:30 on January 27, 2005, Improved Technical Specification (ITS) 3.8.9, Condition A, was entered for having one alternating current (AC) electrical power distribution subsystem [EB], inoperable. The potential loss of both 4160V ES Buses due to a single failure was considered analogous to a loss of redundancy associated with the AC Electrical Distribution Subsystem. ITS 3.8.9, Condition A, requires that the AC electrical power distribution subsystem be restored to operable status within 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />.

At 22:00 on January 27, 2005, ITS 3.8.1, Condition A, was entered. Breakers 3211 and 3212 were declared inoperable in order to implement a modification (Engineering Change 60150) to remove the single failure vulnerability associated with the OPT.

At 01:37 on January 28, 2005, Engineering Change 60150 was implemented and breakers 3211 and 3212 were declared operable. The actions of ITS 3.8.1, Condition A, were no longer applicable.

At 01:50 on January 28, 2005, ITS 3.8.1, Condition A, was entered. Breakers 3205 and 3206 were declared inoperable in order to implement a modification (Engineering Change 60155) to remove the single failure vulnerability associated with the BEST.

At 01:52 on January 28, 2005, the actions of ITS 3.8.9, Condition A, were no longer applicable.

The potential for OPT protective relay actuation from a fire-induced cable fault or watt-hour meter failure was eliminated by modification.

At 06:41 on January 28, 2005, Engineering Change 60155 was implemented and breakers 3205 and 3206 were declared operable. The actions of ITS 3.8.1, Condition A, were no longer applicable.

This report is being submitted pursuant to 10CFR50.73(a)(2)(ii)(B) and 10CFR50.73(a)(2)(ix)(A).

SAFETY CONSEQUENCES

The design basis capability of the OPT and BEST is for each transformer to be able to feed both 4160V ES buses. The design intent is that the most limiting condition caused by actuation of the 51B and 51BN relaying would be loss of offsite and onsite power to only the one 4160V ES bus where the fault occurred that caused the actuation of these relays. A preliminary failure modes and effects analysis identified six single failure scenarios where power (offsite and onsite) would be lost to both 4160V ES buses. Five of these six scenarios applied to the design basis 4160V ES bus alignment where the OPT or BEST are feeding both 4160V ES buses. Such single failures taken coincident with a LOCA/LOOP could compromise the ability to mitigate the consequences of such an event. One of these six scenarios applied to the normal 4160V ES bus alignment where 4160V ES Bus "A" is powered from the OPT and 4160V ES Bus "B" is powered from the BEST.

The core damage risk associated with this event was evaluated using input from simulator runs, walkdowns, and Probabilistic Risk Assessment (PRA) insights. Unlike design basis assessments, which are based on mitigating a design basis event (LOOP/LOCA) with any single failure, a probabilistic assessment of core damage is based on best estimates for the likelihood of event occurrence and equipment failures. Using PRA, the frequency of occurrence for a design basis LOOP/LOCA coincident with a bus fault is less than that which requires further analysis.

The risks associated with a failure involving the watt-hour meter or its circuitry were assessed based on a random failure of the circuitry; a failure of the subject circuitry caused by a bus fault; or, fire in the control room or switchgear rooms. The initial conditions are a normal operating plant at 100 percent power and a normal 4160V ES bus alignment where 4160V ES Bus "A" is powered from the OPT and 4160V ES Bus "B" is powered from the BEST. As previously mentioned, one single failure scenario has been identified that could result in locking out both 4160V ES buses in the normal alignment. The failure is a bus fault on the 4160V ES Bus "A" with a consequential failure of the CT circuitry to the OPT breakers. The bus fault can be a breaker failure or a fire.

Non-Fire Initiated Failure Mode The only non-fire initiated failure mode susceptible to locking out both 4160V ES buses with a normal plant configuration is a 4160V ES Bus "A" failure which causes a consequential failure of the CT circuitry. The result would be a loss of both 4160V ES buses due to a lock-out of both offsite and onsite power sources with a subsequent loss of all AC powered safety-related equipment. Direct current [EJ] and inverter [EF] powered equipment would remain powered from the safety-related batteries [EJ, BTRY]. Due to the plant specific configuration at CR-3, the plant would remain online at 100 percent power with the primary impact being a loss of normal makeup [CB]. After isolating letdown, simulator experience indicates that the Reactor Coolant System [AB] would be stable for about three hours until the low pressurizer level would direct a plant action.

Other complications would involve a loss of reactor building (RB) [VA] and control complex (CC) [VI] cooling. The plant response to the loss of CC cooling would be very similar to a Station Blackout (SBO) event, except that Emergency Operating Procedure EOP-12, "Station Blackout," would not be entered immediately since the plant would not trip. Based on room heat-up calculations, there is about 80 minutes before the Emergency Feedwater Initiation and Control (EFIC) System [BA], rooms exceed design temperature limits. Beyond this time, EFIC control is unpredictable and can potentially compromise the Emergency Feedwater (EFW) system [BA] capability. If SBO actions are taken to ventilate the EFIC rooms, up to four hours would be available, limited by batteries. A loss of EFIC is assumed to fail Emergency Feedwater Pump EFP-2 (steam driven) [BA, P]. EFP-3 (diesel driven) [BA, P] would still be available however, operator action may be required to open the injection valves. Auxiliary Feedwater Pump FWP-7 (electrically driven by an independent diesel) [SJ, P] is not impacted by EFIC and would be available as long as it is started prior to battery depletion. Lastly, RB temperatures are expected to exceed ITS limits for normal operation but are not expected to challenge accident design limits.

Recovery of the 4160V ES bus would involve the determination that 4160V ES Bus "B" is not faulted, resetting the lockouts and loading the EGDGs.

Feedback from the Maintenance Department indicates that a diagnosis could take up to one and a half hours. After resetting the lockouts, most operator actions for plant recovery can be accomplished from the control room. Given the plant would have remained stable, there were potential recovery actions, and there was at least one train of mitigating equipment available, the risk of this scenario is very low.

Fire Initiated Failure Mode The only fire initiated failure mode that is susceptible to locking out both 4160V ES buses with a normal plant configuration is a fire in the 4160V ES Bus "A" switchgear room. Current fire modeling practices indicate that hot gas layer formation is unlikely based on a relatively low heat release rate, coupled with the small volume of cables, and that only the cabinets directly adjacent to the CT circuitry need to be counted as initiators. The worst case scenario could cause a loss of offsite power and a plant trip, assuming that the offsite power feeds could also be impacted in addition to the CT circuitry. This scenario would be an SBO with the plant being cooled using EFW and natural circulation. CR-3 has three electrically independent delivery methods to provide core cooling (EFP-2, EFP-3 and FWP-7). After four hours, the batteries deplete and EFP-2 is assumed to fail based on the inability to reliably control the Once Through Steam Generator [AB, SG] level.

As in the previous case, room heat-up could fail EFIC control in about 80 minutes, compromising EFP-2 and EFP-3. If this occurs, FWP-7 is still available. Control Rod Drive room heat-up is not expected to exceed temperature limits for FWP-7 valve controllers. Manual action to reset the EGDG output breaker to recover from this event is expected to be implemented in about 25 minutes, but no later than 30 minutes, from the start of the event. The fire risk is very low based on the current fire modeling practices, the availability of mitigating equipment and the potential to recover power to the unaffected bus.

Based on the above discussions, the identified single failure vulnerability does not represent a reduction in the public health and safety. The risk significance of the identified single failure vulnerability has been determined to be very low based on the initiating event frequency, the ability to recover without initiating a plant transient, and the availability of diverse mitigating systems and equipment that would remain unaffected by the single failure.

The OPT, BEST and EGDGs remain operable during the duration of the identified condition, providing power to both 4160V ES buses. Therefore, this event does not meet the Nuclear Energy Institute (NEI) definition of a Safety System Functional Failure (NEI 99-02, Revision 2).

CAUSE

Two causes have been identified for the single failure vulnerability. First, procedures in effect at the time the OPT and BEST were installed required the design engineer to consider failure effects requirements of structures, systems and components, including a definition of those events and accidents which they must be designed to withstand. Neither CR-3 engineering management nor design engineers interpreted this to require a failure modes effects analysis. Second, inadequate technical rigor was exercised during the design, verification, and CR-3 acceptance of the modification packages developed for the OPT and the BEST by the CR-3 Architect Engineer. The function of the 518/51BN protective relaying was not fully understood by the design team. The functions were believed to be nonsafety-related. Moving the protective relaying into the existing, but common metering circuit, was done without regard to creating cross train vulnerability.

Original Configuration Originally, the Crystal River Unit 1/Crystal River Unit 2 (CR-1/CR-2) start-up transformer and CR-3 start-up transformer [EB, XFMR] were the two independent power supplies to the 4160V ES buses as shown in Figure 3. The 518/51BN relays, CTs and connecting wiring for the 4160V ES Bus "A" feeder breakers (3205/3211) and 4160V ES Bus "B" feeder breakers (3206/3212) were completely isolated. If the 51B/51BN relays associated with these breakers actuated, then the faulted 4160V ES bus would be de-energized by opening all possible power supply breakers (including the EGDG breaker) and locking them out until they were manually reset. There was wiring that connected CTs from each train into a common metering circuit. No control function was provided by the metering circuit and the CTs provided electrical isolation from the actual 4160V ES bus. No single failure mode could create a loss off offsite and onsite power to both the 4160V ES buses. This design met the single failure criteria.

OPT Design Change Single failure vulnerability was introduced in 1990 when the protective relay circuits for OPT breakers 3212 and 3211 were modified by Modification Approval Record (MAR) 89-08-11-03 as part of the interface for functionally replacing the CR-1/CR-2 start-up transformer with the OPT.

The 51B/51BN relays were taken out of the dedicated CT circuits and connected into the metering circuit that used CTs on both the "A" and "B" trains (see Figure 4). This violated the single failure and train separation criteria.

BEST Design Change Single failure vulnerability was introduced again in 1993 when the protective relay circuits for BEST breakers 3206 and 3205 were modified by MAR 91-03-23-01 as the final part of the interface for functionally replacing the CR-3 start-up transformer with the BEST. The 51B/51BN relays were taken out of the dedicated CT circuits and connected into the metering circuit that used CTs on both the "A" and "B" trains (see Figure 5). Although less susceptible to a single failure due to a change in the location of the ground, single failure and train separation criteria were still violated.

CORRECTIVE ACTIONS

1.� Modifications (Engineering Changes 60150 (OPT) and 60155 (BEST)) were implemented to disconnect the CT circuits to the watt-hour meters and remove the common return path of both train's CT currents through the watt-hour meters. This configuration restores electrical and physical separation between the two trains.

2. The protective relaying schemes for the breakers in the following table were reviewed for single failure criteria compliance. No violations of the single failure criteria were identified.

4160V ES bus offsite source breakers 3205, 3206, 3207, 3208, 3211 and 3212 4160V EGDG output breakers 3209 and 3210 4160V/480V transformer breakers 3220, 3221 and 3222 480V breakers 3310 and 3311 480V ES bus cross-tie breakers 3390 and 3391 3. Other actions associated with this event are being addressed in the CR-3 Corrective Action Program in Nuclear Condition Report 149507.

PREVIOUS SIMILAR EVENTS

No previous similar events involving single failure vulnerability of the 4160V ES buses have been reported to the NRC by CR-3.

FIGURES

Figure 1 - OPT Protective Relay Scheme — Simplified Drawing Figure 2 - BEST Protective Relay Scheme — Simplified Drawing Figure 3 - Original Configuration of 4160V ES Buses - Simplified Drawing Figure 4 - 4160V ES Buses After OPT Installation - Simplified Drawing Figure 5 - 4160V ES Buses After BEST Installation- Simplified Drawing

ATTACHMENTS

Attachment 1 - Abbreviations, Definitions, and Acronyms Attachment 2 - List of Commitments FIGURE 1 OPT PROTECTIVE RELAY SCHEME - SIMPLIFIED DRAWING ATTACHMENT 3A

OPT B BUS FEED

A BUS FEED BKR 3212 BKR 3211 51BN 51BN I) EXIST.

G/1/' ..._yi tG GROUND 51B 51B C B A r---1 L_J WATT 1

METER I

.1 51B: PHASE OVER CURRENT RELAY 51BN: RESIDUAL NEUTRAL GROUND

OVER CURRENT RELAY

FILE: ATTACHMENT3A.DWG FIGURE 2 BEST PROTECTIVE RELAY SCHEME - SIMPLIFIED DRAWING ATTACHMENT 3B

BEST

BEST EXIST.� B BUS FEEDA BUS FEED GROUND� BKR 3206 BKR 3205 51BN� ' 51BN ) )51B� 51B r--1� 71 I� I� I� I I� I� I I� � pH ir� 1 I L_J� L_J r--

WIT

I�HO UR 1 � METERJ1.I 51B: PHASE OVER CURRENT RELAY 51BN: RESIDUAL NEUTRAL GROUND

OVER CURRENT RELAY

FILE ATTACHMENT3B.DWG FIGURE 3 ORIGINAL CONFIGURATION OF 4160V ES BUSES - SIMPLIFIED DRAWING CR-1 & 2CR-3 Start-Up AAA/Start-UpTransformer A11/NA ivnnTransformer Original Configuration 1� CR-3 Unit Aux xfmr I 4160V Unit Buses 3207 3205 ( 3211 3208 3206 ( 3212 4160V ES BUS 3A 3209�3210 ii A EDG�B EDG FIGURE 4 4160V ES BUSES AFTER OPT INSTALLATION - SIMPLIFIED DRAWING CR-3 Start-Up NAIA/ OFFSITE AAA/ Transformer PWR XFMR ^ AAA AAAA After OPT Installation in 1990 MAR 89-08-1.1-03 CR-.3 Unit Aux xfmr 4160V Unit Buses 3207 3205 ( 3211 3208 3206 ( 3212 4160V ES BUS 3A 4160V ES BUS 3B 3209�3210 ii A EDG�B EDG FIGURE 5 4160V ES BUSES AFTER BEST INSTALLATION- SIMPLIFIED DRAWING BACKUP AAA/�OFFSITE AAA/

ES XFMR AAAA PWR XFMR AAAA

After OPT Installation in 1991 MAR 89-08-11-03 CR-3 Unit Aux xfmr 1--,-,,.,p 4160V Unit Buses 3207 3205 3211�3208 3206 ( 3212 4160V ES BUS 3A 4160V ES BUS 311 3209�3210 ii A EDG�B EDG

CFR

CR-1/CR-2 CR-3

SBO

V

NOTES:

ATTACHMENT 1

ABBREVIATIONS, DEFINITIONS AND ACRONYMS

Alternating Current Backup Engineered Safeguards Transformer Control Complex Code of Federal Regulations Current Transformer Emergency Feedwater Initiation and Control Emergency Feedwater Pump Emergency Feedwater System Emergency Diesel Generator Emergency Operating Procedure Engineered Safeguards Feedwater Pump Improved Technical Specifications Loss of Coolant Accident Modification Approval Record Loss of Offsite Power Nuclear Energy Institute Nuclear Regulatory Commission Offsite Power Transformer Progress Energy Florida, Inc.

Probabilistic Risk Assessment Reactor Building Station Blackout volt Improved Technical Specifications defined terms appear capitalized in LER text {e.g., MODE1 } Defined terms/acronyms/abbreviations appear in parenthesis when first used {e.g., Reactor Building (RB)}.

ATTACHMENT 2

LIST OF COMMITMENTS

The following table identifies those actions committed to by PEF in this document. Any other actions discussed in the submittal represent intended or planned actions by PEF. They are described for the NRC's information and are not regulatory commitments. Please notify the Supervisor, Licensing & Regulatory Programs, of any questions regarding this document or any associated regulatory commitments.

SECTION

No regulatory commitments are being made in this submittal.