IR 05000261/2010009

IR 05000261-10-009, on 04/19/2010 - 06/2/2010; H. B. Robinson Steam Electric Plant, Unit 2; Augmented Inspection

ML101830101
Person / Time
Site:	Robinson
Issue date:	07/02/2010
From:	Reyes L Region 2 Administrator
To:	Mccartney E Carolina Power & Light Co
References
IR-10-009
Download: ML101830101 (63)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION uly 2, 2010

SUBJECT:

H. B. ROBINSON STEAM ELECTRIC PLANT - AUGMENTED INSPECTION TEAM REPORT 05000261/2010009

Dear Mr. McCartney:

On June 2, 2010, the U.S. Nuclear Regulatory Commission (NRC) completed an Augmented Inspection at your H. B. Robinson reactor facility. The enclosed report documents the inspection results, which were preliminarily discussed on April 26, 2010, with Mr. Robert Duncan, Vice President Nuclear Operations, yourself, and other members of your staff. A public exit meeting was conducted on June 2, 2010.

The Augmented Inspection Team (AIT) was established to review the causes, safety implications, and your staffs actions for an event that occurred on March 28, 2010, which involved fires in electrical equipment, a reactor trip, a subsequent safety injection actuation, and an Alert emergency declaration. The team developed a sequence of events, reviewed related records, interviewed operators and individuals involved with the fire and plant response, and conducted walkdowns of affected areas. Initially, a Special Inspection Team (SIT) was dispatched to the site on March 30 to review the event. Additional information obtained by the SIT indicated the significance of the event was greater than initially understood, therefore the AIT was established.

Based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program, and the significance of this operational event, an NRC AIT was dispatched to the site on April 19, 2010, in accordance with Inspection Procedure 93800, Augmented Inspection Team. The inspection focus areas are detailed in the Augmented Inspection Charter (Enclosure 2). In addition to evaluating the facts and circumstances surrounding the event, the team reviewed the results of your companys event investigation which included cause determinations and proposed corrective actions.

The Augmented Inspection was chartered as a fact finding effort. It is not the responsibility of an AIT to determine compliance with NRC rules and regulations or to recommend enforcement actions. Therefore, the performance issues identified in this report will require additional NRC inspection and further review prior to determining what enforcement action, if any, is appropriate. In addition, the AIT was not charted to assess the adequacy of actions to restart

CP&L 2 the Robinson facility. The NRC continues to monitor and inspect ongoing activities associated with plant restart and the resolution of problems identified during the AIT.

The team concluded this event was initiated by an electrical fault in a cable which had limited or minimal design and installation margins. The event was further complicated by additional equipment malfunctions and ineffective operator actions that failed to mitigate an excessive cool down of the reactor coolant system. This resulted in an automatic safety injection actuation and an increase in the probability of reactor coolant pump seal failure due to the reduction in seal cooling capabilities. In addition, actions by the operators following the fire and during the post trip recovery, directly led to the second fire in electrical equipment and conditions that required an Alert emergency declaration.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosures will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Should you have any questions concerning this inspection, please contact us.

Sincerely,

/RA/

Luis A. Reyes Regional Administrator Docket No.: 50-261 License No.: DPR-23

Enclosures:

1. Inspection Report No. 05000261/2010009 w/Attachments:

2. Augment Inspection Team Charter dated April 16, 2010

REGION II==

AUGMENTED INSPECTION TEAM Docket No: 50-261 License No: DPR-23 Report No: 05000261/2010009 Facility: H. B. Robinson Steam Electric Plant, Unit 2 Locations: 3581 West Entrance Road Hartsville, SC 29550 Dates: April 19, 2010 - June 2, 2010 Team Leader: R. Haag, Chief Construction Projects Branch 3 Division of Construction Projects Inspectors: S. Currie, Reactor Engineering Examiner, NRR F. Ehrhardt, Senior Reactor Inspector, RII P. Fillion, Senior Reactor Inspector, RII J. Hanna, Senior Reactor Analyst, RII J. Hickey, Senior Resident Inspector, Robinson L. Miller, Senior Operations Examiner, RII R. Monk, Senior Resident Inspector, Watts Bar Unit 1 P. Pieringer, Reactor Operations Engineer, NRO Approved by: Leonard D. Wert, Director Division of Reactor Projects Enclosure 1

SUMMARY OF FINDINGS

IR 05000261/2010009, 04/19/2010 - 06/2/2010; H. B. Robinson Steam Electric Plant, Unit 2;

Augmented Inspection.

An Augmented Inspection Team (AIT) was dispatched to the site on April 19, 2010, to assess the facts and circumstances surrounding a plant trip that occurred on March 28, 2010. The AIT was established in accordance with NRC Management Directive 8.3, NRC Incident Investigation Program, and implemented using Inspection Procedure 93800, Augmented Inspection Team.

The inspection was conducted by a team of inspectors from the NRCs Region II office, senior resident inspectors from Robinson and Watts Bar, an inspector from the NRC Office of Nuclear Reactor Regulation (NRR), and an inspector from the NRC Office of New Reactors (NRO.) The team identified 14 issues that will require additional NRC inspection. These issues are tracked as unresolved items in this report.

NRC-Identified and Self-Revealing Findings

None

Licensee-Identified Violations

None.

EXECUTIVE SUMMARY H. B. Robinson Steam Electric Plant, Unit 2 NRC Inspection Report 05000261/2010009 The purpose of the AIT was to inspect and assess the facts and circumstances surrounding the March 28, 2010, plant event that involved fires in electrical equipment, a reactor trip and subsequent safety injection actuation, and an Alert emergency declaration. During this event two separate fires occurred approximately four hours apart.

The first fire was caused by a fault in a 4160 volt cable. The breaker designed to isolate and minimize the consequences of a fault at this particular location did not operate due to a long standing problem that had not been corrected. The magnitude of the fault caused damage to the unit auxiliary transformer which initiated a transfer of power source to the 4160 volt buses.

Because of the voltage decrease associated with the fault, one of two safety buses transferred to an emergency diesel generator power supply. Following isolation of the fault, two of the five 4160 volt buses and the equipment powered by these buses were lost.

An equipment malfunction and unexpected equipment response resulted in a reduction in reactor coolant pump (RCP) seal cooling capabilities and an increased probability of seal failure.

Component cooling water to the RCP thermal barrier heat exchangers was isolated when a valve closed during the transfer of power to the emergency diesel generator. Operators did not expect the valve to close, in part, because their training, including simulator training, failed to model the equipment response. The closed valve was not identified for 39 minutes, at which time it was opened. Seal injection to the RCPs was stopped due to low volume control tank level which occurred because the automatic transfer of the charging pumps suction source to the refueling water storage tank failed to occur. A prior modification to the control circuit was improperly implemented, which prevented the automatic transfer on low volume control tank level. Because cooling to the RCP thermal barrier heat exchanger was restored before seal injection stopped, cooling to the RCP seal was maintained throughout the event.

Control room operators failed to properly monitor and address key reactor parameters following the first fire. Following the reactor trip and unavailability of power to certain plant equipment, steam flow continued and was not controlled in the manner expected following a normal plant trip. The ongoing steam flow caused a cooldown of the reactor coolant system (RCS).

Operators failed to recognize the magnitude of the cooldown and corresponding decrease in RCS pressure and, as a result, a safety injection occurred. The cooldown was eventually stopped when the main steam isolation valves closed following an unexpected loss of a 120 volt instrument bus.

A second electrical fault and fire was caused by inappropriate recovery actions. Approximately four hours after the first fire, operators attempted to reset the generator lockout relay without first ensuring the cause of the lockout was cleared. This re-energized a bus damaged by the first fire and caused another electrical fault and fire, which resulted in significant damage to plant equipment. An Alert emergency classification was made due to a fire affecting the A and B DC buses.

This event was significantly complicated by equipment malfunctions and ineffective operator performance. The first fire was caused by a fault in a cable that did not meet many of the specifications in the design change which installed the cable. Equipment performance problems challenged RCP seal integrity due to reduced seal cooling capabilities. This increased the overall risk of the event. While the operators were slow to recognize the problems with RCP seal cooling, they eventually took actions that ensured seal cooling was maintained throughout the event. The AIT viewed the operators not recognizing the magnitude of the RCS cooldown and taking actions to mitigate to the cooldown as a major contributor to the overall significance of this event. The RCS cooldown was eventually stopped by an unexpected equipment occurrence, rather than expected operator actions.

The AIT concluded that the event did not adversely affect the health and safety of the public.

There were no measurable radiological releases associated with the event, and no Technical Specification Safety Limits were approached or exceeded.

The licensees event review investigation satisfactorily addressed the issues associated with the event. The licensees investigation contained observations and findings that were similar to those reached by the AIT following an independent review of the event.

REPORT DETAILS

Introduction and Charter The NRC conducts AIT reviews of significant operational events at facilities licensed by the NRC. The Robinson AIT was established to inspect the facts, conditions, and probable causes associated with the event that occurred during the evening of March 28, 2010. Initially the NRC dispatched a Special Inspection Team (SIT) on March 30, 2010, to review the event, but escalated to an AIT based on additional information which indicated the risk significance of the reactor trip and subsequent transient was higher than originally determined. This report contains the inspection results from both the SIT and AIT.

During this inspection team members analyzed information to determine the causes, conditions, and circumstances pertaining to the event. They independently reviewed computer data and control room chart recorder printouts and held discussions with many of the operators involved in the incident. One member of the AIT directly observed some of the recovery actions following the first fire, activities that led to the second fire, and subsequent emergency response activities.

The AIT directly examined plant equipment, observed investigative and maintenance activities, and discussed equipment operation with operators involved in the event. The inspection emphasized fact finding and the determination of probable causes. Additionally, two members of the AIT inspected remedial training provided to the operating crews to address performance problems identified during the event. Enclosure 2 of this report contains the AIT charter.

1.0 Description of Event Before the event, the Robinson electrical distribution system was in an at power configuration and the 4160 volt portion of the system was aligned with Buses 1 and 4 powered from the Unit Auxiliary Transformer (UAT), Bus 5 supplied from Bus 4 through breaker 52/24, Bus 2 supplied from Bus 1, and Bus 3 powered from the Start-Up Transformer (SUT). Figure 1 below shows a simplified schematic of the Robinson electrical distribution system. This figure, along with the system descriptions in Section 2.0 of this report, will aid in understanding the details of this event.

Figure 1, H. B. Robinson Simplified Electrical Distribution

On March 28, at 18:52 Eastern Standard Time (EST), a fault occurred in a 4160 volt (V)

(referred to as 4 kV for the remainder of the report) feeder cable from Bus 4 to Bus 5.

The output breaker in Bus 4 that feeds Bus 5 (breaker 52/24) failed to open as designed because control power was not available to the breaker. The fault caused Bus 4 voltage to lower, which decreased the speed for the RCP that was powered from Bus 4. This caused the flow in RCS Loop B to decrease and initiated an automatic reactor trip. The UAT was internally damaged from the high fault current and generated a sudden pressure trip signal. This caused the generator lockout relays to actuate and resulted in the main generator output breakers opening and the fast transfer of power to Bus 4 from the UAT to the SUT. Following the fast transfer, power to Bus 4 was supplied by the SUT via breaker 52/19.

The fault condition persisted following the fast transfer, causing a decrease in voltage on Bus 3 and 480 V safety-related Bus E-2, which is fed from Bus 3. This under voltage condition caused Bus E-2 to separate from Bus 3, the B Emergency Diesel Generator (EDG) to automatically start and supply power to Bus E-2, and loads to sequence on to Bus E-2. The under-voltage condition on 4 kV Bus 3 also caused the 480 V Bus 3 main supply breaker to open, de-energizing Motor Control Center (MCC) 4.

Several seconds following the fast transfer, breaker 52/19 tripped open on over-current.

This cleared the fault and ended the first fault event. The sequence of events for the first fault took place over a 12 second period of time (18:52:22 to 18:52:34.) Figures 2 and 3 below are photographs showing the damage sustained to cables and conduits as a result of the first electrical fault. Following the first fault, the configuration of the electric plant was as follows:

• 4 kV Buses 1, 2, and 3 were powered from the SUT

• 480V Bus E-2 was powered by the B EDG

• 4 kV Buses 4 and 5 were de-energized

• 480V Buses 3, 4, and 5 were de-energized

• MCC 4, 8, 11, 12, 13, 14, 15, 17, and 21 were de-energized After the disruption to the electric plant and reactor trip, a series of equipment problems and operator performance issues increased the overall significance of the event. The following equipment conditions existed after the reactor trip:

• When Bus E-2 momentarily lost power, FCV-626, the CCW return valve from the RCP thermal barrier heat exchanger closed and isolated CCW flow to all the RCP thermal barriers.

• When MCC 4 de-energized, all Moisture Separator Reheater (MSR) Drain Tank Alternate Drain valves and MSR Timer valves failed open, providing a flow path for main steam to the main condenser via the MSR Shutoff valves and MSR reheater tubes. This steam flow resulted in a cooldown of the RCS. Additionally, power was unavailable to the MSR Shutoff valves, preventing the valves from being remotely closed from the control room.

Operators started two charging pumps at 18:53 as directed by the emergency operating procedures (EOPs), but failed to properly monitor pressurizer level or RCS temperature.

At 19:00 an automatic safety injection (SI) occurred due to low RCS pressure caused by the cooldown. The suction of the charging pumps did not transfer from the volume

control tank (VCT) to the refueling water storage tank (RWST) as VCT level decreased (setpoint is 12.4 inches of water in VCT) because of an equipment failure. Operators did not identify this failure to automatically transfer for 46 minutes. At 19:24 the B RCP high bearing temperature alarm was received in the control room. At 19:25 power was lost to 120 V Instrument Bus 3. The loss of power to Instrument Bus 3 satisfied the logic for the Main Steam Isolation Valves (MSIVs) to close and ended the on-going release of steam to the condenser, stopping the RCS cooldown.

A high bearing temperature alarm for the A RCP was received at 19:30. At 19:31 operators identified that FCV-626 was closed and re-opened the valve, which restored CCW cooling to the RCP thermal barrier heat exchangers. FCV-626 was closed for 39 minutes before operators recognized this condition and reopened the valve. The post event review indicated that charging and seal injection flow was lost at 19:37 because the VCT was empty. The AIT noted that RCP cooling was maintained during this event because FCV-626 was opened six minutes before seal injection flow was lost. At 19:46 operators identified that the suction of the charging pumps did not automatically transfer from the VCT to the RWST. At 19:51 operators in the control room manually aligned the suction of the charging pumps to the RWST.

The second electrical fault and fire, which also damaged plant equipment, was caused by inappropriate recovery actions following the first fire. At 21:26 operators transitioned from the EOPs to General Plant Procedure (GPP)-004, Post Trip Stabilization. At 22:34 operators attempted to reset the generator lockout relays per GPP-004. Because the UAT sudden pressure trip signal still existed, attempting to reset the generator lockout relays caused another fast transfer of 4 kV Bus 4 from the UAT to the SUT, reenergized Bus 4, and caused a fault at breaker 52/24. The electrical fault and corresponding arc flash damaged surrounding equipment. Breaker 52/19 opened on over-current, cleared the fault, and ended the second fault event. Figure 4 below is a photograph showing damage to Bus #4 resulting from the second fault event.

Following the second fault, alarms received in the control room indicated that both safety-related 125 volt DC battery buses had grounds. The shift manager subsequently declared an Alert emergency action classification based on a fire affecting safety systems required to establish or maintain safe shutdown. The licensee notified off-site organizations of the Alert classification within the required timeframes and properly established the emergency response facilities. The Alert was terminated at 01:46 on March 29 when the termination criteria were met.

Figure 2 - Damaged Conduits from Bus #4 to Bus #5 Figure 3 - Damaged Conduits Entering the Top of the Bus #

Figure 4 - Back of Bus #4 Containing Breaker 52/

1.1 Risk Significance of Event Management Directive 8.3, Incident Investigation Program, describes the NRCs process for investigating significant operational events involving reactor facilities. This document specifies a risk-informed approach for determining when the agency will commit additional resources for further investigation of an event. The risk metric used for this decision is conditional core damage probability (CCDP), a measure of the instantaneous risk of core damage. To evaluate this event, the NRC analysts used the Standardized Plant Analysis Risk (SPAR) Model for H. B. Robinson, Revision 3.50 dated March 9, 2009.

Subsequent to initiating the SIT, the NRC learned of several operational errors made during the event response and of the potential loss of seal cooling to the reactor coolant pumps. The analysts modeled this condition in the SPAR model by reflecting that CCW flow to the reactor coolant pumps was secured concurrent with a failure of seal injection.

As this would produce an overly conservative result not reflective that seal cooling/injection was maintained during the actual event, the analysts modeled a recovery action based on the following factors: 1) 20 minutes to seal damage on a loss of seal cooling, 2) elevated operator stress, and 3) poor work practices. Based on this additional information the dominant accident sequence became an RCP seal LOCA followed by operators failing to execute ECCS recirculation. An initiating event assessment performed in the Graphical Evaluation Model mode yielded a CCDP estimated to be 4.2E-5 indicating that the event was of substantial risk significance and warranted conducting an AIT.

2.0 System Descriptions 2.1 On-site Power Distribution System The on-site power distribution system has two primary functions: 1) provide a means of transmitting the main electrical generator output to the distribution grid for use by consumers and 2) provide electrical power for plant components during normal, shutdown, and abnormal conditions.

The main electrical generator serves as the primary source of auxiliary electrical power when the unit is on-line. Power from the main electrical generator is supplied via the UAT, to several 4 kV buses. When the main electrical generator is not available, the SUT serves as the primary source of power to the 4 kV buses. The 4 kV buses supply power to components with large motors such as reactor coolant pumps and feedwater pumps. All 4 kV buses at Robinson are classified as non safety-related. The 4 kV buses also supply 480V buses via step down station service transformers. Two safety-related EDGs, A and B, are available to supply power to critical components on the 480V E-1 and E-2 buses if a loss of off-site power occurs. Additionally, a dedicated shutdown diesel generator (DSDG) is available to power dedicated shutdown components on the dedicated shutdown (DS) bus if all the other power supplies are not available. The 480V buses supply components such as CCW pumps, charging pumps and SI pumps as well as MCCs. The MCCs supply power to smaller loads such as motor operated valves, instrument buses and lighting panels.

Robinson has two primary DC buses, A and B. The DC buses supply both safety-related and non-safety-related loads. The non safety-related loads are separated from the safety-related bus by a safety-related breaker which is designed to open and isolate any faulted non-safety-related component from safety related loads.

2.2 Chemical and Volume Control System (CVCS)

The functions of the CVCS include: 1) maintaining RCS water inventory, 2) adjusting soluble boron concentration in the RCS coolant to control reactor power, 3) providing for continuous cleanup of the RCS coolant, and 4) supplying seal injection to the RCPs.

These functions are accomplished by continuously supplying and removing a portion of the reactor coolant from the RCS in the closed loop CVCS system. The charging portion of the CVCS system takes coolant from the VCT and returns it to the RCS via one or more of three positive displacement charging pumps. VCT level is automatically maintained by a level control system. If VCT level is not maintained by normal system interactions and decreases to 12.4 inches, the charging pump supply automatically shifts from the VCT to the RWST. The RWST is a very large makeup water supply and ensures a constant source of coolant is available for charging.

Charging flow is returned to the RCS via two paths: 1) directly to an RCS loop, and 2)indirectly to the RCS via RCP seal injection. Seal injection flow enters the RCP between the thermal barrier assembly and the pump bearing. A portion of the seal injection flow returns to the RCS after passing down through the thermal barrier assembly. Another portion of the seal injection flow returns to the VCT after flowing up past the pump bearing and #1 seal, providing lubrication and cooling for the bearing and seal. The #1 seal is one of three shaft seals which provide a barrier between the RCS and the containment atmosphere. If seal injection flow is lost, RCS coolant flows up past labyrinth seals in the thermal barrier assembly and is cooled as it passes over the thermal barrier cooling coils.

2.3 Component Cooling Water System The primary function of the CCW system is to remove heat from safety-related components and transfer that heat to the service water system. The CCW system also provides continuous cooling water flow to the individual RCP thermal barrier heat exchangers. As described above in section 2.2, the RCP #1 seals are normally cooled and lubricated by continuous seal injection flow. If seal injection flow is lost, RCS water will flow up past the thermal barrier heat exchanger to the #1 RCP seal. However, RCS water normally must be cooled by the thermal barrier heat exchanger prior to reaching the #1 RCP seal in order to prevent the seal from overheating and sustaining damage.

Therefore, to prevent damage to the #1 RCP seal either seal injection from the charging system or CCW flow through the RCP thermal barrier heat exchanger must be available.

CCW from each of the RCP thermal barrier heat exchangers is returned to a CCW heat exchanger via a common return line. This line exits containment and can be isolated by motor operated valve FCV-626, Thermal Barrier Outlet Isolation Flow Control Valve.

FCV-626 automatically closes on high CCW flow coming from the RCP thermal barrier heat exchangers or a Containment Isolation Phase B signal. The high CCW flow closure is provided to protect against a failed thermal barrier heat exchanger and the resulting

flow of RCS coolant into the CCW system.

3.0 Operations The team conducted an overall and independent review of Operations to determine if licensee staff responded properly during the event, procedures were adequate, the plant reference simulator was modeled correctly, and to better understand the licensees decision making process. The following areas were specifically addressed and are discussed in more detail in the following sections.

• The licensees response to the reactor trip, SI actuation, and the isolation of DC grounds.

• Use, coordination, and adequacy of emergency, abnormal, and alarm response procedures, as well as other plant operational procedures that were used or should have been used during the event.

• Staffing conditions and requirements in the control room and their potential contribution to the operational errors that occurred.

• The extent and effectiveness of operator training with respect to the concurrent performance of emergency and abnormal procedures related to fires.

• Differences between the plant control room and the plant reference simulator that may have impacted operator actions.

• The licensees decision making process for the events leading up to the second fire event.

Additionally, the team reviewed the licensees corrective actions, causal analysis, and extent of condition, with respect to Operations.

3.1 Operator Performance

a. Inspection Scope

The team conducted an independent review of control room activities to determine if licensee staff responded properly during the events. With respect to operator awareness and decision making, the team was specifically focused on the effectiveness of control board monitoring, communications, technical decision making, and work practices of the operating crew. With respect to command and control, the team was specifically focused on actions taken by the control room leadership in managing the operating crews response to the event. The team performed the following activities in order to understand and/or confirm the control room operating teams actions to diagnose the event and implement corrective actions:

• Conducted interviews with control room operations personnel on shift during the event.

• Reviewed procedures, narrative logs, event recorder data, system drawings, and plant computer data.

• Observed a simulated plant response to this event as demonstrated on the plant reference simulator.

• Reviewed the crews implementation of emergency, abnormal, and alarm procedures as well as Technical Specifications.

• Reviewed Operations administrative procedures concerning shift manning and procedure use and coordination.

• Reviewed Operations procedures in use at the time of the second fire.

b. Observations and Findings

The team determined that operators exhibited weaknesses in fundamental operator competencies when responding to the event. Specifically, the team determined that the operating crew did not identify important off-normal parameters and alarms in a timely manner, resulting in a failure to recognize an uncontrolled RCS cooldown and a potential challenge to RCP seal cooling.

Additionally, the team determined that crew supervision did not exercise effective oversight of plant status, crew performance, or site resources.

Monitoring of Plant Parameters and Alarms Through a review of plant data, the team determined that the crews response to the first event was not effective in stabilizing the plant. Through interviews and review of plant data, the team determined that the crew did not recognize the magnitude of the RCS cool down caused by an on-going steam demand. The RCS cool down rate exceeded the limit of 100o/hr in any one hour period as specified in Technical Specification (TS)3.4.3, RCS Pressure and Temperature (P/T) Limits. The fact that the RCS cooldown rate exceeded the limiting value specified in TS 3.4.3, and the requirement to evaluate the actions contained in TS 3.4.3, was not recognized by the crew at any time during or after the cooldown. Based on interviews, the Reactor Operator (RO) and Control Room Supervisor (CRS) assessed the cool down rate as being consistent with what was experienced during simulator training for an RCP trip followed by a reactor trip. The RCS cool down continued until Instrument Bus 3 was inadvertently de-energized (approximately 33 minutes after the start of the first event), which caused the MSIVs to close, isolating the steam generators from the steam header.

Based on the sequence of events, a review of plant data, and operator interviews, the team concluded that the crew did not recognize that VCT level was decreasing, a low VCT level alarm had annunciated, and automatic swapover of the charging pump suction from the VCT to the RWST failed to occur, until indicated level in the VCT had decreased to approximately 2-3 inches and charging flow had degraded. Once the crew identified this condition, the RO attempted to manually align the suction of the charging pumps to the RWST but made an error when performing the alignment. The error left the suction of the charging pumps aligned to the VCT. The Shift Technical Advisor (STA) determined the alignment was incorrect and the RO corrected the error. The crew did not reference APP-003-E3, VCT HI/LO LVL, which provided direction to manually transfer the charging pump suction to the RWST. RCP seal cooling was maintained because the crew reopened FCV-626 to restore CCW cooling to the RCP thermal barrier heat exchanger approximately 6 minutes before depletion of the VCT. However, high pump bearing temperature alarms were received on all three RCPs. The high temperature alarms subsequently cleared after operators reopened FCV-626.

Based on operator interviews, the team concluded that, following implementation of Emergency Operating Procedures (EOPs), the operators did not complete a satisfactory

review and evaluation of alarm conditions prior to the second event. Instead, the operating crew entered GP-004, Post Trip Stabilization, and attempted to reset the generator lockout relay without using the information in the Annunciator Panel Procedures (APPs) to completely and accurately assess abnormal electric plant status.

GP-004 is a normal operating procedure and is written with the assumption that the plant is in a normal (undamaged) configuration. The crew was not aware that a sudden pressure fault signal from the UAT was still applied to the generator lockout circuit logic, as indicated by a locked in UAT fault trip alarm (APP-009-B6, AUX TRANSF FAULT TRIP). The attempted reset reenergized Bus 4 and caused a fault at breaker 52/24, initiating the sequence for the second fire. The team concluded that if the crew had performed a thorough control board walkdown, additional electric plant APPs and/or AOPs could have been identified and implemented before exiting to a normal operating procedure (GP-004).

Additional review by the NRC will be required to determine if the licensees programs resulted in untimely identification and investigation of abnormal plant parameters and unexpected main control room alarms. This review will also determine whether the crews monitoring of plant parameters and alarms, and use of associated procedures, represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. The URI is identified as05000261/2010009-01, Monitoring of Plant Parameters and Alarms. Additionally, further review by the NRC will be required to determine if the RCS cooldown rate exceeding the limiting value specified in TS 3.4.3 represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. The URI is identified as05000261/2010009-02, RCS Cooldown Rate Exceeds Technical Specification 3.4.3 Limit.

Command and Control Based on interviews and a review of plant data, the team determined that the Shift Manager (SM) and STA became distracted from oversight of the plant, which includes awareness of major plant parameters such as RCS temperature and pressurizer level, during the first event. The team determined that this situation occurred, in part, due to the SM and STA becoming engaged in simultaneous Emergency Action Level review.

Based on interviews, the team determined that the SM did not effectively manage the frequency and duration of crew updates and crew briefs during the early portion of the event. Crew updates became so frequent that they interrupted the implementation of emergency procedures and distracted the operators from timely progression through the Path-1 EOP.

Based on interviews and a review of plant data, the team concluded that the SM and CRS did not ensure monitoring and diagnosis of key major plant parameters, such as RCS temperature, pressurizer level, and VCT level, by other control room crew members.

The team determined, through interviews, that the CRS was unaware that an Auxiliary Operator (AO) assigned to the shift, but not assigned to the Fire Brigade, was available to perform local operator actions contained in the Path-1 EOP. As a result, the B battery charger did not get restarted within 30 minutes of power loss as required by Path-1 (Restart battery chargers within 30 minutes of power loss using OP-601). Per the

Path-1 basis document, this step is a plant specific step relative to the design of the station batteries and the batteries at Robinson are rated for a one hour duty cycle. In order to ensure that the batteries are not completely discharged, the procedure directs restart of the battery chargers within 30 minutes. From plant data, the team determined that the B battery charger was de-energized for 39 minutes.

Based on a review of plant data, the team concluded that the management expectation for establishing positive control of equipment configuration was not implemented by the operating crew. For example, the supply breaker to faulted 4kV Bus 5 was not tagged open following the first event and the fire detection system for the 4kV switchgear room was not reset after the fire watch was secured.

Through interviews and a review of plant data and alarm response procedures, the team determined that the SM and CRS did not ensure that sufficient information necessary to assess abnormal electric plant status was collected and evaluated prior to performing steps within a procedure that assumed a normal electric plant configuration. This observation is discussed in more detail in Section 3.1, Operator Performance.

Based on interviews, the team determined that the SM did not use technical resources available in the Outage Command Center (OCC) for performing an assessment of damage to the electric plant before the crew reset the generator lockout relays. The team noted that such an assessment may have determined that reset of the generator lockout relays should not have been attempted, in part because the UAT sudden pressure condition, which caused the generator lockout, was still present.

Resource Utilization Through interviews, the team determined that the Balance of Plant (BOP) operator concurrently performed Abnormal Operating Procedure (AOP)-041, Response to Fire Event, during the first event. The team observed that AOP-041 contains numerous steps to coordinate on-site and off-site fire brigade response and notifications. The team determined that having a licensed operator perform AOP-041, concurrent with the CRS and RO performing emergency operating procedures, is a licensee expectation in accordance with OMM-022, Emergency Operating Users Guide. Through interviews, the team determined that because the BOP operator was performing AOP-041, he was unavailable to assist the control room team in recognizing and diagnosing off-normal events and conditions for approximately the first 30 minutes of the first event.

During interviews, the two operators responsible for panel operation (the RO and CRS)consistently noted the unavailability of a third person (the BOP licensed operator) to perform independent panel checks. The team noted that during conditions of minimum manning, using the BOP operator to concurrently perform certain AOPs may hinder or prevent him or her from assisting the CRS and RO in stabilizing the plant during events that challenge the control room crew. Additional review by the NRC will be needed to determine if the licensees utilization of operators, during conditions of minimum control room manning, is adequate during complex events. This review will also determine if this issue represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. The issue will be identified as URI 05000261/2010009-03, Utilization of Operators During Events Requiring Use of Concurrent Procedures.

3.2 Training and Simulator

a. Inspection Scope

The team conducted interviews with control room operations personnel on shift during the event and observed a simulated plant response to this event as demonstrated on the plant reference simulator. As part of the interviews, the operators were asked how their training affected their actions during the event. Their feedback provided the basis for further evaluation in the following areas:

• The team assessed the use and impact of simulator training on operator performance. The team reviewed simulator lesson plans containing elements similar to those experienced in the event.

• The team evaluated corrective actions related to simulator training and reviewed the following remedial simulator scenarios:

- Remedial simulator scenarios #1, Lesson ID REMTRG1, Course Code LOC0015R, 4/22/2010

- Remedial simulator scenarios #2, Lesson ID REMTRG2, Course Code LOC0015R, 4/22/2010

- Remedial simulator scenarios #3, Lesson ID REMTRG3, Course Code LOC0015R, 4/22/2010

- Static Scenario #1, Lesson ID SSS5, Course Code LOC0015R of 4/22/2010

• The team assessed crew performance by reviewing simulator crew evaluation forms from the period between February, 2008 and February 2010. The team also reviewed individual requalification examination reports from February 11, 2010.

b. Observations and Findings

The team concluded that training contributed to an incomplete understanding of the plant response during the event by the crew.

Simulator fidelity A review of simulator performance and event data by the team confirmed one simulation deficiency which had been identified by the licensee as part of their event review. When power to safety-related 480 volt Bus E-2 was transferred to the emergency diesel generator, FCV-626 (thermal barrier heat exchanger outlet isolation flow control valve)closed unexpectedly. As discussed in more detail in the Section 4.5, Unexpected Closure of FCV-626, the as-built plant configuration resulted in the valve closing on a loss of power. This response was not obtained in the simulator because the simulator modeling of FCV-626 was based solely on CCW flow through the valve and did not take into account power to the valve operator and associated control circuit. Consequently, in simulator scenarios which included a loss of power to Instrument Bus 4, this valve remained open.

Because the plant reference simulator did not demonstrate expected plant response for a loss of Instrument Bus 4, the team identified the need for additional NRC review to determine the adequacy of fidelity of the plant reference simulator for conducting loss of

component cooling system control manipulations and plant evolutions. This review will also determine if this issue represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. The URI is identified as05000261/2010009-04, Fidelity of Plant-Referenced Simulator.

Operator knowledge The team identified one potential cognitive bias displayed by operators during the event.

This bias appears to have been introduced during a recent training cycle. During operator interviews, two crew members specifically stated that they knew safety injection was imminent for the event because the B RCP was not running. When questioned as to how they knew this, they indicated that the RCS cooldown rate during the event was consistent with what they had seen on the simulator. One operator specifically mentioned that safety injection is an expected response following loss of an RCP with a subsequent reactor trip. These operators believed a safety injection was normal for the plant conditions experienced (loss of an RCP) and did not seek out information that would disprove this belief. From the teams evaluation of the sequence of events, it appeared that an expedient response to the RCS cool down rate would have minimized the potential for an automatic safety injection.

From interviews with training personnel and a review of training scenarios the team identified a simulator scenario, run within the last year, involving loss of an RCP. This simulator scenario, LOCT 05-3, emphasized that loss of an RCP would result in a cool down and subsequent safety injection due to high steam line differential pressure.

During the scenario a reactor trip from 50% power was initiated and, at this power level, the reactor trip and loss of an RCP resulted in a safety injection. From interviews and review of additional training lesson plans, the team determined that in the majority of training scenarios when RCP conditions degrade, the crew rapidly reduced reactor power in anticipation of needing to trip the unit off-line. After a power reduction, other conditions were then introduced in the scenarios to cause the crew to manually trip the reactor and secure the RCP. The lower power condition that is trained on results in a high steam line differential pressure safety injection. However, the plant conditions simulated in these training scenarios are different than the plant conditions experienced during the March 28 event. Plant conditions at full power actually decrease the potential for a safety injection. The team found no evidence that operator training had included a discussion or simulation of the plant response to similar failures at higher power levels.

The simulator training facilitated the operators assumption that the cool down and safety injection experienced during the event were understood and acceptable.

Simulator Scenario Complexity Through interviews, the team determined that simulator scenarios as complex as the events that occurred have not been presented to operators during continuing training.

The team concluded that not doing so was a missed opportunity to challenge procedures, simulator fidelity, and operator expertise so that weaknesses and improvements were proactively identified. The team observed that the licensee had not conducted scenarios that required the BOP operator to perform concurrent procedures (that significantly decreased the BOPs involvement with EOPs and control board monitoring for extended periods) in parallel with other tasks. Conducting such scenarios would potentially have identified risks or vulnerabilities in operating strategies.

Performing this type of training could also have provided valuable feedback to operations supervisors regarding the nature of the challenges to command and control they might encounter during these types of events. The team noted that the three simulator training scenarios developed as part of the remedial training package to be conducted for all operating crews contained this degree of complexity.

Effectiveness of Operator Performance Feedback To assess the extent of condition for the operator performance issues demonstrated during this event, the team reviewed a sample of simulator crew evaluation forms spanning the period of February 2008 to February 2010. The team identified multiple examples of operating crew weaknesses identified by training, relative to monitoring and control of major plant parameters. Of the six packages reviewed, four contained comments summarized as follows:

• February 27, 2008 - unaware of steam dumps open; no attempt at RCS temperature control

• March 3, 2008 - crew not clear if steam dumps actuated

• February 19, 2009 - pressurizer level control post-trip was not anticipated; S/G level control needed improvement

• February 24, 2009 - slow to identify steam dump malfunction; post- trip trends of associated parameters not provided The team noted that even though the evaluations highlighted the operators responsibility for monitoring and controlling major plant parameters, this emphasis was not effective in achieving the level of performance necessary to stabilize the plant following the uncontrolled cooldown that occurred during this event. The team concluded that additional inspection is warranted to determine if the licensees corrective action program is effective in capturing and addressing operating crew performance weaknesses. The team noted that the licensee also identified this issue regarding operating crew performance standards as part of their event investigation. This review will also determine if this issue represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. The URI is identified as05000261/2010009-05, Corrective Action for Operating Crew Performance Issues.

3.3 Procedure Content and Use

a. Inspection Scope

The team reviewed the licensees implementation of emergency, abnormal, alarm and normal operating procedures used during the event. The review focused on the effectiveness of the procedures in addressing the event and technical accuracy of the content. The EOP network was compared with the Westinghouse Owners Group (WOG) emergency procedure guidelines. Use of procedures was observed in a simulated plant response to this event as demonstrated on the plant reference simulator.

Information obtained from interviews with the operating crew, system descriptions, event sequence of event, plant computer data and narrative logs were used to support these reviews.

b. Observations and Findings

Emergency Operating Procedures From interviews, the team determined that the control room operators, in responding to the event, relied exclusively on actions and guidance explicitly described in EOPs. The operators did not consider mitigating actions that would have stabilized the plant that were not explicitly contained in these procedures, such as shutting the MSIVs. The emergency procedures being implemented centered on the Path-1 EOP.

From a review of the plant procedures used by operators to respond to this event, the team determined that certain Path-1 procedure steps required operators to rely on their knowledge because these steps did not contain detailed (rule-based) guidance. The team observed that Path-1 is a flow diagram that assists with diagnostics but does not consistently provide acceptance criteria and alternate actions. The team determined that, in general, implementation of the Path-1 EOP relies more heavily on operator knowledge-based behavior versus the rule-based behavior emphasized in WOG Emergency Response Guidelines. The team noted that common industry practice among Westinghouse technology plants is to utilize a two-column page format for EOPs and to also provide more explicit detail regarding specific parameters to be checked and specific components to manipulate within each step.

The team observed that EOPs did not contain explicit guidance to fully isolate ongoing steam flow in all cases. For example, End Path Procedure (EPP) Foldout A Step 6 MSR Isolation Criteria does not contain additional contingency actions in the event the specified action cannot be taken or is not effective (i.e. loss of power to MSR steam supply valves). During interviews, operators stated that they had been trained in the simulator to send local operators to close MSR valves as a contingency action.

However, this action is not listed in the Foldout A procedure and no additional or alternate action that could be performed from the control boards, such as closing the MSIVs, is specified. Additionally, Path-1 Turbine Tripped does not contain additional steps that operators might be reasonably expected to take in order to accomplish the intent of the step, such as closing the MSIVs, in the event that the specified contingency actions of manually tripping the turbine and running back the turbine are not successful.

The team also identified an inconsistency between the Path-1 Basis Document and the licensees emergency operating procedure users guide regarding the immediate operator action of SI Initiation. Path-1 EOP does not explicitly list parameters or conditions to be checked in order to determine if a safety injection is required (requiring both the operator performing the immediate action and the CRS who is reading the procedure to rely on their knowledge). However, the Path-1 Basis Document provides an interpretation of this step that states, in part, that a safety injection is required if RCS inventory is decreasing in an uncontrolled manner and exceeding all available makeup flow. OMM-022, Emergency Operating Procedures Users Guide Section 8.3.1, Item 10, lists parameters and values that operators are expected to check when performing this immediate action step. The team noted that this step in OMM-022 does not specify checking RCS parameters directly related to RCS inventory, such as pressurizer level, as described in the Path-1 basis document.

The team reviewed plant data from the first event and determined that pressurizer level decreased off-scale. Based on interviews, the team also determined that operators did not recognize the magnitude and rate of the pressurizer level decrease caused by the ongoing RCS cool down. Consequently, the team identified the need for additional NRC review to determine the adequacy of OMM-022 with respect to the immediate operator action of checking whether a safety injection is required. This review will determine whether the inconsistency between the Emergency Operating Procedures Users Guide and the Path-1 Basis Document is a performance deficiency. An Unresolved Item will be opened pending completion of this review. The URI is identified as05000216/2010009-06, Adequacy of Emergency Operating Procedure Background Documents.

Other Operating Procedures The team observed that procedure GP-004 Post Trip Stabilization contained a step to reset the generator lockout relays but did not contain steps, cautions, or notes that prompt operators to ensure the inputs are clear prior to attempting a reset.

Although AOP-024, Loss of Instrument Buses was not used, and was not required to be used per the licensees procedure use guidelines during this event, the team noted that the procedure does not address the effect of a loss of an instrument bus on the main steam flow channels that input into the Main Steam Line Isolation Signal.

Additionally, AOP-024 does not address the loss of CCW flow to the RCP thermal barrier heat exchangers (FCV-626 closure).

The team reviewed the circumstances which resulted in the fire in and subsequent failure of the A Main Condenser Vacuum Pump. The pump failed because seal water to the pump, which is supplied by demineralized water, was lost for approximately three and a half hours prior to the pump failure. The loss of power following the first fire caused the loss of demineralized water. The Main Condenser Vacuum Pump establishes and maintains condenser vacuum to provide a heat sink used for decay heat removal following a reactor trip. The team observed that the licensee does not have a procedure to address loss of seal water makeup to the main condenser vacuum pumps.

Use of such a procedure could have prevented the fire and associated damage to this equipment. As a result of this observation, the team identified the need for additional NRC review to determine if procedures should have been available to address a sustained loss of seal water makeup to the main condenser vacuum pumps. Additional review by the NRC will be needed to determine whether the lack of a procedure for loss of seal water to the main condenser vacuum pumps is a performance deficiency. An Unresolved Item will be opened pending completion of this review. The URI is identified as05000261/2010009-07, Loss of Seal Water Results in Failure of the A Main Condenser Vacuum Pump.

3.4 Licensee Investigation and Corrective Actions (Operations)

a. Inspection Scope

The team reviewed portions of the licensees Significant Adverse Condition Investigation Report for the event related to Operations in order to 1) independently assess the licensees investigation of the event, 2) verify the licensee had appropriately reviewed

plant and operator response to the event, and 3) to assess the adequacy of the licensees corrective actions.

Additionally, following the event, the licensee developed and conducted week-long remedial training for each operating crew. The training was designed to address operational performance gaps demonstrated during the event, including annunciator panel procedure use, control board monitoring, command and control, and concurrent use of AOPs when performing EOPs. NRC inspectors observed portions of the remedial event response training provided to three operating crews, including classroom and simulator training and simulator evaluations.

b. Observations and Findings

The team concluded that the licensees investigation report for this event adequately addressed the Operations issues identified during this event. The team verified the licensee adequately explained operator response to the event and identified factors contributing to operator performance. The team determined that the licensee adequately identified and documented causes specific to the event as well as immediate and proposed corrective actions for identified discrepancies. The team concluded that the remedial training provided to operating crews was effective in identifying and correcting individual and crew response associated with operational performance gaps.

4.0 Equipment Performance 4.1.1 Fault in Cable between 4 kV Bus 4 and Bus 5

a. Inspection Scope

To determine the circumstances surrounding the fault in the cable that led to the first electrical disturbance and subsequent reactor trip, the team performed the following activities:

• Determined details of the cable construction including conductor size, insulation thickness and material, and type of shielding using the manufacturers data sheet

• Compared the cable construction to system requirements and standard industry practice

• Reviewed relevant portions of the plant modification that installed the cable

• Viewed the site of the cable fault

• Reviewed cable records to determine where other similar cables are installed in the plant

• Interviewed engineering staff involved with the electrical distribution systems and components

• Reviewed the licensees causal analysis for the cable fault

• Evaluated the licensees proposed corrective actions

b. Observations and Findings

The cable that faulted did not meet many of the specifications for the design change that installed the cable. This contributed to the cable failure. The cable, manufactured by the Rome Cable Corporation (Rome), was installed in 1986 when 4 kV Bus 5 was installed

as an extension of Bus 4 per Plant Design Change Number DCN-851. The cable, identified as C21344A, served as the interconnection between 4 kV Buses 4 and 5 and was comprised of two conductors for each of three phases. The cable was installed in two steel conduits, with each conduit containing all three phases. As noted in Section 2.1, all 4 kV buses at Robinson are non safety-related.

The Bill of Materials for DCN-851 indicated that the cable should be in accordance with Standard Specification L2-E-035 for 5,000 Volt Power Cable. However, the Bill of Materials did not indicate a purchase order number for the cable that faulted, as it did for other cables installed by the modification, such as 3/c No. 12 AWG cable. Records reviewed by the team indicated that the cable came from reel number HBR-13505.

Differences between Standard Specification L2-E-035 and the actual installed Rome cable are listed below:

• L2-E-035 called for coated copper conductors. The installed cable had uncoated conductors.

• L2-E-035 called for all cables to be provided with an outer jacket. The installed cable did not have a jacket.

• L2-E-035 called for cable insulation and jacketing that was self-extinguishing and non-propagating with regard to fire as described in IEEE 383-1974, Type Test of Class 1E Electric Cables, Field Splices, and Connections for Nuclear Power Generating Stations. The Rome catalogue data made no claim as to fire propagation properties. The event demonstrated that the cable lacked fire propagation properties because 1) the cable ignited following the fault, 2) the cable did not self extinguish after the fault was denergized, and 3) flame was propagated along the cable.

• L2-E-035 called for 133 percent insulation level and insulation shielding if specified in the purchase order. The installed cable did not have either of these features.

The cable consisted of single conductor 500 MCM uncoated copper with 130 mils of cross-linked polyethylene insulation rated for continuous operation at 5 kV and 90 degree Celsius. The insulation thickness was determined from the overall cable diameter and from the licensees measurement of conductor diameter. The manufacturers catalogue information (SPEC 7155 dated January 1, 1991) stated that an insulation thickness of 120 mils is suitable for applications requiring 100 percent insulation levels. However, due to the high-resistance grounding scheme used on the Robinson 4 kV electrical system, an insulation level of 133 percent or 173 percent was required, depending on how long a ground fault could remain on the system. The significance of not having adequate insulation thickness was that, should a single line to ground fault occur the voltage on the two un-faulted phases would exceed the rating of the insulation.

The cable did not have a jacket. The significance of not having a jacket was that the cable insulation was more vulnerable to damage during installation. Also, the jacket, if installed, would have provided a buffer between the insulation and grounded metal parts, such as the conduit or bus enclosure.

The cable did not have an insulation shield. When an insulation shield is not installed, the electric field will be partly in the insulation and partly in whatever lies between the insulation and ground. This situation could be conducive to corona if a thin layer of air lies between the insulator surface and ground, which can lead to insulation deterioration.

IEEE 666-1991, Design Guide for Electric Power Service Systems for Generating

Stations, Section 12.3.6 states: Power cables rated 5 kV and over should be equipped with insulation shield. The significance of not having a grounded insulation shield was that voltage stress on the insulation was not symmetrical and uniform around the circumference, but rather greater at points where the insulation contacted a grounded surface, such as a metal conduit, than at other points around the circumference.

The following information indicated that shielded cable was originally intended for this cable: 1) Design Change Notice No. 6 to DCN-851 changed the termination detail from one depicting the grounding of shield wires to one with no shield wires, and 2) installation instruction 4.35 directed installation of a stress cone for cable C21344A, which would be needed only for a shielded cable.

Cognizant licensee engineers stated that the Rome cable installed as part of the Bus 5 modification was different than other 4 kV cable installed at Robinson and was used only for the Bus 4 to Bus 5 connection and the feeder from Bus 5 to station service transformer 2E. During the inspection, the licensee did not present any documentation explaining or justifying why the installed cable for the Bus 5 modification was different than Standard Specification L2-E-035 and the typical cables installed in the plant. The team reviewed the 4 kV cables connected to Buses 1 through 5, and found this statement to be correct. All the 4 kV cables connected to Buses 1 through 5, except for the two cables mentioned above, met or exceeded standard specification L2-E-035, with at least 133 percent insulation and insulation shield.

In addition to construction details of the faulted cable, the team reviewed various design considerations related to the cable. The ampacity of two 500 MCM, 90 degree Celsius, cables installed in conduit in free air is 954 amperes. The team estimated the maximum continuous load on Bus 5 as 493 amperes; 216 amperes for the 1750 HP Circulating Water Pump and 277 amperes for the 2000 kV station service transformer. The overcurrent relays were set at 1000 amperes. Therefore, the cables were not overloaded during normal operation. The conduits were the correct size for the cables installed within them. The number of bends in the conduits did not exceed the recommended maximum number of bends. Therefore, pulling tension limits should not have been exceeded during installation. This did not preclude the possibility that the three single conductors became twisted as the cable was pulled through three 90 degree bends. The licensees Event Review Team (ERT) visually examined the faulted cable and the station service transformer 2E feeder cable and determined the three single conductors were twisted. Twisting of one conductor around the other two conductors could result in jamming of the cables in the conduit since the combined diameter of the twisted cables would be greater than the inside diameter of the conduit. The twisting would have led to excessive pulling force being applied during cable installation. The required pulling force is proportional to the side wall pressure exerted on the cable at a bend.

Because of the extensive damage resulting from the length of time the fault was energized, the failure mechanism could not be determined with absolute certainty. The licensees causal analysis determined with a fair degree of certainty that the initial fault occurred at a point where the conduits terminate at the top of Bus 5 switchgear. After consideration of the above facts and review of the licensees causal analysis, the team concluded that the failure mechanism probably involved one or more of the following factors:

• Degradation of the insulation at the surface of the cable due to corona

• Damage to the insulation due to inadvertent twisting of the three conductors during the pulling-in process resulting in excessive side-wall pressure at one or more of the three 90 degree bends in the conduit

• Rubbing of the cable against the conduit or switchgear top plate due to turbine building vibration A secondary fault at the Bus 4 cable compartment for circuit breaker 52/24 was caused by plasma gas migrating inside the conduit and through a hole in the conduit seal, along with terminations that were not taped. The ERT postulated that the hole was caused by pressure built up in the conduit as a result of the fault. The ERT further postulated that this secondary fault at circuit breaker 52/24 created permanent degradation of the insulation at that location. All of the cable within the compartment was completed destroyed when Bus 4 was reenergized about four hours after the initial fault was cleared.

The licensee stated that corrective actions related to the cable failure would be to replace the Rome cable feeding station service transformer 2E before plant startup. The licensees Significant Adverse Condition Investigation Report for the event states that a search, using catalog identification numbers, was made across the Progress Energy fleet for this type of cable or similar cable and none was found. The licensee did not believe revisions were needed to the design control process because the process had been changed earlier to preclude the problems described herein, i.e. lack of proper control over purchasing and field changes.

The team concluded that the apparent root cause of the initial cable failure and subsequent associated short-circuits was poor quality control over the non-safety-related modification process for installing the cable. A cable of lesser quality than other 5 kV cables installed throughout the plant was installed as a substitute during this modification. The cable terminations were not taped and the cable was not restrained to prevent rubbing.

The consequence of the cable fault was a reactor trip. Because of the magnitude of the electrical fault, the reactor trip would have occurred regardless of whether bus tie circuit breaker 52/24 was fully functional. The resultant voltage transient decreased RCP speed which lowed RCS flow and initiated a reactor trip. This occurred faster than the time delay overcurrent protective relays associated with circuit breaker 52/24. Additional review by the NRC will be needed to determine whether the cable installation represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. The issue will be identified as URI 05000261/2010009-08, Deficiencies in Non Safety-Related Cable Installation.

4.2 Failure of Circuit Breaker 52/24 to Open

a. Inspection Scope

The team reviewed the work history and preventative maintenance records for circuit breaker 52/24 and viewed the damaged cable entrance compartment and surrounding area. The team also interviewed licensee personnel knowledgeable of facts associated with the failure of circuit breaker 52/24 to open as designed to clear an electrical fault.

b. Observations and Findings

Circuit breaker 52/24 is the non safety-related tie circuit breaker between 4 kV Bus 4 and Bus 5. Following an electrical fault on cabling between this breaker and Bus 5 as described in Section 4.1, the breaker failed to open to clear the fault due to a lack of control power.

The team reviewed equipment records related to circuit breaker 52/24 and determined that Work Request 357740 was written in November 2008 to repair the closed position indicating light located on the front of the circuit breaker. Because the closed position light would not illuminate after the light bulb was replaced, licensee personnel assumed the problem involved the socket for the bulb. Although the licensee had subsequently developed a work order to repair the socket, the licensee had not performed any additional repairs up to the time of the event. A number of opportunities existed to identify the source of the problem, including additional work requests and walkdowns by the system engineer. The additional work requests were canceled to the work order and the system engineer failed to recognize the potential impact of the failed indicating light regarding breaker operation.

Following the event, the licensee determined that one of the control power fuses in the breaker trip circuit was failed. Laboratory examination by the licensee revealed that the fuse had a cracked internal element. The licensees ERT found that the overcurrent relays and the circuit breaker were fully functional. The failed fuse caused the breaker trip circuit to be deenergized, resulting in the indicating lamp being off and preventing the circuit breaker from tripping.

Operations, Maintenance, and Engineering personnel did not fully understand the significance of the deenergized breaker indicating light. Operations personnel did not request an engineering assessment when they reviewed the work order. However, because station engineering was independently aware of the condition, it is not evident that a request for an engineering assessment would have resulted in a different outcome.

The broken fuse, style LPN-RK-30SP, was manufactured by Bussman Division of Cooper Industries. As part of their corrective actions for this problem, the licensee checked the resistance of 16 fuses of the same style to determine whether any incipient degradation was taking place. The tested group included in-service fuses of various sizes as well as three new fuses. The licensee determined all the fuses had acceptable resistance readings. The licensee stated they would also provide training to appropriate plant personnel regarding this event and expectations for response to circuit breaker indicating lamps being off when they should be on. (Note: On April 14, 2010, the NRC issued Information Notice 2010-09, Importance of Understanding Circuit Breaker Control Power Indications, which described the problem with circuit breaker 52/24 control power).

Section 4.1 states that, because of the high magnitude of the fault current, a reactor trip would have occurred as a result of the March 28 event, regardless of whether circuit breaker 52/24 was fully functional. However, for potential faults resulting in smaller currents, proper operation of circuit breaker 52/24 would prevent a reactor trip.

The team concluded the licensee failed to understand the possible implications of circuit breaker 52/24 indicating light being off and should have pursued the issue in a timely manner. The problem existed for approximately 17 months until this event revealed the circuit breaker was unable to isolate a fault condition. Additional review by the NRC will be needed to determine whether the failure to correct, in a timely manner, a problem with the indicating light for circuit breaker 52/24 and the underlying problem with the control power fuse represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. The issue will be identified as URI 05000261/2010009-09, Failure to Repair Circuit Breaker 52/24 Resulting in Breaker Being Unable to Operate.

4.3 Performance of Protective Relays

a. Inspection Scope

The team reviewed the performance of the following protective relays to determine if they functioned properly for the conditions experienced during the events:

• Inverse time overcurrent relays at circuit breakers 52-19, 52-20 and 52-24

• The sudden pressure relay at the unit auxiliary transformer

• Undervoltage relays for 4 kV Bus 4 and 480 V safety-related Buses E1 and E2

• Ground detection relay for the 4 kV high-resistance grounded system (UAT Y-winding)

The team reviewed sequence of event recorder data, relay set points, time-current characteristic curves, data from pre and post-event calibrations, fault current calculations, and the ratings for the distribution transformer and resistor used in the system grounding scheme. The team also reviewed the transfer of power for 480 V safety-related Bus E-2 from the start-up transformer to the B EDG. This item was reviewed to ensure the plant respond as expected to the disruptions in the electrical systems, with particular attention on relay actuations.

b. Observations and Findings

The protective relay system performed as designed and did not contribute to any of the problems associated with the two events. The team determined the electrical systems performed as designed when power to Bus E-2 was transferred to the B EDG.

4.4 Charging Pump Suction Valves Control Circuits

a. Inspection Scope

To determine the circumstances surrounding the failure of the charging pump suction control valves to transfer suction from the VCT to the RWST, the team performed the following activities:

• Reviewed the control circuit for valve LCV-115B and LCV-115C

• Reviewed portions of the work order that installed a comparator in the VCT level instrumentation loop

• Reviewed the sections of the manufacturers technical manual for the comparator to assess the required configuration for this application

• Reviewed the testing performed following replacement of the comparator

• Reviewed the quality assurance process regarding selection and installation of components

• Interviewed engineers and managers regarding the root cause of the problem and proposed corrective actions

b. Observations and Findings

Following the cable fault and resultant reactor trip, VCT level decreased and reached a low level set point that should have automatically transferred the suction source for the running charging pump to the RWST. The transfer did not take place as designed. The control circuitry which implements this transfer utilizes two VCT level transmitters. When each transmitter senses a low level, it energizes a relay via a comparator. When both relays are energized, and their contacts are closed, the circuit for opening the charging pump suction from RWST valve (LCV-115B) should be made up and the valve should open. Then, when LCV-115B opens, a signal is generated to close the VCT suction valve (LCV-115C.)

One of the relays in the LCV-115B circuit was driven by an older style Hagan level comparator, and the other relay was driven by a newer style NUS comparator. Different NUS comparator configuration options, such as electromechanical relay or solid state output, can be made by placing plug-type jumpers at different locations on the circuit board. The licensees post-event troubleshooting revealed that the NUS comparator was not properly configured when it was installed in 2008. The NUS comparator should have been configured to have its output function operate in the solid state mode and energize the control relay when a low level was sensed. When the comparator was configured in 2008, the placement of jumpers resulted in an electromechanical relay output, which was only capable of de-energizing the control relay upon low level. As a result, the control relay driven by the NUS comparator was in the energized state when level in the VCT was normal. When level in the VCT decreased below the level at which the suction to the charging pumps should have transferred, the associated valves did not reposition because the relay driven by the NUS comparator was de-energized and the valve open circuit was not made up.

The licensee did not detect the incorrect configuration of the NUS comparator after installation because of the limited scope of the post-installation testing. When the new comparator module was calibrated the bistable trip light responded as intended, satisfying the test acceptance criterion. The output contacts were not checked during the calibration and the licensee did not perform an integrated test, such as simulating a low VCT level, to confirm the two valves repositioned.

The licensee replaced the VCT level Hagan comparator with an NUS comparator as part of a larger project to provide a replacement for obsolete Hagan comparators. Licensee engineers stated that about 80 percent of the Hagan comparators had been replaced with NUS comparators at the time of the AIT inspection. The team questioned the extent of condition for potential similar errors in replacement comparators, i.e. incorrect placement of jumpers and inadequate testing for detecting errors. The licensee noted that comparators used to perform reactor protection system functions, safety injection

functions and certain other functions were subject to Technical Specification surveillance testing, which provided a check of the comparator output contacts. The licensee also pointed out that the circuit in question may have been unique in that only one of the comparators used in the two-out-of-two logic had been changed to the new NUS module. If two NUS modules had been installed, both containing the incorrect configuration for the jumpers, the transfer from VCT to RWST suction would have taken place with a normal VCT level and the problem would have been self revealing. The licensee stated that many control functions using the new NUS modules would alarm when the bistable actuates, making a similar problem self revealing.

The licensee controlled the substitution of NUS comparators for Hagan comparators under the plant modification process using Engineering Evaluation EE-92-144. The licensee controlled component removal and installation within the maintenance process.

The installation of the comparator for the charging pump suction transfer control circuit was accomplished under Work Order 011162348 in September 2008. Work order instructions directed an I&C technician to refer to the calibration procedure to determine the desired comparator configuration and refer to NUS instruction book EIP-M-DAM800 to determine the placement of jumpers necessary to implement that configuration. The placement and removal of jumpers was translated to work instructions which were reviewed and verified by an I&C system engineer.

The licensee stated their planned corrective actions would include a review of all control circuits incorporating NUS comparators to confirm these circuits will operate properly. In cases where a review indicates proper operation cannot be assured, the licensee stated that appropriate testing will be performed. In addition, the process for implementing any future NUS comparator installations will be strengthened to preclude the problems described above.

The team determined the failure of the suction for the charging pumps to automatically transfer from the VCT to the RWST upon low level in the VCT was caused by an error in the work instructions describing the placement of jumpers when a VCT level comparator was replaced. Additionally, the licensees post-maintenance testing was not adequate to detect the problem. Additional review by the NRC will be needed to determine whether these problems represent a performance deficiency. An Unresolved Item will be opened pending completion of this review. The issue is identified as URI 05000261/2010009-10, Failure of Charging Pump Suction Valves to Automatically Transfer Due to Errors in Implementing an Instrumentation Component Upgrade.

4.5 Unexpected Closure of FCV-626

a. Inspection Scope

In order to determine the reason valve FCV-626 closed during the first electrical fault and fire, the team reviewed plant drawings, plant procedures, work orders, corrective action reports, simulator operation and interviewed personnel associated with the operation of FCV-626.

b. Observations and Findings

Valve FCV-626 is located in the combined CCW return from the three RCP thermal barrier heat exchangers. In its normal open position it allows CCW flow to pass through the thermal barrier heat exchangers, providing backup cooling to the RCP seals in the event of a loss of the primary cooling flow (seal injection) from the charging pumps.

There are two close functions for FCV-626: 1) closes on high flow in the return line which is indicative of a rupture of a thermal barrier heat exchanger (RCS to CCW system leak) and 2) closes in response to a Phase B containment isolation signal. The valve has no automatic opening functions.

The valve closed when power to safety-related 480 volt Bus E-2 was transferred to the emergency diesel generator. The valve remained closed for approximately 39 minutes before the operators recognized the condition, reopened FCV-626, and restored CCW cooling to the RCP thermal barrier heat exchangers. Plant staff knew that FCV-626, a motor operated valve, was powered from Bus E-2 via MCC 6. However, plant staff, including operators, was unaware that FCV-626 would close on a momentary loss of power. Additionally, the simulator was modeled such that FCV-626 remained open when power to Bus E-2 was momentarily interrupted.

The high flow closure function of FCV-626 is accomplished using flow orifice FE-626, which is located in the thermal barrier return line, and provides flow switch FIC-626 with a hydraulic input to operate high and low flow contacts. When the high flow contact opens, relay FIC-626X is de-energized and closes a contact in the closing circuit for the motor operator of FCV-626, thereby closing FCV-626.

Plant procedure EDP-008, Instrument Buses, incorrectly indicated that the power source for the flow switch FIC-626 control circuit was from Instrument Bus 1. The power for the FIC-626 control circuit is from Instrument Bus 4. Instrument Bus 4 is fed from MCC-6, which also feeds motor operated valve FCV-626. When Bus E-2 transferred to the EDG, both valve FCV-626 and relay FIC-626X lost power for approximately 10 seconds.

During this time interval, relay FIC-626X repositioned to its de-energized state, which closed a contact in the close circuit of valve FCV-626. When Bus E-2 reenergized, valve FCV-626 immediately began to close, which sealed in contacts to completely close the valve. The close circuit was sealed in before relay FIC-626X reset to its energized state.

The team concluded that the most likely cause of the time delay for the relay to reset was a constant voltage transformer located between the relay and MCC-6.

The safety significance of inadvertent shutoff of RCP thermal barrier cooling water is discussed in Section 1.1 of this report.

The team reviewed historical data associated with the control circuit for valve FCV-626 and determined the licensee had at least two potential opportunities to discover the behavior of FCV-626 on a loss of power. The first opportunity was in 2005 while implementing Engineering Change (EC) 59456. While performing the EC, workers encountered wiring issues as documented in NCR168221. The licensee subsequently determined that flow switch FIC-626, which was previously thought to be powered from Instrument Bus 1, required no power to operate. As part of that investigation, it was noted that relay FIC-626X was powered from Instrument Bus 4. Wiring discrepancies existed in some of the associated drawings, but were either not noted or not pursued.

Additionally, the licensee did not update EDP-008, which continued to show Instrument Bus 1 Breaker 16 as the power source for the flow switch FIC-626 control circuit. The licensee has written NCR 391995 to correct these deficiencies.

The second opportunity occurred in 2008 during the performance of OST-163, Safety Injection Test and Emergency Diesel Generator Auto Start on Loss of Power and Safety Injection (refueling). As part of the test, 480V safety-related Bus E-2 was transferred to the B EDG. Data recovered by the licensee indicated that FCV-626 closed at approximately the same time the B EDG re-energized Bus E-2. However, during the test, CCW system flow, which was being provided by the opposite train of power, was stable and RCPs were not running. Thus sufficient information was available to recognize that flow perturbations were not the cause for FCV-626 closing. About two hours later, the valve was reopened. The licensee either did not identify or did not pursue the unexpected behavior of FCV-626.

The ERT entered the problems described in this section into the corrective action system. One corrective action will clarify the drawings associated with the control circuitry of FCV-626 and make some minor corrections to current drawings. Another corrective action will implement a modification to prevent inadvertent closure of valve FCV-626 for a momentary loss of power. The licensee stated the modification would be implemented prior to restart of the plant from the refueling outage.

Additional review by the NRC will be needed to determine whether the design of FCV-626, which caused the valve to close during a momentary loss of power, represents a performance deficiency. An Unresolved Item will be opened pending completion of this review. This issue is identified as URI 05000261/2010009-11, FCV 626, RCP Thermal Barrier Outlet Isolation CCW Valve, Unexpected Closure.

In order to better understand the reason FCV-626 closed during a momentary loss of power, the team reviewed the licensing bases for the CCW system, including FCV-626.

The team reviewed correspondence between the licensee and the NRC regarding NUREG 0737, Clarification of TMI Action Plan Requirements, Item II.K.3.25, Power on Pump Seals. This TMI item required the licensee to determine the consequences of a loss of RCP cooling due to a loss of offsite power lasting two hours. In their correspondence, the licensee stated that no modifications were necessary because the CCW system is still operable during a loss of offsite power (powered from the emergency buses) and provides flow to the RCP thermal barrier heat exchangers. They also stated that the B and C CCW pumps are automatically (requiring no operator action) started by a station blackout signal during a loss of offsite power event.

Additional review by the NRC will be needed to determine if the behavior of RCP seal cooling following a loss of offsite power event is consistent with the description provided by the licensee in NUREG 0737 correspondence and if any differences represent a violation. An Unresolved Item will be opened pending completion of this review. The issue is identified as URI 05000261/2010009-12, NUREG 0737 Response From Licensee to the NRC Describing the Behavior of RCP Seal Cooling Following a Loss of Offsite Power Event.

4.6 Grounds on Both A and B Train Direct Current (DC) Buses

a. Inspection Scope

The team reviewed the circumstances which resulted in simultaneous electrical grounds on both the A and B DC buses. Additionally, the team reviewed the licensees response to the grounds, including ground isolation efforts. The team reviewed completed procedures, log entries, system drawings, performed a system walkdown and interviewed personnel.

b. Observations and Findings

When the operator attempted to reset the generator lockout relay, Bus 4 was re-energized and a fault occurred at circuit breaker 52/24. The fault caused extensive internal and external damage to the rear of the cubicle for circuit breaker 52/24. The control room subsequently received several alarms including both battery charger A/A-1 and B/B-1 trouble annunciators. An auxiliary operator verified both the A and B batteries had ground indications. In support of fire fighting efforts, operators isolated control power to 4 kV Buses 4 and 5 at 23:10, clearing the ground on the B DC bus.

Operators successfully cleared the A DC bus ground by removing DC power from the main generator hydrogen supervisory panel at 00:00. This panel is located approximately three feet behind the rear of the circuit breaker 52/24 cubicle and was visibly damaged when the fault at Bus 4 was reenergized. The external damage included several status light lenses that were melted.

The team determined the fire induced grounds did not affect the operation of any safety related components. This conclusion was supported by a review of post event battery voltage trends and a lack of evidence of components that did not function as a result of DC power degradation. The operators isolated the source of the DC grounds in a timely manner using appropriate procedural guidance (OMM-003, Fire Protection Pre-Plans/Unit No. 2, and OMM-035, Ground Isolation.) The licensee entered Condition Report 390082 documenting the DC grounds in their corrective action program.

4.7 Performance of Radiation Monitors R11/12

a. Inspection Scope

The team reviewed the circumstances which resulted in a report of smoke coming from the containment atmosphere radiation monitors sample pump. The team reviewed completed procedures, drawings, log entries, and performed a system walkdown.

b. Observations and Findings

The Safety Injection and containment isolation Phase A that occurred shortly following the first event caused the R-11/R-12 containment air and plant vent radiation monitor sample supply and return line isolation valves to close, as designed. At 20:12, control room operators secured the R-11/R-12 radiation monitor sample pump due to reports from field personnel that the pump was smoking. Subsequent investigation by the licensee determined the low sample flow switch, which should have stopped the sample pump when the sample lines were isolated, failed to operate. The smoke was caused by

slipping of the drive belt between the sample pump and drive motor. The belt slipped because the pump stalled due to high sample line backpressure when the supply and return line isolation valves closed. The team reviewed maintenance records to determine whether there had been any outstanding work orders written against the flow switch and found none.

The licensee confirmed R-11/12 would function properly with a normal sample flow path, returned the radiation monitors to service, entered CR 390076 into their corrective action program, and initiated a work request to repair the low sample flow pressure switch during the refueling outage.

4.8 Dedicated Shutdown Diesel Generator (DSDG)

a. Inspection Scope

The team reviewed the circumstances which resulted in a failure of the DSDG to start.

The team reviewed completed procedures, log entries, system drawings and performed a system walkdown.

b. Observations and Findings

At 18:52 on March 28, the DS bus was automatically de-energized, as designed, due to undervoltage on 4 kV Bus 3. As a result, the DSDG support equipment, such as the starting air system compressor and battery charger, lost power. Based in part on adequate starting air pressure, the licensee considered the DSDG available for the purpose of assessing on-line risk. The log reading normal minimum value for starting air pressure is 165 psig and operators were monitoring this parameter twice per day.

At 14:41 on March 31 the licensee attempted to start the DSDG and re-energize the DS bus to maintain adequate DSDG support parameters such as starting air pressure and battery voltage. Starting air pressure had decreased to 100 psig and the DSDG did not start.

The licensee successfully started the DSDG on April 1 at 13:40 by pressurizing the DSDG starting air receiver tank using high pressure air bottles. Both the E-1 and E-2 safety buses were energized during this time with Bus E-1 powered from off-site power and Bus E-2 supplied from EDG B. The licensee entered Condition Reports 390954 and 390958 into their corrective action program.

Additional review by the NRC will be required to determine if the DSDG was available when credited in the licensees risk assessment during the plant cooldown to Mode 4.

This review will also determine whether this issue represents a performance deficiency.

An Unresolved Item will be opened pending completion of this review. The issue will be identified as URI 05000261/2010009-13, Dedicated Shutdown Diesel Generator Failed to Start Due to Low Starting Air Pressure.

4.9 Power Interrupted to Instrument Bus 3

a. Inspection Scope

The team reviewed the circumstances which resulted in an inadvertent de-energization of Instrument Bus 3. The team reviewed completed procedures, log entries, and system drawings. The team also interviewed personnel and performed a system walkdown.

b. Observations and Findings

At 18:52 on March 28 the B battery charger de-energized due to loss of power to Bus E-2. Per Path-1, control room operators subsequently dispatched an Auxiliary Operator (AO) to restore the B battery charger. As the AO entered the battery room he made inadvertent contact with the handle for the B Inverter Supply Breaker 72/MCC-B (1K).

The contact resulted in breaking the handle off of the breaker. Based on the timeframe when the AO entered the battery room and the time when Instrument Bus 3 was unexpectedly loss, the licensees ERT concluded the contact with the breaker caused the loss of Instrument Bus 3. The Auxiliary Operator recognized the damage to the breaker handle and continued to complete the restoration the B battery charger. The B battery charger was restored at 19:31. Upon exiting the battery room the AO verified the B inverter was operating correctly and reported the damage to the breaker handle.

A review of plant data indicated Instrument Bus 3 was de-energized at 19:25 and re-energized at 19:27. The loss of Instrument Bus 3 power deenergized the High Steam Flow bistables in the Engineered Safety Features system. This condition, coincident with an RCS Low Tavg signal due to the RCS cooldown, generated a Main Steam Line Isolation signal, automatically closing all MSIVs and terminating the RCS cooldown.

Based on interviews with the AO, no actions were performed to reset or reclose the B Inverter Supply Breaker. The licensee generated Work Order 01735191 to repair the broken breaker handle. The licensee performed troubleshooting activities to determine the cause of the two-minute interruption in instrument bus power, but was unable to detect any problems. The licensee was continuing to perform troubleshooting at the time this report was written. The licensee entered Condition Report 390070 into their corrective action program.

Additional review by the NRC will be needed to assess the adequacy of the licensee troubleshooting efforts and evaluate any problems that may be identified. This review will also determine whether any performance deficiencies exist. An Unresolved Item will be opened pending completion of this review. The issue will be identified as URI 05000261/2010009-14, Unexpected Loss of Instrument Bus 3 for Two Minutes.

4.10 Security Equipment Response

a. Inspection Scope

The team reviewed the response of security related equipment to the events. The team reviewed completed procedures, drawings, security log entries, and performed a system walkdown.

b. Observations and Findings

The security equipment responded as expected during the events.

4.11 Licensee Investigation and Corrective Actions (Equipment)

a. Inspection Scope

The team reviewed portions of the licensees Significant Adverse Condition Investigation Report for the event related to equipment performance in order to 1) independently assess the licensees investigation of the event, 2) verify the licensee had appropriately reviewed equipment performance during the event to identify problem areas requiring further review, 3) ensure the root cause(s) of equipment problems were identified, and 4)assess the adequacy of the licensees corrective actions.

b. Observations and Finding The team determined that the licensees event investigation and Significant Adverse Condition Investigation Report used valid investigative tools such as support/refute, cause/effect, barrier analysis, etc. to fully understand the scope of problems. The team concluded that the licensees investigation report adequately addressed the cable fault, the failure of circuit breaker 52/24 to trip, and the failure of the charging pump suction valves to swap from the VCT to the RWST. The team verified the licensees explanation for these items was reasonable. Causes specific to the equipment problems were identified and as appropriate, corrective actions were initiated for identified discrepancies. Corrective actions for the remaining equipments issues discussed in this report are also addressed by the licensees corrective action program. The team verified that corrective actions, of an immediate nature, were appropriate flagged in the corrective action program as required to be implemented prior to plant restart.

5.0. Fire Protection 5.1 Fire Brigade and Control Room Response

a. Inspection Scope

The team reviewed operator and fire brigade performance to determine the effectiveness of personnel in mitigating the effects of the fire. The team interviewed the shift manager, licensed operators and fire brigade members, and reviewed procedures and control room logs. The team specifically reviewed the following aspects of the fire response: 1)entry into the fire area, 2) use of fire-fighting equipment, 3) brigade leader command and control, 4) communications between the fire brigade and control room, 5) searches for fire victims and fire propagation, 6) smoke removal, and 7) use of pre-fire plans.

b. Observations and Findings

As noted in Section 1.0, the first electrical fault started at 18:52 on March 28 and continued for 12 seconds before being isolated. Resulting fires were observed in the

4 kV electrical Bus 5 area and cubicle for circuit breaker 52/24. At the same time, the control room received fire alarms B76 (4 kV Switchgear Room Fire Alarm) and B77 (Fire Detection Panel FDAP-A2 Master Fire Alarm).

At 18:54 the SM and STA returned to the control room with a report of fire at 4 kV Bus 5 that they had received from security officers. At 18:56 the control room entered AOP-41, Response to Fire Event, and requested off-site fire department assistance. For the next 30 minutes the BOP operator was dedicated to the fire response activity. At 18:57 the fire was initially extinguished at 4 kV Bus 5 through use of two dry chemical extinguishers, but the fire re-flashed several times afterwards. At 19:02 two security officers entered the 4 kV switchgear room to check for security patrols. The two officers saw flames from the vents at the back of breaker cubicle 52/24. The security officers requested that a security station call the control room to inform them of the condition because the officers were unable to contact the control room. The two security officers obtained two fire extinguishers and discharged them into the upper and lower vents at the back of circuit breaker 52/24. The fire was extinguished within 3-5 minutes and security officers remained in the room to assess the condition. At 19:18 the security officers exited the 4 kV switchgear room, went to the ground floor of the turbine building, and informed the Fire Protection Auxiliary Operator of their actions.

The team observed that plant personnel concurrently fought two fires in close physical proximity (4 kV electrical Bus 4 with circuit breaker 52/24 is located in the turbine building one floor above 4 kV electrical Bus 5.) However, the team noted that the fire at circuit breaker 52/24 was fought by security officers without the initial awareness or knowledge of either the fire brigade or the control room. Although the team noted that the control room staff and fire brigade personnel were notified of the fire at circuit breaker 52/24 after the fact, the lack of communication/coordination between the security officers and the fire brigade and control room during the fire fighting put the officers at personal risk and could have complicated the response to the event had they been unsuccessful in extinguishing the fire. While the outcome was positive, the licensees ERT also identified the potential danger involved with non-fire brigade individuals fighting fire at high voltage electrical equipment without the knowledge or consent of the fire brigade or the control room.

At 19:30 the offsite fire department arrived on scene. The fire brigade members and offsite firefighters were directed to perform inspection of 4 kV switchgear room and the rooms for 480V Buses E1 and E2. At 19:38 the Fire Protection Auxiliary Operator notified the Fire Brigade Incident Commander via radio that security officers had put out a fire at circuit breaker 52/24. At 20:34 the fire brigade reported ALL CLEAR for the 4 kV switchgear room and the 4 kV Bus 5 area.

At 22:34 operators attempted to reset the generator lockout relays which restored power to 4 kV Bus 4 and initiated another electrical fault. At 22:40 a fire was reported in circuit breaker 52/24, procedure AOP-41 was re-entered, and the fire brigade was re-activated.

Significant damage occurred at the back of circuit breaker 52/24 and to surrounding equipment due to the arc flash and fire. At 22:55 fire brigade members entered the 4 kV switchgear room to fight the fire at circuit breaker 52/24. At 23:01 the fire brigade extinguished the fire at circuit breaker 52/24, began ventilation of the area and then set a re-flash watch. On March 29 at 01:00 thermal imaging results indicated temperatures of

4 kV Bus 4 were low enough to secure the reflash watch and at 01:30 the fire brigade declared ALL CLEAR.

The team concluded that operator performance during the event was effective in identifying the source of the fire, initiating fire fighting activities, and monitoring the operability of plant equipment. The team concluded that the licensees fire brigade adequately controlled and extinguished the fire. Based upon interviews, the team determined that communications between the control room, fire brigade leader, and fire brigade members were acceptable throughout the fire-fighting efforts.

5.2 Fire Protection System Response

a. Inspection Scope

The team reviewed the performance of the fire protection detection and suppression equipment to determine its effectiveness in responding to the two fires. The team 1)observed the physical arrangement of the affected equipment, 2) verified that fire extinguishers and hose stations were provided at their designated locations and that they were available to combat the fire, and 3) verified that passive fire protection features (electrical raceway barriers, fire doors, fire dampers, steel fire proofing, and penetration seals) performed as designed.

b. Observations and Findings

When the fire ALL CLEAR notification was made at 20:34 the re-flash watch was secured. However, control room operators did not reset the fire detection equipment to allow detection of any subsequent fires in the area. The team determined that the failure of the control room operators to reset the fire detection equipment following the first fire had minimal impact on the response to the second fire event. This conclusion was based on fact that within six minutes of the electrical fault, the fire at circuit breaker 52/24 was reported, AOP-41 was re-entered, and the fire brigade was activated. The fire was subsequently extinguished within 5 minutes of the fire brigade entering the room.

The team verified that the fire detection equipment (thermal and ionization detectors) in the 4 kV switchgear room functioned properly during the event. There was no fire detection capability in the 4 kV Bus 5 area. The team noted that neither the 4 kV switchgear room nor the 4 kV Bus 5 area have automatic suppression systems (e.g.,

Halon or CO2). Fire suppression capabilities for these areas are provided by fire hose stations and hand held fire extinguishers.

The team concluded that fire detection equipment performed in accordance with plant design. The team concluded that fire brigade and control room personnel use of fire protection equipment was adequate to control and extinguish the fire.

5.3 Other Fire Protection Items

a. Inspection Scope

The following items not specifically covered in section 5.1 and 5.2 were reviewed:

• Historical fires during the past 10 years that had occurred within the protected area

• Types of fire that occurred during the event and the extent of damage from the fire, specifically the electrical cables in the overhead above 4 kV Bus 5

• Corrective actions for the fires and any follow-up items

• Relevant Operational Experience items

b. Observations and Findings

The licensees assessment of the fire protection response to the event was adequate and related problems were entered into the corrective action program. The team identified sections of fire-damaged electrical cable located in the overhead of 4 kV Bus 5 to be saved for possible further evaluation. The team reviewed the licensees previous response to NRC Information Notice 2002-27, Recent Fires at Commercial Nuclear Power Plants in the United States, to assess the licensees approach to extinguishing electrical fires and determined the response was adequate.

6.0 Emergency Preparedness

a. Inspection Scope

The team reviewed the licensees implementation of the emergency preparedness (EP)procedures used during the event. The review focused on the circumstances surrounding the events to determine if the licensees EP classification and notifications were appropriate and timely. The team interviewed members of the licensees EP organization and other individuals involved with EP aspects of the event. The team reviewed the event timeline, logs, statements by individuals who responded to the event, the Robinson Emergency Action Level (EAL) matrix, event notification worksheets, and other documents related to EP classifications.

b. Observations and Findings

In order to determine the appropriateness of the EP classifications following the first and second fires, the team performed a detailed assessment of the event timeline with particular attention to those activities that are entry points for the EAL matrix. The first fire at 4 kV Bus 5 was discovered at 18:53 by two security officers. The security officers informed the SM and STA who were returning to the control room. At 18:56 control room operators sounded the fire alarm, activated the fire brigade, and requested offsite assistance. The fire brigade responded to 4 kV Bus 5 and reported smoke, but no visible fire. At 19:04 the fire was reported out by the Fire Brigade Incident Commander. Based on this sequence of events the team determined that the first fire event did not meet the criteria in the EAL matrix for an EP classification.

The second fire event occurred when 4 kV Bus 4 was reenergized as a result of the attempt to reset the generator lockout relays. Personnel in the plant heard and saw

indications of the electrical fault at circuit breaker 52/24 and reported this information to the control room. When the fire and electrical fault occurred, several alarms were received which revealed both A and B batteries had ground indications. Based on the information the team reviewed, the control room was notified of fire and smoke in the 4 kV switchgear room at 22:39. Upon entering the 4 kV switchgear room at 22:55, the fire brigade identified flames and damage at circuit breaker 52/24. The SM made an Alert EP classification at 23:00. The SM stated that he made the classification based on criteria HA2.1 of the EAL matrix for a fire in a Table H-1 area affecting the A and B DC buses. The fire brigade reported the fire was extinguished at 23:01.

The team determined the licensee did not assess the second fire event and the corresponding effects on plant equipment for possible entry conditions into the EAL matrix in a timely manner. Based on the sequence of events described above, the team determined that the second fire event met the HA2.1 criteria in the EAL matrix for an Alert classification at 22:39. While the untimely Alert classification is not a violation of regulatory requirements, the licensees procedural expectations and industry guidance for making EP classifications within 15 minutes from the time EAL entry condition information /indications becomes available to the control room staff were not satisfied.

The licensee entered the untimely Alert classification into their corrective action program.

The untimely Alert classification will be an input to the licensees Emergency Preparedness Cornerstone Drill and Exercise Performance Indicator as a missed opportunity.

Notifications of the Alert to the State and Counties were made twelve minutes after the event classification. Notification to the Nuclear Regulatory Commission Operations Center was made 40 minutes after the event classification. The team determined that notifications to the State and Counties and to the Nuclear Regulatory Commission Operations Center were timely and accurate.

Following the Alert classification at 23:00, notification of the licensees Emergency Response Organization was delayed 29 minutes due to equipment failures. However, the Emergency Response Facilities were properly manned and declared operational within the required 60 minutes from event declaration. The team determined that the manning of Emergency Response Facilities was timely to support control room personnel in troubleshooting activities, restoration of electrical distribution lineups, and stabilization of plant parameters.

The Alert event was terminated at 01:46 on March 29, 2010. The decision to terminate the event was based on the following: 1) no public issues existed that would necessitate the continued activation of the State and County Emergency Operations Facilities and 2)the licensees Outage Control Center had established a technical focus and was aligned for the recovery activities. The team determined that termination of the Alert event at 01:46 was appropriate.

7.0

Exit Meeting Summary

On April 26, 2010, preliminary results from the teams initial onsite portion of the inspection were presented to Mr. Robert Duncan, Vice President Nuclear Operations, Mr. Eric McCartney, and other members of his staff. On June 2, 2010, the Region II Deputy Regional Administrator and the Augmented Inspection Team Leader presented

the results of the inspection in a public meeting at Coker College to Mr. Eric McCartney, and other members of his staff. Although proprietary information may have been reviewed during the inspection, no propriety information was included in this inspection report.

s:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

G. Attarian, Manager, Major Projects

C. Castell, Supervisor, Licensing/Regulatory Programs

C. Dick, Operations

J. Edwards, Training

P. Gaffney, Design Engineering Manager

C. Georgeson, Principal Engineer, NED

D. Grant, Operations

T. Hooper, Lead Engineer, NED

T. Koschmeder, Senior Engineer

B. McCabe, Manager, Nuclear Regulatory Affairs

T. McNamara, Lead Engineer, Transmission Operations

K. Moore, Lead Engineer

T. Natale, Director, Nuclear Work Management

D. Nunnally, Emergency Preparedness Supervisor

K. Riley, Supervisor, Major Projects

R. Rishell, Manager, Probabilistic Risk Assessment

E. Roberts, Superintendent of Operator Training

J. Seymour, Operations

D. Simms, Operations

L. Smith, Operations

J. Stephenson, Fleet Emergency Preparedness Manager

B. Stuckey, Senior Technical Support Specialist

S. West, Superintendent Security

R. Williamson, Engineer

M. Woodrum, Major Projects

LIST OF ITEMS

OPENED

Opened

05000261/2010009-01 URI Monitoring of Plant Parameters and Alarms. (Section 3.1)

05000261/2010009-02 URI RCS Cooldown Rate Exceeds Technical Specification 3.4.3 limit. (Section 3.1)

05000261/2010009-03 URI Utilization of Operators During Events Requiring Use of Concurrent Procedures. (Section 3.1)

05000261/2010009-04 URI Fidelity of Plant-Referenced Simulator. (Section 3.2)

05000261/2010009-05 URI Corrective Action for Operating Crew Performance Issues. (Section 3.2)

05000261/2010009-06 URI Adequacy of Emergency Operating Procedure Background Documents (Section 3.3)

05000261/2010009-07 URI Loss of Seal Water Results in Failure of the A Main Condenser Vacuum Pump (Section 3.3)

05000261/2010009-08 URI Deficiencies in Non Safety-Related Cable Installation (Section 4.1)

05000261/2010009-09 URI Failure to Repair Circuit Breaker 52/24 Resulting in Breaker Being Unable to Operate (Section 4.2)

05000261/2010009-10 URI Failure of Charging Pump Suction Valves to Automatically Transfer Due to Errors in Implementing an Instrumentation Component Upgrade (Section 4.4)

05000261/2010009-11 URI FCV 626, RCP Thermal Barrier Outlet Isolation CCW Valve, Unexpected Closure (Section 4.5)

05000261/2010009-12 URI NUREG 0737 Response From Licensee to the NRC Describing the Behavior of RCP Seal Cooling Following a Loss of Offsite Power Event (Section 4.5)

05000261/2010009-13 URI Dedicated Shutdown Diesel Generator Failed to Start Due to Low Starting Air Pressure (Section 4.8)

05000261/2010009-14 URI Unexpected Loss of Instrument Bus 3 for Two Minutes (Section 4.9)