ML11221A297

521- R326C B&W Tech Cross Training - Chapter 15 Rancho Seco Loss of ICS Power

ML11221A297
Person / Time
Site:	Rancho Seco
Issue date:	05/01/2008
From:	NRC/HR/ADHRTD/RTTB-PWR
To:
References
Download: ML11221A297 (36)

Text

Pressurized Water Reactor B&W Technology Crosstraining Course Manual Chapter 15.0 Rancho Seco Loss of ICS Power

TABLE OF CONTENTS 15.0 RANCHO SECO LOSS OF ICS POWER ..................................................................... 1 15.1 Introduction ............................................................................................................ 1 15.1.1 ICS Power Supplies ........................................................................................ 1 15.1.2 Main Steam System ........................................................................................ 2 15.1.3 Main Feedwater System ................................................................................. 2 15.1.4 Auxiliary Feedwater System ............................................................................ 2 15.1.5 High-Pressure Injection System ...................................................................... 3 15.2 Loss of ICS DC Power ........................................................................................... 3 15.2.1 Initial Conditions .............................................................................................. 3 15.2.2 Initiating Even.................................................................................................. 3 15.2.3 RCS Overcooling ............................................................................................ 3 15.2.4 Safety Features Actuation System (SFAS) Actuation ..................................... 4 15.2.5 Partial RCS Repressurization ......................................................................... 5 15.2.6 Restoration of ICS dc Power and Plant Stabilization....................................... 7 15.3 Major Issues ......................................................................................................... 10 15.3.1 ICS Power Failure ......................................................................................... 10 15.3.2 Makeup Pump Failure ................................................................................... 11 15.3.3 RCS Overcooling .......................................................................................... 12 15.3.4 PRA Insights .................................................................................................... 13 15.4 References ........................................................................................................... 13 APPENDIX Sequence of Events ........................................................................................................... 14 LIST OF FIGURES Figure 15-1 ICS Power Supplies Figure 15-2 Main Steam System Figure 15-3 Main Feedwater System Figure 15-4 Auxiliary Feedwater System Figure 15-5 High Pressure Injection System Figure 15-6 Dominant Core Vulnerability Sequence Event Tree Figure 15-7 Dominant core Damage Sequence Event Tree USNRC HRTD 15-i Rev 05/2008

This page intentionally blank.

USNRC HRTD 15-ii Rev 05/2008

15.0 RANCHO SECO LOSS OF ICS POWER Learning Objectives:

1. Describe the symptoms of an overcooling transient.

2. Explain how the loss of power to the ICS control stations resulted in:

a. An overheating event and

b. A subsequent overcooling event

3. Describe the effect of operating a multi-stage centrifugal pump without a suction source.

15.1 Introduction On December 26, 1985, Rancho Seco experienced a loss of ICS power causing a reactor trip and rapid cooldown of the reactor coolant system. This incident was not the first time Rancho Seco had experienced problems with its ICS power supplies. A reactor trip and excessive plant cooldown occurred on January 5, 1979, resulting from a personnel error that caused a ground fault in the ICS DC power supplies. The plant response to the 1979 event was almost identical to the December 26, 1985 event. On March 20, 1978, a reactor trip and excessive plant cooldown occurred resulting from a light bulb from a backlighted push button being dropped into the socket grounding the NNI DC power. A complete loss of NNI DC power resulted. This is referred to as the light bulb incident and is considered the most severe overcooling transient to ever occur at Rancho Seco and is referenced in several vendor, licensee, and regulatory documents. Although ICS power was not lost during the light bulb incident, the design of the power supplies is virtually identical. A brief description of Rancho Seco system differences follows.

15.1.1 ICS Power Supplies As shown in Figure 15-1, there are redundant 120-vac supplies to the ICS. One of the supply sources is from a 120-vac vital bus, and the second supply is a non-vital 120-vac bus. Each of the 120-vac sources supplies positive and negative 24-vdc power supplies through switches S1 and S2. The +24-vdc power supplies and the vdc power supplies are auctioneered, and the highest voltage provides power to the ICS buses. A power supply monitor is installed and monitors the output of each power supply as well as the auctioneered output. Because of the inability to predict the response of the ICS under degraded voltage situations, the power supply monitor will open switches S1 and S2 if any monitored voltage drops to 22 Vdc.

Internally, the ICS will convert the +/- 24 Vdc to a -10 to +10 Vdc that is used by the system. When this signal is sent to the control valves, the v demand represents a fully USNRC HRTD 15-1 Rev 05/2008

closed signal; the +10-v signal represents a fully open signal; the zero signal will position the valve to 50% open. The valves of interest in this transient are:

1. Atmospheric dump valves (ADVs)

2. Turbine bypass valves (TBVs)

3. Main feedwater valves

4. Auxiliary feedwater valves The main feedwater pump turbine speed is also controlled by the ICS. In this control circuit, a 0- to +10-v signal is used. The signal range of 3.4 - 7.3 Vdc corresponds to the minimum to maximum range of feed pump speeds.

15.1.2 Main Steam System The main steam system (Figure 15-2) supplies steam from the once-through steam generators (OTSGs) to the turbine-generator, the turbine driven main feedwater pumps, the dual drive auxiliary feedwater pump, and other plant auxiliary systems. Three atmospheric dump valves (ADV) and two turbine bypass valves (TBV) are installed on each header to remove excess reactor coolant system energy during a turbine trip or load rejection. The valves have a total capacity of 50 percent, with equal amounts of heat removal capacity available by dumping steam to the atmosphere or bypassing steam to the condenser.

These valves are controlled by the ICS. Rancho Seco normally operates with four of the six ADVs manually isolated to prevent overcooling following a reactor trip.

15.1.3 Main Feedwater System The discharge of the turbine driven main feedwater pumps (Figure 15-3) is routed to the OTSGs via high-pressure feedwater heaters, startup feedwater regulating valves, and the main feedwater regulating valves and their associated block valves. As previously stated, the main feedwater pump speed, the startup regulating valves, and the main feedwater regulating valves are controlled by the ICS.

15.1.4 Auxiliary Feedwater System The auxiliary feedwater (AFW) system is shown in Figure 15-4 and consists of redundant pumps and flowpaths to each steam generator. One of the two pumps is motor driven, and the other pump is dual driven. The dual driven pump is powered by a turbine on one end of the pump shaft and a motor on the other end. The pumps normally receive suction from the condensate storage tank (CST). Auxiliary feedwater flow to the steam generators is controlled by parallel valves in each supply line. One of the two valves receives a signal from the ICS, and the second valve receives a signal from the safety features actuation system (SFAS).

USNRC HRTD 15-2 Rev 05/2008

15.1.5 High-Pressure Injection System The high-pressure injection system (Figure 15-5) is typical of the 177 FA high-pressure injection (HPI) systems. As shown, the pumps normally receive suction from the makeup tank and discharge to an RCS cold leg via the makeup flow control valve. If an emergency core cooling actuation signal is received, the following changes will occur in the system:

1. The makeup tank outlet valve closes.

2. The borated water storage tank suction valves open.

3. The four HPI motor operated valves open.

4. The HPI recirculation isolation valves close.

15.2 Loss of ICS DC Power 15.2.1 Initial Conditions The plant had been returned to power following a two-day outage for repairs. Power had been escalated to 76% (712 MWE) and stabilized. RCS Tavg was at its normal value of 582°F, and system pressure was 2150 psig. The pressurizer level was at its normal value of 220 inches.

15.2.2 Initiating Even The loss of ICS dc power was caused by a failure of the power supply monitor which opened switches S1 and S2 (Figure 15-1). The loss of input power resulted in a zero voltage on the +/- 24-vdc busses in the ICS. At zero volts, the main feedwater regulating valves and the startup regulating valves assumed a 50% open position. Since voltage was lost, the main feedwater pumps dropped to minimum speed. With the reactor at 76%

power and almost no feedwater flow to the once-through steam generators for RCS heat removal, an overheating event started even though the turbine bypass and atmospheric dump valves were 50% open due to the loss of ICS power. With reduced heat removal, the reactors energy causes an increase in RCS temperatures. The increase in temperatures results in an increase in pressurizer pressure and pressure level. The pressurizer spray valve was manually opened in an effort to lower RCS pressure.

15.2.3 RCS Overcooling Due to the net undercooling, the reactor tripped on high RCS pressure sixteen seconds after the loss of ICS power. The RCS pressure peaked about one second later at a value of 2298 psig. Several of the steam generator safety valves lifted and then reseated. The reactor trip signal also tripped the turbine generator. The operators closed the pressurizer spray valve when the reactor tripped, in anticipation of RCS cooldown and depressurization. With the reactor at decay heat levels, and a heat removal capacity of greater than 25% due to the half-open TBVs and ADVs, an overcooling transient began.

USNRC HRTD 15-3 Rev 05/2008

Both AFW pumps actuated about the time the reactor tripped due to the low MFW pump discharge pressure. These pumps began to supply AFW flow to both steam generators through the half open AFW (ICS controlled) flow control valves.

The operators recognized that power had been lost to the ICS about two minutes after the reactor trip, but they did not initially understand the plant response to this loss of power.

The operators also recognized the beginning of an overcooling transient due to the 50%

demand to the TBVs, ADVs, and the AFW valves. Realizing that these components could not be operated from the control room due to the loss of power, equipment operators were dispatched to close manual isolation valves. The operators could have closed the valves from the remote shutdown panel. However, they over-looked that method.

Operators in the control room noticed pressurizer level decreasing and fully opened the "A" high-pressure injection valve for more makeup flow to the RCS. The makeup tank level began to rapidly decrease, and the operators opened the BWST suction valve to the makeup pumps and started the "B" HPI pump.

Believing that significant main feedwater (MFW) flow (chart recorder also failed to mid-scale) existed, the operators tripped both MFW pumps. The AFW system was supplying about 1000 gpm to each steam generator at this time. With failed steam valves and excessive auxiliary feedwater flow, the RCS pressure and temperature continued to decrease.

15.2.4 Safety Features Actuation System (SFAS) Actuation In a little less than three minutes after the reactor trip, RCS pressure decreased from 2298 psig to the SFAS actuation setpoint of 1600 psig, and pressurizer level had decreased from 220 in. to 15 in. The actuation signal opened all four HPI motor operated valves and placed a second make-up pump in service. Also, the makeup tank outlet valve and the pump recirculation valves were closed. Selected emergency equipment, including the motor driven AFW pump, were automatically shed from the vital buses and sequence loading of SFAS equipment began. The AFW (SFAS) valves came fully open. Even with full high pressure injection flow, RCS pressure continued to decrease.

The control room operators, recognizing that the AFW flow was excessive, initiated an override of the SFAS signals to the AFW (SFAS) valves and closed them. However, the AFW (ICS) flow control valves remained at the 50% position. Meanwhile, the motor driven AFW pump sequenced back onto its vital bus.

During this time frame, several people checked the ICS power cabinets. It was realized that all four 24-vdc power supplies were de-energized; however, no one seemed to recognize that switches S1 and S2 were open.

Because of continued overfeeding and excessive steam dumping, the cooldown of the RCS continued. When steam generator pressure decreased to 500 psig, the running USNRC HRTD 15-4 Rev 05/2008

condensate pumps began to supply feedwater to the OTSGs. The feeding by the condensate pumps added approximately 1000 gpm flow to each steam generator. The RCS temperature had cooled 100 degrees in the first 7 min. following the reactor trip.

Later, the RCS pressure decreased to a low of 1064 psig, and the pressurizer water level dropped off-scale. Subsequent evaluation indicated that a small steam bubble formed in the upper head region of the reactor vessel.

15.2.5 Partial RCS Repressurization The transient continued with the pressurizer level off-scale low, and with full high-pressure injection in progress. The operators outside of the control room were working feverishly to isolate the sources of released steam and the excessive AFW flow. Although pressurizer level was off-scale, the RCS subcooling margin was substantial (85 degrees and increasing). The subcooling margin began to increase prior to the reactor trip and did not decrease to the pre-trip value of 40 degrees at any time during the transient. The high subcooling margin while the pressurizer level was off-scale was an indication to the operators that the pressurizer had not completely emptied, but the pressurizer was empty for approximately 3 minutes.

Although the cooldown continued, the flow from the high-pressure injection system apparently began to refill the pressurizer, although the level was still below the indicating range. RCS pressure also began to increase from a low point of 1064 psig. The continued cooldown, combined with the RCS pressure increase resulted in conditions that exceeded the B&W pressure/temperature limits for pressurized thermal shock of the reactor vessel.

However, the nil ductility transition temperature technical specifications limits were not violated.

The control room operators throttled the HPI flow slightly as RCS pressure and subcooling margin continued to increase. The cooldown had now decreased steam generator pressures to about 435 psig, causing the main steam line failure logic to actuate.

This actuation closed the feedwater flow control valves, stopping flow from the condensate pumps. The flow had lasted for approximately two minutes.

Nine minutes after being dispatched by the control room, the operators at the TBVs and the ADVs reported that the valves had been isolated. However, the non-licensed operator at the AFW (ICS) flow control valves was experiencing some difficulty in closing them. He used the valve handwheel to partially close the "B" AFW (ICS) flow control valve, although he thought he had completely closed the valve. As a result, the flow continued to the "B" steam generator, decreased by about 40 percent. He then went to the "A" AFW (ICS) flow control valve and closed it with the valve handwheel. Closing this valve caused the flow through the "B" AFW flow control valve to increase because much of the flow that had been going through the "A" AFW (ICS) flow control valve was apparently being redirected through a line cross connecting the two valves. However, the operator believed that the "A" AFW flow control valve was only 80 percent closed since he could still see about 1/2-inch of unaffected valve stem. Using a valve wrench, he applied additional force USNRC HRTD 15-5 Rev 05/2008

to the valve, which resulted in failure of the manual operator, whereupon the valve reopened. As a result, local manual control of the valve by the valve handwheel was no longer possible.

In the meantime, a second non-licensed operator arrived at the "B" AFW (ICS) flow control valve and fully closed it completely. The first operator then called the control room and was told to close the "A" AFW manual isolation valve. Since it would not move, even after he applied a valve wrench, it remained open. The "A" AFW (ICS) flow control valve also remained open until ICS power was restored. Because of its location, the second operator found it expeditious to jump a controlled area fence approximately 6 feet high when going from the "B" to the "A" AFW (ICS) flow control valves. This appeared to have saved about 2 minutes.

Meanwhile, in the control room pressurizer level was back on-scale and increasing so that operators started to throttle all the HPI valves to slow the increase in RCS pressure.

The subcooling margin was 170 °F.

The operators opened the HPI pump, SFAS-controlled, recirculation valves to prevent the pumps from overheating when flow was subsequently further throttled. However, the suction valve from the makeup tank was still closed at this time. Recirculation flow was sent to the makeup tank, which soon filled, and the relief valve began to discharge to the flash tank.

The operators in the control room stopped the "C" reactor coolant pump (RCP) and the "A" HPI pump at an RCS temperature of 418 °F.

The Shift Supervisor, Shift Technical Advisor, and the Senior Control Room Operator had earlier discussed whether the AFW pumps should be tripped. The emergency procedures had been modified after the cooldown transient of October 2, 1985 to require that the AFW pumps be tripped during an overcooling transient if the steam generator could not be isolated by shutting valves. The Shift Supervisor, however, made the decision to delay tripping the AFW pumps. The operators were concerned that AFW might not be available, when later required, if the AFW pumps were tripped.

Meanwhile, the chart recorders indicated that the A steam generator was overfilling with the overflow entering the steam lines. The safety parameter display system (SPDS) video screen also showed steam generator levels and this indication was later reported to have indicated the steam generators were not full. The "A" steam generator actually filled to the top of the steam shroud and began to spill water into the steam annulus and into the main steam line for about 7 minutes until ICS power was restored. The AFW flow rate to the "A" steam generator at this time was off-scale high (greater than 1300 gpm). (A later evaluation and inspection showed there was no apparent damage to the main steam lines or the turbine driven AFW pump as a result of this overflow).

USNRC HRTD 15-6 Rev 05/2008

The makeup tank (MUT) was still receiving the HPI pump recirculation flow and, in turn, was relieving to the flash tank. The control room operators, therefore, closed the suction valve from the borated water storage tank in an attempt to mitigate the high level in the MUT, forgetting the suction line from the MUT was shut. This action isolated the suction to the makeup pump, the "A" HPI pump (which had been stopped earlier), and the "A" low pressure injection (decay heat removal) pump, which was in recirculation.

While the steam releases had been isolated earlier, the "A" AFW (ICS) flow control valve was still open which produced an RCS cooldown rate of approximately 200 °F per hour. The RCS subcooling margin peaked at 201°F at 4:39 a.m. at an RCS temperature of 390°F and an RCS pressure of 1430 psig. This was about 800 psig higher than the pressure limit for the pressurized thermal shock region at this temperature.

Finally, at 4:40 a.m. the backup Shift Supervisor had returned back to the control room after having helped to isolate the steam release and discovered that switches S1 and S2 to the ICS dc power supplies were tripped to the OFF position.

15.2.6 Restoration of ICS dc Power and Plant Stabilization Twenty-six minutes after it was lost, ICS dc power was restored when switches S1 and S2 were turned back to the ON position. With power restored, normal remote control of ICS equipment in the control room also was restored. Shortly after ICS power was restored, the Senior Control Room Operator (SCO) called the NRC Operations Center and reported an Unusual Event. The SCO briefly described the event and promised to call back later with additional details. The operators were now able to stop the RCS cooldown and continue to depressurize out of the pressurized thermal shock region. The main items of interest during this period were the damage to the makeup pump, which subsequently released radioactivity, the illness of the backup Shift Supervisor, and an additional loss of ICS dc power.

When power to the ICS was restored, apparently all the ICS-controlled valves shifted to the manual mode and received a demand signal to go fully open, a condition that was unexpected by the operators. However, the isolation valves for the TBVs and ADVs had been closed and the "B" AFW (ICS) flow control valve had been shut with the handwheel.

The control room operators immediately shut all open ICS-controlled valves, including the open "A" AFW (ICS) flow control valve, from the control room. All AFW flow to both steam generators was now stopped, and the RCS began to heat up. The lowest RCS temperature reached was 386°F. (The plant had cooled by 180°F in 26 minutes.)

At this time, RCS pressure was being reduced to achieve conditions outside the pressurized thermal shock region. The operators were directed to disengage the manual handwheel on the "B" AFW (ICS) flow control valve and to open the isolation valves for the ADVs and TBVs so that the ICS could completely resume control of these valves. The "A" steam generator level decreased below the steam shroud shortly after the "A" AFW (ICS) flow control valve was closed.

USNRC HRTD 15-7 Rev 05/2008

The RCS cooldown had been arrested, so operators stopped the "B" HPI pump and closed the open HPI injection valves ("A" and "B"). However, they left the makeup pump operating. The "A" HPI pump had been stopped earlier. The operators attempted to restore normal makeup flow through the makeup valve. However, the makeup isolation valve could not be opened from the control room because the operators did not reset one of the SFAS isolation signals for this valve.

A short time after stopping the "B" HPI pump, the operators noticed a loss of reactor coolant pump seal flow (they were alerted by an alarm and low flow indication) and restarted the "B" pump to reestablish seal flow. They checked the valve lineup to the seals and again stopped the "B" makeup pump. Again, flow to the seals stopped and the "B" HPI pump was restarted. What the operators did not realize was that the makeup pump was severely damaged and could not supply adequate reactor coolant pump seal injection flow.

(The "A" decay heat removal pump was apparently not damaged since it was operating with its recirculation line open and therefore discharging back to its own suction.)

Coincident with this seal flow problem, the auxiliary building stack radiation monitor alarmed. A smoke alarm was also received that isolated the auxiliary building ventilation system. The inadequate seal flow and radiation alarms were apparently all caused by the failure of the makeup pump that had been operating for about 10 min. with both suction valves (BWST and MUT) closed. At 5:00 a.m., the operators in the control room heard a loud noise and observed that the makeup pump ammeter was reading only about 1/3 of normal running current. It was then that they realized that the makeup pump had been damaged. They also discovered that both makeup pump suction valves were closed and immediately opened the suction valve from the MUT in the hope of preventing further damage to the pump. Opening the valve allowed water to spill from the damaged makeup pump onto the makeup pump room floor. The operators closed the valve after approximately 450 gallons had spilled.

The failed makeup pump had only a single stop-check valve that isolated the RCS from the failed makeup pump seals. In addition, the makeup pump was isolated from the operating HPI pump recirculation line by only a single stop-check valve. Consequently, there was some concern on the part of the Control Room Operators that this failure could lead to a small-break LOCA. Therefore, the Shift Supervisor sent two non-licensed operators to enter the makeup pump room and isolate the makeup pump by closing the locked-open manually operated suction valves, discharge valves, and the recirculation line isolation valve. The makeup pump room contained airborne radioactivity, and contaminated water on the floor. Although the non-licensed operators wore some protective clothing, they did not wear respirators or high top boots because none were available near the pump room entrance. They performed a radiation survey before entering the makeup pump room, however, no assessment was made of particulate or gaseous radioactivity until after they entered.

USNRC HRTD 15-8 Rev 05/2008

After isolating the makeup pump, the operators entered the west decay heat removal cooler room to attempt to open the SFAS-actuated makeup isolation valve by hand. This valve still had a close signal so they were unable to open it. The operators later found that the SFAS signal had not been reset for the makeup isolation valves at the B safety features panel. (Following actuation of the SFAS, the makeup valve close signal must be cleared at both the "A" and "B" safety features panels.) The operators then went back into the makeup pump room briefly to check its status and then left the area. (Both operators were monitored and whole body counted on the morning of December 26. The results showed that they had not received a significant radiation dose from the entry into the makeup pump room.)

Meanwhile, the backup Shift Supervisor became ill in the control room and collapsed in front of the control panel. He had assisted in isolating the ADVs, which are located outside where the weather was cold and damp. At this time an additional Senior Control Room Operator arrived at the plant. He was not scheduled to be on shift and had arrived early to do some paperwork. When he reached the control room, he turned his attention to the backup shift supervisor who had become ill. After discussing the situation with the Shift Supervisor, he called an ambulance. The supervisor was later transported to the hospital and later released. The supervisors illness diverted the attention of the control room operators and resulted in the loss of the supervisor, although it did not have a significant effect on the incident. (The utility stated, during the investigation, that based upon the medical diagnosis at the hospital and other available information, there was no indication that drugs or alcohol were involved with the illness.)

After calling the ambulance, the off-duty SCO answered the Emergency Notification System (ENS) phone when the NRC Operations Center called and requested an update on the plants initial report. After the SCO briefed the NRC Operations Center, he was requested to maintain an open line. The open line was maintained until the Unusual Event was terminated. Operators in the control room were intent on stabilizing the plant and bringing all systems and parameters to normal where possible. The RCS had now depressurized out of the pressurized thermal shock region and a 3-hour soak at the existing RCS temperature and pressure (870 psig and 428°F) was begun in accordance with B&W guidelines. Operators began to drain the overfilled steam generator to the condenser to reestablish MFW flow with the main condensate pumps.

The Shift Supervisor, concerned about the habitability of the auxiliary building after the ventilation system shutdown, decided to restart the ventilation system. However, a smoke detector alarm in the radiological waste area prevented the ventilation system from operating. The smoke detector in the radiological waste area is believed to have detected smoke from the reactor building radiation monitor, which overheated when its suction was isolated by the SFAS actuation. Efforts to start the auxiliary building ventilation system were finally successful, and ventilation from the auxiliary building to the atmosphere was restored. (The maximum permissible radionuclide concentration at the site boundary was later calculated to be less than one-fifth of the maximum permissible concentration for 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. The whole body dose to a person hypothetically at the site boundary during the event USNRC HRTD 15-9 Rev 05/2008

would have been no greater than 0.2 mRem. The thyroid dose would have been zero mRem. These results are well within Rancho Seco Technical Specification limits.)

After assisting in isolating the makeup pump, a non-licensed operator noticed he had lost his security badge. Thus, he was no longer able to open doors to the areas that require a badge for entrance. After reporting the loss to the control room, he was escorted by a security guard to the control room where he remained until a spare security badge was brought to him about 20 minutes later.

The SFAS signal was also bypassed at 6:06 a.m. At approximately 6:10 a.m., the plant was stabilized. The main steamline failure logic had been inhibited and the steam generators were being fed by the main condensate pumps.

At 6:11 a.m., a momentary ICS or Fan Power Failure alarm occurred. The S1 and S2 switches remained closed and the alarm cleared without operator action. No equipment response was noted.

At 6:14 a.m., a third ICS or Fan Power Failure alarm was received. The ICS-controlled valves again received a 50 percent demand signal. The operators immediately reset switches S1 and S2 to restore ICS power. This caused the ICS-controlled valves to receive a 100 percent demand signal. The operators then closed the valves remotely from the control room.

The Plant Superintendent relieved the Shift Supervisor as Emergency Coordinator and manned the Technical Support Center (TSC) at 7:15 a.m. Meanwhile, several gallons of water had spilled onto the TSC floor. The water came from a drain on a pilot-operated valve in the fire main when a fire alarm was received and the normally dry fire header was pressurized with water. There was no release of water from the fire main header and the spilled water had no significant effect on this incident. The Emergency Coordinator terminated the Unusual Event at 8:41 a.m.

15.3 Major Issues The post event evaluation and investigation resulted in the compilation of an action list that consisted of 14 sections, with several items contained in each section. Three of these sections are discussed in the following paragraphs.

15.3.1 ICS Power Failure There are redundant dc power supplies for the ICS, whose outputs are auctioneered to provide +24 Vdc and -24 Vdc power for use in various modules within the ICS. The outputs of each of the +24 Vdc and -24 Vdc power supplies are monitored by a single power supply monitor (PSM) module, which will alarm in the control room if the output voltage of any of the dc power supplies drops to 23.5 Vdc. The PSM also senses the auctioneered output of each pair of power supplies and will trip open the input switches, S1 and S2, if the voltage USNRC HRTD 15-10 Rev 05/2008

on either output bus drops to 22 Vdc. This results in interrupting all dc power within the ICS. This is a designed response to prevent voltage fluctuations from causing the ICS to behave in an erratic or unpredictable manner.

Early findings determined that the PSM trip setpoint was drifting. Significant to the event was the finding that small amounts of resistance, as little as one ohm, in series with the sensed voltage to the PSM could cause tripping of the PSM. During the troubleshooting process it was found that the time delay drop of switches S1 and S2 was much shorter than the manufacturer's tolerance.

This time delay feature may explain why the intermittent ICS or fan power failure alarm received at 06:11 did not result in a trip. During the investigation, it was reported that this alarm had also occurred intermittently on at least two occasions in the weeks before the trip, although no documentation of these alarms could be found and no work requests to investigate them could be located. The fact that the time delays on S1 and S2 were shorter than designed (which is in the conservative direction) may have contributed to the loss of ICS power in this event. Had they been within the design tolerance, the voltage fluctuation caused by an improperly crimped wire may not have been long enough in duration to cause S1 and S2 to trip.

During inspections of the physical wiring in the ICS cabinets, and in an attempt to identify the source of a possible increased input resistance, the technicians found that the input lead wire to the +/-24 Vdc monitor had an improperly crimped connection at the +24 volt bus bar in cabinet one. In fact, the lug fell free from the wire when it was disconnected from the bus. This wire is an internal cabinet wire installed by the vendor prior to delivery of the cabinets.

Following the discovery of the improperly crimped connection, the Quality Control group inspected the remainder of the factory and field wire terminations in ICS and NNI cabinets, and found numerous deficiencies in the quality of the wire terminations in these cabinets. The root cause for the loss of the ICS power was identified as the loose connection. The factory bus wiring has been replaced with current standard wiring.

15.3.2 Makeup Pump Failure At time 04:16:57, the makeup tank isolation valve shut on Safety Features Actuation System (SFAS) initiation due to reactor pressure falling below 1600 psig. During the subsequent SFAS recovery, the valve was not reopened. At 04:30 operators secured the A HPI train which included shutting the supply valve from the Borated Water Storage Tank (BWST). They did not realize that this action also isolated the suction to the Makeup Pump. Upon securing the B HPI train at 04:42, RCP seal injection flow decreased toward zero. The B HPI was restarted, the lineup verified, and the pump secured again. Once again RCP seal injection flow decreased toward zero. At 05:00, operators statements indicate that a loud noise was heard in the plant. The operators then realized that the makeup pump was without suction and tripped the pump and opened the makeup tank USNRC HRTD 15-11 Rev 05/2008

isolation valve. The makeup pump had run for nearly thirty minutes with its suctions to both the BWST and the makeup tank closed. However, by this time a pump seal had been badly damaged and over 1200 gallons of water from the makeup tank spilled to the pump room floor prior to reclosing the makeup tank isolation valve.

Operating procedures for the HPI, Makeup and SFAS systems were reviewed. No procedural direction could be found instructing operations to reopen the makeup tank isolation valve during SFAS recovery or otherwise ensuring an adequate source of water to the makeup pump. In addition to the absence of a procedural requirement to prevent this occurrence, there were no warning or control devices such as a pump low suction alarm or trip, nor interlocks between the makeup tank or BWST outlet valves and the pump to ensure adequate suction existed.

The root cause of the damage to the makeup pump was determined to be inadequate procedures. The procedural inadequacy is evidenced in this regard by the lack of guidance for operators to recover from SFAS initiation. A contributory cause of the damage to the pump was personnel error, as evidenced by the operators performing valve operations which isolated the suction of the makeup pump while the pump was running.

15.3.3 RCS Overcooling The reactor trip and overcooling event was initiated by a loss of ICS power and subsequent repositioning of valves that regulate the rate of RCS heat removal. When the ICS DC power supply tripped, voltage output from the hand/auto stations that control main feedwater (MFW) regulating and startup valve positions and MFW pump speed failed to mid-scale (0 Vdc), resulting in a reduction in MFW flow. Within 16 seconds, the loss of heat transfer in the once-through steam generators (OTSGs) caused a reactor trip on high reactor coolant system (RCS) pressure. The hand/auto stations for the ICS controlled auxiliary feedwater (AFW) valves, the atmospheric dump valves (ADVs), and the turbine bypass valves (TBVs) also failed to mid-scale, resulting in those valves opening to their 50% demand position.

Steaming through the ADVs and TBVs was a steam demand significantly in excess of decay heat generation. Additionally, both AFW pumps started due to low MFW pump discharge pressure and were delivering feedwater to the OTSGs within a few seconds of the reactor trip. This excessive heat transfer condition created an RCS cooldown rate in excess of technical specification limits that was not fully controlled for 26 minutes.

Remote control was also lost. No procedural guidance was in place to identify other points of remote control (such as the Appendix R remote shutdown panel), or toward recovering power to the ICS.

The issue of potential loss of ICS power has been raised in numerous vendor and regulatory documents over the past 8 years as well as being included as an independent casualty in B&Ws Abnormal Transient Operating Guidelines (ATOG).

USNRC HRTD 15-12 Rev 05/2008

The deficiency in this area was the lack of procedural direction aimed at ICS recovery, or guidance toward alternate points of control which will more quickly mitigate a transient.

The root cause was determined to be failure to implement changes addressed in 1980 concerning the potential cooldown affects of the turbine bypass valve failure mode during a loss of ICS. At that time the utility was alerted to the design deficiency and was requested to make corrections be made in a timely manner.

15.3.4 PRA Insights All major events are reviewed by Oak Ridge National Laboratory from a PRA standpoint. Each year NUREG/CR-4674, Precursors to Potential Severe Core Damage Accidents, publishes the results of this review. The Rancho Seco results were published in the 1985 issue. The NUREG lists the major sequences that result in core vulnerability and core damage.

The major core vulnerable sequence is shown in Figure 15-6. The sequence involves a transient (initiated by the loss of ICS power), a reactor trip, success of AFW, success of Main Feedwater, a challenge to the PORV ( the valve was opened to control RCS pressure), success of the reseating of the PORV, and failure of the HPI system (one pump was rendered inoperable by the incorrect suction lineup). The conditional probability for this sequence is 1.766E-4.

The major core damage sequence is shown in Figure 15-7. The sequence involves a transient initiator, successes of AFW and MFW, failure of the PORV to reseat, and HPI.

The conditional probability of this sequence is 3.0E-5/Rx-yr.

15.4 References

1. U.S. Nuclear Regulatory Commission. Loss of Integrated Control System Power and Overcooling Transient at Rancho Seco on December 26, 1985, USNRC Report NUREG 1195, February 1986.

2. Docket 50-312, Rancho Seco Nuclear Generating Station Unit 1, Resolution of Issues Regarding the December 26, 1985 Reactor Trip, February 19, 1986.

3. Sacramento Municipal Utility District Incident Analysis Root Cause 85-41, March 19, 1986.

USNRC HRTD 15-13 Rev 05/2008

APPENDIX - SEQUENCE OF EVENTS INITIAL CONDITIONS Average temp 582°F. RCS pressure 2150 psig.

Reactor Power 76%. ICS in full automatic control.

Note: Rancho Seco does not have main steam isolation valves.

TRANSIENT INITIATOR - LOSS OF ICS DC POWER 04:13:47 Loss of ICS DC power (power supply monitor failed).

04:13:+ Main feedwater flow decreasing rapidly. MFPs to minimum speed. RCS pressure increase, spray valve opened manually.

04:14:01 AFW initiated on low MFP discharge pressure (<850 psig).

PLANT TRIP AND START OF COOLDOWN 04:14:03 Reactor trip on high RCS pressure. PZR spray closed.

04:14:04 RCS pressure peaks at 2298 psig. Six OTSG safety valves open and later reseat.

04:14:06 Second AFW pump starts on low MFP discharge pressure.

04:14:06 Peak hot leg temperature of 606.5°F reached.

04:14:+ Operators start to perform E.01 - reactor trip letdown isolated. Operators start E.02 - vital systems verification 04:14:11 AFW flow to both OTSGs via 50% AFW (ICS) valves.

04:14:25 PZR level decreasing. A HPI MOV opened to increase makeup flow to the RCS.

04:14:30 Overcooling symptoms noted. Loss of ICS DC power has positioned the following valves to 50%:

1. Turbine Bypass Valves

2. Atmospheric Dump Valves

3. AFW Flow Control Valves 04:14:48 Makeup tank level decreases due to excessive RCS makeup. MU pump suction shifted to BWST.

04:15:04 B makeup (HPI) pump started for additional makeup.

04:16:02 AFW flow to OTSGs >1000 gpm. MFW flow indicating about 3 million pounds per hour. Actual MFW flow is zero because of MFP speed. In addition, the main feedwater stop valves were closed. However, a flowpath from the MFP to the OTSGs is available through the startup main feedwater valves.

USNRC HRTD 15-14 Rev 05/2008

SEQUENCE OF EVENTS (continued)

SFAS ACTUATION - COOLDOWN - DEPRESSURIZATION 04:16:57 RCS pressure decreased from 2298 psig to 1600 psig. SFAS actuated at the setpoint of 1600 psig. PZR level dropped from 220 inches to 15 inches. The SFAS actuation opened all 4 HPI MOVs to predetermined positions. The following equipment was also actuated by the SFAS signal:

1. MUT outlet valve closed - BWST supply to HPI opened.

2. MU pump recirculation path isolated.

3. AFW valves to the 100% position.

4. LPI/DHR pumps start.

5. Both EDGs start.

04:16:59 The "A" HPI pump started by SFAS. "B" HPI already running. Both HPI pumps and the MU pump supplying MU to RCS.

04:17:10 AFW (SFAS) flow control valves manually closed.

04:17:15 A & B emergency air conditioning units auto start. Significant increase in control room noise level.

04:17:27 Motor driven AFW pump auto sequenced back to the vital bus. Dual drive AFW running on steam source.

04:18:58 RCS temperature less than 500°F.

04:19:00 Pressurizer emptied. Steam bubble in reactor vessel head.

04:19:15 Emergency air conditioning stopped to reduce noise level.

04:20:00 STA to turbine deck to determine lifting relief valves.

04:20:00 PZR level off scale low. Subcooling margin of 85 degrees and increasing.

04:20:+ Technician sent to check ICS power supplies. All four 24-vdc power supplies were de-energized. The automatic bus transfer (ABT) did not transfer. The power supply to the ICS would be inspected by three people during the next 20 minutes without discovering that switches S1 and S2 feeding the 24-vdc supplies were open.

04:20:20 OTSG pressures at 500 psig. Feedwater into the OTSGs from condensate pumps via open startup feedwater valves. An additional 1000 gpm feedwater flow to the steam generator.

04:21:25 Minimum RCS pressure of 1064 psig (RCS temperature of 464°F) is reached.

USNRC HRTD 15-15 Rev 05/2008

SEQUENCE OF EVENTS (continued)

PLANT REPRESSURIZATION 04:21:+ RCS cooldown continuing. However, flow from the HPI pumps starts to increase RCS pressure even though pressurizer level is off scale low.

04:22:00 B&W pressure and temperature limits for PTS are exceeded; however, technical specification NDT limits are not exceeded. Operator starts to throttle HPI flow.

04:22:50 OTSG pressures decrease to 435 psig. Steam line break logic actuated.

Main and startup regulating valves are closed. Feedwater flow from the condensate pumps is terminated.

04:23:00 Atmospheric dump valves and turbine bypass valves were shut locally.

(Manual handwheels used.)

04:23:10 "B" AFW (ICS) flow control valve partially closed. Operator thought valve was fully closed. Flow has decreased by 40%.

4:25:30 HPI recirculation valves to makeup tank opened. HPI pump suction being supplied by BWST.

4:26:22 "A" AFW flow control valve closed locally. Operator thinks that the valve is only 80% closed.

04:26:47 Pzr level back on scale. Subcooling margin is 170 degrees. Operators throttle HPI to slow the increase in RCS pressure.

04:28:00 Makeup tank level off scale high. Makeup tank relief valve lifts.

04:28:00 RCP C stopped at an RCS temperature of 418°F.

04:28:43 RCS letdown restored.

04:28:59 A HPI stopped.

04:29:40 Operator damages A AFW flow control valve in an attempt to close valve

>80% closed. Operator directed to close AFW manual isolation.

04:29:40 RCS pressure peaks at 1616 psig. RCS temperature is 418°F.

04:29:45 C and D HPI valves are closed in order to reduce the repressurization while temperature is decreasing.

04:30:00 An unusual event is declared.

04:30:30 Plant is depressurized using PZR spray in an attempt to restore PTS limits.

04:33:20 B AFW flow control valve closed by second operator. AFW flow to B OTSG is stopped.

04:33:20 A OTSG filled to top of steam shroud. Water begins to spill into the steam lines. Flow into the OTSG is in excess of 1300 gpm.

04:35:+ The A HPI suction valve from the BWST is closed in an effort to lower makeup tank level. However, the makeup tank outlet valve is still closed.

04:36:+ The manual AFW isolation valve cannot be closed by the operator.

04:39:00 RCS subcooling margin reached a peak of 201 degrees and began to decrease (RCS temp = 390°F, RCS press = 1430 psig). This is approximately 800 psig into the PTS region.

USNRC HRTD 15-16 Rev 05/2008

SEQUENCE OF EVENTS (continued)

ICS POWER RESTORATION AND PLANT STABILIZATION 04:40:00 The Backup shift supervisor finds switches S1 and S2 in the OFF position.

The switches were closed. The valve stations reverted to the HAND position.

All valves with the exception of the AFW flow control valves had been isolated. The control room operators closed the AFW (ICS) flow control valves.

04:40:+ With AFW flow isolated, the RCS starts to heat up. The lowest RCS temperature was 386°F. The RCS had cooled down 180 degrees in 26 minutes.

04:41:00 Operators report that the A AFW manual isolation valve is stuck open.

Operators are directed to disengage the handwheel for the B AFW flow control valve and to return the ADVs and TBVs to service.

04:41:10 A OTSG level below the steam shroud.

04:42:42 B HPI pump stopped. The makeup pump remains in service.

04:42:56 Operators closed the A and B HPI MOVs.

04:43:50 RCP seal injection low flow.

04:43:54 B HPI pump restarted to reestablish RCP seal injection flow.

04:40:+ Steam leakage from the damaged makeup pump causes an auxiliary building stack radiation monitor alarm. Makeup pump damaged due to a lack of suction. Radioactive release is within Tech. Spec. limits.

04:50:19 B HPI pump stopped.

04:50:30 B HPI pump restarted in response to a low RCP seal flow alarm. The operators have not realized that the makeup pump has been damaged.

04:52:+ Backup shift supervisor collapsed in control room. This operator had assisted in closing the ADV and TBV manual isolations.

05:00:+ Control room operators hear a loud noise. They observe that the makeup pump ammeter is reading low and realize that the pump has been damaged.

05:00:10 Makeup pump is tripped. Makeup tank outlet valve is opened allowing 450 gallons of water to spill out of the damaged pump. Makeup tank outlet valve is reclosed.

05:05:+ RCS pressure decreased out of the PTS region. A 3 hour3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> soak is initiated.

(RCS temp = 428°F, RCS press = 870 psig).

USNRC HRTD 15-17 Rev 05/2008

SEQUENCE OF EVENTS (continued) 05:05:+ Ambulance called for backup shift supervisor.

05:27:+ Two auxiliary operators are contaminated while isolating the damaged makeup pump. Operators did not follow proper radiological safety procedures.

05:29:+ Operators are unable to open makeup isolation to the RCS. It was later found that the SFAS signal had not been cleared from the valve.

05:29:04 Second reactor coolant pump stopped.

06:06:00 Operators bypass SFAS.

07:15:+ Plant superintendent relieves shift supervisor as emergency coordinator.

08:41:+ The unusual event is terminated.

USNRC HRTD 15-18 Rev 05/2008

Figure 15-1 ICS Power Supplies This page intentionally blank Figure 15-2 Main Steam System This page intentionally blank Figure 15-3 Main Feedwater System This page intentionally blank Figure 15-4 Auxiliary Feedwater System This page intentionally blank Figure 15-5 High Pressure Injection System This page intentionally blank Figure 15-6 Dominant Core Vulnerability Sequence Event Tree This page intentionally blank Figure 15-7 Dominant core Damage Sequence Event Tree This page intentionally blank