ML20151X391

Loss of Power and Water Hammer Event at San Onofre,Unit 1 on November 21,1985

ML20151X391
Person / Time
Site:	San Onofre
Issue date:	01/31/1986
From:	NRC - INCIDENT INVESTIGATION TEAM
To:
References
NUREG-1190, TAC-62010, NUDOCS 8602120393
Download: ML20151X391 (200)

Text

- - _

NUREG-1190 Loss of Power and Water Hammer Event at San Onofre, Unit 1, on November 21,1985 U.S. Nuclear Regulatory Commission i[ .....

/

f I

i

! 8602120393 860131 j ADOCK 0500 6 i

{DR

. . _ _ . . . - - . - . . . , . . , - . . . . _ - . . . . . _ , - . . . . . . . . , , . . . . , , . , - , ~ . _ . . _ _ . . . , , - _ , - , , . , , - . _ , , , - . - _ , . , - - - , . . - _ . . . . , . . .

NOTICE Availability of Reference Materials Cited in NRC Publications Most documents cited in NRC publications will be available from one of the following sources:

1. The NRC Public Document Room,1717 H Street, N.W.

Washington, DC 20555

2. The Superintendent of Documents, U.S. Government Printing Office, Post Office Box 37082, Washington, DC 20013-7082 I

3. The National Technical Information Service, Springfield, VA 22161 Although the listing that follows represents the majority of documents cited in NRC publications, it is not intended to be exhaustive.

Referenced documents available for inspection and copying for a fee from the NRC Public Docu-ment Room include NRC correspondence and internal NRC memoranda; NRC Office of Inspection and Enforcement bulletins, circulars, information notices, in:pection and investigation notices; Licensee Event Reports; vendor reports and correspondence; Commission papers;and applicant and licensee documents and correspondence.

The following documents in the NUREG series are available for purchase from the GPO Sales Program: formal NRC staff and contractor reports, NRC-sponsored conference proceedings, and NRC booklets and brochures. Also available are Regulatory Guides, NRC regulations in the Code o/

Federal Regulations, and Nuclear Regulatory Commission Issuances.

Documents available from the National Technical Information Service include NUREG series reports and technical reports prepared by other federal agencies and reports prepared by the Atomic Energy Commission, forerunner agency to the Nuclear Regulatory Commission.

Documents available from public and special technical libraries include all open literature items, such as books, journal and periodical articles, and transactions. Federal Register notices, federal and state legislation, and congressional reports can usually be obtained from these libraries.

Documents such as theses, dissertations, foreign reports and translations, and non-NRC conference proceedings are available for purchase from the organization sponsoring the publication cited.

Single copies of NRC draft reports are available free, to the extent of supply, upon written request to the Division of Technical Information and Document Control, U.S. Nuclear Regulatory Com-mission, Washington, DC 20555.

Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are maintained at the NRC Library, 7920 Norfolk Avenue, Bethesda, Maryland, and are available there for reference use by the public. Codes a.1d standards are usually copyrighted and may be purchased from the originating organization or, if they are American National Standards, from the American National Standards Institute,1430 Broadway, New York, NY 10018.

l NUREG-1190 Loss of Power and Water Hammer Event at San Onofre, Unit 1, on November 21,1985 ateYu shed u U.S. Nuclear Regulatory Commission Washington, D.C. 20555 p>"'=m, s /

AUXILIARY TRANSFORMER C" TURBINE GENERATOR FEEDWATER PACIFIC OCEAN BEACH CONDENSATE STORAGE TANK MEZZANINE e .- -

sp - .* .

N, 4; ppm w- g-

+'

,;;.;$- ;;y f

'^

J~x s,%; (-x,

, -, ;p y.ph

'* ~'

,, s , -  ;-+, .

.j

..+n

q # ,_ -e a w4mg WiWWNa.i Q f[4 N- 'A ,

l- A'. , , , , , -: x . 4 H$

. ,* ** ~* M' CONTAINMENT 4 < ,

e

'_ 1 de,gi SPHERE

• ENCLOSURE A4 . ,

A"? -7[

^

' fsj _ ;. .. y .

AUXILIARY

" , I,b FEEDWATER ,

E".{' '

STORAGE #r ' , s *f/ .

_ 7 f TANK ',/ 4 8

[. > $,Y .

~. f;. _fgl

-~ -c:

g;j n' ,y]

• f

• f,,

si . !

- - p es . ,;

+' ~M--

~, 2

[W ~,

l.

p W;

bf(j !, , . ,

AuntLIARY *5 I -

s m' .

THANSFORMERS -

D A'L A AND B  :- -

j [ . *?( *

, DIESEL tp>n' W ,u ' .a,,

.E GENERATOR BUILDING

,' f s

~

_;Q ' ,

t w

w . . ,,., 'y 4

! MAIN STEP-UP s 4 .

' j' * ' "r #

d#

~*

i TRANSFORMER I J f[ g;:

gQ

[. ' ,

~ ^'A",.' ?$)' z yw  ;,

~

~ -

l j ' *

' $$ '&. }2,lifllQ Y O$ ~ R. A  ::'L'2 2 = u CONTROL EAST FLASH EVAPORATOR UNIT EAST FEEDWATER HEATER PLATFORM ROOM Aerial Photo of San Onofre Unit 1 Looking West

m ABSTRACT On November 21, 1985, Southern California Edison's San Onofre Nuclear Generat-ing Station, Unit 1, located south of San Clemente, California, experienced a partial loss of inplant ac electrical power while the plant was operating at 60 percent power. Following a manual reactor trip, the plant lost all inplant ac power for 4 minutes and experienced a severe incidence of water hammer in the feedwater system which caused a leak, damaged plant equipment, and chal-lenged the integrity of the plant's heat sink. The most significant aspect of the event involved the failure of five safety-related check valves in the feed-water system whose failure occurred in less than a year, without detection, and jeopardized the integrity of safety systems. The event involved a number of equipment malfunctions, operator errors, and procedural deficiencies. This report documents the findings and conclusions of an NRC Incident Investigation Team sent to San Onofre by the NRC Executive Director for Operations in confor-mance with NRC's recently established Incident Investigation Program.

6 iii

TABLE OF CONTENTS Page Abstract............................................................. iii The NRC Team for the San Onofre Event of November 21, 1985........... ix Acronyms and Abbreviations........................................... x

1. INTRODUCTION................................................... 1-1

2. DESCRIPTION OF TEAM ACTIVITIES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1 2.1 General Approach........................................ 2-1 2.2 Interviews and Meetings................................. 2-1 2.3 Plant Data.............................................. 2-2

3. NARRATIVE OF THE EVENT......................................... 3-1 i 1

3.1 Plant Status............................................ 3-1

3. 2 Electrical Ground Fault................................. 3-2 3.3 Loss of Power........................................... 3-3 3.4 Four-Minute Blackout...... ............................. 3-6

3. 5 Conflicting Requirements................................ 3-8

3. 6 The Water Hammer..... ............ ..................... 3-9 3.7 S team Generators Boil D ry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10 3.8 Plant Cooldown.......................................... 3-11 3.9 Isolating the Leak...................................... 3-13

4. DESCRIPTION OF PLANT SYSTEMS................................... 4-1 4.1 General 0esign.......................................... 4-1 4.2 Main Steam System....................................... 4-1 4.3 Steam Generator Blowdown System......................... 4-2 4.4 Ma i n Feedwa te r Sys tem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 4.5 Condensate System....................................... 4-3 4.6 Aux il ia ry Feedwa te r Sys tem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4 4.7 Sal t Water Cool i ng Sys tems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-5 4.8 Turbine Plant Cooling Water System...................... 4-6 4.9 Chemical and Volume Control System...................... 4-7 4.10 Residual Heat Removal System............................ 4-8 4.11 Safety Injection System................................. 4-R 4.12 Electrical Distribution System.......................... 4-9 4.12.1 General Description.............................. 4-9 4.12.2 220 KV System.................................... 4-10 4.12.3 4160-Volt System................................. 4-11 4.12.4 480-Volt System....... . ......................... 4-13 4.12.5 120-Volt AC System.'..... ........................ 4-14 v

,i.i. ..-i.-' -

TABLE OF CONTENTS (Continued)

Page 4.13 Sa feguard Load Sequencing Sys tem. . . . . . . . . . . . . . . . . . . . . . . . . 4-15 4.14 Diesel Generators......................................... 4-16 4.15 Containment Building Isolation............................ 4-17 4.16 Stati on Pe rs onne l . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4- 17 4.16.1 Administrative Organization...................... 4-17 4.16. 2 On-Shi f t O rgani za ti on. . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18 4.16.3 Human Factors Affecting Operator Performance...................................... 4-19 4.17 Control Room Indication.................................. 4-21 4.17.1 Loss of Vital Bus 4.............................. 4-21 4.17. 2 Los s o f Uti l i ty 8 us . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-22 4.17.3 Data Recording Systems.... ...................... 4-23 4.18 NRC Emergency Response Organization....................... 4-24

5. LOSS OF POWER EVALUATION....................................... 5-1 5.1 Electrical System Operations Leading to Auxiliary Transformer C Di f ferential Relay Trip. . . . . . . . . . . . . . . . . . . 5-1 5.1.1 Relay Trip...................................... 5-3 5.1. 2 Discussion...................................... . 5-3 5.2 Unit Trip and Loss of All AC Power...................... 5-4

6. WATER HAMMER EVALUATION........................................ 6-1 6.1 Plant Conditions Leading to Water Hammer................ 6-2 6.2 Water Hammer-Induced Damage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-4 6.2.1 Piping and Piping Support Damage................ 6-4 6.2.2 Feedwater Loop 8 Flow Control Station Damage.................................. 6-5 6.2.3 AFW Piping Damage............................... 6-5 6.2.4 Valve Malfunctions and Damage................... 6-5 6.3 NRC Eval uations of Water Hammer. . . . . . . . . . . . . . . . . . . . . . . . . 6-7 6.3.1 H i s to ry and Focus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-7 6.3.2 SONGS-1 Water Hammer History and Evaluations. . . . 6-9 6.4 Valve Inservice Testing................................. 6-9 6.5 Feedwater System Check Valve Maintenance. . . . . . . . . . . . . . . . 6-12 6.6 Feedwater Train Noise Investigation..................... 6-12 6.7 Valve Failure-Related Findings.......................... 6-13 vi

TABLE OF CONTENTS (Continued)

Page

7. HUMAN FACTORS EVALUATION....................................... 7-1 7.1 7.2 Introduction...................................... 7-1 SONGS Pers onnel Pe r fo rmance. . . . . . . . . . . . . . . . . . . . . ...... . . . . . . . .7-1

7. 2.1 Ope ra to r Pe rfo rmance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1 l 7.2.2 Other Site Personnel Performance. . . . . . . . . . . . . . . . . 7-8 7.3 P ro c e d u re s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7-. .11. . .

7.3.1 ENS Communications Problems...................... 7-11 7.3.2 NRC Incident Response Plan Implementation........ 7-15 8.

OTHER EQUIPMENT AND SYSTEM EVALUATIONS......................... 8-1 8.1 Auxiliary Transformer C Secondary Side. . . . . . . . . . . . . . . . . . . .

8.2 Safety Injection 8-1 8.3 SLSS Remote SurveillanceAnnunciator........................... .. 8-1 8.4 Flash Evaporator Panels........................ 8-2 Unit...................................

8.5 Turbine Breakable Diaphragms (Rupture . 8-2 8.6 Reactor Coolant Pump B Thrust 8earin Disks).............. 8-2 Temperature Indication. . . . . . . . . . . . . . g

...................... 8-3 8.7 Steam 8.8 Generator Blowdown Isolation........... ............ 8-3 8.9 RHR Valve Interlock.......................... .... 8-3 Even t Reco rd ing Sys tems . . . . . . . . . . . . . . . . . . . . . . . . . . . ........ ...... 8-3 8.10 Emergency Notification System....................

8.11 Safeguards System................................ ........ 8-4

........ 8-4

9. PRINCIPAL FINDINGS AND CONCLUSIONS............................. 9-1 APPENDIXES A Memorandum from W. J. Dircks, Executive Director for Operations, to the Commission, " Investigation on November 21, 1985 Event at San Onofre Unit 1 Will Be Conducted by an Incident Investigation Team IIT "

November 22, 1985...........................(....).,.............. .

A-1 B

Plant Conditions When Water Hammer Occurred and

! Es tima ted Piping Support Loads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1 i

{ C Regulatory Review of Potential for Water Hammer l at San Onofre Unit 1............................................ C-1 i

vii

I THE NRC TEAM FOR THE SAN ONOFRE EVENT OF NOVEMBER 21, 1985 Thomas T. Martin, Team Leader Matthew Chiramal

.l William G. Kennedy

Wayne D. Lanning Aleck W. Serkiz i- S'.even K. Showe

~

i-TEAM SUPPORT STAFF Walter E. 011u, Division of Technical Information and Document = Control i

i 4

4 4

i f.

I I

r I

i l

j' 'I ix l 4

ACRONYMS AND ABBREVIATIONS ac Alternating Current ACO Assistant Control Operator AFW Auxiliary Feedwater ARMS Area Radiation Monitoring System ASB Auxiliary Systems Branch (NRC)

ASME American Society of Mechanical Engineers ATWS Anticipated Transient Without Scram BWR Boiling Water Reactor CB Circuit Breaker CCW Component Cooling Water C0 Control Operator CRS Control Room Supervisor CV Control Valve CVCS Chemical and Volume and Control System dc Direct Current EAL Emergency Action Level

, ENS Emergency Notification System l E0 Emergency Officer E01 Emergency Operating Instruction EPIP Emergency Plan Implementation Procedure ERMS. Effluent Radiation Monitoring System ESF Emerguncy Safeguards Features FCV Flow Control Valve FEMA Federal Emergency Management Agency FI Flow Indicators FW Feedwater FWS Feedwater Support ,

GDC General Design Criteria gpm gallons per minute HQDO Headquarters Duty Officer (NRC)

HVAC Heating, Ventilation, and Air Conditioning HX Heat Exchanger IIT Incident Investigation Team IRC Incident Response Center IST Inservice Testing kV Kilovolts kVA Kilovolts Ampere kW Kilowatts Ibf Pounds Force LCV Letdown Control Valve LCV Level Control Valve LER Licensee Event Report LMFW Loss of Main feedwater LOB Loss of Bus LOCA Loss of Coolant Accident LOP Loss of Offsite Power LP Low Pressure LR Level Recorder mA Milliamperes MCC Motor Control Center MFW Main Feedwater M0 Maintenance Order X

MOD Motor-0perated Disconnect MOV Motor-0perated Valve MVA Megavoit Amperes MWe' Megawatt (Electric)

MWt Megawatt (Thermal)

NPE0 Nuclear Plant Equipment Operator NRC Nuclear Regulatory Commission NRR Office of Nuclear Reactor Regulation (NRC)

NSSS Nuclear Steam Supply System OA/FA Oil-Air / Forced Air 01 Operating Instruction ORMS Operational Radiation Monitoring System PORV Power Operated Relief Valves POV Power-0perated Valve ppb parts per billion ppm parts per million PR Pressure Recorder PT Potential Transformer PWR Pressurized Water Reactor PZR Pressurizer RCS Reactor Coolant System RD0 Regional Duty Officer RHR Residual Heat Removal R0 Reactor Operator RPF0 RWST Reactor Plant First Out (Annunciators)

Refueling Water Storage Tank SCE Southern California Edison Company SDG&E San Diego Gas and Electric Company SEB Sphere Enclosure Building SER Safety Evaluation Report SG Steam Generator SGWH Steam Generator Water Hammer SIS Safety Injection System SLSS Safeguard Load Sequencing System SONGS-1 San Onofre Nuclear Generating Station, Unit 1 SRI Senior Resident Inspector SR0 Senior Reactor Operator SSAM Shift Supervisor's Accelerated Maintenance (Order)

SST Station Service Transformer STA Shift Technical Advisor T,y, Average Temperature TBCW Turbine Building Cooling Water TI Temperature Indicators TIC Temperature Indicating Control TPCW Turbine Plant Cooling Water T

ref Reference Temperature TR Temperature Recorder USI Unresolved Safety Issue V Volts VAC ac voltage VCT Volume Control Tank VDC dc voltage WOG Westinghouse Owner's Group YR Steam Flow / Feed Flow / Level Recorder xi

)

1 INTRODUCTION The San Onofre Nuclear Generating Station, Unit 1, operated by the Southern California Edison Company (SCE), is a 450 MWe Westinghouse pressurized water reactor located on the Pacific Ocean, approximately four miles south of San Clemente, California. The plant received an NRC operating license in 1967.

At 4:51 a.m., on November 21, 1985, the plant was operating at 60 percent power, when a ground fault was detected by protective relays associated with a trans-former which was supplying power to one of two safety-related 4160V electrical buses. The resulting isolation of the transformer caused the safety-related bus to de-energize, which tripped all feedwater and condensate pumps on the east side of the plant. The pumps on the west side of the plant were unaffected since their power was supplied from another bus. The continued operation of the west feedwater and condensate pumps, in combination with the failure of the east feedwater pump discharge check valve to close, resulted in over pressurization and rupture of an east side flash evaporator low pressure heater unit. The operators, as required by emergency procedures dealing with electrical systems, tripped the reactor and turbine generator. As a result, the plant experienced its first complete loss of steam generator feedwater and inplant ac electrical power since it began operation.

The subsequent 4-minute loss of inplant electric power started the emergency diesel generators (which by design did not load), de-energized all safety-related pumps and motors, significantly reduced the number of control room instrument indications available for operators to diagnose plant conditions, produced spurious indications of safety injection system actuat.!an, and caused the NRC red phone on the operator's desk to ring. Restoration of inplant electric power was delayed by an unexpected response of an automatic sequence that should have established conditions for delayed remote-manual access to offsite power still available in the switchyard.

The loss of steam generator feedwater was the direct result of the loss of power to the two main feedwater and one auxiliary feedwater pump motors, and the designed 3-minute startup delay of the steam powered auxiliary feedwater pump. The loss of the feedwater pumps, in combination with the failure of four additional feedwater check valves to close, allowed the loss of inventory from all three steam generators and the partial voiding of the long horizontal runs of feedwater piping within the containment building. The subsequent automatic start of feedwater injection by the steam powered auxiliary feedwater pump did not result in the recovery of steam generator level because the backflow of steam and water to the leak in the evaporator carried the auxiliary feedwater with it. Later, operators isolated the feedwater lines from the steam genera-tors, as required by procedure, unknowingly initiating the process of refilling the feedwater lines in the containment building. Before all feedwater lines were refilled, a severe water hammer occurred that bent and cracked one 1-1

. o

feedwater pipe in the containment building, damaged its associated pipe supports and snubbers, broke a feedwater control valve actuator yoke, stretched the studs, lif ted the bonnet, and blew the gasket from a 4-inch feedwater check valve. The damaged check valve developed a significant steam-water leak, the second leak in the event.

The second leak, in combination with an earlier inadvertent re establishment of steam generator blowdown, caused all three steam generator water levels to drop below indicating levels. Steam from all three steam generators fed the leak, because of the absence of individual main steam isolation valves.

Despite these problems, operators later succeeded in recovering level indication-in the two steam generators not directly associated with the feedwater piping leak. With the reestablishment of steam generator levels, the operators safely brought the plant to a stable cold shutdown condition, without a significant release of radioactivity to the environment (the pre-existing primary to second-ary leak was not exacerbated) and without significant additional damage to plant equipment.

On the day following the event, and in conformance with the recently established Incident Investigation Program, the NRC Executive Director for Operations sent an NRC Team of technical experts to the site. (For the directive establishing the Team, see Appendix A.) The original five-member Team, augmented by an addi-tional staff member, was selected because of its broad experience in operating plant event analyses, with individual Team members having specific knowledge and experience in operations, human factors, electrical and reactor systems, and water hammer phenomena. The Team was directed to (1) determine what happened; (2) identify the probable causes; and (3) make appropriate findings and con-clusions to form the basis for possible follow-on actions. This report docu-ments the results of the Team's efforts in identifying the circumstances and causes of the event, together with its findings and conclusions.

I The scope of this fact-finding effort was limited to the circumstances surround-ing the events of November 21, 1985 including operator and NRC actions, equip-ment damage and malfunctions, equipment maintenance and testing history, and regulatory involvement.

Section 2 describes the methodology used by the Team to collect and evaluate information about the event. Section 3 provides a narrative and detailed sequence of events, reconstructed from analysis of operator and NRC interviews and logs, event recorders, and system descriptions. Data to confirm operator observations were sketchy, at best, because the loss of station power inter-rupted operations of crucial recording instruments.

Section 4 provides a summary description of how San Onofre, Unit 1 mechanical and electrical systems involved in this event function and interact. Under-standing the major differences between this plant and more recently designed pressurized water reactors will clarify the basis for operator actions.

Section 5 discusses the performance of plant electrical equipment, the mal-function of which initiated the sequence of events and complicated the operator response to it. Sections 5 through 8 relied heavily on available results from Southern California Edison's analysis and troubleshooting activities of equip-ment involved in the event.

l 1-2 l

1

Section 6 and its associated appendices discuss the results of the Team's review of the development of conditions that resulted in the water hammer, and the resulting damage it caused. This section also examines the maintenance and testing history of the check valves which failed, and the communications between NRC and SCE on (1) the inservice testing program for safety-related check valves, t (2) the design of the auxiliary feedwater system, and (3) the prevention or mitigation of water hammer effects.

I Section 7 discusses the performance of operators and NRC staff during the event and the human factors considerations which affected them.

Section 8 discusses additional noteworthy equipment and system problems that occurred during the event.

Finally, Section 9 presents the Team's principal findings and conclusions, based on information available to the Team at the time the report was compiled.

It should be noted that the root causes of selected equipment failures are still under investigation. However, with the possible exception of the identifica-tion of the root cause of the feedwater system check valve failures, it is unlikely that new significant information relative to what happened and why will be developed.

1-3

z I

2 DESCRIPTION OF TEAM ACTIVITIES 2.1 General Approach In general, the investigative methods used by the San Onofre Incident Investigation Team were based on the experience and methodology developed by the Incident Investigation Team for the Davis-Besse event of June 9,1985. To assure continuity and consistency in Team activities, one member from the Davis-Besse Team served on the San Onofre Team.

The Team tollected and evaluated a variety of information to determine the sequance of operator, plant, and equipment responses during the event and the causes of equipment malfunctions and operator errors. The sequence of responses was difficult to reconstruct because digital data from the Technical Support Center computer was unavailable. Instead, the sequence was developed primarily from interviews with personnel involved in the event. The Team inspec'.ed the equipment which malfunctioned, the equipment and piping damaged by water hammer, and the control room instrumentation and controls. The Team also interviewed plant management and NRC Region V personnel at the site about their knowledge of plant response and operator activities. Personnel at Southern California Edison Corporate headquarters and NRC headquarters were also interviewed to explore the design and regulatory history of the auxiliary feedwater system (as it related to water hammers) and the inservice testing program for Unit 1.

The equipment which malfunctioned or contributed to the water hammer event was quarantined soon after the event so that troubleshooting and examinations could be performed systematically, and so that evidence concerning the root cause of failures would not be lost or destroyed. The root causes, which have yet to be definitively established in only a few cases, are being determined by Southern California Edison (SCE) personnel and equipment vendors using procedures agreed upon by the Team.

2. 2 Interviews and Meetings The event occurred the week before Thanksgiving and about a week before a major planned plant outage, disrupting SCE staff plans for work and vacations. The prompt initiation of the investigation by the Team upon its arrival the weekend before the holiday, and the magnitude of the overall troubleshooting and in-vestigative effort, tended to overwhelm the available onsite technical staf f.

To the SCE staff's credit, they were always cooperative and open with their ideas and information.

The formal investigation began on the morning of November 23 when the Team met with SCE personnel to obtain an overview of their understanding of the event. Following this meeting, the Team began interviews with operating person-nel. The Team placed a high priority on interviewing personnel on duty at the time of the event to learn about the actions they took and the observations they made. The Team recognized that the quicker these interviews could be held, the rore information those being interviewed would remember.

2-1

The Team split into two groups to accelerate the completion of the interviews.

Operator interviews were completed in two days. All interviews and some  ;

meetings were recorded by stenographers who prepared and typed transcripts and '

~

made them available to the Team and interviewees on the following day. Those interviewed had the opportunity to review the transcripts and complete errata sheets when necessary.

Interviews were, in general, scheduled with personnel in decreasing order of their seniority within the shift, beginning with the Shift Superintendent and proceeding to those less senior. The rationale for this sequence was to move from general to specific information. Thus, the Team obtained information on overall plant operations before obtaining information on the detailed actions of specific operators.

Some personnel were interviewed more than once when the Team needed additional clarifying information. Table 2.1 contains a listing of the 51 interviews and meetings the Team conducted.

2.3 Plant Data Because of the nature of this event, the plant data available to evaluate this event was significantly deficient in comparison to information normally available after a reactor trip. No precise digital data were available before or during the event. The only data permanently recorded for post-trip evaluation included:

1. Strip charts from the trend recorders *

2. Oscillograph trace recordings

3. The strip chart from the control room event recorder

4. Logs maintained by operators, emergency coordinators, emergency services, and security personnel.

The Foxboro computer, which is part of the Technical Support Center and normally provides accurate time recordings of plant conditions, was not available because of the troubleshooting activities in progress before the reactor trip. A description of this data collection equipment is found in Section 4.17.3.

SCE provided the Team with numerous color photographs documenting plant layout and the as-found condition of equipment. These photographs, some of which are I

included in this report, contributed significantly to the Team's understanding and evaluation of the event. The Team also examined over 600 documents and l other pieces of evidence during its investigation. All the information used by the Team has been catalogued and will be placed in the local and NRC Public Document Rooms in a special San Onofre file.

l

• The majority of these were inoperable during the 4-minute loss of ac power.

2-2

Table 2.1 Interviews and Meetings Conducted by the San Onofre Team Date Time Meeting / Interview 11/23/85 10:15 Entrance Meeting on Incident Investigation 11/23/85 13:15 Interview of Electrician 11/23/85 14:00 Iiterview of Test Technician 11/24/85 08:55 Interview of Supervisor of Coordination, Acting Unit Superintendent 11/24/85 08:57 Interview of Shift Supervisor, Midnight Shift 11/24/85 10:20 Interview of Control Operator 11/24/85 11:30 Interview of Control Room Supervisor 11/24/85 12:20 Interview of Shift Technical Advisor 11/24/85 13:35 Interview of Nuclear Plant Equipment Operator 11/24/85 14:00 Interview of Nuclear Plant Equipment Operator 11/24/85 14:20 Interview of Assistant Control Operator, Control Room Operator (trainee) 11/24/85 15:10 Interview of Assistant Control Operator 11/24/85 15:50 Interview of Nuclear Plant Equipment Operator 11/25/85 09:25 Interview of NRC Senior Resident Inspector 11/25/85 09:30 Interview of Shift Superintendent, Day Shift 11/25/85 10:15 Interview of Control Room Supervisor, Day Shift 11/25/85 10:25 Interview of Control Room Operator, Day Shift 11/25/85 14:15 Interview of Supervisor, Fire Protection Services 11/25/85 15:00 . Manager of Safety Emergency Preparedness 11/25/85 15:40 Interview of NRC Resident Inspector

• Transcripts were made of all meetings and interviews listed.

2-3

i Table 2.1 Interviews and Meetings Conducted by the San Onofre Team (continued)

(

11/25/85 16

55 Interview of NRC Resident Inspector 2 11/25/85 18:10 Interview of Fire Captain, San Onofre Fire

]L Department, Shift Captain

! 11/26/85 10:30 Interview of NRC Resident Inspector 11/26/85 13:00 Water Hammer Conference l 11/26/85 16:00 Interview of Emergency Medical Technician 2

11/26/85 16:30 Interview of Fire Watch Rover j 11/26/85 18:10 Interview of Assistant Control Operator, Control Room Operator (trainee) 11/27/85 08:15 Interview of Fire Watch Rover

11/27/85 10

05 Interview of Station Manager i 11/27/85 11:18 Meeting of SONGS Security, Station Security Manager, Computer Supervisor, Computer Engineer (Safeguards Information) 11/27/85 14:00 Interview of Chemistry Supervisor I

12/11/85 09:00 Interview of Manager of Station Operations '

12/11/85 10:00 Intervicw of Mechanical General Foreman j 12/11/85 11:07 Interview of Shift Superintendent

]

j 12/12/85 08:10 Interview of Plant Superintendent, Unit 1 12/12/85 09:55 Interview of Shift Superintendent

12/12/85 11

00 Interview of Health Physics Technicians 12/12/85 11:30 Interview of Health Physics Technician 12/12/85 13:20 Interview of Nuclear Plant Equipment Operator 1 12/12/85 16:40 Interview of Health Physics Supervisor 12/12/85 17:25 Interview of Health Physics Technician j 12/13/85 07:45 Interview of Control Operator i

e 1 2-4 1

l a

i

~

I 1

Table 2.1 Interviews and Meetings Conducted by the l San Onofre Team (continued)  !

12/13/85 08:40 Interview of Nuclear Operations Assistant 12/13/85 13:00 Transcript of Proceedings in Rosemead, California 12/18/85 10:10 Interview of NRC Task Manager 12/18/85 11:20 Interview of Former NRC Auxiliary Systems Branch Engineer 12/18/85 13:00 Interview of NRC Duty Officer 12/18/85 15:05 Interview of NRC Systematic Evaluation Program Manager 12/20/85 10:10 Interview of NRC Engineer 12/20/85 11:46 Interview of Senior Reactor Events Analyst 12/26/85 13:25 Interview of Assistant Director for Operating Reactors 12/26/85 15:00 Interview of Assistert ;irector for Safety Assessment 12/27/85 09:00 Interview of NRC Engineer, Engineering Issues Branch 12/27/85 10:28 Interview of Acting Branch Chief, Mechanical Engineering Branch 12/27/85 15:25 Interview of Deputy Director, Division of Human Factors Technology 12/27/85 15:48 Interview of Assistant Director for Generic Issues 2-5

3 NARRATIVE OF THE EVENT This section provides a step-by-step description of events in the early morning hours of November 21, 1985, at San Onofre, Unit 1, that led to a loss of inplant power and a severe incidence of water hammer. The detailed record of events normally provided by the plant computer was not available because the computer had not been reset following earlier electrical ground troubleshooting activi-ties. The NRC Team based this narrative and the sequence of events listed in Table 3.1 on a composite of operator and management interviews, operator logs, readings from the plant's oscillograph trace recorder, and interpretations of strip charts from plant trend recorders.

The interviews indicate that various plant personnel heard a series of unex-plained loud " bangs" and muffled " booms" during the event. In addition, secur-ity personnel saw flashes of light above a transformer in the site's restricted area. To the extent possible, the sights and sounds heard by the plant staff have been correlated with the response of plant parameters recorded by the instruments previously mentioned.

3.1 Plant Status On November 20,1985 at 11:30 p.m. ,* the midnight shif t of operators assumed control of the San Onofre Nuclear Generating Station Unit 1. During the day j shift, the generator load had been decreased from 407 MWe to 250 MWe (licensed power is 450 MWe) because of a tube leak in one part of the main condenser.

Saltwater from the Pacific Ocean is used to cool the condenser and a small amount of it was leaking into the condenser. One of the circulating water pumps was stopped so that the leak could be fixed. With only one-half of the condenser available to condense steam, the generator load (and steam flow rate) had been reduced accordingly.

The saltwater leak into the condenser contaminated the feedwater with corrosive chlorides, which would adversely affect the steam generator tubes. To reduce the concentration of chlorides in the steam generator, the rate of secondary blowdown had been increased from about 40 to about 100 gallons per minute for each steam generator. (Blowdown is the water being removed from the steam generators.) The blowdown flow rate was unusually high, and will adversely affect steam generator level recovery later because it was equal to about two-thirds of the total flow to each steam generator that could be provided by the auxiliary feedwater system.

Except for saltwater leakage into the condenser, no other conditions were affect-ing plant operations. However, both pressurizer power-operated relief valves (PORVs) were isolated (a condition permitted by the plant's Technical Speciff-cations) because one valve was leaking and the block valve for the other PORV had failed a surveillance test. No other ongoing tests or changes to the plant status were planned. The oncoming shift of five control room operators (2 SR0s

• All times represent Pacific Standard Time unless otherwise specified.

3-1

and 3 R0s) and two plant equipment operators had barely completed shift turnover procedures when, at 10 minutes before midnight, an alarm sounded in the control room indicating that a 100 percent ground was detected by the ground detector on 4160-volt bus IC (An electrical ground is an undesirable path for current from the power source to some other conducting body.)

3.2 Electrical Ground Fault Bus 1C is a 4160-Volt safety related bus through which electrical power flows to safety-related equipment (e.g. , the feedwater/ safety injection pump, the charging pump, the auxiliary feedwater pump) and other equipment during normal plant operation. Power is normally supplied to bus 1C from offsite, through the station switchyard and, in an emergency, can be supplied from one of two backup diesel generators. The 4160-Volt bus 2C is the backup for bus 1C in that it powers the redundant train, i.e., other side, of safety related equip-ment. Even though an electrical ground was present on bus 1C, such a condition does not interrupt power to the equipment and thus the operation of plant equip-ment was routine.

The alarm did not identify the precise location of the ground. It could have existed anywhere from the transformer secondary windings to the motor of a safety-related pump or other loads connected to bus IC. The development of a second ground, however, could result in an electrical fault and in protective relay actuation and isolation of the faulted circuit (e.g., loss of power to the bus); or if the ground was in a motor, it could damage it. Consequently, after midnight, two electrical technicians were called to the site to assist in troubleshooting the ground.

After the alarm annunciated indicating the ground on bus IC, the reactor oper-ators referred to a written ground fault procedure for locating and isolating the ground.

For several hours, the operators swapped, started and stopped equip-ment according to the procedure in an attempt to locate the ground. The Super-visor of Coordination, a member of Unit 1 management (the Acting Plant Superin-tendent), who is a licensed SRO, was called by the Shift Superintendent and came to the site at about 3:00 a.m. to provide assistance. The-Shift Superintendent thought the situation would require de-energizing bus IC, a decision that would af fect power operation and the Technical Specifications which govern plant oper-ations. (The Technical Specifications require that the plant be shut down if bus 1C is de-energized for a period of 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> or more.)

After the Supervisor of Cnordination arrived, the actions taken and the remain-ing options available to isolate the ground were discussed. Operators had com-plied with the ground fault procedure to the extent of having tested all equip-ment connected to bus IC, except for the west main feedwater pump. In order to test the pump, which required that it be removed from service, the Unit 1 elec-trical load had to be reduced to ensure continued operation without a reactor trip.

Accordingly, preparations were made to stop the feedwater pump by reduc-ing the unit load from 250 to 150 MWe and realigning support systems.

In the meantime, the two electrical test technicians had arrived onsite. After verifying that the ground was not in any load connected to the bus (except for the west feedwater pumn), the operators followed the ground fault procedure, until the step that required that bus 1C be de-energized, flowever, at this 3-2

point the Shift Superintendent, with suggestions from the electrical test tech-nicians and with his management's concurrence, improvised a troubleshooting l method not included in the procedure. The proposed method of troubleshooting I was seen as having an advantage in that bus 1C need not be de-energized.

Instead of de energizing bus IC, the improvised troubleshooting method involved the operators connecting it in parallel with bus IA, and disconnecting its nor-mal power supply (transformer C). This paralleling operation was performed three times. When the power supply from transformer C to bus IC was disconnected, the ground alarm cleared, which indicated to the operators that the ground was on the secondary side of transformer C feeding bus IC. This finding occurred at about 3:30 a.m., and eliminated the need to remove the west main feedwater pump from the bus since the fault was known to be on the transformer side of the bus. Subsequently at 4:30 a.m., with bus 1C aligned to receive power from bus 1A rather than transformer C, the unit load was increased from 150 MWe to 250 MWe. Transformer C with the ground fault still present was left energized and supplying bus 2C. This unusual switchgear alignment is not covered in plant procedures.

Bus 1C had not been de-energized because the operators knew that this would invoke a Technical Specifications Action Statement, a condition that they believed should always be avoided. This situation was discussed several times with the Supervisor of Coordination. Plant Technical Specifications require {

that buses IC and 2C always be energized (except during refueling); if bus 1C is de-energized, the Action Statement requires that it be returned to operable status within 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> or that the plant be shutdown.

The two electrical technicians were dispatched to visually inspect transformer C and its switchgear in an attempt to locate the ground. It was while they were inspecting the switchgear that they heard a loud " boom". At the same time a security guard caw a flash of light from the vicinity of the top of the trans-former near the technicians. It was 4:51 a.m.

3.3 Loss of Power In the control room, operators also heard the " boom" while the background noise from rotating machinery wound down amidst the sounds of numerous alarms and the illumination from rows of annunciator lights. Immediately, all five operators surveyed the alarm panels and control boards to diagnose the situation. The Shif t Superintendent noticed that transformer C had been isolated by differen-tial relay protection, which resulted in a loss of power to bus 2C. From among the multiple, nonassociated alarms displayed on the panels, he identified the transformer C differential relay alarm on the turbine first-out panel and quickly determined that vital bus 4 had been de-energized. He directed operators to trip (shut down) the reactor pursuant to the immediate actions specified in the procedure for loss of this vital bus. Two operators pushed each of the two manual reactor trip buttons (only one of which is required to trip the reactor) and another operator then pushed the unit trip button. Within 20 seconds the operators had identified the initiator of the event, and had manually tripped the reactor and turbine generator in accordance with the plant procedure fnr loss of the vital bus. At that moment, only emergency lights illuminated the control room as normal station lighting went black when inplant ac power was lost.

3-3

Immediately following the reactor trip, the red telephone, part of the emergency notification system connected directly to NRC headquarters in Bethesda, Maryland, rang in the control room. Although pre occupied with examining the control boards and alarm panels, and while watching the reactor operators respond to the reactor trip, the Shift Superintendent finally answered the telephone. The l voice on the telephone was that of the NRC Duty Officer in Bethesda (evidently the phone had rang at both ends). The Shift Superintendent was puzzled as to why NRC wanted information about or could possibly have known that a reactor i trip had occurred less than one-half minute before. The exchange of questions and answers that ensued between the Unit 1 control room and the NRC Duty Officer was a mixture of miscues and incomplete communication. Finally, the Shift Superintendent implied to the Duty Officer that he was too busy to talk, told him that he would call him back, and told him the unit was stable. The NRC Duty i

Officer did not ask the control room operators about which emergency class was declared.

Evidently, when bus 1C lost power, the emergency notification system responded .

to the power interruption by signaling the red telephone in the control room and at the NRC Operations Center to ring. Each party thought that the other party had called, and as a consequence, meaningful dialogue was not established.

During a second telephone call, the Shift Superintendent indicated that an Alert '

would probably be declared, but subsequent evaluation led to the declaration of an Unusual Event pursuant to the site's emergency plan. However, the event was not reported to the NRC as an Unusual Event, although the control room operators and site personnel knew that this emergency action level had been declared. As a result of the mixed communications, the NRC Duty Officer believed that an Alert had been declared. Subsequently, the NRC requested that an open, continu-ous telephone line be maintained to the control room, an arrangement which is normally reserved for an Alert declaration. The Shift Superintendent did not understand the need for such a request, particularly since it distracted him from his responsibilities in the control room. He then requested the Super-visor of Coordination to communicate with the NRC. Later, when the NRC Resident Inspector arrived in the control room at about 6:00 a.m., he was requested to handle the open telephone line with the NRC.

If the plant had been in a normal switchgear alignment, isolating transformer C would have de energized buses 1C and 2C. But because of the abnormal alignment, the isolation of transformer C de-energized only bus 2C, because bus 1C was powered by the main turbine generator via bus 1A. Thus, all inplant power was not instantaneously interrupted. About one-half of the equipment lost power when bus 2C (i.e., transformer C) was lost, and 20 seconds later the remaining equipment lost power when the unit was manually tripped. As a result, the west feedwater pump (on bus IC) continued to operate af ter the east feedwater pump (on bus 2C) lost power.

At about 4:St. am, in the 4kV switchgear room located below the turbine building mezzanine level, a roving fire watch had come to relieve the fire watch sta-l tioned in the rocm. Shortly thereafter, they heard a cannon-like sound and I

felt the floor vibrate. One of the fire watches described the noise as a "muf-fled howitzer." The test technicians outdoors described it as a loud " boom."

During the 20-second interval that the west main feedwater pump continued to l operate, the single 12-inch diameter check valve at the discharge of the east l 3-4

feedwater pump stuck open. As a result, the high pressure (1300 psig) discharge water from the west pump flowed backwards through the east feedwater pump and overpressurized the east condensate piping and components (design pressure is only 350 psig).

The low pressure flash evaporator condenser had been overpressurized by the backflow of feedwater from the west feedwater pump. The feedwater at hign tem-perature and pressure ruptured a tube in the flash ev,aporator condenser and  !

ballooned the rectangular shell of the flash evaporator until a 20-foot long 2-foot-wide fishmouth split occurred along the welded seam and relieved the pressure. Water and steam covered the east part of the turbine building and activated fire alarms several levels below the turbine deck.

Post event investigation revealed that a power supply cable from transformer C to bus 1C had failed. The cable had shorted and caused two large openings in the metal jacket covering the wires. When the cable failed, a loud noise, per-haps an electrical arc, occurred. Thus, the noises heard by the flre watches and test technicians may have been the rupture of the flash evaporator located on the turbine building deck above the 4kV room, or the failure of a cable in a cable tray located in the turbine building, or some combination of the two.

As the overhead lights went out, two alarm lights appeared on the fire protec-tion annunicator panel located in the 4kV room and started blinking, an indica-tion that there was a fire. One of the alarms was for the hydrogen seal area and the other was for the lube oil storage area. One fire watch carefully went out into the darkness with his flashlight to investigate the noise and alarms.

He instructed the other fire watch to report the fire alarms to the control room by telephone.

Several attempts by the fire watch in the 4kV room to call the control room failed because both numbers on his call sheet were busy. Finally, he called the site fire department, assuming that there was a fire. The fire-watch officer at the site fire department asked him to go outside to see if there was a fire.

When he opened the door he saw what he thought was smoke and the silhouettes of operators running around. He reported his observations to the officer, who then instructed him to go back and ask one of the operators where the fire was located. When he reported back to the officer that the operators were too busy to talk to him, the officer indicateo he would dispatch the fire brigade.

The fire apparatus, a Mack 50-foot telesquirt, triple-combination pumper, arrived at the turbine building within 3 minutes. Instead of a fire and smoke, they found only steam. They also discovered that the foam deluge system had activated around the lube oil storage tank due to steam escaping from the traps on the feedwater pump shaft seal. They remained on the scene to support the operators in their recovery activities.

When the operators pushed the unit trip button, all inplant ac power was lost.

Consequently, the west feedwater pump stopped and may have reversed its rota-tional direction because its discharge check valve also stuck open and did not prevent backleakage from the steam generators. Reverse rotation could have damaged the pump. (Note - The feedwater pumps are also the safety injection 3-5

s pumps.) The trend recorder data show that' the potential existed for both trains of the condensate system to be overpressurized by the backflow from the steam generaturr. (Post-event examinations revealed that all five check valves be-tween the pumps and the steam generators failed to perform their safety function of preventing backleakage.) For this event, the failed tube in the flash evap-orator condenser probably relieved the pressure surges and prevented further overpressurization.

3.4 Four-Minute Blackout To an observer on thg cliff above the beach overlooking the site at 4:51 a.m. ,

Unit 1 would have appeared mysteriously dark and the site abnormally silent. ,

(Units 2 and 3 were both shutdown for refueling.) When bus 2C lost power from -

transformer C, emergency diesel generator 2 started and achieved rated speed s Emergency diesel generator 1 started when bus 1C de energized after the main generator was tripped. According to the design for the onsite emergency power supply, the emergency diesels do not automatically provide power to emergency equipment after a loss of offsite power or de-energization of buses IC and 2C.

Thus, until operators manually load the diesel generators or restore offsite power to the buses, San Onofre Unit 1 is designed to be without any inplant ac power. Only the plant dc system and the inverter-supplied (battery powered) vital buses remain operational. Without any ac power, the normal lighting in the control room and throughout the plant is lost.

', After the control room operators manually tripped the plant, they immediately cocpleted the necessary actions from memory. The Control Room Supervisor then 1, read aloud the appropriate steps from the Emergency Operating Instructions (E0I) for a reactor trip or safety injection. The third step of the procedure requi} es operators to verify that buses IC and 2C are energized. The operators l found'that they were de energized. The E01 being read from directed the opera-l tors to the E01 for loss of all ac power. The second step in the latter proced-ure requires the operator to determine if there is an indication of a safety j injection (SI) actuation on the annunciator window located on the first-out '

alarm panel. The alarm window and other instrumentation indicated that there was an SI actuation. However, the SI alarm was spurious. Because the operators took appropriate steps to verify that SI had not occurred, the spurious nature of the SI alarm was quickly determined. .

The next step in the E01 was to verify that the automatic sequencer had operated upon loss of voltage. The sequencer automatically realigns circuit breakers such that only an operator action to close a single breaker is necessary to '

restore inplant ac power via the main station transformer and offsite power.

However, the light that would indicate that the sequence was completed and that the final breaker should be closed to restore power did not illuminate. The operators waited for more than 2 minutes for the sequence to be completed and then assumed the sequencer had failed. They then took manual action to actuate the necessary circuit breakers.

At about 4:54 a.m., the operators first attempted to complete the sequence to restore offsi g power to buses 1C and 2C. All control room personnel watched the operator on the electrical board fail four times to close the electrical breakers necewary.,to restore power to the station. Finally, on the fifth attempt, and 4 minutes af ter power was lost, the buses were energized and the station blackout was over. !g i

6 s , , f k

3-6

The blackout could have been longer if operator attempts to restore offsite power had not been successful. The E0I for loss' of ac power lacks criteria for how long the plant can be without power before the diesel generators are to be loaded. In this case, the operators as required by procedures restored power from the switchyard rather than connecting the buses to the emergency diesel generators. (Had priority been given in the E0I to restoring power using the emergency diesel generators, power could have been. restored in less than a min-ute.) In this 4-minute period, the feedwater lines started to empty, and steam pockets were formed in the feedwater piping, because of the five faulty check-valves.

The loss of the vital bus caused by the loss of transformer C caused multiple alarms on the annunicator systems and loss of indication and trend recorders in the control room. Remembering the large number of annunciator lights on the panels, an operator indicated in an interview that he was shocked to see that the reactor had not tripped (with normal power- distribution, the loss of trans-former C results in a reactor trip automatically). With few exceptions, all 3 the trend recorders failed and the recorded data for critical system parameters were lost during the blackout. Although these instruments are normally used by the operators to control the plant, adequate information from other instruments )

(

was available to the operators to understand the event and maintain plant safety. {

However, trends of critical parameters during the blackout phase were not avail-able to guide operator actions. Fortunately, the blackout lasted only 4 minutes and no additional complications occurred.

The clock in the control room stopped for the 4 minutes while the power was lost. Subsequently, this clock was used to identify times when log entries for subsequent operator actions were made. Accordingly, the times entered in the logs and given by the operators in interviews had to be adjusted.

After the reactor trip, the water levels in the three steam generators dropped below the setpoint for actuating the auxiliary feedwater system. The turbine-driven auxiliary feedwater pump received a start signal within seconds after the reactor trip. Although an actuation signal existed, the motor-driven pump does not receive a start signal until power is available. Because safety bus 1C was de energized, the motor-driven pump could not start.

The turbine-driven auxiliary feedwater pump started and began .its warmup cycle.

Unlike turbine-driven auxiliary feedwater pumps at most other plants, this pump is designed to increase speed gradually for about 3 minutes before it delivers water to the steam generators. (This warmup period minimizes the potential for overspeed trips inherent in turbine-driven pumps when accelerated rapidly to full speed.) However, until the warmup cycle was completed, there was a total loss of feedwater to the steam generators.

After power was restored, the E01 for loss of ac power returned the operators to the E01 for reactor trip or safety injection. Following the E0I for reactor trip response, the operators closed the atmospheric steam dumps, thereby stop-ping a brief steam relief from the steam generators. At this time flow from the auxiliary feedwater system was verified to be about 135-150 gpm to each of the steam generators. The E01 then required that the main feedwater flow path be isolated. This action resulted in the closure of the regulating valve and the motor-operated isolation valve in each feedwater line. This action termi-nated the flow of auxiliary feedwater backwards to the ruptured flash evaporator 3-7

s and redirected it to the steam generators. Thus, thez heat sink provided by the steam generators / emergency feedwater was degraded only temporarily.

Until the motor-operated valve in the feedwater piping to each steam generator was closed, the water in these pipes was draining backwards through the conden-ate system, to the leak. Draining, or empty,ing these pipes is supposed to be prevented by the five check valves in the feedwater system. Their specific purpose, by design and safety function, is to keep the pipes filled with water and to prevent backleakage to the condensate system, especially the auxiliary feedwater flow which is introduced into the feedwater piping downstream of the check valves. Thus, without such isolation, the auxiliary feedwater was flowing back through the condensate system to the ruptured flash evaporator.

3.5 Conflicting Requirements A loud " boom" awoke the Shift Technical Advisor (STA) just after he had gone to sleep at about 4:30 a.m. His sleeping quarters are located in a mobile home between Units 1 and 2/3, within the site's protected area. Earlier that night he had been assisting in troubleshooting the electrical ground, primarily by interpreting and evaluating the Technical Specification requirements for the ac power supply. After the noise awoke him, he tried to contact the control room, first on his two-way radio and then using the plant telephone system. Unable to contact anyone, he got dressed and went to the control room, arriving there about 5:02 a.m. according to the control room clock. On his way he noticed steam coming from the flash evaporator. At that point, he did not know that ac power had been lost. He assumed his normal duties after a reactor trip by exe-cuting the E01 containing the Critical Safety Function Status Trees. These are part of the emergency operating guidelines developed by the Westinghouse Owners Group and are used in monitoring those safety functions related to the mainte-nance of the various barriers that prevent the release of radioactive material to the environment.

The Supervisor of Coordination, who had entered the control room during the period power was lost, asked the Shif t Superintendent if it.would be okay if he reset the radiation monitors which were still alarming. (Note-on a loss of power, the radiation monitors fail in a mode which isolates the containment building, including the steam generator blowdown.) Receiving permission, he reset the radiation alarms. As a result, without operator awareness, steam generator blowdown was automatically re-established at about 100 gpm per steam generator. Within a few minutes, as he monitored the status trees, the STA in-formed the Shift Superintendent that the steam generator levels were low, but not low enough to be in the alert range.

Meanwhile, in response to decreasing pressurizer water level, a reactor opera-tor started a charging pump. The second charging pump then started automatic-ally in response to the low charging header pressure. The suction of both pumps.

then automatically switched to the refueling water storage tank when a low level setpoint was reached in the volume control tank. The suction to the pumps switched several times between the volume control tank and the refueling water storage tank until it was manually switched by'the operators to the refueling water storage tank late in the event to borate for cold shutdown. The water level in the pressurizer was decreasing toward off-scale low, an_d the pressurizer pressure was decreasing toward the low pressure setpoint (1735 psig) for safety injection actuation.

3-8

f L

Beginning at about 5:00 a.m. , the operators had two conflicting interests or concerns to deal with prior to getting control of the reactor coolant pressure.

First, on the primary coolant side, both the pressurizer level and reactor cool-ant temperature were low and decreasing. Second, on the secondary system, all three steam generator water levels were low and decreasing. The primary param-eters indicated an abnormally high cooldown during natural circulation conditions in the reactor coolant system--something like a steam line break--but without a

! steam break. The operators were concerned and discussed the situation with the Supervisor of Coordination. The operators did not want the steam generators to go dry, but at the same tir.1e, the operators wanted to minimize the heat trans-fer in the steam generators in order to recover primary system level and pressure.

The operators recognized that the reactor coolant system was being overcooled.

In order to prevent the system from reaching the setpoint for safety injection

{

actuation, a reactor operator directed another operator controlling the auxiliary feedwater system to stop the flow. Accordingly, the throttle valves were closed terminating auxiliary feedwater flow. The steam generator water levels then decreased at a faster rate. Noting this condition, the Shift Superintendent, who was maintaining an overview of control room activities, directed the oper-ator to restore auxiliary feedwater flow. Based on the recollection of the operator, auxiliary feedwater was restored to an indicated level of about 25 gpm to each steam generator in about 10 seconds.

Also during this period, a nuclear plant equipment operator was dispatched to the turbine building to manually close the two 24-inch steamline block valves to minimize the cooldown. Closing these valves is normally done after a reactor trip because of secondary-side steam leaks. Unlike other plants with remotely operated main steamline isolation valves, these valves cannot be remotely oper-ated from the control room. They must be clos,? by equipment operators using a large hand-held, air-operated wrench.

3.6 The Water Hammer At about 5:07 a.m. , the equipment operator had just started to close one of the l

I steamline block valves located on the turbine building mezzanine level when he-heard a bang, felt a concussion wave, and was engulfed by a cloud of steam. He ran from the area, but not before the steam had soaked his clothing. When he reached the control room he reported that the main steam line had broken. The control room operators had also heard the bang. However, what the control room operators had actually heard, and what the equipment operator had witnessed, was a failed feedwater check valve caused by a thermal-hydraulic phenomenon known as water hammer.

As the auxiliary feedwater pumps refilled the feedwater piping to the steam generators, conditions were being established for a phenomenon that can generate destructive forces greater than 150,000 pounds- fo rce. Since the feedwater pip-ing to the steam generators had drained because of the failed check valves, the pipes contained water and steam at high temperature and pressure from the steam generators.

As the auxiliary feedwater system filled the piping with relatively cold water, an instability occurred at the steam / water interface, which created a slug of water in the steam space. The slug accelerated at great speed, as steam was condensed in front of the slug, until it encountered an obstruction 3-9

I or'a change of direction in the piping, such as at an elbow or closed valve.

Upon contact, the slug imparted its energy to the pipingBecause with the of force of a the long hammer blow, i.e., a condensation-induced water hammer.

(203 feet) horizontal layout of the feedwater piping to the B steam generator i and other sustaining conditions, this piping experienced the water hammer. The forces from the water hammer displaced the 10-inch diameter feedwater piping, distorted its original configuration, caused an 80-inch' crack, and damaged pipe hangers and snubbers. In seconds, the one-half inch thick piping was irrevers-ibly damaged--the 80-inch crack, 30 percent through the wall at places, indicates how close the pipe had come to splitting open.

Outside the containment building, the forces associated with tne water hammer were forceful enough to stretch 10 one-half-inch diameter bolts holding the bonnet on a 4-inch bypass check valve by about one-half inch. All of the bolts were stretched into an hour glass shape. The steam and water from the check -

valve body to bonnet interface had sufficient force to blow away the insulation '

from all the piping located 360 degrees around the check valve. The steam es-caping through the gap between the bonnet and valve body was felt 25 feet away by the nuclear plant equipment operator who was closing the steamline block valve.

The design of the steam system at Unit I has the three steamlines joined into a common pipe (or steam header) inside the containment building without any valves to prevent. simultaneous blowdown of all three steam generators should a leak in a steamline or a feedwater line occur. Hence, the leak from the B feedwater bypass check valve located outside the containment building communicated with all three steam generators, via the steam header and B feedring, and their steam inventories were vented via the leak to the atmosphere. In adMition, the auxil-f ary feedwater flow to B steam generator escaped from this let.'t instead of going to the steam generator.

3.7 Steam Generators Boil Dry The effects of the water hammer and the failed bypass check valve were not indi-cated in the control room. Based on the report by the nuclear plant equipment operator, the control room operators thought that there was a steamline break.

However, they continued to follow the E0I for reactor trip response in a system-atic manner because their instrumentation did not reflect a major steamline break. With both charging pumps operating at full flow, the reactor coolant pressure and pressurizer level recovered, and control of primary pressure was regained.

At this point, the E0I required that the B reactor coolant pump be started for pressurizer spray control. Shortly af ter starting the pump, a thrust bearing high temperature alarm sounded. Although the operators believed it was a false indication, discussions with the Supervisor of Coordination led to a decision 20 minutes later to start the A and C pumps and shutdown the 8 pump.

When the two reactor coolant pumps were started, the steam-generator levels were about equal, but low on the wide range level indicators. The operators recalled that the level went off-scale low in all three steam generators shortly after the A and C' pumps were started. The Shift Superintendent noticed a sharp decrease in steam pressure, and recalling earlier the report by the Theplant equip-auxiliary ment operator, declared he thought-that-there was-a steam leak.

3-10

feedwater flow rate was maintained at about 25 gpm (indicated) to each steam generator in order to assure the heat sink was maintained, although there was no indicated level in any steam generator. All three steam generators were

' essentially dry; however, the conditions for a " dry" steam generator in the E0I were not satisfied. The Control Room Supervisor reviewed the E0I for loss of secondary coolant but, based on secondary system conditions, the criteria were not met for beginning the procedure (e.g., steam pressure was above 400 psig).

l The E0I for responding to a steam generator low level has three conditions that i

must exist simultaneously for a steam generator to be considered dry. If either the wide range water level indicates greater than zero, or the reactor coolant loop temperature difference is greater than zero, or if auxiliary feedwater flow to the steam generator is 25 gpm or more, the steam generator is considered to be effective in removing decay heat and is not considered dry. The operators did not recall any time during the event that at least one of the three condi-tions did not exist. However, none of these parameters are recorded on a strip chart.

The STA continued to cycle through the E0I for the Critical Safety Function Status Trees. He also maintained a vigilant watch on the steam generator levels. Because the narrow range level indication was below 10 percent, the E0I referred him to the EDI for responding to steam generator low level. As he went through the E01, step 3 required that the steam generator blowdown be isolated--

a significant discovery.

Prior to this time, about 100 gpm had been draining from each steam generator as blowdown. This was the dominant contributor to the loss of water inventory and level in the steam generators. The STA informed the Control Room Supervisor-of this important discovery, and the blowdown was secured at about 35 minutes after it had automatically been re-established when the radiation monitor alarm had been reset. The continuous blowdown from the steam generators had escaped recognition by the operators in analyzing the reasons for the cooldown and steam generator low levels. (The status of the blowdown system is not indicated in the control room.)

By this time, it was obvious to the Shift Superintendent that the plant had to be placed in cold shutdown in order to isolate and correct the leak. At this point, the control room operators did not know that a water hammer had occurred and that it had caused the steam leak outside containment and severe damage to.

feedwater piping and hangers inside the containment building.

3.8 Plant Cooldown Another discussion took place in the control room between the operators and the Supervisor of Coordination concerning how best to cool the reactor to cold shut- i down. Although the procedures called for a normal 50 percent level in the steam generators, the operators concluded that restoring such a high level would ex-acerbate the cooldown transient, and could result in exceeding the Technical' Specification cooldown rate of 100-degrees Fahrenheit per hour. All steps in the E0I for a reactor trip response had been completed and the final step re-ferred the operators to the standard operating procedure for changing operating modes from power to hot standby. Reactor coolant conditions indicated that they had passed through this procedure already, and that it would be appropriate to go to cold shutdown from hot standby.

3-11

1 The operators decided to maximize the cooldown rate in order to depressurize quickly and isolate the leak while maintaining sufficient margin to the Techni-4 cal Specification limit. Consequently, the auxiliary feedwater flow rate was increased to the A and C steam generators from about 25 to 40 gpm indicated.

The flow rate to 8 was not increased because the B reactor coolant pump had been secured, minimizing the primary-to-secondary heat transfer and the need for additional feedwater. The levels in the A and C steam generators increased slowly, but level in the B steam generator remained off-scale low. It was then apparent to the opera *. ors that the leak was associated with the B steam gener-j -ator. At this point, they established a cooldown rate of about 60-degrees  !

, Fahrenheit per hour an1 started the preparations that had to be completed prior J to establishing decay neat removal using the residual heat removal (RHR) system.

Based on the STA's rev ew of the Critical Function Status Trees', he noticed that l

containment building pressure was slightly positive when it should have been negative. lie so infort,ed the Shif t Superintendent, who then determined that the i line which normally vents air from the containment building had been isolated by the radiation monitor when power was lost. Thus, the leakage from air-operated valves inside the building had caused the pressure increase.

j Earlier during the event, the saltwater cooling system had been aligned to the 4

heat exchangers in the turbine plant cooling system when containment building

) cooling was re-established. The RHR system requires the full capacity of.the j saltwater cooling system for a rapid cooldown. Thus, the saltwater flow to the turbine plant cooling system had to be terminated and redirected to the RHR 5,

system. An alternate arrangement involving the intake screen wash pumps was i then used to supply cooling water to the turbine plant cooling water heat exchangers.

The normal cooling water to the turbine plant cooling water heat exchangers is provided by the circulating water system. However,. numerous attempts to estab-lish the conditions for starting the circulating water. pumps failed due to abnormal conditions in the main condenser. The temperature in the condenser was about 200 degrees Fahrenheit (normal is about 100*F). When a vacuum was applied to the tube side of the condenser to ensure that the water box was filled, the circulating water would flash to steam, and the vacuum interlock for starting the circulating water pumps could not be satisfied. Evidently, j the steam traps and other secondary sources of hot water and steam overheated l

the condenser in the absence of circulating cooling water. At the time, the l operators did not understand why the tubes could not be' filled with water.

One of the last preparations made prior to opening the isolation valves to the RHR system was a containment building entry to close the valve in the bypass line around a RHR pump. The bypass line provides an alternate hot leg injection path following a loss of coolant accident. Closing the valve is a normal oper - 4 4

ation and ensures the maximum flow rate through the RHR heat exchangers, i.e., !

faster cooldown. l 1

At about 9:2^ a.m., all preparations had been made and-the operators unsuccess- l l

fully attem;.;ed to open the suction and discharge valves that isolate the RHR system from the reactor coolant system. The valves are interlocked'such_that ,

l the reactor coolant pressure must be well below the design pressure of the RHR system before the valves will open. The operators investigated and reviewed 3-12

the as-built drawings and confirmed that the pressure setpoint was 400 psig.

The reactor coolant system pressure was about 370 psig at the time. The oper-ators assumed the interlock had failed and overrode the relay in the interlock logic and opened the valves. Later it was found that a procedural deficiency l and related training misled the operators to believe that the interlock had l failed.

At 9:41 a.m. , about 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> after the event started, RHR cooling was established and the Emergency Coordinator terminated the Unusual Event. The major task re-maining was isolating the leak from the failed check valve.

3.9 Isolating the Leak At about 6:00 a.m. , the day shift operating crew began arriving at the control room. Station management, engineers, NRC Resident Inspectors, and other person-nel also came to the control roon.. The shift turnover that normally begins at 7:00 a.m. (and is usually completed by 7:30 a.m.) did not occur until 10:00 a.m. because of the event. The day shift and most of the other personnel re-mained outside the control room in the Technical Support Center and supported the recovery operations, including the isolation of the leak.

Attempts to identify the location of the leak began about 5:30 a.m. An Assist-ant Control Operator dressed in a steam suit (i.e., protective clothing for environmental protection against fires, steam, hazardous materials, etc.) in-spected the mezzanine area. Two members of the fire brigade accompanied him into the area, which was standard procedure for rescue operations in a hostile environment. Radiological surveys had previously been completed by two health physicists.

The first entry in the steam environment lasted only about 2 minutes because of the heat. The second entry was made with additional protective clothing. The operator was then able to determine that the leak was near the feedwater bypass regulating valve. But his visibility and mobility were so restricted that he did not attempt to isolate the leak. He returned to the control room at about 8:00 a.m. , and informed the Shift Superintendent of his findings.

Based on the status of the plant, the operators concluded that there was not an urgent need to isolate the leak. The conditions in the reactor coolant system were stable, and water levels had been reestablished in steam generators A and C.

The leak was effectively removing decay heat, with a resultant steady cooldown rate.

At 10:45 a.m., the manual valve in the B steam generator main feedwater line and a manual valve in the bypass piping were closed, which isolated the leak.

The auxiliary feedwater system refilled 8 steam generator. The unit entered mode 5 for a refueling outage at about 3:00 p.m.- a week sooner than planned.

After the shift change, operators that had been on duty were debriefed by site management to ascertain the sequence of events. During the debriefing, it be-came apparent to the operators and management that a water hammer had occurred, and that an inspection of the systems inside the containment building was required.  ;

It was during this containment entry, early the next day that operators found evidence of damage to the feedwater piping to the B steam generator. The 3-13

inspection revealed displaced pipe, missing and damaged insulation,. and damaged pipe supports resulting from thermal hydraulic forces not previously considered in the design of that piping.

Thus, what'had begun early in the shift as an attempt to isolate an electrical-problem, led at 4:51 a.m. to a temporary loss of inplant ac power, a condition

• which, combined with five failed check valves, subsequently resulted in an incidence of water hammer powerful enough to challenge the integrity of the j safety-related feedwater system.

)

t

.A 4

[

1 d

1 k

i d

3-14 i-

Table 3.1 Chronological Sequence of Events Initial Conditions at Unit 1, November 21, 1985

- Saltwater leaking .into the main condenser at 5 x 10 -3 gpm

{ - Unit operating at 60 percent reactor power to facilitate search for condenser leak

- South circulating water pump shut down to allow entry into south condenser water boxes

- Steam generator blowdown ongoing at about 100 gpm per generator to minimize chloride buildup -

- Electrical ground troubleshooting in progress; ground determined to be located on auxiliary transformer C supply to 4kV bus 1C

- Bus IC power supply shifted to 4kV bus IA, powered from the output of the main generator through auxiliary transformer A Auxiliary transformer C remained energized, supplying power to 4kV bus 2C, while personnel inspected electrical equipment

- Fox 3 critical function monitor system recording function disabled because of previous power interruption during ground isolation effort Transient Initiator 04:51:11 Auxiliary transformer C differential relays detected a phase-to phase fault current in excess of 1500 amps and actuated trips in associated circuit breakers to isolate the transformer.

Circuit breakers 4032 and 6032 opened to isolate auxiliary trans-former C from the 220kV switchyard.

Circuit breaker 12C02 opened to isolate auxiliary transformer C from 4kV bus 2C.

Systems Response / Operator Actions 04:51:11+ Bus 2C de-energized, de-energizing the following selected loads:

East feedwater pump Southeast condensate. pump Northeast condensate pump East heater drain pump Vital 120VAC bus 4 ENS phone began to ring along with all the.other alarms associa c3 with the trip of the auxiliary transformer as the Shift Supervisor entered the control room.

15

Table 3.1 Chronological-Sequence of Events Diesel generator 2 started automatically on loss of 4kV bus 2C, but did not load automatically, per design.

East feedwater pump discharge check valve failed to seat as the de-energized pump coasted dowi..

Running west feedwater pump pressurized the east condensate-feedwater i I

heater train.

East flash evaporator condenser tubes became overpressured, ruptured and overpressurized the evaporator shell, causing the shell to develop a-fishmouth opening approximately 20 feet 1.ong and 2 feet wide. The accompanying noise was described as a " muffled howitzer."

04:51:31 Operators manually tripped the reactor in response to loss of vital 120VAC bus 4, as required by procedure, due to wholesale loss ~of control room instrumentation. The reactor trip initiated a turbine trip.

04:51:32 Operators pushed the unit trip button, opening main transformer out-put circuit breakers 4012 and 6012, auxiliary transformer A and B output circuit breakers 11A04 and 11804, and tripping the turbine.

04:51:32+ All inplant power was lost, except for 120VAC vital buses carried by inverters.

All inplant lighting was lost, except for battery powered emergency lighting.

Letdown, steam generator blowdown and the containment sphere mini-purge isolation valves shut on loss of power.

Diesel generator 1 started automatically on loss of 4kV bus IC, but did not load automatically, per design.

Station loss-of-voltage automatic transfer scheme initiated to allow backfeed of offsite power through the main and auxiliary transformers.

Security access control equipment malfunctioned following automatic transfer to alternate pov?r supply.

Elactric and steam powered auxiliary feedwater pumps received auto-matic initiation signals on low steam generator level, due to level drop following reactor trip and turbine stop valve closure. The electric-driven pump started later, after electric power was re-stored. )

The steam turbine-driven pump began a 3 -minute warmup period.-

All three steam generator feed regulating valves shut to 5 percent flow position in automatic response to a reactor trip. i 3-16

Table 3.1 Chronological Sequence of- Events As the west feedwater pump stopped, its ' discharge check l valve and the'. check valve downstream of the regulating valve of the C steam generator failed to seat. At.the-same time, the discs in each check valve downstream of regulating valves to A and B steam generators settled to the bottom of their respective valve bodies. All three i

steam generators. began to empty their feedwater lines to the east l

l flash evaporator condenser because of the tube rupture.

Shift Superintendent picked up spuriously ringing ENS phone, informed the NRC Headquarters Duty Officer (HQD0) of the reactor trip and loss of power, promised to call back, responded to questions, stated that offsite power was available and that the plant was stable and tripped, and again promised to call right back.

i Operators verified that rod bottom lights energized, indicating the reactor had tripped.

East and west main feedwater pump shaft seal drain. trap vents were observed to be blowing excessive steam and water.

The fire watch in the 4kV switchgear room received a fire alarm from the lube oil reservoir area, observed steam in' the area and called station emergency services.

East condensate-feedwater train condensate relief was observed to be blowing steam.

Main feedwater pump suction and discharge temperatures increased to approximately 400 F.

i Operators responded to a spurious annunciation and sequencer light indication of initiation of the safety injection system, but determined that plant parameters did not require operation of the system and that the system had, in fact, not actuated.

Station emergency services dispatched a fire truck to Unit 1.

Operators observed that the 18kV system isolation light actuated, indicating that the first phase of loss of voltage auto transfer scheme had been completed.

Operators attempted to reset the unit trip lockup bus to enable back-feed of power from the switchyard, but the reset failed, apparently due to the timing of the attempt before the main generator no-load motor-operated disconnect was fully opened. The operator did not verify that the reset was effective.

Operators found security access controls were not responsive and utilized planned procedures, personnel', and hardware to compensate.

04:55+ Steam turbine-driven auxiliary' feedwater pump completed its warmup l cycle and began to deliver approximately 130 gpm-AFW flow (indicated 1 3-17

Table 3.1 Chronological Sequence of Events flow was about 110 gpm/SG) at outside ambient temperature to main feedwater lines just downstream of the three feedwater control sta-tions. Reverse flow in the main feedwater line carried AFW to the condensate system.

Operators decided that the station loss of voltage automatic transfer l scheme had failed and attempted to complete the sequence from the l control room.

{

Operators discussed energizing buses using the running but unloaded diesel generators. Operators decided to energize buses using the preferred offsite power source. ,

The first attempt to close 220kV switchyard circuit breaker 4012 failed because an operator did not push the synchronizing check-bypass pushbutton.

04:55:13 The second attempt to close 4012 succeeded when the operator correctly depressed the pushbutton, but it immediately tripped free because the unit trip lockup bus had not been reset.

04:55:15 The third attempt to close 4012 had the same results as the second attempt.

An operator reset the unit trip lockup bus.

The first attempt to close 220kV switchyard circuit breaker 6012 failed because an operator had again not depressed the synchronizing check-bypass pushbutton.

04:55:24 The second attempt to close 6012 succeeded, backfeeding power from the 220kV switchyard, which had remained energized, to auxiliary transformers A and B.

Operators closed the feeder circuit breaker from auxiliary trans-former A to 4kV bus IA, re-energizing 4kV bus IA and IC. (The tie breaker between bus IA and 1C had never been opened. )

Operators closed the feeder circuit breaker from auxiliary trans-former 8 to 4kV bus IB and from bus 18 to 2C. Operators subsequently completed re-energization of the station by powering the remaining de-energized 480VAC buses.

The electric powered auxiliary feedwater pump started with a 20-second delay upon regaining power, due to the continued presence of-a steem generator low level signal, and increased AFW flow to approximately 155 gpm per steam generator / indicated flow was about 135 gpm/SG).

Letdown automatically reinitiated on return of power, but the charging pumps remained tripped.

3-18 1

Table 3.1 Chronological Sequence of Events Atmospheric steam dumps actuated on return of power, but operators shifted steam dump operations to automatic pressure control, thereby securing steam dumps.

l Operators shut feedwater isolation valves MOV-20, 21 and 22 and l feedwater. regulating valves FCV 456, 457 and 458, as required by l procedure, unknowingly stopping further voiding of steam generator feedwater lines and starting the refilling process at a rate of approximately 155 gpm per steam generator.

The Supervisor df Coordination reset radiation monitor alarms that were received because of loss of power. Resetting the monitor for steam geaerator blowdown re-initiated blowdown for each steam genera-tor at atout 100 gpm.

Letdown isolated automatically on low pressurizer level.

4 Operators checked pressurizer level and pressure as required by pro-cedure, found level and pressure were low and decreasing, at about 5 percent and 1880 psig, respectively, and became concerned that plant cooldown could be excessive or cause safety injection.

04:58 Operators started the south charging pump to raise pressurizer ' level.

04:59 The north charging pump started automatically on low charging header pressure with one charging pump running.

05:00 The suction of both charging pumps shifted automatically between VCT

] and RWST and back as the level cycled through VCT low level set points.

Operators verify proper operation of AFW pumps.

05:01 The Shift Supervisor called HQD0 on ENS to provide information on the plant transient. He completed a call 2 minutes later, indicating that they would probably declare an alert and close it out in the same call, that he needed to go, that he was still dealing with the problem and would call back.

Operators started reactor coolant pump B to provide a source for pressurizer sprays for pressure control.

05:02 Operators terr.inated AFW flow to the steam generators to minimize RCS cooldown; ther. subsequently resumed AFW flow to all steam generators at a rate of about 40 gpm per generator (indicated flow was about l 25 gpm/SG).

The STA arrived in the control room.

A plant equipment operator was dispatched to manually close main steam block valves to reduce plant cooldown.

3-19

Table 3.1 Chronological Sequence of Events 05:06 An unusual event was declared onsite. (Licensee Emergency Plan Tab D-1-1.) State and local offsite agencies were informed. A Prompt Notification Report of the declaration of en Unusual Event was not made to NRC.

05:07 A loud " bang" was heard. The nuclear plant equipment operator, sent to shut the main steam block valves, heard a water hammer and observed l steam on the turbine building mezzanine. The operator left the mezza-nine without shutting the main steam valves.

05:08 Circuit breaker 4012 wc: closed by an operator utilizing the synchronizing check-bypass pushbutton 05:09 The reactor cooling pump B thrust bearing high temperature alarm annunciated.

05:10 The control room received a report of a steam leak on the feedwater mezzanine from a dripping wet operator, who had just returned from that location.

I Letdown valves opened af ter pressurizer level rose above 10 percent.

05:12 Operators shut the turbine plant cooling water (TBCW) supply valve for containment sphere air coolers and started a TBCW pump. An operator was dispatched to re-establish TBCW flow to containment sphere air coolers.

, 05:17 Charging pump suction was shifted to the RWST to start boration for cold shutdown.

, 05:20 Operators reset the safeguards sequencers and secured the unloaded diesel generators.

Operators secured the lube oil reservoir foam system and fire pump, after confirming that the system should not have actuated.

05:24 Operators started reactor coolant pump A.

05:27 Operators started reactor coolant pump C.

The wide range level indication dropped off-scale low in all three steam generators.

05:28 Operators stopped reactor coolant pump B.

f Operators decided to establish rapid controlled cooldown of RCS at

about 100 F/hr to stop the assumed steam leak.

Operators increased flow'to steam generator A and C from about 40 gpm

to about 70 gpm each. AFW flow to the B steam generator was maintained at about 40 gpm.

3-20 e

_ _ _ - _ - - . _ _ - - _ _ - . _ . - - _ _ . _ - - - - - _ _ _ - - - - - - - - - - - _ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ^ - - - - - -

Table 3.1 Chronological Sequence of Events 05:30 Blowdown from steam generators was secured by reducing the setpoint on the radiation monitor.

Wide range water level indication returned on-scale in A and C steam generators.

Operators commenced periodic air sampling for radioactivity in the vicinity of the steam leak. The highest sample reported showed 5 X 10 10 uCi/cc.

. Personnel wearing steam suits made two attempts to identify and secure the source of the steam leak.

05

45 The turbine generator was placed on a turning gear. ,

Operators shut steam generator blowdown micro-valves.

l HQD0 called SONGS-1 on the 2NS'to check plant status and establish an open line. The shift superintendent was asked to call back once he could get someone assigned to maintain an open line.

05:46 Safety injection was-blocked.

The north charging pump was secured.

unknown Sandbags were placed at the entrance to the chemical feed room to prevent water from flowing across the floor into the electrical switchgear rooms.

05:48 SCE's emergency coordinator called HQDO, was persuaded of the need to establish an open ENS line to NRC, but was told of phone problems and that he would be called back.

1 05:57 Operators stopped boration using RWST.

05:58 Operators noted containment sphere pressure was slightly positive; found that containment sphere mini purge valve CV-10 had not been re-opened after the radiation monitors were reset following restor-ation of power; and opened CV-10, allowing containment sphere pres-

, sure to return to its' normal, slightly negative condition.

HQDO succeeded in establishing an open line between-the site emergency coordinator, the NRC regional duty officer and the headquarters Incident Response Center. The line would remain open until released by NRC. j The HQD0 notified FEMA of the declaration of an Alert.

06:15 ' Operators, unable to start the circulating water pumps due to igh-condenser temperature and. steam in the water boxes, aligned s .twater

-cooling to the turbine plant cooling water heat exchangers. j 06:30 Operators started emergency boration for cold shutdown. 1 3-21

Table 3.1 Chronological Sequence of Events 07:00 Operators secured emergency boration.

07:02 The Plant Superintendent assumed the role of emergency coordinator and changed the Unusual Event declaration basis to Emergency Plan Tab. G-1-2.

07:44 A monitoring team dispatched to measure potential offsite radio-activity determined downwind site boundary radiation levels to be less than 0.1 mrem /hr.

07:47 Operators started the second emergency boration to assure 5 percent shutdown in mode 5.

07:55 Emergency boration was secured. .

08:00 Entered Mode 4; operators still believed there was a steam leak.

08:35 Operators secured the turbine-driven auxiliary feedwater pump due to low steam pressure.

08:36 Operators aligned screen wash pumps to supply cooling for the turbine plant cooling water heat exchangers.

08:37 Operators aligned salt water cooling pumps to provide maximum component cooling water heat exchanger cooling.

09:00 Operators started a third component cooling water pump in preparation for initiating RHR.

09:10 Operators attempted to open RHR suction valves, M0V-813 and 834, but pressure interlock had not yet cleared, although RCS pressure was well below 400 psig.

Air sample from the chemistry sample room determined that noble gas activity was 1.87 times the maximum permissible level.

09:12 Containment sphere entry was made to isolate the hot leg.recircula-tion flow path by shutting RHR-004.

09:18 Operators overrode the high pressure interlock and opened MOV-813 and 834.

09:20 Operators stopped vacuum pumps.

09:30 Operators shut RHR-004.

09:35 Operators started the West RHR pump.

09:38 Operators started the East RHR pump.

09:40 The Unusual Event was terminated and both RHR pumps were in service.

3-22

Table 3.1 Chronological' Sequence of Events 10:00 Shift turnover began and continued sequentially until all positions were briefed and properly relieved.

10:45 Feedwater leakage was manually isolated.

11:15 A work order was issued to repair the security system affected by

. moisture from the leak.

( -

i 13:20 Steam generator sample : hcwadactivityinAandCless_ghanthe j threshold of detectability; activity in B was 2.87 x 10 uCi/ml.

14

06 Operators restarted the 480V room air conditioner.

14:10 Operators isolated a dc ground on control power to FCV-456 and 457.

j 14:36 Operators commenced RCS degassing.

15:08 The plant entered Mode 5. '

f NOVEMBER 22, 1985 01:00 Operators entered the containment sphere and identified damaged pipe supports and insulation on the B stean generator feedwater line. -

i

16

41 Operators secured electric auxiliary feedwater pump.

17:32 Operators secured filling the AFW tank from Unit 2 and 3.

l 21:40 Operators manually closed the main steam isolation valves.

22:45 Operators transferred water from A to B steam generator, using the blowdown lines.

I I

i O

4 4_

3-23

--e , -r>r .-w - - - . - , ~ , , ,,y .- , , - - ,, y , , ,

4 DESCRIPTION OF PLANT SYSTEMS 4.1 General Design The nuclear steam supply system (NSSS) for the San Onofre Nuclear Generating Station, Unit 1, is a pressurized water reactor (PWR) supplied by the Westinghouse Electric Company. The plant was designed and constructed by the Bechtel Power Corporation. Licensed power level is 1337 MW(t), yielding a l net electrical output of approximately 450 MW(e).

The reactor coolant system (RCS) consists of a reactor pressure vessel and three parallel heat transfer loops. Each loop contains a model 27, U-tube steam generator and a reactor coolant pump. RCS pressure control and overpressure protection are provided by a pressurizer connected to loop 8.

Pressurizer spray is provided from loops A and 8. Normal operating pressure is -

2085 psig.'

As corrective action for steam generator tube corrosion, approximately 8.6 percent of the tubes have been plugged and approximately 57 percent have been sleeved. This plugging and sleeving has resulted in lower-than-design RCS flow and reduced steam generator effective heat transfer surface. Further, to pro-vide a sess hostile operating environment for the tubes, RCS temperature was reduced from design temperature. Control and protection setpoints were adjusted to accommodate these changes in operating temperature and flow. Current pro-grammed average RCS temperature at 100 percent power is 551.5 F. A self-imposed limit by SCE restricts power to 93 percent based on steam generator moisture carryover considerations.

4.2 Main Steam System The main steam system distributes steam from the thiee steam gene'rators to the high pressure turbine, moisture separator-reheaters, turbine-driven auxiliary feedwater pump, steam dump and other miscellaneous loads.

As shown in Figure 4.1, the steam generators discharge into a common ring header within the containment building, which then connects to the two main steam lines. The main steam lines penetrate the containment building and distribute steam to the main turbine and other loads. Each main steam line has a manually operated (air wrench) block valve (MSS 301 and 302), which is used for maintenance isolation and to limit cooldown rates by isolating leaky valves and steam traps after plant shutdown.

A branch line is connected to each main steam line, outside of the containment building, upstream of the manual block vaives. Each branch line contains five self-actuating code safety valves and two steam dumps which discharge to the atmosphere. The pneumatically operated atmospheric steam dumps are controlled j

by the steam dump system in response to: ('..) reactor coolant temperature during j

load rejections or tyrhine trips and (2) steam pressure during startup or shut- '

down operations. Steam to the turbine-driven auxiliary feedwater pump comes from one of the main steam lines outside of the containment building upstream of the manual block valve. I 1

4-1

Connections to the main steam li~nes between the manual block valves and turbine stop valves include a cross-connect line and supply lines to the four moisture separator-reheaters and condenser steam dump valves CV 3 and 4. The condenser steam dump valves are operated by the steam dump system in response to the same control signals as the atmospheric steam dumps, except that they are automati-cally disabled on low condenser vacuum.

Since there are no check valves or automatic isolation valves to isolate the steam generators from each other, a steam or feedwater leak can affect all three steam generators.

4.3 Steam Generator Blowdown System The steam generator blowdown system continuously removes (blows down) water from the steam generators. This blowdown helps maintain the purity of steam generator water by removing contaminants (both chemical and radiochemical) which tend to concentrate in the steam generators. These contaminants could contribute to steam generator tube degradation.

The blowdown flowpath shown in Figure 4-2 runs from the steam generator bottom blowdown connections, out of the containment building, through manual angle throttling valves FWS 512, 516, and 521, into a common header. Flow from the common header runs through remotely operated angle check valve CV 100 to the blowdown tank. Flow from the blowdown tank can be directed either to the cir-culating water discharge or to the liquid radwaste holdup tanks, depending upon the activity level of the blowdown. An alternate flowpath bypasses the blow-down tank through CV 100 A and B directly to the circulating water outfall.

Radiation monitor R-1216 continuously samples blowdown water. Tha monitor normally analyzes a common sample from all t6ree steam generators but may be realigned to sample each steam generator individually. If R-1216 senses high radiation or loses control power, it alarms in the control room and automati-cally isolates blowdown by closing CV 100, 100A and 100B.

4.4 Main Feedwater System The main feedwater system normally provides heated feedwater at a controlled rate to the three steam generators. The auxiliary feedwater system (Section 4.6) connects to the main feedwater piping to each steam generator just prior to their containment building penetrations. In its normal mode, main feedwater is net a safety system. Because the two main feedwater pumps are part of the safety injection flowpath (Section 4.11), the pumps and all equipment required to realign their suctions and discharges are safety-related.

As shown in Figure 4.3, each main feedwater pump receives flow from a separate train of low pressure feedwater heaters in the condensate system through a pneumatic / hydraulic isolation valve. Each feedwater pump discharges through a check valve and a pneumatic / hydraulic isolation valve to a first point (high pressure) feedwater heater. The first point heater outlets join into a common header to supply the feedwater control stations for the three steam generators.

Flow to each steam generator is through an individual control station consisting of a motor-operated isolation valve, an air-operated main feed regulating valve, a check valve and a manually operated isolation valve. A small bypass line containing manual isolation valves, an air-operated bypass flow control valve, 4-2

and a check- valve is-used for low flow conditions such as startup and shutdown operations. Each main feedwater line from the motor-operated isolation valves to the steam generator is seismically qualified.

The main feedwater pumps are powered by 3,500 hp electric motors supplied from l 4kV buses 1C and 2C. Seal water is supplied from the discharge of the conden-sate pumps. Main feedwater. pump discharge check valves FWS 438 and 439 prevent-l reverse flow through an idle pump. These and the other-main feedwater system l check valves are manufactured by MCC Pacific.

I l

The first point or high pressure heaters receive extraction steam through nonreturn check valves from the high pressure turbine. Tube side design

, pressure.is 1367 psig. Each first point heater has a shell. side relief which i discharges to the blowdown tank.

i

! The following feedwater system parameters are indicated and recorded in the

control room:

i 1. Main feedwater pump suction pressures and temperatures.

! 2. Pressure and temperature in the common portion of the main feedwater system at the outlet of the first point heaters.

3. Feed flow, steam flow and level for each steam generator.

i After a reactor trip, main feedwater regulating valves FCV 456, 457 and 458 are j

automatically throttled to five percent flow, which is sufficient for decay heat removal, but minimizes overcooling due to excessive 'feedwater addition.

' The reactor trip response procedure directs the operators to isolate the main feedwater flowpath, after verifying auxiliary feedwater flow, by closing the main and bypass regulating valves (FCV 456, 457, 458, 142, 143 and 144) and the

[ motor-operated feedwater header isolation vah es (MOV 20, 21 and 22).

l A safety injection system signal automatically shuts the main feedwater

} regulating valves, the bypass valves and the motor operated isolation valves.

4.5 Condensate System .

i 4

The-condensate system transfers deaerated condensate from the main condenser hotwells through two parallel trains of low pressure ~ feedwater heaters to the suction of the main feedwater pumps. Steam extracted from the main turbine

] provides the heat source for the feedwater heaters.

1 As shown in Figure 4.4, the system begins with the four condensate pumps which take suction from the hotwells'of condensers E-2A (north) and E-28 (south).

1 The four hotwells are tied toge'.her by a 24-inch line which also connects to i the suctions of all condensate pumps. The discharges of the pumps are tied i together af ter their respective check valves and then the system splits into two trains. Each train consists of:

1. An air ejector condenser.

2. A turbine gland sealing steam condenser.

] 3. A flash evaporator unit containing a flash evaporator, a 5th point heater and drain cooler and a 4th point heater and drain cooler.

4. A 3rd point heater.

5. A 2nd point heater.

A i 4-3 k.

The system is nonsafety-related up to the pneumatic-hydraulic isolation valves in the main feedwater pump suctions.

The condensate pumps are tripped on loss of voltage on bus 1C and 2C or a safety injection system signal. Pump seal water is provided from the discharge of the condensate pumps.

The air ejector and gland steam condensers are shell and tube heat exchangers with condensate flowing through the tube side. Tube side design pressure is 350 psig.

The flash evaporators are combined ur.its consisting of a flash. evaporator in a common housing with the 5th and 4th point low pressure feedwater heaters and drain coolers. The flash evaporators are designed to produce fresh water from sea water but have not been used for several years. Condensate does, however, still flow through the tubes of the evaporator condenser. Design pressure of the tube side of the flash evaporator condenser and low pressure feedwater heaters is 350 psig and shell side design pressure is 15 psig. A shell side relief valve on each flash evaporator set at a nominal 15 psig discharges to the blowdown tank.

The 2nd and 3rd point feedwater heaters are also designed for 350 psig tube pressure. Shell side reliefs from the 2nd and 3rd point heaters discharge to the blowdown tank. A water box thermal expansion relief valve is provided on each 3rd point feedwater heater and discharges to an open drain line and to che atmosphere under the 3rd point heater. Nonreturn check valves are provided in the extraction steam lines for all feedwater heaters except the 5th point.

4.6 Auxiliary Feedwater System The auxiliary feedwater system (AFW) is a safety system designed to assure a reliable heat sink for the reactc.r during abnormal or emergency conditions when main feedwater is not available. The AFW system also provides feedwater to the steam generators during normal startup, shutdown and hot standby conditions.

, Either of the redundant AFW pumps can supply sufficient flow to the steam gen-l erators for decay heat removal present 3 1/2 minutes after a trip from full power.

The AFW system shown on Figure 4.5 consists of a motor-driven pump, a turbine-driven pump, an auxiliary feedwater tank, and associated piping, valves and instrumentation. Flow is from the AFW storage tank through individual suction lines to the two auxiliary feedwater pumps. The pumps discharge through inde-pendent check and isolation valves to the auxiliary feedwater flow control valves for the three steam generators. Each flow control valve can receive flow from the motor-drivea and the turbine-driven auxiliary feedwater pump discharge header.

FCV 2300 and 3300 control AFW flow to the A and C steam generators, respectively, while FCV 2301 and 3301 control flow to the B steam generator. Each flow con-trol valve discharges through check and isolation valves to the main feedwater piping of its respective steam generator between the main feedwater control stations and the feed line containment building penetration. Auxiliary feed-water flow is then through the main feedwater piping into the steam generators.

The system is automatically actuated if narrow range level in two of three steam generators indicates less than 5 percent. Narrow range level is expected 4-4

to shrink to less than 5 percent after a trip from full power.

The system may

-also be initiated manually.

The auxiliary feedwater tank is a seismically qualified 240,000 gallon tank with 150,000 gallons reserved for AFW. All non-AFW penetrations are above the 150,000 gallon level. Normal makeup is from the Unit 2 makeup demineralizer or l

the Unit 2 condensate storage tank and requires operator action to install the requisite fire hoses to provide a flow path.

The turbine-driven auxiliary feedwater pump is a centrifugal pump driven by a i single-stage turbine supplied by the turbine division of the Worthington Corporation. Steam for the turbine is from the west main steam header, upstream of the maintenance shutoff valve, and the turbine exhausts to the atmosphere. The pump will provide a 300 gpm flow at a design head of 1093 4

psig. The turbine is provided with an automatic 3 1/2 minute startup sequence to remove ani water from the steam supply line and turbine casing. Flow fror this i

pump is not available until the sequence is complete. The pump is i automatically tripped on low suction pressure. Suction and discharge pressures are indicated in the control room.

The motor-driven auxiliary feedwater pump is a centrifugal pump driven by a i

250-hp electric motor powered from 480-volt bus 1, which can be powered by the emergency diesel generators. The capacity of the motor-driven pump is 235 gpm at 1058 psig. Pump trips include low suction pressure, motor overcurrent or loss of power and low discharge pressure, the latter to protect the motor from pump runout. Suction pressure, discharge pressure and motor amps are indicated in the control room.

] Auxiliary feedwater flow control valves FCV 2300, 3300, 2301 and 3301 are i

divided into two trains. Train A consists of FCV 2300 and 2301 and train B

{ consists of FCV 3200 and 3301. FCV 2300 and 3300 to steam generators A and C are manually preset to 100 percent open and FCV 2301 and 3301 to steam generator 8 are manually preset at 50 percent open. These settings are designed to limit AFW flow at normal steam generator operating pressures to less than 150 gpm per steam generator to minimize the potential for steam generator water hammer. Auxiliary feedwater flow to each steam generator is indicated in the' control room.

4.7 Salt Water Cooling Systems

, The salt water cooling systems shown on Figure 4.6 provide a heat sink for both

safety and nonsafety-related loads. The systems consist of the condenser and

' circulating water system, the screen wash and salt water cooling pumps, and the turbine plant and component cooling heat exchangers.

l The nonsafety-related condenser and circulating water system provides a heat sink to cool and condense exhaust steam from the low pressure turbines and is )

the normal cooling supply to the turbine plant cooling water heat exchangers i (see Section 4.8). Water from the Pacific Ocean is drawn'into the intake I structure, where traveling screens prevent trash and debris from entering'the pumps. The circulating water pumps take suction from the intake structure and discharge through the four condenser water boxes and back to the ocean through the outfall piping. During normal operation, the north pump supplies the north waterboxes of both condenser sections and the south pump supplies the south i

l 4-5 i

1

. , , _ . _ . ._, < - ~ - - - - - - , - -,---~---r - - " - ' ' " ~ ~ ^ ~ ~ ~

waterboxes. The system is designed to allow reduced power operation with one:

circulating water pump off. Under these conditions, either the north or south waterboxes may be isolated and drained for personnel entry to identify and repair saltwater leaks. Some of the circulating water pump flow is normally diverted to cool the turbine building cooling water heat exchangers. The circu-lating water pumps have a design capacity of 173,000 gpm each and are powered by 1500-hp electric motors. The north pump is supplied from 4kV bus 1C and south pump from bus 2C. The condenser waterboxes are connected to a vacuum priming system to ensure that they are full before starting a circulating water pump. Pressure switches PS 99 and 100 prevent the start of the circulating water pumps unless they sense at least a 20-inch vacuum indicating that the waterboxes are full.

Normal supply to the non M aty-related turbine plant cooling water heat exchangers is from i.he circulating water pumps through power-operated valves (POV) 7 and 8. if the circulating water pumps are not available, the turbine plant cooling water heat exchangers can be supplied from either the safety-related saltwater cooling pumps or the nonsafety-related screen wash pumps shown in Figure 4.7. Turbine plant cooling water is important because, in addition to normal secondary plant cooling loads, it is the cooling water supply to the containment building cooling units.

The safety-related component cooling water heat exchangers are normally supplied by the salt water cooling pumps. The saltwater cooling pumps are automatically started on a safety injection signal. The pumps are powered by 100-hp electric motors supplied from 480V buses 1 and 2. If the saltwater cooling pumps are unavailable, the screen wash pumps can be manually aligned as a backup.

The normal function of the screen wash pumps is to backwash the traveling screens of the intake structure to wash off any debris or trash which has accumulated I and to minimize differential pressure across the screens. The pumps are powered by 100-hp motors supplied from 480V buses 1 and 2. The screen wash pumps can be manually aligned to provide a backup source of cooling water to the component cooling water and the turbine plant cooling water heat exchangers.

4.8 Turbine Plant Cooling Water System The turbine plant cooling water (TPCW) system, shown on Figure 4.7, is a non-safety, closed-loop system which provides cooling water for turbine auxiliaries f and the containment building cooling and ventilating units. The heat sink for the system is normally circulating water, but the saltwater cooling or screen wash pumps can be manually aligned as backups.

The system consists of a turbine plant cooling water tank, two pumps, two heat l exchangers and associated valves, piping and instrumentation. The normal flow-path in the closed-loop system is from the turbine plant cooling water tank, through one of the pumps and its cooler to the supply header. Cooling water is distributed to the various loads and is returned to the tank. Some of the loads include circulating water pump motor bearing coolers, steam and feedwater sample coolers, turbine lube oil coolers, instrument and service air compressors, gen-erator hydrogen coolers, and containment building cooling and ventilating units.

The turbine plant cooling water pumps are powered by 350-hp electric. motors supplied from 4kV busses IC and 2C. System piping to the containment building 4-6

____-____a

l l

l cooling and ventilating units, between isolation valves CV 515 and 516, is safety related. If both TPCW pumps are off, it is possible for elevated por-tions of the TPCW system inside the containment building to drain. To minimize the potential for classical water hammer in the building cooling and ventilating units on restoration of flow, the TPCW flowpath to the building is isolated by the operator prior to starting a TPCW pump. The manual isolation valve is then throttled open slowly to restore building cooling.

4.9 Chemical and Volume Control System The chemical and volume control system (CVCS), shown on Figure 4.8, is a safety-related auxiliary system with both normal operating and post-accident functions. Normal operating functions include purifying the reactor coolant system (RCS); maintaining proper water inventory in the RCS; providing seal water for the reactor coolant pump seals; adjusting RCS boron concentration; maintaining the proper concentration of corrosion-inhibiting chemicals in the RCS; and filling and pressure-testing the RCS. A safety injection signal starts the preferred charging pump to previde borated water from the refueling water storage tank (RWST) for conditions when RCS pressure remains greater than safety injection system discharge pressure.

The basic CVCS flowpath is from RCS loop A cold leg, through the regenerative heat exchanger and letdown orifices, to the residual heat removal (RHR) heat exchangers. The cooled and depressurized letdown flow then leaves the contain-ment building when it is purified by demineralizers before being sprayed into the hydrogen gas space of the volume control tank (VCT). RHR temperature re-corder TR600 indicates letdown temcerature in the control room. The charging pumps take suction on the VCT and discharge through a flow control valve to the tube side of the regenerative heat exchanger and then to the loop A cold leg.

Reactor coolant pump seal injection is from the discharge of the charging pumps tnrough seal injection filters and individual flow control valves to the seals of the three reactor coolant pumps. Makeup of demineralized water and/or boric acid is supplied to the charging pump suction. '

The two centrifugal charging pumps have a capacity of 136 gpm at 2300 psig.

The pumps are operated from the control room and are powered from 4kV buses 1C and 2C.

Some CVCS components are provided with interlocks or automatic actuations based on plant conditions:

1. Letdown isolation valve LCV 1112 shuts if pressurizer level decreases to 10 percent but opens on restoration of level or loss of power.

2. Orifice block valves CV 202, 203 and 204 are closed by a safety injection signal (SIS) or loss of power.

3. Pressure control valve PCV 1105 fails shut on loss of power.

4. Charging pump suction transfers from the VCT to the RWST on SIS or VCT low level; for a VCT low level transfer, suction is automatically returned to the VCT when level is restored.

5. If a pump is running, the nonrunning charging pump starts automatically on low charging header pressure or overcurrent trip of the running 4-7

pump. Pumps which trip on loss of power do not automatically restart when power is restored.

6. Safety injection starts the preferred charging pump and opens charging flow control valve FCV 1112.

Control power for CVCS letdown components is supplied from the utility and vital buses as follows:

1. 120 VAC utility bus - LCV 1112, CV 202, 203 and 204

2. Vital bus 1 - CV 525

3. Vital bus 4 - PCV 1105

4. Vital bus 5 - CV 526 1

4.10 Residual Heat Removal System l The residual heat removal system (RHR) removes reactor decay heat and reactor coolant system (RCS) sensible heat during operations below 350 F. During operations at normal RCS pressure and temperature, the RHR heat exchangers are used as part of the chemical and volume control system (CVCS) to cool RCS letdown prior to purification. RHR system piping is also part of the hot leg injection flowpath during-long-term recirculation cooling following a major loss of coolant accident. The RHR sysf.em shown in Figure 4.9 consists of two pumps and two heat exchangers located inside the containment building. Flow during cooldown and refueling operations comes from a loop hot leg, through the RHR pumps and heat exchangers, to a loop cold leg. The low temperature and pressure RHR system is normally isolated from the RCS by a series of motor-operated isolation valves (MOV 813, 814, 833 and 834). An interlock is provided from pressurizer pressure instrumentation that permits opening valves 813 and 834 when pressure is less than about 370 psig. The interlock clears if pressure increases to about 400 psig, preventing the valves from being opened. Cooling water for the RHR heat exchangers comes from the component cooling water system (CCW).

To provide a hot leg injection flowpath, for use during long-term recirculation cooling, RHR pump "A" manual bypass valve RHR 004 is normally left open. The recirculation flowpath enters the RHR system at the outlet of the RHR heat exchangers. Flow is then in the reverse direction through the heat exchangers to the b pass

/ around pump "A" (RHR 004) and into the hot leg through MOVs 813 and 814. Prior to initiating normal RHR cooling, operators enter the containment building to close RHR-004.

4.11 Safety Injection System The safety injection system is designed to mitigate the consequences of a loss of coolant accident. It injects borated water for initial core cooling and subsequently recirculates and cools spilled reactor coolant and injection water from the containment building sump fcr long-term cooling.

The safety injection flowpath, shown on Figu.re 4-10, is from the refueling water storage tank (RWST) to the safety injection pumps, main feedwate, pumps and the safety injection pentration to all three loop cold legs. An additional flowpath is from the RWST to the preferred charging pump and through the normal charging path.

4-8

Safety injection is automatically actuated by a signal from either low pres-surizer pressure (2/3) or high containment building pressure (2/3); it may also be initiated manually. Upon actuation, the main feedwater pumps are tripped and isolated from the condensate and feedwater systems and are realigned to take suction from the safety injection pumps, which provide borated water from the refueling water storage tank. Feedwater pump discharges are realigned to the safety injection header, the three cold leg injection valves open, and the l feedwater pumps are restarted. The feedwater pump minimum flow is automatically transferred to the RWST by a . safety injection signal to provide pump protection in case RCS pressure remains above the 1175 psig shutoff head of the pumps.

The main feedwater pumps are 3500-hp notor-driven pumps capable of providing 10,500 gpm flow in their safety injection-alignment. Power to the feedwater i

pumps is from 4kV buses 1C and 2C.

i To improve the reliability of the feedwater pump realignment, valves HV 851 A and B and liv 853 A and B are equipped with bonnet vent valves. These vent i

valves were added to vent the pressure between the double disks to prevent disk ~

binding and impaired movement. Additionally, a small hole was drilled in the disks of main feedwater pump discharge check valves FWS 438 and 439 to prevent the formation of a hydraulic lock and improve the reliability of HV 851 A and B.

Because RCS pressure may remain above feedwater pump discharge pressure under some conditions, a safety injection automatically realigns the CVCS charging system to provide an immediate injection of borated water. Charging pump suc-tions are realigned to the RWST and the preferred charging pump starts to pro-t vide flow through charging tiow control valve FCV 1112. The operator may man-1 ually align the charging pump flow to the cold leg injection penetrations.

I i During normal cooldown operations when RCS pressure is less than 500 psig, Technical Specifications require establishing two positive barriers between the RCS and the feedwater and condensate systems to prevent the flow of unborated water into the RCS. These barriers may be motor-operated valves, when closed and tagged with power removed; pneumatic-hydraulic valves, when closed and tagged with the respective hydraulic block valves closed; or manually operated valves, when locked closed or tagged.

4.12 Electrical Distribution System 4.12.1 General Description The SONGS Unit 1 electrical distribution system consists of the main transformer, auxiliary transformers A, B and C (which interface with the 220kV switchyard and the inplant electrical system), and the inplant electrical system (composed of the main and emergency diesel generators, the 4160V, 480V and 120V ac buses and loads, and the 125V de system). Figure 4.11 shows the arrangement of the 220kV switchyard circuit breakers and buses for Unit 1. Figure 4.12 shows the j arrangement of the main generator, auxiliary transformers, main transformer, 4

4160V syster and 480V system. Figure 4.13 indicates the arrangement of 120V ac

buses and the 125V dc system The main generator is a 500MVA unit that supplies its output power at 18kV to-the main transformer and auxiliary transformers A and B. The main transformer-3 is a 485MVA unit which supplies the main generator output to the San Diego Gas i

J i

4-9 i

l and Electric (SDG&E) and Southern California Edison (SCE) nower transmission systems through the 220kV SONGS switchyard. The main transformer steps up the 18kV generator voltage to the switchyard 220kV. Auxiliary transformer A and auxiliary transformer B are each rated 10/12.5MVA (OA/FA) and step down the 18kV generator voltage to supply inplant 4160V buses 1A and 18, respectively.

The main and auxiliary transformers are directly connected to the generator bus. The main transformer output is connected to the northeast and northwest 220kV switchyard buses through circuit breakers (CB) 4012 and 6012. Auxiliary transformer A supplies 4160V bus 1A through circuit breaker 11A04, and auxiliary transformer B supplies bus 18 through CB 11804.

The main generator 18kV bus has a no-load motor-operated disconnect (MOD) switch, which can be used to isolate the generator from the transformers, and which )

allows buses 1A and IB to be backfed from the switchyard through the main and auxiliary transformers. (The latter feature is provided to meet the delayed-access circuit requirement of General Design Criterion (GDC) 17.)

Auxiliary transformer C is the immediate access circuit (required by GDC 17) between the switchyard and the in plant safety-related electric distribution system. Auxiliary transformer C is a 50MVA unit with one primary winding (designated as H winding) and two secondary windings (designated as X and Y windings). The H windings of auxiliary transformer C are connected to the switchyard through circuit breakers 4032 and 6032 and step down the 220kV from the switchyard to 4160V. The 4160V X winding supplies bus IC through circuit breaker 11C02, and 4160/ Y winding supplies bus 2C through 12C02. A current-limiting reactor and a bypass breaker are provided on each of the X and Y wind-ing outputs. The reactors are used only when the associated diesel generator is being load tested; they are used to reduce any current associated with a fault during this mode of operation. Tie breaker 11001 is provided between bus 1A and bus 1C and tie breaker 12C01 between bus IB and bus 2C.

The two emergency diesel generators, each rated at 6000kW, are designed to sup-ply 4160V buses 1C and 2C during emergency conditions when offsite power is unavailable to the buses. The 4160V buses 1C and 2C supply the unit's 480V switchgears 1, 2 and 3 through station service transformers. These switchgears in turn supply nine motor control centers (MCC's 1, lA, 18, 1C, 2, 2A, 2B, 3, 3A) and all 480V loads. One set of such loads includes the battery chargers that are associated with 125V dc buses 1 and 2.

Unit 1 has seven 120V ac vital buses and one 120V ac utility bus which supply power to various instrumentatitq and control, systems. Vital buses 1, 2, 3, 3A, 5, and 6 are normally supplied by inverters off the 125V dc buses. Vital bus 4 and the ut.ility bus are supplied by the unit's 480V system (MCC 2 is the normal supply and MCC 1 is the alternate) through single phase transformers.

4.12.2 220kV System During normal plant operation, circuit breakers 4012 and 6012, asrociated with the main transformer, and circuit breakers 4032 and 6032, associated with  !

auxiliary transformer C, are closed. These breakers can be operated (i.e.,

closed or opened) from the control room by control switches located on the vertical panel. These breakers are automatically tripped by protective relays associated with the switchyard and with their associated t.ansformers. One such protective relay is the differential relay associated with auxiliary 4-10

transformer C, which on actuation will trip open circuit breakers 4032 and 6032.

Similarly, circuit breakers 4012 and 6012 open automatically upon actuation ~of protective relays associated with the main generator, main transformer, and auxiliary transformers A and B. Actuation of the unit trip or undervoltage on both buses 1C and 2C trip open breakers 4012 and 6012. Actuation of the above-mentioned protective trip relays associated with the main and auxiliary trans-formers and the. main generator would also actuate the generator lock-up and associated alarm buses.

4.12.2.1 Closing 220kV Circuit Breakers for Backfeeding Circuit breakers 4012 and 6012 are associated with generator output to the 220kV l switchyard. Normally > these breakers are closed to supply generator output to the switchyard. However; during situations when auxiliary transformer C is unavailable and the unit'is tripped, the only available path for offsite power to supply the station distribution system is through the main and A and B auxil-iary transformers. In this mode of. operation, the 18kV bus must be isolated '

1 from the main generator. This isol'a tian and alignment of the distribution system are automatically parformed by the " loss-of-voltage auto transfer" scheme. At the completion of the automatic transfer scheme, operators are required Jo man-ually close breakers 4012 and 6012 to complete the backfeed operation. In order to close the breaker manually, the operator must:

1. Depress the unit trip reset pushbutton in order to remove all trips on the breakers.

2. Insert the synch. selector switch in the appropriate control location and turn it on.

3. Depress the synch. check bypass pushbutton.

] 4. Turn the circuit breaker control switch to the close position.

These four steps are required to close the first switchyard circuit breaker, normally CB 4012 associated with the 220kV northeast bus. To close the second breaker, CB 6012, only steps 2 and 4 are required. Step 1 is no longer neces-l sary since unit trip reset has reset the generator lock-up, and step 3 should

j. not be used nrder to allow the synch. check relay 25X to perform its function of verifying ,..chronism between the northeast and northwest 220kV buses. (When the main generator is operating, synchronizing check relay 25X provides a per-missive to assure that the generator output is within limits with respect to the running 220kV system.)

i 4 4.12.3 4160-Volt System Bus 1A is normally supplied by auxiliary transformer A through breaker 11A04.

It can also be tied to bus 1C through tie breaker 11C01 and be supplied by auxiliary transformer C during startup or whenever needed. The loads on-this

bus are the motors for reactor coolant pumps A and C.

Auxiliary transformer 8 normally supplies Bus 1B through breaker 11804. Its alternate supply path is from auxiliary transformer C to bus 2C and tie breaker 4

12C01. The loads connected to this bus are the reactor coolant pump 8 motor, and the main and spare exciters of the main generator.

i Bus IC, a safety-related bus, is normally supplied by the X winding of auxiliary transformer C via circuit breaker 11C02.- It can be supplied by auxiliary transformer A through bus 1A and tie breaker 11C01. Emergency diesel l

4-11

generator 1 can also supply this bus through its associated breaker 11C14. The following loads are connected to bus IC.

1. West feedwater pump motor

2. North circulating water pump motor

3. West safety injection pump motor

4. Southwest condensate pump motor

5. Charging pump B motor

6. Northwest condensate pump motor

7. West heater drain pump motor

8. Normal lighting transformer feeder

9. Station service transformer (SST) I feeder

10. Station service transformer 3 feeder

11. South turbine plant cooling water (TPCW) pump motor Bus 2C is the other safety-related 4160V bus which is normally supplied from i the Y winding of auxiliary transformer C through breaker 12C02. It can be supplied through tie breaker 12C01 from bus 18. Its associated emergency diesel generator 2 can also supply the bus through breaker 12C15. The loads connected to this bus are as follows.

1. East feedwater pump motor

2. South circulating water pump motor

3. East safety injection pump motor

4. Southeast condensate pump motor

5. Charging pump A motor

6. Northeast condensate pump motor

7. East heater drain pump motor

8. North TPCW pump motor

9. Station service transformer 2 feeder

10. Standby lighting transformer feeder

11. Switchyard feeder

12. Alternate feeder to SST 3 Tie breakers 11C01 and 12C01 can be opened and closed from the control room.

They can also close automatically on station loss of voltage auto transfer sequence. They trip and lockout on safeguard load sequencing system (SLSS) operation. (The SLSS actuates and sequences the various emergency safeguard features in the event of a safety injection signal (SIS), loss of offsite power (LOP), loss of 4160V bus IC/2C (LOB), or SIS and LOP.) The tie circuit breakers are provided with overcurrent protection.

The source breakers to buses 1A and IB (11A04 and 11804) can also be operated from the control room. They too automatically close during the station loss of voltage auto transfer scheme. These circuit breakers trip on overcurrent protection actuation, on actuation of protective features provided for the associated transformers and generator, and on unit trip.

Source breakers 11C02 and 12C02 for buses 1C and 2C can be opened and closed from the control room. These circuit breakers do not have an auto close feature and trip on overcurrent, auxiliary transformer C protection (such as differential relay) and during the station loss-of-voltage auto transfer scheme.

4-12

Diesel generator breakers 11C14 and 12C15 can be closed from the control room

-during a loss of power (LOP), provided the diesel generator is at valid voltage and frequency, the respective bus 1A-1C or bus -1B-2C tie breaker and auxiliary transformer reactor bypass breaker are open, and the initiating LOP signal has been reset at the SLSS surveillance panel.

Buses IA and 18 are provided with undervoltage alarms and undervoltage relays whose contacts are used in the trip circuits of the reactor coolant pump motors.

Buses IC and 2C also have undervoltage alarms and relays. Undervol+. age relay actuation on buses-1C and 2C will trip certain connected loads, initiate the loss'of voltage auto transfer scheme, and start the associated emergency diesel generators. Buses 1C and 2C are each provided with a ground detector relay, which alarms on a 10 percent ground (the set point is 21 volts), and a dual scale (10 percent and 100 percent) ground voltmeter located on the control room -

electrical board. Buses 1A and IB do not have ground detectors that directly monitor grounds on the bus, but have ground detector relays and voltmeters that monitor the secondary neutrals of auxiliary tranformers A and B. The alarm in this case is set at 6.9 volts. '

h.

4.12.4 480-Volt System

• y ',

The 480V systam is supplied power from the 4160V system through station service transformers sSST). SST 1, connected to 4160V bus IC, supplies 480V bus 1; SST 2, connected to 4160V' bus 2C, supplies 480V bus 2; and SST 3, normally connected to 4160V bus IC with 4160V bus 2C as an alternate source, supplies 480V bus 3.

SST 3 source breakers 11C11 and 12C11 are designed to be shed on undervoltage of 4160V bus 1C ar.d 2C, respectively.

The following loads are connected to 480V bus 1.

toxiiiary feedwater pump G105 Sphere Enclosure Building (SEB) normal ventilation fan A40 SEB normal exhaust fan A42 North screenwash pump North saltwater cooling pump East condanser vacuum pump North refaeling pump North component cooling pump East RHR pump West fire pump East recirculation pump South instrument air compressor Pressurizer heater group A Pressurizer heater group C Battery charger set A MCC's 1, 1A, IB, 1C This bus is provided with a ground detector system (a relay at the bus and a

dual-scale voltmeter in the control room). The bus voltage is indicated in the control room and undervoltage relays are also provided. Sus undervoltage actuation trips ali feeder breakers,.except the MCC feeders, instrument air compressor and battery charger.

4-13

r The loads on 480V bus 2 are :

East fire pump

. South screenwash pump West condenser vacuum pump Component cooling pump South' refueling pump West RHR pump South saltwater pump SEB vent supply fan A41 SEB exhaust fan A43 Instrument air compressor West recirculation pump Pressurizer heaters groups B and D Battery charger set B MCC's 2, 2A, 28 The ground detection and bus voltage monitoring for this bus are similar to those of bus 1. Also similar is that undervoltage on the bus will trip all feeder breakers except MCC feeders, instrument air compressor and battery -

charger.

The following loads are connected to 480V bus 3.

Turbine auxiliary lube oil pump Auxiliary saltwater cooling pump North instrument air compressor Auxiliary cooling pump Boric acid batching heaters South component cooling pump Switchgear room HVAC distribution panel MCC's 3, 3A Ground detection and voltage monitoring, similar to those on the other 480V buses, are provided. On a bus undervoltage condition the breakers trip loads from the bus for the cooling water pump, turbine auxiliary lube oil pump, auxiliary cooling pump, boric acid batch heaters, and auxiliary saltwater cooling pump.

4.12.5 120-Volt AC System This system is designed to provide a reliable, regulated and redundant source of single phase 120V ac power for olant controls and instrumentation. -As shown on Figure 4.13, it consists of inverters, transformers, regulators, automatic transfer switches, seven vital buses, and one utility bus.

Inverters 1, 2 and 3, connected to 125V dc bus 1, are the normal source of power to vital buses 1, 2, 3 and 3A. These buses transfer automatically to the 37.5kVA transformer if the inverter voltage or frequency deviates from pre-set limits.

Similarly, vital buses 5 and 6 have inverter 5 as the normal supply with the 7.5kVA transformer as back-up. Vital bus 4 is normally supplied from the_regu-lated 7.5kVA supply, with automatic transfer to the 37.5kVA back-up supply.

Utility bus transfers from the 37.5kVA transformer to lighting switchboard back-up supply. The two single phase transformers which supply power to vital 4-14

buses 1, 2, 3, 3A, 4, and utility bus, are supplied 480V from MCC 1 and MCC 2 through a manual master transfer switch (the normal supply being MCC 2).

4.13 Safeguard Load Sequencing System (SLSS)

,The primary function of the safeguard load sequencing system (SLSS) at SONGS 1 is to detect and react to low pressurizer pressure, high containment building pressure, and 4160V bus 1C and/or 2C undervoltage signals. The SLSS actuates

!. and sequences emergency safeguard features (ESF) in the event of a safety in-jection signal (SIS), loss of offsite power (LOP), loss of 4160V bus IC or 2C (LOB), or a simultaneous SIS and LOP (SISLOP).

The SLSS is composed of two independent and redundant sequencer trains desig-nated as sequencer 1 and sequencer 2. Each sequencer is composed of one logic cabinet, one termination cabinet with two cable assemblies, and-one remote sur-veillance panel. The sequencers receive power from the 125V dc system.

Each sequencer logic uses a two-out-of-three low pressurizer pressure or high containment pressure scheme to initiate an SIS. The LOP and LOB logic is a one-out-of-two bus undervoltage arrangment. Each sequencer has redundant sub-channel X and subchannel Y with inputs from pressurizer pressure, containment pressure, and 4160V buses 1C and 2C undervoltage relays. Both subchannels must actuate in order for the sequencer to actuate.

The two remote surveillance panels, one for each sequencer, are mounted in the control room on their associated diesel generator control board. Each panel contains the SIS manual initiation and reset pushbuttons and switch, the LOP manual initiation and reset pushbuttons and switch and eight status lamps. Six of the eight lamps are located on the right-hand side of the panel and they give the status of SLSS load group A through F. These lamps are normally il-luminated and go off when their respective load group is sequenced on. One of the remaining two lamps is a normally energized lamp indicating availability of the sequencer power supply. The final lamp is a " door closed" indicator that will extinguish if any one of the four sequencer doors is open.

The load groups are associated with the load sequencing that is automatically initiated by the SLSS upon a SISLOP signal. The load groups are designated as follows:

Load Group A - Time 0 seconds. This load group is designed to perform its function immediately with no timing ~ circuitry involvement. (There is a 10-second allowance for the diesel generators to reach rated voltage and frequency.)-

Load Group B - Time 11 seconds. This load group is designed to load onto the safety-related buses 1 second after the diesel generator output breaker closes.

Load Group C - Time 12 seconds. This load group is designed to load

2. seconds af ter diesel generator breaker closure.

Load Grop D - Time 21 seconds. This load group is designed to be energized- 11 seconds after the diesel generator output breaker is closed.

4-15

• Load Groups E and F are spares and not used.

The functioning of the SLSS and its load groups for various conditions are as follows:

1. SIS initiation with offsite power available will cause

a. a reactor trip and a unit trip,.

b. the start of both diesel generators, and

c. acutation of all SI loads in all the load groups through the sequencer without any timing sequence.

2. SIS initiation with loss of offsite power will cause

a. a reactor and unit trip,

b. both diesel generators to start,

c. load stripping from 4160-V and 480-V buses, and, i

d. the diesel generator output breakers to close and initiate load sequencing of load groups A through D.  ;

3. Loss of offsite power, LOP, will cause

a. a reactor trip and a unit trip, and

b. both diesel generators to start.

4. Loss of 4160V bus 1C or 2C, LOB, will start the associated diesel generator.

When conditions 1 and 2 above occur, the load group A through D status lamps at the remote surveillance panels will be off, indicating that the load groups have sequenced on. Under condition 3, only the group A status lamp at each panel will extinguish. Under condition 4, only the group A lamp on the panel associated with the lost bus will go off.

4.14 Diesel Generators The two redundant diesel generators prov'de a reliable standby source of power to safety-related equipment if offsite sources are not available. Either diesel generator can supply sufficient safety-related loads to respond to accident conditions or place and maintain the reactor in a safe shutdown condition.

The diesels are 8375-hp, turbo-charged units supplied by Transamerica-Delaval.

Each 4KV generator has a continuous rating of 6000kW. with a ten percent overload rating for 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. The generators start automatically on a safety injection signal or loss of power signal (LOP). However, the generators are not automatically connected to their respective buses and loaded sequentially unless both safety injection and loss of power conditions exist.

Fuel oil for the generators is stured in two 50,000 gallon tanks. Each gener-ator also has a day tank containing a fechnical Specification minimum of 290 gallons for a 45-minute supply at full load. A low day tank level starts an ac powered fuel oil transfer pump. Fuel is supplied to the diesel injectors by an engine-driven pump, with a dc powered standby. Lubricating oil is 4-16

provided by an engine-driven pump with an ac motor-driven backup. Each diesel has an oil cooler supplied by the diesel cooling water system.

The dedicated cooling water system for each generator circulates coolant from the engine and its auxiliaries through a fan-cooled, water-to-air heat exchanger (i.e., a radiator). Coolant is circulated by an engine-driven pump and the radiator fans are driven by an ac motor. A diesel can operate unloaded, without overheating, for 39 minutes with no power to the radiator fan motor.

The redundant air-starting system for each diesel consists of two compressors, two receivers and associated valves, piping and auxiliaries. The receivers are designed to provide five start attempts w.ithout recharging.

l Power to the ac powered diesel auxiliaries comes from diesel generator motor control centers (MCC)-1B and 28. These MCCs ultimately receive power from 4kv buses IC and 2C, which are normally powered from offsite, but receive standby power from the diesel generators.

4.15 Containment Building Isolation The purpose of the containment building isolation system is to minimize the release of radioactive materials to the environment by automatically isolating all nonessential systems that penetrate the building. Isolation is initiated by a containment building high pressure or safety injection signal. Main and mini purge ventilation isolation also occur if high containment building radio-activity is sensed by radiation monitor R-1212. Automatic isolation valves are provided in all normally open, nonessential systems that penetrate the contain-ment building. These valves fail shut on loss of power or operating air.

Control power for the containment building isolation signal actuation logic and building isolation valves (except for main purge valves POV 9 and 10, and mini-purge valve CV 10) comes from either 125 VDC buses 1 and 2 or from inverter-supplied 120 VAC vital buses. Control power for POV 9 and 10 and CV 10 comes from 480 VAC bus 1. Containment building radiation monitor R-1212 receives control power from 120 VAC vital bus 4, which is not inverter supplied.

During power operations the containment mini purge flowpath through CV-10, 40 and 116 is normally open to maintain a slight negative atmospheric pressure inside the building. This flowpath is required because of air leakage from valves and instruments in the building. CV-10 fails shut on loss of power and remains shut after power is restored. CV-10, 40 and 116 receive an isolation signal when radiation monitor R-1212 alarms or loses control power. The radia-tion monitor must be reset before the mini purge isolation valves can be reopened.

Normally shut containment building main purge valves POV 9 and 10 are isolated by locked-shut manual valves in series except during cold shutdown and refueling.

4.16 Station Personnel This section describes the personnel organizations at the San Onofre site involved in the November 21, 1985 event.

l 4.16.1 Administrative Organization The administrative organization of San Onofre is shown in Figure 4.14. Most of these departments are divided into those supporting Unit 1 and those who support 4-17

both Units 2 and 3. The site organization consists of approximately 1,500 permanent staff members and 1,000 contractor employees. They support the operation of all three units at the site. In maintenance, approximately 90 people work exclusively on Unit 1, 300 work on both Units 2 and 3, and 150 people divide their time among all three units. The plant Operations Depart-ment functions under routine conditions to operate all three units. Of the 300 plant operations personnel, approximately one-third are assigned to Unit 1.

The on-shif t organization is responsible for the immediate concerns involved in operating all three units.

4.16.2 On-Shift Organization The on-shift organization comprises several different groups working around-the-clock on the unit to produce electricity. Personnel from several other depart-ments, in addition to those from the operations staff, work on-shift. The on-shift work assignments routinely come from administrative supervisors, but all activities that affect the unit are coordinated through the Shift Supervisor.

l The Shift Superintendent is responsible to the Operations Department Plant Superintendent at Unit 1 for safe operation of the plant under all conditions.

3 To support this responsibility under routine conditions, the Shift Superinten-dent has authority over all personnel whose activities affect Unit 1. There are a minimum of six on-shift operators, in accordance with plant technical specifications and site procedures, who report to the Shift Superintendent (see Figure 4.15).

The Shift Superintendent interacts with other organizations in support of his work. The Shift Superintendent communicates with the Technical Department through the Shift Technical Advisor (STA). The STA provides technical informa-tion or recommendations to assist the operating crew. The Shift Superintendent can also initiate on-shift maintenance without going through the administrative organization by issuing a Shift Supervisor's Accelerated Maintenance (SSAM) order. Finally, the Shift Supervisor exercises authority by assuming the duties of Emergency Coordinator during the initial stages of emergencies. The Shift Superintendent holds an SR0 license.

The next senior position on shif t is the Control Room Supervisor (CRS), who also holds an SR0 license and normally exercises the Control Room Command Function for the Shift Superintendent. This function is formally assigned to a single operator who is then responsible for exercising the necessary decision and command authority for overall activities or operations affecting either the safety of the general public, station personnel, or the plant. The Control Room Supervisor is required to remain in the control room, unless relieved, to maintain a broad perspective on operations by not becoming too involved with any single operational activity, and to assume the authority and duties of the Shift Superintendent in the Shift Superintendent's absence from the control room. The CRS is also expected to make a complete tour of the plant once during the shift. In an emergency, when the Shift Superintendent assumes the Control Room Command Function, the CRS becomes responsible for implementing the appro-priate procedures and for directing other operators.

The Control Operator (CO), the next senior position in the chain of command, holds a reactor operator (RO) license. The C0's primary responsibility is the 4-18

l safe and efficient operation of assigned equipment. For Unit 1, the C0 is responsible for all plant equipment. This responsibility is carried out through direction of licensed and non-licensed operators at the unit. Although all the other personnel on duty are expected to make plant tours each shift at some time during the shift, the CO is required to stay in the Control Room for the entire shift. For a plant trip or transient, the C0 takes control of the pri-mary and electrical control panels.

The Assistant Control Operator (ACO) is responsible for carrying out the direc-tions of the CO, but must also assist in directing the activities of licensed and nonlicensed operators at the unit. The AC0 holds an R0 licerse and remains in the control room, except for the required plant tour. During a trip or tran-sient, the AC0 takes control of the secondary control panels, nonessential loads, turbine generator shutdown and Emergency Plant Implementation Procedure (EPIP) duties.

The remaining two members on shift are nonlicensed Nuclear Plant Equipment Operators (NPE0s), who take directions from the Control Operators and spend most of their time outside the control room making inspection tours and keeping logs on equipment operations. The NPEOs are responsible for keeping the Control Operators aware of plant conditions. In the event of a plant trip or transient, NPE0s are required to contact the control room for directions.

The on-shift operators remain as a unit through shift rotations and training and consider themselves an operating team in running the plant. Five shifts are employed, with three on-shift around-the-clock, one in training, and one providing extra personnel to cover on-shift absences or additional support.

On-shift personnel are also supported with extra personnel. The Shift Technical Advisor (STA), who is available 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> per day, holds a SR0 license. The STA reports to the control room if a trip or transient occurs and advises the Shift Superintendent. Operator trainees, in addition to the shift staff, often fulfill shift responsibilities under the supervision of the on-shift operator. Also, off-shift operators are available to provide support beyond that within the capability of the on-shift personnel. Thus, on-shift personnel have regularly scheduled assistance and as-needed additional resources to support them.

In emergency conditions, including the declaration of an Emergency Plan Action Level, the Shift Superintendent becomes the Emergency Coordinator until relieved by a senior administrative manager: the Plant Superintendent, Operations Manager, or Station Manager. The Emergency Coordinator is responsible for notifications, evaluations, protective actions, event classification, exposure control, and emergency response coordination. When activiated, all organizations have repre-sentatives working directly for the Emergency Coordinator to provide any i.ucessary assistance.

4.16.3 Human Factors Affecting Operator Performance Several factors affect operator performance. These include the control room where they perform most of their actions, the training the operators receive, and the procedures that they follow. These subjects are addressed in the fol-lowing sections.

4-19

i-4.16.3.1 Control Room The Unit 1 control room is a relatively conventional design with center J-shaped console surrounded on three walls by vertical panels. The back of the control room has desks for operators and doors to the offices of the shift superintendent and the nuclear operators assistant. The back wall.of the control room has a glass window for viewing the control room from the adjacent Technical Support Center. Figure 4.16 shows the control room from the Shift Superintendent's office door and Figure 4.17 is the floor plan for the control room and adjacent l offices showing major panels and controls significant to this event. The con-i trol room has not undergone any significant upgrades required by NRC's TMI Action Plan. The site's detailed control room design review plans to upgrade the con-

't trol room were scheduled for submittal to the NRC in December 1985.

l The panels showing indications that played a significant role in the event are shown in the section addressing personna performance, Section 7, Figures 7.2, 7.3, and 7.4. Figure 7.1 shows the - antation of these specific panels within i the control room. For a detailed discussion of control room indications and data recording systems, see Section 4.17.

I 4.16.3.2 Training Qualification at each position of the shift crew includes completion of specific individual training and on-the-job experience. Several positions also require

, classroom training. The NRC's R0 and SRO 'icensing programs are included in j the licensee's training program, but are supplemented by additional training and experience.

}

Operators are trained for a week annually on the simulator at Zion Nuclear '.

Power Station. Although the Zion simulator is not identical to the Unit 1

] control room, it is reprogrammed to model Unit I characteristics and behavior for this training. The significant advantage of simulator training is that each shift trains as a team and practices the teamwork essential to responding to trips and transients. An STA is included in the shift's simulator training, while trainees or other personnel are not.

4.16.3.3 Procedures The emergency operating procedures at Unit 1 are based on the NRC-approved ga-neric Westinghouse Owner's Group technical guidelines. Since the TMI accident, a significant shift in the philosophy of operations during emergencies has oc-J curred in the nuclear industry. Previously, procedures were written based on

! two assumptions: (1) that one event would cause an emergency and,-(2) that l operators could quickly diagnose the emergency's cause. TMI demonstrated viv-idly that both assumptions could be wrong. Since then, the industry has concen-

trated on writing procedures to focus on maintaining functions necessary to plant safety, rather than those focusing on specific events. This new function or symptom orientation, in its pure form, is not viewed as efficient for address-

, ing well-understood minor upset conditions. Therefore, hybrid approaches have i been developed, among which is the Westinghouse Owners Group (WOG) technical guidelines. The Unit 1 procedures employ the WOG approach of attempting to respond to specific events with " optimal recovery guidelines" while requiring that checks be made on the status of a set of " critical safety functions."

l l 4-20

The emergency procedures, beginning with " Reactor Trip or Safety Injection" procedure 501-1.0-10, are implemented when the reactor trips or should have tripped, when a safety injection occurs, or if there is a loss of electrical power. The procedure requires operators to verify proper system trips.for the reactor and turbine, to check electrical power available, and to check SI. If these checks identify abnormalities with the reactor-trip or electrical power, the operator is then directed to use the emergency operating instructions (E0Is) for anticipated transients without scram or loss of all AC power, as appropriate.

If SI actuated, the procedures direct operators to ensure the line up of neces-sary systems and then perform diagnostic checks and plant stabilization until SI can be terminated. If SI did not actuate, the operators are directed to follow the routine trip response procedure. The " Reactor Trip Response E0I,"

i procedure 501-1.0-11, provides guidance on achieving initial and then long-term plant stability.

Simultaneous with the use of these procedures, operators evaluate the Critical Safety Functions Status Trees (procedure 501-1.' 0-1) . The procedure sets and directs priorities for subcriticality, core cooling, heat sink availability, reactor coolant system integrity, containment building integrity, and reactor coolant system inventory. Depending on the status and priority of these, operators are directed to procedures designed to restore the necessary critical safety functions. The evaluation of the critical function status trees is typically performed by the STA, while the operators are performing the other E0Is. The STA periodically reports the status trees status to the Shift Superintendent as an independent check of the safety condition and success of operator actions.

4.17 Control Room Indication The majority of the plant's instrumentation and control power supply require-ments are provided by the 120V ac buses. Many indicators, recorders and meters in the control room are dependent on these buses. Two of thase buses, vital bus 4 and the utility bus, are supplied by the plant'ac distribution system and can lose power under total loss of ac power conditions. On loss of power to these buses, the control room indicators, recorders and'other instru-mentation that are s pendent on these busec, fail or w ivst into different states. A detailed list of the control room componer U sIfected and the state of their failure are provided below.

4.17.1 Loss of Vital Bus 4 The following is the list of indicators and recorders which are supplied power from vital bus 4 and their status upon loss of bus power. Control room I

instruments off this bus are identified by a green dot on the instrument.

Status After Instrument L.oss of Power Indicators:

Vital bus 4 voltage indication light off Analog rod position indication system off Pressurizer pressure indicator PI 434 fails low RCP seal water return temperature indicators fails Icv TI 1116 A, B & C 4-21

' Letdown TI 1103 8 . fails low Letdown flow FI'1104 fails low Charging flow FI 1112 fails low Charging line temperature-TI-1103 A fails low ARMS channels R 1231, R 1232, R 1233, alarms R 1234, R 1235, R 1236 & R 1237 ORho channels R 1211, R 1212, R 1214, alarn R 1215, R 1216, R 1217 & R 1218 ERMS channels R 1250 R 1251, & R 1252 alarms Stack iodine monitoring channels R 1219, alarms R 1220 & R 1221 RCP seal water return cross-tie to off; valves VCT valves CV 410 & 411 indications fail closed Recorders:

Pressurizer level LR 430 fails low Pressurizer pressure '(narrow range) PR 430 fails low Pressurizer pressure (wide range) PR 425 fails low Pressurizer liquid and vapor TR 430 fails low VCT level LR 1100 fails low RCS loop Tave TR 401 fails low RCS cold leg TR 402 chart drive stops; indication good Tave/ Tref TR 405 fails low SGs steam flow, feed flow, level fails as-is YR 456, 457 & 458 Steam and feedwater pressure R 8 chart drive stops; indication. good Condenser backpressure PR 480 fails low Radiation monitor RLR 1200 faf1s low RCPs bearing temp TRC 446- fails to print / advance Charging and SI pump bearing temp fails to print / advance TRC 1119 Component cooling water heat exchanger chart drive stops; outlet temp TR 606 indication good 4.17.2 Loss of Utility Bus The following is the list of indicators and recorders which are supplied power from the utility bus and their status upon loss of bus power. Control room instruments off this bus are identified by a yellow det on the instrument.

Status After Instrument Loss of Power Indicators:

Utility bus voltage indicator light off Pressurizer power relief line temp fails low TIC 433A

• Pressurizer safety line temp fails low TIC 433 B & C Pressurizer spray valves PCV 430C&H off; valve indicating lights functional 4-22 4

RCP seal water return valves off; valves PCV 1115 A, B & C indicating lights fail open RCP seal water valve CV 291 off; valve indicating lights fails ocen RCPs seal bypass valve CV 276 fails; valve indicating lights fails closed

, Letdown control valve LCV 1112 off; valve indicating lights fails open Letdown orifice isolation valves off; valves CV 202, 203 & 204 indications fail closed Charging isolation valves off; valves CV 304 & 305 indications fail closed Excess letdown isolation valve offs; valve CV 287 indication light fails closed Pressurizer relief tank vent valves off; valves CV 542 & 543 indications fail closed Steam dump to condenser valves off; valves CV 3 & 4 indications functional Recorders-RCP seal leakoff FR 1117 and 1118 fails low RCP seal water supply temp chart drive stops; TR 1111 indication continues Turbine metal temperatures fails to print / advance R 2 and R 3 Misc bearing temperatures fails to print / advance R 4 and R 4A Generator and transformer temp fails to print / advance R5 Generator gas and exciter temp fails to print / advance R6 Feedwater pump suction pressure temp. fails as is; and temperature R7 press. fails high Average feedwater temp fails as is; chart TR 456 drive fails Condensate flow FR 480 fails low RCP vibration detectors fails. low 4.17.3 Data Recording Systems The function of data and event recording is provided by control room data re-corders, a control room event recorder, a critical function monitoring system, and the oscillograph system. All of these, except the oscillograph system, are routinely used for diagnosing the causes of unscheduled reactor trips and to ensure that safety-related equipment functioned properly. The oscillograph j system records transients on the electrical distribution system immediately i following a system disturbance and provided valuable information on this event.  ;

The control room chart a orders at Unit I are the common analog recording devices for plotting ina./idual or small groups of parameters against time.

Several types of chart recorders are used with different power supplies for the chart drive and pens. Some recorders are powered from uninterruptible inverter-powered 120 VAC vital buses, while the other recorders are not. l l

l 4-23

i The control room event recorder is a 40-track discrete event recorder that i

accurately records the trip and reset of 38 trip bistables associated with the i reactor and turbine generator. Of the other two tracks, one indicates the recorder's speed, while the second is a spare. The power supplies for tho

, event recorder are from safety related sources. The paper drive for the

'. recorder normally operates at 3/4 inches pe- hour, but on a trip it switches to

[ a speed that corresponds to 3/4 inches per second. The event recorder was

! replaced in 1984 after 14 years of service.

The critical function monitoring system consists of a small general purpose computer, the FOX III, providing on-line display of selected plant parameters in different formats; it has a xenon prediction capability. Its periodic recording of plant parameters is also available for review after any trip.

The i system has hard-copy printers and CRT displays. One CRT, located near the i window in the Technical Support Center (TSC), is visible from the control room. ,

l

) The system's computer and all but one of the printers are located in the TSC l i adjacent to the control room. The other printer is located in the Emergency Operations Facility (EOF). Under normal conditions, the system records the current value of all parameters each minute and maintains this information for j 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. Upon sensing a trip, the system records another 5 minutes and then '

automatically prints out data for the 25 minutes prior to the trip and 5 minutes I following. The system can also track and print specific parameters on user

! request at 1 second intervals. The normal power supply for the system is the j MCC IC bus powered from the 4kV 1C bus; the central processor has an internal battery backup to perserve main memory for short power outages.

To support analysis of electrical transients, the oscillograph system records

) 29 data points of the electrical system. The system is not normally considered i

part'of the plant's event recording systems because it primarily deals with the i

output of the reactor plant's turbine generator. This system records phase-to-phase and neutral voltages for the 4kV buses and the turbine generator, as well 1

as the position )f the control valve, stop valve, and reactor scram breakers.

' The response time of the system is designed to record electrical faults and, therefore, the system is fast enough that the tracings show actual voltage sine waves associated with the 60-hertz bus frequency. The oscillograph system is not normally recording but when one of ten initiators occurs, the system records its sensor data in sequence. The trips are (1-5) undervoltage setpoints on the 4kV buses and the generator, (6) over voltage at the generator, (7) generator frequency out of normal band, (8) generator stator ground, (9) main transformer

neutral current, and (10) a manual control switch. The device records for a fixed time, then calibrates all channels with a standard signal, and resets for another initiation.

4.18 NRC Emeroency Response Organization When immediate NRC notification is required, the first person called at NRC is the Headquarters Duty Officer. Several state and local organizations are also kept informed of events at the plant. Because these agencies usually require notification prior to the NRC, the NRC's notification may be up to I hour or 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> after an event, depending on the plant's classification of the event.

Calls are usually made to the NRC using the dedicated-line Emergency Notifica-tion System (ENS). Phones are installed at Unit 1 in the control room, the Shift Superintendent's office and two nearby offices (see Figure 4.17).

4-24

The NRC Duty Officer serves as the initial NRC notification channel to the plant for minor events through emergencies. The Headquarters Duty Officers stand 12-hour shifts and control the ENS. They do not have detailed knowledge of each plant reactor but have general training on each reactor type. The initial notification is passed by the Headquarters Duty Officer to the appropriate NRC Regional Duty Officer and the region initially has the lead responsibility for the NRC's response.

The Regional Duty Officer notifies the appropriate regional official. The level i

of response is based in part on the plant's classification of the event in accor-dance with their NRC-approved and tested emergency plan. -Based on an evaluation of plant conditions and adequacy of the licensee response, the Regional Adminis-trator, in consultation with the Headquarters Emergency Officer, recommends

! actions necessary to place NRC in the appropriate response mode. The region has the re,sponsibility to activate NRC's formal response organizations when a licensee declares an " Alert" by placing NRC in " standby" status. This response level activates the regional and headquarters incident response centers but the j region retains the lead responsibility. Further levels of response are imple-mented based on guidelines described in NUREG-0845, " Agency Procedures for the NRC Incident Response Plan,"

4 i

~

i i I

\

4 1

l 4-25

- 5 s -

I"-  ;

7tu

',m u

= ;A-a-- 2 __

I g We , ".

p h n " ,- _

J"i "o

m. V J".

". h ,

"i m ". h n =

E mV N.

m m e

. k Te s

m t s

E' y

.v h N

.v E

S m

a

= t e

S

.h

_- s a i n

a x__ x,

_ M x_

. k s a

1 4

e r

u

.v h r i m ig

@e F

~ - I ii, t a t a _

'r I'  !' I' 1

6,'r L

- ng nw p* @s. 1I i

Iil 1 m

r E u

0 GG O0 t IIR I

T A

NNS H N K AC L

NE LTN UIS sA CO OAT R IR TW CE T

0 A OA T

R W K

. N A .

T i/ k uJ OO I

M O

D W

O L n B w C o F

x -

d w

l o

B

_ X N r o

_ t 1

2 5

ar e

_ x e n

HE GV L

_ " 'x

,c S2 G IE HL 8 _ Um W1 F 5 m

NN1 O

sAIA ET O 2 i,.

1

'L A a e

oD I t

t CA R A-

- S A

t GL i e 2 NL IA v TI C C 4 AT LU UO E

- e r

C RR u IE CT g O A i

TW F g

• ' ll*l I. * .

!gsa +Eg3E8 *I R

N T O M R O

W TI S WDN O Nl0 won OlO l

R OAS O AII O W! e.

I WE g T

A R

E O*e LI Ba1 OS0 aA c

y L OMT X B

O o

l NT t

B m

0 1

x N T 0 T 0 0A O a E 4 A T Q s

G R R A i m R M

A E

T S - ~ -

G G I

I I G N

I I R I w

I -

oE Of l

OII W f

D 5 l A

E F

D E

E F

V e f

F

,d"

!=  !- F n a a C 5 5 5 s a a a a a I I I I I I s a a a a E E 1 E E E

! i  : i i i i -

i i E

2 i

c-2

.g I  ! $  ! n 8

Dx! ml 2I s 7I ll r"'

i' lllr01-ii iQ,;

ee se 'ii

&XIEH ;! Bli ;! M!

@ ,!!-l mli ll 2 I

4 I

i l 11 j!n mi ,,il ,il 4-28

Il! 11!

=in ein i k h5$ Ak I. , :h , I.

x -

x O, --

=

\,fg_!

A i-  ! ,

E-s  ! r . l j j a__ l 1 el __s y

'

• Rg i Rg - p Is iv ,,g la

,A ,

A i__  ;; = s o o, = ::.. __i

. a i: 7 '

% gw 3 e!i t

gg 5s

, . . i. . . .E 1  !- ii iv .

li.  !, 1 i s i

. - l i 3

.g ig it ' a

!! 11* , .l Y l(  !! = /Q 9  !! l(

o __ N/ o __

l - l

.- .- ic

1 1 I T T T l

• = p I r -

y - p 4- 4. 4. 4.

W

( =( "

! ( -

( -

e e a s 1

, 4-29 t

. - . . . _ . ~ _ . . . - , . . . . _ . - , . . . - , _ _ , . _ _ . _ . . . . . - . - . . . . - _ _ _ _ _ . _ _ _ , _ - . _ _ _ . - _ _ . _ , , _ _ . . . . _ _

C-) c #-

) c -3 ln a---- -=

a-------___-----_u.--__

ii8 E5 5 e

J.JI i

5 5 5 lise:i lie:cti lim:e:1 blt blt Et Et e- e- e- E e

05 ., 2 _

,, e 2

/ "' ie o I L n w .

n

-o ,

(  !

> > li =

' ., m

. 1.

• 5  != e

• I

=

s -

2 1  :: ):

n n ill Il 3 !hl1 i

gg s y sa -

a i i! = w  ::

if X I

-cil 1

= 'l

?I -

/\

11 4-30

, , , , 11 J b i k 5

_sy *!

g- ~

l I

$: M  :

il i s ga _x_ga Age Als R

T I I I I I I I

+ 1 l + + 1 I &

n

!E I I 5 i Hl l!$  !

3 $l $ +H ll2E. $a E

I I, ,I I, i I I I i I

I. ,_ o o

h h p

e i; 1

]:

19 [l I! ]

I a + l 000W00 I I I

@ l-- i a ,

m 1  :.: :.: :.: I" I' TTT - N.

Ji EE :9 g .

i _

,g lN g

p g

- [, -

~

p g

Qy

' 0-X i d) noi

l El- @" ,l q.

o i*i l.l - .!) .-

o Q E!

g,._

4-33

A b$ ~ "I I ""l' is gl5 ll ;r .

EI I- :3:

e "5 ejG

~~ {

li h

$E a e sn -

O

,5

- 5, En ik z e

er 3

?.

t

~%- e 25 eg @

5 s'l i s.

?

Sb  !!

si Elux q (

?-

Esm U

ISW', l' 4-34

si is is il il il ,s

s N:In Xt ei:la Zt H:In Zt ei:11 es:II el :li I ~i e ~5 I - i l[i n e@H$s:l 8

_I!!a5 8 @ @ 9:9 _,

s n

a e

n a g =[I !

u Elea .i -

Mie= 'I -

z; ir! <

e, B~: z z i a

• r

-hn I{

s a

~

g 3 G.  ; Dj 3  !

! .I ..ar ., ! .im .i i  :

-, r_.  ;,: "b ,

y. y.

3 e :e:

I!

.h Ell ai

_h 9 G) G) l Xs ya s.!

!!l b 1[I 4-35

t I

i SANTIAGO #1 JL 220 KV NORTH EAST BUS f s' 3' )

s' CONTINUE 0 p p TO SONGS 2 tr 3 IM I4 h h 4022 5 4032 1 est! 3 a *: O P I4 n

= ; I4o I4 n

(

3 p p I4 IMh I4 n

6 I mit 4 es22 8 esu p D p (4 n (40 (4"ECI 0

220 MV NORTH WEST BJS ll-

'i H il H F '

'i Hil H F llCHTNING ARRESTOR 18ANI TRANSFORMER NVVW WW AUX.TRAN1 "A* AUX TRANS. "s" AUX. TRJWIL T NVVW NVVW N\/ VW u WW WW WW mAm samRAt0R it if if 1f TO TO TO TO 4100V 4tM V 4tM V 41epV eustA susis susic suste Figure 4.11220 KV System 4-36

04 S

8 l3

~!

• T

. m 8

, _ = = +T+-e - ,= =

-3 lII -

-4 T lhuffa"i"it 5i gyj  ; non;,.J , ;

, +T+-O =Ll"o'fa

'g*y ,

• J +74 jj'" "' I=U ,

g + A+ { ~4 T ? @'" ".?" ~*i* = "gi 5 = s 3 l

=

ggg-4^j" gg tasva.ns.m nv ans g

n -

e M .

4 ~$ .-yy. J

+^+ 5+ E

-e-T+J-W k

47M .NEm!Ym I. = = $

gi: .=: 4 .m ,

E&

. o E

e! mi V . '""8 "l'""

+ ^gM .c I >

. <<3.l'f* ,

( l g

5 = 9 E l 5, +; .

l  ! l >

! 4 34 5 4k; = @i.A"a" i =-

1

+T* u

= a mij s g.

w Nb b C g- a; = +yg ,,i,g.yg

.=

sE y&

~5 HD y : g u,- . .... . ma

--*-7 :

=

=, g 8

-4 ^B  ; ennse.s ,

sE5.g =

Ej 4I g,3! "

2 I

4^l * '

g.

es R s i o. n == .a o

• q 5s= ' $2 -9

, / ._ y v. > R S a. = == m

. g

-*-7 : {}- ..,  ; -*-7+-e - ,=,=

+r+- -*-7 % ....= =

s!j@

g, r,

3 5

4-37

hhLYtte. I '

u TO BE mctia - IE M E I - TO D E - MM I

nia ) n,a.T T I T T o T T "

n,=) nm) nm) => ) nm) n=>

T)nm)

-ma. .m.,

57 c=aa mm.n c=am.a 57 3 7 c.m.,

uaan vi -

nic c=ns.e

.T -

37 e

w co uww mu,e_a l

NWwT.w ..,an=.w w l w n.

s. m m -

~ 5 pn umsv .a s lE EE lE umar .a r Et

=

usar.s SunTCme0aa0 f%

l --- .- ._

+o o o o o c gfie.cc a 1),...

1),. .. . .. 3.<  !).. 3., 3.< ).<).<

l

_ 1,. . . . . . .

( Figure 4.13 120 V AC and 125 V DC Systems l

l i

i

lll I!! !!! !!!

I I I il

.?

l l

1ll

=

l a

j -

I

~

8 l

gl g

m I!

Il 4-39

T R R

ATE NO T I tWMA E E cUIPR uPUE P N QOE n _

i o -

t a _

i z _

n a

g r

T O N

• E D L R O L" T L" N O R f t

TN O OR A R O i

FE RMS I R TO T STN T h I

H TN I TOV NOR I T N A I A I S SOR I

S I ORE P O R n R C U C ~-T SCPE A

E P S 0 O O U

S 5 1

4 e

r u

g i

F T R R

ATE N BT O

I EWtA t

CUwR UPuE P N eOE O0 RR S"

i8

c f-wa_

~

-~ ' mj%.f m.T2:$~'-

. f

?

1? ~ I y J,R ~ - n. t.,h :;t

% d,.v.2 .

- ..-'< Q

' s )

e

._'. t M~

4._e'm.M~ W m $]'

g

.. i c

c. f J 5 f

,[ , .

f

. __ w r,I.

. e O

"u

~

,..

• i

. .,Y.%. J s i

. I mif~-

c. _ *L . Q^ . ',.

~. 't n

Q- .l .- I

~ e d

_r ... .

A4.;~3;.pk,  :.;W-e t

n

.D'

- rC .

T~- i n

r

% e h j.1 L a *.

~

S f

t p

u

, &g _, ,, h,,. 4_

i

- h

- J

, S D

i l m ,

rI 0 'K , _

o r

0 s - ,

F .

_ ( -

h- u-s ,u1 _

_ n1 r- m _

5 \ - o -

o aE =.

y. R

~

.' =j - l a 3I N. -

o r

i, t n

- 4 W o Q C 1

6,.

wd r-e-

,n ~:h- _ , S

,~, G z .- d2 N

.mi'4.g, %n. g

- ; e

+

, _.- ~

. i .

_a O cr ,a J

S i .w-l . , i.

.5. 3 i C,

~

~ .

.p9P.w(lad .

a 6

1

.w

~ .

7 '

4

- _ ~

c,,,g :> i .w e

\

L.p/)7;. ('  %

,,a :,

r u

g

(*4, p- - y.g n i i

^

,k F f !t! \!

lt yQ&. H. i. gh s; .

1. i.

yg&'s> ,

,h " ' -l 1

• Y Yl [ P b

+S

,l

,.3 fg g~' Y ^ 4 W

W AM T8 E

C H

_T L U8 B'

F E O O C

(

).

- p

_ 4  ; ,

Y C ..

ni EC a .

G .

R E

I s E

E.

t. a

- - "T_

Wm T XS T s OY FS E

N b e N c O u n S fi f

I T N

E O

B ., t m

E n

_ Tr e .

e

_ n c

_ E P

U j

a S d

'T T F

e A

S, t d n

[*- a

(*

• m fI ' o m

o

.u R

.T t l L Eu o

~ n.O e

TP t

r W"p.C T R C P . n W

E G

u o ER '

s o

. C I

R m e

m_

E s

i E

n 7 a 1 m A ls er ms l A T oL nN aP vA o

T T

U n

E s

u l a_

t s

c 4

e AT N tE e o Aed E4 bW T P 2 r Es Ps M10 TO E s

a s  ?

7 u

OA WT AL A IWU T

E R

U P

s o

s e_ ig SC 0 P r n 6 F

~ TN5 u a o L I

e-

'..e.[

C O

AT Y

A R

T T

s T

T i

s l s_

E u

TW I

A M Mu O iu

, eP U TU C a t.

I A0 E0 RC A g

l f DLA i

f s t

u WE W

O1f i w

a OA Ns n

S x L T Ea E

a HIN i U0 P 1 u

A A

P PT!

MU II D e E EE OG i

,TEe l fA c

C CAA T a

A f S Tt R S t COE A R i

E Do E

A u

E Et us Rn U R $

E R

P

},

R I

I I

I OA Co Es 4,

R T

E NI U

IB VD ,

TI T NC AE r d l

SW W

L"" '

't l+:., i4l aIA j j !l

5 LOSS OF POWER EVALUATION 5.1 Electrical System Operations Leading To Auxiliary Transformer C Differential Relay Trip On Wednesday, November 20, 1985, at 23:50, an alarm indicating a ground on 4160-V bus IC was received in the control room at SONGS Unit 1. The ground detector voltmeter indicated a 100 percent or full scale ground. In response to the alarm, the operators referred to Operating Instruction (01) S01-9-7, which deals with 4160-V and 480-V bus and feeder faults. The procedure first has operators attempt to locate and isolate the ground fault in the loads con-nected to the bus. This required operators to shift services to redundant equipment powered from sources other than bus 1C, and stop equipment powered from bus IC. For example. the north charging pump connected to bus 2C was started and the south charging pump on bus 1C was stopped; the operators veri-fied that the ground detector indication did not clear, and then restored the original equipment alignment by restarting the south charging pump and securing the north charging pump. In the case of service station transformers SST 1 and SST 3, alternate supply paths were established to energize the respective 480-V buses prior to disconnecting the SSTs. (Refer to Figures 4.11, 4.12, and 4.13 for details of the electrical distribution system.)

By 00:20, November 21, all lead circuits on bus 1C, except for that of the north circulating water pump and the ' west feedwater" pump,-had been tested;

~

however, the ground fault had not been located. Unit 1 was operating at 250 MWe at this time and the operators received permission to reduce load to 150 MWe in order to stop the west feedwater pump. One of the steps required by the procedure (501-9-7, step 6.2.7) is to check the fuses of the ground detector potential transformer (PT), because a blown fuse will cause a spurious ground fault indication. This step was to be performed if the ground on the bus could not be los nted by deenergizing all feeder circuits. In this instance, the operators proceeded to take this step prior to isolating the two remaining loads. The PT fuses were verified to be good and the PT was returned to service.

At 00:24 the unit was down to the desired 150 MWe. Around this time, the con-trol room operators closed the bus 1A-1C tie breaker 11C01, thus parallelling auxiliary transformers A and C in order to verify that the ground fault was valid and not just a problem with the bus IC ground detector. With the buses parallelled, the ground detector associated with auxiliary transformer A also indicated the fault, thus confirming the existence of the fault. The buses were parallelled for approximately 5 seconds this time.

At 01:30 the south circulating water pump on bus 2C was started (it had been secured earlier to enable a search for salt water leakage into the condenser),

and the north circulating pump on bus IC was stopped. The ground did not clear and the north pump was restarted. At 02:40 the south circulating pump t

was again stopped due to increasing chlorides in the steam generators. At approximately 02:45, an electrical technician was given anproval to check the ground detector on bus 1C. The technician measured the voltage across the 5-1

ground detector relay and confirmed t u voltage to be 215 volts (210 volts is nominally 100 percent on the ground oo ctor voltmeter), thus verifying that a ground fault existed and that the grouna detector was functioning properly.

The technician informed the Shift Superintendent of his finding. Following discussions with control room personnel, the technician checked the PT associated with the ground detector for ground. To do this, the tie breaker between bus 1C and bus 1A (11C01) was again closed.

At this point auxiliary transformer C and auxiliary transformer A were again operating in parallel, connected to the inter-tied buses 1C and 1A. The ground detectors of bus IC and of auxiliary transformer A both indicated the ground fault. The PT of bus 1C was then disconnected, but the ground was still indicated on the other detector. This test eliminated the PT as a possible location of the fault. The PT was returned to service. At this time, 03:35 the auxiliary transformer C source breaker to bus IC, 11C02, was opened by the i control room operator; bus 1C remained energized from bus IA, and the ground ,

indication on both indicators cleared. (In this instance, the two transformers  !

remained in parallel for approximately 2 minutes.) This action isolated the j ground from bus 1C and indicated that it was somewhere between the X winding of I auxiliary transformer C and its circuit breaker 11002. After further discus-sions between control room personnel and the technician, the technician checked the synchronizing PT of bus 1C to further isolate the location of the fault.

To do this, the source breaker 11C02 was reclosed, again paralleling auxiliary transformers A and C for approximately 5 minutes, and the synchronizing PT was disconnected. The fault did not clec . The PT was returned to service and the source breaker was re-opened from the control room isolating the fault. Having determined that the fault was not on bus IC and that the west feedwater pump would not have to be deenergized, at 04:00 the control room operators commenced increasing power and by 04:30 the unit was again at 250 MWe. Meanwhile, the electrical technician and crew were visually inspecting the electrical circuit between auxiliary transformer C and bus IC, the area around the transformer and the X winding's reactor and associated bypass breaker. They were preparing to rack out (remove) the reactor bypass electrical breaker after receiving permis-sion from the control roce.

At this point, the electrical distribution system status was as follows:

Auxiliary transformer C was energized and supplying 4160-V bus 2C; i.e., circuit breakers 4032, 6032 and 12C02 were closed, Generator at 250 MWe and breakers 4012 and 6012 were closed, Auxiliary transformer A was supplying 4160-V buses IA and IC; i.e.,

breakers 11A04 and 11C01 were closed, Auxiliary transformer 8 was supplying 4160-V bus 18, SST 3 and associated 480-V bus 3 were being supplied by 4160-V bus 2C through breaker 12C11, Lighting transformer was on normal supply off bus 1C, South circulating water pump had stopped and both feedwater pumps were running, 120-V ac utility bus was on alternate supply off the lighting switchboard, which in turn was being supplied by 4160-V bus IC, and

• All other electrical-system alignment was normal.

5-2

5.1.1 Relay Trip At 04:50 auxiliary transformer C tripped on actuation of transformer differen-tial relay. Later observation at the relay panel confirmed that the differen-tial trip involved phase B and phase C. Phases B and C trip targets nad dropped, indicating that a fault current in excess of approximately 1500 amperes was involved.

Control room operators received multiple alarms when auxiliary transformer C tripped, among them the turbine first-out alarm for auxiliary transformer C differential trip. Upon loss of the transformer, 4160-V bus 2C and all loads connected to it, including vital bus 4, were lost. Diesel generator 2 automati-cally started upon loss of voltage to bus 2C. The control room personnel observed the loss of vital bus 4 by the loss of its potential indicating light with the green dot on the vertical panel. Based upon this, the reactor was manually

! scrammed; the unit trip pushbutton was also manually actuated.

5.1.2 Discussion In responding to the ground indication on 4160-V bus IC, the operators correctly followed Operating Instruction 501-9-7 in attempting to locate the ground on the load circuits of all connected loads, up to the point which required them to de-energize the feedwater pump motor circuit.

The 01 did not instruct the operators-to use the ground detector on an unfaulted bus (bus 1A ground detector, in this case) to verify the authenticity of the detector on another bus which is indicating a fault. Nor did the OI direct operators to tie a faulty bus to a fur:ctioning bus as a means of transfering power or load. Yet the operators did close tie breaker 11001.three times when a ground fault was present on bus IC. When bus 1C and 1A were thus connected, auxiliary transformer A and auxiliary transformer C (X winding) were operating in parallel. The transformers were operated in parallel three times for periods ranging from 5 seconds to 5 minutes.

Operating two transformers in parallel supplying a faulty 4160-V bus is not appropriate for the following reasons:

1. The ground fault on the ungrounded delta-connected transformer secondary is being supplied by a neutral grounded wye-connected transformer secondary, thus providing a fault current path back to a source.

2. The 4160-V switchgear is not rated to handle phase-to phase or short-circuit faults should they occur while the buses are energized from the switchyard through two transformers in parallel. (This is the same reason why current-limiting reactors are installed in auxiliary transformer C leads while the diesel generators are run in parallel with the transformer.)

3. Feeding a fault from two sources will exacerbate the situation and could lead to the fault tripping both sources.

4. Parallel operation of transformers, especially ones with unequal impedances, would lead to circulating currents being set up between them.

5-3

OI S01-9-7 did not include specific warnings regarding parallel operation of transformers and other sources while a fault exists in the electrical distribu-tion system. (Momentary parallel operation between functioning sources should be permitted; however, where continued parallel operation is required, the system should be designed for it.)

5.2 Unit Trip and Loss of All AC Power On depressing the unit trip pushbutton, the following events occur.

the turbine trips the generator exciter field breaker opens the 4160-V bus IA source breaker 11A04 opens 4160-V bus 1B source breaker 11804 opens an auxiliary relay (UT-X) energizes which in turn trips generator output 220-KV breakers 4012 and 6012 (and energizes another auxiliary j relay for local breaker failure backup protection).

During the event, the unit trip pushbutton was depressed 19 seconds after j auxiliary transformer C tripped. At this point, when breakers 4012 and 6012 opened and power from the switchyard through auxiliary transformers A and 8 to 4

4160-V buses 1A, 18 and 1C was isolated, all ac power to the in plant ac distri-bution system was lost. Only the plant dc system and the inverter powered vital buses (all battery powered) remained operational. The utility bus was now without power. The 4160-V bus 1C undervoltage relays actuated and, with the existing loss of bus 2C, initiated generator lock-up and the loss of voltage auto transfer scheme. Both emergency diesel generators received a start signal and started (diesel generator 2 was already running from a previous signal).

] The control room operators were following Emergency Operating Instruction (E0I)

S01-1.0-10 for reactor trip or safety injection. Step 3 of this instruction requires verification that buses IC and 2C are energized. With these buses i

deenergized, the operators turned their attention to the loss of all ac power instruction, S01-1.0-60. Step 1 of this E0I requires verification of offsite power at the 220-KV switchyard, which the operators did by reviewing the-vol-tage indication in the control room. Step 2 requires a check for SI initiation on the reactor panel first-out annuciator window 2 (RPF0-2). The alarm window was lit, indicating SI actuation; however, this was a spurious alarm and the operators were aware that it would come in on loss of ac power. (See section 8.2 for an explanation of this spurious actuation.) Operators in the control room then checked the SLSS load group lights on the SLSS surveillance panels located on the vertical diesel generator boards and observed that they were not lit, indicating SI had initiated. (See section 8.3 for a discussion on the spurious indication of SLSS load group lights.) The operators then checked the status of pressurizer pressure and containment building pressure, determined that an SI was not warranted and concluded that SI had not initiated.

At step 3a of the E0I, the operators are expected to wait for the automatic operation of the loss of voltage auto transfer scheme to complete and the end-of-sequence light to light. However, the end-of-sequence light did not illuminate. An operator depressed the unit trip reset pushbutton, to enable

. closure of 220-KV circuit breakers 4012 and 6012, but the reset did not occur.

because this action was apparently taken before the generator 18-KV motor-operated disconnect (MOD) opened. The operator at the electrical board checked the-l status of the generator MOD, which was, by then, open as required, and'the 5-4

status of the source breakers to buses 1A and IB (11A04 and 11804) and the tie breakers of buses 1A-1C and 18-2C (11001 and 12C01), which, contrary to expectations, were not all closed. The E01 did not address this particular situation and the control room operators decided to recover power from the switchyard to feed the 4160-V buses one at a time. They attempted to close 220-KV circuit breaker 4012. The first attempt failed because the operator did not depress the synch check bypass pushbutton (see section 4.12.2.1 for a des-cription of closing operation of this breaker). The second and third attempts are believed to have failed because the unit trip reset had not reset the gener-ator lock-up, and breaker 4012 tripped free when the two attempts were made.

(These two attempted closures and trips of breaker 4012 are seen in the oscillograph traces.) After the failures in closing breaker 4012, the operator at the J console again pushed the unit reset button. The operator at the vertical panel tried to close breaker 6012. The first attempt failed because he did not depress the synch check bypass button. The second attempt was success- l ful, following the activation of the synch bypass button. The operator then l closed breaker 11A04 energizing buses 1A and IC (bus tie breaker 11001 was already closed), followed by closing 11804 and 12C01. Thus, four minutes af ter loss of all ac power, power was returned to the station distribution system.

The evaluation of the operator response to the unit trip, loss of all inplant ac power and spurious indication of safety injection actuation, is discussed in Section 7.

The operators were expecting the end-of-sequence light indicating the comple-tion of the auto transfer scheme 2 minutes after loss of ac power. However, based on a review of the scheme, the time required could be closer to 4 minutes.

The shooting actual of time required the transfer will have to be determined during the ongoing trouble-scheme.

The operators' lack of knowledge regarding the timing of this scheme points out inadequacies in SCE's previous test program and in the training operators received on the response characteristics of this system. I The difficulties experienced by the operators during their attempts to recover power from the 220-KV switchyard indicate inadequacies in.their training and understanding of the following:

1

1. Unit trip and generator lock-up reset.

2. 220-KV circuit breaker closing operation, with and without synchronizing check.

3. Loss of voltage auto transfer scheme.

Although both diesel generators were ready and available to supply emergency power to the station during this period, the operators did not use them since 220-KV offsite power was available. Had the operators been unsuccessful in closing either circuit breaker 4012 or 6012, they stated that they would have used the diesel generators to supply power to the station. The procedures are not clear as to when this is to be done.

Just prior to auxiliary transformer C differential trip, the electrical techni-cian was given permission to rack out (remove) the transformer X winding reac-tor bypass breaker. Had he done so, the control room operators would not have been able to close the output breaker of diesel generator 1 because of the elec-trical interlock between the two breakers. A review of the interlock circuit shows this to be a design deficiency.

5-5

m. ___- ._

6 WATER HAMMER EVALUATION This section discusses the water hammer which occurred at SONGS-1 on November 21, 1985, its underlying causes and the damage incurred. Since failed check valves in the feedwater piping were the underlying cause, this section also discusses valve maintenance and inservice testing related to these valves. To clarify the discussions that follow, a brief review of water hammer phenomena and com-monly accepted definitions are provided.

Hydraulic instabilities occur frequently in piping networks as a result of changes in fluid velocity or pressure. Some of the better understood occur-l rences include induced flow transients due to starting and stopping pumps, opening and closing valves, water filling voided (empty) lines, and pressure changes due to pipe breaks or ruptures. As a consequence of the change in fluid velocity or pressure, pressure waves are created which propagate through-out the fluid within the piping network and produce audible noise, line vibra-tions and, if sufficient energy transfer occurs between the pressure wave and the pressure boundary, structural damage to piping, piping supports and attached equipment. More specifically, this pressure transient is a fluid shock wave in which the pressure change is the result of the conversion of kinetic energy into pressure waves (compression waves) or the conversion of pressure into kinetic energy (rarefaction waves). Regardless of the underlying causes, this phenomena is generally referred to as water hammer. The more severe the out-come, the higher the likelihood is of the event being called a water hammer.

Water Hammer Definitions

1. " Classical water hammer" generally identifies a fluid shock, accompanied by noise, which results from a sudden, near instantaneous, stoppage of a moving fluid column. Unexpected valve closures, back flow against a check valve, and pump startup into voided lines where valves are closed downstream are common examples of underlying causes leading to " classical" water hammer and are generally well understood.

Analytical methods have been developed to predict loads for this type of fluid hammer and include the effects of initial pressure, fluid inertia, piping dimensions and layout, pipewall elasticity, fluid bulk modulus, valve operating characteristics (time to open or close), etc.

2. " Condensation-induced water hammer" results when cold water (such as auxiliary feedwater) comes in contact with steam. Conditions conducive to this type of water hammer are: an abundant steam source and a long empty horizontal pipe run being refilled slowly with cold water. The cold water will draw energy from the steam with the rate of energy transfer being governed by local flow conditions. As the steam condenses, addi-tional steam will flow countercurrent to the cold water, and as the pipe fills up (i.e., the void decreases) the steam velocity will increase, setting up waves on the surface of the water, eventually entraining water and causing slug flow. Slug flow will entrap steam pockets and promote 6-1

significant heat transfer between the steam and colder water. Figure 6.1 illustrates (in simplified form) the flow conditions which would come about during the refilling of a voided horizontal feedwater line. Once slug flow conditions commence, a steam pocket will suddenly condense, creating a localized depressurization instantaneously. The resulting pressure imbalance across the slug (approximately 700 psi at SONGS-1) causes the slug to accelerate away from the source of pressure and toward the region of condensation.

Condensation is extremely rapid and predicting the exact location is im-possible. When the water slug suddenly strikes water in a previously

, filled pipe, it produces a traveling pressure wave which imposes loads of l the magnitude that would be induced by " Classical" water hammer in the piping network. This phenomenon, called " condensation-induced" water hammer, occurred at SONGS-1.

Predicting loads associated with this type of water hammer is extremely difficult because of the interactive and complex hydrodynamic and heat transfer phenomena which proceed the sudden condensation. Void fraction (or how empty the pipe is) and subcooling (or how much colder the water is than the saturation temperature of the steam when steam and water come in contact) are two important parameters currently used in models for pre-dicting this type of water hammer occurrence and its associated loads.

3. " Steam generator water hammer" (SGWH) is a condensation-induced water hammer which has occurred principally in pressurized water reactors (PWRs) l with steam generators having a top feedring for feedwater injection. The l

underlying causes are similar to those discussed above (e.g., voiding of l the horizontal feedring and feedwater piping immediately adjacent to the steam generator and subsequent injection of cold water). Damage from SGWH has generally been confined to the feedring and its supports and to the steam generator feedwater nozzle region. However, damage to feedwater line snubbers and supports has also occurred at other plants. A SGWH resulted in a fractured weld in a feedwater line at Indian Point Nuclear Power Plant, Unit 2 in 1972.

6.1 Plant Conditions Leading to Water Hammer Plant conditions at 50NGS-1 which led to a steam condensation-induced water hammer included the voiding of long horizontal lengths of feedwater lines (which allowed for a backflow of steam from all steam generators before opera-tors isolated the FW lines by closing MOVs-20, 21, and 22) and the subsequent refilling of the FW lines with relatively cold (i.e., less than 100 F) AFW.

Table 3.1 highlights approximate times and events which are pertinent to esta-blishing water hammer conditions and which are referred to later in this dis-cussion. Figures 4.3, 4.4, 4.5, G.2, 6.3, and 6.4 describe the flow circuits, valves and other equipment affected by this water hammer.

Upon detection of the fault on C auxiliary transformer, relay protection de-energized 4KV bus 2C, de-energizing the east-side main feedwater (MFW) pump FWS-G-3A (Table 3.1). The continued. operation of west-side MFW pump FWS-G-3A, due to the unusual electrical alignment, plus the failure of the east-side MFW pump discharge check valve FWS-438 to seat, resulted in the overpressurization and failure of the east flash evaporator tube and shell side. The subsequent 6-2

unit trip de energized the west-side MFW pump and denied power to the electric-driven auxiliary feedwater (AFW) pump AFW-G-10S. With the cessation of flow to the steam generators, the failure of check valves FWS-438 and FWS-439, and the failure of the three check valves in all the feedwater control circuits (valves FWS-346, FWS-345, and FWS-338), a path was provided to blowdown all three steam generators through their respective feedwater lines to the atmosphere through the failed flash evaporator.

Tle; drop in the steam generator water level following the unit trip initiated tts AFW system, but the electric pump was de-energized and the steam-driven AFW pump AFW-G-10 took 3 minutes to deliver flow, because of a programmed warmup period for the turbine. Thus, for 3 to 4 minutes no flow was being provided to the steam generators and the leaking check valves permitted the horizontal feedwater lines to void. Further, the initiation of AFW flow at about 135 gpm i

from the steam-driven pump was not effective in halting the voiding, because

! flow was being carried away from the steam generator by the steam blowing down l from the steam generators, through the failed check valves in all three FW con-trol stations and out the leak in the flash evaporator.

Figure 6.5a presents data from temperature recorder TR-456, which records the temperature of fluid in the feedwater line in the common header just upstream of the FW flow control stations. The recorder and pen failed due to the loss of station power but, following re-energization, indicated a sharp rise in temperature (indicative of the presence of steam in the FW line), followed by a rapid drop in temperature (indicative of the injection and mixing of AFW in the fluid passing the sensor), followed by a tailing off of temperature (indicative of the closure of the motor operated isolation valves). In addition, tempera-

, ture recorder TR-96 (Figure 6.5b) data from sensors upstream of the east and west MFW pump suction locations demonstrated a similar temperature increase to about 385 F and then a drop in temperature. In order to have the rise shown by TR-96 (which is about 20 feet lower than TR-456 and has an intervening volume of about 2500 gallons of 300 to 365 F water) a blowdown of a significant portion of the feedwater lines upstream of the FW regulating valves must have

occurred.

Following restoration of unit power, the motor-driven AFW pump (AFW G-105) started automatically, increasing the indicated AFW flow rate to a preset rate of 155 gpm per steam generator. However, all three steam generator levels continued to drop since the FW check valves remained open, the main steam sys-tem had not been isolated, and steam generator blowdown had not been isolated.

Subsequently, operators, following an emergency operating procedure for reactor trip response, isolated the failed FW check valves by shutting the three FW control isolation valves, MOV-20, 21, and 22 at approximately 04:55. Isolation of the.feedwater trains occurred before the water hammer in loop B.

Subsequent to the isolation of the main FW loops, and recognition in the control room that both AFW pumps were delivering water, the operators become concerned 4

for the over-cooling of the primary reactor coolant system and the decrease in pressurizer (PZR) level, at which time operators decreased AFW flow from 155 gpm to zero, and then back up to 40 gpm. The estimated time for AFW flow reduction is 05:00. Thus, refilling the FW lines downstream of the flow control stations was halted and then resumed at a much lower flow rate. This decrease in AFW flow rate adversely influenced the pipe refilling characteristics, particularly in Loop B.

6-3

The slow refilling of the FW lines within the containment building continued from approximately 05:00, when AFW flow was first throttled back to when the water hammer was reported to have occurred at 05:07 by a plant equipment operator. As noted previously, conditions conducive to steam-condensation water hammer in the feedwater lines were present for quite some time. The gross failure of upstream check valves which permitted water to drain from the feedwater lines and be replaced with steam was the underlying cause for water hammer. Leaky chack valves have been previously cited in reports of other water hammer occurrences. Five check valves are known to have been failed at SONGS-1 on November 21.

Estimates of feedwater line voiding and refilling are discussed in Acpendix B.

Figure 6.6 illustrates hypothesized refilling conditions in loop B prior to the water hammer. The much larger voided volume of loop B and manual control of AFW injection following closure of M0Vs-20, -21, and -22 (p wticularly total stoppage of AFW) were major factors in establishing conditions which triggered this water hammer. Appendix B discusses the refill aspects in more detail and j also provides estimates of pipe void conditions when water hammer occurred and l I

provides estimates of water hammer loads based on damage done to piping supports.

Post-event water hammer load calculations estimate that a 1-2 percent void I existed when the steam pocket collapsed; the probable location of that steam pocket is that shown in Figure 6.6.

6.2 Water Hammer-Induced Damage The next four sections detail water hammer-induced damage to Loop B feedwater piping and supports, the FW Loop B flow control station, and to Loop B auxiliary feedwater (AFW) piping and feedwater system check valves.

6.2.1 Piping and Piping Support Damage Damage to the Loop B FW piping was confined to plastic yielding of the NE elbow and to a visible crack on the outside of the pipe, extending approximately 80 inches axially. The crack penetrates approximately 30 percent of the pipe wall at its deepest point from the outside and approximately 25 percent on average.

Damage to supports, in some instances, was severe. This section provides a narrative and pictorial description of the damage visible after the FW piping insulation was removed.

Figure 6.7 shows the FW Loop B piping layout and identifies the piping support stations where damage occurred. This figure also provides directional orienta-tion and indicates piping dimensions. Figure 6.8 shows principal areas of damage, indicates how the pipe moved, and highlights the narrative and picto-rial "walkdown" which follows. Table 6.1 provides a narrative description of the piping and sup ort station damage shown in Figures 6.9 through 6.30, and identifies those support stations.

The water hammer forces were sufficiently large to damage pipe supports and piping and to transmit loads through the containment building penetration structure outward to the B Loop feedwater regulating station. No damage was evident to the steam generator B feedring or nozzle region that can be attri-buted to water hammer, nor was there evident damage or movement to the piping between support H00C and the steam generator B feedwater nozzle (Figure 6.9).

6-4 l

6.2.2 Feedwater Loop 8 Flow Control Station Damage The water hammer originating in the feedwater line within the containment build-ing generated Loop a water B flow control slug which transmitted a pressure wave upstream to the station. Check valves FWS-346 and FWS-378, downstream of the control valves, were designed to prevent backflow, although post-event in-spection revealed that the closure disk for FWS-346 was laying in the bottom of the valve chamber. Thus, any closed valve would be subjected to the water hammer loads.

In addition to check valve FWS-378, the flow control valve (FCV)

FCV-457 and motor operated valve MOV-20 were subjected to the water hammer loads, because they Operating had been(E01).

Instructions closed by operators following procedures in the Emergency Figure 6.30 shows steam generator 8 feedwater piping and valving at the Loop B control station (outside the containment building). Figures 6.31 through 6.34 illustrate the piping and valve conditions found after the occurrence of water hammer.

The 2-inch bleed bypass line suffered minor damage. Figure 6.35 illustrates the relative positions of the main FW (10-inch line) and FW bypass (4-inch line) flow lines and the damaged check valve locations.

Because check valve FWS-378 was intact and operational, it was subjected to water hammer loads and absorbed much of the water hammer energy whereupon the bonnet studs yielded and the gasket was forced outward against the studs (Fig-ures 6.36 and 6.37). The failure of the gasket relieved much of the internal pressure, tion. thereby minimizing damage to other equipment and valves at this sta-However, FCV-457 did incur damage to the flow actuator yoke, as shown in Figures 6.38 and 6.39; and disassembly revealed a bent valve stem.

Further discussions and illustrations of valve conditions (which were found upon disassembly following this event) are provided in Section 6.2.4.

6.2.3 AFW Piping Damage The AFW injection point to the main feedwater piping at SONGS-1 occurs in the "Figures breezeway" upstream 6.40 and 6.41. of the containment building steel shell, as shown in tie into the main feedwater (MFW) line.The AFW lines run horizontally and then verticall containment building penetration function. Figure 6.40 also shows the Loop B Water hammer loads were imposed on AFW Loop B piping, as is evident from the pipe motion recorded in Figures 6.42 through 6.45. Although pipe movement extended several hundred feet upstream, there was no evidence of piping damage.

6.2.4 Valve Malfunctions and Damage Post event disassembly and examination of valves that contributed to water hammer conditions confirmed that check. valve failures were the underlying causes(s) for the occurrence of water hbmmer. Inspection findings identified the following valve conditions:

Valve Description As-Found FWS-345 MFW Reg Check Disc separated from hinge SG A arm, disc stud broken (threaded portion).

6-5

Valve Description As-Found FWS-346 MFW Reg Check Disc separated from hinge SG B arm, disc stud deformed.

FWS-398 MFW Reg Check Disc nut loose.- Disc SG C partially open. Disc caught inside of seat ring.

FWS-438 FWP Discharge Check Disc nut loose. Disc partially open. Disc caught on inside of seat ring.

FWS-439 FWP Discharge Check Disc nut loose. Disc partially open. Anti-rotation lug lodged under hinge arm.

The following observations were common to the MFW regulator check valves:

- 1. Severe wear was evident under the hinge arm; the wear spots were bright. The wear was caused by the disc anti-rotation lugs rotating  ;

against the hinge arm. l

2. The surface around the hinge arm stem hole contacting the disc nut )

was recessed, as if the nut had been ground into the hinge arm forming " ball and socket"~like surfaces.

3. The underneath surface of the bonnet stop was heavily hammered.

4. There is evidence that the disc nut was pinned to the disc.

5. All the valves had anti-rotation lugs welded to the disc.

The following observations were common to the main feedpump discharge check valves (FWS-438 and FWS-439):

1. The disc nuts were loose.

2. The discs were partially open.

3. The disc nuts and washers were in place.

4. Neither the disc nut nor the disc stud was drilled to' accommodate the locking pin.

5. There is no evidence,on the disc stud of its being hammered against the bonnet stop.

The following sections illustrate valve damage.

6.2.4.1 Swing Check Valve FWS-346 This 10-inch swing check valve in feedw te' line B'is the first backflow barrier between SG B and the Loop B flow control .tation. Figure 6.46. illustrates a typical flange-end swing check valve design of the type extensively used in the 50NGS-1 feedwater system.

Disassembly of FWS-346 revealed ~that the closure disk was laying in the bottom of-the cavity, as shown in Figure 6.47. Thus, there was no back flow barrier 6-6

to prevent drainback until the operators manually closed valves MOV-20 and FCV-457 at approximately 9 minutes into the transient.

Figures 6.48 through 6.54 show FWS-346 components and the valve body cavity.

The seat surface of the disc face did not exhibit noticeable damage (see Fig-ure 6.49), although there were two cuts (on the mating face in the valve body) as seen in Figure 6.48. Damage to the disc nut pin is shown in Figure 6.50.and the worn hinge pin hole is shown in Figure 6.51. SCE personnel stated that the nut had previously been pinned and evidently worked loose over a period of time. The worn pin hole and damaged disc stud are evidence of continual opera-tion with a loosened or lost nut. Figure 6.52 and 6.53 show the bottom side of the bonnet and the' disc stop cast into the bonnet. Damage caused by continual impact during operation is evident. Figure 6.54 is a composite photograph of the FWS-346 valve cavity, looking towards SG B. Scratch marks at the bottom support the hypothesis that the disc had been laying free in the bottom of the valve bodf for some time.

As noted above, post-event inspections of the check valves in feedwater loops A and C (FWS-345 and FWS-398) also revealed a failed condition.

6.2.4.2 Swing Check Valve FWS-378 FWS-378, a 4-inch swing check valve of similar design to FWS-346, is the valve location at which the gasket failed. Figures 6.55 through 6.58 show the valve after disassembly. It did not appear to be damaged internally as a result of the water hammer, although, the disc seating surface displayed multiple cracks across the sealing surface. The deformed gasket which failed and relieved water hammer pressure loads is shown in Figure 6.55. Figure 6.58 shows one of the elongated bonnet studs which had been stretched about 1/2 inch.

6.2.4.3 Swing Check Valves FWS-438 and FWS-439 FWS-438 and FWS-439 are 12-inch swing check valves located upstream of the east-side and west side FW pump, respectively. - The failure of FWS-438 to fully close is believed to have resulted in the overpressurization and rupture the i

east-side flash evaporator.

Figure 6.59 illustrates FWS-438 "as-assembled" and "as found" following post-event examination. The nut had loosened and the disk had rotated the anti-rotation " nub" under the hinge arm, thereby preventing the disc from returning to the full seal position. Figures 6.60 and 6.61 show the topside of the FWS-438 check disc. Evidently one nub had oeen subjected to impact for some time (Fig- .

ure 6.60). There was no provision for a locking pin 'in disc nut pin FWS-438.

Post-event inspection also revealed a similar failure of the west-side FW check valve (FWS-439).

6.3 NRC Evaluations of Water Hammer 6.3.1 History and Focus During the early 1970s, the NRC staff became awarc of the increasing frequency of water hammer occurrences in nuclear power plant systems and developed con-cerns for the potential challenge to system integrity and operability that 6-7

. , , , , - - - _ 3 -r_,-, y -t , , _ . +, - --e,+y-~ . .y-

could result from these incident's. For pressurized water reactors, the major contributor to the increased frequency of these incidents was a phenomena called steam generator water hammer (SGWH) (defined in the opening of this section). The significance of these events varied from plant to plant; how-4 ever, NRC was concerned that a severe SGWH might develop that would cause a complete loss of feedwater and would, therefore, effect the ability of a plant to cooldown after a reactor trip.

l Following the SGWH that occurred at Indian Point Unit 2 in 1972, which resulted i in a circumferential weld idilura in one of the feedwater lines, NRC sent inquiries and questions to all utilities (including SCE) requesting design and i operational information describing design features for avoiding SGWH. In 1978, the generic subject of water hammers was declared to be an unresolved safety

issue (USI A-1), and therefore, received increased NRC and industry attention.

SGWH is associated with the draindown of top feedrings and, therefore, NRC attention was directed at the feedring design, and internal SG components near the FW nozzle. Experience had revealed that internal damage to the feedring and supports could occur. Thus, the modifications implemented to prevent SGWH generally required installation of J-tubes to delay draindown of feedrings, short horizontal runs of FW piping adjacent to the SG feedwater nozzle to mini-mize the magnitude of water hammers, limits on AFW flow rates to avoid too rapid an injection of cold water, etc. In general, attention focused on the internal structure and design of the steam generator rather than on conditions in the FW lines and flow control stations.

The NRC was aware of the possibility of developing condensation-induced water -

i hammer extending back into the feedwater piping as a result of line voiding '

because of a water hammer occurrence at the KRSK0 plant in Yugoslavia in 1979.

i Limited information on that event suggested that leaky check valves or preoper-ation pump testing (i.e., start and trip tests), or both, were the underlying causes. However, similar occurrences had not been reported for U.S. plants and, apparently check valve failures were not considered a significant contrib-utor to feedwater svitem water hammer by NRC. The Team conducted interviews of NRC staff involved in the resolution of USI-Al and reviewed staff reports gener-l ated by tho. The Team did not identify any citable references, decisions or

! discuss 4ns specifically related to why the scope of the staff's activities

, associated with the resolution of USI A-1 did not include the potential for and prevention of feedwater water hammers resulting from voiding of main feedwater lines due to leaky feedwater check valves. Implicit in the reliance NRC placed

on "J" tubes to prevent steam generator feedring voiding, thereby preventing i SGWH, was an assumption that feedwater system check valves do not leak. As a i result, the Team conducted additional interviews of NRC staff involved in approv-ing licensee inservice testing programs for safety-related valves. However, the i

Team failed to identify citable references, decisions, or discussions which

- demonstrated that inservice testing programs for safety-related feedwater system j check valves were specifically to be reviewed against criteria that would assure NRC could rely upon the check valve integrity to prevent voiding of feedwater J lines. It appears that the NRC did not consider feedwater piping water hammers q due to failed check valves to be a substantial contribution to the body of reported i occurrences and, therefore, did not pursue this issue further.

A compilation of water hammer occurrences, underlying causes, systems affected, and corrective actions taken is contained in NUREG/CR-2509 (Reference 1).

6-8 i

NUREG-0918 (Reference 2) summarizes (1) the causes of SGWH, (2) various mea-sures employed to prevent or mitigate SGWH, and (3) the nature and status of modifications that have been made at each operating pressurized water reactor plant. NUREG-0927, Rev. 1 (Reference 3) summarizes technical findings relevant to USI A-1, " Water Hammer," and provides insights into means to minimize or eliminate further water hammer occurrences. NUREG-0993, Rev. 1 (Reference 4) contains the regulatory analysis for USI A-1, and documents public comments received and staff response or actions taken in response to those comments on the resolution for this USI.

6.3.2 SONGS-1 Water Hammer History and Evaluations There have been a number of water hammer occurrences at SONGS-1, dating back to 1969.' Five such occurrences are summarized in NUREG/CR-2509 (Reference 1),

three of which involve the feedwater system. The April 29, 1972, January 14, 1974, and May 14, 1979 occurrences damaged FW control valves and piping supports in the main steam and feedwater lines. However, none of these events were attributable by SCE to SGWH, but rather to improper installation of piping supports, inadequate design, faulty flow controller hardware, etc.

Correspondence between the NRC and SCE dating back to 1975 (References 5 to 14) clearly indicates NRC and SCE preoccupation with the prevention of steam gener-ator water hammer (SGWH). NRC concluded (Reference 13) that backfitting SONGS-1 steam generators (to install J-tubes on the feedrings) was not warranted because of: (a) existing design features (i.e., the small horizontal length of FW pip-ing at the SG nozzle was not long enough to allow a steam pocket to form), and (b) that SGWH had never occurred at SONGS-1. Since SGWH had never occurred (although it had occurred in other Westinghouse PWRs), the probability of occur-rence was judged to be very low. As a-result, NRC also waived the need for estimating SGWH loads on FW piping and supports (Reference 14). Followup inter-views by the Team with NRC staff who were involved in previous SONGS-1 water hammer reviews supported the above findings that NRC's concerns were with the prevention of SGWH and not gross voiding of the feedwater lines.

The potential for cold AFW injection to cause water hammer was also included in the Team's evaluation of the references cited and in followup reviews (Refer-ences 15 to 19). The establishment of an upper AFW flow rate limit (i.e., 1_50 gpm per SG), as recommended by Westinghouse (Reference 16), was considered ade-quate and accordingly, operators were alerted in the E0Is not to exceed this flow limit whenever the feedring was uncovered to avoid creating SGWH conditions.

Team members met with SCE staff on December 13, 1985 (Reference 20) and SCE reviewed the SONGS-1 AFW system in terms of the original designs, upgrades (i.e., seismic upgrades), TMI Action Plan requirements, etc. These discussions also delved into prior water hammer occurrences and evaluations (References 5-to 15). Although this_ meeting did not uncover any significant new information, it illustrated the belief that SGWH would not occur, since it had not occured previously, despite numerous feedring uncovery transients, and that limits on AFW flow rate were sufficient to preclude such an occurrence.

6.4 Valve Inservice Testing The ASME Boiler and Pressure Vessel Code,Section XI, which specifies valve inservice testing (IST) requirements for these feedwater check valves, states:

6-9 i

Valves.shall be exercised to the position required to fulfill their function unless such operation is not practical during plant operation . . . Valves that cannot be exercised during plant operation shall De specifically identified by the owner and shall be full-stroke exercised during cold shutdowns.

Full-stroke exercising 'during cold shutdowns for all valves not full-stroke exercised during plant operation shall be on a frequency determined by the intervals between shutdowns as follows: for intervals of 3 months or longer, exercise during each shutdown; for intervals of less than 3 months, full-stroke exercise is not required unless 3 months have passed since last shutdown exercise.

Additionally, the NRC staff position on cold shutdown testing of val'ves is as follows:

1. The licensee is to commence testing as soon as the cold shutdown.

condition is achieved, but not later than 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> after shutdown, and continue until complete or the plant is ready to return to power.

2. Completion of all valve testing is not a prerequisite to return to power.

3. Any testing not completed during one cold shutdown should be per-

! formed during any subsequent cold shutdowns starting from the last test performed at the previous cold shutdown.

1 In 1977, SCE submitted its IST program to the NRC for review and approval. The

, program was revised in 1979 and 1983 in response to NRC comments and resubmitted to the NRC 'in January 1984. The IST program for SONGS 1 hes not yet bien ap-proved by tha NRC (as of January 1386). NRC staff interviewed by the Team at-l tribute the delay in approval of the program to (1) the long-term existence of j open issues for which SCE did not propose timely resolution and to (2) NRC re-view scheduling problers. SCE implemented the proposed program while awaiting

, NRC approval.

The feedwater system check valves are all periodically tested in the closed j position. The main and bypass feedwater regulating check valves are'normally tested in cold shutdown (Mode 5) and the feedwater pump discharge check valves j are tested in hot standby (Mode 3).

The Team reviewed the operating history for Unit 1 for the period from November 1984 (last refueling outage) to November 1985~to ascertain when IST could have been performed for the feedwater check valves (i.e., during plant shutdown last-ing more than 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br />). The periods that IST could have been performed include:

November 1984 Startup after refueling outage February 9-25, 1985 17 days

, May.1-11, 1985 10 days 3

August 21-30, 1985 10 days j -September 19-23, 1985 5 days The IST surveillance recceds were then reviewed for each of the valves. These

,> records indicate that ~zhe valves were last tested as follows:

6-10 i

, _ , - - . . , y + - . - .-- , , , - , ,-% ~ - , , ,p.-.,.4_ ... , ,_,- ,,. . . ,-,,m , _ , - - - ~

Valve Description Last Tested Pass FWS-438 East feedwater pump discharge check November 1984 yes FWS-439 West feedwater pump discharge check November 1984 yes FWS-345 A loop FCV-456 check February 1985 yes FWS-346 B loop FCV-457 check February 1985 yes FWS-398 C loop FCV-458 check February 1985 yes FWS-379 A loop CV-142 check October 1984 yes FWS-378 8 loop CV-144 check October 1984 yes FWS-417 C loop CV-143 check October 1984 yes There are 121 valves that are subject to IST during cold shutdown. Although IST was performed during each outage, all of the valves were not tested.

Consequently, the feedwater valves were tested only one time since October IP'"'.

The available opportunities for valve IST were not always fully utilized due to higher priority operational requirements.

1 Surveillance test procedures for verification of check valve closure for the i

' main feedwater pump discharge valves (FWS-438 and FWS-439) require one main feedwater pump to be running while the other pump is stopped. The discharge valve at the idle pump is then opened and the pressure is monitored between the pump and its discharge check valve. An increase in pressure or an operator observation that the pump is rotating backwards would indicate that the check valve is not closed. While providing reasonable assurance of check valve closure, this testing method also subjects the low pressure pump suction piping to some relatively high discharge pressures if the check valve failed to close (as in the November 1985 event) and thus damage is possible to such components as the flash evaporator. Testing with the idle pump suction valve shut would provide a more rigorous test.

Surveillance test procedures for verifying that the other main feedwater check valves are closed require testing to be performed during plant cold shutdown-with the steam generators filled to above the feedring. The motor operated valve upstream of the check valves is closed and the drain valve between this valve and the associated check valve is opened. The column of water in the steam generator provides approximate,y 4.5 psi differential pressure across the valve to provide the closing force on the check valve disc. The procedure states that this section of piping is to be drained, and that little or no flow from the drain should be verified. This test procedure leaves the surveillance operator to make the decision about how much flow is "little" and thus indica-tive of positive verification of check valve closure. The IST records do not provide a means of determining if flow occurred or its extent, or for verifying that complete valve cavity drainage occurred before a determination was made that "little, or no flow" occurred.

Valves FWS-345 and FWS-346 failed the IST on February 24, 1985 when tested dur-ing Mode 5 (cold shutdown). Maintenance work orders were prepared to repair both valves. However, on February 26, 1985, a "Non-routine and Increased Fre-quency IST" was performed during Mode 3 (hot standby), and the valves passed. I During Mode 3 the steam generator pressure increased the force available to  !

seat the check valves (to approximately 700 psia) and thereby have enabled them to pass. Testing at the higher differential pressure provides a more rigorous test.

The work orders were then cancelled and no corrective maintenance was performed.

6-11

6.5 Feedwater System Check Valv'e Maintenance The main feedwater pump discharge check valves (FWS-438 and FWS-439) were last disassembled, inspected and reassembled on May 5, 1980. All internals and seating surfaces were reported to be in good condition. No additional recorded maintenance since that time has been found.

Maintenance on the main feedwater regulator check valves (FWS-345, FWS-346, FWS-398) dates back to the SONGS-1 refueling outage in 1975. The maintenance history can be summarized as follows:

1975 Refueling:. Inspected three main feedwater regulator check valves and installed new internals in all three check valves.

1977 Refueling: Inspected and installed new internals in all three check valves.

I 1978 Refueling: Inspected all three check valves, found no problems, cleaned parts, and reassembled valves.

1980 Refueling: Inspected all three check valves, replaced cotter pins, washers and nuts on flappers, and reassembled.

SCE personnel could not identify any other completed maintenance work orders for any of the check valves in the feedwater system. However, a cancelled work order was found for FWS-345 (M0 #84103233000) and FWS-346 (o0 #84103232000),

the feedwater regulatory valve check valves for steam generators A and B, re-spectively. These work orders originated about October 26, 1984 because each valve had failed IST according to the problem stated on the work order. The work orders were cancelled on October 28, 1984, although the reasons for their cancellation were not stated. Further, a review of the completed surveillance test results for October 1984 indicated that both valves passed. The Team was not able to identify the reasons for these apparent discrepancies.

A 1976 SCE incident report summarized a failure of the 8 feedwater check valve (FWS-346) that was identified by an unusual noise near the feedwater regulator valve (similar to the noise in June 1985 discussed below). When the valve was inspected, workers found that the disc had fallen to the bottom of the valve body (as happened during this event). These inspection activities would have been authorized by a maintandnce work order, but SCE could not retrieve a related work order.

6.6 Feedwater Train Noise Investigation On June 24,1985 (Reference 21), a rapping noise was noticed in the breezeway between the containment sphere and the turbine building. Initial investigation with the aide of a stethoscope indicated that the rapping metallic sound appeared to originate from the vicirity of feedwater block valve FWS-342, just downstream of the 10-inch feedwater check valve FWS-346. The block valve was radiographed and the valve manufacturer contacted in an attempt to determine the source of the noise. In a subsequent evaluation (Reference 21), the pos-sible causes of the noise were believed to be: (1) a loose hinge pin in check valve FWS-346; (2) a loose disc is check valve FWS-346; (3) a rattling disc in block valve FWS-342; and, cavitation originating in the area of the check or block valve. Although the exact cause was not determined, SCE concluded that the possible causes did not threaten plant safety and scheduled further testing.

6-12

A presentation was made to the Onsite Review Committee (OSRC) on July 18, 1985.

The presentation is based on an earlier memorandum which summarizes the investigation and concludes:

Station Technical feels that it is safe to continue to operate the Unit since the failure of either valve (block or check) does not decrease the margin of safety of the plant. The components of the valve are large enough to pose only a very remote chance of traveling to and damaging the Steam Generator tubes.

The OSRC concurred with this conclusion. On September 3, 1985, SCE's Nuclear Safety Group reviewed the minutes of the Committee meeting and concluded that the station was taking appropriate action related to the B feedwater train noise.

Maintenance orders were generated to inspect the block and check valves during the next available opportunity, which occurred in October 1985 when the plant was shut down for another reason. However, it was decided to delay the inspec-tion of the valves until the refueling outage scheduled to begin at the end of November 1985, because no variation in the feedwater noise sound level or fre-l quency had been noted.

6.7 Valve Failure-Related Findings Check valve failures caused by partial disassembly while in service do not ap-pear to be unique to SONGS-1 or to the valve manufacturer (MCC Pacific). A limited review of Licensee Event Reports (LER) indicates that these valve fail-ures are not unique. The Team reviewed 33 LERs that identified check valve failures in feedwater systems; these indicated that 11 check valves failed be-cause the disc failed to close or because the disc retaining nuts, studs or locking pins failed to allow proper operation of the check valves, resulting in system backflow. None of these LERs specified that the failed check valves were manufactured by MCC Pacific, who was the supplier of the failed check valves at SONGS-1.

Failure of FWS-438 and FWS-439, the main feedwater pump discharge check valves, may have been due to inadequate valve design, since the disc-retaining nut was not provided with a positive locking device that should have reduced the pro-bability of the disc working loose and wedging into the valve seat and failing open. Additionally, excessive clearances between the hinge and disc assembly allowed the disc to rotate past the anti-rotation devices.

The failure of FWS-346, the B feedwater header check valve, may have been caused by the inadequate hardness of the disc attaching stud, which allowed the threads to strip and the end to mushroom over, conditions contributing to the ultimate valve failure. However, the service conditions (i.e., flow-induced vibration) experienced by this valve may also be a major contributor to failure. Such vibration was suspected during the June, 1985, investigation, and as indicated by the significantly mushroomed end of the valve disc-attaching stud. Failure of FWS-345 and FWS-398, the A and C feedwater regulator check valves, may have been due to similar service conditions.

The cracks in the seating surface of FWS-378, the 4-inch check valve in the B loop bypass line, appear to be service related. However, these cracks may be due to the significant forces on the valve from the water hammer.

6-13

. Failure of the yoke of FCV-457, the 8 feedwater regulating-valve, was probably due to lack of sufficient support or bracing of the valve operator during the pipe movement when water hammer loading occurred.

MCC Pacific Valve Co. Bulletin 400 (1978) cautions owners to' check the opera-tional environment and to avoid conditions that lead to high turbulence which can damage valve internal parts and shorten valve life. This same Bulletin also states (under " Service Recommendations") that:

Service in systems involving rapid and frequent flow reversals, pulsation or excessively turbulent flow should be avoided.

Locating swing check valves away from elbows, equipment, etc.

-within the piping system can often minimize or eliminate problems caused by this type of application. Suspected pro-blem systems should be reviewed with the valve manufacturer before selecting and purchasing swing check valves.

The 1975 SONGS-1 refueling maintenance records for the MFW check valves stated:

Main feedwater check valves internal discs, hinge arms and hinge pins were replaced. Excessive wear was noted on the hinge pins and hinge arms.

Despite this finding,- the Team did not find evidence.that the valve manufacturer was contacted as a followup to any previous check valve deteriorations. .'. ' ' h e r ,

the internals were periodically replaced'and operation resumed.

Spinning of discs and induced damage by turbulence in swing check valves are not a new phenomenon. An article in, Power, February 1983, states:

Spinning of the disc by fluid forces has injured many swing check valves. Sometimes the disc stem has worn completely through, allowing the disc to float downstream. Anti-rotation pins (Fig. 202, p S-42) can prevent this.

This .is the type of valve failure discussed in Section 6.3.4.

These IST procedures do not provide a positive means to detect a damaged, or degraded check valve. Only periodic disassembly, inspection and maintenance can provide such assurance.

6-14

. .r - .

References

'I

1. R. L. Chapman, et al, " Compilation of Data Concerning Known and Suspected

-Water Hammer Events in Nuclear Power Plants, CY1969-May 1981,"

NUREG/CR-2059, May 1982.

2. Anderson, N. and Han, J. T., " Prevention and Mitigation of Steam Generator Water Hammer Events in PWR Plants," NUREG-0918, November 1982.

3. Serkiz, A. W., " Evaluation of Water Hammer Occurrence in Nuclear Power -

Plants, Technical Training Relevant to USI A-1," NUREG-0927, Rev. 1, i March, 1984. ,

4. Serkiz, A. W. , " Regulatory Analysis for USI A-1, " Water Hammer,"

NUREG-0993, Rev. 1, March 1984.

5. K. P. Baskin, Southern California Edison Company (SCEC) letter to R. A.

Purple,NRC,

Subject:

" Secondary System Fluid Flow Instability,"

l July 14, 1975.

6. K. P. Baskin, SCEC, letter to A. Schwencer, NRC,

Subject:

Susceptibility to Steam Generator Feedwater Line Water Hammer Events," December 27, 1977.

l

7. J. B. Moore, SCEC, letter to R. H. Engelken, NRC-R/V,

Subject:

" Degraded 1

Conditions of Hydraulic Snubbers," November 6,1978.

8. K. P. Baskin, SCEC, letter to D. L. Ziemann, NRC,

Subject:

"Information Requested on Feedwater Lines," July 3, 1979.

t

9. J. G. Haynes, SCEC, letter to D. L. Zeimann, NRC,

Subject:

"Information Requested Concerning Steam Generator Water Hammer," August 31, 1979.

10. D. L. Ziemann, NRC, letter to J. H. Drake, SCEC,

Subject:

Steam 1

i Generator Water Hammer," September 12, 1979.

j 11. K. P. Baskin, SCEC, letter to D. L. Ziemann, NRC,

Subject:

" Steam j Generator Water Hammer," February 14, 1980.

12. J. A.

Dearien,

EG&G Idaho, letter to R. E. Tiller, 00E,

Subject:

" Steam Generator Water Hammer Technical Evaluation, San Onofre Unit 1," March 31, i

1980.

13. D. L. Ziemann, NRC, transmittal to R. Dietch, SCEC, Subjert: " Steam Generator Water Hammer," April 22, 1980.

! 14. J. G. Haynes, SCEC, letter to D. L. Ziemann,~NRC,

Subject:

" Steam i Generator Water Hammer San Onofre Nuclear Generating Station Unit 1,"

4 April 22, 1980.

15. K. P. Baskin, SCEC, transmittal to D. M. Crutchfield, NRC,-

Subject:

" Automatic Initiation of Auxiliary Feedwater System SONGS-1," March 6, 1981.

i j 6-15

__._.___.______.__m__ _ . _ . _ _ - _ . _ _ _ . - - _ _ _ _ _ _ _

' 16. Technical Bulletin 75-7, " Water Hammer in Steam Generator Feedwater Lines," Westinghouse Nuclear Services Division, March 9, 1977, Exhibit 89-1.

17. D. G. Eisenhut, NRC, to J. H. Drake, SCEC, Subjecti "

Auxiliary Feedwater System Flow Requirements," November 15, 1979.

18. D. M._ Crutchfield, NRR, to R. Dietch, SCEC, " Auxiliary Feedwater System Automatic Initiation and' Flow Indications (TMI Action Plan Item II.E.1.2),

November 18, 1982.

I 19. W. A. Paulson, NRC to K. P. Baskin, SCEC, " Auxiliary Feedwater System Technical Sepcifications," October 21, 1984.

20. Transcript (Set 3-044), December 13, 1985 Meeting at Rosemead, California.

I

21. Exhibit No. 355, 8 Feedwater Noise Investigation, provided by SCE on December 11, 1985.

i l

1 i

i b

l l

1 s

6-16 i

Table 6.1 Description and Corresponding Illustrations of Feedwater Pipe Damage Following SONGS-1 Water Hammer Description of Component, Support Figure Damage, Motion, Etc. Location (s)*

6.9 This snubber station, the closest to the H00A SG 8, showed no visible damage or pipe H008 movement. The feedwater pipe turns H00C vertically, and at an angle, to rise approximately 10 feet to mate with the SG feedwater inlet nozzle.

6.10&6.11 These support stations were the first HOOD that showed damage (or movement) caused H005 by water hammer. H006

( 6.12 View of FW pipe elbow at northeast corner Downstream

showing dent in pipe that resulted when the H006 pipe hitting the concrete corner and then rebounding.

! 6.13 View of pipe (looking south) showing HOOG movement of approximately 12 inches, slippage of vertical support pads off channel beam structures and downward drop of fit pipe.

I 6.14 View of support HOOG looking in opposite HOOG direction from Figure 6.13.

6.15 View of horizontal and vertical support H00H pads displaced southward approxiately 12 inches.

6.16 Evidence of first lateral motion (east- 120 ward); note deformed vertical structure, and then axial rebounding which displaced pipe supports approximately 12 inches south-ward.

6.17 Close up of scratch marks which show 120 evidence of water hammer forces driving

{ pipe inward and then displacing it 1 northward.

~

6.18 Further evidence of axial displacement of H00J FW pipe on east side wall looking north towards support H00J.

f

• See Figure 6.7 for support locations and identification.

i 6-17 i

1

~

Table 6.1 Description and Corresponding Illustrations of Feedwater Pipe Damage Following SONGS-1 Water Hammer (continued)

Description of' Component, _ Support Figure Damage, Motion, Etc. Location (s)*-

' 6.19, 6.20- A series of photos illustrating damage HOOK 6.21, 6.22 incurred at the support structure down-6.23, 6.24 stream of the southeast elbow. The damage 4

incurred by the structure clearly illustrates the magnitude of pipe motion which occurred during the water hammer pulse.

Permanent set (i.e., bend) in FW at elbow 6.25 near sup-pipe. View at elbow from support i

HOOK and looking west toward support ports HOOK l

' HOOL. Pipe has been bent laterally and HOOL.

south from support HOOL to SE corner i

elbow.

I View showing lateral movement (westward) HOOL I 6.26

! of pipe which resulted in sheared vertical support structure.

,' 6.27 View of spalled concrete and support plate HOOL damaged by water hammer, nuts were. loosened and bolts were missing in wall plates.

i' H00M 6.28 View of piping and support damage just downstream of where FW B line takes a j

90 bend to exit the containment building.

.I Note: (a) bent white stucture showing evidence of eastward motion, (b) vertical l

displacement of pipe until restrained l

vertically by white structure.

l A wide angle-view of supports H00M and 'H00M j- 6.29 H014 1 H014. Snubber at H014 was bent westward and FW piping was driven upward, Shows vertical support bending caused by H015 f 6.30 l

vertical motion of FW piping just down-stream o' containment penetration C-3G.

[

i *See Figure 6.7 for support locations and identification.

i 1

4 6-18

i. .

7"f f 'g TTgW" Nyg_-"

g 1/ -

7 +-

4 a gyyt v -yF "-"T-TT-"F"' **M' "E%r- -7 -+7-' ST?-r-"r4 W94 W*Ud""

F= f$ '?-'Md^f-i3y= WW7--v

1 1

1 U

CONDENSATION SURFACE

\ h M

& L STEAM  %

-Ww 3

a. Stop Valve Has Closed and Refill Starts jOdD If T

REPLENISHMENT OF CONDENSED STEAM _

,,l$.

waan -:::sX:i3!8::s::: -  :::$N,_:,:_:,

b. Cold Water Has Filled Bottom of Pipe I - AFW m

-'N 2  ; :I i:M3;  !

w:

b u

c. Pipe is Nearly Full and Surface Waves Form - AFW P

T STIAM V010$

'sigg$$$2saggliMig[N b

d. Slug Flow Conditions are Established -AFW l

Figure 6.1 Filling of a Voided Feedwater Line l l

l l

l 6-19

h E

I -

~

ani I eg 185a

- a en

> N ii - a B E.

a =

- a=

_s g

~

e E l I

e Ee eB e

e n 5 a

5*

a n

h a R

5 Ov , O-(- i 0-X i ,, ,,

-X 5 x

Ed N$ Mf M$

2 n

VJ kN kN z 8

g N d 5 d &

t t &

g E d a a

& y

- K ,

i_s:

/ i it!

4 5 5 kt o

o y ,

l vnme , unime I

6-20

SG "C" C0mTROLS

SG "C" FAILED GASKET h STRETCHED BOLTS 4

x,wS v s x " BYPASS LWE rwS 372 144 Q'rwS 375 CONCRETE TUR9mE WALL YOKE

, CLOSURE DISK tw C0erROLS FOUND IN BOTTOM /

OF VALVE CAVITY /

U

[ RA0V d '

MIwS

/

IL j ($AAAE CONCtPT AS FOR ~8h FCV p 67 p 20 342 f

1. A M

NM VrwS f

y I *

• AII ~E~

2 WNRM a

Flow  : SC *A" C0ttTROLS Y 7 SG "A" FROAS FEEDwATER TR as Figure 6.3 SONGS-1 Loop ~B" Steam Generator Flow Control Station

l

_ - a

=

=

N3 N3 4 2 2 2 N=

nA 3

3 3

3

^*.0,3 1

3

=

m I l " t e

s y

3 3

0 1

= S

)

w_3 A.3 W F

A E

(

t r

t e

a N,,,

N= b 0 2

3 d F

w e

e y

r i

a l

i x

u A

1 M

- S G

3 N

Al N-O

= S 4

- 6

,. e r

3 u

ig F

s i~ f d..

a r-" e7 w

a h= :

9:

AU E,FEEDinin E rz. .Tcw,P

~

minisur avSi end 'l

. w3 g .q MLM50 - . , ,.

^

.2 l Dagv%A7 n  : :. ,. ,

V' +

q; W

- ~

e y .;j'p;%g!?e;: f ._  : 7 ' 1::. f 9p g _. , 3 ; , .

,,_ M,;.p;n 3

'Tr.  ::,9  !!'i g'i t F 'e gp.g,pgg s  ; fi11 ra E g. ..F' 114 ' ;tiIl_ e lit!i!

il I l' l l 'fl' lIll P l~ l l'Il

.! 'i Il l I :lN ;Il k Yi l i-f ll.

j' f 3 !N 2 . h !l u0ll Ill l  ! l f$7h  ! l I . oh  !  ! ,  ! ah

! l H!  ! Ili 'l ll .,

lll l!i i!! l'l _l.I Ifff3"FMd3 f i,_l i__ l'!!. , -

!  ! lif !i!

l I  !  ! l ll lI l l l_m'i bf2 'N.d !l h I' I k'N4 l ll: : bl l .

i. b )l ,

! !! <l ll

!!WI l l i i 1 1 l Xi l' !1L1!il$fT "i !!

Il 0 FW' LOOP "B" l'SOLATEb U U6:30 ill I

., il l' l r- !l l fl' i.

g-~t' i i r ng l' Iil q; o

. > r i i , u

$ il  !;  ; ~T , i j

~

4 h 7Aul /jq, It'  ! I h.

q i

1j l

~l

, 1. -l j t 4 i dil (ti;; Litl tilt i ljy ti

,l i 1- l

l i i .

l l I l' hll COOLING CONTINUES _ i . l' l _

i

[

, i l J! l fi  !

l!y 6 plli;t 4h 4 -

i i i 1 i I

l lll111?Th! luli, li li.1 .il! 11; Isasar

4. t "l;Ti .f i >

p~- WATER HAMMER REPORTED @ 05:07 ~~ I e i- W8%{T]l FH l T

E 'I~

Il ll liylyl lll '

/!

1 1

i i:

l j q '

/ l300lll~ 375l

Loq , 3as t l I' 400 '

, I t. !425 '

}

[ i l'l

! A V / .l ll1111 1 111 1 r --pPERATORS ISOLATE VIA MOV-20 li. fill >MM. i l' i p' l l; ji h #

fi-_. R Pl CO'OL OF AFW @DO N IJE TO I ,;

~

,i ,i w ! li i i BACKFLOW 150 gpm ll l

!b W i

Oli5 AM! 'I!! >;n I I <! . I ,

l' IllllIlliilllllillllihil [j ! j

,! lill!! it l, i f ijf

~

!h llll l l l l !I lll llll il ll LI j i  !!!! il !lll ,

IMl H! !llI @ hi !N I 4 IIN S l  ; -

bdORD'ER lj;!""E"Y

!i! jill!illlli.

i t i ! . 4 AM i : 1>>j' 1.jsll i

g..LM]lli! I Il l j !'! TRANSIENT COMMENCES AT 04:51llillliilIij:itilllil!.Iliill

. iiil

't 'lil, to :I I l ! !!'i '- >

ii' ith

i! lls!! .

i j _i :t!.

' l'l i

'!! (' ll

1. !

llij sil, lll ie l;ll i ll!!ll '!!>

i tit !'ll.l .

l iii' 'l

!!al !, l

'll l!!!

,it! !jli ll. ll ..

. l .l 't l' ! i

.!l! Ql'l :ij  :

Figure 6.5a Temperature Recorder TR-456

,, . . ;;;' C. R O 2 2?IyNcn7 C,~ ' '~ _ 4

'C4'2"

( g. 7 l?.IO- C3 3 A StL.FV_ES${ EAST SIDE ELCC G,-3A .500 TD'dPJ FW PUMP

=

F- j9 GCEA) GGG SOC, fY2 CSS } WESfblDE ,

D^

~~'

FU2.PLE G-36 30C. TC/tIP; FW PUMP g

, g 6 CO' sg g,

N, j,

, g N. / \

' I ' ' '

GREEN

/[,' -

i -

PURPLE.

~~

\

Ns ,

/ \\

' N.

e s/ .

- Y

/' RED 'N g\,

a r>

ss

S

/.

O \\ "%

2, G-3B

\

if

!l ig,

\ q p ',l

f'=~

'#' "$, V/ffp

'l /

f I

$ \  %. , 6 ' J JCT'

,U ' .* g 8 ' ytf ., .

hkay

~,

j

~~ p,

]

t. . '.

ly .

I 1 SARoyal

/

4/ A '

o o

r,

(..s fliCqrg;.;y / 78/F jf

.- i ,

- ' & SUCI l & P f "

[b

- , BLUE T / ._

G-3A

\.

0

\ .

s

. \ >

. s / g ,,

t O \

~

~

-y N

- ', / ,

+

t g ,. -

I C__.__, I i Figure 6.5b Recorder R-7

l l

l 1

SLOW REFILLING OF HORI2ONTAL PIPE

a. Loop "B" Pipe 90% Full When AFW Stoppoo I'

8ACKFLOW OF STEAM TO REPLACE STEAM BEING CONDENSED CRITICAL FLOW CONDITIONS BEGIN TO FORM l

}'.- .!  !!! . !!!,[ I!'!

..::i:i:::;.:.:. ..::i:i:?;j$$.h':!::. .:::: : : ::.;. - : . 2:+ i

b. Wave Instabilities Being Formed Prior to Bridging of the Vertical Elbow

c. Elbow Has Bridged and Vertical Leg Filled

~

.i

d. Probable Steem Pocket Locatior Just Prior to Loop ~B" Water Hammer Figure 6.6 Refilling of Loop "B" Leading to Water Hammer 6-3

%\

b-4  %=E a

\

s-

' /

g

'* \

' / \

\

s

\ e' i E

\ '

$4 h y / 4l i, i

.s

+

+ fr e

?

Y,

</  :

1 w

. + / c i

m

!\ //g N .y.

' \ ,,

ye \ r "-

/

i

'7 4 + \ ff

\ .

' x *

'#d\

c\

\

6-26

4 a er SUPPORI NOOK ELBOW DENTED W THE INSIDE AND SLIGHTLY SULGED -

\ ] ,  ;-

} ,.. a nae:* .,

' l l naa r ss v

/

- A = = = = . .

J VIEW A-A l

l /

i 0

N ^

N N N

N N

N*7 uty- N

-P Cow . m n N

--- Ossi WaTEa lenauREn UPWAA0 g /

CoNnGunanoW as0VERIENT y # ~ PIPING

1. 33MIIVEffD OF PIPNIG p N

/ qp DOUUN

/

N COIITANIRAENT N PENETRATION C3G N

Figure 6 8 Overview of Feedwater Piping Et Support Damage Due to Water Hammer

r

~lr~f M M~Ji1 N

VERTICAL /SEMlVERTICAL

[" RUN OF FW LINE "B" TO SG "B

LOOKING WESTWARD.*

3mL

"(I V4

,m l

SUPPORTS HOOA. HOOB,

1. I

, & HOOC PROVIDE A RIGID SUPPORT. NO DAMAGE WAS

=~

~""* OBSERVED AT THIS SUPPORT STATION.

} l Figure 6.9 FW Line "B" Support Station

' Note: See Figure 6.7 for Direction Orientation, 6-28 A .

w fi I NOTE LATERAL Ogy;b @ MOVEMENT OF

. .ge asa p . #. I. }MPIPE WESTWARD APPROXIMATELY 8 INCHES, AND 1 DISPLACEMENT OF q PIPE INWARD p O RDS CONCRETE i , '8

1: j f

I i

~

. r i Figure 6.10 FW Line "B" Loop Support HOOD l

l

)

I r , j

y. v.-

'k. ' *

^'

hy - ; '".g NOTE BENT BRACKET

{l". *~

af

• g DUE TO LATERAL

[ c^; y '; .. ~

'.. y PIPE MOTION: VIEW

[f ** .s m 3 -i p -

.p, i .**- # ~ LOOKS WESTWARD TOWARDS SUPPORTS p

y, ' y. 77; /] g .-- t ris

' ^

HOOA, HOOB, EtHOOC.

(:

p ',.

R '

^

f Q* t; ..

'l . .,

N* 1 . jf * 'W -l t y s, p- _ ,

~

! l . .

t " 4 ja jy

~

s ay u[.Lh.wa,s ;i .

w u.

Figure 6.11 FW Line "B" Loop Supports H005 & H006 6-29

CONCRETE CORNER LOOKING TOWARDS SG "B" DENT IN FW PIPE 4--+9 l

CORNER ELBOW WHICH BULGED APPROXIMATELY

% INCH S" p ,, ,

~ -. .c

- i;

-: - ua x.:. s?gsw ,u _(~:Sgig, e

i Figure 6.12 Pipe Downstream From H006 Support llfyl/*~

~

alll? '. ' .

7-g

,) BULGED ELBOW

.>;j[ +

[, ((;D"yqqn~ ,g A~'  % , ,; ,Q } -. .

~~ *

~

? l, . . t 'Q : .

. r i%

IN ,,

3) ' , _ l s*

s.f

. ~ts i -3;  ;,L % .s..

N, % 9fQs nf A'*h;ne g$ ?:>c ,t l **

• ' ' )f(

,.e g.f ..d e

NOTE PREVIOUS SUPPORT d

• POSITION: PIPE WAS MOVED p- AXIALLY SOUTHWARD

. - -. _ c[im. - :w. .

.1j Figure 6.13 FW Line "B" Loop Support HOOG 6-30

l l

l l

, 4#- 2 3 s .

[g2. .

- $gt;

.0 f 7 , .gui -

,1 t *.

Nd 3

r i

k T:c:

b

{l:L[ $.

Iiau E  !

Figure 6.14 FW Line "B" Loop Support HOOG, Looking Northward m;xA7

, PIPE SUPPORT PADS 1

7 l NOTE DEFORMED VERTICAL CHANNEL DUE TO IN BOARD (WESTWARD) PIPE IMPACT s v- DUE TO WATER HAMMER b[!;>91jpl?,-

,,m s y; ild%id.A,-;;Q , .-, -.' . LL. >

Figure 6.15 FW Line "B" Loop Support HOOH 6-31

l y v7 g,ry:; m - .

y WW+.a :, a f C 'I, a

w -  %'

l ,

'/

')

l t-f

I

k. 4

( -

i;,' ' <

i 1

i l ~4 m q

i: :s 1% U .; A Figure 6.16 FW Line "B" Loop Support Point 120 (Which is 20'-6" South of Support HOOH)

(- j![E yg

.>i.:

1 i- ,

T .,

b,- '

- SCRATCH MARKS SHOW

' Dl N'

EVIDENCE OF !NITIAL WATER HAMMER LOADS DRIVING FW LINE INWARD (EAST) AND LATERAL (SOUTH) FOLLOWING REBOUNDING AT THE

' .: ,? . DOWNSTREAM ELBOW.

b

g l

,' @~  :

,j !vi e 1 a '?)/( - .i j

" ' ~

.) q _:n

( '-'

. . i. b .-

g

'nl ^'-

~ ~"f

- . , ...i a. .

Figure 6.17 FW Lino "B" Loop Support Point 120

, 6-32

FW LINE "C" 4

HOOJ 1i!

Il\

i Figure 6.18 FW Line "B" Loop Support HOOJ l

t t

?

l f' a \

c;; -  %

v -g ^

\

A VIEW LOOKING j  ; SOUTH: FW LINE

=s HAS MOVED INWARD

& UPWARD

.; m '

J I1

'l

! I k <!

ree 1 '

4 1s l

. . . :.Lik:Laauu ~ i Figure 6.19 FW Line "B" Loop Support Station HOOK j

i 6-33

-_ ..- - _.. ,_ _ - ~ - - . _ - + _ _ _ _ _ . - -

-l .i

~~

}

3 2

+

3, _, ]

1

+ ~< ,

s i

f T #

t 3 '

l

'. :0 i.

J . .: j [ .

}

{

-g. gem . , .

I. .:w I AM G U TO S PPOR S TION HOOK '

L'in"N ?"na's' '^" '

WATER HAMMER Figure 6.21 Local Damage at Support Figure 6.20 FW Line "B" Support Station HOOK HOOK

e

?

l 4

CLOSEUP VIEW, LOOKING ,

DOWNWARD AT HOOK l l

Y l 1

i

\

i i

Figure 6.22 Closeup of Damage at Support j Station HOOK i

( ; g.

- ger7.mmmm _. ~

- j ^.

gx jG f ..a j- ,

M ;/q (( i

, Ms, y.

. ,m, >

. g'. pf

. 3.9,jj A

v* ~ ' - . .

cl ,

+ i

- -:;7"" .

+..

'ri;.g .. , ~, ~

(.. / .- $_

j '

/(

i 4 ,.

^

-w ugyg; Q Figure 6.23 Torn Pipo Support Structure HOOK 6-35 l

- , - - - , - ~ . - - . _ . - . _ - . . _ , - , . . . . . - - , . .-

(j'_ ao 1:

b l

.; . .I; ;

1  ;  : -

. . ~ *,, . .. % ilF l , :,3 ' ' -

Li g.

'}) ,

._m.A l Figure 6.24 Torn Pipe Support Structure HOOK g-

[.'  !

0 ,

i:

e

!: , j

'n .:.

iff ,

Figure 6.25 Outward (or Lateral) Set in FW Line "B" Looking West Toward Support HOOL 6-36

Y W "h '" b }_

"e .9:%

T ^% k wy -

' SHEARED VERTICAL SUPPORT SPALLED CONCRETE w

s' i  ;

L 1: z.

hn

.* " ^{31 .'

!.&

I[t h e ?

I_ ~'

, ON Figure 6.26 FW Line "B" Sheared Support at Support HOOL l

I l

MISSING ANCHOR BOLT i

s. v.-

J .

' _ ej f l

?

:n I

it -

. LOOSENED BOLTS

~#

, t.H .,

(slLp; v w.., .

w ww~.

j ,

m 7., .

,, )

} ,

y  : -;;

l-s.,

j~ '*

c , ;. 3e ) -

c!Q:-fA.

, , j 2

.. _ Y l

Figure 6.27 FW Line "B" Wall Support Structure at Support HOOL i

.. 6-37 l

I,

-_-_m_w-mhw%~m- --ww- _W_ _.W _ m. _ _ _ . _ w-

ELBOW

' ~

l! .

.w

~

NOTE SUPPORT STRUCTURE BENT EASTWARD AND FW PIPE BEING RESTRAINED FROM i

FURTHER VERTICAL

.a MOVEMENT BY SUPPORT HOOM

--l , (AT TOP).

) (<, M

-, ~

L _. __=,.

j

• W~~q y; '

. A

.I - ,

Figure 6.28a FW Line "B" Loop Support at Support HOOM

}~ g.

' Y ((

t ..

f;F 3

s. .u.4 :y

+

( 4 ,,.q i; p . .

k.. .i -

h i

Figure 6.28b FW Line "B" at Support H014 and HOOM 6-38

tT~ '

2 g s_

_.I'\ ,

j p 1

M

. .~.

I NOTE BENT VERTICAL

! ROD DUE TO UPWARD MOTION OF PIPE l

t" em ,

u.

'h s.,-. a . \. . .

Figure 6.29 Pipe Hanger Damage at Support Station H015 6-39

v'

\ i s  ;.

a

/ e e

d.

e o s k

, i 0

, (>

U 5 f

\

k i

E

\

z s*

N*o e 3

l b: d, /  !

i

• +/.,

5

\ b

'a l

l

// '\

2 5

l 6-40

NOTICE CIRCUMFERENTIAL CRACK IN CONCRETE "y TURBINE BUILDING WALL w.

k;l

,,1. ,

l l

v e

Figure 6.31 FW Loop "B" Penetration at Turbine Building Wall j

l r en :r .

I

.L ci >/

[k .

, vie 3

i d & .

i j I, f

4 I / 1 l h i  !

a

• l FWS-342 FWS 346

) Figure 6.32 Inspection of FW Valving i

6-41 I

--_-_--__~.._-.----____~~.m_< -

S )

g -

. -s . ,

R.13 t

p, ,

?

u

+.

ff

. y( '

s

.. =

v-3 }g}hl,ng

%I

a. ,

.+

p' g-I ey

1. 'g

, p kk'~, , .,k!

• i ik Y ff. j'

~

h; f<4

• q3 MWh b .

FCV-457 MOV-20 j Figure 6.33 inspection of FW Piping 5

e .

. q,

'h

. g ._ ~ #

{_#

g' 5 4 7,, g: -1, >

• a ;k;d l ,

~, 3% q l

b s j -d :1 "Ni"N[' , ,

1  %,,.,g , ~ , - #9~ , -

l .,. .b 'Q) l n -

f-i i ,-c- INOR HANGER

! g 3-l}r r.'" AMAGE VISIBLE

1. g;;. T , .[': .f/ )

i l

~i i

f y'

WV  %?s i

~~t f,

?

i

. 2 $ :aWI i.

l Figure 6.34 2-inch Bleed Line at FW Loop "B" 6-42 i

,$1 e

T ,

i b{

g

~w; g jph

[,-  !(_

e+-

r %s.37g 8

icnll" VALVE)

/

$d:

~

%sm, . $~

Y Ch ',.

l '

l . 'g.;-' A'h' ,* 1 lt 'v'h . . .'

( f'epVALygj ~ ;L (:: ,l;. ,

$;7.

/

99

p4 7- ' .,-

3-g:e , -

yf,3 7-

• A<- V %Q;.

l l

Figure 6.35 Overview of Valves FWS-342, FWS-346, FWS-376 and FWS-378 in FW Loop "B" Control Station i

l l

}

1 s

6-43

pg 31 -

9

^

3,% '. 4 '

?

,.,. t ;; .;

_-y

,' (~sd f

.---[' L'

, _f s . -

s b -A__ s - ,

l ', a C 4 1-

/

m.m Figure 6.36 FWS-378 Failed Bonnet

,u  %

{f 31_

NOTE EXTENDED i- BOLTS AND

• 4 GASKET EXTRUDED sr AGAINST BOLTS

s. .

~ ~~t BY WATER HAMMER LOADING

- 99 n V ,l.g, \ .

.( s.

8 i

,y. <

.. :s

. L ._ M '

Floure 6.37 Closeup of FWS 378 Donnet 6-44

.;= 3

.q i k,4 f } 4 1

7 F l,.A7

'N .

s,

'f f%M?y. ,e s: $

' ff. _. QQ'f(;' '" _ _,

Figure 6.38 Damage to FCV-457 Actuator Yoke l

s e

. [

M i

M

' ' 5 *

.. 4 ,

- .s- :ka . :

Figure 6.39 Closeup of FCV-457 Yoke Damage .

I 6-45 L.. __

COMTAIMMEfBT SHELL g gf .

14 NET T N

, n

,.. x t .

s o' -

a

)

P NG 1

7e Figure 6.40 Junction of AFW & MFW at SONGS-1 (FW Loop "B")

CONTAINME q~ <

y ', ..ao .

!? ~ -r i

Y a-Figure 6.41 AFW Piping Run & Routing to Main FW Lino (Loop "B")

, r,- %

I wo- ,

n, .p

,.ci '%4."F' l

1 4.L-,-

, 4

=t

, m -

p= u ,

c. , , . .

,) %

au . . . -

a.

s Figure 6.42 AFW Loop "B" Support J

,h.,

.s* -

'1444 . '. r- ' l' ' r - I .[W'  ?

4

, 4, --

, gs ,

j,]( [ FA %

o:w. ame. -

, ., p -

s.

,p. . _

\g '

4

+ -

.i '4

' -f.

.- .'.Qy"l'h.';_.

'L. *

, %;o f 5K s _. . Q ,

R,; -<' j ^ :.i<- Q .

a.

.>e 1

4 t 7.. ..

F 9(:4 *s yg '

4,

\. , .

x. .., . .: -

..I,

v. & vag-%'

um:

- , u~, w .a t _. ..a^.

• • ~

t."

Figure 6.43 AFW Loop "B" Motion Due to Water Hammer Loads 1

6-47

\

\~

c; - ,

4.

~

L

n L  !. ,.

k&; 4 &n e

r: c. j 2-l .

't

~

s,-

{ .

t .

i %f l

1 a _u.c . -,, ;

Figure 6.44 AFW Loop "B" Piping Displacement Due to Water Hammer Loads

@ 'l , p ,,,

. [~r 9

v,

'?$

j -.

i i

Figure 6.45 AFW Loop "B" Piping Displacement Due to Water Hammer Loads 6-48 l

l VALVE BONNET BONNET STUD A a LNNNNNM9 eke r- g y g rg

-HINGE ANTI ROTATION NUB Q

I f

y -

CLOSURE DISK--

• W w //e v/w

\ VALVE BODY Figure 6.46 Typical Swing Check Valve With Internal Hanger, as Assembled.

6-49

,7 m, hO/ / N s

\

////lW l

L.

I I

l -

/

ANTI-ROTATION BARS s

- ,/

VALVE FWS- 346 AS ASSEMBLED J, -

,' l L/, O 4:

II':*[b.

1 2

1

-p WORN PIN HOLE IMPACTED PIN t

l WORN THREADS I

hmwim, y gM VALVE FWS 346 AS FOUND Figure 6.47 Check Valve FWS-346 6-50

,...,,.s.. mn :m 4

._3 T

.y-

.= 7

g 2C. NOTE DAMAGE l

,a (i.e., CUTS) TO DISC j SEATING SURFACE S

l E Wi i

)

. .)

h

/

Figure 6.48 FWS-346 Seat Ring Damage e4%'* Te ?i'YfTe # ' w77, -J % 'f ~rs' s - '

NLff ;4'b,m%Q$'4,s$$se$

npr?>? s

' ' ~ {l!,',11 '~~~',',i

.Asf~ s3.u,,d#,/ygs . &Sw~ y;;x:p.;;p 9 ; 7 ,m!<s ,~,' .

,s. s e, ,

~. .

s s ,s

?g,p,k/

hk '

k?h '

' ~

s %y.tski u .

.y m 7spp,F

/ ,re s ip,WMDP&q

-C p .t

• [s/

~ f ; 4,/<% -s,;(sd:u p y is %l"s~%

f*- r %

f -- ., ,fc j; e c q( %( ,:},

v

. % ' '% . s .

p-c?-Gp #9ym% e y/*y~ft 'm w .% , r7=f % y % 0 ~,9.

J, , &&* 4 e; s--

-t /s .3 3,ggv 9 -m e4

} f )1 0 h Y,

p f,. / /yr* }7,'p.,

w 1WhC!Qs :,v.,r,"~,m-y: , L s V .N/ ?/: ? 5;.; %/

y Ni. h 1;

} ' f _ x :% N AiN, ,ND k

Figure 6.49 FWS-346 Disc Face 6-51

3 Qls >%f;f}g _,

! ?ff677'{M

$ ~q$ %

? _

y ;>

l'.  ; /I:

t .I %Iw-j-ig.l~j%l .

p, . ,s j ; -

sig .i % .

hj!%f ^

..% NOTE IMPACT '

[f g j';f', g / ' DAMAGE AND ,

7 {,g- WORN THREADS.

'gf% -;!gIi' ,-

NUTWAS MISSING

j. ,

%. .% p ' [ % .- -

L t1,% j' ,

\

lA .gf f of ,% g f - ANTI-ROTATION

1. DEVICES s , ll % '., < ,

T 'jlhb Vrj e, l f"

$ , I#* N [9. ., WORN HINGE .

3 [ PIN HOLE

% ?+ flll:l l:I'g;?i-? ,-  ;"'

k' w n

t i .

a vlj_e ja  ;-G ,

lv A f

9g(?

g\. f ** it e,y* i i ;g

j. 9%di; b,sJ Ig&%

Figure 6.50 Damage to FWS-346 Disc Figure 6.51 Worn and Elongated FWS-346 Nut Pin Hinge Pin Hole l __ _.._. . _ _ _ _ _ . _ _ _ . . _ . . _ . _____

fjTgJpf ATj' S.gr

' ' ' ']q ,

i:. -

4j f i,

,l2 -

• i g.i -

~~

p NOTE DAMAGED .

s DISC STOP I y(- v 4* .e - -

ATTRIBUTED TO 9I s, [ CONTINUED

' ~ IMPACT BY es CHECK DISC r- -a. l Y

4  ?

4' ..

• ': l

, ,. i ;,,_,

Al

(:b h , h ,',::c' n,17  ; '

q'~= s , x ip

..J,Q .. ~s-L J: .ciusd,,JG! : ' ~,

Figure 6.52 FWS-346 Bonnet l

1 l

l l

) "

I j

i i

l e '

- 'd i,

1

- ,E p - ,w , ~:. c . ..

I  ;, ..it

,pMI : '*

i

-;zj, y k . ' , .

1. c ,, ' '!?M

) Figure 6.53 Closeup of FWS-346 Disc Stop 4

I 6-53 ll

e

,6

[ 'I'[*%^ y, OjJ P,".%

g

+

v..,

O Y

E?n g 0 g o k d.

."4

.# # W OO -

or e

o 6

5 o

d A YO 9<

$ d#

h7d j #

2

,< O z O ll 0

6 6-54

../-

1 i .

.;p.

o f h 7,

.h [

< *, , g r..

.g' '1,

~

' p' .

^

Figure 6.55 FWS-378 Check Valve Internals r ryp, rt ., 7tp *.:

-Mj",-f;g;fgs' II ,

y;,: +

~ *;}l i { < .

NOTE RADIAL RESTRAINT PROVIDED TO GASKET BY BONNET BOLTS

~t I

.s

• J Figure 6.56 FWS-378 Gasket Following Water Hammer 6-55

I

, *t y,

.. .fs ~

,,4

, e.f>,s?) y s

b ,

+- .

• 4

, ,,f.

. . M e, NOTE SEVERAL N# HAIRLINE CRACKS ON SEALING FACE y I jfQ _

Figure 6.57 FWS-378 Disc Face NOTE ELONGATED THREAD SECTION AT UPPER END OF STUD

.,Y YghR6:q) a:':a ac..au_ ,_a , , _

Ficure 6.58 FWS-378 Stud and Nut Removed From Bonnet 6-56

~

l/ O /

~

~

\

I

) @

's I /

" %. w - f VALVE FWS.G AS ASSEMBLED o*"  %

~~

s O ' ,

J .

Y 46N '

STUB \

, CAUGHT \

4 UNDER HINGE O

I RO ATED FOUND

- 15

~% 'y -

l VALVE FWS-G AS FOUND q Figure 6.59 Feedwater Swing Check Valve FWS 438 '

6-57

i

_ _ __ . m A

l tv

. s b \

i i ' , ,,->

g l g

. 'g e ,. f'

'

• NOTE 0 " FLATTENED l ,

POSITIONER 2

" NUB" g .,

k ,, '

o t .

]

~' %4.

!,,gf' l ~ ~ j' i

Fi0ure 6.60 FWS-438 Closure Disc 4

r

.. 3

. W.s

.~

/ :N~.L

-s

. s

' Tq,Q4 j.

l -

t .,

i

,.  %[ _%5

] [,

i I .

< , Y, l

l .+ .

r t i

7 I :j

.x ,

t

/

l I a- ..).'

l 4 Figuro G.61 FWS 438 Disc Holding Nut Arran00mont l

\

6-58 i

7 PERSONNEL PERFORMANCE AND HUMAN FACTORS EVALUATIONS 7.1 Introduction This section assesses the response to this event by utility and NRC personnel and the human factors issues affecting their performance. Other organizations, such as local, State, and other Federal authorities, were notified of the event, but did not play a significant role, so their participation is not addressed.

7. 2 SONGS Personnel Performance The utility personnel that responded included the on-shift operating crew, extra operators who were called in, and security and emergency services personnel.

Their performance and the human factors affecting their performance are dis-cussed in the general order of their involvement with the event. The Team interviewed most personnel involved and based the following assessment primarily on transcripts of those interviews.

7.2.1 Operator Performance The utility's immediate response to this event was made by the on-shif t operat-

, ing crew. The crew was larger than normal because a Control Operator trainee and an extra NPE0 were also on duty.

7.2.1.1 Ground Isolation Actions i Upon receipt of the ground alarm, the control room operators first identified i the source of the alarm, used the alarm procedure to identify the appropriate response procedure, and then followed ground isolation procedure 501-9-7, "4160V and 480V Bus and Feeder Faults." Extra personnel, including one SRO-i licensed member of management, and specialists in troubleshooting and repairing grounds were called to assist the operating crew during this phase.

The ground isolation process proceeded as directed in the procedure through the minor loads until operators reached th? steps to test large loads, which verbatim are as follows:

6.2.5 REDUCE Unit load as necessary and DE-ENERGIZE the major equip-l ment such as Circulating Water Pumps, Feedwater Pumps, etc.

One at a time and observe the ground meter, s N0fE: When a grounded circuit has been identified, it should be left de-energized until the problem has been

{ identified and repaired.

i 6.2.6 When the grounded circuit is identified, then initiate a Maintenance Order to have the circuit inspected and tested.

1 7-1 1

l l

6.2.7 M the ground is on 1C or 2C Bus and cannot be located by de-energizing feeder circuits, then CHECK and RENEW, if necessary, the Ground Detector Potential Transformer fuses because a blown fuse will cause a false ground indication.

6.2.8 g the ground is apparently on the Bus, then with the concurrence of the Shift Superintendent, REDUCE Unit load as necessary, TRANSFER the Bus load to other Buses and DE-ENERGIZE the 4160V Bus.

CAUTION If 4160 Volt Bus 1C or 2C is Inoperable in Modes 1-4, tIien restore the Inoperable Bus within 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> or be in Cold Shutdown within the next 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />. (Tech.

Spec. 3.6).

6.2.9 Initiate a Maintenance Order to have the Bus inspected and the source of the ground eliminated.

The actions operators took are identified in the narrative of the event, Section 3.

The procedure provided clear direction to the operating staff. Based on inter-views, the Team determined that management permission to reduce power was granted, j power was reduced and equipment was realigned to allow the testing of major equipment. Subsequently, all major equipment, except the feedwater pump, was tested and the grounded circuit remained unidentified. With the exception of some electrical instrumentation, the remaining potential locations of the grounded circuit were safety-related equipment, the deenergization of which would place the plant in a Technical Specification Action Statement, requiring that the equipment be restored to an operable status in a short period of time, or that the plant be shut down.

At this point, the operating staff decided to deviate from the procedure, de-spite the fact that there was no valid immediate need to do so. A plan of action was developed and discussed with plant management, who approved its im-plementation. As a result, the following steps outside the procedure occurred:

1. Operators checked the ground detector potential transformer fuses before the procedure required that they be checked.

2. Bus IC was paralleled with bus 1A momentarily to use an alternate ground indication to verify the bus 1C ground.

The period of parallel operation of auxiliary transformers A and C was short, but inappropriate, since auxiliary transformer A already had an installed ground path, i.e., a second ground existed which could cause accelerated degradation of the existing problem.

Based on interviews with those involved, the Team determined that the operating staff became aware early in the morning of the event that the plant would prob-ably have to be shut down. The ground looked as though it was either on the feedwater pump or bus IC itself. All required permissions, equipment alignments, power reductions, etc., were completed, satisfying the prerequisite to secure the feedwater pump. Management had been and was consulted again on how to deal 7-2

with the ground. The interpretation of the Technical Specifications was dis-cussed and those involved recognized that the deenergization of the feedwater pump, bus IC, or auxiliary transformer C, would probably require the plant to be shut down.

Preparations were made to shut down the plant when, based on further discussions, the operating staff again decided to implement an ad hoc process to deal with the ground. This decision was made with the understanding that the planned activities were not described in the procedures and that the steps were intended to locate any ground that could be isolated without affecting the operating status of the plant.

As a result, the following steps outside the procedure occurred:

1. Bus 1C was again paralleled to Bus IA and the PT disconnected to test the PT as a possible location of the ground.

This action again paralleled auxiliary transformers A and C, this time for about two minutes, and was inappropriate, as indicated above.

2. With the buses paralleled and the ground location still not identified, the tie breaker between bus 1C and auxiliary transformer C was opened; as a result, the indicated ground on bus IC disappeared.

3. With the ground now known to be on auxiliary transformer C, the tie breaker was again shut to support further attempts to localize the ground.

This action again paralleled auxiliary transformers A and C, this time for about five minutes, and was also inappropriate, as indicated above.

1 4. Subsequently, the tie breaker between bus 1C and auxiliary transformer C i

was reopened, but the transformer was left energized while technicians went out into the plant in a further attempt to find the ground.

The ground still existed on an energized circuit, sustaining the vulnera-bility of the plant while inspections were performed that did not require the transformer to remain energized. Further, inspecting grounded ener-gized circuits involves potentially significant personnel hazards. Deener-gizing auxiliary transformer C would take the plant into an Action State-ment and, as discussed above, might lead to a plant shutdown.

Following identification of the ground on auxiliary transformer C, the trans-i former was lef t energized and inspections were conducted to further localize l

the ground. If the ground location had been identified and found isolable from j the transformer, the plant would not have had to be shut down.

Both the failure to follow procedures and the actions taken that were not specified in the procedures were considered reasonable by the operating staff.

Evidently, their training, plant knowledge and procedures did not alert them to the dangers of tying the grounded IC bus with the normally grounded 1A bus or working around grounded equipment.

In addition, Procedure 501-9-7 does not provide Instructions for the methods and steps to identify ground faults on a transformer. Af ter the operators 7-3 1

__---__-___-_-_____I

localized the ground to the C transformer by steps outside the procedure, they had no training or procedural guidance on the significance of the ground loca-tion or the urgency with which the transformer should be de-energized.

As background, Technical Specification Action Statements can be entered inten-tionally or because of malfunctions, and normally allow time for operators to resolve problems before plant shutdown is required. Valid purposes for occa-sional voluntary entry into Action Statements include preventive maintenance, troubleshooting, and surveillance testing.

Based on the above discussion, it appears that the operators inappropriately deviated from procedures to unnecessarily delay entry into a Technical Specifi-cation Action Statement. Had the procedures been followed, the ground would have appeared to be on bus IC, the bus would have been de-energized, and the plant would subsequently have been shut down.

7.2.1.2 Recognition of the Loss of Vital Bus 4 and Reactor Trip When the initial alarms were received with the transformer trip, the operators assumed, based on their experience and training, that they were attributable to the loss of a vital bus. This was confirmed when they checked the vital bus indications on the electrical auxiliaries panel (Figure 7.4) and found that one of the vital bus availability lights was out. The operators needed to quickly identify which bus was lost in order to determine appropriate actions because the loss of some buses requires an immediate reactor trip whlie the loss of others does not. Efforts to quickly determine which bus was not available were hampered by labels which were too small to be read from where operators were positioned. The operators had to either go to the panel to read them or count the indicator lights on the panel to determine which bus was lost. (Note: the lights are in an unusual order--1, 2, 3, 3a, 4, utility, 5, 6.) However, the operators were able to quickly identify the bus lost, and tripped the reactor about 20 seconds after the loss of the vital bus 4.

1 1 7. 2.1. 3 SI Verification The next operator action of concern was their verification of safety injection (SI) actuation. The second step in procedure 501-1.0-60, " Loss of all AC Power,"

is to verify SI actuation. The procedure specifies that the method of verifying SI is to refer to the SI Actuation alarm (RPF0-2) on the first-out panel. When the procedure was read aloud, the operators' first response was that no SI had occurred, even though RPf0-2 was alarming. This SI actuation alarm was known by some operators to be unreifable because spurious alarms had been experienced with a previous loss of some buses. As a result, the alarm was initially assumed to be spurious. However, it was then discovered that the SLSS remote survell-Iance panel also indicated an SI signal. This panel (Figure 7.2) indicated 51 actuation had occurred by indicating starting and loading of the diesel gener-ators, a condition that only occurs with a loss of power and coincident SI.

This indication, together with RPFO-2 alarm, confused the operators and caused them to check the SI system operation. It was determined that SI was neither in progress nor necessary under existing plant conditions.

While the procedure specifies one method of verifying 51 actuations, it does not Gi ve operators guidance on alternate methods to evaluate whether SI occurred, nor is there a caution that spurious SI indicators will occur on loss of certain 7-4

electrical buses. The operators' duties and responsibilities instructions in-clude guidance not to take actions based solely on a single indication but to use other indications to confirm readings. Although the operators did use other indicators, the procedures provided no specific guidance in this regard.

During interviews, some operators also demonstrated a lack of detailed knowledge concerning the operation of the SLSS.

7.2.1.4 Electrical Power Recovery The next difficulty the operators experienced was with the station loss of volt-age auto-transfer scheme. The system has three primary indications associated with its operation (Figure 7.3). The first was received normally, indicating that the 18kV system had been isolated. The next expected indication was the i

"open" light for the generator's motor-operated disconnect (MOD) (Figure 7.4), l which was also received. The last indication, before operators would normally i take action to recover power, was the end of sequence indicator. This last l light was not received as expected by the operators. Af ter waiting for what '

the operators thought was enough time, they considered the sequencer faulty and took manual actions. Contrary to the procedures, the buses were re-energized bus by bus rather than all buses being tied together before the system was ener-gized. The operators were unsuccessful in three attempts to return power to the site using the 220kV circuit breaker (CB) 4012; the first attempt with the 220kV CB 6012 was also unsuccessful. The second attempt with CB 6012 was suc-cessful and the buses were then energized. CB 4012 was closed later.

The investigation to date has not determined whether or not the system operated as designed. However, no failures of this system have been found. It appears, at this time, that the operators did not know the time required for the auto-matic system to operate. The decision to manually re energize the buses rather than follow the action described in the procedure seems reasonable, but it re-sulted in errors caused by the lack of detailed information that was implicit in the procedure.

The attempts to close breakers CB 4012 and CB 6012 failed for two reasons.

First, the initial attempt to close both breakers was not performed properly.

Before energizing one of the dead buses, the interlock checking the synchroniza-tion across the breaker had to be overridden. The operator did not use the bypass pushbutton for this purpose on his initial attempt to close each breaker.

The 4kV breakers normally manipulated by operators do not contain this inter-lock feature so the operator was not famfilar enough with this unusual operation to remember the bypass without being reminded once during each attempt. The same operator later used the bypass to close CB 4012, even though both sides of the breaker were energized and the interlock should not have been defeated.

The second attempt at closing CB 4012 failed because the initial electrical-trip signal had apparently not been reset properly. For the lockup bus to be reset, the turbine *0enerator motor-operated disconnect (M00) had to have completed its opening. Since the operator attempted the reset early in the event, and because the MOD opens relatively slowly, the first attempts to reset the lockup bus were probably not successful. Further, the operator did not verify that the resets were successful by either observing the light around the reset button itself or by observing the clearing of the alarm on the vertical panel. The procedure, had it been followed, would direct operators to reset the lockup bus after the MOD had opened. Because the procedure was not followed in the order specified, the initial attempts to reset the lockup bus were not successful.

7-6

These operator actions demonstrate their lack of knowledge of the operation of the synchronization bypass switch, the lockup bus reset circuit, and the station loss of voltage automatic transfer scheme.

7. 2.1. 5 Use of the Diesel Generators Using the diesel generators to recover electrical power was considered by the operators during the loss of inplant power. Because operator training and the procedure for the use of the diesels (501-1.0-60) gives priority to available off site power as the preferred source, the operators continued their ef forts to re-anergize the buses from the switchyard. Because inplant power was lost for appr1ximately 4 minutes, the delay in re energizing the buses raises the ques-tion 6: to how long the operators would have continued in their attempts before using the available diesel generators. The procedure, however, provides no guidance on how long the attempt to restore power from offsite power sources should continue. The Shift Superintendent told the Team that he was aware of the difficulties of restoring power from offsite and would not have waited much longer before acting to load the diesels.

The procedure utilized by the operators did not specify the maximum period of diesel generator operation while unloaded, with or without ac-powered auxil-laries. The radiator fan motors must be powered in about 39 minutes to ovuld diesel overheating, and long-term diesel operation at no load is known to be deleterious to diesel operability. The previous version of Procedure 501-1.71, Rev. 2 (Dec. 1981), had the necessary guidance, but it was dropped when the procedure was revised.

7.2.1.6 Initial Response to Indications Af ter the reactor and turbine trips, the behavior of the primary and secondary systems was evaluated based on the operators' experience and training. The review of major plant parameters revealed to the operators that the pressurizer level and the steam generator levels seemed lower than expected, although both were not unreasonably low given the operators' understanding of the plant status.

The low pressurizer level was attributable to a rapid cooldown, although it was also recognized that charging had stopped with the loss of power. The concern

! about the lower-than expected steam generator levels was offset by the realiza-l tion of the design delay in initiating auxiliary feed flow from the steam-driven l auxiliary feedwater pump and that the motor-driven auxiliary feedwater pump was off during the loss of power.

Operators dealt with pressurizer level first by manually starting a charging pump. The second pump automatically started by design in response to the low discharge header pressure. The trip of pressurizer heaters on low level (less than 10 percent) increated operator concern for level and pressure control and raised the possibility of an SI initiation.

The decreasing pressurizer level was evaluated as being caused by the cooling of the RCS by the steam generators. Accordingly, the control operator respon-sible for the RCS directed the operator at the auxiliary feedwater controls to

" cut of(" auxiliary feedwater (AFW) flow to avoid an 51. The concern was based on the decreasing pressurizer level and knowledge that a low pressurizer level would lead to a further decrease in RCS pressure, possibly to the 51 actuation setpoint. Subsequently, AFW flow was reduced from approximately 135 gpm (indi-cated) to zero for about 10 seconds before the Shift Superintendent directed 7-6

that AFW flow be re-established at 25 gpm, because of his concern that the steam l generators would become dry. (Note: steam generator level was already low and decreasing.) This indicated rate of flow (actual rate was determined later to be approximately generator " dry". 40 gpm) met the procedural criteria to avoid declaring a steam A " dry" steam generator is defined in E01 S01-1.3-3, " Response to Steam Generator Low Level," as not having any of the following:

1. Wide range above zero inches.

2. RCS loop differential temperature greater than zero.

3. Feed flow equal to or greater than than 25 gpm.

These actions demonstrated steam generator status. The a reasonable concern for the RCS status over the interaction between the operator responsible for the RCS and the operator at the auxiliary feedwater controls is expected and the Shift Superintendent's actions demonstrated his " big picture" view of plant operations.

7.2.1.7 Decision Not To Recover Normal Steam Generator Water Level Af ter pressurizer level was recovered, reactor coolant pumps A and C were started, and with steam generator water level still observable, the plant was in a relatively stable condition with the cooldown continuing. The procedural guidance for the low steam generator water level (less than 10 percent on narrow range) is to raise level to 50 percent on the narrow range (501-1.3-3). The procedure the reactor contains no cautions concerning the effect of this level change on coolant system. Several operators, the Shift Superintendent and the STA discussed the inadvisability of this action and what direction to take.

There was a known steam leak in the mezzanine area but not specific information as to whether the leak was in the steam or feedwater systems or which loop was affected.

However, operators observed that although steam pressure was low, the leak was not large enough to depressurize the steam system and was not caus-ing an excessive cooldown.

(A steam generator pressure less than 400 psig is the point at which operators refer to procedures for loss of secondary coolant.)

The operators decided that a cooldown could be conducted safely, with a com-fortable margin to the 100-degrees per-hour limit in the Technical Specifications.

This cooldown was initiated by increasing the feed flow to the A and C steam generators since they had their reactor coolant pumps running. This process continued until the plant was placed on RHR.

7.2.1.8 RHR Initiation Operating Instruction 501-4-9, " Residual Heat Removal System Operation," con-tains a prerequisite (3.6), and a step (6.1.8) to close RHR-004, the hot leg recirculation bypas. valve around the east RHR pump. The valve is normally open to allow a flow leg recirculation use.path around the RHR pumps to support the post accident hot The relatively routine RHR use can be accommodated by shutting the valve, which must be locally operated within the containment build-ing. The operators considered that the plant was well enough under control that the procedure should be followed.

RHR alignment encountered difficulties because of a pressure interlock. To protect the relatively low pressure RHR system from the high RCS pressures, one of the RHR inlet isolation valves, MOV-813, and one of the RHR outlet isolation valves, MOV-834, have an opening interlock based on pressurizer pressure. When 7-7

. _ _ .. -_ _ - -. = _ - - .- -- - -- - - - . - - _ _ _ . .- __

the operator attempted to open MOV-813 per step 6.1.6 of S01-4-9, it would not i open. The operators considered entering the containment building to operate

{ the valve, but dropped the idea when they found that the interlock relay was i readily accessible. The RCS pressure indicated in the control room was 370 psig.

j Since this pressure was below the operators' understood relay setpoint of i 400 psig, the relay was locally operated using a contact follower button to I allow the opening of MOV-813. The relay also prevented MOV-834 from opening

and was again overridden.

f Post-event investigation found that the relay was aligned properly and would not reset until pressure dropped to approximately 367 psig (see section 8.8).

The reason operators believed the set point was 400 psig was attributable to their procedures and training. The procedural reference to 400 psig-is in-cluded in step 6.20 of 501-3-5, " Plant Shutdown from Hot Standby to Cold Shut-j down," in 501-4-9 as prerequisite 3.5, and the note following step 6.1.3. It is also included in training study guide number 22. Therefore, the operators had no reason not to expect the relay b allow the valve to operate under exist-1l ing plant conditions.

7.2.1.9 Continuation of Feed Flow to 8 Feedwater Line The operators' joint decision to continue feeding the 8 feedwater line after a l

1eak was identified was based on concerns for the safety of personnel attempt-

ing to locate and isolate the leak. Their intent was to cool the line feeding i l the break and thereby change the escaping fluid from steam to water. However, i unknown to the operators, this action could have created conditions for a second water hammer (from the continued injection of cold water into a horizontal feed

]a line with steam voids). Although there is procedural guidance on a maximum feed-l water flow rate into a steam generator with a low water level to avoid steam

] generator water hammer, there is no guidance and apparently no training for i operators on an acceptable flow rate into a voided feedwater line to prevent -

I feedline water hammer.

i l 7.2.2 Other Site Personnel Performance l This section addresses the performance of site personnel other than operators l who participated in the event. They include the STA, emergency coordinator, i emergency services, and security personnel. This section also addresses the

post-trip review.

I 7.2.2.1 Shift Technical Advisor's Performance l

The STA participated significantly in this event and took part in major deci-1 sions made by the control room operators. After assisting in the search for j the ground fault, the STA informed the Shift Superintendent that he was going j to bed. He had gone to sleep in his trailer at about 04:30, when he was awak-

ened by a loud noise. After attempting to reach the control room by phone j twice unsuccessfully, he started dressing. He also inspected Unit 1 from the l STA trailer and observed nothing unusual. He then heard on the two-way radio j an announcement that a fire truck was headed to Unit 1, and started for the 1 control room in earnest, arriving at about 04

58, according to the control room

! clock. Since that clock had been de-energized for about 4 minutes during the 1

loss of power, the STA arrived 11 minutes after initiation of the event. The

! STA is supposed to arrive in the control room within 10 minutes of such an event.

l l

! 7-8 I

i

He exceeded that requirement at San Onofre because he had not been alerted that the event had occurred.

On arrival, the STA began performing his normal post-trip task of checking the

" Critical Safety Function Status Trees," S01-1,0-1. His only concern was that the steam generators were below the 10 percent water level on their narrow range instruments. Since that condition was common following reactor trips, the STA reported it to the Shift Superintendent and they both agreed to look into it later. From that point, the STA made periodic independent checks of conditions in progress, referred to the Critical Safety Function Status Trees, participated in control room decisions, and checked the guidance of other procedures. The STA contributed to the decision to cooldown at less than the maximum possible l rate, specifically to stay well within the 100 F per hour limit. A rapid cool-down plan was considered, along with recovering steam generator level, after the report of the steam leak in the mezzanine. But, because the plant was relative 19 stable even with the reported leak, the STA recommended that rather than going through another transient that the plant continue to cooldown within the 100 *F per hour limit. Once this alternate plan was adopted, the cooldown was controlled with auxiliary feedwater flow. He checked the procedure for low steam generator level to see if there was anything else they should do to re-cover level and found the reminder to secure steam generator blowdown. The operators then recognized that the steam generator blowdown at about 100 gpm for each steam generator had been inadvertently re-established shortly after the restoration of electric power, and immediately isolated the blowdown by reducing the radiation monitor setpoint. (The status of steam generator blow-down is not indicated in the control room.)

By all reports the STA performed the useful function of independently monitor-ing operating crew activities and made a positive contribution to the shift's performance.

7.2.2.2 Emergency Coordinator The Emergency Coordinator's responsibilities are normally fulfilled by the Shift Superintendent until he is relieved by an operation's management representative.

The acting Unit Superintendent, an individual normally outside the group con-sidered operation's line managers, was on site to assist the Shift Superinten-dent in troubleshooting the electrical ground on the IC 4kV bus. This individ-ual returned to the control room during the 4-minute power loss and, after power was recovered and the situation was evaluated as an Unusual Event, took over the position of Emergency Coordinator.

NRC was never officially notified of the Unusual Event declaration, a responsi-bility of the Emergancy Coordinator, althcugh several phone discussions were held between the site and the NRC. The likely cause of this oversight is that the official declaration to the NRC was simply missed.

The Emergency Coordinator provided the first full explanation of plant conditions to the NRC. The explanation included three major errors which contributed to NRC's confusion. First, the reactor trip was explained as resulting automati-cally from the loss of the transformer. Second, the plant was confirmed to be in an Alert status and third, the feedline leak was initially described as a leak in the main steam system as opposed to a steam leak of unknown origin.

7-9

Thus, there were inadequacies iri the manner in which NRC was notified and in how the event was described.

7.2.2.3 Other Support Services Other support services were provided by emergency service personnel, health physicists and security personnel. Emergency services personnel responded to the event when the possibility of a fire was reported and subsequently partici-pated in attempts to locate and secure the source of the steam leak. No diffi-culties were noted in the performance of the emergency services personnel.

Health Physics personnel took samples as requested. With the loss and restora-tion of power, security systems were affected. Site security personnel and operators recognized the situation and implemented planned compensatory measures.

No significant problems were encountered by plant staff in gaining required access to safety-related equipment, and the procedures appeared to work smoothly.

No difficulties were noted in the performance of health physics and security personnel.

7.2.2.4 Post-Trip Review The unavailability of the critical function monitoring system provided by the Foxboro III computer impeded the post-trip review process due to the lack of data to accurately reconstruct the event. The operators' failure to have the computer reset after its power supply was interrupted during the troubleshooting activities and after the 4-minute loss of power negated the data gathering and recording functions of the system. Consequently, the trend recorders provided the only plant thermal-hydraulic parameter data available for the evaluation of the system responses. As the name implies, these recorders provide only quali-tative information. During the 4-minute loss of ac power, the computer and most trend recorders did not record useful data.

In discussions with SCE technical representatives, the Team observed that, on occasion, some site personnel who generally evaluate plant data lacked a suffi-ciently inquiring attitude. As a result, certain significant underlying reasons for system response or component performance were not detected until brought to SCE's attention by the Team. Examples include (1) the Team's reluctance to accept that the flash evaporator failure was not in some way connected to the water hammer led the Team to hypothesize a connection, which when tested, led to the discovery of the first failed feedwater pump discharge check valve; (2) the Tearr's reluctance to accept the first explanation of the similarities of east and west feedwater pump suction temperature traces led to the discovery of the second failed feedwater pump discharge check valve; and (3) the Team's reluctance to accept cparatcr log entrier %ich could not be confimed on every pertinent recorder trace led to the confirmation of the Team's conclusion that a reactor coolant pump had been started earlier than reported. It appears that SCE's process for evaluating and following up events may not be sufficiently thorough, and not systematic enough, to identify all failed components and root causes of failures.

The SCE's post-trip review report has not been completed. (The unit will be in a scheduled refueling outage for several months.)

7-10

7.3 NRC Emergency Response Performance NRC involvement in the November 21, 1985, event was limited to receiving the initial notification of the plant transient, alerting appropriate staff members, establishing communication links, dispatching resident inspectors to the site, asking questions, evaluating information, notifying other Federal agencies, monitoring SCE activities, and answering questions.

7.3.1 ENS Communications Problems The communications problems between San Onofre Unit I and the NRC can be divided into malfunctions of ENS system hardware and poor communications on the part of site and NRC personnel. The hardware problems with the ENS are discussed in section 8.10 and the site's communications performance deficiencies are dis-cussed in section 7.2.2.2. This section addresses the NRC's part in the com-munications problems.

1 Several of the reasons for poor communications on the ENS are NRC's responsibil-ity. The NRC pursued a line of questions that focused on details rather than on the overall plant status. The information provided to NRC frequently was j not understood because the recipients lacked site-specific background information and were therefore confused about plant conditions. The NRC asked leading ques-tions that produced answers of little value or created misconceptions about the sequence of events.

The NRC did not establish an open phone line to the site for over an hour. NRC Resident Inspectors took over the communications function from SCE personnel and then transferred the phone pickup location to a point remote from the information available in the control room. Finally, the NRC used the ENS phone for purposes other than obtaining information about the plant, i.e., a redundant retelling of information for each new NRC participant who came '

on the line in the ongoing discussion.

7.3.1.1 The Focus of NRC Inquiries in reviewing the transcript of ENS communications, it became obvious to the Team that questions asked by NRC characteristically focused on detail, when frequently a better question would have provided an overall perspective of the plant's status and the sequence of events which produced it.

For example, NRC learned that the plant had experienced a loss of power and that the reactor had been manually tripped. Instead of asking why the plant had to be manually tripped, NRC asked whether they lost flow. The affirmative response by SCE led NRC to conclude that the reactor protection system (RPS) must have failed, because a loss of flow should produce an automatic trip. In reality, a partial loss of puwer nad led the operators to manually trip the plant, which led to a loss of power to reactor coolant pumps, which in turn led to the loss of flow. Nothing was wrong with the RPS.

NRC's information gathering approach arises from the common practice of attempt-ing to understand events in a causal form. This event-based reasoning usually works well in an engineering environment, where engineers evaluate the effects of component or system failures, i.e., when reasoning from an event forward.

The approach fails when the task involves reasoning backwards and the initial cause is incorrectly diagnosed or when there is more than one significant cause.

Because faulty reasoning was a major contributor to problems in the response to 7-11

1 the TMI accident, NRC required the industry to develop another approach for l operator control-during emergencies. This approach has operator procedures j focus on symptoms or safety-functions as the basis for emergency operator actions.

This same concept may be of benefit to NRC staff involved in ENS communications during emergencies.

i 7.3.1.2 NRC Lack of Plant-Specific Knowledge NRC at times misunderstood statements by SCE because the recipients lacked knowl-edge about the unique design and operation features at San Onofre Unit 1. For example, the plant does not automatically load diesels on a loss of power and,

under these conditions, plant-specific procedures require use of off-site power, if it is avalaible. The NRC Duty Officer assumed that the diesel generators should automatically load on loss of power, as is common in the industry, and i therefore assumed that the recovery involved loading the diesels. The site 1

identified the specific transformer lost, but that information was not useful

because the NRC did not know the plant-specific electric bus configuration or identifiers. When the steam leak was reported, the NRC asked which side of the j main steam isolation valves the leak was on. This plant does not have main

. steam isolation valves and must manually shut block valves.

i These and other examples of plant-specific knowledge deficiencies caused mis-understandings that contributed to NRC confusion on what happened.

7. 3.1. 3 Leading Questiogs NRC frequently asked leading questions designed to elicit responses that would i support assumptions about how the plant was designed and operated, or hypothesis j about the sequence of the event. Unfortunately, the SCE staff did not always

( catch the errors in logic, and therefore did not correct them; on occasion SCE

! appeared to confirm inaccurate information, as in the following excerpt:

NRC: OK. Did diesels pick up?

Site: The diesels started.

NRC: Did they loai?

Site: No. They don't automatically load here.

NRC: OK. So that was part of the recovery process j Site: Right. L NRC misinterpreted the response to the implied last question as confirming that i the inplant buses had been recovered by loading the diesel generators. The

! inplant buses were recovered using the preferred source, the switchyard, as l designed and required by procedure. Unlike most other reactor plants, these j diesel generators start on loss of power, but do not automatically load unless J

a safety injection occurs.

1 i

i 7-12 i

Another example follows:

NRC:

Now, the basis for the Alert, that's because you had a loss of offsite power? l Site: It was based on that, that's correct.

NRC: You did lose off-site power?

Site:

We lost off site power to the site for approximately 15 minutes.

NRC:

Was that only to Unit 1 or to the entire station?

Site: No. Only to Unit 1.

NRC: Are you out of the Alert now?

Site: We have not gotten out of the alert right now. We're still eval-uating. As I indicated to you we have a break in the main steam system.

And we're evaluating whether we're in a UE [ Unusual Event] due to a rapid depressurization of the main steam system.

NRC: OK. So you're currently in an Alert.

Site: Right.

Later, after NRC conversation occurs. informs FEMA that the plant was in an " Alert," the following NRC:

The licensee has been emergency borating, they have aux feed-water, is operating satisfactorily. The licensee, correct me if.

I'm wrong, declared an Alert...

Site: No sir, I declared an UE.

NRC: An Unusual Event?

Site: Unusual Event.

NRC: Are you in an Unusual Event?

Site: No Alert was declared.

NRC: Not declared.

Site: We were never in an Alert. We are still in an UE. And we are probably close to getting out of it, now that we've-isolated our steam leak.

The combination of leading questions and confirmation of misinformation signif-icantly confused NRC's understanding of the event and plant status. In the first instance, the site appeared to understand the term " alert" as a generic name for emergencies.

This may be because the individual responding to NRC questions does not normally fill the role of Emergency Coordinator. However, Communication difficulties of this kind are attributable to the-inadequate 7-13 '

training of ENS system users on the subject of how to use the question and answer process to obtain appropriate, timely information.

7.3.1.4 SCE's Inability to Support an Open ENS Line The NRC perceived statements by plant personnel to indicate that they could not support a continuous open ENS line. The first call occurred during the loss of ac power and the Shift Superintendent hurried the NRC off the line to get back to the plant. Near the end of the second call, there was a concern in the con-trol room for a possible SI and the Shift Superintendent again cut the call short. In the third call almost an hour after the trip, originated by the NRC Duty Officer, the Duty Officer requested that an open line be maintained. When he discovered he was speaking with the Shift Superintendent, he suggested that the Superintendent call back when they could support the open line, even though the Shift Superintendent volunteered to provide a summary of plant status. The Superintendent then agreed to call back. In the next call, the site's Emergency Coordinator stated that he did not understand the event yet and asked if he could call back in 15 minutes. (However, the NRC continued to question him regarding the cause of the event and in response he provided information which later turned out to be inaccurate.)

The NRC Headquarters Duty Officer and Emergency Officer took the first two calls to mean that the site did not have sufficient personnel to support an open ENS, which seemed to be confirmed by the subsequent calls. This conclusion resulted in limiting the initial communications with the site unnecessarily.

7.3.1.5 NRC Site Communicators

.Another communication difficulty resulted because NRC Resident Inspectors re-lieved more knowledgable plant operators as ENS communicators. This substitu-tion deprived the NRC staff on the ENS of access to personnel knowledgeable about the plant and capable of interpreting plant data. Further, NRC site per-sonnel used a telephone remote from the control room to communicate with NRC headquarters. (See Figure 4.17 for the location of the phone used in the NRC .

consultation room.) The NRC rationale for the use of this phone was to minimize crowding in the control room. As a result of this relocation, runners had to be used, as available, to take requests to the control room and return with the relevant data. This cumbersome arrangement resulted in the inability of NRC staff on the ENS to monitor plant conditions in a timely manner.

Because SCE had staff and the responsibility to support an open ENS line, it was unnecessary and inappropriate for NRC to assume the role of the lone communicator.

7.3.1.6 ENS Conferencing The ENS system was used by NRC for functions beyond initial notification and information collection from the site. As each new NRC participant was brought into the conversation, he was briefed about what had happened. During these discussions, little new information was revealed and the site communicator was frequently a passive participant. When briefings were not being conducted, the conversation frequently turned into a conference evaluation of past data, some-times to the exclusion of soliciting new information from the site. Both uses of the ENS would have been better handled on separate telephone lines and doing so would have allowed the establishment of systematic processes for obtaining 7-14

information needed by the NRC to understand the event. The situation appears to arise because NRC lacks policy and guidance on the proper use of the ENS.

7.3.2 NRC Incident Response Plan Implementation During the event, the NRC partially staffed the Region V and Headquarters Incident Response Centers (IRC) but did not formally enter a standby response mode, as described in NRC Manual Chapter 0502, "NRC Incident Response Program."

Early communications between the site and NRC were not effective in developing an understanding of the event. The need for additional information and clari-fication of previously established facts was recognized by both the Region V Duty Officer (RD0) and Headquarters Emergency Officer (E0). The RDO informed the Senior Resident Inspector (SRI) of plant status and his concerns, and th e SRI volunteered to go to the site to determine what was going on and then call back. The E0 requested that additional staff members go to the headquarters IRC to establish an open communications line with the site to further evaluate event information. Subsequently, an open line was established between the site, the Headquarters IRC and the Region V IRC. As the morning unfolded, additional NRC personnel staffed these three communications centers.

SCE indicated in the second call that an Alert emergency classification declara-tion would probably be made on a subsequent phone call, and that it would prob-ably be closed in the same call. During later phone discussions, SCE did not notify NRC of the declaration of an Unusual Event. At 05:06, and shortly after L 06:00, SCE personnel appeared to confirm that the plant was still in an Alert.

Based on this information, the HQD0 notified the Federal Emergency Management Agency of the event at San Onofre, Unit 1.

Generally, a site declaration of an Alert would precipitate an NRC response mode transition to standby. This transition can be authorized by an executive team member, or Regional Administrator, or in the event of their unavailability, by the E0. Approximately 15 minutes after the apparent confirmation of an Alert, the Regional Administrator was bridged into the open line and, in response to his request for a summary of what had happened, learned that the site was in an Unusual Event. With this clarification of event classification, and recognition of the current status of NRC response, no formal activation of the incident response plan or transition to standby was warranted. The activities being performed by NRC were appropriate to a normal response mode.

7-15

// l l l l l I Fig. 7.4 Fig. 7.3 Fig. 7.2 J r

,k i

l Figure 7.1 Control Room Orientation of Figures 7.2,7.3,

• and 7.4 amams a amma m .

I sj SEQUENCER O LIGHTS OUT INITIALLY t,

t '

i. . , -

Figure 7.2 Safeguard Load Sequencing System (SLSS)

Surveillance Panel 7-16

C8 4012 y;mn- = .e BREAKER ,a ; w *i-CONTROL F-a y AND INDICATOR .,

" SYNC. '"

BY-PASS" g;j AUX.

CB 6012 ^

BREAKER -

C CONTROL i:

"b- og/ .. '

AND -

s INDICATOR

,  : +.

' " LOSS OF 220KV AUTO

"?

s TRANSFER

'~ END OF SEQUENCE"

" LOSS OF 220KV 18KV SYSTEM Figure 7.3 Electrical Auxiliaries Panel (left side) ISOLATED" VITAL BUS AVAILABILITY LIGHTS (1, 2, 3, 3A, 4, UTILITY AN D 5, 6)

MOD OPEN INDICATOR 7 Tl W " '

N P

$ Cl W EB ' ,g AUX. ~

TRANSFORMER s s-i' C

} l3 IiY' ads [e* s W{f c

5 - ,

~[l

+

s..

Figure 7.4 Electrical Auxiliaries Panel (right side) 7-17

8 OTHER EQUIPMENT AND SYSTEM EVALUATIONS This section identifies and evaluates other equipment and systems that had problems during the November 21, 1985 event. The performance of the equipment and the root cause of the problems are also included.

8.1 Auxiliary Transformer C Secondary Side In the process of searching for the ground f ault on 4160-V bus IC, plant per-sonnel had located it on the X winding (secondary side) of auxiliary transfor-mer C between the transformer and the bus. At 03:35, the operators had isolated the fault from bus IC, but kept the transformer with a ground energized. At 04:51, the auxiliary transformer C differential relay actuated, tripping and isolating the transformer. After the trip, the targets on the differential relay identified that phase 8 and C trip had occurred. Subsequent investigation has located the fault to be in one of the interconnecting cables between the transformer and bus IC. The interconnecting cables are 3/c-750 Kc mil aluminum armored cables, which run in cable trays located in the turbine building. The cable tray which contained the faulted cable section was located directly be-neath a feedwater pipe flange, which appears to have leaked for sometime. Fig-ures 8.1, 8.2, 8.3 and 8.4 show the details of the damaged cable section. A detailed examination of the damaged cable will be performed by SCE.

The root cause of the transformer differential protection actuation is believed to be the phase-to phase fault in the cable section which was damaged. The cause of the cable failure is under investigation; however, the intrusion of water into the cable could have contributed to its ultimate failure.

8.2 Safety Injection Annuciator During the event, when the unit was without ac power, the reactor plant first-out annunciator window 2 (RPF0 2) alarmed indicating SI initiation. The control room operators reviewed plant conditions and concluded that the alarm was spur-ious, and that SI had not initiated and was not required. Subsequent investiga-tion has determined that the alarm relay that actuates the SI annunciator window is ac power dependent and will erroneously alarm on loss of that power. This is a design deficiency.

8.3 SLSS Remote Surveillance Panels During the event when control room operators were verifying SI actuation, the SLSS remote surveillance panel load group status lights were reviewed for indication of SI actuation. The load group status lights on both sequencer panels indicated SI actuation. (Section 4.13 describes the operation of the status lights.) This indication of the SI actuation was also determined to be spurious, but did cause some confusion to the operators during the event. This spurious actuation of the SLSS surveillance panel lights is under investigation by SCE.

8-1

8. 4 Flash Evaporator Unit During the event, the east condensate header was overpressurized causing a catastrophic failure of the east flash evaporator shell.

As shown in Figures 8.5 and 8.6, the evaporator unit consists of a flash evapo-rator in a common housing with the 4th and 5th point low pressure feedwater heaters and drain coolers. The flash evaporators have not been used for several years and extraction steam to them has been isolated. The evaporator condenser is, however, still part of the condensate system flowpath. Design pressure of the flash evaporator condenser, 4th and 5th point low pressure feedwater heater tubes is 350 psig, while the shell side design pressure is 15 psig. The low pressure feedwater heaters were in service on November 21.

When bus 2C deenergized and the east main feedwater pump tripped, failed dis-charge check valve FWS 438 allowed the west main feedwater pump to pressurize the east condensate header. This pressure caused a tube failure in the east evaporator condenser which pressurized the flash evaporator shell resulting in the failure of the shell shown on Figures 8.7 and 8.8. The evaporator heater tubes are visible on Figure 8.8. After the loss of all inplant ac power, the remaining (west) main feedwater pump coasted down, and failed main feedwater regulating valve check valves (FWS 345, 346, and 398) allowed backflow from all steam generators through the failed east and west main feedwater pump discharge check valves (FWS 438 and 439) to the failed tube in the east flash evaporator condenser. This backflow continued until the operators closed motor-operated feedwater header isolation valves 20, 21, and 22, and main feedwater regulating valves FCV 456, 457, and 458.

SCE personnel have partially disassembled the east flash evaporator unit to determine the extent of damage. Figure 8.9 shows the unit with the evaporator heater, flash chamber, and south water box removed. The flash evaporator condenser tubes are visible in this view. Figure 8.10 shows a rupture of one of the evaporator condenser tubes. SCE is continuing its investigation into the damage to the east flash evaporator unit.

Helium leak checks were performed on all east feedwater heaters, revealing no leakage beyond that expected from normal operation. The west feedwater heaters will be leak-tested prior to returning the unit to service.

The failure of the flash evaporator had no direct safety significance.

8.5 Turbine Breakable Diaphragms (Rupture Disks)

During the event, steam was observed issuing from the low pressure turbine breakable diaphragms. As shown on Figure 8.11, each low pressure turbine has four breakable diaphragms designed to protect the turbine casing from over-pressurization. The diaphragms, made of thin lead, are designed to break if turbine exhaust pressure, normally subatmospheric, reaches 5 psig. The diaphragms are supported against external atmospheric pressure and normally seal the turbine casing against air inleakage. All diaphragms were intact prior to the November 21 event. '

8-2 I' -

2_______

Four of the diaphragms ruptured during the event, three on low pressure turbine 1 and one on low pressure turbine 2. Rupture of the diaphragms is not considered unusual for conditions existing after a loss of all ac power with continued energy addition into the main condenser and is of no safety significance.

8.6 Reactor Coolant Pump B Thrust Bearing Temperature Indication Reactor coolant pump B was started at 05:01 and at 05:09 the thrust bearing high temperature alarm was received in the control room. When the operators checked the reading, the temperature indicator appeared to have failed high.

After discussion, the operators decided to accept the indication reading as valid and started the RCPs A and C and stopped B. Subsequent investigation has determined that the temperature detector failed resulting in the high temperature indication. This failure is considered a random failure, not associated with the event.

8.7 Steam Generator Blowdown Isolation On loss of power, the radiation monitors fail in a mode which isolates the con-tainment building, including steam generator blowdown. After power was restored, blowdown resumed when radiation monitors in the control room were reset. A re-view of the design and operation of the operational radiation monitoring system (ORMS), shows that blowdown isolation functioned as designed. However, the status of steam generator blowdown is not indicated in the control room, and the operators did not recognize that steam generator blowdown was re-established when the radiation monitors were reset.

8. 8 RHR Valve Interlock During the event, initial attempts at opening RHR system valves MOV 813 and MOV 834 were unsuccessful. It was assumed that the pressure permissive inter-lock was malfunctioning and the valves were opened by manually depressing the interlock relay. Subsequent investigation had determined that the permissive performed as designed. The procedure was found to be imprecise and information provided in training did not correspond to actual plant setpoints. As a result, the operators did not correctly understand the response of the pressure-permissive interlock.

8.9 Event Recording Systems The majority of the plant capability to record data needed to analyze the event was not available. During troubleshooting for the ground on the IC 4kV bus, the '

power to the Critical Function Monitoring Systems general purpose computer (the Technical Support Center computer), the FOX III, was momentarily interrupted, causing loss of the automatic minute-interval storage of plant data. The system was not reset by the operators. Consequently, when the reactor trip occurred hours later, the 25 minutes of pre-trip data automatically printed was for plant conditions not related to this event. The system did not provide useful infor-mation until reset after the event at approximately 08:30.

I In addition to loss of the FOX III system, nine control room chart recorders {

lost power to their chart drives for the duration of the loss of station power.

In seven cases, power for the recording pens was not lost. The recorders l

l 8-3 l

continued to record parameters values but without the chart drive. As a result, only the minimun and maximum parameter readings are available for the period the power was off. A summary of the recorders affected include:

Chart Drive Lost For: Pen Status:

All Steam Generator Continued marking, steam flow, feed flow and water level.

Pressurizer pressure, Wide range pressure wide and narrow range, indication lost only, level, water temperature.

RCS Cold leg temperature. Continued marking.

T ave /T ref. Failed low.

These failures limited data available to monitor plant behavior in the Technical Support Center and hampered efforts to reconstruct the event and to evaluate system performance after the event.

However, the operators had sufficient instrumentation during the event to follow their procedures and ensure plant safety. The loss of instrumentation and the ability to trend parameters would have become more important if the event had been of longer duration or had involved additional complications.

8.10 Emergency Notification System During the event, the Emergency Notification System (ENS) red prone in the con-trol room rang spuriously while the plant was experiencing electric power prob-lems. Since the spurious ringing apparently coincided with power system trans-ients, a review and investigation of the power supply system of the ENS was conducted. Figure 8.12 shows the power supply arrangement. The root cause could not be determined and is considered as random noise-induced signals.

l However, this spurious operation distracted control room personnel and led, in part, to mixed communications and confusion between SCE and NRC personnel.

8.11 Safeguards System The automated safeguards access control system had two malfunctions during this event. Nevertheless, operators and security personnel implemented appropriate planned compensatory measures. The Team identified no signifi-cant problems that operators had in obtaining access to safety-related plant equipment.

8-4

-.%,.,...3 ,

/ '

. ~:

W. .

/ , . T.J .

4 CW' ' . .~g;p

.,.,^.# -

x ;' -

-- .y e

v. 1

.-y- :4+

. , . , .:  ;' . mi

.; :1

~<

v, .1

.,, .7,

,,a .

L

,.5 '

'*3 _

. . :. ; . ::u.caa aadk i k' . .h .-

Figure 8.1 Cable Length Showing Damage c ,.rw ~.  ;

,rP ; .

• g ' - , .4 ,

f w . s . .

-' \ -

\..

~.j!f f- .-

s. .

4..

../.: 3 . .;

y , py, i i1 ,

o

.,

• gy

. .c-

.4 1

- r N

N' g.y%j -

fry _ , ) ' - ..

,L 4 fi ," "i

['. ~ .

, [- : _.

g1- f.. c,z. . , . )

. g. 6 '.  : 4 ~'

• vjur k ...., . . . . .

.. .. . . s - '*' .

Figure 8.2 End View Showing 3/C 750-Kc mil Copper Conductor, Neoprene Jacket and Aluminum Armor 3

8-5

. _ _ . _ _ _ . . . _ . _ . _ _ _ _ _ _ _ . . _ _ ~ - _ - _ _ _ . _ . . . . . _ _ - . . - . _ _ . - _ . . . . _ . . . , _ _ _ _ _ . _ _ . . _ _

-Q xNs

.g

/

/ ,

~. . .- -

/  ; --

~

, 'g;=, j

-r..

s GW ),- anew

, e.:

.v_ -

s;.

m.

)@3 1

3

- ^g .

l

~

, .,c

1J' ,' 1

~7. _.j

,; 9

~

.uL -

l

. 1 u;p

~

.. ~ . :., c49

_ . u _ ,w m&, .m Figure 8.1 Cable Length Showing Damage g...,, _

M

.a g:

a .-'

f ..

{". # '

_'

• i -

.1, .

. .p ;

s.

i

j. l+ .

q ..e

g. s-g]

. y

,7

' m ;..y 4

, - 4, - - ...

.: s ; - -

h, '

<\..- 4

.e. #

'Ik D

, .}

,. e . 4,

s. .,,.g .

..-(

..g -;,..... ww:p"g

.f ;.

z* .

t

,h#... . . , , . . . . 's.t S*u M. e'. .

Figure 8.2 End View Showing 3/C 750-Kc mil Copper Conductor, Neoprene Jacket and Aluminum Armor l

i 8-5

,. . , q x ; e W ' ' Y;a + ' ' "

~~

'"t.:.. w, f ., _, .f r .

. t

' .Q ,' ; _ *)_ i j; %f *

,  ;; : - ~*

- z. -~. !  : 53:,

. . . . :"  :,y

. .,e

- . . ,:1

%,&.Q; p );, . . - :- , , " , :.

. / ,;. ,, =

M .~ ,,

'9

~

.. u-

~  ?'

m, ,q

, jy;L

, Q ,.,j, ( + ,

.o -

p,4.,>- wouw ~ ,; y ;n,a .

a[g] .;-- 4

,, o

. ..(. . a . n, .- ',; k

• Figure 8.3 Cable Section When Phase-to-Phase Faults Occured

%h q . . ~

E

., . .a. ,

'i

. W '.-' ' ;.n,

. .[__ <

sq '

. , j) u,

.q..

.T.~+ Rg

x, .

,

s ;,5s - -

~ : :,.

e a

;

. ~q +

4 ,;

-.+a . %,. . . ~ . _ ;a Tt

?b - .

4 y

, b.g+. g;;ux 3-,O f.m'4f ... 3., '

&.. .n.a ..t

, +

., ";.1

- (. ; . : ,::( yy

' g eS. *' ,- .

i

,  ?

~ , F' 'j

. , a.

O[ wu

~

Figure 8.4 Close-Up of Damage Caused by the Fault 8-6

H 1

)

Q D W b

'J w

n E u

O I V v 8e P s o

r e s

?

s t

o m r n o t

$ u O s a

l F

a r

o p

n "

! a v

a

/s E HTt S l h A l n

t S

n LENo F

a s

a m

l n a r t

r n F o a t E

T n

r A

se o a E H

8 M e A

f M

t n c n

o g n o

A H

C t

a t

W w r F

4 t

s i

a a

n H S S e i n o A -

o r L r

a a

v r

l e

v t

t g

i n

S 5

o ]

8 i

r u

g e

F dc _

i. d Q) i ' 4  !

5 5

• Q E .2 I

Eg Il* 55 e

{s as lst

-- II k

,a N ei 3

ms N *B  :

I=I

-= S 2

/ k a

e

1. os

'  ?

e 5 s

W k.6 R

(

8-8

.,+CQ \ s . _

p  ; N,;*b x I

Figure 8.7 East Flash Evaporator Unit (Southwest View)

c

up.- _ ,

I ,

A

_hl

,; , -q.

c

'A.

J A

A

'I a-g /"

p.

g

- .g 4 1. c. "

Q

[7 '

k ,

s E1 _ A~u hu ..

Figure 8.8 East Flash Evaporator Unit Showing Shell Failure (South View) 8-9

' \\

\

wiih " ._

gy -

l

{ a E  :;

m  :

kbsm .U ~

[$ 33:

.= ..

4 ~rr

"[~~,f; f . ,

Figure 8.9 Flash Evaporator Unit with Evaporator Heater Flash Chamber and South Waterbox Removed (Northwest View) 1

~.

. ... w ' ..

, ;< [ ti?j'.' fr y?' .(

ac c , ,'s % .1Qpaej g .; p -

~ ,

'a- -;, y 1

-p, p g

._i 2 .

Figure 8.10 Failed Evaporator Condenser Tube 8-10

N 4 N W<

1 MIGN PRESSURE TunemE

. O RUPTURED V

RU'TURED

LOW PRESSURE 1 TURelWE U40AMAGEO RUPTURE 0 i

U4 DAMAGED RUPTURED t

LOW PRESSURE 2 TURenE Ufe0AMAGED U4 DAMAGED Figure 8.11 Arrangement of Low Pressure Turbine Breakable Diaphragms 8-11

l l

22e av AUXILLARY TRANSFORMER C 4 A 4160V BUSIC 4160V BUS 2C 11CD2 12CD2 SWCR NO- 1 I av^c 5*ca ao- 2 I 'v^c ItC10 12C10 h 1102 y 1202 MCC#1 480VAC MCC 72 400VAC NORMAL POWER ALTERNATE SUPPLY POWER SUPPLY r --- - g NC; t : 240 ,l MANUAL TRANSFER SWITCH 1 2 4 l l LOCATED IN 4KV ROOM)

L _ _. ___ _J

~ TO SCE TELECOM. SYSTEM 3NVA EXiDE BATTERIES A W W 48&12B740V TRANS.

2 HG3 5 144AH g 3

, , j ,

RO3

^ l 120 VAC I m BATTERY a L COMMUNICAtl0N 12 CHARGER OtSTR. PANEL M LO N 3

- p 48V. 50A FUSE PANEL FUSE PANEL 120V140y N0; 1 40-2 10.30 BREAKER LORAIN PANEL -

INVIRTER _

WBA 102 81 1 500 VA BACK UP OC POWER SUPPLY VA,C SUPetr

' r r-------------------

g D O

{ ine'? =

i l RED PHONtS 15 TOTAll BREAXER BOX le IENS SYSTEM)

I SCE EQUIPMENT  : T ATiti EQUIPMENT I

Figure 8.12 Power Supply Scheme for the ENS 8-12

l 9 PRINCIPAL FINDINGS AND CONCLUSIONS The event that occurred at San Onofre, Unit 1 on November 21, 1985, was signifi-cant because (a) all inplant ac power was lost for 4 minutes; (b) all steam generator feedwater was lost for 3 minutes; (c) a severe water hammer was ex-perienced in the feedwater system which caused a leak, damaged plant equipment and challenged the integrity of the plant's ultimate heat sink; (d) all indi-cated steam generator water levels dropped below scale; and (e) the reactor coolant system experienced an acceptable but unnecessary cooldown transient.

In addition, other aspects which contributed to the complexity of the event and to the burden placed on the operators included: ' a rupture in a flash evapora-tor unit; spurious and incomplete instrumentation indications; fire alarms and system actuations; and malfunctions in the automated security equipment.

The Team has concluded that the most significant aspect of the event was that five safety related feedwater system check valves degraded to the point of inoperability during a period of less than a year, without detection, and that their failure jeopardized the integrity of safety-related feedwater piping.

The root causes of the check valve failures have not been determined and are still under review by SCE and its contractors. Potential contributors to this problem include inadequate maintenance, inadequate inservice testing, inade-quate design, and inadequate consideration of the effects of reduced power operations.

Maintenance records for these valves were either missing or lacked specificity on what was done. Inservice testing records for these valves were inconsistent; the testing procedure was not rigorous; the test acceptance criteria were subjective; the testing frequency was open-ended; and, the tests did not assure detection of the failures found. These check valves and valves of similar design have a history of like failures. Finally, reduced power operations at Unit I are now routine because of steam generator tube plugging and sleeving, and the reduced feedwater flow may have increased the susceptibil-ity of check valve components to hydraulic-induced vibration.

In addition to this major conclusion on the underlying cause of the event, the Team has made the following related findings and conclusions. There is no significance to the order in which they are presented.

1. The primary cause for the water hammer in the feedwater piping was the failure of multiple check valves in the feedwater system. These failures permitted the piping to empty and fill with steam before the motor-operated feedwater isolation valves were closed. Although the steam condensation-induced water hammer ~ occurred in only one feedwater line, the potential existed for water hammer to occur throughout the safety-related portions of the feedwater system.

2. The failures of the five check valves in the feedwater system provided a mechanism for potential common mode failure of the heat sink provided by the three steam generators. The failed check valves permitted high pres-sure steam and water from the steam generators to flow back to the low pressure condensate system; the backflow carried with it the auxiliary 9-1

feedwater flow necessary to maintain the heat sink provided by the steam generators. Operator actions were necessary to stop the backleakage and prevent a more serious sequence of events.

3. Long horizontal runs of feedwater piping with the potential for voiding are particularly susceptible to destructive steam condensation-induced water hammers. Further, operators are not provided the means for detecting the voiding of these lines or given guidance on appropriate ways to deal with the situation. Design or procedural changes may be warranted.

4. The flash evaporator failed when overpressurized by the discharge flow of an operating feedwater pump due to the partial loss of power and a stuck open feedwater pump discharge check valve that should have prevented the backflow.

5. The timing of the five check valve failures could not be ascertained with certainty. The Team concluded that all check valves had failed prior to the event because the missing parts to the valves were not found in the inspected feedwater piping after the event. Noise from the B steam gener-ator feedwater piping, evident to plant personnel since June 24, 1985, supports the conclusion that the feedwater control station check valve in the B feedwater line had failed earlier. The inspection of the steam gen-erators has not yet been completed by SCE.

6. The surveillance procedure for testing the check valves in the Inservice Testing (IST) program lacked adequate methods and objective acceptance criteria for determining whether check valves are closed. Thus, although the check valves had been tested within the past year, operators may have misinterpreted the test results. Furthermore, the IST is not designed to detect developing conditions that may lead to the failure of the check valves, e.g., loose disks and stud nuts.

t 7. The NRC had not completed its review of SCE's Inservice Testing Program.

The initial program was submitted in September 1977 and revised in its entirety on January 24, 1984. Disagreement between SCE and NRC on resolu-tion of certain open issues and scheduling problems with NRC's review have substantively contributed to this delay.

8. The resolution of the Unresolved Safety Issue, USI A-1, " Water Hammer,"

did not specifically address the prevention and mitigation of the conse-quences of condensation-induced water hammers in feedwater piping upstream of the feedring. Interviews of NRC staff involved in resolution of water hammer issues failed to develop citable references, decisions, or discus-sions that provided a basis for excluding further consideration of feed-water piping water hammer. However, in the regulatory analysis of the resolution of USI A-1, the staff acknowledged that elimination of water hammers is not feasible, that the frequency of water hammers had been sub-stantially reduced by changes in design and operations, and that studies of water hammer had revealed a significantly lesser safety concern than previously hypothesized. It appears that further consideration of water hammers due to main feedwater line voiding was not pursued due to a lack of reported occurrences in U.S. plants.

9. NRC's reliance on "J" tubes to delay the development of conditions neces-sary to support steam generator water hammer implicitly assumes that feed-water check valve integrity would be maintained to prevent steam generator 9-2

feedring voiding. However, corresponding regulatory requirements to ensure that these check valves performed this safety function were not part of the resolution of the water hammer issue. ,

10.

The root cause for the loss of power was a phase-to phase fault of an electrical cable from auxiliary transformer C to bus IC. The underlying reason for the cable failure has not yet been determined; however, it appears that the cable may have become wetted by a long-term flange leak from the feedwater system, running above the cable tray.

l 11.

The plant is designed to experience an extended loss of inplant ac power on loss of offsite power without safety injection. Operators are required to restore power from the switchyard or to load the diesel generators to restore inplant power. SCE's Emergency Operating Instructions on loss of ac power lack guidance on how long operators can attempt to restore power from offsite sources before the diesel generators should be loaded follow-ing a loss of inplant ac power, or how long the diesel generators can run unloaded without overheating, if their ac powered radiator fans remain de-energized.

12.

The station loss of voltage auto transfer scheme for establishing the delayed access to offsite power may not have functioned as designed. SCE evaluations are continuing.

13.

The multiple spurious indications early in the event that a safety injec-tion actuation had occurred, added to the confusion of the situation and unnecessarily increased the burden on the operators. Operators diagnosed plant conditions and appropriately disregarded these indications. The safety injection annuciator will always incorrectly alarm on a loss of ac power. This is a design deficiency.

The cause of the spurious indication on both safeguard load sequencer system panels is still unknown.

14.

The operating staff, with the concurrence of management, did not follow appropriate procedures when troubleshooting the electrical ground. Their actions unnecessarily delayed entry into Technical Specification Action Statement requirements that could require plant shutdown.

15.

Once the electrical ground was located on the feeder from auxiliary trans-former the C to bus auxiliary IC, the operators did not a w essively pursue isolating transformer. Instead, they opted to leave the transformer energized while technicians performed inspections that did not require the transformer to be energized.

16.

The operators' actions, after the transformer trip, were consistent with their training. However, in the Team's judgment, some operators lacked detailed plant knowledge in the following areas:

Cautions associated with paralleling transformers.

Requirements for resetting unit generator trips.

The process for operating 220KV circuit breakers.

Expected indications and timing of the loss of voltage automatic transfer scheme.

Setpoints for residual heat removal system pressure interlock.

Expected indication and meaning of lights on SLSS sequencer panels.

Operability of diesel generators with auxiliary transformer C reactor coil bypass breakers removed.

9-3

These deficiencies may be due to inadequate operator training and/or procedures.

17. On occasion, some site personnel who generally evaluate plant data lacked a sufficiently inquiring attitude. As a result, certain significant indi-cations of underlying reasons for system response or component performance were not detected until brought to the attention of SCE by the Team. It appears that SCE's process for evaluating and following up events may not be sufficiently thorough and systematic to assure that failed components are detected and adequately explained.

18. The status of the steam generator blowdown system is not indicated in the control room. The reestablishment of blowdown when the radiation monitors were reset was not recognized and adversely contributed to the cooldown of the reactor coolant system and to the delay in recovering the steam gener-ator' levels.

19. During the loss of all inplant ac power, sufficient information was avail-able in the control room to enable the operators to follow their procedures and ensure plant safety. However, control room operators had failed to have the Technical Support Center computer reset following electrical ground troubleshooting activities. This failure disabled the computer's ability to record new plant data and thereby denied the operators access to pre-trip and post-trip trends that would have assisted real time and post-event analysis and evaluation. Had the station blackout been of longer duration, or involved additional complications, operator responses and the functions provided by the Technical Support Center could have been hampered by the lack of trend data.

20. Station maintenance records are incomplete, difficult to locate and, when available, lack sufficient detail to determine what was done.

21. The spurious ringing of the NRC red phone at the beginning of the event has not been explained, but it distracted control room personnel and con-tributed to the confusion in the communications between SCE and NRC.

22. ENS communications between NRC and SCE were not effective because: (1) the NRC Duty Officer was not knowledgeable about the unique design of the plant and, therefore, misinterpreted operator responses to questions; (2) commun-ications with the plant were initially limited because statements by plant operators incorrectly implied that sufficient personnel were not available to support the establishment of an open line; (3) NRC asked leading ques-tions and operators sometimes did not correct, and in some cases appeared to confirm, inaccurate information; (4) NRC questions characteristically focused on details rather than on the " big picture"; (5) NRC cluttered the communications channel with repetitive discussions about the sequence of events as additional NRC personnel came on the line to the exclusion of obtaining new plant information; (6) NRC resident inspectors relieved more knowledgeable plant operators as ENS communicators and reestablished com-munications at a location remote from real time plant information; and, (7) plant operators failed to inform the NRC of the declaration of an unusual Event.

23. There were two malfunctions of the automated security access control equip-ment; however, site personnel implemented appropriate planned compensatory measures, thereby precluding a safety-safeguards interface problem.

9-4

c

24. There was-no significant release of radioactivity.

It must be recognized that this' report was compiled prior to completion of all required inspections and evaluations of equipment involved in the event. SCE's continuing diagnostic efforts have unearthed additional information nearly daily; however, this information has been easily integrated into the Team's understand-ing of the incident and in most cases has confirmed long-held hypotheses on the sequence'of events. Future reports from SCE will incorporate the findings of-those studies which are not yet complete.

9-5 L

4 APPENDIX A Memorandum from W. J. Dircks, Executive Director for Operations, to the Commission, November 22, 1985

APPENDIX A

[os =eg'o UNITED STATES

~g E o NUCLEAR REGULATORY COMMISSION

$ e $ WASHINGTON, D. C. 20555

.....* NOV 221985 MEMORANDUM FOR: Chairman Palladino Commissioner Roberts Commissioner Asselstine Comissioner Bernthal Commissioner Zech FROM: William J. Dircks Executive Director for Operations

SUBJECT:

INVESTIGATION OF NOVEMBER 21, 1985 EVENT AT SAN ON0FRE UNIT 1 WILL BE CONDUCTED BY AN INCIDENT INVESTIGATION TEAM (IIT)

At about 5:00am on November 21, 1985, San Onofre Unit 1 experienced a loss of an auxiliary transformer. Subsequently, a partial loss of electrical power occurred and the control room lighting was lost. The reactor was manually scramed which resulted in a short-term loss of all AC power. A sizeable, unisolable leak was then identified in the feedwater system which is used to maintain steam generator levels, and other failures were experienced in the plant equipment. The plant is now in cold shutdown. There were no releases and adequate core cooling was maintained at all times.

Because of the nature and complexity of this event, I have requested AE0D to take the necessary action to send a five member IIT of technical experts to the site to: (a) fact find as to what happened; (b) identify the probable cause as to why it happened; and (c) make appropriate findings and conclusions which would form the basis for any necessary follow-on actions.

The team will report directly to me and is comprised of: Thomas T. Martin, Director of the Division of Engineering and Technical Programs, Region I; Mr. Waynt. Lanning, Chief, Incident Investigation Staff, AE0D; Mr. Steven Showe, Chief, PWR Training Branch, IE - Chattanooga; Mr. William Kennedy, Safety Operational Engineer, Division of Human Factors, NRR; and Mr. Matthew Chiramal, Chief. Engineering Section, AE0D. The team was selected on the basis of their knowledge and experience in the fields of reactor systems, reactor operations, human factors, and power distribution systems. Team members have no direct involvement with San Onofre Unit 1. The team is currently enroute to the site.

A-1

The licensee has agreed to a request by Jack Martin, Regional Administrator, to preserve the equipment in an "as-found" state until the licensee and the NRC Team have had an opportunity to evaluate the event. The licensee has also agreed to maintain Unit 1 in a shutdcwn condition until concurrence is received frem the NRC to return to power.

The IIT report will constitute the single NRC fact-finding investigation report.

It is expected that the team report will be issued within 45 days from now.

Wi iam J. Dircks xecutive Director for Operations cc: SECY OPE OGC ACRS OPA Regional Administrators l

l l

A-2

APPENDIX B 1

Plant Conditions When Water Hammer I Occurred and Estimated Piping Support Loads l

l

w 4

APPENDIX B i

This appendix deals with voiding conditions when wat.~ hammer occurred at SONGS-1, the refilling process and the estimated war.(.. hammer loads that resulted as based on' analyses of damage to piping supports. These estimates have been used to develop the findings reported in section 6.

PLANT CONDITIONS WHEN WATER HAMMER OCCURRED Estimated Voiding of Loop B The void present in loop 8 feedwater pipe downstream of motor-operated isolation valve MOV-20 when water hammer occurred was estimated several ways:

1. Sequence of events and volumes

2. Hydrodynamic instability 3.

Evidence of water hammer load at FWS-378 j

The first method relies to a large extent on operator recollections and fill volume calculations; the second method relies on current theories related to steam-condensation water hammer phenomenon and the flow instabilities leading to steam pocket collapse; the third method relies on calculations of the loads required to plastically yield (i.e., stretch) the bonnet studs of FWS-378 (the 4-inch check valve in the Loop B flow control train). This last method is the most reliable. Determining the void fraction is necessary for estimating steam-condensation water hammer loads. Since the water hammer load that affected FWS-378 was a traveling wave (i.e., " classical" water hammer) the extent to which the bonnet studs were elongated can be used to back calculate the impulse wave and the reflected wave.

Sequence-of-Events and Volumes Method The lower estimate of void fraction can be calculated assuming that:

1. MOV-20 was closed at 04:55 2.

AFW flow had increased to 155 gpm* (two AFW pumps were operating)

3. AF.

flow was reduced to zero at 05:00 and then reset to 41 gpm at 05:00:15

4. The feedwater pipe downstream of MOV-20 was not completely voided when MOV-20 closed

5. The water hammer occurred at 05:07.

• Based on correcting indicated values from flow calibration data found after the event.

B-1

The total injected AFW is calculated to be 1051 gallons; the piping volume downstream of MOV-20 is calculated to be 860 gallons. These calculations indicate a full line when the water hammer occurred, that is, a zero percent void fraction.

The upper estimate of void fraction can be calculated from the following assumptions:

1. MOV-20 was closed at 04:55  !

2. The AFW flow was 135 gpm

3. AFW flow was reduced to zero at 04:59:30 and restored to 25 gpm at 04:59:45

4. AFW flow was constant at 25 gpm* until the water hammer

5. The feedwater line downstream of MOV-20 was completely empty when MOV-20 closed.

The volume of AFW injected under these assumptions is 789 gallons, or a liquid volume fraction of 84.5 percent.

This method yields an estimated void fraction of zero to 15.5 percent.

Hydrodynamic Instability Methods The refilling of the feedwater piping is a transient controlled by hydrodynamic instabilities. Initial refill conditions will be determined by whether or not the pipe " runs full," which is a function of AFW injection flow rates.

Experiments have show (Reference 1) that a critical Froude number must be reached for the pipe to run full (see below). This was not the case for 150 gpm injection flow and, therefore, initial AFW injection filled the lower portion of the horizontal feedwater pipes for all FW flow circuits. In other words, steam existed along the top of the entire line as cold AFW filled up the bottom.

The simultaneous presence of steam and colder water in the horizontal FW pipe results in mass and energy transfer taking place; this transfer results in condensation on the water surface and the pipe wall. The colder water in the bottom of the pipe acts as a heat sink drawing heat from the top of the pipe; the pipe acts as a conducting fin drawing the latent heat of condensation to the colder bottom where the cold AFW is laying. As steam condenses locally, it is replenished from the steam generator (SG) and hydrodynamic instabilities are et up on the water surface. As the water level in the pipe rises (refilling i continuous following closure of MOV-20), a more pronounced surface hydraulic interaction is set up, with transition from stratified flow occurring (Figure 6.1.c). As the void fraction decreases, the backflow of steam corresponding to condensation on the water surface and pipe wall will become high enough to result in a transition to slug flow; this condition is called a critical void fraction.

Two correlations available for estimating this critical void fraction as a function of steam flow are the Taitel-Duckler (Reference 2) and Wallis-Dobson (Reference 3) correlations. These correlations are fundamentally the same and since the Wallis-Dobson correlation has been used in previous NRC studies

• 0perator observed value.

B-2

related to SGWH (Reference 4), this correlation was used to estimate the void fraction existing at the time of water hammer occurrence.

A lower estimate of critical void fraction of 4 percent was calculated, (assum-ing condensation only on a semi quiescent water surface). Including condensation on the upper pipe wall in the calculations raised the estimated critical void fraction to 15 percent. Both estimates (Reference 5) were based on hand calcu-lational models, which will overestimate void fraction. (A calculation taking into account time and pipe position transients, including pipe wall heat capa-city effects, would be required to refine these estimates.) .It should also be noted that these void fraction estimates are based on the total horizontal pipe length (i.e, the 203 feet of B feedwater line between the vertical elbow next to the steam generator back to FWS-346).

Limiting Load Method The water hammer force was back-calculated at FWS-346 by estimating the force necessary to stretch the bonnet bolts approximately 0.5 inches; that force would require an internal pressure of 16,000 psi. Since water hammer wave reflection will essentially double the load (References 6 and 7), the initial impact pressure from a traveling slug was estimated to have been 8000 psi.

This load would correspond to a void fraction of 22 percent. Other calculations related to structural support damage (see above) revealed lower forces and support an estimated void fraction of less than 1 percent when the water hammer occurred.

In summary, these three methods arrive at overlapping values of void fraction, with a probable range of between 1 and 15 percent and support the hypothesis that the water hammer occurred just about when the Loop B horizontal piping run was nearing complete refill.

Flow Conditions When Water Hammer Occurred For steam condensation-induced water hammer to occur, several hydrodynamic phenomena must precede it. First, the pipe must fill (this is a function of refill rate ard whether the pipe will run full), the steam-water interface (which controls the rate of heat transfer between the steam and cold water),

the hydrodynamic flow conditions which change as the pipe fills (see Figure 6.1) from stratified flow to slug flow. Eventually a steam pocket is entrapped, which then collapses, and accelerates a water slug in the direction of pressure inbalance.

The SONGS-1 water hammer can be attributed to the refill transient (e.g. , the {

time required to refill the voided lines and existing flow conditions and l operator actions. Initially, the AFW flow rate (after the MOVs were closed) was 155 gpm. This flow rate is too low to maintain full pipe flow and can be deduced from Froude number considerations. The Froude number (a dimensionless parameter) is the ratio of fluid inertial forces to gravity effects and has been long used to model wave effects (i.e., the " hydraulic jump" phenomena, or wave cresting in open channel flow). The Froude numbe. necessary to fully fill a circular horizontal pipe (with some water running ahead of the filled section) is approximately 0.5 (Reference 1).

B-3

.. _ _ _ _ o

For an AFW injection flow rate of 155 gpm, the Froude number is 0.13 (averaged over the total FW pipe cross-sectional area) and is 0.02 for an AFW flow rate of 25 gpm. Thus, the FW pipe (which is horizontal) will not run full during the refill transient (see Figure 6.6) and the variation of AFW injection during the time preceding the water hammer is important.

The'AFW piping refill transient can be examined simplistically as a filling of voided pipe volumes. During the 5 minutes following closure of MOVs -20,

-21, and -22, more than sufficient AFW had been injected to totally fill loops A and C. However, the horizontal piping run in loop B was only 90 percent full (Table B-1) and at this point the operators reduced AFW rate to zero and then back to 41 gpm.

Loops A and C would have been completely filled in about 3.5 minutes since they have about 50 percent of the volume of loop B. Thus, this simplistic fill vol-ume approach correlates well with the fact that operators throttled back AFW at about 05:00 following detection of overcooling of the reactor coolant systems.

SGs A and C were receiving cold water and reinitiating cooldown (all three SGs had essentially boiled dry previously).

Reducing the AFW flow rate to 41 gpm had several adverse ef fects on the con-tinued refilling of loop B: (1) reverting the refill process to a quiete'r (or more gradual) hydraulic condition, (2) allowing the steam void at the top of the pipe to propogate backwards to MOV-20 and (3) allowing the cold AFW to stay in contact with the hot steam for a longer period of time. Following resump-tion of AFW injection, calculations indicate that loop B would have been com-pletely filled at about 05:04, suggesting that a steam bubble may have been trapped during the refilling and that the bubble collapsed later.* The water hammer occurred at 05:07.

Examination of flow conditions existing at the vertical upturn elbow region at these low Froude numbers (Fr = 0.03 at 41 gpm) is important to an understanding of local flow conditions just prior to the water hammer. The low Froude num-bers which enhance a quiet filling condition versus slug flow may be-a signifi-cant factor in averting water hammer in loops A and C. SCE provided informa-tion (Reference 8) related to air-water tests conducted at Creare, Inc. which were designe' M simulate refilling of the horizontal FW pipe at SONGS-1.

These exper p- showed that for Froude numbers of 0.02 to 0.12:

As the liquid level neared the top of the pipe (void fraction of a few percent) the air gap at the vented end of the pipe is bridged by a single slug and water immediately starts filling the vent riser. Most of the remaining air stays trapped in the pipe when filling is. continued.

This quiescent refilling, coupled with rapid bridging of the elbow region to fill the vertical pipe at low void fractions (i.e. , < 10 percent), is likely the reason that loops A and C did not experience a water hammer. At SONGS-1 the steam pocket could have been swept out due to a not perfectly horizontal pipe run. The Creare air-water tests were very carefully run to ensure a perfectly level horizontal pipe run.

• Uncertanties in the calculation assumptions may have contributed to this result.

B-4

~

The total stoppage of AFW and resumption at 41 gpm when loop 8 was 90 percent full delayed fluid bridging of the elbow and increased the time that steam and cold water remained in contact. Whether total flow stoppage or the significant AFW flow reduction were the critical factors leading to a water hammer cannot be determined without a refined thermal-hydraulic refilling analyses. Nonethe-less, operator actions which delayed the total refilling of loop B enhanced the probability the steam condensation-induced water hammer would occur.

The hypothesized loop B refill conditions prior to the occurrence of water ham-mer are shown in Figure 6.6. A 0.5-inch gap at the top of loop B horizontal FW pipe corresponds to a void of approximately 2 percent. The actual position of the steam pocket at the time of collapse is unknown. However, pipe displace-ments and loads experienced at the damaged supports suggest that the slug was formed in the horizontal run of loop B piping just upstream from the vertical piping run and then accelerated upstream. This postulated low void fraction condition is supported by the calculated structural damage loads discussed below.

Another well point beyond of interest is that conditions conducive to water hammer existed 05:07. Feedwater leakage was manually isolated at 10:45 on November 21, 1985 (Table 3.1). Although backflow of steam for loops A and C would have been blocked by the closure of the MOVs, the failed gasket in 4-inch check valve FWS-378 continued to provide an open backflow path until manual isolation (via closure of FWS-376) was accomplished.

COMPARISON OF ESTIMATED LOADS WITH DAMAGE INCURRED Souther California Edison (SEC) provided the Team with the following comparison of estimated water hammer loads with observed and measured damage incurred (Reference 9).

data taken following the water hammer incident. Figure 6.62 shows measured piping Characteristics of Damage Forces Based on the survey of the horizontal pipe displacements between the as-found and the design configurations (Figure 8.1), it is obvious that the major force exerted on the pipe is opposite to the direction of normal feedwater flow. This force is caused by the water hammer induced pressure wave propogating in the direction opposite that of the normal feedwater flow. A reflected pressure wave was generated following transmission of the first wave which loaded valves FCV-457 and FWS-378.

From Figure B.1 it is observed that the largest displacement is along the longest run of the pipe (between support location 100 [H00G] and 140

[H00K].) This displacement is a result of the large impulse load in this section of the line, coupled with increased line flexibility after the support damage done by the water slug. It is also in this region that the i

FW piping material incurred an 80-inch axial crack. The crack started on the outside of the pipe and had an approximate 25 percent thru wall penetration [from the outside wall (Figure 6.8)].

Pipe Support Damage The minimum forces needed to damage some of the pipe supports and the maximum forces that some intact pipe supports can withstand are estimated in this section.

B-5

This estimate provides an upper bound and a lower bound of the force exerted on the pipe during the water hammer event. The pipe support locations and identification numbers for various suppcru are shown in Figure 6.7. A brief description of the calculated loads and findings is provided below.

1. H00A - Based on the fact that the base plate was pulled out from the wall and the welded dummy support (to which the snubber is attached) was not damaged, it is estimated that the force parallel to the snubber axis was between 40,000 to 90,000 lbs.

This translates into a force of 45,000 lbs. to 101,000 lbs.

parallel to the pipe axis.

2. H006 - Based on the fact that the snubber was functionally tested and found to be damaged, it is estimated the force 3 exerted parallel to the axis of the snubber is 40,000 lbs.

This translates into a force parallel to the pipe of at least 44,000 lbs.

3. H00J - Based on the fact the dummy support was sheared from the pipe, the force parallel to the pipe is estimated between 62,000 lbs. and 164,000 lbs., depending on the manner of' loading.

4. HOOK - Based on the severe damage of the pipe guide, the force I needed to deform the structure is estimated to be around 112,000 lbs. parallel to the axis between support locations 150 and 170.

However, because the energy loss in collapsing the structure and creating the dent is not considered in the analysis, the actual force may be as high as 180,000 lbs.

5. HOOL - Based on the fact that the dummy stub was sheared off and the various base plates were pulled off or damaged during the event, the minimum force needed to cause such damage ranges from 36,000 to 82,000 lbs, depending on the nature of the loading.

Timing History Analysis The time-history analysis is another way to estimate the force exerting on the pipe. This method is an iterative process, starting with an estimated force and then comparing the calculated horizontal pipe displacement with the observed horizontal displacement. The solution is the force at which the calculated displacement is close to the observed displacement.

The time-history analysis performed by Bechtel (Reference 9) showed that the impulse force, which resulted in a 12-inch displacement of the long north-south run of pipe, is about 160,000 lbs.

Analysis Results Based on the time-history analysis and the observed damage to support HOOK, it is clear that the force exerted on the segment of the horizontal pipe between the locations 150 and 180 ranges from 112,000 lbs. to 180,000 lbs. Also, the maximum force exerted on the pipe segment, to which the supports H00A and H006 are attached, is limited to 101,000 lbs. Based on B-6

the experiments done by Swaffield and Phil (Reference 10) the pressure wave transmits only 90 percent of its strength downstream of a 90 degree pipe elbow. Also, the pressure wave tends to decrease as the energy of the impulse is dissipated during propogation. Therefore, the forces exerted on H00A, H006, and HOOK are consistent and are probably caused by the same pressure wave propogating opposite to.the direction of the feedwater flow.

Assuming the energy loss due to damage on the' pipe support system between the locations of pipe support HOOK and check valves FWS-346 is negligible, the force at the location of FWS-346 can be estimated by increasing the force at the location 160 by 10 percent. This 10 percent is again based on the data discussed in Swa1 field and Phil (Reference 10). This increase will result in a force of 123,000 to 198,000 lbs. at FWS-346. The corre-sponding impact pressure for this force ranges from 1600 to 2600 psi.

Using the methods developed by the CREARE in NUREG-0291'(Reference 4) for NRC's prior water hammer analysis, an impact pressure of 1600 to 2600 psi would be equivalent to a void fraction of 1.5 percent to 2.5 percent, as-suming the subcooling of the slug formed by steam-condensation was in the range of 85 'F to 200 F.

The data related to the dents around the first elbow of the horizontal pipe (at the NE corner) and around pipe support location 140 were not used in the above analysis because of the uncertainties in determining the force needed to cause indentation resulting from impact against the concrete.

This large uncertainty is composed of various analysis uncertainties re-lated to pipe property, force direction, plastic deformation model, and concrete characteristics before and after the impact.

Therefore, based on this more detailed analysis of the damage forces calculated at various damaged pipe supports, it was concluded that:

1. The force exerted on the pipe between support locations 40 and 90 ranges from 45,000 to 101,000 lbs. The actual force was probably closer to 101,000 lbs. than to 45,000 lbs.

2. The force exerted on the pipe between locations 150 and 180 ranges from 112,000 to 180,000 lbs.

3. The impact pressure of a water slug, or the surge pressure of the pressure wave, is estimated to range from 1600 to 2600 psi.

4.

The void fraction in the main feedwater line at the time of water hammer is estimated to range from 1.5 to 2.5 percent, depending on the subcooling of the water slug. This void fraction is estimated for a total FW pipe length of 203 feet.

These calculations further refine the void fraction estimates discussed above and indicate that very low void fractions existed in the horizontal line just upstream of the vertical elbow leading to steam generator B (i.e., 1-2 percent void).

8-7

Table B.1 SONGS-1 Feedwater Flow System Parameters and Volumes Design Parameter Data Feedwater Pipe Schedule (Loops A,B,&C) 10-inch, Schedule 60 Inside Diameter 9.75 inches Estimated Length of Horizontal Pipe:

Loops A & C 118 feet Loop B 221 feet Estimated Volume of Horizontal Pipe:

Loop A 459 gal.

Loop B 857 gal.

Estimated Volume of Vertical Pipe Rise at SG B 60 gal.

Estimated Refill Characteristics Loop B:

04:55 to 05:00 @ 155 gpm = 775 gals > 857 gal.

Est'd void fraction @ 05:00 = 10% in Horizontal Pipe 05:00 to 05:07 @ 41 gpm = 236 gal.

Total Volume Injected = 1011 gal. > (857+60); pipe is full Loop A or C:

04:55 to 05:00 @ 155 gpm = 775 gal. > 459 gal.

> 459+60 gal. i B-8

References

1. Wallis, G. B.,

et al., " Conditions for a Pipe to Run Full When Discharing Liquid into a Space 1977, pp. 405-413.

Filled with Gas," Journal of Fluid Dynamics, June 2.

Y. Taitel and A E. Duckler, "A Model for Predicting Flow Regime Transitions in Horizontal and Near Horizontal Gas-Liquid Flow," AICHE Journal, Vol. 22, pp. 47-55.

3.

G. B. Wallis and J. E. Dobson, "The Onset of Slugging in Horizontal Stratified 1973.

Air-Water Flow," J. of Multiphase Flow, Vol 1, pp.173-193, 4.

J. A. Block, et al., "An Evaluation of PWR Steam Generator Water Hammer,"

NUREG-0291, U.S., NRC, 1976.

5.

Letter from F. M. Joos (CREARE, Inc.) to C. Chiu (Southern California Edison) on December 16, 1985. Exhibit 427.

6.

H. Rouse, Engineering Hydraulics, John Wiley & Sons, Inc., New York, 1949.

7. V. C. Streeter and E. B.- Wylie, Hydraulic Transients, McGraw-Hill Book Company, New York, 1967. '

8.

Joos, 1986.

F. M. (CREARE, Inc. ), to Chiu, C (SCEC) letter dated January 6, i I

9.  !

Bechtel Power Corporation Calculations MC-077-002, " Hydraulic Transient Load on MFW and AFW Piping at Vicinity of SG-8," December 15, 1985.

Exhibit 495.

10. Swaffield, J. A. and Phil, M., "The Influence of Bends of Fluid Transients Propagated 603-14, Vol. in Incompresible Pipe Flow," Proc. Instr. Mech. Engrs., pp.

183.

(

B-9

s' f

% / \

$ /

\

/ /I

\ /g \ #

8

/

\l g#

/ f l5 I "

I /\

m iE f

3

\ / 'l \/ i

\

9 I% i s l gs =

\ /

/

I

\ / I [

,i a

\ i 2

\ e

/

\ li I

i

1. llgI

' 5

' hSh')'g -

e 3 i B-10

APPENDIX C Regulatory Review of the Potential for Water Hammer at San Onofre Unit 1 t .

APPENDIX C NRC's concerns related to the potential for water hammer at SONGS-1 date back to 1975. The correspondence summarized below provides a historical perspective of these concerns.

5/13/75 - NRC informs the licensee (Southern California Edison) that the poten-tial and consequence of secondary system water hammer needs to be analyzed; that uncovering the steam generator feedring and subsequent operation of the

' auxiliary feedwater system may cause a water hammer; that changes in plant design or operation necessary to prevent water hammers or assure system inte-grity should be identified for NRC evaluation.

7/14/75 - SCE informs the NRC staff that changes in design or operation of the feedwater system are not necessary; that feedwater flow is continued through periods of low steam generator level; that the layout of the feedwater lines external to the steam generators will minimize the magnitude of a water hammer; that auxiliary feedwater flow is not automatically initiated; that analysis of feedwater piping using dynamic forcing functions modeling water hammer j phenomena have not been performed.

{

l 9/2/77 - NRC informs SCE of the staff conclusion that keeping steam generator feedwater lines and feedwater spargers full should preclude the occurrence of .

water hammer and that SCE should propose plant design and procedural modifica-tions to minimize probability of SGWH.

12/27/77 - SCE informs staff no modification to plant design is warranted and that administrative controls wii! be imposed on operations to assure feedwater is added slowly af ter the feedwater sparger is uncovered.

5/25/79 - NRC requests information on the design of feedwater lines including questions on the history of feedwater line water hammers.

6/18/79 -SCE provides requested information.

7/3/79 - SCE provides additional information on feedwater line water hammer experiences to supplement response of 6/18/79.

8/2/79 - NRC requests that SCE provide additional information pertaining to susceptability of plant to SGWH per telephone conversation.

8/31/79 - SCE provides requested information on susceptability to SGWH and indicates flow meter indication at low flows is not available.

9/12/79 - NRC acknowledges plant operating history does not show that SGWH has occurred at the plant, but requests additional information to provide further assurance that SGWH would not occur in the future and that surveillance procedures would be adequate to detect water hammer or damage from water hammer, if it were to occur.

C-1 APPENDIX C

2/14/80 - SCE responded to the NRC request and informed the staff that the steam generator feedwater spargers have been uncovered many times without SGWH; that administrative controls are in place to reduce the frequency of uncovering the spargers; that transient and accident analysis were not affected by these con-trols; that the impact of automating the auxiliary feedwater system would be evaluated; that visual inspections would be conducted if water hammers occur; that there had been no loss of offsite power with the plant operating; and that further evaluations of the potential for an appropriate corrective action for water hammer would be performed.

4/15/80 - SCE confirms discussions with the NRC staff that the evaluation to be performed by SCE of water hammer would not include SGWH, but would include the development of forcing functions for classic water hammers, such as valve closure and pump start.

4/22/80 - NRC forwards the safety evaluation report (SER) relating to the potential for water hammer in plant feedwater lines and documents the SCE commitment to evaluate further the potential magnitude of water hammers. The SER concludes that the potential for SGWH is sufficiently low to permit con-tinued plant operations. The basis for this conclusion is stated to be a review of operating history and related operational and procedural characteristics of the feedwater system which showed that although conditions conducive to SGWH l have been encountered, SGWH had not occurred.

10/16/80 - SCE informs NRC of plans to automate and upgrade the auxiliary feedwater system in response to TMI lessons learned requirements. A discussion of revised administrative controls to prevent SGWH were not included since they .

were discussed in the correspondence of 2/14/80. l 11/25/80 - SCE provided its promised evaluation of the effects of " classic" type water hammers on feedwater piping in the plant. It concluded that classical water hammer has no significant effect on piping stress and support loads and that existing administrative controls are adequate.

3/6/81 - SCE provides an evaluation of the potential for SGWH with the automated auxiliary feedwater system; indicates that uncovering the feedring cannot be prevented; that flow limits are required because conditions conducive to SGWH cannot be eliminated for the auxiliary feedwater system; that new flow meters enabled improved administrative controls on flow rate; and, that auto-mation of the auxiliary feedwater system with these controls does not increase the probability of inducing SGWH.

3/82 - NRC issues NUREG-0918, which summarized the resolution of the concern for SGWH at operating pressurized water reactors. It stated, in part, that:

San Onofre 1 has short horizontal feedwater pipe (less than 3 feet) leading to the SG inlet. SGs still use the " unmodified" feedring with bottom-discharge holes. The auxiliary feedwater flow at the plant can only be started manually: this allows the plant operator to feed the SGs with heated main feedwater whenever possible.

The staff has accepted the present implementation at San Onofre 1.

However, this matter will be reexamined if any SGWHs occur at the plant in the future.

C-2 APPENDIX C

1 (NUREG-0918 referenced the NRC staff SER produced in April 1980. )

It is clear from reviewing the prior correspondence cited above, and from followup interviews with NRC staff who were involved in previous SONGS-1 water hammer reviews, that the thrust of NRC concern was directed at the prevention and mitigation of consequences of SGWH and not at preventing gross voiding of the FW lines.

SONGS-1 AFW System Evaluations The potential for cold AFW injection to produce SGWH was included in the NRC and SCE water hammer evaluations discussed above and cited in Section 6.

Enclosure 2 of Reference 22 of Section 6 provides a review of those evaluations 4 and reaches the conclusion that establishing an upper flow rate limit (i.e.,

150 gpm/SG) on AFW (as recommended in Westinghouse's Bulletin 75-7, Reference 23 of Section 6) would prevent too rapid an injection of cold water into the feedring and, therefore, avoid SGWH. Operators were cautioned in the E0Is not to exceed this upper AFW flow limit whenever the steam generator feedring was uncovered so as not to set up conditions for water hammer.

In 1979, as a result of NRC's Bulletins and Orders Task Force review of operating reactors following the TMI-2 accident, SCE was requested to provide information regarding AFWS flow requirements at SONGS-1 (Reference 1).

Enclosure 1 of Reference 2 discusses the basis for AFWS flow requirements and includes plant transient analyses dealing with the following transients and accidents identified by NRC staff:

1. Loss of main feedwater (LMFW)

2. LMFW with loss of offsite ac power

3. LMFW with loss of onsite and offsite ac power

4. Plant cooldown

5. Turbine trip with and without bypass

6. Main steam isolation valve closure

7. Main feed line break

8. Main steam line break

9. Small break LOCA

10. Other transient or accident conditions not listed above.

It should be noted that none of these analyses considered the additional compli-cations which might arise from failure of feedwater system check valves, includ-ing the potential blowdown of steam generators or the failure of the low pres-sure piping in the feedwater system.

NRC completed review of AFWS automatic initiation and flow indication (THI Action Plan Item II.E.1.2) for SONGS-1 in 1982 (Reference 3) and approved the AFWS Technical Specifications in 1984 (Reference 4).

The Team met with SCE staff on December 13,1985 (Reference 5), at which meet-ing SCE reviewed the SONGS-1 AFW system in terms of the original designs, up-grades (i.e. , seismic upgrades), TMI Action Plan requirements, etc. These dis-cussions also delved into prior water hammer occurrences and SGWH evaluations.

Although this meeting did not reveal any significant new information, it again illustrated the belief that SGWH would not occur within the steam generator, since it had never occurred at SONGS-1, despite numerous transients involving uncovered feedrings, and limits on AFW flow rate were sufficient to preclude such an occurrence.

C-3 APPENDIX C

References

1. D. G. Eisenhut, NRC to J. H. Drake, SCEC, " Auxiliary Feedwater System Flow Requirements," November 15, 1979.

2. K. P. Baskin, SCEC, transmittal to D. M. Crutchfield, NRC, " Automatic Initiation of Auxiliary Feedwater System SONGS-1," March 6,1981.

3. D. M. Crutchfield, NRR to R. Dietch, SCEC, " Auxiliary Feedwater System Automatic Initiation and Flow Indications (TMI Action Plan Item II.E.1.2),

November 18, 1982.

4. W. A. Paulson, NRC to K. P. Baskin, SCEC, " Auxiliary Feedwater System Technical Specifications," October 21; 1984. ,

5. Transcript (Set 3-044), December 13, 1985 Meeting at Rosemead, California.

C-4 APPENDIX C

U S NUCLt AM HE GUL ATOa. COMui55 SON REPOR T NUMutu f.ssgaea e, TsOC. eed vos JWe , se ea, A

. .a FORM 335 asnC. i Bell 2RAPHIC DATA SHEET NUREG-1190

> t ...a.

3 TITLE ANO SU8TsTLE e HECne,t h,4 ACCT 55 tun NuwetM

/

Loss of Powe and Water Hammer Event at San Onofre, Unit 1 * "" "'"

" ' P"" "

on November , 1985 " "

/

r l " '"

. &u f ,.0a.5, January 1986

, oAu ..,fe r 55u o

-ONr f lu Aa Jany6ry 1986 9 PHO Cie T A5miWOMn UNI, NUM9E R

$ 9 E*FORutNG ORGANilATION N AME ARio MAeLaNG ADDHE 55 fiar*wer /* Codrs Incident Investigation Team Executive Director for erations [ , ,, ,,,, ,,

i U.S. Nuclear Regulhtory ommission Washington, D.C. 20555

,, SPON50RsNG OHGANs2 Af SON N Auf AND W AsteNgDont s5 tsachee le Cees

'/ 1 24 f.PE OF HEPOR T Incident Investigation Team \

Executive Director for Operat' ions Technical U.S. Nuclear Regulatory Commis'. ion 'i= Piaim Covaatou-~ ~ ~<d Washington, D.C. 20555 n suu u . NTA.,Nori5 iA A.5raACT,m e, ,

T On November 21, 1985, Southern California dison's San Onofre Nuclear Generat-ing Station, Unit 1, located south 'of San flemente, California, experienced a partiallossofinplantacelectricaipowdrwhiletheplantwasoperatingat g

60 percent power. Following a manual re/ctor trip, the plant lost all inplant ac power for 4 minutes and experienced' severe incidence of water hammer in the feedwater system which caused a le ( damaged plant equipment, and chal-lenged the integrity of the plant's he ' sink. The most significant aspect of the event involved the failure of fivg safgty-related f check valves in the feed-water system whose failure occurred ip less\than a year, without detection, and jeopardized the integrity of safety yystems.\The event involved a number of equipment malfunctions, operator errors, and procedural deficiencies. This report documents the findings and cdnclusions hf an NRC Incident Investigation Team sent to San Onofre by the NRCfxecutive Diepctor for Operations in confor-mance with NRC's recently establis ed Incident In estigation Program.

15e EG Y WORDS AND DOCout er ANAL.585 15h DE SCHeP IONS

\

loss of power lp water hammer j San Onofre, Unit 1 j ll 4

3

.. A... A.,<,,.5,A,.. N,

, , 5g'u,..

, . , u . s 5.. ,< a ,,eN ,, No .. ~ ...s.5 Uncia s,,s,i fied Unlimited , , 5, Co.. . . C , .s , . .C A , ,o, o P,m ,

UEdEssified s

UNITED STATES sncea ,coaruet,ssnat.

m5m

• HE5 NO

- NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555 g;{Egy OFFICIAL BUSINESS PENALTY FOR PRIVATE USE. 4300 p'AR{078877 1

POL I l tr 1 g ,,, " ' on tiyj?y ' opvol'ec

,B soc

,G T og, R.pg kURtc CC 2055$

c __ - _ _ _ - _ _ - _ _ _ _ _ - _ - _ _ _