ML20134A814

Loss of Main and Auxiliary Feedwater Event at the DAVIS-BESSE Plant on June 9, 1985

ML20134A814
Person / Time
Site:	Davis Besse
Issue date:	07/31/1985
From:	NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO)
To:
References
NUREG-1154, NUDOCS 8508150428
Download: ML20134A814 (103)

Text

,

.. s l'

NUREG-1154 Loss o" Main and Auxiliary Feedwater Event at the Davis-Besse Plant on June 9,1985 U.S. Nuclear Regulatory Commission g tka "' G%

+

s..../

l SS0**A0$$R$!80$$46 5 PDR

I l

t ,

i, 1

NOTICE Availability of Reference Materials Cited in NRC Publications Most documents cited in NRC publ; cations will be available from one of the following sources:

1. The NRC Public Document Room,1717 H Street, N.W.

Washington, DC 20555

2. The Superintendent of Documents. U.S. Government Printing Of fice, Post Office Box 37082 Washington, DC 20013 7082

3. The National Technical Information Service, Springfield, VA 22161 Although the listing that follows represents the majority of documents cited in NRC publications, it is not intended to be exhaustive.

Referenced documents available for inspection and copying for a fee from the NRC Public Docu-ment Room include NRC correspondence and internal NRC memoranda; NRC Office of inspection and Enforcement bulletins, circulars, information notices, inspection and investigation notices; Licensee Event Reports; vendor reports and correspondence; Commission papera; and applicant and licensee documents and correspondence.

The following documents in the NUREG series are available for purchase from the NRC/GPO Sales Program: formal NRC staff and contractor reports, NRC-sponsored conference proceedings, and NRC booklets and brochures. Also available are Regulatory Guides, NRC regulations in the Code of Federal Regulations, and Nuclear Regulatory Commission Issuances.

i l Documents available from the National Technical Information Service include NUREG series reports and technical reports prepared by other federal agencies and reports prepared by the Atomic Energy Commission, forerunner agency to the Nuclear Regulatory Commission.

Documents available frcn public and special technical libraries include all open literature items, such as books, journal and periodical articles, and transactions. Federal Register notices, federal and state legislation, and congressional reports can usually be obtained from these libraries.

Documents such as theses, dissertations, foreign reports and translations, and non-NRC conference proceedings are available for purchase from the organization sponsoring the publication cited.

lingle copies of NRC draf t reports are available free, to the extent of supply, upon written request to the Division of Technical information and Document Control, U.S. Nuclear Regulatory Com-mission, Washington, DC 20555. l Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are maintained at the NRC Library, 7920 Norfolk Avenue, Bethesda, Maryland, and are available there for reference use by the public. Codes and standa ds are usually copyrighted and may be purchased from the originating organization or, if they are American National Standards, from the l American National Standards Institute,1430 Broadway, New York, NY 10018.

Loss of Main and Auxiliary ,

Feedwater Event at the Davis-Besse Plant on June 9,1985 Manuscript Completed: July 1985 Date Published: July 1985 U.S. Nuclear Regulatory Commission Washington, D.C. 20555 p.....,,

k /

1

~l: l L

l 1

ABSTRACT On June 9, 1985, Toledo Edison Company's Davis-Besse Nuclear Power Plant, located in Ottawa County, Ohio, experienced a partial loss of feedwater while the plant was operating at 90% power. Following a reactor trip, a loss of all feedwater occurred. The event involved a number of equipment malfunctions and i extensive operator actions, including operator actions outside the control room.

Several operator errors also occurred during the event. This report documents j the findings of an NRC Team sent to Davis-Besse by the NRC Executive Director for Operations in conformance with the staff proposed Incident Investigation Program.

I J

l i

1 i

i 1

l I

1 i

i iii i

i --

/

TABLE OF CONTENTS I

Page I Abstract . . . . . . . . . . . . . . . . . . . .. .... . . iii The NRC Team for the Davis-Besse Event of June 9, 1985.... ...... vii Acronyms . . . . . . . . . . .. . .. . . . . ... . .. . . . ix

1. Introduction. . . . . . . . . ... . . . ... . .. . . . 1-1

2. Description of Fact Finding Efforts . . . . . .. . .. . . . 2-1 2.1 General . . . . . . . . . . .. . . . . . .. ... 2-1 2.2 Interviews and Meetings . .... . . . .. . . . . . . 2-1 2.3 Plant Data. . . . . . . . . .. . . . . .. . . ... . 2-3 2.4 Quarantined Equipment . . . .. . . . .. . .. .. . . 2-3

3. Narrative of the Event . . . .. . . . ... . . .. . . 3-1 3.1 Shift Change . . .. .... . . . .. . .. . . . 3-1 3.2 Reacter Trip-Turbine Trip .. . . . . . . . . . . . . 3-2 3.3 Loss of Main Feedwater . . . .. . . . . . .. . . . . 3-4 3.4 Loss of Emergency Feedwater . . . . . . .. . . . . . 3-4 3.5 Reactor Coolant System Heatup . . . . ... . . . .. . 3-6 3.6 Operator Actions . . . . . . . . . . . . .. . . .. . 3-6 3.7 PORV Failure . . . . .. . . .. .. . .. . . .. . . 3-9 3.8 Steam Generator Refill . . . . . . . . . . . . . . 3-10 3.9 Emergency Plan. . . . . . .. . . .... .... . 3-12

4. Description of Plant Systems. . .. . . . ... . .. . . . 4-1 4.1 General Design . . . . . ... . . . ... . .. . . 4-1 4.2 Main Steam System. . . . .. . . .. . . . .. . . 4-1 4.3 Main Feedwater System. . . ... . . . . . . . .. . . 4-2 4.4 Auxiliary Feedwater System . .. . .. . . . . . . . 4-3 4.5 MU/HPI Cooling Systems . . . .. . . . . . . . . . .. . 4-4 4.6 Steam and Feedwater Rupture Control System (SFRCS) . . . 4-5 4.7 Pressurizer Pilot-Operated Relief Valve (PORV) . . . . . 4-6

5. Equipment Performance . . .. . .. . .. . ... . . . . . 5-1 5.1 Pre-existing Conditions . . .. . . . .. . .. . . . . .5-1 5.1.1 Safety Parameter Display System . . . .. . . . .5-1 5.1. 2 Source Range Nuclear Instrumentation . . . . .. .5-1 5.1.3 Startup Feedwater Pump .. . . . .... . . . . .5-2 5.1. 4 Control of Main Feedwater Pumps . . . .. . . . . .5-2 5.1.5 Flux / Delta Flux to Flow Reactor Trip Instrumentation . . . . .. . . ... . . . . . . 5-3 v l l

TABLE OF CONTENTS (Continued)

Page 5.2 Equipment Problems That Occurred During Event. . .... 5-3 5.2.1 Control of Main Feedwater Pumps . . . . . . . . . 5-3 5.2.2 Closure of Both MSIVs, Spurious SFRCS Actuation . 5-4 5.2.3 Main Steam Safety Valves, Atmospheric Vent i Valves .... ....... ... . ...... 5-5 5.2.4 AFW Trains Nos. I and 2 Turbine Overspeed Trips . 5-6 5.2.5 AFW Containment Isolation Valves . . .. . . . 5-8

5. 2. 6 Main Steam Supply Valve to AFPT No. 1 . ... . 5-10 5.2.7 Source Range Nuclear Instrumentation . .. . 5-11

5. 2. 8 Pilot Operated Relief Valve (PORV) ...... . 5-11 5.2.9 Startup Feed Control Valve for OTSG No. 2 . .. 5-13 5.2.10 Recovery and Control of Both AFW Turbines . . . . 5-13 5.2.11 AFW No. 1 Suction Transfer . . .. ..... . 5-15 5.2.12 Turbine Turning Gear . . . . ... . .. . . . 5-16 5.2.13 Control Room HVAC . . . . . . .... . . . .. 5-16 5.2.14 Turbine Bypass Valve .............. 5-16 l

6. Human Factors Considerations ................ ... ..... .. 6-1 1

6.1 Operator Performance...... . .................. ... 6-1 i 6.1.1 Licensed Operators.......... ....... ........... 6-1 l 6.1. 2 Procedural Compliance.... ...................... 6-3 6.1. 3 Operator /STA Interaction........................ 6-4 6.1.4 Emergency Noti fication. . . . . . . ... ........ ..... 6-5 6.1. 5 Equipment Operators..... ....... ........... .. 6-5 6.2 Other Man-Machine Interface Considerations...... ...... 6-6 6.2.1 PORV Position Indication....... ................ 6-6 6.2.2 Safety Parameter Display System. ....... ....... 6-7 6.2.3 Plant Communications............................ 6-7 6.2.4 AFW Pump Turbine Overspeed Trip............. ... 6-7

6. 3 Personnel Issues....................................... 6-8

7. Safety Significance . ... . .. . .. . .. . . .... . 7-1

8. Findings and Conclusions. . . ................ 8-1 I

Appendices A Memorandum from W. J. Dircks, Executive Director for Opera-tions to the Commission, " Investigation of June 9, 1985 Event at Davis-Besse Will Be Conducted by NRC Team,"

June 10, 1985 . ...... . ... ... . ..... . A-1 B Toledo Edison Company Intra-Company Memorandum, " Guidelines to Follow When Troubleshooting or Performing Investigative Actions into the Root Causes Surrounding the June 9, 1985 Reactor Trip," June 13, 1985 . . . . . . . . . . . . . . B-1 vi

I THE NRC TEAM FOR THE DAVIS-BESSE EVENT OF JUNE 9, 1985 Charles E. Rossi, Team Leader J. T. Beard T. Larry Bell Wayne D. Lanning TEAM SUPPORT STAFF Stephen Burns, Office of the Executive Legal Director Walter E. 011u, Division of Technical Information and Document Control i

vii

r ACRONYMS AFPT Auxiliary Feedwater Pump Turbine AFWS Auxiliary Feedwater System ARTS Anticipatory Reactor Trip System ASME American Society of Mechanical Engineers AT0G Abnormal Transient Operating Guidelines B&W Babcock & Wilcox BWST Borated Water Storage Tank CRAM Count Rate Amplifier Module CST Condensate Storage Tank EAL Emergency Action Level ECCS Emergency Core Cooling System E00 Emergency Duty Officer EPRI Electric Power Research Institute FSAR Final Safety Analysis Report F/V Frequency-to-Voltage HED Human Engineering Deciciency HPI High Pressure Injection HVAC Heating, Ventilating, and Air Conditioning I&C Instrumentation and Control ICS Integrated Control System LPI Low Pressure Injection MFP Main Feed Pump MFW Main Feedwater MSIV Main Steam Isolation Valve MSSV Main Steam Safety Valve MU/HPI Makeup /High Pressure Injection NI Nuclear Instrumentation NPSH Net Positive Suction Head NSSS Nuclear Steam Supply System OTSG Once-Through Steam Generator PORV Pilot Operated Relief Valve PWR Pressurized Water Reactor RCP Reactor Coolant Pump RCS Reactor Coolant System SFAS Safety Features Actuation System SFRCS Steam and Feedwater Rupture Control System S/G Steam Generator SPDS Safety Parameter Display System STA Shift Technical Advisor S/U Startup SUFP Startup Feedwater Pump USAR Updated Safety Analysis Report ix j

1 1

1 INTRODUCTION The Davis-Besse Nuclear Power Station, Unit 1, operated by the Toledo Edison Company, is located on Lake Erie in Ottawa, County, Ohio, approximately six miles northeast of Oak Harbor, Ohio. At 1:35 a.m. , on June 9,1985, one of the two main feedwater pumps at Davis-Besse tripped (i.e., stopped) on overspeed while the plant was operating at 90% power. Thirty seconds later the reactor and turbine were automatically tripped on high reacter coolant system pressure.

Soon after the reactor tripped, both main steam isolation valves spuriously closed, resulting in a loss of steam to the second main feedwater pump. Subse-quent to this complete loss of main feedwater, an operator error, malfunctions of two redundant valves in the safet,-related auxiliary feedwater system, and overspeed trips of the two redundant, steam turbine-driven auxiliary feedwater pumps resulted in loss of all sources of feedwater to the steam generators.

Separate actions by operators were required to (1) correct the initial operator error, (2) open the valves which malfunctioned, and (3) reset the overspeed trips of the turbine-driven auxiliary feedwater pumps. Actions outside the con-trol room were required to open the valves and place the pumps in operation.

While operators acted to restart the safety-related auxiliary feedwater system, operator actions outside the control room were also taken to place a nonsafety-related, electric motor-driven, startup feedwater pump in service. The plant's two steam generators had essentially boiled dry before feedwater from any source became available to them. Further, a number of additional equipment problems complicated the event. Nevertheless, operators were successful in bringing the plant to a stable shutdown and in preventing any abnormal releases of radio-activity and any major damage to the plant.

On the day following the event, and in conformance with the staff-proposed Inci-l dent Investigation Program, the NRC Executive Director for Operations sent an i NRC Team of technical experts to the site. (For the directive establishing the i Team, see Appendix A.) The Team, composed of four staff members, was selected

because of its broad experience in operating plant event analyses, with indivi-I dual Team members having specific knowledge and experience in operations, in-4 strumentation and controls, and reactor systems. The Team was to (1) fact-find as to what happened; (2) identify the probable cause as to why it happened; and (3) make appropriate findings and conclusions to form the basis for possible follow-on actions. This report documents the Team's efforts in identifying the circumstances and causes of the event together with its findings, and conclusions.

l The scope of this fact-finding effort was limited to the circumstances concern-j ing the event of June 9, 1985, including operator actions and equipment mal-

functions. Section 2 describes the methodology used by the team to collect j and evaluate information about the event.

Sections 3 and 6 of this report discuss what was learned about operator per-formance. Through interviews with the operators on duty at the time of the l

event, the Team obtained a considerable amount of information about operator i

performance and capabilities during circumstances that required an extensive

' 1-1

- - - - . -- -- ._- .. - - - . - -L

range of operator actions in a relatively short period of time to bring the plant to a safe, stable condition. A number of the findings and conclusions in Section 8 are based on the Team's evaluation of operator response to the event.

Section 4 provides an overview of how the reacter systems function and interact, as well as a description of the safety systems involved in this event.

Section 5 discusses the Team's review of Toledo Ediscn Company's efforts to determine the causes of the equipment malfunctions as well as summaries of their maintenance experience related to that equipment. The Team did not, however, extensively review Toledo Edison's management performance record, quality assur-ance program, maintenance procedures, or history of regulatory compliance. A number of significant pieces of equipment were either not in service prior to the event or malfunctioned during the event. The Team used summaries provided by Toledo Edison of their past evaluations, troubleshooting, testing, and maintenance related to the equipment that malfunctioned in reaching a number of the findings and conclusions discussed in Section 8.

Section 7 contains a discussion of the safety significance of the event, includ-ing a summary of information on the consequences of loss of feedwater events at Davis-Besse as a function of mitigating systems available. Section 8 presents the Team's findings and conclusions.

Based on this report, it is expected that the NRC Executive Director for Opera-tions will identify and assign specific NRC offices the responsibility for sub-sequent actions related to this event.

l i

a 1

1-2

2 DESCRIPTION OF FACT FINDING EFFORTS 2.1 General The Team collected and evaluated information to determine the sequence of oper-ator, plant, and equipment responses during the event and the causes of equip-ment malfunctions. The sequence of these responses was determined primarily by interviewing personnel who were at the plant during the event and by reviewing plant data for the period immediately preceding and during the event. The Team also toured the plant to examine the equipment which malfunctioned, the equip-ment that was key to mitigating the transient, and the control room instrumenta-tion and controls. The Team also interviewed plant management personnel and NRC Region III personnel who arrived at the site soon after the plant was sta-bilized about their knowledge of the plant response and operator actions. The root causes of equipment malfunctions in most cases have yet to be definitively determined. The root causes are being established through systematic trouble-shooting performed by Toledo Edison personnel and equipment vendors using pro-cedures agreed upon by the Team.

As with all commercial nuclear power plants, a considerable amount of informa-tion on plant response and specific equipment actuation can be obtained from records automatically generated in the form of analog recordings and digital printouts. These records accurately indicate the chronological sequence for such things ss the starting and stopping of pumps and the opening and closing of valves, as well as the time response of key plant parameters. By correlat-ing plant records with personnel statements on their actions and observations, the Team was able to compile a picture of the key aspects of the event.

The equipment which malfunctioned was quarantined so that troubleshooting and corrective actions could be performed systematically. This ensured that infor-mation on the root causes of each malfunction could be obtained.

2.2 Interviews and Meetings The Team placed a high priority on interviewing personnel on duty at the time of the event to learn about the actions they took and the observations they made. It was recognized that the quicker these interviews could be held, the more information those being interviewed would remember. The Team held meet-ings with Toledo Edison and NRC Region III personnel to obtain an overview of the sequence of events from their analyses and evaluations of plant data. The Team also met with Toledo Edison and Region III staff members to agree upon a course of action for troubleshooting the equipment which malfunctioned and to discuss the results of the troubleshooting efforts. The root causes for equip-ment malfunctions cannot be determined until the troubleshooting ef forts are completed.

All interviews and meetings about the sequence of events during the plant trans-ient, the course of action for troubleshooting equipment which had malfunctioned, and the cause for the equipment malfunctions were recorded by stenographers and 2-1

typed transcripts were prepared. The Team also took pictures of key plant equip-ment and made tape recordings of pertinent discussions during the tour of the control room and plant. A record was not made of discussions between the Team and Toledo Edison or Region III personnel about obtaining documents relating to the Davis-Besse plant design / operation or pertaining to schedules. Nor did the Team record the first meeting with Toledo Edison and Region III staff members on the morning of June 11, 1985 to explain the Team's objectives and plans.

The formal fact-finding effort began in the afternoon of June 11 when the Team met with Toledo Edison personnel to obtain an overview of their understanding of the event. The Team then met with Toledo Edison and Region III staffs to learn about specific design features of the Davis-Besse plant important to understanding the event. The Team was given an overview of the design of the Davis-Besse main steam system, main feedwater system, auxiliary feedwater sys-tem, and steam and feedwater rupture control syster. Questions concerning the plant design were answered.

Following these overview meetings on the sequence of events and plant systems, the first of a number of meetings was held on the course of action to be taken with the equipment which malfunctioned. The decisions and actions taken are discussed in Section 2.4.

Although the highest priority was given to interviews with the personnel on duty during the event, shift scheduling made it more convenient to interview available NRC Region III staff first. On June 11, the Team discussed the event with Region III personnel to obtain their overview from observations and evalua-tions made when they arrived at the site following the event. The NRC Resident Inspector was the first NRC representative to arrive at the site at approximately 3:20 a.m. on June 9. He described his observations of the plant status when he arrived and discussed a sequence of events which he had prepared from plant records.

Interviews with operating personnel began on the morning of June 12. The gen-eral approach for scheduling interviews was to talk to personnel in decreasing order of their seniority within the shift, beginning with the shift supervisor and proceeding to those less senior. The rationale for this sequence was to move from general to specific information. Thus, the Team obtained information on the overall plant operation and then obtained information on the detailed actions of specific operators. The scheduling of interviews and meetings was also based on the availability of personnel and the progress of Toledo Edison in developing plans for troubleshooting the equipment.

The interviews with the plant personnel generally covered the following areas.

The interviewee was asked to describe his position in the plant organization and to discuss his background and experience. The interviewee was then asked to describe the event beginning from the time he first realized that an abnormal plant condition existed. The interviewee was questioned on the actions he per-sonally took during the event, his observations of plant responses during the event, and his observations of the actions of others. Following a " walk-through" of actions and observations made during the event, the team asked the interviewee questions on the use of procedures, the value of training, perceptions of the adequacy of plant maintenance, and whether NRC regulations or procedures inter-fered with maintaining plant safety during this event or at other times.

I 2-2 I

1 l The shift supervisor, assistant shift supervisor, the two licensed reactor oper-ators, the shift technical advisor, and the administrative assistant to the shift supervisor were interviewed individually. The four non-licensed equipment operators who performed key actions outside the control room during the event were interviewed as a group. Those being interviewed in some cases had either l a supervisor or an attorney (or both) present during the interview. Except in rare cases, only the interviewee responded to questions during the interview.

Plant management and NRC Region III personnel who went to the site soon after the event were interviewed in the same general manner as plant operating person-nel. In addition to the NRC Resident Inspector discussed above, those interviewed included the NRC Senior Resident Inspector, the Plant Manager, the Operations Superintendent, and the Operations Supervisor. Specific questions related to the plant organization, maintenance, and other issues not directly related to the sequence of events varied, depending upon the particular experience and position of the individual being interviewed.

Some personnel were interviewed more than one time when the Team needed to ob-tain additional clarifying information. Table 2-1 contains a listing of the interviews and meetings conducted by the Team.

2.3 Plant Data The following plant records were used in determining the times at which key events occurred during the transient:

(1) Sequence of Events Monitor Printout (2) Alarm Printout (3) Data Acquisition Display System Printout (4) Analog curves generated from the digital information from the Data Acquisition Display System The Sequence of Events Monitor and the Alarm Printout are both functions of the Plant Process Computer. The Sequence of Events Monitor records the change of state of the major digital inputs, such as equipment and key system trips. The Sequence of Events Monitor records events to the nearest five milliseconds and provides the most accurate time recording of all available plant records. The Alarm Printout lists both digital and analog information when parameters reach a predetermined alarm state. The digital points are scanned once per second and analog points are scanned at varying intervals (either at 1 , 5 , 15 , 30 ,

or 60-second intervals). The Alarm Printout indicates the time that a parameter either exceeds the alarm limit or returns to within the limit. The time resolu-tion, however, is determined by the scanning interval.

The Data Acquisition and Display System is part of the Technical Support Center equipment and maintains a 24-hour record of plant parameters for event analyses.

Key plant variables are reccrded with a scan rate of once per second. Data from this system were available to the Team both in tabular form and, for selected variables, in the form of graphs. The times for the various events in the sequence of events were taken from the above plant records.

l 2.4 Quarantined Equipment l

On June 10, 1985, NRC Region III issued a Confirmatory Action Letter indicating, I among other things, that Toledo Edison would not perform any additional work 2-3 l

on equipment that malfunctioned during the event until the Team could review the proposed actions. The Team met with Toledo Edison and Region III represen-tatives on the af ternoon of June 11 to ensure agreement on the quarantined equip-ment list and to establish a course of action for determining the root causes of the equipment malfunctions. Toledo Edison was asked to develop plans for performing the troubleshooting in a systematic, controlled manner. A primary concern was to ensure that adequate records would be maintained on the "as-found" condition of equipment. In this meeting, any item that had failed during the June 9 event was placed on the quarantine list, except where Toledo Edison was able to justify removal on the basis of not being related to the safety concerns of the event.

In subsequent meetings, Toledo Edison and the Team agreed upon the list of equip-ment which should be handled under the procedures for quarantined equipment. As the evaluation of the event evolved, the list was modified. For example, steam line traps and drains thought to have malfunctioned and caused water buildup in the steam lines upstream of one set of turbine bypass valves were included. The malfunctioning of the traps and drains then became a possible root cause for water hammer damage to the turbine bypass valve. The steam generator atmos-pheric vent valves were also added to the list when a question arose regarding their potential malfunctioning or improper use during the event.

In a meeting on June 14, Toledo Edison presented plans for controlling the equip-ment troubleshooting. They provided " Guidelines to Follow When Troubleshooting or Performing Investigative Actions Into the Root Causes Surrounding the June 9, 1985 Reactor Trip," which delineated the general procedures to be followed for troubleshooting each piece of equipment. Toledo Edison's original document was revised as a result of discussions at this and subsequent meetings. The proce-dures agreed upon required maintenance work orders to be based upon a specific action plan for each piece of equipment. The action plans were to contain hypotheses and probable causes of failure or abnormal operation for each piece of equipment. The action plans were to include an analysis of information con-cerning the operation of the equipment during the event, a review of the main-tenance, surveillance and testing history for the equipment, and plans for deter-mining the probable causes for the equipment malfunctions observed. Specific statements indicating where equipment vendor representatives were to be used in the troubleshooting also were to be provided in the action plans.

The troubleshooting procedures required that all as-found conditions, such as damaged components or setpoint adjustments, be documented. Retention and complete traceability for components and equipment requiring replacement were to be main-tained. Toledo Edison agreed to notify the NRC when the deter.nination of the root cause of the malfunction or failure of a piece of equipment was made. It was agreed that the results of the troubleshooting process, root cause determi-nations, and supporting justification were to be presented to the NRC as soon as practical.

The Team did not approve the individual action plans, but did review and comment on the plans for the most significant equipment which malfunctioned during the event. Region III personnel monitored Toledo Edison troubleshooting efforts to ensure that the general guidelines for the troubleshooting and the specific equipment action plans were followed.

2-4 i

On July 11 and 12,1985, the Team met with Toledo Edison personnel to discuss the status of the troubleshooting efforts. The information available on the g

cquipment malfunctions at the time this report was prepared is discussed in Section 5.

The " Guidelines to Follow When Troubleshooting or Performing Investigative l Actions into the Root Causes Surrounding the June 9, 1985 Reactor Trip" with 1 the attached list of quarantined equipment (" Equipment Freeze" list) appears as Appendix B.

I 2-5

6/20/85 10:10 am Interview of NRC Region III Senior Resident Inspector 6/20/85 5:30 pm Meeting on Troubleshooting for the Auxiliary Feed Pumps Overspeed Trips 6/21/85 9:00 am Interview of Plant Manager and Operations Superintendent 6/21/85 10:30 am Meeting on Valves AF 599 and AF 608; Troubleshooting for the AFPT Overspeed Trip Throttle Valve Problem; and Sequence of Events 6/21/85 2:00 pm Interview of Administrative Assistant to the Shift Supervisor 6/21/85 5:30 pm Interview of NRC Region III Senior Resident Inspector 6/27/85 10:35 am Meeting on Troubleshooting for the Auxiliary Feed Pumps Overspeed Trips, the PORV, Spurious Closure of the MSIVs, and the Startup Feed Valve SP-7A.

7/9/85 9:30 am Meeting on Design and Operation of the Steam and Feedwater Rupture Control System 7/9/85 11:20 am Meeting on Miscellaneous Plant Design Details and Equipment Capacities 7/9/85 1:25 pm Interview of Operations Superintendent on Availability of Selected Procedures and Actions Required to Re-gain Main Feedwater Flow 7/9/85 2:40 pm Meeting on Pilot Operated Relief Valve Controls 7/10/85 9:10 am Meeting on Design and Operation of the Steam and Feedwater Rupture Control System 7/10/85 1:12 pm Meeting on Sequence of Events 7/11/85 9:22 am Meeting on Design and Operation of the Steam and Feedwater Rupture Control System 7/11/85 12:00 pm Meeting on Operator Training Related to Steam and Feedwater Rupture Control System Manual Actuation 7/11/85 4:25 pm Meeting on Status of Troubleshooting Activities 7/12/85 9:10 am Meeting on Status of Troubleshooting Activities 2-7

s 4

4 i

3 NARRATIVE OF THE EVENT U

]

This detailed description of the Davis-Besse loss-of-feedwater event focuses j attention on the operator actions which prevented a potentially serious event,

both in terms of safety and economics, from occurring. From their normal oper-

) ating routine, the operators were plunged abruptly into a high stress situation i requiring complicated responses outside the control room. Furthermore, these

{ activities unfolded early on a Sunday morning when additional technical exper-

) tise from either onsite or offsite was at a minimum.

i In view of the importance of the operator actions, the narrative of the event which follows is based upon a composite of the operator interviews performed by the Team. The narrative is written to reflect the operators' descriptions of their actions, observations, and thoughts during the event. The team decided j that this would best convey the effects of stress, training, experience, team-I work, and impediments on operator performance. There are undoubtedly lessons to l be learned about what operators are likely to do during a serious event which l'

are not easily summarized, but which perhaps can be inferred from the descrip-tions of what occurred during this particular event.

The sequence of events listed in Table 3.1 is based on the plant process com-J puter printouts (alarm and sequence of events) and the data acquisition display

! system (DADS) computer printouts. The trends of important primary and secondary

! coolant system parameters are shown in Figures 3.1 through 3.3 as a function of time. Figure 3.1 shows the reactor coolant system pressure, average tempera-j ture, and the pressurizer level. Figures 3.2 and 3.3 show the pressure, level j and flow for each of the two steam generators.

i

! 3.1 Shift Change i

j On June 9, 1985, the midnight shift of operators assumed control of the Davis- t Basse nuclear power plant. The oncoming shift included four licensed operators, four equipment operators, an auxiliary operator, and an administrative assistant.

The shift supervisor and the assistant shift supervisor are licensed senior reactor operators and the most experienced members of the operating crew. Both ware at the plant before it was issued an operating license in April 1977. The two reactor operators, who were responsible for the control room, had decided between themselves who would be responsible for the primary-side and who would take the secondary-side work stations. The secondary-side operator has been a licensed reactor operator for about two years; the primary-side operator was licensed in January 1985. He had previous nuclear Navy experience and was an equipment operator before being licensed. Prior to the morning of June 9, n2ither reactor operator had been at the controls during a reactor trip at

! Davis-Besse.

The four equipment operators are a close-knit group, three of whom had been operators in the nuclear Navy. Their experience at the plant ranges from three to nine years, averaging six-and-one-half years per operator. Equipment operators receive directions from the control room operators to manipulate and troubleshoot equipment in tre reactor auxiliary building and the turbine i

3-1

- - - - - - -. - _ _ - _ - - - - _-- - - l

building. Generally, equipment operators occupy this position temporarily as they participate in a development program leading to the position of licensed

operator. However, two equipment operators did not intend to become licensed operators.

The shift turnover on June 9 was easy--there were no ongoing tests or planned changes to plant status. The plant was operating at 90 percent of the full power authorized in the license granted by the NRC in April 1977, to minimize the potential for an inadvertent reactor trip (i.e., shutdown) due to noise on primary coolant flow instrumentation.

All the major equipment control stations were running on automatic except the No. 2 main feedwater pump. As a result, the integrated control system instru-ments were monitoring and controlling the balance between the plant's reactor coolant system and the secondary coolant system.

Since April 1985, there had been control problems with both main feedwater pumps.

Troubleshooting had not identified nor resolved the problems. .In fact, a week earlier, on June 2,1985, both feedwater pumps tripped unexpectedly after a reactor trip. After some additional troubleshooting, the decision was made to

, not delay startup any longer, but to put instrumentation on the pumps to help diagnose the cause of a pump trip, if it occurred again. As a precaution, the number two main feedwater pump was operating in manual control to prevent it from tripping and to ensure that all main feedwater would not be lost should the reactor trip.

) Some operators were uneasy about going up to power with problems in the feedwater j pumps, but they complied with the decisions made by their management.

During the first hour of the shift, the operators' attention and thoughts were directed to examining the control panels and alarm panels, and performing

instrument checks and routine surveillances associated with shift turnover.

Thus, at 1:35 in the morning, the plant generator was providing electricity to I.

the Ohio countryside. The secondary-side operator had gone to the kitchen  ;

where he joined an equipment operator for a snack. The other reactor cperator was at the operator's desk studying procedures for requalification examinations.

The assistant shift supervisor had just left the kitchen on his way back. to the control room af ter a break. The shif t supervisor was in his office outside the control room performing administrative duties.

I 3.2 Reactor Trip - Turbine Trip The assistant shift supervisor entered the control room (shown in Figure 3.4) and I was examining one of the consoles when he noticed that main feedwater flow was decreasing and that the No. 1 main feedwater pump had tripped. Since the No. 2 I'

feedwater pump was in manual control, it could not respond to the integrated control system demand automatically to increase feedwater flow.

The " winding down" sound of the feedwater pump turbine was heard by the reactor operator in the kitchen, and by the administrative assistant and the shift supervisor, both of whom were in their respective offices immediately outside

the control room. They headed immediately for the control room--the event had j begun.

3-2 j

1 The secondary-side reactor operator ran to his station and immediately increased the speed of the No. 2 main feedwater pump to compensate for the d: crease of feedwater flow from the No. 1 pump. The primary-side operator had already opened the pressurizer spray valve in an attempt to reduce the pressure surge resulting from the heatup of the reactor coolant system due to a decrease in feedwater flow.

The plant's integrated control system attempted automatically to reduce reactor /

turbine power in accordance with the reduced feedwater flow. The control rods were being inserted into the core and reactor power had been reduced to about 80 percent. At the same time the primary-side reactor operator held open the pressurizer spray valve in an attempt to keep the reactor coolant pressure below the high pressure reactor trip set point of 2300 psig (normal pressure is 2150 psig). However, the reduction of feedwater and subsequent degradation of haat removal from the primary coolant system caused the reactor to trip on high reactor coolant pressure. The operators had done all they could do to prevent the trip, but the safety systems had acted automatically to shut down the nuclear reaction.

The primary-side operator acted in accordance with the immediate post-trip actions specified in the emergency procedure that he had memorized. Among other things, he checked that all control rod bottom lights were on, hit the reactor trip (shutdown) button, isolated letdown from the reactor coolant system, and started a second makeup pump to anticipate a reduced pressurizer inventory after a normal reactor trip. Then he waited, and watched the reactor coolant pressure to sea how it behaved.

The secondary-side operator heard the turbine stop valves slamming shut and knew the reactor had tripped. This " thud" was heard by most of the equipment operators who also recognized its meaning and two of them headed for the control room. Almost simultaneously, the secondary-side operator heard the

( loud roar of main steam safety valves opening, a sound providing further proof that the reactor had tripped. The lifting of safety valves after a high power rsactor trip was normal. Everything was going as expected as he waited and watched the steam generator water levels boil down--each should reach the normal post-trip low level limit of 35 inches on the startup level instrumentation and hold steady.

i The shift supervisor joined the operator at the secondary-side control console and watched the rapid decrease of the steam generator levels. The rapid feed-water reduction system (a subsystem of the integrated control system) had closed the startup feedwater valves, but as the level approached the low level limits, the startup valves opened to hold the level steady. The main steam safety valves closed as expected. The system response was looking "real good" to the shift supervisor.

The assistant shift supervisor in the meantime opened the plant's looseleaf crergency procedure book. (It is about two inches thick, with tabs for quick roference. The operators refer to it as emergency procedure 1202:01; the NRC refers to it as the ATOG procedure - Abnormal Transient Operating Guidelines.)

As he read aloud the immediate actions specified, the reactor operators were rosponding in the affirmative. After phoning the shift technical advisor (STA) to come to the control room, the administrative assistant began writing down what the operators were saying, although they were speaking faster then she could write.

3-3

The STA was working a 24-hour shift and was asleep when awakened by a telephone call from the shift supervisor, which was followed immediately by the call from the administrative assistant. (The STAS are provided an apartment-type room in i the administrative building, which is outside the protected area about one-half mile from the plant. According to procedures, they must be able to get to the control room within 10 minutes of being called. ) He had detected a sense of urgency in the telephone calls and so he ran out of the building to his car for the drive to the site. He was anxious himself--this was his first reactor trip since becoming a shift technical advisor in January 1985.

3.3 Loss of Main Feedwater Although the assistant shift supervisor was loudly reading the supplementary actions from the emergency procedure book, the shift supervisor heard the main steam safety valves open again. He knew from experience that something was unusual and instinctively surveyed the control console and panels for a clue.

He discovered that both main steam isolation valves (MSIVs) had closed--the first and second of a list of unexpected equipment performances and failures that occurred during the event.

The secondary-side operator was also aware that something was wrong because he noticed that the speed of the only operating main feedwater pump was decreasing.

After verifying that the status of the main feedwater pump turbine was normal, he concluded that the turbine was losing steam pressure at about the same time that the shift supervisor shouted that the MSIVs were closed. All eyes then turned up to the annunciators at the top of the back panel. They saw nothing abnormal in the kind or number of annunciators lit after the reactor trip. The operators expected to find an alarm indicating that the Steam Feedwater Rupture Control System (SFRCS, pronounced S-FARSE) had activated. Based on their knowledge of previous events at the plant, they believed that either a partial or full actuation of the SFRCS had closed the MSIVs. However, the SFRCS annunci-ator lights shown in Figure 3.5 were dark. The MSIVs had closed at 1:36 a.m.

and they were going to stay closed. It normally takes at least one-half hour to prepare the steam system for reopening the valves.

The No. 2 main feedwater pump turbine, deprived of steam, was slowly winding down. Since the MSIVs were closed and there was limited steam inventory in the moisture separator reheaters, there was inadequate motive power to pump feed-water to the steam generators. At about 1:40 a.m. the discharge pressure of the pump had dropped below the steam pressure which' terminated main feedwater flow.

3.4 Loss of Emergency Feedwater )

The secondary-side operator watched the levels in both steam generators boil down; he had also heard the main steam safety valves lifting. With'out feedwater, he knew that an SFRCS actuation on low steam generator level was imminent. The SFRCS should actuate the auxiliary feedwater system (AFWS) which in turn should provide emergency feedwater to the steam genereL us. He was trained to trip manually any system that he felt was going te tr: i automatically. He requested and received permission from the shift snp%"itsi to trip the SFRCS on low level to conserve steam generator invent +r , 3 the AFWS would be initiated oefore the steam generator low-level setpoint was reached, s

3-4

He went to the manual initiation switches at the back panel and pushed two buttons to trip the SFRCS. (The SFRCS control panel is shown in Figure 3.6.)

' H2 inadvertently pushed the wrong two buttons and, as a result, both steam g;nerators were isolated from the emergency feedwater supply. He had activated the SFRCS on low pressure (the top pair of buttons in Figure 3.6) for each steam generator instead of on low level (the fourth pair of buttons from the tcp). By manually actuating the SFRCS on low pressure, the SFRCS was signalled that both generators had experienced a steamline break or leak and the system rssponded, as designed, to isolate both steam generators. The operator's enticipatory action defeated the safety function of the auxiliary feedwater system--a common-mode failure and the third abnormality to occur within 6 minutes after the reactor trip.

The operator returned to the auxiliary feedwater station expecting the AFWS to actuate and provide the much-needed feedwater to the steam generators that were boiling dry. Instead, he first saw the No. 1 AFW pump, followed by the No. 2 AFW pump trip on overspeed--a second common-mode failure of the auxiliary feed-water system and abnormalities four and five. He returned to the SFRCS panel to find that he had pushed the wrong two buttons.

The operator knew what he was supposed to do. In fact, most knowledgeable ptople in the nuclear power industry, even control room designers, know that the once-through steam generators in Babcock & Wilcox-designed plants can boil dry in as little as 5 minutes; consequently, it is vital for an operator to be able to quickly start the AFWS. There could have been a button labeled simply "AFWS--Push to Start." But instead, the operator had to do a mental exercise to first identify a signal in the SFRCS that would indirectly start the AFW system, find the correct set of buttons from a selection of five identical sets located knee-high from the floor on the back panel, and then push them without being distracted by the numerous alarms and loud exchanges of information b2 tween operators.

The shift supervisor quickly determined that the valves in the AFWS wee improp-orly aligned. He reset the SFRCS, tripped it on low level, and corrected the operator's error about one minute after it occurred. This action commanded the SFRCS to realign itself such that each AFW pump delivered flow to its asso-ciated steam generator. Thus, had both systems (the AFWS and SFRCS) operated properly, the operator's mistake would have had no significant consequences on plant safety.

The assistant shift supervisor, meanwhile, continued reading aloud from the emergency procedure. He had reached the point in the supplementary actions j that require verification that feedwater flow was available. However, there '

was no feedwater, not even from the AFWS, a safety system designed to provide fcedwater in the situation that existed. (The Davis-Besse emergency plan identifies such a situation as a Site Area Emergency.) Given this condition, the procedure directs the operator to the section entitled, " Lack of Heat Transfer." He opened the procedure at the tab corresponding to this condition, but left the desk and the procedure at this point, to diagnose why the AFWS had failed. He performed a valve alignment verification and found that the isola-tion valve in each AFW train had closed. Both valves (AF-599 and AF-608) had failed to reopen automatically after the shift supervisor had reset the SFRCS.

H2 tried unsuccessfully to open the valves by the push buttons on the back panel.

H2 went to the SFRCS cabinets in the back of the back panel to clear any trips 3-5

in the system and block them so that the isolation valves could open. However, i there were no signals keeping the valves closed. He concluded that the torque  !

switches in the valve operators must have tripped. The AFW system had now suf-fered its third common-mode failure, thus increasing the number of malfunctions to seven within 7 minutes after the reactor trip (1:42 a.m.).

3.5 Reactor Coolant System Heatup Meanwhile, about 1:40 a.m. , the levels in both steam generators began to decrease below the normal post-reactor-trip limits (about 35 inches on the startup range). ,

The feedwater flow provided by the No. 1 main feedwater pump had terminated. '

The flow from the No. 2 main feedwater pump was decreasing because the MSIVs were closed, which isolated the main steam supply to the pump. With decreasing feedwater flow, the effectiveness of the steam generators as a heat sink for removing decay (i.e., residual) heat from the reactor coolant system rapidly decreased. As the levels boiled down through the low level setpoints (the auxiliary feedwater should automatically initiate at about 27 inches), the average temperature of the reactor coolant system began to increase, indicating a lack of heat transfer from the primary to the secondary coolant systems.

When the operator incorrectly initiated SFRCS on low pressure, all feedwater was isolated to both steam generators. The reactor coolant system began to heat up because heat transfer to the steam generators was essentially lost due to loss of steam generator water level.

The average reactor coolant temperature increased at the rate of about 4 degrees Farenheit per minute for about 12 minutes. The system pressure also increased steadily until the operator fully opened the pressurizer spray valve (at about 1:42 a.m. ). The spray reduced the steam volume in the pressurizer and temporarily interrupted the pressure increase. The pressurizer level 1 increased rapidly but the pressurizer did not completely fill with water. As I the indicated level exceeded the normal value of 200 inches, the control valve for makeup flow automatically closed.

At this point, things in the control room were hectic. The plant had lost all feedwater; reactor pressure and temperature were increasing; and a number of unexpected equipment problems had occJrred. The seriousness of the situation was fully appreciated.

3.6 Operator Actions By 1:44 a.m. , the licensed operators had exhausted every option available in the control room to restore feedwater to the steam generators. The main feed-water pumps no longer had a steam supply. Even if the MSIVs could be opened, the steam generators had essentially boiled dry, and sufficient steam for the main feedwater pump turbines would likely not have been available. The turbines for the AFW pumps had tripped on overspeed, and the trip throttle valves could not be reset from the control room. Even if the AFW pumps had been operable, the isolation valves between the pumps and steam generators could not be opened from the control room, which also inhibited the AFWS from performing its safety function. The likelihood of providing emergency feedwater was not certain, even if the AFW pump overspeed trips could be reset and the flow path estab-lished; for example there was a question as te whether there was enough steam remaining in the steam generators to start the steam-driven pumps. Unknown to 3-6

the operators, the steam inventory was further decreased because of problems controlling main steam pressure. The number of malfunctions had now reached eight.

Three equipment operators had been in the control room since shortly after the rtactor tripped. They had come to the control room to receive directions and to assist the licensed operators as necessary. They were on the sidelines w&tching their fellow operators trying to gain control of the situation.

The safety-related AFW equipment needed to restore water to the steam generators hid failed in a manner that could only be remedied at the equipment location and not from the control room. The affected pumps and valves are located in locked compartments deep in the plant.

The primary-side reactor operator directed two of the equipment operators to go to the auxiliary feedwater pump room to determine what was wrong--and hurry.

The pump room, located three levels below the control room, has only one entrance: a sliding grate hatch that is locked with a safety padlock (Fig-ure 3.7). One of the operators carried the key ring with the padlock key in his hand as they left the control room. They violated the company's "no running" policy as they raced down the stairs. The first operator was about 10 feet ahead of the other operator who tossed him the keys so as not to delay unlocking the auxiliary feedwater pump room. The operator ran as fast as he could and had unlocked the padlock by the time the other operator arrived to help slide the hatch open.

The operators descended the steep stairs resembling a ladder into the No. 2 AFW pump room. They recognized immediately that the trip throttle valve had tripped (see Figure 3.8). One operator started to remove the lock wire on the handwheel while the other operator opened the water-tight door to the No. 1 AFW pump. He also found the trip throttle valve tripped and began to remove the lock wire from the handwheel.

The shift supervisor had just dispatched a third equipment operator to open AFW isolation valves AF-599 and AF-608. These are chained and locked valves, as shown in Figure 3.9, and the shift supervisor gave the lock-valve key to the operator before he lef t the control room. He paged a fourth equipment operator over the plant communications systems and directed him also to open valves AF-599 and AF-608. Although the operators had to go to different rooms for each valve, they opened both valves in about 3 minutes. They were then directed to the AFW pump room.

As the operators ran to the equipment, a variety of troubling thoughts ran through their minds. One operator was uncertain if he would be able to carry cut the task that he had been directed to do. He knew that the valves he had to open were locked valves, and they could not be operated manually without a ksy. He did not have a key and that concerned him. As he moved through the turbine building, he knew there were numerous locked doors that he would have to go through to reach the valves. He had a plastic card to get through the card readers, but they had been known to break and fail. He did not have a set of door keys and he would not gain access if his key card broke and that concerned him too.

3-7 1

The assistant shift supervisor came back into the control console area after having cleared the logic for the SFRCS and he tried again, unsuccessfully, to open the AFWS isolation val"as. At this point, the assistant shift supervisor made the important decision to attempt to place the startup feedwater pump (SUFP) in service to supply feedwater to the steam generators. He went to the key locker for the key required to perform one of the five operations required to get the pump running.

The SUFP is a motor-driven pump, usually more reliable than a turbine-driven pump, and more importantly, it does not require steam from the steam generators to operate. The SUFP is located in the same compartment as the No. 2 AFW pump.

But since the refueling outage in January 1985, the SUFP had been isolated by closing four manual valves and its fuses were removed from the motor control circuit. This isolation was believed necessary because of the consequences of a high energy break of the non-seismic grade piping which passes through the two seismic qualified AFW pump rooms. Prior to January 1985, the SUFP could be initiated from the control room by the operation of a single switch.

The assistant shift supervisor headed for the turbine building where he opened the four valves and placed fuses in the pump electrical switchgear. This equip-ment is located at four different places; in fact, other operators had walked through the procedure of placing the SUFP in operation and required 15 to 20 minutes to do it. The assistant shift supervisor took about 4 minutes to perform these activities. He then paged the control room from the AFW pump room and instructed the secondary-side operator to start the pump and align it with the No. 1 steam generator.

The two equipment operators in the AFW pump rooms had been working about 5 min-utes to reset the trip throttle valves when the assistant shift supervisor entered the room to check the SUFP. The equipment operators thought that they had latched and opened the valves. However, neither operator was initially successful in getting the pumps operational. Finally, after one equipment operator had tried everything that he knew to get the No. 1 AFW pump operating, he left it and went to the No. 2 AFW pump where the other operator was having the same problem of getting steam to the turbine. Neither operator had pre-viously performed the task that he was attempting.

The assistant shift supervisor went over to assist the equipment operators and  ;

noticed immediately that the trip throttle valves were still closed. Apparently, the equipment operators had only removed the slack in attempting to open the valve. The valve was still closed and the differential pressure on the wedge disk made it difficult to turn the handwheel after the slack was removed, thus necessitating the use of the valve wrench. A third, more experienced operator had entered the pump room and used a valve wrench to open the trip throttle valve on AFW pump No. 2. Without the benefit of such assistance the equipment operators may well have failed to open the trip throttle valves to admit steam to the pump turbines.

The third equipment operator then proceeded to the No. 1 AFW pump trip throttle valve. The valve had not been reset properly and he experienced great diffi-culty in relatching and opening it because he had to hold the trip mechanism in the latched position and open the valve with the valve wrench. Because the trip mechanism was not reset properly, the valve shut twice before he finally opened the valve and got the pump operating.

I 3-8 l

3.7 PORV Failure Prior to being informed by the assistant shift supervisor that the SUFP was available, the secondary-side operator requested the primary-side operator to rsset the isolation signal to the startup feedwater valves in preparation for starting the SUFP. In order to perform this task, the operator left the con-trol console and went to the SFRCS cabinets in back of the control room. As ha re-entered the control panel area, he was requested to reset the atmospheric vant valves. As a result of these activities the primary side operator esti-meted that he was away from his station for 20 to 30 seconds. (In fact, he was cway for about two minutes.)

While the operator was away from the primary-side control station, the pressur-l izer PORV opened and closed twice without his knowledge. The pressure had in-creased because of the continued heatup of the reactor coolant system that rssulted when both steam generators had essentially boiled dry.

According to the emergency procedure, a steam generator is considered " dry" when its pressure falls below 960 psig and is decreasing, or when its level is balow 8 inches on the startup range (normal post-trip pressure is 1010 psig and post-trip level is 35 inches). The instrumentation in the control room is inadequate for the operator to determine with certainty if these conditions exist in a steam generator. The lack of a trend recorder for steam generator pressure makes it difficult to determine if the steam pressure is 960 psig and decreasing. The range of the steam generator level indicator (Figure 3.10) in the control room is 0-250 inches, a scale which makes determining the 8-inch isvel difficult. The safety parameter display system (SPDS) was intended to provide the operators with these critical data, but both channels of the SPDS w;re inoperable prior to and during this event. Thus, the operators did not know that the conditions in the steam generators beginning at about 1:47 a.m.

ware indicative of a " dry" steam generator, or subsequently, that both steam gsnerators were essentially dry.

When both steam generators are dry, the procedure requires the initiation of c ke-up/high pressure injection (MU/HPI) cooling, or what is called the " feed-cnd-bleed" method for decay heat removal. Even before conditions in the steam g;nerators met these criteria, the shift supervisor was fully aware that MU/HPI cooling might be necessary. When the hot-leg temperature reached 591*F (normal post-trip temperature is about 550*F), the secondary-side operator recommended to the shift supervisor that MU/HPI cooling be initiated. At about the same time, the operations superintendent told the shift supervisor in a telephone discussion that if an auxiliary feedwater pump was not providing cooling to one steam generator within one minute, to prepare for MU/HPI cooling. However, the shift supervisor did not initiate MU/HPI cooling. He waited for the equipment operators to recover the auxiliary feedwater system.

The shift supervisor appreciated the economic consequences of initiating MU/HPI ccoling. One operator described it as a drastic action. During MU/HPI, the PORV and the high point vents on the reactor coolant system are locked open, which breaches one of the plant's radiological barriers. Consequently, radio-active reactor coolant is released inside the containment building. The plant w:uld have to be shut down for days for cleanup even if MU/HPI cooling was suc-cassful. In addition, achieving cold shutdown could be delayed. Despite his d21ay, the shift supervisor acknowledged having confidence in this mode of core l 3-9

cooling based on his simulator training; he would have initiated MU/HPI cooling if "it comes to that."

The primary-side operator returned to his station and began monitoring the pres-sure in the pressurizer, which was near the PORV setpoint (2425 psig). The PORV then opened and he watched the pressure decrease. The indicator in front of him signaled that there was a closed signal to the PORV and that it should be closed (Figure 3.11). The acoustic monitor installed after the TMI acci-dent was available to him to verify that the PORV was closed, but he did not look at it. Instead, he looked at the indicated pressurizer level, which ap-peared steady, and based on simulator training, he concluded that the PORV was closed.

In fact, the PORV had not completely closed and, as a result, the pressure decreased at a rapid rate for about 30 seconds.

The operator did not know that the PORV had failed. He believed the RCS depres-surization was due either to the fully open pressurizer spray valve or to the feedwater flow to the steam generators. He closed the spray valve and the PORV block valve as precautionary measures. But subsequent analyses showed that the

)

failed PORV was responsible for the rapid RCS depressurization. Two minutes '

later, the reactor operator opened the PORV block valve to ensure that the PORV was available. Fortunately, the PORV had closed by itself during the time the block valve was closed. The failed PORV was the ninth abnormality that had occurred within 15 minutes after reactor trip.

3.8 Steam Generator Refill At about 1:50 a.m. the No. 1 atmospheric vent valve opened and depressurized the No. I steam generator to about 750 psig when the SFRCS signal was reset by the primary-side operator. The vent valve for the No. 2 steam generator had been closed by the secondary-side operator before the SFRCS signal was reset.

The indicated No. 1 steam generator level was less than 8 inches. The corre-sponding pressure and indicated level in No. 2 steam generator were about 928 psig and 10 inches, respectively. The indicated levels continued to decrease until the secondary-side operator started the SUFP after being informed by the assistant shif t supervisor that it was available and after the other operator had reset the isolation signal to startup feedwater valves.

Although the flow capacity of the SUFP is somewhat greater, approximately 150 gallons per minute (gpm) were fed to the steam generators because the startup valves were not fully opened. Essentially all the feedwater from the SUFP was directed to the No. I steam generator. At about 1:52 a.m. , the pressure in the No. 1 steam generator increased sharply while the indicated water level stopped decreasing and began slowly to increase. Since there was little feedwater sent to the No. 2 steam generator, its condition did not change significantly.

I The trip throttle valve for No. 2 AFW pump was opened by the equipment operators at about 1:53 a.m. After the SFRCS was reset and tripped on low level by the shift supervisor, the AFWS aligned itself so that each AFW pump would feed only its associated steam generator, i.e., the No. 2 AFW pump would feed the No. 2

steam generator. Thus, the No. 2 AFW pump refilled the No. 2 steam generator l and its pressure increased abruptly to the atmospheric vent valve relief set j point. The turbine governor valve was fully open when the trip throttle valve  ;

i 3-10  ;

l

was opened and the pump delivered full flow for about 30 seconds until the oper-ator throttled the flow down.

The No.1 trip throttle valve was opened by the equipment operator about 1:55 a.m.

and feedwater from the AFWS flowed to the No. 1 steam generator. However, the No. 1 AFW pump was not controlled from the control room but controlled locally by the equipment operators.

The equipment operators controlled the pump locally using the trip throttle valve. One operator manipulated the valve based on hand signals from the oper-ator who was outside the No.1 AFW pump room communicating with the control

! room operator. For two hours the AFW pump was controlled in this manner by the operators. Their task was made more difficult from the time they first entered j s tha AFW pump room by the intermittent failures of the plant communication i,ta-tion in the room.

With feedwater flow to the steam generators, the heatup of the reactor coolant system ended. At about 1:53 a.m. the average reactor coolant temperature peaked at about 592*F and then decreased sharply to 540*F in approximately 6 minutes (normal post-trip average temperature is 550*F). Thus, the reactor coolant j system experienced an overcooling transient caused by an excessive AFW flow

, from the condensate storage tank. The overfill of the steam generators caused

the reactor coolant system pressure to decrease towards the safety features

! actuation system (SFAS) setpoint of 1650 psig. To compensate for the pressure j dacrease, and to avoid an automatic SFAS actuation, at approximately 1:58 a.m.,

the primary-side operator aligned one train of the emergency core cooling system

! (ECCS) in the piggyback configuration. In this configuration the discharge of l the low pressure injection pump is aligned to the suction of the high pressure injection pump to increase its shutoff head pressure to about 1830 psig. At i about the time the train was actuated, the combination of pressurizer heaters, I makeup flow, and reduction of the AFW flow increased the reactor coolant pres-l sure above 1830 psig. As a result, only a limited amount (an estimated 50 gal-lons) of borated water was injected into the primary system from the ECCS.

At 1:59 a.m., the No. 1 AFW pump suction transferred spuriously from the con-densate storage tank to the service water system (malfunction number 10).

This action was not significant, but it had occurred before and had not been corrected. Similarly, a source range nuclear instrument became inoperable j after the reactor trip (malfunction number 11) and the operators initiated i emargency boration pursuant to procedures. (Note: One channel had been in-operable prior to the event.) The source range instrumentation had malfunc-tioned previously and apparently had not been properly repaired. Also, the i control room ventilation system tripped into its emergency recirculation mode (malfunction number 12), which had also occurred prior to this event,

, The steam generator water levels soon exceeded the normal post-trip level and

{ the operator terminated AFW flow to the steam generators. The subcooling i margin remained adequate throughout this event. The event ended at about 2  ;

! o' clock in the morning, twelve malfunctions and approximately 30 minutes after l it began.

}

3-11 i

3.9 Emergency Plan The shift technical advisor (STA) entered the control room about 15 minutes after the reactor trip and at about the time the SUFP was started. The STA was not required during this event in the manner that was envisioned, i.e. , to pro-vide technical advice and independent oversight. Instead, he provided adminis-trative assistance to the shift supervisor by consulting the emergency plan.

He also made the initial call to the NRC Operations Center at 2:11 a.m., after the plant was stable.

At that time, no emergency class was declared. In the telephone call to NRC, the fact that all feedwater had been lost during the event was reported. How-ever, the fact that the steam generators had essentially emptied and that no sources of feedwater had been available for nearly 12 minutes was not reported.

The STA had not been in the control room at the start of the event and did not have a total understanding of what had occurred. It is likely that had the plant not been brought to a stable condition quickly, and had plant safety fur-ther degraded, knowledgeable personnel in the control room would be focused on recovery efforts and would not want to take time to discuss the plant status with the NRC Operations Center.

At 2:26 a.m., the STA telephoned the NRC Operations Center to indicate that an Unusual Event had been declared. From questions asked by the NRC Operations Officer, he learned that either an alert or site area emergency had existed during the event. Plant conditions did not warrant any emergency classifica-tion at that time, although the shift supervisor had declared an Unusual Event primarily to ensure that additional personnel were made available at the plant to aid in evaluating the event and to maintain stable plant conditions.

From the interviews of those on duty during the event, it appears that all knowledgeable personnel were occupied in stabilizing the plant and, thus, not available to quickly and adequately inform the NRC Operations Center during the period that the plant had no feedwater. Although there appears to have been no intent to withhold information from the NRC Operations Center, there also was no appreciation that prompt, clear notification of the severity of an event to the NRC is essential for the NRC to perform its required functions. It should also be noted that plant management was informed of the event, including its severity, much more rapidly than the NRC. However, lengthy continued conversa-tions with plant management did not occur as would be the case when an open telephone line is maintained with the NRC Operations Center.

3-12

. . . . . - - - - .-. - . -- - - . - _ ~

I Table 3.1 Chronological Sequence of Events

! Initial Conditions Unit operating at 90% power Number One Main Feedpump (MFP) in automatic control Number Two Main Feedpump in manual control One Source Rar.ge Nuclear Instrumentation Channel inoperable Safety Parameter Display System (SPOS) inoperable, both channels 1

Transient Initiator I

j' *01:35:00 #1 MFP Trips

Control' system causes MFP flow increase; MFP turbine trips on j overspeed.

f Systems Response / Operator Actions to Partial Loss of Main Feedwater i

j 01:35:01 Unit runback initiated toward 55% at 50%/ min.

01:35:21 Operator increases the speed of #2 MFP turbine. Pressurizer (Prz)

spray valve manually opened to 100%.

! 01:35:30 Reactor Trip & Turbine Trip--RCS High Pressure (2300 psig) from j 80% power.**

j *01:35:31 Computer recorded Steam and Feedwater Rupture Control System (SFRCS),

5 full trip on low level, actuation channel 2.

I 801:35:31 Both Main Steam Isolation Valves (MSIVs) start to close.

01:35:34 SFRCS actuation signal automatically clears.

l a01:35:36 MSIV #2 has closed.

j *01:35:37 MSIV #1 has closed.  ;

With both MSIVs closed, the source of steam for #2 MFP turbine is l isolated. Steam from main steam piping and moisture separator i reheaters allowed #2 MFP to provide adequate flow for about 41s minutes.

! 01:35:45 Pressurizer spray valve closed.

01:35:56 Once Through Steam Generator (OTSG) levels at normal post-trip level (35 inches).

801:40:00 OTSG 1evels begin to fall from the normal post-trip level. l t

" Unexpected or off-normal response.

• • As part of normal reactor trip procedure, operator isolated RCS letdown and started second RCS makeup pump, to maintain pressurizer level.

3-13

Table 3.1 Chronological Sequence of Events (continued)

System Response / Operator Actions to Complete Loss of Main Feedwater .

01:41:04 SFRCS OTSG #1 low level (26.5 in.) full trip, actuation channel 1; this actuation causes Auxiliary Feedwater Pump (AFP) #1 to be aligned to draw steam from, and provide feed to, OTSG #1.

• 01:41:08 The control room operator attempted to manually initiate SFRCS; however, he incorrectly actuated the SFRCS on low steam pressure instead of the desired low steam generator level. He performed the manual actuation by depressing the top switch in each column of manual actuation switches for the two SFRCS actuation channels.

Therefore, each SFRCS actuation channel sensed that its associated steam generator was inoperable and that the opposite OTSG was intact.

SFRCS actuation channel 1 then attempted to align its associated AFW train (#1) to draw steam from only, and to provide feed only to, OSTG #2. SFRCS actuation channel 2 attempted to align its associated AFW train (#2) to draw steam from only, and to provide feed only to, OTSG #1. Both SFRCS actuation channels also closed their associated OTSG/AFW containment isolation valves. That is, SFRCS actuation channel 1 isolated OTSG #1 by closing valve AF-608; actuation channel 2 isolated OTSG #2 by closing valve AF-599. These OTSG/AFW isolation actions prevented any auxiliary feed flow from reaching either OTSG.

Per the SFRCS design, valves that had been positioned by the low level trip on SFRCS channel 1 were repositioned by the higher priority low pressure trip. The AFP 1 steam supply valve from OTSG #1, MS-106 had started open in response to the SFRCS actuation channel 1 low level trip. Following the low pressure trip, the valve should have continued opening to its full open position before it cycled closed. The entire open/close stroke time should have been about 50 seconds. *MS-106, however, returned to its closed position in about 18 seconds.

i 01:41:13 SFRCS actuation channel 2 tripped on OTSG #2 low level. Since the i low pressure trip already present had priority, no change in compo-nent actuation occurred.

• 01:41:31 Auxiliary Feedwater Pump Turbine (AFPT) #1 tripped on overspeed.

t

• 01:41:44 AFPT #2 tripped on overspeed.

System Response / Operator Actions to Complete Loss of All Feedwater 01:42:00 Manual reset of SFRCS OTSG low pressure actuation.

• AF-599, AF-608 should have reopened automatically, but did not

• An attempt was made to reopen AF-599 and AF-608 from the main 1

control panel, but the valves did not respond.

3-14

l Table 3.1 Chronological Sequence of Events (continued) 01:42:00 Pzr. Spray valve opened.

01:43:55 Assistant Shift Supervisor went to SFRCS cabinets (behind the control room area), opened the doors, and operated the " operating bypass" for the SFRCS (" Initiate Reset and Block," used for normal plant cooldowns) in an attempt to reset any automatic safety signals to AF-599 and AF-608.

l l *The valves remained closed. ,

• 01:44 Equipment Operators were dispatched into the plant to operate the following equipment:

(1) Two Equipment Operators were sent to the Auxiliary Feedwater Pump turbines to manually restore the AFW pump to service.

(2) The Assistant Shift Supervisor left the control room to make

! the startup feed pump available for service. This required I opening the pump suction valve, the pump discharge valve, and

! two cooling water valves. In addition, the control fuses for l

the 4160-volt pump motor circuit breaker were required to be installed.

j (3) Two equipment operators were sent to open OTSG Auxiliary Feed-

! water Isolation Valves AF-599 and AF-608. These valves are the containment isolations for the AFW system. The operators moved the valves from the closed position, and the motor j operators opened the valves.

j 01:44:50 RCS Makeup flow decreases as the makeup flow control valve, MU-32, j' modulates closed based on pzr. level being above setpoint (200 inches).

l 01:45:50 AFPT #2 overspeed trip reset locally.

01:46:29 OTSG#1 Atmospheric Vent Valve opened.

01:46:30 AFPT #1 trip throttle valve re-latched and valve opened (overspeed trip not cleared). Speed controlled locally throughout event.

i a

l 01:47:33 OTSG #1 below 960 psig and decreasing. ,

y 01:47:48 OTSG #2/AFW isolation valve, AF-599, opened locally, l-j 01:48:08 OTSG #1 atmospheric vent valve closed.

01:48:49 Pzr. PORV opens (first time) at 2433 psig (2425 setpoint).

l

• 01:48:51 OTSG #2 below 960 psig and decreasing. (Both OTSGs now " dried out," according to criteria in plant emergency procedures related to MU/HPI cooling.)

4 I

l i

! 3-15 i

! l

Table 3.1 Chronological Sequence of Events (continued) 01:48:52 Pzr. PORV has closed at 2377 psig (2375 setpoint).

01:49:28 OTSG #1/AFW isolation valve, AF-608, opened locally.

01:50:09 Pzr. PORV opens (second time) at 2434 psig.

01:50:12 Pzr. PORV has closed at 2369 psig.

01:50:13 OTSG #1 Atmospheric Vent Valve opened; OTSG #1 pressure decreases rapidly toward about 750 psig.

01:51:17 OTSG #1 level falls below eight inches (MU/HPI cooling criterion).

• 01:51:18 Pzr. PORV opens (third time) at 2435 psig; did not close.

01:51:23 Startup feed pump motor started.

01:51:30 Obtained flow from startup feed pump to OTSG #1.

01:51:42 Operator started to close Pzr. PORV block valve as RCS pressure fell through 2140 psig.

01:51:42 RCS Loop #1 reaches a minimum pressure of 2081 psig.

Loop #1: T-hot = 588.6*F; Tave = 587.5 F.

01:51:43 Pzr spray valve closed.

01:51:49 Acoustic monitor indicates less than 20% flow through PORV/ block valves.

01:53:00 RCS loop #1 T-hot reaches peak value of 593.5*F.

01:53:22 AFW Train #2 has significant flow, with control locally via the trip-throttle valve.

01:53:25 RCS Tave reaches peak value of 592.3*F.

01:53:35 OTSG #2 returns to above 960 psig.

01:53:56 PORV Block Valve reopened by operator.

01:54:45 OTSG #1 returns to above 960 psig.

01:54:46 AFW Train #1 has significant flow, with control locally via the trip throttle valve.

01:56:58 OTSG #2 Atmospheric vent open; OTSG #2 below 960 psig and decreasing.

01:57:05 OTSG #1 below 960 psig and decreasing.

3-16

Table 3.1 Chronological Sequence of Events (continued)

• 01:57:53 Low suction pressure developed on AFP #1; 34 seconds later (01:58:27),

suction pressure was recovered.

01:58 Tave passed through the normal post-trip temperature. The cooldown had lowered RCS pressure to about 1720 psig. Operators manually started the HPI pump #1 in the pigg>back mode (LPI pump 1 supplying the suction to the HPI pump 1) to maintain pressurizer pressure and level. A slight amount of water (about 50 gallons) was injected.

01:58:08 RCS loop #1 reaches a minimum pressure of 1716 psig.

Loop #1: T-hot = 546.6*F; Tave = 546.2*F.

01:58:28 OTSG #1 Atmospheric vent closed.

01:58:33 AFW Train #1 flow reduced to control OTSG 1evel.

• 01:58:40 AFP #1 suction automatically transferred from the condensate storage j tank (CST) to the service water system. The operator realigned to CST.

! 01:58:57 AFPT#1 overspeed trip reset.

! 02:01 When AFPT #2 was returned to service, the control room operator i controlled the pump in manual rather than returning it to Automatic.

! 02:01:13 AFW Train #2 flow reduced.

i 02:02:27 OTSG #1 returns to above 960 psig.

02:02:30 OTSG #2 returns to above 960 psig.

02:04 Plant conditions essentially stable.

I Additional Complications

1. Control Room HVAC system spuriously tripped to its emergency mode.

2. The operator attempted to override / reset the automatic close signal to the l

OTSG #2 startup feed control valve SP-7A. The reset light for this valve j did not come on, indicating that control of the valve had not been regained.

The control room operators therefore believed that flow from the S/U feed-pump went only through SP-7B to OSTG #1 and not through SP-7A to OTSG #2.

3. Upon energization, the remaining source range nuclear instrumentation channel failed off-scale low. All control rods were verified to be fully inserted, and emergency boration was initiated.

{ 4. The main turbine did not go into its turning gear.

1 J 5. When vacuum was restored and the MSIVs opened, a water slug damaged one f I of the main turbine bypass valves.  !

! 3-17 l

Table 3.1 Chronological Sequence of Events (continued)

Notes

1. The above sequence of events is based upon combining information obtained from plant computer printouts and operator interviews.

I Adequate subcooled margin was available throughout the transient. The 2.

l Reactor Coolant Pumps remained in operation. The Quench Tank contained the discharges from the PORV.

3. There is a question regarding the operation of the atmospheric vent valves.

b 4

9 3-18

_ - - - . - - .. - ~ - - - .

(

320 - 620 i i i i i i i 2500 i i l ,,

298 -

610 -

ACS Pressuse \ -

2390

/

~s \T 256 -

eco - / \ -

2280

/ \

' l \ /

• / -

224 590

/ k p 2170

i. 8 l ./ -\

\ <j 3 xJ

-: 192 t

a 5a0

~

4q "

f

>. / ./ -\- 20s0 a e

1 o l s(.

/ -/  !

o-i

/ ./ \. e

=

j\-

3 160 -4 570 -

/

1950 3.

w t t l ./ k. m L 2 5 l

. / / \ r k

/ ./- 1840 l 128 -

560 -

e -\. RCS Temperature . o a k N. __// 5 5

5 / -

96

=

y550

\

/

f (-

m0

\

\ /

68 -

540 - -% ressureer Levei P \./

• 1620 32 -

530 - -

1510

' ' ' ' ' ' ' ' ' ' 1400 C- 520 A M 1:35 31 1:38.31 1:41:31 1:44 31 1.47.31 1:50-31 1:53:31 1.56:31 1:59:31 2:02:31 Time Figure 3.1 Reactor Cool. ant System and Pressuruer Response as a Functon of Ti.~. i em 9,1985 l

I r

i i

l r

r

{ )hppk( wolf retawdeeF puratS rotareneG maetS enO rebmuN g g g g s

!, a s a a a I i i I r I 1 1 i I 1

,; ) mpg ( wolf retawdeeF yrailixuA rotareneG maetS enO rebmuN g

l a E s a a a W s s l 1 i I I I I

/ l ~

g t

)r-

/ _

w g -

> ~ .-

i I

\. l a C

l I 9 i

/

sn l

~

k N s

d s

! / N -

! N /

N- $

g g

T _.(...

! l.s.s -

e t

's.s 3-b'k_

e-f

l .

1. . '(

a

'b.. e  :

7 h

f_ sy (

} E l

l

./ l A t

i

• ) L g 1

" l_ u6 ta '

P a i

( I - 2 a.

/.s./ n, e

1 / - = t 4 .

t2 l_ $ [

1

,a I -

a I

. s, - w  :

/

c. ,

,i s- -

W 1

( ),,....--

\' _ -

• h t, / l )', k =

. i :, ... ......, .i.

~

. ,,,, .e_ . . .-. e u.

i

.,........,.... ....,.-._. _ .................a..=..

, ~ ... l i

I I s.

5 e...

,9 ..  ;

ob s

! igisp( erusserP rotareneG maetS enO rebmuN i

3 1 l l l l l i I I 1 3 8 o

$ $ $. $. M 8._ 0 M -

I )sehcnh level rotareneG maetS enO rebmuN j i

l i

C-20 f  !

- - -- . - - ---tt- n+m---p#---g -.wi-q. y-. - -+ --,# e .-..-.ry --- w-----+4--<m- 9 --

) mpg ( wolf retawdeeF yrailixuA rotareneG maetS owT rebmuN g s a a a a a s s a I f i I l i 1 i I I 1

)hppk( wolf retawdeeF putratS rotareneG maetS owT rebmuN g g g g a s a s a W i I i i/ i I I

)t 1

t

- g p -

N. I ~

\ s Ns k 2 N s

9 '.

N E. s - 'j

/

m g s'N s -

y o .

< \ g s

/ E N '-

\

k

> \ m e e

\ \

8 3,

%c.L. l-k.

l I, F l l

/ i i- s-e i '

4 I 5 e.[ ,l

)

k,

• eA,-

_ g e

• l 1l T 3 4_ y K 3 3 R

l

\  !

I

~'

(

f5 k

/

~

b- '

e 3 r h / ~

,S, ' g f -

m E

- .% a o V% /

/ -

4 t

\ >/*****...... i (fj e

s.'- .. -.e W. ...

• • • t* l

..... l W u

/

1 , , , . ..... .. .. .. ..... . . . . . . . \< . ,

.------------------k.

...........,,.L..  %

---

=======a,==.=.....

i i i i i i s

e

{ i oy k b b b N k a b N s lg. sol erusserP rotareneG maetS owT rebmuN I I I I I I I I I I I o -

M $ 9 8 M 8 N

)sehcnO level rotsteneG maetS owT #ebmuN l

0-21 F

i I -: f2" r7iry-

~[ -

j ,%. A

_ - [' g C 2;H - w fp c,,

,_, )~

< -,- h4, ~ggag:M 4, {a[

" " TRIP M ASTE R

=

fy n: , iEE j a it <-

couriRw Tuna vaie a, uLn y .u,ip

'b[d jj,a l  ;' ?' - - . . ,

s ,  ;-

t Y, -

ti <- ACP 14

• Af.P 1-2 e mit Tatt uitTAst i  ; i

' ' ' ' i I ,

3gQ .' .. j Tetr TRIP .

- -:[ ]

.u. . -

asw; i r;;; .

gM l' --

! E OktD DB 1111 000 5 0000.JO 2: 0:'g 1 i e4  :- su

• gre: ,((

-,7

{* (( . T- ii ~MM

{s,m 4

@:D ,, -

.~ gg aur nur 7.1{ Gj

, C"

. N-V -YA,... A., C~

e , s -:I/ 4 -j .

,, ,L

~g

_ _\ a ;

,u i

,a p.; . .

i Figure 3.4 Davis-Besse Control Room ,,

.:f@ .

-y. ,

l '4  ;[

I E $

W <

i 1

1 y .. , , n .~ >- L S. ,

.-i .

• CRT

(' _

n ?6 ' '. #

i j

Figure 3.5 Annunciator Panel l t

i i

I i .

l

i 5"::

7 w a r v r :.54.: ;>

~,s p~ i;4: -

P 3n%

~

F

~Ed:  ; fli  ! ..

.N u . t[ggtt(

' c .* .

4

,, ;5  ; ' h;~ j ,RY E G I. iii e a W I I $ 5 Figure 3.7 Grate Leading to Auxiliary Feedwater Pump Room

{

~

5 -

l Figure 3.6 Manual Initiation Switches for SFRCS

-.m.---.,_ar -.s---- -s-.-L- _ _ -u--- __,m_- --

+

l

l  :- .

, 2 l

r >

a E

if I i

3 l c' i.

~'

sg

! \ 6m

\

m b

= E x

I I

.: v 7 ;_

?

\

.. , (# ..

n u, '

4 i

E.  !

r 00 n

I .+

t 8

.Lff .g t

i 1

4 .

i

'l i

e., .

3-24

i

,Q, m ,

EfWE 59tE EM EF av E-a HP ggD E )5!iUT i . a m r.. c, "EU!Y$VlY RE VV I . _.. . .

7--

" i: AUTO i i

150 ,

~

ICS RCil .

-[. -

y; Ht$ 'RC2-6 2 .c.

3 i

[J, 7

[

e

" 8 _.

PRESSURIZER 1

1- ' ~

• SPRAY

! L, ,

L .

t.

[ ._4 LI N i

Figure 3.10 Control Room Indication of Figure 3.11 Control Station for Pressurizer PORV Steam Generator Level

_ __ ~ ._ _. - __ _ _ _

i i

4 OESCRIPTION OF PLANT SYSTEMS i 4.1 General Desian The Nuclear Steam Supply System (NSSS) for the Davis-Besse plant was supplied l by the Babcock & Wilcox Company. The NSSS, shown in Figure 4.1, consists of two heat transport loops with each containing a hot leg, a once-through steam l

l generator (OTSG), and two cold legs. Water from the OTSG is returned to the I reactor vessel by the reactor coolant pumps, with one pump located in each cold leg. Reactor coolant system (RCS) pressure is maintained by an electrically heated pressurizer that is connected to one of the hot legs. During normal operations, the pressurizer contains a 700-cubic foot steam bubble that exerts a pressure of approximately 2150 psig on the RCS. Protection against overpres-surization is provided by the pliot operated relief valve (PORV) and two code The pilot operated relief valve discharges to a quench tank.

j safety valves.

The two code safety valves discharge directly to the containment building.

The reactor design power level is 2,772 MW(t), which is also the design power level for the station and all components. At a power level of 2,772 MW(t),

the net station electrical output is 906 MW(e).

4.2 Main Steam System f

1 l The main steam system functions to deliver superheated steam from the steam

) generators (OTSGs) to the main turbine and required plant auxillaries. As

shown in Figure 4.2, the system begins with the outlet piping from the steam generators and passes through the containment building to the main steam isola-l tion valves (MSIVs). Protection against overpressurization for the steam gener-

! ators is provided by 18 code safety valves (9 per steam generator) located on the system piping upstream of the MSIVs, and two atmospheric vent valves (one j per steam generator) which act as relief valves. The atmospheric vent valves are controlled by the integrated control system (ICS) and aid in controlling '

]

7 steam pressure if a large transient occurs when the unft is in service, if

! condenser vacuum is lost, or if the M51Vs are closed. Connections upstream of each main steam Isolation valve supply steam to the redundant turbine-driven auxiliary feedwater pumps. Either system header is capable of supplying either turbine; however, the auxiliary feedwater pump turbine normally receives steam I from its associated steam header.

The piping downstream of the MSIVs contains non-return valves that prevent re-verse flow when steam generator pressures are not equal. From the non return <

] valves, steam flows to the high pressure turbine and secondary systems, such as the air ejectors,

}

During normal operations, the main steam system valves are not required to

change position; however, reactor trips and steam and feedwater rupture control system actuations cause changes in valve position. When the reactor trips, OTSG pressure rises rapidly resulting in the actuation of the steamline safety valves.

! The integrated control system (IC$) biases the steam generator pressure control l

setpoint to a value higher than the normal steam header pressure control value I 4-1

to minimize the cooldown of the reactor coolant system. Once the ICS gains control of the steam pressure, the safety valves should close.

The steam and feedwater rupture control system (SFRCS) also changes the position of the main steam system valves. If a SFRCS actuation signal is received, the following changes can occur in the system:

1. The MSIVs close.

2. The atmospheric vent valves close l

3. The steam supply valves open to supply steam to the auxiliary feedwater pump turbines.

4.3 Main Feedwater System The main feedwater system, Figure 4.3, begins with the cross-connected deareator storage tanks. Each of these tanks has a capacity of 64,000 gallons and provides the required net positive suction head (NPSH), i.e. pressure, for the booster feedwater pumps. The booster feedwater pumps are driven through a gear reducer 4

by the main feedwater pump turbines and function to increase system pressure to satisfy the suction requirements for the main feedwater pumps. The direct-driven main feedwater pumps increase feedwater pressure to a value greater than steam

generator pressure and discharge through the high pressure feedwater heaters to the feedwater regulating valves.

l Two parallel valves are used to govern the flow of feedwater to each OTSG. The first of the two valves is called the startup control valve and regulates feed-water flow from 0% power to approximately 15% power. Startup control valve SP-78 supplies the #1 OTSG, and startup control valve SP-7A supplies the #2 OTSG. When the startup control valves reach the 80% open position, the main feedwater regulating block valves open, and flow is also controlled by the main i feedwater regulating valves. The main feedwater regulating valves control feed-j water flow during the power escalation from 15% to 100%. The pressure drop

across the valve network is monitored and used to control main feedwater pump

! turbine speed. From the outlet of the feedwater regulating valves, the feed-water travels to the OTSGs via a motor-operated main feedwater isolation valve.

Main feedwater is added to the OTSG through the external main feedwater ring and the main feedwater nozzles.

A separate auxiliary feedwater ring is used for the addition of auxiliary feed-i water flow. After entering the steam generator, auxiliary feedwater is sprayed on the tubes to enhance natural circulation when reactor coolant pumps are not running and to minimize thermal shock to the steam generator.

^

When the plant is in mode 3 (Hot Standby), a motor-driven startup feedwater pump is used to maintain steam generator level. The startup feedwater pump l receives its suction from the deareator storage tanks and discharges to the steam generator main feed rings via the high pressure feedwater heaters, the feedwater regulating velves, and the main foodwater isolation valves. After reactor criticality is achieved, power is escalated to about 1% and a main feedwater pump is placed in service. When the main feedwater pump is in ser-i vice, the startup feedwater pump is shutdown and isolated from the main feed-

water system. Startup feedwater pump isolation includes the closing of the l suction, discharge, and the cooling water isolation valves. All of these valves are located in the turbine building and must be locally operated, in addition i

1 1

4=2

1 l

to the manual operation of the pump isolation valves, the breaker control power fuses are removed as a safety precaution. This prevents the operation of the pump with its suction supply isolated.

The startup feedwater pump is designed to deliver feedwater flow at approximately 200 gpm with a steam generator pressure of 1050 psig. Electrical power is sup-plied to the pump motor from the non-Class 1E distribution; however, the pump power supply may be manually transferred to the diesel generator busses if re-quired. Operation of the startup feedwater pump in of f-normal situations re-quires the manual opening of the suction, discharge, cooling water inlet and outlet valves, and the installation of the breaker control power fuses.

If the reactor trips, the feedwater system is controlled by the rapid feedwater reduction system which closes the main feedwater regulating valves and positions

! the startup control valves to a position that allows proper OTSG level control.

These actions are taken to prevent excessive cooling of the RCS caused by over-feeding the steam generators. This system also increases the speed of the oper-l j ating main feedpump turbine (s) from a normal value of 4400 rpm to 4600 rpm.

I In addition to the control actions described above, the steam and feedwater rupture control system (SFRCS) closes the main feedwater regulating valves, the i startup regulating valves, and the main feedwater isolation valves when certain abnormal plant conditions are detected.

I I

4.4 Auxiliary Feedwater System i

i The auxiliary feedwater system (AFW), Figure 4.4, is designed to remove the i core's decay heat by the addition of feedwater to the steam generators following I a reactor trip, if main feedwater is not available. The system consists of redundant turbine-driven auxiliary feedwater pumps and associated piping. Three (

suction sources are available to the AFW pumps: the deareator storage tanks, the condensate storage tank (CST), and the service water system. The CST serves i

as the normal suction source for the system; however, if a low suction pressure condition is sensed, the AFW suction will automatically transfer to the service water system. Manual action would be required to transfer suction to the deareator storage tanks.

j When the AFW system is actuated by the steam and feedwater rupture control sys-4 tem (SFRCS) on signals other than low steam generator pressure, the steam to

drive the AFW pump turbine and the discharge of each pump are aligned with the i associated steam generator. Each of the AFW pumps is rated at 1050 gallons per minute (gpm) when pumping against a steam generator pressure of 1050 psig; j 250 gpm of the 1050 gpm is used for recirculation flow.

l The #1 pump supplies the #1 OTSG via motor-operated valves AF-360, AF-3870, and j AF-608. The feedwater supply for #2 OTSG is from the #2 pump through valves AF-388, AF-3872, and AF-599. However, if the SFRCS is actuated on low OTSG pressure, the flow path of the system is altered to prevent the feeding of a ruptured steam generator. The isolation of feedwater to the faulted steam gen-erator is accomplished by closing the AFW cuntainment isolation valve (AF-599

]

l or AF-608). Feedwater to the intact steam generator is supplied by both The pumps steam 1 through the appropriate cross-connect valve (AF-3869 or AF-3871).

supply valves for the turbine-driven pumps are also realigned to provide steam for both pumps f rom the intact steam generator. The following listing gives the

)

position of the AFW system valves during various SFRCS actuations

i 4-3 i

I I

NORMAL SYSTEM ALIGNMENT Open valves - AF-360, AF-388, AF-599, AF-608 Closed valves - AF-3869, AF-3870, AF-3871, AF-3872, MS-106, MS-106A, MS-107, MS-107A SFRCS LOW LEVEL ACTUATION Optn valves - AF-360, AF-388, AF-3870, AF-3872. AF-599, AF-608, MS-106, MS-107 Closed valves - AF-3869, AF-3871, MS-106A, MS-107A SFRCS ACTUATION #1 OTSG LOW PRESSURE

Open valves - AF-360, AF-388, AF-3869, AF-3872, AF-599, MS-106A, MS-107 i Closed valves - AF-608, AF-3871, MS-106, MS-107A SFRCS ACTUATION #2 OTSG LOW PRESSURE l

Open valves - AF-360, AF-388, AF-3870, AF-3871, AF-608, MS-106, MS-107A Closed valves - AF-3869, AF-3872, AF-599, MS-106A, MS-107 i

The SFRCS is also described in section 4.6.

4.5 MU/HP! Coolina Systems Makeup /High Pressure Injection (MU/HPI) core cooling (also called PORV cooling or feed and bleed core cooling) involves the use of the makeup and purification system, the high pressure injection system and, at the operator's discretion, 1

the low pressure injection system. These th-ee systems are shown in Figure 4.5.

' The system contains two multistage centrifugal makeup pumps rated at 150 gpm, with a discharge pressure of approximately 2500 psig. Two suction sources are

) available to the pumps; the makeup tank and the borated water storage tank (BWST).

j During normal operations, the makeup pumps supply seal injection and control pressurizer level by discharging into the RCS via the makeup flow control valve j (HU-32). The discharge of the makeup pumps enters the RCS through one of the high pressure injection penetrations. When feed and bleed operations are re-quired, plant procedures require the positioning of the three-way suction valve (MU-3971) to the BWST suction source, fully opening the makeup flow control

valve, and the starting of both makeup pumps.

1 The high pressure injection pumps (HPI) are a part of the emergency core cooling system and are not in service during normal operations. The system consists of i

redundant pumps and four injection paths into the cold legs of the RCS. The l pumps receive their suction from the BWST and have a shutoff head of 16J0 psig.

When these pumps are used in the feed and bleed mode of core cooling, both pumps are started and the discharge paths into the RCS are opened. However, in order to supply a flow of cooling water to the core, RCS pressure must be less than the shutoff head of the HPI pumps or the pumps must be " piggy-backed" to the l discharge of the low pressure injection pumps at described below.

{

The low pressure injection (LPI) pumps are also a part of the emergency core l l cooling systems. The LPI pumps recotve a suction from the BWST and discharge '

via the decay heat removal coolers (not shown in Figure 4.5) into the reactor l

4-4

(

l v2ssel. The pumps are rated at 3000 gpm with a discharge pressure of approxi-mately 150 psig. The shutoff head of the pumps is about 200 psig. Plant pro-ccdures allow the. discharge of the LPI pumps to be aligned to the suction of the HPI pumps by opening valves OH-62 and DH-63. This alignment increases the discharge pressure of the HPI pumps from 1630 psig to approximately 1830 psig cnd allows HPI flow at a higher RCS pressure.

When the feed and bleed mode of core cooling is required, plant procedures call l

for starting the makeup pumps and the high pressure injection pumps. After the pumps are in service, the pressurizer pilot-operated relief valve, the pres-surizer vent, and the hot leg vents are opened. The HP!/LPI piggy-back mode l

j of operation is not specifically addressed in the loss of subcooling margin or i the overheating sections of plant procedures but may be aligned at the discre-tion of the operator. All the required bleed and feed alignments are performed in the control room.

4.6 Steam and Feedwater Rupture Control System (SFRCS)

The steam and feedwater rupture control system (SFRCS) is provided in the plant dssign as an engineered safety features actuation system for postulated tran-sient or accident conditions arising generally from the secondary (steam gen-

) eration) side of the plant, because the OTSGs serve as the heat sinks for the j rsactor power. The SFRCS senses loss of main feedwater (MFW) flow, rupture of an MFW line, and rupture of a main steamline. It also senses loss of all forced coolant flow in the primary system.

The safety function of the SFRCS is to provide safety actuation signals to squipment that will: 1solate the steam flow from the OTSGs, isolate the MFW flow, and start and align the AFW system. The SFRCS also provides output signals to the turbine trip system and to the Anticipatory Reactor Trip System

(ARTS).

In the event of loss of MFW pumps or a main feedwater line rupture, the OTSGs l' would start to boil dry, and, if action is not initiated promptly, there would be no motive steam available for the turbine-driven AFW system and the OTSGs would be lost as heat sinks. As soon as the MFW pump discharge pressure falls below the pressure in the OT5G (i.e., reverse differential pressure across a check valve) by a predetermined value, the SFRCS provides safety actuation sig-nals to close the main steam isolation valves (MSIVs), close t,?e MFW stop and control valves, and start AFW. The SFRCS also receives OTSG low level signa 19 which are diverse from the reverse differential pressure signals.

In the event of steamline pipe ruptures, when the main steam pressure drops, the SFRCS will close both MSIVs and the MFW stop and control Scives. The description of the SFRCS in the Updated Safety Analysis Report (USAR) Sec-tion 7.4.1.3 does not mention the SFRCS closure (or re-opening) of the AlW containment isolation valves (AF-608 and AF-599), although the design does include such features. The AFW is also initiated and both AFW trains are cligned to draw steam only from, and to provide feed only to, the unaffected

" intact" OTSG.

In the event of loss of all four reactor coolant pumps (RCPs), forced cooling flow of the reactor coolant system would be lost and AfW flow is needed to en-hence natural circulation flow. Therefore, the SFRCS senses the loss of four RCPs and automatically initiates AFW, 4-5

Figure 4.6 depicts the channelization of the SFRCS. There are two Actuation Channels, each of which contains two identical logic channels. Within each Actuation Channel, one logic channel is ac powered and the other logic channel is de powered. The field wiring at the actuated equipment is such that gener-ally both logic channels must " trip" (i.e., a two-out-of-two AND logical arrange-ment) to actuate most equipment, which is referred to as a " full trip." How-ever, some equipment is actuated by a " half trip" (i.e. , only one logic channel of an actuation channel has tripped). For example, the atmospheric steam vent valves are closed by " half trips."

4.7 Pressurizer Pilot Operated Relief Valve (PORV)

At the top of the pressurizer as shown in Figure 4.1, there are two code safety

! valves which vent directly to the containment atmosphere, a high point vent line, and the pilot operated relief valve (PORV) with its associated upstream block valve.

The PORV block valve is a manually-controlled motor-operated valve, equipped with position instrumentation including a position alarm.

The PORV is a style HPV-SN solenoid-controlled pilot-operated pressure relief valve manufactured by the Crosby Valve and Gage Company. It is the Team's under-standing that Davis-Besse is the only B&W-designed PWR that has a Crosby PORV.

The Crosby PORV is operated by the reactor coolant system pressure via a solenoid-operated pilot valve and therefore does not involve any pneumatic power

, (instrument air or nitrogen). Electric power is used for the solenoid control device. To actuate the PORV, the solenoid is energized. This action allows the use of reactor coolant system pressure to open the main disc of the valve.

I l The controls for the PORV include features for automatic operation, manual open.

l l

manual close, and lock open. All operations to open the PORV involve energizing a control relay which in turn energizes the PORV solenoid. In automatic, the pressure channel's bistable would close one set of contacts above the high pres-i sure setpoint (2425 psig) and would close another set of contacts below the low pressure setpoint (2375 psig). When the high pressure setpoint is reached, the control relay is energized and an electrical seal-in circuit is energized. When the low setpoint is reached, an auxiliary relay is operated which in turn inter-l rupts the valve-open seal-in circuit, a

In manual control, the circuit is designed for momentary-only operation uf the switch to the valve-open position. The seal-in circuit will hold the valve open if the pressure is above the low pressure setpoint. To lock open the PORV (as would be done for MU/HP! cooling), manual control switch would be rotated to the " lock open" position. The control circuitry would maintain the PORV solenolo energized regardless of RCS pressure. To manually close the PORV, the control switch must be rotated to the " auto" position and the control switch pushed inward. This action causes both control relays to be de-energized and the seal-in circuit to be de-energized, which in-turn causes the PORV solenoid to be de-energized.

Shown in Figure 3.11, the indicators for the PORV include: control power avall-able light (blue), automatic (white), PORV open (red), PORV close (green), lock open (amber). The PORV open/close lights are operated by a limit switch operated by the PORV solenoid plunger (1.e., the output of the electric solenoid; the i

4-6 1

l

l I

i mechanical input to the PORV), All of these position lights are PORV command indicators, in that they indicate only the position that the electric controls hive commanded for the PORV. Only the acoustic monitor is a direct indicator of the flow condition through the PORV/ block valve path.

The acoustic monitor for the PORV was installed as one of the post-TMI safety improvements. Two redundant accelerometer sensors are mounted on the discharge piping. Each sensor channel provides a signal to drive the remote 0-100% (open)

PORV position meter on the post-accident monitoring (PAM) panel, and an adjust-cble position signal switch to drive the remote PORV open/ closed lights on the PAM panel. The Team was told that the adjustable switch was set such that the r:d (open) light would be energized if the flow signal is greater than 22% of the full flow value.

If PORV/ block valve flow is less than 22%, the red (ope.1) light would be turned off and the green (closed) light would be energized. The meter could be used to obtain more precise position / flow information. The PAM panel is a separate ptnel mounted about 7 feet to the left of where the reactor operator assigned to the primary system would be standing. Both redundant red / green PORV indicat-ing lights are easily visible to the operator if he turns his head. However, the 0-100% meters are relatively small, i.e., about a 3-inch tall vertical edge-mounted meters. To read this meter, the operator would have to step a ptce or two toward the PAM panel.

4-7 l

PRESSURIZER --

8 -

(SEE INSERT) s

, s , .~

l 'i i

t .

AUX -

FEED + i l lNLET l

. , 1 MAIN  : HOT LEG 11 I

FEED -+ OTSG OTSG INLET I l l

_,-s a . _ ..- ""

I SURGE ---*

LINE 's e E] e REACTOR COOLANT c " _ - -M.--.>

PUMP i

i k I.

I

.I I

d

~

T k Q,q -.

~ s

,,f ,,

s./ + - '

COLD REACTOR PRESSURIZER INSERT LEG s VESSEL N

\ ,o.,,,,,

g l g s wer,v m ve no,m a ve=1 nozas

- =m. om.

V - ,t 1 1 /\ /

j ... .c.

ta m wws d W vtS4EL tuPPostT THEnteoWELL g SAasPLsNG NolILE

, waren sumoLa

,_ ,el i

'-oi

,. j

'r= ='==rne \

eve cal o, =

U_- :: ,,

p

\

l eunes L norns Figure 4.1 Davis-Besse Nuclear Steam Supply System 4-8

SAFETY VALVE (9/HDR) Q ATMOSPHERIC VENT CONTAINMENT  ! Q l TO MAIN TURBINE l l gg,y h [ MS-106 h ( MS-107A

1. 1 OTSG V

TRIP TRIP 7 THROTTLE THROTTLE

l l GOVERNOR l l GOVERNOR

\ AFW \ AFW TURWNE TURWNE

1. 1 #2 SAFETY VALVE l k [ MS-106A  % ( MS-107 -

CONTAINMENT I 1

Q TO MAIN TJRBINE l l MSIV

! - ATMOSPHERIC

~

m l VENT

1. 2 OTSG V

Figure 4.2 Main Steam System

( ( DEAREATOR TANKS IC BOOSTER BOOSTER FEED PUMP

$ 3 FEED PUMP

's '

TUR88NE TURBINE DRIVER DRIVER MAIN FEED MAIN PUMP f FEED MMN o PUMP MAIN FEEDWATER FEED REGULATING n n n BLOCK VALVE

-- -- -- SP-7s TO OTSG #1

<> <J ss START UP FEEDWATER HIGH PRESSURE STAR UP PUMP VALVE FEEDWATER HEATERS rm rs n

- - - SP-7A

. __ _. TO OTSG #2 NOTE. SYSTEM VALVES h0T INVOLVED IN STARTUP THE 6 VENT OMITTED V V V REGULATING FOR CLARITY VALVE Figure 4.3 Main Feedwater System l

l

SERVICE Ap.3870 WATER u

N = To orso ni AF-360 AF

1. 1 FROM AF CST i 3869

?

FROM mm AF DEAREAToR 3871 h

AF-388 AF N h AF

=- To oTso n2 ji 3872 599 SERVICE WATER Figure 4.4 Auxiliary Feedwater System

t' BWST

' I f FROM MAKEUP TANK Y MAKEUP U-32 MU-3971 rA Y

=

Ji HIGH PRESSURE DH-63 INJECTION

,9 N TO RCS COLD LEGS 3pTO

• REACTOR VESSEL Qs2 DH-62 LOW PRESSURE INJECTION PUMPS 3 TO RE ACTOR Q L VESSEL Figure 4.5 Makeup /HPl Cooling System

ACTUATION CHANNEL #1 1 l SotiC = l LOGIC l

[

REV. dp SWITCHES C INPUT CHANNEL l RELAY l OUTPUT T 7 BUFFERSg l ,j g

DRIVERS RELAYS LOW PRESS. SWITCHES C D g  !

RCP MONITORS C D l l

• g i tociC I g e CHANNEL .., g g-I l C l C 3 l l C 3 i l I

?

G I

m l l

- l LOciC I l

C 1 CHANNEL l 3 a2 l C-- - 3 I l I I C '

I I + - -

l  !

+

- l l

- . s l MOV LOciC "AND" l l l 1

C  ? l CHANNEL l l C i I l l C l l l

~

INSTRUMENT SENSING ACTUATION CHANNEL #2 CH AN N EL" Figure 4.6 SFRCS Block Diagram

5 EQUIPMENT PERFORMANCE This section identifies and discusses the equipment problems related to the June 9, 1985 event. The section is divided into two parts: pre-existing condi-tions, and problems that occurred during the event. The fact that such a large number of failures occurred and that several common mode failures occurred during l this event are major concerns. Effective evaluation of operating experience l related to the equipment performance and effective troubleshooting, maintenance, i

and testing of the equipment would likely have prevented many of these failures.

5.1 Pre-existing Conditions This section discusses equipment conditions known to have existed prior to the onset of the June 9 event.

1 5.1.1 Safety Parameter Display System (SPDS)

At Davis-Besse, the safety parameter display system (SPDS) is also referred to as the "ATOG display" (for Abnormal Transient Operating Guidelines which were developed by the Babcock & Wilcox (B&W) Owners' Group). This display system summarizes the most safety-significant plant variables on TV-screen displays in real-time.

The SPDS was inoperable prior to, and remained out of service throughout the event. Both independent SPDS display units were inoperable due to separate but similar problems in the data transmission system between the control room terminals and their respective computer processors. The failures were inter-mittent in nature. The diversity of the SPDS display sources (Ramtek and Chromatics display devices) has often allowed at least one display to remain operable. However, it is the Team's understanding that the SPDS at Davis-Besse has not proven reliable; Toledo Edison indicated that the failure rate of these units is higher than is acceptable.

5.1. 2 Source Range Nuclear Instrumentation The Davis-Besse design includes redundant nuclear instrumentation channels (NIs) for each of the overlapping regions

power range, intermediate range, and source range. For the source range, two channels are provided. For a plant restart, the Technical Specifications require that both source range NIs be operable.

If, after startup, a source range NI is lost, power operations may continue.

Problems with NI-l (source range channel 2) have existed prior to the initial plant startup in 1977. On June 4, 1985, NI-l was declared inoperable because (even with its detector high voltage supply turned off) it read a steady 1 decades (counts per second) greater than the redundant channel. Further, the NI-1 channel seemed to be experiencing intermittent count rate spiking problems whether its detector high voltage-was on or off.

! 5-1 e

l

Since January 1985, five maintenance work orders had been initiated for NI-1 due to these two problems. In each case, t,he Technical Specifications survell-lance test was performed and the channel was declared operable. The cause of the problems was not definitively identified.

The other source range nuclear channel (N1-2, channel 1) has also had a his-tory of unresolved intermittent problems. For example, on March 25, 1985, NI-2 failed off-scale low (less than 1 X 10 1 counts per second). When the reactor protection system cabinet door was opened, the count rate level indication returned to normal. The Technical Specification surveillance test was then performed and the channel declared to be operable without any troubleshooting effort. Similarly, on April 13, 1985, NI-2 failed off-scale low. Prior to performing any troubleshooting, the instrumentation and control (I&C) technician tapped on the front of a module in the cabinet and the count rate returned to normal. After a visual inspection, the Technical Specification surveillance test was performed and the channel was declared again to be operable. No additional I troubleshooting was performed.

5.1. 3 Startup Feedwater Pump At the Davis-Besse plant, the steam turbine-driven AFW system is not used nor-mally for plant startup or shutdown. Instead a separate, non-safety-related electric motor-driven startup (S/U) feedwater pump was provided as part of the original plant design. The avai.1 ability of the S/U feedwater pump has been an important consideration in Toledo Edison's assessment of the AFW system relia-bility and in assessing equipment which could be used to mitigate events involv-ing loss of main feedwater. In October 1984, Toledo Edison advised the NRC of a previously unanalyzed condition regarding the S/U feed pump. The associated piping had not been designed or analyzed for a postulated high (or moderate) energy line break. Since the non-safety-related S/U feed pump is in the same room as one of the safety-related AFW pump turbines, and the associated S/U feedwater piping runs through both AFW pump rooms, Toledo Edison proposed cer-tain compensatory actions. These actions included staticning an operator in the room during operation of the S/U pump and closing certain manual isolation valves. In January 1985 the NRC found the proposed compensatory actions acceptable and Toledo Edison implemented them, as well as the removal of fases in the control circuit for the breaker.

Thus, making the S/U pump available for service involves five separate manual actions at four different locations in the plant: (1) a pump suction valve must be opened; (2) the pump discharge valve must be opened; (3) two pump cooling water valves must be opened; and (4) the control fuses for the 4160-volt circuit breaker for the pump motor must be re-installed.

5.1.4 Control of Main Feedwater Pumps During the 1984 refueling of the plant (plant restart took place in January 1985), replacement control equipment for the main feed pumps (MFPs) was installed to improve plant performance following a plant trip.

During a previous event on April 24, 1985, when operating at 98% power, a flux /

delta flux to flow automatic reactor trip occurred. Approximately 8 seconds later, MFP No. I turbine tripped unexpectedly. The cause of the MFP trip was 5-2

never positively identified. Additionally, several MFP instruments were recali-brated. (Note: there have been difficulties in obtaining proper speeds for rapid feedwater reductions. For example, prior to April 24, 1985, the target speed was thought to be set to 4800 rpm, when in fact it was actually set to 5150 rpm. Following the April 24, 1985 trip, the target speed was thought to have been readjusted to 4600 rpm, when in fact it was actually set to 5000 rpm.)

During a plant trip on June 2, 1985, both MFPs tripped unexpectedly 4 seconds after the plant trip. Several possible failure causes were postulated by Toledo Edison's staff; however, troubleshooting was not able to substantiate any of these. Following the June 2, 1985 trip, further adjustments were made to the target speed voltage for the rapid feedwater reduction control system.

l Although some of Toledo Edison's staff expressed reservations, Toledo Edison's senior management decided not to delay the plant startup to resolve the MFP control problem. It was decided that one MFP would be operated in automatic and the other MFP would be operated in manual. During the plant startup with the plant at 56% power and increasing, the No. 1 MFP (in automatic control) tripped causing a plant runback. As a result, additional testing was performed on June 5 and 6, 1985.

1 i

5.1. 5 Flux / Delta Flux to Flow Reactor Trip Instrumentation At Davis-Besse, the reactor protection system design includes a reactor trip on the ratio of reactor coolant flow to neutron flux / delta flux. The flow por-tion of this instrumentation had been experiencing some " noise" problems. The magnitude of this noise was sufficient to reach the trip setpoint if the reac-tor was operated at 100% power (as had occurred on April 24, 1985); however, the instrumentation was considered to be " operable."

j It is the Team's understanding that this " noise" problem has existed since new flow instrumentation was installed during the 1984 refueling outage. Toledo Edison's efforts to resolve this problem had not been effective. As a result, it was decided that the reactor would be operated at 90% power to avoid further reactor trips due to the " noise." Had this problem been resolved, the plant would have most likely been much closer to 100% power at the onset of the June 9, 1985 event.

t 5.2 Equipment Problems That Occurred During the Event This section discusses the equipment problems that occurred during the June 9, 1985 event. Each problem is described, followed by the results to date of Toledo Edison's root cause determination. Related background information is also given. A couple of the problems are not of major safety significance but are included in order to convey the overall situation with respect to the prob-lems that the operators faced during the event. A brief summary of these equip-ment failures is provided in Table 5.1.

5.2.1 Control of Main Feedwater Pumps (MFPs)

While the Davis-Besse plant was operating at a steady 90% power level on June 9, 1985, the transient was initiated by a spontaneous and substantial speed increase of the No. 1 MFP and the subsequent MFP trip on overspeed.

5-3

- , , ,-w- - - - , - . - - - - g -- -

Toledo Edison's troubleshooting plan for this item was " Action Plan for Main-feed Pump Control System," Action Plan No. 8, dated June 18, 1985. Prior to the initiation of troubleshooting activity, Toledo Edison's hypothesis was that the root cause for this failure involved one or more of the following conditions:

1. Loose connections associated with the electrical circuitry for the MDT-20 control system.

2. A circuit board malfunction.

3. Hydraulic / mechanical control problem.

During a meeting on July 12, 1985, Toledo Edison discussed the status of the i troubleshooting of this item. The frequency-to-voltage converter (F/V) module I on circuit board number 4 was found to be faulty.

The circuit board was sent to the General Electric (GE) factory in Fitchburg, MA. The factory confirmed that the F/V converter had failed in a manner which GE classified as a " random failure." However, Toledo Edison has not presented an engineering report to support the conclusion that the circuit board failure was the root cause for the overspeed of MFP No. 1.

5.2.2 Closure of Both MSIVs, Spurious SFRCS Actuation Early in the June 9,1985 event, both main steam isolation valves (MSIVs) closed, causing the loss of the main steam source for operating the second MFP.

The MSIVs are tripped closed automatically by the safety features actuation system (SFAS) and by the steam and feedwater rupture control system (SFRCS).

During the June 9, 1985 event, there was no annunciator indication in the con-trol room of either a partial or full actuation of the SFAS or the SFRCS at the time of MSIV closure. Initially, Toledo Edison believed that both MSIVs had failed due to causes unrelated to other systems. Currently, Toledo Edison believes that the MSIVs were responding to a spurious full trip of SFRCS Actua-tion Channel #2 on OTSG low level which the alarm print shows occurred a few seconds earlier.

Toledo Edison's troubleshooting plan for this item was " Low Steam Generator Level Trip of SFRCS," Action Plan Nos. 5, 6, and 7, dated June 22, 1985.

Toledo Edison's hypothesis, based upon information from the nuclear system vendor Babcock & Wilcox, was that the SFRCS trip was caused by the OTSG level transmitter's response to a rapid oscillatory pressure wave phenomenon that occurs in the OTSGs subsequent to the closure of the main turbine stop valves.

During a meeting with the Team on July 11, 1985, Toledo Edison indicated that no actual troubleshooting had started on this item. Thus, Toledo Edison has  :

l not presented an engineering report to support the results of the root-cause determination.

During the 1984 refueling outage, the OTSG 1evel transmitters providing level control and level indication on the main control panel were changed from B&W/

Bailey Model BY transmitters to Rosemont model 1153 transmitters. These trans-mitters share OTSG taps and sensing lines with the level transmitters which provide the OTSG level input signals to the SFRCS.

5-4

The Bailey Model DY trcnsmitters require a volume displacement to operate the bellows. Toledo Edison believes that this volume displacement served to absorb (dampen) some of the oscillatory pressure phenomenon in the instrument sensing lines. The replacement Rosemont transmitters require no significant volume displacement for their operation. Toledo Edison believes that the resultant loss of damping in the sensing lines due to the new transmitters may have caused the SFRCS level transmitters to sense the pressure phenomenon to a degree that spurious trip signals were generated.

The modifications completed during the refueling outage that may affect this equipment include:

a. replacement of the amplifier and calibration boards within the level trans-mitters for the SFRCS to meet equipment nualification needs, i b. replacement of the low level bistable modules for the SFRCS level input I

channels with dual high/ low bistables,

c. opening of auxiliary feedwater pump turbine (AFPT) steam crossover valves on all AFW system actuations; this modification was later functionally removed after water hammers were experienced, and

d. modification of the OTSG blowdown valves.

Toledo Edison stated that prior to the 1984 refueling outage there had not been i any spurious actuations of the SFRCS on level, but that during the five months i between the January 1985 restart and the June 9,1985 event, the plant experi-enced two spurious partial actuations of the SFRCS on OTSG low level following turbine trips. These actuations were made more difficult to analyze because

the SFRCS seemed to automatically reset itself after a few seconds. The SFRCS design does not include seal-in features to maintain the safety actuation sig-nal until deliberate reset action is taken by the reactor operator.

The time delays associated with the main annunciators in the control room may I

not have indicated the actuation and reset times accurately. Further, the alarm

! printer apparently does not distinguish well between a partial and a full actua-i tion of an SFRCS logic actuation channel, and the sequence of events monitor

might print " full trip" for either a partial or full actuation.

Following the first of these spurious actuations (on April 24, 1985), the main-tenance work order called for running the monthly Technical Specification i surveillance test while checking for anomalies; none were found. Following the i

second spurious actuation (on June 2, 1985), the maintenance work order called i for testing the alarm logic to determine why a full-trip alarm occurred when i l only a half-trip existed. In the process of re-connecting a connection opened l for troubleshooting, the problem cleared, and no further effort was made to l troubleshoot the equipment.

l l 5.2.3 Main Steam Safety Valves, Atmospheric Vent Valves i

! After the reactor trip on June 9, 1985, all 18 of the main steam safety valves apparently lifted. This determination is based upon the fact that all the l

canvas exhaust hoods were later found to be missing; they apparently were forced l

5-5

off by the exhausts during the event. Subsequent to the trip, repeated lifts of one or more of the safeties on each steam header were experienced intermittently

, for several minutes, resulting in pressure swings of approximately 50 psi.

There were also several periods when steam header pressure swung over 100-250 psi for several minutes. Toledo Edison believes that this degree of pressure change was abnormal.

The Toledo Edison's troubleshooting plan for this item was " Report on Main Steam Header Pressure" Action Plan No. 16, dated June 25, 1985. Toledo Edison's hy-potheses were that the unexpected header pressure swings could have been caused 1

by an extended blowdown of one or more safety valves on each steam header, by leakage steam flow past the safety valves, by malfunction of the atmospheric vent valves, or by malfunction of the controls for the atmospheric vent valves.

During a meeting on July 11, 1985, Toledo Edison discussed the status of the troubleshooting of this item. Two ICS modules were found to be out of calibra-tion. These discrepancies would have caused the atmospheric vent valves to have opened at about 1030 psig instead of 1015 psig, which is the ICS setpoint for vent valve control in a post-trip situation when the turbine bypass system i

is not available. Further, the bore size of the inlet piping to the main steam safety valves was found to be smaller than the manufacturer (Dresser) stated.

However, neither of these results is believed to explain the conditions observed on June 9, 1985, and the troubleshooting is continuing.

As of this time, Toledo Edison has not presented an engineering report to support the root cause determination for the pressure change.

5.2.4 AFW Trains No. I and No. 2 Turbine Overspeed Trips During the initial acceleration of the AFW pump turbines (AFPTs), both tripped

, on overspeed. This caused a complete loss of auxiliary feedwater.

, Toledo Edison's troubleshooting plan for this item was " Auxiliary Feed Pumps Overspeed Trips," Action Plan Nos. 1A and IB, dated June 20, 1985.

Toledo Edison's primary hypothesis is that steam, which had condensed in the supply lines, formed saturated water slugs which went through the turbine gover-nors, flashed in the nozzles of the turbines and caused overspeed. Alternate or contributing hypotheses included: " double start" of AFPT #1 in that it j was rolling on steam from OTSG #1 via valve MS-106 prior to receiving steam '

from OTSG #2 via valve MS-106A, a sudden decrease in pump loads due to an abrupt closing of AFW containment isolation valves AF-608 and AF-599, governor problems, and loss of pump suction.

During a meeting on July 12, 1985, Toledo Edison discussed the status of the troubleshooting of this item. Both AFPT governors were inspected by the manu-facturer (Woodward); no problems were found. Analysis shows that large amounts of steam would condense in the steam supply pipes to the AFPTs, especially in the crossover lines (e.g., OTSG #2 to AFPT #1).

Toledo Edison has discussed the possibility of condensate causing turbine over-i speed with the vendor (Terry Turbine). Tests apparently had been conducted several years ago. In one test case, after steam was flowing and the turbine 5-6 i

I was running, the injection of 50-600 lbm (pounds mass) of water caused the tur-bine to bog down. In another test, when water was injected into the steam during the starting of the turbine, the water went through the governor, caus-ing it to open further and allowing more oater to pass. At the turbine nozzles, the water flashed and caused the speed to increase significantly. The tests were terminated prior to reaching the overspeed trip setpoint. Toledo Edison's discussions add credibility to the primary hypothesis. Toledo Edison stated that a search of the manufacturer's technical manual and the vendor service letters yielded no suggestion of this overspeed' potential or a minimum steam quality (dryness) specification.

Although the troubleshooting is not complete because of the need for hot func-tional tests (plant Mode 3), Toledo Edison believes that the root cause has been determined to be the primary hypothesis. The Mode 3 tests are expected j

to be confirmatory. As of this time, Toledo Edison has not presented an engi-

! neering report to support this root cause determination.

In an effort to improve AFW system reliability, Toledo Edison modified the actuation logics such that, for all AFW actuations, each AFW turbine would draw steam through redundant parallel valves (i.e., from both OTSGs via valves MS-106, 1 MS-106A for AFPT #1 and valves MS-107, MS-107A for AFPT #2, as shown on Fig-

ure 4.2, rather than only from its associated OTSG). This modification, how-1 ever, resulted in the occurrences of some water hammer events. Toledo Edison 1

then re-modified the actuation logics so that they would be functionally simi-lar to the previous configuration. The water hammer events are consistent with the hypothesis that hot water collects in the lines to the pump turbines.

However, Toledo Edison did not address potential adverse affects of operating on the steam crossover lines alone.

The review of the AFW design indicates that the AFW steam crossover lines (i.e. ,

those associated with the opposite OTSG for each AFW turbine and steam ad.nission valves MS-106A and MS-107A) have long horizontal runs. Toledo Edison believes that these conditions are likely to have resulted in several hundred pounds of saturated hot water.

When Toledo Edison initially explained the June 9,1985 event to the vendor of the AFW turbines, the vendor indicated that the overspeed trips could have been caused by hot (saturated) water entering the AFW governors / turbines, flashing to steam, and causing the turbine to overspeed. Toledo Edison has stated that overspeed trips at four different plants appear to have been caused by this phenomenon. While it appears that the vendor had been aware of this overspeed susceptibility, it is not clear whether the vendor had advised Toledo Edison or any other nuclear users of the turbines.

For a postulated break of one main steamline, the steam crossover valves (MS-106A and MS-107A) and lines are provided in the design so that the AFP turbines can draw steam from the OTSG not affected by the break. Toledo Edison stated that i

the AFW system had never been tested in this configuration, i.e., AFPT #1 draw-ing steam only from OTSG #2 via MS-106A, or AFPT #2 drawing steam only from )

OTSG #1 via MS-107A. Testing of the AFW system in this accident configuration l 4 would be expected to have revealed the steam condensation problems and the overspeed tripping prior to an actual operating event.

i I

i 5-7

._- __ ___ _ _ . . ~ ~ . _ . . , _ _ _ . _

i 5.2.5 AFW Containment Isolation Valves During the June 9, 1985 event, AFW containment isolation valves AF-608 and AF-599 could not be reopened from the control room, either automatically or manually, following their inadvertent closure. This caused the complete loss of the AFW safety function by blocking the flow of both AFW trains to either OTSG.

Toledo Edison's troubleshooting plan for this item is " Auxiliary Feedwater Sys-tem Valve Problem Analysis (AF-599 and AF-608)" Action Plan No. 12, dated June 14, 1985. Toledo Edison's hypotheses for this problem included: improperly adjusted torque switch bypass contacts, improper torque switch settings, improper torque switch setting calculation, improper torque switch installation, wrong or improperly adjusted spring packs, and failure of motor brakes to release.

On June 21, 1985, during a meeting with NRC, Toledo Edison reported the results of the root cause determination. The number of handwheel turns to the point where the bypass contacts for the torque switches are opened were found to be improper for both valves. For one valve (AF-608), the bypass contact was set at nearly the value in the procedure (8 turns vs. 9 turns), but the specified setting is believed to be too early in the opening cycle. Premature opening of the bypass contacts can result in torque switch actuation which trips the valve motor (i.e., the load on the motor is greater than the torque switch setting).

This load may be higher because of the high differential pressure (dp) across the valve at the time the torque switch bypass contact opened. For the second valve (AF-599), the bypass contact was grossly misadjusted from the value speci-fled in the procedure. Toledo Edison stated that the procedure was " bulky and difficult" and, therefore, such an error should not be unexpected.

Toledo Edison stated that the bypass switch settings had been increased in the past few years based upon a Torrey Pines study. Toledo Edison's current con-sultant, retained for the troubleshooting activities (MOVATS, Inc.), suggested that a higher bypass switch setting (at least 10%) is necessary to overcome the high differential pressure across the valve, and even a higher value should be considered if more margin is needed.

The Team expressed concern that this root cause aetermination was primarily based upon analysis and did not involve tests that reproduced the failure.

Because of the implications on other motor-operated valves at the Davis-Besse plant and other plants, the Teea suggested also that Toledo Edison confer with the valve designer / manufacturers (Limitorque and Velan) to determine if they i concur with this root cause.

I j

Toledo Edison issued Revision 2 of the troubleshooting plan, dated June 26, 1985, to provide testing with a differential pressure across the valve.

During a meeting on July 12, 1985, Toledo Edison discussed the status of the troubleshooting of this item. They had discussed the preliminary root cause with both the valve operator manufacturer (Limitorque) and the valve manufac-turer (Velan). Neither disagreed with the possibility that the opening torque switch bypass contacts had been specified at too small a value, i

5-8

Toledo Edison has now conducted tests with about 1000 psid across the valves (AFW pump side high) to attempt to reproduce the failure. The AF-608 valve failed one of three tests at 1050 psid. The AF-599 valve passed a test at 350 psid, just barely passed at 750 psid, and failed to open twice successively l at 1050 psid.

Discussions revealed that Torrey Pines had specified settings from the start of valve stem motion, whereas M0 VATS specifies settings from valve disc movement.

Due to the gap between the stem and the disc, the difference between these two motions could be as great as 10% of total valve travel.

Toledo Edison has also completed calculations and confirmed that the valve operator has sufficient force to open the valve against high differential pres-sures of 1050 psid (2.9 hp vs. 4.0 hp available).

l Toledo Edison also found that for AF-599 the spring pack locknut was installed backwards and that no setscrew was installed. For AF-608, the spring pack was lightly pre-loaded. Toledo Edison believes that these discrepancies were not significant with regard to the June 9 failures.

Toledo Edison believes that the root cause of the AF-608 and AF-599 valve malfunctions has wide applicability at the Davis-Besse Plant and could affect other plants also. They are currently considering specifying the setpoint for i the open torque switch bypass contact at 90% of the full-open position for all l valves at the plant.

l As of this time, Toledo Edison has not presented an engineering report to I

support the final result of the root cause determination.

During the discussion Toledo Edison stated that the safety function for the i valve had been incorrectly specified as only to c!ose, not to open or reopen.

For a postulated main steamline break upstream of the MSIV, both OTSGs would initially depressurize. This is shown in Figure 15.4.4-3 of the USAR. Low OTSG pressure would actuate the SFRCS and cause both MSIVs and both AFW con-i tainment isolation valves (AF-608 for OTSG #1 and AF-599 for OTSG #2) to close.

l Because the MSIV would close, the " intact" OTSG would repressurize. The repres-surization should reset the SFRCS actuation and cause the automatic re-opening l of the associated AFW containment isolation valve to allow AFW flow so that the i OTSG could be used as a heat sink. Thus, valves AF-608 and AF-599 and the asso-l cisted SFRCS have two safety functions: to close to isolate the affected OTSG cnd to open to allow use of the unaffected OTSG. Review of the auxiliary feed-

[ water system and the SFRCS designs revealed, and discussions with Toledo Edison L

confirmed, that neither the SFRCS nor the auxiliary feedwater system meet the single failure criterion with respect to opening an AFW containment isolation valve to feed an intact steam generator.

The valves had never been tested with a differential pressure across the valve which is likely to be the condition for certain postulated accidents. With no differential pressure present, the tests may not reveal an improper setting of the bypass contacts around the torque switches, an improper torque switch set-ting, or an improperly sized motor. It should be noted that testing of valves with differential pressure is not generally done within the industry.

l l

5-9 l

During a previous event on March 2, 1984 at Davis-Besse, the AF-599 valve auto-matically closed and later could not be re-opened with the controls provided for the valve in the control room. The valve had to be handcranked open locally during the recovery phase of that event (as was also the case on June 9,1985).

In March 1984, Toledo Edison's corporate engineering staff decided that, although the valve inspection found no causes, and no attempt had been made to reproduce the failure by a test, the valve must have driven itself too far closed and could not re-open. Therefore, the specified closing torque switch setting for the valve must be improper. On this basis, the specified closing torque switch was changed to cut off the motor at a smaller closing torque value. Subsequent testing of the valve (without a differential pressure across the valve) did not show any problems but likewise did not demonstrate that the problem had been corrected. No further action was pursued, and the valve was returned to service and declared to be " operable."

5.2.6 Main Steam Supply Valve to AFPT No. 1 Valve MS-106 is the main steam supply valve from OTSG #1 to auxiliary feed pump turbine (AFPT) #1. It was in its normally closed position just prior to the June 9, 1985 event. Six minutes into the event (at 01:41:04) Actuation Chan-3 nel #1 of the SFRCS actuated on low level and initiated the start of AFPT #1 on steam from OTSG #1. That is, MS-106 started to open. Four seconds later, the SFRCS was actuated manually on low pressure. Such an actuation (low pressure on OTSG #1) has priority and would signal MS-106 to re-close. The design of i the valve control circuitry is such that the valve should have completed its I opening stroke (25 seconds) and then returned to the closed position (another 1 25 seconds). Review of the plant data after the event revealed that MS-106 was fully closed at 19 seconds after it started to open. This value suggests that l the valve stopped and/or switched direction in mid-stroke, contrary to the design intent.

Toledo Edison's troubleshooting plan for this item was " Auxiliary Feed ' ump Turbine Main Steam Inlet Isolation Valve (MS-106) Problem Analysis," Act.3n Plan No. 27, dated June 25, 1985. Toledo Edison developed six hypotheses which included: an open motor field circuit which could have caused the motor oper-ator to overspeed; and five different open-circuit malfunctions of seal-in circuits, pressure switches, control relays, torque switches, or limit switches which could have caused the valve to reverse direction at some intermediate position. l Toledo Edison issued Revision 1 to the troubleshooting plan, dated July 3, 1985, to reflect the possibility of wiring errors associated with the SFRCS modifica-tions, improper AFP low suction pressure switch operation, improper steam supply low pressure relays, and torque switch or bypass contact misadjustments.

During a meeting on July 12, 1985, Toledo Edison discussed the status of the troubleshooting of this item. The actual troubleshooting is complete except for testing under reactor system operating conditions. The troubleshooting found:

l

1. A loose wiring connection.

2. A wiring discrepancy in the MS-106 controls, i

5-10 l

3. A wiring discrepancy in the motor starter.

4. An unnecessary gap between the spring pack locknut and the outer thrust washer.

5. A cocked packing gland flange.

6. The opening torque switch bypass contact set to open too early.

7. Inoperable MS-106A position alarms in the control room.

At this time, the troubleshooting has not produced a conclusive root cause for the June 9, 1985 malfunction, and thus, a Toledo Edison engineering report to support the results of the root cause determination is not available.

5.2.7 Source Range Nuclear Instrumentation One of the two redundant source range nuclear instrumentation channels (NIs)

(Channel 2, NI-1) was inoperable prior to and throughout the June 9,1985, event.

About 16 minutes into the event, the neutron level had decreased to the top of the source range. Upon energization, the second source range NI (Channel 1, NI-2) failed; it went off-scale low (i.e., less than 1 X 10 1 counts per second).

Toledo Edison's troubleshooting plans for this item were " Action Plan Report for NI-1 Source Range Channel," Plan No. 15A-1, 15A-2, dated June 17, 1985 and "NI-2 Count Rate Level Indication Failure Analysis," Plan No. ISB, dated June 17, 1985. Toledo Edison's hypotheses for the failure of NI-1 are: an intermittent problem within the Count Rate Amplifier Module (CRAM), extraneous counts being introduced from various external sources, and resonant cable lengths. The hy-pothesis for NI-2 is that the detector high voltage or the input signal to the CRAM is being interrupted by a bad relay contact, loose wiring, and/or loose components.

i At a meeting on July 11, 1985, Toledo Edison discussed the status of the trouble-shooting of this item. For NI-1 (inoperable prior to the event), Toledo Edison has observed periods of elevated count rates that seemed to come and go, and i

Toledo Edison has obtained some baseline data. Technical assistance on noise problems with pulse-type instrumentation is being obtained from Ohio State Uni-

, versity. No root cause has been identified and troubleshooting is continuing.

For NI-2 (failed low during event), Toledo Edison's efforts have not reproduced the failure. No root cause has been identified and troubleshooting is continu-

ing. Toledo Edison is considering revising the troubleshooting plan.

As of this time, Toledo Edison has not presented an engineering report to support the results of the root cause determination.

The problems with the source range NIs which occurred prior to the event are ,

discussed in Section 5.1.2.

l 5.2.8 Pilot Operated Relief Valve (PORV)

During the June 9, 1985, event, the pilot operated relief valve (PORV) operated automatically three times. In the first operation, the valve opened at about I

l 5-11 l

the proper pressure (the setpoint is 2425 psig), was open for about 3 seconds, and re-closed at about the proper pressure (2375 psig ). The second operation was similar except that the closing pressure was slightly lower. In the third operation of the PORV, it did not re-close. Review of the data for the quench tank pressure and level indicates that the flow was not terminated until the block valve was closed. The PORV block valve was closed by the operator when system pressure had fallen to about 2140 psig, 24 seconds after the PORV had opened. The operator re-opened the block valve a little over 2 minutes later.

At this time, it appeared that the PORV was closed.

Toledo Edison's troubleshooting plan for this item was " Review of the Operation of the PORV," Action Plan No. 10, dated June 22, 1985. It appears that Toledo l Edison's primary hypothesis is that differential thermal expansion of the valve disc and the body caused the PORV to become stuck.

^

Other hypotheses were: valve mechanical malfunction, solenoid linkage broken or corrosion buildup, and stick-ing caused by foreign material.

Toledo Edison approved Revision 1 to the troubleshooting plan on July'3, 1985.

A major change was the addition of a summary of the operating experiences with PORVs sticking open at six other PWRs (i.e., pressurized water reactors, the same type reactor as Davis-Besse) due to a wide variety of causes. These PORVs were manufactured by vendors different from that for Davis-Besse.

During a meeting with NRC on July 11, 1985, Toledo Edison discussed the status

of the troubleshooting of this item. The plan has virtually been completed, i

including disassembly and inspection of the valve, without identification of j the root cause for failure.

t When the Team inquired as to what the manufacturer advised, Toledo Edison i replied that Crosby had recommended additional tests in two areas: (1) the PORV control circuits, and (2) functional PORV tests at 600 psi and full RCS pressure. Toledo Edison stated that the PORV manufacturer is not surprised that no cause has been found. During the Electric Power Research Institute's i (EPRI) PORV valve testing, tests were conducted under a variety of conditions, but each test consisted of only a single operation of the PORV. Toledo Edison states that during the EPRI testing there were one or more failures of the PORV to close and, although investigation (s) were conducted, no cause was ever deter-mined. During the June 9, 1985 event, the PORV did not fail until the third operation. During a 1977 event at Davis-Besse, the PORV operated nine times and then failed. These points suggest that the EPRI test results may not be representative of PORV operation and that the results should be used with caution.

Further, the apparent situation that the causes of some PORV failures may not ever be known raises again the question of the need for better protection against PORV failures, i.e., automatic block valve closure.

1 Toledo Edison is currently reviewing Revision 2 to the PORV troubleshooting plan to provide for checking the controls and actual PORV lif ts. Toledo Edison is also considering that if a failure cause for this PORV cannot be identified, a new PORV may be procured that would be tested paior to installation.

As of this time, Toledo Edison has not presented an engineering report to l

support the results of the root cause determination.

i I

5-12 1 _ _ _. - _ _ - _ _ _ _ .

l The review of the PORV maintenance and operating history reveals that the mechanical operation of the valve had not been tested and that the valve had not otherwise been operated for over 2 years, 9 months prior to the June 9, 1985 event.

5.2.9 Startup Feed Control Valve for OTSG No. 2 During full power operation, the startup (S/U) feed control valves (SP-78 and SP-7A) are fully open. Upon SFRCS actuation, the S/U feed control valve to each OTSG is automatically closed. This action occurred 5\ minutes after the plant trip on June 9.

In anticipation of returning the startup feed pump to service, the operator attempted to override / reset the SFRCS so that the S/U feed control valves could i be re-opened. However, the reset light for SP-7A did not come on, indicating j that the operator had not regained control of the valve. Based upon the appa-rent lack of reset for SP-7A, the control room operators believed at the time that flow from the startup feed pump went through SP-78 to OTSG #1 only and not through SP-7A to OTSG #2.

Toledo Edison's troubleshooting plan for this item was "Startup Feed Valve SP-7A

Problem Analysis," Action Plan No. 18, dated June 22, 1985. Their hypothesis is that the valve actually functioned properly, but the indication of SFRCS reset for this valve was not available due to a burned out indicator bulb. The alter-native hypothesis, i.e., that the valve did not respond correctly, would be addressed by the collection of plant data to show if there was flow i

through the valve.

During a meeting with NRC on July 12, 1985, Toledo Edison discussed the status of the troubleshooting of this item. The actual troubleshooting has been com-pleted, and Toledo Edison believes that a final conclusion regarding this valve has been reached. Simulated SFRCS output signals show that the S/U feed control valve SP-7A closed on demand, that the override / reset features functioned prop-erly, and that the valve re-opened when operated from the control room. Plant data shows that during the June 9, 1985 event, SP-7A actually opened to about 12% and the measured flow was about 1.5% of full S/U flow.

Therefore, Toledo Edison has concluded that the valve performed properly during the event, and that only the reset indicator failed due to a burned out bulb.

However, as of this time, Toledo Edison has not presented an engineering report to support the results of this root cause determination.

5.2.10 Recovery and Control of Both AFW Turbines During the June 9, 1985, event, equipment operators were dispatched to restore both AFW trains. The equipment operators had difficulty resetting the turbine

, trip-throttle valves which had tripped due to overspeed. Further, there was difficulty restoring proper speed control. The control room operator attempted i to regain control repeatedly. The efforts were not successful. During the I

event, the AFW #1 turbine increased to about 2200 rpm, which is well below full speed and was insufficient to pump feedwater into the pressurized 0TSG. The equipment operators continued to operate the trip-throttle to control speed, arc encountered difficulties. The linkage for the trip-throttle valve for 5-13 l

)

AFPT #1 apparently disengaged twice, causing the valve to slam shut. Control I of both AFW turbines was performed locally throughout the event, l Toledo Edison's troubleshooting plans for this item were "AFPT Overspeed Trip Throttle Valve Problem," Action Plan 1D, dated June 20, 1985 and "AFPT Manual /

Auto-Essential Control Problem," Action Plan 1C, dated June 24, 1985. Toledo Edison's hypotheses regarding the difficulty in re-latching the overspeed trip throttle included: (1) the tappet of the turbine trip mechanism malfunctioned, (2) the trip hook relatching spring was inadequate, and (3) there were mechan-l ical difficulties related to the trip hook pivot point or to the linkage mechanism. Toledo Edison's hypotheses regarding difficulty opening the trip throttle valve included the possibility that the valve may not be correctly balanced or adjusted for opening against the steam generator pressure. Toledo Edison's hypothesis regarding the difficulty that the control room operator experienced in regaining AFPT control was directly attributable to the inabil-ity to re-latch the trip-throttle valve linkage properly and the difficulty and delay in opening the trip-throttle valve.

- During a meeting with NRC on July 11, 1985, Toledo Edison discussed the status of the troubleshooting under plan No. 10, which covers the re-latching of the turbine overspeed trip mechanism and difficulties in opening the turbine trip-throttle valve. Figure 5.1, from Toledo Edison's action plan, illustrates the

pertinent aspects related to this problem. Except for some tests to be done under full steam pressure (plant mode 3), the actual troubleshooting has been

! completed. Toledo Edison believes the root cause for this item has been deter-mined. All mechanisms have been checked and found to be properly adjusted, with all mechanism pivot points and components free to operate. The equipment operators who were involved during the June 9, 1985, event have been involved in every step of this troubleshooting activity. Toledo Edison stated that these equipment operators now believe that there was no mechanical problem with the mechanism, but rather that they did not know how to perform the necessary actions, j It is physically possible to pull the connecting arm sufficiently far (to the left in Figure 5.1) to be able to barely re-engage the trip hook at the trip-l throttle valve but not reset the overspeed trip back at the other end, where the overspeed tappet and manual trip lever are located. This end of the con-

,' necting arm is behind the governor and is not easy to see.

The equipment operators had been trained and certified in all the specified areas related to the AFW systems. The overspeed trips are tested monthly and have to be reset each time; however, the test is conducted with low pressure auxiliary steam (235 psig), and the procedure emphasizes getting the trip hook and latchup lever at the trip-throttle valve together. Since the Technical Specification surveillance test is performed by one out of six operating shif ts on a rotating basis, it is possible that these equipment operators had not had sufficient hands-on experience, even at the lower auxiliary steam pressure.

During the event, when a more experienced individual arrived later at AFPT #1, the trip was immediately relatched properly, the trip-throttle re-opened, and the AFP made operable. Had this operation been performed originally by a more experienced operator, the AFP would have been available when the isolation valve (AF-608) was re-opened, i.e., about 5 minutes before flow was actually acquired.

For the #2 AFPT, the equipment operator reset the overspeed trip mechanism 5-14

properly and in a timely manner. However, some resistance was experienced as the trip-throttle valve was being opened. The equipment operator seemed not to know what to do at that point. After a more experienced operator arrived some minutes later and used a valve wrench to open the trip-throttle valve fully, the AFP #2 was operable.

In summary, the delay in recovering both AFW trains following the overspeed trips is attributed to less-than-adequate hands-on training under full steam pressure conditions.

As of this time, Toledo Edison has not presented an engineering report to support the results of the root cause determination.

Problems had been experienced previously with resetting the trip-thruttle valves properly.

l 5.2.11 AFW No. 1 Suction Transfer l

During the June 9, 1985, event, AFW train #1 provided significant flow (>400 gpm) for about a 3\-minute period between about 01:55 and 01:59 a.m. At about the end of this period, the operator reported that the low suction pressure alarm had come on and the suction source was being automatically transferred from the condensate storage tank to the service water system. The plant traces also show a sharp speed reduction (nearly all the way to zero rpm) at about the same time, suggesting a spurious closure of the trip-throttle valve. Just after this time (01:59), the overspeed trip was properly reset, and the trip-throttle valve re-opened. The control room operator manually returned the suction to the condensate storage tank.

Prior to this transfer (i.e. , at 01:58), the alarm data shows an actual low suction condition had developed, had lasted for 34 seconds, and cleared itself.

Toledo Edison's troubleshooting plan for this item was " Inadvertent Auxiliary Feedwater Pump #1 Suction Supply Transfer from Condensate Storage Tank to Service Water Supply," Action Plan No. 26, dated June 26, 1985. Toledo Edison's primary hypotheses include: suction pressure switches associated with AFW #1 setpoints were out of specification, suction pressure switches actuated due to vibration.

l the low suction pressure alarm was out of specification, the common AFW strainer was clogged, the AFW #2 low pressure switches were out of specification, or an actual low suction pressure situation was induced. Other hypotheses include:

momentary loss of power to suction valves AF-786 and SW-1382, suction strainer I S-201 was clogged, and manual suction transfer to the service water system.

l l

During a meeting with NRC on July 11, 1985, Toledo Edison discussed the status of the troubleshooting on this item. Some of the troubleshooting steps have been completed but none of the findings establish the cause of the suction transfer. Troubleshooting is continuing.

As of this time, Toledo Edison has not presented an engineering report to support the results of the root cause determination.

5-15

5.2.12 Turbine Turning Gear After the plant had stabilized from the June 9, 1985, event, it was noticed that the main turbine had not gone onto its turning gear. Since the same problem had been experienced recently and blown fuses had been found then, the shift super-visor dispatched a worker to replace these same fuses. As the Team understands it, the fuse replacement alleviated the immediate problem.

The Team agreed with Toledo Edison that it was not necessary to have this item on the quarantine list or to develop a special troubleshooting plan.

5.2.13 Control Room HVAC System Toledo Edison stated that during the event the control room heating, ventilation, and air conditioning (HVAC) system tripped spuriously and went into its emer-gency mode of operation. This type of actuation had occurred on previous occa-sions and did not appear to be unique to this event.

The Team agreed that it was not necessary to have this item on the quarantine list or to develop a special troubleshooting plan.

5.2.14 Turbine Bypass Valve About 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> after the reactor trip on June 9, 1985, the condenser vacuum was re-established and MSIVs were re-opened. At this time one of the turbine bypass valves was damaged severely. The control room operators heard a loud cracking sound like that heard typically from a water hammer. Subsequent inspection showed that both the valve yoke and housing were cracked. Additionally, the valve stem thread dimension was questionable and the pin connector was found in contact with the sleeve assembly.

Toledo Edison's troubleshooting plan for this item was " Turbine Bypass Valve 2-2

]

4 (SP13A2) Problem Analysis," Action Plan Nos. 9a and 9b, dated June 18, 1985. Toledo i Edison's hypothesis is that the damage was most likely caused by a combination of a water hammer and mis-assembly of certain valve internals. An alternate hypothesis is that the valve positioner malfunctioned.

During a meeting on July 12, 1985, Toledo Edison discusseo the status of the troubleshooting of this item. It has been determined that, while all the tur-bine bypass valves had been rebuilt in 1980 under the supervision of the manu-facturer, only this valve was rebuilt again in 1982 (and without the benefit of supervision by the manufacturer). Further, this is the only bypass valve found to have a modified valve stem with a cotter pin. The inspection results identifieJ 11 discrepancies:

1. Clogged strainer (ST3).

2. Deformed strainer (ST3A) and failed steam trap.

3. Short piston actuator travel length, 1 9/32 inches vs. the design value of 1 9/16 inches.

t 5-16

._ - __ . ._ . - ~ _ = - . . _ _ - _ - - -. - --.-__-.- -_ - - _ _ - - . - . . - ._

1 l

4. Discoloration of the yoke at the break location.

5. Broken positioner linkage.

6. Valve stem previously scored seriously in a vise.

7. Valve Activator stem extension piece bent.

8. Main plug separated from stem, found in bottom of valve body.

9. Belleville springs and spacers found jammed together on pilot plug.

10. Cotter pin and washer for main plug missing.

l

11. Three inches of water in bottom of valve.

Toledo Edison re-confirmed that there is no program at Davis-Besse for periodic maintenance or testing of the turbine bypass valves. The valve parts have been sent to Fisher for destructive testing and evaluation of possible failure causes.

Investigation into the temperature (about 140 F) difference between the two i

steam lines downstream of the MSIVs is continuing to determine how much steam l condensed. The associated steam traps and drains are also being investigated ,

l for proper operation.

! Toledo Edison maintains that the root cause of this failure is a combination of a water hammer and valve misassembly.

1 As of this date, Toledo Edison has not presented an engineering report to support the results of the root cause determinations.

1 i

I e

f I

i 4

I f

' 5-17

! i i

_. --.- -_ - ~_.-- - __ - - - - _ . _ . . . - ~ _ _ - _ _ - - . . _ _ . - . . . - . _ - - - - - . . - . . . .

.l Table 5.1 Summary of Equipment Troubleshooting Results NATURE OF ITEM FAILURE PROBABLE ROOT CAUSE CONNENTS

1. Main Feedwater Turbine Overspeed Control System Electronic Circuit Pre-existing Control System 1 Card Failure Problems Have Not Been Resolved

2. Closure of MSIVs Spurious Actuation Not Identified Troubleshooting Activities of SFRCS Have Not Yet Begun

3. Steam Safeties, Atmos. Abnormal Pressure Not Identified Vents Control

4. Aux. Feedwater Turbines Overspeed Condensate Flow to Turbines From Testing with Plant Hot Needed

, Steam Supply Lines During Turbine to Verify Cause Start i

5. AFW Containment Isolation Would Not Re-Open Improper Settings for Torque Valves Switch Bypass Contacts

6. Steam Supply Valve to Short Cycle Not Identified Failure Could Not Be Reproduced.

AFPT #1 Improper Torque Switch Bypass Contacts Could Be Problem i

7. Source Range MI Failed, Low Not Identified Failure of One of Two Channels Could Not Be Reproduced i

8. PORY Did Not Close Disassembly of Valve and Testing Cause May Never Be Identified of Control System Failed to Reveal Cause

9. .. S/U Feedwater Control Did Not " Reset" Indication Problem Only -

Valve Indicator Lamp j

I 10. Recovery of AFP Turbine Trip-Throttle Valve Lack of Operator Training Not a Hardware Problem i l

Operational Diffi-culties t

Table 5.1 Summary of Equipment Troubleshooting Results (Continued)

NATURE OF ITEM FAILURE PROBABLE ROOT CAUSE COMMENTS

11. AFP #1 Suction Transfer Transfer to Service Not Identified Water

12. Turbine Turning Gear Did not Engage Troubleshooting Not Reviewed by Team

13. Control Room HVAC Spurious Transfer Troubleshooting Not Reviewed to Emergency Mode by Team

16. Turbine Bypass Valve Fractured Water Hammer, Valve Mis-Assembly Cause of Water Hammer Not Yet Known l

l i

M

. TRIP HOOK REPOSITION SPRING m '

a O

Trip Hook

?({  ?

Latch-Up Lever p-- Manual ..

e Y; Valve Trip '

Trip Lever -

t:3 '

Mechanism g i$ dji LINK SPRING

'f I

af. Tappet p, e.

Turbine Trip Mechanism Steam in Steam

> > To Turbine Figure 5.1 AFPT Trip-Throttle Valve 5-20 ,

1 l

6.0 HUMAN FACTORS CONSIDERATIONS This section discusses the human factors aspects of the June 9 event. The following discussion of personnel performance and man-machine interfaces are related to how these human factors considerations affected the event. When appropriate, potentially adverse effects are described that could have affected the event. The information was obtained through interviews with plant operators and their management, and by a walk-through of the control room and other parts of the plant where relevant equipment is located.

6.1 Operator Performance Based on the details of operator actions described in Section 3, it should be evident that both the licensed (control room) and non-licensed (equipment) operators performed well as a coordinated group to mitigate a complex event involving multiple malfunctions. The operators, particularly the assistant shift supervisor and senior equipment operator, performed timely corrective actions from outside the control room and prevented a potentially more serious event. However, noteworthy operator errors also occurred. Two were made by licensed operators and two were made by equipment operators.

6.1.1 Licensed Operators The actions taken by the licensed operators during the event indicated that they were generally cognizant of plant conditions and responded to them in a deliberate manner. For example, the first operator error involving manual initiation of the steam and feedwater rupture control system (SFRCS) occurred only after the reactor operator had requested and received permission from the shift supervisor to initiate the SFRCS system. The second operator error oc-curred when the shif t supervisor did not initiate MU/HPI cooling at the point required by the emergency procedure. The shift supervisor believed that initi-ating MU/HPI cooling was not needed nor required because restoration of auxiliary feedwater flow was imminent and he did not recognize that the criteria in the procedure for initiating MU/HPI cooling existed.

The operator's attempt to initiate SFRCS manually was anticipatory. He tried to initiate the SFRCS before it automatically initiated on low steam generator water level. Instead he inadvertently pushed the two buttons labeled low steam pressure. A Davis-Besse operator described such anticipatory actions as a kind of preventive medicine. That is, if the safety system is going to trip, manually tripping it earlier would provide its benefit sooner. For this event, manual initiation of the SFRCS would initiate the AFWS before the steam generator low water level set point was reached. Hence, less inventory would be lost and the AFWS would restore the steam generator inventory to normal levels faster.

Operators at some other utilities are also trained to manually initiate safety svstems when automatic actuation is imminent. While this practice is normally conservative and proper, this event indicates that it introduces an opportunity for incorrect operator actions either because of a lack of knowledge of plant conditions or through mistakes in implementation. In this case, the layout of 6-1

the control panel contributed to the event, but had the equipment worked pro-perly, this operator error would not have had a major effect on the seriousness I or consequences of the event.

1 As noted above, the layout of the SFRCS, buttons contributed to the operator I error. The SFRCS manual switches, shown in Figure 3.6, are arranged in two columns of five switches. Each column represents one actuation channel of the SFRCS and each button represents a different parameter; for example, low steam pressure or low water level. During recovery activities on June 9, the operator should have pushed the fourth button from the top of each column to initiate SFRCS on low water level in both steam generators rather than the two top buttons (which initiates the SFRCS on low steam pressure).

Further, it should be noted that to initiate SFRCS for a depressurized steam generator, the operator currently has to operate pushbuttons on a diagonal rather than a horizontal line. For example, to manually initiate SFRCS on low pressure in one steam generator using the push buttons in Figure 3.6, the operator must push the top button in one column and the second button from the top of the other column, depending on which steam ger.erator had the low pres-sure. Toledo Edison was considering a revised layout in which the pushbuttons for low pressure actuatian for either steam generator would be in a horizontal line. Had this high priority deficiency been corrected in this manner prior to the June 9 event, the operator's error would have isolated only one steam gen-erator and the other steam generator would have been available as a heat sink.

In addition to the diagonally aligned buttons, other human factors considera-tions of concern are the location of the SFRCS panel and the labels identifying the buttons. The SFRCS panel is located on the back panel--behind and below the main control console shown in Figure 3.4. It is not easily visible from the operator's normal work station and the operators must walk around the main control console to manipulate the switches. The pushbuttons are small and their labels do not clearly describe their function. For examples, the label "SG 1-1 LOW STM PRESS" means that pushing the button initiates SFRCS channel 1, which isolates the feedwater to No. 1 steam generator because it has low pres-sure (e.g., steamline break or leak). (See Figure 3.6.) Similarly, the button labelled "SG 1-1 LOW WTR LVL" means to initiate SFRCS channel 1 on low level in No. 1 steam generator and start the No. 1 AFW pump to feed the steam genera-tor because it has low water {evel (e.g. , loss of feedwater). The labelling of controls for manually initiating one of the most important systems at Davis-Besse should have been unambiguous.

In January 1985, Toledo Edison advised the HRC that the SFRCS manual initiation pushbuttons had been. identified as one of the principal items needing human engineering improvements.* A design change of these pushbuttons was expected to 1 be completed by the end of the fifth refueling outage, scheduled in late 1986.

Toledo Edison had given a high safety significance to the correction of this deficiency because it was recognized that an operator error could inadvertently block any SFRCS actuation. The operator error postulated in the Toledo Edison analyses had the same result as the operator error that occurred during the June 9 event, i.e., feedwater was not supplied to both steam generators. During

• Letter from R. Crouse, Toledo Edison Company, to J. F. Stolz, NRC, Docket 50-346, January 31, 1985.

6-2

a meeting with the Team, Toledo Edison informed the Team that current control room operator training emphasized the proper technique for manual initiation of SFRCS but did not emphasize the potential consequences of an incorrect initiation.

The operator on duty during the event, however, who pushed the wrong buttons indicated that this was the first time that he had manually actuated the SFRCS or had ever been at the control panel during an SFRCS actuation. Further, he indicated that he had received no specialized classroom or simulator training on correctly initiating the SFRCS. Furthermore, the simulator at the Babcock &

Wilcox facility where the reactor operator received training did not include the SFRCS. This situation indicates a lack of thoroughness of training and provides a further incentive for plant-specific simulators.

The operations superintendent indicated that during the event he directed the shift supervisor by telephone to prepare for MU/HPI cooling within 1 minute if the shift supervisor was unable to establish feedwater flow with an AFW pump.

This instruction was based on the operations superintendent's knowledge that analyses showed that if make-up cooling was established within half an hour, there was a good probability that the core would not become uncovered, and that a serious situation would be avoided, even with only the startup feedwater pump (SUFP) operating.

During the time that the shift supervisor was discussing the initiation of -

MU/HPI cooling with the operations superintendent, the secondary-side operator twice suggested this mode of cooling to the shift supervisor. In addition, emergency procedure 1202.01 specified that MU/HPI cooling be initiated if (and when) both steam generators were " dry" and there was no feedwater. However, as noted previously, the shift supervisor did not initiate MU/HPI cooling because he believed that restoration of auxiliary feedwater flow was imminent. During this period he was on the telephone and, as a result, a delay in a decision regarding this mode of core cooling occurred and time was made available for the equipment operators to restore auxiliary feedwater flow. The decision by the shift supervisor was also influenced by a reluctance to release reactor coolant into the containment because of the cleanup and extended shutdown associated with it. In this regard, when the shift supervisor was asked why he did not initiate MU/HPI cooling, he noted:

Well, it's a pretty drastic step. And I wanted to wait until I was -- well, I didn't want to do it prematurely. I wanted to wait until I was at the point that was required by the procedures to do that.

6.1. 2 Procedural Compliance .

Emergency Procedure 1202.01 is the Davis-Besse version of the-B&W Owners Group's Abnormal Transient Operating Guidelines (ATOG) and had been implemented in January 1985. From interviews conducted by the Team, the operators have confi-dence in this procedure to help them mitigate the consequences of plant events.

For example, when asked how this procedure compared with pre-TMI procedures, the shift supervisor replied, Before we would have to be in maybe two, three, four procedures.

This is the only document that we have to pull out. And it will 6-3

. _ _ _ - __-__ .l

.- . . . . . _. _ - . - - _ _ = -_ -_. -

i lead you step by step through the procedure until you do discover j your fault.

l This procedure was followed during the event. The Team believes it was imple-

mented correctly until the decision point was reached in the section entitled

" Lack of Heat Transfer" regarding the initiation of MU/HPI cooling. With both steam generators dry, i.e., pressure is below 960 psig and decreasing or water level is less than 8 inches and with no feedwater, the procedure indicates i that MU/HPI cooling is to be initiated. However, based on the operator inter-views, it appears that none of the control room operators were fully aware that i; both steam generators were " dry" as defined by the emergency procedure, i

j At Davis-Besse, the available instrumentation did not provide clear information

! to'the operator that the steam generators were dry. For example, steam generator

! pressure is not recorded in the control room for trending purposes. As a j result, when the steam generator pressure reaches 960 psig, the-operator must

remember whether the pressure has been decreasing or whether a sudden depressuri-l zation has occurred. Further, steam generator levels are indicated in the j control room on a scale of 0 to 250 inches on the startup level instrumentation shown in Figure 3.10. Given this scale, the operator cannot read "8 inches" on the instrument accurately, even if the instrument is accurate at the low range of its scale.

1 I

Both the steam generator pressure and water level can be trended using the safety parameter display system (SPDS); however, it was inoperable prior to the June 9 event. The inoperability of the SPDS and the lack of adequate indi-cation of steam generator conditions contributed to the control room operators j not knowing that the steam generators were dry, which resulted in their failure

to follow the emergency procedure.

i Further, because the equipment operators restored auxiliary feedwater flow to

! the steam generators in approximately 12 minutes, the shift supervisor's delay i in implementing MU/HPI cooling did not lead to adverse consequences. However, i the time to restore the feedwater is a critical factor regarding the serious-

! ness of this event, and further delays in restoring feedwater could have had potentially serious safety consequences, as discussed in Section 7.

l This event points out a natural reluctance on the part of-operators to initiate j any action which could result in long plant shutdowns or other major economic 4 consequences. That is, the operator can be expected to explore and discuss all l available options and to take the time to assure the need before initiating a l " drastic" action. This consideration should be recognized and reflected through I precise and clear instructions in any procedure which addresses the need for j such " actions."

6.1.3 Operator /STA Interaction

)

Neither the shift supervisor nor any of the other licensed operators requested the assistance of the shift technical advisor (STA) during the event. One reason for not doing so is the fact that the STA was not in the control room j when the event occurred. (Note: He is allowed 10 minutes to reach the control j room after being called.) Moreover, the event occurred so rapidly that it i was essentially over when he did arrive. In summery having the STA available i

6-4

._. = - __ -.- . . - -

was a post-THI improvement to provide the shift supervisor with additional tech- l nical expertise, but his potential assistance and guidance were not available i nor required during this event. f I

6.1.4 Emergency Notification After the plant was stabilized, the shift supervisor's attention turned toward the actions specified in the emergency plan, such as notifying the NRC, and the local sheriff. He requested that the STA perform the notifications, which was L

the only responsibility that the STA was assigned during the event. The transcribed telephone discussions with the NRC operations officer, indicate that in the initial call, the STA did not provide an adequate description of the event because of lack of sufficient knowledge. Subsequently, additional calls were made. During the third telephone call, at 2:26 a.m., the STA l informed the NRC that an Unusual Event had been declared.

The shift supervisor, who is also the emergency duty officer (E00) on this shift, declared an Unusual Event. Although he recognized that the emergency plan identified the total loss of feedwater event as a Site Area Emergency, the plant was no longer in this emergency action level, and he concluded that it was not an apprcpriate emergency class. He declared an Unusual Event primarily to assure that sufficient technical and maintenance support personnel would come to the site for event analyses and to ensure that the plant remained

! stable. The shift supervisor indicated there was some confusion as to the i correct classification or if any classification was required because the

! emergency plan was silent on how to determine the emergency action level if it changed during the event.

i At Davis-Besse, the emergency plan is initially implemented by the shift l supervisor, who also has primary responsibility for ensuring that the plant is maintained in a safe condition. Thus, because of the competing priorities of directing attention to necessary recovery actions to obtain a safe and stable plant or of reviewing the emergency plan and initiation of its actions, there could be a substantial delay in implementing the emergency plan. This delay, in turn, may affect the timely identification of the proper emergency action level and appropriate notifications. If the 'une 9 event had been more complex and continued longer, it is likely that the emergency classification and noti-

' fication would have been substantially delayed and would have lacked accurate l

details because knowledgeable personnel during this shift were involved with activities to obtain a safe and stable plant condition.

6.1. 5 Equipment Operators l

The control room operators dispatched two equipment operators to reset the overspeed trip for each AFW pump, and accordingly, to restore this equipment to First, service. The recovery from an overspeed trip is a two-step process.

the overspeed trip must be reset, and then the trip throttle valve must be latched. The trip throttle valve may be latched without resetting the overspeed trip; however, overspeed trip protection would then not be available. One

' equipment operator went to the No. 1 AFWP turbine, while the other operator went to the No. 2 AFWP turbine.

One operator had successfully reset the overspeed trip and had latched the No. 2 trip throttle valve. However, he had not turned the handwheel the required The number of revolutions to unseat the valve and admit steam to the turbine.

6-5

fact that there was steam pressure of about 900 psig at the valve made the valve difficult to open. Furthermore, the operator was extremely cautious in attempting to open the valve. In attempting to avoid any potentially damaging or adverse actions, he failed to apply enough force to the handwheel to open the i

valve.

The other operator latched the No.1 trip throttle valve but failed to proper-ly reset the trip. Again, a large differential pressure existed across the trip throttle valve, but the operator partially opened the valve and, as a result, the turbine speed increased to about two-thirds its normal speed. At this speed, the discharge pressure of the pump was not high enough to feed the steam generators. After several unsuccessful attempts to increase the pump speed, he went to assist the other operator at the No. 2 AFW pump, i The assistant shift supervisor, who had come to the AFW pump rooms, told the equipment operators that the No. 2 trip throttle valve was still closed and it had to be opened to admit steam to the No. 2 AFW pump turbine. Meanwhile, after having opened the AFWS isolation valves (See Section 3) a third, more experienced, senior equipment operator entered the AFW pump room and used a valve wrench to open the trip throttle valve. He then proceeded to the No. 1 i

! turbine, and again using the valve wrench, fully opened the trip throttle valve. '

l As noted previously, at this point the No. 1 trip throttle valve had not been reset. The senior equipment operator correctly reset the overspeed trip and latched the trip throttle valve. The No. 1 AFW pump turbine was then returned to service.

The experience of the assistant shift supervisor and the senior equipment operator were instrumental in their returning the AFW system to service.

The failure of the equipment operators to initially reset the overspeed trips and open the trip throttle valves was due to their lack of knowledge and experience. (Note: The Training Coordinator stated that the equipment operators had been trained on how to reset and latch the trip throttle valves.) If the equipment operators had been able to quickly reset and had opened the trip throttle valves, auxiliary feedwater flow would have been available approxi-mately 5 minutes earlier.

6.2 Other Man-Machine Interface Considerations This section discusses man-machine interfaces that were important or that could have been important during the event.

l 6.2.1 PORV Position Indication t

The PORV control station (shown in Figure 3.11) indicated to the operator that the PORV had closed after the third opening, when in fact the PORV had failed to close. The indication showed PORV solenoid plunger position and control signal status. However, these indications are indirect and are not necessarily representative of actual PORV positions. As a result, the operator did not i

know that the PORV had failed to close when he closed the PORV block valve as a precautionary action. Thus, proper operator action was taken and the PORV posi-tion instrumentation was not an important factor in mitigating this event.

One of the post-TMI requirements was installation of acoustic monitors for detecting a failed-open PORV. Although this monitoring system was available in 6-6

the control room, the operator did not use it, even after he reopened the PORV block valve. One important reason for not referring to this instrument is believed to be the fact that the acoustic monitors are located on the post-accident panel which is about 7 feet away from the PORV control station. The 3-inch high and -inch wide meters cannot be read from this distance. Thus, the operator has to leave his control station to read the acoustical instrumen-tation and he did not do this.

6.2.2 Safety Parameter Display System The safety parameter display system (SPDS) at Davis-Besse was also a post-TMI improvement to provide the operator unambiguous information on the status of the plant. The system has the capability of displaying a full range of relevant plant parameters and trends on demand by the operator. Although the SPDS has two channels or trains, both were inoperable prior to the event. At Davis-Besse, the system has a reputation for being so unreliable that the operators do not depend upon it.

There are specific references in the " Lack of Heat Transfer" section of the emergency procedure that require the SPDS or hand plots (which are not practi-cal) to be used during an event. However, the SPDS is not required to be ope-rable by the Davis-Besse technical specifications.

The SPDS was needed during the event to trend RCS pressure and temperature and OTSG pressure and level because the corresponding steam generator instrumenta-tion in the control room was inadequate to properly implement the plant pro-cedures required. The SPDS, as noted previously, was not available nor was it required to be available by NRC requirements.

6.2.3 Plant Communications The plant communication system was a significant contributor to the proper and prompt mitigation of the event. The control room operators used the Gaitronics System to direct the equipment operators to various places in the plant to correct and operate equipment. Without the communications system, a number of operator actions would have been delayed or prevented, such as when: (a) the assistant shift supervisor informed the control room operator that toe SUFP was available after he had made the SUFP system operable; (b) the more experienced senior equipment operator was paged and directed to go to the AFW pump room where he opened the trip throttle valves and started the AFW pumps; and (c) after the AFW pumps were running, they had to be controlled manually at the pumps by the equipment operators in response to directions communicated from the control room.

6.2.4 AFW Pump Turbine Overspeed Trip The AFW pump turbine overspeed trips could not be reset in the control room; action had to be taken at the equipment. The trip throttle valve is a manual valve and the associated linkages must be manually manipulated at the AFW pump by an operator. If, for example, the AFW pump room became uninhabitable, overspeed trips of the AFW pump turbines could not be reset and the AFWS would remain unavailable.

6-7

6.3 Personnel Issues There are a number of management-labor situations affecting personnel morale at Davis-Besse--the most talked about being ongoing contract negotiations.

Further, some licensed operators resent a Toledo Edison dress code requiring that they wear uniforms. According to the operators' interviews, neither plant morale nor contract negotiations have had any adverse impact on plant 1 operations and maintenance. A good deal of mutual respect exists among the people working at the plant. "They have worked many years in nuclear power, so everybody is competent," according to one operator. They are concerned about losing their jobs, for example, if the NRC recommended that the plant not go back on line. In general, the operators were skittish towards NRC. As one operator put it, "I am more uncomfortable in this room this morning than I was in the auxiliary feedpump room Sunday morning and I had the whole plant on top of me Sunday morning."

Throughout the course of its fact-finding efforts, the Team met with and t

interviewed Toledo Edison managers, operators and support personnel. The Team could not infer from their comments or from their actions on June 9 that management-labor issues at Davis-Besse adversely affected operator performance.

i  !

l l

i l

6-8 I

4

7 SAFETY SIGNIFICANCE A total loss of feedwater is a significant event. It can have severe consequences if actions to ensure prompt and effective recovery are not taken. The conse-l quences and significance of the June 9 event could have been far different had Edditional equipment failed, had additional personnel errors been made, or had i recovery otherwise been delayed. Thus, there are many possibilities and differing sequences which could have affected the safety significance of this i transient.

The time margins and consequences of alternate sequences remain under study by the NRC. However, based upon what happened during the event, and on the analy-ses of the consequences of loss of feedwater events provided by Toledo Edison, the Team was able to gain a perspective on the safety significance of the event, on the time available for its mitigation, and on the effects of various combina-tions of equipment available for mitigation.

l j When a reactor trips, decay heat must be removed. The preferred heat removal

, path is through the steam generators. If this path is not available, direct core cooling must be initiated. If decay heat is not removed from the reactor

! coolant system more rapidly than it is produced, temperature and pressuie will

! increase. The pressure rise would be limited by the PORV and primary safety j valves, but when the pressure rise is limited or reduced by these valves, reactor

! coolant is lost. If this loss continues and the inventory is not made up from external sources, eventually the core will become uncovered and fuel damage will result. Thus, the parameters which assume importance in this mode of cooling are the system pressure and the pressure capabilities (and flow) of the systems available to provide makeup cooling water to the reactor coolant 3

system.

In reality, over a period of approximately 15 minutes, the Davis-Besse steam

' generators boiled essentially dry. As a result, the reactor heat sink was lost and reactor temperature and pressure increased. Eventually a reactor coolant t

temperature r" 594*F was reached, which corresponds to a saturation pressure of 1460 psig. (that is, if the system pressure decreased to or below 1460 psig, l

bulk boiling of the water in the core would result). The high pressure injec-tion (HPI) pumps at Davis-Besse have a pressure capability of approximately

{

1630 psig. When operated in the piggy-back mode with the low pressure injection This i pumps, the HPI pumps have a maximum pressure of approximately 1830 psig.

l higher pressure corresponds to a reactor coolant saturation temperature of 623'F.

Had feedwater not been restored or other mitigative actions taken, extrapolations indicate that the reactor coolant temperature would have reached 623*F about 20 minutes after the loss of feedwater, or approximately 13 minutes after the steam generators essentially boiled dry.

1 (1) through l As previously indicated, decay heat can be removed in two ways:

l the boiling action in the steam generators and (2) through release of coolant i

through the PORV and safety valves with makeup water (MU/HPI cooling mode).

Feedwater to the steam generators can be supplied by the main feedwater system,

[

7-1

.. _ _ _ . _ . _ _ - _ _ . _ _ _ _ ..~ _ _ . _ _ _ _ _ . _ . _ - , _ _

two steam-turbine driven AFW pumps or by the electric-motor driven startup feed-water pump. Pumps available for use in the MU/HPI cooling mode include the two reactor makeup pumps and the two high pre;sure injection pumps discussed above.

In fact, the two makeup pumps can provide flow to the primary system even when the primary system pressure is at the safety valve setting. The HPI pump, how-ever, as previously discussed, cannot.

On June 9, flow from the auxiliary feedwater pumps promptly reversed the tem-perature rise in the reactor coolant system. However, these pumps require steam for their operation. In this particular event, the motor-driven startup feedwater pump was available, thus steam availability for the AFW pump turbines was assured. Even if this pump were not available, the auxiliary feedwater pump turbines could possibly be started with the high pressure steam stored in dry steam generators. Calculations have indicated that stored steam at a pres-sure of 1000 psig would be sufficient to start the pumps (Ref. 1).

Another factor influencing the ability to start the AFW pumps is leakage of steam, if the startup feedwater pump is not available. Leakage of steam through leaking steam line safety valves or release of steam by misoperation or leakage of the steam line atmospheric vent valves could have affected steam availability for restarting the auxiliary feedwater pump turbines. It should be r.ated that subsequent to a reactor trip which occurred on June 2, 1985, seven main steam safety valves and one atmospheric vent valve were found to be leaking. In the June 9 event, steam header pressure swung over 100 to 250 psi for several minutes for unknown reasons after closure of the main steam isolation valves (MSIV).

Since the main steam safety valves routinely lift for reactor trips at high power at Davis-Besse, valve leakage or failure to fully reseat is not unlikely.

The Team considered available analyses of loss of feedwater events. Among these was a report prepared for Toledo Edison by EDS Nuclear Inc., " Davis Besse Unit No.1 Auxiliary Feedwater System Reliability Analysis Final Report," dated December, 1981 (Ref. 2). This report asserts that adequate core cooling and the prevention of fuel damage at Davis-Besse can be accomplished in the follow-ing two ways whenever the main feedwater flow or the reactor coolant system

forced circulation has been interrupted

j j 1. Availability of full flow from at least one of the redundant AFWS l turbine-driven pumps to one steam generator within 10 minutes of the initial loss of main feedwster or loss of forced circulation. l

[ 2. Availability of main feedwater startup pump flow to one steam gener-ator, combined with availability of primary coolant makeup flow from at least one makeup pump, manual opening of the pressurizer pilot operated relief valve, and isolation of reactor coolant system let-down within 30 minutes of the loss of feedwater.

During the fact-finding effort, Toledo Edison provided the Team with a report dated June 22, 1981, entitled " Engineering Summary Report of a Complete Loss of Feedwater Transient Analyses for Davis Besse, Unit 1" (Ref. 3), prepared by Babcock & Wilcox (B&W) Company. It was marked as a " Draft" with a note that "This Document is Presented as Preliminary and For Information Only. This Docu-ment Does not Serve as a Licensing / Procedure Base Document For Davis-Besse."

The Team reviewed the analyses contained in the document in an effort to assess

the capability of systems at Davis-Besse to mitigate a loss of feedwater event 1

7-2

i with failures in the systems needed for mitigation. It should be noted that such events are beyond the design basis for the plant. Following is a discus-sion of the results from the B&W report.

l Using normal conservative licensing assumptions, and assuming neither operator actions nor AFW, the core begins to be uncovered at approximately 37 minutes and is completely uncovered by 41 minutes following complete loss of feedwater, j With the assumption of a " realistic" decay heat curve, opening the PORV and manual initiation of both makeup pumps by the operator at 30 minutes extends the time for beginning to uncover the core from 37 minutes to over 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.

i Operator action at 30 minutes to open the PORV, to manually initiate one makeup pump and to place the startup feedwater pump in operation would prevent the core from becoming uncovered. Initiating both makeup pumps and the startup feedwater pump at 30 minutes without opening the PORV would also prevent the core from becoming uncovered.

Tables 7-1 and 7-2, reproduced from the B&W report (Ref. 3), summarize the con-sequences of loss of all feedwater with various equipment tised for mitigation.

Loss of offsite power is also assumed in these analyses. It should be noted that the tables show that the startup feedwater pump was assumed available for all cases in which analyses showed the mitigative actions to be successful. It should also be noted that in the June 9 event, the plant was initially at 90%

power and not, as assumed in the tables, at full power, and main feedwater from main feedpump 2 continued for approximately 5 minutes after reactor trip. Also, both auxiliary feedwater pumps were returned to service and the startup feedwater pump was placed in service in less than 20 minutes.

j Toledo Edison submitted a report to NRC entitled " Analysis of a Complete Loss of Feedwater Transient for the Davis-Besse Nuclear Power Station Unit 1" (Ref. 4).

This report analyzed a complete loss of feedwater under two circumstances:

(1) no operator actions and (2) operator actions at 30 minutes to open the PORV, to manually initiate one makeup pump and to place the startup feedwater pump in operation. The conclusions were the same as those in the June 22, 1981, Babcock & Wilcox report (Ref. 3) discussed above.

In assessing the safety significance of the June 9 event, a review of the analyses of loss of feedwater transients indicates that loss of feedwater is an event where mitigation is required within approximately 30 minutes to I hour. On June 9, ample equipment was available to fully mitigate the transient in less than 20 minutes. Both safety-related auxiliary feedwater trains were available, the startup feedwater pump was available, both reactor coolant makeup pumps were  !

available, and the operator had the capability to open the PORV and makeup flow-control valve from the control room. The equipment of most value for the event, however, had been placed in service only through relatively complex actions out-side the control room. The startup feedwater pump appears to have been partic-ularly important in that it is capable of ensuring steam availability for the turbine-driven auxiliary feedwater pumps even after the steam generators are dry, and according to the previously discussed analyses, it can be used in combination with one reactor coolant makeup pump to prevent the core from being uncovered even without the safety-related auxiliary feedwater pumps.

The key safety significance of the event, however, is the fact that multiple equipment failures occurred, initiating a transient beyond the design 7-3

j basis of the plant. Each of the following, without corrective operator actions, would have defeated operation of the safety-related auxiliary feedwater l; system:

1. The operator error in SFRCS actuation on low pressure.

2. The failure of the auxiliary feedwater system containment isolation valves to reopen after their inadvertent closure.

3. The overspeed tripping of the auxiliary feedwater pumps.

The event demonstrates the susceptibility of redundant equipment to common mode failures and reiterates the importance of " defense in depth" to reactor safety.

Excellence in equipment maintenance, thoroughness in identifying the basic causes for system malfunctions, thoroughness in testing systems under conditions for j which they may have to perform, and excellence in operator training are all

! required to ensure safety. The value of diversity and prompt and effective operator action in accomplishing key safety functions are particularly evident from this event.

I

! l i l i l l

7-4

= - - - - - - - . - . .. - _ _ _ . - _ .- .-. . . -.. .

l l

Table 7.1 Alternate Operator Actions at Davis-Besse for Loss of All Feedwater and Offsite Power Transient Operator action required 1.0 ANS decay heat at 30 minutes 1.2 ANS decay heat (realistic cases)

Number of makeup pumps 2 2 1 2 1 actuated at 30 minutes Yes No No Yes Yes l

PORV opened at 30 minutes Electric startup feedwater

[ Yes Yes Yes Yes Yes i pump actuated at 30 minutes 4

Success of action to miti- Yes 50%* Yes Yes gate accident ~ 50%*

j c

• Chance of success.

l a

i i

i i

I J

l e

a I

i i

i i

I J

t

+

) 7-5

Table 7.2 Summary of Alternate Operator Actions at Davis Besse for Loss of all Feedwater and Offsite Power Operator action required 1.0 ANS decay heat at 30 minutes 1.2 ANS decay heat (realistic cases) j Number of makeup pumps 2 1 2 1 1 2 2 1 2 1 1 2 2 1

i actuated at 30 minutes l

j PORV opened at 30 minutes No No Yes Yes Yes Yes No No Yes Yes Yes Yes No

! Electric startup feedwater Yes Yes Yes Yes No No Yes Yes Yes Yes No No No

, pump actuated at 30 minutes

, Success of action to miti- No No 50%* No No No Yes 50%* Yes Yes No No No

gate accident 4

y

• Chance of success.

cn

}

4

]

l i  :

9 b

I 4

__ _ . _ __ . _ _ . - . . _ . . _ _ _ _ _ _ _ _ _ _ _ = _ _ _.

I l

References

1. L. E. Roe, Toledo Edison to R. N. Reid, NRC, transmitting "An analysis on the capability of a dry and isolated steam generator to start an auxiliary feedwater pump 30 minutes af ter the loss of all main and i

auxiliary feedwater." June 23, 1979.

l

) 2. R. P. Crouse, Toledo Edison to T. N. Novak, NRC, Transmitting " Davis-Besse l

No.1 Auxiliary Feedwater System Reliability Analysis Final Report."

Docket 50-346. December 31, 1981.

f 3. Draft " Engineering Summary Report of a Complete Loss of Feedwater

! Transient Analysis for Davis-Besse Unit 1," Babcock & Wilcox Company l (582-7151-14-00). Docket 50-346. June 22, 1981.

4. L. E. Roe, Toledo Edison, to H. R. Denton, NRC, Transmitting " Analysis of

' a Complete Loss of Feedwater Transient for the Davis-Besse Nuclear Power Station, Unit 1." Docket 50-346. June 15, 1979.

l P

I i

l t

l i

i 6

(

4 7-7 1 \

i

8 PRINCIPAL FINDINGS AND CONCLUSIONS The Team's findings and conclusions are based upon an evaluation of the following:

1. Information from interviews of Toledo Edison (licensee) and NRC Region III personnel;

2. Plant data recorded during the event;

3. Information from meetings with the licensee and Region III personnel;

4. The licensee's troubleshooting action plans for equipment that malfunctioned;

5. Information obtained from the equipment troubleshooting activities; and

6. Available analyses of the consequences of loss of feedwater events at Davis-Besse.

It must be recognized that the root cause determination process is critically important and is not yet completed by the licensee. Table 5.1 summarizes the results to date for each equipment problem. The final results could, of course, revise the information in this report and perhaps raise important additional aspects or issues.

The Team has concluded that the underlying cause of the loss of main and auxi-liary feedwater event of June 9,1985, was the licensee's lack of attention to detail in the care of plant equipment. The licensee has a history of perform-ing troubleshooting, maintenance and testing of equipment, and of evaluating operating experience related to equipment in a superficial manner and, as a result, the root causes of problems are not always found and corrected.

Engineering design and analysis effort to address equipment problems has i frequently either not been utilized or has not been effective. Furthermore, I operator interviews made clear that equipment problems were not aggressively addressed and resolved beyond compliance with NRC regulatory requirements.

In addition to this major conclusion on the underlying cause of the event, the Team has made the following findings and conclusions. There is no significance to the order in which they are presented.

(1) The key safety significance of the event is that multiple equipment failures occurred, result.ng in a transient beyond the design basis of the plant. These failures included several common-mode failures affecting redundant safety-related equipment.

(2) If the failure of only the safety-related equipment could have been prevented, the event would not have been so serious or so complicated.

(3) If the safety-related auxiliary feedwater system equipment had functioned in accordance with system design requirements, the operator error in ini-  ;

tiating the steam and feedwater rupture control system on low steam pres-sure rather than low steam generator level would have been corrected in less than a minute and would not have had a significant effect on the course of the plant transient.

8-1

~

(4) Based on the licensee's current hypotheses for the causes of the auxiliary

,' feedwater system containment isolation valve and pump malfunctions, the causes could have been detected and corrected prior to the event by i straightforward tests. Such tests had apparently never been run during the life of the plant.

(5) The licensee's lack of effective engineering for determining the proper settings for valve torque switch bypass contacts and improper imple-

mentation of specified settings were the probable causes of the auxiliary feedwater system containment isolation valve malfunctions. Furthermore, this problem likely exists with other valves at Davis-Besse and could exist at other plants.

(6) Neither the SFRCS system nor the auxiliary feedwater system at the Davis-Besse plant meet the single-failure criterion for all design basis acci-l dents.

(7) The availability of the electric motor-driven startup feedwater pump sig-nificantly improved the safety margin for the plant during the event. The capability to promptly place an electric motor-driven pump and associated valves for supplying auxiliary feedwater in service from the control room would have significantly increased the safety margin for the plant during the event.

(8) The operators' understanding of procedures, plant system designs, and spe-cific equipment operation, and operator training all played a crucial role in their success in mitigating the consequences of the event. However, if the equipment operators had been more familiar with the operation of the auxiliary feedwater pump turbine trip-throttle valve, auxiliary feedwater could have been restored several minutes sooner.

(9) The locked doors and valves in the plant had the potential for signifi-cantly hampering operator actions taken to compensate for equipment malfunctions during the event and were a significant concern to the equipment operators.

(10) The operators did not initiate MU/HPI cooling (feed and bleed) immediately ,

upon reaching plant conditions where MU/HPI cooling is required by the emergency procedures. MU/HPI cooling was delayed because of the belief that restoration of feedwater was imminent and a reluctance to release reactor coolant to the containment structure. The operators and plant management believed that analyses for Davis-Besse indicated that 30 min- I utes was available before actions were required to prevent the core from beginning to uncover.

(11) If the manual initiation features of the SFRCS had originally been properly designed with regard to human factors considerations, such as labeling and placement, it is likely that no operator error in this initiation would have occurred. Further, if only the previously identi-fied human engineering deficiency regarding SFRCS manual initiation on low pressure had been corrected prior to this event, the operator's erroneous

initiation would likely have resulted in isolation of only one steam i

generator from auxiliary feedwater, f

! 8-2 l - -- - _. . -_ --. - _- .- - - .-

(12) The event was not reported to the NRC Operations Center in a manner reflecting the safety significance of the event. The more serious the event, the more operator involvement required to maintain plant safety.

For example, if the June 9 event had been protracted, knowledgeable personnel would not have been available to maintain an open telephone line with the NRC.

(13) Although the PORV is involved in the recovery from certain plant tran-sients, its reliable operation has not been established by a suitable test program nor is its operational readiness verified by a periodic surveil-lance test.

(14) The post-THI improvements: Temperature-saturation meters, additional training on transient behavior, and ATOG emergency procedures made a positive contribution to the mitigation of the event. Of these, training on transient behavior was the most important. The PORV flow acoustic monitor was not used by the operators. Because the shift technical advisor was not in the control room at the time the event began, and because the transient occurred so rapidly, he did not provide technical advice to the shift supervisor.

(15) Thorough integrated system testing under various system configurations and plant conditions as near as practicable to those for which the system is required to function during an accident is essential for timely detection and correction of common mode design deficiencies.

(16) For plant events involving conditions outside the plant design basis, operator training and operator understanding of systems and equipment are key to the success of mitigating actions taken by the operators. It is not practical to rely on detailed step-by-step procedures for such events.

(17) Operators at other plants may be reluctant to initiate MU/HPI cooling ,

(feed and bleed) or similar actions without a delay to reconfirm the need I and to consider less severe alternatives.

(18) The instrumentation available in the control room during the event was not adequate to clearly inform the operators that the criteria for MU/HPI cooling had been reached. The only practical alternative was the SPDS, I which was not available, nor was it required to be available.  !

8-3

i l

[

3

\] ij UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D. C. 20555

\...../ JLW 1 0 1985 l

l .

MEMORANDUM FOR: Chairman Palladino Commissioner Roberts Commissioner Asselstine Commissioner Bernthal Commissioner Zech FROM: William J. Dircks Executive Director for Operations

SUBJECT:

INVESTIGATION OF JUNE 9, 1985 EVENT AT DAVIS BESSE WILL BE CONDUCTED BY NRC TEAM About 1:30am on June 9, 1985, a loss of feedwater transient occurred at Davis Besse. The reactor tripped from 90% power on the loss of a main feedwater pump; the other feedwater pump was lost because of an inadve'rtent MSIV isolation.

Both turbine-driven auxiliary feedwater pumps failed to operate as designed.

It is understood at this time that the steam generators reached a " dry-out" condition and that the pressurizer PORY actuated. Feedwater was restored within approximately 12 minutes. Instrumentation indicated that adequate subcooling was maintained at all times.

Because this event has potential safety implications worthy of further study l I haveofrequested team technical experts AEODtotothetake sitethe to:necessary(action a) fact-find as to to send what a small (4 mem happened;

. (b) identify the probable cause as to why it happened; and (c) make appropriate findings and conclusions which will form the basis for possible follow-on actions. The team will report directly to me and is composed of:

Dr. Ernie Rossi, Team Leader (IE), Mr. Wayne Lanning (AE00), Mr. J. T. Beard (HRR), and Mr. Larry Bell (Reactor Training Center, Chattanooga). This team was selected on the basis of their knowledge and experience in the fields of.

operations, instrumentation and control, and reactor systems. Because the team has not completed incident investigation training, 01 is providing assistance regarding investigation techniques and support. The team will .

leave in the evening of June 10, 1985 and will be onsite early June 11, 1985.

The licensee has been requested, to the extent practical, to preserve the equipment in an "as-found" state and to have personnel and records available.

A-I

k

. The Commission  !

1 i

a j They have agreed to this request to the Regional Administrator. The team's

! report will constitute the single fact-finding investigation report. It is expected that the team's report will be issued no later than 30 days from now.

J i

j (SipedWmism 1.Dircks  ;

I William J. Dircks Executive Director for Operations 1

cc: SECY i OPE j OGC 1 ACRS I

l l

1 i

i .

j i

)

i 1

I I

i i

i 1

o I

i i

i I

! A-2 i

1 APPENDIX B e

' DATE f 1,NrggCOMPANY MEMORANDUM June 13, 1985

'O June 15, 1985. Rev. 1 l

I June 15, 1985, Rev. 2 Action item Lead Individuals June 19, 1985, Rev. 3 i

' *aou June 21. 1985 Rev. 4 SusJE CT J. K. Wood-_ dt h '

! Guidelines to Follow When Troubleshooting or Performing Investigative l Actions into the Root Causes Surrounding the June 9. 1985 Reactor Trip i

)

l 3

j For each item on the Equipment Freeze list (Attachment 1), an action

plan shall be developed for investigative or troubleshooting work which j 1 provides the basis for the Maintenance Work Order. Personnel (lead and/

{

or support) developing the action plan shall have knowledge of the design 1 criteria of the specific area being considered. Vendor engineering i

support will be utilized as necessary to accomplish this requirement.

i When used, vendor assistance shall be documented.

i Troubleshooting and investigative activity shall be preceded by event

evaluation and analysis to determine hypothesis (ses) and probable causes of failure or abnormal operation. Analysis and evaluation shall proceed

4 as follows: i 8

1 I

{ a. Collect and analyze known information/ operational data for condi-

! tions prior to, during and after the transient.

l l b. Review maintenance and surveillance / testing history. I i 2  ;

c. Develop a summary of data including a and b above that support any i proposed probable cause of failure or abnormal operation.

j d. Conduct a change analysis (i.e., what has changed since the last known successful operation of the system or equipment).

e. Based on above Items a-d develop primary and alternate hypoth-

! esis(ses) for the root cause of the problem.

l f. Develop plans for testing the probable causes/ hypothesis (i.e.,

checks verifications, inspections, troubleshooting, etc.). In l

developing inspection and troubleshooting plans, care must be taken ,

l to insure when possible that the less likely causes/ hypotheses (ses) remain testable. When planning troubleshooting activity try to simulate as closely as practical the actual conditions under which the system or component f ailed to operate properly on June 9,1985.

g. Document the above in a report.

It is very important that the performance of our investigations do not in any way result in the loss of any information due to disturbances of components or systems. Investigations need to be conducted in a logical, ,

well thought-out and documented manner. To avoid the loss of information l l t

l B-1

Juna 13, 1985 June 15, 1985, Rev. 1 June 15, 1985 Rev. 2 June 19, 1985, Rev. 3 i

June 21, 1985, Rev. 4 I Page 2 4

and to assure the capture of reliable information, the following guide-lines in addition to the requirements of AD 1844.00 need to be addressed and followed when initiating and implementing an NWO.

l 1 1. All action plans for trouoleshooting and investigative work shall be reviewed with NRC personnel prior to implementation.

2. All MW0s relating to the 6/9/85 trip investigation shall be handled as NSR.

3. Troubleshooting and repair shall be accomplished on separate MW0s.

4. MWO's are to be approved by the Action Item Lead individual and reviewed by QC prior to their implementation. Copies of MW0s, when approved by the Action Item Lead Individual, shall be forwarded to D. J. Mominee (Stop 3070). It is the Lead Individual's responsi-bility to assure that the investigative actions are appropriate, I suf ficient, properly defined, documented, and data is preserved.

5. Only those MWO's approved by the Action Item Lead Individual and QC may be worked on any of the "f rozen systems" identified on the attached list.

6. Assure that only current drawings and controlled vendor manuals are used.

7. Consider the need for vendor representatives. Vendor representatives should be used to assist in troubleshooting if appropriate expertise is not available in-house. The representatives will need to be given specific guidance for what they are and are not to do. Vendor representatives must follow the guidelines of this memorandum and 1 requirements of the Maintenance Work Order.

8. The MWO must clearly document the scope, affected equipment, and the  ;

desired objective of the investigative activity.

9. The sequence of activity needs to be documented on the HWO or proce-l dures specified in the MWO. If the sequence can be determined prior to the activity being performed, define that sequence and provide a checkoff for each step. If the desired sequence cannot be determined prior to the activity, as a minimum define the fundamental sequence to be taken and document each specific step as it is performed.

10. Document on the HWO all as found conditions. Visual inspect and document any missing, loose or damaged components, note positions 1 (open, closed, up, down, knob settings, switch positions, setpoints, etc.) abnormal environmental conditions, operation of cooling de-vices, water leaks, oil leaks, loose fittings, cracks, evidence of overheating or water damage, cleanliness, bent tubing, fluid levels, 1 jumpers, lifted wires, etc. Describe the overall condition or C-2

Jun 13, 1985 June 15, 1985, Rev. 1 June 15, 1985, Rev. 2 ,

June 19, 1985, Rev. 3 June 21, 1985, Rev. 4 Page 3 appearance. Whenever possible, use photographs to document as found conditions. When considered necessary, retain a sample of fluids or their residue for further analysis.

11. When discrepancies are noted during the investigation, stop work and notify the Action Item Lead Individual. Document the deficiency.

The Lead individual must sign off on the discrepancy prior to contin-uing the investigation.

i 12. Document the results of the investigation on the MWo.

13. Prior to starting any repair activities the Action Item Lead Individual must document that all investigations have been properly completed.

14. No equipment is to be shipped off site without prior approval of

, 1 Nuclear Facility Engineering and Quality Engineering for including appropriate hold and witness points. Use the "Q" purchase order process to obtain these approvals.

NOTE: In all cases, applicable procedure must be followed. The requirements of this memorandum must be communicated to craft personnel to avoid any confusion or misunderstandings during this investigative period.

15. All failed or removed components / equipment shall be retained for ongoing review and examination. Complete traceability shall be maintained.

t The NRC shall be notified when the determination of the root cause of the 1 malfunction / failure has been made. As soon as practical, the results of the troubleshooting process, root cause determinations and justification will be presented to the NRC (e.g., next day in a meeting).

The NRC shall be advised as soon as practical of plans and schedules for corrective action work, prior to the work being perfermed.

NOTE: Any communication with the NRC personnel will be coordinated through John Wood.

JKW/SGW/bjs Attachment B-3

l l 6/15/85 Rsv. 1 8:00 a.a.  :

i 6/19/85 Rev. 2 12:30 p.m.

6/20/85 Rev. 3 9:30 a.m.

6/21/85 Rev. 4 8:30 a.m.

Attachment 1

Page 2 I l  !

i i

( The licensee agreed to complete a walkdown outside Containment of the l j Main Steam System by appropriate personnel to identify any additional damage that may have been caused by water hammer.

l The Fact-Finding Team stated that

a. If required for safety, work shall proceed. l i

I l b. Surveillance Requirements of the Technical Specifications should I be satisfied.

1 l c. The team should be advised of any actions taken in the two areas above.

! Scw/bjs l

4 t

i 2 i l

l k

• 1 i

i I

i i  !

i,  !

i i

1 4

1 I

i I

4 i  :

r f

t 0-5 t

r- 1 u a ocau auva,o., c-. .o, , . . ~. . ,vo. . . .% r,oc .- .. ~. ,

7o..' .=

7"o'"3$' elSLIOGRAPHIC DATA SHEET

.. . .. . , .oc, ,osi o, , .. . . . . .. . NUREG-1154

, , , , a . o .v. , , , o , u... 6...

Loss of Na and Auxiliary Feedwater Event at the  ;

Davis-Besse ant on June 9,1985 . o . . . . . .c. , c o. ...o

.0,,. ....

....o.... .

July l/y 1985

. o . . .. .c. . ...u . o

.so ,.. , v...

July t! 1985

. . . . . . . _ . . . . , . . . , , . . . . , ._....,,,,__,.c. . ..e.. c . . . .. . .i..... ....

j Incident Investigati Team -

%..a Executive Director fo perations ...,- o.*~

l U.S. Nuclear Regulator omission Washington, D.C. 20555

( ., l-so....e.,

. . ~ ~ m. . ,e e.e . ~ ., . , ,e, . . . . ,o . . . . . ,e .a. . .. , , , . c . .... .

% /

f Fact Finding j Same as 7 6 f .. . oo cow ...o ,,-e .e m.see, f

.. v,.a .. ,.... ,o,.. 7 1

l

[

,,...,..o.n..,

3 ,.

4 OnJune9,1985,ToledoEdisonCompany\5 Davis esse Nuclear Power Plant, located

in Ottawa County, Ohio, experienced a p5rtialgloss of feedwater while the plant ,

was operating at 90% power. Followingaheadortrip,alossofallfeedwater l occurred. The event involved a number of~ ipment malfunctions and extensive ,

4 operator actions, including operator actio outside the control room. Several l operator errors also occurred during the eGe . This report documents the findings

of an NRC Team sent to Davis-Besse by ther NRC'Txecutive Director for Operations ,

j in conformance with the staff-proposed incident (Investigation Program. ,

i / \

i

/  %.

g 1r_

1 I ' k 1

a i

6 l' 6

%(

l

.. ooco..,,.,. ..........c.v. u x... o..

. . ., ,.g .

! g l

I k T Unlimited

'. .tCo.i, e C.. ,8,C A rlo, i 'D'n3assified

(

. .. . . . . . . . . o, . . . n o r . ,

,,,........a.

. . . .. a 1 .U.S. OcytapprMt FatMt!M CFFICEe 1981 641 721sf074 i

I UNITED STATES ,oun,, cta,, ,a,, t NUCLEAR REGULATORY COMMISSION rostaca * *<< s ra'o WASHINGTON, D.C. 20555 w."s'.". o' c OFFICIAL BUSINESS PENALTY FOR PRIVATE USE, $300 12055507ag77 US ARC I IAN19A19819C

- UI 0 T DR-PDR NUREG kASFINGTON DC 20555 i

I i

i 1

i l

4 e

r, ,

a 2

3

}

l 4

i I

_ _ _ _ _ _ . _ - _ _ - _ _ _ _ _ - _ _ _ _ - _ _ - _ _ - _ _ . - _ _ _ _ _ _ _ _ _ _ _ _ - _ - _ - _ _ - _ _ _ _ _ - - _ _ _ _ _ _ _ _ _ _ _ -