ML20029D008

Forwards Preliminary Case Study Rept, Human Performance Aspects of Operating Events with Bypass or Defeat of Engineered Safety Features, for Review & Comment

ML20029D008
Person / Time
Issue date:	04/15/1994
From:	Holahan G NRC OFFICE FOR ANALYSIS & EVALUATION OF OPERATIONAL DATA (AEOD)
To:	Oconnor W BABCOCK & WILCOX CO.
References
NUDOCS 9405030247
Download: ML20029D008 (3)

Text

W 151994 Mr. William T. O'Connor, Chairman Babcock and Wilcox Owners Group 5teering Committee 1790 Rockville Pike, Suite 525 Roc.\\ville, MD 20852 Subje::t:

Preliminary Case Study Report on Human Performance Aspects of Operating Events With Bypass or Defeat of Engineered Safety Features

Dear Mr. O'Connor:

A preliminary AE0D case study report, "Hu w rarformance Aspects of Operating Events with Bypass or Defeat of Engineerec Se t/ Features," is enclosed. The study contains observations, lessons leerned, ev:lusions, and recommendations based on eight events from late-1991 to present The study was conducted as a followup to se N ted issues raised in Case Study AE00/C92-01, " Human Performance in Operating svents," (NUREG-1275, Vol. 8).

We believe that issues raised in the study are significant and merit attention.

in accordance with our peer review process, prior to the finalization and distribution of our case study report, we are providing you, various industry groups, and experts in the field of human performance, with a copy of the preliminary report for review and comment. We request that you focus your review primarily on the accuracy and completeness of the observations and analyses contained in the report. The conclusions and recommendations are provided for your information in order that you may understand the i

significance well place on these issues and, therefore, obtain a more complete picture of the total report.

Changes to the report will be made if the l

underlying information is in error, or new additional information is provided.

We ask that comments be provided by letter.

Since we wish to finalize and issue the report shortly, we ask that any comments be received by us with 45 days from receipt of this preliminary report.

Should you require additional time beyond that point, please let us know; otherwise it will be assumed that you have no comment.

If you or your staff have any questions regarding this study, please feel free to contact me or John Kauffman at (301) 492-4436.-

Sincerely, Q S Q 47 Q 15 Original Signed by:

PDR Gary M. Holahan, Director yivision of' Safety Programs on, 4'Abb@K

~'"% h[h iffice for Analysis and Evaluation

,,j w

of Operational Data

~7~G

Enclosure:

As stated bec w/o enclosure:

See attached j(_ gy_fqp@ tdOO3038,1 Distribution:

See attached

• See previous concurrence:

h, e

R0AB R0AB C:ROAB DSP JKauffman:mmk Glanik JRosenthal VBenaroya GHolahan a 1 3/31/94*

3/31/94*

3/31/94*

JV/94 4/f/94

< j {p f

g a

i

- L' 0-Mr. William T. O'Connor bec w/o enclosure:

W. T. Russell, NRR E. S. Beckjord, RES-A. C. Thadani, NRR M. M. Slosson, NRR R. C. Jones, NRR J. S. Wermiel, NRR R. M. Gallo, NRR A. E. Chaffee, NRR

,i T. T. Martin, R-I S. D. Ebneter, R-II J. B. Martin, R-III L. J. Callan, R-IV K. E. Perkins, R-IV/WCF0 L. A. Reyes, NRR S. A. Varga, NRR G. C. Lainas, NRR H. N. Berkow, NRR D. B. Matthews, NRR J. W. Roe, NRR J. A. Zwolinski, NRR J. E. Dyer, NRR E. G. Adensom, NRR S. C. Black, NRR DISTRIBUTION w/ enclosure:

Public

Central! File 4 DISTRIBUTION w/o enclosure:

ROAB R/F DSP R/F JKauffman Glanik JRosenthal VBenaroya PBaranowsky Dross EJordan LSpessard KRaglin, TTC MTaylor, ED0 RSavio, ACRS

-?

t Mr. William T. O'Connor, Chairman Babcock and Wilcox Owners Group

/

)

Arizona Public Service

/

j Mail Station 9878

-411 North Central Phoenix, AZ 85004

/

\\

Subject:

Preliminary Case Study Report on Human Performance Aspects of j

Operating Events With Bypass or Defeat of Engineered Safety Features

/

Dear Mr. O'Connor:

A preliminary AE00 case study report, " Human Performarfce Aspects of Operating Events with Bypass or Defeat of Engineered Safety Features," is enclosed. The studycontainscbservations,lessonslearned, conc 7usions,andrecommendations based on eight events from late-1991 to present The stuiy was conducted as a followup to seleqted issues raised in Case Study

)

t AE00/C92-01, " Human Performance in Operating / vents," (NUREG-1275, Vol. 8).

We believe that issues raised in the study a're significant and merit

/

attention.

In accordance with our peer review pro [ss, prior to the finalization and distribution of our case study report we are providing you, various industry groups,-andexpertsinthefieldofJumanperformance,withacopyofthe preliminary report for review and mment. We request that you focus your review primarily on the accuracy d completeness of the observations and analyses contained in the report The conclusions and recommendations are provided for your information i order that you may understand the significance well place on the e issues and, therefore, obtain a more complete H

picture of the total report.

Changes to the report will be made if the underlying information is i error, or new additional information is provided.

We ask that comments be pr ided by letter.

Since we wish to finaliz and issue the report shortly, we ask that any comments be received by us with 45 days from receipt of this preliminary report.

Should you r uire additional time beyond that point, please let us know; otherwise it w 1 be assumed that you have no comment.

If you or your st f have any questions regarding this study, please feel free to contact me or ohn Kauffman at (301) 492-4436.

Sincerely, Gary M. Holahan, Director Division of Safety Programs Office for Analysis and Evaluation of Operational Data En osure: As stated cc w/o enclosure:

See attached Distribution: See attached g

ROABpb -

R0ABh p

AB DSP D:DSP JKatiffman:mmk GlarH JRosenthal VBenaroya GHolahan 3/3 /94 3/J(/94 q/ )/94 3/ /94 3/ /94 6

c'

. +

DRAFT - FOR COMMENT April 21,1994'

- AEOD/C94-xx A

P p

CASE STUDY REPORT-I HUMAN PERFORMANCE ASPECTS OF OPERATING EVENTS WITH BYPASS OR DEFEAT OF ENGINEERED SAFETY FEATURES MARCH 1994 k

q Prepared by:.

John V. Kauffman

i JV

-n -

(.

.i Reactor Operations Analysis' Branch Omce for Analysis and Evaluation of Operational Data-U.S. Nuclear Regulatory Commission

1 4

DRAFT - FOR COMMENT April 21,1994 -

~

4 CONTENTS l

R FO R EWO R D.......................................................l v

A B B RE VIATI ONS.............................................. j 1

I NTR O D UCTI O N..............................................

1 2

B A C K G R O UN D '..............................................

-1 3

REVIEW OF OPERATIONAL EVENTS.............................. 2 3.2 La S al l e U ni t 2..........................................

3.1 Crystal River Unit 3.........................

3

~

3.3 Nort h Anna Unit 2..........................................

4 3.4 - Fermi Unit 2 6

3.5 Hatch Unit 1 6

3.6 Oco n e e Unit 3.................................

7

................ 7 3.7 Wolf Creek.......

3.8 South Tex as Unit 2..........................................

8 i

a 4

OBSERVATIONS AND LESSONS LEARNED......................... 9

'4.1 Design and Control of Engineered Safety Features................... 9 '-

4.2 - Control Room Instrumentation and Control 4.3 Trai ni ng................................................. ~ 10 -

10 4.4 Operator' Attitudes About Engineered Safety Feature Actuations........

11 4.5 Maintenance, Surveillance, and Testing...........................

- 12 '

4.6 Post.Even t R eviews..........................................

12 5

CO NCLUS IONS..............................................

13' 6

RECOM M EN DATIONS..........................................

15 7

REFERENCES 16 APPENDICES Appendix A Reactor Trip With Valid. Safety Injection Actuation iii

e DRAIT - FOR COMMENT April 21,1994 FOREWORD The Office for Analysis and Evaluation of Operational Data (AEOD) performed a study of operator actions in the control of engineere' lety feature (ESF) equipment, and found significant issues that need industry and regulatory attention. The staff found improvements resulting from past industry and regulatory efforts, such as procedural guidance and training for high-pressure injection termmation, and concluded that similar efforts would also yield benefits for other practices and other ESFs.

AEOD staff recognizes that the plants in the study recovered from the ESF failures resulting from operator action. If any defeat had continued, the event recovery guidance in emergency operating procedures would likely have caused operators to attempt to restore ESF function. Nevertheless, these events are legitimate precursors to more serious events and clearly indicate the need for better operator control of ESFs. The AEOD staff found the need for an integrated effort to improve ESF related operator activities in interrelated areas such as the following:

industry management support operator training. including simulator training emergency operating procedures and abnormal operating procedure analyses (to I

identify the best advice for procedures) emergency operating procedures and abnormal operating procedure upgrades (to incorporate improvement in procedures)

J Two other areas warrant detailed consideration. The first area includes possible design changes to minimize unneeded ESF actuations or undesirable consequences from ESF actuations. Minimizing the number of challenges to operators or the perceived need to intervene quickly during an event should lead to better ESF control. The second area includes the investigation of the human performance aspects of events to evaluate ther~ighly those conditions for which it is appropriate to bypass or defeat ESFs.

Thorough evaluations will enable licensees to find improvements that could be made in areas such as instruments, procedures, and training.

Our study highlights the need for improvement in ESF control and the need for a new commitment to improve human performance in mitigating accidents.

Gary M. Holahan, Director Division of Safety Programs Office for Analysis and Evaluation of Operational Data v

9 DRAFT - FOR COMMENT April 21,1994 ABBREVIATIONS AEOD Analysis and Evaluation of Operational Data (NRC Office for)

AFW auxiliary feedwater AOP auxiliary oil pump ATWS anticipated transient without scram BWR boiling water reactor EDG cmergency diesel generator EOP emergency operating procedure ERT event review team [ licensee's]

ESF engineered safety feature IIPCI high-pressure coolant injection IIPI high-pressure injection IN Information Notice LER licensee event report MFW main feedwater MOV motor-operated valve NEI Nuclear Energy Institute NRC U.S. Nuclear Regulatory Commission NSO nuclear station operator NUDOCS Nuclear Documents System

]

PTL pull-to-lock i

RCS reactor coolant system RCIC reactor core isolation cooling RO reactor operator RWCU reactor water cleanup SCSS Sequence Coding and Search System SI safety injection SRO senior reactor operator STA shift technical advisor

)

4 TMI-2 Three Mile Island Unit 2 TS Technical Specifications vii

y DRAFT - FOR COMMENT April 21,1994 q

1 INTRODUCTION Appropriate control of engineered safety features (ESFs) is an essential element of j

reactor safety, as evidenced by the Three Mile Island Unit 2 (TMI-2) and Chernobyl Unit 4 accidents, in which operators defeated ESFs that could have prevented or mitigated the accidents.

The study objectives are to (1) find operating events that involved inappropriate bypass or defeat of ESFs from late 1991 to the present, (2) find and evaluate contributing.

l factors, (3) and develop lessons learned for the U.S. Nuclear Regulatory Commission (NRC) and industry to use in improving the control of ESFs.

Current operating U.S nuclear power plants were designed and constructed with' considerable defense in depth, including automatic emergency core cooling systems with emergency power supplies, reactor containments with automatic isolation capability, and automatic reactor protection systems. The high reliability, diversity, and redundancy of these and other defenses directly influence the level of reactor safety.

u The operating crew can compromise the high reliability,' diversity, and redundancy of these defenses by making an error that results in common-cause failure of some portion -

of these defenses. Operating experience suggests that human intervention may be a dominant failure mode. For example, operators experienced difficulties controlling safety injection (SI) in at least 3 (25 percent) of 12 reactor trips and valid sis since December 1991 (see Appendix). This inappropriate defeat or bypass of ESFs during events is of great interest because the operator may not be able to recover frorn such errors in complex, stressful situations. All ESF defeats in the study were recovered. It is likely that if any defeat had continued, the event recovery guidance in emergency operating procedures (EOPs) would have caused operators to attempt to restore ESF q

i function.

.J 2

BACKGROUND The TMI-2 accident in 1979 increased the industry's awareness of the importance of control of ESFs and led to many improvements, such as bette'r EOPs and implementation of the shift technical advisor (STA). An event at Crystal River in -

December 1991 indicated that' control of ESFs continued to challenge operators and that the lessons learned from evaluating operating experience might lead to improved operator performance and reactor safety.

The NRC issued Information Notice (IN) 92-47, " Intentional Bypassing of Automatic Actuation of Plant Protective Features," to alert licensees to the.importance of formal' criteria and training for the limitations on bypassing plant protective features. The a

NRC's Office for Analysis'and Evaluation of Operational Data's (AEOD) staff began -

~

this study after observing later events at North Anna Unit 2 and Fermi Unit 2 in which j

1 q

1

'I

. - ~

4 DRAFT - FOR COMMENT Aoril 21,1994 licensee actions to address IN 92-47 were not fully effective in preventing inappropriate defeat of ESFs.

The appropriate control of ESFs is a complicated topic influenced by many variables including design and operating philosophy, training, regulations and other requirements, procedures, regulators, and utility management. Control of ESFs is also complicated by economic and safety considerations such as the need to perform required or beneficial maintenance and testing, minimize personal injury and unnecessary equipment damage, prevent unwanted actuations or isolations for operational convenience (such as when transferring electrical power supplies or during testing). and minimize the effect of spurious actuations on plant operation and power generation.

An operator controlling an ESF may need to choose among many, possibly conflicting, goals. Ideally, these choices are well planned, and are governed by Technical Specifications (TS), plant operating procedures such as EOPs, and plant administrative control policies and related implementing procedures such as those for equipment tagging, lifted wire and jumpers, surveillance and tests, and general conduct of operations and maintenance.

The AEOD staff reviewed operational events from December 1991 to March 1994 that involved inappropriate bypass or defeat of ESFs. The staff found these events by searching databases, including licensee event reports (LERs) coded in the Sequence Coding and Search System (SCSS), and in the Nuclear Documents System (NUDOCS) text search system. AEOD investigated many of the events on site as part of its human performance study program. The reviewers did not likely find all such events from the databases and expect that licensees did not recognize or report some events.

3 REVIEW OF OPERATIONAL EVENTS 3.1 Crystal River Unit 3 At 3:09 a.m. on December 8,1991, the plant was starting up after a short maintenance outage, when a slow loss of reactor coolant system (RCS) pressure became apparent to the operators (Ref.1). The actuator for the pressurizer spray control valve had failed significantly open but continued to indicate that it was closed. The operators did not realize why the RCS pressure was decreasing until the pressurizer spray line isolation (block) valve was closed about an hour later. The reactor tripped on low pressure. As pressure continued to decrease, a member of the operating crew bypassed automatic ESFs (high-pressure injection [HPI), emergency feedwater, emergency diesel generators

[EDGs], and partial containment isolation) actuation for about 6 minutes (the actuation bistables were tripped, indicating an actual demand, while the ESFs were in the bypass condition for about 16 seconds).

A crew member bypassed ESFs before the cause of the decreasing plant pressure was:

understood. This action was not directed by abnormal or emergency procedures and was 2

.- i DRAFT - FOR COMMENT April 21,1994 not directed by shift supervisors, who did not learn that ESFs were bypassed for several minutes. The operators returned the ESFs to automatic initiation mode when annunciators and " management on shift" alerted shift supervisors to the ESF bypass.-

HPI and the other systems then activated. Operators later established manual control of the HPI system to maintain RCS pressure above 103.0 bar (1500 psig).

The crew had difficulty with command, control, and communications. The operators failed to use the annunciator response procedure for low RCS pressure. An operator bypassed ESFs without direction or concurrence by shift supervisors and shift supervisors did not learn that an ESF was bypassed for about 6 minutes. The shift supervisor made a late declaration of an unusual event and related notifications. The shift turnover process did not ensure that all crew members were informed of recent observed changes in the behavior of the pressurizer spray valve. During the startup, the " management on shift" (a manager with senior reactor operator [SRO) qualification) helped bring the event to an end by noting that ESFs were bypassed and by recommending that the pressurizer spray isolation valve be closed.

The staff found weaknesses in procedures. The annunciator response procedure for low RCS pressure addressed responses to control circuit faults, but did not cover appropriate actions to diagnose and correct the cause of the pressure decrease. One of the station's abnormal procedures contained such guidance, including directions to close the pressurizer spray line isolation valve Operators did not execute all applicable steps of that abnormal procedure because ESF termination criteria had been met.

3.2 LaSalle Unit 2 At 8:47 a.m. on April 20,1992, a regenerative heat exchanger relief valve in the reactor water cleanup (RWCU) system lifted while the RWCU system was being shut down (Ref. 2). The resultant RWCU high-differential flow isolation signal was erroneously bypassed by a nuclear station operator (NSO) for 3-% minutes.

Several weeks earlier, an RWCU isolation had occurred because of a spurious RWCU high-differential flow signal. Both RWCU containment isolation valve motors failed when the RWCU system was isolated. Licensee managers criticized the operators for allowing the spurious isolation. The motors were replaced, and a testing program was established to verify motor-operated valve (MOV) limit switch settings as the plant power level increased during subsequent operation.

On April 20, Unit 2 was at 20 percent power. An NSO shut down the RWCU system as part of the testing to verify MOV limit switch settings. This shut down was accomplished by closing the system return valve before stopping the RWCU pumps which was in reverse order of the actions in the procedure substep. About a minute later, RWCU high-differential flow alarmed, indicating the start of a 45-second delay timer preceding RWCU isolation. At LaSalle, RWrU high-differential flow alarms were common while starting up or shutting down the system.

3

+

DRAFT-FOR COMMENT April 21,1994-The NSO wanted to preserve the test and obtained the shift foreman's permission to bypass the automatic ESF closure of the RWCU containment isolation valves. A second NSO bypassed the RWCU isolation,- and reported a continuing RWCU differential flow of about 360 liters per minute (95 gpm).

About 3 minutes later, the operators determined that the RWCU high-differential flow alarm was not spurious. An equipment attendant had reported flow through an RWCU regenerative heat exchanger relief valve, and a third NSO had found the level increasing in the reactor building equipment drain tank. The lead NSO asked the shift control roo.m engineer and the shift foreman how they wanted to isolate the RWCU system.

Both agreed to allow the automatic RWCU isolation although the special test procedure stated that valve operation without thermal overload protection (as was the case with automatic operation) could damage the motor or the valve if the limit switches had drifted because of thermal expansion. The operators returned the RWCU bypass key switch to normal position, allowing the RWCU to automatically isolate, which terminated the loss of inventory from the RWCU through the open relief valve.

A resident inspector performed a followup inspection of activities preceding the bypass of RWCU (Ref. 3). The inspection report documented that RWCU high differential j

flows were common during reactor heatup and cooldown and that managers decided to j

bypm,s spurious RWCU isolation signals. From the time the reactor went critical at about 6 a.m. until about 9:30 a.m., the licensee bypassed the RWCU isolation function four times, the longest period being for about 30 minutes. As shown by the April 20 event, an RWCU relief valve could be open resulting in a loss rate of about 360 liters per minute (95 gpm) of reactor coolant from the RWCU system with no other j

immediate indications other than the " spurious" high differential alarm.

3.3 North Anna Unit 2 At 7:16 a.m. on April 16,1993, a control problem in the main generator voltage

-I regulator actuated a field-forcing (overexcited) generatc annunciator (Ref. 4). The unit operator responded by attempting to lower the level of generator excitation. Seventeen seconds after the annunciator was received, a generator differential lockout occurred, causing the main generator, turbine, and reactor to trip.

Control room staff implemented the EOPs for plant recovery. While performing the reactor trip recovery procedure, the control room staff placed the motor driven auxiliary feedwater (AFW) pump control switches in pull-to lock (PTL) and closed the steam supply valves to the turbine driven AFW pump for about 18 minutes. The procedure reader discovered the system misalignment and inoperability when he reached a later procedure step that returns AFW to a standby configuration. Shift supervisors immediately returned the AFW to a standby, operable configuration. Required heat sink conditions for the reactor were maintained while AFW was inoperable.

The AEOD staff investigated human performance aspects of the event and found that 4

DRAFT - FOR COMMENT April 21 1994 1

AFW was disabled while a valid AFW automatic initiation signal (Lo-Lo Steam Generator Level) was present. Shift supervisors intended for the operator to throttle AFW valves to stop flow, which would have precluded further AFW flow without operator action.

The automatic start function of the AFW pumps was defeated for 18 minutes, unknown to shift supervision, and before meeting the criteria in the applicable EOP.

The licensee's response to IN 92-47 was not effective in ensuring appropriate control of AFW.

When the RCS average temperature decreased below its no-load value, the operators were unsure of which EOP step (s) to use to respond. This was after the operators had passed the step dealing with RCS cooldown in Procedure 2-ES-0.1,

" Reactor Trip Response." The operators could not decide whether to use the

" Response Not Obtained" column of a previous step, or to continue in the -

procedure. Ultimately, the operators interpreted a prior " verify" step as allowing them to switch over from AFW to main feedwater (MFW).

The AFW system was not designed for automatic reopening of the motor-operated discharge valves, or for automatic. restart of AFW pumps in PTL. These capabilities would allow the operators to control the system while maintaining its automatic safety function if needed later in an event. EOPs gave guidance to maintain minimum required AFW (throttled) flows, depending on conditions, with steam generator level below the' AFW actuation setpoint. Methods to stop or throttle AFW operation below its automatic initiation setpoint would effectively disable the AFW, an ESF system under a valid demand, and necessitate operator action for reinitiation (except for a SI).

The operator who placed the AFW pumps in PTL was concerned about possible pump degradation if the pumps were to run on recirculation flow, as would happen if the pump discharge valves were closed. The operator believed that flow could be restored more quickly and reliably, if needed, by starting the pumps than by opening MOVs.

3.4 Fermi Unit 2 While investigating an event that occurred on August 13,1993 (Ref. 5), the AEOD team learned of a similar event in November 1992. Therefore, the team also interviewed the principal operators in the earlier event to compare operator response between the events.

In this earlier event, a loss of feedwater caused a reactor trip and reactor pressure vessel Level 2 and 3 isolations and actuations. The operators throttled high-pressure coolant injection (IIPCI), reactor core isolation cooling (RCIC), control rod drive, and standby -

5

DRAFT-FOR COMMENT April 21, 1994 feedwater flows. A reactor operator (RO) took HPCI out of service after the reactor water level rose above the HPCI actuation setpoint. Operators shut down the HPCI by placing the HPCI auxiliary oil pump (AOP) in PTL after tripping the HPCI turbine.

Thesc actions were included in the HPCI operating procedure for HPCI shutdown.

Ilowever, the AOP remained in ITL for about 15 minutes, indicating that the crew did not expeditiously complete the remaining procedure steps to return HPCI to a standby, armed configuration. The procedure does not specify a time limit for completing these later actions.

With the HPCI AOP in ITL, operators would have to reactuate HPCI manually if needed because HPCI is inoperable without the AOP to supply hydraulic fluid during IIPCI startup. HPCI is typically designed to trip on Level 8 ar..

utomatically reinitiate on Level 2.

A licensed RO stated (Ref. 6) that allowing HPCI to be tripped by the Level 8 signal was not desired. For example, it might appear to indicate poor control of the event by the operators and might contribute to a high cooldown rate. He also stated that it was

" common practice" not to reset HPCI after operators had tripped it and " turned off the oil pump." On the same day, a nuclear training supervisor stated that this was not accepted practice by the training department.

The licensee did not identify defeat of the HPCI while reviewing the event. Therefore, it did not report those aspects of the event to the NRC, and missed an opportunity to prevent recurrence of similar ESF defeats at Fermi.

3.5 Hatch Unit 1 At 7:33 a.m. on September 30,1992, the reactor was manually tripped followed by a manual trip of the main turbine several seconds later (Ref. 7). HPCI and RCIC automatically actuated. The licensee later found that HPCI did not inject because the cactor water level was restored before the injection valve opened. The licensee had quickly restored the level using the "A" reactor feed pump and RCIC, and then used RCIC to maintain the reactor water level.

The resident inspectors observed portions of the licensee's event review team (ERT) investigation of this event, including the critique with the involved operators after the reactor trip. The inspectors noted several issues for additional review (discussed below).

Shortly after the scram, the shift supervisor directed that HPCI be "taken off." An operator placed the IIPCI AOP switch in the ITL position, which temporarily disabled the HPCI system. The shift supervisor immediately corrected this improper action and returned the pump to an available status. After the scram, the operators stated that the extensive training on anticipated transient without scram (ATWS) scenarios in the simulator may have contributed to this error. As part of the ATWS actions, HPCI injection is " term.inated and prevented" by placing the AOP switch in PTL, position in 6

DRAFT - FOR COMMENT April 21,1994 accordance with procedures. This technique is also used to prevent HPCI reinitiation following inadvertent HPCI initiation.

On the afternoon of September 30,1992, control room and refueling floor personnel were advised of apparently inoperable refueling floor ventilation dampers. The licensee halted fuel movement and, on investigation, found that control switches had been improperly placed in the "open" position instead of the " automatic" position at about 1:30 a.m. on September 30,1992, during manipulations of refueling floor ventilation systems. The involved operator used a procedure that required the dampers to be verified open.

The inspectors and ERT concluded that the procedure could easily be misinterpreted to require positioning of the control switches to the open position. The administrative controls for the switches were inadequate to minimize inadvertent or inappropriate bypassing of the safety function of the dampers. In most cases at Hatch, use of a keylock switch is required to override an automatic ESF component by a control room switch.

The inoperable dampers were not immediately identified during the scram recovery because the ESF actuations were quickly reset. Therefore, the licensee did not recognize the failure of the dampers to shut until the post-trip review of safety parameter display system data for the transient.

3.6 Oconec Unit 3 On January 26,1993, the reactor tripped from 100-percent power (Ref. 8). During the trip recovery, while the operators transferred from the emergency feedwater to the MFW pumps, a loss of automatic initiation of both emergency feedwater flow paths resulted when both emergency feedwater control valves were not placed in automatic mode as directed by procedure. The condition was first discovered about 5-% hours later by an RO reviewing the control board.

The licensee attributed the inappropriate operator action to improperly following the correct procedure. Another contributing factor was the failure of training instructors to clearly emphasize management expectation to check or initial check-off blanks in the Abnormal Procedures as the steps are completed. The licensee also stated that the correct procedure was improperly followed in a similar previous event. In that event, the HPl system was activated, which violated TS requirements for low-temperature overpressure protection. An HPI pump breaker was racked in, and the discharge valves were not deactivated.

3.7 Wolf Creek On May 8,1993, Wolf Creek entered Mode 3 (Hot Standby) from Mode 4 (Hot Shutdown) with both motor-driven auxiliary feedwater pumps rendered inoperable by their handswitches being in PTL (Ref. 9). This situation remained uncorrected for about

)

13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br />.

7

DRAIT 'FOR COMMENT April 21,1994 The licensee attributed the root cause to failure of the control room operating crews to control plant work activities such that administrative requirements could be thoroughly considered. The licensee also stated that operators did not follow the administrative procedures for control board walkdowns and shift turnover, in that two shift changes occurred without adequate discussion of the requirements for the motor-driven auxiliary feedwater pumps.

As immediate corrective action, managers clarified their expectations for turnovers, control room professionalism, and communications with operating crews. The licensee changed the procedures to render the mode change review process more effective.

3.8 South Texas Unit 2 On December 24,1991, a feedback linkage on the Loop A pressurizer spray valve became disengaged, allowing instrument air to force the spray valve to the open position (Ref.10). Shortly thereafter, the reactor automatically tripped because of low pressurizer pressure, accompanied by an SI and containment isolation. After the plant was stabilized, the operators left the EOPs. When instrument air was restored to the reactor containment building, the Loop A pressurizer spray valve failed open again.

Pressurizer pressure dropped below the nominal SI setpoint but an automatic block / reset feature of the SI actuation circuitry prevented the safety logic from sending an automatic SI signal. By design, all of the SI trains were blocked when the reactor trip breakers were open and S1 had niready actuated.

The licensee determined that the operators did not manually initiate SI because they incorrectly assumed that the EOP criteria for Si continued to apply even though the EOP had been exited. About 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> after exiting the EOP, the reactor trip breakers were closed, which automatically reset SI actuation capability.

The licensee later concluded that the plant was in violation of TS 3.3.2 from the time the EOPs were exited to the time when the reactor trip breakers were closed. South Texas TS require'two of three SI automatic circuits to be operable in Modes 1 through 4. The licensee attributed the cause of the TS violation to the operators exiting the EOP with all SI trains blocked because the operators did not consider that the design feature of the SI block would place the plant in this condition. The licensee revised the EOPs to call for resetting the SI permissive before exiting the procedure.

4 OBSERVATIONS AND LESSONS LEARNED Events involving inappropriate bypass or defeat of ESF can be of great safety significance as demonstrated by the accidents at Three Mile Island and Chernobyl.

Defeat or bypass of ESFs' can be a common-mode failure mechanism for these otherwise highly reliable systems, while operator recovery from such errors in complex, stressful situations is highly uncertain.

8

DRAFT - FOR COMMENT April 21,1994 Although many improvements in human performance have enhanced the control of ESFs, certain aspects could be further improved by considering the following lessons learned.

1 4.1 Design and Control of Engineered Safety Features 4.1.1 Most fluid system ESFs are designed to supply adequate flow for design basis j

events, assuming a single failure. However, the amount of flow needed for such events would exceed that needed for less significant events or when the assumed single failure does not occur. Because many ESFs lack automatic level or flow controls after initiation, many of these "overdesigned" ESFs may have adverse consequences (e.g., full AFW injection may lead to an excessive cooldown) without operator intervention during these less significant events.

4.1.2 "Overdesigned" ESFs contribute to operator intervention during events, which has led to errors that disabled the system (e.g. North Anna Unit 2). This need for operator action could be reduced by design and TS changes to minimize undesirable ESF actuations, such as only starting EDGs on bus undervoltage conditions rather than starting and running EDGs unloaded as now frequently occurs for loss-of-coolant accident or SI signals. Delays in restoring or rearming the system after it is shutdown have disabled the ESF (e.g., Fermi Unit 2).

4.1.3 Instruments that actuate ESFs may be unreliable during certain plant states or activities, as demonstrated by the event at LaSalle Unit 2. People tend to ignore or disable nuisances that continually " cry wolf."

4.1.4 If the operator acts to control an ESF during an event, TS requirements may not be met or may not clearly address the situation (e.g., manually throttling AFW after a reactor trip may lead to a loss of required, automatic AFW flow).

4.2 Control Room Instrumentation and Control After the TMI accident, the industry spent much effort such as in detailed control room design reviews to improve the man-machine interface, including instrumentation and control, to better support operator performance. Despite these efforts, the events in the study revealed weaknesses in instrumentation and controls.

4.2.1 The operators rely on instruments for decision making that may be inadequate for certain events. For example, an open RWCU relief valve at I.aSalle caused a 360 liter per minute (95 gpm) loss of reactor coolant that was not recognized by the operators because of prior experience of erratic indication. The instruments at Crystal River Unit J did not reveal the cause of reactor depressurization.

4.2.2 In some events, instrumentation and controls for ESF equipment misled operators regarding the status of the equipment. For example, at Hatch Unit 1, the control switches for automatic isolation dampers were in the open position which defeated 9

,e DRAFI'- FOR COMMENT April 21,1994 the automatic safety function of the dampers, without sufficient visual or audible cues to alert the operators to this defeat. At South Texas Unit 2, the operators did not consider the automatic block feature of the SIlogic before exiting the EOP, apparently because of insufficient cuing that SI was automatically blocked.

4.2.3 Instruments and annunciators did not sufficiently alert operators to misaligned ESFs at North Anna Unit 2, Hatch Unit 1, and Oconee Unit 3.

4.3 Training The training that operators receive is obviously a powerful influence on their response to events. Simulator training is particularly important to the operator's " mental modeling" of systems, their responses, and their interfaces with other systems. In addition, the simulator is where operators demonstrate and practice their responses to transient and accident conditions. This may well be their predominant experience base for responding to such events. Operator performance in real events, therefore, offers insights into the associated training program. Similarly, simulator training sessions provide opportunities to better support operator performance, for example, by identifying procedural or instrumentation enhancements.

4.3.1 Training operators to take specific actions, such as to disable HPCI by placing its AOP in PTL for boiling-water reactor (BWR) ATWS events, during particular types of events has contributed to similar (inappropriate) operator actions in other situations (e.g., Hatch Unit 1). These negative transfers of training illustrate the human tendency to revert to successful prior behavior patterns during a stressful situation.

4.3.2 Inappropriate operator actions involving the bypass or defeat of an ESF during simulator training have remained uncorrected in some cases, and may have contributed to similar inappropriate ESF defeat during events at North Anna Unit 2 and Fermi Unit 2. The switch from AFW to MFW has been particularly problematic, as shown by events at Oconee Unit 3, North Anna Unit 2, and Palo Verde Unit 3 (Ref.11).

4.3.3 Weaknesses during ESF bypass events suggest areas where training may be

)

improved.

Control room communications: Informal terms at Hatch Unit 1, lack of repeat-j backs or acknowledgements at North Anna Unit 2, and lack of feedback regardmg completion of actions at Crystal River Unit 3.

i Procedural usage: Procedures not used at Crystal River Unit 3 and LaSalle Unit 2, poorly executed at North Anna Unit 2, and misinterpreted at LaSalle Unit 2.

I 10

DRAFF - FOR COMMENT April 21,1994 Supervisory watch standing practices and team-building: shift supervisors (including the STA) unaware of operator bypass of ESF at Oconee Unit 3, North Anna Unit 2, and Crystal River Unit 3; lack of supervisory response to annunciators and other indications of bypassed ESF at North Anna Unit 2);

inappropriate defeat of an ESF at the direction of the SRO/STA at LaSalle Unit 2, and weaknesses in communications and procedural usage described above.

Verification of successful ESF operation, and completion of ESF safety function before it is reset: at Hatch Unit 1, operators did not verify that ESF functioned before reset and did not manually initiate equipment that should have automatically initiated. At South Texas Unit 2, operators did not manually -

initiate SI because they incorrectly assumed that the EOP criteria for SI termination continued to apply even though the EOP was exited earlier.

Plant policies and procedures (including EOPs) for control of ESFs: weaknesses in understanding policies and procedures for control of ESFs at LaSalle Unit 2, North Anna Unit 2, and South Texas Unit 2. Operators stated in interviews the training was passive, not active. For example, in a simulator session, if an operator was directed to " secure system 'A'" and performed this action

" correctly" by shutting valves in the system, he would not be informed by the instructor. Unless he performed unacceptable actions and was corrected, he might finish the session with the erroneous idea that other (inappropriate) ways of

" securing" the system, such as by placing pumps in PR, were also acceptable.

4.4 Operator Attitudes About Engineered Safety Feature Actuations 1

Factors that contribute to operators' views concerning ESF actuations include the operators' training, licensee management and NRC performance indicators and precedents (what is " counted" and what has been rewarded or punished in the past), peer pressure, individual pride, the reliability of the ESFs, and whether an ESF actuation benefits the particular situation.

4.4.1 Many factors contribute to the belief among some operators that ESF actuations are " bad" and should be avoided or minimized where possible. For example, ESF actuations are reportable, may be included in the performance indicators for the plant, may " unnecessarily" complicate an event, may have adverse consequences (e.g., may contribute to excessive cooldown), or may interrupt normal support system operation such as by isolating BWR shutdown cooling.

4.4.2 Licensee trainers and managers and NRC personnel frequently evaluate operators' responses in real events and simulator scenarios. Allowing an ESF injection system to automatically trip on high level, for example, may be considered as poor operator control or performance. Similarly, quick recovery and restoration of lost functions may be considered good performance and may be needed because nuclear power plants are complex with highly interactive systems.

Plant managers at LaSalle criticized operators for not defeating an ESF after an 11

~

^

DRAFF - FOR COMMENT April 21 1994 g

9 event where a valid ESF actuation resulted in equipment damage because equipment was set up improperly.

4.43 Based on the above, plant operators may perceive the need to intervene quickly during events or rapidly changing conditions. However, both the decisions made and the actions taken during this early intervention are prone to human error.

Therefore, countries such as France and Germany have designed systems with automated controls such that operator action is not needed and is discouraged (except to initiate failed automatic features or protect equipment or personnel) for periods of 10 to 30 minutes after a reactor trip.

4.5 Maintenance, Surveillance, and Testing 4.5.1 Errors during maintenance, surveillance, and testing have disabled automatic safety features at Wolf Creek and Hatch Unit 1.

4.5.2 Testing of automatic safety systems frequently requires defeat of the system. This has contributed to errors. For example, operators at LaSalle Unit 2, routinely defeated an ESF (RWCU isolation) for " testing" when encountering perceived equipment or instrument problems. When reactor coolant was lost outside of containment, the crew had been conditioned that defeat of the ESF was allowable, and defeated it, in this case, "living with" a problem introduced poor operating practices that later manifested themselves in improper performance during a loss-of-coolant event.

4.6 Post-Event Reviews When inappropriate bypass-of-ESF events are not recognized, they are not reported as such. When this occurs, the plant loses an opportunity to improve safety performance, and the industry is deprived of an opportunity to learn, and may be led to a false sense of security that such events do not happen or are very rare.

4.6.1 During or after AEOD site visits, the staff found five cases where the licensee had not discovered or recognized an inappropriately bypassed ESF: the Fermi Unit 2, November 1992 event, and four defeats of RWCU isolation at LaSalle 2. These inappropriate operator actions involving bypass or defeat of an ESF were not licensee identified or corrected.

4.6.2 Because some ESF bypass events were not recognized or reported, such events are more prevalent than indicated by the operating experience data bases.

Technical Specifications 4.7 12

3 J

DRAFT - FOR COMMENT April 21,1994 The AEOD staff studied the written guidance (see 4.3) for operator response to the situation or event, and other associated requirements such as TS.

TS typically address the operability of equipment before an event better than they do operability afterward. TS typically do not address well the need for the operator to control or inhibit ESFs during scrams and transients. These actions may be allowed or directed by EOPs but contrary to TS. Licensees have recognized similar situations and handled them in a variety of ways including making licensing interpretations, invoking 10 CFR 50.54 (x) and (y), or obtaining prior NRC approval of EOP actions in the safety evaluation of the liceasee's EOP submittal. Other situations, however, may not have been recognized or may have been resolved in a manner that appears illogical or contradictory. For example, Typical BWR TS require HPCI and RCIC to be operable in Modes 1,2, and 3. If operators take manual control of both systems after a trip, a strict interpretation could be that HPCI and RCIC were inoperable because they were incapable of performing their intended function (injecting rated flow) in the intended manner (automatically).

At North Anna Unit 2, the AFW function was disabled by placing the motor-driven pumps in PTL mode and isolating the turbine-driven pump steam supply while a valid AFW automatic initiation signal was present. Shift supervisors intended for the operator to throttle AFW valves to stop flow, which would have precluded further AFW flow without operator action. The licensee considered this intended action as acceptable

" operator control" of the system.

y 5

CONCLUSIONS The AEOD staff reviewed recent operating experience associated with this issue and offers the following conclusions:

5.1 Plants continue to rely on operators to control safety systems even though the measures taken to ensure appropriate control have weaknesses that have been demonstrated by operating experience.

5.2 Many individual corrective actions taken after TM1-2 have not been fully effective or have not been effectively implemented throughout the industry. These conclusions include observations of instrumentation and control deficiencies which TM1-2 actions did not address explicitly, but may be inferred to have been I

addressed as part of detailed control room design reviews or Regulatory Guide 1.97 efforts.

Engineering expertise on shift (STA or SRO): ESFs were inappropriately defeated without the knowledge of the STA and SRO, who did not learn of them for some time. In one example, ESFs were inappropriately defeated at the direction of the SRO/STA. Another operating crew did not 13 1

ev

+

DRAFT-FOR COMMENT April 21i1994 manually initiate SI because they incorrectly assumed that the EOP criteria for SI termination continued to apply even though the EOP was exited earlier (see 4.3).

Operating experience review programs: The licensee inappropriately defeated ESFs after receiving a generic communication on the control of ESFs. Some licensees took insufficient actions in response to the generic communication. However, the generic communication may not have sufficiently highlighted and detailed the concerns on this topic.

Instruments: Operators had no direct indication of open valves that caused either a loss of reactor coolant or a loss of primary system pressure. In one example, an ESF was bypassed when instruments were not viewed as reliable for existing plant or system conditions (see 4.2).

Alarms: Annunciators did not sufficiently alert other operators and supervisors to the defeat or bypass of ESFs. In one example, an ESF failed to respond during an event, and the operators were not alerted. In-another example, the operators did not consider the automatic block feature of the SI apparently because the Si automatic block was not sufficiently cued (see 4.2).

Controls: Poor control switch design led to a situation where an ESF was inadvertently defeated. The switch design did not visually indicate that the ESF was disabled. The system was not designed to provide an alarm when misaligned (see 4.2).

5.3 Aspects of some other regulatory programs were not fully effective or were not effectively implemented throughout the industry.

Event reviews: A licensee conducting a post-trip review as called for in Generic Letter 83-28, failed to identify inappropriate defeat of an ESF.

Inappropriate ESF defeats at another licensee were not identified until questioning by the NRC (see 4.6).

Operator training: In many instances, licensed operators took inappropriate actions, did not use procedures appropriately, did not effectively supervise or communicate, did not investigate alarms, had difficulty implementing EOPs, or did not ensure that automatic safety functions were performed (see 4.3).

Technical Specifications: The TS for certain plants do not address well the need for the operator to control or inhibit ESFs during events. This ambiguity has caused confusion when actions may be allowed or directed by EOPs but may be contrary to TS. Also, operator response to events 14 1

1

]

-o -

DRAFT-FOR COMMENT April 21,1994 may be impeded because the licensee is required to invoke 10 CFR 50.54 (x) and make related notifications during an event.

6 RECOMMENDATIONS The case study clearly indicates that ESF reliability requires appropriate and reliable operator action and that operators have experienced numerous problems in controlling ESFs.

Problems with human intervention in ESF action continue because the industry has not established an integrated, comprehensive program addressing system design, operator guidance, training, and management involvement to optimize ESF reliability.

6.1 The NRC's Office of Nuclear Reactor Regulation and the nuclear industry (through the Nuclear Energy Institute [NEI) and the Owners Groups' committees on EOPs) should initiate an integrated and comprehensive evaluation of efforts to establish high level licensee management policies and guidance on ESF operations, control, and termination establish detailed operator guidance and training on appropriate means to control and terminate E5Fs establish clear guidelines on the-relation of TSs to the operation and control of ESF systems make bettet use of automation for ESF logic and controls. In particular, the focus should be on minimizing the burden on the operator, for example by not starting systems unnecessarily and by adding simple level controls such that the need for operator intervention is reduced where feasible.

6.2 NRC and licensees should carefully evaluate licensees' administrative controls and training to control ESFs. They should evaluate the effectiveness of the controls-and training by scrutinizing operating experience; (i.e., post-event evaluation of operator responses to actual plant operating events, associated written guidnnce, and training). They should also perform root cause analyses to identify and correct contributing specific and programmatic weaknesses.

15

DRAFT - FOR COMMENT April 21,1994 7

REFERENCES 1.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to T. M. Novak, " Human Performance Study Report - Crystal River Unit 3 (12/08/91)," January 30,1993.

2.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to T. M. Novak, " Human Performance Study Report - LaSalle Unit 2 (4/20/92),"

June 16,1993.

3.

U.S. Nuclear Regulatory Commission, inspection Report No. S')-374/92-011, LaSalle Unit 2, June 1,1992.

4.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to G. M. Holahan, " Human Performance Study Report - North Anna Unit 2 (4/16/93)," June 23,1993.

5.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to G. M. Holahan, " Human Performance Study Report - Fermi Unit 2 (8/13/93),"

August 13, 1993.

6.

R. A. Spence, U.S. Nuclear Regulatory Commission, memorandum to J. V. Kauffman, HPCI Operation at Fermi Unit 2," December 1,1993.

7.

U.S. Nuclear Regulatory Commission, Inspection Reports No. 50-321 and 366/92-22, Edwin I. Hatch, November 6,1992.

8.

Duke Power Company, Licensee Event Report 287/93-001, Oconee Unit 3, February 25,1993.

9.

Wolf Creek Nuclear Operating Corporation, Licensee Event Report 482/93-010, Wolf Creek, June 2,1993.

10.

Houston Lighting and Power Company, Licensee Event Report 499/91-010-01, South Texas Unit 2, September 23,1992.

11.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to D. F. Ross, " Human Performance Study Report - Palo Verde Unit 3 (2/4/93),"

April 19,1993.

16

y=

l mA DRAFT - FOR COMMENT April 21,1994 APPENDIX A Reactor Trip With Valid Safety Injection' Actuation

~g December 1991 to Present i

Plant /Date Licensee Event Comments Report No.

Crystal River Unit 3 91-018 See report" 12/08/91 South Texas Unit 2 91-010 See report' 12/24/91 Indian Point Unit 2 92-002 Reactor trip when main feed regulating valve 01/27/92 failed closed. SI because of high steam flow and low temperature.

+

Trojan 92-008 While in hot standby, two main steam isolation 02/23/92 vahes did not open on demand, causing SI on high main steam line differential flow Palo Verde Unit 2 93-001 Steam generator tube rupture, manual reactor 03/14/92 trip, difficulties with EOPs and radiation I

monitors Ft. Calhoun 92-023 A stuck open pressurizer code safety resulted 07/03/92 in SL Operators may have terminated SI v.kh insufficient subcooling margin because of.

Instrument problems

• North Anna Unit 2 92-007 Main steam trip valve failt ' shut, reactor trip, 08/06/92 SI Sequoyah Unit 2 92-011 Reactor trip, SI because of radio transmission 08/21/92 in sensitive plant area Robinson Unit 2 92-017 Loss-of-offsite power, turbine runback, reactor 08/22/92 trip, manual St Palo Verde Unit 2 92-006 Control element assemblies inserted, turbine 11/13/92 trip and reactor trip, bypass valves opening contributes to SI -

Palo Verde Unit 3 93-001 Reactor trip on low steam generator level,-

02/G4/93 AITV cold water injection and bypass vahes.

~,'

opening led to SI McGuire Unit 2 93-008

. Partial loss of-offsite power led to reactor trip 12/27/93 and loss-of-offsite power; cooldown caused SI Events evaluated as having operator difficulty with the control of Si capabihty.

A-1

'+

_m u

---c.