ML20029D054

Human Performance Aspects of Operating Events W/Bypass or Defeat of Esf

ML20029D054
Person / Time
Issue date:	04/21/1994
From:	Kauffman J NRC OFFICE FOR ANALYSIS & EVALUATION OF OPERATIONAL DATA (AEOD)
To:
Shared Package
ML20029D049	List: ML20029D048 ML20029D054
References
AEOD-C94-XX, NUDOCS 9405030339
Download: ML20029D054 (21)

Text

7 o

]

DRAIT - FOR COMMENT April 21,1994 AEOD/C94.xx CASE STUDY REPORT llUMAN PERFORMANCE ASPECTS OF OPERATING EVENTS WITil llYPASS OR DEFEAT OF ENGINEERED SAFETY FEATURES MARCil.'994

,) /)()h 6 k k O30125660 ok i f

Prepared by:

John V. Kauffman Reactor Operations Analysis Branch Office for Analysis and Evaluation of Operational Data U.S. Nuclear Regulatory Commission 9405030339 940415 PDR ORG NEXD PDR

4

(

DRAFT - FOR COMMENT April 21,1994 CONTENTS FO R EWO R D...........

v ABBREVIATIONS

. vii i

INTRODUCTION 1

2 IIACKGROUND 1

3 REVIEW OF OPERATIONAL EVENTS......................

2 3.1 Crystal River U ni t 3......................................... 2 3.2 I;tSalle Unit 2.........

3 3.3 North Anna Unit 2...

4 3.4 Fermi Unit 2 6

3.5 Hatch Unit 1 6

3.6 Oconec Unit 3....

7 3.7 Wolf Creek....

8 3.8 South Texas Unit 2................

8 4

OllSERVATIONS AND LESSONS LEARNED 9

4.1 Design and Control of Engineered Safety Features 9

4.2 Control Room Instrumentation and Control 10 4.3 Tra i n i ng..................................................

10 4.4 Operator Attitudes About Engineered Safety Feature Actuations......

11 4.5 Maintenance, Surveillance, and Testing......

12 4.6 Post-Event Reviews...

12 5

CONCLUSIONS..........

13 6

RECOMMENDATIONS..........

15 7

REFERENCES 16 APPENDICES Appendix A Reactor Trip With Valid Safety Injection Actuation 1

111

a DRAFT-FOR CO51 MENT April 21,1994 k

FOREWORD The Office for Analysis and Evaluation of Operational Data (AEOD) performed a study of operator actions in the control of engineered safety feature (ESF) equipment, and found significant issues that need industry and regulatory attention. The staff found improvements resulting from past industry and regulatory efforts, such as procedural guidance and training for high-pressure injection termination, and concluded that similar efforts would also yield benefits for other practices and other ESFs.

AEOD staff recognizes that the plants in the study recovered from the ESF failures resulting from operator action. If any defeat had continued, the event recovery guidance in emergency operating procedures would likely have caused operators to attempt to restore ESF function. Nevertheless, these events are legitimate precu' sors to more serious events and clearly indicate the need for better operator control of ESFs. The AEOD staff found the need for an integrated effort to improve ESF related operator activities in interrelated areas such as the following:

industry management support operator training, including simulator training emergency operating procedures and abnormal operating procedure analyses (to identify the best advice for procedures)

)

emergency operating procedures and abnormal operating procedure upgrades (to incorporate improvement in procedures)

Two other areas warrant detailed consideration. The first area includes possible design changes to minimize unneeded ESF actuations or undesirable consequences from ESF actuations. Minimizing the number of challenges to operators or the perceived need to intervene quickly during an event should lead to better ESF control. The second area j

includes the investigation of the human performance aspects of events to evaluate thoroughly those conditions for which it is appropriate to bypass or defeat ESFs.

Thorough evaluations will enable licensees to find improvements that could be made in areas such as instruments, procedures, and training.

Our study highlights the need for improvement in ESF control and the need for a new commitment to improve human performance in mitigating accidents.

Gary M. Holahan, Director Division of Safety Programs Office for Analysis and Evaluation of Operational Data i

v l

e DRAFF-FOR COMMENT April 21,1994 AllIIREVIATIONS AEOD Analysis and Evaluation of Operational Data (NRC Office for)

AFW auxiliary feedwater AOP auxiliary oil pump ATWS anticipated transient without scram BWR boiling-water reactor EDG emergency diesel generator EOP emergency operating procedure ERT event review team [ licensee's]

ESF engineered safety feature HPCI high-pressure coolant injection HPI high-pressure injection IN Information Notice LER licensee event report MFW main feedwater MOV motor-operated valve NEl Nuclear Energy Institute NRC U.S. Nuclear Regulatory Commission NSO nuclear station operator NUDOCS Nuclear Documents System PTL pull-to-lock RCS reactor coolant system i

RCIC reactor core isolation cooling RO reactor operator RWCU reactor water cleanup SCSS Sequence Coding and Search System SI safety injection SRO senior reactor operator STA shift technical advisor TMI-2 Three Mile Island Unit 2 TS Technical Specifications vii

4

[:

DRAIT - FOR COMMENT April 21,1994 i

1 INTRODUCTION Appropriate control of engineered safety features (ESFs) is an essential element of reactor safety, as evidenced by the Three Mile Island Unit 2 (TMI-2) and Chernobyl Unit 4 accidents, in which operators defeated ESFs that could have prevented or mitigated the accidents.

The study objectives are to (1) find operating events that involved inappropriate bypass i

or defeat of ESFs from late 1991 to the present, (2) find and evaluate contributing factors, (3) and develop lessons learned for the U.S. Nuclear Regulatory Commission (NRC) and industry to use in improving the control of ESFs.

Current operating U.S nuclear power plants were designed and constructed with considerable defense in depth, including automatic emergency core cooling systems with emergency power supplies, reactor containments with automatic isolation capability, and automatic reactor protection systems. The high reliability, diversity, and redundancy of these and other defenses directly influence the level of reactor safety.

The operating crew can compromise the high reliability, diversity, and redundancy of these defenses by making an error that results in common-cause failure of some portion of these defenses. Operating experience suggests that human intervention may be a dominant failure mode. For example, operators experienced difficulties controlling safety injection (SI) in at least 3 (25 percent) of 12 reactor trips and valid sis since December 1991 (see Appendix). This inappropriate defeat or bypass of ESFs during events is of great interest because the operator may not be able to recover from such errors in complex, stressful situations. All ESF defeats in the study were recovered. It is likely that if any defeat had continued, the event recovery guidance in emergency operating procedures (EOPs) would have caused operators to attempt to restore ESF function.

2 BACKGROUND The TMI-2 accident in 1979 increased the industry's awareness of the importance of control of ESFs and led to many improvements, such as better EOPs and implementation of the shift technical advisor (STA). An event at Crystal River in December 1991 indicated that control of ESFs continued to challenge operators and that the lessons learned from evaluating operating experience might lead to improved operator performance and reactor safety.

The NRC issued Information Notice (IN) 92-47, " Intentional Bypassing of Automatic Actuation of Plant Protective Features," to alert licensees to the importance of formal criteria and training for the limitations on bypassing plant protective features. The NRC's Office for Analysis and Evaluation of Operational Data's (AEOD) staff began this study after observing later events at North Anna Unit 2 and Fermi Unit 2 in which 1

DRAFT - FOR COMMENT April 21,1994 licensee actions to address IN 92-47 were not fully effective in preventing inappropriate defeat of ESFs.

The appropriate control of ESFs is a complicated topic influenced by many variables including design and operating philosophy, training, regulations and other requirements, procedures, regulators, and utility management. Control of ESFs is also complicated by economic and safety considerations such as the need to perform required or beneficial maintenance and testing, minimize personal injury and unnecessmy equipment damage, prevent unwanted actuations or isolations for operational convenience (such as when transferring electrical power supplies or during testing), and minimize the effect of spurious actuations on plant operation and power generation.

An operator controlling an ESF may need to choose among many, possibly conflicting, goals. Ideally, these choices are well planned, and are governed by Technical Specifications (TS), plant operating procedures such as EOPs, and plant administrative control policies and related implementing procedures such as those for equipment tagging, lifted wire and jumpers, surveillance and tests, and general conduct of operations and maintenance.

The AEOD staff reviewed operational events from December 1991 to March 1994 that involved inappropriate bypass or defeat of ESFs. The staff found these events by searching databases, including licensee event reports (LERs) coded in the Sequence Coding and Search System (SCSS), and in the Nuclear Documents System (NUDOCS) text search system. AEOD investigated many of the events on site as part of its human performance study program. The reviewers did not likely find all such events from the databases and expect that licensees did not recognize or report some events.

3 REVIEW OF OPERATIONAL EVENTS 3.1 Crystal River Unit 3 At 3:09 a.m. on December 8,1991, the plant was starting up after a short maintenance outage, when a slow loss of reactor coolant system (RCS) pressure became apparent to the operators (Ref.1). The actuator for the pressurizer spray control valve had failed significantly open but continued to indicate that it was closed. The operators did not realize why the RCS pressure was decreasing until the pressurizer spray line isolation (block) valve was closed about an hour later. The reactor tripped on low pressure. As pressure continued to decrease, a member of the operating crew bypassed automatic ESFs (high pressure injection [HPI), emergency feedwater, emergency diesel generators

[EDGs], and partial containment isolation) actuation for about 6 minutes (the actuation bistables were tripped, indicating an actual demand, while the ESFs were in the bypass condition for about 16 seconds).

A crew member bypassed ESFs before the cause of the decreasing plant pressure was understood. This action was not directed by abnormal or emergency procedures and was 2

DRAFT - FOR COMMENT April 21,1994 not directed by shift supervisors, who did not learn that ESFs were bypassed for several minutes. The operators returned the ESFs to automatic initiation mode when annunciators and " management on shift" alerted shift supervisors to the ESF bypass.

HPI and the other systems then activated. Operators later established manual control of the HPI e stem to maintain RCS pressure above 103.0 bar (1500 psig).

The crew had difficulty with command, control, and communications. The operators failed to use the annunciator response procedure for low RCS pressure. An operator bypassed ESFs without direction or concurrence by shift supervisors and shift supervisors did not learn that an ESF was bypassed for about 6 minutes. The shift supervisor made a late declaration of an unusual event and related notifications. The shift turnover process did not ensure that all crew members were informed of recent observed changes in the behavior of the pressurizer spray valve. During the startup, the " management on shift" (a manager with senior reactor operator (SRO] qualification) helped bring the event to an end by noting that ESFs were bypassed and by recommending that the pressurizer spray isolation valve be closed.

The staff found weaknesses in procedures. The annunciator response procedure for low RCS pressure addressed responses to control circuit faults, but did not cover appropriate actions to diagnose and correct the cause of the pressure decrease. One of the station's abnormal procedures contained such guidance, including directions to close the pressurizer spray line isolation valve. Operators did not execute all applicable steps of that abnormal procedure because ESF termination criteria had been met.

3.2 LaSalle Unit 2 At 8:47 a.m. on April 20,1992, a regenerative heat exchanger relief valve in the reactor water cleanup (RWCU) system lifted while the RWCU system was being shut down (Ref. 2). The resultant RWCU high-differential flow isolation signal was erroneously bypassed by a nuclear station operator (NSO) for 3-% minutes.

Several weeks earlier, an RWCU isolation had occurred because of a spurious RWCU high-differential flow signal. Both RWCU containment isolation valve motors failed when the RWCU system was isolated. Licensee managers criticized the operators for allowing the spurious isolation. The motors were replaced, and a testing program was established to verify motor-operated valve (MOV) limit switch settings as the plant power level increased during subsequent operation.

On April 20, Unit 2 was at 20 percent power. An NSO shut down the RWCU system as part of the testing to verify MOV limit switch settings. This shut down was accomplished by closing the system return valve before stopping the RWCU pumps which was in reverse order of the actions in the procedure substep. About a minute later, RWCU high-differential flow alarmed, indicating the start of a 45-second delay timer preceding i

RWCU isolation. At LaSalle, RWCU high differential flow alarms were common while starting up or shutting down the system.

j 3

DRAFT - FOR COMMENT April 21,1994 The NSO wanted to preserve the test and obtained the shift foreman's permission to bypass the automatic ESF closure of the RWCU containment isolation valves. A second NSO bypassed the RWCU isolation, and reported a continuing RWCU differential flow of about 360 liters per minute (95 gpm).

About 3 minutes later, the operators determined that the RWCU high-differential flow alarm was not spurious. An equipment attendant had reported flow through an RWCU regenerative heat exchanger relief valve, and a third NSO had found the level increasing in the reactor building equipment drain tank. The lead NSO asked the shift control room engineer and the shift foreman how they wanted to isolate the RWCU system.

Both agreed to allow the automatic RWCU isolation although the special test procedure stated that valve operation without thermal overload protection (as was the case with automatic operation) could damage the motor or the valve if the limit switches had drifted because of thermal expansion. The operators returned the RWCU bypass key switch to normal position, allowing the RWCU to automatically isolate, which terminated the loss of inventory from the RWCU through the open relief valve.

A resident inspector performed a followup inspection of activities preceding the bypass of RWCU (Ref. 3). The inspection report documented that RWCU high differential flows were common during reactor heatup and cooldown and that managers decided to bypass spurious RWCU isolation signals. From the time the reactor went critical at about 6 a.m. until about 9:30 a.m., the licensee bypassed the RWCU isolation function four times, the longest period being for about 30 minutes. As shown by the April 20 event, an RWCU relief valve could be open resulting in a loss rate of about 360 liters i

per minute (95 gpm) of reactor coolant from the RWCU system with no other immediate indications other than the " spurious" high differential alarm.

3.3 North Anna Unit 2 At 7:16 a.m. on April 16,1993, a control problem in the main generator voltage regulator actuated a field-forcing (overexcited) generator annunciator (Ref. 4). The unit operator responded by attempting to lower the level of generator excitation. Seventeen seconds after the annunciator was received, a generator differential lockout occurred, causing the main generator, turbine, and reactor to trip.

Control room staff implemented the EOPs for plant recovery. While performing the reactor trip recovery procedure, the control room staff placed the motor-driven auxiliary feedwater (AFW) pump control switches in pull-to-lock (PTL) and closed the steam.

supply valves to the turbine driven AFW pump for about 18 minutes. The procedure reader discovered the system misalignment and inoperability when he reached a later procedure step that returns AFW to a standby configuration. Shift supervisors immediately returned the AFW to a standby, operable configuration. Required heat sink conditions for the reactor were maintained while AFW was inoperable.

The AEOD staff investigated human performance aspects of the event and found that 4

t, DRAFT - FOR COMMENT April 21,1994 AFW was disabled while a valid AFW automatic initiation signal (Lalo Steam Generator Level) was present. Shift supervisors intended for the operator to throttle AFW valves to stop flow, which would have precluded further AFW flow without operator action.

The automatic start function of the AFW pumps was defeated for 18 minutes, unknown to shift supervision, and before meeting the criteria in the applicable EOP.

The licensee's response to IN 92-47 was not effective in ensuring appropriate control of AFW.

When the RCS average temperature decreased below its no load value, the operators were unsure of which EOP step (s) to use to respond. This was after the operators had passed the step dealing with RCS cooldown in Procedure 2-ES-0.1,

" Reactor Trip Response." The operators could no' decide whether to use the

" Response Not obtained" column of a previous step, or to continue in the procedure. Ultimately, the operators interpreted a prior " verify" step as allowing them to switch over from AFW to main feedwater (MFW).

The AFW system was not designed for automatic reopening of the motor-operated discharge valves, or for automatic restart of AFW pumps in PTL. These capabilities would allow the operators to control the system while maintaining its automatic safety function if needed later in an event. EOPs gave guidance to maintain minimum required AFW (throttled) flows, depending on conditions, with steam generator level below the AFW actuation setpoint. Methods to stop or throttle AFW operation below its automatic initiation setpoint would effectively disable the AFW, an ESF system under a valid demand, and necessitate operator action for reinitiation (except for a SI).

The operator who placed the AFW pumps in PTL was concerned about possible pump degradation if the pumps were to run on recirculation flow, as would happen if the pump discharge valves were closed. The operator believed that flow could be restored more quickly and reliably, if needed, by starting the pumps than by opening MOVs.

3.4 Fermi Unit 2 While investigating an event that occurred on August 13,1993 (Ref. 5), the AEOD team learned of a similar event in November 1992. Therefore, the team also interviewed the principal operators in the earlier event to compare operator response between the events.

In this earlier event, a loss of feedwater caused a reactor trip and reactor pressure vessel Level 2 and 3 isolations and actuations. The operators throttled high pressure coolant injection (IIPCI), reactor core isolation cooling (RCIC), control rod drive, and standby 5

DRAFT-FOR COMMENT April 21,1994 l

feedwater flows. A reactor operator (RO) took HPCI out of service after the reactor water level rose above the HPCI actuation setpoint. Operators shut down the HPCI by placing the HPCI auxiliary oil pump (AOP) in ITL after tripping the HPCI turbine.

These actions were included in the HPCI operating procedure for HPCI shutdown.

However, the AOP remained in ITL for about 15 minutes, indicating that the crew did not expeditiously complete the remaining procedure steps to return HFCI to a standby, armed configuration. The procedure does not specify a time limit for completing these

.i later actions.

With the HPCI AOP in ITL, operators would have to reactuate HPCI manually if needed because HPCI is inoperable without the AOP to supply hydraulic fluid during HPCI startup. HPCI is typically designed to trip on Level 8 and automatically reinitiate on 12 vel 2.

A licensed RO stated (Ref. 6) that allowing HPCI to be tripped by the Level 8 signal was not desired. For example, it might appear to indicate poor control of the event by the operators and might contribute to a high cooldown rate. He also stated that it was

" common practice" not to reset HPCI after operators had tripped it and " turned off the oil pump." On the same day, a nuclear training supervisor stated that this was not accepted practice by the training department.

The licensee did not identify defeat of the HPCI while reviewing the event. Therefore, it did not report those aspects of the even'. to the NRC, and missed an opportunity to prevent recurrence of similar ESF defeats at Fermi.

3.5 Hatch Unit 1 i

At 7:33 a.m. on September 30,1992, the reactor was manually tripped followed by a manual trip of the main tt2rbine several seconds later (Ref. 7). HPCI and RCIC automatically actuated. The licensee later found that HPCI did not inject because the reactor water level was restored before the injection valve opened. The licensee had i

quickly restored the level using the "A" reactor feed pump and RCIC, and then used RCIC to maintain the reactor water level.

The resident inspectors observed portions of the licensee's event review team (ERT) investigation of this event, including the critique with the involved operators after the i

reactor trip. The inspectors noted several issues for additional review (discussed below).

Shortly after the scram, the shift supervisor directed that HPCI be "taken off." An operator placed the HPCI AOP switch in the PTL position, which temporarily disabled the HPCI system. The shift supervisor immediately corrected this improper action and returned the pump to an available status. After the scram, the operators stated that the extensive training on anticipated transient without scram (ATWS) scenarios in the simulator may have contributed to this error. As part of the ATWS actions, HPCI injection is " terminated and prevented" by placing the AOP switch in PTL position in 6

x DRAFT - FOR COMMENT April 21,1994' accordance with procedures. This technique is also used to prevent HPCI reinitiation following inadvertent HPCI initiation.

On the afternoon of September 30,1992, control room and refueling floor personnel were advised of apparently inoperable refueling floor ventilation dampers. The licensee halted fuel movement and, on investigation, found that control switches had been improperly placed in the "open" position instead of the " automatic" position at about 1:30 a.m. on September 30,1992, during manipulations of refueling floor ventilation systems. The involved operator used a procedure that required the dampers to be verified open.

The inspectors and ERT concluded that the procedure could easily be misinterpreted to require positioning of the control switches to the open position. The administrative controls for the switches were inadequate to minimize inadvertent or inappropriate bypassing of the safety function of the dampers. In most cases at Hatch, use of a keylock switch is required to override an automatic ESF component by a control room switch.

The inoperable dampers were not immediately identified during the scram recovery because the ESF actuations were quickly reset. Therefore, the licensee did not recognize the failure of the dampers to shut until the post-trip review of safety parameter display system data for the transient.

3.6 Oconee Unit 3 On January 26,1993, the reactor tripped from 100-percent power (Ref. 8). During the trip recovery, while the operators transferred from the emergency feedwater to the MFW pumps, a loss of automatic initiation of both emergency feedwater flow paths resulted when both emergency feedwater control valves were not placed in automatic mode as directed by procedure. The condition was first discovered about 5-% hours later by an RO reviewing the control board.

I The licensee attributed the inappropriate operator action to improperly following the correct procedure. Another contributing factor was the failure of training instructors to clearly emphasize management expectation to check or initial check-off blanks in the Abnormal Procedures as the steps are completed. The licensee.also stated that the correct procedure was improperly followed in a similar previous event. In that event, the HPI system was activated, which violated TS requirements for low-temperature overpressure protection. An HPI pump breaker was racked in, and the discharge valves 1

were not deactivated.

3.7 Wolf Creek On May 8,1993, Wolf Creek entered Mode 3 (Hot Standby) from Mode 4 (Hot Shutdown) with both motor-driven auxiliary feedwater pumps rendered inoperable by their handswitches being in PTL (Ref. 9). This situation remained uncorrected for about 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br />.

7

~

DRAFF - FOR COMMENT April 21,1994 The licensee attributed the root cause to failure of the control room operating crews to control plant work activities such that administrative requirements could be thoroughly considered. The licensee also stated that operators did not follow the administrative procedures for control board walkdowns and shift turnover, in that two shift changes occurred without adequate discussion of the requirements for the motor-driven auxiliary feedwater pumps.

As immediate corrective action, managers clarified their expectations for turnovers, control roo n professionalism, and communications with operating crews. The licensee changed the procedures to render the mode change review process more effective.

3.8 South Texas Unit 2 On December 24,1991, a feedback linkage on the Loop A pressurizer spray valve became disengaged, allowing instrument air to force the spray valve to the open position (Ref.10) Shortly thereafter, the reactor automatically tripped because oflow pressurizer pressure, accompanied by an SI and containment isolation. After the plant was stabilized, the operators left the EOPs. When instrument air was restored to the reactcr containment building, the Loop A pressurizer spray valve failed open again.

Pressurizer pressure dropped below the nominal Si setpoint but an automatic block / reset feature of the Si actuation circuitry prevented the safety logic from sending an automatic SI signal. By design, all of the SI trains were blocked when the reactor trip breakers were open and Si had already actuated.

The licensee determined that the operators did not manually initiate SI because they incorrectly assumed that the EOP criteria for Si continued to apply even though the l

EOP had been exited. About 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> after exiting the EOP, the reactor trip breakers were closed, which automatically reset SI actuation capability.

The licensee later concluded that the plant was in violation of TS 3.3.2 from the time the EOPs were exited to the time when the reactor trip breakers were closed. South Texas TS require two of three Si automatic circuits to be operable in Modes 1 through 4. The licensee attributed the cause of the TS violation to the operators exiting the EOP with i

all SI trains blocked because the operators did not consider that the design feature of the Si block would place the plant in this condition. The licensee revised the EOPs to call for resetting the SI permissive before exiting the procedure, i

4 OllSERVATIONS AND LESSONS LEARNED Events involving inappropriate bypass or defeat of ESF can be of great safety significance as demonstrated by the accidents at Three Mile Island and Chernobyl,

)

Defeat or bypass of ESFs can be a common-mode failure mechanism for these otherwise highly reliable systems, while operator recovery from such errors in complex, stressful situations is highly uncertain.

8

DRAFT - FOR COMMENT April 21,1994 Although many improvements in human performance have enhanced the control of ESFs, certain aspects could be further improved by considering the following lessons i

learned.

4.1 Design and Control of Engineered Safety Features 4.1.1 Most fluid system ESFs are designed to supply adequate flow for design basis events, assuming a single failure. However, the amount of flow needed for such events would exceed that needed for less significant events or when the assumed single failure does not occur. Because many ESFs lack automatic level or flow controls after initiation, many of these "overdesigned" ESFs may have adverse consequences (e.g., full AFW injection may lead to an excessive cooldown) without operator intervention during these less significant events.

4.1.2 "Overdesigned" ESFs contribute to operator intervention during events, which has led to errors that disabled the system (e.g. North Anna Unit 2). This need for operator action could be reduced by design and TS changes to minimize undesirable ESF actuations, such as only starting EDGs on bus undervoltage conditions rather than starting and running EDGs unloaded as now frequently occurs for loss-of-coolant accident or SI signals. Delays in restoring or rearming the system after it is shutdown have disabled the ESF (e.g., Fermi Unit 2).

4.1.3 Instruments that actuate ESFs may be unreliable during certain plant states or i

activities, as demonstrated by the event at LaSalle Unit 2. People tend to ignore or disable nuisances that continually " cry wolf."

4.1.4 If the operator acts to control an ESF during an event, TS requirements may not be met or may not clearly address the situation (e.g., manually throttling AFW after a reactor trip may lead to a loss of required, automatic AFW flow).

4.2 Control Room Instrumentation and Control After the TMI accident, the industry spent much effort such as in detailed control room i

design reviews to improve the man-machine interface, including instrumentation and control, to better support operator performance. Despite these efforts, the events in the study revealed weaknesses in instrumentation and controls.

4.2.1 The operators tely on instruments for decision making that may be inadequate for I

certain events. For example, an open RWCU relief valve at LaSalle caused a 360 liter per minute (95 gpm) loss of reactor coolant that was not recognized by the operators because of prior experience of erratic indication.. The instruments at Crystal River Unit 3 did not reveal the cause of reactor depressurization.

1 4.2.2 In some events, instrumentation and controls for ESF equipment misled operators

.l regarding the status of the equipment. For example, at Hatch Unit 1, the control

]

switches for automatic isolation dampers were in the open position which defeated 1

9

.l J

DRAFT - FOR COMMENT April 21,1994 the automatic safety function of the dampers, without sufficient visual or audible cues to alert the operators to this defeat. At South Texas Unit 2, the operators did not consider the automatic block feature of the Si logic before exiting the EOP, apparently because of insufficient cuing that SI was automatically blocked.

4.2.3 Instruments and annunciators did not sufficiently alert operators to misaligned ESFs at North Anna Unit 2, Hatch Unit 1, and Oconee Unit 3.

4.3 Training The training that operators receive is obviously a powerful influence on their response to.

events. Simulator training is particularly important to the operator's " mental modeling" of systems, their responses, and their interfaces with other sy. stems. In addition, the simulator is where operators demonstrate and practice their responses to transient and accident conditions. This may well be their predominant experience base for responding to such events. Operator performance in real events, therefore, offers insights into the associated training program. Similarly, simulator training sessions provide opportunities to better support operator performance, for example, by identifying procedural or instrumentation enhancements.

4.3.1 Training operators to take specific actions, such as to disable HPCI by placing its AOP in PTL for boiling water reactor (BWR) ATWS events, during particular types of events has contributed to similar (inappropriate) operator actions in other situations (e.g., Hatch Unit 1). These negative transfers of training illustrate the human tendency to revert to successful prior behavior patterns during a stressful situation.

4.3.2 Inappropriate operator actions involving the bypass or defeat of an ESF during simulator training have remained uncorrected in some cases, and may have j

contributed to similar inappropriate ESF defeat during events at North Anna Unit 2 and Fermi Unit 2. The switch from AFW to MFW has been particularly problematic, as shown by events at Oconee Unit 3, North Anna Unit 2, and Palo Verde Unit 3 (Ref.11).

4.3.3 Weaknesses during ESF bypass events suggest areas where training may be improved.

Control room communications: Informal terms at Hatch Unit 1, lack of repeat-backs or acknowledgements at North Anna Unit 2, and lack of feedback regarding completion of actions at Crystal River Unit 3.

Procedural usage: Procedures not used at Crystal River Unit 3 and LaSalle Unit 2, poorly executed at North Anna Unit 2, and misinterpreted at 12Salle Unit 2.

10

DRAFT - FOR COMMENT April 21,1994 Supervisory watch standing practices and team-building: shift supervisors (including the STA) unaware of operator bypass of ESF at Oconee Unit 3, North Anna Unit 2, and Crystal River Unit 3; lack of supervisory response to annunciators and other indications of bypassed ESF at North Anna Unit 2);

inappropriate defeat of an ESF at the direction of the SRO/STA at LaSalle Unit 2, and weaknesses in communications and procedural usage described above.

Verification of successful ESF operation, and completion of ESF safety function before it is reset: at Hatch Unit 1, operators did not verify that ESF functioned before reset and did not manually initiate equipment that should have automatically initiated. At South Texas Unit 2, operators did not manually initiate SI because they incorrectly assumed that the EOP criteria for SI termination continued to apply even though the EOP was exited earlier.

Plant policies and procedures (including EOPs) for control of ESFs: weaknesses in understanding policies and procedures for control of ESFs at LaSalle Unit 2, North Anna Unit 2, and South Texas Unit 2. Operators stated in interviews the training was passive, not active. For example, in a simulator session, if an operator was directed to " secure system 'A'" and performed this action

" correctly" by shutting valves in the system, he would not be informed by the instructor. Unless he performed unacceptable actions and was corrected, he might finish the session with the erroneous idea that other (inappropriate) ways of

" securing" the system, such as by placing pumps in PTL, were also acceptable.

4.4 Operator Attitudes About Engineered Safety Feature Actuations Factors that contribute to operators' views concerning ESF actuations include the operators' training, licensee management and NRC performance indicators and precedents (what is " counted" and what has been rewarded or punished in the past), peer pressure, individual pride, the reliability of the ESFs, and whether an ESF actuation benefits the particular situation.

.j l

4.4.1 Many factors contribute to the belief among some operators that ESF actuations are " bad" and should be avoided or minimized where possible. For example, ESF actuations are reportable, may be included in the performance indicators for the j

plant, may " unnecessarily" complicate an event, may have adverse consequences (e.g., may contribute to excessive cooldown), or may interrupt normal support system operation such as by isolating BWR shutdown cooling.

4.4.2 Licensee trainers and managers and NRC personnel frequently evaluate operators' responses in real events and simulator scenarios. Allowing an ESF I

injection system to automatically trip on high level, for example, may be considered as poor operator control or performance. Similarly, quick recovery and restoration of lost functions may be considered good performance and may be needed because nuclear power plants are complex with highly interactive systems.

Plant managers at LaSalle criticized operators for not defeating an ESF after an 11

~

DRAFT-FOR COMMENT April 21,1994 event where a valid ESF actuation resulted in equipment damage because equipment was set up improperly.

4.4.3 Based on the above, plant operators may perceive the need to intervene quickly during events or rapidly changing conditions. However, both the decisions made and the actions taken during this early intervention are prone to human error.

Therefore, countries such as France and Germany have designed systems with automated controls such that operator action is not needed and is discouraged (except to initiate failed automatic features or protect equipment or personnel) for periods of 10 to 30 minutes after a reactor trip.

4.5 Maintenance, Surveillance, and Testing 4.5.1 Errors during maintenance, surveillance, and testing have disabled automatic safety features at Wolf Creek and Hatch Unit 1.

4.5.2 Testing of automatic safety systems frequently requires defeat of the system. This has contributed to errors. For example, operators at LaSalle Unit 2, routinely defeated an ESF (RWCU isolation) for " testing" when encountering perceived equipment or instrument problems. When reactor coolant was lost outside of containment, the crew had been conditioned that defeat of the ESF was allowable, j

and defeated it. In this case, "living with" a problem introduced poor operating practices that later manifested themselves in improper performance during a loss-of-coolant event.

4.6 Post-Event Reviews When inappropriate bypass-of-ESF events are not recognized, they are not reported as such. When this occurs, the plant loses an opportunity to improve safety performance, and the industry is deprived of an opportunity to learn, and may be led to a false sense of security that such events do not happen or are very rare.

4.6.1 During or after AEOD site visits, the staff found five cases where' the licensee had not discovered or recognized an inappropriately bypassed ESF: the Fermi Unit 2,-

November 1992 event, and four defeats of RWCU isolation at LaSalle 2. These inappropriate operator actions involving bypass or defeat of an ESF were not licensee identified or corrected.

4.6.2 Because some ESF bypass events were not recognized or reported, such events are more prevalent than indicated by the operating experience data bases.

4.7 Technical Specifications 12

o DRAFT - FOR COMMENT April 21,1994 The AEOD staff studied the written guidance (see 4.3) for operator response to the situation or event, and other associated requirements such as TS.

TS typically address the operability of equipment before an event better than they do operability afterward. TS typically do not address well the need for the operator to control or inhibh ESFs during scrams and transients. These actions may be allowed or directed by EOPs but contrary to TS. Licensees have recognized similar situations and handled them in a sariety of ways including making licensing interpretations, invoking 10 CFR 50.54 (x) a, d (y), or obtaining prior NRC approval of EOP actions in the safety evaluation of the licercec's EOP submittal. Other situations, however, may not have been recognized or may have been resolved in a manner that appears illogical or contradictory. For example, Typical BWR TS require HPCI and RCIC to be operable in Modes 1,2, and 3. If operators take manual control of both systems after a trip, a strict interpretation could be that HPCI and RCIC were inoperable because they were incapable of performing their intended function (injecting rated flow) in the intended manner (automatically).

At North Anna Unit 2, the AFW function was disabled by placing the motor-driven pumps in PTL mode and isolating the turbine-driven pump steam supply while a valid AFW automatic initiation signal was present. Shift supervisors intended for the operator to throttle AFW valves to stop flow, which would have precluded further AFW flow without operator action. The licensee considered this intended action as acceptable

" operator control" of the system.

5 CONCLUSIONS The AEOD staff reviewed recent operating experience associated with this issue and offers the following conclusions:

5.1 Plants continue to rely on operators to control safety systems even though the measures taken to ensure appropriate control have weaknesses that have been i

demonstrated by operating experience.

5.2 Many individual corrective actions taken after TMI-2 have not been fully effective or have not been effectively implemented throughout the industry. These

)

conclusions include observations of instrumentation and control deficiencies which TMI-2 actions did not address explicitly, but may be inferred to have been addressed as part of detailed control room design reviews or Regulatory Guide 1.97 efforts.

Engineering expertise on shift (STA or SRO): ESFs were inappropriately defeated without the knowledge of the STA and SRO, who did not learn of them for some time. In one example, ESFs were inappropriately defeated at the direction of the SRO/STA. Another operating crew did not 13 j

+

DRAFT - FOR COMMENT April 21,1994 manually initiate Si because they incorrectly assumed that the EOP criteria for SI termination continued to apply even though the EOP was exited earlier (see 4.3).

J Operating experience review programs: The licensee inappropriately defeated ESFs after receiving a generic communication on the control of ESFs. Some licensees took insufficient actions in response to the generic communication. However, the generic communication may not have sufficiently highlighted and detailed the concerns on this topic.

Instruments: Operators had no direct indication of open valves that caused either a loss of reactor coolant or a loss of primary system pressure. In one example, an ESF was bypassed when instruments were not viewed as reliable for existing plant or system conditions (see 4.2).

Alarms: Annunciators did not sufficiently alert other operators and supervisors to the defeat or bypass of ESFs. In one example, an ESF failed to respond during an event, and the operators were not alerted. In i

another example, the operators did not consider the automatic block feature of the SI apparently because the Si automatic block was not sufficiently cued (see 4.2).

Controls: Poor control switch design led to a situation where an ESF was inadvertently defeated. The switch design did not visually indicate that the ESF was disabled. The system was not designed to provide an alarm when misaligned (see 4.-2).

5.3 Aspects of some other regulatory programs were not fully effective or were not effectively implemented throughout the industry.

Event reviews: A licensee conducting a post-trip review as called for in.

l Generic Letter 83-28, failed to identify inappropriate defeat of an ESF.

Inappropriate ESF defeats at another licensee were not identified until q

questioning by the NRC (see 4.6).

q Operator training: In many instances, licensed operators took inappropriate actions, did not use procedures appropriately, did not effectively supervise or communicate, did not investigate alarms, had difficulty implementing EOPs, or did not ensure that automatic safety functions were performed (see 4.3).

Technical Specifications: The TS for certain plants do not address well the need for the operator to control or inhibit ESFs during events. This ambiguity has caused confusion when actions may be allowed or directed by EOPs but may be contrary to TS. Also, operator response to events 14 1

DRAIT-FOR COMMENT April 21,1994

)

may be impeded because the licensee is required to invoke 10 CFR 50.54 (x) and make related notifications during an event.

6 RECOMMENDATIONS The case study clearly indicates that ESF reliability requires appropriate and reliable operator action and that operators have experienced numerous problems in controlling ESFs.

Problems with human intervention in ESF action continue because the industry has not established an integrated, comprehensive program addressing system design, operator guidance, training, and management involvement to optimize ESF reliability.

6.1 The NRC's Office of Nuclear Reactor Regulation and the nuclear industry (through the Nuclear Energy Institute [NEI] and the Owners Groups' committees on EOPs) should initiate an integrated and comprehensive evaluation of efforts to establish high level licensee management policies and guidance on ESF operations, comrol, and termination establish detailed operator guidance and training on appropriate means to control and terminate ESFs l

establish clear guidelines on the relation of TSs to the operation and l

control of ESF systems make better use of automation for ESF logic and controls. In particular, the focus should be on minimizing the burden on the operator, for example by not starting systems unnecessarily and by adding simple level controls such that the need for operator intervention is reduced where feasible.

6.2 NRC and licensees should carefully evaluate licensees' administrative controls and training to control ESFs. They should evaluate the effectiveness of the controls and training by scrutinizing operating experience; (i.e., post-event evaluation of operator responses to actual plant operating events, associated written guidance, i

and training). They should also perform root cause analyses to identify and correct contributing specific and programmatic weaknesses.

15

4-DRAFT-FOR COMMENT April 21,1994 7

REFERENCES 1.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to T. M. Novak, " Human Performance Study Report - Crystal River Unit 3 (12/08/91)," January 30,1993.

2.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to T. M. Novak, " Human Performance Study Report - LaSalle Unit 2 (4/20/92),"

June 16,1993.

3.

U.S. Nuclear Regulatory Commission, inspection Report No. 50-374/92-011, LaSalle Unit 2, June 1,1992.

4 J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to G. M. Holahan, " Human Performance Study Report - North Anna Unit 2 (4/16/93)," June 23,1993.

5.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to G. M. Holahan, " Human Performance Study Report - Fermi Unit 2 (8/13/93),"

August 13,1993.

6.

R. A. Spence, U.S. Nuclear Regulatory Commission, memorandum to J. V. Kauffman, "HPCI Operation at Fermi Unit 2," December 1,1993.

7.

U.S. Nuclear Regulatory Commission, Inspection Reports No. 50-321 and 366/92-22 Edwin I. Hatch, November 6,1992.

8.

Duke Power Company, Licensee Event Report 287/93-001, Oconec Unit 3.

February 25,1993.

9.

Wolf Creek Nuclear Operating Corporation, Licensee Event Report 482/93 010, Wolf Creek, June 2,1993, 10.

Houston Lighting and Power Company, Licensee Event Report 499/91010-01, South Texas Unit 2, September 23,1992, 11.

J. E. Rosenthal, U.S. Nuclear Regulatory Commission, memorandum to D. F. Ross, "Iluman Performance Study Report - Palo Verde Unit 3 (2/4/93),"

April 19,1993.

16

f a

o DRAFT - FOR COMMENT April 21,1994 APPENDIX A Reactor Trip With Valid Safety Injection Actuation December 1991 to Present Plant /Datt Licensee Event Comments Report No.

Crystal River Unit 3 91-018 See report' 12/08/91 South Texas Unit 2 91-010 See report

• 12/24/91 Indian Point Unit 2 92-002 Reactor trip when. main feed regulating vahe 01/27/92 failed closed. Si because of high steam flow and low temperature Trojan 92-008 While in hot standby, two main steam isolation 02/23/92 valves did not open on demand, causing Si on high main steam line differential flow Palo Verde Unit 2 93-001 Steam generator tube rupture, manual reactor 03/14/92 trip, difficulties with EOPs and radiation monitors Ft. Calhoun 92-023 A stuck open pressurizer code safety resulted 07/03/92 in SI. Operators may have terminated S1 with insufficient subcooling margin because of

)

instrument problems

• North Anna Unit 2 92-007 Main steam trip valve failed shut, reactor trip, 08/06/92 S1 Sequoyah Unit 2 92 011 Reactor trip, Si because of radio trar.smission 08/21/92 in sensitive plant area Robinson Unit 2 92-017 less-of-offsite power, turbine runback, reactor j

08/22/92 trip, manual Si Palo Verde Unit 2 92-006 Control element assemblies inserted, turbine 11/13/92 trip and reactor trip, bypass valves opening j

contributes to Si Palo Verde Unit 3 93-001 Reactor trip on low steam generator level, 02/M/93 AFW cold water injection and bypass valves -

opening led to St McGuire Unit 2 93-008 Partial loss-of-offsite power led to reactor trip 12/27/93 and loss-of offsite power; cooldown caused Si

&cnts evaluated as having operator difficulty with the control of Si capabihty.

A-1