ML20070S234
| ML20070S234 | |
| Person / Time | |
|---|---|
| Site: | Sequoyah  |
| Issue date: | 02/18/1994 |
| From: | Jeffery Lynch SCIENCE & ENGINEERING ASSOCIATES, INC. |
| To: | Eckenrode R Office of Nuclear Reactor Regulation |
| Shared Package | |
| ML20070S213 | List: |
| References | |
| CON-FIN-J-2014 NUDOCS 9405230291 | |
| Download: ML20070S234 (8) | |
Text
'
~
c}
N Science A AimV Engineering i.
Associates. Inc.
y s n.va I a.
,,3 8 of Ni v.na.eq+ s.v. w a o n tni
- l*.
P:W.,$ ;i h e February 18,1994 U.S. Nuclear Regulatory Commission Office of Nuclear Reactor Regulation Human Factors Assessment Branch Attn: Richard J. Eckenrode Mail Stop OWFN 10 D24 Washington,DC 20555 r
Subj: Contract NRC43-93-036 Task 7 Subtask 7 Sequoyah Event Investigation Report Data under FIN J2014 (fee recoverable)
Dear Mr. Eckenrode:
The final Sequoyah Event Investigation Report detailing infonnation and findings from the site visit conducted on September 15-16,1993 is forwarded.
Sincerely yours, SCIENCE & ENGINEERING ASSOCIATES,INC.
i(.
f-arr
- 1QPr et Manager Enclosure i
i 9405230291 94051g yDR ADOCK 05000327 PDR i
4 Memorandum To:
Jim Lynch Science and Engineering Associates, Inc.
From:
Valerie Barnes, Linda Radford Performance, Safety and Health Associates, Inc.
Date:
February 11, 1993 Re:
Sequoyah Event Investigation Trip Report
Background
This trip report describes the activities and findings of a site visit to Sequoyah Nuclear Plant (SQN) conducted on September 15-16,1993 to investigate the human performance aspects of the December 31,1992 dual-unit trip. The focus of the investigation was on understanding the root cause(s) and contributory factors of the human errors that occurred in the course of the event, with particular emphasis on the Unit 2 operators' decision to use the normal boration method when the relevant emergency operating procedure (AOI-34, Emergency Boration, Revision 7) required emergency boration. The licensee's response to the human performance deficiencies during the event and the effectiveness of the corrective actions also were of interest. The functioning of the plant equipment and hardware systems over the course of the event is documented in detail elsewhere, and so is not repeated here.
Methods The methods used in this brief investigation consisted of document reviews, interviews with plant personnel and observation of two scenarios in the SQN training simulator. Appendix A lists the documents reviewed. The licensee staff interviewed primarily were those who had been directly involved in the event and were present in the control room as it evolved.
Findings It appeared to the investigators that the root cause of the human performance deficiencies during the event was an interaction of inadequate shift staffing and less than adequate operator training. However, there also were several other factors that appear to have contributed to the operators' responses. These factors include high levels of noise and the impact of the noise on the operators' ability to communicate, the design of the conttol room, 1
l i
operator experience levels, and attributes of the SQN procedures program. These root and contributory factors are discussed below, Shift staff' g: To understand the human performance aspects of the December 31,1992 m
event, it is important to understand the configuration of the SQN control rooms for Units 1 and 2 and their shift staffing practices at the time of the event. The control room can be described as configured in the shape of a barbell, with the bulk of the control panels for each j
unit at the ends of the barbells in the typical horseshoe configuration. On the walls of the
" waist" of the barbell are controls and displays for common systems between the two units, as well as the interfaces for several systems associated with the individual units. Also in the waist of the barbell are raised platforms for the workstations of the Shift Operations Supervisor (SOS) and the two Assistant Shift Operations Supervisors (ASOSs) assigned to each unit. The ASOS workstations face into the horseshoe areas of their respective units.
The distance from the SOS's desk to the horseshoe area for either unit is approximately 30'.
On-shift control room staffing is typically one SOS for both units, one STA (holding an
)
SRO) for both units, one ASOS assigned to each unit, a support ASOS for both units who is not required to hold an SRO license, and two ROs for each unit, for a total of nine licensed operators per shift. The SOS and unit ASOSs are required to hold Senior Reactor Operator (SRO) licenses while the operators in the Control Room Operator (CRO) and Operator at the Controls (OAC) roles are expected to hold Reactor Operator (RO) licenses. Only three ROs per shift are required by the SQN technical specif cations (TS), rather than the two per unit (for a total of four) that SQN has attempted to schedule.
SQN has experienced difficulty in consistently staffing four RO positions per shift. In the two years prior to the December 31 event, the licensee was subject to budget cuts that led to a major reduction in force. As a consequence, one full class of Assistant Unit Operators (AUOs - non-licensed operators) in training for RO licensing was cut. In addition, SQN has had difficulty recruiting AUOs to enter the licensing program. One interviewee indicated that obtaining an RO license is unattractive because of the rigorous preliminary testing required of applicants for the program and the high level of responsibility that ROs must carry. Plant managers interviewed indicated that they believe that potential reemits are discouraged from seeking the license and a career in nuclear operations because of the uncertain future of the commercial nuclear power industry in the United States. Further, the plant managers and other interviewees noted that ROs are needed in support positions, such as the procedures group, and will seek opportunities to work in these other areas rather than staying on-shift because of the disadvantages of shift work. As a result of the limited pool of ROs and RO complaints about excessive forced overtime, SQN had developed a practice of allowing AUOs in training to serve in the CRO position during normal operations, under the supervision of a licensed operator, prior to the event.
On the night of the December 31 event, the RO scheduled for Unit 2 called in sick, so an AUO was assigned to the CRO role for that shift. Consequently, the shift complement for that night consisted of an SOS, a support ASOS who was assigned to work with the staff in 2
the switchyard, another support ASOS who was responsible for the STA role in an emergency, the two unit ASOSs, two ROs in Unit I and an RO and an AUO in Unit 2.
Because the AUO assigned to the CRO role for Unit 2 was unlicensed, however, he was unable to manipulate controls during the event.
Interviews with the SOS, the Unit 2 ASOS, and the Unit 2 RO and AUO on-shift that night indicated that the substitution of the AUO for the scheduled RO and the limitations on the AUO's ability to take actions played an important role in the errors that occurred.
Specifically, the Unit 2 crew was delayed in taking manual control of the auxiliary feedwater system (AFW) after the reactor trip because of the need to perform other recovery actions.
This delay allowed an RCS cooldown to occur, creating the need for emergency boration.
Because the ASOS was concerned that the RO could not simultaneously monitor the emergency boration process and take manual control of the AFW system (controls fer the two systems are approximately 10' apart on the panels in the horseshoe), he allowed the RO 1
to use the normal boration process, which further complicated the recovery.
l It appeared to the investigators that the SQN practice of assigning only one SOS and one
]
STA for both units also contributed to the errors that occurred during the dual-unit trip. For j
example, the SOS stated that he was distracted from supervising the Unit 2 operations by supervising Unit 1, answering incoming phone calls, performing the emergency plan implementing procedure and making the required notifications. Although he maintained awareness of the activities in Unit 2, he was unable to provide the close supervision of the crews' actions and decisions that are practiced in the simulator. The Unit 2 ASOS indicated that greater availability of the SOS for the type of consultation and supervision he was trained to expect might have prevented the boration error, if the SOS had been there to monitor his decisions and actions. The ASOS also stated that the limited availability of the SOS increased his stress level from a sense of increased individual responsibility, which may have affected his judgment over the course of the event. In addition, the use of only one STA for both units meant that performance of the status trees for Unit 2 was intermittent and that the STA also was of limited availability for consultation with the Unit 2 ASOS. It appeared to the investigators that independent supervision of the unit operators' actions is compromised during a dual-unit trip at SQN by staffing the shift with only one SOS and one STA.
Training: As implied in the foregoing, weaknesses in training also appeared to contribute to the human performance problems during the event. The weaknesses include deficiencies in the training simulator and in the curriculum.
The SQN training simulator simulates only one unit. Consequently, the crew does not have the opportunity to experience and practice responses to a dual-unit trip. For example, an SOS cannot practice dividing his time and attention between the two units or making staff allocation decisions in a dual-unit trip scenario. Operators cannot practice coord' mating actions on the common equipment in the " waist" of the barbell, nor performing the actions that pertain to the other unit on the panels at "their" end of the central corridor. In addition, noise levels from all of the alarms that are activated during a dual-unit trip cannot be 3
experienced. Mock ups or other methods to create dual-unit training scenarios were not used because dual-unit events were considered to be of very low probability.
Further, prior to the December 31 event, operators were not trained with the minimum shift composition (i.e., one RO on a unit) allowed by TS and dual-unit trip scenarios were not included in the training curriculum because of the simulator limitations. The staffing configuration used for simulator training is an SOS, an ASOS for each unit, a control room operator (CRO) who is primarily responsible for the secondary side of the assigned unit during transients and an Operator at the Controls (OAC) who is responsible for primary side operations during emergencies. A trainer typically plays the STA role.
One consequence of being unable to practice dual-unit trip responses or responding with minimum shift staffing is that the crew is unable to learn to cope with the added stresses associated with a dual-unit trip or the workload associated with having only one RO on a unit. As a result, the crews do not have the opportunity to develop strategies for performing their tasks under those conditions. For example, immediately after the units tripped, the SOS sent the support ASOS out of the control room to assess the cause of the trips and to supervise any recovery actions required in the switchyar, o in the plant. If the SOS had the opportunity to practice a dual-unit trip, particularly wit ' mimum staffing, he might have been aware that the support ASOS's resources could have been better used by assigning him the CRO role in Unit 2.
The investigators noted other limitations in the realism of the two scenarios that were i
observed in the training simulator during the site visit. For example, the ceiling of the simulator is much higher than that of the control room to allow for the visitors' gallery windows so that lighting conditions differ between the simulator and control room. As the simulator lighting is configured now, there is no possibility of simulating the lighting changes j
that occur during a station blackout. Noise levels in the simulator also are much lower than in the control room because there is no HVAC background noise simulated nor the capability to simulate high levels of alarm noise during a dual-unit event or the sounds of the control room printers. Sounds from the turbine building that were audible in the control room during the December 31 event also cannot be simulated for training purposes. In addition, the scenarios observed did not include multiple, insistent phones ringing at the SOS's desk as was reported to occur during events by the interviewees. Again, it appeared to the investigators that as a result of this lack of realism, the operators were not prepared for the lighting changes and had not developed strategies for communicating through the noise in the control room during the initial minutes of the December 31 event. Further, a lack of practice with responding to multiple telephone calls while also being required to supervise the operators' responses to the trip on each unit and perform the emergency plan event classification and notifications may have set the stage for the difficulties the SOS experienced on the night of December 31.
Research as well as experience in nuclear power and related industries has shown that repeated exposure to stressful conditions and practice with responding to them, until responses become nearly automatic (i.e., overlearned), is a successful method of assisting persons to overcome maladaptive stress responses, such as an inability to concentrate on a 4
task or the tendency to focus in on one aspect of the situation to the exclusion of others (i.e.,
s tunnel vision). It is likely that the Unit 2 crew's relative inexperience with dual-unit trips and with responding to emergency events with minimum shift staffing contributed to the judgment errors that occurred during the December 31 event.
Other Performance Shaping Factors: In addition to the shift staffing and training deficiencies, other performance shaping factors seemed to have affected the crews' responses at various points of time in the course of the event. These additional factors included the control room design, high levels of noise, the assumptions underlying the design of the procedures and operator experience levels.
The size of the control room made it difficult for the SOS to maintain command and control of both units simultaneously, particularly with the high levels of alarm noise during the initial minutes of the event. For example, the Unit 2 ASOS stated that, although he and the Unit 2 RO could communicate at the boards by shouting, he could only see the SOS's mouth moving, but could not hear his words from the horseshoe. However, even without the noise, it would be difficult for the SOS to closely monitor two crews' actions simultaneously from 30' away: If the SOS moved closer to one of the units, he would necessarily be out of touch for a time with the crew in the other unit.
Another issue that may have contributed to the difficulties the operators experienced is the assumption that two ROs would be available on a unit to perform the accident mitigation procedures. A discussion with the SQN Emergency Operating Procedures (EOPs)
Coordinator revealed that when validating the 1991 revision to the EOPs in the simulator, only the reactor trip procedure was validated with one RO at the boards. The remainder of the EOPs were validated with two RO's, based on the assumption that dual unit trips are highly unlikely and the third RO required by TS would be available to support the accident unit.
l Scenarios based on the December 31 event have since been run in the simulator with only one RO, and the EOP Coordinator reponed that the EOPs and abnormal operating procedures that were used during the event could be perfonned " comfortably" there.
However, as noted above, the simulator cannot provide a realistic simulation of the December 31 event or other dual-unit trips. Consequently, it is unclear that the simulator scenarios used to validate the EOPs have assured that they are usable under dual-unit trip /TS staffing conditions.
A second aspect of the event that may be attributable to the procedures program is that the thermal barrier booster pumps (TBBPs) failed to start when the shutdown boards reloaded.
An investigation by SQN personnel found that the handswitches for the TBBPs were mis-positioned prior to the event because of an error in the applicable operating procedure. As is the case for most U.S. plants, normal and abnormal operating and maintenance procedures are not as thoroughly verified and validated as are EOPs at SQN, with the result that technical inaccuracies in these categories of procedures are more common than in EOPs.
5
d Finally, differences in the experience levels of the operators assigned to the two units on the night of the event also may have contributed to the Unit 2 crew's dif0culties. Interviews with the Unit 1 ASOS and RO in the OAC role during the event revealed that these two individuals have worked closely together for many years. Further, the Unit 1 RO in the OAC role had been licensed for over ten years and is an experienced pilot, which may mean that he has had extensive practice at performing tasks under stressful conditions. In contrast, the Unit 2 ASOS and RO did not have a long history of working together and the RO had been licensed only for a little over three years. Because of the differences in their team histories, it is likely that the Unit I crew had developed comfortable, predictable and familiar methods of communicating and working together, whereas the Unit 2 crew may not have yet developed familiar patterns that would enhance their performance as a team during the stressful conditions of the event.
In addition, the experience level of the Unit 2 RO was reported by the Unit 2 ASOS to have strongly influenced his approach to handling the event. He told the investigators that he was aware that the RO was comparatively inexperienced, and that fact, in combination with the lack of another RO in the CRO role, caused him to be concerned with carefully managing the RO's workload and to attempt to provide closer supervision and support to the RO than he would have with a more experienced operator. In his self-analysis of his performance during the event, the ASOS attributed part of the rationale for his decision to allow the use of normal boration rather than emergency to this factor. He also stated that the RO's relative inexperience, combined with the lack of a licensed CRO and the limited support he had available from the SOS significantly increased his stress level during the event.
Findings Regarding the Licensee's Response to the Event It appeared to the investigators that the licensee's analysis of and response to the event may not have reflected an adequate understanding of the complexity of the event from a human performance perspective. The analysis and subsequent management actions in response to the event seemed to focus narrowly on the individuals involved rather than taking a systems perspective. As a result, it is unclear that the corrective actions taken by the licensee
~
provide assurance that similar human performance deficiencies will be avoided in future events.
Management actions that focused on the operators involved in the event included l
several weeks of interviews with groups of plant managers in which the SOS, Unit 2 ASOS and RO were required repeatedly to recall their actions and thoughts during the course of the event, l
removing the SOS and the Unit 2 ASOS and RO from shift work and requiring them to develop a briefing to present to each shift detailing the lessons to be learned from the event, poor performance ratings for the SOS and the ASOS that resulted in them being e
permanently taken off shift work and reassigned to other duties.
6 i
m - j
l4 l
-1 Other corrective actions were e
a commitment to the NRC to maintain four ROs on each shift, a one time addition of a scenario to the operator re-qualification training program that e
is similar to the December 31 event, in that it required the shift to respond with only one RO in the unit, increased management emphasis on procedural adherence and self-checking, e
reinforced by disciplinary action for individuals who commit errors or fall to follow procedures.
l The plant managers interviewed are relatively new to their positions and were either hired or promoted after the December 31 event. The Operations Manager and Plant Manager both indicated that they believe the actions of the operators during the event were symptomatic of a lack of discipline among SQN personnel and that they were hired with a mandate from upper TVA management to change the " lax operational culture" at SQN, as they described it.
The Operations Manager, in particular, indicated that he is intent on creating a sense of individual responsibility and accountability for formality of operations in his department.
It is unclear to the investigators, however, that these corrective actions are adequate to prevent a reoccurrence of the types of human performance problems that occurred in Unit 2 on December 31. For example, although SQN has committed to maintaining four ROs per shift, they continue to experience difficulty in recmiting RO trainees from the AUO ranks and have been barred by the operators' union from hiring experienced operators from outside TVA to train for SQN licensing. They have hired and are training SRO-level personnel from Trojan, but plant managers did not describe a long-term plan for enhancing their recruiting program for ROs. Second, neither the plant managers nor training personnel indicated that any plans are underway to revise the training program pennanently to include scenarios for minimum shift staffing, to redesign the simulator in order to be able to model dual-unit trips or to develop other methods for simulating the dual unit trip experience. Further, there have
_~
been no efforts and none are planned to re-validate the EOP set under minimum shift staffing conditions. And, finally, the workload associated with the job tasks of the SOS position during a dual-unit trip has not been evaluated or the job tasks redesigned. As a result, for example, administrative controls over telephone calls into the SOS from non-critical personnel during an event have not been considered.
7
_