ML20043D673

Trip Rept of Onsite Analysis of Human Factors of Event at Catawba Unit 1 on 900320 (Overpressurization of RHR Sys)

ML20043D673
Person / Time
Site:	Catawba
Issue date:	05/31/1990
From:	Harbour J, Meyer O EG&G IDAHO, INC., IDAHO NATIONAL ENGINEERING & ENVIRONMENTAL LABORATORY
To:	NRC
Shared Package
ML20043D667	List: ML20043D666 ML20043D673
References
NUDOCS 9006110050
Download: ML20043D673 (25)

Text

...

,,.o a_4 6

l 1

j 1

TRIP REPORT

)

Jerry L. Harbour Orville R. Meyer On. Site Analysis of the Human Factors of an Event at Catawba Unit 1 on March 20, 1990 i

(0verpressurization of RHR System)

Investigative Team f

Jerry Harbour, INEL/EGSG Idaho

• George Lanik, NRC/AE00 Orville Meyer, INEL/EGLG Idaho Ann Ramey Smith, NRC/RES

• Team Leader May 1990 INEL/EG&G Idaho, Inc.

9006110050 900531 i

cDR ADOCK 050C0413

.k.

PDC

~

n 1

) '

ACKNOWLEDGMENTS 4

Appreciation is expressed for the cooperation of the Catawba Station staff, in particular the Unit I control room operators who were on duty during the day shift on March 20 and who freely provided information 3

concerning their observations, thinking, and actions.

The acquisition of on site information was also greatly facilitated by Curtis Rapp, Reactor Engineer, Region 11, William Orders Senior Resident, Region 11, and Mark lesser, Resident inspector, Region 11.

Thanks are also due to Steve Johnson, certified PWR Operator Examiner, for assistance in operations

'>1ysis-and to Elaine Beardall for her efficient text processing.

l I

I l

l L

l 1

l i

i

-p

1 EXECUTIVE

SUMMARY

On March 20, 1990, at about 0930, Catawba Station Unit 1 experienced an overpressurization of the Residual Heat Removal System (RHR) and Reactor Coolant System (RCS) during the procedure to initially pressurize the RCS to 100 psig following a refueling outage.

As part of the AE00 program to investigate the human factors aspects of operational events, a team was sent to the site.

The team leader was George Lanik of AE00; other team members were Ann Ramey-Smith from the Human factors Branch, i

RES, and Orville Meyer and Dr. Jerry Harbour from the Idaho Natioaal Engineering Laboratory (INEL).

The team spent the day of March 27 at the site and gathered data from discussions, plant logs, and interviews with control room operators and other station staff.

The team also had access i

to the draft results of the then on going investigation by NRC Region 11 personnel.

This trip report provides a review of the details of the event, a preliminary analysis of the human factors issues that were relevant to the event, and a summary of the findings from the event analysis.

Catawba Unit I had completed an approximate seven week refueling outage and reinstalled the reactor vessel head.

The control room night shift on March 20 completed the initial fill and ver.t and had vented water and gas (bubbly flow) from the reactor head vent for one to two hours hours longer than on previous fill and vent operations.

At the time of the shift change (0700 hours0.0081 days <br />0.194 hours <br />0.00116 weeks <br />2.6635e-4 months <br />), the pressurizer was 97% full, the RHR system was in operation with pump A operating and pump B in standby (suction valve open), and the Chemical and Volume Control System (CVCS) l l

was in operation with approximately equal charging and letdown flow of 50 Figure 1, Solid Plant Operation, illustrates the RCS, RHR and CVCS gpm.

configurations.

The on coming day shift initiated the pressurization of the RCS at about 0705.

The pressurizer fill was topped off until water exited the PORVs.

The PORVs were then shut and their control placed in the Low Temperature Over Pressure Protection Mode. The RCS makeup flow from centrifugal charging pump 1B was increased to 100 GPM and the letdown flow I

decreased to 30 GPM.

The target RCS pressure was 100 psig, 11

Similar previous pressurizations had required four to six hours to

• 't reach 100 psig.

Due to the gases trapped in the steam generator U tubes, the pressure rise is exponential and is not detectable over the early, longer part of the charging period.

The operators had three indicators for monitoring RCS pressure (two wide range (WR), 0 3000 psig, and one low range [LR), 0 800 psig) which were being closely monitored for a detectable rise in RCS pressure. However, unknown to the control room operators on duty, all three RCS pressure instrument transmitters were still isolated af ter the welding of the tubing fittings during the The two WR RCS pressure instruments are the sensors for refueling outage.

the Low Temperature Over Pressure Protection Mode for the PORVs.

Possibly due to the more extended venting of the reactor head by the At 0938 previous shift, the RCS pressure rose faster than anticipated.

the RHR pump suction relief valve lifted and limited RCS pressure to 455 psig maximum and the RHR pump discharge pressure to 625 psig maximum.

These pressure rises were not observed by the operators although the RHR discharge pressure indicator was operable.

The RHR suction relief valve remained open passing the RCS charging flow to the Pressurizer Relief Tank (PRT).

The rising PRT level indication was observed by the operators who began a search for the leakage path from the RCS.

The open RHR suction relief valve and abnormally high RHR pressurt were discovered, leading to the realization that the RCS had pressurized.

No annunciators alarmed during this sequence since the maximum RHR pressure that was reached was slightly below the actuation setpoint of the prnssure switch.

The factors that affected personnel performance during this event are discussed below:

Plannina and Schedulino Activities during a refueling outage at Catawba were scheduled and tracked by an Integrated Scheduling organization.

Integrated Scheduling had issued a Work Request to replace tubing fittings with socket welds on the RCS pressure instruments during the refueling lii l

outage and had scheduled a functional test with the RCS pressurized prior to Mode 4.

The need for these pressure instruments to be operable after fill and vent and prior to initial pressurization of the RCS was overlooked by both Integrated Scheduling and the Instrumentation and Electronics (IAE) planners.

No formal independent review of outstanding Work Requests was made prior to initial fill and vent.

Taaaina of Out-of Service Control Room Instruments The operability of control room instruments was the responsibility of There is a tagging procedure at Catawba, but it was not the

.lAE.

practice of lAE to always place a tag on an inoperable control room Since the three RCS pressure indicators in the control indicator.

room were not tagged and since the three indicators provide apparent redundancy, the operators were led into a cognitive trap of relying exclusively on these three indicators to detect pressurization of the i

RCS.

Systems Monitorino In addition to the RCS pressure indicators, there were other means Both the letdown available for detecting the rise in RCS pressure.

(CVCS) pressure and the RHR pump discharge pressure rose in response to the rise in RCS pressure.

Both of these pressure indicators were located near the RCS pressu"e indications since the RCS pressure indicators are grouped with the CVCS and RHR indicators and controls for a single operator.

Initiation of a significant change in state of a system should prompt the operator to monitor the overall indicated status of the system, in this event, the survey should have included the pressures in the CVCS and RHR systems as well as the RCS.

it is axiomatic that such operational surveys will reveal nothing abnormal most of the time. However, these surveys under normal conditions can provide the images of normal operation that permit the operator to detect the unexpected abnormality, iv L

1 J

Operator's Initial Diaonosis l

l The increasing PRT level indication alerted the operators that the RCS response was abnormal. However, the fact that the operators were unaware that the three redundant, diverse RCS pressure instruments were isolated had created a cognitive trap for the operators and their initial diagnosis assumed that the RCS was not pressurized.

e The input of the previously uninvolved RCS system engineer may have helped break this trap.

Variation in the Performance of Procedures The initial fill and vent of the RCS was performed in accordance with the applicable procedure. However, venting of gas from the reactor vessel head vent had continued for a longer time interval before the vent valve was closed than on previous RCS initial fill and vent evolutions.

The greater volume of gas removed from the RCS may have contributed to an earlier rise of RCS pressure af ter charging was initiated. The principle of " change analysis" is that any change '

execution of a procedure can create unforeseen effects.

In this event, application of the principle may have alerted the operators to the possibliity of an earlier rise in RCS pressure.

1 L

y l

L, i

p

j l

e 0 24 w

g oi s 'g C c 8

O 4

2 ) =o o

}

~

=

k

.-W f

I' d

g 6

) )Al g

l G:

01:

n g

~'

~"

• ^~

~

• O

CE CE i; 04

i 04:

01:

x x

01:

).

gg D.

I

+

i l

j t

L___&

h 01: 01: 01:

m,i f y 17 I

3 4

I a

ge 4

38 mt 1L m:

m:

T T

~

r m:

m:

l7 E

3 t m:$

g f-i gg l

3 e

ir

• • t*

01:01:

o p p Q

h!h g

o m

gg h

l s f k.5

II; s

-g LL NC 1

.i<

ry gtg Q_

4 y

Q Q

s vi a-,,,, - - -

-e-e

-r--

e

gWyg,g l

1

$.. ' [.; '

f

.y :

d i

' TABLE OF CONTENTS x:

I Pace No,

-ACKNOWLEDGMENTS-

^

CXECUTIVE S WMARY l

I Introduction

').0 M ',

'2.0' Description of Investigation 1

s 1

I.

2.1 Background

4 l

2.2: Event Description t

9 2.3 Analysis 2.3.1 Planning and Scheduling of 9

f Equipmer.t Outages 2.3.2 Procedures for Out of Service Instrument

-10 j

2.3;3 Systems Monitoring 13 2.3.4 Operator Recovery from the Event 13 4

1 16 3.0 Summary.of.-Findings

.l l

~

vi T

Figure'1 Solid: Plant 0perations 11 Figu're 2.Tagout' ProceduresJfor Control Room Instruments 5

14 Ifigur'e3 Input-Action Models m*

Table'I Normal Control Room Crew Complement 3

3 i

Ta' le II l Catawba Personnel Interviewed b

r 2

.vii q

2L. b -.

{' )

h i

1.0 INTRODUCTION

y v.:

(l s -

'i 11 Purpose y

j a

W Thr purpose of this site' visit on March 27.was to gather information

~

fregarding the overpressurization; event at Catawba Unit 1 on March 20, 1990.

This was the second of a_ planned series of studies to be conducted by NE/AE00 for the purpose of analyzing the factors that affect human.

perfo'aance during operational events.

1. 2 '

Scope e,.1 g

1 This study addresses the factors cifecting human performance which rtsu',ted in inoperable RCS pressure instruments and the.operatcrs' reliance on them during the filling and initial pressurization of the RCS.

The discussion also addresses the racognition of, and the recovery frofa, the overpressurization event, j

13 Team Composition i

The team leader was George Lanik from the Nuclear Regulatory Commission's i

_(NRC)_-Office.for the Analysis-and Evaluation of Operational Data.

The

.}

'other team members were Ann Ramey Smith, NRC's Office of Nuclear

' Regulatory Research, Human Factors Branch, and Dr.-Jerry Harbour and

,s Orville Meyer, both from the Idaho National Engineering Laboratory / EG&G j

Idaho Human' Factors Research Unit.

2.0 DCSCRIPTION OF INVESTIGATION 21-

Background

i The Catawba Station is located in South Carolina and consists of two i

JWestinghouseT4-loop PWRs, each of 1129 MWe net capacity.

Unit 1 entered commercial: operation in 1985, Unit 2 in '1986.

8 1

k 5

1

.. ~.

The normal 3 n'it 11and Unit'2'are, operated fro?2 a common control room.

I U

The non licensed

control room crew complement is shown in Table'!.

Both the Unit I and

+

-operators are not preassigned to individual units.

If Jthe Unit 2 supervisors are qualified Shift Technical Advisors (STA).

an event occurs on one unit,-the supervisor from the second unit becomes the STA for the unit in an emergency condition.

Personnel interviewed by-the event analysis team are listed in Table II.

On March 20, Unit'I was ending a sevan week refueling outage which had been entered on January 27,.1990.

The cient occurred about 2 1/2 hours into the day shift.

The previous shift had completed the fill and vent of the RCS with the pressurizer level at 97% and the PORVs open.

In order to-permit the operation of pressurizing the RCS to 100 psig and jogging RC

, umps to be initiated ar.; ceinpleted by the same shift crew, the night shift had elected.to use the remaining few hours of their shift to continue venting of the reactor vessel head.

The plan was for the day shif t to top off the fill of the RCS until water flowed through the open PORVs, close the PORVs, and continue charging the RCS until an RCS pressure of 100 psig was reached.

The off going shift briefed th'e day shif t on the plant status and previous activities, but the possible significance of the continued venting of the reactor vessel head was not discussed.

The day. shift crew expected that it would require 4 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> to attain 100 psig RCS pressure with the pressure rise undetectable until late in that time interval because of the' exponential increase of RCS pressure with time. However, the crew was unaware that the RCS pressure instruments (Two WR, 0 3000 psig indication, and one LR, 0-800 psig indication) were in inoperable because their root valves were Inadvertently left closed.

L addition to indication, the two WR instruments provide Low Temperature l

Over Pressure Protection (LTOP) for the RCS by automatic opening of the pressurizer PORVs ar:d provide.cverpressure protection for the RHR system

-by auto-isolation of the suction valves.

The LTOP protection mode of the PORVs was therefore not available, unknown to the crew.

The A train of the RHR system was in operation.. The suction valve for the 8 train of the L

L RHR system was also open.

L Z

~

.u TABLE;I.

u.,.

E.

v 7 '.( *

+

Normal Contreal Room Crew Complement-

~ "

o i

-A.

' Control Room i

-Contro1' Room Shift Supervisor (SS)

Control Room Senior Reactor Operator (CRSRO)

Nine (typically) Non Licensed Operators (NL0s) j.j B.

Unit 1

[

I Unit Supervisor (US)

.i Operator.at the Controls (0ATC) k Balance of Plant Operator _ (BOP) l C.

Unit 2 a

Unit' Supervisor (US)

-- i

Operator.at the Controls (0ATC)

.l Balance of Plant Operator (B0P) 1 1

~ TABLE 11~

I Cat'awba Personnel Interviewed a

T. 0. Williams, SS. (on duty on Unit 1.during-the event) i M. T. Lee. B0P (on duty on Unit I during the event) i R. B.'Abernathy, Integrated Scheduling V

R.!N. Casler, Superintendent of Operations /NP S.'T. Rose, Nuclear Production JJ. H. Knuti, Operations W. W. McCollough, Mechanical Maintenance R. T. Simril, Jr., Operations 3

1

[-

.2 22 Event Description x

he following event time line-sequence was constructed based upon

~

)

. interviews with;the control' room operators and upon review of the

-initialled and-dated control room copy of the Initial Fill Procedure and-the plant process computer records.- The results of the event analysis by-j l

the station staff and by the NRC Region II and Resident-Inspectors personnel were also available and very informative.

1/27/90 Unit 1 enters refueling shutdown.

o l

2/7/90-Root valves are closed by lAE technicians to isolate RCS Wide Range

-[

o (WR) Pressure Instruments-lNCP5120 and INCP5140 and RCS Low Range-l The-(LR) Pressure Instrument INCP5142 per Work Request 5491 IAE-1.

. instrument sensing line tubing fittings are to be replar.d by socket-weld fittings per Work Request 1493 MES 1.

J No permanent record or tag-out of the inoperability of these three l

o j

pressure instruments-is made-in the control room, i.e., no out-of-service tag is hung on the indicators.

The IAE group is considered to have operational responsibility for instruments in~the j

application of Catawba Nuclear Station Directive 3.1.1 (0P), " Safety Tags and Delineation Tags."

(A sequence scheduling error had been made in the writing and

-issuance of the above Work Requests. Completion of the Work The Requests was identified as being required prior to Mode 4.

scheduling error had a comon mode effect on all three pressure instruments (2 WR and 1 LR).]

2/21/90 Socket weld installations completed on all three pressure o

instruments.

Root valves remain closed on the two WR and the LR NC

-pressure instruments.

4 o

^ ' -

y n:

2i 3/,12LM y j *, 1207i Loy

. reactor vessel head studs tensioned; Unit 1 enters Mode 5.

^

a

.c 13/13/90 1300 o

Entered'the fill procedure for the RCS system.

a t'..

L3/20/90 i

0419.

.o ~

reactor vessel head vents are closed.

(The operators who wer'

. interviewed stated that venting from reactor vessel head vent was-I

/

Lcontinued'for approximately three' hours vs approximately one hour on previous fills of the RCS system.)

0700'

o; Control room operators shift change.

Plant cond.. tons are:

7 i

'o; RCS -fill is.nearly complete as ' evidenced by pressurizer level instrument reading of 97%.

1 i

o PORVs~are open.

L L

Train 1 A of the RHR system in~ operation.

Pump suction isolation-o yalve for Train'B is open.-

i The.CVCS system is in operation with centrifugal charging pump 18 -

-o

_ running and charging and letdown flow balanced at approximately 50 K

gpm.-

-[ Operators are unaware.that both WR _and the LR RCS pressure

' instruments are inoperable due to the' root values being closed.)

Charging flow increased to approximately 110 gpm to complete the o-fill'of the RCS system.

5 l

i

~w

(/

.C

!.0708 ;The following sequence takes place:

Y ~o :

The Pressurizer Relief Tank (PRT), level increases indicating to the q

operatorst that: the pressurizer is over flowing through the PORVs.

v

~

The PORVs are closed and the low Temperature Over-Pressure

- o Prot'ection Mode for the PORVs is selected.

(In this mode the PORVs controls are set to open the PORVs automatically upon receipt of a

' actuation signals from the two WR RCS pressure instruments in the:

event of high RCS pressure.- The two WR RCS pressure instruments also provide overpressure protection for the' RHR system by initiating an auto-isolation at 600 psig.

Since'the two WR pressure instrumentsareinoperable,thisprotectionisinoperable.]

Charging flow is adjusted to approximately 100 gpm and letdown flow o

to approximately 30 gpm to begin pressurizing the RCS system to 100 psig.

The operators are expecting this pressurization ta require four to six hours, based on previous experience.

Since the pressure rise is controlled by the pressurization of the gases trapped in the U-bends of the steam generator tubes, the operators expect that it will require two or more hours to detect a pressure increase in the NC system.

(P = MRT/V relation creates an exponential rate of rise of NC pressure.)

After this adjustment, the operators begin observing the two WR and o

the L.R RCS pressure instruments in order to detect the'first indication of increasing pressure in the NC system.

(There are three factors present at this time that could prompt

. operators to check the'opershility of the three pressure instruments, i.e., (1) the plant has been in a shutdown condition for nearly two months with work in progress in containment and no opportunity to observe the operability of the three RCS pressure instruments, (2) the two WR instruments are providing the NC low temperature overpressure protection, and (3) operability checks are marginally possible by checking for the static head pressure of j

s

][

[

y u

p

3

i. ; ' 7' " approximate 1y' 28 psig in the!RCS'systen (1% of 0 3000 range on WR, 3-1/2% of 0 800 range on LR).and, more positively, by also observing 6

the RHR system pump discharge pressure which is a low range pressure instrument.)

The licensed RO designated as the Unit 1 Operator at the Controls o

A licensed R0

.j (0ATC).is controlling the RCS pr'essurization.

1 designated as the Unit-1 Balance of Plant (B0P) operator is on duty

-in the. control room.

The Unit 1 supervisor is in-the plant

.lj

-directing a test of the main' feed pump turbine lA cverspeed control s.

The control room shift supervisor and control room senior reactor operator, who are licensed SR0s, are in the control room.

Other activitier in the control room on Unit I are SSPS response time testing,~which periodically activates the annunciator horn, and valve stroke testing of the Letdown isolation Valves.

(The-i j

operators indicated in the interviews that these were distractions but that they did not directly interfere with the monitoring-and control of the RCS pressurization.)

i l -

l L

0938 The RCS and the connected RHR system have pressurized to the degree o.

that suction relief valve on RHR train B has opened.

This is 2-1/2 L

hours after initiation of charging vs the expected four to six This reduces the RCS pressure to approximately 175 psig and hours.

the relief valve remains open passing the net charging flow from the

[

CVCS system to the PRT.

(The setpoint for this relief valve is 465 psig and it is large enough to relieve the combined flow of the charging pumps.)

The'RHR pump 1A High Discharge Pressure annunciator does n.gl alarm

~

o.

I',

(setpoint 579 psig) and operators do not observe the abnormally high I

RHR ressure instrument indications because of their concentration on ins three inoperable RCS pressure indicators.

(later analysis concludes that the RHR discharge pressure reached a conservative maximum of of. 625 psig, possibly less since the RHR high pressure annunciator did not alarm.]

L 7

4

~

s

n

- c w' N f 7 0945' v[bh*,o;

' Operators observe 1an: increasing-leveltin the PRT.-

4 y--

h o.

Charging'is reduced to 95 gpm. -Unit supervisor is notifled.

'PRT l

pump down'linet,p is checked since previous'shif t had_ reported

[,

problems with PRT pump down.

PRT pump down is verified.' PRT level j

-continues to' increase.-

]

J a1

-2 1000 Operators svipect PORVs may-be leaking into the PRT.

o PORVs-isolated' singly, but PRT level continues to increase, s..

o i

s

.-1015

. System' diagrams are pulled and examined.

The RCS systems engineer, o

who was in theicontrol room on unrelated matters, participates in-the'diagiostics' and recalls an NRC Information Notice on interfacing systems lus:of coolant.

System diagrams are reviewed for possible leakage paths.

-o1

. Operators-observe that the RHR pump discharge pressure is abnormally

-high at 375 psig.

Since the RHR pump head;is approximately 200 psig, this indicates that-the RCS may be= pressurized to approximately 175 psig.

o The 0ATC directs an equipment operator to check RHR system suction relitf valves located in the containment.

l o

Inicreased letdown flow to 120 gpm and gradually reduced charging-flow to depressurize RCS and RHR system to approximately 200 psig as L: <

indicated on the RHR pump discharge pressure instrument.

f, '

'1020 L

~ o The equipment. operator finds the relief valve on the suction of the

'B Train RHR system-is discharging water as evidenced by sound and vit ation.

I:

8 u

i la 1

( {.

,s The-B Train.RHR system is; isolated.

The.& Train RHR suction relief y o, -

valve remains: connected to the PCS.

y :v

[The control room notifies staticn management and an investigation is initiated. The WR and LR RCS pressure instruments are unisolated, it' is estimated'that the RCS pressure peaked at 455 psig.

The RHR discharge piping reached a maximum of 625 psig which is above the 600 psig design pressure for operation but within ASME Code limits.)

2.3 Analysis 2.3.1 Plannino and Schedulina of Eauioment Outaoes The direct cause of this event was the failure to unisolate and verify the operability of the three RCS pressure instruments after completion of the' welding of the fittings and before entering the fill and vent procedure. The sequencing of the operations of isolating. welding, and unisolating were under the control of the Integrated Scheduling activity during the refueling outage.

An Operations Outage Information Request Sheet had been filled in by the RCS systems engineer indicating that the work was required prior to entering Mode 5 (tensioning of reactor vessel head studs) and that a functional test was required prior to entering Mode 4 (hot-operations).

The Work Request issued by Integrated S;heduling to install the socket welds on the three RCS pressure instruments was scheduled to be-done during the refueling outage and was completed by Mechanical Maintenance prior to setting the reactor vessel head.

A functieaal check of the pressure instruments was scheduled to be completed I

prior to entry into Mode 4 when RCS pressure was applied to the instrument sensors.

The isolation and unisolation of the-sensors to be performed by lAE were identified only as notes on the Work

^

Request. The notes were reviewed by an IAE planner and documented

- on a Standing Work Request (SWR) 6114 which is used to assure that the RCS pressure instrumentation is aligned and operable prior to Mode 4.

A formal 9

s k

7

e' review of outstanding work and of the' maintenance data base was not

.4 performed prior to initiating fill and vent and initial pressurization of the RCS.

The station management's investig tion of this event has determined that a chang'e should be made to the scheduling mechanisms.

This change will provide for a review of activities and work completion required for entry into Gach of Modes 16, inclusive, and for changing from the RCS filled to not filled conditions and vice versa.

To be effective, this listing should not rely solely upon the required completion conditions on the work requests or an error on a work request will not be detected.

These listings should receive a review that is independent of the release of the work requests.

The station management has indicated that they will also change the controlling procedure for unit shutdowns so that it will require a signoff from lAE that the RCS wide range pressure instruments are valved in and operable prior to settir.; the reactor vessel head.

This' action is specific to the oversight causing this event only.

2.3.2 Procedures for Out of-Service Instruments Catawba Nuclear Station Directive 3.1.1, " Safety Tags and Delineation Tags," is applicable to the removal of control room instruments from service. However, IAE has operational responsibility for the control room instruments and it was not their practice to always hang a tag on the instrument indicator _in the control room but to sometimes place the tag elsewhere, it was a normal practice for IAE to advise.the control room of outages and restorations, but there was no formal procedure or permanent control room record.

The operators had instructions to r.ot place their own tags on out of-service indicators.

10

>. M y' '

Consequehtly,L ti.cre was no.out-of; service tag or record h the control: room to alert! the operators that. the LR-and both WR pressur i instruments were inoperable..Such tags and records would have f

provided an' independent safety barrier against entering a sensitive operation while-relying'on indicators that were inoperable because of the IAE work.

Their value rests on the fact that they are additional, independent' barriers and upon the fact they initiate a check for operability at the time when plant conditions as observed by the aperators require' operability of the instruments. An unplanned event could create a situation where the control room operators need or use an instrument at a time which cannot be foreseen by planners.

In such a situation, an outage tag directs the operators to use an alternate operable instrument or restore the instrument.

See figure 2.

As illustrated in Figure 2, planning and scheduling is a predictive

_ process where a sequence of activities is developed with key events pre-established, e.g., initial fill and vent of the RCS, where a

certain conditions must be satisfied before the sequence can continue. Planning and scheduling is absolutely essential Curing operations such as a refueling outage and has been developed by the nuclear intJstry into a reliable process.

The value of a procedure where the control room operators place their own tag on an instrument which is to be taken out of service is that it is an independent reminder to the operators to check for ' operability before_again relying on the instrument.

In this event at Catmha "

wculci have been a simple task for an operator to visually verify that the socket welds were in place and the root valves cpened.

Ir is not a substitute for plar,aing and scheduling and for quality work by maintenance personnel.

Most often before removing an out-of-service tag, the check that the control room operator performs consists of receiving assurance from maintenance that the instrument is ready for service.

11 1

5,

' l[:

--m

.t '

^ '

].

(

f~

~

Figure'~ 2. Tcgout - Proce:dares;fer Control Room : _

.lastruments:

14 ;-

~

s i-Control room reliance on Actual instrument plant

-+-

conditions Clearance of I,. -.

instrument

{

tag-out Control room d

tag-out Predicted.

1, i

~~

]

restoration conditions g

t 4

l 8

d, i

Planning i

instrument and t.--

outage scheduling I

f l

i

l2L3.3 -Svstems Monitorino

y..;

• L "i

As noted in th'e Event Description,LSection 2.2, the rise in RCS pressure was observable-on the letdown pressure indicator and the RHR I

pump dis' charge pressure.

The fact that this was not observed by the operators.is understandable since they had two independent WR RCS pressure instruments and a separate NR RCS pressure instrument which provided some diversity.

However, the use of the RHR pump' discharge pressure after the rise in PRT level to deduce that the RCS pressure was approximately 175 psig and not zero illustrates a useful rule of

" systems monitoring."

\\

An-accepted rule of systems monitoring is that when a significant change in operating conditions is occurring, then the available indications on all the affected systems should be surveyed.. In this event, the initiation of pressurization of the RCS would not only affect the RCS but also the interfacing CVCS and RHR systems.

On almost all occasions, such surveillance will reveal only the normal effect on the interfacino systems, but-such observations provide a baseline reference for the operators.

On the occasion such-as'during this event where there is an anomaly, the anomaly may be then more readily detectable.

Even without previous experience in this RCS pressurization operation, this type of surveillance would probably have noted the RCS pump discharge pressure rise.

This type of surveillance is sometimes described as attaining and aaintaining a

" process image" which means an overail view and understanding of the process.

2.3.4 Ooerator Recovery from the Event The operators' confidence that they were receiving reliable, redundant, arf diverse readings of RCS pressure not only permitted the RCS pressare "ise to be undetected but also was the basis for their explanation of the rise in PRT level and the actions the operators selected to resolve the rise in PRT level.

13 l,

c..y l

11 1-:

figure 3:-

_n

~

INPUT - ACTION MODELS.

Selected Explanation' Approximate ~

Input.

Action (s)

(Hypothesis)

Sionals Times Normal charging Monitor WR and LR 0708 - 0945 RCS pressure

.RCS insttumentation

. of RCS indicatars 0 0 H-

' Leak in RCS or (1)* Check PORVs for 0945 - 1008 Rapid increase in connected' system' leakage PRT level (2) RCS makeep flow AND reduced and check

~

for other RCS or RHR RCS pressure leaks indicators 9 0 (3) RHR suction relief valves checked for flow.

(1) RHR suction RCS pressure 1008 - 1112 Passing flow rather relief valve must reduced than just leaking in 2

RHR suction relief have been lifted by high pressure, valve therefore, RCS pressure must also be high AND (2) RCS pressure RHR pump discharge indicators still isolated pressure 9 380 psig following maintenance AND-RCS pressure indicators 9 0 u

• Numbers indicate sequential actions / hypotheses.

r

• '. s,

e figure 3 illustrates selected actions of the operators' based on y

their understanding of detected input signals, as repre<ented by various control room instrumentatie readings.

for example, from g

0708 - 0945, the operators' observed the RCS system pressure k

indicators at 0 psi (an input signal). Their explanation or hypothesis for this detected input was that the RCS was charging 9

Their selected action (representing a decision), was to normally.

L continue normal monitoring of the VR and t.R RCS instrumentation.

Although their explanation and selected action was to prove faulty, it was based on, or initiated by, a false signal (i.e., RCS pressure

[

I indicators at 0 psi).

Only at 0945 when they detected a rapid increase in PRT ' eve? (representing a su ond, and conflicting input i

signal), did they <.hange their initial hypothes',s f rom RCS chirging g

normally, to a leak was present in the RCS or connected syttem.

Based on tMs second explanation, the operators' selected actions

]

sould conform to this new (yet still incorrect) understanding oi:

that J

the system state.

Finally, from 11'08 - 1112, they detected other input signals (e.g., RHR aischarge pressure at 380 psi) that forced P

,et another explanation of the system state (this time correct) which

-&g resulted in actions being selected that returned the plant condition M

I to a normal configuration.

The information that the RHR suction relief valve was " passing flow" (operator's terminology) rather than merely leaking led to the explanation that the RHR pressure was high.

Even then, according to

[

the operators recollections, it was dif ficult to disbelieve the three E

It was only separate instrument readings of 0 po g RCS pressure.

when someone providsd an explanation, i.e., the recollection that all EE

=

three pressure instruments had been valved out at one time during the F

{

refueling outage, +as there acceptance of the conclusion that the 0 psig RCS pressure readings were false and that the RCS was 7

presseri nd.

15 M em

[,

v 3.0'

SUMMARY

OF FINDINGS ft 1

The findings from the analysis of this event can be classified as follows:

k P1anning and Scheduling

-1.

A mistake in planning and scheduling of maintenance, repair or modification can ' lead to scenarios that would be considered nearly incredible, particularly if t*.e mistake is carried forward from an outage into Mode 5'and beyond. There are defenses-against the effects of.a scheduling mistake which can be added to.the scheduling activities; the independent listing of requirements that must be met before initiating a significant change in plant conditions (entering Mode 5, initiating filling of the RCS, etc.) which is being considered by the Catawba Station staff, is an effective defense.

2.

Tagging of Out-of-Service Control Room Instruments

Administrative' procedures that require that information is available to the control room operators on current plant status provide an L

l independent' defense egainst scheduling mistakes.

Reliable and immediate knowledge of current plant status is ir.portant if an unplanned event occurs that could not be foreseen by the scheduling activity, Knowledge of current status of control room' instrunientation is particularly important because if the operator can t

l be misled into the use of a false signal a progression of explanations and actions by the operator is initiated that may be c

diffir. ult to interrupt.

A false signal leads to a false hypothesis j"

l which leads to ineffective actions, or worse.

/

Sy.tems Monitoring 3.

s H

Maintenance by the operator of 2n overall surveillance of tne systems U

under his control can provide a final defense against many types of problems, including in this event, the existence of f alse sfinals.

l

.It is not an infallible means of defense, but it is of value on its 16 I'

1 gf hy

l

'/

~

own merit in providing the operator with his own image-of the normal D'

et

~ response of the systems.

Abnormal responses may, then, be more readily apparent.

4.

Operaturs' initial Diagnosis ca 1

The increasing PRT level indication alerted the operators that the RCS' response wcs abnormal. However, the fact that the operators were unaware that the three redundant, diverse pressure instruments were isolated had created a cognitive trap for the operators and their initial diagnosis assumed that the RCS was not pressurized.

The entry of the previously uninvolved, RCS system engineer may have e

helped break this trap.

5.

Variation in the Performanco of Procedures The initial fill and vent of the RC't was done in conipliance with the applicable procedure.

However, venting of gas from the reactor ve:sel head vent had continued for a longer time interval before the vent velve was closed than on previous performances of the RCS initial fill and' vent.

The greater volume of gas removed from the-RCS may have contributed to an earlier rise-of RCS pressure after charging was initiated.

The principle of " change analysis" is that any change in executior, of a procedure can create unforeseen

effects, in this event,' application of the principle may have alerted the operators to the possibility of an-earlier rise in RCS-pressure.

!l-1 17

.-----~.:---- --------.-.

.