ML20056G249

Forwards Accident Sequence Precursor Analysis of Plant LER 483/92-011 Performed by ORNL Contracted by Nrc,For Comments

ML20056G249
Person / Time
Site:	Callaway
Issue date:	08/05/1993
From:	Wharton L Office of Nuclear Reactor Regulation
To:	Schnell D UNION ELECTRIC CO.
Shared Package
ML20056G250	List: ML20056G249 ML20056G434
References
NUDOCS 9309020317
Download: ML20056G249 (23)

Text

'

k

'2 Docket No. 50-483

. August 5, 1993 Mr. Donald F. Schnell Senior Vice President - Nuclear Union Electric Company Post Office Box 149 St. Louis, Missouri 63166

SUBJECT:

REQUEST FOR COMMENTS ON THE PRELIMINARY DRAFT OF NUREG/CR-4674,

" PREG'RSORS TO POTENTIAL SEVERE CORE DAMAGE ACCIDENTS: 1992, A STATUS REPORT" Enclosed is a copy of an accident sequence precursor (ASP) analysis of Callaway Licensee Event Report (LER) No. 483/92-011 performed by the Oak Ridge National Labor.atory (ORNL), contracted by the NRC. The ASP analysis will be part of NUREG/CR-4674 scheduled to be published in September 1993. We are interested in any comments you may have on this preliminary analysis. Your response, which is strictly voluntary, should address the following three areas; report characterization of conceivable plant responses to the event, representation of safety system configuration, and analysis assumptions regarding equipment recovery. Any responses received by August 31, 1993, will be considered by AE00 and ORNL in the final draft of NUREG/CR-4674. We recognize that the timeliness of this response may be an inconvenience, and genuinely appreciate your efforts to work with us. Should you have any questions, please contact me at (301) 504-1396.

Sincerely, ORIGINAL SIGNED IW:

L. Raynard Vbarton, Project Manager Project Directorate III-3 Division of Reactor Projects III/IV/V Office of Nuclear Reactor Regulation

Enclosure:

As stated cc: w/ enclosure See next page DISTRIBUTTON Docket File ATGody,Jr. BBartlett, SRI NRC & Local PDRs GHolohan, AE0D LRWharton PDIII-3 Reading OGC JKing JRoe ACRS(10)

JZwolinski PDIII-3 Gray JHannon Region III, DRP MRushbrook FManning AE0D OFFICE PD3Y3fh4 PD3-3: INTERN PD3-3:PE _ PD3-3:PD N

, NAME MRubr'oh JKing/jfk/bj[~d RWhartd / JHannonIN DATE Q/ 4/93 P,/ 4'/93 6 / 4 /93 j' /If93 l ,

0FFICIAL RECORD 1 aggyi DOCUMENT NAME: G:\CALLAWAY\ ASP.LTR -

PQ kkhh.e2 a r

ord}

. m.

gr$8sssnmnas88lk_

s ~

i

~

b Mr. D. F. Schnell Callaway Plant Union Electric Company Unit No. I cc:

Cermak Fletcher Associates Mr. Bart D. Withers 18225 Flower Hill Way #A President and Chief Gaithersburg, Maryland 20879-5334 Executive Officer Wolf Creek Nuclear Operating Corporation Gerald Charnoff, Esq. P.O. Box 411 Thomas A. Baxter, Esq. Burlington, Kansas 66839 <

Shaw, Pittman, Potts & Trowbridge 2300 N. Street, N.W. Mr. Dan I. Bolef, President Washington, D.C. 20037 Kay Drey, Representative t Board of Directors Coalition .

Mr. S. E. Sampson for the Environment Supervising Engineer, 6267 Delmar Boulevard Site Licensing University City, Missouri 65130 Union Electric Company -

Post Office Box 620 Fulton, Missouri 65251 U.S. Nuclear Regulatory Commission ,

Resident Inspectors Office RR#1 Steedman, Missouri 65077 Mr. Alan C. Passwater, Manager .

Licensing and Fuels i Union Electric Company

• Post Office Box 149 St. Louis, Missouri 63166 .

Manager - Electric Department Missouri Public Service Commission '

301 W. High Post Office Box 360 '

Jefferson City, Missouri 65102 i Regional Administrator  !

U.S. NRC, Region III 799 Roosevelt Road Glen Ellyn, Illinois 60137 Mr. Ronald A. Kucera, Deputy Director

• Department of Natura' Resources P.O. Box 176 Jefferson City, Missouri 65102 i

8 ENCLOSURE '

l PRELIMINARY l

e 0.1 LER Number 483/92-011  !

i i

Event

Description:

Loss of main control board annunciators Date of Event: October 17, 1992 i

Plant: Callaway i

0.1.1 Summary '

t At 0100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> on October 17,1992, during restoration from the replacement of a failed power supply f that disabled 76 main control board (MCB) annunciator windows, a short-circuit caused nine power i supply fuses to blow, failing the annunciator system. Four blown fuses in the field contact power -

supplies were found and replaced about one hour later. He operators assumed this fuse _ replacement returned the annunciator system to operation, although anomalous behavior was still being observed. In actuality.164 annunciator windows remained inoperable. He remaining failed fuses were found and ,

replaced, and the annunciator system tested and confirmed operable at 1937 hours0.0224 days <br />0.538 hours <br />0.0032 weeks <br />7.370285e-4 months <br />. The conditional core [

damage probability estimated for this event is 4.3 x 105 This estimate is considered conservative. He t relative significance of this evmt compared to other postulated events at Callaway is shown below in  ;

Fig.1. t IE.R 483/92-011 t i

l 1Er7 1E 6 1E 5 1E 4 IE 3 1E-2 [

l i I v i i I i n i

precursor cutoff kNM- =m 360 HEP Fig.1. Relative event significance of LER 483/92-011 compared with other potential events at Callaway.

i I

LER NO: 483/92-011 l 1 PRELIMINARY  !

d

• -- e- _.________I -

PRELBIINARY 0.1.2 Event Description On October 16,1992 at 1840 hours0.0213 days <br />0.511 hours <br />0.00304 weeks <br />7.0012e-4 months <br />, with the unit at 100% power, an annunciator field contact power supply failed because of an internal transformer short. De power supply frilure caused 76 MCB ,

annunciator windows to illuminate. At 0058 hour6.712963e-4 days <br />0.0161 hours <br />9.589947e-5 weeks <br />2.2069e-5 months <br />; on October 7,1992, the failed power supply was replaced and all applicable annunciator windows cleared. At 0100, during restoration from the power supply replacement, a short-circuit occurred and all four field contact power supply output fuses blew.

His resulted in the loss of the entire MCB annunciator system. Approximately 370 annunciator windows were illuminated. Numerous plant computer alarms were also affected. By 0156 hours0.00181 days <br />0.0433 hours <br />2.579365e-4 weeks <br />5.9358e-5 months <br /> the blown fuses had replaced and power had been restored to the MCB armunciator system. Upon restoration of power, the illuminated annunciators cleared and the critical problems with the system were considered corrected.

The operations crew performed lamp tests on all the annunciator panels, which they assumed verified the operability of the system.

However, anomalous annunciator system operation was still being observed; the problems were considered minor and plant personnel determined they could await analysis by the morning shift. During the morning shift, unexpected annunciator system operation continued to be observed, and additional troubleshooting began. At 1300 hours0.015 days <br />0.361 hours <br />0.00215 weeks <br />4.9465e-4 months <br />, a bad logic power supply was found and replaced. At -1630 hours, instrumentation and control technicians determined that five additional logic power supply fuses had been blown, apparently at the same time that the first four field power supply fuses blew. Dese fuses were replaced by 1800 hours0.0208 days <br />0.5 hours <br />0.00298 weeks <br />6.849e-4 months <br />, and testing to confirm annunciator operability was completed by 1937 hours0.0224 days <br />0.538 hours <br />0.0032 weeks <br />7.370285e-4 months <br />. Following the replacement of the four fuses at 0156 hours0.00181 days <br />0.0433 hours <br />2.579365e-4 weeks <br />5.9358e-5 months <br /> (when the annunciator system was believed to be operable),164 annunciator windows had remained inoperable.

Lack of knowledge of the annunciator system on the pan of the plant personnel resulted in an inadequate assessment of the event, failure to declare an Alert when the system failed, and failure to terminate plant activities which could have resulted in unnecessary challenges to plant systems (a liquid radwaste release, a 345 kV line tag out, and turbine stop valve surveillance testing).

0.1.3 Additional Event-Related Information The Callaway annunciator system is designed to monitor 1400 alarm points using field contacts, which either open or close, to alert operators in the control room by illuminating an annunciator window and sounding an audible alarm. Individual alarm points that are grouped on a system basis also feed the plant computer and the alarm printer.

He system has four power supplies connected to a 125Vdc station battery to power the 1400 field alarm contacts. Rese power supplies have common (paralle!-connected) inputs and outputs, and each power supply input and output is protected by 1 A slow-blow fuse. Here are also 14 logic power supplies that receive their input power from one of the two 125Vdc station battery systems, one of which is common to the field contact power supplies discussed above. He logic power supplies provide five different voltages to the system and each has a protective fuse associated with its voltage. None of the fuses (78 total) have local indication or indicating lights to monitor their operability. De arrangement of the power supplies is shown in Fig. 2. i i

l LER NO: 483/92-011 1 2 PRELBfINARY i

1 l

o .

PRELIMINARY l

Gem ET& lese

'*t.%.

- l 5 2 -l

+- y-u T * * * " ' -

- rm, e

= '

• AM':.'*. *

/

Wv uso ,, T< l T.

w m

- ~ <- s,-

~' ~

r\,

we nn.

• **"" c=;lll'"

/ ...

.-./

a

?*

_ ,=r_

,,. l'l e -- u

- ,w-

- - ;g ) t* 24

- ~

_ l j .i e Fig. 2. Callaway annunciator power supplies.

Annunciators which were inoperable prior to 0100 on October 17,1992 (when one power supply was failed) are shown in Fig. 3. Annunciators which were inoperable following the fuse replacement at 0156 hours0.00181 days <br />0.0433 hours <br />2.579365e-4 weeks <br />5.9358e-5 months <br /> are shown in Fig. 4.

0.1.4 Modeling Assumptions The event has been modeled as a potential reactor trip, loss of offsite power, and small-break loss-of-coolant accident (LOCA) during the 18.5 h period starting at 0100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> on October 17,1992, when multiple power supply fuses were blown. Based on a review of the plant operations in progress during the event and the loss ofinformation available to the operators, the frequency of inhicing events, the probability of the operator failing to actuate manually-am**4 systems, and the probability of not recovering initially failed systems were revised as shown in the Branch Frequencies / Probabilities section of the Conditional Core Damage Probability Calculation sheets included with this analysis (changes are shown to the right of * >

• symbols).

LER NO: 483/92-011 3 PRELafINARY

PRELIMINARY l

l W!>i4

• 4 G .

n 8

$ N NN* d

'b l

N. N ./ NI/

M / /k /

!$ . ....  : $* X 4*d  ;

=

a d s a x < =

+- X w4 ,

=

W E W M !i s X *4M= 4 i

= > > b W = 4 4 2 4

• 4 r E l  : H i F 5 li 5 I

b $ Y h N h A N Y wommwxpr x E+ X ci

!! I N N 2

• j H !!F 5 >

a sg s ss/ss $ 3/ .

b 0 9 9 E I* fsf f$ A

- p 3 X *8$ '(' s n a i m = t a d . 5 4 a m e s > = -

Mi k e m s w

- e B B $ 4 4 e W > N H > N e > B B 4 H

=

MXMM E re r

Fig. 3. Annunciators failed prior to 0100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> on October 17,1992.

LER NO: 483/92-011 4 PRELIMINARY

PRELBIINARY

/h h l E

h! l E! l [. i f h

/l% h /%

2 2 .=5

!I N h /%/ %% 5 $ 3 h! h! d!N N N/\g\ /\ e/  % .. .. \/

$ b h 3

N N $~fb b

/ N/%/ '\/h\

85 trl / \

%) ./

m 8 4 5 N  :

EilEiWlM T N y%/%/ I

\,,/\ l j

M N N/ /k I.

'T Q%/%/  %)f%/ , q D 7  %

A/\ M

~ '

/\/N ih w%/%/ NyN/ ,, W Ng\/g\N M' \ \ b/l5 /

\ N ,/

%/%/

\ \ / / %

p *

• e Fi > .s k eDX EE.

m a s eu s l =

XXXXXX i

. .. ene . ,,,, e>

CI

!4

. v i1 ,Ei s M

= w 4

.a N $l M 3 ju ju M W

$ [3

= t >

rs

,(

N

=

4 4 A

N !N r N I

@ N A,ry

. xj. y E $

RT ss =

$ XXXXX

= 5 + = * + = = S XXX e

5 + s a w d

tXXX a

= -m+ m x <! - S XXX ix Eg. 3. Annunciators failed prior to 0100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> on October 17,1992 (cont.).

LER NO: 483/92 011 5 PRELBIINARY

4 -

-1 PRELBUNARY t l

1

< = u a - s

\tf .

E 5 /~N @ '

i

• Jia a :t Jr 4 i M i is , IN t i

3 h N 3 b _g i g Id I) m @ j e

i a

!U fE i $ b  !

4 u

f I # h b 'h h a c

&" bb b &" S" k

/l%

• @ 5 % tG  % %

8 < . u o w w

. a '!, ,n vy . -

n

• AA -

's 2 o. H

. 2 8

1. k h $ $

• k ,

a 4 4- 4 5 $ $ *e

• M M o o M. M. l i

g e .at w $ $ g g g 3 ,3  !

r% ,.3 . .

= 4 4 4 $ $ $ -

E t t  :

. . o u . - ..u . w '

i I

Fig. 3. Annunciators failed prior to 0100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> on October 17,1992.

LER NO: 483/92 4 11 6 PRFLIMINARY e

O

d e

PRELIMINARY l

i; ,

i si eXMsieXs l

! i i iqi  ; i e:w HXMX 1 i I=in X  ! i MM s XXM  :

i i 1415 X i MMMHM ,

i m ini XX i m isisi< W l

(l*l*l l X MX

! EiHill l i sInt XXM si<!! XX

• X X N ilH sisi i i dirX dl*la i i

si is X mi i i

<ls i ti X  ;

r iisists!st i !H e X

• i i iIsisigl 1 li 5 HHM i ,

i E m w XX e  :  :

iisisiMXl:

iIsisi MX Iti l

i MHM c !5

- 9I 191 e HXXX

=F M R I i M 4 XMM XXXXXX i

?is < alx >

XXXXXX i x * < sin > .

=

XXXXM gli- a>4 X

=

1:1 XX  : n i- + x n w XX E XX H  :

I XX s XX a i sic < X: 4 i

XXXX s X i laixiclu < r

=

XXXX s X i<!<lcis X t Fig. 3. Annunciators failed prior to 0100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> on October 17,1992.

LER NO: 483/92-011 7 PRELIMINARY <

t

?

1

1 PRELBfINARY i

i i

y

$ N N 34

• d  !

J =

GN k k

• d '

5  % /

E.

m E

hk > ,h s  !

A/N / '

E > k >;

k' / N /\k  !

T /N/

. e u . . .

N N '

/ N, s <

0 >

93 hl h I E '

/ \/ \

k k >  !

2

'! h Il N It E

• >k/ s, s k > h N /T / \ N

$ $tr g)l/\ I

(

lt! 7, lll lll > o

/\, s >u ' ~ ltl- = l E sb k . h .k

• E, ,

h k h h !i h 2 Itl 3 lg

- ~, ~,  %, h f

. t

. e \/

N 0 a si m. m.  : -

,1 4 ,\=\/'a $ q,N/ q s

=;;  !

S li W lii,,s

/

s.,a 8 5  !. \

f .4 t

> 0 V, 4

, i:i  != k4 s -V.N, ^N,'  !

- p 3 -

,N.?. ,/s s A. ( ,Y.  :

g',y' ,' y'g's 3 g 3 g e 'g.

s- s s, m s, l

e a e a > s e XX.y n .yy s v.y

s. .

- . a a e X.yyy. > n.

=

>X e e ik =XX > XX e I

i Fig. 4. Annunciators failed after 0156 hours0.00181 days <br />0.0433 hours <br />2.579365e-4 weeks <br />5.9358e-5 months <br /> on October 17,1992.  !

i LER NO: . 483/92-011 g PRELBIINARY  !

L 4

PRELIMINARY 2 3II

/ \m g '\

4 k k :/ A*N E NN d fs g s / T/

I I 8 j

\ /s /

I

,Ns , + k k kh S N hs

\/ \ .s \ /s/ .. ..

5 Y $ N, e\

N Ns $ $ $ h f. s f \ , __

/ N/ %/g \

G N' I h a b b b

~~ ~

\ 's. / / \ g 'N / i /

f b b s , s - si s / N

\;;;g^; # ., OM M#

,*s, L s_ / \/ s s s A. <' . .r \/ \/

N'/

gkN,

,*1

/N f s f s [

I

- \

N

/ vi v s h ,,

I h > t 3,. ,

i

\

1 I h D8 h@sq' g h

> s E I  !

T /'\ /

D8

/ s/ \

I

\ g/

/ s

o  : e 5

,, T /s s/\ / \.fb' . . ,

I h ,9 Nr,k t- Wi 3

,2 w Es F F M N

. s. /s. a / g ,rs <

0 Y IS ,P' se$v sd 8

-s 8 d s

s. /\. e n n i s 8

59 8 s,.n als 9 se ej<

3 J';,$'l _ _

sp e.fs%s ss f

)

• ,9 s e m e s W l l 5 ,g 4 4 W Js ,( l 4

a ( $ sG $8 $ \g/ s 2

s \> s (

/\)/a N/

c $

a ( $ $ E ,s

( 2

- [s ( (  % $ f a w e

e r}s u

$ W [s o =*

z

[e[s (

\,

a u

, (s o .

g Fig. 4. Annunciators failed after 0156 hours0.00181 days <br />0.0433 hours <br />2.579365e-4 weeks <br />5.9358e-5 months <br /> on October 17,1992.

LER NO: 483/92-011

9. PRELIhDNARY i P

.o

. a PRELIMINARY  :

t i

= m s s . .

j

., a, s.- a

/ \

v ,i m.

a ;I *H

.$ , -.,I ,$., i

\/s/

3 h d -

igB.s,h s  :

\ / E 1'

m c =

1 %

,1 e

};

e ,% ,n, .i.;. , n, i

- g o  ;

+ B ,, m sv ,

w-

~

,s s.< - s.- ,a ,

.o *: .n diu  :

,r s'l.

,I

,r t

e ill m g ,Niss/ .i.n si .

1 e a s g ..

r

$a y

I

• 8 U O w w

(

~

/ %

iji ',g' F;

, f;, =

is a g /y /(\

A l

s-x vv *. . ,

AAA5s N/N -T k k

< a u o - . .

. u . . . t l

l Fig. 4. Annunciators failed after 0156 hours0.00181 days <br />0.0433 hours <br />2.579365e-4 weeks <br />5.9358e-5 months <br /> on October 17, 1992.  !

LER NO: 483/92-011 i

to PRELIMINARY O

-1 PRELIMINARY  !

mi>lmi(j 5;>t515l$ I III i [' $ '

! m=jal4l 5!Ni> >lN 4 4 ,

s = = = (  !!=i+ 314 4 134[ l

\ /

! m' = ; m l al l $ 9 f!4i* $ 4 ,k

• El$l  !

! NI l@lIl4 4 f i [4

! ( $ C'

$ l' Hl[ E E f s 4 9 *jaX  ! (in '

5 4 > $

4 a i e N 4 >lElg 5

-i..l_N W'd 4 ' D !'I E $

E %l > $l9l 5 ,

t

! > E E Ml8 g s 4 4 t' $ ? t i

>lElElMl l 5 4 C' C 4 4 l l i

i > Elst IgIY : s e ai s < a 5 > ElE l E l, ,,

B 414 4 $ $1 ,

'lN *!fl 4lC Cl4 $

E l

• lN

= > kl l CB'i

! a 4 4 4 a

• ClNik'*l4 C' i T > <5 'M >

x > < sm >

- *- x x. sis s s .

• E $ E I E E I > D >' N 8 E k

• k N H s ',= X Ff  !

~

i H E E

• 4 H )

5 N E' E 3 4 3 5 $ 44 W $ ( 8 1

M N N 5 5 5 I 4 M $ N 4l i M N s s s s s 'gA 4 C 's (

ng. 4. Annunciators failed after 0156 hours0.00181 days <br />0.0433 hours <br />2.579365e-4 weeks <br />5.9358e-5 months <br /> on Osober 17,1992 (cont.).

i LER NO: 483/92-011-11 . PRFLIMINARY 1

es

PRELBfINARY

~

0.1.5 Analysis Results i He conditional core damage probability estimated for the event is 4.3 x 108 The dominant core ,

damage sequence, highlighted on the event tree in Fig. 5, involves a postulated small-break LOCA with .

failure to initiate high-pressure recirculation. His analysis was performed using screening human error probabilities and with limited information concerning the activities that were in progress at the time of the event. As such, the analysis is considered conservative.

Additional information concerning this event is included in Augmented Inspection Team Report  ;

50483/92018 (DRP), 'Callaway Loss of Annunciators Event," October 16-19, 1992.

4 i i

P tocA RT arw ww woi HF9 op SEO END h no part l 4

, og l 71 CD 72 CD I Ok 73 CD  ;

74 CD ,

i o< i 75 CD l

76 CD j

77 CD  ;

i 78 ATWS 5 i i .

Fig. 5. Dominant core damage sequences for LER 483/92-011. .;

I LER NO: 483/92-011 l t

. 12 PRELB11 NARY t 1

l

PRELBf1 NARY .

. CoelTIcetAL CORE DAMAGE PROEASILITY CALCULATIDWS Event Identifter: 483/92-011 Event

Description:

Loss of esin control tmerd arrsneletors Event Date: 10/17/92 Plant: 'Cattaway 1 thAVAILA81LITY, DURATIQu= 18.5 NON-RECDVERABLE INITIATIbG EWKT PROSABILITIES TRAkS 7.6E LOOP 4.8E-03 LOCA 3.2E 06 MQUEu2 CONDITIONAL PROEA8!LITT S*S End Stats /Initistor Probabttity i CD i

TRAK $ 3.0E-06 LOOP 6.6E 06 LOCA 3.4E 05  !

Tote 1 4.3E 05 '

s ATVS TRANS 2.2E-06 LOOP C.0E+0D LOCA T.0F-08  ;

ToteL 2.2E-06 ,

I' SEQUENE CoelT10kAL pro & ABILITIES (PROSA3tLITT OROER)

SeqJence End State Prob N Rec **

71 (OCA *rt afw hPI MPR/-NPI CD 3.2E-05 T.5E 01 53 LOOP -rt/toop EMERG. POWER -efw/emers. power -porv.or.srv. chat t CD 4.0E-06 5.2E-01 KAL.LOCA ep. rec (st)  ;

11 TRANS -rt sfw pory.or.srv.chal t PORV.DR.SRV.RERAT -NPI NPR/ CD 2.4E 06 6.2E 01

-WI 72 LOCA -rt -afw pl CD 2.1E-06 6.3E-01 54 LOOP -rt/ loop EMERG. POWER -efu/eeerg. power -pory.or.srv.chell - CD 1.4E-06 5.2E 01 M AL.LOCA EP. REC to itAWS et ATWS 2.2E-06 1.2E 01 ,

• • non-recovery credit for edited case 4

5taJEW2 CONDJfickAL PROEABILITIES (SEQUEkCE ORDER)

• 5eSJence .f.nd State - Prob . N tec**

11 TRAKS -rt -efw pory.or.srv.chall PORV.0R.Sav.tESEAT -WI NPR/ CD 2.4E 06 4.2E-01

-NPI 18 TRAKS rt - -

ATWS 2.2E-06 1.2E-C1 53 LOOP -rt/ loop ENERG. POWER -afw/seerg. power -pory.or.srv.chall . CD . -4.0E-06 5.2E 01 KAL.LOCA ap. rec (st) l 54 LOOP -rt/ loop EMERG. POWER -afu/emerg. power +pory.or.srv.chall - - CD 1.4E-06 5.2E-01 i K AL.LOCA EP. REC 71 LOCA -rt afw *MPI ' Wit /-NPI : .G 3.2E 05 7.5E 01 72 LOCA -rt -afw uPI CD J2.1E-06 6.3E-01

• • non-recovery credit for edited cas.e -

Event Identifier: 483/92 011  !

LER NO: 483/92-011 13 PRELBiINARY

+.  !

~

' I PRELIMINARY i Note: For unevellebilities, canditionet probability values are differentist values which reflect the  :

added risk dae to fattures essecleted with an event. Parenthetical values indicate a redaction in  ;

risk compared to a slalter period without the esisting feltures. ~

EEGUEeM NCDCL: c:\sep\1999;purhoeet. cap  ;

BRANCN MODEtt c:\eep\1999\tattwy.stl.  ;

PROSASILITT flLE: . .c \eep\1989\pwr, bet 1. pro . -;

No Recovery Limit GRANCN FR!GUEeCIES/PeneAe]((T}[g~

Branch -System  ; son-tecoy . apr Fait i

TRANs '3.5E.06 > 4.1E-03 . 1.0E+00

-Branch modet _IRITOR '

Initiator Freq: 3.5E 04

• 4.1E LOOP .. 71.6E-05 > 4.9E-04 5.3E '

Branch madets INITOR Initietor f req: 1.6E 05 > 4.9E-04 LOCA . . . .

2.4E-06 > 2.3E 05 4.3E 01 > 7.5E-01 Branch model:

^

Ir! TOR '.

Inittster Freq: '2.4E 06 > 2.3E 05 rt '2.8E-04 1.2E-01 {

rt/toop 0.0E+00 1.0E+00 i EMEtc. POWER 2.9E-03 > 2.9E-03 8.0E-01 > 1.0E+00 ,

Branch Modet: 1.of.2 '

Train 1 Cond Prob: 5.0E 02 Train 2 Cond Prob: 5.7E 02 i afw - 3.8E-04 2.6E-01  ;

afu/smers. power 5.0E-02 3.4E-01 i MFW 1.0E+00 > 1.0E*00 7.0E 02 > 2.2E-01 1.0E-03 >

3.0E-02

• Branch model 1.of.1+opr  !

Train 1 Cord Prob: 1.0E+00 pory.er.ory.che t t 4.0E-02 1.0E+00 PotV.OR.spV.ttsEAT 2.0E-02 > 2.0E-02 1.1E-02 > 4.2E-01 Brench model: 1.0F.1 trein 1 . Cand Prob: 2.0E-02 porv.er.srv resest/emerg. power 2.0E 02 1.0E+00 ,

SEAL.LOCA 2.7E-01 > 5.5E-01 1.0E+00 l srench Modet: 1.or.1 Train 1 Cond Prob: 2.7E 01 > 5.5E-01 ep. rec (st) 5.8E-01 1.0E+00 1

, EP.tEC 2.5E 02 > 2.5E-01 1.0E+00 Branch modet: 1.0F.1 l 1

Train 1 Cond Prob

2.5E 02 > 2.5E-01 l

MPI 1.0E-03

• 8.0E 03 8.4E*01 -

aranch mode 1: 1.or.2 i

Train 1 Cond Prob: 1.0E-02 > 8.0E-02 i Train 2 Cond Prob: 1.0E-01 l 4PI(F/B)- 1.0E-03 > 1.T-03 -8.4E 01 1.0E-02 > l 7.0E 02 trench mode 1: 1.0F.2+cpr:

Train 1 Cond Prob: ' - 1.0E-02 Train 2 Cond Prob: -1.0E 01 .. --

MPR/ hPI ~

1.5E-04 > 1.5E 04 < 11.0EM- 1.0E 03 >

' 1.0E 01 .. .: . . .. ,

' trench modeIt' 1.0F.2+epr_ ' . . ..-

Train 1 Cond Prob: -

1.0E 02 -

l

~

Train 2 Cond Prob: . 1.5E 02-PORY.CPEN- . 2.0E-02 > 2.0E 1.0E+00- - 4.0E-04 >

5.0E-02 Branch model . t'.0F.1+epr I Train 1 Cond Preb 2.0E 02

• branch andet file-

• • forced . .

Event Identifier: 483/92 011

. LER NO: 483/92-011 1

g _ PRELBfINARY i

LICENSEE EVENT REPORT (LER)

.. cut, o. , '

oocirt .u .t. a, e.a a, I Callaway Plant Unit 1

" *' olslololel4 l: l 1 l or l o l 7 A Loss Of Main Control Board Annunciators Caused By Blown Power Supply Fuses During Maintenance Was Not Declared An ALERT Due To Lack Of System Knowledge

,-. e.. .. ,e. u .c. .

. .->. , e . e ,, , c+ e ..e,re. -m wo ..

aipHW DAY vi.AR YEAA l Arv. nonw DAY ytaA FAcm37vanames DOCKET wi.manencs s*0VD.tTtAt o t. 1 oIslololo! ! I 1lo 1l7 9 !2 9l2' -

ol1l1 -

olo 1l1 1l3 9l2 olslolo!ol l l e eef Tws ntroset as s.naaTTto Punswurt to Tws maouswwt rts or se com : acasse a.m er sw #se iss et to to scani ro4o6m 60.7 awawi 73.7 s aw  !

PowtA 20406wn tel 60.344sM11 Go.73WaH.t

- - 78.71tas {

trvat rict 1 l0 l0 remoswnismi so temici so.taama>=4 T OTwen shearree I somosweimi 1 4 wn.i

] so.tswcrisi f .nwa>w

] 60.?swcru.wsu so.nwa> .

- Ansoner e=n rs e, ew s.

f 20 acEwnml 60.73wa6ind &c.73wQteri VOL.UPfrMY  !

ver= arm cowf acT moa Tws up tiri m r -o f m~.r.

uau Thomas P. Sharkey, Supervising Engineer, Site Licensing

[

Coe*MITr one u=t som gAcm comesomfwt past6mf DEAcetatp per vwa arpo*T f ots 3l3l4 6l3l6l.l:l3l3l6 caust f s ree et e rni YpY v Ns caust e Tse L coempoutwT ra X I B J X P 0 3 5 N 4

l f  !  ! ! !  ! ! ! l l l l l l l SUPPLt thf at AtPo8tf EAPECTC not asym

cay , v taA cretcTe s6samsson vt3 99 eso. e.====ee DPtCTC SututLDo** DATO eso DATE(168 A.s~cT u i.ac - m=m_ _ . , . . . ,

j At 0100 CDT, on 10/17/92 during restoration from replacement of a failed power supply, all four

' field contact power supply output fuses blew causing all RK system Main Control Board (MCB) annunciatcrs to become inoperable. Because only 371 of 683 (MCB) annunciators lit, the licensed '

4 operaton incorrectly believed that those annunciators which had remained dark were operable, 4

Derefore, an ALERT was not declared as recuired by plant Emergency Action Levels. The fuses

] were successfully replaced at 0156. ne plant was in Mode 1 - Power Operations at 100 percent reactor power at the time of the event.

ne cause of the initial failure of the power supply was a short in the power transformer internal to

] the field power supply. During restoration following replacement of this power supply, a short '

occurred while removing jumpers, causing the fuses to blow. The operators failed to declare an  ;

AI_ERT because inadequate knowledge of the RK system led them to believe that some annunciators '

remained operable.

l Training will be provided to 7:rsonnel on the operation of the annunciator system. Actions to be taken in case of annunciator failures have been detailed in procedures. A modification will be evaluated to )

! (

improve the reliability of field power supplies and provide detection of power supply failures to the

operating crews.

t I

J.

LICENSEE EVENT REPORT (LER)

TEXT CONTINUATION -

7Acastym mt m OOCKET kWEEA 52 tra m eacg ni aAMafn he Callaway Plant Unit 1 ols!alolol4lsl sl2 .

oltli .

elo ol2 o* ol7 TEJ[? Of cisme esses e senaus. ese anseerus 8AC Som So&A'eH171 BASIS FOR A VOLUNTARY REPORT:

On 10/19/92, at approximately 1240 CDT, utility engineers reviewing plant operation data from 10/16/92 and 10/17/92, determined that, between 0100 and 0156 on 10/17/92, all of the Main Control Board (MCB) RK system annunciators (l) were inoperable. A phone call was made at 1320, on 10/19/92, in accordance with 10CFR50.72(b)(v) to report an event that resulted in a major loss of emergency assessment capability. This report is being made voluntarily to address root cause and corrective action for the loss of annunciators and the failure to declare an ALERT.

PL ANT CONDITIONS AT TIME OF EVENT:

i Mode 1 - Power Operations 100 percent reactor power DESCPJPTION OF EVENT:

At 1840, on 10/16/92, an annunciator (RK system) field contact power supply (2) failed, causing approximately 76 MCB annunciator windows to be lit. At 0058, on 10/17/92, the power supply was i replaced and all applicable annunciators cleared.

At 0100, durin 4 output fusesblew, (3)gcausingrestoration from the power supply replacement, all four field contac all RK system MCB annunciators to become inoperable. This resulted in 371 of 683 MCB annunciators to be lit. Although loss of all RK system annunciators is considered an <

ALERT under the plant's Emergency Action levels, the licensed operators incorrectly believed that those annunciators which had remained dark were operable. 'Ihe licensed operators were also not aware that all four power supply output fuses had been blown. Therefore, an ALERT was not declared on 10/17/92.

Troubleshooting by the Instrumentation and Controls (I&C) technicians revealed the four blown field power supply fuses. They were successfully replaced at 0156. Other fuses in the logic cabinets (4) of the annunciator system had also failed sometime daring the 0100 restoration, but were not initially discovered. Therefore,164 of the annunciators (those with reflash capabilities) remained inoperable, although the work document was signed off as complete.

l

LICENSEE EVENT REPORT (LER)

. TEXT CONTINUATION pery anaamg gis DDCLET muuttA E2) t f a avvw,f a te r emot i3 TEAA m Rty es,Mmem evo Callaway Plant Unit 1 olslolelol4lsl: sl2 .

oltl1 -

olo ol: o' ol7 TIAT e' one, apese a semiares. ese . - h4C Foren M4A aH178 During the day shift on 10/17/92, I&C technicians and the system engineer continued to troubleshoot what was originally believed to be individual annunciator window problems. At 1400, a logic power supply fuse was replaced, reducing the number of inoperable annunciators to 135. At 1800, an additional seven fuses in the logic power supplies were replaced. At 1937, all RK system annunciators were retested and verified operable. '

r ROOT CAUSE: l A. Failure to Declare an ALERT The failure to declare an ALERT at 0100, when all RK system annunciators were lost, can be attributed to inadequate knowledge by plant personnel of how the annunciator system functions. i Although the licensed operators involved with this event were aware that an ALERT should be called if all annunciators were lost, the fact that about half of the annunciators failed in an unlit state led the licensed operators to believe that those annunciators remained operable. There is no MCB indication of a totalloss of RK system annunciators. >

i Inadequate trammg exists on the annunciator system for engineers and Operations Department personnel. The lack of knowledge did not allow the determination that the blown fuses resulted in ,

a loss of all MCB annunciators.

)!

B. Equipment Failures The cause of the initial failure of the field power supply was a turn-to turn short in the power transformer internal to the field power supply. An evaluation has determined that a short which  !

occurred while removing jumpers following the power supply replacement caused the four field contact power supply fuses to blow. The logic power supply fuses were probably also blown as a result of this short.

C.  :

Contributing Factors Several other factors were determined to have contributed to this event. '

i I

1 E

~

, ilCENSEE EVENT REPORT (LER) t TF - J 1;uNTINUATION

.m m. n, oocm o mo, n. ~. r. . , ..ce m VEAA Medt AfW eaAepa ero Callaway Plant Unit 1 olslololol4lsl2 sl -

olilt -

olo ol4 on ol7 rw e . - - < uw.,nn

1. Lack of Communication.

Prior to the troubleshooting to replace the failed field power supply, there was no pre-job briefing between the operating crew and the I&C techrdeians, planner and engineer performing the work. In addition, the fact that all four power supplies fuses were blown was not conveyed to the licensed operators by the personnel working in the RK system cabinets located behmd the MCB.

2. Supervision There was no direct supe: vision of the I&C technicians during the power supply replacement. An engineer and a planner were providing technical assistance to the technicians, but their supervisory responsibilities were not clearly defined.

3. Work Controls A caution existed in the work package to warn personnel that a loss of all MCB annunciators would require the declaration of an ALERT. However, only the planner who prepared the ,

work request and the operating crew read the work package and were famihar with the  !

caution noted thereon. The specific fuses that were replaced during the job performance were not noted on the work completion form.

4 Retest Considerations

)

No retest was specified on the work document for the field power supply replacement. The  ;

e retest performed measured voltage across the field contact power supply outputs and performed a lamp test of the system. However, this did not reveal that the logic power supply fuses were blowm.

5. System Design The design of the annunciator system requires the power supplies to be connected in parallel.

Thus, temporary jumpers are required whenever one of the power supplies is being replaced.

The configuration also causes difficulty in troubleshooting the system, and tends to make individual logic power supply failures undetectable. This design is in part attributable to the fact that the annunciator system is non-safety related.

.l i

LICENSEE EVENT REPORT (LER)

. TEXT CONTINUATION '

..cun - n, ooce .m m na - .r. .

.m o.

rw - e euefut to Callaway Plant Unit 1 ol slo lolol4lsl: sl2 -

olil1 .

olo ols o.,ol7 i ron . - ==c uw nn CORRECTTVE ACTIONS:

A. Failure to Declare an ALERT Training will be provided to Operations and Engineering personnel to assure a greater level of expertise on the operation of the annunciator system. Actions to be taken in case of annunciator failures have been detailed in Operations Department procedures.

B. Equipment Failures A modification will be evaluated to improve the reliability of RK system field power supplies, improve DC power redundancy and provide detection of power supply failures to the operating crews.

C. Contributing Factors l

The circumstances surrounding this event have been reviewed with the individuals involved to j ensure management expectations are understood. In order to enhance future power supply replacement work practices, the following actions are being taken:

j

1. A guidance has been developed for retesting of RK system power supplies. This will ensure l

that Engineering personnel are contacted to determme the scope of retest.

l

2. Requirements for direct field supervision of critical maintenance activities will be clarified. i i

3 The requirements for pre-job briefings for critical maintenance activities will be defined and communicated to appropriate personnel.

l

4. Work completion documentation nWstM with the fuse replacement has been upgraded to l document the fuses which were replaced. J l

SAFETY SIGNIFICANCE: i 1

Compensatory alarming and non-alarming indications were available to the control room operators throughout the event. These included:

a Engineering Safety Features (ESF) Status Panels with alarm indication for safety-rtlated valves, pumps, and breakers which enables the operators to assess ESF sys+em status and perforrrance.

l l

LICENSEE EVENT REPORT (LER)

TEXT CONTINUATION

, ~ , . _ , , , , -- .

Callaway Plant Unit 1 olslololol4l l2 sl2 -

oltli -

olo ole ce ol7 a Safety Parameter Display System (SPDS) to assist the operators to assess the ouet and severity of accident conditions.

l s Digital Rod Position Indication, Control Rod Group Demand Indication, Power Range Nuclear Instruments, and the automatic Reactor Protection System (RPS) to enable the operators to initiate a manual reactor trip, if required, or to be made awsre of an automatic RPS actuation.

s Partial Trip Status Panel to indicate a potential or actual RPS or ESF actuation signal is present.  ;

a Permissive / Interlock status panel for OT delta T rod stops, overpower stops, steam dump arming, and condenser availability.  !

s Radiological Release Information System (RRIS) and the RM-11 Radiation Monitoring panel were available to assist the operators in monitoring meteorological data and radiological monitoring systems in the case a release has occurred.

s MCB analog indications of power, pressures, temperatures, levels, flows, stive positions, etc.

to assist the operators in controlling the various plant systems.

m Plant computer CRT displays and alarm typer for approximately 2,836 field input computer points.

1 Corrective maintenance on the annunciator system was near completion at the time of the unplanned loss of the field contact power supplies. The operators had previously undergone a crew brief of this planned maintenance and were aware of the risk oflosing additional annunciators. Therefore, they  !

lud a heightened awareness of the MCB indications and a desire to maintain steady state plant  ;

, conditions by avoiding any distractions or operator induced transients. Due to the loss of the annunciators, the licensed Shift Supervisor delayed the scheduled weekly testing of the main turbine to i preclude any change in the plant's steady operation.

Even though the annunciator system is not safety related, the importance of the annunciators to the operators is recognized. The loss of non-safety related annunciators alone for this event did not pose a threat to the public health and safety.

i PREVIOUS OCCURREN.CES: ,

None. l l

l i

- r  !

~

ilCENSEE EVENT REPORT (LER)

TEXT CONTINUATION

,cus, ni ,,.-.e....

I " em om on sw m sua

..c u.

arjep are Callaway Plant Unit 1 o!slololol4lsl sl2 .

olSlt -

olo ol7 o' ol7 sure ~ . - . .u m uunan FOOTNOTES:

The system and component codes li;ted below are from IEEE Standard 805-1984 and 803A-1984, respectively.

(1) System - IB, Component - ANN '

(2) System - IB, Component - JX Manufacturer - PANALARM i t

Model #70-IDC-2 (3) System - IB, Component - FU (4) System - IB, Component - CAB 4

I t

i GB

-f. uavlTED STATES f'pe t8Cg%, NUcLEAn REGULATORY COMMisslON .

j((, 7

-  : W 3 ;, naciou in

aa?n$$.',$

Docket No. 50-483 NOV I O 1952 g

[p Union Electric Company ATTN: Mr. Donald F. Schnell

[> '

Senior Vice President - Nuc ear ' }I' ' -

Post Office Box 149 - Mail Code 40' .

St. Louis, MO 63166 V

Dear Mr. Schnell:

The enclosed report refers to a special onsite review by INRC Augmented Inspection Team (AIT) October 19 through 25, 1992, relativ@ to the October 16, 1992 loss of control room annunciators and failure to recognize the operational effects of the annunciator loss at the Callaway Nuclear Power Station. The AIT was composed of Messrs. R. A. Westberg, B. L. Bartlett, and T. D. Reidinger of this office; Mr. R. A. Spence of the Office for Analysis and Evaluation of Operational Data (AE00); and Messrs. L. R. Wharton and F. P. Paulitz of the Office of Nuclear Reactor Regulation (NRR). The report also refers to the followup activities of your staff and to the discussion of our findings with Mr. D. F. Schnell and others of your staff at the conclusion of the inspection.

The enclosed copy of our AIT report identifies areas examined during the inspection. Within these areas, the inspection consisted of a selective examination of procedures and representative records, observations, and interviews with personnel.

The AIT was fortned to gather information on the event. Specifically, the AIT examined licensee response to the event, root causes of the loss of annunciators and other equipment failures, and event classific1 tion and reporting.

The AIT determined that the actual safety significance of the event was minimal. No significant operational safety parameters were approached or exceeded. There was no release of radiation. The AIT identified four root causes for the event. Removal of any one of these causes would have significantly attigated the event. The root causes were as follows:

1. Poor Communications / Teamwork. 'L '

2. Lack of a Questioning Attitude / Complacency. (([/[f

[ l,$ .

3. Inadequate Knowledge of Annunciator System. g $ gl.2 d.i d 4 *- hi

4. Less than Adequate Work Performance. @/f,a j,m/ q i

~

c7e m u c. .Uww 7 f"<~T a.4 g./y,. - hwkSy 4 .. .

p w . a'.L 4 n; , . 4L M'f Ly*; , 9'4 i

~ cL0t~M-# YAb'

S' '*" ' W n , ,

,.u m, s

. . i l

Union Electric Company -

2-It is not the responsibility of an AIT to determine compliance with NRC rules and regulations or to recommend enforcement actions. These aspects will be reviewed in a subsequent inspection.

~

In accordance with 10 CFR 2.790 of the Commission's regulations, a copy of this letter and the enclosed inspection report will be placed in the NRC Public Document Room, se will gladly discuss any questions you have concerning this inspection.

Sincerely, r

ML Y William L. Forne'y, puyDirech Division of Reactor Projects

Enclosures:

1. Inspection Report No.

50-483/92018(DRP)

2. October 20, 1992 Confirmatory Action Letter

3. AIT Charter

4. Personnel Contacted See Attached Distribution h

e

. . j NOV 10 532 '  ;

Union Electric Company ~

Distribution cc w/ enclosure:

G. L. Randolph, Vice President, Nuclear Operations  ;

J. V. Laux, Manager Quality  :

Assurance  :

T. P. Sharkey, Supervising Engineer, Site Licensing l DCD/DCB (RIDS) i OC/LFDCB  ;

Cesident Inspector, RIII  :

Region IV  ;

Resident Inspector, Wolf Creet K. Drey f Chris R. Rogers, P.E.

Utility Division, Missouri  !

Public Service Commission l Gerald Charnoff, Esq. l R. A. Kucera, Deputy Director, 1 Department of Natural Resources  !

The Chatrisan Coarnissioner Rogers ,

Connissioner Curtiss  !

Commissioner Remick l Commissioner de Plangue ,

D. C. Trimble, Jr., OCM i D. A. Ward, ACRS  !

J. M. Taylor, EDO .

J. ' H. Sniezek, DEDR  :

G. E. Grant, EDO T. E. Murley, NRR 1 J. G. Partlow, NRR J. W. Roe, NRR  !

J. A. Zwolinski, NRR l J. N. Hannon, NRR  ;

W. T. Russell, NRR 3 C. E. Rossi,-NRR ,

j R. L. Spessard AE00 i E. L. Jordan, AE00  :

4 A. B. Davis, RIII i C. J. Paperiello, RIII  !

1 M. W. Hodges, RI A. F. Gibson, RII' .i S. J. Collins, RIY  :

. K. E. Perkins, RV )

I a

1 1

)

.. - - _ . - _ _ _ _ _ . ~-_____

I

. j e

4 1

i I

I 1

l l

l O

k AUGMENTED INSPECTION TEAM REPORT .

I,

^ -' , ,4 4 -

4 - . #' , .

' m ,,j . :7 ~> ml:t..-.e . . . . . .: :_gwe' .m.a: . . 3_we, , pf .,w,s.4.<-

a,? ;;>

. > . s . ; %. . ~ 7, a m.a..+.,"a~

a. . .

.x. ; ?*,.,

- , v~,.c--* -

..~,.%,. g .,% e s.

, v v . .,;;.m,:... .:

.,Q G. . , <.r. , .

,s: 4, ...,4. g , .:s ;y: . ~;< r wh. a+ : (, p. ;:,,,2.s;.9,;,,

w.- .~v + c. p;m..

- , . . ,~..,3g,.,.,,.

c;~,. : y"e$y;:: a,;>N.%;q;.,, .g .,.;;. n.x gr ,.< r .,+w.:

A(- .?- ** Q s-.p:. 2.G

^

v * ~' " w m * ;p M,;i 'f;m m;,w;x.,,;3 r.y.w:w .

. . m,. . r:e

4 m n w g.a

. - . . . WL , .

, ~;*g.eg=C r " -,.. Y- .' .- ,:

a ; s a, m m ,.; g ,

- x " e_,w> : m ~ x

+

47, o., ,:"o.

n-> Muns-

? r. .:. ^ w~?

=

,,  ;<;a ,-  ;,, ;*3.s s ,

s , - ' .y,; ,

a g ,- ,

v +

s -

f i

i CALLAWAY LOSS OF ANNUNCIATORS EVENT OCTOBER 16 - 19, 1992 i a

INSPECTION REPORT NO. 50-483/92018(DRP) 6

)

i

.t n ,y "

] _A

. . 5 L

U.S. NUCLEAR REGULATORY COMISSION REGION III 9

Report No. 50-433/92018(DRP) License No. NPF-30 Docket No. 50-483 Licensee: Union Electric Company Post Office Box 149 - Mail Code 400 St. Louis, MO 63166 Facility Name: Callaway Nuclear Power Station Inspection Conducted: October 19 through 25, 1992 Inspectors: B. L. Bartlett, DRP T. D. Reidinger, DRS R. A. Spence, AEOD L. R. Wharton, NRR F. P. Paulitz, NRR Approved By: Wdb N II10f92-R. A. Weltberg, Team LeaderM Date Plant Systems Section Approyect By: h A-

. N."Jackiw, Chief II!/o[9E Date eactor Projects Section 3A Insoection Sumary:

.Insoection on October 19-25. 1992 (Recort No. 50-483/92018(DRp))

Areas Insoected: Special Augmented Inspection Team I conducted in response to the loss of control room ann (AIT) inspection unciators at the Callaway Nuclear Power Station on October 16, 1992. The review included validation of the sequence of events, deterutnation of the root cause for the annunciator loss and equipment failures during the event, evaluation of licensee response to the event, and evaluation of the licensee's event classification and reporting.

Results:. No operational safety parameters wre approached or exceeded. The i AIT concluded that the root cause of the initial power supply failure was '

random failure of its power transformer; the root cause of the subsequent blown fuses was personnel error; and the overall root causes for the event were poor comunications/ teamwork, lack of a questioning attitude / complacency, inadequate knowledge of annunciator system, and a less than adequate work performance.

. _ _ _. ... _ _ . ~ _ _ _

t

?

TABLE OF CONTENTS ,

i 1.0 Introduction ....................................................... 1 i

1.1 Event Summary ... . . ... .. . ...........................................

1

~

1.2 AIT Formation ....................................................... I r 1.3 AIT Charter ........................................................ 1 I

l 2.0 B a c ka round In fo rma t i o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.1 S y s t em De s c r i e t i o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 '

2.2 Precursors to the Event ............................................ 3 s

l 2.3 Seou e n c e o f Even t s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 l 3.0 E v e n t R e s o on s e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 .

3.1 Doerator Resoonse .................................................. 5 i 3.2 M a n a c e r i a l P e rfo rma n c e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.3 Human Performance Issues ........................................... 9 4.0 Eo u i ome n t F a i l u re s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14  !

4.1 An al ys i s o f Root C au s e Det e rmi n at i on . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

[

1 4.2 Corrective Action .................................................. 15 i

5.0 Eve nt Cl a s s i fi c a t i on and Reoo rt i na . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 I

1 I

6.0 Safety Stanificance ................................................ 18 1 7.0 Overall Conclusions ................................................ 18 I

7.1 C au s e o f Eou i pment Fa il u re s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 t

[

7.2 Ro ot C a u s e s for t he Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 r 8.0 Ex i t I n t e rv i ew . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21  !

t i

t I

?

t i

'I i

i t

1 1

5 e'

tt - e . - - _ _ _ - _ _ . - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ , _ _ _ . _ _ . _ _ _ _ _ _ . _ _ _ _

. Eetails ,

1.0 Introduction 1.1 Event Summarv On October 19, 1992 at approximately 12:40 p.m., Callaway engineers reviewing operational data from October 16 and 17 discovered that at 1:00 a.m. on October 17, all main control room annunciators had been inoperable and the operators had been unaware of this condition.

1.2 AIT Formation On October 19, 1992, senior NRC managers determined that an AIT was warranted to gather information on the loss of annunciators, partial loss of annunciators, and failure to recognize the operational effects which occurred during the event. An AIT was formed consisting of the following personnel:

Team Leader: R. A. Westberg, Team Leader, Division of Reactor Safety (DRS)

Team Members: B. L. Bartlett, Senior Resident Inspector Callaway Site, Division of Reactor Projects R. A. Spence, Reactor Systems Engineer, AE00 i T. D. Reidinger, License Examiner, DRS i

L. R. Wharton, Licensee Project Manager Callaway Site, i NRR F. P. Paulitz, 11C Engineer, NRR One member of the AIT, the Senior Resident, was on site on October 19, 1992.

The full AIT arrived on site October 20, 1992. In parallel with formation of the AIT, RIII issued a Confirmatory Action Letter CAL) (Enclosure 2) on October 20, 1992, which confirmed certain licensee (actions in support of the team inspection.

1.3 AIT Charter

~

A charter was formulated for the AIT and transmitted from W. L. Forney to R. A. Westberg on October 20,1992, (Enciciure 3) with copies to appropriate i EDO, NRR, AE00, and RIII personnel. The AIT's objectives were to: (1) I conduct a timely, tho. rough, and systematic inspection related to the event, ,

(2) assess the safety significance of the event and communicate to Regional and Headquarters management the facts and safety concerns related to the event  ;

such that appropriate followup actions are taken, and (3) collect, analyze, and document factual information and evidence sufficient to determine the cause(s), conditions, and circumstances pertaining to the event.

1 1

I 1

The AIT completed its charter and was terminated on Saturday, October 24 1992.

2.0 Backaround Infomation 2.] System Descriotion .

The annunciator system was designed by the Riley Corporation. The system is designed to monitor 1400 alars points using field contacts which either open or close, to alert operators in the cor. trol room by illuminating an annunciator window and sounding an audible alare. Individual alars points ,

that are grouped on a system basis also feed the plant computer for display on '

the cathode-ray tube and the alars printer.

l l

FELS 848. m v N a.s. we

& l

@ f f n

== ' = = m. m a=> ame

• ' ~

/gw Wvm ,,,

lT

. a 9 ~

yw 3

=nn.

E.

g ==. / e"1%'

. ";;lP" .

L. . /

i T*

l h.- =,""-

- E -::: _ a

• = es- k* .%,"$' j s  :. W, -  ;

==l -

1 f ,

an e ..

s

. e '

Annunciator Power Supplias l I

The system has four power supplies connected to a 125Yde station battery to power the 1400 field alarm contacts. These power supplies have common 1 2

4 l

1

~ (parallel connected) inputs and outputs, and each power supply input and output is protected by a one ampere " slow blow" (delaye'd opening) fuse. There are also 14 logic power supplies that receive their input power from one of  !

two 125Vdc station battery systems, one of which is conson to the field contact power supplies discussed above. The logic power supplies prov.ide five  !

different voltages to the system and each has a protective fuse associated with its voltage. None of the fuses (78 total) have local indication or indicating lights to monitor their operability.

2.2 Precursors to the Event On the evening of October 16, 1992, the plant was at 100% power,1224MW electric. No major plant evolutions were in progress and no major pieces of plant equipment were out-of-service. No technical specification action statements were in effect.

The AIT's charter limited the inspection to this one event. However, the AIT noted that previous events of loss of almost all of the annunciators had occurred. There have been 12 previous power supply failures during the last nine years that caused partial losses of operability of the annunciator system.

2.3 Secuence of Events In order to validate the sequence of events associated with the loss of annunciators, plant computer, related power supplies, and fuse failures, the AIT conducted interviews with licensee management, operations personnel, and instrument and control (I&C) personnel cognizant of the event. . Licensee documentation and event review meeting summaries were also reviewed to determine the actual sequence of events.

At 6:40 p.m., approximately 76 annunciators illuminated. The control room operators immediately verified that no plant trip, transient, or other evolution had caused the large number of alarus. This condition was initially diagnosed as either a blown fuse or a power supply probles. The shift supervisor (SS) notified the I&C shift technicians and requested engineering support from the Engineering Duty Officer (EDO). At approximately 7:00 p.m.

the plant sanager (MCP) was also notified (he was still on-site). At 7:26 p.m., the two I&C technicians replaced one fuse on logic bay power supply No.

RK045E1. However, the fuse replacement did not correct the problem or change the status of any of the annunciator windows.

The 1&C technicians identified a problem with the Multiplex (NJX) cabinet field power supply No. 2 (RK045DI). The 1&C planner reported on site at 8:45 p.m. and began to research previous work requests (WRs) related to annunciator problems. The backup system engineer was contacted and arrived on site at g:00 p.m.. The I&C technicians obtained a replacement power suppl warehouse and bench tested it in the I&C shop for about one hour. yThefrom the planner and system engineer met in the back of the control' room at the power supply cabinet to view the degraded power supply, discuss the replacament and any necessary installation precautions. The 11C planner and backup :ystem engineer continued to discuss the power supply installation with the 11C '

3 e

a

' technicians in the I1C shop. The 11C crew (planner, backup system engineer, and technicians) reviewed drawings and previous WRs to develop an appropriate jumpering configuration. -

At approximately 11:00 p.m., the sidnight (OWL) shift relieved the 3:00 to J 11:00 (PM) shift. The OWL shift I&C technicians were briefed on the WR work instructions by the 11C planner. At 11:15 p.m., the ILC crew entered the control room to obtain SS approval for the WR. The OWL shift SS verified that a specified caution was on the WR and authorized the work. The caution stated if all four power supplies were inoperable that an Alert should be declared.

The SS contacted the MCP and informed his of the plans to replace the failed field power supply.

On October 17,1992 at 12:58 a.m., field power supply No. 2 was replaced and all annunciator window lights cleared. At 1:00 a.m., a short circuit occurred coincident with removal of the output jumpers from the terminal blocks.cf the power supply. The short circuit caused fuses to blev on all four field power supplies resulting in a loss of the main control roos annunciator system.

Numerous annunciators (approximately 360) were illuminated. Also affected were numerous plant computer alarms. Fuses for the field power supplies were obtained from the ILC shop and the previously failed power supply. During replacement of the fuses, there was a problem which caused the fuses to blow l again at 1:24 a.m.. Additional fuses were obtained from the warehouse, and at 1:55 a.m., the I&C technicians successfully replaced the four blown fuses in the field power supplies and restored power to the main control roos annunciator system. Upon restoration of power, the illuminated annunciators ,

i cleared and the critical probless with the system were considered corrected. l The operations crew perfomed lamp tests on all the annunciator panels, which l they assumed verified operability of the system. The SS signed the WR, l indicating completion of work on the annunciator system.

The OWL shift crew continued to observe anomalous annunciator operation at this time. The SS considered the remaining annunciator problems sinor, which could wait to be analyzed on the 7:00 a.m. to 3:00 p.m. (AM) and PM shifts. j i

At approximately 7:00 a.m., the AM shift relieved the CWL shift. Operating crews were told to closely monitor control panels due to continuing annunciator problems. Sometime between 7:00 a.m. and 7:30 a.m. on October 17, l 1992, the control ,mos called 11C and indicated that the annunciators were not normal because a desineralizar alars came in and would not stop flashing when it was acknowledged. At the same time, a reactor coolant system level alam whi:h is normally defeated during power operations had also come in. At 8:00  !

a.m., the MCP called the SS concerning the status of the annunciator probles. l Discussions were held regarding the need for and authorization to call in I additional operations and engineering support to resolve the annunciator problem. The SS contacted the supervising engineer and IEC technicians to again troubleshoot the annunciator system. During the remainder of the shift, troubleshooting of the annunciator system continued.

At 9:32 a.m., operations can' ducted a lamp test of the annunciator windows and observed a panel with only half bright illumination. At 1:00 p.m., the IEC senior system engineer identified a bad logic power supply which was 4

l

suosequently replaced. Operations performed additional local equipment tests to verify the operability of the various trouble annunciators. However, the associated alarus did not activate.

At 4:30 p.m., ItC determined that five additional logic power supply fuses had been blown. These fuses were assumed to have blown at.1:56 a.m. when the four field power supply fuses blew because they were all connected (powered by the same !?5Yde station battery). The fuses were replaced at about 5:00 p.m. and all logic power supplies were verified by a testing scheme developed by the senior system engineer using input relay and driver logic cards. Testing continued until 7:37 p.m. and the event ended when the control room annunciator system, including the plant computer inputs, were verified as completely restored and fully operable.

Based on their review, the AIT determined the time line provided by the licensee associated with the annunciator loss was complete and accurate.

3.0 Event Reseense 3.1 Ooerator Resconse To determine what actions the operators took in response to the event and the suitability of these actions, the AIT reviewed plant logs, appropriate plant emergency and normal operating procedures, and interviewed the operators involved in the event.

Operator actions during the loss of all annunciators were considered less than adequate. Since the operating crews did not comprehend the assessment capability loss that occurred, they respcnded to the event in a less than systematic manner, without the use of all available information or procedures to implement compensatory operator actions. As the operators had not been <

adequately trained in the partial or total loss of the annunciator logic, annunciator power supply systems, or the plant computer system, the operators experienced momentary confusion during the cascading annuaciator failures on the various panels. Thus, the reactor operators (Ros) were not aware of the ,

l extent of the loss of the annunciators, due to the number of invalid computer 1 alarms that were printed and only a portion of the annunciator windows illuminating. The operators monitored redundant control board instrumentation inmediately after the losses of annunciators at 6:40 p.m. on October 16 and at 1: 00 a.m. and 1:24 a.m. on October 17, 1992. ,

i Although no procedural guidance existed, the OWL shift SS appropriately directed that no. power level changes be initiated after the loss of annunciators equipment because they were concerned about being able to monitor plant status. The two Ros, on their own initiative, increased the frequency of their respective main control board panel walkdowns after the initial annunciator loss. However, this was not performed continuously, nor were additional Ros assigned to augment the on-shift Ros.

The operators' inadequate unde'rstanding of the annunciator systas resulted in the following: -

j 5

4 e

l O

~

a not declaring an alert when all annunciators were lost devoting insufficient manpower to corrective actions not implementing available abnormal procedures on the loss of the plant computer when plant computer was partially lost continuing a liquid radwaste release during the event Signing the WR as completed while 163 annunciators still remained inoperable a

devoting shift management resources to a 34SkV line tag out  !

performing turbine stop valve surveillance testing during the event ignoring some plant computer alarus not taking adequate control room or plant compensatory measures devising unapproved, informal, and ad hoc annunciator functional testing which resulted in a false impression of the operability of the annunciator system, when in fact, 163 annunciators were inoperable.  !

3.2 Manacerial Performance 3.2.1 Initial Shift Sueervision involvement When it was concluded that there was a probles with the annunciator system, the SS called I&C and the MCP. Normally the SS would call the EDO but since he had still might seenbe theon MCP in the control room a, bout an hour before he thought he site. Thus, when he called the MCP and informed him that approximately room.

76 annunciators were inoperable, the~ MCP came to the control The SS then called the EDO at home and infonned him of the situation.

During this time frame, the SS called the engineering duty supervisor and requested that an I1C engineer be sent to the site to assist the !&C technicians.

When the 11C technicians determined that the'125Yde field contact power supply  :

had failed, the SS requested that an 11C planner be sent to the site to help '

plan any WRs needed to replace the power su,pply. The IEC planner arrived on site and began to research previous WRs. As part of a documentation review on the computer, the 11C planner found a previous WR for replacing the power supply.

it.

It was still on the computer when the PM shift SS walked by and read At about this time, the next shift operating crew was arriving to take the OWL shift. The SS informed the OWL shift SS of a caution in the previous WR which stated that if all four power supplies were inoperable that an Alert should be declared. The OWL shift SS informed the ILC planner that when the new WR was writttn that he wanted a similar caution placed in it.

l 6

l I

When the power supply fuses blew at 1:00 a.m., the SS went to the back of the control room to find out what had occurred. He was informed that they had

' blown some fuses." No one informed the 55 which fuses had been blown or what "

the effects of this would be; however, the SS did not ask for this infomation.

When the EDO and the MCP had talked with the PH shift SS, they had requested that they be notified of any change in status. However, this request was not recorded in the turnover log. This information was passed on to the OWL shift I

SS during the shift turnover. The OWL shift SS failed to notify either of these individuals of the additional annunciators that had become inoperable.

During the shift turnover briefing, the OWL shift SS had directed his R0s to pay additional attention to the control boards and delayed performance of a scheduled surveillance. However, when the additional annunciators became .

inoperable at 1:00 a.m., no additional personnel were brought in to observe the control room boards, no plant announcement was made to alert plant personnel to pay additional attention to plant status, and the equipment operators were not directed to increase monitoring of their assigned equipment status.

The SS and the crew did evaluate plant conditions in an effort to determine whether they met the conditions for an alert status. Due to the misconception that only one half of the annunciators were inoperable since only half were illuminated, the SS decided that they were not in an alert. The OWL shift STA also joined the discussion of emergency action level (EAL) conditions and

' agreed with the SS that an alert level was not reached. However, the STA accepted the hypothesis that only one half of the annunciators were inoperable

. without additional independent verification. The STA also did not discuss what equipment was affected with I&C, review the annunciator system, or 1

question which annunciators were affected.

The licensee's internal prob'lem identification and resolution system is known ,

as the 505 (Suggestion, Occurrence, Solution) program. The SOS program is utilized by the licensee to identify opportunities for improvement, ecployee {

safety concerns, and issues of regulatory significa.nce. The SS should have generated a SOS following the failure of the initial power supply and for the annunciator problems.  ;

l 3.2.1.1 Conclusions t

a. The SS did not understand the operation of the' annunciator system, 1 and that unilluminated annunciators could be inoperable (except i for one RO on the PM shift, none of the individuals involved in '

this event were knowledgeable in the failure mode of the

' annunciators). Both the SS and the STA were convinced that since some annunciators were illuminated, some power was still available to the annunciator syste.m.

I

b. The SS lacked a questioning attitude. He did not fully pursue the question of which fuses had blown when he talked to'the IEC

technicians.

7 I i

i I I l

l

1 1

c. The OWL shift SS failed to notify either the EDO or the MCP of the  ;

additional annunciators that were inoperable after the shift change,

d. The inclusion of the caution in the power supply replacement WR I was a positive initiative; however, the misconception on how the annunciator system worked and the lack of a questioning attitude negated this action. j

e. The SSs should have directed that 50Ss be written documenting the l night's events.

f. The shift logs did not identify the need to notify the EDO and the MCP of changing plant conditions 3.2.2 Subsecuent Plant Manaaement involvement following the initial power supply failure at 6:40 p.m. on October 16, 1992, the SS called the MCP who was still on the site. The MCP went to the control room and received a briefing. The MCP was infonned that a field contact power supply failure had occurred but that only about 76 annunciators were affected.

The SS and MCP concurred that the indication for an emergency classification of alert was not satisfied. The MCP discussed the situation with the SS and briefly observed the I&C technicians performing troubleshooting activities. l The MCP then left the control room and eventually left the site. The SS '

called the EDO at his residence and informed his of the current plant status. l The decision was again reached that the energency action level for an alert i was not satisfied. Both the EDO and the MCP specifically requested that they l be informed of any change in plant status. This information was passed on verbally from the PM SS to the OWL shift SS. i 4

At approximately 11:30 p.m., the MCP called the SS and was given a status on the replacement of the failed power supply. The MCP informed the SS that if any additional assistance was required that he should go ahead and call them in. .

i Between 8:00 and 8:30 a.st, on October 17, 1992, the MCP called the SS to get l an update. The MCP was informed that the power supply was replaced and that  !

during the replacement approximately 360 annunciators had come in. When the 1 MCP learned that the EDO had not been informed of the night's events he ]

directed that the EDO be informed of the annunciator status. The MCP then i offered additional help to the on-shift staff but since the annunciators had l been

• repaired,* the SS did not perceive a need for any additional help. When '

the SS called the EDO, he informed him of the current plant status but neglected to inform his of the approximately 360 annunciators that had been lost, since he thought the EDO had previously been informed. There was no additional discussion with the MCP or the EDO concerning whether an alert should has* been declared.

No further plant s.anagement involvement occurred until October 19, 1992.

Discussions were held in the regular morning status meeting at 6:45 a.m.. As a result of those discussions the licensee decided to meet with some of the 8

l l

l

. . l

~ individuals involved and gather more information. This meeting was held at 10:00 a.m. and was attended by the MCP and the EDO. During this meeting it was determined that it was very likely that all control room annunciators had  !

been inoperable from 1:00 a.m. to 1:56 a.m. on October 17, 1992. Subsequently  !

the licensee made a final determination that the annunciators had indeed been l inoperable and made an Emergency Notification System phone call to the NRC. I 3.2.2.1 Conclusions

a. The lack of knowledge on the annunciator system which existed in i the shift crews also existed in plant management.

b. The shift crew's failure to keep the EDO and to a lesser extent the MCP fully informed of the status of the annunciators contributed to management's failure to realize the extent of the situation.

c. The MCP and the EDO repeatedly offered additional assistance to the on-shift crews. In addition, management authorized the SSs ,

involved to call out any needed assistance without additional management approval. The SSs involved repeatedly declined the offer.

d. Even though the licensee believed that 50 percent of the annunciators were inoperable, additional operaters were not assigned to verify plant conditions. When the SSs failed to request additional personnel to perform this verification, plant management should have directed that it be performed.

3.2.3 Onsite Review Committee There was no Onsite Review Committee (ORC) involvement. This event never got to ORC because an alert was never declared; therefore, ORC never got a chance  ;

to review this incident, immediately prior to, during, or inmediately following the event.

3.3 Human Performance Issues The were a number of latent factors identified during the event. A discussion of these factors follows. - -

3.3.1 , Teamwork and Consnunications The' licensee had developed a teamwork training program, T61.TEAN.8, with the ,

National Academy for Nuclear Training, in May 1990 to help operators recognize their individual and team strengths. All operators had received this training to enhance control room teamwork, nurture the control room team culture, and to strengthen control room team performance. However, the teamwork among the operators and between the operators and the I&C personnel during this event was less than adequate. .

M

Dissenting RO concerns regarding the decision not to declare an alert were not dealt with appropriately by two SSs. The PM shift SS reportedly told an RO that they were not in an alert, but would not provide a rationale or discuss .

it further. The OWL shift SS repeatedly convinced his R0s that declaring an '

alert was not necessary. The two Ros who were concerned that the intent of the EAL regarding the declaration of an alert on the partial loss of annunciators did not forcefully present or pursue their concerns with the SS, when more than half of the annunciators were out-of-service.

There was less than adequate consunication between control room operators and the 11C technicians and systems engineer, control room operators and equipment operators, and the S5s and plant management. For example, the I1C technicians and backup system engineer did not clearly inform any of the operating crew that their actions resulted in the blowing of all the output fuses, which then rendered all four field power supplies inoperable. Thus, the Ros were not aware of the extent of the loss of the annunciators. The Ros were not always informed of when I&C was going to rform work that affected the control board annunciator responses. j The control room operators did not communicate the extent or signif'icance of the loss of control room annunciators to the equipment operators. The equipment operators were rarely used to verify the validity of annunciator and computer alarms and were not directed to increase the monitoring frequency of plant equipment. l The SS's failure to recognize the complete loss of annunciators at 1:00 a.m.

October 17, 1992, resulted in an alert not being declared and licensee management, the NRC, and other government agencies not being notified. This failure also resulted in a lack of managerial and technical expertise available to adequately address the loss of annunciators.

There was a less than adequate job pre-briefing on the annunciator WR by the backup system engineer, the 11C planner, or the SS. For example, the 11C technicians and the backup systes engineer had not read the WR. However, the caution, discussed.

relative to the loss of all power supplies requiring an alert, was During the day shift annunciator light testing on October 17, 1992, a R0 determined that four annunciator panel sections illuminated'at only half intensity. Although the RO informed the control room supervisor, this information was not coanunicated to the 11C personnel or SS for approximately three hours, during which time the operators thought that only five annunciator windows were out-of-service.

3.3.2 Cem and and Control .

The SS on the October 17, 1992 OWL shift did not exhibit a sufficiently.

questioning attitude. He did not adequately question or take charge of the.

11C personnel to determine which power supplies had been shorted out. He did not direct continuous monitoring of main control board indications. .He did not direct equipment operators to continuously monitor the plant equipment.

He did not inform the plant management or the NRC of the annunciator failures. i 10 I

e

r

. .. }

t

~

- The OWL shift SS did not use the dual role STA as an STA on the annunciator problem. I Instead the STA perfomed as a field supervisor on an unrelated turbine surveillance procedure. .

1 t

' There were also problems on the October 17th AM shift. For example, the R0s independently developed an unapproved ad hoc annunciator functional test without management oversight. However, the control room supervisor with the SS's approval did re-initiate I&C repair efforts after learning that five i annunciator windows showed invalid alarms.

The I&C group on the OWL shift was composed of two technicians, an !&C engineer, and an I&C planner. There was no management direction as to the single point of accountability or contact for the operators. The technicians ,

I had different opinions of who was in charge (system engineer or senior I&C t technician); consequently, no one infomed the operators of the loss of all four field power supplies, despite the caution discussed at the beginning of  !

the work. Contributing to the failure was the misconception in the operation i

of the annunciator system; !&C personnel convinced themselves that since some j

annunciators were illuminated some power was still available to the annunciator system. Therefore, they.did not ade  !

i unilluminated annunciators were receiving power.quately question whether the  !

3.3.3 Procedures i

All the operators interviewed indicated that they did not use any procedures

to respond to the loss of annunciators.  !

Plant procedures did not address the symptoms for a partial or tctal loss of '

the RK system (alars annunciators) or partial loss of the plant computer.

There was no abnormal procedure that provided appropriate actions to respond to this type of event. There was no guidance on the maintenance of a steady state reactor power level after a loss of annunciators.

Even though the plant computer was responding with numerous false data points, e.g. " Safety Injection pump A in lockout,' abnormal procedure. No.

OTO-RJ-00001, " Loss of Plant Computer," was not used. However, this procedure did not sensitize the operator for the need to take compensatory measures or identify the parameters, indications, or equipment for increased operator monitoring. Although the R0s lost confidence in the validity of the plant computer alarm indication, they were not aware of the extent of the loss of the annunciators and did not adequately use .the. alars response procedures effectively to verify the proper plant actions. There was no procedural guidance on the need to verify plant computer alares against annunciator window alarus during partial system failures.

Only a limited number of STAS were trained to use OTS-RJ-00001, " Restoration of Plant Computer Failures." This procedure refers to OTO-RJ-00001, " Loss of Plant Computer," for actions if plant computer failures occur during. -

restoration.

operability of the STAS indicated plant computer. that this could be used for the determination of e

11 o

O

' The Emergency Plan Implementing Procedure, EIP-ZZ-00101, ' Classification of Emergencies," does not include the loss of the plant computer as a specific criteria for the declaration of any emergency classification. Yet, the plant computer provides moro alars information than the annunciators do.

The operators questioned had many different views as to what a " plant transient" meant as used as a criterion in the " Indications

• section for a Site Area Emergency due to a loss of annunciators. For example, some thought a power level change was sufficient to increase the emergency classification while others did not.

The work request did not contain a systematic troubleshooting plan to determine the cause of annunciator system failure or a post maintenance testing method with acceptance criteria.

The operators did not have a list to determine which annunciator windows were inoperable on loss of specific power supplies.

The plant's technical specifications contained references to certain annunciator alarn . These included: T -T deviation (TS 4.1.1.4.b), rod position deviation monitor (TS 4.1.3.2f axiaT flux difference monitor (TS 4.2.1.1.b), and quadrant power tilt ratio (TS 4.2.4.1). The plant procedures consider these alarms inoperable when the plant computer or the appropriate HUX power supply is inoperable. The operators began tracking these technical specification related alarus at the 1:00 a.m. October 17 loss of annunciators and verified the operational data. t The procedural deficiencies were considered to be contributors to the root causes of the event. For example, had there been procedures, the lines of comunication would have been established, the questions to ask would have been provided, and the required knowledge of the annunciator system would have been provided or referenced.  !

3.3.4 Trainina Discussions with the licensee regarding classroom and simulator training i courses regarding a total or partial loss of plant annunciators or partial loss of the plant computer indicated the following:  ;

3.3.4.1 Classroom Trainino No specific training on a partial or total loss of annunciators.or partial  !

loss of the plant computer has been conducted. '

There was no training specifically addressing the operation of the annunciator system. Training on specific annunciation windows was included.with the training on the. system affected by the alara.

There was no specific operator, engineering, or management classroom training on the annunciator system. This resulted in the operators being unaware that  ;

open and closed logic or field contacts cause either illuminated or unilluminated failed annunciator windows.

12 e

6

~

Operators were not trained that individual annunciator panel lamp tests are sufficient only for testing the lamps and deternining the partial loss of a PK bus, but inadequate as an annunciator system functional test. The operators were not trained that individual alars logic or driver cards, or logic power supply failures would not manifest self-revealing symptoms.

The operators were not trained that a specific field or logic power supply energized annunciator windows in multiple alarm panels, despite several previous power supply failures. The operators had no training or technical information on which annunciator windows were inoperable on loss of a specific field or logic power supply.

The simulator was not modeled to simulate the partial or total loss of all main control board annunciators or the plant computer, or to enable effective training on the abnormal or diagnostic loss of plant computer procedures, OTO-RJ-00001 and 0T5-0-00001.

The 11C technicians had been trained on the operational characteristics of the annunciator system including the loss of the annunciator system upon loss of the power supplies. They had also been specifically instructed on the caution relative to declaring an alert if all four power supplies were lost. During the event, this knowledge was,not used.

While training was not specified as a root cause of the event, it was a contributor to the lack of knowledge of the annunciator system. l 3.3.4.2 Simulator Scenario One dynamic simulator scenario (DS-28) was developed in response to an event at Nine Mile Point, Unit 2, and administered to all operating crews during the 91-5 cycle of operator requalification. This was initiated by the partial loss of the PK01 bus which resulted in the loss of horn and window lights in four balance of plant annunciator panel sections. The simulator scenario subsequently cascaded through multiple primary and secondary system malfunctions compounding the loss of annunciators. -

There were no abnormal procedures that addressed the partial loss of the PK system. The annucciator response procedure, OTA-RLRK017, for window 178, "PK01/02/03/04 TROUBLE," immediate action did not direct a response to the loss of the annunciators affected. It only describes actions to be taken to determine which PK bus was lost. In the scenario, the operators have to deduce that the annunciators were lost based on observing changes in control board indications. While this challenged the operators' knowledge-based reasoning capability in a training scenario, the lack of guidance can delay operator response in an actual loss of annunciators caused by a PK bus failure.

The annunciator system failure response from this scenario was different from I i

that in the October 17, 1992 event. The simulator scenario training conducted on the crews was ineffective in properly responding to this event because of its lack of similarity. -

13

3.3.5 Human performance Investication The licensee conducted an investigation and questioned the operators, but did not have all the operators prepare individual statements after the event; therefore, it did not appear to constitute a rigorous human performance 4 investigation. l 3.3.6 Operatino Crew Stress tevel The event occurred at 1:00 a.m. during the second day of an OWL shift. The operators stated that they had no unusual fatigue at the time.

3.3.7 Man-Machine Interface l

The main control annunciator fieldboard has no annunciator or indication of a failed or logic {

had to deduce the failure. power supply to alert the operators. The operatsrs j The space available to the 11C technicians to perform the replacement of the field power supply was confined.

4.0 Ecuiement Failures  :

l Between 7:30 p.m. and 11:00 p.m. on October 16, I1C technicians made voltage measurements on the 14 logic and four field power supplies. As a result, one 0.5A fuse was replaced on a logic power supply and the voltage of one field power supply was determined to be unacceptable (low). l t

i Between 11:00 p.m. on October 16 and 7:00 a.m. on October 17, 1992, a replacement field power supply was obtained from the warehouse and bench tested. The WR. indicated that the power supply was to be replaced and to repair the fuses.

During the replacement, it was necessary to jumper both the input and output field supplies between two of the power supplies that were j physically between the one that was being replaced. While setting up to  ;

remove the jumpers, the I&C technicians reported that an electrical are was i

observed and additional annunciator windows illuminated. The four field power i supplies had blown input fuses. Tatse fuses are rated one ampere and are the slow blow type.

Additional fuses for the fleid power supplies were obtained from the 11C shop and the previously failed power supply. During replacement of the fuses, ,

t there was a problem with one of the fuses holders. This caused a delay in the fuse placement which caused the fuses to blow again. (It is necessary to replace them all in a short time so that the load is shared.) Additional fuses were obtained fr the warehous,e, inserted without a delay, and the ,

i system was thought to bi restored at 1:56 a.m..

A logic power supply was replaced between 1:30 p.m. and 2:00 p.m. and between 3:00 p.m. and 5:00 p.m. five Igic power supply . fuses were replaced. The  ;

voltage was measured on the 14 logic power supplies and the four field power  !

supplies. Subsequently, all measurements indicated normal voltages. Further '

functional testing indicated that the annunciator system was operable.

14 l

~

l i

4.1 Analysis of Root Cause Determination The AIT determined tne root cause of the equipment failures as follows:

l

a. The failure mode of the field power supply was low voltage. The l

' intermediate cause of the low voltage was failure of the primary field windings of the power transformer. The root cause of this I failure will be determined by the licensee and reported as part of e the CAL response.  :

b. The most probable intermediate cause of the blown fuses in the '

field power supply was grounding of a temporary jumper clip during .

replacement of the failed power supply.~The root cause appeared to be personnel error.  !'

i i

c. The most' probable root cause of the blown fuses in the logic power .

supplies was a current surge generated when the field power-  !

supplies were lost. i i

About 25 hours2.893519e-4 days <br />0.00694 hours <br />4.133598e-5 weeks <br />9.5125e-6 months <br /> passed from the initial identification of the event until the system was fully operable. The difficulty in determining the extent of the '

failure > and in restoring the system operability were as follows:

A field contact power supplies are wired in parallel 5

the instruction manual was inadequate the system drawings did not completely identify the inter-relation between the effect of field power supply failure and the logic power supplies the ground detector also ties the system together the !&C engineers who had the greatest knowledge of the system '

were not available when the event occurre '

.t 4.2 Corrective Actions -

+

The licensee tried to reproduce the electrical fault ti. ,t occurred during~the I field contact power supply failure but was unable to identify how it occurred.

Because of previous problems with the annunciator system, in 1991, a request  !

for resolution was proposed to modify the annunciator system to separate the .l system so that three groups of 400 alarus and one group of 200 alarus are i independent from each other. The modification would prevent system  !

interaction so that failures would be confined to one group, therefore they.  :

would be ensier to analyze and repair. Construction Modification package No. j 91-1037 has been issued to address the above problem and is pra, posed to be implemented after the 1993 refueling outage.  ;

15 e

~~

p y -

y m m

Implementation of this modification and the identification of which annunciators are on which power supply should prevent a similar event from happening in the future.

9 5.0 Event Classification and Reportina Emergency Implementing Procedure, EIP-22-00101, Attachment 1, addresses loss of annunciators as follows:

Initiating Condition Indication (s) Emergency Classification Most or All Annunciator Panels ALERT Alarms (Annunciators) RK014 through RK026 Lost are not operable i

The operators have been trained at Callaway to use only the criteria listed in the " Indications

• section as the trigger for the declaration of the emergency specified.

Additional consideration of the intent of the initiating condition listed is discouraged.  ;

In this case, while the " Initiating Condition' -

accurately reflects the guidance in NUREG 0654 that an alert should be declared if most annunciators are lost, the " Indications Section" lists loss of all annunciator panels on the sain control board as the only criteria.

This created a dichotomy and effectiv ;y narrowed the initiating condition to all annunciators lost since the indications column did not accurately reflect the intent of the initiating condition.

The operators interviewed had different opinions of the need for declaring an emergency classification and on the percentage of annunciators necessary to be iner.erable before the declrration sust be made. This ranged from 51% to 75%.

An opinion was also expressed that a smaller loss of the safety systec annunciators instead of a certain percentage of the total annunciators would have more ;afety significance. Several of those interviewed originally had .

the c? inion that an alert was not necessary, but after the fact concluded that tne extra expertise and manpower that the declaration of alert would have brought to bear on the probles would have been worthwhile.

At 6:40 p.u- 4 illuminated, o '-tober 16,1992,193 annunciators were lost (approximately 76 e<1t dark). The RO explained to his SS that he saw half of the annunciatorx riash then go dark ex:ept for the 76 that sta He had experienced previous annunciator power supply failures,yed recognized illuminated.

this as such, and expected that scoe of the unilluminated annunciatcrs had also failed. He noted that many of the plant -computer alarms were unreliable.

Although the R0 had little confidence in the annunciators or the plant computer systems, he did not enter the " Loss of Plant Computer" abnormal procedure. He ashed his SS why they should not be in an alert. The SS reportedly explanation. responded that they were not in an alert, but offered no

~

At 1:00 a.m. on October 17, 1992, all annunciators wers lost.. Another RO was concerned that an alart was not declared based on counting 360 of the 683 16 ll 6

annunciators being illuminated. This cperating crew was unaware that the unilluminated annunciator panels were also out-of-service. The plant computer again provided obviously unreliable alarms. Several discussions between the R0s and SS regarding the proper emergency declaration occurred, but an alert was not declared. At 1:L6 a.m. on October 17, 1992, all but 163 annunciators were returned to service, although the operators were aware of only five annunciator windows with problems at this time.

If an alert had been declared in recognition of the 1:00 a.m. loss of annunciators, it would have restined in effect until operation of theadequate post-maintenance testing proved the satisfactory annunciators.

was made. However, no emergency declaration of any kind As a result, the operating crew did not summon management or sufficiently knowledgeable technical experts to expeditiously resolve the loss of the annunciator problems. The operators did not adequately inform plant management of the extent of the loss of annunciators during the event. The licensee also did not take many of tne compensatory measures that may have been expected as a result of this type of event. The licensee did not sttff the Technical Support Center during the event. Additional licensed operators were not called upon to continuously monitor plant instrumentation. Equipment operators were parameters in thenotplant.

directed to increase monitoring status of equipment The number of annunciator windows illuminated was not and included in operator photographs taken. logs, despite the fact that window counts were made i plan and did not clearly indicate the power supplies that had blown An fuses.T unrelated turbine stop valve surveillance test was performed while 163 annunciators were unknowingly out-of-service. Repair of the annunciator i

system was halted prematurely for most of the morning of October 17, 1992, as '

a result of less than adequate post-maintenance testing.  :

The licensee faxed a statement to the NRC Operations Center at 12:47 a.m. CDT 1 and made p.m. EDT onan official Emergency Notiffcation System event notification at 2:14 October 19, 1992, which was taken as ENS 24453. The report was submitted pursuant to 10 CFR 50.72.b.1.v., which requires the reporting of a major loss of emergency assessment capability. 10 CFR 50.72.a specifies that the NRC Operations Center be informed immediately after the  ;

l notification of state and local governments of the declaration of an alert, and not later than one hour after the time the licensee declares an emergency.

10 CFR 50.72.b.1 specifies that the NRC Operations Center be informed as soon as practical and in all casec within an hour of the event. Thus, while the contents of this report were adequate, it was about 60 hours6.944444e-4 days <br />0.0167 hours <br />9.920635e-5 weeks <br />2.283e-5 months <br /> late. )

The Missouri State Emergency Management Agency Director was informed of the event at 3:30 p.m. ca October 19, 1992. The presiding Commissioners for Callaway, Osage, Gasconade, and Montgomery, Missourt counties were informed of the event between 3:35 p.m. and 4:10 p.m. on October 19, 1992. The Mayor of Fulton, Missouri was informed of the event at 10:00 a.m. on October 20, 1992.

Thus, the licensee's insediate and subsequent actions related to an emergency classification were less than adequate. ~

17 O

~

6.0 Safety Sionificance .

The AIT concluded that no operational parameters were approa - cd de exceeded and that there wert no radiological consequences to this event. However, the AIT had concerns with the following:

1. The delay in detecting and responding to this the event.

2. The potential for delay in detecting and responding to a plant transient or other events while annunciators are out-of-service.

3. The potential for equipment that is damaged or out-of-service not being readily identified.

4. Callaway's operators were trained to use annunciators during transients and did not thoroughly understand or pursue the significance of the event.

5. The proper declarations and reports had not been made as required.

7.0 Overall Conclusions 7.1 Cause of EouiDeent Failures The AIT determined that the causes of the equipment failures were as follows:

a. The intermediate cause of the power supply failurt appeared to be failure of its power supply transformer.

b.

The failure of the fuses in the field power supplies and the logic power supplies was due to personnel error.

7.2 Root Causes for the Event The AIT used an events and causal factors charting technique to develop the root causes of the event. The event was considered to be the initial loss of all annunciators, the partial loss of annunciators until Saturday at 7:37 p.m.,

loss.

and the failure to recognize the opers.fional effects of the annu; cistor The team identi.fied four root causes for the event. Absent any one of these causes, the event would have been significantly attigated. The root causes were as follows:

7.2.1 Poor Conwunications/ Teamwork.

This existed between the SS and the EDO, the SS and the plant manager, the SS and the IEC technicians, and the SS and the shift crew. It also existed in the control room logs. For example:

18

~

l

\

l

4 The I&C technicians, I&C engineer, and I&C planner did not inform I

the SS what fuses blew and what the effects were.

The OWL shift SS did not call the EDO or the MCP after the 1:00 a.m. event. This occurred despite the fact that both the EDO and MCP had asked to be called if there were any changes.

The AM shift SS did not frform the EDO of the I:00 a.a. event.

The RO turnover sheets (Fri PN to OWL) did not mention that dark annunciators could also passibly be inoperable.

The SRO above turnover sheet did not inform the OWL shift SS of the comment.

The SSs did not adequately brief the R0s or the e operators on IEC actions or changing conditions. quipment 7.2.2 Lack of a Duettionino Attitude /Comolacency.

There were numerous times during this event that a more questioning attitude could have prevented the event or its consequences. For example:

Less than adequate questioning by the SS/ Operating Supervisor to the IEC technicians in regard to which power supplies and fuses were inoperable.

Less than adequate questioning by the SS/ Operating Supervisor with '

regard to generic comments made by the IEC technician on what the extent of the problems was.

R0 logs did not address the significance or number of annunciators that were illuminated.

Several R0s questioned the decision that an alert was not required several times but allowed themselves to be convinced otherwise.

The operators, the I&C technicians, the engineer, and the work planner all thought that the annunciator system was still '

receiving power from someplace since some annunciators were illuminated; however, no effort was made to verify this assumption. '

7.2.3 Inadecuate Knowledee of Annunciator system.

There were numerous individuals involved with the event that lacked adequate knowledge of the annunciator system which hampered their ability to make appropriate decisions. For Example:

19  !

e

The operators, . engineers and management involved received less than adequate training on the annunciator system to permit an understanding of the symptoms of its failure. One RO who had experienced power supply failures previously learned through that experience, but did not pass on his knowledge during shift turnover. ,

IEC personnel had received training on the annunciator system, but did not use that knowledge to conclude that all annunciators had been lost.

There was a lack of available procedures on the partial or total loss of the annunciator system, or the partial loss of the plant cor.puter.

Neither the operators nor the IEC personnel had correlations between the annunciator windows and the power supplies to aid in the determinttion of which annunciator windows were inoperable.

7.2.4 Less than Adecuate Work Performance, There were numerous instances where work performance contributed to an inability to appropriately dia Further, since the WR was non gnose and respond to the annunciator problem.

safety related, it did not receive the same level of attention as a safety related WR would have received. For Example:

Loss of all four field power supplies was apparently a result of the jumpering operation during replacement of power supply No. 2.

There was a less than adequate pre-job briefing. Only the caution about the potential of an alert was noted to the 11C group by the SS.

There was inadequate post-maintenance testing after the four field power supply fuses were replaced, which resulted in the operators' belief thet only five anaunciator windows were still out-of-service.

The WR did not contain a systematic troubleshooting plan and it was completed without clearly identifying which fuses and power sup;.!!es were replaced or affected.

a The I&C technicians and the engineer did not read the WR which contained four power the caution supplies about were lost. the declaration of an alert if all i

l l

l l

20 i

?

o . l 8.0 Exit Interview

• t The team met with licensee repre.4entatives (denoted in enclosure 4) on October ;

24, 1992, and sumarized the purpsse, AIT charter items, and findings of the  !

inspectten. The team discussed the likely informational content of the  !

inspection report.with regard to documents or processes reviewed by the team l during the inspection. The licensee did not identify any'such documents or i processes as proprietary.

j l

l l

i a

r

(

\'

P r

f f

21 em '

ENCLOSUPI 2

, s.e asa w UovtTED STATES f $ NUCLEAR REGULATORY COMMisslON i t nEciose m

'- l tesnoossvcovmoao g cesse stova. sounois seier .

CONFIRMATORY ACTION LETTER OCT 2 01992 Union Electric Company ATTN: Mr. Donald F. Schnell CAL-RIII-92-012 Senior Vice President - Nuclear Post Office Box 149 - Mail Code 400 St. Louis, MO 63166

Dear Mr. Schnell:

This confirms the conversation tin October 19, 1992, between Messrs. William forney and Robert Greger of my staff and you and Mr. Gary Randolph of your staff 1992. related to the loss of annunciators which occurred on October 16 - 17, With respect to the Callaway plant satters discussed, we understand that you will perform the following actions:

1.

Conduct an investigation to detemine the causes of the annunciator failures and the failure of shift personnel to recognize the extent of these failures, and to evaluate the decision making and comunications associated with the event.

2.

Place the power supply which failed in quarantine until released by the NRC's Augmented Inspection Team (AIT).

3.

Maintain documentary evidence of your investigation effort and make this available to the AIT.

4 Evaluate these most recent equipment failures and staff actions in light of past equipment failures and staff performance to determine if additional actions are necessary.

5.

Provide within 30 days to NRC Region III a documented evaluation of the take. issues including corrective actions you have taken or plan to above None of the actions specified herein should be construed to take precedence over actions which you feel necessary to ensure plant and personnel safety.

If your understanding differs from that set forth above, please call me imediately. I Issuance of this Confimatory Action Letter does not preclude issuance of an Order formaliz'.ng the above commitinents or requiring other ,

actions on tne part of Union Electric Company. Nor does it preclude NRC .from

).gg~g f

• g V trY 1

k OCT 2 01992  :

Union Electric Company 2 '

taking enforcemer.t action for violations of NRC requirements that may have. '

prompted the issuance of this letter.

1 Sincerely,

• y-A. Bert Davis ,

Regional Administrator  ;

i Distribution cc:

G. L. Randolph, Vice President, Nuclear Operations

~

J. V. Laux, Manager Quality t Assurance i Tom P. Sharkey, Supervising '

Engineer, Site Licensing  !

~

DCD/DCB (RIDS)

OC/LFDCB ,

Resident Inspector, RIII  :

Region IV Resident Inspector, Wolf Creek l K. Drey l Chris R. Rogers, P.E. '

Utility Division, Missouri  !

Public Service Comission Gerald Charnoff Esq. ,

Thomas Baxter Esq. '

R. A. Kucera, Deputy Director,  !

Department of Natural Resources  ;

State Liaison Officer i J. M. Tay1or, EDO  !

J. H. Sniezek, DEDR H. L. Thompson, DEDS ,

T. E. Murley, NRR J. G. Partlow, NRR i

J. W. Roe, NRR ,

J. A. Zwolinski, NRR I E. L. Jordt.n, AEOD  !

J. Lieberman, OE i

, J. R. Goldber '

J. N. Hannon,g, OGC

.NRR R. J. Strasma, RIII L. R. Greger, RIII i R. A. Westberg, RIII  !

i I

,, . _ _ . _ _ _ _ _ _ _ _ _