ML20155F749

Independent Safety Evaluation Svcs Project Rept 3-96, Investigation of E Diesel Generator In-Operability Event

ML20155F749
Person / Time
Site:	Susquehanna
Issue date:	07/24/1996
From:	Miltenberger J PENNSYLVANIA POWER & LIGHT CO.
To:
Shared Package
ML20155F723	List: ML20155F720 ML20155F749 ML20155F766
References
FOIA-98-81 3-96, NUDOCS 9811060104
Download: ML20155F749 (26)

Text

_ ._ . . . - _ c.n -

, a~:

l .

INDEPENDENT SAFETY EVALUA TION SERVICES PROJECTREPORT3-96 1

1 Inves*tigation ofEDieselGeneratorIn-operabilityEvent -

o 4 9

I Report Date: 7/24/96 l /J.WMi%!j#

Managm-iouwpendent Safety Evaluation Services File A16-3 981106'0104 981103 EX BIi PDR FOIA SORENGEN90-81 PDR PAGE Od PAGE(S)

, Q[((067 0IOf V

- JLA.-3& 1996 14*47 US tf(C SUSQ RES OFFICE 717 542 4573 P.03 s'.. ..

1 l

TABLE OF CONTENTS 1.0

SUMMARY

....................................................................................................1 1.1 C o n c l u s i o n s .... ..... . . . . .. . .. .. . . .. .. ... . .. .. .. ... . .. .. . .. .. . . . ... . .. . .. . . .. . . ... .. ... ... . .. ..... ... . . . .. 1 1.2 Rec o nime n dati on s ......................... ....... . . ............ ... .. .. . ..........'.... ............. .. . 3 .

2.0 DESCRI PTIO N OF EVENT ............ .............. ... ........... . .. .......... ..'.... 5 -

3.0 DESCRIPTION

OF SYSTEM .. .. ... . .... ........ ..... . .... ..... ................... 8 4.0 HEALTH AND SAFET( OF THE PUB LIC............................ ................................. 9 5.0 MISALIGNMENT EVENT OF JUNE 1 4, 1 9 9 6 . .. .. . . .. .. . . . . ... .. ... . .. .. . . .. . .. . . .. .. ... .. ... . . . . 1 2 5.1 Nucle a r Plan t Operator........ ... .......... ...................... ................................. 12 6.2' PCO and Shift S u pe rvision ............................................ ................................ 14 8.0 O P E RATO R R O U N DS .. .. ..:... ... .... .... ....... .. .. .. ......... .... ............ ...... .. ........... .. ....... ... . 17 6.1 Misalign ment of E DG Not Deteeted........................ ...................................... 17 6.2 Operator Rounds Need More Management Attention ..... ........ ..... ...........18

7. 0 S U RVEl L LA N C E S . ' ..... .... .. .... . ........ .. .... ....... ............................... .. ......... .. ........ 21 8.0 HU M A N FACTO RS CO NSID ERATIONS............................................. .. .........a. 23 d

e EX IBIT EDG715.rpt PAGE OFMh PAGE(S)

%_ c _ ,

,, .,_m ,... , _ _ . _ _ ,

1.0

SUMMARY

During the period from June 14,1996, through July 4,1996, the E DG was technically inoperable because a circuit breaker had been improperly aligned. The Limiting Condition for Operation for operating with three diesel generators was exceeded and a Technical Specification Violation occurred. The incorrect circuit breaker line up was identified by an operator rnaking his scheduled rounds. During the twenty day interval over 80 operator rounds and three formal surveillances were conducted without detecting the problem.

+

The Independent Safety Evaluation Services (ISES) group was assigned to investigate the incident in parallel with an Event Review Team (ERT) composed of line managers.

ISES shared data with the ERT. However, ISES did an independent analysis and drew its own conclusions. The ISES examined the misalignment event, the conduct of operator rounds, and the cor) duct of surveillances. !SES also evaluated the impact of the event upon the health and safety of the public.

1.1 Conclusions ,

The principal conclusions were: '

1.1.1 The adverse impact of the event upon the health and safety of the public was slight.

e The ability to cope with a LOCA/ LOOP accident was degraded because operator action would have been required to maintain the E DG in servim

. Had a LOCA/ LOOP occurred, the E DG would have started and carried its loads for at least the ten minutes postuleted in the design bases for the ECCS systems to function with no operator action. Initial actions including core flooding would have occurred without degradation. At some indeterminate later time operator action *

' would have been required to keep the E DG in service.

• The probability of a LOCA/ LOOP event is very small.

1.1.2 The principal cause of the E DG misalignment event of June 14, was inadequate performance by the nuclear plant operator (NPO), who was performing the 4.16 KV breaker alignment in the E DG building.

• The operator confused the breaker in cubicle 6 on panel OA510 with breaker 0A510'06, which is located in cubicle 1.

• The procedure contains a caution note and the breakers are labeled. Had the operator read the note and examined the panel he would have found a breaker EDG715.rpt EX BIT 1

. PAGE Orch PAGE(S)

p, go as%%a,me

+.

es u snseen w a m g2 m mesu e. C ~ -

l . labeled "0A51006 FEEDER BREAKER T.O TEST FACILITY XFMR" in cubicle 1.

.Either, he did not do this or he did not understand what he saw. He also failed to

.' read and (or) understand the label on the breaker located in cubicle 6 which reads, i "0A51005 FEEDER BREAKER TO MCC 08565". For some reason his mindset was

! that he was to move the breaker located in cubicle 6.

1 <

j 1.1.3 Shift Supervision erred by not confirrning the breaker misalignment before j

e

. granting permission to the NPO to open the knife switch and move the breaker. j e A misalignment of a 4.16 KV ESS breaker had been reported. .

• Misalignments of ESS breakers are very rare.

. No one was sent to the scene to back up the NPO.

1.1.4 Shift Supervision also erred by not conducting an initial status control investigation the day the mispositioning event occurred.'

1.1.5. The event raises questions about the quality of operator rounds.

• Over 80 operator rounds were completed by 15 NPOs before the misalignment was detected. The operators were not specifically required to check the breaker

. alignment and no one did.

• Review of data from the security computer shows that some operators are spending very little time in the E DG building. Eleven rounds were completed in less than 3 minutes 30 seconds. Three minutes and thirty-four seconds were required for a team of ISES engineers to walk briskly through the building sighting each required item on the rounds' sheet but taking no readings and performing no actions.

• Comparison of data from round sheets and the plant process computer shows that in

, 19 cases operators had made a dieck mark on the round sheet indicating that they had tested the alarms dn panel OC577E. However, the panel trouble alarm was not recorded by the computer.

. Fourteen of the alarm recording discrepancies occurred on the rourids of two operators who showeid 0 hits for 6 and 8 rounds respectively. The s.ame two individuals made 9 of the 11 rounds which were completed in less than 3 minutes 30 ,

seconds. .

1.1.6 The fact that the misalignment was not detected by three formal surveillances is

, a serious concem.

. Three operators initialed surveillance sheets stating that breaker OA51005 was ,

racked in and open.

EDG715.rpt E IT 2 PAGE Oh PAGE(S)

, -e . . . . ~ , . ~ ..m l .. .

. There was no breaker in the cubicle in question.

Questions arise concerning the validity of the rest of the surveillances that were done by the persons in question. ,

1.1.7 The existence of surveillance records and operator testimony that do not agree with discovered breaker positions raises issues that are beyond the competence

. of ISES to investigate. The scene has shifted from technical and operational issues to questions of malfessence. .

i Three separate' operators initialed weekly surveillances dated June 20. June,27, and July 4,1996, confirming that breaker 0A510051 E Supply to 08565 was 'OPEN and RACKED IN".

. The slignment of panel OA510 on July 4, when the error was discovered, was exactly as it had be a left on June 14, when the error was made. Specifically, the transfer .

breaker was located in the OA51006 position and no breaker was installed in position l

. 0A51005. ,

• No evidence was found indicating that any breaker in panel OA510 had been moved in the interval between June 14, and July 4. . - -

. One NPO admitted that he had made an error and had misinterpreted the conditio.n of the breaker.

. The other two NPOs stated that they had seen the proper lights lighted on breaker 0A51005 and that they had physically checked that the breaker was properly racked f

v in.

The testimony of two NPOs does not agree with the preponderance of evidence which indicates that n'o breaker was installed in position OA51005 at the time that the two NPOs stated that they saw one there.

1.1.8 During the course of the investigation various human factors weaknesses were l' identified. A list is found in Section 8. .

L 1.2 Recommendations l

ISES recommends that:

l ,

1.2.1 The performance of the specific operators upon whose rounds the alarm discrepancies occurred be evaluated.

EDG715.rpt EX IBlT 3 PAGE OF[EIgllL PAGE(S)

es m '

uemu c m een usmaen as ww v.a.

1.2.2 The attention devoted by Management to the quality of operator rounds be .

increased.

1.2.3 An investigation of the surveillance issue be commissioned by a person (persons) skilled in malfeasance type work. (in progress.)

1.2.4 A detailed check be cond'ucted to ensure that the major cir'cuit breakers and other key components are in 'he t proper configuration.

1.2.5 The human factor wea'knesses identified in Section 8.0 be addressed. l

' ~

0 ,e e e 4

e ,

e 5 0 e

9

@ g e

a

• e 4

4

• a f

e

~

EDG715,rpt EX IT 4

!, PAGE OhPAGE(S)

g_ _ _ _ .. g g g ._ _ _

- . . ~. -

. j i' o

2.0 DESCRIPTION

OF EVENT I On July 4,1996, an operator making his rounds noticed that the breaker alignment for ,

the E Diesel Generator (DG) was not correct. The E DG was in service in place of the ,

D DG, which was being overhauled. l 1

Specifically, breaker 0A51005 on 4 KV Bus panel 0A510 in 'the E DG building was missing. This is the breaker'that supplies power to the E diesel auxiliaries if offsite power is lost.. A breaker was installed in the cubicle that provides power from the E DG ,

to th'e D' Engineered Safeguard System (ESS) bus. The breaker that supplies power to' the E DG test bus when the E DG is not aligned for service was installed in its normal p'sition o and was racked out.

The situation was that that the E DG would have operated normally in the event of an automatic start caused by loss of an ESS bu's or by a Loss of Coolant (LOCA) event.

l However, if a Loss of Offsite Power (LOOP) event had occurred no power would have been available to the E DG auxiliaries.

l The E DG was declared INOPERABLE and the applicable Limiting Condition of -

Operation (LCO) was entered. The breaker lineup was placed in the proper configuration, a surveillance test was completed and the E OG was returned to -

operable status.

1 The misalignment occurred on June 14,1996. The operator had placed the E DG l auxiliary supply breaker in the cubicle that supplies power to the D ESS buses. The correct procedure is to leave the auxiliary supply breaker in place and use the test bus breaker to supply power to the affected ESS bus. The result was that the station operated with three OPERABLE diesel generators for the 20 days between June 14, and July 4. The Technical Specification Limit of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in this configuration was exceeded.

Subsequent investigation showed that on two occasions during the 20 day period an l ,

additional diesel was taken out of service for briof periods - the longest was 2 minutes reducing the station to TWO operable diesel generators. No, LCOs were exceeded during these outages.

During the ensuing 20 days shiftly operator rounds of the E DG building were made and three formal surveillances of the diesel generator breaker alignment were completed.

The mis' alignment was detected on the 20th day during a normal operator round.

l EDG715.'rpt EXpBIT 5 PAGE. Fu0h PAGE(S)

4

3.0 DESCRIPTION

OF SYSTEM The following description will be useful in understanding the event.

The Susquehanna Station was built with four diesel genera' tors, wh'ch supply .

emergency power to the ESS buses of both units. In order to avoid unnecessary -

outages a fifth diesel was built. The installation was such that the fifth diesel could be substituted for any of the other four engines. The fact that the fifth engine was in service would be transparent to the operators in the control room. (i.e., if DG E were substituted for, say, DG D the controls and indications would be the same as though . ' -

DG D were in service.)

The basic schematic is shown in Figure 1.

. FIGURE 1 ,

sm, c

..... 7a h =

s e** n e s, *W n $ ,

gs, um u ,,..

,,,f ,

.. , o  : ...

' SA)eua se

,64410A gA261M

= 1A tSs a e

~^ '

i -. .. .

I: , uli. = "'"" wu. >S...

.u , . , .

,- i  ! ,,;;;  ; .

,,l,  : , , ,c ,, -

2

. .) i . ,,,,,

• • "58:!

s! . . ) .;Io

• • "7

• ,eaacuse.m.l  !  :

g, u . .. ...... . . . . . . ,

J m. i. -

I

.. >+ . . .

!l '

. x, u,

mum

- 9 ,se,,a, se--en ew i t

""*( w . .

.M."

e n- d 8. =

! esMS j . sA,04 1

- sal aus 28

':E* 'f M

, . 4, dE .hM o . . .

uniem osus JR- .= "'a ""^ "i

• v. w wa.

$N688 a

' *' .f,',,. ' * * * *'

. 4 OA 6100 a

,o a ao est aw !

@ = .. ,,, g -

axwo moy w onesovoon

l g,0 t F30U881%

D E

h EDG715.rpt EX BIT b 6 PAGE Oh PAGE(S)

_c _%-. _ . _. ...

Figure 1 shows the' breaker configuration used to connect the E DG to the va,rious buses.

l The design makes'it impossible to connect two diesel generators to.the same bus at the sam'e time. This is done by physically moving the transfer breakers of the affected diesel and of DG E. Consider Figure 1. In order to align the E DG to the A ESS buses, '

the transfer breaker of the A DG (OA510A01) must be moved from the output of the A DG to position 0510A02 and the transfer breaker of the E DG must be moved from the test position (OA510-06) to position OA51001. When the alignment is made the I above breakers are closed. They establish the current path'. Electricity can now flow ,

from the E DG to the 1 A and 2A ESS buses via OA510-01 and 0A510A02. The actual '

output' breakers, which close to connect the diesels to the ESS buses, are located on the ESS buses downstream of panels OA510 and 0A510A through D. The only time ,

that the transfer breaker on panel 0A510 serves as the E DG output breaker is when the E DG is being tested by supplying power to transformer OX207.

1 Notice that buses 08565 and 08566, the E DG auxiliary power buses are normally j supplied by offsite power from Transformer T-10 or T-20. N offsite power were lost, '

auxiliary power would come from the E DG via breaker 0A510-05, which would shut.

l Notice also that the transfer breaker of the E DG changes number from 05A10-06, to i 01,02,03, or 04 depending upon which bus is being supplied. In the normal l l

configuration where the E DG is not in service the E DG output breaker is in the test l position and is numbered 0A51006. . I 1

i l

l l

l l

EDG715.rpt EX IBIT 7 PAGE / Oh PAGE(S)

l

. j Figure 2 shows the physicallayout of 4KV bus pane 10A510. l FIGURE 2

. . 0A510 -

Cubicle 1 2 3 4 5 6 7 8

~

Breaker 06 01 03 07 ~02 05 04 OA510- -

Use -

Test E ESS ESS ESS AUX ESS -

D/G- A C B E DIG D i -

l l Breaker 0A510-07 in cubicle 4 is a spare. Cubicie 8 houses the, PT transformers for the 3

breakers. .'

It is important to know that, while 8 breaker cubicles are available, only T$O breakers are physically present, viz. the transfer breaker and the auxiliary power supply breaker.

The E DG Auxiliary Power Supply Breaker, OA510-05, is installed in cubicle 6. When -

4 the E DG is out of service, the transfer breaker. resides in cubicle 1, where it is named, .

OA510-06. However, when the E DG is in service the transfer breaker moves to cubicle

. 2,3, 5 or 7 as applicable where it becomes breaker 0A510-01,03, 02 or 04 respectively. - -

The misalignment of June 14, occurred when the operator confused cubicle #6 with breaker 0A510-06. He moved breaker 0A510 05 from cubicle #6 to cubicle #7. He did not touch breaker 0A510-06, which was housed in cubicle #1.

. Please note that the diagram shows breakers OA510-01 through 07 with a hyphen. The.

above description has been written to match the diagram. In the following text the

. hyphen will be eliminated unless a verbatim quote is being cited.

We will consider the impact upon the health and safety of the public and then a'ddress the specifics of the event. -

• I &

.I EDG715.rpt EXHIBIT h 8 PAGE N OFE PAGE(S)

. - -_- -. :- - . -- .-_- - - _ , . - . _ = - . . - . - - - . .

_m , , ,.g m - -.

l

• l

. 4.0 HEALTH AND SAFETY OF THE PUBLIC Section 3/4.8 ELECTRICAL POWER SYSTEMS of BASES section of the TECHNICAL-SPECIFICATIONS defines the purpose of the AC power sources as follows:

The OPERABILITY of'the AC and DC power sources and associated distribution I systems duririg operation ensures that sufficient power will be available to supply the safety related equipment required for (1) the safe shutdown of the facility and

-(2) the mitigation and control of accident conditions within the facility.......... .

+

The ACTION requirements specified for the levels of degradatiori of the power sources provide restrictions upoa continued facility operation commensurate with 1 the level of degradation..The OPERABILITY of the power sources are ' I consistent with the initial condition assumptions of the accident analyses and are based on maintaining at least three of the onsite AC and the corresponding DC l power sources OPERABLE during accident conditions coincident with an

! assumed loss of offsite power and single failure of one other onsite AC l source..:... - -

l

.1 The E DG was not OPERABLE for the period from June 14, through July 4, because power would not have been available to the E DG auxiliary equipments were offsite -

I power to have'been lost. The situation was that the postulated single failure had occurred and only three die'sel engines were operable. The station would have been able to cope with the design accident assuming no further degradation. Obviousy, the safe operating envelope was degraded because the station operated with only three operable diesel engines for longer than the allowable 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> 8. -

We will now examine the implications from the viewpoint of the health and safety of the public.

We will note in passing that as long as offsite power was not lost the engine would have functioned normally. That means that if either of the D ESS buses were to be lost for some reason the diesel generator would have picked'up the load and AC power would have been restored.

. However, the purpose of the diesel engines is to supply power in the event that offsite power fails. ISES has examined the E DG system and has concluded that the engine would have started and supplied power to the D ESS buses in the event of a Los's of

. offsite power (LOOP) event. The limiting auxiliary loads ar's the fuel oil trartsfer pump, which supplies makeup fuel to the day tank, ano the E DG building ventilation fans.

The day tank contains sufficient fuel to supply the engine for at least one hour. A low level alarm exists, which would alert the operators that action is required. The engine proper is cooled by emergency service water. It should run indefinitely without any building cooling fans. The purpose of the fans is to cool the switch gear and associated EDG715.rpt EXHIBIT s

, ., PAGE M OFh.PAGE(S)

control circuitry. The opinion of the DG system engineers is that the engine would not -

be limited by building temperature. (i.e., it would run out of fuel before temperature became a problem.) ISES was unable to find any calculation that defined the length of time that the machine could run without building cooling. In the absence of a -

calculation on the ventilating system capabilities, ISES assumed that the engine would start and carry its loads for 10 minutes,'which is the design criterion for the ECCS systems to operate with no operator action. ,

A low priority alarm in the control room would alert the operators that power had been lost to bus OA565 .Whether or not this alarm would be noticed and acted upon in the chaos surrounding a loss of offsite power event or, much worse, a loss of offsite power- .

combined with a loss of coolant accident is not known Assuming that the'alerm were seen .and acted upon, it would have been necessary to remove breaker DA51006 from cubicle 1 and insert it into cubicle 6 in order to restore auxiliary power. The breaker in cubicle 7 could r)ot be replaced in cubicle 6 without shutting down the E DG because it would be supplying power to the bus. The bottom line is that a great deal of operator action would have been required under conditions of stress to restore the E D.G auxiliary bus to service. Based upon the above ISES will analyze the safety implications based upon the assumption that the E DG will supply the D ESS buses for l ten minutes.

l .

Three scenarios will be considered: -

l .

1

• Loss of offsite power (LOOP) with no other complications, which is the ability to safely shutdown the facility.

• Loss of coolant accident (LOCA). and I

• Loss of coolant accident combined with loss of offsite power (LOCA/ LOOP).

^

l .

Loss of Offsite Power .

A LOOP event could easily be handled if the D ESS bus were de4nergized on both units. Water could be added using the HPCI or RCIC systems, steam could be sent to i the suppression pool via the SRVs and heat could be rejected to the spray pond using l

a RHR loop in suppression pool cooling. When temperatures were within limits, normal l- shutdown cooling could be placed in service. If HPCI and RCIC were not available, the reactor could be depressurized using the SRVs, makeup could be provided using core spray and heat removed by suppression pool cooling and then shutdown cooling.

l .

. The ability to safely shutdown the facility was never in jeopardy.

l l

I EDG715.rpt EXHBIT 10 PAGE/h 0FMPAGE(S)

asvnsa scsa -

us m. m e wr u ,u m m e.-

. " Loss of Coolant Accident .

A LOCA without loss of offsite power would not be affected by loss of E DG power to the E DG auxiliary loads. The auxiliary loads would remain supplied by offsite power and the E DG would function normally. .

Loss of Coolant Accident and Loss of Offsite Power .

The LOCNLOOP is the worst case design bases accident for the station. in this situation all four diesel engines supply power to the ESS buses and all ECCS pumps start. The design assumption is that three engines are available when the accident -

occurs.

N one assumes thst DG E would h4ve operated for ten minutes, all four engines would have t:een avai5bla :.t the start of the event and the initial actions including core .

flooding would have occurred without degradation. A later failure of the E DG would leave the station with three engines which would meet the design criteria. N one

' assumes that one of the other diesels fails, three buses would have been powered for the critical first few minutes of the event. The second engine would be lost after the initial cor0 flooding had occurred.

I After the initial flooding has occurred the decay heat can be removed with a much ~

smaller volume of water. It is quite likely that the pumping can be cut back to a level that can be sustained by two diesel generators.

There is no question of the fact that the ability of the station to cope with the LOCNLOOP accident was degraded. The mitigating circumstance is that in, the worst case the design number.of three diesel generators would have been available for the initial stages'of the accident. In view of the very low probability of a LOCA/ LOOP accident the actualimpact on the health and safety of the public was slight. -

EDG715.rpt EXH IT 11

, PAGE / OFh PAGE(S)

-- - .. .. .~ _ ~. a. . ... ...  :.;- -

5.0 MISAUGNMENT EVENT OF JUNE 14,1996 The , event involved shortcomings by the Nuclear Plant Operator (NPO), by the Plant Controls Operator (PCO) and by Shift Supervision.

5.1 Nuclear Plant Operator .

The governing procedure was OP-024-004 Revision 14, TRANSFER AND TEST MODE OPERATIONS OF DIESEL GENERATOR E. Part of the alignment is made in the E DG building and part is made in the D DG. building. The first action step reads,as follows: .

l l

NOTE: Breaker will not close !n following step.

3.1.5 At OA510, Diesel Generator E 4.16 KV Switchgear, PLACE Generator E supply to 08565 Breaker 0A51005 control switch to CLOSE position and RELEASE to reset breaker logic for automatic closure on undervoltage at 08565.

The operator then proceeds to the D DG building where he performs steps 3.1.6 through 3.1.25, which include shifting the D DG transfer breaker from position OA510D01 to p' osition OA510D02 (step 3.1.18). .

The procedure then states:

NOTE:, Following steps refer to one transfer breaker used in five different locations. Normally it is located at 0A510, Diesel Generator E 4.1.6 KV Switchgear, Test Facility Cubicle and numbered 0A51006.

3.1.26 REMOVE Transfer Breaker CA51006 and PLACE in DG E for DG A (B)

(C) (D) breaker cubicle. (Breaker now OA51001 (02) (03) (04).)

i

• 3.1.27 RACK IN Transfer Breaker 0A51001 (02) (30 (04).

J.

3.1.28 CLOSE Transfer Breaker 0A51001 (02) (03) (04) Control and Trip DC t

Power Knife Switch.

3.1.29 CLOSE Transfer Breaker 0A51001 (02) (03) (04).

On June 14, the operator successfully completed steps 3.1.5 through 3.1.25. He .

proceeded to the E DG building to complete the procedure. When he got to 4KV bus panel OA510 he reported to the control room that he'had found breaker 0A51006 racked in with the DC control power knife switch closed. Considerable discussion ensued involving the NPO, the PCO and Shift Supervision. The. conclusion was that

~

. the. alignment procedure had not noen properly completed the last time that the E DG EDG715.rpt - 0" 12 PAGE /d Oh PAGE(S)

,__ _m- _. , _ . , . ._ -.,

i was removed from service, viz. breaker 0A51006 had been left racked in with the knife switch closed. The NPO was grarited permission to open the knife switch and to reposition the OA51006 breaker. The NPO then completed the alignment p' rocedure and the diesel was successfully tested supplying the D ESS buses. The diesel ran properly because power to the auxiliary loads was being supplied from offsite power.

Condition Report 96 D707 was written documenting that a status control event had occurred. The Operability Determination for CR 96-0707 concluded that oper' ability of

- the diesel engines had not been affected and that no LCO was applicable.

In reality the operator was looking at CUBICLE 6, which houses breaker 0A51005. The knife switch was indeed closed becauseAhe breakerwas properly aligned to feed the E- -

DG auxiliary loads.

The error is puzzling for'two reasons. Film, the operator had completed Step 3.1.5, which requires him to cycle the OA51005 breaker control switch to reset the breaker logic. He reported no difficulty in performing this step. Since the other steps. of the 4

procedure were done properly and since he reported his problem with Step 3.1.26 to this control room, ISES has no reason to think that Step 3.1.5 was not done correctly.

In short, he had successfully identified the breaker in cubicle 6 as Breaker 0A51005 before proceeding to the D DG room.

Second,'the note before Step 3.1.26 stat'es that breaker OA5106 is normally located in-the test facility cubicle. The test facility cubicle is really cubicle 1. Had the operator

. read the note and. examined the panel he would have found a breaker labeled

'OA51006 FEEDER BREAKER TO TEST FACILITY XFMR' in cubicie 1. Either, he did not do this or he did not understand what he saw. He also failed to read and (or) understand the label on the breaker located in cubicle 6 which reede, '0A51005 FEEDER BREAKER TO MCC 09565". For some reason his mindset was that he was to move the breaker located in cubicle 6.

It is worth noting at this time that the labeling on the breakers in panel OA510 is not identical with the words in the procedure. Step 3.1.5 calls breake'r 0A51005 " Generator E supply to O'B565'. The breaker label reads, '0A51005 FEEDER BREAKER TO MCC 08565". Step 3.1.26 calls' breaker 0A51006, " Transfer Breaker 0A51006*. The panc!

label says, '0A51006 FEEDER BREAKER TO TEST FACILITY XFMR".

ISES did not interview the operator and will not speculate upon what was in his mind when he failed to correctly identify the breaker. (Extensive interviews were conducted by the Event Review Team.) ISES did confirm that the operator had been property ilualified for return to shift in the NPO position on March 12,1996, and that he had received training on E diesel breaker switching on November 22,1995. Clearly, some human engineering factors contributed to the event. The nomenclature and labeling are confusing. Still, the fact remains that the evolution has been successfully performed many times in the past. If one understands'the physical installation, he can follow the intent of the procedure. (The Manager-ISES had no trouble walking through EDG715.rpt EXHIBIT 13 PAGE h OdPAGE(S)

~

e- - - ... m ...

the procedure at Penel OA510. He noticed, however, that the labels on the breakers do

• i riot agree with the nomenclature in the procedure and he had to study the panels and

think about what was expected.) For some reason the operator in question i experienced a mental block when the time came to reposition breaker 0A51006. ,

i The conclusion is that a major caut;e of the event was inadequate performance by the

{ operator. The contributing causer were explored by the ERT during the hterview

process.

i .. .

5.2 PCO and Shift Supervialon , ,

The PCO was informed of the misalignment and he discussed the situation with the l Unit Supervisor. The Shift Supervisor participated in the discussion.

{

1 '

5.2.1 Misalignment Was Not Confrmed -

l l.

! The P'CO and Shift Supervision'were informed that an electrical system misalignment i

had been found and no one was sent to the scene to confirrh the condition. The I j , situation was that an NPO was aligning a system that is important to plant safety. He

. reported that he had found an ESS circuit breaker in the wrong configuration. The problem was discussed at .some length using the plant page system and the NPO was i' '

given permission to proceed. ,

i The item was documented in Condition Report 96-0707 which says:  !

j' Step 3.1.26 was about to be performed which has you remove the breaker from

.. cubicle OA510 06 (feed to 0X207 for running the D/G on the test bus) and place  !

it in the breaker cubicle for the diesel that is going to be substituted for. The -

! NPO found the breaker in OA510-06 to be racked in with the knife switch closed.

L .

i . The Operability Determination Section of CR 96-0707 states: (sic.).

Breaker 0A510 06 being racked in with its DC knife switch closed did not render any of the 'A' through 'D' D/Gs inoperable. The breaker is closed when running the 'E' D/G on the test bus and does not affect the other diesels.

CR 707 was' written by the STA using information provided by the Unit Supervisor.

Clearly, the understanding in the control room was that the NPO had found the test breaker racked in with its control circuits energized.

Finding a 4.16 KV ESS circuit breaker out of the proper alignment is a highly unusual occurrence. ISES checked the SOORs and CRs for the past five years and found no EDG715.rpt EXHIBIT 14 PAGE/7 0FhPAGE(S)

_._ _~ ~ _ - - _ _ . _ _ _ _ _ .. _ . _ ._ _ _ _ __. __ _ ._

f.. ,. -

, similar events. It should have prompted an inquiry. As a minimurp the misaligned condition should have been verified.

No verification was made. No one in the control room walked out to check the situation and no other operator was dispatched to the scene. Had the AUS or an experienced NPO been sent to the E DG building before the breaker alignment was changed, it is t

alrnost certain that the incident would have beer' avoided. The NPO was physically in the E DG byllding making the breaker alignment between 0432 and 0509. The AUS should have been available to assist the NPO. The AUS had been scheduled for random drug testing that shift. However, testing did not occur until after the,even[ He -

departed the south gate at 0541 end returned at 0559. If the AUS were tied bp - -

elsewhere during the event, other resources could have been made available.

l lSES concluded that Shift Supervision erred by not confirming the breaker misalignment before granting permission to open the knife switch and move the breaker. -

5.2.2 Status Control Event Not Promptly Investigated

Another point is that the incident was classifier., as a status control event by the Shift Supervisor. The status control issue was not pursued by the shift on watch at the time of the event. The shift completed the E DG line up and started the engine for the ,

required surveillance test. Th'e NPO was in the E DG building until 0643. The AUS exited the site for random drug testing at 0541 and returned at 0559. No depositions were taken and the AUS was not dispatched to the scene.to investigate before the shift went home. The status control investigation was passed on to the next shift.

. The oncoming Shift Supervisor initiated a status control investigation form. The first action item is to direct the AUS to " commence an immediate investigation and documentation of occurrence *. The AUS visited the E DG building to oversee the test

, run of the machine. He did not investigate the status control event. A review team was appointed and a meeting was held on the Monday, June 17, which was the first working ,

day following the occurrence. As of July 4, the status control investigation had not been

. completed.

Had the status controlinvestigation been pursued vigorously by either the offgoing or the oncoming shift, it is ipite likely that the misalignment would have been discovered the day of the event.

. 5.2.3. Summary -

ISES concluded that Shift Supervision had two opportunities to either prevent the event or to catch the error quickly, viz. :

EDG715.rpt E 15 PAGE/glBITOh 2 PAGE(S)

l ess-q~p gpi x es:vs gnrmarmgs ninch w su aa e.a l ' 1. Dispatch another operator to the scene to confirm the breaker status before l changing the alignment.

2. C6nduct an initial status control investigation on the day the event' occurred.

Neither was acted upon. -

E 9

. 4, e

. g B

1 l

l 9

8 6

S 0

e .

G 4

e G

l l

EDG715.rpt EX BIT 16

.. PAGE/ Odd _ PAGE(S)

.__ .. - ._ .._ . . = . , ... ._

6.0 . OPERATOR ROUNDS ,

. Shiftly operator rounds were completed every day that the E DG was inoperable. The problem was no.t detected Ontil July 4, when an operator on rounds noticed that the OA510 panel did not look right. By the time the misalignment was detected a total of 82 .

rounds (two per shift) had been made by 15 different operators.

8.1 Misalignment of E D'G Not Detected ,

ISES obtaineci the security computer records at d the completed round sheets. ISES confirmed that the person who had completed the round sheet had actually , l entered the building at a time corresponding to the time window on the' log I sheet. (Actual times are not recorded on the log. Rounds are taken each day betwe,en 1900 and 0100, 0100 and 0700,0700 and 1300 and 1300 and 1900.)' ISES also confirmed that the required readings had been recorded on the log sheets. Of particular interest,' readings were recorded for 125V DC battery charger (OD596) '

voltage and current on overy round sheet implying that each NPO had entered the lower level of the E DG building where the OA510 panel. is located. Thus, each NPO on  ;

rounds had an opportunity to detect the misalignment. 3 I

The specific round sheet covering the diesel generators is Ol-PL-0171, NPO PLANT

~

l LOG: DIESEL GENERATOR AND ESS XFMR'S. No specific instructions for verifying j electrical alignments are included in OI.PL 0171. However the generalinstructions j found on page i specify the following with regard to electrical panels: ,

. At load centers and motor control centers obsente:

a. Indicating lights are on; change bulbs if necessary. ,

b. Protective relaying status. ,

c. Temperature trends from one tour to next.

d. Protective covers (doors) installed (closed) with all fasteners secured.

The operator is responsible for making a general check of electrical pan,els. He is not j specifically instructed to confirm that a particular breaker alignment or confguration

, exists. No specific, heck of the 0A510 panel is included on the round sheet.

An operator can confirm the E DG switching alignment rather simply. If the E DG is i substituting for t'1e D DG, the indications will be: - .

• The t hite iight indicating that the E DG is available for emergency should be .

' illuminated on panel OC512E-D. The four OC512E-A (8, C, D) panels are directly i opposite panel OA510. -

i EDG715.rpt Eyl BIT 17 PAGECMh _OFd[gt PAGE(S)

v:rm m - ve.w 2, x e.a r. .

4 e Indicating lights will be lighted on cubicles 6 and 7 on panel OA510 indicating that control power is available and that breaker 0A51005 and breaker 0A51004 have control power and are racked in. An amber. and blue light should show on cubicle 6 for 0A51005 indicating that the auxiliary power supply breaker is racked in and open. A blue and red light should show for 0A51004 on cubicle 7 indicating that the power supply breaker to DG D has been racked in, has control power and is closed.

The misalignment was found on July 4, when the NPO realized that the lights on panel OA510 did not look right.

~

An operator on rounds could have told at a glance that two breakers were not lighted '

on panel 0A510 without entering the OA510 panel area. A bit more effort would have - ' {

been required to observe that the single lighted breaker was not in cubicle 6. . l Verification of the white E DG aligned on panel OC512E-D is not good enough. It merely indicates that the switching on panel DC512E-D is correct.

l The operators could not have failed to be aware of the fact that the E DG was i substituting for the D. They tour the D DG room on the same rounds that they tour the E DG building. The log sheet is set up so that the operator normally tours the diesels in the sequence of A through E. Entries on the D DG section of the rounds' sheet are marked o/s (out of service) for each of the shifts in question. The operators knew that ,

the E DG was substituting for the D. It appears that no one made a point of looking at '

the switchgear and confirming that the breaker alignment was correct. Such a check ,

was not specifically required and it was not done. - -

8.2 Operator Rounds Need More Management Attention

' 4 The fact that 15 operators over the cou'rse of 82 rounds failed to detect the E DG

, misalignment raises questions as to the quality of operator rounds. As noted above, ISES reviewed the time spent in the E DG building by the NPOs performing operator rounds. The data are plotted on Figure 3.

4 EDG715.rpt EMBT 1g PAGEN Oh PAGE(S)

[ ; .'

l

l. .

, a l- Figure 3.

Duration of NPQ Rounde in the E LWG effette to 7/4/90 ,

l assao num ,w.w e ma m.a. . ns me. ean.w

' o m eo com I ,, ,,

r . .

]. ,,

E- *

] omeo ,

I a,. .

g o ,2 Do om eo * * - ' -

,,,,, In I a L a h In n li I Ila I lilh

! I I I !! I I I ! l 1 ! LI I I I I I I I I I I I I Times ranged between 1 minute 49 seconds and 35 minutes 33 seconds. The ave, rage was 8 minutes 28 seconds. Eleven rounds were completed in less than 3. minutes and 30 seconds. .

9 ISES walked through the E DG round procedure. After locating each of the specific l items on the checkoff list and determining what was necessary to perform each, two j

• !SES engineers conducted a timed walk-through of the rounds. Three minutes and I thirty-four seconds were required to walk briskly along the most direct path through the l building physically sighting each required indication and component but taking no l readings and making no allowance for required actions such as alarm testing or filter i blowdown. .

l i

One is forced to question how a person could walk the same path, recording the l readings and performing the various tasks in less than three-and-one-half minutes.  !

Yet,13.4 % of the docurnented rounds were so donel - i ISES inspected the completed round sheets and noted that without exception all required readings had been recorded and all required non. reading items had been

, checked as completed. In each case the round sheet had been initialed by the NPO. l EDG715.rpt EXHIBIT 19 PAGEM Oh PAGE(S) i

f ,

It is possible to verify that one of the required actions was actually done. The rounds' sheet requires that the operator test the alarm at Panel 00577E, which is located in the lower level of the E DG building. The acceptance criteria is that'the alarm reflash in the Control Room at panel 0C653. This alarm is recorded on the plant process computer.'

l lSES obtained the' computer data for the times in question and found that the alarm had not been tested on 19 shifts. The data show that in 19 cases an operator checked a log sheet indicating that he had completed an action that had not been done.

Six operators were involved. Key data are in Table 1. ,

i TABLE 1 -

Operator Rounds Alarm Minimum Maximum

. Time Time -

Performed Not in Building in Building Recorded (Minisec.) (Min./sec.) -

1 6 1 5:25 9:37

'2 -

10 '

1 3:17 23:01 3 6 6 1:49 3:13 '

4 8 8 2:57 4:35 .

5:25 '

5 .

4 2' 3:08 6 4 1 7:51 30:24 ..

The persons with the shortest ranges of time in the building were Operators 3 and 4, neither of whom completed any of the required alarm tests. Knowing that neither-16dividual did the alarm test it is reasonable to inquire what other evolutions they may

. have omitted. The same question could be asked of Operator 5, who tested the alarms ,

on only two of the four shifts m.onitored and whose times in the tsuilding were short. All but one of the 11 rounds completed in less than 3 minutes 30 seconds were done by Operators 3,4 and 5.

No pattern is evident for the other operators. Operato'rs 1,2 and 6 were in the building 5 minutes 25 seconds,16 minutes 35 seconds and 7 minutes 51 seconds the times the alarms were missed. .

The above figures raise serious doubts about the quality of rounds being performed by

, some nuclear plant operators. ISES recommends that the sp'ecif' c performance of

. Operators 3,4 and 5 be evaluated. ISES also recommends that significantly more management attention be paid to the performance of operator rounds in general.

t EDG715.rpt EXyl IT

• l 20 PAGE AG OM PAGE.(S)

o-- e e- - ... ..,_ : r -

_m-

, . 1 j

l 7.0 SURVEILLANCES Three separate operators initialed weekly surveillances dated June 20, June 27, and

. July 4,1996, confirming that breaker 0A51005,1E Supply .to 08565, was 'OPEN and RACKED IN." It is almost certain that the breaker was not in the correct position. The alignment of panel OA510 on July 4, when the error was discovered, was exactly as it had been left on June 14, when the error was made. Specifically, the transfer breaker ,

was located in the OA51006 position and no breaker was jnstalled in position OA51005.

'This isses was pursued at length by the ERT team. The operators were interviewed - -

including walk-through at the scene. The following interview data were provided by the ERT team leader. .

The NPCs involved on June 20, and June 27, stated that they checked the breaker and found it open and racked in. Both stated that the proper lights were lighted on breaker 0A51005. We will call these persons NPO-1 and NPO-2 and will revisit them below.

The person involved in the July 4, surveillan'ce, NPO-3, stated that the lights were not lighted. NPO-3 saw the red semaphore on the switch and thought that the alignment was correct. He considered calling the control room and then decided that it was all right. He did not physically check that the breaker was in place and racked in. As a -

matter of interest, NPO-3 was in the building long enough to have checked thoroughly.

He spent 30 minutes and 17 minutes on his two tours through the E DG building the ,

night of July 3 - July 4. The bottom line is that NPO-3 did not do the surveillance correctly. He admitted his error freely to the ERT team. Details of the error can be .

obtained from the ERT. Unlike the NPO who made the misalignment on June 14, NPO-3 did not report any problem to the control room and no one had the opportunity to provide help.

All three NPOs stated that they had conducted the surveillance with the procedure in hand. The usual practice is for the PCO to provide copies of the applicable parts of s'urveillance procedures to the NPOs at the start of a shift. The NPCs verify the lineups

, and mark up the copies in the field as they do their rounds. When they are done they sign off the master procedure in the control room. That is what occurred during the surveillances under discussion.

NPO-1 stated that he had found both lights lighted at breaker 0A51005. He then stated that he had opened the breaker door and confirmed that the breaker was present.

ISES examined the security records and observed that NPO-1 had made two visits to the E DG building on the night shift of June 19 - June 20. The first was from 20:28:13 until 20:30:02, a total of 1 minute 49 seconds. The second was from 02:22:36 until 02:24:47, a. duration of 2 minutes 11 seconds. NPO-1 spent a total of 4 minutes inside the E DG building on the shift in question. During those 4 minutes he completed two sets of rounds. ISES verified that all required readings had been entered in the log and EDG715.rpt E HJI IT 21

. PAG 7 Oh PAGE(S) '

mma ww w tee ssa se m m m en c.a that the log was initialed. NPO-1 stated that he also removed and replaced the cover on breaker 0A51005. ISES does not believe that it is physically possible to properly

.' complete two sets of E DG rounds and open, reclose and properly secure a breaker

, door in four minutes.

, NPO-2 stated that he had observed the proper lights lighted on breaker 0A51005. He also stated that he had confirmed the breaker to be racked in by looking through the peep hole at the bottom of the cabinet door. NPO-2 had also completed two sets of rounds taking all required readings. NPO-2's times in the building were 5 minutes and 29 seconds and 5 minutes and 58 seconds for a total time of 11 minutes and 27 seconds. It takes only a few seconds to check a breaker racked in by looking through '

the peop. hole. ISES concluded that a competent NPO could have completed the tasks .

cited in 11 minutes.

The only possible way for the testimony of NPO-1 and NPO-2 to be correct is if a .

breaker had been installed in position OA51005 at the time that they made their surveillances. That means that someone would have had to install a breaker sometime subsequent to June 14, and then remove it again prior to July 4. Neither ISES nor the ,

ERT found any evidence that such's movement occurred. As noted above, the breaker configuration found on July 4, is identical to that left on June 14.

ISES finds it inconceivable that a person deliberately moved a heavy circuit breaker from position OA51006, installed it in position OA51004 arid then some time later .

returned it to position OA51006. There are far easier ways for a knowledgeable person to tamper with the power plant.

The section on Operator Rounds established the fact that the required alarm test on i

panel OC577E was not completed by several operators even though each had checked and initialed a log sheet indicating that he had properly completed the test. NPO-1 was the person who had failed to do the alarm check on 6 of 6 rounds in the sample (Operator 3 cited in Section 6.2). This fact severely.orodes the credibility of NPO-1.

. NPO-2 had no discrepancies in alarm panel testing.

ISES conclusion is that the testimony of NPO-1 and NPO-2 does not agree with the preponderance of evidence, which Indicates that no breaker was installed in position 0A51005 at the time tha,t NPO-1 and NPO-2 stated that they saw one there. This is not a technical or operational issue.

ISES recommends that a person skilled in malfeasance investigation be assigned to the case. '

e EDG715.rpt EXHlBIT 22

, PAGidd OM PAGE(S)

.-. __ . - -. . . - . . _ _ . - . - - . . - _ _ _ _ . - ~ _ _ . _ _ _ - _ _ . . . .. .. . . . . _

JLA 1996 15:00 uS NRC SUSO RES OFFICE 717 542 4573 P.26

• ? .* *'. ,

8.0 HUMAN FACTORS CONSIDERATIONS The following human factors items were noted during the course of the investigation.

• Confusion exists in the nomenclature between cubicles and breakers on the OA510 panel.

. 1

• The cubicles on panel 0A510 are not clearly labeled. It might be helpful to install signs at the top of each reading something like: E DG TEST on cubicle 1 A DG . .

SUPPLY on cubicle 2, E DG AUXILIARY POWER SUPPLY on cubicle 6, etc.

• Discrepancies exist between the text in Procedure OP424-004 and the breaker labels on Panel 0A510. .

• The portable radios do not work in the E DG building forcing the NPOs to communicate with the control room using the page system. .

e. The cord on the page telephone located by the OA510 panel is not long enough to

. allow an operator to talk to the control room while looking at the front of the panel.

It is recommended that the above items be reviewed for possible corrective action.

l I

l . '

i

. b EDG715.rpt EXHIBIT 23 PAGEh0h PAGE(S)