IR 05000282/1989032

Augmented Insp Team Insp Repts 50-282/89-32 & 50-306/89-32 on 891227-900109.No Violations or Deviations Noted.Major Areas Inspected:Sequence of Events,Root Causes of Equipment Failure,Operator Response & Licensee Event Followup

ML20006E983
Person / Time
Site:	Prairie Island
Issue date:	02/09/1990
From:	Miller H NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML20006E979	List: IR 05000282/1989032 ML20006E978 ML20006E984
References
50-282-89-32, 50-306-89-32, NUDOCS 9002270036
Download: ML20006E983 (24)

Text

$ ;.,47

~

j

,

.

'

-

.

-

Y

...U. S.-NUCLEAR REGULATORY COMMISSION-

.

.

,

'

>

..

REGION III'

>

.

>

Repo'rts Nos. 50-282/89032(DRS); 50-306/89032(DRS)

I

.

Docket Nos. 50-282; 50-306

' License No.'DPR-42; DPR-60

^!

,:

'

Licensee: Northern' States Power Company

'414 Nicollet Mall

'

Minneapolis, MN 55401 Facility lName:. Prairie Island Nuclear Generating Plant, Units 1 and 2 Inspection At:

Prairie Island Site, Red Wing, MN-Inspection Conducted: December. 27, 1989 through January 9, 1990

,

t AIT Members: R. A. Westberg, Team Leader

-D. S. Butler

't P..L. Hartmann i

D. C. Dilanni p Approved By:

H.-J. Miller, Director'

N Division of Reactor Safety Date Inspection Summary:

'

Inspection on' December 27, 1989 through January 9,-1990 (Report Nos.

.

J 50-282/89032(ORS); 50-306/89032(DRS))- Special Augmented Inspection Team (AIT)"

inspection to review the circumstances surrounding the Unit 2 reactor trips of

December;21 and 26', 1989, including:

sequence of events; root causes of equipment failure; operator-response; and licensee event follow-up and review'

- ( 93702).'

Results: No' violations or deviations.were identified in any of the areas inspected. No significant operational safety parameters were-approached or s

exceeded. Root causes of equipment failure were not the result of design

!

-deficiencies. The operators response to the events conformed to procedures or

[%

were conservative. The licensee's: event follow-up and review was methodical,

conservative and technicelly sound.

The-AIT'noted-only.one weakness during.the course of the inspection. This was-in the area of trending.

It appears to the AIT that there was sufficient evidence (cause unidentified MG. set output breaker tripping, see Section

,

!

5.c(4)) to indicate that a selective tripping problem was developing between

,

.the MG sets protective relaying.

9002270036 900212

'

'

PDR ADOCK 05000202 O

PNV lj

'

+

.

<jfs y

<

,4

,,

-

m

1. (%

.

m.

.. :Q " ',

y

-

M-

%,,y

<, -Q;C: e <,

' '

'

'

'

'

A,

,

-

>

o w

--.,

<

-. t

-

N Q,j.

?' ?

h_!

~ #

. l, "-,

l- ! '_

,

_

.fi

'

5_

,

- ",

-_h

'

xa

q m

n

.

.

,

Md4,5 /

?-

-

'

d

,

'

_..cw,,s m." 3

<

g r

y m:?,

- ^ ^

,

,

in w n.

c n

?

c n

.

gn7 g

_-y

-- -

-q--

+-

. !!.; 4...,'

ma,

'

E

< g

,..

._

,:

a,>

-

,

,

a...

. -,

,

a

>

-

u.

8:

s. 5. d.

,

'

r i

m '?

,

.

.

-.

,

L::

-.... N N

,

-

,

i n

n.

m;

-

,

• ~*

v

,

,

4.,

,

~,

-s

>

,

r t : -

,

,

_-(

-

t

_y I

h e.

h 4 r

,

a. ; j }[.h_

'l.' h.

,

.

_'z..

,

,

,

.

>>

'

% n1 -

O

~

,

-

T w,

'

-

,

, e

,

~

.f

'

' 'f mAUGMENTED INSPECTIONLTEAM REPORT:

~ l

'

1. -

'

,

.

m

.

-

'

s

,,

,

5

,

'

_~

g a

17:

.

f

.us p..:

@

t y,

,;

a

,.

~

a

_

,

_

_,.

,

v:-

- m a

., c, at a

.

e

.

,

>

.

,'

GO:

i p

s

.

u

,

.

- -

a

'

'

..i

,.

.

'r ft

-[

3

,

,

,

d[ $[

r-ein

.-.

,.

.

.-

.

.

I

..

'

4.,7 '

- ; w.

.'U, St NUCLEAR REGULATORY COMMISSION-M

.

<

s u a ::

,#

' q

[h)

_'

,..v, i

a-

-

.

p

-1 i

.

$

',

w; E'

.=

q'

t I

-

, ;) i -_'3 l s

-

~

f

b-

'

4 i

5{.{1;

,

q

'

s

_

. - - -

,r,.

,

s Y

I,"

-

,

)

x',

.

,n

,

- _

.,.

.

.

. -

.

.

..

' 1,

. w r_

b J '.

_

>

- PRAIRIE ISLAND: UNITJ2: REACTOR TRIP'WITH PARTIAL LOSS 7

'

es

' m. +, z

'

a

+

y

.,,

.

tl.i i spm..I e/'W~

0F 0FFSITE. POWER 4"

'

+

m-

,,-

m,

.~

..

-

,

1 JANUARY-26,(1990

• >

<4

.,

a

'7/

q,,

,7,Hl'i)?,,

g$

.,

T

..

INSPECTION REPORTS NO:

,

\\ ;" >;

gV R c50-282/89032(DRS); 50-306/89032(DRS)~.

\\\\

.;.

,

.

.

.

'

.

Qy

.7 g

g..

a t

Q&

_

bc

., ?

'

,

I

.Jr.

?

-

'

w"',

,r-[

'

+

,..t

,

g l y 7. '

y <

f

' '.,, '

v

.f

'

s'

,

.e I F a (

,&.;

x e-

..

,

y

{ -

., '

-

s

')

j

,4, ppw&,*:R x q

. 1, A -<

1. ,jV~

i, ' 'j ',J,i;

~- m

'

i

'

..

=

)

,

.,;

-v v y

--,-

-se-c

- - - - '- '

' ' -

+

jf;,..Li.

'

,

,

,

,

'

,

. 4-TABLE OF CONTENTS

-

-

.

. 1 '.-.

Event-Summary.................................................

1-

.

2h Persons Contacted..............................................

1-

'

,

3..

S e q u e n c e o f Ev e n t s............................................ -. 2,

,

.

14 '..

Op e ra t o r R e s p o n s e............................. -................. 4 5.'.

Equipment Failures - Causes and Consequences................... 5 6.

4160 Volt Non-Safeguards Bus Power Supplies.............>..... 17-

'

7.

Licensee Event Followup'and Review............................

>

8. --

Potenti al Tamperi ng Revi ew.................................... 20 9.

E x i t M e e t i n g................................................... 21

.

,

)

' Attachments:-

- 1 AIT. Charter

. 2.

-Licensee / Confirmatory Action Letter i

3.

Event 1SequenceLTime Line

>

4.

Rod Control - System

- 5.

MG SetiTesting Summary

-

- 6.

One Line-Electrical Diagram 7.

Outdoor Air-Blast Circuit Breaker-8.

Simplified' Breaker-Cross-Sectional View o

r

' l

,

s-(

c h

]( -.4,

.

.

f

.

-1.

Event Summary

'At 12:32 p.m. (CST), December _ 26, 1989, while operating at 100 percent power the Unit 2 reactor scrammed as a result of a negative flux _ rate.

The apparent cause of the negative flux rate was the dropping of a control rod or rods.

In additioni both of the Rod Drive Motor Generator output breakers were found open. One motor generator output breaker also opened under similar conditions during an event which took place on December _21, u

1989.,

During the event, both reactor coolant pumps and the main condenser-circulation pumps lost power because their alternate power supply was not available.

This was caused by an electrical isolation of. Bus No. 1 in the i

substation due to the slow opening of the C phase Unit 2 Turbine Generator i

output breaker which resulted in a partial loss of offsite power to Unit 2.

-

The secondary systems operated as designed with the transfer of decay heat l

from the steam generators via the atmospheric steam dump valves and the

,

auxiliary feedwater system.

Both emergency diesels started but did not

load since power was not lost to safeguard Bus Nos. 25 and 26.

l

Power was restored to the reactor coolant pumps by 2:25 p.m. which restored

'

forced circulation and ended the event.

The Senior Resident Inspector (SRI) informed Region III of the trip and communication was established with the plant via the NRC Operations Center to monitor plant conditions.- Region III, the SRI, and the Office of Nuclear Reactor Regulation (NRR) met telephonically and an NRC Augmented

,

Inspection Team (AIT) was deemed appropriate to review the event. The AIT,

!

headed by a senior regional electrical specialist, utilized the SRI already f

onsite'and additional personnel from Region III and NRR. This' report

' describes the AIT activities,and findings. The Team Charter is included as

Attachment No. 1.

R On' December 27, 1989, Region III issued a Confirmatory Action Letter (CAL)

'l to Northern States Power Company covering the actions' planned regarding this event.

The CAL is included as Attachment No. 2.

The AIT independently-verified licensee performance of these actions. Based on the licensee's investigation into the root cause of the events and completion of the AIT charter, the CAL was retracted on January 11, 1990.

J 2.

Personnel Contacted

-

l

Northern States Power Company f

C. Larson, Vice President, Nuclear Generation R. Jenson, Senior Vice President, Power Supply t

>

L. Taylor, Vice President, Transmission and Inter-Utility L. Eliason, General Manager, Nuclear Plants

'

• E. Watzl, Plant Manager

• S. Kollmann, Superintendent, Plant Maintenance

,

J. Leveille, Senior Nuclear Engineer G. Eckholt, Nuclear Support Services

• R. Lindsey, Assistant to the Plant Manager

,

. - - -.

..

. - -

i

in is... U,

=

,

p

'

.

y

• D. Mendele, Gineral Superintendent, Engineering and Radiation Protection-R. Sloss, Technical Engineer

R. Conklin, Supervisor, Security and Services

M. Wadley, Shift Manager

. M. Klee, Superintendent, Quality Engineering

• K. Beadel1, Superintendent, Technical Engineering P. Hellen~, Technical Engineer R. Gould, Supervisor, Relay Maintenance M.' Hammer, System Relay Specialist (Monticello)

J. Curtis, Electrical Maintenance Supervisor G. Aandahl, Principal Production Engineer D. Woytcke, Supervisor, Relay Maintenance K. Lennon, Electrical Maintenance K. Mustaphi, Electric Protection Engineering il I

United States Nuclear Regulatory Commission

• R. W. Cooper, Chief, Engineering Branch R. N. Gardner, Chief, Plant Systems Section T. O'Connor, Resident Inspector, Prairie Island W..Axelson, Chief, Division of Reactor Projects, Branch 2,

.

-[

• B. Burgess, Chief, Projects Section 2A j

• Indicates..those personnel attending the January 9,1990 telephone exit.

.3.

Sequence of-Events j

Background L

The sequence of events for the December 21 and 26 events are nearly identical.

j Both events resulted from a reactor. trip, Bus 1 lockout and loss of onsite i

nonessential electrical Bus Nos. 21, 22, 23 and 24. -A detailed-discussion

T is provided below. Discussion of rod control, motor-generators and the

}j root cause of the unit trips are discussed in'Section 5.

-a.

December 21, 1989 Event-

!

!

The event began about 2:22 a.m. (CST) on December 21, 1989 with a'

!

I reactor trip and resultant turbine trip approximately 30 seconds

!

later. Operators noted the negative flux rate "first out" annunciator

~j had locked in which indicated it was the first reactor trip signal

!

. presented to the reactor protection logic.

About 10 cycles (167 msec) following the Unit 2 main generator output

._

breakers opening, a Bus 1 (345kV substation) lockout. occurred.

This completely removed power to Bus Nos. 21, 22, 23 and 24 (see Attachment

,

6). The unit tr.ip removed the 2M (auxiliary) transformer normal'

'

,

power supply to these buses, and the Bus 1 lockout removed the alternate power supply which would have normally re-energized Bus

,

'

,

Nos. 21-24 automatically by breaker operation.

Bus Nos. 21 and 22

)

(4kV) supply the Reactor Coolant Pumps (RCP) and the Main Feed Pumps (MFP).

Bus.Nos. 23 and 24 supply various 4kV and 480V loads in the

!

!

I

._

b

p.-

g-3 g g-p

...

h

'

lant, such as rod control, condensate and condenser circulation p

pumps. As per Emergency Operating-Procedures, the plant was verified in natural circulation to: remove decay heat via addition of auxiliary

feedwater to the steam generators and the use of the atmospheric steam dumps.

The Essential Safeguards Bus Nos. 25 and 26 lost their normal power source from the 2RS (reserve) transformer when the Bus I lockout occurred. As designed, the automatic transfer logic selected and closed in the first alternate power supply for each bus; the. Bus Nos.

'16-25 bus. tie breaker (16-8) for Bus No. 25, and the cooling tower-supply breaker (26-1) for Bus No. 26.

The Emergency Diesel Generators-(EDGs) both started due to the momentary undervoltage signal on Bus Nos. 25 and 26, but did not load onto either bus because the buses were re-energized promptly from the alternate supply. Therefore, no essential electrical loads were affected throughout the event..The-

. EDGs ran for approximately 38 minutes and were stopped once it was determined that the electrical supplies to Bus Nos. 25 and 26 were a

reliable.

Initially the operttors did not know the cause of the trip, beyond the

!

"first out" indication, because the control room has no substation

breaker indications other than the generator output breaker control

.<wi tche s. Once they realized.a Bus No I lockout had-occurred, the.

operators believed the cause of the reactor trip was from a main generator breaker opening, resulting from an electrical fault in the

.-,

substation. To ascertain the status of substation breakers, the

'

control room either asks the load dispatcher or sends an operator to the substation relay house. With this scenario, a "first out" indica-tion would not be a negative flux rate trip. Most likely, the turbine

,

trip-reactor trip or steam generator low level reactor trip (due to t-loss-of feedwater pumps) would be the suspected "first out" annunciator

.

.from events precipitated f. rom a loss of the turbine generator.

'

After one hour into the event, since initial actions-had been taken i

and the plant was stabilized in' natural circulation cooling, the

'

'

sequence of events printout was reviewed. -Additional personnel had begun to arrive which supplemented this review. These personnel realized that the source of the reactor trip could not have been from i

the main generator output breaker opening since the sequence of events printout showed the negative flux rate trip occurred prior to the main generator output breakers opening. The root cause of.the reactor trips will be discussed in Section 5.

In addition to identifying,the cause of the trip, the operations crew y

attempted to restore power to nonessential Bus Nos. 21, 22, 23, and f

24. This was done by utilizing Unit 1 power sources and is discussed m

in detail in Section 6.

During the December 21 event, the re-ener-gizing of these buses proceeded cautiously, since the cause of the Bus No. I lockout was unknown and a concern existed that an electrical fault could be reintroduced during the energization of these buses, if proper care was not utilized.

At 5:31 a.m., the No. 23 and No. 24 buses were energized from Unit 1 sources.

The No. 21 condensate pump was started at 5:33 a.m. and the

..

.

.

m.

.

.

s 4 y.1,

..

!

l c

b No. 21 circulation pump was started at 5:54 a.m.-

The No. 21 and No, i

>

22 buses were energized at 5:44 a.m., with the A RCP started at 5:52'

a.m., and the B RCP started at 6:21 a.m.

This resulted in. forced circulation with the condenser removing decay heat.

b.

December 26,1989 Trip The sequence of' events for the December 26 trip was nearly identical

'

to the December 21 trip, except for the timing between each step. This

-

was due to the experience' gained by the plant during the December 21 trip. Most notable was the immediate conclusion that the cause of the trip was a negative rate flux trip from a rod protection system anomaly and~not induced from a main generator output breaker opening.

Also,

.;

the restoration of power during this event occurred more promptly, given the understanding of the previous event. 0ne problem that occurred while restoring power to Bus Nos. 23 and 24 is discussed in Paragraph 5 of this report.

A' minor anomoly was encountered in this restoration relative to overspeed of the No. 22 Turbine-Driven Auxiliary Feed Pump

-

during its removal. from service. LThis was not an operability problem and was discussed in' detail within Inspection Report 282/89031;

'

306/89031(DRP).

4.

Operator Response The AIT directly observed the operations crew response to the December 26, 1989, reactor trip, within minutes after the event initiated.- The AIT also discussed each event in detail with operations, technical engineering,

' electrical, maintenance, and managerial personnel. The AIT concluded that the appropriate-operator response in each event was to' verify that the reactor tripped and to implement Emergency Operating Procedure -(E0P) "1E-0

,

'

Reactor Trip" and,i at step 4, since a safety injection was not required, enter (EOP) IES-0.1 " Reactor Trip Recovery." Procedure IES-0.1 verifies

'or initiates corrective action to obtain stable core cooling by ensuring

'

adequate auxiliary feedwater flow, stable primary pressure and steam generator (heat sink) availability. At step 10 with'no RCPs running, the-operators verified natural circulation by Attachment A to the procedure.

• This was done promptly in-both cases.

Step 11 checks for source' range-indication. During the December 21 event both channels of source range

. indication were erratic; therefore the operators declared both source range detectors inoperable and the detectors were replaced. The balance of plant recovery was-to perform administrative notifications, shutdown secondary-systems, restore the 345kV breaker lineup and consider the need to perform a cooldown.

'

Du_ ring both events, these actions were promptly initiated and implemented

'

even though the December 21 event occurred during the backshift when staffing was at a minimum. With the cause of the reactor trip initially being unknown, the threat of distraction to troubleshoot the cause or to focus'on restoring nonessential power was great. However, this was not evident in the response of operations personnel to the event. They focused appropriately on verification of the reactor's shutdown status and providing l

adequate natural circulation for core cooling. Troubleshooting was appro-priately left to auxiliary personnel who supplemented the operations crew.

>

,

n y,4,.. +

,

c

-

.,

Following interviews with involved personnel and review of th'e required E0Ps, the AIT concluded that Operations response was appropriate.

In' addition, the E0Ps utilized by the licensee permitted prompt assessment and recovery from.the reactor trip and allowed the licensed operators to devote their attention to reactor safety.

5.

-Equipment Failures

'

a.

345 KV Distribution Circuit Breaker No. 8H13 (1) Background The 345kV distribution at Prairie Island consists of 345kV transformer lines, air circuit breakers-( ACBs), motor-operated disconnects, and metering and control devices (see Attachment 6).

There are six 345kV lines coming into the substation.

One line-comes from each of the two main transformer, one line comes from

the Blue Lake Generating Stction, one line comes from the Byron-I substation, and two lines come from the Red Rock substation.

Each

!

of the incoming lines is-separated from the buses and other lines o

by two ACBs.

There are two incoming itnes and three breakers which

~

i comprises a " breaker-and-one-half" isolation scheme. Two motor-

'

operated disconnects are provided for. each breaker for isolation during maintenance.

The ACBs are General Electric Type ATB-362-7 Outdoor Air-Blast circuit breakers'.

These breakers have the capability of inter '

rupting at their maximum current rating within two cycles (33 msec)

i after receiving a trip signal.

The breaker consists of three

'

interrupters per phase containing two sets of contacts each of which will provide six breaks.in series (see Attachment 7).

Three

>

lower control valves are operated electrically and transmit the

!

open or close signal to the individual interrupters through a

-)

pneumatic-mechanical linkage.

,

(2) ACB Operation When power is applied to the-trip coil (see-Attachment-8),.the-magnetic holding force in the control solenoid drops and spring f

pressure drives the armature away from the pole piece and against y

the pilot piston causing it to move to its extreme end of travel.

g This closes the vent under the vertical operating rod piston and

-

permits supply air-to pressurize this volume.

This causes an i

equal pressure on both sides of the vertical operating piston and since the lower side of the piston is slightly smaller than the (

-

upper side, causes a low tension force in the vertical operating i

rod. This lower tension is not sufficient to hold the control

-

valve piston in each interrupter against interrupter pressure and the valves open allowing interrupter air to pressurize the area under the blast valve piston. As the blast valve piston starts moving, it pushes the main driving piston toward the open position and starts opening the blast valve to extinguish the arc expected during the opening operation.

As the blast valve piston continu9s l

pushing the contact driving piston toward the open position, a

)

m

'

i 4_

.

h

.

.

..

l port in the' cylinder wall is. exposed from which air flows between lthe two pistons.. The blast valve piston changes directions and drives the blast valve toward'the closed position while the main driving piston continues in the same direction pulling the linkageL

,<

r and rotating the moving contacts to the fully opened position, b

The resistor switch contacts are held in the closed position by

'

a latch on the standard switch and by a flat on the operating cam in the preinserted switch until the main-contacts are almost fully open. When the resistor switch _is released it is spring driven to

,

the full open position. The blast valve piston drives the blast

"

valve closed just after the resistor switch is opened, completing the opening operation.

(3) Breaker Failure Relay Scheme eI The breaker failure relay scheme used at Prairie Island consists

'

F of an overcurrent fault detector, an input circuitry-initiate logic, a timer circuit, and an output circuit which energizes:a lockout relay. -The purpose of the overcurrent fault detector is to identify that the breaker's main contact is in the closed position during a fault. Operation of the.overcurrent unit and-the initiating circuits energizes the timer circuit.

If the breaker fails to open within 10 cycles (167 msec),-the output circuit is

,

energized and sends a signal to the lockout relay. The lockout relay contacts operate all the breakers which are electrically adjacent to the failed breaker.

-

-(4), Equipment Operation During December 21 Event

.

'

When Unit 2 scrammed on December 21, a trip signal was sent to the substation to open 345 kV Breaker Nos. 8H13 and 8H14.

Breaker No. 8H14 tripped, but C phase on Breaker No. 8H13-failed to open within two cycles (33 msec)~which caused its breaker failure lockout relay (86BF/8H13) to operate after ten cycles (167 msec).

It sent trip signals 'via Relay No. 86B1, to-open Breaker?Nos. 8H7, 8H16, 2RSY and 2RSX.

Breaker Nc,. 8H7 failed.

to open and its breaker failure lockout relay (86BF/8H7) operated.

,

This caused Breaker No. 8H8 to open and subsequently Breaker Nos.

8H27 and 8H28 to open at the Red Rock substation.

(5) Equipment Operation During December 26 Event The December 26 event was similar to the December 21 event except Breaker No. 8H7 operated properly and Relay No. 86BF/8H7 did not operate or-cause Breaker No. 8H8 to open.

(

(6) Licensee Actions Subsequent to the December 21 trip, the 345 kV breakers in the substation were inspected and tested.

Breaker Nos. 8H13, 8H14, and 8H7 passed the tests (open within 33 msec after receiving a trip signal) and were put back in service.

Since the weather (-22 degrees F.) was thought to be a factor, heaters were installed or, the controls. The heaters were under administrative control and D

'

,

i-.

.

.

.

t ;p (

,

,

i:

!

I were to be in operation when the temperature-went below 0 degrees s.-t F.

Unit 2 was put back on the grid on' December 23 and operated

.

.

until the December 26, 1989 trip.

-.'

,

p On December 27, 1989, a meeting was held to establish a-task

'

force to resolve the concerns generated by the second failure of breakers in the 345 kV substation. The following task force

>

objectives were established:

l m

(a) Establish justification for returning the unit to service

using 8H14 breaker.

(b) Replace main generator breakers (parallel plan).

(c)

Identify root cause for breaker failure and repair.

Prove through testing that 8H14 is acceptable.

(d) Prove and document that relay scheme in substation is designed correctly and operated correctly.

(e) Obtain samples of grease used in breakers to ascertain if

'

slow breaker operation may have been caused by the grease.

In addition, the task force also assigned action items, such as:

contacting General Electric for assistance; contacting other e

utilities, EEI, and INP0 for information; arranging for the-services of an ex-GE field engineer with experience in GE Air Blast breakers; and collating all maintenance / testing records for Breaker Nos. 8H13 and 8H7.

On December 29, 1989, the AIT witnessed the test of Breaker No.

.

8H13 in accordance with a.special.-procedure, No. 8H13.SPC, in

.;

an effort to repeat the previous failures..This procedure

required the breaker to be instrumented to monitor.various:

parameters of the breaker while the breaker was loaded with a current value simulating the load on'the breaker at the time of e

failure.

It would be tripped by the lockout relay.

,

However, prior to the completion of the test, the test was modified. This was because anomalies were discovered when the lower pilot valve control assembly covers were removed.

It was observed that the clearance between the pilot' valve and the trip r

armature on C phase was inadequate.

The conclusion was that this condition, in conjunction with high armature push-off. force, could explain the slow operation of C phase during the previous failures.

Procedure No. 8H13.SPC was modified to test C phase only and when it was tripped, C phase breaker failure occurred.

The recording oscillograph indicated C phase opened slowly and that all relaying l

operated properly.

(Substation Bus No. 1 Lockout.)

On December 30, 1989, the AIT witnessed the disassembly of the lower pilot valve and armature assembly.

The pilot valve and l

,

.

(QW--

_.

,,a, p

j

>

t

. cylinder were found undamaged and appeared normal. However,

!

inspection disclosed that the armature had unusual wear patterns

!

in the paint which indicated that the armature had been binding.

'

>

e

!

i

To further verify that the armature was binding, the pilot

<

control valve and armature assembly were reinstalled using the

e original shims.

Push-off force measurements were taken and

'

compared with the as-left measureroent of 32 pounds from the last

major maintenance activity. The new measurements varied from 40

to 56 pounds.

The breaker was also exercised several times and t

the pilot valve to armature clearance was measured. While.035

to.040 inches is specified, the new measurements varied from i

L

.005 to.015 inches.

o l

Based on this evidence, the licensee concluded that binding of the

,

j armature in the trip coil assembly was the immediate cause of the slow operation of C phase.

l The trip coil assembly will be inspected and analyzed for the root r

cause of failure.

The licensee committed to report the results

,

of this failure analysis in the 30 day report required by the CAL,

!

(7) Review of Breaker Procurement

-

The circuit breakers at the Prairie Island Unit No. 2 substation are of the air blast type, manufactured by General Electric which

'

are designed to operate outdoors.

The breakers were ordered in 1969 for delivery in 1971. The circuit breakers in-_the substation

s

-

were procured under a typical specification for commercial grade i

equipment in which the manufactured product was to conform to and be tested in accordance with the latest applicable ASA, IEEE, and (

NEMA star.dards. The design parameters consisted of the following:

(a) voltage rating 345 kv,

'

(b) current rating 2000 amps at 65cc, (c) interrupting capacity 25,000 MVA, I

(d)

interrupting time 2 cycles (33 msec),

'

(e) the-circuit breaker to be installed in a 60 cycle 345,000 volt circuitry in a WYE connection solidly grounded, and

!

(f) ambient temperature range, minimum minus 30*C to plus 40*C.

The circuit breaker manufacturer also furnished detailed drawings, service manuals and recommended spare parts list.

The AIT, after reviewing the procurement specification and the vendor quntation, concluded that the purchase order specification and the procurement process was typical for procurement of circuit breakers for non-safety grade application.

The application of

.

the ASA, IEEE, or NEMA standards for pre-installation testing was

!

u

. - - -

.

- -

.

. - _ _ _ _ _ _.

yf 4 -

e.

,

i-

'l tp p;

consistent for the demonstration of the operable qualities of these

components.. In addition, from the discussion with the licensee and review of breaker maintenance ptocedures and records, the nine

,

,

I.

circuit breakers in the substation have been adequately serviced

.

g'

and maintained throughout their service life.

On this basis, the

'

partial loss of off-site power during the reactor ~ transients of December 21, and 26, 1989, did not appear to be due to any inade-i quacies in procurement or pre-installation testing of the suspect

circuit breakers, p

(8) Breaker History L

[

The task force compiled the maintenance, testing, and failure

>

history cf Breaker Nos. 8H13 and 8H7 as one of the action items

!

for their investigation.

Review of the data produced by this.

L item indicated that there were no previous maintenance or testing

,

L anomalies.

Further, review of past cold weather trips that coincided with substation electrical problems only produced one

'

example of a trip when the temperature was below zero (October

1978).

Based on this information, the licensee concluded that i

the temperature did not have an effect on the breakers during the December 21 and 26 events, b.

Control Rod Drive System

>

(1) Rackground

-

The rod control system (see Attachment 4) is a solid-state I

electrical control system that holds and moves the shutdown rods

and control rods.

Rod movement is initiated by the control rod t

drive mechanism (CRDM). A CRDM is an electromagnetic stepping device which sequentially energizes and deenergizes three coils I

to move a control rod up or down one step.

The operating coil

!

stack consists of three independent coils which include a

!

stationary gripper coil, a moveable gripper coil and a lift coil.

During steady state plant operation, the CRDMs hold.the

,

control rods fully withdrawn from the core.

In this holding

,

mode, only the stationary gripper coil is energized on each

-

mechanism. If power to the stationary gripper coil is removed.

.

as during a reactor trip, the control rods will fall into the t

core.

In addition, if the stationary gripper current approaches zero for any pulled rod, that rod will likewise fall into the core.

!

Electrical power for the CRDMs is provided by two a-c motor-

'

generator (MG) sets through the reactor trip breakers to the

'

three solid state power cabinets.

Each power cabinet receives signals from the logic cabinet.

The power cabinet controls the

current to the independent coil groups by five silicon controlled

,

rectifier (SCR) bridges.. Each SCR bridge has a control circuit that maintains the desired current through the coils. A feedback signal is received from a sampling resistor which is in series

!

with each CRDM coil. The actual current is compared with the demand s1 nal received from the logic cabinet for either high

!

9 i

-

.

.

-

_

.

i

,

e.,

v r e !.g. a

,

i current,. low current, or zero current.

Thel power cabinet contains circuits for failure detection. A

" Power Cabinet Urgent Failure" light and/or annunciator will occur for any number of the following conditions:

(a) A regulator failure - when actual coil current does not match demand current.

(b) A phase failure - when a SCR bridge is malfunctioning or failed.

-

l4 (c) A logic failure - when zero current is ordered to both the.

stationary and moveable gripper coils simultaneously.

-

-(d) A multiplexing error - when more than one group of rods attempts to move.

L (e) A loose or missing circuit card.

!

An urgent failure locks all rods in that power cabinet by

energizing both stationary and moveable gripper coils to prevent

'

the dropping of that power cabinet control rods.

Individual banks of rods may be moved, provided their groups are not associated with the failed power cabinet.

(2) Rod Control System Failure Analysis The' licensee formed a rod control system task force following the dropping of shutdown control rods E3 and Ill during MG set

'

troubleshooting activities on December 30, 1989.

These rods had previously dropped into the core during MG set troubleshooting

.

activities performed on December 22, 1989, following the

'

December 21, 1989 unit trip.

Review and observation of the licensee's troubleshooting and CRDM testing indicates that the two reactor trips were caused by a dropped rod. Analysis of the scrams indicated that Shutdown Bank A, Group 1, Rod E3 was falling into the core slightly ahead of the other rods..During testing activities, the licensee pulled L

all eight shutdown rods to > 30 steps (to clear rod bottom lights).

On several occasions, rods 13 and 111 (same shutdown bank and group as E3)-dropped into the core.

The CRDMs for E3 and Ill were tested and no anomalies were identified.- The licensee also performed additional checks of the rod control phase sequence, x

coil inductance and coil resistance.

In each check no problems h,

were identified n ace E3 and Ill had dropped on several occasions,

"

the licensee expanded the testing to include checks of all the control rods and no anomalies were identified. A chart recorder

,

'

was then c.onnected to monitor the sampling resistor current of E3, Ill, B8 and D10.

Review of the chart recordings after dropping E3 and 111 indicated that the stationary gripper current had momentarily decreased to zero and no power cabinet urgent failure alarm was present. The licensee continued troubleshooting and

w

_,

g.

,

( '. - * t.

{

.

e

,

determined that there was a 0.4 ohm resistance path between the i

F neutral and ground. The rod control power cabinets were then

isolated from the MG sets and each other. Troubleshooting deter-i

..

mined the current sampling resistor.for rod C9 was providing the

'

ground path.- The licensee replaced the resistors for C9, K5 i

!

.(located on the same sampling board as C9), E3 and Ill. The

neutral to ground resistance returned to normal. The Unit 1

neutral to ground potentials were 100 mVdc and 2.5 Vac.

Following

<

L the resistor replacement, the Unit 2 neutral to ground potentials

E returned to approximately the same values as Unit 1.

.

The control rod timing traces that were performed on April 27, g

1989 showed that for rods 09 and K5 the stationary gripper current

was exhibiting some electronic noise. Traces made prior to

!

(s replacing the C9 sampling resistor on January 1,1990 indicated i

p the same noise.

Replacement of the sampling resistor cleared up

,

the. stationary current noise for C9 aM K5 (both power cabinet

.

E 2AC). The traces for E3, Ill (both Dam.c cabinet IAC) and 13

!

[

(power cabinet 180) did not show any signs of noise similar to i

C9 and K5. Thus, a defective sampling resistor in one power

!

cabinet did not affect the other two power cabinets.

,

The rod control system was reinstrumented to monitor the following i

points in Cabinet IAC:

'

(a) Stationary gripper currents j

(b) Group C SCR bridge output (V coil)

l

'

i (c) Reference voltage (V ref)

,

(d) Neutral to ground potential

The rods were again pulled above 30 steps on January 7, 1990.

The following occurred during this holding period:

(a) Rods: E3 and Ill fell into the core

!

(b) No power cabinet urgent alarm was received (c) V ref went low, as did V coil.

l The licensee determined the urgent alarm card had failed; this i

would prevent an alarm condition from energizing the stationary-

~

and moveable coils to prevent the rods from dropping.

The licensee replaced the urgent alarm card, regulator card, phase y

control card, firing card and I/O card in the V ref signal path

+

f associated with rods E3 and Ill.

The urgent alarm cards in

'

power cabinets 2AC and IBD were tested and determined to be satisfactory.

Satisfactory CRDM testing of shutdown rods E3

and Ill was completed. The licensee will continue to monitor the rod. control cabinets and will inform the NRC when they plan

-

to remove the instrumentation. The licensee has also planned

,

further failure analysis on the cards that were replaced.

,

- -.. _

-

O n

C-e

!

'

s

Based on the above, the AIT concurs with the licensee's conclusion

[

that the two reactor scrams were the result of a malfunctioning urgent

'

alarm card in Rod Control Power Cabinet IAC.

I c.

Rod Control Motor-Generator (MG) Sets

!

(1) Background The power'to the rod control system is supplied by two MG sets.

-;

Each MG set consists of a 150hp motor driving a 260 Vac, 3 phase, l

synchronous generator. A flywheel is used to maintain the speed during momentary power interruptions.

The two MG sets are operated in parallel (for reliability purposes) and either machine can supply the necessary power requirements of the rod control system. The MG output breakers may.be tripped manually or.

_

automatically by overexcitation, reverse overcurrent, breaker

!

overcurrent, or a trip of an MG's respective supply breaker.

Opening of an MG's supply breaker and the above protective v

relay trips are annunciated in the Control Room as "1/2 Motor-Generator Sets Tripped."

-

r (2) MG Set Failure Analysis The testing of the MG sets was designed to preserve the as-found'

,

condition of the machines and to determine the plant and/or equip-

ment conditions that caused the MG output breakers to open. A

'

trip of the MG output breaker with no targets indicates that the

'

breaker most likely tripped on field overexcitation (over voltage). A trip with a target indicates the breaker tripped on

'

instantaneous or thermal (12-T) reverse overcurrent..The MG sets

-

are over-designed and could provide a maximum output current of

>900A. There was no. indication of excess current. The Unit 2 MG set output current when operating the machines in. parallel was l

approximately 25A per machine at 255 Vac.

Unit 1 machines were

outputting approximately 20 A per machine at 265 Vac. The MG set

'

input breakers were always found closed, thus they did not initiate

,

,

the tripping of the output breakers.

>

The following is the as-found condition of the Unit 2 MG set output t

breakers for the following dates:

l Date MG Output Breaker Status

December 21,1989 (2:22am)

1. 21

1. 22 6 Reactor Scram Wen 5tTosed

,

• Negative flux rate trip

'No relay

.

' Dropped rod (s)-

targets

.

December 21, 1989 (10:00 pm)

.

!

'Teactor Shutdown

.

'

' Initiated troubleshooting

' Checked all protective relays except reverse overcurrent

"

'MG regulator waveforms normal for both machines

' Pulled shutdown rods

,

!

'

-

'

p.. o e, Deember 22,1989 (1:00 am)

1. 21

1. 22

• Reactor Shutdown 5Bpen 35en

' Shutdown rod (s) dropped

'No relay

'"A" phase

'

' Initiated troubleshooting targets instantaneous

' Replace #21 regulator reverse current

'MG regulator waveforms target normal for both machines

• Pulled shutdown rods

' Soaked MG sets for approximately L

i.

five hours prior to startup, no. problems L

December 26,1989(12:32pm)

1. 21

1. 22

!

Ukeactor Scram F0 pen 5~ pen

' Negative flux rate trip

'No relay

• No relay

,

[

' Dropped rod (s)

targets targets

.

'

The licensee developed a comprehensive testing schedule to determine the problems with the MG sets. A Metha-Tech Automatic

,'

l Oscillograph was connected to both MG sets and was set up to automatically monitor 16 channels of information.

If an event occurred, the data was captured and-transferred to a hard disc.

c For continuous monitoring, a BMI voltage analyzer was connected p

to preserve any system perturbations that did not trigger the event recorder. See Attachment 5 for a summary of the licensee's

test results.

The installed regulator waveforms were obtained and compared to the vendor manual.

The waveforms appeared to be similar.

The, AIT noted that the 21 MG regulator waveforms were slightly faster than the 22 MG regulator waveforms.

The exciter field pulses from 21 MG were typically running 0.30 msee faster than the 22 MG exciter pulses. This. implies that the 21 MG was the controlling p

MG and that the generator time constant may be faster (underdamped j

when compared to 22 MG time constant) than the 22 MG time constant.

(

This supports why the 21 MG set responded faster to output changes and caused the 21 MG output breaker to open on overexcitation for1 tests (g).. (h), and (1).

Tests (a), (b), (c), (d), (e), and (f)

were tests that simulated a complete loss of a function.

Subse-

,

!.

quent tests of current,-potential, feedback and power transformers L

(megger and resistance checks) did not identify any defective components.

The MG set vendor manual discussed the importance of setting up the' protective relaying to provide adequate selective tripping

,

when operating the MG sets in parallel. Selective tripping is

\\

the means by which an abnormality in one MG set operating in

/

parallel will cause itself to trip out and not trip out the functional MG set.

The vendor r.;anual described two abnormalities.

First is the failure of one MG set causing a voltage decay. This is easily detected by the reverse current (IRV) relay in the abnormal MG set. The functional MG set would see the failed parallel MG set as an additional load causing current to flow

.

!

!

L,

l

y*

...

,

e into the generator through the failed MG set IRV relay. The second failure mechanism causes a rise in MG output voltage and is much harder to provide selective tripping protection.

The increase in voltage causes a reverse current to flow in the functional MG set and trips out the wrong MG set on reverse overcurrent and eventually trips the malfunctioning MG set output breaker on field overexcitation (overvoltage). The vendor manual described in detail how to set the overexcitation and reverse overcurrent trips to prevent the two abnormalities from occurring.

The protective relaying as-found setpoints were within their calibratio't tolerance.

However, the setpoints were not being reviewed to ensure they continued to match the dynamic characteristics of the two MG sets.

(3) Event Review The December 21, 1989 event is best described by the loss of prime mover tests. Both MG sets would have lost power when the Switchyard Bus 1 lockout occurred. Test (h) best reproduces this event where 21 MG output breaker tripped open on field overexcitation and 22 MG output breaker remained closed.

The December 22, 1989 dropping of the shutdown rod (s) and opening of both MG output breakers is best described by the loss of 21 MG feedbacktests(c)and(e).

The only difference was that the December 22 opening of the 22 MG output breaker was initiated by the "A" phase instantaneous reverse current trip in lieu of the

"A" and "C" phase time overcurrent trips in tests (c) and (e).

The December 26, 1989 event is best described by the loss of 21 MG feedback tests (c) and (e).

The only difference was that the opening of the 22 MG output breaker was probably the result of overexcitation (no targets) in lieu of the "A" and "C" phase time overcurrent trips in tests (c) and (e).

The December 22 and 26 openings of both output breakers appears to be the result of inadequate protective relaying selectivity and voltage regulator time constant adjustments. These adjustments were probably acceptable over the first eight p)

years of operation. Review of the performance history records indicate the IRV relays had their setpoint changed in 1982. The performance history records (see Subsection (4)) also indicate that 22 MG has had a history of tripping beginning in 1986.

(4) MG Set Performance History The following history records were reviewed:

(a) WR F0769-FL (2/82) - changed rod drive MG IRV setpoints

,

(both units).

(b) WR K0903-FL (1/86) - 22 MG tripped IRV "A" phase l

l

,

,

._

-.

_ _ _ _ _.

_ _ _ _ _ _

- *..

y,;

-

(s,

• '

+

,

instantaneous current, could not locate a problem, i

q (c) WR K4364-FL (7/86) - 22 MG tripped on IRV current, could

.,

not' locate a problem.

L (d) WR K8033-FL (12/86) - 21 MG not maintaining voltage, replaced two regulator transformers.

!

(e) WR L9538-FL (12/87) - 22 MG tripped on instantaneous IRV current, p

[

21MG IRV 22MG IRV V

i[

As-found 0.105 0.185 As-left 0.106 0.106

(f) WR'M1572-FL (2/88) - 22 MG tripped on IRV "A" ahd "C" phase instantaneous current, 21MG IRV 22MG IRV As-found-0.100 0.088 As-left 0.092 0.094

!

(g) WR M1714-FL (3/88) - 22 MG tripped on IRV overcurrent, installed new regulator unit, replaced transistors and

,

'1 some capacitors on old board and returned old regulator

_

p to stores.

NOTE: The work request did not identify.in the work completed area whether the' regulator waveforms were checked and if the time constant adjustments were made, c

(h) WR M3463-FL (6/88) - 22 MG tripped on IRV overcurrent, 21MG IRV 2?MG IRV As-found 0.59(A) 0.090(C)

0.178(A) 0.094(C)

As-left 0.108(A) 0.090(C)

0.109(A) 0.094(C)

!

'(1) WR N2731-FL (4/89) - perform maintenance' on MG-relays per maintenance procedure PI-2MG, " Routine EPS Maintenance of Unit-2 MG 21 and 22."

NOTE: This procedure is a relatively new procedure that

'was developed by the MG set system engineer and protective f.

relaying personnel.

Starting in the year 1986 there is a trend that the 22 MG was-J tripping consistently on reverse current.

It appears from the performance history records that the root cause of the 22 MG tripping events was not clearly identified.

It also appears i

that during the last four (4) years there was an indication that the protective relaying and voltage regulator time constant

'

-15 i

C

h l,a Ec

I

.

adjustments were not matchics the MG set dynamic characteristics.

}

E and ccmplicated the dropped rod events.

,

l

-

.

The licensee acknowledged that their trending program was in (h the developmental stages and indicated they were working at

!

improving this program.

'

s

[

(5) MG Design s

' '

The AIT reviewed the MG design to ensure that no other symptoms i

-

and/or components could have induced the MG tripping patterns.

!

It is the opinion of the AIT that this design was acceptable

'

,

for this application.

[

(6) Corrective Action

$

'

L

.

The licensee installed a new voltage regulator in each machine.

p The protective relaying and generator time constant was determined

',

for each machine and adjusted to prevent a malfunctioning machine

from tripping the functional machine..In addition, permanent

,

,

monitoring equipment has been installed to monitor both MGs until

!

t the next refueling outage.

'

f (7) Conclusions Based on the above, the AIT has concluded that the MG sets did

>

-

not initiate the reactor scrams on December 21 and 26, 1989, t

It appears that the problems that developed in the protective relaying selectivity and voltage regulator adjustments were

,'

initiating MG set tripping on their own' accord. As a result, j

these components no longer matched the dynamic characteristics

'

of the two machines.

-,

p (8) 10 CFR 50.72 (b)(2)(ii) Reportability

,

During a review of the Unit 2 reactor operations log, the AIT

'

became cognizant of the December 22 dropping of all shutdown bank rods during troubleshooting activities (reactor was

>

t L

shutdown).

Following a discussion of 10 CFR 50.72(b)(2)(ii)

reporting requirements with the licensee, the AIT was informed

'

that the licensee did not consider the event as reportable since-

there was no manual or automatic actuation of the RPS. Thus, if

-

a reactor scram occurred from a loss of MG set power or rod o

control' system failure and the reactor was not in the power range of nuclear instrumentation,- the reactor scram would not be reportable by their interpretation of the reporting requirements.

[

This issue also applies to the 10 CFR 50.73(a)(2)(iv) reporting l

[

requirements.

'

d.

Reactor Trip Breaker During troubleshooting activities, the "A" reactor trip breaker (Westinghouse DB-50) failed to close. The cause was the undervoltage coil's misalignment which ultimately would not allow the trip bar to

-

L

__

l

..A.

,

!-

i

?

reset, and allow for closure of the breaker.

This failure did not

.

involve the safety function of opening when called upon to do so.

Both i

'

reactor trip breakers were replaced with DB 50 breakers refurbished by Westinghouse. The AIT witnessed time response testing of the breakers, including separately tripping the breaker via the under voltage coil and the shunt trip, and concluded that the replacements had been adequately time response tested.

The misalignment appears to have resulted from mechanical agitation

$

during the repeated cycling of reactor trip breakers during trouble-

,

'

shooting activities.

The licensee sent the breaker to Westinghouse facilities for root cause evaluatior and rebuilding. The AIT requested

,

'

the results of this evaluation to be included in the 30 day report required by the CAL.

6.

4160 Volt Non-Safeouards Bus Power Supplies a.

Background During both the December 21 and 26 events, normal power was lost to

.

Bus Nos. 21, 22, 23 and 24 (see Attachment 6).

Bus Nos. 21 and 22

'

power the Reactor Coolant Pumps (RCP), and the Main Feed Pumps (MFP).

Bus Nos. 23 and 24 power various non-safeguards 4.16 kV loads, and supply various 480V loads through a 480V bus distribution to individual Motor Control Centers.

Bus Nos. 23 and 24 carry non-safeguards equip-ment, such as the Rod Control System, Condensate and Condenser circulation pumps.

For Bus Nos. 21 and 22 alternate power is normally supplied automatically by the 2RS transformer via the 2RX (Bus 21) and 2RY

,

(Bus 22) legs. Auto transfer occurs when the normal' supply breakers

'

open on undervoltage (Breakers 21-1 and 22-1) and the 2RS supply breakers close (Breakers 21-4 and 22-4).

The RCPs would remain de-energized until automatically restarted. A reactor trip signal is generated from either Bus 21 or 22 sensing an undervoltage.

.

During both events Dus Nos. 21 and 22 were immediately de-energized.

This resulted from the Bus 1 (345kV)-lockout in the substation, which tripped open 2RSY and 2RSX and de-energized the 2R supply legs (2RY and 2RX). ihroughout both events these buses functioned appropriately, and no anomalies were encountered during the energization of Bus Nos.

21 and 22. During the December 21 event, these buses were energized approximately three hours following the reactor trip. During the December 26 trip, Bus Nos. 21 and 22 were energized approximately one hour and forty minutes following the reactor trip. The shorter time for the restoration of power was due to the experience gained during p

the December 21 event.

During the December 26 event, the operators were initially unsuccessful in the attempt to energize Bus Nos. 23 and 24 through the 12RYBT bus tie. With normal electrical supply from the 2MY leg of the Unit 2M (Auxiliary) transformer unavailable, operators placed normal supply

'

L Breaker Nos. 23-2 and 24-1 in pull-to-lock to prevent isolation of the power leg.

The alternate supply breakers were also placed in

y -

"

,,.,

e

!

pull-to-lock to isolate the buses, prior to closing in the upstream

,,

i-power supply,

!

The Prairie Island 4kV breakers all contain an anti pump circuit which is desianed to interrupt the close signal to a breaker, once L

the breaker closes. The purpose of the anti pump coil (Y) is to prevent pumping of the closing mechanism when closing against a faulted circuit. With the exception of Breaker Nos. 21-4, 22-4, 23-9, 24-9, (and the associated breakers for Unit 1) placing the breaker in the pull-to-lock position defeats the anti pump feature, and allows closure of the breaker.

On the December 21 event, operators closed the 12RYBT and powered

!

Bus Nos. 23 and 24 without any complications.

Breaker Nos. 23-9 and

,

24-9 (along with 23-2 and 24-1) were placed in pull-to-lock as normal procedure for a de-energized bus. Once the bus tie was closed (12RYBT), Breaker Nos. 23-9 and 24-9 were removed from pull-to-lock, and each was closed energizing its respective bus. This was done at 5:31 a.m. following the unit trip.

During the December 26 event, when operators followed the same process l

of providing power through bus tie 12RYBT, Breaker Nos. 23-9 and 24-9

!

did not close when the operator removed the breaker control switches (CS) from pull-to-lock to the closed position. The breakers were returned to pull-to-lock and the situation was assessed by electrical engineering personnel who were present in the control room. Although not immediately recognized, technical engineering personnel later determined that a unique anti pump circuit for these breakers prevented the anti pump coil from being de-energized (i.e. reset) unless the Unit 2 main generator output breaker 86 lockout relay had been reset.

-The operators then reset the 86 relays (2G and 2GT) first in the substation relay house, then in the control room as required, and Breaker Nos. 23-9 and 24-9 were closed without incident.

Although the alternate power source breakers for Bus Nos. 21 and 22 have the same requirement to reset the 86 relays, no problems were encountered. This was due to the prior resetting of the 86 relays for Bus Nos. 23 and 24.

The AIT reviewed several aspects of this inability to close the 23-9

!

and 24-9 breakers.

!

(1) Verification of design i

On January 3, 1990, the team members reviewed and witnessed a F

test to verify that the anti pump circuit would prevent closure

):-

of Breakers No. 23-9 and No. 24-9, with the electrical conditions that were present during the December 26 event.

Specifically, loads were removed from Bus No. 24 by transferring the 480V load to Bus No. 23 via several 480V bus ties and isolating the 480V supplies from Bus No. 24. All electrical supplies were isolated to Bus No. 24, specifically the 2M and 2R transformer supplies.'#%

The test had two purposes:

to prove Breaker No. 24-9 would automatically close onto Bus No. 24 without a 2G 86 lockout; and i

l l

b t

_

r c

.

.

!

l to show the breaker could not be closed with a 2G 86 lockout I

L signal present. The 2G lockout was tested. The 2GT lockout was not tested; however, this function is identical to the 2G i

function.

p The first phase of testing was accomplished by removing the 2G lockout which was provided only to Breaker No. 24-9.

As expected, once removed, the 24-9 breaker automatically closed since Breaker No. 24-1 (2M supply) was open.. The second phase was completed by inserting a 2G 86 relay lockout signal with Breaker No. 24-9 in

,

L pull-to-lock recreating the conditions present on December 26.

.

The operator attempted to close Breaker No. 24-9; however, the

!

!

breaker would not close with the 2G 86 lockout relay energized thus proving the assumption.

.

The AIT reviewed the testing, schematics for breaker logic and i

discussed the operation of the breakers in detail with the j

licensee and concluded the breakers operated as designed.

'

'

s (2) Root Cause

.

The cause of the 23-9 and 24-9 breakers riot closing when attempted was the result of not resetting the 2G and 2GT 86 lockcut relays prior to the attempt to close the breakers. This design was not understood until significant effort to understand the anomaly was made during the December 26 event.

During the December 21 event, the load dispatcher directed the plant to close the 8H13 and 8H14 main generator output breakers (which requires the 2G and 2GT breakers to be reset) shortly af ter the trip, because the breakers were leaking air.

Cold weather routinely results in the operating air integrity of air operated breakers to be diminished.

Since this was done during the December 21 event, the 86 relays (2G and 2GT) were reset early into the event, and the energizing of Bus Nos. 23 and 24

from the IR bus tie (12RYBT) was successful. The timing of resetting the 2G and 2GT relay was the only factor in re-energizing

,

Bus Nos. 23 and 24.

'

A substation Bus No I lockout in conjunction with a unit trip had not occurred at Prairie Island in the 16 years of operation to the best of record and recollection of the licensee staff.

Thus this

' design was not known or appreciated.

The team reviewed several procedures regarding this concern.

Operation Procedure C20.5 (Rev. 9), Plant 4.16V system, Section 4.1, " Energizing a de-ener-gized bus," was reviewed.

This is a generic procedure to provide guidance for restoring a de-energized bus. This general guidance is to remove loads, isolate supply breakers, identify any bus fault, restore an electrical supply and reload the bus. One criteria is to verify lockouts have been reset. This was written and interpreted to mean reset bus lockouts. Temporary Memo (TM)

90-01 was issued January 4,1990 for this procedure.

The memo discusses the need for resetting generator lockouts. On January 12, 1990, this guidance was added to the E20.5 procedure by Revision 40.

n

-

-

.

o 1.,

,

Yb h

The reactor trip recovery procedure IES-0.1, Rev. 8 Step 17h...

requires that restoration of the substation must have load

'l dispatcher permission. Action 3 of step 17h requires resetting F<

the (2)G and (2)GT relays.

This step is step 17 of 19 steps,

,

and is well into the recovery phase. The team concluded there i

U was no particular guidance other than general unit recovery j

associated with this step.

s The AIT concluded that the root cause vas a lack of understanding of the unique operating circuit configuration of Breaker Nos.

!

23-9, 24-9 by operating personnel, i

The guidance provided operators by procedures did not address E.

the main generator 86 lockout relay reset requirement. With recognition of this aspect of the breaker operating circuit, the provision for integrating generator lockout reset with

'

l'

alternate power availability for Bus Nos. 23 and 24 could have been made.

'

[

7.

Licensee Follow-Up and Review

._

.

As a result of the events, the licensee established three Task Forces to

,

t P

investigate the root causes of the problems with the substation breakers, l

!

the rod drive motor generator sets, and the rod control system.- The AIT i

reviewed the adequacy of the licensee's program for the troubleshooting, testing,-and analysis of the equipment quarantined by the December 27, 1990 i

i CAL through direct observation, review of daily status reports, meetings i

with the licensee, and telephonic meeting between the site, RIII, and Headquarters.

The AIT concluded that the licensee's investigation was methodical,

<

L conservative and technically sound.

Further, the licensee personnel involved in the investigation of the events were knowledgeable and extremely capable.- The AIT also concluded that the depth of the licensee's investiga-tion of the rod drive motor generator problems reestablished the greater part of the original design basis for this system and the knowledge of the

.

system that was gained will serve the licensee in good stead in.the future.

Therefore, the licensee's assumptions on the cause of the events (beginning December 21,1989), troubleshooting, testing, corrective actions, and final conclusions were appropriate throughout these events.

8.

Potential Tampering Review The licensee reviewed the possibility of personnel tampering with the

.

reactor controis or the switchgear at the substation, during the two trips y~

(December 21 and 26,1989).

This review considered the commonality of personnel during the three periods or the possibility of a disgruntled employee who, with adequate technical knowledge, would tamper with the i

controls. The licensee reviewed the personnel present in the control room

~

and the turbine building and concluded that except for one supervisory

,

engineer, there was no commonality of personnel during the periods in v

question. The licensee also reviewed the possibility of operating personnel having a possible grievance that could trigger a potential tampering of

.

W,.-c 3.

,

>

,'

.m g

,

equipment and concluded that both trips were the result of. equipment

'

D malfunction rather than any personnel tampering..Any further review

~

would entail interviewing personnel.

The licensee has no basis for such action at this time. On this basis, the' licensee has discontinued further

'

review into this matter. However, the licensee has assured the AIT that L

'if new information does surface, the review will be. reopened. The AIT

'

agrees with the licentee's assessment of the potential tampering by'NSP's-

'

operating personnel.

f 9.

Exit Meeting-The inspectors' met'with licensee representatives (denoted in Paragraph.1).

[

on January 9,1990, and-summarized the' purpose, scope, and findings of.the inspection.-.The inspectors discussed the likely informational content of

-.

b, the inspection report with regard to documents or processes reviewed by the

inspectors.during the inspection.

The licensee did not identify any.such documents or processes as proprietary.

,

'

n

.-

.

1

_.