ML17056B794

Insp Rept 50-410/92-81 on 920324-28.Violations Noted. Major Areas Inspected:Offsite & Onsite Power Sys & CR Annunciation Power Supplier

ML17056B794
Person / Time
Site:	Nine Mile Point
Issue date:	04/08/1992
From:	Beall J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I)
To:
Shared Package
ML17056B793	List: ML17056B792 ML17056B794
References
50-410-92-81, NUDOCS 9204200057
Download: ML17056B794 (60)

See also: IR 05000410/1992081

Text

U. S. NUCLEAR REGULATORY COMMISSION

REGION I

50-410/92-81

Qg~k~N

50-410

i

n

N

NPF-69

~ie~n~

acilit

Name

In

eci nAt:

~nducted;

Niagara Mohawk Power Corporation

301 Plainfield Road

Syracuse,

New York

13212

Nine Mile Point Nuclear Power Station, Unit 2

Scriba, New York

March 24-28, 1992

I~us ectors:

S. Hansell, Operations Engineer, DRS

J. Ibarra, Senior Instrument and Controls Engineer, AEOD

D. Brinkman, Senior Project Manager, NRR

ther

on ri

tin

NR

Personnel:

bserver

Approved by:

W. Schmidt, Senior Resident Inspector, DRP

R. Laura, Resident Inspector, DRP

P. Eddy, State of New York

'8

J.

Beall, Team Leader, DRS

pl

r

r

E

'

date

9204200057

920410

PDR

• DOCK 05000410

8

PDR

0

f

TABLE

F

NTENT

~Pa e

EXECUTIVE SUMMARY ......................................

2

1 e0

INTRODUCTION

~

~

~

~

~

~

~

a

~

~

~

~

~

~

~

~

~ '

t

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

3

1.1

1.2

The AIT Scope and Objectives ...........................

3

AIT Process

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

'

~

~

~

~

~

~

o

3

2.0 'OSS OF OFFSITE POWER AND CONTROL ROOM ANNUNCIATORS

~

~

~

4

2.1

2.2

2.3

2.4

Overview of Offsite and Onsite Power Systems ......... ~...

History and Design of Control Room Annunciation Power Supplies

.

Chronology ofEvents.........................-....

Highlights of the Event....

~ .,.................

~,

~

~

~

~

~

~

4

~

~

~

~

4

~

~

~

5

~

~

~

~

5

2.4.1

Loss of Control Room Annunciators

2.4.2

Loss of Offsite Power ..............-.....

~ ..

~

~

~

5

~ ~,o'

3.0

PERSONNEL AND NUCLEAR PL'ANT SYSTEMS PERFORMANCE

~

~

~

~

~

~

8

3.1

Equipment Performance

~

~

~

~

9

3.1.1

Power Supplies to Control'Room Annunciators......

3.1.2

Instrument AirSystem........;........

~ ..

~

3.1.3

Division IIIEDG Trip".......,...........

'..

~

~

~

9

9

10

3.2

Procedure Adequacy

3.3

Personnel Performance ..... ~...............

~

~

~

~

~

~

~

11

12

3.3.1

Auxiliary Boilers Relay Calibration Work Package

3.3.2

Meter and Test Technicians..............

3.3.3

Control Room Operators

12

13

13

4.0

GENERIC IMPLICATIONSOF THIS EVENT ....................

14

5.0

LICENSEE CORRECTIVE ACTIONS .........................

14

5.1

Immediate Corrective Actions.................

5.2

Short Term and Long Term Corrective Actions...'..'.

14

15

1

Table of Contents

6@0

CONCLUSIONS

o

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

~

o

1 5

7.0 'ANAGEMENTMEETINGS...............................

16

TABLE 1 - Chronology of Events

APPENDIX A - NRC Augmented Inspection Team Charter

APPENDIX B - Persons

Contacted

APPENDIX C - Documents Reviewed

FIGURE 1 - Nine Mile Point Unit'2, Simplified Electrical Distribution

EXE

TIVE

MMARY

On March 23, 1992, at about 10:00 a.m., licensee personnel inadvertently tripped one of the

two lines supplying offsite power to Nine Mile Point Nuclear Power Plant Unit 2, which was

shutdown with about one-third of the core offioaded.

The line loss led to a loss of control

room annunciators

and the declaration of an alert.

The second offsite power line also was

inadvertently tripped causing a total loss of offsite power.

Several minutes later, one of the

two running emergency diesel generators

(EDGs) tripped, due to loss of cooling water.

An augmented

inspection team (AIT) was dispatched by the NRC to determine the

circumstances

that led to this event, its causes,

safety significance and generic implications,

and the adequacy of the licensee's

response

to the event.

The AITbegan its assessments

on

March 24, 1992, completed its onsite reviews on March 28, 1992, and presented

its

preliminary findings in a public exit meeting on April 1, 1992.

A technician initiated the event by bumping contacts closed while replacing a glass relay

cover.

Human factors problems contributed to the inadvertent performance error.

One of the two sources of power to the control room annunciators

(UPS 1A) lost power and

was unable to transfer to another source due to an internal fault. The fault had been

identified 14 days before the event, had been assigned

a "C" priority (less than seven days),

and had been scheduled for work 2 days after the event.

This work delay indicates a need

for review of oversight of work prioritization and scheduling.

The remaining annunciator power source (UPS 1B) was not able to sustain the load, tripped

on overload within 30 seconds,

and control room annunciators

were lost.

The current design

represents

a continued vulnerability to a loss of control room annunciators during shutdown

or plant transients (high alarm and annunciator loads) ifcoincident with the loss of the non-

safety UPS 1A,

The internal UPS 1A fault was found to be a failed internal logic battery.

These batteries

were replaced following the IITwith a planned replacement cycle of about 15 months.

The

bad battery was sent to the vendor by the licensee for a failure analysis.

The earlier than

expected failure (seven months vice planned fifteen months'eplacement

interval) requires

followup.

Operator performance was acceptable.

The total loss of offsite power was caused by a

combination 'of inadequate work package plant impact assessment

and inaccurate technician

information.

Weaknesses

in management

support to operations also contributed.

The Division IIIEDG tripped during the event due to the loss of service water cooling.

This

trip disclosed a previously unrecognized

design vulnerability to a sequential loss of offsite

power sources.

That is, the offsite lines are lost not simultaneously, but separated

by

15 seconds or more.

In this scenario,

should the first EDG fail or be out of service (as on

March 23), then the Division IIIEDG willtrip unless it is followed by rapid operator action.

This represents

a potential generic issue for plants with EDGs that don't supply their own

auxiliary and support systems.

The consequences

of this event were minimal. The facility was in refueling with the reactor:

cavity flooded.

The reactor core and the reactor coolant system were unaffected, no

equipment was damaged

and no radioactivity was released.

D

TIN

Upon being informed of the loss of offsite power and control room annunciators

at

Nine Mile Unit 2 on March 23, 1992, the NRC Region I Regional Administrator and

senior management

from the Office of Nuclear Reactor Regulation (NRR) and the

Office for Analysis and Evaluation of Operational Data (AEOD) determined that an

Augmented Inspection Team (AIT) should be formed to review and evaluate the

circumstances

and significance of this occurrence.

The basis of the NRC concern was

the apparent inadequacies of management

controls of maintenance activities that

.

allowed the event to occur.

Accordingly, an AITwas selected, briefed, and the AIT

leader dispatched to the site on March 24, 1992.

~ 1.1

Th

AIT c

nd

ectiv

The charter for the AIT (Appendix A) was finalized on March 24, 1992.

The

charter directed the team to conduct an inspection and accomplish the

following objectives:

Conduct a timely, thorough and systematic review of the circumstances

surrounding the event, including the sequence of events that led to and

followed the March 23, 1992 loss of offsite power and control room

annunciator;

2.

Collect, analyze and document relevant data and factual information to

determine the causes,

conditions and circumstances

pertaining to the

event, including the response

to the event by the licensee's operating

staff;

3.

Assess

the safety significance of the event and communicate to

Regional and Headquarters

management

the facts and safety concerns

related to the problems identified; and

4.

Evaluate the licensee's review of and response

to the event and

implemented corrective actions.

t.2

A TP

During the period March 24-28, 1992, the AIT conducted an independent

inspection, review, and evaluation of the conditions and cir'cumstances

associated

with this event.

The team inspected the event-initiating relays, the

offsite power supply breakers,

control room annunciators'ower

supplier, and

related indications in the control room;

held discussions

and formal interviews

with personnel involved in this event; reviewed relevant records including

computer printouts before, during, and after the event, and trends of pertinent

4

plant parameters;

and evaluated the adequacy of established

procedures,

management

oversight, and personnel training.

Appendix B is the list of

personnel contacted by the AITand Appendix C is the list of documents

reviewed by the AIT members.

This inspection was conducted in accordance with NRC Manual Chapter 0513,

Part III, Inspection Procedure 93800, Regional Instruction 1010.1 and

additional instructions provided in the AITcharter.

2.0

F

FF ITEP WERAND

NTR LR

M

IAT R

2.1

rview f ffi

n

ni

P wr

m

Nine Mile Point Unit 2 receives power from two offsite 115 kV lines, called

line 5 and line 6.

The breakers for these lines are located in the Scriba

.

switchyard approximately one-half mile from the site.

Figure

1 is a simplified

one-line diagram of the portion of the electrical distribution involved in the

,March 23, 1992 event.

At the time of the event, line 5 was supplying power to the Division I and III

buses through transformer RSST-A with R-50, motor-operated

disconnect

switch MDS-3 an'd the Division I and IIIbreakers all shut.

The auxiliary

boilers were also being powered from line 5 with MDS-10, MDS-5, and the

auxiliary boilers breaker shut,

The Division II bus was being supplied by

line 6 through R-60, MDS-4, transformer RSST-B and the Division II breaker.

Open were MDS-20, the Division IIIbreaker from transformer RSST-B and

all the EDG breakers to their respective buses.

The Division I EDG was out

of service for planned maintenance

and the Division II and IIIEDGs were

aligned for automatic start on loss of power to their respective buses.

The Division IIIEDG is dedicated to supply power to the High Pressure

Core

Spray (HPCS) system.

The remaining safety-related equipment is divided

between Divisions I and II. The HPCS system was not required to be

operable at the time of the event, but was available as a backup for adding

water to the reactor coolant system and the flooded reactor cavity.

2.2

Hi t

D in f

nr]R

m

nni inPwr

lier

The Nine Mile Point Unit 2 control room annunciators

are powered jointly

from two uninterruptible power supplies (UPS), UPS-1A and UPS-1B.

Upon

loss of one UPS, the annunciator power supply circuit of the other continues to

supply all the loads alone. Ifan overload condition results, the remaining

power supply would trip, as it did on March 23, 1992, and all control room

annunciators would become deenergized.

I

Each UPS can supply the annunciators by using an ac or a dc (station

batteries) input. A separate

maintenance

ac source also can be used to supply

the annunciator circuit directly in case of UPS failure or planned UPS

maintenance.

Each UPS unit has a control logic module which monitors

voltage and which initiates automatic transfers to maintain power to the

annunciators.

Each UPS control logic module also contains internal battery

packs (three "D" cells per pack) for certain functions.

Proper voltage from

these internal batteries was necessary for annunciator power to be transferred

from the maintenance

supply and back to the UPS.

For additional details on UPS design,

see NUREG-1455, "Transformer Failure

and Common-Mode Loss of Instrument Power at Nine Mile Point Unit 2'on

August 13, 1991."

2.3

Chr nol

of Event

The AITcompiled independently a detailed chronology of events by

interviewing cognizant personnel, reviewing relevant records including

computer printouts before, during, and after the event, and trends of pertinent

plant parameters.

This detailed chronology is provided in Table 1,

2.4

Hi hli hs f h

Ev nt

f

n r I R

m Annuncia

r

On March 9, 1992, UPS-1A automatically transferred power, as

designed,

to the annunciators

to the maintenance

source,

This transfer

occurred due to a voltage fluctuation caused by testing on another

circuit. The UPS-1A control logic sensed

the voltage drop and

properly initiated the transfer.

Licensee personnel were unable to return the annunciators

to their

normal source.

The cause of this problem was not known and a Work

Request (WR 201538) was initiated with a "C" priority (less than seven

days) to troubleshoot the failure'o transfer.

The loss of line 5 (see Section 2.4.2) on March 23, 1992, deenergized

the annunciator circuits powered from UPS-1A.

Power remained

available from the large station batteries but, as had been identified two

weeks previously, UPS-1A was not able to transfer to another supply.

For a short period of time (less than 30 seconds),

UPS-1B maintained

the control room annunciators.

Because the plant was shutdown, many

alarms and annunciators were energized causing a high load on the

circuit. UPS-1B was unable to continue to supply the demand alone,

and the annunciator circuit tripped on overload.

Power was restored to the control room annunciators after about

80 minutes.

2.4.2 ~ffff

P

On March 20, 1992, meter and test technicians began calibrating relays

using an electrical preventive maintenance procedure (No. S-EPM-

GEN-2Y070).

The work package had been prepared

a month earlier,

but had been postponed

as part of the work package review activities

which were corrective actions following the Nine Mile Point Unit

1

loss of ultimate heat sink (see NRC Inspection Report 50-220/92-80).

The plant impact assessment

in the work package did not address

effects should any of the relays actuate during the job.

On March 23, 1992, a technician inadvertently bumped contacts of the

backup overcurrent protection relay on the "B" phase of the auxiliary

boilers supply breaker.

At the time, the technician had completed

calibration of the relay and was replacing the glass cover.

The meter

and test personnel

noted that a relay trip had occurred, notified the

control room, and were directed to come to the c'ontrol room to

explain.

~

The auxiliary boiler relay trip (type 86) had also tripped the next

breaker up the circuit (see Figure 1), R-50.

The relay logic included

two sets of contacts which effectively mimicked the power circuit up to

MDS-5.

Since R-50 and MDS-10 were closed, a current path existed

which picked up another relay (type 94), which tripped R-50. At the,

time of the event, MDS-20 was open so no current path existed to

actuate the type 94 relay for the line 6 breaker, R-60.

The trip of R-50 deenergized all equipment being supplied from line 5.

This equipment included UPS-1A, two of the three air compressors,

the

Division I bus and the Division IIIbus.

After about 10 seconds,

the

Division IIIEDG successfully completed an automatic start and

restored power to the Division IIIbus.

After about 20 seconds,

control

e

room annunciators were lost due to the inability of the remaining power

supply (UPS-1B) to sustain the load.

The Division I bus, which had

included one of the two running service water pumps, remained

deenergized

since its dedicated EDG was out of service for planned

maintenance.

Control room operators took a number of initial actions in response to

the event.

These actions included dispatching an operator to the refuel

floor to continuously monitor water level, debriefing the meter and test

personnel on the cause of the event, and declaring an Alert in

accordance with the licensee's Emergency Plan.

About 10 minutes after the event began, the last running air compressor

tripped due to loss of cooling water caused by the loss of line 5. Air

pressure,

already dropping, began to fall more rapidly.

Operators were

concerned

as to the continued leak tightness of the air bladder seals

around the outside of the reactor vessel and in the submerged

main

steam lines.

About 18 minutes after line 5 tripped, control room operators attempted

to restore power to the deenergized

equipment (including the

annunciators) by closing MDS-20 and using power from line 6.

Control room operators

had called the utility's offsite power control

center in Syracuse

and had been told that the travelling switchyard

operator (based about 20 miles south of the plant in Fulton) would be

radioed to go to the Scriba switchyard and close the offsite breakers.

Syracuse personnel estimated a half hour response

time.

The decision to use line 6 followed discussions with the meter and test

personnel who stated that there were no problems with this approach.

This operation also received the concurrence of the senior control room

supervisor.

The Operations Manager and the Plant Manager were also

present at the time with the latter being briefed prior to assuming his

role as the Emergency Director at the Technical Support Center.

The closure of MDS-20 completed the controls logic associated

with the

still tripped type 86 relay for, the auxiliary boiler; this resulted in the

trip of R-60 and the loss of line 6. At this time, Unit 2 had lost all

offsite power (LOOP).

After about 10 seconds,

the Division II EDG successfully completed an

automatic start and reenergized

the Division II bus,

Shortly thereafter,

SWP-1B automatically restarted which restored service water to

Division II loads.

Operators also promptly restarted RHS-P1B to

restore shutdown cooling.

There was no apparent rise in reactor

coolant system temperature during the less than two minute interruption

in shutdown cooling flow.

Service water cooling remained interrupted to the running Division III

'DG due to a potential design deficiency (see Section 3.1.3).

The

Division IIIEDG tripped due to loss of cooling water after about seven

minutes.

Control room operators, in conjunction with meter and test personnel,

determined that failure to reset the initial tripped type 86 relay had

caused the opening of R-60 upon MDS-20 closure.

The type 86 relay

was reset and the Fulton-based operator arrived at the Scriba

switchyard and reclosed the R-50 and R-60 breakers.

The initial

attempt to restore power to the line 5 loads was not successful

because

MDS-3 would not close.

Operators initiated a work request to

troubleshoot MDS-3 but successfully

used a different lineup (from

line 6) to restore power.

No problems were found during

troubleshooting efforts and MDS-3 was later closed successfully.

Upon restoration of the control room annunciators,

operators noted high

reactor building sump level alarms.

The levels were found to be the

result of normal leakage which continued while the sump pumps were

deenergized.

After power w'as restored to the pumps, sump levels

"

returned to normal.

The licensee restored the plant to a normal shutdown lineup and, after

discussions with the NRC and offsite organizations,

terminated the

Alert.

30

PER

L AND

LEAR PL NT

Y TEM

PERF

RMAN E

The AIT assessed

the performance of the personnel and the plant systems before,

during, and after the event.

The findings of the AITare grouped into three broad

categories:

Equipment Performance;

Procedure Adequacy; and Personnel

Performance.

i mn Perfrm

The equipment performed as expected, with the exception of the power

supplies to the control room annunciators,

the instrument air system, and the

Division IIIEDG. Also, MDS-3 initiallyfailed to close, but it was inspected

by the licensee,

no problems were found, and it closed successfully on the next

attempt.

3.1.1

w r

i

R

m

As discus'sed in Section 2.4.1, control room annunciator power from

UPS-1A was from the maintenance supply at the time of the event.

Loss of line 5 deenergized

the maintenance

supply and the power

supplies from UPS-1B sustained the load for less than 30 seconds

before tripping and deenergizing the control room annunciators.

A WR

had been initiated on March 9, 1992, with a "C" priority (less than

7 days) to troubleshoot the inability of UPS-1A to transfer, but no work

had begun as of March 23, 1992.

Post-event investigation determined that one of the small internal

battery packs had failed and, after battery replacement,

UPS-1A was

able to transfer properly from the maintenance

supply.

The licensee

sent the failed battery to a vendor for a failure analysis.

The licensee

planned replacement interval for these batteries was 15 months.

The

failure of one battery after about seven months requires followup.

3.3.2

The loss of line 5 deenergized

two of the three air compressors

and air

pressure

began to drop.

Operators took various actions as valves began

to drift to their failed (on loss of air) positions causing, in some cases,

system transients.

One such transient resulted in the loss of cooling

water to the remaining running air compressor which tripped and

increased

the rate of air pressure

loss.

Operators were also concerned about the possibility of leakage through

the seals around the reactor vessel and in the main steam lines.

The

seals are designed not to fail catastrophically on loss of air but do use

air pressure to enhance leak tightness.

An operator was immediately

dispatched

to continuously monitor water level in the flooded reactor

cavity.

10

Although there was no evidence of seal leakage during the event, the

operators were concerned about possible seal leakage or failure due to

loss of air pressure.

Operators were also required to take various other

actions to mitigate transients caused by loss of air pressure.

No backup

air supply, such as a diesel air compressor for the system or

pressurized

bottles for the seals was provided during the refueling

outage.

3.1.3

Divi i n III D

The Division IIIEDG started,

as designed,

due to undervoltage on the

Division IIIbus caused by the loss of line 5. The EDG supplied the

bus until it tripped on high jacket water temperature

caused by the loss

of service water flow following the loss of line 6.

At the time of the event, service water was available to the Division III

EDG from service water pumps (SWP) on the Division I bus (SWP-1A)

and the Division II bus (SWP-1B).

The loss of line 5 deenergized

the

Division I bus, tripping SWP-1A.

The Division I EDG was out of

service undergoing planned maintenance

so the Division I bus remained

deenergized

until power was restored to it"from an offsite source.

Service water continued to be supplied to the running Division IIIEDG

from SWP-1B, which was powered from the Division II bus being-

energized from line 6.

The service. water supply valves to the Division IIIEDG automatically

shut on low water pressure but require operator action to open.

This

automatic closure feature is blocked for the first minute of Division III

EDG operation.

The time delay is sufficient for the Division I and

Division II EDGs, ifstarted simultaneously with the Division IIIEDG,

to come up to speed and sequence

the service water pumps onto the

buses and restore system pressure.

The design, therefore, assumed

the

simultaneous

loss of offsite power (LOOP) and start of all EDGs.

'he

March 23, 1992 event, involved a sequential loss of offsite power

with 18 minutes between the loss of line 5 and line 6.

The trip of

line 6 deenergized

the Division II bus, tripping the remaining running

service water pump (SWP-1B).

The Division II EDG successfully

started automatically, reenergized

the Division II bus, and the safety-

related components (including SWP-1B) on the bus were reenergized

in

the proper sequen'ce.

The interruption in service water pressure

and

11

flow caused the closure of the Division II service water supply valve to

the running Division IIIEDG since the one minute duration closure

block on the valve had begun 18 minutes earlier when line 5 was lost.

The Division IIIEDG tripped due to loss of cooling'water about

seven minutes later.

The trip of the Division IIIEDG demonstrated

a previously not

recognized vulnerability to a sequential LOOP.

The interrelationship in

the controls logic of the three EDGs would lead to the loss of cooling

water to the Division IIIEDG in the following generalized

scenario:

~

Loss of the offsite line supplying Division IIIand Division I (or

Division II) buses, coincident with main generator trip (or bus

transfer not available);

~

The Division IIIEDG starts successfully but the other EDG fails

(or is not in service);

~

Loss of remaining offsite line 15 seconds or more later.

The possibility that a sequential LOOP may be more limiting than a

simultaneous LOOP is a potential generic issue.

The above scenario

also indicates the net to reassess

the NMP-2 emergency core cooling

system (ECCS) with respect to this previously not recognized

vulnerability.

3.2

Pr

edure Ad

ac

In general,

the licensee's

procedures

were adequate,

and they provided

direction to control the event,

However, the plant electrical procedures

were

cumbersome

and in some cases contained generic information in lieu of

specific details.

An example is, the system operating procedures for the

115 kV Switchyard, 13.8 kV/4.16 kV/600V, and Emergency A.C.

Distribution systems all contain the same generic information for a "Loss of

Bus" event.

This type of procedure format does not ensure a consistent

operational practice between all operating crews.

The overhead alarm response procedure for the tripped backup overcurrent

relay No. 852426, "Aux Boiler Transformer BU LKO Relay Trip" did not

contain the relay impact on 115 kV line 5 or 6.

12

Procedure N2-OP-72, "Standby and Emergency A.C. Distribution," was the

only procedure that specifically stated to reset lockout relays (86 overcurrent

devices) prior to energization of the 4.16 kV 1E busses.

Operating procedures

N2-OP-70, "Station Electrical Feed and 115 kV Switchyard," and N2-OP-71,

"13.8 kV/4160V/600V A.C. Power Distribution," did not direct the operators

to reset tripped relays prior to energizing electrical busses or distribution lines.,

The operating procedure used to cross-tie 115 kV Line 6 to reserve station

service transformer 1A, N2-OP-70 section H.5.0, did not reference the

protective interlock trip from the auxiliary boiler transformer regular or

backup overcurrent relays (86 devices).

The procedure used to return 115 kV Line 5 to service is an example of a

cumbersome procedure.

Procedure N2-OP-70 section 7,0 provides direction to

first, go to procedure N2-OP-71 section H, and perform steps 1.0.a through

1.0.n or H.5.0 as applicable.

The operator returns back to procedure N2-OP-

70 for nine steps.

After completing the nine steps, procedure N2-OP-70

directs the operator back to procedure N2-OP-71 section H.6.0.

The first step

in N2-OP-71 section 6.0.a. refers the operator back into procedure N2-OP-70

section E,

N2-OP-70 section E.2.0 and 3.0 provide direction to energize

reserve station service transformer 1A. When the reserve station service

transformer 1A is energized,

the operator returns to procedure N2-OP-71

section 6.0.b. to complete the restoration of power to the 13.8 kV and

4.16 kV buses.

3.3

Per

nnel Perf rmance

3.3.1

Auxilia

iler

Rela

li

i n W rk Packa

e

The work package prepared for the calibration of the auxiliary boiler

backup overcurrent protection. relays was not adequate.

The package

did not contain all the drawings needed to understand

the effects should

any of the relays be actuated.

The plant impact assessment,

which was

part of the work package, did not address

the impact of inadvertent

relay actuation during the work. Work package review activities,

which were corrective actions following the Nine Mile Point Unit 1

loss of ultimate heat sink (see NRC Inspection Report 50-220/92-80),

did not identify and correct these deficiencies.

Similar problems were found in other relay work packages,

but a

review of a sample of non-relay work packages did not identify similar

deficiencies.

13

3.3.2

M

r

n

T

T

hni i

The performance of the meter and test personnel during the event was

weak.

The technician working at the relay and replacing the cover did not

know the effects of inadvertent relay actuation.

The technician

performance error during cover replacement was inadvertent and, due

in part, to human factor problems.

Only a small positioning error was

necessary

to bump the contacts closed and the technician had to lean

over in a kneeling position to replace the cover on the relay which was

just a few inches from floor level.

The prompt, forthright disclosure of the performance error helped

control room operators understand

the cause of the loss of line 5.

Due

to the concurrent loss of control room annunciators,

operators were not

able to identify immediately the cause of the event and thought initially

that it had originated offsite.

The ASSS asked the meter and test personnel, who had come to the

control room to explain the trip, ifthere was any problem in closing

MDS-20 preparatory to using power from line 6 to reenergize lost

buses,

The meter and test personnel inaccurately stated that there was

no problem in the proposed course of action.

Closure of MDS-20

caused

the trip of R-60 and the loss of line 6 (LOOP), since the initial

relay trip had not been reset.

3.3,3

n r l R

m

The licensed operators'esponse

to the loss of 115 kV offsite electrical

power was good.

The shift correctly classified the loss of control room

annunciators

as an Alert. The operators

made a reasonable

decision to

restore electrical power to the Division I distribution system from

115 kV line 6.

The factors affecting their decision were:

1) continued

loss of all control room overhead annunciators;

2) dropping air pressure

resulting in the possibility of leaking main steam line plug and refuel

cavity seals, which could have led to a loss of water in the reactor

cavity and spent fuel pool; 3) known delays, of at least one half hour,

to restore the normal electrical power to reserve station service

transformer "A" from the offsite 115 kV switchyard; 4) the auxiliary

boiler transformer was electrically isolated (the overcurrent relay that

was inadvertently tripped was a protective device for the auxiliary

boiler transformer); 5)

when asked by the assistant

station shift

supervisor (ASSS), the relay personnel informed the operators that

14

there was no threat to 115 kV line 6 ifthey used Line 6 to restore

power to reserve station service transformer 1A; 6) the relay senior

tester had ten years experience at NMP-2, was familiar with the relay

protective trips, and was the technical expert in the area of relay

knowledge.

The restoration of power was good even though the electrical system

operating procedures

were cumbersome.

The procedure problems

details are discussed in section 3.2.

The station shift supervisor (SSS) exhibited good command and control

while conducting the plant restoration.

Specifically, the restoration of

the residual heat removal (RHR) shutdown cooling system was timely.

The operators'ecisions

for plant restoration were influenced by some

circumstances

beyond their control.

Examples are listed below.

~

The restoration of the 115 KV electrical power required a

traveling operator, based out of Fulton, New York.

~

The overhead annunciator alarms were lost 20 seconds after the

loss of 115 kV line 5.

~

The plant did not have a backup contingency for the loss of

instrument and service air.

4.0

ENERI

IMPLI ATI N

F THI EVENT

The AIT reviewed this event for generic implications and identified one item that has

potential generic implications,

The trip of the Division IIIEDG (see Section 3.1.3)

demonstrated

that a sequential LOOP might be more limiting, in some cases,

than an

'nstantaneous

LOOP.

5.0

LI EN EE

RRE TIVE A TI N

5.1

Immediate

rrective A i n

1.

Immediately following the event, the plant manager issued a stop work

order with respect to all relay work.

2.

A review of all relay work requests

was ordered.

15

3.

An assessment

organization was formed to investigate the circumstances

leading to this event, plant response

and personnel action before,

during, and after the transient, and the root cause of the event.

These

actions included assessing

possible modifications to prevent recurrence

of the Division IIIEDG trip.

5.2

h

T rm

n

T rm

iv A'

In addition to the immediate actions described in Section 5.1, the licensee

implemented or announced

planned corrective actions to address

weaknesses

and concerns identified following the event.

These short term and long term

corrective actions were in the following areas:

work control, meter and test

technician job performance,

operator performance, Division IIIEDG

reliability, and control room annunciator power supply reliability.

The licensee was requested

to document and discuss these short term and long

term corrective actions by letter to the NRC within 30 days of receipt of this

report.

The effectiveness of the corrective actions will be reviewed as part of

the routine inspection program.

6.0

NL

I N

The AIT concluded that the cause of the initiating event (relay trip of line 5) was an

inadvertent technician performance error caused, in part, by human factors problems.

The loss of control room annunciators

was caused due to a continued vulnerability in

the current design during shutdown or transient conditions.

Delays in addressing

a

known problem in one source of power to the annunciators

(UPS-1A) indicates a need

for review of oversight of prioritization and scheduling.

The AIT identified the following weaknesses

in management

support of operations:

1)

Acceptance of delays in operation of offsite power supply breakers inherent in

the use of a travelling operator based in Fulton, New York.

2)

Absence of a backup air supply during the refueling outage with air 'pressure

being used in reactor vessel and main steam line seals.

3)

Acceptance of cumbersome,

generic procedures which require operators to use

concurrently several procedures during an event such as a LOOP.

16

The Division IIIEDG was not required to mitigate this event, but the trip of the EDG

demonstrated

a generic and previously not recognized vulnerability (see Section 3.1.3)

that requires followup.

II

The consequences

of this event were minimal because the reactor core and the reactor

coolant were unaffected by this event, there was no equipment or structural damage,

and no radiation was released.

7.0

ANA EMENT MEETIN

The licensee management

was informed of the scope of this AITduring an entrance

meeting on Tuesday, March 24, 1992.

The licensee management

was briefed of the

inspection observations routinely and at the conclusion of onsite review on Saturday,

March 28, 1992.

A public exit meeting was conducted on April 1, 1992 at 1:00 p.m. at the licensee's

training facilities with licensee representatives

identified in Appendix C to discuss the

preliminary inspection findings.

The licensee acknowledged the inspection findings

and provided the results of their assessment

of the event and the short and long term

corrective actions for both units.

TABLE 1

HR N L

Y

FE

3/09/92

UPS 1A transferred automatically to maintenance power supply due to voltage fluctuation

associated

with energization of main turbine EHC system.

Attempts to manually transfer UPS 1A back to its normal source were unsuccessful.

Work

Request (WR) 201538 was initiated with a seven-day priority to resolve problem.

3/20/92

Meter and test technicians began calibrating relays using electrical preventive maintenance

procedure (No. S-EPM-GEN-2Y070)

3/23/92

E

Initial Plan

n itions:

The plant was in the refueling mode with the reactor vessel head removed and the reactor,

cavity flooded to normal refuel level.

Approximately on-third of the core had been

transferred to the spent fuel pool, but no refueling activities were in progress.

Emergency

diesel generator (EDG-EGS1) and associated

ECCS systems were out-of-service for planned

maintenance.

All'other equipment was in a normal shutdown lineup for existing plant

conditions.

The electrical lineup is shown on Figure 2.

1008

Auxiliary boiler overcurrent protection relay actuated during cover replacement by

technician following calibration,

At this time offsite 115 kV power line 5 tripped

which deenergized

UPS-1A and the Division I and the Division IIIsafety-related

4 kV

boards.

Loss of the 4 kV boards stopped the Division I service water pump (SWP-

1A) and instrument air compressor (C1A).

After about 10 seconds,

EDG-EG2 successfully completed an automatic start and

restored power to the Division III4 kV board.

After about 20 seconds,

control room

annunciators were deenergized

due to inability of the remaining power supply (UPS-

1B) to sustain the load.

1009

Operator sent to refuel floor, confirmed no drop in water level, established

communications with the control room, and remained there to monitor level.

1016

Licensee declared Alert in accordance with the emergency plan due to loss of control

room annunciators.

1018

The remaining running air compressor

(C1B) tripped due to loss of cooling water,

Table

1

1026

Operators closed MDS-20 in attempt to reenergize UPS-1A and the Division I 4 kV

board from offsite 115 kV power line 6.

This resulted in loss of line 6 due to failure

to first reset relay which initiated the original trip of line 5. At this time, the site had

'ost all offsite power (LOOP).

Loss of line 6 deenergized

the Division II 4 kV board which tripped SWP-1B (the

only remaining service water pump) and RHS-P1B (the pump providing shutdown

cooling). About 10 seconds later, EGS-EG3 successfully completed-on automatic

start and restored power to the Division II 4 kV boards.

1027

SWP-1B automatically restarted per design, restoring service water for Division II

, loads.

1028

RHS-P1B restarted

(shutdown cooling restored).

1033

EDG-EG2 tripped on high jacket water temperature.

1044

Operations personnel reset auxiliary boiler overcurrent protection relay which had

initiated trip of line 5 and, later, line 6.

1046

Offsite power available to Division II 4 kV board (powered at this time from the

running EDG-EG3).

1055

Initial attempts to restore power to UPS-1A, the Division I 4 kV board and the

Division III4 kV board were not successful

(MDS-3 would not close);

1131

UPS-1A restored which restored control room annunciators,

1136

Instrument air pressure

restored to normal.

1144

Division I 4 kV board reenergized

(via line 6 through the auxiliary boiler

transformer).

1221

Division III4 kV board reenergized

(via line 6 through RTX-1B).

1245

EDG-EG3 removed from Division II 4 kV board (powered at this time from line 6);

I

1307

SWP-P1A restarted.

1317

Alert terminated.

~0.8 RINGO

4

O

if***4

APPENDIX A

UNITED STATES

NUCLEAR REGULATORY COMMISSION

REGION I

475 ALLENDALEROAD

KING OF PRUSSIA, PENNSYLVAN(A19406.1415

MAR 24 892

MEMORANDUMFOR:

Marvin W. Hodges, Director

Division of Reactor Safety

Charles W. Hehl, Director

Division of Reactor Projects

FROM:

SUBJECT:

Thomas T. Martin

Regional Administrator

AUGMENTED INSPECTION TEAM (AIT) CHARTER - LOSS

OF OFFSITE POWER WITH COMPLICATIONS

You are directed to perform an Augmented Inspection Team (AIT)review of the causes,

safety

implications, and associated

licensee actions which led to the inadvertent loss of offsite power

and control room annunciations

at Nine Mile Point Station Unit 2.

The basis of the NRC

concern is the management

control of maintenance

activities that allowed the event to occur.

,The inspection shall be conducted

in accordance

with NRC Manual Chapter 0513, Part III,

Inspection Procedure 93800, Regional Instruction 1010.1

and additional instructions

in this

memorandum.

DRS is assigned

responsibility for the overall conduct of this inspection.

DRP is assigned

responsibility for resident

inspector

and clerical support and coordination with other NRC

offices. J. Beall is designated

as the onsite Team Leader.

Team composition is described

at the

end of this memorandum.

Team members are assigned to this task until the report is completed

and will report to Mr. Beall.

OBBJR

V

The general objectives of this AITare to:

Conduct a timely, thorough, and systematic review of the circumstances surrounding the

event, including the sequence ofevents that led to and followed the March 23, 1992, loss

of offsite power and control room annunciator panels;

b.

Collect, analyze,

and document relevant data and factual information to determine the

causes,

conditions, and circumstances

pertaining to the event, including the response

to

the event by the licensee's operating staff;

MAR 2 4 1392

Marvin W. Hodges

Assess

the

safety

significance of the

event

and

communicate

to

Regional

and

Headquarters

management

the facts

and

safety

concerns

related

to the problems

identified; and

d.

Evaluate the appropriateness

of the licensee's review of and response

to the event and

implemented corrective actions.

PE

FTHEIN P

The AIT should identify and document the relevant facts and determine the probable causes of

the event.

It should also critically examine the licensee's

response

to the event.

The Team

Leader shall develop and implement a specific, detailed plan addressing

this event.

The AIT should:

a.

b.

Develop a detailed chronology of the event;

Determine the root causes of the event and document equipment problems,

failures,

and/or personnel errors which directly or indirectly contributed to the event.

Potential items to be considered:

~

Operator actions during and following the event;

~

Outage

planning includirig adherence

to controls

and

schedule

to minimize

shutdown risk.

~

Management and administrative controls in place before, during and following the

event.

Coordination of maintenance

activities before and during the event, including

worh planning and control.

Assess the performance of the UPS system during the event.

Sensitivity to plant conditions.

~

Communications among plant personnel

and with/within control room.

I

~

~

VAR 2 4 1992

Marvin W. Hodges

C.

Determine the expected

response of the plant and compare it to the actual response.

Assess plant response including but not limited to the following:

~

EDG IIItripping

~

Cause of delays in restoring offsite line five

~

Cause of the reactor building sump alarms

d.

e.

Assess the adequacy of the responses of the operations and technical support staffs to the

event and the initial licensee analysis.

Assess licensee actions in restoring power.-

I

Determine the management response including the scope and quality ofshort-term actions

and gather information related to the long-term actions intended to prevent recurrence of

this event, including internal and external communications/dissemination

of licensee-

identified concerns and corrective actions.

f.

Determine the relationship ofprevious events or precursors, including site-related, ifany,

to this event.

g.

Determine the potential generic implications of this event, such as recommend

lessons

learned, and the necessity for generic industry communications.

,"i~HEDULE

The AIT shall be dispatched

to Nine Mile Point Station Unit 2 so as to arrive and commence

the inspection on March 24, 1992. A written report on this inspection shall be provided to me

within three weeks of completion of the onsite inspection.

TEAM

MP

ITI N

The assigned Team members are as follows:

Team Manager:

Onsite Team Leader:

Onsite Team Members:

M. Hodges, DRS

J. Beall, DRS

S. Hansell, DRS

D. Brinkman, NRR

J. Ibarra, AEOD

homas T. Martin

Regional Administrator

MAR 24 892

'arvin

W. Hodges

CC:

W. Kane, DRA

W. Hehl, DRP

C. Cowgill, DRP

L. Nicholson, DRP

Team Members

R. Lobel, OEDO

J. Calvo, NRR

R. Capra, NRR

K. Abraham, PAO

L. Spessard,

AEOD

E. Rossi, NRR

W. Lanning, DRS

J. Durr, DRS

L. Bettenhausen,

DRS

SRI NMP-1

APPENDIX B

PER

N

NTA TED

MhawkP wr

ti n

~Nme

C. Beckham

J. Burton

T. Collins

M. Conway

R. Crandall

K. Dahlberg

J. Darling

A. DeGracia

S. Doty

E. Dragomer

J. Endries

M. Eron

C. Gerberich

D. Hanczyk.

D. Hosmer

A. Julka

D. Kinney

H. Lockwood

D. Lomber

M. McCormick, Jr.

R. Menikheim

J. Pavel

F. Peters

L. Pisano

R. Reynolds

R. Slade

J. Spadafore

R. Sylvia

C. Terry

T. Tomlinson

D. White

S. Wilczek, Jr.

/itin

, Unit 2

Unit 2

4

Manager, QA Operations, Unit 2

Manager, QA Operations, Unit 1

Technician, Meter and Test

Station Shift Supervisor, Unit 2

System Engineer

Unit 1 Plant Manager

Technician, Meter and Test

Operations Manager, Unit 2

General Supervisor, Electrical Maintenance

Station Shift Supervisor, Unit 2

President,

Niagara Mohawk Corporation

Assistant Station Shift Supervisor, Unit 2

Control Room Operator, Unit 2

Control Room Operator, Unit 2

Unit 2 Manager Outage/Wk Control

Supervisor, Electrical Design, Unit 2

Operations Planner

Technician, Meter and Test

Control Room Operator, Unit 2

Unit 2 Plant Manager

Supervisor, Meter and Test

Site Licensing Engineer

Technician, Meter and Test

Unit 1 Outage Manager

Control Room Operator, Unit 2

General Supervisor, Training Operations,

Program Director, ISEG

Executive V.'. - Nuclear

V. P. - Nuclear Engineering

Supervisor, Reactor Engineering, Unit 2

Assistant to the Plant Manager, Unit 2

V. P. - Nuclear Support

Appendix B

N

l

R

l

mmi

i n

R, Capra

C. Cowgill

M. Hodges

~ L. Nicholson

• W. Schmidt

Project Director, NRR

Branch Chief, RI

Director, DRS

Section Chief, DRP

Senior Resident Inspector

• Denotes those present at the exit meeting on April 1, 1992, attended by the public and

news media.

The team also held discussions with other licensee management,

operations,

. maintenance,

engineering and quality assurance

personnel.

~

~

APPENDIX

D

ENT

REVIEWED

1.

Chief Shift Operator and Station Shift Supervisor logs for March 23, 1992.

2.

Copies of written statements provided by personnel involved in the event.

3.

Computer alarm logs for 1008 hours0.0117 days <br />0.28 hours <br />0.00167 weeks <br />3.83544e-4 months <br /> - 1308 hours0.0151 days <br />0.363 hours <br />0.00216 weeks <br />4.97694e-4 months <br /> on 03/23/92

4.

Administrative Procedure AP-5.2.5, Rev.01, Work In Progress (WIP)

5.

Administrative Procedure AP-5.4, Rev.04, Conduct of Maintenance

6.

Administrative Procedure AP-5.4.2, Rev. 02, Troubleshooting,

7.

Administrative Procedure AP-5.5, Rev. 02, Work Control

8.

Administrative Procedure AP-5.5.1, Rev. 06, Work Request

9.

Nuclear Division Interfacing Procedure NIP-ECA-01, Rev. 03, Deviation Event Report

10,

Generation Administrative Procedure GAP-OPS-01, Rev. 00, Administration of

Operations

1 1,

Electrical Preventive Maintenance Procedure S-EPM-GEN-2Y070, Revision 1, dated

05/17/88; with Data Sheet (Attachment 10.3) for Equipment Piece No. 2 ABS-X1,

dated 03/23/92; Test Results Information Sheet for 2ABS-X1, dated 03/23/92; and

Work- In Progress

Data Sheet for 2 ABS-X1, dated 03/20/92.

12.

Work Control Monitoring Program Plan for Nine Mile Point Unit One and Unit Two,

.

Revision 0, March 17, 1992.

13.

Nine Mile Point Unit 2 Outage/Work Control Department Instruction Shutdown Safety,

Revision 00 (draft).

14.

Administrative Procedure AP-5.2.3, Revision 03 "Preventive Maintenance Program.

15.

Surveillance Reports 92-15000 dated 03/12/92 and 92-15001 dated 03/20/92.

16.

Surveillance Reports 92-23001 dated 03/12/92 and 92-23002 dated 03/20/92.

17.

Deviation/Event Reports 1-92-Q-0770, 1-92-Q-0771, 1-92-Q-0772, 1-92-Q-0786, 1-92-

Q-0787, 1-92-Q-0818, 1-92-Q-0819, 2-92-Q-0949, 2-92-Q-0950,

1-92-Q-0812, 2-92-Q-0802.

Appendix C

.

2

18.

Training Plans for March 9, 1992 Site Management Meeting for training on the Work

Control Process Monitoring Program.

19.

Work In Progress Data Sheets 2TMI-TE157, 2TMB-T1CIA, 2TMI-TE153, Remote

Shutdown 2CES~PNL405, 2FNR-CRNI, 2SFC-STRS, 2MSS~V1A, 2CND-IPNL287,

2TMI-SE133, 2HDL-LT22A, 2TMI-TE152, Remote Shutdown 2CES~PNL405.

20.

Work Requests

Nos. 190344, 199188, 200401; 201294, 201292, 201538, 200568,

199163, 184946, 177320, 195594, 200150, 200347, 198008, 200523, 200140, 193168,

177162, and 200182.

21.

Operating Procedure N2-OP-70, Rev. 2, Station Electrical Feed and 115KV

Switchyard

22.

Operating Procedure N2-OP-71, Rev. 3, 13.8KV/4160V/600V A.C. Distribution

System

23.

Operating Procedure N2-OP-72, Rev. 4, Standby and Emergency A.C. Distribution

System

P

24.

NMP-2 Training Response

to the NMP-2 Alert on March 23, 1992

25.

Alarm Response

Procedures

(ARP)

852426.Aux Boiler Transformer BU LKO Relay Trip

852434 Reserve Station Transformer 1A Loss of Voltage

852431 Motor Operated Circuit Switcher 2YUC-MDS5 Open

852533 Aux Boiler Transformer Loss of

Voltage'52453

Aux Boiler Transformer Backup Transfer Trip

26,

Drawing 12177-ESK-8YUC05, 115KV Transfer Trip 2nd Alternate, Sh.

1 and 2

27.

Drawing 12177-ESK-8YUC03, 115KV Ckt Switcher 2YUC-MDS5 Cont

28.

Drawing 12177-ESK-8YUC04, 115KV Transfer Trip 1st Alternate, Sh.

1 and 2

29.

~ Drawing 12177-ESK-8SPR14, XFMR 2ABS-X1 RLY

30.

Drawing 12177-ESK-8SPR10,

XFMR 2ABS-Xl Backup Prot

5

31.

Drawing 12177-ESK-8SPR12, XFMR 2ABS-X1 Pri Prot

32.

Drawing 12177-ESK-8SPR11,

XFMR 2ABS-Xl Fault Press Prot

33.

Drawing 12177-EE-1EA-10, 115KV Swyd One Line Diagram

~

~

4

Appendix C

3

34.

Drawing 12177-ESK-8NNS07, XFMR 2ABS-X1 4KV Winding Prot

35.

Drawing 12177-ESK-SNPS07,

BUS 2NPS-SWG002 Supply ACB2-5

'6.

Drawing 12177-ESK-SNNS16, BUS 2NNS-SWG018 Supply CB18-2

37.

Drawing LSK-24-8.2N, Normal Station Service (13.8 KV) Breaker Controls

38.

Drawing LSK-24-5.3A, Auxiliary Boiler Transfer Protection

39.

Drawing LSK-24-7.2C, 115KV Motor Operated Circuit-Switcher Control

40.

Drawing LSK-24-5.2C, '115KV Line Protection Transfer Trip

k

41.

Drawing LSK-24-8.60, 4.16KV Normal Station Service Breaker Control

42.

Nine Mile Point Nuclear Station Unit 2, One Line Diagram, 115KV Switchyard Offsite

Power Sources Backup Protection, Sheets

1 and 2, Provided on March 26, 1992

43.

Nine Mile Point Nuclear Station Unit 2, One Line Diagram, Present Arrangement

.

UPS1A & 1B Power Feeds to NSSS Annunciator Panel 2CEC-PNL630, Provided on

March 26, 1992

44.

Nine Mile Point Nuclear Station. Unit 2, One Line Diagram, Present Arrangement

UPS1A & 1B Power Feeds to BOP Annunciator Panels 2CEC-PNL858 & 2CEC-

PNL833, Provided on March 26, 1992

45.

Exide Electronic Drawing C1061373, Logic Supply and Relay Panel UPS MKII-A

FIGURE

1

NINE MILE POINT UNIT 2

SIMPLIFIED ELECTRICAL DISTRIBUTION

LINE 5

LINE 6

SCRIBA

SWITCHYARD

NMP-2

SITE

R-50

MDS10

MDS-20

R-60

. MDS-5

MDS-3

TRANSFORMER

MDS-4

RSST-A

TRANSFORMER

AUX BOILERS

RSST-B

TRANSFORMER

EDG

DIVISION I

EDG

DIVISION III

EDG

DIVISION II

=