Text

Insp Rept 50-220/92-80 on 920222-0304 Re 920221 Event. Major Areas Inspected:Inadvertent Isolation of Ultimate Heat Sink
	ML17056B732
Person / Time
Site:	Nine Mile Point
Issue date:	03/14/1992
From:	Eapen P; NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I)
To:	;
Shared Package
ML17056B731	List: ML17056B730; ML17056B732;
References
	50-220-92-80, NUDOCS 9203260315
	Download: ML17056B732 (62)
	v • d • e

U. S. NUCLEAR REGULATORY COMMISSION

REGION I

~ei~rt

g

QaakkMu.

Ei

N

50-220/92-80

50-220

DPR-63

~i~ngg

.~PN

N

In

tion At

QoOnl~ced

~ln pec~tr

'iagara

Mohawk Power Corporation

301 Plainfield Road

Syracuse, New York 13212

Nine MilePoint Nuclear Power Station, Unit 1

Scriba, New York

February 22 - March 4, 1992

G. Barber, Senior Resident Inspector, DRP

R. Bhatia, Reactor Engineer, DRS

D. Brinkman, Senior Project Manager, NRR

her

n ri

in

NR

Per onnel:

Qb

server'pproved

by:

W. Schmidt, Senior Resident Inspector, DRP

C. Beardslee,

Reactor Engineer Intern, DRS

=P. Eddy, State of New York

.E

Dr. P.. Eapen, Team

der,

Chief, Systems Section, DRS

Date

~l<<i: Pl

E

l

E

'

PDR "'>>>> 9-"orgy

ADOCK 0500022

PDR

TABLE OF

ONTENT

Page

EXECUTIVESUMMARY:.;;...;..;..........................;3

1.0

INTRODUCTION

1.1

The AITScope and Objectives

1.2

AITProcess

~

'

~"

~

~"

~

~ 4

~

~ 4

~

~ 5

2.0

ISOLATIONOF ULTIMATEHEAT SINKEVENT..................

5

2.1

2.2

2.3

Screen House Bay Gate Operation

Chronology ofEvents......

~ . ~............

Highlights of the Event....................

~

'

~

~ 5

~

~ 6

~

~ 6

3.0

PERSONNEL AND NUCLEAR PLANT SYSTEMS PERFORMANCE ..., .. 9

3.1

3.2

3.3

3.4

Equipment Performance

Procedure Adequacy

Personnel Performance ..

Management

Assessment

~

~ .. 10

.. 10

~

.

13

4.0

GENERIC IMPLICATIONSOF THIS EVENT

5.0

THE LICENSEE'S IMMEDIATEACTIONS

~

14

~

~ 'e

~

1 5

6.0

CONCLUSIONS ~.................,..............,.....,

15

7.0

MANAGEMENTMEETINGS...............................

16

TABLE 1 - Chronology of Events

I

FIGURE 1 - Screen House Bay Level Sketch

FIGURE 2 - Normal Flow Configuration of the Screen House Bay

FIGURE 3 - Reverse Flow Configuration of the Screen House Bay

APPENDIX A - NRC Augmented Inspection Team Charter

APPENDIX B - Persons

Contacted

APPENDIX C - Documents Reviewed

I

EXECUTIVE UMMARY

On February 21, 1992, at about 8:30 a.m., licensee personnel inadvertently isolated Nine-

Mile Point Nuclear Power Plant Unit 1 from Lake Ontario, the unit's ultimate heat sink, by

closing all gates that let the water from the lake to the plant's service water bay.

One

service water pump and two circulating water pumps were running at the time of this event.

The water level in the bay rapidly decreased

below the level assumed in the Unit 1 licensing

basis.

The bay level was reduced for about six minutes before the operators opened B gate

restoring water to the bay from the lake.

The running service water pump was secured by

the operators

as it lost suction and.cavitated.

The emergency service water pump was

started as required by procedures,

and this pump had to be secured immediately due to low

discharge pressure.

The only noticeable change in plant conditions due to the brief isolation

of the ultimate heat sink was a two degree Fahrenheit increase in the reactor building closed

loop cooling system.

The reactor had been shutdown since February 16, 1992, and the

reactor system was depressurized

with reactor coolant temperature

at about 143 degrees

Fahrenheit at the time of this event.

An augmented inspection team (AIT) was dispatched by the NRC to determine the

circumstances

that led to this event, its causes,

safety significance and generic implications,

and the adequacy of the licensee's

actions before, during, and after the event.

The AIT

began its assessments

on Saturday, February 22, 1992, and completed its onsite reviews on

February 28, 1992. The AITpresented

its preliminary findings in a public exit meeting on

March 4, 1992.

The AIT concluded that the causes for this event were:

(1) Failure to follow the established

work control process by various levels of personnel in multiple licensee

groups; (2)

Inadequate

management

over view to assure that the workers understood

and followed

established procedures;

(3) Inadequate communications within and among organizations

participating in work activities; and (4) Failure to adequately consider risks associated

with

test activities that affected multiple systems during shutdown conditions.

The consequences

of this event were minimal because

the reactor core and,the reactor

coolant were unaffected by this event, no equipment was damaged,

and no radiation was

released.

The operator response

to the decrease

in level in the intake bay was good.

However, the work control breakdown that led to this event was significant because it caused

the unit to isolate from its ultimate heat sink.

The screen

house bay level dropped below that

required for the safe operation. of the pumps and caused

the Unit to be in an unanalyzed

condition.

The adequacy of the instrumentation in the bay and the licensee's failure to-consider risk

associated

with test activities that affected multiple systems during shutdown conditions were

identified as items that may have potential generic implications.

l.~

Upon being informed of the inadvertent isolation of,Nine Mile Unit 1 from Lake

Ontario on February 21; 1992, the NRC Region I Regional Administrator and senior,

management from the Office of Nuclear Reactor Regulation (NRR) and the Office for

Analysis and Evaluation of Operational Data (AEOD) determined that an Augmented

Inspection Team (AIT) should be formed to review and evaluate the circumstances

and significance of this occurrence.

The basis of the NRC concern was the apparent

inadequacies of management controls of maintenance activities that allowed the event

'o occur.

Accordingly, an AIT was selected, briefed, and the AIT leader dispatched

to the site on February 22, 1992.

1.1

The AIT

c

and Ob'ective

The charter for the AIT (Appendix A) was issued on February 24, 1992.

The

charter directed the team to conduct an inspection and accomplish the

following objectives:

Conduct a timely, thorough and systematic review of the circumstances

surrounding the event, including the sequence of events that led to and

- followed the isolation of the ultimate heat sink;

2.

Collect, analyze and document relevant data and factual information to

determine the causes,

conditions and circumstances

pertaining to the

event including the response of the licensee's staff to the event;

3.

Assess the safety significance of the event and communicate to

Regional and Headquarters

management

the facts and safety concerns

related to the problems identified; and

4.

Evaluate the licensee's review of and response

to the event and

implemented corrective actions,

II

During the period between February 22, 1992 and March 4, 1992, the AIT

conducted

an, independent inspection, review, and evaluation of the conditions

and circumstances

associated

with this event.

The team inspected the gate's

and the pumps in the screen house bay and related indications in the control

room;

held discussions

and formal interviews with personnel involved in this

event; reviewed relevant records including computer printouts before, during,

and after the event, and trends of pertinent plant parameters;

and evaluated the

adequacy of established

procedures,

management

oversight, and personnel

training.

Attachment B is the list of personnel contacted by the AIT and

-Appendix C is the list of documents reviewed by the AIT members.

This inspection was conducted in accordance with NRC Manual Chapter 0513,

Part III, Inspection Procedure 93800, and additional instructions provided in

the AITcharter.

2.0

I

LATI N

F

TIMATEHEAT INK EVENT

2.1

creen H use Ba

ate

rati n

Lake Ontario provides the source of water to Nine Mile Unit 1 to cool the

reactor during'normal operation and accident conditions.

One of the five gates

in the unit's screen

house lets water from the lake to the screen house bay

where 19 pumps take suction for safety and non-safety related purposes.

Figure

1 shows the normal and minimum required bay levels as well as the

elevations where various pumps take suction from the screen

house bay.

The flow of water from the lake to the screen house bay during normal screen

bay gate configuration is depicted in Figure 2.

Water is drawn from the lake

at a location about one thousand feet away from the shore.

The water'nters

the bay through the intake tunnel and gates A and B. The water from the bay

is returned through gate C and the discharge tunnel to a location in the lake

about five hundred feet away from the shore.

A specially designed intake

structure removes debris and other floating bodies from the incoming water.

During winter months, the intake structure has a potential to experience ice

build up and block the flow area.

To prevent such ice build up,

the operators

reverse the flow of water using procedure N1-OP-19.

-Figure 3 depicts the

screen bay gate configuration during reverse flow operation.

In this

configuration water is drawn from the lake through the discharge tunnel and

flows to the bay through gate D, mixed with hot water from the condenser,

and returned to the lake through gate E to the intake tunnel and the intake

structure.

The water that is returned to the lake is warmer and removes ice

build up on the intake structure.

The licensee also refers to this operation as

de-icing operation of the intake structure.

The gates are massive concrete blocks that are typically 6 feet by 9 feet and

weigh about 5,400 pounds.

Electric motors raise and lower gates A

through D.

Gate E is hydraulically controlled.

The controls (push buttons)

for the gates are located in the screen house.

2.2 ghhlf

4

The AITcompiled independently a detailed chronology of events by

interviewing cognizant personnel, reviewing relevant records including

computer printouts before, during, and after the event, and trends of pertinent

plant parameters.

This detailed chronology is provided in Table l.

2.3

Hi hli hts f the Even

The push button that opens the D gate was inoperable for a long time and the

licensee personnel used a piece of a wooden ruler to make the electrical

contact for the motor that operates

the gate.

A work order 163740 was is'sued

on February

10, 1992, to fix this push button circuitry.

1

The implementation of this work request was not fully in accordance with the

licensee's

established

procedures.

For example, maintenance personnel did not

adequately detail the required work and operations personnel did not specify

the required procedures for post maintenance

test.

The electrical circuitry for gate D was fixed adequately on February

10, 1992.

However, during this work, the licensee personnel identified an undocumented

electrical jumper (wire) that bypassed

the mechanical tension overload

protection switch from the drive motor circuit.

On February 11, 1992, a deficiency event report (DER) 1-92-0267 was

generated

to resolve this concern,

as required by the licensee procedures.

However, this DER.was not processed

in a timely manner, and an adequate

operability determination was not made by the operations personnel

as required

.

by the licensee's procedures.

A temporary modification (5395) was initiated

on February

11, 1992, to obtain an engineering resolution for this

undocumented jumper.

The undocumented jumper, the DER, and the

temporary modification were not documented in the work request,

as required

by the licensee's procedure.

Consequently, control room operators were

deprived of this critical information when they reviewed the work request.

This work request was not closed, again as required by the procedure,

when

the original work was completed on February 10, 1992.

On February

11, 1992,

the original work request was revised to restore the

wiring in gate D circuitry to the original design by adding the statement

"Return wiring to normal configuration per dwg C 22303 C, sh. 3,'-'he

personnel'involved in the work understood that the change was to remove the

undocumented jumper from gate D circuitry. However, the brevity, lack of

specificity, and details made this addition to appear no more than a routine

system restoration after fixing the push button circuitry on gate D. Additional

instances of failure to follow licensee procedures

occurred during'the

processing of this change.

For example, the-maintenance

general supervisor

did not review and approve this change; the senior reactor operator did not

review this change adequately for plant impact; the shift supervisor did not

review the change for post maintenance

test requirements;

and no detailed

work instructions or troubleshooting steps were specified.

During early afternoon on February 12, 1992, the jumper was removed

without waiting for the resolution of the DER issued on February

11, 1992, as

required by the licensee's procedures.

-Electrical maintenance

reported the

'completion of the revised work to the station shift supervisor and informed

him that the ability of this gate to close during reverse flow operation cannot

be assured with the mechanical overload switch in the circuit.

The station shift supervisor ordered immediately the reinstallation of the

jumper through the emergency temporary modification process,

and a new

DER (1-92-0286) 'was issued to document and evaluate this action.

All

departments

involved agreed that operation of the gate D could only be tested

~

safely during shutdown conditions and decided to postpone the work on gate D

until the next scheduled unit shutdown in fall 1992.

None of these decisions

or the removal/reinstallation of the jumpers were documented in the work

request.

8

The unit was automatically shutdown on February

16, 1992, due to an

unrelated problem with the turbine stop valves. After this unplanned shutdown,

engineering and maintenance personnel discussed

the possibility of gate D

work in the morning outage work planning meetings.

However, due to the

objections from operations,

this work was not scheduled.

On February 21, 1992, the reactor remained shutdown, depressurized

with the

reactor coolant temperature of about 143'F.

The gate D work was again

discussed at the 6:45 a.m. outage work planning meeting.

The operations

personnel initially objected to this work. However, the operations supervisor

agreed to discuss testing of gate D further with maintenance.

This action

caused inadequate communications between maintenance

and operations and

led to the bypass of the normal work control review process.

At about 7 a.m.,

the station shift supervisor (SSS) agreed to put screen house bay in reverse

flow configuration to allow testing of the gate under differential pressure.

A

specially marked-up electrical drawing (referred to as blue markup), which

was previously approved by another station shift supervisor,

was issued to

electrical maintenance who began the gate D work at about 8:20 a.m.

Electrical maintenance personnel removed the undocumented jumper from the

motor circuitry. The operations personnel cycled the gate 2-3 feet in the

downward direction successfully and then closed the gate fully. An attempt to

reopen gate D was unsuccessful.

This caused the inadvertent isolation of the.,

Unit from the lake at 8:29 a.m.

The isolation occurred while two circulating water pumps (125,000 gpm each)

and one service water pump (20,000 gpm) were removing water from the bay.

As a result the water level in the bay rapidly decreased.

At 8:29 a.m. the control room received the tunnel high differential alarm and

then the screen bay low level alarm, and the control room ordered the opening

of gate D immediately.

A non-licensed operator held the "UP" button closed,

while an electrician held a jumper across the tension overload switch and

commenced

the opening of D gate.

Additionally, the control room ordered

a

licensed operator to open B gate also immediately.

Between 8:30 and

8:35 a.m., both B and D gates were being opened.

gt takes about five

minutes to fully open the gates).

At 8:32 a.m. the running service water pump

cavitated, and it was immediately secured by the operators.

The operators also

attempted to start emergency service water pump No. 11, as required by the

licensee's procedure (SOP-7, Loss of Service Water).

However, this pump

was immediately secured

due to low discharge pressure resulting from low bay

level.

The operators also secured circulating water pump No. 11 to reduce

water removal rate from the bay.

At 8:35 a.m. the bay level was returned to normal.

The licensee started both

emergency service water pumps successfully at 8:38 a.m., and at 8:44 a.m.,

service water pump No. 11 was started and returned to service successfully.

Subsequently,

at 8:45 a.m., both emergency service water pumps were secured

and the bay and other equipment were restored to shutdown operation

configurations.

During this event, the level decreased

from normal 243'0" to 229'" or

about nine feet below the minimum level 238'" for safe operation for about

six minutes.

The control room has an alarm for the bay level.

However, the

set point and the location of this instrumentation in the bay are such that the

alarm is annunciated

when the level is 18" below the required level of238'"

for safe pump operation.

The unit's safety analysis requires a minimum water

level of 238'" in the screen house bay.

Therefore, this event caused the

facility to be in an unanalyzed condition.

The only observed system change was a two degree fahrenheit increase in the

temperature of the reactor building closed loop cooling system.

No

radioactivity was released

and no equipment or structural damage was

observed.

At 9:30 a.m. the license'e conducted a detailed debrief with all personnel

involved in the event.

A stop work order was issued for Unit 1 at 10:30 a.m.

DER 1-92-Q0390 was initiated immediately to document and resolve this loss

of ultimate heat sink event.

The licensee notified the senior resident inspector and the NRC operations

center at 11:30 a.m. and 12:25 p.m., respectively.

Subsequently,

the licensee verified that the non-safety related service water

pump and the safety-related emergency service water system pump were

operable by performing detailed surveillance tests.

3.0

PER ONNEL AND

CLEAR PLANT Y TEM

PERF

RMAN E

The AITassessed

the performance of the personnel

and the plant systems before,

during, and after the event.

The findings of the AITare grouped into three broad

categories:

Equipment Performance;

Procedure Adequacy; and Personnel

Performance.

I

10

3.1

i m

P rf rmance

The equipment performed as expected with the exception of the gate D and the

bay level alarm.

The licensee personnel expected that the gate D would open

from full closure position in spite of the mechanical tension overload switch in

the gate circuitry while the bay is in reverse flow operation.

However, this

switch inhibited the motor from opening the gate as intended.

The bay level

alarm in the control room that annunciated only after the level decreased

18"

below that was required for safe pump operation.

Service water pump No. 11-

and the emergency water pump No. 11 that were affected by the event were

successfully verified operable by the licensee using appropriate surveillance

procedures.

No equipment or structural damage occurred as a result of this

event.

3.2

Procedur

Ad

uac

In general, the licensee's

procedures

were adequate,

and they did not

contribute to this event.

However, procedure (SOP-7) required the operators

to turn on the safety-related emergency service water pumps when the bay

level was below'he suction level for these pumps.

The licensee acknowledged

this concern and agreed to review this matter for resolution prior to the start

up of unit 1 ~

3,3

Pers

nnel P rf rmance

The licensed operators'esponse

to the decreased

level in the intake bay was

good.

However, prior to the event, the performance of various levels of

personnel in multiple licensee organizations, including operators,

during the

preparation, implementation, and documentation of work request 163740 was

inadequate.

The following instances of failure to follow established

procedures by licensee

personnel exemplify a breakdown in the work control process:

(1)

-

Maintenance personnel did not adequately describe the work on the

initial work request issued on February

10, 1992, as required by

licensee administrative procedure AP-5.5, steps 5.1 and 5.2 (See

Appendix C for complete title of documents).

(2)

Operations personnel did not specify the surveillance procedures

to be

used for post maintenance

test, as required by licensee procedure AP-

5.2.4.

11

(3)

Maintenance workers did not record the undocumented

electrical

jumper identified during the conduct of work in the work request, or in

the DER issued to resolve it, as required by the trouble shooting

procedure AP-5.4.2, Section 2.

(4)

The personnel did not close the work request when the work was

completed on February

10, 1992, as required by licensee procedure

AP-5.5.1, step 5.17.

(5)

The change to the work request that authorized the removal of the

undocumented jumper was not reviewed and approved by the initiating

department general supervisor,

as required by the licensee's work

request procedure AP-5.5.1, step 4.10.

(6)

The work request change was not reviewed by the SSS on

February 11, 1992, for post maintenance

tests (PMT) requirements,

as

required by procedure in AP-5.5.1, step 4.3.

(7)

The work request was not adequately reviewed by the senior station

'perator (SSO) for plant impact as required by the work request

procedure AP-5.5.1, step 4.4.

(8)

No troubleshooting procedure was prepared for the test of the gate D

on February 21, 1992, as required by the work request procedure AP-

5.5.1, step 3.19 and troubleshooting procedure AP-5.4.2.

(9)

No post maintenance

test was specified by maintenance for the gate D

work on February 21, 1992, as required by AP-5.5.1, step 4.3.

(10)

The initial DER was processed

inadequately and an operability

determination was not made,

as required by procedure NIP-ECA-01,

section 6.1.2.

(11)

The removal of the undocumented jumper was not recorded in the work

request,

as required by the work request procedure AP-5.5.1.

(12)

The undocumented jumper was removed without authorization from

engineering, contrary to licensee procedure AP-6.1.

12

(13)

- Communications between operations and maintenance were very

informal and incomplete.

Maintenance and engineering did not

adequately explain the details of the work. involved to operations.

This

was contrary to the licensee's work in progress procedure AP-5.2.5,

step 5.2.2, which requires the department supervisor/chief to conduct a

pre-work brief with appropriate personnel.

(14)

The maintenance personnel bypassed

the work control center and went

straight to the SSS on duty, contrary to procedure.

Due to inadequate

documentation of prior work on the work request,

inadequate

communications,

and a lack of a questioning attitude, the SSS

authorized work without fully being aware of the plant impact of the

requested work. This SSS authorization is required in work-in-progress

procedure AP-5.2.5, step 4.1.

(15)

The licensee work request procedure AP-5.5.1, step 4.3 requires the

SSS to perform technical reviews on work requests

when work control

personnel do not perform the review.

The SSS did not perform an

adequate technical review for this work request.

(16)

Step 4.3 of the licensee's work in process procedure AP-5.2.5 requires

the Assistant Station Shift Supervisor (ASSS) to ensure that the post

maintenance

testing requirements

are appropriate to verify 'equipment

function.

Since the maintenance personnel bypassed

the process, it

appeared

that this ASSS function was not executed for this work

request.

(17)

Communications within operations were inadequate.

The chief shift

station operator (CSO) was aware of the closing of gate D and some of

the potential impacts of closing the gate during reverse flow.

However, he failed to communicate that to shift management.

The

ASSS was aware of operation's objections to testing gate D, but he did

not communicate that to the SSS who approved the test.

(18)

Various levels of personnel and supervision in the maintenance

organization did not fully understand

the work request process

and the

significance of various signatures on work request form.

13

3.4

Mana emen

A

men

The AITreviewed management involvement and control for the activities

related to the event.

The team assessed

the adequacy of the implementation of

the licensee's

management

expectation for the work control process.

The team

also reviewed the self assessments

and personnel training in this area.

The AITobserved that the licensee management

expectations regarding the

strict adherence

to established

procedures were not effectively communicated,

monitored, or controlled by supervisors and management during work request

process implementation.

The existence of effective management

and

supervisory assessments

to verify adequate implementation of established

procedures

and constructive feed back to enhance the process were not

evident.

The team observed inadequate

management involvement in assuring the

direction of licensed reactor operator activities.

While the licensee's

procedure GAP-OPS-01

states that the general supervisor of operation is

responsible for the operating shift activities through the SSS, the team

observed that this direction was primarily provided by the CSO and not by the

SSS or the general supervisor of operation.

The team concluded that the lack

of emphasis of the above procedural requirement was a weakness

in the

management of operational activities.

While the licensee conducted special training for questioning attitudes and the

new work philosophy, training appeared

to be lacking to indoctrinate workers

in the bases of procedures

and the role of established

procedures in assuring

operation of the unit within its licensing and regulatory bases.

The licensee's quality assurance

department surveillances identified repeated

instances of failure to follow procedures

and inadequate work requests.

Based

on the team's observations, it appeared

that the licensee's

response

was not

effective in correcting the root causes for these failures.

14

4.0

ENERIC IMPLI ATI N

OF THIS EVENT

The AITreviewed this event for generic implications and identified two items that

- have potential generic implications:

(1) The adequacy of the instrumentation in the

bay; and (2) The licensee's failure to adequately consider risks associated

with the test

activities that affected multiple systems during shutdown conditions.

The team noted that the instrumentation in the screen house bay is neither safety

related nor designed to assure a high level of reliability. However, operators

use the

information from this instrumentation to make decisions regarding the screen house

bay level and the availability of the ultimate heat sink.

Additionally, the location of

the bay level instrumentation was such that it alarmed the control room only after the

bay level decreased

below that is required to ensure adequate

suction for the pumps.

The team observed that the decision to remove the jumper, and the testing necessary

to determine the effects of this action were not adequately reviewed and approved by

the licensee's

operations,

maintenance,

and engineering organizations.

When

maintenance personnel discovered the jumper, they were concerned

because it was not

indicated on the design drawing.

Therefore,

they believed that the jumper was not

part of the original design or was added by a subsequent plant modification. Their

initial inclination was to remove the jumper, however, they knew that this gate

operated successfully for years with the jumper installed.

The engineering

department was contacted,

and a testing strategy was developed.

However, this

strategy did not include the use of an integrated test procedure,

and therefore, did not

receive multi-departmental review.

The licensee's

focus to restore the gate's

electrical configuration per the design was proper.

However, their failure to verify

the design by using an integrated test procedure was a definite weakness.

The team noted that a test could have been developed for shutdown conditions that

included risk reducing compensatory

measures,

such as, (1) using the screen house

overhead crane to augment the D gate lift'withthe jumper removed,

(2) establishing

minimum decay heat levels, (3) establishing minimum cooling water pump

alignment, (4) establishing additional fore-bay level monitoring with operators

stationed. to communicate level changes,

and (5) having a pre-established

course of

action, ifcompensatory

measures

were less than fully successful.

The failure to

generate

an integrated test procedure resulted in a failure to properly consider

shutdown risks.

I'

5.0

THE LI EN EE'S IMMEDIATEA TI N

Immediately following the event, the licensee implemented the following actions:

1.

The acting plant manager issued a stop work order that essentially stopped all

activities, except for the surveillance and other actions required by the unit's

license.

2.

A review of all work requests

was ordered.

3.

.An assessment

organization was formed to investigate the circumstances

leading to this event, plant response

and personnel action before, during, and

after the transient, and the root cause of the event.

The team found the licensee's review of and response

to the event and implemented

corrected actions acceptable.

6.0

NL SINS

The AIT concluded that the causes of the inadvertent isolation of Lake Ontario from

Nine Mile Unit 1 were:

1 ~

Failure by various levels of personnel in multiple licensee groups to follow the

established work control process,

2.

Inadequate

management

oversight to assure that the workers understood

and

followed established procedures,

3.

Inadequate communications within and among organizations participating in

work activities, and

4.

Insensitivity to shutdown risk among multiple licensee organizations.

The consequences

of this event were minimal because

the reactor core and the reactor

coolant were unaffected by this event, no equipment or structural damage,

and no

radiation was released.

The operator response

to the decreased

level in the intake bay was good.

The AIT

concluded that the licensee's procedures

were adequate

and did not contribute to the

event.

However, the work control breakdown that led to this event was significant

because it caused the unit to isolate from its ultimate heat sink causing the screen

house bay level to drop below that was required in the Unit's licensing bases.

C

16

7.0

MANA EMENT MEETIN

The licensee management

was informed of the scope of this AITduring an entrance

meeting at 2:00 pm on Sunday, February 23, 1992.

The licensee management

was,

briefed of the inspection observations routinely and at 10:00 a.m. on Friday,

February 28, 1992.

A public exit meeting was conducted on March 4, 1992 at 10:00 a.m., at the

licensee's training facilities with licensee representatives

identified in Appendix C to

discuss the preliminary inspection findings.

The licensee acknowledged

the inspection

findings and provided the results of their assessment

of the event and the short and-

long term corrective actions for both units.

TABLE 1

HR N L

Y FE

2/10/92

Work Request (WR) 163740 completed and Raise Push Button Circuit on the gate D was

fixed.

While WR 163740, consisting of troubleshooting, was being performed, undocumented

temporary modifications were discovered.

2/11/92

Deviation event report (DER) 1-92-Q-0267 was initiated by electrical maintenance

to identify

that the mechanical tension overload switch was bypassed contrary to the design drawing.

WR 163740 was revised to require the removal of the tension overload bypass on Gate D

circuitry

2/12/92

Revised WR 163740 completed.

Tension overload switch was put back in the gate D motor

circuitry. Limited travel test completed.

Reported to Station Shift Supervisor (SSS) that the jumper was removed.

The SSS orders the reinstallation of the bypass using the emergency temporary modification

process.

( Maintenance was not sure that the D gate willopen without the bypass during a

de-icing operation.)

DER 1-92-Q0286 was issued to address reinstallation of the bypass per procedure.

2/21/92

Ini ial Plan

nditi n

The plant was in cold shutdown with reactor temperature at 143 degrees F, pressure at

0 PSIG.

Major equipment in operation:

Service water pump No. 11

Circulating water pumps No. 11 & No. 12

Shutdown cooling loop No. 11

Reactor building closed loop cooling (2 loops)

Turbine building closed loop cooling (1 loop)

Table

1

Allother equipment was in a normal shutdown line-up for the existing plant conditions.

0645

Operations personnel

stated that they'ay not be able'to support electrical work on .

- gate D.

Operations management

agreed to review the matter

0700

Electrical Maintenance personnel entered the control room to discuss gate D

maintenance with the SSS.

0730

Intake flow to reverse flow configuration per electrical maintenance

request.

0751

Blue mark-up 1-92-50111 issued to electrical maintenance on screen house gates C &

D.

. 0820

Operations working with electrical maintenance on gate D work request for

troubleshooting.

0820- Electrical maintenance personnel removed the undocumented jumper from the gate.

'829

control circuitry. The electrical maintenance

and operations personnel involved in the

testing of the gate D first cycled the gate 2-3'n the downward direction.

The result

of the test was satisfactory.

Then the gate D was fully closed.

An attempt to reopen

gate D was successful.

0829

Tunnel high differential pressure alarm.

0830

Notified by "C" operator that gate D is closed.

- Circulating water intake level low alarm H2-1-3.

- No, 12 circulating pump removed from service.

- "C" operator instructed electrical maintenance to reinstall jumper.

0830- A non-licensed operator held the up button closed while an electrician held a jumper

0835

across

the tension. over load switch and commenced opening of D gate.

In-plant "E" operator opened gate B to restore level.

0832

Service water header'pressure

low alarm H1-4-2.

- service water pump No. 11 removed from service..

CSO attempted to start the emergency service water pump No. 11.

No discharge

pressure

was observed in the Control Room, the pump was immediately shut down.

Table

1

0833

Fire header pressure low H2-2-8 clear.

- electrical fire pump on.

0835

Screen house intake level normal, alarm H2-1-3 clear.

0838

Emergency service water pumps Nos.

11 & 12 on.

- reactor building service water header pressure normal.

0840

Vented No. 11 service water pump.

0844

Service water pump No. 11 on.

- turbine building service water header pressure normal.

0845

11 & 12 emergency, service water pumps off.

0850

Electric fire pump off.

0900

Water intake flow returned to normal..

- Breathing air compressor

restarted (tripped on low service water pressure).

- Fish screen closed and drain valve opened on No. 12 circulating water pump.

- No. 12 water box vents opened.

0902

Attempted start of service water pump No. 12 following venting, high starting current

observed (prolonged); service water pump No. 12 in Pull-to-Lock.

0930

The licensee conducts a debrief with all personnel involved with the event

0935

Yellow hold-out placed on No. 12 Service Water pump.

1130

The licensee informs the NRC Senior Resident Inspector about the event

1225

The licensee notifies the NRC operations center as required by 10 CFR 50.72.

Later licensee assessed

Unit 2 for similar problems.

250

DIESEL

EMERGENCY

CONTAIN.

DIESEL

SERVICE

GENERATOR

SERVICE

SPRAY

FIRE MATER

ELECTRIC

WATER

COOLING

MATER

RAW

PUMP

FIRE

PUMP

PUMPS

WATER PUMP

PUMPS

WATER

CIRCULATING

MATER

PUMPS

240

230

220

I

s

I

s

is l

~

s

~

s

I I

'

I

~

j

I

P

s

~

i

~

I I; I.

Q~ Ie-

s

I i:.Ili:I

~

I'I

I

1

I

s

!i

s

~

I

s

I

s

~

s

I

j

s

~

I

s

~

iI!li

I

s

~

I

s

'

s

~

s

S

s

~

s

~

s

~

i I'I

I

s

~

s

I

I':s(

~

~ -- -i

~

I

s

~

) I:

s

I

~

i---LI

s

I

I i

l

s

~

I

I j

.

I

'

i

s

ljj

I

s

~

I

s

.232

~

I

s

s j '

i

s

I

l

I

'ssl'.I

s

~

I

I..l si.

~

s

s I I

s

I

~

I li il

~ i:II!I

~

2QQj'

I III ( j

I

I:Is

~: i!

}

s

I

.

sI

I

s

l.i.'

i! I I

.'I i

I

~ '

s

~

.'2N'!

~

s

Is,

~

s

r~

I

~

s

~

s

I

~

s

~

s

I

~

s

I

S

s

~

s

~

s

I

I i

~

s

I

s

I I!

I

l i '

'.

s .',

I

s

i

I',II,'i

s

~

s

~

I

I;!!jI

221'-9'1I

245'-3II

Lake High Level

ay of Event

L'evel 243'-10"

Low Lake Level

238'-6"

Plant Analyzed Level

229'-6" Estimated

Level Drop Down

This Event

Level Alarm 220'-6"

21'.

il

!

I

< css

Figure

1

Suction Levels of Pumps in the Screen

House

Bay

DISCHARGE

~ -.TUNNEL

INTAKE

TUNNEL

I

I'

L

UJ

CD

CC

CD

C)

Cc'J

I

r

1

I

r

iC

I

Ir

I

~

I

~ 1

Lf-8

FROM

D .i'

I

P RIMARY

FORE BAY

C1I

W

I

o

C)O

I

SECOHDARY

FOREBAY

'r

~"

I

r'

r ~

I

'i

r,

FOR tIORMAL OPERATION:

I

GATES A, 8 & C

~ OPEN

I

GATES 0 & E

~ CLOSED

TO TEMPER WATER THAT

IS T00 COLD, PARTIALLY

OPEN GATE

E"

OUTLINE OF

SCREEN

& PUMP HOUSE

I

1r

~ r

~r1,

FIRE PUMPS

SERYICE WATER PUMPS

CIRCULATING WATER PUMPS

COHDEHSERS

TO COHDEHSERS

Figure

2

- Normal

Flow Configuration of the Screen

House

Bay

REVERSED PLOW IN

SPECIAL OPERATIONS

IH TAKE AND-

-> DISCHARGE<

> TUNNELS

I

~ I' rl'1

I

C

I

~rQ

I

II

I

,li

. I

ti

I

A

i

',1

~ q

I

~

i..:

D

IIt

'I

I

'irl

~ 'i

t\\ ~

W

( O~M

Ca~ W

p

,.I

I

.r.

t

>'1

~ ~

irt

I'

I

.I

lt

//.

~ I

F

1

t'li

\\

!.

I 'I

!

W

I

W

1

~ W

OC

X

Q

CDlrl

CD o

O

cc

pll'i

rtt

.Ii>'tgii.

\\

t

.I

/, 1

~ 1

1

I

STOP LOGS

'I<

i4)g

Sr

'Itri

I~

l,~

i"(

.;1<,

O< o

lrl

O o

O

I

0

CD

o W

I

,u3hEe(

';!.

,t'I~,

t

~

rt

1"

, ~

,

PART PLAN AT UPPER LEVEL

REVERSEO FLOW OPERATION:

GATES, A, 8 & C CLOSED

GATES 0 & E OPEN

4 SLUICE

I-

GATE -.. ';,:

4c

t '

tt

tlr I

'r

tt,

~ I

I

ONE CIRCULATING WATER PUMP

I

WELL & OHE SERVICE WATER

PUMP WELL SMOWH UNWATERED

FOR

P UMP MAIN7 EN AH CE

IJ

FROM AHD TO COHDEHSERS

Figure

3

- Reverse

Flow Configuration of the Screen

House

Bay

gpss

AEONS,

(4

Mp

a

0

Cy

I

n0

'I

!..

C

Vl

0

Y/

+>>*++

APPENDIX A

UNITED STATES

NUCLEAR REGULATORY COMMISSION

REGION I

476 ALLENDALEROAD

KING OF PRUSSIA, PENNSYLVANIA19406

February

24,

1992

MEMORANDUMFOR:

Marvin W. Hodges, Director, Division of Reactor Safety

Charles W. Hehl, Director,-Division of Reactor Projects

FROM:

Thomas T. Martin

Regional Administrator

SUBJECT:

AUGMENTED INSPECTION TEAM (AIT) CHARTER-

INADVERTENTISOLATIONOF NINE MILEPOINT

STATION, UNIT 1 ULTIMATEHEAT SINK

You are directed to perform an Augmented Inspection Team (AIT) review of the causes,

safety implications, and associated

licensee actions which led to the inadvertent isolation of

the ultimate heat sink at Nine Mile Point Station, Unit 1 on February 21, 1992.

The basis of

the NRC concern is the apparent inadequacies of management

controls of maintenance

activities that allowed the event to occur. The inspection shall be conducted in accordance

with NRC Manual Chapter 0513, Part III, Inspection Procedure 93800, Regional

Instruction 1010.1 and additional instructions in this memorandum.

DRS is assigned responsibility for the overall conduct of this inspection.

DRP is assigned

responsibility for resident inspector and clerical support and coordination with other NRC

offices.

Dr. P. K. Eapen is designated

as the onsite Team Leader.

Team composition is

described at the end of this memorandum.

Team members will work for Dr. Eapen and are

assigned

to this task until the report is completed.

f!K C

El

The general objectives of this AIT are to:

a.

Conduct a timely, thorough, and systematic review of the circumstances

surrounding

the event, including the sequence of events that led to and followed the

February 21, 1992 isolation of the ultimate heat sink;

b.

'ollect, analyze, and document relevant data and factual info'rmation to determine the

causes,

conditions, and circumstances

pertaining to the event, including the response

to the event by the licensee's

operating staff;

Marvin W. Hodges

Charles W. Hehl

C.

Assess the safety significance of the event and communicate to Regional and

Headquarters

management

the facts and safety concerns related to the problems

identified; and

d.

Evaluate the licensee's review of and response

to the event and implemented

corrective actions.

S

OPE

FTHEIN PE TI N

The AIT should identify,and document the relevant facts and determine the probable causes

of the event. It should. also critically examine the licensee's

response to the event.

The

Team Leader shall develop and implement a specific, detailed inspection plan.

The AIT should:

a.

Develop a detailed chronology of the event;

b.

Determine the root causes of the event as a result of the AIT's evaluation and

document equipment problems, failures, and/or personnel errors which directly or

indirectly contributed to the event.

Potential items to be considered:

Licensee staff actions before, during and following the event.

Configuration controls; including previous modifications and event related

system alignments.

Management oversight and administrative controls in place before, during and

~ following the event.

~

Coordination of maintenance

and operations activities before and during the

event.

~

Adequacy and implementation of troubleshooting procedures.

~

'Licensee staff sensitivity to plant conditions.

Schedular impacts due to the then pending restart of the unit.

c.

Determine the expected

response of the plant and compare it to the actual response,

due to the isolation of the ultimate heat sink.

Marvin W-. Hodges

Charles W. Hehl

d.'.

Determine the adequacy of the responses of the operations and technical support staffs

to the event and the initial licensee analysis,

and decisions on NRC notification

including event classification and reportability.

I

Determine the management

response including the scope and quality of short-term

actions and gather information related to the long-term actions intended to prevent

recurrence of this event, including internal and external communications/dissemination

of licensee-identified concerns

and corrective actions.

Determine the relationship of previous events or precursors, ifany, to this event.

Determine the potential generic implications of this event and recommend

lessons

learned, necessity for generic industry communications,

etc.

5CH¹EE

The AIT shall be dispatched to Nine Mile Point Station Unit 1 so as to arrive and commence

the inspection on February 23, 1992.

A written report on this inspection shall be provided to

me within three weeks of completion of the onsite inspection.

TEAM

OMPOSITION

The assigned Team members are as follows:

Team Manager:

Onsite Team Leader:

Onsite Team Members:

Wayne Lanning, DRS

P. K. Eapen,

DRS

S. Barber, DRP

R. Bhatia, DRS

D. Brinkman, NRR

Thomas T. Martin

Regional Administrator

Marvin W. Hodges

Charles W. Hehl

CC:

W. Kane, DRA

C. Cowgill, DRP

L. Nicholson, DRP

Team Members

R. Capra, NRR

J. Calvo, NRR

R. Lobel, OEDO

K Abraham, RI

APPE

IX B

PER

N

NTA TED

MhwkP w

'

~Nm-

Pg+ii~n

J. C. Aldrich

P. Allen

D. Althouse

W. Bandla

H. Barrett

C. Beckham

K. Belden

T. Bockman

R. Burtch, Jr.

P. Candella

A. Curran, Jr.

K. Dahlberg

M. Dooley

J. Endries

C. Fischer

D. K. Greene

D. Hosmer

R. E. Jenkins

F. LePine

E. Lighthall

P. Mazzaferro

M. McCormick, Jr.

B. Mercier

L. E. Pisano

D. Reynolds

J. Rizzo

G. Roarick

A. Salemi

B. Sherman

J. Spadafore

B. Sylvia

T. Syrell

C. Terry

R. Tessier

S. Wilczek, Jr.

H. Wysocki

QA Unit 1

"C" Operator

System Engineer

Unit //1, OPS

General Supervisor, Operations

Mgr, QA OPS - Unit 2

Assistant Station Shift Supervisor

Chief, Shift Operator

Mgr. Nucl. Comm.

Site Engineer

Site Licensing

Plant Mgr. Unit 1

Tech Supp. NMP1

President, Niagara Mohawk Corporation

~ General Supervisor, Maintenance

Mgr Licensing

Unit 2 Mgr Outage/Wk Control

QA/Operating Experience

Electrical Maintenance Schedules

Lead Site Engineer

Outage Coordinator

Plant Mgr - Unit 2

Electrician

WCC/OMG

Electrical Maintenance Supervisor

In-plant "E" Operator

Station Shift Supervisor

Director, Emerg. Prep.

Electrician

ISEG

Executive V. P. - Nuclear

Electrician

V. P. - Nuclear Eng.

Mgr OPS

V. P. - Nuclear Support

Electrical Systems Engineer

Appendix B

S Nuclear Re ul t

mmi sion

R. Capra

C. Cowgill

C. Hehl

W. Lanning

R. Laura

S. Young

~ W. Schmidt

D. Brinkman

Project Director, NRR

Branch Chief, RI

Director, "DRP, RI

Deputy Director, DRS

Resident Inspector

Sr. Resident Insp'ector

Sr. Resident Inspector

Sr. Project Manager, NRR

Denotes those present at the exit meeting on March 4, 1992, attended by the public and

news media.

The team also held discussions with other licensee management,

operations,

maintenance,

engineering and quality assurance

personnel.

PPE

IX

REVIEWED

1.

Chief Shift Operator and Station Shift Supervisor logs for February 20, 1992 and

February 21, 1992

2.

Copies of written statements provided by personnel involved in the event.

3.

Internal memorandum to:

R. L. Tessier, From: 'H. T. Barrett, dated: February 21,

1992, Subject:

"Accountability Meeting on Loss of Heat Sink"

4.

Station Shift,Supervisor Instructions dated 02/20/92

5.

Vnit 1 Daily Work Schedule for 02/19/92 - 02/22/92

6.

Work Request No. 163740

7.

Blue Mark-up 1-92-50111

8.

Deviation/Event Reports 1-92-Q-0267, 1-92-Q-0286, 1-92-Q-0390

9.

Operating Procedure Nl-OP-19, Rev. 18, Section 2.0, Reverse Flow:

Screen house

Operation

10.

Condenser vacuum strip chart for 02/21/92

11.

Computer alarm logs for 0746 hours0.00863 days 0.207 hours 0.00123 weeks 2.83853e-4 months - 1017 hours0.0118 days 0.283 hours 0.00168 weeks 3.869685e-4 months on 02/21/92

12.

Shutdown cooling heat exchanger inlet and outlet temperature strip chart for 02/21/92

13,

Special Operating Procedure Nl-SOP-7, Rev. 01, Service Water Failure

14.

Drawing No. C-18318-C,- Screen & Pump House Service Water Plan Above El 256'0"

15.

Drawing No. C-18319-C, Screen & Pump House Service Water

16.

Drawing No. C-18320-C, Screen & Pump House Service Water

Plan Above'l 233'0"

Sections

1-1 and 9-9

17,

Drawing No. C-18321-C, Screen & Pump House Service Water Section 2-2

18.

Drawing No. C-18322-C, Screen & Pump House Service Water Section 3-3

-

19.

Drawing No. C-18323-C, Screen & Pump House Service Water Section 4-4

Appendix C

20.

21.

22.

23.

Drawing No.

8-8

C-18324-C, Screen & Pump House Service Water Section 5-5

F

C-18325-C, Screen & Pump House Service Water Section 6-6

C-18326-C, Screen &Pump House Service Water Section 7-7

C-18327-C, Screen & Pump House Service Water &Fire Prot. Section

4

24.

Drawing No. C-18328-C, Screen & Pump House Sealing Water &Fire Prot. Plan

Above El 256'0"

.

25.

Drawing No. C-18329-C, Screen & Pump House Sealing Water & Fire Prot. Section

1-1

26.

Drawing No. C-18330-C, Screen & Pump House Fire Prot. Details in Diesel Driven

Fire Pump Room Plans & Sections

27.

Drawing No. DEN 16334, 2500 G.P.M. Fire Pump Elevation 15 HN-410EF-4 Stage

Vertical Turbine Pump

28.

Drawing No. F-63022-C, Sheet 1, Service Water Reactor & Turbine Bldgs., P & I

Diagram ASME Section XIBoundary Diagram

29.

30.

Drawing No. F-63022-C, Sheet 2, Reactor Bldg. Closed Loop Cooling System ASME

Section XI Boundary Diagram

Drawing No. F-63022-C, Sheet 3, Turbine Building Closed Loop Cooling System

ASME Section XI Boundary Diagram

31.

Drawing No. C-18022-C, Sheet

1, Service Water Reactor &Turbine Bldgs P &I

Diagram

32,

33.

Drawing No. C-18022-C, Sheet 2, Reactor Bldg. Closed Loop Cooling System P &I

Diagram

Drawing No. C-18022-C, Sheet 3, Turbine Building Closed Loop Cooling System P &

I Diagram

34.

Drawing No. C-18022-C, Sheet 4, Waste Buildings Closed Loop Cooling System P &

I Diagram

35.

36.

Maintenance History For Screen House Gates, 01/09/80 to 02/04/92

Document No. SDBD-502, Rev. 0, Service Water System Design Bases Document

Appendix C

37.

Drawing No. C-19441-C, SH. 1, SH.4, SH.4A, 3A, Elementary Wiring Diagram 600

Volt Power Board 176 Power Circuits

38.

Drawing No. C-22303-C, SH.3 Interconnection Diagram,. 600 Volt Power Boards 176

39.

Vendor Drawing No. F26192, 46B, Repair Part Catalogue,

Safety Suggestion for

Crane &Hoist, Lubrication Chart, Frequency,

General Instruction for Shepard Niles

'oller

Bearing Hoist

40.

Drawing No. C-22009-C, SH.3, SH.6 Interconnection Wiring Diagram Instrumentation

Systems for Service and Cooling Water

41.

Drawing No. C-18022-C, Service Water Reactor &Turbine Bldgs. P &I Diagrams

42.

Alarm Response

Procedures

(ARP)

N1-ARP-H2(1-3) Circ Water Pump Intake Level Low

N1-ARP-H1(4-2), Service Water Pump Low Hdr Pressure

N1-ARP-H3(4-1), Screen house Low Seal Pressure

N1-ARP-H2(2-8), Fire Header Low Pressure

Nl-ARP-H1(4-3), Screen Wash Pump

43.

Administrative Procedure AP-5.2.5, Rev.01, Work'In Progress (WIP)

44.

Administrative Procedure AP-5.4, Rev.04, Conduct of Maintenance

45.

Administrative Procedure AP-5.4.2, Rev. 02, Troubleshooting

46.

Administrative Procedure AP-5.5, Rev. 02, Work Control

47.

Administrative Procedure AP-5.5.1, Rev. 06, Work Request 48.

Generation Administrative Procedure GAP-OPS-02, Rev. 00, Control of Equipment

Markups

49.

Administrative Procedure AP-6.1, Rev. 03, Control of Equipment Temporary

Modifications

50.

Nuclear Division Interfacing Procedure NIP-ECA-01, Rev. 03, Deviation Event Report

51.

Generation Administrative Procedure GAP-OPS-01, Rev. 00, Administration of

Operations

ML17056B732

Text

Navigation menu

Search