ML20236K445

Forwards Augmented Investigation Team Rept 50-440/87-14 on 870617-20.Enforcement Action Re Errors in Plant Design,Const & Testing Forthcoming.Record Copy

ML20236K445
Person / Time
Site:	Perry
Issue date:	07/30/1987
From:	Norelius C NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:	Edelman M CLEVELAND ELECTRIC ILLUMINATING CO.
References
NUDOCS 8708070121
Download: ML20236K445 (2)

See also: IR 05000440/1987014

Text

__

t

-

JUL s o 1997

Docket No. 50-440

The Cleveland Electric

Illuminating Company

i

ATTN:

Mr. Murray R. Edelman

I

Vice President

Nuclear Group

'

Post Office Box 5000

i

Cleveland, OH 44101

1

Gentlemen:

This refers to a special onsite investigation review conducted by the NRC

Augmented Investigation Team (AIT) on June 17-20, 1987, relating to the

unexpected scram following a loss of the "A" RPS bus and an unexpected closure

of the outboard Main Steam Isolation Valves.

This also refers to discussions

of findings with yourself, Mr. A. Kaplan, and others of your staff during the

course of and at the conclusion of the onsite AIT review on June 20, 1987.

I

We are concerned about the series of errors that were made that allowed the

plant to be designed, constructed, and tested without identifying that the

'

plant was not built in accordance with the Final Safety Analysis Report.

Enforcement action with respect to these errors will be forthcoming in a

subsequent report.

The enclosed copy of the Augmented Inspection Team report identifies areas

examined during this review.

Within these areas the investigation consisted

of a review of representative records, observations, and interviews with

personnel.

l

In accordance with 10 CFR 2.790(a) a copy of this letter and the enclosure

will be placed in the NRC Public Document Room.

We will gladly discuss any questions you have concerning this inspection.

Sincerely,

ORIGINAL SIGNED BY C. E. NORELIUS

Charles E. Norelius, Director

DivisionofReactorProjects

Enclosure:

Augmented Investigation

Team Report No. 50-440/87014(DRP)

i

See Attached Distribution

l

RIIIp

RIII

NRR

R

RII

Kd d55 fo

ey

Mu

Burdick

N ris us

,

w

-

y

w

.

[g ,o e0 7aLgkA

<co d Cm

""

i

e

-

t

.

The Cleveland Electric

2

. JUL 3 0 1987

!

Illuminating Company

1

l

1

1

1

cc w/ enclosure.

F. R. Stead, Manager, Perry

Plant Technical Department

M. D. Lyster, Manager, Perry Plant

Operations Department

Ms. E. M. Buzzelli, General

Supervising Engineer, Licensing

and Compliance Section

i

DCS/RSB (RIDS)

I

Licensing Fee Management Branch

Resident Inspector, RIII

Harold W. Kohn, Ohio EPA

Terry J. Lodge, Esq.

James W. Harris, State of Ohio

Robert M. Quillin, Ohio

,

Department of Health

State of Ohio, Public

Utilities Commission

R. L. Spessard, AE0D

,

l

T. M. Novak, AE0D

l

T. G. Colburn, NRR

l

W. D. Lanning, NRR

I

l

i

.

i

f

o

m.,

.m,

ty

3c

.m

,.

7

,

1

i

!

)

U.S. NUCLEAR REGULATORY COMMISSION

'I

Region III

4

-

i

Augmented Investigation Team

I

l

Report No. 50-440/87014(DRP)

License NPF-58

l

Docket No. 50-440

Licensee:

The Cleveland Electric Illuminating Company

Post Office Box 5000

l

Cleveland, OH 44101

Facility Name:

Perry Nuclear Power Plant, Unit 1

Inspection At:

Perry Site, Perry, Ohio

Inspection Conducted:

June 17 through June 20, 1987

Inspectors:

Kevin A. Connaughton

Division of Reactor Projects

David E. Hills

Division of Reactor Safety

J. E. Mauck

Nuclear Reactor Regulation

Gerald F. O'Dwyer

DivisionofReactorProjects

i

'

Michael E. Parker

Division of Reactor Projects

Richard C. Knop, Team Leader

Division of Reactor Projects

Approved By:

E

7/u//7

i

ReactorProjects,Branc 1

Dat'e /

!

1

,h

s t ud 67014-w

_ _ _ . .

>

_ _ _ _ _ _ _ - _

.I

,. a y

cwrf:

o

in x

w

b

..

,

,

.

't'

1

l

x

.,

,

.

>

.

i

Table of Contents

"

-

,

\\

f

4

-

,

,

{

+

<t

.

<

L

1.

General Discussions

,/

'

,

,

-

'

r,

',

';

Initial Followup'of Event

') '

/

i.'/

'

a.

t

"

b.

Scram Chronology

.

\\

>

/ 1,

.e

c, ,

,

.

,

.

'

!

c.

AIT Formation

i

/'

'

e

<

i

. <

.

V

f

l

2.

Design Description '- Main Steam Isoladion Valve ConWo'l

..

7'i

,

!

'

t

\\

l

3.

Discussion of Design Error

l

'

4.

Preoperatismal Tr'st Frogram Review

\\1+

a.

Procedure Preparation

e

b.

Procedure Test Results

l', g

.

i

s

q

,

~

c.

Followup on Previous Work

9

j, .

<

i

)

.

1

l

'

-

.

<

5.

Re-circulation Pump "B" Feiltere to Start

"

'\\

' < -

,

n ,

t.

s

6.

Licensee Event iteport Re;f ew 0'P'otential Precursors

j

• '

t

,1

-

7.

Electrical Protection As!embii, s

e

'

'

7

t

-

( ,

8.

Post Scram Evaluation

~

' '

,

>

'

E

i

I

! t

. ;'

,'

9.

Interim hinistrative Controls for Respnding to the Loss of a dre.}le

RPS Electrical Bt's

,e

/ 0

-

>

>

,

,

.-

10.

Exit Interviews

i

9

.

, . , ,

,

,

"

/'

11.

Conclusions

".

..I

\\.

.

12.

Attachment - Commitment Letter Dated June 20,'19?]

's

)

t,

j

l

e

\\.

'

'

'

f -,

,

y.,

" _ > s .i

ri

i

.

y

i

J

s

f

,)

-

.

Q

l}

+

1

~

.

.-

,n

e

%

)

,

(

3

,

i

1

'

. .y

er

o w],,-

-l

$

,

a

_

y

y ~,

37.gn

m-

m

.

- .

-

-

.

.

.,

,

,

1

r-'

r

(j , j .

g4

sa

f h 1. p GeneralDis5udion

.

a.

InitialFoQow-upofEvent

'

,

L

/

On' June 17, 1967, the NRC was notified by the licensee of a reactor

'

,

scram that occurrsd at 11:39 a.m,~EDT that morning.

'

'

,

!

)

L

>

As a result of the reactor trip and the unexplained closure and

'

subsequent re-opening of the outbnard main steam isolation valves

'

f-

>

(MSIVs), an inspector was dispatched to'the site to review the

'

'

,'_

'

cirr.umst,ances of the event.'

The licensee had provided verbal

> assurances to Pegion.III that the reactor would not be restarted

l

pending review enf approval by the NRC.

i

'

Upor., arriving on site, the inspector met with operations and

'

engineeringpersonneltodeterminecurrentplantstatusandreview

the licensee s initial determination of the cause of the event.

The

,

plent was in Mode 3, hot shutdown, at the time and proceeding to

,

cold shutdown.

The licensee's initial determination was that the

>

, reactor scram was caused by the closure of the outboard MSIVs.

l

The irspector reviewed the following documents and drawings to try

l

to datermine the cause of the MSIV closure:

'

'

I

10 CFR 50.72 Event Report

+

Unit Logs

i

Cor.dition Report

c

<

Final Safety Analysis Report

1

!

Nuclear Steam Supply Shutoff System Power Distribution

I

i

Reactor Protection System M-G Set 5001A

'

Reactor Protection System M-G Set 5001B

Nuclear Steam Supply Shutoff System Main Steam Line Isolation

+

Valves - Inboard

Nuclear Steam Supply Shutoff System Main Steam Line Isolation

l

i

Valves - Outboard

l

Nuclear Steam Supply Shutoff System logic, Logic A and C

'

Nuclear Steam Supply Shutoff System Logic, Logic B and 0

After reviewing these documents and drawings with the licensee it

was evident that the closure of the outboard MSIVs was due to the

loss of Reactor Protection System (RPS) Bus "A".

Each MISV has two

!

!

_ _ _ _ _ _ _ _ _ _ _ _

3

-

. . .

,, ,

,,

w

- ,

.

,

.

,

solenoid operated pilot valves.

The MSIVs are designed to fail

closed upon loss of power to the pilot solenoids.

Perry's current

as-built condition has both pilot solenoid valves for the outboard

MSIVs powered from RPS Bus "A".

Therefore, upon loss of RPS Bus "A",

the outboard MSIVs closed, resulting in the reactor scram.

At the time of the review the licensee had already conducted testing

of the power supply to the outboard MSIV pilot solenoids and verified

the configuration identified on as-built drawings was correct.

Review of the power su) ply for the inboard MSIV pilot solenoids

identified that both tie A and B solenoids for all four inboard

MSIVs were powered from RPS Bus "B".

This was subsequently verified

by the inspector on June 18,"B".1987, during the performance of a test

to simulate loss of RPS Bus

This test verified that all power

was lost to both the A and B pilot-operated solenoid valves for the

inboard MSIVs.

Initial review of the Final Safety Analysis Report (FSAR) identified

that Section 6.2. states, "Each main steam isolation valve is served

by two independent p'ilot valves, each of which is powered from an

independent source.

Contrary to the requirements stated in the FSAR, the inboard and

outboard MSIV pilot solenoids are not powered from independent

sources,

b.

Scram Chronolooy

Time

Event Description

.

1100

Recirc pumps shifted to slow speed per startup

procedure to enter natural circulation test.

1134

Tripped recirc pumps A and B per startup procedure.

1139

RPS Bus A de-energizes when EPA breaker trips.

The

following occur:

Outboard MSIV pilot valves de-energize, MSIVs

)

close.

Reactor scram due to MSIV closure.

l

IRM, Div 1, SRMs, Div 1 - lose power.

l

l

'

B0P isolation (outboard) signal.

RWCll/MSL drain outboard isolation.

i

l

'

l

1141

Transferred RPS Bus A to alternate supply. Outboard

j

l

MSIVs opened.

1

2

!

1

.

. . . .

-

..

..

.

,

f

g -

- ,

.

, , ,

,

.

,,

,

,

,,

,

1143

Operator closes outboard MSIVs with manual switch.

1152

Reset Reactor scram.

I

1156

Reset tripped EPA, transferred RPS Bus A back to normal

power supply; reset half RPS trip.

Ccmmenced BOP

isolation recovery.

1156

Opened main condenser vacuum breakers due to lack of

i

steam supply.

'

l

1156

Commenced cycling SRVs to control Reactor pressure.

1200

Inboard MSIVs close on low vacuum signal.

1322

Received reactor low level 3 scram due to void collapse

and shrinkage.

!

l

1327

Reset RPS scram signal.

l

1

'

Legend

RPS - Reactor Protection System

i

EPA - Electrical Protection Assembly

i

!

MSIV - Main Steam Isolation Valve

IRM - Intermediate Range Monitor

i

'

SRM - Source Range Monitor

l

B0P - Balance of Plant

1

RWCU - Reactor Water Cleanup System

i

i

MSL - Main Steam Line

j

SRV - Safety Relief Valve

c.

AIT Formation

Based on the unexpected closure of the MSIV Valves on loss of a

j

single RPS bus trip, an Augmented Investigation team was formed

and aispatched to the site on June 18, 1987.

An entrance meeting was held on June 19, 1987.

The charter of the

team was to perform a fact-finding review of the events, to

j

communicate these facts to regional and headquarters personnel, to

"

identify any potential generic safety concerns, and to document the

results of the onsite review.

!

!

3

_-

[3 8"OM1 f

f,

2

ND*

AW

?-7 YYIOk.N N' ; h4 **'f

(

b

f3

i ' l

4

4

e

f

r

"

i

- 2.-

Design Description: ' Main Steam Isolation Valve Control

Each steam line has two containment isolation valves,- one inside and one -

outside the containment barrier. The isolation valves are spring loaded,

pneumatic piston-operated globe valves - designed to fail closed on loss:

of pneumatic pressure or loss of power to the power solenoids.. Each valve

has an air. accumulator to assist.in the closure of the valve.upen loss of

the air supply or electrical power to the pilot solenoids and failure of

the loaded spring.

Each MSIV has two control. solenoids, "A" and "B",-both

of which must be de-energized for the valve to close.

During normal

operation, both solenoids are energized. The "A" solenoid should be

powered from RPS bus "A" (Distribution Panel P001), and the "B" solenoid

from RPS bus "B" (Distribution Panel P002).

Power is provided to the

solenoids via MSIV control switch contacts; thus, each MSIV control.

switch contact section contains wiring associated with both RPS buses

" A" and "B".

There is no physical separation provided between these

circuits.

In fact, all wiring associated with a given control switch is

color coded either yellow qif the MSIV is a " Division 1/ outboard" valve)

or blue (if the MSIV is a ' Division 2/ inboard" valve).

There is no color.

distinction made between circuitry associated with redundant RPS buses in

this application.

3.

Discussion of Design Error

l

The anomaly regarding the power to the KSIV pilot solenoid valves

j

originated with General Electric drawing "GE828E445CA" dated May 12, 1976.

This drawing de$ and both solenoids of the inboard MSIVs powered from R

icted both solenoids of the outboard MSIVs powered from

l

RPSA(C71-P001

B (C71-P002). However, Revision 2 dated February 14, 1977 provided for

independent power sources to each solenoid of a MSIV (RPS d power to

solenoid A and RPS B power to solenoid B).

This change was made by-

General Electric to prevent loss of a single RPS bus causing the

closure of the outboard or inboard MSIV (e.g. ,iates Incorporated) for the

the June 17, 1987 event at

Perry).

The Architect Engineer (Gilbert Assoc

Perry Plant failed to correctly transpose the. power supply revision

rovided in GE 828E445CA, Revision 2, to the Gilbert Associates Incorporated

p(GAI) elementary diagram B-208-013 (sheet number 5).

This error was

carried throughout the later revisions and as a result non-independent

,

l

power supplies were p(attached) the licensee committed to modify the pow

rovided to the dual solenoids at the MSIVs.

By letter

dated June 20, 1987

j

supplies to the MSIV pilot solenoids.

This modification will preclude an

MSIV isolation caused by the loss of a single RPS oower supply and will

'

follow the design guidelines provided by General Electric and the designs

'

presently installed at other BWR 6 plants (e.g. , Grand Gulf, River Bend,

and Clinton).

In addition, the modified design will be in agreement with

the Perry FSAR, Sections 6.2.4.2.1,.8.3.1.1.5.4, andFigure7.3-3(Sheet 4).

This modification will be completed prior to startup following a July 12,

1987 outage.

Furthermore, the licensee has committed to discuss the

details of this modification with the NRC staff.

The staff was concerned that other transposing errors could have been made

by GAI during the transposing process.

By letter dated June 20, 1987, CEI

4

_ _ _ _ _ _ _

____

-__-_

_ _ _ _ _ - - _ - - _

v,

a m pa s ~ . ,

.

~ ry mm e -

<m

x.

m w m

-m

s wav y

,

..

committed to complete a consistency review of General Electric drawings

versus GAI drawings for the C71 (Reactor Protection), C51 (Neutron

= Monitoring), and B21H (Nuclear Steam Supply Shutoff) systems.

This review

was to be completed by June 22,1%7, and all attendant changes to the

drawings to be completed by June 26, 1987. A summary of the results of

this review will be sent to the NRC. On June 20, 1987, approximately

j

half of the system drawings had been reviewed with no apparent errors

-j

discovered.

However, discrepancies were noted in such areas as footnote

-1

numbering, contact status representation, and a relay designation.

The

{

relay designation was picked up in late 1986 and as a result DCP-86-P216

l

was issued on December 23, 1986, that will correct the applicable GAI

'

'

drawings.

The other discrepancies were considered ~to be correctly

transposed in that they were done according to GAI footnote and contact

!

representation philosophy.

Based on the review of the interim results

and the commitments provided by CEI, the staff concludes that this concern

is resolved.

our review of the MSIV pilot solenoid power configuration the staff

I

During%at within panels H13-P691, 692, 693

noted

and 694 acceptable isolation

I

ard separation was not provided.

Specifically,theisolationconcernwas

divisional contact to contact isolation at relay K7A,B,C,0.

This concern

will be resolved when the power supply configuration for the MSIV solenoids

l

is modified.

At that time the divisional isolation will become coil to

'

contact which is acceptable.

The contact to contact divisional isolation

will be removed because the power supply for all of the B solenoids will

become RPS B power which is within the same division as logic contacts 78

i

and 70. The reverse is true for all of the A solenoids.

The separation

concern involved intruder divisional wiring in panels H13-P691, 692, 693,

and 694.

This wiring was placed in conduits that were not terminated as

close as practicable to the common device (K7A-D relays).

As a result of-

this incorrec+. termination, the intruder divisional wiring is physically

touching the predominate panel divisional wiring which does not follow the

ouidelines of Regulatory Guide 1.75 or the commitments provided in the

Perry FSAR Chapter 7 and 8.

In the June'20 letter, the licensee has

committed to install a permanent barrier between the intruder divisional

'

wiring and the predominate panel divisional wiring prior to startup

following the July 12, 1987 outage.

The staff finds this commitment

acceptable and, therefore, the separation concern will be resolved.

4.

Preoperational Test Program Review

a.

Procedure Preparation

The inspector reviewed the as-built drawing verifications to

ascertain whether this process should have identified the design

error.

Since the actual plant configuration matched the plant

electrical schematics and wiring diagrams the inspector concluded

,

that this process would not have identified the problem.

l

A review of Preoperational Test Procedure IC71-P002 by the inspector

indicated that Section 6.3 had not been written in accordance with

plant design drawings, but coincided with the system description

given in the FSAR.

i

1

,

1

5

1 -

>

. ~

ap

+m

y

-

.

+

,s

1

]

j

It should be noted that as a result of earlier NRC identified

i

violations in the Preoperational Test Program, various supplemental

l

Assurance Review,g Special Project Plan 1102, '! Test Procedure

programs includin

had been implemented to ensure test adequacy.

l

1

Activities associated with these programs should have been

$

sufficient to identify the conflicts between the system design

I

depicted in the Final Safety Analysis Report (FSAR) and the as-built

l

design which should have been tested in the preoperational test

i

procedure.

q

The inspector's review of the Management Procedure Review Team (MPRT)

documentation indicated that the review did identify that the

preoperational procedure did not reference all applicable drawings,

but did not identify that the FSAR and the applicable drawings did

not match.

The MPRT accepted the response from the preo>eration

writer that the applicable drawings were identified in tie initial

checkout and run-in (IC&R) procedure.

It should be noted that the

.

IC&R procedure tested the HSIV circuitry in accordance with the

i

drawings and not in accordance with the FSAR description.

The MPRT

'

failed to identify this fact as well.

1

In summary, the reoperation test as written matched the FSAR

description, but did not match the as-built condition of the plant.

The review process did not identify this discrepancy.

b.

Procedure Test Results

The inspector further reviewed Test Procedure IC71-P002 results

l

package documentation to determine why actual test performance did

not identify any problems.

Since the preoperational te:t procedure

did not reflect the as-built design it should have been impossible

to perform as written.

However, the inspector noted that the System

Test Engineer (STE) had in fact signed off the applicable steps as

,

l

satisfactorily completed.

Furthermore, these steps were Quality

l

Assurance witness points and thus had also been verified by a Perry

Plant Quality Assurance (QA) Inspector.

These individuals are no

,

longer employed at the Perry Plant.

However, the licensee provided

i

the NRC inspector with documentation of interviews they conducted

l

!

with the STE and QA Inspector following discovery of this problem.

This documentation indicated that the STE had been confused in regards

1

l

to the MSIV solenoid lights on Control Room panels H13-P622 and

4

H13-P623.

Both the "A" and "B" solenoid lights for the outboard

i

MHVs are . contained on panel H13-P623 while panel H13-P622 contains

these for the inboard MSIVs.

The STE had thought however, that the

!

panel H13-P623 contained the lights for all the "A" solenoids while

i

panel H13-P622 contained the lights for the "B" solenoids.

Although

these panels are now clearly labeled for each of these solenoid lights,

the inspector verified through review of Field Deviation Disposition

Request KL1-942 that they were not labeled as such at the time of the

test.

Furthermore, light suffice nomenclature appearing on the inside

of the panels above each of the solenoid lights all contained an "A"

t

!

in the number on panel H13-P623 and "B" on panel H13-P622.

The STE

,

6

l

-

1

J

J t . 3 N:. .,}

.' '

y. ,

p

y

'

$'p.,

@ -i @.

My

f

,

9

did not realize that these numbers referred to the lights themselves

instead of the solenoid designations.

Thus, during testing when

RPS MG Set "A" output breaker was opened causing t1e "A" and "B"

solenoid lights for the outboard MSIVs on panel H13-P623 to extinguish,

the STE thought the "A" solenoid lights for both the inboard and

outboard MSIVs had extinguished.

He also verified that the solenoid

lights on panel H13-P622 all remained energized.

The inspector

,

'

reviewed the Jumper and Lifted Lead Log for the time period of the

'

test against electrical schematic and wiring diagrams to ensure that

Nuclear Steam Supply Shut-off System (NSSSS) logic input relays would

'

have remained energized upon loss of the RPS Bus.

The inspector

verified that jumpers installed would have kept panel H13-P622 MSIV

solenoid lights energized under these conditions.

This confirms what

the STE and QA Inspector claim they saw.

In addition, a note in the

'

chronological test log verifies they saw lights on only one panel

extinguish.

Thus, the STE mistakenly signed off the steps indicating

that the "A" solenoids were off while the "B" solenoids were on.

The QA Inspector believed assurances from the STE that the lights

were correct.

A similar explanation was provided by the licensee

for the case of opening RPS MG Set "B" output breaker.

However,

,

l

this explanation still does not address that the procedure steps

specifically identify both "A" and "B" solenoids as being found or.

each of the panels.

Therefore, these panel designations in the

procedure steps would have had to have been ig'that the test procedure

'

nored during the test.

Actual conduct of the test failed to identify

did not match the as-built design of the plant.

Accordingly, this

process did not note discrepancies with the FSAR and thus process

did not identify the plant design error.

!

l

During the review of the Jumper and Lifted Lead Log the inspector

!

notedthatthetestprocedureitselfdidnotidentifythejumpers

1

l

used to ensure the NSSSS logic input relays remained energized during

the test.

The jumpers were installed and documented in the IC&R test on

April 16, 1985.

The IC&R testing was reviewed by the TPRC and the

remainder of the test was released on April 24, 1985.

Therefore,

the jumpers were reviewed for future impact on subsequent

preoperational testing in conjunction with the TPRC review.

'

Furthermore, the IC&R test was included in the preoperational test

results package and thus the jumpers were again available for TPRC

review for impact on testing.

The inspector has no further concerns

regarding jumpers and lifted leads during the preoperational test.

c.

Followup on Previous Work

Asaresultofthevariouserrorsinconjunctionwiththe

preoperational test, the remaining work performed by the involved

individuals became suspect.

Therefore, the licensee performed a

review to determine the entire scope of the individuals' involvement

in the preoperational test program.

The STEs who improperly wrote or

performed the test steps were both identified to have worked in

.

7

_ _-

tll

JgfW4

@d

f19

i (% 4@# ^

y

i

s i ; y

(

+Q,

sjgx id)

Q(3

ph

JWA

'

'

i'

e

r

t

o

'

.a

conjunction with only one 'other preoperational: tes't procedure.

The results package for this procedure,1C71-P001 " Reactor Protection

Systen " was then reviewed by the licensee to verify that each step

waswrlttenandperformedtotheappropriatedrawingsandthatthe

test procedure addressed all testing commitments.

Furthermore, the

licensee verified that current surveillance instructions adequately

address the logic originally demonstrated in the preoperational

test.

The licensee found that the QA. Inspector involvement was a

l

much wider extent.

Various activities were undertaken by the licensee

i

to develop an assessment of the QA Inspector's performance.

Personnel

I

records were reviewed to ensure adequate experience and' qualifications.

In addition, ten percent of the technical specification instrumentation-

inspected by the QA Inspector during initial calibration was compared

{

against subsequent surveillance tests to ensure adequacy of inspection

results.

Other testing witnessed by the QA Inspector was compared to

Licensee Event Reports to confirm that his work did not contribute to

1

significant safety deficiencies.

Inspection reports were reviewed

,

l

to ascertain his knowledge of system configuration and instrument

acceptability.

Finally, nonconformance reports by the QA Inspector

were reviewed for adecuacy of technical content and quality assessment.

The licensee concludec through these various reviews that remaining

i

work by these individuals does not constitute any significant problem.

j

In summary, the Preoperational Test Program should have been able to

identify the discrepancy between the as-built design and the FSAR.

j

This then should have led to discovery of the design error.

However,

due to an inadequate pre-operational test procedure, and the

,

'

subsequent inadequate performance and quality assurance witnessing

of this test procedure the discrepancy was not identified.

Extensive

reviews required by Special Project Plan 1102 failed to identify the

testing problems.

As a result, the licensee has completed a review

to ensure remaining work by the involved individuals does not

represent any serious concern.

5.

Recirculation Pump "B" Failure to Start

The inspector interviewed maintenance personnel who had been involved with

troubleshooting and reviewed plant electrical schematic diagrams to

ascertain the cause of Recirculation Pump "B" failure to start.

Maintenance personnel had determined that diode D2B in Low Freguency Motor

l

Generator Set Panel 1B33-P001B, located in the Auxiliary Building, had

sheared in half.

The sheared diode acted like an electrical short causing

current to bypass the K1338 starting sequence auto transfer relay.

Failure

of this relay to energize, in turn, caused a Pump Motor Breaker 5B failure

to close.

This diode normally acts as a surge suppressor in the pump start

circuitry.

Upon removal of this diode the pump started satisfactorily.

The diode was subsequently replaced by maintenance >ersonnel.

It could not

be positively determined what caused the diode to slear.

The licensee

indicated that this failure had not occurred at the Perry Plant in the

.

past and thus the inspector concluded that this was an isolated occurrence.

1

,

!

The inspector also verified that relay K133B is no longer energized once

j

the pump is started.

Thus, failure of the diode will have no effect upon

l

8

y

,-w

.

n

mm ~w

, < ngw p.w

+,.

~

,

,

i

1

o

1

,

the pump while it is running: The inspector also verified this for relay ~

-

K113B which is in parallel to relay K133B and diode D28.

The inspector has

no other concerns in this area. .

.;

6.

Licensee Event Report Review _ Potential-Precursors

-

e

Review of Perry Licensee Event Reports (LERs)-identified four LERs and

i

six instances in which the loss of-an RPS Bus has occurred previous to

this event.

The LERs are:

1

1

LER 86-044

August 6,1986, ' Loss of RPS Bus "A" as a result of a

!

failed capacitor in the EPA process control board.

LER 86-050

August 20, 1986, Loss of RPS Bus "B" as a result of

inadvertent de-energization of RPS Bus B normal power

l

supply.

• -

LER 86-071

October 24, 1986, Transfer of RPS Bus "A" power

supply resulting in unexpected B0P isolations.

LER 86-072

October 25, 28, and 29,1986, Loss of RPS Bus "A" as

a result of voltage fluctuations while on the

alternate power supply.

a.

A review of these LERs was performed to determine.if any of the six

events were precursors that should have alerted the licensee to the

l

current design of the MSIV pilot solenoid valve' power supply.

This review consisted of determining the plant. status, and

i

specifically the position of the MSIVs at the time of each event to

^

determine if any of these events were repeat occurrences of the

,

June 17, 1987 event.

This consisted of reviewing the applicable

i

LERs, Unit Logs, and Test Procedures.

Based on the information reviewed and personnel interviews the

inspectors came to the conclusion that during each of the previous

l

events the reactor was in either cold shutdown or hot shutdown and

3

'

the MSIVs were closed.

As such, the licensee was not specifically

'

alerted to the loss of power to the MSIV pilot solenoid valves for

either the inboard or outboard MSIVs.

,

l

The inspectors have requested the licensee review each of the

specific instances identified in the above LERs to determine what

information was available to the operators, and if this condition

could have been recognized as a result of that information.

1

.

b.

Further review of LER 86-044 dated August 29, 1986, identified that

J

the licensee committed to change out all. EPA electronic process

control boards for both RPS Bus A and B Motor-Generator sets and

their alternate power supplies.

Review of the' commitment tracking

system and work orders indicates that to date the licensee has only

changed out the EPA control boards for the alternate power supplies.

.

9

__

_

.

.

M'

w

.e

v ; / wep-ms:9

g uw:g

ye p y ra wy 4 '

i%9ayr,

p

4

..

AsaresultoftheEPAtrigonJune 17, 1987, the licens'ee has-

changed out the RPS Bus "A

EPA circuit boards for the normal power

sup?ly and performed surveillance testing to verify conformance to

Tecinical Specifications.

This testing was witnessed by. the

inspectors.

'

LER 86-044 alu identifies that as a result of the capacitor

i

failures in the circuit board the licensee determined that the

manufacture has assigned a three year de-energized storage shelf

'

life.

During replacement of the circuit boards for RPS Bus'"A"

the inspectors observed that the circuit board had a shelf life

expiration date of December 25, 1991.

This shelf life exceeds the

three years identified in the LER.

The licensee'was able to provide

adequate documentation to substantiate the extended shelf life and

!

assure the inspector that this evaluation was per their established

shelf-life program and conforms to applicable military standards.

The inspectors have expressed concern to the licensee that they are

expected to comply with any commitments in the LER or provide'a

revision to the LER to clarify or modify any subsequent changes.

7.

Electrical Protection Assemblies

The initiating device for the reactor scram of June 17, 1987, has been

determined to be a tripped Electrical Protection Assembly (EPA) breaker

in the RPS Bus "A".

This device is a General Electric circuit breaker

which provides undervoltage, overvoltage, and underfrequency protection.

Two EPA breakers are provided in series for both the normal and backup

RPS power supplies for both RPS Bus "A" and'RPS Bus "B".

The EPA breaker

that tripped was 1C715003C which is second in series between the RPS M-G

set "A" and the distribution panel.

Reactor operators initially transferred the RPS Bus "A" to the alternate

power supply after the reactor scram and reset the reactor scram.

A

short time later the reactor operators reset the tripped EPA breaker and

transferred power supply back to the normal RPS supply.

The EPA breaker

5003C subsequently performed satisfactorily while in service.

l

On June 18, 1987, the licensee performed surveillance instruction

SVI-C71-T5230 to determine the as-found settings of the RPS "A" normal

power supply EPA breakers, and found it necessary to perform breaker

adjustments to bring the settings into specification to comply with

Technical Specification requirements.

On June 19, 1987, the inspector witnessed the testing of the replacement

logic card for EPA 1C71-S003A by Surveillance Instruction (SVI)-C71-T5230,

Revision 2, " Reactor Protection System (RPS)--Electrical Power Monitoring

Calibration / Functional for 1C71-S003A and 1C71-S003C." The card failed

the SVI and was replaced with a new logic card. Subsequently, both EPAs

IC71-S003A and IC71-S003C (with a new logic card in each) passed

SVI-C71-T5230.

These boards were initially quarantined by the NRC pending

evaluation of their failure mechanism.

10

.

1

ed

%s ea;p m +

<

p%B;Vf p-N-

/

6}, m i% wt &

r , W,y

me

%

.

v

After a visual ~. inspection and review of the licensee's ' esting of tiie

t

boards the NRC released the quarantine.

The circuit boards are to be

sent to the manufacturer, General Electric, for further testing to

determine the specific failure mechanism. The licensee'has committed to

provide additional information into the failure mechanism upon completion

of this testing.

On ' June 20, 1987, the inspector witnessed the successful retesting of EPA

1C71-5003B, by SVI-C71-5232,- Revision 2, "RPS--Electrical Monitoring

Calibration / Functional for IC71-S003B and 1C71-S003D."

Subsec uentl

i

IC71-5003D also passed SVI-C71-5232.

All EPAs in the affectec RPS .y'd" EPA

l)

power supply were replaced and retested satisfactorily and all EPAs in the

unaffected RPS "B" power supply were retested satisfactorily.- All testing

j

witnessed was performed correctly and no violations or_ deviations were

j

identified.

j

8.

PostScramEvaluation(93702)

)

On June 19 and 20,1987, the inspectors reviewed the licensee's post

j

scram evaluation documented in the Post Scram Restart Report 1-87-9

J

dated June 18, 1987.

The report provided a listing of key, plant

j

parameters along with their values prior to the scram, maximum and

'

minimum values during the transient, and post scram stable values.

Additionally, the report included a chronology of events and an

evaluation of plant response develo)ed from the Sequence-of-Events

,

recorder (SER), written statements ay on-shift operating personnel,

i

Emergency Response Information System (ERIS) archived data, and the

,

control room log.

These information sources were' attached to the report.

The licensee's evaluation of the event was-determined to be complete and

consistent with the findings of the AIT relative to plant response and

the identification of items requiring further evaluation and/or remedial

action prior to plant restart.

Specifically, the following items were

identified:

3

a.

The tripping of the "A" RPS Equipment protection assembly which

initiated the event.

b.

The discrepancy between the FSAR and the as-built MSIV pilot

solenoid power supply wiring configurations.

c.

The failure of the "B" reactor recirculation pump to manually.

start during scram recovery.

d.

The 100 degree / hour limit on reactor coolant system (RCS)

cooldown rate was exceeded at the bottom reactor vessel her

drain in the first hour following the scram.

The resolutions of items a. and b. above are discussed in Paragraphs 7

j

and 3 of this report, respectively.

Regarding item c., the licensee determined that a diode in the

reactor recirculation pump start circuitry had failed.

The diode was

replaced and the pump satisfactorily retested prior to plant restart on

June 22,1987.

This is further discussed in Paragraph 5.

'i

11

_

_ _ - _ _

__

4

Y

j?

h !'Y J

(f"F'

%^ *WW. !

~ b ?}Nt}

t 2$

ll

h

W *ff

?Q{Wh

,f

j * c

_

R*N 's

3.'$5

4

'n

w

3

]

.

d

Regarding~ ites d. , on June 18, 1987, the licensee performed an

-

engineering evaluation of the RCS cooldown rate exceedence.as required by

o

Perry Technical Specification' 3.4.6.1.

The evaluation concluded that the-

.

cooldown rate of 130 degrees / hour experienced following the scram had no

"

impact on fatigue or brittle fracture considerations.

The cooldown-

transient was determined to be bounded by existing analyses for control

rod drive mechanisms which are designed for up to 309 cycles with a

cooldown rate of 200 degreas/ hour.

The inspector reviewed the

engineering evaluation which was documented via facility Change Request

(FCR) No. 07036 and found it to be acceptable.

9.

Interim Administrative Controls for Responding to the Loss.of a Single

RPS Electrical Bus

On June 21, 1987 the inspector reviewed changes to System Operating

-Instruction (50l}-C71, "RPS Power Supply Distribution (Unit 1)'" and Off

Normal Instruction (0NI)-C71-2, " Loss of One RPS Bus (Unit 1) which were

issued by the licensee between June 19 and 21, 1987.

The procedure

changes were issued to reflect the as-built MSIV solenoid power supply

wiring configuration including plant response to the de-energization of. a -

single RPS electrical bus.

Additionally, the inspector verified by

direct observation that an information. tag had been placed on the RPS'

power supply transfer switch which instructed operators not to use the

switch unless the MSIV control switches have been verified /placed in the.

closed position.

This instruction was provided to preclude inadvertent

reopening of the MSIVs upon RPS' bus re-energization.

10.

Exit Interviews

The AIT met with the licensee representatives several times during the

inspection and at the conclusion of the inspection on June 20, 1987.

The AIT summarized the scope and results of the inspection.

The AIT

leader stated that after discussions with Regional and NRR management,

the plant would be allowed to restart based on

i

a.

The NRC's review of the 10 CFR 50.59 analysis by the licensee

H

indicating that there was no unreviewed safety question.

1

b.

A commitment by the licensee to accomplish certain tasks.

These

commitments are documented in the attached June 20, 1987 memo.

11.

Conclusions

!

The AIT's conclusions of the circumstances surrounding tne unplanned

)

closure of the MSIVs following loss of a Single RPS bus are as follows:

a.

The design error occurred in 1977 when GAI did not pick up on a

design change to power each of the redundant solenoids from separate

RPS busses.

(Note subsequent review by CEI did not identify any

other errors in RPS circuitry.)

t

12

'

J

_ _ _ _

.';

.

-.

i

I

.

.

b.

The wiring error was not identified during preoperational testing

j

due to a series of errors involving the Procedure Writer, the System

Test Engineer, the Management Procedure Review Team, and the Quality

Assurance Engineer.

Other work performed by the

personnel involved

was reviewed in detail.

No other significant pro)lems were

identified.

c.

The licensee intends to modify the circuitry during a July 12, 1987

outage.

In the interim, a review of 10 CFR 50.59 evaluation

indicates that there is no unreviewed safety question.

f

!

l

.

l

l

I

13

.