Text

Augmented Insp Rept 50-440/87-24 on 871104-09.No Violations or Deviations Noted.Major Areas inspected:871029 & 1103 MSIV Closure Failures,Including Root Cause Safety Significance & Maint History
	ML20148T461
Person / Time
Site:	Perry
Issue date:	01/22/1988
From:	Connaughton K, Eick S, Lanksbury R, Ornstein H, Shaw H, Stefano J, Wright G; NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:	;
Shared Package
ML20148T459	List: IR 05000440/1987024;
References
	50-440-87-24, NUDOCS 8802030312
	Download: ML20148T461 (53)
	v • d • e

.

,

U.S. NUCLEAR REGULATORY COMMISSION

REGION III

Report No:

50-440/87024(DRS)

Docket No:

50-440 License No:

NPF-58 Licensee: Cleveland Electric 111uminating Company Post Office Box 5000 Cleveland, Ohio 44101 Facility Name:

Perry Nuclear Power Plant, Unit 1 Inspection At:

Perry Site, Perry, Ohio inspection Conducted:

November 4 through 9, 1987 NRC Augmented 1nspection Team

'SD k~A

/

Inspectors: Team Leader:

R. D. LhnkTbury t l 2 2h9 (Date)

@D

-

hk Team Members:

K. A.

onnaughton llulst

~(Date)

A d fed S. D. Eick

' /"-/f 8 (Date)

SD-

$,,

$tTL Ornstein t l22lD (Date)

hJ) Isv'

f,s dTSha'

L l13 lD (Date)

fJ. J. $ efamr\\{n LL il 21lO (Date)

,

/

Approved By:

G.T/. Wri ht, Chief

/, J} 6 e Operations Branch (Date)

8802030312 890122 POR ADOCK 03000440

PM

_ _ _ _ - _

_ - _ ___

.

Inspection Summary inspection on November 4 throuch 9,1987 (Report No. 50 440/87024(DRS))

Areas Inspected:

Special Augmented Inspection Team (AIT) inspection conducted in response to the Main Steam Isolation Valve (MSIV) closure failures of October 29, 1987, and November 3,1987, for Perry Unit I and related activities. The review included root cause determination, safety significance, maintenance history, similar previous occurrences, and broader industry implications.

Results: No violations or deviations were identified; however, the licensee has committed to additional and expanded surveillances of the MSIV's and continued investigation efforts to attempt to pinpoint the failure rechanism involved in the slow closure time,

L.

.

..

.. _. -.

_

- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.

EXECUTIVE SUMMARY On October 29, 1987, the Perry Nuclear Power Plant was in the process of completing their Startup Test Program and was perfonning stroke time testing of the Main Steam Isolation Valve (MSIV's) when the inboard valve in the "0" main steam line failed to close within the maximum value delineated in the facility technical specifications.

Two other MSIV's also failed, including the outboard MSIV in the "D" main steam line.

In all cases, subsequent stroke times for these three MS!V's were within acceptable values. The licensee initially declared the MS!V's inoperable. However, based on the acceptable

,

stroke times achieved af ter the second try, later declared them operable. The licensee believed that the failures were the result of impurities in the MS!V actuator control unit and that the impurities had apparently been dislodged and/or expelled during MISV operation.

Plant operation and the Startup Test Program were continued with the stipulation that additional stroke time tests on the MSIV's, to confirm their operability just prior to the perfomance of the full reactor isolation startup test be performed.

On November 3,1987, while performing the additional stroke time testing of the MS!V's both the inboard and outboard MSIV's in the "D" main stea.n line again exhibited unacceptable stroke times.

The licensee reported the "ailure of the two MSIV's to the NRC and commenced an orderly shutdewn. As a result of this event Region !!! dispatched on Augmented Inspection Team (AIT) to the site the following day.

The licensee evaluated potential component failures and from this developed a carefully planned disassembly and troubleshooting progran. As a part of this troubleshooting program the licensee disassembled the MS!V actuator control i

units from the three MSIV's that had previously failed. The results of this disassembly and inspection revealed that the Ethylene Propylene Diene Monocer (EPDM) elastomers contained within the Autocatic Switch Company (ASCO) dual solenoid valves had been significantly degraded by exposure to high temperature and possibly hydrocarbons, An annular dimple was also observed on the seat material and resulted in part of the seat material being extruded into the exhaust orifice. This dimple, together with the deteriorated state of the seat matcrial, indicated that the exhaust seat could be held in an "energized" position even though the solenoids had been deenergized, and would prevent the control air from being exhausted to atmosphere and therefore prevent the MS!V from closing.

The AIT concluced that the rrost probable root cause of the observed MS!V's failure to close on October 29, 1987, and again on November 3, 1987, was a malfunction of the ASCO Podel NP-8323A20E three-way dual solenoid valve caused by deterioration and degradaticn of the Ethylene Propylene Diene Monomer (EPDM)

discs in the ASCO dual solenoid valve due to exposure to a high terperature env i rcr,ce nt.

The high temperature environment was the result of several steam (

leaks in the vicinity of the failed valves. The second most probable cause of the deteriorated and degraded EPDM discs appears to be hydrocarbon intrusion into the valve, or a combination of high temperature and hydrocarbon intrusion.

The licensee subsequently replaced or rebuilt all eight MSIV dual solenoid valves. The plant was restarted on November 13, 1987, and the Startup Test Program, including the full reactor isolation startup test, was successfully ccepleted.

iii l

-

,

.

AUGMENTED INSPECTION TEAM (AIT) REPORT 50-440/87024 Page No.

I.

Introduction

A.

Synopsis of Events 1-2 B.

Augmented Inspection Team (AIT) Fonnation

C.

AIT Charter 2-3 0.

Persons Contacted

!!. Description - MS!Y Slow Closure of October 29 and November 3, 1987 A.

Narrative Description 3-5 8.

Sequence of Events 5-7 III. Failure Mechanism Analysis 7-9 IV.

Investigative Efforts A.

System Descriptions 1.

Instrument / Service Air 9-10 (Portions Pertaining to MSIV's Only)

2.

Main Steam Isolation Valves (MSIV's)

10-12 B.

Evaluation of Safety Significance 1.

Immediate Safety Significance

2.

Other Safety Significance 12-14 C.

Effect of Maintenance Activities 1.

MSIV Maintenance History

2.

Service Air (SA) and Instrument Air (IA)

14-15 Maintenance History 0.

Operations Activities 1.

Operator Response 15-17 2.

Impact of Concurrent Surveillance Activities 17-18 on MSly Performance E.

Troubleshooting Activities and Results 18-22 Y.

Recent Events Involving MSIV Slow Closure / Failure to Close A.

Perry Events

B.

Industry Events 1.

Solenoid Valve Related MSIV Failures 22-25 2.

Solenoid Valves Not Related to MSly Failures 25-26 VI. AIT Conclusions 26-27 VII A!T Recomrendations 27-32 Vill Analysis Plan for EPDM Solenoid Components 32-33 IX Exit Interview 33-34 X

Startup Review A.

Prior to Startup

B.

Following Startup

iv l

.

ATTACHMENTS Attachment No.

Description

Confirmatory Action Letter (CAL)

Augmented Inspection Team (AIT) Charter

Air System P&ID

.

Main Steam Line Isolation Valve (MSIV)

l Cross-Section

MSIV Control Unit Schematic Drawing

Cutaway Drawing of an ASCO 8320 Solenoid Yalve

Sketch of Disk Holder Seated Against the Valve Exhaust Orifice

Restart Authorization l

V l

-

_ _ _ _ _ _ _ _ _ _ _ _ _ _

.

I.

INTRODUCTION l

A.

Synopsis of Event On October 29, 1987, while at approximately 76% power and in the process of corrpleting the Startup Test Program, one of the Main Steam Isolation Valves (MS!V's) in the "0" main steam line (inboard)

at the Perry Nuclear Power Plant, Unit 1, was found to have a stroke J

time greater than the c:aximum allowable value delineated in the facility technical specifications. As a result of this failure, each l

of the other seven MS!V's were tested.

Two of these MS!V's, one in the "B" main steam line (outboard) and the remaining one in the "D" main steam line (outboard) also exhibited unacceptable stroke times.

In all cases subsequent stroking of the MSIV's resulted in stroke tirnes within the technical specification range of allowable values.

Ini.,ially the three MSIV's were declared inoperable, but based upon the inability to recreate the failures and the subsequent satisfactory MSIV performance the licensee declared the MSIV's operable. Based upon discussions between the licensee. NRC Region III, and the Office of Nuclear Reactor Regulations (NRR) management the licensee agreed to perform additional individual fast closure tests on the MSIV's to confim their operability just prior to the performance of the full reactor isolation startup test.

On November 3,1987, while perfoming the MSIV fast closure operability checks, the inboard and outboard MSIV's in the "0" main steam line again exhibited stroke tines in excess of the technical specification maximum value. These two MSIV's were among the three that had exhibited the same problem on October 29, 1987. The MSIV in the "B" main steam line that had failed on October 29 showed an acceptable stroke time during this test.

The licensee reported the failure of the two MSIV's to the NRC and correnced an orderly shutdown.

B.

Augmented Inspection Team (AIT) Formation On November 3,1987, the Perry Senior Resident Inspector (SRI)

inforred Region III that while observing the licensee's perfomance l

of the MS!V operability check in preparation for the full reactor I

isolation startup test, that two MSIV's had again failed to close proper!y.

Subsequent to the report of this event, Region !!! eval-uated the information and detemined that the criteria for dispatching an AIT had been met. Assistance f rom NRR was requested in several specialized areas including air systems and valves.

This assistance was provided by Dr. H. L. Ornstein, Senior Reactor Engineer (AE00).

H. K. Shaw, Senior Mechanical Engineer (NRR), and J. J. Stefano, Femi Project Manager and fomally Perry Project Manager.

In addition, Region III provided expertise in operations and plant raintenance by assigning K. A. Connaughton, SRI, S. D. Eick, Reactor Inspector, and R. D. l.anksbury, Acting Chief. Test Programs Section as Team Leader.

All of these individuals arrived on site on the norning of Noveeber 4, 1987. Concurrent with the AIT activities Region III issued a Confir-natory Action Letter (CAL) RIII-87-019) which was received by the licensee on November 4, 1987.

The CAL confirred certain actions to be taken by the licensee in support of the AIT and also confirmed

- _ _ _

_ - _ _ _ _ _ _ _ _ - _ _ - _ _ _ _ - _ _ _ _ _ - _ - _ _ - _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.

that the plant would not be restarted without the concurrence of the Regional Administrator or his designee.

The CAL is Attachment 1 to this report.

C.

A!T Charter On November 3,1987, a draf t charter for the A!T was far.,.ulated with a list of preliminary questions to be pursued and a list of general areas to be investigated:

Failure of MS!V's to closc/close within Technical Specification

limits.

Safety significance, root cause(s).

Interaction of prior maintenance activities to the event.

Safety implications if actual Group ! isolation signal had

been present.

History of any previous problems.

Broader implications.

Event reporting.

A finalized A!T Charter was issued on November 5, 1987.

This Charter is Attachment 2 to this report.

D.

Persons Contacted Cleveland Electric !11uminating Corpany (CE!)

A.Kaplan, Vice President, Nuclear Group

M. D. Lyster, General Manager, Perry Plant Operations Department (PP00)

F. R. Stead, Directory Perry Plant Technical Departrent (PPTD)

E. Riley, Director, Nuclear Quality Assurance Department (hCAD)

C. M. Shuster Director, Nuclear Engineering Department (NED)

R. A. Newkirk, Manger, Technical Section, PPTD

'V K. Higaki, Manager, Outage Planning Section PPOD

W. E. Coleman, Manager, Operations Quality Section, hQAD

B. D. Walrath, Manager. Engineering Projects Support Section NED

D. R. Green Manager, Electrical Design Section, NED

E. M. Bu:zelli, Manager, Licensing and Cocpliance Section, PPTD

S. J. Wojton, Manager, Radiatien Protection Section, PPTD

K. R. Pech, Manager, Mechanical Design Section, NED

R. A Stratman, Manager, Operations Department PP00 W. R. Kanda, Jr., Manager, Instrumentation and Control Section, PPOD

T. A. 01eksiak, Jr., Lead Supervisor, Maintenar,ce Section. PPCD

V. J. Concel, Lead System Engineer. Technical Section, PPTD

S. F. Kensicki, Technical Superintendent, PPTD

G. A. Dunn, Supervisor, Licensing and Corpliance Section. PPTD K. F. Russell, Shif t Supervisor, Operations Section. PP00 M. W. Gnyrek, Senior Operations Coordinator, Operations Section, PPOD

. _ _ _ _ _ _ _ _ _ _

. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _

_ _ _ _ _ _

i o

.

>

.

J. P. Eppich, Senior Project Engineer, Mechanical Dosign Section, NED G. W. Heffner, Supervisor of Media Relations i

P. J. Arthur, Nuclear Steam Supply System Lead, Technical Section, a

PPTD

'

General Electric

J. J. Sheehan, Operations Manager Automatic Switch Company (ASCO)

l K. Thomas, Sales Engineer

Ralph A. Hiller Company l

J. dancy, Sales Engineer

'

Denotes those attending the exit meeting on November 9,1987,

a In addition to the above, other merr.bers of the Perry staff were contacted by the AIT.

!!. DESCRIPTION - MSIV SLOW CLOSURE OF OCTOBER 29 AND NOVEMBER 3, 1987 i

A.

Narrative Description J

]

On Thursday, October 29, 1987, at about 6:35 p.m. (EST) while Perry Unit I was operating at approximately 761 power, the licensee

)

perfomed a fast closure test of the "D" inboard main steam line

,

-

MSIV (1B21-F00220) as part of Startup Test Instruction (STI)

l 1821-025A, "MSIV Function test". When the control switch fur

1B21-F0022D was placed in the 'CLOSE" position, the valve failed I

to start closing for approximately 18 seconds. At that point the l

valve stroked closed for a total stroke time, including the 18 second a

delay, of 22.8 seconds.

Technical Specification 3/4.4.7 requires i

that the MS!Y's close within a time frame of 2.5 to 5 seconds and

Technical Specification 3/4.6.4 requires that they close within 5 seconds. The licensee wrote a Level 1 test exception (failure to j,

rrect the Level 1 acceptance criteria of the STI) and at 6:42 p o.

reopened 1821-F0022D. At 7:00 p.m. the Unit Supervisor declared

,

1821-F00220 inoperable based upon its slow closure time. The appli-cable action staterrents were then entered in accordance with Technical Specifications Limiting Condition for Operation (LCO) 3.4.7.a and LCO 3,6.4.a..

LCO 3.4.7 a. requires that with one HSIV inoperable, either restore that MSIV to operable status within 8 hcurs or isolate the affected main steam line by closing and deactivating an MSIY in that r.ain steam line. LCO 3.6.4.4. requires that with one containment isolation valve (MS!V) inoperable, either restore thr: valve to operable status within 4 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months or isolate the affected penetration by use of at least one deactivated, closed, valve.

Subsequently, the decision was made to re stroke the IB21-F00220 MS!V.

This was accceplished twice - once at 9:03 p.m. with a resultant stroke time of 3.2 seconds and again at 9:06 p.m. with a resultar.c stroke tine

_ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ - _ _ _ _ _ _

_ - _ _ _ _ _ _ _ - _ _ _ - _ _ _ _ _ _

.

!

.

f J

of 2.9 seconds. Also, subsequent to the failure of the 1821-F0022D MSIV the licensee convened the Plant Operations Review Comittee (PORC)

'

to evaluate the situation.

(

i

At 9:44 p.m. the "D" outboard main steam line MSIV (1821-F00280) was

)

fast closure tested with a resultant closure time of 77 seconds. Again.

[

at 9:52 p.m., MSIV 1821-F0028D was stroked with a resultant closure

'

time of 3.2 seconds. As a result of this second failed MSIV the licensee

!

made the decision to test the remaining MS!Y's. This was accomplished

!

between 9:53 p.m. and 10:20 p.m. and, with the exceptioa of MSIV i

-

1021-F00288, all showed acceptable closure times. The stroke time for

i

'

the 1821.F0028B MSIV, was found to be 11.9 seconds.

A second test of this valve resulted in a closure tine of 3.9 seconds.

i

!

In accordance with Technical Specification LCO 3.4.7.a. and 3.6.4.a.

the licensee isolated the "0".uin steam line at 10:40 p.m.

This

'

was accomplished by closing / verifying closed the 1821-F00280 MSIV, i

the before seat drain valves and the'MSIV Leakage Control System I

Isolation valve. These valves were then deenergized.

l

j The licensee's FORC initiated a review of the situation and concluded

that the MSIV's were operable based on successul stroke time tests l

subsequent to the initial failures.

From the observed MSIV behavior,

!

l the licensee believed that the failures were due to the presence of

!

J impurities in the MSIV ectuator control unit and that the impurities

)

were apparently dislodged and/or expelled during MSIY operation. Based I

on their review, at approximately 11:10 p.m., MSIV's 1821-F00220,

!

1821-F00280, and 1821-F00288 were declared operable.

At 11:40 p.m.

!

the "0" main steam line was restored to an operable status and at

.

12:10 a.m. on October 30, 1987, the licensee made a 4 hour4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months report on

[

l the slow closure of the MSIV's as required by 10 CFR 50.72(b)(2)(iii).

l l

In a discussion between the licensee and NRC management on October 30, i

1987 a concern was expressed to the licensee that while a plausible i

explanation for the MSIV failures had been provided, additional assur-

ances of continued MS!Y operability were warranted pending further

!

evaluation.

To address this concern, the licensee agreed to perform j

additional MS!V stroke time testing prior to the perfomance of the i

full reactor isolation startup test which was then scheduled to be l

perfomed within the following seven days.

On Tuesday, November 3,1987, at about 11:45 a.m. (EST) the licensee i

decreased power to 80% in order to perform the additional MSIV stroke j

time testing. At 11:57 a.m. the 1821-F00220 MS!V was tested with a resultant stroke tire of 18 seconds. Based upon this, the Unit

,

I Supervisor, at 11:58 a.m., declared 1821-F00220 inoperable. Using

!

managerent guidance previously provided this MSIV Was re-stroked at i

11:59 a.m. with a closure time of 3.0 seconds and declared operable

!

at 12:00 p.m. by the Unit Supervisor. At 12:12 p.m. the 1821-F00280 MS!Y was stroked and failed to close in the 2 minutes and 49 seconds

'

that the control switch was held in the "CLOSE" position. The control j

switch was allowed to return to its normal position of "AUT0" and was

then taken back to the "CLOSE" position. The MSIV ther, closed in 3.4 seconds. Even though the MSIV had closed within acceptable limits on j

i

'

.

!

.

<

i the second closure attempt the Unit Supervisor declared 1821-F00280 e

inoperable. Subsequently, at 12:30 p.m. the o'ecision was made to I

declare the "D" main steam line inoperable. The other six MS!V's

!

]

were stroke time tested with acceptable results.

i

.

'

Within the hour following these MS!V failures, another discussion was i

held between hRC and licensee management personnel. During this f

'

difcussion, the licensee infomed NRC management of its intent to

-

increase power and perform the full reactor isolation startup test

!

,

thereby placing the unit in Hot Shutdown within the time limits of l

l Technical Specifications 3/4.4.7 and 3/4.6.4. The licensee was

'

,

infomed that this course of action was considered to be both noncon-l servative and contrary to the intent of the technical specification, j

Under the circumstances, technical specifications intended that an

orderly plant shutdown be conducted to minimize the potential for

1 challenging the inoperable MS!V's.

Based upon this discussion, the l

'

licensee agreed to perform an orderly reactor shutdown. At 1:30 p.m.

l the licensee infomed the System Operation Center of the intended i

j plant shutdown and at 1:37 p.m. ccnmenced a nomal plant shutdown. At i

i 1:53 p.m. and 1:54 p.m. the 1821-r00220 and 1821-FO028D MSIV's.

'

{

respectively, were fast closed, itS!Y IB21 F00220 had a stroke time of I

3.4 seconds and 1821-F00280 had a stroke time of 3.3. seconds. This

[

'

{

was done to conply with Technical Specification LCO 3.4.7.a. and j

I'

3.6.4.a. requirements to isolate the affected line. At 1:55 p.m. the i

licensee made an Emergency Notification System (ENS) notification on f

!

the slow closure times of the MSIV's and on the plant shutdown in

[

j accordance with 10 CFR 50.72(b)(2)(tti) and 10 CFR 50.72(b)(1)(i)A.

l

l I

i j

R.

Sequence of Events and Opustor Actions

'

I

!

At the AIT's request. a chronology of events related to the MSIV l

failures on October 29 and hovember 3,1987, was assembled by the

<

licensee.

The chronology, which included MS!V performance data and

[

l operator actions, was verified to be accurate by A!T personnel j

I through review of operating logs. Technical Specification LCO

tracking systm documentation, interviews with licensee operating f

l personnel, and inspector observation of MS!Y surveillance testing sonducted on Novertber 3,1987.

The chronology was as follows:

[

,

f h0TE: All times are in Eastern Standard Tire.

October 29. 1987

}

l 1835 Stroked MS!Y 1821.F00220 for Startup Test Instruction f

!

(STI)-B21-025A Section 8.3.

Valve did not begin to close I

for 18 seconds. Level 1 Test Exception Report written.

[

'

1842 Re-opened 1821-F00220.

!

1900 Ceclared 1821-F0022D inoperable based upon a total closing

)

tire of 22.8 seconds.

Entered associated LCOs.

,

!

21C3 Re-stroked 1821-F00220 tire to close 3.2 seconds.

)

,

i

[

!

I

.

2106 Stroked 1821-F0022D again - time to close 2.9 seconds.

2144 Stroked 1821-F0028D - time to close 77 seconds.

2152 Re-stroked 1821-F0028D - time to close 3.2 seconds.

?153-Stroked remaining MSIV's. All satisfactory witn the 2220 exception of 1821-F00288.

Found 1821-F0028B had an initial slow stroke time of 11.9 seconds, second stroke was 3.9 seconds.

2240 Isolated "0" Main Steaa Line (lSL). MSIV 1821-F028D deenergized.

2310 All MSIV's were verified to stroke within 3-5 seconds.

Could not repeat the inicial condition causing MSIV to slow close. Based on licensee management review the decision was made to declare all MS1V's operable.

2340 Restored "0" MSL.

October 30, 1987 0010 Made 4 hr. report on slow closing MSIV's in accordan:e with 10 CFR 50.72(b)(2)(iii).

November 2, 1987 1942 Corrmenced Surveillance Instruction (SVI)C71-T0039, "MSL

Isolation Valve Closure Channel Functional" (10% stroke -

partial closure - RPS).

2142 Completed SVI C71-T0039 - Satisfactory.

'

November 3, 1987 1145 Decreased power to 80% to stroke MSIV's.

1154-Stroked MSIV's.

1222 1157 1821 F00220 took 18 seconds to close.

1158 Unit Supervisor dedared 1821-F0022D inoperable.

1159 1821-F0022D restroked in 3.0 seconds.

1200 Unit Supervisor declared 1821-F00220 operable, l

1212 1821-F00280 did not close in the 2 minute 49 seconds that (

the control switch was in "close".

Took switch back to l

"Auto", then to "close", valve shut in 3.4 seconds, l

1212 Unit Supervisor declared 1821-F00280 inoperable.

l

_.

.

1230 Declared MSL "0" inoperable based on repeated failure of 1821-F00220 and 1821-F0028D to stroke in required time.

1330 Informed System Operation Center of intended plant shutdown.

1337 Comenced a normal reactor shutdown.

1353 Closed 1821-F0022D 3.4 seconds.

1354 Closed 1821-F0028D 3.3 seconds.

1355 Made ENS notification on slow closing MSIV's and plant shutdown in accordance with 10 CFR 50.72(b)(2)(iii) and 10 CFR 50.72(b)(1)(i)A.

111 FAILURE MECHANISM ANALYSIS After the second event on November 3, 1987, the licensee convened a team of individuals from various departments including representatives of Gilbert Associates (the architect engineer) and General Electric (GE).

The charter of this team was to develop a list of components whose failure would result in the observed behavior of the MSIV's. After developing this list, the known facts were used to evaluate the probability associated with Of these twenty-four, nineteen (19)y-four each of the potential component failures.

Their analysis yielded twent (24) potential component failures.

were evaluated as unlikely failures, one (1) was evaluated as a moderate proba-bility failure, and four (4) were evaluated as likely failures.

The four likely failures and the one moderate probability failure can be grouped together into a category involving the ASCO dual solenoid valves on the MSIV actuator air control units and the air system feeding them.

The twenty-four potential component failures and their associated probabilities of causing the observed behavior were as follows:

Failure of the Automatic Switch Company (ASCO) Model 8323 three-way dual

solenoid valve (fast closure)

Instrument air system quality

Obstructions / foreign materials in air lines / accumulators

One or both of the solenoid's of the dual solenoid valves for each

of the MSIV's failed to decouple (mechanically separate) upon de-energization

+ Solenoid valve exhaust port blocked Failure of the Norgren two-way control valve Hydraulic speed control failure MSIV internal binding Swagelock fittings improper installation / assembly / leakage Failure of the ASCO Model 8323 three-way solenoid valve (slow closure / test)

Valve packing too tight Failure of the Norgren four-way control valve Valve lineup of instrument air header system Control unit wiring and termination fail'Jre resulting in a hot short Glazed contacts on control and relay components

'

.

Relay failure or incorrect operation resulting in misoperation of the MSIV's Panel control switch failure or misoperation Limit switch settings incorrect or inoperable Mis-wiring for indication of instrumentation or switches Data acquisition failure Procedural error for testing High steam flow /high reactor power interaction Incorrect reassembly and installation of the control unit Actuator binding / stem binding Likely failure

+ Moderate probability failure In conjunction with the above, the AIT also evaluated potential failure modes and concluded that the most probable component failure was the ASCO dual solenoid valve.

In addition, the AIT evaluated the above analysis performed by the licensee and agreed with the methodology and conclusions reached. Subsequent to this analysis the licensee provided a written proposal for troubleshooting the MSIV's to the AIT for concurrence. After evaluation and coment by the AIT, a carefully planned disassembly and troubleshooting program was generated.

The focus of the troubleshooting was to gather more data with regard to the failures postulated as probable or likely. This was accomplished by performing various tests of the air system, including particulate counts, dew point measurement and analysis of air samples for hydrocarbons, and by disassembly of various portions of the MSIV actuator air control units.

A discussion of the inspection process for the control units and corre-sponding results is provided in Paragraph IV.E. of this report.

The licensee evaluated the facts gathered during the troubleshooting program and concluded that they had substantiated their original evaluation that the most probable failure mechanism was the ASCO dual solenoid valve.

The licensee reconvened the original failure analysis team and tasked them with developing a list of potential failure modes of the ASCO dual solenoid valve and the corresponding probability of each of these modes. Their anal sis yielded a total of nine (9) potential failure modes. Of these, one 1) was evaluated as likely, two (2) were evaluated as possible, and six 6) were evaluated as unlikely.

The nine potential failure modes and their associated probabilities of causing the observed behavior are as follows:

Local high temperature has caused deterioration of EPDM seal materials

Blockage of the dual solenoid valve exhaust port with tape Jaming of kinematic components 0xidation of EPDM compound used in the gaskets, seals, and disc seal

+

materials l

Residual magnetism following coil de-energization l

Wrong materials l

Lockseal vapors

+ 0-ring / lubricant interaction

'

Corrosion within solenoid enclosure

--

-- _

,

.

.-

.

Likely failure

+ Possible failure The AIT reviewed the conclusions of the failure analysis team and agreed with their assessment with one exception. The AIT considered that deteriora-tion due to hydrocarbon attack of the EPDM sealing materials within the ASCO dual solenoid was a likely probable cause and that information available did not invalidate this concern. The licensee had obtained an air semple and had it analyzed for hydrocarbons with negative results.

However, this alone did not preclude a previous contamination of the air system with hydrocarbons nor did it preclude an introduction of hydrocarbons from a source upstream of the air supply line such as from the use of non-approved pipe thread sealants or lubricants.

IV.

INVESTIGATIVE EFFORTS A.

System Descriptions 1.

Instrument / Service Air (Portions Pertaining to MSIV's Only).

A drawing of the service air and instrument air system is shown on Attachment 3.

The service air system for each unit consists of one motor driven compressor with an integril intercooler and aftercooler, an air intake filter silencer, lube oil subsystem, filters, condensate traps, controls, a receiver tank and a piping network for distribution throughout the plant. A cross tie header between Perry Unit 1 and Perry Unit 2 is included in which distri-bution connections to the various plant areas are provided. During normal operation, the service air systems for the two units are cross connected with one compressor running and the other in the automatic standby mode.

If the service air system pressure drops below 110 psig the standby service air compressor starts automatically.

Separate instrument air systems are provided for each unit to supply clean, dry, oil free air for control purposes throughout the plant. The system is designed to meet the guidelines of ANSI Standard MC-11-1 (ISA-57.3) with the exception that the maximum allowable particle size for air to safety related equipment is specified to be less than or equal to 40 microns ~

The nomal supply of air to the instrument air system is from the respective service air system for the unit and the instrument air compressor for each unit is used as a backup. The service air compressor is operated continuously to provide a constant output pressure of 125 psig. The instrument air system for each unit also includes an after cooler (integral with the compressor), a receiver tank, a prefilter, an air dryer, an afterfilter, and a piping network for distribution throughout the plant. All instru-ment air leaving the receiver tank passes through the filters and the air dryer. The Unit 1 and 2 instrument air distribution systems are cross-tied.

- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _.

.

If the instrument air system pressure drops below 90 psig the instrument air compressor starts automatically and maintains system pressure in the 90 psig to 100 psig range. A diaphragm operated isolation valve is provided in the air supply line from the service air system. This valve closes automatically when the instrument air system pressure drops below 90 psig and may be manually opened by a switch in the control room when the system pressure rises above 90 psig.

The output of the last downstream afterfilter is directed to numerous places throughout the plant including the accumulators and control units for the MSIV's.

2.

Main Steam Isolation Valves (MSIV's)

Two Main Steam Isolation Valves (MSIV's) are welded in a horizontal run of each of the four main steam line pipes; one valve is as close as possible to the inside of the drywell and the other is just outside the containment.

Attachment 4 shows a main steem line isolation valve.

Each is a 26 inch Y pattern, globe valve. The main disc or poppet is attached to the lower end of the stem. Nomal steam flow tends to close the valve, and higher inlet pressure tends to hold the valve closed. The bottom end of the valve stem closes a small i

pressure balancing hole in the poppet. When the hole is open, it acts as a pilot valve to relieve differential pressure forces t

on the poppet.

Valve stem travel is sufficient to give flow areas past the wide open poppet greater than the seat port area.

The poppet travels approximately 90 percent of the valve stem travel to close the main seat port area; the last 10 percent of valve stem travel closes the pilot valve.

A 45 degree angle permits the inlet and outlet passages to be

!

streamlined. This minimizes pressure drop during normal steam flow and helps prevent debris blockage. The valve stem penetrates

,

the valve bonnet through a stuffing box that has two sets of replaceable packing. A lantern ring and leakoff drain are located

!

between the two sets of packing. To help prevent leakage through the stem packing, the poppet backseats when the valve is fully open.

Attached to the upper end of the stem is an air cylinder that opens and closes the valve and a hydraulic dashpot that controls its speed. The speed is adjusted by a valve in the hydraulic return line bypassing the dashpot piston.

Valve closing time is adjustable to between 3 and 10 seconds.

The air cylinder is supported on the valve bonnet by actuator support and spring guide shafts. Helical springs around the spring guide shafts close the valve if air pressure is not available.

The valve is operated by pneumatic pressure and by the action of compressed springs.

The control unit is attached to the air cylinder. This unit is shown en Attachment 5 ano contains air

a

.

control valves and solenoid operated valves.

Part 4 of Attachment 5 is ti.e main pilot control valve (dual solenoid valve). This valve consists of a valve body with a solenoid attached to either end (see Attachment 6).

The dual solenoid valve provides control air to operate the four-way control valve (part 1) and the two-way control valve (part 3) and is used for opening and for fast closure of the MSIV. When both of the solenoids on the dual solenoid valve are energized the incoming solenoid air supply is directed through the valve body to shift the four-way control valve and the two-way control valve to the open position.

In the open position the four-way control valve ports air through the three-way control valve (part 2) to the underside of the MSIV actuator piston while at the same time venting the over piston area of the MSIV actuator to atmosphere. With the two-way control valve in the open position the exhaust path through it to atmos-phere is closed.

For a fast closure of the MSIV both solenoids de-energize shutting off f'e control air to the fcur-way control valve and the two-way cor ol valve and venting them both to atmosphere. When this occurs both valves will shift to the closed position.

In the closed position the four-way control valve now directs air to the over piston area of the MSIV actuator and vents the under piston area to atmosphere. The two-way control valve now is in the closed position and alsu vents the under piston area of the MSIV actuator to atmosphere.

In this condition the MSIV is closed both by air pressure and by the helical valve springs.

Slow closure capability (used for test purposes) of the MSIV is accomplished through the use of the single solenoid valve (part 5).

When the MSIV is open and the solenoid for the single solenoid valve is energized, air is directed to the three-way control valve (part 2) causing it to shift to the closed position.

In this position the air that was directed to the under piston area of the MSIV actuator from the four-way control valve is stopped and a vent path for the under piston area is opened up through an air metering valve (part 9).

The over piston area is still vented to atmosphere through the four-way control valve.

In this configu-ratkn the air trapped in the under piston area is slowly bled off through the metering valve allowing the MSIV to slowly close.

Remote manual switches in the control room enable the operator to operate the valves. Operating air is supplied to the valves from the Instrument / Service Air System. An air tank (accumulator)

,

i between the control valve and check valve provides backup operating air.

The main steam line isolation valves are designed to close under accident environmental conditions of 330'F for one hour at drywell pressures of 30 psig m6ximum and -14 psig minimum.

In addition, l

they are designed to remain closed under the following post-l accident environmentel conditions:

l a.

330*F for an additional 2 hours2.314815e-5 days 5.555556e-4 hours 3.306878e-6 weeks 7.61e-7 months at drywell pressure of 15 l

psig maximum.

I

o

.

,

b.

310 F for an additional 3 hours3.472222e-5 days 8.333333e-4 hours 4.960317e-6 weeks 1.1415e-6 months at 15 psig maximum.

c.

250*F for an additional 18 hours2.083333e-4 days 0.005 hours 2.97619e-5 weeks 6.849e-6 months at 15 psig maximum, d.

250'F to 100'F ramp during the next 99 days at 15 psig maximum.

B.

Evaluation of Safety Significance 1.

Immediate Safety Significance Based upon the absence of plant conditions requiring an automatic main steamline isolation, the excessive MSIV stroke times did not have insnediate safety significance. Had a main steamline isola-tion been required, isolation of the "D" main steamline may not have occurred within the timeframe assumed in the accident analysis for the Perry plant. The safety significance of such an occurrence is further discussed in the following paragraph, 2.

Op.erSafetySignificance In response to the question of whether or not the accident analysis bounded thc event which occurred on November 3,1987, when both the inboard and outboard MSIV's in one of the four main steam lines failed to close within the 5 second time required in the plant Technical Specifications, the licensee was tasked to perform an analysis to evaluate the safety significance of this event.

There was no additional safety significance attributable to the other MSIV that failed to close since the redundant MSIV in that line closed within the prescribed technical specification values.

The two MSIV's in one main steam line (line "D") that failed to close within the required tire were identified as 1821-F00220 and 1821-F00280.

The 1821-F00220 (inboard) MSIY took 18 seconds to close; tne 1821-F0028D (outboard) MSIV did not close until the valve switch was recycled in the control room (approximately 2 minutes 40 seconds). Both General Electric (GE) and Gilbert Associates (GAI), the Perry Architect Engineer, assisted the licensee in the performance of this analysis.

First GE determined that two accident scenarios and three transients deWribed in the Final Safety Analysis Report (FSAR)

took credit for closure of the MSIVs.

These events were the following:

1) Main steam line break outside containment 2)

Inside containment breaks which cause reactor water level to reach the Level 1

Pressure regulator failure transient

Loss of condenser vacuun transient

Loss of AC power transient The bounding event was determined to be the main steam line break outside containment, since that event would permit the largest amount of activity to reach the site boundary. Therefore, GE was tasked with determining what the mass flow would be for a main steam line break outside containment given the as found

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __

.

O conditions that existed on November 3, 1987 (i.e., three main steam lines isolated within proper times, and the remaining main steam line isolating in 18 seconds).

The analysis was performed using the GE "SAFE 06" Code, an NRC approved Code which had been

'

previously used by Perry in the ECCS performance analyses (FSAR Chapter 6).

It should be noted that the mass release determined by this Code was much less than the mass release discussed in FSAR Section 15.6.4.4 for the main steam line break outside con-tainment due to the conservative assumptions used in the FSAR analysis (assuming that level rise time is 1.0 second; that steam-water mixture quality is a constant 7.0%, and that the system pressure remains constant at 1060 psig throughout MSIY closure).

In addition, GAI was asked to perfonn two additional calculations.

The first calculation considered the mass release given in the FSAR (FSAR page 15.6-10) for the first 5.5 seconds of the event and then using the GE supplied flow data after 5.5 seconds with

.

one main steam line open. The second calculation used the GE supplied data throughout the event.

For each calculation two

i results were determined.

First the postulated amount of radia-tion which would be released in the 18 seconds it took for the 1821-F00220 (inboard) MSIV to isolate on November 3, and second the postulated total time it would take with one main steam line unisolated before 10 CFR Part 100 limits (i.e., Iodine dose of 300 Rem) were exceeded.

It was assumed for these calculations that there would be no plateout or hold up time for the release and that no fuel failure would occur.

conclusions were drawn (g the FSAR mass release the following For the calculation usin EB = Exclusion Boundary):

EB lodine dose with 18 second single MSIV closure - 192 Rem EB Iodine dose with 79 second single MSIV closure - 300 Rem For the calculation using the GE data the following conclusions were drawn:

,

EB Iodine dose with 18 second single MSIV closure - 82 Rem

EB Iodine dose with 120 second single MSIV closure - 300 Rem As shown above for either calculation the slow closure (18 seconds)

of the 1821-F00220 MSIV on November 3 would not have resulted in a release exceeding 10 CFR Part 100 guidelines. Also, depending upon which calculation was used, the plant would have had between 79 and 120 seconds to isolate that line under accident conditions prior to exceeding 10 CFR Part 100 guidelines based on the assump.

tions given previously.

Therefore, the licensee concluded that the 18 second slow closure of the 1821-F00?20 (inboard) MSly had

j been shown to be within the bounds of the accident guidelines, l

The NRR technical staff reviewed the calculations performed by GE and GA! addressing the MSIV slow closure event which occurred on November 3, 1987, in the "0" main steam line, and found the

'

!

,

- - -

- -..

--

, - - _ _ _ _ _ _, _ -... _ -. _ _.. _ _ _ _,,

.

,

.

licensee's assumptions and conclusion that the accident guide-lines of 10 CFR Part 100 would not have been exceeded had a LOCA occurred during the time the inboard MSIV (1821-F00220) rimained open, to be reasonable.

C.

Effect of Maintenance Activities The AIT reviewed the licensee's maintenance history of the MSIV's, the Service Air (SA) and Instrument Air (IA) Systems.

This included: (1)

a review of work orders (WO's) that had been performed on the systems since January 1985, (2) the testing that was performed as the result of these maintenance activities, (3) interviewin the licensee's staff with respect to the maintenance performed, and (g) the existing material

condition of the affected MSIV's and interconnected instrument air as it could affect the MSIV closure functions.

1.

MSIV Maintenance History Approximately sixty (60) W0's were reviewed to determine maintenance history on the MSIV's for the past two years.

Numerous maintenance activities had been performed on the valves in recent months such

as lapping the valve seats, machining valve poppet seats, adjusting limit switch settings and retorquing packing glands.

These W0's

'

were followed up with appropriate post-maintenance testing and acceptable LLRT results.

No anomalies could be seen that could be considered as contributing to the MSIV closure function failure.

During the inspection, the removal of the MSIV actuator control units for the three failed MSIV's was observed and a visual inspection of the other MSIV's was done to assess the material condition and environment these valves were subjected to.

Results

of this inspection are detailed in Section IV.E. of this report.

This work, performed per WO 87-9293, WO 87-9324 and WO 87-9285, was done in an expeditious and efficient manner.

,

2.

Service Air (SA) and Instrument Air (!A) Maintenance History In reviewing WO's it became apparent that a number of air system problems had been experienced over the past two years.

Various air system supplied valves (none related to MS!V operation) were found to have dirt, desiccant, sand and/or rust in them that pre-vented proper valve seating and operation.

Past problems with the quality of the IA system had been attributed to either not meeting the system dewpoint recuirement of -40*F or not meeting the system particulate requirement of no particles greater than 40 microns.

Although the potential for detrimental effects to the MSIV's and associated equipment existed, the licensae indicated that based

!

on a review of the system, that they determined that the contam-ination was apparently insufficient to cause detrimental effects on the MS!V's and interfacing equipment.

Not meeting the dew point requirement caused moisture to be introduced into the air system.

Particulate introduction stemed from the afterfilters being: (1) bypassed (due to inadequate

'

. --. _..

- - _ - - _,

. _ _ _ _ _ _

_

_ - _ _

. _ _ _.. _..

.

. _ _ _

__

_._

_

_ _ _ _

o

!

t

,

.

I procedures)fma(2)overdueforelementchangeout,or(3)the I

repetitive intenance) task for filter change out had been

,

missed due to leaking isolation valves. Also, per the recommen-dation of the vendor, the desiccant had been changed from a

mixture of silica gel and activated alumina to 100% activated

alumina. The silica gel desiccant was found to break down into

[

,

i silica sand and cause plugging of the filter.

This was a main

contributor to the various air system supplied valves not seating l

i properly.

At the time of this inspection the licensee was not performing

routine inspections of the IA system prefilters.

The only i

,

requirement for prefilter change out or possible problem ident-

.

'

l ifier was a high differential pressure (10 psid) alarm across the filter. The differential i

recomended by the vendor) pressure was monitored once per day (a l

With no visual inspections being i

.

i performed there existed a possibility that the filters could j

l develop a hole and that the alarm point of 10 psid would never

!

i be realized. Because Perry's IA and SA systems were supplied by l

lubricated compressors, the systems prefilters had the function

'

of filtering oil or oil aerosols and preventing any form of hydro-carbons from entering the desiccant and ultimately the air system.

a

!

Hydrocarbons have been shown to degrade certain elastomers, such L

j as EPDM, that are utilized in the ASCO solenoid valves on the l

MSIV control units.

l

I Preventive maintenance on the IA system afterfilters was a semi-

[

j annual "repetitive task" that entailed doing a visual inspection (

for degradation. A particulate count (40 micron limit) and a dew

!

i point check (-40*F) were done on a yearly basis with a desiccant

!

visual examination done on a semi-annual basis. To improve the l

quality of the IA system and therefore minimize the potential

!

a l

for introducing hydrocarbons into the air system, the licensee

!

agreed to establish a requirement in their preventive maintenance program which will include periodic replacement of the IA system i

prefilters and semi-annual visual inspections. Dew point and I

,

j particulate sampling of the IA system will continue in accordance I

with the existing plant administrative procedure witn unacceptable i

results being evaluated and system blowdowns being conducted i

!

until satisfactory results are obtained. The implementation of i

new maintenance practices along with the continued dew point and

[

particulate sample should provide the licensee with a reliable

[

j met.ns for determination of the air quality of the IA system.

(

-

y D.

Operations Activities

[

f i

1.

Operator Response l

)

The AIT reviewed the event chronology discussed in aragraph II.B.

l J

against the requirements of the licensee's technica specifications I

i as well as applicable operating and administrative procedures and

!

j determined that actions taken by operating personnel met the

[

requirements.

The AIT also reviewed licensee actions for class-l l

ifying and reporting the MS!V failures to the NRC pursuant to

,

.

!

f

,

4 e

b

-

.- - ---- -

F.

.-

.

10 CFR 50.72. and determined that the events were reported under the appropriate reporting criteria and within the required time-frames. These findings, however, hinge upon the assumption that following the initial MSIV stroke time failures and subsequent acceptable MSIV stroke time tests on October 29, 1987, that the licensee correctly detemined that the affected MSIV's had been restored to operable status.

Based upon the additional MSIV stroke time failures on November 3, 1987, and the root cause(s)

of the MSIV failures identified and discussed in Paragraph VI of this report, the licensee's MSIV operability determination on October 29, 1987, does not appear, in hindsight, to have been well supported.

The AIT reviewed licensee nomal, offnomal, and emergency operating procedures to determine whether or not appropriate guidance was provided for operator response to the MSIV stroke time test failures on November 3,1987.

The following procedures were reviewed:

Plant Emergency Instruction (PE!)-B13. "Reactor Pressure

Vessel Control" Off-Normal Instruction (ONI)-Nil, "High Energy Pipe Break

'

Outside Containment" Plant Emergency Instruction (PEI)-D17, "Radiation Release

Control" System Operating Instruction (501)-821, "Nuclear Steam Supply Shutoff, Automatic Depressurization, and Nuclear Steam Supply Systems (Unit 1)."

Under the circumstances which existed on November 3, 1987, following the MSIV stroke time test failures, operators were provided adequate procedural direction via 501-821 to manually reattempt MSIV closure.

Step-by-step direction was provided for manipulating the MSIV controls to affect fast or slow manual MSIV closure. 'In the event that manual fast closure attempts had not succeeded, operating procedures could have been utilized to manually slow-close the MSIV's in the test mode. With the valves closed in the test mode and the MSIV test pushbutton held in the depressed position, the valves would have remained closed indefinitely, permitting evaluation of available options and, if deemed necessary, the performance of additional actions to secure the valves in the closed position (e.g. shutting down the plant and securing the instrument air supply to the MSIV actuators.)

For circumstances requiring an automatic NSIV closure signal, or where specified plant instruments indicated significant steam leak (s) isolable utilizing one or more MSIV's, the AIT detemined that operators would have been directed, by procedure, to verify and/or close the appropriate MSIV's. Additionally, instruction was provided for responding to the much more serious types of events in which conditions required a main steam line isolation and multiple MSIV failures resulting in unisolable main steam

,

.

line(s). Activation of the licensee's emergency response plan was directed for these more serious types of events.

Based upon a review of the licensee's operator training and requalification program, i.he AIT determined that licensed operators were provided classroom and simulator training in the utilization of PEl-B13, ONI-N11, PEl-D17 and S01-021.

During initial training, operators were provided approximately 100 hours0.00116 days 0.0278 hours 1.653439e-4 weeks 3.805e-5 months of simulator instruction and 80 hours9.259259e-4 days 0.0222 hours 1.322751e-4 weeks 3.044e-5 months of classroom instruc-tion which included plant transients and accidents requiring the use of these instructions.

Training to these and other PEls and ONIs covered entry conditions, immediate operator actions, and supplemental actions.

While the circumstances surrounding the November 3, 1987, MSly stroke time failures did not require entry into these instructions, inspector observation of operator actions during the event indicated that the operators had a good understanding of the opera'ing and surveillance test procedures in use and that procedural requirements were being adhered to.

2.

Impact of Concurrent Surveillance Activities on MSIV Perfomance The AIT reviewed a list of surveillance tests in progress at the time of the MSIV stroke time test failures. The list was compiled by the licensee and verified accurate by review of the list against operating log entries over the timeframes of interest. At the time of the October 29, 1987, MSIV failures, the following surveillance tests were in prorress:

Surveillance Instruction No.

Ti tle B21-T0187-R

"ECCS Reactor Water Level Channel Functional" E22-T0195-C

"ECCS Suppression Pool Water Level High Channel C Functional" E22-T1202

"HPCS Pump Discharge Flow Low Channel Functional" M16-T2001

"Drywell Vaccuum Breaker Isolation Yalve Operability Test" M17-T2002

"Containment Vaccuum Relief Valve Operability Test" At the time of the November 3, 1987, MSIV stroke time test failures, the following surveillance tests were in progress:

- -_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_ _ _ _ _

.

Surveillance Instruction No.

Title B21-T0369-A

"Safety Relief Valve Pressure Actuation Channel Functional" C51-T0026

"APRM Flow Biased Power / Flow Verification" Based upon the root cause(s) of the MSIV failures discussed in Paragraph VI of this report and the review of the foregoing Surveillance Instructions, the AIT concluded that the perfonnance of these surveillances had no bearing on the MSIV failures.

E.

Troubleshooting Activities and Results Af ter the event of November 3,1987, the licensee convened a team of specialists, including representatives of GAI and GE, to determine the potential components whose failure would fit the observed facts. Their analysis yielded twenty-four (24) potential component failures.

Nine-teen (19) of these were evaluated to be unlikely, one (1) was evaluated as a potential failure, and four (4) were evaluated as likely failures.

The four likely failures and the one potential failure all fell into a category involving the ASCO dual solenoid valves or the air system feeding them. This analysis was used in developing a troubleshooting plan.

During the Entrance Meeting on November 4, 1987, the requirements of the CAL (Attachment 1 of this report) were reinforced - specifically that no work was to be performed on the specified components / systems without the concurrence of the AIT team leader.

Subsequent to this the licensee provided a written proposal for troubleshooting the MSIV's to the AIT for concurrence.

After evaluation and comment by the AIT a carefully planned disassembly and troubleshooting program was generated.

In conjunction with the above, the AIT also independently evaluated potential failure modes and concluded that the most probable component failure was the ASCO dual solenoid valve.

To determine the cause of the mis-operation of the MS!Y control systems, three MSIV actuator control units were removed and disassembled. These were the units on MSIV's 1821-F00220 (inboard), 1821-F00280 (outboard),

and 1821-F002SB (outboard). All three units were designed and con-structed identically. The B21-F00280 MSIV was the valve that failed to close until cycled a second time during one of the events, while the other two had not meet the Technical Specification requirements for closure times. Prior to any work on the MSIV's being performed, a visual examination of all eight MSIV's was performed to document the as found conditions.

The material condition found in the control unit connections, air control valves, and the ASCO solenoid valves in the control unit was as follows:

1.

The first control unit to be removed and disassembled was 1821-F0022D.

Prior to removal, the MSIV was opened (ASCO dual solenoids energized) and voltage checks were made to determine

E e

.

the as found conditions. No anomalies were noted.

In addition, air blows were also performed on the MSIV actuator air supply, the solenoid air supply and the MSIV accumulator. These tests included collection of any exhausted material by pillow case, a particle count check, and a dew point check. No negative results were reported for the pillow case air blows and the dew point checks indicated that they were less than -40*F.

The particle count checks were performed by blowing the air through 0.45 micron filter paper.

Several of these were sent to an indepen-dent laboratory for particle size measurement and characterization.

Particle sizes were reported in excess of the 40 micron limit for the instrument air system coninitted to by the licensee.

The licensee noted that because the sampling methodology allowed for potential contamination of the samples from outside air and from handling that these results were indeterminate. The particles collected were characterized into three basic types: white translucent, rust in color, and black metallic. The sample sizes were too small to allow further analysis. After removal of the control unit, it was taken to a work area where it was connected to a regulated (90 psig) nitrogen supply and a test box that allowed the solenoids to be energized and de-energized.

The control unit was then cycled several times.

In each case the control unit functioned per design with no anomalies being noted.

Metallic shavings and a dirt-like substance were discovered at the 1-5/8" inlet port to the MSIV actuator and the swage lock input fitting had deep grooves or etched scratches. The ASCO dual solenoid valve was disassembled and no foreign materials

,

l were found in the valve internals. All body gaskets were flat-l tened, brittle, degraded, and showed evidence of being exposed i

to high temperature. The body gaskets were found to be adhering to the brass valve body and when peeled from the valve body left portions of their EPDM material behind. The B solenoid coil was rusty, apparently due to moisture intrusion.

Both the A and 8

!

coils were meggered and checked for continuity with no anomalies l

noted. The EPDM disc on the solenoid operated disc holder was i

found to be hardened and somewhat deformed. An annular dimple l

was observed on the EPDM disc of the disc holder.

This was l

caused from the disc holder being pushed against the raised

!

(cone-like) exhaust orifice of the solenoid valve body (see Attachment 7) causing the orifice to cut into the seat material.

I l

This resulted in part of the seat material being extruded into the exhaust orifice. This dimple, together with the deteriorated

)

state of the disc material, indicated that the <iisc holder could

be held in an "energized" position even though the solenoid had

!

been de-energized, and would prevent the control air from being exhausted to atmosphere and therefore prevent the MS!V from i

l closing.

2.

The control unit for valve 1821-F0028B was the next to be removed l

and disassembled. When the 1-5/8" stainless steel air supply l

oiping was removed from the control unit, metal filings were discovered on internal threads together with an unknown material.

l l

l

l t

a

.

O This material was later analyzed using Infrared Spectrophotometry (IR), and was determined not to be "Neverseeze" lubricant (commonly used for making up air system joints) and to possibly be "Rectorseal" thread sealant.

However, no evidence of foreign material in the control unit internals was discovered during the dismantling process. When the control unit was removed from the MSIV a

"puddle" of unknown fluid was found in one of the actuator air ports.

Subsequent analysis using IR identified the fluid as silicon lubricant.

Air blows were also perfomed on the MSIV actuator air supply and the solenoid air supply.

These tests included collection of exhausted material by pillow case, a particle count check, and a dew point check.

The results of this testing was similar to that reported above for MSIV 1821-F00220.

When the ASCO dual solenoid valve was disassembled, small amounts of dirt / grease (possibly 0-ring lubricant) mixture and some unidentified particles (possibly metal shavings) were found in the exhaust port and the internal thread of the exhaust and intake ports. Material galling was discovered at the ferrule area in the T-fitting connected to the solenoid valve inlet and the air supply port.

Both the upper and lower cylinder connection ports were smeared with substantial amounts of blackish grease (possibly 0-ring lubricant), but there was no foreign material found inside the solenoid valve assembly or in the pilot air i

line. Neither solenoid A nor B sub-assemblies contained foreign material but all body gaskets (0-rings) were brittle, degraded, flattened, and showed evidence of being exposed to high temperature (per the ASCO representative who inspected them). As with the previous control unit the body gaskets were also found to be

,

!

adhering to the brass valve body and when peeled away portions of the gasket material remained adhering to the valve.

The sole-

'

noid coil surfaces were slightly discolored possibly because of high temperature exposure.

Both the A and B coils were meggered and checked for continuity with no anomalies noted.

Inspection

,

of the EPDM material on the disc holder revealed conditions, i

including the annular dimple, similar to that reported in section 1 above for the 1821-F0022D control unit.

>

,

'

3.

The third control unit removed was for MSIV 1821-F00280. While witnessing the removal of the control unit for MSly 1821-F00288, a member of the AIT noted the presence of a piece of duct tape over the exhaust port of the 1821-F0028D ASCO dual solenoid valve.

'

l This finding was significant in that if the exhaust port is plugged, the MS!V would not be aDie to close. As a result of this finding the remaining MSIV's were inspected but no other similar conditions were noted.

In order to allow testing to detemine if this duct tape contributed to the problems exhibited by the 1821 "00280 MSIV, the AIT instructed the licensee to leave the tape in an undisturbed state.

In addition, it was requested that the valve be tested in the disassembly work area to deter-mine what effect, if any, the duct tape had on the solenoid vahes operation. Af ter its removal and transport to the work

l t

- _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.

. _ _ _ _ _. _ ___ __ _ _ __ ____ __ __ __ _ _ _________ ____ _______.

_ _ - _ _ _ _ _ _ _ _ _ _ _

_ _ _ _ _ _ _ _ _ - _ _

Y

.

area the control unit was mounted on a test rig, connected to a regulated (90 psig) nitrogen supply and a test box.

The control

!

unit was energized, simulating the MSIV being in the open position, and allowed to sit for approximately two hours and fifteen minutes.

This wait period was to allow the solenoids and valve body to heat up to an equilibrium value.

The equilibrium value was approxi-mately 130*F in an ambient temperature of approximately 85'F.

It was hoped that by allowing the valve to sit and heat up that the

original failure could be recreated on the bench. When the control

<

unit was de-energized it worked per design.

It was observed that

!

the duct tape covering the ASCO dual solenoid valves exhaust port I

acted like a flap and lifted away from the port, except for one i

'

point of attachment, and allowed the valve to exhaust to atmo-

,

sphere.

The tape was then removed and examined.

The examination revealed that the tape had been in place for some tine. The tape no longer had the flexibility of new tape and remained in i

its installed shape even af ter removal.

The tape also had i

become so porous that when held up to a light source pinpoint holes could be seen.

In addition, the sticky side of the tape that had not been attached to the valve body had collected dust and dirt.

The AIT concluded that based upon the test performed and the examination of the tape that it had not been a contributor to the nbserved behavior of the 1B21-F0028D MSIV. The licensee's j

investigation into the origin of the duct tape revealed that it had probably been put in place during a previous maintenance outage as a cleanliness barrier.

The material condition of the control unit air connections and

.

the ASCO dual solenoid valve was similar to the condition found in the two earlier ones but to a different degree.

It appeared that high temperature had caused a more severe degradation of (

i 1821-F00280.

.

Other valves in the 1821-F00280 control unit were then disassembled.

Small amounts of dirt and some metallic particles or shavings were

'

found inside the air control valves, but no foreign matter was i

!

found in the dual solenoid valve. With the exception of the ASCO l

dual solenoid valve, the operability of the control unit was

,

!

believed to be unimpaired by these small particles of contaruination.

!

WithrespecttotheASCOdualsolenoidvalve,thoughnoforeign

,

matter was found inside of the valve, the failure of the MSIV s t

'

due to this could not be totally eliminated since the foreign material could have conceivably been blown out of the exhaust port during subsequent operation.

The licensee evaluated the data gathered as a result of the trouble-

-

shooting program and concluded that the root cause of the failures

'

!

of October 29, 1987, and Novedor 3,1987, was a failure of the l

respective MSIV's ASCO dual solenoid valve.

This failure was l

attributed to the hardening and dimpling of the EpDM seat material as the result of exposure to a local high temperature environment caused by steam leakt in the vicinity of the control units. As part of their corrective action the licensee rebuilt some of the l

ASCO dual solenoid valves.

Inspection of the remaining solenoid l

.

valves that were disassembled indicated that their material con-dition was significantly better than the three that had been installed on the MSIV's that had failed.

Seat impressions were noted on the valve seat, however, the dimpling condition evident on the other valves was not on any of these valves.

V.

RECENT EVENTS INVOLVING MSly SLOW CLOSURE / FAILURE TO CLOSE A.

Perry Events The AIT reviewed MSIV fast closure stroke time test results for MSIV testing conducted since operating license issuance and prior to October 29, 1987. These test results included tests conducted to satisfy technical specification surveillance test requirements and startup tests involving MSIV closure.

Based upon this review, a total of 78 individual MSIV fast closures were identified. Two instances were identified in which individual MSIV's exceeded their maximum allowable stroke time.

One occurred following the loss of offsite power startup test conducted on May 10, 1987, when MSIV IB21-F00288 closed in 5.1 seconds; the second occurred during surveillance testing conducted on August 10, 1987, when MSIV 1821-F0028C closed in 5.3 seconds.

Following each of these occurrences, adjustments were made to the HSIV fast stroke speed controllers and the valves were retested with satisfactory closure times (4.0 - 4.6 seconds).

While MSIV IB21-F00288 was among those MSIVs which failed on October 29, 1987, the AIT could not determine whether the earlier failure on May 10, 1987, was due to the same root cause(s). The small magnitude by which the stroke time was exceeded on May 10, 1987, did, however, suggest that the failures were not related.

The root cause of the failure of MSIV 1821-0028C on August 10, 1987, appeared to be more clearly unrelated to the October 29 and November 3, 1987, MSly failures. Aside from the fact that this valve did not experience subsequent failures, inspection of the ASCO dual solenoid valve internals for MSly 1821-F0028C showed little, if any, degradation.

!

The licensee had experienced additional failures of solenoid valves similar to the ASCO dual solenoid valve used on the MSIV control unit.

i On July 30, 1986, (LER 86-030) the licensee reported the failure of

a similar ASCO dual solenoid valve in the containment vacuum relief system. The valve (model 8-HB-8320 A9) was found to be "leaking air due to an accumulation of dust in the valve seating area." The LER noted that "a similar solenoid valve connected to the same airline in the same panel was then inspected in order to determine if it was

'

experiencing a similar problem.

No further problems were identified."

A review o-! Work Order No. 860010560 by the AIT showed that the

!

adjnent valve had been removed, cleaned and rebuilt.

The wcrk order indicated that the maintenance staff had found small amounts I

of black dust in the body of the seeand valve.

B.

Industry Events 1.

Solenoid Valve Related MSIY Failures

___ _____ _ ________________-____-__ __

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_ _ _ _ _ _ _ _ - _ _

_ _

- _ _ _ _ _ _ _ _ _ _ _ _

.

There have been a multitude of solenoid valve failures at U.S.

nuclear power plants. With regard to solenoid valves used for MSIV closure there have been several dozen failures. Some of i

these events are reported below along with descriptions of notifications that the NRC provided and a discussion of the actions taken by the licensee in response to those notifications.

The following failures occurred between 1970 and 1980:

Dresden-2 9 failures Hatch-1 5 failures Hatch-2 1 failure Haddam Neck 1 failure Lacrosse 1 failure Millstone-1 3 failures Monticello 3 failures Nine Mile Point-1 2 failures Oyster Creek 3 failures Peach Bottom-3 2 failures Pilgrim-1 3 failures Quad Ci t'.es-1 5 failures Quad Cities-2 2 failures Trojan 1 failure Vermont Yankee 4 failures Zion-1 2 failures Zion-2 4 failures These failures were reported in NRC Inspection and Enforcement (IE) Circular 81-14 "Main Steam Isolation Valve Failures to Close",

November 5, 1981. The circular recommended that holders of con-i struction pemits: 1

"Evaluate MSIV control system design in light of both successful and unsuccessful industry experience";

2 - Consider design changes where appropriate to ensure high reliability and to minimize or eliminate the comon-mode failure i

potential present in current designs."

No written response to the circular was required and the AIT is

!

unaware of any action taken by CE! as a result of it.

IE Infumation Notice (!N) 86-57, "Operating Problems with Solenoid Operated Valves at Nuclear Power Plants", July 11, 1986, presented information about a September 27, 1985, event at Brunswick-2 in which 3 out of 8 MS!V's failed to fast close. As with the event

,

at Perry, two of the MSIV's were in one main steam line. The event at Brunswick was suspected to most likely have been caused by hydrocarbon contamination of the air system and high ambient temperature conditions, degrading the Ethylene Propylene Diene l

Monomer (EPDM) valve seating and seal material. The information l

notice stated that Brunswick was replacing the solenoid valves l

EPDM internals with Viton since EPDM is unsuitable in air systems that are not designed to "oil free" specifications.

EPDM absorbs r

l hydrocarbons resulting in swelling and loss of mechanical properties.

The information notice discussed Viton's superior high temperature

perfomance when compared to EPDM.

Viton's disadvantage was also

'

r i

.

!

.

!

l noted, i.e., it is less resistant to radiation than EPOM (by a factor of 10).

IE IN 86-57 noted ASCO's recommendation that Viton be used for applications where radiation levels do not exceed 20 megarads.

The infonnation notice also addressed chloride contamination of other MS!V solenoid coils, and the failure of several scram discharge valves at Brunswick which were caused by excessive amounts of silicone lubricant. CEI reviewed IE IN 86-57 and determined that no action was necessary because:

(a) The use of Viton seals vs. EPCM had already been investigated

,

'

at Perry, and the use of such seals was consistent with Perry's Equipment Qualification (EQ) Program requirenents.

The original ASCO dual solenoid valves that were used for Perry's MSIV's had Viton seals, seats an! gaskets, but were changed to EPDM because of EQ concerns.

(b) The licensee's maintenance program ar.d adherence to ASCO's installation and maintenance instructions were expected to

,

'

prevent the problems noted in the infonnation notice, i.e.,

"high temperature ambient conditions, inadequate maintenance

'

program on short-lived components, and the excessive use of l

lubricants during maintenance." The response also noted that

'

oil free air is used at Perry and that there is no danger of l

hydrocarbon buildup. Upon contacting General Electric (GE)

!

the licensee was informed that GE believed the problems dis-l cussed in IE IN 66-57 were due to hydrocarbon contaminants j

and not high temperatures; that the EPDM materials used in the MSIV's passed high temperature and radiation EQ testing; and that the unit at Grand Gulf (which had similiar ASCO

!

solenoid valves) had not experienced any problems due to high temperatures.

In addition, GE recommended that the i

air system used by Perry be designed to oil free specifica-tions thus eliminating the possibility of hydrocarbon contaminants.

IE IN 85-17 and 85-17 Supplement 1. "Possible Sticking of ASCO Solenoid Valves" March 1, 1985, and October 1, 1985, described the failure of ASCO solenoid valves which resulted in the failure of three (3) MS!V's at Grand Gulf Unit 1 to fast close.

Those solenoid valves had Viton seals, seats, and gaskets. ASCO attributed the failures to high-temperature sticking which resulted from a foreign substance which collected at the lower core / plug nut interfaces.

Definitive identification of the foreign substances were not accomplished due to the small amounts of material that was collected.

GE recomended that Grand Gulf replace the potentially contaminated MS!V solenoid valves and periodically examine and clean them.

Subsequently, Grand Gulf replaced all 8 of the solenoid valves with similar environmentally qualified ones having EPDM (as opposed to Viton)

seals, seat and gaskets.

The EPDM valves did not stick when subjected to the same conditions that caused the Viton valves to stick.

.

CEl's followup determined that no action was required since there were no solenoid valves with Viton used in safety-related applications at Perry; and the ASCO solenoid valves used for the MSIV's were qualified and had EPDM internals rather than Viton, 2.

S_olenoids Valves Not Related to MSIV Failures IE IN 80-11 "Generic Problems with ASCO Valves in Nuclear Applications Including Fire Protection Systems," March 14, 1980, discussed the problems of having oil in contact with EPDM parts which are internal to ASCO solenoid valves. The notice stated that there is a potential for failure of solenoid valves having EPDM internals due to traces of oil from oil based thread lubricants and traces of oil from instrument air compressors.

The infortna-tion notice cited ASCO's recommendation that EPDM elastomers found in EQ qualified ASCO NP-1 solenoid valves be replaced with Viton kits. Attached to the information notice was a letter from EG&G Idaho, Inc. which described fifty failures of solenoid valves, citing common mode failures due to oil or other foreign material in the air supply system.

In addition, it noted that 18 percent of the failures found were caused by high temperatures and humidity resulting in electrical failure. The licensee's response to IN 80-11 indicated that similar Class 1E qualified ASCO solenoid valves (NP-1) having EPDM would be rebuilt with Viton kits. Similarly certain ASCO 8320 solenoid valves would also have the EPDM replaced with Viton.

The licensee's review package for IE IN 80-11 also included an ASCO service bulletin on the subject (dated April 1,1980).

That bulletin stated that "If pipe thread sealant is properly applied, and if ASCO NP-1 solenoid valves are properly installed in an oil free instrument air system, there should be no need to replace the ethylene propylene elastomeric parts with Viton kits.

If there are traces of compressor oil in the system, steps should be taken to eliminate it, to prevent damage to other components in the system."

IE IN 81-29 "Equipment Qualification Testing Experience "

September 24, 1981; and IE IN 82-52 "Equipment Environmental Qualification Testing Experience - Updating of Test Sunnaries Previously Published in IN 81-29," December 21, 1982, discussed problems with ASCO solenoid valves in which Viton elastomer seals deteriorated under high radiation exposure.

IE IN 82-52 recommended that licensee's should review their system require-ments for Viton compatability in view of ASCO's recornendation that Viton seals should not be used in applications where expo-sures are in excess of 20 megarads (EPDM being the recommended replacement for Viton).

The licensee's review of these infomation notices for applicability to the MSIV control unit's noted that Perry's ASCO solenoid valves already had EPDM seals and, therefore, the information notices were not applicable.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_ _ _ _ _ _

.

c

.

i IE IN 84-23 "Results of the NRC Sponsored Qualification Nethodology

Research Test on ASCO Solenoid Valves," April 5, 1984, highlighted the fact that two ASCO solenoid valves which had undergone natural aging had failed EQ tests. The valves were heated in an air oven at 140'F for three years. The valves were pressurized with nitro-gen and the coils were continuously energized. One of the failure mechanisms involved was the sticking of the EPDM to the valve's metallic parts. The failure of the other naturally aged valve was attributed to the accumulative degradation of the EPDM diaphram.

The licensce's review of this information notice focused upon the fact that the MSIV control unit contained different ASCO solenoid valves (NP-8320 and NP-8323) which were fully qualified in accor-dance with testing perfomed by GE. As a result the licensee concluded that the information presented in IN 84-23 was not applicable to their MSIV control unit.

t Infortnation Notice 85-08, "Industry Experience on Certain Materials I

used in Safety-Related Equipment" dated January 30, 1985, addressed

the environmental qualifications of ASCO solenoid valves having Viton and EPDM parts in addition to addressing the use of elas-j tomers and epoxy coatings in personnel air locks, hydrogen recombiners and oil storage tanks.

The information notice stated the conditions under which the NRC considered Viton and EPOM to be environmentally qualified.

,

The licensee's review of the information notice (relating to the

'

MSIV contiol unit) noted that all of the valves in the control

'

unit contained EPDM parts, and the valves were fully qualified in a:cordance with GE equipment qualification report NEDC-30800.

I As a result the licensee concluded that no action was required

,

in response to the information notice.

l

'

VI. AIT CONCLUSIONS The rust probable root cause of the observed MSIV's failure to close on I

October 29, 1987, and again on November 3, 1987, Was a malfunction of the ASCO Model NP-8323A20E three-way dual holenoid valve caused by deterioration

,

and degradation of the Ethylene Propylene Diene Monomer (EPDH) discs in

!

the ASCO dual solenoid valve due to exposure to a high temperature environ-

[

<

ment.

The high temperature environment was the result of several steam

leaks in the vicinity of the failed valves. The second most probable

cause of the deteriorated and degraded EPOM discs appears to be hydrocarbon l

<

intrusion into the valve, or a combination of high temperature and hydro-r carbon intrusion.

i All evidence collected during the investigation indicated that the event l

was probably caused by the failure of the ASCO dual solenoid valve to shift

'

i to the de-energized position. The evidence collected included the following:

f f

a.

The MS!V's in question stuck open during one comand, but closed I

l within the Technical Specification requirement in responding to

!

the next comand, j

'

j

!

-..

..

_ _ _ _ _ _ _ - _ _ _ _ _

_ _ _ _ _ _ _ _ _ _ - _ _ _

- _ _ _ _ _ _ _ _ _ -

__

_ - _ _ _ - _ _ _ _ _

.

b.

The design of the control unit is such that the simultaneous failure of more than one of the air control valves would be required to cause

the observed failures,

,

c.

The EPDM disc on the solenoid operated disc holder in the MSIV's in question was found to be hardened and somewhat deformed.

d.

An annular dimple had formed on the EPDM disc.

This dimple, together with the deteriorated state of the disc material, indicated that the disc holder could be held in an "energized" position during the

!

de-energizing comand, and would prevent the control air from being exhausted to atmosphere and, therefore, prevent the other air control valves from shifting to the proper position to vent the underside of the MSIY actuator piston to atmosphere and to port control air above the HSIV actuator piston.

The EPDM disc material was qualified for service temperatures up to 140 degrees Fahrenheit using clean, dry air.

From the state found on the disc, it is suspected that the qualified service limits of the L SM material may have been exceeded.

Plant records showed that steam, at temperatures of 300 degrees Fahrenheit or higher, leaked from the 1821-F00220 MS!V during September 1987 and from leakage c]ntrol system valves in the vicinty of the 1821-F0028B and 1821-F00280 MSIV's in early 1987. However, the evidence was not conclusive enough to determine whether this deterioration was caused by the high steam temperature alone, by the interaction of EPDM and hydro-carbons released from the instrumentation air system, or by the action of both.

The AIT also concluded that while the licensee was very responsive after the event of November 3,1987, and proceeded in a methodical, well thought out, manner in determining the probable root cause, their lack of action in starting to formulate a troubleshooting program prior to the second failure, to validate their theorized failure mechanism, was not as conser-vative as the NRC would If ke to see.

In addition, the licensee's plan af ter the November 3,1987, event to increase power and perfom the full reactor isolation startup test, thereby place the unit in Hot Shutdown, is considered by the A!T to be both nonconservative and contrary to the intent of the technical specifications.

VII. AIT RECOMMENDATIONS A.

Failure Mechanism Investigation *

The AIT recommends that in order to more definitively determine the root cause(s) of the ASCO dual solenoid valve failures the licensee should have sophisticated laboratory analysis performed on the solenoid valves from the MSly control units that failed, parts which were removed from the solenoid valves which were rebuilt, and the air samples which were taken from the inlet and outlet lines from the control units.

Some of the possible contributors to, or root causes of, the solenoid valve malfunctions which could be revealed by such analysis (in addition to high temperature, or possibly pointing towards synergism with high temperature) are:

-

_ _ _ _ _ _ _ _ _

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

l l

.

1.

Impurities in the instrument air, such as:

a.

Hydrocarbon from:

(1)

the service air compressors.

(2)

temporary air compressors which were used in containment (July / August 1987),

(3)

pipe threading materials in the air system, (4)

improper lubricant or excessive lubricant on the solenoid valves from manufacture, installation or maintenance, b.

Desiccant from previous air system malfuntions (incorrect filter installation) or mis. operation (bypassing of the filters),

c.

Ofrt, shavings / particles, pipe sealant, weld or soldering debris from incorrect installation or maintenance operations (e.g., pipe threading, gasket, seats, sys tem).

d.

Dirt, scale. oxides, etc. from the manufacturing of the air system components or from subsequent corrosion or surface oxidation of air system components, air Moisture from the air system; e.g. temporary (July /

e.

compressors which were used in containment August 1987), or moisture intrusion from the environment; e.g. steam leaks, etc.

2.

Inadequate cycling:

ASCO recomends cycling the solenoid valves to prevent sticking. ASCO Bulletin 8003, "Installation and Maintenance Instructions" notes the following:

"Preventive Maintenance a.

Keep the medium flowing through the solenoid operator or valve as free from dirt and foreign material as possible, b.

While in service, the solenoid operator or valve should be operated at least once a month to insure proper opening and closing" 3.

Aging of elastomers:

possibility that the elastomers are breaking d >n due to excess age.

Shelf life information does not appear to be readily available regarding ASCO solenoid valves and rebuild kits.

It is known that the elastomers used in many ASCO solenoid valves have short shelf lives - however, it did not appear that the ASCO solenoid valves and rebuild kits at Perry had records or caution labels noting any limitations in this regard.

.

B.

ASCO Design The AIT recomends that in view of the fact that there have been a large number of ASCO solenoid valve failures in safety systems, and that 1) the fdlure mechanisms have not been fully understood, and 2) the design alerances, design characteristics, design calculations and operating.aargins of the solenoid valves have not been made avail-able to the licensees or the NRC, that NRC should take actions to obtain in depth design information from ASCO needed to assure satis-factory operations of such valves (e.g., ASCO has not responded to the question of what is the maximum air stream particle size the solenoid valves can handle).

C.

Potential Generic Technical Specification Deficiencies The AIT recomends that the issue'of rapid repair of steam leaks and

'

the avoidance of high localized temperatures which can lead to degrad-ation and failure of seemingly qualified safety equipment should be i

addressed by the NRC. The technical specifications and LC0's regarding

containment and steam tunnel temperatures may require modifications.

The existence of steam leaks at the Perry plant prior to October 29, i

1987, is believed to have been one of the initiating events of the failures of the MSIV's to function properly. The technical specifica-

,

tions at most plants are predicated upon gross averages of containment and steam tunnel temperatures without consideration of localized high

,

temperatures.

'

I D.

Equipment Qualification Testing *

The AIT recommends that work be done by the licensee and NRC to assure that EQ testing properly accounts for normal plant operating conditions (including normal operation of equipment), anticipated transients or equipment malfunction, design basis accidents, and combinations thereof.

!

The direct applicability of the current EQ tests to the actual plant i

operating and accident conditions is suspect.

EQ testing of the MS!V

control unit solenoid valves entailed a 1000 hour0.0116 days 0.278 hours 0.00165 weeks 3.805e-4 months elevated temperature test.

That test included cycling the valves once every 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months .

Such l

a test is not indicative of a long period high temperature soak as may l

have occurred at the Perry plant prior to the October 29 failures.

The cycling during the EQ testing would minimize the likelihood of

valve failure due to disk to seat sticking, as evidenced by proper

valve operation after the initial failures.

]

E.

, Instrument Air System Deficiencies The AIT recormends that plant specific instrument air system j

deficiencies at Perry, at noted below, should be corrected:

!

1.

System Design Criteria l

It is recomm nded that the licensee review the instrument air

!

system, and the components which are cependent upon it and take action to assure that there is a match between the needs of the

components and the quality of the instrument air delivered to

l

i

!

_ - -, - --_- - __ --.. _. _. -

- - _ _ - - _ _

. _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

-

.

the components.

In addition, the plant specifications and the FSAR should be modified to assure compatibility between the system and the components / systems which use and/or depend upon instrument air.

The instrument air system was originally supposed to meet ANSI standard MC 1.11 - 1976 (ISA-57.3) "Quality Standard For Instrument Air" which limits particulate size to 3 microns. However, the NRC granted the Perry plant relaxation from the ANSI /ISA 3 micron requirement to 40 microns, based upon General Electric document 22A2537-Revision 2 "Field Cleaning and Cleanliness of Nuclear Power Plant Components," November 1979, which defines instrument quality air as "compressed air dried to a dew point of -40*F at the supply pressure and passed through an oil trap and a less than or equal to 50 micron filter to remove oil and foreign particles."

The 40 micron requirement is not consistent with the needs of all

j equipment using the instrument air (e.g., air compressor seal air system has a maximum allowable particle size of 10 microns at

Perry - see item 3.c. below).

j 2.

Air System Degradation by Use of Temporary Compressors l

!

The AIT recommends that the licensee institute appropriate controls i

to ensure that the quality of the air in the air system, and therefore the air operated components it supplies, is not degraded i

by the use of temporary compressors.

It is possible that air operated components may have been degraded at Perry during periods in which the containment air system was isolated and a temporary portable air compressor was used. The presence of particulates

!

or impurities introduced by such activities would probably not l

show up in "air blows" months later. However, the presence of such contaninants could have caused degradation and malfunctions

,

of air-operated equipment several months after their introduction

to the system.

3.

Inadequate Maintenance and Surveillance Testing *

i The AIT recomends that the licensee review each of the following

'

i issues and take appropriate action:

a.

There was no prefilter inspection program (at the time of this inspection there was no comitment to inspect the prefilter until the pressure drop across it became excessive-10 i

psid). A blown or incorrectly installed prefilter would not be identified by this criteria - this could be an especially important deficiency in view of the fact that l

the prefilter is the primary defense against hydrocarbnn l

intrusion from the compressor or intake, b.

Desiccant column inspection is inadequate - semiannual

!

surveillance involves visual inspection of desiccant in the

!

column. The inspection is limited to viewing the desiccant l

on the very top of the column.

However, the material which is observed is the most recently added material which may

_ - _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - - _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _

.

have little resemblance to the older and possibly degraded desiccant which is below it; in addition, there did not appear to be a firm comitment for providing desiccant i

column change out.

c.

In accordance with the manufacturer's data, the maximum allowable particle size for the air compressors' seal air system is 10 microns. Failure of the seal air system could result in the intrusion of oil into the instrument air system. Consequently, the presence of particulates in the

'

instrument air system in excess of 10 microns has the potential for degrading the compressor's seal system

'

thereby leading to gross contamination of the instrument air system.

(Such a contamination coupled with a blown or improperly installed prefilter could cause major air system problems.)

d.

Dew point is noted daily near the dryers, with an instrument air sample being drawn anr.ually from downstream of the afterfilter.

The sample's particle count is also taken,

i however it is not checked for hydrocarbons or other contam-inants.

It appears that no testing is done to check for

'

hydrocarbons or specific contaminants in the air system.

'

e.

The licensee's acceptance of instrument air having particulates i

in excess of component (vendor) design requirements indi-

,

cated their lack of understanding of the problem; e.g., in a November 9, 1987 letter (PY-CEI/0!E-0288L, Edelman to

'

Davis), the licensee stated that "very small quantities of

,

particles greater than 40 microns were identified which i

indicates acceptable air system quality.

Therefore, it is

,

a very low probability that the particles had an adverse affect upon the solenoid valve operation."**

Safety Accumulators i

The AIT recomends that the licensee review MSIV surveillance I

testing (and other testing involving accumulators as applicable)

i for adequacy. The surveillance testing of the HSIV's do not test the adequacy of the backup safety accumulators.

The safety

,

j related check valves are not tested frequently to assure their operability upon a loss of instrument air. Accumulator pressures

are not monitored, therefore, a malfunctioning check valve is not readily detectable.

(Although this inspection was confined to MSIV accumulators, it is believed that accumulators for other safety systems at Perry nay have similar deficiencies.)

>

Note:

As described in IN 85-17 and its supplement, other BWRs have experienced sticking of safety related ASCO solenoid valves; similiar failures at Grand Gulf were attributed to failures due I

to "nicroscopic particles" which were found in the valves.

The licensee's acceptance of a less than desirable air system and their lack of adequate attention to NRC generic comunications is of concern to the NRC.

31

!

__ _ _ - _ - _ _ _ _ _

_ _ _ _ _ _ _ _ _ _ _ _ - _

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.

!

.

o l

F.

Potential Generic !ssue Infomation Dhsemination The AIT recommends that in the short term an infomation notice be issued to alert the industr to the more current failures and what

kinds of failure mechanism (y) are postulated to exist, what should be s

looked for if solenoids are disassembled, and a recommended testing program that can be used to help detect failures prior to these occurring during an actual transient.

In the longer tem, a Bulletin i

should be conside:ed to require specific actions by licensee's to

!

nitigate future failures if further information indicotes the speci-l fic actions to be taken.

Issuance of this Bulletin should be held j

in abeyance until further information can be gathered regarding the

'

failure mechanism (s) so that adequate corrective actions can be t

developed.

In addition, consideration should be given to alerting

industry self improvement groups (such as INPO) that industry (

initiative needs to be taken to resolve this issue.

As described in section V.B. of this report there have been numerous

failures of ASCO solenoid valves over the past 15 to 20 years. At i

various times during this period the NRC has issued several foms of comunications to alert the industry to these potentially significant failures. However, as evidenced by the fact that these failures

,

continue to occur, it appears that the industry has not been aggressive in correcting the problems.

Items A, D and E.3., above were discussed at an NRC/CEI meeting on November 10, i

i 1987.

CEI management indicated that the faulty maintenance and surveillance practices would be corrected and that a test program would be implemented.

'

VIII ANALYSIS PLAN FOR EPDM SOLEN 0ID COMP 0NENTS

,

!

i After completion of the licensee's troubleshooting program and evaluation

,

of the data collected, the licensee proposed a number of corrective actions

that they intended to implement (reference letter PY-CET/0!E-02896, dated

,

11/13/87, Edelman to Davis). Among these was an analysis plan for the EPOM

,

'

!

solenoid components. This plan entailed chemical analysis of the removed

,

'

elastomer materials at a molecular level to determine if changes had occurred from its original state. The plan also entailed a comparison i

of the physical properties of the removed elastomer materials to that of l

,

j new materials to detemine the extent of degradation and reduced perfomance.

l In developing this plan the licensee utilized current industry experience i

with ASCO solenoids, including a similar event at Brunswick in 1985, to

>

provide guidance.

The data gathered from this analysis plan combined with

other industry experiences would be utilized to determine a final root l

'

cause for the previous events.

The following is an outline of the analysis program:

I A.

Samples

'

j j

1.

Unused elastomer gasket material J

2.

Used elastomer from pilot solenoids which did not fail

3.

Used, degraded elastomer material from failed AV O dual

!

solenoids

4.

ASCO dual solenoid valve bodies with elastomer residue

i t

l

I t

i

.

.-

_

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - - _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

__ __-

__

.

B.

Physical Testing i

1.

Profilimetric analysis to compare indentations in EPDM discs t

(sample nos. 2 and 3).

2.

Optical Microscopy to determine the presence of foreign material, or loss of material from surfaces.

3.

Hardness testing to compare with original specifications.

4.

Compression set to compare with unused material and note

performance degradation, j

C.

Chemical Testing

)

'

1.

Infrared Spectrophotometry survey to detemine carbonile content.

This will provide information about the mode of attack (organic

~

acids from the presence of hydrocarbons) and extent of

,

oxidation.

2.

Scanning Electron Microscopy /X-Ray Dispersion Spectrometry to

,'

confirm or negate copper-catalyzed accelerated oxidation (which was a postulated failure mode at Brunswick.)

D.

Environmental Testing Six new dual coil solenoids will be sent to a laboratory for

additional environmental testing.

The solenoids will be placed i

,

in three separate environmcntal chambers (two per chamber) at

'

various elevated temperatures in an energized condition.

The solenoids will remain energized for predetermined times in an

attempt to determine the temperature and continuously energized time at which the solenoids do not perform their function.

The test duration has been set at 92 days.

I The licensee's proposed schedule for the completion of the above is that c

Item C.1 would be complete by the end of January 1988, with the remaining

s items to be complete by the end of March 1988. The licensee also

.

'

i comitted that a test plan for Item D would be provided by November 23,

!

)

1987, and that interim test results would be provided as they become

!

available.

l The AIT reviewed the proposed plan and found it acceptable. As noted in Section V of this report, the most probable failure mechanism of the ASCO l

dual solenoid valve is the deterioration and degradation of its EPDM

!

i components due to high temperatures.

The second most likely failure

!

mechanism is the same as the first but with hydrocarbon interactird with

!

the EPDM cceponents as the cause of the deterioration and degradation. A

cocbination of both of these failure mechanisms is also a possible cause.

!

The results of the above program should validate which of these failure i

'

mechanisms was the root cause of the ASCO dual solenoid's inability to shift to the de-energized position.

!

f IX. EXIT INTERVIEW l

The AIT net with licensee representatives (denoted in Paragraph !.D.)

i informally throughout the inspection period and at the conclusion of the inspection on Novenber 9,1987, and sumarized the sccpe and findings of j

'

the inspection activities.

I I

i

'

i f

-

_ _ _ _ _ - _ _ _ _ _ _ _ _ _ _

_ _ _ _ -

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_ _ _

_ _ _ _.

p.

.

'

.

,

I The AIT also discussed the likely inforfrational content of the inspection report with regard to documents or processes reviewed by the inspectors during the inspection.

None of the areas expected to be contained in the report were identified by the licensee as proprietary. The licensee acknowledged the findings of the inspection.

,

.

X STARTUP REVIEW

'

1

.)

,

On November 10, 1987, thhlicensee met with.nembers of the Region IIP a staff, and members of Headquarters staff, in Pagion III to discuss theb 's'

plans for startup and to obtain NRC approval. As a result of this

-

meeting the licensee comitted to perform a notnber of actions both prh.,r to startup and subsequent to startup. Thesececmmitments are detained in -

^

a letter (PY-CEI/01E-0~.89 L) dated Novutbtr 13,1W(actions:r from Edelman, CC1, I

to Davis, NRC. The following is a summary'cf We

'

'

.

!

A.

Prior to Startup ReplacetneentirecontruiunitforML./1821-F0b28Dwithanew j

1.

'

unit and the, ASCO dual solenoids on i4SIV's 1821-F0022D and i

1RI-F007?A with new ASCO's. The N ainihg file (5) ASCO dual

.

f

'

i

!

solenoids would N rebuilt.

y x

,

2.

Replace @e ASCO single solenoff-(slow closure) cnA151V j

1821-F002dB.

'

,

4

3.

Perform n evaluation of all other ASCO solenoid Salves classified as Class IE used in harsh environment applications

'

in th3 plant.

t

-

,

.% -

Evaluate other equipment in the vicinity of the steam leaks

.+

that occurred near MSIV's 1821-FM220,1821-FCO28D, and 18f)-F0028B to assess any impact that these s wam leaks may (

have had.

"

5.

Determfne the, historical readings of the permanent steam tunnel and crywell temperature einentiin the vicinity of the MSIV's l

'

!

and detentine a baseline fo" each element and criteria to be used to indicate onset of ) steam leak in the monitored area.

Es tablis,h a procedure with actions to be taken upon exceeding a

,t thresVold value.

j

t

,

i i

6.

Install temporary temperature elements in the viudity of the

!

ASCO dual solenoids and on the solenoids and vable bodies

,

theorelves in both the steam tunnel and the drywell. Develop t

baseline data for these elements and an interin temperature

threshold, L

,

7.

Perform a test to verify that air does not flow between the air f

!

compressor reduction gear vpnts and the air compressor intake.

'

!

i

,

-

.

L

l

!

'

's

-

,

...

~

_ _ _ _ _ _ _ _ _-______-__-___________ _ ___ _ _ _ - ______

-_.

_

-___ ______ ____ _ _

.

8.

Following Startup

?

1.

Perform a laboratory anal of LSe EPDM degradation (ysis to confirm the failure mechanism high temperature / hydrocarbon attack).

.

This item is further discussed in Section VIII of this report.

2.

Establish a preventive maintenance program for periodic t

replacement of the instrument air system prefilters.

In addition ndd a generic precaution to air system work orders regarding the use of thread lubricants and sealants.

3.

Until the first refueling outage perform a monthly ASCO dual solenoid operability test and a quarterly fast closure time test. Prior to exceeding a six (6) month period, an inspection of the ASCO dual solenoid experiencing the highest temperature profile shall be performed.

4.

Complete a review of all known steam leaks in the plant which l

could affect Class 1E equipment and evaluate for any effect on their long term qualified life. Also, complete a review of

'

potentially high temperature area environments of all Class 1E

'

~ '.

solenoids and other equipment with EPDM sub-components where

'

elastomer compression set or degradation could result in

'

equipment not being able to perform its intended function.

,

On November 13, 1987, Region III released the licensee from CAL RIII-87-019 (Attachment 1) t,ased on their corrective actions, comitments, and the preliminary results of the AIT inspection. Region III also concurred with

'

'-

the licensee's reqaest to allow the plant to startup and proceed with their

'

Startup Test Program. The above was documented in a letter (Attachment 8)

from Davis to E41 man dated November 13, 1987.

,

Subsequent to the restart of the Perry Plant and completion of the Startup

^

Test Program, asiother MSIV dual solenoid valve failed. On November 3, 1987,

'

while performing the expanded monthly operability test for the dual solenoid

valves, the dual solenoid valve for MSIV 1821-F0022B failed to change state when de-energized.

The licensee shutdown the plant, and the NRC dispatched an AIT to the site.

The findings of that inspection will be documented in L

Inspe W on Report No. 50-440/87027.

i

l l

l x

!

l

[

e

.

{hh*h CONFIRMATORY ACTION LETTER

.,.

.

I s

ATTACHMENT 1

'

MW4 1987 l

t

.

Docket No. 50-440 i

Docket No. 50-441

!

The Cleveland Electric illuminating l

Company ATTN: Mr. Murray R. Edelman

,

Vice President

!

'

Nuclear Group Post Office Box 5000

,

'

Cleveland, OH 44101

,

Gentlemen:

This letter confirms the telephone conversation on November 3,1987, between (

Mr. Greenman and others of this office and Mr. A. Kaplan of your staff a

,

regarding the Main Steam Isolation Valve (MSIV) failures occurring at the t

Perry Nuclear Power Plant Unit 1 on November 3,1987. With regard to the

'

!

matters discussed, we understand that you will-

i

!

1.

Take those actions necessary to ensure that complete documentary evidence

of the "as found" condition of equipment being inspected is maintained.

l i

'

2.

Provide a step by step troubleshooting program to establish the root

,

l cause of the MSIVs failure to meet acceptance criteria.

3.

Not disturb any components that offer a potential for being the root I

cause including power sources, switches, solenoids, and the air system i

directly feeding the MISVs until that action is approved by the NRC AIT

!

tean leader, i

4.

Except as dictated by plant safety, advise the NRC AIT Leader prior to conducting any troubleshooting activities. Such notification should

!

>

i be provided soon enough to allow time for the team leader to assign an

inspector to observe activities.

5.

Submit to NRC Region !!! a formal report of your findings and conclusions l

within 30 days of receipt of this letter.

'

i None of these actions should be construed to take precedence over actions which you feel necessary to ensure plant and personnel safety.

l We also understand that Perry Nuclear Power Plant Unit I will not be made

critical without the concurrence of the Region III Regional Administrator or l

his designee, j

I t

l CONFIRMATORY ACTION LETTEL

l

!

_

.

- _ _ _ - _ _ _ _ _ - _ _ _.

.

'(

CONFIRMATORY ACTION' LETTER

-

. c.

.

.-

The Cleveland Electric Illuminating 2 gg e y Company Please let me know imediately if your understanding differs from that set out above,

Sincerely

.

t Oracinal eigned by.

L Bert Davia

.

A. Bert Davis Regional Administrator cc:

F. R. Stead, Manager, Perry

,

i Plant Technical Department M. D. Lyster, Manager, Perry Plant

,

Operations Department

!

Ms. E. M. Buzzelli, General i

Supervising Engineer, Licensing and Compliance Section DCD/DCB (RIDS)

Licensing Fee Management Branch l

Resident Inspector, RIII Harold W. Kohn, Ohio EPA j

Terry J. Lodge, Esq.

James W. Harris, State of Ohio Robert M. Quillin, Ohio Department of Health State of Ohio, Public j

Utilities Comission

-

J. M. Taylor, DEDO I

T. E. Murley, NRR

!

J. Lieberman, OE i

R. Cooper, EDO W. Lanning, NRR

,

]

F. Miraglia, NRR

'

G. Holahan, NRR

.

M. Virgilio, NRR j

J. Partlow, NRR

K. Connaughton, SRI l

J. Strasma, RIII i

.

i A1 RIII RII}y

\\

D RII RIII RI I R I

\\

sw sca Knop /jp Forn'ty

/

t Gre an P

Tello avfs U/ )

ll l1 1 i

CONFIRMATORY ACTION LETTER

,

...

- -. - -

-

- - -. - -

--

_ _ - _ _ _ _ _ _ - _ _ - _ - _ - - _ _ _

- _ _

m.;,

m-

'

,

.

[

/geA *8 0g#*,

UNITED ST Af ts

,

NUCLEAR CEGULATORY cOMMissl0N

[

%

ATTACIMENT 2

"

,#<

ngo,oN m (..,.....j)

,,,moosevet, mono

!

s'

l corn cu.vn. itu~ois som

j

.

WW 5 1*,M l

T b

i

,

'

/

!

Mf.MORANDUM FOR:

R. D. Lanksbury. Team Leader,' Perry Augmented Inspection l

Team (AIT)

.

FROM:

Edward G. Greenman, Deputy Director, Division of Reactor

!

Projects

!

i i

l SUBJECT:

AIT CHARTER

b

Enclosed for your implementation is the Charter developed for the i

,

inspection of the events associated with the Perry MSIV failures which

'

occurred on October 29 and November 3, 1987. This Charter was prepared in

accordance with the NRC Incident Investigation Manual and the draft AIT

implementing procedure issued for use on October 2,1987. As stated, the I

objectives of the AIT are to communicate the facts surrounding this event

!

to regional and headquarters management, to identify and comunicate any

!

generic safety concerns related to this event to regional and headquarters j

management, and to document the findings and conclusions of the onsite

inspection.

If you have any questions regarding these objectives or the j

enclosed Charter, please do not hesitate to contact either myself or R. Knop of my staff.

l l

MCC Edward G. Greenman, Deputy Director Division of Reactor Projects

Enclosure:

AIT Charter I

REGION W

$>

I e

tee moosavatt moao e,

OLEN ELLYN,ILUNoll 40lH

%,...../

NOV Docket No. 50-440 Docket No. 50-441 The Cleveland Electric Illuminating Company ATTN: Mr. Murray R. Edelman Vice President Nuclear Group Post Office Box 5000 Cleveland, OH 44101 Gentlemen:

SUBJECT:

CONFIRMATORY ACTION LETTER No. CAL-RIII-87-019 On October 29, 1987, and again on November 3,1987, several Main Steam ! solation Valves (MSIV's) failed to close within the maximum allowable time as delineated in the Perry Technical Specifications. As a result of these events an Augmented Inspection Team (AIT) was dispatched to the site and a Confirmatory Action Letter (CAL-RI!!-87-019) was issued on November 4,1987, to document our understanding that you would perform the following actions pursuant to the review of the MSIV failures:

1.

Take those 4ctions necessary to ensure that complete documentary evidence of the "as found" condition of equipment being inspected is maintained.

2.

Provide a step by step troubleshooting program to establish the root cause of the MSIVs failure to meet acceptance criteria.

.

3.

Not disturb any components that offer a potential for being the root cause including power sources, switches, solenoids, and the air system directly feeding the MISVs until that action is approved by the NRC AIT team leader.

4.

Except as dictated by plant safety, advise the NRC AIT Leader prior to conducting any troubleshooting activities. Such notification should be provided soon enough to allow time for the team leader to assign an inspector to observe activities.

5.

Submit to NRC Region !!! a fonnal report of your findings and conclusions within 30 days of receipt of this letter.

The CAL also specified that the plant would not be restarted without the concurrence of the Regional Administrator or his designe __.. _ _ _

_

,

,. -'

e

,

.

o gy 13 M The Cleveland Electric Illuminating

Company l

[

With respect to Items 1 through 4 you have completed all of the specified actions and these have been evaluated by the AIT. Their report will be issued shortly.

With regard to Item 5 we understand you will submit to Region III a formal report i

as specified.

[

>

Based on a review of your corrective actions and commitments as specified in 13, 1987 l

y(our letters dated November 9, 1987 (PY-CE!/0!E-0288L), and November PY-CEI/0!E-0289L), and based on the preliminary results of our AIT inspection,

!

we believe you have established adequate plans for continued safe operation of j

the plant and for final resolution of this matter. Therefore I concur with your

request to startup the Perry plant and proceed with your Startup Test Program.

Sincerely, i

Original Signed By A. Bert Davis j

A. Bert Davis

!

Regional Administrator

cc:

F. R. Stead, Manager, Perry Plant Technical Department M. D. Lyster, Manager, Perry Plant j

Operations Department t

Ms. E. M. Burrelli, General l

Supervising Engineer, Licensing and Compliance Section DCD/DC8 (RIDS)

i Licensing Fee Management Branch

!

Resident Inspector, RI!!

l Harold W. Kohn, Ohio EPA

[

.

Terry J. Lodge, Esq.

James W. Harris, State of Ohio Robert M. Quillin, Ohio j

Department of Health

-

State of Ohio, Public

!

Utilities Comission

'

R. Cooper, EDO W. Lanning, NRR F. Miraglia, NRR j

G. Holahan, NRR

M. Virgilio, NRR

J. Partlow, NRR J. Strasma, RIII

'

RI!!

RIII RIII RIII RIII RI!

RIII Chrhotteos Mker f

l L k bu /mc t

iello D

Pap)/87 ho us i

11/ 0 /87 11/0/87 11/g /87 11g/87 11/g /87 11//

11/S/87

S

???

Fo

/87 11

/d

v t e Inspection Report - Perry - 1987024
80 \| 81 \| 82 \| 83 \| 84 \| 85 \| 86 \| 87 \| 88 \| 89 \| 90 \| 91 \| 92 \| 93 \| 94 \| 95 \| 96 \| 97 \| 98 \| 99 00 \| 01 \| 02 \| 03 \| 04 \| 05 \| 06 \| 07 \| 08 \| 09 \| 10 \| 11 \| 12 \| 13 \| 14 \| 15 \| 16 \| 17 \| 18 \| 19 \| 20 \| 21 \| 22 \| 23 \| 24 \| 25 \|
1987 Quarterly Integrated 1Q (0) 2Q (0) 3Q (0) 4Q (0) Assessments Mid Cycle End of Cycle 8(0) 9(0) 10(0) 11(0) 12(0) 13(0) 14(0) 15(0) 16(0) 17(0) 18(0) 19(0) 20(0) 21(0) 22(0) 23(0) 24(0) 25(0) 26(0) 27(0) Security Operator Exams Emergency Planning Other

IR 05000440/1987024

Text

REGION III

Enclosure:

Navigation menu

Search