ML20148T073

Forwards Augmented Insp Team Charter for Insp of Events Associated w/871029 & 1103 MSIV Failures

ML20148T073
Person / Time
Site:	Perry
Issue date:	11/05/1987
From:	Greenman E NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:	Lanksbury R NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
Shared Package
ML20148M398	List: ML20148M396 ML20148T073
References
NUDOCS 8805110199
Download: ML20148T073 (9)

Text

F.-

I*

,[ aseg% ventso st:ts:

NUCLEAR RECULATORY COMMissl0N

[ ,h nion m in ATTAC10ENT 2

7.  ! too moosavatt moAo 2 ottN sLLvn,iLLimois setat WW5 1*JI j .

MEMORANDUM FOR:

/(l R. D. Lanksbury, Team Leader,' Perry Augmented Inspection Team (AIT)

FROM: Edward G. Greenman, Deputy Director, Division of Reactor Projects

SUBJECT:

AIT CHARTER Enclosed for your implementation is the Charter developed for the inspection of the events associated with the Perry MSIV failures which occurred on October 29 and November 3, 1987. This Charter was prepared in accordance with the NRC Incident Investigation Manual and the draft AIT implementing procedure issued for use on October 2, 1987. As stated, the objectives of the AIT are to cormunicate the facts surrounding this event to regional and headc.uarters management, to identify and comunicate any generic safety concerns related to this event to regional and headquarters management, and to document the findings and con:1usions of the onsite inspection. If you have any questions regarding these objectives or the enclosed Charter, please do not hesitate to contact either myself or R. Knop of my staff.

MM Edward G. Greenman, Deputy Director Division of Reactor Projects

Enclosure:

AIT Charter cc w/ enclosure:

A. B. Davis, R!l!

C. J. Paperiello, RIII F. Miraglia, NRR J. Partlow, NRR C. Rossi, NRR G. Holahan, NRR L'. Lanning, NRR M. Virgilio, NRR R. Cooper, EDO K. Connaughton, SRI l

b(II I '- N.)

J.

1 1 Perry MSIY Stroke Time Failure Augmented Inspection Team (AIT) Charter Investigate:

1. Failure of MSIVs to close/close within Technical Specification limits.

2. Safety Significance, Root Cause(s).

3. Interaction of prior maintenance activities to the event.

4. Safety implications if actual Group 1 isolation signal had been present.

5. History of any previous problems.

6. Broader Implications e.g. other systems, other valve / components.

7. Event Reporting.

8. Conclusions.

Questions for Perry AIT

1. Failure of MS!Vs to close/ clos within Technical Specification limits.

(10/29/87and11/03/87) 1.1 What was the sequence of events?

1.2 What were the closure times generated during the surveillance?

1.3 What operator actions were taken during the event? Were they appropriate?

1.4 Is there a history of any previous problems (e.g. 10/29 event, etc) with the MSIVs?

1.5 Did the RPS logic makeup per design during the surveillances?

1.6 What additional testing was being performed?

2. SafetySignificance,RootCause(s).

2.1 Was there any imediate safety significance from this event? If so, what was significant?

2.2 What was the root cause of the event?

3. Interactions of maintenance activities to the event.

3.1 What is the past and present maintenance history of the MS!Ys?

3.2. What is the maintenance history of the Service Air (SA) and InstrumentAir(IA).

3.3 khat testing was perfomed as the result of maintenance activities?

3.4 What is the material condition of the affected valves and inter-connected instrument air and control systems as it would affect the valve closure function?

4. Safety implications if actual Group I isolation signal had been present.

4.1 Does the licensee have procedure in place to handle this event?

4.2 Are they adequate?

4.3 Have the operators been trained on them?

i h

i . ..

J' ; '

4.4 Does the accident analysis bound this event?  !

4.5 What actions were taken by the operators?

4.6 Was the event properly categorized?

4.7 Was the event reported as required?

5. History of ab, previous problems.

5.1 Have there been previous events similar to this? <

5.2. If there were previous events was the licensee aware of them? i 5.3 If not, why not?

5.4 Is there information available on other similar events?

5.5 Have there been any IEIN's or IER's issued or similar subjects?

5.6 Is there information avaiable from other sites of similar problems?

6. Broader Implications.

6.1 Is a IEIN or IEB warranted or a result of this event?

6.2 Are there other valves or instruments that require investigation?

6.3 If the problem lies e'xternal to the MS!Y's, are ther0 generic implications? e.g. for other plant systems or other plants with same components.

7. Conclusion.

'f 7.1 What corrective actions are propcsed, and art they adequate?

7.2 Examine generic implications to other plants and advise NRC management subsequent to the site inspection.

7.3 Document inspection findings in accordance with draft manual chapter 0325.

t l

i

. . ..s.. '

i ATTACHMENT 3 AIR SYSTEM P&ID l

i .

'.e

' 13 .

I g @h :t s .

k b.* Ml f?  :

\' &'s fsY.%jb i

', E=

r  ?

(r $_rM,;

..s. 4

) %)Mr

-.- .Ll.p:fl ,

s ll !D. ..].5 g D'.' _#

h, ,,(,n,l g,,

r . ap t -l i! 18 .!'- "-

,, Ih

• k,

, g'4 U,L- > 'W i.igf ,

I' ik

! Ijil J i >b I'r  ;  !

'l tl

!  ; in rd e cs r e: y CT' I ,. ..!.!; . i ,ou, !v! e _$ 1- - (;n-

3. m -s f

' W, i OId o i ,I o

4 Ey.7 j ,

> YI ' *---- @

I W; h., )!e,, j/i'!e # 11 6 n

f+

-,,}1l.

8 . 33'"

yf{s' l ,&

c h -

Jl f. ,

-p AT

'h 9 Is

., 1A. :

3

c-e p r-e_I .I i p -

J l yf f i ,, -

~3.

}(,-.ci,h s

I

/-

c $ 'O iA d 9 N1. h,i,r;--

i G

I I I' 11 l [, f: jt 7.l..fr...q -

3

'e 'b 5m  :' g.....s.....q-E I(dA {

,' r

• c**id* * ' * . . .>..I' '

8 * ' M' ' .

]  ;

l 'A 5 S .

il Y

I ".

g..p,

. i N 41 &.4

<~

\

\.1

.l f t N. n" . l.<..(.t, ,

i ',i 'e.3 N i

i i -l11. . a S, - ,

.,l ['?

• 4 c,-

l,.

l!M. _lI jl Il7ge..,s ll,'pJ@i J d' lc g-w, i )', . .o<

g.7 Ihes:pd,' . n -

V u~

i 3Dsp .$ f3 9 4 ,

il . #.y . . @l:pj! , j If lll.3][it.

,, ,. l[ '*c.N $.

8

~ , . (_

o l,;Ik. .g/'h h..g'[id.,E!i[

. ., p !, , e

,T 1 -

E q . .h.. '

i ea

- a. ..: g ;.,,il

,q , :e ,.s.s .g _ .p,.. . . ..gg. g, 4[K.. ,. . 3

. ,i i :,

. p.i

,g3..j............,gij,g;{L [h; EF,i i1* . I .#p  : , i! ;;I;

-l  ! i .$

terlid -

- ', l 1 4

i

'------ i i, , , 1, y

, ,4  :

- " ~ -

-" ill  : . . . . . . . . . . . . . . .' "cc :::: r.1 ill

1 .

ATTAQOENT 4 I . . 1 1

• • AIR CYLINDER HYDRAULIC I

es l'

ei a

g

}

V,e ouM Por SPRING GUIDE t

h l

M '

l< .

lll ll l ACTUATOR SUPPORT AND 9PtED CONTROL VALVE ll 9PRING GUIDE SHAFT gg y s.

= II ,,

HELICAL 9PRINGS

_.__j l' .

ij ,

lI lt jo  ? $PRING St AT MEMBER ll STEM PACKING s W

/ Lt AE OFF CONNECTION

{N

"[ / 50NNtfWOLTS 5f ifh BJhhfT LANTtRN RING CLEARANCE N

. u _

~

x

'x (r fH Q[ . ,

PILOT SPRING

=

lhh -

PILOT gooy gly P$PPET IPLUG. MAIN DISKI N W AIN V ALVI st AT

.- 1 PILOT SE AT

'N

/

FLOW _

PERRY MUCLEAR POWER PLANT G THE CLEVELAND ELECTRIC lLLUMINATING COMP ANY Main S'.eam Isolation Valve Figure 5.4-8

, ---m - -

ggmus a l . . .* -

O t'- , ' '-=.a ommiessen wes)=so m *'.a=m==e=s=e =m== ATTACHNDff 5

• ' e ,

O s

<=e=ss=mu~ssa~see~m n <==sw=ta,emme me~w = m l

o

@ (e mean,mnet twg.nse P%et optah'e6inna ess we Laesseem m *

' @ &*===a wmsnm swo mem  ;

,=evema=m.w v)wte.we. s w m wwm .

'@ eus

(*s*ecum==.re.tewsmeeerumnwanttimeomapsee mm)est m neconset onnen menesus * ,

@ tT sewmaut awassesan eges annum, una thstas. .

l

• • essomnem3esas m wo man,ca.

@ I'rutteam team Pses acem,' msg (eeman

, truen caem) ws = ===aven. te.

@, & -(to cassgegas see wen.m, tapennen .

w vast ei.)ws e mere = enweenee. . .

@ l' n a == t= namn.ma , t u.e

=== %

$ Deca mut (1Nonwisasm et omag)  % 3 - I

@ A88 STWWut was LPWesnata = 974eQ Ana m E t- =-- w em anai mg gqlll l ll tlr3 .

• as csmann we gangf 9 t e $4 m

to tytie ** -tuna gertoi Ct sea, ion tanta n "-

to now mw .orus guetties tt see u .,

a m a tueter == nmun .cwsw treism u (gru:m et aan c1 estru te ortu arten tuetw -stvem setttm u tt wem ortu wwt me h

g M

" G G 4

}

.a g# ,

er innac tweaas cm. nami asst .

a (o.a. wet oos csona a newes e seus unnm)

QS y

\ I I c . ,-

s.t. _ .- -

96 wes ea nt, u.

~

t_c m

\ /- ~*

i 1 - I

)

g enase cawrwe.Yterm aasum em an. 6a no n.c.

o 8

j -

( m I E /, .) -

- 8 D m iI

! tarem v se-.mamuta sa mn. ee saAt.,

y[ .

lN ,,,,,,

p ..

i

• • ma m ut8""" """"#'"' ' # "

l l ~rN.ep . 7 f.'M Y esa = = w M r e m=6 m i er j i l ma w=u 7 . . . . .. .. . q

..k -

i .

t e 1 i ._

es.=4 ese teamst tse==

I 13%O@v4 C ..

I K*1 l

t

l l

',..'t.

• ATTAQOENT 6  !

. __ l l

utanw o 'A' ~ '

l pk j

. .. I i;.

e. \- t i

l .-

. , ,i i i; s  ;

v> N

> l ,

[ '

d /

h 4

h I$f

' ~ ~

c>.~,/

't a

Ww R

,/  :

'%l.,,

1 J/ . n t N o-4mc,

{

\X l

)

f,?.'.?" " 8

• d

[..; t u is

$j[X/[ 3 suuwo s-

-[ --i 4 o

t. E iI tl E

! /,. --;

I ys g' [!

I i I __ b D'

.7_ -- l; g I!

s

[r i.

[ -L ';

Sheet 1 of 2

u

- +

,- . i.; m [ I' I l

[g __

r 1 _ _

r 4' ,__

.is l

[j ,

+ l

, j_ _

p .)

'[

ri 5{I id . . . . .

A k 4 t i 6 =m 0 f' 9 f

t,il, 8e l 4 k A( S

+j.Si k

?" ""3.') _

?2, 13 ^ , r.ny .

y

\. ... . .l ,h!";ny Y $

s, to.; wm a

T.f -

c I

/

I' D '

E{4 r

. . +.

1 ( t w

(* '

x 4

< ") L a N

't E i $ 4 9 1 8 V

• $ I$

l' s*

'i' I

v

, k 2

_) 'A s t

' ~

c 4

F=

,. r' kommm,

• u a m)  %

>00 ) ,.

c vm Q A

I I

) T a

l o { ,

t l

l

[T5rf Qf"T ,

I Sheet ? of 2 i

. _ . _ _ . ~ _ - . . _ . . . . _ . . _ ._ - - - . . . .. _ .- . . _ _ . _ . _

q. yv _ ~- w- r p : , . 4. ; :. 3;

~ '

i1.. , ;l

'"' *

• T

,leg ,

-r (l>

.[

- ATLCHENT ,7 - ~, _ f x

\

t.

DISC HOLDER SKETCM. , -

s!

'l ) .

I <s j ,4i 3

,\

[ ,

b. ,s -y \.s: i s s .' '

) <

s 7s .

DISK HOLDEL"LEGD .(

(GUIDE!') N '; 5 X $ I'

\ \

, 7.\

/

! _ ANNUIAR DIMPLE

-  ! EPDM DISC s x <

, y , 3

't 1 DISK HOLDEE ,  !

1 I t;$. -(,

DISC HOLDER GENERAL APPEAPANCE s; (

i 1-

?((.t'

! I 6 \.

i ,

4 t

EXHAUST "10 CYLINDER RETORT ,

/ , '

_ FIDM DISK

,g UTSC HOLDER s.

&\\\\ x\\W/

aw:aN

& vss La CROSS SECTION OF DISC IN ITS SEATED POSI1174 i

I h

1 A

.q.

k I

- . , , ._- ) '

r- y .,

, , {pf.:').

LM".TE) STATES ATTAQDfENT 8

[pa Metqf;'N NUCLEAR MOULATORY COMMISSION qg

' . y 4

,1-  :

j '- b7.SiON lil' 799 R.1ostvtLT moAo J' 1 OLEN ELWN, ILUNOlf. 80117 J *.se*

g 1313N Docket No. 50-440 Docket No. 50-441 The C1' _ land Electric Illuminating

. Compqy ATTN: Mr. Murray R. Edelman s Vice President Nuclear Group Post Office Box 5000 Cleve?and, OH 44101 Gentlemen:

SUBJECT:

CONFIRMATORY ACTION LETTER No. CAL-RIII-87-019 On October 29, 1987, and again on November 3, 1987, several Main Steam Isolation

, Valves (MSIV's) failed to close within the maximum allowable time as delineated

< in the Perry Technical Specifications. As a result of these events an Augmented i

Inspection Team (AIT) was dispatched to the site and a Confirmatory Action Letter

, (CAL-RIII-87-019) was issued on November 4,1987, to document our understanding that you would perform the following actions pursuant to the ~ review of the MSIV failurts:

' 1. Take those actions necessary to ensure that complete documentary evidence of the "as found" condition of equipment being inspected is maintaint.d.

2. , Provide a step by step troubleshooting program to establish the root cause c$ the MSIVs failure to meet acceptance criteria. *

/

3. Pat disturb any components that offer a potential for being the root cause

/ inchdf37, ppr sources, switches, solenoids, and the air system directly 1 eeding @e; MI5'!s until that action is approved by the NRC AIT tean leader.

1) j 4. Except as dictated by plant safety, advise theSuch NRC AIT Leader prior to notification should be i

cons'ucting any troubleshooting activities.

provided soon enough tp allow time for the team leader to assign an inspector to observe w:tivities.

\ ,

s

5. Submit to NRC Region III a fermal report of your findings and conclusions within 30 days of receipt of this letter.

s, The C4L also specified that the plant would not be restarted without the concurrence of the Regional Administrator or his designee.

N

(

( g N ~ h )-

.+

/. .

The Cleveland Electric Illuminating 2 g 13 ON Company i

With respect to Items 1 through 4 you have completed ahl of the specified actions and these have been evaluated by the AIT. Their report will be issued shortly.

With regard to Item 5 we understand you will submit to Region III a formal report as specified.

Based on a review of your corrective actions and comitments as specified in your letters dated November 9,1987 (PY-CEI/01E-0288L), and November 13, 1987 (PY-CEI/01E-0289L), and based on the preliminary results of our AIT inspection, we believe you have established adequate plans for continued safe operation of the plant and for final resolution of this matter. Therefore I concur with your request to startup the Perry plant and proceed with your Startup Test Program.

Sincerely, Original Signed By A. Bert Davis A. Bert Davis Regional Admir.istrator cc: F. R. Stead, Manager, Perry Plant Technical Department M. D. Lyster, Manager, Perry Plant Operations Department Ms. E. M. Buzzelli, General Supervising Engineer, Licensing and Compliance Section DCD/DCB (RIDS)

Licensing Fee Management Branch Resident Inspector, RIII Harold W. Kohn, Ohio EPA -

Terry J. Lodge, Esq.

James W. Harris, State of Ohio Robert M. Quillin, Ohio Department of Health State of Ohio, Public Utilities Comission R. Cooper, EDO W. Lanning, NRR F. Miraglia, NRR G. Holahan, NRR M. Virgilio, NRR J. Partlow, NRR J. Strasma, RIII RIII RIII RIII RIII RII RII RIII L kb /mc t Chr YHotimos M fer No 'Tu s iello D 11/p/87 11/g /87 Pap )/87 11//%/87 11/0/87 11 /87 11/f/87 11//

Fo 1__ ___ /87 117 /8i

<-G ? . .

f.

• e,g* ATTACll!!ENT 3 FEB 101953 Docket No. 50-440 i

The Cleveland Electric Illuminating Company ATTN: Mr. Alvin Kaplan Vice President Nuclear Group 10 Center Road Perry, OH 44081 Gentlemen:

The enclosed report refers to the~special onsite review conducted by an NRC Augmented Inspection Team (AIT) composed of R. D.1.anksbury, S. D. Eick, and G. F. O'Dwyer of this office and H. L. Ornstein (AE00), and S. D. Alexander (NRR) on November 29 through December 4, 1987. The review was in response to the recent failure of one Main Steam Isolation Valve to stay closed. Operation of the Perry Nuclear Power Plant, Unit 1, is authorized by NRC Operating License No. NPF-58. The essence of our findings were discussed with Mr. A. Kaplan and others of your staff at the conclusion of the inspection.

The enclosed copy of our inspection report identifies areas examined during the inspection. Within these areas, the inspection consisted of a selective examination of procedures and representative records, observations, and interviews with personnel.

The major purpose of the AIT was to conduct a timely, thorough, and systematic inspection of the event in order to determine the cause(s), conditions, and circumstances pertaining to it, and to communicate to NRC management the facts and safety concerns related to the event. While primarily a fact finding mission, issues identified by the AIT may be examined for possible enforcement in subsequent inspections.

In accordance with 10 CFR 2.790 of the Commission's regulations, a copy of this letter and the enclosed inspection report will be placed in the NRC Public Document Room.

We will gladly discuss any questions you have concerning this inspection.

Sincerely,

,......<..t . -

^ '

C 'ubert H J. Mille ~r, D'irector Division of Reactor Safety l

f'd k gE losure: y WAugmented Inspection Team gortNo. 50-440/87027(DRS)

See Attached Distribution RIII RIII, RIII Illh I.I I RIII Rill I l

M ..

m N Lanksbury/lc # 0' Dwyer es Eick

/

illips (d  % Mi r g Eersey ChPp(sto imos 2/'i /88 +/r/fi kg t ,

NRR NRR ,

Palb Abf k Ornstein Alexhnder 1

w v'

~ -

e3 ,

'rg~" : 1 " p30C The Cleveland Electric Illuminating 2 Company Distribution cc w/ enclosure:

F. R. Stead, Director, Perry Plant Technical Department M. D. Lyster, General Manager, Perry Plant Operations Department Ms. E. M. Buzzelli, Manager Licensing and Compliance Section M. R. Edelman, Nuclear Vice President, Centerior Energy DCD/DCB (RIDS)

Licensing Fee Management Branch Resident Inspector, RIII Harold W. Kohn, Ohio EPA Terry J. Lodge, Esq.

James W. Harris, State of Ohio Robert M. Quillin, Ohio Department of Health State of Ohio, Public Utilities Commission I.

ne.. '. . a ., ,

a U.S. NUCLEAR REGULATORY COWi!SSION REGION III Report No: 50-440/87027(DRS)

Docket No: 50-440 License No: NPF-58 Licensee: Cleveland Electric Illuminating Company Post Office Box 5000 Cleveland, Ohio 44101 Facility Name: Perry Nuclear Power Plant, Unit 1 Inspection At: Perry Site, Perry, Ohio Inspection Conducted: November 29 through December 4, 1987 NRC Augmented Inspection Teat:

Inspectors: Team Leader- . D . L . a b u ry 2hh9 (Date)

~n~bf Team Members . F. wyer 2 /9he (Date)

RD Date)

H L. O n ein 2hhe (Date)

D % xander Q.. 2 f9/te 7

(Date)

Approved By: . 07! fe f

Operations Branch (Dats)

Inspection Sumary Inspection on November 29 through December 4, 1987 (Report No. 50 /,40/87027(DRS))

Areas Inspected: Special Augmented Inspection Team (AIT) inspection conducted in response to the Main Steam Isolation Valve (MSIV) closure failures of g November 29, 1987, on Perry Unit 1, and related activities. The review Y y

T 4-= , ,

y . ., .

I included root cause determination, safety significance, maintenance history, and broader industry implications.

Results: No violations or deviations were identified; however, the licensee has committed to additional and expanded surveillances of the MSIV's and maintenance activity practices in order to preclude subsequent similar failures.

i l

11 1

EXECUTIVE SUINARY On November 29, 1987, the Perry Nuclear Power Plant was in the process of performing a Main Steam Isolation Yalve (MSIV) fast closure operability check when the inboard MSIV in the "B" main steam line would not stay closed. The licensee was performing the operability checks as the result of commitments made due to previous problems with MSIV closures that occurred on October 29, 1987, and again on November 3,1987. The operability check consisted of depressing the slow closure "test" push button and allowing the MSIV to fully close. The control switch was then placed in the "close" position and the "test" push button released. If the fast closure solenoid operated valve changes state per design, the MSIV will remain closed; if it fails to change state, the MSIY will reopen. The licensee unsuccessfully performed the above test a second time and then attempted to fast close the MSIV two times with no success. Subsequently, the licensee reported the failure to the NRC and commenced an orderly shutdown. Following the shutdown, licensee personnel and the Senior Resident inspector made a drywell entry to observe the MSIV during a fifth closure attempt. During this test the valve stayed in the open position until the ASCO dual solenoid valve was gently tapped. The MSIV responded by closing with a normal stroke time. As a result of this event, Region III dispatched an Augmented Inspection Team (AIT) to the site the same day.

The licensee evaluated potential component failures and failure modes and from this, developed a carefully planned disassembly and troubleshooting program.

As a part of this troubleshooting program the licensee disassembled the failed ASCO dual solenoid valve. The results of this disassembly and inspection revealed the presence of a sliver of foreign material and two smaller particles of foreign material in the "B" colenoid housing assembly. This material was later proven to be Ethylene Propylene Diene Monomer (EPD'i) from one of the 0-rings in the valve that was replaced as part of the corrective action to the November 3, 1987, event. No signs of other solenoid valve degradation were evident. The licensee evaluated the results of the troubleshooting program and concluded that the root cause of the failure of the MSIV to close was mechanical binding of the ASCO dual solenoid valve by the sliver of EPDM material. The mechanical binding resulted in the exhaust seat being held in an "energized" position even though the solenoids had been de-energized, and therefore, prevented the control air from being exhausted to atmosphere and prevented the MSIV from closing.

The AIT concluded that the root cause of the observed MSIV failure to stay closed on Noverrber 29, 1957, was a malfunction of the ASCO Model No.

NP-8323A20E three-way dual solenoid valve caused by pieces of the body gasket (EPCM 0-ring) falling into the "B" solenoid valve sub-assembly and ultimately interfering with the ability of the ASCO dual solenoid valve to shift to the de-energized position. The source of the body gasket material was deteriorated and degraded EPDM 0-rings that were replaced as part of the November 3,1987, event. This foreign material was introduced as a result of the maintenance activities by the licensee to rebuild the dual solenoid valve. The rebuild activity did not conpletely disassemble the "B" solenoid and therefore, the presence of the foreign material went undetected. The licensee subsequently replaced all eight MSIV dual solenoid valves with new ones. The plant was restarted on December 8. 1987.

iii

v

. . ,= ..

s Augmented Inspection Team (AIT) Report 50-440/87027 Page No.

I Introduction A. Synopsis of Event 1 B. AIT Formation 1 C. AIT Charter 2 D. Persons Contacted 2-3 II Description - MSIY Failure to Close on November 29, 1987 A. Narrative Description 3-4 B. Sequence of Events 4-5 III Failure Mechanism Analysis 6 IV Investigative Efforts A. Main Steam Isolation Valves (MSIV's) Description 6-8 B. Evaluation of Safety Significance 8 C. Operator Response 8 D. Troubleshooting Activities and Results 8-9

1. Material Condition of the Affected Valves 9-10

2. Laboratory Analysis of Particles Found in the Solenoid Valve 10-11

3. Analysis of Instrument Air System Air Quality 12-13

4. Licensee's Conclusions 13-14 V Recent Events Involving MSIV Slow Closure / Failure to Close 14-15 VI AIT Conclusions 15-16 VII AIT Recommendations 16 VIII Exit Interview 16-17 IX Startup Review A. Prior to Startup 17 B. Following Startup 18 iv

7.<

"J: ..- .. ,

t ATTACHMENTS Attachment No. Description 1 ConfirmatoryActionLetter(CAL) 2 Augmented Insepetion Team (AIT) Charter 3 Main Steam Isolation Valve (MSIV)

Cross Section 4 MSIV Control Unit Schematic Drawing 5 Cutaway Drawing of an ASCO 8323 Solenoid Valve 6 Samples Submitted for Analysis 7 Particles Found in Solenoid Valve 8 Particles Nucleation Site on 0-Ring 9 Restart Authorization I

y

I INTRODUCTION A. Synopsis of Event On November 29, 1987, while at approximately 80% power and in the process of performing a Main Steam Isolation Valve (MSIV) fast closure operability check, the inboard MSIY in the "8" main steam line at the Perry Nuclear Power Plant, Unit 1, would not stay closed.

Each of the other seven MSIV's were successfully tested. The licensee was performing the operability checks as the result of comitments made due to previous problems with MSIV closure that occurred on October 29, 1987, and again on November 3,1987 (see Augmented Inspection Team Report 50-440/87024). The operability check consists of depressing the slow closure "test" push button and allowing the MSIV to fully close. The control switch is then placed in the "close" position and the "test" push button released. If the fast closure, dual solenoid operated, valve changes state per design the MSIV will remain closed; if it fails to change state the MSIV will reopen. The licensee unsuccessfully performed the above test a second time and then attempted to fast close the MSIV two times with no success. Because of a previous comnitment made by the licensee as a result of the past MSIV failures, the licensee reported the failure to the hRC and proceeded to shutdown the reactor to allow investigation of the event. Following the shutdown, licensee personnel and the Senior Resident Inspector made a drywell entry to observe the MSIV during a fifth closure test. During this test the MSIV again failed to close during the approximate three minute period in which the cor. trol switch was held in the "close" position. The dual solenoid operated valve was then gently tapped and the MSIV responded by closing in a normal stroke time (approximately 3 seconds). The MSIV was cycled one more time and exhibited a nomal closing stroke time.

B. AIT Formation Subsequent tc the report of this event, Region III evaluated the data provided by the licensee on November 29, 1987, and detemined that the criteria for an AIT existed. Assistance from the Office of Nuclear Reactor Regulation (NRR) was requested in several specialized areas including air systems and solenoid valves. This assistance was provided by Dr. H. L. Ornstein, Senior Reactor Engineer (AE0D) and S. D. Alexander, Reactor Engineer (NRR). In addition, Region III provided expertise in operations and plant maintenance by assigning G. F. O'Cetyer, Perry Plant Resident Inspector, S. D. Eick, Reactor Inspector, and R. D. Lanksbury, Reactor Inspector and Team Leader.

Mr. Lanksbury and Ms. Eick arrived on site the evening of November 29, 1987, Dr. Ornstein arrived on November 30, 1987, and Mr. Alexander arrived on December 1, 1987. Concurrent with the AIT activities, Region III issued a Confirmatory Action Letter (CAL-RIII-87-24) which l was received by the licensee on November 30, 1987. The CAL confirmed certain actions to be taken by the licensee in support of the AIT and also confirmed that the plant would not be restarted without the concurrence of the Regional Administrator or his designee. The CAL is Attachment 1 to this report. .

l \

l l

i l

1 ".' . .,- '

I

. 1 C. AIT Charty On November 30, 1987, a draft charter for the AIT was formulated -

with a list of preliminary questions to be pursued and the following list of general areas to be investigated:

Failure of MSIV's to close/close within Technical Specification limits.

Rootcause(s).

Interaction of previous maintenance /rt: placement activities to the event.

Broader implications.

Event reporting.

A finalized AIT Charter was issued on December 2, 1987. This ,

Charter is Attachment 2 to this report.

D. Persons Contacted Cleveland Electric Illuminating Company (CE!)

• A. Kaplan, Vice President, Nuclear Group

• F. R. Stead, Director, Perry Flant Technical Department (PPTD)

E. Riley, Director, Nuclear Quality Assurance Department (NQAD)

R. A. Newkirk, Manager, Technical Section, PPTO V. K. Higaki, Manager, Outage Planning Section, Perry Plant Operations Department (PP00)

W. E. Coleman, Manager. Operations Quality Section, NQAD B. D. Walrath, Manager, Engineering Projects Support Section, Nuclear Engineering Departinent (NED)

D. R. Green, Manager, Electrical Design Section, NED

• E. M. Buzzelli, Manager, Licensing and Compliance Section, PPTD S. J. Wojton, Manager, Radiation Protection Section, PPTD K. R. Pech, Manager, Mechanical Design Section, NED R. A. Stratman, Manager, Operations Department, PPOD W. R. Kanda, Jr. , Manager, Instru.nentation and Control Section, PP00 V. J. Concel, Lead System Engineer, Technical Sectinn, PPTD E. F. Parker, Supervisor, Mechanical Maintenance Quality Section, PPTD P. J. Arthur, Nuclear Steam Supply System Lead, Technical Section, PPTD G. A. Dunn, Supervisor, Licensing and Compliance Section, PPTD S. W. Litchfield, Equipment Qualification Lead, Engineering Projects Support Section, NED A. J. Polland, Lead Quality Engineer, Mechanical Maintenance Quality Section,NQAD General Electric (GE)

T. R. McIntyre, Manager, Perry Engineering R. S. Tunder, Senior Engineer, Plant Materials Technology 2

~ .; ,

9 Automatic Switch Company (ASCO)

K. Thomas, Sales Engineer Ricerca, Inc.

R. L. Cryberg, Manager, Analytical Services W. R. Bramstedt, Group Leader, Analytical Services

• Denotes those attending the exit meeting on December 4,1987.

In addition to the above, other members of the Perry staff and Ricerca staff were contacted by the AIT.

II DESCRIPTION - MSIV FAILURE TO CLOSE ON NOVEMBER 29, 1987 A. Narrative Description On November 29, 1987, at approximately 1:57 a.m. (EST) while Perry Unit I was operating at approximately 80% power, the licensee performed a MSIV fast closure operability check in accordance with Surveillance Instruction (SVI) -C71-T0039, Revision 1, "Main Steam Line Isolation Valve Closure Channel Functional." The licensee was performing the operability checks as the result of commitments made due to previous problems with MSIV closure that occurred on October 29, 1987, and again on November 3, 1987, (see AIT Report 50-440/87024).

The operability check consisted of depressing the slow closure "test" pushbutton and allowing the MSIV to fully close. The control switch is then placed in the "close" position and the "test" pushbutton released. If the fast closure, dual solenoid operated, valve changes state per design the MSIV will remain closed; however, if it fails to change state the MSIV will reopen. At approximately 2:39 a.m., inboard MS!Y 1B21-F00228 was slow closed in accordance with SVI-C71-T0039, Step 5.1.36.b.b.. Upon reaching the fully closed position, the MSIV control switch was placed in the "closed" position and the "test" pushbutton released. The MSIV reopened indicating that the dual solenoid valve had not changed state. The operators verified that the solenoids had de-energized by observing their status lights in the control room. At 2:42 a.m. the operators again performed the slow closure operability check and again the MSIV reopened. Subse-quently, they attempted to fast close the MSIV two separate times with no success. Based upon this the Unit Supervisor, at 2:45 a.m.,

declared MSly 1821-F0022B inoperat,le. Per a comitment made as a result of the October 29, 1987, and November 3, 1987, MSIV failures the licensee commenced an orderly reactor shutdown. At 3:32 a.m. the System Operation Center was informed of the intended plant shutdown and at 3:35 a.m. the licensee made a courtesy four hour Emergency Notification System (ENS) report on the failure of MSIV IB21-F0022B.

At 11:14 a.m. the reactor was scramed and placed in Hot Shutdown.

Following the shutdown, licensee personnel, accompanied by the Senior Resident inspector (SRI), made a drywell entry to inspect the MSIV and observe it during a closure attempt. At 1:03 p.m. a third attempt to fast close MSIV 1821-F00228 was made with no success. Personnel located at the MSIV verified, by monitoring the electrical terminals, 3

that the fast closure solenoids had de-energized. After holding the control switch in the "close" position for approximately three minutes the fast closure solenoid's valve body was gently tapped. MSIV 1821-F0022B then closed in approximately three seconds. Technical Specification 3/4.4.7 requires that each MSIV close within a time frame of 2.5 to 5 seconds and Technical Specification 3/4.6.4 require that they close within 5 seconds. At 1:06 p.m. the operators opened and fast closed MSIV 1821-F0022B one more time and observed that it operated properly. At 5:20 p.m. the reactor reached Cold Shutdown.

B. Sequence of Events and Operator Actions At the AIT's request, a chronology of events related to the MSIV failure on November 29, 1987, was assembled by the licensee. The chronology, which includes MSIV performance data and operator actions, was verified to be accurate by AIT personnel through review of operat-ing logs, condition reports, ERIS Limiting Condition for Operation (plots, Technical SpecificationLC0) tracking and interviews with licensee operating personnel and staff. The chronology was as follows:

NOTE: All times are in Eastern Standard Time.

November 8, 1987 2048 Rebuild work on MSIV 1821-F00228 complete per Work Order (WO) 87-9464 November 10, 1987 2324 MSIV IB21-F00228 slow closed per Systen Operating Instruction (501)-B21, "Nuclear Steam Supply Shutoff, Automatic Depressurization, and Nuclear Steam Supply Systems (Unit 1)". MSIV control switch placed in the "close" position and test pushbutton released. MSIV verified to have remained closed indi-cating that the fast closure solenoids had de-energized and changed state. All p accomplished in accordance with W0 87-9464,  ;

Revision 3. MSIV IB21-F0022B fast closed per Surveillance Instruction (SVI)-B21-T2001, "MSIV Full Stroke Operability Test," as a retest for WO 87-9464. It closed satisfactorily in 3.2 seconds.

November 16, 1987 0535 Startup Test Instruction (STI)-B21-0025B,  !

"Full MSIY Closure," was done at 96%

power (all MSIV's fast closed satisfac-torily incluaing MSIV 1821-F0228).

November 29, 1987 0157 Commenced SVI-C71-T0039, Revision 1, "Main Steam Line Isolation Valve Closure Channel Functional." All MSIV's cycled properly except MSIV 1821-F0022B, which is described below.

4 b

j- . v i~~

i

_0239 Slow closed MSIV 1821-F00228 per SVI-C71-T0039. MSIV IB21-F00228 reopened, indicating that the dual solenoid valve did not change state as required when the control switch was placed in the "close" position. Operators verified that the fast closure solenoids had de-energized.

0242 Slow closed MSIV 1821-F00228 and again, the MSIV reopened. Operators verified that the fast closure solenoids had de-energized.

Operators twice attempted to fast close MSIY 1821-F00228 by placing the MSIV control switch in "close". The fast close solenoids de-energized but the valve did not close.

0245 Declared 1821-F0022B inoperable and reactor shutdown commenced per a licensee commitment made following MSIV dual solenoid valve failures on November 3, 1987.

0332 Informed System Operations Center of intended plant shutdown.

0335 Made a courtesy 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> ENS report on failure of 1821-F0022B to remain closed.

1114 Reactor taken to Hot Shutdown by scramming the reactor.

1303 Af ter reactor shutdown, operators placed the control switch for MSIV 1821-F0022B in "close" but the MSIV did not close. Personnel verified locally that the fast close solenoids had de-energized. Approximately 3 minutes after the switch was placed in "close",

MSIV 1821-F0022B's solenoid valve was gently tapped and it then closed properly in approximately 3 seconds.

1306 Operators opened and fast closed MSIV IB21-F0022B, one more time, and it operated correctly.

1720 Reactor in Cold Shutdown.

6

! 5

, j III FAILURE MECHANISM ANALYSIS After the event of November 29, 1987, the licensee convened a team of individuals from various departments including representations of Gilbert Associates (GAI) (the architect engineer), General Electric (G.E.),

Automatic Switch Company (ASCO) (the solenoid valve vender), and Ralph A.

t Hiller Company (the control unit vendor). The charter of this team was to develop a list of components whose failures would fit the observed behavior of the MSIV. After developing this list, the known facts were used to evaluate the probability associated with each of the potential component failures. The team also evaluated the list of failures that had been developed as part of the root cause analysis performed as the result of the October 29 and November 3, 1987, events (see AIT Report 50-440/87024 fordetails). Their analysis concluded that the only component failure that fit the observed behavior was a failure of the ASCO Model NP-8323A20E three-way dual solenoid valve. The tear next developed a list of potential failure modes of the ASCO dual solenoid valve and the corresponding proba-bility of each of these modes. Their analysis yielded a total of eight (8) potential failure modes. Of these, two (2) were evaluated as likely and six (6) were evaluated as unlikely. The eight potential failure modes and their associated probat;ilities of causing the observed behavior are as follows:

• Improper di.sassembly or reassembly of dual solenoid valve during rebuilding

• One or both solenoid coil assemblies failed to reposition Poor instrument air cuality Non-qualified seal elastomers Local high tecperature deterioration of EPDM seal materials Blockage of the dual solenoid valve exhaust port with tape Deoradation of 0-ring lubricant Dual solenoid valve mounted at a 45 angle vs vertically as recomended by the vendor

• Likely failure Subsequent to this analysis the licensee provided a written proposal for l troubleshooting the MSIV to the AIT for concurrence. After evaluation I and comment by the AIT a carefully planned disassembly and troubleshooting l program was generated. In conjunction with the above, the AIT also I independently evaluated potential failure modes for the MSIV's and concluded that the failure of the ASCO dual solenoid valve was the only reasonable choice.

IV INVESTIGATIVE EFFORTS A. Mein Steam Isolation Valves (MSIV's) Description Two Main Steam Isolation Valves (MSIV's) are welded in a horizontal run of each of the four main steam line pipes; one valve is as close as possible to the inside of the drywell and the other is just outside the containment.

6 i

I -

i

\ ,- . ..

I ,

Attachment 3 shows a main steam line isolation valve. Each is a 26

inch Y pattern, globe valve. The main disc or poppet is attached to the lower end of the stem, Nonnal steam flow tends to close the valve, and higher inlet pressure tends to hold the valve closed. The bottom end of the valve stem closes a small pressure balancing hole in the poppet. When the hole is open, it acts as a pilot valve to relieve differential pressure forces on the poppet. Valve stem

, travel is sufficient to give flow areas past the wide open poppet greater than the seat port area. The poppet travels approximately 90 percent of the valve stem travel to close the main seat port area; the last 10 percent of valve stem travel closes the pilot valve.

A 45 degree angle permits the inlet and outlet passages to be streamlined. This minimizes pressure drop during nonnal steam flow and helps prevent debris blockage. The valve stem penetrates the valve bonnet through a stuffing box that has two sets of replace-able packing. A lantern ring and leakoff drain are located between the two sets of packing. To help prevent leakage through the stem packing, the poppet backseats when the valve is fully open.

Attached to the upper end of the stem is an air cylinder that opens and closes the valve and a hydraulic dashpot that controls its speed.

The speed is adjusted by a valve in the hydraulic return line bypass-ing the dashpot piston. Valve closing time is adjustable to between 3 and 10 seconds. The air cylinder is supported on the valve bonnet by actuator support and spring guide shafts. Helical springs around the spring guide shafts close the valve if air pressure is not available.

The valve is operated by pneumatic pressure and by the action of compressed springs. The control unit is attached to the air cylinder.

This unit is shown on Attachment 4 and contains air control valves and solenoid operated valves. Part 4 of Attachment 4 is the main pilot control valve (dual solenoid valve). This valve consists of a valve body with a solenoid attached to either end (see Attachment 5).

The dual solenoid valve provides control air to operate the four-way control valve (part 1) and the two-way control valve (part 3) and is used for opening and for fast closure of the MSIV. When both of the solenoids on the dual solenoid valve are energized the incoming solenoid air supply is directed through the valve body to shift the four-way control valve and the two-way control valve to the open position. In the open position the four-way control valve ports air through the three-way control valve (part 2) to the underside of the MSIV actuator piston while at the same time venting the over piston area of the MSIV actuator to atmosphere. With the two-way control valve in the open position the exhaust path through it to atmos-phere is closed. For a fast closure of the MSIV both solenoids de-energize shutting off the control air to the four-way control valve and the two-way control valve and venting them both to atmos-phere. When this occurs both valves will shift to the closed position. In the closed position the four-way control valve now directs air to the over piston area of the MSIV actuator and vents the under piston area to atmosphere. The two-way control valve now is in the closed position and also vents the under piston area of 7

l

the MSIV actuator to atmosphere. In this condition the MSIV is closed both by air pressure and by the helical valve springs. .

Slow closure capability (used for test purposes) of the MSIV is accomplished through the use of the single solenoid valve (part 5).

When the MSIV is open and the solenoid for the single solenoid valve is energized, air is directed to the three-way control valve (part 2) causing it to shift to the closed position. In this position the air that was directed to the under piston area of the MSIV actuator from the four-way control valve is stopped and a vent path for the under piston area is opened up through an air metering valve (part 9). The over piston area is still vented to atmosphere through the four-way control valve. In this configuration the air trapped in the under piston area is slowly bled off through the metering valve allowing the MSIV to slowly close.

Recote manual switches in the control room enable the operator to operate the valves. Operating air is supplied to the valves from the Instrument / Service Air System. An air tank (accumulator) between the control valve and check valve provides backup operating air.

B. Evaluation of Safety Significance Based upon the absence of plant conditions requiring an automatic main steam line isolation, the failure of the MSIV to close would not have any immediate safety significance. Had a main steam line isola-tion been required, isolation of the "Bd main steam line would have still occurred through the closure of the outboard MSIV, 1B21-F00288, which showed acceptable stroke time drring testing and no evidence of failure of the dual solenoid valve.

C. Operator Response The AIT reviewed the event chrono'cgy discussed in paragraph II.B.

against the requirements of the licensee's technical specificatiors as well as applicable operating and administrative procedures and determined that actions taken by operating personnel met the require-ments. The licensee declared MSIV 1821-F00228 inoperable, placed the reactor in Hot Shutdown within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />, Cold Shutdown within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, and notified the NRC of the failure and shutdown pursuant to a licensee comitment made following the MSIV dual solenoid valve failures of October 29, 1987, and November 3, 1987. In addition, the licensee contacted the Senior Resident Inspector and made a courtesy ENS report. The AIT determined that this event was not imedictely reportable pursuant to 10 CFR 50.72.

D. Troubleshooting Activities and Results To determine the cause of the mis-operation of the MSIV control system, the suspect MSIV 1B21-F00228 ASCO dual solenoid valve was removed and disassembled. Prior to the removal of the dual solenoid valve, a visual examination and electrical checks were made to document the as-found conditions. No anomolies were noted. The 8

l

material conditions found in the ASCO dual solenoid valve and related air system were as follows:

1. Material Condition of the Affected Valve The disassembly of the ASCO dual solenoid valve from MSIY IB21-F0022B was observed. The dual solenoid valve had been i rebuilt approximately one month earlier, with the exception of the "B" solenoid subassembly. At the time the corrective actions for the November 3, 1987, event were being implemented, the licensee only had three new dual solenoid valves available for replacing the eight being removed from the MSIV's. They were able to obtain sufficient rebuild kits to allow rebuilding the five remaining dual solenoid valves. The "B" solenoid subassembly w6s not rebuilt because it would have required cutting the "B" solenoid coil wires to gain access. Since the coil is not part of the rebuild kit this would have effectively ruined the dual solenoid valve and thus prevented its use. In addition, the licensee's determination of the root cause failure of the dual solenoid valves from the November 3,1987, event did not include any malfunction related to the components contained within the "B" solenoid subassembly.

The licensee disassembled the dual solenoid valve with extreme care and in accordance with the approved procedure. All condi-tions found were recorded, including a photographic record. GE and ASCO representatives also observed the process closely and examined the parts. Each part was put in a container and marked for identification. khen the "B" coil subassembly was disassem-bled, three pieces of foreign material were found on the top of the core (see Attachment 5). The three pieces of foreign matertul were stuck to the top of the core assembly and had to be peeled off. Visual, including microscopic, examination of the elastomer parts revealed no unusual wear or degradation.

Internal moving parts and elastomers did have a thin coating of a material believed to be the standard silicone lubricant.

Other conditions noted were that: (1) the EPDM disc pads showed the characteristic seat impressions (but not the annular dimple observed on the failed solenoid valve's seats from the November 3, 1987, event), appeared shiny and, according to the ASCO represen-tative, somewhat hardened as compared to new elastomers; (2) the Bisco Locaseal material used to seal the coil housing electrical lead conduit tubes had become activated indicating that the dual solenoid valves had been exposed to a neutron flux (core leakage and/or streaming paths). Perry personnel stated that all the inboard MSIV dual solenoid valves had activated Locaseal material which suggests that the normal neutron flux in the vicinity of them is fairly extensive during operation.

After completion of the disassembly and inspection of the MSIV IB21-F0022B dual solenoid valve the dual solenoid valves for MSIV's IB21-F0022C and 1B21-F00220 were disassembled and visually 9

examined (the 1821-F0022C solenoid valve was pre'viously rebuilt and the IB21-F0022D solennid valve was replaced with a new valve as a result of the November 3, 1987, event). The results of this inspection yielded similiar results to that discussed above for the IB21-F0022B dual solenoid valve. The licensee also perfomed a dimensional analysis of the three solenoid valves, including measurement of spring constants. The results of this analysis were submitted to ASCO, who provided written certifica-tion that all measurements were within design tolerances. The AIT requested copies of the design drawings so that an independent review could be performed, however, ASCO refused to provide these drawings to the AIT or to the licensee.

G9eral observations noted during the disassembly of the solenoid valves were as follows:

(1) The disassembly procedure referenced the procedure for maintenance of system cleanliness. Although no discrepancies were noted in this area, this procedure was not imediately available for reference.

(2) Contrary to ASCO installation instructions in Bulletin NP 8323, these dual solenoid valves are installed at approx-imately a 45 degree angle from vertical instead of vertical as prescribed and they are installed without a "street elbow" fitting in the exhaust port.

2. Laboratory Analysis of Particles Found in the Solenoid Valve The licensee was able to obtain the services of e local laboratory, Ricerca Incorporated of Painesville, Ohio, to conduct sophisti-cated non-destructive, destructive, and consumptive analysis of samples which were obtained from the November 29,1987, event.

On December 1, 1987, several members of the AIT accompanied Perry personr.el to the Ricerca laboratory facility to discuss non-destructive and destructive testing of samples which were obtained from the early November valve refurbishing, as well as particle samples which were obtained from air-line discharge samples sub-sequent to the November 29 failure. A listing of the samples which were given to Ricerca for testing appears in Attachment 6.

The licensee and Ricerca were well equipped to initiate testing on short notice because of the background work and planning that had been done in order to analyze samples that had been obtained from the previous Perry MSIV failures (October 29 and November 3, 1987). Ricerca and Perry staff had spoken to knowledgable personnel at other organizations that had been involved in the laboratory analysis of samples obtained from ASCO solenoid valves that had failed at Brunswick (USNRC, Franklin Research Institute, CP&L Shearon Energy Center Laboratory).

One of the most substantial discoveries that Ricerca made was the finding of a particle "nucleation site" on the EPDM 0-ring 10

which had been removed from the failed solenoid during the early November 1987 refurbishment. This finding was made using a Zeiss universal stereo widefield optical microscope (SV-8) equipped with a source of indirect reflected light. Attachment 7 shows the particles that were found in the solenoid valve.

Attachment 8 shows an 0-ring that had been removed from the solenoid valve during the early November refurbishment. The suspected "nucleation site" of the largest particle is also shown on Attachment 8.

Using the aforementioned microscope and optical micrographs, the Ricerca staff was able to determine the nominal sizes of all 3 particles that had been recovered from the failed solenoid valve. The nominal dimensions of those particles are as follows:

Length Width Depth Particle 1 4.2 mm 0.8 mm 0.3 mm Particle 2 0.6 m 0.4 mm 0.1 m Particle 3 0.9 mm 0.3 mm 0.3 m Analysis of the particles, 0-rings, discs, etc., was done using Scanning Electron Microscopy (SEM), Energy Dispersive X-ray Spec-troscopy (EDS), Backscatter Electron Imaging (BEI), and for sone samples, compression and hardness testing. The analysis and tests were conducted to enable comparison between unused off-the-shelf specimens and the materials that had been removed from the Perry plant subsequent to the November 3, 1987, event.

Molecular level examination of the three particles that were found in the solenoid valve and the EPDM 0-ring, which were removed during the ear 1y November 1987 refurbishment, clearly showed that they were made of the same EPDM material. The spectral plots (EDS) obtained from the particles and the 0-rings revealed the presence of: silicon (presumably from the silicon lubricant used in the valve assembly, either Parker Super 0-lube used by CEI, or Dow Corring lubricant No. 550 oil used by ASCO); copper and zine (the origin of which was presumably migration / contact from the solenoid valve's brass parts); and iron, nickel and chromium (also believed to have been introduced from contact with the solenoid valve's internal parts). The absence of large amounts of aluminum tends to discount the presence of desiccant (activated alumina).

Examination for any hydrocarbons associated with the particles and 0-rings revealed the presence of only a few hydrocarbon groups. This is believed to have resulted from the elastomer's manufacturing, and to not be indicative of large scale oil intrusion (e.g., compressor oil) into the instrument air system.

The analysis also found traces of other elements (lead, sulfur, calcium, potassium, chlorine) in the particles and 0-rings.

Those elements were believed to have been introduced in the elastomer during the manufacturing and/or curing process as trace metal catalysts and cross binding agents.

11

3. Analysis of Instrument Air System Air Quality In order to assess the quality of the instrument air supplied to the MSIV control units, the licensee conducted instrument air system blow down and particle count tests on December 4,1987.

One AIT member witnessed the tests. Instrument air samples were obtained for three one minute duration blows at the 3/8" and 1 5/8" lines which lead to the MSIV 1821-F0022B control unit. The three one minute samples from the 3/8" line had counts of 8, 6, and 5 particles which were 3 microns or less. There were no larger sized particles detected. The three one minute samples from the 1 5/8" line had counts of 35 and 40 particles which were 3 microns or less and counts of 1, 0, and 3 particles in the 3 to 10 micron range, with no larger sized particles detected.

The results of these tests indicate that on December 4, 1987, the instrument air system supplying MSIV 1821-F0022B control unit had particle counts which did not meet the ISA 7.3 require-ments but did meet the Perry nuclear plant's Final Safety Analysis Report commitment for air quality.

Ricerca analyzed many of the air samples that were obtained following the November 3, 1987, event. Analysis conducted included Infrared Spectroscopy (IR) for identification of organic compounds, particulate sizing, and gas chromatography for identification and quantification of condensable hydro-carbons. A listing of the samples that were examined is included as Attachment 6.

The air samples obtained as a result of the November 3,1987, event from the supply 'ines connected to the dual solenoid valves and natedactuators as MSIV-4ofthrough the failed MSIV's 7 and control MSIV-12 units through 19 (samples desig-),

on Attachment 6 contained particles in excess of 40 microns (see Section IV.D.2).

The analysis of samples MSIV-5, 6,13,15 and 19 revealed the l

presence of fibrcus materials. The licensee believed that the fibrous materials were the result of adverse sampling conditions in the drywell. Subsequent samples which used improved sampling

' techniques did not reveal any fibrous materials. In order to quantify particulate levels in the containment atmosphere that may have led to contamination of instrument air system samples, i the licensee took containment air samples in the vicinity of the MSIV 1821-F00228 control unit. The results of the particle count were as follows:

Size Number of Particles 0- 3 microns 815 l 3 - 10 116 10 - 20 35 l 20 - 30 15 l 30 - 40 7

> 40 7 12

' ,. i.:

Preliminary laboratory results of samples MSIV-4 through 7 and MSIV-16 and 17 characterized the particulates as three distinctly different types: white translucent, rust colored, and black metallic. Further analysis of those particulates has not yet been provided. A sub-sequent report characterized the majority of the particles greater than 40 microns from samples MSIV-4 through 7 and 12 through 19 as:

1 clear, crystalline-like 2 whitt , cloudy 3 dark with relatively few metallic-like particles and fibers.

The licensee concluded that the majority of the fibers found were dust particles that were due to room air, as opposed to having the instru-ment air supply as their source. The licensee also alluded to the possibility that sample contamination could also have been a source of the large particles, however, there does not appear to be any strong basis to discount the instrument air system as the source of the particles.

The analysis also noted that instrument air samples MSIV-1 and 3 revealed the presence of silicone lubricant and partially degraded thread sealant, "Rectorseal," but did not reveal the presence of "Neverseeze" thread lubricant. Because the method used to obtain air samples MSIV 1 and 3 was "crude", it could not be definitively established that the silicone oil and the "Rectorseal" that were found in the samples had come from the instrument air system, as opposed to being residue which carte from the fittings during the valve removal operation.

Two additional instrument air systm samples, MSIV-10 and 11 were taken at the containment penetr6 tion on November 7, 1987.

Those samples were analyzed for hydrocar usns, using gas chrom-a tography. Both samples revealed hydrocarbon levels to be less than or equal to 0.1 part per millior . One sample had no detect-able hydrocarbons. The results indicated an absence of large scale hydrocarbon contamination of the instrument air system at the containment penetration (on November 7, 1987).

4. Licensee's Conclusions The licensee evaluated the data gathered as a result of the troubleshooting program and concluded that the root cause of the failure on November 29, 1987, was a failure of the MSIV's ASCO dual soienoid valve. This failure was attributed to the binding of the "B" solenoid sub-assembly due to the presence of foreign material in the solenoid core assembly area. The source of the foreign material was attributed to a degraded 0-ring from which a sliver of material separated and fell into the core assembly area. Since this portion of the solenoid was not previously disassembled during its refurbishment, its presence was not detected. The licensee also concluded that this root cause 13

4 appeared to be different than the root cause of the previous MSIV solenoid failures (October 29, 1987, andNovember3,1987).

V. RECENT EVENTS INVOLVING MSIV SLOW CLOSURE / FAILURE TO CLOSE NRC Inspection Report No. 50-440/87024,Section V, discusses a number of events that have transpired, and their associated industry comunication, within the industry over the past 15 to 20 years involving MSIV slow closures or failures to close. This report also discusses the failures of several MSIV's to close within the maximum allowable time as delineated in the Perry Technical Specifications. The Perry events were initiated when, on October 29, 1987, the Perry Nuclear Power Plant was in the process of completing their Startup Test Program and was perfonning stroke time testing of the MSIV's. During this testing the inboard valve in the "0" main steam line failed to close within the maximum value delineated in the facility technical specifications. Two other MSIV's also failed, including the outboard MSIV in the "D" main steam line. In all cases, subsequent stroke times for these three MSIV's were within acceptable values. The licensee initially declared t'e PSIV's inoperable. However, based on the acceptable stroke times achie <ed after the second try, later declared them operable. The licensee believed that the failures were the result of impurities in the MSIV actuator control unit and that the impurities had apparently been dislodged and/or expelled during MSIV operation. Plant operation and the Startup Test Program were continued with the stipulation that additional stroke time tests on the MSIV's, to confirm their operability just prior to the performance of the full reactor isolation startup test be performed. On November 3, 1987, while performing the additional stroke time testing of the MSIV's both the inboard and outboard MSIV's in the "D" main steam line again exhibited unacceptable stroke times. The licensee reported the failure of the two MSIV's to the NRC and comenced an orderly shutdown.

As a part of their troubleshooting program the licensee disassembled the MSIV actuator control units from the three MSIV's that had previously failed. The results of this disassembly and inspection revealed that the Ethylene Propyiene Diene Honomer (EPDM) elastomers contained within the ASCO dual solenoid valves had been significantly degraded by exposure to high temperature and possibly hydrocarbons. An annular dimple was also observed on the seat material and resulted in part of the seat material being extruded into the exhaust orifice. This dimple, together with the deteriorated state of the seat material, indicated that the exhaust seat could be held in an "energized" position even though the solenoids had been de-energized, and would prevent the control air from being exhausted to atmosphere and therefore prevent the MSIV from closing.

The licensee subsequently replaced or rebuilt all eight MSIV dual solenoid valves. The plant was restarted on November 13, 1987, and the Startup Test Program, including the full reactor isolation startup test, was successfully completed.

One occurrence of an MSIV to close has occurred since the issuance of NRC Inspection Report No. 50-440/87024. This occurred at LaSalle, Unit 1 on December 17, 1987. The plant was in hot shutdown following a reactor scram resulting from a feedwater transient. The licensee was in the 14

i._

l process of closing the MSIV's to allow repair work of balance of plant equipment. The method being utilized to close the MSIV's consisted of I pressing the "test" pushbutton to allow the MSIV to slow close, taking the l hand switch to the "close" position, and then releasing the "test" push- i button. Nonnally this would result in the MSIV sta l when this was done for MSIV IB21-F0028C (outboard) ying closed, it reopened. The however, licensee attempted to close the valve again but with no success. They also verified that the status lights for the dual solenoid valve's solenoids indicated that they had de-energized. Subsequent to the failure, the licensee removed and disassembled the ASCO dual solenoid valve. Examina-tion of the valve internals revealed that the interfacing surfaces of the core assembly and the plugnut assembly (see Attachment 5) of the "B" solenoid had a yellowish, sticky, substance coating them. When the interfacing surfaces of these components were pressed together the core assembly would hang from the plugnut assembly with no support. No other anomalies were noted. This failure mode is consistent with a MSIV failure that occorred at Grand Gulf in 1985 (reported in Information Notice 85-17 and 85-17, Supplement 1) in which a simile

• substance was found coating the interfacing surfaces of the "B" solenoid core assembly and plugnut assembly. LaSalle subsequently replaced all eight MSIV dual solenoid valves.

VI AIT CONCLUSIONS The root cause of the observed MSIV failure to close on November 29, 1987, was a malfunction of the ASCO Model NP-8323A20E three-way dual solenoid valve caused by pieces of the body gasket (Ethylene Propylene Diene Monomer (EPDM) 0-rings) falling into the "B" solenoid valve sub-assembly and ultimately interfering with the ability of the dual solenoid valve to shif t to the de-energized position. The source of the pieces of body pasket material were deteriorated and degraded EPDM 0-rings that were replaced as part of the November 3,1987 event (Inspection Report No.

50-440/87024). This foreign material was introduced as a result of the maintenance activities by the licensee to refurbish the dual solenoid valves. The "B" solenoid valve assembly was not completely disassembled during the refurbishment and therefore the foreign material was r.ot detected. Subsequent to the valve's refurbishment the valve operated successfully four times. It is hypothesized that during the fifth opera-tion one or more particles lodged between the core assembly and the wall (see Attachment 5) thereby preventing proper valve operation. The tapping of the valve probably loosened the particle (s) enough to allow the spring forces to overcome the static friction / binding forces caused by the particle (s) which may have been lodged between the sliding core and the stationary solenoid base sub-assembly.

All evidence collected during the investigation indicated that the event was caused by the failure of the ASCO dual solenoid valve to shif t to the de-energized position. The evidence collected included the following:

r The MSIV in question failed to stay closed during two slow closure operability checks and failed to close during three fast closure attempts.

b. On the fifth closure attempt the MS!V stayed open for the approximate 15 m

i three minute period in which the control switch was held in the "close" position until the solenoid valve body was gently tapped.

At that time the valve closed in a normal stroke time (approximately 3 seconds),

c. The design of the control unit is such that the simultaneous failure of more than one of the air control valves would be required to cause the observed failure.

The AIT also concluded that the licensee was very responsive to the event of November 29, 1987, and proceeded in a methodical, well thought out, cpproach to resolving the root cause of the event.

VII AIT RECOP94ENDATIONS A. The AIT recommends that the NRC take actions to assure that ASCO provides the NRC with pertinent information necessary to assure proper operation of such safety-related valves in accordance with plant safety analysis. Specific design, operation, and maintenance infomation should be obtained from ASCO. (It may be necessary to establish a special/ vendor inspection team to ascertain some of the infomation which ASCO has been reluctant to provide.) Examples of such information include:

Valve design specifications including acceptable tolerances on moving parts.

Material specifications including shelf-life and inservice life expectancy.

Design analysis including force analysis showing design margin for operation in the presence of "dirty air" Valve cycling frequency required to assure successful operation Maintenance of internal cleanliness during assembly / disassembly Sticking due to overheating caused by local hotspots from steam leaks and/or faulty / damaged / missing themal insulation or inadequate HVAC Effects of these conditions on qualified life (EQ)

Effects of prolonged exposure to neutron flux

b. The AIT recomends that the information notice discussed in Inspection Report 50-440/87024 to inform utilities of the recent MSIV failures be expanded to stress the merits of replacing safety-related solenoid i operated valves rather than refurbishing them.

! The infomation notice should also stress that it is importan' that if the option to rebuild the ASCO's is taken that it should be a complete rebuild to preclude a problem similar to that encountered at Perry on November 29, 1987.

VIII EXIT INTERVIEW The AIT net with licensee representatives (denoted in Paragraph I.D.)

infomally throughout the inspection period and at the conclusion of the inspection on December 4,1987, and summarized the scope and findings of l

the inspection activities.

16

~

l.- ',',

~

i.

The AIT also discussed the likely infomational content of the inspection report with regard to documents or processes reviewed by the inspectors during the inspection. None of the areas expected to be contained in the report were identified by the licensee as proprietary. The licensee acknowledged the findings of the inspection. -

IX STARTUP REVIEW On December 4,1987, the licensee met with members of the AIT and a representative cf Region III management to discuss their plans for startup and to obtain NRC approval. As a result of this meeting the licensee comitted to perform a number of actions both prior to startup and subsequent to startup. These comitments are detailed in a letter (PY-CEI/01E-0296L) dated December 4,1987, from Edelman, CEI, to Davis, NRC. The following is a summary of these actions:

A. Prior to Startup ,

1. Replace the ASCO dual solenoid valves on all eight MSIV's with new ASCO's.

2. Cycle all eight new ASCO dual solenoid valves a minimum of ten times as part of tht. retest activities.

B. Following Startup

1. Institute administrative controls for all future Class IE ASCO solenoid valve work to require the use of new valves or the complete disassembly and cleanout to ensure that no particles are introduced during the rebuild process.

2. Complete the corrective actions previously discussed in a letter (PY-CEI/01E-0289L) dated November 13, 1987, from Edelman, CEI, to Davis, NRC, with the following clarifications:

e. The dual solenoid valve inspection discussed on page 5 of the enclosure will be performed during ar outage of opportunity prior to the end of October 1988,

b. The modified monthly slow closure surveillance test discussed on page 5 of the enclosure will be performed on a staggered basis as follows: i (1) Until the January 4,1988, outage, the test will be perfomed weekly, staggered between the inboard and outboard MSIV's.

(2) for a one (1) month period following the January 4, 1983, outage this test will be performed once every two (2) weeks, again staggered between the inboard and outboard MSIV's.

4

?

17 m ._ _

- '~-

. .n ,

op .q - '

.. .:a ,

. . - t s

^

'l i

s

\

3-l l

5 On December 4,1987, 1Reg)

RIII.87-24.(Attachment based iononIIItheir released corrective the licensed actions, f ru r, CAL ' l commitments, and the preliminary results of the AIT inspection. '

l Region III also concurred with the licensee's request to allow l

, the plant t., startup. The above was documented in a letter i (Attachment 9) from Davis, NRC, to Edelman, CEI, dated December 8, i 1987.

i J

P l

4

/ l t

i  !

I

~

L 18 ,

l _ _ - - . .- - -. . _ _ - . -__. --_ - _ - _ _ _ . _ _ _ _ .

~, 7 tp s

CONFIRMATORY ACTION LtTTER g.

. .  ?

.. , s

~

Attuchment #1 NOV e ten CAL Rill-87-24 Docket No. 50-440 Docket No. 50-441 The Cleveland Electric Illuminating Company ATTN: Mr. Murray R. Edelman l

Vice President '

l Nuclear Group Post Office Box 5000 Cleveland, OH 44101 '

Gentlemen:

This letter confirms the telephone conversation on November 'a0,1987, Mwein Mr. H. Miller of this of fice and Mr. A. Kaplan of your stafi regarding the Main Stean isolation Valve (MSIV) failure occuritag at the Perry Nuclear Power Plant Unit I on November 29, 1987. With regard to the m;tters discussed, we understand that you will:

\

1. Take those actions necessary to ensure that complete documentary evitence of the as found" condition of equipment being inspected is it.aintained, s

2. Provide a step by step troubleshooting program to establith the root cause of the M51V failure.

3. Not disturb any components that offer a potential for bein] the root -

cause including power sources, switches, solenoids, and the air system directly feeding the MSIVs until that action is approved by the 1cader of the NRC Augnented Inspection Team (AIT) which has responsibility for examining this matter. We recognize that, per t,ur authorization of November 29, 1987, the air pack was removed from MIIV IB21-F022B and the asse:iated ASCO dual solenoid 3-way pilot valoc was disassembled vith

) the AIT team leader present.

4. Except as dictated by plant safety, advise th0 NRC AIT tom lude pri;r to conducting any troubleshooting activities. Such notification stould be provided soon enough to allow time for the te m leader to assign an inspector to observe ectivities.

5. Subnit to NRC Regior. III a formal report of your findings and coelesioni, withir. 3C days of receipt of this letter. iou may extend the submitti,1 ht+ c' t6 t f or:41 rcr ort rua s td ir. thc hE Corfirr ; te , A:tiv !.e t:(-

c)t(t !.;vu.tir 4, 19 f] , to withir 30 crys of rectipt cf this lett+.

\

CONr]g.U. TORY ACTIO!: LfiTf6

CONFIRMATORY ACTION LETTER The Cleveland Electric Illuminating Company 2

g,g , , g .

We also understand that Perry Nuclear Power Plant Unit I will not be made critical witnout the cori,urrence of the Region Ill Regional Administrator or his designee.

Issuance of this Confinnatoiv 1. tion Letter does not preclude the issuance of t an order requiring implementauon of the above conrnitments.

Please let me know imediately if your understanding differs from that set out above. ,.

3 Sincerely, Ceirir.d cir.td t:7

1. bn Dais A. Bert Davis Regional Adninistrator cc: f. R. Stead, Manager, Perry Plant Technical Department M. D. Lyster, Manager, Perry Plant

i Operations Department Ns. E. M. Buzzelli, General

, Supervising Engineer, Licensing and Compliance Section DCD/DCB (RIDS)

Licensing fee Management Branch Resident Inspector, Rll!

Harold W. Kohn, Ohio EPA Terry J. Lodge, Esq.

James W. Harris, Stau. of Ohio Robert M. Quillin, Ohio Department of Hehlth State of Ohio, Public Utilities Comist,ior.

R. Cooper, EDO W. Lanning , NRP.

F. Miraglia, NR0 G. Holahan, NRR M. Virgilio, NRR J. Partlow NR?

t. Conr.auchton, SRl J.

Strasr.a . R111

. Sucharshi, Rll) 1

" ic t . EM

c. Liettrr:r, 6 ,;

, R;11 01 Rlll Rll! PL1 R M F,lll Rlll Rll R I .l l Rll)

-7r:n.:y'...

C 1 21rno rce /,9. tra ,

4l. 14

~

w.:3,;h '

i l hm )cht G-Es c.r.

ik a!r:

Tch MNhut h pv iellr [nh

.6 .A V i r c il i c./ hM. t J . ll Tl 0 ,i '. i %' a ,T" '$-

.R > y. 7 Je- -

4 to:,r x ::ce i mlu, tInu.

j

'_2IL_f Y _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - - _ _ _ - - _ . - - - - - - - - - -

)., e

• pt # "*% U881780 STAfts

.. *k NUCLEAR CEGULATORY COAsMIS$10N ItEQlON lll Attachment $2 g ,f

• p 799 moosavtLT mo Ao

• $ eLEN ELLYN ILLIMols set 37 DECe2tm HEMORANDUM FOR: R. D. Lanksbury, Team Leader, Perry Augmented Inspection Team (AIT)

FROM: Hubert J. Miller Director, Division of Reactor Safety

SUBJECT:

AIT CHARTER Enclosed for your implementation is the Charter developed for the inspection of the events associated with the Perry MSIV failure which occured on November 29, 1987. This Charter was prepared in accordance with the NRC Incident Investigation Manual and the draft AIT implementing procedure issued for use on October 2,1987, and is based on the discussions you had with Region III personnel on November 30, 1987. As stated, the objectives of the AIT are to communicate the facts surrounding this event to regional and headquarters management, to identify and communicate any generic safety concerns related tn this event to regional and headquarters management, and to document the findings and conclusions of the onsite inspection.

If you have any questions regarding these objectives or the enclosed Charter, please do not hesitate to contact either myself or Monte Phillips of my staff.

/,, , q, s . G i

Hube t J. Miller, Director Div1sion of Reactor Safety

Enclosure:

AIT Charter

, cc w/ enclosure:

l A. B. Davis, RIII I C. J. Paperiello, RIII F. J. Miraglia, NRR J. C. Portlow, NRR I

C. E. Rossi, NRR l G. Holahan, NRR W. D. Lanning, NRR M. J. Virgilio, NRR

! R. W. Cooper, EDO l K. A. Connaughton, SRI l

i

s c .

PERRY MSIV STROKE TIME FAILURE AUGMENTED INSPECTION TEAM (AIT) CHARTER INVESTIGATE:

1. Failure of MSIVs to close/close within Technical Specification limits.

2. RootCause(s).

3. Interaction of prior maintenance / replacement activities to the event.

4. Broader implications, e.g., other systems, other valve / components.

5. Event reporting.

6. Conclusions.

QUESTIONS FOR PERRY AIT

1. Failure of MSIVs to close/close within Technical Specification limits.

(11/29/87) 1.1 What was the sequence of events?

1.2 What were the closure times generated during the surveillance? .

1.3 What operator actions were taken during the event? Were they appropriate?

2. RootCause(s) 2.1 What was the root cause of the event?

3. Interaction of prior maintenance / replacement activities to the event.

3.1 What is the material condition of the affected valves and inter-connected air systems as it would affect the valve closure function?

l 3.2 What is the affect of the Asco valve refurbishment / replacement on the valve failure?

4. Broader implications, e.g., other systems, other valve / components.

4.1 Is a IEIN or IEB warranted as a result of this event?

4.2 Are there other valves or instruments that require investigation?

1 1

t

c ,

5. Event reporting.

5.1 Was the event reported as required?

6. Conclusions.

6.1 What corrective actions are porposed, and are they adequate?

6.2 Examine generic implications to other plants and advise NRC management subsequent to the site inspection.

6.3 Document inspection findings in accordance with draft manual chapter 0325.

I i

I I

1 l

• ATTAQDDT #3 I I i A u

• AIM CYLINDE R

. m . i HYDRAULIC gl ,

DASH POT

/ ( 'l SPRING GUIDE

\

ll SPE ED CONTROL VALVE ACTUATOM $UPPORT AND

~

HEllCAL SPRINGS e h l

\ kl 1 \: s I. C. }

ll ll ll

i SPRING SE AT MEnA3E R h h STEM PACKING s

(

1 / / LE AK OF F CONNECTION

/ BONNET tl0LTS J f I."L9 70 5 r Ph f l y BDNNET L ANTERN RING CLEARANCE u s I -

jw e

.ut N, m x fEh =

PILOT SPRING N PILOT  % ,

gony f

P5PPET l PLUG.M AIN DISKI t

,, MAIN VALVE SE AT PILOT SE AT N

FLOW PERRT NUCLEAR POWER PLANT G THE CLEVELAND ELECTRIC lLLUMIN ATING COMP ANY Hain Steam Isolation Valve Figure 5.4-8

4 M ems samminamass

. wen *

• I '.

0 (*e- se.mTwt. o ,

• a -

seween.mus)= = c.a ween e , ATIAQMENT (* ,

@ g e = = t m aw m T a s.ase o w ,e c -

Omtecsema tammin imustisss se cm,iseases on, a

@ (o Tub est.Weroti 19ss,tse Pgst putshst. 4485 t.4 De L4 tsestet m -

• @ ga has eman.haht an am OPtehtm taum(heane 9%et (JhfTT*A teu(1ts84m taptender.turTOs OB 4%CEO

'@ (* weet ==n.gassee womwn msg (geomaa esse come, wwu we. m eenanneet omrimi m (pesee . .

@ sT emessit Cawnssessut Phas empeeL tanst (glated.

1hma Comm) esos e issonnue,ca.

h [e putteam tauremosta nas twese 'tmug(eresus '

wits cams) isrs m emiatom. te.

@, adtammaates (se ChsMuSAR Pte Titannae, tweesges .

er Tiet es)niet m tienen tervenatsas .

@ {' aan esmans tout tm aan,gts. cwt w tueussa

=== m ereta m#t (fustatesEn m Cmteg) &- +

l

• p% ~

@ ase tver.4 tiana mewwu m ovw se CWd & ="

am r"es>en a E_!ill we l i llflf E

• a*

• 4 Snes o t a ta k

TD CPtw M4 -CUM 4 %sTTO4 C1 eaaage g ,gton gg ToItew ww .ertha turitten Lt 664 f.) -~

Kt 7-0 10 tuetat wa(% chilt . cum 4 tefus O ($ir100 C1 een C1 Gamri) m . m m m w . . m ,.0 ,,,ce . s .mn .

I p [ $j .- {

gg

. * =- a r,

~

d U

1 a g-, /

a u

(%84a.8918D0 Cpthsha & G8%su6 8 wstssacea) {

qt ,

5--

tatutagss etmetet toew egosassuS*".e* *

• t's uus.68 at, u C

'*. 1 e

y [ u T \ [T w

( l I

= ._._j -

-E !, ,\ -

eq uus so sa.u 8

[ _ R, ..-

. . - - e y

w v~ 9 O r m

! =a.= cama Yins = masuta --~

y[ a i .

i= ==.Ts. =, r. . . . . .i g I '==a s

--E f

+

i 'a'*

. l i

p ..- . . .. , I i i l

,,, m ,, m e.,u.

i j .r

,e,...o -. r.,..= _cs, .

uc -. e. e,i ~ ~ -

j i

,,1,-___......,

i _ =

- .... , 3 g

• 9erest has tehAPt451 %upen gggq teTTH. m ipit4 eneTweg ,*

K.1

. . . .. ..... ..

• 4 e

y .

• ATTAQOENT #5 e as4 man suamas n-l sot.cwoto "A " -

h } (f C#k.i V'

I l

a j

l

+

. d

'lj' I e ..

ll "

.- l f

W f W Y

{

C i

ll y

.d.

yg

/

>lf' 4,

c.g,ug - -

O

) t,>,

. - t_

0 f

)' % w%. p

/ Y%ug

(

'gr J g'*j> j & f%,

,/

, <[ - 1 d' I

p/, _

w j o 4mc, s - Pt ayd "*' %

r l, l ,

jg !II

, , i ,

i

' c M -

iI l

_ stiver, el l' l, I o ,,3 fc..A I

he,e,

. e ,,;,- u.

ru

[ !I Sotmac '4 ~

[{j~~!ji o i!

--bg j cm. --g

][, --. y t ll q 1, l __

o 3--- ,

i s

[{

T <

,-t Sheet 1 of 2

>). . ' *, oc

. . n il __ T I L g@ :__

g i  ! __ ,

II '

u 3 i-- _._

R l_

lI E r I3 I it.l g . . . .

NQ 4 w 9 t e

• I s I*

o

'h k

O k's}x

• d D a a :1 a 3

! d?j N $c.h., , R.

c 4

N o

Trop 2 'my Y c o

)

/-

. . - . Y t ' f 1 IE 4 o

ew st i k 44 4 N

e e I s a

.h t, ,Il b

. '4 2  %  %

4 a

_ N 'A\ a >

• c

-- 71 p 5 _ . _ _

k v

[ Ik-w , E g

Trop c 1'o 3

(

Q

{

o '$ 6

. . . - .... ~

Sheet 2 of 2 l ^

} , *, ' ,

Att chae:t #6 SAMPLE DATE/ TIME DESCRIPTION ANALYSIS MSIV-1 11/6/87:1115 B21-F028B Deposits from 1 5/8" air hose. IR MSIV-2 11/6/87:1545 B21-F028B exhaust port (unknown fluid) IR MSIV-3 11/6/87:1115 Fitting from B21-F028B v/ foreign mat'l IR inside (black solids and oily fluid)

MSIV-4 11/6/87:2101 B21-F022D: = 0.1 ft.3 solenoid supply PSC collected on 0.459 filter paper.

MSIV-5 11/6/87:2108 B21-F022D: = 0.1 ft.3 solenoid supply PSC collected on 0.459 filter paper.

MSIV-6 11/6/87:2125 B21-F022D: = 0.1 ft.'3 actuator supply PSC collected on 0.459 filter paper.

MSIV-7 11/6/87:2135 B21-F022D: = 0.1 ft.3 actuator supply PSC collected on 0.45p filter paper.

MSIV-8 11/7/87:0800 Rectorseal ta Thread sealant sample. IR te MSIV-9 11/7/87:0800 Neverseeze Thread lubricant sample IR MSIV-10 11/7/87:0730 P52-F556: Instr. air at Containment GC penetration (outside). 10 min. blov-dovn, 5 min. purge of sampler.

MSIV-11 11/7/87:0745 P52-F556: Instr. air at Containment GC penetration (outside). 10 min. blov-dovn, 15 min. purge of sampler.

MSIV-12 11/7/87:1151 B21-F028B: Solenoid supply, =0.lf t.3 PSC on 0.459 particulate filter.

MSIV-13 11/7/87:1202 B21-F028B: Solenoid supply, =0.lft.3 PSC on 0.45p particulate filter.

MSIV-14 11/7/87:1214 B21-F028B: Actuator supply, =0.lft.3 PSC -

on 0.459 particulate filter.

MSIV-15 11/7/87:1220 B21-F028B: Actuator supply, =0.lft.3 PSC on 0.45p particulate filter.

MSIV-16 11/7/87:1503 321-F028B: Solenoid supply, =0.lft.3 PSC on 0.45u particulate filter.

~

G

. '. ' a -

SAMPLE DATE/ TIME DESCRIPTION ANALYSIS MSIVI-17 11/7/87:1521 821-F028B: Solenoid supply, a0.5ft.3 PSC on 0.45p particulate filter.

MSIV-18 11/7/87:1537 B21-F028B: Actuator supply, .0.1ft.3 PSC on 0.45p particulate filter.

MSIV.19 11/7/87:1553 B21-F028B: Actuator supply, a0.5ft.3 PSC on 0.45p particulate filter.

MSIV-20 11/16/87:1600 B21-F028D: Solenoid valve body.

MSIV-21 11/16/87:1600 Solenoid Rebuild Kits (3 kits v/ elastomer parts)

MSIV-22 11/16/87:1600 B21-F028A: Gaskets, disc, core assembly.

MSIV-23 11/16/87:1600 B21-F022B: Gaskets, disc, core assembly.

MSIV-24 11/16/87:1600 B21-F028B: Gaskets, disc, core assembly.

MSIV-25 11/16/87:1600 B21-F022C Gaskets, disc, core assembly.

MSIV-26 11/16/87:1600 B21-F028C: Gaskets, disc, core assembly.

MSIV-27 11/16/87:1600 B21-F022D: Gaskets, disc, core assembly.

MSIV-28 11/16/87:1600 B21-F028D: Gaskets, disc, core assembly.

MSIV-29 11/30/87:1230 B21-F022B Plungers, solenoid "B" MSIV-30 11/30/87:1230 B21-F022B: Core Assembly, seats & gaskets.

MSIV-31 11/30/87:1230 B21-F022B: Valve body.

MSIV-32 11/30/87:1230 B21-F022B: Foreign Material from Plunger area "B" solenoid (3 particles total)

MSIV-33 11/30/87:2200 B21-F022B: Instrument Air grab sample from 2" supply.

MSIV-34 11/30/87:2215 B21-F022B: Instrument Air grab sample from Solenoid supply line.

MSIV-35 12/03/87:1230 B21-F022B: 0-ring removed from "A" solenoid side of shuttle valve.

MSIV-36 12/03/87:1230 B21-F022B: Valve body gasket.

MSIV-37 12/03/87:1230 B21-F022B: Adapter nut and 0-ring.

MSIV-38 12/03/87:1230 B21-F022B: "A" solenoid plug nut 0-ring.

s *

• Attachment #7 l ,

l I

i

.b l i l

,J 4

J 1

.l 4

i

.Tu..

., ~ e ' .;-  ?

y , .re . ,e A-

, l'

,sec ** - *Y7 1, 8

4.g

,_ . , ~ i e ' '

wg ,

s u 8%g. . i . u. . g. . .

3 . . ,+x l

' ~ seyv.w i s  ;

, p. .m,-v%.  :.n . x c- ~

.s .f.,.r .1< e 4

.2,*,- i n .,

Y f... %m N --

c -

... y~

-m. . .

r

• y e .. i 7

• [ f;g .4 s.

y

. 7.! '

f 4

,o. . a 4 .

'#%p:*.' 9 gg$ 3 ye' . $ , : .k ' ..

1

,.=g ; ; , Yq,,$.p .

e ., -

l v,;7-2.%r:..as,Q",v '

(

)

in?.$ 'e

1 i

j MSIV 32 9.8X 1mm = 102 pm

] Particles 1, 2 & 3 i

i i  !

l i

[

i ,

l-l I L i Figure 1. MSIV 32 Optical Microscopy i 4

q l,

I l

l 1

i.

i ,.

J i i

i f

J l

( 1 s

• Attechnent #8 l

, . .. s l , m ., . ,

l I I l

i .

'w, l

l J l 3, i l

l

/

/

/

'A I

i i

f'. ,.:. ;

. . ~ ,n i

.j,

,I a. 9.BX 1rna = 102 pm  ;

1 i

l

! l  !

\

! I i

I I

j l

p I  !

I I

I i

t I

b. 25.8X 1mm = 39 pm t

! I i  :

Figure 4. MSIV 23 0-ring Optical Microscopy l 1 1 1

I

(_ . I

(

s 5 N88 W"'

,- *[/,9**"'*%,k NUCLEAR REGULATORY COMMISSION Attachment #9

f. > q neoeow m 5

• 790 moostvsLT mead sism sLLvn eLLmois sein DEC a1987 Docket No. 50-440 Docket No. 50-441 The Clevaland Electric Illuminating Company ATTN: Mr. Murray R. Edelman Vice President Nuclear Group Post Office Box 5000 Cleveland, OH 44101 Gentlemen:

SUBJECT:

CONFIRMATORY ACTION LETTER NO. CAL-Rill-87-24 On November 29, 1987, a Main Steam Isolation Valve (MSIV) failed to close. This is the same type of event which occurred on October 29 and November 3,1987, for which Confirmatory Action Letter No. CAL-Rill-87-19 was issued. As a result of the most recent event, an Augmented Inspection Team (AIT) was dispatched to the site and a Confirmatory Action Letter (CAL-Rill-87-24) was issued on November 30, 1987, to document our understanding that you would perfom the following actions pursuant to the review of the MSIV failure:

1. Take those actions necessary to ensure that complete documentary evidence of the as found" condition of equipment being inspected is maintained.

2. Provide a step by step troubleshooting program to establish the roo, cause of the MSIV failure.

3. Not disturb any components that offer a potential for being the root cause including power sources, switches, solenoids, and the air system directly feeding the MSIVs until that action is approved by the leader of the NRC AIT which has responsibility for examining this matter. We recognize that, per our authorization of November 29, 1987, the air pack was removed from MSIV 1B21-F0228 and the associated ASCO dual solenoid 3-way pilot valve was disassembled with the NRC AIT Team Leader present.

4. Except as dictated by plant safety, advise the NRC AIT Team Leader prior to conducting any troubleshooting activities. Such notification should be provided soon enough to allow time for the Team Leader to assign an ir.spector to observe activities.

5. Submit to NRC Region 111 a fomal report of your findings and conclusions within 30 days of receipt of the November 30, 1987, Confirmatory Action Letter. You may extend the submittal date of the fomal report requested in the NRC Confirmatory Action Letter dated November 4,1987, to within 30 days of the receipt of the Ncvember 30, 1987, Confirmatory Action Letter.

2 Nw k w.

n

(

,,,' o 'c

- *4 '

DEC s;;t-,

The Cleveland Electric Illuminating 2 Company ,

The November 30, 1987, Confirmatory Action Letter also specified that the plant would not be restarted without the concurrence of the Regional Administrator or his designee.

With respect to Items 1 through 4, you have completed all of the specified actions and these have been evaluated by the AIT. Their report will be issued shortly. With regard to Item 5, we understand you will submit to Region III a formal report as specified.

Based on a review of your corrective actions and commitments as specified in your letters dated December 4,1987 (PY-CE!/01E-0296L), and November 13, 1987 (PY-CEI/0!E-0289L), and based on the results of our AIT inspection, we believe you have established adequate plans for continued safe operation of the plant and for final resolution of this matter. Based on these reviews and conclusions, NRC concurrence was given in a telephone conversation between H. Miller and A. Kaplan on December 4, 1987 to restart the Perry Plant. This letter confirms that conversation.

Sincerely,

.o -

,0.Ad il m A. Bert Davis Regional Administrator See Attached Distribution

~

4 ,

, s. ~e .

ge

• D ,,s. .e J

~ " ' '

'The Cleveland Eleci;ric l'Iluminating 3 Company cc: F. R. Stead, Manager, Perry Plant Technical Department M. D. Lyster, Manager, Perry Plant Operations Department Ms. E. M. Buzzelli, General Supervising Engineer, Licensing and Compliance Section DCD/DCB (RIDS)

Licensing Fee Management Branch

• Resident Inspector, Rll!

Harold W. Kohn, Ohio EPA Terry J. Lodge, Esq.

James W. Harris, itate of Ohio Robert M. Quillii., Ohio Department of Health State of Ohio, Public Utilities Comission R. Cooper, EDO W. Lanning, NRR F. Miraglia, NRR G. Holahan, NRR M. Virgilio NRR J. Partlow, NRR J. Strasma, RIII I

L l