ML20148M396
Jump to navigation
Jump to search
| ML20148M396 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 03/28/1988 |
| From: | Miller H NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | Martin T Office of Nuclear Reactor Regulation |
| Shared Package | |
| ML20148M398 | List: |
| References | |
| NUDOCS 8804050356 | |
| Download: ML20148M396 (3) | |
See also: IR 05000440/1987024
Text
{{#Wiki_filter:.
VAR 2 81988
MEMORANDUM FOR: T. T. Martin, Acting Associate Director for Inspection and
Technical Assessment, NRR
FROM: Hubert J. Miller, Director, Division of Reactor Ssfety,
Region III
SUBJECT: IMPLEMENTING RECOMMENDATIONS FROM'TWO PERRY AIT INSPECTIONS -
'
Attached are copies of two recent AIT reports describing the failure of several
Main Steam Isolation Valves (MSIVs) at Perry to close or ; lose within the
maximum allowable time delineated in the facility technh. ' specifications.
In addition to the events described in the attached reports, at least one other
related event has occurred. This event occurred at LaSalle Unit 1.
Contained within the body of these reports are a list of recommendations
made by the AIT. As indicated in the attached reports and their recommen-
dations, problems with Automatic Switch Company (ASCO) solenoid operated
valves (S0Vs) have been relatively widespread. Each affected licensee appears
to be dealing with these problems in a piecemeal manner. The NRC, in turn,
is largely constrained to dealing with licensees individually as problems
arise. Several past generic communications on these problems have been issued
but they appear to have had little, if any, lasting effect as evidenced by the
fact that the problems continue to occur. Adding to the problem is the fact
that the vendor, in dealing with individual licensees, has not been cooperative
in all cases. It would appear that the most effective and appropriate approach
to resolving the issues raised by the AIT would be to pursue them with one or
more of the industry self-improvement groups, such as NUMARC. This course of
action would help assure that adequate, coordinated, resources were brought to
bear in promptly resolving the issues.
Attached for your convenience is a sumary of the recomendations made
by the AIT that 1nvolve action by the NRC if they are to be utilized.
Also condensed from the AIT reports are the basis for the recomendations.
While several of the recomendations would appear to be most appropriately
handled by the industry, the third recommendation would also appear to warrant
further review and evaluation by NRR. With regard to the fourth recomendation,
Region III has submitted a proposed Inforr.ation Notice. Region III will
continue to follow the actions of LaSalle and Perry as they continue to work
toward final resolution of the events that occurred at their facilities.
.E!!Ai. SSiED in BU25.I J. MilliR
b Hubert J. Miller, Director
Division of Reactor Safety
See Attached Distribution
L
I
s ury/mc
RI11/&
PWIlips .Wr
PIII
t
RI
C iss t mos
R
[
i ler
f $U \\
8804050356 880328 0 [[,g/
DR ADOCK 0500
>
-
__ - _ _ _ _
MAR 2 a 13gg
T. T. Martin 2
Distribution
Attachments:
1. Summary of Perry AIT
Recommendations Requiring
NRC Action
2. AIT Report No. 50-440/87024(DRS)
3. AIT Report No. 50-440/87027(DRS)
cc w/o attachments:
A. B. Davis, RA
E. G. Greenman, DRP
-
-
Equum mmumm __.-mmmmmmW
.
Attachment 1 -
SUMMARY OF PERRY AIT RECOMMENDATIONS REQUIRING NRC ACTION ;
* Action should be taken to obtain indepth design infonnation from ASCO
regarding their SOVs. The basis of this recommendation is that the
failure mechanisms of tne large number of ASCO S0V failures in safety
systems have not been fully understood and the design tolerances,
characteristics, calculations, and operating margins of the 50V have
not been made available to the licensees or the NRC. This information
appears to be important to developing a full understanding of the
various failures and to identifying with confidence what maintenance
and testing is required of these valves given their apparent fragile j
design. ;
*
The rapid repair of steam leaks and avoidance of high localized
temperatures which could lead to degradation and failure of seemingly ,
qualified equipment should be addressed. The recommendation also ,
suggests that technical specifications and LCOs regarding containment i
and steam tunnel temperatures may require modification. The basis of ,
this recomendation is that it is believed that the existence of
steam leaks at Perry was a precursor tc the initial set of MSIV
failures that occurred.
* The equipment qualification (EQ) testing of ASCO S0Vs should be
revisited to assure that the testing properly accounts for normal
plant operating conditions (including normal operation of equipment),
anticipated transients or equipment malfunction, design basis
accidents, and combinations thereof. The basis of this recomendation
is that the EQ testing and qualification of the Perry ASCO S0Vs ,
'
may have not reflected actual usage of the equipment under day-to-day
conditions that could normally be expected to occur (e.g., steam
leaks, cycling interval well in excess of daily, etc.). As a result,
problems with the S0Vs under these conditions are only now coming
to light instead of being detected prior to their qualification.
*
An Information Notice (IN) should be issued to alert the industry )
,
to the more current failures and what kinds of failure mechanism (s
are postulated to exist, what should be looked for if S0Vs are ,
disassembled, and a recommended testing program that can be used to l
help detect failures prior to those occurring during an actual
transient as discussed above. It appears that licensees in general,
are not following the intent of ASCO's recommendation for cycling
the S0Vs periodically to ensure that they will function and that,
based on another ASCO failure that occurred in Region III (not
discussed in the attached reports), licensees are not familiar with
all the potential failure mechanisms and therefore overlook evidence
of the failure during their inspection. In addition, consideration
should be given to alerting industry self-improvement groups (such
as NUMARC) that industry initiative needs to be taken to resolve this
issue. The recomendation riso suggests that a Bulletin should be
considered if further infc mation indicates the specific actions
licensees should take. The basis of this last recommendation is that
the NRC over the past 15-20 years, has issued several forms of
comunications to alert the industry to these potentially significant
failures that are occurring. However, as evidenced by the fact that
these failures continue to occur, it appears that the industry has not
been aggressive in correcting the problems.
- - . - - - . . - .
. .
.
_ . . . . _ _ .
"k-
F
* ~
~ }
ATTACH 1!ENT 2
e e
i -
Docket No. 50-440
The Cleveland Electric Illuminating
Company
ATTN: Mr. Alvin Kaplan
Vice President
Nuclear Group
10 Center Road
Perry, OH 44081
Gentlemen:
The enclosed report refers to the special onsite review conducted by an NRC
Augmented Inspection Team (AIT) composed of R. D. Lanksbury, K. A. Connaughton,
and S. D. Eick of this office and H. L. Ornstein (AE0D), H. K. Shaw (NRR), and
J. J. Stefano (NRR) on November 4 through 9,1987. The review was in response
to the failure of several Main Steam Isolation Valves to close within the maximum
allowable time as Jelineated in the Perry Technical Specifications. Operation
of the Perry Nuclear Power Plant, Unit 1, is authorized by Operating License No.
NPF-58. The essence of our findings were discussed with Mr. A. Kaplan and others
of your staff at the conclusion of the inspection.
The enclosed copy of our inspection report identifies areas examined during
the inspection. Within these areas, the inspection consisted of a selective
examination of procedures and representative records, observations, and
interviews with personnel.
The major purpose of the AIT was to conduct a timely, thorough, and systematic
inspection of the event in order to determine the cause(s), conditions, and
circumstances pertaining to it, and to comunicate to NRC management the
facts and safety concerns related to the event. While primarily a fact finding
mission, issues identified by the AIT may be examined for possible enforcement
in subsequent inspections.
In accordance with 10 CFR 2.790 of the Commission's regulations, a copy of l
this lettcr and the enclosed inspection report will be placed in the NRC
Public Document Room.
We will gladly discuss any questions you have concerning this inspection.
Sincerely,
s
q.. .
.. .. . . . . . , ,
Hubert J. Miller, Director
Division of Reactor Safety
':3 9)G3 &f),
Encio re: Mgmented Inspection Team
Repor No.50-440/87024(DRS)
See Attached Distribution
RIII g RII!. RIII RIII RIlI % RIII RIII l
bWi fc S M 8 W iV J
La sbury/lc Cornaughton E ht F Chr'M ssotirno MflTer i
ns Sha t an ,, ..
PO
. _ _ _ _
..
.
.. ,
.
' e , * . rg
-
. u
, ~r .
The Cleveland Electric Illuminating 2 - 3AN 22 IS'Mr
Company ,
cc w/ enclosure:
F. R. Stead, Director, Perry
Plant Technical Department
M. D. Lyster. neral Manager,
Perry Pio. erations
Department
Ms. E. M. Buzzelli, Manager
Licensing and Compliance
Section
M. R. Edelman, Nuclear
Vice President,
Centerior Energy
DCD/DCB (RIDS)
Licensing Fee Management Branch
Resident Inspector, Rlll
Harold W. Kohn, Ohio EPA
Terry J. Lodge, Esq.
James W. Harris, State of Ohio
Robert M. Quillin, Ohio
Department of Health
State of Ohio, Public
Utilities Comission
i
l
I
l I
l
r
- - - - - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _ _ _ f]
'
8 . ,
,
..
9
e ig . .
. . tiy>A M 9,
. .
' - '
U.S. NUCLEAR REGULATORY COMMISSION
( REGION III
Report No: 50-440/87024(DRS)
Docket No: 50-440 License No: NPF-58
Licensee: Cleveland Electric Illuminating Company
Post Office Box 5000
Cleveland, Ohio 44101
Facility Name: Perry Nuclear Power Plar.t. Unit 1
Inspection At: Perry Site, Perry, Ohio
Inspection Conducted: November 4 through 9, 1987
NRC Augment Inspection Team
Inspectors: Team Leader:
hR. D. LbnkTbury
Co d
ll22l%B
Q.D
K.9.Donnaughton
(h
Team Members: llufst
(Date)
9. b. BcJ
h.D.Eick *
'/25/6B
(Date)
kn
. . Ornstein lln/W
,
(Date)
dCK. Sha h ll2.2 / tv
S
(Date)
O
.J.
dd\fn
w ro ~ - ilulr<
j (Date)
Approved By: G. rih (Chef / 12 E 4
Operations Branch (Date)
'L- i
pozd30 3 p-
,
1.
"
, .
s
.
. .; .. i 3
. .
. - e
.
Inspection Summary
Inspection on November 4 through 9,1987 (Report No. 50-440/87024(DRS))
Areas Inspected: Special Augmented Inspection Team (AIT) inspection conducted
in response to the Main Steam Isolation Valve (MSIV) closure failures of
October 29, 1987, and November 3,1987, for Perry Unit I and related
activities. The review included root cause determination, safety
significance, maintenance history, similar previous occurrences, and broader
industry implications. l
Results: No violations or deviations were identified; however, the licensee i
has committed to additional and expanded surveillances of the MSIV's and '
continued investigation efforts to attempt to pinpoint the failure mechanism
involved in the slow closure time.
l
1
11 -
-
.
..; n9
.
.
' ~ '
EXECUTIVE SUPNARY
On October 29, 1987, the Perry Nuclear Power Plant was in the process of
completing their Startup Test Program and was perfoming stroke time testing
of the Main Steam Isolation Valve (MSIV's) when the inboard valve in the "D"
main steam line failed to close within the maximum value delineated in the
facility technical specifications. Two other MSIV's also failed, including the
outboard MSIV in the "D" main steam line. In all cases, subsequent stroke
times for these three MSIV's were within acceptable values. The licensee
initially declared the MSIV's inoperable. However, based on the acceptable
stroke times achieved after the second try, later declared them operable. The
licensee believed that the failures were the result of impurities in the MSIV
actuator control unit and that the imourities had apparently been dislodged
and/or expelled during MISV operation. Plant operation and the Startup Test
Program were contirued with the stipulation that additional stroke time tests
on the MSIV's, to confim their operability just prior to the perfomance of
the full reactor isolation startup test be performed.
On November 3,1987, while performing the additional stroke time testing of
the MSIV's both the inboard and outboard MSIV's in the "D" main steam line
again exhibited unacceptable stroke times. The licensee reported the failure
of the two MSIV's to the NRC and commenced an orderly shutdown. As a result
of this event Region III dispatched an Augmented Inspection Team (AIT) to the
site the following day.
The licensee evaluated potential component failures and from this developed a
carefully planned disassembly and troubleshooting program. As a part of this
troubleshooting program the licensee disassembled the MSIV actuator control
units f rom the three MSIV's that had previously failed. The results of this
disassembly and inspection revealed that the Ethylene Propylene Diene Monomer
(EPDM) elastomers contained within the Automatic Switch Company (ASCO) dual
solenoid valves had been significantly degraded by exposure to high temperature
and possibly hydrocarbons. An annular dimple was also observed on the seat
material and resulted in part of the seat material being extruded into the
exhaust orifice. This dimple, together with the deteriorated state of the
seat material, indicated that the exhaust seat could be held in an "energized"
position even though the solenoids had been deenergized, and would prevent the
control air from being exhausted to atmosphere and therefore prevent the MSIV
from closing.
The AIT concluded that the most probable root cause of the observed MSIV's
failure to close on October 29, 1987, and again on November 3,1987, was a
malfunction of the ASCO Model NP-8323A20E three-way dual solenoid valve caused
by deterioration and degradation of the Ethylene Propylene Diene Monomer (EPDM)
discs in the ASCO dual solenoid valve due to exposure to a high temperature
environment. The high temperature environment was the result of several steam
leaks in the vicinity of the failed valves. The second most probable cause of
the deteriorated and degraded EPDM discs appears to be hydrocarbon intrusion
into the valve, or a combination of high temperature and hydrocarbon intrusion.
The licensee subsequently replaced or rebuilt all eight MSIV dual solenoid
valves. The plant was restarted on November 13, 1987, and the Startup Test
Program, including the full reactor isolation startup test, was successfully
completed.
iii ,
i
l 1 i
. - - _ _
,
-
.
*
,. ..-,,.
,
.
.
*
.. >
AUGMENTEDINSPECTIONTEAM(AIT) REPORT
50-440/87024
Page No.
I. Introduction 1
A. Synopsis of Events 1-2
B. Augmented Inspection Team (AIT) Fonnation 2
C. AIT Charter 2-3
D. Persons Contacted
II. Description - MSIV Slow Closure of October 29
and November 3, 1987
A. Narrative Description 3-5
B. Sequence of Events 5-7
III. Failure Mechanism Analysis 7-9
IV. Investigative Efforts
A. System Descriptions
1. Instrument / Service Air 9-10
(Portions Peirtaining to MSIV's Only)
2. Main Steam Isolation Valves (MSIV's) 10-12
B. Evaluation of Safety Significance
1. Immediate Safety Significance 12
2. Other Safety Significance 12-14
C. Effect of Maintenance Activities
1. MSIV Maintenance History 14
2. Service Air (SA) and Instrument Air (IA) 14-15
Maintenance History
D. Operations Activities
1. Operator Response 15-17 ,
2. Impact of Concurrent Surveillance Activities 17-18
on MSIV Performance ;
,
E. Troubleshooting Activities and Results 18-22
I
V. Recent Events Involving MSIV Slow Closure / Failure to Close
A. Perry Events 22
B. Industry Events
1. Solenoid Valve Related MSIV Failures 22-25
2. Solenoid Valves Not Related to MSIV Failures 25-26
VI. AIT Conclusions 26-27
VII AIT Recommendations 27-32
VIII Analysis Plan for EPDM Solenoid Components 32-33
IX Exit Interview 33-34
X Startup Review
A. Prior to Startup 34 c.
B. Following Startup 35
- IV
l
.-
-
,
,n-
' -
. .
.*
>,.*p-
. *
, . .
'
ATTACHMENTS
Attachment No. Description
1 Confirmatory Action Letter (CAL)
2 Augmented Inspection Team (AIT) Charter
3 Air System P&ID
4 Main Steam Line Isolation Yalve (MSIV)
Cross-Section
5 MSIV Control Unit Schematic Drawing
6 Cutaway Drawing of an ASCO 8320
Solenoid Valve
7 Sketch of Disk Holder Seated Against
the Valve Exhaust Orifice
8 Restart Authorization
,
Y l
"
! .
.
...9
.
. . '.
I. INTRODUCTION
A. Synopsis of Event
On October 29, 1987, while at approximately 76% power and in the
process of completing the Startup Test Program, one of the Main
Steam Isolation Valves (MSIV's) in the "D" main steam line (inboard)
at the Perry Nuclear Power Plant, Unit 1, was found to have a stroke
time greater than the maximum allowable value delineated in the
facility technical specifications. As a result of this failure, each
of the other seven MSIV's were tested. Two of these MSIV's, one in
the "B" main steam line (outboard) and the remaining one in the "D"
main steam line (outboard) also exhibited unacceptable stroke times.
In all cases subsequent stroking of the MSIV's resulted in stroke
times within the technical specification range of allowable values.
Initially the three MSIV's were declared inoperable, but based upon
the inability to recreate the failures and the subsequent satisfactory
MSIV performance the licensee declared the MSIV's operable. Based
upon discussions between the licensee, NRC Region III, and the Office
of Nuclear Reactor Regulations (NRR) management the licensee agreed
to perform additional individual fast closure tests on the MSIV's to
confirm their operability just prior to the performance of the full
reactor isolation startup test.
On November 3,1987, while perfoming the MSIV fast closure operability
checks, the inboard and outboard MSIV's in the "0" main steam line
again exhibited stroke times in excess of the technical specification
maximum value. These two MSIV's were among the three that had
exhibited the same problem on October 29, 1987. The MSIV in the "B"
main steam line that had failed on October 29 showed an acceptable
stroke time during this test. The licensee reported the failure of
the two MSIV's to the NRC and comenced an orderly shutdown.
B. Auamented Inspection Team (AIT) Formation
On November 3,1987, the Perry Senior Resident Inspector (SRI)
infomed Region III that while observing the licensee's perfomance
of the MSIV operability check in preparation for the full reactor
isolation startup test, that two MSIV's had again failed to close
properly. Subsequent to the report of this event, Region III eval-
uated the information and determined that the criteria for dispatching
an AIT had been met. Assistance from NRR was requested in several
specialized areas including air systems and valves. This assistance
was provided by Dr. H. l.. Ornstein, Senior Reactor Engineer (AE00),
H. K. Shaw, Senior Mechanical Engineer (NRR), and J. J. Stefano, Femi
Project Manager and fomally Perry Project Manager. In addition,
Region III provided expertise in operations and plant maintenance by
assigning K. A. Connaughton, SRI, S. D. Eick, Reactor Inspector, and
R. D. Lanksbury, Acting Chief, Test Programs Section as Team Leader.
All of these individuals arrived on site on the morning of November 4,
1987. Concurrent with the AIT activities, Region III issued a Confir-
matory Action Letter (CAL) RIII-87-019) which was received by the
licensee on November 4, 1987. The CAL confirmed certain actions to '
be taken by the licensee in support of the AIT and also confimed
.
.
*
...rg.
,
.
*
* '
that the plant would not be restarted without the concurrence of the
Regional Administrator or his designee. The CAL is Attachment 1 to
this report.
C. AIT Charter
On November 3, 1987, a draft charter for the AIT was formulated with
a list of preliminary questions to be pursued and a list of general
areas to be investigated:
*
Failure of MSIV's to close/close within Yechnical Specification
limits.
*
Safety significance, root cause(s).
*
Interaction of prior maintenance activities to the event.
Safety implications if actual Group I isolation signal had
been present.
History of any previous problems.
Broader implications.
Event reporting.
A finalized AIT Charter was issued on November 5, 1987. This
Charter is Attachment 2 to this report.
D. Persons Contacted
Cleveland Electric Illuminating Company (CEI)
*A.Kaplan, Vice President, Nuclear Group
*M. D. Lyster, General Manager, Perry Plant Operations Department (PP0D)
*F. R. Stead, Director, Perry Plant Technical Department (PPTD)
E. Riley, Director, Nuclear Quality Assurance Department (NQAD)
C. M. Shuster, Director, Nuclear Engineering Department (NED)
*R. A. Newkirk, Manger, Technical Section, PPTD
*V. K. Higaki, Manager, Outage Planning Section, PPOD
*W. E. Coleman, Manager, Operations Quality Section, NQAD
*B. D. Walrath, Manager, Engineering Projects Support Section, NED
*D. R. Green, Manager, Electrical Design Section, NED
*E. M. Buzzelli, Manager, Licensing and Compliance Section, PPTD
*S. J. Wojton, Manager, Radiation Protection Section, PPTD
*K. R. Pech, Manager, Mechanical Design Section, NED
*R. A Stratman, Manager, Operations Department, PPOD
W. R. Kanda, Jr., Manager, Instrumentation and Control Section, PPOD
*T. A. Oleksiak, Jr., Lead Supervisor, Maintenance Section, PPOD
! *V. J. Concel, Lead System Engineer, Technical Section, PPTD l *S. F. Kensicki, Technical Superintendent, PPTD l *G. A. Dunn, Supervisor, Licensing and Compliance Section, PPTD ] K. F. Russell, Shift Supervisor, Operations Section, PPOD .
M. W. Gmyrek, Senior Operations Coordinator, Operations Section, PPOD
i '
2 \
i
l l i
-
. .
,
.
-
. ., . - i ,, .
.
.
. .
.
J. P. Eppich, Senior Project Engineer, Mechanical Design Section, NED
G. W. Heffner, Supervisor of Media Relations
P. J. Arthur, Nuclear Steam Supply System Lead, Technical Section,
PPTD
General Electric
*J. J. Sheehan, Operations Manager
Automatic Switch Company (ASCO)
K. Thomas, Sales Engineer
Ralph A. Hiller Company
J. Nancy, Sales Engineer
* Denotes those attending the exit meeting on November 9, 1987.
In addition to the above, other members of the Perry staff were
contacted by the AIT.
II. DESCRIPTION - MSIV SLOW CLOSURE OF OCTOBER 29 AND NOVEMBER 3,1987
A. Narrative Description
On Thursday, October 29, 1987, at about 6:35 p.m. (EST) while Perry
Unit I was operating at approximately 76t power, the licensee
performed a fast closure test of the "D" inboard main steam line
MSIV (IB21-F0022D) as part of Startup Test Instruction (STI)
1821-025A, "MSIV Function Test". When the control switch for
1821-F0022D was placed in the "CLOSE" position, the valve failed
to start closing for approximately 18 seconds. At that point the
valve stroked closed for a total stroke time, including the 18 second
delay, of 22.8 seconds. Technical Specification 3/4.4.7 requires
that the MSIV's close within a time frame of 2.5 to 5 seconds and
Technical Specification 3/4.6.4 requires that they close within 5
seconds. The licensee wrote a Level 1 test exce
meet the Level 1 acceptance criteria of the and STI)ption (failure to
at 6:42 p.m.
reopened 1821-F00220. At 7:00 p.m. the Unit Supervisor declared
iB21-F0022D inoperable based upon its slow closure time. The appli-
cable action statements were then entered in accordance with Technical
Specifications Limiting Condition for Operation (LCO) 3.4.7.a and LC0
3.6.4.a.. LCO 3.4.7.a. requires that with one MSIV inoperable, either
restore that MSIV to operable status within 8 hours or isolate the
affected main steam line by closing and deactivating an MSIV in that
main steam line. LC0 3.6.4.a. requires that with one containment
l l isolation valve (MSIV) inoperable, either restore the valve to operable l
status within 4 hours or isolate the affected penetration by use of
at least one deactivated, closed, valve.
l I ! Subsequently, the decision was made to re-stroke the 1821-F0022D MSIV.
This was accomplished twice - once at 9:03 p.m. with a resultant stroke
time of 3.2 seconds and again at 9:06 p.m. with a resultant stroke time
i 3
f
.
..
,.,;.
.
.
- - *
of 2.9 seconds. Also, subsequent to the failure of the 1821-F00220
MSIV the licensee convened the Plant Operations Review Comittee (PORC)
to evaluate the situation.
At 9:44 p.m. the "D" outboard main steam line MSIV (1821-F00280) was
fast closure tested with a resultant closure time of 77 seconds. Again,
at 9:52 p.m., MSIV 1821-F0028D was stroked with a resultant closure
time of 3.2 seconds. As a result of this second failed MSIV the licensee
made the decision to test the remaining MSIV's. This was accomplished
between 9:53 p.m. and 10:20 p.m. and, with the exception of MSIV
1821-F00288, all showed acceptable closure times. The stroke time for
the IB21-F0028B MSIV, was found to be 11.9 seconds. A second test of
this valve resulted in a closure time of 3.9 seconds.
In accordance with Technical Specification LC0 3.4.7.a. and 3.6.4.a.
the licensee isolated the "D" main steam line at 10:40 p.m. This
was accomplished by closing / verifying closed the 1821-F0028D MSIV,
the before seat drain valves and the MSIV Leakage Control System
Isolation valve. These valves were then deenergized.
The licensee's PORC initiated a review of the situation and concluded
that the M51V's were operable based on successul stroke time tests
subsequent to the initial failures. From the observed MSIV behavior,
the licensee believed that the failures were due to the presence of
impurities in the MSIV actuator control unit and that the impurities
were apparently dislodged and/or expelled during MSIV operation. Based
on their review, at approximately 11:10 p.m. , MSIV's 1821-F0022D,
1821-F0028D, and 1821-F00288 were declared operable. At 11:40 p.m.
the "D" main steam line was restored to an operable status and at
12:10 a.m. on October 30, 1987, the licensee made a 4 hour report on
the slow closure of the MSIV's as required by 10 CFR 50.72(b)(2)(iii).
In a discussion between the licensee and NRC management on October 30,
1987, a concern was expressed to the licensee that while a plausible
explanation for the MSIV failures had been provided, additional assur-
antes of continued MSIV operability were warranted pending further
evaluation. To address this concern, the licensee agreed to perform
additional MSIV stroke time testing prior to the performance of the
full reactor isolation startup test which was then scheduled to be
performed within the following seven days.
On Tuesday, November 3,1987, at about 11:45 a.m. (EST) the licensee
decreased power to 80% in order to perform the additional MSIV stroke
time testing. At 11:57 a.m. the 1821-F0022D MSIV was tested with a
resultant stroke time of 18 seconds. Based upon this, the Unit
Supervisor, at 11:58 a.m. , declared 1821-F0022D inoperable. Using
canagement guidance previously provided this MSIV was re-stroked at
11:59 a.m. with a closure time of 3.0 seconds and declared operable
at 12:00 p.m. by the Unit Supervisor. At 12:12 p.m. the 1821-F002CD
MSIV was stroked and failed to close in the 2 minutes and 49 seconds
that the control switch was held in the "CLOSE" position. The control
switch was allowed to return to its normal position of "AUT0" and was
then taken back to the "CLOSE" position. The MSIV then closed in 3.4
seconds. Even though the MSIV had closed within acceptable limits on
l I
4
,
?
.
.
-
.
'.'..y.
.
,
.
.
. .
.
the second closure attempt the Unit Supervisor declared IB21-F0028D
inoperable. Subsequently, at 12:30 p.m. the decision was made to
declare the "D" main steam line inoperable. The other sb: MSIV's
were stroke time tested with acceptable results.
Within the hour following these MSIV failures, another discursion was
held between NRC and licensee management personnel. During this
discussion, the licensee informed NRC management of its intent to
increase power and perform the full reactor isolation startup test
thereby placing the unit in Hot Shutdown within the time limits of
Technical Specifications 3/4.4.7 and 3/4.6.4. The licensee was
informed that this course of action was considered to be both noncon-
servative and contrary to the intent of the technical specification.
Under the circumstances, technical specifications intended that an
orderly plant shutdown be conducted to minimize the potential for
challenging the inoperable MSIV's. Based upon this discussion, the
licensee agreed to perform an orderly reactor shutdown. At 1:30 p.m.
the licensee infonned the System Operation Center of the intended
plant shutdown and at 1:37 p.m. commenced a normal plant shutdown. At
1:53 p.m. and 1:54 p.m. the 1821-F0022D and 1821-F0028D MSIV's,
respectively, were fast closed. MSIV 1821-F0022D had a stroke time of
3.4 seconds and 1821-F0028D had a stroke time of 3.3. seconds. This
was done to comply with Technical Specification LC0 3.4.7.a. and
3.6.4.a. requirements to isolate the affected line. At 1:55 p.m. the
licensee made an Emergency Notification System (ENS) notification on
the slow closure times of the MSIV's and on the plant shutdown in
accordance with 10 CFR 50.72(b)(2)(iii) and 10 CFR 50.72(b)(1)(i)A.
B. Sequence of Events and Operator Actions
At the AIT's request, a chronology of events related to the MSIV
failures on October 29 and Nover'>er 3, 1987, was assembled by the
licensee. The chronology, whici included MSIV performance data and
operator actions, was verified to be accurate by AIT personnel
through review of operating logs, Technical Specification LC0
tracking system documentation, interviews with licensee operating
personnet, and inspector observation of MSIV surveillance testing
conducted an November 3, 1987. The chronology was as follows:
NOTE: All times are in Eastern Standard Time.
October 29, 1987
1835 Stroked MSIV 1B21-F0022D for Startup Test Instruction
(STI)-B21-025A Section 8.3. Valve did not begin to close
for 18 seconds. Level 1 Test Exception Report written.
1842 Re-opened 1821-F0022D.
1900 Declared 1821-F00220 inoperable based upon a total closing
time of 22.8 seconds. Entered associated LCOs.
2103 Re-stroked 1821-F0022D - time to close 3.2 seconds.
5
I
}. *
-
'c. ,
. *q'
: . -
t a . -
2106 Stroked IB21-F0022D again - time to close 2.9 seconds.
i 2144 Stroked IB21-F0028D - time to close 77 seconds.
.
2152 Re-stroked IB21-F0028D - time to close 3.2 seconds.
2153- Stroked remaining MSIV's. All satisfactory with the
2220 exception of IB21-F00288. Found 1821-F0028B had an
initial slow stroke time of 11.9 seconds, second stroke
was 3.9 seconds.
2240 Isolated "D" Main Steam Line (MSL). MSIV 1821-F028D
deenergized.
2310 All MSIV's were verified to stroke within 3-5 seconds.
Could not repeat the initial condition causing MSIV to
slow close. Based on licensee management review the
decision was made to declare all MSIV's operable.
2340 Restored "D" MSL.
October 30, 1987
0010 Made 4 hr. report on slow closing MSIV's in accordance with
10 CFR 50.72(b)(2)(iii).
November 2, 1987
1942 Comenced Surveillance Instruction (SVI)C71-T0039, "MSL
! solation Valve Closure Channel Functional" (10% stroke -
partial closure - RPS).
2142 Completed SVI C71-T0039 - Satisfactory.
November 3, 1987
1145 Decreased power to 801 to stroke MSIV's.
1154- Stroked MSIV's.
1222
1157 1821-F0022D took 18 seconds to close.
1158 Unit Supervisor declared IB21-F00220 inoperable.
1159 1821-F0022D restroked in 3.0 seconds.
1200 Unit Supervisor declared 1821-F00220 operable.
1212 1821-F0028D did not close in the 2 minute 49 seconds that
the control switch was in "close". Took switch back to
"Auto", then to "close", valve shut in 3.4 seconds.
1212 Unit Supervisor declared 1B21-F0028D inoperable. f
, '
I
l 6
b
[ ,
.
.
, . ,3
.
.
. . .
1230 Declared MSL "D" inoperable based on repeated failure of
1821-F0022D and 1821-F0028D to stroke in required time.
1330 Informed System Operation Center of intended plant
shutdown.
1337 Connenced a normal reactor shutdown.
1353 Closed 1821-F0022D 3.4 seconds.
1354 Closed IB21-F0028D 3.3 seconds.
1355 Made ENS notification on slow closing MSIV's and plant
shutdowninaccordancewith10CFR50.72(b)(2)(iii)and10
CFR 50.72(b)(1)(i)A.
III FAILURE MECHANISM ANALYSIS
After the second event on November 3,1987, the licensee convened a team
of individuals from various departments including representatives of
Gilbert Associates (the architect engineer) and General Electric (GE).
The charter of this team was to develop a list of components whose failure
would result in the observed behavior of the MSIV's. After developing this
list, the known facts were used to evaluate the probability associated with
each of the potential component failures. Their analysis yielded twent
(24) potential component failures. Of these twenty-four, nineteen (19)y-four
were
evaluated as unlikely failures, one (1) was evaluated as a moderate proba-
bility failure, and four (4) were evaluated as likely failures. The four
likely failures and the one moderate probability failure can be grouped
together into a category involving the ASCO dual solenoid valves on the MSIY
actuator air control units and the air system feeding them. The twenty-four
potential component failures and their associated probabilities of causing
the observed behavior were as follows:
* Failure of the Automatic Switch Company (ASCO) Model 8323 three-way dual
solenoid valve (fast closure)
* Instrument air system quality
* Obstructions / foreign materials in air lines / accumulators
* One or both of the solcnoid's of the dual solenoid salves for each
of the MSIV's failed to decouple (mechanically separate) upon
de-energization
+ Solenoid valve exhaust port blocked
Failure of the Norgren two-way control valve
Hydraulic speed control failure
MSIY internal binding
Swagelock fittings improper installation / assembly / leakage
Failure of the ASCO Model 8323 three-way solenoid valve (slow closure / test)
Valve packing too tight
Failure of the Norgren four-way control valve
Valve lineup of instrument air header system
Control unit wiring and termination failure resulting in a hot short
Glared contacts on control and relay components
7
.
*
..,..'.,.
.
.
. . .
Relay failure or incorrect operation resulting in misoperation of the
MSIV's
Panel control switch failure or misoperation
Limit switch settings incorrect or inoperable
Mis-wiring for indication of instrumentation or switches
Data acquisition failure
Procedural error for testing
High steam flow /high reactor power interaction
Incorrect reassembly and installation of the control unit
Actuator binding / stem binding
* Likely failure
+ Moderate probability failure
In conjunction with the above, the AIT also evaluated potential failure
modes and concluded that the most probable component failure was the ASCO
dual solenoid valve. In addition, the AIT evaluated the above analysis
performed by the licensee and agreed with the methodology and conclusions
reached. Subsequent to this analysis the licensee provided a written
proposal for troubleshooting the MSIV's to the AIT for concurrence. After
evaluation and coment by the AIT, a carefully planned disassembly and
troubleshooting program was generated.
The focus of the troubleshooting was to gather more data with regard to
the failures postulated as probable or likely. This was accomplished by
performing various tests of the air system, including particulate counts,
dew point measurement and analysis of air samples for hydrocarbons, and
by disassembly of various portions of the MSIV actuator air control units.
A discussion of the inspection process for the control units and corre-
sponding results is provided in Paragraph IV.E. of this report.
The licensee evaluated the facts gathered during the troubleshooting program
-
and concluded that they had substantiated their original evaluation that the
most probable failure mechanism was the ASCO dual solenoid valve. The
licensee reconvened the original failure analysis team and tasked them with
developing a list of potential failure modes of the ASCO dual solenoid
valve and the corresponding probability of each of these modes. Their
analysis yielded a total of nine (9) potential failure modes. Of these,
one (1) was evaluated as likely, two (2) were evaluated as possible, and
six (6) were evaluated as unlikely. The nine potential failure modes and
their associated probabilities of causing the observed behavior are as
follows:
* Local high temperature has caused deterioration of EPDM seal materials
Blockage of the dual solenoid valve exhaust port with tape
Jaming of kinematic components
+ 0xidation of EPDM compound used in the gaskets, seals, and disc seal
materials
Residual mar etism following coil de-energization
Wrong materials
Lockseal vapors
+ 0-ring / lubricant interaction
Corrosion within solenoid enclosure
8
-
.
*
.
.,.,.
.
.
. . .
* Likely failure
+ Possible failure
The AIT reviewed the conclusions of the failure analysis team and agreed
with their assessment with one exception. The AIT considered that deteriora-
tion due to hydrocarbon attack of the EPDM sealing materials within the ASCO
dual solenoid was a likely probable cause and that information available
did not invalidate this concern. The licensee had obtained an air sample
and had it analyzed for hydrocarbons with negative results. However, this
alone did not preclude a previous contamination of the air system with
hydrocarbons nor did it preclude an introduction of hydrocarbons from a
source upstream of the air supply line such as from the use of non-approved
pipe thread sealants or lubricants.
IV. INVESTIGATIVE EFFORTS
A. System Descriptions
1. Instrument / Service Air (Portions Pertaining to MSIV's Only)
A drawing of the service air and instrument air system is shown
on Attachment 3. The service air system for each unit censists
of one motor driven compressor with an integral intercooler and
af tercooler, an air intake filter silencer, lube oil subsystem,
filters, condensate traps, controls, e receiver tank and a piping
network for distribution throughout the plant. A cross tie header
between Perry Unit 1 and Perry Unit 2 is included in which distri-
bution connections to the various plant areas are provided. During
nomal operation, the service air systems for the two units are
cross connected with one compressor running and the other in the
automatic standby mode. If the service air system pressure drops
below 110 psig the standby service air compressor starts
automatically.
Separate instrument air systems are provided for each unit to
supply clean, dry, oil free air for control purposes throughout
the plant. The system is designed to meet the guidelines of ANSI
Standard MC-11-1 (ISA-57.3) with the exception that the r.aximum
allowable particle size for air to safety related equipment is
specified to be less than or equal to 40 microns.
The nomal supply of air to the instrument air system is from the
respective service air system for the unit and the instrument air
compressor for each unit is used as a backup. The service air
compressor is operated continuously to provide a constant output
pressure of 125 psig. The instrument air system for each unit
also includes an af ter cooler (integral with the compressor), a
receiver tank, a prefilter, an air dryer, an afterfilter, and a
piping network for distribution throughout the plant. All instru-
ment air leaving the receiver tank passes through the filters and
the air dryer. The Unit I and 2 instrument air distribution
systems are cross-tied.
9
_
..
,'. o., . .
.
.
. . .
If the instrument air system pressure drops below 90 psig the
instrument air compressor starts automatically and maintains
system pressure in the 90 psig to 100 psig range. A diaphragm
operated isolation valve is provided in the air supply line
from the service air system. This valve closes automatically
when the instrument air system pressure drops below 90 psig and
may be manually opened by a switch in the control room when the
system pressure rises above 90 psig.
The output of the last downstream afterfilter is directed to
numerous places throughout the plant including the accumulators
and control units for the MSIV's.
2. Main Steam Isolation Yalves (MSIV's)
TwoMainSteamIsolationValves(MSIV's)areweldedina
horizontal run of each of the four main steam line pipes; one
valve is as close as possible to the inside of the drywell and
the other is just outside the containment.
Attachment 4 shows a main steam line isolation valve. Each is
a 26 inch Y pattern, globe valve. The main disc or poppet is
attached to the lowtr end of the stem. Nomal steam flow tends
to close the valve, and higher inlet pressure tends to hold the
valve closed. The bottom end of the valve stem closes a small
pressure balancing hole in the poppet. When the hole is open,
it acts as a pilot valve to relieve differential pressure forces
on the poppet. Valve stem travel is sufficient to give flow
areas past the wide open poppet greater than the seat port area.
The poppet travels approximately 90 percent of the valve stem
travel to close the main seat port area; the last 10 percent of
valve stem travel closes the pilot valve.
A 45 degree angle pemits the inlet and outlet passages to be
streamlined. This minimizes pressure drop during normal steam
flow and helps prevent debris blockage. The valve stem penetrates
the valve bonnet. through a stuffing box that has two sets of
replaceable packing. A lantern ring and leakoff drain are located
between the two sets of packing. To help prevent leakage through
the stem packing, the poppet backseats when the valve is fully
open.
Attached to the upper end of the stem is an air cylinder that
opens and closes the valve and a hydraulic dashpot that controls
its speed. The speed is adjusted by a valve in the hydraulic
return line bypassing the dashput piston. Valve closing time
is adjustable to between 3 and 10 seconds. The air cylinder is
supported on the valve bonnet by actuator support and spring
guide shafts. Helical springs around the spring guide shafts
close the valve if air pressure is not available.
The valve is operated by pneumatic pressure and by the action
of compressed springs. The control unit is attached to the air
cylinder. This unit is shown on Attachment 5 and contains air j
10
t
y .-
.
: ..
....,c
(,, .
. . .
s
control valves and solenoid operated valves. Part 4 of Attachment
<
'
5 is the main pilot control valve (dual solenoid valve). This
valve consists of a valve body with a solenoid attached to either
end (see Attachment 6). The dual solenoid valve provides control
air to operate the four-way control valve (part 1) and the two-way
control valve (part 3) and is used for opening and for fast
closure of the MS!Y. When both of the solenoids on the dual
solenoid valve are energized the incoming solenoid air supply is
directed through the valve body to shift the four-way control
, valve and the two-way control valve to the open position. In the
open position the four-way control valve ports air through the
three-way control valve (part 2) to the underside of the MSIY
actuator piston while at the same time venting the over piston
area of the MSIV actuator to atmosphere. With the two-way control
valve in the open position the exhaust path through it to atmos-
phere is closed. For a fast closure of the MSIV both solenoids
de-energize shutting off the control air to the four-way control
valve and the two-way control valve and venting them both to
atmosphere. When this occurs both valves will shift to the closed
position. In the closed position the four-way control valve now
directs air to the over piston area of the MSIV actuator and vents
the under piston area to atmosphere. The two-way control valve
now is in the closed position and also vents the under piston
area of the MSIV actuator to atmosphere. In this condition the
MSIV is closed both by air pressure and by the helical valve
springs.
Slow closure capability (used for test purposes) of the MSIV is
accomplished through the use of the single solenoid valve (part 5).
When the MSIV is open and the solenoid for the single solenoid
valve is energized, air is directed to the three-way control valve
(part 2) causing it to shift to the closed position. In this
position the air that was directed to the under piston area of
the MSIV actuator from the four-way control valve is stopped and
a vent path for the under piston area is opened up through an air
metering valve (part 9). The over piston area is still vented to
atmosphere through the four-way control valve. In this configu-
ration the air trapped in the under piston area is slowly bled off
through the metering valve allowing the MSIV to slowly close.
Remote manual switches in the control room enable the operator
to operate the valves. Operating air is supplied to the valves
from the Instrument / Service Air System. An air tank (accumulator)
between the control valve and check valve provides backup
operating air.
The main steam line isolation valves are designed to close under
accident environmental conditions of 330*F for one hour at drywell
pressures of 30 psig maximum and -14 psig minimum. In addition,
they are designed to remain closed under the following post-
accident environmental conditions:
a. 330'F for an additional 2 hours at drywell pressure of 15
-
psig maximum.
11
'
t
'
h , . ~. .. . . .
,
.
,.
...
b. 310'F for an additional 3 hours at 15 psig maximum.
f c. 250*F for in additional 18 hours at 15 psig maximum.
d. 250'F to 100'F ramp during the next 99 days at 15 psig
maximum.
.
B. Evaluation of Safety Significance
1. Imediate Safety Significance
'
Based upon the absence of plant conditions requiring an automatic
main steamline isolation, the excessive MSIV stroke times did not
have imediate safety significance. Had a main steamline isola-
tion been required, isolation of the "D" main steamline may not
have occurred within the timeframe assumed in the accident analysis
for the Perry plant. The safety significance of such an occurrence
is further discussed in the following paragraph.
2. Other Safety Significance
In response to the question of whether or not the accident analysis
bounded the event which occurred on November 3, 1987, when both
the inboard and outboard MSIV's in one of the four main steam
lines failed to close within the 5 second time required in the
plant Technical Specifications, the licensee was tasked to perform
an analysis to evaluate the safety significance of this event.
There was no additional safety significance attributable to the
other MSIV that failed to close since the redundant MSIV in that
line closed within the prescribed technical specification values.
The two MSIV's in one main steam line (line "D") that failed to
close within the required time were identified as IB21-F0022D and
IB21-F0028D. The IB21-F00220 (inboard) MSIV took 18 seconds to
close; the 1821-F00280 (outboard) MSIV did not close until the
valve switch was recycled in the control room (approximately 2
minutes 40 seconds). Both General Electric (GE) and Gilbert
Associates (GAI), the Perry Architect Engineer, assisted the
licensee in the performance of this analysis.
First GE determined that two accident scenarios and three
transients described in the Final Safety Analysis Report (FSAR)
took credit for closure of the MSIVs. These events were the
following:
! 1) Main steam line break outside containment
2) Inside containment breaks which cause reactor water level
to reach the Level 1
3) Pressure regulator failure transient
4) Loss of condenser vacuum transient
5) Loss of AC power transient
The bounding event was determined to be the main stean line break
i
outside containment, since that event would pemit the largest
' amount of activity to reach the cite boundary. Therefore, GE
was tasked with detemining what the mass flow would be for a
i main steam line break outside containment given the at found 1
12
F .
.
. ,'.~,.
.
.. .
conditions that existed on November 3,1987 (i.e., three main
steam lines isolated within proper times, and the remaining main
steam line isolating in 18 seconds). The analysis was perfomed .
using the GE "SAFE 06" Code, an NRC approved Code which had been j
previously used by Perry in the ECCS performance analyses (FSAR i
Chapter 6). It should be noted that the mass release detemined
by this Code was muc5 less than the mass release discussed in
FSAR Section 15.6.4.4 for the main steam line break outside con-
tainment due to the conservative assumptions used in the FSAR
anclysis (assuming that level rise time is 1.0 second; that
steam-water mixture quality is a constant 7.0%, and that the ;
system ressure remains constant at 1060 psig throughout HSI'l
closure .
In addition, GAI was asked to perform two additional calculations.
The first calculation considered the mass release given in the
FSAR (FSAR page 15.6-10) for the first 5.5 seconds of the event
and then using the GE supplied flow data after 5.5 seconds with
one main steam line open. The second calculation used the GE
supplied data throughout the event. For each calculation two
results were detemined. First the postulated amount of radia-
tion which would be released in the 18 seconds it took for the
1B21-F0022D (inboard) MSIY to isolate on November 3, and second
the postulated total time it would take with one main steam line
unisolated before 10 CFR Part 100 limits (i.e., Iodine dose of
300 Rem) were exceeded. It was assumed for these calculations
that there would be no plateout or hold up time for the release
and that no fuel failure would occur.
For the calculation using the FSAR mass release the following
conclusions were drawn (EB = Exclusion Boundary):
EB lodine dose with 18 second single MSIV closure - 192 Rem
EB Iodine dose with 79 second single MSIV closure - 300 Rem
for the calculation using the GE data the following conclusions
were drawn:
EB Iodine dose with 18 second single MSIV closure - 82 Rem
EB Iodine dose with 120 second single MSIV closure - 300 Rem
As shown above for either calculation the slow closure (18 seconds)
of the 1821-F03220 MSIY on November 3 would not have resulted in
a release exceeding 10 CFR Part 100 guidelines. Also, depending
upon which calculation was used, the plant would have had between
79 and 120 seconds to isolate that line under accident conditions
prior to exceeding 10 CFR Part 100 guidelines based on the assump-
tions given previously. Therefore, the licensee concluded that
the 18 second slow closure of the IB21-F00220 (inboard) MSIV had
been shown to be within the bounds of the accident guidelines.
The NRR technical staff reviewed the calculations perfomed by
GE and GAI addressing the MSIV slow closure event which occurred
on November 3, 1987, in the "D" main steam line, and found the
13
. . .[. .3
.
-
. .
licensee's assumptions and conclusion that the accident guide-
lines of 10 CFR Part 100 would not have been exceeded had a LOCA !
occurred during the time the inboard MSIV (1821-F00220) remained l
open, to be reasonable.
C. Effect of Maintenance Activities
l
The
Service AITAirreviewed
(SA) and the licensee's
Instrument maintenance
Air (IA) history
Systems. This of the1)MSIV's,(thl
included:
a review of work orders (WO's) that had been performed on the systems !
since
of these January 1985, (2)
maintenance the testing)that
activities, was performed
(3 interviewing as the
the licensee's result
staff
with respect to the maintenance perfomed, and (4) the existing material
condition of the affected MSIV's and interconnected instrument air as
it could affect the MSIV closure functions.
1. MSIV Maintenance History
Approximately sixty (60) WO's were reviewed to determine maintenance
history on the MSIV's for the past two years. Numerous maintenance
activities had been performed on the valves in recent months such
as lapping the valve seats, machining valve poppet seats, adjusting
limit switch settings and retorquino packing glands. These W0's
were followed up with appropriate post-maintenance testing and
acceptable LLRT results. No anomalies could be seen that could
be considered as contributing to the MSIV closure function failure.
During the inspection, the remoial of the MSIV actuator control
units for the three failed MSIV's was observed and a v 0ual
inspection of the other MSIV's was 6 e to assess the material
condition and environment these valves were subjected to. Results
of this inspection are detailed in Section IV.E. of this report.
This work, perfomed per W0 87-9293, WO 87-9324 and WO 87-9285,
was done in an expeditious and efficient manner.
2. Service Air (SA) and Instrument Air (IA) Maintenance History
In reviewing WO's it became apparent that a number of air system
problems had been experienced over the past two years. Various
air system supplied valves (none related to MSIV operation) were
found to have dirt desiccant, sand and/or rust in them that pre-
vented proper valv. seating and operation. Past problems with the
quality of the IA system had been attributed to either not meeting
the system dewpoint requirement of -40*F or not meeting the system
particulate requirement of no particles greater than 40 microns.
Although the potential for detrimental effects to the MSIV's and
associated equipment existed, the licensee indicated that based
on a review of the system, that they determined that the contam-
ination was apparently insufficient to cause detrimental effects
on the MSIV's and interfacing equipment.
Not meeting the dew point requirement caused moisture to be
introduced into the air system. Particulate introduction stemmed
from the afterfilters being: (1) bypassed (due to inadequate i
I
,
14
,
I
*
.
*
.. . . . . .
,
.
.
.
.
procedures), (2) overdue for element change out, or (3) the
-
repetitive (maintenance) task for filter change out had been
missed due to leaking isolation valves. Also, per the recomen-
dation of the vendor, the desiccant had been changed from a
mixture of silica gel and activated alumina to 100% activated
alumina. The silica gel desiccant was found to break down into
silica sand and cause plugging of the filter. This was a main
contributor to the various air system supplied valves not seating
properly.
At the time of this inspection the licensee w'as not perfoming
routine inspections of the IA system prefilters. The only
requirement for prefilter change out or possible problem ident-
ifier was a high differential pressure (10 psid) alarm across the
filter. The differential
recomended by the vendor)With.
pressure wasinspections
no visual monitoredbeing
once per day (as
performed there existed a possibility that the filters could
develop a hole and that the alarm point of 10 psid would never
be realized. Because Perry's IA and SA systems were supplied by
lubricated compressors, the systems prefilters had the function
of filtering oil or oil aerosols and preventing any form of hydro-
carbons from entering the desiccant and ultimately the air syster.
Hydrocarbons have been shown to degrade certain elastomers, such
as EPDM, that are utilized in the ASCO solenoid valves on the
MSIV control units.
Preventive maintenance on the IA system afterfilters was a semi-
annual "repetitive task" that entailed doing a visual inspection
for degradation. A particulate count (40 micron limit) and a dew
point check (-40*F) were done on a yearly basis with a desiccant
visual examination done on a semi-annual basis. To improve the
quality of the IA system and therefore minimize the potential
for introducing hydrocarbons into the air system, the licensee
agreed to establish a requirenent in their preventive maintenance
program which will include periodic replacement of the IA system
i prefilters and semi-annual visual inspections. Dew point and , '
particulate sampling of the IA system will continue in accordance
with the existing plant administrative procedure with unacceptable
results being evaluated and system blowdowns being conducted
until satisfactory results are obtained. The implementation of
new maintenance practices along with the continued dew point and
particulate sample should provide the licensee with a reliable
means for determination of the air quality of the IA system.
D. Operations Activities
i 1. Operator Response
The AIT reviewed the event chronology discussed in paragraph II.B.
against the requirements of the licensee's technical specifications
j as well as applicable operating and administrative procedures and i detemined that actions saken by operating personnel met the i requirements. The AIT also reviewed licensee actions for class-
ifying and reporting the MSIV failures to the NRC pursuant to
l
15
1 i
} .
.
'
.....,.
.
.
.. .
10 CFR 50.72, and determined that the events were reported under
the appropriate reporting criteria and within the required time-
frames. These findings, however, hinge upon the assumption that
following the initial MSIV stroke time failures and subsequent
acceptable MSIV stroke time tests on October 29, 1987, that the
liceqsee correctly detemined that the affected MSIV's had been
restored to operable status. Based upon the t.dditional MSIV
stroke time failures on November 3,1987, and the root cause(s)
of the MSIV failures identified and discussed in Paragraph VI of
this report, the licensee's MSIV operability determinatior, oa
October 29, 1987, does not appear, in hindsight, to have been
well supported.
The AIT reviewed licensee nomal, offnomal, and emergency
operating procedures to determine whether or not appropriate
guidance was provided for operator response to the MSIV stroke
time test failures on November 3,1987. The following procedures
were reviewed:
*
Plant Emergency Instruction (PEI)-B13 "Reactor Pressure
Yessel Control"
*
Off-Nomal Instruction (ONI)-N11 "High Energy Pipe Break
Outside Containment"
*
Plant Emergency Instruction (PEI)-D11. "Radiation Release
Control" System Operating Instruction (501)-B21 "Nuclear
Steam Supply Shutoff, Automatic Depressurization, and
Nuclear Steam Supply Systems (Unit 1)."
Under the circumstances which existed on November 3, 1987,
following the MSIV stroke time test failures, operators were
provided adequate procedural direction via 501-B21 to manually
reattempt MSIV closure. Step-by-step direction was provided for
manipulating the MSIV controls to affect fast or slow manual
MSIV closure. 'In the event that manual fast closure attempts
had not succeeded, operating procedures could have been utilized
to manually slow-close the MSIV's in the test mode. With the
valves closed in the test mode and the MSIV test pushbutton held
in the depressed position, the valves would have remained closed
indefinitely, permitting evaluation of available options and, if
deemed necessary, the performance of additional actions to secure
the valves in the closed position (e.g. shutting down the plant
and securing the instrument air supply to the MSIV actuators.)
For circumstances requiring an automatic HSIV closure signal, or
where specified plant instruments indicated significant steam
leak (s) isolable utilizing one or more MSIV's, ti.e AIT detemined
that operators would have been directed, by procedure, to verify
and/or close the appropriate MSIV's. Additionally, instruction
was provided for responding to the much more serious types of
events in which conditions required a main steam line isolation ,
and multiple MSIV failures resulting in unisolable main steam :
16
,
~
i- .
.
..
,
*
. l ., . ;
. .
.. .
line(s). Activation of the licensee's emergency response plan
was directed for these more serious types of events.
Based upon a review of the licensee's operator training and
requalification program, the AIT determined that licensed
operators were provided classroom and simulator training in the
utilization of PEI-B13, ONI-Nil, PEI-D17 and 501-021. During
initial training, operators were provided approximately 100
hours of simulator instruction and 80 hours of classroom instrec-
tion which included plant transients and accidents requiring the
use of these instructions. Training to these and other PEls and
ONIs covered entry conditions, immediate operator actions, and
supplemental actions.
While the circumstances surrounding the November 3, 1987 MSIV
stroke time failures did not require entry into these instructions,
inspector observation of operator actions during the event
indicated that the operators had a good understandina of the
operating and surveillance test procedures in use and that
procedural requirements were being adhered to.
2. Impact of Concurrent Surveillance Activities on MSIV Perfomance
The AIT reviewed a list of surveillance tests in progress at the
tinte of the MSIV stroke time test failures. The list was compiled
by the licensee and verified accurate by review of the list against
operating log entries over the timeframes of intere3t. At the
time of the October 29, 1987, MSIV failures, the following
surveillance tests were in progress:
Surveillance
Instruction No. Title
B21-T0187-R "ECCS Reactor Water Level Channel
Functional"
E22-T0195-C "ECCS Suppression Pool Water Level i
High Channel C Functional"
E22-T1202 "HPCS s.ap Discharge Flow Low Channel
Functional"
M16-T2001 "Drywell Vaccuum Breaker Isolation
Valve Operability Test"
M17-T2002 "Containment Yaccuum Relief Valve
Operability Test"
At the time of the November 3,1987, MSIV stroke time test
failures, the following surveillance tests were in progress:
,
(
l
17
_ . _ _ _ _ _ _ _ _ _ _
,
-
1 . .
...
,
.L..., *
, .
,
. .
.
Surveillance
Instruction No. Title T
'
B21-T0369-A "Safety Relief Valve Pressurt
Actuation Channel Functional"
C51-T0026 "APRM Flow Biased Power / Flow *} ,
Verification" '
-
Based upon the root cause(s) of the MSIV failures discursed in
Paragraph VI of this report and the review of the foregoing
Surveillance Instructions, the AIT concluded that the perfonnance
of these surveillances had no bearing on the hSI'/ failures s
,
E. Troubleshooting Activities and Results N ,
Af ter the event of November 3,1987, the licenste convened a tehjf ,
specialists, including representatives of GAI and GE, to cetehr;ne 'the N ",
potential components whose failure would fit the observed facts, Their
analysis yielded twenty-four (24) potential component failurds ?Nine-
teen (19) of these were evaluated to be unlibly, one (1) was baluated
as a potential failure, and four (4) were evaluat.ed as likely failurcs.
The four likely failures and the one potential failure all fell into
a category involving the ASCO dual solenoid valves cr the air system
feeding them. This analysis was used in developing a troubleshooting
plan.
During the Entrance Meeting on November 4,1987, the rednirements of
the CAL (Attachment 1 of this report) were reinforced - specifically
that no work was to be performed on the specified components / systems
without the concurrence of the AIT team leader. Subsequent to this
the licensee provided a written proposal for troubleshooting the MSIV's
to the AIT for concurrence. After evaluatien and comment hv the All a
carefully planned disassembly and troubleshooting progra'.uw generated.
In conjunction with the above, the AIT also independentiy enluated
potential failure modes and concluded that the most probhple domponent
" i"
i failure was the ASCO dual solenoid valve.
'
.
l
To determine the cause of the mis-operatn.*n of the hW/ control syn, . ems, s
- three MSIV actuator control units were removed and dtsassemt t ad. These
l
were the units on MSIV's 1821-F0022D (inboard),1821-MG28b foutboaro), i
'
l
and IB21-F0028B (outboard). All three units were des hned anl con-
structed identically. The B21-F0028D MSIV was the vulve that f911ed
to close until cycled a second time during one of the events, while
the other two had not meet the Technical Specification reau1r:h.uits
for clostre times. Prior to any work on the MSIV's teing perfonned,
a visual examination of all eight MSIV's was y rformed t,o document the
as found conditions. The material condition fund in the centrol enit
.
connections, air control valves, and the ASCO solenoic valves in the
l control unit was as fellows: i
1. The first control unit to be removea an'd uisasserabled was
i 1821-F0022D. Prior to removal, the MSIV war opened (ASCO dual i solenoids energized) and voltage checks were made to detennine . 3
i
'
18
l
-. . - - _ _ _ _. - _ _ . -- _ ,
M- ~ . c W' . ( M; .' gy t
y .'a c i
'
I
( x e
T \
.,s p( .
j
-
,
S,
t ,' ' t
1 ,the as found conditions. No anomalies were noted. In addition,
5Wiv'. I 1
air blows were also perfonned on the MS!Y actuator air supply,
the solenoid air supply and the MSIV accumulator. These tests
) ,
included collection of any exhausted material by pillow case, a
i particle count check, and a dew point check. No negative results
'
were reported for the pillow case air blows and the dew point
, h checks indicated that they were less than -40'F. The particle
w count checks wcre performed by blowing the air through 0.45
r.icern filter pper. Several of these were sent to an indepen-
c'en laborato y for particle size measurement and characterization.
Particle sizes,were reported in excess of the 40 micron limit
for the instrwet air system committed to by the licensee. The
licensee noted tilat bitruse the sampling methodology allowed for
'
potential contaminaion of the samples from outside air and from
handling t. hat these realts were indetenninate. The particles
collected were characterized into three basic types: white
translucent, rust in color, and black metallic. The sample
sizes were'too smell tr.,' allow further analysis. After removal
of the contrc.1 unit, it was taken to a work area where it was
connectedsto e icoulated (90 psig) nitrogen supply and a test
box thi.t uTiweis the solenoids to be energized and de-energized.
The control uc,13 was then cycled several times. In each case
'
thc control un t TJnctioned per design with no anomalies being
noted.
. Metallic shavings and a dirt-like substance were discovered at
the 1-5/8" inlet port to the MSIV actuator and the swage lock
input fitting had deep grooves or etched scratches. The ASCO
!
'
\ dual solenoid valve was disassembled and no foreign materials
>
'
1 were fouid in the valve internals. All body gaskets were flat-
u ! tened, br a tle, degraded, and showed evidence of being exposed
\
to higb temperature. The body gaskets were found to be adhering
< to the brass valve body and wtien peeled from the valve body lef t
3 ,
portions of their EPDM material behind. The B solenoid coil was
'
raty, apparently due to moisture intrusion. Both the A and B
coils were meggered and checked for continuity with no anomalies
notert. The EPDM disc on the solenoid operated disc holder was I
four.d to be hardened and somewhat deformed. An annular dimple !
, was observed en the EPDM disc of the disc holder. This was
caused from :td disc holder being pushed against the raised
x (cone-like) exhaust orifice of the solanoid valve body (see
Attachment ?) causing the orif'ce to cut into the seat material.
This resulted in part of the seet material being extruded into
the exhaust orifice. 1 tis dimpie, together with the deteriorated
stm of the disc materfal, indicated that the disc holder could
be held in an "energized' rnition even though the solenoid had
beer de-energized, and would prevent the control air from being
,
. '
exhausted to atmosphere and therefore prevent the MSIV from
closing.
2. The control unit f or valve 1821-F0028B was the next to be removed
; and disassembled. When the 1-5/8" stainleu steel air supply
piping was removed from the control unit, metal filings were
'
4-
discovered on internal threads together with an unknown material.
'
19
_ ___. - _ _ _ _ - _ _ _ _ _ _ _ _ - - _ _ - - _ - _ _ _ - - _ _ _
r
- __ _ a
. .
__ _ - _ _
.
.. . . .
.
. . . . _
.
'
.
s . . . - -
,
.
.
-
, .
This material was later analyzed using Infrared Spectrophotometry
(IR), and was determined not to be "Neverseeze" lubricant (commonly
used for making up air system joints) and to possibly be "Rectorseal"
thread sealant. However, no evidence of foreign material in the
control unit internals was discovered during the dismantling
process, When the control unit was removed from the MSIV a
"puddle" of unknown fluid was found in one of the actuator air
ports. Subsequent analysis using IR identified the fluid as
! silicon lubricant.
Air blows were also perfonned on the MS!Y actuator air supply and
the solenoid air supply. These tests included collection of
exhausted material by pillow case, a particle count check, and a
dew point check. The results of this testing was similar to
that reported above for MSIV 1821-F00220.
When the ASCO dual solenoid valve was disassembled, small amounts
of dirt / grease (possibly 0-ring lubricant) mixture and some
unidentified particles (possibly metal shavings) were found in
the exhaust port and the internal thread of the exhaust and
intake ports. Material galling was discovered at the ferrule
area in the T-fitting connected to the solenoid valve inlet and
the air supply port. Both the upper and lower cylinder connection
ports were smeared with substantiai amounts of blackish grease
(possibly 0-ring lubricant), but there was no foreign material
found inside the solenoid valve assembly or in the pilot air
line. Neither solenoid A nor B sub-assemblies contained foreign
material but all body gaskets (0-rings) were brittle, degraded,
flattened, and showed evidence of being exposed to high temperature
(per the ASCO representative who inspected them). As with the
previous control unit the body gaskets were also found to be
adhering to the brass valve body and when peeled away portions
of the gasket material remained adhering to the valve. The sole-
noid coil surfaces were slightly discolored possibly because of
high temperature exposure. Both the A and B coils were meggered
and checked for continuity with no anomalies noted. Inspection
of the EPDM material on the disc holder revealed conditions,
including the annular dimple, similar to that reported in section
1 above for the 1821-F0022D control unit.
3. The third control unit removed was for MSIV IB21-F00280. While
witnessing the removal of the control unit for MSIV 1821-F0028B,
a member of the AIT noted the presence of a piece of duct tape
over the exhaust port of the 1821-F0028D ASCO dual solenoid valve.
This finding was significant in that if the exhaust port is
plugged, the MSIV would not be able to close. As a result of
this finding the remaining MSIV's were inspected but no other
similar conditions were noted. In order to allow testing to
detennine if this duct tape contributed to the problems exhibited
by the 1B21-F0028D MSIV, the AIT instructed the licensee to leave
the tape in an undisturbed state. In addition, it was requested
that the valve be tested in the disassembly work area to deter-
mine what effect, if any, the duct tape had on the solenoid
valves operation. After its removal and transport to the work
20
- - - - - - - - - - - - - - _ - - - _ - - - - - - _ _____._
4. .
.
l
,......
,
.
' .
.
-
. .
area the control unit was mounted on a test rig, connected to a
regulated (90 psig) nitrogen supply and a test box. The control
unit was energized, simulating the MSIV being in the open position,
and allowed to sit for approximately two hours and fifteen minutes.
This wait period was to allow the solenoids and valve body to heat
up to an equilibrium value. The equilibrium value was approxi-
mately 130*F in an ambient temperature of approximately 85'F. It
was hoped that by allowing the valve to sit and heat up that the
original failure could be recreated on the bench. When the control
unit was de-energized it worked per design. It was observed that
the duct tape covering the ASCO dual solenoid valves exhaust port
acted like a flap and lifted away from the port, except for one
point of attachment, and allowed the valve to exhaust to atmo-
sphere. The tepe was then removed and examined. The examination
revealed that the tape had been in place for some time. The
tape no longer had the flexibility of new tape and remained in
its installed shape even after removal. The tape also had
become so porous that when held up to a light source pinpoint
holes could be seen. In addition, the sticky side of the tape
that had not been attached to the valve body had collected dust
and dirt. The AIT concluded that based upon the test performed
and the examination of the tape that it had not been a contributor
to the observed behavior of the 1B21-F0028D MSIV. The licensee's
investigation into the origin of the duct tape revealed that it
had probably been put in place during a previous maintenance
outage as a cleanliness barrier.
The material condition of the control unit air connections and
the ASCO dual solenoid valve was similar to the condition found
in the two earlier ones but to a different degree. It appeared
that high temperature had caused a more severe degradation of
1821-F0028D.
, Other valves in the 1821-F0028D control unit were then disassembled. '
Small amounts of dirt and some metallic particles or shavings were
found inside the air control valves, but no foreign matter was
found in the dual solenoid valve. With the exception of the ASCO
dual solenoid valve, the operability of the control unit was
believed to be unimpaired by these small particles of contamination.
With respect to the ASCO dual solenoid valve, though no foreign
matter was found inside of the valve, the failure of the MSIV's
due to this could not be totally eliminated since the foreign
material could have conceivably been blown out of the exhaust
j port during subsequent operation.
The licensee evaluated the data gathered as a result of the trouble-
shooting program and concluded that the root cause of the failures
of October 29, 1987, and November 3, 1987, was a failure of the
respective MSIV's ASCO dual solenoid valve. This failure was
attributed to the hardening and dimpling of the EPDM seat material
as the result of exposure to a local high temperature environment
caused by steam leaks in the vicinity of the control units. As
part of their corrective action the licensee rebuilt some of the
ASCO dual solenoid valves. Inspection of the remaining solenoid
i i 21 __
_ _ - _ _ _ _ _ _ _
.
. . :. . ..
.
.
*
.
l
'
'
valves that were disassembled indicated that their material con-
,
dition was significantly better than the three that had been
'
installed on the MSIV's that had failed. Seat impressions were
noted on the valve seat, however, the dimpling condition evident
on the other valves was not on any of these valves.
V. RECENT 2 VENTS INVOLVING MSly SLOW CLOSURE / FAILURE TO CLOSE
A. Perry Events
The AIT reviewed MSIV fast closure stroke time tett results for MSIV
testing conducted since operating license issuance and prior to
October 29, 1987. These test results included tests conducted to
satisfy technical specification surveillance test requirements and
startup tests involving MSIV closure. Based upon this review, a total
of 78 individual MSIV fast closures were identified. Two instances
were identified in which individual MSIV's exceeded their maximum
allowable stioke time. One occurred following the loss of offsite
power startup test conducted on May 10, 1987, when MSIV 1B21-F0028B
closed in 5.1 seconds; the second occurred during surveillance testing
conducted on August 10, 1987, when MSIV 1B21-F0028C closed in 5.3
seconds. Following each of these occurrences, adjustments were made
to the MSIV fast stroke speed controllers and the valves were retested
with satisfactory closure times (4.0 - 4.6 seconds).
While MSIV IB21-F0028B was among those MSIVs which failed on October 29,
1987, the AIT could not determine whether the earlier failure on May 10,
1987, was due to the same root cause(s). The small magnitude by which
the stroke time was exceeded on May 10, 1987, did, however, suggest
that the failures were not related.
The root cause of the failure of MSIV IB21-0028C on August 10, 1987,
appeared to be more clearly unrelated to the October 29 and hovember 3,
1987, MSIV failures. Aside from the fact that this valve did not
experience subsequent failures, inspection of the ASCO dual solenoid
valve internals for MSIV 1821-F0028C showed little, if any, degradation.
The licensee had experienced additional failures of solenoid valves
similar to the ASCO dual solenoid valve used on the MSly control unit.
On July 30,1986, (LER 86-030) the licensee reported the failure of
a similar ASCO dual solenoid valve in the containment vacuum relief
system. The valve (model 8-HB-8320 A9) was found to be "leaking air
due to an accumulation of dust in the valve seating area." The LER
noted that "a similar solenoid valve connected to the same airline
in the same panel was then inspected in order to determine if it was
experiencing a similar problem. No further problems were identified."
A review of Work Order No. 860010560 by the AIT showed that the
adjacent valve had been removed, cleaned and rebuilt. The work
order indicated that the maintenance staff had found small amounts
of black dust in the body of the second valve.
B. Industry Events
1. Solenoid Valve Related MS!V Failures
22
_
- _ _ _ _ _ _
.
-
.
, .; . . . .
.
.
.
.
. .
There have been a multitude of solenoid valve failures at U.S.
nuclear power plants. With regard to solenoid valves used for
MSIY closure there have been several dozen failures. Some of
these events are reported below along with descriptions of
notifications that the NRC provided and a discussion of the
actions taken by the licensee in response to those notifications.
The following failures occurred between 1970 and 1980:
Dresden-2 9 failures
Hatch-1 5 failures
'
Hatch-2 1 failure
Haddam Neck 1 failure
Lacrosse 1 failure
Millstone-1 3 failures
Monticello 3 failures
Nine Mile Point-1 2 failures
Oyster Creek 3 failures
Peach Bottom-3 2 failures
Pilgrim-1 3 failures
Quad Cities-1 5 failurer
Quad Cities-2 2 failures
Trojan 1 failure
Vermont Yankee 4 failures
Zion-1 2 failures
Zion-2 4 failures
These failures were reported in NRC Inspection and Enforcement
(IE) Circular 81-14 * Main Steam Isolation Valve Failures to Close",
November 5, 1981. The circular recommended that holders of con-
struction permits: 1 "Evaluate MSIV control system design in
light of both successful and unsuccessful industry experience";
2 - Consider design changes where appropriate to ensure high
reliability and to minimize or eliminate the common-mode failure
potential present in current designs."
No written response to the circular was required and the AIT is
unaware of any action taken by CEI as a result of it.
IE Infonnation Notice (IN) 86-57, "Operating Prot'lems with Solenoid
Operated Valves at Nuclear Power Plants", July 11, 1986, presented
information about a September 27, 1985, event at Brunswick-2 in
which 3 out of 8 MSIV's failed to fast close. As with the event
at Perry, two of the MSIV's were in one main steam line. The
event at Brunswick was suspected to most likely have been caused
by hydrocarbon contamination of the air system and high ambient
temperature conditions, degrading the Ethylene Propylene Diene
Monomer (EPDM) valve seating and seal material. The information
notice stated that Brunswick was replacing the solenoid valves
EPDM internals with Viton since EPDM is unsuitable in air systems
that are not designed to "oil free" specifications. EPDM absorbs
hydrocarbons resulting in swelling and loss of mechanical prcperties.
The infonnation notice discussed Viton's superior high temperature ,
'
perfomance when compared to EPDM. Viton's disadvantage was also
23
a
I
_ _ _ _ _ _ _ _ _ _ . _
-
.
<
.. .... .
'
*
)
. .
'
. .
< noted, i.e., it is less resistant to radiation than EPDM (by a
:
factorof10). IE IN 86-57 noted ASCO's recomendation that
'
Viton be used for applications where radiation levels do not
exceed 20 megarads. The information notice also addressed
chloride contaminstion of other MSIV solenoid coils, and the
failure of several scram discharge valves at Brunswick which were
caused by excessive amounts of silicone lubricant. CEI reviewed
IE IN 86-57 and detemined that no action was necessary because:
(a) The use of Viton seals vs. EPDM had already been investigated
at Perry, and the use of such seals was consistent with
Perry's Equipment Qualification (EQ) Program requirerients.
The original ASCO dual solenoid valves that were used for
Perry's MSIV's had Viton seals, seats and gaskets, but were
changed to EPDM because of EQ concerns.
(b) The licensee's maintenance program and adherence to ASCO's
installation and maintenance instructions were expected to
prevent the problems noted in the infomation notice, i.e.,
"high temperature ambient conditions, inadequate maintenance
program on short-lived components and the excessive use of
lubricants during maintenance." The response also noted that
oil free air is useo at Perry and that there is no danger of
hydrocarbon buildup. Upon contacting General Electric (GE)
the licensee was informed that GE believed the prelems dis-
cussed in IE IN 86 57 were due to hydrocarbon contaminants
and not high temperatures; that the EPDM materials used in
the MSIV's passed high temperature and radiation EQ testing;
and that the unit at Grand Gulf (which had similiar ASCO
solenoid valves) had not experienced any problems due to
high temperatures. In addition, GE recomended that the
air system used by Perry be designed to oil free specifica-
tions thus eliminating the possibility of hydrocarbon
contaminants.
IE IN 85-17 and 85-17 Supplement 1. "Possible Sticking of ASCO
Solenoid Valves" March 1, 1985, and October 1, 1985, described
the failure of ASCO solenoid valves which resulted in the failure
of three (3) MSIV's at Grand Gulf Unit 1 to fast close. Those
solenoid valves had Viton seals, seats, and gaskets. ASCO
attributed the failures to high-temperature sticking which
resulted from a foreign substance which collected at the lower
core / plug nut interfaces. Definitive identification of the
foreign substances were not accomplished due to the small
i
amounts of material that was collected. GE recomended that
Grand Gulf replace the potentially contaminated MSIV solenoid
valves and periodically examine and clean them. Subsequently,
Grand Gulf replaced all 8 of the solenoid valves with similar
environmentally qualified ones having EPDM (as opposed to Viton)
seals, seat and gaskets. The EPDM valves did not stick when
subjected to the same conditions that caused the Viton valves to
stick.
'
24
b rW
.
_ _ _ . _ _ _ _ _ _ _ _ . ..
.
'
, . ,'. . . . , .
.
*
.- .
CEI's followup determined that no action was required since there '
were no solenoid valves with Viton used in safety-related
applications at Perry; and the ASCO solenoid valves used for the
MSIV's were qualified and had EPDM internals rather than Viton.
2. Solenoids Valves Not Related to MSIV Failures
IE IN 80-11 "Generic Problems with ASCO Valves in Nuclear
Applications Including Fire Protection Systems," March 14, 1980,
discussed the problems of having oil in contact with EPDM parts
which are internal to ASCO solenoid valves. The notice stated
that there is a potential for failure of solenoid valves having
EPDM internals due to traces of oil from oil based thread lubricants
and traces of oil from instrument air compressors. The informa-
tion notice cited ASCO's recomendation that EPDM elastomers
found in EQ qualified ASCO NP-1 solenoid valves be replaced with
Viton kits. Attachtd to the information notice was a letter from
EG&G Idaho, Inc. which described fifty failures of solenoid valves,
citing common mode failures due to oil or other foreign material
in the air supply system. In addition, it noted that 18 percent
of the failures found were caused by high temperatures and humidity
resulting in electrical failure. The licensee's response to IN
80-11 indicated that similar Class 1E qualified ASCO solenoid valves
(NP-1) having EPDM would be rebuilt with Viton kits. Similarly
certain ASCO 8320 solenoid valves would also have the EPDM replaced
with Viton.
The licensee's review package for IE IN 80-11 also included an
ASCO service bulletin on the subject (dated April 1,1980). That
bulletin stated that "If pipe thread sealant is properly applied,
and if ASCO NP-1 solenoid valves are properly installed in an oil
free instrument air system, there should be no need to replace
the ethylene propylene elastomeric parts with Viton kits. If there
are traces of compressor oil in the system, steps should be taken
to eliminate it, to prevent damage to other components in the
system."
IE IN 81-29 "Equipment Qualification Testing Experience,"
September 24, 1981; and IE IN 82-52 "Equipment Environmental
Qualification Testing Experience - Updating of Test Sumaries
Previously Published in IN 81-29," December 21, 1982, discussed
problems with ASCO solenoid valves in which Viton elastomer
seals deteriorated under high radiation exposure. IE IN 82-52
recomended that licensee's should review their system require-
ments for Viton compatability in view of ASCO's recomendation
that Viton seals should not be used in applications where expo-
sures are in excess of 20 megarads (EPDM being the recomended
replacement for Viton).
The licensee's review of these information notices for
applicability to the MSIV control unit's noted that Perry's ASCO
solenoid valves already had EPDM seals and, therefore, the
information notices were not applicable, j
!
I
25
I
'
,
.. . . < . ,
r ,. .
'
.
: .
,
IE IN 84-23 "Results of the NRC Sponsored Qualification Methodology
} Research Test on ASCO Solenoid Valves," April 5,1984, highlighted
J_ the fact that two ASCO solenoid valves which had undergone natural
aging had failed EQ tests. The valves were heated in an air oven
at 140*F for three years. The valves were pressurized with nitro-
gen and the coils were continuously energized. One of the failure
,
mechanisms involved was the sticking of the EPDM to the valve's
metallic parts. The failure of the other naturally aged valve was
i
attributed to the accumulative degradation of the EPDM diaphram,
The licensee's review of this information notice focused upon the
,
fact that the MSIV control unit contained different ASCO solenoid
valves (NP-3320 and NP-8323) which were fully qualified in accor-
dance with testing perfomed by GE. As a result the licensee
concluded that the information presented in IN 84-23 was not
applicable to their MSIV control unit.
Information Notice 85-08, "Industry Experience on Certain Materials
used in Safety-Related Equipment" dated January 30, 1985, addressed
the environmental qualifications of ASCO solenoid valves having
Viton and EPDM parts in addition to addressing the use of elas-
tomers and epoxy coatings in personnel air locks, hydrogen
recombiners and oil storage tanks. The information notice stated
the conditions under which the NRC considered Viton and EPDM to
be environmentally qualified.
The licensee's review of the information notice (relating to the
MSIV control unit) noted that all of the valves in the control
unit contained EPDM parts, and the valves were fully qualified
in accordance with GE equipment qualification report NEDC-30800.
As a result the licensee concluded that no action was required
in response to the information notice.
VI. AIT CONCLUSIONS
The most probable root cause of the observed MSIV's failure to close on
October 29, 1987, and again on November 3,1987, was a malfunction of the
ASCO Model NP-8323A20E three-way dual solenoid valve caused by deterioration
and degradation of,the Ethylene Propylene Diene Monomer (EPDM) discs in
the ASCO dual solenoid valve due to exposure to a high temperature environ-
ment. The high temperature environment was the result of several steam
leaks in the vicinity of the failed valves. The second most probable
cause of the deteriorated and degraded EPDM discs appears to be hydrocarbon
intrusion into the valve, or a combination of high temperature and hydro-
carbon intrusion.
All evidence collected during the investigation indicated that the event
was probably caused by the failure of the ASCO dual solenoid valve to shift
to the de-energized position. The evidence collected included the following:
a. The MSIV's in question stuck open during one command, but closed
within the Technical Specification requirement in responding to .
the next command.
i t i ,
26 r
I
- _ _ _ _ _ _ _
..'..:.*
.
The design of the control unit is such that the simultaneous failure
* *
b.
of more than one of the air control valves would be required to cause
the observed failures.
c. The EPDM disc on the solenoid operated disc holder in the MSIV's in
question was found to be hardened and somewhat deformed.
I d. An annular dimple had formed on the EPDM disc. This dimple, together
with the deteriorated state of the disc material, indicated that the
disc holder could be held in an "energized" position during the
de-energizing comand, and would prevent the control air from being
exhausted to atmosphere and, therefore, prevent the other air control
valves from shifting to the proper position to vent the underside of
the MSIV actuator piston to atmosphere and to port control air above
the MSIY actuator piston.
The EPDM disc material was qualified for service temperatures up to 140
degrees Fahrenheit using clean, dry air. From the state found on the disc,
it is suspected that the qualified service limits of the EPDM material may
have been exceeded. Plant records showed that steam, at temperaturcs of
300 degrees Fahrenheit or higher, leaked from the IB21-F00220 MSIV during
September 1987 and from leakage control system valves in the vicinty of
the IB21-F0028B and 1821-F0028D MSIV's in early 1987. However, the evidence
was not conclusive enough to determine whether this deterioration was caused
by the high steam temperature alone, by the interaction of EPDM and hydro-
carbons released from the instrumentation air system, or by the action of
both.
The AIT also concluded that while the licensee was very responsive after
the event of November 3,1987, and proceeded in a methodical, well thought
out, manner in determining the probable root cause, their lack of action
in starting to formulate a troubleshooting program prior to the second
failure, to validate their theorized failure mechanism, was not as conser-
vative as the NRC would like to see. In addition, the licensee's plan
after the November 3,1987, event to increase power and perfom the full
reactor isolation startup test, thereby place the unit in Hot Shutdown, is
considered by the AIT to be both nonconservative and contrary to the
intent of the technical specifications.
VII. AIT RECOMMENDATIONS
A. Failure Mechanism Investigation *_
The AIT recommends that in order to more definitively determine the
root cause(s) of the ASCO dual solenoid valve failures the licensee
should have sophisticated laboratory analysis perfomed on the solenoid
valves from the MSIV control units that failed, parts which were
removed from the solenoid valves which were rebuilt, and the air
samples which were taken from the inlet and outlet lines from the
control' units.
Some of the possible contributors to, or root causes of, the solenoid
valve malfunctions which could be revealed by such analysis (in
addition to high temperature, or possibly pointing towards synergism
with high temperature) are: l
27
[
u
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
~
y. ,
.
., .
' -
7. . :
. .
- .
.
1. Impurities in the instrument air, such as: -
a. Hydrocarbon from:
(1) the service air compressors,'
(2) temporary air compressors which were used in
containment (July / August 1987),
(3) pipe threading materials in the air system.
(4) improper lubricant or excessive lubricant on
the solenoid valves from manufacture,
installation or maintenance,
b. Desiccant from previous air system malfuntions
(incorrect filter installation) or mis-operation
(bypassingofthefilters).
c. Dirt, shavings / particles, pipe scalant, weld or
soldering debris from incorrect installation or
maintenance operations (e.g., pipe threading, gasket,
seats, system).
d. Dirt, scale, oxides, etc. from the manufacturing of
the air system components or from subsequent corrosion
or surface oxidation of air system components.
e. Moisture from the air system; e.g. temporary air
compressors which were used in containment (July /
August 1987),ormoistureintrusionfromthe
environment; e.g. steam leaks, etc.
2. Inadequate cycling: ASCO recommends cycling the solenoid valves
to prevent sticking. ASCO Bulletin 8003, "Installation and
Maintenance Instructions" notes the following:
"Preventive Maintenance
a. Keep the medium flowing through the solenoid operator or
valve as free from dirt and foreign material as possible,
b. While in service, the solenoid operator or valve should be
operated at least once a month to insure proper opening and
closing"
3. Aging of elastomers: possibility that the elastomers are breaking
down due to excess age. Shelf life information does not appear to
be readily available regarding ASCO solenoid valves and rebuild
kits, it is known that the elastomers used in many ASCO solenoid
valves have short shelf lives - however, it did not appear that
the ASCO solenoid valves and rebuild kits at Perry had records or
caution labels noting any limitations in this regard.
28
- '
*
. ...;-
.
.
,
* '
B. ASCO Design
. .
The AIT recomends that in view of the fact that there have been a
large number of ASCO solenoid valve failures in safety systems, and
that 1) the failure mechanisms have not been fully understood, and
2) the design tolerances, design characteristics, design calculations
and operating margins of the solenoid valves have not been made avail-
,
able to the licensees or the NRC, that NRC should take actions to
-
obtain in depth design infomation from ASCO needed to assure satis-
factory operations of such valves (e.g., ASCO has not responded to
the question of what is the maximum air stream particle size the
solenoid valves can handle).
C. Potential Generic Technical Specification Deficiencies
The AIT recommends that the issue * of rapid repair of steam leaks and
the avoidance of high localized temperatures which can lead to degrad-
ation and failure of seemingly qualified safety equipment should be
addressed by the NRC. The technical specifications and LCO's regarding
containment and steam tunnel temperatures may require modifications.
The existence of steam leaks at the Perry plant prior to October 29,
1987, is believed to have been one of the initiating events of the
failures of the MS!V's to function properly. The technical specifica-
tions at most plants are predicated upon gross averages of containment
and steam tunnel temperatures without consideration of localized high
temperatures.
D. Equipment Qualification Testing *
The AIT recommends that work be done by the licensee and NRC to assure
that EQ testing properly accounts for nomal plant operating conditions
(including normal operation of equipment), anticipated transients or
equipment malfunction, design basis accidents, and combinations thereof.
The direct applicability of the current EQ tests to the actual plant
operating and accident conditions is suspect. EQ testing of the MSIV
control unit solenoid valves entailed a 1000 hour elevated temperature
test. That test included cycling the valves once every 24 hours. Such
a test is not indicative of a long period high temperature soak as may
have occurred at the Perry plant prior to the October 29 failures.
The cycling during the EQ testing would minimize the likelihood of
valve failure due to disk to seat sticking, as evidenced by proper
valve operation after the initial failures.
E. Instrument Air System Deficiencies
The AIT recommends that plant specific instrument air system
deficiencies at Perry, as noted below, should be corrected:
1. System Design Criteria
It is recommended that the licensee review the instrument air
system, and the components which are dependent upon it and take
action to assure that there is a match between the needs of the
components and the quality of the instrument air delivered to
29
._. _ ___ ._ . _ _ _ _ _ _ _ _ _ _ _ _ _
..',...,-
,
.
*
.. .
the components. In addition, the plant specifications and the
FSAR should be modified to assure compatibility between the system
and the components / systems which use and/or depend upon instrument
air.
The instrument air system was originally supposed to meet ANSI
standard MC 1.11 - 1976 (ISA-57.3) "Quality Standard For Instrument
Air" which limits particulate size to 3 microns. However, the
NRC granted the Perry plant relaxation from the ANS!/ISA 3 micron
requirement to 40 microns, based upon General Electric document
22A2537-Revision 2 "Field Cleaning and Cleanliness of Nuclear
Power Plant Components," November 1979, which defines instrument
quality air as "compressed air dried to a dew point of -40'F at
the supply pressure and passed through an oil trap and a less than
or equal to 50 micron filter to remove oil and foreign particles."
The 40 micron requirement is not consistent with the needs of all
equipment using the instrument air (e.g., air compressor seal air
system has a maximum allowable particle size of 10 microns at
Perry - see item 3.c. below).
2. Air System Degradation by Use of Temporary Compressors
The AIT recommends that the licensee institute appropriate controls
to ensure that the quality of the air in the air system, and
therefore the air operated components it supplies, is not degraded
by the use of temporary compressors. It is possible that air
operated components may have been degraded at Perry during periods
in which the containment air system was isolated and a temporary
portable air compressor was used. The presence of particulates
or impurities introduced by such activities would probably not
show up in "air blows" months later. However, the presence of
such contaminants could have caused degradation and malfunctions
of air-operated equipment several months after their introduction
to the system.
3. Inadequate Maintenance and Surveillance Testing *
The AIT recommends that the licensee review each of the following
issues and take appropriate action:
l a. There was no prefilter inspection program (at the time of
this inspection there was no comitment to inspect the
prefilter until the pressure drop across it became excessive-10
psid). A blown or incorrectly installed prefilter would
. l
not be identified by this criteria - this could be an
i especially important deficiency in view of the fact that ! the prefilter is the primary defense against hydrocarbon ! intrusion from the compressor or intake.
b. Desiccant column inspection is inadequate - semiannual
i
surveillance involves visual inspection of desiccant in the
! column. The inspection is limited to viewing the desiccant
- on the very top of the column. However, the material which
1s observed is the most recently added material which may
l
30
' l
- _ _ _ ; .
+
, 'm'
.
,
. .
have little resemblance to the older and possibly degraded
g desiccant which is below it; in addition, there did not
- appear to be a fim comitment for providing desiccant
column change out,
c. In accordance with the manufacturer's data, the maximum
allowable particle size for the air compressors' seal air
system is 10 microns. Failure of the seal air system could
result in the intrusion of oil into the instrument air *
. system. Consequently, the presence of particulates in the
instrument air system in excess of 10 microns has the
potential for degrading the compressor's seal system
thereby leading to gross contamination of the instrument
air system. (Such a contamination coupled with a blown or
improperly installed prefilter could cause major air system
problems.)
d. Dew point is noted daily near the dryers, with an instrument
air sample being drawn annually from downstream of the
afterfilter. The sample's particle count is also taken,
however it is not checked for hydrocarbons or other contam-
inants. It appears that no testing is done to check for
hydrocarbons or specific contaminants in the air system.
e. The licensee's acceptance of instrument air having particulates
in excess of component (vendor) design requirements indi-
cated their lack of understanding of the problem; e.g., in a
November 9, 1987 letter (PY-CEI/01E-0288L, Edelman to
Davis), the licensee stated that "very small quantities of
particles greater than 40 microns were identified which
indicates acceptable air system quality. Therefore, it is
a very low probability that the particles had an adverse
affect upon the solenoid valve operation."**
4. Safety Accumulators
The AIT recomends that the licensee review MSIV surveillance
testing (and other testing involving accumulators as applicable)
for adequacy. The surveillance testing of the MSIV's do not
test the adequacy of the backup safety accumulators. The safety
related check valves are not tested frequently to assure their
operability upon a loss of instrument air. Accumulator pressures
are not monitored, therefore, a malfunctioning check valve is
not readily detectable. (Although this inspection was confined
to MSIV accumulators, it is believed that accumulators for other
safety systems at Perry may have similar deficiencies.)
** Note: As described in IN 85-17 and its supplement, other BWRs have
experienced sticking of safety related ASCO solenoid valves;
siiniliar failures at Grand Gulf were attributed to failures due
to "microscopic particles" which were found in the valves. The
licensee's acceptance of a less than desirable air system and
their lack of adequate attention to NRC generic comunications
is of concern to the NRC.
31
.
*
.
.:..,-
.
.
.
.
F. Potential Generic Issue Information Dissemination
The AIT recommends that in the short term an infomation notice be
issued to alert the industr to the more current failures and what
kinds of failure mechanism s(y) are postulated to exist, what should be
looked for if solenoids are disassembled, and a recommended testing
program that can be used to help detect failures prior to these
occurring during an actual transient. In the longer tem, a Bulletin
should be considered to require specific actions by licensee's to
mitigate future failures if further information indicates the speci-
fic actions to be taken. Issuance of this Bulletin should be held
in abeyance until further information can be gathered regarding the
failure r'echanism(s) so that adequate corrective actions can be
developed. In addition, consideration should be given to alerting
industry self improvement groups (such as INPO) that industry
initiative needs to be taken to resolve this issue.
As described in section V.B. of this report there have been numerous
failures of ASCO solenoid valves over the past 15 to 20 years. At
various times during this period the NRC has issued several foms of
comunications to alert the industry to these potentially significant
failures. However, as evidenced by the fact that these failures
continue to occur, it appears that the industry has not been aggressive
in correcting the problems.
* Items A, D and E.3., above were discussed at an NRC/CEI meeting on November 10,
ICP CEI management indicated that the faulty maintenance and surveillance
preuices would be corrected and that a test program would be implemented.
VI!! ANALYSIS PLAN FOR EPDM SOLEN 0ID COMPONENTS
After completion of the licensee's troubleshooting program and evaluation
of the data collected, the licensee proposed a number of correctivt actions
that they intended to implement (reference letter PY-CET/0!E-0266, dated
11/13/87, Edelman to Davis). Among these was an analysis plan for the EPDM
solenoid components. This plan entailed chemical analysis of the removed
elastomer materials at a molecular level to detemine if changes had
occurred from its original state. The plan also entailed a comparison
of the physical properties of the removed elastomer materials to that of
new materials to detemine the extent of degradation and reduced performance.
In developing this plan the licensee utilized current industry experience
with ASCO solenoids, including a similar event at Brunswick in 1985, to
provide guidance. The data gathered from this analysis plan combined with
other industry experiences would be utilized to detemine a final root
cause for the previous events. The following is an outline of the analysis
program:
A. Samples
1. Unused elastomer gasket material
2. Used elastomer from pilot solenoids which did not fail
3. Used, degraded elastomer material from failed ASCO dual
solenoids
4. ASCO dual solenoid valve bodies with elastomer residue
32
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ -
'
.
*
.. , ....-
,
*
.
. *
B. Physical Testing
1. Profilimetric analysis to compare indentations in EPDM discs
(sample nos. 2 and 3).
2. Optical Microscopy to detemine the presence of foreign
material, or loss of material from surfaces.
3. Hardness testing to compare with original specifications.
4. Compression set to compare with unused material and note
perfonnance degradation.
C. Chemical Testing
1. Infrared Spectrophotometry survey to detennine carbonile content.
This will provide information about the mode of attack (organic
acids from the presence of hydrocarbons) and extent of
oxidation.
2. Scanning Electron Microscopy /X-Ray Dispersion Spectrometry to
confirm or negate copper-catalyzed accelerated oxidation
(which was a postulated failure mode at Brunswick.)
D. Environmental Testing
Six new dual coil solenoids will be sent to a laboratory for
additional environmental testing. The solenoids will be placed
in three separate environmental chambers (two per chamber) at
various elevated temperatures in an energized condition. The
solenoids will remain energized for predetermined times in an
attempt to determine the temperature and continuously energized
time at which the solenoids do not perform their function. The
test duration has been set at 92 days.
The licensee's proposed schedule for the completion of the above is that
Item C.1 would be complete by the end of January 1988, with the remaining
items to be complete by the end of March 1988. The licensee also
committed that a test plan for item D would be provided by November 23,
1987, and that interim test results would be provided as they become
available.
The AIT reviewed the proposed plan and found it acceptable. As noted in
Section V of this report, the most probable failure mechanism of the ASCO
dual solenoid valve is the deterioration and degradation of its EPDM
components due to high temperatures. The second most likely failure
mechanism is the same as the first but with hydrocarbon interaction with
the EPDM components as the cause of the deterioration and degradation. A
combination of both of these failure mechanisms is also a possible cause.
The results of the above program should validate which of these failure
mechanisms was the root cause of the ASCO dual solenoid's inability to
shift to the de-energized position.
IX. EXIT INTERVIEW
The AIT met with licensee representatives (denoted in Paragraph I.D.)
infonnally throughout the inspection period and at the conclusion of the
inspection on November 9,1987, and sumarized the scope and findings of
the inspection activities.
33
.
_ _ _ - _ _ _ _ _ _ _
-
,
..,o..,
,
.
. *
v
The AIT also discussed the likely infonnational content of the inspection
report with regard to documents or processes reviewed by the inspectors
during the inspection. None of the areas expected to be contained in the
report were identified by the licensee as proprietary. The licensee
acknowledged the findings of the inspection.
X STARTUP REVIEW
On November 10, 1987, the licensee met with members of the Region III
staff, and members of Headquarters staff, in Region !!! to discuss their
plans for startup and to obtain NRC approval. As a result of this
meeting the licensee committed to perfonn a number of actions both prior
to startup and subsequent to startup. These comitments are detailed in
a letter (PY-CEI/0!E-0289 L) dated November 13, 1987, from Edelman, CEI,
to Davis, NRC. The following is a sumary of these actions:'
A. Prior to Startup
1. Replace the entire control unit for MSIV 1821-F0028D with a new
unit and the ASCO dual solenoids on MSIV's IB21-F0022D and
IB21-F0022A with new ASCO's. The remaining five (5) ASCO dual
solenoids would be rebuilt.
2. Replace the ASCO single solenoid (slow closure) on MSIV
1821-F0028B.
3. Perfom an evaluation of all other ASCO solenoid valves
classified as Class 1E used in harsh environment applications
in the plant.
4. Evaluate other equipment in the vicinity of the steam leaks
that occurred near MSIV's IB21-F0022D,1821-F00280, and
1B21-F0028B to assess any impact that these steam leaks may
have had.
5. Determine the historical readings of the permanent steam tunnel
and drp ell temperature elements in the vicinity of the MSIV's
and detennine a baseline for each element and criteria to be
used to indicate onset of a steam leak in the monitored area.
Establish a procedure with actions to be taken upon exceeding a
threshold value.
6. Install temporbry temperature elements in the vicinity of the
ASCO dual solenoids and on the solenoids and valve bodies
themselves in both the steam tunnel and the drywell. Develop
baseline data for these elements and an interim temperature
threshold.
7. Perfonn a test to verify that air does not flow between the air
compressor reduction gear vents and the air compressor intake.
34
-
.
*
..f..
.
. .
.
<
B. Following Startup
1. Perfonn a laboratory anal
of the EPDM degradationhightemperature/hydrocarbonattack).
(ysis to confinn the failure mechanism
This item is further discussed in Section VI11 of this report.
2. Establish a preventive maintenance program for periodic
replacement of the instrument air system prefilters. In
addition, add a generic precaution to air system work orders ,
'
regarding the use of thread lubricants and sealants.
3. Until the first refueling outage perfonn a monthly ASCO dual
solenoid operability test and a quarterly fast closure time
test. Prior to eaceeding a six (6) month period, an inspection
of the ASCO dual solenoid experiencing the highest temperature
profile shall be performed.
4. Complete a review of all known steam leaks in the plant which
could affect Class IE equipment and evaluate for any effect on
their long term qualified life. Also, complete a review of
potentially high temperature area environments of all Class IE
solenoids and other equipment with EPDM sub-components where
elastomer compression set or degradation r~ H result in
equipment not being able to perform its * ...ded function.
On November 13, 1987, Region !!! released the licensee from CAL RI!!-87-019
(Attachment 1) based on their corrective actions, comitments, and the
prelimnary results of the AIT inspection. Region III also concurred with
the licensee's request to allow the plant to startup and proceed with their
Startup Test Program. The above was documented in a letter (Attachment 8)
from Davis to Edelman dated November 13, 1987.
Subsequent to the restart of the Perry Plant and completion of the Startup
Test Program, another MSIV dual solenoid valve failed. On Nover.ber 3, 1987,
while performing the expanded monthly operability test for the dual solenoid
valves, the dual solenoid valve for MSIV 1821-F00228 failed to change state
when de-energized. The licensee shutdown the plant, and the NRC dispatched
an AIT to the site. The findings of that inspection will be documented in
Inspection Report No. 50-440/87027.
!
35
i
}}