ML20236Y300
| ML20236Y300 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 12/04/1987 |
| From: | Edelman M CLEVELAND ELECTRIC ILLUMINATING CO. |
| To: | Davis A NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| References | |
| PY-CEI-OIE-0296, PY-CEI-OIE-296, NUDOCS 8712110227 | |
| Download: ML20236Y300 (16) | |
Text
____ _ - _____
. t.. j / m ai i r,! a f };rd vTg Q b d ni j,,Oy t
t :
- ~ n e s=j T 1s i [
P!
"JCrTDlO
(>}
v 4. F U F ) a-1.! D 1
s < -,..-
40 v
.s_<-
1 3s u
,s,
e s
P.O.
DOX 97 m
PERRY, OHIO 44081 m
TELEPHONE (216) 259-3737 a
ADDRESS-to CENTER ROAD Serving The Best Location in the Nation Hurray R. Edelman PERRY NUCLEAR POWER PLANT sR vicF PRFMDf NT V
NUG EAR
/
December 4, 1987 PPINDy PauuiG lTif Tl PY-CEI/0IE-0296 L iM_
[
</ Thh N
ptt Mr. A. Bert Davis
~
/
Regional Administrator, Region Ill f1LE
\\
U.S. Nuclear Regulatory Commission 799 Roosevelt Road Glen Ellyn, Illinois 60137 Perry Nuclear Power Plant Docket No. 50-440 Augmented Inspection Team (AIT)
Status and Commitments
Dear Mr. Davis:
This letter provides a preliminary description on the sequence of events, troubleshooting, and conclusions surrounding the single MSIV solenoid operated valve problem identified on November 29, 1987.
It also contains corrective actions and commitments made to the AIT on December 4, 1987.
The formal report required by your Confirmatory Action letter dated November 30, 1987 will be submitted on or before December 30, 1987.
Based upon the information provided to date, we request your concurrence to restart the plant.
If you have any questions, please feel free to call.
Very truly y urs Murray R. Edelmdn Senior Vice President Nuclear Group MRE:njc Enclosure cc:
K. Connaughton T. Colburn Document Control Desk m\\
/N Y
\\\\
\\
8712110227 871204 PDR ADOCK 0500 0
j P
uc 91187
Enclosure PY-CEI/0IE-0296 L Page 1 of 15 I.
Executive Summary On November 29, 1987, the Perry Nuclear Power Plant was conducting periodic stroke tests of the Main Steam Isolation Valves (l;SIV). The tests were performed by fully closing each MSIV individually utilizing the test solenoid, followed by taking the control switch to close.
Performance of this test verifies proper operation of the MSIV dual solenoid, since the MSIV vill only remain closed if the dual solenoid de-energizes and properly depositions. This testing method was implemented following previous MSIV solenoid problems which occurred earlier in the month.
During this test, the 1B21-F022B valve failed to remain closed after two separate attempts using the sequence described above.
In accordance with commitments documented in our letter, PY-CEI/0IE-0289 L dated November 13, 1987, the plant was shutdown and notifications made to the NRC Operations Center and Senior Resident Inspector. All other seven MSIVs satisfactorily passed the test requirements.
On November 29 and 30, an NRC Augmented Inspection Team (AIT) arrived onsite. A troubleshooting plan was established and implemented. The plan included various MSIV solenoid valve disassembles, chemical analysis of valve materials, visual inspection of valve parts, Instrument Air System analysis and microscopic examinations. The root cause analysis conclusion is that a sliver of foreign material, discovered in the 1B21-F022B solenoid valve body, caused mechanical binding of the valve. This binding caused the solenoid to remain in the normally energized position which then prevented the air bleed off from the MSIV actuator and thus valve (MSIV) closure.
The foreign material was Ethylene Propylene Diene Monomer (EPDH) and was demonstrated to have come loose during the previous valve rebuild activity.
Corrective actions included complete replacement of all MSIV solenoid valves with new valves.
Enclosuro PY-CEI/0IE-0296 L Page 2 of 15 II.
Event Description A.
Chronology of Events On November 29, 1987 at 0240, modified Surveillance Instruction, SVI-C71-T0039 " Main Steam Line Isolation Valve (HSIV) Closure Channel Functional" vas being performed on 1B21-F022B, Main Steam Line (HSL)
B inboard MSIV. The test was performed by fully closing each MSIV individually, utilizing the test solenoid, followed by taking the control switch to close. Performance of the test in this manner verifies proper operation of the HSIV dual solenoid, since the HSIV vill only remain closed if the dual solenoid de-energizes and properly depositions (see Diagrams 1 and 2).
When the 1B21-F022B control switch was taken to the closed position (following slow closure), the valve stroked open, indicating a failure of the dual solenoid. Approximately three minutes later, the test was repeated with the same result. Two more attempts were made utilizing only the control switch. The valve failed to close during both attempts and was subsequently declared inoperable.
At 0410, a reactor shutdown was commenced in accordance with commitments documented in our letter PY-CEI/01E-0289 L dated November 13, 1987.
In addition, at 0509, MSL B vas isolated in accordance, with Technical Specification Action requirement 3.6.4.a.
After j
reducing reactor power, a manual scram was initiated at 1114.
Operators commenced a plant cooldown in preparation for entering l
Operational Condition 4,. Cold Shutdown.
l S
l l
l l
Enclosure PY-CEI/0IE-0296 L Page 3 of 15 During the plant cooldown, personnel entered the Dryvell to visually inspect the 1B21-F022B valve while operators again attempted to stroke the valve using the control switch. At 1303, with assistance from dryvell personnel (providing a slight tap on the solenoid body),
the valve was successfully stroked closed. The valve was then reopened and at 1306, again stroked satisfactorily from the control switch. By 1720, the plant was in cold shutdown.
On November 30, the Nuclear Regulatory Commission (NRC) issued a Confirmatory Action Letter (CAL) detailing various actions to take in preparation for an NRC Augmented Inspection Team (AIT). The team arrived on site November 29 and 30.
B.
Troubleshooting Activities On November 30, the 1B21-F022B dual solenoid valve (ASCO Hodel Number NP-8323A20E) was removed from the field, disassembled and visually inspected. While the visual inspections were underway, a troubleshooting plan was developed. The plan's purpose was to conduct a thorough, timely and systematic investigation of the circumstances surrounding the HSIV failure. The plan was comprised of three major segments; (1) collection and documentation of pertinent information (2) analysis of that data for root cause determination and (3) implementation of corrective action necessary to prevent recurrence. The troubleshooting plan and associated work orders were' presented to and approved by the NRC AIT. The details of the plan's'first segment and the associated test results are discussed in Section III below. Root cause determination and corrective actions are discussed in Sections IV and V respectively.
Enclosure PY-CEI/0IE-0296 L Page 4 of 15 III. Information Collection and Documentation of Results This segment included visual and microscopic examinations of various valve parts and foreign materials. In addition, chemical analysis was performed on these samples which included destructive and consumptive tests.
Specifically:
A.
Tests conducted on the 1B21-F022B solenoid included the following:
1.
The solenoid was disassembled and visual inspections performed.
The visual inspections were intended to identify surface defects, impurities or the presence of foreign material.
Results:
l A sliver and two other small segments of foreign material were discovered in solenoid valve body (see Diagram 1).
The seating surface was also inspected and no evidence found of hardening of the discs or 0-rings. The discs and 0-rings remained flexible and non-rigid. There was only slight dimpling of the exhaust port disc and a minor depression on the inlet port disc. There was no resemblance to the extruded hardened dimples observed on the seats identified during the November 3, 1987 event.
2.
Chemical analysis was performed of the foreign material identified above. The chemical analysis included infrared analysis to determine material composition, and microscopy, intended to enhance the ability to conduct visual inspections.
___m__
._m-- - _. - -
Enclosure-PY-CEI/0IE-0296 L-i Page 5 of 15 i
Results:
a.
The infrared analysis indicated the presence of EPDM and small amounts of silicone on'the sliver and on the only other particle found capable of analysis.
Both i
materials are part of the solenoid valve assembly. This analysis also concluded that there vas-no. evidence of oxidation. Oxidation if present, would be indicative of the presence of hydrocarbons.
b.
The microscopy indicated that the material was black in color, spongy'in construction and exhibited a compression set.
similar to EPDM material. This supports the conclusion that the material was EPDM.
c.
Microscopy Dimensional Analysis of the foreign-material'and that of the pre-rebuild 0-ring from the 1B21-F022B' solenoid,
_ valve were compared. The results indicated that the foreign material vas similar to a gouge identified in a pre-rebuild 0-ring in it's shape and overall dimension (0.8 mm X 4.2 mm X 0.3 mm).
d.
An infrared analysis was performed on the pre-rebuild 0-ring and confirmed that the foreign material organic chemical composition matched that of the 0-ring (EPDH).
4 l
l
Enclosure PY-CEI/0IE-0296 L Page 6 of 15
- e. Further chemical composition analysis for metalics was performed by Scanning Electron Micrograph (SEM) on all particles (e.g. foreign material) and the pre-rebuild 0-ring.
This analysis provided further confirmation of material source by comparing trace metal catalysts and cross linking agents present in the samples. The results of this analysis document that the microstructure of the 0-ring and sliver are the same.
- f. Hieroscopy evaluation of both the pre-rebuild and existing 0-rings was performed and verified no other potential sources of the foreign material.
B.
.The 1B21-F022C and F022D solenoid valves were disassembled and visual inspections performed.
(The F022C solenoid valve was previously rebuilt while the F022D solenoid valve was completely replaced with a new valve as a result of the November 3, 1987 event).
Results:
No foreign materials were identified.
Only minor seat impressions were visible on the disk (similar to IB21-F022B disk).
C.
Evaluations conducted on the pre-replacement 1B21-F022A solenoid valve included seat leakage tests followed by disassembly and visual inspection. Results:
No seat leakage was observed. No significant foreign materials (other than small particulate) or other discrepancies were identified.
. 7 Enclosure PY-CEI/0IE-0296 L Page 7 of 15 D.
Air blows were performed on the Instrument Air System followed by dev point and particle count analysis.
Results:
1.
The air blov was conducted using a pillovcase as the filter medium. Although slight discoloration was noted, no visible particulate was identified. Additional particulate count analysis demonstrated that Instrument Air quality was within design requirements. The largest particle measured was in the less than 20 micron range.
2.
A dev point check was performed at the air supply line to the F022B solenoid. The results (-42 degrees F) were satisfactory.
3.
Samples were collected and results concluded that no hydrocarbons were present within the Instrument Air System.
1 E.
A review was performed on solenoid valve construction and dimensional analysis for thermal affects which may have contributed to the solenoid valve failure. The solenoid valve spring constants including any resulting heat effects on these values were also evaluated to determine if they had contributed to the solenoid valve I
failure.
Results:
The vendor has provided written certification documenting that as i
measured spring characteristics and valve dimensions are within
- j design tolerances and are bounded by the results of the Environmental
)
i Qualification Tests In addition, it was determined that the particles found are of sufficient size to become lodged between sliding surfaces around the solenoid core inhibiting movement and causing the valve to malfunction.
i a
l
4 Enclosure PY-CEI/0IE-0296 L Page 8 of 15 F.
A review of the Air Quality Standard for the Instrument Air System was performed.
Included in this review was a justification of previous data collected on November 7, 1987 vhich documents greater than 40 micron size particles.
Results:
Because of the probability of some contamination occurring due to the large amount of exposed sample handling involved with the filter paper collection method, it is not easily ascertained what percentage of these results were due to contamination.
It is suspected that the majority of fibers found are dust particles from room air. As a more accurate method, the particle counter instrument was recalibrates to detect and quantify particles in the 40-50 micron and >50 micron range and was used to test the validity of the filter paper collection results. The results of this analysis concluded that Instrument Air quality was within design requirements for l
particulate.
i 6
I
-- - - - - - - - - - - - ~ ^ - - - - - - - - - - - - ----
^
Enclosure PY-CEI/0IE-0296 L Page 9 of 15 IV.
Data Analysis and Root Cause Determination A.
Conclusion t
A detailed root cause analysis was performed and a draft report was provided to the NRC AIT. It concluded that the root cause of the HSIV failure was binding of the "B" solenoid (refer to Diagram 1) due to the presence of foreign material.
During visual examination, a sliver of foreign material (EPDH) was discovered in the 1B21-F022B solenoid valve body core cavity. The material was demonstrated to have come from rebuild activities conducted during the previous MSIV outage (early November 1987).
This sliver was inadvertently dropped into the core area of the "B" solenoid and since that portion of the solenoid was not disassembled it was not detected until the current investigation.
Inspection of the "B" solenoid core area required removing the "B" coil. However, the leads for the "B" coil vould have required cutting to remove the coil. By the time this solenoid valve was being rebuilt, the root cause had already been determined to be heating of EPDH material.
Consequently it was decided that the "B" solenoid core area vould not require inspection.
The sliver of foreign material caused mechanical binding of the solenoid valve.
This binding caused the solenoid to remain in the normally energized position which prevented the air bleed off from the HSIV actuator and thus valve (HSIV) closure.
4
a Enclosuro PY-CEI/0IE-0296 L Page 10 of 15 B.
Supporting Analysis Hieroscopic and infrared analysis performed on this foreign material and on the pre-rebuild 0-ring for the 1B21-F022B valve support this i
root cause determination. A summary of other analysis demonstrates no other reasonable cause or e. contributing factor.
Specifically; 1.
Visual inspections of other solenoid valves and associated parts revealed no other discrepancies which could have contributed to the valve failure.
2.
Instrument air samples and analysis identified no problems which could have contributed to the solenoid failure.
Although earlier filter collection samples identified some amount of particles in excess of the 40 micron limit, subsequent samples using a particulate counter found the air quality within design requirements. Consequently, particulate in the Instrument Air System are not believed to be a contributing factor to the solenoid valve failure.
The system is designed to meet the guidelines of ANSI Standard MC-11-1 (ISA-S7.3) with the exception that the maximum allovable particulate size for air to safety related equipment is 40 microns. Recent Instrument Air analysis has confirmed the dev point, hydrocarbon levels and particulate to be within required limits.
3.
A variety of destructive and chemical tests were performed on many pre-rebuild and new solenoid valve parts. The results of these tests identified no discrepancies which could have contributed to the solenoid valve failure.
l
Enclosure PY-CEI/01E-0296 L Page 11 of 15 C.
Relationship of November 29, 1987 Event to Previous Events MSIV solenoid valves experienced failures to shift in the past (refer to our letter PY-CEI/0IE-0288 L d.ated November 9, 1987 and 0289 L dated November 13, 1987).
On October 29 and November 3, MSIVs failed to close during testing activities, However, in these instances the root cause was determined to be failure of the solenoid valves due to EPDM elastomer degradation of the seating surface. This degradation was believed to be caused by elevated temperatures resulting from steam leaks in the vicinity of the MSIV air packs.
The root cause of these previous events is unchanged by any of the current post event investigations completed to date. This conclusion is based on the following.
1.
Temporary instrumentation mounted on the ASCO valve and contact readings taken during the 11/29/87 Dryvell entry indicated temperatures below the maximum allovable values documented in the existing Environmental Qualification Report.
2.
Subsequent disassembly of the 1B21-F022B ASCO solenoid valve and examination of elastomers indicted no evidence of hardening of the discs or 0-ring. The discs and 0-rings remained flexible l
and non-rigid. There was only slight dimpling of the exhaust port disc and a minor depression on inlet port disc. These indications are expected due to the operational design of the solenoid valve. In addition, 0-rings installed during the rebuild were removed freely.
Enclosure PY-CEI/0IE-0296 L Page 12 of 15 V.
Conclusions and Corrective Actions This report describes the event chronology, troubleshooting efforts and evaluations performed to determine the cause of ivents on November 29, 1987 when one Perry Unit 1 Main Steam Isolation Valve (MSIV), 1B21-F022B, failed to remain closed during performance of a surveillance test to verify the HSIVs would fast close on command. The cause, based on data currently available, was failure of an Automatic Switch Company (ASCO)
Hodel NP8323A20E 3-vay dual solenoid valve. The root cause was binding of the "B" solenoid due to the presence of foreign material in the solenoid core area. The source of foreign material has been identified by visual examination and laboratory testing as a sliver of EPDM material which separated from an 0-ring. This occurred during the solenoid rebuild process conducted subsequent to the events of November 3, 1987. This sliver was inadvertently dropped into the core area of the "B" solenoid and, since that portion of the solenoid was not disassembled, it was not detected until the current investigation. This root cause appears to be different than the root cause of the previous HSIV solenoid failures (October 29 and November 3, 1987).
Throughout the November 29 event, the outboard B HSL isolation valve remained operable and capable of performing its intended safety function.
There was no loss of safety system function.
Consequently, the event was not safety significant.
To prevent recurrence, all HSIV dual solenoid valves have been replaced with new valves and will be cycled 10 times as part of the retest activity prior to reactor startup.
In addition, administrative controls vill be instituted on future class 1E ASCO solenoid valve work to require the use of new valves or complete disassembly and cleanout to ensure no particles are introduced during the rebuild process'.
Enclosure PY-CEI/0IE-0296 L Page 13 of 15 Previous corrective actions discussed in our earlier letter, PY-CEI/0IE-0289 L dated November 13, 1987, will be completed as planned with the following clarifications:
1.
The dual solenoid valve inspection discussed on page 5 of the enclosure vill be performed during an outage of opportunity prior to the end of October 1988. This is based on the complete replacement of all MSIV solenoid valves discussed above.
2.
The modified monthly slow closure surveillance test discussed on page 5 of the enclosure vill be performed on a staggered basis as follows:
Until the January 4, 1988 outage, the test vill be performed a.
veekly, staggered between the inboard and outboard MSIV's.
b.
For a one month period following the January outage this test vill be performed once every two weeks, again staggered between the inboard and outboard MSIV's.
i I
4 PY-CEI/k)$E-0296 L
' ' ~ ~
f Page 14 of 15
...~.~~: m I
.sotmoto "A
l y
b f
l p
J t
L' j{"
Nl l
8 i
g,
.t l
.l i
\\'-
}
/
y l
4 M l
,,r
- 8 4
s
'(
t (.v:y
)
.,,: s
\\
'8l r llj h lb N"*
h
~
'f.
{,y 1
/
.]
\\
7 ['
i f;
Loc.xroca ci
.I
?fl' l&'
I td l
. ll
^
FOREIGN U
M ATERIA L 1
. b,,
1 y
r I
7
> {-
k jI
.i h.
i
/
i
(
(Y4
{ 't, i ji
.foLENQ/D "8 "
}1 l j 9
,g
--7
}
fl i
L a
p t.
- Y
--f, N,
E y &
.j F
b
/t
__J GL
'I 3
g
[ 'l
[
l Diagram I r
s
=
Enclosure PY-CEI/01E-0296 L Page 15 of 15 98998 p
-.,. =
n 9
0 1
t 4
yk,k;eco<t mu 1
~%
b-
=i
- Cg- ~ 9> gwgB
-(hrr-o n
s.,t c
I kk k
gJ, e
s a
k D
s s
4 s
3 s
ts T=
k f l Y
- I i
i s
n Lt CO!L coa 3
s F
7 4,
kNE& f ??,^.
y. \\.\\.'
b
-a g
f f~
h a
/
c j
r n
..,x c
e o
b i
h*
?
5 I
g i
e l
.r..,. 7,,
nu e
l
-gy le a
l
__i
[E
. t__
o I
i g
1
(
__1 1
E i-I~
bl Diagram 2 e
1
_-