IR 05000206/1989031

Insp Rept 50-206/89-31 on 891007-1122.Violations Noted.Major Areas Inspected:Automatic Switch Corp Solenoid Failures in Unit 1

ML20005F271
Person / Time
Site:	San Onofre
Issue date:	12/22/1989
From:	Johnson P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V)
To:
Shared Package
ML20005F265	List: IR 05000206/1989031 ML20005F264
References
50-206-89-31, IEIN-88-043, IEIN-88-43, NUDOCS 9001160110
Download: ML20005F271 (10)

Text

p1

'

!

-

1.

,

O g

h --

.,,

U.S. NUCLEAR REGULATORY COMMISSION REGION V-

_ -.

.

Report No.

50-206/89-31-Docket No.

50-206 License'No.

DPR-13 Licensee:-

Southern California Edison Company Irvine Operations Center-23 Parker Street Irvine, California 92718

'

Facility Name:

San Onofre Unit-1 Inspection at:

San Onofre, San Clemente, California Inspection conducted: October 7 through November 22, 1989 Inspector: -

C. W.'Caldwell, Senior Resident Inspector

• N'

bA /n A/.b/E J Approved By:

P. H. Johnson, Chief

.

U Date Signed Reactor Projects Section 3 Inspection Summary.

' Inspection on October 7 through November 22, 1989 (Report No. 50-206/89-31)

Areas Inspected:

Special resident inspection of ASCO solenoid failures in Unit 1.

Inspection procedures 30703, 92701, 92702, 93702 were utilized.

.--

Safety Issues Management System (SIMS) Items:

None Results:

Significant Safety Matters:

i

!

As.a result of this inspection, the inspector concluded that there were significant deficiencies in the licensee's program for handling corrective actions.

In particular, problems were identified involving i

inadequate dissemination of information about component problems to the

-

organizations responsible for evaluations, resolution of root cause y

determinations and submission of supplemental licensee event reports

!

(LERs), review of component failures for potential 10 CFR Part 21 l

applicability, follow-through on commitments made in LERs, oversight of outstanding items, and review of component failures for generic

,.

I implications.

9001160110 891226 l'

{DR ADOCK 05000206

'

t PDC u-

.

-

..

.-

-.

.

_ -

-

',

.v-

,

,

.

.

.

,

,

.g.

Summary of Violations:

- 7 --

--

Two violations were identified.

The first involved inadequate corrective-actions related to failures of Automatic Switch Company (ASCO)ilure to solenoid valves.

The second violation was related to the licensee's_fa comply with Technical Specification 3.0.3 by initiating a plant shutdown-within one hour when the normal HLR flow path was determined to be inoperable.

Open Items Summary:

During this report period, one unresolved item was closed; two enforce-ment items were opened.

,

.

I

%

a

-

.

.

.

,

..-.

.

.

.

-.

_

.

.-

.

,

.

.

.

,

.

DETAILS 1.

Persons Contacted 7"

-

Southern California Edison Company

-

e

• H. Ray, Vice President, Nuclear Engineering, Safety, and L4 censing

• R. Bridenbecker, Vice President and Site Manager

• H. Morgan, Station Manager D. Shull Jr., Nuclear Oversight Manager, NES&L

!

• R. Krieger, Operations Manager

.

L L. Cash, Maintenance Manager

• J. Reilly, Technical Manager M. Merlo, Nuclear Engineering Design Manager, NES&L P. Knapp, Health Physics Manager

-

D. Peacor, Emerge cy Preparedness Manager P. Eller, Securit Manager D. Herbst Qualit Assurance Manager, NES&L

'

D.Stoneclpher,QalityControlManager,NES&L

'

C. Chiu, Assistant Technical Manager J. Schramm, Operations Superintendent, Unit 1 J. Patterson, Assistant Maintenance Manager, Unit 1

,

'

R. Plappert, Compliance Manager

'

R. : Baker, Compliance Engineer, NES&L San Diego Gas and Electric Company J. Winter, Site Representative

• Denotes those attending the exit meeting on November 6, 1989.

The inspector also contacted other licensee employees during the course of the inspection, including operations shift superintendents, control room supervisors, control room operators, QA and QC engineers, compliance engineers, maintenance craftsmen, and health physics engineers and technicians.

2.

Background In a letter dated March 18, 1975, the NRC indicated that a review of system capabilities to assure that boron precipitation would not compromise long-term core cooling should be performed at San Onofre.

After considerable correspondences between NRC and the licensee, normal and alternate hot let recirculation (HLR) flow paths were developed by the licensee.

NRC approval of these flow paths was provided in a Safety Evaluation Report dated July 16, 198..

.

-

__.

._

.

.

-

.

.

.

_

o

,

,.,

,;

'.

.

.

'

.

.

!

. Section 6.3.2 of the Updated Unit 1 San Onofre FSAR indicates that the

,

safetyinjectionsystem(SIS).iscom)risedoftwoessentiallyseparate subsystems.

Those are the SIS for tie initial injection of borated water to the reactor coolant system (RCS), and the recirculation system for subsequent cooling and recirculation of spilled water back-e UniRCS for long-term removal of residual core heat.

Subsequent to the %ddition of a HLR flow path, Section 6.3.2.1.2 of the FSAR was amended to describe the recirculation flow paths, including the means for preventing, boron precipitation.

It indicated that the recirculation system _ consisted of a normal and alternate cold leg recirculation (CLR) flow path, and a normal and alternate HLR flow path.

In 1987, a number of environmental qualification (EQ) deficiencies were.

,

identified in Unit I by the NRC.

As a result, the licensee undertook a major effort to identify all EQ related problems in the Unit.

Corrective actions for most of the deficiencies identified were implemented by SCE.

However some system upgrades were identified for the HLR system pending

.

completIonofdesignchangesandprocurementofqualifiedcomponents.

These upgrades were planned for the Cycle XI refueling outage, as detailed in several letters, including an October 2,'1989 letter to the NRC concerning full term operating license open items.

Regarding these upgrade plans for the HLR system, a memorandum dated August 16, 1989 providedajustificationforcontinuedoperation(JCO)forHLRcomponents requiring EQ upgrades.

In that JCO, interim qualification of affected

,

L

. components in the normal HLR flow path was-established until the Cycle XI-outage.

However, in that JC0, the licensee indicated that the alternate HLR path was examined and was determined not capable of being qualified on an interim basis.

The licensee also indicated in that JC0 that the probabilistic impact of having available only one EQ HLR path

,

!:

was considered acceptable for the duration of the current fuel cycle L

until EQ and single failure upgrade modifications are implemented.

l

.0n August 24, 1989, the resident inspector was informed by the licensee j-that on the previous day, SCE was performing an in-service test (IST) of

_

l the charging flow control valve (FCV)-1112.

The test required that

'

l'

charging and letdown be isolated.

As a result, the licensee attempted to shut the downstream charging isolation valve CV-304, but was unable to do so.

CV-304isthenormalchargingisolatlonvalveandisasafety related component required to shut when normal HLR is required approximately eight hours after a loss of coolant accident.

Normal HLR is through CV-305, which opens to direct flow to the pressurizer auxiliary spray line.

Valves CV-304 and CV-305 are parallel valves supplied from a common header.

Therefore, if CV-304 fails to close, as much as 400 gpm of charging water intended for HLR will be bypassed to the loop A cold leg.

The licensee indicated that no Technical Specification requirements were applicable to this valve.

It was further stated, however, that since it was considered an important component, a 72-hour administrative action was imposed at 3:22 p.m. on August 23, 1989, to either fix the valve or initiate a plant shutdown.

l

,

.

. ~

.

-

.

-,

s

,

..

'

.

,_.,

>

..

,

Subsequent investigation by the licensee determined that CV-304 did not i

close because its solenoid actuator valve, 51-VCC-HY-1304, had failed to close when deenergized.

This solenoid valve was an Automatic Switch Company (ASCO) solenoid valve, model number 206-380.

The valve was subsequently fixed on August 24, 1989,- and the administrat17aWion' ~

statement was exited.

Since there had been e history of ASCO solenoid valve failures at San Onofre and at other licensed facilities, the inspector reviewed the details surrounding this event further.

Significant historical events related to this issue are provided below:

Between March and September 1987, Unit I had four ASCO solenoid

-

valves fail a total of five times.

December 12, 1987 - Licensee event report (LER)87-016 was submitted

-

to the NRC, explaining that valve failures on independent trains of

-

multiple systems had occurred as a result of ASCO solenoid valve problems.

It was concluded in the LER and in an interim root cause evaluation (dated November 10,1987) for associated nonconformance l

reports-(NCRs) that the presence of a thin hard film formed between the top.of the slug and the slug housing was the cause for these i

L solenoid valve failures.

The root cause evaluation also identified i

the following:

"As soon as the film forms a seal that has a large enough surface area, the valve will fail to actuate and cannot vent the air from the actuator.

Only after the seal is broken (by mechanical agitation, removal of the air supply, or cooling of the film) will the valve operate." The root cause evaluation also indicated that the licensee had contacted ASCO concerning the valve

'

I failures.

In response, ASCO claimed that no lubricants or

-

preservatives had been applied to the valves.

In this LER, the

'

licensee committed to submit a suaplemental LER when'the root cause l

of these ASCO valve failures was known and to test the ASCO solenoid i

valves weekly or secure them in their safety related positions.

l February 17, 1988 - The Supplier Quality Assessment Section wrote a

-

letter to ASCO identifying the thin hard film as the root cause of the valve failures. The licensee asked ASCO to review their processes to determine the cause of the film.

,

March 11, 1988 - ASCO responded to the above letter by indicating

-

that they apply a thin coat of Dow Corning 550 lubricant on some 0-rings, to the core face, and on the stem area of the subject valves to ensure proper break-in.

At this time, licensee personnel assigned to this project realized that the Dow Corning 550 lubricant was the root cause of the valve failures.

A memorandum for file was L

issued on April 22, 1988 explaining ASCO's assessment and also stressing that the Dow Corning 550 lubricant should be tested for L

this failure mechanism.

April 1988 - The Nuclear Engineering & Construction (Projects) group

-

l installed new solenoid valves for valves CV-304 and CV-305.

l l

l 1

..

- -

.

.

.

.

.

...

.-

. --

.--

. - -

.

.-.

.

.

.

i

. -.

,,

.,

.-

'.

'

-

-

,

,

.

Projects personnel involved were unaware that the ASCO valves.were considered to be problem components by some individuals within SCE.

June 21, 1988 - The Nuclear Safety Group (NSG) issued.a _"V.iAlation

-

and Operating Event Review Checklist" to request that tFe Indepen-dent Safety hn ineering Group (ISEG) check for industry feedback reports on ASC equipment.

In this request, it was identified that an NSG open item was established until a revision to.thts LER was submitted with a program / procedure / practice to keep ttr ASCO solenoids clean,

. June 23, 1988 - NRC Information Notice 88-43, " Solenoid Valve

-

o

"

Problems," was issued by the NRC.

This Notice identified ASCO solenoid valve failures at the Perry Plant which appeared to be caused by the Dow Corning 550. TheDowCorningproductlitergture quoted in this Notice indicated that this product gels at 200 C in 14 months and that the time decreases exponentially with increasing

'

temperature.

'

L August 5, 1988 - Nuclear Engineering recommended to Station

-

Maintenance that a note be added to procedure 501-I-8.171, TCN 0-1,

"ASCO Solenoid Valve Overhaul Procedure," to emphasize that Dow Corning 550 Lubricant should be applied cautiously when used as a

-

coating for gaskets because heavy a)plications could cause a

!

normally energized valve to stick wien de-energized. This message was transmitted as a " Safety Improvement" with a disposition.

requested within 30 days.

October 1988 to May.1989 - A number of communications took place

-

within SCE.

However, resolutions or corrective actions were not identified by these items of correspondence.

May 19,.1989 - Station Technical personnel identified the Dow

-

-Corning 550 as the cause of the ASCO solenoid valve failures and requested that this be put on the Control of Problem Equipment (COPE) list,' and that corrective action be identified so that the supplemental LER 1-87-016 could be completed.

They specifically requested that actions be taken to ensure that no ASCO valves using

-

Dow Corning 550 are installed and that no Dow Corning 550 is applied to ASCO valves during maintenance.

August 23, 1989 - Charging isolation valve S1-VCC-CV-304 failed to

-

close when required during a quarterly inservice test, because ASCO solenoid valve S1-VCC-HY-1304 failed to realign when de-energized.

3.

Inspection Findings The inspector held numerous discussions with licensee personnel in order to determine why the solenoid valve for CV-304 failed despite all of the past problems with ASCO solenoid valves as described above.

The inspec-tot discussed this problem with station compliance personnel, who believed that a supplemental LER was delinquent.

However, they also believed that a positive root cause determination for the valve failures

_.

.

-

.

._ _ _ _ _ _ _ _

_

.. _

_

_

_

,

.-

,.

'

<

.

,

.

'

had not been identified.

The inspector also noted that the ASCO solenoids were not on the Control of Problem Equipment (COPE) list and that they had been unconditionally qualified for use at San Onofre by the-COPE program.

SincethesevalveswerenotontheCOPElist,ghje L

qualifications were not questioned by the EQ pro; ram.

In7as; the

.

___

ins >ectornotedthattheEQprogramconsideredtieASCOsolenoldvalves

,

l to se qualified (including HY-1304) and that the vafve housings had been

-

tested and were found to operate (energized) at 200 C.

-

As a result of this review, the inspector concluded that thunderlying

,

cause for the failure of CV-304 was that ASCO solenoid valve models of concern (206-380 and 206-832) were not added to the licensee's COPE program upon failure of valves from April 1987 through November 1987.

In fact, these valves were not added to the COPE list until September 1,-

1989.

The COPE program appeared to be the principal mechanism to-get information concerning the problem ASCO solenoid valves to the Projects.

.

In addition to the root cause previously discussed, the inspector found a number of contributing factorc leading to the failure of CV-304.

Inadeounte Dissemination of information about valve problems l'

The COPE procedure had a requirement for personnel to provide

-

l pertinent information concerning equipment failures to the COPE list.

However, there was no s>ecific requirement for interfacing programs such as the NCR or LEt programs to make input into the COPE program, and it was not apparent that all personnel knew to provide information to the COPE program.

'

The experience review program did not get the word disseminated to

-

appropriate personnel regarding Information Notice 88-43. This was apparently due to the fact that ASCO solenoids ~ were of a different model than those discussed in the Notice, even though both models involved the use of lubricant on normally energized solenoids.

The EQ program did not have anything in the files on ASCO solenoids.

-

l This was apparently due to the fact that NCRs for valve failures were not supplied to the EQ program in the 1987-88 time frame.

(The inspector considered that it might be worth reviewing old NCRs to see if other potential input to the EQ program might exist.

This was discussed with the licensee.)

Untimely resolution of the problem An NSG review (on August 5, 1988) of procedure 501-I-8.171 identi-

-

fled the need to revise the procedure, and requested a response within 30 days.

However, the procedure was not revised to reflect the NSG's comments until September 7, 1989.

This was more than a year later.

The NCR procedure did not have any specific provisions to ensure

-

timely resolution of root cause determinations.

,

, - -

-__

_

_

. _ _

.

. _ _

_

._

_ _ _ _ _

.

y

.

..

3,

... ~.

Inadequate, untimely reportability of the issue A su)plemental LER to LER 87-016 had not been submitted identifying

-

the )ow Corning 550 lubricant as the root cause of the failures.

'This was apparently due to the fact that the root caUte N d~ nit' BHT fully documented and approved by Station Technical.

Thlswas L

despite the fact that the cognizant engineer and other station technical personnel close to the issue had determined tAat Dow Corning 550 was the problem.

This was also despite t k fact that NRC Notice 88-43 (issued in June 1988) identified similar problems and highlighted the use of Dow Corning 550 lubricant.

,.

The LER procedure did not have any specific requirements to ensure

-

timely submission of followup LERs.

(In fact a further review

.

indicatedthattherewereatotalo?15LERsInwhichtheneedfora su)plemental LER had been identified, but for which a sup)1emental-LEt had not been submitted.

Eight of these supplemental LERs were -

-

expected to be submitted in 1988.)

'

These ASCO valve failures had not been reviewed for potential 10 CFR

-

Part 21 applicability.

The inspector found that component failures were not normally reviewed with regard to Part 21, but were made by other means such as 10 CFR 50.73, if applicable.

(The licensee indicated that this was done to avoid duplicate reporting.) The requirements to provide information for items that are of potential Part 21 applicability were not clearly defined in other procedures such as the root cause or NCR procedures.

Ineffective oversight of the issue Edison's internal tracking system followed this issue, but it

-

appeared that-the backlog of higher priorities prevented this from being resolved sooner.

-NSG had been tracking this issue but did not appear to be effective

-

in getting timely resolution of the issue.

hadequategenericevaluations Other ASCO solenoid models (e.g., model numbers NP-8316 and NP-8320)

-

did not appear to have been evaluated from a generic standpoint when this issue first occurred.

The root cause procedure, 50123-XV-31, did not provide guidance for

-

the generic aspects of equipment failures.

4.

Safety Significance As indicated by these deficiencies, adequate steps were not taken with respect to these solenoids to assure that the cause of the condition was determined; corrective action also was not taken to preclude repetition of the failures.

The consequences of this were that when the solenoid for CV-304 failed, a diversion of flow from the normal HLR path would

-

..

.

.,

- -.

-.

.

.

.. -..

--

.

.

.

,

...

.

~

have occurred should it have been necessary to initiate HLR in a post-LOCA environment.

In addition, the alternate HLR path was not fully available as evident by the August 16,-1989 JC0 which indicated that the alternate HLR path had been examined and could not be qua]JflgCan.an -_

'

interim basis.- The licensee's failure to take proper correclive actions to preclude additional failures of the ASCO solenoid valves is an-

apparentviolation(50-206/89-27-01).

_

,

When CV-304 failed, the licensee initiated a 72-hour admintstrative action statement to repair solenoid valve HY-1304 or commence a reactor shutdown.

However, the intent was that the Technical Specification (TS)

would be applicable to the HLR flow path when it was required by the NRC.

TS 3.3.1, for the safety injection and containment spray systems, states that the reactor shall not be made or maintained critical unless a number of conditions are met.

These Limiting Conditions-for Operation (LCOs)

include the operability of two recirculation pumps, the recirculation heat exchanger, two charging pumps, ion statement regarding the and valves and interlocks associated

-

with these systems.

A specific Act inoperability of system valves is not provided.

In the absence of a specific action statement concerning valves in the HLR flow path, TS 3.0.3 was applicable.

TS 3.0.3 specifies that within one hour action shall be initiated to place the Unit in a Mode in which the specification does not apply, by placing it in at least hot standoy within the next six hours.

The licensee did not initiate a plant shutdown after valve CV-304 was found to be inoperable, but maintained Unit 1 in operation until the

valve was repaired on August 24, 1989.

The failure to comply with TS 3.0.3 is considered a violation (50-206/89-31-02).

-

5.

Licensee Followup

,

After the failure of CV-304, the inspector discussed the concerns related to this event with the licensee.

As a result, on September 3, 1989, the Corporate QA Organization initiated an investigation of. the issues.

The inspector discussed the findings (provided in a report) after completion of the review and learned that QA found many of the same underlying problems as the inspector.

In addition, QA found a number of other problems associated with the failure of the ASCO solenoids.

For example, the commiwent in LER 87-16 to perform weekly testing of the valves (subject of the LER) had been neglected for a period of time.

The inspector considered that QA had made a comprehensive effort to determine the causal factors for the failures of the solenoid valves.

In addition, corrective action requests (CARS) were issued when required.

These will be reviewed by QA when the responses from the responsible organizations are obtained.

During this investigation it was discovered that the COPE 3rogram, EQ, ISEG, and procurement all considered these ASCO valves to )e fully qualified for installation in Unit 1.

As a followup to the inspector's original concerns, the licensee performed an in-depth review of all ASCO solenoid valves used in Units 1, 2, and 3 and determined that there were no other solenoids subject to the failure mode found for CV-30 _

8;

,_

,5

,

.

.

' 6.

Mitigating Circumstances Upon reviewing the details surrounding the ASCO solenoid failures and compliance with the TS requirements, the inspector considered that there-wereanumberofmitigatingfactorsleadingtotheprobleel{TsWssed-~~

above.

Those were the following:

It was not explicit in any correspondence between the NRC and SCE

-

that a TS change was required when the HLR feature warr-added.

SCE did not consider the TS to cover the HLR system when CV-304

-

failed; however, conservatism was exhibited by the fact that the l

plant was placed in a 72-hour administrative action statement to fix the solenoid for CV-304 or shut the Unit down.

After CV-304 failed, SCE realized that the HLR system should be

-

included in the TS, and was planning to request a TS change.

!

-

The quality oversight organization undertook a major effort to

^

-

determine the root cause(s) for the solenoid valve failure (though j

not before the issue was identified by the inspector)

Several problems (including the EQ deficiencies previously

-

discussed) were identified during the design bases and single failure reviews of the HLR system.

The licensee exhibited l

conservatism in taking action to implement repairs to these l

components. That effort was encouraged by the NRC.

'

7.

Followup Of Previously Identified Items (92701)

l

!

(Closed) Unresolved Item (206/89-24-01), "ASCO Solenoid Valve Failure" This item is closed based upon the discussion above.

B.

Exit Meetina (30703)

,

On November 6,1989, an exit meeting was conducted with the licensee i

representatives identified in Paragraph 1.

The inspectors summarized the inspection scope and findings as described in the Results section of this report.

The licensee acknowledged the inspection findings and noted that appro-

,

priate entrective actions would be implemented where warranted. The licensee did not identify as proprietary any of the information provided I

to or reviewed by the inspectors during this inspection.

.

I