ML17304B358

LER 89-007-01:on 890503,deficiencies Found During Installation of Relays Constituted Reportable Condition. Caused by Inadequate Methodology of Applying Epoxy Matl to Relay Coils.Relays Replaced.Part 21 related.W/890725 Ltr

ML17304B358
Person / Time
Site:	Palo Verde
Issue date:	07/25/1989
From:	Haynes J, Shriver T ARIZONA PUBLIC SERVICE CO. (FORMERLY ARIZONA NUCLEAR
To:	NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
References
REF-PT21-89-122-000 192-00501JGH-TD, 192-501JGH-TD, LER-89-007, LER-89-7, PT21-89-122, PT21-89-122-000, NUDOCS 8908010124
Download: ML17304B358 (18)

Text

I ACCELERATED DlRIBUTION DEMOYSWQTIO.'i SYSTEM REGULATORY INFORMATION DISTRIBUTION SYSTEM (RIDS)

ACCESSION NBR:8908010124 DOC.DATE: 89/07/25 NOTARIZED: NO DOCKET FACIL:STN-50-530 Palo Verde Nuclear Station, Unit 3, Arizona Publi 05000530 AUTH. NAME AUTHOR AFFILIATION SHRIVER,T.D. Arizona Public Service Co. (formerly Arizona Nuclear Power HAYNES,J.G. Arizona Public Service Co. (formerly Arizona Nuclear Power RECIP.NAME RECIPIENT AFFILIATION

SUBJECT:

LER 89-007-01:on 890503,Potter & Brumfield relay malfunctions.

W/8 DISTRIBUTION CODE: IE28T COPIES RECEIVED:LTR ( E'NCL i SIZE:

TITLE: Licensee Event Report (LER) & Part 21 Rept Combination (50 D t)

NOTES:Standardized plant.. 05000530 RECIPIENT COPIES RECIPIENT COPIES ID CODE/NAME LTTR ENCL ID CODE/NAME ,LTTR ENCL PD5 LA 1 1 PD5 PD 1 1 DAVIS,M. 1 1 INTERNAL ACRS MICHELSON 1 1 ACRS MOELLER 2 2 ACRS WYLIE 1 1 AEOD/DOA 1 1 AEOD/DS P/TPAB 1 1 AEOD/ROAB/DSP 2 2 DEDRO 1 1 IRM TECH ADV 1 1 IRM/DCTS/DAB 1 1 NRR/ADSP DIR 1 1 NRR/DEST/ADE 8H 1 1 NRR/DEST/ADS 7E 1 0 NRR/DEST/CEB 1 1 NRR/DEST/ESB 8D 1 1 NRR/DEST/ICSB 7 1 1 NRR/DEST/MEB 9H 1 1 NRR/DEST/MTB 9H 1 '

NRR/DEST/PSB 8D 1 1 NRR/DEST/RSB 8E NRR/DLPQ/HFB 10 1

1 1

1 NRR/DEST/SGB NRR/DLPQ/PEB '0 8D 1 1

1 1

NRR/DOEA/EAB 11 1 1 NRR/DOEA/GCB 1 1 NRR/DREP/RAB 10 1 1 NRR/DREP/RPB 10 2 2 NRR/DRIS/SIB 9A 1 1 NRR/DRIS/VIB 1 1 R NRR/DRP/1-2 DIR 1 1 VER, E 1 1 NUDOCS-ABSTRACT 1 1 REG FILE 02 1 1 I REGION 1 1 1 0 2 1 1 REGION 3 1 1 REGION 4 1 1 REGION 5 1 1 RES MORISSEAU,D 1 1 RES/DSIR/EIB 1 1 RES/DSR/PRAB 1 1 RGN5 FILE 01 1 1 EXTERNAL: EG&G WILLIAMS,S 4 4 FORD BLDG HOY,A 1 1 INPO RECORD CTR 1 1' L ST LOBBY WARD 1 1 LPDR 1 NRC PDR 1 1 NSIC MAYS,G 1 1 NSIC MURPHYIG A 1 1 NOTES 1 1 NCtXE K) ALL "RIDS" RECIPIENIS PLEASE HELP US IO REDOCZ HASTE CDÃIACr IHE DOCUMENZ COVET DES KI RQCN Pl-37 (EÃZ. 20079) K) EZJMQXB YOUR MME PKN DZPHGBDTICN ZZSXS FOR DOCUMEMI'8 KRJ DON'T NEZDt FULL TEXT CONVERSION 59 ENCL 58 REQUIRED'OTAL NUMBER OF COPIES REQUIRED: LTTR

Arizona Public Service Company P.O, BOX 53999 ~ PHOENIX, ARIZONA 85072-3999 192-00501-JGH/TDS/DAJ July 25, 1989 U~ ST Nuclear Regulatory Commission NRC Document Control Desk Washington, D.C. 20555

Subject:

Palo Verde Nuclear Generating Station (PVNGS)

Unit 3 Docket No. STN 50-530 (License No. NPF-74)

Licensee Event Report 89-007-01 File: 89-020-404 Attached please find Supplement Number 1 to Licensee Event Report (LER) No.

89-007-00 prepared and submitted pursuant to 10CFR50.73. In accordance with 10CFR50.73(d), we are herewith forwarding a copy of the LER to the Regional Administrator of the Region V office.

This report is also being submitted pursuant to 10CFR21 and includes information requested in 10CFR21.21(b)(3). In accordance with 10CFR21.21(b)(2), three copies of this report are being provided to the Director, Office of Nuclear Reactor. Regulation.

If you have any questions, please contact T. D. Shriver, Compliance Hana'ger at (602) 393-2521.

Very tru y yours, J. G. Haynes Vice President Nuclear Production JGH/TDS/DAJ/kj Attachment cc: W. F. Conway (all w/a)

D. B. Karner E, E. Van Brunt, Jr.

T. E. Hurley (3 copies).

J. B. Hartin T. J. Polich H. J. Davis A. C. Gehr INPO Records Center Potter 8 Brumfield

~5oeoio}24 8907~.'5 -0 FDIC ADCICK 0 OOI.I I-'DC

II NRC Form 344 U.S. NUCLEAR REGULATORY COMMISSION I(t 53/

APPROVED OMB NO. 31400104 LICENSEE EVENT REPORT {LER) EXPIRES: 4/31/SS (II PA 4 3/

05000530ior07 FACILITY NAME DOCKFT NUMBER (2)

Palo Verde Unit 3 TITLE Ici Potter and Brumfield Relay Malfunctions EVENT DATE ISI LER NUMBER (6) REPORT DATE (7) OTHER FACILITIES INVOLVED (4)

MONTH OAY YEAR YEAR 44QVctvTIAI. ,: neve~ OAY YEAR FACILITYNAMES DOCKET NUMBER(S)

NVMSER NVMSErt MONT N/A 0 5 0 0 0 0 503 89 8 9 0 0 7 01 07 2 5 8 9 N/A 0 5 0 0 0 OPE RAT I NO THIS REPORT IS SUBMITTED PURSUANT T 0 THE REOUIREMENTS OF 10 cFR (lt ICnrctt onr or morr o/ tnr Ioiiowinpi Ill MODE (SI 20,402(4) 20.405(c) 50.7 3(c I (2) I w) 73.71(SI POWER 20,405 ( ~ II I I (0 50.34(c) Ill 50,73(r) (2)(r) 73.71(cl I.EYEL 0 0 0 20,405 (~ I I I I (41 50.34(c) (2) 50.73 (4)(2)(riiI OTHER ISprciry in Apttrrct priow mr/in Tort /IIIC Form 20.40SN) ill(iii) 50.73(rl(2)(i) 50,73( ~ ) (2 l(rii))I A) 366AI 20,405( ~ All(iv) 50.73( ~ )I2)IS) 50.73( ~ l(2) (r(ii)(S) 20.405(cl(l l(v) 50 7 3(c) (2) I iii) 50.73( ~ )(2)(c) 10CFR21 LICENSEE CONTACT FOA THIS LER (12)

NAME TELEPHONE NUMBER AREA CODE Timothy D. Shriver, Compliance Manager 602393 2521 COMPLETE ONE LINE FOR EACH COMPONEN'7 FAILURE DESCRIBED IN THIS REPORT (13)

CAUSE SYSTEM COMPONENT MANVFAC REPORTABLE MANUFAC.

TVAEA TO NPROS CAUSE SYSTEM COMPONENT TVRER JE RL Y P 2 9 7 N ,%@ryr.g..."ijyiFA'ot

.F<<2nrr'v cPv SUPPLEMENTAL REPORT EXPECTED Ilci MONTH OAY YEAR EXPECTED SUBMISSION DATE HS)

YES /II ycL comprrtr EIIPECFEO $ VdMISSIOiY DATE/ NO ABETRAcT ILrmrt to /400 coven r ~, rooroiimrtrry A/tron <<noir corer typrwnttrn /iiirU (14I On May 3, 1989 at approximately 0730 MST, Palo Verde Unit 3 was in a refueling outage with the core off-loaded when APS determined that deficiencies discovered during the installation of Potter and Brumfield (P&B) relays constituted a reportable condition pursuant to 10CFR21 and 10CFR50.73. The P&B relays are utilized in the PVNGS Engineered Safety Features Actuation Systems and cause safety-related components to actuate when de-energized.

On August 3, 1988, APS reported a deficiency in the P&B MDR series relays (Reference LER 528/88-018). As a result, APS and P&B re-designed the relays for installation during the PVNGS Unit 1, 2, and 3 refueling outages. During post installation testing of the relays in Unit 3 on April 24 and 25, 1989 and prior to declaring the relays operable, it was discovered that approximately twenty-five percent of the new model relays malfunctioned.

The cause of the relay malfunctions has been determined to be an inadequate methodology of applying an epoxy material to the relay coils to preclude contamination of the rotor and stator mating surfaces in the relay internals.

The epoxy causes the rotor and stator to bond which results in the relay failing to operate.

NRC r rim 344

II NRC form 3SSA U,S, NUCLEAR REOULATORY COMM/SS/ON

)883)

LICENSEE EVENT REPORT (LERI TEXT CONTINUATION APPROVED OM8 NO 3)50-0)CS EXP)RES/ 8/3l /88 OOCKET NUMEER l?)

LER NUMEER (8) ~ ACE,)3)

SEQUENTIAL idly rl S V IS IO N NUM S/I NUM S/I Palo Verde Unit TEXT /// moro looco I/ roo//mN/, vso ~ 3 P/f/C form 38)LS'p/ I l 7) o This report is also being provi'ded pursuant to the provisions of 10CFR21. The s o o o 5 3 0 8 9 0 0 701 02 oF 07 narrative below includes the information requested by 10CFR21.21(b)(3);

however, it is being formatted to report this event in accordance with the requirements of 10CFR50.73.

DESCRIPTION OF WHAT OCCURRED'.

Initial, Conditions:

The following plant conditions existed when the event described in this LER was determined to be reportable at approximately 0730 HST on Hay 3, 1989.

Palo Verde Unit 3 was in a refueling outage with the core (AC) off-loaded to the Spent Fuel Pool (ND).

Reportable Event Description (Including Dates and Approximate Times of Hajor Occurrences):

Event .Classification: Condition which could 'have prevented the fulfillment of a safety function.

I Note: This section includes information requested by '10CFR21 concerning the nature of the defect and dates on which information was obtained/developed.

On Hay 3, 1989 at approximately 0730 HST, APS determined that deficiencies discovered'uring the installation of Potter and Brumfield relays (RLY) in Unit 3 constituted a reportable condition.

pursuant to 10CFR21 and consequently 10CFR50.73.

Prior to the event described in .this LER, on August 3, 1988 APS reported a defect in Potter and Brumfield HDR series relays being utilized at PVNGS (Reference LER 528/88-018). As corrective action to prevent recurrence, APS and Potter and Brumfield designed replacement HDR series relays to be installed during each Unit's refueling outage. The re-designed relays were being installed during the Unit 3 first refueling outage. During the post installation testing of the replacement relays on April 24 and 25, 1989, several of the relays would not rotate to their de-energized position. Of forty-two (42) relays tested in Unit 3, ten (10) relays did not operate properly. Five (5) of the malfunctioning relays seized and the other five (5) operated slowly. The malfunctioning relays were installed in the "B" Train Nuclear Steam Supply System Engineered Safety Features Actuation System (NSSS ESFAS)(JE).

NAC ~ /IAM SSSA I9 83 ~

0 NRC Ferrrr 3SSA U.S. NUCLEAR REGULATORY COMMISSION r943l LICENSEE EVENT REPORT ILER) TEXT CONTINUATION APPROVED OM8 NO 3150 010l EXPIRES: 8I31/88 FACILITY NAME III OOCXET NVMSER l?l LER NUMSER ISI PAGE ISI SEQUENTIAL RAI REVISION NVMSEII NVM Palo Verde Unit TEXT IIF INFIF NMce II NEvrrrNE vw C.

~ 3 HRC Forrrr 38SA 3I I IT) o s o o o 5 3 0 8 9 Status of structures, systems, or components that were inoperable. at 0 0 701 0 3 OF07 the start of the event that contributed to the event:

Prior to the installation of the replacement relays in Unit 3, the "B" Train NSSS ESFAS system was inoperable for the scheduled performance of a "B" Train electrical (EB) outage.

D. Cause of each component or system failure, if known:

Note: This section includes information requested by 10CFR21 concerning the nature of the defect and dates on which information was developed.

An extensive investigation. of the Potter and Brumfield (P&B) relay failures was conducted. Personnel from P&B and an independent testing laboratory (HI-REL Labs) assisted APS engineering personnel with the investigation.

The relay failures do not appear to be isolated to a particular model number, which would suggest a common mode failure. P&B Engineering and guality Control management personnel inspected the failed relays at PVNGS while they were installed in the NSSS ESFAS cabinet (CAS). Following the in situ inspection, the failed relays were removed. Five (5) relays were provided to P&B for their failure analysis. HI-REL Labs management inspected several relays at PVNGS. HI-REL was provided two (2) relays for an independent verification of the failure mechanism.

During the investigation of the cause of the relay malfunctions, APS and HI-REL Labs personnel discovered the presence of an, epoxy material on some of the coil rotor and stator metallic surfaces.

The epoxy material, which is utilized for coil insulation, was determined by APS and HI-REL Labs personnel to have caused the rotor and stator surfaces to bond together preventing the free rotation of the rotor by spring pressure when the coil is de-energized. (See Section I.E and I.K for further information concerning the operation of the relays.) The epoxy material was confirmed to be present on the samples inspected by P&B on April 27, 1989. The material was confirmed to be epoxy by HI-REL Labs and P&B on April 28, 1989.

E. Failure known:

mode, mechanism, and effect of each failed component, if The HDR relay malfunctions occur when the relays do not change position after they are de-energized. Normally, when the coils are de-energized, the rotor rotates approximately 30 degrees due to spring force. However, during the identified failures, the spring

~ AC ~ ONM ESSA I9 83 ~

0~

NRC Form 348A U.S. NUCLEAR REGULATORY COMMISSION 18 83 I LICENSEE EVENT REPORT (LER) TEXT CONTINUATION APPROVED OM8 NO 3150 0104 EXPIRES: 8/31/88 FACILITY NAME 111 POCKET NUMEER l1)

LER NUMEER l8> ~ AGE 13I yEAR,@ SEQUENT/AL NUM EA

<~+:. REVAK)N L< NUM44/4 Palo Verde Uni t 3 o s o o o 5 3 0 89 00 7 0 1 04oF 0 7 TEXT /// moro Fooco o /oooood. VFP dddoooo/H/IC Form 3//8A8/113) force was not able to return the rotor to its de-energized position. The relays were "sticking" in their energized position.

This condition resulted in the relay contacts not properly changing state. The consequence of the relay failures is that the related safety equipment would not be actuated as required.

F. For failures of components with multiple functions, li'st of systems or secondary functions that were also affected:

The information concerning the function of the relays is discussed in Section I.K.

G. For 'failures that rendered a train of a safety system inoperable, estimated time elapsed from the discovery of the failure until the train was returned to service:

The information required above is not considered appropriate for the particular event being reported in this LER. However, in general, it takes approximately 8-12 hours to replace a failed relay and conduct appropriate retests to return safety systems to full operability.

H. Method of discovery of each component or system failure or procedural error:

The relay failures were discovered during post installation testing of the relays as discussed in Section I.B.

Cause of Event:

Based on Potter and Brumfield's evaluation of the relays from Palo Verde, the manufacturing process led to the relay failures. The manufacturing process required epoxy to be used in touch up applications without the epoxy being cured. The uncured epoxy flowed onto the rotor and stator mating surfaces. The heat from the normally energized relays cured the epoxy, binding the relays in the energized position.

Safety System'esponse:

Not applicable - there were no safety system responses and none were necessary.

K. Failed Component Information:

Note: This section includes information requested by IOCFR21 concerning the identification of the firm supplying the basic component and the number and location of the relays at Palo Verde.

'yoC I DAM 3444

<9 83<

0 NRC Pol~ 355A U.S. NUCLEAR AEOULATOAY COMMISSION 19.831 LICENSEE EVENT REPORT (LER) TEXT CONTINUATION APPROVED OM8 ND 3)SO OII)e EXPIRES', 8)31)88 FACILITY NAME III DOCKET NUMEEA 13)

LEA NUMEER Ie) PACE LTI

.P'1'- 55OUENTlAL ll5 V l5 lO N ssUMPTR NUMPSR Palo Verde Unit 3 o s o o o 5 3 0 8 9 0 7 0 1 0 5oF 0 7 TExT iis sls<<p Fosse is sp<<ssesE ssse sspss<<M) HRc F<<sss 385A'si I IT)

The malfunctioning relays are manufactured by Potter & Brumfield and are used in equipment supplied to Palo Verde by Combustion Engineering (CE) and General Atomics (GA). The relays consist of a rotary actuator mechanism with the contact sections mounted in insulating rings on top. The actuator mechanism embodies a stator assembly on which two relay coils are mounted. The two coils are connected in series inside the relay. When the coils are energized, a rotor turns through an arc of approximately 30 degrees. This operates the contact section on the extension of the rotor shaft. The travel of the rotor is confined to a 30-degree arc between the stator faces and the stop ring. Two springs return the rotor to the stop ring when the coils are de-energized. This also returns the contacts to their normal position. Thus, the relays provide an energized and a de-energized position. When the relay repositions to the de-energized position, various valves (V),

pumps (P), motors (MTR), etc. would be actuated.

The relays are supplied in a variety of sizes, coil voltage ratings, and contact numbers. At Palo Verde, nine (9) different re-designed relays are being utilized. The relays that failed in Unit 3 were Models MDR-7061, 7062, and 7063 in the NSSS ESFAS cabinet. However, due to the similarities in construction and materials, all Potter and Brumfield models could be subject to the same failure mechanism. No new model relays have been installed in Palo Verde Units 1 and 2.

The MDR relays are used in three systems at PVNGS. These systems are:

i) The Nuclear Steam Supply System Engineered Safety Features Actuation System (NSSS ESFAS)(JE).

ii) The Balance of Plant Engineered Safety Features Actuation System (BOP ESFAS)(JE).

iii) The Reactor Trip Switchgear (RTSG)(AA)(JD).

The NSSS ESFAS uses the MDR relays as actuation relays. They are used to control valves and motors and to provide indication. There is a total of 62 relays used in each NSSS ESFAS train. At two trains per unit, this adds up to a total of 372 relays used in the NSSS ESFAS systems for the three Palo Verde units.

The BOP ESFAS uses the MDR relays as .actuation relays to provide control of motors, valves, dampers (DMP), and emergency diesel generators (EK) (DG) fo1lowing an. actuation signal. Each BOP ESFAS train has 30 MDR relays. At two trains per unit, this adds up to a total of 180 relays in the BOP ESFAS systems for the three 'PVNGS units.

lssC ~ Oiv )eee s9 $ )i

41 NRC Form 344A U,S, NUCLEAR REGULATORY COMM/$$ ION I9 83I LICENSEE EVENT REPORT (LER) TEXT CONTINUATION APPROVED OMS NO 3150&104 EXPIRES/ 8/31/88 FACILITY NAME III OOCKET NUMSER l21 LER NUMSER I4)

SEQUENT/AL QP> rlEVrSION NVM44A rer NVM44A Palo Verde Unit,3 o s o o o 53 08 9 0 0 7 0 1 06 oF 0 7 TEXT lllmore oooce ro no//rror/. //oo /oreooo//YRC Form 3%AS/ II2 I The reactor trip switchgear uses one HDR relay for each reactor trip breaker. The relay is used to provide an indication signal to the Plant Protection System (PPS)(JC) after a reactor trip breaker has opened. Failure of an HDR relay in this application would not prevent the reactor trip breaker from performing its safety function of opening. There are 4 reactor trip breakers in each unit. This leads to,a total of 12 HDR relays used in the reactor trip switchgear (RTSG) systems (AA) at PVNGS.

II. ASSESSMENT OF THE SAFETY.CONSEQUENCES AND IHPLICATIONS OF THIS EVENT:

Note: This section contains the information requested by IOCFR21 concerning the nature of the safety hazard which is created or could be created.

It should be noted that the malfunctioning relays were discovered during post installation testing in Unit 3 prior to their being returned to service. There are no new model relays installed in Palo Verde Units 1 and 2. Therefore, the relays were never relied upon to perform a safety-related function., However, the failure of a relay in the ESF to properly rotate by spring tension upon being de-energized by a valid safety system actuation signal would have prevented the associated valves, pump motors, etc. from operating as required for a safe plant shutdown. The failure of the relays in the RTSG to properly rotate results in erroneous, indication of reactor trip breaker (BKR) position to the PPS and in the Control Room. There are no other components which perform the same function as the relays that would be available during an event.

III. CORRECTIVE ACTIONS:

This section contains the information requested by 10CFR21 concerning the corrective action which has been, is being, and will be taken; the organizations responsible for the corrective action; and the length of time for accomplishing the corrective action.

A. Immediate:

As immediate corrective action, replacement of the Potter and Brumfield relays in Unit 3 was stopped in order to investigate the problem.

B. Action to Prevent Recurrence:

APS is returning all potentially defective replacement relays to PEB for dissassembly, inspection, and testing. Potter and Brumfield has corrected their manufacturing process to control the use of epoxy and ensure epoxy is applied and cured prior to assembly of the coil assembly and stator assembly. All assemblies NAC ~ Ilorrr EOOA 19 43 ~

0 NRC Form 844A V.S. NUCLEAR REOVLATORY COMMISSION I9 8S I LICENSEE EVENT REPORT (LER) TEXT CONTINUATION APPROVED OMB NO S)50 0)04 EXPIRES: 8/8)/88 FACILITY NAME )1) DOCKET NVMSER IS)

LER NVMSER )4) ~ AOE ISI YEAR .'XII SSOVSNTIAL:..o> oooco II /4)Iooed, 4>> d///m/mo/I H/Ic FomI sRs48/ I)1) will be inspected by Potter and Brumfield to ensure there is no evidence of epoxy on mating. surfaces prior to final assembly.

IV. PREVIOUS SIMILAR EVENTS:

A previous similar event was reported in LER 528/88-018. Since the failure mechanism previously reported was different than the failure mechanism reported in this LER, the, previous corrective action would not have prevented this event.

'vRC FORM S4OJ

<9 4) ~