ML20059D070

Intervenor Exhibit I-MFP-138,consisting of Nonconformance Rept, & Rev 00,NCR DC1-92-EM-N010,dtd 920729

ML20059D070
Person / Time
Site:	Diablo Canyon
Issue date:	08/21/1993
From:	AFFILIATION NOT ASSIGNED
To:
References
OLA-2-I-MFP-138, NUDOCS 9401060396
Download: ML20059D070 (27)

Text

- -.- _-. - .- .- - - -- -- _ - -- ._. -..--. - _ - _ _ - _

4

,e -

i rs-tas (3/st) @_y 7g gy 3_ggg - L f 93 'N$' 22'

q -Qf j'93 fff- /3 E smuommusunct

'l aspour 3-Er estive Dates 12/31/90

1. aca gt , ,g,g 2. quality Problem asport W- '.!f applicable)

3. Item / Activity Unit 1 RT Due to MFW PP 1-1 T'tp hfarease .gq pm 7q pc to

{ s. anacription of sceneentermones:

} j Unit 1 Trio from loss of Main Feed Water Pumo 1-1. This is considered a i 1 nonconformance per 0AP 158. paragraph 2.1.2. Uwu a r

! T 1 .

l ,

J A

Y

I

n .

l .. m u.

y Dr..nis.u .. c. e. m i d % . (,/s) . . .r ni m m u. te l H Phi Edk!AN 03/06/92 iklush[#kddwsAg! 0 er. Serv. 03/06/92 f A 12. NGt Evalust[en Attached en the NCR Test continuation ammets j m 1. Plant Conditions III. Cause of Probles T. Corrective Actions 1

A II. hustription of Problem II. Analysis of Problem T1. Additional Raformation L

T Y 16. Estiasted 15. Responsible E S 13. Trend Other Completten Ora tien ,

c  : cose. NjQ.lQ11 l1.lll ll-lll Date Nuclerr.a$*n'gineering j j s N/A 09/30/92 LConstruction Servi <

! I R 16. 30CFR50.73 17. Potentially 30CFR21 18. 10CFR$0.9 19. Reference other '

e E neport.nie seportante aspersable soportamie, if app.

l A P res IXl me i1 Ya l I ne. IX ) fu I1 as [Xl INP0 Network L o

R to. assis see attached i T I

A Initial 21. fine Limit 22. Mathed 13. Notified By 16. Time 25. Date i R a Report 4 hrs. telephone E 1 Shift Supervisor 12:13 PST 03/06/92  ;

y L retie g 26. no , aired 27. Time L4mit ,s.. In sie. 29. Date 1 1 E T 1 a pert Yes E l nei1 30 days 1-92-002-00 04/03/92 l w v so. other Asencies Notified None '

31. Seearts l a

R i n 4

U 32. Diairman ( /s) 33. Date 40. Other (p/s) 61. Date i

P A P

HJPhillips y

04en /c B/2 6/ 97 BMGrosse 6eEr/P777tC#et

36. 46 (p/

• SS ., Data - 61. Other (p/s) l P , 63. Data s R LJStifte ' 8/27/92.

NSKoe lish h &

I as. Reg. case. hr licPP' 9.

Datay f 64. Other (p/s) 67. Date

]

P5AC 64.

enlF (PI*IPGDahan Meettag Dates

' M/ I ~

t 49 00MPAAC Notaficataan Dates

Review . . .

2

Corrective to. Ceeplete - TRC Dinirman (p/s) 51. Date St. 46 Verification by (p/s) $3. Date j Action i

9401060396 930821 PDR ADOCK 05000275

. hist ritaat ion G PDR NFC Meterials PSRC Secretary 1 Manager. QA Station / Hydro Construction Initiater s'

Plant Manager, DCFP TES Appropriate QC e ODNPRAC Secretary Authertsed Inspecter. Other

j Engineering if ePP!1 cable , Other a cm cto(cm%

i

S

• • 200*3DOd 70101 ** '

RD-27-1952 18516 FR0ri

. . TD S2239061

• P. 82 so.ans (smes .

, - are.:.:::Y" .e..

1

s. = jg=. g . g . g y;- i. ====,=

= - > l 3.

StamiasteretFUnit 1 RT Due to NFW PP 1-1 T fs awasemme ~

s. ammes gases et ammmedammenses

[

j Unit 1 Trf n frem fnu of Main T =' Water Pusm 1-1. This is considered a nonconfornience per 04P 155 paragraph 2.1.2.

1 A

T .

t _

g _ /7 I e, .a. _ - m. -- d -- a -

s. anne

-_ ^- -

a.edgeseus ag. gyte) 33. ante N Ph a

Q

1. 51 sat Gen a rr. amourepuess.ians. e snenen _Suuse af III.

9 L tv. annivene w seenen estesesse assicas -

r

' vs. masa w i - musu== -

a n. m.ms is. mas e as. a.., ma.

i an.s Mtq.lq4 111II name me.,ans g s Il.tiI me Nuclear engineerfng N/A 09/30/92 a a as. nesynne.ra 13. seseessaasy seersas hConstrvetion Servia es l el *

=.iti"=* i i k'i'f"=*mi as. sam ass.e as. antesesse y*

L a as. asess 2"!T* = m a45*#*ri

&- 1 See attached _

8 i.su.i

• • • • 8

.. u =. a .. .,

(# 4 hrs, t a Tenh== D ift M rvisor 12:13 PST '

03/06/92-E

• I T

@j sesWT eri! 3Yday i N 't-00 W 1 So. Wuher aggesses bel 8fA88 04/N92' None

13. ammeres _ _ _ _ _

C

• .____ t k

e - as, e .4 A l F A pi an. m.a. ao. emner HJPhilliDs

~

w in B/2 6f tZ- an l BMGrosse , s 8/.#

l e " '

j

' m en we) " as! asa - 6a. esG G t(Wo f g ustifter as, assa

, m. w.> n. i ALNSKoe b ish ==e w.i .

5. hr M M N. 3884 es. 8her @ ) ,

i w=

ar . .

. =n usurammersz-----

47 hee  !

l ihrvossive ' te. emap8ste

• EDS Gateman We) j metten ' H. Bute E gh veestaasu as ty (pfs) j l 53. Sese

.W. Masasamadas

. i act staterials .

i 9temmeer, 94 abation/Hydre " _ _ _ .4em P3st ^ S i

i Plass manager. Bert tus 3asuster i '

eguptet - -, lassaarland3aspesser, c_-: ge

, l Sugsamortag l

is applisaM e a$er

,! Maar I, ,

. Z 1

j AUG 27 '92 18:18

-! PAGE.EB2 I

) ]W ??,2Wd .- # *CE]E "U--

' ~

~ H I'

^

^--'1'~^ ^' "

~ ' ~ ~ ^ ~ ' " ^

e e

MftttMt Rfouuteert eeweaamaa Docket No. "5 Y$~ 0l{ Otiscnst inn. No &[h -

In the matter of [A*/F/(;_(t$ t&21 h ft [dM$ (C 23f' . _ _ .

IDC:T:F!tD V _

40ra tn , _ _ __ _ . T ' / E '

<{jia7;9 _

htin r of e

C--r . O _

Can:rn:t:t.katy bh.

_ ! $ ons h?>k T',2l#$$

Waness Other Reportet tt t,4 t' f l

l 1

I i

d NCR DC1-92-EM-N010 Rev. 00 Jul.v 29, 1992 MANAGEMENT

SUMMARY

l On March 6, 1992, at 1032 PST with Unit 1 in mode 1 a reactor trip was initiated as a result of a Low-Low Level on the 1-3 Steam Generator.

Operators were in the process of returning the lube oil cooler to service on l the 1-1 main feedwater pump. As part of the process.an operator lifted the grate to gain access to some valves. When the grate was closed, the vibration apparently caused the inverter feed..., both speed channels on the 1-1 nain feedwater pump to fail. An automatic transfer to inverter associated with the 1-2 main feedwater pump failed to automatically occur and power.was lost to both speed channels on the 1-1 main feedwater pump. The loss of the speed channel caused the 1-1 MFW pump to go to a " maximum speed" condition.

At approximately 10:33 a unit trip initiated and all 12 and 4KV busses transferred to Startup power. At approximately the same time, Diesel Generator 1-1 started automatically. Following tha trip the RCS cooldown continued until the main turbine was relatched.

A non-emergency four-hour report required by 10 CFR 50.72(b)(2)(ii) was made 7 to the NRC. An Event Response Plan (ERP) was developed by plant personnel to investigate the event, identify the root cause, review and document all immediate corrective actions required to verify the unit readiness for '

resumption of power operation.

The root cause of this event was determined to be due to the original PG&E power supply design (Pre-1989) for the Lovejoy speed probes being a single l

channel design which was inconsistent with the overall dual channel design philosophy of the Lovejoy system.

The contributory cause was a letter dated 2/27/86 written by a technician EJdressing the potential for lose of both speed channels due to the loss of a single power supply that was not answered. However, the problem identification / resolution process has been refined since 1986 and today, this concern would have been documented, investigated and answered on an Action Request per Administrative procedure C-12.

To prevent recurrence NECS Electrical Engineering will issue a departmental memo to describe the event as a lesson learned.

l As prudent actions NECS Engineering will: (1) provide a design for the-120VAC l

power supply needed for the Lovejoy speed probes on the Main Feedwater Turbine needed to replace the failure prone Abacus inverters and (2) will review DCM S-23 to determine its consistency with the design philosophy referenced in this NCR, to propose a change.

92NCRWP/92EMN010.PGD Page 1 of 25 l

l l

~

NCR DC1-92-EM-N010 Rev. 00' July 29, 1992 NCR DCO-92-EM-N010 Unit 1 Mactor trip due to Main Feed Water Pump 1-1 Trip.

I. Plant Conditions

! Unit I was in Mode 1 (Power'0peration)' at 100% power.

II. Description of Event A. Sumary:

On March 6, 1992 at 1032 PST a Unit 1 trip was initiated due to'a low-low Level in Steam Generator 1-3. The low-low water level steam generator condition was_ caused by a Main Feedwater Pump (MFWP) 1-1 trip.

B.

Background:

I The MFW inverters were installed by NECS GC under DCN 41705/42705 '

in November of 1989 (Ref:-AR.# A0161985/A0172926 WO # ,

C0060409/C0067073). The inverters performed'normally'until'May 27, 1990 when the first failure occurred'on inverter IYFW22.

Steam condensate running into the inverter.and shorting out two power transistors, was assumed to be the only cause of failure.

The transistors were replaced and the. inverter was tested at 500 watts for one hour and-then reinstalled.

On June 27, 1990 a-second failure occurred on inverter IYFW21:and a new inverter (Ref:' stock code # 73-7636)_from the warehouse stock was installed. The failed inverter was repaired by replacing the driver and logic cards and returned to the.

warehouse.

On April 25, 1991 the third failure occurred on IFW12 and a new inverter from the warehouse was installed. The' failed inverter was repaired by replacing the power supply. card and was. returned.

to the warehouse.

On May 18, 1991 the fourth failure occurred on~IYFW11 and an inverter from the warehouse was installed. -The failed inverter was sent back to Abacus Controls for failure analysis. Abacus j Controls determined that the power supply card had failed and in.a communication to Electrical Maintenance Engineering dated June 27, 1991, Abacus Controls determined that the power supply. card was" i

designed for operation from 200-250 VDC source, while ours 1s- ,.

supplied by 135 VDC. Due to-the failures at DCPP,- Abacus Controls' 92NCRWP/92EMN010.PGD Page 2 of 25 ~

p .

i l .

i 5

- NCR'DC1-92-EM-N010 Rev. 00-July 29,~1992 I i

g r

came to the conclusion that the input switching transistor Q1 was ,

operating with insufficient; base current. Their' solution =to this- ,

design flaw was to change the;value of R3 in a11,120 VDC power.

supply cards which meant a simple ~ schematic and parts list revision. ,

~

On October 20 1991, the fifth failure occurred on' inverter.IFW21 and an inverter from the warehouse was: installed.: The failed: -

inverter was sent'back_to Abacus Controls for analysis and again.  ;

it was determined that failure of the inverter was due to the-power supply card.

On October 26, 1991 the sixth failure occurred on inverter IYFW22, and an inverter from the warehouse was installed. Again the failed inverter was sent back to Abacus Controls for analysis and R3 upgrade. Abacus. Controls determined that this failure was due to- .'

the power module and was not related to the previous power supply.

j card failures.

On December 3,'1991, the. seventh failure occurred on inverter. <

IFW11 and also a failure to transfer. due:to the relay' contacts .

being miswired. Two driver cards and a power module'were replaced.

h and the inverter returned to service. Functionalitesting resulted- 1 in intermittent failure to. transfer caused by the'Agastat relay-plunger being improperly installed. The agastat relays were  ;

replaced and the auto-transfer functions:were satisfactorily  ;

tested. A Quality Evaluation (QE)'was written for root cause and  !

corrective action. Electrical Maintenance. Engineering contacted-Abacus Controls and suggested that they find 3 way to remove the excessive heat being generated by'R4 through.R7:on the driver  :;

cards. On December 5, 1991 in a letter from Kevin' Muller.of -

Abacus Controls stated that they were upgrading the 452-4-125-M--

NMN which would have the R3 upgraded, remove R4 through RT from the driver card and mount equivalent resistance in the open area of the chassis. An RPE was immediately requested to allow this- j modification.

On December 28, 1991, the eighth failure occurred on inverter .

IYFW12 and an inverter from the warehouse with the R3. upgrade by-the vendor was installed. The failed = inverter was repaired by-replacing a driver card, power supply card,' power module and R3 upgrade (implementing DCN 47165). The unit:was put back-in the warehouse and by January 30, 1992 all inverters.in; service and.

power supply cards in stock were upgraded completing'DCN: ~

~

47165/48165.

92NCRWP/92EMN010.PGD. ,Page. 3 of 25.-

--___ _____, , . - . , . . - , - - - - , - , - - - - - - , , , - 4,---w,...m[w,r- .,m ,,y,i , ,..,..he~ h -c ., . , , m 9 .w , , . . . , .

1 NCR DC1-92-EM-N010 Rev. 00 July 29, 1992 ]

l On March 6, 1992, the ninth failure occurred on inverter IYFW11 when an operator dropped a grate adjacent to the inverter panel and a solder connection opened up causing excessive current to be l drawn blowing both DC input fuses. .The auto transfer also failed because of debris in a relay contact which resulted in loss of power to both Lovejoy speed probes and tripped the feedwater. pump.

! C. Event

Description:

l On March 6,1992, while the operators were returning a lube oil

! cooler to service on MFWP 1-1, an operator lifted a small grate to l look at the oil cooler transfer valve. After completing his..

j inspection, the operator lowered the grating to approximately a forty five degree angle and let it fall close. At that time he heard the feedwater pump local alarm and observed the alarm for the MFWP 1-1 inverter undervoltage flashing. When the grate was closed, the inverter powering the speed chamels on MFWP 1-1 failed. A transfer to the inverter associated with the MFWP 1-2 l failed to " automatically" occur and power was lost to both speed channels on the 1-1 MFWP. The loss of the speed channels caused the 1-1 MFWP to go to a " maximum speed" condition.

At 10:29 PST the control room received a "FEEDWATER PUMP 1-1 SYSTEM PROBLEM" alarm. This was the first indication in the control room of a problem with the 1-1 MFW pump. The MFW pump 1-2 backed down due to the increased flow .

At 1030 PST the Main Feedwater Pump (MFWP) 1-1 tripped.-

Indications from the vibration monitoring equipment on the pump indicated that the turbine tripped on overspeed. Operators immediately began reducing the Unit load and stabilized it at approximately 510 MWe.

At approximately 1032 PST the reactor tripped due to a low-low level on the Steam Generator 1-2 and the low Tavg setpoint reached (554 degree F). A feedwater isolation initiated due to a reactor.

trip with a Low Tavg coincidence. At approximately the same time, the MFW pump 1-2 tripped on a high discharge pressure when the feedwater isolation occurred. Both motor driven Auxiliary Feedwater pumps started due to the trip of the MFW pump.

At approximately 10:33 a unit trip initiated and all 12 and 4KV busses transferred to Startup power. At approximately the same time, Diesel Generator 1-1 started automatically.

92NCRWP/92EMN010.PGD Page 4 of 25

l

.- NCR DC1-92-EM-N010 Rev. 00 July 29, 1992 i

Following the trip the RCS cooldown continued until the main turbine was relatched. The minimum Tavg reached was approximately l

l 520 degree F. Once the turbine was relatched the RCS Tavg was

! returned to T-no load.

At approximately 12:13 PST a non-emergency four-hour report required by 10 CFR 50.72(b)(2)(ii) was made to the NRC.

An Event Response Plan (FRP) was developed by plant personnel to investigate the event, identify the root cause, and review and document all immediate corrective actions required to verify the l unit readiness for resumption of power operation.

l D. Inoperable Structures, Components, or Systems that Contributed to the Event:

MFWP 1-1 iaverter and transfer relay.

E. Dates and Approximate Times for Major Occurrences:

l

1. March 6, 1992; 10:28 PST: Control room receives a "FWP 1-1 Turbine Sys. Problem."

2. March 6, 1992; 10:30 PST: Main Feedwater Pump.1-1 Trips.

3. March 6, 1992; 10:32 PST: Event / Discovery date. Unit 1 Reactor Trips due to a low-low water level on Steam Generator 1-3,

4. March 6, 1992; 12:13 PST: A non-emergency four-hour report required by 10 CFR 50.72(b)(2) l (11) was made to the NRC.

F. Other Systems or Secondary Functions Affected:

10% STEAM DUMP VALVES.

l

1. Following the reactor trip, three of the 20 percent steam l dump valves (PCV 19,20,21) were cycling off their seats for  ;

approximately 1 or 2 minutes. The loss of the MFWP 1-1 i caused the loss of load signal C-7B and as designed the '

modulating solenoid' valves energized and swapped the 10  !

percent steam dump valves from the SG pressure controller to I the load rejection controller. The rapid increase in pressure from the load rejection I/P output (0.5 psig) to the SG pressure control I/P output (3 psig) was enough to cause 92NCRWP/92EMN010.PGD Page 5 of 25 i

NCR DC1-92-EM-N010 Rev. 00 -

-July 29, 1992 the volume booster to overshoot and open the three 10 percent dump valves. PCV 22 did not open. An investigation revealed '

that this valve lift was due to an overly reactive air volume boost and valve positioner system. Various improvements are being considered to improve the 10 percent steam dump valves performance stability. ]

Therefore, with the exception of the following two issues the control system operated as designed:

1. Subsequent to the reactor trip , PCV 19/20/21 briefly opened even though steam generator pressure was below the control system setpoint at 1035 psig.

2. Subsequent to the event, the output of PCV-22 S/G pressure control I/P was observed to be 1.0 psig versus the desired 3.0 psig, thereby explaining why PCV -22 did not o' n.  !

DIESEL GENERATOR START. 1 Diesel Generator 1-1 started automatically due to a voltage  ;

transient on the vital bus during the transfer to startup power. l This D/G start experience had the same characteristics as other l D/G starts on bus transfers. Bus "H" is usually lightly loaded '

which causes the voltage to decay slower on this BUS than the other busses when the auxiliary breaker is opened. +In this case it took about 1.8 seconds from the time the auxiliary breaker opened to when the startup breaker closed. Busses "G" and "F" took about 0.5 and 0.8 second respectively. This long decay time allowed the diesel start relay sufficient time'to close it's contact and start the diesel. Busses "G" and "F" decay time was not long enough to allow the diesel start relay to time out. As a confirmatory measure relay 27HHB1 was bench tested. (See WO C0097640). In addition the relay that checks 25% bus voltage to allow the autotransfer was checked on "H" bus and was found to be set correctly. The as found relay drop out voltage was in-accordance with maintenance procedure E-50.21. No other problems were noted.

G. Method of Discovery:

The event was apparent to the control room operators when they received a "FWP 1-1 Turbine System Problem" alarm and the Main Feedwater Pump 1-1 tripped. The operators immediately began reducing Unit 1 turbine load to about 510 MWe. A reactor trip was initiated due to a low-Low Level on the 1-3 Steam Generator.

92NCRWP/92EMN010.PGD Page 6 of 25

~

NCR DC1-92-EM-N010 Rev. 00 July 29, 1992 H. Operator Actions:

Operators immediately began reducing power and stabilized the unit load at 510 MWe. A post event analysis of operator response was conducted by the DCPP Operations section and operator actions taken in response to the loss of MFWP 1-1 were appropriate and in accordance with abnormal procedure AP 15.

I. Safety System Responses:

1. The reactor trip breakers (BKR)(JC) opened.

2. The control rod drive mechanisms (CRDM)(DRIV)(AA) allowed the control rods to drop into the reactor.

3. The main turbine (TRB)(TA) tripped.

4. The motor driven auxiliary feedwater pump (P)(BA) started automatically and delivered water to all steam generators as required.

III. Cause of the Event A. Immediate Cause:

Operators were in the process of returning a lube oil cooler to service on the 1-1 Main Feedwater pump. An operator lifted a ,

small grate above the lube oil cooler to look at the transfer valves. After completing his inspection, the operato* lowered the grating to approximately a forty five degree angle and let it. fall close. At that time, he heard the feedwater pump local alarm and observed the alarm for the MFWP 1-1 inverter undervoltage flashing. When the grate was closed, the inverter feeding both speed channels on the 1-1 MFWP failed. A transfer to the inverter associated with the MFW 1-2 pump failed to

" automatically" occur and power was lost to the speed channels on the 1-1 MFW pump. The loss of the speed channel caused the 1 MFW pump to go to a " maximum speed" condition which later resulted in the MFWP trip due to overspeed. The reactor tripped several minutes later due to a low-low water level on steam generator 1-3.

92NCRWP/92EMN010.PGD Page 7 of 25 4

'NCR DC1-92-EM-N010 Rev. 00 \

July 29, 1992

.B. Determination of Cause:

1. Human Factors: N/A

2. Equipment / Materia 1:

a. Material Degradation:-

On March 6,1992, Unit 1 MFWP 1-1 suffered a loss of power to both speed probes. This loss of power occurred at the exact same time that.an operator dropped a' grate near panel CTFW11. During the troubleshooting that . '

followed under Work Order C0097599,.it was discovered that there was no power on the output of.MFP 1-1 inverter IYFW11. Adcitionally.it was discovered'that both SA fuses on.the IYFW11 inverter were-blown.

I"verter-IYFW11 was taken to the shop for troubleshooting and repair. Upon inspection of.one of the 20798 power modules, both 47 ohm, resistors, had been discolored showing obv.ious_ overheating. After removing-the module from:the chassis it was further noted that the connections between the diode and the power transistor had opened up. The effect of the connection opening was to cause excessive. current to'be drawn,  :

blowing the DC input fuses and shutting down.the inverter. . It. appears that the connection was warm and then excessive vibration caused-it to open. The .

excessive. vibration could:have been~ caused when the grate was dropped leading to the immediate alarm which followed as observea.by the operators. However the failure of the connection'could.have been due to metal fatigue of the solder in the connection, the dropping of the grate being of sufficient magnit"de to complete the fracture. Electrical Maintenance tried to simulate the.

conditions that caused the trip by dropping the grate '

several times. The auto transfer failed because of debris in a relay contact.

DCP E-47458 adds a separate power' supply for one of the two Lovejoy system speed probes on each MFW pump.- Both speed probes are presently supplied by the feedwater inverters. This design change will prevent a~ loss of both speed probes.if the inverter system fails again.

The failure of this inverter caused the MFW pump to go to a " maximum speed". condition whichLresulted in the-i pump trip.

92NCRWP/92EMN010.PGD Page 8 of 25

- .. . - - . - -- . - . . -._ - . - . _ ~ - . . ._ . . - . .. - .

1

. l NCR DCli 92-EM-N010 Rev. 00

.~ Juiy 29,: 1992 b.. Design:

What are the design issues involved in utilizing ani

-inverter rated 250 Viin a 125 V: application?,~Is.there  :

-adequate cooling?. Improper biasing.of the. transistors? i After the-fourth failure,-the1 inverter was returnedito' Abacus Controls forLanalysis. They determined that the~  !

power supply ;1rd had failed because it was designed for ;i operation from a-200-250 VDC-source while?our unit was-operating from'a=135 VDC_ source.- They concluded that. .

the input switching transistor Q1 was'being operated..

with insufficient base current.;TheirLsolution-to'this' i design flaw.was to change the value of R3 in 'a11 120.VDCL power supply cards which:resulted-in DCN E-47165/48165. .

Although:there were.no failures of inverters withithe R3.  !

change until the most recent one, Lit was obvious that ,

~ there wasLa significant problem with heat building ^1n - -

the card' cage..' Abacus Controls was again contacted-and:

~

we requested that they come up'with a way to remove excessive heat being generated by R4 tthrough.R7 on the--

driver-cards. .Their' response was to upgrade the inverter which would have the-R3 change and;also remove

- the R4 through' R7 from the' driver: card ~and mount :

equivalent resistance in the open area of the~ chassis'..

An RPE was immediately requested to allow this' design modification to be implemented. -

c. Installation:.

An investigation made on the reason why'the lugs in the ,

- transfer' relay-were'too big to fit into:the! terminal-relay revealed that original construction and now current specs'do not show the'use~of compression type

, wire termination's.. It has.become standard. practice in NECS and DCPP Electrical Maintenance.that when a:

compression' type of termination:is supplied with a ~

component, that the compression termination: plates are' .

. removed and acring lug termination is made using the-original,screwLhole.:Thelcontactors.in-question were- i supplied with. compression. type termination. hardware 1that~ j was removed to' allow for a'ringilug to; be"used. However, i the screw size. required a: larger lug to be used than

' there was room to fit into the'teminal relay.-

1 92NCRWP/92EMN010.PGD. Page 9 cif 25 '

> 1

-yy*

. -yvy rw*-- ""i--,m-v y i-r-~r -" --*w --

g---+t-

f NCR DC1-92-EM-N010 Rev. 00 -

l July 29, 1992

d. Manufacturing:

f N/A.

e. Preventive Maintenance:

N/A r

j f. Testing:

l Do we periodically test the power supply transfer scheme? If so why didn't previous PMT's discover the transfer problem.?

l When the original design was installed in 1989/1990 they l did post-modification testing on the inverters and on l

the transfer scheme. In both cases their te' ing were successful. The reason they were successful is that in one case they had no problem, and in the other case there was a contact that was wired normally open.instead l of normally closed. That was coupled with an Agastat relay that had an instantaneous contact plunger that was assembled improperly. The two errors resulted in a circuit which operated correctly during testing, i

g. End-of-life failure:

N/A C. Immediate Cause:

The immediate cause of this event was due to the low-low water level on Steam Generator 1-3.

D. Root Cause:

The original PG&E power supply design (Pre-1989) for the lovejoy speed probes is a single channel design and was inconsistent with the overall dual channel design philosophy of the Lovejoy system.

E. Contributory Cause:

l The contributory cause was a letter dated 2/27/86 written by'a technician addressing the potential for loss of-both speed

' channels due to the loss of a single power supply that was not answered. However, the problem identification / resolution process l has been refined since 1986 and today this concern would have been 92NCRWP/92EMN010.PGD Page 10 of 25 j 1

~

NCR DC1-92-EM-N010 Rev. 00 July 29, 1992 documented, investigated and answered on an Action Request per Administrative procedure C-12.

IV. Analysis of the Event A. Safety Analysis:

Inadvertent loss of all main feedwater flow is a condition II event as described in the Final-Safety Analysis Report (FSAR)

Update. This type of event has been analyzed in FSAR Section 15.2.8, " Loss of Normal Feedwater." The reducing of load by the operators to approximately 510 MWe to stabilize the unit and the automatic start of the auxiliary feedwater pumps ensured that an adequate supply of water was provided to the steam generators for the cooldown of the reactor per design. Thus the health and safety of the public were not adversely affected, and there were no adverse consequences or safety implications resulting from this event.

B. Reportability:

1. Reviewed under QAP-15.B and determined to be non-conforming in accordance with Section'2.1.2.

2. Reviewed under 10 CFR 50.72 and 10 CFR;50.73 per NUREG 1022-and determined to be reportable in accordance with ,10 CFR 50.73(a)(2)(iv). The report associated with this NCR is LER 1-92-002-00.

3. This problem does not require a 10 CFR 21 report, since.it is being evaluated under 10 CFR 50.72(b)(2)(ii) and it is a class II equipment.

4. This problem does not require reporting via an INPO Nuclear Network entry, however a voluntary entry will be made.

5. Reviewed under 10 CFR 50.9 and determined the event was not reportable under 10 CFP 50.9 since this event is being reported under 10 CFR 50.73.

6. Reviewed under the criteria of AP C-29 requiring the issue.

and approval of an OE and determined that no.OE.is required.

92NCRWP/92EMN010.PGD Page 11 of 25

l .

NCR DC1-92-EM-N010 Rev. 00 July 29, 1992 V. Corrective Actions A. Immediate' Corrective Actions:

1. An Event Response Plan ERP 92-002 was' initiated to address-the immediate corrective actions required-to restart Unit 1 and is hereby made part of the NCR to facilitate complete- ,

documentation.

2. Install DCP # E-48458 in Unit 2 ASAP, to provide a separate power supply for one of the two Lovejoy system speed probes.

on each MFWP.

RESPONSIBILITY: R. JOHANSEN DEPARTMENT: I&C Tracking AR:- A0260610, 1 STATUL. COMPLETE.

3. Replace the pump power supply with new modules.and inspect ,

the contractors in the transfer scheme for both Unit 2 MFWP.  !

RESPONSIBILITY: J. RAPPA DEPARTMENT: Electrical Maintenance  !

-Tracking AR: A0260704, STATUS: COMPLETE.

B. . Investigative Actions:.

1. Electrical Maintenance and NECS to review the' design'of.the MFWP power supoly scheme to improve its reliability and simplify the design of the Lovejoy speed probes.

RESPONSIBILITY: B. Grosse DEPARTMENT: NECS Electrical.

Tracking AR: A0260772 AE #: 1 STATUS: COMPLETE.

i

2. NECS Engineering to go back and look at their design review process and determine if there is any improvement that can be made into the present system to increase reliability.

i RESPONSIBILITY: Bruce Grosse DEPARTMENT: NECS Electrical.

Tracking AR: A0260772 AE #: 2 STATUS: COMPLETE.

3.

Investigate why the lugs in the transfer relay are too big.

92NCRWP/92EMN010.PGD Page 12 of 25 i

l NCR DC1-92-EM-N010.Rev. 00 Jub 29,1992 RESPONSIBILITY: J. Nystrom DEPARTMENT: Work Planning Center.

Tracking AR: A0260772 AE #: 04 STATUS: COMPLETE.

C. Corrective Actions to Prevent Recurrence:

NECS Electrical Engineering will issue a departmental memo to describe the even* =s a lesson learned.

RESPONSIBILITY: B. Grosse DEPARTMENT: NECS Electrical.

Tracking AR: A0260772, AE # 03 STATUS: COMPLETE.

Outage Related? No OE Related? No NRC Commitment? Yes CMD Commitment? No D. Prudent Actions (not required for NCR closure)

1. Provide a new design for the 120VAC power supply needed for the Lovejoy speed probes on the Main Feedwater Turbine, to replace the failure prone Abacus inverters. Detailed design results should be IAW agreements reached in 03/19/92 meeting with NECS-EE(BMG) et.al.

NECS Electrical Engineering and Electrical Maintenance to review the four design change options of the feedwater pump turbine speed sensing and control power supply and determine which option should be used.

RESPONSIBILITY: D. Moyer DEPARTMENT: Electrical Engineering.

Tracking AR: A0262790 STATUS: COMPLETE.

2. Review DCM S-23 to determine its consistency with the design philosophy referenced in this NCR and propose a change.

t RESPONSIBILITY: B. Grosse ECD: 08/30/92 DEPARTMENT: NECS Electrical.

Tracking AR: A0270003, 92NCRWP/92EMN010.PGD Page 13 of 25 i

4  %

NCR DC1-92-EM-N010 Rev. 00 -

July 29, 1992 VI. Additional Information A. Failed Components:

Inverter model number IYFW11 (Abacus Controls Inc. Model No. 452-4-125-M-NMN) failed.

B. Previous Similar Events:

NCR 1-90-TI-007. Manual Reactor Trip due to Loss of feedwater to the Steam Generator. The root cause of this event was unknown. However the most probable cause was either a non repeatable Solid State Protection System (SSPS) card failure or an inadvertent actuation caused by I&C technicians working on the SSPS racks. There was no corrective action taken to prevent recurrence; however a precautionary measure was taken to replace the SSPS cards A213 and A517. This precautionary measure would not have prevented this event.

NCR 1-91-TI-N045. Reactor trip from Turbine Trip due to Steam Generator 1-3. The root cause of this event was due to the failure of the " track and hold board" circuit board the Lovejoy feedwater pump control system which caused the pump speed to increase until " low selected" at the preset (6100 RPM) level of the manual start up station. This sudden increase in feedwater .

pump speed overfed steam generator 1-3 due to the selected mode of valve control resulting in a high level P-14 actuation causing a turbine trip and reactor trip.

The corrective action to prevent recurrence was to provide an Operations Incident Summary.

As a prudent action the TRG recommended that DCPP and GO system engineers that a long term replacement for the Lovejoy speed control system be considered.

This corrective action and precautionary measure would not have prevented this event.

92NCRWP/92EMN010.PGD Page 14 of 25

I

=

~

NCR DC1-92-EM-N010 Rev.-00'

. July 29,<1992 C. Operating Experience. Review:

1. NPRDS:

Not applicable.

2. NRC Information Notices, Bulletins, Generic Letters:-

N/A.

3. INPO SOERs'and SERs: N/A D. Trend Code:

Responsible department NE, and cause code'C1.

E.- Corrective Action Tracking:

1. The tracking action request is A0260772.-

. 2.- Are the corrective actions outage relatedt No.,

l F. Footnotes and Special Comments: .

None.

G.

References:

1. Q0009378

2. A0260474

3. A0260704

4. A0260481

5. Licensee Event-Report (LER) 1-92-002.

6. Event Response Plan.(ERP) 92-3.

7. FWP inverters history..

8. Analysis of operator response.

9. ' Background write-up

10. Case study of the event.- '

92NCRWP/92EMN010.PGD. Page 15- of 25

4 NCR DC1-92-EM-N010 Rev. 00 -

July 29, 1992-

11. Letter dated 02/27/86 from K.P. Lucien to R.P. Gilbreath.

Subject:

Lovejoy System, Unit 2.

12. Power supply design change options.

13. NECS lessons learned letter dated 07/22/92. CHRON 193257.

14. Event and causal factor chart.

15. Letter from Maintenance Maintenance Manager to Managers and directors - Subject Operability of equipment' required by Technical Specifications.

H. TRG Meeting Minutes:

1. On March 13, 1992, the TRG convened and considered the following items:

a. The NCR write-up was reviewed for editorial content.

b. Review of the investigations conducted by Electrical Maintenance.

1. Simulation of the event. The grate was dropped a number of times to recreate the event.

2. Reason why the power supply and inverter failed.

3. Inverter heat problem. i 1

c. Discussion on the timeliness-issue. l

d. Discussion and establishment of the root cause. The root I cause of this event was established to be due to the vendor poor design which included a weak connection whose failure resulted in a loss of the inverter.

e. Establishment of investigative actions as follows: I

1. Install DCP # E-48458 in Unit 2 ASAP, to provide a separate power supply for one of the two Lovejoy system speed probes on the MFWP. Status: Complete.

l

2. Replace the pump power supply with new modules and l inspect the contractors in the transfer-scheme for both Unit 2 MFWP. Status: ' COMPLETE.

92NCRWP/92EMN010.PGD Page 16 of 25 l .. .

NCR DC1-92-EM-N010 Rev. 00 Jul3 23, 1992

3. NECS and Electrical Maintenance to review the design of this MFWP power supply scheme and recommend what should be done.

4. The MFWP inverters on the Unit and Unit 2 have been replaced.

f. Associated QEs need to be reviewed and discussed.

g. Some updates need'to be added to this NCR write-up. The background write-up to be provided by the Electrical section; the AMSAC issue by the I&C section; the diesel generator start additional write up by Steve Foat and the determination of cause by Dale Moyer.

TRG TO RECONVENE ON THURSDAY MARCH 19. 1992 AT 10:00 AM.

2. On March 19,-1992 at 9:00 AM the TRG reconvened and considered the following items:

-a. Electrical section director to meet with Engineering to determine actions to be taken to' correct the inverter problem. Reliability invited to the meeting.

Three options available:

1. Put a simple AC source.

2. Leave one speed probe on transfer.to back-up and the other speed probe on AC..

3. In 1R5/2R5 I&C a design change will be installed which consists of speed probes analyzing digital board in the lovejoy system that will have the capability in failing in an "as is condition". The board which is in place now requires a specific-default.value upon loss of power.

This design change was to be put in 1R4/2R4, but because of certain reasons, it has been postponed to 1R5/2R5. It-is now ready to be installed.

b. Talked to the supplier about the heat problem on the inverter.

92NCRWP/92EMN010.PGD Page 17 of 25

NCR DC1-92-EM-N010 Rev. 00 l July 29,-1992

! c. Discussion on the solder melting point.and the h' eat l problem. Evidence of heat and cold solder joint in the -

l connection.

d. Reliability of the inverter is not acceptable.

l l e. Write an INP0 Network entry on this event. Regulatory i Compliance is-to make the network entry on this event.

t l f. Discussion on the design and manufacturing of the l inverter.

g. Discussion on the oversized lug. Two criteria to be- '

l considered, the wire size and the. screw terminal' size.

i

h. WPC to follow up with'GC construction to identify the design and procedures for the termination duriag the -

original construction / installation of the inverter's relays.

1. Previous failures to transfer are documented on a QE.  ;

j. Review oflthe NCR write up.~ $

! k. Two issues TRG needs to rev.iew before reconvening on this event. One is timeliness and the other is the-previous corrective actions on inverters failure. -One thing needs to be considered is that we should have written off the power supply and get a different power source.

l i

1. In the previous failures the reason we did not change i the power supply was because of money.

TRG TO RECONVENE ON APRIL 16, 1992'AT 10:00 AM.' l

3. On April 17, 1992 9 10:00 AM the TRG reconvened and i considered the following items: 1

a. Federman and Grosse to be invited to attend the next TRG. QC to notify. i

b. Incorporate in the NCR, the Operations director's response to the operator's response regarding the loss of main feedwater.

92NCRWP/92EMN010.PGD Page 18 of 25 i

NCR DC1-92-EM-N010 Rev. 00 July 29, 1992

c. The electrical section manager'gave a slide presentation to better illustrate the failure of the main feedwater j pump inverter. '

d. Need to incorporate in the NCR:

1. Something about the surveillance conducted by QA -

No explanation on the QE surveillance. If QA/QC  !

found something wrong they need to call the i responsible person.  ;

2. Lesson learned: when you see multiple failures, redesign.

e. Discussed the root cause of the Main feedwater pump trip.

1. As found condition: relay open.

2. It.is now believed that the vibration induced-caused the solder joint to separate.

f. Discussed the immediate corrective actions.

g. Discussed the chronology of events.

1. Review of NCR DC2-89-N021' o Root cause incomplete clearance process.

o Corrective actions

h. Electrical section will give Regulatory Compliance a copy of the writeup prepared for Warren Fujimoto.  ;

i. Reviewed corrective actions,

j. Will established more root cause and corrective actions when the next TRG reconvene. At that time should have l detailed root cause'and corrective actions from l Engineering.

l

k. Discussed the chronology of events, including the March 6, 1992 event.

1. Discussed the root cause and potential corrective actions. -;

92NCRWP/92EMN010.PGD Page 19 of 25 l

l l

. 1 t

NCR DC1-92-EM-N010 Rev. 00 July 29, 1992

m. Discussed pro and con of rapid and inaccurate. )

information and slow and accurate information. l

n. QA surveillance discussed.  !

I'

o. By the next TRG should have:

1. Complete root cause and corrective actions prepared for review.

2. Contributory causes.

3. Barrier analyses.

4. Chronology.of. events.

1 L

5. Initial design. review. -

p. Consider an amendment of the LER.

TRG TO RECONVENE ON MAY 5. 1992. ,

l

4. On May 5, 1992 9 1:30 PM the.TRG reconvened and considered l the following items:

l

a. An event and causal.' factor prepared by engineering was reviewed by the TRG. .The Lovejoy system was installed ]

i around 1986. l

b. The root cause of this' event was established. The two-speed probes had a single power supply rather.than two power supplies. The final root cause established is as follows: The original:.PG&E power suppl., design (Pre-1989) for tne.Lovejoy speed probes is a single channel design and is inconsistent with the overall dual channel design philosophy.of the Lovejoy' system.

l c. Causal factor. The 1989 NCR identified as root cause

! d. The Lovejoy system was originally designed-with two speed probes which redundancy. However,gave some flexibility this philosophy and has not'been carried out in the present design.

e. Question was asked "why in 1989 when we performed the FMEA on this design we did not catch this problem? .

92NCRWP/92EMN010.PGD Page 20 of .25 l

l t

NCR DC1-92-EM-N010 Rev. 00 July es, 1992

f. Trend code is: NECS as the responsible department and the cause code is C1.

g. The following two investigative actions were established:

1. Electrical Maintenance and NECS to review the design of the MFWP power supply scheme to improve its reliab"'ty and sinplify the design of the lovejoy speed probes.

2. NECS Engineering to go back and look at their design review process and determine if there is any improvement that can be made into the present system to increase reliability.

h. Take credit for the joint meeting between NECS electrical and electrical maintenance .

Reconvene the design review. Finalize the root cause and 1.

continue to work on the corrective actions.

j. Covered the inverter failure which has been discussed in the component failure analysis. The inverters were not part of the NPRDS.

k. The following two issues have not been discussed by this TRG:

1. Timeliness versus overall general issue.

2. Important to Safety versus safety related.

Somebody or some group will r.eed to address them.

1. It was recognized that the issue of class B program needs to be addressed and it is known that the I maintenance rule has mandated that this issue be l addressed. However, it will have to be addressed, since l the maintenance rule will be implemented by 1996.

m. The following documents will be referenced in the NCR:

Q0009378; A0260474; A0260704; A0260481 TRG TO RECONVENE ON MAY 28, 1992 AT 10:00 AM.

92NCRWP/92EMN010.PGD Page 21 of 25 3

4 c

I NCR DC1-92-EM-N010 Rev. 00 -

July 29, 1992 l

l

5. On May 28, the TRG reconvened and considered the following items:

a. Reviewed NECS answer to their investigative action.

b. NECS presented a slightly modified version of their last event and causal factor chart in regard to the root Cause.

c. NECS also presented a letter dated 2/27/86'from a GC technician expressing concerns on the design, fabrication and component-shortcomings of the Love Controls Corporation (LCC) Feedwater Pump' Speed Control System.

d. The discussion on this-letter centered on the subject .

"when somebody addressed a concern why was no-action  ;

taken?". The answer to that question, underlinr2 that R the method of resolving problems then, is much different- i that it is today. Today, an ATAR or some form of action j may have been taken and the concerns' investigated. This  :

letter was only found last week.  :

e. It was decided not to take any further action, regarding

! the determination'of "what if that letter was answered"..

f. On the basis of this letter, the root cause established i at the last meeting was reviewed and discussed. However, i j no change to the root cause was made. l l

g. Three ideas for corrective actions were proposed:

l 1. Send a memo to Design Engineering that talk about a t

i system based approach which requires.to concentrate I on the failure when a FMEA is performed on a'DCP.

Need to determine if it affects other components.

i

2. Document the changes in the design process (Start i

up and test program). .

The NECS Engineering representative. indicated that the process has been improved by the creation of an independent design review group which was formed:in 1990. Prior to this time design _ reviews were performed by members'of the same group. NECS E-3.6 DC provides a procedural guidance for design reviews, including a series of checklists which prompt the reviewer to look at the design impact on 92NCRWP/92EMN010.PGD' Page 22 of 25

I p, .

~

NCR DC1-92-EM-N010 Rev. 00 July.29, 1992 a more global scale. - However,~ B0P equipments are not included in these checklists. These checklists are to be reviewed when BOP equipment can be addressed with respect'to-DCN considerations.. AP C-55 should be used-'as a guide.

3. Take. credit for a task force put together by W.

Fujimoto which will look' at the overall feedwater

' design system.. Reliability Engineering will be provide information on the systems,

h. Specific to this incident the Electric Maintenance-director in concert with other maintenance directors will' elevate the conscientiousness in regard to issues that require intense maintenance work after a design change has been resolved. l

1. Question as a prudent action. -Do we need to remove the.

inverters'from the plant?

TRG TO RECONVENE ON 6/26/92 FOR FINAL ROOT CAUSE AND CORRECTIVE ACTIONS.

6. On June 26, the TRG reconvened and considered.the'following. I items:

l

a. Agreement that 8. Grosse will become.a voting. member of .

this TRG, since he has got significant interest in this~ ~

NCR. ,

! b. It was agreed that the root ciause'as presently. stated

( was correct and.no modification of corrective' action was

! necessary due to the February 27, 1986 letter. presented .

l at the last TRG. ,

c. Discussed the letter dated February 27, 1986, from K.P.

Lucien, a GC technician to R.P.lGilbreath on the subject of the Lovejoy System'on Unit ^ 2, which led to the slightly modified root cause. .

This letter was presented by Electrical' Engineering on May 28,.1992 with a slightly modified root cause!on the basis of the review of this letter. -It was then agreed that although this letter may have prevented.this event-if actions had been taken, it did not change.the root cause of this event as presently stated.- The philosophy.

of this letter is. centered on "when somebody addressed.a concern why was no action'taken?".- The-answer to that.

92NCRWP/92EMN010.PGD. Page 23 of 25

NCR DCl-92-EM-N010 Rev. 00

• July 29, 1992 question, underlines that the method of resolving problems then, is much different that it is today.

Today, an ATAR or some form of action would be taken and the concerns investigated.

d. NECS Electrical Engineering distributed copies of the power supply design change options for the feedwater cump turbine speed sensing and control.

e. The various options were discussed and subsequently it was recommended by NECS Electrical Engineering to get together with Electrical Maintenance to review the four design change options of the feedwater pump turbine speed sensing and control power supply and determine which option should be used.

f. The rollowing prudent actions were ir.itiated:

a. Provide design for 120VAC power needed for the Lovejoy speed probes on the Main Feedwater Turbine, replacing the failure prone Abacus inverters.

b. Review DCM S-23 to determine its consistency with the design philosophy referenced in this NCR and if required, propose a change to the DCM.

g. The following corrective action to prevent recurrence was initiated:

NECS Electrical Engineering will issue a departmental memo to describe the event as a lesson learned.

RESPONSIBILITY: NECS Electrical ECD: 0,/10/92

h. The following investigative action was initiated:

Investigate why the lugs in the transfer relay were too wide for the contactor.

RESPONSIBILITY: WPC ECD: 07/10/92 l

1. The TRG adjourned and decided to:

l RECONVENE ON 07/29/92 AT 10:00 AM TO REVIEW THE NCR WRITEUP AND THE RESULTS OF THE INVESTIGATIVE ACTIONS.

l 92NCRWP/92EMN010.PGD Page 24 of 25

NCR DCl-92-EM-N010 Rev. 00 l July 29, 1992 ]

7. The TRG reconvened on 7/29/92 and considered the following items:

a. Review of the status and answers of the investigative and corrective actions to prevent recurrence :

1. Discussed the memo from NECS regarding the i corrective action to prevent recurrence.

2. Discussed.the answer from J. Nystrom.regarding the i lugs in the transfer relay being oversized..

3. Discussed the' answer to investigative action no. 1,. -

regarding Electrical Maintenance and NECS review of  ;

the MFWP power supply scheme design.

b. Reviewed and made comments on the NCR write-up.

NCR to the PSRC by August 31 1992. ,

NCR Closure. ECD 09/30/92 I. Remarks:

None.

J. Attachment (s):

l 92NCRWP/92EMN010.PGD Page 25 of 25

.