ML20029A294
| ML20029A294 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 07/07/1987 |
| From: | Lambert W A COOPER BESSEMER CORP. |
| To: | Paul Prescott NRC |
| Shared Package | |
| ML20029A292 | List: |
| References | |
| FOIA-90-442 NUDOCS 9102110220 | |
| Download: ML20029A294 (119) | |
Text
{{#Wiki_filter:- .- . . . . - - . --- . . - -
' ' )
CooPWRElpSEMt.HhdDihaockTING
@E 9ulyA .7,'<1987
~ ., I Dur Ref: QCC-3534, ;
Mr. Peter Prescott,
.Nuclear Reactor. Engineer 3 , isiI1on Regulatory.com Hc11stop EW3321 '
.Weshington, D.C.DF555/ - !
150bjectfgANP,Pal(1,t,';;U,1J,,,0,1,[s'e1,jener,atori 0 ear .Hr.. .Prescottt1 We understand the
'nine power rod be,t aring" .the NRCd[d for the purpose of inspection remove i to ,
insure its'tetegrity'. -Cooper Dessemar very strongly, recat very ,,
'strongly, recomends i sock is carried out9.egainst'such here"is"always"4 disasser,bly.
grave riskd'o'en such unnecessary f introdt!cing a - problem that does not otherwise-exist. Curre'ntly'there are approximately 124 hours'of running time on.the' - new number nirc rod bearing of which nearly 90 hours have been at - , 100% or greater load. "The crankshaf t is protected from domsge due7to bearing, failura by virtue of temparature detectors .which..from the,"
'rocont effcetively. exporten:e There are of.nothe n'urber two indications:of anymain dis essbearing and ic.
{r.functioi:s very all indichtins " aft ~e'st'to 'a"p' r operly asseabled and running'beafact,\ ring. Therefore va, as the engina manufact.urer, are totally.oppposed to the suc Ae1pested rc:cyal of this bearing Ling 066061y.'th0t3ht.,beadng _ js, running justa'.),Ll.lght.! to confirei what ,the;ehgj,n,e ' sw-We u'nderstofd that if an f6tpection# is 'id-be 'undert'aten' it kill'o'cc~ur this wtik. Therefdre,s will you. kindly _ pass on this recernsndation S
.icinediato,1y_toJh,,o,s,e.,,d,tge,ty concerned with, .the ., current' si,tuation',at.
.ANPP.L ,
Incid'e'nta11f,'-thiiTn~gine"is Tick' running 'Very 'well7 add' sill ~sh6rtly complete a ,twe,ntpfo,st. hour run, prior, to the 35, consecutive, start r , test.) Very"tEul f/h'y yov/s;*.S .$ h f/ k "H. A.~GEert. Manager /
. Quality Control cc: 'J. By w m .. . ... _ _ . . .
'ANPP.PlentHanjer; J11e500391/P2a Uo:otn Avenoe .
Orove c'ty rennWeta 16127 d+'
- Nu USCORPORCT H12l .t$34o00Tetet 63),
iNTEdnIt'k'N NEk$h RESS0b YT64'ORlWN COMpaF.osons'. POWER ENCIN'ES I 9102110220 901219 4 PDR FOIA QUILD90-442 PDR ,
., . . - - .- , . - - - , .~
' ' ' ' ~ ~
1 ^ .gfy*llV)dff/
. , ~ < loll W3uJ,se::lectric Coil r,41 Cmr/iam Lve PO Eur 2nco Cdumtus CH C21 (614)4W-1200 Tva 6874Gt7 MCGEDCOL TWX 810 4321G34 ,e
{ l Mr. Daniel E. Sachs, Mail Station 6086 - ARIZONA NUCLEAR PO'n*ER PROJECT j P. O. Box 52034 i Phoenix, AZ 85072-2034 i novembec 5, 1987 l
SUBJECT:
Preliminary draft of report on Failure Analysis Dear Mr. Sachst Enclosed is an advance copy of my portion of the Analysis Report. John Zeiner is in the process of asse:bling the photographs and lab data for inclusion in the final report. The final report vill be submit ted as soon- as these attachments can be assembled. If you have any suggestions or questions regarding the analysis, it is not too late to incorporate them in the final report. Regards,
& V T. J. Lorenz Division Chief Electrical Engineer Enclosure .
w
,,e-AtMLYSIS OF ROTATING FIELD FAILURE .
FOR , AR120tM NUCLEAR PCHER PROJECT PALO VERDE GENERATING STATION NOVEMBER 5, 1987. REFERENCES PURCFMSE ORDER 33504414 PURCHASE ORDER 33504356 .
,1 l
tMTICtML ELECTRIC C0ll DIVI SI CN - l OF. MAGNETEK, INC H 9 g
., y , y- - -.-
- tM110r+M. ELECTRIC C0!L DIVISIO4 0F t%Ct4ETEK, INC was requested to investigate the f ailure of rotating field coils of the Parsons 0.0 MVA Generators at the Palo Verd2 generating plant of the Arizona Nuclear Power - '* Project.
These field coils f ailed by separation of the outer layers of turns, allowing the conductors to move away from the field coil winding and into the inter polar space.
- Two of the f ailed pole assemblies were shipped to the NATIONAL ELECTRIC C0ll (NEC) laboratory at Columbus, Ohio for this investigation. At this laboratory the field winding was carefully removed from the pole steel and observations made of the coil construction and condition.
The following are the points of investigation pursuedt
- 1. By calculation, 9 stimate the disruptive forces that might cause the field winding to come apart as it did. -
- 11. Measure the strength of the adhesive at various places around the failed specimens.
III. Measure the strength of en specimens of adhesive of the same type as used in the falled coils. IV. Measure the strength of proposed substitute adhesives. V. Make a microscopic examination of the adhesive joints and report on
- a. Mode of adhesive f ailure observed
- b. Quality of adhesive application
- c. Confirm the chemistry of the adhesive.
VI. Draw conclusions as to Ehe probability of f ailure of the remaining field coils of this set. e l l l 1
- J 9 j$
6 e , r
. Q
}
- {
i'
.. CALCULATION OF FORCES D4 THE ADHES19ES The field coils of this machine were wound with .204 x .204 copper conductor that had been insulated with enamel and a wrapping of ,
DACRON / GLASS yarn. The steel pole piece on which this conductor is wrapped i has a cross section d' wide and 40-7/8' lengthwise and a radial winding space between insula tors of 5-1/2'. The pole pieces are bolted to a rim having a diameter of approximately 44-1/2'. The pole to the center is' insulated with layers of rag paper /Hylar sheets glued steel. ;
, i The ileid winding consists of 11 layers, each with up to 25 ' turns. An adhesive is painted on as the turns are wound on ine pole steel.
The eleven layers build to a coil about 2-1/2' thick and the 25 turns fills the 5-1/2"47'. the coil is about between pole insulating collars. The overall length of between J013 and 1028 lbs.The ccabined weight of pole steel and coni !s- . l The operating speed of this generator is 400 rpm; the maximum overspeed ' during testing is to be 750 rpm. FORCES TAKING A COIL SIDE AS A BODY { l Taking in) the coil side as a unit, we have a body of copper (.321 lbs per cu l having dimensions taken to be 2-1/2 x 5-1/2 x 47*. The center of gravity ofthis rotation; thisradius body 11esis at at a radius of 25.54" from the conter of an angle of 9.58 degrees from the pole axis. When rotating at 600 rpn, b+cause of inertia effects each pound of ! l material will have a centrifugal effect of 261 lbs that must be testrained by the combination o.f pressure against the pole head and by the adhesive material. At 750'rpw this force will be 408 lbs for _each pound of coil system. A coil caxial side of these dimensions has a weight of 4.414- lbs 'per-inch of length, or 20725 lbs for the 47' assumed length. The radial force
. per overspeedofofaxial inch length (s. thus 1152 lbs at 600 rpm and 1000 lbs at the 750 rpm. ,
The pole head is at an angle of 9.58 degrees
- to the direction of this force vector, resu' ting in a pressure on the-head of 98.6% of the total f orce. At overspe'ed the pressure on the pole head,is
.986 X 1800 lbs for each on theaxial inch, and insulati'ag for the 2.5
- wide surf ace a unit pressure of 710 psi collar.
The force normal- to the pole axis is proportional to the sin of 9.58 degrees or 14.6% of the total . At 600 rpm then, the glue area 5-1/2* by l' must X wi ths 1152 /5.5 tand a tensile force at the pole insulation interf ace of .166
= 34.8 psi average. At overspeed this is 54 psi.
l l ___._m____.__. . _ _ _ .
-The adhesive choice-and' polo' insulation must be madP to~be able to survive this tensilo. loading of 54 psi. A discussion of adhesivo strengths is-provided later in this analysis. '
Should this glue line f ail, the resulting force vectors willl distort the ' coll _ side. Since one face is_ supported by the pole head, the coil side can
~
only move in- one way, that~ being in ' the manner: of a beam having a ' thickness of 2-1/2. inches, a width of 5-1/2 inchesi and a length of 47 inches. He are neglecting the end reac tion,. so can use formulas for simple be ams . - Calculations show that> the weight ~ per inch = of the 21/2 x 5-1/2 crocs section , under the 400 rpm rotational forces, and resolved to the 9.50 degree angle,is 191.7-lbs per inch. This simple beam would then have a stress.in- the outermost . fiber of 9237 psi. Al though copper does not have a ' distinct yleid point, this stress is considerably above ~ the design objective of 2000:pst used by some engineers. It is the glue attachment to the pole body that mak,e this high loading successful.' Using 3 modulus of- elasticityf of 17 million, one calculates a lif ting of the coil from the pole by_ .10_ inches under normal operating speeds. (This _- assumes'no plastic yielding of the conductors). :At the overspeed these figures are 14,400 psi . extreme fiber , tension, and a' lif t of .14 inches, again assuming elastic. deformations only. In the real world the copper would yield and the coil movement would be greatly in excess of these numbers; the actual values would depend on the metallurgical properties of the copper conductors. One can also calculate -the shearing focce between-layers of conductors, assuming elastic deformation. The greatest shearf force occurs at the ends of -the beam, halfway thre*:gh the-layers. The value of this shear; force is Just under 500 psi at the 600 rpm condi tion. It is essential that the adhesion between = the coil and the pole steel be maintained if coil integrity is to be preserved. S O J l t 1 i I 1 1 1
.--.a...-_-_.....
._ . . ~ . . . . . . . ,. _ .
The ' f ailures expwitnced 'have been by: separa tiont ofithe _ outermost = one ori _two layers _of turns.' Adjacent turns havo remained ' bonded togeth2r- for thoi rnos tf par t . ' Fortheoutermost'.Iayerof' turns,_being;.205*' thick,-5-1/2 wide)(or.lesst , and~ assumed to be 7 4 ' long, we find the-center of-gravity at:a radius of-25.70and at an angle'of"12.32 degrees to the pole axis. At:-the normal? speed of.608 rpm, the radial: force vector i s 263-lbs'per Ib of weight. iWe find the weight of: one axial inch of a the .265 -x 5-1/2 section -to be .36 - Ibs, so the radial force vector iso.36X 263 = 95 lbs per inch axially. . At the. of f set angle of 12.32 ~ degrees,-- the force cceponent normal to the' pole head is 92.6 lbs and the cmponentf at:right angles to that 1s. 26.23: lbs per inch. l - The area of adhesive to resist this 20'.23 lbs;isl1*' x 5-1/2.', so' the _ stress on the adhesive averages 20.23=/ 5.5.= 3.68 psi. , The first two layers taken ._together pulltagainst the adhesive'withiabout - twice that load,-amounting _to about 7_ psi.; 4 Since th'e mode of_ failure-is that of. rupture of the glue in this portions of thw coil, we~ can assume that' ior some reason'lthe local, capability of _. the adhesive to'wi thstand tensile' forces was less. than _7 psi. l Wi thout - the adhesive to hold ' the wires in place,L the conductor by ! tsel f'- is grossiy inadequate- to maintain it5 post tlon. AKING A SINGLE CONDUCTOR-The most- highly- stress adhesive Joint is theLone _ restraining;the. conduc tor- 4 at the 1argest radius. This is the:one'Justaunder the' pole head _'.on;the' outermost layer of the coll . This conductor <has the dimensions-of .205 x .205. Its center of gravityl ! is at = a radius of 28.47-inchest the- f orce .mul tiplier -at 600 rpm is' 290/3'.
- The vector is at an angle ct it.14-degrees f rom the_ pole: axis. This ;
resul ts in a force against' the' pole head: of 2.85 lbs..perjinch axially, and. .1
- a tension on-the glue Joint,off.56 lbs per inch. Tha loading:of thel -I adhesive in tension is then.56 / .205 = 2.75: psi.,
At . werspeed of ' 750 -rpm ; this - stress will be .56% -higher or :4.3. psi .a LThis is not a large stress, providing the; glue system is-intact. 'If ~1tlis not', then depending on~_the size of the deficient: area,s.the strength of the copper conductor must be utilized to maintain coll l integrity.-
~
'[
4
*b- 9 * *P ' '
- b 4 %c , s A+ -
3 g g 4 'u Jd ee g 4, g 3,g y g o,,- , -, . . . , . , - - , - e, .~,,--y , ,., ,g,, , .g',.. g ,m.. ....,y., 4y v.,h r 4 ,, ,, , , ,
STREtGTH OF ADHESIVE:0F' FAILED COILS The windings were reinoved from t,he pole- assemblies by cutting transversely-across the end of. the coil so as to allow removal of layers intact as much - l as possible. Because cd delamination in service it'was nott always possible l to obtain intact layers. - To determine the inherent strength of the adhesive, specimens were cut-from the layers and sawed to generate double shearstest specimens. See the Xerox imprint of a typical specimen. Eachief these specimens was pulled , to destruetion in a tensile testing machine '.-
- R For the flest pole assembly t eceived the tensile ~ tests' resul ts weret. .
-1 Pounds tension : Shear psi- '
206 502
+
209 510-203 495-233 568 172 420. 246 "600
-186- _454 ,
' 142 -346 l 156- -380 l 41 100 l 44 107 :( Average 167.1 -408 < Not included in the above were specimens- that: came apart 'during specimen _, preparation. '- For the second pole _assemb1y received six specimens were prepared fromL each of the available layers -(9)l the' tensile-test resus-were Pounds tension _. Shear- (psi): i
' Layer #1 (nearest 69 148 Pole 000 .
----. : specimen. broke in preparation 1
Dody)
~
30 .73 , 132 _322
- 161 393-96 234' Layer #2 87 212 119 290' --
146 356- , 45 110 222 - SA l- s 208 507
. . . ~ . - .---. --. . . , - . . . -. . . . . .. . . - . ~ . . - - . . - .-
~
/4 Layer!N3 .
321. 783 D
-59' -1 A4 43- '
1992
-i-42' .,192 I 22- 54 141 - 344:
Layer N4 104 254 314 7662 ! 8
-28 I- < j '
80: '195: 103- ' 2" S . 46 1 l'. W Layer.h5-66 -144 118 268 ! 88 215
- 56 137 81 198 .
157 393- . s Layer N6 171 417 < 213- 520 , 248 605 , le 24 X -) 33 88 e6 -- l Layer N7 162- -395 93 ?227 162 395
'29 71 :-
=63. 154
- 42' .10 2 .,
Layer N8 97L 237 64 156 - 4 46- ~ 112. l - 41 .108 200- 244-
-84 . -205 - o e Layer N9 18 44 000 -- >
[
=71 173
- 22. 54.
- 59 14A-
.-- 200 i Layer N10. _. ..This layer removed by others prior to receipt'by NEC.- Layer N11 - And this, Iayer also.
. As with the first coil, .several specimens cama apart during specim2n preparation. The lowest readin0 obtaintd by test was-20 psi shear strength. Tho largest was 783 psi. Note the general decrease in strength for the outermost layers. ,
Ill STRENGTH OF NEW ADHES!VE SPECIMENS STERt.ING D-111A , SHEAR TEST Pounds Tension Shear (psi) at failure of based on two surfaces pull test l' x .205'
'159 388 188 459 110 268 97 237 l 236 527 210 512 96 234 185 451 274 648 204 498 Average 174 pounds 424 psi shear CROSSED HIRE TENSION TEST -
An additional test was perfe ...;J using crossed wire pairs. Pounds pull at f ailure - 18 15 14 25 28 18 22 21 20 24 , Average 19.7 pounds, equivalent to 468 psi tensilei strength based on .205 x .205 glue area i e e w
_ ; 7--
.;>,. u IVn STRDiGTH. OF C440!DATE RES!NS:
It has been proposed'that Armstrong A-701 be used in place of the Sterling; 0111A. Specimens using the A-701-were evaluatedLat -thel same _ time 'and in the.same manner as-.the_ Sterling-specimens A701= Shear; Test
- 4
- l Pounds Tension L*
;,. v ff; 573 , .
, y 436 +
-487 ,
'594 688 ggp; ,
-i 575'
=d57-588 d45.
Fuerage 633 pound-equals 1544 psiishear strengt'h. This is three times 45 strong.' no the ' Sterling D111A. TENSILE -TEST A7011and-~ 300-20 '
.Crossed p m testsLwere Sterling 300-20 resin also performed on the Armstrong A-701 and':en the Armstrong A701 : Sterling 300-20 91 118 -
74 '81. 59 80 90 :96 45 80 68 105, 41 94- , ' -76 103 90- 93 Avgt 74.7 lbs' 96 lbs.- Tensile strength based on area of.. 205 x . 205 t 1778 psi .2284 psi l
'l s 1 h
- )
, A,
Frcen these tests we conclude tha t - the Armstrong A-701 is about .4 times as ( strong as the Dl!!A system presently be'ng used, and that the new Sterling i 300-20 would be about 5 times as' strong as tha present' resin. . l However, i t will be shown in the sumary that the presently used D-lilA has adequate strength if properly applied. 1 I V - MICROSCOPlc EXAMINATION , The report of Lyle Brostrom of Ashland Chemical is attached. Mr. Brostrom reports extensive adhesion f ailure at the resin / glass interf ace. He sees li ttle evidence of f ailure of the resin body. He says " sigalficant strength was lost because of dry joints *. i There appeared to be less resin in those sections that had dalaminated than in those sections that had not delatninated.. An infrared spectrophotometer was used on samples of resin, one from the delamination portion, the' other from the coil part not delaminated. These. two curves show similar chemistry of - the two samples. e f i
lI t- 93 ,
', +-
stktiARY . .
\
- 1. The -adhesive, used has adeq0 ate. strength to wi thstand thelorce thit1
- will be encountered, both;in normalE service and during testing..
Location.: A.t pole body:interf ace Tensions- 54 lbs alcoverseed
... .;l
.Inside outer layer- 3.6-lbs tension?
Inside'second layer - 7L .lbs tension' ' Top most turn 3- -Ilbs'tensioni ,
,; r Aver. tensile strength-'of: Sterling D111A resin,t neut; :460 psli tebsilk.
Minimum shear strength = of Sterling -D11.!A resin, news 234 psi: shear < - Minimum shear strength of undamaged specimens: 20 psi I CONCLUSIONS.
.The resin system is adequate. >
~
f '- ( The problem appears to'be a c'ombination of-insu Kicient resir Llo-fill:the 3
. voids and poor' wet-out of the, resin / glass interface. Both'of these;are:
-production. variables. .
There -is no easy way to inspect ccenpleted coilsL to-find those su~bjec tito this failure mode. ' i s REFERENCES
' ' Formulas for Stress 'and Strain?, Roark and Young,LHcGraw Hill ,-Inc.
' Mechanics of Haterials', J.L. Robinson, John Wiley sand Ton's '
'The Science of. Adhesive Joints', U.J.Bikerman,JAcademid Press a NEC Specifications: 12A92 ? Procedure for ' determining *!/'-Block.
Requirements-on Sallent Pole rtotors" FORMULAE Centrifugal force / weight = 28.-4E-6 X radius- X '(rpm {^2 . . , Force component normal'to pole axise-Centrifuge.1 force XJsin(force sngle) - i Extreme fiber stress = force / inch X (length)^2 :/(bXh^2): X .75 Lift-off , = 1.428E-6. X !(rpm) ^2 X ' sin (f orc e angl e) .X !^4 / h^2 X radius / Modulus of elasticity l l ( , , , - _ . ~ - - - - ~a~~ ,- M-~ -~ ' ~ '
___ , . - . _ . . __. ~ _ _ ._._. _ -_ _ .. . _ r; n ... . r
..g-PROBLEM EVALUATION PTL NO.
( .p ,wdomo,,,vmcro,g gg h.) SUPPLY SYSTEM REQUEST , ' 3 - ORicm AloR _, . . . .. 6 m- . , f 4 o d L E m a' ' f?.ifortTLo ' c rJ t.J M,# 2. ~d#65 E L PLANT PROBLEM 0ESCRIPTCH A STO C V C o U E /t 't kl 6 76 J ^ 5 f i 'l f** 't '"' o 'O E 'l^ "'^ rt'. fedcarA ric G idillATo tt. 0 H *YS ( !V l '1 E *\ A t 5 0 c e ATt o a r 4 T 86 si Lt. r es,s /TocEW #dtt- Root C.A0 5 E ' l$5vl5 of THC Roo ' c A vs a _ Av/ALg s e.t AMAL9 Sit; 'r ei E F,t Ec re // a f s L Pbwm A w o ML A s rf' TtWr Mh3 h'Ms is . c ndrJ C Adut. FA c ~rtn. e ot o.rn r o e. c s,J u n. rte. c.a ttita rov t Atrica m a c nT . S isit.A u. n st e.s s' v r tet e
- t. 4 m v s.md. ,
O f i mJ o e c. A *14 4 . O f C D M M f^'4h' d ^J 5 ' (I'd = CCCNUATIONSHEETS 8' YES O No ORIG:NAJOR NAME (PPai) .
- 58. SPECSROCEDURE.REGULATORETC. W160fR MA.
O NO ORIGINA ORSGMA Ri.0 ATE - MEL EPH CONTINU ATCN SHEET G YES f -
'e t I li TEEATTAtticd PA GF$
MEL MANUFACTURER CODE ' MEL SYSTIM NO.
- L/A fl.100.5 CUDA,TC'l4 jg Qg S'GNATURE.S g ATE A/
e 9. 2 s U 1. 6 . f? 8' _ ' SHIFT MANAGER EVENT DATEniME
" #6 . 8 8 %F0WE3 C[.Ch^. i b%30 VArq PCCMADISPOSMON ' O YES F NON RE?OkTABLE REQUIREMENT 10CFR
. AFFROVAL E!OUEST ^ 7 hO
- O PQiENTIALLY REFORTABLE OTHER O REPORTABLE o
TECH. SPEC.VOLATCN O YES I NO YlTALMWR O YES -- YNO SH:FTlAWGEM) ATE - i LCO EtaEREO O YES Y NO VUL WRNO. *' -- -- j T S NO. _ . . , I
- 4
+ MANAGEMENT REVIEW COleMITEE , ,
j
%% cy' hzW u;ik[. m,,y ggfu Gvb& b6_y s,nj &g 9!
. ggs #3 -
COMMENTS 7 Ecy,. !/ ,a3 pg.g procwg 3 f- f y Ma~&( c'#- P "tr 9 o . 2mf y* ~ h to$ Snc$ui AquwlrMf c,, t., w $1PTLs
~
m $/.id Yre->45$Ntt w ,#62' /etelut '*)' PROELEM RESOLLTTION METHOO "p Luo%f** Ti svoladtfo, 46</av.e'/ 7f
~
ltlalp, O PDF G ROR O NONEREQUlF.EDCLOSEPER gC ( O NCR O uwR y
-O ISCR 6 b-g.
O TER O PRF 39 PTLENTRY (Lus "y1M K CD POR O RFTS O CM O --SCN - 0 OUjER(UST)
-l 1-4 r 9 MDR
/75 ACTICN 00E DATE -
I h 7
/
RESPONSIBLE DE?ARTMENTiORGANIZATCH -_ , ' RXSU REVIEWREQUG.50 O YES JD NO: / gf. INDUSTRIAL SAFETY /BISK ASSESSMEFfr REQUIRED C YES W NO IMPLEMENTING DCC.60.On PTL NO. - zi 24ho 3 PtAffT)Me@'SI'Y TUFeUATE PER CLOSURE DATE l CL OUT REVIEW SIGNATURE.0/JE e , _,
-M'
'. s .
. e
. . 1989 DIESEL GENERATOR PROBLEMS
* - ~
l , l .% \ j , I. E'f30PSIS l l This study was undertaken in an attempt to determir.e if there d i were any lessons to be learned or coromon elements in the I reported diesel generator problems that could be discerned by 1 looking at the problems from the perspective of root cause determination. ' OEA assessment 89005 (OER 89064A) of December 4, iso 9;. Standby AC Diesel Generator Maintenance and Testing lookedt a-t many of the same problems along with others from an earlier time f rame. That study was performed to determine if there were recurring generic problems experienced with certain types of compenents such as relays, switches, valves, etc. Several recommendations uere made which were directed towards correction of noted- specific problems. The purpose of this assessment is to evaluate programmatic issues associated with the selection process for root cause- . analysis, effectiveness of the root cause analysis program, - and areas that may have a common causal factor that might be identified by viewing the problems and causal factors from a macroscopic perspective. The first step in the process was to break the problems into groups that could be easily defined and , more or less, to separate the more serious problems f rom the less serious problems. Areas checked were
- c. . engines, generators, instruments and controls, fuel systems,
*n air systems and HVAC systems and the problems.were grouped as follows: .
TOTAL BCAS OEA o Failure to start or run 15 7 2 o INAC problems 9 0 o Fuel oil / fuel oil chemistry 14 1 o Air syst'em problems 8 3 o Software problems 24 7 documentation problems o Other hardware problems 16 3 1 TOTALS 86 21 3 From the above information it was determined that between 8 and 9% of all plant prob 2 ems reported in 1989 were associated with the 3 DGs or their support systems. Of the reported 89 problems associated with the diesel generators 21 were assigned to be analyzed for root cause which is about 2% below the average for all plant problems (25%). Of the 21 assigned problems , 13 .had been analyzed for root cause as -of 1 March 1990. Root causes were reported as follows:
- Installation Deficiency - not installed to design 1 Design Deficiency - problem not anticipated 3
,, Design Deficiency - Specification LTA 2 Design Deficiency - Assumed Risk 1 Nanagement Programs LTA - C. A. not implemented 1 Management Programs LTA - No plan 1 g - -
. - 1989 DIESEL GEf3ERATOR PROBLEMS _ .
5 . Management Programs LTA - Supervisory O/S LTA 1 orocedure Deficiency - Didn't cover situation 2 Andeterminate 2 i Incomplete 1
- Scme problems have more than one root cause k
- 11. OVERALL CONCLUSIONS
.g..
1 .- The three WSP-2 diesel generator units are problem intensive uhen compared to the rest of the plant equipment and processes.
- 2. Other than the four identified groups of problems .
that might call for further analysis, no l sisnificant trends in the problemi could be discerned.
- 3. During review of the PERs that were closed without benefit of a root cause analysis they we r t.
scrutinized for any evidence of a PER that should have been elevated to the PDR/MDR/NCR status so that a root cause would have been conducted. No specific cases were ' c.J t1.81 JuM smisen t .w
..g the method of assigning apecific cases for action N. or analyses is faulty. Hceever, it was noted that in some cases it would appear that when thc PER arfived at the Management Review Committee Meeting it was sponsored by an individual armed with a good explanation of the problem, causes, and corrective actions which resulted in direction tc ' carry on'.
In retrospect some of the corrective actions and causal factors may have been lost from the record as a result of this type of actwn. Followc is a list of several examples:- l a. PER 289-002 reports the loss of a sample bomb i into the HPCS DG fuel storage tank. The PER directs an evaluation of the sampling techniques which was probably done but the results are not documented and any fallout corrective action items are not tracked,
- b. PER 289-094 reoorts a failed damper drive motor with no further explanation as to cause or corrective action.
- c. PER 289-160 documents a case - of failure to implement corrective action from a NRC Notice. The PER provides no_ evidence that the work was scheduled or completed,
- d. PER 289-303 reports a restricted fuel oil fill line to DO-TK-1B. The PER directs that an 4 N^ O
. 1989 DIEcEL GENERATCR PROBLEMS ?
Y inspection of the line be docim,ented. Marked on the PER is the statement
- 5/8 - NDE probe shows no blockage - line clear' This is a case where
. failure to determine root cause may provide 4
embarressment in the future should the problem recur at some inopportune time.
. c. PER ** 209-446 rep;rts a breaker trip end 86
. lockout of the HPCS DG. The PER directs (1) trainino for operators, (2) review of ^he-procedu.es, (3) operator counseling, and contains the PTL numbers of the corrective actions. In retrospect, a root cause analysis may have better documented the problem, .
Its causes, and the cornctive actions, particularly if the root cause was not accurately identified and the event repeats itself. '.
- f. FER 289-530 reports the incorrect landing of a test jumper by a test engineer during surveillance testing. The only statement of explanation on the PER is 'the test was rerun successfully see PER 25)-0530'. While it was probably obvious at the
- t. :. a ' . .v. L'.a ... o b h .a .,e, . I ,' ttit o;.Act , the
,p problem, 4.s causes_, and recommended' corrective vW . actions are probably lost forever. *-
+
I. SUSDMRY OF RECOSNENDATIONS
- 1. To reduce the rate of recurrence of the problems the overdue root cause analyses should be completed and, in the future, a larger proportion of the reported problems should be subjected to root cause analysis.
This is cased on two premises:
- a. In comparison to other plant equipment, the number of analyzed. problems on the safety related diesel generators should be at least as high as those analyzed on the rest of the plant (25% in 1989),
- b. Because of the number of problems associated with the diesel sencrators, a higher level of ef fort should be expended in trying to analyze the problems to prevent recurrence.
- 2. The five cases of mis-positioned valves and dampers
,. (94, 223, 294, 633, 891) should be subjected to a
.f root cause analysis as a . group to determine i f-there is a common cause associated with these events.
.~-
. 1989 DIESEL GENERATOR PROBLDtS
. l
. i 1
Y 3. The four cases of inability of the three , diesel oil transfer pumps to meet AS'tE Pump and Valve testing requirements (197, 523, 753, 906) should be subjected to a root cause analysis to ' determine if there is a common cause associated with these events.
- 4. Management should consider, in some cases, g.. .
performing a root cause analysis 'just for the
. record'.
- 5. There u'ere three events (348, 407, 483) reported of a standby diesel generator failure to come up to speed in the required 10 seconds during testing.
At the time of this report, the one case essigned for analysis had not yet been completed and the ' other two were not assigned for analysis. This is . en area u'here en analysis of all three events under - one cover might prove to be beneficial. IV. APPENDIX A. List of all Problem BV6106 tion Requests consicered in this report, g w s O-lC ?. s;f . l
- a. .. e e.,- ~..-,.--v ,-.y,,-.,- ..-- . , .- ---
', 1989 DIESEL GENERATOR PROBLEMS w '
/ V. .DESCRIPT1QN QF THE ILVfET_S JLY GRCUP
. DIESEL GENERATOR FAILS TO START OR RUN Follows is a listing of problems grouped under this category for ch/ch a formal root cause was performed (or is still open unitingsto be performed). No inference to 'reportability' or
' f ailure ' s.hould be taken by the fact that an event is included in this category. Events were grouped for convenience only. No attempt was made to perform en evaluation of events uhere-a root cause analyt.is had not been made .
NCR 289-0322 HPCS DG RCA by: TECH - DATE OF EVENT: 5/10/89 This event is not a diesel generator problem as such and is not counted in the total. This problem has to do with a design problem of degraded voltage and undervoltage relay settings which also effects the diesel generators. c, RCOT CAUSE(S): NONL a & . . FOR 289-0356 HPCS DG RCA BY: OEA , DATE OF EVENT: 5/13/89 This event occurred during a special test of the HPCS diesel generator that was testing the ability of the unit to start cn one air header (50% start capacity). The unit failed to start. This test was performed af ter modifications to the air start headers which separated the previously cross connected headers to provide a truly redundant starting air system. Making the problem very dif ficult to diagnose was the fact that three separate problems lead to the failures to start. First was the servo booster header select shuttle valve (DSA-V-80) which was stuck in the left header select position. This prevented governor boost when trying to start on the right air heeder. It should be noted that until the air headers wers isolated from each other there was never a reeson for the shuttle valve to operate and its existence was unnecessary for a normal line up start. Secondly.. the starting air pressure control valve (DSA-PCV-1C) on the lef t heeder was not allowing the required vo'.ume of air to enter the admission valve which prevented a start when attempting a start on the left header (it should be noted that this condition would probably not have been noticeable during a
., normal two header start). Thirdly, the air inlet port to the- ",j'. servo booster was about 90% clogged and the inlet oil check-valve to the servo booster was not closing which resulted in sluggish or ineffective operation of the servo booster. All of these problems were the result of rust / dirt in the Ot C
_ , _ . . . - _ -- ., ~, _. , . _ . -
i '. *.. 1989 DIESEL GENERATOR PROBLEMS .
,h) starting air system except for the oil inlet check valve problem. ,
ROOT CAUSE(S): 1
- 1. Equipment - Design - Specification LTA The euto drain trap specified for the DSA air dryers was not a good choice for this application as it uas not
.cbemically compatible with the process and did not operate es desired. This problem is nce resolved.
- 2. Personnel - Management programs LTA - Cort ective action not implemented. In view of the many -documented cases indicating that operation with water in the diesel 4 generator air starting systems results in starting fa!1ures, the one year it took to resolve the deficient trap .'roblem and the failure to inspect and cle:.n the system 'fter a known wa t e r ingestion is all indicative '
of failure to heed precursors and take timely corrective action.
- 3. Personnel - Management programs LTA - No planning.
Voiding of the Priority 2 MTR stating that the drain was clogged in hopes that the problem would be solved at . some future date by the installation of a modification
,]) .
indicates a lack of understanding of the importance ,of i the components and a lack of sensitivity to plant conditio'ns.
- MDR 289-0379 DG2 RCA BY: OEA DATE OF EVENT: 5/19/89 The preliminary investigation conducted on the evening of the event concluded that the trip was the result of an improperly latched overspeed trip mechanism. This conclusion was reached more by a process of elimination than by direct -
evidence. There is a possibility that during the work on top of the engines that the latch was some hos disturbed and that it may have been latched in a ' semi-unlatched' condition or more likely it was unletched and because of some mechanical interference encountered from one of the new heads failed to travel to the full- trip position until it vibrated loose during operation. Follow-up investigation of all available information was inconclusive. There is no evidence to indicate that the ' semi-latched' mechanism was the cause and there is no evidence to indicate that it was not the cause. Follow-up testing of the generator set has. not l repeated the event and inspection of the overspeed device on
.WR AV-1971 disclosed no reported discrepancies. ;h -;)
ROOT CAUSE(S): Indeterminate - assumed to be a latching problem with the overspeed trip mechanism as a result of j major maintenanc_e conducted on top of the engines. 1
. 1989 DIESEL GENERATOR PROBLEMS MDR 289-0384 DG2 RCA BY: TECH
'DATE OF EVENT: 6/20/89 During implementation of PPM 7.4.8.1.12.5 the DG2 start times uere greater than (the required) 10 seconds.
This is the only information availabic on this incident. No root cause analysis has been performed to date. MDR 28,9sO463 HPCS DG RCA BY: GEN. ENG. DATE OF EVENT: 6/14/$9 This event occurred during idle speed testing of the HFCS DG when it was noted that the voltage regulator overexcited and drove the generator voltage to levels in excess of 5000 volts. The problem was determined to be caused by the failure of the voltage regulator to disconnect uhile the - engine was at idic speed. . RCOT CAUSE(S): Design deficiency - problem uts not anticipated. MDR 289-0487 DG1 RCA BY: OEA DATE OF EVENT: 7/17/S9 4 DG1 tripped on high temperature as a result of the disk Jf coming adrif t f rom the shaf t in SW-V-214 and 'f ailing to opeo. RCOT CAUSE(S): Design deficiency - failed to anticipate problem. As a result of this event the valves in question (the Standby Service Water inlet valve to the engine cooling water heat exchangers) cere eliminated from the system which removed many feilure modes including the one reported. MDR 298-0781 _HPCS DG RCA BY: TECH D TE ,OF EVENT: 5/22/89
/ /.?'?
/
On while performing T.S.S. 7.4.1.1.2.12 the HPCS diesel generator load appeared to drift during a 5 minute period while operating in parallel with the utility. Pccer slowly increased from a steady state load of 2700 KW to 3440 KW where it remained for about five minutes until it- ca s reduced back to 2700 KW by operator action follouing normal operating methods. As no other indications of fault were noticed the surveillance testing ues completed. The system . engineer was called in and a detailed inspection was rtade of ! the unit, No problems could be found and the unit uns l restarted and egain tested. On 2/4/90, while performing T.S.S. 7.4.8.1.1.2.12, the unit again assumed load eithout i operator action, this time as the set was being placed on the l
.,. line in parallel. The load increased rapidly to 3415 KW and l y then, probably as the result of the operator trying to regain i control of the unit by lowering speed (load),.the unit i unloaded and tripped out .on reverse power. A detailed trouble shooting plan was prepared and implemented by NR AS- I y --
'vr- -= e- y -4w-m-- ,-w -
g- e- %ww9vyy- ur- -
-,*f gg ny- "
.,.t 1989 DIESEL GENERATOR PROBLEMS N
i.J 4403 to check out the electronic circuitry associated with speed and load control. The droop-iso switch was high on the list of suspected parts from the first incident and again was suspect from the symptoms noted. When the trouble shooting guide looked at the droop-iso switch it ur.s determined that the switch was not making good contact and was determined to be the problem. I ! RQOT' C AUSE ( S ) : Indeterminate - without disassembly of the switch it is not possible to make a failure analysis. The switch in question was jumped out cf the circuit until a design change can be made to replace the switch with a-new cni. Failure analysis is planned when the old switch is made an ilabl e . CF. 289-0752 DG2 RCA BY: TECH DATE OF EVENT: 9/25/89 ) l During surveillance testing and uhile paralleled to the grid uith the load at 1100 SW lt was noted that the VARs were pegged dounscale. The operator attempted to rectify the prcblem by raising voltage on the unit with no results. The unit was tripped of f the line and again brought on line with a little higher incoming voltage. All conditions were noted to be proper and the S.T. was completed without further ih prcblems. ', ROOT CAU$E(S): The root cause has not yet been approded, however, it would appear to be a design error that allows ths voltase regulator to saturate and become' unresponsive. OTHER PROBLEMS IN THIS CATEGORY FEF. 289-0407 - repeat event of MDR 289 0348 included in that evaluation P E.:. 28 9 - 0 4 4 6 - HPCS DG tripped on reverse power - PTL item PE:. 289-0468 - DG1 ' start times not verified during test. -no action required. PEF. 289-0483 - DG1 slow to come up to speed, same as 289-0348 MTR action. P Er. 289-0533 - HPCS DG tripped during testing due to a jumper that was installed for the test. - no action required. PER 289-0910 - DG1 had high vibratlon alarms during testing. No action required.
.,g.:;
s
-__m.______-.__:m
. 1989 DZESEL GENERATOR PROBLEMS l
. \
)NS :
i wer2 three events (348, 407, 483) reported of 4 2re to come up to speed in the required time for the
!by diesel generators. At the time of the report the
- ase assigned for analysis had not yet been completed 1
- he other two uere not assigned for analysis. This l 1 area where an analysis of all three events under
- over might prove to be beneficial. i 7 .
$k!$b 9
1 [
3
- i
- 1989 DIESEL GENERATOR PROBLEMS .
3 INAC PROBLEMS . There cere no INAC prob 1 cms that were subjected to Roct,- Cause Analysis in 1989. Follows i sa list of all reported FWAC
- problems: .,
1 FlB EO.:. EPl P_B91!L.EJ IQ ! 0094- D>tA-AD-51" DA>lPER FOUND OPEN brdR 0293. %- DEA-PS-32 NUTS MISSING FROM COVER hf4R 0294 DMA-DPS-31 INST FOUND VALVED OUT . hfaR 0459 DMA-DPS-31 WIRING ERROR htAR 0469 DEA RMS DRAWING ERROR HFTS 0633 DEA-FN-51 DAMPER'FOUND OPEN OTHER 0815 HVAC NO TEST hMDE OF }WAC PTL 0869 HVAC HICH TEMPS FROM ASHFALL TER 0891 fWAC DAMPER FOUND CLOSED Ff4R The four cases (94, 294, 0633. 0891) of mis-positioned valves
- or dampers may present an area that should be evaluated as a group t o determine if there is a problem in this area.
. l 9 .
9 6 e J 4 i e A
1989 DIESEL GENERATOR PROBLEMS FUEL OIL / FUEL OIL CHEMISTRY PROBLEMS There was only one PER on the subject of fuel oil or fuci oil chemistry that ues subjected to a root cause analysis in 1969.- MDR 289-0091 FUEL OIL TANKS RCA BY: TECH I EV'dNT DATE: 3/6/89 Bacte[i and funsi found in the fuel oil storage tanks as the J result of testins. This MDR is still under investigation and i no root cause or recommended corrective actions have been ! made to date. RCOT CAUSE(S): IN PROCESS _ OTHER PROBLEMS IN THIS CATEGORY . PER NO. EPN PROBLEJJ B 1 0002 DO-TK-2 . FUEL SPECIFICATIONS OTHER 0197 DO-P-2 FLOW OUT OF RANGE NOME 0303 DO-TK-1B FILL LINE RESTRICTED OTHER 0305 DO-TK-1B WATER IN FUEL PTL 0358 FUEL DELIVERY DID NOT MEET SPEC NONE . g ! 0523 DISCHARGE PRESSURE OUT OF RANGE MWR 9 0537 DO-P - 1 B FUEL NIGHT ORDERS.NOT FOLLOWED .OTHER 0604 FUEL TEST RESULTS NOT RECEIVED ON TIME NONE' 0753 DO-P-1B FLOW OUT OF RANGE OTHER j 0820 FUEL FAILED FILT CLEAN TEST OTHER 0906 DO-P-1A D/P AND FLOW OUT OF RANGE OTHER 0957 FUEL CONTRACTOR PROBLEMS NONE 0968 FUEL TEST IMPROPERLY REMOVED FROM FSAR PDF CONCLUSIONS
- 1. There were four problems (197, 523, 753, 906) reported in
, the three fuel oil transfer pumps meeting the ASME' Pump and Valve testing requirements .in 1989. This is-en area where a ccabined analysis might point out. generic ;
problemd with either the pumps, the test methods, or the j test criteria. ! O h 4 ROD $
l
. 1989 DIESEL GENERATOR PROBLEMS $ # AIR SYSTEM PROBLEMS l l
bOR 289-041 DSA VALVES RCA BY: TECH 1 EVENT DATE: 1/28/89 l This event determined that several valves in the DSA system for the two standby diesel generators had been installed backwards f rom' the* drawing requirements. This condition was determ4ned to not have en ef fect on operabi)1ty of the diesel generators. ROOT CAUSE(S): Manufacturing _. not installed ' according to plan. PDR 289-056 DSA-V-4A RCA BY: OPS EVENT DATE: 1/29/89 , During Surveillance Testing the subject valve was found in the open position as opposed to the required closed position. The valve in question is the air receiver isolation valve for the spare receiver and (at the time) was intenced to maintain the sparc receiver in a backup condition in the' cese of a problem with the on line receiver, m ROOT CAUSE(S): Procedure LTA - did not cover situation W and Personnel error-distracted. % MDR 289-918 DSA-M-6B2/1 RCA BY: TECH EVENT DATE: 11/27/89 During routine overhaul of the subject air motor it uns found that the planet sear shaf t had sheared off near_ the spline of the shaft. The motor had been inoperabic for some undetermined period of time. This air motor was-installed on DG-ENG- 1 B 2 . The root cause was determined to be indeterminate as there was not enough evidence to indicate why the shaf t had broken, although it is assumed to be-a some form of problem with the turbine, gear train or Bendix sear which made the starter drag enough so that the engine imparted torque on the starter shaf t. ROOT CAUSE(S): Indeterminate i OTHER PROBLEMS IN'THIS CATEGORY PER NO. EPN PROBLEM g
.,, 0187 DSA-RV-3 WRONG SET POINT RFTS c.p 0223 DSA-V-2B VALVE FOUND CLOSED PTL 0573 AIR COMP AIR COMPRESSOR DRIVER WOULD NOT STOP' MWR 0591 AIR . DEWPOINT OUT OF SPEC HIGH .\fm 0799- RECEIVER AIR RECEIVERS NOT GRCONDED TER l
% ,n
- n. , , , - -,- , - - - , , ,
1989 DIESEL GENERATOR PROBLEMS
?Q ty -
CONCLUSIONS i
- 1. No indication of common problems could be discerned uhen 1 coking at the air systems. Prob 1cm 223 might be :
included in the proposed analysis of mis-positioned valves and dampors discussed under HVAC problems. . 4 e 4 0 I e 4 4 l l I h
.*,g l
*)
. e,'
198'9 DIESEL GENERATOR PROBLEMS SOFTWARE AND DDCUMENTATION FROBLEMS TECH SPEC ERRORS RCA BY: TECH 362 OF EVENT: 3/23/89 nt is concerned with a documentation problem where nical Specifications were in error. This item is analysis has not yet been en and a" root cause
'd.
- CAUSE(S): IN PROCESS .
TECH SFEC ERRORS RCA BY: TECH ,, OCA OF EVENT: 3/23/89 rnt is identical to the one covered on NCR 289-062 ', h[f0 id will be addressed in that document. g '; fc T CAUSE(S): IN PROCESS
-260 WRONG SWITCH CONFIGURATION RCA BY: QA E OF EVENT: 8/26/S9 is still open and no root cause has been .
$)
ent el ted to date,
>T CAUSE(S): IN PROCESS SECURITY VIOLATION RCA BY: ADM 1-318 TE OF EVENT: 6/29/89 overhaul of the diesel generator sets the door leading out side of the DG building was lef t propped open which 3 access to the vital fuel storage area via the HPCS DG Af ter some consideration this was determined not to be '
Spec violaticn so no root cause analysis was performed s OT CAUSE(S): NONE 0G1,2,3 RCA BY: LIC 9-401 TE OF EVENT: 5/26/89 analysis it was determined that Building 56, as a ary building, was not considered in the design basis at if the building collapsed as the result of a wind , the debris could clog the air intake for all three generators. This event is still open and no root has been formulated to date. OT C AUSE( S) : IN PROCESS . RCA BY: GEN,
;9-417 DO/K55 sTE OF EVENT: 6/7/89 i
1989 DIESEL. GENERATOR PROBLEMS The relay setting was changed by Engineering Direction but '
'h the' drawings were not changed.
RCOT CAUSE(S): Management Programs LTA - Supervisory
., . Oversight LTA PDR 289-648 . INJECTOR OVERHAUL RCA SY: QA DATE OF EVENT:* 9/18/89 The oderhaul of used fuel injectors as performed by a contractor was not performed under the . required QA surveillance because the specification did not specify the QC "
level of the injectors. The root cause was determined to be Personnel Error-Procedure not followed (this is a level B step and not a root cause) so no root cause was assigned and (by definition) no valid corrective action OSists. RCOT C AUSE( S ) : IN PROCESS . OTHER PROBLEMS IN THIS CATEGORY PER NO. EPN PROBLEM TO 0104 TEST TEST NOT VALID FRF . 0121 CONTACTOR FAILED SEISMIC QUALIFICATIONS FTL fQ 0220 52N/D01 , RELAY DRAWING ERROR :. F.FTS O 0275 DO-V-56 VALVE NEEDS TO BE RECEIPT INSPECTED ?ONE 0279 HPCS-DG STARTUP TESTING INADEQUATE FOC' 0296 HPCS-DSA NO RELIEF VALVES INSTALLED IN SYS TER 0337 DG2 BDC WIRING ERROR M'E 0375 DG1,2 NEED JUMPER TO PERFORM S.T. NONE 0406 DG1 18 NONTH S.T. MISSED NONE 0440 2"D.O. LINE WAS CUT WITHOUT DOCUMENTATION MIR 0450 DLO-HX-2A BOLTS ARE 1/2' TOO SHORT MIR 0465 DG 1/7 DC WIRING ERROR MIR 0503 HPCS DG JUMPER INCORRECTLY LANDED NONE - 0759 M512 DRAWING ERRORS F.FTS 0873 DG2/K10 RELAY. DRAWINGS CONFLICT MAR 0884 DGl.2 NO DATA ON CABLE RUN TEMPERATURES FTL 0956 RLY SCR WRONG QC CLASS OTHER CONCLUSIONS This is the largest group of problems in the collection (24 of 86) eith seven of those problems considered significant enough to demand a root cause analysis. As of 1 March,1990 only one of the seven had been properly analyzed and corrective action recommendations made. Completion of the remaining sin analyGes would be required to- make a determination if there was a common cause associated with
- r. . these problems. By definition, unless' the events are 6 analyzed and corrective action - taken, there - is. an increased probability of repeat events. ,
l paco 15
ll *
. . 1989 DIESEL GENERATOR PROBLEMS Dd HARDWARE PROBLEMS NOT OTHERWIbi! GROUPED MDR 2S 9-299 MISC. RELAYS RCA BY: TECH EVENT DATE: 5/25/89 This hDR reported sever 61 relays in the plant, most associated with the diesel generator ciectrical output secticn, in uhich there were noted some metal parts not noted in other. e relays of the same make and model.
ROOT CAUSE(S): Procedure LTA - Did not cover the situation. ,, hDR 2E9-402 CYLINDER LINER LEAR RCA SY: OEA EVENT DATE: S/23/89 During testing following overhaul of DG2 it was noted that - uster was leaking f rom one of the power packs on DG-ENG-1B1
- uhich was initially assumed to be coming from the head. The powerpack removed f rom cylinder No. 19 of DG-ENG-1B1 had a crack at the upper joint of the liner to water jacket weld.
This was determined by the vendor (CEECO) who had assembled the powerpack (using a new liner) to be caused by a ' bad ueld'. The root cause in this case was indeterminate as it ^ was a problem in the manuf acturers process. The liner was 't eM returned to the vendor who replaced it without charse, RDOT CAUSE(S): Indeterminate hDR 289-628 DG-ENG-1B1 RCA BY: TECH EVENT DATE: 7/24/89 Durir.g surveillance testing of DG-ENG-1B1 the operator detected the odor of diesel fuel on the dip stick while checking lubricating oil level in the engine sump. The requirement to smell the dip stick for diesel fuel is part of the surveillance procedure. Once reported, the staff de' wmined that the fuel supply line to the fuel injector of
., der number 17 was leaking at the rate of about two drops p' second. GM/EMD engines have- been made for years with all or mest of the fuel supply and return lines integral to the block or inside the cylinder -head covert,. -This design significantly reduces the hazards of fuel induced fires and makes the engine cheaper to manufacture. Inherent to this desiSn is the problem of fuel leaks into the lubricating oil system resulting in lubricating oil r.ilution. It was determined that the leak in this case was' the result of a metal to metal seal 'being marred during assembly.
ROOT CAUSE(S): Design Deficiency - Assumed Risk de OTHER PROBLEMS IN THIS CATEGORY NOTE
._ _ . ., c -r. .-~ - . . . - _.. ~ . . . . _ . . . . , , _ _ _ , , , . . , . , , . . - . , _ - _ . .
. 1989 DIESEL GENERATOR PROBLEMS t
0/ . None of the following problems resulted in a DG failure to start or a failure to run PROBkEJj TO EE.R d REN 0135 DG1 U BOLT MISSING FROM OIL LINE RFTS 0160 DG1,2 LINES VIBRATE NuR 0180 - SB-1s - 54 DEFECTIVE SWITCHES PTL 024 2. . n D01. 2 ^IL LEVEL SWITCH BAD TER
- 0330 'DG1 O/S TRIP WITH SHORT BOLTS RFTS 0331 DG1 ROCKER ARM SUPPORT CRACKED hMR 0352 HPCS DG SPURIOUS ALARMS TER __*
0399 SPARE HEAD IMPROPERLY DRILLED BY MFR. OTHER 0400 DG2 CYLINDER LINER LEAKING SEE 402 NONE 0506 DG1 LOW CYLINDER COMPRESSION RFTS 0521 DG2 LOW CYLINDER COMPRESSION NONE 0671 DG2 SPURIOUS ALARMS AND INDICATIONS FMR 0819 BKR-N1-3 CAN'T CLOSE AFTER DG TEST hMR
- This condition was subjected to an informal root cause analysis by OEA and TECH Staff. The cause was determined to be a dra- error. Recommendations were made (and impicmented)ging to resolve the problem.
COSCLUSIONS , b No conclusions could be formulated based on the above information other than to conclude that the ' prob 1 cms appgar to be random 1'y dispersed at d unrelated.
. .'.g'.
e Me em ., 9 *
... ){ '
1989 DIESEL GENERATOR PROBLEMS . l
' j D.
u
. APPENDIX A DIESEL GENERATOR PROBLEMS PER LIST PEP 2.8h EPN PROBLEM .TO E CODE 002 DO-TK-2 " FUEL SPECIFICATION OTHER FO 0041 s ' '
DSA-V-S IN BACKWARDS FOR TECH DSA 0056 - DSA-V-4A VALVE FOLWD GPEN PDR OPS DSA 0062 GENERIC TECH SPEC ERRORS - NCR TECH DCC 0064 GENERIC TECH SPEC ERRORS PDR TECH DOC . 0091 FUEL OIL BACTERIA & FUNGI hDR TECH FO 0094 DMA- AD-51 DAMPER FOUND OPEN FMR MA:N hTAC 0104 TEST TEST VALIDITY PRF TECH DOC 0121 CONTACTOR FAILED SEISMIC QUALS PTL QA DOC . U BOLT MISSING FROM OL RFTS TECH RARD 0135 DG 1
- 0160 DG 1,2 LINES VIBRATE FMR TECH RARD 0180 SB-1s 54 DEFECTIVE SWITCHES PTL OPS RARD 0187 DSA-RV-3 uRONG SET POINT RFTS GEN. DSA 0197 DO-P-2 FLOW OUT OF RANGE NONE FO 0220 52X/DCd RELAY DRAWING ERROR RFTS GEN. DOC 0223 DSA-V-2B VALVE FOUND CLOSED PTL OPS DSA 0242 DG1,2 OIL LEVEL SWITCH BAD TER #9 RARD .
0260 RMS-DG2 % BONG SW1TCH CONFIG. MDR DCC h, ., 0275 DO-V-56 NEEDS RECEIPT INSPECTION'NONE DOC 0279 HPCS DG STARTUP TESTS INADEQUATE POC TECH DCC 0293 DE -PS-32 NUTS MISSING FROM COVER bMR MAINT HVAC 0294 DMA-DPS-31 INST FOUND VALVED OUT hMR MAINT }NAC 0296 HPCS DSA NO RELIEF VALVES TER TECH DOC 0299 RELAYS NETAL PARTS IN RELAYS FOR TECH RARD 0303 DO-TK-1B FILL LINE RESTRICTED OTHER TECH FO 0305 DO-TK-1B WATER IN FUEL PTL TECH FO 0318 DG1,2 SECURITY VIOLATION PDR ADM' DOC 0322 HPCS DG DEGRADED VOLTAGE STARTS NCR TECH 0330 DG1 O/S TRIP WITH SHORT BOLTS RFTS TECH HARD - 0331 DG1 ROCKER ARM SUPPORT CRACK FNR TECH RARD 0337 DG2 BDC WIRE ERROR FMR TECH DOC 0352 HPCS DG SPURIOUS ALARMS TER TECH RARD 0355 HPCS DG ENGINE WENT TO 900 RPM PTL TECH D3 0356 HPCS DG FAILED TO START W/ONE H. h0R TECH D3 0375 DG1,2 NEED JUMPER FOR S.T. NONE COC 0379 DG2 TRIPPED AT 80% LOAD MDR OEA DG 0384 DG2 10.3 VICE 10 SECOND START MDR TECH D3 0358 FUEL DELIVERY DID.NOT MEET SPE NONE FO 0399 SPARE HEAD IMPROPERLY DRILLED OTHER QA RARD 0400 DG2 CYLINDER LINER LEAKING NONE KARD 0401 001,2,3 BLDG 56 COULD CLOG INTAKE PDR LIC DOC 0402 DG1 CYLINDER LINER LEAR h0R OEA MARD
, 0406 DG1 18 MONTH S.T. MISSED NONE CCC ;. 0407 DG2 10.4 VICE 10 SEC START NONE DG ~'
0417 DG/K55 RELAY DRWG CHANGES NOT PDR GEN. COC 0440 2"DO LINE-CUT W/O DOC MWR BPC DOC 0446 HPCS DG rRIPPED ON REVERSE POTER PTL OPS DG e .n = ,e
,. o i
-* I
, 1989 DIESEL' GENERATOR PROBLEMS
' \
) 0450 DLO-HX-2A BOLTS 1/2' TCC SHORT FMR TECH DOC 0459 DbtA-DPS-31 WIRING ERROR hMR TECH hTAC 0463 HPCS DG VOLTAGE IN ENCESS OF 5000 FDR GEN. DG 0465 , DG1/7 DC WIRING ERROR hMR TECH DOC 0468 DG1 NO START TIME RECORDED NONE DG 0469 DEA RMS DRAWING ERROR EFTS TECH HVAC 0483 DG1 10.3 VICE 10 SEC START hMR TECH DG 0487 DG1 -
m TRIP ON HIGH JCW TEMP hDR OEA DG
. DG1 LOW CYLINDER COMPRESS PTTS GEN. RARD 0506 3 0521'
- DG2 LOW CYLINDER COMPRESS NONE RARD 0523 DO-P-1B LOW DISCHARGE PRESS . FMR TECH FO 0530 HPCS DG JLOlPER INCORRECTLY LAND NONE DOC 0533 HPCS DG TRIP FRON JUMPER INSTAL NONE DG 0537 FUEL NIGHT ORDERS NOT FOLLOW OTHER OPS FO 0573 HPCS DSA CONP DRIVE WOULD NOT STOP hMR TECH DSA 0591 DSA DEWPOINT OUT OF SPEC hMR NONE DSA 0604 FUEL TEST RESULTS NOT RECEIVED NONE FO -
0628 DG2 FUEL LEAK TO LUBE OIL FOR TECH RARD - 0633 DEA-FN-51 DAMPER FOUND OPEN OTHER TECK4VAC 0648 SPARES 41 INJECTORS O/H W/O QC PDR QA DOC 0671 DG2 SPURIOUS ALARMS AND IND. bMR MAIN HARD 0752 DG2 VARS DOWNSCALE - TRIP bOR TECH DG 0753 DO-P-1B FLOW HIGH . OTHER FO 0781 HPCS DG LOAD DRIFTED HIGH h0R TECH DG .'
.- 0759 N512 DRAWING ERRORS RFTS GEN. DOC 33.;- 0799 DSA AR RECEIVERS NOT GROUNDED 'TER TECH DSA 0815 HVAC NO TEST htADE OF HVAC PTL LIC HVAC 0819 BKRLN1-3 CAN'T CLOSE AFTER DG TES hMB MAIN RARD' 0820 FUEL OIL FAILED F/C TEST OTHER TECH FO 0669 HVAC HIGH TEMP FROM ASHFALL TER TECH HVAC 0873 DG2/R10 RELAY DWG CONFLICT bM'R TECH DOC 0884 DGl.2 NO DATA ON CABLE TEMPS PTL TECH DOC.
0891 HVAC DANPER FOUND CLOSED hMR TECH hTAC 0906 DO-P-1A D/P AND FLOW OUT RANGE OTHER OPS FO 0910 DG1 HIGH VIBRATION liONE DG i 0918 DSA MTR AIR MOTOR BROKEN SHAFT MDR TECH DSA - 0956 RLY SCR WRONG QC CLASS OTHER OPS DOC 0957 FUEL OIL CONTRACTOR PROBLEMS NONE FO 0968 FUEL OIL REMOVE TEST FROM FSAR PDF HP/CH FO 'l
..e d .
6 d a
.b NF ; *
- g c1 :d.- c .
b,k'I'. k6rsEG e i 4 h l A t E 1 A A h ve gE u. 2 rf W B r gw 2 -
. .~
'a !
Y ( ' a
- v.
- 8 1 9'
4
. s *
)- .
; a a a
e - 03s441 wt uloom M 4
I O b5
-{.h.9'e g -n Ab 8 A
L - - , -, g I oc. O
^3
_f.: I d
=:
g
,,, a
*9 __ g u
dw --_ , a, _. -
's- .
b Y
- a. .
i e U
- I d I h i i i i i e i DMI WI A1230*I1A %d 4
4
1 l I 4 j i
\
50 g M a. d ::.tIk
-c l y .;.,
N6r!IE .
-. ,e 5
A 5 4
- e
~x
.x ~
G ** gg N e8 g -- ? g W l r 8: Yt _1 .a g-E} i, 1
- .B fu #
A q.< 1.. . I
~
D, , l
$ $ $ b 4 4 i a a h h
h N N! b i i 1
. . - . - - . . . , - . - . - . . . . . . ~ . - . . . . - . . . . . . . - - . . . - . . . - , . . . - . . . . - - . . - . . . ._
- .=- -- . . _ ..
l _I 1 .
- . -t
{ , c t 2 t. C . ' E-a.
. r ,n : Q. ,!s!
.a j '$~ ES$ +
g 1 . .! r 4
.J g l 1
- 1; E
~=
.y 9 2 gi8 i .
x ' R t '
. 4, s f.e.
e a - t i . _h , i 4
- 4 i 1
Ai t a ' e
'M o N 9
9' e 2 2. - 3 . g. .g ' p-
'l h ,
DMI WI AllDOW M .; 4
'4 .-.-.-m,.,- --.m,..- ,.e., .;-,,g,rr.,.. , . , , , , , , , ,w,_=m.,-_ . , _ , , . , , , , . _ , ,,.,m,, ..,.m,....._,, ,,,..7,--, ,r., , , c e i 0,wm-,dvem e. r
II O
+
H 5 f I. EI8
-e t :.
$($Ih 4 e
d + 5 A , q-
.. s W
is - g l as - S n q ,. 6} . _s e ~
! 9~I, d
.a A
_ r,
$ b I $ I e i i i i e i 333441 W1 A11*iOM Md ll _ . _ _ . . _ . - _ _ . _ _ _ . .
d p +W N q,iki Na r . 7
~
as Ex s W 7$ r.
]i A y
r a .
~5 ..
- m. ,
N - 4 s d
- a Dh1 N! Micom y I ,
e
i 1 l l l l l l II g 94 a
- R.EeN 64 ', .
N6 dH e s 1 4
~t ay ,
= $
E *' ).I gy --- ,s o$. 5 j- $ s d ag ~q O l N k N t h -, l m'
+
i l r 1 I M , l 4
=
M
; ~
$ $ $ h e'
i i i e i DYA41 WI M130TIA M l l l l l l a
t i O t h.$II,
-n hh,e'*~i h rh g 9
, g ig - '. b' 4
)t , .
.g N te E a
{!
~j n 4
g - s *g sg e a fW e he
. y':
= .
1 ~ 4 g g .- 8
- d i *
- gy,W1 W1 A1100W M l.
l 1
t , 1 te bEiit ri e 6s d. K6r!!S
- s a
J .
}
va N H g, g av e 32 ' \ .a s d I a.
, J. .
)
~
1 A. -- 4
} ,
3 4, Jc 8 o a a 8 8
,d.
a, 4 e e - - oyA1 W1 ul00W M 4 A- ,
/
a kS M-z yr .an -u 3 k ' <'4 'e a
% E .s t nan ._4 _
g ese w _J T
~ %
t d V5 u
!( ~Q ag --
s fe 5 d 1 N e 1 g 1 I N 2 2 2 3 #
*e i i i i a i i OYA1WIMt%TMXd 3
i . G
.4 0 i ~ OW ' 2/f-- efs sioo .
;5 182IOR i Y ATTENTIOM REQUIRED MORNING REPORY - REGION V AUGUST 8.1990
-7 LICEW.!E/ FACILITY:. NOTIFICATION- TELEPHONE CALL FROM PLANT MANAGER WNP-2 91 JECT: 10 CFR PART 2I REPORT
-D M MET NO. 50-397.
REP %TTABLE EVENT NUMBERi N/A 4 f DIf CUSSIf : CN AUGUST 6 1990.i-THE LICENSEE INFORMED'THE REGIGNAL OFFICE OF THE INTENT TO FILE'A PART 2I REPORT ON MANUFACTURING DEFECTS IDENTIFIED AS A RESULT OF THEIR -INVESTIGATION INTO RECENT. PROBLEMS WITH SHORTED
. WINDI AGS IN THEIR EMERGENCY DIESEL GENERATORS. THE GENERATORS WERE ORIGINALLY MANUFACTURED BY A SMALL
-COMPANY SY THE NAME OF PORTEC. THAT COMPANY WHICH HAS CHANGED HANDS SEVERAL TIMES IS PRESENTLY CALLED NEI PEEBLES AND HAS APPROXIMATELY I4 EMPLOYEES.
THE MANUFACTURING DEFECT INVOLVES THE TECHNIQUE FOR THE APPLICATION OF A POLYESTER RESIM TO ' THE BRAIDED
- DACRON GLASS COVERING THE SOUARE COPPER WIRE AS IT -IS WOUND ON THE ROTOR OF THE' GENERATOR. .THE TECHNIQUE HAS RESULTED IN SHORT CIRCUITS BETWEEN WINDINGS THAT CAN LEAD TO'OAMAGE TO THE WINDINGS OR VIBRATION OF THE GENERATOR. THE SHORTED WINDINGS CAN LE. IDENTIFIED THROUGH THE USE OF AN A .C. VOLTAGE DROP TEST:,HOWEVER.
2THIS TEST IS NOT USUALLY DONE EXCEPT AT THE' MANt*FACTURING FACILITY. THE-LICENSEE DESCRIBED THb GENERATOR' PROBLEMS TO'THE NRC STAFF AT.A MEETING IN NRC REGION V ON JULY 26 1990. THE LICENSEE ~ALSO-DESCRkBED THE' GENERATOR AND TEST IN A JUSTIFICATION FOR CONTINUED OPERATION SUMMITTED TO THE NRC BY LETTER DATED JULY.27 1990. THE LICENSEE BELIEVES'THAT.THE WATTS BAR AND LAGUNA VERDE FACILITIES MAY HAVE GENERATORS MANUFACTURED BY THIS VENDOR. -AN OPERATING EXPERIENCE REPORT IS BSING PREPARED FOR IMPO.
' REGIONAL 1 ACTION: RaffTINE FOLLOWUP
- , CONTACT: LC. BOSTED 1509) 377-2627 L.~ CONSTABLE (FTS) 463-374S- f- _ j -
,g _ g io-3 ' ,
4xwssin n A&&- M M r P" ':
'I?N gjkisTw67
. p, s sea 1 lo f.s -
- }}g;, e, laud 1
, ,y
%l gio*l&'
\. O %
. g g.-
?)
..e-n-
- i g
?
91rr EXECUTIVE
SUMMARY
MDR 290-499 ' DIFSEL OENERATOR SHORTED POLE WINDINOS 2 The root cause analysts of this event includes the failed p' ole windings on'both the DMilon I tad the_'DMslon 11 diesel generators as well as the ones that faded afte'r - rewinding followin; Supply System procedures. In addition,: the noted scuffing of - ! the generator slip rings'and the epparent; discrepancy in the setting.of the slip ring 1
~
L safe:y gap screw is addressed. ; a CONCl.1)SIONS l
- 1. Operation of dol and DO2 with minor defe:ts of tum leakage current (due_
to inadequate _or improper initial manufacturing processes) have not produced . any noticeable symptoms L during1 operation _ and .. has not indicated 1any-- progressive performance deterioration, t
~
~
- 2. Re-winding of the DOI coils after discovery of'the_ defects through the AC 3'oltage Pole Drop Test was a conservat ive action.
- 3. Disassembly oi ine machine Land root lcause' analysis' determined that the; ;
i fa.ilure mechanism of the: pole winding l Insulation was a manufacturing defect !' and ruled out aging or operational events as a causative' factor.
. I q .t. Continued operation oflDO2 hith one coil that indicates.later turn leakage is; g acceptable as long as-the vibrationiand DC field current parameters'do-not l*
show a tiend. Indicating progressive expansloc :of= the inter tum' leakage?
' ~
current ed: The performanae test criteria required by Plant Technic'al Specifications and the agreed upon additional testing and monitoring continue to be met withi . no decrease in generator perfonnance.
- 5. The- process of winding 'and/or curing of the original' polesindings was-somehow defective in that the insulation was abrided prior to encapsulation _
and not fully encapsulated which resulted In an insulation scheme that would ..-f not, in localized areas, prevent shorting of. the" sindings,- In addition,Lthe.- adhesive used was not _the required material;specified by 'the Engineering-Specification in force at the time of manufacture,
- 6. The process .of winding and/or curing of the rewound poles (performed by1 e the: Supply System) appeared .to have . produce-d a better product than the original poles, however, the process is;still somewhat defective in that it does not result -in complete 1 encapsulation _ in' all areas nand the bunching and abrading-of thel glass 'is: stillKtaking place lapparently as a result of the -
method of tensioning the wire during the winding process.
- 7. The . setting of the slip ring-safe"y gap screw at .0S0" may have resulied in some damage to the DO-1 field _windingsJhowever, no indication of arc over- I on DO-2'could be found. Setting of the safety gap scre_w at a g9p greater-S-
W: than the 1000 Volt. setting- does not seem to. bc: justified either by the - engineering infonnation available or by common sense. :t-c
- Pape l ,T A
. . . i S
W l l BOOT CAUSES
- 1. Equipment - Manufacturing Error - Not made per Des!grs. The original pole windings were not wound in accordance with the design requirements in that a polyester resin was used to encapsulate the windings vice _the required Annstrong A401 one part epoxy resin.
- 2. Equipment - Design Deficiency - Specification LTA. The process used to wind the pole pieces allowed bunching and abrading of the Dacron glass covering on the winding wire and did not provide adequate direction to assure proper encapsulation (wetting) by the epoxy restn.
CO_S.IB11LtLT1Sp CAUSES
- l. Procedures - LTA- cimits LTA. The CVI documents used to maintain the Diesel Generators contains the wrong specification for setting the slip ring safety gap screw. ,
WARUSA }LAGON (Things that are not right but are not, in themselves, a root cause of the problem at hand.)
- 1. Personnd - Management Programs LTA- Work- Practices LTA. Installation 3 of the sup ring brushes with the pig tails trained into the wrong side of the brush rigging demonstrates poor work practices and poor supervisory f) .
oversight. RECOMMENDATIONS
- 1. Evaluate the slip ring safety gap screw setting for the Diesel Generators, estabush the proper set point, provide direction for Geld setting, if required, and correct the various CVI documents to reflect the proper data. GEN ESO
- 2. Proside a method for assunng that, in the event the Supply System ever rewinds another generator pole, tensioning of th_e new wire is accomplished with out damage to Dacron / glass covering and that the method. of applying the epoxy to the wire assures 100% penetration and wetting of l
the Dacron / glass. GEN ENO l l l l
- MDR 290-499 PAGE $ OF .
CATEGOR.111 ROOT CAUSE 6NALYSIS
SUMMARY
DO-1 & DG-2 SliORTED FIELD WINDINOS 1.0 1MPLEM ENTINo p_OCUMENT: MDR 290-499 m)T_1.MJiyo plPARTMENTj OEA INITIATOA KIDDER JLMT:2093 ELENT PATE: 6/20/90 REPORT PATE; DEFINITIONS EVENT - a plant situation consisting of a sequence of actions that resuh in desiation from the expected CAUSAL FACTOR - a deScient performance-shaping factor--There - are 13 performance shaping factors: verbal com:nunication, wTitten communication, interface design or equipment condition, emircnmental conditions, work schedule, work practices, work organizatloa/plarming, supervisory methods, training /qualibation methods, training /quali0 cation content, change management, resource management, and managerial methods. ROOT CAUSE - the set of an c'.ents fundamental causal factors CONTRIBUTING CAUSE - a condition, while not sight, was not, in itself capable of creating an event , SLIP RING END - the end of the generator in which ~ the slip rings are installed. This is the only obvious way to determine one end of the generator from the other. The Slip ring End (SRE) of the generator is located at the South end of the generator as it is installed at WNP-2. OPPOSITE SLIP RINO END - the end of the generator opposite to the slip ring end. The generator thrust bearing is enclosed in the Opposite End (OE) which is the North end of the generator as it is installed at WNP-2. DO-GEN-dol ~- The MEL equipment piece number for the Generator portion of the Dhiston I Standby _ Diesel Generator. The prime movers for this generator are two Diesel Engines coupled in tandem. The North engine is DO-ENO-1A1 and the South Engine is DO-ENO-1A2. DO-GEN-DO2 - The MEL equipment piece number for the Generator portion of the Division 11 Standby Diesel Generator. The p' rime movers for this generator are two Diesel Engines coupled in tandem. The North engine is DO-ENO-1B1 and the South Engine is DO-ENO-1B2. VIBRATION ALARMS - There are three permanently installed vibration detectors associated with the Diesel Generator unit. One' installed on each engine and identined as Excessive Vibration Eng. No. I and Exog:lve Vibration Eng. No.2. The other detector is installed on the generator and is identified as Excessive Vibration Gen. TEMPERATURE ALARMS - There are three separate teroperature alarm sensors - associated with the Generator. They are the stator winding high temperature, the OE bearing cup -tempeature and the SRE bearing outer ring temperature. These alarm functions alarm. a. .common flag at the Engine,.Control Panel
. . marked ", Gen Stator or l
7
-y
.. l 1 MDR 290-499 l PAGE 6 0F .
l caused the alarm, there is no method of reading the temperature directly and the areas monitored cannot be accessed by portable instruments. 20 }ET. ROD 1lCTION_j The purpose of_ this report is to document the analysis of the shoned turn windings found on poles 2 and 7 of the field (rotor) of the Dbision 1 Diesel Generator and pole 6 of the Dhision 11 Diesel Generator. 2.1 DyeApAgn of the Generator The standby diesel Eenerators as installed at WNP-2 consist of two redundant sets, each of which consists of a generator, two engines connoted in tandem, a static extter, and necessary fluid systems to suoport and cool the units. Each diesel generator is installed in a controDed tmo,, kere room where -air temperature and tur-Jdity is controlled at optimum it t. Se generator is- equipped with space heaten which keep it warm while in mdby condition and' the engines are eqdpped with a keep warm system. The generator is designed to proside emergency backup power to the reactor emergency cooling systems Ln the event of a loss of off site power and would be expected to carry loads at or near the generator name plate rating without interruption for periods of up to 30 days, if necessary. Testing of the generator usuary is carried out by placing the generator in paraDel with the grid and runr.ing it at its rated capacity and power factor. During annual tests it is subjected to the actual cinergency demand conditions of quick start and support of design , loads on its isolated bus. ne generator is a 4650 kw, .S power factor. ' volt AC, 60- Hz -unit which operates at 900 RPM. _ The generator field ha 5 pole (coils) and is on the rotor. The omput vol' age of the generator is controDe separately housed solid state vohage regtdator which has a maximum output of 125 volts DC at 142 -Amps. The eight pole coils are electrically p.owered by connecting _ two coils series-(ISO degrees apan) to make up a complete 2 pole set and' 4 . pole sets are wired- in parallel through the rotor shaft shp rings. The field windings wound on the rotor.are Layer Wcund-Coil, Pole +, Class F temperature rise typical _of windings found on Sallent Pole Synchronous Generators with high centrifugal rotor forces. The coilinsulation temperature class is "H" (180 degrees C). The wire Is wrapped with'a double layer of unvarnished Dacron glass insulation designed to be encapsulated with a one part epcxy paint-on resin (Armstrong A-701). The conductor size is #5 square and the d aL121.yolts_DC and 142.4 Amps with a maximum output capability of field is tated,See EP drawing A6695 (page ly65 Am L.J ) for a schematic for the 8 pole
~hnding. No'rmal opMg~roltage across the ield windings is dependent on many factors and is difBcult to predict, however, the amperage through the field windings, l under repeatable conditions is predictable and, at fullload, runs between 125 and 135 Amps (i.e. 30-40.wps per pole) depending on generator output voltage and power factor.
~
2.2 Operabilirv Testin2 and Manufacturina Testina l The ability of the generator to sustaln required output voltage and stability is demonstrated annually in accordance with REGULATORY GUIDE 'l.108 and IEEE - Standard 387-1977 paragraph 6.6 (periodic testing) as modified by plant specific technical speelBeations. Routine periodic testing of the diesel generator units demonstrate continued satisfactory ability of the units to meet required demands 1
.- - = - MDR 290-499 PAGE 7 OF . Maintenance of the generator units meet or exceed the requirements of IEEE 3S7- ; 1977 paragraph 6,7 and include; aD of the manufacturer's recommendations for inspection and maintenance. Included in this maintenance is a determination of the stator and field insulation ' resistance to ground which is considered to be a reUable method ima4suring the integrity of the electrical Insulation. _ Industry-standards lead one to beliv ithat if the urdt is periodically tested to its limits, is weU maintained,- and has me minima) electrical leakage to ground through its various components (as - determined by pe: iodic testing) then the machine is considered to be in a condition - to meet its required demands, pole voltage drop testing of the field windings is not a required operability test of the above REOl'IDE and IEEE Standard. The test is carried or h accordance with IEEE 115 and is intended to show the integrity of the insulation between tums on the pole. It is a normally a post manufacturing test and was conducted as pan of the testing performed after bearing repair of the Number One Diesel Generator. Both AC and DC voltage drop tests were conducted to assure that no winding damage had occurred dudng the bearing failure event. The r>rocess comh11 Amp.ly_ of placing a potential of 20 Volts DC or 120 Volt _s AC_ano1Liht._tE0_miltof a Jole set and detmnining the voltage drop across each winding. The total vcitage drop must, of course, equal the source voltage, however, if one cou has less voltage drop than the other then it can be assumed that there is some short circuiting or leakage of the windings of that con taking place. The 120 Volt AC test is more , sensitive in that cou inductive reaction coupled with copper resistance establishes.a-vector impedance that amplifies the individual con voltage drop if shorted or leaking' turns are present. The AC voltage drop test showed that cod 2 and 7 of DO-1 had tum to tum shons or leakage present. This testing is very sensitive and can show ] very minor leakage conditions which may not effect the ability of the unit to operate at rated conditions. 2.2 Effects p_f Shoned Tum O_peration Operation of the generator with shoned field windings can result _in several scenarios depending on the cause of the shorts. In the event of shorts that result from aging (time related degradation resulting from environmental or operational conditions) the condition will continue to-degrade unless the causative factors are accurately identified and corrective action taken to mitigate the condition (s). Under these conditions, funher deterioration may be mitigated, however, the condition will never improve and may continue to deteriorate at some slower rate depending on the amount of damage already incurred and the amount of short circuiting actually taking place. The amount of short circuiting that takes place under normal operating conditions is not readily predictabic, however, because of the very low potential (usually less than 4 volts) between turns, the existence of inner tum voids, the resistance of the carbonized fault area, and the tendency for the current to take the path ofleast. resistance, i.e. through the copper wire, it is generally assumed that short circuiting or leakage between turns (except in the case f gross shorting) only takes place during conditions of high mechanical or electr al stren. In the event of a short ciretdt that develops as the result of a Daw in the insulation scheme (determined to be the case at WSP-2) the short circuiting will initially be confined to the area of 1 faulty insulation, again usually occurring during conditions of high electrical or mechanical stress, will be limited by the extent of faulty insulation and will grow,if at all, slowly as a fun, tion ofleakage current destruction of any weak insulation areas and the frequency and severity of the causative stress conditions. y 4 w . m .- ,_r~.-.. rr% - ., s . - , ,, u ar m . , , a . s, n
- v. : .a , .a
. l MDR 2M-499 PAGE S OF . j degradation is not mJtigated by neighboring good insulation in the damage a;te area.
With severe damage propagation through the coll, the damaged coil's magnetic field becomes weaker than its neighboring poles. Since the magnetic field is a function of { amperes current times the number of turns and the current through each pole is l determined by the series resistance of two coils (i.e. are damaged and the others are ' not) the magnette - field across the rotor for that pole set becor' nes off balanced. Init!alindication of this condition is usually an increased vibration level. More severe shorting can also be detected by increased field (toior) current, however this is less sensitive due to the parallel arrangement with the three other pole sets. When the number 2 pole of the DG1 field winding (the worst case of inner tum leakage found to- date at WNp-2) was dissected it was determined that the leaka;e was occurring in one area located between layers !! and 12 and levels 13 and 14 (see attached sketch) which had the potential for removing 26 turns from the coil,if fully shorted, out of 451 total turns with an effective length of 43.25 inches each or somethin; less than 6 percent of the total winding length. The actual area of the damage site was something less - than 25 square inches and involved four windir.gs with the damage at the intersecting 4 corners. The leakage clicult developed in a: area where the laid on double Dacron glass had not been fully encapsulated with the - one part epoxy adhesive during the winding process either because there was r.ot. enouth epoxy applied or it was improperly cured which resulted in the squeezing out of the applied adhesive, or some combination of both conditions. There is r.o evidence of deterioration of the epoxy or glass present except, of course, in the immediate area of arcing. Where fuD encapsulation of the wire took place there' remains _ a good insulation system. There is the possibility that if the arcing were allowed to proceed as it had, the area would have grown in size as the result of destruction of good epoxy. Based on evidence that the deficient insulation has existed since initial instal'ation, the rate of progressive deterioration has been very. slow to present. 2,3 SCOPE The scope of this report is limited to the failure and events leading up to the failure of DG-1 as reported by PER 290-499. In addition, associated-problem reported by PER 290-533 is incorporated into this document. This investigation was conducted and the analysis made using avaDable plant records, discussions with plant personnel, consultation with industry . experts, v'.sual inspections, basic root cause determination methodology, failure analysis, and events-and causal factors analysis. In addition, an event precursor search was made. 3.0 .DETAll.ED NARRATIVE OF THE EVENT. 3.1 DISASSEMBLY AND INSPECTION OF PARTS During re-assembly of the Division I (DGI) diesel'tenerator after repairs made necessary by fauure of the generator bearings _(see NCR 290-411) a test was performed to determine the integrity of the pole windings. prior to final assembly. The test (IEEE I15), which consists of applying a 120 volt AC current across each set of poles and recording the voltage drop across each coll, determined that on DG-1 there were two defective poles which were number 2 and number 7, number 2 being the worst of the two (see Page ). Pole number 7 was tagged and held for later examination, if reouired. Further testing at 20 Volts DC indicated that the DC .
....._ ,. . . . ..s. .. .w. . ~ c .s .1 . . . . . . _ .. t ~ . _.
MDR 290-499I
. PAGE 9 OF .
defective and was non-conclusive in'regards-to pole numNr 7 (see Page ). _ ne poles _were removed from the rotor and preparations were made to rewind all_ . eight cous. Because it was the pole with Indications of highest turn to turn leakage,L pole number 2 was taken spart tum by turn to try to dhcover.the source and cause of the indicated leakage. As the tums were removed it was moted that the left side' (looking towards the sUp rings) tums were unexpectedly easier _to remove than the; right side turns and that in the areas where the turns could be separated by hand: the insulation was Ughter in color than the areas where tools were needed to separate; ': the tums. MA rough map -(see Page ; .) shows the general area of much, lighter; colored insulation and where the tumsiwere more easily separated / See pole photo No. 6 for thei darker colors and poley photo? No.l l' for; the' Ught.er colon found. l Men the fault ~ area was uncoveted.it was loc.ated = on the"left side of. the' pole' ; i- be: ween levels ~13 and '14 and between layers 11; and 12 (see Page ._ and pole: ; j photos No, 3 & 4). The- fault had occurred at the Juncture of, the four turns, ' - Further disassembly of the pole showed .noDother obvious defectsi LMicroscopicl d;
- examination of the insulation on removed sections of' wire shwed that in the areas -
of darkest color (see Photo No. );the Dacron glass was weU encapsulated and the: ! i color of the copper bar showed through the glass and epoxy. tin the areas of Ughter color (see Photo No. ) the glass was not encapsulated and< numerous bner turn soids existed in the glass and epoxy 1 mixture. . The epoxy -In this7 aria was yeDow - t green in color. Dere were no obvious aging' embrittlement or cracking of the epoxy.- c
- - Discussions with the epoxy
- manufacturer confirmed that an aging mechanhm was not.- i i operative, When the number _2 pole of'the- DGlifield winding-.(the worst case of inner tum leakage found to date at WNP-2);was dissected it was detennined that'the- J leakage was occurring in one . area located between layerspillan_dil2 tad levelst13 1 and 14 (see attached sketch) which had the potential for removing 26 tums from the q '
coil,if fuUy shorted, out of 451 total: turns with an effective length 'of.43.2S inches ~ each or something less than 6 percent of:the total winding length.RThe actual ~ area - 1 !: of the -damage site was something less -than '.25 square inches >and involved fouri l p Mndings with the - damage at the intersecting 4: comers. iThe leakage circult? . developed in an_ area where the laid 'on : double Dacron glass hadinot been fully i encapsulated mith the ~one' part ' epoxy adhesive during theewinding process either [ because there was not. enough epoxy applied 1or it1was . improperly 7 cured which. 1 ! resulted in the squeezing out of the appUed adhesive, or-some combination 'of both 4 conditions (see Photo.No. ) There was no . evidence of detedoration of the epoxy j or glass such as fretting, abrasion or overheating present:except,;of course,lin-the 'j - immediate area of arcing. Where full encapsulation .of the wire had occurred there ; i
- remains a good insulation system.
4 ne total area of-aB the defects was' something less than 1/4 square Inch and . extended for about -S to 10 inches along the plane of the one.large ' defect. ' The. " L smauer defects (see Photo No.- ) show more precisely the faDure mode.:. It would - i appear that amongst the many areas:of deficient encapsulation there were some areas c!~ that were adjacent to oncL on the n_ ext wire that .resulted in some electricalleakage. '.. Inspection of some of the failed-insulation andLthe adio.ining weu encapsulated areas-4 by a representative of the manufacturer of the one part epoxy resin _(Armstrong A-701) confirms that . the : breakdown in insulation resulted ' from an . error in the , application of the epoxy or during the bake cycle of the coil (see Page ).L 1 > Based on the information gained from inspection of the dhassembled DG-1 rotor it I was determined .that the condition found =could be ~ generic .to DO-2 which was manufactured by the.same factcry at the same time. : Utilizing the 120 VAC pole _ 7
+ " -
- - -, - - - , - . , . , - . .,~ s L: . ~ ~ . ;u , . . . _ . , .
1 MDR 290-499 PAGE 10 OF . l some form of leakage or short (see Page ), however, not to the same extent as was I noted in pole nu:nber 2 of DO-1. No evidence of heating, bulging or other signs of deterioration was visible during the inspection. 3.2 OPERABILITX HISTORY Ol pol MLI2 PO2 Both DOI and DO2 have- been able to meet the required output requirements throughout the last five years. As indicated in the attached graph of field current measurements at full load operation the e is no Indication of change or degrading trend. Vibrathnal data also show a level pattem of perforrnance at full load conditions. Tnus performance of both dol and DO2 with shorted or leakage turns in one or two pSe coils has not noticeably affected the ability of the units to meet the acceptan:e crite-ia of the required operability tests. 3.3 FAllt'RE ANALYSIS Failure analysis was performed jointly by contractor personnel and Supply System personnel. See the attached report (see Page ) for detaDs of the analysis perfonned by the centractor personnel. A.REAS/COND:TIONS ANALYZED
- 1. Vibrator, history of both dos
- 2. Exciter ar.perage history of both dos
- 3. Engine lead history of both dos
- 4. Search f:r operational events that could contribute to field winding voltage spikes
- 5. Inspectica a:d analysis of the slip ring safety screw condition and settings.
- 6. Cond". ion of windings as found on poles 2 and 7 and on pole 8 after
- i. rewinding.
l 7. Inspectica and analysis of areas of Insulation found' to be suspect on the winding wife during dissection of the coils. S. Evaluatien of scuff marks found on both dos slip rings during inspections. 3.3.1 Vibration Analysis Review of generator vibration history disclosed that the methods used to determine vibration on the ge 1erator bearing.: are sound and, in the hands of well trained technicians, should produce usable and reproducible data. Most of the data that is j collected is not trended and, in many instances, is not suitable for trending as it
; varies so much that the data becomes unbelievable. The manufacturers specification (see Page ) for generator vibration as measured at any point is .003 inches
. maximum amplitude peak to peak at 900 RPM. This criteria equates to the Supply l System acceptance criteria of .157 In/See Peak Velocity. (not frequency dependent). Review of the recorded data for DOI during its last surveillance (despite the fact that it suffered a bearing failure eight running hours later) did not indicate any significant vibration at any frequency, none at 2X RPM as would be expected from an electrical unbalanco problem, and only minor indications that there was an Impending bearing failure (this at 5X RPM and 8X RPM) (see Page ) on the Opposite Slip Ring End and the Slip Ring End bearings. Readings obtained from DO2 within one naning hour of the inspection th" discloied shorted pole windings
. a j
-p
-' e
/,;
} . -
. 1 MDR 290-499.
PAGE II OF .o j 1 d resonance, mechanical -looseness -or' electdcal unbalance = at -2X1RPMP ~ oni the horizontal plane of-the Opposite Slip Ring End bearing. It would be expected that
~
an electrical unbalance problem created by a weak pole would: effect both ends of the machine nearly equally. This is not the case with'DO2. Longiterm trends of vibration indicated that-(see Page ) DOI has shown no'significant trends over a five year period and (see Page ) the data for DO2 Is inconclusive because of the sparsity of data gathered at comparable loading conditions and;the poor. quality of the data ' that does exist.- 3.3.2 Field Current Analysis - Review of field current readings for a five year pedod indicateL (see Pagen ;.and:
) that there has been no significant shift in:' field currents under ' conditions ;of-comparable loading. Discussiens vith Electrical Engineers, _both within the Supply System and from rdthoutilndicate that because of the number of poles (eight)'on the' dos it would be unlikely that a change in field current would be notlecable with the-amount of field winding leakage found on the WNP-2 mach!nes and that indications' of electncal unbalance would be sery obvious before any indications of excessive Eeld current were noted.
3.3.3. Engine Load History AnCysis Cylinder exhaust temperatures as recorded by-thermocouples at the exhaust-manifoldsg l are a good bdication of' lead.'on each cylinderi "By. averaging the 20 cylinder iemperatures a - good approxication of engine load ' can be0 obtained.' Tnese temperatures are = routinely reccrded at WNP for the purposeLof Edetecting- faulry-injectors and to assure that loads between sides of an indhidual engine are equally loaded and also to assure that the two engines of an indhidual generator set are-approximately evenly loaded. Aserage cylinder temperatures were plotted for both 4 generators (see Page ) in ar. attempt to see if(there -had been any; unexplained _ ' shifts in loading under comparable electrical output loads, the; intention being that if , there had been a . shift it mfght provide a clue- as to when the field windings had staned to shon. So significant changes were noted in; average cylinder temperatures over a five year period.
~_
3.3.4 Search for operational events that' could contribute to. field. winding voltage ; spikes. A search of documentation that eculd provide' operational history:of events that could I. have caused field winding voltage spikes was conducted. No :docuinented cas'e was - l found that outlined such an event.- NCR 290-411 documents evidence of at least - one out-of-phase _ loading - event on =D O-1 and. discussions = wiIfi plant personnel
~
Bisclose that there -have been several such events of varying intensity, none of which _
- were remembered to be particulativ bad or that resulted in a breaxer-trip. None of the personnel interviewed. remembered ever rurdng the DGs on the gria during a.
period of time when there could have'been a lightening-strike and most indjeated
~
that it would be unlikely that the' dos would be operated under such conditions if _ there were -any other options as this . type of operation: was addressed by an -NRC i Generic letter several years ago. The lack of documentation does- not mean that
~
there has never been a voltage spike event, however, it does IHdicate thatlTIIIe"re~h7 Fn"8ife irwaTfidny mud m apparence and that1t'nuwdTRTTr:i5Vern at the tune E-
'6f the evedt -
9 9 e e u t s, e.. A s. .:* .. . ' .2 a 1..- t e
~. . - - - .- .- - - - - . . . . - . - - . -- ..
'e. ~
- MDR 290-199 j
- PAGF 12 OF ' ..-
- 4, l
The DOI safety gap . screw was removed for Inspection on 7/12/90 The safety gap-screw consists of two 3/S"-16 studs aboutcl-1/2" to 2" long, each with one end a slotted and the other cut flat with a hacksaw or other similar device.1 The electrode n end (as opposed to the slotted end).ls not cut square and the cut marks are obsious _, on the surface. -When ' received, the . electrode _ end and a portion; of the threads = j towards the slotted end about .1/2 to 3/4" long was unvarnished and the electrode 1 i t surface was covered with a Ught coat of oxidation; l Initial examination under a 10 - j and.20 Lpower hand held microscope revealed lseveral: arc strikes In' one area.- Dur!ng attempts to remove the oxide layer the three -smaDest are strikes were lost . under the 10X microscopei Follow-up' examination)(see Photo Number ') disclosed - I ! several h!gh voltage, low current strikes. When-the generator was disassembled the~ :j safety screw setting was.at about .050". Review of surveillances performed'annuaDy -!' per PPM 7.4.5.1.1.2.16; Diesel Generator Dol hiectrical Inspection, Step :13. indicates that the ' Safety Gap Screw has been maintained: at .0$0" (nominal) since initial
.I stanup. Review of the manufacturers test data;0n Page. -) reveals that the initialJ d
- setting Af ter dielectric testing was-established t.; .u62".
The dielectric strength of air is 56 volts / mill at STP (see Lndustrial Electricity , J..$1. Nanon & B.J,' Gelmine; Page 290, . Van Nostand Reinbold Co., N.Y., N.Yc 1951). Die!ectrie strength is that ability 1of- a material -(orc gas) to resist being punctured ! electrie.tuy and is. expressed in Volts per Mill. j X volts /.0S0"-S6 volts /IE-03," or 6SSd Volt _s for a air gap o' 0$0".) c _ . Funher investigation revealed that :the~ Diesel Generator Technical Manual-(CVIl02- i 53-00 65-1-1 {$ee Page })' states that the safety _ gap screw ?"has beeni set: at the
- proper clearance 'to suit safety
- requirements, neset the gap periodically Lto:.080",
NEl Engineering Specification ET 2.4.2.S; STANDARD COMMERCIAL TESTS--- SYNCHRONOl's OENERATORS WITH 2 OR MORELBEARINOS' paragraph 3.2 j (s(e Page ) states:. 1
- Spark gap of all slip ring assemblies is pre-adjusted tot 050 in. prior to assembly J and Dielectdc tests. Upon completion of Dielectric testing per 2.9,i the spark L gap shall- bemadjusted to_ a gap that ~ allows 1 spark-over at the overvoltage; corresponding to 60% of 'the _ voltage - applied .during Djelectrici test or 1000V,
- whichever is higher....
For 1000 Volts (assuming standard pressure and ~temperature): . ; 1000 Volts /X Inches-86 Volts /IE-03 or .0116 inches? 3.3.6 Condition of windings as found on poles. 2 and 71and on pole 8 afterl - rewinding. :i d
'a INSPECTION OF POLE NUMBER TWO -
Pole number two was the first pole:to be ' dissected 'as-ittwas the pole with the-1 lowest voltage drop recorded during the initial pole drop test of the rotor (s'ee Page'- !
). Prior to dissecting the coil attempts were made'to localize the shoned windings: i by applying high-voltages and amperages.to the coil and allowing it to heat up. At o none point the coil was aHowed to' smoke for. several minutes in an attempt to bum . j J out the short; Thennography performed' during. these heating; tests wat inconclusive : '
'j 1
c MDR-290 1499 PAGE 13 OF . of the windings after dissection must be tempered with the. knowledge that the tests'
]
probably [mpaned much more stress. on the cod than wotdd ever be seen during~ q normal or off normal operation of the generator. -
]i During the. dissection of-the coil it was noted that there were several different colors !
apparent on the indMdual lays of wirei seemingly dependent on location in the cod (see Photo Number ). The right side of the cou generally had the daiker colored wires and the left side (the lighter colors 'and there also was some differences noted-- with changes in depth (see Photo: Number and :)s It:was also noted that the. ease of removing _the wires from the bundle was rnuch greater on the left side as; j opposed to the right side:which required more effon to strip a:given _ wire. --Small' bumed areas were noted =on the intersectioniof four wires (see Pagen . )-and these - wires were removed for later laboratory- evaluation. The general. condition of= the coil as a whole was considered to be good with no obsious faults (other than the noted _ ; bumed area) were noted, j INSPECTION OF ' POLE NUMBER SEVEN Pole number seven was also subjected to testing with elevated voltages and currents" ', in an attempt to localize the leaking windings to no avail. This coil was t_ransported , from Spckane to the WNP-2 site where mit was : disassemb_ led ; by c a J groupl ofj _ consultants (see Page - and ) with experience in defective electrical insulation and; M its location. 1.ayer to layer areas were closely inspected and any; suspect: areal ] examined with a microscope. Turn to ' turn areas 1were, for the rnost part, not i inspected. Overall the cou was determined to' be inicxcellent' shape and.no faults = were found. Photographs _were obtained of all suspect areas..and the suspectMres: , were examirsed in the laboratory for evidence of shoning; The c'onclusion drawn from 4 this examination was that the electrical leakage present in the coilvas microscopic in . nature and that there 1was no indication of deterioration. of,the cod or insulation ! L . present .that could beround. (, INSPECTION OF POLE NUMBER EIGHT l - Pole eight:was rewound at Eastern Electric Company in Spokan_e and, after baking,' ~ L failed the AC Yoltage drop test. No attempt was' made to localize the failed area-l and the cou was returned to the WNP-2 site for analysis. ;The cut ends of the cod L were t' rimmed and ' cleaned (see Photo Numberi ') and h1gh resistance Ohm readings 4 i .were made in an attempt to localize the Lleakage. .High resistance?(200 > 300 - MEGOHMS) leakage was common across the cou when measured with. a Simpson VOM at 9 volts DC output (su Page ). After the Ohm'
~
I readings were taken .at nl+ volts-.the wire to Mre : rcsistance was checked with.a ' Megger at 100. volts DC neadings across several layers of wire recorded at 300-
~
MEGOHMS were not uncoinmon. 33.7 INSPECTIOS and analysis of areas of ~ INSULATION found-to be suspect after removal from the coils. Wire removed from- 'cou number two rs subjected : to 'several.. different' tests.~ Photographs of the fault areas were taken se Photo Number - :thm - -) and the general condition of the -insulation _ throughout the 'coilLwas utablished. 4 A simple 1
, y
MDR 290-499 PAGE 14 OF . removed from the area of faulty Imulation disclosed a signincant difference In weight . of the insulation and epoxy mix. The insulation as applied on each strand of site l consists of a double layer of _ Da:ron Glass applied without varnish to the copper wire. During installation of the ste to the pole it is coated with an one pan epoxy { i ! (Armstrong A-701) which is a hip.!y viscous material (2000 to 3000 Poise). Once i the wire has been totally wound ccto the_ pole it is placed into an oven and baied I at 350 degrees F for S to 10 houn. Six of the eight poles atte::pted resulted in a satisfactory AC voltage drop test. Microscopic examination of the " good" insulation and the " bad" insulation from p:le number 2 revealed that the " bad" insulation contained a large amount of voids and that the glass had not been wetted during the encapsulation process (see Photo Nurnber and ). Where tums lay side by side and there has been no encapsulati:n of the glass the only insuhtion provided is the air gap existing as a function of de thickness of the glass wra; ping. One layer of glass is about .00N" thick, assurr.I four layers of glass (two isyers on each Mre) the total thickness would be .0:.6" which (inL air at standard temperatures and - pressures) would are at 137.6 vo:3. 137.6 volts is considerab'y less than layer to layer potential of NS volts afforted by the 6SSO volt protecton provided by the safety gap screw set at .0S0" but, sculd be well protected at the 35 volts afforded by the 1000 velt protection with the safety gap set at .0116". 3.3.S Evaluation ci scuff marks fou-d en sUp rings, it was noted on both dos that there were scuff marks on both generator's sup rir.Is . (see Photo Number and ) wh!:h were considered to possib:7 be a syrnptom of shorted field winding tums. Several theories were proffred to explain the scuff marks, some of which arei improper brush r:aterial, rotor dration, improper tensioning . of the brushes, a-changing electrical Gux within the f.eid under othemise steady state conditions, Seld excitation with the rotor stationary, !!ghtening strikes, welding on or near the feld windings, and out-of-round sUp rr.gs. During disassembly, it was noted that the brushes were insta' led incorrectly i- fSe of the eigh orush holders. The brush lead had been routed under the bruth tensioning device which, under conditions of advanced wear, could have prevented the brush from being properly tensioned. No other reasons cou)d be determined for the scuff marks on the SU; rings. 3.3.9 Inspection, test run and evahation of DG2. As a result of an inspection peder:ned on DO-2 ft was determined that pole number six (see Page ) of that set indicated some shorted Mndings. No other unusual conditions were noted. Iniew of field curre- l seepage ) and vibration readings (see Page ) disclosed no unusual degrading conditions present. Inspection of the safety gap screw removed frora l>O-2 showed no indication of are over (see Page and ). The st'ety gap screw was set at .0M" (as directed by procedure) when found and was reumed to a dimension of .062". A 72 hout test run of DG-2 was performed with ae generator at full load during which time output wave form (see Page ), vibratic: trends (see Page **), and Eeld current trends (see Page
) were monitored. No unusual or off-normal conditions were noted during the test run. Upon comp'ation of the test run another AC pole drop test was performed (see Page ) to 5.te if the indicated shorted winding condition had worsened, No indication of degraistion was noted and no further changes in voltage-drop across the coil was noted.
3.3.10 Investi; n of Shorted Ph vnund by Supply System Prmsses.
, I i
.l
. l l
MDR 290-499- i PAGE !S OF , i Upon completion of re-winding the eight poles of DO-2 two of the poles failed to l pass the AC voltage drop test after the poles were remounted on the rotor. The poles were numbered four and eight. Pole number four ~was dissected in Spokane where it was determined that there were no apparent reasons for the shorting,- No l attempt was made to localize the defective areas other than to subject the coil to ) destructively high voltages and currents and obsene- hot spots and smoke. Pole l number eight was transported back to the WNP-2 site .where it was subjected to- ) testing to determine the extent - and . location of shorted windings as ; well at mierescopic exandnation of the insulation from selected areas of the pole. Insulation resistance checks of pole eight windings at nine volts-DC disclosed a generally poor inst 2ation scheme throughou'. (see Page ) with three tum to turn dead shorts,407 tums wth less than 1- MEG Ohm (13,000 to S50,000 Ohms) and 12S tums with greater than 1 MFO Ohm. Microscopic examination of selected sections of the wire-resefed abrasion and separation of the glass Obers and poor encapsulation of the retninbg fibers whjeh apparently stemmed from jast not applying ^a suitable amount of resin to the glass. Inspection of the remaining portion of pole number four disclosed that a portion of the sinding indicate excessive voiding between hyers and between tums while the - remening tums were well bonded. Keview of the sinding records show that the un-bonded sectics was wound on one shift and the remaining sections were completed on e.other shift which would indicate that there was at least one person involved in
- j the re-sinding operation who'did not fully understand the importance-of the epoxy in the insulation scheme.
Renew of the winding process showed that the new wire is removed from the wire sped by unwinding after which the wire is immediately turned towards the work site by pissbg it over a piece of pipe covered with mica. From there the wire passes thrc;gh a tensioning device that prc. ides for proper tightness of the wire on the pole piece. Enmination of the site frorn the spool, after the tuming point, and after the tensioner indicated a large amount of contaminants (dirt) on the glass (the wire had been sitting on the winding device fro ~ several days. since the last pole' had been wotmd) and areas of the glass matting that had been bunched, cut or abraded during the process of turning it or tensioning it. His same bunching of the glass matting had been observed on both the original windings and on poles wound by the Supply System. 3.4 EVENTS AND CAUSAL FACTORS See the Events and Causal Factors Chart on page for event-and causal factor sequence. Two signiGeant causal fae: ors were identified:
- 1. The process for insulating the field winding wire is somewhat defielent as written in the portec Engineering .SpeelGeation and will not produce a quality product on a repeatable basis. Some improvement was noted in the process when modified by .he Supp'. Splem to include application of epoxy to the wire as it came off the spool
- 2. The original setting of the slip ring safety gap screw at .080" was incorrect per the ponec Engineering Specifications and the failure to incorporate the proper setting into the Vendor's Technical Manual prevented the Supply System from detecting the problem. The incorrect setting of the safety gap screw set up the machine for eventual problems
- r~ - .a , ., ,, . - . . . . . , ., , o n v : - . m - . .a . . . . . .
t MDR 290-499. l PAGE 16 OF . encountered,in all probability, would not brave been destructive to a properly installed insulation scheme, 4.0 CONClxElONS
- 1. Operation of dol and DO2 with minor defects of turn leakage current (due .
to inadequate or improper initial manufacturing processes) have not produced - any noticeable syrnptoms during . operation arid .has - not Indicated any "; progressive performance deterioration.
- 2. Re-winding of the dol cous after discovery of the defects through the AC t Voltage pole Drop Test was a conservative setion.
L Disassembly of the machine and root cause analysis determined that the failure mechanism of the pole winding insulation was a manufacturing defect and niled out aging or operational events as a causative factor. 4 Continued operation of DG2 with one coil that indicates inter turn leakage is acceptable as long as the vibration and DC field cunent parameters do not show a trend indicating progressive - expansion .of the inter tum leakage current and: The performance test criteria required by Plant Technical Specifications - and the agreed upon additional testing .and monitoring continue to-be met with no decrease in generator performance. >
- 5. The process of winding and/or. curing of the original pole windings was somehow defective in that the insulation was abraded prior to encapsulation .
and not futy encapsulated which resuhed in an insulation scheme that would not, in loca'! zed areas, prevent shorting of the windings.
- 6. The process of winding and/or curing of the rewound poles-(performed by ~
the Supply System) appeared to have produced a better product than thr original poles, however, the process is stH1 somewhat defective in that it does not result in complete encapsulation in all areas and _ the bunching and abrading of the glass is still taking place.
- 7. The setting of the silp ring safety gap screw at .0S0" ma.y have resulted in some damage to the DO-1 fleid windings, however, no indication of are over on DO-O could be found; Setting of the safety gap screw at a gap greater than the 1000 Volt setting does not seem to be justified either by the -
engineering information available or by common sense.
~,,- o 1
-i l
have produced a better product than the ' original poles, however, the process is = still somestat defective in that it does not result in complete encapsulation in all areas and the bunching and abrading of the glass is , still taking place. j
)
- 7. The setting of the slip ring safety gap screw at .050" .rray have resulted in some damage to the DG 1 field sindings, however, no indication of are over on DO-2 could be found. Setting of the safety gap screw at a gap greater-than the 1000 Volt setting does not seem to be justified either by the engineering information available or by common sense.
l 5.0 ] LOOT cal'5ES
- 1. Equipment - Installation Error - Not made per Design; The original pole '
windings were not wound in accordance with the design requirements in that a polyester resin was used to encapsulate the windings vice -the required Armstrong A-701 one part epoxy resin.
- 2. Equipment - Design De.".ciency - Specification LTA. The process used to wind the pole pieces a!' owed bunching and abrading of the ~ Dacron glass covering on the sindin; wire and did not provide adequate direction _to -
assure preper encapsulation (wetting) by the epoxy. resin. l 6.0 CONTRI.BUTING CAUSE3 ' l
- 1. Procedures - LTA- Llrr2s LTA. The CVI documents used to maintain the:
Diesel Generators contains the wTong specification for setting the sy ring safety gap screw. 7.0 WARUSA MOON (Things that are not right but are not, in themseh;es, a root cause of the prob:em.at hand.)
- 1. Per annel - Management Programs LTA- Work Practices. LTA. Insta"ation of ti e slip ring brushes with'the pig tails trained into the wTong side of the brust rigging demonstrates poor work practices and poor supenisory overu;ht.
3 i l-
4_m.E . .h_, - a a m l a$ sp. n g,e 4 h k fy N [$p$f;)p.- l[ g* N; "
%h - hi4 .Q" Bi(Nj,i '
5$ :" y 7
'I
) !
lr W4 wy
,J! J' f ,
, s<ff..!
- II ';. D Ih 'f
;-) g *{j, -
wa ,?j 'a e gng. e 4 ;. 75 m,j ., ,- v. 7 k ,l
~'
b g Y < _
' N,
$g$ ap 8 %{I}k M-u -~
, .- : a
~
i fE$$h. -w an Y ,' . $- r
,k~ "k,I'5 ' r n
- k. " t
( , me :, : y'
- (f x$ a.s..? '- -f.\']6' % ,
; au Qip; g l% \ , ;:
gdk , 3 - a y#4' . . .
$v6?lp$*>;;\
y
~
hp M . { ,
~ c
.. A a - ,
g% G .,; *,h "^;f .. p' afb
. ft .
g
'% i ggy qg ya, ~
, u.;4 .
g~,g;'; .
. :J ;+y
'#s .-
a
,P h Q;h; :r n -
., R
$k 'y -
?)ihf$A
_,~ I;' " ' 9 ,, t
^
hN ' ren -
- n. %? -
,g 1,. . ~ ,,
f
,,_y, , - - -
4 4 [
. % 'N ,
r .
"P 1.
- s
.;< e&g;. & d @ipfi~ ' 6 -
fkA 1 g %%;-%w . W W e< w .y n
%; pM ,:ea, 4 a
; fp p ..
j = g s. p ' yM6gg . ,
. g;.g - ;
, p, _. , j ' 'fp l
.. f 't.
,g.s.ig s ,
s
,y d< ,, i j ; $
4 3 y
* : ef -
.~, ilf.h9 *h 3
f,h [*hl ! .
,3 s,vm. g : 34g
~
-a Y ':'%i. - t.a}
s . ,. 2 . .,... $. g> q p, w fu
't [
..g y ;i; , , .- ,5 y
.y.pggg4 7.e , ,
w.
- m. sen.gs $sAe &,h t
+g m g =
w ..
.g:, A.c 7.' <
a...wmmh $p
%.m
,,,a
. m,.w s
*gG '.y , . . -7 , j,p, g* '
- vq f
,i-
'y
- ifi.,'$~0!;e'F yg y',le.
a.ya.,. .w. . ..;;. c ;.. w ;. ..m .r -: yy n i 3
' 'i., % 3 : T;;. ? k, . *, f*g,:'..* ' ' - 4 ,. -
i , b. i.
. .- Y
,) ;fl,
- h* * '
y[-3[ hk llr { ,, ' .'.., * % ' $.? , , ;
~
'f.h;g.%*
- H
'f m ,.o ;' ~ ~ ,q;pr .-
.'5
..g, Y,7; ic., ; - 3 l g
%;yv,!
^ ;-
*g ?,i.. .
3 :'1.ri':';., f. 4
. "d-l.
S s n
* .)&Y:;7.:.
i ;s. 3.i.:::?- .h',ud....':$' \( 4 .
~m kgQ b y
?
cew.". ..7...,), g . 's . . , .fp@' . .. ._. '. :h' h4;?5:k
- ap',.4.., e,. .?. 5 M. 'TM. infis '. '. ?
...c
,b.# .
h,.(.s,. S *=4
, k.":(s. .g'. g*T- ;. :j ' ~
- ~
~~ ,y.yr . .
.a . .t ,,- ,
,e .. *5',*'- f;D;9.".:, / .'S },!l:7. .' ' '; j N
. n a. y pp; at2 _ .;gg:-J M3 y D $.
%p.gM %*a'
@kY.hr .pq0Wm a i y . 2"
" 4; Jc l
Y@~@9,mgt #. k # dfgsseml?; _ gi m e 5'Cf" " g g. a ;
- RV-Af@ge.*gmt.h .~ a ?~ v, 8 pJ
, .e ~ t ' I c k , h y \A nE ' 4E p l a_ y, s ur.{;ymx j p on y p_ y
'a
~,._
. as Av rn
$ 23 l [ETE4MM M ems 1/M(WM EifltGNSEEWiff
- l ffQfff-i :
4 s;y ::
- n. b .i . , *1 ,
.N+ 9~
}E fg a f' ; , ff!5
- i i %N .
kff hd W h um) ; psykh , f wy o: y a mly yy ; qqu&>+}}, in h am me . gro' m^a l h. if d s2}! w i
.2 n , ' ' .,~. y E
$y '13 m@M$w$Md<
e WiMgffh dW ti it 2 p : %
.f p 2.
'r 4 s
3 g% ; spw * 'wri i t e
.. ,e ey 34% fg&g'i 7 p : gr. J p.ggghge s$bh 4 . p4 ,' ,% 2 n i k, n way : < g st eg 4;lv}& .
s XTtu u.- un r, - y y, c y,,
; . 8 .
(, i G ,ya
~
i e &O W %V ' ' [' g .E f,L , k ed
, g 1
["- sc. i ,2). , y a
;h.. My@ g p' a '
MT 4 . .
. - t, < gGj Q 1g a-g,
$(
p v A ; gu, , sR 1: r;& .
~
t ea
-Y . , . . f '
ij 72 <y I $.w a .
= t il {' n
{ , yajif- @#
I CJN! gj s y 2 q ! s .
. 4 :;
* ~ g j, 1 i I Y S 'E ?
L gj~ . f, l ] . !') -
.s 8 f tl 3 l
]
I r l
$ h' h.cm;I
$ P Sa %+ at /
d EsA.+(q#b5[g
' W* '9p%
4
'5 og .e,Y%f.C $( psor, ge
.w.+ 2g..
.9.be .
.3 5 I:#@
1
- h. kk" aq? -
m
- 2. -
'SU~
s
- eu# *
Aj 3' 3 i 5/ pp!.qdhu, R h ^ ' A4- ' 3' O f'3'f l;$ f 3 h
- g. ;, p;ge, g
, ;u 2 ET > ${r;c .
ga. p
% p cW-. .-
- y. -~ x@ a$ $. 5' : 6# ?n
. c a.
\
w{g$m ks t ((p$ re Ef. in . k ) y, e < '6 - < I2i[ xGihq h. %lhh E$h / ' h pIS* y
%s lps
'1-w ._ e.s .
1 j
%f 2l&un$t;W
. ~
M
^
..r ' N #*
vIlfM4 339 'r% n ., inm4h
., ys.A %.gr q.ygg5 . . : .
, .e u
.;TQQd %* '~ = * ' q ,o Q;f';Qt) h j?
t' .' ' .
'" l l;;)i' sl H
'~' "
ty ~ 8
~.
',ih,Ika w sN
$h n
N
1; ,t g ,' ..f.f., I f
i % I
.W ,;
~~ .; . sg ;
76,nP- v .(n*/jkj(.). W Y wg -,o .e' s p.3 A Ql' w
's ' mm.
', @0 3
t nk:u: ., 4> ~ ,in [ sw 1 $$ , f [J a-8 eR um:rsimmmrM:2 amu mwrasmpantwm1mmix 4 , .- , ~
e M a M
- ,
- 1
. . i
. . . %' em . . ( *= I #,d . ., e
,.. 4g k kn.a j
f4* . 1
,.afs4.,y/,f.
.; . '. ' i '
s 3
/ .gy .. s.
.F 4$
/~p '
. . ik.R . . ~- %
" ~.I hi 4$..[Y&afh . .f. c i
9 f f;- %, f .,..
. t Yfphh# ,.:..-
~
. .' - . , , . .. :f r. -
? -
- ..e. #: 7. N' -
!$jd ' f.v%%wRsifR. 3 fii
^
\s.
^
34$N f i
$. hf -
+:..p ~ easy & yg4 . /
,3 xe6 c kw ~
.j-
"df G2hf*,L$g,g?l,.+
e u , O,. i? 14,e pp... l ' W+ -[ i [f e a .n ,u. u,. t -c- 3 m
~
, g :i .,,, a
.a. un
.g k Y.$ ;' yr w
.a m. J my h*
k'I J. , yQ"Q%f.[b,g r;,'$ 12 . .
"hM.(.=-~
k mo W wit e, ? }h' '/ ! . ti -
.N (pa@s w- .
g, f - 4I'fp.;~
.. ~ u p 4.; &x > 3
- u
.s yng? ,
f e .. k . ,;$E.,
$'(,. f -
e*" . - s -
. n ;a -
s q g 7 PJ !J +[ t.s ^ n ~ g V?$fm l , f *.., W it 3 [-4 gg 1 . e
'I . t yg, g &r_ o e -. . . p j ma
$&h$
t ""
. mi -
- i!"l.
a N T 3 n
5 nape wg ; i$giil$g%@p!!!# q kdt j - g, (:*fqy y
- b :-
p f u en Q-w}g BgT f e 1 ( .
>;k f
phls. i g w n ;a
$DQ U e k; y ,h [ q? w 4 ; -
. 1 hlh$$lkf5 h 22Wi!Y ' ., hh kk~'"
ig M ,g@wa ~' nu & 9%%ns$%ar . y
- %l\W "
y
- e. -
,. ,un, o s ,
.p .
c peas - mp';= . . A am v / jil" P ,7
) .
.h. . " ir - y,, ,., .(
i 1 h%k ,1;p,.5 4R"' '
- 4. f. .y
.. se'.lgE A r %ams w.p,.
aa
. ., 4
.a. ; J 4; .(l ; <-
9 {m p;u~
.1. ni ,3 :g; j y 1
- M
.g~
n
;>. *[r ' -
F' ,
~
.* [F .I
- f ..
v >::w . ,4%);>s ajt .n i t
,A db- a;p[aot,t (6 -
g 4 ,, /. .,' 5 h ? i; 3
*lC.C?,gw. ^ h c)
'^ ~
khh$.h._hbd(fti{ili j ' ._ .a;',_ E
.c wg . ,,., .
" *^
t'
- sy r
0 , L
e 4 e g y
. x n.,~9 9p ,
y . f<Is !hWQ DQ M [ by > 4lWQ+$g#N*$)es!b yaJ#ds db
~
hjf N!hfIlY. i hp@)-s
%+%,
4 oe sm e
;e d p M s.q Pl ff.
c
; s 1
. , ,o 4
y? a '" %Y p p l2 p, 3 5 A (
- h. $v ;Aw.m y);, y ,'30'iC& $%,
S, h a.C' t wl;-. h. ^ b,Nb
;a ' ok ,'
1
.G
,s 1 w< w
^
Aw53% p 3 .w[%3f'A"$g$[ n a y' M k 4 M fth.hf 8, & x W eg s~ ik s ~ . '.J - ~!
$5hg f ?V y .'jf . -g, QgAg%g t'a ' Agn,d &gd, .ig n %
wa[x u ' #
$$hF4
$w$4$$P.#j(%al" 3.g., .
aaL\, s Add a.-
%~ y y, y ag",
a
*
- gr , g , - ,
g- g3g
;hyl - '
is " 3 i ~ (.p[3 7 -
%s? u 9 2 y oj ? 5 F@(edk,,g M
!t .
P' My" n. Q,.,.. e.f y 1 I P % %, o ,, z
#%5fsgs 3 t: ?) ~ 'a
; et2 ggp $ #;w lj G,Ei
$jfjnllq";,
$s ew +g;g,,.. se jg M ' fty* e ;[L jl{
j$$gy Qb d4 -
$' :w - lRn
$5p bg
%,5.<qq;)g~p s ,. g q >
,.y
.: n
-' t ;
u) ce < y h3 g - h?* . G N
- A
- I 3
e
- a. - .
, v . <, w, ;
' I -
d$ p 575NN$ 4 m uw r ! m' g~~ f l ssp r
$fr
'g f .' '
1
*j
?
- E e'
[{l h> Eksh' e h '#) krL.,,l < ang($u%',T=
$dI s >j i '
Ef h h 5II
% Api!! 73 w sl: c kkflf!!$!hhg{.if mm e
I f, 4 f 2 y t-Yq 'b" p i-
+ s} Y '
'y ( t h~n.. }
g s ( x b
- 3 u
..s.~n. . .
- ~. ..-.
3 .,-y .. . , , ~ . . - . -,
,e h
'"' ' *'* ' ' i
Q,.. :s!' ,fTi '
-7'e -
g , I i
] . . --
. e
,- t;;
} e
+ 4 e c; ,
. 4.
A l J L 1.e, . j !I j 9{..it' 4.k!
- s. ,
o
-8 .;,.
. y. ? s.
y 1 l
==s 1-I e **h 4m e+=me- - .
. Artearop , s; %WJf #e*T P F **'** #%4de p=WW Wg[g).ma e 4 #. 4 -#=
. -. . , . _ . _ . . - , . . .4.. - . . , , - . - .. . - - , . . , . . . ._ _ , _ , . . .... . . - , . . , , , . ,
, , - - a - a - 6
.- j g -- .
**[
k; '; ; -;' . .: ~.
. u.
i
; , ;% .y, '
4 p< a
.NY!'y,'E.s ..% . ;
,,. o 44,c
~
49 ; $. .. .
.* % .%g
}- :' u. C'. . . % '
,.&g5' A h',
i 3 2N +,:yp& ,Q (('
.F f'
. . . p .
V f %* f' l. ( 2 m ,' , t_ qf A.
@ p. {fgg -
g4 in
+{$
7 Y, ?* w
~
t g , w I .e W lh , g'i . , ,
*l ,
.n QQ5 l
%a% b-.],[ r&T ' f
?
- ~
.i u - -
L . ;,:;o
- l ^
. s: n
- > < 1; .
' *
- k
,h
~
.. fg ,, ..
= ,.s~
~ ,
..- + s r
g~
~~.. ' .
..e
;.[ut y.%x':e 4
'gy -$,- ,
Q?s.N.kN m, -e ef . s,n ( 5 c d, ? f.. 3 ' r _, , , , , ,
.Pge Ed_ -
'j 1 4% -
t
,.g.
7e F p i c.'
,j' 4 b-g p W %
g,i [' A'
< .i i i
' ..y,t\.5)b ; y.
, .. . a .
og,'hy&gi' t i c n s .,#,' ,, Q{5IAE[iTE:2ItSEdd5ssib3Ebhh jps ,ee
- g. gfde j1 l -
gy u. w o 's t ,m e g a, i s, u ,m , , e. j T g'l;y :t. 3,; e +h: t c m e ,,s F q., ; 'y(" aep e,~ _ , ,, , m ,, t a 4 1 o - :- oe sou oo 1. , t, . , , , , s ,. . - I$If
%aE
- t. dt f% T 4 's
- i) Pu % ; ; p, , , "$ ,.;
rec w o s. , WE 11 g (
}h, 'i VI N5t E IS $ @hM$ 53Y6N 'k. d$ .
i 7 lt 's),o. 2 ,
,2
. : . e 2: m '_
( ** . % 4% . 'p [ i kE.5 ~g
$, ,s q'Q l l; <
m ,.-3 .. S> i ?
&[d,%rp b w e . a;- a Y, a
e- 3..
,. a 1 .
s 4 G Td M a n ,, ? ;~ : a; u, g m.t)By.._. Al . y' , 9 a 1D ( .J , a.~. !
i 53?MNa@hh% %
'81C _ of s.
l:- o
, '\,- ; .,
s jy g
.n V v e. 9% L'
; 7 ,3 . . i .;
y .;..
) ', j,.
i ; ,k. f( u i Af(. ,_' J,' 3
,e i, i 1
_ _ 'd T
$,,,g.gJg I
.affEATiEE$$FECyl@@@l$MBJ%1 w
j mura a ~ s 3a r A ; ~ ev w , c ec ,3 s,ac f ' ' "' e d M o c n - g ,, ,,, p .. . . re ,, _. , ,, e { &y ) p[g:Es T;;~g% 5 1g r M
<<~ o u u , . s m , ,, , ,, , . ,
y' M l kyy 7,kg{ v ? ' 'v cae n n. c o o u .a t '\
.J qq -
tg]y
. e ~ a a u..e ..ra- g
% wi r< ,. ,s., w t.3 5 :s Ib - s:7 i k w a m m m za w z - - w , fg i
y$fg-$g[f-* 2 ); ' [,, i :
~
_.= -
- ~y y a.
' ~
D $ q; .f ' y,
+
k,
> t p w u p J [' '
3 , 1 i p'[ q g 5 l c.g;3 1 *.s e- . 9,1 T. : /
. af J,rfiIh y L't t j.&
t/J g eh.% a dl e t. . .
, $ k .
(j 1 3 h $ h g < f s u -
- b
4 0 W I e P9 M*
?
., 6og ' .
ac",up&se*S}A4
'"*wd Mig xsc(
2 9 '1
!,& ,1,f 7'd l 1 .
f4 M
%.0 y
")g!
y E;4 e
%wglgg
$splhbi@eap#
?
MedGPd6fM;tzEMgjg f
$1'lh,$n:af 4
R.3.\c,.aa N,ERI! .m
- v. .
,\ .
4 ~ _ '~ - . _ _ _ . _ _ _ _ _ _ _
~.
'=
6 l
^
l thg>Q gga. >) 4
; i I
r o-g,ig[ ' [* k ' E $. (. b
- y '
l
- p. -
~ -- .- .
l? i ~ $$ g!!$g$n y o. M ys , 4 .* T i rw/$$dbb ap b, $N
^
l e J ., I h h?$1 .
- ., 1 g t -
$?g "i .
5 .c .. s
.} v3 Wg -
.2 e
!.il>
i b t 1, h a >h ma
~
0 i L
^
f . t:e m m m E 2 m r.s 7 m ta a m urzsim m p ,g --g
.s t,...z_ u_
'g -'
s ga y r '
$,?$f'g4 "q497
~'-
'S
{, W
. &x.:ff ?* ..,
.qq P.;9;4 4
p? T x." 3 t
?
jd,..tsf;,ndh
.n ~ %- .g 3
a . t l $l
- .a nQ.:a*.a wa$,l$a'ER;re,,.a
- . i&dt*L2,,29.$.h?pa n
9il w JI,0 .w a; u<*; . , N.1 r*-
~k
, :y j 1
t weswnemoeswwn _. _ ;= - k ,; i my. . b C,) yd s i,ip R . a c S ArrrS C A P S c ae %i $x; ' i
,e s.
sg- - s
#4 *l ejs %
- v, *
, ( $ $
g . li, fQw: AL *si; L}Q N8I5 #
?
k_, - 9 --
, y T- cezz m
.hj' i
- fR '.
E-a . a {j t-3.y! m" x J
. , 4,{
7 gr x 4
' ;p 5 ,
*.a y
x 6 - (gg -
' ~
41 V g.
.#' w. ,.
- 2 .
e
'l.. ; $ ' f , .. -:Y- '
.A3ygg- 99
- & Ydynp - ,-
f f f 5. , h
*K'.,
'- .- ; , j_
[, w.' *
, f .s
, i s
- Se L. A , , , , , , M*f E% >
r l NCR 290-411
] Pago 10 of .
_ATEGORY 11 B0_01 C AUSE bNAlgyElg
SUMMARY
pG-1 GEARING DILURE I.0 ltilkEJ$ PLT _IEG DOCU4[;t[T1 NCR 290-411 INITIAT_Ilq DEPAETliEN U OEA INITIATORt KIDDER EXTt2093 EVENT DAT01 5/27/90 BEPORT DATE1 8/7/90 DEFINITIONS EVENT - a plant situation consisting of a sequence of actions that result in deviation from the. expected l It0 MAN PERFORMANCE - an individual's response _(appropriate or inappropriate) to the perceived needs of the situation INAPPROPRIATE ACTION - human performance that is inappropriate for the situation at hand CAUSAL FACTOR - a deficient performance-shaping factor-- There are 13 performance shaping factors: verbal communication, written communication, interface design or equipment condition, environmental conditions, work schedule, work practices, work organization / planning, super <isory methods, training / qualification methods, training / qualification content, change management, resource management, and managerial methods. ROOT CAUSE - the set of an events fundamental causal factors CONTRIBUTING CAUSE - a condition, while not right, was not, in itself capable of creating an event SLIP RING END - the end of the generator in which the slip rings are installed. This is the only obvious way to determine one end of the generator from the other. The Slip ring End (SRE) of the generator is located at the South end of the generator as it is installed at WNP-2. OPPOSITE SLIP RING END - the end of the generator _ opposite to the slip rir.g end. The generator thrust bearing is enclosed in the Opposite End (OE) which is the North end of the generator as it is installed at WNP-2. DG-GEN-DG1 - The MEL equipment piece number for the Generator portion of the Division I Standby Diesel Generator. The prime movers for this generator are two Diesel' Engines coupled in tandem. The North engine is DG-ENG-1A1 and the South Enw ine is DG-ENG-1A2. VIBRATION ALARMS - There are three permanently installed vibration detectorc associated with the Diesel Generator unit. One installed on t ach engine and identified as Excessive vibration Eng. No. 1 and Excessive Vibration Eng. No.2. The other detector is installed on the generatcr and ,
. l . l l
NCR 290-411
. Page 11 of . ,
I is identified as Excessive Vibrati.on Gen. TEMPERATURE ALARMS - There are three separate temperature alarm sensors associated with the Generator. They are the stator winding high temperature, the OE bearing cup temperature and the SRE bearing outer ring temperature. These alarm functions alarm a common flag at the Engine Control Panel marked " Gen Stator or Bearing Temp fli g h " . There is no way of knowing which of the three functions caused the alarra, there is no method of reading the , temperature directly and the areas monitored cannot be accessed by portable instruments.
2.0 JNTRODUCTION
The purpose of this report is to document the analysis of the bearing fallure that resulted in a minor fire et '; h e south end of the Division I Diesel Generator at the Searing referred to in this report as the " Slip Ring End" or "SRE" Bearing. The generator is a 4400 )N , 4160 Volt, 60 ilz unit manutactured by Portec (NEI) driven by tandem diesel engines. The thrust bearing located at the opposite end from the slip rings (OE) of the generator rotor is a double tapered roller bearing (see Page 16) made by Timken and is designed to accept the design thrust of the machine as well as provide seismic restraint for the generator rotor which weighs 23,600 pounds and rotates at 900 RPM during 60 llz operation. The SRE bearing is a cylinder roller bercing (see Page 28), manufactured by Torrington, designed to allow axial movement of the rotor so as to compensate for thermal growth of the rotor. The rotor position in the stator is fixed at the Timken (OE) end and the expected rotor thermal growth of about .060" is toward the SRE bearing (see Page 16). The scope of this report is limited to the failure and events leading up to the failure of DG-1 as reported by PER 290-411. In addition, associated problems reported by PERs 390-455 and 290-498 are incorporated into this document. A related problem concerning shorted rotor poles is addressed separately in MDR 290-490 Problems generic to the other diesel generators, ai noted, will be reported herein, however, extensive application (beyond the Diesel Generators) of the root cause and lessons learned to other equipment is not within the intended scope of the report. This investigation was conducted and the analysis made using available plant records, discussions with plant personnel, visual inspections, basic root cause determination methodology, barrier analysis, fault tree analysis, events and causal factors analysis. and selected portions of the
!! PES evaluation system. In addition, an event precursor l search was made, l
l
1 l NCR 290-411 I Page 12 of . 3.0 DETAIL @ NARR ATIVE QE IfiG EVENT2 On Sunday, May 27, 1990 the #1 diesel generator was 6 hours into the 24 hour run as part of PPM 7.4.8.1.1.2.5B. At 1742 while taking individual cylinder temperature readings the operator noticed a change in pitch in the sound coming from the engines. Thinking that the Main control Room had adjusted load he continued taking readings. Shortly thereafter, he noticed a different smell and heard "a rumbling coming from the 1A2 engine". He - proceeded to the control panel to see if there were any changes. While observing the panel the " Generator Stator or Bearing Temperature" alarm came in. The alarm was silenced anu the operator called the main control room to report the alarm. While on the phone the operator heard the engine start to rumble again and the " Excessive Vibration Generator" alarm came in. The control room operator tripped the diese) generator and at the same time two maintenance men working in the engirm room reported a fire in the engine area. The operator reported the fire to the Main Control room and then proceeded to the engine area where he noted " fire coming out of the generator bearing of the 1A2 engine". The fitt was immediately extinguished with a dry chemical fire extinguisher. The fire re-flashed one time before being completely suppressed. The unit was shutdown and tagged out by the operators and no further work was Jone on the machine. On 5/28/90 a detailed inspection was initiated of the generator and engines for signs of damage or causal factors. The following conditions were notedt
- a. The . SRE bearing oil level was ainu t 1/4" below the low mark on the oil level sight glass (see Photo Number 4).
NOTE Without knowing exactly where to look for the high and low level marks on the gauge glass it was nearly impossible to deiermine where those points located by viewing the gauge glass through- were the observation port provided in the coupling gurrd,
- b. The oil in the SRE bearing oil sight glass showed two distinct colors of fluid present in the sight glass (see Photo Number 4).
- c. The SRE bearing thermocouple appear ed to be kinked at its exit point from the thermocouple well (see Photo Number 6).
- d. The OE bearing oil level was about 3/4" below the low mark in the oil level sight glass (see Photo
NCR 290-411
. Page 13 of .
Number 2). NOTE The OE bearing oil level sight glass had three different marks ground into it, the highest of the three was the intended " low level" mark as determined by actual measurements (see Photo Number 2),
- e. The OE bearing bracket dowel pins appeared to be bent or bcnken.
- f. There appeared to be an excessive amount of oil on the floor and structural parts under or near ' both ge..arator couplings.
9 Some smoke stains and evidence of fire extinguishing agent were noted on the SRE bearing cover.
- h. Small caution signs posted adjacent to the oil 1cvel gauge glasses (see Photo Numbers 1 & 3) were obviously noticeable and agreed in . content (on proper oil levels) with similar direction contained in the PPM being used to operate the unit, however, neither agree with the exact directions contained in the CVI Manual 02-53-00,68,1(1-1); Drwg D09236 which indicates that the high and low level marks are 3/8" apart.
A review of the operators log revealed the following information (quoted word for word) had been recorded just before and during the event: 5/25/90 0300 Completed adding 21 gallons each to engines 1A1 and 1A2 of Nalco 2100 Corrosion Inhibitor 2000 Performed PPM 8.3.126 on Div I Diesel - #2B1/1 air motors failed _to start and engage. 2200 Attempted to-locally start Div I Diesel for 10 min flush of oil systems - failed to start -maintenance looking into problem. 5/26/90 1805 L/V Div I diesel for local start for 10 min flush of oil system. 1839 Attempted to started Div I diesel per 2.7.2D
- diesel failed to start - DLO-P-11A1 tripped on thermal overload and DLO-P-2A1 failed to pick up Lin standby.
NCR 290-411 Page 14 of . 2150 L/U Div I diesel for local start per 2.7.2D i 2200 Locally started Div I -diesel - Engine ran for 10 sec. and then S/D for no apparent reason - Maintenance looking'into problem.- 4 5/27/90 0125 L/U1for loca) start 0129 Started Div I diesel successfully.- ;
-t 0135 Sec Div I Diesel -0145 Commenced PPM 8.3.126'- D01 Air Motor Test 0203 Completed PPM 8.3.126, L/U for; auto start -
0443 Started DG#1-per T.S.S 7.4.8.1.1.22 0510 Eng oil Iv1 ok 0510 Loaded DG#1 i 0617 Went to idle per step 48
.- l 0630 Eng oil-Iv1 ok
'I 0635 Stopped DO#1
" Low Air- Press" alarm due ,to Press SW Isol.
Press OK-0515 Added water to.DCW Exp. Tks. 1227 Started Dg#1 per-LOP /LOCA Test 7.4.8.1.1.2.&3.'" Low Air Hdr" Pres Alarm. Started' Air Comps-to 2458 1255 DLO-P-11Al' tripped. Roset. l 1402 DLO'P-11A1 tripped.. Reset. 1305 Started 2 Hr. test run i 1513 DLO-P-11A1 tripped. Reset. 1640 DLO-P-11A1 tripped. Reset. 1717-DLO-P-11A1 tripped. Reset.- : 1743" Received " Generator' Stator or-bearings temp" alarm:
- called . control- room. -
while- on phone- with
. control-room " excessive.-vibrationL ~ gen" came -- i n .
Control room punched out the diesel = - wlille ; m'_ ' m.m- _m-- --
NCR 290-411 rage 15 of .
, i i
operator was on phone with control room maintenance men working in area discovered fire in the inboard bearing of the 1A2 engine. Fire extinguisher
. (chemical) was applied and fire was suppressed.
3.1 LAIT,URl bNALYSLE NOTE Reference to the diesel ger9rator sketch on page 16 will assist in understanding the following descriptions. On 5/29/90 disassembly of the SRE bearing was initiated to determine if the bearing could be repaired or replaced without removing the generator from the building. After the coupling guard was removed an attempt to check generator-to-engine alignment determined that the generator could not be rotated by application of reasonable force. Rough measurements indicated that the generator rotor had dropped approximately .031" and was renting on the bearing cover labyrinth oil seal. Because of these conditions, it was not possible to determine if there existed any misalignment of the generator to engine prior to the failure using post failure diagnostic techniques. Review of the alignment records af ter completion of the last alignment made in 1987 indicates that alignment, at that time, was well- within the design limits. Once the SRE bearing- cover and bearing locknut were removed it was noted that the generator shaft was mis-located about 1/2 to 3/4" towards the OE and that the SRE bearing surfaces were badly spalled. The bearing separator (see Photo Number ) was fractured in several places and was removed with the fingers. Once the separator was removed it ras noted that the level of the lubricating oil in the bearing housing during cool down of the machine could be determined by the presence of a carbon residue line established between the very hot air space and the residual oil where a high oxidation rate has occurred-and left a film of gelled oil remaining at the interf ace point. This level was recorded by photographs (see Photo Number 15 and page 35
). "
lo damage could be seen on the shaft sleeve under the inner ring of the bearing so attempts were made to remove the bearing. The outer ring, rollers and separator assembly were successfully removed. It was not possible to remove the inner ring and it was determined that further efforts to do so would probably destroy-the shaft sleeve so work was stopped and preparations were made to remove the generator for repair in the shop. The generator was removed from the building and shipped to a i
~ . _ . _ . _ _ _ . _ . _ _ . . _ . _ . _ . . . . . _ . . _ . . _ . ~ . _ . . . . . _ . - _ _ . . _ . . . _ _ . _ . . . . _ . _ _ . . . _ _ _ . .
1 _NCR 290-411 Page 16 of . 1 i repair shop in Spokane, Washington where it was' disassembled and repaired. During the disassembly further damage was , found within the machine.- 1 Attempts to obtain thrust readings on the rotor determined that the rotor was frozen in the axial and radial directions even with the SRE bearing removed. When the OE bearing housing was removed it was determined that-this bearing was . also badly damaged. The locknut for this bearing-was located about 1/2" from the bearing and the shatt' side tab' was . sheared off which had allowed the locknut to rotate. , The J bearing had turned on the shaft as well as within the bearing housing.
'1 The oil levels determined by observing the carbon residue' line established between the very hot air' space and the-residual oil where a high oxidation rate hr.s : occurred and ,
lef t' a film of gelled oil remaining at the interface point i indicated that the rollers of the OE bearing had not been-immersed in oil at the time the machine. was shut down (see Photo Number 8). It was noted that the "O" ring groove ' machined into the OE inside and outside bearing covers (per design) also had-ta ~ matching groove - cut- into _the bearing: bracket (see Photo Number 7). These " extra" "O". ring grooves-do not show - on the construction drawings. Calculations performed by Generation Engineering indicate that = the-."O" ring was only crushed about .002" with the:" extra" groove cut
- in the bracket and the "O" ring would not have.' been an-effective seal under these conditions.-
O-Y
-.-,----.wm r,-a-o,--..-.--.m.--w-,-- . - - - m e.. . ...-e--.b . . . - , . . - , , , - - - . , - . ~ ,w.,4 y , -# r- < + - . - ** =M ,'..e
l NCR 2 90- d 11
.J ..1
. _. S 1 _ G 1 N,1 R A - R e a a e .t e. 0 ,-
l 1 j
. * -l) fjAM[ . INSIDE '
PEARING EEARING BRACKET CPPCS:TE END COVER INSIDE OUTSIDE BEARING I t. "ING OUTSIDE IEAg]NO SLIP COVER / OVE: COVE
, EEi S
)
i
. , . ..e h ROTOR VINDINGS
, (y c=1- , ,, _, - "
l- i o '"
~~
t l
- - _ /
N _ / \
. SHAFT p
dck %I i l e a-1-I - b 1 l
% . _< - t _.-
_ _r "- ' "
'[ NJ -
,t i
~o \- OIL SUMP O!L SL %
DOUELE 'O' RING G ReoVf- Gil SIGHT ET All GLASS CIL 5:3HT TD -- _y GLASS _ CD ) g
, [- -
CRADLE x1 t DIESEL ENGINE COUPLED TO BOTH ENDS
'/ 'g f s'~f ' ' -
'. .' .'a L.L iff :)
'//_/O,/,///V//,////
m ,____ ,_ c _
?
3 _J' 'L i m*: - - r...... f!
- c=m].,@ -ew-e-
\
_ m : ..
,s'$< )
. !$$ h;I3h E
[ew..- 2"%='-a '"'; ,(,ig%;> g 4,p~uunuw.r . . .. . .y,,r ; ,
,,/,,:
m.um,,:,4 n . g s -
- e- r csu,'.,/, , /, ,
e- ,- - , , , -,,- , - - ,- y - - , -
4 NCR 290-411 Page 18 of . There was a significant amount of oil present in the generator cradle . cavity and a light coat of oil present on the rotor and stator windings. It is considered possible thet some or all of the oil on the windings could originate at the air start motor exhaust hoses that exhaust under the engine couplings at the rear end of the engines. Most certainly, the large majority of the oil noted on the deck in this area originated at the air start motor exhaust. The oil in the cradle cavity is considered to have originated from opill over of tne OE bearing sump through the labyrinth seal during events of high oil level and/or foaming of the oil in the bearing cavity or through the defective "O" ring seal. Measurements were taken to determine if the marks scribed on the bearing oil level gauge glasses were in agreement with Drawing D-09236. Both low level marks on the glasses were within a very short distance of being correct, if one discounts the several incorrect marks filed into the OE bcaring sight glass frame (see Page 39). Results of an analysis of the lubricating oil by an independent laboratory (see Page 63-74) disclosed the following information: SRE BEARING Many particles show evidence of high temperatures generated at their formation. Bronze particles are from bushings or bearing cage. Heat affected particles and friction polymers evince high load / stress and high t erope ra t ure . Severs sliding wear results from breakdown of oil film at moving contacts. OE BEARING Dark oxides indicate insufficient lubrication. Roller fatigue is indicated by laminar particles. The observed two phase solution in the SRE sight glass (see Photo Number 4) is the demarcation line between the oil in the sump, which had been severely overheated, and the relatively cool and un-oxidized oil remaining in the sight glass. Because of the extensive damage to the OE bearing it was necessary to remove all parts of that bearing, including the locknut, with a cutting torch. There was extensive damage to the shaft under the inner race and the threads for the locknut, all of wnich had to be turned down and rebuilt with welding and machining. Both bearing thermocouples were shop tested to determine if - 1 l
. NCR 290-411 Page 19 of .
they wert open. Both thermocouples indicated good, however, it was noted that both the SRE and the OE thermocouple leads were not connected according to the vendors drawing (see Page 30-32). Review of the applicable drawings indicate that the alarm function for the SRE and the OE cearing alarm would have been inoperable. As such the alarm received at the time the machine wai shut down would either have had to been spurious or f ron. the stator winding thermocouple. ANALYSIS OF SRE (TORRINGTON) BEARING FAILURE Examination and analysis of the SRE bearing disclosed that the separator (bronze) (see sketch on page 27 for details of bearing constructioni had been rolled into the roller track and prevented seizure of the bearing (see Photo Number 18) . The rollers showed long term axial thrust application indicating that thrust had been applied from the direction of l the generator opposite slip ring end (see Photo Number 19), i The tapering of the rollers (see Photo Number 21) is indicative of gradually increasing axial misalignment of the rollers to the inner ring which reduced the active bearing surface and extruded metal towards the unloaded section (see Page 28). Because of the absence of very high temperatures this extrusion wculd have taken a relatively long time (tens of hours) to occur. The build up of metal on the OE side of the rollers (see Photo Number 21) is a combination of outer ring shoulder metal as the result of extrusion mixed with portions of the incorporated parts of the bronze separat'or. Fault tree analysis (see Page 25) of the- SRE bearing discloses that the failure mode of the bearing is. fatigue due to overload beyond the designed capability of the bearing to accept axial thrust as exhibited by the spalling condition of the outer ring (load bearing element) and the tapering of the rollers. This finding is confirmed by the bearing consultant who independently examined the bearing (see Page 40) and the lubricating oil analysis (see Page 63-74). ANALYSIS OF OE (TIMKEN) BE?. RING FAILURE The loaded cone, rollers and cup (see page 26 for nomenclature) were all badly burned and the rollers were flattened on one side (see Photo Numbers 9 & 10 and Page &
). The cage was broken in several places and the cone ribs were extruded in the direction of thrust (towards the generator). The unloaded cone, rollers and cup were dented (cee Photo Number 14) as the result of introduction of foreign material assumed to have originated- at the loaded face) and showed signs of fluting (see Photo Number 14)
(electrical arcing while the unit is rotating) which occurred after the bearing failed and probably occurred during coast down of the machine after 't was tripped. Fault tree l- analysis .(see Page 27) discloses that'the failure mode of the l OE bearing is lack of adequate lubrication as exhibited-by (_ the burned and scored rollers and the absence of indications v . . , . . . _ . .
~ i NCR 290-411
.Page 20 of .
l l of f atigue or excessive long term denting. This finding is I confirmed by the bearing consultant who independently. examined the bearing (see Page ' 40) and the; lubricating oil analysis (see Page 63-74). After the rotor had been .romoved from the stator it.became i evident that there was some moderate damage to - the end turn " wedges (see Photo Number 22) and one top stick (see Photo Number 23) according to-the.Portec (NEI) representative the damage to the stator was the result of either slipping poles-or synchronizing the machine out of - phase. The damage seen i is not considered to be a result of the: bearing problem nor a possible cause of the bearing' problem.- It was also noted that all of the dowel pint. holding the OE bearing bracket in place were either sheared of f or badly bent - (see Pages' 81 & 82). This condition was determined-to:be the result of the bearing failure as there was no evidence on - the OE bearing. I that would indicate that it had been run - for- an ' extended period of time in e angularly misaligned condition _ and the, ' f allure mode for the dowel pins was . determine to be ductile , f ailure indicating a singular overload event .as opposed to' a - fatigue failure created by vibration oryrepeated minor events (see Page 77. ; During reassembly,.it was found that two of .the rotor _ poles b i were shorted turn to turn and it~ became necessary to rewind tlie rotor poles. This condition is not considered . to be related to the subject of this report and is addressed' in' MDR a 290-499. i 3.2 EVENI AND CAUSAL EACTORS ANALYSIS j A search of the operating logs - and maintenance history for i-the machine was conducted to- establisht the . event and -causal factors analysis. The findings are listed asifollows: L i CHRONOLOGICAL RECORD'OF ALARMS AND MWRS: > DG-1 # 1 3/4/85 MWR AX7810 OE# bearing # has oil' leak, re-torque and refill. oil ' 9/17/85 DG-1 Logbook OE# bearing oil went low after engine was. running 3/20/85 MWR AWO934 OE# bearing has oil leak,~replece "O" ring 3/10/86 .MWR AV4392 OE# bearing oil low, added-oil
~
5/9/86 .DG-1 Logbook major overhaul? r
~
12/8/86- MWR AV8063 OE# bearing; oil.too'high, drain down I 4
-- - ,. ,_ .,_.,.,.,, _ _ m._, , - , - . . . _ . . . . _ - - ,,.m..m - . - _ - - - ,
NCR 290-411 Page 21 of . 6/3/87 MWR AT0236 OE# bearing sight glass has no reference marks on it, rotate glass 180 degrees 7/18/87 DG-1 Logbook OEN bearing is focming, shut down to drain oil 8/11/87 MWR AT1234 OE# bearing has low oil level, add oil 1 9/25/07 MWR AT1'/44 OE# bearing 1/2" below low mark, add oil 11/6/87 MWR AT2189 OE# bearing low oil IcVel, "added oil, a good amount of oil beneath bearing indicates leakage problems" 11/30/87 MWR AT2434 OE# bearing oil is too high, also there are two sets of marks on gauge glass , 12/2/87 MWR AT2522 OEN bearing was leaking oil because level was too high, drain some oil 12/13/87 MWR AU8811 OE# oil below lower mark, add oil 1/5/88 MWR AT2870 OE# bearing oil is low due to , leaking sight glass i 1/7/88 MWR AT2910 OEN bearing oil 1cvel low due to slow leak 1/11/88 MWR ATP961 OE# bearing low oil level -- Voided see MWR AT2910 1/P1/88 MWR AT3138 OE# bearing low oil leve), added one pint 2/25/88 DG-1 Logbook Number 2 engine, excessive vibration alarm 2 5/23/88 DG-1 Logbook Trip on 86 lockout 5/24/88 DG-1 Logbook Gen Stator or bearing Hi temp alarm 5/25/88 DG-1 Logboo!; Gen Stator or bearing Hi temp alarm , 6/13/88 DG-1 Logbook Gen Stator or bearing Hi temp alarm 6/27/88 MWR AV1765 SRE# bearing low oil level, added to mid point 10/25/88 DG-1 Logbook Added oil to OC bea'ing r
l l NCR 290 411 Page 22 of . , 5/14/89 DG-1 Logbook Gen Stator or bearing Hi temp alarm 5/18/89 MWR AS0381 OE# bearing has low oil level, add oil 6/8/89 DG-1 Logbook Number 2 engine, excessive Vibration ' alarm 6/8/89 DG-1 Logbook Number 2 engine, excessive vibration alarm (second alarm, same day) 6/14/89 DG-1 Logbook Number 2 engine, excessive vibration alarm 7/2/89 DG-1 Logbook Number 2 engine, excessive vibration alarm 8/12/89 DG-1 Logbook Number 2 engine, excessive vibration alarm 8/13/89 DG-1 Logbook Number 2 engine, excessive vibration
- sealed in, would not clear. Added oil to both generator bearings to bring oil to proper operating level.
11/20/89 DG-1 Logbook Number 2 engine, excessive vibration alarm - would not clear ' 12/15/89 DG-1 Logbook Number 2 engine, excessive vibration alarm, cleared after 15 min.
# Due to the configuration of the machine and a lack of a common language, the Opposite From. the Slip Ring End becring found on the north end of the generator is referred to in many different ways by operations and maintenance personnel.
In this report it will always be referred to as the OE bearing despite the description used in the referenced document. The. Slip Ring' End bearing will be referred to at the SRE bearing. 1 l l l I
,r, - , -v, , ,_.eer, , ~~,-.v n- , ,s - , - - , - , , - - - - . ~
pact 73 of EVENTS AND CAUSAL FACTORS i I
.. NCR 290-411 -
WLEt$ CTRERV18E INDitATED
"" ALL EvDTS APPLT To DE PIMlWO 3/4/85 9/17/45 LCV 3/20/85 OIL 3/10/46 OIL 12/4/48 CIL I PALL LLLM ,
- OIL A n13 -+ LEAX, REPLACI -e LEVEL LCV,
- 100 MICH, -
A00 C!L CE DolxE RVN 'O' IING ADP CIL DMIN 6/3/47 NO 7/1%/47 Clt 4/11/41 Clt 1/27/47 CIL 11/6/87 LCY
.-.+ P.17 . P.tM S -. P C A P. I NO , - LCV ADD
- t/2' LCV, -* O!L ADD CIL -
CN CA0E TRAIN 8 05.E CIL ALD CIL Olt ex FLCCR 11/23/47 12/2/42 O!L 12/13/47 1/b/44 O!L 1/2/44 OlL
--+ Clk 700 - 100 H!OM, -* Clk 100 LOW
- LOV, ADD ~+ LCT, ADD -
E!OX, tu!N CRAIN AD Clt ett CIL 2 PAAXI CN 00 00 LEAXING SMLL LEAX
',$ 1/11/44 clL 1/21/84 Olt 2/21/44 k0 $/23/84 $/24/44 CD
'l >" -+ L;i. CD -+ LCV ADDD I -+ 2 DO HI -+ TRIP CN 14
- hl TI.'t? - -
- m. V C 1 D P!NT V!LufICN LOCXOUT ALV..
$ /21/4 4 CD 6 /13/44 CD 6 /2 7/44 3RF 10/25/44 l/14/41 C D H I T EP.r LCV OIL,
- Let O!L A0D -+ HI TIrJ -
-. M: 7 I:".P --+ +
AL.V ALAF. ADO 10 MAX Olt ALRfl r-t/14/69 LCV 6/4/41 NO 2 6/4/49 NO 2 t/14/e' =t 1/2/41 NC 2
~+ C l t L LT.J. ,
- D<0 M I -+ LVO HI + 2 DC . -+ bt HI -
CD O!L VIEuTICW V1312xD TODAY VlBAATICX V12AATICN 4/12/41 NQ 4/13/49 N0 4/13/49 ADOED 11/20/81 30 12/15/49 NO
-+ 2 DC HI -+ 2 DS HI -+ clk 70 BOTH -e 2 DO HI -e 2 bt HI -
V1 T.?AT 1:N VILuTION ET.MINos VIEuflCN . VIEuTICN VCW'T CLEAR
, ,2, , , , o m . ,2, , , 0 0,N ,,2,,,0 D, ,,2,,,4 s.c:
x - x! m.r - H: vis - XiPPro -. ritt eX sax BEARINo ALM l Page_ f
l l l NCR 290-411 i Page 24 of . l 3.2 PRECURSOR SEARCH In addition to the Event and Causal Factors Analysis a review of past failures of the Standby Diesel Generators was made to determine if there were previous failures that could have been caused by the same conditions. The only other breakdown found to have been caused by the same type of failure mode was at the vendors shop during the " Type Qualification Test" of the sets when the OE bearing failed due to lack of lubrication, reportedly caused by allowing the oil level to i drop beluw the low level mark, specified as 5 7/8" measured from the shaft center line (see Page 35). Review of vibration readings taken over several years indicate no evidence that would lead one to believe that a bearing failure was about to occur. Discussions with vibration experts at NWL disclose that our methods of measuring and recording vibration readings would not normally be expected to provide indication of impending failure of a bearing under the circumstances we have experienced. ,
4.0 CONCLUSION
S
- 1. The SRE bearing failed due to extrene axial misalignuent caused by applying thrust from the opposite end of the generator (rotor was thrusted towards the south) which lead to a classic overload failure of the bearing. .
(see Page 29)
- 2. The oil level in the SRE bearing was high enough to provide adequate lubrication to the bearing during and after the failure of the bearing despite the fact that it was somawhat below the prescribed low level mark (see Photo Number 16 and Page 39).
- 3. The application of unacceptable thrust to the SRE bearing was caused by failure of the generator thrust (OE) bearing.
- 4. The OE' bearing failed due to two or more lubrication deficiency events, the last of which lead to total failure of the bearing.
- 5. The long term leakage of oil from the OE bearing cavity coupled with an unacceptably restrictive operating oil level band (3/8") and the propensity of the oil to foam if the oil level is too high (greater than 1/8" above the low mark) set up the unit for eventual failure.
- 6. The OE bearing oil level sight glass has two marks ground or filed into it and one mark cut into it with a hacksaw or similar instrument (see Photo Number 2) . The highest of the three marks (the hacksaw cut) was the design low levc1 mark, however, there was no way of determining this -
i NCR 290-411 Page 25 of . other than to measure the mark from the center line of . the shaft (nearly impossible to do without disassembling the coupling- guard and the coupling) or to ask someone I who should have known. To rely on field interpretation- ' of such a critical parameter, with no other problems present, most probably would- have resulted in eventual lubrication insufficiency at -the bearing.
- 7. The long term leakage from the OE bearing cavity was caused by an improperly machined "O" ring grove in the OE bearing bracket which precluded proper compression of the "O" ring and allowed a slow leak to exist, both when the machine was running and when it was-idle.
- 8. The vibration . alarms recorded over the years are.
considered to be caused by our' inability to properly set the switches. The - switches are not sensitive enough to predict an event as experienced. Therefore the series of vibration alarms . received. on Engine 1A2 are not considered pertinent to the analysis of the failure of the generator. *
- 9. The temperature alarms associated with the generator i bearings should have given warning of impending bearing failure and allowed shutdown of the unit before extensive-damage was incurred.
1
- 10. There appears to be a marked difficulty exhibited' in communicating long term problems between the operating staff and the maintenance and technical staff.
5.0 RO_OT CAUSES i
- 1. Equipment -
Manufacturing Error - Not made- per design. The "O" ring groove machined into the OE i bearing bracket of DG1 was not in accordance with the design _ drawings and ultimately lead to failure--of the generator bearing.
- 2. Personnel -
Management- Programs . - Failure: to heed precursors. Failure to communicate and act upon .long term problems of. oil leakage f rom the bearing and difficulties.in ~ maintaining levels in the. bearing reservoir help set up the conditions that lead to-eventual oil starvation of the-bearing.
- 3. Equipment -
Design Deficiency - Loss of moni to' ring : Alertness. The . temperature alarm , instrumentation, the vibration alarm instrumentation.and the oil. sump level indicator s on- the ' generators are all -somewhat deficient in t'neir ability to perform the intended function of. alerting the: operating: -- pe rsonn el to'- abnormal or off standard conditions.- i
,.mn,mn.. c. . ,- ,. , ,a.n-wsnm , ,,. c. ,..m -
,an.n. , . . , . . ,+ ..,-,-.y+w, , --
NCR 290-411
, Page 26 of ,
- 4. Equipment - Design deficiency - Specification less than adequate. The prescribed 3/8" operating band for the bearing oil reservoir is impractical and nearly impossible to maintain.
6.0 GONTRIBUTE G CAUSfa
- 1. The choice of a lubricating oil in the diesel generator bearings that has a tendency to foam under operating conditions for which the unit is designed restricted the usable oil level operat.ing band to about 1/8".
7.O REgottlENDATIONS
- 1. Perform an engineering feasibility study and cost analysis of the Diesel Generator bearing lubrication which should include, among other things, the following aspects:
- a. Incorporation of a forced lubrication system for each bearing that would include a pump, filter, cooler and low level alarm.
System should be designed to fail ~such that an alarm will sound upon loss of flow and the oil level in the bearing H at the presently prescribed operating level to allow for corrective action prior to bearing failure.
- b. Investigate and provide, if available, an oil that hae less propensity for foaming for use in the bearing lubricating system OR increase the capacity of the sump so that the allowed 1/8" operating band represet.ts at least two gallons of oil-OR provide specific written direction that makes foaming oil the preferred method of operation for the bearings.
- c. Incorporation of an oil temperature thermocouple, indicator and alarm for each bearing,
- d. Provide a method for adding oil to the bearings-with the unit- in operation, this operation should be accomplished without jeopardizing the life or health of the person performing the task.
- e. Provide an' accurate method of determining the actual bearing oil level commensurate with good human factors engineering 1
i i NCR 290-411 l , Page 27 of . i . ! i principles.
- 2. Provide'an indicator to display bearing metal
. temperatures- monitored . by -the - presently installed metal temperature thermocouples at_~both diesel I generator bearings. Establish normal, alert and alar'm temperatures for inclusion into plant . I l procedures.
- 3. Remove -the " extra "O" ring. groove" _from the I outside surface;of the'OE-bearing bracketLof-D01.
- 4. Investigate the presently. installed . vibration det.ectors on_the diesel generators and the coupled d
engines. Either provide improvements that would make them = a viable method _ of___ detecting incipient problems with the units, replace - _them :with - a better unit, or delete them as a source. of f alce~' security. 8.0 RCA PREPARED BY: KIDDER- , i TEAM MEMBERS: W.S. DAVISON _ S.G. WILMAN J.C. MOWERY T.E. ALTON H.P. HUSTON _ C.H. GARREN.' w
- BEARING - NOMENCI.hTURE
* . .Page no of BE ARIN G_ WIDTH (T) _
4
$)F jg '
' "('t CUP DACK FACE R ADIUS(r)
N ' ' - CUP FRONT FACE RA0lus ' n CUP DACX FACE 3 C)JP \ys
% CUP FRONT FACE CAGE ,. . / CONE BACK FACE RIB CONE FRONT FACE RIB
%l ROLLER
+ CONE B ACK FACE j
! y,5 --CONE BACK FACE RADIUS (R)
CONE FRONT FACE : /j ,0NE C
/ h CONE FRCNT FACE RA01US CONE WIDTH (B)
- g -
3 - w
.c:
, ..;s
..,, c:
y o C L9 6 , u s 2 o o Ec a E O ' J c. D
/// /
s
, I
[ I l l
~
y N - CAGE Cl.E AR ANCE l ' Reprinted thrwgh the cour .esy of The Timken CoreFJ j
-)56
- , = . . . - . . . - - - . a .. .. -. - . -. . - ,- , .. . _ , - - - , , . . . . - . _ = . .
in> eso._ is ., I STRALGHT ROLLER BEARING W10TH _ yO. D. CORNER i OUT E R Risc -,,- fr ~ ';4.' ROLLER - INNER RING ngi- a m=- l
'-k ~ f$
'a . a c:n .
- 7. . . pum
;'n f f _ w -
wsOaC cuaER
- i ' .-'.
"s. *s&V ouT s:cE f
; -'S ;
OI AM ET ER -' ..- 1 [,p d' ' d_ SHOULDERS i BORE [ ,',
"m'r-1 b -m":+9 a . . 5g., -
t {\ .
- : N-gg'gg' .
, fdI44 SEPARATOR
' ~ '
4- ' M*U T FACE 1- _ . Ti s { _ _ . _. __ 4
. . - :__ . . . . i. .
l h BEAR]NG SRE lj ' RING F ALLURE MODE- NCR 290-411
- DVER -
PAGE S Of' s EE ARING BRprii.t T
/ / / / / / / / / / /.J / / / i ,
- 9. . a i
[:95+tm DUTER RING I 37r: RE ROLLER rA' UR- - i j e .m . l INNER RING s NNNNNNg%ANNNNNNNNNNNNNN
<Q .
'////////////// i hj,.;r mi fG8:M '
b aN 1 7
? F AlUR E u - -
L l-
$NNNNNNN g $%NNNNNNNNNNNNNN
lit 11ll Pa;t : # g_. ull ga
, 1,
, _ __ g I; 1 l' Os .i i g -
.c I
id
!I '
i
.j,il
_. e sis ii . ii .
^
.~
i
'? 1 y i '
.j y, ),e ! I 't s .
~
I ' g.l l
}la{
alf.' i rgt j. y ng _ ,, 1 ... . 1.; 1 qk 1 xy - - - . ys If '
,ji l Il 1, l
)~
= iljt
.g; i-ill
- 1 ii- !o::.,.i %o, t&
ilh 7R.. 3 e , 'l I . i i e mp ~ y, ij .
!! l !. -.
4
- 4. i .
k.f k 'lj ets _ 1 lQo i ___'l-
.maas tgo$
i$ i z lk)'i
}
' j .J__ '
o *s'4 a h Z {s'io u ., - 5 I" l T ~ c ..' I., 3 i.s, m . o m I1 i fii i , to D ~ ~
< t O l. 'a l [
. F. -
l .- O ![ LI e .
. 9 n {' ,
11 m
~
!! ( !! .I i ,
l.. 5
$ l-<. 1= I o.
NCR 299-411
' :tn rtret crrecs.cmstwriuc t<a@: Duett "** ** #
r et stat:R 31 [p~~fy f r1 r9 m I i l 818 fil ' l 'l 'l 21 'l 8161 'l 'l *I $$ "I 'i $5 8fi4'I3 @ ,
.A [fI *i 31 al 1
as cco 37 97 96 ts a 18 "2 f~.i ~ l ']
- [ J. l .1 ls%.co fi's [ ser j I
, , , ,1 [ t j i ] co-a [ t j ij tio3[~~rT '~1 b b_ o o 5 o O Y Y 4, p HH q% TO REMOTE ANNUNCIATOR 8
- 4 si-
- -__a t
5:T f ----
. 0-M h-~ -d h-TO LOCAL ANNUNC1270R RETE:ENCE 4-EvA-3 R2
[*543-EH2 L R20 STANDBY D!ESEL GENER/. TOR E543 CH4 P10 E542-EVA-3 R6 GENERATOR STATOR AND 3C HING TEMP e gga;.gy3 7 pc ALARM C]RCU1T A S FQUND
.h
.)
NCR 2%=411 ti.4 cttet careta ccustattwt tatsmocovett DC STAT,CR [1AC,
. r r,- --
- r 1: 1 81 'I '] l i l
- l
- l '] ['18121'l516l2181'l59"liiiiiii'i @
97 96 37 i {~ ~ 7 l 0 l ( lag g 34) l T1 3 tI 12 (~ % l 9] c: s [ t j i ] uo r [ e j i _j no 3 [ a l ( _] M 4 + TO REMOTE #MJNCIATOR lH HH jH-b
$s$ m
!b !b !b TO LOCAL AN'MtCIATOR
' REFbkENCE E543-0VA-3 R2 E543-SH2 R20 STANDBY D]ESEL GENERATOR E543 SH4 R10 E543-2VA-3 R6 GENERATOR STATOR AND ICAR]NG TEMP E540-2VB-7 R9 ALARM CIRCUIT A3 DESIGNED _, W --
$~ . i'- u.
Page_ '2- _of .' .
- ___ l
! xwa /:0, l ttrERM/RETERM OATA SHEET 650 b'A 3 hs $6@ l l Drawi'ig No.' ( Plant l E 13 JH4 A', l IMPL/MEL. No. W d r.t j 6 )'t. is_._ kg. _M' b :l l Cletrance No, l i 'Q I %-ded- Mi
- d. Iel e en>4f,hyL l 3h -5 M 2- _l I
.a.e.n ak x I Location
/I PW1,!p i def3/ '9hSk 1%
l N~__ V A g r;vil l M nitial . RE7its l t;:s _? - c _ l'/e ri f ied l l lV6rifiUl l linitiajj By Ev i Cceren's l
+mi.a'. Nc. linit'all
'M4 rwm ,* / L l l l '
}.
l I ff.lI 20 5 l 4' . J ., j t'A WICA &ns tM,
/M v,N u
4 I l l l l (
.G"}4:(W M ST
?g l' y, )
.Q, > l l l ~~
4 3 l'.uu<G/.w - l ; tY-" j //8 l
~ -
y{$fyI-{i j . I
- i f
i $%d .u<C 8xM,2: -TU l l -l
.i. I
,,~
w l l e l h/4,7 \ W ~C,~<,S n,,w m,!Cll dh~kb
.I v-- ,.. !
l l f I Q'- \ y/g@y,9y,fx 0%g n lj l i n'y
, n ., I I
ib A, (.2W w-- i6-(* c l
/Wm 6 J a .:..I l
I l l sgp
.1 '~~ i .a '
l .I l#'1 l I // J _l I Nc4ITMd7 4.& I En2 r.4*. l l l .l d, I ~ 2 1 l l l '/ l l 1 l ] f I l i 2 l i I f I J J l l I I 1 l I l l 1 l I I J. I 1 I I I I I l I I I l i 1 I I I I l 1 I I l i I { I I I I l l I I i __l I I I i i i 1 I i I l _l l l I i i_ l 1 I i l i _l l I I l e . arks , Ac.just .ents or Additional Da*a fis Necessar/: 4 bu} l b r_(L._ l
% ds t 1 \%\~OC0v5-Q1 Rb komee ~1k'e- ,
n> tl
&b; KTB M. N/6( boa PlW4m . I
_ Reviewed By y 6fm'ThVle.( / O 5 M / 7~' l l Attachment C eactnvueca g ens,cnnuwsem t
~
6/ IJ$ Pa > b h8h,
* - Odwe[Ainan 4vur'- 'h W' E $43.S' #- )
y'8 ( 'D 6 1 Pe,e g 4 i
,f j
\d li-3
/5 l -
i t4' . t!*'*
- p s 7
- l. Lg --g2L
< ^ " '" e , .4 W> '
s
' E44w -
c e o < [ f *\[, ,e> if, ,. .,,, 7.c . . --]
)
I f'l I g$alSi 1 lDG/ 9's - lD%t~b1 ,
-), n . ' :e~;'
h, g, . . '
\
',b -
- " ...e tp.A sw.o
\
J
,423. <~ .- (
dc:4 - a
t . . .. Ep !97p :ph'.h'sQsy5 7m Hig A J.e,g1:* i +. 'dh-k;:d w g
.. ~ y;ft
";; y;Fl,*g.K yqQ)@Nd+g)N? L.s,q,5MWlf.fy&hX,]:.EW9;5
'*'.s.:i s.
3 -db 4 w .t s- : r'h- &n.w %. eh us U n; , y. :
- e :~-m&. !-e'.
- >- a ;
s.u.W>ip.:. f.h.&; 7,.w"$$;fm:$Yli'[n:hh-@:.h,g\.4"'% WW t.d/ -2 a x.u . $
- c. ,,c m ww. ..a.m.g2;g,glg
..f,N. w.wl,@.:..,a..,. p +gg gg . 4p g ,w sm&g 'y,. p.,p...,1_ g_m.m. g y go .,.g,-
a,, . x . .a. :.. . . ,g i
;S.M _n,n>u e.
,s.u e a _. .
8 .7 6 i 5 i 4 ; 3 i 2 : I i w e 9 D-
.D
- e. - , .
.- g l
'g 4S 4AS ' mo s
? an
~- *"C# ==e .
. , e a,=@ .. e '
- . C a
""e:
s ==u. O> ~ "" ""* 7" .- I C1
,, g 7 -
__ 4 _ - _ - - - -
..- - m_ __ __ _.
ryp, ,,,,,,, Q as -s.n ers s.
- m .ars :
RA18IAL BEARING ,
. THRUST BEARING GENERATOR BEARING DETAIL' 3
}
+
NOTES
- e ,.s.amn om . _-
- === e. ame
.t,a.
s.amm e..
. ,s sem ==.e=3 = arw . e mar.
w v asee == .e m w. m *
. .s coma ees--e asse o.ss e
.e
- e. - .n ..~
s - e g _ g.,g u .+, .,...
- em. ; .mn ==. . e e , e, s
. .. ,,,r,.as . . -
.. r. . a-- .-
g A-- em au nm ..-.-- = . s u m e ww
.w e.w <.. r.aw ew s.ers .
-a sv.n v _
y.,-- e e ,a.. m _ - - - - DIESE*. GENERATOR -
~,"" 02-DG-ENG-DG-2:
.I ,l m BEARING DIL LEVEL 'I l .l ! l. l . l A. .n m -. p.
.evp 4 . m ., a. + n .w.nn -
- 3.rv n -: < 4 %
i 4. . -l- . ~7 3 (,
'I 5 .A 1 . 'l
- l. -
7- _ _ _ l
}. _
, , , , wg - g---- -- - ~ ~ - - ~- - * ~ - - -
u~ ^ -
. w.
. m ..er=
v~.; u ,. s
' !
- J, ' ,1. N; < U p 3 *' 'tkT d.%w .,
G ,
.,, ' ,o ; ".d@; . .c/-. . . .w lWMe -
-p.;, ,, e i',p.. I u) 4
,- =ROLLERLBEARING" FAILURES
%%.. N,';Mjhggf 3 v,. .-
,.,'w,',.bQ
.M% r@ -
'p '
* < t. . t.r * ,,.,,,
suAlso
-5
$ re 3*/"2r/he,'.
y + ret 1/@ HIMI '
.P O.'r;('k'.o
- vM de us4 g\O.
;*y,
! t tw i E
r %'$'. .p&,h,'dT.N
,I Y n
l-N.?hhi(!b,f,'h.h 1.' 'll.i.;'jy(*tAf .s u.:, f ~t 4 y g>::,W%en e . t.w . 1 _ y I
;.DM*k%dh/
s ' jinn. f]E/, N.c l. e.m.n . I M' Os h:,.,*v.. y . 'M' 'N '"=*- "*'" "*" '#"
.mj 1, %*d*A'e"s#f 14.W) %" 1m i
i a i$$ . e., rP'E.t.k $e '.*J .w m &H 5? h h. k c '4,
, .t y y
.d .
gg
?A?.ftn!l.
-^
- e . rr a - r. cx m o 3 P.!!4 r
Ces:Daatl4J 3 r-mfd 3r RC i t *'.11 N ' Ig'. G N )M M C
-; I Ak >t =;
?mmuqip$6Y
- v. .: M m x >
hT i
?1.9%
Y[W'*
$:PMIN 'h, W 5'4'flf$[6 6 )92 Q P.
j I i I X'ld. : I
) I stiv3:30 c1 J
h* p cm&*$- f'fk h' "#$M
- t ' 8AFD 'I.h3 PITAk c'
MUI8;'O . [f
- ,& "i .u$
t ; c ; A i M@, e.,t, . e * (,"1", A,; ' inf*
"i: 1
;t . 4;. d
' *A q Wl.E. - t1 &. YE .
Fi.'dd,s'jy,.lk[gr is ifie ..w Q4 9thpf.; *"t g?,] TA 9 nun ,l!vhm ns mreu
.:e ' .3*,*y c% #3.%,I8 iE nm3u. uni b h <ry,i'ugs #i i 1.:st -SAvt3 'lettD D3s ' t.at r - taria t > ttvtu!
- .v.. n h,,:,j " nuns l P Mt t'5
',.m,
*:,:. pd
.~e b* Y
~*
d 'ss.1.
-t j i g
?
hf,q l 1 l * -l'
* -ls : l.'
$ yg.s [ /, 6*e , y"r.6,v,. , ,qa., 3 ' I y nt vnftTb 1 a a F{- I n.g Omit:T!
'* CMll 0.((- #
s nTiso :/.4A.tAilex fAngc - FAf(GV ? Rt LAtl3X*.DT ?.! 1 A'.15 MUt M,hI.idd M e-5 k.h,.#0 @i'@T$$c f' (% M q '"*' ; L 1 JL J L t OC J CA 8A t
%*y&.,y)h*
' [h O u n w w. e es
'M g r -
\ ,
W}hMh h cW 6:Id,D I yV.4 " Mc/N/Y,8; ho, p ' ?"' ( M y
- r, .T'
ixturrictet t ro ev:x
- p :ttsivt 4,g'.td ( LVMicayt . LVIAI CANT : kV&Algut ' OvhttAD . F AIL 40 '
- g;n,,,[( '
y .s c. e ; e us , e se a M,h , '*gjg*- sk th 7 h ).9 # ? --
)
i IT4; @' f.
*g ft$lg -g ( ,
- CC3CDTMitD C'Avolwo DA*M D -
RISE $P474 '- ' f a'.t!
/9 f . s. ? 3 8, LtAD r.n.us MkLm PAta ~ CM1 04*.7tliv!Cx8 O MM IAtsst.L:55
.(G f.hd..$
!F ;. , E0/d$r i#f z' :i -
. dyf*@ 2
-j j j.
~
_l l-M@5YNhhbkNhhhk$h
... .rc .. . . */.;. %,e. ;wS,<.c.::
uyet:. y
.a e ;Q - h... - r i m ino '
umts on sattuun ts 3 Txnnuria' isnuutm%imno ul unvn- u an:vu - n:m l
. g v n A t j .
t -s t' ' "; L ;(- )
;f < .-Ax, * .q - };
s.t aw ,<.m 3 Wq r-'x.. m tum 3 r - nnon n, Se r.
.t omr W ,%. F.4 M,re.,.W x.fp%. nennyt
! e..
- was a .vW.o...,g ,a e ?.
w u nition rmin 3.. o .. s. . o -; ( ; e
,. ,y
.M .M$9h'.$:$g
. .;tp:a
, :: 'z.5y:,. g h j.
x.Q :q e < l m r _ ,,,, 3 l
; yl
.%;fW.
- . t wet ua
_ . . - t
=ta
.>i '
4 9
'u jys,3 _,- - - -
!I
=
~ ;
~ . yw J is.
~
0 9
/
N ^~ 9
/
5 s\
~ 0 9
/
/ 2 2
/e .
/
4~,; 4 0 9
' /
r\'- 1 1 _ / 1 s k. 3 0 N .
... /
S G 1
' \
. .s 1
\- /
\
\ / 2
\ e 0
\
DS R E A/
\
/
i l/' j /
/1 9
{ l 7, 1 R T j
- i
/ 9 O
~ E !
i 8 I I
/
M i
- 7 1
3 A '
/
bR N 2
.~
1 A .\ 9 8 x
/ -
P
@ XN 0
-~
0 2
/1 1
f - 9 8
)
j A T / -..
/
/
1 2
.R 0 1
B I 9 V / 8
- /
9 b / 9 8
) .
~
~
~
~
/
/
7 2 8 9 8
^ ' y
/
z 7 s L A a
. - / i
. 8
/
s X A 6 4 2 0 8 6 4 2 _ 1 1 1 1 0 0 0 0 3 0 0 0 0 0 0 0 0 - (pg - s a
~
- g. d OF
~ ~
] 0 . 9 E 2 /
.% 9
/
x 3.' 5
- Ns-s x',
0 xN
- 0 _
/ _
A 2 2
/ _
-[f. e. 4 0
2 v
/
9 1 1 Z I R _ s / 3 O K% I I
' 0 S
x ~
/1 2
.~
h
- e. ' 1
/
( -- /,/. , 2 - 0
._. ,, , 9 S '
/
4 R / 1 E s._ 1 T -.,.- 9 8
" /
' 7
- j/ i',' /
1 A .i - 2 R < 1 A i ,j/ 9 8 P \
\ i
\
,- i /
0 a/ , ./. 2
/ \.' /
5) $( N i
! / . \ 1 1
T O j l-
!j
.\ 9 l
l I i. . 3, 8 E T f A / I _ _ A
,i /
1 2 V
/
R 0 1 2
- B I 9 -
8 V / -
/
. 9 9
8 ( / 7
,/
}
, 2
.x /
~ -
,b/ 8
,.' j '
9 8 s '
< /
- 2 A /
7
.N ,
I 9
. N s /
8 A I 8
/
6 X A 6 4 2 0 8 6 4 2 0 P. 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -
) _
k _ i
pip = ' E Lol __..
=.I m ra.cge,,.,
/. m-z, i,. . s -
~; =
e$. .
, gr ;
n 7 - *- / / ! / ! tj, (,z 8
/ ,
? 'k 4
,f_- / x, .
/ e j
_ra/ -
,,,-f. .s
. g, ]~~ a
~ c w-'
s
- x
,,f
'i
.s,..
g ?sp *}' ' - }
^~\ I El
n ail =
eN \ kT' b 1 M.- - q' N,7m -' :% wugwN,,. '
,-,,^,~
3 y. 7 . s'
\ \, ' ~~
'MG%t@ y .{}
s N x My -gi - - w~ - 3,, ,
, \.g.gpst .m y' g
h,M . k I ._ S. I \ ( . \ AQba'~' Q\ m \ \ s/. g .% 0
.$ 'd
~
\
.\ -\' \
$))hk[&hW6' '
S u\ y;: 9%;p l
\ \ \A< i s h , i n g-4 N ,g \:n 3,c;.g:. n Cf) _.L4; *N.
w i m,,,.m -
, e, - -
cc
\. w--
e A~\ wf C. q (p
- / --@ ,.r,3 x~
- n. ,
\ \ \ ;\ \; \'MQ:
\:. =. . ,. ~ : no - G _M f, .
u s~$ .. ,0 kQ- = .E - C, G. i] +C'. 3 *4G,',3 E, ag o e o o o~c~y, 9 a t i
')
c 59e s
. , - - +
.e s e m .-.e wy wv '- -
t m.,__ .
. G. ; . - - :
DG1 ~ CYLINDER TEMPS AND DC AMPS ' i FULL LOAD OPERATION 3
! DC AMPS -
CYL TEMP & KW-160 s 5----------------------- , F ;e . *._.. .w_ _ _.4'_ . %_.._ % . . p.__.-$._ N K_w k
- t. -M ---t. K t 2 ..$_' *_N. *- -% ',14 0
[ 41 r'8'e ^ _ _ _ _ _ . . _ _ _ _ . __ ._. .. Y _ U W ~~ '_ _9 .-_-_ ~ h cC _ _"~~~~ _w _.- 120 1 4 i s
- _ . _ _ _ . . _ _ - - _m_
_~
~
~ =
"~
- ~~ 100 :l t
._- - - - " - - ' " - ~ '
3 i
- . = .. . - . _ _
g9 . f
. t_ . ; _ a _ _._ . _.._ _ _
.p _. - _ _ _
. _ _ . _ _ _ . _ . . . _ ._ .. .. _ . _ . ~- _ . . _ . - - - -
60-49 1
...._...:~.:..-.--._=.
- 3. 4_ ; _=t-=t-t t-i -i-,,, 4. :
_ . . _ . _ . - _ _ _...___=____ .. . 20
- e .
' ?'
' ' ' ' ' ' ' ' ' ' ~'
0: O 3 TIME - , i ,i c o. i - s
--*- K W DC AMPS : W, ..
4
- ENG~1: CYL - .ENGi2TCYL .
'I 4 %
. - . ~ m9 I I' . JUNE 1985 TO JUNE 1990 I^ e.[, <
?
{'- : - [ ll _ v j.
. u- ,pg.. --wt. s.- - -- 6
~% -- , g .
._iy ._-4-*t b M.H T _ -- - -________________c-_ '
.. 3 - SPECtAL SEARINGS :: BALL SCREWS :: PRECISION REVOLVING EQUIPMENT i [, $s
- h wDd B precision beari, cs 223 PINE STREET (P.O. BOX 451), CARLTON, OREGON 97111 TELEPHONE (503)S52 6012 JUE 21, 19 9 ')
MR. DONNELL R. KIDDER, ENGINEER l WASHINGTON F"3LIC PJa'ER SUPPLY SYSTEM PO E0X 968 >:.:!L DROP 956B RICHLAND, WA 99352 SUBMCT: GENI.uTOR BEARINGS FAILURE YOUR *1JRCRASE ORDER #208850 DEAR MR KIDDIR, i IT IS >Pl CONSIDERED OPINION THAT THIS FAILURE WAS PRIMARLY. DUE TO LACR
,,.. OF LUBRICATION, *.iHICH STARTED AT THE TAPERID ROLLER BEARING END (THRUST j., -) BEARING) THIS ALLOWED THE THRUST LOAD TO TRANSFER TO THE CYLINDRICAL ROLLIR END, AS THE TAPERED ROLLER SEARING INCREASED IN END PLAY THE CYLI.\T)RICAL RD:.LER BEARING WAS SUBJECTED TO FURTHER THRUST LOAD AND '
MISALIGNED LO.O CAUSING THE ERONZE F.ETAINER TO EREAK UP. THE BRONZI PARTICALS STOPPID TOTAL WELDING 0F THE ROLLEP,S -TO THE RACES IN THE CYLINDRICAL ROLLER BEARING. HOWEVER, THE TAPPERED ROLLER BEARING ACTUALLY WELDD 'SOME OF THE ROLLERS TO THE RACES WHICH INDICATE 3 TEMPIR-ATURES IN EXCESS OF 1500* F. YOURS VERY TR"I.Y, ff G,?!C W ' R.E. BROWNING RB/cs
.G l9y .
-{
li-DG-1 SHORTEJ WINDlhGS l
- POLES FOUND WITH INDICATIONS OF WINDING 'TO WINDING CURRENT LEAKAGE
- DECIAION TO REWIND ALL EIGHT. POLES-
- TIME CONSUMING' AND SUBJECT TO ' MANUFACTURING ERRORS
- ROOT CAUSES
- POLE COIL MANUFACTURING DEFECTS. WINDINGS NOT FULLY-
' ENCAPSULATED DURING ASSEMBLY - ALLOWS TURN TO' TURN . h-LEAKAGE.
I - APPARENTLY CONFINED .TO LOCALIZED AREA.
!! - NOT SUBJECT TO TIME-RELATED DEGRADATION UNDER NORMAL
) OPERATING CONDIT!ONS. -
- DESIGN DEFECTS , I
-- SPARK GAP.- SCR5'iWS IMPR,PERLY SET - ALLOWS ARCING OVER DURibtG OFF- NORMAL OPERATION. ,.
- CV! DOCUMENTS REFLECT ~ IMPROPER SETTING.
L. Y O .
- -DIESEL GENERATOR! NO.1:- 1 ROTOR FIELD POLE # 2. ,
u
?-
m \- '
\
s N ,
'1 e
1 ; ; ,
+=_++ ,
, , o y~
N']I k _, ii i
~3 4 N'*t#
! =IS ..
- 4. ,
/
j 4 llki'~ Shetted Turns Location:: Bdween Layers 11/12 & Levels,18/19 - -l Appmdmately One Foot in from OSRE > ', of Pc:'e Behind Wedge Bearir)g Phle 4 j_ Additional Evidence of Arc Spotting on-Insulation for ~ 18 inches & Two-4 One Inch Lcog Locations of: Square' :-
- Wire Edge Shorting.
a
DG BEARIAG L=ALLL'Rt: EXTENSIVE DAMAGE ,
*BOTH BEARINGS DESTROYED
*ALL FOUR OIL LABYRINTHS BADLY SCORRED
*ALL FOUR OIL SLINGERS. DESTROYED
. THRUST BEARING JOURNAL' BADLY SCARRED I ROOT CAUSES *O~ RING GROOVE ON THRUST BEARING - BRACKET -
elMPROPER . MACHINED ALLOWED OIL ~ LEAKAGE - 18 EVENTS OF. LOW OIL AND 2 '
- FAILURE .TO HEED PERCURSORS ATTEMPTS TO ' SOLVE' PROBLEM DID .NOT.. RESULT :IN OF ROOT CAUSE AND CORRECTION OF- THE PROBLEM.'
ADDRESSED SYMPTONS.
*lNEFFECTIVE OR INOPERATIVE. INSTRUMENTATION: VE-
~-BOTH VIBRATION ALARMS. TOO SENSITIVE TO GIVE BEARING- THERMOCOUPLES IMPROPERLY. .. WIRED,INOPERATI WARNING
-NO TEMPERATURE INDICATORS PROVIDED
-BEARING OIL LEVEL INDICATION ' SUBJECT TO . FIELD-
-INTERPETATION L
- POOR DESIGN
~~
1/8" OPERATING BAND -ON OIL SUMP- LEVEL FOR THRUST BEARINGS, 3/8" FOR- RADIAL BEARINGS
-USE OF AN OIL THAT FOAMS AT LEVELS SPECIFIED AS P FOR THE BEARINGS
_ .m._-
-we_
- w. we em u i-
-~ "~=-_____.-__.:___ __
.l I
\
-l BEARING JRE BEARING F AILURE MODE - NCR 290-411 COVER PAGE OF BEARING BRACKET
///////////////4 DUTER RING 3 GRE ROLLER 7A~"LUR--
s INNER RlNG 3NNNNNNNh g NNNNNNNNNNNNNNN
//////////////// e&mM5 '
Med , 1 Tr-A7TTR - c, -
- A. ._ UR -_
u - L -, i_ s l- _. x .
\\Mh N\\NNNN\\\\NNN i
m , - . , ~ . , , . , , , . - 4y>c-a ,,s-i-, -,-,+--4,,w e c w~v- ,s-- y , . . , , - ee p++m., , .p-
290-411
] .<, _ ]' _
_ - . ]C -
~p
~1 ~}'
- ~ NCR P A G E ,_ ,_.O F ,,
l i TRAME "
" ' FC ARING BRACKET '
IfAR!IG B
]NSint DMDE COVER _
OPPOSITE END " 'j SLIP vtR - ER OUTSIDE BEARING RINGS DVER .- ROTOR VINDINGS n
.-j _ , g- t;.. ,
w to- --i 2 rj a u e_ ,, b _
. SHAFT
_ ?. b k C - < - a L e i. , , - "
's t3' ll
%s x .._r? .-
. NJ N
" -O!L SUMP OIL SUMP~* '
OlL SIGHT'
-- OUILE 'O' RING Gl' ASS
'E T All d
OlL SIGHT tr-) GLASS f_._ I L CRADLE
.A DIESEL ENGINE COUPLED TO BOTH CNDS l
~ .g ,
6 s a 4 0 l^ ---~ m 7 ==== = 7 j s (fI F- L NOCR RCTA1NCR 7 - EirmPM 1
- m o t_ u s m : F3g T^= !
mwweA
~-
y a ,
/E - _ _ _ TNNmo p em
, j c j,
. =e----:c n- - .g.=.g~
, ~/ /,
/// / /
- Y ".T !.. ?. 2 I // / /
$k?!khW!!!!!!$ .!!kb.?k?!!A.
I e
1 8
\
\
b k .--
.w n \
\
.$ j 3 \
i i 5
\
t
+
\
1
,p', .,
' \
i i. j h' .
\
f;f g$$ 7e 5 l a 4:
\ 'y f' ,
,N< &s s;ifC' e, .
\
1 3 _
;( )e \
\
Q ,, i 4
...f, >.
. / Li s A" l p h i
'. y j
l&!k :~ _ g %. '
l [N ww
' tl_
i/t gx1 - y o 7 g flz $ fh;u N
- p} ~f n, ,
.?' .l 's A w 4ht'f h ri # _4$.h w*[qjp'#/
l 4 h4 a , k. x o 3..% % W- .
..nt yn< $gh eI n.1 Ah.d2ibt,p?%5? %:a ?P
.' gpfg *4. j l9 M
- r. k yy m.dsnu a
St w w~. e ,yk=:, e gg
.;. m a n a i'; a % 3AYBlik j n
.L , ,(
g (f , e l $p " ' T. :' f E
. ,$h+iN
; ?f .-
; s l '* ,lf 4^ f
' ': ey ( h* l m
m I. th \l { A p y ,j . ( -a, u,s,, a, -
=
s. v m ,3 . g g ?s: g.ym i! jis g yl @& 1 QMG'Igp ,)[/ff.-
;j "m
I 'iWpriB ,dffi <. --
$)gg; {g, {;f @*&y?I t fd%#Mggf;(j ' 7
~
8 : . EEINT&EmiE2asadff THEFEf2mFETErdHEZR l'
t
- f i :
]p A
7 7 r p
- 5
,% *~
^ ^* '
g :{
. ..,,, f ,c ;i. .
j T ,
, c' de
~ . e , ,,
. ; .s;- , l4 e ; 1
- l
- I '
r;; , a p 97}Ly m p3a l I i ; si 1 '
+ j.
t - t . WINSTON & STRAWN oC L STRCCT. N W wAt " Po% 0 C 20oC5 350
;ter: am?oo i-
. ,vte s d etCT D A trotr 440$ta mt64w u-f(LECO*iE A 204) 37t 691C 202/371-5703 .
.-September 27,-1990
[REEDOM.0F liiiOETl0N t ACT RE$'.T foZA90 - VV2 of Freedom-of Information.
/O '
' and Publication Services. office of Administration U.S. Nuclear Regulatory' Commission- [ Washington, D.C.- 20555 ' i FREEDOM OF INFORMATION MT REQUEST 4
Dear-Sir:
Pursuant - to the provisions of' the: Freedom of Information Act; , (5 U.S.C.-$552)'and the Nuclear Regulatory Commission'W-("NRC") ' policios and regulations (10 C.F.R. 59. 3 - gt; gAq. )l,1I .~hercby. request that copies of the folloving records be_ furnished _'to me: (1). LAny document that-pertains ~toffailuresfo'f, j. problems with or defects"inigeneratorsi_ ~
-manufactured by'E.P. Portec,1Inc.,.Electri'c?
Products Division;_Ftrsons Peebles-Electric l ( JProducts;' Parsons Peebles Electric Products l Inc.; lor NEI^Peebles' Electric Products, Inc . ,.- l Lwhich are : a part of:- an emergency diesel generator. ' at a nuclear / power; plant
.t (2) Any-document.that' pertains toifailures'of, i problemsLwith or defects in any electric.
generator: pole -of.: any.. emergency diesel generator- ' at a nuclear power plant. _ As used herein, "NRC", refers to the. Nuclear' Regulatory-Commission, and-any offici;al',_: employee,;special gnvernmentL employee, consultant, .or advis~ory ccmmittee thereoftas well as any.other person, entity or governmental body providing:
. . . 1
/ f) 9 ' [ /,, h / ;
,4 t r m n , p- :
=
I Director 3 September 27, 1990
' Pago - 2 -
i information or comments germaine to this request- for publicly l available records, l An used herein, " record" or " document" means any paper,_ tape, microfiche or other medium by which information covered by the instant request is recorded or-stored. Whenever any " record" or .I
" document" is requested,-the request refers to any' draft copics ;
of the final versionLof the document, or any copy of the same ' document which is not an exact duplicate thereof ( .qtg2, with marginal comments) . If the NRC should determino~that any record requested above, in whole or in_part, shall be exempt from nondisclosure pursuant to regulation, I request that the non-exempt portion of the record be provided and that every reasonable effort be made to segregate exempt from non-exempt portions of the record, pursuant to 10 C.F.R. 59.19. I would appreciate your prompt response within ten working
~
days of the receipt of this request as provided by 10 C.F.R. 59.8 and the NRC's policies. I agree to pay such fees as required under 10 C.F.R. 59.33 et seg. for the scarch, review and provision of the requested records. Please advise me in advance if it is believed these fees will exceed $500.00,- If you have any questions regarding this request, please contact me at (202) 371-5876. Thank you for your assistance. Sincerely,
&ffk Claudia C. Guild l
I l
,}}