ML20212P181

Insp Rept,South Reactor Feedwater Pump Turbine Failure

ML20212P181
Person / Time
Site:	Fermi
Issue date:	02/14/1986
From:	Meils E NUCLEAR ENGINEERS & CONSULTANTS, INC.
To:	NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
Shared Package
ML20212P164	List: IR 05000341/1986005 ML20212P161 ML20212P181
References
CON-NRC-05-85-157-1, CON-NRC-5-85-157-1 NUDOCS 8609030022
Download: ML20212P181 (14)

Text

.

Enclosure 2 Technical Assistance to Support Region 111 Inspection at Fermi, Docket 50-341 ,

1 asi: Order tt 15 NPC-05-85-157-1 Inspection Report South Reactor Feedwater Pump Turbine Failure Prepared by: Eric L. Meils P.E.

Nuclear Engineers and Consultants (NEAC), Inc.

15807 Crabbs Branch Way, Ste. 17 Roc k vi l l e , Maryland 20855 (301)330-0553 8609030022 860826 PDR ADOCK 05000341 G PDR FEB 141W6

6 Table of Contents

' Page Sect 2cn c

Executive Summary.........................................

History of South Reactor Feed Pump l urbi ne. . . . . . . . . . . . . . . . 4 Details of the Inspection................................. 5 Inspection Findings....................................... 8 9

Conclusions...............................................

Documents Reviewed........................................ 10 Personnel Interviewed..................................... 13 I

i i

?

4 Executive Summary

~I h e investigation into the causes of the July 1985 failure of the Fermi 2 South Reactor Feedwater Pump Turbine (SRFPT) was conducted from January 21, 1986 to February 5, 1986. The SRFPT is a nonsafety-related component and the investigation centered on allegations that the licensee and its contractor were not responsive to persistent high vibration on the SRFPT. It was alleged that the licensee did not perform adequate trouble shooting, they only repl aced parts, and did not search for the root causc of problems, and further that this practice extended to safety-related systems.

This investigation dealt only with the failure of the SRFPT and thc allegations related to this piece of equioment.

Documente of SRFPT installation, subsequent maintenance, preoperational testing, and operating and annunciator response procedures in effect at the time of the f ailure were examined.

The findings of this i nvesti g a ti on are as follows:

1) The initial installction documentation for the SRFPT is deficient. Three witness points were not signed off by the work supervisor dealing with acceptability of the foundation, levelness of the turbine, and proper torquing of the foundation bolts.

2) The turbine to pump ali gnment documents contain contradictory data and alignment was not performed according to procedure.

3) The startup organization f ailed to communicate to the operations organizatian special warmup procedures utilized to clear thc high vibration problem experienced during the preop test of the turbine.

4) The operations personnel failed to f ull y investigate vibration alarms occurring during the turbine runs for power ascension testing.

L) The actual cause of the failure and resulting extensive damage was due to a variety of errors - no si ngle item can be determined to be the root cause unless it is designated as a failure to correct ex cessi ve vibration of the SRFPT observed during the system preop test.

6) The current program for the repai - of the SRFPT is adequate.

i

7) The licensee has stated i. hat a corrective action program would be t rapl emented to help preclude similar failures. This program has not, to date, been formalized. Follow up by the resident intpectors in recommended.

4 History of the South Reactor Feed Pump Turbine 1972 - First factory test aborted due to high vibration.

1972 - Second factory test satisfactory following shaft balance.

1974 - Instellation begun at site by Wismer b becker.

1980 - High vibration trips removed from trip circuitry.

1981 - DeLaval modi fications to shroud (upgrade modification).

1981 - Rotor returned to DELTEX to machine out rust pitting on journal.

1981 - Rotor reinstalled at site per Bechtel procedure.

1983 - 4 runs for preoperational testing.

1) High vibrations (11-18 mils) at 1650-1710 rpm, test aborted.

2) Temporary vibration instrumentation installed. High vibrations at 1650-1710 rpm, test aborted.

3) Special warmup procedure utilized. Vibration normal throughout full operating range.

4) Special warmup procedure utilized in conjunction with preap test. Preop test completed for turbine.

1985 - Turbine coupl ed to pump, feeding reactor for power ascension testing.

7/5 @ 0215 - No problems noted.

7/7 0 1730 - No problems noted.

7/8 @ 0159 - High vi br ati on alarms, turbine HP stop valve drain line found broken following run.

7/12 @ 0815 - High vibration alarme cont i nual l y coming in, turbine HF stop valve drain line found broken for the second ti me , speed probe tips found nicked off by up to 1/8".

7/20 @ 1520 - Started and tripped.

7/21 e 1900 - Started and tripped.

7/21 0 2030 - High vibration alarms in and out, of increasing duration.

7/22 0 1250 - High vibration alarms, turbine failure.

i l

Details of the Inspection the investigation examined the various portions in the history of the South Ecactor Feedwater Pump Turbine (SRFPT) in an effert to determi ne the cause of the failure and any contributing factors.

The details to follow are presented in the order in which the investigation proceed. Observations are presented here as well as later in the Conclusions section of this report.

The first area examined was the preoperational testing of the SRFPT. The preop testing was performed with steam from the auxiliary boiler and the pump was uncoupled from the turbine.

During the first run, excessively high vibrations were experienced at appr ox i mat el y 1670 rpm. The machine s critical speed was noted as 3070 rpm as calculated by the vendor. The Detroit Edison Company's (DECO) Startup Engineering Assistance group ( SUE A .' requested the Engineering Research Department (ERD),

also of DECO, to perform vibration measurements to pinpoint the source of the vibration. During the second i n st r un.en t ed run, ex cessi vel y high vibrations were also noted and the run was terminated. In the conclusions of the ERD report for this run it was noted that the vibration was caused by an internal rub or excessive unbalance. A special warmup procedure was recommended by SUEA and a diagnostic procedure incorporating the special warmup req _tirements was perf ormed during the third run. During this run, no unusual vi br at i on was experienced. The final ren for precp testing was per f ormed utilizing the preop test prvccdure augmented with the special warmup steps again with no excessive vibrations noted. A memo was issued by SUEA to the preop test engineer in charge of the SRFPT stating that the operating procedure should be amended to include a precaution to avoid 1670 rpm and to include the special warmup steps in the system operating procedure (SOP). The SOP was never revised and the Feedwater system was accepted as is.

It should be noted that the North Reactor Feedwater Pump Turbine experienced no similar vibration problems eventhough it is an identical unit.

The next area examined was of the operational and maintenance events immediately preceding the turbine failure (sec History for timr frame). Aside from normal preventive maintenance, nothing was done to the SRFPT from the conclusion of the preop testing in late 1984 until the sequence of runs for power ascension testing in July 1985. During these July runs, many problems were experienced and trouble shooting by DECO concentrated on speed i n d i c a t i cr. problems. During these runs, much time was spent at or about the 1670 rpm high vibration point noted during preop.

1 i

l Vi bration alarms were repeatedl y recei ved. Operations personnel checked the plant computer which indicated low vibration levels.

7 hey' concluded from this information that the high vibration j alarms were spurious. Operators werc never dispatched locally to l verify the validity of the alarms. The annunciator response I procedure- in effect at the time calls for-the operator to trip the turbine' if vibration exceeds 4 mils or an unusual noise develops. Good operating practice is to dispatch an operator to ,

locally assess the validity of a suspect alarm. In the time frame of these runs, the SRFPT high pressure stop valve drain line broke twice. The engineering decision wcc that the lines were inadequately supported. Additionally, the speed probes failed and were pulled showing the tips to be nicked off by as much as 1/8". The corrective action was to replace tho -pickups with new and increase the probe tip to speed gear clearance from 0.005 to 0.020". Upon replacement of the probes an excessive amount of RTV sealant was used on the bearing cap resulting in insufficient front journal bearing crush. The excessi ve . use of sealant contributed to the e:: tent of the damage. The repeated alarms, broken drain line, and destroyed speed pickups should have been sufficient warning that serious vibration problems existed with the SRFPT.

The next area examined was the current repair program for the

.SRFPT. The work request travcler was reviewed in an attempt to assess the thoroughness and completeness of the work performed and to assess the readiness for operation of t,he turbine. The resultant conclusion is that the work performed was thorough and that all the required work to prepare the turbine for operational readiness has either been performed or is planned. The only reservation being in the area of post-maintenance testing of the turbine. As of the conclusion of the site portion of this i nvecti gati on on 2/5/86, a comprehensive corrective action statement had not been formalized by the utility. It is DECO's intent to produce such a statement.

During the perfcemance of the current repair program, a survey was performed of the foundation of the SRFPT. The turbine is supported by three pedestals. Two on either side of the turbine axis support the turbine casing and the third supports the uncoupled end (front) bearing housing. The survey showed the turbine casing pedestals sloped 0.OOS"/ft rear high, and _the front bear ing pedestal essentially level. It is possible that roter and casing thermal expansion during turbine heatup may have induced a mild shaft bow. This was not icentified by DECO as a possible' contributor to the initial internal rub experienced during preop testing and the rub which ultimately led to turbine failure. The deficient construction of the turbine pecestal directed attention to the initial construction documentatic, for

the SRFPT.

lhe initial installation documentation for the SRFPT was e::amined to determine if the turbinc failure was caused by a construction def i ci ency. The initial installation documents for the SRFPT were found to be missing signatures for the completion of several steps of the installation procedure. These steps are for the verification cf acceptability of the f oundati on , the l evel ness chec t: of the turbine, and the torquing of the f oundation bolts.

It is probable that due to these mincing checks in the installation procedure that the excessive slope of the turbine casing pedestal s was created during initial construction.

The maintenance history of the SRFP1 from initial installation to the time of turbine f ailure was reviewed. The documentation r el ated to the. turbine to pump alignments contain conflicting i nf or mati on. The alignment performed just prior to the turbine failure was not performed according to procedure and was only performed for parallel alignment but not angular alignment. The documentation related to the disassembly and reassembly of the turbine in 1981 was f ound to be complete and detailed enough to verify that the turbine internal clearances were within design limits following reassembly.

Inspection Findings The findings of this investigation, as presented below, are the result of the review of documents presented by the licensee and of interviews conducted with DECO personnel and their contractors.

1) The initial installation documentation was not properly completed - three witness points wer e not signed. These points were for the acceptability of turbine foundation, levelness of the turbine cating, and for proper tor quing of the foundation bolts.

2) The turbine to pump alignment documents contain contradictory date. The last alignment prior to failure was not performed according to procedure.

3) The startup organization failed to communicate to the operations organization, special warmup procedures utilized to clear the high vibration problem experienced during the preop test of the turbine.

4) The operations pcrsonnel failed to fully investigate vibration alarms occurring during turbine runs.

5) The actual cause of the failure and resulting extensive damage wat duc to a variety cf errors - nc single item can be determined to be the root cause unless it is designated as a failurc to correct a deficiency observed in the system preop test.

6) The current program for the repair of the SRFPT is adequate.

7) The licensee has stated that a corrective action program would be implemented to help preclude similar failures. This program has not, to date, been formalized. Follow up by the resident inspectors is recommended.

Conclusient The conc l usi ons presented her e are bated on factual information pr ovi ded by the licensee during the course of the investigation.

The f ailure of the SRFP1 in July of 1985 was the culmination of a series of errors and poor practices. The turbine failure occurred while being run by the operations personnel and they are primarily responsible for the failure eventhough they were " set up" by pre. i ous events and decisions. The operator's failure to properly respond to high vibration alarms led directly to the failure. The f ailure of the Startup Organization to first identify the cause of the internal rub and fix it, and secondly to fail to pass on key operati onal requirements for the machine, were the c.c s t significant portion of the " set up". The error in the turbine pedettal construction could have induced a mild warp in the turbine shaft due to unequal growth of the rotor rel ati ve to the cccing during heatup. The turbine to pump alignment could have played a part in the vibration problem, however, the turbine would not have exhibited different vibration characteristics during a normal versus special warmup sequence as experienced by the SRFPT.

The current maintenance repair program for the SRFPT is adequate and thorosgh. All internal clearance measurements were taken and proved to be within acceptable limits. The difference in pedestal slopes het been corrected. A /cndcr representative was present throughout the major portions of the turbine repair and reassembly.

The licensee has stated that they will implement a c or rec ti ve action program to hel p ensure that a si mi l ar failure does not occur. As of the end of the on site investigation, the program had not yet been formalized.

It is my opinion that Maintenance is providing Operations with a good turbine. Successful operation will depend on an aggressive post meintenance testing program, attention to turbine super v i sor y alerms, and an adequate inservice surveillance program.

Documents Eeviewed DECO Purchos2 ng Depar tment Inspection Division, Inspection Repori No. 6, for Reactor Feed Pump Turbine Drives, dated 5/9/72 DECO Purchasing Department inspection Division, Inspection Report No. 9, for Reactor Feed Pump Turbine Drives, dated 7/17/72 Wismer L Becker Operation Prcecess T ravel er for the Installation of the Ecuth Reactor Feed Pump and Turbine, dated 6/13/78.

Wismer F. Becker Operation Procesc Traveler for the Disassembly of the South Reactor Feed Pump, dated 5/21/79.

Wismer L Becicr Operation Process Tr av el er for the Reassembly of the South Reactor Feed Pump, dated 7/1/82.

Wismer t Becker Operation Process Traveler for the Installation of the North Reactor Feed Pump and Turbine, dated 6/13/78.

Wismer L Becker Operati on Process Traveler for the Disassembly of the North Reactor Feed Pump, dated 5/21/79.

Wi smer & Becker Operation Procesc Traveler for the Reassembly of the North Reactor Feed Pump, dated 7/1/82.

Feeduater Sys_ tem T ernover Package from Construction to Startup, Turnover No. 03-0342P, dated 4/11/79.

Feedwat er System Turnover Package f rom Construction to Startup, Turnover No. 03-0342P, dated 11/6/81.

Project Engineering Meeting Minutes, date 11/10/80, subject RFP &

RFFT Super vi sor y Instrumentation; RFPT Protective Trips.

Punchlist Card No. N21-OO3 00008, Disassembly and Reassembly of the SRFPT, for period 8/81 - 1/82.

Peactor Feedwater System Preop Test, PRET.N2100.OO1 rev. O Test E:: c ept i on Disposit. ion Report (TEDR) No. 19, Preop Test N2100.OO1 ' Reactor Feedwater), Perform diagnostic test.to collect vibration data on the SRFPT Letter to J. Cagle (System Test Engineer) from J. Contoni (SUEA),

subject Diagnostic Test of the SRFPT, dated 9/28/83

4 Letter to J. Cagle (System Test Engineer) f rom J. Contoni (SUEA),

subject SRFPT, dated 10/3/83 Letter to P. P. Achar ya (Generation Enginecring) from R. F.

Drol:enrhire (Engi neer i ng Research), subject Vi bration Analysis of Fermi II SRFPT During September, 1983, dated 10/19/83 Wori Order No. 5505D4 Align ERFPT L pump, dated 5/1/84.

Work Order No. 98G727 t or the removal / reinstall ation of the SRFPT front bearing cap for IOC ens.mination of speed prober, dated 7/22/85.

Work Order No. 276518 f or the removal / reinstallation of the NRFPT front bearing cap and ILC wori. on speed pickups, dated 8/8/85.

Devi at i on / Event Report No. NP-85-0384, Trip of South Reactor Feed Pump and Governor End Bearing Damage Inspection Report of SRFPT damage by Adrian Chatrer (DeLaval),

dated July 28, 1985 Letter to J. D. Leman (Supt. Maintenance / Modifications) from C.

J. Cassise (Maintenance General Foreman), subject SRFPT damage, dated 8/12/85.

Letter to F. E. Agosti from SRFPT Task force, subject K-T analysis of the failure of the SRFPT, dated 8/20/85 Engineering Evaluation Request (EER) No.85-732, SRFPT lower case 0.063" low on thrust bearing end.

Potential Design Change (PDC) No. 4689, SRFPT lower case is 0.063" low on the thrust bearing end.

Engineering Evaluation Request (EER) No.85-733, SRFPT sol e pl ate for front bearing bracket.

Potential Design Change (PDC) No. 4690 rev. A, SRFPT sole plate for front beari ng bract:et. Adjust slope to compensate for slope in turbine shell.

As Duilt Notice (ABN) No. 4690-1 rev. O, Rework of SRFPT front bearing pedestal sol e pl ate.

Temporary Modification No. 86-0007 to correct wiring errors of the SRFP/SRFPT vibration sensor / computer interface, dated 1/24/86.

6 Annunc i ator Responsc Pr ocedure (ARP) 5D57, South RFP/RFPT Shaft 1 rouble, rev. O annunci ator Response Proccdure (ARP) 5DD7, South RFP/RFPT Shaft Trouble, r e v. 1 System Operating Procedure 23.107, Reactor Feedwater System, rev. O System Operating Procedure 23.107, Reactor Feedwater System, rev. 4 System Operating Procedure 23.107, Reactor Feedwater System, rev. 8 fic a ntenanc e Procedure 35.000.107, Horizontal Rotating Equipmcnt Alignment, rev. 1

o a Personnel Interviewed S. Booker - Maintenarite Engineer J. Contoni - Lead Engineer, Mechanical R. Gies - Consultant Mechani c al Problems A. Peluso - Systems Engineer M. Ni chol e - Maintrnance Staf f E. Lapeta - Systemt Engineer A. Geschwind - Systems Enginec-r A. Boyd - Maintenance Foremen R. Mack - Vibration Analysis S. Herd -- Operating Dept. Procedure Coordinator