ML20234C982
| ML20234C982 | |
| Person / Time | |
|---|---|
| Site: | Fermi  |
| Issue date: | 05/21/1987 |
| From: | Beckman D PARAMETER, INC. |
| To: | |
| Shared Package | |
| ML20234C924 | List: |
| References | |
| NUDOCS 8707070053 | |
| Download: ML20234C982 (11) | |
Text
-
50-341/87020 Technical Review Report Inspection Follovup of Failure of the South Reactor Feed Pump Turbine Fermi, Unit 2, Nuclear Power Plant NRC Task No. 157-01-025-A May 15 - 21, 1987
~
8707070053 870629 PDR ADOCK 05000341 0
1 50-341/87020 Prepared for:
Prepared by:
USNRC Region III Parameter, Inc.
799 Roosevelt Road 13380'Watertown Plank Rd.
Glen Ellyn, IL 60127 Elm Grove, WI 53122 NRC Liaison Personnel:
Authored by:
E. Greenman D. Beckman W. Rogers Prisuta Beckman Assocs.
M. Parker i
i W
i 2
A.
50-341/87020 Fermi, Unit 2, South Reactor Feedpump Failure Executive Summary On May 13, 1987, the Fermi 2 South Reactor Feedpump Turbine (SRFPT) failed during plant startup. The operators unsuccessfully attempted to avoid a low reactor pressure vessel (RPV) level scram using the Reactor Core Isolation Cooling (RCIC) and Standby Feedwater Systems.
The combined systems had insufficient capacity for the 18% reactor power level resulting in an automatic scram on low RPV level.
Loss of Feedwater procedures have been revised to improve operator guidance for such situations.
Inspection coverage by an NRC consultant began at about 1600 hours0.0185 days <br />0.444 hours <br />0.00265 weeks <br />6.088e-4 months <br /> on May 15, 1987, continued on dayshift and backshifts through May 21, 1987, and involved about 59 inspection hours.
The licensee believes the cause of failure to be an inadequate valve lineup procedure which resulted in a turbine high pressure casing drain being closed during startup. This would have permitted accumulation of water in the HP casing to induce severe vibrations / oscillations when RFP critical speed was reached during startup.
The licensee's root cause analysis used a combination of simplified Kepner Tregoe analysis and engineering judgement.
The' conclusions were highly inductive and supported largely by circumstantial evidence.
Turbine disassembly was observed by the consultant.
Lack of detailed work procedures resulted in some minor problems, e.g., failure to remove all casing bolting prior to attempting to lift casing, etc.
Turbine damage was moderate and is expected to require 3-4 weeks to repair.
The licensee proposed power escalation greater than 50% using only the North RFP and responded to RIII concerns regarding RFP reliability and potential common mode failures.
Special pump monitoring provisions and corrective actions for prior RFP lube oil water contamination were instituted.
Procedure revisions to improve licensee response to anomalous RFP operation were issued.
Power escalation and NRFP restart were attempted on 5/18.
Personnel and equipment performance were observed by the consultant and were acceptable until RFP lube oil water contamination recurred and the pump was shutdown pending correction.
The licensee believes the water contamination resulted from inadvertently placing a waterlogged lube oil filter in service prior to pump restart.
The licensee's programs for control of balance of plant system alignment and status appear to be somewhat informal, contributing to this problem, e.g., the licensee was unable to immediately and conclusively establish the status of the above filter and lube oil system alignment without physical verification of conditions.
NRFP oil system cleanup continued into 5/20; the source of water intrusion could not be conclusively determined.
A special sample plan was instituted and the pump restarted.
While investigating anomalous NRFP vibration computer indications, the licensee found that the signal amplifier for the vibration j
3 l
1
,1 50-341/87020 monitoring circuits was deenergized (and had apparently been so throughout the current period of pump operations).
The monitors were reenergized and pump operation continued through about 2750 rpm (critical speed is approximately 3,000 rpm; rated speed is about 5,000 rpm).
Pump operation was suspended when reactor power was reduced to investigate abnormal reactor coolant leakage.
Significant observations made by the Consultant during these activities included:
a.
Although in many respects the licensee's programs and performance were acceptable, the confidence level in NRFP reliability was degraded by repetitive problems.
b.
The repetitive and recurring problems above resulted from deficiencies F
in the extent of action plan scope and implementation.
c.
The licensee's support staff appeared to be subject to subtle schedule pressures resulting in premature suspension of some corrective / preventive actions resulting in recurrence of prior problems and a reluctance to j
pursue items not already identified by the management approved action J
plan.
d.
Similarly, the licensee's control of nonsafety-related, balance of plant systems appears to be less rigorous th~an desirable as indicated by recurring problems with system and equipment status management.
Further, the licensee's various engineering and technical support departments are generally treated as the ultimate authorities for "on the spot" problem dispositions providing informal, oral disposition of problems permitting deviation from the nonsafety-related procedural requirements.
The informal nature of such dispositions and the absence of documentation and prompt supervisory / management review and procedure changes provides i
opportunities for loss of management control of plant activities potentially important to safety, e.
Although the individual licensee staff members encountered during this inspection were generally competent and displayed affirmative, safety oriented performance, evidence of departmental parochialism and lack of leadership and coordination were observed.
The action plans were based on department task assignments without clear, central leadership below the plant manager level.
This appeared to contribute to the oversights leading to the equipment problem occurrence and recurrence.
Detailed Report The Fermi 2 Nuclear Power Plant, South Reactor Feed Pump Turbine (SRFPT) failed and tripped at about 1700 hrs., 5/13/87, causing a plant trip from about 18%
during power escalation.
The operators attempted to prevent a plant trip 4
50-341/87020 using RCIC and Standby Feedwater Systems, and were unable to maintain reactor level, resulting in an automatic scram on Low RPV Level.
RCIC and Standby Feedwater can support only about 6% power operation, resulting in a decreasing level after loss of the SRFP.
Abnormal Operating Procedure, No. 20.107.01, j
Loss of Feedwater / Feedwater Control, did not adequately address. actions to j
be taken in such instances and has since been revised and confirmed acceptable by the Resident Inspector.
Inspection coverage by an NRC consultant began at about 1600 hours0.0185 days <br />0.444 hours <br />0.00265 weeks <br />6.088e-4 months <br /> on Ma 15, j
1987, continued on dayshift and backshifts through May 21, 1987, and invu'ved j
about 60 inspection hours.
1.
South Reactor Feed Pump Failure Review j
The cause of the SRFP failure could not be absolutely determined.
The turbine experienced severe vibration (13+ mils; 4.0 mil limit), a sheared
' drain throttle valve drain line, a damaged drain line hanger and an RFP trip apparently due to vibration effects on the governor trip circuits.
j The licensee used a simplified Kepner Tregoe analysis to identify potential root causes.
Once the initial listing of 16 potential root causes were identified, the licensee used less rigorous engineering judgement and inductive methods to choose what they believed to be the six most likely root causes.
Further licensee review concluded that the Riost accumulation of water in the turbine first (HP) probable cause was an stage casing causing undampened vibrations / oscillations when critical turbine speed (critical speed approximately 3070 rpm + 10%; rated speed approximately 5000+ RPM) was reached.
The turbine venBor was consulted on 5/18 and found the licensee's conclusions plausible.
The water accumulation was apparently caused by a defective return-to-service valve lineup procedure, resulting in the turbine HP casing drain valve (F0078) being lef t closed after maintenance.. The licensee's investigation included reverification of system valve lineups and operator interviews.
The interviews indicated that the pump had been isolated per the Feedwater i
System S0P No. 23.107, Section 4.17, on the day shift of 5/12.
The system was restored during the 0000-0800 shift on 5/13.
50P 23.107 did not include a corresponding return to service section.
The operators indicated that the pump was returned to service using the pre-warmup-procedure.
Section 4.17 closes various manual valves,-including F0078 but the pre-warmup procedures do not verify / reopen them.
The Feedwater 50P restoration procedures have been corrected to ensure that return to service lineups will be controlled by the licensee's abnormal lineup i
administrative procedures.
The consultant observed t'he SRFPT disassembly and damage inspection from 5/15 - 17..
Licensee maintenance activities were generally acceptable except for content and use of detailed disassembly and inspection procedures.
The maintenance procedures were general in 5
50-341/87020 nature and required extensive use of turbine drawings for disassembly.
During shift turnover on 5/15, the oncoming maintenance crew was advised that the turbine upper casing half was unbolted and ready to remove.
Minor difficulties were then encountered in disassembly in that two large (approximately 2") casing bolts were not removed prior to the first attempt to lift the casing.
Damage to the SRFP turbine was moderate, involving steam and oil seal damage, shaft damage due to seal contact, and diaphragm and rotor diaphragm damage resulting from rubbing contact. No significant shaft deformation (bowing) or damage (lost blades, etc.) occurred.
The rotor and accessories were shipped to the turbine vendor on 5/18; repairs are expected to take 3-4 weeks.
The licensee employed a failure analysis consultant to eva'luate the SRFPT bearings for event damage and potential contribution to event causes.
The licensee's consultant concluded that the bearings performed nominally and showed no indications of being cause contributors.
The SRFP Limitorque actuated turbine exhaust damper was also found damaged following the transient.
The butterfly type damper had apparently overtravelled due to an actuator limit switch rotor failure and damage to the damper's external mechanical stop.
The damper was repaired on 5/17-18.
The Resident Inspectors followed up o.c details of the Limitorque limit switch rotor failure.
2.
Power Operation With only One RFP Operable The licensee proposed power escalation through the 50% power plateau with only the North RFP available to RIII and responded to RIII concerns regarding NRFP reliability and potential common mode failures.
The licensee initiated a 35 point action plan to address current and prior concerns and operational / maintenance problems.
Major aspects of that plan included:
The licensee's failure analysis of the SRFPT and related followup were directed at confirming that no potential common mode failures or precursors affecting the NRFPT existed.
An engineering review was conducted for the last (August,1985) j overhaul and alignment of the NRFPT and the results were presented i
to the inspector.
The results were considered nominal by the licensee.
During the 1985 work above, the licensee found that the RFP units 4
underwent substantial dimensional changes when vacuum was applied to the exhaust casing, including lifting (up to 0.050") of the low pressure end foundation (flex foot to sole plate joint).
The pump vendor recommended immediate modification of the pump foundations to control the uplift due to vacuum.
The licensee's evaluation disagreed with the vendor's.
The licensee stipulated that the above NRFP alignment results compensated for the uplift phenomena pending eventual modification of the foundations.
6
50-341/87020
{
~
The inspector reviewed the detailed alignment data, vendor correspondence, and internal licensee correspondence finding that the 1986 alignment had apparently been accepted "as-is" with respect to the uplift problem.
The licensee's disposition was based largely on acceptable, vibration free operation of the turbine in the interim.
Historical and current vibration data showed nominal pump i
performance.
The licensee did not discretely evaluate the effects of uplift on offnormal turbine operation.
)
Prior problems with water contamination of RF.P lube oil and pump seal wate ' control were reportedly corrected.
The NRFP lube oil 1
had been found to be water contaminated during the week of 5/11.
The licensee believed that the water leaked from the pump shaft water seals back through bearing oil seals into the return oil.
Abnormal seal oil pump and tank level control operation had been identified as a potential cause and short term corrective actions were taken.
The licensee reverified the feedwater system valve lineup to: (a) determine whether any other valves (in addition to the above Snuth turbine drain valve) were mispositioned and, (b) provide assurance that the system was properly aligned for operation on only the North RFP.
When queried by the consultant regarding the performance of a hands on inspection of the North RFP and turbine to ensure general integrity of equipment (no obviously lense nuts, bolts, connectors; visual conditions satisfactory, etc.), the licensee conducted such an inspection.
All conditions were reported to be satisfactory.
A special pump / turbine monitoring program was instituted by the licensee to ensure that all critical NRFP parameters would be j
monitored from pump warnup through critical speed operation.
The plan included manually logged data in the control room and pump rooms, three sources of remote pump vibration monitoring data, acquisition of a fourth set of pump vibration signatures for spectral analysis, use of various control room recorders and the process computer for additional data acquisition, and local s
measurement of turbine drain line temperatures to ensure that the drains flowed freely.
A written checklist and data sheets were provided and a preshift briefing was conducted.
The consultant observed an attempted restart of the NRFP on 5/18, 1600-2400 shift.
The pump performed acceptably through warmup and rollup.
Control room activities included a preshift briefing for shift operators and special data takers.
The power escalation and pump restart were well controlled and control room discipline was good.
At about 6% reactor power with the NRFP feeding the RPV, a routine lube oil sample found 3000 ppm water contamination (confirmed by a second sample) and the pump was shutdown.
Maximum desirable water content is 100-500 ppm.
7 50-341/87020 a
~
Following initial reduction of water concentrations from the prior (5/11-14) oil contamination episode, sampling frequencies were apparently reduced from once every two hours on 5/12-14 to about once per shift.
A routine sample taken about 0900 hrs., 5/18, found 82 ppm water content.
Licensee investigation through 5/19 concluded that a RFP lube ~ oil filter was placed in service at about 1330 hrs. on 5/18 prior to RFP startup.
That filter /coalescer may have been waterlogged from prior lube oil purification activities under water contamination conditions.
The pump was restarted about 1730 hrs. and the water contamination-i
.found at about 2000 hrs.
Apparently, no samples were taken between the times of the 0900 hrs. sample, the 1330 hrs. filter lineup change,.
and the pump start.
The licensee's control of balance of plant system alignment and equipment status appears to be somewhat informal.
During immediate J
. followup to this episode of lube oil water contamination,'the licensee was unable to conclusively establish whether the filter above had been
~
cleaned prior to being placed on line and what the precise lube oil system lineup had been prior to the current shift, i.e., whether an improper valve lineup could have caused the contamination.
On 5/20, the licensee continued purifying the RFP lube oil with a temporarily installed centrifugal pacifier and attempted to identify the source of the contamination. The licensee purged and partially drained the system, attempting to remove fugitive water contamination.
The lube oil coolers were pressure tested with no water leakage identified.
Pump seal water system,. lube oil system connections, and other potential water sources were investigated.
Although the temporary centrifugal lube oil purifier was successful in j
reducing the water content to within acceptable limits, the source of water could not be conclusively established.
The licensee postulates that residual water in the system from the prior (5/11-14) contamination episode and the filter discussed above are the most likely sources.
The filter is equipped with a float operated drain valve to automatically dump collected water; the licensee believes this valve malfunctioned and caused an accumulation of residual water in the filter that was freed when the filter was placed on line.
Further, lube oil sampling and purification on 5/19-20 indicated that a significant amount of water was stratified in the NRFPT sump.
Water concentrations were reduced to acceptable levels by the dayshift on 5/20 and pump operations resumed.
The licensee implemented a lube oil sample plan, taking samples from ten points-in the pump oil system on a 1-2 hour frequency.
Minor transient lube oil particulate and water concentrations were observed but were either within procedural-limits or determined to be acceptable for short term operation by the licensee's engineering staff.
As of 5/20, the filter /coalescer elements had not yet been cleaned or replaced and the temporary centrifugal purifier continued to be used for routine sump purification.
8 4
50-341/87020 e
During operation on 5/18 and restart on 5/19, the NRFPT shaft eccentricity indication on the process computer and a vibration monitoring microcomputer showed stable values of 4 to 5+ mils with a procedural limit of 2 mils.
The licensee's vibration engineering specialist acknowledged that the reading was anomalous but not problematic in that it was stable.
Eccentricity would be indicated by an oscillating indication.
The licensee believed that a slight but acceptable error existed in the monitor.
The licensee further acknowledged that the Feedwater System 50P and Alarm Response Procedures should be revised to reflect more appropriate criteria for eccentricity.
Subsequently, the vibration engineer reviewed the control room computer vibration indications, finding them abnormal (repetitive negative values, etc.).
Further investigation by the licensee determined that the signal amplifier feeding the two computers was deenergized, apparently since the last system calibration.
As a result, all NRFP operation from 5/18 through the afternoon of 5/20 had been conducted without real time vibration monitoring.
The inspector was advised that the spectral monitoring equipment had stored valid data (did not require the signal amplifier) but that it was electronically stored and not available for real time monitoring.
The NRFP was successfully run up to about 2750 rpm through about 1600 hrs on 5/20, at which time reactor power was reduced to investigate abnormal reactor coolant leakage.
Durir.g an exit meeting on 5/21 with senior licensee management (including the DECO Executive Vice President), the consultant presented the following summary observations, regarding the licensee actions in support of their proposal to conduct power operations in excess of 50% power with only one RFP available:
a.
Operation at high power conditions with only one feed pump available is technically feasible but requires a high level of confidence in the single feed pump to minimize the chances of potentially severe plant and reactor water level transients due to feed pump failure.
In many respects the licensee's programs and performance were commendable.
Management tools such as formal action plans, formal schedules, internal communication, control room and operational discipline, etc. appeared effective.
b.
However, although the licensee's plans to achieve this high confidence level were extensive, the repetitive and recurring problems above were indicative of less than adequate control of balance of plant systems and action plan implementation.
c.
The licensee's support staff appeared to be subject to subtle schedule pressures resulting in premature suspension of some corrective / preventive actions resulting in recurrence of prior problems, e.g. lube oil water contamination.
9 i
i l
i
q 50-341/87020
)
e In other cases, efforts to ensure the high readiness and integrity of the equipment were not sufficiently thorough, e.g., failure to clean or replace the contaminated filter /coalescer prior to use, failure to ensure that vibration monitoring equipment was in service prior to pump operation (especially in light of the extensive vibration monitoring program instituted), etc.
d.
Similarly, the licensee's control of nonsafety-related, balance of I
plant systems appears to be less rigorous than desirable.
Examples include the difficulties in establishing SRFPT drain line status immediately following the 5/13 event, difficulties in establishing RFP lube oil systea status immediately following the 5/18 water contamination episode, the deenergized vibration monitoring power supply, and the attempted lift of the SRFPT upper casing prior to complete unbolting, etc.
j l
The licensee's various engineering and technical support departments are generally treated as the ultimate authorities for "on the spot" problem dispositions.
In several observed cases, these departments provided informal, oral disposition of problems permitting deviation from the procedural requirements, sucn as the disagreement between shaft eccentricity readings and lube oil contamination limits with procedure criteria.
Although the problems dispositioned were technically classifiable as nonsafety-related, the problems co'uld impact the safe operation of the reactor.
The inspector was advised that the dispositions would typically be documented and subject to management review as part of the Deviation / Event Report for the SRFPT Failure and that procedure changes would be made if then deemed appropriate.
The informal nature of such dispositions and the absence of prompt supervisory / management review and procedure changes provides opportunities for loss of management control of plant activities potentially important to safety.
e.
Although the individual licensee staff members encountered during this inspection were generally competent and displayed affirmative safety oriented performance, evidence of departmental parochialism and lack of leadership and coordination were observed.
Some interdepartmental action plan items (e.g., NRFPT startup monitoring program) required plant manager intervention to " assign" duties when departmer,t level initiatives were unsuccessful.
Similarly, the action plans were based on department task assignments without clear, central leadership below the plant manager level.
The inspector believes this contributed to the oversights leading to the problems discussed above.
10
p i
l Attachment ?
50-341/87020 i
References:
=
Deviation / Event Report (DER)87-168, SRFPT Trip & Scram, 5/14/87 j
Root Cause Analysis, SRFPT Turbine Failure, Problem #051487 Action Items List & Schedules, N&S Reactor Feed Pumps, 5/15-20/87 Failure Analysis Assocs. Inc., Evaluation Report on Lube Oil Contamination and SRFPT Bearing Condition, 5/19/87 l
Operating Procedure 23.107, Reactor Feedwater System, Revisions 20 & 21 Abnormal Operating Procedure 20.107.01, Loss of Feedwater/Feedwater Control, Revision 5 General Operating Procedure 22.000.05, Hot Standby to Rated Power
{
Alarm Response Procedures 5.d.27 and.57, N & S RFP Shaft Trouble, Revisions 1 &2 Maintenance Procedure 35.000.30, Reactor Feed Pump Turbine Maintenance, Revision 3 Maintenance Procedure 35.000.31, Reactor Feed Pump Maintenance, Revision 4 Work Request and Maintenance Package PN21-276529, Repair of SRFPT, 8/10/85 Work Request & Maintenance Package, 016A051387, Repair of SRFPT, 5/13/87 i
Reactor Feed Pump and Turbine Technical Manual, TDI Model KJ NSS & Control Room Logs, 5/14-20/87 NRFPT Special Monitoring Logs, 5/18-20/87 RFP Lube Oil Sample / Analysis Logs, 5/19-20/87 DER NP-86-181, Oversize Clearances on SRFPT, 5/3/86 Transamerica DeLaval, Inc. (TDI) Letter to DEC0, RFP Vacuum Deflection, 8/20/86 TDI Alignment Data Report, North RFP Turbine, 8/30/85 TDI Letter to DEC0, Modifications - RFP Vacuum Uplift, 9/4/87 Progress Report, A.D. Peluso, RFPT Vacuum Uplift, 8/22/86 EDP-6128, Modification of RFPT Sole Plates. 8/22/86 11
.