05000498/LER-2001-001

From kanterella
Jump to navigation Jump to search
LER-2001-001, Failure of Essential CoolinqWater Pump 1C Following Maintenance
South Texas Unit 1
Event date: 09-24-2001
Report date: 11-19-2001
Reporting criterion: 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
4982001001R00 - NRC Website
v • d • e

DESCRIPTION OF EVENT

Overview - On September 17, 2001, Unit 1 was in Mode 1 in coastdown operations in preparation for a refueling outage. At 0200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />, the Allowed Outage Time for the Unit 1 Train C Essential Cooling Water pump was entered to begin work to overhaul the pump for corrective maintenance. On September 21, 2001, following the maintenance overhaul, the Train C Essential Cooling Water pump lost discharge pressure during a dynamic fill and vent evolution. The pump ran for approximately 10 minutes with what was thought to be adequate cooling and lubrication water flow. At approximately 2106 hours0.0244 days <br />0.585 hours <br />0.00348 weeks <br />8.01333e-4 months <br />, the pump was manually stopped when output pressure dropped to zero. The pump was observed to come to an abrupt stop. Subsequent investigation into the loss of discharge pressure identified that the No. 2 pump bearing and associated bearing housing exhibited evidence of severe overheating. Lubricating flow orifices could not be seen due to the extent of the bearing damage and steel wires were found on the inside of the bearing.

Because pump repair work was projected to extend beyond the 7-day Allowed Outage Time for the Train C Essential Cooling Water pump, which would expire on September 24, 2001, at 0200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />, enforcement discretion was requested to permit repairs to continue without requiring a shutdown of the unit. The Nuclear Regulatory Commission granted enforcement discretion on September 23, 2001, at 1800 hours0.0208 days <br />0.5 hours <br />0.00298 weeks <br />6.849e-4 months <br />. Maintenance and testing were completed and the Train C Essential Cooling Water pump was restored to operable status and Technical Specification Limiting Conditions for Operation 3.5.2, 3.5.6, 3.6.2.1, 3.6.2.3, 3.7.3, 3.7.4, 3.7.7, and 3.7.14 were exited on September 26, 2001, at 0905 hours0.0105 days <br />0.251 hours <br />0.0015 weeks <br />3.443525e-4 months <br />.

Detailed Event Description - The Train C Essential Cooling Water (ECW) pump had been placed on increased frequency testing in May 2001 due to decreased output. Results of the testing confirmed that developed head had degraded from the reference value. On September 17, 2001 work began to overhaul the pump and increase its operating performance. During day shift, the pump was removed from the ECW system, open pipe ends were covered to prevent entrance of foreign materials, and the pump was transported to the Mechanical Maintenance shop. The pump was disassembled on night shift. As part of the planned maintenance, several parts of the pump were taken to the blast yard where they were sandblasted and a Belzona coating applied per an associated design change package. The parts included the outer columns, inner columns (enclosing tubes), diffuser, suction volute, and impeller. Inspection of the pump internals then began as outlined in the maintenance procedure work instructions. This inspection continued through the next three shifts.

During day shift on September 19, 2001, the remaining parts were returned from the blast yard, inspections were completed and parts were readied for reassembly. In order to prepare the upper and lower inner columns (enclosing tubes) for reassembly, the scale buildup on each end's mating surface was removed. A wire cup brush attached to an air grinder was used to remove the scale buildup. The inside of the tubes was not cleaned following completion of the wire brushing. Reassembly of the pump began during the night shift. As part of the check for cleanness of parts, Mechanical Maintenance personnel visually checked the inside of both the upper and lower enclosing tubes, however no foreign material was detected due to the layer of dark-colored uneven scale inside the tubes.

DESCRIPTION OF EVENT (continued) A step to ensure cleanness and condition of parts was a check off in the pump maintenance procedure and was also a signoff step in the work order data package. However, the maintenance procedure did not refer the user to the Control of System Cleanness during Maintenance procedure, which describes cleaning techniques and provides cleanness acceptance criteria.

The work order controlling the pump maintenance contained a reference to the system cleanness requirements and identified the appropriate cleanness class for the ECW system. The reference section of the work order listed the Control of System Cleanness procedure as the source document for the cleanness class. The Control of System Cleanness procedure contains Addenda describing the acceptance criteria for cleanness as well as the cleaning methods to be used by craftsmen.

The Control of System Cleanness during Maintenance procedure also requires the work planner to establish a Maintenance Verification Point (MVP), where appropriate, in the work control document.

The MVP heightens awareness for an important task. Although the procedure requires the work planner to insert an MVP for the visual verification of acceptable cleanness for the type of work performed on the ECW pump, MVPs were not established in the work package.

Reassembly of the pump was completed during day shift of September 20, 2001, and the pump was transported back to the Train C system and set in place. On night shift, the motor was set in place and alignment of pump and motor began. The alignment, coupling, and repacking of the pump were completed during day shift of September 21, 2001. The shaft rotated freely by hand and the Equipment Clearance Order was released.

The Train C ECW Pump was started locally from the Train C Switchgear Room for a Post Maintenance Test during night shift on September 21, 2001. This start was being performed in accordance with the ECW system operating procedure, which governs the fill and vent of the system.

Pump discharge pressure immediately increased to 65 psig and flow was indicated on system heat exchanger return flow indicators. Local operators, who were at the pump and in the Switchgear Room, reported a good start based on a discharge pressure of about 60 psig, water flow from the Lube Water Filter Common Outlet Vent Valve, adequate seal leakoff coming from the pump packing, good oil level, and stable motor current at approximately 60 amperes.

As required by the fill and vent procedure, Instrumentation & Control (I&C) Technicians were asked to vent the lube water flow and filter differential pressure gauges. After venting, both gauges indicated zero. When this condition was reported to the Control Room, the operator at the pump was requested to recheck pump seal leakoff. The operator and maintenance personnel verified sufficient seal leakoff flow. At that point, the l&C Technicians were again asked to vent the lube water flow and filter differential pressure gauges. Both gauges still read zero after this second venting.

Shortly thereafter, and approximately 10 minutes after pump start, the pump discharge pressure dropped from 63 to 0 psig. A noise was heard in the area of the pump which personnel in the area believed to be the check valve closing. The mechanics noted seal leakoff flow stop. The local operator at the pump requested an immediate shutdown. System flows as indicated on the heat exchanger return flow indicators also were noted to have dropped to zero. The local operator in the DESCRIPTION OF EVENT (continued) Switchgear Room then opened the pump's breaker, and the mechanics observed the pump shaft come to an abrupt stop.

The pump was removed and disassembled on September 22, 2001. It was determined that significant damage had occurred to the Train C ECW pump. Bearing No. 2 was bonded to the pump shaft and the bearing housing of the "spider assembly" was out-of-round due to overheating. The spider assembly is a structural member of the pump casing that maintains the shaft in the centerline of the pump and contains the No. 2 bearing housing. The bearing flow orifices could not be seen due to the extensive bearing damage. The lower pump shaft also exhibited evidence of extreme overheating.

Investigation determined that the wire cup brush, used during pump overhaul, threw filaments during the normal course of use into the inside of the inner columns (enclosing tubes). The wire brush filaments (foreign material) caused insufficient cooling/lubricating flow by blocking or obstructing the bearing flow orifices. The lack of cooling caused significant heating and thermal expansion of the bearing and the shaft which in turn caused the shaft to bind in the shaft casing, ultimately causing the pump shaft to stop.

CAUSE OF EVENT

Root Cause:

The cause of the Train C Essential Cooling Water pump failure following overhaul maintenance was foreign material intrusion. The root cause of the foreign material intrusion was less than adequate implementation of the Foreign Material Exclusion program.

The wire cup brush threw filaments, during the normal course of use, into the inside of the inner columns (enclosing tubes). The wire brush filaments (foreign material) caused insufficient cooling/lubricating flow by blocking or obstructing the bearing flow orifices.

Foreign material prevention activities described in station procedures were not followed. No actions were taken to prevent the filaments from entering the inner columns during cleaning of the mating surfaces. No actions were taken to clean the inside of the inner columns prior to reassembly to remove the wire filaments.

Contributing Causes:

  • The work package, pump maintenance procedure, and cleanness procedure, as implemented during the task, did not provide sufficient controls to prevent the introduction of foreign material.
  • The craftsmen did not follow the cleanness procedure providing direction and control for preventing foreign material from entering plant equipment/systems. The cleanness procedure was not directly referenced in the pump maintenance procedure and was only listed as a source document in the work order.

CAUSE OF EVENT (continued)

  • Work planners and Supervisors did not provide Maintenance Verification Points (MVP) in the work instructions to heighten personnel awareness to the cleanness inspection as required by the cleanness procedure.
  • The performance of the cleanness inspection was inadequate. The visual inspection, as performed, could not detect the small wire filaments that were masked by the buildup of uneven scale lining the inside of the columns, poor lighting and the long length (10 feet) of the columns.

CORRECTIVE ACTIONS

Remedial:

The pump repairs and return to service testing were completed satisfactorily. The work package included additional precautions, steps, and verifications to ensure adequate component and system cleanness. The pump bearings were flushed prior to returning the pump to the field. Once the pump was reinstalled in the field additional lubrication water flow testing was performed to ensure the flow path was clear and adequate flow achieved. The Train C Essential Cooling Water pump was declared operable on September 26, 2001, at 0905 hours0.0105 days <br />0.251 hours <br />0.0015 weeks <br />3.443525e-4 months <br />.

Compensatory:

This event was discussed with all maintenance supervisors. These discussions reinforced the requirement that work packages be reviewed to assure that required Maintenance Verification Points were included. This action was completed on October 5, 2001.

Corrective:

1. Revise the Essential Cooling Water system operating procedure to require an Essential Cooling Water pump to be stopped if the associated lube water flow indicator does not indicate greater than 3.0 gpm flow within 45 seconds after a pump start, including pump starts for Fill and Vent evolutions. This action was completed on October 12, 2001.

2. Revise the ECW pump maintenance procedure to require specific cleaning instructions for the inside of the columns (enclosing tubes) prior to reassembly of the pump, and incorporate Maintenance Verification Points as required by the Control of System Cleanness during Maintenance procedure. This action will be completed by November 29, 2001, prior to the overhaul of any other ECW pumps.

3. Brief work planners, supervisors, and craftsmen on their responsibilities for cleanness requirements, inspections, and MVPs as per the requirements of Control of System Cleanness during Maintenance procedure. This action is to be completed by November 29, 2001.

CORRECTIVE ACTIONS (continued) 4. Reinforce station expectations for Foreign Material Exclusion practices during formal training. This action is to be completed by April 25, 2002.

ANALYSIS OF EVENT

This event is reportable per 10 CFR 50.73(a)(2)(i)(B) due to exceeding the 7-day Allowed Outage Time of Technical Specification 3.7.4, Essential Cooling Water System. The Nuclear Regulatory Commission granted enforcement discretion for this condition on September 23, 2001 at 1800 hours0.0208 days <br />0.5 hours <br />0.00298 weeks <br />6.849e-4 months <br />.

The quantitative change in core damage probability associated with the Essential Cooling Water pump being out of service for up to an additional 5 days was compared to a qualitative assessment of risk associated with a plant shutdown to comply with Technical Specification 3.7.4. The additional 5 days was preferable to the potential consequences associated with a plant shutdown, and there was no increase in radiological risk. This event was reviewed for risk impact and found to be risk insignificant since the conditional core damage probability (CCDP) was approximately 8.24E-7.

The Allowed Outage Time for the Train C Essential Cooling Water pump expired on September 24, 2001 at 0200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br />. Essential Cooling Water pump 1C repairs continued without shutting down the Unit, which is a condition prohibited by Technical Specification 3.7.4. The Nuclear Regulatory Commission granted enforcement discretion to allow continued plant operation while Train C ECW pump repairs were completed. Enforcement discretion was also granted from the Action Statement shutdown requirements of the Technical Specifications associated with the systems supported by Essential Cooling Water:

ADDITIONAL INFORMATION

The following compensatory measures were implemented during the period of enforcement discretion:

The following equipment was verified to be operable or functional during the extended allowed outage period:

  • The Positive Displacement Charging Pump (maintenance and/or testing prohibited).

ADDITIONAL INFORMATION (continued) The following equipment/lineups were verified to be operable or functional during the extended allowed outage period:

  • The circuits required by Technical Specification 3.8.1.1.a (circuits between the offsite transmission network and the onsite Class 1 E Distribution System).
  • The two remaining onsite power sources required by Technical Specification LCO 3.8.1.1.b (standby diesel generators and each separate fuel tank).
  • The two supporting Essential Cooling Water loops required by Technical Specification LCO 3.7.4.
  • The circuit between the 138KV offsite transmission network, via the Emergency Transformer, and the onsite Class 1 E Distribution System.
  • The 138KV line from Blessing to STP or the 138KV line from Lane City to Bay City (both must be in service).
  • Maintenance activities in the switchyard that could directly cause a Loss of Offsite Power event were prohibited.
  • Maintenance activities that could result in an inoperable open containment penetration were prohibited.

Additional Specific Actions Implemented to Address the Train C ECW Pump Failure

  • In the event of a Loss of Offsite Power (LOOP), the STP procedure for the loss of any 13.8 kV or 4.16 kV Bus provides the specific steps required to cross-connect the Train C Engineered Safety Features (ESF) bus power to an alternate emergency power source by back-feeding through an ESF transformer to the Emergency Bus. Loading on the ESF Bus is restricted to the 480 VAC ESF Load Centers, one centrifugal charging pump and its associated cubicle cooler, two Class 1 E battery chargers and the 25 KVA TMI inverter.
  • Instructions have been prepared for Operations personnel to use a permanently installed piping cross-connect that exists between the ECW trains. These instructions detail actions to take in the event of a LOOP that will allow for one ECW pump to supply cooling water to two Standby Diesel Generators (SDGs). Additional instructions state the necessary steps to allow for one ECW pump to supply cooling water to two trains of Essential Chillers. These actions would only be implemented under emergency conditions.

ADDITIONAL INFORMATION (continued)

  • Surveillance and planned maintenance activities with the potential to cause a reactor/turbine trip or on-site power interruption in both units were suspended to minimize any potential challenge to safety systems.
  • The Control Room personnel were briefed on the above compensatory measures, and Control Room Log entries describing these compensatory measures were made.

Retrieved from "https://kanterella.com/w/index.php?title=05000498/LER-2001-001&oldid=201700"