05000498/LER-2006-002

LER-2006-002, Simultaneous Inoperab lity of Two Essential Chilled Water Trains

South Texas, Unit 1
Event date:	03-31-2006
Report date:	05-30-2006
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
4982006002R00 - NRC Website

A. REPORTABLE EVENT CLASSIFICATION

This event is reportable pursuant to 10 CFR 50.73(a)(2)(i)(B) as a condition prohibited by Technical Specifications (TS). The Limiting Condition for Operation (LCO) for TS 3.7.14 states that at least three independent essential chilled water system loops shall be operable.

The LCO for TS 3.0.3 states:

When a Limiting Condition for Operation is not met, except as provided in the associated ACTION requirements, within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> action shall be initiated to place the unit in a MODE in which the specification does not apply by placing it, as applicable, in:

a. At least HOT STANDBY within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, b. At least HOT SHUTDOWN within the following 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, and c. At least COLD SHUTDOWN within the subsequent 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />.

Unit 1 entered TS 3.0.3 due to simultaneous Train A and Train C essential chiller inoperability and remained in the action longer than one hour.

B. PLANT OPERATING CONDITIONS PRIOR TO THE EVENT

Unit 1 was operating at 100% power at the time of discovery.

EVENT

Essential Chilled Water Train C was inoperable. When Essential Chiller 12A failed to start automatically, there were two inoperable trains of essential chilled water and the plant entered TS 3.0.3.

TIMES

Unit 1 was operating at 100% power on March 31, 2006. Essential Chilled Water Trains A and C were in service. At 10:31, Essential Chilled Water Pump 11B, Essential Chiller 12B, and Electrical Auxiliary Building (EAB) HVAC Train B were started, and EAB HVAC Train C was secured to support a scheduled equipment rotation from Train C to Train B. At 10:51, Essential Chilled Water Train C was declared inoperable because the Essential Chiller 12C breaker failed to open when the control room hand switch was moved to the "stop" position. Standby Diesel Generator 13 was also declared inoperable because an interlock would not allow the diesel generator output breaker to close while the Chiller 12C breaker was closed.

Essential Chiller 12A had been in service since March 18, cycling on and off as necessary to maintain required chilled water temperature. At 11:24, Chiller 12A cycled off. At 12:04, the plant operator discovered that Chiller 12A was not running even though the chilled water outlet temperature was above the chiller automatic start set point. At 12:07, Essential Chilled Water Train A was declared inoperable and the plant entered TS 3.0.3 because two trains of essential chilled water were inoperable simultaneously. At 12:17, the operator gave Chiller 12A a "stop" signal followed by a "start" signal. Chiller 12A started successfully and chilled water outlet temperature began to decrease. At 13:10, an electrical maintenance technician "slightly agitated" the 69X relay for the Chiller 12C breaker and the breaker opened. At 15:06, Essential Chiller 12C was declared operable following satisfactory maintenance and testing.

Thus, the plant was in TS 3.0.3 for approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> and 59 minutes.

Electrical Maintenance performed several operational troubleshooting checks after Chiller 12A was started at 12:17. Following the completion of those checks, Chiller 12A was secured at 21:31 and shutdown troubleshooting checks were performed.

When no fault was identified, the three most likely components were replaced to bound the problem. The chiller was started at 03:11 on April 1 for calibration of the low water temperature switch and post-maintenance testing. Chiller 12A was declared operable at 08:56 on April 1.

OR PROCEDURAL OR PERSONNEL ERROR

Operators recognized that Essential Chiller 12C failed to trip and that Essential Chiller 12A failed to start.

A. FAILURE MODE, MECHANISM, AND EFFECTS OF EACH FAILED COMPONENT

Troubleshooting the Essential Chiller 12C circuit breaker revealed that auxiliary contact 5-7 on relay 69X had not closed as required to cause the breaker to open.

No fault was identified on Chiller 12A, however, the three most likely components were replaced to bound the problem.

B. CAUSE OF EACH COMPONENT OR SYSTEM FAILURE

The Chiller 12C breaker failed to open because auxiliary relay 69X contact 5-7 did not close. Voltage was present at the relay, the coil was energized, and contact 5-7 should have closed. High resistance and contamination found on contact 5-7 prevented the contact from closing, and can be attributed to normal operation of the relay over a long period of time. The apparent cause was lack of instruction to periodically inspect and clean the auxiliary relay.

The exact cause of electrical component failure in Essential Chiller 12A is unknown.

STP has experienced intermittent failures of essential chillers in the past where the precise cause of the problem could not be identified. This is attributed to the chiller analog control circuit not being fault-tolerant, and insufficient data and diagnostic capability provided by the analog control circuit hamper troubleshooting efforts. The low chilled water temperature switch, low chilled water flow switch, and 14TR relay that were removed from Essential Chiller 12A were sent to an off-site vendor for failure evaluation.

OF COMPONENTS WITH MULTIPLE FUNCTIONS

The Essential Chilled Water System provides chilled water to the following safety- related air handling units (AHUs):

n Main supply in Electrical Auxiliary Building (EAB) n Control room envelope in EAB n Electrical penetration space emergency AHUs in EAB n Reactor makeup water pump cubicle in Mechanical Auxiliary Building (MAB) n Boric acid transfer pump cubicle in MAB n Essential chiller area in MAB n Chemical and volume control system valve cubicles in MAB n Radiation monitor room in MAB n Spent fuel pool pump cubicle in Fuel Handling Building (FHB) n Containment sump isolation valve cubicle in FHB n Engineered safety features pump cubicles in FHB Standby Diesel Generator 13 was also declared inoperable because an interlock would not allow the diesel generator output breaker to close while the Chiller 12C breaker was closed.

D. FAILED COMPONENT INFORMATION

Chiller 12C breaker relay 69X is a 125 vdc, type AR, ultra-high-speed style #AR1457C80A01 manufactured by Westinghouse.

A. SAFETY SYSTEM RESPONSES THAT OCCURRED

No safety system responses occurred.

B. DURATION OF SAFETY SYSTEM TRAIN INOPERABILITY

Essential Chilled Water Train C and Standby Diesel Generator 13 were inoperable for approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> and 15 minutes, and Essential Chiller Train A was inoperable for approximately 21 hours2.430556e-4 days <br />0.00583 hours <br />3.472222e-5 weeks <br />7.9905e-6 months <br /> and 55 minutes.

C. SAFETY CONSEQUENCES AND IMPLICATIONS

The risk associated with the Essential Chiller 12A and 12C failures is based upon functional/non-functional times, not upon operable/non-operable times. There were no overlapping non-functional times for the two chillers. However, Diesel Generator 13 was non-functional while the Chiller 12C breaker was incapable of opening from the time of discovery at 10:51 until the Chiller 12C breaker was opened at 13:10. The Incremental Conditional Core Damage Probability (ICCDP) for this event is 5.95E-09 and represents a very small change in risk to the operation of STP. ICLERP was not calculated due to this low ICCDP value.

Run CDF/yr CDF/hr dCDF/hr Time ICCDP ACRUN Zero 7.49E-06 8.55E-10 -- -- -- DG 13 2.82E-05 3.22E-09 2.36E-09 1:13 2.88E-09 DG 13 and Chiller 12A 3.10E-05 3.54E-09 2.69E-09 0.13 5.83E-10 DG 13 2.82E-05 3.22E-09 2.36E-09 0:53 2.09E-09 Chiller 12C 9.58E-06 1.09E-09 2.3904E-10 1:41 4.02E-10 Total ICCDP 5.95E-09 This event did not have an adverse effect on the health and safety of the public.

IV. CAUSE OF THE EVENT

The event being reported is a condition prohibited by TS in that Unit 1 entered TS 3.0.3 due to simultaneous Train A and Train C essential chiller inoperability for longer than one hour. The cause of each train failure was stated in Section II.B above.

V. CORRECTIVE ACTIONS

Corrective actions for Essential Chiller 12A:

• Replaced the low chilled water temperature switch, low chilled water flow switch, and 14TR relay.

• Sent the low chilled water temperature switch, low chilled water flow switch, and 14TR relay that were removed from Essential Chiller 12A to an off-site vendor for failure evaluation.

Corrective actions for Essential Chiller 12C:

• Replaced relay 69X in switchgear El C/9

• Create PMs to clean, inspect, adjust, and burnish contacts on Westinghouse auxiliary relays type AR ultra-high-speed for GQA high and medium components.

• Develop new procedure to clean, inspect, adjust, and burnish the contacts of auxiliary relays type AR ultra-high-speed.

• Clean, inspect, or replace AR ultra-high-speed relays for the component cooling water pumps, containment spray pumps, centrifugal charging pumps, auxiliary feedwater pumps, essential cooling water pumps, and essential chillers in the Class 1 E 4.16 kV switchgear in Units 1 and 2 at the next scheduled maintenance.

Additional Actions STP Nuclear Operating Company is aware that the essential chillers are not meeting reliability expectations. An Essential Chiller Reliability Team was formed on April 3, 2006 to develop and implement a set of compensatory measures, and to make recommendations for long-term essential chiller reliability.

The team's recommendations are based upon review of the following areas:

• Equipment operating practices

• Current and previous revisions to the operating procedures

• Previously identified corrective actions

• Review of maintenance history

• Planned essential chilled water and essential cooling water LCOs

• Previous essential chiller events The team developed the following compensatory actions to minimize essential chiller LCO durations:

1. Perform essential chilled water and essential cooling water LCO activities around the clock to minimize out-of-service time.

2. During the week preceding an essential chilled water LCO, perform a multidiscipline walk down consisting of Engineering, Maintenance, and Operations personnel on the chillers that will be left operating to ensure that they are in good condition.

3. After an essential chiller is started, have a Plant Operator monitor for approximately ten minutes to ensure proper operation. This is because the essential chillers do not cool for about four minutes after they are started. After four minutes of operation take a set of logs on the chiller. This will ensure that the unit is operating correctly prior to leaving the area.

4. Minimize three train essential chiller operations and maximize load on the essential chillers during three train operation by having two trains of EAB and CRE ventilation in service and by starting any other available fans.

5. Evaluate scheduled essential cooling water and essential chilled water LCOs to determine if activities can be scheduled so as to minimize LCO time (group work together).

Emphasize the importance of returning this equipment to service as quickly as possible.

6. During performance of sequencer testing, do not start the third chiller. Secure the train that is to be tested, and be prepared to perform the surveillance actuation in a timely manner. Calculations support running one train of EAB ventilation for short durations ( 7. Implement corrective actions to improve reliability by replacing parts identified during the Common Cause Evaluation performed in 2004.

8. Only start the essential chilled water pump on the idle train to support performance of the surveillance. This will reduce the number of times the essential chillers are cycled.

9. Have contingency work packages planned in advance with parts identified to perform troubleshooting and repair for all of the essential chillers.

The Essential Chiller Reliability Team also determined that the following additional actions will support long-term essential chiller reliability:

1. Evaluate single train operation of EAB ventilation, which would require only one essential chiller to be in service, and ensure that the chiller has sufficient load.

2. Check Essential Chiller 22A compressor oil drain line piping for misalignment in all directions and verify that the 6-1/4 inch dimension has been maintained between the oil drain line flange and the base of the compressor discharge flange.

A common cause analysis of all essential chiller events that occurred in 2005 was completed in February 2006. Additional corrective actions are currently being taken as a result of that Significant Condition Adverse to Quality:

1. Revise planned maintenance activities so that diagnostics testing is performed on the TCMs prior to installation so that manufacturing defects can be found and corrected.

This testing should be performed after completing all modifications that are performed at STP (e.g., installing standoffs and new potentiometers).

2. Provided input to the vendor inspection process concerning the failures of the essential chiller parts in Events 2 and 4 from the 2005 Common Cause and Events 3, 5, 7, 8, and 14 from the 2004 Common Cause so that future inspections of Trentec can identify and prevent the noted manufacturing errors.

3. Changed the scope and intent of essential chiller 78-week PMs to require the check of the terminations be performed last after all other maintenance has been completed and that the PM specify that all terminations on the chiller be checked.

4. Changed the maintenance procedure so that when replacing the hot gas bypass and prerotational vane motors, the motors are inspected/tested in the shop prior to installation. Inspections/tests include checking contact resistance values and verifying they change state, checking for proper rotation, and removing the back cover to verify there is no oil seepage.

5. Evaluate the availability and costs associated with new fault-tolerant digital chiller controllers and present a modification request to the Project Review Team.

6. Establish a Pre-installation Testing/Inspection Program similar to our current Post Maintenance Testing process so that more manufacturing defects can be found prior to the installation of parts in the plant.

VI. PREVIOUS SIMILAR EVENTS

There have been no other violations of TS 3.0.3 in the past three years.

There were fifteen chiller events (not LERs) in 2004, nine in 2005, and three in 2006.