05000382/LER-2012-005
| Waterford 3 Steam Electric Station | |
| Event date: | 06-01-2012 |
|---|---|
| Report date: | 01-25-2013 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition |
| 3822012005R02 - NRC Website | |
REPORTABLE OCCURRENCE
On 6/1/2012 at 04:26 CDT, Operations declared Auxiliary Component Cooling Water (ACCW) [BS] Train A inoperable due to excessive seat leakage through air operated temperature control valve ACC-126A [TCV]. The Component Cooling Water (CCW) [BS] Technical Specification 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> shutdown action was entered. The valve was unable to meet its low leakage safety function when closed by flow demand. Following maintenance, ACCW Train A was declared operable on 6/2/2012 at 14:30 CDT.
Similar conditions have occurred since October of 2011. Engineering evaluation has determined the cause to be wearing of the valve stem bushings for ACC-126A. Since the condition is only recognized when placing the system in standby, the recurring condition created several occasions where past operability of ACCW Train A was not assured. Train B has not exhibited this problem. Analysis of past operability has determined that the impact on safety significance of ACC-126A failing to fully close is minimal.
This condition is being reported pursuant to the requirements of 10 CFR 50.73(a)(2)(i)(B) (Operation Prohibited by Technical Specification), 10 CFR 50.73(a)(2)(11)(B) (Unanalyzed Condition), 10 CFR 50.73(a)(2)(v)(D) (Condition That Could Have Prevented Fulfillment of a Safety Function), and 10 CFR 21 (Reporting of Defect).
INITIAL CONDITIONS
During this time period, Waterford Steam Electric Station Unit 3 (Waterford 3) was operating in Mode 1, stable at or near 100% power. Both trains of ACCW were aligned for normal operations with no plant protection system actuation signals present.
EVENT DESCRIPTION
On 6/1/2012, Operations was transferring the cooling of CCW from the Wet Cooling Tower (WCT) [BS] to the Dry Cooling Tower (DCT) [BS] in preparation of securing ACCW Train A for planned maintenance. As required by procedure, Operators attempted to close air operated valve (AOV) ACC-126A using the manual mode of operation of the valve controller. With the controller output at zero, Operators noted that system flow indicated over 1000 gpm, indicating the valve did not fully close. ACCW Train A was declared inoperable and troubleshooting commenced.
An AOV diagnostic test was performed on ACC-126A, however the valve operator I/P transducer output signal stopped at 13.8 psig (15.00 psig +/- 0.12 psig required to close valve). A second test revealed that the I/P transducer calibration had drifted as compared to the last calibration performed on 10/11/2011 under Work Order (WO) 00292841. The modulation function of the transducer was not affected. Based on this information, Engineering and Maintenance determined that the I/P transducer required replacement. A replacement I/P transducer was not immediately available, therefore the decision was made to install the old I/P transducer from ACC-126A which was removed on 10/11/2011. After successfully calibrating the replacement I/P transducer, ACC-126A was returned to service for post-maintenance testing and subsequently declared operable.
The modulating closed function of ACC-126A is a safety function since it assures adequate WCT water inventory will exist for seven days post Loss of Coolant Accident (LOCA). Engineering Change EC-38218 (performed 6/15/2012 under CR-WF3-2012-2870) documents that 110 gpm of leakage past ACC-126A when the demand flow is 0 gpm would not result in exhausting the design basis WCT water inventory in the event of a worst case design basis accident under worst case meteorological conditions. The calculation established by correlation that the Ultimate Heat Sink (UHS) [BS] remains capable of performing its design basis safety function for the prescribed mission time as long as ACC-126A leakage is limited such that the system jockey pump maintains at least 16.4 psig system high point pressure. Plant operation outside of these conditions would represent an unanalyzed condition.
A review of ACCW Train A system operation for the past three years has found several instances of inability to maintain system high point pressure with the jockey pump, all since October 2011. These resulted in replacement of the I/P transducer. These instances are listed in the following table.
Train A I/P Transducer Replacements 10/10/2011 ACC-126A I/P Transducer was replaced under (W0-292841) due to issues with calibration (CR-WF3-2011-6951).
Removed I/P Transducer: S/N# A776947-6, was provided under P0-10117835 purchased from Callaway. Date of Manufacture April 2006.
Installed I/P Transducer: S/N# 60050-6, was provided under P0-10329852 purchased from Callaway. Date of Manufacture September 2010.
Removed I/P Transducer successfully past bench calibration, and was determined to be acceptable for re-installation.
6/1/2012 ACC-126A I/P Transducer was replaced under WO-312796 as ACC System flow was observed to be 1400-2000 gpm while securing ACCW Train 'A' (CR-WF3-2012-2692).
Removed I/P Transducer: S/N# 60050-6, was provided by Argo Turboserve Corporation under P0-10329852 purchased from Callaway. Date of Manufacture September 2010.
Installed I/P Transducer: S/N# A776947-6, was provided under P0-10117835 purchased from Callaway. Date of Manufacture April 2006. This I/P Transducer was originally removed from ACC-126A under WO-292841 during a complete accessory replacement due to issues during calibration. This I/P Transducer was calibrated afterwards and deemed to be functioning acceptable.
7/5/2012 ACC-126A I/P Transducer was replaced under WO-320439 due to ACCW Train 'A' system high point pressure dropping to 0 psig when ACCW pump A was secured (CR- WF3-2012-3217). Spray flow was verified locally at wet cooling tower A. The jockey pump was running.
Removed I/P Transducer: S/N# A776947-6, was provided under P0-10117835 purchased from Callaway. Date of Manufacture April 2006.
Installed I/P Transducer: S/N# 87678-8, was provided under P0-10351691 purchased from Callaway. Date of Manufacture March 2012.
7/9/2012 ACC-126A I/P Transducer was replaced under WO-319967 due to ACCW Train 'A' high point pressure appeared to be oscillating between 0 PSIG and 5 PSIG for several minutes and slowly rose over the next several minutes with the average of the oscillations around 7 PSIG.
Removed I/P Transducer: S/N# 87678-8, was provided under P0-10351691 purchased from Callaway. Date of Manufacture March 2012.
Installed I/P Transducer: S/N# 87678-9, was provided under P0-10351691 purchased from Callaway. Date of Manufacture March 2012.
7/28/2012 Inspection of ACC-126A valve internals found worn in-board and out-board stem bushings, wom shaft, and damaged t-ring. All wom or damaged parts were replaced under WO 320439. Post maintenance testing and operability surveillances demonstrated that full function and operability had been restored. It was demonstrated during post maintenance testing that ACC-126A closed fully with valve operation in automatic mode.
A review of ACCW Train B system operation for the past three years has found no instances of inability to maintain system high point pressure with the jockey pump. I/P transducers were replaced as planned maintenance, as listed in the following table.
Train B I/P Transducer Replacements 2/17/2012 ACC-126B I/P Transducer was replaced under WO-51673380 during a required PM (CR- WF3-2012-00789).
Removed I/P Transducer: S/N# A903874-2, was provided by Enertech under purchase order WPY00584. Date of Manufacture July 2000.
Installed I/P Transducer: S/N# 60050-5, was provided by Argo Turboserve Corporation under P0-10329852 purchased from Callaway. Date of Manufacture September 2010.
S/N# 60050-5, failed during post maintenance testing.
2/18/2012 ACC-126B I/P Transducer was replaced due to the newly replaced I/P transducer failing during instillation calibration (2/17 and 2/18 replacements are during the same maintenance period).
Installed I/P Transducer: S/N# A903874-2, was provided by Enertech under purchase order WPY00584. Date of Manufacture July 2000.
This condition first occurred on ACC-126A in October, 2011. It has since occurred three times, the most recent on July 9, 2012. Waterford 3 has not been able to obtain replacement I/P transducers which do not exhibit calibration drift at the high end. High end drift affects the full close function. The modulate function is unaffected.
On 7/10/2012, compensatory measures were implemented under CR-WF3-2012-3280 to maintain operability of ACCW Train A by establishing a proceduralized manual action to locally close ACC-126A post accident based on proceduralized criteria. This action, along with previously established system monitoring, provides continued assurance of operability.
In analyzing the multiple attempts to repair ACC-126A, the decision was made to perform an internal valve inspection during a planned train maintenance outage.�This decision was based on as-found diagnostic testing and the inability to achieve ACCW High Point Header Pressure above 17.4 psig using the local valve handwheel.
As a planned contingency discussed by the site during challenge meetings, internal parts were verified on-hand.
The as-found condition of ACC-126A internals were: 1) damaged t-ring, 2) worn in-board and out-board stem bushings, and 3) worn shaft. All worn or damaged items were replaced. Post maintenance testing and operability surveillances demonstrated that full function and operability had been restored. It was demonstrated during post maintenance testing that ACC-126A closed fully with valve operation in automatic mode even though the I/P transducer installed had known drift at the high end of the span. This demonstrates that the known problem found on the I/P transducers did not, in itself, cause excessive seat leakage.
This reported condition is entered into the site corrective action program as CR-WF3-2012-2692 and CR-WF3- 2012-3280.
SYSTEM DESIGN
The ACCW System is a dual train (Trains A and B) system which supplements the heat removal of the Dry Cooling Towers (DCTs) in cooling the CCW System by providing cooling water to the shell side of the CCW Heat Exchangers. The ACCW system removes heat load from the CCW systems (at the CCW heat exchangers) after CCW leaves the DCTs and rejects the heat load to the atmosphere via the Wet Cooling Towers (WCTs). The ACCW system is designed to operate during normal operation and following a design basis event or accident, including Loss of Coolant Accidents (LOCA) or Main Steam Line Break (MSLB) inside the Containment. Upon receipt of a Safety Injection Actuation Signal (SIAS), the ACCW removes sufficient heat from the CCW to ensure that the CCW supply temperature does not exceed its design basis maximum temperature of 115°F by providing sufficient flow through the CCW heat exchanger while minimizing the water consumption of the WCTs. ACCW flow control is facilitated by the throttling of valve ACC-126A(B) via temperature control loop CC ITAC7075A(B).
With the system in standby, operability is maintained by the operation of a jockey pump to preclude void formation.
ACC-126A is a 12 inch, Class 2 butterfly valve. The valve is equipped with a Fisher model 486U-15-40 double acting piston actuator. ACC-126A is a fail-as-is AOV and is provided with a backup nitrogen accumulator, if needed, to perform is safety related functions. The actuator accessories include two Fisher 67CFR filter regulators, a Masoneilan 8005N I/P transducer and a Fisher 3570 positioner. The air supply for ACC-126A is supplied either by the Instrument Air System [LD] or Nitrogen Accumulator [LK].
ACC-126A is controlled by temperature controller CC-ITIC7070A [TC] on Control Room panel CP-33. Two meters and six pushbuttons are mounted on the controller face. The left side meter indicates setpoint temperature over a range of 50 to 130°F. The right side meter shows controller output over a range of 0 to 100 percent. Valve movement is proportional to the output signal. The AUTO/MAN pushbuttons backlight when in automatic/manual control. The SETPT RAISE/LOWER pushbuttons change the setpoint temperature as indicated by the setpoint meter and backlight when setpoint achieves 130/50 °F, respectively. The OUTPUT RAISE/LOWER pushbuttons change controller output in manual only and backlight when output achieves 100/0 percent, respectively.
During normal operation, ACC-126A is in the AUTOMATIC mode of operation. The WCTs are utilized during the summer months while the DCTs are relied upon during the winter months. Typically the temperature setpoint on CP-33 is between 84°F - 87°F during the summer to ensure the DCT Fans do not start. ACC-126A is required to regulate flow through the shell side of the CCW Heat Exchanger such that constant outlet temperature is maintained. ACC-126A is controlled by temperature controller CC ITIC7070A on CP-33. The tube side outlet temperature (CC ITE7075A) [TC] of the CCW heat exchanger controls the valve. If the temperature exceeds the setpoint for CC ITAC7075A1 during normal operation the valve begins to control flow. Under accident conditions, ACC-126A is required to modulate this flow to the WCT until post accident heat can be removed by the DCTs alone. (NOTE: The modulation position of ACC-126A may be at the closed or near closed position reducing evaporative losses to maintain WCT water inventory for the entire accident.) ACC-126A is provided with a nitrogen accumulator to perform this function during a loss of instrument air.
On SIAS, the setpoint is automatically raised to 115°F provided WCT Basin temperature is above 74°F. This change can be observed on the controller setpoint meter. Raising the setpoint preserves WCT Basin inventory during an accident to meet the 30 day post-LOCA requirements.
CAUSAL FACTORS
A Root Cause Evaluation with failure modes analysis was conducted. The evaluation determined that "Preventive Maintenance Less Than Adequate" was the root cause.
The failure of the in-board and out-board bushings was the direct cause of ACC-126A valve internal leakage exceeding the maximum permissible ACCW leakage flow rate. Excessive bushing wear due to ACC-126A's process conditions was not indicated by industry operating experience. An inadequate preventive maintenance strategy ultimately allowed the bushing to degrade to the point where internal valve misalignments occurred, allowing leakage past ACC-126A and degrading other valve internal components. In addition, failure modes analysis and corrective maintenance performed on 02/03/2009 and 02/02/2011 were not able to identify the bushings as degraded components, resulting in unplanned maintenance and unplanned Technical Specification action entries.
The bushings were inspected and a preliminary analysis was completed by Engineering using a new bushing as comparison. The initial examination concluded the most likely failure mode of the degraded bushings was due to excessive wear over an extended period of time.
During this extended period, the misalignment of the disc contacted the t-ring with sufficient force to facilitate the incipient stages of tearing. Therefore, the misalignment of the valve disc, coupled with accelerated t-ring degradation resulted in excessive seat leakage past ACC-126A.
The above discussion explains how the direct cause created the problems with ACC-126A; however, a secondary set of events occurred that masked recognition of the cause of the leakage past ACC-126A during periods of time the t-ring was degraded. The I/P transducer did on some occasions, combined with the degraded bushings, cause ACCW A High Point Header Pressure to drop below acceptable levels. In addition, incomplete testing (other than as-found AOV diagnostic testing and post maintenance testing) resulted in the inability to detect subtle degradation within ACC-126A. These tests would have provided more internal operating experience to revise necessary preventive maintenance to prevent excessive leakage past ACC-126A. An example of this would have been high point header pressure checks. If high point header pressure checks would have occurred, a trend could have been established that could have captured subtle disc misalignments as it occurred.
The Masoneilan 8005N I/P Transducer, based on industry operating experience, is susceptible to foreign material (exceeding 5 microns in size) causing drifting and output pressure lock-ups. In addition, the I/P calibration did not exhibit a complete shift in span, indicating that this drifting may be due to a physical obstruction within the I/P transducer.
CORRECTIVE ACTIONS
- The I/P transducer for ACC-126A was replaced with a calibrated I/P transducer.
- Initiated a compensatory measure for operability establishing criteria for system parameter monitoring which ensures recognition of degradation when placing system in standby.
- Initiated a compensatory measure for operability establishing a proceduralized manual action to locally close ACC-126A post accident based on proceduralized criteria.
- Sent one of the Masoneilan 8005N I/P transducers to the vendor for evaluation.
- Initiated work order to inspect ACC-126B t-ring and bushings based on as-found conditions of ACC-126A.
- Implemented ACCW high point pressure checks with ACC-126A(B) closed to validate that current leakage limits through ACC-126A(B) are not exceeded.
- Utilizing Design Engineering processes, determined a Preventative Maintenance (PM) strategy to inspect/replace valve internals for ACC-126A(B). Initiated follow-up actions to develop and implement the PM strategy.
- Established and documented design basis requirements (i.e. leakage criteria) for ACC-126A(B).
- Initiated a modification to replace the Masoneilan 8005N I/P Transducers with a more reliable/robust transducer.
SAFETY SIGNIFICANCE
Engineering performed an analysis of the effect of seat leakage values for ACC-126A on safety significance over the previous three years in Condition Report CR-WF3-2012-3644. This analysis utilized known plant data to conservatively bin the seat leakage assumed into three groups of assumed maximum leakage flow rates during known periods of time. Actual site meteorological data (wet and dry bulb temperature) was utilized to determine the worst case meteorological conditions for each time period. As the actual meteorological conditions were less than assumed in the design basis, calculations demonstrated that the DCTs were capable of removing more heat from the CCW system under these conditions than assumed in design basis.
The engineering analysis concluded that sufficient WCT inventory would have been available post accident over the entire time period analyzed to meet the WCT inventory design basis. There were no impacts to nuclear, radiological, or industrial safety.
10 CFR 21 REPORTING The failure of the Masoneilan I/P transducers is being conservatively reported pursuant to 10 CFR 21 requirements. While the symptoms experienced and the on-site analysis performed suggests a possible manufacturing deficiency, the vendor did not confirm that a manufacturing deficiency did exist. Additionally, the results from the engineering analysis of safety significance did not meet the criteria for substantial safety hazard.
As described in the Event Description section of this report, the Masoneilan 8005N I/P Transducers experienced multiple cases of calibration drift at the high end of the span and one case of lockup during calibration. These all occurred on valve ACC-126A (Train A).
Waterford 3 sent one of the transducers to the vendor, Argo Turboserve Corporation, for evaluation. The vendor report was received by Waterford 3 Procurement on 6/15/2012. The vendor report documents an as-found condition of a missing interior part, the nozzle ball, and states the item will not function without the ball in place.
The vendor report does not attribute the deficiency as due to manufacturing.
On 07/26/2012, a Masoneilan technician performed troubleshooting on-site on three Masoneilan 8005N I/P Transducers utilizing the Waterford 3 calibration bench. Of the three I/P transducers tested, one exhibited lock ups during calibration (failed calibration), one failed to function completely (non-responsive), and the final transducer calibrated satisfactorily (however, some non-linearity was noted). During the inspection, the field technician identified small metal shavings between the magnetic coil and magnet which may be contributing to the drift; however, this was not confirmed by the vendor. The troubleshooting was indeterminate as to cause.
Identification Data for Failed Masoneilan 8005N I/P Transducers Serial Number Date of Manufacture 87678-8 March 2012 A776947-6 April 2006 60050-6 September 2010 60050-5 September 2010 The root cause evaluation did find that two installed Masoneilan 8005N I/P Transducers did perform satisfactorily over a long period of time. One is still in service, the other removed from service only due to preventive maintenance strategy.
Identification Data for Good Performing Masoneilan 8005N I/P Transducers Serial Number Date of Manufacture Instillation Period A901955-1 December 1997 January 2000 to February 2009 (Train A) A903874-2 July 2000 January 2006 to Present (Train B) To evaluate safety significance for 10 CFR 21 reporting, the same I/P transducer failure of ACC-126B (Train B) due to span drift at the high end must be assumed to have occurred concurrent with the Train A failure.
As documented in the previous Safety Significance section, the Train A I/P transducer failure combined with the actual cause of bushing degradation did not impact nuclear safety. The Event Description section also documents that the known problem found on the I/P transducers did not, in itself, cause seat leakage in excess of design. ACC-126B (Train B) did not exhibit any seat leakage problems during the three years evaluated. If ACC- 126B is assumed to experience the same span drift at the high end for it's UP transducer concurrent with ACC 126A, no increase in seat leakage will occur and Train B WCT inventory remains unaffected.
Therefore, even with the same assumed failure occurring on both safety trains of ACCW, there is no impact to nuclear safety.
SIMILAR EVENTS
Corrective action program data for the past three years was searched for similar failures. The applicable similar events are documented in the Event Description section where they were evaluated for past operability.
ADDITIONAL INFORMATION
Energy industry identification system (El IS) codes are identified in the text within brackets [ ].