05000382/LER-2009-003
| Docket Numbersequential Revmonth Day Year Year Month Day Yearnumber No Na 05000 | |
| Event date: | 02-22-2010 |
|---|---|
| Report date: | 04-29-2010 |
| Initial Reporting | |
| ENS 45862 | 10 CFR 21.21, Notification of failure to comply or existence of a defect and its evaluation |
| 3822009003R01 - NRC Website | |
REPORTABLE OCCURRENCE
This condition is a Licensee Event Report (LER) to report a 10CFR21 issue of generic interest. This condition meets the reporting criteria of 10CFR21 because reduced reliability of Tyco model' E7024PB relays has been established in Waterford 3 Corrective Action Program, the root cause was determined to be manufacturer fabrication less than adequate, and Waterford 3 has Tyco model E7024PB relays in the warehouse.
The 10CFR21 two day report was made on 4/21/10 at 1035 EDT under notification number EN45862. This is the follow up 30 day written notification.
INITIAL CONDITIONS
This LER identifies multiple Tyco model E7024PB electro-pneumatic time delay relay failures that have occurred at different initial conditions. Of interest, four of the eight failures resulted in Waterford 3 entering into Unplanned Shutdown Limiting Conditions of Operation (LCO) of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> (one additional was a 7 day LCO).
EVENT DESCRIPTION
Waterford 3 has observed a higher than expected failure rate of normally energized Tyco model E7024PB electro-pneumatic time delay relays produced since 2007. Seven of the eight failures were observed on 2007 or newer date code relays and one was observed on a 2004 date code relay. Waterford 3 had 33 Tyco model E7024PB time delay relays installed in the plant. The Agastat E7000 series timing relay consists of two basic operating types, On-Delay and Off-Delay.
The On-Delay models provide a delay period of contact re-positioning on energization and no time delay upon de-energization. The Off-Delay models provide a delay period of contact re-positioning on de-energization and no time delay upon energization. These relays may be ordered with various electrical characteristics suitable for use specific to each application. The early-in-life failures have been observed on Tyco model E7024PB relays designed as an Off-Delay, 4 pole double throw, .
125VDC operating coil, with a time delay range of 0.5 to 5 seconds.
All failed relays were replaced. Post failure bench testing at Waterford 3, Tyco, and independent Failure Analysis has not consistently repeated the failures.
CAUSAL FACTORS
A Root Cause Analysis was completed using multiple investigative techniques (Kepner-Tregoe Problem Solving Analysis and Failure Modes Analysis). The root cause of the early in life relay failures is manufacturer fabrication less than adequate. No common specific fabrication issue was identified that would encompass all the relay failures, but the Root Cause Analysis identified eight potential fabrication issues that could not be eliminated during the analysis process. These include:
- Cracked contact blocks on relays 09150078, 09030683, and 08350005
- Resistance change to contacts due to clamping force on terminal block
- Incorrect recycle spring installed causing 10 CFR Part 21 notification
- Contact block force spring adjustments (Allen set screw) out of tolerance
- Loose terminal points discovered on two spare relays
- Screw inside the lower contact block assembly (contact set 8-12) is not tight / backed out from stationary contact 0921
- Foreign Material in contact block most likely from manufacturing
- Mechanical binding of armature and plunger due to foreign material The eight early-in-life normally energized Tyco model E7024PB electro-pneumatic time delay relay failures with serial numbers 04030244, 08040634, 08040636, 08270633, 09030683, 08350005, 07160240, and 09150078 can be grouped as follows:
- Date code 0804 High Pressure Safety Injection (HPSI) Pump B (09/11/2008) and Chemical Volume Control (CVC) Charging Pump A (10/13/2008) Tyco indicated the failures were due to worker error in fabrication and testing (force balance).
An additional failure occurred during a bench test. This Tyco Relay 08040635 bench test failure (04/13/2009) cause was indeterminate.
- Date code 0903 CVC Charging Pump B (03/13/2009) Southwest Research Institute (SwRI) performed failure analysis and found foreign material in relay.
- Date code 0827 Battery Exhaust Fan A (06/01/2009) Date codes 0808-0835 were identified by the Tyco 10CFR Part 21 Notification dated 11/21/08 as potentially having an incorrect spring installed during fabrication. The 08270633 relay was returned to Tyco per 10CFR21 and Tyco reported the correct springs were installed. Tyco could not duplicate the failure.
CAUSAL FACTORS CONTINUED
- Date code 0835 HPSI Pump B (06/22/2009) The date code 0808 through 0835 relays that were sent back to the vendor per the Part 21 notification, re-certified by the vendor, and returned to Waterford 3 as new relays and the 0835 date code relay failed subsequently. The 08350005 Relay was sent to SwRI for failure analysis with result that the failure could not be duplicated, but minor foreign material was identified.
- Date code 0403 Train B Integrated Emergency Diesel Generator/Engineering Safety Features Test multiple components (05/15/2008) Bench testing of the failed relay by Electrical Maintenance determined that this normally energized relay would change state if actuated shortly after being energized; however, after an extended period of time (-1 hour) the relay would not change state. This was evaluated as evidence of a heat related failure mechanism specific to this relay. This relay was inadvertently disposed and a formal failure analysis could not be performed.
- Date code 0915 Low Pressure Safety Injection (LPSI) Pump B (2/22/10) Tyco witnessed relay test at Waterford 3, which confirmed failure mechanism. Tyco returned to manufacturing facility with relay (09150078) to perform failure analysis. This is in the Waterford 3 corrective action process.
- Date code 0716 Battery Exhaust Fans B (2/27/10).
The relay failure mechanism could not be confirmed at Waterford 3.. Relay (07160240) was returned to Tyco to perform failure analysis. This is in the Waterford 3 corrective action process.
CORRECTIVE ACTIONS
The Root Cause Analysis has identified multiple required corrective actions. The most pertinent are as follows:
o Revised Tyco relay pre-installation testing to include a 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> burn in time and cycling the relay 10 times.
o For installed Tyco relays of specific date codes and risk significance, the testing frequency has been increased to proactively assess reliability.
o Implementing modification to replace all Tyco E7024PB relays in safety related applications.
SAFETY SIGNIFICANCE
The Waterford 3 EOOS model was used to calculate the risk using the Waterford 3 Revision 4 Change 1 Probabilistic Risk Analysis (PRA). The zero maintenance risk for the Revision 4 Change 1 PRA is 2.292E-06/year.
The 2007-2009 Tyco date coded model E7024PB relays installed have 293 successful actuations out of a total of 300 actuations or 97.67% successful actuations. This results in a failure rate of 2.33E-2. A risk analysis was performed using this 2.33E-2 failure rate. The analysis shows that the only relays having potential risk significance are those associated with the HPSI pumps A and B, Emergency Feedwater (EFW) pump B, and Chilled Water (CHW) pump A.
If these relays were to fail, the failure to start would be readily detected by the operators, who are trained and frequently practice in training where the pumps would be started manually using the switches on the main control room panels. Existing procedures require verifying these pumps start and direct manually starting the pumps if they did not start automatically. The operator actions for manually starting these pumps were conservatively assumed to have a 1.1E-2 or 2.8E-3 failure probability, depending on the accident sequence. Using these manual actions, the internal events Core Damage Frequency (CDF) is 2.885E-6/yr, which is an increase of 5.93E-7/yr with respect to the baseline CDF of 2.292E-6/yr. The risk impact for external events is 2.30E-7/yr for a total risk impact of 8.23E-7/yr. All the Tyco model E7024PB relays have been reviewed to confirm that the potential relay failures can be mitigated by simple manual action. Thus, the potential reduced reliability of the relays is not risk significant.
The Public Health and Safety was not adversely affected by this condition. The relay failures were identified during surveillance or bench testing with the redundant train operable. There were no system or train actuations or initiations resulting from these relay failures. There was not a common mode failure mechanism identified for the relay failures. This condition had no impact on industrial safety or radiological safety. No personnel were injured during these events. The relays are installed outside the radiological controlled areas. No dose was received as a result of these failures.
SIMILAR EVENTS
As stated in the event description, Waterford 3 has observed a higher than expected failure rate of normally energized Tyco model E7024PB electro-pneumatic time delay relays produced since 2007.
Seven of the eight failures were observed on 2007 or newer date code relays and one 2004 date code relay. This LER is an outcome of multiple similar events.
ADDITIONAL INFORMATION
None _/