IR 05000382/2009010

IR 05000382-09-010; 07/20/2009 - 10/19/2009; Entergy Operations, Inc; Waterford 3 Steam Electric Station; Special Inspection

ML093100238
Person / Time
Site:	Waterford
Issue date:	11/05/2009
From:	Caniano R Division of Reactor Safety IV
To:	Kowalewski J Entergy Operations
References
IR-09-010
Download: ML093100238 (44)

Text

UNITED STATES NUC LE AR RE G UL AT O RY C O M M I S S I O N ber 5, 2009

SUBJECT:

SPECIAL INSPECTION TO EVALUATE MULTIPLE FAILURES OF ENGINEERED SAFETY FEATURES ACUTATION SYSTEM RELAYS AT WATERFORD 3; NRC INSPECTION REPORT 05000382/2009010

Dear Mr. Kowalewski:

On July 24, 2009, the U.S. Nuclear Regulatory Commission (NRC) completed the on-site portion of a Special Inspection at your Waterford 3 facility. The enclosed inspection report documents the inspection results, which were discussed on July 24 at a preliminary exit meeting with you and members of your staff. A re-exit was conducted by telephone on August 24, 2009, with you and members of your staff. A final exit meeting was conducted by telephone on October 19, 2009, with C. Arnone and other members of your staff.

This inspection reviewed the circumstances surrounding six failures of safety-related relays in the ten-month period from September 2008 through June 2009. The inspection team reviewed selected procedures and records, observed activities, and interviewed personnel to assess the effectiveness of actions taken to identify and resolve problems which led to these failures and to assess your actions to address the extent of condition and extent of cause. The team also evaluated your maintenance and testing program for safety-related relays to assess its adequacy and to determine if the program provided appropriately for age management of these relays.

The enclosed report documents two NRC-identified findings of very low safety significance (Green) which were determined to involve violations of NRC requirements. Additionally, a licensee-identified violation which was determined to be of very low safety significance is listed in this report. Because of the very low safety significance associated with these findings and because the findings were entered into your corrective action program, the NRC is treating the NRC-identified findings as non-cited violations consistent with Section VI.A.1 of the NRC Enforcement Policy.

If you contest any violation documented in this report, you should provide a written response within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN.: Document Control Desk, Washington, D.C. 20555-0001; with copies to the Regional Administrator, Region IV; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Senior Resident Inspector at the Waterford 3 facility.

Entergy Operations, Inc. -2-In addition, if you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Senior Resident Inspector at the Waterford 3 facility. The information you provide will be considered in accordance with Inspection Manual Chapter 0305.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Roy J. Caniano, Director Division of Reactor Safety Docket: 50-382 License: NPF-38

Enclosure:

Inspection Report No. 05000382/2009010 w/Attachments

REGION IV==

Docket: 05000382 License: NPF-38 Report: 05000382/2009010 Licensee: Entergy Operations, Inc.

Facility: Waterford Steam Electric Station, Unit 3 Location: Hwy. 18 Killona, LA Dates: July 20 - October 19, 2009 Inspector: E. Ruesch, Reactor Inspector, Plant Support Branch 2 Contractor: G. Nicely Approved By: Neil O'Keefe, Chief Engineering Branch 2 Division of Reactor Safety Attachments:

1. Supplemental Information 2. Special Inspection Team Charter 3. Information Requests-1- Enclosure

SUMMARY OF FINDINGS

IR 05000382/2009010; 07/20/2009 - 10/19/2009; Entergy Operations, Inc; Waterford 3 Steam

Electric Station; Special Inspection This report documents the results of a special inspection to assess the response of the licensee to multiple failures of Tyco Agastat relays. The inspection team was composed of a region-based reactor inspector and an NRC contractor. Two Green non-cited violations were identified and one licensee-identified violation was noted. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609, "Significance Determination Process." Findings for which the significance determination process (SDP) does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Summary of Events and Inspection Results The NRC conducted a special inspection to better understand the circumstances surrounding the recent history of Tyco Agastat E7000-series relay failures at Waterford 3, particularly the six failures of model E7024PB relays which occurred during the ten months from September 2008 to June 2009.

This inspection identified that the licensee's programs for preventive maintenance and performance trending for safety-related relays did not conform to industry standards or adequately address industry operating experience. This condition impeded the licensee's ability to identify degraded performance and failure trends. Relay failures that were identified were not always categorized or evaluated appropriately. The team was concerned that the licensee may not have adequately communicated with the relay manufacturer and the NRC concerning unusual failure rates among some specific lots of relays. The team also concluded that the licensee extended the replacement interval significantly beyond the vendor's recommended life without an adequate technical basis.

Cornerstone: Mitigating Systems

Green.

The team identified a violation of 10 CFR 50, Appendix B, Criterion XVI,

Corrective Action, because the licensee failed to perform a root cause analysis and implement corrective actions to prevent repetition of a significant condition adverse to quality. Specifically, multiple failures of Agastat E7024PB relays that were installed in or designated for safety-related applications constituted a significant condition adverse to quality. The evaluations for the individual relay failures were narrow and did not identify the adverse trend until eight relays had failed in service and seven had failed pre-installation bench tests over a two-year period. The failure of these relays would prevent auto-starting of critical equipment during a loss of offsite power, potentially creating a substantial safety hazard.

The failure of the licensee to recognize that the adverse trend in failures of Agastat E7024PB relays constituted a significant condition adverse to quality, to perform a root cause evaluation, and to initiate corrective actions to prevent recurrence is a performance deficiency. This performance deficiency is more than minor because it is associated with the mitigating systems cornerstone attribute of equipment performance because it affects the availability and reliability of systems which respond to initiating events to prevent undesirable consequences. Using Inspection Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, the performance deficiency was determined to require a Phase 2 analysis because of the potential for a loss of safety system function. A Phase 2/Phase 3 Significance Determination was performed by an NRC Senior Reactor Analyst. Based on a bounding analysis, the analyst quantitatively determined that the actual change in core damage frequency (CDF) due to the increased failure rate of Agastat E7024PB relays would be less than 4.0E-7/year. Therefore, this performance deficiency was determined to be of very low safety significance (Green).

This performance deficiency was determined to have a Problem Identification and Resolution cross-cutting aspect in the Corrective Action Program component because the licensee failed to periodically trend and assess information from the Corrective Action Program and other assessments in the aggregate to identify programmatic and common cause problems. P.1(b) (Section 4OA3.4.2.b.1)

Green.

The team identified a violation of 10 CFR 50, Appendix B, Criterion III,

Design Control, which occurred when the licensee inappropriately extended the service life of 322 safety-related Tyco/Agastat series E7000 time-delay relays without having an adequate technical basis. Specifically, the licensees engineering justification for extending the qualified life beyond the manufacturer-recommended ten years considered only degradation due to thermal aging; it failed to consider other known modes of degradation in accordance with applicable industry standards. Further, the team identified that a performance monitoring program intended to assess any increased failure rate due to this change was inappropriately canceled.

The failure of the licensee to perform a complete analysis of aging effects as required by industry standards in extending the qualified life of safety-related Agastat E7000-series relays is a performance deficiency. This performance deficiency is more than minor because it is associated with the mitigating systems cornerstone attribute of design control because it affects the availability and reliability of systems which respond to initiating events to prevent undesirable consequences. Using Inspection Manual Chapter 0609.04, Phase 1

- Initial Screening and Characterization of Findings, this performance deficiency was determined to be of very low safety significance (Green) because it is a design or qualification deficiency confirmed not to result in loss of operability or functionality. Specifically, only one of the identified relay failures had occurred beyond the recommended 10-year service life; this failure did not result in the failure of multiple redundant trains of safety-related equipment . This finding was determined not to have a cross-cutting aspect because it is not indicative of current licensee performance. (Section 4OA3.5.1.b.1)

Licensee-Identified Violations

A violation of very low safety significance, which was identified by the licensee, has been reviewed by the team. Corrective actions taken or planned by the licensee have been entered into the licensees corrective action program. This violation and the corrective actions are listed in Section 4OA7 of this report.

REPORT DETAILS

4OA3 EVENT FOLLOWUP

1. SPECIAL INSPECTION ACTIVITIES

The NRC conducted this special inspection to review the circumstances surrounding the recent history of Tyco Agastat E7000-series relay failures at Waterford 3, particularly the six failures of model E7024PB relays which occurred during the ten months from September 2008 to June 2009. This review was conducted in order to assess the effectiveness of the licensees actions taken to identify and resolve problems which led to these failures, and to assess the actions taken to address the extent of condition and extent of cause for these failures.

The Special Inspection Team conducted this inspection in accordance with Inspection Procedure 93812, Special Inspection. The team reviewed procedures, logs, corrective action documents, design and maintenance records, and procurement records for the equipment of concern. The team interviewed key station personnel regarding the individual relay failures, the cause evaluations conducted in response to these failures, the maintenance and testing program which provides for age management of the affected relays, and the procurement and bench testing procedures applicable to the affected relays. The team observed a receipt inspection of two relays of the affected model and conducted walk-downs in the locations where the relays are installed to evaluate environmental conditions. The charter for the Special Inspection Team, which describes the scope of the inspection in greater detail, is provided in Attachment 2 to this report.

2. DESCRIPTION OF EVENT AND CHRONOLOGY

2.1 Event Summary On June 22, 2009, during a scheduled performance of surveillance procedure OP-903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification, on Emergency Diesel Generator (EDG) Train B, the B High Pressure Safety Injection (HPSI) pump failed to start as designed on a valid test signal from the Engineered Safety Features Actuation System (ESFAS). Operations personnel successfully started the HPSI Pump B using the manual hand-switch. The failure of the HPSI pump to start automatically resulted in one of the two redundant trains of the Emergency Core Cooling System (ECCS) being declared inoperable. During troubleshooting, the licensee determined that the failure of HPSI Pump B to auto-start was the result of the failure of ESFAS relay SIX7 (EG EREL2392 N) to change state when de-energized. The failed relay was replaced and successfully retested. The licensee initiated condition report CR-WF3-2009-03102 to evaluate the failure.

The SIX7 relay is a normally-energized, time delay on drop-out, Agastat model E7024PB004 relay, manufactured by Tyco Electronics. Bench testing of the failed relay, S/N 08350005, revealed that several internal contacts had failed open and would not close. The failed relay had been bench tested successfully prior to being installed and had performed satisfactorily during one regularly scheduled surveillance test prior to its June 22, 2009, failure. This failed relay had been installed in April 2009 as a corrective action for a previous failure of the SIX7 relay in September 2008.

2.2 Preliminary Risk Significance In the ten months prior to the June 22, 2009, failure of ESFAS relay SIX7, Waterford 3 had experienced five additional failures of Agastat model E7024PB ESFAS timing relays, including one earlier failure of relay SIX7. The NRC performed a preliminary evaluation of the risk significance of these six failures in terms of conditional core damage probability (CCDP). The CCDP analysis assumed the failure to auto-start of all risk-significant components that relied on an Agastat model E7024PB relay during an ESFAS actuation, that in the majority of cases an operator would recognize the failure and manually start the affected equipment, and that the ESFAS system was affected by the increased rate of relay failures for a one-year period. The total incremental CCDP calculated using these assumptions was 3.8E-6. NRC Management Directive 8.3, NRC Incident Investigation Program, directs the consideration of a Special Inspection when the estimated CCDP is greater than or equal to 1E-6.

The NRC determined that due to the repeat nature of this problem, the estimated risk from the affected relays, and the potential for additional common cause failures in the presently installed population of these relays at the facility, a Special Inspection was warranted to assess the cause of the increased relay failure rate, the potential impact of further relay failures, and the corrective action taken. Further, the NRC determined that additional inspection was warranted to determine if the licensees extension of qualified life for Agastat E7000-series relays contributed to the problem.

2.3 Component Description Agastat E7000-series relays are Class 1E electro-pneumatic timing relays currently manufactured by Tyco Electronics, formerly manufactured by Amerace Corporation.

These relays are manufactured with time delay on pick up (on-delay) or time delay on drop out (off-delay), double pole or four-pole double throw, AC- or DC-coil configurations with a variety of time delays. The model which experienced the recent increased failure rate at Waterford 3 is model E7024PB, a time delay on drop out, four-pole double throw, DC-coil relay with a selectable time delay of 0.5 to 5 seconds.

The date of manufacture of these relays is indicated by the first four digits of the relay serial number, with the first two digits indicating the year and the second two digits indicating the week; the date codes referred to in this inspection report correspond to these four digits (e.g., a relay with a date code of 0804 was manufactured the fourth week of 2008). The vendor recommends a replacement schedule of these relays based on a qualified life of 25,000 operations or 10 years from date of manufacture, whichever occurs first.

2.4 Sequence of Events The team developed a detailed sequence of events and organizational response timeline. The timeline includes applicable events and actions before, during, and following the six ESFAS relay failures in 2008 and 2009 as well as events surrounding the licensees extension of qualified life for many Agastat E7000-series relays. The timeline was generated primarily from work orders and maintenance records, procurement records, and corrective action program documents with enhancements and clarifications provided by interviews with licensee staff. A summary list of the failures is

provided in Table 1 on Page 11. This review satisfied inspection activities associated with Special Inspection Charter Item 1.

It is important to note that while the team reviewed other relay failures which occurred during the five-year period which was reviewed, the team determined that the adverse trend for increased failures only applied to E7024PB relays. Therefore, these other failures have not been included in this chronology. The function of each failed relay is listed in Table 1.

October 30, 2003 The failure of EG EREL2392 Q, an Agastat E7024PB relay with date code 9948, resulted in the failure to start of Component Cooling Water (CCW) Pump AB during performance of the Train B Integrated Emergency Diesel Generator (EDG)/Engineered Safety Features Test. The licensee determined the most likely cause of this relay failure to be a cracked coil housing.

February 4, 2004 During calibration of EG EREL2392 K, an Agastat E7024PB relay with date code 9948, the as-found timing test result did not meet the acceptance criteria. After adjustments, consistent timing could not be maintained. The licensee determined the most likely cause of this relay failure to be a cracked coil housing. In its analysis of this failure, the licensee incorrectly stated that this was the first time that a cracked coil housing had caused a relay failure.

May 8, 2006 The spontaneous failure of SSD EREL2390 E, an Agastat E7024PB relay with date code 0144, resulted in an undervoltage annunciator being received in the control room. There were no adverse impacts to equipment operation. The failed relay was replaced; no failure analysis was performed.

December 5, 2006 During calibration of 4KV EREL2388 D, an Agastat E7024PB relay with date code 8640, the as-found timing test result did not meet the acceptance criteria. After adjustments, consistent timing could not be maintained. The failed relay was replaced; no failure analysis was performed.

August 20, 2007 During calibration of EG EREL2342 K, an Agastat E7024PB relay with date code 9948, the relay failed to change state when de-energized. The vendor determined that the most probable cause of the failure was a blackened area on the contact surfaces which may have caused contact failure. No reason for the blackened contacts was identified.

August 30, 2007 An extent of condition review following the August 20, 2007, failure of EG EREL2342 K, determined that of 22 Agastat E7024PB relays with date code 9948 that had been

procured and installed, only two had been shown to demonstrate acceptable performance in service. Of the remaining 20, five had failed in service, 12 had been found out-of-tolerance during the most recent calibration, and three had not been calibrated since installation. Table 2 on page 18 summarizes these failures.

September 12, 2007 During calibration of EG EREL2392 L, an Agastat E7024PB relay with date code 9948, the as-found timing test result did not meet the acceptance criteria. After adjustments, consistent timing could not be maintained. The relay was replaced. During its previous calibration in March 2004, this failed relay had been found out-of-tolerance by over 200 percent (time delay set point is 2.0 seconds +/- 25 percent; as-found delay was 6.03 seconds). The relay had been recalibrated and returned to service. The vendor determined that the most probably cause of the failure was a blackened area on the contact surfaces which may have caused contact failure. No reason for the blackened contacts was identified May 15, 2008 The failure of 4KV EREL2388 D, an Agastat E7024PB relay with date code 0403, caused the failure of an undervoltage signal, resulting in the failure to auto-start of twelve safety-related components and the failure to actuate of one annunciator during a Train B Integrated Emergency Diesel Generator/Engineered Safety Features Test. Bench testing of the failed relay determined that this normally energized relay would change state as designed when de-energized if actuated shortly after being energized.

However, after being energized for greater than approximately one hour, the relay would not change state. This relay had been installed as a replacement following the December 5, 2006, relay failure. Since installation, the relay had performed successfully during one functional test on December 6, 2006. The licensee determined the apparent cause of this failure to be heat degradation in the continuously energized state.

September 11, 2008 The failure of EG EREL2392 N, an Agastat E7024PB relay with date code 0804, resulted in the failure of High Pressure Safety Injection (HPSI) pump B to auto-start during performance of the Train B Emergency Diesel Generator and Subgroup Relay Operability Verification test. This relay was returned to Tyco Electronics for analysis.

October 13, 2008 The failure of EG EREL2343 A, an Agastat E7024PB relay with date code 0804, resulted in the failure of Coolant Charging (CVC) Pump A to auto-start during performance of the Train A Emergency Diesel Generator and Subgroup Relay Operability Verification test.

This relay had been installed on August 20, 2008, and had never successfully performed during a functional test. This relay was returned to Tyco Electronics for analysis.

October 14, 2008 While replacing relay EG EREL2343 A following the October 13, 2008, failure, two of three replacement relays obtained from warehouse stock were determined to be defective due to loose terminal points; both degraded relays had date codes of 0835.

October 27, 2008 As a result of the October 14, 2008, discovery of two relays with defective terminal points in warehouse stock, the licensee placed the four remaining spare Agastat E7024PB relays on hold. The licensees inspection of these spare relays determined that two of the four relays were defective due to loose terminal points; these relays also had date codes of 0835.

November 4, 2008 The licensee issued a Corrective Action Request (CAR) to Tyco Electronics requesting an investigation into the cause, extent of condition, potential reporting, and development of corrective and preventive action for the Agastat E7024PB relays with date code 0835 noted to have defective terminal points.

November 25, 2008 Tyco Electronics issued a 10 CFR 21 report regarding an Agastat E7024PB relay, date code 0813, which was manufactured with an incorrect recycle spring. Coincident with this Part 21 report, on November 26, 2008, Tyco Electronics issued a recall of Agastat E7024PB relays with date codes ranging from 0808 through 0835; this recall included seven relays with a 0835 date code that had been sold to Waterford 3 and two which had been procured by Waterford 3 from another station. Four of these recalled relays were the subject of the November 4, 2008, Corrective Action Request issued to Tyco by the licensee. It was later determined that none of the relays procured by Waterford 3 had an incorrect recycle spring installed.

January 28, 2009 Tyco Electronics provided Waterford 3 with results of the failure analysis performed on the 0804 date code relays which had failed on September 11, 2008, and October 13, 2008. The report concluded that these two relay failures were due to terminal blocks that had been incorrectly adjusted outside of the manufacturing specifications during assembly. Tyco Electronics cited operator error as the cause for this incorrect terminal block adjustment.

March 13, 2009 The failure of CVC EREL2393 A, an Agastat E7024PB relay with date code 0903, resulted in the failure of Coolant Charging Pump B to auto-start during performance of the Train B Emergency Diesel Generator and Subgroup Relay Operability Verification test. The failed relay was sent to Southwest Research Institute for failure analysis. The laboratory determined that the most probable cause of failure was foreign material in the relay introduced during the manufacturing process. This relay had been installed on January 27, 2009, and had never successfully performed during a functional test.

April 13, 2009 While performing a pre-installation bench test on an Agastat E7024PB relay, date code 0804, one of two sets of contacts failed to change state on initial energization. This relay

had been previously installed in the plant in the Charging Pump B application (EG EREL2393 A) and removed on January 27, 2009, due to concerns with date code 0804 relays. The relay had been sent back to the vendor for testing and returned to Waterford 3 on March 31, 2009, with no problems noted by the vendor.

June 1, 2009 The failure of EG EREL2342 L, an Agastat E7024PB relay with date code 0827, resulted in the failure of Train A Battery Room Exhaust Fans to auto-start during the performance of the Train A Emergency Diesel Generator and Subgroup Relay Operability Verification test. In its reportability review for this failure, the licensee noted, the failed relay is manufactured by AmeraceA search of CRs since June 2000 using Amerace revealed only 2 CRs and neither identified a relay failure. The team determined that this search was inadequate because the relays were no longer manufactured by Amerace; this narrowly scoped search may have resulted in the failure of the licensee to identify the adverse trend in relay failures.

June 10, 2009 The licensee initiated Condition Report CR-WF3-2009-02862 which identified that several Condition Reports associated with relay failures had not been appropriately evaluated to determine if the conditions were Maintenance Rule functional failures. See Section 4OA7 for further discussion.

June 22, 2009 The failure of EG EREL2392 N, an Agastat E7024PB relay with date code 0835, resulted in the failure of High Pressure Safety Injection Pump B to auto-start during performance of the Train B Emergency Diesel Generator and Subgroup Relay Operability Verification test. The failed relay had previously been sent back to the manufacturer for testing following Tyco Electronicss November 26, 2008, recall of Agastat E7024PB relays. The relay had been returned to Waterford 3 with no problems noted by the vendor.

July 8, 2009 The licensee initiated a Condition Report CR-WF3-2009-03448 identifying an adverse trend of six failures since 2008 in which Agastat E7024PB relays failed to change state when de-energized.

July 10, 2009 During a bench test of two Agastat E7024PB relays with date codes of 0850 and 0851, the relays were found to be defective. The relays failed to change state after being energized for 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />. The licensee determined this condition to be not reportable based on testing methodology.

July 20, 2009 During a Quality Assurance observation, the licensee identified that they had failed to properly evaluate the impact of six failures of Agastat E7024PB relays on maintenance

effectiveness. Further, the licensee identified that they had failed to take appropriate corrective actions or correctly identify the extent of condition after this failure was identified on June 10, 2009. See Section 4OA7 for further discussion.

Agastat E7024PB Relay Failures October 2003 - July 2009 Date Failure Mode Code Affected Equipment (Known or Most Probable)

Oct 30, 2003 9948 CCW Pump AB Cracked Coil Housing Feb 4, 2004 9948 Switchgear Air Handlers Cracked Coil Housing May 8, 2006 0144 Undervoltage Annunciator Not analyzed Dec 5, 2006 8640 4KV Bus Undervoltage Not analyzed Aug 20, 2007 9948 Switchgear Air Handlers Blackened Contacts Sep 12, 2007 9948 Battery Room Exhaust Fans Blackened Contacts May 15, 2008 0403 4KV Bus Undervoltage Heat Degradation Sep 11, 2008 0804 HPSI Pump B Misadjusted Terminal Blocks Oct 13, 2008 0804 CVC Pump A Misadjusted Terminal Blocks Oct 14, 2008 0835 Not installed; bench test failure Loose Terminal Points Oct 14, 2008 0835 Not installed; bench test failure Loose Terminal Points Oct 27, 2008 0835 Not installed; bench test failure Loose Terminal Points Oct 27, 2008 0835 Not installed; bench test failure Loose Terminal Points Mar 13, 2009 0903 CVC Pump B Foreign Material Apr 13, 2009 0804 Bench test failure; previously Not analyzed installed in CVC Pump B Jun 1, 2009 0827 Battery Room Exhaust Fans Analysis in progress Jun 22, 2009 0835 HPSI Pump B Analysis in progress Jul 10, 2009 0850 Not installed; bench test failure Analysis in progress Jul 10, 2009 0851 Not installed; bench test failure Analysis in progress Abbreviations:

CCW - Component Cooling Water A relay with Date Code XXYY HPSI - High Pressure Safety Injection was manufactured the YYth week CVC - Coolant Charging of year XX.

Table 1

3. ROOT CAUSES OF RELAY FAILURES

3.1 Determination of Root Causes

a. Inspection Scope

The team reviewed the licensees root cause evaluation of the adverse trend in failures of Agastat E7024PB relays, completed on August 5, 2009. The team examined the licensees methods for determining the causes of the recent increased failure rate and the root and contributing causes identified. This review, together with the below review of corrective actions, satisfied inspection activities associated with Special Inspection Charter Item 2.

b. Discussion The licensee performed a root cause analysis which examined the six in-service failures of Agastat E7024PB relays between September 2008 and June 2009. The licensee identified one root cause and two contributing causes for these six failures. Several of the prior-to-installation relay failures which occurred during the same time period were listed in the licensees report and were noted as evidence of the root cause, but were not specifically addressed as part of the causal analysis. Failures which occurred prior to September 2008 were not discussed.

The licensee determined that since 2008, Tyco Electronics produced E7024PB relays with manufacturing variations that affected performance after a period of energization.

Therefore, the licensee determined the root cause of the six in-service failures to be manufacturer fabrication that is less than adequate to preclude component failure.

The licensee identified a contributing cause of inappropriate Condition Report classification based on the significance of the failures. The licensee determined, using guidance in Corrective Action Program procedures, that the adverse trend of Agastat E7024PB relay failures should have been identified on March 17, 2009, during a review of Condition Report CR-WF3-2009-1190 associated with the March 13, 2009, relay failure which resulted in the failure of Coolant Charging Pump B to auto-start. The licensees failure to identify the adverse trend as a significant condition adverse to quality is further discussed in Section 4OA3.4.2.b.1 of this report.

The team determined that the licensees root cause analysis was of adequate breadth and depth to sufficiently identify causal factors and to draw reasonable conclusions as to the causes of the six in-service failures. However, the scope of the evaluation did not include all of the relay failures in the time period, pre-installation bench test failures were not addressed, and not all identified failures were analyzed.

3.2 Corrective Actions

a. Inspection Scope

The team reviewed the corrective actions which the licensee initiated to correct the root and contributing causes of the adverse trend in Agastat E7024PB relay failures and to prevent recurrence of the condition. This review, together with the above review of the causes, satisfied inspection activities associated with Special Inspection Charter Item 2.

b. Discussion The licensee identified 12 immediate corrective actions, three interim actions, 20 short term corrective actions, and six long term corrective actions to correct and prevent recurrence of the identified increased rate of Agastat E7024PB relay failures. Short term corrective actions included the development of a modification to replace all identified high priority Agastat E7024 normally energized relays with relays of a different design.

Long term corrective actions included the development and implementation of a modification to replace all Agastat E7024 relays.

The team determined that the corrective actions and corrective actions to prevent recurrence identified by the licensee were appropriate both in scope and in timeliness.

The team determined that the planned effectiveness review, required by the licensees Corrective Action Program for all corrective actions to prevent recurrence, would be adequate to verify that the implemented corrective actions resulted in the desired outcome; the success criteria associated with this effectiveness review were adequate to ensure prompt identification of a continued or reemerging adverse trend.

4. EXTENT OF CONDITION

4.1 Failure and Reportability Analyses

a. Inspection Scope

The team reviewed the failure modes identified for each of the 19 hard failures of Agastat E7024PB relays that occurred from October 2003 through July 2009. The team assessed the scope of the licensees cause evaluation, including whether common-cause failure modes were identified, and the corrective actions taken for any identified common-cause failures. Additionally, the team assessed whether any of these failures should have resulted in notifications to the NRC. This review satisfied inspection activities associated with Special Inspection Charter Item 7.

b. Discussion Of the 19 hard failures of Agastat E7024PB relays that occurred from October 2003 through July 2009, 12 occurred in service. The remaining seven failed during bench testing prior to installation in the plant.

In-Service Failures At the end of the on-site portion of this inspection, cause analyses had been completed for eight of the 12 relays which had failed in service and cause analyses were in progress for the two most recent failures; two of the failures were not analyzed. Failure analyses were performed variously by the licensee, by Tyco Electronics, and by Southwest Research Institute. The results of these analyses identified the most likely cause of two failures to be a cracked housing, two to be blackened contacts, one to be heat degradation, two to be terminal blocks incorrectly adjusted at the factory, and one to be foreign material internal to the relay which had most likely been introduced during the manufacturing process. See Table 1 for a list of these failures.

The two relays that failed due to a cracked housing and the two with blackened contacts shared a date code of 9948. After the third of these four failures was identified on August 20, 2007, and entered into the corrective action program as CR-WF3-2007-02961, the licensee performed an extent of condition review. This review identified that the 22 Agastat E7024PB relays which had been procured by Waterford 3 with date code 9948 had demonstrated a trend of unreliability. Of the 19 of these that had been installed in the plant for at least one calibration interval, five had failed in service (two of these five failures occurred prior to the period covered by this report) and 12 had experienced timing set-point drift out of specification during the most recent calibration interval. A summary of the date code 9948 relays is provided in Table 2. The licensee began a campaign to replace the 17 date code 9948 relays still installed in the plant. On September 12, 2007, an additional in-service failure occurred during a functional test of one of the 12 relays that had previously exhibited a large set-point drift. Prior to identifying the negative trend, the licensee had determined the condition identified in CR-WF3-2007-02961 to be not reportable; reportability was not revisited following identification of the adverse trend.

Two Agastat E7024PB relays with date code 0804 which failed on September 11 and October 13, 2008, were returned to the manufacturer for analysis. The manufacturer determined that the most likely cause of this failure was terminal blocks which had been incorrectly adjusted at the factory. The manufacturer and the licensee concluded that this condition was not reportable under the requirements of 10 CFR Part 21 because they believed operator error during assembly did not constitute a defect. The team was concerned that errors during the manufacturing process of these relays which resulted in failures may have constituted a manufacturing defect which required reporting to the NRC per 10 CFR Part 21. This issue is further discussed in Section 4OA3.4.1.b.1 of this report.

Failures Prior to Installation When replacing the Agastat E7024 relay which failed on October 13, 2008, the replacement relay obtained from the warehouse by maintenance personnel was noted to have a loose terminal point. Two more relays were obtained from the warehouse; one of these also had a bad terminal point. This condition was documented in the Corrective Action Program as CR-WF3-2008-04782. The licensee quarantined the four Agastat E7024PB relays remaining in the warehouse pending inspection. Following this inspection, two of these relays were identified to have the same defective terminal points. The licensee issued a Corrective Action Request to Tyco Electronics requesting an investigation into the cause, extent of condition, potential 10 CFR Part 21 reporting, and development of corrective and preventive actions. The four defective relays, all with date code 0835, were returned to Tyco Electronics. The April 13, 2009, failure of a date code 0804 relay was not analyzed by the licensee for failure mode.

Two failures of Agastat E7024PB relays to change state during pre-installation testing occurred on July 10, 2009, during a bench test designed to challenge the potential heat-related failure mechanism of these relays prior to installation. This test had recently been developed in response to the previous relay failures and was based on thermography scans taken in the B Auxiliary Relay Cabinet in June 2009. Because the maximum relay coil operating temperature was measured to be between 150 and 165 degrees Fahrenheit, the test was designed to conservatively raise the ambient temperature of the relay operating environment such that the coil temperature was

maintained greater than 167 degrees Fahrenheit during the test. This required that the ambient temperature be raised to 153 degrees Fahrenheit for the duration of the 8-hour test. These two test failures were entered into the Corrective Action Program as CR-WF3-2009-03502. During its cause analysis following these test failures, the licensee concluded that the failures were caused by the elevated ambient temperatures of the test environment. The licensee further concluded that the test did not appropriately simulate operating conditions for these relays because under normal conditions, only the coils operate at the temperatures created by the test environment; the contact blocks operate at significantly lower temperatures. The team identified the following problems with the licensees analysis:

• The licensee noted that the maximum ambient temperature expected under normal and accident conditions is 104 degrees Fahrenheit and determined that therefore, in future testing, the ambient temperature should be limited to 104 degrees Fahrenheit and the coil temperatures monitored for information only.

However, the licensee based this on temperatures in the room, not inside the unventilated cabinet which contained multiple heat-generating electronic components. Seasonal temperature variations were also not considered.

• The licensee noted that the Tyco Agastat E7024PB product specification manual lists the ambient temperature for the normal operating environment of the relays as70-104 degrees Fahrenheit. However, the licensee failed to note that the specifications also list a maximum temperature of 156 degrees Fahrenheit for the qualified life of the relay. During the licensees test, the ambient temperature did not exceed this maximum temperature.

• The licensee concluded that consideration of 10 CFR Part 21 applicability was not necessary because the failures occurred after testing performed outside of design specifications. However, as demonstrated above, the relays were tested within their worst-case design parameters and should not have been expected to fail.

At the end of the on-site portion of this inspection, the licensee planned to send one of the two relays which failed during the July 10, 2009, test to Tyco Electronics for further analysis.

.1 Potentially Inadequate Evaluations for Reportability

Introduction.

The team identified an unresolved item (URI) to further evaluate whether two examples where the licensee's evaluations of relay failures and decisions not to report the conditions to the NRC constituted a violation. Specifically, on September 11, 2007, the licensee identified that 17 Agastat E7024PB relays with date code 9948 had either failed in service or exhibited excessive set point drift. On October 14, 2008, the licensee identified four defective Agastat E7024PB relays with date code 0804 during pre-installation bench testing. The licensee did not adequately evaluate to determine whether these failures were required to be reported to the NRC.

Description.

The team identified two examples where the licensees evaluations of relay failures may not have been adequate to address NRC reporting requirements.

Example 1 On August 20, 2007, while performing calibration on an Agastat E7024PB relay associated with safety-related ventilation systems, the licensee found that relay contacts would not operate as designed. On September 11, 2007, during an extent of condition review following this failure, the licensee identified that Agastat E7000-series relays with a date code of 9948 had repeatedly been identified as being out-of-tolerance during routine calibrations and had experienced five hard failures. Out of 22 purchased by Waterford 3, only two relays from this date code had demonstrated acceptable performance. Table 2 summarizes the performance problems for the relays in this group. Based on this information, the licensee concluded that relays with this date code were unreliable and initiated action to replace all date code 9948 relays at their next scheduled calibration.

Prior to identifying this adverse trend, the licensee had determined the August 20, 2007, failure which initiated the extent of condition review to be not reportable. The licensee failed to reconsider the condition for reportability after recognizing the full extent of the problem.

The team determined that these parts were basic components as described in 10 CFR 21.3. Procurement documents for this group of relays specified that these relays had a qualified life of 10 years, and would maintain a setpoint within plus or minus 10 percent. Since the relays exhibited unacceptable performance well before reaching 10 years, it appeared that the licensee's evaluation should have concluded that the condition constituted a deviation as defined in 10 CFR 21.3.

The team determined that these relays were installed in both trains of the Engineered Safety Features Actuation System (ESFAS), with functions to auto-start safety-related mitigating systems. In some cases, date code 9948 relays with unacceptable performance were installed in redundant equipment (e.g. both trains of high pressure safety injection pumps). Therefore, it was possible that this condition may have involved a substantial safety hazard as defined in 10 CFR 21.3 and NUREG 0302, "Remarks Presented (Questions /Answers Discussed) at Public Meetings to Discuss Regulations (10 CFR Part 21) for reporting of Defects and Noncompliance," Revision 1. It was also possible that this condition may have involved a potential for loss of safety function as described in 10 CFR 50.72(b)(3)(v) and 10 CFR 50.73(a)(2)(v).

Example 2 On October 14, 2008, while performing pre-installation bench testing on a spare relay intended to replace a failed Agastat E7024PB relay, maintenance personnel noted that the replacement relay obtained from the warehouse had a loose terminal point. Three of the remaining six spare relays also had a bad terminal point. The licensee identified these relays as defective and returned them to Tyco Electronics for a cause evaluation. All four defective relays shared a date code of 0835.

On November 11, 2008, the licensee incorrectly determined that the defects in four 0835 date code relays were the same condition as had been previously captured in Condition Report CR-WF3-2008-4765 which had been written for

date code 0804 relays which failed on September 11 and October 13, 2008. The licensee documented in Condition Report CR-WF3-2008-4782 that a reportability evaluation for the date code 0804 relays would satisfy the requirements to evaluate the date code 0835 relays for reportability. Subsequently on January 28, 2009, Tyco Electronics concluded that the cause of the failures of the two 0804 date code relays was misadjusted terminal blocks. Since this was a different condition that the loose terminal points noted on four date code 0835 relays, the licensee appears to have failed to evaluate the date code 0835 condition for reportability.

The relays in this example were designated for the same uses as those described in Example 1, and therefore also were basic components. The failure of these relays prior to use also appeared to constitute a deviation which had the potential to involve a substantial safety hazard. However, because these relays had never been installed in the plant, this condition did not involve a potential for loss of safety function as described in 10 CFR 50.72(b)(3)(v) and 10 CFR 50.73(a)(2)(v).

The team compared the licensee's reportability evaluation procedures and determined that the licensee's procedures did not incorporate all applicable NRC guidance that related to these examples. In particular:

• The licensee did not include consideration of single failures in addition to the condition being evaluated

• The procedure guidance evaluated whether the failure had created a substantial safety hazard of a loss of safety function, rather than whether the condition could have created one

• The procedure did not address whether the failure involved a loss of the potential loss of accident mitigation functions when considering whether a substantial safety hazard was involved

• The procedures did not address evaluating spare parts which were found to have problems during pre-installations checks to determine whether a deviation from technical requirements was involved On September 3, 2009, the licensee provided three position papers to the NRC documenting positions on Part 21 reporting (reference ML092520027). The licensee stated that they had met their responsibilities based on guidance in NUREG 1022, Event Reporting Guidelines: 10 CFR 50.72 and 50.73, Revision 2 and 10 CFR 21.21.

This finding is being treated as an unresolved item (URI) pending determination by the NRC staff whether the licensees failure to evaluate and report the trend of Agastat E7024PB relay unreliability constituted a violation of 10 CFR 21: URI 05000382/2009010-01, Failures to Evaluate Adverse Conditions for Reportability to the NRC.

Agastat E7024PB Date Code 9948 Relay Failures as of September 11, 2007 Serial Number Equipment ID Affected Equipment Failure Type 99480080 EG EREL2343 B Containment coolers, Trains A/C Out of tolerance 99480081 EG EREL2342 M Chilled water, Train A Failed and replaced 99480082 EG EREL2392 H Control room air handler, Train B Out of tolerance 99480083 EG EREL2392 N HPSI pumps B & AB Not yet recalibrated 99480084 EG EREL2342 K Switchgear fans, Train A Failed and replaced 99480085 EG EREL2342 L Battery exhaust fans, Train A Out of tolerance 99480086 EG EREL2392 L Battery exhaust fans, Train B Out of tolerance 99480087 EG EREL2392 J Ventilation fans, Train B Out of tolerance 99480088 EG EREL2342 M Chilled water, Train A Out of tolerance 99480089 EG EREL2392 K Switchgear fans, Train B Failed and replaced 99480090 EG EREL2392 R CCW pump B Calibration satisfactory 99480091 EG EREL2343 E Undervoltage; Boric acid pumps Out of tolerance 99480092 EG EREL2392 E Containment spray pump B Failed and replaced 99480093 EG EREL2342 P HPSI pumps A & AB Out of tolerance 99480094 EG EREL2342 R CCW pump A Out of tolerance 99480095 EG EREL2392 M LPSI pump B Out of tolerance 99480096 EG EREL2343 A CVC pump A & AB Calibration satisfactory 99480097 EG EREL2393 C EFW pump B Out of tolerance 99480098 EG EREL2393 A CVC pump B Out of tolerance 99480099 EG EREL2392 Q CCW pump B & AB Failed and replaced 99480100 EG EREL2393 D Containment coolers, Trains B/D Not yet recalibrated 99480101 EG EREL2392 Q CCW pump B & AB Not yet recalibrated Abbreviations:

CCW - Component Cooling Water Data are from Condition Report CVC - Coolant Charging CR-WF3-2007-02961.

EFW - Emergency Feed Water HPSI - High Pressure Safety Injection LPSI - Low Pressure Safety Injection Table 2

4.2 Impact of Relay Failures

a. Inspection Scope

The team performed an assessment of the circumstances of each relay failure. This included a review of plant conditions and surveillance tests in progress at the time of each failure, the consequences of each relay failure, the time since the previous successful demonstration of the safety function of each failed relay, and the potential for operator recovery actions from each failure. This review satisfied inspection activities associated with Special Inspection Charter Item 6.

b. Discussion Prior to the licensee identifying an overall adverse trend in relay failures on July 8, 2009, the licensee evaluated the failures as isolated problems. The impact of the failures in the aggregate was not considered.

The team noted that the licensee had no trending mechanism to identify a pattern of relay failures. Failures were trended only with the supported system, spreading the failures among multiple systems. A previous program for trending the performance of relays was discontinued in 2001, which contributed to the failure of the licensee to promptly identify the failure trend. As part of the discussions with the team, licensee personnel indicated that their Maintenance Rule Program had failed to record any of these relay failures as actual or potential functional failures. This is further discussed in Section 4OA7 of this report.

.1 Failure to Identify an Adverse Trend in Failures of Time-Delay Relays

Introduction.

The team identified a violation of 10 CFR 50, Appendix B, Criterion XVI, Corrective Action, because the licensee failed to perform a root cause analysis and implement corrective actions to prevent repetition of a significant condition adverse to quality. Specifically, multiple failures of Agastat E7024PB relays that were installed in or designated for safety-related applications constituted a significant condition adverse to quality. The evaluations for the individual relay failures were narrow and did not identify the adverse trend until eight relays had failed in service and seven had failed pre-installation bench tests over a two-year period. The failure of these relays would prevent auto-starting of critical equipment during a loss of offsite power, potentially creating a substantial safety hazard.

Description.

On May 15, 2008, during performance of a Loss of Offsite Power Test, several components failed to actuate as designed. The cause of these failures was determined to be a failed Agastat E7024PB relay in the Engineered Safety Features (ESF) Actuation System. Condition Report CR-WF3-2008-2352 was initiated to document this condition. The preliminary cause of the failure was listed as heat-related degradation; however, contrary to licensee procedural requirements, the relay was disposed of prior to a formal failure analysis being performed.

On September 11, 2008, during the performance of the Emergency Diesel Generator and Subgroup Relay Operability Verification, High Pressure Safety Injection Pump B failed to auto-start due to the failure of an Agastat E7024PB relay in the ESF Actuation System. Condition Report CR-WF3-2008-4304 was initiated to document this failure.

The cause of the failure was subsequently determined to be terminal blocks that had been improperly adjusted at the factory.

On October 13, 2008, during the performance of the ESFAS Subgroup Relay Test -

Operating, Charging Pump A failed to auto-start due to the failure of an Agastat E7024PB relay. The licensee initiated Condition Report CR-WF3-2008-4765 to document this failure. The cause of this failure was subsequently determined to be terminal blocks that had been improperly adjusted at the factory.

On March 13, 2009, during the performance of the ESFAS Subgroup Relay Test -

Operating, Charging Pump B failed to auto-start due to the failure of an Agastat E7024PB relay. The licensee initiated Condition Report CR-WF3-2009-1190 to document this condition. The most probable cause of this failure was subsequently determined to be foreign material in the relay assembly. On March 17, 2009, the licensee classified this condition report as Category B, requiring a lower-tier apparent cause evaluation. However, the licensee later concluded that based on the requirements of the Corrective Action Program procedures, the Condition Report should have been classified as a Category A, indicating a significant condition adverse to quality and requiring performance of a root cause analysis and initiation of corrective actions to prevent recurrence.

On June 1, 2009, during the performance of ESFAS Subgroup Relay Test - Operating, Train A Battery Exhaust Fans failed to auto-start due to the failure of an Agastat E7024PB relay. This failure was documented in Condition Report CR-WF3-2009-2674.

Failure analysis was in progress at the time of this inspection.

On June 22, 2009, during the performance of Emergency Diesel Generator and Subgroup Relay Operability Verification, High Pressure Safety Injection Pump B failed to auto-start due to the failure of an Agastat E7024PB relay. This failure was documented in Condition Report CR-WF3-2009-3102. Failure analysis was in progress at the time of this inspection. On June 25, 2009, the Condition Report Review Group inappropriately classified this condition report as Category B. Based on requirements of the licensees Corrective Action Program procedures, the licensee later concluded that the Condition Report should have been classified as a Category A, indicating a significant condition adverse to quality and requiring a root cause analysis and corrective actions to prevent recurrence.

On July 8, 2009, the licensee identified an adverse trend in failures of Agastat E7024PB relays as a significant condition adverse to quality. This adverse trend was documented in Condition Report CR-WF3-2009-3448 and was classified as Category A in accordance with the licensees Corrective Action Program procedural requirements. The team determined that the licensee should have identified this adverse trend by October 2008.

Analysis.

The failure of the licensee to promptly recognize that the adverse trend in failures of Agastat E7024PB relays constituted a significant condition adverse to quality, to perform a root cause evaluation, and to initiate corrective actions to prevent recurrence is a performance deficiency. This performance deficiency is more than minor because it is associated with the mitigating systems cornerstone attribute of equipment performance because it affects the availability and reliability of systems which respond to initiating events to prevent undesirable consequences. Using Inspection Manual

Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, the performance deficiency was determined to require a Phase 2 analysis because of the potential for a loss of safety system function. A Phase 2/Phase 3 Significance Determination was performed by an NRC Senior Reactor Analyst. The analyst evaluated the actual failures during the previous 12 months both separately and collectively, considered the potential failure of all equipment vulnerable to the failures, and analyzed dual failures that had the potential to prevent the EDG sequencer from functioning. Based on a bounding analysis, giving credit for operator recovery actions, the analyst quantitatively determined that the actual change in core damage frequency (CDF) due to the increased failure rate of Agastat E7024PB relays would be less than 4.0E-7/year. Further, the analyst determined that external events would not have contributed a significant amount to the overall risk and that the core damage scenarios did not involve more than a negligible potential for a large early release. Therefore, this performance deficiency was determined to be of very low safety significance (Green).

This performance deficiency was determined to have a Problem Identification and Resolution cross-cutting aspect in the Corrective Action Program component because the licensee failed to periodically trend and assess information from the Corrective Action Program and other assessments in the aggregate to identify programmatic and common cause problems. P.1(b)

Enforcement.

Title 10 CFR 50, Appendix B, Criterion XVI, Corrective Action, requires, in part, that measures be established to assure that conditions adverse to quality are promptly identified and corrected. In the case of significant conditions adverse to quality, these measures shall assure the cause of the condition is determined and corrective action is taken to preclude repetition. The identification of the significant condition adverse to quality, the cause of the condition, and the corrective action taken shall be documented and reported to appropriate levels of management. Contrary to the above, from September 2008 through June 2009, Waterford 3 failed to identify as a significant condition adverse to quality, to assure the cause of the condition was determined, and to ensure corrective actions were taken to preclude repetition. Specifically, multiple failures of Agastat E7024PB relays between May 15, 2008, and June 22, 2009, intended to actuate safety-related mitigating systems, were treated as individual conditions adverse to quality. Because this finding was determined to be of very low safety significance (Green) and was entered into the licensees corrective action program as CR-WF3-2009-3448, this violation is being treated as a non-cited violation consistent with the NRC Enforcement Policy: NCV 05000382/2009010-02, Failure to identify an adverse trend in failures of time-delay relays.

5. AGE MANAGEMENT OF RELAYS

5.1 Agastat E7000-Series Relay Service Life

a. Inspection Scope

The team reviewed the licensees technical justification for extension of the manufacturer-recommended 10-year service life for Agastat E7000-series relays. The team reviewed vendor documentation and testing methodology used for the basis of its recommended service life and compared it to the licensees justification for life extension. The team evaluated the potential impact of this life extension on individual

component performance and on system reliability. This review and evaluation satisfied inspection activities associated with Special Inspection Charter Item 3.

b. Discussion Vendor specifications list the qualified life of Agastat E7000-series relays as 25,000 operations or 10 years from the date of manufacture, whichever occurs first. The manufacturer recommends relay replacement at the end of this 10-year qualified life, and performance of periodic monitoring of relays during the 10-year life to monitor for substantial changes in timing delay which may indicate premature end-of-life.

The industry standard for determining the qualified life of Class 1E electrical equipment used in nuclear plants, IEEE 323-1974, IEEE Standard for Qualifying Class 1E Equipment for Nuclear Power Generating Stations, states that electromechanical equipment such as relays shall be operated during testing to simulate the expected mechanical wear and electrical contact degradation which would occur over the qualified life of the equipment. NUREG-0588, Interim Staff Position on Environment Qualification of Safety-Related Electrical Equipment, states that the degrading influences discussed in this IEEE Standard and the electrical and mechanical stresses associated with cyclic operation of equipment should be considered and included as part of the aging programs. It further states that equipment located in general plant areas outside containment where equipment is not subjected to a design basis accident environment should be qualified to the normal and abnormal range of environmental conditions postulated to occur at the equipment location.

In April 1995, the licensee completed Problem Evaluation/Information Request (PEIR)

DE-57 which recommended an extension of service life for all Agastat E7000-series relays installed at Waterford 3 beyond that recommended by the manufacturer. The team determined that the licensees evaluation for extension of qualified life for Agastate E7000-series relays was based on incomplete testing and relied on a methodology which did not conform to industry standards. Therefore, while not a contributor to the recent trend in infant mortality failures of Agastat E7024PB relays, the team determined that the licensees justification for extension of service life beyond the 10 years recommended by the manufacturer did not have an adequate technical basis.

.1 Inappropriate Extension of Qualified Service Life of Agastat Relays

Introduction.

The team identified a violation of 10 CFR 50, Appendix B, Criterion III, Design Control, which occurred when the licensee inappropriately extended the service life of 322 safety-related Tyco/Agastat series E7000 time-delay relays without having an adequate technical basis. Specifically, the engineering justification for extending the qualified life beyond the manufacturer-recommended 10 years considered only degradation due to thermal aging; it failed to consider other known modes of degradation in accordance with industry standards. Further, a performance monitoring program intended to assess any increased failure rate due to this change was inappropriately canceled.

Description.

In April 1995, the licensee completed Problem Evaluation/Information Request (PEIR) DE-57 to address concerns with the qualification of optional equipment, such as auxiliary switches, used on Class 1E Agastat E7000-series relays. As part of this evaluation, the Equipment Qualification group requested that an evaluation of the

service life of these relays be performed to minimize unnecessary component replacements. The evaluation in PEIR DE-57 identified "two significant deficiencies" with the vendors qualification document ES-1000, Environmental Test Report on Agastat E7000 Series Timing Relays, which was the basis document for the vendors qualified service life for these relays. The first deficiency noted by the licensee was that the service life for normally energized relays was not addressed. The second was that the report specified a seemingly short qualified life for the relatively mild ambient conditions of the qualification test. PEIR DE-57 provided an evaluation, specific to the 322 Agastat E7000-series relays installed at Waterford 3, to determine the expected service life for both energized and de-energized relays. The report concluded the following, conflicting with the manufacturers recommendations for applications at Waterford 3:

• The replacement of Agastat E7000-series timing relays should be based on the installation date rather than the manufacturing date code as recommended by Tyco Electronics/Amerace.

• The calculated service or expected design life of 103 for installed relays ranged from 16.74 to 39 years; the calculated service or expected design life of the remaining 219 installed relays ranged from 40 to 122 years.

Based on these conclusions, the report recommended that replacement tasks for each relay be either extended or deleted, provided that periodic calibration checks were performed to provide trend data to anticipate end-of-life failures. As a result of the DE-57 recommendations, action taken in Condition Report CR-98-0591 extended or deleted periodic maintenance tasks for replacing Agastat E7000-series relays.

The team was concerned that the revised replacement intervals significantly exceeded the vendors qualified service life for these relays. The industry has had considerable experience with end-of-life relay failures earlier than expected. This operating experience was factored into the vendor's recommendation. The team noted that PEIR DE-57 evaluated the service or expected design life of the Agastat E7000-series relays by applying Arrhenius methodology for thermal aging using empirically-determined aging data for the construction materials and expected relay service temperatures. The physical locations of all cabinets housing these relays were identified and the interior cabinet temperatures were recorded using data loggers. Relay service temperatures were established based on these measured cabinet temperatures. The construction materials most susceptible to thermal aging were determined and analyzed for service life at the established temperatures. The team reviewed these methodologies and the results of the licensees calculations and identified the following deficiencies:

• The cabinet interior temperatures were obtained during the winter months; the corresponding ambient room temperatures were not recorded. The team determined this to be non-conservative because normal room temperatures were approximately 65 degrees Fahrenheit but analysis showed that the room temperatures could be as high as 104 degrees Fahrenheit under both normal and accident conditions. These higher ambient temperatures would increase the cabinet interior temperatures correspondingly.

• The evaluation based the calculated service or expected design life solely on temperature-related aging. Other environmental conditions which may occur during normal and abnormal conditions at the installed equipment location and the expected operating duty cycle of the relays were not considered.

The team noted that IEEE 323-1974, IEEE Standard for Qualifying Class 1E Equipment for Nuclear Power Generating Stations, states that during testing to determine qualified service life, electromechanical equipment such as relays shall be operated to simulate the expected mechanical wear and electrical contact degradation. NUREG-0588, Interim Staff Position on Environment Qualification of Safety-Related Electrical Equipment, states that the degrading influences discussed in this IEEE Standard and the electrical and mechanical stresses associated with cyclic operation of equipment should be considered and included as part of the aging programs; it goes on to state that equipment located in general plant areas outside containment where equipment is not subjected to a design basis accident environment should be qualified to the normal and abnormal range of environmental conditions postulated to occur at the equipment location.

The team concluded that the licensee did not have an adequate technical basis to demonstrate that they could safely extend the qualified life of safety-related Agastat E7000-series relays. Also, the team identified that PEIR DE-57 had specified that periodic calibration results were trended to anticipate end-of-life failures, but this trending was discontinued. The lack of trending appeared to have contributed to the fact that the licensee had not identified the failure trends discussed in this report more promptly.

Further, the excessive calibration drift experienced in most of the date code 9948 relays onsite was likely indicative of impending failure, but the licensee failed to recognize this, and instead attempted to recalibrate the relays. Additional discussion of industry standards and operating experience is located in Section 4OA3.5.2 .

Analysis.

Extending the qualified life of safety-related Agastat E7000-series relays without having an adequate technical basis is a performance deficiency. This performance deficiency is more than minor because it is associated with the mitigating systems cornerstone attribute of design control because it affects the availability and reliability of systems which respond to initiating events to prevent undesirable consequences. Using Inspection Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, this performance deficiency was determined to be of very low safety significance (Green) because it is a design or qualification deficiency confirmed not to result in loss of operability or functionality. Specifically, only one relay failure had occurred beyond the recommended 10-year service life; this failure did not result in the failure of multiple redundant trains of safety-related equipment. This finding was determined not to have a cross-cutting aspect because it is not indicative of current licensee performance.

Enforcement.

Title 10 CFR 50, Appendix B, Criterion III, Design Control, requires, in part, that design control measures shall provide for verifying or checking the adequacy of design, such as by the performance of design reviews, by the use of alternate or simplified calculational methods, or by the performance of a suitable testing program.

Where a test program is used to verify the adequacy of a specific design feature in lieu of other verifying or checking processes, it shall include suitable qualifications testing of a prototype unit under the most adverse design conditions.

Contrary to the above, in April 1995, Waterford 3 failed to verify or check the adequacy of design by performing design reviews or by implementing a suitable testing program to justify the extension of qualified life for 322 Tyco/Agastat E7000-series timing relays.

Specifically, the licensee: did not perform suitable qualifications testing of a prototype unit under the most adverse design conditions; did not account for some known modes of degradation; did not account for normal and abnormal operating conditions; and did not maintain a trending program to monitor for indication of impending end-of-life relay failures. Because this finding was determined to be of very low safety significance (Green) and was entered into the licensees corrective action program as CR-WF3-2009-03813, this violation is being treated as a non-cited violation consistent with the NRC Enforcement Policy: NCV 05000382/2009010-03, Inappropriate extension of qualified service life of Agastat relays.

5.2 Operating Experience and Industry Guidance

a. Inspection Scope

The team reviewed industry guidance and operating experience surrounding service life extension and unexpected failures of safety-related relays. The team assessed how the licensee incorporated this guidance and operating experience into its design control, maintenance and testing practices, and replacement schedule. This review satisfied Special Inspection Charter Item 4.

b. Discussion The team determined that guidance from several industry standards that related to aspects of this inspection scope had not been fully incorporated into station procedures and preventative maintenance templates at Waterford 3. Specifically:

• IEEE 323 -1974, IEEE Standard for Qualifying Class 1E Equipment for Nuclear Power Generating Stations, defines qualified life as the period of time for which satisfactory performance can be demonstrated for a specific set of service conditions. The standard stipulates that any qualification by analysis (i.e., not performing testing) must include justification of methods, theories, and assumptions used, that demonstration of qualification of Class 1E equipment must include assurance that the severity of the qualification methods equal or exceed the maximum anticipated service requirements and conditions, and that relays shall be operated during testing to simulate the expected mechanical wear and electrical contact degradation. The team noted that Waterford 3 did not consider Class 1E Agastat E7000-series timing relays to have a qualified life and did not consider the applicable degrading influences and the cyclic operation of the relays in determining the service or expected design life (see Section 4OA3.5.1.b.1).

• IEEE 1205-2000, IEEE Guide for Assessing, Monitoring, and Mitigating Aging Effects on Class 1E Equipment Used in Nuclear Power Generating Stations, provides guidelines for assessing, monitoring, and mitigating aging degradation effects on Class 1E equipment used in nuclear power generating stations. After this standard was issued, Waterford 3 did not establish an adequate program to assess, monitor, and mitigate aging effects on Class 1E Agastat E7000-series timing relays.

• NUREG/CR-4715, An Aging Assessment of Relays and Circuit Breakers and System Interactions, notes that relays are affected by aging and that signs of aging are observable by the ninth and tenth years, with an indication that relays are experiencing increases in failure rates from the seventh through the eleventh year. Additionally, continuously energized relays fail approximately 60 percent more frequently than in relays with de-energized coils, 47 percent of timing relay failures were associated with set point drift, and periodic replacement of timing relays could be used to reduce the effects of age-related degradation. As-found set point trending is recommended to detect adverse trends in set point drift, inrush current measurements are recommended to detect degradation, and thermography is recommended to detect hot spots.

The team concluded that Waterford 3 did not address these failure rates or trends when extending the replacement period of Agastat E7000-series timing relays beyond the manufacturer recommended ten years. Further, the licensee did not adequately trend failures and take appropriate actions to determine causes of relay failures due to excessive timing drift.

• NUREG/CR-5762, Comprehensive Aging Assessment of Circuit Breakers and Relays, recommends infrared temperature measurement, inrush current and vibration testing of timing relays to detect degradation. The team determined that periodic maintenance performed on timing relays at Waterford 3 does not include these tests.

• NRC Information Notice 84-20, Service Life of Relays in Safety-Related Systems, identified that the service life for a relay used in a normally energized application is significantly shorter than when used in a cycled or normally de-energized application. It is believed that the shorter service life of the normally energized relays can be attributed to accelerated thermal aging of relay components. Further, even relays used only in a cycled or normally de-energized application may have a service life considerably shorter than the plant life.

• EPRI TR-102067, Maintenance and Application Guide for Control Relays and Timers, recommends that timing set point calibration be checked and adjusted every three years and that as-found timing data from each calibration be trended, with a continuously increasing/decreasing trend indicating a set point drift problem that would likely require replacement of the relay. For relays used in normally energized applications, a thermal survey using an infrared temperature instrument is recommended to be performed and the results trended.

This guide further stated that while relays, especially those used in safety related applications, are generally located in areas where mild environmental conditions exist, even these mild environments, when combined with the operating duty, can cause significant aging stresses. These stresses can be classified into four categories: environmental, thermal, mechanical, and electrical. These stresses will tend to cause some relay materials to degrade with time. The symptoms of such degradation may include changes in response time (e.g., change in the time delay set point for time delay relays), in coil characteristics (e.g., coil resistance,

inrush and holding currents, and insulation resistance), or in contact characteristics (e.g., resistance across closed contacts, dielectric capability).

Waterford 3 did not perform timing set point calibration every three years for all of the installed Agastat timing relays. Rather, the team determined that these calibrations are performed with periodicities ranging from 3 to 27 years. Infrared temperature scans at Waterford 3 are only performed on an as-required basis.

Further, the licensee did not evaluate the applicable degrading influences and the cyclic operation of the relays in determining the service or expected design life.

• EPRI TR-106857-V31, Preventive Maintenance Basis - Relays - Timers, states that timing relay drift is caused by changes in the plunger and bellows mechanisms and is likely to cause the pneumatic relay to be out of tolerance after just a few years and to be unable to be re-calibrated after 20-40 years depending on the ambient temperature. The guide recommends thermography be performed annually, as-found testing and calibration be performed every 4 to 10 years depending on criticality and service conditions, functional testing be performed every 2 years, and replacements made as-required. As discussed in Section 5.3, while the licensees preventative maintenance program for Agastat E7000-series relays was originally based on this EPRI guideline, the program was subsequently revised and many of the recommended tasks were eliminated or their frequencies extended.

The team also determined that applicable station, fleet, and industry operating experience had not been appropriately incorporated into station procedures.

Specifically:

• In November 2002, Entergy fleet Condition Report LO-NOE-2002-00317 identified simultaneous failures of Agastat relays in opposite trains of safety-related equipment at another Entergy plant and noted an abnormal number of relay failures discovered during testing. The report suggested that the relay replacement program should be replacing relays at a frequency shorter than the expected service life such that failures are not expected and are unusual occurrences. Waterford 3 reviewed the Condition Report and concluded that there is no useful information in the attached CR.

• In September 2003, Entergy fleet Condition Report LO-NOE-2003-00537 identified multiple failures of Agastat E7000-series relays during prior-to-installation bench testing at another Entergy site. Specifically, two relays, 12 and 14 years from date of manufacture, did not perform as expected during a pre-installation bench test. The Condition Report notes that the station had previously concluded that the service life of normally energized Agastat E7000-series relays was 10 years from the date of installation, not from the date of manufacture as recommended by the vendor. The service life of most normally de-energized relays had been extended to 23 years from the date of installation.

Shelf life had not been considered; that station initiated action to reevaluate shelf life of these relays.

• In April 2003, Entergy fleet Condition Report LO-NOE-2003-00603 identified operating experience at another facility related to safety-related Agastat relays found in service which had exceeded their qualified life. The report noted confusion over whether the qualified life of relays was considered from the date of manufacture or the date of installation. Further, the report recommended that if a stations replacement frequency is based on date of manufacture, stations should review program provisions to ensure the requirement is adequately implemented. The team was unable to determine that the Waterford 3 performed the recommended review.

• In February 2004, Entergy fleet Condition Report LO-NOE-2004-00050 identified operating experience at another facility in which newly-manufactured Agastat E7000-series timing relays procured in 2001 were manufactured with incorrect springs which could result in improper operating characteristics. The team was unable to determine if Waterford 3 evaluated this condition for applicability.

5.3 Periodic Maintenance and Age Management

a. Inspection Scope

The team examined the licensees maintenance and testing program for Agastat E7000-series relays and evaluated whether it provided for appropriate age management. The team assessed the adequacy of this program and whether the licensee was following the program provisions. This review satisfied the requirements of Special Inspection Charter Item 5.

b. Discussion The licensees maintenance and testing program is contained in Procedure EN-DC-324, Preventive Maintenance Program, and the associated Preventative Maintenance Basis Template, EN-Relay-Timing. The guidance in these documents was originally based on guidance provided in EPRI Technical Report TR-106857-V31, Preventive Maintenance Basis - Relays - Timing. This report recommended that for timing relays such as the Agastat E7000-series relays, thermography be performed annually to identify higher-than-normal temperature components, as-found testing and calibration be performed every four to ten years depending on criticality and service conditions, functional testing every be performed every two years, and relay replacement be performed as-required.

The licensee subsequently modified these procedures to perform this maintenance according to the following schedules:

• Thermography: as-required for high duty cycle applications and based on operator observations or other testing; not applicable for low duty cycle applications.

• As-found testing and calibration: 6 to 9 years depending on criticality and service conditions.

• Functional testing: 3 years for high critical applications; not applicable for low critical/low duty cycle applications; not recommended for low critical/high duty cycle applications.

EPRI technical report TR-102067, Maintenance and Application Guide for Control Relays and Timers, recommends timing set point calibration be checked and adjusted every 3 years. This guidance also recommends trending of the as-found set point from each calibration to provide predictive indication of imminent failure. The team noted that as-found testing and calibration intervals for the installed Agastat E7000-series timing relays ranged from 3 to 27 years and that trending of as-found timing data from each calibration had been discontinued in the second quarter of 2006, after which the station relied on the Condition Report process to identify adverse trends in setpoint drift. The program was adjusted such that when a timing relay is found out of specification, a condition report would be generated and the condition would be evaluated. If the relay drift is determined to be correctable and the timing demonstrates repeatability after recalibration, then the relay would be determined to be acceptable and returned to service.

The team determined that the lack of formal trending impeded the stations ability to identify degrading trends in relay performance. Further, no actions were taken to increase frequency of calibration of relays that demonstrated excessive set point drift.

The team noted as an example of this condition the adverse trend of excessive drift and premature failure of Agastat E7024PB relays with date code 9948 (see section 4OA3.4.1.b).

Additionally, PEIR DE-57 recommended that timing relay replacement tasks be extended or deleted provided that periodic calibration checks were performed. These replacement tasks were extended or deleted as recommended, but the associated periodic calibrations were not initiated. The licensee identified this deficiency in Condition Report CR-98-0591 and initiated periodic calibration checks. However, the team identified that testing of the Agastat E7000-series relays repeatedly demonstrated as-found timing delays which were out specification. In response to this excessive drift, the licensee initiated ER-W3-1998-0132-001A, Analysis of Manufacturer Tolerance for the 1E Agastat Timers, which performed a control circuit review for all Class 1E Agastat E7000-series relays to determine if the manufacturers repeat accuracy of +/-10 percent could be widened to permit the relay calibration interval to be extended. The review found that in all but 19 of the 322 relays, it was acceptable to increase the set point tolerance to +/-25 percent. For the 19 relays that were identified where the time delay would impact the design basis function of the equipment or system, the set point tolerance was maintained at +/-10 percent.

The team determined that Waterford 3s preventive maintenance and testing program for Agastat E7000-series timing relays does not conform to industry standards. As discussed in Section 4OA3.5.1.b.1 of this report, Waterford 3s applications of E7000-series relays depart substantially from manufacturer specifications. Relay replacements are performed at intervals from 18 to greater than 40 years despite industry operating experience indicating significant increases in failure frequencies of relays aged 7-11 years. The licensee does not maintain trending data of as-found timing delays. Relay failures as a result of excessive set point drift are entered into the corrective action program, but in most cases these relays are recalibrated and returned to service with no evaluation of previous trends. Industry standards and operating experience recommend more frequent timing tests and calibrations than are performed at Waterford 3 as well as additional testing, such as inrush current signature trending, to assist in determining

degradation. This guidance has not been incorporated into Waterford 3s preventative maintenance program.

4OA6 EXIT MEETING SUMMARY

On July 24, 2009, the team presented preliminary inspection results to Mr. J.

Kowalewski, Site Vice President, and members of licensee staff. Licensee staff confirmed that no proprietary information was reviewed by the team.

On August 24, 2009, the team presented additional inspection results telephonically to Mr. J. Kowalewski and other members of licensee staff.

On October 19, 2009, the team presented the final inspection results telephonically to C.

Arnone, Plant Manager and other members of licensee staff. The licensee acknowledged the inspection findings.

4OA7 LICENSEE-IDENTIFIED VIOLATION

The following violation of very low safety significance (Green) was identified by the licensee and is a violation of NRC requirements which meets the criteria of the NRC Enforcement Policy, NUREG-1600, for being dispositioned as NCVs.

• Title 10 CFR 50.65(b)(1) requires that safety-related structures, systems, and components that are relied upon to remain functional during and following design basis events shall be included in the scope of the Maintenance Rule monitoring program. Contrary to this, in 2006 the licensee eliminated the safety-related Agastat E7024PB Engineered Safety Features Actuation System relays from the scope of its maintenance rule program, resulting in relay failures not being appropriately evaluated as functional failures. The licensee identified this condition on July 20, 2009, in Condition Report CR-WF3-2009-3698. This finding is of very low safety significance because it does not represent the loss of safety function of a system or train and does not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event.

SUPPLEMENTAL INFORMATION

KEY CONTACTS

Licensee

C. Arnone, Plant Manager

M. Barreto, Engineering

E. Brauner, Engineering Fix-It-Now Supervisor/Root Cause Team Lead

K. Christian, Director Nuclear Safety Assurance

K. Cook, Operations Manager

T. Drews, Engineering

B. Ford, Licensing (Entergy Corporate)

M. Groom, Engineering

M. Haydel, Engineering Supervisor

J. Kowalewski, Site Vice President

B. Lindsey, Maintenance Manager

D. Marble, Maintenance Supervisor

D. McElroy, Engineering

R. Murillo, Licensing Manager

K. Nichols, Director Engineering

R. Putnam, Manager Engineering Programs

J. Ridgel, Quality Assurance Manager

W. Steelman, Licensing Engineer

O. Tucker, Engineering Supervisor

B. Ward, Quality Specialist

J. Williams, Licensing Specialist

NRC

J. Clark, Chief, Reactor Projects Branch E, Region IV

G. Cwalina, Senior Allegations Coordinator, NRR

M. King, Senior Reactor Systems Engineer, NRR

R. Latta, Senior Reactor Inspector, Region IV

D. Overland, Senior Resident Inspector, Region IV

G. Replogle, Senior Project Engineer, Region IV

M. Runyan, Senior Reactor Analyst, Region IV

J. Thompson, Senior Reactor Operations Engineer, NRR

ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000382/2009010-01 URI Failures to Evaluate Adverse Conditions for

Reportability to the NRC (Section 4OA3.4.1.b.1)

Opened and Closed

05000382/2009010-02 NCV Failure to identify an adverse trend in failures of

time-delay relays (Section 4OA3.4.2.b.1)

-1- Attachment 1

05000382/2009010-03 NCV Inappropriate extension of qualified service life of

Agastat relays (Section 4OA3.5.1.b.1)

DOCUMENTS REVIEWED

Condition Reports

CR-98-0591

CR-WF3-2001-01358

CR-WF3-2006-00481

CR-WF3-2006-02107

CR-WF3-2007-02961

CR-WF3-2008-02352

CR-WF3-2008-04304

CR-WF3-2008-04765

CR-WF3-2008-04782

CR-WF3-2008-05786

CR-WF3-2009-01190

CR-WF3-2009-01649

CR-WF3-2009-02674

CR-WF3-2009-02693

CR-WF3-2009-03102

CR-WF3-2009-03448

CR-WF3-2009-03630*

CR-WF3-2009-03698*

CR-WF3-2009-03742*

CR-WF3-2009-03813*

CR-WF3-2009-03821*

CR-WF3-2009-03822*

CR-WF3-2009-03823*

CR-WF3-2009-03824*

CR-WF3-2009-03825*

• - CR written in response to an issue raised as part of this inspection

Procedures

EN-LI-118, Root Cause Analysis Process, Revision 10

EN-LI-119, Apparent Cause Evaluation (ACE) Process, Revision 8

ME-007-005, Time-Delay Relay Setting Check and Adjustment, Revision 13

OP-903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification,

Revision 303

OP-903-094, ESFAS Subgroup Relay Test - Operating, Revision 014

ENS-DC-300, Periodic Maintenance Program, Revision 1

-2- Attachment 1

ENS-DC-324, Periodic Maintenance Program, Revision 5

EN-LI-121, Entergy Trending Process, Revision 8

EN-MP-112, Shelf Life Program, Revision 2

W2.301, Identification, Evaluation, and Reporting Process for 10CFR21 Compliance, Revision

LP-122, "Reporting of Defects and Noncompliances Under 10 CRF 21," Revision 2

EN-LI-102, "Corrective Action Process," Revision 13

Calculations

ER-W3-00-0088-00-00, Trentec Report No. 61393.0, Nuclear Environmental Report for Various

Agastat Components, Revision 0

ER-W3-1998-0132-002, Update of EDB to reflect new setpoint limits as approved by ER-W3-

1998-0132-001, Revision 0

ER-W3-1998-0132-001, Analysis of Manufacturer Tolerances for the 1E Agastat Timers,

Revision 0

Drawings

07-3398-97, Panel CA Wiring, Revision B

07-3398-98, Panel CC Wiring, Revision B

07-3398-99, Panel CE Wiring, Revision C

Letters

Letter from Tyco Electronics to all nuclear customers: Notification of Limited E7024 Relay

Recall (CII-012), dated 11/26/08

Letter from Tyco Electronics to Tom Drews: RMA D7475Q - 2 ea. E7024PB004, SN 08040634

and 08040636 - Relays returned for analysis and root cause investigation, dated 1/28/09

Other Documents

Problem Evaluation/Information Request (PEIR) DE-57, Qualification and Service Life of

Agastat series E7000 timing relays, dated May 1995

Vendor Manual TD-A348.0025, Agastat 7000 Series, Revision 2

-3- Attachment 1

Vendor Report ES1000, Environmental Test Report on Agastat E7000 series timing relays,

dated April 1980

PM Basis Template, EN-Relay-Timing, Revision 2

Operating Experience Documents and Industry Guidance

NUREG/CR-4715, An Aging Assessment of Relays and Circuit Breakers and System

Interactions, dated June 1987

Part 21 Report 1995-15-3, Nine Mile Point Part 21 - Wylie Laboratories Report, 06/23/1995

Information Notice 92-77, Questionable Selection and Review to Determine Suitability of

Electro-pneumatic Relays for Certain Applications, 1992

IEEE 323, IEEE Standard for Qualifying Class 1E Equipment for Nuclear Power Generating

Stations, 1974

IEEE 1205, IEEE Guide for Assessing, Monitoring, and Mitigating Aging Effects on Class 1E

Equipment Used in Nuclear Power Generating Stations, 2000

Regulatory Guide 1.89, Qualification of Class 1E Equipment for Nuclear Power Plants, 1974

NUREG/CR-5762, Comprehensive Aging Assessment of Circuit Breakers and Relays, dated

March 1992

NUREG-0588, Interim Staff Position on Environment Qualification of Safety-Related Electrical

Equipment, dated July 1981

Information Notice 82-04, Potential Deficiency of Certain Agastat E7000 Series Time-Delay

Relays, 1982

Information Notice 84-20, Service Life of Relays in Safety-Related Systems, 1984

EPRI TR-102067, Maintenance and Application Guide for Control Relays and Timers, 1993

EPRI TR-106857-V31, Preventive Maintenance Basis - Relays - Timers, 1998

-4- Attachment 1

UNITED STATES

NUC LE AR RE G UL AT O RY C O M M I S S I O N

Eric A. Ruesch R- E1G-I O N I V

2 EAST LAMAR BLVD , SU I TE 400

AR LI N GTON , TEXAS 76011-4125

July 13, 2009

MEMORANDUM TO: Eric

A. Ruesch, Reactor Inspector

Plant Support Branch 2

Division of Reactor Safety

FROM: Roy

J. Caniano, Director

Division of Reactor Safety

SUBJECT: SPECIAL INSPECTION CHARTER TO EVALUATE THE

WATERFORD-3 ESFAS RELAY FAILURES

A Special Inspection Team is being chartered in response to the Waterford-3 Engineered Safety

Features Actuation System relay failures. You are hereby designated as the Special Inspection

Team Leader. Mr. Gerald Nicely, an engineering inspection contractor, is designated as a team

member. The Senior Reactor Analyst assigned to support the team is Mike Runyan.

A. Basis

From September 2008 to June 2009, Waterford-3 experienced six failures of Tyco

(Agastat) Model E7024PB time delay relays. The licensee believes that at least three of

the failures were caused by manufacturing problems, including: 1) installation of an

incorrect spring; 2) an incorrect force applied to a component during manufacturing; and

3) foreign materials. The manufacturer recommends a 10 year service life, but the

licensee has extended the service life to almost 30 years in some applications. It

appears that two of the known failures involved relays that had exceeded the

recommended 10 year service life.

The licensee uses 33 of these relays in safety-related applications. They are primarily

used in auto-starting circuits for safety-related components. Two of the failures involved

the failure of the Train B high pressure safety injection pump to automatically start during

surveillance testing.

This Special Inspection Team is chartered to review the circumstances related to

Waterford 3's history of Tyco E7000-series relay failures, particularly model E7024PB

failures, and to assess the effectiveness of the licensee=s actions for resolving these

problems. The team will also assess the licensees actions towards addressing the

extent of condition and extent of cause, as well as evaluating the potential need for

generic communications.

-1- Attachment 2

Eric A. Ruesch -2-

B. Scope

The team is expected to address the following:

1. Develop a chronology (time-line) related to events related to E7000-series relays.

This timeline should include all relevant changes made to maintenance, testing,

and replacement of relays. A listing of relay failures within the past five years

should also be included.

2. Evaluate the licensees progress in developing root or apparent cause

evaluations for the known failures and assess the adequacy of the licensees

corrective actions.

3. Review the basis for the 10 year vendor service life recommendation (obtain

vendor report) and determine if the licensee had an adequate technical basis to

extend the service life without impacting the reliability of the system.

4. Evaluate the licensee's response to industry operating experience related to

these relays to determine whether the licensee took timely and effective

measures to address the issues.

5. Evaluate the licensees maintenance and testing program for these relays and

how this provides for age management. Assess the adequacy of these programs

and whether the licensee is following the program provisions.

6. In concert with the senior reactor analyst, collect data as necessary to support a

significance determination for each relay failure. This should include an

assessment of the circumstances of each failure (i.e., plant conditions, testing

mode), what equipment would not function, time since previous successful

performance, and the potential for operator recovery actions.

7. Determine if the licensee met NRC reporting requirements for failure, including

CFR Part 21 and 10 CFR 50.73.

8. Make recommendations as to whether there may be a need for generic

communications concerning the failures.

C. Guidance

Inspection Procedure 93812, ASpecial Inspection,@ provides additional guidance to be

used by the Special Inspection Team. Your duties will be as described in Inspection

Procedure 93812. The inspection should emphasize fact-finding in its review of the

circumstances surrounding the events. It is not the responsibility of the team to examine

the regulatory process. Safety concerns identified that are not directly related to the

events should be reported to the Region IV office for appropriate action.

-2- Attachment 2

Eric A. Ruesch -3-

The Team will report to the site, conduct an entrance, and begin inspection no later than

July 20, 2009. While on site, you will provide daily status briefings to Region IV

management, starting on Tuesday, July 21, 2009. Regional management will coordinate

with the Office of Nuclear Reactor Regulation to ensure that all other parties are kept

informed. A report documenting the results of the inspection should be issued within

days of the completion of the inspection.

This Charter may be modified should the team develop significant new information that

warrants review. Should you have any questions concerning this Charter, please

contact me at (817) 860-8180.

-3- Attachment 2

Information Request

July 9, 2009

Waterford 3 Special Inspection on ESFAS Relay Failures

Inspection Report 05000382/2009010

This inspection will focus primarily on the six ESFAS relay failures which occurred between

September 2008 and June 2009, but will also include a review of all Tyco/Agastat 7000-series

relay failures which have occurred over the past five years.

Please provide the following by July 13, 2009:

1. Summary list of all Tyco (Agastat) 7000-series relays installed in the plant. For each,

include the model number, indicate whether the relay is installed in a safety-related

application, and provide the date of installation and expected service life.

2. Timeline of all events related to Tyco (Agastat) 7000-series relays from January 2004 to

present. This timeline should include relay failures, relay replacements and reason for

replacements, changes/revisions to maintenance and/or test procedures, and generic

and/or vendor communications.

3. Corrective Action Documentation and Cause Evaluations for the six ESFAS relay

failures which occurred between September 2008 and June 2009. If not contained in the

cause evaluations, provide the following for each failure:

a. Circumstances of each failure including plant conditions, testing mode, failure

mode, method of recovery, and affected equipment,

b. Time since last prior successful performance, and

c. Operator actions taken to recover from the failure.

4. Tyco/Agastat vendor report which includes the basis for the 10-year service life

recommendation.

5. Documentation of basis for extending the service life of 7000-series relays beyond the

10-year vendor recommendation.

6. Periodic maintenance schedule, testing or surveillance schedule, and/or replacement

schedule for 7000-series relays. Include all applicable procedures.

-1- Attachment 3

Supplemental Information Request

July 16, 2009

Waterford 3 Special Inspection on ESFAS Relay Failures

Inspection Report 05000382/2009012

This information request supplements the information request sent on July 9, 2009. Please

provide the following documents to the team prior to or upon its arrival onsite on July 20, 2009:

7. Procurement history of Tyco/Agastat 7000-series relays from 2004 to present, including

calculations/specifications used to determine requirements for purchase. For failed

relays, provide purchase orders, receipt inspection documentation, and documentation

of pre-installation bench tests (if applicable). Include the stations receipt inspection

procedure and bench test procedure.

8. Safety function evaluations (or equivalent) for Tyco/Agastat 7000-series relays installed

in safety-related applications. Also provide evaluations for any Tyco/Agastat 7000-

series relays not classified as safety-related but installed in safety-related systems.

9. For each installed relay location, provide design bases of the cabinet and/or room

ventilation system(s) including the calculated maximum temperature and humidity

ranges maintained during normal operation, worst-case operational occurrences, and

design basis events.

10. Data sheets and/or summary data for as-found and as-left conditions for periodic

maintenance performed on Tyco/Agastat 7000-series relays from January 2004 to

present. To enable the team to review trends, these data should include required set

points, as-found set point drift from the previous performance of the maintenance

procedure, and time since previous performance of the maintenance procedure.

11. System health report(s) (or equivalent) for Tyco/Agastat 7000-series relays. If these

relays are not tracked separately, provide system health report(s) (or equivalent) which

would document and/or track/trend failures of or deficiencies with Tyco/Agastat 7000-

series relays.

-2- Attachment 3