05000346/LER-2003-008

LER-2003-008,

Davis-Besse Unit Number 1
Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(ix)(A) 10 CFR 50.73(a)(2)(v), Loss of Safety Function
3462003008R00 - NRC Website

DESCRIPTION OF OCCURRENCE:

The Safety Features Actuation System (SFAS) 1JE) at the Davis-Besse Nuclear Power Station (DBNPS) is designed to automatically prevent or limit fission product and energy release from the core, to isolate the containment vessel and to initiate the operation of Engineered Safety Features equipment in the event of a loss of coolant accident. The SFAS consists of four identical redundant sensing and logic channels and two identical redundant actuation channels. Each sensing channel includes analog circuits with analog isolation devices, and each logic channel includes trip bistable modules with digital isolation devices. The isolated output of the trip bistable module is used to comprise coincidence matrices with the terminating relays within the actuation channel of the SFAS.

The trip bistables monitor the station variables and normally feed continuous electrical (fail-safe) signals into two-out-of-four coincidence matrices.

Should any of the station variables exceed their trip setpoints, the corresponding bistables in each of the four channels will trip and cease sending output signals. If two of the four channel bistables monitoring the same station variable cease to send output signals, the corresponding normally-energized terminating relays on all channels will trip.

The SFAS is a fail-safe, de-energize to trip, system. Therefore, if the power supply to a channel is lost, that channel will trip, reducing the system coincidence matrices from two-out-of-four to one-out-of-three mode. The terminating relays of sensing and logic channels 1 and 3 must both be de-energized to activate safety actuation channel 1. Similarly, sensing and logic channels 2 and 4 must both be de-energized to activate safety actuation channel 2. The terminating relays (also known as output relays) (JE-RLY] act on the actuation control devices such as motor controllers and solenoid valves.

The DBNPS has experienced failures of the SFAS output relays in the past few years. These failures, attributed to age degradation of the relays, are typically failures of the relay coils where the coil open-circuits and de- energizes, resulting in closure of the output relay contacts and providing a 'half-trip" condition. In this 'half-trip

• condition, a trip of the complementary relay in the corresponding SFAS channel would cause the actuated component to go to its SFAS desired position. While no actuations of SFAS equipment occurred as a result of these failures in the recent past, and although the half-trip of a channel is a conservative design feature response, it is an undesired operational condition.

In approximately 1997, the relay manufacturer ended production of the relays used as SFAS output relays and divested itself of the tooling needed to produce these relays. In July of 1998, it was realized that the existing DBNPS stock of spare SPAS output relays would only last for several years at the observed failure rate. However, because these relays were no longer being manufactured, the lack of stock of new relays represented a concern for the DESCRIPTION OF OCCURRENCE: (Continued) remainder of plant life. Therefore, in the Fall of 1999, plane were developed to acquire replacement relays, and replace approximately 250 of the 286 output relays during the Thirteenth Refueling Outage (13RFO) scheduled to begin in the Spring of 2002. Since a direct replacement for the relays was no longer manufactured, relay manufacturer Deutsch was contacted to acquire suitable, non-safety grade replacement relays, which were then qualified for safety- related service by Wyle Laboratories, an independent testing lab, to the specifications supplied for use in a safety-grade application. The new Generation 3 (03) relays (Deutsch Model 4CP36AF) were shipped to the DBNPS in September of 2000.

During pre-installation functional testing at the DBNPS in December 2001, an intermittent failure of one 03 relay was identified in which the relay failed in a non-conservative manner. The relay's normally closed contacts did not repeatedly close when the relay coil was de-energized. The failed relay was examined and the cause of failure was determined to be a mechanical defect in the relay that occurred during manufacturing. Because only one failure was observed during bench testing, it was believed this failure was a random failure. During 13RFO, the approximately 250 Generation 1 (01) and Generation 2 (G2) relays originally installed in the SFAS were removed and replaced with the new 03 relays. The old G1 and G2 relays were stored in the Instrument and Control Maintenance shop.

Upon receipt of a Wyle Laboratory report on July 10, 2002, stating the December 2001 failure was due to an isolated, random failure caused during the manufacturing process, the fact that this conclusion was based upon a sample size of one was questioned by DBNPS Plant Engineering. Since there was no conclusive evidence that the defect was limited to a single relay, it was determined by Plant Engineering that additional evidence was necessary to justify the conclusion. Therefore, 10 additional G3 relays were sent to Wyle Laboratories for further analysis. These 10 relays passed all additional functional testing, and therefore no destructive testing of the 10 relays was performed as had been done for the failed relay. The results of the additional functional testing were documented in a Wyle Laboratory report dated August 9, 2002.

On February 27, 2003, during preparation for performing the integrated test of SFAS Train 2 during the continuing outage, a second G3 relay failed to reset appropriately, which raised the concern of a possible common mode failure with the new relays. This second failed relay along with a sample of the G3 relays was sent to Wyle Laboratory for inspection. The results of this inspection revealed that the contact rating of the G3 relays were incompatible with the configuration of the SFAS. Specifically, it was discovered that the G3 relay contacts are rated at 5 amperes (amps) at 30 volts direct current (VDC) resistive load. When installed in the SFAS, these contacts experience a nominal voltage of either 120 volts alternating current (VAC) or 125 VDC, or approximately four times the rated load. Additionally, most of the relay contacts are used to operate either a solenoid or another relay coil, introducing an inductive component of the load.

▪ f NRC Form 366A) (17) DESCRIPTION OF OCCURRENCE: (Continued) As a result of the discovery that the 03 relays were incompatible with the SFAS, the G3 relays were removed from the SFAS. The previously installed G1 and 02 relayo were removed from storage, and those that were successfully bench-tested were reinstalled in the SFAS along with additional 01 relays.

These relays will be fully tested as part of restart activities.

It was initially believed that this issue was not reportable because none of the 03 relays were known to be installed when the SPAS was required to be operable per Technical Specification (TS) 3.3.2.1, Safety Features Actuation System Instrumentation. However, after further review, on July 8, 2003, with the plant in Mode 5, it was discovered that five of the G3 relays had been installed during the Thirteenth Operating Cycle, which concluded on February 16, 2002. The relays installed were as follows:

• Relay 2K12J installed 5/2/2001 for CV5010E, Containment Hydrogen Analyzer 2 Discharge Valve

• Relay 4K23D installed 6/1/2001 for CV5076, Containment Vacuum Relief Isolation Butterfly Valve

• Relay 21(27D installed 8/20/2001 for RC240B, Pressurizer Sample Line Isolation Valve

• Relay 4K23B installed 10/17/2001 for P43-3, Component Cooling Water Pump 3

• Relay 3K25A installed 11/6/2001 for CS1530, Containment Spray Auto Control Valve 1.

The G3 relays had been procured to replace the older 01 relays. Once the 03 relays were approved for use and placed into etock, they were installed instead of the older 01 relays upon failure of a G1 relay. This was done to monitor operation and evaluate performance prior to the wholesale change out that was planned during 13RFO.

APPARENT CAUSE OF OCCURRENCE:

The originally installed Couch Type 4C relays (G1) are small, hermetically sealed rotary relays designed primarily for military usage. These relays were designed with a rugged balanced arm motor assembly and a rotary contact mechanism that ensures stable operation during conditions of bumping, vibration, shock, and acceleration. These relays were widely used in missile and aircraft applications. The contacts of the relays are rated for 5 amps at 30 VDC. Correspondence between the relay manufacturer, the DBNPS Architect/ Engineer, the vendor of the SFAS (Consolidated Controls) and Toledo Edison (original licensee for the DBNPS) in the early 19708 indicate that the relay manufacturer successfully evaluated the performance of the relays at 6 amps at 125 VDC inductive loads (making but not breaking). However, the SFAS vendor did not adequately document the contact rating discrepancy to allow for later changes in the procurement of replacement relays.

APPARENT CAUSE OF OCCURRENCE: (Continued) During the procurement process for the 03 replacement relays, the DBNPS procurement engineer did not follow procedures and perform an equivalency evaluation as required for the new relays. Instead, due to weak procedural requirements governing the performance of equivalency evaluations, the engineer delegated the vendor of the relays the responsibility for performing an equivalency review. While the DBNPS design specification for the SFAS specified the proper relay contact ratings, this information had been removed from the procurement package by the procurement engineer on the basis that the relays were purchased by part number rather than by the design specification.

The relay vendor qualified the new relays to the requirements set forth in the DBNPS purchase order, which specified the new relays be the same in form, fit, and function as those previously supplied. Since the purchase order did not specify a relay contact voltage rating, the manufacturer of the 03 replacement relays recognized that the original 01 relays had contacts rated for 5 amps at 30 VDC, and manufactured the replacement 03 relays with the same contact ratings. The relay vendor qualified the new 03 relays to these same requirements, resulting in the 03 relays having contacts rated for 5 amps at 30 VDC instead of 6 amps at 125 VDC.

A review of the G3 relay internals indicates significant differences in construction of the G3 relays when compared to the previously installed G1 and G2 relays. The 03 relays are essentially a relay within a relay, where a small cubical relay was installed in the same relay casing as the original 01 and 02 relays. This small cubical relay is a rocker style relay, as opposed to the original G1 and G2 rotary design. The relay contacts inside the small 03 cubical relay have less internal clearances than the original 01 and G2 rotary design, making them more susceptible to contact arcing at higher voltages.

ANALYSIS OF OCCURRENCE:

Because the G3 relays could have failed in a non-conservative manner, any SFAS channel where these relays were installed during the operating cycle was potentially incapable of performing its designated safety function.

Therefore, for the five installations listed above, the associated functional units were inoperable per TS 3.3.2.1, and per TS Table 3.3-3 Action 11, with these output logic channels inoperable, the associated components should have been tripped within one hour or be at least Hot Standby within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />. These actions were not taken while the plant was in Mode 1 with the relays installed; therefore this condition represents operation prohibited by the Technical Specifications, which is reportable per 10CFR50.73(a)(2)(i)(B).

Reporting of single causes that could have prevented the safety function of trains or channels in different systems is required by 10CPR50.73(a)(2)(ix)(A). Since the affected trains of equipment were in different systems, no potential loss of safety function for the entire system occurred, and so this issue is not reportable under 20CFR50.73(a)(2)(v).

However, because a single cause (improper procurement of replacement relays) potentially rendered the affected individual trains incapable of performing their safety function, this issue is reportable per 10CFR50.73(a)(2)(ix)(A).

ANALYSIS OF OCCURRENCE: (Continued) Each month during power operation, all four channels of the SPAS are functionally tested to demonstrate functionality and operability of the system. The normal testing that was performed with these five 03 relays installed did not reveal any abnormal relay operation (it should be noted that these monthly functional tests do not apply normal operating voltages to the contacts). However, these five 03 relays were satisfactorily tested during the performance of the integrated SPAS time response test performed during 13RFO on February 11, 2003 (actuation channel 1) and on March 7, 2003 (actuation channel 2), where normal operating voltage is applied to the relay contacts. This testing reasonably indicates that the SPAS equipment would have performed its safety function under accident conditions.

The consequences of a non-conservative failure of any of these five relays would result in an SFAS-actuated component not actuating when required. The component could still be manually actuated as operators perform post-actuation safety system verification, but the particular SFAS auto-initiation function would be impaired. Since the relays were not installed in redundant channels, the redundant SFAS-actuated component remained capable of performing its designated safety function in the event of an actuation of the SFAS.

Additionally, the procedure utilized in the event of an SPAS actuation directs the verification of a proper SPAS actuation. A failure of the SPAS output relay would not prevent manual actuation of the SFAS-actuated component.

CORRECTIVE ACTIONS:

All the 03 relays have been removed from the SFAS and replaced with successfully bench tested 01 or G2 relays. These relays will be fully tested as part of restart activities.

Procedure NOP-CC-7002, Procurement Engineering, will be revised as follows to:

• Strengthen the requirement that instructs Procurement Engineering personnel to perform an equivalency review

• Establish procedural controls that allow a vendor to perform an equivalency evaluation

• Establish clear guidance that instructs Procurement Engineering personnel on when to review plant design parameters when performing an equivalency evaluation.

This procedure revision will be complete by November 26, 2003.

FAILURE DATA:

There have been no LERs in the past three years involving improper procurement of Technical Specification equipment at the DBNPS. Furthermore n there have been no recent DBNPS Corrective Action Documents of a similar nature involving installation of inappropriately procured equipment that rendered Technical Specification equipment inoperable.

Energy Industry Identification System (Ells) codes are identified in the text as [XX].

NP-33-03-00B-00 CRS 03-05402, 03-03232, 03-02725