05000346/LER-2009-001
| Docket Number | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident |
| 3462009001R01 - NRC Website | |
Energy Industry Identification System (EIIS) codes are identified in the text as [XX].
DESCRIPTION OF EVENT:
Initial Plant Conditions:
On October 13, 2009, the Davis-Besse Nuclear Power Station (DBNPS) was operating in Mode 1 at approximately 100 percent power. Service Water [BI] Train 2 was inoperable per Technical Specification (TS) Limiting Condition for Operation (LCO) 3.7.8 at the start of the event for planned maintenance. This resulted in the following equipment to also be inoperable per the associated Technical Specification Limiting Conditions for Operation: Containment Air Cooler [BK] 2, Containment Spray [BE] Train 2, Emergency Diesel Generator [EK] 2, Component Cooling Water [CC] Train 2, Control Room Emergency Air Temperature Control System [VI] Train 2, Low Pressure Injection [BP] Train 2, High Pressure Injection [BQ] Train 2, and Auxiliary Feedwater [BA] Train 2.
System Description:
The Containment Air Cooler (CAC) System is comprised of three air cooler units [BK-FCU] located within the Containment Vessel [NH]. Two of the three units are used for both normal and emergency cooling. Each unit consists of finned tube cooling coils [BK-CCL] and a direct-driven fan [BK-FAN].
The fans are designed to operate under normal conditions at full/fast speed, controlling the ambient air temperature to a maximum of 120 degrees F with two units operating. A ductwork system distributes the air over and around equipment in Containment that produces or releases heat. Cooling water for the air cooler units is supplied by the Service Water System, and a modulating temperature control valve [BI-TCV] is provided in the discharge line from each cooler. CAC 3 is designed as a swing unit that can be aligned as either Train 1 or Train 2.
In the event of a Loss of Coolant Accident, the Safety Features Actuation System [JE] will start the Containment Air Cooling fans in slow speed, and the modulating control valve is interlocked to fully open to allow full water flow through each cooler.
Technical Specification (TS) Limiting Condition for Operation (LCO) 3.6.6 requires two Containment Spray trains and two CAC trains be operable while the plant is operating in Modes 1, 2, 3, and 4. With one required CAC train inoperable, TS LCO 3.6.6 Condition C requires within 7 days the inoperable train be restored to operable status. With two required CAC trains inoperable, TS LCO 3.6.6 Condition E requires one of the required trains be restored to operable status in 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. If these actions and associated completion times cannot be met, TS LCO 3.6.6 Condition F requires the plant be placed in Mode 3 in 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and Mode 5 in 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />.
TS LCO 3.6.3 requires each containment isolation valve be operable while the plant is operating in Modes 1, 2, 3, and 4. With a penetration flow path inoperable that has only one containment isolation valve, TS LCO 3.6.3 Condition C requires within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> the affected penetration flow path be isolated by use of at least one closed and de-activated automatic valve, closed manual valve, or blind flange.
DESCRIPTION OF EVENT: (continued)
Event Description:
On September 28, 2009, during performance of the CAC 3 18-month Surveillance Test, when the CAC fan was attempted to be shutdown from slow speed while aligned to electrical train 2, conflicting information was received as to whether the fan had shutdown or remained operating in slow speed.
When the Service Water Outlet Valve for CAC 3 was closed it immediately reopened, which also indicated a defective contact relay in the fan slow speed run circuit. CAC 3 was declared inoperable per TS LCO 3.6.3 Condition C and CAC 2 was placed in service. The two Service Water Outlet Valves for CAC 3 were also declared inoperable as Containment Isolation valves per TS LCO 3.6.3 Condition C due to the slow speed contact preventing the valves from being placed in the closed position, and manual isolation valves were closed to comply with the TS LCO.
During review of the issue with the CAC 3 fan slow speed contact relay a reference to NRC Information Notice 92-19, "Misapplication of Potter & Brumfield MDR Relays," was identified. This Information Notice describes misapplications of Potter and Brumfield MDR Relays like those installed in the CAC fan circuitry. Specifically, when the relay's own contacts are used in the electrical circuit to reset the relay coil, a relay race condition can occur. Consequently, the relays would fail in an intermediate mechanical position due to a partial alternating current waveform being applied to the relay's reset circuit. An earlier FirstEnergy BETA Laboratory failure report (completed July 21, 2009) from a CAC fan 1 relay failure in June 2009 showed that a short or partial alternating current waveform to the reset coil of the MDR relay could place the relay in an intermediate mechanical state (neither open nor closed).
On October 13, 2009, a review of the CAC fan wiring diagrams showed that the fan control circuit utilizes Potter and Brumfield MDR 4121 rotary relays [BK-RLY], to provide a start signal for slow speed operation and to provide an interlock to open the Service Water Inlet and Outlet Valves. The MDR relays in the CAC circuitry use their own contacts to reset the relay coil. Upon receipt of an emergency signal, the two CAC fans connected to the essential buses will automatically start and run in slow speed. The issue with the MDR relays may cause the CAC fan to fail to start due to the relay stopping in an intermediate mechanical position. In order for this failure to occur, the timing has to be just right.
The stopping of the relay in an intermediate mechanical position was able to be reproduced under laboratory conditions. When the relay fails in an intermediate position, the normally closed and normally open contacts are all in the open state. The relays used in the CAC fan circuitry are latch-in style relays, so once the relay is operated it will remain in that condition until a trip signal is received.
Because of this potential to stop in an intermediate position, both operating CAC trains, which were operating in fast speed, were declared inoperable at 1207 hours0.014 days <br />0.335 hours <br />0.002 weeks <br />4.592635e-4 months <br /> on October 13, 2009, and TS LCO 3.6.6 Condition E was entered. Additionally, with Service Water Train 2 and thus Containment Spray Pump 2 inoperable, TS LCO 3.6.6 Condition G was also entered. This required entry into LCO 3.0.3, which required actions to be initiated within one hour to place the unit in Mode 3 within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />. At 1222 hours0.0141 days <br />0.339 hours <br />0.00202 weeks <br />4.64971e-4 months <br />, CACs 1 and 2 were started in slow speed to eliminate the potential of the relay failing in an intermediate position, thereby restoring operability of CAC 1, and TS LCO 3.6.6 Conditions E and G and LCO 3.0.3 were exited. Service Water Train 2 operability was restored on October 13, 2009, at 1556 hours0.018 days <br />0.432 hours <br />0.00257 weeks <br />5.92058e-4 months <br />, which also restored the operability of CAC 2.
DESCRIPTION OF EVENT: (continued) An extent of condition review identified approximately 2 dozen locations at the DBNPS that use these relays. However, only the four CAC fan circuits (CAC 1, CAC 2, CAC 3 as 1, and CAC 3 as 2) use the relay's own contacts to reset the relay coil.
CAUSE OF EVENT:
The cause of the Potter and Brumfield MDR 4121 rotary relays to fail in mid-position was a misapplication of the relays. These relays were not designed to have the coil wired through its own contacts, and in doing so, in some instances the contacts can de-energize the coil before the relay has completely moved the contacts to the spring holding position. This causes all the relay contacts to remain open and not in the desired state. The relay manufacturer makes other relays that are specifically designed to use its own contacts to set or reset the latch and unlatch coils, and the physical point gap of the contact is manufactured to allow the correct timing of the device.
When NRC Information Notice 92-19 was issued in 1992 to describe this issue with the Potter and Brumfield MDR rotary relays, there were no MDR relays used at the DBNPS. In 1999, the MDR relays were identified as a replacement for Westinghouse Latching Type AR relays for which the mechanical latch attachments had been discontinued and the magnetic replacements were not seismically qualified. The Potter and Brumfield MDR rotary relays were installed in the CAC fan circuits in 2000 and 2001. Due to an inadequate design change, the operating experience with the Potter and Brumfield MDR rotary relays was not effectively used to prevent the problems identified.
ANALYSIS OF EVENT:
While laboratory tests of a single relay were successful in reproducing failures to the intermediate mechanical state, this condition occurs with a low incident rate in the field. Each of the eight relays (one to provide a start signal for slow speed operation and one to provide an interlock to open the Service Water Inlet and Outlet Valves) are tested at least twice during monthly surveillance testing, and this event manifested itself only a few times since the relays were installed eight to nine years ago. It is unlikely that both CACs would fail to switch to slow speed during an accident condition. The Containment Spray System, which also serves to cool containment during accident conditions and for which two trains are required to be operable per TS LCO 3.6.6 during Modes 1 through 4, was unaffected by the Potter and Brumfield MDR rotary relay issue. Therefore, this issue was of very low safety significance.
Reportability Discussion:
TS LCO 3.6.6 requires two Containment Spray trains and two CAC trains to be Operable in Modes 1 through 4. Condition C allows one CAC train to be inoperable for up to 7 days, and Condition E allows two CAC trains to be inoperable for up to 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. With these relays installed, rendering all CACs inoperable, the plant operated for a period longer than allowed by TS LCO 3.6.6, which is reportable per 10 CFR 50.73(a)(2)(i)(B) as operation in a condition prohibited by the Technical Specifications.
Likewise, with all CACs inoperable, this issue represents an event or condition that could have prevented fulfillment of the safety function of a system needed to mitigate the consequences of an accident, which is reportable per 10 CFR 50.73(a)(2)(v)(D). On July 1, 2010, at 1358 hours0.0157 days <br />0.377 hours <br />0.00225 weeks <br />5.16719e-4 months <br />, the NRC Operations Center was notified of the loss of safety function as required by 10 CFR 50.72(b)(3)(v)(D).
CORRECTIVE ACTIONS:
Upon discovery of this condition, CAC fans 1 and 2 were switched from fast speed to slow speed operation to eliminate the potential the fans would fail to start in slow speed during accident conditions.
An Engineering Change was issued on November 4, 2009, to modify the control circuitry of the Containment Air Cooler fans so that contacts from the MDR relays are not used to de-energize their own coils. Instead, auxiliary contacts have been installed on the slow speed starters for the CAC fan motors and the circuits re-wired to eliminate the use of MDR relay contacts to de-energize its own relay coil.
A change to procedure NOP-CC-2007, "Part/Component Equivalent Replacement Packages," was made to include guidance to include an Operating Experience Review as an element in selecting a replacement part or component.
A review was performed to ensure the extended operation of the Containment Air Coolers in slow speed and the resultant increased Containment temperatures did not adversely affect the qualified life of equipment in Containment.
PREVIOUS SIMILAR EVENTS
A previous event in June 2004 when CAC 2 failed to start in slow speed during monthly Surveillance Testing was determined to be the stopping of the MDR relay in the intermediate position. At that time, the most probable causes were considered to be a mechanical problem with the relay, even though disassembly at an internal laboratory did not find any deficiencies. The suspect relay was replaced, and the CAC declared operable following successful post maintenance testing. A similar event also occurred in March 2008 when the CAC Service Water inlet isolation valve would not remain closed when CAC fan 1 was shutdown during system realignment.