05000285/LER-2007-002
| Fort Calhoun Station | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(vii), Common Cause Inoperability |
| 2852007002R00 - NRC Website | |
BACKGROUND
The raw water (RW) system at Fort Calhoun Station (FCS) provides cooling water from the ultimate heat sink (Missouri River) to heat exchangers that supply component cooling water to safety related systems. There are four RW pumps powered by 4160 volt (V) motors. The RW pump motors are powered from the 4160V vital busses.
The circuit breakers with the condition described in this LER were manufactured by Asea Brown Boveri (ABB) as a retrofit application, replacing the originally supplied General Electric (GE) Magna-Blast circuit breakers in 1995. The model number associated with RW pump circuit breakers is ABB 5VKBR-250. This modification affected all of the station's 4160 V circuit breakers. The part which has broken is described as the Mechanism Operated Contact (MOC) offset rod (see Figure 1). The function of the rod is to transmit circuit breaker operating mechanism state change, through rod movement, to provide linear motion to a plunger which in turn operates a circuit breaker position switch (MOC switch) mounted in the stationary part of the circuit breaker cubicle.
The MOC linkage offset rod is made of American Iron and Steel Institute (AISI) C-1018, (plain) cold rolled carbon steel, and has a nominal diameter of 5/16 inch. The rod has a smooth outer surface that was plated with zinc and a dichromate coating for corrosion protection. The rod has nominally 1 inch of cut thread (5/16-18) on each end. The rod is oriented approximately vertical inside the circuit breaker frame. The bottom of the rod is actuated by the rotation of a jackshaft through a clamped crank link and shoulder stud with a pivoting rod attached to the MOC linkage offset rod. There is approximately a 1-7/8 inch horizontal centerline offset between the pinned ends. The upper end of the rod connects to a pinned bell crank (using an identical rod end to the bottom connection) which causes a plunger to rise and fall.
The MOC switch provides feedback to various control circuits as to the open/close status of the circuit breaker. When the circuit breaker has been elevated to the fully connected position (racked in), the switch will be actuated whenever the circuit breaker is closed. The MOC switch is actuated by the plunger interlock (plunger) mounted on the top of the circuit breaker mechanism (see Figure 2).
The MOC switch has a number of "a" contacts (closed when the circuit breaker is closed and open when the circuit breaker is open) and "b" contacts (open when the circuit breaker is closed and closed when the circuit breaker is open). The MOC switch for the RW pump circuit breakers provides a signal to open the discharge valve from the pump when the circuit breaker is in the CLOSED position. If the MOC offset rod breaks, the circuit breaker successfully closes, but the MOC contacts do not change state. This causes the discharge valve to remain closed when the RW pump motor circuit breaker closes and prevents the associated pump from delivering water to the RW header.
EVENT DESCRIPTION
On January 25, 2007, the 4160V circuit breaker for RW pump AC-10B closed on demand but the MOC switch did not actuate. The failure was determined to be a broken MOC offset rod. Inspection of the MOC offset rod determined that the rod had failed due to cyclic fatigue. The MOC offset rod was replaced and AC-10B was returned to service. A visual inspection of the other three RW pump circuit breakers, performed at that time, did not reveal any similar damage.
On February 8, 2007, a similar event occurred where the circuit breaker RW pump AC-10C closed on demand, but the MOC switch did not actuate. The failure was determined to be failure of the MOC offset rod in the associated 4160V circuit breaker. The MOC offset rod on the 4160V circuit breaker for AC-10C also failed due to cyclic fatigue. A condition report was initiated for engineering to review the potential for a common mode failure.
Engineering reviewed the number of cycles of the circuit breakers for the RW pumps. The circuit breaker for AC-10B had over 1200 cycles when its MOC offset rod failed. The circuit breaker for AC-10C had over 1700 cycles when its MOC offset rod failed. The circuit breakers for AC-10A and AC-10D each had over 1600 cycles. Based on the similar numbers of cycles, engineering initially determined that continued operation of AC-10A and AC-10D without circuit breaker failure could not be assured. All other safeguards associated circuit breakers were verified to have less than 1000 cycles (the limit established for MOC offset rod operability) and were not considered to be susceptible to circuit breaker failure at that time.
RW pumps AC-10A and AC-10D were declared inoperable on February 8, 2007 at 1342 CST based on the number of circuit breaker cycles for these two components in relation to the two circuit breakers that failed.
With three RW pumps inoperable, FCS entered technical specification (TS) 2.0.1 which requires the plant be placed in hot shutdown within six (6) hours. The potential for a common mode failure of the MOC offset rods in the 4160V circuit breakers could have prevented operation of the RW pumps to fulfill the required design function to remove residual heat during a design basis accident. An 8-hour notification was made to the Headquarters Operation Office (H00) per 10 CFR 50.72(b)(3)(v)(B).
On February 8, it was not known if the other two RW pump circuit breakers could fail by this same mechanism.
The MOC offset rods of the circuit breakers for RW pumps AC-10A and AC-10D were subsequently inspected and tested.
Non destructive examination (NDE) determined that the rods had no cracks. Destructive testing by a certified facility determined that the rods would not have failed for several hundred additional operations of the circuit breakers. Testing in the facility also determined that no pre-existing cracking existed in the MOC offset rods for AC-10A and D. As a result of the testing and examination results it was determined that the RW system had been capable of performing its design safety function. Therefore, this event was not reportable under 10 CFR 50.72(b)(3)(v)(B) as previously reported and the verbal report to the 1-100 was retracted on March 15, 2007.
While no verbal report criteria is applicable to this condition this event is reportable per 10CFR 50.73(a)(2)(vii).
CONCLUSION / ROOT CAUSE
The ABB retrofit circuit breakers were designed in 1994 and placed in service at FCS in 1995. No indication of problems related to the MOC offset rods occurred until their failure in 2007.
The ABB circuit breakers were designed as a roll-in replacement for the originally supplied GE Magna-Blast circuit breakers. ABB Services Company obtained a Class 1E qualified circuit breaker mechanism from another branch of ABB and then designed the interfaces such as the circuit breaker carriage, interlocks, and MOC linkage required to mate the ABB circuit breaker with the GE switchgear. Since ABB was not the original equipment manufacturer for the GE switchgear, a number of assumptions were made during the design phase involving the design of these interfaces. To accomplish the reverse engineering process, ABB Services Company procured two spare GE type M-26 switchgear cells (one for the 1200 ampere circuit breakers and one for the 2000 ampere circuit breakers) to allow measurements of critical characteristics and to ensure proper fit of the replacement circuit breaker. ABB Nuclear then performed qualification/dedication services to allow the retrofitted circuit breakers to be used in a Class lE nuclear safety related application.
ABB Services Company is no longer in business and design records related to the design of the retrofit are not available for review. ABB Nuclear responsibilities have been assumed by Westinghouse Electric Company (WEC). WEC personnel were contacted regarding this investigation, but were unable to produce information related to the original design of the MOC linkage.
Original qualification testing of the ABB circuit breakers demonstrated successful completion of mechanical aging tests to 2000 operating cycles. Routine inspection of the MOC offset rods is not part of the preventive maintenance requirements established by the vendor for the circuit breakers. The MOC offset rods are not on the recommended spare parts list that was provided by ABB, so it is not considered that their failure prior to 2000 circuit breaker operating cycles would have been expected by ABB. Prototype circuit breakers were mechanically aged as part of the qualification testing performed by Wyle Laboratories for ABB to 2000 operating cycles.
During the qualification testing, the circuit breakers were housed in a switchgear cell and demonstrated satisfactory performance of the MOC linkage. The qualification testing did not include (nor would have been expected to include) operation of the circuit breaker in the test configuration. The qualification testing to 2000 cycles was performed in accordance with American National Standards Institute (ANSI) C37.59 requirements to perform "mechanical endurance testing to at least the between-servicing levels." The recommended service level is 2000 operating cycles. In reviewing the qualification test reports, there are no indications of failure of the circuit breaker subcomponents during the mechanical aging test.
As explained earlier, the MOC linkage is used to operate the MOC device installed in the switchgear. Industry experience with the external MOC devices has shown that the circuit breaker mechanism can stall (i.e., operation can be slowed significantly) due to interaction with the MOC during circuit breaker operation. In the events described with the RW pump circuit breakers, the circuit breakers successfully closed even when the MOC linkage failed.
In normal operation, the circuit breaker is racked vertically into the switchgear, the top of the circuit breaker MOC plunger is spaced 1/8-inch away from the end of the switchgear MOC actuating rod when the circuit breaker is in the open position.
For testing, a feature is provided to operate the circuit breaker without sending 4160V power to the load. In this "test" configuration, the circuit breaker is racked down and partially withdrawn from the cubicle. An offset metal "test flag" device (explained below) is installed to bridge the vertical gap between the MOC plunger bolt on the top of the circuit breaker and the end of the switchgear MOC actuating rod.
Available information suggests that neither Omaha Public Power District (OPPD) nor ABB Services Company recognized the additional stresses that would be imposed on the MOC linkage in the circuit breaker due to the usage of the test flag device. Processes involving the identification of critical interface/operating configurations when specifying material procurement requirements failed to identify the usage of the test flag as a critical constraint. This is considered the root cause for the MOC offset rod failure.
The test flag is a metal device weighing approximately two pounds that imposes additional stresses on the MOC linkage when used. The test flag shape causes the normally linear forces ('y' direction) between the circuit breaker MOC plunger and the switchgear MOC actuating rod to have a side force component ('x' direction) that increases the frictional loading on the MOC plunger and the switchgear MOC switch actuator. The weight of the test flag is also substantial enough that during the high speed operation of the circuit breaker, significant additional loading on the MOC linkage is imposed by the test flag. Breaker timing tests on the ABB circuit breakers demonstrate closure times of about 30 milliseconds (ms) and opening times of about 15 ms, which are estimated to generate accelerations on the order of 10 gravities (g) to 100 g on the circuit breaker plunger, test flag and MOC switch plunger masses. The resultant inertial loads are judged to be significant contributors to the fatigue failure of the rods.
The original OPPD purchase contract for the replacement circuit breakers, it was required that "...the circuit breakers, with interrupter, are otherwise identical to and interchangeable with the existing GE Magna-Blast air-magnetic circuit breakers..." However, there was no specific mention of the test flag or a requirement to perform production testing with the test flag in the contract. This testing would not have resulted in a failure of the MOC offset rod since the production testing did not require mechanical endurance testing, but would have prompted ABB to consider the forces imposed by the test flag on the MOC linkage in the circuit breaker design.
ABB did perform production testing of each circuit breaker provided to OPPD that included placing the circuit breaker in the test position in the prototype switchgear cell but this did not include the installation of the test flag. Improved communication between OPPD and ABB on the use of the test flag could have resulted in the identification of this as a design consideration. OPPD did not identify the use of the test flag as a critical interface in the contract specifications.
The OPPD design process relies on knowledge and experience of personnel conducting design activities to accurately identify all critical operating characteristics. The contract refers to operating the circuit breaker in the test position and has specific requirements for connecting the control power to the circuit breaker in the test position, but does not mention the use of the test flag.
The visual examination of the failed rods shows that both rods failed in the same general area of the thread. The use of die cut threads in the MOC offset rod introduced high intensity stress locations which served as fatigue crack initiation sites.
The selection of this fabrication method by ABB caused the effects of high stress loading on the MOC offset rods to manifest themselves earlier than if rolled threads had been selected. This is considered a contributing cause for the MOC offset rod failure.
The test flag installation at FCS was not adequately controlled to ensure that critical gaps were maintained when the test flag was used. This is considered a contributing cause for the MOC offset rod failure. Operating Instruction (OI)-EE-1A, "4160 Volt Circuit Breaker And Control Switch Operation," provides instructions for placing a 4160V circuit breaker in the test position. This procedure instructs the electricians to install the test flag loosely to avoid damage to the circuit breaker. A tightly installed test flag significantly increases the stress loading of the MOC linkage. Field testing of the MOC actuator spring force shows that the force required to move the MOC actuator increases significantly (1/8-inch movement past the normal compression results in a 20 to 25 percent increase in force) and could cause the spring to become fully compressed and present a "hard-stop" condition for the MOC actuator.
A review of operating experience indicates that other facilities have developed preventive maintenance tasks to periodically measure the travel of the switchgear MOC actuating rod and to measure the force required to operate the rod.
OPPD has been unable to establish that institution of these practices would have prevented these events.
CORRECTIVE ACTIONS
Until a long term replacement strategy is developed; 1. The MOC offset rods in the circuit breakers of RW pumps AC-10C, A and D were replaced with rods having less than 1000 cycles of operation.
2. The other affected 4160V safety related circuit breakers having more than 1000 cycles of operation have been rebuilt with rods having fewer than 1000 cycles of operation.
3. An administrative method to track number of cycles on MOC offset rods has been developed.
OPPD is developing a replacement strategy for the MOC offset rods. Implementation of this strategy will be controlled by the corrective action system.
SAFETY SIGNIFICANCE
The MOC for the RW pump circuit breakers provides a signal to open the discharge valve from the pump when the circuit breaker is in the CLOSED position. Failure of the offset rod for the RW pump circuit breaker, for AC-10B and AC-10C, caused the discharge valve to remain closed when the circuit breaker CLOSED and prevented the respective RW pump from delivering water to the RW header. When the second offset rod failed, it was determined that a common mode failure existed and the other two raw water pumps were declared inoperable, placing the FCS outside Technical Specification (TS) guidance for allowable inoperable Raw Water pumps. TS 2.0.1 was entered until sufficient RW pumps could be restored to operation to meet TS requirements.
Other potentially affected circuit breakers have been examined to determine the impact of failure to actuate the MOC switch.
After the second RW pump MOC offset rod failure (AC-10C), it was apparent that the potential for failure of the remaining RW MOC offset rods existed (in the breakers for AC-10A and AC-10D). A failed offset rod, in the case of the RW pump circuit breakers, results in a failure of the RW pump discharge valve to automatically open when the pump starts (circuit breaker closes). The MOC offset rods for these pumps had 1671 and 1688 cycles, respectively. This is in excess of the judged threshold for failure of 1000 cycles and these RW pumps were then considered to be inoperable. With three (3) RW pumps inoperable, FCS entered TS 2.0.1.
The rods removed from AC-10A and AC-10D were subsequently replaced and the removed rods were examined and found to be free from crack-like indications. Destructive testing indicated that there was no imminent risk of MOC offset rod failure that would have prevented the RW pump discharge valves from opening when the RW pump started.
The failure of the MOC offset rod will cause the MOC switch to return to (or remain in) its circuit breaker open position.
With the circuit breaker actually closed, the normally closed MOC switch contacts would remain closed and the normally open contacts would remain open. This is opposite of their position with the circuit breaker closed and the MOC offset rod intact.
A review of each MOC switch contact for all safety related 4160V circuit breakers was done to evaluate the impact of a failed MOC offset rod. This evaluation determined an adverse result could have occurred if a MOC offset rod had failed for one of the safety related 4160V/480V in-house transformers. These circuit breakers are interlocked with their respective 480V main secondary circuit breakers that connect the in-house transformers to their 480 volt bus. A MOC offset rod failure in the 4160V circuit breaker feeding one of these transformers would trip the 480 volt bus feeder circuit breaker and block it from being reclosed until the MOC rod was replaced. This would result in the loss of one 480V bus.
The loss of a 480V bus is an analyzed condition. However, the circuit breaker cycle counts for all six (6) in-house transformers is less than 225 cycles, which is well below the 1000-cycle threshold where MOC offset rod failure would be expected to occur.
A potential worst case condition that could have also occurred is that of a MOC offset rod failure in the circuit breaker of the largest 4160V safety bus load (reactor coolant pump motor) and a subsequent failure of the associated undervoltage relay contact for that load. A failure of the MOC offset rod for the RC-3C reactor coolant pump motor breaker would have allowed a running RC pump to remain on the bus. In the event that the diesel generator (DG-1) had started on bus undervoltage and had the undervoltage auxiliary relay for RC-3C failed to trip the RC-3C circuit breaker, a subsequent closure demand of the DG circuit breaker on the bus could have resulted in DG failure from overload. However, the loss of a 4160V bus and or DG is an analyzed condition. Additionally, the circuit breaker cycle count on all RC pumps is 253 cycles or less, which is well below the 1000-cycle threshold for expected MOC rod failure. Therefore, there was minimal effect on the health and safety of the public.
SAFETY SYSTEM FUNCTIONAL FAILURE
This event does not result in a safety system functional failure in accordance with NEI-99-02.
PREVIOUS SIMILAR EVENTS
There have not been any other instances of a similar nature resulting in a failure of circuit breakers at the Fort Calhoun Station in the last three years.
Figure 1 MOC Offset Rod in Clouded Area Figure 2 Test Flag (Link) Configuration