05000289/LER-2002-001
| Docket Number | |
| Event date: | |
|---|---|
| Report date: | |
| 2892002001R00 - NRC Website | |
instrumentation, and control loads. The system consists of four buss sections: Vital Buss A (VBA), VBB, VBC, and VBD, each supplied from a static inverter (inverter). The inverters are supplied normally from the 480-Volt system through rectifiers with an uninterrupted transfer to a 125-Volt DC source.
At the time of the event, TMI Unit 1 was not critical, with the Reactor Coolant System temperature greater than 250 degrees Fahrenheit. The plant was in a refueling outage and had replaced the station auxiliary transformers earlier in the outage. The new auxiliary transformers have Load Tap Changers (LTC), which can vary the incoming AC voltage to the inverters.
There were no structures, components, or systems that were inoperable at the start of this event, which contributed to this event.
EVENT DESCRIPTION
Summary On December 1, 2001, VBB *[EF/BU] was lost due to the failure of 1B inverter IEF/INV11. The VBB failure occurred during the performance of surveillance testing of the emergency diesel generator, which provides backup power in the event of a loss of off-site power. During the time the inverter was being supplied by the emergency diesel generator, the inverter input cycled between the AC input and the DC input several times. After the seventh transfer to the AC source, the 1B inverter failed due to a blown input fuse.
The inverters were procured from SCI (Solidstate Controls, Inc.) for TMI-1 in the early 1970s. During the analysis of this event, it was determined that the inverters did not meet the design specification for the maximum AC supply voltage. Multiple design changes and proposed design changes since 1984 treated symptoms of inverter cycling but failed to address either the design documents or the inverter design.
I � DOCKET (2) PAGE (3) LER NUMBER (6) The installed inverter 1B was susceptible to cycling for the voltage transients to which it was exposed.
This was because the inverters were supplied with 460/120 versus 480/120 transformers. This resulted in an overvoltage disconnect setpoint of 495 versus 506 volts. Supply AC voltage with voltage transients (noise) exceeding the overvoltage disconnect setpoint caused cycling of the inverter 1B from AC to DC to AC. The step change in voltage during DC to AC transfer was estimated to be 130 VDC to 140+ VDC with voltage transients. This step change can drive the Constant Voltage Transformer (CVT) into the saturation portion of its excitation curve. A saturated CVT can result in high enough current to open input fuse FU2 and shutdown the inverter 1B.
The root causes are that the original design of the inverter did not meet the maximum voltage specification, the design documents did not reflect actual high voltage limits, and these non-conformances were not identified by the inverter factory acceptance testing and documentation when the inverters were accepted from SCI. Contributing causes included design document errors that could have been corrected, high AC supply voltage to the inverters, lack of engineering rigor during design changes, and poor implementation of the Corrective Action process.
The extent of condition is that inverter 1A may have also been susceptible to this condition. The 1D and 1C inverters were not as susceptible to this failure due to the light loads powered by these inverters. The 12/1/01 event showed that the inverters would not all fail at the same time because while the 1B and 1D inverters were cycling, only the 1B inverter failed due to a blown fuse. The 1D inverter cycled for several minutes without adverse consequences.
The risk was evaluated as acceptable (GREEN significance level). The at-risk period included a time- period during the outage prior to heat-up (approximately seven days), and the period of time the plant was greater than 250 degrees Fahrenheit (F). The period when the plant was greater than 250 degrees F was 41.7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />.
Corrective actions address the design discrepancy that led to the high voltage, the procurement process, implementation of a modification to reduce the AC supply voltage to the inverters, actions to address proper implementation of the corrective action process, and actions to address the lack of engineering rigor in preparing design changes.
Condition Statement The deficient condition is that inverter 1 B design was not adequate for the voltage transients to which it was exposed. Supply AC voltage with voltage transients (noise) exceeding the overvoltage disconnect setpoint caused cycling of the inverter 1 B from AC to DC to AC. The step change in voltage during DC to AC transfer was estimated to be 130 VDC to 140+ VDC with voltage transients. This step change can drive the CVT into the saturation portion of its operating curve. A saturated CVT can result in high enough current to open input fuse FU2 and shutdown of inverter 1 B.
Loss of the inverter causes loss of its associated vital buss. Loss of a single vital buss has minor impact.
Loss of multiple or all vital busses causes a reactor trip, initiates ESAS, removes ESAS override capability, affects control of emergency feedwater, and causes loss of accident and post-accident monitoring instrumentation. This event was determined to be reportable on January 15, 2002, as a condition that could have prevented the fulfillment of the safety function of systems that are needed to remove residual decay heat.
Event Description Detail Class 1 E inverters were procured from Solidstate Controls, Inc. (SCI) for TMI-1 in the early 1970s.
They were supplied with an AC input rating of 460 VAC versus vendor design documents specifying 480 VAC. Furthermore, the inverters did not meet the design specification for the maximum AC supply voltage.
The inverters have cycled from AC to DC during plant shutdown, when grid voltage was high and loading was light. However, this cycling has not caused the FU2 fuse to trip. This was an indication that the source AC voltage was too high, but did not cause the event under investigation.
DOCKET (2) LER NUMBER (6) PAGE (3)
NUMBER NUMBER
05000289 2002 � -- � 001 � -- � 00 5 � OF � 11 Multiple design changes and proposed design changes since 1984 treated symptoms of inverter cycling but failed to address either the design documents or the inverter design.
The input fuse (FU2) for Inverter 1B was blown on 12/1/01 during 1303-11.10 surveillance testing resulting in loss of inverter 1B and vital buss VBB.
ROOT CAUSE and OTHER CAUSAL FACTORS The root cause of this event is that the original inverters did not meet specifications and the design documents were incorrect. The purchase specification identified the inverters to be supplied with an AC input range of 460 VAC +/- 10% (414 to 506 volts). The inverters did not meet the purchase specification requirements for maximum AC supply voltage. Furthermore, the vendor documentation did not identify the actual voltage limit. An error precursor for this root cause was that the over- voltage transfer point, specified at 506 VAC, was not identified as a Critical Characteristic of Design and as such was not recognized as an important design feature. This contributed to the failure to recognize that the originally supplied inverters did not meet the requirements of the purchase specification. The design inputs used by the Modification team for replacement of the Auxiliary Transformers did not indicate that the inverters had a lower than specified high voltage transfer point.
One of the design inputs used in the development of the setpoint for the high side of the Auxiliary Transformer Tap Changer control range was to maintain 480 VAC System Voltages below 506 volts.
The fact that the inverters did not meet the requirements of the purchase specification, allowed the design of the normal system voltages to be set such that they approached the transfer point. If the voltage was at or near the setpoint, repeated transfers could and did occur.
There were also five (5) additional causal factors that contributed to this event. These causal factors are described below:
Conditions occurred on April 1986 and October 1988 during which several inverters were transferring between the normal AC and the alternate DC sources. The situation was caused DOCKET (2) I LER NUMBER (6) PAGE (3) 1 � NUMBER I REVISION 6 OF � 11 2002 -- � 001 � -- � 00 by a higher than designed AC voltage being supplied to the overvoltage disconnect circuit. No loss of vital busses occurred during these events. In both cases the inverter vendor was contacted and corrective actions were discussed. In both cases procedure changes and setpoint changes were made to respond to the conditions. The information received from the vendor was not captured in the design change process. Updating the Vendor Manual and inverter drawings to specify identified design information may have prevented future misinterpretation of inverter design voltage and overvoltage transfer setpoint.
2. In 1995 a modification was made to the inverter overvoltage disconnect circuit. During this process, design information previously obtained from the vendor was not placed on the inverter drawings or in the vendor manual. Thus, an opportunity to document and evaluate the inverter voltage limitations was missed.
3. As part of the design of the Auxiliary Transformer Replacement modification (performed during the refueling outage during the 4th quarter of 2001), the taps on the 4160/480 VAC transformers feeding the safety related 480 VAC busses were changed to 4055/480 to increase voltage to improve voltage to safety-related motors. This change also increased voltage to the inverters.
A voltage regulation study selected a normal control band of load tap changer (LTC) 4214- 4270 VAC in concert with the transformer tap change to provide optimum voltage to safety related motors and other loads. On October 27, 2001, during preventive maintenance on an inverter, high AC voltage transfers were being experienced. No failure of the inverter resulted, and the issue was not pursued in a timely manner. Thus, corrective action to address this condition was not initiated or processed in a timely manner, which resulted in a missed opportunity to identify this design deficiency.
4. As a result of the high AC voltage transfers being experienced in casual factor 3 above, troubleshooting showed that the inverter high voltage input disconnect was occurring at a voltage less than expected. The solution recommended by the engineering change process was to lower the LTC voltage range to 4162-4214 VAC. The engineering change process did DOCKET (2) LER NUMBER (6) PAGE (3) 05000289 2002 � -- � 001 � -- � 00 7 � OF � 11 not address the fact that the LTC control band would not be bounding because it would be lower than the upper range of 4300 VAC for the emergency diesel generator (EDG). In hindsight, the engineering change process should have evaluated the EDG voltage range.
This condition alone did not cause the inverter supplying VBB failure. Lowering the maximum EDG voltage to 4218 VAC maximum from 4300 VAC may not have prevented the inverter failure, since the EDG voltage was estimated to be as high as 4212 during the inverter failure event.
A design change in 1995 installed the dead band to the over-voltage disconnect to reduce cycling in the inverters. The 1995 design change also increased the possibility of CVT saturation because it added a delay for the AC disconnect contactor to dropout. This delay was recognized during a visit by the inverter vendor in November 2001, during troubleshooting of the inverters. A design change made in November 2001 removed the dead band for the overvoltage disconnect. Although this modification decreased the possibility of CVT saturation by improving the speed of the overvoltage disconnect, removal of the dead band increased the possibility of cycling when AC voltage is near the setpoint. Review of the EDG voltage during the 1B inverter failure shows that even if the maximum dead band had remained, cycling may have occurred.
CORRECTIVE ACTIONS
Immediate and Interim Corrective Actions:
1. VBB was re-powered from an alternate AC power source, Regulated Transformer "A", TRA on December 1,2001.
2. The input fuse, FU2, for Inverter 1B was replaced and the Inverter 1B was returned to service powering VBB on December 2, 2001.
3. The Emergency Diesel Generator voltage control bands were changed to 4100-4150 VAC on December 3, 2001. This reduced the inverter input voltage when the buss was supplied by the DOCKET (2) LER NUMBER (6) PAGE (3)
NUMBER NUMBER
05000289 2002 � -- � 001 � -- � 00 8 � OF � 11 EDG.
4. The surveillance test which identified the problem with VBB was satisfactorily performed on December 3, 2001.
5. A modification to install manual reset pushbuttons for the inverters was installed on December 4, 2001.
Long Term Corrective Actions to Prevent Recurrence:
1. Update drawings/vendor manual to reflect actual configuration. (Completion date April 30, 2002) 2. Complete the installation of 480/460 VAC transformers for each inverter. (Completion date April 30, 2002) 3. Verify the current procurement process would prevent this type of error. (Completion date April 30, 2002) Additional Corrective Actions:
1. Implement new guidelines for review of future design changes to ensure engineering rigor in the design change process. (Completed) 2. Review corrective action process expectations with engineering on timeliness of initiation/processing and evaluation of extent of condition to assess the inadequate corrective action associated with this event. (Completed) 3. Review Engineering Change Request (ECR) deficiencies with engineers in staff meetings.
Reinforce expectations regarding review of modification documents and fully evaluating the impact of a design change. These reviews will stress engineering rigor in the design change process. (Completed) 4. Train engineering personnel in the new Exelon design change process. (Completion date April 30, 2002) DOCKET (2) LER NUMBER (6) PAGE (3) 05000289 2002 � -- � 001 � -- � 00 9 � OF � 11
ASSESSMENT OF SAFETY CONSEQUENCES
The loss of VBB had no safety consequences other than a loss of instrumentation powered from the vital buss, since the loss of VBB did not result in any automatic component actuation. However, since the condition associated with the loss of VBB had some applicability to other vital power supplies, an extent of condition evaluation was performed to determine if the other vital busses would fail under similar conditions. It was determined that VBA may have also been susceptible to this condition. VBC and VBD were not as susceptible to this failure due to the light loads powered by these busses. The loads on VBC and VBD are approximately half of the loads on the VBA and VBB.
Based on discussions with the original equipment manufacturer, the light loads would not have caused the inverters to blow fuses. The 12/1/01 event demonstrated that only the more heavily loaded busses would fail. Even though both the 1B and 1D inverters were cycling, only the 1B inverter failed due to a blown fuse. The 1D inverter cycled for several minutes without adverse consequences.
The assessment of safety consequences involves the period of time from November 24, 2001, when the inverters were modified to remove the dead band until the emergency diesel generator output voltages were modified on December 3, 2001. This is the period of time VBA and VBB were susceptible to blown fuses on their respective inverters.
During this period of time, plant conditions were changing. These changing plant conditions are divided into two time periods for evaluation. The first time period is when the RCS temperature was below 250 degrees Fahrenheit and the method of decay heat removal was the decay heat removal system (DHR). The second time period was when the RCS temperature was greater than 250 degrees, during which time the method of decay heat removal was the once through steam generators (OTSG). The time period durations were approximately seven days for the first period, and 41.7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> for the second period.
First Time Period Evaluation A qualitative evaluation of the risk that would have been incurred had there been a LOOP from I _ DOCKET (2) LER NUMBER (6) PAGE (3) November 24, 2001 through December 1, 2002 was performed. This evaluation took credit for an operable decay heat removal system (with a redundant train available) and an initiating condition of the reactor coolant system temperature being less than 250 degrees Fahrenheit. The evaluation also assumed the LOOP and resultant cycling of the inverters resulted in a loss of VBA and VBB. VBC and VBD were assumed to remain operable throughout the event.
The qualitative evaluation concluded that the risk of core damage from a LOOP and subsequent loss of VBA and VBB was small because the time available to complete the operator action to restart a decay heat removal system was more than an hour, due to the very low core decay heat level.
Second Time Period Evaluation A second evaluation was performed to determine the risk significance that would be incurred if a LOOP occurred with the subsequent loss of all vital busses with plant RCS temperature above 250 degrees Fahrenheit. An initial condition of having RCS heat removal via the OTSGs was assumed. It was also assumed that there would be automatic HPI initiation' if the RCS temperature was above 250 degrees Fahrenheit.
The second evaluation was performed quantitatively using the TMI PRA. The results of the evaluation found that the risk of core damage and large early release as a result of the inverter cycling problem was small.
For 41.7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> time of concern, the incremental conditional core damage probability, ICCDP for a loss of offsite power event = 7.31 e-7, which is less than 1.0e-6, (the color criteria limit for Green ICCDP), for a risk significance level of Green.
Please note that automatic HPI initiation is not enabled below approximately 329 degrees Fahrenheit. This evaluation assumes that operators would align and start a HPI pump if a LOOP occurred during this condition.
The time duration in which this condition existed was approximately 9 hours1.041667e-4 days <br />0.0025 hours <br />1.488095e-5 weeks <br />3.4245e-6 months <br /> and thus is not a significant factor to the overall risk.
DOCKET (2) LER NUMBER (6) PAGE (3) 05000289 2002 � -- � 001 � -- � 00 11 � OF � 11 For the 41.7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> time of concern, the incremental conditional large early release probability, ICLERP for a loss of offsite power event = 4.91e-8, which is less than 1.0e-7 (the color criteria limit for Green ICLERP), for a risk significance level of Green.
Although the consequences of a LOOP with loss of all vital busses could be significant, the amount of time the plant was exposed to this condition was very limited. Thus the probability of the event was sufficiently small that the overall risk significance was within the Green level.
SIMILAR EVENTS
TMI Unit 1 has had no LERs over the past 3 years that had similar inverter problems. Previous inverter failures at TMI-1 were reviewed. Some of the failures were identified as opportunities to correct design documents, but none of the previous events had the same conditions as this event.
Additionally, a review of industry operating experience showed that there were no similar events at other sites.
- The Energy Industry Identification System (ENS), System Identification (SI) and Component Function Identification (CFI) Codes are included in brackets, [SI/CFI] where applicable, as required by 10 CFR 50.73 (b)(2)(ii)(F).