05000286/LER-2007-001
| Docket Number | |
| Event date: | 4-03-2007 |
|---|---|
| Report date: | 6-4-2007 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation 10 CFR 50.73(a)(2)(v), Loss of Safety Function |
| 2862007001R00 - NRC Website | |
Note: The Energy Industry Identification System Codes defined in IEEE Std 803-1983 (components) and IEEE Std 805-1984 (Systems) are identified within the brackets {}.
DESCRIPTION OF EVENT
On April 3, 2007, at approximately 53% reactor power, after a power reduction and shut down of the 31 Main Boiler Feedwater Pump (MBFP) {P}, while preparing to perform maintenance for the 31 MBFP indicator control power supply {JX}, the logic control power supply for the 32 MBFP was unexpectedly lost resulting in the loss of main feedwater (FW) {SJ} flow. Control room (CR) operators manually initiated a reactor trip (RT) {JC} at approximately 0417 hours0.00483 days <br />0.116 hours <br />6.894841e-4 weeks <br />1.586685e-4 months <br />, after observing rapidly decreasing steam generator (SG) {AB} levels. All control rods {AA} fully inserted and all required safety systems functioned properly. The plant was stabilized in hot standby with decay heat initially being removed by the main condenser {SG} through the main steam (MS) {SB} dump valves. At approximately 0446 hours0.00516 days <br />0.124 hours <br />7.374339e-4 weeks <br />1.69703e-4 months <br />, operators closed the MS isolation valves (MSIVs) {ISV} due to low reactor coolant system average temperature. There was no radiation release. The Emergency Diesel Generators (EDGs) {EK} did not start as offsite power remained available. The Auxiliary Feedwater (AFW) {BA} pumps automatically started as expected due to SG low level from shrink effect. An investigation into the cause of the event and.a post transient evaluation was initiated.
During plant startup from an outage there were several problems with the power supply for the 31 MBFP Lovejoy control system {JK} one of which was the +5 volt display power supply of Train B. A decision was made to replace the power supply and since its wiring was soldered the power supply was required to be isolated to perform the work. Because both the 31 and 32 MBFP Lovejoy control system power supply wiring was in the same electrical box, a trip risk was possible and a decision was made by plant management to take the 31 MBFP out of service and isolate the power to the 31 MBFP Lovejoy control panel. At approximately 0320 hours0.0037 days <br />0.0889 hours <br />5.291005e-4 weeks <br />1.2176e-4 months <br />, operators began a 50 MW power reduction to support shutting down the 31 MBFP to replace its Lovejoy display power supply. Prior to the event all Control Rods were withdrawn from the reactor core and in Auto, AFW pumps were in standby, the EDGs were in standby, off-site power was in service, the 32 MBFP was in-service with its Foxboro Control System in manual, and 31 MBFP was shutdown. Instrumentation and Control (I&C) personnel were to replace the 31 MBFP + 5 volt Train B power supply in order to maintain two trains of power supply for MBFP Lovejoy indications. The power supply replacement required de-energizing the instrument power supply for 31 MBFP Train B. The Field Support Supervisor (FSS) prepared tagouts and identified 118 volt supply breaker 10 in Instrument Bus Panel 31A per plant drawings which was verified by Engineering as the correct supply. Troubleshooting prior to the de-energization indicated the Train A power supply was normal when it was paralleled with the Train B power supply. At approximately 0414 hours0.00479 days <br />0.115 hours <br />6.845238e-4 weeks <br />1.57527e-4 months <br />, the circuit breaker {BKR} was opened to de-energize the 31 MBFP Lovejoy power supply on instrument bus 31A, {EE} circuit 10. At that time the indicated 32 MBFP speed reduced to approximately 800 rpm, but with 32 MBFP discharge pressure ramping up. As a result of the unexpected response; Operators closed the circuit breaker for instrument bus 31A circuit 10 and attempted to increase 32 MBFP speed manually with no response. The AMSAC Alert alarm activated (indicating low FW flow) and the MBFP recirculation valve opened and SG levels were observed decreasing. With an apparent loss of FW control and decreasing SG levels, a manual RT was initiated.
NRC FORM 366AU.S. NUCLEAR.REGULATORY COMMISSION 0-200-0 FACILITY NAME (1) DOCKET (2) LER NUMBER (6) PAGE (3) On April 3, 2007, at 0507 hours0.00587 days <br />0.141 hours <br />8.382936e-4 weeks <br />1.929135e-4 months <br />, a four hour non-emergency notification was made to the NRC (Log Number 43272) for a reactor trip while critical and included the eight hour non-emergency notification for actuation of the AFW system. The RT was reported under 10 CFR 50.72(b)(2)(iv)(B) and the AFW actuation reported under 10 CFR 50.72(b)(3)(iv)(A). The event was recorded in the Indian Point Energy Center corrective action program (CAP) as CR-IP3-2007-01775.
FW to the SGs is supplied by two steam turbine driven MBFPs with an oil control system. The control oil system adjusts the position of the high pressure and low pressure stop and governor valves based on a signal from the MBFP speed control system. The speed of each MBFP turbine is controlled with a combination of electric, pneumatic and hydraulic controls. The Lovejoy Control System uses a microprocessor to convert the electrical output of the Foxboro Control System into an equivalent control oil pressure signal to control the Westinghouse MBFP's turbine speed control system. The control oil system provides varying pressure to modulate (control) the MBFP governor valves by positioning the governor valve's servomotor relay piston. Each Lovejoy control system has two logic power supplies and two display power supplies. The controller for the 31 MBFP utilizes the supply from 118 VAC Instrument Bus 31A for its Train A power supplies and 118 VAC.
Instrument Bus 32A for its Train B power supplies. The outputs from each controllers' logic power supplies are connected through diodes to a common output (auctioneered) that supplies the microprocessor. At any given time the power supply with the higher voltage is supplying the microprocessor and the other is in standby. The display power supplies are configured with the same design. The preventive maintenance (PM) for the Lovejoy control system is to test and calibrate the MBFP Lovejoy speed control in accordance with the vendor manual.
The vendor manual contains calibration procedures for the system's electronics and pneumatic controls but there are no PM procedures specified for the Lovejoy power supplies. There is no action to verify that each power supply is capable of supplying the load. Lovejoy issued an Advisory Bulletin following this event that recommends replacing the power supplies as they approach 10 years.
Troubleshooting the MBFP Lovejoy control system subsequent to the RT found that the 32 MBFP Train A + 5 volt logic power supply was degraded in a way that when it was not supplying the load, its output voltage was normal, but when loaded its output voltage dropped to +1 volt.
Cause of Event
The direct cause of the RT was decreasing SG levels due to loss of 32 MBFP FW control as a result of the failure of the 32 MBFP Train A Lovejoy control logic power supply when the 31 MBFP instrument bus power supply was isolated. The 32 MBFP Train A Lovejoy logic power supply failure was due to a bad connection on pin 4 of the power supply voltage regulator. When the 31 MBFP Train B Lovejoy power supply was de-energized, the 32 MBFP Train A Lovejoy logic power supply was unable to supply the logic microprocessor load and the voltage degraded low. The degraded voltage caused the microprocessor to fail and resulted in loss of 32 MBFP control. The root cause was a failure of an auctioneered power supply that was not self revealing and was undetectable until it was required to carry load.
Typically power supplies degrade due to aging of their electrolytic capacitors and operate satisfactorily until failure. Prior to the event the power supply voltages were verified and indicated satisfactorily until the 32 MBFP Train A Lovejoy logic power supply was loaded by itself. The degradation of the power supply was not a typical or expected failure mode. With the power supplies that are not tested under load but in the auctioneered,configuration, the power supply with the higher voltage will block the weaker from carrying the load.
Engineering believed that a PM to replace the Lovejoy power supplies on a time based interval was not required since the Lovejoy controller system has indications for power supply failure and any failure would be indicated and the power supply replaced at that time. The Lovejoy vendor documents did not identify any required maintenance on the power supplies internal components except voltage calibrations. A contributing cause was controlled documents relating to the design of the Lovejoy system did not indicate that each 118 volt Instrument Bus supply had connections to both Lovejoy controllers. A risk based decision was made to isolate circuit 10 in Instrument Bus panel 31A based on controlled documents that indicated that this circuit was the supply for the 31 MBFP only. Had the documents identified that this circuit actually supplied both Lovejoy controllers and could impact the operating MBFP a different decision may have been made. The Lovejoy power supply whose output voltage dropped low due to a bad connection on pin 4 was manufactured by ACDC Electronics {A397}, Model ETV 351.
An extent of condition review determined that there are other power supplies that could be vulnerable to failures.. A recent power supply failure on a Unit 2 MBFP suction pressure transmitter determined that Indian Point had not implemented a power supply refurbishment and replacement program. Corrective actions for that event were to develop and implement an Instrument Power Supply PM and a capacitor program. Sufficient time has not transpired for these corrective actions to be fully implemented.
Corrective Actions
The following corrective actions have been or will be performed under Entergy's Corrective Action Program to address the cause and prevent recurrence
- Replaced the 31 MBFP Lovejoy Train B +5 volt display power supply and the 32 MBFP Lovejoy Train A +5 volt logic power supply.
- Establish the Instrument Power Supply PM that will replace all power supplies in the Lovejoy system. Scheduled completion date is July 31, 2007.
Replacement of power supplies will commence in the next refueling outage scheduled for the spring of 2009.
- Perform an assessment to identify other auctioneered power supplies and verify if PMs confirm that the power supplies can carry their load.
Scheduled completion date is August 31, 2007.
- Revise as necessary the Lovejoy PMs at both units to test that each power supply is capable of supplying its load. Scheduled completion date is August 31, 2007.
- Correct controlled documents associated with the Lovejoy MBFP control system (Drawings 9321-F-39893, 9321-LL-30412 sheet 14, 9321-LL-30412 Sheet 14A; description of Instrument Bus 31A Circuit 10 and Instrument Bus 32A circuit 10 in 3-COL-FW-1; System Description 21.0, "Main Feedwater"). Changes are scheduled to be implemented by August 31, 2007.
Event Analysis
The event is reportable under 10CFR50.73(a)(2)(iv)(A). The licensee shall report any event or condition that resulted in manual or automatic actuation of any of the systems listed under 10CFR50.73(a)(2)(iv)(B). Systems to which the requirements of 10CFR50.73(a)(2)(iv)(A) apply for this event include the Reactor Protection System (RPS) including RT, AFWS actuation and MSIV closure.
This event meets the reporting criteria because a manual RT was initiated at 0417 hours0.00483 days <br />0.116 hours <br />6.894841e-4 weeks <br />1.586685e-4 months <br />, on April 3, 2007, and the AFWS actuated as a result of the RT. Although the MSIVs were closed due to low reactor coolant system average temperature, the action was not to mitigate the consequences of the event. The failure of the MBFP speed controller power supply did not result in the loss of any safety function.
Therefore, there was no safety system functional failure reportable under 10CFR50.73(a)(2)(v).
PAST SIMILAR EVENTS
A review of the past two years of Licensee Event Reports (LERs) for unit 3 events that involved a RT from loss of a power supply identified no applicable LERs.
However, unit 2 had three LERs in the past two years that reported RTs as a result of power supply failures. LER-2006-003 reported a RT due to a mismatch of reactor power to turbine load during a power reduction from a loss of heater drain tank (HDT) pumps. The HDT pumps were lost due to a failure of the power supply for the HDT level transmitter (LT). The direct cause of the HDT LT failure was a faulty capacitor. The root cause was a programmatic weakness with classification and management of critical PM tasks. LER-2006-005 reported a RT due to a generator exciter trip caused by a Generrex power supply failure. The power supply failed due to a bad common ground connection. LER-2007-004 reported a RT due loss of FW flow as a result of a failure of the power supply for the MBFP suction pressure transmitter. The power supply failed due to capacitor aging. The unit 2 LER-2007-004 included a CA to develop and implement at IPEC an Instrument Power Supply PM in accordance with the Entergy Nuclear South (ENS) PM Template. The Instrument Power Supply PM CA also included auctioneered power supplies discovered in this event. The CA for LER-2007-004 could have prevented this event but the CA had not yet been implemented due to the short time period from the previously reported event (LER-2007-004).
Safety Significance
This event had no effect on the health and safety of the public. There were no actual safety consequences for the event because the event was an uncomplicated RT with no other transients or accidents. Required primary safety systems performed as designed when the RT was initiated. There were no risk related components out of service at the time of the RT. The main steam isolation valves were manually closed by operations as a result of low RCS temperature that was due to excessive cooling as a result of a new core with low decay heat. Reactor core cooling through the SG was controlled by use of the main steam power operated relied valves (PORVs) rather than by use of the steam dumps to the condenser. A condition of excessive heat removal from the SGs is bounded by the analysis in FSAR Section 14.2.5, "Rupture of a Steam Pipe," which bounds excessive FW flow with reactor at zero power analyzed in Section 14.1.10, "Excessive Heat Removal Due to FW System Malfunctions." The AFWS actuation was an expected reaction as a result of low SG water level due to SG void fraction (shrink), which occurs after automatic RT from full load. There were no significant potential safety consequences of this event under reasonable and credible alternative conditions. The loss of FW flow for this event was bounded by the analysis in FSAR Section 14.1.9, "Loss of Normal FW." The AFWS actuated and provided required FW flow to the SGs. The AFW capacity is sufficient to provide the minimum required FW flow to the SGs. For this event the RCS pressure remained below the set point for pressurizer PORV or code safety valve operation and above the set point for automatic safety injection actuation. Following the RT, the plant was stabilized in hot standby.