05000286/LER-2017-004

LER-2017-004, Reactor Trip Due to Main Generator Loss of Field

Indian Point 3
Event date:	11-03-2017
Report date:	12-20-2017
Reporting criterion:	10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation
2862017004R00 - NRC Website

LER 17-004-00 for Indian Point Unit 3, Regarding Reactor Trip Due to Main Generator Loss of Field

ML17363A212
Person / Time
Site:	Indian Point
Issue date:	12/20/2017
From:	Vitale A J Entergy Nuclear Northeast
To:	Document Control Desk, Office of Nuclear Reactor Regulation
References
NL-17-151 LER 17-004-00
Download: ML17363A212 (7)

comments regarding burden estimate to the Information Services Branch (T-2 F43), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by e-mail to used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

05000-286 Indian Point 3 2017 - 00 - 004 Note: The Energy Industry Identification System Codes are identified within the brackets { }.

DESCRIPTION OF EVENT

On November 3, 2017, at 2022 hours0.0234 days <br />0.562 hours <br />0.00334 weeks <br />7.69371e-4 months <br />, with reactor power at 100 percent, the Indian Point Unit 3 (IP3) Control Room operators received a Turbine Trip First Out Annunciator {ALM} and an automatic reactor trip {JC}, initiated by a main generator {TB} Lockout Relay 86BU {RLY, 86} trip. The 86BU relay trip was due to actuation of the main generator Loss of Field Relay 40 {RLY, 40}, and resulted in a direct trip of the 345 kilo-Volt (kV) generator output breakers 1 and 3 {EL, BKR, 52}. All control rods {AB} fully inserted and all required safety systems functioned properly. The plant was stabilized in hot standby with decay heat being removed by the main condenser {SG}. There was no radiation release. The emergency diesel generators {EK, DG} did not start, as offsite power remained available and stable. The Auxiliary Feedwater System (AFWS) {BA} automatically started as expected on steam generator (SG) {AB, SG} low level as a result of void fraction (shrink) effect. Indian Point Unit 2 (IP2) was unaffected and remained at 100 percent power.

The Generator Protection System protects the main generator from internal and external faults by tripping the field excitation breaker {BKR, 41} and the 345 kV generator output breakers 1 and 3. These circuit breakers are tripped by the Primary (86P) and Backup (86BU) Generator Lockout Relays {RLY, 86}, which also initiate a turbine trip {JJ}. The Turbine Protection System energizes solenoid valves 20/AST and 20/ASB {TG, PSV} to dump the autostop oil when a turbine trip is required. This removes the autostop oil pressure, allowing the turbine stop valves {TG, SHV} to close by spring action. A turbine trip can be actuated by a: (1) main generator trip, (2) reactor trip {JC}, (3) safety injection {BQ} actuation, or 4) manual trip. The Primary and Backup Generator Lockout Relays (86P and 86BU) provide the main generator trip signals to energize the 20/AST and 20/ASB solenoid valve for a turbine trip.

The November 3, 2017 reactor trip event was reported to the Nuclear Regulatory Commission (NRC) in a 4-hour non-emergency notification under 10 CFR 50.72(b)(2)(iv)(B) for an actuation of the Reactor Protection System (RPS) when the reactor is critical, and included an 8-hour notification for a valid actuation of the RPS and AFWS under 10 CFR 50.72(b)(3)(iv)(A) (Event Log No. 53052). This event notification was updated on November 6, 2017 to revise the report to reflect the actual Reactor Trip First Out Annunciator, which was for Generator Lockout Relay actuation of a turbine trip. The initial notification incorrectly stated that the reactor tripped on 33 SG low level. As previously described, SG low level is an expected post reactor trip transient condition. The event was recorded in the Indian Point Energy Center (IPEC) Corrective Action Program (CAP) as CR-IP3-2017-05133. A post transient evaluation was initiated and completed on November 7, 2017.

Prior to the November 3, 2017 reactor trip event, on August 31, 2017, the IP3 Control Room had received an Exciter Trouble alarm {ALM}. The operators dispatched to investigate discovered that the Drawer Operative light for one of the two Thyristor Firing Module drawers (Module B) was extinguished, indicating a loss of pulse. They also noted an acrid odor in the vicinity of the drawer and burn marks on the underside of the drawer. The Thyristor Firing Modules are part of the Main Generator Exciter System {TL, EXC}. No main generator capability was lost at the time, and IP3 remained online at full power. However, the Exciter Trouble alarm could not be cleared. So Engineering consulted the equipment vendor, Siemens-Allis {S188}, to understand the consequences of the discovered conditions. The vendor stated that the two Thyristor Firing Module drawers are parallel and redundant, and that full generator capability could still be maintained while operating on a single drawer. In addition, the vendor did not recommend opening the failed Thyristor Firing Module drawer because the vendor could not be certain of the material condition of the system, and opening the drawer may introduce a risk of tripping the generator. A Critical Decision Paper (CDP) and an Operator Decision Making Instructions (ODMI) document were prepared using vendor-provided information to assist plant management in determining whether or not to enter a comments regarding burden estimate to the Information Services Branch (T-2 F43), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by e-mail to used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

Indian Point 3 2017 - 00 - 004 05000-286

3. LER NUMBER

forced outage to repair the failed Thyristor Firing Module drawer. The decision was ultimately made to wait until the next forced outage or refueling outage to effect the repair. To support this outage repair work, a work order was prepared and the necessary replacement parts were staged.

The Main Generator Exciter System supplies the Direct Current (DC) field excitation current for the generator. It automatically maintains the generator output voltage and controls reactive load in accordance with the setting determined by the operator. It also maintains the generator output within the design capability to keep the generator in synchronism with the electrical transmission grid system. The Exciter System is a rotating, brushless type system that consists of a permanent magnet generator (PMG) {PMG}, an Alternating Current (AC) generator {GEN}, and a rotating rectifier assembly {RECT} mounted on a common shaft. The PMG provides 120 Volt AC, 420 Hertz (Hz), 3-phase power to the exciter field through the field breaker and four Power Amplifiers {AMP}. Two Thyristor Firing Modules control the firing (on) times of the Power Amplifier thyristors (SCRs) {SCR}, and by varying (delaying) the point in the AC sine wave at which the SCRs are pulsed, the magnitude of the current flow to the main generator exciter field can be changed. Each of the two Firing Modules has an internal +/-15 VDC power supply {JX} which is energized by the PMG. The 120 VAC, 420 Hz, 3-phase supply from the PMG is stepped down to 90 VAC inside the Firing Module and is rectified to a DC voltage. The +/-15 VDC voltage is used to power all of the other circuit cards inside the Firing Module. Once the main generator spinning at its rated 1800 revolutions per minute (RPM), the power supply output is always 15 VDC, regardless of generator load.

Two controls are used to transmit and adjust the main generator output terminal voltage demand signals to the Thyristor Firing Modules and Power Amplifiers. These are Base Adjust and Voltage Adjust. The Base Adjuster {EC} is used to generate the signal that determines a base or fixed value for excitation and serves as the means of adjusting voltage and reactive load in the manual mode. The Voltage Adjuster {EC} is used to adjust and, in conjunction with the Automatic Voltage Regulator (AVR) {90}, maintains a predetermined generator output voltage or reactive load. When the AVR is off, only the signal from the Base Adjuster is allowed to be applied to the two Thyristor Firing Modules. With the AVR in service, the AVR output is connected to the Firing Modules. The AVR output signal will add to or subtract from the base signal to maintain the desired main generator output terminal voltage as set by the Voltage Adjuster.

Following the November 3, 2017 reactor trip event, the AVR was removed from service and the Thyristor Firing Module B that had failed on August 31, 2017 was inspected. A visual inspection and subsequent bench testing revealed that internal components of the failed Thyristor Firing Module were damaged. Specifically, one of the phase transformers (T1) used to step down the 120 VAC, 420 Hz, 3-phase supply from the PMG to 90 VAC was damaged and four of the six input diodes (D1, D2, D3, and D5) for the +/-15 VDC power supply were damaged.

The T1 transformer was replaced, along with the other two phase transformers (T2 and T3). The power supply containing the failed diodes was also replaced with a spare power supply. Prior to returning it to service, the repaired Thyristor Firing Module B was tested for functionality. A Failure Modes Analysis (FMA), Fault Tree Analysis, and troubleshooting plan were used to identify and test different segments of the system, as well as the individual subcomponents. Each identified failure mode was systematically eliminated through testing. The AVR system as a whole underwent a full functional test prior to return to service in order to verify its proper operation.

The FMA and Fault Tree Analysis focused on the Main Generator Exciter, PMG, AVR, and their subcomponents for investigation, which collectively work to maintain field excitation in the main generator. The FMA identified a total of 48 potential failure modes, and these were reviewed independently by an Engineering consultant for reasonableness of the basis of each failure mode. Troubleshooting eliminated all but one failure mode, which could neither be eliminated nor confirmed. This potential failure mode describes a condition where the degraded/failed Thyristor Firing Module B caused a failure of the in service/operating parallel redundant Thyristor Firing Module A. This potential failure mode was supported by the investigation and testing of the failed components, which revealed that the power supply internal to Module B did not isolate upon failure and may have comments regarding burden estimate to the Information Services Branch (T-2 F43), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by e-mail to used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

05000-286 Indian Point 3 2017 - 00 - 004

3. LER NUMBER

affected the operation of Module A since the two modules share multiple connection points (i.e , the modules are not electrically isolated). A more detailed discussion of the failure mode follows:

During normal full power operations, the delta connected phase transformers T1, T2, and T3 in each Thyristor Firing Module drawer provide 3-phase 90 VAC at the input to the Firing Module power supply. This 90 VAC is rectified to 90 VDC, and Zener diodes regulate voltage to produce 15 VDC between the +15 VDC and common (COM) rails, and between the COM and -15 VDC rails. The COM rails are floating (i.e., not grounded). The Module A and B power supplies are each provided reverse current protection via blocking diodes on their +15 VDC and -15 VDC rails. Additionally, power supply relay K1 is provided to isolate the +15 VDC and -15 VDC output rails when the input voltage is not adequate. The COM rails do not have blocking diodes and are not isolated by the K1 relay, and as such, the COM rails are not electrically isolated on a power supply failure. This power supply circuit design vulnerability, when combined with the failures of the Module B input diodes and T1 transformer, created a low output impedance condition on the COM rail of the failed Module B power supply.

Because the COM rails are shared, and the COM rail of the failed Module B power supply remained unisolated, the low output impedance condition resulted in a backfeed current path to ground via the Module B drawer chassis and damaged T1 transformer core and winding insulation. The backfeed current placed an additional load on the operating Module A power supply, which caused the voltage on the shared COM rail to degrade. The voltage output of the Module A power supply eventually fell below 15 VDC, which caused the pulse generators to fail to provide the required output pulse train to sustain proper main generator field excitation.

Since this final remaining failure mode could not be proven by testing, the Engineering consultant suggested the implementation of a monitoring plan to allow for additional information processing during and following plant start up. This monitoring plan is currently in place via the installation of a high speed recorder, and has been incorporated in an ODMI.

An extent of condition (EOC) review was conducted to determine where the same or similar conditions may exist, but the adverse impact has not yet occurred. For this event, the EOC was limited to the IP2 and IP3 main generator AVRs where a failure could result in a direct plant trip either by a loss of excitation or by other means. The assessment concluded that the extent of condition at IP3 has a low associated risk due to the extensive troubleshooting and testing that occurred on the Exciter System and AVR following the November 3, 2017 reactor trip event, and since the system was returned to service with no known deficiencies. At IP2, although the AVR is of a different design, the recent operating history of the AVR will be reviewed to ensure that any similar conditions are properly evaluated and addressed.

CAUSE OF EVENT

The direct cause of the loss of main generator field was a failed Thyristor Firing Module drawer which affected proper operation of the redundant Thyristor Firing Module drawer. The loss of the main generator field actuated Generator Protection System Backup Lockout Relay 86BU, which in turn initiated the turbine trip and reactor trip.

The root cause of this event was that the IP3 AVR Firing Module power supplies have a latent design vulnerability where shared common output nodes are not isolated after a failure. The failure of the components in the Thyristor Firing Module B resulted in a degraded output from both Thyristor Firing Modules that caused the main generator exciter field to collapse. Evidence obtained while troubleshooting the cause of this event led to a plausible sequence of events and failure mode determination.

Significant contributing causes to this event are:

1. Inconsistent evaluation of vendor recommendations. This contributed to this event by not properly comments regarding burden estimate to the Information Services Branch (T-2 F43), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by e-mail to used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

- 00 - 004 2017 validating the vendor recommendation that was used during CDP development. This was also a contributor in that the decision not to upgrade the AVR, contrary to vendor recommendations, allowed the design vulnerability to remain in the system, creating an operating experience knowledge gap by continued use of equipment that has reached obsolescence within the industry.

2. Ineffective implementation of the critical decision process. This contributed to this event by removing the opportunity to challenge assumptions and smaller decisions that factored into the options of the final CDP.

The design documents for the AVR did not contain sufficient detail that would have allowed the organization to identify the latent design vulnerability. If the design vulnerability had been recognized in the recommendations from the CDP, the station would have been given the opportunity to make a different decision.

CORRECTIVE ACTIONS

The following corrective actions have been or will be performed under the Entergy CAP to address the causes of this event.

• Following the IP3 reactor trip event, repairs were made to the failed components and the system was returned to service with no known deficiencies.

• A temporary modification was implemented under Engineering Change (EC)-74798 to install monitoring equipment and record certain key I P3 main generator AVR operating parameters. This monitoring equipment will ensure that important system parameters are logged to assist in any troubleshooting which would follow a future failure of the AVR.

• An ODMI for CR-IP2-2017-04158 (Revision 2) is in effect to provide guidance for operation of the AVR with the Generator Exciter monitoring plan. This ODMI requires monitoring of generator parameters for abnormal operation, including the monitoring equipment installed under EC-74798, and provides direction on actions to be taken in the event that one or more generator parameters deviate outside of their normal operating ranges, or if thyristor power supply voltage is unstable.

• As part of the final system functional test, it was verified that a Thyristor Firing Module drawer could be electrically disconnected without creating a significant system disturbance. This will make future troubleshooting and isolation easier in the event of recurrence of this or a similar failure.

• Revise the critical decision process to require justification of assumptions and evaluate decisions regarding equipment issues for inclusion in the risk register.

• A plant modification is proposed to eliminate the latent design vulnerability condition by electrically isolating the AVR Firing Module power supplies upon a failure, which was the intended original system design.

• Review the recent operating history of the IP2 AVR to ensure that any similar conditions are properly evaluated and addressed.

EVENT ANALYSIS

The event is reportable under 10 CFR 50.73(a)(2)(iv)(A). The licensee shall report any event or condition that resulted in manual or automatic actuation of any of the systems listed under 10 CFR 50.73(a)(2)(iv)(B). Systems to which the requirements of 10 CFR 50.73(a)(2)(iv)(A) apply for this event include the Reactor Protection System (RPS) {JC}, including reactor trip, and AFWS actuation. This event meets the reporting criteria because an automatic reactor trip was initiated on November 3, 2017 at 2022 hours0.0234 days <br />0.562 hours <br />0.00334 weeks <br />7.69371e-4 months <br /> and the AFWS was automatically actuated on a valid low SG water level signal due to shrink effect.

Indian Point 3 05000-286 comments regarding burden estimate to the Information Services Branch (T-2 F43), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by e-mail to used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

Indian Point 3 05000-286 - 00 PAST SIMILAR. EVENTS A review was performed of the past five years for IP2 and IP3 Licensee Event Reports (LERs) that reported a reactor trip resulting from a failure of the Main Generator Excitation System, including the exciter and voltage regulator. The review identified IP2 LER 2012-006 that reported an automatic reactor trip which occurred on June 6, 2012. The reactor trip resulted from a turbine trip due to a trip of Backup Generator Lockout Relay 86BU on loss of main generator field excitation. Investigation determined the 86BU relay actuation was triggered by relay 62BU1/AUX {RLY, 62}, which serves as a time delay for the KLF-40 loss of field relay {RLY, 40}. The actuation of the loss of field relay was in response to a loss of generator field excitation from the Generrex voltage regulator system {TL, EXC}. The Generrex is the IP2 main generator exciter regulating system manufactured by the General Electric Company {G080}.

The direct cause of the reactor trip was loss of generator field excitation due to failure of the Generrex C-Phase Trigger Generation Card. The root cause was indeterminate, but was most likely due to premature failure of the U5 operational amplifier (op-amp) on the C-Phase Trigger Generation Card causing the U3 and U6 op-amps to also degrade. Corrective actions included replacement of the C-Phase Trigger Generator and AC/DC Gate cards with new cards which were then calibrated and monitored for proper operation. The failed C-Phase Trigger Generator card was shipped to a vendor for an equipment failure analysis, and it was later confirmed that the card failed due to failure of the op-amps.

SAFETY SIGNIFICANCE

This event had no effect on the health and safety of the public. There were no actual safety consequences for the event because it was an uncomplicated automatic reactor trip with no other transients or accidents, and the required primary safety systems performed as designed. The AFWS actuated and provided required feedwater flow to the SGs. The AFWS actuation was an expected reaction to the low SG water level caused by SG void fraction (shrink).

This occurs after a reactor trip due to main steam {SB} back pressure that results from the rapid reduction of steam flow following turbine control valve {TA, FCV} closure. A reactor trip with the reduction in SG level and AFWS actuation are conditions for which the plant is analyzed. This event was bounded by the analyzed event described in IP3 Updated Final Safety Analysis Report (UFSAR) Section 14.1.9, Loss of Normal Feedwater. The AFWS has adequate redundancy to provide the minimum required flow assuming a single failure. The UFSAR analysis demonstrates that the AFWS is capable of removing the stored and residual heat plus reactor coolant pump {P} waste heat following a loss of normal feedwater event, thereby preventing over-pressurization of the Reactor Coolant System (RCS) {AB} and preserving reactor coolant inventory.

The analysis in UFSAR Section 14.1.8, Loss of External Electrical Load, concludes that an immediate reactor trip on a turbine trip is not required for reactor protection. A reactor trip on a turbine trip is provided to anticipate probable plant transients and to avoid the resulting thermal transient. If the reactor {AC} is not tripped by a turbine trip, the over temperature delta temperature (OTDT) or over pressure delta temperature (OPDT) trip would prevent safety limits from being exceeded. This event was bounded by the analyzed event described in UFSAR Section 14.1.8.

The response of the plant is evaluated for a complete loss of steam load or a turbine trip from full power without a direct reactor trip. The analysis shows that the plant design is such that there would be no challenge to the integrity of the RCS or main steam system {SB} and no core safety limit would be violated.

For this event, all control rods inserted as required upon initiation of the reactor trip. The RCS pressure remained below the setpoint for pressurizer power operated relief valve (PORV) {AB, RV} and code safety valve {AB, RV} operation, and above the setpoint for automatic safety injection actuation. Following the reactor trip, the plant was stabilized in hot standby with decay heat being removed by the main condenser.

2017 - 004