05000483/LER-2015-004, Auxiliary Feedwater Control Valve Inoperable Due To Faulty Electronic Positioner Card
| Callaway Plant Unit 1 | |
| Event date: | 08-11-2015 |
|---|---|
| Report date: | 10-12-2015 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition |
| Initial Reporting | |
| ENS 51308 | 10 CFR 50.72(b)(2)(iv)(B), RPS System Actuation, 10 CFR 50.72(b)(3)(iv)(A), System Actuation |
| 4832015004R00 - NRC Website | |
1. DESCRIPTION OF STRUCTURE(S), SYSTEM(S) AND COMPONENT(S):
The auxiliary feedwater (AFW) system [EIIS system identifier BA] automatically supplies feedwater to the steam generators to remove decay heat from the reactor coolant system upon the loss of the main feedwater (MFW) [EIIS system identifier Si] supply. The motor-driven AFW pumps (MDAFPs) [EIIS component identifier P] start automatically upon low-low steam generator water level in any steam generator, upon trip of both turbine-driven MFPs, upon actuation of Anticipated Transient Without Scram Mitigation System Actuation Circuitry (AMSAC), or upon actuation by the Loss of Coolant Accident (LOCA) sequencer or shutdown sequencer. The turbine-driven AFW pump (TDAFP) is automatically started by steam generator water level low-low in any two steam generators, undervoltage on either of the 4160V safety- related busses, or upon actuation of AMSAC. All three AFW trains can also be manually actuated.
The MDAFPs (PALO1A and PALO1B) supply flow to the steam generators through a normally open motor-operated flow control valve [EIIS component identifier FCV] that automatically throttles flow. A downstream flow orifice prevents pump run-out conditions under all steam generator pressure conditions. One MDAFP at full flow is sufficient to remove decay heat and cool the unit to residual heat removal (RHR) entry conditions for a normal plant shutdown. The AFW system design is such that it can perform its function following a feedwater line break (FLB) between the MFW isolation valves and the reactor containment building, combined with a loss of offsite power following turbine trip, and a single active failure of the TDAFP. This results in the minimum assumed flow to the intact steam generators. One MDAFP is not, by itself, capable of delivering 100% of the AFW flow assumed in the FLB safety analysis of record in FSAR Section 15.2.8.
One MDAFP would deliver to the broken MFW header at a flow rate throttled by the motor-operated flow control discharge valve until the problem was detected, and flow terminated by the control room operators. Sufficient flow would be delivered to the intact steam generator by the throttled flow from the motor-driven AFW pump feeding the affected steam generator (SG) plus the equally split flow to two intact SGs from the other MDAFP.
In order for the MDAFPs to be operable during normal power operation while the AFW system is in automatic control or above 10% RTP, the applicable discharge flow control valves (ALHV0005 and ALHV0007 for PALO1B; ALHV0009 and ALHVO011 for PALO1A) must be operable and verified to be in the fully open position. Upon actuation of the AFW system, these flow control valves either throttle or isolate flow from the 'A' and '13' MDAFPs to their respective SGs, depending on the accident scenario and the level of TDAFP flow available. During a postulated design basis accident (DBA) scenario where SG pressure is initially low, then increases, the AFW flow control valves would actuate to a more open position as SG pressure increases to maintain a constant rate of AFW flow. However, this additional actuation of the AFW flow control valves to a more open position, following their initial actuation to throttle or isolate, is not credited in any accident analysis. As documented in Calculation AL-29, Rev. 003B, "Auxiliary Feedwater System Performance During a Feedline Break," increases in SG pressure are assumed to result in decreased AFW flow. Additional actuation of the AFW flow control valves to a more open position to maintain a constant AFW flow rate is not credited.
Additional throttling of the AFW flow control valves is required following a DBA during the subsequent plant cooldown to RHR entry conditions. The AFW flow control valves are not credited to prevent pump run-out conditions for the MDAFPs. A flow orifice downstream of each AFW flow control valve is credited for AFW pump run-out protection.
Failure of the TDAFP after the flow control valves have already throttled, or isolated, such that they would then be required to reopen is not a postulated scenario in the accident analysis. Per the Callaway licensing basis described in FSAR Sections 3.1.1.2 and 15.2.8, failure of the TDAFP is only postulated to occur as a single failure assumption upon initial demand, and as such, the MDAFP flow control valves will remain open to provide the required flow from the MDAFPs to the SGs.
The MDAFPs also have a normal operation function and remain operable with the applicable discharge flow control valves throttled by operator action from the control room to maintain steam generator levels during plant heatup, cooldown, or if started due to an Auxiliary Feedwater Actuation Signal (AFAS) or manually started in anticipation of an AFAS. Therefore, correct valve positions in these plant states below 10% RTP vary depending upon plant conditions and are verified per Technical Specification (TS) Surveillance Requirement (SR) 3.7.5.1. SR 3.7.5.3 verifies that AFW can be delivered to the appropriate steam generator in the event of any accident or transient by demonstrating that each automatic valve in the flow path actuates to its correct position on an actual or simulated actuation signal. This SR includes the requirement to verify that each MDAFP motor-operated discharge flow control valve (ALHV0005 and ALHV0007 for PALO I B, and ALHV0009 and ALHVO011 for PALO1A) limits the flow from the associated MDAFP to its respective steam generator to less than or equal to 300 gpm. Each MDAFP flow control discharge valve is required to be operable per TS 3.7.5 for MODES 1, 2, and 3.
ENERGY INDUSTRY IDENTIFICATION SYSTEM COMPONENT AND SYSTEM INFORMATION The Energy Industry Identification System (EllS) component and system identifiers for the components described herein are as follows:
System: BA, Auxiliary Feedwater System Components: P, Pump; FCV, Flow Control Valve System: SJ, Main Feedwater System
2. INITIAL PLANT CONDITIONS:
Prior to ALHV0007 failing to open from the main control board (MCB), Callaway Plant was stable in MODE 1, with 1248 MWe output. At 01:39 on August 11, 2015, an unexpected turbine trip and reactor trip occurred in response to an offsite transmission line fault. As a result of these trips, an automatic actuation of the auxiliary feedwater system occurred. When control room operators attempted to take manual control of the 'B' train motor-driven auxiliary feedwater pump (MDAFP) discharge control valves (subsequent to acceptable operation of the valve, initially, in response to the automatic AFW actuation), they were unable to reopen valve ALHV0007 using the main control board control station. The extent of condition and the full ramifications for ALHV0005 were discovered while the plant was in MODE 3 (Hot Standby) at normal operating temperature and pressure.
3. EVENT DESCRIPTION:
Prior to the event, Callaway Plant was stable in MODE 1 with 1248 MWe output. At 01:39 on 8/11/2015, an unexpected turbine trip and reactor trip occurred due to an offsite transmission line fault on the Montgomery-Callaway 8 switchyard line.
An automatic Auxiliary Feedwater Actuation Signal (AFAS) was generated. The turbine-driven auxiliary feedwater pump (TDAFP) and both motor-driven auxiliary feedwater pumps (MDAFPs) started, with the TDAFP primarily satisfying the water level requirements of the steam generators. All four flow control valves for the two MDAFPs, including ALHV0005 and ALHV0007, closed as a result of the flow supplied by the TDAFP. In accordance with the normal operation procedure, manual operator action was later initiated to transfer flow to the MDAFPs in order to secure the TDAFP. Steam generator (SG) levels were restored to within the control band.
In preparation for securing the TDAFP, a control room operator was unable to reopen ALHV0007, MDAFP 'B' to SG 'A,' using the main control board (MCB) control station at 04:56 on 8/11/2015. An Operating Technician (OT) was dispatched locally to the valve and found it fully closed on its seat. The OT partially opened the valve using the handwheel; however, the control room operator was still unable to regain remote electrical control of the valve and the valve was declared inoperable. The valve was then locally closed with the handwheel and left closed. Local manual control of ALHV0007 was maintained. After valve ALHV0007 was declared inoperable, Job 15003479 was initiated to troubleshoot. Upon preliminary investigation, electricians observed that the electronic positioner card was charred. The electronic positioner card was then replaced with a "hybrid" electronic positioner card (explained later in this Event Description) and valve control was restored. Further investigation of the failed electronic positioner card determined that the BR2 bridge rectifier had burned completely off of the positioner's circuit card.
In 2014, the Limitorque Modutronic Model 10A electronic positioners for the four MDAFP discharge control valves, as made by the original equipment manufacturer (OEM), were reverse-engineered for Callaway by a vendor, Nuclear Logistics, Inc. (NLI), due to obsolescence issues. The control cards in the `13' train MDAFW flow control discharge valves (ALHV0005 and ALHV0007) were replaced during Refuel 20 (October 2014) with the new reverse-engineered circuit cards, referred to by make and model number as NLI-10-LM-P Revision 0 cards. The 'A' MDAFW train flow control valves were not modified during Refuel 20.
On 12/3/2014, a plant trip occurred due to a failure of main generation excitation transformer XMB01. An AFAS was generated and all three AFW pumps started. In this event, flow control valve ALHV0005, MDAFP '13' to SG 'JD,' did not move from its initial open position upon receipt of the AFAS and the '13' MDAFW train was declared inoperable at 00:35 on 12/3/2014. The other three MDAFP flow control valves operated as expected. A failed BR2 bridge rectifier was found on the positioner card (installed during Refuel 20), and the failed card was replaced with an identical NLI-10-LM-P Revision 0 positioner card. No other damage to valve or operator components was found. The '13 MDAFW train was declared operable at 21:56 on 12/3/2014. This failure was initially determined to be due to infant mortality based on Callaway's review of the rectifier data sheet and subsequent failure analysis by NLI.
Subsequent to the 12/3/2014 event, both ALHV0005 and ALHV0007 performed normally during routine surveillance testing until the failure of ALHV0007 occurred on 8/11/2015 during the turbine trip and reactor trip associated with an offsite transmission line fault. Valve ALHV0007 closed during the initial transient because all three auxiliary feedwater pumps had started.
In response to the December 2014 failure of ALHV0005, NLI revised their circuit card design to increase the rating of the BR2 bridge rectifier from 1.5 amps to 4.0 amps. On 4/21/2015, NLI provided a failure report noting the bridge rectifier design change; however, they did not identify the rectifier as undersized. They noted an excessive-current failure due to heat buildup, with potential causes cited as locked rotor, F 1 terminal to F2 terminal short, or misapplication of external voltage.
NLI did not refer to any inadequate sizing issues. Subsequent manufacturing of this positioner will be to the new NLI-10- LM-P Revision I circuit card design. However, to meet Callaway's immediate needs, the vendor refurbished several NLI- 10A-LM-P Rev. 0 positioner cards and replaced the 1.5 A BR2 bridge rectifiers with 4.0 A rectifiers, introducing what is referred to as a "hybrid" NLI-10A-LM-P Revision 0 positioner card.
The electronic positioner cards for both ALHV0005 and ALHV0007 were replaced with two of these "hybrid" positioner cards prior to returning the plant online following the 08/11/2015 trip. As of the date of this LER, the 'A' MDAFW train flow control valves (ALHV0009 and ALHVO011) still have the OEM positioner cards installed (i.e., not the ones reverse engineered by NLI). Callaway has one "hybrid" positioner available in stock, but has not purchased any of the NLI-10A- LM-P Rev. 1 positioner cards (which have been redesigned to remove the design deficiency) to date. All remaining stock of the NLI Rev. 0 electronic positioner cards (with the 1.5A bridge rectifiers) have been removed from storeroom stock and quarantined.
During the period of time between 11/18/2014 (MODE 3 entry at 02:32 after Refuel 20) and 12/3/2014 when the 'B' MDAFW train was inoperable for the mitigation of the limiting Design Basis Accident (DBA) for AFW (i.e., a feedwater line break, FLB) due to an improperly functioning control card installed on ALHV0005, the redundant AFW trains were inoperable as follows:
- 'A' MDAFW train was inoperable, but available, on 12/1/2014 (00:59 - 02:07, available with a dedicated Maintenance Rule operator);
- 'A' MDAFW train was inoperable and unavailable on 12/1/2014 (07:37 - 08:13) and on 12/1/2014 (11:36 - 12:25).
- TDAFW train was inoperable, but available, from 11/18/2014 (02:32 MODE 3 entry) through 11/20/2014 (02:30) except that TDAFW train was non-functional on 11/19/2014 at 08:37 for tagging; and
- TDAFW train was inoperable and unavailable on 11/22/2014 (16:44 - 20:06) and 12/1/2014 (13:58 - 14:08).
4. ASSESSMENT OF SAFETY CONSEQUENCES:
Impact During Plant MODE 1 above 10% RTP During normal operation above 10% RTP, the MDAFP flow control valves are required by TS 3.7.5, "Auxiliary Feedwater System," to be in the fully open position in order for the corresponding train of AFW to be operable. Upon actuation of the AFW system, the MDAFP flow control valves are credited to either throttle or isolate flow from the 'A' and 'B' MDAFPs to their respective SGs, depending on the accident scenario and the level of TDAFP flow available. No accident analysis sequence requires that the MDAFP flow control valves reopen, after closing from their initial fully open position above 10% RTP, in order for the AFW system to fulfill any of its specified safety functions.
In the case of the ALHV0005 failure on 12/3/2014, the valve was unable to throttle or isolate flow through automatic flow- limiting operation or remote manual operation. As a result, from the time ALHV0005 was left energized on 11/18/2014 until the failed NLI-10A-LM-P Rev. 0 positioner card was replaced on 12/3/2014, the 'B' MDAFW train would not have been able to fulfill all of its specified safety functions above 10')/0 RTP for the FLB analysis where automatic flow-limiting operation to the faulted SG is required.
After the ALHV0005 failure on 12/3/2014, the NLI-IOA-LM-P Rev. 0 positioner card was replaced with an identical NLI- 10A-LM-P Rev. 0 positioner card with the same degraded condition associated with the undersized bridge rectifier.
However, ALHV0005 then operated normally with the degraded condition present until 8/12/2015 when the NLI-10A-LM-P Rev. 0 positioner cards on both ALHV0005 and ALHV0007 were replaced with the NLI-10A-LM-P "hybrid" positioner cards. After the ALHV0005 NLI-IOA-LM-P Rev. 0 positioner card was replaced on 8/12/2015, the removed card was tested by energizing it in a controlled environment simulating the most limiting DBA conditions to determine how much additional, energized operation time the NLI-10A-LM-P Rev. 0 positioner card could have sustained before failure. The previously installed NLI-10A-LM-P Rev. 0 positioner card, before being replaced on 8/12/2015, could have continued operating for a time period greater than the required AFW system mission time. In addition, both ALHV0005 and ALHV0007 performed as expected during a controlled plant shutdown in response to Technical Specification 3.4.13 on excessive RCS leakage (reported in LER 2015-001-00 and Event Notification 51253) and the manual AFAS that subsequently occurred on 7/23/2015 (reported in LER 2015-002-00). This gives confidence that ALHV0005 would have been able to throttle or isolate at all times such that the ability of the ALHV0005 to perform its specified safety functions in MODE 1 above 10% RTP would not have been challenged by the degraded condition present from 12/3/2014 to 8/12/2015.
In the case of the ALHV0007 failure on 8/11/2015, the valve successfully demonstrated the ability to throttle or isolate flow.
Therefore, the ability of ALHV0007 to perform its specified safety functions in MODE 1 above 10% RTP would not have been challenged by the degraded condition that existed for ALHV0007 since Refuel 20.
Impact During Plant MODE 1 below 10% RTP, and MODES 2 and 3 When placing the AFW system in service as part of a controlled plant cooldown or heat up, the MDAFP flow control valves are remotely manually closed prior to starting the 'A' and 'B' MDAFPs, as allowed by TS Surveillance Requirement 3.7.5.1.
During this evolution, core decay heat decreases to the point that short-term, automatic AFW flow to the SGs is not required to maintain adequate SG level. As discussed in the 5/23/1997 NRC letter attached to Technical Specification Task Force Traveler TSTF-245-A Revision 1, it is recognized that the AFW system "may be used during startup of the plant, normal shutdown, and hot standby conditions and that it is controlled [sic] band operated during these conditions in the manual mode of operation. In such situations, the AFW system is considered OPERABLE with regards to the limiting condition for operation and the TS definitions of OPERABLE/OPERABILITY." This provision is consistent with the Note associated with SR 3.5.7.1, as amplified by the associated TS Bases.
The hypothetical initiation of an FLB, coupled with a limiting single failure of the TDAFP while ALHV0007 was in a manually closed position below 10% RTP and in the described degraded condition, would have required ALHV0007 to be opened manually to meet the AFW system specified safety function of providing the required flow to the SGs during the accident scenario. This would have been accomplished through local manual operator action to open the valve using the actuator handwheel. The automatic flow-limiting circuit demand signal would not have opened ALHV0007 due to the failure of the NLI-10A-LM-P Rev. 0 positioner card.
The most limiting scenario for MODES 1-3 under 10% RTP, while manual control of the MDAFP valves is allowed, is a design basis accident occurring immediately after plant shutdown when core decay heat is at its peak. In that situation, if the MDAFP flow control valves had already been remotely manually closed, operators would have over 45 minutes to open ALHV0007 before reaching 10% level in the 'A' SG. A water level of ten (10) % is the minimum SG level that will prevent thermal shock to the SG internals after re-establishing flow. The value of 45 minutes was conservatively calculated by assuming feedwater to all four SGs is simultaneously unavailable. The amount of time available before reaching 10% SG level increases rapidly as the postulated time of the event after reactor shutdown increases. The normal operation guidance in procedure OTN-AL-00001 for manually starting the AFW pumps directs the operator to "OPEN discharge control valves as needed.
Similarly, for accident conditions, the Emergency Operating Procedure (EOP) guidance in E-0, "Reactor Trip or Safety Injection," for ensuring proper AFW valve alignment directs the operator to "align AFW valves as necessary." EOP ES-0.1, "Reactor Trip Response," can also be entered from E-0. Step 6 of ES-0.1 checks SG feed flow status. If total feedwater flow is necessary using AFW." Similarly, EOP FR-H.1, "Response to Loss of Secondary Heat Sink," directs the operator to "CHECK AFW valve alignment" if AFW flow has not been established. Field training for operation of the MDAFW flow control valves is covered under qualification standard, EOS-SAE-18, to manually isolate feedwater to a faulted steam generator in accordance with EOP E-2, on the Secondary Watchstation qualification card.
Continued manual local operator use of ALHV0007 to gradually throttle flow using the valve actuator handwheel would also be needed during the following plant cooldown to RHR entry conditions. Local manual operator action to control ALHV0007 in this scenario is a reliable and credible operator action that can be completed within 45 minutes. However, based on the FLB analysis sensitivity cases described below, one MDAFW pump feeding two intact SGs is sufficient to meet the acceptance criteria of an FLB event. In view of this "best estimate analysis" from Westinghouse, it would not have been necessary for ALHV0007 to throttle or isolate flow at any time in order for the AFW system to successfully mitigate a DBA.
Based on the above, the ability of the 'EC MDAFW train to perform its specified safety functions in MODES 1, 2, and 3 below 10% RTP would not have been challenged by the degraded condition of ALHV0007. In addition, due to the data obtained from testing performed on the ALHV0005 NLI-10A-LM-P Rev. 0 positioner card that was replaced on 8/12/2015, the ability of the AFW system to perform its specified safety functions would not have been challenged by the degraded condition present of ALHV0005 from 12/3/2014 to 8/12/2015. It has already been established that the AFW system would not have been able to perform all of its specified safety functions from 11/18/14 to 12/3/14 due to the degraded condition for ALHV0005 prior to the replacement of the NLI-10A-LM-P Rev. 0 positioner card on 12/3/2014.
Crediting of Local Manual Operator Action FSAR Section 10.4.9.1.1 notes that the AFW system has the capacity to be operated locally as an alternate, redundant means of feedwater control, in the unlikely event that the control room must be evacuated.
The guidance of NRC Information Notice 97-78, "Crediting of Operator Actions in Place of Automatic Actions and Modifications of Operator Actions, Including Response Times," was used to evaluate the acceptability of the use of local manual actions below 10% RTP during a DBA. Information Notice 97-78 provides a list of nine items to be addressed for manual operator actions. In addition to the procedural guidance and training discussed above, the other seven items were addressed in the past operability evaluation for this event. In the manual AFAS event that occurred on 7/23/2015 (reported in LER 2015-002-00), local manual operator action was successfully used to actuate ALHVO011 following procedural guidance when it failed to remote manually open. During that event, local manual action to open ALHVO011 was completed within 8.5 minutes.
Feedwater Line Break (FLB) Analysis Sensitivity Cases Using "best estimate analysis" assumptions in a post-event analysis performed by Westinghouse for the conditions present during plant operation between 11/18/2014 and 12/3/2014, one MDAFW train would have been able to mitigate any event requiring decay heat removal until RHR operation. The limiting DBA (FLB) for AFW operation would have been mitigated by a single train of MDAFW feeding two intact steam generators. If certain key analysis inputs are set to their nominal values, the conservatisms built into the current analysis of record (AOR) for the FLB accident in FSAR Section 15.2.8 (with three SGs receiving MDAFW flow) render the AOR more limiting with respect to the analysis acceptance criteria than the situation under consideration where, due to the failed open position of flow control valve ALHV0005 and the postulated failure of the TDAFP, only 2 SGs receive MDAFW flow.
This reanalysis includes the use of a low-low SG water level reactor trip setpoint of 5% of narrow range span (NRS). The nominal SG low-low setpoint (field bistable settings) for an adverse containment environment is 21% of NRS. The nominal setpoint (21% of NRS) accounts for the worst case Channel Statistical Allowance (CSA) of 19.8% of NRS and also contains setpoint margin of 1.2% of NRS. The AOR uses a safety analysis limit (SAL) of 0% of NRS to account for these worst case channel errors. The CSA is calculated using multiple factors, including sensor and rack drift terms and environmental allowance (EA) biases. The amount of drift that had occurred after calibration during Refuel 20 until 12/3/2014 was essentially zero. The worst case EA biases were also found to be very conservative. By using more realistic numbers from accident models, the EA biases were reduced significantly. These adjustments brought the CSA from 19.8% of NRS to 10.8% of NRS.
Acceptable FLB analysis results were obtained from Westinghouse (i.e., no hot leg saturation) with two steam generators receiving MDAFW flow and with the steam generators at 5% of NRS when the reactor trips, which is below the 10.2% of NRS analysis limit obtained with a reduction of drift terms and EA biases. Two new cases have been analyzed, with and without offsite power. The assumed reactor trip setpoint, combined with a better-estimate decay heat model, with a better- estimate main steam safety valve model (which allows earlier secondary-side pressure relief), and with a reduced feedwater line fill time (as this volume is assumed to be emptied due to the break and needs to be refilled with AFW prior to any additional inventory reaching the SGs), coupled with the assumption that only 2 SGs receive MDAFW (as opposed to 3 in the AOR), yield an FLB transient that demonstrates hot leg saturation is precluded. Furthermore, these modeling changes result in accident consequences that are bounded by those of the current AOR for the plant (i.e., more margin to hot leg saturation is seen for these two new cases).
The degraded condition, determined to be a vendor design deficiency, was such that the bridge rectifiers for the valve operator motor fields for 'B' train valves ALHV0005 and ALHV0007 could have and did fail, preventing further electrical operation of the valves from the main control board. The failure of ALHV0007 resulted in an inability to remotely feed the 'A' steam generator from the 'B' MDAFP. However, the ALHV0007 valve could still be manually manipulated locally using the valve handwheel. During the 8/11/2015 plant event, operators chose to continue feeding the 'A' steam generator using the TDAFP via its discharge valve, ALHV0008, TDAFP to S/G 'A.' AFW performance requirements during an FLB bound the mitigation function requirements performed by the AFW system for a Main Steam Line Break, Loss of Non-Emergency AC Power, Loss of Normal Feedwater, SG Tube Rupture, and small break LOCA.
Risk Impact This event was also evaluated with the Callaway PRA model. The evaluation determined the incremental conditional core damage probability (ICCDP) of this event was less than 1E-06; therefore, this event was of very low risk significance. Use of the PRA model to evaluate the event provides a comprehensive, quantitative assessment of the potential safety consequences and implications of the event, including consideration of alternative conditions beyond those analyzed in the FSAR.
5. REPORTING REQUIREMENTS:
This LER is submitted pursuant to 10 CFR 50.73(a)(2)(i)(B) to report a condition prohibited by the Technical Specifications and pursuant to 10 CFR 50.73(a)(2)(v)(D) to report a condition that could have prevented the fulfillment of the safety function of structures or systems that are needed to mitigate the consequences of an accident. The 'B' MDAFW train was inoperable for 15 days which exceeded the TS 3.7.5 Condition C Completion Time of 72 hours. During the 15-day 'B' train inoperability there were times when the required AFW flow for the licensing basis FLB analysis was not available. The event reported herein did not involve an unanalyzed condition that significantly degraded plant safety.
With respect to 10 CFR 50.73(a)(2)(v)(D), i.e., a condition that could have prevented the fulfillment of the safety function of structures or systems that are needed to mitigate the consequences of an accident, Section 3.2.7 of NUREG-1022 Revision 3 allows the use of engineering judgment to determine whether a single train of a system (like AFW) could have performed the system's accident mitigation function for FSAR Chapter 6 and 15 events. The following guidance highlights this consideration:
"As a result, for SSCs within the scope of this criterion, a report is required when 1) there is a determination that the SSC is inoperable in a required mode or other specified condition in the TS Applicability, 2) the inoperability is due to one or more personnel errors, including procedure violations; equipment failures; inadequate maintenance; or design, analysis, fabrication, equipment qualification, construction, or procedural deficiencies, and 3) no redundant equipment in the same system was operable.
"For systems that include three or more trains, the inoperability of two or more trains should be reported if, in the judgment of the licensee, the remaining operable trains could not mitigate the consequences of an accident.
"Single, independent (i.e., random) component failures are not reportable if the redundant component in the same system did or would have fulfilled the safety function.
During the period of time between 11/18/2014 (MODE 3 entry after Refuel 20) and 12/3/2014 when the 'B' MDAFW train was inoperable for the mitigation of an FLB due to an improperly functioning control card installed on ALHV0005, the redundant AFW trains were inoperable as follows:
- 'A' MDAFW train was inoperable, but available, on 12/1/2014 (00:59 - 02:07, available with a dedicated Maintenance Rule operator);
- 'A' MDAFW train was inoperable and unavailable on 12/1/2014 (07:37 - 08:13) and on 12/1/2014 (11:36 - 12:25).
- TDAFW train was inoperable, but available, from 11/18/2014 (02:32 MODE 3 entry) through 11/20/2014 (02:30) except that the TDAFW train was non-functional on 11/19/2014 at 08:37 for tagging; and
- TDAFW train was inoperable and unavailable on 11/22/2014 (16:44 - 20:06) and 12/1/2014 (13:58 - 14:08).
During these short windows of TDAFW train unavailability, only a single MDAFW train was operable. Since two MDAFW trains or one TDAFW train are required to mitigate an FLB per the FSAR-described safety analysis, this event is reported pursuant to this criterion as a loss of safety function.
With respect to 10 CFR 50.73(a)(2)(ii)(B), i.e., an unanalyzed condition that significantly degraded plant safety, the identified condition was determined to not be reportable per this criterion.
NUREG-1022 Section 3.2.4 Revision 3 provides the following reporting guidance:
"The level of significance of these cases generally corresponds to the inability to perform a required safety function.
For instance, accumulation of voids that could inhibit the ability to adequately remove heat from the reactor core, particularly under natural circulation conditions, has an effect similar to a condition that could prevent the fulfillment of the safety function of the AFW system.
Beyond the examples given in 1983, an example of an event reportable as an unanalyzed condition that significantly degraded plant safety would be the discovery that a system required to meet the single failure criterion does not do so.
In another example, if fire barriers are found to be missing, such that the required degree of separation for redundant safe shutdown trains is lacking, the event would be reportable as an unanalyzed condition that significantly degraded plant safety. On the other hand, if a fire wrap, to which the licensee has committed, is missing from a safe shutdown train but another safe shutdown train is available in a different fire area, protected such that the required separation for safe shutdown trains is still provided, the event would not be reportable.
As discussed above under the section on Feedwater Line Break (FLB) Analysis Sensitivity Cases, using "best estimate analysis" assumptions in a post-event analysis performed by Westinghouse for the conditions present between 11/18/2014 and 12/3/2014, one AFW train would have been able to mitigate any event requiring decay heat removal until RHR operation. The limiting DBA (i.e., an FLB) for AFW operation would have been mitigated by a single train of AFW feeding two steam generators. Therefore, the event reported herein did not involve an unanalyzed condition that significantly degraded plant safety.
6. CAUSE OF THE EVENT:
Following the AFAS on 12/3/2014, all AFW pumps started and three of the four MDAFP flow control valves closed, as expected, due to the flow from the TDAFP. ALHV0005 failed to close due to the damaged bridge rectifier on the NLI-10A- LM-P Rev. 0 positioner card. Prior to that failure, the 'B' MDAFP had been most recently run on 11/18/2014 during Refuel 20 and both ALHV0005 and ALHV0007 operated satisfactorily. Following that surveillance, ALHV0007 was left fully open and de-energized. However, following Surveillance Job 13506361, ALHV0005 was left energized because the ZS/3 limit switch contacts did not open, as designed, with the valve in the fully open position,. The ZS/3 limit switch contacts should open and de-energize the valve when the valve is in the fully open position. Valve energization was evident by movement of ALHV0005 during subsequent TDAFP runs on 11/19/2014 and 11/20/2014. The extended energization of the bridge rectifier circuit led to accelerating aging and eventual bridge rectifier failure prior to being called upon on 12/3/2014. All three of the other MDAFP flow control valves operated as expected throughout this event.
Following the 12/3/2014 event, the NLI-10A-LM-P Rev. 0 positioner card on ALHV0005 was replaced with an identical card with the same undersized bridge rectifier condition. The issue with ZS/3 limit switch contacts was also resolved during post-maintenance position indication retests on 12/3/2014. After the NLI-10A-LM-P Rev. 0 positioner card replacement on ALHV0005, no further failures of this valve occurred due to the degraded condition. ALHV0005 then operated normally with the degraded condition present until 8/12/2015 when the NLI-10A-LM-P Rev. 0 positioner cards on both ALHV0005 and ALHV0007 were replaced with the NLI-10A-LM-P "hybrid" positioner cards.
The direct cause of the ALHV0005 failure was a failure of a bridge rectifier on the valve's electronic positioner circuit card.
Because the failure of the bridge rectifier is known to be from a design deficiency, ALHV0005 was not operable from 11/18/2014 to 12/3/2014.
In the case of the ALHV0007 failure, the valve successfully demonstrated the ability to throttle or isolate flow on 8/11/2015.
As a result of the AFAS on 8/11/2015, all AFW pumps started automatically. Both ALHV0005 and ALHV0007 stroked closed to control the high flow rate from all three AFW pumps running. The control room operators subsequently attempted to open ALHV0005 and ALHV0007 to secure the TDAFP. ALHV0005 opened, but ALHV0007 did not move due to the damaged positioner card bridge rectifier. ALHV0007 had been energized during the three-hour window following its initial closing such that sufficient degradation had accumulated for the bridge rectifier to fail. This failure to reopen does not call into question the ability of ALHV0007 to stroke closed from its normally open position at any time during or prior to the plant trip on 8/11/2105. Therefore, the ability of the AFW system to perform its specified safety functions in MODE 1 above 10% RTP would not have been challenged by the degraded condition that existed for ALHV0007 since Refuel 20. In addition, all three of the other MDAFP flow control valves operated as expected throughout the event.
The NLI-10A-LM-P Rev. 0 positioner card associated with both of these failures uses a Rectron RB157 bridge rectifier to convert 120-VAC input to a full-wave rectified signal that is applied to the field of the actuator motor. This bridge rectifier is rated for 1.5 amps at 25°C and for lesser currents at higher temperatures (de-rated values). For example, at 60°C the current rating is 1.0 amp and at 100°C the current rating is 0.4 amps. The ambient room temperature, the heat rise in the valve operator compartment, and the heat produced by the bridge rectifier all contribute to the operating temperature of the component and affect its current rating. Assuming a room temperature of 30°C and a valve operator compartment heat rise of 18°C, the operating temperature of the bridge rectifier was at least 48°C. Assuming an operating current of 1.5 amps, the bridge rectifier on the installed NLI-10A-LM-P Rev. 0 card was operated outside of the de-rated current curve.
Operation of the Rectron RB157 bridge rectifier in excess of its de-rated values resulted in accelerated aging of the bridge rectifier each time it was energized. The Rectron RB157 bridge rectifier is comprised of four diodes in a consolidated package. The failure modes, however, are similar to a traditional bridge rectifier constructed from four diodes in a non- consolidated package. The individual diodes are constructed as bi-polar devices with P-N junctions. When operating in the forward region of the P-N junction at excessive currents, the diode will eventually degrade to the point of failure due to short circuiting. At this point, all electrical control of ALHV0005 and ALHV0007 was lost during their respective failures.
Degradation due to excessive current occurs when the bridge rectifier is energized which only occurs during periods when the MDAFP flow control valves are not in the fully open position during valve modulation or stroking. When in the fully open position, the MDAFP flow control valves are designed to be de-energized.
The positioner cards installed in ALHV0005 and ALHV0007 were changed during Refuel 20 (fall 2014) from the Limitorque Modutronic positioner cards to the NLI-10A-LM-P Rev. 0 positioner cards, introducing a degraded condition by using bridge rectifiers with a lower than required current rating for the operating conditions. The failure of ALHV0005 occurred shortly after Refuel 20 on 12/3/2014. The ALHV0007 failure occurred approximately eight months later on 08/11/2015. Both failures were due to cumulative degradation of the NLI-10A-LM-P Rev. 0 positioner cards. The degraded condition on both valves existed from the time the NLI-10A-LM-P Rev. 0 positioner cards were installed during Refuel 20 until the time they were replaced with the NLI-10A-LM-P "hybrid" positioner cards on 8/12/2015.
The root cause of the card failures was determined to be a vendor design deficiency. The reverse-engineered electronic positioner cards were not made to design specifications. Contributing to this, the vendor did not test the rectifier circuit to challenge the circuit's ability to satisfy design requirements, nor did the Callaway design engineer require the vendor to conduct a proof test to verify the design. The responsible Callaway design engineer and qualified reviewer also misunderstood their roles and responsibilities with regard to reviewing the vendor qualification testing document. The electronic positioner card failure on ALHV0007 on 8/11/2015 was a repeat occurrence of the failure of ALHV0005 on 12/3/2014. The design deficiency was not recognized during the evaluation of the previous failure in the corrective action program.
7. CORRECTIVE ACTIONS
New positioner control cards with a higher rated wattage resistor and a higher rated amperage rectifier (4.0A) were installed on 8/11/2015 and 8/12/2015 on ALHV0007 and ALHV0005 under Jobs 15003479.560 and 15002444.500, respectively. The electronic positioner cards were replaced with refurbished "hybrid" electronic positioner cards. The "hybrid" electronic positioner is a more robust design than the NLI Rev.0 electronic positioner. The 1.5A bridge rectifier has been replaced with a 4.0A bridge rectifier. The "hybrid" electronic positioners were not proof tested before installation; however, the bridge rectifier circuit, which is the focus of this report, has since been reproduced and proof tested in an onsite laboratory.
Additional vendor testing also successfully demonstrated the ability of a Revision 0 "hybrid" card to withstand continued operation for 100 hours under full load conditions at 60°C. The valve operator motor field was actually loaded to 2.5 amps even though full load would have been 1.5 amps. These measures provide assurance the installed configuration on ALHV0005 and ALHV0007 meet or exceed the original design.
Through review of the design of the field rectifier it was determined that, by replacing the 1.5A rectifier with a 4.0A rectifier, the electronic positioners have design margin equivalent to the original electronic positioners; therefore, the "hybrid" electronic positioners meet the required specification. Future installations of the reverse-engineered electronic positioner will use the one remaining "hybrid" positioner card in stock, or the new NLI NLI-10A-LM-P Rev. 1 design, which will be manufactured with the 4.0A rectifier. Permanent removal of the NLI Rev. 0 electronic positioners ensures the known degraded bridge rectifier circuits will not be reinstalled in the plant.
The issue with the ZS/3 limit switch contacts was also resolved during post-maintenance position indication retests on 12/3/2014.
8. PREVIOUS SIMILAR EVENTS:
`A' MDAFW train flow control discharge valve to the 'C' steam generator, which required the replacement of a faulty electronic circuit card in the Foxboro rack in the back cabinet area of the main control room (different card) and involved the failure to recognize unsatisfactory test data.