05000440/LER-2007-004

LER-2007-004, Automatic Reactor Protection System Actuation Due to Feedwater Control Power Supply Failure

Docket Number
Event date:	11-28-2007
Report date:	02-15-2008
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(iv)(a)
4402007004R01 - NRC Website

Energy Industry Identification System Codes are identified in the text as [XX].

INTRODUCTION

On November 28, 2007, at 0732 hours0.00847 days <br />0.203 hours <br />0.00121 weeks <br />2.78526e-4 months <br />, the Reactor Protection System (RPS) [JC] automatically actuated in response to a turbine control valve fast closure signal caused by failure of the Digital Feedwater Control System (DFWCS) [JB] power supplies. At the time of the event, the plant was in Mode 1 (i.e., Power Operation) with the reactor operating at 100 percent of rated thermal power (RTP). There was a subsequent loss of feedwater flow to the reactor presture vessel (RPV) [RPV] and reactor water level decreased until the High Pressure Core Spray (HPCS) [BG] system automatically started. The Reactor Core Isolation Cooling (RCIC) [BN] system started, but tripped 13 seconds later. At 0839 hours0.00971 days <br />0.233 hours <br />0.00139 weeks <br />3.192395e-4 months <br />, notification was made to the NRC Operations Center (ENS Number 43808) in accordance with 10CFR50.72(b)(2) (iv)(A), Emergency Core Cooling System (ECCS) discharge into the reactor coolant system, 10CFR50.72(b)(2)(iv)(B), Actuation of the reactor protection system when the reactor is critical, and 10CFR50.72(b)(3)(iv)(A), Valid actuation of several specified systems. This event is being reported in accordance'with 10CFR50.73(a)(2) (iv)(A) as an event or condition that resulted in automatic actuation of RPS, HPCS, RCIC, Division 3 Emergency Diesel Generator (EDG) [EK], Division 3 Emergency Service Water (ESW) [BI], Division 1 ESW, and Containment isolation valves [JM].

Engineering analysis performed subsequent to the RPS actuation revealed that the RCIC system had been inoperable since January 21, 2006. This condition is being reported in accordance with 10CFR50.73(a)(2)(i)(B), any operation prohibited by Technical Specifications (TS).

This report also satisfies the Operational Requirements Manual section 7.6.2.1, which requires a Special Report submittal following an ECCS actuation and injection into the reactor coolant system.

EVENT DESCRIPTION

On November 28, 2007, the plant was operating in Mode. 1, Power Operation at 100 percent RTP.

All EDG and ECCS Systems were operable. The feedwater system was aligned with the reactor feed pump turbines (RFPT) A & B in automatic level control. The motor-driven feedwater pump (MFP) was in standby readiness. The RCIC and HPCS suction flow paths were aligned to the suppression pool to satisfy a TS Limiting Condition for Operation for an inoperable condensate storage tank (CST)level transmitter.

At 0732 hours0.00847 days <br />0.203 hours <br />0.00121 weeks <br />2.78526e-4 months <br />, an RPS actuation occurred resulting in an automatic reactor scram. All control rods fully inserted. The RPS actuation was the result of a Turbine Control Valve Fast Closure signal.

The RPS scram signal occurred as a result of failed power supplies in the DFWCS.

Both RFPTs tripped and the MFP did not start. Approximately 10 seconds later, RPV water level decreased to Level 2, 130 inches above top of active fuel (TAF). Containment isolation occurred with isolation of all required valves. Both Reactor Recirculation [AD] pumps tripped as designed.

The RCIC and HPCS systems started. The Division 3 EDG which supplies emergency electrical power to HPCS also started as designed, but did not load onto the bus. The RCIC system tripped 13 seconds after starting on low RCIC pump suction pressure prior to reaching rated flow. The HPCS system was used to restore and maintain reactor water level in a band of 150 to 215 inches above TAF. The lowest RPV water level reached during the event was 109.5 inches above TAF.

From 0735 to 0740 hours0.00856 days <br />0.206 hours <br />0.00122 weeks <br />2.8157e-4 months <br />, the operators attempted to start the MFP and use it to control RPV water level. The MFP minimum flow valve could not be opened due to loss of the DFWCS power supplies. The operators also attempted to perform a quick start of RFPT A, but were unable to control the turbine on its potentiometer. The attempts were unsuccessful since power was lost to the DFWCS system. RPV water level control remained on the HPCS system by cycling the HPCS injection valve. The HPCS system injected to the RPV nine times over the next 2.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />.

At 0815 hours0.00943 days <br />0.226 hours <br />0.00135 weeks <br />3.101075e-4 months <br />, RCIC suction was realigned from the Suppression Pool to the CST.

The DFWCS primary power supply voltage recovered at 0832 hours0.00963 days <br />0.231 hours <br />0.00138 weeks <br />3.16576e-4 months <br />. This permitted opening of the MFP minimum flow valve and the MFP was restarted. RPV level control was transitioned to the MFP. At 0902 hours0.0104 days <br />0.251 hours <br />0.00149 weeks <br />3.43211e-4 months <br />, the primary power supply degraded further to the point that it caused RPV water level high logic Signals that tripped the MFP. The primary power supply did not recover again.

The operators started the RCIC system manually at approximately 0850 hours0.00984 days <br />0.236 hours <br />0.00141 weeks <br />3.23425e-4 months <br />. Subsequent to RCIC pump start, the system experienced flow variations from 50 to 100 gallons per minute (gpm) and tripped on low pump suction pressure at 0908 hours0.0105 days <br />0.252 hours <br />0.0015 weeks <br />3.45494e-4 months <br />. At 0915 hours0.0106 days <br />0.254 hours <br />0.00151 weeks <br />3.481575e-4 months <br />, the operators re-started the RCIC system with the flow controller in manual control. Discharge flow was directed to the RPV and CST simultaneously. The RCIC system was then used for RPV level and pressure control.

At 1310 hours0.0152 days <br />0.364 hours <br />0.00217 weeks <br />4.98455e-4 months <br />, RPV pressure had decreased to allow the operators to, transition RPV level control from RCIC to the feedwater booster pumps. At 1630 hours0.0189 days <br />0.453 hours <br />0.0027 weeks <br />6.20215e-4 months <br />, the operators placed shutdown cooling in operation. Mode 4, Cold Shutdown was reached at 1715 hours0.0198 days <br />0.476 hours <br />0.00284 weeks <br />6.525575e-4 months <br /> when the RCS temperature was decreased to less than 200 degrees Fahrenheit.

CAUSE OF EVENT

The Turbine Control Valve Fast Closure RPS actuation signal was caused by a failure of both DFWCS power supplies. Failure of the DFWCS power supplies de-energized two feedwater control relays and supplied an invalid Level 8 signal to the main turbine system. The invalid signal caused the turbine control valves to 'fast' close resulting in an RPS actuation signal. The reactor shutdown automatically and was not the result of operator actions.

When the power supplies failed, power was lost to DFWCS relays that, by design, are de-energized by an RPV Level 8 signal, Low Feed Flow, or RPV Level 3 in the Feed Water. Control logic circuits.

There were no actual RPV Level 8, Low Feed Flow or RPV Level 3 conditions prior to the RPS actuation. The MFP and associated feedwater system controls became unavailable due to the loss of DFWCS power.

The secondary DFWCS power supply (1C34K0677) experienced a gross failure. The primary DFWCS power supply (1C34K0676) was in a degraded condition where the output voltage was within acceptable voltage limits but not able to carry a nominal electrical load. The low voltage system alarm and low voltage LEDs on the power supply modules were not illuminated to indicate the presence of a faulted condition. When the secondary DFWCS power supply experienced a gross failure on November 28, 2007, the primary DFWCS power supply was unable to carry the load and all of the input and output modules were disabled causing the RPS actuation.

Failure analysis performed by the FirstEnergy Nuclear Operating Company's Beta Laboratory found that transformer T2 on the 24VDC converter board of both the primary and secondary power supplies contained a defect. A common external fault did not exist that could have caused both power supplies to fail. There were two separate power supply failures that caused the DFWCS malfunction.

Failure of the RCIC system to continue operating as designed was caused by incorrect tuning parameter settings for the RCIC flow controller introduced following maintenance performed on January 21, 2006. A calibrated replacement flow controller was installed. The controller's Rate setting was changed from 0.01 to 9.25 out of a possible range of "N/A" (or 0.01) to 10. The changed tuning parameters made the system more sensitive to noise in the flow signal and increased the level of RCIC control response. This factor resulted in rapid changes in system flow during RCIC startup (i.e., flow oscillations) and in noise oscillations in the RCIC pump suction pressure instrumentation that were sufficient to drop pressure to the trip unit actuation setpoint.

The RCIC system trip on low suction pressure at 0908 hours0.0105 days <br />0.252 hours <br />0.0015 weeks <br />3.45494e-4 months <br /> was caused by complicated flow dynamics presented by the dual discharge to the RPV and CST for RPV level and pressure control implemented by the operators. The dynamics presented a high demand on the RCIC controls when operating with the flow controller in Automatic. The demands resulted in flow variations that ultimately tripped the RCIC pump.

The implementation of incorrect RCIC flow controller tuning parameters was caused by exempting the controller settings from configuration control since they were considered operational adjustments. The governing documents for performing RCIC flow control loop adjustments lacked the requisite configuration control to assure reliable RCIC performance consistent with its design basis. Critical vendor guidance was missing from the RCIC system tuning instruction, specifically, a prohibition against use of Rate setting adjustment for the flow loop application, and unclear acceptance criteria distinction between the flow loop and speed loop components. In 1987, the RCIC controller set points were removed from the Master Setpoint List without establishing adjustment limits in the tuning procedure. In 1999, performance data from start-up testing had been removed from the tuning instruction. The tuning instruction allowed for in field adjustments as necessary and set up the conditions where procedural barriers were, removed and the station became dependent on knowledge and expertise to properly tune the RCIC flow controller.

Knowledge deficiencies in personnel responsible for determining and implementing tuning parameters for the RCIC flow controller played a significant role with interpreting the RCIC test results. The engineering staff lacked the requisite training/experience to define RCIC system acceptance criteria and direct process control tuning activities. The Instrumentation and Control (I&C) staff also lacked the requisite training and experience to define process control tuning set points and process control loop response acceptance criteria. Over time, the station expertise had eroded such that the maintenance activity resulted in unrecognized controller adjustments outside of the expected ranges.

�

EVENT ANALYSIS

Based on the following information, this event is considered to be of very low safety significance.

The reactor protection system functioned as designed. All 177 control rods inserted to the full-in position and within the required scram time. While the scram was successful, the event included a loss of feedwater and failure of the RCIC system to start and run. The HPCS system responded as designed and was used to maintain RPV water level until the RCIC system was restored.

The event, including the plant response, is bounded by evaluation in the plant's Updated Safety Analysis Report (USAR). No plant parameter due to this automatic scram from 100 percent power challenged the transients described in the USAR Chapter 15, Accident Analysis. The specific transients that apply are:

152.3 Turbine Trip 15.2/ Loss of Feedwater Flow 15.3.1 Recirculation Pump Trip The transients are incidents of moderate frequency. Loss of all feedwater flow is classified as an Anticipated Operational Transient, Event 20 in the Perry Nuclear Safety Operational Analysis, Appendix 15A.

A probabilistic risk assessment (PRA) performed for the scram, including loss of the RCIC system and feedwater, calculated the incremental conditional core damage probability (ICCDP) to be 6.5E-.

08. The Incremental Large Early Release Probability (ICLERP) by definition can not be greater than the ICCDP, thus ICLERP is less than 6.5E:08. Transients with a core damage probability less than 1.0E-06 and a large early release probability less than 1.0E-07 are not considered to be significant risk events.

The RCIC system is not an Engineered Safety Feature System. RCIC system operation is credited for several transient's described in the USAR Chapter 15. The avail6bility of the RCIC system contributes to the reduction of overall plant risk. The RCIC system is designed to operate either automatically or manually following RPV isolation, when accompanied by loss of feedwater system flow to provide adequate core cooling and control RPV water level. The RCIC system is designed to initiate within 30 seconds and discharge 700 gpm flow over a reactor pressure range of 165 to 1215 pounds per square inch absolute (psia).

The fact that the RCIC system did not continue to run after receiving its start signal on November 28, 2007, demonstrated that the flow controller settings established when they were last adjusted would have prevented the RCIC system from performing its design function. As a result, the RCIC system is considered to have been inoperable beginning on January 21, 2006. Since then, compliance with TS LCO 3.5.3, "The RCIC system shall be OPERABLE" was not assured. RCIC system inoperability in Mode 1 and Modes 2 and 3 (with reactor steam dome pressure > 150

• pounds per square inch gauge (psig)) istonsidered to be an operation prohibited by TS LCOs 3.5.3, 3.0.4, and 3.5.1; each is reportable under 10CFR50.73(a)(2)(i)(B). ' As noted above, RCIC system inoperability also impacted compliance with TS 3.5.1, ECCS Operating, Action B. Required Action B.1 would have to be performed whenever HPCS is inoperable in Mode 1. Required Action B.1 states to verify by administrative means that the RCIC system is operable within one hour. There were numerous instances since January 21, 2006 where the HPCS system was inoperable and Required Action B.1 was performed. However, without knowing that RCIC was inoperable, the intent of Required Action B.1 could not be assured. Since compliance with Required Action B.1 could not be assured, the instances would be considered an operation prohibited by TS and reportable under 10CFR50.73(a)(2)(i)(B).

A bounding probabilistic risk assessment (PRA) was performed for having the RCIC system inoperable as described above: The PRA calculated the incremental conditional core damage probability (ICCDP) in this case to be 1.4E-07. The Incremental Large Early Release Probability (ICLERP) is calculated as 15 percent of ICCDP, which results in an ICLERP of 2.1E-08.

Configurations with a core damage probability less than 1.0E-06 and a large early release probability less than 1.0E-07 are not considered to be significant risk events.

The Operational Requirements Manual section 7.6.2.1 requires a Special Report to be submitted following an ECCS actuation and injection into the reactor coolant system. The report shall include a description of the circumstances of the actuation and the total accumulated actuation cycles to date. Additionally, the current value of the usage factor for each affected safety injection nozzle shall 15‘e provided when its value exceeds 0.70. The HPCS system was used for level control and injected into the RPV on nine occasions during this event over a period of 2.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> at an RPV pressure range of approximately 930 to 550 psig. Prior to the scram, there had been 33 HPCS injections over the life of the plant. The addition of nine HPCS injections from the scram response brings the total number of HPCS injections to 42. This exceeds the number of transient events (40) assumed in USAR Section 3.9.1.1, Design Transients. An engineering analysis was performed to calculate the HPCS nozzle fatigue usage factor for 70 injections. The fatigue usage factor is the limiting factor for evaluating the number of cycles of HPCS injection into the reactor. The fatigue usage factor for 70 injections is 0.2761. Therefore, the occurrence of 42 HPCS injections has not resulted in exceeding the allowable usage factors for the HPCS nozzle. Since the HPCS injection nozzle usage factor is less than 0.70, an exact value has not been calculated for inclusion in this report.

CORRECTIVE ACTIONS

The old style Foxboro Model FPS-400-24 (P/N P0917LY) DFWCS power supplies that failed were replaced with the new style FPS-400-24 (P/N P0922YU) power supplies. An additional power supply, using a different model for added diversity and defense in depth (Lambda P/N LZSa-500-3), was added to each redundant pair of power supplies in the DFWCS and in the Digital Reactor Feedpump Turbine Speed Control System.

A voltage status alarm on each new Foxboro power supply was connected to the process alarm for DFWCS.

Load testing and line regulation preventative maintenance tasks will be created for the DFWCS power supplies. Baseline thermography will be included to support the predictive maintenance of the power supplies.

The RCIC system was re-tuned during startup from the forced outage. The RCIC flow controller settings were reset to values utilized and demonstrated to be successful at the 1987 Startup Test RCIC RPV injection settings.

A comprehensive engineering analysis of the RCIC system performance prior to, during, and after the November 28, 2007, RPS actuation was performed. Plant startup with the RCIC system inoperable on December 6 and 7, 2007, is documented in Perry LER 2007-005. Based on the results of the analysis, the restoration of tuning parameters to startup testing values, and successful completion of post maintenance testing, the RCIC system was declared operable and the plant exited TS 3.5.3, Condition A on December 21, 2007, at 0155 hours0.00179 days <br />0.0431 hours <br />2.562831e-4 weeks <br />5.89775e-5 months <br />.

A two second time delay on the low suction pressure trip signal has been installed to prevent spurious actuation of the low suction pressure trip during RCIC starts.

Instrumentation and Control (I&C) work instruction ICI-C-E51;4003, RCIC CONTROL SYSTEM TUNING, will be revised to specify limits to perform RCIC flow loop tuning/controller setting changes. Additional changes will be made to l&C instructions ICI-B17-008, P&I CONTROLLER DIAL CALIBRATIONS, and ICI-B16-015, BAILEY TYPE 701 CONTROLLER to provide for configuration management of the RCIC flow controller settings.

A training needs analysis will be performed for engineering personnel with respect to process controller tuning. A job-task analysis will be performed for I&C to encompass the lessons learned from the RCIC events, especially for tuning of the RCIC flow controller. The results will be incorporated into the engineering and I&C technician training programs, as appropriate.

PREVIOUS SIMILAR EVENTS

A review of LERs for the past 3 years found one event where an RPS actuation was caused by loss of feedwater. LER 2007-001 documents an automatic reactor scram from 31 percent power which occurred on May 15, 2007, as a result of decreasing reactor water level experienced during testing of the DFWCS. The cause was attributed to a design logic error in the DFWCS that prevented the controls from tracking correctly. Corrective actions were implemented to change the software and improve the test procedure. These actions would not reasonably be expected to have prevented the power supply failures which caused the November 28, 2007, event. No similar events were identified for RCIC.

On January 20, 2006, the RCIC flow controller failed to control the pump when placed in the automatic mode. A calibrated replacement flow controller was installed. The controller rate, gain, and reset settings were changed in the tuning process. The changed settings made the flow controller too responsive and contributed to the RCIC problems after the November 28, 2007, event.

COMMITTMENTS

There are no regulatory commitments contained in this report. Actions described in this document represent intended or planned actions, are described for the NRC's information, and are not regulatory commitments.

�