05000220/LER-2009-003

LER-2009-003, Manual Scram and High Pressure Coolant Injection Following a Loss of Feedwater Level Control Due to Firmware Deficiency

Docket Number
Event date:	10-05-2009
Report date:	12-04-2009
Reporting criterion:	10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation
2202009003R00 - NRC Website

A. PRE-EVENT PLANT CONDITIONS:

Prior to this event, Nine Mile Point Unit 1 (NMP1) was operating and stable at 100 percent power with no inoperable systems affecting this event.

B. EVENT:

At 1156 on October 5, 2009, the annunciator for Feedwater Control System trouble alarmed in the control room. Reactor water level was observed to be rising. The operators took manual control of the shaft-driven feedwater pump flow control valve (FCV) in an effort to control reactor water level. Although the FCV was given four close demand signals, reactor water level continued to rise. At 1158, the operators manually scrammed the reactor from approximately 100 percent rated power, in anticipation of an automatic reactor scram. All control rods fully inserted as required. Following the manual reactor scram, the High Pressure Coolant Injection (HPCI) system automatically initiated on low Reactor Pressure Vessel (RPV) water level as designed. At 1159, RPV water level was restored above the HPCI low level actuation setpoint and the HPCI system initiation signal was reset. After the reactor scram and turbine trip, the turbine bypass valves operated properly to control reactor pressure.

The immediate cause of the event was loss of control of the feedwater FCV, which resulted in an increasing feedwater flow rate and rising reactor water level until the operators scrammed the reactor.

The HPCI system actuation signal on low RPV level is an expected occurrence following a reactor scram due to water level shrinkage. The HPCI system is an operating mode of the feedwater system and is not an Emergency Core Cooling System (ECCS).

There was no impact on Nine Mile Point Unit 2 (NMP2) from this event.

This event involved the manual actuation of the Reactor Protection System (RPS), which resulted in a reactor scram, and the automatic initiation of the HPCI system due to reactor low water level. The notifications per 10 CFR 50.72(b)(2)(iv)(B) for RPS actuation and 10 CFR 50.72(b)(3)(iv)(A) for HPCI actuation were completed on October 5, 2009 at 1441.

THE EVENT:

There were no inoperable components or systems that contributed to this event.

D. DATES AND APPROXIMATE TIMES OF MAJOR OCCURRENCES:

October 5, 2009:

1154T The feedwater FCV was operating normally at approximately 85 percent open when the valve started to drift open. The demand signal from the master feedwater controller responded by sending a decrease signal. The valve continued to drift open.

1156T The annunciator for Feedwater Control System Trouble alarmed in the control room.

1157T The feedwater FCV was at approximately 92 percent open. The operators took manual control of the valve and sent four lower demand signals. The feedwater FCV continued to open.

1158T The feedwater FCV was at approximately 95 percent open and reactor water level continued to rise.

The operators manually scrammed the reactor. The HPCI system automatically initiated on low RPV level.

1159T RPV level was restored to above the HPCI system low level actuation setpoint and the HPCI system initiation signal was reset.

E. OTHER SYSTEMS OR SECONDARY FUNCTIONS AFFECTED:

The HPCI system was initiated on low RPV level following the reactor scram due to reactor water level shrinkage. The HPCI system is a mode of operation that uses equipment from the Feedwater system to perform its function. The HPCI system is not an ECCS.

F. METHOD OF DISCOVERY:

This event was discovered by the operators when the Feedwater Control System Trouble annunciator alarmed in the control room.

G. MAJOR OPERATOR ACTION:

Upon discovery of the feedwater FCV slowly drifting open, the operators changed the FCV controls from automatic to manual and attempted to close the FCV manually by giving it four close demand signals. When this was unsuccessful, the operators manually scrammed the reactor in anticipation of an automatic scram. After the scram, the operators verified all rods were fully inserted. No other actions were required to support shutting down the reactor.

H. SAFETY SYSTEM RESPONSES:

All safety systems responded per design. There was no loss of offsite power to the onsite emergency buses, the HPCI system initiated as designed, and the ECCS systems were available but not called upon to support the safe shutdown of the reactor.

II. CAUSE OF EVENT:

The root cause of the event falls under NUREG-1022 Cause Code B (Design, Manufacturing, Construction/Installation).

In the spring 2009 refueling outage, a Control Components, Inc. (CCI) QuickTrak II system was installed for the shaft­ driven feedwater pump FCV. This system consists of a pneumatic digital valve controller and a high-capacity servo valve positioning device. The root cause of the event was a programming error in the vendor-supplied firmware logic that prevented the proper operation of the transfer function of the FCV positioner when the operating positioner spool became mechanically bound. Instead, the FCV continued to open and raise reactor water level despite being given four close demand signals. It was determined that the most likely cause of the positioner spool binding would have been a very small particle of foreign material (FME), not visible to the human eye. No FME was actually found inside the positioner during the post-scram inspection.

The QuickTrak II system is only used for the NMP1 shaft-driven feedwater pump FCV. NMP2 does not have a shaft-driven feedwater pump or a feedwater FCV of the design used at NMPl; thus, NMP2 is not susceptible to the type of failure that occurred at NMP1.

This event was entered into the Nine Mile Point corrective action program (Condition Report 2009-006370).

III. ANALYSIS OF THE EVENT:

This event is reportable in accordance with 10 CFR 50.73(a)(2)(iv)(A) as an event or condition that resulted in manual or automatic actuation of any of the systems listed in paragraph 10 CFR 50.73(a)(2)(iv)(B). Both the RPS and HPCI system (an operating mode of the feedwater system) were actuated during this event. Both systems are listed in 10 CFR 50.73(a)(2)(iv)(B).

Except for the FCV controller and positioning device, there were no equipment failures associated with this event. Plant systems performed per design. Plant parameters, other than the reactor water level, remained within normal values throughout the event. There was no loss of offsite power to the onsite emergency buses, HPCI initiated as designed, and the ECCS systems were available but not called upon to support the safe shutdown of the reactor. Had this event occurred at low power, the shaft driven pump would have supplied water to the reactor at a faster rate and the operators may not have been able to perform the manual scram prior to reaching the reactor high level setpoint. However, the results would be the same; i.e., reactor scram with RPV water level shrinkage and HPCI initiation. It is therefore concluded that had a design basis accident occurred coincident with this event, even at low power, plant systems would have responded per design to mitigate the accident. Based on the above considerations, the safety significance of this event is very low, and the event did not pose a threat to the health and safety of the public or plant personnel.

This event affects the NRC Regulatory Oversight Process (ROP) Index for Unplanned Scrams. Due to this scram, the Unplanned Scram Index value will be 0.8 compared to the Green-to-White threshold value of greater than 3. This reduction will not result in entry into the "Increased Regulatory (White) Response Band.

IV. CORRECTIVE ACTIONS:

A. ACTION TAKEN TO RETURN AFFECTED SYSTEMS TO PRE-EVENT NORMAL STATUS:

The spool piece and cage in the FCV main positioner were replaced, the standby positioner was inspected, and the FCV was placed back in service. The plant was restored to full power on October 9, 2009.

B. ACTION TAKEN OR PLANNED TO PREVENT RECURRENCE:

Immediate actions taken to prevent recurrence were to enable the FCV controller software setting for indication of a positioner spool piece binding and to increase the Position Excess alarm sensitivity. This will provide the operators with an early indication of positioner degradation. The operations procedures were also improved to give direction to the operators on how to swap positioners if certain positioner alarms come in. The firmware for the FCV controller will be upgraded to ensure that conditions that indicate potential degradation in the performance of the positioner will result in a transfer to the redundant positioner.

A. FAILED COMPONENTS:

The feedwater FCV controller and positioning device are the only components that failed during this event.

B. PREVIOUS LERs ON SIMILAR EVENTS:

was due to a failed positioner, except one with a diaphragm design. The failure was due to a lack of the service life being defined during the design process. This style positioner was replaced with the QuickTrak II system design currently in use.

C. THE ENERGY INDUSTRY IDENTIFICATION SYSTEM (EIIS) COMPONENT FUNCTION IDENTIFIER AND

SYSTEM NAME OF EACH COMPONENT OR SYSTEM REFERRED TO IN THIS LER:

COMPONENT IEEE 803 IEEE 805 PART

COMPONENT IDENTIFIER SYSTEM IDENTIFICATION NUMBER

Feedwater Pump P SJ FCV positioner FCV SJ CCI QuickTrak II, CV-7 (Control Components, Inc.) FCV actuator FCV SJ HPCI Pump P BJ Turbine Bypass Valves V JI

D. SPECIAL COMMENTS:

None