05000389/LER-2003-003
| St. Lucie Unit 2 | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| 3892003003R00 - NRC Website | |
FACILITY NAME (1)
DOCKET
05000389, LER NUMBER (6)
Description of the Event
On June 11, 2003, St. Lucie Unit 2 was in Mode 1 operation at approximately 22 percent reactor power. The 2A main feedwater pump was supplying feedwater to the 2A and 2B steam generators (SGs) via the low power feedwater control system (LPFWCS) with the 15 percent bypass valves (EIIS:SJ:LCV), LCV-9005 (A train) and LCV-9006 (B train), in automatic operation. The main feedwater regulating valves (EIIS:SJ:FCV), FCV-9011 (A train) and FCV-9021 (B train), were out of service because station personnel were investigating feedwater flow anomalies in the main feedwater regulating valve control system.
At approximately 0122 hours0.00141 days <br />0.0339 hours <br />2.017196e-4 weeks <br />4.6421e-5 months <br />, personnel in the vicinity of LCV-9005 heard a loud noise. Control room operators received the 2A SG level high/low alarm as the 2A SG water level decreased below 60 percent level. During attempts to remotely operate LCV-9005 manually, personnel observed the valve's stem move, but no corresponding increase in feedwater flow was observed to the 2A SG.
As noted earlier, the 2A steam generator main feedwater regulating valve was out of service, so the control room reactor operators (ROB) restored flow to the 2A SG by throttling open MV-09-3, the 100 percent motor-operated feedwater bypass gate valve, as permitted by plant off-normal operating procedure ONOP 2-0700030, "Main Feedwater.' As the level increased in the 2A SG, the control room ROs closed MV-09-3, but the flow recorder still indicated flow to the SG. Although not proceduralized, the Assistant Nuclear Plant Supervisor (AMPS) made the decision to close HCV-09-1A, the main feedwater isolation valve (MFIV), to terminate the level increase. SG 2A water level turned and began to decrease, and at approximately 0125 hours0.00145 days <br />0.0347 hours <br />2.066799e-4 weeks <br />4.75625e-5 months <br /> the AMPS opened HCV-09-1A and the control room ROs opened MV-09-3 to maintain level. Subsequently, MV-09-3 was closed, but stalled at approximately 7 percent open when the motor's thermal overload protection devices tripped. At approximately 0127 hours0.00147 days <br />0.0353 hours <br />2.099868e-4 weeks <br />4.83235e-5 months <br />, with SG 2A level increasing, the AMPS again closed HCV-09-1A in the attempt to control level. At approximately 0129 hours0.00149 days <br />0.0358 hours <br />2.132936e-4 weeks <br />4.90845e-5 months <br />, the 2A SG water level swelled to the SG high-high setpoint of 88 percent. The plant response included the automatic trip of the operating 2A main feedwater pump, the automatic turbine trip, and the automatic reactor protection system (RPS) anticipatory loss-of-load reactor trip.
All rods fully inserted and all safe shutdown equipment operated as designed. The operators performed standard post-trip actions and stabilized the plant in Mode 3.
Automatic initiation of the auxiliary feedwater system did not occur and was not required due to the power history and power level present at the trip. Auxiliary feedwater was manually initiated to supply water to the SGs. However, if required, main feedwater was available to supply the SGs with water.
Cause of the Event
Two separate Event Response Teams (ERTs) were formed to analyze the event for both equipment issues and human performance issues. With respect to the equipment issues, the failure of the valve stem for LCV-9005 was the initiating event that led to the reactor trip. A metallurgical inspection of the fractured stem indicated that the causative failure mechanism was fatigue. The orientation of the fracture indicated that the causative low stress/high cyclic loads were primarily bending in nature, and was indicative of vibration. Piston ring wear appears to have contributed to the inservice vibration. Prior to unit restart, LCV-9005 was repaired, and its sister valve LCV-9006 was disassembled and inspected. The stem to plug connection for LCV- 9006 was loose and the locking pin was sheared, a condition that is attributable to FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) St. Lucie Unit 2 � 05000389 _ NUMBER NUMBER the application of improper torque to the stem to plug connection. LCV-9006 was reassembled with a new valve stem and plug assembly. The vendor manuals for LCV-9005 and LCV-9006 will be updated to include the proper torque requirements for the stem to plug connections. FPL is developing enhanced periodic maintenance criteria for LCV-9005 and LCV-9006 for both Units 1 and 2 under the Preventive Maintenance (PM) program. FPL evaluated extent of condition by reviewing St. Lucie Units 1 and 2 Fisher control valves for common cause failure mechanisms. The results of the review concluded that the application, service, inspection frequency, and/or design of the valves reviewed are much less susceptible to vibration induced fatigue failures.
Investigation into the inability to fully close MV-09-3 revealed that the motor operator design was marginal if the valve is manually cycled beyond its anticipated normal operation. Although the sizing of the thermal overload devices was acceptable based on design considerations (i.e., full stroke of the valve), additional operating margin would be desirable when jogging the valve to set the required flow. Prior to unit restart, the thermal overload devices for MV-09-3, and its sister valve MV-09-4, were replaced with larger thermal overload devices to provide greater operational margin. Post maintenance testing was completed satisfactorily. FPL also plans to replace the St. Lucie Unit 1 100 percent feedwater bypass thermal overload devices.
With respect to human performance issues, St. Lucie Operations and industry standards stress that operators should manually trip the reactor when plant parameters cannot be controlled and are approaching automatic trip setpoints. A manual reactor trip was not initiated during the approximately eight-minute period following the failure of LCV-9005.
The ERT determined that inadequate command and control of operator activities during the feedwater transient was the root cause of the failure to perform an anticipatory manual reactor trip. The licensed Nuclear Plant Supervisor (NPS) and ANPS failed to maintain the proper overview of the transient. The ANPS stepped out of his supervisory oversight role when he became actively involved with valve manipulations on the control board. The NPS was initially located outside the control room observing the local main feedwater control system troubleshooting activities.
However, when the NPS returned to the control room he became involved in mitigating the failure and lost focus on overall plant status. Symptomatic of the breakdown in the command and control structure, the licensed board ROs were not actively providing parameter and equipment status, and procedural guidance was not being effectively communicated among crewmembers during the event. The NPS and ANPS failed to meet expectations for both control room oversight and for conservative decision-making.
Other key contributing factors to this event include:
- Procedural guidance was inadequate. The off-normal operating procedure was written for main feedwater operation and did not address the LPFWCS. The ANPS made a cognizant knowledge-based error when he attempted to control SG water level by manipulating the MFIV. Additionally, there was little margin between the criteria for a manual trip and the automatic SG high level trip setpoint with respect to performing a successful anticipatory trip during rapidly changing conditions. Furthermore, use of the 100 percent bypass valve was inappropriate at low power levels.
- Expectations for developing operating contingency actions for degraded conditions that could lead to a plant transient were not clearly established, nor was there procedural guidance to ensure consistency in developing these actions or communicating them to subsequent crews. Contingency actions for a loss-of-feed FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) St. Lucie Unit 2 05000389 were discussed during shift meetings during this time, however, these discussions did not contain sufficient detail to prevent or preclude inappropriate operator actions.
In order to address the human performance root cause and contributing factors all control room operators were required to attend an Operations Management brief and simulator exercise prior to standing watch. The exercise reinforced Management expectations for the ANPS/NPS oversight role, procedure use, conservative decision making, and development of detailed contingency actions for postulated abnormal conditions. Similarities between this event and INPO significant operating event reports (SOERs 96-01 and 94-01) were reviewed along with the new procedural requirements for the reduced manual SG high level trip criteria and use of the 100 percent feedwater bypass valves. Additionally, the NPS and ANPS involved in the event were removed from active watch standing until they completed a comprehensive remediation program.
Operations departmental policies 515, "Control Room Oversight," and 516, "Conservative Decision Making," were developed to clarify Management expectations with regard to control room oversight and conservative decision making. Operations departmental policy 513, "Abnormal/Degraded Operating Conditions," was developed to provide formal guidance on evaluating abnormal and degraded conditions as well as establishing compensatory and contingency actions for these conditions. These policies have been communicated to all licensed operators and operations training instructors. The St. Lucie Training department developed Simulator Exercise Guide 0814066, Revision 01, "Ops Teamwork and Performance Coaching Scenario," that reinforce the Operations departmental policies discussed above, as well as reinforce clear and precise communications between control room team members. This training module has been incorporated into Licensed Operator Continuing Training (LOCT).
Analysis of the Event
This condition is reportable under 10 CFR 50.73(a)(2)(iv)(A) as an event that resulted in manual or automatic actuation of the reactor protection system (RPS).
There are two steam generator feedwater pumps in the main feedwater system. Both pumps are required for 100 percent power operation. One pump is capable of sustaining 60 percent power operation. During low flow conditions, a recirculation system allows feedwater from the pump discharge to be diverted back to the condenser to protect the pump from overheating.
The feedwater control portion of the system consists of two separate trains, one for each SG. Both SGs have their own feedwater regulating system consisting of a 100 percent flow capacity electro-pneumatic feedwater regulating valve, a 15 percent flow capacity electro-pneumatic bypass valve, and a 100 percent motor-operated bypass valve for emergency operation. Automatic control is the normal mode of operation for these flow control valves with manual-remote capability available from the control room if necessary. Normally, for low power levels, feedwater control is via a 15 percent bypass valve using signals from the LPFWCS. The LPFWCS uses inputs from SG level, neutron flux and feedwater temperature. These parameters provide a stable SG level program for low power operation and provides finer level control during startup, particularly during turbine synchronization, than either manual or the main feedwater automatic control system.
During normal operation, feedwater control is via the 100 percent flow capacity electro-pneumatic feedwater regulating valves. The motor-operated 100 percent flow bypass valve is designed to provide an alternative method of SG level control during main feed regulating valve on-line maintenance or malfunctions. SG level control using the 100 percent motor-operated feedwater bypass gate valve would be difficult and would not be considered a normal operating situation. Therefore, when controlling flow with the 100 percent feedwater bypass valve, the 100 percent bypass is used to get near the desired flowrate, and fine tuning is performed with the 15 percent feedwater bypass valve. At low power levels, loss of both the 15 percent bypass valve and the main feedwater regulating valve would result in the inability to control SG level via the 100 percent feedwater bypass valve, with the most likely result being a manual or automatic reactor trip.
During the SL2-14 refueling outage, post maintenance testing was performed on the low power feedwater, steam bypass, and main feedwater control systems following maintenance activities, although integrated system performance was not verified prior to returning these systems to service. This resulted in Operations being challenged by the on-going troubleshooting efforts associated with the set up of these control systems during plant start up. During this event, the main feedwater regulating valves were out of service to support the troubleshooting activities associated with the main feedwater control system. The LPFWCS was operating in automatic mode when the stem for LCV-9005 failed. Attempts to recover SG level and stabilize the situation via the 100 percent bypass valve and MFIV were unsuccessful and resulted in an automatic reactor trip. Prior to returning the unit to power, FPL conducted comprehensive integrated testing on the main feedwater control, low power feedwater control, and steam bypass control systems. FPL is evaluating secondary plant control system post maintenance testing requirements for outages.
Operating experience for LCV-9005 was examined. Based on time-in-service criteria, LCV-9005 was originally scheduled to be inspected during the spring 2003 SL2-14 refueling outage. However, during the outage a review was performed and the inspection was deferred until the following refueling outage. FPL issued a Condition Report to evaluate the decision-making process for deferring outage work scope.
After the event, St. Lucie performed a more detailed review of the Condition Report, INPO OE, INPO EP/X, and work history databases for relevant history on Fisher Type HPD control valves. No relevant industry data with respect to stem failures was found. The Condition Report review revealed no past issues with Fisher valve stem failures at St. Lucie. Work order history identified actuator, actuator positioning arm, and positioning pin replacements along with positioner calibration, but no stem failures were identified in that review.
Analysis of Safety Significance Reactor trips are analyzed events, and all safe shutdown equipment operated as designed. Although the operators chose to use auxiliary feedwater as the water supply for the SGs post trip, if required the main feedwater system was available to feed the SGs. The analyzed loss of load transients are more limiting than the transient experienced during this event. Therefore, this event had no adverse effect on the health and safety of the public.
FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3)
Corrective Actions
1. The plant was stabilized in Mode 3.
2. For St. Lucie Unit 2, LCV-9005 was repaired and 9006 was disassembled, inspected, valves LCV-9005 and 9006 will be inspected prior to startup from the next refueling or short-notice outage of sufficient duration.
3. The control room crews were briefed on the event prior to standing watch. The topics included the ANPS/NPS oversight role, procedure usage, conservative decision making, and development of detailed contingency actions for postulated abnormal conditions.
4. The NPS and ANPS involved in the event were removed from active watch standing until a comprehensive remedial program was completed.
5. The Units 1 and 2 procedures for main feedwater (ONOP 1/2-0700030) were revised to provide guidance on the use of the 100 percent feedwater bypass valves MV-09-3 and MV-09-4 during recovery of loss of feedwater transients. The manual trip criteria for SG high-high level was reduced from 85 to 80 percent level.
6. Management expectations for control room crews were enhanced by Operations departmental policies 513, "Abnormal/Degraded Operating Conditions," 515, "Control Room Oversight," and 516, "Conservative Decision Making.
7. Simulator Exercise Guide 0814066, Rev. 1, "Ops Teamwork and Performance Coaching Scenario," was developed and incorporated into the LOCT program.
8. Maintenance procedures for LCV-9005 and LCV-9006 will be developed and vendor manuals will be updated with stem to plug torque requirements.
9. The overload devices for Unit 2 valves MV-09-3 and MV-09-4 were replaced under valves MV-09-3 and MV-09-4.
10. St. Lucie is developing enhanced periodic maintenance and inspection criteria/frequency for LCV-9005 and LCV-9006 for both units. St. Lucie is developing the model PM work orders that will implement these revised requirements.
11. St. Lucie is evaluating secondary plant control system post maintenance testing requirements for outages.
12. St. Lucie is evaluating the decision-making process for deferring outage work scope.
Additional Information
Failed Components Identified Tag / Component: � LCV-9005 / Level Control Valve For FW Reg Station (FCV-9011) 15 Percent Bypass Manufacturer: � Fisher Control Valve Model No: � Type HPD, 4 inch Similar Events None