05000389/LER-2003-006

LER-2003-006,

Docke ( Num C
Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
3892003006R00 - NRC Website

Description of the Event

On December 20, 2003, St. Lucie Unit 2 was in Mode 1 at 100 percent power. At 0948 hours0.011 days <br />0.263 hours <br />0.00157 weeks <br />3.60714e-4 months <br />, St. Lucie Unit 2 automatically tripped from full power on loss of generator field current. Both steam generator levels decreased to below their auxiliary feedwater actuation signal (AFAS) setpoints. The AFAS-1 signal was generated 10 seconds after the AFAS-2 signal. The 2C turbine-driven auxiliary feedwater (TDAFW) pump [EIIS:BA:P] started approximately 9 seconds after AFAS-2 and then tripped approximately 38 seconds later. Control room annunciation and local inspection confirmed that the mechanical overspeed trip mechanism had tripped.

A similar event occurred on April 1, 2003 when the 2C TDAFW pump experienced an overspeed trip after successfully starting on demand. At the time, the most probable cause was determined to be problems with the trip mechanism engagement due to worn parts. The .worn parts allowed vibration-induced slippage causing a turbine trip without an overspeed condition. However, the similarities between the April 2003 and December 2003 event raised questions with respect to the previous conclusion that the trip was due to the mechanical overspeed trip device. An event response team (ERT) was formed to investigate the cause of the overspeed trip.

The AFW system consists of a condensate storage tank (CST), two electrical motor-driven AFW pumps, one TDAFW pump, and manual, check, solenoid, and motor- operated valves that provide for flow to the steam generators. The AFW system uses secondary steam to drive the TDAFW pump. The TDAFW pump is capable of supplying AFW flow to the steam generators for the total expected range of steam generator pressure by means of a turbine controlled by a Woodward PG-PL mechanical governor (EIIS:BA:65].

Each motor-driven pump is lined up to feed one steam generator with a cross connection to enable the routing of the flow to each steam generator. The turbine- driven pump supplies feedwater to both steam generators by means of two separate lines each with its own control valve and each sized to pass the required flow. AFW valves needed for system operation under design bases events have control switches in the control room. Each of the motor-driven auxiliary feedwater pumps utilize a Class 1E AC power supply (4.16 kV safety related bus). The turbine-driven pump train relies strictly on a DC power supply.

The AFW system is provided with complete sensor and control instrumentation to enable the system to automatically respond to a loss of steam generator inventory. A separate AFAS is generated for each steam generator, AFAS-1 for steam generator 2A and AFAS-2 for steam generator 2B. AFAS-1 controls the 2A AFW pump and starts the 2C TDAFW pump by opening "A" side steam admission valve MV-08-13. Similarly, AFAS-2 controls the 2B AFW pump and starts the 2C TDAFW pump by opening the "B" side steam admission valve MV-08-12.

• Upon low steam generator level, an AFAS 330-second time delay is actuated,.. If the applicable steam generator water level increases to reset the low level actuation bistable before the AFAS time delay expiresithe time delay resets and the AFW system is not actuated. If the AFAS time delay expires while steam generator level is below the AFAS low level actuation bistable (first) reset, then the AFW system receives an AFAS. Provided that the AFAS logic does not identify the steam generator or its associated feedwater supply header as being ruptured, the signal starts the AFW pumps to supply a minimum feedwater flow of 275 gpm to the steam generator at 1000 psia.

The AFW system supplies water to the steam generator until the steam generator level II: � 355J1 ti-201111 FACILITY NAME (1)

DOCKET

LER NUMBER (6) St. Lucie Unit 2 05000389 increases to the second AFAS reset setpoint, where the AFW pump discharge valves automatically close thereby stopping flow to the steam generators.

Cause of the Event

The ERT developed a detailed troubleshooting plan to identify the cause of the 2C TDAFW pump overspeed trip. Five main categories of potential causes were identified:

1) overspeed linkage unlatching inadvertently, 2) condensate in the steam supply lines overwhelming the governor capabilities, 3) governor valve binding, 4) erratic governor response, and 5) variation in turbine load. These potential causes were based on in-house expertise, onsite vendor and industry expert input, and industry Operating Experience (OE).

Inspection of the mechanical overspeed trip mechanism and governor valve operation ruled out potential causes 1 and 3 above. To check the other causes, testing was devised to replicate the overspeed event by initiating a staggered AFAS actuation by opening the "A" side steam admission valve 10 seconds after the "B" side steam admission valve was opened.

The 2C TDAFW pump was started by opening the "B" side steam admission valve, MV-08- 12, and the "B" steam generator feed valve, MV-09-12, simultaneously. The 2C TDAFW pump turbine speed increased rapidly to 3075 rpm, where the governor took control, the speed dropped back to about 2325 rpm, and then the turbine ramped up and stabilized at 3750 rpm for about four seconds. This is the expected start response.

After the "A" side steam admission valve, MV-08-13, and the "A" steam generator feed valve, MV-09-11, were opened, the turbine continued to operate with minor speed oscillations for approximately two seconds at which time the mechanical overspeed linkage was observed to trip. This test confirmed that the earlier event was a valid overspeed trip of the 2C AFW pump.

A second test was performed to determine if condensate build-up in the steam supply line was causing the turbine to overspeed. The turbine was started manually with both steam admission valves open and run for several minutes to sweep condensate from the steam supply lines. After the preconditioning run, the first test was repeated.

During the second test, the turbine came up to speed as before but did not trip. The speed trace was very smooth with very little perturbation or speed oscillation. This test indicated that in the absence of significant condensate, the turbine governor was controlling correctly.

The steam supply lines are kept warm to minimize condensate formation by the use of steam bypass and drain valves. In order to maximize the ability to maintain the condensate build-up in the steam supply piping as low as possible with the current design, the philosophy of setting the steam bypass valves was changed. Instead of setting the valves such that a rolling turbine would stop, the valves were set such that once the turbine was stopped, it would not begin to roll. If the turbine was rolling prior to a start, it would not be able to catch the start transient and would trip on overspeed. This change in philosophy allowed the net effect of the bypass valves to be increased. At the end of the testing sequence, the final bypass valve positions were deemed to be acceptable based on successful testing and the adjustment of the governor stability setting.

Further testing of the staggered AFAS scenario led to the realization that opening ' the second steam admission valve could still introduce condensate into the running turbine, thereby challenging the turbine governor's ability to control the perturbation. Prior to the first steam admission valve opening, the initial pressure NA Nava t1-2111111 FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) St. Lucie Unit 2 05000389 in the piping downstream of the steam admission valves is slightly above atmospheric.

These gate valves have a full stroke time of approximately 27 seconds. Due to the pressure differential, when the first valve opens the flow is choked and most of the condensate that may be accumulated upstream of the valve will either flash to steam or rapidly be flushed down the pipe. The initial steam flow rates are very high. By the time the governor starts reacting to control the speed of the turbine (about three seconds), most of the remaining condensate upstream of the valve and all the condensate downstream of the valve would have passed through the turbine.

When the second steam adMission valVe opens, the downstream piping is presstlrized.

Any condensate upstream of the second valve will not flash to steam. The condensate will flow relatively slow based on gravity and steam flow rate. The steam flow in the second line will be approximately half of the total flow. This condensate will upset the turbine speed and consequently governor control when it is introduced into the turbine. The governor will still attempt to control the required speed of the turbine. However, if the turbine governor is not adequately tuned, then divergent oscillations would lead to overspeed.

The 2C TDAFW pump utilizes a self-contained mechanical/hydraulic Woodward PG-PL governor to control turbine start and constant speed operation. Governor stability can be adjusted by a compensating needle valve on the hydraulic servo unit. This needle valve is adjusted whenever the governor oil is changed to flush air out of the governor hydraulic control lines and to establish a stable governor set-up.

Procedure IMP-09.01, "2C Auxiliary Feedwater Pump Governor Oil Change Instruction," controls the oil change and set-up process. However, the procedure was based on the vendor manual guidelines, and lacked the specific details needed to ensure consistent governor set-up. Governor oil is changed out on an 18-month frequency, having been last changed in May 2003. During troubleshooting after the overspeed trip, the compensating needle was found to be 3/8 turn open. A compensating needle valve adjustment was performed with vendor support and checked at normal operating speed.

The needle valve was left at 5/32 turns open. After this adjustment was made, the 2C AFW was tested successfully in both staggered start alignments. Indication of condensate in the steam supply lines was still evident, however, the governor responded to the perturbation and quickly returned speed to nominal. It was apparent from this evolution that the governor set-up procedure was inadequate.

The normal Technical Specification surveillances used to demonstrate operability of the AFAS and AFW pumps were inadequate. Although the AFAS design could result in a staggered opening of the 2C TDAFW pump steam admission valves, the system response to this scenario was not tested. After polling several other utilities with CE designed plants, it became obvious that the staggered start configuration at St. Lucie is unique. Only one other plant had an original AFW system design that produced a staggered start similar to St. Lucie. The design at that plant was changed to simultaneous start in the 1995/1996 time frame. The occurrence of successful 2C TDAFW pump performance in response to dual, staggered AFAS initiations at St. Lucie contributed to the perception that each failure was due to a discreet component issue and not a long-standing design inadequacy.

Based on these findings, two root causes were identified for the 2C TDAFW pump overspeed trip after receipt of valid AFAS signals: 1) inadequate design of the AFAS start logic and 2) inadequate design of the steam supply piping. Specifically, the original.AFAS design did not fully consider the effect of a staggered start on the turbine-driven pump. The steam admission piping is configured such that sufficient 111: NU 2551C1-7111113 FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) St. Lucie Unit 2 05000389 condensate builds up to cause a perturbation to the turbine and governor when the second steam admission valve opens.

The contributing causes were identified as:

• Less than adequate adjustment of the turbine governor (type PG-PL). The governor compensating needle valve was set too open causing the governor control to become unstable and divergent when challenged by a staggered AFAS start.

• The procedure for setting the compensating needle valve was not adequate for setting the governor adjustment for the staggered start design condition.

• The steam supply line bypass and drain valves were not positioned to minimize condensate in the steam supply lines.

• Surveillance testing did not test system response to a staggered AFAS initiation.

Analysis of the Event

This event is reportable under 10 CFR 50.73(a)(2)(i)(B) as "any operation or condition which was prohibited by the plant's Technical Specifications... " The AFAS staggered-start and steam line condensate design inadequacies existed since the commencement of commercial operation of St. Lucie Unit 2. Although there are instances where the 2C AFW pump successfully operated during a AFAS staggered-start demand in the past, surveillance and maintenance procedures did not demonstrate that a staggered-start AFAS demand would not result in a 2C AFW pump overspeed trip. This latent defect resulted in periods of 2C AFW pump inoperability greater than the TS allowed outage time.

Analysis of Safety Significance FPL performed testing to validate the 2C TDAFW pump response to demand signals. When the first steam admission valve opened, the 2C AFW pump governor maintained control of the turbine during the initial introduction of condensate through the turbine.

Opening the second steam admission valve introduced additional condensate, and if the turbine was at full speed, the governor may have been unable to control the perturbation and cause a 2C AFW pump overspeed trip. However, opening the second steam admission valve effectively purged the condensate in the steam supply lines.

Subsequent operator actions to restart the pump would not be subject to a similar pump trip because significant amounts of condensate would not be present in the lines. FPL validated this behavior by the testing conducted as a part of this event analysis.

Based on the probabilistic safety assessment (PSA) of this condition, risk is dominated by a total loss of feedwater (both main feedwater and AFW). For the purposes of the risk assessment, the AFAS demand is assumed to result in an overspeed trip of the 2C TDAFW pump. In this scenario, the operators would enter 2-EOP-01, "Standard Post Trip Actions," which would lead them to 2-EOP-06, "Total Loss of Feedwater." Step 8.A of 2-EOP-06 would direct the crew to 2-0NP-09.02, "Auxiliary Feedwater," or to Appendix G of 2-EOP-99, "Local Operation of the 2C Auxiliary Feedwater Pump." 2-ONP-09.02, Appendix A would normally be used for restoration of the 2C TDAFW pump unless the pump could not be started from the control room following resetting an overspeed trip.- The operators would take approximately 12 to 15 minutes to transition through 2-EOP-6 to 2-ONP-09.02 or Appendix G. It is assumed that it would take another 10 to 12 minutes to complete Appendix A or Appendix G BIC RIX 315111-21111 FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) St. Lucie Unit 2 05000389 actions and restore auxiliary feedwater flow. Since a trip of the 2C TDAFW pump is alarmed in the control room, one operator would typically begin the restoration steps in 2-ONP-09.02 in parallel with the remaining crew performing the procedural requirements of 2-EOP-01 and 2-EOP-06. The timing assumptions for the operator actions to diagnose the loss of feedwater event, take the actions to reset the 2C TDAFW pump overspeed trip, and reestablish flow to both steam generators are based on drills by two different crews and a simulator exercise. This total action time (less than 30 minutes) is well within the estimated 60-minute time until core damage would occur as a result of a total loss of feedwater. The 60-minute time to core damage is consistent with the assumptions used in development of the baseline St. Lucie PSA. A scoping number of 0.05 for the probability of operator's failure to reset the 2C TDAFW pump and deliver water to either the .A or B steam generator was estimated using a tool similar to the EPRI HRA Calculator. The CDF risk increase is below 1.0E4/Yr.

Based on the above, this condition had no adverse impact on the health and safety of the public.

The St. Lucie Unit 1 TDAFW pump is similar to the St. Lucie Unit 2 TDAFW pump with several significant differences. Based on differences in design and operating experience on Unit 1, FPL identified no immediate actions for the 1C TDAFW pump.

However, the 1C TDAFW pump was tested using a sequential opening of the steam admission valves to validate that there was not a common concern. Potential changes to the 1C TDAFW AFAS logic, supply piping and enhancements to the 1C AFW governor set-up procedure will be addressed as part of the long-term corrective actions for this LER.

The St. Lucie Units 1 and 2 emergency diesel generators (EDGs) are possibly susceptible to a governor stability adjustment issue although different model governors are used. However, this is not likely because the maintenance procedures for setting the gain and reset of the EDG governor are more methodical than the current TDAFW turbine governor maintenance procedure. Additionally, the EDGs are tested during Safeguards testing in a manner that emulates the design basis events.

As such, instability of engine speed due to inappropriate set-up would be visible during this testing.

No further generic implications were identified during the investigation of this event.

Corrective Actions

1. The 2C TDAFW pump is being tested with the steam admission valves opening in series similar to the recent plant trip scenario (MV-08-12 then MV-08-13 at least 10 seconds apart to allow the pump to reach full speed). Also, the reverse opening sequence pump start is also being performed. This testing is being performed in increased frequency to validate that the success of the testing is not a function of time between tests. This testing was performed initially after a four hour cooldown (A-B), then an 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> cooldown (A-B), then 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> (A-B), then 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> (B-A), then 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> (A-B), then 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> (B-A), then weekly for a month, then bi-weekly for a month, and finally incorporated into the regular monthly surveillance testing. Beginning with the weekly testing, only one series (A-B or B-A) needs to be performed, each test alternating to the opposite series the following test. TDAFW pump 2C speed will be recorded on a high-speed recorder for the first two months of augmented testing. This activity is being tracked by Procedure OP-2-0010125A, Data Sheet 30.

RL MIN 355A 11-21111i FACILITY NAME (1) DOCKET NUMBER (2) LER NUMBER (6) PAGE (3) 2. FPL has modified Procedure 2-0700050, "Auxiliary Feedwater Periodic Test," to specify the method for setting the steam admission bypass needle valves to optimize condensate removal and heating of the downstream piping.

3. FPL has modified Procedure 2-0700050, "Auxiliary Feedwater Periodic Test," to specify staggered starting for the monthly surveillance tests.

4. FPL will modify Procedure IMP-09.01, "2C Auxiliary Feedwater Pump Governor Oil Change Instruction," to clearly delineate the 'method and criteria for setting the governor compensating needle to ensure the optimum stability setting is established.

5. FPL is evaluating a modification to the AFAS start logic for the St. Lucie Units 1 and 2 TDAFW pumps to simultaneously open both steam admission valves.

Additional Information

The December 20, 2003 St. Lucie Unit 2 automatic trip details are documented in LER 50-389/2003-005-00, "Automatic Reactor Trip Due to Loss of Turbine Generator Excitation.

Failed Components Identified Component Tag:

� 2C TDAFW Pump Governor Manufacturer: � Woodward Model Number: � PG-PL Similar Events Vacuum," documents an earlier, April 1, 2003 overspeed trip of the running 2C TDAFW pump.

ROTI 351011-20111