05000286/LER-2009-004
| Docket Number | |
| Event date: | 5-28-2009 |
|---|---|
| Report date: | 4-21-2010 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation 10 CFR 50.73(a)(2)(v), Loss of Safety Function |
| 2862009004R01 - NRC Website | |
Note: The Energy Industry Identification System Codes are identified within the brackets {}.
DESCRIPTION OF EVENT
On May 28, 2009, at 5:50 hours, while at approximately 100% steady state reactor power, Control Room operators received vibration alarms {JK} (B.F.P. Turbine 32 Vibration & Thrust) and notification of an unusual noise on the 32 main boiler feedwater pump (MBFP) {SJ} bearing 3. The 32 MBFP was taken to idle and as per design the 32 MBFP discharge recirculation valve (FCV} opened. The 31 MBFP speed stopped increasing even though the Start-Up speed control was set 86%. At approximately 6:17 hours, steam generator (SG) {AB} levels decreased from 45 percent to approximately 38 percent.
While in automatic, all four main feedwater (FW) regulating valves (FRV) {FCV} opened and SG-31, 33 and 34 water levels started increasing. FW flow to the 32 SG was not increasing and SG water level was not increasing. At approximately 6:20 hours, Operators reduced power to approximately 61%. FRV-31, 33 and 34 were responding in Auto to control SG level, while the 32 FRV remained at 100 percent demand signal. At 6:22 hours, SG-32 level was high and increasing. At approximately 6:23 hours, with 32 SG water level at approximately 71 percent, operators placed the 32 FRV in manual in an attempt to control SG level but the level reached the high level trip set point {JB} initiating a turbine trip (TT) {JJ} that resulted in an automatic reactor trip (RT) {JC}. All control rods {AB} fully inserted and all required safety systems functioned properly. The plant was stabilized in hot standby with decay heat being removed by the main condenser {SG}. There was no radiation release. The Emergency Diesel Generators {EK} did not start as offsite power remained available. The Auxiliary Feedwater (AFW) System {BA} automatically started as expected due to Steam Generator low level from shrink effect. The event was recorded in the Indian Point Energy Center corrective action program (CAP) as CR-IP3-2009-02494 and CR-1P3-2009-02710. A post transient evaluation was initiated and completed on May 28, 2009. The root cause evaluation performed under CR-IP3-2009-02710 was revised based on questions identified in CR-1P3-2009-04393.
Review of conditions prior to the event determined there were problems with the FRVs and MBFP in the March 11, 2009 shutdown for the unit 3 cycle 15 refueling outage.
During the shutdown, the 32 MBFP was taken out of service and power reduced to approximately 38 percent. During the shutdown the 31 MBFP speed increased to assume the flow required but the MBFP speed did not go above 4000 RPM. The percent FW demand signal was in excess of the FW start-up signal, which was set at 85 percent.
Decreasing 31 MBFP discharge pressure caused reduced FW flow that resulted in decreasing SG levels. The resulting transient caused the FRVs to fully open which resulted in a rise in SG levels which overshot their normal operating levels. The control room operators switched to manual control of the 31 and 32 FRVs to stabilize level and a normal shutdown for the outage continued. This event was recorded in the CAP as CR-IP3-2009-00730 which included a corrective action (CA) that required a full stroke check of the MBFP high pressure (HP) governor valve. Maintenance was to perform the check in accordance with procedure 0-TUR-402-MFW,"Main Boiler Feed Pump Turbine Inspection." The actual data recorded did not include the stroke displacement of the HP governor valve itself even though the procedure required recording this data. The data recorded included the displacement of the governor hydraulic servo motor but not the governor valve itself. Additionally, the procedure did not contain specific steps to check for "slop" or looseness in the actuating linkages between the governor hydraulic servo motor and the governor valve. On April 15, 2009, work on the 31 MBFP HP governor linkages was completed by a vendor in accordance with procedure 0-TUR-402- MFW. However, the vendor did not complete the procedure sections which require recording governor valve stroke. The need for governor valve stroke data was added to the procedure to ensure that the valve would provide an acceptable steam supply for satisfactory operation of HP steam for the MBFP.
This occurs in both the level control (LC) and flow control (FC) sections of the three element control configuration. A total demand signal is then sent to components that adjust FRV stroke, thereby positioning them as required by the control system. The SGWLC system uses proportional and integral controllers {JB} manufactured by NUS Corporation {N430}.
An extent of condition (EOC) review was performed and determined that the MBFP hydraulic control system is similar between unit 2 and 3, as is the Lovejoy speed control and FW controllers. The FRV controllers are also the same for both units.
The Lovejoy speed control system is unique to the MBFP turbines. The problems with relay spring constants and linkage looseness are applicable to the Main Turbines however, the size and configuration of these components on the Main Turbine are sufficiently different as to exclude them from being considered a EOC of the MBFP control problems. The AFW turbine driven pumps have control systems that rely on mechanical driven components and not hydraulic components.
Cause of Event
The direct cause of the RT was a TT from a high SG-32 level. The cause of the high SG level was overfeed of the 32 SG by the 31 MBFP due to the inability of the 31 MBFP turbine to operate at higher speeds on HP steam and due to the 32 FRV SG water level controller (LC) going into saturation. The limited ability of the 31 MBFP to operate on HP steam was due to improper HP governor valve stroke.
The root causes (RC) for the RT were: RC1) Ineffective problem solving.
Troubleshooting and problem solving techniques were not effective and resulted in missed potential failure modes and incomplete understanding of the impact of a deficient high-critical balance-of-plant (BOP) component on plant operation, and RC2) Poor vendor oversight. Poor vendor oversight as a result of diminished maintenance standards allowed the vendor to close the work package without completing the governor valve stroke readings. This was a missed opportunity to identify and correct the improper HP governor valve stroke.
Contributing causes (CC) were: CC1: Inadequate procedures. Procedure 0-TUR-402-MFW HP/LP overlap tolerances were excessive based on previous vendor recommendations to minimize valve performance instability. In addition, the procedure did not specify to check the linkages for excessive clearance which can cause binding and limit the stroke of the HP governor valve. CC2: Procedure adherence. The MBFP vendor failed to document the 31 MBFP HP governor valve stroke settings as required by procedure 0-TUR-402-MFW. Maintenance personnel did not meet procedure use and adherence expectations during on-line adjustments of the HP governor valve and the As-Left HP governor valve stroke measurements were not documented following field adjustments, CC3: Low FW flow in 32 SG feed line. With all four FRVs full open, the 32 FRV flow was 5-10% lower than the other three FRVs flow. This condition caused the 32 FRV controller to go into saturation due to excessively short reset time, and 32 FRV controller could not recover in time to avert the SG high level trip actuation, CC4:
Ineffective corrective actions. A CR Corrective Action referenced scope add Form 259 to verify full stroke of 31 BFPT control valves from the CR via MBFP speed control Lovejoy % FW signal and % startup signal. The scope add form only required checking the signal from the CR to the I/P and never required a check of the HP governor valve stroke so the outage WO never measured the valve stroke.
Corrective Actions
The following corrective actions have been or will be performed under Entergy's Corrective Action Program to address the cause and prevent recurrence:
- Engineering personnel have completed attendance of Kepner-Tregoe (K-T) training based on Engineering Standard EN-MS-S-012-MULTI and have used the K-T problem solving technique to address problems on the 23 Component Cooling Water pump and the 32 AFW pump.
- New SG Water Level (SGWL) Controller settings were developed and are being evaluated for their appropriateness for implementation. Schedule due date is September 30, 2010 for final engineering determination.
- The need for additional vendor oversight was assessed and determined to be necessary and CR-IP2-2009-02629 initiated for adverse trend for inappropriate vendor oversight. The additional vendor oversight is being implemented by AFI-1 (Supplemental Worker Performance) under the Maintenance Improvement Plan and by Outage Management under the Supplemental Personnel-Critical Maintenance Identification and Mitigation white Paper from the fleet vendor oversight initiative. The evaluation performed under CR-IP2-2009-02629 identified additional corrective actions to improve performance.
- Maintenance personnel were coached on management expectations on the requirements of procedure EN-MA-126, "Control of Supplemental Personnel," relating to oversight of contractors.
- This event was incorporated into the Maintenance Supervisor continuing training especially highlighting oversight of contractors.
- Maintenance, Operations, System Engineering and Outage Management personnel were briefed on the root cause and lessons learned from this event especially in the areas of vendor oversight, inadequate procedures, informal communications and complacency.
- Procedure EN-MA-125 (Troubleshooting Control of maintenance Activities) was reviewed against industry standards and lessons learned from this event to ensure it provides sufficient guidance regarding team composition requirements when troubleshooting complex or significant problems. The review determined the procedure is in alignment with industry standards and no changes were necessary.
- A TEAR was generated to ensure that a sufficient core group of individuals within the Maintenance Department are proficient in the use of the K-T method of problem resolution.
- Maintenance, and Engineering personnel were briefed on Management expectations for effective problem solving using K-T methods and other available resources including use of formal procedures, processes, and industry experts when troubleshooting significant emergent issues. The briefing reinforced the expectation that an effective team approach is needed for troubleshooting complex problems. Engineering has completed its briefing. Maintenance is scheduled to complete briefing department personnel after the spring refueling outage.
- Procedure 0-TUR-402-MFW was revised to always require stroke measurement of HP governor valves, and the linkage inspection step was revised to specify a check of the linkages for excessive clearance to ensure there is no binding to limit the stroke of the HP governor valve. The procedure was also revised to reduce the HP/LP governor valve overlap tolerances based on vendor and engineering recommendations to include the addition of Entergy verification signoffs at critical steps.
- Management expectations were provided to planners when creating or modifying work orders that have CRs attributed to them that they must ensure the instructions resolve the issue in the CR.
- Procedure adherence was included in the pre-outage just-in-time presentation for vendor personnel for the next refueling outage in the spring of 2010.
- Coaching was provided to the responder to CA-5 of CR-IP3-2009-00730 to ensure CAs in the response are completed as written.
- A work request was initiated to be scoped into the next refueling outage to open the 31 MBFP HP governor valve (steam side) to determine if any internal damage or other factors contributed to this event.
- An engineering Change Request (ECR) was initiated to examine the HP governor valve stroke specifications, including the HP governor valve minimum stroke requirement.
Event Analysis
The event is reportable under 10CFR50.73(a)(2)(iv)(A). The licensee shall report any event or condition that resulted in manual or automatic actuation of any of the systems listed under 10CFR50.73(a)(2)(iv)(B). Systems to which the requirements of 10CFR50.73(a)(2)(iv)(A) apply for this event include the Reactor Protection System (RPS) including RT and AFWS actuation. This event meets the reporting criteria because an automatic RT was initiated at 06:23 hours, on May 28, 2009, and the AFWS actuated as a result of the RT. The RT did not result in the failure of any primary system to function properly. Therefore, there was no safety system functional failure reportable under 10CFR50.73(a)(2)(v). On May 28, 2009, at 07:12 hours, a 4-hour non-emergency notification was made to the NRC for an actuation of the reactor protection system while critical and included an 8-hour notification under 10CFR50.72(b)(3)(iv)(A) for a valid actuation of the AFW System (Event Log # 45098).
Past Similar Events
A review was performed of the past three years of Licensee Event Reports (LERs) for unit 3 events that involved a RT from a MBFP failure or malfunction of a FW FCV.
One potential LER was identified. LER-2007-001 reported a manual RT due to decreasing SG levels as a result of the loss of FW flow caused by the failure of the 32 MBFP Train A control logic power supply. The cause of this event was a failed auctioneered power supply for the Lovejoy control system for MBFP control. The cause of the event reported in LER-2007-001 was not the same as this event therefore the corrective actions would not have prevented this event
Safety Significance
This event had no effect on the health and safety of the public.
There were no actual safety consequences for the event because the event was an uncomplicated reactor trip with no other transients or accidents. Required primary safety systems performed as designed when the RT was initiated. The AFWS actuation was an expected condition as a result of low SG water level due to SG void fraction (shrink), which occurs after a RT and main steam back pressure as a result of the rapid reduction of steam flow due to turbine control valve closure.
There were no significant potential safety consequences of this event under reasonable and credible alternative conditions. A RT and the increase in SG level is a condition for which the plant is analyzed. This event was bounded by the analyzed event described in FSAR Section 14.1.10, "Excessive Heat Removal Due to Feedwater System Malfunctions.
Excessive FW additions are an analyzed event postulated to occur from a malfunction of the FW control system or an operator error which results in the opening of a FW control valve. The analysis assumes one FW valve opens fully resulting in the excessive FW flow to one SG. For the FW system malfunction at full power, the FW flow resulting from a fully open control valve is terminated by the SG high level signal that closes all FW control valves and trips the MBFPs and the main turbine. A TT initiates a RT. The analysis for all cases of the excessive FW addition initiated at full power conditions with and without automatic rod control, show that the minimum DNBR remains above the applicable safety analysis DNBR limit, the primary and secondary side maximum pressures are less than 110% of the design values, and all applicable Condition II acceptance criteria are met. For this event, rod control was in automatic and all rods inserted upon initiation of the RT. The AFWS actuated and provided required FW flow to the SGs. RCS pressure remained below the set point for pressurizer PORV or code safety valve operation and above the set point for automatic safety injection actuation. Following the RT, the plant was stabilized in hot standby.