05000483/LER-2009-002

LER-2009-002, TDAFP Failed to Start During Surveillance Test

Docket Number Sequential Revmonth Day Year Year Month Day Yearnumber No.
Event date:	05-25-2009
Report date:	11-05-2009
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v)(B), Loss of Safety Function - Remove Residual Heat 10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident 10 CFR 50.73(a)(2)(v), Loss of Safety Function 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition
Initial Reporting
4832009002R01 - NRC Website

A. REPORTABLE EVENT CLASSIFICATION

10CFR50.73(a)(2)(i)(B) requires reporting of any operation or condition prohibited by a plant's Technical Specifications (TS). Following a failed surveillance test on 25 May 2009, the Turbine-Driven Auxiliary Feedwater Pump (TDAFP) was determined to have been inoperable longer than permitted by the Required Action for restoring an inoperable TDAFP per the associated TS Limiting Condition of Operation (LCO). Consequently, Callaway was in a condition prohibited by Technical Specifications.

10CFR50.73(a)(2)(v) requires the reporting of a condition that could have prevented the fulfillment of a safety function. The Train A Motor-Driven AFW Pump (MDAFP) was briefly removed from service for scheduled surveillance in the same period the TDAFP was later determined to have been inoperable.

Inoperability of two Auxiliary Feedwater (AFW) trains meets the criteria for a condition that could have prevented fulfillment of a safety function. Specifically, this condition is reportable under 10CFR50.73(a)(2)(v) Criterion B for a system needed to remove residual heat and 10CFR50.73(a)(2)(v) Criterion D for a system needed to mitigate the consequences of an accident.

Furthermore, when the failure of a MDAFP is postulated as an assumed single limiting failure in conjunction with the inoperability of the TDAFP, only a single MDAFP would be available for mitigation of a Feedline Break (FLB). Safety acceptance criteria for the FLB analysis would not be met in this scenario. Consequently, the extended period of time that the TDAFP was determined to be inoperable represents an unanalyzed condition that significantly degrades plant safety and is therefore reportable under 10CFR50.73(a)(2)(ii)(B) as discussed in Section 2.C.

B. PLANT OPERATING CONDITIONS PRIOR TO THE EVENT

The plant was in MODE 1, Power Operation, at 100% reactor power.

START OF THE EVENT AND THAT CONTRIBUTED TO THE EVENT

No structures, systems, or components were inoperable at the start of the event which contributed to the event.

D. NARRATIVE SUMMARY OF THE EVENT, INCLUDING DATES AND APPROXIMATE TIMES

At 1123 on 25 May 2009, the TDAFP [EllS system: BA, component: P] was taken out of service for scheduled surveillance of the AFW Actuation System [El IS system: JE]. This requires entry into Condition C of TS LCO 3.7.5.

Callaway TS 3.7.5 requires three Auxiliary Feedwater (AFW) trains to be Operable in MODES 1, 2 and 3.

When one AFW train is declared inoperable for a reason other than loss of steam supply or loss of essential service water, Required Action C.1 of TS LCO 3.7.5 requires the Operability of that AFW train to be restored within a Completion Time of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />.

During this surveillance, the slave relay performed as expected, thus verifying the actuation capability of the Turbine-Driven Auxiliary Feedwater Actuation Signal (AFAS). Both steam supply valves to the TDAFP turbine opened as expected. However, the Trip & Throttle Valve (TTV) [El IS system: BA, component: HCV; Callaway designator: FCHVO312] did not open. As a result, the steam flow path to the TDAFP turbine remained closed, and the TDAFP did not start. The pump was declared inoperable and non-functional, and immediate action was taken to investigate and repair the valve. The surveillance procedure was performed again with the same result, and the turbine control relays were visually verified to operate correctly. The slave relay electrical circuitry was confirmed to be functional.

TTV FCHVO312 is closed by a spring mechanism. The valve actuator is designed to open the valve from a normally closed and unlatched position. Upon receiving a start signal, the actuator rotates the spindle, causing the sliding nut and latch lever to travel up the spindle in the closed direction. The rising sliding nut and latch lever compress the spring mechanism until the valve latches to the trip lever. The closed torque switch is set to trip once the valve is latched. The actuator then reverses direction and opens the valve until stopped by a limit switch, permitting steam to flow to the TDAFP turbine.

Visual inspection of the TTV confirmed that the actuator of the TTV had moved approximately 70% towards the latched position but had stopped prior to latching. The torque switch was found in the open position. This indicates the actuator experienced excessive drag during operation, causing the torque switch to open and de-energize the actuator motor. The sliding nut and screw spindle squeaked audibly during manual operation of the valve.

After the spindle and sliding nut of the TTV were lubricated, the TDAFP was successfully started with valve FCHVO312 operating as designed. Following these repair actions, the slave relay test was ultimately completed at 2256 on 25 May 2009. At that time, the TDAFP was declared Operable, and Condition C of TS LCO 3.7.5 was exited.

The TTV is stroked at least two times each quarter during surveillance testing, and all surveillance tests between the start of Cycle 17 on 6 November 2008 and 4 May 2009 were completed satisfactorily. The valve became inoperable between the 4 May 2009 and 25 May 2009 surveillances. The exact time it became inoperable cannot be determined, but it is reasonable to assume that valve FCHVO312 was inoperable for greater than the 72-hour Completion Time of Required Action C.1 of TS LCO 3.7.5.

Because the TTV did not latch and open properly, the TDAFP would not have been able to perform its automatic and remote manual safety-related functions. The TDAFP could have been placed into service with local manual operation of FCHVO312. In this case, an Operator would manually open the TTV, thereby supplying steam to the TDAFP turbine.

A scheduled surveillance was performed on the Train A MDAFP [EllS system BA; component P] during the period the TDAFP was assumed, but unknown, to be inoperable. Between 2023 and 2130 on 24 May 2009, the Train A MDAFP was inoperable and not functional for valve testing. In addition, from 1023 to 1110 on 25 May 2009, the Train A MDAFP was inoperable but functional for slave relay testing and pump surveillance testing.

Removing the Train A MDAFP from service with another AFW train inoperable would have required entry into TS LCO 3.7.5 Condition D for two inoperable AFW trains. The Actions of this Condition require entry into MODE 3, Hot Standby, within six hours and entry into MODE 4, Hot Shutdown, within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. The times(s) when Condition D should have been in effect was much less than the 6-hour completion time for entry into MODE 3. However, concurrent inoperability of two of the three AFW trains meets the criteria for a condition that could have prevented fulfillment of a safety function. Further, the extended inoperability of the TDAFP, considered in conjunction with an assumed single failure of a MDAFP, represents an unanalyzed condition that significantly degrades plant safety. These reportability considerations, loss of safety system function and an unanalyzed condition that degrades plant safety, are further addressed in Section 2.0 of this LER.

E. METHOD OF DISCOVERY OF EACH COMPONENT, SYSTEM FAILURE, OR PROCEDURAL ERROR

The failure of the TDAFP TTV was discovered during surveillance testing of the Train A Auxiliary Feedwater Actuation System Slave Relays. Concurrent inoperability of two AFW trains was discovered upon post-event reviews of the TDAFP TTV failure.

An inspection of FCHVO312 performed on 9 September 2009 revealed an assembly error in the coupling for FCHVO312. This valve assembly error was a result of a failure to adhere to written instructions. This assembly error is discussed further in Section 3.

2.� EVENT DRIVEN INFORMATION

A. SAFETY SYSTEMS THAT RESPONDED

No safety systems responded to these conditions. No safety systems were required to respond to these conditions.

B. DURATION OF SAFETY SYSTEM INOPERABILITY

Due to the failure of FCHVO312, the TDAFP became inoperable at a time between the 4 May 2009 and 25 May 2009 surveillances, a duration of at most 21 days. As stated, the exact time and date at which TDAFP became inoperable cannot be determined, but it is reasonable to assume that the duration of inoperability was greater than the 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> permitted by Required Action C.1 of TS LCO 3.7.5.

The valve had satisfactorily completed all surveillance tests between the beginning of Cycle 17 operation (6 November 2008) and 4 May 2009. The TDAFP is considered to have been in a degraded but operable condition for that period.

The Train A MDAFP was inoperable for testing for 67 minutes on 24 May 2009 and for 47 minutes on 25 May 2009. With the TDAFP inoperable during those tests, two AFW trains were simultaneously inoperable for a combined 114 minutes, or 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and 54 minutes.

C. SAFETY CONSEQUENCES AND IMPLICATIONS OF THE EVENT

The inoperability of the TDAFP in conjunction with an actual or assumed failure or unavailability of a MDAFP would adversely impact the mitigation capability or assumed plant response to any of several postulated events described in Section 15.2 of the Callaway Final Safety Analysis Report (FSAR). The Loss of Normal Feedwater (LONF), Loss of Non-Emergency AC Power (LOAC) and FLB events credit the automatic delivery of AFW within 135 seconds of event initiation. Although manual operator actions would recover the inoperable TDAFP, initiation of AFW flow would be delayed relative to the time credited for automatic initiation of the TDAFP in the Licensing Bases analyses. The manual recovery action would be sufficient to prevent core damage for these sequences. However, further assessment (as summarized below) is necessary to determine the impact of the TDAFP inoperability on the licensing bases analyses documented in FSAR Section 15.2. In addition, the TDAFP is credited to provide secondary-side heat sink inventory during Small Break Loss of Coolant Accident (SBLOCA) sequences described in Section 15.6 of the Callaway FSAR. Further assessment of the impact of the TDAFP inoperability on SBLOCA sequences, as well as on Station Blackout (SBO) response, is provided below.

LOSS OF NORMAL FEEDWATER AND LOSS OF NON-EMERGENCY AC POWER Should a postulated LONF/LOAC event occur with the TDAFP inoperable, a single failure could leave the plant reliant upon a single MDAFP to provide secondary side inventory and heat sink. This condition is outside of the licensing bases analysis of record. During steam generator replacement, a best-estimate analysis of the LONF/LOAC events was performed that demonstrates that a single MDAFP is sufficient for these events when better-estimate inputs are used. Based on the best­ estimate analysis, it can be concluded that the extended inoperability of the TDAFP did not represent an unanalyzed condition that significantly degraded plant safety for the LONF/LOAC sequences.

FEEDLINE BREAK

With regard to a postulated FLB, if both MDAFPs are available, applicable acceptance criteria for the FLB transient can be met. However, if the failure of a MDAFP is postulated as an assumed single limiting failure in conjunction with inoperability of the TDAFP, the safety analysis acceptance criteria for the FLB analysis cannot be met. Specifically, one of the acceptance criteria in the analysis of record is to ensure that hot leg saturation is precluded. Should the plant be reliant on a single MDAFP following a FLB, hot leg saturation would not be precluded. Therefore, the extended inoperability of the TDAFP would represent an unanalyzed condition that significantly degrades plant safety for the FLB accident sequence.

SMALL BREAK LOSS OF COOLANT ACCIDENT

The TDAFP is credited to provide secondary side heat sink inventory during SBLOCA accident sequences. Although this credit is taken during the quasi-equilibrium phase that occurs prior to clearing the loop seal in the SBLOCA sequence, the SBLOCA response is generally not impacted by changes in AFW flow. Even if only one MDAFP (delivering to two steam generators) is available, the effect on the SBLOCA analysis results can be shown to be negligible. Therefore, extended inoperability of the TDAFP would not represent an unanalyzed condition that significantly degraded plant safety for the SBLOCA sequence.

STATION BLACKOUT

The inoperability of the TDAFP would impact the response to a Station Blackout event. There was approximately an 80-minute supply of water in the steam generators at the time this condition was discovered. Testing has shown that Operations is able to manually start the TDAFP in 20 minutes.

This ensures adequate water is supplied to the steam generators to maintain the integrity of the reactor coolant system and for shutdown of the reactor.

With regard to risk significance, since AFW flow would be recovered prior to depletion of the initial inventory, risk evaluations have found that the condition addressed in the LER did not pose an unacceptable increase in core damage frequency. However, based on the above-described impact of an inoperable TDAFP on the FLB analysis, assuming the concurrent inoperability of a MDAFP (as an assumed additional single failure), the extended inoperability of the TDAFP constituted an unanalyzed condition that significantly degraded plant safety. Further, for the periods when both the TDAFP and the Train A MDAFP were simultaneously inoperable, this constituted a condition that could have prevented the fulfillment of safety functions (removing residual heat and mitigating the consequences of an accident) based on the requirement for two MDAFPs to mitigate an FLB accident.

3. CAUSE(S) OF THE EVENT AND CORRECTIVE ACTION(S)

After the original Root Cause Analysis was completed and LER 2009-002-00 was submitted, a second Root Cause Analysis team was formed on 1 September 2009 to reevaluate the root causes for this event.

This analysis was aided by a planned maintenance outage for the TDAFP that allowed further inspection of FCHVO312. The findings of this second Root Cause Analysis are summarized in this section.

The TDAFP failed to start on 25 May 2009 because valve FCHVO312 did not open. The failure of FCHVO312 to open was caused by excessive friction which resulted in the opening of the closed torque switch. Causal Factors for this event include the loss of lubrication effectiveness and an assembly error of the TTV. The three Root Causes associated with these Causal Factors and the associated Corrective Actions to Prevent Recurrence (CATPR) are discussed below.

INADEQUATE PREVENTATIVE MAINTENANCE FREQUENCY

One Root Cause of the loss of lubrication effectiveness has been identified as an inadequate existing Preventative Maintenance (PM) Basis frequency for routine TTV lubrication, based on the component operating temperatures and environmental conditions.

The PM basis for TTV lubrication frequency extensions had been based primarily on equipment out-of- service time, not on the specific combination of grease properties and operating conditions. The FCHVO312 coupling temperature has been measured at 300 degrees F, approaching the 350 degrees F temperature limit of the grease used, Mobilgrease 28. Although the operating temperature for the FCHVO312 coupling is within the temperature limit of Mobilgrease 28, Callaway should have considered this operating temperature when evaluating stem lubrication frequencies. Contributing to the event is the 18-24 month TTV lubrication frequency suggested in the Electric Power Research Institute (EPRI), Terry Turbine Maintenance Guide, AFW Application, TR 1007461, Final Report, November 2002, which may not be applicable when the steam supply exists up to the TTV, which is the case with FCHVO312.

The CATPR for this Root Cause is to develop an appropriate PM frequency for TTV lubrication based on grease type and system operating conditions. Additional actions include the approval of a more effective high-temperature grease, using a more rigorous process to review PM Basis changes, and providing training to the personnel who establish PM Basis frequency.

FAILURE TO PERFORM THE PROCEDURE SECTION DIRECTING VALVE LUBRICATION

The second Root Cause of the loss of lubrication effectiveness has been identified as the failure to perform Section 6.3 of procedure MPM-FC-QK001, Auxiliary Feedwater Pump Turbine Annual Inspection, during the last refueling outage, Refuel 16 (RF16). Section 6.3 of MPM-FC-QK001 directs the lubrication of FCHVO312. As stated in LER 2009-002-00, during performance of this procedure in RF16, Maintenance personnel believed the intent of Section 6.3 had been satisfied previously and did not perform Section 6.3 as scheduled. The TTV installed in RF16 had most recently been lubricated upon being rebuilt in September 2007.

Two CATPRs have been developed for this Root Cause. The first CATPR is to revise the work instructions for the replacement of FCHVO312 to verify the performance of lubrication requirements by sign-off as a critical step. The second CATPR is to develop standalone PMS for performing the necessary valve lubrication of FCHVO312, rather than having the lubrication steps contained within the TDAFP turbine inspection activities. These standalone PMS will provide specific direction regarding the application method, quantities and location of grease.

INCORRECT INSTALLATION OF THE THRUST WASHER FOR FCHVO312 In addition to the Root Causes associated with the lack of lubrication effectiveness, an assembly error of the TTV was also identified as a Root Cause of the event. In the TDAFP inspection performed on 9 September 2009, the thrust washer on the TTV was discovered to have been installed incorrectly.

When installed correctly, the beveled side of the thrust washer is oriented 'down' in order to avoid interference between the washer and the bottom edge of the valve coupling. Contrary to procedure instructions and vendor drawings, the beveled side was found oriented 'up'. This incorrect orientation may have caused an increased amount of galling between the washer and the spindle end, contributing to the overall resistance imposed on the valve actuator prior to latching. A new thrust washer was correctly installed during the September 2009 maintenance outage.

The installation of the thrust washer with the beveled side of the washer in the incorrect orientation is contrary to work instructions. The instructions for thrust washer installation contain adequate guidance for the correct orientation of the thrust washer. However, these instructions were not followed as written.

A Root Cause Analysis had been performed in 2008 to address deficiencies regarding work instruction use and adherence. The CATPRs from this 2008 analysis were not in effect at the time the thrust washer was installed, but have since been implemented. Additional actions to improve work instructions for the FCHVO312 rebuild have been identified and implemented.

In addition to the Causal Factors associated with the three Root Causes discussed above, a third Causal Factor identified that improvement is needed in organizational focus regarding AFW work activities. No Root Cause or CATPR is associated with this Causal Factor, but contributing causes were identified. These contributing causes include imprecise communication that occurred between Maintenance supervision and Engineering personnel in the assignment and completion of work associated with this event. Maintenance, Engineering and Planning personnel were briefed to improve the quality of these communications. In addition, unlike some other risk significant systems, the AFW system did not have a dedicated work coordinator for refueling outages. Dedicated work coordinator(s) will be assigned for major AFW system activities during refueling outages in the future.

4. PREVIOUS SIMILAR EVENTS

Prior to 1995, Nebula EP-1 grease was used which resulted in grease hardening and poor valve performance. Fuses in the FCHVO312 control circuit were opening as a result of valve binding due to grease hardening. Callaway changed from Nebula EP-1 grease to Mobilgrease 28, a higher temperature rated grease, in 1995. Since that time, while lubrication frequencies have changed, there have been no documented valve issues related to lubrication concerns until this event on 25 May 2009.

5. ADDITIONAL INFORMATION

Callaway is designed to have only one TDAFP and one TTV. The TDAFP and its associated TTV are unique components in regard to their use and function at Callaway Plant. FCHVO312 is the only Gimpel valve installed in Callaway.

Even though FCHVO312 is a uniquely designed valve, it can reasonably be considered to be within the rising stem Motor Operated Valve (MOV) category. Typical rising stem MOVs at Callaway undergo specific MOV testing to determine the stem factor, which is an indication of the effectiveness of the torque to thrust conversion. However, MOV testing on FCHVO312 is substantially different than testing on other rising stem valves. Due to the construction of FCHVO312, actuator output torque or thrust cannot be directly measured in the close direction.

For the typical rising stem MOVs, the program engineer has a trending database that tracks changes in the coefficient of friction between the stem and stem nut of the valve. Any adverse trends in the measured stem factor are identified and actions taken to promptly correct lubrication issues before they impact valve performance.

Trip & Throttle Valve FCHVO312 is a motor-operated, 4-inch, partially balanced globe valve.

Manufacturer: Gimpel Machine Works (EPIX code G153) Model Number: JO#74-12238 83-14239.

The system and component codes listed below are from the IEEE Standard 805-1984 and IEEE Standard 803A-1984 respectively.

System: JE, Engineered Safety Features Actuation System System: BA, Auxiliary/Emergency Feedwater System (PWR) Component: HCV, Hand Control Valve System: BA, Auxiliary/Emergency Feedwater System (PWR) Component: P, Pump