IR 05000483/2011007

IR 05000483-11-007, on 03/21-05/13/11, Callaway Plant, Special Inspection for Turbine-driven Auxiliary Feedwater Pump Bearing Degradation

ML111780808
Person / Time
Site:	Callaway
Issue date:	06/27/2011
From:	Geoffrey Miller NRC/RGN-IV/DRP/RPB-B
To:	Heflin A Union Electric Co
References
IR-11-007
Download: ML111780808 (35)

Text

UNITED STATES NUC LE AR RE G UL AT O RY C O M M I S S I O N R E GI ON I V 612 EAST LAMAR BLVD , SU I TE 400 AR LI N GTON , TEXAS 76011-4125 June 27, 2011 Mr. Adam C. Heflin, Senior Vice President and Chief Nuclear Officer Union Electric Company P.O. Box 620 Fulton, MO 65251 Subject: CALLAWAY PLANT - NRC SPECIAL INSPECTION REPORT 05000483/2011007

Dear Mr. Heflin:

On May 13, 2011, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Callaway Plant on the circumstances surrounding discovery of a degraded bearing condition of the turbine-driven auxiliary feedwater pump on February 8, 2011. Based on the risk and deterministic criteria specified in NRC Management Directive 8.3, NRC Incident Investigation Program, including possible generic implications, the NRC initiated a special inspection in accordance with Inspection Procedure 93812, Special Inspection. The basis for initiating the special inspection and the focus areas for review are detailed in the special inspection charter (Attachment 2). The determination that the inspection would be conducted was made by the NRC on February 24, 2011, and the onsite inspection started on March 21, 2011. The enclosed report documents the inspection findings that were discussed on May 13, 2011, with you and members of your staff.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures, root cause investigation and records, observed activities, and interviewed plant personnel. The turbine-driven auxiliary feedwater pump was found to have a degraded thrust bearing due to running below minimum flow requirements. A low speed and low flow postmaintenance test procedure did not adequately capture vendor required minimum flow requirements. The test procedure did have a minimum flow precaution but the inspectors determined this was not followed during performance of the procedure. The bearing was replaced and an evaluation of its condition determined that the degraded condition of the bearing would have supported the required safety function of the pump. The inspection team reviewed your corrective actions and determined that, if properly implemented; they will enhance the continued safe operation of the Callaway Plant and prevent recurrence of testing the pump below minimum flow requirements.

This report documents six NRC-identified findings of very low safety significance (Green). All six of these findings were determined to involve violations of NRC requirements. However, because of their very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations, consistent with Section 2.3.2 of the NRC Enforcement Policy. If you contest the violations or the significance of the noncited violations, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission,

Union Electric Company -2-ATTN: Document Control Desk, Washington, D.C. 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 612 E. Lamar Blvd, Suite 400, Arlington, Texas, 76011-4125; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at the Callaway Plant. In addition, if you disagree with the cross-cutting aspect assigned to any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Callaway Plant. The information you provided will be considered in accordance with Inspection Manual Chapter 0305.

In accordance with 10 CFR 2.390 of the NRC's Rules of Practice, a copy of this letter, its enclosures, and your response if you choose to provide one, will be made available electronically for public inspection in the NRC Public Document Room or from the NRCs document system (ADAMS), accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html. To the extent possible, your response should not include any personal privacy or proprietary, information so that it can be made available to the public without redaction.

Sincerely,

/RA/

Geoffrey Miller, Chief Project Branch B Division of Reactor Projects Docket: 50-483 License: NPF-30 Enclosure: NRC Inspection Report 05000483/2011007 w/Attachments:

Attachment 1: Supplemental Information Attachment 2: Special Inspection Charter Attachment 3: Timeline CC: Distribution via Listserv

Union Electric Company -3-Electronic distribution by RIV:

Regional Administrator (Elmo.Collins@nrc.gov)

Deputy Regional Administrator (Art.Howell@nrc.gov)

DRP Director (Kriss.Kennedy@nrc.gov)

Acting DRP Deputy Director (Jeff.Clark@nrc.gov)

DRP Deputy Director (Troy.Pruett@nrc.gov)

DRS Director (Anton.Vegel@nrc.gov)

DRS Deputy Director (Thomas.Blount@nrc.gov)

Senior Resident Inspector (David.Dumbacher@nrc.gov)

Resident Inspector (Jeremy.Groom@nrc.gov)

Branch Chief, DRP/B (Geoffrey.Miller@nrc.gov)

Senior Project Engineer, DRP/B (Rick.Deese@nrc.gov)

Senior Project Engineer, DRP/B (Leonard.Willoughby@nrc.gov)

Project Engineer, DRP/B (Greg.Tutak@nrc.gov)

Project Engineer, DRP/B (Nestor.Makris@nrc.gov)

CWY Administrative Assistant (Dawn.Yancey@nrc.gov)

Public Affairs Officer (Victor.Dricks@nrc.gov)

Public Affairs Officer (Lara.Uselding@nrc.gov)

Project Manager (Mohan.Thadani@nrc.gov)

Acting Branch Chief, DRS/TSB (Dale.Powers@nrc.gov)

RITS Coordinator (Marisa.Herrera@nrc.gov)

Regional Counsel (Karla.Fuller@nrc.gov)

Region IV State Liaison Officer (Bill.Maier@nrc.gov)

Congressional Affairs Officer (Jenny.Weil@nrc.gov)

OEMail Resource RIV/ETA:OEDO (Stephanie.Bush-Goddard@nrc.gov)

ROPreports R:_REACTORS\_CW\2011\CAL2011007-RP-MLC.doc ADAMS: No 7Yes 7 SUNSI Review Complete Reviewer Initials: RWD 7 Publicly Available 7 Non-Sensitive Non-publicly Available Sensitive RIV:RI/DRP/C NRR:PM/DORL C:DRP/B MChambers JPolickoski GMiller

/RA/ /RA/ /RA/

6/24/2011 6/26/2011 6/27/2011 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax

U.S. NUCLEAR REGULATORY COMMISSION

REGION IV

Docket: 05000483 License: NPF-30 Report: 05000483/2011007 Licensee: Union Electric Company Facility: Callaway Plant Location: Junction Highway CC and Highway O Fulton, MO Dates: March 21 through May 13, 2011 Inspectors: M. Chambers, Resident Inspector, Cooper Nuclear Station D. Dumbacher, Senior Resident Inspector, Callaway Plant L. Micewski, Resident Inspector, Diablo Canyon Power Plant J. Polickoski, Project Manager, NRR M. Runyan, Senior Reactor Analyst Approved By: G. Miller, Chief, Project Branch B Division of Reactor Projects 1 Enclosure

SUMMARY OF FINDINGS

IR 05000483/2011007; 03/21-05/13/11; Callaway Plant; Special inspection for turbine-driven auxiliary feedwater pump bearing degradation.

The report covered one week of onsite inspection and in-office review through May 13, 2011. A resident inspector and project manager performed the inspection with assistance from a project engineer and a senior reactor analyst. Six Green noncited violations were identified. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process,

Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

Green.

The inspectors identified a noncited violation of Technical Specification 5.4.1.a involving a failure to follow procedures in the development of Procedure OTS-FC-0006, TDAFW Pump Post-Maintenance Test Run on Aux Steam. Specifically, the licensee failed to incorporate turbine-driven auxiliary feedwater pump vendor manual precautions, limitations, and technical information in Procedure OTS-FC-0006, which resulted in the axial unloading, rolling element ball skidding, and subsequent degradation to the turbine-driven auxiliary feedwater pump inner outboard thrust bearing. Following discovery during planned maintenance and as immediate corrective actions, the licensee declared the turbine-driven auxiliary feedwater pump inoperable, entered the applicable Technical Specification Limiting Condition for Operation, replaced the oil and bearings, restored the pump to operability, and initiated Callaway Action Request 201101042 to perform a root cause analysis.

This finding is more than minor because it affected the equipment performance attribute of the Mitigating Systems Cornerstone and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. The finding was evaluated using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, and was determined to be of very low safety significance (Green) because there was not a design or qualification deficiency that resulted in a loss of operability or functionality, it did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time, it did not represent an actual loss of risk significant equipment, and it did not affect seismic, flooding, or severe weather initiating events. This finding has a cross-cutting aspect in the area of human performance associated with the work practices component because the licensee failed to ensure procedural adherence in the establishment of the turbine-driven auxiliary feedwater pump postmaintenance test procedure H.4(b)(Section 4.0).

Green.

The inspectors identified a noncited violation of Technical Specification 5.4.1.a involving five examples of failure to follow

Procedure OTS-FC-0006, TDAFW Pump Post-Maintenance Test Run on Aux Steam. Specifically, operators failed to follow an existing total flow precaution in Procedure OTS-FC-0006 which resulted in the axial unloading, rolling element ball skidding, and subsequent degradation to the turbine-driven auxiliary feedwater pump inner outboard thrust bearing. Following initial condition discovery during planned maintenance and as immediate corrective actions, the licensee declared the turbine-driven auxiliary feedwater pump inoperable, entered the applicable Technical Specification Limiting Condition for Operation, replaced the oil and bearings, restored the pump to operability, and initiated Callaway Action Request 201101042 to perform a root cause analysis.

These findings were more than minor because they affected the equipment performance attribute of the Mitigating Systems Cornerstone and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences.

The finding was evaluated Using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings. These findings were determined to be of very low safety significance (Green) because there was not a design or qualification deficiency that resulted in a loss of operability or functionality, they did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time, they did not represent an actual loss of risk significant equipment, and they did not affect seismic, flooding, or severe weather initiating events. This finding has a cross-cutting aspect in the area of human performance associated with the work practices component because the licensee failed to ensure procedural adherence in the implementation of the turbine-driven auxiliary feedwater pump postmaintenance test procedure H.4(b)(Section 4.0).

Green.

The inspectors identified a noncited violation of 10 CFR Part 50,

Appendix B, Criterion V, Instructions, Procedures, and Drawings, regarding the licensees failure to follow the requirements of Procedure APA-ZZ-00500,

Appendix 3, Past Operability and Reportability Evaluations. Specifically, the inspectors identified that the past operability evaluation for the turbine-driven auxiliary feedwater pump used a nonconservative calculation of mission time that did not take into account all design and licensing basis functions when determining the mission time. The licensee entered this issue into their corrective action program as Callaway Action Request 201102431 and updated its mission time analysis to account for the turbine-driven auxiliary feedwater pumps specified safety function to bring the plant to a safe shutdown condition.

This finding is greater than minor because if left uncorrected, it would have the potential to lead to a more significant safety concern because systems that may be inoperable may not be recognized and that it impacted the Mitigating Systems Cornerstone attribute of human performance in that the failure to accurately understand the auxiliary feedwater system mission time affected the mitigating systems objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the finding is determined to have very low safety significance because it did not result in the loss of safety function of any technical specification required equipment. The cause of this finding is related to the problem identification and resolution cross-cutting component of corrective action program because licensee personnel failed to thoroughly evaluate conditions adverse to quality and perform adequate operability determinations P.1(c)(Section 4.0).

Green.

The inspectors identified a noncited violation of 10 CFR Part 50,

Appendix B, Criterion III, Design Control, for a failure to adequately determine safety related equipment oil leakage acceptance criteria that was used in operator logs. Specifically, the 2008 licensee fluid leak management program calculations to determine the mission time assessments related to oil leak rates of safety related pumps and motors were nonconservative when added to Procedure ODP-ZZ-0016E, Appendix 1, Equipment Operator General Inspection Guide. The licensee evaluated this issue in Callaway Action Request 201102431 and calculated new conservative oil leak rates for the affected equipment.

This finding is more than minor because if left uncorrected it has the potential to lead to a more significant safety concern. Specifically, the failure to adequately evaluate and determine an appropriate lube oil leak rate to maintain safety related equipment operability affects the equipment performance attribute of the Mitigating Systems Cornerstone and could have impacted the availability of mitigating equipment if left uncorrected. The finding was evaluated using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, and determined to be of very low safety significance since the as-found condition of the safety related equipment reviewed back to August 15, 2007, found no oil leakage rates that would have caused a loss of system safety function. This finding was not reflective of current licensee performance and therefore, has no cross-cutting aspect (Section 4.0).

Green.

The inspectors identified a noncited violation of Technical Specification 5.4.1 for a failure to adequately establish and implement procedures required by Regulatory Guide 1.33, Appendix A, Section 9,

Procedures for Performing Maintenance. Specifically, the preventative maintenance schedule to perform periodic lube oil analysis established an 18 month frequency without adequate justification resulting in a failure to promptly detect bearing degradation in the turbine-driven auxiliary feedwater pump. The licensee evaluated this issue in Callaway Action Request 201101042 and has corrective actions to review the lube oil analysis frequency and reduce it to at least a 9 month frequency.

This finding is more than minor because if left uncorrected it has the potential to lead to a more significant safety concern. Specifically, the failure to adequately evaluate and determine an appropriate lube oil monitoring schedule resulted in the failure to promptly detect a degraded bearing in the turbine-driven auxiliary feedwater pump affecting the equipment performance attribute of the Mitigating Systems Cornerstone and could have impacted the availability of mitigating equipment if left uncorrected. The finding was evaluated using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, and determined to be of very low safety significance since the as-found condition of the degraded bearing would not have caused a loss of system safety function.

The finding has a cross-cutting aspect in the area of human performance associated with the decision making component, in that, the licensee failed to use conservative assumptions in the decision to extend the turbine-driven auxiliary feedwater pump lube oil monitoring interval to 18 months H.1(b)(Section 4.0).

Green.

The inspectors identified a noncited violation of 10 CFR Part 50,

Appendix B, Criterion XVI, Corrective Action, associated with the licensees failure to promptly identify and correct a condition adverse to quality.

Specifically, the licensee reduced the scope of preventative maintenance for the turbine-driven auxiliary feedwater pump overhaul during Refueling Outage 16 without proper justification, resulting in the failure to perform required pump maintenance. This issue was entered into the licensees corrective action program as Callaway Action Request 201102407 and the pump has been scheduled to be overhauled during the next refueling outage.

This finding is more than minor because, if left uncorrected, corrective action deficiencies would have the potential to lead to a more significant safety concern.

The failure to perform required maintenance could allow equipment degradation affecting the equipment performance attribute of the Mitigating Systems Cornerstone and could have impacted the availability of mitigating equipment if left uncorrected. Using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, this finding was determined to be of very low safety significance because the degraded condition did not result in a loss of operability or functionality. The inspectors determined that the finding has a cross-cutting aspect in the area of human performance associated with the work control component because the licensee does not appropriately coordinate work activities by incorporating actions to address the impact of changes to work scope on the plant such that nuclear safety is supported H.3(b)(Section 4.0).

Licensee-Identified Violations

None.

REPORT DETAILS

1.0 Special Inspection Initiation On February 8, 2011, during planned performance of a lube oil sample, the turbine-driven auxiliary feedwater pump outboard bearing was found to have been degraded.

The degraded bearing could have challenged the capability of the pump to perform its safety related function. In accordance with Management Directive 8.3, NRC Incident Investigation Program, the NRC determined that a special inspection was warranted, in part, based on the potential safety significance and because of potential generic issues associated with improper lubrication of bearings.

The team conducted their reviews in accordance with NRC Inspection Procedure 93812, Special Inspection, including ensuring that the NRC response to the event was consistent with the potential safety significance. The charter for the special inspection is provided as Attachment 2.

2.0 Turbine-Driven Auxiliary Feedwater Pump Description The turbine-driven auxiliary feedwater pump is a steam-driven pump which utilizes a Terry turbine as a prime mover supplied by steam from steam generators B and C.

Upon initiation of an auxiliary feedwater actuation signal, the pump will start and feed all steam generators. The pump is a horizontally mounted six stage pump driven by a 1378 horsepower Terry steam turbine. The pump is rated at 1145 gpm, 3850 rpm, and a total head of 3450 feet of water at 95°F. The function of the pump is to provide water to the steam generators under accident conditions, especially when alternating current electric power is not available, such as in a station blackout situation. The pump is normally run for short periods about three times per quarter with this operation normally in a low flow (approximately 140 gpm) recirculation mode. The pump has a plain radial ball bearing at the inboard end (towards the turbine) and two angular contact bearings in a thrust bearing configuration at the outboard end. The outboard bearings are stacked back-to-back to provide thrust load carrying capacity in both axial directions (parallel to pump shaft).

Historically, the licensee has operated the pump via different methods. The following descriptions explain the difference between speed, pressure, and flow conditions when testing this pump.

Main steam driving the turbine (high speed) with the pump flow path to the steam generators (low system resistance) nets a result of the highest flow with a relatively low discharge pressure (i.e., lower than while only operating on a recirculation flowpath). This is the design flow path, but it is not tested at power since the test would introduce cold water into the steam generators (adding positive reactivity to the reactor).

Main steam driving the turbine (high speed) with the pump recirculation flow path to condensate storage tank nets a result of less flow and a higher discharge pressure than while feeding the steam generators. This is the normal flow path used for surveillance testing the pump.

Auxiliary boiler steam driving the turbine (low speed because of low steam pressure) or low speed testing with main steam driving the turbine with the recirculation flow path to the condensate storage tank nets the lowest flows and low discharge pressures.

3.0 Discovery of the Degraded Bearing On February 8, 2011, planned preventive maintenance Job 09509875 was scheduled to change and sample the oil contained in the bearing reservoirs for the turbine-driven auxiliary feedwater pump. The oil drained from the outboard pump bearing reservoir was noted to be degraded (very dark). This condition could have eventually challenged the capability of the pump to perform its required safety related function. This occurrence represented the third consecutive oil sample where dark oil and particulates had been found in the turbine-driven auxiliary feedwater pump outboard pump bearing housing, with the first event in September 2009 and the second event in May 2010.

The dark oil was attributed to wear metal contamination in the oil, and an initial apparent cause of incorrect oil level in the outboard bearing reservoir, was assigned. As a result, the licensee took immediate actions to clean the reservoir, ship the oil offsite for analysis, replace the outboard bearings, and slightly raise the outboard bearing housing oil level. Analysis of the darkened oil found high levels of iron, copper, aluminum, lead, and zinc. This initial cause was later evaluated not to be the root cause of the darkened oil and inspectors focused their inspection efforts away from this cause and towards the actual root cause.

The outboard bearing consists of two stacked angular thrust bearings. They were removed and inspected. Wear was found on the inner outboard bearings inside raceway, on the rolling elements, and on the cage roller pockets. The inner outboard bearing inner raceway wear and the rolling element cage were the sources of the wear metal debris that caused the dark oil. Both outboard bearings were replaced with new bearings. The worn turbine-driven auxiliary feedwater pump outboard bearings were shipped to the original equipment manufacturer for cause investigation. The vendor analysis of the bearings identified rolling element skidding, likely due to axial unloading of the inner outboard bearing, as the cause of the observed bearing degradation and darkened oil. During skidding, the lubrication film layer was insufficient to maintain proper clearance at the moving part contact points, resulting in metal to metal contact.

This metal to metal contact produced the wear particulates observed in the oil. From this point forward, the degradation mechanisms acting on the bearings were from particulate interaction with the rolling elements, heat generated by the metal to metal skidding, and the metal to metal skidding forces imposed on the bearing cage and raceways. The apparent cause was then revised from incorrect oil level to rolling element skidding.

Five starts occurred after the May 2010 event where the pump was operated at slow speed (1280-2320 rpm) per Procedure OTS-FC-00006, TDAFP Post-Maintenance Test Run on Auxiliary Steam. The total time duration of operation at slow speed in May and June 2010 was 213 minutes. The low speed and low flow testing resulted in running the pump below vendor recommended speeds and minimum flow which significantly contributed to the ball skidding degradation of the thrust bearing. The team compiled a complete sequence of events surrounding the darkened oil which is attached as 3 of this report.

4.0 Review of Root Cause Analysis, Extent of Condition, and Corrective Actions

a. Inspection Scope

The team reviewed the licensees root cause analysis to determine if it was conducted to a level of detail commensurate with the significance of the problem. As part of their review, the inspectors interviewed engineering and operations personnel. The team reviewed the licensees corrective actions to ensure they addressed the extent of condition and whether they were adequate to prevent recurrence. In particular, the team reviewed station procedures and processes and whether similar equipment could have been damaged by running below normal speeds.

The team interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program. Additionally, the team interviewed the root cause team members and the members from the licensees Corrective Action Review Board.

The team also evaluated the events leading to and the licensee response to the Callaway turbine-driven auxiliary feedwater pump bearing degradation. In order to review each area of the special inspection charter, the team reviewed calculations, design documents, licensing documents, work orders, modification packages, and corrective action documents. The team evaluated licensee compliance with the applicable regulatory requirements and applicable codes and standards. The team interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program. The team assessed licensee implementation of their corrective action program, design controls, and procedure implementation.

b. Observations and Findings

Observations The inspectors reviewed the licensee root cause analysis that utilized a seven step root cause analysis process as directed by Procedure APA-ZZ-00500, Corrective Action Program, Appendix 12, Significant Adverse Condition - Significance Level 1. The root cause analysis team developed an equipment fault tree to assist with determining failure modes for turbine-driven auxiliary feedwater pump oil degradation and an event and causal factor chart for the sequence and timeline of the events. Information was developed based on interviews, job history, vendor investigation, and condition report records. Causal factors were developed based on if removed or prevented, the incident would not have occurred or its consequences would be mitigated. Causes were developed from the identified applicable conditions. The licensee also used change analysis, hazard/barrier analysis, and safety culture component assessment methodologies to further determine the cause(s), as necessary.

The licensee determined that the most probable root cause of the event was an inadequate technical assessment in the approval of procedures that operated the turbine-driven auxiliary feedwater pump at low flow operation at low speed.

The corrective actions to prevent recurrence included deleting Procedure OTS-FC-00006, TDAFP Post-Maintenance Test Run on Auxiliary Steam, and revising Procedure ETP-FC-00001, Calibration of Terry Turbine Governor Valve Actuator. This was necessary to prevent pump degradation that occurred when the pump was operated at less than rated speed and flow rates. Procedure OTS-FC-00006 ran the turbine on auxiliary steam during low speed and flow operation in Mode 4.

Procedure ETP-FC-00001 was planned to be revised to incorporate criteria on flow rate and time limit acceptance criteria for running the pump for governor postmaintenance test without pump degradation.

Extent of Condition and Extent of

Cause.

The extent of a root cause of, inadequate technical assessment in the approval of procedures that operated the turbine-driven auxiliary feedwater pump at low flow operation at low speed, could apply to all licensee equipment. However, the extent review was limited to systems monitored by the mitigating systems performance indicators. These systems include the diesel generators, essential service water, component cooling water, chemical and volume control, residual heat removal, high pressure coolant injection, fuel oil and the auxiliary feedwater systems.

All the components in these systems are electric motor-driven at constant speed except the turbine drive of the turbine-driven auxiliary feedwater pump and the diesel generators. The turbine was included in the extent of cause due to its variable speed function. The licensee will review and revise the turbine operating procedures to limit low speed testing.

The extent of other causal factors includes the procedural noncompliance to the flow limitation during performance of the turbine-driven auxiliary feedwater pump Mode 4 testing at low speed and flow. Callaway Action Request 201101042 took credit for actions from two previous procedural noncompliance issue resolutions. The first was Callaway Action Request 200803101, which addressed deficiencies regarding work instruction use and adherence. The second was Callaway Action Request 201008004, which addressed an issue of supervisors and managers not consistently reinforcing the standards necessary to achieve higher levels of performance. The licensee credited corrective actions taken in Callaway Action Requests 200803101 and 201008004 as making significant improvements in this area and it continued to be an area of reinforcement with field observations. Errors in this area were trended on at least a quarterly basis with additional reinforcement taken when a negative trend developed.

Based on this progress and the error precursors surrounding the use of Procedure OTS-FC 00006 during Refueling Outage 17, the licensee concluded no further corrective action was necessary.

Enhancements and Other Issues. As part of their investigation, the licensee identified other issues and enhancements that would improve the reliability of the turbine-driven auxiliary feedwater that included:

• Revise the warehouse stock description for replacement thrust bearings to have replacement bearings issued and installed as paired sets, to ensure the obsolete and new model bearings, both approved for use in the pump are not mixed.

• Revise instructions and procedures to ensure that new bearings are prelubricated.

• Revise work instructions for bearing temperature limitations and cooldown requirements when heated for interference fit installations.

• Revise lessons learned into training to prevent silicon sealant used on pipe threads from being introduced into the bearing housing and the oil. Prior lube oil analysis history had found small amounts of silicon in the oil.

• Install sight glasses on the turbine- and motor-driven auxiliary feedwater pump bearing housings in conjunction with the constant level oilers. This will allow positive verification of bearing housing oil level at all times. The current configuration depends on the constant level oiler adjustment to set bearing housing oil level.

A minor violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, was identified by the inspection team for the licensees failure to follow work instructions during constant level oiler adjustment on the turbine-driven auxiliary feedwater pump bearings that resulted in the bearing housing oil level being above the maximum limit from September 2009 through May 2010. The violation is minor due to the high oil level not impairing adequate lubrication of the bearings.

• Delete the use of Alvania EP2 grease as an aid to hold the bearing retaining coil springs in place during pump bearing assembly. In May 2010 dark oil was found in the turbine-driven auxiliary feedwater pump outboard bearing with the cause due to first time use of grease to hold the retaining springs in place.

The inspectors identified a minor violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the licensees failure to perform an engineering evaluation prior to implementation of the first time use of Alvania EP2 grease to help retain the helical bearing retaining springs in place in the pump outboard bearing housing following maintenance performed September 2009. Callaway Action Request 201004714 identified the use of this grease as the apparent cause of degraded oil in this bearing housing from the September 2009 installation as discovered in May 2010. More specifically, the licensees work instructions for Job 09006170.500 completed on September 9, 2009, directed the use of a small amount of Alvania EP2 grease to help retain the helical springs within the holes on the turbine-driven auxiliary feedwater pump outboard bearing.

The use of this grease was not approved per an engineering evaluation and was not documented in the condition report system as required by plant procedures.

The grease was later determined to be incompatible with the Turbo 32 oil in the bearing housing and resulted in dark oil being discovered when the next sample was taken. Callaway Action Request 201004714 noted that use of Alvania EP2 grease to hold the springs in place during end cover installation (Job 09006170.500) was a first time practice and is not part of pump overhaul Procedure MTM-AL-QP002, Turbine-Driven Auxiliary Feedwater Pump. A corrective action removed the step that applied this grease from the work instructions for Job 09006449.600. As an immediate corrective action, the licensee cleaned the grease off the springs and bearing housing cover prior to reassembly in May 2010. The licensee initiated Callaway Action Request 201004714 to perform the apparent cause evaluation and remove Alvania EP2 grease from the pump overhaul job as described above, Callaway Action Request 201102391 to evaluate the lack of engineering evaluation, and Callaway Action Request 201102337 to evaluate the extent of condition to the motor-driven auxiliary feedwater pumps.

• Develop and maintain availability of just-in-time training on communicating precautions for working on the pump, critical steps in disassembly/reassembly of the pump and associated equipment, and lessons learned for the turbine-driven auxiliary feedwater pump to planners and technicians.

• Develop more stringent engineering reviews of the postmaintenance testing process. The inspection team questioned the OTS-FC-00006, TDAFW Pump Post-Maintenance Test Run on Aux Steam, 50.59 evaluation due to the problems caused by the slow speed and low flows during the postmaintenance test procedure performance raising the question if this was a test outside the current licensing basis. However, on further review, the inspectors noted that there was a precaution against running the pump below minimum flow that, if followed, could have prevented further ball skidding bearing degradation. The inspectors concur with the licensee review that determined their 10 CFR 50.59 process worked as written.

• Accelerate vibration analysis frequency from 18 months to match the oil change frequency. The oil change frequency is to be evaluated for a frequency of 18 months to a frequency of 9 months. The inspection team reviewed these performance monitoring frequencies and determined that they were not appropriately developed. The inspectors identified a noncited violation for this failure to adequately evaluate these maintenance activity frequencies and documented this as NCV 05000483/2011007-05 Failure to Establish Preventative Maintenance Schedule to Protect Safety-Related Equipment from Undetected Degraded Conditions, in this report.

• Perform an evaluation on the outboard bearing housing oil level of the pump and either restore the oil level to the original design or process a design change to approve direct lubrication for both the inboard and outboard bearing housings.

This is to address the licensees immediate action that raised the oil level in the outboard bearing to directly lubricate the roller bearings.

Findings In their reviews associated with the root cause and its supporting documentation, the team identified the following six violations:

(1) Failure to Follow Procedure in the Establishment of a Turbine-Driven Auxiliary Feedwater Pump Post Maintenance Test Procedure

Introduction.

The inspectors identified a Green noncited violation of Technical Specification 5.4.1.a involving a failure to follow procedure in the development of Procedure OTS-FC-0006, TDAFW Pump Post-Maintenance Test Run on Aux Steam.

Specifically, the licensee failed to incorporate vendor manual precautions, limitations, and technical information in Procedure OTS-FC-0006.

Description.

On May 16, 2010, during Refueling Outage 17, licensee management directed that a turbine-driven auxiliary feedwater pump postmaintenance test procedure be established and implemented prior to Mode 3 entry to provide assurance of turbine-driven auxiliary feedwater pump operability for the pump surveillance test. On May 17-18, 2010, licensee engineering, operations, and training personnel initiated, developed and reviewed new Procedure OTS-FC-0006. The intent of Procedure OTS-FC-0006, as approved, was to run the turbine-driven auxiliary feedwater pump utilizing auxiliary steam from the auxiliary boiler while in Mode 4, due to the unavailability of any other source of steam, and to ramp the pump up and down in speed as follows: first to 1280 revolutions per minute (rpm) to check speed governor control setpoints; raise speed to 3850 rpm logging speed, flow rate, oil, steam, and discharge pressures; lower speed to the idle setting of 1280 rpm; and then trip the pump. On May 20-21, 2010, and June 8-9, 2010, the turbine-driven auxiliary feedwater pump was run five times utilizing Procedure OTS-FC-0006 for a combined run time of 213 minutes over those five runs (the runs were 42, 35, 34, 79, and 23 minutes respectively). In the inspectors review of data logged over the five pump runs, the highest recirculation (total)flow rate achieved was 84 gpm at 2320 rpm.

On February 8, 2011, the licensee discovered dark oil with metal particulates in the pump outboard bearing oil sump during planned preventive maintenance to sample and change the oil (the first oil sampling since the Refueling Outage 17 bearing replacement in May 2010). Visual observation of the outboard bearing assembly indicated surface damage in localized areas in the steel bearing races, steel rolling elements, and rolling element brass cage of the inner bearing of the outboard thrust bearing assembly with the outer bearing found to be in good condition. For reference, the turbine-driven auxiliary feedwater pump outboard bearing assembly consists of two (an inner and outer)

SKF 7411 angular contact bearings stacked back-to-back. The licensee declared the turbine-driven auxiliary feedwater pump inoperable, entered the applicable Technical Specification Limiting Condition for Operation, replaced the oil and bearings, restored the pump to operability on February 11, 2011, and initiated Callaway Action Request 201101042 to perform a root cause analysis.

Licensee, pump vendor, and bearing vendor evaluation of the degraded inner outboard thrust bearing and the particulate contaminated oil determined that axial unloading of the inner bearing due to low flow/low speed operations from May and June 2010 caused insufficient stress between the bearing rolling and contact surfaces (known as ball or rolling element skidding) and thereby caused the observed bearing degradation and darkened oil. The licensee completed a past operability determination of this condition as documented Callaway Action Request 201101042 stating that despite the observed degradation, the bearing would have lasted at least 50 hours5.787037e-4 days <br />0.0139 hours <br />8.267196e-5 weeks <br />1.9025e-5 months <br /> in comparison with a bounding turbine-driven auxiliary feedwater pump mission time of 5.64 hours7.407407e-4 days <br />0.0178 hours <br />1.058201e-4 weeks <br />2.4352e-5 months <br />.

Following a review of the events described above and evaluating the technical adequacy of the condition evaluation, the inspectors reviewed Procedure OTS-FC-0006 as used in May and June 2010; Procedure APA-ZZ-00101, Processing Procedures, Manuals, and Desktop Instructions; Callaway Plant Procedure Writers Manual; Manual M-021-0061, Instruction Manual for Centrifugal Pumps; plant computer pump run and data logs; as well as conducted interviews with licensee and operations personnel involved in the development and execution of Procedure OTS-FC-0006. Following these reviews and interviews, the inspectors determined that the pump vendor technical manual provided specific guidance on Page 3 of Addendum 1 stating, Considering the potential detrimental effects of extended periods of low flow operation (cavitation damage, noise, vibration, wear and bearing and shaft failure), the following minimum flow rates are required for your pump: 120 gpm for run times under 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and 350 gpm for runs from 1-3 hours. The inspectors review also revealed pump vendor technical manual guidance on Page 24 and 25 to start the turbine and bring it up to speed rapidly, and the pump should be shut down rapidly to protect the internal wearing parts Further inspector review of Procedure OTS-FC-0006 revealed that none of the pump vendor technical manual precautions and limitations were incorporated into this new procedure except for an inadequately worded Precaution 3.8 stating: PAL02, Turbine Driven AFW Pump, should NOT be run at less than 120 gpm total flow. Run time should be limited to less than 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> during this test.

Further review of Procedure OTS-FC-0006 and pump run data logs revealed that despite vendor guidance to rapidly bring the pump up and down from speed, both the procedures direction and pump run data validated that turbine-driven auxiliary feedwater pump speed was stair-stepped up and down at many plateau speeds where these speeds were maintained for minutes at each occurrence. Inspector review of Procedure APA-ZZ-00101 as the procedure writers governing procedure for how Procedure OTS-FC-0006 was to be developed revealed that Step 4.4.4.b directed use of the Procedure Writers Manual to create this new procedure. The Procedure Writers Manual makes three distinct references in Steps 7.5.1, 7.5.3, and 7.7.1 for the use and inclusion of vendor technical manual guidance to include: major equipment operating precautions and limits recommended by the appropriate vendor manuals, limitations identified in approved vendor manuals, and procedure instruction used for maintenance-related activities that delineate vendor technical information should be applied as follows Finally as revealed in the interviews with licensee engineering and operations personnel, the inspectors determined that the procedure developers did not reference the pump vendor technical manual in the development of Procedure OTS-FC-0006 and did not perform an engineering evaluation to review any potential relief from pump vendor technical manual guidance to justify wording in Procedure OTS-FC-0006.

Analysis.

The inspectors determined that the licensees failure to utilize relevant vendor information as required by Procedure APA-ZZ-00101 in the establishment of Procedure OTS-FC-0006 was a performance deficiency. This finding was more than minor because it affected the equipment performance attribute of the Mitigating Systems Cornerstone and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences (i.e. core damage). This finding was evaluated using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings. This findings was determined to be of very low safety significance (Green) because there was not a design or qualification deficiency that resulted in a loss of operability or functionality, it did not create a loss of system safety function or of a single train for greater than the technical specification allowed outage time, it did not represent an actual loss of risk significant equipment, and it did not affect seismic, flooding, or severe weather initiating events. This finding has a cross-cutting aspect in the area of human performance associated with the work practices component because the licensee failed to ensure procedural adherence in the establishment and implementation of the turbine-driven auxiliary feedwater pump postmaintenance test procedure H.4(b).

Enforcement.

Technical Specification 5.4.1.a requires, in part, that written procedures shall be established and implemented covering activities listed in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978, which include the planning and performance of maintenance procedures for safety-related equipment. Contrary to the above, on May 18, 2010, procedure reviewers failed to implement Procedure APA-ZZ-00101, Step 4.4.4.b in using the Procedure Writers Manual to incorporate major equipment operating precautions, limitations, and technical information from the appropriate vendor manual in the establishment of Procedure OTS-FC-0006. Because the finding is of very low safety significance and has been entered into the corrective action program as Callaway Action Request 201101042, this violation is being treated as a noncited violation, consistent with Section 2.3.2 of the NRC Enforcement Policy:

NCV 05000483/2011007-01, Failure to Follow Procedure in the Establishment of a Turbine-Driven Auxiliary Feedwater Pump Postmaintenance Test Procedure.

(2) Failure to Follow Procedure for Turbine-Driven Auxiliary Feedwater Pump Postmintenance Testing

Introduction.

The inspectors identified five examples of a Green noncited violation of Technical Specification 5.4.1.a involving failure to follow Procedure OTS-FC-0006, TDAFW Pump Post-Maintenance Test Run on Aux Steam, which resulted in the axial unloading, rolling element ball skidding, and subsequent degradation to the turbine-driven auxiliary feedwater pump inner outboard thrust bearing.

Description.

As described in Section 1R3.b(1), the turbine-driven auxiliary feedwater pump was run with auxiliary steam driving the turbine while still in Mode 4, to provide assurance of turbine-driven auxiliary feedwater pump operability for the Mode 3 pump surveillance test. Procedure OTS-FC-0006 was used to guide the operators in the pump runs. On May 20-21, 2010, and June 8-9, 2010, the turbine-driven auxiliary feedwater pump was run five times utilizing Procedure OTS-FC-0006 for a combined run time of 213 minutes over those five runs (the runs were 42, 35, 34, 79, and 23 minutes respectively). In the inspectors review of data logged over the five pump runs, the highest recirculation (total) flow rate achieved was 84 gpm at 2320 rpm.

Inspectors reviewed licensee documentation and conducted interviews with licensee engineering and operations personnel and determined that operators failed to follow Precaution 3.8 restricting minimum total flow to 120 gpm during all five pump runs. The highest flow rate achieved during any of the runs was 84 gpm. Also, operators failed to follow Step 5.1.24.g of Procedure OTS-FC-0006 which provided guidance for management of low oil pressure due to the low pump speed. Inspectors concluded that these failures contributed to the detrimental conditions observed in the oil and bearing.

Analysis.

The inspectors determined that the licensees failure to follow the minimum total flow precaution of Procedure OTS-FC-0006 was a performance deficiency.

Specifically, operators failed to stop the first of five evolutions once it was realized that minimum flow could not be achieved. This finding is more than minor because it affected the equipment performance attribute of the Mitigating Systems Cornerstone and affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences (i.e. core damage). Using Inspection Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, this finding was determined to be of very low safety significance (Green) because there was not a design or qualification deficiency that resulted in a loss of operability or functionality, it did not create a loss of system safety function or of a single train for greater than the technical specification allowed outage time, it did not represent an actual loss of risk significant equipment, and it did not affect seismic, flooding, or severe weather initiating events. This finding has a cross-cutting aspect in the area of human performance associated with the work practices component because the licensee failed to ensure procedural adherence in the implementation of the turbine-driven auxiliary feedwater pump postmaintenance test procedure H.4(b).

Enforcement.

Technical Specification 5.4.1.a requires, in part, that written procedures shall be established and implemented covering activities listed in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978, which include the planning and performance of maintenance procedures for safety-related equipment. Contrary to the above, on May 20-21, 2010, and June 8-9, 2010, equipment operators failed to apply Procedure OTS-FC-0006, Precaution 3.8, to not run the turbine-driven auxiliary feedwater pump at less than 120 gpm total flow. Because the finding is of very low safety significance and has been entered into the corrective action program as Callaway Action Request 201101042, this violation is being treated as a noncited violation, consistent with Section 2.3.2 of the NRC Enforcement Policy:

NCV 05000483/2011007-02, Failure to Follow Procedure for Turbine-Driven Auxiliary Feedwater Pump Post Maintenance Testing.

(3) Failure to Follow Procedure Results in Inadequate Past Operability Determination

Introduction.

The inspectors identified a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, regarding the licensees failure to follow the requirements of Procedure APA-ZZ-00500, Appendix 3 Past Operability and Reportability Evaluations. Specifically, the inspectors identified that the past operability evaluation for the turbine-driven auxiliary feedwater pump, as documented in Callaway Action Request 201101042, used a nonconservative calculation of mission time that did not take into account all design and licensing basis safety functions, when determining the mission time.

Description.

On March 18, 2011, the licensee completed a past operability determination for the degraded inner outboard thrust bearing for the turbine-driven auxiliary feedwater pump as documented in Callaway Action Request 201101042. This request initiated performance of a root cause analysis of the event where a low flow/low speed postmaintenance test of the turbine-driven auxiliary feedwater pump performed in May and June 2010 caused axial unloading of the inner bearing and insufficient stress between the bearing rolling and contact surfaces (known as ball or rolling element skidding) and thereby caused the observed bearing degradation and darkened oil. This past operability determination evaluated the mission time by considering the coping period of 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> for the turbine-driven auxiliary feedwater pump during a station blackout event to be the most conservative time requirement. The past operability evaluation also noted a Final Safety Analysis Report, Chapter 15, design basis accident time of 1.64 hours7.407407e-4 days <br />0.0178 hours <br />1.058201e-4 weeks <br />2.4352e-5 months <br />, based on the loss of normal feedwater transient.

Following a review of the past operability evaluation, the inspectors reviewed Procedure APA-ZZ-00500, which provides the guidance used by operations and engineering staff to perform operability determinations. The inspectors noted that 1, Paragraph 6 requires, in part, that when the design and licensing basis function is understood, assess the degraded or nonconforming condition and determine if the condition prevented the structures, systems, and components from performing any of those functions for the duration of the mission time. Additional general guidance from Paragraph 6 for performing past operability evaluations states, in part, that In order to perform a past operability evaluation, it is necessary to understand the design and licensing basis function of the structures, systems, and components in addition to the mission time. Procedure APA-ZZ-00500, Appendix 3, further states that guidance typically comes from one or more of the sources of information, including, but not limited to: technical specifications including bases, Final Safety Analysis Report, accident analysis bases documents, and that the probabilistic risk analyses.

Following the past operability determination and procedural review, the inspectors identified that the calculated mission time was nonconservative, as the 1.64 hours7.407407e-4 days <br />0.0178 hours <br />1.058201e-4 weeks <br />2.4352e-5 months <br /> previously mentioned accounts only for the accident phase and failed to account for the turbine-driven auxiliary feedwater pumps required safety function for the recovery phase. The recovery phase accounts for the time when auxiliary feedwater is needed until shutdown cooling of the plant with the residual heat removal system is established.

Therefore, the inspectors identified that the past operability determination did not account for the full design and licensing basis of the turbine-driven auxiliary feedwater pumps specified safety function due to a nonconservative calculation of the turbine-driven auxiliary feedwater pumps mission time. The calculated mission time referenced the accident phase only and omitted the recovery phase required to ensure that safe shutdown of the plant is achieved and maintained.

After being notified of this discrepancy, the licensee initiated Callaway Action Request 201102431 and performed Revision 1 of the past operability determination dated March 31, 2011, which correctly evaluated the more conservative mission time as the sum of 1.64 hours7.407407e-4 days <br />0.0178 hours <br />1.058201e-4 weeks <br />2.4352e-5 months <br /> for the accident phase and 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> additional time for the recovery phase, resulting in a mission time of 5.64 hours7.407407e-4 days <br />0.0178 hours <br />1.058201e-4 weeks <br />2.4352e-5 months <br />. The inspectors determined that Revision 1 of the past operability determination adequately assessed the turbine-driven auxiliary feedwater pump mission time.

Analysis.

The inspectors determined that the licensees failure to follow the requirements of Procedure APA-ZZ-00500, was a performance deficiency. Specifically, the inspectors identified that the past operability determination for the turbine-driven auxiliary feedwater pump used a nonconservative calculation of mission time that did not take into account all design and licensing basis safety functions when determining the mission time. The finding is greater than minor because if left uncorrected, it would have the potential to lead to a more significant safety concern because systems that may be inoperable may not be recognized, and that it impacted the Mitigating Systems Cornerstone attribute of human performance in that the failure to accurately understand the auxiliary feedwater system mission time affected the mitigating systems objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. The finding was evaluated using Inspection Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings. The finding is determined to have very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, it did not create a loss of system safety function or of a single train for greater than the technical specification allowed outage time, it did not represent an actual loss of risk significant equipment, and it did not affect seismic, flooding, or severe weather initiating events. The cause of this finding is related to the problem identification and resolution cross-cutting component of corrective action program because licensee personnel failed to thoroughly evaluate conditions adverse to quality and perform adequate operability determinations P.1(c).

Enforcement.

Title 10 of the Code of Federal Regulations Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, requires, in part, that activities affecting quality shall be prescribed by documented instructions or drawings, of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions or drawings. Procedure APA-ZZ-00500, Appendix 3, Past Operability and Reportability Evaluations, requires the licensee to perform a past operability evaluation with an understanding of the design and licensing basis safety function of the structures, systems, and components in addition to the mission time. Contrary to this requirement, the past operability determination for the turbine-driven auxiliary feedwater pump did not fully account for all design and licensing basis functions, resulting in a nonconservative calculation of mission time. The calculated mission time referenced the accident phase only and omitted the recovery phase required to ensure safe shutdown of the plant is achieved and maintained. Because the finding is of very low safety significance and has been entered into the licensees corrective action program as Callaway Action Request 201102431, this violation is being treated as a noncited violation consistent with Section 2.3.2 of the Enforcement Policy: NCV 05000483/2011007-03, Failure to Follow Procedure Results in Inadequate Past Operability Determination.

(4) Failure to Calculate and Implement Conservative Safety Related Equipment Oil Leakage Operability Criteria

Introduction.

The inspectors identified a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for a failure to adequately determine safety related equipment oil leakage acceptance criteria that was then used in operator logs.

Specifically, the 2007 licensee fluid leak management program calculations to determine the mission time assessment related to oil leak rates of safety related pumps and motors were nonconservative when added to Procedure ODP-ZZ-0016E, Appendix 1, Equipment Operator General Inspection Guide.

Description.

The safety related equipment evaluated for operability limits of oil leakage are pumps and motors for the emergency core cooling system, containment spray system and the component cooling water system. The emergency core cooling system consists of the centrifugal charging pumps and motors, the residual heat removal motors, and the safety injection pumps and motors.

On March 22, 2011, during a walkdown of the auxiliary feedwater, component cooling water, and spent fuel pool cooling pumps as part of the special inspection charter for extent of condition the inspectors questioned if observed oil leakage and current oil bubbler level for the inboard bearing on component cooling water pump D had any impact on the operability of this equipment. The inspectors questioned if the amount of oil pooling in the pump packing leakoff area would prevent the component cooling water pump safety function for a 30 day mission time.

In response to the inspectors questions and to determine operability of component cooling water pump D the licensee initiated Callaway Action Request 201102287 and determined that the oil leak would not have prevented the pump from running for the 30 day mission time. The inspectors investigation found that the licensee does not have a program to monitor oil additions (oil leakage) other than reviewing work documents or operator logs. This raised a concern that it would be difficult to precisely determine the actual oil leak rate of the component cooling water pump D. However, on February 18, 2011, the licensee performed an extent of condition review for the turbine-driven auxiliary feedwater pump degraded oil issue with respect to oil level settings for pumps with TRICO oilers. A picture taken for this review showed the inboard oiler for component cooling water pump D to be approximately 75 percent full. The oil level observed on March 22, 2011, during the inspectors walkdown, was compared to determine the difference between the levels. The oil leak rate was determined to be 1.6 drops per hour. The inspectors reviewed the operability calculation that conservatively determined the acceptable oil leak rate to be 1.8 drops per hour.

The inspectors also reviewed the operator logs oil leak rate operability limit evaluation for the safety related pumps and motors that may become inaccessible during an accident requiring them to run with no oil addition for their mission times. The inspectors reviewed oil leak operability limit evaluation document, Callaway Action Request 200608098, and found the evaluation for the component cooling water pump to be incorrect resulting in a nonconservative operability oil leak limit of 5.3 drops per hour.

The inspectors then questioned the accuracy of the operability limits for other equipment in this evaluation: charging pumps, safety injection pumps, containment spray pumps and residual heat removal pumps. The licensee addressed this question in Callaway Action Request 201102434, determined new, more conservative limits for the equipment and reviewed data back though 2007 that verified none of the affected equipment had been inoperable due to oil leakage during this period.

In Callaway Action Request 200608098, Action 4, engineering established oil leakage limits for operability that were incomplete and did not accurately meet design. In Action 9 these limits were placed in Procedure ODP-ZZ-0016E, Appendix 1, Equipment Operator General Inspection Guide, resulting in an inaccurate procedural guidance for operators. The inspectors found this nonconservative 2007 evaluation was not reflective of current licensee performance and therefore, this finding has no cross-cutting aspects.

The inspectors did determine that licensee past performance failed to provide complete, accurate and up-to-date operability limits for the operator procedures resulting in nonconservative guidance for safety related pump and motor oil leakage limits. These nonconservative limits affected the Mitigating Systems Cornerstone equipment reliability attribute because if left uncorrected could have allowed adverse oil leakage rates to remain undetected, masking inoperable safety related equipment.

Analysis.

The inspectors determined that the failure to adequately evaluate conservative oil leakage operability guidance used in operator logs is a performance deficiency. This finding is more than minor because if left uncorrected it has the potential to lead to a more significant safety concern. Specifically, the failure to adequately evaluate and determine an appropriate lube oil leak rates to maintain safety related equipment operable affects the equipment performance attribute of the Mitigating Systems Cornerstone and could have impacted the availability of mitigating equipment if left uncorrected. The finding was assessed using Manual Chapter 0609.04, Phase 1 -

Initial Screening and Characterization of Findings, and was determined to be of very low safety significance since the as-found condition of the safety related equipment reviewed back to August 15, 2007, found no oil leakage rates that would have caused a loss of system safety function. This finding was not reflective of current licensee performance and therefore, has no cross-cutting aspect.

Enforcement.

Title 10 of the Code of Federal Regulations Part 50, Appendix B, Criterion III, Design Control, requires, in part, that design basis for components are correctly translated into procedures and that deviations are controlled. Contrary to this, from August 15, 2007, until March 22, 2011, the NRC identified that the licensee failed to adequately evaluate and translate into operator logs conservative oil leakage limits that ensured operability of the emergency core cooling system, containment spray system, and component cooling water system pumps and motors. This finding is of very low safety significance and is in the licensees corrective action program as Callaway Action Request 201102434. This finding is characterized as a noncited violation consistent with Section 2.3.2 of the NRC Enforcement Policy, and is identified as:

NCV 05000483/2011007-04, Failure to Calculate and Implement Conservative Safety Related Equipment Oil Leakage Operability Criteria.

(5) Failure to Establish Preventative Maintenance Schedule to Protect Safety-Related Equipment from Undetected Degraded Conditions

Introduction.

The inspectors identified a Green noncited violation of Technical Specification 5.4.1 for failure to adequately establish and implement procedures required by Regulatory Guide 1.33, Appendix A, Section 9, Procedures for Performing Maintenance. Specifically, the preventative maintenance schedule to perform periodic lube oil analysis established an 18 month frequency without adequate justification resulting in a failure to promptly detect bearing degradation occurring in the turbine-driven auxiliary feedwater pump.

Description.

As part of the special inspection of the turbine-driven auxiliary feedwater pump, the inspectors reviewed pump trend information from oil analysis, vibration data, and pressure data. The inspectors noted intervals for the oil samples, oil changes, and vibration testing data extended up to 18 months, greater than frequencies normally seen in the industry. The vendor manual recommends changing the oil every 3 months and recommends periodic oil analysis but has no frequency specified. When the inspectors questioned these intervals the licensee provided a preventative maintenance basis document performed in 2007 that extended the oil analysis frequency to 72 weeks (18 months) and explained the vibration testing was performed every 72 weeks in accordance with code test requirements. The inspectors verified the vibration testing was in compliance with the 2001 version of the ASME Operation and Maintenance Code for a class B pump, in a standby system that has requirements to perform quarterly tests to check flow, pressure, and speed and a comprehensive test every two years that check flow, pressure, speed, and vibration.

The inspectors reviewed the basis change for PM0812171, pump oil change/sample on the turbine-driven auxiliary feedwater pump that extended the job from 48 weeks to 72 weeks in 2007. The document notes that the water content for the previous three years of oil sampling has been acceptable. The reason given justifying the change was, The frequency for this activity is consistent with expert panels composed of knowledgeable individuals from the Electric Power Research Institute (EPRI), member utilities, and manufacturers that suggested an oil analysis every 3 months for continuously running critical pumps, and every 18 months for non critical pumps. The reference referred to Electric Power Research Institute Manual, Preventive Maintenance Basis, Volume 13: Horizontal Pumps, TR-106857-V13, 1997. The inspectors review of this manual found a horizontal pump preventative maintenance template chart that listed, for a type two pump (Critical, Low Duty Cycle, Severe Service Condition such as High Pressure Coolant Injection Pump, Auxiliary Feedwater Pump, or Safety Injection Pump) in the oil analysis column, a recommended frequency of 18 months. In the body of the manual the inspectors also found recommendations that the licensee did not follow in their evaluation to extend the job interval. The licensee failed to: consider the vendor manual recommendations, determine the basis of the previous 42 week frequency, or review past sample data other than water analysis.

Other test parameters such as wear metals, viscosity, and other contaminants could also impact the frequency of testing and were not considered.

The inspectors also noted that the horizontal pump maintenance template recommended monthly vibration testing for this class of pump though the licensee was currently performing vibration testing on an 18 month frequency. The licensees root cause report states that other nuclear power stations perform vibration testing of their turbine-driven auxiliary feedwater pumps on a shorter frequency, as short as quarterly. A vibration test frequency shorter than yearly would provide additional benefits in frequency analysis and trending of the equipment health.

The licensees corrective actions will review the oil change preventative maintenance frequency and at a minimum adjust frequency to change and sample the oil every nine months. There is an additional action to change vibration testing frequency to match the oil analysis frequency.

The licensee had recent opportunities to correct the inadequate preventative maintenance bases. The first occurred during the 2009 extent of cause of the turbine-driven auxiliary feedwater trip and throttle valve failure due to reduction in scope of a preventative maintenance activity, the second during the September 2009 black oil investigation, and the third during the May 2010 black oil investigation. Each of these events provided the licensee an opportunity to review its oil analysis and oil change frequency. These missed opportunities demonstrate a cross-cutting aspect in the area of problem identification and resolution in that the licenses corrective action program failed to thoroughly evaluate these problems such that the resolutions address extent of condition with the turbine-driven auxiliary feedwater pump preventative maintenance activities and frequencies. Failure to closely monitor the condition delayed the discovery of a degraded condition affecting the equipment performance attribute of the Mitigating Systems Cornerstone and could have impacted the availability of the turbine-driven auxiliary feedwater pump if left uncorrected.

Analysis.

The inspectors determined that failure to establish a conservative maintenance schedule to monitor the turbine-driven auxiliary feedwater pump lube oil condition is a performance deficiency. This finding is more than minor because if left uncorrected it has the potential to lead to a more significant safety concern. Specifically, failure to adequately evaluate and determine an appropriate lube oil monitoring schedule resulted in failure to promptly detect a degraded bearing in the pump affecting the equipment performance attribute of the Mitigating Systems Cornerstone and could have impacted the availability of this mitigating equipment if left uncorrected. The finding was assessed using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, and was determined to be of very low safety significance since the as-found condition of the degraded bearing would not have caused a loss of system safety function. The finding has a cross-cutting aspect in the area of problem identification and resolution associated with the corrective action program component, in that, the licensee failed to thoroughly evaluate problems such that the resolutions address extent of conditions when they failed to review the turbine-driven auxiliary feedwater pump preventative maintenance scope and frequency during previous adverse conditions P.1(c).

Enforcement.

Technical Specification 5.4.1.a requires that written procedures as described in Regulatory Guide 1.33, Revision 2, Appendix A, be established, implemented and maintained. Regulatory Guide 1.33 Appendix A, Section 9, Procedures for Performing Maintenance, Subsection b, requires that preventative maintenance procedures and schedules be developed to include inspections of equipment and replacement of items that have a specific lifetime. Contrary to this, between September 2007 and March 25, 2011, the NRC identified that the licensee failed to establish and implement a preventative maintenance schedule frequency to ensure reliability of the turbine-driven auxiliary feedwater pump bearings. This finding is of very low safety significance and is in the licensees corrective action program as Callaway Action Request 201101042. The finding is characterized as a noncited violation consistent with Section 2.3.2 of the NRC Enforcement Policy, and is identified as: NCV 05000483/2011007-05, Failure to Establish Preventative Maintenance Schedule to Protect Safety-Related Equipment from Undetected Degraded Conditions.

(6) Inadequate Extent of Cause Results in Missed Safety-Related Pump Overhaul

Introduction.

The inspectors identified a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, associated with the licensees failure to promptly identify and correct a condition adverse to quality. Specifically, the licensee reduced the scope of preventative maintenance for the turbine-driven auxiliary feedwater pump overhaul during Refueling Outage 16 without proper justification, resulting in the failure to perform required pump maintenance.

Description.

In 2009, Callaway Plant experienced a failure of the turbine-driven auxiliary pump steam supply trip and throttle valve to open. The trip and throttle valve had its lubrication and inspection preventative maintenance scope reduced during the 2008 Refueling Outage 16, resulting in the valve not being properly lubricated and subsequently failing. The subsequent failure investigation extent of cause review determined the cause to be an isolated occurrence. Therefore, the licensee did not review other Refueling Outage 16 turbine-driven auxiliary feedwater pump preventative maintenance activities for similar issues.

During review of the February 8, 2011, degraded pump bearing, the licensee identified that the complete overhaul of the turbine-driven auxiliary feedwater pump during Refueling Outage 16 had been modified to only perform mechanical seal replacement and bearing replacement. Inspection or overhaul of the pump internals was not performed, nor was a work order initiated for a complete overhaul of the pump after the work scope was reduced. The licensees program required a pump overhaul after either 500 pump starts or 1000 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.805e-4 months <br /> of operation. The inspectors questioned if failure to perform the 2008 pump internal inspection portion of the job had exceeded any program requirements. The licensee reviewed pump history and determined the last overhaul of the pump was in 1995 and there had been 535 starts since then. The licensee also determined that the station health program to monitor pump hours and starts had not been maintained.

Review of the Refueling Outage 16 pump overhaul reduction in scope found no documentation approving the scope change. The job was not completed since parts (the rotating assembly) were not obtained prior to the outage. The pump internal overhaul was deleted and documented that it was considered for Refueling Outage 17 pending further evaluation; however, no evaluation was performed. The inspectors identified that the reduction in scope of the maintenance activity which resulted in the missed overhaul was similar to the cause of the 2009 valve failure. The reduction in maintenance scope occurred during the same refueling outage and on the same component, and therefore should have been identified during the extent of cause evaluation for the failed trip and throttle valve. As a result of the inspectors observations, the licensee included inadequate corrective action in evaluation of prior issues with the turbine-driven auxiliary feedwater pump as a contributing cause in the root cause evaluation for the February 2011 degraded pump bearing.

The inspectors concluded that completion of the turbine-driven auxiliary feedwater pump overhaul could have potentially identified and fixed issues with rotor centralization, shaft alignment, and identified any parts showing service life limitations. These were all potential contributing causes to the ball skidding and the licensee planned corrective actions to check these during the next overhaul and inspection of the pump internals.

Additional corrective actions planned by the licensee included revising the turbine-driven auxiliary feedwater pump preventative maintenance basis overhaul frequency based on historical pump runs and using the vendor pump life calculation to ensure future pump overhaul frequencies would be performed at conservative intervals. The pump was also scheduled to be overhauled at the next opportunity, Refueling Outage 18, and the pump service life was to be monitored by the plant health reporting tool.

Analysis.

The inspectors determined that the failure to identify a condition adverse to quality was a performance deficiency. This finding is more than minor because the failure to perform required maintenance in a timely manner affected the equipment performance attribute of the Mitigating Systems Cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, Phase 1 - Initial Screening and Characterization of Findings, this finding was determined to be of very low safety significance because the degraded condition did not result in a loss of operability or functionality. The inspectors determined that the finding has a cross-cutting aspect in the area of human performance associated with the work control component because the licensee did not appropriately coordinate work activities by incorporating actions to address the impact of reductions to work scope on the plant such that nuclear safety is supported H.3(b).

Enforcement.

Title 10 of the Code of Federal Regulations Part 50, Appendix B, Criterion XVI, Corrective Action, requires, in part, that measures shall be established to assure that conditions adverse to quality are promptly identified and corrected. Contrary to this, from July 16, 2009, to March 25, 2011, the measures established by the licensee failed to identify a condition adverse to quality. Specifically, an inadequate extent of condition review for the work scope reduction for the turbine-driven auxiliary feedwater pump resulted in the failure to perform required pump maintenance. Because this finding is of very low safety significance and is in the licensees corrective action program as Callaway Action Request 201102407, it is characterized as a noncited violation consistent with Section 2.3.2 of the NRC Enforcement Policy:

NCV 05000483/2011007-06, Inadequate Extent of Cause Results in Missed Safety-Related Pump Overhaul.

5.0 Review of Operating Experience

a. Scope

The team reviewed internal operating experience by obtaining a list of plant corrective action documents related to the auxiliary feedwater system and selecting those documents related to lubrication and bearing problems. The team further reviewed the licensees review of industry operating experience for the auxiliary feedwater system and pump bearing and lubrication problems. The team review included inspection of the licensees operating experience program and specific review of related operating experience from the root cause investigation for the February 8, 2011, discovery of degraded oil and bearings in the turbine-driven auxiliary feedwater pump.

For external operating experience, the Operating Experience Branch in the Office Nuclear Reactor Regulation provided the results of keyword searches related to black oil, bearing failures, and lubrication findings associated with turbine-driven pumps. The NRC Operating Experience Branch provided a list of licensee event reports, NRC Information Notices, NUREG documents and other operating experience information.

The team selected operating experience information that was applicable to this inspection and reviewed whether the licensee had addressed the items in their root cause analyses related to these events or had processed the information through their operating experience program. As part of their review, the inspectors performed an auxiliary feedwater system walkdown to determine if applicable industry operating experience had been incorporated into system design and maintenance practices.

b. Findings and Observations

No findings of significance were identified.

6.0 Potential Generic Issues

a. Scope

The team evaluated the degradation of Callaways turbine-driven auxiliary feedwater pump bearings to determine whether any potential generic issues should be communicated to the industry (e.g., information notices, generic letters, and bulletins).

b. Findings and Observations

The team determined that this issue did not warrant a generic communication based on the types of problems encountered.

4OA6 Meetings, Including Exit

On March 25, 2011, the inspection team presented the preliminary results of this inspection at the end of the onsite week to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of his staff who acknowledged the findings.

The team verified that no proprietary information was retained.

On May 13, 2011, the team leader presented the final results of the inspection to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of the licensee staff who acknowledged the findings. The team verified that no proprietary information was retained.

On June 27, 2011, the team leader conducted a final exit meeting with Mr. S. Maglio, Manager, Regulatory Affairs, and other members of the licensee staff who acknowledged the findings. The team verified that no proprietary information was retained.

1: Supplemental Information 2: Special Inspection Charter 3: Timeline

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

Matt Hall, Manager, Engineering Systems

Jeff Imhoff, System Engineer

Sam Morris, Engineer, Eng T-Performance

Jesse Pitts, Engineering Lead, Eng T-Performance

Adam Schnitz, Associate Engineer, Licensing

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

Failure to Follow Procedure in the Establishment of a Turbine-

05000483-2011007-01 NCV Driven Auxiliary Feedwater Pump Postmaintenance Test Procedure (Section 4.0)

Failure to Follow Procedure for Turbine-Driven Auxiliary

05000483-2011007-02 NCV Feedwater Pump Postmaintenance Testing (Section 4.0)

Failure to Follow Procedure Results in Inadequate Past

05000483-2011007-03 NCV Operability Determination (Section 4.0)

Failure to Calculate and Implement Conservative Safety

05000483-2011007-04 NCV Related Equipment Oil Leakage Operability Criteria (Section 4.0)

Failure to Establish Preventative Maintenance Schedule to

05000483-2011007-05 NCV Protect Safety-Related Equipment from Undetected Degraded Conditions (Section 4.0)

Inadequate Extent of Cause Results in Missed Safety-Related

05000483-2011007-06 NCV Pump Overhaul (Section 4.0)