IR 05000285/2010006

IR 05000285-10-006, on 3/22/10 - 6/30/10; Fort Calhoun Station; Special Inspection Into Turbine-Driven Auxiliary Feedwater Pump Failures to Start

ML102250215
Person / Time
Site:	Fort Calhoun
Issue date:	08/12/2010
From:	Clark J NRC/RGN-IV/DRP/RPB-E
To:	Bannister D Omaha Public Power District
References
IR-10-006
Download: ML102250215 (48)

Text

UNITED STATES NU C LE AR RE G UL AT O RY C O M M I S S I O N R E GI ON I V 612 EAST LAMAR BLVD , SU I TE 400 AR LI N GTON , TE XAS 76011-4125 August 12, 2010 Mr.David J. Bannister, Vice President and Chief Nuclear Officer Omaha Public Power District Fort Calhoun Station FC-2-4 P.O. Box 550 Fort Calhoun, NE 68023-0550 Subject: FORT CALHOUN PLANT - NRC SPECIAL INSPECTION REPORT 05000285/2010006

Dear Mr. Bannister:

On June 30, 2010, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Fort Calhoun Station to evaluate the facts and circumstances surrounding the failure to start of the turbine-driven auxiliary feedwater pump, FW-10 due to an inadequately protected reset lever. Based upon the risk and deterministic criteria specified in NRC Management Directive 8.3, "NRC Incident Investigation Program," including possible generic implications, the NRC initiated a special inspection in accordance with Inspection Procedure 93812, "Special Inspection." The basis for initiating the special inspection and the focus areas for review are detailed in the Special Inspection Charter (Attachment 2). The determination that the inspection would be conducted was made by the NRC on March 11, 2010, and the on-site inspection started on March 22, 2010. The enclosed report documents the inspection findings that were discussed on June 30, 2010, with Mr. Jeff Reinhart, Site Vice President and other members of your staff.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed plant personnel.

This report documents three self-revealing findings and one NRC-identified finding of very low safety significance (Green). All four of these findings were determined to involve violations of NRC requirements. However, because of their very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations, consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the violations or the significance of the noncited violations, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 612 E. Lamar Blvd, Suite 400, Arlington, Texas, 76011-4125; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the

Omaha Public Power District 2 NRC Resident Inspector at the Callaway Plant. In addition, if you disagree with the crosscutting aspect assigned to any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Fort Calhoun Station.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, and its enclosure, will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRCs document system (ADAMS).

ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/ RVA for Jeffrey A. Clark, P.E.

Chief, Project Branch E Division of Reactor Projects Docket: 50-285 License: DPR-40 Enclosure: NRC Inspection Report 05000285/2010006 w/Attachments:

Attachment 1: Supplemental Information Attachment 2: Special Inspection Charter Attachment 3: Timeline of Activites Relating to 2009 and 2010 TDAFW Pump Failures Attachment 4: Final Significance Determination Evaluation Jeffrey A. Reinhart Site Vice President Omaha Public Power District Fort Calhoun Station FC-2-4 Adm P.O. Box 550 Fort Calhoun, NE 68023-0550 Mr. Bill R. Hansher Manager (Acting) - Nuclear Licensing Omaha Public Power District Fort Calhoun Station FC-2-4 Adm.

P.O. Box 550 Fort Calhoun, NE 68023-0550 David A. Repka Winston & Strawn 1700 K Street, NW Washington, DC 20006-3817

Omaha Public Power District 3 Chairman Washington County Board of Supervisors P.O. Box 466 Blair, NE 68008 Ms. Julia Schmitt, Manager Radiation Control Program Nebraska Health & Human Services Division of Public Health P.O. Box 95026 Lincoln, NE 68509-5026 Ms. Melanie Rasmussen Radiation Control Program Officer Bureau of Radiological Health Iowa Department of Public Health Lucas State Office Building, 5th Floor 321 East 12th Street Des Moines, IA 50319 Chief, Technological Hazards Branch FEMA, Region VII 9221 Ward Parkway Suite 300 Kansas City, MO 64114-3372

Omaha Public Power District 4 Electronic distribution by RIV:

Regional Administrator (Elmo.Collins@nrc.gov)

Deputy Regional Administrator (Chuck.Casto@nrc.gov)

DRP Acting Director (Tony.Vegel@nrc.gov)

DRP Acting Deputy Director (Troy.Pruett@nrc.gov)

DRS Director (Roy.Caniano@nrc.gov)

DRS Acting Deputy Director (Jeff.Clark@nrc.gov)

Senior Resident Inspector (John.Kirkland@nrc.gov)

Resident Inspector (Jacob.Wingebach@nrc.gov)

Branch Chief, DRP/E (Jeff.Clark@nrc.gov)

Senior Project Engineer, DRP/E (Ray.Azua@nrc.gov)

Project Engineer (William.Schaup@nrc.gov)

FCS Administrative Assistant (Berni.Madison@nrc.gov)

Public Affairs Officer (Victor.Dricks@nrc.gov)

Public Affairs Officer (Lara.Uselding@nrc.gov)

Branch Chief, DRS/TSB (Michael.Hay@nrc.gov)

Project Manager (Lynnea.Wilkins@nrc.gov)

RITS Coordinator (Marisa.Herrera@nrc.gov)

Regional Counsel (Karla.Fuller@nrc.gov)

Congressional Affairs Officer (Jenny.Weil@nrc.gov)

OEMail Resource ROPreports DRS/TSB STA (Dale.Powers@nrc.gov)

OEDO RIV Coordinator (Margie.Kotzalas@nrc.gov)

Executive Technical Assistant (John.Giessner@nrc.gov)

File located: R:_REACTORS\_FCS\2010\FCS2010006RP-MLC.docx ML102250215 SUNSI Rev  : Yes

Yes No ADAMS Reviewer Initials RVA Compl. No Yes Publicly Avail : Yes No Sensitive Sens. Type Initials RVA

No RIV: SRI:DRP/B SRI:DNMS/LVFO SPE:DRP/E C:DRP/E MLChambers LWilloughby RVAzua JAClark

/RA/ E /RA/ E /RA/ /RA/ RVA for 08/09/2010 08/09/2010 08/09/2010 08/12/2010 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax

U.S. NUCLEAR REGULATORY COMMISSION

REGION IV

Docket: 05000285 License: DPR-40 Report: 05000285/2010006 Licensee: Omaha Public Power District Facility: Fort Calhoun Station Location: 9610 Power Lane Blair, NE 68008 Dates: March 22 through June 30, 2010 Inspectors: M. Chambers, Senior Resident Inspector, Cooper Nuclear Station L. Willoughby, Senior Resident Inspector, Las Vegas Field Office, Division of Nuclear Material Safety J. Kirkland, Senior Resident Inspector, Fort Calhoun Station J. Wingebach, Resident Inspector, Fort Calhoun Station M. Williams, Reactor Inspector, Plant Support Branch 2 D. Loveless, Senior Reactor Analyst Approved By: J. Clark, Chief, P.E., Project Branch E Division of Reactor Projects 1 Enclosure

SUMMARY OF FINDINGS

IR 05000285/2010006; 3/22/10 - 6/30/10; Fort Calhoun Station; Special inspection into turbine-

driven auxiliary feedwater pump failures to start.

The report covered one week of on-site inspection and in-office review through June 30, 2010.

Four resident inspectors performed the inspection with assistance from two region based inspectors including a senior reactor analyst. Four Green noncited violations were identified.

The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance Determination Process." The crosscutting aspect was determined using Inspection Manual Chapter 0310, Components Within the Cross-Cutting Area. Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

Green.

A self-revealing noncited violation of 10 CFR 50 Appendix B,

Criterion XVI, Corrective Action, occurred for the licensees failure to assure that a condition adverse to quality was corrected. Specifically, five instances were identified where the licensee failed to correct an adverse configuration design which allowed the turbine-driven auxiliary feedwater pump FW-10 exhaust backpressure trip reset lever to be bumped and unlatched which would have prevented the pump from starting when required. The failure to correct this adverse condition resulted in the turbine-driven auxiliary feedwater pump reset lever becoming unlatched and causing the pump to trip off during a surveillance test start attempt on February 17, 2010. The licensee entered this issue in their corrective action program as Condition Report CR-2010-0813.

The finding is more than minor because it is associated with the equipment performance attribute of the Mitigating Systems Cornerstone, and affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events. Using Manual Chapter 0609.04,

"Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function of a single train for greater than the technical specification allowed outage time. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Fort Calhoun Phase 2 pre-solved table item, Turbine-driven Auxiliary Feedwater Pump Fails to Start, the inspectors determined this finding to be potentially risk significant. A Phase 3 analysis was performed and it was determined that the finding was of very low risk significance. This finding has a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensees periodic trends and assessments did not recognize the significance of precursor events related to bumping the reset lever and prompt action to prevent further problems with the turbine-driven auxiliary feedwater pump FW-10 P.1(b) (Section 1.2.1).

Green.

The team identified a noncited violation of Technical Specification 5.8.1.a regarding the licensees failure to implement written procedures as recommended in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978.

Paragraph 3.l of Appendix A requires procedures for startup, shutdown and operation of the auxiliary feedwater system. Specifically, the licensee had no procedural guidance to verify full engagement of the turbine-driven auxiliary feedwater pump FW-10 exhaust backpressure trip mechanism when latched.

This resulted in the licensees failure to identify the partially latched condition of the exhaust trip mechanism which subsequently vibrated loose during a surveillance test causing a start failure of the pump, on February 17, 2010. The licensee entered this deficiency in their corrective action program as Condition Report CR 2010-0813.

This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences.

Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The finding has a crosscutting aspect in the area of problem identification and resolution associated with operating experience because the licensee failed to implement and institutionalize operating experience through changes to station operating procedures when they failed to incorporate industry information to verify the turbine-driven auxiliary feedwater pump is fully latched P.2(b) (Section 1.2.2).

Green.

A self-revealing noncited violation of Technical Specification 5.8.1.a was identified regarding the licensees failure to implement and maintain the applicable procedures recommended in Regulatory Guide 1.33, Revision 2,

Appendix A, February 1978. Paragraph 9.a of Appendix A requires that such maintenance that can affect the performance of safety-related equipment be properly preplanned and performed in accordance with documented instructions.

Specifically, the licensee failed to have an adequate procedure for ensuring air was vented from the auxiliary feedwater pump control oil system following maintenance. As a result, the turbine-driven auxiliary feedwater pump failed to start during the February 26, 2009, operability test. The licensee has entered this issue into their corrective action program as Condition Report CR-2009-0905.

The finding is more than minor because it is associated with the Mitigating System Cornerstone attribute of procedure quality and adversely affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage). Using Manual Chapter 0609.04, "Phase 1 Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a design deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification required train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a crosscutting aspect in the area of human performance associated with conservative assumptions due to the licensee failing to identify possible unintended consequences of high points in a control oil system tubing design change that could become air bound and interfere with fast starts of the turbine-driven auxiliary feedwater pump H.1(b)

(Section 1.3.1).

Green.

A self revealing noncited violation of 10 CFR, Part 50, Appendix B,

Criterion III, Design Control, occurred when the licensee failed to ensure that the design basis of certain structures, systems and components were translated into specifications, drawings, procedures, and instructions when implementing Engineering Change 45105. Specifically, this design change reduced the turbine-driven auxiliary feedwater pumps margin between the pump discharge pressure and the pumps high discharge pressure trip set-point resulting in an April 6, 2009, high pump discharge pressure trip during a scheduled surveillance test start. The licensee entered this issue in their corrective action program as Condition Report CR-2009-1611.

The inspectors determined the finding is more than minor because it is associated with the design control attribute of the Mitigating Systems Cornerstone, and adversely affected the cornerstone objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 -

Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function of a single train for greater than the technical specification allowed outage time. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Fort Calhoun Phase 2 presolved table item, Turbine-driven Auxiliary Feedwater Pump Fails to Start, the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. A Phase 3 analysis was performed and it was determined that the finding was of very low risk significance. The finding has a crosscutting aspect in the area of human performance because the licensee failed to use conservative assumptions in decision making when a nonconservative design margin was approved and implemented on the turbine-driven auxiliary feedwater pump H.1(b)

(Section 1.3.2).

Licensee-Identified Violations

None

REPORT DETAILS

REACTOR SAFETY

Cornerstones: Initiating Events, Mitigating Systems, Barrier Integrity, and Emergency Preparedness 1.0 Special Inspection Scope On February 17, 2010, the turbine-driven auxiliary feedwater pump (FW-10) tripped off following a start demand signal during a monthly operability surveillance test. The FW-10 steam turbine exhaust backpressure trip reset lever was noticed to trip shortly after the pump start. In accordance with Management Directive 8.3, NRC Incident Investigation Program, the NRC determined that a special inspection was warranted, in part, based on the potential safety significance and because of previous problems with this pump that were not conclusively resolved.

The inspection charter required the team to:

(1) review the circumstances related to the discovery of the degraded condition,

(2) assess the licensees determination of cause and effectiveness of actions taken to resolve and prevent recurrence of these problems, and

(3) assess the effectiveness of licensee programs to maintain the physical condition of the turbine-driven auxiliary feedwater pump. The team evaluated if the licensee took appropriate actions to address these issues including extent of condition, extent of cause, and common cause questions. The inspectors reviewed the licensees seismic design basis to ensure the seismic robustness of the turbine-driven auxiliary feedwater pump and that the surrounding structures were maintained to ensure their function under design-basis conditions.

The team conducted their reviews in accordance with NRC Inspection Procedure 93812, "Special Inspection Procedure." The special inspection team reviewed procedures, corrective action documents, as well as design and maintenance records for the equipment of concern. The team interviewed key station personnel regarding the events, reviewed the root cause analysis, and assessed the adequacy of corrective actions. The team walked down and inspected the equipment in the field. A list of specific documents reviewed is provided in Attachment 1. The charter for the special inspection is provided as Attachment 2.

2.0 Review of the Turbine-driven Auxiliary Feedwater Pump Reset Lever Trip February 17, 2010 Background On February 17, 2010, the turbine-driven auxiliary feedwater pump FW-10 exhaust backpressure trip reset lever was seen to unlatch and trip when FW-10 was started for Surveillance Test OP-ST-AFW-004, Auxiliary Feedwater Pump FW-10 Operability Test. The turbine-driven auxiliary feedwater pump is a steam-driven pump which utilizes a Coffin turbine as a prime mover supplied by steam from the steam generators.

The turbine has an exhaust backpressure trip mechanism consisting of a trip piston actuated by a bellows connected to the turbines exhaust line, a trip latch and a reset lever. A high backpressure of 35 psig +/- 10 psig (normal backpressure is 4 to 6 psig)would cause the trip piston to extend pushing up on the trip latch lever, unlatching the trip lever latch plate from the reset lever pin. When the reset lever pin is released a spring pulls the reset lever against the mechanical governor causing the turbine to shut down.

In response to the FW-10 trip the licensee performed troubleshooting to check for blockage of the steam exhaust pipe, examined the backpressure trip linkage and checked the calibration of the backpressure trip piston. It was visually verified that there was no blockage in the accessible portions of the steam exhaust piping. Examination of the trip linkage found no abnormalities and calibration of the trip piston noted the as-found setpoint was in tolerance. The calibration work records did note that there is normally a few thousandths of an inch gap between the backpressure trip piston plunger and the trip latch, but no gap was visible, and a piece of paper could not be slid between the plunger and the trip latch. Interference between the trip piston plunger and the trip lever would interfere with a full latch between the trip lever and the reset lever.

Immediately prior to the trip, Surveillance Test IC-ST-IA-3009, Operability Test of IA-YCV-1045-C and Close Stroke Test of YCV-1045, was performed. Surveillance Test IC-ST-IA-3009 results in workers being in close proximity to the reset lever, which if inadvertently bumped could have left the reset lever not fully engaged with the trip latch.

A bump causing a partial unlatching between the trip latch and reset lever would increase the risk of normal vibrations causing a trip during the pump start. Based on this troubleshooting and three successful test runs FW-10 was returned to service.

Further troubleshooting on March 9 and 19, 2010, verified that the backpressure trip mechanism will trip if the trip latch or reset lever are bumped or vibrated. A reenactment of the partially unlatched condition was performed. The trip latch and reset lever were manually placed in a partially unlatched condition, and the pump was started. After the pump start, the trip latch and reset lever unlatched, tripping the pump off. This demonstrates that if the trip lever latch and reset lever pin are partially unlatched, the pump will trip when normal pump startup vibration occurs.

The licensee implemented an interim corrective action to control access to the cage that contains FW-10. A memo was issued on March 17, 2010, to operations personnel and management, ensuring that any plant personnel entering the FW-10 cage area are required to have a face to face briefing with the shift manager. The purpose of the briefing is to accomplish the following: 1) to increase personnel awareness of the exhaust backpressure trip mechanism location; 2) provide a review of operating experience associated with bumping the trip mechanism; 3) for personnel entering the FW-10 cage area, ensure that they have established basic measures to prevent bumping the backpressure trip mechanism; and 4) to establish a communication protocol to notify the control room when the personnel leave the FW-10 caged area. The process also requires that a spotter be present to ensure that all personnel and equipment remain at least six inches away from the exhaust backpressure reset lever. Finally, after personnel have left the caged area, operations personnel will verify the FW-10 backpressure trip lever is in its proper reset and fully latched condition. Longer term corrective actions are to install a design change to remove or protect the FW-10 backpressure trip mechanism and revise the FW-10 procedures to verify that the backpressure trip is fully engaged following any shutdown of FW-10.

a. Inspection Scope

The team evaluated the events leading up to and the licensee response following the failure to start of the Fort Calhoun turbine-driven auxiliary feedwater pump FW-10, on February 17, 2010. In order to review each area of the special inspection charter issued on March 11, 2010, the team reviewed calculations, design documents, licensing documents, work orders, modification packages, operational procedures, and corrective action documents. The team evaluated licensee compliance with the applicable regulatory requirements and applicable codes and standards. The team interviewed key station personnel from operations, design and system engineering and the corrective action program. The team assessed the licensees implementation of their corrective action program, design controls, and procedure implementation.

b. Findings

===.1

Introduction.

A Green self-revealing noncited violation of 10 CFR 50 Appendix B,===

Criterion XVI, Corrective Action, occurred for the licensees failure to assure that a condition adverse to quality was corrected. Specifically, five instances were identified where the licensee failed to correct an adverse configuration design which allowed the turbine-driven auxiliary feedwater pump FW-10 exhaust backpressure trip reset lever to be bumped and unlatched which would have prevented the pump from starting when required. The failure to correct this adverse condition resulted in the turbine-driven auxiliary feedwater pump FW-10 reset lever becoming unlatched causing the pump to trip off during a surveillance test start attempt on February 17, 2010.

Description.

On February 17, 2010, turbine-driven auxiliary feedwater Pump FW-10 exhaust backpressure trip reset lever was seen to unlatch and trip when started for Surveillance Test OP-ST-AFW-004, Auxiliary Feedwater Pump FW-10 Operability Test. The licensee initiated a significant condition adverse to quality Condition Report CR-2010-0819 to investigate the trip of the turbine-driven auxiliary feedwater pump due to an apparently inadequately latched exhaust backpressure trip mechanism.

The attached timeline of activities shows that the reset lever has been bumped and tripped five times, starting in 2001 up to a period between the January 20, 2010, successful FW-10 run and the February 17, 2010, unsuccessful FW-10 surveillance start. The bumping precursor events were different from this event in that the precursor bumping immediately tripped the lever and the turbine was not running. Two of the precursor events occurred when the plant was online resulting in FW-10 being unavailable and an unplanned entry into Technical Specification 2.5(1)B which requires the inoperable pump to be restored to service within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. Two of the events occurred during the performance of surveillance Procedure IC-ST-IA-3009, Operability Test of IA-YCV-1045-C and Close Stroke Test of YCV-1045. This test requires personnel to stand in close proximity to the turbine exhaust backpressure trip reset lever to reach an overhead valve. This surveillance test had also been performed immediately prior to the February 17, 2010, unsuccessful start attempt for the FW-10 operability test.

The licensee indicated in their root-cause analysis that the reset lever may have again been bumped, this time not enough to trip, but enough to partially unlatch the mechanism. The licensees root cause analysis determined, The FW-10 backpressure trip reset lever (FW-64-RL) is susceptible to lateral forces that can result in the trip latch (FW-64) and reset lever (FW-64-RL) becoming partially unlatched. The licensee also identified as a contributing cause that, the corrective action trending process prescribed in SO-R-2, Condition Reporting and Corrective Action, and FCSG-24, Corrective Action Program Guideline, was not effective in identifying and reducing or eliminating the susceptibility of accidentally bumping the backpressure trip latch (FW-64) or reset lever (FW-64-RL).

Sometime between the last successful FW-10 pump run on January 20, 2010, and the failure to start on February 17, 2010, the reset lever was apparently bumped enough to partially unlatch it. In this partially latched condition, FW-10 was inoperable and unavailable to support the Mitigating System Cornerstone objective of ensuring the availability and reliability of systems that respond to initiating events. The licensee has implemented interim corrective actions to verify the trip lever is fully latched after any person has entered the FW-10 cage. There is a corrective action plan to implement a design change to remove or protect the trip lever to prevent any recurrence of this failure.

Analysis.

The performance deficiency associated with this finding involved the licensees failure to correct repeated tripping of the turbine-driven auxiliary feedwater pump reset latch that demonstrated the susceptibility of the mechanism to bumping.

The finding is more than minor because it is associated with the equipment performance attribute of the Mitigating Systems Cornerstone, and affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function of a single train for greater than the technical specification allowed outage time. The finding required a Phase 2 analysis.

When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Fort Calhoun Phase 2 presolved table item, Turbine-driven Auxiliary Feedwater Pump Fails to Start, the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The senior reactor analyst performed the Phase 3 analysis (Attachment 4) and it was determined that the finding was of very low risk significance. This finding has a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensees periodic trends and assessments did not recognize the significance of precursor events related to bumping the reset lever and prompt action to prevent further problems with the turbine-driven auxiliary feedwater pump FW-10

P.1(b).

Enforcement.

Title 10 CFR, Part 50, Appendix B, Criterion XVI, Corrective Action, requires, in part, that conditions adverse to quality are identified and corrected. Contrary to this, repeated trips of the turbine-driven auxiliary feedwater pump FW-10 exhaust backpressure trip mechanism, due to bumping, occurred without the condition being corrected. Because the violation was of very low safety significance, was not repetitive or willful, and was entered into the licensees corrective action program as Condition Report CR-2010-813, this violation is being treated as a noncited violation, consistent with the NRC Enforcement Policy Section VI.A.1: NCV 05000285/2010006-01, Failure to Correct Repeated Tripping of the Turbine-Driven Auxiliary Feedwater Pump FW-10.

===.2

Introduction.

The team identified a Green noncited violation of Technical===

Specification 5.8.1.a regarding the licensees failure to implement written procedures as recommended in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978.

Paragraph 3.l of Appendix A requires procedures for startup, shutdown and operation of the auxiliary feedwater system. Specifically, the licensee had no procedural guidance to verify full engagement of the turbine-driven auxiliary feedwater pump FW-10 exhaust backpressure trip mechanism when latched. This resulted in the licensees failure to identify the partially latched condition of the FW-10 turbine exhaust backpressure trip mechanism which subsequently vibrated loose during a surveillance test causing a start failure of FW-10, on February 17, 2010.

Description.

On December 17, 2009, the turbine-driven auxiliary feedwater pump FW-10 was run per OP-ST-AFW-3011, Auxiliary Feedwater Pump FW-10, Steam Isolation Valve, and Check Valve Tests. The backpressure trip mechanism is reset per step 7.60 but does not require verification of a fully latched condition. On December 18 and 26, 2009, and on January 20, 2010, FW-10 was run per surveillance test Procedure OP-ST-AFW-004, Auxiliary Feedwater Startup and System Operation, and while the backpressure trip latch was not manipulated there were no procedural steps to verify the backpressure trip was fully latched after operation. On February 17, 2010, the turbine-driven auxiliary feedwater pump FW-10 exhaust backpressure trip reset lever was seen to unlatch and trip when started for Surveillance Test OP-ST-AFW-004. The licensee initiated a significant condition adverse to quality Condition Report CR-2010-0819 to investigate the trip of the turbine-driven auxiliary feedwater pump due to an inadequately latched exhaust backpressure trip mechanism.

The inspectors noted that industry guidance on turbine-driven auxiliary feedwater pump equipment recommends a daily walkdown to perform, among other checks, a verification of proper system and turbine valve lineup, paying particular attention to the turbines . . . trip latch engagement . . . . The inspectors have found similar instances where the licensees singular use of a Curtis turbine-driven auxiliary feedwater pump has led them to not fully utilize relevant operating experience and industry recommendations for similar equipment. In this case the failure to procedurally require verification of a fully latched condition of their trip mechanism resulted in the February 17, 2010, failure to start of the turbine-driven auxiliary feedwater pump and a resulting loss of availability and reliability of a system that is required to respond during an event to prevent undesirable consequences, an objective of the Mitigating Systems Cornerstone. The licensee has initiated interim corrective actions to control access to the FW-10 cage, including actions to verify that the exhaust backpressure trip mechanism is fully latched after any cage entry. In addition, the licensee has taken actions to revise operation procedures for FW-10 to require verification that the exhaust backpressure trip mechanism is fully latched.

Analysis.

The failure to have an adequate procedure to assure that the auxiliary feedwater pump FW-10 exhaust backpressure trip mechanism is fully latched, following maintenance and/or operational activities on or near the pump, is a performance deficiency. This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The finding has a crosscutting aspect in the area of problem identification and resolution associated with operating experience because the licensee failed to implement and institutionalize operating experience through changes to station operating procedures when they failed to incorporate industry information to verify the turbine-driven auxiliary feedwater pump is fully latched P.2(b).

Enforcement.

Technical Specification 5.8.1.a requires that licensees establish, implement and maintain the applicable procedures recommended in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978. Section 3.l. of Appendix A requires procedures for startup, shutdown and operation of the auxiliary feedwater system.

Contrary to the above, until February 17, 2010, the licensee turbine-driven auxiliary feedwater pump FW-10 operating procedures and access control procedures did not verify that the pumps turbine exhaust backpressure trip mechanism was fully latched.

Because of the very low safety significance of this finding and because the licensee has entered this issue into their corrective action program as Condition Report 2010-0813, this violation is being treated as a noncited violation in accordance with Section VI.A.1 of the Enforcement Policy: NCV 05000285/2010006-02, "Failure to Verify That the Turbine-driven Auxiliary Feedwater Pump Exhaust Backpressure Trip Lever was Fully Latched.

3.0 Review of other Recent Turbine-driven Auxiliary Feedwater Pump Failures Background In addition to the February 17, 2010, trip of the turbine-driven auxiliary feedwater pump FW-10 on high turbine exhaust backpressure, two other FW-10 failures have occurred recently. On April 6, 2009, FW-10 tripped approximately one minute after the pump started during an increased frequency surveillance test start using Procedure OP-ST-AFW-004, Auxilary Feedwater Pump FW-10 Operability Test.

Earlier that year, on February 26, 2009, FW-10 failed to start when performing Procedure OP- ST-AFW-004 for postmaintenance testing following Construction Work Order 322162-01 maintenance.

Failure to Vent Air Following Maintenance.

Licensee Root Cause Analysis 2009-0905 investigation into the February 26, 2009, failure to start of FW-10 noted that a 2001 design change, EC 14994, FW-10 Reliability Enhancements, created a new high point in the control oil system. As part of the design change, the external sweated copper tubing on the lube oil system and control oil system was replaced with stainless steel tubing and Swagelok fittings. During this replacement, the craft installed the tubing using prefabricated runs and field routed the tubing as necessary. As a result, the lube oil line from the mechanical oil pump to the governor speed control was installed with two high points, one high point in the line supplying the oil relay and one drip line supplying the governor hood. When this change was made in 2001 engineering did not know that the oil tubing routing was a critical factor. The concerns with high points in a system are: 1) that during idle periods, air can be introduced into the system through a leak in one of the tubing connections; or 2) if the tubing is removed for maintenance and the fluid in the system drains from any high points. As a result, an air slug can be introduced into the system during the initial operation of the affected system. The inspection team has noted at other plants that work instructions, vendor manuals and operational experience on turbine-driven auxiliary feedwater pump governor hydraulic systems require various methods to ensure air is removed prior to returning the equipment to operational status. On February 26, 2009, work was performed to change the mechanical governor speed setpoint to raise turbine speed slightly. This required removing tubing connected to the governor hood.

Following maintenance there were no instructions to ensure air was vented from the control oil system and though a maintenance run was performed prior to the operability run it was not sufficient to remove the trapped air. When the turbine was started for the operability test, the air slowed the control oil system response resulting in a slow sluggish turbine start that was then terminated by plant operations.

Inadequate Design Margin On February 26, 2009, a design change to increase the discharge pressure of the turbine-driven auxiliary feedwater pump FW-10 from 1170 psig to 1210 psig was implemented by increasing the turbine governor speed setpoint. When initial governor speed adjustment was completed and the pump was run to check the result, discharge pressure was found slightly above the desired 1210 psig with actual pressure being 1240 psig. Due to the work being performed within a 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> technical specification limiting condition for operation the licensee investigation concluded that there was reluctance to perform another adjustment to lower speed due to time pressure. Instead, it was decided to accept the 1240 psig pump discharge pressure and another design change was made to document 1280 psig as the maximum discharge pressure limit.

This design change also evaluated the new smaller 170 psig margin to the high pump discharge pressure trip setpoint of 1450 psig and incorrectly concluded it was adequate.

The licensee staff had no information on the normal turbine startup transient response affect on discharge pressure relative to the steady state pump discharge pressure.

During a subsequent pump surveillance test FW-10 tripped when the high discharge pressure trip setpoint was reached during the pump startup transient.

a. Inspection Scope

The team evaluated the events leading to and the licensee response following these two failures of the Fort Calhoun turbine-driven auxiliary feedwater pump. In order to review each area of the special inspection charter issued on March 11, 2010, the team reviewed calculations, design documents, licensing documents, work orders, modification packages, and corrective action documents. The team evaluated licensee compliance with the applicable regulatory requirements and applicable codes and standards. The team interviewed key station personnel from operations, design and system engineering and the corrective action program. The team assessed licensee implementation of their corrective action program, design controls, and procedure implementation.

b. Findings

===.1

Introduction.

A Green self-revealing noncited violation of Technical Specification 5.8.1.a===

was identified regarding the licensees failure to implement and maintain the applicable procedures recommended in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978. Paragraph 9.a of Appendix A requires that such maintenance that can affect the performance of safety-related equipment be properly preplanned and performed in accordance with documented instructions. Specifically, the licensee failed to have an adequate procedure for ensuring air was vented from the turbine-driven auxiliary feedwater pump control oil system following maintenance. As a result, the turbine-driven auxiliary feedwater pump failed to start during an operability test.

Description.

On February 26, 2009, the turbine-driven auxiliary feedwater pump FW-10 failed to start (achieve normal discharge pressure and flow) when given a start signal during an operability test. This test was performed as part of the postmaintenance testing following Construction Work Order 322162-01 maintenance to increase pump discharge pressure slightly by adjusting the governor setpoint and, therefore, operating speed of the pump.

In response to the failure of FW-10 to start, operations personnel declared the pump inoperable and entered Technical Specification 2.5(1)B, with a 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> limiting condition for operation. Based on troubleshooting immediately following the event, the cause for the failure of the pump was indeterminate. The pump was later declared operable at 11:55 p.m. on February 26, 2009, based on no deficient components identified during troubleshooting and subsequent multiple successful pump starts.

The licensees Condition Report CR-2009-0905 root cause analysis determined that FW-10 failed to start because air was present in the turbines control oil system. The air in the control oil system slowed down the response of the oil relay piston that opens the turbines governor valve. A modification to the FW-10 oil control system was installed in 2001 that replaced the control oil systems copper tubing with stainless steel. During this modification, new stainless steel tubing was installed with high points that would not self-vent following maintenance that could have allowed air into the system. Engineering staff did not understand that this was a critical factor when implementing the design change and resulted in an oil system that could not self-vent and therefore, was sensitive to air binding.

During the February 26, 2009, maintenance to adjust turbine speed, the tubing between the control oil system and the mechanical governor was disconnected to adjust the governor which allowed air to enter the control oil system. The licensee did not have a procedure in place to vent air from the control oil system following the maintenance.

This resulted in a failure to start during a surveillance of a safety-related system and additional unavailability time due to troubleshooting following the failure. This affected the Mitigating System Cornerstone objective of ensuring availability and reliability of systems that respond to initiating events. It is a standard industry practice to vent air from control oil and hydraulic systems following maintenance that introduces air into the system. On systems with high points that are difficult to vent, additional effort must be taken to assure all air is vented from the system before attempting to restore operability.

The licensee had no programmatic procedures/processes to address air venting following maintenance on FW-10. The licensee corrective actions to address this issue did replace the tubing to remove the high point and make the system more likely to self-vent.

Analysis.

The performance deficiency associated with this finding involved the licensees failure to have adequate instructions in Construction Work Order332162-01 to ensure air was vented from the control oil system following maintenance. The finding is more than minor because it is associated with the Mitigating System Cornerstone attribute of procedure quality and adversely affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage). Using Manual Chapter 0609.04, "Phase 1 Initial Screening and Characterization of Findings," the finding was found to have very low safety significance (Green) because it was not a design deficiency; did not represent loss of a safety function, loss of a single train for greater than its allowed outage time, or loss of a non-technical specification train of equipment; and did not screen as potentially risk significant due to a seismic, flooding, or severe weather initiating event. The finding has a crosscutting aspect in the area of human performance associated with conservative assumptions due to the licensee failue to identify possible unintended consequences of high points in a control oil system tubing design change that could become air bound and interfere with fast starts of the turbine-driven auxiliary feedwater pump H.1(b).

Enforcement.

Technical Specification 5.8.1.a requires that licensees establish, implement, and maintain the applicable procedures recommended in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978. Paragraph 9.a of Appendix A requires that such maintenance that can affect the performance of safety-related equipment be properly preplanned and performed in accordance with documented instructions. Contrary to this requirement, on February 26, 2009, the licensee did not provide a process to ensure air was vented from the turbine-driven auxiliary feedwater pump control oil system during maintenance. As a result, the turbine-driven auxiliary feedwater pump failed to start during an operability test. Because this violation was of very low safety significance and it was entered into the licensees corrective action program as Condition Report CR-2009-0905, this violation is being treated as a noncited violation in accordance with Section VI.A.1 of the Enforcement Policy:

NCV 05000285/2010006-03, Failure to Vent Control Oil Following Maintenance Results in Failure of the Turbine-driven Auxiliary Feedwater Pump to Start.

===.2

Introduction.

A self revealing, Green, noncited violation of 10 CFR, Part 50, Appendix B,===

Criterion III, Design Control, occurred when the licensee failed to ensure that the design basis of certain structures, systems and components were translated into specifications, drawings, procedures, and instructions when implementing Engineering Change 45105. Specifically, this design change reduced the turbine-driven auxiliary feedwater pumps margin between the pump discharge pressure and the pumps high discharge pressure trip setpoint resulting in the April 6, 2009, high pump discharge pressure trip during a scheduled surveillance test start.

Description.

The turbine-driven auxiliary feedwater pump, FW-10, had a pneumatic speed control system that was removed to improve pump reliability. This changed the turbine control system from a variable speed drive to a constant speed drive.

Implementation of this design change required the licensee staff to determine a mechanical governor speed setpoint to allow the pump discharge pressure to meet auxiliary feedwater system head and flow requirements. The licensee staff did not properly determine the auxiliary feedwater hydraulic friction losses and as a result, when implemented in the June 2008, design change EC 34435, it resulted in a low mechanical speed limiting governor speed setpoint of 7600 rpm. This left FW-10 operating with a discharge pressure near the low limit of the pumps surveillance test acceptance criteria.

This may not have left enough design steam generator flow in the event the pump minimum recirculation flow valve failed open.

On February 26, 2009, FW-10 turbine speed was raised to 7900 rpm by design change EC 45105 to obtain a discharge pressure of 1210 psig. These actions were required to obtain a discharge pressure design margin to the surveillance test minimum pump discharge pressure acceptance criteria and addressed required forward flow with the minimum recirculation valve open. However, once again the 7900 rpm pump speed did not provide the expected pump discharge pressure results when a steady state discharge pressure of 1240 psig was reached, 30 psig above the target discharge pressure of 1210 psig. Due to the 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> technical specification action statement time limit it was decided to revise the acceptance criteria in design change EC 45105 to allow a new maximum pressure limit of 1280 psig. While this prevented the need for further adjustments to the mechanical governor speed setpoint it resulted in only a 170 psig margin to the FW-10 high pump discharge pressure trip set point of 1450 psig. Licensee staff lacked knowledge on typical transient pressure response that could be expected during the startup transient of the turbine-driven auxiliary feedwater pump compared to the steady state discharge pressure and this resulted in the staff making the nonconservative judgment that the 170 psig margin was adequate.

During an emergency start of the turbine-driven auxiliary feedwater pump, the steam admission valve opens, allowing a sudden inrush of steam into the governor valve and turbine. This speeds up the turbine at the same time the attached mechanical governor is speeding up and pressurizing the control oil system. This typically results in some amount of overshoot past the steady state speed until the control system brings speed back down to the setpoint. Since the turbine is driving the pump, these speed changes are seen as pump discharge pressure changes in the auxiliary feedwater piping downstream of the pump. The pump high discharge pressure trip functions as the turbine overspeed protection device and protects downstream piping from overpressure if a loss of speed control occurs. A normal startup transient may have some momentary slightly higher pressures than steady state pump discharge pressure, further reducing the margin between pump discharge pressure and the high pump discharge pressure trip setpoint. The amount of startup transient overshoot will have some variation depending on variables related to turbine and piping standby temperature, inlet steam quality, and other conditions. The inspectors noted that the licensees turbine-driven auxiliary feedwater pump performance monitoring does include a recommended industry practice of recording pertinent transient data. Review of the recorded transient data allows for a detailed evaluation of a surveillance test, trending evaluation of a specific test from a series of tests, and supports troubleshooting required to evaluate the malfunction or degradation of components or system problems due to abnormal transients.

Following the February 26, 2009, design change that increased pump discharge pressure, FW-10 had been on an increased frequency of weekly surveillance runs with six successful surveillance tests performed up until April 6, 2009. Then FW-10 tripped approximately 1 minute after the pump started due to high discharge pressure. The 170 psig margin coupled with an increased startup transient overshoot resulted in the pump trip. The licensee did not have the transient pressure data to conclusively determine if 170 psig was an adequate margin. This resulted in a run failure of the turbine-driven auxiliary feedwater pump and additional unavailability to correct the inadequate design margin which impacted the Mitigating Systems Cornerstone objective to ensure the availability, reliability and capability of systems that respond to initiating events. The inadequate margin was introduced on February 26, 2009, and existed until April 6, 2009. Though there were six successful weekly runs prior to the seventh, the lack of knowledge on the startup transient pressure changes indicates the trip could have occurred on any of the starts and the empirical knowledge that one out of seven starts could result in a failure during this exposure period. When a Phase 3 risk analysis was reviewed with this failure and recovery probability during the six week exposure period it was determined that there was still a very low risk significance associated with this event.

Analysis.

The performance deficiency associated with this finding involved the licensees failure to assure that the turbine-driven auxiliary feedwater pump design margin was correctly maintained during design changes affecting pump discharge pressure. The inspectors determined the finding is more than minor because it is associated with the design control attribute of the Mitigating Systems Cornerstone, and adversely affected the cornerstone objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage). Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function of a single train for greater than the technical specification allowed outage time. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Fort Calhoun Phase 2 presolved table item, Turbine-driven Auxiliary Feedwater Pump Fails to Start, the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The senior reactor analyst performed the Phase 3 analysis, Attachment 4, and determined that the finding was of very low risk significance. The finding has a crosscutting aspect in the area of human performance because the licensee failed to use conservative assumptions in decision making when a nonconservative design margin was approved and implemented on the turbine-driven auxiliary feedwater pump H.1(b).

Enforcement.

Title 10 CFR, Part 50, Appendix B, Criterion III, Design Control, requires, in part, that established measures shall assure that the design basis is correctly translated into specifications. Contrary to that, on February 26, 2009, design change EC 45105 adjusted the speed of FW-10 to 7900 rpm resulting in an inadequate margin to prevent a high discharge pressure trip of FW-10, that occurred April 6, 2009. This condition was corrected April 6, 2009, when the FW-10 high discharge pressure switch setpoint was raised from 1450 psig to 1600 psig to restore margin. Because the violation was of very low safety significance, was not repetitive or willful, and was entered into the licensees corrective action program as Condition Report CR-2009-0905, this violation is being treated as an noncited violation, consistent with the NRC Enforcement Policy Section VI.A.1: NCV 05000285/2010006-03, Turbine-driven Auxiliary Feedwater Pump Trip Due to Inadequate Design Margin.

4.0 Review of Root Cause Analysis, Extent of Condition and Corrective Actions On February 17, 2010, the licensee established a root cause analysis team to investigate the facts and identify the causes associated with the failure of the turbine-driven auxiliary feedwater pump FW-10. The team conducted their review in accordance with Procedure NOD-QP-19, Cause Analysis Program. The licensees procedure requires the team to:

• Identify and validate root and contributing causes

• Conduct an extent of condition review

• Determine extent of cause

• Develop corrective actions The licensees root cause analysis Revision 1 of Condition Report CR-2010-0813, Steam Driven Auxiliary Feedwater Pump (FW-10) Tripped Off, was completed on April 24, 2010.

As part of the special inspection charter the inspectors also reviewed the following documents:

• CR-2009-0905, Steam Driven Auxiliary Feedwater Pump (FW-10) Failed to Achieve Normal Discharge Pressure and Flow

• CR-2009-1611, Steam Driven Auxiliary Feedwater Pump (FW-10) Tripped Off

• CR-2010-0813, Steam Driven Auxiliary Feedwater Pump (FW-10) Tripped Off

• CR 2010-0910, NRC-6 Performance Indicator, MSPI Auxiliary Feedwater System, Turned FCS Red Root Cause Methodology The licensee performed a root cause analysis to identify the causes of the failures of the turbine-driven auxiliary feedwater pump in 2009 and 2010. As part of their review, the licensee performed extent of condition and extent of cause reviews. The root cause utilized a structured root cause analysis method in accordance with Procedure NOD-QP-19, Cause Analysis Program. The analysis includes the event sequence (timeline), a systematic method to indentify the root/contributing causes (events and casual factors, barrier analysis, fault tree analysis, change analysis, etc.)

and the method used to gather data and what data was gathered.

Root Cause and Corrective Actions to Prevent Recurrence Condition Report CR-2009-0905. The root cause analysis for Condition Report CR-2009-0905 determined through troubleshooting, causal factor analyses and re-enactment that the root cause of the February 26, 2009, FW-10 failure to start was, An air slug trapped under the FW-10 oil relay piston slowed the oil relay pistons resonance time. A contributing cause to the event was, The high point created in the FW-10 oil system during installation of DC 14994 may contribute to prolonged fast speed starts of FW-10. The high point in the oil system tubing was not identified by the licensee, but by a Coffin Turbo Pump representative hired to assist in troubleshooting FW-10. Corrective actions to prevent recurrence include a design change that eliminated the high points and tubing slope to assist with self-venting, and procedure changes to direct operators to allow up to 2 minutes for the pump to start prior to tripping the turbine. The licensee evaluated their loss of feedwater accident scenario and increased the analyzed FW-10 start times from 50.9 seconds to 2 minutes.

Air in any oil or hydraulic control system would be expected to affect that systems performance. However, the licensee investigation determined there was no extent of condition when it found no operational experience with electrohydraulic controlled pumps and valves or the emergency diesel generator governor failures related to air intrusion.

The extent of the root cause investigated a slug of air in oil affecting electrohydraulic control of pumps and valves and emergency diesel generators and while no instances of air problems were found the licensee did identify a similar cause of foreign material adversely impacting operation of a control system. This resulted in a corrective action to protect against foreign material contamination of electrohydraulic pumps/valves and emergency diesel generator governors.

Condition Report CR-2009-1611. The root cause analysis for Condition Report CR-2009-1611 used event sequence, causal factor analyses and failed barrier identification to determine the root cause of the April 6, 2009, FW-10 trip on startup was, EC 45105 (FW-10 Speed Limiting Governor Setting) and FDCR 45607 (Revise the Acceptance Criteria I EC 45105) reduced the discharge pressure margin to trip setpoint resulting in FW-10 tripping on high discharge pressure. Due to a lack of knowledge of the operating characteristics of FW-10, licensee staff made the judgment that the reduced discharge pressure margin to trip setpoint was adequate. Corrective actions included increasing the discharge pressure margin and additional guidance for control of engineering design changes.

The licensee determined there was no extent of condition since there was no internal operating experience of similar pumps that had failed due to high discharge pressure.

There is an extent of cause that other plant configuration changes or engineering analysis have had an adverse impact on equipment, plant processes, or human performance. This had been addressed by corrective actions that improved definitions of engineering changes, transferred responsibility for determining the type of change process to be used from system to design engineering, and disallowed the use of field design changes for significant changes, Condition Report CR-2010-0813. The root cause analysis for Condition Report CR-2010-0813 used event sequence, causal factor analyses and failed barrier identification to determined the root cause of the February 17, 2010, FW-10 failure to start to be, The FW-10 backpressure trip reset lever (FW-64-RL) is susceptible to lateral forces that can result in the trip latch (FW-64) and reset lever (FW-64-RL) becoming partially unlatched. Internal operating experience found that since the year 2000, FW-10 has been inadvertently tripped by bumping the trip latch or reset lever five times. A contributing cause was an ineffective trending process that should have identified and reduced, or eliminated, the susceptibility of accidentally bumping the backpressure trip.

Corrective actions to prevent reoccurrence include an interim action that controls access to FW-10 and verifies the trip mechanism to be fully latched prior to leaving the FW-10 area, a design change to remove or protect the backpressure trip mechanism, enhancements to the station trending program, a procedure revision to include the interim access controls into the licensee equipment status control procedure and revisions to all FW-10 operating procedures to verify the backpressure trip latch is fully engaged following shutdown of FW-10.

The FW-10 trip due to the apparent backpressure trip unlatching has an extent of condition that includes any Fort Calhoun Station procedure that operates the turbine-driven auxiliary feedwater pump. While the extent of cause would include any mechanical trip device at the plant that could be bumped or forced, a review of other plant equipment did not find any mechanical devices with similar vulnerable characteristics.

Condition Report CR-2010-0910. As a result of the three turbine-driven auxiliary feedwater pump FW-10 failures since February 2009 the licensees internal performance indicator for the auxiliary feedwater system turned red. The root cause analysis for Condition Report CR-2010-0910 determined that the root cause of these three recent FW-10 failures was due to the fact that the licensee, lacks the technical knowledge of the FW-10 control systems and operating characteristics that is required to ensure that the Engineering Change process does not introduce errors when making configuration changes to FW-10. Corrective actions include additional training for licensee engineering staff.

a. Inspection Scope

The team reviewed the licensees root cause analysis to determine if it was conducted to a level of detail commensurate with the significance of the problem. The team reviewed the licensees corrective actions to ensure they addressed the extent of condition and whether they were adequate to prevent recurrence. The team interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program.

b. Findings and Observations

The inspectors determined that the licensees analysis accurately captured the root causes of the events. The inspectors found that the corrective actions to prevent recurrence would likely ensure improved turbine-driven auxiliary feedwater pump reliability. Effectiveness reviews were implemented by the licensee to ensure the corrective actions implemented were sufficiently robust to address the root causes. The inspectors did note that some of the root cause analysis reports were narrowly focused and failed to consider all potential causal factors and some corrective action enhancements. Specifically, the inspectors noted the following:

• The licensees review of the three turbine-driven auxiliary feedwater pump failures in Condition Report CR 2010-0910 root cause analysis correctly determined the common thread between the causes of these pump failures was that the licensee lacks the technical knowledge of the FW-10 control systems and operating characteristics that is required to ensure that design changes do not introduce errors when making configuration changes to FW-10. The corrective actions the licensee proposes to correct this condition is additional training for the system and design engineering personnel. However, effective training on operating characteristics would be hampered by the lack of licensee performance monitoring during FW-10 starts that prevent the licensee from knowing the baseline operating characteristics of turbine transient performance.

The inspection team informed the licensee of operating experience, industry standard practices and industry guides that recommend these practices. As a result, the licensee initiated Condition Report CR 2010-2424 to document this issue and track proposed changes to allow instrumentation of FW-10.

• The root cause investigation into the backpressure trip lever unlatching was narrowly focused on the bumping of the latch as the most likely cause.

However, licensee troubleshooting was unable to duplicate a partially latched condition by bumping the trip or reset levers. The inspectors noted that two other possible causes are the possibility of interference between the trip piston plunger and the trip lever or an inadequate full latch when reset. The possibility of a partial latch during reset will be prevented by procedure revisions to all FW-10 operating procedures to verify full latch following operation of FW-10. The design change to remove or protect the backpressure trip mechanism will address the bumping cause. The root cause did evaluate the condition of the trip piston touching the trip lever but did not consider this a likely cause. However, the team noted that any interference between the trip piston and the trip lever would prevent the trip lever latch plate from fully engaging with the reset lever pin. The inspection team discussed this with the licensee and as a result, actions were added to specify a specific gap dimension and tolerance between the FW-10 backpressure trip piston and backpressure trip lever to ensure a gap remains and the piston plunger cannot interfere with the trip latch engagement.

• The inspectors noted that the use of communication memo as an interim action to control access to the FW-10 cage was a minor violation of Technical Specification 5.8.1.a that requires written procedures be implemented as recommended in Regulatory Guide 1.33, Revision 2, Appendix A. section 1.c.

equipment control. The communications memo was being used as an interim measure to control access the FW-10 cage when previous access control measures had been proven ineffective. Since the memo appeared to be used as a method to maintain FW-10 operability Technical Specification TS 5.8.1 was applicable. When informed of this issue, operations reissued the memo to control access to the FW-10 cage using the programmatic requirements which included a 50.59 screening. These controls will remain in place until the corrective action is completed that will revise SO-G-113 (Equipment Status Control) to proceduralize the memo guidance on FW-10 access control.

• The root cause investigation into the backpressure trip stated that vibrations from the running pump could contribute to the trip unlatching. The team questioned the licensee if this would affect the seismic qualification or robustness of FW-10 and if there was any possible concern with impact on the sensitive backpressure trip mechanism from external objects during a seismic event. As a result the licensee verified the seismic adequacy of FW-10 and documented this review in Revision 1 to the Condition Report CR-2010-0813 root cause analysis report.

5.0 Review of Evidence Preservation Process

a. Scope

The team reviewed the licensees process for quarantine of affected components and equipment to preserve important characteristics needed for critical failure analysis.

Evidence is needed to determine failure cause and determine appropriate corrective actions to prevent recurrence of failures.

The team reviewed the licensees history of evidence preservation issues documented in the condition report database. In particular, the team reviewed station procedures, processes and determined if the licensee process was effective at evidence preservation. The team interviewed key station personnel from operations, design, and licensing, and the corrective action program.

b. Observations and Findings

The licensee identified deficiencies in its evidence preservation process in Condition Report CR 2009-6897 when evidence was lost during an apparent cause investigation of a failed air accumulator. Though the investigation was completed without the evidence, additional actions were taken to make improvements in the sites NOD-QP-19, Cause Analysis Program, guidance procedure. These improvements were attempts to specify the requirements for the collection and protection of evidence as it relates to casual analysis at Fort Calhoun. However, using NOD-QP-19 to control evidence is of limited use when the station is in the discovery phase of an equipment problem that usually involves operations and maintenance personnel and procedures. The licensee Quality Assurance department noted this issue in February of 2010 and it was documented in Condition Report CR 2010-0799 that the procedure improvement corrective action for Condition Report CR 2009-6897 did not directly address the apparent cause of insufficient guidance for plant personnel involved in the planning of work where the need for preserving physical evidence is required. It recommended that processes for evidence collection and preservation be determined and incorporated into station guidance related to work planning, troubleshooting, and conduct of maintenance.

However, the licensees response considered the changes to the cause analysis program procedure as adequate and noted that Condition Report CR 2009-6897 does have an effectiveness review to check for any physical evidence lost, or compromised condition reports, in about one year.

The team also interviewed plant operations personnel on evidence preservation during the February 17, 2010, trip of the turbine-driven auxiliary feedwater pump, FW-10, and noted that operations personnel were aware of the importance of evidence preservation by minimizing operation or manipulation of failed components as much as possible consistent with keeping the plant in a safe configuration. In the case of the FW-10 trip, this resulted in operations personnel deciding not to reset the tripped exhaust backpressure reset lever but to wait on an appropriate investigation team to examine the equipment. These efforts are expected to reduce the chances for lost evidence in future Fort Calhoun cause analysis investigations.

6.0 Review of Operating Experience

a. Scope

The team reviewed internal operating experience by obtaining a list of plant corrective action documents related to the auxiliary feedwater system and selecting those documents related to failures of the turbine-driven auxiliary feedwater pump. The team inspected the licensees review of industry operating experience for the turbine-driven auxiliary feedwater pump. The team inspection included a specific review of related operating experience during the root cause investigation for the February 17, 2010, trip of the turbine-driven auxiliary feedwater pump.

For external operating experience, the NRC Operating Experience Branch provided the results of keyword searches related to exhaust backpressure trips, such as: trip mechanisms unlatching due to wear or worn edges of the mechanism (trip tappet/head lever), inadequate engagement of the mechanism (trip tappet nut/head lever) due to improper setup, inadequate engagement due to failure to properly reset the mechanism, or excessive vibration unlatching trip mechanisms; overshoot of normal operating speed resulting trips due to water, binding, or improper setpoints; speed control problems, slow starts or overspeed trips due to air in control oil or hydraulic control systems following maintenance. The NRC Operating Experience Branch provided a list of licensee event reports, NRC information notices, NUREG documents and other operating experience information. The team selected operating experience information that was applicable to this inspection and reviewed whether the licensee had addressed the items in their root cause analyses related to these events or had processed the information through their operating experience program.

b. Findings and Observations

Using a Coffin turbine to drive the auxiliary feedwater pump is unique in the United States commercial nuclear industry. Due to Fort Calhoun having the only Coffin turbine in the US nuclear industry the inspection of the licensee industry operational experience searches rarely turn up relevant information. However, the NRC operating experience branch did supply relevant operational experience from plants that use different turbines to drive their auxiliary feedwater pumps and relevant hydraulic controls experience. One example was a January 2003 operational experience account of an auxiliary feedwater pump which tripped on overspeed following a demand start due to a design margin that had the overspeed trip setpoint too close to operating speed. This is very similar to the Fort Calhoun April 6, 2009, high discharge pressure trip during a pump start due to inadequate design margin. Other relevant recommended standard industry practices such as transient performance monitoring is not performed at Fort Calhoun and has contributed to a lack of understanding by the station personnel of the normal transient responses of the Coffin turbine-driven auxiliary feedwater pump. There were several experiences associated with failures of feedwater and main steam hydraulically operated isolation valves due to inadequate venting of oil systems following maintenance.

Information Notice 86-07 describes a loss of speed control on a standby diesel generator due to inadequate filling and venting of the governor hydraulic control system following maintenance. These are relevant to the licensees 2009 FW-10 failure to start following inadequate venting of control oil following maintenance.

7.0 Potential Generic Issues

a. Scope

The team evaluated the failures of Fort Calhouns turbine-driven auxiliary feedwater pump on whether any potential generic issues should be communicated to the industry (e.g., information notices, generic letters, and bulletins).

b. Findings and Observations

The team determined that this issue warrants a generic communication informing other licensees of the types of problems encountered. Specifically, the team determined that, though the design of the Coffin turbine-driven auxiliary feedwater pump trip is unique in the US nuclear power plant industry, unlatching of the a trip mechanism from bumping, wear, or improper latching is similar to industry events with other turbine-driven auxiliary feedwater pump overspeed trip mechanisms. The team will discuss this issue with the NRC Office of Nuclear Reactor Regulation for issuance on an information notice. The team did not identify any other potentially generic safety issues during the inspection.

4OA6 Meetings, Including Exit

On March 26, 2010, the team presented the preliminary results of this inspection at the end of the onsite week to Mr. David Bannister, Vice President Nuclear and Chief Nuclear Officer, and other members of his staff who acknowledged the findings. The team verified that no proprietary information was retained.

On June 3, 2010, the team leader presented the final results of the inspection to Mr. Jeff Reinhart, Site Vice President, and other members of the licensee staff who acknowledged the findings. The team verified that no proprietary information was retained.

On June 30, 2010, the Project Branch 3 Chief presented the revised results of the inspection to Mr. Jeff Reinhart, Site Vice President, and other members of the licensee staff who acknowledged the findings. The team verified that no proprietary information was retained.

1: Supplemental Information 2: Special Inspection Charter 3: Timeline associated with TDAFW Pump problems 4: Final Significance Determination Evaluation

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

R. Acker, Licensing Engineer, Licensing Department

J. Cate, Supervisor, System Engineering Department

J. Drahota, Supervisor, Reliability Engineering

H. Faulhaber, Manager, Nuclear Engineering Division

M. Ferm, Manager, Operations Support

M. Frans, Manager, System Engineering Department

J. Gasper, Manager, Design Engineering Department

J. Goodell, Manager, Nuclear Performance Improvement and Support Department

D. Guin, Supervisor, Regulatory Compliance

D. Haas, Design Engineer, Design Engineering Department

R. Haug, Manager, Training Department

J. Herman, Manager, Engineering Programs Department

E. Jun, System Engineer, System Engineering Department

E. Matzke, Licensing Engineer, Licensing Department

T. Matthews, Manager, Nuclear Licensing Department

T. Nellenbach, Plant Manager

T. Pilmaier, Manager, Performance Improvement Department

B. Venters, Superintendent, Maintenance Department

J. Zagata, Reliability Engineer, Reliability Engineering Division

S. Swearngin, Supervisor, Mechanical Engineering Division

NRC Personnel

J. Robles, Reactor Systems Engineer NRR/DIRS/IOEB Operating Experience Branch

R. Sigmon, Reactor Systems Engineer NRR/DIRS/IOEB Operating Experience Branch

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

050000285/2010006-01 NCV Failure to Correct Repeated Tripping of the Turbine-driven Auxiliary Feedwater Pump FW-10 (Section 1.2.1)

050000285/2010006-02 NCV Failure to Verify that the Turbine-driven Auxiliary Feedwater Pump Exhaust Backpressure Trip Lever was Fully Latched (Section 1.2.2)

050000285/2010006-03 NCV Failure to Vent Control Oil Following Maintenance Results in Failure of the Turbine-driven Auxiliary Feedwater Pump to Start (Section 1.3.1)

050000285/2010006-04 NCV Turbine-driven Auxiliary Feedwater Pump Trip Due to Inadequate Design Margin (Section 1.3.2)

Attachment 1