IR 05000483/2009009: Difference between revisions
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
||
| Line 1: | Line 1: | ||
{{Adams|number = ML092730656}} | {{Adams | ||
| number = ML092730656 | |||
| issue date = 09/30/2009 | |||
| title = IR-05000483-09-009 on 06/22/2009 - 08/27/2009 for Callaway Plant; Preliminary White Finding | |||
| author name = Chamberlain D D | |||
| author affiliation = NRC/RGN-IV/DRP | |||
| addressee name = Heflin A C | |||
| addressee affiliation = AmerenUE | |||
| docket = 05000483 | |||
| license number = NPF-030 | |||
| contact person = | |||
| case reference number = EA-09-200 | |||
| document report number = IR-09-009 | |||
| document type = Inspection Report, Letter | |||
| page count = 61 | |||
}} | |||
{{IR-Nav| site = 05000483 | year = 2009 | report number = 009 }} | {{IR-Nav| site = 05000483 | year = 2009 | report number = 009 }} | ||
| Line 7: | Line 22: | ||
[[Issue date::September 30, 2009]] | [[Issue date::September 30, 2009]] | ||
EA-09-200 Mr. Adam C. Heflin, Senior Vice President and Chief Nuclear Officer AmerenUE P.O. Box 620 Fulton, MO 65251 Subject: NRC SPECIAL INSPECTION REPORT 05000483/2009009 - CALLAWAY PLANT; PRELIMINARY WHITE FINDING | EA-09-200 | ||
Mr. Adam C. Heflin, Senior Vice President and Chief Nuclear Officer AmerenUE P.O. Box 620 Fulton, MO 65251 Subject: NRC SPECIAL INSPECTION REPORT 05000483/2009009 - CALLAWAY PLANT; PRELIMINARY WHITE FINDING | |||
==Dear Mr. Heflin:== | ==Dear Mr. Heflin:== | ||
On September 2, 2009, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Callaway Plant to evaluate the facts and circumstances surrounding the failure to start of the turbine-driven auxiliary feedwater pump due to an inadequately lubricated trip throttle valve. Based upon the risk and deterministic criteria specified in NRC Management Directive 8.3, "NRC Incident Investigation Program," including possible generic implications, the NRC initiated a special inspection in accordance with Inspection Procedure 93812, "Special Inspection." The basis for initiating the special inspection and the focus areas for review are detailed in the Special Inspection Charter (Attachment 2). The determination that the inspection would be conducted was made by the NRC on June 17, 2009, and the onsite inspection started on June 22, 2009. The enclosed report documents the inspection findings that were discussed on September 2, 2009, with you, and members of your staff. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, observed activities, and interviewed plant personnel. The enclosed report discusses one finding that appears to have low to moderate safety significance (White). As described in Section 1R2 of this report, the NRC concluded that the failure to adequately lubricate the turbine-driven auxiliary feedwater pump trip throttle valve during Refueling Outage 16 resulted in the pump's failure to start on May 25, 2009. The safety significance of this finding was assessed on the basis of the best available information, including influential assumptions, using the applicable Significance Determination Process and was preliminarily determined to be a White (i.e., low to moderate safety significance) finding. Attachment 4 of this report provides a detailed description of the preliminary risk determination. The finding is also an apparent violation of NRC requirements and is being considered for escalated enforcement action in accordance with the Enforcement Policy, which can be found on the NRC's Web site at http://www.nrc.gov/reading-rm/doc-collections/enforcement. Union Electric Company 2 In accordance with NRC Manual Chapter 0609, "Significance Determination Process," we intend to complete our evaluation using the best available information and issue our final determination of safety significance within 90 days of the date of this letter. The significance determination process encourages an open dialogue between the NRC staff and the licensee; however, the dialogue should not impact the timeliness of the staff's final determination. | On September 2, 2009, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Callaway Plant to evaluate the facts and circumstances surrounding the failure to start of the turbine-driven auxiliary feedwater pump due to an inadequately lubricated trip throttle valve. Based upon the risk and deterministic criteria specified in NRC Management Directive 8.3, "NRC Incident Investigation Program," including possible generic implications, the NRC initiated a special inspection in accordance with Inspection Procedure 93812, "Special Inspection." The basis for initiating the special inspection and the focus areas for review are detailed in the Special Inspection Charter (Attachment 2). The determination that the inspection would be conducted was made by the NRC on June 17, 2009, and the onsite inspection started on June 22, 2009. The enclosed report documents the inspection findings that were discussed on September 2, 2009, with you, and members of your staff. | ||
The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, observed activities, and interviewed plant personnel. | |||
The enclosed report discusses one finding that appears to have low to moderate safety significance (White). As described in Section 1R2 of this report, the NRC concluded that the failure to adequately lubricate the turbine-driven auxiliary feedwater pump trip throttle valve during Refueling Outage 16 resulted in the pump's failure to start on May 25, 2009. The safety significance of this finding was assessed on the basis of the best available information, including influential assumptions, using the applicable Significance Determination Process and was preliminarily determined to be a White (i.e., low to moderate safety significance) finding. Attachment 4 of this report provides a detailed description of the preliminary risk determination. The finding is also an apparent violation of NRC requirements and is being considered for escalated enforcement action in accordance with the Enforcement Policy, which can be found on the NRC's Web site at http://www.nrc.gov/reading-rm/doc-collections/enforcement. Union Electric Company 2 In accordance with NRC Manual Chapter 0609, "Significance Determination Process," we intend to complete our evaluation using the best available information and issue our final determination of safety significance within 90 days of the date of this letter. The significance determination process encourages an open dialogue between the NRC staff and the licensee; however, the dialogue should not impact the timeliness of the staff's final determination. | |||
Before we make a final decision on this matter, we are providing you with an opportunity to (1) attend a Regulatory Conference where you can present to the NRC your perspective on the facts and assumptions the NRC used to arrive at the finding and assess its significance, or (2) submit your position on the finding to the NRC in writing. If you request a Regulatory Conference, it should be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least one week prior to the conference in an effort to make the conference more efficient and effective. If a Regulatory Conference is held, it will be open for public observation. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of your receipt of this letter. If you decline to request a Regulatory Conference or submit a written response, you relinquish your right to appeal the final Significance Determination Process determination, in that by not doing either, you fail to meet the appeal requirements stated in the Prerequisite and Limitation sections of Attachment 2 of Manual Chapter 0609. Please contact Mr. Vincent Gaddy at (817) 860-8141 within 10 business days of the date of this letter to notify the NRC of your intentions. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision. The final resolution of this matter will be conveyed in separate correspondence. Because the NRC has not made a final determination in this matter, no Notice of Violation is being issued for these inspection findings at this time. In addition, please be advised that the characterization of the apparent violation described in the enclosed inspection report may change as a result of further NRC review. The report also documents four NRC-identified findings, which were evaluated under the significance determination process as having very low safety significance (Green). These findings were determined to involve violations of NRC requirements. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the noncited violations in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 612 E. Lamar Blvd., Suite 400, Arlington, Texas 76011-4125; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspectors at the Callaway Plant. In addition, if you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Callaway Plant. The information you provide will be considered in accordance with Inspection Manual Chapter 0305. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosures will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC=s document Union Electric Company 3 system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html. | Before we make a final decision on this matter, we are providing you with an opportunity to (1) attend a Regulatory Conference where you can present to the NRC your perspective on the facts and assumptions the NRC used to arrive at the finding and assess its significance, or (2) submit your position on the finding to the NRC in writing. If you request a Regulatory Conference, it should be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least one week prior to the conference in an effort to make the conference more efficient and effective. If a Regulatory Conference is held, it will be open for public observation. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of your receipt of this letter. If you decline to request a Regulatory Conference or submit a written response, you relinquish your right to appeal the final Significance Determination Process determination, in that by not doing either, you fail to meet the appeal requirements stated in the Prerequisite and Limitation sections of Attachment 2 of Manual Chapter 0609. | ||
Please contact Mr. Vincent Gaddy at (817) 860-8141 within 10 business days of the date of this letter to notify the NRC of your intentions. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision. The final resolution of this matter will be conveyed in separate correspondence. | |||
Because the NRC has not made a final determination in this matter, no Notice of Violation is being issued for these inspection findings at this time. In addition, please be advised that the characterization of the apparent violation described in the enclosed inspection report may change as a result of further NRC review. | |||
The report also documents four NRC-identified findings, which were evaluated under the significance determination process as having very low safety significance (Green). These findings were determined to involve violations of NRC requirements. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the noncited violations in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 612 E. Lamar Blvd., Suite 400, Arlington, Texas 76011-4125; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspectors at the Callaway Plant. In addition, if you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Callaway Plant. The information you provide will be considered in accordance with Inspection Manual Chapter 0305. | |||
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosures will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC | |||
=s document Union Electric Company 3 system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html. | |||
Sincerely,/RA/ Dwight Chamberlain, Director Division of Reactor Projects Docket: 50-483 License: NPF-30 | Sincerely,/RA/ Dwight Chamberlain, Director Division of Reactor Projects Docket: 50-483 License: NPF-30 | ||
| Line 22: | Line 52: | ||
Attachment 1: Supplemental Information Attachment 2: Special Inspection Charter Attachment 3: Timeline Associated With Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve FCHV0312 Attachment 4: Preliminary Significance Determination Evaluation Mr. Luke H. Graessle Director, Operations Support AmerenUE P.O. Box 620 Fulton, MO 65251 E. Hope Bradley Manager, Protective Services AmerenUE P.O. Box 620 Fulton, MO 65251 Mr. Scott Sandbothe, Manager Regulatory Affairs AmerenUE P.O. Box 620 Fulton, MO 65251 R. E. Farnam Assistant Manager, Technical Training AmerenUE P.O. Box 620 Fulton, MO 65251 | Attachment 1: Supplemental Information Attachment 2: Special Inspection Charter Attachment 3: Timeline Associated With Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve FCHV0312 Attachment 4: Preliminary Significance Determination Evaluation Mr. Luke H. Graessle Director, Operations Support AmerenUE P.O. Box 620 Fulton, MO 65251 E. Hope Bradley Manager, Protective Services AmerenUE P.O. Box 620 Fulton, MO 65251 Mr. Scott Sandbothe, Manager Regulatory Affairs AmerenUE P.O. Box 620 Fulton, MO 65251 R. E. Farnam Assistant Manager, Technical Training AmerenUE P.O. Box 620 Fulton, MO 65251 | ||
Union Electric Company 4 J. S. Geyer Radiation Protection Manager AmerenUE P.O. Box 620 Fulton, MO 65251 John O'Neill, Esq. Pillsbury Winthrop Shaw Pittman LLP 2300 N. Street, N.W. Washington, DC 20037 Missouri Public Service Commission P.O. Box 360 Jefferson City, MO 65102-0360 Deputy Director for Policy Department of Natural Resources P.O. Box 176 Jefferson City, MO 65102-0176 Mr. Rick A. Muench, President and Chief Executive officer Wolf Creek Nuclear Operating Corporation P.O. Box 411 Burlington, KS 66839 Kathleen Logan Smith, Executive Director and Kay Drey, Representative, Board of Directors Missouri Coalition for the Environment 6267 Delmar Boulevard, Suite 2E St. Louis, MO 63130 Mr. Lee Fritz, Presiding Commissioner Callaway County Courthouse 10 East Fifth Street Fulton, MO 65251 Director, Missouri State Emergency Management Agency P.O. Box 116 Jefferson City, MO 65102-0116 Mr. Scott Clardy, Administrator Section for Disease Control Missouri Department of Health and Senior Services P.O. Box 570 Jefferson City, MO 65102-0570 Union Electric Company 5 Certrec Corporation 4200 South Hulen, Suite 422 Fort Worth, TX 76109 Mr. Keith G. Henke, Planner II Division of Community and Public Health Office of Emergency Coordination Missouri Department of Health and Senior Services 930 Wildwood Drive P.O. Box 570 Jefferson City, MO 65102 Chief, Technological Hazards Branch FEMA Region VII 9221 Ward Parkway, Suite 300 Kansas City, MO 64114-3372 Union Electric Company 6 Electronic distribution by RIV: Regional Administrator (Elmo.Collins@nrc.gov) Deputy Regional Administrator (Chuck.Casto@nrc.gov) DRP Director (Dwight.Chamberlain@nrc.gov) DRP Deputy Director (Anton.Vegel@nrc.gov) DRS Director (Roy.Caniano@nrc.gov) | Union Electric Company 4 J. S. Geyer Radiation Protection Manager AmerenUE P.O. Box 620 Fulton, MO 65251 | ||
John O'Neill, Esq. Pillsbury Winthrop Shaw Pittman LLP 2300 N. Street, N.W. Washington, DC 20037 | |||
Missouri Public Service Commission P.O. Box 360 Jefferson City, MO 65102-0360 Deputy Director for Policy Department of Natural Resources P.O. Box 176 Jefferson City, MO 65102-0176 Mr. Rick A. Muench, President and Chief Executive officer Wolf Creek Nuclear Operating Corporation P.O. Box 411 Burlington, KS 66839 Kathleen Logan Smith, Executive Director and Kay Drey, Representative, Board of Directors Missouri Coalition for the Environment 6267 Delmar Boulevard, Suite 2E St. Louis, MO 63130 Mr. Lee Fritz, Presiding Commissioner Callaway County Courthouse 10 East Fifth Street Fulton, MO 65251 Director, Missouri State Emergency Management Agency P.O. Box 116 Jefferson City, MO 65102-0116 Mr. Scott Clardy, Administrator Section for Disease Control Missouri Department of Health and Senior Services P.O. Box 570 Jefferson City, MO 65102-0570 | |||
Union Electric Company 5 Certrec Corporation 4200 South Hulen, Suite 422 Fort Worth, TX 76109 Mr. Keith G. Henke, Planner II Division of Community and Public Health Office of Emergency Coordination Missouri Department of Health and Senior Services 930 Wildwood Drive P.O. Box 570 Jefferson City, MO 65102 Chief, Technological Hazards Branch FEMA Region VII 9221 Ward Parkway, Suite 300 Kansas City, MO 64114-3372 Union Electric Company 6 Electronic distribution by RIV: Regional Administrator (Elmo.Collins@nrc.gov) Deputy Regional Administrator (Chuck.Casto@nrc.gov) DRP Director (Dwight.Chamberlain@nrc.gov) DRP Deputy Director (Anton.Vegel@nrc.gov) DRS Director (Roy.Caniano@nrc.gov) | |||
DRS Deputy Director (Troy.Pruett@nrc.gov) Senior Resident Inspector (David.Dumbacher@nrc.gov) Resident Inspector (Jeremy.Groom@nrc.gov) Branch Chief, DRP/B (Vincent.Gaddy@nrc.gov) Senior Project Engineer, DRP/B (Rick.Deese@nrc.gov) | DRS Deputy Director (Troy.Pruett@nrc.gov) Senior Resident Inspector (David.Dumbacher@nrc.gov) Resident Inspector (Jeremy.Groom@nrc.gov) Branch Chief, DRP/B (Vincent.Gaddy@nrc.gov) Senior Project Engineer, DRP/B (Rick.Deese@nrc.gov) | ||
CWY Site Secretary (Dawn.Yancey@nrc.gov) Public Affairs Officer (Victor.Dricks@nrc.gov) Team Leader, DRP/TSS (Chuck.Paulk@nrc.gov) RITS Coordinator (Marisa.Herrera@nrc.gov) Regional Counsel (Karla.Fuller@nrc.gov) Congressional Affairs Officer (Jenny.Weil@nrc.gov) OEMail Resource DRS STA (Dale.Powers@nrc.gov) OEDO RIV Coordinator (Leigh.Trocine@nrc.gov) ROPreports WilliamJones@nrc.gov Gregory.Bowman@nrc.gov Mark.Haire@nrc.gov Christi.Maier@nrc.gov Nick.Hilton@nrc.gov June.Cai@nrc.gov John.Wray@nrc.gov MaryAnn.Ashley@nrc.gov Gerald.Gulla@nrc.gov Alexander.Sapountzis@nrc.gov Robert.Summers@nrc.gov Doug.Starkey@nrc.gov File located: R:_Reactors\CW\2009\2009009RP-JRG.doc ML092730656 SUNSI Rev Compl. Yes No ADAMS Yes No Reviewer Initials VGG Publicly Avail Yes No Sensitive Yes No Sens. Type Initials VGG RIV: RI:DRP/B RI:DRP/C SRI:DRP/B SRA ACES JGroom MChambers DDumbacher DLoveless MHaire /RA-E/VGG for /RA-E/VGG for /RA-E/VGG for /RA/ /RA/WJ for MH 9/15/09 9/22/09 9/16/09 9/21/09 9/23/09 OE NRR C:DRP/B D:DRP GBowman MCunningham VGaddy DChamberlain /RA-E/VGG for /RA-E/VGG for /RA/ /RA/ 9/23/09 9/28/09 9/28/09 9/30/09 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax 1 Enclosure U.S. NUCLEAR REGULATORY COMMISSION REGION IV Docket: 05000483 License: NPF-30 Report: 05000483/2009009 Licensee: Union Electric Company Facility: Callaway Plant Location: Junction Highway CC and Highway O Fulton, MO Dates: June 22 through August 27, 2009 Inspectors: J. Groom, Resident Inspector, Callaway Plant M. Chambers, Resident Inspector, Cooper Nuclear Station D. Dumbacher, Senior Resident Inspector, Callaway Plant D. Loveless, Senior Reactor Analyst Approved By: D. Chamberlain, Director Division of Reactor Projects 2 Enclosure | CWY Site Secretary (Dawn.Yancey@nrc.gov) Public Affairs Officer (Victor.Dricks@nrc.gov) Team Leader, DRP/TSS (Chuck.Paulk@nrc.gov) RITS Coordinator (Marisa.Herrera@nrc.gov) Regional Counsel (Karla.Fuller@nrc.gov) Congressional Affairs Officer (Jenny.Weil@nrc.gov) OEMail Resource DRS STA (Dale.Powers@nrc.gov) OEDO RIV Coordinator (Leigh.Trocine@nrc.gov) ROPreports WilliamJones@nrc.gov Gregory.Bowman@nrc.gov Mark.Haire@nrc.gov Christi.Maier@nrc.gov Nick.Hilton@nrc.gov June.Cai@nrc.gov John.Wray@nrc.gov MaryAnn.Ashley@nrc.gov Gerald.Gulla@nrc.gov Alexander.Sapountzis@nrc.gov Robert.Summers@nrc.gov Doug.Starkey@nrc.gov | ||
File located: R:_Reactors\CW\2009\2009009RP-JRG.doc ML092730656 SUNSI Rev Compl. Yes No ADAMS Yes No Reviewer Initials VGG Publicly Avail Yes No Sensitive Yes No Sens. Type Initials VGG RIV: RI:DRP/B RI:DRP/C SRI:DRP/B SRA ACES JGroom MChambers DDumbacher DLoveless MHaire /RA-E/VGG for /RA-E/VGG for /RA-E/VGG for /RA/ /RA/WJ for MH 9/15/09 9/22/09 9/16/09 9/21/09 9/23/09 OE NRR C:DRP/B D:DRP GBowman MCunningham VGaddy DChamberlain /RA-E/VGG for /RA-E/VGG for /RA/ /RA/ 9/23/09 9/28/09 9/28/09 9/30/09 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax 1 Enclosure U.S. NUCLEAR REGULATORY COMMISSION REGION IV Docket: 05000483 License: NPF-30 Report: 05000483/2009009 Licensee: Union Electric Company Facility: Callaway Plant Location: Junction Highway CC and Highway O Fulton, MO Dates: June 22 through August 27, 2009 Inspectors: J. Groom, Resident Inspector, Callaway Plant M. Chambers, Resident Inspector, Cooper Nuclear Station D. Dumbacher, Senior Resident Inspector, Callaway Plant D. Loveless, Senior Reactor Analyst Approved By: D. Chamberlain, Director Division of Reactor Projects 2 Enclosure | |||
=SUMMARY OF FINDINGS= | =SUMMARY OF FINDINGS= | ||
IR 05000483/2009009; 06/22/09 - 08/27/09; Callaway Plant; Special inspection into turbine-driven auxiliary feedwater pump failure to start. The report covered one week of onsite inspection and in office review through August 27, 2009. | IR 05000483/2009009; 06/22/09 - 08/27/09; Callaway Plant; Special inspection into turbine-driven auxiliary feedwater pump failure to start. | ||
The report covered one week of onsite inspection and in office review through August 27, 2009. | |||
Two resident inspectors performed the inspection with assistance from a senior resident inspector and a senior reactor analyst. One apparent violation and four green noncited violations were identified. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance Determination Process." Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006. | Two resident inspectors performed the inspection with assistance from a senior resident inspector and a senior reactor analyst. One apparent violation and four green noncited violations were identified. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance Determination Process." Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006. | ||
| Line 33: | Line 73: | ||
===A. NRC-Identified and Self Revealing Findings=== | ===A. NRC-Identified and Self Revealing Findings=== | ||
===Cornerstone: Mitigating Systems === | ===Cornerstone: Mitigating Systems=== | ||
* TBD. The team identified a self-revealing apparent violation of Technical Specification 3.7.5, "Auxiliary Feedwater System," due to the failure to adequately lubricate turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. During May 25, 2009, surveillance testing, the turbine-driven auxiliary feedwater pump did not start as expected due to hardened grease on the valve spindle of FCHV0312. The previous lubrication preventative maintenance had been missed and lack of lubrication increased friction between the sliding nut and spindle preventing FCHV0312 from opening. Following lubrication FCHV0312 and the turbine-driven auxiliary feedwater pump tested satisfactorily. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200904216. This finding is greater than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function because the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Callaway Plant Phase 2 pre-solved table item "Turbine Driven Auxiliary Feedwater Pump Fails to Start," the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The preliminary outcome of the Phase 3 significance determination analysis, Attachment 4, determined the finding was of low to moderate safety significance. The inspectors determined that this finding had a crosscutting aspect in the area of human performance associated with the work practices component because the licensee failed to follow the procedural guidance provided when changing the scope of a preventive maintenance task [H.4(b)](Section 1R2). | * TBD. The team identified a self-revealing apparent violation of Technical Specification 3.7.5, "Auxiliary Feedwater System," due to the failure to adequately lubricate turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. During May 25, 2009, surveillance testing, the turbine-driven auxiliary feedwater pump did not start as expected due to hardened grease on the valve spindle of FCHV0312. The previous lubrication preventative maintenance had been missed and lack of lubrication increased friction between the sliding nut and spindle preventing FCHV0312 from opening. Following lubrication FCHV0312 and the turbine-driven auxiliary feedwater pump tested satisfactorily. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200904216. | ||
This finding is greater than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function because the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Callaway Plant Phase 2 pre-solved table item "Turbine Driven Auxiliary Feedwater Pump Fails to Start," the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The preliminary outcome of the Phase 3 significance determination analysis, Attachment 4, determined the finding was of low to moderate safety significance. | |||
The inspectors determined that this finding had a crosscutting aspect in the area of human performance associated with the work practices component because the licensee failed to follow the procedural guidance provided when changing the scope of a preventive maintenance task [H.4(b)](Section 1R2). | |||
: '''Green.''' | : '''Green.''' | ||
The team identified a noncited violation of Technical Specification 5.4.1.a, "Procedures," for the failure to provide adequate procedural guidance for the lubrication of auxiliary feedwater pump turbine trip throttle valve FCHV0312. The inspectors found that 2002 corrective actions to improve the lubrication procedure were not fully developed and the procedure lubrication guidance was ambiguous in that it did not specify the amount of lubricant to apply or what valve subcomponents to lubricate. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200905032. This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not have a crosscutting aspect since the 2003 lubrication procedure revision was not reflective of current licensee performance (Section 1R3). | The team identified a noncited violation of Technical Specification 5.4.1.a, "Procedures," for the failure to provide adequate procedural guidance for the lubrication of auxiliary feedwater pump turbine trip throttle valve FCHV0312. The inspectors found that 2002 corrective actions to improve the lubrication procedure were not fully developed and the procedure lubrication guidance was ambiguous in that it did not specify the amount of lubricant to apply or what valve subcomponents to lubricate. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200905032. | ||
This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not have a crosscutting aspect since the 2003 lubrication procedure revision was not reflective of current licensee performance (Section 1R3). | |||
: '''Green.''' | : '''Green.''' | ||
The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control," for the failure to adequately evaluate the use of Mobile 28 grease for the turbine-driven auxiliary feedwater pump trip throttle valve. The licensee's 1995 evaluation included no documentation for the appropriate relubrication interval of the valve. Additionally, the inspectors identified that the valve exhibited temperatures ranging from 235°F to near 300°F compared to the 215°F valve temperature used in the evaluation. The inspectors questioned if the use of Mobile 28 grease was appropriate since operating experience suggests that Mobile 28 grease has a tendency to thicken and harden at temperatures exceeding 250°F and elevated temperatures increased the lubricant's tendency to lose oils and could result in increased stem friction. Following questioning by the inspectors, the licensee initiated Callaway Action Request 200905067 and Request for Resolution 200905651 to determine if Mobile 28 grease was an appropriate lubricant for valve FCHV0312 (Section 1R3). This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of design control and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not to have a crosscutting aspect since the inadequate 1995 lubrication evaluation was not reflective of current licensee performance (Section 1R3). | The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control," for the failure to adequately evaluate the use of Mobile 28 grease for the turbine-driven auxiliary feedwater pump trip throttle valve. The licensee's 1995 evaluation included no documentation for the appropriate relubrication interval of the valve. Additionally, the inspectors identified that the valve exhibited temperatures ranging from 235°F to near 300°F compared to the 215°F valve temperature used in the evaluation. The inspectors questioned if the use of Mobile 28 grease was appropriate since operating experience suggests that Mobile 28 grease has a tendency to thicken and harden at temperatures exceeding 250°F and elevated temperatures increased the lubricant's tendency to lose oils and could result in increased stem friction. Following questioning by the inspectors, the licensee initiated Callaway Action Request 200905067 and Request for Resolution 200905651 to determine if Mobile 28 grease was an appropriate lubricant for valve FCHV0312 (Section 1R3). | ||
This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of design control and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not to have a crosscutting aspect since the inadequate 1995 lubrication evaluation was not reflective of current licensee performance (Section 1R3). | |||
: '''Green.''' | : '''Green.''' | ||
The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," regarding the licensee's failure to follow the requirements of Callaway Procedure APA-ZZ-00500, "Corrective Action Program." Specifically, licensee personnel failed to initiate Callaway action requests for adverse conditions of high hand wheel forces, galled subcomponents, and hardened, gritty grease found during the 2007 rebuild of the spare turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. The licensee has entered this issue into their corrective action program as Callaway Action Request 200905053. This finding is greater than minor because, if left uncorrected, failure to fully utilize the corrective action program could become a more significant safety concern. The inspectors determined that this finding impacted the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The cause of this finding is related to the problem identification and resolution crosscutting component of the corrective action program because licensee personnel failed to implement a corrective action program with a low threshold for identifying issues [P.1(a)](Section1R3). | The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," regarding the licensee's failure to follow the requirements of Callaway Procedure APA-ZZ-00500, "Corrective Action Program." Specifically, licensee personnel failed to initiate Callaway action requests for adverse conditions of high hand wheel forces, galled subcomponents, and hardened, gritty grease found during the 2007 rebuild of the spare turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. The licensee has entered this issue into their corrective action program as Callaway Action Request 200905053. | ||
This finding is greater than minor because, if left uncorrected, failure to fully utilize the corrective action program could become a more significant safety concern. The inspectors determined that this finding impacted the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The cause of this finding is related to the problem identification and resolution crosscutting component of the corrective action program because licensee personnel failed to implement a corrective action program with a low threshold for identifying issues [P.1(a)](Section1R3). | |||
: '''Green.''' | : '''Green.''' | ||
The team identified a noncited violation of Technical Specification Limiting Condition for Operation 3.0.4 for entering Mode 3 with the turbine-driven auxiliary feedwater pump inoperable. Specifically, on November 3, 2008, while in Mode 4 for Refueling Outage 16, an unexpected overspeed trip of the turbine occurred during postmaintenance testing. Callaway operations staff inappropriately concluded that a water slug from the auxiliary steam line was the cause of the turbine overspeed. Following entry into Mode 3, during preparations for turbine-driven auxiliary feedwater pump testing, the licensee found the servo control valve installed during the outage was faulty. When questioned by the inspectors, the licensee determined that the faulty servo control valve discovered in Mode 3 was responsible for the overspeed of the turbine-driven auxiliary feedwater pump that occurred in Mode 4 and that the equipment was inoperable during the mode change that occurred on November 4, 2008. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200905313. This finding is greater than minor because it is associated with the Mitigating Systems Cornerstone attribute of equipment performance and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and | The team identified a noncited violation of Technical Specification Limiting Condition for Operation 3.0.4 for entering Mode 3 with the turbine-driven auxiliary feedwater pump inoperable. Specifically, on November 3, 2008, while in Mode 4 for Refueling Outage 16, an unexpected overspeed trip of the turbine occurred during postmaintenance testing. Callaway operations staff inappropriately concluded that a water slug from the auxiliary steam line was the cause of the turbine overspeed. Following entry into Mode 3, during preparations for turbine-driven auxiliary feedwater pump testing, the licensee found the servo control valve installed during the outage was faulty. When questioned by the inspectors, the licensee determined that the faulty servo control valve discovered in Mode 3 was responsible for the overspeed of the turbine-driven auxiliary feedwater pump that occurred in Mode 4 and that the equipment was inoperable during the mode change that occurred on November 4, 2008. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200905313. | ||
This finding is greater than minor because it is associated with the Mitigating Systems Cornerstone attribute of equipment performance and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and | |||
Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The inspectors determined that this finding has a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee failed to fully evaluate the overspeed of the turbine-driven auxiliary feedwater pump that occurred on November 3, 2008 [P.1(c)](Section 4OA2). | Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The inspectors determined that this finding has a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee failed to fully evaluate the overspeed of the turbine-driven auxiliary feedwater pump that occurred on November 3, 2008 [P.1(c)](Section 4OA2). | ||
===B. Licensee-Identified Violations=== | ===B. Licensee-Identified Violations=== | ||
None | None | ||
| Line 52: | Line 105: | ||
==REACTOR SAFETY== | ==REACTOR SAFETY== | ||
Cornerstones: Initiating Events, Mitigating Systems, Barrier Integrity, and Emergency Preparedness | Cornerstones: Initiating Events, Mitigating Systems, Barrier Integrity, and Emergency Preparedness 1R1 Special Inspection Scope On May 25, 2009, during a planned slave relay test of the auxiliary feedwater actuation system, the turbine-driven auxiliary feedwater pump failed to start on demand. Subsequent troubleshooting determined that the trip throttle valve for the turbine did not actuate properly which prevented steam from being admitted to the turbine. Electrical continuity checks revealed that the torque switch on the valve operator tripped which stopped the motor-operated valve actuator. Inspection of the trip throttle valve showed that the lubricant used on the valve spindle and sliding nut was dried. In accordance with Management Directive 8.3, "NRC Incident Investigation Program," the NRC determined that a special inspection was warranted, in part, based on the potential safety significance and because of potential generic issues associated with lubricating motor-operated valves. | ||
1R2 Review of the Failure of Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve Background On May 25, 2009, Callaway Plant operators performed Procedure OSP-SA-0007A, "Train A AFAS Slave Relay Test." The procedure is designed to demonstrate that the auxiliary feedwater actuation system slave relays are operable and capable of starting the turbine-driven auxiliary feedwater pump. The turbine-driven auxiliary feedwater pump is a steam-driven pump which utilizes a Terry turbine as a prime mover supplied 7 Enclosure by steam from steam generators B and C. The steam line to the turbine contains normally closed, motor-operated, spring loaded trip throttle valve FCHV0312. Upon initiation of an auxiliary feedwater actuation signal, the trip throttle valve operator moves in the closed direction in order to latch the mechanical linkage of the valve stem. A limit switch is used to detect when the trip throttle valve has reached the latched position. Once in the latched position, the direction of the motor operator is reversed and the trip throttle valve opens admitting steam to the governor flow control valve, which is normally fully open, starting the turbine-driven auxiliary feedwater pump. During the performance of Step 6.2.8 of Procedure OSP-SA-0007A, the turbine-driven auxiliary feedwater pump did not start as expected. Following the failed start, the licensee discovered that valve FCHV0312 had traveled approximately 70 percent of the closed stroke in an attempt to latch the valve operator but had stopped prior to latching the valve. Troubleshooting was conducted on May 25, 2009, under Job 09003598, Task 910. During that job, the licensee determined that the closed torque switch had opened which stopped the motor-operated valve actuator. Callaway operators then declutched and manually operated the trip throttle valve actuator. During manual operation of the actuator, the sliding nut and screw spindle were observed to have an audible squeak while moving. A sample of the grease taken from the valve spindle was later examined using infrared spectrometry. The results of that analysis showed that the chemical composition was comparable to that of Mobile 28 grease and showed no other traces of either zinc or lithium which are found in Exxon Nebula EP-1 or EP-2 greases, the other two greases used on valve stems at the Callaway Plant. Those results indicated that the grease found on the valve FCHV0312 on May 25, 2009, was not mixed with other, non-approved greases. Following troubleshooting, the licensee performed Job 09003598, Task 510, to lubricate the trip throttle valve. During that job, the licensee cleaned, inspected, and lubricated the trip throttle valve sliding nut, screw spindle, split coupling and trip linkages. The valve was manually operated during the performance of the job to allow access to all valve components. The mechanical maintenance technician who operated the valve noted that excessive forces were necessary to manipulate the valve and an audible squeak could be heard as the valve spindle rotated. As the valve was lubricated, the maintenance staff observed that the valve hand wheel became easier to manipulate and the audible squeak subsided. Once the valve was adequately lubricated, the licensee performed postmaintenance testing under Job 09003598, Task 920, to confirm the valve could operate as designed. Missed Lubrication of Valve FCHV0312 | The inspection charter required the team to: (1) review the circumstances related to the discovery of the degraded condition, (2) assess the licensee's determination of cause and effectiveness of actions taken to resolve and prevent recurrence of these problems, and (3) assess the effectiveness of licensee programs to maintain the physical condition of the turbine-driven auxiliary feedwater pump trip throttle valve including the licensee's lubrication and valve replacement programs. The team evaluated if the licensee took appropriate actions to address these issues including extent of condition, extent of cause, and common cause questions. The inspectors reviewed the licensee's Generic Letter 89-10 program to ensure appropriate testing was being performed that would demonstrate the turbine-driven auxiliary feedwater pump trip throttle valve's ability to function under design-basis conditions. | ||
The team conducted their reviews in accordance with NRC Inspection Procedure 93812, "Special Inspection Procedure." The special inspection team reviewed procedures, corrective action documents, as well as design and maintenance records for the equipment of concern. The team interviewed key station personnel regarding the events, reviewed the root cause analysis, and assessed the adequacy of corrective actions. The team walked down and inspected the equipment in the field and spare equipment in the warehouse. A list of specific documents reviewed is provided as Attachment 1. The charter for the special inspection is provided as Attachment 2. | |||
1R2 Review of the Failure of Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve Background On May 25, 2009, Callaway Plant operators performed Procedure OSP-SA-0007A, "Train A AFAS Slave Relay Test." The procedure is designed to demonstrate that the auxiliary feedwater actuation system slave relays are operable and capable of starting the turbine-driven auxiliary feedwater pump. The turbine-driven auxiliary feedwater pump is a steam-driven pump which utilizes a Terry turbine as a prime mover supplied 7 Enclosure by steam from steam generators B and C. The steam line to the turbine contains normally closed, motor-operated, spring loaded trip throttle valve FCHV0312. Upon initiation of an auxiliary feedwater actuation signal, the trip throttle valve operator moves in the closed direction in order to latch the mechanical linkage of the valve stem. A limit switch is used to detect when the trip throttle valve has reached the latched position. Once in the latched position, the direction of the motor operator is reversed and the trip throttle valve opens admitting steam to the governor flow control valve, which is normally fully open, starting the turbine-driven auxiliary feedwater pump. During the performance of Step 6.2.8 of Procedure OSP-SA-0007A, the turbine-driven auxiliary feedwater pump did not start as expected. Following the failed start, the licensee discovered that valve FCHV0312 had traveled approximately 70 percent of the closed stroke in an attempt to latch the valve operator but had stopped prior to latching the valve. | |||
Troubleshooting was conducted on May 25, 2009, under Job 09003598, Task 910. During that job, the licensee determined that the closed torque switch had opened which stopped the motor-operated valve actuator. Callaway operators then declutched and manually operated the trip throttle valve actuator. During manual operation of the actuator, the sliding nut and screw spindle were observed to have an audible squeak while moving. A sample of the grease taken from the valve spindle was later examined using infrared spectrometry. The results of that analysis showed that the chemical composition was comparable to that of Mobile 28 grease and showed no other traces of either zinc or lithium which are found in Exxon Nebula EP-1 or EP-2 greases, the other two greases used on valve stems at the Callaway Plant. Those results indicated that the grease found on the valve FCHV0312 on May 25, 2009, was not mixed with other, non-approved greases. | |||
Following troubleshooting, the licensee performed Job 09003598, Task 510, to lubricate the trip throttle valve. During that job, the licensee cleaned, inspected, and lubricated the trip throttle valve sliding nut, screw spindle, split coupling and trip linkages. The valve was manually operated during the performance of the job to allow access to all valve components. The mechanical maintenance technician who operated the valve noted that excessive forces were necessary to manipulate the valve and an audible squeak could be heard as the valve spindle rotated. As the valve was lubricated, the maintenance staff observed that the valve hand wheel became easier to manipulate and the audible squeak subsided. Once the valve was adequately lubricated, the licensee performed postmaintenance testing under Job 09003598, Task 920, to confirm the valve could operate as designed. | |||
Missed Lubrication of Valve FCHV0312 Callaway replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system (Preventive Maintenance 0824900). The turbine-driven auxiliary feedwater pump trip throttle valve that failed to open on May 25, 2009, was replaced under Job 0551578 during Refueling Outage 16 (October 2008). The valve was replaced with a similar valve that was removed from service during Refueling Outage 13 in April 2004 and refurbished in September 2007. Replacement Preventive Maintenance 0824900 did not have specific lubrication instructions. Lubrication of valve FCHV0312 is accomplished by Preventive Maintenance Task 0810863 which is completed every refueling outage and was scheduled for Refueling Outage 16 as Job 07506359. The maintenance supervisor assigned to coordinate Job 07506359 made a job routing request, to engineering 8 Enclosure inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced. Although not specifically referenced, the portion of the job requested to be closed without action was Section 6.3, "Trip Throttle Valve Lubrication and Inspection." The motor-operated valve system engineer replied to the job routing request that the valve had been diagnostically tested satisfactorily and that no additional inspection was needed. The engineer did not consult with the technician requesting closure and did not verify that the lubrication job task being closed was completed by the installation procedure. Consequently, the lubrication portion of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was not performed. Since the valve was not lubricated during Refueling Outage 16, the last lubrication of the valve prior to the May 25, 2009, failure occurred during the refurbishment of the valve in September 2007. The lack of lubrication on valve FCHV0312 resulted in increased friction between the sliding nut and spindle which prevented the valve from opening on demand May 25, 2009. | |||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
| Line 67: | Line 129: | ||
=====Introduction.===== | =====Introduction.===== | ||
The team identified a self-revealing apparent violation of Technical Specification 3.7.5, "Auxiliary Feedwater System," due to the failure to adequately lubricate turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. | The team identified a self-revealing apparent violation of Technical Specification 3.7.5, "Auxiliary Feedwater System," due to the failure to adequately lubricate turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. | ||
=====Description.===== | =====Description.===== | ||
On May 25, 2009, Callaway Plant operators performed surveillance testing Procedure OSP-SA-0007A, "Train A AFAS Slave Relay Test," to test the slave relays used to start the turbine-driven auxiliary feedwater pump. During the performance of Step 6.2.8 of Procedure OSP-SA-0007A, the turbine-driven auxiliary feedwater pump did not start as expected. During troubleshooting, the licensee discovered that the turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 had traveled in the closed direction in an attempt to latch the valve operator but had stopped prior to completing its stroke. Electrical continuity checks revealed that the motor-operated valve closed torque switch had opened. Callaway operators declutched and manually operated the trip throttle valve actuator. An audible squeak was observed during manual operation indicating mechanical binding within the sliding nut and valve spindle. Following troubleshooting, the licensee performed Job 09003598 to lubricate the trip throttle valve. Once the valve was adequately lubricated, the licensee successfully performed postmaintenance testing. | On May 25, 2009, Callaway Plant operators performed surveillance testing Procedure OSP-SA-0007A, "Train A AFAS Slave Relay Test," to test the slave relays used to start the turbine-driven auxiliary feedwater pump. During the performance of Step 6.2.8 of Procedure OSP-SA-0007A, the turbine-driven auxiliary feedwater pump did not start as expected. During troubleshooting, the licensee discovered that the turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 had traveled in the closed direction in an attempt to latch the valve operator but had stopped prior to completing its stroke. Electrical continuity checks revealed that the motor-operated valve closed torque switch had opened. Callaway operators declutched and manually operated the trip throttle valve actuator. An audible squeak was observed during manual operation indicating mechanical binding within the sliding nut and valve spindle. Following troubleshooting, the licensee performed Job 09003598 to lubricate the trip throttle valve. Once the valve was adequately lubricated, the licensee successfully performed postmaintenance testing. | ||
9 Enclosure Callaway replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system. The turbine-driven auxiliary feedwater pump trip throttle valve that failed to open on May 25, 2009, was replaced during Refueling Outage 16 in October 2008. The valve was replaced with a similar valve that was removed from service during Refueling Outage 13 in April 2004 and refurbished in September 2007. The replacement procedure did not have specific lubrication instructions. Lubrication of the valve was scheduled to be performed a few days after replacement as Job 07506359. The maintenance supervisor assigned to coordinate the lubrication incorrectly assumed that the valve was adequately lubricated since it had been recently replaced. Callaway Procedure APA-ZZ-00320, "Work Execution," Section 4.12, allows for a job to be canceled if it is determined that the work is not necessary or has been completed by another job. The maintenance supervisor initiated a job routing request to engineering inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced. Although not specifically referenced, the portion of the job requested to be closed was Section 6.3, "Trip Throttle Valve Lubrication and Inspection." | 9 Enclosure Callaway replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system. The turbine-driven auxiliary feedwater pump trip throttle valve that failed to open on May 25, 2009, was replaced during Refueling Outage 16 in October 2008. The valve was replaced with a similar valve that was removed from service during Refueling Outage 13 in April 2004 and refurbished in September 2007. The replacement procedure did not have specific lubrication instructions. Lubrication of the valve was scheduled to be performed a few days after replacement as Job 07506359. The maintenance supervisor assigned to coordinate the lubrication incorrectly assumed that the valve was adequately lubricated since it had been recently replaced. Callaway Procedure APA-ZZ-00320, "Work Execution," Section 4.12, allows for a job to be canceled if it is determined that the work is not necessary or has been completed by another job. The maintenance supervisor initiated a job routing request to engineering inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced. Although not specifically referenced, the portion of the job requested to be closed was Section 6.3, "Trip Throttle Valve Lubrication and Inspection." | ||
Callaway engineering replied to the job routing request that the valve had been diagnostically tested satisfactorily and that no additional inspection was needed. The engineer's response only answered the specific question asked by the job routing request and did not examine the procedural requirements that were the subject of the request. The engineer did not consult with the technician requesting closure and did not verify that the lubrication job task being closed was completed by the installation procedure. Based on the input received from engineering, the mechanical maintenance supervisor closed the lubrication portion of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection." The mechanical maintenance supervisor failed to identify that the entire work scope of the superseded work document was included in the superseding work document which did not meet the requirements of licensee Procedure APA-ZZ-00320. Since the valve was not lubricated during Refueling Outage 16, the last lubrication of the valve occurred during the refurbishment of the valve in September 2007. The lack of lubrication on valve FCHV0312 resulted in increased friction between the sliding nut and spindle which caused the valve not to open upon demand on May 25, 2009. | |||
The licensee initiated Significant Condition Adverse to Quality Callaway Action Request 200904216 to investigate the failure of the turbine-driven auxiliary feedwater pump to start due to an inadequately lubricated trip throttle valve. The licensee's root cause analysis determined that the failure to lubricate valve FCHV0312 was due to the failure to fully review the closure of the lubrication portion of Procedure MPM-FC-QK001 during Refueling Outage 16. The licensee also determined that while the actual timing of the failure could not be determined, it was reasonable to assume that the turbine-driven auxiliary feedwater pump was inoperable for a time frame greater than the technical specification allowed completion time. | |||
Long term corrective actions were implemented to revise the replacement preventive maintenance procedure to include a lubrication section. Additionally, the licensee identified several enhancements to their lubrication program including an evaluation of the preventive maintenance frequency and an evaluation of lubricants used on valve FCHV0312. The inspectors noted that the corrective actions identified in the root cause analysis did not address programmatic issues concerning work execution and a lack of protocol for the initiation and response to job routing requests. Specifically, 10 Enclosure Procedure APA-ZZ-00320 allows for a single individual to close incomplete preventive maintenance tasks. | |||
=====Analysis.===== | =====Analysis.===== | ||
The performance deficiency associated with this finding involved the licensee's failure to ensure valve FCHV0312 was adequately lubricated such that it remained operable. Specifically, Section 6.3 "Trip Throttle Valve Lubrication and Inspection," of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was closed without adequate review. The lack of lubrication resulted in increased friction within the valve which caused the valve not to open on May 25, 2009. This finding is greater than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function because the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, | The performance deficiency associated with this finding involved the licensee's failure to ensure valve FCHV0312 was adequately lubricated such that it remained operable. Specifically, Section 6.3 "Trip Throttle Valve Lubrication and Inspection," of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was closed without adequate review. The lack of lubrication resulted in increased friction within the valve which caused the valve not to open on May 25, 2009. This finding is greater than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function because the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Callaway Plant Phase 2 pre-solved table item "Turbine Driven Auxiliary Feedwater Pump Fails to Start," the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The senior reactor analyst performed the Phase 3 analysis, Attachment 4, and determined that preliminarily, the finding was of low to moderate safety significance. | ||
"Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Callaway Plant Phase 2 pre-solved table item "Turbine Driven Auxiliary Feedwater Pump Fails to Start," the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The senior reactor analyst performed the Phase 3 analysis, Attachment 4, and determined that preliminarily, the finding was of low to moderate safety significance. The inspectors determined that this finding had a crosscutting aspect in the area of human performance associated with the work practices component because the licensee failed to follow the procedural guidance provided when changing the scope of a preventive maintenance task [H.4(b)]. | |||
The inspectors determined that this finding had a crosscutting aspect in the area of human performance associated with the work practices component because the licensee failed to follow the procedural guidance provided when changing the scope of a preventive maintenance task [H.4(b)]. | |||
=====Enforcement.===== | =====Enforcement.===== | ||
Technical Specification 3.7.5, "Auxiliary Feedwater System," requires, in part, that three trains of auxiliary feedwater shall be operable in Modes 1, 2 or 3. The technical specifications required that if one train of auxiliary feedwater is inoperable for greater than 72 hours, actions be taken to be in Mode 3 within 6 hours and Mode 4 within 12 hours. Contrary to the required action statements, on May 25, 2009, the turbine-driven auxiliary feedwater pump train was found to be inoperable due to a lack of lubrication of trip throttle valve FCHV0312. Subsequent review determined that the lack of lubrication resulted in the turbine-driven auxiliary feedwater pump being inoperable for greater than 72 hours. Pending determination of the finding's final safety significance, this finding is identified as Apparent Violation (AV) 05000483/2009009-01, "Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve." | Technical Specification 3.7.5, "Auxiliary Feedwater System," requires, in part, that three trains of auxiliary feedwater shall be operable in Modes 1, 2 or 3. The technical specifications required that if one train of auxiliary feedwater is inoperable for greater than 72 hours, actions be taken to be in Mode 3 within 6 hours and Mode 4 within 12 hours. Contrary to the required action statements, on May 25, 2009, the turbine-driven auxiliary feedwater pump train was found to be inoperable due to a lack of lubrication of trip throttle valve FCHV0312. Subsequent review determined that the lack of lubrication resulted in the turbine-driven auxiliary feedwater pump being inoperable for greater than 72 hours. Pending determination of the finding's final safety significance, this finding is identified as Apparent Violation (AV) 05000483/2009009-01, "Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve." | ||
1R3 Review of Lubrication and Valve Replacement Program for Valve FCHV0312 Valve Lubrication The licensee has two preventive maintenance tasks that lubricate valve FCHV0312. The first is Preventive Maintenance 0810863 which is a Terry turbine equipment check and is performed every refueling outage. The second is Preventive 11 Enclosure Maintenance 0818214 which is a turbine-driven auxiliary feedwater pump equipment check that is performed every 72 weeks. The scheduling of the two preventive maintenance tasks is such that Preventive Maintenance 0810863 is performed every refueling outage and Preventive Maintenance 0818214 is performed in the middle of the operating cycle. The requirements to lubricate the trip throttle valve are contained in Step 6.3 of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection." That procedure directs mechanical maintenance personnel to clean, inspect and lubricate the valve at the sliding nut, trunnion screws, the screw spindle, the trip linkages and pins, and at the split coupling. The frequency of lubrication is consistent with industry standards and vendor guidance which recommends a minimum 18 to 24-month lubrication frequency. | |||
The trip throttle valve is lubricated with Mobile 28 grease which is a high temperature, anti-wear grease composed of a polyalphaolefin synthetic base fluid with an organo-clay thickener. Mobile 28 grease, designed for lubrication of splines, screws and worm gears, falls between the National Lubricating Grease Institute Grade No. 1 and No. 2. The recommended operating temperature is -55°C to 180°C (-67°F to 356°F) with appropriate relubrication intervals. Mobile 28 is recommended by the trip throttle valve vendor as an acceptable lubricant. The licensee first began using Mobile 28 grease for valve FCHV0312 in 1995 under Request for Resolution 16006A. That evaluation determined that Mobile 28 was an acceptable lubricant based on the valve vendor's recommendations and based on the operating temperatures and physical characteristics of the lubricant. | The trip throttle valve is lubricated with Mobile 28 grease which is a high temperature, anti-wear grease composed of a polyalphaolefin synthetic base fluid with an organo-clay thickener. Mobile 28 grease, designed for lubrication of splines, screws and worm gears, falls between the National Lubricating Grease Institute Grade No. 1 and No. 2. The recommended operating temperature is -55°C to 180°C (-67°F to 356°F) with appropriate relubrication intervals. Mobile 28 is recommended by the trip throttle valve vendor as an acceptable lubricant. The licensee first began using Mobile 28 grease for valve FCHV0312 in 1995 under Request for Resolution 16006A. That evaluation determined that Mobile 28 was an acceptable lubricant based on the valve vendor's recommendations and based on the operating temperatures and physical characteristics of the lubricant. | ||
| Line 87: | Line 156: | ||
Valve FCHV0312 is used as a steam admission valve for the turbine-driven auxiliary feedwater pump. The valve is physically arranged downstream of two air operated valves that provide a source of steam from steam generators B and C. Each line is equipped with a bypass line that allows steam to be present at the trip throttle valve to maintain the line warm and free of moisture. Because valve FCHV0312 is a steam admission valve, it is constantly exposed to elevated temperatures. When reviewing operating experience, the team found that Mobile 28 grease, in tests performed for NUREG/CR-6750, "Performance of MOV Stem Lubricants at Elevated Temperature," experienced physical characteristic changes when exposed to elevated temperatures. | Valve FCHV0312 is used as a steam admission valve for the turbine-driven auxiliary feedwater pump. The valve is physically arranged downstream of two air operated valves that provide a source of steam from steam generators B and C. Each line is equipped with a bypass line that allows steam to be present at the trip throttle valve to maintain the line warm and free of moisture. Because valve FCHV0312 is a steam admission valve, it is constantly exposed to elevated temperatures. When reviewing operating experience, the team found that Mobile 28 grease, in tests performed for NUREG/CR-6750, "Performance of MOV Stem Lubricants at Elevated Temperature," experienced physical characteristic changes when exposed to elevated temperatures. | ||
Of particular concern in the operating experience was the observation that Mobile 28 changed from bright red in color to almost black when heated and that the lubricant appeared to have thickened and hardened. Additionally, an absorption test identified that elevated temperatures (around 250°F) increased the lubricant's tendency to lose oils and could result in increased stem friction. The team noted that the licensee's evaluation of the use of Mobile 28 grease was bounded at 215°F but actual operating conditions at valve FCHV0312 were approximately 300°F. The team reviewed the licensee's lubrication procedure and found that no guidance was provided as to the quantity of grease to use, how to properly purge the system of old lubricant, and at exactly what lubrication point new grease should be applied. At Callaway, these particular details are considered "skill of the craft." During the root cause investigation, the licensee interviewed mechanical maintenance staff and confirmed there was not consistent information provided on the amount of lubricants to use or the specific points to lubricate. The root cause analysis associated with Callaway Action Request 200904219, Enhancement 2.0, recommended revisions to Preventive Maintenance 0810863 and 0818214, to specify lubrication locations, amount and type of lubricant to use, and instructions to manually stroke the valve while lubricating through 12 Enclosure the trunnion screws to assure the trip throttle valve is fully lubricated across the entire length of the spindle. | Of particular concern in the operating experience was the observation that Mobile 28 changed from bright red in color to almost black when heated and that the lubricant appeared to have thickened and hardened. Additionally, an absorption test identified that elevated temperatures (around 250°F) increased the lubricant's tendency to lose oils and could result in increased stem friction. The team noted that the licensee's evaluation of the use of Mobile 28 grease was bounded at 215°F but actual operating conditions at valve FCHV0312 were approximately 300°F. | ||
The team reviewed the licensee's lubrication procedure and found that no guidance was provided as to the quantity of grease to use, how to properly purge the system of old lubricant, and at exactly what lubrication point new grease should be applied. At Callaway, these particular details are considered "skill of the craft." During the root cause investigation, the licensee interviewed mechanical maintenance staff and confirmed there was not consistent information provided on the amount of lubricants to use or the specific points to lubricate. The root cause analysis associated with Callaway Action Request 200904219, Enhancement 2.0, recommended revisions to Preventive Maintenance 0810863 and 0818214, to specify lubrication locations, amount and type of lubricant to use, and instructions to manually stroke the valve while lubricating through 12 Enclosure the trunnion screws to assure the trip throttle valve is fully lubricated across the entire length of the spindle. | |||
When reviewing the licensee's lubrication program, the inspectors examined the licensee's storage of lubricants and noted several expired consumables (grease, lubricants, oils, etc.) within the mechanical maintenance shop. The inspectors questioned the mechanical maintenance supervisor to determine if potential existed for use of expired lubricants in safety related applications. Based on interviews, the inspectors determined that the potential to use expired consumables in safety related applications exists since there is no procedural guidance that requires verification of a product's expiration date prior to use, it is simply an expectation. The licensee reviewed the work history associated with these products and verified that these materials were not used during work on safety related components. This issue was entered into the licensee's corrective action program as Callaway Action Request 200905038. Additionally, the inspectors observed that the licensee's trip throttle valve lubrication specification within the plant equipment database called for Exxon Nebula EP-1 which is not a vendor approved grease for the trip throttle valve. The team verified that current work instructions and procedures require the use of Mobile 28 grease. The team considered this a minor administrative issue and the licensee entered the discrepancy into their corrective action program as Callaway Action Request 200905032. Valve Replacement and Refurbishment Currently, the licensee replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system (Preventive Maintenance 0824900). The frequency of replacement is consistent with industry standards and vendor guidance which recommends a minimum 6-year replacement frequency. Callaway's frequency is more conservative because internal operating experience has demonstrated that the trip throttle valve develops too much seat leakage during the fourth operating cycle which results in increased corrosion of safety related components on the 1988 foot elevation of the auxiliary buildings as well as an increase in nonoperating bearing oil temperatures. The procedure used to rebuild the valve is contained within the preventive maintenance task. The scope of the work during a valve refurbishment includes: | When reviewing the licensee's lubrication program, the inspectors examined the licensee's storage of lubricants and noted several expired consumables (grease, lubricants, oils, etc.) within the mechanical maintenance shop. The inspectors questioned the mechanical maintenance supervisor to determine if potential existed for use of expired lubricants in safety related applications. Based on interviews, the inspectors determined that the potential to use expired consumables in safety related applications exists since there is no procedural guidance that requires verification of a product's expiration date prior to use, it is simply an expectation. The licensee reviewed the work history associated with these products and verified that these materials were not used during work on safety related components. This issue was entered into the licensee's corrective action program as Callaway Action Request 200905038. | ||
Additionally, the inspectors observed that the licensee's trip throttle valve lubrication specification within the plant equipment database called for Exxon Nebula EP-1 which is not a vendor approved grease for the trip throttle valve. The team verified that current work instructions and procedures require the use of Mobile 28 grease. The team considered this a minor administrative issue and the licensee entered the discrepancy into their corrective action program as Callaway Action Request 200905032. | |||
Valve Replacement and Refurbishment Currently, the licensee replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system (Preventive Maintenance 0824900). The frequency of replacement is consistent with industry standards and vendor guidance which recommends a minimum 6-year replacement frequency. Callaway's frequency is more conservative because internal operating experience has demonstrated that the trip throttle valve develops too much seat leakage during the fourth operating cycle which results in increased corrosion of safety related components on the 1988 foot elevation of the auxiliary buildings as well as an increase in nonoperating bearing oil temperatures. The procedure used to rebuild the valve is contained within the preventive maintenance task. The scope of the work during a valve refurbishment includes: | |||
* Disassembly of the valve | * Disassembly of the valve | ||
* General inspection of all valve components | * General inspection of all valve components | ||
| Line 114: | Line 189: | ||
14 Enclosure The inspectors reviewed Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," and found that the corrective actions suggested by Callaway Action Request 200203228 were not fully developed and that the procedure, as written, provided ambiguous guidance on how to lubricate valve FCHV0312. Specifically, the inspectors noted that the procedure only gives the option of repositioning the valve to access certain components but does not require the valve be manually mid-positioned to ensure adequate lubrication. The procedure does not specify what components of concern are to be lubricated when the valve is manually repositioned. The inspectors also observed during a system walkdown that the valve design is such that the valve spindle, in addition to the valve coupling, is not fully accessible from a static position since the sliding nut obstructs access to the spindle threads. While the sliding nut is equipped with Zerk type grease fittings that are ported to allow the lubricant to be applied to the spindle, the entire length of the spindle could only be lubricated through this point if the valve was manually operated while grease was injected at the fitting location. The inspectors also noted the procedure does not specify lubrication locations or the amount of lubricant to use and how to achieve a proper purge of old lubricant. The inspectors questioned engineering and maintenance staff on how these issues were addressed during trip throttle valve lubrications and were told that this level of work was within the skill of the craft. | 14 Enclosure The inspectors reviewed Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," and found that the corrective actions suggested by Callaway Action Request 200203228 were not fully developed and that the procedure, as written, provided ambiguous guidance on how to lubricate valve FCHV0312. Specifically, the inspectors noted that the procedure only gives the option of repositioning the valve to access certain components but does not require the valve be manually mid-positioned to ensure adequate lubrication. The procedure does not specify what components of concern are to be lubricated when the valve is manually repositioned. The inspectors also observed during a system walkdown that the valve design is such that the valve spindle, in addition to the valve coupling, is not fully accessible from a static position since the sliding nut obstructs access to the spindle threads. While the sliding nut is equipped with Zerk type grease fittings that are ported to allow the lubricant to be applied to the spindle, the entire length of the spindle could only be lubricated through this point if the valve was manually operated while grease was injected at the fitting location. The inspectors also noted the procedure does not specify lubrication locations or the amount of lubricant to use and how to achieve a proper purge of old lubricant. The inspectors questioned engineering and maintenance staff on how these issues were addressed during trip throttle valve lubrications and were told that this level of work was within the skill of the craft. | ||
In addition to the inadequacies identified by the NRC, the licensee's root cause investigation team also identified that the lubrication procedure for the turbine-driven auxiliary feedwater pump trip throttle valve has several inadequacies that could lead to improperly lubricating the valve. The licensee's review included interviews with mechanical maintenance staff which confirmed there was not consistent information provided on the amounts of lubricants to use or the specific points to lubricate. The root cause analysis associated with Callaway Action Request 200904219 recommended Enhancement 2.0 to revise the valve lubrication procedure to specify lubrication locations, amount and type of lubricant to use and instructions to manually stroke the valve while lubricating through the trunnion screws to assure the trip throttle valve is fully lubricated across the entire length of the spindle. | In addition to the inadequacies identified by the NRC, the licensee's root cause investigation team also identified that the lubrication procedure for the turbine-driven auxiliary feedwater pump trip throttle valve has several inadequacies that could lead to improperly lubricating the valve. The licensee's review included interviews with mechanical maintenance staff which confirmed there was not consistent information provided on the amounts of lubricants to use or the specific points to lubricate. The root cause analysis associated with Callaway Action Request 200904219 recommended Enhancement 2.0 to revise the valve lubrication procedure to specify lubrication locations, amount and type of lubricant to use and instructions to manually stroke the valve while lubricating through the trunnion screws to assure the trip throttle valve is fully lubricated across the entire length of the spindle. | ||
=====Analysis.===== | =====Analysis.===== | ||
| Line 128: | Line 203: | ||
=====Introduction.===== | =====Introduction.===== | ||
=== | === | ||
The team identified a green noncited violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control," for the failure to adequately evaluate the use of Mobile 28 grease for the turbine-driven auxiliary feedwater pump trip throttle valve. | The team identified a green noncited violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control," for the failure to adequately evaluate the use of Mobile 28 grease for the turbine-driven auxiliary feedwater pump trip throttle valve. | ||
=====Description.===== | =====Description.===== | ||
| Line 135: | Line 210: | ||
Valve FCHV0312 is used as a steam admission valve to the turbine-driven auxiliary feedwater pump. The valve is physically arranged downstream of two air operated valves that provide steam from the Steam Generators B and C. Since valve FCHV0312 is used as a steam admission valve, it is continuously exposed to high temperatures. The disposition of Request for Resolution 13409A determined that no change should be implemented since the proposed lubricant replacements were not acceptable greases for addressing long term hardening due to high temperatures at valve FCHV0312. | Valve FCHV0312 is used as a steam admission valve to the turbine-driven auxiliary feedwater pump. The valve is physically arranged downstream of two air operated valves that provide steam from the Steam Generators B and C. Since valve FCHV0312 is used as a steam admission valve, it is continuously exposed to high temperatures. The disposition of Request for Resolution 13409A determined that no change should be implemented since the proposed lubricant replacements were not acceptable greases for addressing long term hardening due to high temperatures at valve FCHV0312. | ||
On April 24, 1995, the licensee initiated Request for Resolution 16006A to change the grease used on trip throttle valve FCHV0312. Request for Resolution 16006A referenced Request for Resolution 13409A and noted that the Nebula EP-1 grease currently in use was not recommended for applications where continuous high temperatures can cause oil evaporation, resulting in hardening of the grease. The request for resolution documented that the valve vendor was contacted and recommended a high-temperature lithium-based grease such as Mobile 28. Mobile 28 grease is a high-temperature, anti-wear grease composed of a synthetic base fluid with a non-soap thickener. The final disposition of Request for Resolution 16006A approved the use of Mobile 28 grease in place of Exxon Nebula EP-1 for valve FCHV0312 since it was recommended by the valve manufacturer and the temperature of the valve FCHV0312 screw spindle was approximately 215°F which is within the temperature range recommended by the grease manufacturer. The inspectors reviewed Request for Resolution 16006A and noted that the evaluation provided no documentation as to what constitutes an appropriate relubrication interval as recommended in the vendor data sheet for Mobile 28. Additionally, the inspectors questioned if the assumed temperature of 215°F at the Valve FCHV0312 screw spindle was consistent with the current plant configuration. The licensee measured the temperature of various components on Valve FCHV0312 on June 23, 2009. Those 16 Enclosure measurements indicated that the valve exhibited temperatures ranging from 235°F at the sliding nut to 272°F at the valve coupling. Additional measurements were taken on June 29, 2009, following a planned surveillance of the turbine-driven auxiliary feedwater pump. Those measurements showed temperatures near 300°F at the valve coupling. Since the valve displayed temperatures exceeding those in Request for Resolution 16006A, the inspectors questioned if the use of Mobile 28 grease was appropriate, particularly since operating experience performed by Idaho National Lab and documented in NUREG/CR-6750, "Performance of MOV Stem Lubricants at Elevated Temperature," suggested Mobile 28 grease has a tendency to thicken and harden at temperatures exceeding 250°F. Additionally, the test performed in NUREG/CR-6750 identified that elevated temperatures increased the lubricant's tendency to release oils and could result in increased stem friction. Following questioning by the inspectors, the licensee initiated Callaway Action Request 200905067 and Request for Resolution 200905621 to determine if Mobile 28 grease was an appropriate lubricant for valve FCHV0312. The results of that analysis determined that the thread angle of the stem for valve FCHV0312 is similar to that of two of the stems tested in NUREG/CR-6750. While the coefficient of friction was determined to increase for those stems, the licensee determine that the maximum postulated coefficient of friction is bound by the current thrust and torque analysis for valve FCHV0312. The final conclusion of the Callaway action request was that while the recorded temperatures on valve FCHV0312 exceeded those evaluated for in Request for Resolution 16006A, the use of Mobile 28 grease is appropriate when coupled with an adequate relubrication interval. At the close of the inspection, the licensee was still evaluating the appropriate preventive maintenance frequency for lubrication of the valve. | On April 24, 1995, the licensee initiated Request for Resolution 16006A to change the grease used on trip throttle valve FCHV0312. Request for Resolution 16006A referenced Request for Resolution 13409A and noted that the Nebula EP-1 grease currently in use was not recommended for applications where continuous high temperatures can cause oil evaporation, resulting in hardening of the grease. The request for resolution documented that the valve vendor was contacted and recommended a high-temperature lithium-based grease such as Mobile 28. Mobile 28 grease is a high-temperature, anti-wear grease composed of a synthetic base fluid with a non-soap thickener. The final disposition of Request for Resolution 16006A approved the use of Mobile 28 grease in place of Exxon Nebula EP-1 for valve FCHV0312 since it was recommended by the valve manufacturer and the temperature of the valve FCHV0312 screw spindle was approximately 215°F which is within the temperature range recommended by the grease manufacturer. | ||
The inspectors reviewed Request for Resolution 16006A and noted that the evaluation provided no documentation as to what constitutes an appropriate relubrication interval as recommended in the vendor data sheet for Mobile 28. Additionally, the inspectors questioned if the assumed temperature of 215°F at the Valve FCHV0312 screw spindle was consistent with the current plant configuration. The licensee measured the temperature of various components on Valve FCHV0312 on June 23, 2009. Those 16 Enclosure measurements indicated that the valve exhibited temperatures ranging from 235°F at the sliding nut to 272°F at the valve coupling. Additional measurements were taken on June 29, 2009, following a planned surveillance of the turbine-driven auxiliary feedwater pump. Those measurements showed temperatures near 300°F at the valve coupling. Since the valve displayed temperatures exceeding those in Request for Resolution 16006A, the inspectors questioned if the use of Mobile 28 grease was appropriate, particularly since operating experience performed by Idaho National Lab and documented in NUREG/CR-6750, "Performance of MOV Stem Lubricants at Elevated Temperature," suggested Mobile 28 grease has a tendency to thicken and harden at temperatures exceeding 250°F. Additionally, the test performed in NUREG/CR-6750 identified that elevated temperatures increased the lubricant's tendency to release oils and could result in increased stem friction. | |||
Following questioning by the inspectors, the licensee initiated Callaway Action Request 200905067 and Request for Resolution 200905621 to determine if Mobile 28 grease was an appropriate lubricant for valve FCHV0312. The results of that analysis determined that the thread angle of the stem for valve FCHV0312 is similar to that of two of the stems tested in NUREG/CR-6750. While the coefficient of friction was determined to increase for those stems, the licensee determine that the maximum postulated coefficient of friction is bound by the current thrust and torque analysis for valve FCHV0312. The final conclusion of the Callaway action request was that while the recorded temperatures on valve FCHV0312 exceeded those evaluated for in Request for Resolution 16006A, the use of Mobile 28 grease is appropriate when coupled with an adequate relubrication interval. At the close of the inspection, the licensee was still evaluating the appropriate preventive maintenance frequency for lubrication of the valve. | |||
=====Analysis.===== | =====Analysis.===== | ||
The performance deficiency associated with this finding involved the licensee's use of nonconservative assumptions in the 1995 evaluation of Mobile 28 grease as a lubricant for valve FCHV0312. This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of design control and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," | The performance deficiency associated with this finding involved the licensee's use of nonconservative assumptions in the 1995 evaluation of Mobile 28 grease as a lubricant for valve FCHV0312. This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of design control and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," | ||
the issue screened as having very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not have a crosscutting aspect since the inadequate 1995 lubrication evaluation was not reflective of current licensee performance. | the issue screened as having very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not have a crosscutting aspect since the inadequate 1995 lubrication evaluation was not reflective of current licensee performance. | ||
=====Enforcement.===== | =====Enforcement.===== | ||
| Line 166: | Line 245: | ||
=====Enforcement.===== | =====Enforcement.===== | ||
Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," requires, in part, that activities affecting quality shall be prescribed by documented instructions or drawings, of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions or drawings. Contrary to the above, on September 14, 2007, the licensee failed to enter several adverse conditions identified during rebuild of their safety related turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 into their corrective action program as required by Section 4.1 of Procedure APA-ZZ-00500, "Corrective Action Program," Revision 44, that states a Callaway action request be initiated for a condition that could credibly impact nuclear safety, radiological safety, personnel safety, or plant reliability. Because of the very low safety significance of this finding and because the licensee has entered this issue into their corrective action program as Callaway Action Request 200905053, this violation is being treated as a noncited 19 Enclosure violation in accordance with Section VI.A.1 of the Enforcement Policy: NCV 05000483/2009009-04, "Failure to Enter Conditions Adverse to Quality Associated with the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve into the Corrective Action Program." | Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," requires, in part, that activities affecting quality shall be prescribed by documented instructions or drawings, of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions or drawings. Contrary to the above, on September 14, 2007, the licensee failed to enter several adverse conditions identified during rebuild of their safety related turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 into their corrective action program as required by Section 4.1 of Procedure APA-ZZ-00500, "Corrective Action Program," Revision 44, that states a Callaway action request be initiated for a condition that could credibly impact nuclear safety, radiological safety, personnel safety, or plant reliability. Because of the very low safety significance of this finding and because the licensee has entered this issue into their corrective action program as Callaway Action Request 200905053, this violation is being treated as a noncited 19 Enclosure violation in accordance with Section VI.A.1 of the Enforcement Policy: NCV 05000483/2009009-04, "Failure to Enter Conditions Adverse to Quality Associated with the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve into the Corrective Action Program." | ||
1R4 Review of Root Cause Analysis, Extent of Condition and Corrective Actions On June 8, 2009, the licensee established a root cause analysis team to investigate the facts and identify the causes associated with the failure of the turbine-driven auxiliary feedwater pump trip throttle valve. The team conducted their review in accordance with Procedure APA-ZZ-00500, Appendix 12, "Significant Adverse Condition." The licensee's procedure requires the team to: | |||
* Identify and validate root and contributing causes | * Identify and validate root and contributing causes | ||
* Conduct an extent of condition review | * Conduct an extent of condition review | ||
* Determine extent of cause | * Determine extent of cause | ||
* Develop corrective actions | * Develop corrective actions The licensee's final root cause analysis was completed on July 16, 2009. | ||
Root Cause Methodology | Root Cause Methodology The licensee performed their analysis utilizing a structured root cause analysis method in accordance with Procedure APA-ZZ-00500, Appendix 12, "Significant Adverse Condition - Significance Level 1." A fault tree was developed for FCHV0312 to determine the valve's failure mechanism. An event and causal factors chart was created to document the sequence of events and the casual factors that led to the failure of valve FCHV0312. The results of that analysis determined valve FCHV0312 was not lubricated properly because, during Refuel 16, the lubrication portion of Procedure MPM-FC-QK-001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was not performed which resulted in increased valve spindle friction. This increased friction caused the valve's torque switch to trip which prevented the valve from opening. | ||
Root Cause and Corrective Actions to Prevent Recurrence | Root Cause and Corrective Actions to Prevent Recurrence The licensee determined that the root cause of the event was that the maintenance supervisor did not review the full scope of the replacement preventive maintenance procedure when closing the lubrication portion of Procedure MPM-FC-QK-001, "Auxiliary Feedwater Pump Turbine Annual Inspection," during Refueling Outage 16. Procedure APA-ZZ-00320, "Work Execution," Section 4.12, required the maintenance supervisor to perform a review to ensure the entire work scope of the superseded work document is included in the superseding document. | ||
As a corrective action to prevent recurrence, the licensee proposed revision to the replacement preventive maintenance procedure to incorporate valve lubrication requirements. | As a corrective action to prevent recurrence, the licensee proposed revision to the replacement preventive maintenance procedure to incorporate valve lubrication requirements. | ||
| Line 189: | Line 270: | ||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
The team reviewed the licensee's root cause analysis to determine if it was conducted to a level of detail commensurate with the significance of the problem. As part of their review, the inspectors interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program. Additionally, the team interviewed the root cause team members and the members from the licensee's Corrective Action Review Board. The team reviewed the licensee's corrective actions to ensure they addressed the extent of condition and whether they were adequate to prevent recurrence. In particular, the team reviewed station procedures and processes to determine if any other motor-operated valves may have been improperly lubricated. | The team reviewed the licensee's root cause analysis to determine if it was conducted to a level of detail commensurate with the significance of the problem. As part of their review, the inspectors interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program. Additionally, the team interviewed the root cause team members and the members from the licensee's Corrective Action Review Board. | ||
The team reviewed the licensee's corrective actions to ensure they addressed the extent of condition and whether they were adequate to prevent recurrence. In particular, the team reviewed station procedures and processes to determine if any other motor-operated valves may have been improperly lubricated. | |||
====b. Observations and Findings==== | ====b. Observations and Findings==== | ||
The inspectors determined that the licensee's analysis accurately captured the root cause of the event. Since the event was determined to be caused by an individual 21 Enclosure human error, the inspectors noted that the licensee appropriately identified a need to implement a defense-in-depth mechanism to ensure the turbine-driven auxiliary feedwater pump trip throttle valve is adequately lubricated. The inspectors found that the corrective action to prevent recurrence would likely ensure that the valve is fully lubricated in the future. Effectiveness reviews were implemented by the licensee to ensure the corrective actions implemented were sufficiently robust as to address the root cause. The inspectors did note that the final root cause analysis report was narrowly focused and failed to consider all potential causal factors associated with the failure of the turbine-driven auxiliary feedwater pump trip throttle valve. | The inspectors determined that the licensee's analysis accurately captured the root cause of the event. Since the event was determined to be caused by an individual 21 Enclosure human error, the inspectors noted that the licensee appropriately identified a need to implement a defense-in-depth mechanism to ensure the turbine-driven auxiliary feedwater pump trip throttle valve is adequately lubricated. The inspectors found that the corrective action to prevent recurrence would likely ensure that the valve is fully lubricated in the future. Effectiveness reviews were implemented by the licensee to ensure the corrective actions implemented were sufficiently robust as to address the root | ||
cause. The inspectors did note that the final root cause analysis report was narrowly focused and failed to consider all potential causal factors associated with the failure of the turbine-driven auxiliary feedwater pump trip throttle valve. | |||
The team found that the final root cause analysis lacked adequate justification to exclude several of the identified enhancements and other issues as contributors to the event. Specifically, the inspectors noted the following: | The team found that the final root cause analysis lacked adequate justification to exclude several of the identified enhancements and other issues as contributors to the event. Specifically, the inspectors noted the following: | ||
* The replacement procedure for valve FCHV0312 did not have specific lubrication instructions. The team confirmed through interviews that typical replacement procedures include instructions to restore the replaced equipment to a functional status. The difference between the valve FCHV0312 replacement procedure and typical replacement procedures caused the mechanical maintenance supervisor to assume the valve was adequately lubricated at completion of the replacement. | * The replacement procedure for valve FCHV0312 did not have specific lubrication instructions. The team confirmed through interviews that typical replacement procedures include instructions to restore the replaced equipment to a functional status. The difference between the valve FCHV0312 replacement procedure and typical replacement procedures caused the mechanical maintenance supervisor to assume the valve was adequately lubricated at completion of the replacement. | ||
* Engineering did not perform an adequate review of the job routing request inquiring if additional inspection of the valve was required. The lack of technical rigor was largely attributable to the fact that no protocol exists on how job routing requests should be evaluated and a fundamental misunderstanding between departments on the intention of such requests. | * Engineering did not perform an adequate review of the job routing request inquiring if additional inspection of the valve was required. The lack of technical rigor was largely attributable to the fact that no protocol exists on how job routing requests should be evaluated and a fundamental misunderstanding between departments on the intention of such requests. | ||
* Communications between maintenance and engineering lacked sufficient detail to ensure the required preventive maintenance tasks for valve FCHV0312 were accomplished. Additionally, the inspectors noted several potential flawed defenses and latent organizational weaknesses which were not documented in the final report. Specifically, the inspectors noted that: | * Communications between maintenance and engineering lacked sufficient detail to ensure the required preventive maintenance tasks for valve FCHV0312 were accomplished. | ||
Additionally, the inspectors noted several potential flawed defenses and latent organizational weaknesses which were not documented in the final report. Specifically, the inspectors noted that: | |||
* The replacement program for valve FCHV0312 allows for the valve to be not adequately lubricated if the replacement and lubrication preventive maintenance tasks are performed out of sequence. Reliance is placed on proper scheduling that increases the probability of equipment problems due to human error. | * The replacement program for valve FCHV0312 allows for the valve to be not adequately lubricated if the replacement and lubrication preventive maintenance tasks are performed out of sequence. Reliance is placed on proper scheduling that increases the probability of equipment problems due to human error. | ||
* Procedure APA-ZZ-00320 provides ambiguous guidance on how to close work documents. Additionally, the procedure, as written, allows for single individuals to close preventive maintenance tasks. The lack of a required second or peer check when closing incomplete preventive maintenance tasks could result in less than adequate technical review. | * Procedure APA-ZZ-00320 provides ambiguous guidance on how to close work documents. Additionally, the procedure, as written, allows for single individuals to close preventive maintenance tasks. The lack of a required second or peer check when closing incomplete preventive maintenance tasks could result in less than adequate technical review. | ||
* The lubrication frequency for valve FCHV0312 does not provide sufficient margin in that if a preventive maintenance task is missed or deferred, the potential exists to impact equipment operation. | * The lubrication frequency for valve FCHV0312 does not provide sufficient margin in that if a preventive maintenance task is missed or deferred, the potential exists to impact equipment operation. | ||
22 Enclosure The team noted that the comprehensiveness of the final root cause analysis was impacted by a failure of the licensee to interview all pertinent individuals and fundamental errors in the use of systematic analytical techniques such as TapRoot and event and causal factor charting. Additionally, less than adequate management oversight of the root cause process, such as management sponsor and Corrective Action Review Board reviews, allowed the final root cause to be issued without thoroughly evaluating all significant causal factors. The items identified by the inspection team were discussed with the licensee. As a result, the licensee initiated Callaway Action Request 200906143 to examine the quality of the root cause analysis performed and to determine if programmatic issues in cause evaluation exist. 1R5 Review of Operating Experience a. Scope The team reviewed internal operating experience by obtaining a list of plant corrective action documents related to the auxiliary feedwater system and selecting those documents related to lubrication and maintenance of the turbine-driven auxiliary feedwater pump trip throttle valve. The team further reviewed the licensee's review of industry operating experience for the auxiliary feedwater system and lubrication problems with motor-operated valves. The team review included inspection of the licensee's operating experience program and specific review of related operating experience during the root cause investigation for the May 25, 2009, failure of the turbine-driven auxiliary feedwater pump trip throttle valve. For external operating experience, the NRC Operating Experience Branch provided the results of keyword searches related to motor-operated valve stem lubrication issues, Terry turbine Gimpel 4-Inch valves, and findings associated with turbine-driven pumps. The NRC Operating Experience Branch provided a list of licensee event reports, NRC information notices, NUREG documents and other operating experience information. The team selected operating experience information that was applicable to this inspection and reviewed whether the licensee had addressed the items in their root cause analyses related to these events or had processed the information through their operating experience program. As part of their review, the inspectors performed an auxiliary feedwater system walkdown to determine if applicable industry operating experience had been incorporated into system design and maintenance practices. | 22 Enclosure The team noted that the comprehensiveness of the final root cause analysis was impacted by a failure of the licensee to interview all pertinent individuals and fundamental errors in the use of systematic analytical techniques such as TapRoot and event and causal factor charting. Additionally, less than adequate management oversight of the root cause process, such as management sponsor and Corrective Action Review Board reviews, allowed the final root cause to be issued without thoroughly evaluating all significant causal factors. | ||
The items identified by the inspection team were discussed with the licensee. As a result, the licensee initiated Callaway Action Request 200906143 to examine the quality of the root cause analysis performed and to determine if programmatic issues in cause evaluation exist. | |||
1R5 Review of Operating Experience | |||
====a. Scope==== | |||
The team reviewed internal operating experience by obtaining a list of plant corrective action documents related to the auxiliary feedwater system and selecting those documents related to lubrication and maintenance of the turbine-driven auxiliary feedwater pump trip throttle valve. The team further reviewed the licensee's review of industry operating experience for the auxiliary feedwater system and lubrication problems with motor-operated valves. The team review included inspection of the licensee's operating experience program and specific review of related operating experience during the root cause investigation for the May 25, 2009, failure of the turbine-driven auxiliary feedwater pump trip throttle valve. | |||
For external operating experience, the NRC Operating Experience Branch provided the results of keyword searches related to motor-operated valve stem lubrication issues, Terry turbine Gimpel 4-Inch valves, and findings associated with turbine-driven pumps. The NRC Operating Experience Branch provided a list of licensee event reports, NRC information notices, NUREG documents and other operating experience information. The team selected operating experience information that was applicable to this inspection and reviewed whether the licensee had addressed the items in their root cause analyses related to these events or had processed the information through their operating experience program. As part of their review, the inspectors performed an auxiliary feedwater system walkdown to determine if applicable industry operating experience had been incorporated into system design and maintenance practices. | |||
====b. Findings and Observations==== | ====b. Findings and Observations==== | ||
| Line 217: | Line 313: | ||
While comparing the licensee's Generic Letter 96-05 response to Calculation FC-20, "Manual Operation of FCHV0312," the inspectors identified that the licensee's value listed for differential thrust to open was 3172 pounds-force but the licensee's generic letter response indicated the design differential thrust to open valve FCHV0312 is 4350 pounds-force. Additionally, dynamic testing from Refueling Outage 16 in October 2008 determined that the differential thrust to open with error was 5316 pounds-force. Subsequent review by the licensee determined that the value contained within the plant equipment database was not up to date and contained a nonconservative value for design differential pressure thrust to open. While the information contained in the licensee's database was not up to date for the current plant configuration, the current required open thrust was determined to be bounded by the maximum allowed differential pressure thrust for valve FCHV0312 which is 7106 pounds. The licensee entered the discrepancy into their corrective action program as Callaway Action Request 200905239. | While comparing the licensee's Generic Letter 96-05 response to Calculation FC-20, "Manual Operation of FCHV0312," the inspectors identified that the licensee's value listed for differential thrust to open was 3172 pounds-force but the licensee's generic letter response indicated the design differential thrust to open valve FCHV0312 is 4350 pounds-force. Additionally, dynamic testing from Refueling Outage 16 in October 2008 determined that the differential thrust to open with error was 5316 pounds-force. Subsequent review by the licensee determined that the value contained within the plant equipment database was not up to date and contained a nonconservative value for design differential pressure thrust to open. While the information contained in the licensee's database was not up to date for the current plant configuration, the current required open thrust was determined to be bounded by the maximum allowed differential pressure thrust for valve FCHV0312 which is 7106 pounds. The licensee entered the discrepancy into their corrective action program as Callaway Action Request 200905239. | ||
24 Enclosure 1R7 Potential Generic Issues a. Scope The team evaluated the failure of Callaway's turbine-driven auxiliary feedwater pump trip throttle valve due to inadequate lubrication to determine whether any potential generic issues should be communicated to the industry (e.g., information notices, generic letters, and bulletins). | 24 Enclosure 1R7 Potential Generic Issues | ||
====a. Scope==== | |||
The team evaluated the failure of Callaway's turbine-driven auxiliary feedwater pump trip throttle valve due to inadequate lubrication to determine whether any potential generic issues should be communicated to the industry (e.g., information notices, generic letters, and bulletins). | |||
====b. Findings and Observations==== | ====b. Findings and Observations==== | ||
| Line 224: | Line 323: | ||
==OTHER ACTIVITIES== | ==OTHER ACTIVITIES== | ||
{{a|4OA2}} | {{a|4OA2}} | ||
==4OA2 Identification and Resolution of Problems== | ==4OA2 Identification and Resolution of Problems (71152== | ||
) | |||
====a. Inspection Scope==== | ====a. Inspection Scope==== | ||
| Line 233: | Line 333: | ||
=====Introduction.===== | =====Introduction.===== | ||
The team identified a green noncited violation of Technical Specification Limiting Condition for Operation 3.0.4 for entering Mode 3 with the turbine-driven auxiliary feedwater pump inoperable. | The team identified a green noncited violation of Technical Specification Limiting Condition for Operation 3.0.4 for entering Mode 3 with the turbine-driven auxiliary feedwater pump inoperable. | ||
=====Description.===== | =====Description.===== | ||
| Line 240: | Line 340: | ||
While in Mode 3, the licensee performed Jobs 06113631 and 07506205 to test the turbine-driven auxiliary feedwater pump using the normal steam supply. During performance of the jobs, the instrumentation and controls engineer noted that the 25 Enclosure channel two, governor valve position indicator was erratic. Troubleshooting was performed and it was found that the servo control valve replaced during Refueling Outage 16 was faulty. At 4:50 p.m. on November 5, 2008, the licensee entered the applicable technical specification and initiated repairs to the turbine-driven auxiliary feedwater pump. Postmaintenance testing was performed and the licensee exited the technical specification at 12:15 p.m. on November 6, 2008. | While in Mode 3, the licensee performed Jobs 06113631 and 07506205 to test the turbine-driven auxiliary feedwater pump using the normal steam supply. During performance of the jobs, the instrumentation and controls engineer noted that the 25 Enclosure channel two, governor valve position indicator was erratic. Troubleshooting was performed and it was found that the servo control valve replaced during Refueling Outage 16 was faulty. At 4:50 p.m. on November 5, 2008, the licensee entered the applicable technical specification and initiated repairs to the turbine-driven auxiliary feedwater pump. Postmaintenance testing was performed and the licensee exited the technical specification at 12:15 p.m. on November 6, 2008. | ||
The team reviewed corrective action documents associated with the turbine overspeed and the faulty governor servo control valve. The inspectors questioned if the two corrective action documents were related and specifically, if the cause of the turbine overspeed that occurred in Mode 4 could have been related to the faulty governor servo control valve discovered while the unit was in Mode 3. Additionally, the team questioned if the turbine-driven auxiliary feedwater pump could have been inoperable during the mode change that occurred on November 4, 2008, in violation of Technical Specification Limiting Condition for Operation 3.0.4. The licensee reviewed the Refueling Outage 16 work history associated with the turbine-driven auxiliary feedwater pump governor and found that currents for the servo position sensor were lower than expected and the abnormal readings indicated that the remote servo valve was not operating properly. Based on this evidence, the licensee concluded that the faulty servo control valve discovered on November 5, 2008, was responsible for the overspeed of the turbine-driven auxiliary feedwater pump that occurred in Mode 4 and that the equipment was inoperable during the transition to the Mode 3 change that occurred on November 4, 2008. | The team reviewed corrective action documents associated with the turbine overspeed and the faulty governor servo control valve. The inspectors questioned if the two corrective action documents were related and specifically, if the cause of the turbine overspeed that occurred in Mode 4 could have been related to the faulty governor servo control valve discovered while the unit was in Mode 3. Additionally, the team questioned if the turbine-driven auxiliary feedwater pump could have been inoperable during the mode change that occurred on November 4, 2008, in violation of Technical Specification Limiting Condition for Operation 3.0.4. | ||
The licensee reviewed the Refueling Outage 16 work history associated with the turbine-driven auxiliary feedwater pump governor and found that currents for the servo position sensor were lower than expected and the abnormal readings indicated that the remote servo valve was not operating properly. Based on this evidence, the licensee concluded that the faulty servo control valve discovered on November 5, 2008, was responsible for the overspeed of the turbine-driven auxiliary feedwater pump that occurred in Mode 4 and that the equipment was inoperable during the transition to the Mode 3 change that occurred on November 4, 2008. | |||
=====Analysis.===== | =====Analysis.===== | ||
| Line 250: | Line 352: | ||
==4OA3 Event Follow-up== | ==4OA3 Event Follow-up== | ||
{{IP sample|IP=IP 71153}} | {{IP sample|IP=IP 71153}} | ||
(Closed) Licensee Event Report (LER) 05000483/2009002-00: Turbine-Driven Auxiliary Feedwater Pump Failed to Start During Surveillance Test On May 25, 2009, the Callaway plant turbine-driven auxiliary feedwater pump failed to start during a planned surveillance run. The licensee determined that the failure of the turbine-driven auxiliary feedwater pump was due to an inadequately lubricated trip throttle valve. The valve was inadequately lubricated because the licensee inappropriately closed the lubrication portion of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," during Refueling Outage 16. Subsequent review by the licensee determined that while the exact timing of the valve failure could not be determined, but it was reasonable to assume that the degraded condition existed for a period greater than the Technical Specification allowed completion time for the auxiliary feedwater system. Consequently, the event resulted in a reportable event per the requirements of 10 CFR 50.73(a)(2)(i)(B), any operation or condition which was prohibited by the plant's Technical Specifications. Additionally, since the motor-driven auxiliary feedwater pump train A was inoperable just prior to discovery of the degraded condition, the event was determined to be reportable per 10 CFR 50.73(a)(2)(v), as a condition that could have prevented fulfillment of a safety function and 10 CFR 50.73(a)(2)(ii)(B), as an analyzed condition that significantly degraded plant safety. The licensee submitted a licensee event report on July 21, 2009. The inspectors reviewed the licensee's submittal and determined that the report adequately documented the summary of the event including the potential safety consequences and corrective actions required to address the performance deficiency. The inspectors identified a self-revealing violation of Technical Specification 3.7.5, "Auxiliary Feedwater System." The enforcement aspects of the violation are discussed in Section 1R2 of this report as AV 05000483/2009009-01, "Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve." This LER is closed. | (Closed) Licensee Event Report (LER) 05000483/2009002-00: Turbine-Driven Auxiliary Feedwater Pump Failed to Start During Surveillance Test On May 25, 2009, the Callaway plant turbine-driven auxiliary feedwater pump failed to start during a planned surveillance run. The licensee determined that the failure of the turbine-driven auxiliary feedwater pump was due to an inadequately lubricated trip throttle valve. The valve was inadequately lubricated because the licensee inappropriately closed the lubrication portion of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," during Refueling Outage 16. Subsequent review by the licensee determined that while the exact timing of the valve failure could not be determined, but it was reasonable to assume that the degraded condition existed for a period greater than the Technical Specification allowed completion time for the auxiliary feedwater system. Consequently, the event resulted in a reportable event per the requirements of 10 CFR 50.73(a)(2)(i)(B), any operation or condition which was prohibited by the plant's Technical Specifications. Additionally, since the motor-driven auxiliary feedwater pump train A was inoperable just prior to discovery of the degraded condition, the event was determined to be reportable per 10 CFR 50.73(a)(2)(v), as a condition that could have prevented fulfillment of a safety function and 10 CFR 50.73(a)(2)(ii)(B), as an analyzed condition that significantly degraded plant safety. The licensee submitted a licensee event report on July 21, 2009. The inspectors reviewed the licensee's submittal and determined that the report adequately documented the summary of the event including the potential safety consequences and corrective actions required to address the performance deficiency. The inspectors identified a self-revealing violation of Technical Specification 3.7.5, "Auxiliary Feedwater System." The enforcement aspects of the violation are discussed in Section 1R2 of this report as AV 05000483/2009009-01, "Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve." This LER is closed. | ||
{{a|4OA6}} | {{a|4OA6}} | ||
==4OA6 Meetings, Including Exit On June 26, 2009, the team presented the preliminary results of this inspection at the end of the onsite week to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of his staff who acknowledged the findings. | ==4OA6 Meetings, Including Exit== | ||
The team verified that no proprietary information was retained. On September 2, 2009, the team leader presented the final results of the inspection to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of the licensee staff who acknowledged the findings. The team verified that no proprietary information was retained. | |||
On June 26, 2009, the team presented the preliminary results of this inspection at the end of the onsite week to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of his staff who acknowledged the findings. The team verified that no proprietary information was retained. | |||
On September 2, 2009, the team leader presented the final results of the inspection to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of the licensee staff who acknowledged the findings. The team verified that no proprietary information was retained. | |||
=SUPPLEMENTAL INFORMATION= | =SUPPLEMENTAL INFORMATION= | ||
| Line 276: | Line 382: | ||
: [[contact::S. Petzel]], Engineer, Regulatory Affairs | : [[contact::S. Petzel]], Engineer, Regulatory Affairs | ||
: [[contact::J. Pitts]], Supervising Engineer, Performance | : [[contact::J. Pitts]], Supervising Engineer, Performance | ||
: [[contact::S. Sandbothe]], Manager, Regulatory Affairs | : [[contact::S. Sandbothe]], Manager, Regulatory Affairs | ||
==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED== | ==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED== | ||
Open | Open | ||
: 050000483/2009009-01 AV Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve | : 050000483/2009009-01 AV Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve | ||
(Section 1R2) | (Section 1R2) | ||
===Opened and Closed=== | ===Opened and Closed=== | ||
: 050000483/2009009-02 NCV Failure to Maintain an Adequate Lubrication Procedure for Valve FCHV0312 (Section 1R3) | : 050000483/2009009-02 NCV Failure to Maintain an Adequate Lubrication Procedure for Valve FCHV0312 (Section 1R3) | ||
: 050000483/2009009-03 NCV Failure to Adequately Evaluate the Use of Mobile 28 Grease for the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve (Section 1R3) | : 050000483/2009009-03 NCV Failure to Adequately Evaluate the Use of Mobile 28 Grease for the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve (Section 1R3) | ||
: 050000483/2009009-04 NCV Failure to Enter Conditions Adverse to Quality Associated with the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve into the Corrective Action Program (Section 1R3) | : 050000483/2009009-04 NCV Failure to Enter Conditions Adverse to Quality Associated with the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve into the Corrective Action Program (Section 1R3) | ||
: 050000483/2009009-05 NCV Failure to Ensure Turbine-Driven Auxiliary Feedwater Pump is Operable Prior to Entry into Mode 3 (Section 4OA2) | : 050000483/2009009-05 NCV Failure to Ensure Turbine-Driven Auxiliary Feedwater Pump is Operable Prior to Entry into Mode 3 (Section 4OA2) | ||
Attachment 1 | Attachment 1 | ||
===Closed=== | ===Closed=== | ||
: [[Closes LER::05000483/LER-2009-002]]-00 LER Turbine-Driven Auxiliary Feedwater Pump Failed to Start During Surveillance Test (Section 4OA3) | : [[Closes LER::05000483/LER-2009-002]]-00 LER Turbine-Driven Auxiliary Feedwater Pump Failed to Start During Surveillance Test (Section 4OA3) | ||
==DOCUMENTS REVIEWED== | ==DOCUMENTS REVIEWED== | ||
CALLAWAY ACTION REQUESTS | CALLAWAY ACTION REQUESTS | ||
| Line 317: | Line 427: | ||
: 200904840 | : 200904840 | ||
: 200905004 | : 200905004 | ||
: 200905032 | : 200905032 | ||
: CALCULATIONS | : CALCULATIONS | ||
: NUMBER TITLE REVISION | : NUMBER TITLE REVISION | ||
: FC-20 Manual Operation of FCHV0312 0 | : FC-20 Manual Operation of FCHV0312 0 | ||
: ZZ-224 MOV Sizing Calculation 11 | : ZZ-224 MOV Sizing Calculation 11 | ||
: DRAWINGS | : DRAWINGS | ||
: NUMBER TITLE REVISION E-23FC23(Q) Schematic Diagram Aux FWP Turbine Trip throttle Vlv 19 M-22AL01(Q) Piping and Instrumentation Diagram Auxiliary Feedwater System 33 | : NUMBER TITLE REVISION E-23FC23(Q) Schematic Diagram Aux FWP Turbine Trip throttle Vlv 19 M-22AL01(Q) Piping and Instrumentation Diagram Auxiliary Feedwater System 33 | ||
| Line 328: | Line 438: | ||
: ASA 900# Trip Throttle Valve Top Mechanism 3"-900# Inlet and 3"-900# Outlet with Hard Packing, Oil Trip, Double Leakoff, Hard Trip, Corner Packing A P-4975 Parts List Top Mechanism Trip Throttle Valve with Hard Packing 0 P-4979 Parts List Top Mechanism Trip Throttle Valve with Hard Packing Auxiliary Parts 0 MODIFICATION PACKAGES | : ASA 900# Trip Throttle Valve Top Mechanism 3"-900# Inlet and 3"-900# Outlet with Hard Packing, Oil Trip, Double Leakoff, Hard Trip, Corner Packing A P-4975 Parts List Top Mechanism Trip Throttle Valve with Hard Packing 0 P-4979 Parts List Top Mechanism Trip Throttle Valve with Hard Packing Auxiliary Parts 0 MODIFICATION PACKAGES | ||
: NUMBER TITLE REVISION | : NUMBER TITLE REVISION | ||
: MP 05-1003 Trip Throttle Valve Gear Ration Change 100:1 0 | : MP 05-1003 Trip Throttle Valve Gear Ration Change 100:1 0 | ||
: Attachment 1 PROCEDURES | : Attachment 1 PROCEDURES | ||
: NUMBER TITLE REVISION | : NUMBER TITLE REVISION | ||
| Line 365: | Line 475: | ||
: OTN-AL-00001 Turbine Driven Auxiliary Feedwater Pump Trip/Throttle Appendix A Valve Trip Check and Reset 1 | : OTN-AL-00001 Turbine Driven Auxiliary Feedwater Pump Trip/Throttle Appendix A Valve Trip Check and Reset 1 | ||
: OTN-AL-00001 Turbine Driven Auxiliary Feedwater Pump Trip/Throttle Appendix A Valve Trip Check and Reset 2 | : OTN-AL-00001 Turbine Driven Auxiliary Feedwater Pump Trip/Throttle Appendix A Valve Trip Check and Reset 2 | ||
: OTS-FC-00004 Tripping Sequence of Auxiliary Feedwater Pump Turbine 9 OPERATING EXPERIENCE INFORMATION South Texas Project Unit 2, | : OTS-FC-00004 Tripping Sequence of Auxiliary Feedwater Pump Turbine 9 OPERATING EXPERIENCE INFORMATION | ||
: South Texas Project Unit 2, | |||
: LER 2007-001-00 | : LER 2007-001-00 | ||
: South Texas Project Unit 2, | : South Texas Project Unit 2, | ||
| Line 379: | Line 490: | ||
: NRC | : NRC | ||
: RIS 2001-015, Performance of | : RIS 2001-015, Performance of | ||
: DC-Powered Motor-Operated Valve Actuators NUREG/CR-6750, Performance of MOV Stem Lubricants at Elevated Temperatures, October 2001 NUREG/CR-6806, MOV Stem Lubricant Aging Research, March 2003 Part 21 Report 2001-35, Material Substitution of Grade 1212 Steel in place of grade 1018 carbon steel specified by designed | : DC-Powered Motor-Operated Valve Actuators NUREG/CR-6750, Performance of MOV Stem Lubricants at Elevated Temperatures, October | ||
: 2001 NUREG/CR-6806, MOV Stem Lubricant Aging Research, March 2003 Part 21 Report 2001-35, Material Substitution of Grade 1212 Steel in place of grade 1018 carbon steel specified by designed | |||
: JOBS | : JOBS | ||
: W219154 | : W219154 | ||
| Line 388: | Line 500: | ||
: 09003598 | : 09003598 | ||
: 09502426 | : 09502426 | ||
: MISCELLANEOUS | : MISCELLANEOUS | ||
: Request for Resolution 16006A, Change Grease on FCHV0312 Trip Throttle Valve, April 24, 1995 Letter from Jack Donohew, NRC to Garry Randolph, Union Electric Company, Subject: Callaway Plant, Unit 1 - Generic Letter 96-05, "Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves," Dated May 30, 2001 Letter from Mel Gray, NRC to Garry Randolph, Union Electric Company, Subject: Request for Additional Information - Generic Letter 96-05, "Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves," Callaway Plant, Unit 1, Dated June 18, 1999 | : Request for Resolution 16006A, Change Grease on FCHV0312 Trip Throttle Valve, April 24, 1995 Letter from Jack Donohew, NRC to Garry Randolph, Union Electric Company, Subject: Callaway Plant, Unit 1 - Generic Letter 96-05, "Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves," Dated May 30, 2001 Letter from Mel Gray, NRC to Garry Randolph, Union Electric Company, Subject: Request for Additional Information - Generic Letter 96-05, "Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves," Callaway Plant, Unit 1, Dated June 18, 1999 | ||
: Attachment 1 Letter | : Attachment 1 Letter | ||
| Line 394: | Line 506: | ||
: ULNRC-04075, Docket Number 50-483, Callaway Plant Union Electric Company NRC Generic Letter 96-05 Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves, Dated July 28, 1999 Letter | : ULNRC-04075, Docket Number 50-483, Callaway Plant Union Electric Company NRC Generic Letter 96-05 Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves, Dated July 28, 1999 Letter | ||
: ULNRC-04430, Docket Number 50-483, Callaway Plant Union Electric Company NRC Generic Letter 96-05 Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves, Dated March 30, 2001 | : ULNRC-04430, Docket Number 50-483, Callaway Plant Union Electric Company NRC Generic Letter 96-05 Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves, Dated March 30, 2001 | ||
: Attachment 2 | : Attachment 2 | ||
: June 17, 2009 | : June 17, 2009 | ||
: MEMORANDUM TO: Jeremy Groom, Resident Inspector, Team Leader Projects Branch B | : MEMORANDUM TO: Jeremy Groom, Resident Inspector, Team Leader Projects Branch B | ||
: Division of Reactor Projects | : Division of Reactor Projects Michael Chambers, Resident Inspector | ||
: Projects Branch C | : Projects Branch C | ||
: Division of Reactor Projects | : Division of Reactor Projects | ||
: FROM: | : FROM: | ||
: Dwight Chamberlain, Director /RA/ AVegel for | : Dwight Chamberlain, Director /RA/ AVegel for | ||
: Division of Reactor Projects | : Division of Reactor Projects | ||
: SUBJECT: | : SUBJECT: | ||
: CHARTER FOR SPECIAL INSPECTION INVOLVING THE FAILURE OF THE TURBINE DRIVEN AUXILIARY FEEDWATER PUMP TO START AT | : CHARTER FOR SPECIAL INSPECTION INVOLVING THE FAILURE OF THE TURBINE DRIVEN AUXILIARY FEEDWATER PUMP TO START AT | ||
: THE CALLAWAY PLANT | : THE CALLAWAY PLANT | ||
: In response to the identification that the turbine driven auxiliary feedwater pump did not start and may not have been able to perform its function potentially due to the motor operated trip throttle valve stem being improperly lubricated between May 4 and May 25, 2009, a special inspection will be performed. | : In response to the identification that the turbine driven auxiliary feedwater pump did not start and may not have been able to perform its function potentially due to the motor operated trip throttle valve stem being improperly lubricated between May 4 and May 25, 2009, a special inspection will be performed. | ||
: You are hereby designated as the special inspection team leader. | : You are hereby designated as the special inspection team leader. | ||
| Line 415: | Line 526: | ||
: The licensee took corrective action to fix (lubricate) the valve that day which took approximately 12 hours to accomplish. | : The licensee took corrective action to fix (lubricate) the valve that day which took approximately 12 hours to accomplish. | ||
: The turbine driver auxiliary feedwater pump was subsequently tested and restored to an operable status. | : The turbine driver auxiliary feedwater pump was subsequently tested and restored to an operable status. | ||
: A regional Senior Reactor Analyst (SRA) preliminarily estimated the Incremental Conditional Core Damage Probability for this issue to be 7.3 x 10-6, which falls in the overlap region between baseline and special inspection. | : A regional Senior Reactor Analyst (SRA) preliminarily estimated the Incremental Conditional Core Damage Probability for this issue to be 7.3 x 10 | ||
-6, which falls in the overlap region between baseline and special inspection. | |||
: Although the risk for this event is in the overlap region, a special inspection will be performed since there may be generic issues with lubricating motor-operated valves. UNITED STATESNUCLEAR REGULATORY COMMISSIONREGION IV612 EAST LAMAR BLVD, SUITE 400ARLINGTON, TEXAS 76011-4125 | : Although the risk for this event is in the overlap region, a special inspection will be performed since there may be generic issues with lubricating motor-operated valves. UNITED STATESNUCLEAR REGULATORY COMMISSIONREGION IV612 EAST LAMAR BLVD, SUITE 400ARLINGTON, TEXAS 76011-4125 | ||
: Attachment 2 B. Scope | : Attachment 2 B. Scope | ||
| Line 430: | Line 542: | ||
: CFR 50.73. | : CFR 50.73. | ||
: Also determine if the licensee has plans to issue a Licensee Event Report to document this issue. 18. Review the licensee's compliance with the Technical Specifications. | : Also determine if the licensee has plans to issue a Licensee Event Report to document this issue. 18. Review the licensee's compliance with the Technical Specifications. | ||
: Attachment 2 C. Guidance | : Attachment 2 C. Guidance Inspection Procedure 93812, "Special Inspection," will be used during this inspection. | ||
: The inspection should emphasize fact-finding in its review of the circumstance surrounding this event. | : The inspection should emphasize fact-finding in its review of the circumstance surrounding this event. | ||
: It is not the responsibility of the team to examine the regulatory process. | : It is not the responsibility of the team to examine the regulatory process. | ||
| Line 442: | Line 553: | ||
: Should you require support for the final determination of the risk significance of any issue, contact David Loveless at (817) 860-8161. | : Should you require support for the final determination of the risk significance of any issue, contact David Loveless at (817) 860-8161. | ||
: Should you have any questions concerning this guidance, contact me at (817) 860-8141. | : Should you have any questions concerning this guidance, contact me at (817) 860-8141. | ||
: Attachment 3 Timeline associated with TDAFW Pump Trip throttle Valve FCHV0312 | : Attachment 3 Timeline associated with TDAFW Pump Trip throttle Valve FCHV0312 | ||
: Date ActivityJuly 8, 1986 Turbine-driven auxiliary feedwater pump trip throttle valve fuses are discovered blown due to lack or improper valve lubrication. May 1, 1987 | : Date ActivityJuly 8, 1986 Turbine-driven auxiliary feedwater pump trip throttle valve fuses are discovered blown due to lack or improper valve lubrication. May 1, 1987 | ||
: RFR 3715A approves 18-month frequency for Preventive Maintenance 0818214. | : RFR 3715A approves 18-month frequency for Preventive Maintenance 0818214. | ||
| Line 530: | Line 641: | ||
: 10:47 p.m. Turbine-driven auxiliary feedwater pump started for slave relay test. May 25, 2009 | : 10:47 p.m. Turbine-driven auxiliary feedwater pump started for slave relay test. May 25, 2009 | ||
: 10:56 p.m. Turbine-driven auxiliary feedwater pump declared operable. June 17, 2009 NRC Region IV approves a special inspection charter to understand the circumstances surrounding the failure of the turbine-driven auxiliary feedwater pump. June 22-26, 2009 NRC Special Inspection Team at Callaway Plant. | : 10:56 p.m. Turbine-driven auxiliary feedwater pump declared operable. June 17, 2009 NRC Region IV approves a special inspection charter to understand the circumstances surrounding the failure of the turbine-driven auxiliary feedwater pump. June 22-26, 2009 NRC Special Inspection Team at Callaway Plant. | ||
: Attachment 4 PHASE 3 ANALYSIS FAILURE OF AUXILIARY FEEDWATER PUMP TRIP AND THROTTLE VALVE | : Attachment 4 PHASE 3 ANALYSIS FAILURE OF AUXILIARY FEEDWATER PUMP TRIP AND THROTTLE VALVE | ||
: Summary of Significance Determination | : Summary of Significance Determination The senior reactor analyst completed a Phase 3 analysis using the plant-specific Standardized Plant Analysis Risk (SPAR) Model for Callaway, Revision 3.50, including the external events initiators. | ||
: The exposure period of 11 days represented half the time from the last successful run of the pump plus the repair time. | : The exposure period of 11 days represented half the time from the last successful run of the pump plus the repair time. | ||
: The analyst estimated the nonrecovery for the failed valve at 21 percent. | : The analyst estimated the nonrecovery for the failed valve at 21 percent. | ||
: The final result was calculated to be 1.8 x 10-6 indicating that the finding was of low to moderate risk significance (White). | : The final result was calculated to be 1.8 x 10-6 indicating that the finding was of low to moderate risk significance (White). | ||
: Details | : Details | ||
: A. Summary of Issue On May 25, 2009, Callaway Plant operators performed surveillance testing Procedure | : A. Summary of Issue On May 25, 2009, Callaway Plant operators performed surveillance testing Procedure | ||
| Line 557: | Line 667: | ||
: APA-ZZ-00320, "Work Execution," Section 4.12, allows for a job to be canceled if it is determined that the work is not necessary or has been completed by another job. | : APA-ZZ-00320, "Work Execution," Section 4.12, allows for a job to be canceled if it is determined that the work is not necessary or has been completed by another job. | ||
: The maintenance supervisor initiated a job routing request to engineering inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced. | : The maintenance supervisor initiated a job routing request to engineering inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced. | ||
: Although not specifically referenced, the portion of the job requested to be closed was Section 6.3, "Trip Throttle Valve Lubrication and Inspection." Callaway engineering replied to the job routing request that the valve had been diagnostically tested satisfactorily and that no additional inspection was needed. | : Although not specifically referenced, the portion of the job requested to be closed was Section 6.3, "Trip Throttle Valve Lubrication and Inspection." | ||
: Callaway engineering replied to the job routing request that the valve had been diagnostically tested satisfactorily and that no additional inspection was needed. | |||
: The engineer's response only answered the specific question asked by the job routing Attachment 4 request and did not examine the procedural requirements that were the subject of the request. | : The engineer's response only answered the specific question asked by the job routing Attachment 4 request and did not examine the procedural requirements that were the subject of the request. | ||
: The engineer did not consult with the technician requesting closure and did not verify that the lubrication job task being closed was completed by the installation procedure. | : The engineer did not consult with the technician requesting closure and did not verify that the lubrication job task being closed was completed by the installation procedure. | ||
| Line 574: | Line 685: | ||
: Specifically, the licensee closed without adequate review, Section 6.3 "Trip Throttle Valve Lubrication and Inspection," of Procedure | : Specifically, the licensee closed without adequate review, Section 6.3 "Trip Throttle Valve Lubrication and Inspection," of Procedure | ||
: MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection." The lack of lubrication resulted in increased friction within the valve operating stem which caused the valve not to open on May 25, 2009. | : MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection." The lack of lubrication resulted in increased friction within the valve operating stem which caused the valve not to open on May 25, 2009. | ||
: C. Significance Determination Basis 1. Phase 1 Screening Logic, Results and Assumptions In accordance with NRC Inspection Manual Chapter 0612, Appendix B, "Issue Screening," the analyst determined that the failure to lubricate valve FCHV0312 at appropriate intervals was a licensee performance deficiency. | : C. Significance Determination Basis | ||
: 1. Phase 1 Screening Logic, Results and Assumptions In accordance with NRC Inspection Manual Chapter 0612, Appendix B, "Issue Screening," the analyst determined that the failure to lubricate valve FCHV0312 at appropriate intervals was a licensee performance deficiency. | |||
: The issue was more than minor because it was associated with the mitigating systems cornerstone attribute of equipment performance and affected the cornerstone objective of ensuring the availability and reliability of systems that respond to initiating events to prevent undesirable consequences. | : The issue was more than minor because it was associated with the mitigating systems cornerstone attribute of equipment performance and affected the cornerstone objective of ensuring the availability and reliability of systems that respond to initiating events to prevent undesirable consequences. | ||
: The analyst evaluated the issue using the Significance Determination Process (SDP) Phase 1 Screening Worksheet for the Initiating Events, Mitigating Systems, and Barriers Cornerstones provided in Manual Chapter 0609, | : The analyst evaluated the issue using the Significance Determination Process (SDP) Phase 1 Screening Worksheet for the Initiating Events, Mitigating Systems, and Barriers Cornerstones provided in Manual Chapter 0609, | ||
| Line 580: | Line 692: | ||
: This finding affected the Mitigating Systems Cornerstone. | : This finding affected the Mitigating Systems Cornerstone. | ||
: The analyst determined that the finding represented a loss of system safety function because Attachment 4 the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. | : The analyst determined that the finding represented a loss of system safety function because Attachment 4 the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. | ||
: Therefore, a Phase 2 estimation was conducted in accordance with Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations." | : Therefore, a Phase 2 estimation was conducted in accordance with Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations." | ||
: 2. Phase 2 Risk Estimation In accordance with Manual Chapter 0609, Appendix A, Attachment 1, "User Guidance for Phase 2 and Phase 3 Reactor Inspection Findings for At-Power Situations," the Senior Reactor Analyst evaluated the subject finding using the "Risk-Informed Inspection Notebook for Callaway Nuclear Generating Station, Unit 1," Revision 2.1a. | : 2. Phase 2 Risk Estimation In accordance with Manual Chapter 0609, Appendix A, Attachment 1, "User Guidance for Phase 2 and Phase 3 Reactor Inspection Findings for At-Power Situations," the Senior Reactor Analyst evaluated the subject finding using the "Risk-Informed Inspection Notebook for Callaway Nuclear Generating Station, Unit 1," Revision 2.1a. | ||
: The following assumptions were made: | : The following assumptions were made: | ||
a. The identified performance deficiency occurred in October 2008 when the licensee failed to perform appropriate preventive maintenance on valve FCHV0312. | |||
: However, the valve continued to be stroked about twice monthly and pass surveillance through May 4, 2009. | : However, the valve continued to be stroked about twice monthly and pass surveillance through May 4, 2009. | ||
: Therefore, the deficiency only began to affect plant risk at some point between May 4 and the time of failure on May 25, 2009. b. The failure was identified during a test of the turbine-driven auxiliary feedwater pump conducted on May 25, 2009. | : Therefore, the deficiency only began to affect plant risk at some point between May 4 and the time of failure on May 25, 2009. | ||
b. The failure was identified during a test of the turbine-driven auxiliary feedwater pump conducted on May 25, 2009. | |||
: The last successful test that opened valve FCHV0312 was completed on May 4, 2009. | : The last successful test that opened valve FCHV0312 was completed on May 4, 2009. | ||
: Valve FCHV0312 was repaired and returned to service approximately hours later on May 25, 2009. c. In accordance with Manual Chapter 0609, Appendix A, Attachment 2, "Site Specific Risk-Informed Inspection Notebook Usage Rules," Rule 1.1, "Exposure Time," the analyst evaluated the time frame over which the finding impacted the risk of plant operations. | : Valve FCHV0312 was repaired and returned to service approximately hours later on May 25, 2009. | ||
c. In accordance with Manual Chapter 0609, Appendix A, Attachment 2, "Site Specific Risk-Informed Inspection Notebook Usage Rules," Rule 1.1, "Exposure Time," the analyst evaluated the time frame over which the finding impacted the risk of plant operations. | |||
: The analyst determined that the performance deficiency affected plant risk for 11 days. | : The analyst determined that the performance deficiency affected plant risk for 11 days. | ||
: Therefore, the exposure time used to represent the time that the performance deficiency affected plant risk in the Phase 2 estimation was greater than 3 but less than 30 days. d. In accordance with Appendix A, Attachment 1, Step 2.1.3, "Find the Appropriate Target for the Inspection Finding in the Pre-solved Table," the analyst determined that the appropriate target for evaluating this performance deficiency was "TDAFP Fails to Start." | : Therefore, the exposure time used to represent the time that the performance deficiency affected plant risk in the Phase 2 estimation was greater than 3 but less than 30 days. | ||
: Therefore, the analyst utilized the pre-solved table associated with the SDP notebook to perform the estimation. e. The analyst gave no operator action credit as discussed in Manual Chapter 0609, Appendix A, Attachment 1, Table 4, "Remaining Mitigation Capability Credit." | d. In accordance with Appendix A, Attachment 1, Step 2.1.3, "Find the Appropriate Target for the Inspection Finding in the Pre-solved Table," the analyst determined that the appropriate target for evaluating this performance deficiency was "TDAFP Fails to Start." | ||
: Therefore, the analyst utilized the pre-solved table associated with the SDP notebook to perform the estimation. | |||
e. The analyst gave no operator action credit as discussed in Manual Chapter 0609, Appendix A, Attachment 1, Table 4, "Remaining Mitigation Capability Credit." | |||
: The requirements to have procedures in place and to have trained the operators in recovery under similar conditions for such credit were not met. | : The requirements to have procedures in place and to have trained the operators in recovery under similar conditions for such credit were not met. | ||
: The dominant sequences from the notebook were documented in Table 1. | : The dominant sequences from the notebook were documented in Table 1. | ||
| Line 601: | Line 718: | ||
: LBDC-TDAFW-MDAFW-FB 7 3 | : LBDC-TDAFW-MDAFW-FB 7 3 | ||
: LBDC-TDAFW-MDAFW-EIHP 7 4 | : LBDC-TDAFW-MDAFW-EIHP 7 4 | ||
: LBDC-TDAFW-TESW 5 | : LBDC-TDAFW-TESW 5 | ||
: Using the pre-solved worksheet, the result from this estimation indicated that the finding was of moderate safety significance (YELLOW). | : Using the pre-solved worksheet, the result from this estimation indicated that the finding was of moderate safety significance (YELLOW). | ||
: However, the analyst determined that this estimate did not include a full coverage of the risk related to the failure identified, particularly because of the shorter exposure time, the affects of potential recovery, and the affect that certain fire initiators would have on the specific condition. | : However, the analyst determined that this estimate did not include a full coverage of the risk related to the failure identified, particularly because of the shorter exposure time, the affects of potential recovery, and the affect that certain fire initiators would have on the specific condition. | ||
: Therefore, a Phase 3 evaluation was conducted to better assess the risk of the finding related to internal initiators and fully assess the risk related to external initiators. | : Therefore, a Phase 3 evaluation was conducted to better assess the risk of the finding related to internal initiators and fully assess the risk related to external initiators. | ||
: 3. Phase 3 Analysis | : 3. Phase 3 Analysis Assumptions The following assumptions were made to support this Phase 3 analysis: | ||
: 1. The Callaway plant-specific SPAR, Revision 3.50, including the external event initiators, was the best tool for quantifying the risk of the subject performance deficiency. | : 1. The Callaway plant-specific SPAR, Revision 3.50, including the external event initiators, was the best tool for quantifying the risk of the subject performance deficiency. | ||
: 2. Trip and throttle valve FCHV0312 failed to latch upon demand on May 25, 2009, preventing the start of turbine-driven auxiliary feedwater pump PAL02. | : 2. Trip and throttle valve FCHV0312 failed to latch upon demand on May 25, 2009, preventing the start of turbine-driven auxiliary feedwater pump PAL02. | ||
| Line 632: | Line 747: | ||
: The pump failed to start during testing on May 25, 2009, at 11:41 a.m. | : The pump failed to start during testing on May 25, 2009, at 11:41 a.m. | ||
: Trip and throttle valve FCHV0312 was repaired and the pump returned to service on May 25, 2009 at 10:56 p.m. | : Trip and throttle valve FCHV0312 was repaired and the pump returned to service on May 25, 2009 at 10:56 p.m. | ||
: Attachment 4 | : Attachment 4 | ||
: The team evaluated the failure mode of the valve and determined that it was unknown exactly when the valve was no longer in a condition to perform its risk significant function. | : The team evaluated the failure mode of the valve and determined that it was unknown exactly when the valve was no longer in a condition to perform its risk significant function. | ||
: In accordance with the Risk Assessment of Operational Events Handbook, Section 2.3, the exposure time for a component that fails from a degradation mechanism that gradually affects the component should be one- | : In accordance with the Risk Assessment of Operational Events Handbook, Section 2.3, the exposure time for a component that fails from a degradation mechanism that gradually affects the component should be one- | ||
| Line 652: | Line 767: | ||
: Therefore, diagnosis credit was adjusted for "Extra Time." | : Therefore, diagnosis credit was adjusted for "Extra Time." | ||
: However, following proper diagnosis, there would not be more than five times nominal for manipulation of the valve. | : However, following proper diagnosis, there would not be more than five times nominal for manipulation of the valve. | ||
: Therefore, the analyst assumed "Nominal time" for completing the action. * Stress: A station blackout scenario with a loss of all feedwater places the operators in at least a high level of stress during both diagnosis and action. | : Therefore, the analyst assumed "Nominal time" for completing the action. | ||
* Stress: A station blackout scenario with a loss of all feedwater places the operators in at least a high level of stress during both diagnosis and action. | |||
: Multiple competing priorities, sudden onset of stress, and the knowledge that the consequences of this task represents a threat to plant safety clearly places the operators under a high level of stress. | : Multiple competing priorities, sudden onset of stress, and the knowledge that the consequences of this task represents a threat to plant safety clearly places the operators under a high level of stress. | ||
: Because the stress would not persist for long periods of time nor place the Attachment 4 operators under a threat to their physical well being, the analyst determined that the stress would not be at the extreme level. * Complexity: The analyst determined that the diagnosis and recovery of this specific failure were moderately complex. | : Because the stress would not persist for long periods of time nor place the Attachment 4 operators under a threat to their physical well being, the analyst determined that the stress would not be at the extreme level. | ||
* Complexity | |||
: The analyst determined that the diagnosis and recovery of this specific failure were moderately complex. | |||
: While it would be clear that the valve was not open, the cause of the failure would not be clear. | : While it would be clear that the valve was not open, the cause of the failure would not be clear. | ||
: The valve is unusual, in that it must be manipulated in the closed direction to latch prior to opening. | : The valve is unusual, in that it must be manipulated in the closed direction to latch prior to opening. | ||
: The opening of the valve with dried out lubricant would be physically straining and difficult. | : The opening of the valve with dried out lubricant would be physically straining and difficult. | ||
: A higher level of complexity might have been considered, however, this complexity was affected mostly by the environment and misleading nature of the situation. | : A higher level of complexity might have been considered, however, this complexity was affected mostly by the environment and misleading nature of the situation. | ||
: Therefore, the analyst dealt with these issues under the ergonomics shaping factor. * Procedures: | : Therefore, the analyst dealt with these issues under the ergonomics shaping factor. | ||
* Procedures | |||
: | |||
: The analyst determined that the emergency operating procedures would properly focus on the turbine-driven auxiliary feedwater pump and the valve alignments. | : The analyst determined that the emergency operating procedures would properly focus on the turbine-driven auxiliary feedwater pump and the valve alignments. | ||
: Under normal conditions, these procedures, combined with operator skills and knowledge, would be sufficient to recover the pump. | : Under normal conditions, these procedures, combined with operator skills and knowledge, would be sufficient to recover the pump. | ||
: The complications in performing the subject recovery would not be expected to be addressed in procedures. | : The complications in performing the subject recovery would not be expected to be addressed in procedures. | ||
: Therefore, the analyst determined that the procedures were of nominal level for both diagnosis and action. * Ergonomics: The analyst determined that the biggest impediment to successful recovery was the ergonomics of this situation. | : Therefore, the analyst determined that the procedures were of nominal level for both diagnosis and action. | ||
* Ergonomics | |||
: The analyst determined that the biggest impediment to successful recovery was the ergonomics of this situation. | |||
: Valve FCHV0312 would not be in the expected condition when an operator arrives in the pump room. | : Valve FCHV0312 would not be in the expected condition when an operator arrives in the pump room. | ||
: The analyst noted that the same performance deficiency that would have caused the valve to fail to open automatically would also impair its recovery. | : The analyst noted that the same performance deficiency that would have caused the valve to fail to open automatically would also impair its recovery. | ||
| Line 682: | Line 804: | ||
: Table 2 provides the calculations used to apply the performance shaping factors and the odds ratio. | : Table 2 provides the calculations used to apply the performance shaping factors and the odds ratio. | ||
: The resulting HRA non-recovery value was 21 percent. | : The resulting HRA non-recovery value was 21 percent. | ||
: TABLE 2Manually Open Trip and Throttle Valve 312 Performance Shaping Factor Diagnosis Action | : TABLE 2Manually Open Trip and Throttle Valve 312 Performance Shaping Factor Diagnosis Action | ||
: PSF Level Multiplier PSF Level Multiplier | : PSF Level Multiplier | ||
: PSF Level Multiplier | |||
: Time: Extra Time 0.10Nominal 1.0Stress: High 2.0High 2.0Complexity: Moderately Complex 2.0Moderate 2.0Experience: Nominal 1.0Nominal 1.0Procedures: Nominal 1.0Nominal 1.0Ergonomics: Misleading 50.0Poor 10.0Fitness for Duty: Nominal 1.0Nominal 1.0Work Processes: Nominal 1.0Nominal 1.0 | : Time: Extra Time 0.10Nominal 1.0Stress: High 2.0High 2.0Complexity: Moderately Complex 2.0Moderate 2.0Experience: Nominal 1.0Nominal 1.0Procedures: Nominal 1.0Nominal 1.0Ergonomics: Misleading 50.0Poor 10.0Fitness for Duty: Nominal 1.0Nominal 1.0Work Processes: Nominal 1.0Nominal 1.0 | ||
: Nominal 1.0E-02 1.0E-03 Adjusted 2.0E-01 0.0 Odds Ratio 1.7E-01 3.8E-02 | : Nominal 1.0E-02 1.0E-03 Adjusted 2.0E-01 0.0 Odds Ratio 1.7E-01 3.8E-02 | ||
: Composite 20 40 | : Composite 20 40 | ||
: Failure to Manually Open Valve 312 and Start TDAFW Probability: 2.1E-01 The licensee used the EPRI HRA calculator with the HCR/ORE (cognitive) and THERP (execution) modules to evaluate the failure to recover the pump. | : Failure to Manually Open Valve 312 and Start TDAFW Probability: | ||
: 2.1E-01 The licensee used the EPRI HRA calculator with the HCR/ORE (cognitive) and THERP (execution) modules to evaluate the failure to recover the pump. | |||
: Their nonrecovery probability was 1.6E-2. | : Their nonrecovery probability was 1.6E-2. | ||
: For the reader's edification, the nominal failure rate for an action requiring diagnosis is considered to be 1.1E-2 per demand. | : For the reader's edification, the nominal failure rate for an action requiring diagnosis is considered to be 1.1E-2 per demand. | ||
| Line 694: | Line 818: | ||
: Attachment 4 Alternative Recovery Evaluation The analyst determined that the | : Attachment 4 Alternative Recovery Evaluation The analyst determined that the | ||
: SPAR-H method may not be the best tool for evaluating the nonrecovery from the failure of valve FCHV0312 because it does not readily assess the impact of the performance deficiency on the recovery actions. | : SPAR-H method may not be the best tool for evaluating the nonrecovery from the failure of valve FCHV0312 because it does not readily assess the impact of the performance deficiency on the recovery actions. | ||
: Therefore, the analyst developed the following event tree to better describe the recovery actions: | : Therefore, the analyst developed the following event tree to better describe the recovery actions: | ||
: ACTIONOperator Fails toOpen ValveFORCEDControl Room Failsto Direct ForcedOpeningVALVE-FAILInitial Operator Failsto OpenDISPATCHControl Room Failsto Diagnose andDispatch OperatorIE-TDAFW# | : ACTIONOperator Fails toOpen ValveFORCEDControl Room Failsto Direct ForcedOpeningVALVE-FAILInitial Operator Failsto OpenDISPATCHControl Room Failsto Diagnose andDispatch OperatorIE-TDAFW# | ||
: END-STATE-NAMES1 | : END-STATE-NAMES1 | ||
| Line 706: | Line 830: | ||
: FCHV-0312 Fails to Open on Demand2009/08/13Page 1 | : FCHV-0312 Fails to Open on Demand2009/08/13Page 1 | ||
: The initiator for this event tree is the demand for a start of the turbine-driven auxiliary feedwater pump with the subsequent failure of Valve FCHV0312 to latch. | : The initiator for this event tree is the demand for a start of the turbine-driven auxiliary feedwater pump with the subsequent failure of Valve FCHV0312 to latch. | ||
: Given this initiator, the top events are described as follows: | : Given this initiator, the top events are described as follows: | ||
: Dispatch: | : Dispatch: | ||
: This top represents the initial diagnosis and actions by the main control room operators in: following their emergency operating procedures; deciding local evaluation of the valve condition is required; and sending an operator out to the pump room. | : This top represents the initial diagnosis and actions by the main control room operators in: following their emergency operating procedures; deciding local evaluation of the valve condition is required; and sending an operator out to the pump room. | ||
| Line 772: | Line 896: | ||
: TABLE 4Phase 3 Results | : TABLE 4Phase 3 Results | ||
: SPAR Licensee PRA | : SPAR Licensee PRA | ||
: SPAR-EE Baseline 5.65E-5 4.22E-5 2.80E-3 Case 1.95E-4 2.06E-4 2.86E-3 Delta 1.38E-4 1.64E-4 6.10E-5* 11-Day Exposure 4.17E-6 4.93E-6 1.84E-6* Including External 9.24E-6 1.09E-5 | : SPAR-EE Baseline 5.65E-5 4.22E-5 2.80E-3 Case 1.95E-4 2.06E-4 2.86E-3 Delta 1.38E-4 1.64E-4 6.10E-5 | ||
: 21% Non-recovery 1.93E-6 2.27E-6 1.84E-6** 1.6% Non-recovery 1.76E-7 1.92E-7 1.40E-7 * | * 11-Day Exposure 4.17E-6 4.93E-6 1.84E-6 | ||
: SPAR-EE run included NRCs nonrecovery of 0.21 ** SDP Result. | * Including External 9.24E-6 1.09E-5 | ||
: 21% Non-recovery 1.93E-6 2.27E-6 1.84E-6** 1.6% Non-recovery 1.76E-7 1.92E-7 1.40E-7 | |||
* | |||
: SPAR-EE run included NRCs nonrecovery of 0.21 | |||
** SDP Result. | |||
: Analyst's best estimate. | : Analyst's best estimate. | ||
: Table 5 documents the major sequences contributing 97 percent of the change in core damage frequency. | : Table 5 documents the major sequences contributing 97 percent of the change in core damage frequency. | ||
| Line 785: | Line 913: | ||
: 2.03E-6 3.4 LOCDNK4 8 Loss of dc Bus NK04, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.98E-6 3.3 LOCDNK1 8 Loss of dc Bus NK01, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.98E-6 3.3 | : 2.03E-6 3.4 LOCDNK4 8 Loss of dc Bus NK04, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.98E-6 3.3 LOCDNK1 8 Loss of dc Bus NK01, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.98E-6 3.3 | ||
: Attachment 4 LOOP 15 Loss of Offsite Power, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.09E-6 1.8 | : Attachment 4 LOOP 15 Loss of Offsite Power, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.09E-6 1.8 | ||
: FRI-A1A 2-12 Fire in the Auxiliary Building, Loss of all Feedwater, Failure of Feed and Bleed. 6.05E-7 1.0 LOOP 16-43-10 Station Blackout, loss of TDAFW, and Reactor Coolant Pump Seal LOCA. 6.04E-7 1.0 | : FRI-A1A 2-12 Fire in the Auxiliary Building, Loss of all Feedwater, Failure of Feed and Bleed. 6.05E-7 1.0 LOOP 16-43-10 Station Blackout, loss of TDAFW, and Reactor Coolant Pump Seal LOCA. 6.04E-7 1.0 | ||
: The analyst noted that all three methods provided very similar results. | : The analyst noted that all three methods provided very similar results. | ||
: This indicated that the primary difference between the NRC and the licensee results was the recovery assumptions. | : This indicated that the primary difference between the NRC and the licensee results was the recovery assumptions. | ||
| Line 792: | Line 920: | ||
: Large Early Release In accordance with the guidance in Inspection Manual Chapter 0609, Appendix H, this finding would not involve an increased risk of a large early release of radiation because Callaway has a large, dry containment and the sequences contributing to a change in the core damage frequency did not involve either a steam generator tube rupture or an inter-system loss of coolant accident. | : Large Early Release In accordance with the guidance in Inspection Manual Chapter 0609, Appendix H, this finding would not involve an increased risk of a large early release of radiation because Callaway has a large, dry containment and the sequences contributing to a change in the core damage frequency did not involve either a steam generator tube rupture or an inter-system loss of coolant accident. | ||
: Licensee's Evaluation The licensee evaluated the risk associated with the failure to start of turbine-driven auxiliary feedwater pump PAL02 using their site-specific probabilistic safety assessment model, as documented in PRAER 09-333, Revision 0. | : Licensee's Evaluation The licensee evaluated the risk associated with the failure to start of turbine-driven auxiliary feedwater pump PAL02 using their site-specific probabilistic safety assessment model, as documented in PRAER 09-333, Revision 0. | ||
: The licensee's final value for the increased risk (ICCDP) was 1.34 x 10-7. | : The licensee's final value for the increased risk (ICCDP) was 1.34 x 10 | ||
-7. | |||
: The analyst reviewed the PRAER and determined that the assumptions and/or models used provided very similar results with one exception. | : The analyst reviewed the PRAER and determined that the assumptions and/or models used provided very similar results with one exception. | ||
: The licensee's model and the SPAR models provided very similar results when evaluating risk related to the failure of turbine-driven auxiliary feedwater pump PAL02. | : The licensee's model and the SPAR models provided very similar results when evaluating risk related to the failure of turbine-driven auxiliary feedwater pump PAL02. | ||
| Line 799: | Line 928: | ||
: The licensee used the EPRI HRA calculator to evaluate the human error probability and produced a value of 1.6E-2. | : The licensee used the EPRI HRA calculator to evaluate the human error probability and produced a value of 1.6E-2. | ||
: This nonrecovery is about nominal for an action that requires diagnosis under rather ideal conditions. | : This nonrecovery is about nominal for an action that requires diagnosis under rather ideal conditions. | ||
: The analyst determined that the following conditions should have affected the licensee's results, but did not appear to be considered: | : The analyst determined that the following conditions should have affected the licensee's results, but did not appear to be considered: | ||
: 1. The Technical Support Center would not have been available prior to core damage during most conditions. | : 1. The Technical Support Center would not have been available prior to core damage during most conditions. | ||
: The licensee noted that the center could Attachment 4 be manned within 30 minutes during the working day. | : The licensee noted that the center could Attachment 4 be manned within 30 minutes during the working day. | ||
: However, in the analyst's judgment, they could not have effectively made an independent evaluation and response within the remaining time available. | : However, in the analyst's judgment, they could not have effectively made an independent evaluation and response within the remaining time available. | ||
: 2. There would be significant competing priorities - Recovery of two failed diesels | : 2. There would be significant competing priorities - Recovery of two failed diesels | ||
- Offsite power recovery - Recovery of the TDAFW pump | - Offsite power recovery - Recovery of the TDAFW pump | ||
: 3. The environmental conditions were less than ideal - No emergency lighting in the pump room | : 3. The environmental conditions were less than ideal - No emergency lighting in the pump room | ||
- Hot, humid pump room - Crawling required to reach valve in some instances | - Hot, humid pump room - Crawling required to reach valve in some instances | ||
: 4. The ergonomic conditions were significantly degraded - Standing on governor valve insulation - Footing in the vicinity of the mechanical trip mechanism - Off even standing for several potential vantages | : 4. The ergonomic conditions were significantly degraded - Standing on governor valve insulation - Footing in the vicinity of the mechanical trip mechanism - Off even standing for several potential vantages | ||
- Reaching for valve handle - Operator can't see valve stem while manipulating | - Reaching for valve handle - Operator can't see valve stem while manipulating | ||
: 5. There were misleading indications that may have negatively impacted results - Operator may determine valve is bound - Operators may try to reset thermal overloads - Valve must be operated in the closed direction to open | : 5. There were misleading indications that may have negatively impacted results - Operator may determine valve is bound - Operators may try to reset thermal overloads - Valve must be operated in the closed direction to open The biggest concern the analyst expressed was that the operator might believe the valve was mechanically bound and incapable of being manipulated manually. | ||
: Under this case, he may have been redirected by licensed operators to attempt recovery of the emergency diesel generators or offsite power instead of focusing on the turbine-driven auxiliary feedwater pump. | : Under this case, he may have been redirected by licensed operators to attempt recovery of the emergency diesel generators or offsite power instead of focusing on the turbine-driven auxiliary feedwater pump. | ||
: The licensee argued that recovery of secondary cooling would be the highest priority during these scenarios. | : The licensee argued that recovery of secondary cooling would be the highest priority during these scenarios. | ||
| Line 827: | Line 955: | ||
: Action FCHV0312 3.80E-02 | : Action FCHV0312 3.80E-02 | ||
: 4.00E-03 | : 4.00E-03 | ||
: 5.00E-02 | : 5.00E-02 | ||
: 1.81E-01 4.59E-02 1.05E-01 1.80E-01 1.62E-01 2.03E-01 4.09E-01 1.81E-01 1.87E-01 | : 1.81E-01 4.59E-02 1.05E-01 1.80E-01 1.62E-01 2.03E-01 4.09E-01 1.81E-01 1.87E-01 | ||
: Table A2Multiple Parameter Sensitivities Increase Sticking Probability Decrease Sticking Probability Low Sticking Probability Never | : Table A2Multiple Parameter Sensitivities Increase Sticking Probability Decrease Sticking Probability Low Sticking Probability Never | ||
: Fails | : Fails | ||
| Line 834: | Line 962: | ||
: 0.1 0.2 0.3 0.40.1 0.2 0.3 0.4 0.1 0.2 0.3 0.4 | : 0.1 0.2 0.3 0.40.1 0.2 0.3 0.4 0.1 0.2 0.3 0.4 | ||
: FALSE | : FALSE | ||
: 1.18E-01 2.03E-01 2.88E-01 3.73E-016.63E-02 1.14E-01 1.61E-01 2.09E-01 2.68E-02 4.59E-02 6.51E-02 8.43E-021.60E-01 | : 1.18E-01 2.03E-01 2.88E-01 3.73E-016.63E-02 1.14E-01 1.61E-01 2.09E-01 2.68E-02 4.59E-02 6.51E-02 8.43E-021.60E-01 | ||
: Appendix-A4-2 Appendix to Attachment 4 | : Appendix-A4-2 Appendix to Attachment 4 | ||
: Table A3Sensitivity Evaluation Adjustments to Dominant Sequences for Timing Initiator Sequence Number CDFContributionSequence Initiator Functional Losses AdjustmentsModifiedCDF LOOP 16-45 1.95E-0532.3%SBO | : Table A3Sensitivity Evaluation Adjustments to Dominant Sequences for Timing Initiator Sequence Number CDFContributionSequence Initiator Functional Losses AdjustmentsModifiedCDF LOOP 16-45 1.95E-0532.3%SBO | ||
: No 1.95E-05 | : No 1.95E-05 | ||
| Line 852: | Line 980: | ||
: No 2.23E-07LOOP 16-43-10 1.51E-070.2%SBO | : No 2.23E-07LOOP 16-43-10 1.51E-070.2%SBO | ||
: No 1.51E-07EQ3 2-15 1.20E-070.2%LOOP | : No 1.51E-07EQ3 2-15 1.20E-070.2%LOOP | ||
: AFW-FB No 1.20E-07 SUM | : AFW-FB No 1.20E-07 | ||
: SUM | |||
: 5.98E-05 | : 5.98E-05 | ||
: 5.05E-05 TOTAL 6.04E-05 Percent 99.0%Total Delta CDF: 5.1E-05/year Appendix-A4-3 Appendix to Attachment 4 | : 5.05E-05 TOTAL 6.04E-05 Percent 99.0%Total Delta CDF: 5.1E-05/year | ||
: Appendix-A4-3 Appendix to Attachment 4 | |||
: Table A4 Recovery Time Estimates Activity Time Units Licensed Operators Following Emergency Procedures: 10minutes Licensed Operators Contact/Recall Plant Operator: 5minutes Licensed Operators Brief Plant Operator: 3minutes Plant Operator Travels to the Pump Room: 8minutes Plant Operator Assesses Condition: 5minutes Plant Operator Contacts Licensed Operators for Instructions: 4minutes Plant Operator Returns to Valve: 2minutes Plant Operator Makes Determination Regarding Valve Functionality: 1minute | : Table A4 Recovery Time Estimates Activity Time Units Licensed Operators Following Emergency Procedures: 10minutes Licensed Operators Contact/Recall Plant Operator: 5minutes Licensed Operators Brief Plant Operator: 3minutes Plant Operator Travels to the Pump Room: 8minutes Plant Operator Assesses Condition: 5minutes Plant Operator Contacts Licensed Operators for Instructions: 4minutes Plant Operator Returns to Valve: 2minutes Plant Operator Makes Determination Regarding Valve Functionality: 1minute | ||
}} | }} | ||
Revision as of 07:08, 25 August 2018
| ML092730656 | |
| Person / Time | |
|---|---|
| Site: | Callaway  |
| Issue date: | 09/30/2009 |
| From: | Chamberlain D D NRC/RGN-IV/DRP |
| To: | Heflin A C AmerenUE |
| References | |
| EA-09-200 IR-09-009 | |
| Download: ML092730656 (61) | |
Text
September 30, 2009
Mr. Adam C. Heflin, Senior Vice President and Chief Nuclear Officer AmerenUE P.O. Box 620 Fulton, MO 65251 Subject: NRC SPECIAL INSPECTION REPORT 05000483/2009009 - CALLAWAY PLANT; PRELIMINARY WHITE FINDING
Dear Mr. Heflin:
On September 2, 2009, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Callaway Plant to evaluate the facts and circumstances surrounding the failure to start of the turbine-driven auxiliary feedwater pump due to an inadequately lubricated trip throttle valve. Based upon the risk and deterministic criteria specified in NRC Management Directive 8.3, "NRC Incident Investigation Program," including possible generic implications, the NRC initiated a special inspection in accordance with Inspection Procedure 93812, "Special Inspection." The basis for initiating the special inspection and the focus areas for review are detailed in the Special Inspection Charter (Attachment 2). The determination that the inspection would be conducted was made by the NRC on June 17, 2009, and the onsite inspection started on June 22, 2009. The enclosed report documents the inspection findings that were discussed on September 2, 2009, with you, and members of your staff.
The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, observed activities, and interviewed plant personnel.
The enclosed report discusses one finding that appears to have low to moderate safety significance (White). As described in Section 1R2 of this report, the NRC concluded that the failure to adequately lubricate the turbine-driven auxiliary feedwater pump trip throttle valve during Refueling Outage 16 resulted in the pump's failure to start on May 25, 2009. The safety significance of this finding was assessed on the basis of the best available information, including influential assumptions, using the applicable Significance Determination Process and was preliminarily determined to be a White (i.e., low to moderate safety significance) finding. Attachment 4 of this report provides a detailed description of the preliminary risk determination. The finding is also an apparent violation of NRC requirements and is being considered for escalated enforcement action in accordance with the Enforcement Policy, which can be found on the NRC's Web site at http://www.nrc.gov/reading-rm/doc-collections/enforcement. Union Electric Company 2 In accordance with NRC Manual Chapter 0609, "Significance Determination Process," we intend to complete our evaluation using the best available information and issue our final determination of safety significance within 90 days of the date of this letter. The significance determination process encourages an open dialogue between the NRC staff and the licensee; however, the dialogue should not impact the timeliness of the staff's final determination.
Before we make a final decision on this matter, we are providing you with an opportunity to (1) attend a Regulatory Conference where you can present to the NRC your perspective on the facts and assumptions the NRC used to arrive at the finding and assess its significance, or (2) submit your position on the finding to the NRC in writing. If you request a Regulatory Conference, it should be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least one week prior to the conference in an effort to make the conference more efficient and effective. If a Regulatory Conference is held, it will be open for public observation. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of your receipt of this letter. If you decline to request a Regulatory Conference or submit a written response, you relinquish your right to appeal the final Significance Determination Process determination, in that by not doing either, you fail to meet the appeal requirements stated in the Prerequisite and Limitation sections of Attachment 2 of Manual Chapter 0609.
Please contact Mr. Vincent Gaddy at (817) 860-8141 within 10 business days of the date of this letter to notify the NRC of your intentions. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision. The final resolution of this matter will be conveyed in separate correspondence.
Because the NRC has not made a final determination in this matter, no Notice of Violation is being issued for these inspection findings at this time. In addition, please be advised that the characterization of the apparent violation described in the enclosed inspection report may change as a result of further NRC review.
The report also documents four NRC-identified findings, which were evaluated under the significance determination process as having very low safety significance (Green). These findings were determined to involve violations of NRC requirements. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the noncited violations in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 612 E. Lamar Blvd., Suite 400, Arlington, Texas 76011-4125; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspectors at the Callaway Plant. In addition, if you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Callaway Plant. The information you provide will be considered in accordance with Inspection Manual Chapter 0305.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosures will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC
=s document Union Electric Company 3 system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html.
Sincerely,/RA/ Dwight Chamberlain, Director Division of Reactor Projects Docket: 50-483 License: NPF-30
Enclosure:
NRC Inspection Report 05000483/2009009
w/Attachments:
Attachment 1: Supplemental Information Attachment 2: Special Inspection Charter Attachment 3: Timeline Associated With Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve FCHV0312 Attachment 4: Preliminary Significance Determination Evaluation Mr. Luke H. Graessle Director, Operations Support AmerenUE P.O. Box 620 Fulton, MO 65251 E. Hope Bradley Manager, Protective Services AmerenUE P.O. Box 620 Fulton, MO 65251 Mr. Scott Sandbothe, Manager Regulatory Affairs AmerenUE P.O. Box 620 Fulton, MO 65251 R. E. Farnam Assistant Manager, Technical Training AmerenUE P.O. Box 620 Fulton, MO 65251
Union Electric Company 4 J. S. Geyer Radiation Protection Manager AmerenUE P.O. Box 620 Fulton, MO 65251
John O'Neill, Esq. Pillsbury Winthrop Shaw Pittman LLP 2300 N. Street, N.W. Washington, DC 20037
Missouri Public Service Commission P.O. Box 360 Jefferson City, MO 65102-0360 Deputy Director for Policy Department of Natural Resources P.O. Box 176 Jefferson City, MO 65102-0176 Mr. Rick A. Muench, President and Chief Executive officer Wolf Creek Nuclear Operating Corporation P.O. Box 411 Burlington, KS 66839 Kathleen Logan Smith, Executive Director and Kay Drey, Representative, Board of Directors Missouri Coalition for the Environment 6267 Delmar Boulevard, Suite 2E St. Louis, MO 63130 Mr. Lee Fritz, Presiding Commissioner Callaway County Courthouse 10 East Fifth Street Fulton, MO 65251 Director, Missouri State Emergency Management Agency P.O. Box 116 Jefferson City, MO 65102-0116 Mr. Scott Clardy, Administrator Section for Disease Control Missouri Department of Health and Senior Services P.O. Box 570 Jefferson City, MO 65102-0570
Union Electric Company 5 Certrec Corporation 4200 South Hulen, Suite 422 Fort Worth, TX 76109 Mr. Keith G. Henke, Planner II Division of Community and Public Health Office of Emergency Coordination Missouri Department of Health and Senior Services 930 Wildwood Drive P.O. Box 570 Jefferson City, MO 65102 Chief, Technological Hazards Branch FEMA Region VII 9221 Ward Parkway, Suite 300 Kansas City, MO 64114-3372 Union Electric Company 6 Electronic distribution by RIV: Regional Administrator (Elmo.Collins@nrc.gov) Deputy Regional Administrator (Chuck.Casto@nrc.gov) DRP Director (Dwight.Chamberlain@nrc.gov) DRP Deputy Director (Anton.Vegel@nrc.gov) DRS Director (Roy.Caniano@nrc.gov)
DRS Deputy Director (Troy.Pruett@nrc.gov) Senior Resident Inspector (David.Dumbacher@nrc.gov) Resident Inspector (Jeremy.Groom@nrc.gov) Branch Chief, DRP/B (Vincent.Gaddy@nrc.gov) Senior Project Engineer, DRP/B (Rick.Deese@nrc.gov)
CWY Site Secretary (Dawn.Yancey@nrc.gov) Public Affairs Officer (Victor.Dricks@nrc.gov) Team Leader, DRP/TSS (Chuck.Paulk@nrc.gov) RITS Coordinator (Marisa.Herrera@nrc.gov) Regional Counsel (Karla.Fuller@nrc.gov) Congressional Affairs Officer (Jenny.Weil@nrc.gov) OEMail Resource DRS STA (Dale.Powers@nrc.gov) OEDO RIV Coordinator (Leigh.Trocine@nrc.gov) ROPreports WilliamJones@nrc.gov Gregory.Bowman@nrc.gov Mark.Haire@nrc.gov Christi.Maier@nrc.gov Nick.Hilton@nrc.gov June.Cai@nrc.gov John.Wray@nrc.gov MaryAnn.Ashley@nrc.gov Gerald.Gulla@nrc.gov Alexander.Sapountzis@nrc.gov Robert.Summers@nrc.gov Doug.Starkey@nrc.gov
File located: R:_Reactors\CW\2009\2009009RP-JRG.doc ML092730656 SUNSI Rev Compl. Yes No ADAMS Yes No Reviewer Initials VGG Publicly Avail Yes No Sensitive Yes No Sens. Type Initials VGG RIV: RI:DRP/B RI:DRP/C SRI:DRP/B SRA ACES JGroom MChambers DDumbacher DLoveless MHaire /RA-E/VGG for /RA-E/VGG for /RA-E/VGG for /RA/ /RA/WJ for MH 9/15/09 9/22/09 9/16/09 9/21/09 9/23/09 OE NRR C:DRP/B D:DRP GBowman MCunningham VGaddy DChamberlain /RA-E/VGG for /RA-E/VGG for /RA/ /RA/ 9/23/09 9/28/09 9/28/09 9/30/09 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax 1 Enclosure U.S. NUCLEAR REGULATORY COMMISSION REGION IV Docket: 05000483 License: NPF-30 Report: 05000483/2009009 Licensee: Union Electric Company Facility: Callaway Plant Location: Junction Highway CC and Highway O Fulton, MO Dates: June 22 through August 27, 2009 Inspectors: J. Groom, Resident Inspector, Callaway Plant M. Chambers, Resident Inspector, Cooper Nuclear Station D. Dumbacher, Senior Resident Inspector, Callaway Plant D. Loveless, Senior Reactor Analyst Approved By: D. Chamberlain, Director Division of Reactor Projects 2 Enclosure
SUMMARY OF FINDINGS
IR 05000483/2009009; 06/22/09 - 08/27/09; Callaway Plant; Special inspection into turbine-driven auxiliary feedwater pump failure to start.
The report covered one week of onsite inspection and in office review through August 27, 2009.
Two resident inspectors performed the inspection with assistance from a senior resident inspector and a senior reactor analyst. One apparent violation and four green noncited violations were identified. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance Determination Process." Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.
A. NRC-Identified and Self Revealing Findings
Cornerstone: Mitigating Systems
- TBD. The team identified a self-revealing apparent violation of Technical Specification 3.7.5, "Auxiliary Feedwater System," due to the failure to adequately lubricate turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. During May 25, 2009, surveillance testing, the turbine-driven auxiliary feedwater pump did not start as expected due to hardened grease on the valve spindle of FCHV0312. The previous lubrication preventative maintenance had been missed and lack of lubrication increased friction between the sliding nut and spindle preventing FCHV0312 from opening. Following lubrication FCHV0312 and the turbine-driven auxiliary feedwater pump tested satisfactorily. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200904216.
This finding is greater than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function because the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Callaway Plant Phase 2 pre-solved table item "Turbine Driven Auxiliary Feedwater Pump Fails to Start," the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The preliminary outcome of the Phase 3 significance determination analysis, Attachment 4, determined the finding was of low to moderate safety significance.
The inspectors determined that this finding had a crosscutting aspect in the area of human performance associated with the work practices component because the licensee failed to follow the procedural guidance provided when changing the scope of a preventive maintenance task H.4(b)(Section 1R2).
- Green.
The team identified a noncited violation of Technical Specification 5.4.1.a, "Procedures," for the failure to provide adequate procedural guidance for the lubrication of auxiliary feedwater pump turbine trip throttle valve FCHV0312. The inspectors found that 2002 corrective actions to improve the lubrication procedure were not fully developed and the procedure lubrication guidance was ambiguous in that it did not specify the amount of lubricant to apply or what valve subcomponents to lubricate. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200905032.
This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not have a crosscutting aspect since the 2003 lubrication procedure revision was not reflective of current licensee performance (Section 1R3).
- Green.
The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control," for the failure to adequately evaluate the use of Mobile 28 grease for the turbine-driven auxiliary feedwater pump trip throttle valve. The licensee's 1995 evaluation included no documentation for the appropriate relubrication interval of the valve. Additionally, the inspectors identified that the valve exhibited temperatures ranging from 235°F to near 300°F compared to the 215°F valve temperature used in the evaluation. The inspectors questioned if the use of Mobile 28 grease was appropriate since operating experience suggests that Mobile 28 grease has a tendency to thicken and harden at temperatures exceeding 250°F and elevated temperatures increased the lubricant's tendency to lose oils and could result in increased stem friction. Following questioning by the inspectors, the licensee initiated Callaway Action Request 200905067 and Request for Resolution 200905651 to determine if Mobile 28 grease was an appropriate lubricant for valve FCHV0312 (Section 1R3).
This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of design control and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not to have a crosscutting aspect since the inadequate 1995 lubrication evaluation was not reflective of current licensee performance (Section 1R3).
- Green.
The team identified a noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," regarding the licensee's failure to follow the requirements of Callaway Procedure APA-ZZ-00500, "Corrective Action Program." Specifically, licensee personnel failed to initiate Callaway action requests for adverse conditions of high hand wheel forces, galled subcomponents, and hardened, gritty grease found during the 2007 rebuild of the spare turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. The licensee has entered this issue into their corrective action program as Callaway Action Request 200905053.
This finding is greater than minor because, if left uncorrected, failure to fully utilize the corrective action program could become a more significant safety concern. The inspectors determined that this finding impacted the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The cause of this finding is related to the problem identification and resolution crosscutting component of the corrective action program because licensee personnel failed to implement a corrective action program with a low threshold for identifying issues P.1(a)(Section1R3).
- Green.
The team identified a noncited violation of Technical Specification Limiting Condition for Operation 3.0.4 for entering Mode 3 with the turbine-driven auxiliary feedwater pump inoperable. Specifically, on November 3, 2008, while in Mode 4 for Refueling Outage 16, an unexpected overspeed trip of the turbine occurred during postmaintenance testing. Callaway operations staff inappropriately concluded that a water slug from the auxiliary steam line was the cause of the turbine overspeed. Following entry into Mode 3, during preparations for turbine-driven auxiliary feedwater pump testing, the licensee found the servo control valve installed during the outage was faulty. When questioned by the inspectors, the licensee determined that the faulty servo control valve discovered in Mode 3 was responsible for the overspeed of the turbine-driven auxiliary feedwater pump that occurred in Mode 4 and that the equipment was inoperable during the mode change that occurred on November 4, 2008. The licensee entered this deficiency in their corrective action program as Callaway Action Request 200905313.
This finding is greater than minor because it is associated with the Mitigating Systems Cornerstone attribute of equipment performance and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and
Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The inspectors determined that this finding has a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee failed to fully evaluate the overspeed of the turbine-driven auxiliary feedwater pump that occurred on November 3, 2008 P.1(c)(Section 4OA2).
B. Licensee-Identified Violations
None
REPORT DETAILS
REACTOR SAFETY
Cornerstones: Initiating Events, Mitigating Systems, Barrier Integrity, and Emergency Preparedness 1R1 Special Inspection Scope On May 25, 2009, during a planned slave relay test of the auxiliary feedwater actuation system, the turbine-driven auxiliary feedwater pump failed to start on demand. Subsequent troubleshooting determined that the trip throttle valve for the turbine did not actuate properly which prevented steam from being admitted to the turbine. Electrical continuity checks revealed that the torque switch on the valve operator tripped which stopped the motor-operated valve actuator. Inspection of the trip throttle valve showed that the lubricant used on the valve spindle and sliding nut was dried. In accordance with Management Directive 8.3, "NRC Incident Investigation Program," the NRC determined that a special inspection was warranted, in part, based on the potential safety significance and because of potential generic issues associated with lubricating motor-operated valves.
The inspection charter required the team to: (1) review the circumstances related to the discovery of the degraded condition, (2) assess the licensee's determination of cause and effectiveness of actions taken to resolve and prevent recurrence of these problems, and (3) assess the effectiveness of licensee programs to maintain the physical condition of the turbine-driven auxiliary feedwater pump trip throttle valve including the licensee's lubrication and valve replacement programs. The team evaluated if the licensee took appropriate actions to address these issues including extent of condition, extent of cause, and common cause questions. The inspectors reviewed the licensee's Generic Letter 89-10 program to ensure appropriate testing was being performed that would demonstrate the turbine-driven auxiliary feedwater pump trip throttle valve's ability to function under design-basis conditions.
The team conducted their reviews in accordance with NRC Inspection Procedure 93812, "Special Inspection Procedure." The special inspection team reviewed procedures, corrective action documents, as well as design and maintenance records for the equipment of concern. The team interviewed key station personnel regarding the events, reviewed the root cause analysis, and assessed the adequacy of corrective actions. The team walked down and inspected the equipment in the field and spare equipment in the warehouse. A list of specific documents reviewed is provided as Attachment 1. The charter for the special inspection is provided as Attachment 2.
1R2 Review of the Failure of Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve Background On May 25, 2009, Callaway Plant operators performed Procedure OSP-SA-0007A, "Train A AFAS Slave Relay Test." The procedure is designed to demonstrate that the auxiliary feedwater actuation system slave relays are operable and capable of starting the turbine-driven auxiliary feedwater pump. The turbine-driven auxiliary feedwater pump is a steam-driven pump which utilizes a Terry turbine as a prime mover supplied 7 Enclosure by steam from steam generators B and C. The steam line to the turbine contains normally closed, motor-operated, spring loaded trip throttle valve FCHV0312. Upon initiation of an auxiliary feedwater actuation signal, the trip throttle valve operator moves in the closed direction in order to latch the mechanical linkage of the valve stem. A limit switch is used to detect when the trip throttle valve has reached the latched position. Once in the latched position, the direction of the motor operator is reversed and the trip throttle valve opens admitting steam to the governor flow control valve, which is normally fully open, starting the turbine-driven auxiliary feedwater pump. During the performance of Step 6.2.8 of Procedure OSP-SA-0007A, the turbine-driven auxiliary feedwater pump did not start as expected. Following the failed start, the licensee discovered that valve FCHV0312 had traveled approximately 70 percent of the closed stroke in an attempt to latch the valve operator but had stopped prior to latching the valve.
Troubleshooting was conducted on May 25, 2009, under Job 09003598, Task 910. During that job, the licensee determined that the closed torque switch had opened which stopped the motor-operated valve actuator. Callaway operators then declutched and manually operated the trip throttle valve actuator. During manual operation of the actuator, the sliding nut and screw spindle were observed to have an audible squeak while moving. A sample of the grease taken from the valve spindle was later examined using infrared spectrometry. The results of that analysis showed that the chemical composition was comparable to that of Mobile 28 grease and showed no other traces of either zinc or lithium which are found in Exxon Nebula EP-1 or EP-2 greases, the other two greases used on valve stems at the Callaway Plant. Those results indicated that the grease found on the valve FCHV0312 on May 25, 2009, was not mixed with other, non-approved greases.
Following troubleshooting, the licensee performed Job 09003598, Task 510, to lubricate the trip throttle valve. During that job, the licensee cleaned, inspected, and lubricated the trip throttle valve sliding nut, screw spindle, split coupling and trip linkages. The valve was manually operated during the performance of the job to allow access to all valve components. The mechanical maintenance technician who operated the valve noted that excessive forces were necessary to manipulate the valve and an audible squeak could be heard as the valve spindle rotated. As the valve was lubricated, the maintenance staff observed that the valve hand wheel became easier to manipulate and the audible squeak subsided. Once the valve was adequately lubricated, the licensee performed postmaintenance testing under Job 09003598, Task 920, to confirm the valve could operate as designed.
Missed Lubrication of Valve FCHV0312 Callaway replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system (Preventive Maintenance 0824900). The turbine-driven auxiliary feedwater pump trip throttle valve that failed to open on May 25, 2009, was replaced under Job 0551578 during Refueling Outage 16 (October 2008). The valve was replaced with a similar valve that was removed from service during Refueling Outage 13 in April 2004 and refurbished in September 2007. Replacement Preventive Maintenance 0824900 did not have specific lubrication instructions. Lubrication of valve FCHV0312 is accomplished by Preventive Maintenance Task 0810863 which is completed every refueling outage and was scheduled for Refueling Outage 16 as Job 07506359. The maintenance supervisor assigned to coordinate Job 07506359 made a job routing request, to engineering 8 Enclosure inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced. Although not specifically referenced, the portion of the job requested to be closed without action was Section 6.3, "Trip Throttle Valve Lubrication and Inspection." The motor-operated valve system engineer replied to the job routing request that the valve had been diagnostically tested satisfactorily and that no additional inspection was needed. The engineer did not consult with the technician requesting closure and did not verify that the lubrication job task being closed was completed by the installation procedure. Consequently, the lubrication portion of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was not performed. Since the valve was not lubricated during Refueling Outage 16, the last lubrication of the valve prior to the May 25, 2009, failure occurred during the refurbishment of the valve in September 2007. The lack of lubrication on valve FCHV0312 resulted in increased friction between the sliding nut and spindle which prevented the valve from opening on demand May 25, 2009.
a. Inspection Scope
The team evaluated the events leading to and the licensee response to the failure to start of the Callaway turbine-driven auxiliary feedwater pump. In order to review each area of the special inspection charter issued on June 17, 2009, the team reviewed calculations, design documents, licensing documents, work orders, modification packages, and corrective action documents. The team evaluated licensee compliance with the applicable regulatory requirements and applicable codes and standards.
The team interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program.
The team assessed licensee implementation of their corrective action program, design controls, and procedure implementation.
b. Findings
Introduction.
The team identified a self-revealing apparent violation of Technical Specification 3.7.5, "Auxiliary Feedwater System," due to the failure to adequately lubricate turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312.
Description.
On May 25, 2009, Callaway Plant operators performed surveillance testing Procedure OSP-SA-0007A, "Train A AFAS Slave Relay Test," to test the slave relays used to start the turbine-driven auxiliary feedwater pump. During the performance of Step 6.2.8 of Procedure OSP-SA-0007A, the turbine-driven auxiliary feedwater pump did not start as expected. During troubleshooting, the licensee discovered that the turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 had traveled in the closed direction in an attempt to latch the valve operator but had stopped prior to completing its stroke. Electrical continuity checks revealed that the motor-operated valve closed torque switch had opened. Callaway operators declutched and manually operated the trip throttle valve actuator. An audible squeak was observed during manual operation indicating mechanical binding within the sliding nut and valve spindle. Following troubleshooting, the licensee performed Job 09003598 to lubricate the trip throttle valve. Once the valve was adequately lubricated, the licensee successfully performed postmaintenance testing.
9 Enclosure Callaway replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system. The turbine-driven auxiliary feedwater pump trip throttle valve that failed to open on May 25, 2009, was replaced during Refueling Outage 16 in October 2008. The valve was replaced with a similar valve that was removed from service during Refueling Outage 13 in April 2004 and refurbished in September 2007. The replacement procedure did not have specific lubrication instructions. Lubrication of the valve was scheduled to be performed a few days after replacement as Job 07506359. The maintenance supervisor assigned to coordinate the lubrication incorrectly assumed that the valve was adequately lubricated since it had been recently replaced. Callaway Procedure APA-ZZ-00320, "Work Execution," Section 4.12, allows for a job to be canceled if it is determined that the work is not necessary or has been completed by another job. The maintenance supervisor initiated a job routing request to engineering inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced. Although not specifically referenced, the portion of the job requested to be closed was Section 6.3, "Trip Throttle Valve Lubrication and Inspection."
Callaway engineering replied to the job routing request that the valve had been diagnostically tested satisfactorily and that no additional inspection was needed. The engineer's response only answered the specific question asked by the job routing request and did not examine the procedural requirements that were the subject of the request. The engineer did not consult with the technician requesting closure and did not verify that the lubrication job task being closed was completed by the installation procedure. Based on the input received from engineering, the mechanical maintenance supervisor closed the lubrication portion of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection." The mechanical maintenance supervisor failed to identify that the entire work scope of the superseded work document was included in the superseding work document which did not meet the requirements of licensee Procedure APA-ZZ-00320. Since the valve was not lubricated during Refueling Outage 16, the last lubrication of the valve occurred during the refurbishment of the valve in September 2007. The lack of lubrication on valve FCHV0312 resulted in increased friction between the sliding nut and spindle which caused the valve not to open upon demand on May 25, 2009.
The licensee initiated Significant Condition Adverse to Quality Callaway Action Request 200904216 to investigate the failure of the turbine-driven auxiliary feedwater pump to start due to an inadequately lubricated trip throttle valve. The licensee's root cause analysis determined that the failure to lubricate valve FCHV0312 was due to the failure to fully review the closure of the lubrication portion of Procedure MPM-FC-QK001 during Refueling Outage 16. The licensee also determined that while the actual timing of the failure could not be determined, it was reasonable to assume that the turbine-driven auxiliary feedwater pump was inoperable for a time frame greater than the technical specification allowed completion time.
Long term corrective actions were implemented to revise the replacement preventive maintenance procedure to include a lubrication section. Additionally, the licensee identified several enhancements to their lubrication program including an evaluation of the preventive maintenance frequency and an evaluation of lubricants used on valve FCHV0312. The inspectors noted that the corrective actions identified in the root cause analysis did not address programmatic issues concerning work execution and a lack of protocol for the initiation and response to job routing requests. Specifically, 10 Enclosure Procedure APA-ZZ-00320 allows for a single individual to close incomplete preventive maintenance tasks.
Analysis.
The performance deficiency associated with this finding involved the licensee's failure to ensure valve FCHV0312 was adequately lubricated such that it remained operable. Specifically, Section 6.3 "Trip Throttle Valve Lubrication and Inspection," of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was closed without adequate review. The lack of lubrication resulted in increased friction within the valve which caused the valve not to open on May 25, 2009. This finding is greater than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as potentially risk significant since the finding represented a loss of system safety function because the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout. The finding required a Phase 2 analysis. When evaluated per Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations," and the Callaway Plant Phase 2 pre-solved table item "Turbine Driven Auxiliary Feedwater Pump Fails to Start," the inspectors determined this finding to be potentially risk significant. The finding was forwarded to a senior reactor analyst for review. The senior reactor analyst performed the Phase 3 analysis, Attachment 4, and determined that preliminarily, the finding was of low to moderate safety significance.
The inspectors determined that this finding had a crosscutting aspect in the area of human performance associated with the work practices component because the licensee failed to follow the procedural guidance provided when changing the scope of a preventive maintenance task H.4(b).
Enforcement.
Technical Specification 3.7.5, "Auxiliary Feedwater System," requires, in part, that three trains of auxiliary feedwater shall be operable in Modes 1, 2 or 3. The technical specifications required that if one train of auxiliary feedwater is inoperable for greater than 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />, actions be taken to be in Mode 3 within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and Mode 4 within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. Contrary to the required action statements, on May 25, 2009, the turbine-driven auxiliary feedwater pump train was found to be inoperable due to a lack of lubrication of trip throttle valve FCHV0312. Subsequent review determined that the lack of lubrication resulted in the turbine-driven auxiliary feedwater pump being inoperable for greater than 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. Pending determination of the finding's final safety significance, this finding is identified as Apparent Violation (AV)05000483/2009009-01, "Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve."
1R3 Review of Lubrication and Valve Replacement Program for Valve FCHV0312 Valve Lubrication The licensee has two preventive maintenance tasks that lubricate valve FCHV0312. The first is Preventive Maintenance 0810863 which is a Terry turbine equipment check and is performed every refueling outage. The second is Preventive 11 Enclosure Maintenance 0818214 which is a turbine-driven auxiliary feedwater pump equipment check that is performed every 72 weeks. The scheduling of the two preventive maintenance tasks is such that Preventive Maintenance 0810863 is performed every refueling outage and Preventive Maintenance 0818214 is performed in the middle of the operating cycle. The requirements to lubricate the trip throttle valve are contained in Step 6.3 of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection." That procedure directs mechanical maintenance personnel to clean, inspect and lubricate the valve at the sliding nut, trunnion screws, the screw spindle, the trip linkages and pins, and at the split coupling. The frequency of lubrication is consistent with industry standards and vendor guidance which recommends a minimum 18 to 24-month lubrication frequency.
The trip throttle valve is lubricated with Mobile 28 grease which is a high temperature, anti-wear grease composed of a polyalphaolefin synthetic base fluid with an organo-clay thickener. Mobile 28 grease, designed for lubrication of splines, screws and worm gears, falls between the National Lubricating Grease Institute Grade No. 1 and No. 2. The recommended operating temperature is -55°C to 180°C (-67°F to 356°F) with appropriate relubrication intervals. Mobile 28 is recommended by the trip throttle valve vendor as an acceptable lubricant. The licensee first began using Mobile 28 grease for valve FCHV0312 in 1995 under Request for Resolution 16006A. That evaluation determined that Mobile 28 was an acceptable lubricant based on the valve vendor's recommendations and based on the operating temperatures and physical characteristics of the lubricant.
Valve FCHV0312 is used as a steam admission valve for the turbine-driven auxiliary feedwater pump. The valve is physically arranged downstream of two air operated valves that provide a source of steam from steam generators B and C. Each line is equipped with a bypass line that allows steam to be present at the trip throttle valve to maintain the line warm and free of moisture. Because valve FCHV0312 is a steam admission valve, it is constantly exposed to elevated temperatures. When reviewing operating experience, the team found that Mobile 28 grease, in tests performed for NUREG/CR-6750, "Performance of MOV Stem Lubricants at Elevated Temperature," experienced physical characteristic changes when exposed to elevated temperatures.
Of particular concern in the operating experience was the observation that Mobile 28 changed from bright red in color to almost black when heated and that the lubricant appeared to have thickened and hardened. Additionally, an absorption test identified that elevated temperatures (around 250°F) increased the lubricant's tendency to lose oils and could result in increased stem friction. The team noted that the licensee's evaluation of the use of Mobile 28 grease was bounded at 215°F but actual operating conditions at valve FCHV0312 were approximately 300°F.
The team reviewed the licensee's lubrication procedure and found that no guidance was provided as to the quantity of grease to use, how to properly purge the system of old lubricant, and at exactly what lubrication point new grease should be applied. At Callaway, these particular details are considered "skill of the craft." During the root cause investigation, the licensee interviewed mechanical maintenance staff and confirmed there was not consistent information provided on the amount of lubricants to use or the specific points to lubricate. The root cause analysis associated with Callaway Action Request 200904219, Enhancement 2.0, recommended revisions to Preventive Maintenance 0810863 and 0818214, to specify lubrication locations, amount and type of lubricant to use, and instructions to manually stroke the valve while lubricating through 12 Enclosure the trunnion screws to assure the trip throttle valve is fully lubricated across the entire length of the spindle.
When reviewing the licensee's lubrication program, the inspectors examined the licensee's storage of lubricants and noted several expired consumables (grease, lubricants, oils, etc.) within the mechanical maintenance shop. The inspectors questioned the mechanical maintenance supervisor to determine if potential existed for use of expired lubricants in safety related applications. Based on interviews, the inspectors determined that the potential to use expired consumables in safety related applications exists since there is no procedural guidance that requires verification of a product's expiration date prior to use, it is simply an expectation. The licensee reviewed the work history associated with these products and verified that these materials were not used during work on safety related components. This issue was entered into the licensee's corrective action program as Callaway Action Request 200905038.
Additionally, the inspectors observed that the licensee's trip throttle valve lubrication specification within the plant equipment database called for Exxon Nebula EP-1 which is not a vendor approved grease for the trip throttle valve. The team verified that current work instructions and procedures require the use of Mobile 28 grease. The team considered this a minor administrative issue and the licensee entered the discrepancy into their corrective action program as Callaway Action Request 200905032.
Valve Replacement and Refurbishment Currently, the licensee replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system (Preventive Maintenance 0824900). The frequency of replacement is consistent with industry standards and vendor guidance which recommends a minimum 6-year replacement frequency. Callaway's frequency is more conservative because internal operating experience has demonstrated that the trip throttle valve develops too much seat leakage during the fourth operating cycle which results in increased corrosion of safety related components on the 1988 foot elevation of the auxiliary buildings as well as an increase in nonoperating bearing oil temperatures. The procedure used to rebuild the valve is contained within the preventive maintenance task. The scope of the work during a valve refurbishment includes:
- Disassembly of the valve
- General inspection of all valve components
- Blue check of valve seating surfaces to ensure 100 percent circumferential contact
- Dimensional measurements and component tolerance verification. Excessively worn parts are replaced
- Gasket replacement and valve lubrication
- Reassembly of the valve 13 Enclosure During the most recent refurbishment of valve FCHV0312 performed under Job W219154 in September 2007, the valve vendor was on site to provide technical assistance. A field service report (FS-47107) was issued following completion of the job and documented several adverse conditions associated with the valve. The team inspected the spare turbine-driven auxiliary feedwater pump trip throttle valve and found that the spare valve's spindle and sliding nut lubricant was thick and had darkened in color when compared to fresh lubricant. The inspectors also observed that there appeared to be evidence of galling of the spindle to valve stem thrust washer surfaces.
a. Inspection Scope
The team evaluated the licensee's programs and procedures for maintaining the turbine-driven auxiliary feedwater pump trip throttle valve. Specifically, the team reviewed procedures associated with valve lubrication and refurbishment, preventive maintenance tasks including preventive maintenance change history, and the licensee's evaluation of the lubricants used and the frequency of lubrication. The team compared the licensee's programs to applicable industry and vendor guidance associated with the trip throttle valve. The team found that the licensee has performed many equipment and program improvements to the turbine-driven auxiliary feedwater pump system that address industry standards and operating experience such as trip throttle valve spindle upgrades, a digital control system upgrade, governor valve stem material improvements, maintenance procedure revisions, and start-up transient performance monitoring and trending. Team interviews with station personnel noted that the licensee actively participates in industry organizations that seek to improve industry auxiliary feedwater system performance.
The team interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program.
b. Findings
===.1
Introduction.
=
The team identified a green noncited violation of Technical Specification 5.4.1.a, "Procedures," for the failure to provide adequate procedural guidance for the lubrication of valve FCHV0312.
Description.
On May 17, 2002, the licensee initiated Callaway Action Request 200203228 to request a change to the preventive maintenance frequency for the turbine-driven auxiliary feedwater pump trip throttle valve. While researching the preventive maintenance requirements for the valve, the lead responder to the Callaway action request documented an additional concern with the trip throttle valve lubrication procedure in that the spindle coupling is not fully accessible in either the fully open or fully closed positions. The Callaway action request recommended that during lubrication, the trip throttle valve be manually mid-positioned to fully lubricate the valve stem and bushings. The licensee initiated Preventive Maintenance AF 6084 to make a change to the lubrication procedure for valve FCHV0312 such that it would address the concerns raised in Callaway Action Request 200203228. On May 16, 2003, the Callaway action request noted that Preventive Maintenance 818214 would be changed to include procedural guidance that the valve may be manually repositioned to allow access to valve components.
14 Enclosure The inspectors reviewed Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," and found that the corrective actions suggested by Callaway Action Request 200203228 were not fully developed and that the procedure, as written, provided ambiguous guidance on how to lubricate valve FCHV0312. Specifically, the inspectors noted that the procedure only gives the option of repositioning the valve to access certain components but does not require the valve be manually mid-positioned to ensure adequate lubrication. The procedure does not specify what components of concern are to be lubricated when the valve is manually repositioned. The inspectors also observed during a system walkdown that the valve design is such that the valve spindle, in addition to the valve coupling, is not fully accessible from a static position since the sliding nut obstructs access to the spindle threads. While the sliding nut is equipped with Zerk type grease fittings that are ported to allow the lubricant to be applied to the spindle, the entire length of the spindle could only be lubricated through this point if the valve was manually operated while grease was injected at the fitting location. The inspectors also noted the procedure does not specify lubrication locations or the amount of lubricant to use and how to achieve a proper purge of old lubricant. The inspectors questioned engineering and maintenance staff on how these issues were addressed during trip throttle valve lubrications and were told that this level of work was within the skill of the craft.
In addition to the inadequacies identified by the NRC, the licensee's root cause investigation team also identified that the lubrication procedure for the turbine-driven auxiliary feedwater pump trip throttle valve has several inadequacies that could lead to improperly lubricating the valve. The licensee's review included interviews with mechanical maintenance staff which confirmed there was not consistent information provided on the amounts of lubricants to use or the specific points to lubricate. The root cause analysis associated with Callaway Action Request 200904219 recommended Enhancement 2.0 to revise the valve lubrication procedure to specify lubrication locations, amount and type of lubricant to use and instructions to manually stroke the valve while lubricating through the trunnion screws to assure the trip throttle valve is fully lubricated across the entire length of the spindle.
Analysis.
The performance deficiency associated with this finding involved the licensee's failure to ensure the procedures required to lubricate valve FCHV0312 were adequate. This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of procedural quality and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not to have a crosscutting aspect since the 2003 lubrication procedure revision was not reflective of current licensee performance.
Enforcement.
Technical Specification 5.4.1.a, "Procedures," required that written procedures be established and implemented covering activities specified in Appendix A, "Typical Procedures for Pressurized Water Reactors," of Regulatory Guide 1.33, "Quality Assurance Program Requirements (Operation)," February 1978. Regulatory Guide 1.33, Appendix A, Section 9.a, required procedures for performance of maintenance.
15 Enclosure Contrary to the above, from May 16, 2003, until June 26, 2009, Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was not adequate to ensure that valve FCHV0312 was adequately lubricated. Because of the very low safety significance of this finding and because the licensee has entered this issue into their corrective action program as Callaway Action Requests 200904216 and 200905077, this violation is being treated as a noncited violation in accordance with Section VI.A.1 of the Enforcement Policy: NCV 05000483/2009009-02, "Failure to Maintain an Adequate Lubrication Procedure for Valve FCHV0312."
===.2
Introduction.
=
The team identified a green noncited violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control," for the failure to adequately evaluate the use of Mobile 28 grease for the turbine-driven auxiliary feedwater pump trip throttle valve.
Description.
On March 2, 1993, the licensee initiated Suggestion Occurrence Solution 199300209 to document that during preventive maintenance, turbine-driven auxiliary feedwater pump trip throttle Valve FCHV0312, failed to open. Action 3 to Suggestion Occurrence Solution 199300209 determined the cause to be hardened Nebula EP-1 grease around the valve spindle and sliding nut. Remedial action for Suggestion Occurrence Solution 199300209 included changing the preventive maintenance frequency from every 24 weeks to every 12 weeks and initiation of Request for Resolution 13409A to evaluate a substitute grease for the actuator sliding nut and spindle. Request for Resolution 13409A requested a change from Nebula EP-1 grease to Shell Narina EP-0 or Aeroshell Grease 5 due to hardening of grease at the valve.
Valve FCHV0312 is used as a steam admission valve to the turbine-driven auxiliary feedwater pump. The valve is physically arranged downstream of two air operated valves that provide steam from the Steam Generators B and C. Since valve FCHV0312 is used as a steam admission valve, it is continuously exposed to high temperatures. The disposition of Request for Resolution 13409A determined that no change should be implemented since the proposed lubricant replacements were not acceptable greases for addressing long term hardening due to high temperatures at valve FCHV0312.
On April 24, 1995, the licensee initiated Request for Resolution 16006A to change the grease used on trip throttle valve FCHV0312. Request for Resolution 16006A referenced Request for Resolution 13409A and noted that the Nebula EP-1 grease currently in use was not recommended for applications where continuous high temperatures can cause oil evaporation, resulting in hardening of the grease. The request for resolution documented that the valve vendor was contacted and recommended a high-temperature lithium-based grease such as Mobile 28. Mobile 28 grease is a high-temperature, anti-wear grease composed of a synthetic base fluid with a non-soap thickener. The final disposition of Request for Resolution 16006A approved the use of Mobile 28 grease in place of Exxon Nebula EP-1 for valve FCHV0312 since it was recommended by the valve manufacturer and the temperature of the valve FCHV0312 screw spindle was approximately 215°F which is within the temperature range recommended by the grease manufacturer.
The inspectors reviewed Request for Resolution 16006A and noted that the evaluation provided no documentation as to what constitutes an appropriate relubrication interval as recommended in the vendor data sheet for Mobile 28. Additionally, the inspectors questioned if the assumed temperature of 215°F at the Valve FCHV0312 screw spindle was consistent with the current plant configuration. The licensee measured the temperature of various components on Valve FCHV0312 on June 23, 2009. Those 16 Enclosure measurements indicated that the valve exhibited temperatures ranging from 235°F at the sliding nut to 272°F at the valve coupling. Additional measurements were taken on June 29, 2009, following a planned surveillance of the turbine-driven auxiliary feedwater pump. Those measurements showed temperatures near 300°F at the valve coupling. Since the valve displayed temperatures exceeding those in Request for Resolution 16006A, the inspectors questioned if the use of Mobile 28 grease was appropriate, particularly since operating experience performed by Idaho National Lab and documented in NUREG/CR-6750, "Performance of MOV Stem Lubricants at Elevated Temperature," suggested Mobile 28 grease has a tendency to thicken and harden at temperatures exceeding 250°F. Additionally, the test performed in NUREG/CR-6750 identified that elevated temperatures increased the lubricant's tendency to release oils and could result in increased stem friction.
Following questioning by the inspectors, the licensee initiated Callaway Action Request 200905067 and Request for Resolution 200905621 to determine if Mobile 28 grease was an appropriate lubricant for valve FCHV0312. The results of that analysis determined that the thread angle of the stem for valve FCHV0312 is similar to that of two of the stems tested in NUREG/CR-6750. While the coefficient of friction was determined to increase for those stems, the licensee determine that the maximum postulated coefficient of friction is bound by the current thrust and torque analysis for valve FCHV0312. The final conclusion of the Callaway action request was that while the recorded temperatures on valve FCHV0312 exceeded those evaluated for in Request for Resolution 16006A, the use of Mobile 28 grease is appropriate when coupled with an adequate relubrication interval. At the close of the inspection, the licensee was still evaluating the appropriate preventive maintenance frequency for lubrication of the valve.
Analysis.
The performance deficiency associated with this finding involved the licensee's use of nonconservative assumptions in the 1995 evaluation of Mobile 28 grease as a lubricant for valve FCHV0312. This finding is greater than minor because it was associated with the Mitigating Systems Cornerstone attribute of design control and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings,"
the issue screened as having very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. This finding did not have a crosscutting aspect since the inadequate 1995 lubrication evaluation was not reflective of current licensee performance.
Enforcement.
Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion III, "Design Control," required, in part, that measures be established to assure that applicable regulatory requirements and the design basis, as defined in paragraph 50.2 and as specified in the license application, are correctly translated into specifications, drawings, procedures and instructions. Contrary to the above, on April 24, 1995, Callaway initiated Request for Resolution 16006A which failed to correctly translate the operating temperature of valve FCHV0312 when evaluating the use of Mobile 28 grease. The assumed temperature of 215°F degrees was nonconservative and operating experience indicated that actual temperatures experienced at valve FCHV0312 could result in thickening of the lubricant and loss of viscosity. Because of the very low safety significance of this finding and because the 17 Enclosure licensee has entered this issue into their corrective action program as Callaway Action Request 200905067, this violation is being treated as a noncited violation in accordance with Section VI.A.1 of the Enforcement Policy: NCV 05000483/2009009-03, "Failure to Adequately Evaluate the Use of Mobile 28 Grease for the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve."
===.3
Introduction.
=
The team identified a green noncited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," regarding the licensee's failure to follow the requirements of Callaway Procedure APA-ZZ-00500, "Corrective Action Program," associated with the turbine-driven auxiliary feedwater pump trip throttle valve.
Description.
On September 10, 2007, the licensee performed Job W219154 to rebuild the spare turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. The scope of the work included as-found inspections, valve disassembly, valve reassembly, and postmaintenance testing. Job W219154 was performed with assistance from a vendor field representative who supervised disassembly, inspection, and rework of the valve. Upon completion of Job W219154, the licensee did not identify any degraded or nonconforming conditions. On September 14, 2007, the valve vendor issued a field service report associated with Job W219154 performed on September 10, 2009. The field service report noted several adverse conditions discovered during the performance of Job W219154 including:
- Hand wheel forces were found to be excessive when the valve was operationally tested.
- Some sticking was noted when the valve was manually stroked by pulling the latch-up lever, with the valve spring removed.
- Sticking was noted when the valve was manually stroked by pulling the pilot valve and stem with the coupling removed.
- The grease in the coupling was found to be dry and gritty. Additionally, grease on the contact areas of the yoke, bore, and sliding nut was found to be hard and gritty.
- The thrust washer and the screw spindle were found to be galled at the contact areas.
- A hard nonmetallic dark grey foreign substance was found on the pilot valve and stem at the bushing contact area.
- Inspection revealed several dimensional and visual discrepancies at the coupling, screw spindle, sliding nut, link pin, and trip hook shaft. The valve vendor also noted that the licensee's auxiliary feedwater pump turbine trip throttle valve overhaul procedure was inadequate with regards to safe disassembly of the valve. The field service report documented that improper disassembly of the valve could possibly result in damage to valve parts and could result in personnel injury. The vendors recommended that prior to any rebuild of the valve, the licensee review and revise their procedure to comply with industry standards. The inspectors found that the 18 Enclosure licensee did revise their overhaul procedure to address the safety concerns and comply with industry standards. Finally, the vendor recommended several preventive maintenance items for valve FCHV0312 including a weekly, manual exercising of the valve and a monthly lubrication with a high temperature, lithium based grease at all fittings in the actuator portion of the valve and at the coupling.
The inspectors reviewed the vendor report associated with Job W219154 and noted that several of the conditions met the requirements specified in Procedure APA-ZZ-00500, "Corrective Action Program," for entry into the corrective action program. Specifically, Section 4.1 required that a Callaway action request be generated for a condition that could credibly impact nuclear safety, radiological safety, personnel safety, or plant reliability. Additional review by the licensee also determined that since the licensee's corrective action program procedure does not require shift manager notification for identification of an adverse condition associated with equipment previously installed in the plant, the possibility exists that the impact on operability for adverse conditions could be overlooked even if there is possible extent of condition concerns.
Analysis.
The performance deficiency associated with this finding involved the licensee's failure to follow the requirements of Callaway Procedure APA-ZZ-00500, "Corrective Action Program." Specifically, licensee personnel failed to initiate condition reports for multiple adverse conditions found during the 2007 rebuild of the spare turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312. This finding is greater than minor because if left uncorrected, the failure to fully utilize the corrective action program could become a more significant safety concern. The inspectors determined that this finding impacted the mitigating systems Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as having very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage times, and did not affect seismic, flooding, or severe weather initiating events. The cause of this finding is related to the problem identification and resolution crosscutting component of the corrective action program because licensee personnel failed to implement a corrective action program with a low threshold for identifying issues P.1(a).
Enforcement.
Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," requires, in part, that activities affecting quality shall be prescribed by documented instructions or drawings, of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions or drawings. Contrary to the above, on September 14, 2007, the licensee failed to enter several adverse conditions identified during rebuild of their safety related turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 into their corrective action program as required by Section 4.1 of Procedure APA-ZZ-00500, "Corrective Action Program," Revision 44, that states a Callaway action request be initiated for a condition that could credibly impact nuclear safety, radiological safety, personnel safety, or plant reliability. Because of the very low safety significance of this finding and because the licensee has entered this issue into their corrective action program as Callaway Action Request 200905053, this violation is being treated as a noncited 19 Enclosure violation in accordance with Section VI.A.1 of the Enforcement Policy: NCV 05000483/2009009-04, "Failure to Enter Conditions Adverse to Quality Associated with the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve into the Corrective Action Program."
1R4 Review of Root Cause Analysis, Extent of Condition and Corrective Actions On June 8, 2009, the licensee established a root cause analysis team to investigate the facts and identify the causes associated with the failure of the turbine-driven auxiliary feedwater pump trip throttle valve. The team conducted their review in accordance with Procedure APA-ZZ-00500, Appendix 12, "Significant Adverse Condition." The licensee's procedure requires the team to:
- Identify and validate root and contributing causes
- Conduct an extent of condition review
- Determine extent of cause
- Develop corrective actions The licensee's final root cause analysis was completed on July 16, 2009.
Root Cause Methodology The licensee performed their analysis utilizing a structured root cause analysis method in accordance with Procedure APA-ZZ-00500, Appendix 12, "Significant Adverse Condition - Significance Level 1." A fault tree was developed for FCHV0312 to determine the valve's failure mechanism. An event and causal factors chart was created to document the sequence of events and the casual factors that led to the failure of valve FCHV0312. The results of that analysis determined valve FCHV0312 was not lubricated properly because, during Refuel 16, the lubrication portion of Procedure MPM-FC-QK-001, "Auxiliary Feedwater Pump Turbine Annual Inspection," was not performed which resulted in increased valve spindle friction. This increased friction caused the valve's torque switch to trip which prevented the valve from opening.
Root Cause and Corrective Actions to Prevent Recurrence The licensee determined that the root cause of the event was that the maintenance supervisor did not review the full scope of the replacement preventive maintenance procedure when closing the lubrication portion of Procedure MPM-FC-QK-001, "Auxiliary Feedwater Pump Turbine Annual Inspection," during Refueling Outage 16. Procedure APA-ZZ-00320, "Work Execution," Section 4.12, required the maintenance supervisor to perform a review to ensure the entire work scope of the superseded work document is included in the superseding document.
As a corrective action to prevent recurrence, the licensee proposed revision to the replacement preventive maintenance procedure to incorporate valve lubrication requirements.
20 Enclosure Enhancements and Other Issues As part of their investigation, the licensee identified multiple "other issues" which are defined as nonsignificant issues that licensee management wishes to capture within the corrective action program. Other issues identified by the licensee included:
- Revise the lubrication program to improve control and storage of lubricants within the plant. The inspectors identified that several expired lubricants were stored within the mechanical maintenance shop. See section 1R5.
- Revise the lubrication procedure for the turbine-driven auxiliary feedwater pump trip throttle valve to include amounts and type of lubricant to use and the need to manually stroke the valve to ensure the valve is fully lubricated.
- Determine if training on lubrication practices is required. Interviews with maintenance personnel revealed that there is not consistent knowledge on the proper use and application of lubricants.
Several enhancements were identified that are considered actions that improve a process or design but do not correct the problem associated with the turbine-driven auxiliary feedwater pump trip throttle valve. Since enhancements are considered as improvements and not actions to correct a problem, there is no procedural requirement to implement an enhancement. Enhancements identified by the licensee included:
- Evaluate a change from Mobile 28 grease to a better performing lubricant for the turbine-driven auxiliary feedwater pump trip throttle valve.
- Evaluate a change to the lubrication periodicity for the turbine-driven auxiliary feedwater pump trip throttle valve.
- Evaluate the work execution procedure and programs to determine if changes are necessary to the review and approval procedure for closing work tasks.
a. Inspection Scope
The team reviewed the licensee's root cause analysis to determine if it was conducted to a level of detail commensurate with the significance of the problem. As part of their review, the inspectors interviewed key station personnel from operations, design and system engineering, maintenance, and the corrective action program. Additionally, the team interviewed the root cause team members and the members from the licensee's Corrective Action Review Board.
The team reviewed the licensee's corrective actions to ensure they addressed the extent of condition and whether they were adequate to prevent recurrence. In particular, the team reviewed station procedures and processes to determine if any other motor-operated valves may have been improperly lubricated.
b. Observations and Findings
The inspectors determined that the licensee's analysis accurately captured the root cause of the event. Since the event was determined to be caused by an individual 21 Enclosure human error, the inspectors noted that the licensee appropriately identified a need to implement a defense-in-depth mechanism to ensure the turbine-driven auxiliary feedwater pump trip throttle valve is adequately lubricated. The inspectors found that the corrective action to prevent recurrence would likely ensure that the valve is fully lubricated in the future. Effectiveness reviews were implemented by the licensee to ensure the corrective actions implemented were sufficiently robust as to address the root
cause. The inspectors did note that the final root cause analysis report was narrowly focused and failed to consider all potential causal factors associated with the failure of the turbine-driven auxiliary feedwater pump trip throttle valve.
The team found that the final root cause analysis lacked adequate justification to exclude several of the identified enhancements and other issues as contributors to the event. Specifically, the inspectors noted the following:
- The replacement procedure for valve FCHV0312 did not have specific lubrication instructions. The team confirmed through interviews that typical replacement procedures include instructions to restore the replaced equipment to a functional status. The difference between the valve FCHV0312 replacement procedure and typical replacement procedures caused the mechanical maintenance supervisor to assume the valve was adequately lubricated at completion of the replacement.
- Engineering did not perform an adequate review of the job routing request inquiring if additional inspection of the valve was required. The lack of technical rigor was largely attributable to the fact that no protocol exists on how job routing requests should be evaluated and a fundamental misunderstanding between departments on the intention of such requests.
- Communications between maintenance and engineering lacked sufficient detail to ensure the required preventive maintenance tasks for valve FCHV0312 were accomplished.
Additionally, the inspectors noted several potential flawed defenses and latent organizational weaknesses which were not documented in the final report. Specifically, the inspectors noted that:
- The replacement program for valve FCHV0312 allows for the valve to be not adequately lubricated if the replacement and lubrication preventive maintenance tasks are performed out of sequence. Reliance is placed on proper scheduling that increases the probability of equipment problems due to human error.
- Procedure APA-ZZ-00320 provides ambiguous guidance on how to close work documents. Additionally, the procedure, as written, allows for single individuals to close preventive maintenance tasks. The lack of a required second or peer check when closing incomplete preventive maintenance tasks could result in less than adequate technical review.
- The lubrication frequency for valve FCHV0312 does not provide sufficient margin in that if a preventive maintenance task is missed or deferred, the potential exists to impact equipment operation.
22 Enclosure The team noted that the comprehensiveness of the final root cause analysis was impacted by a failure of the licensee to interview all pertinent individuals and fundamental errors in the use of systematic analytical techniques such as TapRoot and event and causal factor charting. Additionally, less than adequate management oversight of the root cause process, such as management sponsor and Corrective Action Review Board reviews, allowed the final root cause to be issued without thoroughly evaluating all significant causal factors.
The items identified by the inspection team were discussed with the licensee. As a result, the licensee initiated Callaway Action Request 200906143 to examine the quality of the root cause analysis performed and to determine if programmatic issues in cause evaluation exist.
1R5 Review of Operating Experience
a. Scope
The team reviewed internal operating experience by obtaining a list of plant corrective action documents related to the auxiliary feedwater system and selecting those documents related to lubrication and maintenance of the turbine-driven auxiliary feedwater pump trip throttle valve. The team further reviewed the licensee's review of industry operating experience for the auxiliary feedwater system and lubrication problems with motor-operated valves. The team review included inspection of the licensee's operating experience program and specific review of related operating experience during the root cause investigation for the May 25, 2009, failure of the turbine-driven auxiliary feedwater pump trip throttle valve.
For external operating experience, the NRC Operating Experience Branch provided the results of keyword searches related to motor-operated valve stem lubrication issues, Terry turbine Gimpel 4-Inch valves, and findings associated with turbine-driven pumps. The NRC Operating Experience Branch provided a list of licensee event reports, NRC information notices, NUREG documents and other operating experience information. The team selected operating experience information that was applicable to this inspection and reviewed whether the licensee had addressed the items in their root cause analyses related to these events or had processed the information through their operating experience program. As part of their review, the inspectors performed an auxiliary feedwater system walkdown to determine if applicable industry operating experience had been incorporated into system design and maintenance practices.
b. Findings and Observations
During a system walkdown, the inspectors noted that the orientation of the turbine-driven auxiliary feedwater pump electrical trip solenoid was rotated 180 degrees from the required configuration. With the solenoid in such a configuration, the connecting pin did not rest on the horizontal base of the mounting bracket and was approximately 1 inch below the required position. Additionally, the configuration is such that the solenoid link appeared to be resting on the trip crank pin and the solenoid plunger was at a slight angle which would require additional force to actuate. The licensee evaluated this nonconforming condition and determined that the electrical trip solenoid would still be available to perform its function. The nonconforming condition was entered into the licensee's corrective action program as Callaway Action Request 200905004.
1R6 Review of Generic Letter 89-10 and Periodic Verification Program
a. Inspection Scope
The team reviewed the licensee's Generic Letter 89-10 program for the turbine-driven auxiliary feedwater pump trip throttle valve including the licensee's periodic verification program. As part of their review, the inspectors examined the licensee's response to Generic Letter 89-10, "Safety Related Motor Operated Valve Testing and Surveillance," and Generic Letter 96-05, "Periodic Verification of Design Basis Capability of Safety-Related Motor-Operated Valves." Additionally, the inspectors reviewed the licensee's engineering analysis of the system and dynamic and static testing results to ensure the turbine-driven auxiliary feedwater pump trip throttle valve is adequately designed for and has the ability to function under design-basis conditions
b. Findings and Observations
The team determined that the licensee had not appropriately followed their Generic Letter 89-10 program for the turbine-driven auxiliary feedwater pump trip throttle valve. The licensee's program is based on periodic verification that performs a static diagnostic test every 6 years or four refueling outages and a dynamic test every three refueling outages, for torque-controlled rising-stem valves that do not have at least 25 percent capability margin above their design operating requirements. The testing frequency is established with the assumption that stem lubrication is performed every 18 months as documented in the licensee's Generic Letter 96-05 response. The licensee failed to meet this requirement when valve FCHV0312 lubrication was not performed in Refueling Outage 16 resulting in valve FCHV0312 not being lubricated for approximately 20 months. The performance deficiency associated with the licensee's failure to adequately lubricate the turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 is described in Section 1R2 of this report.
While comparing the licensee's Generic Letter 96-05 response to Calculation FC-20, "Manual Operation of FCHV0312," the inspectors identified that the licensee's value listed for differential thrust to open was 3172 pounds-force but the licensee's generic letter response indicated the design differential thrust to open valve FCHV0312 is 4350 pounds-force. Additionally, dynamic testing from Refueling Outage 16 in October 2008 determined that the differential thrust to open with error was 5316 pounds-force. Subsequent review by the licensee determined that the value contained within the plant equipment database was not up to date and contained a nonconservative value for design differential pressure thrust to open. While the information contained in the licensee's database was not up to date for the current plant configuration, the current required open thrust was determined to be bounded by the maximum allowed differential pressure thrust for valve FCHV0312 which is 7106 pounds. The licensee entered the discrepancy into their corrective action program as Callaway Action Request 200905239.
24 Enclosure 1R7 Potential Generic Issues
a. Scope
The team evaluated the failure of Callaway's turbine-driven auxiliary feedwater pump trip throttle valve due to inadequate lubrication to determine whether any potential generic issues should be communicated to the industry (e.g., information notices, generic letters, and bulletins).
b. Findings and Observations
The team determined that this issue may warrant a generic communication informing other licensees of the types of problems encountered. Specifically, the team determined that the design of the system is such that when the turbine-driven auxiliary feedwater pump trip throttle valve is used as a steam admission valve it exposes the component to prolonged elevated temperatures which requires special consideration in terms of stem lubricant selection and frequency of lubrication. The team will discuss this issue with the NRC Office of Nuclear Reactor Regulation for possible issuance as an information notice. The team did not identify any other potentially generic safety issues during the inspection.
OTHER ACTIVITIES
4OA2 Identification and Resolution of Problems (71152
)
a. Inspection Scope
The team reviewed maintenance, corrective actions, and modification history related to the auxiliary feedwater system to evaluate whether any longstanding issues continued to impact current performance. The team also reviewed the operating experience database and previous inspection reports. The team interviewed the root cause team, the system engineer, and other personnel.
b. Findings and Observations
Introduction.
The team identified a green noncited violation of Technical Specification Limiting Condition for Operation 3.0.4 for entering Mode 3 with the turbine-driven auxiliary feedwater pump inoperable.
Description.
On November 3, 2008, while in Refueling Outage 16, the licensee replaced the servo control valve for the turbine-driven auxiliary feedwater pump governor. During postmaintenance testing in Mode 4, an unexpected overspeed trip of the turbine occurred. Callaway operations staff inappropriately concluded that since the turbine was running using the auxiliary steam system at the time of the postmaintenance test, the cause of the overspeed of the turbine was likely due to a water slug within the system. Without determining an exact cause of the overspeed, the testing was re-performed satisfactorily. Successful performance of the postmaintenance test cleared a mode change restraint which allowed the plant to enter Mode 3 on November 4, 2008.
While in Mode 3, the licensee performed Jobs 06113631 and 07506205 to test the turbine-driven auxiliary feedwater pump using the normal steam supply. During performance of the jobs, the instrumentation and controls engineer noted that the 25 Enclosure channel two, governor valve position indicator was erratic. Troubleshooting was performed and it was found that the servo control valve replaced during Refueling Outage 16 was faulty. At 4:50 p.m. on November 5, 2008, the licensee entered the applicable technical specification and initiated repairs to the turbine-driven auxiliary feedwater pump. Postmaintenance testing was performed and the licensee exited the technical specification at 12:15 p.m. on November 6, 2008.
The team reviewed corrective action documents associated with the turbine overspeed and the faulty governor servo control valve. The inspectors questioned if the two corrective action documents were related and specifically, if the cause of the turbine overspeed that occurred in Mode 4 could have been related to the faulty governor servo control valve discovered while the unit was in Mode 3. Additionally, the team questioned if the turbine-driven auxiliary feedwater pump could have been inoperable during the mode change that occurred on November 4, 2008, in violation of Technical Specification Limiting Condition for Operation 3.0.4.
The licensee reviewed the Refueling Outage 16 work history associated with the turbine-driven auxiliary feedwater pump governor and found that currents for the servo position sensor were lower than expected and the abnormal readings indicated that the remote servo valve was not operating properly. Based on this evidence, the licensee concluded that the faulty servo control valve discovered on November 5, 2008, was responsible for the overspeed of the turbine-driven auxiliary feedwater pump that occurred in Mode 4 and that the equipment was inoperable during the transition to the Mode 3 change that occurred on November 4, 2008.
Analysis.
The performance deficiency associated with this finding involved the licensee's failure to ensure the turbine-driven auxiliary feedwater pump was operable prior to entering Mode 3. This finding is greater than minor because it is associated with the Mitigating Systems Cornerstone attribute of equipment performance and it affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using Manual Chapter 0609.04, "Phase 1 - Initial Screening and Characterization of Findings," the issue screened as very low safety significance because it was not a design or qualification deficiency that resulted in a loss of operability or functionality, did not create a loss of system safety function of a single train for greater than the technical specification allowed outage time and did not affect seismic, flooding, or severe weather initiating events. The inspectors determined that this finding has a crosscutting aspect in the area of problem identification and resolution associated with the corrective action program component because the licensee failed to fully evaluate the overspeed of the turbine-driven auxiliary feedwater pump that occurred on November 3, 2008 P.1(c).
Enforcement.
Technical Specification Limiting Condition for Operation 3.0.4. required, in part, that when a limiting condition for operation is not met, entry into a mode or other specified condition shall only be made when the associated actions of Limiting Condition for Operation 3.0.4, paragraphs a, b or c are met. Contrary to the above, on November 4, 2008, the licensee entered Mode 3 without meeting Limiting Condition for Operation 3.7.5.c, "Auxiliary Feedwater System," and without meeting the associated actions of Limiting Condition for Operation 3.0.4, paragraphs a, b or c. Because of the very low safety significance of this finding and because the licensee has entered this issue into their corrective action program as Callaway Action Request 200905313, this violation is being treated as a noncited violation in accordance with Section VI.A.1 of the 26 Enclosure Enforcement Policy: NCV 05000483/2009009-05, "Failure to Ensure Turbine-Driven Auxiliary Feedwater Pump is Operable Prior to Entry into Mode 3."
4OA3 Event Follow-up
(Closed) Licensee Event Report (LER) 05000483/2009002-00: Turbine-Driven Auxiliary Feedwater Pump Failed to Start During Surveillance Test On May 25, 2009, the Callaway plant turbine-driven auxiliary feedwater pump failed to start during a planned surveillance run. The licensee determined that the failure of the turbine-driven auxiliary feedwater pump was due to an inadequately lubricated trip throttle valve. The valve was inadequately lubricated because the licensee inappropriately closed the lubrication portion of Procedure MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection," during Refueling Outage 16. Subsequent review by the licensee determined that while the exact timing of the valve failure could not be determined, but it was reasonable to assume that the degraded condition existed for a period greater than the Technical Specification allowed completion time for the auxiliary feedwater system. Consequently, the event resulted in a reportable event per the requirements of 10 CFR 50.73(a)(2)(i)(B), any operation or condition which was prohibited by the plant's Technical Specifications. Additionally, since the motor-driven auxiliary feedwater pump train A was inoperable just prior to discovery of the degraded condition, the event was determined to be reportable per 10 CFR 50.73(a)(2)(v), as a condition that could have prevented fulfillment of a safety function and 10 CFR 50.73(a)(2)(ii)(B), as an analyzed condition that significantly degraded plant safety. The licensee submitted a licensee event report on July 21, 2009. The inspectors reviewed the licensee's submittal and determined that the report adequately documented the summary of the event including the potential safety consequences and corrective actions required to address the performance deficiency. The inspectors identified a self-revealing violation of Technical Specification 3.7.5, "Auxiliary Feedwater System." The enforcement aspects of the violation are discussed in Section 1R2 of this report as AV 05000483/2009009-01, "Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve." This LER is closed.
4OA6 Meetings, Including Exit
On June 26, 2009, the team presented the preliminary results of this inspection at the end of the onsite week to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of his staff who acknowledged the findings. The team verified that no proprietary information was retained.
On September 2, 2009, the team leader presented the final results of the inspection to Mr. A. Heflin, Vice President Nuclear and Chief Nuclear Officer, and other members of the licensee staff who acknowledged the findings. The team verified that no proprietary information was retained.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- F. Bagby Jr., Supervisor, Nuclear Maintenance, Mechanical
- L. Beaty, System Engineer
- F. Bianco, Shift Manager, Operations
- M. Covey, Shift Manager, Operations
- D. Davidson, Operations Technician, Operations
- K. Duncan, Shift Manager, Operations
- T. Elwood, Supervising Engineer, Regulatory Affairs/Licensing
- W. Gruer, Operating Supervisor, Operations
- M. Hall, Assistant Manager, Nuclear Engineering
- M. Hoehn II, Acting Supervising Engineer, NSSS Systems
- J. Imhoff, System Engineer
- R. Lane, General Supervisor, Mechanical Maintenance
- S. Maglio, Assistant Manager, Regulatory Affairs
- K. Mills, Manager, Plant Engineering
- S. Petzel, Engineer, Regulatory Affairs
- J. Pitts, Supervising Engineer, Performance
- S. Sandbothe, Manager, Regulatory Affairs
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Open
- 050000483/2009009-01 AV Turbine-Driven Auxiliary Feedwater Pump Inoperable Due to Inadequately Lubricated Trip Throttle Valve
(Section 1R2)
Opened and Closed
- 050000483/2009009-02 NCV Failure to Maintain an Adequate Lubrication Procedure for Valve FCHV0312 (Section 1R3)
- 050000483/2009009-03 NCV Failure to Adequately Evaluate the Use of Mobile 28 Grease for the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve (Section 1R3)
- 050000483/2009009-04 NCV Failure to Enter Conditions Adverse to Quality Associated with the Turbine-Driven Auxiliary Feedwater Pump Trip Throttle Valve into the Corrective Action Program (Section 1R3)
- 050000483/2009009-05 NCV Failure to Ensure Turbine-Driven Auxiliary Feedwater Pump is Operable Prior to Entry into Mode 3 (Section 4OA2)
Attachment 1
Closed
- 05000483/LER-2009-002-00 LER Turbine-Driven Auxiliary Feedwater Pump Failed to Start During Surveillance Test (Section 4OA3)
DOCUMENTS REVIEWED
CALLAWAY ACTION REQUESTS
- 199300209
- 200202491
- 200202509
- 200203228
- 200208110
- 200305288
- 200308175
- 200400798
- 200400884
- 200401780
- 200405270
- 200406231
- 200500187
- 200602955
- 200610423
- 200701152
- 200800798
- 200808775
- 200808777
- 200811884
- 200902515
- 200904077
- 200904216
- 200904840
- 200905004
- 200905032
- CALCULATIONS
- NUMBER TITLE REVISION
- FC-20 Manual Operation of FCHV0312 0
- ZZ-224 MOV Sizing Calculation 11
- DRAWINGS
- NUMBER TITLE REVISION E-23FC23(Q) Schematic Diagram Aux FWP Turbine Trip throttle Vlv 19 M-22AL01(Q) Piping and Instrumentation Diagram Auxiliary Feedwater System 33
- NP-1490 4" - 900# ANSI Trip Throttle Valve Top Mechanism with
- SMB 000 Limitorque Operator, Hard Packing, Double Leakoff, Strainer Mech. Trip, (3) Limit Switches, Solenoid A P-4772 3"
- ASA 900# Trip Throttle Valve Top Mechanism 3"-900# Inlet and 3"-900# Outlet with Hard Packing, Oil Trip, Double Leakoff, Hard Trip, Corner Packing A P-4975 Parts List Top Mechanism Trip Throttle Valve with Hard Packing 0 P-4979 Parts List Top Mechanism Trip Throttle Valve with Hard Packing Auxiliary Parts 0 MODIFICATION PACKAGES
- NUMBER TITLE REVISION
- MP 05-1003 Trip Throttle Valve Gear Ration Change 100:1 0
- Attachment 1 PROCEDURES
- NUMBER TITLE REVISION
- APA-ZZ-00100 Written Instructions Use and Adherence 21
- APA-ZZ-00320 Work Execution 35
- APA-ZZ-00330 Living Preventive Maintenance Program 1 Appendix A
- APA-ZZ-00330 Preventive Maintenance Program 31
- APA-ZZ-00395 Significant Operator Response Timing 11
- APA-ZZ-00500 Significant Adverse Condition - Significance Level 1 5 Addendum 128E
- APA-ZZ-00500 Corrective Action Program 44
- APA-ZZ-00500 Corrective Action Program 48
- APA-ZZ-00542 Event Review and Post Transient Evaluation 14
- APA-ZZ-01250 Operational Decision Making 5
- EC Supp Guide Emergency Coordinator Supplemental Guideline 7
- ECA-0.0 Loss of All AC Power 10
- EDP-ZZ-01126 Lubrication Predictive Maintenance Program 8
- ETP-FC-00001 Calibration of Terry Turbine Governor Valve Actuator 4
- MDP-ZZ-TR001 Planning and Execution of Troubleshooting Activities 4
- MPM-FC-QK001 Auxiliary Feedwater Pump Turbine Annual Inspection 13
- MPM-FC-QK001 Auxiliary Feedwater Pump Turbine Annual Inspection 14
- MPM-FC-QK002 Auxiliary Feedwater Pump Turbine Five-Year Internal Inspection 19
- MPM-ZZ-QA001 Limitorque Actuator Inspection and Lubrication 35
- MTE-ZZ-QA006 Motor Operated Valve Analyst Guide 4
- MTE-ZZ-QA009 MOVATS UDS Testing of FCHV0312 6
- ODP-ZZ-00025 EOP/OTO User's Guide 13
- ODP-ZZ-0016E Operations Technicians Watchstation Practices and Rounds 21
- OOA-SA-C066Z Engineered Safety Feature (ESF) Status Panel SA066Z Alarm
- 1
- Information
- OSP-AL-PV005 Turbine Driven Auxiliary Feedwater Pump and Check Valve 3
- Inservice Test- IPTE
- OSP-SA-0007A Train A AFAS Slave Relay Test 21
- OTA-RK-00026 Turbine Driven Auxiliary Feedwater Pump Start 1 Addendum 128B
- OTA-RK-00026 Turbine Driven Auxiliary Feedwater Pump Speed High or Low 0 Addendum 128E
- Attachment 1
- OTN-AL-00001 Turbine Driven Auxiliary Feedwater Pump Trip/Throttle Appendix A Valve Trip Check and Reset 1
- OTN-AL-00001 Turbine Driven Auxiliary Feedwater Pump Trip/Throttle Appendix A Valve Trip Check and Reset 2
- OTS-FC-00004 Tripping Sequence of Auxiliary Feedwater Pump Turbine 9 OPERATING EXPERIENCE INFORMATION
- South Texas Project Unit 2,
- LER 2007-001-00
- South Texas Project Unit 2,
- LER 2007-002-00 NRC
- IN 2006-29, Potential Common Cause Failure of Motor-Operated Valves as a Result of Stem Nut Wear.
- NRC
- GL 1989-10, Motor Operated Valve Testing and Surveillance NRC
- GL 1996-05, Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves NRC
- IN 1984-010, Motor-Operated Valve Torque Switch Set Below the Manufacturer's Recommended NRC
- IN 1989-11, Failure of DC Motor-Operated Valves to Develop Rated Torque Because of Improper Cable Sizing NRC
- IN 1993-74, High Temperatures Reduce Limitorque AC Motor Operator Torque NRC
- IN 1996-48, Motor Operated Valve Performance Issues
- NRC
- RIS 2001-015, Performance of
- DC-Powered Motor-Operated Valve Actuators NUREG/CR-6750, Performance of MOV Stem Lubricants at Elevated Temperatures, October
- 2001 NUREG/CR-6806, MOV Stem Lubricant Aging Research, March 2003 Part 21 Report 2001-35, Material Substitution of Grade 1212 Steel in place of grade 1018 carbon steel specified by designed
- JOBS
- W219154
- 05515178
- 06523136
- 07506359
- 08510069
- 09003598
- 09502426
- MISCELLANEOUS
- Request for Resolution 16006A, Change Grease on FCHV0312 Trip Throttle Valve, April 24, 1995 Letter from Jack Donohew, NRC to Garry Randolph, Union Electric Company, Subject: Callaway Plant, Unit 1 - Generic Letter 96-05, "Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves," Dated May 30, 2001 Letter from Mel Gray, NRC to Garry Randolph, Union Electric Company, Subject: Request for Additional Information - Generic Letter 96-05, "Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves," Callaway Plant, Unit 1, Dated June 18, 1999
- Attachment 1 Letter
- ULNRC-3548, Callaway Plant Docket Number 50-483, Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves, Dated March 13, 1997 Letter
- ULNRC-04075, Docket Number 50-483, Callaway Plant Union Electric Company NRC Generic Letter 96-05 Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves, Dated July 28, 1999 Letter
- ULNRC-04430, Docket Number 50-483, Callaway Plant Union Electric Company NRC Generic Letter 96-05 Periodic Verification of Design-Basis Capability of Safety-Related Motor-Operated Valves, Dated March 30, 2001
- Attachment 2
- June 17, 2009
- MEMORANDUM TO: Jeremy Groom, Resident Inspector, Team Leader Projects Branch B
- Division of Reactor Projects Michael Chambers, Resident Inspector
- Projects Branch C
- Division of Reactor Projects
- FROM:
- Dwight Chamberlain, Director /RA/ AVegel for
- Division of Reactor Projects
- SUBJECT:
- CHARTER FOR SPECIAL INSPECTION INVOLVING THE FAILURE OF THE TURBINE DRIVEN AUXILIARY FEEDWATER PUMP TO START AT
- THE CALLAWAY PLANT
- In response to the identification that the turbine driven auxiliary feedwater pump did not start and may not have been able to perform its function potentially due to the motor operated trip throttle valve stem being improperly lubricated between May 4 and May 25, 2009, a special inspection will be performed.
- You are hereby designated as the special inspection team leader.
- A. Basis On May 25, 2009, during a planned surveillance run of the turbine driven auxiliary feedwater pump, the pump did not pass its surveillance.
- The trip throttle valve for the turbine did not actuate properly, such that the turbine did not start and come up to speed.
- The torque switch on the valve tripped.
- Inspection of the shaft for the trip throttle valve showed that the lubricant used on the valve shaft was dried.
- The licensee took corrective action to fix (lubricate) the valve that day which took approximately 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> to accomplish.
- The turbine driver auxiliary feedwater pump was subsequently tested and restored to an operable status.
- A regional Senior Reactor Analyst (SRA) preliminarily estimated the Incremental Conditional Core Damage Probability for this issue to be 7.3 x 10
-6, which falls in the overlap region between baseline and special inspection.
- Although the risk for this event is in the overlap region, a special inspection will be performed since there may be generic issues with lubricating motor-operated valves. UNITED STATESNUCLEAR REGULATORY COMMISSIONREGION IV612 EAST LAMAR BLVD, SUITE 400ARLINGTON, TEXAS 76011-4125
- Attachment 2 B. Scope
- 1. Develop a complete sequence of events related to the discovery of the degraded condition. 2. Review operating experience involving prior opportunities to identify and evaluate action implemented at Callaway as a result of operating experience.
- Included in this should be a review of past notices by the valve shaft manufacturer of one lot of defective shafts in the 1990's that they recommended replacing.
- 3. Review the licensee's root cause analysis and determine if it was conducted to a level of detail commensurate with the significance of the problem.
- 4. Determine if the licensee's corrective actions have addressed the extent of condition and assess whether these actions are adequate to prevent recurrence.
- In particular determine if any other motor operated valves may have been improperly lubricated. 5. Examine the valve replacement program and lubricating schedule for how the trip throttle valve (as well as other valves) is tracked to ensure timely lubrication. 6. Collect facts to support an accurate portrayal of exposure time. 7. Collect facts to support proper crediting of the licensee's ability to recover the turbine driven auxiliary feedwater pump within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> as assumed in the risk assessment.
- Ensure to include time needed to procure the lubricant if there was a need to grease the valve during any events. 8. Review the licensee's procedures directing them to manually operate valve with particular attention to whether they were sufficient to direct operators to manual operations in a timely manner and whether they are accomplishable under event-like conditions (i.e., station blackout). 9. Determine whether the licensee's lubrication schedule and procedural scope for the trip throttle valve were within line of those recommended by the Terry Turbine User's Group and industry standards 10. Evaluate whether there was the potential for damaging other parts of trip throttle valve in the method the licensee chose to recover the valve on May 25, 2009.
- 11. Evaluate whether the licensee was thorough in their troubleshooting on May 25, 2009, for evaluating whether other parts of trip throttle valve could have been damaged and caused the failure (in particular the stress point on the screw spindle button). 12. Examine whether the licensee followed industry standards for trip throttle valve rebuilds which they perform. 13. Determine what type of lubricant the licensee was using (lithium based grease) and whether that lubricant is acceptable for the application.
- 14. Examine the licensee's monthly cycle of lubrication and licensee's explanation of how a regularly stroked valve could allow the lubricant to harden. 15. Evaluate the licensee's lubrication procedures for the possibility of mixing grease by not removing old, different types of grease if the licensee has switched lubricants in the past. 16. Determine if the licensee followed their Generic Letter 89-10 program for the trip throttle valve. 17. Verify the licensee met the proper reporting requirements of 10
- CFR 50.72 and
- 10
- CFR 50.73.
- Also determine if the licensee has plans to issue a Licensee Event Report to document this issue. 18. Review the licensee's compliance with the Technical Specifications.
- Attachment 2 C. Guidance Inspection Procedure 93812, "Special Inspection," will be used during this inspection.
- The inspection should emphasize fact-finding in its review of the circumstance surrounding this event.
- It is not the responsibility of the team to examine the regulatory process.
- Safety concerns identified that are not directly to the event should be reported to the Region IV office for appropriate action.
- The team will report to the site and begin inspection no later than June 22, 2009.
- While onsite, you will provide daily status briefings to Region IV management, who will coordinate with the Office of Nuclear Reactor Regulation, to ensure that all other parties are kept informed.
- Depending on the outcome of the inspection, inspection results will be documented in Special Inspection Report 05000482/2009009.
- This report will be issued within 45 days of the completion of the inspection.
- This guidance may be modified should you develop significant new information that warrants review.
- Should you require support for the final determination of the risk significance of any issue, contact David Loveless at (817) 860-8161.
- Should you have any questions concerning this guidance, contact me at (817) 860-8141.
- Attachment 3 Timeline associated with TDAFW Pump Trip throttle Valve FCHV0312
- Date ActivityJuly 8, 1986 Turbine-driven auxiliary feedwater pump trip throttle valve fuses are discovered blown due to lack or improper valve lubrication. May 1, 1987
- RFR 3715A approves 18-month frequency for Preventive Maintenance 0818214.
- May 8, 1987 Turbine-driven auxiliary feedwater pump trip throttle valve fuses are discovered blown due to hardened lubricant. November 10, 1989
- RFR 3715A approves 3-month frequency for Preventive Maintenance 0818214.
- March 2, 1993 The licensee initiated Suggestion Occurrence Solution
- 199300209 to document that valve FCHV0312 did not open during preventive maintenance.
- Action 3 to Suggestion Occurrence Solution
- 199300209 determined the cause to be hardened Nebula grease around the stem and stem nut. March 3, 1993 Request for Resolution 13409 requests change from Nebula
- EP-1 grease to Shell Narina
- EP-0 or Aeroshell Grease 5 due to hardening of grease at valve FCHV0312 due to continuous high temperatures.
- No change is implemented since the proposed replacements are not acceptable greases for addressing long term hardening due to high temperatures at FCHV0312. April 24, 1995 Request for Resolution 16006A approves the use of Mobile 28 grease in place of Exxon Nebula
- EP-1 for valve FCHV 0312. April 25, 1995 During Refueling Outage 7 the licensee installed Valve FCHV0312.
- Postmaintenance testing shows that valve has an open stroke time of 7.905
seconds with a minimum available thrust of 4044 lbf. April 30, 1996 Job W177231 is performed to install valve FCHV0312. May 1, 1996 Postmaintenance testing for valve FCHV0312 shows that valve has an open stroke time of 13.9 seconds with a minimum available thrust of 2795 lbf. April 30, 1996 Job W613037 is performed to install Valve FCHV0312.
- Postmaintenance testing for valve FCHV0312 shows that valve has an open stroke time of
- 23.143 seconds with a minimum available thrust of 4676 lbf. August 8, 1998 Job W194653 performed to rebuild valve FCHV0312. February 24, 1999 Job P629996 is performed to lubricate valve FCHV0312 with Mobile 28 grease. May 18, 1999 Job P633305 is performed to lubricate valve FCHV0312 with Mobile 28 grease.
- August 9, 1999 Job P637319 is performed to lubricate valve FCHV0312 with Mobile 28 grease.
- August 10, 1999 Preventive Maintenance
- 0818214 changes lubrication interval for valve FCHV0312 from every 12 weeks to every 24 weeks. October 6, 1999 Job P620371 performs Preventive Maintenance
- 0810863 (Turbine Skid Inspection). January 25, 2000 Job P642550 is performed to lubricate FCHV0312 with Mobile 28 grease.
- July 10, 2000 Job P651618 is performed to lubricate FCHV0312 with Mobile 28 grease.
- December 26, 2000 Job P658531 is performed to lubricate FCHV0312 with Mobile 28 grease.
- Attachment 3 May 5, 2001 Job P666833 is performed to lubricate FCHV0312 with Mobile 28 grease May 31, 2001 Job P646701 performs Preventive Maintenance
- 0810863 (Turbine Skid Inspection).
- The job does not specifically discuss trip throttle valve lubrication.
- The job also specifies Nebula
- EP-1 grease instead of Mobile 28. November 28, 2001 Job P675506 is performed to lubricate valve FCHV0312 with Mobile 28 grease.
- May 13, 2002 Job P685041 is performed to lubricate valve FCHV0312 with Mobile 28 grease.
- May 17, 2002 The licensee initiated Callaway Action Request
- 200203228 to request a change to the preventive maintenance frequency for the auxiliary feedwater pump turbine trip throttle valve FCHV0312 such that is be worked during refuel only (every 18 months).
- The lead response to the Callaway action request determined that the preventive maintenance bases does not support a change to once a cycle and the auxiliary feedwater pump turbine trip throttle valve preventive maintenance should be performed yearly. October 7, 2002 Job P692333 is performed to lubricate FCHV0312 with Mobile 28 grease.
- November 11, 2002 Job P676907 performs Preventative Maintenance
- 0810863 (Turbine Skid Inspection).
- The job does not specifically discuss trip throttle valve lubrication.
- The job also specifies Nebula
- EP-1 grease instead of Mobile 28. April 14, 2003 Job Preventive Maintenance
- 698598 is performed to lubricate valve FCHV0312 with Mobile 28 grease. April 29, 2003 Lead response to Callaway Action Request
- 200203228 documented a concern with the trip throttle valve stem lubrication in that the coupling is not fully accessible in either the full open or full close position.
- The Callaway action request recommends that the trip throttle valve be manually mid-positioned to fully lubricate the valve step and bushings. July 17, 2003
- The licensee initiated Callaway Action Request
- 200305288 to evaluate the use of Chevron Ulti-Plex Synthetic grease in place of Mobile 28 grease for valve FCHV0312.
- The evaluation of the Callaway action request determined that Mobile 28 grease was an acceptable grease and approved by the valve vendor.
- No change was instituted. January 13, 2004 Preventive Maintenance
- 0818214 is changed from an every 24-week to every 52-week frequency. January 13, 2004 Job P708083 is performed to lubricate FCHV0312.
- The job does not document the type of lubricant used. April 9, 2004 Valve FCHV0312 is rebuilt and the valve spindle is replaced.
- During the rebuild, the valve is lubricated. April 2004 Modification Package 97-1028 is implemented to change the gear ratio in FCHV0312 from 68:1 to 75:1.
- The modification is performed to slow down the stroke of FCHV0312 to prevent overspeed of the turbine driven auxiliary feedwater pump. April 27, 2004 During Refuel 13, Job P628784 is performed to install valve FCHV0312.
- May 1, 2004 Postmaintenance testing shows that valve has an open stroke time of 25.73 seconds with a minimum available thrust of 3358 lbf. May 17, 2005 Preventive Maintenance
- 0818214 frequency is changed from every 52 weeks to every 72 weeks. May 17, 2005 Job P718624 is performed to lubricate valve FCHV0312 with Mobile 28.
- Attachment 3 May 21, 2005 Job P701943 performs Preventive Maintenance
- 0810863 (Turbine Skid Inspection).
- The recently revised procedure does not specifically call for lubrication of the valve trunion screws. November 2, 2005 Job P726097 performs Preventive Maintenance
- 0810863 (Turbine Skid Inspection). January 13, 2006 Valve FCHV0312 is removed from the scope of Callaway's in-service testing program and is considered skid-mounted since it is tested during
Procedure
- OSP-AL-P00002 September 26, 2006 Job
- 05509319 is performed to lubricate FCHV0312.
- The job did not document the type of lubricant used. April 2007 Modification Package 05-1103 is implemented to change the gear ratio in FCHV0312 from 75:1 to 100:1.
- The modification is performed to slow down the stroke of FCHV0312 to prevent overspeed of the turbine driven auxiliary feedwater pump. April 28, 2007 During Refuel 15 Valve FCHV0312 is tested for Valve FCHV0312 which shows the valve has an open stroke time of 18.435 seconds with a minimum available thrust of 2870 lbf. May 6, 2007 Job
- 05517497 performs Preventive Maintenance
- 0810863 (Turbine Skid Inspection).
- The preventive maintenance includes a lubrication of valve FCHV0312. September 13, 2007 Job W219154 is performed to rebuild spare Valve FCHV0312 valve (removed in Refuel 13). October 28, 2008 Job
- 05515178 installs Valve FCHV0312 rebuilt in September 2007 under Job W219154. October 28, 2008 Postmaintenance testing shows that valve has an open stroke time of 17.959 seconds with a minimum available thrust of 1837 lbf. November 5, 2008 Job
- 07506359 is performed; however, the valve is not lubricated per the procedure because section Section 6.3, "Trip Throttle Valve Lubrication and Inspection" is closed. May 25, 2009 11:23 a.m.
- During performance of
- OSP-SA-0007A, the turbine-driven auxiliary feedwater pump fails to start upon demand.
- Troubleshooting revealed that the turbine trip throttle valve failed to latch and the torque switch for the motor-operated valve tripped.
- The licensee enters Technical Specification Limiting Condition for Operations 3.7.5.c.
- Valve FCHV0312 is discovered to have traveled towards the latch position but stopped prior to latching. May 25, 2009 3:30 p.m. Licensee attempts to reperform the slave relay test.
- The turbine-driven auxiliary feedwater pump still did not start. May 25, 2009 4:14 p.m. Job 09003598/500 is taken to in-process.
- This job was written to investigate the valve.
- During this job, the licensee discovered that the "close torque switch" was open.
- Nothing was done to the valve during this job. May 25, 2009 around 4:30 p.m. Conversation takes place in the control room about what could have caused FCHV0312 torque switch to open.
- Licensee decides to cycle the valve actuator manually to check for mechanical binding. May 25, 2009
- 5:07 p.m. Job 09003598/910 is taken to in-process.
- This job is used to manually cycle the FCHV0312 actuator.
- Attachment 3 May 25, 2009
- 5:36 p.m. The operator for FCHV0312 is cycled from the position where is stopped (just prior to latching), to the latched position, then to the full open actuator position.
- The valve is operated by the field supervisor who noted an audible squeak. Based on these indications, the licensee determined that the valve was not adequately lubricated.
- May 25, 2009
- 7:37 p.m. Job 09003598/510 task is taken to in-process.
- This task lubricates valve FCHV0312.
- During lubrication, the mechanic who operated the valve noted that the valve was hard to operate and made a squeak.
- The mechanic noted that he could not operate the valve while standing on the floor, he had to get above the valve.
- As the valve was stroked manually, grease was applied to the valve stem. May 25, 2009
- 9:36 p.m. Turbine-driven auxiliary feedwater pump started for postmaintenance testing. May 25, 2009
- 10:47 p.m. Turbine-driven auxiliary feedwater pump started for slave relay test. May 25, 2009
- 10:56 p.m. Turbine-driven auxiliary feedwater pump declared operable. June 17, 2009 NRC Region IV approves a special inspection charter to understand the circumstances surrounding the failure of the turbine-driven auxiliary feedwater pump. June 22-26, 2009 NRC Special Inspection Team at Callaway Plant.
- Attachment 4 PHASE 3 ANALYSIS FAILURE OF AUXILIARY FEEDWATER PUMP TRIP AND THROTTLE VALVE
- Summary of Significance Determination The senior reactor analyst completed a Phase 3 analysis using the plant-specific Standardized Plant Analysis Risk (SPAR) Model for Callaway, Revision 3.50, including the external events initiators.
- The exposure period of 11 days represented half the time from the last successful run of the pump plus the repair time.
- The analyst estimated the nonrecovery for the failed valve at 21 percent.
- The final result was calculated to be 1.8 x 10-6 indicating that the finding was of low to moderate risk significance (White).
- Details
- A. Summary of Issue On May 25, 2009, Callaway Plant operators performed surveillance testing Procedure
- OSP-SA-0007A, "Train A AFAS Slave Relay Test," to test the slave relays used to start the turbine-driven auxiliary feedwater pump.
- During the performance of Step 6.2.8 of Procedure
- OSP-SA-0007A, the turbine-driven auxiliary feedwater pump did not start as expected.
- During troubleshooting, the licensee discovered that the turbine-driven auxiliary feedwater pump trip throttle valve FCHV0312 had traveled in the closed direction in an attempt to latch the valve operator but had stopped prior to completing its stroke.
- Electrical continuity checks revealed that the motor-operated valve closed torque switch had opened.
- Callaway operators declutched and manually operated the trip throttle valve actuator.
- An audible squeak was observed during manual operation indicating mechanical binding within the sliding nut and valve spindle.
- Following troubleshooting, the licensee performed Job
- 09003598 to lubricate the trip throttle valve.
- Once the valve was adequately lubricated, the licensee successfully performed postmaintenance testing.
- Callaway replaces the trip throttle valve every third refueling outage with a refurbished valve that was previously removed from the system.
- The turbine-driven auxiliary feedwater pump trip throttle valve that failed to open on May 25, 2009, was replaced during Refueling Outage 16 in October 2008.
- The valve was replaced with a similar valve that was removed from service during Refueling Outage 13 in April 2004 and refurbished in September 2007.
- The replacement procedure did not have specific lubrication instructions.
- Lubrication of the valve was scheduled to be performed a few days after replacement as Job 07506359.
- The maintenance supervisor assigned to coordinate the lubrication incorrectly assumed that the valve was adequately lubricated since it had been recently replaced.
- Callaway Procedure
- APA-ZZ-00320, "Work Execution," Section 4.12, allows for a job to be canceled if it is determined that the work is not necessary or has been completed by another job.
- The maintenance supervisor initiated a job routing request to engineering inquiring if additional inspection was required on valve FCHV0312 since the valve had recently been replaced.
- Although not specifically referenced, the portion of the job requested to be closed was Section 6.3, "Trip Throttle Valve Lubrication and Inspection."
- Callaway engineering replied to the job routing request that the valve had been diagnostically tested satisfactorily and that no additional inspection was needed.
- The engineer's response only answered the specific question asked by the job routing Attachment 4 request and did not examine the procedural requirements that were the subject of the request.
- The engineer did not consult with the technician requesting closure and did not verify that the lubrication job task being closed was completed by the installation procedure.
- Based on the input received from engineering, the mechanical maintenance supervisor closed the lubrication portion of Procedure
- MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection."
- The mechanical maintenance supervisor failed to identify that the entire work scope of the superseded work document was included in the superseding work document which did not meet the requirements of licensee Procedure
- APA-ZZ-00320.
- Since the valve was not lubricated during Refueling Outage 16, the last lubrication of the valve occurred during the refurbishment of the valve in September 2007.
- The lack of lubrication on valve FCHV0312 resulted in increased friction between the sliding nut and spindle which caused the valve not to open upon demand on May 25, 2009.
- The licensee initiated Significant Condition Adverse to Quality Callaway Action Request
- 200904216 to investigate the failure of the turbine-driven auxiliary feedwater pump to start due to an inadequately lubricated trip throttle valve.
- The licensee's root cause analysis determined that the failure to lubricate valve FCHV0312 was due to the failure to fully review the closure of the lubrication portion of Procedure
- MPM-FC-QK001 during Refueling Outage 16.
- The licensee also determined that while the actual timing of the failure could not be determined, it was reasonable to assume that the turbine-driven auxiliary feedwater pump was inoperable for a time frame greater than the technical specification allowed completion time.
- B. Statement of the Performance Deficiency The performance deficiency associated with this finding involved the licensee's failure to ensure valve FCHV0312 was adequately lubricated.
- Specifically, the licensee closed without adequate review, Section 6.3 "Trip Throttle Valve Lubrication and Inspection," of Procedure
- MPM-FC-QK001, "Auxiliary Feedwater Pump Turbine Annual Inspection." The lack of lubrication resulted in increased friction within the valve operating stem which caused the valve not to open on May 25, 2009.
- C. Significance Determination Basis
- 1. Phase 1 Screening Logic, Results and Assumptions In accordance with NRC Inspection Manual Chapter 0612, Appendix B, "Issue Screening," the analyst determined that the failure to lubricate valve FCHV0312 at appropriate intervals was a licensee performance deficiency.
- The issue was more than minor because it was associated with the mitigating systems cornerstone attribute of equipment performance and affected the cornerstone objective of ensuring the availability and reliability of systems that respond to initiating events to prevent undesirable consequences.
- The analyst evaluated the issue using the Significance Determination Process (SDP) Phase 1 Screening Worksheet for the Initiating Events, Mitigating Systems, and Barriers Cornerstones provided in Manual Chapter 0609,
- 4, "Phase 1 - Initial Screening and Characterization of Findings."
- This finding affected the Mitigating Systems Cornerstone.
- The analyst determined that the finding represented a loss of system safety function because Attachment 4 the turbine-driven auxiliary feedwater pump PAL02 failing eliminates the capability of the plant to cope with a station blackout.
- Therefore, a Phase 2 estimation was conducted in accordance with Manual Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection Findings for At-Power Situations."
- 2. Phase 2 Risk Estimation In accordance with Manual Chapter 0609, Appendix A, Attachment 1, "User Guidance for Phase 2 and Phase 3 Reactor Inspection Findings for At-Power Situations," the Senior Reactor Analyst evaluated the subject finding using the "Risk-Informed Inspection Notebook for Callaway Nuclear Generating Station, Unit 1," Revision 2.1a.
- The following assumptions were made:
a. The identified performance deficiency occurred in October 2008 when the licensee failed to perform appropriate preventive maintenance on valve FCHV0312.
- However, the valve continued to be stroked about twice monthly and pass surveillance through May 4, 2009.
- Therefore, the deficiency only began to affect plant risk at some point between May 4 and the time of failure on May 25, 2009.
b. The failure was identified during a test of the turbine-driven auxiliary feedwater pump conducted on May 25, 2009.
- The last successful test that opened valve FCHV0312 was completed on May 4, 2009.
- Valve FCHV0312 was repaired and returned to service approximately hours later on May 25, 2009.
c. In accordance with Manual Chapter 0609, Appendix A, Attachment 2, "Site Specific Risk-Informed Inspection Notebook Usage Rules," Rule 1.1, "Exposure Time," the analyst evaluated the time frame over which the finding impacted the risk of plant operations.
- The analyst determined that the performance deficiency affected plant risk for 11 days.
- Therefore, the exposure time used to represent the time that the performance deficiency affected plant risk in the Phase 2 estimation was greater than 3 but less than 30 days.
d. In accordance with Appendix A, Attachment 1, Step 2.1.3, "Find the Appropriate Target for the Inspection Finding in the Pre-solved Table," the analyst determined that the appropriate target for evaluating this performance deficiency was "TDAFP Fails to Start."
- Therefore, the analyst utilized the pre-solved table associated with the SDP notebook to perform the estimation.
e. The analyst gave no operator action credit as discussed in Manual Chapter 0609, Appendix A, Attachment 1, Table 4, "Remaining Mitigation Capability Credit."
- The requirements to have procedures in place and to have trained the operators in recovery under similar conditions for such credit were not met.
- The dominant sequences from the notebook were documented in Table 1.
- Attachment 4 TABLE 1 Failure of Turbine-Driven Auxiliary Feedwater Pump Phase 2 Sequences
- Initiating Event SequenceMitigating Functions ResultsLoss of Offsite Power
- LOOP-AFW-FB 7 6
- LOOP-EAC-TDAFW-FB 7 8
- LOOP-EAC-TDAFW-REC1 6 Loss of Service Water 7
- LSWS-ESW-TDAFW 5 Loss of dc Bus 1
- LBDC-TDAFW-MDAFW-HPR 7 2
- LBDC-TDAFW-MDAFW-FB 7 3
- LBDC-TDAFW-MDAFW-EIHP 7 4
- LBDC-TDAFW-TESW 5
- Using the pre-solved worksheet, the result from this estimation indicated that the finding was of moderate safety significance (YELLOW).
- However, the analyst determined that this estimate did not include a full coverage of the risk related to the failure identified, particularly because of the shorter exposure time, the affects of potential recovery, and the affect that certain fire initiators would have on the specific condition.
- Therefore, a Phase 3 evaluation was conducted to better assess the risk of the finding related to internal initiators and fully assess the risk related to external initiators.
- 3. Phase 3 Analysis Assumptions The following assumptions were made to support this Phase 3 analysis:
- 1. The Callaway plant-specific SPAR, Revision 3.50, including the external event initiators, was the best tool for quantifying the risk of the subject performance deficiency.
- 2. Trip and throttle valve FCHV0312 failed to latch upon demand on May 25, 2009, preventing the start of turbine-driven auxiliary feedwater pump PAL02.
- 3. The best-available information indicated that the failure mode of valve FCHV0312 was a gradual degradation of the Mobil 28 grease stem lubricant in the continuously hot and humid environment.
- As documented in NUREG/CR-6750, "Performance of MOV Stem Lubricants at Elevated Temperature," Mobil 28 grease sheds oil at temperatures above 250F.
- 4. Given Assumption 3, the best assumption is that the Mobil 28 grease dried to a point that it was no longer lubricating the operating stem of Attachment 4 valve FCHV0312 at some point following the successful pump start on May 4, 2009.
- Temperatures during the test, when operating stem temperatures would have been near 300 F, would have accelerated the failure, but not to a great extent given the short period of time that the pump was run.
- 5. Given Assumption 4 and in accordance with Manual Chapter 0609, Appendix A, Attachment 1, Usage Rule 1.1, "Exposure Time," the analyst determined that the exposure time should be estimated by using one half the time from the previous successful test to the failure plus the time to repair the valve and return it to service because the time of inception of imminent failure is unknown.
- 6. Although operation may have been difficult, manual operation of valve FCHV0312 was not inhibited at the time of failure.
- Therefore, manual operation was a viable recovery action.
- 7. The recovery discussed in Assumption 6 was possible for most sequences, provided operators diagnose the condition correctly and determine that manual operation is a viable option and if they do not proceed to other repair options.
- 8. The recovery discussed in Assumption 6 was not possible for sequences involving anticipated transients without scram or main control room abandonment.
- However, for ease of evaluation, recovery was applied to these sequences in the best estimate quantification.
- Adjustments were made as a sensitivity as documented in Assumption 10.
- 9. The best available method to quantify the probability that operators would fail to manually realign valve FCHV0312 was the
- SPAR-H method.
- However, after evaluating several methods of quantifying the probability of nonrecovery, the analysts agreed that there was not currently an appropriate risk tool for evaluating the specific recovery related to the subject performance deficiency.
- 10. The majority of the risk associated with the subject performance deficiency was from station blackout or other sequences that resulted in a complete loss of all feedwater.
- Therefore, the time to recover the turbine-
driven auxiliary feedwater pump was limited to about 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.
- The analyst performed a sensitivity evaluation that indicated adjustments to the recovery time for specific sequences did not significantly affect the result.
- The results of the sensitivity were documented in Table A3 of the Appendix.
- Exposure Period As documented in the main control room log, pump PAL02 was last successfully started for a postmaintenance test on May 4, 2009, at 10:20 a.m.
- The pump failed to start during testing on May 25, 2009, at 11:41 a.m.
- Trip and throttle valve FCHV0312 was repaired and the pump returned to service on May 25, 2009 at 10:56 p.m.
- Attachment 4
- The team evaluated the failure mode of the valve and determined that it was unknown exactly when the valve was no longer in a condition to perform its risk significant function.
- In accordance with the Risk Assessment of Operational Events Handbook, Section 2.3, the exposure time for a component that fails from a degradation mechanism that gradually affects the component should be one-
half the time from the previous successful test plus the repair time.
- One-half the time from the previous test to the failure was calculated to be 10.53 days.
- The repair time was 0.47 days.
- The total exposure time was then calculated to be the sum of these two or 11 days.
- Application of Recovery The analyst evaluated the probability that operators fail to manually open Trip and Throttle Valve FCHV0312 and start Turbine-Driven Auxiliary Feedwater Pump PAL02 using the
- SPAR-H method described in NUREG/CR-6883, "The
- SPAR-H Human Reliability Analysis Method."
- The analyst determined that evaluating the condition of the valve and concluding it was capable of being manipulated was part of the diagnosis because the valve was not in a nominal condition.
- The following performance shaping factors were adjusted from nominal: * Time:
- SPAR rules would indicate that a station blackout with loss of all feedwater would result in core damage in 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.
- The licensee calculated 46 minutes to steam generator dryout for the limiting sequence by calculating the exact heat loading for Cycle 17.
- The licensee's timeline indicated that the valve could have been opened in 21 minutes from the initiation of a station blackout.
- Times provided by the inspection team, shown in Table A4 of the Appendix, indicated approximately 38 minutes for this evolution.
- The analyst noted that some of the licensee's assumptions were optimistic.
- The analyst determined that the operators would have one to two times greater than the nominal time required for diagnosis and that there would be greater than 30 minutes total.
- Therefore, diagnosis credit was adjusted for "Extra Time."
- However, following proper diagnosis, there would not be more than five times nominal for manipulation of the valve.
- Therefore, the analyst assumed "Nominal time" for completing the action.
- Stress: A station blackout scenario with a loss of all feedwater places the operators in at least a high level of stress during both diagnosis and action.
- Multiple competing priorities, sudden onset of stress, and the knowledge that the consequences of this task represents a threat to plant safety clearly places the operators under a high level of stress.
- Because the stress would not persist for long periods of time nor place the Attachment 4 operators under a threat to their physical well being, the analyst determined that the stress would not be at the extreme level.
- Complexity
- The analyst determined that the diagnosis and recovery of this specific failure were moderately complex.
- While it would be clear that the valve was not open, the cause of the failure would not be clear.
- The valve is unusual, in that it must be manipulated in the closed direction to latch prior to opening.
- The opening of the valve with dried out lubricant would be physically straining and difficult.
- A higher level of complexity might have been considered, however, this complexity was affected mostly by the environment and misleading nature of the situation.
- Therefore, the analyst dealt with these issues under the ergonomics shaping factor.
- Procedures
- The analyst determined that the emergency operating procedures would properly focus on the turbine-driven auxiliary feedwater pump and the valve alignments.
- Under normal conditions, these procedures, combined with operator skills and knowledge, would be sufficient to recover the pump.
- The complications in performing the subject recovery would not be expected to be addressed in procedures.
- Therefore, the analyst determined that the procedures were of nominal level for both diagnosis and action.
- Ergonomics
- The analyst determined that the biggest impediment to successful recovery was the ergonomics of this situation.
- Valve FCHV0312 would not be in the expected condition when an operator arrives in the pump room.
- The analyst noted that the same performance deficiency that would have caused the valve to fail to open automatically would also impair its recovery.
- This condition is not fully assessed by the
- SPAR-H tool because it represents an equipment deficiency that would tend to mislead the operators in recovering the pump.
- The pump room has no emergency lighting, is hot and humid.
- Accessing the valve requires crawling or reaching overhead.
- Manipulation of the handwheel involves poor footing and potential interaction with the mechanical overspeed trip element.
- These items alone were sufficient to designate a "Poor" ergonomics rating to the action for this evaluation.
- On the day of the failure, operators originally informed the senior resident inspector that the thermal overloads had tripped and that they were attempting to reset the motor starter.
- The indication was clear that they did not know what the problem with the valve was.
- This indicated an additional negative impact to proper recovery.
- The
- SPAR-H method provides that a rating of "Missing/Misleading" can be applied when the required indication fails to support diagnosis or post-
- Attachment 4 diagnosis behavior.
- Considering the condition of the valve, the analyst determined that operators attempting to manipulate the valve may determine that it was mechanically bound and abandon efforts to open the valve.
- The combined impact of the room conditions, the location of the valve, the potential to negatively impact the pump condition and the misleading indications led the analyst to apply the "Missing/Misleading" attribute to the ergonomics performance shaping factor for diagnosis.
- Table 2 provides the calculations used to apply the performance shaping factors and the odds ratio.
- The resulting HRA non-recovery value was 21 percent.
- TABLE 2Manually Open Trip and Throttle Valve 312 Performance Shaping Factor Diagnosis Action
- PSF Level Multiplier
- PSF Level Multiplier
- Time: Extra Time 0.10Nominal 1.0Stress: High 2.0High 2.0Complexity: Moderately Complex 2.0Moderate 2.0Experience: Nominal 1.0Nominal 1.0Procedures: Nominal 1.0Nominal 1.0Ergonomics: Misleading 50.0Poor 10.0Fitness for Duty: Nominal 1.0Nominal 1.0Work Processes: Nominal 1.0Nominal 1.0
- Nominal 1.0E-02 1.0E-03 Adjusted 2.0E-01 0.0 Odds Ratio 1.7E-01 3.8E-02
- Composite 20 40
- Failure to Manually Open Valve 312 and Start TDAFW Probability:
- 2.1E-01 The licensee used the EPRI HRA calculator with the HCR/ORE (cognitive) and THERP (execution) modules to evaluate the failure to recover the pump.
- Their nonrecovery probability was 1.6E-2.
- For the reader's edification, the nominal failure rate for an action requiring diagnosis is considered to be 1.1E-2 per demand.
- Therefore, the licensee's value is essentially equating recovery to nominal conditions.
- The analyst determined, as documented above, that the operating environment would have been far from nominal conditions.
- Attachment 4 Alternative Recovery Evaluation The analyst determined that the
- SPAR-H method may not be the best tool for evaluating the nonrecovery from the failure of valve FCHV0312 because it does not readily assess the impact of the performance deficiency on the recovery actions.
- Therefore, the analyst developed the following event tree to better describe the recovery actions:
- ACTIONOperator Fails toOpen ValveFORCEDControl Room Failsto Direct ForcedOpeningVALVE-FAILInitial Operator Failsto OpenDISPATCHControl Room Failsto Diagnose andDispatch OperatorIE-TDAFW#
- END-STATE-NAMES1
- OK2
- OK3
- NR4
- NR5
- NR
- TDAFW-RECOVERY -
- Valve
- FCHV-0312 Fails to Open on Demand2009/08/13Page 1
- The initiator for this event tree is the demand for a start of the turbine-driven auxiliary feedwater pump with the subsequent failure of Valve FCHV0312 to latch.
- Given this initiator, the top events are described as follows:
- Dispatch:
- This top represents the initial diagnosis and actions by the main control room operators in: following their emergency operating procedures; deciding local evaluation of the valve condition is required; and sending an operator out to the pump room.
- Successful dispatch leads to the next split.
- Failure means that there is no recovery of the pump.
- The analyst calculated the nonsuccess probability (shown in Table 3 below) using the
- SPAR-H method.
- The analyst noted that the final nonrecovery probability is completely insensitive to the selection of the value for the dispatch top, within a reasonable range, Attachment 4 because the failure probability is so low compared with estimated values for the other tops.
- Valve Fails:
- This top represents the likelihood that the operator dispatched to the pump room fails to initially open the valve.
- The failure modes the analyst is attempting to model with this top include: the operator walking up to the valve and being unable to turn the handwheel from the floor (as a mechanic did on the day of failure); or the operator climbing above the handwheel, determining that the valve is bound tightly enough that he is uncomfortable manipulating the valve on his own accord, and calling the main control room for more guidance.
- If he manipulates the valve (success), obviously, the condition is recovered.
- If he stops and calls the control room, the model then questions the response of the licensed operators.
- This is the most sensitive element of the tree.
- The analyst believed that the conditions would cause this to be a relatively high value because operators are trained at Callaway to stop and contact their immediate supervisor if conditions in the plant are different than expected.
- An initial estimate of 0.8 was used, indicating that the operator would not manipulate the valve without contacting the main control room for additional guidance.
- Forced:
- This top models only the response from the main control room operators upon notification of difficulty in operating valve FCHV0312.
- The analyst attempted to model the answer to, "Will the licensed operators tell the operator to force the valve open at all costs and risk failing the valve, or will they decide to bring in maintenance to work on it?"
- Additionally, this top attempts to indicate the difficulties in having multiple priorities going on at the time.
- Licensed operators would also be attempting to recover both failed diesel generators as well as offsite power.
- Success would send the operator back to the room to take action.
- The failure would lead to nonrecovery as there is not enough time in most sequences to recover via maintenance.
- This top is also very sensitive.
- The analyst used 0.2 as the initial and best estimate.
- Eighty percent of the time the control room will direct the operator to start the pump at all costs.
- There is a percent chance that a more conservative action would be taken by the licensed operators.
- Action:
- This top represents the probability that the operator would be successful in manipulation of the valve once directed to do so.
- The analyst used the
- SPAR-H to determine the best estimate split fraction for this top, as documented in the table below.
- Obviously, success means recovery and failure means nonrecovery.
- The analysis was not very sensitive to changes in this parameter either, including the value that the licensee calculated.
- In fact, setting both Action and Dispatch to the house event "FALSE" (indicating both tops are always successful), only changed the result by 11 percent.
- As shown in the event tree, Sequences 1 and 2 represented successful recovery while Sequences 3, 4 and 5 represented nonrecovery.
- The licensee argued that Sequence 4 should have provided an additional split to show a second try at opening the valve.
- However, in the analyst's judgment, the additional time necessary for such an attempt was not available for the short sequences.
- Attachment 4 The best estimate split fractions for the event tree are shown in Table 3.
- TABLE 3Nonrecovery Model Split Fractions Model Elements Best Event Tree Basic Event Estimate Dispatch
- CR-Action 1.00E-04Valve-Fails Stuck 0.8Forced Force 0.2Action FCHV0312 3.80E-02Nonrecovery Probability: 1.81E-01
- The analyst performed multiple sensitivity calculations using this model.
- Unfortunately, this model shows that the most sensitive part of the significance evaluation is in the unknowns regarding the condition of the valve and how operators will respond to it.
- The analyst conducted single parameter sensitivities as shown in Table A1 in the Appendix to this worksheet.
- Given that only two tops significantly affected the results of the analysis, the analyst conducted multiple parameter sensitivities on only those two, as documented in Table A2.
- The resulting nonrecovery probability for each was highlighted if the associated probability would result in a very low safety significance.
- All other probabilities were associated with low to moderate significance results.
- The analyst determined that the Green/White threshold was crossed at a nonrecovery probability of just under 9 percent.
- Therefore, a nonrecovery probability of 9 percent or greater would represent a finding of low to moderate safety significance.
- Change in Risk The analyst calculated the change in risk related to this performance deficiency using the following three different methods.
- 1. In Method 1, the analyst evaluated the risk utilizing the Callaway SPAR, Revision 3.50, including the external events model.
- The analyst set Basic Event
- AFW-AOV-CC-FTSST, "Steam Supply Valves Fail to Open," to the house event "TRUE," indicating that trip and throttle valve FCHV0312 failed to open.
- Basic Event
- AFW-XHE-XL-FTSST, "Operator Fails to Recover Failure of Steam Supply," was assigned a non-recovery probability of 21 percent.
- The analyst quantified the model and the results are provided in Table 4 below.
- The analyst considered this method to be the best estimate of risk.
- 2. For Method 2, the analyst evaluated the internal risk utilizing the Callaway SPAR, Revision 3.45.
- This was the model available for Callaway at the time of the event.
- The analyst set Basic Event
- AFW-TDP-FS-PAL02, "Turbine Driven Feed Pump PAL02 Fails to Start," to the house event "TRUE," indicating that the pump did not start on demand.
- The analyst Attachment 4 quantified the model and the results are provided in the Table 4 below.
- The analyst used the licensee's surrogate method for quantifying external events to adjust the internal events value.
- 3. For Method 3, the analyst evaluated the internal risk using the baseline and raw values from the licensee's PRA model.
- The results are provided in Table 4 below.
- TABLE 4Phase 3 Results
- SPAR-EE Baseline 5.65E-5 4.22E-5 2.80E-3 Case 1.95E-4 2.06E-4 2.86E-3 Delta 1.38E-4 1.64E-4 6.10E-5
- 11-Day Exposure 4.17E-6 4.93E-6 1.84E-6
- Including External 9.24E-6 1.09E-5
- 21% Non-recovery 1.93E-6 2.27E-6 1.84E-6** 1.6% Non-recovery 1.76E-7 1.92E-7 1.40E-7
- SPAR-EE run included NRCs nonrecovery of 0.21
- SDP Result.
- Analyst's best estimate.
- Table 5 documents the major sequences contributing 97 percent of the change in core damage frequency.
- TABLE 5Dominant Core Damage Sequences Sequence Description CDF % of Total LOOP 16-45 Station Blackout, loss of TDAFW, and failure to recover ac power within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. 1.95E-5 32.3
- FRI-A1A 2-11 Fire in the Auxiliary Building, Loss of all Feedwater, Failure of High Pressure Recirculation. 1.75E-5 29
- FRI-C27 1-16-45 Fire in the Main Control Room leading to Station Blackout, loss of TDAFW and failure to recover ac power within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. 1.04E-5 17.2
- FRI-C27 2-4 Fire in the Main Control Room leading to Abandonment with loss of
- TDAFW. 2.90E-6 4.8
- FLI-FZ2 2-13 Flooding leading to Loss of Service Water System, Loss of Auxiliary Feedwater, Failure to Recover Component Cooling Water
- 2.03E-6 3.4 LOCDNK4 8 Loss of dc Bus NK04, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.98E-6 3.3 LOCDNK1 8 Loss of dc Bus NK01, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.98E-6 3.3
- Attachment 4 LOOP 15 Loss of Offsite Power, Loss of Auxiliary Feedwater, Failure of Feed and Bleed. 1.09E-6 1.8
- FRI-A1A 2-12 Fire in the Auxiliary Building, Loss of all Feedwater, Failure of Feed and Bleed. 6.05E-7 1.0 LOOP 16-43-10 Station Blackout, loss of TDAFW, and Reactor Coolant Pump Seal LOCA. 6.04E-7 1.0
- The analyst noted that all three methods provided very similar results.
- This indicated that the primary difference between the NRC and the licensee results was the recovery assumptions.
- The analyst noted that the non-recovery probability was the only assumption sensitive enough to affect the final result of the significance determination.
- The analyst's sensitivity study indicated that the Green/White threshold would be at a non-recovery value of something less than 9 percent.
- Large Early Release In accordance with the guidance in Inspection Manual Chapter 0609, Appendix H, this finding would not involve an increased risk of a large early release of radiation because Callaway has a large, dry containment and the sequences contributing to a change in the core damage frequency did not involve either a steam generator tube rupture or an inter-system loss of coolant accident.
- Licensee's Evaluation The licensee evaluated the risk associated with the failure to start of turbine-driven auxiliary feedwater pump PAL02 using their site-specific probabilistic safety assessment model, as documented in PRAER 09-333, Revision 0.
- The licensee's final value for the increased risk (ICCDP) was 1.34 x 10
-7.
- The analyst reviewed the PRAER and determined that the assumptions and/or models used provided very similar results with one exception.
- The licensee's model and the SPAR models provided very similar results when evaluating risk related to the failure of turbine-driven auxiliary feedwater pump PAL02.
- The results of the external events evaluations were within a factor of two despite one being produced by the SPAR external events module and the other calculated by assuming a percentage of the internal events model results.
- However, there was a significant disagreement with the nonrecovery probability.
- The licensee used the EPRI HRA calculator to evaluate the human error probability and produced a value of 1.6E-2.
- This nonrecovery is about nominal for an action that requires diagnosis under rather ideal conditions.
- The analyst determined that the following conditions should have affected the licensee's results, but did not appear to be considered:
- 1. The Technical Support Center would not have been available prior to core damage during most conditions.
- The licensee noted that the center could Attachment 4 be manned within 30 minutes during the working day.
- However, in the analyst's judgment, they could not have effectively made an independent evaluation and response within the remaining time available.
- 2. There would be significant competing priorities - Recovery of two failed diesels
- Offsite power recovery - Recovery of the TDAFW pump
- 3. The environmental conditions were less than ideal - No emergency lighting in the pump room
- Hot, humid pump room - Crawling required to reach valve in some instances
- 4. The ergonomic conditions were significantly degraded - Standing on governor valve insulation - Footing in the vicinity of the mechanical trip mechanism - Off even standing for several potential vantages
- Reaching for valve handle - Operator can't see valve stem while manipulating
- 5. There were misleading indications that may have negatively impacted results - Operator may determine valve is bound - Operators may try to reset thermal overloads - Valve must be operated in the closed direction to open The biggest concern the analyst expressed was that the operator might believe the valve was mechanically bound and incapable of being manipulated manually.
- Under this case, he may have been redirected by licensed operators to attempt recovery of the emergency diesel generators or offsite power instead of focusing on the turbine-driven auxiliary feedwater pump.
- The licensee argued that recovery of secondary cooling would be the highest priority during these scenarios.
- However, the analyst pointed out that recovery of alternating current and restoration of a motor-driven pump would fully meet this priority.
- Given the analyst's understanding of the conditions under which recovery would be attempted, the licensee's nonrecovery probability is overly optimistic.
- Appendix-A4-1 Appendix to Attachment 4 APPENDIX
- Table A1Single Parameter Sensitivities Model Elements Best Decrease Single Parameter Increase Single Parameter Event Tree Basic Event Estimate 1 2 3 4 5 6 7 8 Dispatch
- CR-Action 1.00E-04
- 1.00E-05
- 1.00E-03
- Valve-Fails Stuck 0.8 0.2
- 0.9
- Forced Force 0.2
- 0.1
- 0.5
- Action FCHV0312 3.80E-02
- 4.00E-03
- 5.00E-02
- 1.81E-01 4.59E-02 1.05E-01 1.80E-01 1.62E-01 2.03E-01 4.09E-01 1.81E-01 1.87E-01
- Table A2Multiple Parameter Sensitivities Increase Sticking Probability Decrease Sticking Probability Low Sticking Probability Never
- Fails
- FALSE 0.9 0.9 0.9 0.90.5 0.5 0.5 0.5 0.2 0.2 0.2 0.2
- 0.1 0.2 0.3 0.40.1 0.2 0.3 0.4 0.1 0.2 0.3 0.4
- FALSE
- 1.18E-01 2.03E-01 2.88E-01 3.73E-016.63E-02 1.14E-01 1.61E-01 2.09E-01 2.68E-02 4.59E-02 6.51E-02 8.43E-021.60E-01
- Appendix-A4-2 Appendix to Attachment 4
- Table A3Sensitivity Evaluation Adjustments to Dominant Sequences for Timing Initiator Sequence Number CDFContributionSequence Initiator Functional Losses AdjustmentsModifiedCDF LOOP 16-45 1.95E-0532.3%SBO
- No 1.95E-05
- FRI-A1A 2-11 1.75E-0529.0%TRANS
- AFW-HPR 1.00E-028.35E-07FRI-C27 1-16-45 1.04E-0517.2%SBO
- No 1.04E-05FRI-C27 2-4 2.90E-064.8%MCR-EVAC N/A 1.00E-001.38E-05FLI-FZ2 2-13 2.03E-063.4%LOSWS
- AFW-CCW 1.00E-029.69E-08LODCNK4 8 1.98E-063.3%AFW-FB
- No 1.98E-06LODCNK1 8 1.98E-063.3%AFW-FB
- No 1.09E-06ATWS
- 1.01E-072.0%ALL ATWS N/A 1.00E-004.80E-07LOOP 15 1.09E-061.8%AFW-FB
- No 6.05E-07FRI-A1A 2-12 6.05E-071.0%TRANS
- AFW-FB 1.00E-022.88E-08LOOP 16-43-04 6.04E-071.0%SBO
- No 6.04E-07FRI-C27 1-15 3.06E-070.5%LOOP
- AFW-FB No 3.06E-07LOMFW 8 2.81E-070.5%AFW-FB
- No 2.81E-07FRI-TB3 2-16-45 2.23E-070.4%SBO
- No 2.23E-07LOOP 16-43-10 1.51E-070.2%SBO
- No 1.51E-07EQ3 2-15 1.20E-070.2%LOOP
- AFW-FB No 1.20E-07
- SUM
- 5.98E-05
- 5.05E-05 TOTAL 6.04E-05 Percent 99.0%Total Delta CDF: 5.1E-05/year
- Appendix-A4-3 Appendix to Attachment 4
- Table A4 Recovery Time Estimates Activity Time Units Licensed Operators Following Emergency Procedures: 10minutes Licensed Operators Contact/Recall Plant Operator: 5minutes Licensed Operators Brief Plant Operator: 3minutes Plant Operator Travels to the Pump Room: 8minutes Plant Operator Assesses Condition: 5minutes Plant Operator Contacts Licensed Operators for Instructions: 4minutes Plant Operator Returns to Valve: 2minutes Plant Operator Makes Determination Regarding Valve Functionality: 1minute