ML093160122

IR 05000244-09-008; 06/15/2009 - 06/19/2009 and 08/24/2009 - 08/27/2009; R. E. Ginna Nuclear Plant; Special Inspection Team Report

ML093160122
Person / Time
Site:	Ginna
Issue date:	11/12/2009
From:	David Lew Division Reactor Projects I
To:	John Carlin Ginna
Clifford J
References
EA-09-249 IR-09-008
Download: ML093160122 (29)

See also: IR 05000244/2009008

Text

UNITED STATES

NUCLEAR REGULATORY COMMISSION

REGION I

475 ALLENDALE ROAD

KING OF PRUSSIA, PA 19406-1415

November 12, 2009

EA-09-249

Mr. John T. Carlin

Vice President, RE. Ginna Nuclear Power Plant

RE. Ginna Nuclear Power Plant, LLC

1503 Lake Road

Ontario, New York 14519

SUBJECT: RE. GINNA NUCLEAR POWER PLANT - NRC SPECIAL INSPECTION TEAM

REPORT 05000244/2009008; PRELIMINARY WHITE FINDING

Dear Mr. Carlin:

On June 16-20, 2009, and August 24-27,2009, the U.S. Nuclear Regulatory Commission (NRC)

conducted the onsite portions of a special inspection at RE. Ginna Nuclear Power Plant. In

office inspection reviews were conducted in the intervening weeks. The enclosed report

documents the inspection team's findings and observations which were discussed with you and

others members of your staff on August 27, 2009, during a preliminary exit briefing, and on

October 7, 2009, during the final exit meeting.

The special inspection was conducted in response to a turbine driven auxiliary feedwater

(TDAFW) pump overspeed trip on May 26, 2009. The team included a subsequent overspeed

trip on July 2,2009 into its inspection scope. The NRC's initial evaluation of this condition

satisfied the criteria in NRC Inspection Manual Chapter 0309, "Reactive Inspection Decision

Basis for Reactors," for conducting a special inspection. The basis for initiating this special

inspection is further discussed in the inspection team's charter that is included in this report as

Attachment B.

The inspection examined activities conducted under your license as they relate to safety and

compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed

personnel.

This letter transmits one self-revealing finding that, using the reactor safety Significance

Determination Process (SOP), has preliminarily been determined to be White, a finding with low

to moderate safety significance. The finding is associated with the failure to preclude

recurrence of a significant condition adverse to quality associated with the corrosion of the

governor control valve of the turbine driven auxiliary feedwater (TDAFW) pump, that led to a

failure of the TDAFW pump during surveillance testing on July 2, 2009 and was the likely cause

of the overspeed trip on May 26, 2009. Following the July test failure, Ginna replaced the

J. Carlin 2

governor control valve stem and conducted weekly monitoring of the governor control valve

during surveillance testing to identify any potential for stem binding. In addition, corrective

actions included a follow-up inspection of the governor control valve during the fall 2009

refueling outage. There is no immediate safety concern present due to this finding because the

system is now operable and the long term corrective actions are being implemented in Ginna's

corrective action program. The final resolution of this finding will be conveyed in a separate

correspondence.

As discussed in the attached Inspection report. the tinding is also an apparent violation of NRC

requirements. specifically, 10 CFR Appendix B, Criterion XVI. "Corrective Actions." and is

therefore being considered for escalated enforcement action in accordance with the

Enforcement Policy, which can be found on the NRC's Web site at http://www.nrc.gov/reading

rm/doc-collections/enforcementl.

Following a discussion of the preliminary safety Significance of this finding during the initial exit

briefing on August 27,2009. a phone call was held between Glenn Dentel, Branch Chief,

Division of Reactor Projects. and yourself on October 1. 2009. During this call, you indicated

that R.E. Ginna Nuclear Power Plant does not contest the characterization of the rIsk

significance of this finding, and therefore you have declined to further discuss this issue at a

Regulatory Conference or provide a written response. Please note that by declining to request

a Regulatory Conference or submit a written response, you relinquished your right to appeal the

final SOP determination. in that by not doing either, you would not meet the appeal

requirements stated in the Prerequisite and Limitation sections of Attachment 2 of IMC 0609.

You will be advised by a separate correspondence of the results of our deliberations on this

matter.

In addition. the report documents two findings of very low safety significance (Green). The

findings involved violations of NRC requirements. However, because of the very low safety

significance and because they are entered into your corrective action program (CAP). the NRC

is treating these findings as non-cited violations (NeVs) consistent with Section VI.A.1 of the

NRC Enforcement Policy. If you contest any NCV in this report, you should provide a response

within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear

Regulatory Commission, A TIN.: Document Control Desk, Washington, DC 20555-0001; with

copies to the Regional Administrator, Region I; the Director, Office of Enforcement; and the

NRC Resident Inspector at R.E. Ginna. In addition. if you disagree with the characterization of

any finding in this report, you should provide a response within 30 days of the date of this

inspection report. with the basis for your disagreement, to the Regional Administrator, Region I,

and the NRC Resident Inspector at R.E. Ginna Nuclear Power Plant. The information you

provide will be considered in accordance with Inspection Manual Chapter 030?

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its

enclosure, and your response (if any) will be available electronically for public inspection in the

J. Carlin 3

NRC Public Document Room or from the Publicly Available Records (PARS) component of the

NRC's document system (ADAMS). ADAMS is accessible from the NRC Web Site at

http://www.nrc.gov/reading-rm/adams.html(the Public Electronic Reading Room).

Sincerely,

,/;I,/;tJ

D~ew,~~ --fbr-

Division of Reactor Projects

Docket No.: 50-244

License No.: DPR-18

Enclosures: Inspection Report 05000244/2009008

w/Attachment A: Supplemental Information

w/Attachment B: Special Inspection Charter

cc w/encl: Distribution via ListServ

J. Carlin 2

NRC Public Document Room or from the Publicly Available Records (PARS) component of the

NRC's document system (ADAMS). ADAMS is accessible from the NRC Web Site at

httg:llwww.nrc.gov/reading*rm/adams.html(the Public Electronic Reading Room).

Sincerely,

IRAJ James W. Clifford for:

David C. lew, Director

Division of Reactor Projects

Docket No.: 50-244

License No.: DPR-18

Enclosures: Inspection Report 05000244/2009008

w/Attachment A: Supplemental Information

w/Attachment B: Special Inspection Charter

cc w/encl: Distribution via ListServ

SUNSI Review Complete: gtd (Reviewer's Initials)

DOCUMENT NAME: G:\DRP\BRANCH1\Ginna\TDAFW SIT\AED Track Changes Ginna SIT

Report Rev 4 cC.doc

After declaring this document "An Official Agency Record" it will be released to the Public.

To receive I!I copy of this document, indicate in the box: "C' =Copy without attachmenVenclosure "e" = Copy with

attaclhmentlenclosure "N";:; No co ML093160122

OFFICE RI/DRS RI/ENF RI/DRP

NAME CCahill/cc* JClifford/jwc

DATE 10/21109 11/05/09

• see prior concurrence OFFICIAL RECORD COpy

Distribution w/encl:

S. Collins, RA (R10RAMaii Resource) ROPreports@nrc.gov

M. Dapas, ORA (R10RAMaii Resource)

D. Lew, DRP (R1DRPMaii Resource)

J. Clifford, DRP (R1DRPMail Resource)

l. Trocine, RI OEDO

RidsNrrPMREGinnaResource

RidsNrrDorlLpl1-1 Resource

G. Dentel, DRP

N. Perry, DRP

J. Hawkins, DRP

K. Kolaczyk, DRP, SRI

S. Kennedy, DRP, SRI

L. Casey, DRP, RI

M. Rose, DRP, OA

D. Bearde, DRP

1

U.S. NUCLEAR REGULATORY COMMISSION

REGION I

Docket No.: 50-244

License No.: DPR-18

Report No.: 05000244/2009008

Licensee: Constellation Energy, R. E. Ginna Nuclear Power Plant, LLC

Facility: R. E. Ginna Nuclear Power Plant

Location: Ontario. New York

Dates: June 15-19, 2009 and August 24-27.2009

Team Leader: C. cahill, Senior Reactor Analyst, Division of Reactor Safety

Inspectors: S. Kennedy, Senior Resident Inspector, Division of Reactor Projects

K. Kolaczyk, Senior Resident Inspector, Division of Reactor Projects

J. Bream, Project Engineer. Division of Reactor Projects

Approved by: Glenn T. Dentel, Chief

Projects Branch 1

Division of Reactor Projects

2

SUMMARY OF FINDINGS

IR 05000244/2009008; 06/15-19/2009 and 08/24-2'712009; R. E. Ginna Nuclear Plant; Special

Inspection Team Report.

The report covered two on-site inspection visits and related in-office inspection activities by a

special inspection team consisting of a Senior Reactor Analyst. two Senior Resident Inspectors.

and one Project Engineer. One apparent violation (AV) with potential for greater than Green

safety significance and two Green findings were identified. The significance of most findings is

indicated by its color (Green. White, Yellow, or Red) using Inspection Manual Chapter (lMC)

0609, "Significance Determination Process." Findings for which the significance determination

(SDP) process does not apply may be Green or be assigned a severity level after NRC

management review. The NRC program for overseeing the safe operation of commercial

nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4,

dated December 2006.

NRC Identified and Self-Revealing Findings

Cornerstones: Mitigating Systems

Preliminary White: A self-revealing apparent violation (AV) of 10 CFR 50, Appendix B,

Criterion XVI, uCorrective Actions," was identified for the failure to preclude recurrence

of a signIficant condition adverse to quality (SCAQ) associated with the Turbine Driven

Auxiliary Feedwater (TDAFW) pump governor control valve. Specifically, after

identifying corrosion of the govemor control valve stem in April 2005, Ginna did not take

adequate corrective actions to preclude the recurrence of corrosion which led to the

binding of the gavemar control valve and failure of the TDAFW pump on July 2,2009.

In addition, the inspectors concluded that governor control valve stem binding was the

likely cause of the failure of the TDAFW pump on May 26,2009. The overspeed trip of

the TDAFW pump on May 26. 2009. was originally determined by Ginna to be failure of

the governor control system relay valve. Governor control valve stem corrosion is a

SCAQ because corrosion of the stem can IHad to governor control valve stem binding

and failure of the TDAFW pump as discussed in NRC Information Notice (IN) 94-66:

"Overspeed of Turbine-Driven Pumps Caused by Governor Valve Stem Binding" and

other related industry operating experience documents. Immediate corrective actions

inCluded entering this condition in the corrective action program (CAP). conducting a

root cause analysis (RCA). replacing the governor control valve stem, and conducting

weekly monitoring of the governor control valve during surveillance testing to identify

any potential for stem binding. In addition, corrective actions included a follow-up

inspection of the governor control valve during the fall 2009 refueling outage. Ginna will

continue to monitor the govemor control valve under an enhanced TDAFW surveillance

program to ensure TDAFW pump operability.

The finding is more than minor because it is associated with the equipment performance

attribute of the Mitigating System cornerstone and affects the cornerstone objective to

ensure the availability, reliability, and capability of systems that respond to initiating

events to prevent undesirable consequences. Specifically. stem corrosion caused

binding of the governor control valve and led to the failure of the TDAFW pump. This

finding was assessed using IMC 0609 and preliminarily determined to be White (lOW to

moderate safety significance) based on a Phase 3 analysis with a total (internal and

Enclosure

3

external contributions) calculated conditional core damage frequency (CCDF) of B.6E-6.

This finding has a cross-cutting aspect in the area of Problem Identification and

Resolution, Corrective Action Program, because Ginna did not implement a corrective

action program with a low threshold for identifying issues completely, accurately, and in

a timely manner commensurate with their safety significance P.1(a} per IMC 0305].

Specifically, Ginna did not identify issues a~;sociated with corrosion of the governor

control valve within the corrective action program. (Section 2.1.1)

Green: A self-revealing non-cited violation (NCV) of 10 CFR 50, Appendix B, Criterion

XVI, "Corrective Actions," was identified for the failure to establish adequate measures

to correct a longstanding issue associated with steam admission valves leakage. As a

result, the leakage most likely contributed to the build-up of corrosion on the TDAFW

pump governor control valve stem and contributed to the failure of the TDAFW pump on

May 26, 2009, and on July 2, 2009. The steam admission valves had been leaking

since at least 2005. However, G.inna did not take adequate measures to correct the

leakage or minimize the impact of the leakage on governor control valve performance.

Immediate corrective actions included entering this condition in the corrective action

program, conducting a root cause analysis, replacing the governor control valve stem,

and conducting weekly monitoring of the governor control valve during surveillance

testing to identify any potential for stem binding. Additionally, the steam admission

valves were inspected and re-worked and the governor control valve was inspected

during the fall 2009 outage. Ginna will continue to monitor the governor control valve

under an enhanced TDAFW surveillance program to ensure TDAFW pump operability.

Planned corrective actions include replacin~~ the steam admission valves in May 2011.

The finding is more than minor because it is associated with the equipment performance

attribute of the Mitigating Systems cornerstone and affects the cornerstone objective to

ensure the availability and reliability of systems that respond to initiating events to

prevent undesirable consequences. Specifically, leakage through the steam admission

valves can result in continuous wetting of the governor control valve stem and lead to or

accelerate corrosion of the governor control valve. This could result in a stem binding of

the governor control valve and failure of the TDAFW pump. The inspectors evaluated

the signifiCance of this finding using IMC 0609, Attachment 4, "'Phase 1 - Initial

Screening and Characterization of Findings." The finding is of very low safety

significance because it is not a design or qualification deficiency, did not represent a

loss of a safety function of a system or a single train greater than its technical

speCification (TS) allowed outage time, and did not screen as potentially risk Significant

due to external events. This finding has a cross-cutting aspect in the area of Problem

Identification and Resolution, Corrective Action Program, because Ginna did not

thoroughly evaluate problems such that the resolutions address causes and extent of

conditions, as necessary, In a timely manner, commensurate with their significance

[P.1 (c) per IMC 0305]. Specifically, Ginna did not thoroughly evaluate the potential

effect of the steam admission valve leakage on the governor control valve performance.

(Section 2.1.2)

Green: The inspectors identified an NCVof 10 CFR 50, Appendix B. Criterion III,

"Design Control," for the failure to establish measures to ensure that a modification

performed on the governor control valve bushing was a suitable application of materials

for the TDAFW pump. During a review of the RCA associated with the TDAFW pump

failures, the inspectors noted that Ginna did not consider the potential impact of

Enclosure

4

removing some of the hardened layer of the bushing on the corrosion rate of the

governor control valve. Following concerns raised by the inspectors, Ginna inspected

the governor control valve bushing during the fall 2009 refueling outage and observed

corrosion of the bushings. Ginna noted that the corrosion of the bushings appeared to

have been caused by the lapping of the bushing to achieve the increased clearance

between the stem and the bushings. Immediate corrective actions following the

inspection of the governor control valve during the fall 2009 refueling outage included

entering this condition in the CAP and refurbishing the governor control valve with a new

stem and bushing.

The finding is more than minor because it is associated with the equipment performance

attribute of the Mitigating System cornerstone and affects the cornerstone objective to

ensure the availability, reliability, and capability of systems that respond to initiating

events to prevent undesirable consequences. Specifically, lapping of the valve bushing

resulted in an unanticipated corrosion mechanism of the govemor control valve that

impacted the reliability of the TDAFW pump" The inspectors evaluated the significance

of this finding using IMC 0609, Attachment 4. "Phase 1 Initial Screening and

Characterization of Findings." The finding is of very low safety significance because it is

a design or qualification deficiency confirmed not to result in the loss of operability or

functionality. The inspectors determined that this finding has a cross-cutting aspect in

the area of Human Performance, Decision Making. because Ginna did not make a

safety-significant or risk-significant decision using a systematic process, especially when

faced with uncertain or unexpected plant conditions, to ensure safety was maintained

H.1(a) per IMC 0305]. Specifically, Ginna did not use a systematic process such as an

engineering evaluation to properly evaluate the potential impact of removing some of the

hardened layer of the bushing. [H.1 (a) per IMC 0305]

(Section 2.3)

Licensee Identified Violations

None

Enclosure

5

REPORT DETAILS

1. INTRODUCTION

1.1 Background and Event Description

On May 26, 2009, and July 2, 2009, the TDAFW pump tripped on overspeed during the

performance of surveillance testing activities. These were the second and third times

the TDAFW pump had failed a surveillance test in a six month period. The previous test

failure occurred on December 2, 2008, when the TDAFW pump failed to develop

acceptable discharge pressure during a quarterly surveillance test. Details surrounding

the December test failure including Ginna corrective actions are discussed in NRC

Integrated Inspection Report 50-244/2009002. Following each of the three test failures,

Ginna declared the TDAFW pump inoperable and entered the limiting Condition for

Operation for TS 3.7.5 "Auxiliary Feedwater."

Following the May 26, 2009, test failure, Ginna troubleshooting activities were focused

on the TDAFW lubricating oil system where system pressure oscillations were noted by

test personnel prior to the pump overspeed trip. As part of the troubleshooting efforts,

the TDAFW lubricating oil system was drained and refilled with fresh oil, filters in the

lubrication system were examined and replaced, and several components in the

lubricating oil system including the relay and pressure regulating valves were replaced.

Additionally the turbine governor linkage was adjusted to original specifications outlined

in the vendor technical documents. Although several out of speCification and missing

components were identified during this troubleshooting effort, a definitive cause for the

surveillance failure was not identified. Nevertheless, following the successful completion

of post maintenance testing activities, Ginna declared the TDAFW pump operable and

commenced an augmented surveillance testing program for the TDAFW pump that

tested the pump on a weekly basis.

On July 2, 2009, another TDAFW pump overspeed failure occurred when the pump was

undergoing the augmented testing program. Troubleshooting activities following this

failure were focused on the turbine control system, which did not appear to have

functioned properly during the test. Accordingly, the turbine control and relay valves

were disassembled and inspected, and the governor linkage system was adjusted.

Additionally. the lubricating oil system was flushed and examined for particulates.

However, unlike the May 26 failure, these troubleshooting activities identified a definitive

cause, a stuck governor control valve stem, that resulted in the surveillance test failure.

Visual inspection of the stem, which had become bound to its bushing and had to be

forcibly removed, revealed corrosion buildup where the stem contacted the upper valve

bushing. To restore the control valve to an operable status, Ginna replaced the valve

stem and increased the bushing clearances to dimensions specified by the Ginna

TDAFW pump inspection procedure. The pump was successfully tested and declared

operable subject to an augmented testing program.

Ginna assigned an RCA team to investigate the surveillance test failures. The Ginna

RCA team concluded that the primary root cause of the failure of the governor contrOl

valve to control turbine speed on July 2,2009, was binding caused by a corrosion

mechanism that occurred between the valve stem and the valve bushings. The RCA

team also concluded that stem binding appeared to be an intermittent problem and that

Enclosure

6

station members missed opportunities to identify the failure mode during previous

surveillances and failures. As such, the RCA team determined that governor control

valve stem binding could not be ruled out as a possible failure mode during the

December 2008 and the May 2009 TDAFW pump failures.

At Ginna's request, an independent vendor reviewed the issues associated with the

governor control valve stem binding to determine the failure mechanism that was the

cause of the TDAFW pump overspeed trips. The vendor determined that the most likely

cause of the corrosion of the governor control valve stem was fresh-water corrosion

(Langelier corrosion) that was the result of galvanic interactions between the nitrided

case and the base material of the governor control valve stem. The vendor considered

several variables that contributed to the corrosion mechanism such as material

composition of the stem and bushing, stem-to-bushing clearance, steam admission

valve leakage, and TDAFW pump surveillance frequency and duration.

1.2 Special Inspection Scope

The NRC conducted this inspection to gain a better understanding of the circumstances

involving the TDAFW pump overspeed trips during surveillance testing on May 26, 2009,

and on July 2,2009. The inspection team used NRC Inspection Procedure 93812,

"Special Inspection," as a guide to complete their review. Additional inspection and

review activities were outlined in the special inspection team charter, provided as

Attachment B. The special inspection team reviewed procedures, corrective action

documents, work orders, engineering analyses, and the root cause evaluation prepared

by Ginna. In addition. the team conducted equipment walkdowns and interviewed key

plant personnel regarding the discovery and resolution of the condition. A list of site

personnel interviewed and documents reviewed are provided in Attachment A to this

report.

1.3 Preliminary Conditional Risk Assessment

Using IMC 0309, "Reactive Inspection Decision Basis for Reactors," IMe 0609,

"Significance Determination Process," and the Ginna Standardized Plant Analysis Risk

(SPAR) model in conjunction with the Graphical Evaluation Module (GEM). the Region I

Senior Reactor Analyst (SRA) evaluated the increase in conditional core damage

probability for the failure of the TDAFW pump. Based on the nature of the failure and

the subsequent overspeed trip on an instrumented diagnostic run, the condition was

evaluated as being non-recoverable in the event of actual demand.

Based upon best available information, an incremental conditional core damage

probability (ICCDP) in the upper E-6 range (8E-6 per the 82 day exposure period), was

calculated. The exposure period was based on the assumption that the TDAFW pump

would not have started since the last time it passed a surveillance test on March 5 until

May 26 (82 days). The dominant core damage sequence was a station blackout (loss of

offsite power (LOOP) with failure of both emergency diesel generators (EDGs), with no

TOAFW and failure to recover offsite power or an EDG in one hour.

Based upon this conservative conditional core damage probability (CCDP) value, and

having satisfied an IMC 0309 deterministic criterion, the May 26 degraded TDAFW pump

Enclosure

7

condition fell within the Special Inspection to Augmented Inspection Team range for

reactive inspections.

2. SPECIAL INSPECl"ION AREAS

2.1 Review of Maintenance

a. Inspection Scope

The team evaluated the adequacy and completeness of the maintenance on the TDAFW

system, including preventive maintenance, procedural guidance, post-maintenance

testing, and supervisory oversight. The team independently evaluated selected

procedures, preventive maintenance strategies. condition reports (CRs). system health

reports. and associated work orders. In addition, the team reviewed the RCA,

conducted equipment walkdowns and interviewed key station personnel. For the

weaknesses identified, the inspectors verified that appropriate corrective actions have

been planned or taken.

b. Findings and Observations

1. Failure to Preclude Recurrence of a Significant Condition Adverse to Quality

Introduction: A self-revealing AV of 10 CFR 50, Appendix S, Criterion XVI, "Corrective

Actions," was identified for the failure to preclude recurrence of a SCAQ associated with

the TDAFW pump governor control valve. Specifically, after identifying corrosion of the

governor control valve stem in April 2005, Ginna did not take adequate corrective

actions to preclude the recurrence of corrosion which led to the binding of the governor

control valve and failure of the TDAFW pump on July 2, 2009. In addition, governor

control valve stem binding was a fikely failure of the May 26, 2009, TDAFW pump

overspeed trip. This finding was preliminarily determined to be White.

Description: On April 11, 2005, Glnna conducted a periodic major inspection of the

TDAFW pump governor control valve under work order #20401907. During the

disassembly of the valve, mechanics noted corrosion in the bushing area of the stem

plug. The vendor recommended replacing the valve stem and plug due to the corrosion.

However, Ginna did not recognize stem corrosion as a condition adverse to quality and

did not initiate a CR in accordance with station guidance in IP-CAP-1 r "Condition

Reporting." As a result, Ginna did not conduct any further analysis or evaluation to

determine the cause of the corrosion. On July 2, 2009, during surveillance testing, the

TDAFW pump tripped due to overspeed. Ginna formed an Issue Response Team and

developed a comprehensive troubleshooting plan to determine the cause(s) of the

failure. The troubleshooting plan consisted of all possible failure modes, possible

causes of each failure mode, and actions to validate or refute each "failure mode. Upon

disassembly of the governor control valve, the stem was found seized within its bushings

and had to be forcibly removed. Visual inspection showed pitting on the surface of the

stem Where it contacted the upper valve bushing. Following completion of the

troubleshooting activities, Ginnaconcluded that TDAFW pump failure was due to binding

of the governor control valve and that the binding was caused by the build up of

corrosion on the valve stem.

Enclosure

8

Governor control valve stem corrosion is an SCAQ because corrosion of the stem can

lead to govemor control valve stem binding and failure of the TDAFW pump as

discussed in NRC IN 94-66: "Overspeed of Turbine-Driven Pumps caused by Governor

Valve Stem Binding," and other related industry operating experience documents.

Immediate corrective actions included entering this condition in the CAP, conducting an

RCA, replacing the governor control valve stem, and conducting weekly monitoring of

the governor control valve during surveillance testing to identify any potential for stem

binding. In addition, corrective actions included a follow-up inspection of the governor

control valve during the fall 2009 refueling outage. Because additional corrosion was

found during this inspection on September 18, 2009, the control valve was sent to the

vendor to be refurbished. Ginna will continue to monitor the governor control valve

under an enhanced TDAFW surveillance program to ensure TDAFW pump operability.

Ginna also was evaluating longer term corrective actions to address the corrosion

including more frequent governor control valve stem replacement, change of the stem

material, and other modifications to the system design. Planned corrective actions

include replacing the stem admission valves in May 2011.

The inspectors noted that governor control valve stem binding also could have been a

contributor to the December 2, 2008, TDAFW pump failure to develop the minimum

acceptable discharge flow and pressure, and the likely cause of the May 26, 2009,

TDAFW pump overspeed trip. Following these events, Ginna missed opportunities to

identify potential stem binding problems after identifying possible indicators of this failure

mode such as leaking steam admission valves and rust/corrosion on the visible portion

of the governor valve stem. In addition, during troubleshooting efforts in May 2009,

. Ginna missed an opportunity to exercise the governor control valve stem with the linkage

disconnected and lube oil pressure not applied. Operating experience suggested that

cycling the valve by hand without hydraulics applied is a prudent action to validate or

refute governor control valve stem binding.

Analysis: The performance deficiency is that Ginna did not take adequate measures to

correct a condition that had the potential to impact the operability of the TDAFW pump.

Specifically, after identifying corrosion on the governor control valve stem in 2005, Ginna

did not take adequate corrective actions to preclude the recurrence of corrosion which

led to the binding of the governor control valve and failure of the TDAFW pump. The

finding is more than minor because it is associated with the equipment performance

attribute of the Mitigating System cornerstone and affects the cornerstone objective to

ensure the availability, reliability, and capability of systems that respond to initiating

events to prevent undesirable consequences. Specifically, stem corrosion caused

binding of the governor control valve and led to the failure of the TDAFW pump.

In accordance with IMe 0609, Attachment 4, "Phase 1 - Initial Screening and

Characterization of Findings," a Phase 2 risk analysis was required because the finding

represents an actual loss of safety function of a single train for greater than the TS

allowed outage time of 7 days. The Phase 2 risk evaluation was performed in

accordance with IMC 0609, Appendix A, Attachment 1, "User Guidance for Phase 2 and

Phase 3 Reactor Inspection Findings for At*,power Situations." Because the precise time

is unknown for the inception of TDAFW pump inoperability, an exposure time of one-half

of the time period (t/2) between discovery (May 26, 2009) to the last successfully

completed quarterly surveHlance test (March 5, 2009) was used. This t/2 exposure time

Enclosure

9

equals 51 days. Using Ginna's Phase 2 SDP notebook, pre-solved worksheets, and an

initiating event likelihood of 1 year (>30-days exposure time), the inspector identified that

this finding is of potentially substantial safety significance (Yellow). The dominant

sequence identified in the Phase 2 notebook involves a loss of offsite power (LOOP),

failure of both EDGs, and the subsequent loss of the TDAFW pump, with the failure of

operators to restore offsite power within 1 hour: LOOP (2) + EAC (3) + TDAFW (0) +

REC1 (0) = 5 (Yellow). In recognition that the Phase 2 notebook typically yields a

conservative result, a NRC Region I Senior Reactor Analyst (SRA) performed a Phase 3

risk assessment of this finding.

The SRA used Ginna's Standardized Plant Analysis Risk (SPAR) model, Revision 3.45,

dated June 2008. and graphical evaluation module, in conjunction with the System

Analysis Programs for Hands-On Integrated Reliability Evaluations (SAPHIRE). Version

7, to estimate the internal risk contribution of the Phase 3 risk assessment. The

following assumptions were used for this assessment:

1. Based on the guidance provided in the Risk Assessment of Operational Events

Handbook, Revision 1.01, the calculated exposure was determined to be 1198

hrs. The summation of this exposeure time was determined as follows:

• One half the exposure time, (tI2), was applied for the period between

March 5 - May 26, 2009

• The full exposure time, (t). was applied for the unavailability and

troubleshooting from May 26 - May 29, 2009.

• One half the exposure time. (tI2), was applied for the period between

June 25 -July 2, 2009

• The full exposure time, (t), was applied for the unavailability and

troubleshooting from July 2 - July 5, 2009

2. To closely approximate the type of flilure exhibited by the TDAFW pump, the fail

to start basic event <AFW-TDP-FS-TDP> was changed from its baseline failure

probability to 1.0, representing a 100 percent failure to start condition.

3. Based on the nature of the failure. and no recovery procedures in place, there

was no recovery credit assigned to the May failure.

4. The loss of service water initiating event frequency (IE-LOSWS) was increased

from its nominal value of 4.0E-4 to 2.0E*3 to more closely model the risk of a

LOSWS at the Ginna station. This is consistent with Ginna's understanding of

the risk of this event. All remaining events were left at their nominal failure

probabilities.

5, The model was modified by Idaho National Laboratory (INL) to include

convolution correction factors. Convolving the failure distribution eliminates the

simplifying assumption that all failure to run events happen at time=O. Inclusion

of this correction can reduce station blackout (SBO) core damage frequency

(CDF) significantly for plants like Ginna that have low EDG redundancy.

6. Since all observed failures happened upon the initiation of the TDAFW system,

the period between May 29 and June 25 2009, was not included in the exposure

period because the system successfully completed increased, weekly,

surveillance testing. The team could not conclude that the stem binding

condition would occur after the pump had been successfully started and run.

7. Cutset probability calculation truncation was set at 1E-13.

Enclosure

10

Based upon the above assumptions, the Ginna SPAR model calculated an increase in

internal event contribution to conditional core damage probability (CCDP) of 2.9E-6. This

low-ta-mid E-6 delta CCDP value represents a low to moderate safety significance

(White). The dominant internal event sequence involved a LOOP with subsequent

failure of the EDGs (station blackout event), the failure of the TDAFW train and the

failure to recover AC power. The Phase 3 SPAR model results correlate well to the

Phase 2 SDP Notebook dominant core damage sequences.

External Events Risk Contribution

The Ginna Probablistic Safety Assessment (PSA) includes a Level 1 analysis of fires

and flooding external events. The PSA summarizes the fire contribution as representing

approximately 48% of the total (internal and external) core damage frequency, or nearly

half of the annualized risk. The NRC does not have an external event risk model for

Ginna. Consequently, the SRA, after review of the licensee's Individual Plant

Examination of External Events (IPEEE), utilized the licensee's external events

assessment to quantify the fire and flooding events risk contribution for this condition.

Seismic event likelihood was considered to be very low and determined not to be a

significant contributor to the risk of this condition.

The results of the PSA for this condition calculated a CCDP contribution from fire events

at 5E-6 and from flooding at 7E-7. The most significant fire initiated core damage

sequence involved a spectrum of control room fires, with a failure of automatic and

manual suppression, a failure of the TDAFW pump and a failure of the 'C' standby

auxiliary feed water pump for decay heat removal via the steam generators. The most

Significant flooding core damage sequence quantified in the PSA for this condition,

Involved flooding in the relay room, failure of the TDAFW pump and a failure to align

standby auxiliary feed water pump for decay heat removal via the steam generators.

Large Early Release Frequency (LERF) Evaluation

The SRA used IMC 0609, Appendix H, "Containment Integrity Significance

Determination Process," to determine if this finding was a significant contributor to a

large early release. The Ginna containment is classified as a pressurized water reactor

large-dry containment deSign. Based upon the dominant sequences involving LOOP

and station blackout (S80) initiating events, per Appendix H, Table 5.2, "Phase 2

Assessment Factors - Type A Findings at Full Power," the failure of the TDAFW pump

does not represent a significant challenge to containment integrity early in the postUlated

core damage sequences. Consequently, this finding does not screen as a significant

large early release contributor because the close-in populations can be effectively

evacuated far in advance of any postulated release due to core damage, Accordingly,

the risk significance of this finding is associated with the delta CDF value, per IMC 0609,

Appendix H, Figure 5.1, and not delta LERF.

Risk Assessment Summary

The calculated total risk significance of this finding is based upon the summation of

internal and external risk contributions (delta CCDP internal + delta CCDP external (fires

and floods).= delta CCDP total). 2.9E-6 + 5E-6 + 7E-7 = 8.6E-6 delta ceDP.

Enclosure

11

Annualized, this value of 8.6E-6 delta CDF represents a low to moderate safety

significance or White finding.

Licensee's Risk Assessment Summary

Constellation's risk assessment for the given condition, assuming no operator recovery,

resulted in a total delta CDF value of 1.028E-5. This increase in CDF value comprised

of: 4.6E-6 due to intemal events; 0.7E-6 due to intemal flooding events; and 5E-6

fire/external events. Similar to the NRC internal risk contribution, the largest percentage

of intemal risk was derived from station blackout events. The licensee conducted

refinements in their modeling to credit additional recovery options in the event that the

TDAFW pump failed. By crediting these recovery actions, the delta CDF was reduced

from 1.028E-5 to 9.2 E-6, or by approximately 10%. The specific recovery actions are

as follows: .

1. The potential for operators to use standby auxiliary feedwater (SAFW) pump C,

in lieu of the TDAFW pump during fire and flood scenarios which require use of

alternate shutdown procedures was modeled. Although use of an SAFW pump

is not specifically called out in these procedures, the procedures do explicitly

recognize that if the TDAFW pump is not functioning, a loss of secondary heat

sink will result, and direct operators to refer to other emergency procedures for

alternate methods of establishing a heat sink (see ER-FIRE.1 step 4.3.4.2, ER

FIRE.2 step 4.3.12, etc.}. In response to this hypothetical scenario, an

operations Shift Manager (SM) indicated that upon a failure of the TDAFW to

provide flow, a SAFW pump could be manually started by locally opening the

service water suction valve to the pump and locally closing the breaker to the

pump motor. The steps necessary to align an SAFW pump to the steam

generators (SG) are contained in emergency operating procedures (EOP)

Attachment 5.1 'Attachment SAFW'. Since only Busses 14 and 18 are energized

during ER-FIRE scenarios, SAFW pump C would be used.

2. Following the May 26th overspeed trip of the TDAFW pump, new section 2.2

was added to procedure P-15.6, 'Operation of the TDAFW Pump Trip Throttle

Valve.' Revision 00200. This step provides instructions for a 'slow start' of the

TDAFW pump manually, following an overspeed trip of the pump, by using the

trip throttle valve. This allows for starting of the pump even if the governor control

valve is stuck in the full open position following the overspeed trip. This

procedure step was successfully used following both the May 26th failure and

the July 2nd failure. Per Operations management, the shift managers, who

would be directing use of this procedure, were briefed on the updated procedure

to ensure they were aware that new step was available for use and understood

how it was to be performed. A controlled copy of the procedure is located at the

TDAFW pump for use by the operator. Since this procedure was in effect prior

to the July failure exposure period (Le., prior to the last successful TDAFW pump

test on June 25, 2009), it is considered as a recovery for failures of the TDAFW

pump.

Recovery actions identified in #1 above were not quantified by the team however the

evaluation approach appears to be appropriate. Prior to the licensee crediting recovery

actions identified in #2 above, the team was able to verify that the procedures were in

Enclosure

12

place and observed troubleshooting video in which the licensee started the TDAFW

pump with this method. As a result the SRA concluded that the licensee's modeling

demonstrating a risk reduction was appropriate.

Based upon the close comparison between NRC and Constellation risk estimates, no

sensitivity analyses were warranted. The use of V2 to approximate the exposure time

was determined to be appropriate for standby or periodically operated components that

fail due to a degradation mechanism that gradually affects the component during the

standby time period. Inclusion of the unavailability time hours, due to troubleshooting

and repairs, is also appropriate and consistent with the guidance promulgated in Risk

Assessment of Operational Events Handbook, Revision 1.01, dated January 2008.

This finding has a cross-cutting aspect in the area of Problem Identification and

Resolution, Corrective Action Program, because Ginna did not implement a corrective

action program with a low threshold for identifying issues completely, accurately, and in

a timely manner commensurate with their safety significance. Specifically, Ginna did not

identify issues associated with corrosion of the governor control valve within the

corrective actions program. The inspectors concluded that the performance deficiency is

reflective of current performance because Ginna had reasonable opportunities to identify

the issue during troubleshooting in December 2008 and May 2009. P.1(a) per IMC

0305]

Enforcement: 10 CFR 50, Appendix B, Criterion XVI, "Corrective Actions," states, in

part, "Measures shall be established to assure that conditions adverse to quality, such

as failures, malfunctions, defiCiencies, deviations, defective material and eqUipment, and

nonconformances are promptly identified and corrected. In the case of significant

conditions adverse to quality, the measures shall assure that the cause of the condition

is determined and corrective action taken to preclude repetition." Contrary to the above,

after identifying corrosion on the TDAFW pump governor control valve stem on April 11,

2005, which is a significant condition adverse to quality, Ginna did not take adequate

measures to determine the cause and prevent recurrence. The cause of the condition

was left uncorrected and resulted in additional stem corrosion that led to binding of the

governor control valve and the failure of the TDAFW pump on July 2, 2009. In addition,

stem corrosion was the likely cause of the TDAFW pump failure on May 26, 2009. This

issue was entered into Ginna's CAP as CR-:2009-003680 and CR-2009-004577.

Pending final determination of significance, this finding is identified as an AV. (AV

0500024412009008*01: Inadequate Corrective Actions Associated with the TDAFW

Pump Governor Control Valve)

2. Untimely Corrective Actions Associated with Steam Admission Valves

Introduction: A self-revealing NeVof 10 CFR 50, Appendix B, Criterion XVI, "Corrective

Actions," was identified for the failure to establish adequate measures to correct a

longstanding issue associated with steam a(]mission valves leakage. As a result, the

leakage most likely contributed to the build-up of corrosion on the TDAFW pump

governor control valve stem on May 26, 2009 and contributed to the failure of the

TDAFW pump on July 2, 2009.

Description: During a review of failures associated with the TDAFW pump, the

inspectors noted that the steam admission valves had a history of leaking. Based on a

review of the RCA report, industry operating experience, and associated CRs, the

Enclosure

13

inspectors determined that the steam admission valve leakage was a contributing factor

in the development of corrosion of the governor control valve and contributed to the

failure of the TDAFW pump on July 2, 2009, and most likely the failure of the TDAFW

pump overspeed trip failure on May 26, 2009. Based on a review of eRs and interviews

of personnel, the inspectors determined that the steam admission valves had been

leaking since at least 2005.

In July 2005, CR-2005~3660 documented that steam admission valve 3505A was

leaking past its seat. However, Ginna did not take or plan any corrective actions

associated with the issue. By October 2006, steam admission valve 3505A leakage had

increased such that the TDAFW pump turbine was rotating approximately 100 rpm.

Ginna cycled the valve to reduce seat leakage, generated a work order to repair the

valve in May 2008, and conducted an evaluation of past operability. Ginna concluded

that there was no potential for increased consequences with time if the condition

continued. During the May 2008 refueling outage, Ginna conducted a repair of steam

admission valve 3505A. However, this corrective action was not effective; by September

2008, the valve was leaking again although at a reduced rate. The inspectors

determined that Ginna did not recognize or consider the potential impact of the steam

admission valve leakage on the governor control valve stem.

Immediate corrective actions included entering this condition in the CAP, replacing the

govemor control valve stem, and conducting weekly monitOring of the governor control

valve during surveillance te~ting to identify any potential for stem binding. Additionally,

the steam admission valves were inspected and re-worked and the govemor control

valve was Inspected during the fall 2009 outage. Ginna will continue to monitor the

governor control valve under an enhanced TDAFW surveillance program to ensure

TDAFW pump operability. Planned corrective actions include replacing the stem

admission valves in May 2011.

Analysis: The performance deficiency is that Ginna did not adequately address

deficiencies associated with steam admission valve leakage in that the leakage likely

resulted in accelerated corrosion of the governor control valve and contributed to the

failure of the TDAFW pump. The finding is more than minor because it is associated

with the equipment performance attribute of the Mitigating Systems cornerstone and

affects the cornerstone objective to ensure the availability and reliability of systems that

respond to initiating events to prevent undesirable consequences. Specifically, leakage

through the steam admission valves can res.ult in continuous wetting of the governor

control valve stem and lead to or accelerate corrosion of the governor control valve.

This could result in a stem binding of the govemor control valve and failure of the

TDAFW pump. The inspectors evaluated the significance of this finding using IMC 0609,

Attachment 4, "Phase 1 -Initial Screening and Characterization of Findings." The

finding is of very low safety significance because it is not a design or qualification

deficiency, did not represent a loss of a safety function of a system or a single train

greater than its TS allowed outage time, and did not screen as potentially risk Significant

due to external events. The inspectors determined that the steam admission valve

leakage was not a direct cause of the failure of the TDAFW pump overspeed events.

The steam admission valve leakage was a contributing factor in the development of

governor control valve stem corrosion and contributed to the failure of the TDAFW

pump.

Enclosure

14

This finding has a cross-cutting aspect in the area of Problem Identification and

Resolution, Corrective Action Program, because Ginna did not thoroughly evaluate

problems such that the resolutions address causes and extent of conditions, as

r:eecessary, in a timely manner, commensurate with their significance. Specifically,

Ginna did not thoroughly evaluate the potential effect of the steam admission valve

leakage on the governor control valve performance. The inspectors determined that this

issue is reflective of current licensee performance because each time the issue was

identified and a CR was generated represented an opportunity for Ginna to adequately

evaluate the issue and assign appropriate corrective actions. [P.1 (c) per IMC 0305]

Enforcement: 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Actions," states, in

part. "Measures shall be established to assure that conditions adverse to quality, such

as failures, malfunctions, deficiencies, deviations, defective material and equipment, and

nonconformances are promptly identified and corrected." Contrary to this requirement,

from at least July 2005 to July 2, 2009, Ginna did not established adequate measures to

correct longstanding steam admission valves leakage and/or minimize the impact of the

leakage on the TDAFW pump governor control valve performance. As a result, leakage I*

I

through the steam admission valves contributed to the corrosion of the governor control

valve and contributed to the failure of the TDAFW pump on July 2, 2009 and most likely

the failure of the TDAFW pump on May 26, 2009. Because this violation is of very low

safety significance (Green) and Ginna entered this issue into their CAP for resolution as

CR-2009-003680 and CR-2009-004577, this violation is being treated as an NCV

consistent with the NRC Enforcement Policy. (NCV 0500024412009008-02: Inadequate

Corrective Actions Associated with Stearn Admission Valve Leakage)

2.2 Review of Operating Experience

a. Inspection ScoQe

The team reviewed operating experience involving TDAFW pump failures and actions

taken by the Ginna staff to identify and address these types of failures. In addition, the

team examined the specific issues associated with governor control valve stem binding

to assess any new generic issues of industry interest for prompt communication and

dissemination. As part of this evaluation, the inspectors reviewed pertinent industry

operating experience, Ginna's response to NRC Information Notices, and interviewed

key plant personnel.

b. Findings and Observations

No Findings of Significance Identified.

The inspectors determined that there were no new generic issues identified as a result of

this event. However, the inspectors noted that Ginna missed opportunities to utilize

industry operating experience to identify precursors associated with this event and to

conduct effective troubleshooting. Specifically, following the discovery of corrosion on

the exposed portion of the TDAFW pump govemor control valve stem in May 2005,

Ginna did not enter this issue in to their CAP. As a result Ginna missed an opportunity

to revisit operating experience from NRC IN 94-66 conceming governor control valve

stem binding. In several examples in NRC IN 94-66, sites that experienced governor

control valves stem binding identified stem corrosion following valve disassembly. Ginna

Enclosure

15

had originally determined that NRC IN 94-66 was not applicable to Ginna because the

governor control valve was of a different configuration and material as those discussed

in the IN and that corrosion of the stem had not been previously identified. Although this

may have been a reasonable assessment during the original review of the NRC IN 94-66

in 1994, the inspectors determined that once corrosion of the stem was identified in'May

2005, Ginna missed an opportunity to revisit this operating experience. In addition, as

discussed in Section 2.1.2 of this report. Ginna did not utilize Industry operating

experience in evaluating a condition associated with steam admission valve leakage.

NRC IN 94-66, other industry operating experience, and the vendor technical manual

stated that steam admission valve leakage is a cause related to governor control valve

stem binding. However, Ginna did not consider the impact of steam admission valve

leakage on the governor control valve performance. Finally, the inspectors noted that

Ginna missed opportunities to incorporate industry operating experience during

troubleshooting efforts during the May 2009 TDAFW pump failure. NRC IN 94-66

discussed several examples where a freedom of motion test was performed to test for

stem binding with the governor control valve linkage disconnected and hydraulics not

applied. However, during the May 2009 event, Ginna missed an opportunity to conduct

this test and consequently, missed an opportunity to potentially identify the failure mode

prior to the July 2, 2009, overspeed trip. Ginna captured these issues in their CAP

under CR-2009-003680 and CR-2009-004577..

2.3 Review of Root Cause and Extent-ot-Condition

a. Inspection Scope

The team evaluated the adequacy of Ginna's RCA and completed interim corrective

actions. In addition the team evaluated the adequacy of Ginna's initial extent of

condition for the TDAFW pump failures. The team reviewed plant drawings, procedures,

and associated system modifications. In addition, the team conducted a walkdown of

the TDAFW system and interviewed key Ginna personnel.

b. Findings and Observations

Introduction: The inspectors identified an NCV of 10 CFR 50, Appendix B, Criterion III,

"Design Control," for the failure to establish measures to ensure that a modification

performed on the governor control valve bushing was a suitable application of materials

for the TDAFW pump.

Description: During the troubleshooting and repair of the TDAFW pump in July 2009,

Ginna incorrectly evaluated that a vendor service bulletin allowed for larger stem-to

bushing clearance than was prescribed in their current technical documents. Since

there was overlap between the station technical information and the vendor service

bulletin, Ginna decided to increase the clearance, by lapping the bushings, to reduce the

likelihood that corrosion products would cause the stem to bind. To increase the

clearance, Ginna removed some of the hardened layer on the bushing. During a review

of the RCA, the inspectors determined that Ginna did not consider the potential impact of

removing some ot the hardened layer on the corrosion rate of the governor control valve.

The inspectors noted that by removing some of the hardened layer, the material property

of the bushing could have been changed and/or additional micro-cracks could have been

created or exposed that could increase the corrosion rate or potentially cause other

Enclosure

16

problems such as stem-to~bushing galling. The inspectors noted that if the corrosion

rate is greater than predicted, then planned corrective actions may not be adequate to

maintain the TDAFW pump operable. Additionally, the team was concerned, that by

increasing the stem to bushing clearances, the stem would be susceptible to larger

steam exposure and that this could further accelerate the corrosion. Following concerns

raised by the inspectors, Ginna initiated CR-2009-005959 to ensure that all possible

consequences of lapping the TDAFW pump control valve bushing were fully evaluated.

In addition, Ginna assigned corrective actions to inspect the control valve bushing during

the fall 2009 refueling outage. During performance of the refueling inspection under WO

C90623685, pitting of the stem and corrosion of the bushings were observed.

Constellation initiated CR~2009-006765 and noted that the corrosion of the bushings

appeared to have been caused by the lapping of the bushing to achieve the desired

clearance between the stem and the bushings. The development of corrosion on the

bushing reduced the margin that was predicted to ensure adequate stem movement.

Immediate corrective actions following the inspection of the governor control valve

during the fall 2009 refueling outage included entering this condition in the CAP and

refurbishing the governor control valve with a new stem and bushing.

Analysis: The performance deficiency is that Ginna failed to establish measures to

ensure that a modification performed on the governor control valve bushing was a

suitable application of materials for the TDAFW pump. The finding is more than minor

because it is associated with the equipment performance attribute of the Mitigating

Systems cornerstone and affects the cornerstone objective to ensure the availability and

reliability of systems that respond to initiating events to prevent undesirable

consequences. Specifically, lapping of the valve bushing resulted in an unanticipated

corrosion mechanism of the governor control valve that impacted the reliability of the

TDAFW pump. The inspectors evaluated the Significance of this finding using IMC 0609,

Attachment 4, "Phase 1 -Initial Screening and Characterization of Findings." The

finding is of very low safety significance because it is not a design or qualification

deficiency, did not represent a loss of a safety fUnction of a system or a single train

greater than its TS allowed outage time, and did not screen as potentially risk significant

due to external events.

The inspectors determined that this finding has a cross-cutting aspect in the area of

Human Performance, Decision Making, because Ginna did not make a safety-significant

or risk-significant decision using a systematic process, especially when faced with

uncertain or unexpected plant conditions, to ensure safety was maintained. Specifically,

Ginna did not use a systematic process such as an engineering evaluation to properly

evaluate the potential impact of removing some of the hardened layer of the bushing.

H.1(a) per IMC 0305]

Enforcement: 10 CFR 50, Appendix B, Criterion III, "Design Control" states, in part,

"Measures shall also be established for the selection and review for suitability of

application of materials, parts, equipment, and processes that are essential to the safety

related functions of the structures, systems and components." Contrary to the

requirements, in July 2009, Ginna failed to establish measures to ensure that a

modification performed on the governor control valve bushing was a suitable application

of materials for the TDAFW pump. Specifically, lapping of the valve bushing resulted in

an unanticipated corrosion mechanism of the governor control valve that had the

Enclosure

17

potential to impact the performance of the TDAFW pump if not corrected. Because this

violation is of very low safety significance (Green) and Ginna entered this issue into their

CAP for resolution as CR-2009-006765, this violation is being treated as an NCV

consistent with the NRC Enforcement Policy. (NeV 05000244/2009008-03: Failure To

Establish Design Control Measures Associated With The Turbine Driven Auxiliary

Feedwater Pump Governor Control Valve)

2.4 Risk Assessment of the As-Found Condition

a. Insoection Scope

Prior to the initiation of the Special Inspection Team, the Region I SRA performed a

CCDP assessment which conservatively bounded the potential risk significance of the

degraded condition, assuming the TDAFW pump would fail to star on demand to

mitigate the consequences of an event. The initial CCDP estimate was performed in

accordance with IMC 0309, "Reactive Inspection Decision Basis for Reactors." The SRA

used the Ginna SPAR model and associated GEM to evaluate the potential risk

significance of this condition. The results of the IMC 0309 assessment are documented

in report section 2.1.b.1 above.

b. Final Risk Estimate

Following team review and independent verification of the degraded turbine driven

auxiliary feed water pump governor control valve, the team concluded that for

approximately 119B hours, the TDAFW pump would not have been capable of

responding to an event. Consistent with IMC 0609 conditional core damage probability

assessment methodology, this degraded condition resulted in a loss of operability or

safety function, and therefore was of low to moderate (B.6E-6) safety significance.

4. OTHER ACTIVITIeS

40A6 Meetings, Including Exit

On August 27 and on October 7,2009 the team presented the inspection results to Mr.

John Carlin and other members of the R. E. Ginna Nuclear Power Plant staff. The

inspectors verified that none of the materiall~xamined during the inspection is

considered proprietary in nature.

Enclosure

A-1

ATTACHMENT A

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee personnel

J. Carlin, Site Vice President

E. Larson, Plant General Manager

D. Crowley, Senior Engineer

E. Durkish, Associate Engineer

R. Everett, Supervisor, Primary Systems Engineering

T. Harding, Director, Licensing

R. Ruby, Principal Engineer, Licensing

P. Swift, Engineering Manager

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000244/2009008-01 AV Failure to Preclude Recurrence of a

Significant Condition Adverse to Quality

Associated with the Turbine Driven Auxiliary

Feedwater Pump Governor Control Valve.

(Section 2.1.1)

Opened/Closed

05000244/2009008-02 NCV Inadequate Corrective Actions Associated

with Steam Admission Valve Leakage.

(Section 2.1.2)05000244/2009008-03 NCV Failure to Establish Design Control

Measures Associated with the Turbine

Driven Auxiliary Feedwater Pump Governor

Control Valve. (Section 2.3)

I.IST OF DOCUMENTS REVIEWED

In addition to the documents identified in the body of this report, the inspectors reviewed the

following documents and records:

Corrective Actions

CA-2009-002311

Condition Reports

CR-2003-2006 CR-2005-3660 CR-2006-006029 CR-2006-006341

CR-2005-1509 CR-2005-5513 CR-2006-006204 CR-2006-006962

Attachment A

A-2

CR-2007-000876 CR-2008-009911 CR-2009-004590 CR-2009-005959*

CR-2007-001302 CR-2009-003680 CR-2009-004591 CR-2009-006765*

CR-2008-003687 CR-2009-004222* CR-2009-005964

CR-2008-007541 CR-2009-004577

• NRC Identified During Inspection

Corrective Action Tracking System

CATS R007325

Procedures

AP-FVV.1, Abnormal MFW Pump Flow or NPSH, Revision 01702

PT-16QT-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05900

CNG-MN-1.01-1002, Troubleshooting, Revision 0001

CNG-CA-1.01, Corrective Action Program, Revision 0001

CNG-CA-1.01-1004, Root Cause Analysis, Revision 0001

CNG-CA-1.01-1005, Apparent Cause Evaluation, Revision 0100

CNG-CA-1.01-1006, Common Cause Analysis, Revision 0001

CNG-CA-1.01-1007, Trending, Revision 0000

CNG-CA-1.01-1010, Use of OE, Revision 0000

IP-CAP-1, Ginna Condition Reporting, Revision 02800

M-11.5C, Auxiliary Feedwater Pump Motor Mechanical Inspection and Maintenance,

Revision 03000

0-1.1, Plant Heatup From Cold Shutdown to Hot Shutdown, Revision 16301

PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05600. Completed

3/14/08,514108,5/5/08,6/11/08,9/4/08, 1213/08, 1214/08, 12111/08, 12/18/08,2/12/09,

5/26/09, 5/28/09

Surveillance Tests

PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05600, 03/14/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump -. Quarterly, Revision 05700, 05/04/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05700,05/05/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05700. 06/11/2008

STP-0-16-COMP-T, Auxiliary Feedwater Turbine Pump - Comprehensive Test,

Revision 00000, 06/11/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly. Revision 05701, 09/03/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05701, 12/02/2008

PT-16Q-T, Auxiliary FeedwaterTurbine Pump - Quarterly. Revision 05701,12103/2008

PT-16Q-T. Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05701, 12/04/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05702, 12/11/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05702, 12/18/2008

PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05801, 01116/2009

PT-16Q-T, AUXiliary Feedwater Turbine Pump - Quarterly, Revision 05801, 02112/2009

PT-16Q-T, AUXiliary Feedwater Turbine Pump _. Quarterly. Revision 05900,05/26/2009

PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05900,05/28/2009

STP-0-16-COMP-T, Auxiliary Feedwater Turbine Pump - Comprehensive Test,

Revision 00400, 05/28/2009

Attachment A

A-3

Work Orders

W020604989

W020401907

Drawings

DWG No. 33013-1231, Main Steam System, Revision 37

DWG No. 33013-2285, Motor Driven and Turbine Driven Auxiliary Feedwater Pumps

Lube Oil Skid, Revision 17

DWG No. LB-112541, Turbine Control and Quick Start System

DWG No. T040-001A, TDAFW Control Oil System, Revision 0

DWG No. T040-002A, TDAFW Turbine Trip Valve, Revision 0

DWG No. T040-003A, Auxiliary and Standby Aux. Feedwater Systems, Revision 2

DWG No. T040-003B, SAFW System 1-Line Diagram, Revision 0

DWG No. T040-003C, AFW System 1-Une Diagram, Revision 1

DWG No. T040-004A, Auxiliary Feedwater Pump Auto Start Signals, Revision 0

DWG No. T830-004A, Blowdown Isolation Solenoid Control Circuit, Revision 0

Other Documents

Vendor Report, Preliminary Results ofStem Sticking Failure Mechanisms, July 20, 2009

Vendor Report 09-1929 Part 1, Equipment Root Cause Analysis of AFWP Control Valve

Stem Sticking Problem at Ginna Nuclear Power Station

Vendor Technical Question Response, August 26, 2009

Category I Root Causal Analysis, May and July 2009 Turbine Driven Auxiliary Feedwater

Pump (TDAFWP) Failures; Overspeed Trip During Testing

vrD-E9016-4001, Excerpts from EPRI Manual 1007461, Terry Turbine Maintenance

Guide, AFW Application, Revision 000

vrD-G0153-4001, Operating Instructions Easy Flow Body Combined Trip Throttle Valve,

Revision 2

VrD-W0315-4001, Instructions For 465 H.P. Non-Condensing Steam Turbine Serial

Number 26635, Revision 000

VrD-W0315-4002, Service Department Standards Book No. 10, Field Service Manual,

Revision 000

Response to NRC Generic Letter 90-3, Relaxation of Staff Position In Generic Letter 83

28, Item 2.2 Part 2 "Vendor Interface for Safety Related Components, dated

September 18, 1990

CATS 10 R04451, Response to NRC IN 94-66, OVERSPEED OF TURBINE-DRIVEN

PUMPS CAUSED BY GOVERNOR VALVE STEM BINDING

CMM-37-19-9519E, Worthington Turbine Driven Auxiliary Feedwater Pump Hydraulic

Governor Control Valve Maintenance for 9519E, Revision 00200

Constellation Energy Nuclear Generating Group Reply to a Notice of Violation; EA-09

045

Engineering Change Package No. ECP-2009-0146, TDAFW Lube Oil Orifice

Management Review Committee Agenda. August 5, 2009

Purchase Requisition No. 58686, Dresser-Rand Valve Stem

Technical Staff Request 97-199, Leakoff from TDAFW Govenor Valve 9519E

Management Review Committee Agenda, August 5, 2009

Purchase Requisition No. 58686, Dresser-Rand Valve Stem

Attachment A

A-4

Technical Staff Request 97-199, Leakofffrom TDAFW Govenor Valve 9519E

Turbine Driven Auxiliary Feedwater Pump Preventive Maintenance Strategies

Auxiliary Feed Water System, 1st Quarter 200S

Auxiliary Feed Water System. 2nd Quarter 200S

Auxiliary Feed Water System. 3rd Quarter 200S

Auxiliary Feed Water System, 4th Quarter 200S

Auxiliary Feed Water System, 1st Quarter 2009

Operating Experience

OE-200S-000397

OE-200S-000607

OE-200S-000847

oE-20OS-00S60

OE-2008-001296

OE-2009-00212

OE*2009*001178

Oil Analysis

Pump IB Oil Analysis. 2008-03

Pump OB Oil Analysis, 2008-03

Reservoir Oil Analysis 2008-05

Reservoir Oil Analysis 2008-08

Reservoir Oil Analysis 2009-05

Attachment A

A-4

LIST OF ACRONYMS

AV Apparent Violation

CAP Corrective Action Program

CCDP Conditional Core Damage Probability

CDF Core Damage Frequency

CR Condition Report

DRP Division of Reactor Projects

EDG Emergency Diesel Generator

GEM Graphical Evaluation Module

ICCDP Incremental Conditional Core Damage Probability

IMC Inspection Manual Chapter

IN Information Notice

INL Idaho National Labs

• IPEEE Individual Plant Examination of External Events

LERF Large Early Release Frequency

LOOP Loss of Offsite Power

LOSWS Loss of Service Water System

NCV Non Cited Violation

NRC Nuclear Regulatory Commission

NRR Office of Nuclear Reactor Regulation

PARS Publicly Available Records

PSA Probabilistic Safety Assessment

RCA Root Cause Analysis

SAFW Standby Auxiliary Feedwater

SBO Station Blackout

SOP Significance Determination Process

SPAR Standardized Plant Analysis Risk

SRA Senior Reactor Analyst

SCAQ Significant Condition Adverse to Quality

NCV Non-citied Violation

TBD To Be Determined

TDAFW Turbine Driven Auxiliary Feedwater

TS Technical Specification

Attachment A

8-1

Special Inspection Charter

R.E. Ginna Nuclear Power Plant

Failure of the Turbine-Driven Auxiliary Feedwater (TDAFW) Pump

on May 26, 2009 and July 2, 2009

Background:

On May 26, 2009, during routine quarterly surveillance testing of the turbine driven

auxiliary feedwater (TDAFW) system, the TDAFW pump tripped on overspeed. The test

was repeated several hours later and the TDAFW pump tripped again on overspeed.

Fluctuations in oil pressure were observed during the testing.

After extensive troubleshooting, Ginna personnE~1 identified a number of issues but no

definitive cause for the overspeed trips. The lube/control oil system was drained and

cleaned due to the presence of fine particulates; a pressure pulsation dampener

accumulator bladder was replaced due to a below normal pressure condition; the oil

pressure regulating bypass valve was replaced due to potential cycling; a change to

linkage setup was implemented; governor relay valve, and trip and throttle valve parts

were replaced due to out of specification clearances; and a miSSing oil line orifice was

installed. Following these corrective actions, the system was successfully tested and

declared operable.

Constellation is completing a root cause evaluation and the resident inspectors continue

to conduct baseline inspections with assistance from DRS specialists. Testing of the

TDAFW system will be at an increased frequency until Constellation has confidence that

the system is performing as required.

On July 2, 2009, during the performance of the increased frequency surveillance, the

TDAFW pump again tripped on overspeed. Glnna personnel discovered pitting and

corrosion on the stem of the governor control valve during troubleshooting activities to

restore the pump to operable status. This Special Inspection Team had raised concerns

regarding corrosion on the stem of the Governor Control Valve during onsite inspection

activities the week of June 15, 2009.

Basis for the Formation of the SIT:

The failure of the TDAFW pump involved repetitive failures of this safety~related

eqUipment. There have been three failures of the TDAFW pump since December 2008.

On December 2, 2008, a failure was attributed to inadequate implementation of the

preventive maintenance program. SpeCifically, the governor linkages were not

lubricated in March 2008 which resulted in the pump's inability to achieve the required

flow and pressure in December 2008. On July 2, 2009, the failure was preliminarily

determined to be caused by pitting on the stem of the Governor Control valve and overly

tolerances causing binding of the stem. Although the specific failure modes (overspeed

trip, inability to achieve the required flow or pressure) were different, the underlying

cause of inadequate or inappropriate preventive maintenance may be the same.

Attachment 8

B-2

Based upon best available information, the Region I Senior Reactor Analyst (SRA)

conducted a preliminary risk estimate of the May 26 TDAFW pump failure. An

incremental conditional core damage probability (ICCDP) in the upper E-6 range (8E-6

per the 82 day exposure period), was calculated using the Ginna SPAR model.

assuming that the TDAFW pump would not have started since the last time it passed a

surveillance test on March 5 until May 26 (82 days). The dominant core damage

sequence was a station blackout (LOOP with failure of both EDGs,) with no TDAFW and

failure to recover offsite power or an EDG in one hour.

Based upon the preliminary conditional core damage probability estimate of upper E-6

range, in accordance with IMC 0309, this event falls within the region for a Special

Inspection Team.

Oblectives of the Special Inspection:

The objectives of the special inspection are to review and assess: (1) Constellation's

planning and execution of the risk significant work activities on the TDAFW system; (2)

equipment issues related to the TDAFW testing; and (3) Constellation's response to this

significant equipment failure.

To accomplish these objectives, the following will be performed:

1. Evaluate the adequacy and completeness of the maintenance on the TDAFW

system, including preventive maintenance, procedural guidance, post

maintenance testing, and supervisory oversight.

2. Evaluate Constellation's application of pertinent industry operating experience

and evaluation of potential precursors, including the effectiveness of any actions

taken in response to the operating experience or precursors.

3. Evaluate the adequacy of Constellation's response to the TDAFW system

failures, including Constellation's cause analysis and completed interim corrective

actions.

4. Evaluate the adequacy of Constellation's initial extent of condition for the TDAFW

failures, as appropriate.

5. Evaluate the failure modes for potential generic implications including the need

for generic communications

Additionally, the team leader will review lessons learned from the Special Inspection

and, if appropriate, prepare a feedback form on recommendations for revising the

reactor oversight process (ROP) baseline inspection procedures in order to proactively

identify the issues and causes involved with the event.

Guidance:

Inspection Procedure 93812, "Special Inspection", provides additional guidance to be

used by the Special Inspection Team. Team duties will be as described in Inspection

Procedure 93812. The inspection should emphasize fact-finding in its review of the

circumstances surrounding the event. It is not the responsibility of the team to examine

Attachment B

8-3

the regulatory process. Safety concerns identified that are not directly related to the

event should be reported to the Region I office for appropriate action.

The Team will conduct an entrance meeting and begin the inspection on June 8,2009.

While on site, the Team Leader wi" provide daily briefings to Region I management, who

will coordinate with the Office of Nuclear Reactor Regulation, to ensure that all other

parties are kept informed. A report documenting the results of the inspection should be

issued within 45 days of the completion of the inspection.

This Charter may be modified should the team develop significant new information that

warrants review.

Attachment 8