ML093160122
ML093160122 | |
Person / Time | |
---|---|
Site: | Ginna |
Issue date: | 11/12/2009 |
From: | David Lew Division Reactor Projects I |
To: | John Carlin Ginna |
Clifford J | |
References | |
EA-09-249 IR-09-008 | |
Download: ML093160122 (29) | |
See also: IR 05000244/2009008
Text
UNITED STATES
NUCLEAR REGULATORY COMMISSION
REGION I
475 ALLENDALE ROAD
KING OF PRUSSIA, PA 19406-1415
November 12, 2009
Mr. John T. Carlin
Vice President, RE. Ginna Nuclear Power Plant
RE. Ginna Nuclear Power Plant, LLC
1503 Lake Road
Ontario, New York 14519
SUBJECT: RE. GINNA NUCLEAR POWER PLANT - NRC SPECIAL INSPECTION TEAM
REPORT 05000244/2009008; PRELIMINARY WHITE FINDING
Dear Mr. Carlin:
On June 16-20, 2009, and August 24-27,2009, the U.S. Nuclear Regulatory Commission (NRC)
conducted the onsite portions of a special inspection at RE. Ginna Nuclear Power Plant. In
office inspection reviews were conducted in the intervening weeks. The enclosed report
documents the inspection team's findings and observations which were discussed with you and
others members of your staff on August 27, 2009, during a preliminary exit briefing, and on
October 7, 2009, during the final exit meeting.
The special inspection was conducted in response to a turbine driven auxiliary feedwater
(TDAFW) pump overspeed trip on May 26, 2009. The team included a subsequent overspeed
trip on July 2,2009 into its inspection scope. The NRC's initial evaluation of this condition
satisfied the criteria in NRC Inspection Manual Chapter 0309, "Reactive Inspection Decision
Basis for Reactors," for conducting a special inspection. The basis for initiating this special
inspection is further discussed in the inspection team's charter that is included in this report as
Attachment B.
The inspection examined activities conducted under your license as they relate to safety and
compliance with the Commission's rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed
personnel.
This letter transmits one self-revealing finding that, using the reactor safety Significance
Determination Process (SOP), has preliminarily been determined to be White, a finding with low
to moderate safety significance. The finding is associated with the failure to preclude
recurrence of a significant condition adverse to quality associated with the corrosion of the
governor control valve of the turbine driven auxiliary feedwater (TDAFW) pump, that led to a
failure of the TDAFW pump during surveillance testing on July 2, 2009 and was the likely cause
of the overspeed trip on May 26, 2009. Following the July test failure, Ginna replaced the
J. Carlin 2
governor control valve stem and conducted weekly monitoring of the governor control valve
during surveillance testing to identify any potential for stem binding. In addition, corrective
actions included a follow-up inspection of the governor control valve during the fall 2009
refueling outage. There is no immediate safety concern present due to this finding because the
system is now operable and the long term corrective actions are being implemented in Ginna's
corrective action program. The final resolution of this finding will be conveyed in a separate
correspondence.
As discussed in the attached Inspection report. the tinding is also an apparent violation of NRC
requirements. specifically, 10 CFR Appendix B, Criterion XVI. "Corrective Actions." and is
therefore being considered for escalated enforcement action in accordance with the
Enforcement Policy, which can be found on the NRC's Web site at http://www.nrc.gov/reading
rm/doc-collections/enforcementl.
Following a discussion of the preliminary safety Significance of this finding during the initial exit
briefing on August 27,2009. a phone call was held between Glenn Dentel, Branch Chief,
Division of Reactor Projects. and yourself on October 1. 2009. During this call, you indicated
that R.E. Ginna Nuclear Power Plant does not contest the characterization of the rIsk
significance of this finding, and therefore you have declined to further discuss this issue at a
Regulatory Conference or provide a written response. Please note that by declining to request
a Regulatory Conference or submit a written response, you relinquished your right to appeal the
final SOP determination. in that by not doing either, you would not meet the appeal
requirements stated in the Prerequisite and Limitation sections of Attachment 2 of IMC 0609.
You will be advised by a separate correspondence of the results of our deliberations on this
matter.
In addition. the report documents two findings of very low safety significance (Green). The
findings involved violations of NRC requirements. However, because of the very low safety
significance and because they are entered into your corrective action program (CAP). the NRC
is treating these findings as non-cited violations (NeVs) consistent with Section VI.A.1 of the
NRC Enforcement Policy. If you contest any NCV in this report, you should provide a response
within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear
Regulatory Commission, A TIN.: Document Control Desk, Washington, DC 20555-0001; with
copies to the Regional Administrator, Region I; the Director, Office of Enforcement; and the
NRC Resident Inspector at R.E. Ginna. In addition. if you disagree with the characterization of
any finding in this report, you should provide a response within 30 days of the date of this
inspection report. with the basis for your disagreement, to the Regional Administrator, Region I,
and the NRC Resident Inspector at R.E. Ginna Nuclear Power Plant. The information you
provide will be considered in accordance with Inspection Manual Chapter 030?
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its
enclosure, and your response (if any) will be available electronically for public inspection in the
J. Carlin 3
NRC Public Document Room or from the Publicly Available Records (PARS) component of the
NRC's document system (ADAMS). ADAMS is accessible from the NRC Web Site at
http://www.nrc.gov/reading-rm/adams.html(the Public Electronic Reading Room).
Sincerely,
,/;I,/;tJ
D~ew,~~ --fbr-
Division of Reactor Projects
Docket No.: 50-244
License No.: DPR-18
Enclosures: Inspection Report 05000244/2009008
w/Attachment A: Supplemental Information
w/Attachment B: Special Inspection Charter
cc w/encl: Distribution via ListServ
J. Carlin 2
NRC Public Document Room or from the Publicly Available Records (PARS) component of the
NRC's document system (ADAMS). ADAMS is accessible from the NRC Web Site at
httg:llwww.nrc.gov/reading*rm/adams.html(the Public Electronic Reading Room).
Sincerely,
IRAJ James W. Clifford for:
David C. lew, Director
Division of Reactor Projects
Docket No.: 50-244
License No.: DPR-18
Enclosures: Inspection Report 05000244/2009008
w/Attachment A: Supplemental Information
w/Attachment B: Special Inspection Charter
cc w/encl: Distribution via ListServ
SUNSI Review Complete: gtd (Reviewer's Initials)
DOCUMENT NAME: G:\DRP\BRANCH1\Ginna\TDAFW SIT\AED Track Changes Ginna SIT
Report Rev 4 cC.doc
After declaring this document "An Official Agency Record" it will be released to the Public.
To receive I!I copy of this document, indicate in the box: "C' =Copy without attachmenVenclosure "e" = Copy with
attaclhmentlenclosure "N";:; No co ML093160122
OFFICE RI/DRS RI/ENF RI/DRP
NAME CCahill/cc* JClifford/jwc
DATE 10/21109 11/05/09
- see prior concurrence OFFICIAL RECORD COpy
Distribution w/encl:
S. Collins, RA (R10RAMaii Resource) ROPreports@nrc.gov
M. Dapas, ORA (R10RAMaii Resource)
D. Lew, DRP (R1DRPMaii Resource)
J. Clifford, DRP (R1DRPMail Resource)
l. Trocine, RI OEDO
RidsNrrPMREGinnaResource
RidsNrrDorlLpl1-1 Resource
G. Dentel, DRP
N. Perry, DRP
J. Hawkins, DRP
L. Casey, DRP, RI
M. Rose, DRP, OA
D. Bearde, DRP
1
U.S. NUCLEAR REGULATORY COMMISSION
REGION I
Docket No.: 50-244
License No.: DPR-18
Report No.: 05000244/2009008
Licensee: Constellation Energy, R. E. Ginna Nuclear Power Plant, LLC
Facility: R. E. Ginna Nuclear Power Plant
Location: Ontario. New York
Dates: June 15-19, 2009 and August 24-27.2009
Team Leader: C. cahill, Senior Reactor Analyst, Division of Reactor Safety
Inspectors: S. Kennedy, Senior Resident Inspector, Division of Reactor Projects
K. Kolaczyk, Senior Resident Inspector, Division of Reactor Projects
J. Bream, Project Engineer. Division of Reactor Projects
Approved by: Glenn T. Dentel, Chief
Projects Branch 1
Division of Reactor Projects
2
SUMMARY OF FINDINGS
IR 05000244/2009008; 06/15-19/2009 and 08/24-2'712009; R. E. Ginna Nuclear Plant; Special
Inspection Team Report.
The report covered two on-site inspection visits and related in-office inspection activities by a
special inspection team consisting of a Senior Reactor Analyst. two Senior Resident Inspectors.
and one Project Engineer. One apparent violation (AV) with potential for greater than Green
safety significance and two Green findings were identified. The significance of most findings is
indicated by its color (Green. White, Yellow, or Red) using Inspection Manual Chapter (lMC)
0609, "Significance Determination Process." Findings for which the significance determination
(SDP) process does not apply may be Green or be assigned a severity level after NRC
management review. The NRC program for overseeing the safe operation of commercial
nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4,
dated December 2006.
NRC Identified and Self-Revealing Findings
Cornerstones: Mitigating Systems
Preliminary White: A self-revealing apparent violation (AV) of 10 CFR 50, Appendix B,
Criterion XVI, uCorrective Actions," was identified for the failure to preclude recurrence
of a signIficant condition adverse to quality (SCAQ) associated with the Turbine Driven
Auxiliary Feedwater (TDAFW) pump governor control valve. Specifically, after
identifying corrosion of the govemor control valve stem in April 2005, Ginna did not take
adequate corrective actions to preclude the recurrence of corrosion which led to the
binding of the gavemar control valve and failure of the TDAFW pump on July 2,2009.
In addition, the inspectors concluded that governor control valve stem binding was the
likely cause of the failure of the TDAFW pump on May 26,2009. The overspeed trip of
the TDAFW pump on May 26. 2009. was originally determined by Ginna to be failure of
the governor control system relay valve. Governor control valve stem corrosion is a
SCAQ because corrosion of the stem can IHad to governor control valve stem binding
and failure of the TDAFW pump as discussed in NRC Information Notice (IN) 94-66:
"Overspeed of Turbine-Driven Pumps Caused by Governor Valve Stem Binding" and
other related industry operating experience documents. Immediate corrective actions
inCluded entering this condition in the corrective action program (CAP). conducting a
root cause analysis (RCA). replacing the governor control valve stem, and conducting
weekly monitoring of the governor control valve during surveillance testing to identify
any potential for stem binding. In addition, corrective actions included a follow-up
inspection of the governor control valve during the fall 2009 refueling outage. Ginna will
continue to monitor the govemor control valve under an enhanced TDAFW surveillance
program to ensure TDAFW pump operability.
The finding is more than minor because it is associated with the equipment performance
attribute of the Mitigating System cornerstone and affects the cornerstone objective to
ensure the availability, reliability, and capability of systems that respond to initiating
events to prevent undesirable consequences. Specifically. stem corrosion caused
binding of the governor control valve and led to the failure of the TDAFW pump. This
finding was assessed using IMC 0609 and preliminarily determined to be White (lOW to
moderate safety significance) based on a Phase 3 analysis with a total (internal and
Enclosure
3
external contributions) calculated conditional core damage frequency (CCDF) of B.6E-6.
This finding has a cross-cutting aspect in the area of Problem Identification and
Resolution, Corrective Action Program, because Ginna did not implement a corrective
action program with a low threshold for identifying issues completely, accurately, and in
a timely manner commensurate with their safety significance P.1(a} per IMC 0305].
Specifically, Ginna did not identify issues a~;sociated with corrosion of the governor
control valve within the corrective action program. (Section 2.1.1)
Green: A self-revealing non-cited violation (NCV) of 10 CFR 50, Appendix B, Criterion
XVI, "Corrective Actions," was identified for the failure to establish adequate measures
to correct a longstanding issue associated with steam admission valves leakage. As a
result, the leakage most likely contributed to the build-up of corrosion on the TDAFW
pump governor control valve stem and contributed to the failure of the TDAFW pump on
May 26, 2009, and on July 2, 2009. The steam admission valves had been leaking
since at least 2005. However, G.inna did not take adequate measures to correct the
leakage or minimize the impact of the leakage on governor control valve performance.
Immediate corrective actions included entering this condition in the corrective action
program, conducting a root cause analysis, replacing the governor control valve stem,
and conducting weekly monitoring of the governor control valve during surveillance
testing to identify any potential for stem binding. Additionally, the steam admission
valves were inspected and re-worked and the governor control valve was inspected
during the fall 2009 outage. Ginna will continue to monitor the governor control valve
under an enhanced TDAFW surveillance program to ensure TDAFW pump operability.
Planned corrective actions include replacin~~ the steam admission valves in May 2011.
The finding is more than minor because it is associated with the equipment performance
attribute of the Mitigating Systems cornerstone and affects the cornerstone objective to
ensure the availability and reliability of systems that respond to initiating events to
prevent undesirable consequences. Specifically, leakage through the steam admission
valves can result in continuous wetting of the governor control valve stem and lead to or
accelerate corrosion of the governor control valve. This could result in a stem binding of
the governor control valve and failure of the TDAFW pump. The inspectors evaluated
the signifiCance of this finding using IMC 0609, Attachment 4, "'Phase 1 - Initial
Screening and Characterization of Findings." The finding is of very low safety
significance because it is not a design or qualification deficiency, did not represent a
loss of a safety function of a system or a single train greater than its technical
speCification (TS) allowed outage time, and did not screen as potentially risk Significant
due to external events. This finding has a cross-cutting aspect in the area of Problem
Identification and Resolution, Corrective Action Program, because Ginna did not
thoroughly evaluate problems such that the resolutions address causes and extent of
conditions, as necessary, In a timely manner, commensurate with their significance
[P.1 (c) per IMC 0305]. Specifically, Ginna did not thoroughly evaluate the potential
effect of the steam admission valve leakage on the governor control valve performance.
(Section 2.1.2)
Green: The inspectors identified an NCVof 10 CFR 50, Appendix B. Criterion III,
"Design Control," for the failure to establish measures to ensure that a modification
performed on the governor control valve bushing was a suitable application of materials
for the TDAFW pump. During a review of the RCA associated with the TDAFW pump
failures, the inspectors noted that Ginna did not consider the potential impact of
Enclosure
4
removing some of the hardened layer of the bushing on the corrosion rate of the
governor control valve. Following concerns raised by the inspectors, Ginna inspected
the governor control valve bushing during the fall 2009 refueling outage and observed
corrosion of the bushings. Ginna noted that the corrosion of the bushings appeared to
have been caused by the lapping of the bushing to achieve the increased clearance
between the stem and the bushings. Immediate corrective actions following the
inspection of the governor control valve during the fall 2009 refueling outage included
entering this condition in the CAP and refurbishing the governor control valve with a new
stem and bushing.
The finding is more than minor because it is associated with the equipment performance
attribute of the Mitigating System cornerstone and affects the cornerstone objective to
ensure the availability, reliability, and capability of systems that respond to initiating
events to prevent undesirable consequences. Specifically, lapping of the valve bushing
resulted in an unanticipated corrosion mechanism of the govemor control valve that
impacted the reliability of the TDAFW pump" The inspectors evaluated the significance
of this finding using IMC 0609, Attachment 4. "Phase 1 Initial Screening and
Characterization of Findings." The finding is of very low safety significance because it is
a design or qualification deficiency confirmed not to result in the loss of operability or
functionality. The inspectors determined that this finding has a cross-cutting aspect in
the area of Human Performance, Decision Making. because Ginna did not make a
safety-significant or risk-significant decision using a systematic process, especially when
faced with uncertain or unexpected plant conditions, to ensure safety was maintained
H.1(a) per IMC 0305]. Specifically, Ginna did not use a systematic process such as an
engineering evaluation to properly evaluate the potential impact of removing some of the
hardened layer of the bushing. [H.1 (a) per IMC 0305]
(Section 2.3)
Licensee Identified Violations
None
Enclosure
5
REPORT DETAILS
1. INTRODUCTION
1.1 Background and Event Description
On May 26, 2009, and July 2, 2009, the TDAFW pump tripped on overspeed during the
performance of surveillance testing activities. These were the second and third times
the TDAFW pump had failed a surveillance test in a six month period. The previous test
failure occurred on December 2, 2008, when the TDAFW pump failed to develop
acceptable discharge pressure during a quarterly surveillance test. Details surrounding
the December test failure including Ginna corrective actions are discussed in NRC
Integrated Inspection Report 50-244/2009002. Following each of the three test failures,
Ginna declared the TDAFW pump inoperable and entered the limiting Condition for
Operation for TS 3.7.5 "Auxiliary Feedwater."
Following the May 26, 2009, test failure, Ginna troubleshooting activities were focused
on the TDAFW lubricating oil system where system pressure oscillations were noted by
test personnel prior to the pump overspeed trip. As part of the troubleshooting efforts,
the TDAFW lubricating oil system was drained and refilled with fresh oil, filters in the
lubrication system were examined and replaced, and several components in the
lubricating oil system including the relay and pressure regulating valves were replaced.
Additionally the turbine governor linkage was adjusted to original specifications outlined
in the vendor technical documents. Although several out of speCification and missing
components were identified during this troubleshooting effort, a definitive cause for the
surveillance failure was not identified. Nevertheless, following the successful completion
of post maintenance testing activities, Ginna declared the TDAFW pump operable and
commenced an augmented surveillance testing program for the TDAFW pump that
tested the pump on a weekly basis.
On July 2, 2009, another TDAFW pump overspeed failure occurred when the pump was
undergoing the augmented testing program. Troubleshooting activities following this
failure were focused on the turbine control system, which did not appear to have
functioned properly during the test. Accordingly, the turbine control and relay valves
were disassembled and inspected, and the governor linkage system was adjusted.
Additionally. the lubricating oil system was flushed and examined for particulates.
However, unlike the May 26 failure, these troubleshooting activities identified a definitive
cause, a stuck governor control valve stem, that resulted in the surveillance test failure.
Visual inspection of the stem, which had become bound to its bushing and had to be
forcibly removed, revealed corrosion buildup where the stem contacted the upper valve
bushing. To restore the control valve to an operable status, Ginna replaced the valve
stem and increased the bushing clearances to dimensions specified by the Ginna
TDAFW pump inspection procedure. The pump was successfully tested and declared
operable subject to an augmented testing program.
Ginna assigned an RCA team to investigate the surveillance test failures. The Ginna
RCA team concluded that the primary root cause of the failure of the governor contrOl
valve to control turbine speed on July 2,2009, was binding caused by a corrosion
mechanism that occurred between the valve stem and the valve bushings. The RCA
team also concluded that stem binding appeared to be an intermittent problem and that
Enclosure
6
station members missed opportunities to identify the failure mode during previous
surveillances and failures. As such, the RCA team determined that governor control
valve stem binding could not be ruled out as a possible failure mode during the
December 2008 and the May 2009 TDAFW pump failures.
At Ginna's request, an independent vendor reviewed the issues associated with the
governor control valve stem binding to determine the failure mechanism that was the
cause of the TDAFW pump overspeed trips. The vendor determined that the most likely
cause of the corrosion of the governor control valve stem was fresh-water corrosion
(Langelier corrosion) that was the result of galvanic interactions between the nitrided
case and the base material of the governor control valve stem. The vendor considered
several variables that contributed to the corrosion mechanism such as material
composition of the stem and bushing, stem-to-bushing clearance, steam admission
valve leakage, and TDAFW pump surveillance frequency and duration.
1.2 Special Inspection Scope
The NRC conducted this inspection to gain a better understanding of the circumstances
involving the TDAFW pump overspeed trips during surveillance testing on May 26, 2009,
and on July 2,2009. The inspection team used NRC Inspection Procedure 93812,
"Special Inspection," as a guide to complete their review. Additional inspection and
review activities were outlined in the special inspection team charter, provided as
Attachment B. The special inspection team reviewed procedures, corrective action
documents, work orders, engineering analyses, and the root cause evaluation prepared
by Ginna. In addition. the team conducted equipment walkdowns and interviewed key
plant personnel regarding the discovery and resolution of the condition. A list of site
personnel interviewed and documents reviewed are provided in Attachment A to this
report.
1.3 Preliminary Conditional Risk Assessment
Using IMC 0309, "Reactive Inspection Decision Basis for Reactors," IMe 0609,
"Significance Determination Process," and the Ginna Standardized Plant Analysis Risk
(SPAR) model in conjunction with the Graphical Evaluation Module (GEM). the Region I
Senior Reactor Analyst (SRA) evaluated the increase in conditional core damage
probability for the failure of the TDAFW pump. Based on the nature of the failure and
the subsequent overspeed trip on an instrumented diagnostic run, the condition was
evaluated as being non-recoverable in the event of actual demand.
Based upon best available information, an incremental conditional core damage
probability (ICCDP) in the upper E-6 range (8E-6 per the 82 day exposure period), was
calculated. The exposure period was based on the assumption that the TDAFW pump
would not have started since the last time it passed a surveillance test on March 5 until
May 26 (82 days). The dominant core damage sequence was a station blackout (loss of
offsite power (LOOP) with failure of both emergency diesel generators (EDGs), with no
TOAFW and failure to recover offsite power or an EDG in one hour.
Based upon this conservative conditional core damage probability (CCDP) value, and
having satisfied an IMC 0309 deterministic criterion, the May 26 degraded TDAFW pump
Enclosure
7
condition fell within the Special Inspection to Augmented Inspection Team range for
2. SPECIAL INSPECl"ION AREAS
2.1 Review of Maintenance
a. Inspection Scope
The team evaluated the adequacy and completeness of the maintenance on the TDAFW
system, including preventive maintenance, procedural guidance, post-maintenance
testing, and supervisory oversight. The team independently evaluated selected
procedures, preventive maintenance strategies. condition reports (CRs). system health
reports. and associated work orders. In addition, the team reviewed the RCA,
conducted equipment walkdowns and interviewed key station personnel. For the
weaknesses identified, the inspectors verified that appropriate corrective actions have
been planned or taken.
b. Findings and Observations
1. Failure to Preclude Recurrence of a Significant Condition Adverse to Quality
Introduction: A self-revealing AV of 10 CFR 50, Appendix S, Criterion XVI, "Corrective
Actions," was identified for the failure to preclude recurrence of a SCAQ associated with
the TDAFW pump governor control valve. Specifically, after identifying corrosion of the
governor control valve stem in April 2005, Ginna did not take adequate corrective
actions to preclude the recurrence of corrosion which led to the binding of the governor
control valve and failure of the TDAFW pump on July 2, 2009. In addition, governor
control valve stem binding was a fikely failure of the May 26, 2009, TDAFW pump
overspeed trip. This finding was preliminarily determined to be White.
Description: On April 11, 2005, Glnna conducted a periodic major inspection of the
TDAFW pump governor control valve under work order #20401907. During the
disassembly of the valve, mechanics noted corrosion in the bushing area of the stem
plug. The vendor recommended replacing the valve stem and plug due to the corrosion.
However, Ginna did not recognize stem corrosion as a condition adverse to quality and
did not initiate a CR in accordance with station guidance in IP-CAP-1 r "Condition
Reporting." As a result, Ginna did not conduct any further analysis or evaluation to
determine the cause of the corrosion. On July 2, 2009, during surveillance testing, the
TDAFW pump tripped due to overspeed. Ginna formed an Issue Response Team and
developed a comprehensive troubleshooting plan to determine the cause(s) of the
failure. The troubleshooting plan consisted of all possible failure modes, possible
causes of each failure mode, and actions to validate or refute each "failure mode. Upon
disassembly of the governor control valve, the stem was found seized within its bushings
and had to be forcibly removed. Visual inspection showed pitting on the surface of the
stem Where it contacted the upper valve bushing. Following completion of the
troubleshooting activities, Ginnaconcluded that TDAFW pump failure was due to binding
of the governor control valve and that the binding was caused by the build up of
corrosion on the valve stem.
Enclosure
8
Governor control valve stem corrosion is an SCAQ because corrosion of the stem can
lead to govemor control valve stem binding and failure of the TDAFW pump as
discussed in NRC IN 94-66: "Overspeed of Turbine-Driven Pumps caused by Governor
Valve Stem Binding," and other related industry operating experience documents.
Immediate corrective actions included entering this condition in the CAP, conducting an
RCA, replacing the governor control valve stem, and conducting weekly monitoring of
the governor control valve during surveillance testing to identify any potential for stem
binding. In addition, corrective actions included a follow-up inspection of the governor
control valve during the fall 2009 refueling outage. Because additional corrosion was
found during this inspection on September 18, 2009, the control valve was sent to the
vendor to be refurbished. Ginna will continue to monitor the governor control valve
under an enhanced TDAFW surveillance program to ensure TDAFW pump operability.
Ginna also was evaluating longer term corrective actions to address the corrosion
including more frequent governor control valve stem replacement, change of the stem
material, and other modifications to the system design. Planned corrective actions
include replacing the stem admission valves in May 2011.
The inspectors noted that governor control valve stem binding also could have been a
contributor to the December 2, 2008, TDAFW pump failure to develop the minimum
acceptable discharge flow and pressure, and the likely cause of the May 26, 2009,
TDAFW pump overspeed trip. Following these events, Ginna missed opportunities to
identify potential stem binding problems after identifying possible indicators of this failure
mode such as leaking steam admission valves and rust/corrosion on the visible portion
of the governor valve stem. In addition, during troubleshooting efforts in May 2009,
. Ginna missed an opportunity to exercise the governor control valve stem with the linkage
disconnected and lube oil pressure not applied. Operating experience suggested that
cycling the valve by hand without hydraulics applied is a prudent action to validate or
refute governor control valve stem binding.
Analysis: The performance deficiency is that Ginna did not take adequate measures to
correct a condition that had the potential to impact the operability of the TDAFW pump.
Specifically, after identifying corrosion on the governor control valve stem in 2005, Ginna
did not take adequate corrective actions to preclude the recurrence of corrosion which
led to the binding of the governor control valve and failure of the TDAFW pump. The
finding is more than minor because it is associated with the equipment performance
attribute of the Mitigating System cornerstone and affects the cornerstone objective to
ensure the availability, reliability, and capability of systems that respond to initiating
events to prevent undesirable consequences. Specifically, stem corrosion caused
binding of the governor control valve and led to the failure of the TDAFW pump.
In accordance with IMe 0609, Attachment 4, "Phase 1 - Initial Screening and
Characterization of Findings," a Phase 2 risk analysis was required because the finding
represents an actual loss of safety function of a single train for greater than the TS
allowed outage time of 7 days. The Phase 2 risk evaluation was performed in
accordance with IMC 0609, Appendix A, Attachment 1, "User Guidance for Phase 2 and
Phase 3 Reactor Inspection Findings for At*,power Situations." Because the precise time
is unknown for the inception of TDAFW pump inoperability, an exposure time of one-half
of the time period (t/2) between discovery (May 26, 2009) to the last successfully
completed quarterly surveHlance test (March 5, 2009) was used. This t/2 exposure time
Enclosure
9
equals 51 days. Using Ginna's Phase 2 SDP notebook, pre-solved worksheets, and an
initiating event likelihood of 1 year (>30-days exposure time), the inspector identified that
this finding is of potentially substantial safety significance (Yellow). The dominant
sequence identified in the Phase 2 notebook involves a loss of offsite power (LOOP),
failure of both EDGs, and the subsequent loss of the TDAFW pump, with the failure of
operators to restore offsite power within 1 hour: LOOP (2) + EAC (3) + TDAFW (0) +
REC1 (0) = 5 (Yellow). In recognition that the Phase 2 notebook typically yields a
conservative result, a NRC Region I Senior Reactor Analyst (SRA) performed a Phase 3
risk assessment of this finding.
The SRA used Ginna's Standardized Plant Analysis Risk (SPAR) model, Revision 3.45,
dated June 2008. and graphical evaluation module, in conjunction with the System
Analysis Programs for Hands-On Integrated Reliability Evaluations (SAPHIRE). Version
7, to estimate the internal risk contribution of the Phase 3 risk assessment. The
following assumptions were used for this assessment:
1. Based on the guidance provided in the Risk Assessment of Operational Events
Handbook, Revision 1.01, the calculated exposure was determined to be 1198
hrs. The summation of this exposeure time was determined as follows:
- One half the exposure time, (tI2), was applied for the period between
March 5 - May 26, 2009
- The full exposure time, (t). was applied for the unavailability and
troubleshooting from May 26 - May 29, 2009.
- One half the exposure time. (tI2), was applied for the period between
June 25 -July 2, 2009
- The full exposure time, (t), was applied for the unavailability and
troubleshooting from July 2 - July 5, 2009
2. To closely approximate the type of flilure exhibited by the TDAFW pump, the fail
to start basic event <AFW-TDP-FS-TDP> was changed from its baseline failure
probability to 1.0, representing a 100 percent failure to start condition.
3. Based on the nature of the failure. and no recovery procedures in place, there
was no recovery credit assigned to the May failure.
4. The loss of service water initiating event frequency (IE-LOSWS) was increased
from its nominal value of 4.0E-4 to 2.0E*3 to more closely model the risk of a
LOSWS at the Ginna station. This is consistent with Ginna's understanding of
the risk of this event. All remaining events were left at their nominal failure
probabilities.
5, The model was modified by Idaho National Laboratory (INL) to include
convolution correction factors. Convolving the failure distribution eliminates the
simplifying assumption that all failure to run events happen at time=O. Inclusion
of this correction can reduce station blackout (SBO) core damage frequency
(CDF) significantly for plants like Ginna that have low EDG redundancy.
6. Since all observed failures happened upon the initiation of the TDAFW system,
the period between May 29 and June 25 2009, was not included in the exposure
period because the system successfully completed increased, weekly,
surveillance testing. The team could not conclude that the stem binding
condition would occur after the pump had been successfully started and run.
7. Cutset probability calculation truncation was set at 1E-13.
Enclosure
10
Based upon the above assumptions, the Ginna SPAR model calculated an increase in
internal event contribution to conditional core damage probability (CCDP) of 2.9E-6. This
low-ta-mid E-6 delta CCDP value represents a low to moderate safety significance
(White). The dominant internal event sequence involved a LOOP with subsequent
failure of the EDGs (station blackout event), the failure of the TDAFW train and the
failure to recover AC power. The Phase 3 SPAR model results correlate well to the
Phase 2 SDP Notebook dominant core damage sequences.
External Events Risk Contribution
The Ginna Probablistic Safety Assessment (PSA) includes a Level 1 analysis of fires
and flooding external events. The PSA summarizes the fire contribution as representing
approximately 48% of the total (internal and external) core damage frequency, or nearly
half of the annualized risk. The NRC does not have an external event risk model for
Ginna. Consequently, the SRA, after review of the licensee's Individual Plant
Examination of External Events (IPEEE), utilized the licensee's external events
assessment to quantify the fire and flooding events risk contribution for this condition.
Seismic event likelihood was considered to be very low and determined not to be a
significant contributor to the risk of this condition.
The results of the PSA for this condition calculated a CCDP contribution from fire events
at 5E-6 and from flooding at 7E-7. The most significant fire initiated core damage
sequence involved a spectrum of control room fires, with a failure of automatic and
manual suppression, a failure of the TDAFW pump and a failure of the 'C' standby
auxiliary feed water pump for decay heat removal via the steam generators. The most
Significant flooding core damage sequence quantified in the PSA for this condition,
Involved flooding in the relay room, failure of the TDAFW pump and a failure to align
standby auxiliary feed water pump for decay heat removal via the steam generators.
Large Early Release Frequency (LERF) Evaluation
The SRA used IMC 0609, Appendix H, "Containment Integrity Significance
Determination Process," to determine if this finding was a significant contributor to a
large early release. The Ginna containment is classified as a pressurized water reactor
large-dry containment deSign. Based upon the dominant sequences involving LOOP
and station blackout (S80) initiating events, per Appendix H, Table 5.2, "Phase 2
Assessment Factors - Type A Findings at Full Power," the failure of the TDAFW pump
does not represent a significant challenge to containment integrity early in the postUlated
core damage sequences. Consequently, this finding does not screen as a significant
large early release contributor because the close-in populations can be effectively
evacuated far in advance of any postulated release due to core damage, Accordingly,
the risk significance of this finding is associated with the delta CDF value, per IMC 0609,
Appendix H, Figure 5.1, and not delta LERF.
Risk Assessment Summary
The calculated total risk significance of this finding is based upon the summation of
internal and external risk contributions (delta CCDP internal + delta CCDP external (fires
and floods).= delta CCDP total). 2.9E-6 + 5E-6 + 7E-7 = 8.6E-6 delta ceDP.
Enclosure
11
Annualized, this value of 8.6E-6 delta CDF represents a low to moderate safety
significance or White finding.
Licensee's Risk Assessment Summary
Constellation's risk assessment for the given condition, assuming no operator recovery,
resulted in a total delta CDF value of 1.028E-5. This increase in CDF value comprised
of: 4.6E-6 due to intemal events; 0.7E-6 due to intemal flooding events; and 5E-6
fire/external events. Similar to the NRC internal risk contribution, the largest percentage
of intemal risk was derived from station blackout events. The licensee conducted
refinements in their modeling to credit additional recovery options in the event that the
TDAFW pump failed. By crediting these recovery actions, the delta CDF was reduced
from 1.028E-5 to 9.2 E-6, or by approximately 10%. The specific recovery actions are
as follows: .
1. The potential for operators to use standby auxiliary feedwater (SAFW) pump C,
in lieu of the TDAFW pump during fire and flood scenarios which require use of
alternate shutdown procedures was modeled. Although use of an SAFW pump
is not specifically called out in these procedures, the procedures do explicitly
recognize that if the TDAFW pump is not functioning, a loss of secondary heat
sink will result, and direct operators to refer to other emergency procedures for
alternate methods of establishing a heat sink (see ER-FIRE.1 step 4.3.4.2, ER
FIRE.2 step 4.3.12, etc.}. In response to this hypothetical scenario, an
operations Shift Manager (SM) indicated that upon a failure of the TDAFW to
provide flow, a SAFW pump could be manually started by locally opening the
service water suction valve to the pump and locally closing the breaker to the
pump motor. The steps necessary to align an SAFW pump to the steam
generators (SG) are contained in emergency operating procedures (EOP)
Attachment 5.1 'Attachment SAFW'. Since only Busses 14 and 18 are energized
during ER-FIRE scenarios, SAFW pump C would be used.
2. Following the May 26th overspeed trip of the TDAFW pump, new section 2.2
was added to procedure P-15.6, 'Operation of the TDAFW Pump Trip Throttle
Valve.' Revision 00200. This step provides instructions for a 'slow start' of the
TDAFW pump manually, following an overspeed trip of the pump, by using the
trip throttle valve. This allows for starting of the pump even if the governor control
valve is stuck in the full open position following the overspeed trip. This
procedure step was successfully used following both the May 26th failure and
the July 2nd failure. Per Operations management, the shift managers, who
would be directing use of this procedure, were briefed on the updated procedure
to ensure they were aware that new step was available for use and understood
how it was to be performed. A controlled copy of the procedure is located at the
TDAFW pump for use by the operator. Since this procedure was in effect prior
to the July failure exposure period (Le., prior to the last successful TDAFW pump
test on June 25, 2009), it is considered as a recovery for failures of the TDAFW
pump.
Recovery actions identified in #1 above were not quantified by the team however the
evaluation approach appears to be appropriate. Prior to the licensee crediting recovery
actions identified in #2 above, the team was able to verify that the procedures were in
Enclosure
12
place and observed troubleshooting video in which the licensee started the TDAFW
pump with this method. As a result the SRA concluded that the licensee's modeling
demonstrating a risk reduction was appropriate.
Based upon the close comparison between NRC and Constellation risk estimates, no
sensitivity analyses were warranted. The use of V2 to approximate the exposure time
was determined to be appropriate for standby or periodically operated components that
fail due to a degradation mechanism that gradually affects the component during the
standby time period. Inclusion of the unavailability time hours, due to troubleshooting
and repairs, is also appropriate and consistent with the guidance promulgated in Risk
Assessment of Operational Events Handbook, Revision 1.01, dated January 2008.
This finding has a cross-cutting aspect in the area of Problem Identification and
Resolution, Corrective Action Program, because Ginna did not implement a corrective
action program with a low threshold for identifying issues completely, accurately, and in
a timely manner commensurate with their safety significance. Specifically, Ginna did not
identify issues associated with corrosion of the governor control valve within the
corrective actions program. The inspectors concluded that the performance deficiency is
reflective of current performance because Ginna had reasonable opportunities to identify
the issue during troubleshooting in December 2008 and May 2009. P.1(a) per IMC
0305]
Enforcement: 10 CFR 50, Appendix B, Criterion XVI, "Corrective Actions," states, in
part, "Measures shall be established to assure that conditions adverse to quality, such
as failures, malfunctions, defiCiencies, deviations, defective material and eqUipment, and
nonconformances are promptly identified and corrected. In the case of significant
conditions adverse to quality, the measures shall assure that the cause of the condition
is determined and corrective action taken to preclude repetition." Contrary to the above,
after identifying corrosion on the TDAFW pump governor control valve stem on April 11,
2005, which is a significant condition adverse to quality, Ginna did not take adequate
measures to determine the cause and prevent recurrence. The cause of the condition
was left uncorrected and resulted in additional stem corrosion that led to binding of the
governor control valve and the failure of the TDAFW pump on July 2, 2009. In addition,
stem corrosion was the likely cause of the TDAFW pump failure on May 26, 2009. This
issue was entered into Ginna's CAP as CR-:2009-003680 and CR-2009-004577.
Pending final determination of significance, this finding is identified as an AV. (AV
0500024412009008*01: Inadequate Corrective Actions Associated with the TDAFW
Pump Governor Control Valve)
2. Untimely Corrective Actions Associated with Steam Admission Valves
Introduction: A self-revealing NeVof 10 CFR 50, Appendix B, Criterion XVI, "Corrective
Actions," was identified for the failure to establish adequate measures to correct a
longstanding issue associated with steam a(]mission valves leakage. As a result, the
leakage most likely contributed to the build-up of corrosion on the TDAFW pump
governor control valve stem on May 26, 2009 and contributed to the failure of the
TDAFW pump on July 2, 2009.
Description: During a review of failures associated with the TDAFW pump, the
inspectors noted that the steam admission valves had a history of leaking. Based on a
review of the RCA report, industry operating experience, and associated CRs, the
Enclosure
13
inspectors determined that the steam admission valve leakage was a contributing factor
in the development of corrosion of the governor control valve and contributed to the
failure of the TDAFW pump on July 2, 2009, and most likely the failure of the TDAFW
pump overspeed trip failure on May 26, 2009. Based on a review of eRs and interviews
of personnel, the inspectors determined that the steam admission valves had been
leaking since at least 2005.
In July 2005, CR-2005~3660 documented that steam admission valve 3505A was
leaking past its seat. However, Ginna did not take or plan any corrective actions
associated with the issue. By October 2006, steam admission valve 3505A leakage had
increased such that the TDAFW pump turbine was rotating approximately 100 rpm.
Ginna cycled the valve to reduce seat leakage, generated a work order to repair the
valve in May 2008, and conducted an evaluation of past operability. Ginna concluded
that there was no potential for increased consequences with time if the condition
continued. During the May 2008 refueling outage, Ginna conducted a repair of steam
admission valve 3505A. However, this corrective action was not effective; by September
2008, the valve was leaking again although at a reduced rate. The inspectors
determined that Ginna did not recognize or consider the potential impact of the steam
admission valve leakage on the governor control valve stem.
Immediate corrective actions included entering this condition in the CAP, replacing the
govemor control valve stem, and conducting weekly monitOring of the governor control
valve during surveillance te~ting to identify any potential for stem binding. Additionally,
the steam admission valves were inspected and re-worked and the govemor control
valve was Inspected during the fall 2009 outage. Ginna will continue to monitor the
governor control valve under an enhanced TDAFW surveillance program to ensure
TDAFW pump operability. Planned corrective actions include replacing the stem
admission valves in May 2011.
Analysis: The performance deficiency is that Ginna did not adequately address
deficiencies associated with steam admission valve leakage in that the leakage likely
resulted in accelerated corrosion of the governor control valve and contributed to the
failure of the TDAFW pump. The finding is more than minor because it is associated
with the equipment performance attribute of the Mitigating Systems cornerstone and
affects the cornerstone objective to ensure the availability and reliability of systems that
respond to initiating events to prevent undesirable consequences. Specifically, leakage
through the steam admission valves can res.ult in continuous wetting of the governor
control valve stem and lead to or accelerate corrosion of the governor control valve.
This could result in a stem binding of the govemor control valve and failure of the
TDAFW pump. The inspectors evaluated the significance of this finding using IMC 0609,
Attachment 4, "Phase 1 -Initial Screening and Characterization of Findings." The
finding is of very low safety significance because it is not a design or qualification
deficiency, did not represent a loss of a safety function of a system or a single train
greater than its TS allowed outage time, and did not screen as potentially risk Significant
due to external events. The inspectors determined that the steam admission valve
leakage was not a direct cause of the failure of the TDAFW pump overspeed events.
The steam admission valve leakage was a contributing factor in the development of
governor control valve stem corrosion and contributed to the failure of the TDAFW
pump.
Enclosure
14
This finding has a cross-cutting aspect in the area of Problem Identification and
Resolution, Corrective Action Program, because Ginna did not thoroughly evaluate
problems such that the resolutions address causes and extent of conditions, as
r:eecessary, in a timely manner, commensurate with their significance. Specifically,
Ginna did not thoroughly evaluate the potential effect of the steam admission valve
leakage on the governor control valve performance. The inspectors determined that this
issue is reflective of current licensee performance because each time the issue was
identified and a CR was generated represented an opportunity for Ginna to adequately
evaluate the issue and assign appropriate corrective actions. [P.1 (c) per IMC 0305]
Enforcement: 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Actions," states, in
part. "Measures shall be established to assure that conditions adverse to quality, such
as failures, malfunctions, deficiencies, deviations, defective material and equipment, and
nonconformances are promptly identified and corrected." Contrary to this requirement,
from at least July 2005 to July 2, 2009, Ginna did not established adequate measures to
correct longstanding steam admission valves leakage and/or minimize the impact of the
leakage on the TDAFW pump governor control valve performance. As a result, leakage I*
I
through the steam admission valves contributed to the corrosion of the governor control
valve and contributed to the failure of the TDAFW pump on July 2, 2009 and most likely
the failure of the TDAFW pump on May 26, 2009. Because this violation is of very low
safety significance (Green) and Ginna entered this issue into their CAP for resolution as
CR-2009-003680 and CR-2009-004577, this violation is being treated as an NCV
consistent with the NRC Enforcement Policy. (NCV 0500024412009008-02: Inadequate
Corrective Actions Associated with Stearn Admission Valve Leakage)
2.2 Review of Operating Experience
a. Inspection ScoQe
The team reviewed operating experience involving TDAFW pump failures and actions
taken by the Ginna staff to identify and address these types of failures. In addition, the
team examined the specific issues associated with governor control valve stem binding
to assess any new generic issues of industry interest for prompt communication and
dissemination. As part of this evaluation, the inspectors reviewed pertinent industry
operating experience, Ginna's response to NRC Information Notices, and interviewed
key plant personnel.
b. Findings and Observations
No Findings of Significance Identified.
The inspectors determined that there were no new generic issues identified as a result of
this event. However, the inspectors noted that Ginna missed opportunities to utilize
industry operating experience to identify precursors associated with this event and to
conduct effective troubleshooting. Specifically, following the discovery of corrosion on
the exposed portion of the TDAFW pump govemor control valve stem in May 2005,
Ginna did not enter this issue in to their CAP. As a result Ginna missed an opportunity
to revisit operating experience from NRC IN 94-66 conceming governor control valve
stem binding. In several examples in NRC IN 94-66, sites that experienced governor
control valves stem binding identified stem corrosion following valve disassembly. Ginna
Enclosure
15
had originally determined that NRC IN 94-66 was not applicable to Ginna because the
governor control valve was of a different configuration and material as those discussed
in the IN and that corrosion of the stem had not been previously identified. Although this
may have been a reasonable assessment during the original review of the NRC IN 94-66
in 1994, the inspectors determined that once corrosion of the stem was identified in'May
2005, Ginna missed an opportunity to revisit this operating experience. In addition, as
discussed in Section 2.1.2 of this report. Ginna did not utilize Industry operating
experience in evaluating a condition associated with steam admission valve leakage.
NRC IN 94-66, other industry operating experience, and the vendor technical manual
stated that steam admission valve leakage is a cause related to governor control valve
stem binding. However, Ginna did not consider the impact of steam admission valve
leakage on the governor control valve performance. Finally, the inspectors noted that
Ginna missed opportunities to incorporate industry operating experience during
troubleshooting efforts during the May 2009 TDAFW pump failure. NRC IN 94-66
discussed several examples where a freedom of motion test was performed to test for
stem binding with the governor control valve linkage disconnected and hydraulics not
applied. However, during the May 2009 event, Ginna missed an opportunity to conduct
this test and consequently, missed an opportunity to potentially identify the failure mode
prior to the July 2, 2009, overspeed trip. Ginna captured these issues in their CAP
under CR-2009-003680 and CR-2009-004577..
2.3 Review of Root Cause and Extent-ot-Condition
a. Inspection Scope
The team evaluated the adequacy of Ginna's RCA and completed interim corrective
actions. In addition the team evaluated the adequacy of Ginna's initial extent of
condition for the TDAFW pump failures. The team reviewed plant drawings, procedures,
and associated system modifications. In addition, the team conducted a walkdown of
the TDAFW system and interviewed key Ginna personnel.
b. Findings and Observations
Introduction: The inspectors identified an NCV of 10 CFR 50, Appendix B, Criterion III,
"Design Control," for the failure to establish measures to ensure that a modification
performed on the governor control valve bushing was a suitable application of materials
for the TDAFW pump.
Description: During the troubleshooting and repair of the TDAFW pump in July 2009,
Ginna incorrectly evaluated that a vendor service bulletin allowed for larger stem-to
bushing clearance than was prescribed in their current technical documents. Since
there was overlap between the station technical information and the vendor service
bulletin, Ginna decided to increase the clearance, by lapping the bushings, to reduce the
likelihood that corrosion products would cause the stem to bind. To increase the
clearance, Ginna removed some of the hardened layer on the bushing. During a review
of the RCA, the inspectors determined that Ginna did not consider the potential impact of
removing some ot the hardened layer on the corrosion rate of the governor control valve.
The inspectors noted that by removing some of the hardened layer, the material property
of the bushing could have been changed and/or additional micro-cracks could have been
created or exposed that could increase the corrosion rate or potentially cause other
Enclosure
16
problems such as stem-to~bushing galling. The inspectors noted that if the corrosion
rate is greater than predicted, then planned corrective actions may not be adequate to
maintain the TDAFW pump operable. Additionally, the team was concerned, that by
increasing the stem to bushing clearances, the stem would be susceptible to larger
steam exposure and that this could further accelerate the corrosion. Following concerns
raised by the inspectors, Ginna initiated CR-2009-005959 to ensure that all possible
consequences of lapping the TDAFW pump control valve bushing were fully evaluated.
In addition, Ginna assigned corrective actions to inspect the control valve bushing during
the fall 2009 refueling outage. During performance of the refueling inspection under WO
C90623685, pitting of the stem and corrosion of the bushings were observed.
Constellation initiated CR~2009-006765 and noted that the corrosion of the bushings
appeared to have been caused by the lapping of the bushing to achieve the desired
clearance between the stem and the bushings. The development of corrosion on the
bushing reduced the margin that was predicted to ensure adequate stem movement.
Immediate corrective actions following the inspection of the governor control valve
during the fall 2009 refueling outage included entering this condition in the CAP and
refurbishing the governor control valve with a new stem and bushing.
Analysis: The performance deficiency is that Ginna failed to establish measures to
ensure that a modification performed on the governor control valve bushing was a
suitable application of materials for the TDAFW pump. The finding is more than minor
because it is associated with the equipment performance attribute of the Mitigating
Systems cornerstone and affects the cornerstone objective to ensure the availability and
reliability of systems that respond to initiating events to prevent undesirable
consequences. Specifically, lapping of the valve bushing resulted in an unanticipated
corrosion mechanism of the governor control valve that impacted the reliability of the
TDAFW pump. The inspectors evaluated the Significance of this finding using IMC 0609,
Attachment 4, "Phase 1 -Initial Screening and Characterization of Findings." The
finding is of very low safety significance because it is not a design or qualification
deficiency, did not represent a loss of a safety fUnction of a system or a single train
greater than its TS allowed outage time, and did not screen as potentially risk significant
due to external events.
The inspectors determined that this finding has a cross-cutting aspect in the area of
Human Performance, Decision Making, because Ginna did not make a safety-significant
or risk-significant decision using a systematic process, especially when faced with
uncertain or unexpected plant conditions, to ensure safety was maintained. Specifically,
Ginna did not use a systematic process such as an engineering evaluation to properly
evaluate the potential impact of removing some of the hardened layer of the bushing.
H.1(a) per IMC 0305]
Enforcement: 10 CFR 50, Appendix B, Criterion III, "Design Control" states, in part,
"Measures shall also be established for the selection and review for suitability of
application of materials, parts, equipment, and processes that are essential to the safety
related functions of the structures, systems and components." Contrary to the
requirements, in July 2009, Ginna failed to establish measures to ensure that a
modification performed on the governor control valve bushing was a suitable application
of materials for the TDAFW pump. Specifically, lapping of the valve bushing resulted in
an unanticipated corrosion mechanism of the governor control valve that had the
Enclosure
17
potential to impact the performance of the TDAFW pump if not corrected. Because this
violation is of very low safety significance (Green) and Ginna entered this issue into their
CAP for resolution as CR-2009-006765, this violation is being treated as an NCV
consistent with the NRC Enforcement Policy. (NeV 05000244/2009008-03: Failure To
Establish Design Control Measures Associated With The Turbine Driven Auxiliary
Feedwater Pump Governor Control Valve)
2.4 Risk Assessment of the As-Found Condition
a. Insoection Scope
Prior to the initiation of the Special Inspection Team, the Region I SRA performed a
CCDP assessment which conservatively bounded the potential risk significance of the
degraded condition, assuming the TDAFW pump would fail to star on demand to
mitigate the consequences of an event. The initial CCDP estimate was performed in
accordance with IMC 0309, "Reactive Inspection Decision Basis for Reactors." The SRA
used the Ginna SPAR model and associated GEM to evaluate the potential risk
significance of this condition. The results of the IMC 0309 assessment are documented
in report section 2.1.b.1 above.
b. Final Risk Estimate
Following team review and independent verification of the degraded turbine driven
auxiliary feed water pump governor control valve, the team concluded that for
approximately 119B hours, the TDAFW pump would not have been capable of
responding to an event. Consistent with IMC 0609 conditional core damage probability
assessment methodology, this degraded condition resulted in a loss of operability or
safety function, and therefore was of low to moderate (B.6E-6) safety significance.
4. OTHER ACTIVITIeS
40A6 Meetings, Including Exit
On August 27 and on October 7,2009 the team presented the inspection results to Mr.
John Carlin and other members of the R. E. Ginna Nuclear Power Plant staff. The
inspectors verified that none of the materiall~xamined during the inspection is
considered proprietary in nature.
Enclosure
A-1
ATTACHMENT A
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee personnel
J. Carlin, Site Vice President
E. Larson, Plant General Manager
D. Crowley, Senior Engineer
E. Durkish, Associate Engineer
R. Everett, Supervisor, Primary Systems Engineering
T. Harding, Director, Licensing
R. Ruby, Principal Engineer, Licensing
P. Swift, Engineering Manager
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
05000244/2009008-01 AV Failure to Preclude Recurrence of a
Significant Condition Adverse to Quality
Associated with the Turbine Driven Auxiliary
Feedwater Pump Governor Control Valve.
(Section 2.1.1)
Opened/Closed
05000244/2009008-02 NCV Inadequate Corrective Actions Associated
with Steam Admission Valve Leakage.
(Section 2.1.2)05000244/2009008-03 NCV Failure to Establish Design Control
Measures Associated with the Turbine
Driven Auxiliary Feedwater Pump Governor
Control Valve. (Section 2.3)
I.IST OF DOCUMENTS REVIEWED
In addition to the documents identified in the body of this report, the inspectors reviewed the
following documents and records:
Corrective Actions
CA-2009-002311
Condition Reports
CR-2003-2006 CR-2005-3660 CR-2006-006029 CR-2006-006341
CR-2005-1509 CR-2005-5513 CR-2006-006204 CR-2006-006962
Attachment A
A-2
CR-2007-000876 CR-2008-009911 CR-2009-004590 CR-2009-005959*
CR-2007-001302 CR-2009-003680 CR-2009-004591 CR-2009-006765*
CR-2008-003687 CR-2009-004222* CR-2009-005964
CR-2008-007541 CR-2009-004577
- NRC Identified During Inspection
Corrective Action Tracking System
CATS R007325
Procedures
AP-FVV.1, Abnormal MFW Pump Flow or NPSH, Revision 01702
PT-16QT-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05900
CNG-MN-1.01-1002, Troubleshooting, Revision 0001
CNG-CA-1.01, Corrective Action Program, Revision 0001
CNG-CA-1.01-1004, Root Cause Analysis, Revision 0001
CNG-CA-1.01-1005, Apparent Cause Evaluation, Revision 0100
CNG-CA-1.01-1006, Common Cause Analysis, Revision 0001
CNG-CA-1.01-1007, Trending, Revision 0000
CNG-CA-1.01-1010, Use of OE, Revision 0000
IP-CAP-1, Ginna Condition Reporting, Revision 02800
M-11.5C, Auxiliary Feedwater Pump Motor Mechanical Inspection and Maintenance,
Revision 03000
0-1.1, Plant Heatup From Cold Shutdown to Hot Shutdown, Revision 16301
PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05600. Completed
3/14/08,514108,5/5/08,6/11/08,9/4/08, 1213/08, 1214/08, 12111/08, 12/18/08,2/12/09,
5/26/09, 5/28/09
Surveillance Tests
PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05600, 03/14/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump -. Quarterly, Revision 05700, 05/04/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05700,05/05/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05700. 06/11/2008
STP-0-16-COMP-T, Auxiliary Feedwater Turbine Pump - Comprehensive Test,
Revision 00000, 06/11/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly. Revision 05701, 09/03/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05701, 12/02/2008
PT-16Q-T, Auxiliary FeedwaterTurbine Pump - Quarterly. Revision 05701,12103/2008
PT-16Q-T. Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05701, 12/04/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05702, 12/11/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump _. Quarterly, Revision 05702, 12/18/2008
PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05801, 01116/2009
PT-16Q-T, AUXiliary Feedwater Turbine Pump - Quarterly, Revision 05801, 02112/2009
PT-16Q-T, AUXiliary Feedwater Turbine Pump _. Quarterly. Revision 05900,05/26/2009
PT-16Q-T, Auxiliary Feedwater Turbine Pump - Quarterly, Revision 05900,05/28/2009
STP-0-16-COMP-T, Auxiliary Feedwater Turbine Pump - Comprehensive Test,
Revision 00400, 05/28/2009
Attachment A
A-3
Work Orders
W020604989
W020401907
Drawings
DWG No. 33013-1231, Main Steam System, Revision 37
DWG No. 33013-2285, Motor Driven and Turbine Driven Auxiliary Feedwater Pumps
DWG No. LB-112541, Turbine Control and Quick Start System
DWG No. T040-001A, TDAFW Control Oil System, Revision 0
DWG No. T040-002A, TDAFW Turbine Trip Valve, Revision 0
DWG No. T040-003A, Auxiliary and Standby Aux. Feedwater Systems, Revision 2
DWG No. T040-003B, SAFW System 1-Line Diagram, Revision 0
DWG No. T040-003C, AFW System 1-Une Diagram, Revision 1
DWG No. T040-004A, Auxiliary Feedwater Pump Auto Start Signals, Revision 0
DWG No. T830-004A, Blowdown Isolation Solenoid Control Circuit, Revision 0
Other Documents
Vendor Report, Preliminary Results ofStem Sticking Failure Mechanisms, July 20, 2009
Vendor Report 09-1929 Part 1, Equipment Root Cause Analysis of AFWP Control Valve
Stem Sticking Problem at Ginna Nuclear Power Station
Vendor Technical Question Response, August 26, 2009
Category I Root Causal Analysis, May and July 2009 Turbine Driven Auxiliary Feedwater
Pump (TDAFWP) Failures; Overspeed Trip During Testing
vrD-E9016-4001, Excerpts from EPRI Manual 1007461, Terry Turbine Maintenance
Guide, AFW Application, Revision 000
vrD-G0153-4001, Operating Instructions Easy Flow Body Combined Trip Throttle Valve,
Revision 2
VrD-W0315-4001, Instructions For 465 H.P. Non-Condensing Steam Turbine Serial
Number 26635, Revision 000
VrD-W0315-4002, Service Department Standards Book No. 10, Field Service Manual,
Revision 000
Response to NRC Generic Letter 90-3, Relaxation of Staff Position In Generic Letter 83
28, Item 2.2 Part 2 "Vendor Interface for Safety Related Components, dated
September 18, 1990
CATS 10 R04451, Response to NRC IN 94-66, OVERSPEED OF TURBINE-DRIVEN
PUMPS CAUSED BY GOVERNOR VALVE STEM BINDING
CMM-37-19-9519E, Worthington Turbine Driven Auxiliary Feedwater Pump Hydraulic
Governor Control Valve Maintenance for 9519E, Revision 00200
Constellation Energy Nuclear Generating Group Reply to a Notice of Violation; EA-09
045
Engineering Change Package No. ECP-2009-0146, TDAFW Lube Oil Orifice
Management Review Committee Agenda. August 5, 2009
Purchase Requisition No. 58686, Dresser-Rand Valve Stem
Technical Staff Request 97-199, Leakoff from TDAFW Govenor Valve 9519E
Management Review Committee Agenda, August 5, 2009
Purchase Requisition No. 58686, Dresser-Rand Valve Stem
Attachment A
A-4
Technical Staff Request 97-199, Leakofffrom TDAFW Govenor Valve 9519E
Turbine Driven Auxiliary Feedwater Pump Preventive Maintenance Strategies
Auxiliary Feed Water System, 1st Quarter 200S
Auxiliary Feed Water System. 2nd Quarter 200S
Auxiliary Feed Water System. 3rd Quarter 200S
Auxiliary Feed Water System, 4th Quarter 200S
Auxiliary Feed Water System, 1st Quarter 2009
Operating Experience
OE-200S-000397
OE-200S-000607
OE-200S-000847
oE-20OS-00S60
OE-2008-001296
OE-2009-00212
OE*2009*001178
Oil Analysis
Pump IB Oil Analysis. 2008-03
Pump OB Oil Analysis, 2008-03
Reservoir Oil Analysis 2008-05
Reservoir Oil Analysis 2008-08
Reservoir Oil Analysis 2009-05
Attachment A
A-4
LIST OF ACRONYMS
AV Apparent Violation
CAP Corrective Action Program
CCDP Conditional Core Damage Probability
CDF Core Damage Frequency
CR Condition Report
DRP Division of Reactor Projects
EDG Emergency Diesel Generator
GEM Graphical Evaluation Module
ICCDP Incremental Conditional Core Damage Probability
IMC Inspection Manual Chapter
IN Information Notice
- IPEEE Individual Plant Examination of External Events
LERF Large Early Release Frequency
LOOP Loss of Offsite Power
LOSWS Loss of Service Water System
NCV Non Cited Violation
NRC Nuclear Regulatory Commission
NRR Office of Nuclear Reactor Regulation
PARS Publicly Available Records
PSA Probabilistic Safety Assessment
RCA Root Cause Analysis
SAFW Standby Auxiliary Feedwater
SBO Station Blackout
SOP Significance Determination Process
SPAR Standardized Plant Analysis Risk
SRA Senior Reactor Analyst
SCAQ Significant Condition Adverse to Quality
NCV Non-citied Violation
TBD To Be Determined
TDAFW Turbine Driven Auxiliary Feedwater
TS Technical Specification
Attachment A
8-1
Special Inspection Charter
R.E. Ginna Nuclear Power Plant
Failure of the Turbine-Driven Auxiliary Feedwater (TDAFW) Pump
on May 26, 2009 and July 2, 2009
Background:
On May 26, 2009, during routine quarterly surveillance testing of the turbine driven
auxiliary feedwater (TDAFW) system, the TDAFW pump tripped on overspeed. The test
was repeated several hours later and the TDAFW pump tripped again on overspeed.
Fluctuations in oil pressure were observed during the testing.
After extensive troubleshooting, Ginna personnE~1 identified a number of issues but no
definitive cause for the overspeed trips. The lube/control oil system was drained and
cleaned due to the presence of fine particulates; a pressure pulsation dampener
accumulator bladder was replaced due to a below normal pressure condition; the oil
pressure regulating bypass valve was replaced due to potential cycling; a change to
linkage setup was implemented; governor relay valve, and trip and throttle valve parts
were replaced due to out of specification clearances; and a miSSing oil line orifice was
installed. Following these corrective actions, the system was successfully tested and
declared operable.
Constellation is completing a root cause evaluation and the resident inspectors continue
to conduct baseline inspections with assistance from DRS specialists. Testing of the
TDAFW system will be at an increased frequency until Constellation has confidence that
the system is performing as required.
On July 2, 2009, during the performance of the increased frequency surveillance, the
TDAFW pump again tripped on overspeed. Glnna personnel discovered pitting and
corrosion on the stem of the governor control valve during troubleshooting activities to
restore the pump to operable status. This Special Inspection Team had raised concerns
regarding corrosion on the stem of the Governor Control Valve during onsite inspection
activities the week of June 15, 2009.
Basis for the Formation of the SIT:
The failure of the TDAFW pump involved repetitive failures of this safety~related
eqUipment. There have been three failures of the TDAFW pump since December 2008.
On December 2, 2008, a failure was attributed to inadequate implementation of the
preventive maintenance program. SpeCifically, the governor linkages were not
lubricated in March 2008 which resulted in the pump's inability to achieve the required
flow and pressure in December 2008. On July 2, 2009, the failure was preliminarily
determined to be caused by pitting on the stem of the Governor Control valve and overly
tolerances causing binding of the stem. Although the specific failure modes (overspeed
trip, inability to achieve the required flow or pressure) were different, the underlying
cause of inadequate or inappropriate preventive maintenance may be the same.
Attachment 8
B-2
Based upon best available information, the Region I Senior Reactor Analyst (SRA)
conducted a preliminary risk estimate of the May 26 TDAFW pump failure. An
incremental conditional core damage probability (ICCDP) in the upper E-6 range (8E-6
per the 82 day exposure period), was calculated using the Ginna SPAR model.
assuming that the TDAFW pump would not have started since the last time it passed a
surveillance test on March 5 until May 26 (82 days). The dominant core damage
sequence was a station blackout (LOOP with failure of both EDGs,) with no TDAFW and
failure to recover offsite power or an EDG in one hour.
Based upon the preliminary conditional core damage probability estimate of upper E-6
range, in accordance with IMC 0309, this event falls within the region for a Special
Inspection Team.
Oblectives of the Special Inspection:
The objectives of the special inspection are to review and assess: (1) Constellation's
planning and execution of the risk significant work activities on the TDAFW system; (2)
equipment issues related to the TDAFW testing; and (3) Constellation's response to this
significant equipment failure.
To accomplish these objectives, the following will be performed:
1. Evaluate the adequacy and completeness of the maintenance on the TDAFW
system, including preventive maintenance, procedural guidance, post
maintenance testing, and supervisory oversight.
2. Evaluate Constellation's application of pertinent industry operating experience
and evaluation of potential precursors, including the effectiveness of any actions
taken in response to the operating experience or precursors.
3. Evaluate the adequacy of Constellation's response to the TDAFW system
failures, including Constellation's cause analysis and completed interim corrective
actions.
4. Evaluate the adequacy of Constellation's initial extent of condition for the TDAFW
failures, as appropriate.
5. Evaluate the failure modes for potential generic implications including the need
for generic communications
Additionally, the team leader will review lessons learned from the Special Inspection
and, if appropriate, prepare a feedback form on recommendations for revising the
reactor oversight process (ROP) baseline inspection procedures in order to proactively
identify the issues and causes involved with the event.
Guidance:
Inspection Procedure 93812, "Special Inspection", provides additional guidance to be
used by the Special Inspection Team. Team duties will be as described in Inspection
Procedure 93812. The inspection should emphasize fact-finding in its review of the
circumstances surrounding the event. It is not the responsibility of the team to examine
Attachment B
8-3
the regulatory process. Safety concerns identified that are not directly related to the
event should be reported to the Region I office for appropriate action.
The Team will conduct an entrance meeting and begin the inspection on June 8,2009.
While on site, the Team Leader wi" provide daily briefings to Region I management, who
will coordinate with the Office of Nuclear Reactor Regulation, to ensure that all other
parties are kept informed. A report documenting the results of the inspection should be
issued within 45 days of the completion of the inspection.
This Charter may be modified should the team develop significant new information that
warrants review.
Attachment 8