ML020810177
| ML020810177 | |
| Person / Time | |
|---|---|
| Site: | Callaway  |
| Issue date: | 03/21/2002 |
| From: | Howell A Division of Reactor Safety IV |
| To: | Randolph G Union Electric Co |
| References | |
| EA-02-046 IR-02-007 | |
| Download: ML020810177 (57) | |
See also: IR 05000483/2002007
Text
March 21, 2002
Garry L. Randolph, Senior Vice
President and Chief Nuclear Officer
Union Electric Company
P.O. Box 620
Fulton, Missouri 65251
SUBJECT: NRC AUGMENTED INSPECTION TEAM (AIT) REPORT 50-483/02-07 AND
PRELIMINARY WHITE FINDING - CALLAWAY PLANT
Dear Mr. Randolph:
On February 27, 2002, the NRC completed an Augmented Inspection at your Callaway Plant.
The enclosed report documents the inspection findings which were discussed on
February 27, 2002, with you and other members of your staff.
This inspection examined activities conducted under your license as they relate to safety and
compliance with the Commissions rules and regulations and with the conditions of your license.
Within these areas, the inspection consisted of selected examination of procedures and
representative records, observations of activities, and interviews with personnel.
The report discusses an issue that appears to have low to moderate safety significance. The
issue involved the failure, on multiple occasions, to identify and correct a risk significant
condition adverse to quality regarding the degraded condition of the condensate storage tank
diaphragm seal. Foam from the degraded seal was eventually entrained in the auxiliary
feedwater system suction piping and caused an on-demand failure of an auxiliary feedwater
pump, while plant operators reduced reactor power on December 3, 2001. The finding was
assessed using the Significance Determination Process (SDP) and was preliminarily
determined to be White. The finding has a low to moderate safety significance under the SDP
because it involved an increase in the core damage frequency of between 1E-6/year and
1E-5/year.
The failure to promptly identify and correct the degraded diaphragm seal is also an apparent
violation of Criterion XVI of Appendix B to 10 CFR Part 50 and is being considered for
escalated enforcement action in accordance with the "General Statement of Policy and
Procedure for NRC Enforcement Actions" (Enforcement Policy), NUREG-1600. The current
Enforcement Policy is included on the NRCs website at http://www.nrc.gov/what-we-
do/regulatory/enforcement.html.
Before the NRC makes a final decision on this matter, we are providing you an opportunity to
request a Regulatory Conference where you would be able to provide your perspectives on the
Union Electric Company 2
significance of the finding, the basis for your position, and whether you agree with the apparent
violation. If you choose to request a Regulatory Conference, we encourage you to submit your
evaluation and any differences with the NRC evaluation at least one week prior to the
conference in an effort to make the conference more efficient and effective. If a conference is
held, it will be open for public observation. The NRC will also issue a press release to
announce the conference.
Please contact Dr. Dale A. Powers at (817) 860-8195 within 10 days of the date of this letter to
notify the NRC of your intentions. If we have not heard from you within 10 days, we will
continue with our significance determination and enforcement decision and you will be advised
by separate correspondence of the results of our deliberations on this matter.
Since the NRC has not made a final determination in this matter, no Notice of Violation is being
issued for the corrective action program finding at this time. In addition, please be advised that
the number and characterization of the apparent violations described in the enclosed report
may change as a result of further NRC review.
The NRC inspection also identified one additional issue that was evaluated under the risk
significance determination process as having very low safety significance (Green). The NRC
has also determined that a violation was associated with this issue. The violation is being
treated as a noncited violation (NCV), consistent with Section VI.A of the Enforcement Policy.
The NCV is described in the subject inspection report. If you contest the violation or
significance of the NCV, you should provide a response within 30 days of the date of this
inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission,
ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional
Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611 Ryan Plaza Drive,
Suite 400, Arlington, Texas 76011; the Director, Office of Enforcement, U.S. Nuclear
Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the
Callaway Plant facility.
In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter, its
enclosure, and your response will be made available electronically for public inspection in the
NRC Public Document Room or from the Publicly Available Records (PARS) component of
NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at
http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Should you have any questions concerning this inspection, we will be pleased to discuss them
with you.
Sincerely,
/RA/
Arthur T. Howell lll, Director
Division of Reactor Safety
Docket: 50-483
License: NPF-30
Union Electric Company 3
Enclosure:
NRC Inspection Report
50-483/02-07
cc w/enclosure:
Professional Nuclear Consulting, Inc.
19041 Raines Drive
Derwood, Maryland 20855
John ONeill, Esq.
Shaw, Pittman, Potts & Trowbridge
2300 N. Street, N.W.
Washington, D.C. 20037
Mark A. Reidmeyer, Regional
Regulatory Affairs Supervisor
Regulatory Affairs
AmerenUE
P.O. Box 620
Fulton, Missouri 65251
Manager - Electric Department
Missouri Public Service Commission
301 W. High
P.O. Box 360
Jefferson City, Missouri 65102
Ronald A. Kucera, Deputy Director
for Public Policy
Department of Natural Resources
205 Jefferson Street
Jefferson City, Missouri 65101
Otto L. Maynard, President and
Chief Executive Officer
Wolf Creek Nuclear Operating Corporation
P.O. Box 411
Burlington, Kansas 66839
Dan I. Bolef, President
Kay Drey, Representative
Board of Directors Coalition
for the Environment
6267 Delmar Boulevard
University City, Missouri 63130
Union Electric Company 4
Lee Fritz, Presiding Commissioner
Callaway County Courthouse
10 East Fifth Street
Fulton, Missouri 65251
J. V. Laux, Manager
Quality Assurance
AmerenUE
P.O. Box 620
Fulton, Missouri 65251
Jerry Uhlmann, Director
State Emergency Management Agency
P.O. Box 116
Jefferson City, Missouri 65101
Gary McNutt, Director
Section for Environmental Public Health
P.O. Box 570
Jefferson City, Missouri 65102-0570
John D. Blosser, Manager
Regulatory Affairs
AmerenUE
P.O. Box 620
Fulton, Missouri 65251
Union Electric Company 5
Electronic distribution from ADAMS by RIV:
Chairman Meserve (MESERVE)
Commissioner Diaz (JXD2)
Commissioner Dicus (DICUS)
Commissioner McGaffigan (EXM)
Commissioner Merrifield (JMER)
John Larkins, Executive Director, ACRS (JTL)
W. Travers, EDO (WDT)
S. Collins, D/NRR (SJC1)
J. Donohew, Project Manager (JND)
S. Newberry, Director, RES/DRAA (SFN)
M. Mayfield, Director, RES/DET (MEM2)
C. Nolan, OE (MCN)
T. Frye, NRR (TJF)
Regional Administrator (EWM)
DRP Director (KEB)
DRS Director (ATH)
DRS/STA (DAP)
G. Sanborn, (GFS)
Senior Resident Inspector (VGG)
Branch Chief, DRP/B (DNG)
Senior Project Engineer, DRP/B (RAK1)
DRP/TSS (PHH)
RITS Coordinator (NBH)
OEMAIL
Only inspection reports to the following:
Scott Morris (SAM1)
CWY Site Secretary (DVY)
Hard Copy:
Records Center, INPO
DOCUMENT NAME: R:\_CW\CW2002-07RP-TWP.WPD
RIV:DRS\SRA NRR/PM SRI RI D:ACES
- TWPruett:nlh *JNDonohew *TLHoeg *TWJackson *GFSanborn
/RA/ TWPruett - T TWPruett - T TWPruett - E /RA/
03/08/02 03 /08 /02 03 /07/02 03/05 /02 03/13/02
D:DRP DRS/STA D:DRS
- KEBrockman DAPowers ATHowell lll
/RA/ /RA/ /RA/
03/11/02 03/20/02 03/20/02
OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax
- Previously concurred.
ENCLOSURE 1
U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
Docket: 50-483
License: NPF-30
Report No.: 50-483/02-07
Licensee: Union Electric Company
Facility: Callaway Plant
Location: Junction Highway CC and Highway O
Fulton, Missouri
Dates: January 28 through February 27, 2002
Inspectors: Troy W. Pruett, Senior Reactor Analyst (Team Leader)
Timothy L. Hoeg, Senior Resident Inspector
Terry W. Jackson, Resident Inspector
Jack N. Donohew, Project Manager
Approved By: Arthur T. Howell lll
Attachments: Supplemental Information
Augmented Inspection Team Charter
Sequence of Events
System Figures
SUMMARY OF FINDINGS
Callaway Plant
NRC Inspection Report 50-483/02-07
IR 05000483-02-07; on 01/28-02/27/2002; Union Electric Co; Callaway Plant. Augmented
Inspection Report; Problem Identification and Resolution, Modifications. One preliminary White
finding.
The inspection was conducted by regional inspectors and an Office of Nuclear Reactor
Regulation project manager. The inspection identified one apparent violation and one noncited
violation of NRC requirements. The significance of most findings is indicated by their color
(Green, White, Yellow, Red) using Inspection Manual Chapter 0609 Significance Determination
Process. Findings for which the significance determination process does not apply are
indicated by No Color or by the severity level of the applicable violation. The NRCs program
for overseeing the safe operation of commercial nuclear power reactors is described at its
Reactor Oversight Process website at
http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/index.html.
Identification and Resolution of Problems
The team determined that several opportunities were missed to promptly identify and correct a
risk significant condition adverse to quality involving the degraded condition of the condensate
storage tank diaphragm seal. Quality assurance personnel were not actively involved in
providing oversight of the event review team and root cause investigation processes. The
event review team process did not ensure that statements were obtained from all personnel
involved in the event. The corrective action program did not include guidance or expectations
on the assignment of appropriate resources to review the highest classification of significant
conditions adverse to quality. Minimal resources were initially assigned to the root cause
investigation and may have contributed to the delay in identifying the degraded diaphragm seal.
Based on interviews with the licensees staff and a review of the corrective action program
procedure, the team determined that licensed operators were only notified of equipment
deficiencies if the individual discovering the condition believed there was an immediate impact
on nuclear, plant, or personnel safety. Consequently, the potential existed for operability
decisions to be made by non-licensed personnel. The operability evaluation program did not
implement the guidance provided in NRC Generic Letter 91-18, Information to Licensees
Regarding NRC Inspection Manual Section on Resolution of Degraded and Nonconforming
Conditions.
Cornerstone: Mitigating Systems
+ TBD. Between January 1992 and January 31, 2002, several opportunities were missed
to promptly identify and correct a significant condition adverse to quality involving
foreign material in the auxiliary feedwater system and condensate storage tank. The
failure to promptly identify the degraded condition resulted in the failure of an auxiliary
feedwater pump on December 3, 2001. In addition, between January 25 and 29, 2002,
the identification of a significant condition adverse to quality involving the as-found
condition of the degraded diaphragm seal was not reported to the appropriate levels of
management. The multiple examples of missed opportunities to identify a significant
condition adverse to quality was a violation of 10 CFR Part 50, Appendix B,
2
Criterion XVI and also represented a significant human performance cross cutting issue
involving the timely recognition of degraded conditions.
The finding had greater than minor significance because there was a credible impact on
plant safety. Specifically, auxiliary feedwater Pump A failed to run when started by
operations personnel during a plant shutdown. Had a plant event occurred, the potential
existed for foam from the degraded condensate storage tank diaphragm to fail one or
more auxiliary feedwater pumps. The failure of an auxiliary feedwater pump would have
adversely affected the decay heat removal critical safety function. A Significance
Determination Process Phase 3 analysis preliminarily determined that the issue had low
to moderate safety significance (White). This finding was entered in the licensees
corrective action program as Callaway Action Request System Item CARS 200107423.
+ Green. Calculations for auxiliary feedwater pump net positive suction head did not
account for nitrogen saturated water. The failure of calculational methods to verify the
adequacy of net positive suction head requirements for the auxiliary feedwater pumps
was a violation of 10 CFR Part 50, Appendix B, Criterion III.
The failure to account for nitrogen saturated water in the net positive suction head
calculation for the AFW pumps was more than minor because there was a credible
impact on safety in that the available margin of net positive suction head was reduced
by 11 feet. Using Phase 1 of the Significance Determination Process, the issue was
determined to be of very low safety significance because adequate available net positive
suction head remained after accounting for dissolved nitrogen. Therefore, the auxiliary
feedwater pump would have remained available during an actual plant event. The
finding was entered in the licensees corrective action program as Callaway Action
Report System Item CARS 200200485.
TABLE OF CONTENTS
1.0 Description of Event and Chronology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 System Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Event Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Preliminary Risk Significance of Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Sequence of Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.0 Human Factors and Procedural Aspects of the Event . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3.0 Root Cause of Equipment Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1 Auxiliary Feedwater Pump A Root Cause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.2 Corrective Actions Associated With Auxiliary Feedwater Pump A . . . . . . . . . . . 3
3.3 Condensate Storage Tank Diaphragm Root Cause . . . . . . . . . . . . . . . . . . . . . . 8
3.4 Corrective Actions Associated With Condensate Storage Tank Degraded Seal. 9
4.0 Contributing Causes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.1 Operating Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2 Nitrogen Effect on Condensate Storage Tank Diaphragm Seal . . . . . . . . . . . . 13
4.3 Nitrogen Effect on Net Positive Suction Head (NPSH) . . . . . . . . . . . . . . . . . . . 14
4.4 Condensate Storage Tank Modification Corrective Actions . . . . . . . . . . . . . . . 15
4.5 Condensate Storage Tank Seal Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.6 Operability Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.0 Extent of Condition Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.0 Quality Assurance Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6.1 Review of the Quality Assurance Aspects of the Event . . . . . . . . . . . . . . . . . . 21
6.2 Quality Assurance Audits of Foreign Material and Operating Experience . . . . 22
7.0 Risk Significance of Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
8.0 Overall Adequacy of the Licensees Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
9.0 Exit Meeting Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
ATTACHMENT 1 - Supplemental Information
ATTACHMENT 2 - Augmented Inspection Team Charter
ATTACHMENT 3 - Sequence of Events
ATTACHMENT 4 - System Figures
Figure 1 - Auxiliary Feedwater System Simplified Diagram
Figure 2 - Condensate Storage Tank Diaphragm Seal
Figure 3 - Degraded Diaphragm Seal
Figure 4 - Condensate Storage Tank Configuration
Report Details
1.0 Description of Event and Chronology
1.1 System Descriptions
Auxiliary Feedwater System
The auxiliary feedwater (AFW) system has 2 motor-driven pumps (A and B) and
1 turbine-driven pump. The pumps were normally aligned through a common suction
line to the non-safety related condensate storage tank (CST). The essential service
water system was credited as the safety-related source of make-up water to the steam
generators. The AFW pump suction automatically switched from the CST to the
essential service water system following a low level condition in the CST (See Figure 1).
Condensate Storage Tank Diaphragm
The CST diaphragm was a rigid structure that floated on the surface of the water to
minimize the oxygen content of the water in the CST. The diaphragm was
approximately 43 feet in diameter and constructed of rigid closed cell foam laminated
with fiberglass and polyester resin. A seal was attached to the diaphragm and located
between the outer periphery and the inner surface of the CST. The seal was
constructed of soft pliable polyurethane foam covered with a Teflon-coated fiberglass
fabric. The fabric reduced the frictional forces between the seal and the inside wall of
the CST when the water level fluctuated (See Figure 2).
1.2 Event Summary
On December 3, 2001, at 1:15 p.m., operations personnel commenced a reactor
shutdown to repair a leaking main generator bushing. At 10:39 p.m, the Number 4 main
turbine bearing high vibration alarm annunciated in the main control room. At
10:48 p.m., the Number 4 bearing vibration increased to 10 mils and operations
personnel manually tripped the main turbine. Operations personnel prepared to reduce
main condenser vacuum to alleviate the increasing Number 4 bearing vibration. In
anticipation of breaking main condenser vacuum, operations personnel started AFW
Pump B followed by AFW Pump A. At 10:57 p.m., the Number 4 bearing vibration
exceeded 15 mils and operations personnel broke main condenser vacuum. At
10:58 p.m., operations personnel observed reduced pressure and flow from AFW
Pump A and no flow to Steam Generators B and C. Operations personnel started the
turbine-driven AFW pump and dispatched the field supervisor and an equipment
operator to AFW Pump Room A. The field supervisor and the equipment operator
observed that AFW Pump A had no leak-off flow from the outboard pump packing. The
field supervisor left AFW Pump Room A to inspect the turbine-driven AFW pump. Upon
returning to AFW Pump Room A, the field supervisor and the equipment operator
observed that the AFW Pump A outboard pump packing housing was hot to the touch.
The field supervisor contacted the main control room and recommended that AFW
Pump A be secured. At 11:07 p.m., main control room personnel secured AFW
Pump A. Operations personnel continued to cool down the reactor coolant system and
entered Mode 3 at 11:19 p.m.
2
1.3 Preliminary Risk Significance of Event
Following the December 3, 2002, failure of AFW Pump A, the NRC completed an
evaluation of the preliminary risk significance. The analysis determined that the
conditional core damage probability (CCDP) was approximately 1.1E-6. The CCDP is
the probability of core damage over a period of time given a specific plant condition.
The CCDP analysis assumed that there was no potential for common cause failure of
the remaining two AFW pumps and that the duration associated with the failure of AFW
Pump A was less than 30 days. In response to the failure of AFW Pump A, the NRC
determined that a Special Inspection Team (SIT) would assess the causes of the event
during the week of January 28, 2002.
On January 27, 2002, the licensees root cause investigation determined that
polyurethane foam from a degraded CST floating diaphragm may have caused the
failure of AFW Pump A.
On January 28, 2002, the SIT determined that the duration of the degraded condition
could have been greater than 1 year. In addition, the SIT determined that the potential
existed for common cause failure of the AFW system or multiple AFW pumps. The
NRC completed an additional analysis of the event and determined that the new
estimated CCDP was in the range of approximately 5E-5 to 5E-4.
NRC Management Directive 8.3, NRC Incident Investigation Program, required the
consideration of the initiation of an Augmented Inspection Team (AIT) when the
estimated CCDP was greater than or equal to 1E-5. Based on the potential for a
substantial increase in risk stemming from common mode failure implications, the NRC
upgraded the SIT to an AIT on January 31, 2002.
1.4 Sequence of Events
The team developed a detailed sequence of events and organizational response
time-line. The time-line included applicable events and actions before, during, and
following the failure of AFW Pump A on December 3, 2001. The time-line was
generated from control room computer printouts, operator logs, written records, and
interviews with members of the licensees staff. The teams review satisfied the
activities associated with AIT Charter Element 1, Develop a complete description and
sequence of events related to the subject AFW pump failure (including the degraded
CST floating diaphragm), and operator actions taken in response to regain feedwater
flow. The AIT Charter is provided as Attachment 2. The detailed sequence of events is
provided as Attachment 3.
2.0 Human Factors and Procedural Aspects of the Event
a. Inspection Scope
The team reviewed operator actions associated with the failure of AFW Pump A and the
actions taken to restore feedwater flow to Steam Generators B and C. The team
interviewed operations personnel, evaluated control room logs and trend data, analyzed
3
AFW pump performance data, and reviewed control room operating procedures. The
teams review satisfied a portion of the activities associated with AIT Charter Element 5,
Identify any human factor, procedural or quality assurance deficiencies that may have
contributed to the condition.
b. Observations and Findings
No deficiencies were identified with the operator actions associated with the event or
with procedures utilized during the event.
3.0 Root Cause of Equipment Failures
3.1 Auxiliary Feedwater Pump A Root Cause
a. Inspection Scope
The team reviewed the results of the licensees investigation documented in Callaway
Action Request System Item CARS 200107423, Auxiliary Feedwater System Event
Review, to determine if the root cause was of appropriate scope including;
independence, completeness, and accuracy to identify the probable causes of the
failure of AFW Pump A. The team reviewed operations documents, corrective action
documents, maintenance records, and operating experience. The team completed a
walk down of portions of the AFW system and CST. The team also interviewed several
members of the licensees staff. The teams review satisfied a portion of the activities
associated with AIT Charter Element 2, Review the licensees root and probable cause
determination for independence, completeness, and accuracy, including the licensees
assessment of the risk associated with the condition.
b. Observations and Findings
The team determined that the licensees investigation into the failure of AFW Pump A
was of adequate scope and detail to conclude that the pump failure was due to the
entrainment of a piece of foam from the CST diaphragm seal. The foam material
entered the eye of the first stage impeller and produced a localized low pressure region.
The low pressure region caused gas to come out of solution and create voids in AFW
Pump A. The voiding led to a partially air-bound pump which was incapable of
developing the required pump discharge pressure and flow.
3.2 Corrective Actions Associated With Auxiliary Feedwater Pump A
a. Inspection Scope
The team assessed the licensees prompt and long-term corrective actions to address
the root and probable causes of the failure of AFW Pump A. The team reviewed the
licensees root cause analysis, the event review team report, the past operating and
maintenance history for AFW Pump A, surveillance test data, the results of system
walkdown inspections, vendor information, boroscopic inspections of the AFW system,
and inspections of the CST. The team assessed the adequacy of AFW system
inspections by direct observation or by viewing video footage and photographs. The
4
teams review satisfied a portion of the activities associated with AIT Charter Element 7,
Identify and assess the licensees prompt and long-term corrective actions to address
the root and probable causes of the condition.
b. Observations and Findings
The team identified several examples of an apparent violation of 10 CFR Part 50,
Appendix B, Criterion XVI, Corrective Actions. Each of the examples involved a
missed opportunity to promptly identify and correct a significant condition adverse to
quality associated with foreign material in the AFW system and CST.
On December 3, 2001, operations personnel completed a vent-and-fill of AFW Pump A.
Air was vented from the pump casing for approximately 15 seconds. No steam was
released during the venting. Therefore, the team determined that the licensee
appropriately concluded that the failure of AFW Pump A was not due to steam binding.
On December 4, 2001, the licensee completed a walkdown of AFW Train A. The team
reviewed Procedure OSP-AL-00001, AFW Flow Paths Valve Alignment, Revision 5,
which described the proper system alignment for the AFW system. The team also
interviewed operations and engineering personnel involved with the system walkdown.
The team determined that the licensee appropriately concluded that there was no
evidence of an improper valve alignment or an obvious pathway for air-entrainment into
the AFW system.
On December 4, 2001, the licensee convened an event review team to gather
information that would aid in identifying the root cause of the failure of AFW Pump A.
The team reviewed UOTH 01-0047, Event Review Meeting Minutes: High Main Turbine
Vibration, AFAS, and No Flow on A-AFW Pump, and concluded that the report
accurately reflected the sequence of events. However, during interviews with operations
personnel, the team determined that the event review team did not obtain statements
from the two reactor operator trainees who were in the control room at the time of the
event, or the equipment operator who was the first person to monitor and vent AFW
Pump A. Furthermore, the event review team meeting minutes were not distributed to
the control room supervisor, the reactor operator trainees, or the equipment operator for
verification of information.
The team assessed the licensees review of the operational and maintenance history for
AFW Pump A. The team also reviewed the previous 2 years of test data collected with
Procedure OSP-AL-P001A, Motor Driven Aux. Feedwater Pump A In Service Test,
and the previous 2 years of maintenance activities competed on AFW Pump A. The
team determined that the licensee appropriately concluded that past operational and
maintenance activities associated with AFW Pump A did not contribute to the root
cause.
On December 4, 2001, the licensee initiated Callaway Action Request System Item
CARS 200107423, Auxiliary Feedwater System Event Review, to investigate the event.
The licensee classified the significance of Callaway Corrective Action System Item
CARS 200107423 as Level 1 and specified that a formal root cause evaluation should
be completed. The acting mechanical system engineering supervisor and a root cause
5
analyst from the corrective action group were assigned to complete the investigation.
Based on discussions with the licensees staff, the team determined, on the basis of
interviews, that the root cause analyst was not actively involved in the investigation until
mid-to-late January 2002. Based on interviews with the licensees staff and a review of
Procedure APA-ZZ-00500, Corrective Action Program, the team determined that there
were no formal requirements or expectations on the formulation of teams to review the
highest classification of significant conditions adverse to quality.
On December 5, 2001, the licensee started AFW Pump A in order to troubleshoot and
attempt to recreate the failure mechanism. The licensee determined that AFW Pump A
could not meet the minimum flow requirements of the surveillance test procedure and
that the outboard shaft seal stuffing box temperature increased abnormally. The
licensee replaced the outboard shaft seal packing, re-baselined the pump performance
criteria, and satisfactorily completed the pump surveillance test. In addition, the
licensee implemented compensatory measures to perform shiftly vents of all AFW
pumps and decreased the surveillance test interval from quarterly to monthly for each
AFW pump (In January 2002, the licensee decreased the AFW Pump A frequency to
weekly). The licensee determined that the most probable cause of the AFW Pump A
failure was air intrusion into the AFW system. Although no cause had been confirmed,
the licensee determined that the compensatory measures involving shiftly pump venting
and increased testing were sufficient to declare AFW Pump A operable.
Technical Manual M-021000061, Instruction Manual for Ingersoll Rand Centrifugal
Pumps, Revision 25, recommended opening and inspecting the pump when insufficient
pump capacity and stuffing box overheating were experienced. The technical manual
troubleshooting chart identified foreign material as a probable cause for both the
insufficient flow and stuffing box overheating, which were symptoms experienced on
December 3 and 5, 2001. In addition, the guidance in Table 5-2 of Electric Power
Research Institute TR-114612-V2, Pump Troubleshooting, was consistent with the
technical manual information. Nevertheless, as of December 5, 2001, the licensee did
not consider foreign material as a credible failure mechanism. The team determined
that the licensees lack of an evaluation of foreign material in the AFW system or CST
as a possible cause immediately following the event was an example of not promptly
identifying and correcting a significant condition adverse to quality. This missed
opportunity to promptly identify a condition adverse to quality represented an example of
a significant human performance crosscutting issue involving the timely recognition of
degraded conditions. The significance of this cross cutting issue and other examples of
the apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI, is documented in
Section 7.
On December 6, 2001, the licensee formally initiated the investigation into the failure of
AFW Pump A. In addition, the licensee added a system engineer to the root cause
investigation. A vibrations engineer and a licensed operator were utilized on an
as-needed basis.
On December 14, 2001, the licensee contacted the pump vendor and discussed
possible causes of the failure of AFW Pump A. The pump vendor recommended that
the licensee inspect the seal water piping for a possible flow obstruction. Based on
interviews with engineering personnel, the team determined that the licensee continued
6
to believe that foreign material was an unlikely cause since the CST was assumed to be
free of foreign material and because the AFW pumps had been operated successfully.
Consequently, the licensee determined that the inspection of the seal water line could
be delayed until the next regularly scheduled maintenance interval for AFW Pump A.
The team determined that not promptly following the vendors recommendation to
inspect the seal water line for obstructions was an additional example of the significant
human performance cross cutting issue involving the timely recognition of degraded
conditions.
During the January 8, 2002, management meeting, the licensee added individuals from
regulatory affairs and engineering to the root cause investigation. In addition, the
licensee decided to bring the pump vendor and a pump contractor to the site during the
January 14, 2002, maintenance week for AFW Pump A.
Following the January 8, 2002, management meeting, the licensee narrowed the root
cause of the failure of AFW Pump A to three possibilities: (1) low net positive suction
head (NPSH), (2) nitrogen gas disassociation, or (3) foreign material. However, the
licensee continued to believe that foreign material was an unlikely cause since there
was no recent activity which would have introduced foreign material into the CST or the
AFW system. Therefore, foreign material continued to receive a low priority in the root
cause investigation. The team concluded that narrowing the potential root causes to
those listed above was appropriate. However, this conclusion could have been reached
in a more timely manner and causes not related to the event could have been eliminated
sooner. Specifically, the following information was available to the licensee following the
event:
1. Evidence of air during the vent-and-fill process for AFW Pump A suggested that
either air/gas was entrained into the pump, that gas came out of solution, or that
foreign material caused the gas to come out of solution at the eye of the
impeller.
2. System walkdowns performed after the event did not identify any source of air
intrusion.
3. The inability to re-create a failure of AFW Pump A due to air entrainment in
subsequent pump runs on December 5, 2001, demonstrated that air-entrainment
was an unlikely cause of pump failure.
4. The absence of air during shiftly venting indicated that air intrusion was unlikely.
5. Vendor and industry information suggested that foreign material was a possible
cause for the failure of AFW Pump A.
The team determined that the 35-day delay in identifying the potential root causes was
an additional example of the significant human performance cross cutting issue
involving the timely recognition of degraded conditions.
10 CFR Part 50, Appendix B, Criterion XVI, requires, in part, that measures be
established to assure that conditions adverse to quality, are promptly identified and
7
corrected. For significant conditions adverse to quality, the measures shall assure that
the causes of the condition are determined and corrective actions taken to preclude
recurrence. The identification of significant conditions adverse to quality shall be
documented and reported to appropriate levels of management.
During the week of January 15, 2002, the licensee contracted the services of the pump
vendor, Flowserve, and a pump consultant, Dominion Engineering. The consultants
were present during the testing of AFW Pump A on January 15, 2002. The licensee
also inspected the seal water cooling line on AFW Pump A and found a piece of foam
lodged in the orifice. Even though foreign material was identified in the AFW system,
the licensee continued to believe that foreign material was not a credible cause for the
failure of AFW Pump A. The team determined that the failure to promptly identify and
correct the degraded CST diaphragm seal after discovering foreign material in the seal
water cooling line was an example of an apparent violation of 10 CFR Part 50,
Appendix B, Criterion XVI (APV 05000483/0207-01). In addition, this missed
opportunity to promptly identify a condition adverse to quality represented an additional
example of a significant human performance cross cutting issue involving the timely
recognition of degraded conditions.
On January 17, 2002, the licensee began ultrasonic testing of the AFW pump suction
piping to determine if nitrogen gas was coming out of solution and collecting in the
system. The team reviewed Procedure QPC-ZZ-05046, Ultrasonic Examination
Procedure for Determining Liquid Level in Pipes and Components, Revision 0, and
recorded test data obtained between January 17 and 31, 2002. The team determined
that the ultrasonic procedure was adequate, and that the licensees interpretation of the
data was appropriate. The team noted that the licensee believed that air intrusion was
the most probable cause of the AFW Pump A failure between December 4, 2001, and
January 24, 2002. Therefore, the team concluded that the initiation of ultrasonic testing
was untimely in that testing could have commenced at an earlier date to help eliminate
air or gas accumulation in the suction piping as a root cause.
On January 23, 2002, with the assistance of the consultants, the licensee eliminated low
NPSH and air intrusion as potential root causes of the failure of AFW Pump A. The
pump vendor provided a report titled, Evaluation of Auxiliary Feedwater Pump 1A Event
12/3/01, and the pump consultant provided a report titled, Auxiliary Feed Pump Event.
On January 24, 2002, the licensee increased the level of effort associated with the root
cause investigation as preparations were made to inspect the CST for foreign material.
The licensee also decided to complete the AFW system inspections during the regularly
scheduled maintenance window for each train.
On January 26, 2002, the licensee identified foreign material in the CST (See
Sections 3.3 and 3.4).
On January 30, 2002, the licensee initiated boroscopic inspections of AFW Pump B, the
turbine-driven AFW pump, AFW system suction piping, and portions of the AFW system
discharge piping. The team reviewed several work orders and found that the scope of
the inspection activities was appropriate. The following is a sample of work orders
reviewed:
8
+ W220254, Replace Rotating Element on A MDAFP
+ W682227, Suction Piping Inspection
+ W687231, A MDAFP Discharge Piping Inspection
+ W687232, B MDAFP Discharge Piping Inspection
+ W687233, TDAFP Discharge Piping Inspection
On January 31, 2002, the licensee significantly increased the team size and scope of
the root cause investigation.
The team observed portions of the boroscopic inspections and found them to be
appropriate. The licensee identified one additional piece of foam lodged in the AFW
Pump A casing vent line. No other foreign material was identified. The team
determined that the scope of the licensees inspections of the AFW system and CST
were sufficient to ensure the AFW system would perform its safety function.
3.3 Condensate Storage Tank Diaphragm Root Cause
a. Inspection Scope
The team reviewed the licensees root cause investigation documented in Callaway
Corrective Action System Item CARS 200107423, to determine if it was of appropriate
scope including; independence, completeness, and accuracy to identify the probable
causes of the failure of the CST diaphragm. The team reviewed operations documents,
corrective action documents, maintenance records, and operating experience
associated with the AFW pumps and CST. The team completed a walk down of
portions of the AFW system and CST. The team also interviewed several members of
the licensees staff. The teams review satisfied a portion of the activities associated
with the AIT Charter Element 2, Review the licensees root and probable cause
determination for independence, completeness, and accuracy, including the licensees
assessment of the risk associated with the condition.
b. Observations and Findings
On January 26 and 27, 2002, the licensee found foreign material in the CST. The
foreign material mainly consisted of pieces from the diaphragm seal assembly. The seal
materials consisted of a Teflon-coated fiberglass fabric, flexible polyurethane foam,
caulking material, and a piece of fiberglass (See Figure 2).
The team determined that the licensees investigation of the CST diaphragm seal failure
was of adequate scope and detail to conclude that the degraded Teflon-coated fabric
and inner foam material was due to constant nitrogen sparging of the CST at 5 standard
cubic feet per minute. The nitrogen bubbles from the sparging impinged on the
diaphragm seal and increased the wear on a 6-foot section of the approximately
130-foot circumference. The licensee calculated that the impact forces were on the
magnitude of 0.02-lb force on each square inch of the lower surface of the seal at a rate
of approximately 68,700 cycles per impact site over a 4.5-year period. Once the Teflon
coating failed, the same mechanism acted directly on the foam. This failure mechanism
continued to occur until sections of foam became detached and settled on the bottom of
the tank.
9
3.4 Corrective Actions Associated With Condensate Storage Tank Degraded Seal
a. Inspection Scope
The team assessed the licensees prompt and long-term corrective actions to address
the root and probable causes of the failure of the CST diaphragm seal. The team
reviewed the licensees root cause analysis, the past operating and maintenance history
for the seal, and inspections of the CST. The team assessed the adequacy of CST
inspections by direct observation or by viewing video footage and photographs. The
teams review satisfied a portion of the activities associated with AIT Charter Element 7,
Identify and assess the licensees prompt and long-term corrective actions to address
the root and probable causes of the condition.
b. Observations and Findings
The team identified an additional example of the apparent violation of 10 CFR Part 50,
Appendix B, Criterion XVI. Specifically, members of the licensees staff failed to notify
management of a significant condition adverse to quality.
On January 24, 2002, the licensee initiated efforts to inspect the CST. Initially, the
assessment consisted of using a diver to inspect the CST and diaphragm seal. During
the divers inspection, approximately 71 inches of the diaphragm seal was found
degraded (See Figure 3). The diver removed a 25-inch section of foam hanging from
the diaphragm. The diver also found various pieces of foam, Teflon-coated fabric,
caulking, and two rubber pads in the CST sump. The foam found in the CST sump
included a section measuring 12 x 3 x 8 inches, a section measuring 2 x 3 x 8 inches,
and approximately 10 smaller pieces of foam under 1 cubic inch in total volume. No
foam was located outside the sump; however, some caulking and two additional rubber
pads were recovered from the CST floor.
The team reviewed Work Orders W202393, CST Seal Inspection, and W686916,
Camera and Diver Inspection of CST. The diver was given instructions to enter the
CST and assess the integrity of the diaphragm seal cover and the attachment of the
seal to the diaphragm. In the process of communicating the integrity of the diaphragm
seal cover, the licensees staff incorrectly understood that the seal cover had a single
tear at the damaged area and did not have any missing pieces. The fabric found in the
CST sump was incorrectly assumed to have been left in the CST during construction of
the floating diaphragm. This misunderstanding led senior licensee management to
believe the diaphragm seal would not present any impact on plant safety once the
repairs outlined in Work Order W686916 were performed.
On January 30, 2002, following a viewing of video footage with the team, senior licensee
management contacted the diver and questioned the as-found condition of the seal.
Senior licensee management was informed that the Teflon fabric did not have a single
tear, but was missing pieces and had jagged edges.
Even though foam was found in the AFW Pump A seal water line, as well as the CST,
the licensees operability evaluation included in Callaway Corrective Action System Item
CARS 200200489, Complete an Operability Evaluation on the CST Due to the
10
Existence of the Foam Pieces, did not document activities to determine the potential for
foreign material to have been retained in other sections of the AFW system. The team
interviewed senior licensee management to determine what activities were completed by
the licensee following the discovery of foreign material in the CST. Based on the
interviews, the team determined the following:
+ Before entering the CST, senior management had implemented compensatory
measures to declare the CST inoperable and align the AFW suction source to
the essential service water system if significant foreign material was identified.
+ During the morning of January 27, 2002, the licensee had evaluated the effect
on the AFW system from foreign material in the CST. Even though the extent of
the diaphragm seal degradation was not yet known, senior management
concluded that the effect of foreign material on the AFW system was low due to
the number of hours the AFW pumps had operated, the visual clarity of water in
the CST, and the lack of debris on the floor of the CST.
+ During the evening of January 27, 2002, the licensee removed the foreign
material from the CST. Senior management was informed that the material had
been removed and that the CST was clean.
+ During the morning of January 28, 2002, the licensee believed that the removed
material had been in the CST for an extended period of time. Senior
management determined that the Teflon material was not an operability concern
(at this time senior management had been incorrectly informed on several
occasions that the Teflon material had a single tear). In addition, senior
management believed that the foam was not an operability concern based on the
run times for each of the AFW pumps.
+ During the evening of January 28, 2002, senior management questioned
engineering personnel about the Teflon fabric. Senior management was
incorrectly informed that the area where the foam was missing was intact (i.e.,
single tear in material). Senior management determined that the Teflon material
had probably been in the CST since construction.
+ During the afternoon of January 29, 2002, the licensee discussed the results of
inspection activities with the team. The licensee indicated that no inspections of
the AFW system were planned for the near term. The team subsequently
determined that the licensee had planned to inspect small diameter piping
associated with the AFW pumps during the next regularly scheduled planned
maintenance period for each AFW pump train.
+ On January 30, 2002, the team viewed the video footage of the CST and
degraded diaphragm seal with the plant manager. The team determined that the
plant manager had not viewed the footage involving the degraded diaphragm
seal. After viewing the video footage, the team and the plant manager
questioned the validity of the information obtained from the licensees staff
regarding the as-found condition of the diaphragm seal. The plant manager then
re-interviewed the diver and determined that the Teflon material did not have a
11
single tear, but had jagged edges and was missing multiple pieces. Based on
the information obtained from the diver, senior licensee management determined
that the appropriate action would be to shutdown the plant and complete
additional inspections of the CST and AFW system.
The team determined that the incorrect information provided by the licensees staff to
senior management delayed the initiation of the extent of condition review associated
with the discovery of foreign material in the CST. Once initiated, the extent of condition
review was thorough and provided assurances that the AFW system did not contain any
further foreign material. The team determined that the failure of the licensees staff to
correctly report a significant condition adverse to quality to the appropriate levels of
management in a timely manner was an additional example of the apparent violation of
10 CFR Part 50, Appendix B, Criterion XVI.
4.0 Contributing Causes
4.1 Operating Experience
a. Inspection Scope
The team reviewed industry operating experience information related to tank
diaphragms to determine if the licensee applied the data appropriately. The review
consisted of interviewing licensee personnel, searching operating experience
databases, reviewing corrective action documents, reviewing the licensees responses
to operating experience information, and verifying licensee actions taken in response to
applicable operating experience. The teams review satisfied the activities associated
with AIT Charter Element 6, Identify and assess the licensees evaluation of applicable
industry operating experience.
b. Observations and Findings
On December 18, 1991, the NRC issued Information Notice (IN) 91-82, Problems with
Diaphragms in Safety Related Tanks. NRC IN 91-82 reminded licensees that
diaphragms in safety related tanks had finite service lives and could cause various
safety hazards if they fail. On January 21, 1992, the licensee initiated Callaway Action
Tracking System Item CATS 31040, Generate PM for Regular Inspection of CST
Cover, to ensure a maintenance activity to periodically inspect the CST diaphragm seal
was generated. On February 10, 1992, the licensee issued a response to
NRC IN 91-82 which specified that the licensee had experienced some problems with
rubber diaphragm type tank seals and had initiated programs to inspect tank internals
for degradation of diaphragms.
On September 15, 1999, the licensee initiated Callaway Corrective Action System Item
CARS 199901955, PM to Inspect CST Internal Cover Not Generated, in response to
an NRC resident inspector question regarding the condition of the CST diaphragm seal.
The licensee identified that Callaway Action Tracking System Item CATS 31040 had
been closed without performing an inspection of the diaphragm seal. The team
12
determined that the closure of Callaway Action Tracking System Item CATS 31040
without having completed an inspection may have contributed to the delay in identifying
the degraded condition of the CST diaphragm seal and was an additional example of
the apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI.
The licensee contacted the seal vendor and was informed that the only expected
degradation of the diaphragm seal would be wear of the outer Teflon skin covering the
foam seal as it rubbed against the tank wall when level fluctuated. The licensee realized
an inspection should be performed and initiated Generic Work Request G631231125,
Periodic Inspections of CST Diaphragm Seal, to perform periodic seal inspections.
After further review, the licensee determined that a specific inspection activity was more
appropriate than a generic inspection activity and initiated Work Order W201820,
Inspect CST Floating Cover, to inspect the seal in the Spring of 2000. At that time, the
CST seal had not been inspected since its final closeout inspection in 1982. The team
determined that the seal inspection activity per Work Order W201820 was rescheduled
twice (Spring and Summer of 2000) before being performed in October 2000. The
licensees basis for deferring the inspections was to allow enough time for completion of
the seal inspection work. The team determined that the licensees basis for delaying the
inspection did not fully consider the potential consequences of a degraded seal and was
an additional example of the apparent violation of 10 CFR Part 50, Appendix B,
Criterion XVI. In addition, this missed opportunity to promptly identify a condition
adverse to quality represented an additional example of a significant human
performance cross cutting issue involving the recognition of degraded conditions.
On October 17, 2000, the licensee performed a topside visual inspection of the CST
diaphragm seal. The licensee did not thoroughly inspect the outer seal fabric which the
vendor had stated one year earlier was vulnerable to wear failure. The team determined
that the scope of the licensees inspection was inadequate to determine that the
diaphragm seal was intact and was an additional example of the apparent violation of
10 CFR Part 50, Appendix B, Criterion XVI. In addition, this missed opportunity to
promptly identify a condition adverse to quality represented an additional example of a
significant human performance cross cutting issue involving the recognition of degraded
conditions.
The team also determined that the licensee had initiated Preventive Maintenance
Item P663567, Inspect CST Cover, to inspect the topside of the diaphragm seal on a
10-year periodicity. Preventive Maintenance Item P663567 did not include instructions
to inspect the bottom side of the seal assembly. Therefore, future inspections may not
have identified the degradation.
Significant Operating Experience Report 97-01
The team reviewed the licensees response to Significant Operating Experience
Report 97-01, Potential Loss of High Pressure Injection and Charging Capability From
Gas Intrusion. No deficiencies were identified.
13
4.2 Nitrogen Effect on Condensate Storage Tank Diaphragm Seal
a. Inspection Scope
The team reviewed the modifications to the CST that added and later modified a
nitrogen sparging system to control dissolved oxygen. The team reviewed two
modification packages; (1) Request for Resolution 04991, Revision A, Permanent N2
for the Condensate Storage Tank, dated May 4, 1988, to replace the temporary tubing
in CST by permanent connections; and (2) Restricted Modification Package 88-2016,
Revision A, dated November 20, 1992, to install the permanent nitrogen sparger system
in the CST. The documentation for the modification to add the temporary nitrogen
tubing to the CST in 1986 was not available.
b. Observations and Findings
The licensees evaluation of the modifications did not consider the effect of nitrogen on
the CST floating diaphragm. Specifically, the location of the nitrogen piping and how the
nitrogen would diffuse from the sparger and affect the floating diaphragm or the
AFW pump available NPSH was not considered. The licensee concluded, without
documentation, that the release of small quantities of nitrogen gas were insignificant,
not regulated by any state or federal agency, and that no unreviewed environmental
question existed.
There were no drawings of the temporary tubing or the permanent piping in the CST in
the documentation provided by the licensee. There was also no documentation of any
calculations performed to determine, based on the sparger design, the flow rate needed
to saturate the tank water with nitrogen or the effect of nitrogen on the diaphragm seal.
4.3 Nitrogen Effect on Net Positive Suction Head (NPSH)
a. Inspection Scope
The team reviewed Calculation AL-10, Determine the Available NPSH for the Auxiliary
Feedwater Pumps, dated July 21, 1977, as revised in 1980; Calculation AL-13, A
System Model of the AFW System, dated September 14, 1995; and Calculation AL-24,
Determine the Effect of Dissolved Nitrogen on Available NPSH for the Auxiliary
Feedwater Pumps, dated February 6, 2002, to determine if there was adequate NPSH
for AFW Pump A.
b. Findings and Observations
The team identified a violation of 10 CFR Part 50, Appendix B, Criterion III, Design
Control, which involved the failure account for the effect of nitrogen on AFW pump
NPSH.
Calculation AL-10 was completed for the CST and AFW system before the CST was
modified to add a nitrogen sparging system to control dissolved oxygen. Prior to 1988,
the amount of oxygen in the CST water was controlled by recirculating water to the main
condenser hotwell. However, the licensee determined that this method of removing
14
oxygen from the CST resulted in water temperatures during the summer months that
could exceed the design temperature of the AFW system suction piping. To prevent the
high temperature water, the licensee installed modifications to the CST to add a nitrogen
sparging system.
The team determined that the evaluations for the modifications did not address the
affect of the nitrogen saturated water on the pump tolerable volume fraction and the
available NPSH for the AFW pumps. The tolerable volume fraction of gases is a
measure of the quantity of free gases which can be passed by the pump without the gas
volume affecting pump performance (i.e., reducing pump performance to the point of
gas binding the pump). The NPSH required for a pump with a lower tolerable gas
volume fraction will be smaller than the NPSH required for a pump with a higher
tolerable gas volume fraction. NPSH is the minimum suction head required for a pump
to operate. The team determined that the tolerable volume fraction for AFW Pump A
was approximately 5 percent. The licensee determined that the expected volume of free
nitrogen, based on the reduction of pressure from the CST to the pump impeller eye,
was less than 5 percent.
On January 25, 2002, the licensee initiated Callaway Corrective Action System Item
CARS 200200485, Evaluate the Potential Effects of Dissolved Nitrogen on the Auxiliary
Feedwater System, following the identification that nitrogen had not been accounted for
in AFW pump NPSH calculations.
On February 6, 2002, the licensee completed Calculation AL-24 to address the effect of
nitrogen on the available NPSH for the AFW pumps. Calculation AL-24 was reviewed
by the team to determine if the assumptions were conservative and the calculated
available NPSH was correct. The equations used by the licensee were from the
following papers: (1) Daniel W. Wood, et. al., Proceedings of the 15th International
Pump Users Symposium, Application Guidelines for Pumping Liquids that have a Large
Dissolved Gas Content, pages 91 through 98, dated March 1998, (2) Mao J. Tsai,
Chemical Engineering, Accounting for Dissolved Gases in Pump Design, dated July
26, 1982; and (3) C. C. Chen, Chemical Engineering, Optimal System Design Requires
the Right Vapor Pressure. Heres How to Calculate It, pages 106 through 112, dated
October 1983.
Calculation AL-10, specified that the available NPSH for AFW Pump A was 28 feet,
assuming the CST water level was at the suction of the AFW system. The available
NPSH for AFW Pump A at the CST to essential service water system swap-over level
was 35 feet. Calculation AL-10 did not account for dissolved nitrogen because the CST
was not modified with a nitrogen sparger until 1988.
Calculation AL-24 included the effect of nitrogen saturated water, and assumed a
minimum temperature of 50oF, a 5 percent tolerable volume fraction of gas, and the
CST to essential service water system swap-over elevation. The team reviewed
Calculation AL-24 and determined that the reduction in available NPSH for AFW
Pump A, due to the effect of the dissolved nitrogen in the CST, was approximately
11 feet.
15
10 CFR Part 50, Criterion III, Design Control, required, in part, that the licensee
implement design control measures for verifying or checking the adequacy of the design
by the use of calculation methods. The team concluded that the licensees identification
that calculational methods failed to verify the adequacy of NPSH requirements for the
AFW pumps was a violation of 10 CFR Part 50, Appendix B, Criterion III
(05000483/0207-02). The team determined that the failure to account for nitrogen
saturated water in the NPSH calculation for the AFW pumps was more than minor
because there was a credible impact on safety in that the available margin of NPSH was
reduced by 11 feet. Using Phase 1 of the Significance Determination Process, the team
determined that the issue was of very low safety significance (Green) in that adequate
available NPSH remained after accounting for the affect of dissolved nitrogen.
Therefore, the availability of AFW pumps was not effected by the reduction in available
NPSH margin.
4.4 Condensate Storage Tank Modification Corrective Actions
a. Inspection Scope
The team reviewed Request for Resolution 21798, Revision D, Evaluate Permanent
Removal of TAP01 Floating Cover, dated February 1, 2002, to ensure the licensee
appropriately considered any negative consequences associated with the modification.
b. Observations and Findings
The team determined that members of the licensees staff believed that the non-safety
related CST did not have a function important to safety. The equipment qualification
impact review section of the licensing impact review form required that an individual
determine if the activity involved any safety-related structure, system, or component.
The individual was not required to state if the non-safety related structure, system, or
component had a safety function or if it was important to safety. The team noted that
there was a probabilistic risk assessment (PRA) review section on the licensing impact
review form which questioned if there was a potentially PRA-significant change to a
plant system.
The team noted that if the originator of the licensing impact review form did not
understand the safety function of the non-safety component, the answer to the question
about a potential PRA-significant change to a plant system may be incorrect. In this
case, the reviewer did not consider the change PRA-significant. The review was
subsequently re-evaluated by the PRA group. The team determined that the PRA staff
appropriately determined that the modification to remove the diaphragm seal was not
risk significant.
The team interviewed several members of the licensees staff and was informed that
evaluations associated with non-safety related equipment did not receive the same level
of rigor as evaluations associated with safety-related equipment. The team determined
from these interviews that this lack of rigor existed even in those instances in which the
non-safety component had a function which was important to safety.
16
The licensee also determined that not properly acknowledging the relationship between
the non-safety related CST and the safety-related AFW pumps led to non-conservative
decisions that may have contributed to December 3, 2001, event. The licensee planned
to address these concerns by bringing a heightened awareness to the plant organization
through group discussions or training.
4.5 Condensate Storage Tank Seal Maintenance
a. Inspection Scope
The team reviewed design information associated with the expected service life of the
diaphragm seal and the frequency of maintenance and inspection activities.
b. Observations and Findings
The team determined that not all of the original design specifications for the diaphragm
seal were readily available to the licensee during its investigation of the failure of the
diaphragm seal. CST Design Specification 10466-M-109 (Q), Section 5.27.4.3,
referenced a negotiated seal life guarantee and warranty period but did not provide any
actual life expectancy. The licensee was not able to produce any documentation
regarding the seal life or warranty information referenced in the design specifications.
The team determined that the lack of information may have contributed to a decrease in
the sensitivity for the need to complete periodic inspection and maintenance activities on
the diaphragm seal.
4.6 Operability Evaluations
a. Inspection Scope
The team reviewed three operability evaluations associated with the CST and AFW
system to determine the adequacy of the licensees review of degraded but operable
conditions. The team also compared Generic Letter 91-18, Revision 1, Information to
Licensees Regarding NRC Inspection Manual Section on Resolution of Degraded and
Nonconforming Conditions, to the licensees implementing procedures for operability
evaluations. The teams review satisfied a portion of the activities associated with AIT
Charter Element 3, Assess the timeliness and effectiveness of the licensees evaluation
of potential AFW common cause failure stemming from the degraded condensate
storage tank floating diaphragm.
b. Observations and Findings
The team identified several weaknesses associated with the licensees corrective action
and operability evaluation programs.
Procedure APA-ZZ-00500, Corrective Action Program, Section 3.1.1, specified that all
personnel are responsible for the immediate notification of the shift supervisor upon
discovery of a condition that they believe to have an immediate impact on nuclear,
plant, or personnel safety. Section 5.3.1.2 specified that if the condition discovered is
believed to have an immediate impact on operability, reportability, plant or personnel
17
safety, then the originator is to immediately notify the shift supervisor. The team
interviewed personnel assigned to the corrective action group and determined that
equipment deficiencies did not require the notification of the shift supervisor if the
individual believed that equipment operability was not immediately affected. The team
also interviewed the assistant operations superintendent and determined that his
expectations were for personnel to notify the shift supervisor of any equipment
deficiency. The team determined that the guidance and expectations regarding the
notification of the shift supervisor of equipment deficiencies were not consistently
implemented. In addition, the team determined that the opportunity for licensed
personnel to assess operability for degraded equipment conditions may not occur if the
originator (possibly non-licensed individual) believed that an operability concern did not
exist. The team determined that the failure to notify the shift supervisor of deficiencies
associated with plant equipment could have a credible impact on safety. For example,
the shift supervisor was not notified of the significant condition adverse to quality
described in Callaway Corrective Action System Item CARS 200200264, Foreign
Material Found in the AL System and Callaway Corrective Action System Item
CARS 200200485, Evaluate the Potential Effects of Dissolved Nitrogen on the Auxiliary
Feedwater System. The safety significance associated with foreign material in the
AFW system and CST is described in Section 7.
Procedure APA-ZZ-00500, Attachment 7, OPER Disposition, provided guidance for
completing operability evaluations. The guidance consisted of eight steps which
involved:
+ reviewing documentation to understand the issue, assuring the correct individual
was assigned to review the issue,
+ investigating the issue,
+ assigning actions to other individuals if necessary,
+ preparing a response,
+ entering correct information,
+ attaching supporting information, and
+ closing the OPER action.
The team determined that there were no references in Attachment 7 to Generic
Letter 91-18. Consequently, the licensees operability evaluation program did not
provide the following:
+ Guidance to aid the reviewer in determining when a safety evaluation was
required to be considered.
+ A requirement for a licensed operator to be notified when an operability
evaluation was being or had been performed.
18
+ Guidance on the amount of time required to complete an operability evaluation.
+ Guidance on the required level of review of an operability evaluation.
+ Guidance for the completion of safety evaluations associated with compensatory
measures.
+ Guidance for the periodic reviews of open operability evaluations.
The team determined that the failure to have an adequate program for processing
operability evaluations could have a credible impact on safety. For example, the
operability evaluations for Callaway Corrective Action System Item CARS 200200264,
Foreign Material Found in the AL System; Callaway Corrective Action System Item
CARS 200200485, Evaluate the Potential Effects of Dissolved Nitrogen on the Auxiliary
Feedwater System; and Callaway Corrective Action System Item CARS 200200489,
Complete an Operability Evaluation on the CST Due to the Existence of the Foam
Pieces did not provide an adequate basis for continued operability of the AFW system
and the CST. The safety significance associated with these operability evaluations is
described in Section 7.
Callaway Corrective Action System Item CARS 200200264, Foreign Material Found in
the AL System, dated January 15, 2002, involved the discovery of foam in the AFW
Pump A seal water cooling line. The shift supervisor was not notified of this significant
condition adverse to quality. Callaway Corrective Action System Item CARS 200200264
was assigned a Significance Level of 3 (no onsite review committee evaluation
required). The assigner (acting system engineering mechanical supervisor) assigned
the operability evaluation to himself. The operability evaluation had a required
completion date of February 16, 2002, and was completed on January 17, 2002. The
required completion date was well beyond the allowed outage time associated with the
AFW and CST Technical Specifications, which were 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> and 7 days, respectively.
No peer or supervisory review of the finished operability evaluation was completed.
Operations personnel were not provided the opportunity to review the completed
operability evaluation. The team determined that the operability evaluation did not
adequately assess the extent of condition. Specifically, no assessment of the source of
the foreign material or where additional foreign material could be located was
performed. This failure to adequately assess the presence of foreign material in the
AFW system is documented in Section 3.2 as an example of an apparent violation of
10 CFR Part 50, Appendix B, Criterion XVI.
Callaway Corrective Action System Item CARS 200200485, Evaluate the Potential
Effects of Dissolved Nitrogen on the Auxiliary Feedwater System, dated January 25,
2002, involved the identification by the licensee that the effect of dissolved nitrogen on
the AFW system had not been considered. The shift supervisor was not notified of this
significant condition adverse to quality. Callaway Corrective Action System Item
CARS 200200485 was assigned a Significance Level of 3 (no onsite review committee
evaluation required). The operability evaluation had a required completion date of
February 16, 2002, and was not completed before the plant shutdown on
January 31, 2002. The required completion date was well beyond the allowed outage
time associated with the AFW and CST Technical Specifications, which were 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />
and 7 days, respectively. The team determined that the response to the operability
19
evaluation was untimely in that the plant continued to operate beyond the allowed
outage time associated with the AFW system without an assessment of the potential
safety significance of the impact dissolved nitrogen had on the CST and AFW system.
The failure to account for the affect of nitrogen on available NPSH is documented in
Section 4.3 as a noncited violation of 10 CFR Part 50, Appendix B, Criterion III.
Callaway Corrective Action System Item CARS 200200489, Complete an Operability
Evaluation on the CST Due to the Existence of the Foam Pieces, dated January 27,
2002, involved the discovery of foam in the CST. The shift supervisor was notified of
the condition. Callaway Corrective Action System Item CARS 200200489 was assigned
a Significance Level of 2 (onsite review committee evaluation of the root cause
required). The operability evaluation had a required due date of January 27, 2002, and
was completed as required. The licensee determined that the AFW system was
operable because: (1) the material identified in the tank was fragile and not able to clog
an AFW pump orifice, (2) an insufficient flow velocity existed in the CST to cause the
identified material to enter the AFW system, (3) the isolation of secondary plant systems
returning to the CST (minimize recirculation of the CST contents), and (4) successful
completion of AFW pump testing. The team determined that inadequate information
provided by the licensees staff to senior management delayed the initiation of the extent
of condition review associated with the discovery of foreign material in the CST. Once
initiated, the extent of condition review was thorough and provided assurances that the
AFW system did not contain any foreign material. The failure to communicate
information regarding significant conditions adverse to quality is documented in
Section 3.2 as an example of an apparent violation of 10 CFR Part 50, Appendix B,
Criterion XVI.
Corrective Actions
The licensee planed to institute a series of programmatic and procedural enhancements
to improve weaknesses in the operability evaluation program. These enhancements
included:
+ Adding an operability evaluation form to the Callaway Corrective Action System
Item procedure.
+ Attaching completed operability evaluation forms to the associated Callaway
Corrective Action System Item.
+ Requiring operability evaluation forms to specify compensatory actions.
+ Requiring a review of operability evaluations.
+ Supplementing operability evaluation guidance with information from Generic
Letter 91-18.
+ Requiring shift supervisor approval for completed operability evaluations.
+ Tracking operability evaluations with their associated Callaway Corrective Action
System Item.
20
+ Adding criteria for timely completion of operability evaluations.
+ Implementing timely root cause evaluations for Significance Level 1 and 2
events.
5.0 Extent of Condition Review
a. Inspection Scope
The team reviewed documentation to determine to what extent the degraded CST
diaphragm seal may have had on other systems, structures, or components. The
review included visual inspections of the degraded seal and examination of the foreign
material removed from the CST and AFW Pump A. The teams review satisfied the
activities associated with the AIT Charter Element 4, Determine to what extent the
degraded CST floating diaphragm could potentially impact plant equipment. In addition
to the AFW system, this review should assess the potential impact on any other
components and systems, which may be affected.
b. Observations and Findings
The licensee had seven tanks with diaphragm seal devices. Six of those tanks used a
rubber seal assembly and the CST used the rigid diaphragm seal designed by
Conservaflote. The six tanks designed with rubber type diaphragm seals have had
periodic inspections performed in the past and were not expected to fail before signs of
degradation were identified by the licensee. The most risk significant tank (reactor
makeup water tank) had been inspected on a 5-year periodicity and the diaphragm was
replaced in 1998. The team determined that the same or similar condition did not exist
in other tanks at the facility.
The licensee performed laboratory analysis of the foam in order to determine the
potential effect on the steam generators. The analysis concluded that the foam was of a
polyurethane type where degradation (melting) commenced at 428-F and significant
degradation occurred at 513-F. As a result of the analysis and the temperature
environment in the steam generators at the time of the event, the licensee determined
that any foam entering the steam generators would have quickly degraded and become
diluted in the steam generator inventory. The team determined that there were no
apparent effects from the foam on the steam generator internals or instrumentation.
6.0 Quality Assurance Review
6.1 Review of the Quality Assurance Aspects of the Event
a. Inspection Scope
The team interviewed personnel to determine the extent of quality assurance (QA)
personnel involvement in the resolution of the failure of AFW Pump A. The teams
review satisfied the QA activities portion of Charter Element 5, Identify any human
21
factor, procedural or quality assurance deficiencies that may have contributed to the condition.
b. Observations and Findings
The team interviewed the QA manager and determined that QA did not participate in the
event response team meeting for the December 3, 2001, event. In addition, the team
determined that the licensee did not notify QA when event response team meetings
were conducted. Consequently, QA was not provided the opportunity to oversee the
process.
The team also determined that no one from QA was specifically assigned to oversee the
resolution of the root cause analysis and that QA personnel provided limited oversight of
the investigation into the failure of AFW Pump A. Specifically, QAs involvement was
limited to the following: (1) The Callaway Corrective Action System Item was reviewed
as part of the daily screening of corrective action program documents. (2) In early
December 2001, a print-out of the AFW flow control valve response and NRC
Bulletin 88-04, Potential Safety-Related Pump Loss, was provided to the corrective
action group facilitator assigned to assist in the development of the root cause (the
facilitator was not directly involved in the investigation of the failure of AFW Pump A until
mid-to-late January 2002). Bulletin 88-04 involved minimum flow line issues associated
with safety-related pumps and was not related to the failure of AFW Pump A. (3) A
review of the evaluation for returning AFW Pump A to an operable status was
completed. (4) Assessment activities associated with the entry into the CST were
completed.
6.2 Quality Assurance Audits of Foreign Material and Operating Experience
The licensee did not specifically perform audits of the foreign material and operating
experience programs. Instead, the licensee audited the foreign material and operating
experience aspects of whatever audits and assessments were completed. The licensee
provided copies of audits that contained the key words foreign material or operating
experience. The team did not identify any abnormal trends in the foreign material or
operating experience programs.
7.0 Risk Significance of Event
a. Inspection Scope
The team reviewed the licensees risk analysis associated with the failure of AFW
Pump A. The issue was determined to be more than minor because the on-demand
failure of AFW Pump A had an actual impact on plant safety. In addition, the degraded
condition of the diaphragm seal could have affected the availability of the AFW system.
The team completed an NRC Manual Chapter 0609, Significance Determination
Process, Phase 1 analysis of the failure of AFW Pump A. The SDP Phase 1 screening
process required that an SDP Phase 2 analysis be performed because the finding
represented an actual loss of a train of the AFW system. The team also completed an
SDP Phase 3 analysis in addition to the Phase 2 analysis to obtain a better
understanding of the increase in core damage frequency (CDF) stemming from the
identified performance issues. The CDF is the expected frequency of a core damage
22
event. The teams review satisfied the risk assessment activities associated with AIT
Charter Element 2, Review the licensees root and probable cause determination for
independence, completeness, and accuracy, including the licensees assessment of the
risk associated with the condition, and a portion of Charter Element 3, Assess the
timeliness and effectiveness of the licensees evaluation of potential AFW common
cause failure stemming from the degraded condensate storage tank floating
b. Observations and Findings
Potential Impact of Safety Functions
A failure of the AFW system had a significant impact on the decay heat removal safety
function. The AFW system was the third most risk important system for the Callaway
Plant.
Potential for Common Cause Failure of the AFW System
The team inspected the condition of the diaphragm seal once it had been removed from
the CST to determine the extent of the condition of the degraded seal. The team also
interviewed various members of the licensees staff to evaluate the effect the degraded
seal could have on systems supplied by the CST.
The team determined that the degradation of the seal was limited to a 71-inch
circumferential section of the diaphragm. The degraded portion of the diaphragm seal
was located above the single suction source of the AFW system (See Figure 4). The
following table provides a listing of the missing foam portions of the seal.
Degraded Diaphragm Seal Inventory
Amount (length/height/width) Location
2 inch x 3 inch x 8 inch CST Sump
11.5 inch x 3 inch x 8 inch Removed by diver to complete seal
repair
12 inch x 3 inch x 8 inch CST Sump
20.5 inch x 3 inch x 8 inch Total amount of unaccounted foam.
25 inch x 3 inch x 8 inch Hanging from diaphragm and
removed by diver.
The team determined that the amount of unaccounted foam could have been one or
more pieces totaling 20.5 x 3 x 8 inches of material. The foam could have entered the
suction of the AFW system as a single piece of foreign material or as multiple pieces of
foreign material. The team determined that the proximity of the degraded seal location
to the suction of the AFW system increased the likelihood that a piece of foam could
23
enter the AFW system. The team also determined that the length and configuration of
the foam could affect the operation of the AFW system following an event.
The licensees lower bound risk analysis of the degraded diaphragm seal assumed
there was no potential for common cause failure of the AFW system or multiple AFW
pumps because the AFW pump would have recovered without operator action. The
licensees upper bound risk analysis of the degraded diaphragm seal assumed that the
two pieces of foam located in the CST sump, plus one additional piece of foam, could
affect the AFW pumps for a period of 1.8 months. After 1.8 months, only one piece of
foam was assumed to exist. Specifically:
+ The licensee determined that the 25-inch long section of foam removed by the
diver could not have fallen into the CST before January 25, 2002. Specifically,
the section of removed foam was attached to a 6-foot section of foam and the
diver was unable to pull the foam away from the masonite board or tear the
foam. Additionally, the licensee believed that the clearances between the CST
wall and the floating diaphragm were sufficient to prevent the foam from being
pinched off.
+ The licensee determined that the two pieces of foam found in the sump would
not have entered the AFW suction piping once the specific gravity exceeded
1.03. The specific gravity of the foam attached to the diaphragm was
approximately 1.0. The specific gravity of the water saturated foam in the CST
sump was approximately 1.31. The depth of the CST sump was approximately 3
feet. The licensee determined that a linear velocity of approximately 1.5 feet per
second would be required to lift water saturated foam from the bottom of the
CST. The actual linear flow velocity across the CST sump was determined to be
less than 1 foot per second. The licensee chemically analyzed the foam in the
CST sump and determined that the material had undergone hydrolytic
decomposition, indicating that the foam had been in the sump for an extended
duration. The chemical analysis was supported by the recognition that biological
materials had permeated the entire volume of the foam.
+ The licensee evaluated the effect of a single 20.5 inch long piece of foam on the
AFW system. The licensee determined that the elasticity of the foam would have
prevented it from obstructing the suction piping at the CST vortex breaker or the
manual suction isolation valve. The licensee also determined that a single piece
of foam would not have torn into multiple pieces as the foam traveled from the
CST to the suction of an AFW pump.
The team determined that there was a wide variance in the possible effects foam could
have on AFW system performance. Consequently, the team determined that there was
an increased likelihood for common cause failure of the AFW system due to foam from
the degraded diaphragm seal. Specifically: (1) multiple pieces of foam could have
become dislodged from the diaphragm seal. The multiple pieces could have affected
one or more AFW pumps following a demand signal. (2) A single piece of foam could
have separated into multiple pieces as it traversed the AFW suction piping and affected
one or more AFW pumps.
24
Recovery of AFW Pump A
The licensee hypothesized that the foam was extruded through the pump impeller and
that the pump would have self-recovered within a few additional minutes had operations
personnel not secured the pump. The licensee indicated that the pump was recovering
and eventually would have delivered design flow for the following reasons:
+ AFW Pump A motor current was increasing during the event.
+ AFW Pump A pump was vented for approximately 15 seconds allowing gas to
escape. No steam was present during the venting.
+ AFW Pump A was started successfully following the pump vent.
The licensee determined that the pump motor amperage increase during the event
indicated that AFW Pump A was recovering. The team determined that the motor
current increase from 42 to 46 amps, during the 10 minutes the pump operated, did not
support a conclusion that the pump would have recovered without operator intervention.
The team noted that the current increase could have been the result of the rise in
temperature on the outboard stuffing box which may have increased the frictional forces
on the shaft rotor due to the contact with the dry stuffing box packing material.
Moreover, trend data indicated that there was no corresponding increase in pump
discharge pressure or flow.
The team also noted that as a result of operating the pump in a partially air-bound
condition, the pump experienced some minor hydraulic degradation, which required the
licensee to establish new baseline pump characteristics to satisfactorily complete the
required surveillance tests (following the event, AFW Pump A could not attain the
required minimum flow). This information further indicated to the team the significance
of the air-binding event and the potential that the pump may not have been able to
self-recover. In addition, had operations personnel not de-energized the pump, the
potential existed for additional degradation of AFW Pump A.
The team determined that the successful venting and starting of the pump indicated that
the potential existed for operations personnel to recover a failed AFW pump. Two
independent flow paths existed for supplying water to the outboard packing housing.
Even though the seal water cooling line was obstructed with foam, the normal leakage
along the pump shaft should have been sufficient to provide lubricating flow to the
outboard packing housing. The team determined that the licensees human error
probability calculation for recovery of an AFW pump was appropriate. The analysis
considered the available time to diagnose the condition and take actions to restore the
pump. The team also determined that operations personnel possessed the requisite
skills to successfully complete a pump vent. Therefore, the team determined that the
licensee had appropriately determined that the potential for successful recovery of a
failed AFW pump was 0.95.
25
Review of Licensees Risk Calculations
The licensees preliminary lower bound risk analysis assumed that; (1) only one pump
would be affected (no potential for common cause failure), (2) the affected pump would
self-recover, (3) the likelihood that a piece of foam large enough to affect the system
would enter the AFW suction piping was 0.5, and the likelihood that an operator would
fail the pump as part of a recovery action was 2.9E-3. Based on these assumptions, the
licensee determined that the increase in core damage frequency (CDF), excluding
external events and flooding, was approximately 8E-7/year for a 100 percent capacity
factor and 6.8E-7/year for an 85 percent capacity factor. The team determined that the
85 percent capacity factor represented the actual at-power operating history of the
facility. In addition, the team determined that for the 15 percent shutdown interval, the
licensee did not credit the use of the AFW system while the residual heat removal
system was operating with the plant in Modes 4, 5, and 6.
The licensees upper bound case assumed a potential for common cause failure existed
for the first 1.8 months and only a single piece of foam for the remaining 10.2 months.
The licensee based the 1.8 month interval on their determination that the two pieces of
foam found in the CST would have entered the sump within 1.8 months. In addition, the
licensee assumed that the human error probability for recovery of a failed AFW Pump
was approximately 0.05. The licensees upper bound result of the increase in CDF,
excluding external events and flooding, was 5.53E-6/year for a 100 percent capacity
factor and 4.7E-6/year for an 85 percent capacity factor.
The licensees average test and maintenance Probabilistic Safety Assessment (PSA)
model used a loss of offsite power (LOOP) initiating event frequency of 3.9E-2/year
(Electric Power Research Institute TR 110398). The licensees next PSA model update
planned to use a revised LOOP frequency of 2.2E-2/year. The revised LOOP frequency
used the values in EPRI 1000158 (2.8E-2) minus the contribution from coastal LOOP
events and shut-down LOOP events. In addition, the licensee completed a Bayesian
update to reflect plant specific values associated with a LOOP event. Specifically, the
licensee has not had a LOOP event during the previous 17 years of plant operation.
The Bayesian update further reduced the LOOP frequency to 2.2E-2/year. The new
LOOP frequency reduced the baseline average test and maintenance CDF from
2.45E-5/year to 1.59E-5/year.
NRC Phase 2 Risk Assessment
The team completed a preliminary significance determination using the SDP Phase 2
process described in NRC Manual Chapter 0609, Significance Determination Process.
Two analyses were completed. The first analysis assumed that AFW Pump A was
unavailable for 1 year without any credit for recovery actions. The second analysis
assumed that all three AFW pumps were unavailable for 1 year without credit for
recovery actions. Table 2, Initiators and System Dependency for Callaway Nuclear
Generating Station, Unit 1, of the Callaway Plant Phase 2 site specific notebook
required that all initiating event scenarios except a loss of service water and a large
break loss of coolant accident be evaluated for findings affecting a motor driven AFW
26
pump and that all initiating event scenarios except a large break loss of coolant accident
be evaluated for findings affecting the turbine driven AFW pump.
The team analyzed 28 sequences using the site specific notebook and determined that
the dominate initiating event scenarios for a failure of AFW Pump A were transients
without the power conversion system and loss of offsite power. This case produced a
preliminary significance determination of substantial (Yellow).
The team analyzed 45 sequences using the site specific notebook and determined that
the dominate initiating event scenarios for a failure of all AFW Pumps were transients
with and without the power conversion system, loss of offsite power, and loss of a vital
dc bus. This case produced a preliminary significance determination of high (Red).
The team determined that an SDP Phase 3 analysis should be performed because the
results of the Phase 2 analysis may have been overly conservative. Specifically,
mitigation credit was not applied for recovery of a failed AFW pump, all AFW pumps
were assumed to fail in order to account for a potential common cause failure mode,
and the site specific notebook had not accounted for changes in the licensees PSA
model.
NRC Phase 3 Risk Assessment
The team requested that the licensee use their PSA model to calculate the change in
CDF for several combinations of AFW pump failures due to common cause with various
recovery probabilities. The duration for all the combinations was 1 year because no
information was provided which suggested that the diaphragm seal was degraded for an
interval of less than 1 year. The first variable involved adjusting the likelihood that a
piece of foam from the diaphragm entered the suction of the AFW system to 0.25, 0.5,
and 1.0. The second variable involved adjusting the recovery probability of a failed
AFW pump to 0.05, 0.20, and 1.0. The third variable accounted for a change in the
licensees capacity factor from 100 to 85 percent. The team determined that the 85
percent capacity factor represented the actual at-power operating history of the facility.
The lower bound results assumed the duration was 1 year, a potential common cause
failure of the remaining two AFW pumps, the likelihood that a piece of foam from the
diaphragm entered the suction of the AFW system was 0.25, the recovery probability of
a failed AFW pump was 0.05, and the capacity factor was 85 percent. The increase in
CDF, excluding external events and flooding, was approximately 2.1E-6/year.
The upper bound results assumed the duration was 1 year, a potential common cause
failure of the remaining two AFW pumps, the likelihood that a piece of foam from the
diaphragm entered the suction of the AFW system was 1.0, the recovery probability of a
failed AFW pump was 0.2, and the capacity factor was 100 percent. The increase in
CDF, excluding external events and flooding, was approximately 1.1E-5/year.
The teams final determination of the safety significance of the degraded CST
diaphragm seal was based on the following assumptions:
27
(1) An 85 percent capacity factor. In addition, the team determined that for the 15
percent shutdown interval, the licensee did not credit the use of the AFW system
while the residual heat removal system was operating with the plant in shutdown
Modes 4, 5, and 6.
(2) The revised LOOP frequency of 2.2E-2/year.
(3) A potential for common cause failure of the remaining two AFW pumps. The use
of the nominal fail-to-run probability associated with the remaining AFW pumps
was considered inappropriate in that there was an increased likelihood that foam
could affect an AFW pump. Therefore, the team, increased the failure
probability of the remaining motor-driven pump from 4.2E-3 to 0.1 and the
turbine-driven pump from 2.4E-3 to 0.1.
(4) The duration of 1 year. No information was provided which suggested that the
diaphragm seal was degraded for an interval of less than 1 year.
(5) The likelihood of 0.50 that a piece of foam which separates from the diaphragm
enters the AFW system. Two pieces of foam totaling 14 inches in length were
identified in the CST sump. This provided information to suggest that some of
the foam material would not enter the AFW suction piping.
(6) The potential for successful recovery of a failed AFW pump of 0.95. The NRC
analysts determined that the licensees human error probability calculation for
recovery of an AFW pump was appropriate. The steam generator dryout time
for a loss of all feedwater was approximately 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. Adequate time existed to
diagnose the event and take actions to restore at least 1 AFW pump. The team
determined that operations personnel possessed the requisite skills and
knowledge to successfully complete a pump vent. In addition, there were no
environmental or human factor issues which could have affected the ability of
operations personnel to vent and fill an AFW pump.
Given the above assumptions, the team determined that the increase in CDF, excluding
external events and flooding, was approximately 4.3E-6/year. The initiating events with
the greatest contribution to core damage involved a loss of offsite power, loss of service
water, and loss of dc power. For the loss of offsite power initiating events, the dominant
sequences involved a failure to recover ac power within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />, a failure of both
emergency diesel generators, and a failure of the turbine driven AFW pump. For the
loss of service water initiating event, the dominant sequences involved a failure to
recover service water and a failure of the turbine driven AFW pump. For the loss of dc
power initiating event, the dominant sequences involved a failure of the turbine driven
AFW pump, the motor driven AFW pumps, charging pumps, and residual heat removal
pumps.
Consideration of External Events and Flooding
The licensee did not have a PSA model which integrated internal, external, and flooding
events. The contribution from seismic events was negligible because the licensee had
28
adequately resolved the vulnerabilities identified during the completion of their seismic
margins analysis. The CDF associated with fire events was approximately 8.9E-6/year.
The CDF associated with flooding events was approximately 6E-6/year.
The AFW system was the third most risk important system for the Callaway Plant
internal PSA model. The team reviewed the licensees Individual Examination of
External Events document and determined that the AFW system supported the decay
heat removal critical safety function in each of the licensees external event analyses.
The team noted that the percent increase in CDF for the internal events PSA model
(4.3E-6/1.59E-5) was approximately 27 percent. Because of the importance of the AFW
system for both the internal and external event models, the team qualitatively
determined that there was some modest increase in risk associated with the
consideration of external events. The team determined that the increase due to external
events would not likely be sufficient for the performance deficiency to be characterized
as having substantial safety significance (Yellow).
The team determined that the contribution from the failure of the AFW system had a
negligible impact on the large early release frequency. The team reviewed the dominate
large early release frequency sequences affected by failures of the AFW system and
determined that the increase was less than 1E-7/year. The dominate sequences
involved a steam generator tube rupture with a failure to isolate the affected steam
generator and a failure to establish either main feedwater or auxiliary feedwater flow.
Uncertainty
The mean values for the licensees original internal events PSA model, which included
flooding, was 1.95E-5. The 95th percentile of the PSA model was 4.16E-5 and the
5th percentile of the PSA model was 8.51E-6. Given the relatively narrow range between
the 5th and 95th percentiles, the team determined that it was appropriate to use the point
estimates derived from the PSA model quantifications without an additional adjustment
to account for uncertainty.
Conclusions
The team determined that the preliminary safety significance of the degradation of the
CST diaphragm seal was low to moderate (White). Specifically, the estimated increase
in CDF was greater than or equal to 4.3E-6/year and less than 1E-5/year, assuming an
85 percent capacity factor, the revised LOOP frequency, an increased likelihood that a
common cause failure could occur, a 1-year duration for the condition, the likelihood that
a piece of foam which separates from the diaphragm enters the AFW system at least
50 percent of the time, and the potential for the affected AFW pumps to be recovered at
least 95 percent of the time.
29
8.0 Overall Adequacy of the Licensees Response
a. Inspection Scope
The team assessed the observations and findings identified during the inspection in
order to complete an assessment of the overall adequacy of the effectiveness of the
licensees corrective actions in response to the failure of AFW Pump A. The teams
review satisfied AIT Charter Element 8, Review the overall adequacy of the licensees
response to the failure of the AFW pump.
b. Observations and Findings
The team determined that the licensee missed several opportunities to promptly identify
and correct a risk significant condition involving the degraded condition of the CST
diaphragm seal. The team also determined that the multiple failures of the licensee to
identify the degraded CST diaphragm seal was a significant human performance cross
cutting issue involving the recognition of degraded conditions.
Quality assurance personnel were not actively involved in providing oversight of the
event review team and root cause investigation process. The event review team
process did not ensure that statements were obtained from all personnel involved in the
event. The corrective action program did not include guidance or expectations on the
assignment of appropriate resources to review the highest classification of significant
conditions adverse to quality. Minimal resources were assigned to the root cause
investigation and may have contributed to the delay in identifying the degraded CST
diaphragm seal. Based on interviews with the licensees staff and a review of Procedure
APA-ZZ-00500, Corrective Action Program, the team determined that licensed
operators were only notified of equipment deficiencies if the individual discovering the
condition believed there was an immediate impact on nuclear, plant, or personnel safety.
Consequently, the potential existed for operability decisions to be made by non-licensed
personnel. The operability evaluation program did not implement the guidance provided
9.0 Exit Meeting Summary
On February 27, 2002, the team presented the inspection results to Mr. G. Randolph
and other members of his staff at a public exit meeting held at the Callaway Plant.
ATTACHMENT 1
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee
R. Affolter, Vice President Nuclear
J. Blosser, Manager, Regulatory Affairs
S. Bond, Supervisor, Design Engineering
K. Connelly, Engineer, Regulatory Affairs
J. Laux, Manager, Quality Assurance
J. McGraw, Superintendent, Nuclear Engineering
T. Moser, Superintendent, System Engineering
G. Randolph, Senior Vice President and Chief Nuclear Officer
M. Reidmeyer, Supervisor, Regional Regulatory Affairs
M. Taylor, Manager, Nuclear Engineering
M. Waltz, Engineer, Regulatory Affairs
R. Wink, System Engineer
W. Witt, Plant Manager
NRC
L. Ellershaw, Senior Reactor Inspector
M. Franovich, Reactor Analyst
I. Jung, Reactor Analyst
P. Wilson, Senior Reactor Analyst
Y. Huang, Mechanical Engineer
ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
05000483/0207-01 APV Failure to promptly identify and correct a significant
Opened and Closed
050-00483/0207-02 NCV Failure to verify calculational methods.
2
DOCUMENTS REVIEWED
The following documents were selected and reviewed by the inspectors to accomplish the
objectives and scope of the inspection and to support any findings:
Audit and Assessment Reports
Assessment Report AP01-005 and SA-QA-001, Access Adequacy and Implementation of the
Optimized Audit Process, July 2, 2001
Audit Report AP00-008, Fourth Quarter 2000 Quality Assurance Audit Report,
January 11, 2001
Audit Report AP01-001, First Period 2001 Quality Assurance Report, June 26, 2001
Audit Report AP01-002, Second Period 2001 Quality Assurance Audit Report,
October 2, 2001
Audit Report AP01-006, Quality Assurance Audit of the Training, Qualification, and
Performance of Nuclear Division Personnel, February 2, 2002
Audit Report AP98-017, Quality Assurance Audit of Callaway Cycle 10 Reload Design and
Safety Analysis, February 1, 2002
Surveillance Report SP98-001, Self Assessment of the FSAR Review Effort, Callaway Plant,
February 1, 2002
Surveillance Report SP98-016, A Safety Injection Outage, February 1, 2002
Surveillance Report SP99-023, Spent Fuel Pool (SFP) Re-rack Modification, CMP97-1016,
February 2, 2002
Surveillance Report SP99-013, Spent Fuel Pool Rerack Modification, February 2, 2002
Calculations
Calculation AL-10, Determine the Available NPSH for the Auxiliary Feedwater Pumps,
July 21, 1977
Calculation AL-13, A System Model of the AFW System, September 14, 1995
Calculation AL-24, Determine the Effect of Dissolved Nitrogen on Available NPSH for the
Auxiliary Feedwater Pumps, February 6, 2002
3
Corrective Action Program Documents
CARS 199901955, PM to Inspect CST Internal Cover Not Generated
CARS 200001900, B MDAFP Run Stopped Due to Lack of Discharge Flow Indication
CARS 200107423, Auxiliary Feedwater System Event Review
CARS 200200264, Foreign Material Found in AL System
CARS 200200485, Evaluate the Potential Effects of Dissolved Nitrogen on the Auxiliary
Feedwater System, January 25, 2002
CARS 200200489, Complete an Operability Evaluation on the CST Due to the Existence of
the Foam Pieces
CARS 200200669, Flow Rate on the B MDAFP Miniflow Line was Lower Than Expected
CATS 31040, Generate PM for Regular Inspection of CST Cover
Maintenance Documents
Generic Work Request G631231125, Periodic Inspections of CST Diaphragm Seal
Preventive Maintenance Item P663567, Inspect CST Cover
W201820, Inspect CST Floating Cover
W220254, Replace Rotating Element on A MDAFP
W682227, Suction Piping Inspection
W686916, Camera and Diver Inspection of CST
W687231, A MDAFP Discharge Piping Inspection
W687232, B MDAFP Discharge Piping Inspection
W687233, TDAFP Discharge Piping Inspection
W202393, CST Seal Inspection
Miscellaneous Documents
C. C. Chen, Utwin Engineers and Constructors, Inc., Chemical Engineering, Optimal System
Design Requires the Right Vapor Pressure. Heres How to Calculate It, pages 106 through
112, October 1983
4
Chemir/Polytech Laboratory Report, Job 39979, January 28, 2002
CST System Flow Diagram, M-01AP01, Revision E
CST Inspection Document, System Release Exception Form Item Number AP-015,
July 15, 1982
Conservatek Design Drawing, 43-D2, Revision 1
CST Design Specification No. 10466-M-109(Q)
Daniel W. Wood, et. al., Proceedings of the 15th International Pump Users Symposium,
Application Guidelines for Pumping Liquids that have a Large Dissolved Gas Content, pages
91 through 98
Dominion Engineering, Auxiliary Feed Pump Event, January 25, 2002
EPRI TR-114612-V2, Pump Troubleshooting, April 2000
INPO SOER 97-01, Potential Loss of High Pressure Injection and Charging Capability From
Gas Intrusion
Mao J. Tsai, Techon International, Inc., Chemical Engineering, Accounting for Dissolved
Gases in Pump Design, July 26, 1982
Report UOTH 01-0047, Event Review Meeting Minutes: High Main Turbine Vibration, AFAS,
and No Flow on A-AFW Pump, December 4, 2001
Technical Manual M-021000061, Instruction Manual for Ingersoll Rand Centrifugal Pumps,
Revision 25
UOTE 92-048, Callaway Response to NRC Generic Letter 91-82, February 10, 1992
Watson Tomlinson, Evaluation of Auxiliary Feedwater Pump 1A Event 12/3/01,
January 21, 2002
Modification Documents
Request For Resolution 21798, Revision D, Evaluate Permanent Removal of TAP01 Floating
Cover, February 1, 2002
Request For Resolution 04991, Permanent N2 for the Condensate Storage Tank, Revision A,
May 4, 1988
Restricted Modification Package 88-2016, Revision A, November 20, 1992
Procedures
Procedure APA-ZZ-00500, Corrective Action Program, Revision 31
5
Procedure OSP-AL-P001A, Motor Driven Aux. Feedwater Pump A In Service Test,
Revision 27
Procedure OSP-AL-00001, AFW Flow Paths Valve Alignment, Revision 5
Procedure OSP-AL-V001A, Train A Auxiliary Feedwater Valve Operability, Revision 25
Procedure OSP AL-00002, AFW to Steam Generators Flow Path Verification, Revision 3
Procedure OSP-AL-V0002, Auxiliary Feedwater Valve Operability Test, Revision 12
Procedure OSP-AL-V0003, Auxiliary Feedwater Pump Discharge Check Valve Closure Test,
Revision 2
Procedure QPC-ZZ-05046, Ultrasonic Examination Procedure for Determining Liquid Level in
Pipes and Components, Revision 0
Procedure OSP-ZZ-00001, Technical Specification Logs, Revision 37, December 2-5, 2001
Procedure OTG-ZZ-00004, Power Operation, Revision 35
Procedure OTN-AE-00001, Feedwater System Operation, Revision 23
Procedure OTN-AL-00001, Auxiliary Feedwater System Operation, Revision 7
LIST OF ACRONYMS USED
AIT Augmented Inspection Team
CARS Callaway Action Request System
CATS Callaway Action Tracking System
CCDP Conditional Core Damage Probability
CDF Core Damage Frequency
CFR Code of Federal Regulations
CST Condensate Storage Tank
IN Information Notice
LOOP Loss of Offsite Power
NPSH Net Positive Suction Head
PRA Probabilistic Risk Assessment
PSA Probabilistic Safety Assessment
QA Quality Assurance
SDP Significance Determination Process
SIT Special Inspection Team
ATTACHMENT 2
AUGMENTED INSPECTION TEAM CHARTER
January 31, 2002
MEMORANDUM TO: Troy Pruett
Senior Reactor Analyst
FROM: Ellis W. Merschoff
Regional Administrator /RA/ BY TPGwynn
SUBJECT: CHARTER FOR THE AUGMENTED INSPECTION TEAM AT THE
CALLAWAY PLANT
In response to recently developed information stemming from the continuing evaluation of the
impact of the Train A motor-driven auxiliary feedwater (AFW) pump failing to deliver required
flow to the steam generators during a controlled reactor plant shutdown performed on
December 3, 2001, the ongoing special inspection at the Callaway Plant is being upgraded to
an augmented inspection team (AIT). You are hereby designated as the AIT leader.
A. Basis
On December 3, 2001, the licensee manually started the motor-driven AFW pumps
prior to breaking condenser vacuum in anticipation of losing the operating main
feedwater pump during a controlled plant shutdown. After the main feedwater pump
tripped on low vacuum, as expected, the operators noticed the Train A motor-driven
AFW pump was not delivering the required flow to the steam generators. The licensee
manually stopped the A Train AFW pump and started the turbine-driven AFW pump to
provide necessary cooling flow to the steam generators. The initial risk assessment for
this condition indicated an estimated conditional core damage probability (CCDP) of
1.1E-6. As a result of this risk assessment, a Special Inspection Team was initiated in
accordance with NRC Management Directive 8.3, "NRC Incident Investigation Program."
The special inspection started onsite inspection activities at the Callaway Plant on
January 28, 2002.
On January 27, 2002, the licensee's investigation revealed that polyurethane foam from
a degraded condensate storage tank (CST) floating diaphragm may have caused the
pump failure. The revised risk assessment, which takes into account the potential
common cause impact on the AFW pumps, indicates a preliminary estimated CCDP in
the range of about 5E-5 to 5E-4. Management Directive 8.3 requires the consideration
of the initiation of an AIT when the estimated CCDP is greater than or equal to 1E-5.
On the basis of a potential for a substantial increase in risk stemming from common
mode failure implications, the ongoing special inspection is being upgraded to an AIT,
consistent with the guidance in Management Directive 8.3.
2
An AIT will be dispatched to better understand the cause of the AFW pump failure, the
extent of impact on the remaining AFW pumps, and operator actions leading up to and
including the event. The team is also tasked with gaining a better understanding of the
licensees common mode failure analysis as related to their root cause(s). The team
should build on the work already accomplished by the Special Inspection Team.
B. Scope
Specifically, the team is expected to perform data gathering and fact-finding in order to
address the following:
1. Develop a complete description and sequence of events related to the subject
AFW pump failure (including the degraded CST floating diaphragm), and
operator actions taken in response to regain feedwater flow.
2. Review the licensee's root and probable cause determination for independence,
completeness, and accuracy, including the licensee's assessment of the risk
associated with the condition.
3. Assess the timeliness and effectiveness of the licensee's evaluation of potential
AFW pump common cause failure stemming from the degraded condensate
storage tank floating diaphragm.
4. Determine to what extent the degraded CST floating diaphragm could potentially
impact plant equipment. In addition to the AFW system, this review should
assess the potential impact on any other components and systems, which may
be affected.
5. Identify any human factor, procedural or quality assurance deficiencies that may
have contributed to the condition.
6. Identify and assess the licensee's evaluation of applicable industry operating
experience.
7. Identify and assess the licensees prompt and long-term corrective actions to
address the root and probable causes of the condition.
8. Review the overall adequacy of the licensees response to the failure of the AFW
pump.
C. Guidance
This memorandum designates you as the AIT leader. Your duties will be as described
in Inspection Procedure 93800, "Augmented Inspection Team." The team composition
has been discussed with you directly. During performance of the augmented inspection,
designated team members are separated from their normal duties and report directly to
you. The team is to emphasize fact-finding in its review of the circumstances
3
surrounding the event, and it is not the responsibility of the team to examine the
regulatory process. Safety concerns identified that are not directly related to the event
should be reported to the Region IV office for appropriate action.
The team will report to the site, conduct an entrance meeting, and begin inspection on
Thursday, January 31, 2002. Tentatively, the inspection should be completed by close
of business February 2, 2002, with a report documenting the results of the inspection,
including findings and conclusions, issued within 30 days of the public exit meeting.
While the team is on site, you will provide daily status briefings to Region IV
management.
This Charter may be modified should the team develop significant new information that
warrants review. Should you have any questions concerning this Charter, contact Art
Howell III, Director, Division of Reactor Safety at (817) 860-8180.
ATTACHMENT 3
SEQUENCE OF EVENTS
Date and Time Event
July 1982 Plant personnel performed the final CST closeout inspection.
June 20, 1983 Startup Field Report AP-004A initiated to address oxygen
specifications for water from the CST.
July 11, 1983 SWR AP015 installed flanged connections to the CST lower manway
and added a suction and discharge line inside the tank. The
discharge line extended across the tank. CST drawings were not
updated.
January 2, 1985 Temporary Modification 85-M-001 documented CST manway
connections, but the internal tank piping configuration was not
updated.
June 6, 1985 Drawing M-22AP01 was revised to show manway connections;
however, internal tank pipe configuration was not recognized.
September 16, 1991 A modification is requested to sparge nitrogen. The impact of
sparging nitrogen into the CST was not evaluated.
December 18, 1991 NRC issues Information Notice 91-82, Problems With Diaphragms in
Safety-Related Tanks.
January 21, 1992 Corrective Action Tracking System CATS 31040 was generated to
develop a periodic inspection activity of the CST.
February 10, 1992 The licensee issued a response to Information Notice 91-82
(UOTE 92-048) stating that action was initiated to inspect the tank
internals for degradation.
February 1, 1999 Particulate matter was found in CST sampling equipment. Work
Order 197670 was initiated to clean and inspect the inside of the CST
during Refueling Outage 10.
September 15, 1999 Upon questioning by the NRC resident inspector, the licensee
reviewed its records and determined CATS 31040 was closed without
the task being accomplished. CARS 199901955 was initiated.
September 16, 1999 Generic Work Request G631231125 was initiated to inspect the CST
floating diaphragm.
2
Date and Time Event
December 1, 1999 Work Order W201820 replaces Generic Work Request G631231125.
CST floating diaphragm inspection was planned for the Spring
of 2000.
January 27, 2000 Work Order W202393 supercedes Work Order W197670.
October 17, 2000 The licensee performed an limited scope visual inspection of the CST
floating diaphragm and found no degradation.
November 14, 2000 The licensee established a preventive maintenance program to
inspect the CST floating diaphragm every 10 years.
March 31, 2001 Work Order W202393 is deferred from Refueling Outage 10.
December 3, 2001 Operations personnel commenced a reactor shutdown to Mode 3 in
at 1:15 p.m. order to repair a leaking main generator bushing.
December 3, 2001 Operations personnel received high vibration alarms on main turbine
at 10:39 p.m. Bearing Number 4. Vibration levels are recorded as 8.72 mils and
increasing.
December 3, 2001 Operations personnel manually trip the main turbine in response to
at 10:48 p.m. high vibration.
December 3, 2001 Operations personnel manually started AFW Pump B in anticipation of
at 10:56 p.m. breaking main condenser vacuum.
December 3, 2001 Operations personnel manually started AFW Pump A.
at 10:56 p.m.
December 3, 2001 Operations personnel broke main condenser vacuum in response to
at 10:57 p.m. main turbine bearing vibration levels exceeding 15 mils.
December 3, 2001 AFW Pump A ceased to provide flow to Steam Generators B and C.
at 10:58 p.m. Operations personnel started the turbine-driven AFW pump and
dispatched the field supervisor and an equipment operator to AFW
Pump Room A.
December 3, 2001 The field supervisor and an equipment operator observed no leak-off
at 11:01 p.m. flow from the outboard packing gland on AFW Pump A. The field
supervisor left AFW Pump Room A and inspected the turbine-driven
AFW pump.
3
Date and Time Event
December 3, 2001 The field supervisor returned to AFW Pump Room A. The outboard
at 11:05 p.m. packing gland was hot to the touch.
December 3, 2001 The field supervisor recommended that AFW Pump A be secured.
at 11:07 p.m. Control room personnel secured AFW Pump A.
December 3, 2001 The reactor entered Mode 3.
at 11:19 p.m.
December 3, 2001 Operations personnel vented AFW Pump A and observed 15 seconds
at 11:47 p.m. of air (not steam).
December 4, 2001 The licensee convened an event review team to investigate AFW
at 1:30 a.m. Pump A failure.
December 4, 2001 The licensee initiated CARS 200107423, assigned a root-cause
analyst to the event, and classified the event as Significance Level 1.
December 4, 2001 The licensee verified proper system alignment for AFW Pump A.
December 4, 2001 The licensee evaluated Significant Operating Experience
Report 97-01 for potential gas binding of AFW Pump A.
December 4, 2001 The licensee reviewed past operating and maintenance activities for
AFW Pump A.
December 4, 2001 AFW Pump A was started with operations and engineering personnel
present. Normal packing leak-off was observed, but the outboard
packing gland reached a temperature of 190oF after 4 minutes of
operation. The pump was secured, the outboard packing was
replaced, and the licensee re-baselined the inservice testing
performance criteria.
December 5, 2001 AFW Pump A was declared operable. The formal root cause team
was developed with personnel from system engineering and the
corrective action group.
December 6, 2001 The licensee formally initiated the root cause investigation for the
failure of AFW Pump A.
4
Date and Time Event
December 14, 2001 The licensee held a teleconference with the pump vendor to discuss
the pump malfunction. The vendor suggested that an inspection of
the seal water cooling line to the inboard and outboard packing
housing be completed.
January 8, 2002 The licensee conducted a meeting to discuss AFW performance
issues. A member from regulatory affairs, engineering, and two
contractors were added to the root cause team.
January 9, 2002 The licensee narrowed the cause of the event to three possibilities:
(1) low NPSH (2) nitrogen gas disassociation, and (3) foreign material.
Foreign material was not considered a credible failure mechanism.
January 11, 2002 The licensee completed ultrasonic testing of the AFW suction lines.
No significant air or gas accumulation was identified.
January 15, 2002 The licensee performed a surveillance test of AFW Pump A while the
Dominion Engineering and Flowserve representatives were on site.
The licensee found foam material in the orifice of the AFW Pump A
seal water cooling line.
January 17, 2002 The licensee initiated daily ultrasonic testing of the AFW suction lines.
January 23, 2002 The licensee concluded that inadequate NPSH and nitrogen
disassociation were not the causes of air-binding in AFW Pump A.
January 24, 2002 The licensee initiated an inspection of the CST floating diaphragm.
The team size associated with the root cause was expanded to
accommodate the CST inspection.
January 26, 2002 The licensee retrieved approximately 14 inches of foam from the
diaphragm seal from the CST sump. The diver also removed
approximately 25 inches of foam that was hanging from a 6 foot
damaged section of the diaphragm seal. Approximately 20.5 inches
of foam from the 71 inch damaged section was missing.
January 28, 2002 The NRC special inspection team arrived on site.
January 30, 2002 The licensee inspected the small bore piping, first and last stage
impellers, and part of the suction piping for AFW Pump B.
5
Date and Time Event
January 31, 2002 The licensee performed a reactor shutdown to Mode 4 to inspect the
AFW system. The NRC upgraded to an augmented inspection team
(AIT). The licensee increased the scope of the root cause
investigation.
February 1, 2002 The licensee began draining the CST. The licensee initiated foreign
material inspections for the AFW system.
February 2, 2002 The licensee inspected the CST floating diaphragm. The licensee
commenced removal of the CST floating diaphragm and replacement
of the rotating assembly on AFW Pump A.
February 5, 2002 Inspections on the AFW system were completed.
February 6, 2002 The CST is restored. AIT completes onsite inspection activity.
ATTACHMENT 4
SYSTEM FIGURES
Figure 1
Auxiliary Feedwater System
Simplified Diagram
Figure 2
Condensate Storage Tank
Diaphragm Seal
Figure 3
Degraded Diaphragm Seal
Figure 4
Condensate Storage Tank Configuration