ML050350250
| ML050350250 | |
| Person / Time | |
|---|---|
| Site: | Hope Creek  |
| Issue date: | 02/04/2005 |
| From: | Lanning W Division of Reactor Safety I |
| To: | Levis W Public Service Enterprise Group |
| References | |
| EA-05-001 IR-04-013 | |
| Download: ML050350250 (62) | |
See also: IR 05000354/2004013
Text
February 4, 2005
Mr. William Levis
Senior Vice President and Chief Nuclear Officer
P. O. Box 236
Hancocks Bridge, NJ 08038
SUBJECT: HOPE CREEK NUCLEAR GENERATING STATION - NRC SPECIAL
INSPECTION TEAM REPORT NO. 05000354/2004013 AND PRELIMINARY
WHITE FINDING
Dear Mr. Levis:
On December 16, 2004, the US Nuclear Regulatory Commission (NRC) completed a Special
Inspection at the Hope Creek Nuclear Power Plant. The enclosed report documents the
inspection findings which were discussed with members of your staff during a public exit
meeting on January 12, 2005.
This inspection examined activities conducted under your license as they relate to safety and
compliance with the Commissions rules and regulations and with the conditions of your license.
The team reviewed selected procedures and records, observed activities, and interviewed
personnel. In particular, the inspection reviewed your investigation, root cause evaluations,
relevant performance history, extent of condition, potential common cause failures, and
corrective actions associated with the failure of an 8-inch diameter moisture separator drain
tank line on October 10, 2004. The team also independently evaluated the equipment and
human performance issues that complicated the event response, and evaluated the associated
radiological release.
The team concluded that PSEGs overall response to the October 10, 2004, event was
adequate, although the operators were challenged by equipment problems. The team
concluded that none of the problems would have prevented the systems from performing their
intended safety functions.
This report documents one finding that appears to have low to moderate safety significance.
As described in Section 3.3 of this report, this finding involved inadequate evaluation and
corrective action for a degraded level control valve for the A moisture separator drain tank.
The valve malfunctioned several weeks prior to the event and caused the moisture separator
drain system to operate outside its design. This condition resulted in a pipe failure in a moisture
separator drain line on October 10, 2004. In addition to the inadequate evaluation of the level
control valve malfunction weeks before the event, engineers also did not properly consider a
similar occurrence from 1988.
Mr. William Levis 2
This finding was assessed using the reactor safety Significance Determination Process (SDP)
as a potentially safety significant finding that was preliminarily determined to be White (i.e., a
finding with some increased importance to safety, which may require additional NRC
inspection). The finding appears to have low to moderate safety significance because the
condition of the level control valve increased the potential for a plant transient that included the
loss of the normal power conversion system (the main condenser).
We believe that we have sufficient information to make our final risk determination for the
performance issue regarding the inadequate evaluation and corrective action for the level
control valve that malfunctioned. However, before the NRC makes a final decision on this
matter, we are providing you an opportunity to either submit a written response, or to request a
Regulatory Conference where you would be able to provide your perspectives on the
significance of the finding and the bases for your position. If you choose to request a
Regulatory Conference, it should be held within 30 days of the receipt of this letter, and we
encourage you to submit your evaluation and any differences with the NRC evaluation at least
one week prior to the conference in an effort to make the conference more efficient and
effective. If a Regulatory Conference is held, it will be open for public observation. The NRC
will also issue a press release to announce the Regulatory Conference. If you decide to submit
only a written response, such submittal should be sent to the NRC within 30 days of the receipt
of this letter.
Please contact Mr. Raymond Lorson at (610) 337-5282 within 10 days of the date of this letter
to notify the NRC of your intentions. If we have not heard from you within 10 days, we will
continue with our significance determination and enforcement decision and you will be advised
by separate correspondence of the results of our deliberations on this matter.
Additionally, based on the results of this inspection, the team identified three findings of very
low safety significance (Green). Two of these issues were determined to involve violations of
NRC requirements. However, because of their very low safety significance, and because they
have been entered into your corrective action program, the NRC is treating these issues as
non-cited violations, in accordance with Section VI.A.1 of the NRCs Enforcement Policy. If you
deny the non-cited violations noted in this report, you should provide a response with the basis
for your denial, within 30 days of the date of this inspection report, to the Nuclear Regulatory
Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001; with copies to the
Regional Administrator, Region I; the Director, Office of Enforcement, United States Nuclear
Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at
the Hope Creek facility.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its
enclosures will be available electronically for public inspection in the NRC Public Document
Room or from the Publicly Available Records (PARS) component of NRCs document system
(ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/
adams.html (the Public Electronic Reading Room).
Mr. William Levis 3
If you have any questions, please contact Mr. Raymond K. Lorson at (610) 337-5282.
Sincerely,
/RA/
Wayne D. Lanning, Director
Division of Reactor Safety
Docket No. 50-354
License No. NPF-57
Enclosure: Inspection Report 05000354/2004013
w/Attachments
cc w/encl:
G. Barnes, Site Vice President
M. Brothers, Vice President - Nuclear Assessment
M. Gallagher, Vice President - Engineering and Technical Support
W. F. Sperry, Director - Business Support
C. Perino, Director - Regulatory Assurance
M. Massaro, Hope Creek Plant Manager
R. Kankus, Joint Owner Affairs
J. J. Keenan, Esquire
M. Wetterhahn, Esquire
Consumer Advocate, Office of Consumer Advocate
F. Pompper, Chief of Police and Emergency Management Coordinator
J. Lipoti, Ph.D., Assistant Director of Radiation Programs, State of New Jersey
K. Tosch - Chief, Bureau of Nuclear Engineering, NJ Dept. of Environmental Protection
H. Otto, Ph.D., DNREC Division of Water Resources, State of Delaware
N. Cohen, Coordinator - Unplug Salem Campaign
W. Costanzo, Technical Advisor - Jersey Shore Nuclear Watch
E. Zobian, Coordinator - Jersey Shore Anti Nuclear Alliance
Mr. William Levis 4
Distribution w/encl:
S. Collins, RA
J. Wiggins, DRA
E. Cobey, DRP
S. Barber, DRP
M. Gray - DRP, Senior Resident Inspector
K. Venuto, DRP, Resident OA
S. Lee, RI OEDO
R. Laufer, NRR
Region I Docket Room (with concurrences)
K. Farrar, ORA
D. Holody, EO, RI
R. Urban, ORA, RI
F. Congel, OE (OEMAIL)
S. Figueroa, OE (SLF)
M. Elwood, OGC
D. Corlew, ORA
W. Lanning, DRS
J. Lubinski, DRS
R. Lorson, DRS
S. Pindale, DRS
SISP Review Complete: WDL
DOCUMENT NAME: E:\Filenet\ML050350250.wpd
After declaring this document An Official Agency Record it will be released to the Public.
To receive a copy of this document, indicate in the box: "C" = Copy without attachment/enclosure "E" = Copy with
attachment/enclosure "N" = No copy
OFFICE RI/DRS RI/DRS RI/DRS RI/DRP
NAME SPindale/SMP RLorson/RKL WSchmidt/WLS ECobeyEWC
DATE 1/31/05 2/4/05 1/31/05 1/31/05
OFFICE RI/ORA RI/DRP RI/DRS
NAME DHolody/RJU ABlough/ARB WLanning/WDL
DATE 1/31/05 1/31/05 2/4/05
OFFICIAL RECORD COPY
U.S. NUCLEAR REGULATORY COMMISSION
REGION I
Docket No. 50-354
License No. NPF-57
Report No. 05000354/2004013
Facility: Hope Creek Nuclear Power Plant
Location: P.O. Box 236
Hancocks Bridge, NJ 08038
Dates: October 14, 2004 - December 16, 2004
Inspectors: S. Pindale, Senior Reactor Inspector (Team Leader)
S. Dennis, Senior Operations Engineer
T. Burns, Reactor Inspector
E. Knutson, Resident Inspector
J. Wiebe, Reactor Inspector
R. Davis, Materials Engineer
J. Furia, Senior Health Physicist
N. McNamara, Emergency Preparedness Specialist
W. Schmidt, Senior Reactor Analyst
Approved by: Wayne D. Lanning, Director
Division of Reactor Safety
Enclosure
TABLE OF CONTENTS
SUMMARY OF FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
1.0 EVENT DESCRIPTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Event Summary: October 10, 2004, Manual Reactor Scram . . . . . . . . . . . . . . 1
2.0 PLANT RESPONSE: PERSONNEL AND EQUIPMENT . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 Operator Performance and Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1. General - Transient Response and Procedure Adherence . . . . . . . . . . 4
2. Reactor Vessel Level and Pressure Control . . . . . . . . . . . . . . . . . . . . . . 4
3. Training and Simulator Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4. Technical Specification Misinterpretation . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Equipment Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1. HPCI Full Flow Test Valve - Failure to Open on Demand . . . . . . . . . . 10
2. RCIC System Flow Oscillations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3. HPCI System Barometric Condenser Vacuum Pump Trip . . . . . . . . . . 14
4. Reactor Water Cleanup System Pump Trip . . . . . . . . . . . . . . . . . . . . 15
5. Low Reactor Vessel Level Instrument Actuations/Logic . . . . . . . . . . . . 16
2.3 Radiological Release Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.0 MOISTURE SEPARATOR/DRAIN TANK PIPING SYSTEM . . . . . . . . . . . . . . . . . . . . 18
3.1 Design, Operation, and Pipe Failure Details . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2 Historical Chronology - Design and Operational Details and Challenges . . . . 18
3.3 Failure to Evaluate and Correct Degraded Condition . . . . . . . . . . . . . . . . . . . 21
4.0 RISK SIGNIFICANCE OF THE OCTOBER 10, 2004 EVENT . . . . . . . . . . . . . . . . . . . 24
5.0 EVENT ROOT CAUSES AND CAUSAL FACTORS . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.0 EXTENT OF CONDITION AND CORRECTIVE ACTIONS . . . . . . . . . . . . . . . . . . . . . 27
6.1 Extent of Condition Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.2 Corrective Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
7.0 GENERIC ISSUES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
8.0 CROSS-CUTTING ASPECTS OF FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
9.0 EXIT MEETING SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
ATTACHMENT A: SUPPLEMENTAL INFORMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
ATTACHMENT B: SPECIAL INSPECTION TEAM CHARTER . . . . . . . . . . . . . . . . . . . . . . . B-1
ATTACHMENT C: SEQUENCE OF EVENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1
ATTACHMENT D: MOISTURE SEPARATOR/DRAIN SYSTEM CHRONOLOGY . . . . . . . . D-1
ATTACHMENT E: REACTOR VESSEL LEVEL INSTRUMENT DEFINITIONS/RANGES . . E-1
ii Enclosure
SUMMARY OF FINDINGS
IR 05000354/2004013, 10/14/04 -10/22/04, 11/29/04 -12/03/04, 12/15/04 -12/16/04;
Hope Creek Nuclear Power Plant; Special Inspection Team.
This inspection was conducted by full-time and part-time team members, including six regional
inspectors, a resident inspector, a regional senior reactor analyst, and an engineer from the
Office of Nuclear Reactor Regulation. One finding, assessed as a Preliminary White, and three
other Green findings were identified. The significance of most findings is indicated by their
color (Green, White, Yellow, Red) using IMC 0609, Significance Determination Process
(SDP). Findings for which the SDP does not apply may be Green or be assigned a severity
level after NRC management review. The NRCs program for overseeing the safe operation of
commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process,
Revision 3 dated July 2000.
A. NRC Identified and Self-Revealing Findings
Cornerstone: Initiating Events
C Preliminary White. A finding of low to moderate safety significance was
identified where engineering staff did not properly evaluate and correct a
degraded level control valve for the A moisture separator drain tank, as required
by station procedures. In addition, engineers did not properly consider a similar
occurrence from 1988. The level control valve failed 25 days prior to the event
and caused the moisture separator drain system to operate in a condition outside
its design. As a result, an 8-inch pipe in that system failed and caused the event
on October 10, 2004.
This issue is greater than minor because it is associated with the Equipment
Performance attribute of the Initiating Events cornerstone and affected the
objective of limiting the likelihood of events that upset plant stability and
challenge critical safety functions. A Significance Determination Process Phase
2 risk analysis determined that this finding had low to moderate safety
significance based on the increased frequency of a transient with the loss of the
power conversion system initiating event over the 25-day exposure period.
(Section 3.3)
Cornerstone: Mitigating Systems
C Green. A finding was identified as a result of the October 10, 2004, event, and
was a non-cited violation of 10 CFR Part 50, Appendix B, Criterion V
(Instructions, Procedures, and Drawings). Technicians did not comply with a
procedure to properly set a limit switch on a high pressure coolant injection
(HPCI) system injection valve, which is interlocked with the HPCI full flow test
valve. As a result, the full flow test valve did not open as required on initial
demand when control room operators attempted to place the HPCI system in the
pressure control mode of operation. Operators were subsequently successful in
opening the valve about five minutes later after additional actions were taken.
This finding is greater than minor because it is associated with the Equipment
Performance attribute of the Mitigating Systems cornerstone and affects the
iii Enclosure
cornerstones objective to maintain mitigation equipment reliable. This finding is
of very low safety significance because the finding did not represent the actual
loss of the safety function for the HPCI system. Also, reactor pressure remained
relatively stable when the issue occurred and alternate pressure control methods
were available (safety relief valves) if required. (Section 2.2.1)
C Green. A finding was identified as a result of the October 10, 2004, event, and
was a non-cited violation of 10 CFR Part 50, Appendix B, Criterion V
(Instructions, Procedures, and Drawings) in that procedures for operating the
reactor core isolation cooling (RCIC) system at low flow conditions were
inadequate. As a result, while operating the RCIC system during a plant
transient, the system exhibited unexpected flow oscillations in the automatic
mode when control room operators ran the system at low flow conditions.
This finding is greater than minor because it is associated with the Procedure
Quality attribute of the Mitigating Systems cornerstone and affects the
cornerstones objective of ensuring the capability of systems that respond to
initiating events to prevent undesirable consequences. This finding is of very low
safety significance because the finding did not result in the actual loss of the
safety function for the RCIC system. Also, reactor vessel level was maintained
in the appropriate range in accordance with procedures and the HPCI system
was available for reactor vessel level makeup if required. (Section 2.2.2)
C Green. A finding was identified as a result of the October 10, 2004, event in that
PSEG did not effectively implement preventive maintenance for the HPCI system
barometric condenser vacuum pump. As a result, with the HPCI system
operating in the pressure control mode, the vacuum pump tripped twice due to
improper lubrication of the vacuum pump shaft. Due to the vacuum pump
failure, operators removed the HPCI system from service and continued a vessel
cooldown with alternate safety related equipment (safety relief valves).
The finding is greater than minor because it is associated with the Equipment
Performance attribute of the Mitigating Systems cornerstone and affects the
cornerstones objective of ensuring the reliability of systems that respond to
initiating events. The finding is of very low safety significance because per
design basis, with the vacuum pump not available, the HPCI system remained
operable and was able to perform its mitigation function if required.
(Section 2.2.3)
iv Enclosure
REPORT DETAILS
1.0 EVENT DESCRIPTION
1.1 Event Summary: October 10, 2004, Manual Reactor Scram1
On October 10, 2004, at approximately 6:00 p.m., control room operators lowered
reactor power from 100% in response to a substantial increase in main condenser
offgas system flow and a coincident turbine building ventilation exhaust radiation monitor
alarm due to a reported steam leak in the turbine building. Initially, condenser vacuum
was able to be maintained by the steam jet air ejectors (SJAE). While not immediately
known to the operators at that time, a pipe failure had occurred in the drain line from the
A moisture separator (MS), which discharges to the main condenser. At 6:14 p.m.,
because offgas flow continued to increase and steam was noted in the condenser bay
area of the turbine building, control room operators initiated a manual reactor shutdown
(scram) and main turbine trip to reduce the potential consequences of the steam leak.
Following the scram and turbine trip, main condenser vacuum began to decrease
rapidly. In anticipation of the consequential loss of 1) the normal heat sink (main
condenser), and 2) normal reactor vessel level control with the feedwater pumps
(steam/turbine driven), the operators manually opened the turbine bypass valves (while
the condenser was still available) in an attempt to lower reactor pressure to the point
(less than 650 psig) where the condensate pumps would be able to provide water to the
reactor.
As condenser vacuum continued to degrade, the feedwater pumps tripped (as
expected) on a low condenser vacuum signal before reactor pressure could be lowered
to the point where the condensate pumps could be used; and reactor vessel level began
to lower due to the resulting reduction in reactor vessel level makeup. In an effort to
maintain reactor vessel inventory, the operators closed the bypass valves and manually
initiated the reactor core isolation cooling (RCIC) system to maintain reactor vessel level
within the desired level control band. While the operators were in the process of
manually placing the RCIC system in-service, the high pressure coolant injection (HPCI)
system automatically initiated when reactor vessel2 reached the point at which both
RCIC and HPCI automatically start (Level 2).
Reactor vessel level began to recover from the Level 2 setpoint (the lowest vessel level
reached during the transient) with RCIC, and HPCI was secured from reactor vessel
injection. Condenser vacuum continued to degrade and the operators manually closed
the main steam isolation valves (MSIV) in anticipation of an automatic close signal due
to low vacuum. Further operations with the MSIVs closed complicated the post scram
reactor vessel water level and pressure control because the main condenser is the
normal source for heat removal. The operators attempted to place HPCI in service in
the pressure control mode of operation, however, they were delayed for less than 10
minutes due to a valve interlock issue that was subsequently resolved. As reactor
vessel level returned to the normal range, the operators reduced RCIC flow to maintain
1
See Attachment C for a detailed Sequence of Events for the October 10, 2004, event.
2
See Attachment E for a discussion on reactor water level setpoints and definitions.
Enclosure
2
the desired level band and noticed that flow oscillations were occurring with the RCIC
system. As reactor pressure was lowered with HPCI (in pressure control mode), reactor
vessel level control was transitioned to the condensate system, and RCIC was removed
from service.
A normal reactor cooldown rate was established with HPCI until a subsequent
equipment issue (vacuum pump trip) resulted in HPCI being removed from service. The
cooldown was successfully re-established using the safety relief valves (SRV).
Additionally, as expected during RCIC and HPCI operation, suppression pool
temperature increased and the operators placed the residual heat removal (RHR)
system in-service, as directed by emergency operating procedures (EOP). The RHR
system remained in service in the suppression pool cooling mode of operation as the
cooldown continued with the SRVs. Operators stabilized the plant in Hot Shutdown
(average coolant temperature > 200F) at 10:11 p.m. on October 10. The RHR system
was subsequently placed in the shutdown cooling mode of operation, and the plant was
placed in Cold Shutdown (average temperature # 200F) at 5:09 a.m. on October 12.
During the initial event and subsequent cooldown, reactor vessel water level cycled in
response to various plant conditions and operator actions, including main turbine bypass
valve operation, reset of the scram signal, initiation of RCIC and HPCI, and SRV cycling.
Additionally, during the event, the operators noted equipment operational issues with the
reactor water cleanup (RWCU), RCIC, and HPCI systems.
The source of the steam leak was the failure of an 8-inch pipe from the A MS drain
tank line to the main condenser. This failure was the cause for the initial steam leak and
subsequent rapid decrease in condenser vacuum. Post-scram review identified that a
level control valve (LV-1039A) associated with the A MS drain tank had been open in
excess of three weeks, and was the direct cause for the pipe failure. In addition, a
spring can pipe hanger (H25), designed to provide support for an upstream portion of
the failed pipe, was found to have been disconnected. An extension rod, associated
with hanger H25, had worn a hole in the air supply line to LV-1039A due to vibration
over an extended period of time (several years) which caused the LV-1039A valve to fail
open.3
There were no injuries associated with this event. There was a minor radiation release
from the plant that was well below approved regulatory limits. The majority of this
release was monitored by the turbine building exhaust and south plant ventilation stack
radiation monitors.
Operator performance issues and challenges are discussed in Section 2.1; and specific
equipment issues are discussed in Section 2.2.
3
See Attachment D for a MS/drain system and level control valve issue chronology.
Enclosure
3
2.0 PLANT RESPONSE: PERSONNEL AND EQUIPMENT
2.1 Operator Performance and Training
a. Inspection Scope
The team reviewed and assessed licensed operator performance during the transient
and manual reactor scram caused by the pipe failure, including reactor vessel level and
pressure control and the subsequent plant cooldown. In particular, the team focused on
the plant response as reactor vessel level dropped below the Level 2 setpoint for a short
period of time. The team reviewed and evaluated the operators use and adherence to
normal, abnormal, and emergency procedures during transient mitigation and
subsequent plant operations. Additionally, the team assessed adherence to Technical
Specifications and compliance with associated Limiting Conditions of Operations (LCO).
The team interviewed all licensed operators involved in the event to assess operator
performance during the transient and during subsequent operations to cold shutdown.
Items reviewed included the following:
- operator logs;
- narrow and wide range reactor vessel water level graphs;
- sequence of events computer printout;
- narrow and wide range reactor pressure graphs;
- PSEGs root cause evaluations.
The team also reviewed operator actions and communications that occurred during shift
turnover, after the plant had been stabilized in a Hot Shutdown condition, and during the
subsequent plant depressurization and cooldown.
The team reviewed the dynamic response of plant systems, evaluated the transient
response of critical plant parameters, and reviewed the actions taken by the operators in
order to determine if any significant issues existed in the areas of simulator fidelity or
operator training. In particular, the team reviewed PSEGs evaluations regarding HPCI
system operation during the plant depressurization and cooldown; RCIC system
operation during the transient when operators observed system flow oscillations; and
conditions which led to the trip of the B RWCU pump during plant depressurization.
In addition, the team reviewed two PSEG Root Cause Analysis Reports associated with
operational aspects of the events. Specifically, one Root Cause Investigation Report
(Notification/Condition Report 20206606/70041900) evaluated Technical Specification
management; and another Root Cause Investigation Report (Notification/Condition
Report 20206631/70041930) evaluated reactor vessel water level control difficulties.
Enclosure
4
b. Findings
No findings of significance were identified. Several observations and one minor violation
are discussed in the following sections.
1. General - Transient Response and Procedure Adherence
The team found that the on-shift crew displayed adequate command and control during
the initial transient response. Actions directed by the control room supervisor (CRS) and
taken by the board operators were in accordance with procedures. The team also noted
that extra personnel (from the oncoming shift) were utilized in the field to ascertain the
cause of the transient and were effective in communicating pertinent plant condition
information to the CRS and shift manager (SM). The team found that operators entered
the appropriate abnormal and emergency procedures as the event progressed and as
plant conditions changed. The team also found the operators followed procedural
guidance in order to maintain reactor vessel level within the prescribed ranges, however,
the operators were challenged in this area due to equipment performance issues.
2. Reactor Vessel Level and Pressure Control
The team found that operators controlled reactor vessel pressure in the required range
during the transient and subsequent reactor vessel cooldown to Cold Shutdown. The
team also found that the operators ability to control reactor pressure was affected by
mitigation system operational issues (HPCI, RCIC); these equipment issues are
discussed in Section 2.2. Regarding reactor vessel level control, the team found that on
five occasions, reactor vessel level was outside the normal transient response level
band of Level 3 (+12.5 inches) to Level 8 (+54 inches) as prescribed in the EOPs. Each
of the five occasions is described and assessed below. See Attachment E for detailed
information regarding reactor vessel level setpoints, instruments, and definitions. As a
reference, Level 2 (-38 inches), which was the lowest level reached during this transient,
is about 10 feet above the top of the active fuel (which is at -161 inches).
C October 10, 2004 at 6:19 p.m.- Level 2 Automatic Engineered Safety Feature
Initiation and Isolation Signal: Following the initial manual scram and with main
condenser vacuum rapidly degrading, the CRS, anticipating the eventual loss of
the turbine driven reactor feedwater pumps (RFP) on low condenser vacuum,
directed the reactor operators to open the main turbine bypass valves to reduce
reactor pressure to between 600 and 700 psig from approximately 920 psig.
That pressure band was chosen to enable feeding the reactor vessel with the
condensate system pumps (motor driven), whose shutoff head is approximately
650 psig. As vacuum continued to degrade and just prior to the loss of the RFPs
on low condenser vacuum, the CRS directed closure of the bypass valves. The
CRS stated that due to the imminent loss of the RFPs, the action to close the
bypass valves was intended to preserve reactor vessel inventory. At that time,
reactor pressure was not low enough for the condensate system to provide
makeup to the reactor vessel, so the CRS directed manual initiation of the HPCI
and RCIC systems to restore vessel level.
Enclosure
5
The closure of the bypass valves caused an increase in reactor vessel pressure
and a corresponding shrink in vessel level. The bypass valve closure and
associated level shrink combined with the loss of the RFPs caused vessel level
to decrease to Level 2. As the Level 2 setpoint approached, operators manually
initiated RCIC, and HPCI automatically initiated at the Level 2 setpoint. Reactor
vessel level began to increase immediately upon reaching the Level 2 setpoint,
and the operators began to restore reactor vessel level with RCIC to a normal
band of between Level 3 and Level 8, as directed by EOPs, and secured HPCI
injection flow for use in the pressure control mode. The HPCI system was
needed for pressure control because the MSIVs were manually closed (due to
the rapidly degrading condenser vacuum) and therefore, the turbine bypass
valves were not available for pressure control.
The team found that the actions directed by the CRS and taken by the control
room operators were in accordance with abnormal and emergency procedures.
The operators stated the magnitude of the vessel shrink and inventory loss with
bypass valve operation appeared to be larger in the plant than in the simulator.
C October 10, 2004 at 6:46 p.m. - Level 3 Automatic Reactor Scram Signal During
Plant Stabilization: As reactor vessel level recovered to the normal band with
RCIC, HPCI was placed in the pressure control mode of operation. With reactor
vessel level rising to within the normal band of Level 3 to Level 8, the operators
reduced RCIC system injection flow to maintain the normal band. However, as
RCIC flow was reduced, the RCIC system began to experience flow oscillations.
This condition was unexpected by the operators and after about two minutes of
continuous oscillations, RCIC was placed in a minimum flow condition. Reactor
vessel level peaked at about +30 inches before beginning a slow trend
downward with HPCI in the pressure control mode. With reactor vessel level
approaching +20 inches, the operators returned RCIC to service at a reduced
flow rate (350 gpm) in order to maintain a controlled reactor vessel level
increase. However, at this reduced flow rate, RCIC flow oscillations occurred
again (See Section 2.2.2 for a detailed discussion on the RCIC flow oscillations).
Since plant conditions were stable, including reactor vessel level (at about +30
inches), a decision was made to reset the reactor protection system (RPS) Level
3 scram signal, as directed by plant procedures. The scram signal and RPS
were reset, however, the decision to reset the scram was not communicated to
the operators controlling reactor vessel level (with RCIC) and pressure (with
HPCI). Following the scram reset, a slow downward reactor vessel level trend
commenced due to 1) the steam demand from HPCI in the pressure control
mode (which was more pronounced than the operators had expected); 2) the
unexpected flow oscillations with the RCIC system, and 3) the termination of
control rod drive system flow to the reactor vessel (of which the operators
controlling level were not aware). Consequently, in less than two minutes after
the scram reset, reactor vessel level had reached the Level 3 setpoint, and an
automatic RPS scram signal occurred (as designed). The operators responded
promptly to the scram signal and restored level back to the normal band by
increasing RCIC injection flow. Reactor vessel level reached a minimum of +10
inches before being restored to above Level 3.
Enclosure
6
The team reviewed the operators actions leading to the RPS actuation and
concluded there were control room communications and assessment
weaknesses. Although several operators were aware that the scram signal was
being reset, the operators directly involved with reactor vessel level and pressure
control were unaware of the decision. The team found that operator distractions
caused by the RCIC system flow oscillations and underestimation of the steam
demand associated with the HPCI system in the pressure control mode
contributed to the weakness in assessment and communications.
C October 10, 2004 at 9:38 p.m. - Level 8 Automatic Overfill Protection During
Plant Cooldown: With HPCI in the pressure control mode of operation, reactor
pressure at 400 psig, and reactor vessel level at +25 inches on the narrow range
scale, the HPCI barometric condenser vacuum pump circuit breaker opened
(tripped). The operators placed RCIC in service in the pressure control mode of
operation in anticipation of removing HPCI from service due to the vacuum pump
trip. The vacuum pump breakers thermal overload device had actuated
(indicating excessive electrical current), and after no apparent cause for the trip
was found, operators reset the breaker (permitted by procedure) and the pump
was restarted. After approximately five minutes of successful HPCI vacuum
pump operation, RCIC was removed from service. However, after about ten
additional minutes of HPCI operation, the vacuum pump breaker tripped again,
and the operators removed the HPCI pump from service (See Section 2.2.1 for a
detailed discussion of the HPCI vacuum pump issue).
The RCIC system was again placed in the pressure control mode, however, the
RCIC system capacity was insufficient to continue to reduce reactor pressure,
which began to slowly increase (RCIC capacity is about 10% of the HPCI system
capacity). During this period, reactor vessel level increased to approximately
+42 inches on the narrow range instruments; which equated to approximately
+54 inches (Level 8) on the wide range vessel level instruments at this reduced
pressure of 400 psig. Level 8 on the wide range instruments initiates a RCIC trip
signal, and a Level 8 trip of RCIC occurred as designed.
As described in Attachment E, the discrepancy between wide and narrow range
level indication during the reactor coolant system cooldown is expected due to
changes in the density of the reactor vessel inventory as the cooldown
progressed. This was expected by the operators due to the known deviations
between narrow and wide range level instruments experienced during a reactor
vessel depressurization and cooldown. The team noted that once level
decreased below Level 8, the operators again placed RCIC in the pressure
control mode but its capacity was unable to reduce pressure. The operators
then removed RCIC from service and recommenced the plant cooldown with
SRVs to control pressure and the condensate system to control level. The team
found these actions to be acceptable. The team found that given the equipment
operational issues with HPCI and the challenges of maintaining reactor vessel
level in the required band during vessel cooldown, the operators took appropriate
actions in accordance with procedures.
Enclosure
7
C October 10, 2004 at 9:57 p.m. - Level 3 Automatic Scram Signal During Reactor
Cooldown: After RCIC tripped in response to the Level 8 signal noted above,
operators continued the cooldown with SRVs as directed by operating
procedures. Safety relief valve F013R was manually opened with reactor
pressure at 475 psig and remained open for approximately nine minutes until
reactor pressure reached approximately 350 psig. Reactor vessel level had
increased as expected due to level swell when the SRV was opened. Then, the
reactor vessel level began to decrease (due to the inventory reduction from the
open SRV), and the operators closed the SRV when reactor vessel level was at
about +25 inches on the narrow range level indicator. When the SRV went fully
shut, a level shrink occurred as expected and narrow range level decreased to
approximately +7 inches, which is below the Level 3 RPS trip setpoint; and an
RPS actuation occurred. The operators promptly recovered reactor vessel level
above the trip setpoint using the condensate system.
The team reviewed the actions of the operators regarding the use of the SRVs
for pressure control and found that the actions taken were in accordance with
procedures. The team found that the operators discussed the effects of SRV
use on reactor level and had established a level band which they believed, based
on training experience, would have precluded reaching the Level 3 RPS
actuation setpoint. The operators informed the team that although training is
performed requiring SRV usage, it is not typically performed at the lower reactor
pressures experienced in this transient. The team also noted that while using
SRVs during the remainder of cooldown, the operators maintained a higher
reactor level band, and no further Level 3 RPS actuation occurred.
C October 10, 2004 at 10:04 p.m. - Level 8 Automatic Overfill Protection During
Plant Cooldown: Due to the dynamic vessel response and the Level 3 RPS
actuation that occurred while using the SRVs for pressure control, the operators
placed the RCIC system in service in the pressure control mode to continue the
reactor vessel cooldown. However, as previously experienced, RCIC (in
pressure control mode) was not able to effectively reduce reactor pressure due
to its relatively small capacity; and reactor vessel level and pressure began to
slowly rise. Once reactor level reached +42 inches on the narrow range level
instruments (about Level 8 on the wide range level instruments at this low
pressure), the RCIC turbine tripped as expected. The operators then continued
the cooldown with SRVs while maintaining the previously established level band.
With this higher level band on the narrow range (to avoid reaching the Level 3
RPS setpoint), a Level 8 trip signal resulted on the wide range level instruments.
Enclosure
8
The team reviewed the operators actions regarding control of the reactor coolant
system cooldown with RCIC and SRVs and found their actions to be in
accordance with procedures. The team found that due to the inability of the
RCIC system to effectively maintain a cooldown and the dynamic response on
vessel level due to operation of the SRVs, the operators decision to maintain a
higher band on the narrow range vessel level was appropriate. The team also
noted that although operating experience existed regarding SRV usage and the
dynamic effects on reactor vessel, PSEG did not effectively utilize this
information in classroom or simulator training for the operators. PSEG initiated
Notification 20212885 to address this apparent weakness associated with the
operating experience program.
3. Training and Simulator Fidelity
Based on review of the event, associated documents, and interviews of operators, the
team found that simulator modeling and operator training was a factor in the following
areas:
C The response of reactor pressure/level during cycling of turbine bypass valves
and SRVs was more pronounced in the plant than in the simulator.
C The RCIC system exhibited flow oscillations while in automatic under low flow
conditions (about 350 gpm). The simulator did not model this known system
operational challenge; and operators and personnel responsible for simulator
modeling/training were not aware that the system may experience flow
oscillations under low flow/automatic control conditions. This was a procedure
inadequacy issue related to operating experience and is discussed further in
Section 2.2.2.
C The steam demand of the HPCI system (in pressure control) and the affect on
reactor vessel level was more pronounced in the plant than in the simulator.
C Steam leak training scenarios did not typically carry the transient through the
transition from Hot Shutdown to Cold Shutdown with the use of SRVs or HPCI
for the cooldown.
C When the steam leak initially occurred, the steam jet air ejectors (SJAE) were
initially able to maintain condenser vacuum as reactor power was decreased.
For similar simulator scenarios, based upon operator interviews and discussions
with simulator personnel, the SJAEs were not modeled to maintain condenser
vacuum once a steam leak occurred.
The team found that although the operators encountered plant transient responses
which differed slightly from the simulator response they experienced in operator training,
they were able to operate safety systems and other plant equipment in accordance with
procedures to safely mitigate the transient. The team found that PSEGs review of
training and simulator issues was adequate, including corrective actions; and that the
issues were entered into the corrective action system (Notification No. 20212476).
Enclosure
9
4. Technical Specification Misinterpretation
The NRC identified that operators misinterpreted Technical Specification (TS) 3.6.2.3,
which is associated with the residual heat removal (RHR) system. The reactor scram
occurred at 6:14 p.m. on October 10, 2004. During the transient response and as
directed by procedures, the operators placed both loops of the RHR system in the
suppression pool cooling mode of operation (a secondary mode of operation) at 6:31
p.m. The primary mode of operation for RHR is the standby alignment for accident
response in the low pressure coolant injection mode. In accordance with the existing
requirements described in procedure, SH.OP-AP.ZZ-0108(Q), Operability Assessment
and Equipment Control Program, the affected RHR loops were declared inoperable.
The Action Statement for TS 3.6.2.3 requires operators to place the plant in at least Hot
Shutdown within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> and in Cold Shutdown within the next 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. Operators
determined that since the TS LCO had been entered on October 10 at 6:31 p.m., the
requirement to place the plant in Cold Shutdown had to be satisfied within 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br /> (12
hours added to 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />). Operators planned a controlled plant cooldown such that
Cold Shutdown would be reached prior to exceeding the 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />. The operators
achieved Cold Shutdown on October 12 at 5:09 a.m., about 34.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> after the TS
Limiting Condition for Operation (LCO) was initially entered.
However, recognizing that the plant was already in Hot Shutdown at 6:31 p.m. when the
TS LCO was entered (as a result of the reactor scram), the NRC questioned operators
why they were not in the 24-hour LCO Action Statement. Subsequently, PSEG
determined that the potential existed for a missed TS Action Statement and a one hour
report to the NRC, for after-the-fact discovery for this issue, was initiated on October 12
at 7:45 p.m. (NRC Event Notification 41117). The team concluded that operators
misinterpreted, and therefore, did not comply with TS 3.6.2.3 to place the plant into Cold
Shutdown within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of declaring the RHR system inoperable.
The RHR system had been declared inoperable to address a generic concern regarding
a postulated scenario involving a simultaneous design basis loss of coolant accident and
loss of power event that could result in a water hammer upon the subsequent restart of
the RHR system. This issue was identified in response to concerns regarding long term
operation of the RHR system in the suppression cooling mode. However, the team
noted that the postulated scenario described above was not required to be considered
for short term operation of the RHR system in the suppression pool cooling mode
(reference: Response to Task Interface Agreement 2001-14 dated April 28, 2003).
Additionally, during this event, the system remained available for all modes of RHR
operation. Accordingly, the team concluded that the TS mis-interpretation problem was
a violation of minor significance not subject to formal enforcement action in accordance
with Section IV of the NRC's Enforcement Policy. PSEG documented the problem in
their corrective action program as Notification 20206606.
Enclosure
10
2.2 Equipment Performance
a. Inspection Scope
The team reviewed system and equipment performance issues as they related to the
ability of the operators to effectively mitigate the transient and place the plant in a Cold
Shutdown condition as required by Technical Specifications. The team reviewed
operator logs, operating and emergency procedures, plant data, and interviewed
personnel. The team also reviewed prior equipment performance, including
maintenance and testing aspects, to determine whether there were prior opportunities to
identify the problems; and reviewed PSEGs evaluation of the specific deficiencies.
b. Findings
Three self-revealing Green findings and two observations are documented in the
following sections.
1. HPCI Full Flow Test Valve - Failure to Open on Demand
Introduction. A self-revealing finding of very low safety significance (Green) was
identified associated with the HPCI system full flow test valve. A related HPCI injection
valve limit switch was incorrectly set, resulting in the failure of the full flow test valve to
open upon demand. The issue was determined to be a non-cited violation (NCV) of
10 CFR Part 50, Appendix B, Criterion V (Instructions, Procedures, and Drawings).
Description. While responding to the October 10, 2004, event, operators placed the
HPCI system in a standby alignment after its automatic initiation and were in the
process of placing the system in a full flow test (pressure control) mode to begin a
reactor vessel cooldown in accordance with EOPs. When the operator attempted to
open the HPCI full flow test valve F008, it failed to open. The operators made three
additional attempts to open the valve without success.
The operators then checked the system lineup, which included verifying that HPCI
injection valves F006 and HV-8278 were both closed. These valves are interlocked with
the F008 and must both be closed to permit the opening of F008. Although F006 and
HV8278 indicated closed (close light on), the operator depressed both valves close
pushbuttons in an attempt to ensure the interlocks were satisfied for the F008 valve to
open. The operator again attempted to open valve F008 and was successful. The
HPCI system was then successfully placed in the pressure control mode of operation
(within ten minutes).
PSEG determined that F008 failed to open on demand due to an improper adjustment
of the limit switch on valve HV-8278. The HV-8278 limit switch was in the open position
with the valve actually in the closed position; and, therefore, the interlock with F008 was
not satisfied. The closure of HV-8278 is controlled by the motor operator torque switch
(i.e., valve travel stops when the torque switch is actuated). Due to the improperly set
limit switch, the limit switch contacts did not actuate when valve travel stopped. This
condition was not previously identified during routine HV-8278 and F008 testing
because prior testing was done under static (no flow) conditions, and the limit switch had
Enclosure
11
actuated before valve travel stopped. During the event, HV-8278 was closed while there
was still HPCI system flow in the injection line (dynamic conditions), and the torque
switch stopped valve travel slightly sooner due to the higher loading than would have
been experienced under static conditions. The valve was closed, however, it was not
seated enough to allow the limit switch to actuate. The limit switch and torque switch
settings are such that, if set properly, the limit switch would actuate under both static
and dynamic conditions.
Depressing and holding the closed pushbutton on HV-8278 overrides the torque switch,
and the valve motor operator will continue to run until the limit switch actuates.
Following the failed attempts to open F008, control room operators took this action to
depress the close pushbutton for HV-8278, which produced enough additional stem
travel to activate the limit switch.
In evaluating this issue, PSEG identified that the limit switch for the HV-8278 was last
set on May 2, 2003, however, it was not set in accordance with procedural requirements
as stated in SH.MD-EU.ZZ-0011(Q), VOTES Data Acquisition for Motor Operated
Valves. Additionally, PSEG determined that the engineering review of the procedure
results failed to identify the discrepancy in the limit switch settings.
The team found that the technicians did not properly set the HV-8278 limit switch in
accordance with the procedure. The team also found that engineers did not perform an
adequate review of the completed procedure, as required. These failures led to a delay,
of less than ten minutes, in placing the HPCI system in-service when it was required for
transient event mitigation. The team also found that, while the periodic static testing of
the valves did not identify this discrepancy, the valves motor operator design (including
the proper limit and torque switch settings and sequencing) ensures valve operation
under both static and dynamic conditions. Due to system design and configuration, it is
not practical to dynamically test the HPCI injection valves as it would require HPCI
injection flow to the reactor. The team noted that operation of the HPCI in its safety
mode (i.e. injection) was not affected by this problem.
Analysis. The team concluded that the performance deficiency was that plant personnel
did not correctly perform procedure SH.MD-EU.ZZ-0011(Q). Specifically, the technician
incorrectly set the limit switch for HPCI valve HV-8278 on May 2, 2003. The team also
considered that this finding was indicative of a cross-cutting weakness in the area of
human performance (personnel). Additionally, the required engineering review of the
completed procedure did not identify this discrepancy, which was indicative of a cross-
cutting weakness in the area of problem identification and resolution (identification).
The valve failure led to a delay, of less than ten minutes, in the operators placing the
HPCI system in-service when it was desired for event mitigation.
Enclosure
12
This self-revealing finding was greater than minor because it was associated with the
Equipment Performance attribute of the Mitigating Systems cornerstone and affected
the cornerstone objective to maintain mitigation equipment reliable. The team reviewed
this finding using the Phase 1 Significance Determination Process (SDP) worksheet for
mitigating systems and determined the finding was of very low safety significance
(Green), because the finding did not represent the actual loss of the safety function of
the HPCI system. Also, reactor pressure remained relatively stable when the issue
occurred and alternate pressure control methods were available (SRVs) if required.
Enforcement. 10 CFR Part 50, Appendix B, Criterion V (Instructions, Procedures, and
Drawings) requires that, activities affecting quality shall be prescribed by documented
instructions and procedures appropriate to the circumstance and shall be accomplished
in accordance with these instructions. Contrary to the above, PSEG procedure SH.MD-
EU.ZZ-0011(Q) was not accomplished in accordance with the instructions delineated in
the procedure. Specifically, Attachment 10 of the procedure was performed incorrectly
on May 2, 2003, on valve HV-8278. This incorrectly performed procedure led to the
failure of the F008 valve to open on demand during a transient on October 10, 2004.
However, because the violation was of very low safety significance and PSEG entered
the deficiency into their corrective action program under Notifications 20210277 and
20206665, and Order 70041898, this finding is being treated as a non-cited violation
consistent with Section VI.A of the NRC Enforcement Policy. (NCV 05000354/2004013-
01, Failure to Properly Set Limit Switch Settings on HPCI Injection Valve in
Accordance with Procedures)
2. RCIC System Flow Oscillations
Introduction. A self-revealing finding of very low safety significance (Green) was
identified associated with the operation of the RCIC system. Industry operating
experience regarding flow limitations while operating the system in the automatic mode
of operation were not incorporated into operating procedures or training. The issue was
determined to be an NCV of 10 CFR Part 50, Appendix B, Criterion V (Instructions,
Procedures, and Drawings).
Description. During the response to the October 10, 2004, event, operators were in the
process of recovering reactor vessel level with the RCIC system in the automatic mode
of operation (flow control) and in accordance with EOPs. Operators noted system flow
oscillations as they reduced RCIC flow to maintain reactor water level in the normal
range of Level 3 to Level 8. The operators had not seen this system condition before,
either in actual use or in simulator training, and the oscillating flow caused a slight delay
(about 2-3 minutes) in restoring vessel level above the scram setpoint at Level 3.
PSEGs root cause evaluation identified that operating experience (OE) existed in the
turbine vendor document (VTD 323601), which described the condition observed by the
operators, and cautioned against operating the RCIC system in automatic under lower
flow conditions. As stated in the vendor manual, the RCIC system design basis is to
deliver a constant flow rate (600 gpm) in the automatic control mode to the reactor
vessel over a wide range of reactor pressures. However, reducing system flow to below
75% of rated flow (about 450 gpm) in the automatic control mode promotes the
Enclosure
13
likelihood of flow instability. In the event the instability occurs, it was recommended that
the system be placed in the manual mode of operation (speed control).
The team found that these cautions and limitations were not incorporated into the RCIC
operating procedure and therefore were unknown to the operators; and were also
unknown to the operations training department.
Analysis. The team concluded that the performance deficiency was that existing vendor
guidance had not been incorporated into the RCIC system procedure. Specifically,
procedure cautions and limitations did not discuss the potential for system oscillations if
flow was reduced in the automatic mode of operation. The team considered that this
finding was indicative of a cross-cutting weakness in the area of problem identification
and resolution (identification). Additionally, the failure to incorporate operating and
vendor information into procedures and training was indicative of a cross-cutting
weakness in the area of human performance (organization). The failure to incorporate
the information led to mis-operation of the system in the automatic mode of operation
and a slight delay in restoring reactor vessel level above the scram setpoint of +12.5
inches (Level 3). This self-revealing finding was more than minor because it was
associated with the Procedure Quality attribute of the Mitigating Systems cornerstone
and affected the cornerstone objective of ensuring the capability of systems that
respond to initiating events to prevent undesirable consequences. The team reviewed
this finding using the Phase 1 SDP worksheet for mitigating systems and determined
the finding was of very low safety significance (Green), because the finding did not
result in the actual loss of the safety function for the RCIC system. Also reactor vessel
level was maintained in the appropriate range in accordance with procedures and the
HPCI system was available for reactor vessel level makeup if required.
Enforcement. 10 CFR Part 50, Appendix B, Criterion V (Instructions, Procedures, and
Drawings) requires that, activities affecting quality shall be prescribed by documented
instructions and procedures appropriate to the circumstance and shall be accomplished
in accordance with these instructions. Contrary to the above, PSEG procedure HC.OP-
SO.BD-0001(Q), Reactor Core Isolation Cooling System, did not include RCIC system
operating and vendor information which specified cautions and limitations for system
operation in the automatic mode. This led to the inappropriate operation of the RCIC
system when it was needed for transient event mitigation. However, because the
violation was of very low safety significance and PSEG entered the deficiency into their
corrective action program under Notification 20206783 and Order 70041898, this finding
is being treated as a non-cited violation consistent with Section VI.A of the NRC
Enforcement Policy. (NCV 05000354/2004013-02, Failure to Incorporate Operating
Experience for Low Flow Operations of RCIC Into Operating Procedures and
Operator Training)
Enclosure
14
3. HPCI System Barometric Condenser Vacuum Pump Trip
Introduction. A self-revealing finding of very low safety significance (Green) was
identified associated with the HPCI system barometric condenser vacuum pump.
Because of inadequate preventive maintenance on the vacuum pump assembly,
operators removed the HPCI system from service during a plant event after it had been
operating for reactor vessel depressurization and cooldown.
Description. While the HPCI system was operating during the October 10, 2004 event,
the HPCI barometric condenser vacuum pump circuit breaker opened (tripped). The
vacuum pump breakers thermal overload device had actuated (indicating excessive
electrical current). Upon initial investigation, there was no apparent reason for the trip
and the breaker was reset and the vacuum pump restarted (permitted by procedure).
The pump circuit breaker tripped again a few minutes later, at which time the operators
elected to remove the HPCI system from service and continue the cooldown using the
SRVs.
The team reviewed PSEGs evaluation of the issue in the root cause report (Order
70041898) which determined that preventive maintenance instructions for the pump
specified the incorrect lubricant, which led to pump shaft binding and ultimately to a
thermal overload trip of the pump circuit breaker while the pump was in-service.
Specifically, in the lubrication information specified for the pump, a thread sealant was
specified as the lubricant to be installed into the vacuum pump stuffing box instead of
the correct lubricant, which was a multi-purpose lithium grease as specified by the pump
manufacturer. Additionally, PSEG found that the packing gland follower on the pump
was corroded, resulting in binding on the shaft sleeve.
The team also found that a previous trip of the HPCI system vacuum pump breaker had
occurred on July 6, 2004 (Notification 20195868). At that time, the only corrective action
taken was to visually check the pump and breaker for apparent problems. Because no
problems were apparent, the operators reset the breaker, started the vacuum pump,
and ran the HPCI quarterly surveillance test successfully (Procedure HC.OP-ST.BJ-
0001(Q)). The corrective actions were closed to a trend status following a shift manager
(licensed SRO) review. The team reviewed the corrective actions resulting from the
current issue, which included an extent of condition review to include a cross-section of
procedures used at Hope Creek and Salem to ensure that they were appropriately
detailed and contain the correct technical information. Immediate corrective actions
included re-building the HPCI vacuum pump, and re-packing and re-greasing the RCIC
vacuum pump (which also contained the wrong lubricant but had not been adversely
affected). The team determined that the corrective actions were adequate.
Analysis. The team concluded that the performance deficiency was ineffective
implementation of preventive maintenance program for the HPCI barometric condenser
vacuum pump, which led to a pump failure. In particular, the pump preventive
maintenance instructions (Procedure PM031777, and the associated lubrication screen
from the computer-based information tool) were inadequate in that the incorrect
lubricant was specified. This led to pump shaft binding and ultimately a thermal
overload trip of the pump circuit breaker while the pump was in-service. Additionally, the
team concluded that corrective actions from a similar pump trip in July 2004 did not
Enclosure
15
prevent recurrence of the problem. The team considered that this finding was indicative
of cross-cutting weaknesses in the area of human performance (organization) and
problem identification and resolution (corrective action).
The team noted that the vacuum pump is non-safety related support system equipment
for the HPCI pump. Due to the pump failure, operators removed the HPCI system from
service and continued a vessel cooldown with SRVs. The finding is greater than minor
because it is associated with the Equipment Performance-Maintenance attribute of the
Mitigating Systems cornerstone and it affects the associated cornerstone objective of
ensuring the reliability of systems that respond to initiating events. The finding is of very
low safety significance (Green) because per design basis, with the vacuum pump not
available, the HPCI system remained operable and was able to perform its mitigation
function if required. This issue has been entered into PSEGs corrective action program
as Notifications 20206604, 20206634, and 20212882; and Order 70041898.
Enforcement. No violation of NRC regulatory requirements was identified. Although
PSEG did not effectively implement its preventive maintenance program related to the
HPCI barometric condenser vacuum pump, which is a non-safety related piece of
equipment, this aspect of the preventive maintenance program is not an NRC regulatory
requirement. (FIN 05000354/2004013-03, Failure to Effectively Implement the
Preventive Maintenance Program for the HPCI Barometric Condenser Vacuum
Pump)
4. Reactor Water Cleanup System Pump Trip
The team reviewed the details associated with the trip of the B RWCU pump during
October 10, 2004, plant depressurization. The team noted that the nuclear industry has
identified a generic phenomenon associated with RWCU pump trips during reactor
coolant system depressurization. This known issue had been placed on the Hope Creek
operations work-around list as a problem that may degrade the operators ability to
respond to a transient.
The phenomenon involves flashing in the instrumentation line during transient
depressurization conditions that may result in low pump suction flow being sensed by
the transmitter, which in turn will cause a trip of the operating RWCU pump. PSEG
determined during their review of operating experience and through concerns discussed
at Boiling Water Reactor (BWR) Owners Group meetings, that this issue was present in
several other BWRs. In response to previous RWCU pump trips at Hope Creek, and as
indicated by operating experience, PSEG had implemented actions since 2001 to vent
the instrument lines to the flow transmitter to remove entrapped gases. This appeared
to temporarily fix the problem, and the system operated satisfactorily until a March 2003
pump trip. Based upon additional industry initiatives and correspondence, PSEG
initiated several additional corrective actions following a subsequent April 2003 trip.
These proposed actions included revising station procedures to better control system
flow in plant startup and shutdown conditions, re-sloping the instrument tubing lines, and
physically moving the pump suction venturi to a lower elevation.
Enclosure
16
The team found that during the October 10, 2004, event, the RWCU system (a non-
safety related system) was not required for transient mitigation. Based upon the
inclusion of the RWCU on the operator work-around list and the planned and completed
corrective actions, the team concluded that PSEG efforts to date to address industry
concerns associated with the RWCU pumps appeared reasonable.
5. Low Reactor Vessel Level Instrument Actuations/Logic
The team found that as reactor vessel reached Level 2 (-38 inches), two of the four wide
range level instrument channels tripped (channels A and B) and then vessel level
recovered above the Level 2 setpoint. Level channels C and D did not trip. All four of
the level channels provide inputs to the logic for the redundant reactivity control system,
the primary containment isolation system, the nuclear steam supply shutoff system, the
HPCI system, and the RCIC system. The team reviewed the plant computer and
instrument calibration data and found that the level differences sensed between each of
the level channels were within the calibration tolerances permitted by plant procedures.
Additionally, the team noted that the calibration tests for each of the level instruments
had been performed within the required frequency. The team performed a detailed
review of the systems mentioned above and found that isolations, actuations, and
initiations had occurred as expected due to trip of level channels A and B. The
isolations, actuations, and initiations which occurred included:
C HPCI pump initiation;
C Primary containment isolation valve closures;
C RWCU pump suction valve closure; and
C Filtration recirculation ventilation system fans automatically started.
The team also reviewed and verified that the operators performed system lineup
verifications as provided in PSEG procedures HC.OP-AB.CONT-0002(Q), "Primary
Containment," and HC.OP-SO.SM-0001, "Isolation Systems Operation," to identify that
the isolations, actuations, and initiations caused by the level channel trip signals
occurred as expected.
The team noted that PSEG identified that level inputs to some of the control room
computer display system (CRIDS) reactor level display points were outside the
tolerances recommended by the vendor. The team verified through review of instrument
drawings, that the level inputs for CRIDS were unrelated to the level inputs for system
isolations, actuations, and initiations, and were used for indication only.
The team also reviewed Notifications 20214768 and 20214655, which documented the
engineering analysis used to evaluate the differences between the reactor vessel levels
sensed by each of the four level channels. The analysis focused on thermal error,
physical error, design margins, setpoints, and allowable instrument drift for each of the
channels for both wide and narrow range instruments. The analysis concluded that the
level discrepancies which occurred were in all cases either within the error assumptions
of the setpoint calculations or within the available margins to the analytical limits.
Enclosure
17
The team determined, based on the information reviewed, that the reactor vessel level
trip systems functioned as designed and that the isolations, actuations, and initiations
that occurred due to the trip of level channels A and B were appropriate. The team also
reviewed PSEGs recommended actions intended to achieve better accuracy in the level
inputs to the CRIDS system and found them to be adequate. The corrective actions
related to the above referenced notifications were also reviewed by the team, and found
to be adequate.
2.3 Radiological Release Assessment
a. Inspection Scope
The team reviewed data and calculations used to quantify the amount of radioactive
material released following the pipe break and subsequent steam release from the
turbine building on October 10, 2004.
b. Findings
PSEG evaluated the radiological release associated with the event and concluded that
the amount of radioactive material released and its impact on the maximally exposed
individual were both below regulatory limits. The team reviewed PSEGs analyses and
conducted independent assessment, and confirmed that the radiological release was
below regulatory limits.
PSEG technicians obtained two grab samples during the transient, and generated two
gaseous radioactive waste release permits. These permits were accompanied by
calculations on the quantity of radioactive materials released and the impact on the
maximally exposed individual at or beyond the site boundary. The maximum calculated
exposure from beta and gamma radiation dose from noble gases was less than 0.01%
of the limit set forth in PSEGs Offsite Dose Calculation Manual (ODCM) Section
3/4.11.2.2 for quarterly dose limit. The maximum calculated organ dose was 0.74% of
the quarterly limit and 1.02% of the annual limit for the thyroid set forth in ODCM section
3/4.11.2.3 for exposure from radio-iodines and particulates.
PSEG determined that the total amount of radioactivity released (about 9.2 Curies of
noble gas) consisted of both monitored and unmonitored releases (9.1 Curies was
monitored via the South Plant Vent, and an estimated 0.1 Curies was unmonitored).
PSEG determined this information by completing an analysis of the release data and
plant parameters to assess the radiological consequences. The time frame analyzed for
the total radiological release was from about 5:35 p.m. on October 10, 2004, to about
8:30 a.m., on October 11, 2004 (about 15 hours1.736111e-4 days <br />0.00417 hours <br />2.480159e-5 weeks <br />5.7075e-6 months <br />). Operators tripped the turbine about
two minutes after the scram (6:15 p.m. on October 10, 2004), which is when the source
of the steam was isolated. PSEG determined that for the same time analyzed (15
hours), a normal discharge of noble gas through the south plant vent (normal turbine
building release point) would have been about 4.9 Curies. Therefore, PSEG determined
that the event resulted in an additional 4.3 Curies of noble gas than would have been
released during normal plant operations (9.2 minus 4.9).
Enclosure
18
As part of PSEGs review, they also evaluated the radiological impact and conducted a
bounding assessment of a potential unmonitored release. This was done because there
was anecdotal evidence that the turbine building pressure went slightly positive for
several minutes shortly after the pipe failed (the turbine building pressure is normally
slightly negative relative to atmospheric pressure). For example, there were reports by
station personnel that there was condensation/water droplets outside the turbine
building, at building penetrations (doors and material joints). Notification No. 20207055
was written to document this condensation outside the turbine building. Although no
activity was detected in the water droplets, their presence tends to indicate that at some
time during the steam release within the turbine building, the building pressure was not
negative to atmosphere. PSEG bounded the time that the building may have been
slightly positive to be less than one hour (until the steam leak was isolated). In that
time, PSEG calculated the total noble gas quantity within the released steam to be 0.1
Curies.
The team reviewed PSEGs analyses, including assumptions and radiological data and
surveys. The team determined that PSEGs results and assessments of the radiological
consequences for this event were reasonable and appropriate. The team concluded
that the radiological consequences of this event were negligible, and there were no
findings identified in this area.
3.0 MOISTURE SEPARATOR/DRAIN TANK PIPING SYSTEM
3.1 Design, Operation, and Pipe Failure Details
Moisture Separators (MS A and B) each receive main steam after it passes through
the high pressure main turbine. They remove water from the steam before it reaches
the low pressure turbines. The water collects in the respective MS drain tanks. Each
drain tank has a control system to regulate level. As drain tank level increases, the
normal drain valve opens and allows the water to flow to the feedwater heaters. If level
in the drain tank increases further, the respective dump line valve (LV-1039A or LV-
1039B) opens and drains the water to the condenser. If level increases still further, a
turbine trip is initiated to prevent water from entering the steam line to the low pressure
turbines. The operating pressure of each MS is about 160 psig, while the main
condenser is maintained at a vacuum.
3.2 Historical Chronology - Design and Operational Details and Challenges
a. Inspection Scope
The team reviewed the design and operation of the MS and drain system to determine
how the design and operation may have affected the pipe failure. The team also
reviewed the history of design and operational challenges to determine if there were
prior opportunities to identify and correct the conditions that led to the event. A
chronology of some MS/drain system and level control valve issues is provided in
Attachment D.
Enclosure
19
The team reviewed the fabrication code requirements (ANSI B 31.1, 1973 edition with
addenda through winter 1974) to assess compliance with material, welding and
nondestructive test requirements applied during the design and fabrication of the MS
drain line and the attachment of an encapsulation device at the weld of the system pipe
to the condenser nozzle. The team reviewed the stress analysis calculations to assess
compliance with the fabrication code requirements, as originally designed and as
revised subsequent to the installation of the encapsulation in October 1988.
The team also evaluated PSEGs implementation of the flow accelerated corrosion
(FAC) program to determine if the material and operating parameters at the failed
location were considered for inclusion in the program; and whether this specific location
was in the program for periodic inspection.
The team evaluated PSEGs implementation of 10 CFR 50.65 (Maintenance Rule) as
related to the MS drain system (including the level control valve) in order to assess
whether ineffective system monitoring contributed to system performance deficiencies.
b. Findings
The MS drain line is non-safety related piping and is not required to be in the in-service
inspection program. Consequently, the welds in this line were not periodically
inspected. Also, the location of the failure was not in the FAC program because the
alloy steel material (11/4% chromium, 1/2% molybdenum) was not considered susceptible
to corrosion/erosion in single or two-phase flow.
Although PSEG determined that the construction materials and operating variables did
not meet the criteria for FAC program inclusion, technicians had performed some
examination of the piping between LV-1039A and the main condenser. This was done
on several prior occasions when LV-1039A experienced valve seat leakage. PSEG
examined the locations immediately downstream of this valve (as per FAC program
recommendations) using ultrasonic techniques to measure wall thickness. These
thickness measurements had been made during outages in 1997, 1999, and 2003. The
results of this testing confirmed there was no unidentified or unexpected material
corrosion. The wall thickness was verified to be within the acceptance criteria for the
specified pipe material size and schedule. The team reviewed these inspection results
and confirmed these conclusions.
A vacuum leak in October 1988 apparently resulted from line AC-8-GAD-032
movement (same line that failed on October 10, 2004), caused by two-phase flow in the
line as a result of LV-1039A being failed open (due to a disconnected air line). The
vacuum leak developed at the weld where the drain line passed through the condenser
penetration, and occurred after two days of operating with valve LV-1039A failed open.
With this valve open, the MS drain tank water drained into the condenser, and then
allowed steam to enter the line along with the water separated from the steam. The
two-phase flow apparently caused excessive line movement, which flexed the line at the
point where the line was welded to the main condenser penetration (anchor point). The
8-inch line passes through the 10-inch penetration. The leak occurred on the weld that
Enclosure
20
seals the metal donut in the area between the line and the penetration. Since the line
did not rupture and the crack was near the condenser, there was no steam leak during
the 1988 event. Rather, the leak allowed air intrusion between the line and the
penetration into the condenser resulting in a degraded condenser vacuum and elevated
offgas system flow. PSEG implemented modification DCR 4-HM-0494 to correct the
October 1988 vacuum leak by encapsulating the leak site with a welded encapsulation
ring. The modification package documented that the change increased the rigidity of
the penetration connection. The change also moved the effective flex point of the line to
the point where the encapsulation ring was welded to the pipe.
The October 10, 2004, A MS drain tank line failure resulted in the complete severance
of the pipe, approximately 12 - 18 inches from the A condenser shell penetration. The
pipe failure occurred in a seamless 8-inch Schedule 40 (0.322 inch wall thickness) of
ASTM A335 Grade P11 (11/4% chromium, 1/2% molybdenum) alloy steel material. The
break occurred at the toe of the weld, which had been made during the encapsulation
repair of the weld joint that failed in October 1988. The current failure, at the toe of the
weld attaching the encapsulating device to the process pipe, was approximately
21/2 inches upstream of the 1988 crack. There was evidence of some cavitation damage
on the drain pipe inside diameter close to the recent failure but, was not significant. The
inside diameter of the pipe did not exhibit a general wall thinning at any location.
The team determined there were prior opportunities to potentially prevent the October
10, 2004, pipe failure. Specifically, the 1988 encapsulation repair did not address the
root cause for the pipe crack, which appeared to be related to operating the drain line
outside its design. Rather, the encapsulation sealed the vacuum leak but moved the
flex point slightly upstream of the repair.
Another missed opportunity was when the encapsulation was installed in 1988. The
original scope of the controlling modification discussed a request to engineering to
evaluate the need for additional supports on the MS drain line. The associated
installation plan recommended vibration monitors on the piping. However, these
recommendations were not implemented, and represent missed opportunities to
determine whether vibration and line movement were acceptable for the piping
configuration and operation.
Vendor Instruction GEK-37949A, MS and Reheater Drain Systems, stated that the
check valve in the normal drain path (to the feedwater heaters) should be located close
to the branch point for the dump line to minimize the amount of saturated water
upstream of the check valve. The GEK-37949 instruction also stated that two-phase
flow anywhere in the lines upstream of the level control valves can produce pressure
pulsations and uncontrollable level oscillations in the drain tank; and that it was
important that only single-phase liquid exist upstream of the level control valves. The
team found that the location of the check valve associated with the B MS drain piping is
much closer to its branch point than the A MS system. Further, as can be seen in
Attachment D, there were significantly more problems associated with the A vs. the B
MS drain system. While PSEG had previously planned a detailed review of this
information to determine whether system modifications were necessary, no such review
Enclosure
21
was completed. While not directly related to this event, this was a missed opportunity to
improve operation of the A MS drain system.
There were additional historical challenges associated with the MS system. The
chronology in Attachment D identifies four prior reactor scrams (2002, 1998, and two in
1990), caused by turbine trips as a result of a high level in the A MS. Although not
directly related to the October 10, 2004, event, these problems related to overall
response of the MS drain and level control system.
The team determined that PSEG monitored the MS/drain system as required by the
Maintenance Rule. The MS/drain system is properly classified as a low risk system and
is monitored on the plant level; and the system and affected functions had been
monitored as required by (a)(2) of the rule. PSEG evaluated the 1998 and 2002 reactor
scrams to be preventable system functional failures. Neither of those scrams, nor the
combination of the two scrams, required the system to be placed in goal setting, as
required by (a)(1) of the rule. By the end of this inspection, however, PSEG was
evaluating the pipe failure event relative to the preventable system functional failure
perspective. These results will be considered with existing system performance data, as
well as plant level data. The results of this review may potentially require goal setting
for the MS/drain system. PSEG is tracking this evaluation and its results as Operation
4160 of Order 70041898.
3.3 Failure to Evaluate and Correct Degraded Condition
a. Inspection Scope
The team reviewed the details associated with the operation of the MS drain tank and
level control system. In particular, the team focused on Notifications 20203784 and
20204256, which documented the degraded condition of LV-1039A since September 16,
2004. The team reviewed procedures, PSEGs root cause evaluation, and interviewed
personnel in order to evaluate the circumstances and causes that led to the October 10,
2004, MS drain line failure.
b. Findings
Introduction. A self-revealing finding of low to moderate safety significance (Preliminary
White) was identified related to PSEGs failure to adequately evaluate and correct a
degraded condition since September 16, 2004, as required by station procedures. A
preliminary risk analysis determined the finding to be of low to moderate safety
significance based on the increased frequency of a transient with the loss of the power
conversion system initiating event over the 25-day exposure period.
Description. Notification 20203784 was written on September 16, 2004, which identified
that the MS low level alarm was received and the A MS dump valve, LV-1039A, was
noted on CRIDS (computer display) to be about 10% open while the associated valve
controller was receiving an air signal to fully close the valve. The team concluded that
this was the point in time where the valve had been opened for sufficient duration to
Enclosure
22
completely drain the A MS drain tank (valve open and MS low level alarm). A
condenser area entry was made on September 16 to investigate fittings associated with
the air supply line. Engineering and operations personnel discussed this issue, and
engineering responded formally on September 20, stating that there was not an
immediate safety concern.
However, an operator, not satisfied with the September 20 notification response,
initiated another notification (No. 20204256) that same day, stating that the prior
notification addressed only flow accelerated corrosion concerns. Specifically, it did not
address potential impact to the condenser/baffle plate, and the potential impact to the
condenser penetration which had cracked on an earlier occasion (1988) when this same
dump valve had failed open for an extended period of time (resulting in elevated offgas
flow due to increased in-leakage through the crack at the penetration to the condenser).
Again, a formal engineering response, completed on September 22, did not address the
entire concern. Only the first issue of potential internal condenser damage was
addressed, and the response re-stated the original flow accelerated corrosion response.
The responses to both notifications stated that the affected valve and associated piping
would be inspected during the upcoming refueling outage, scheduled to begin around
the end of October 2004.
Neither evaluation considered that two-phase flow could be present from the MS drain
tank (operating pressure - about 160 psig) to the main condenser (operating pressure -
vacuum conditions). The total length of piping from the MS drain tank to the condenser
is about 60 linear feet. This piping was not designed for the dynamic loading that would
accompany two-phase flow. The disconnected hanger (H25), while likewise unknown at
the time, was not available to mitigate the dynamic loading of the lines. The team
concluded that engineerings evaluations associated with the two notifications were
inadequate because the associated MWe reduction due to the leakage, the loss of
water level in MS A and the difference in operating pressures in the MS drain tank and
the main condenser, should have led to the recognition that there was two-phase flow in
the line upstream of LV-1039A.
After about 25 days (September 16 to October 10, 2004) of operation beyond the design
loading capacity of the MS drain tank piping, the 8-inch pipe failed near the condenser
penetration, resulting in a steam leak, manual reactor scram, and loss of condenser
vacuum.
Analysis. The performance deficiency involved the failure to perform an adequate
evaluation to correct the condition or cause of the deficiency as required by PSEGs
Corrective Action Program, NC.WM-AP.ZZ-0002(Q), Corrective Action Process.
Specifically, PSEG did not identify that the failure of LV-1039A caused two-phase flow in
the drain system which resulted in piping fatigue, and represented an increase in the
likelihood of a reactor scram with the loss of the condenser heat sink. This deficiency
was indicative of cross-cutting weaknesses in the area of problem identification and
resolution (evaluation and corrective action).
Enclosure
23
In addition, Engineering and Technical Support Information Communication Protocol
desk top guide, Section 4.4, required that when responding directly to a notification
without the use of an order (e.g., a request for follow-up assessment - RFA) peer and
supervisory review should be documented in the text provided. PSEGs root cause
analysis determined that there was no peer review of the response and the engineers
supervisor reviewed the response, but did not recognize that the response did not
address, as requested, the prior failure of this line under the same conditions. PSEGs
root cause analysis noted that the engineers supervisor and multiple engineers were
not aware of the written guidelines regarding use of the RFA process.
This issue was more than minor because it is associated with the Equipment
Performance attribute of the Initiating Events cornerstone and affected the objective of
limiting the likelihood of events that upset plant stability and challenge critical safety
functions. In accordance with NRC IMC 0609, Appendix A, Significance Determination
of Reactor Inspection Findings for At-Power Situations, the significance determination
process (SDP) Phase 1 required a Phase 2 risk evaluation because the finding
contributed to both the likelihood of a reactor trip and the likelihood that the condenser
heat sink (power conversion system) would not be available due to a loss of condenser
vacuum.
The Region I Senior Reactor Analyst (SRA) estimated the delta (increase) in core
damage frequency (CDF) and large early release frequency (LERF) for this finding
with a modified Phase 2 risk analysis, using the Risk Informed Inspection Notebook for
Hope Creek and IMC 0609 Appendix H, Containment Integrity Significance
Determination Process. The only affected attribute within the notebook was the
transient with the loss of the power conversion system (TPCS) initiating event (IE)
frequency. As such, the TPCS worksheet (Table 3.2) was modified and used for the
evaluation. This was an initiating event finding and therefore the impact of external
events was not evaluated.
The CDF estimation used Table 3.2 with the following assumptions and modifications:
- The exposure time was the 25 days that the valve was failed open.
- The frequency of the occurrence of the pipe break was unknown and the TPCS
IE frequency was increased by one order of magnitude, from a 1 in 100 chance
to a 1 in 10 chance, over the 25 days. (IMC 609 Appendix A, Phase 2 Usage
Rule 1.2)
- No condenser recovery credit was given.
- The credit (failure probability) for the high pressure injection (HPI) safety function
(HPCI and RCIC) was E-3 (1 failure in 1,000 tries) based on the most recent
NRC failure probability information.
Enclosure
24
The LERF estimation used the CDF core damage sequences and assessed them for
a BWR Mark I containment using the following assumptions:
- The early core damage accident sequences were 1TPCS, a low pressure
sequence and 4TPCS, a high pressure sequence. The conditional LERF factors
with a flood containment of 0.1 for a low pressure sequence and 0.6 for a high
pressure sequence were used. PSEGs emergency operating procedures
directed the operators to flood the containment using the fire water system prior
to reactor vessel breach.
- Accident sequence 2TPCS, a long term accident sequence that involves failure
of containment heat removal and ultimately progress to containment failure, was
assumed not to contribute to LERF. It is assumed that effective emergency
response actions can be taken within the long time frame of this accident
sequence.
The risk analysis determined that, given the increased frequency of a TPCS initiating
event over the 25-day exposure period, the finding had low to moderate safety
significance (White) based on CDF and LERF. The CDF, in the low E-6 per year
range (an increased frequency of approximately 1 core damage accident in 600,000
years of reactor operation), was dominated by failures of containment heat removal and
containment venting prior to containment failure (sequence 2TPCS). The LERF, in the
low E-7 per year range (an increased frequency of approximately 1 large early release
in 6,000,000 years of reactor operation), was dominated by two sequences: 1) the
failure of containment heat removal, successful depressurization and containment
venting, followed by a failure of late injection (sequence 1TPCS); and 2) a failure of high
pressure injection systems and failure to depressurize the reactor (sequence 4TPCS).
In discussions with the SRA, the PSEG risk analysis staff agreed with this risk
characterization based on the modified Phase 2 assessment.
Enforcement. There were no violations of NRC regulatory requirements because the
main turbine and extraction steam systems are not safety related. PSEG entered this
finding into their corrective action program as Notification 20206626 and Order
70041898. (FIN 05000354/2004013-04, Failure to Adequately Evaluate and Correct
a Failed Open Level Control Valve in the Moisture Separator Drain System)
4.0 RISK SIGNIFICANCE OF THE OCTOBER 10, 2004 EVENT
The team concluded that the event did not present any actual consequence to the
health and safety of the public because operators successfully shutdown the reactor in
response to the steam leak. The team conducted an initiating event risk assessment to
determine the chance of core damage during the event (conditional core damage
probability). This event risk assessment estimated a conditional core damage
probability in the low E-5 range (approximately 1 in 60,000) using the NRCs
standardized plant analysis risk (SPAR) model for Hope Creek, revision 3.10.
Enclosure
25
The following assumptions were used:
best modeled as a plant transient with loss of the condenser heat sink
(IE-OCHS);
- Given the pipe break and lowering condenser vacuum, the condenser heat sink
was not recoverable; and
- All mitigating systems other than the condenser heat sink and feedwater were
available during the event.
The dominant accident sequence involved core damage due to an assumed inability to
pump water to the reactor after a containment failure. The containment failure would be
caused by the inability to remove decay heat from the containment and failure to lower
containment pressure by venting. The sequence involves successful reactor scram and
high pressure injection (HPCI or RCIC) and reactor depressurization. PSEG also
performed an initiating event assessment for this event and reached a similar, though
slightly lower, conclusion relative to the CCDP (approximately 1 in 200,000 similar
events). The lower CCDP appeared to be due, at least in part, to an assumption in
PSEGs PRA model that, after containment failure, there is a chance that core damage
could be prevented by pumping water to the reactor.
5.0 EVENT ROOT CAUSES AND CAUSAL FACTORS
a. Inspection Scope
The inspection team reviewed PSEGs Root Cause Analysis Report for the MS Drain
Line Failure (Order 70041898). In addition, the team independently assessed the
October 10, 2004, event to determine causal factors and root causes. The team
reviewed data and documentation, and conducted personnel interviews.
The team reviewed the methodology used in the laboratory investigation of the failed
pipe and the detached MS drain tank pipe hanger (H25), and assessed the results of
those tests used to examine the failed parts. Those tests included sectioning of the
failed segments followed by visual, macroscopic and microscopic examination, chemical
analysis, mechanical testing and metallurgical assessments. The laboratory also
conducted an assessment of pipe hanger H25 to determine the possible causes of rod
disengagement from its upper eye nut. The team also reviewed the examination
activities and the assessments of the condition of the threaded rod, eye nut and
corrosion evaluation of the component parts.
b. Findings
During this review, the team assessed and verified that the identified root and
contributing causes were appropriate. PSEGs root cause analysis report associated
with the MS drain line failure identified two primary root causes. One was inadequate
decisions by engineering and management to continue operating the MS system with
the drain valve failed open; PSEG did not have a rigorous process to apply effective
Enclosure
26
decision-making principles to management and engineering decisions in response to
plant conditions that fall below licensing thresholds and/or are not clearly defined by
existing procedures. The second root cause was that operating procedures for the MS
level control system were inadequate to prevent extended operation of the system in the
condition of unstable two-phase flow.
PSEG also identified several contributing causes as listed below:
C The disconnected hanger (H25) was not discovered by any type of inspection,
thereby allowing it to fret through the instrument tray and tubing causing LV-
1039A to fail open;
C The condition of LV-1039A was not monitored to detect further degradation; and
C Appropriate rigor was not applied to the engineering evaluation of the abnormal
condition. Operators raised a concern about the prior pipe failure caused by
operating with LV-1039A open. Engineering did not research that failure and did
not address it in the response.
PSEG sent the failed pipe section and disconnected hanger (H25) for detailed
laboratory failure analysis to supplement their root cause investigation. The results of
the failure analyses are as follows.
C The primary crack in the 8-inch pipe was initiated by high cycle fatigue due to
system vibration. The crack propagated around the majority of the 8-inch pipe
circumference at or near the toe of the fillet weld by fatigue crack growth
combined with ductile tearing.
C There were no material or fabrication process deficiencies that contributed to the
pipe failure.
C Evidence suggested that the 1988 failure mode was fatigue. The existence of
the crack beneath the encapsulation resulted in a higher stress concentration on
the encapsulation to 8-inch pipe joint at the 2004 fracture location.
Laboratory observations regarding the Hanger H25 components were as follows:
C The upper threaded rod was engaged into the eye nut approximately 1/2 inch.
C Although the jam nut may have been in contact with the eye nut at one time, it
was found out of place at the bottom of the threads for a long period of time.
The team reviewed PSEGs root cause analysis report and concluded that the
evaluation was comprehensive and appropriately considered potential causes and
extent of condition for the steam pipe failure, including the problems encountered during
the event. The team determined that PSEG properly identified the causal factors and
root causes for the event. The team also evaluated the results of failure analysis
Enclosure
27
performed on the failed components, as well as PSEGs assessment of those results,
and concluded the identification of the failure mechanism as high cycle fatigue induced
by system vibration (due to the two-phase flow) was reasonable.
6.0 EXTENT OF CONDITION AND CORRECTIVE ACTIONS
6.1 Extent of Condition Review
a. Inspection Scope
The team reviewed PSEGs evaluation to determine whether they appropriately
considered and assessed the extent of condition. In particular, the team focused on
PSEGs review and inspection/examination of similar balance-of-plant (BOP) systems
and components with similar operating characteristics (including normal and off normal
conditions) to ensure they were evaluated for similar degradation. The team also
reviewed PSEGs extent of condition assessment related to non-safety related BOP pipe
hangers and supports, which had not been subject to a formal periodic inspection and
examination.
The team reviewed PSEGs criteria used to include systems and components for
examination as part of the extent of condition effort. The team reviewed the
examination methods (visual, magnetic particle and ultrasonic test), qualification of
examiners, acceptance criteria, and test results. In addition, team members performed
a walk-down of selected portions of the failed system and other BOP piping in locations
where similar piping penetrates the condenser. The team reviewed examination results
of all piping and nozzles inspected internal and external to the condenser.
b. Findings
PSEGs extent of condition evaluations considered both Hope Creek and Salem plants.
Piping
The selection of locations to be non-destructively examined was made to include all
piping that is connected to main condenser nozzles that have a potential for nozzle
and/or piping damage as a result of two-phase flow. This included piping to the
condenser associated with valves that leaked in the past. PSEG reviewed notifications
and work orders that included the following systems: main steam, condenser/feedwater,
extraction steam, and heater drains. As a result of this review, 14 additional condenser
penetrations were identified for inspection. The sample plan also included visual and
magnetic particle examination on all welds on the A and B MS drain lines.
Because the October 10, 2004, pipe failure occurred at the location of a pipe attachment
intended to contain a leak (1988 weld failure - encapsulation), PSEG performed a
search to identify any other encapsulation devices that may have been used to contain
leaks in BOP systems. The team noted that as a result of this examination effort, an
additional encapsulation was identified on the steam seal evaporator relief valve piping.
Enclosure
28
No indications were identified in the vicinity of this encapsulation, or at any other location
in the inspection sample. These lines were examined inside and outside the condenser.
During these inspections and examinations, some indications were identified, for which
PSEG removed/repaired them as necessary. Defects were removed by grinding as
necessary, and were verified as eliminated with the appropriate nondestructive
examination technique before repair activities were completed. Repairs were made to
within the original design specification requirements, and the weld repair locations were
non-destructively tested to verify weld soundness.
Pipe Hangers
PSEG formulated a plan to select and evaluate pipe hangers and supports in a large
sample of BOP steam and high-energy water systems at Hope Creek and Salem. The
selection was based on those systems with similar design, materials, operating
parameters, and were believed to potentially have been exposed to two-phase flow (but
designed for single phase flow). Also, systems were selected which similarly had a
known history of valve leakage, either periodically or continuously, where such operation
would be outside the piping design and potentially result in the application of unanalyzed
forces (static or cyclic) to system components. PSEG performed system and
component corrective action document searches, reviewed industry operating
experience, and conducted interviews to aid in selecting the inspection sample.
Field inspections encompassed validation of integrity of over 5000 hangers in Hope
Creek and Salem. Of this sample, 206 deficiencies were identified. Deficiencies were
identified in the following broad categories; bent rod or support, bottomed-out spring
can, loose support components, signs of excess vibration, or loose jam nuts on pipe
hangers. The majority of the deficiencies were screened as having negligible impact
because the discrepancies were minor and would not affect the function of the support.
These deficiencies were entered into the corrective action process. Of the 206
deficiencies identified, less than ten were considered more than minor and none were
evaluated as having an immediate impact on structural integrity of the associated
system.
The team found the extent of condition reviews to be acceptable both in scope and
detail. The original scope of the reviews were appropriately expanded as new
information became available. The team also reviewed the types of problems found
during the hanger walkdowns; none of the problems appeared to adversely affect the
piping or associated systems.
6.2 Corrective Actions
a. Inspection Scope
The team reviewed the immediate and longer term corrective actions that PSEG
proposed to address the deficiencies identified with this event. The team evaluated the
corrective actions for appropriateness and effectiveness in correcting the causes. The
Enclosure
29
team also reviewed the priority and schedule associated with the corrective actions to
ensure that items requiring resolution prior to plant startup were addressed. In addition,
the team reviewed a sample of the corrective actions that had been completed by the
end of this inspection.
b. Findings
The team determined that PSEGs proposed corrective actions were appropriate. The
team verified that the proposed corrective actions were properly aligned with the
identified root and contributing causes. Specifically, the team determined that the
proposed corrective actions appeared appropriate to address the event causes.
The corrective actions, some of which are described below, are listed according to the
various aspects of the event (engineering issues, pipe failure, disconnected pipe
hanger, equipment issues, and operator and training issues). The primary PSEG
corrective actions included:
Engineering Issues
- Established a formal process for Operational and Technical Decision Making
(OTDM) to apply effective decision-making principles to management and
technical decisions; and conduct training on the OTDM Process.
- Any open Hope Creek operability determinations that may be open at restart
were evaluated. All open Salem operability determinations will be re-evaluated.
- A review of degraded equipment at Hope Creek and Salem was performed,
selected items were re-assessed using the new OTDM Process.
- Improve system engineer walkdowns by developing system specific walkdown
plans and providing training on degradation mechanisms.
- Developed and implemented an Adverse Condition Monitoring procedure, to be
used in conjunction with the OTDM procedure.
- Director of Engineering to review the success of the actions taken to improve
engineering rigor and set expectations for review and documentation of
engineering replies to requests from various sources.
Pipe Failure
- Repaired the failed nozzle and minimize stress intensification factors.
- Revised plant operating procedures (and train operators) for the MS drain piping
to prohibit extended operation with the LV-1039A or LV-1039B valve open
coincident with loss of level in the MS drain tank. This action prevents operating
the drain lines with two-phase flow (outside their design).
Enclosure
30
- Remove the encapsulation found on the steam seal evaporator relief valve and
repair the existing defect.
- An evaluation will be performed to consider relocating the check valve in the A
MS drain piping (in the normal MS drain path - to the feedwater heaters) closer
to the MS drain tank, as per vendor guidance. Since the pipe failure was not
caused by the check valve configuration, this action does not have a direct
impact on the event causes. The team noted, however, that this effort may
improve the response of the A MS drain tank level during transient conditions.
- Dynamic modeling of the system will be performed to evaluate the loads during
normal system operation. Since the failure was not caused by normal operation
of the system, this action does not have a direct impact on the event causes.
However, the team noted that improved understanding of the response of the A
MS drain tank level during transients may provide useful operating insights.
- The feedwater heater operating procedures were revised to limit extended
operation with leaking dump valves or inadequate feedwater heater level control.
Disconnected Pipe Hanger
- Reinstalled H25 (replacing damaged components) in accordance with the
manufacturer installation recommendations; and repaired the instrument tubing
that was damaged by the disconnected H25 hanger.
- Addressed the pipe hanger discrepancies identified during the extent of condition
inspections. Also, about 30 Hope Creek pipe hangers were identified with jam
nut discrepancies. As found hanger rod thread engagement were checked when
the hangers were repaired.
- Establish a formal inspection program for pipe hangers in non-safety related
systems at Hope Creek and Salem. Formal inspection programs already existed
for safety related hangers.
Equipment Issues
C Replaced HPCI vacuum pump/motor and grease the pump using the proper
lubricant. Determined extent of condition associated with RCIC system.
C Changed the site computer lubrication screens for HPCI/RCIC vacuum pumps;
and evaluated the extent of condition of lubrication screen errors.
C Adjusted the limit switches for valve HV-8278. Evaluated additional completed
motor-operated valve diagnostic testing to confirm proper limit switch setting.
Enhanced procedures to provide clear acceptance criteria regarding limit switch
Enclosure
31
setting; and revised engineering procedures related to reviewing limit switch
settings to ensure data is effectively and properly reviewed.
Operator and Training Issues
C Revised HPCI/RCIC procedures to warn operators of potential flow instability.
Trained operators on this phenomenon and the revised procedures. Revised the
simulator to model the flow instability phenomenon for HPCI/RCIC.
C Provided operator training regarding TS usage with focus on the performance
deficiencies where operators misinterpreted TSs.
C Revised procedures to address operator challenges regarding reactor vessel
level control (e.g., provide guidance to allow changing level bands when
controlling pressure with SRVs)
C Implemented simulator upgrade (which was being developed prior to the
October 10, 2004, event), which provides improved modeling of balance of plant
systems.
The team reviewed PSEGs proposed corrective actions; and reviewed a sample of the
completed corrective actions, including the OTDM procedure, selected procedure
revisions, and training corrective actions. The team determined that PSEGs corrective
actions were appropriate to address the identified problems, and confirmed that
corrective actions necessary for plant restart were scheduled to be completed prior to
restart.
7.0 GENERIC ISSUES
During this inspection, no significant issues were identified requiring the issuance of
generic communications to the nuclear industry.
8.0 CROSS-CUTTING ASPECTS OF FINDINGS
Section 2.2.1 describes a finding that involved inadequate human performance
(personnel) as a primary cause. The finding also had a problem identification and
resolution (identification) cross-cutting aspect where engineering review of data did not
identify this performance deficiency.
Section 2.2.2 describes a finding where procedures for operating the RCIC system were
inadequate, and involved problem identification and resolution (identification) as an
underlying cause. The finding also involved a human performance (organization) cross-
cutting aspect where operating and vendor experience were not incorporated into
procedures and training.
Enclosure
32
Section 2.2.3 describes a finding regarding maintenance that involved human
performance (organization) as an underlying causal factor. The finding also involved a
problem identification and resolution (corrective action) cross-cutting aspect since there
was a prior opportunity to correct degraded performance.
Section 3.3 describes a finding where engineering staff did not properly evaluate and
correct a degraded level control valve for the A MS drain tank, and was indicative of a
cross-cutting weakness in the area of problem identification and resolution (evaluation
and corrective action).
9.0 EXIT MEETING SUMMARY
The NRC presented the results of this special inspection to Mr. A. Christopher Bakken,
III, and other members of PSEG management on January 12, 2005. Hope Creek
management acknowledged the findings presented. No proprietary information was
identified.
Enclosure
A-1
ATTACHMENT A
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel:
C. Bauer, Operations Superintendent
M. Bergman, System Engineer
R. Braddick, Operations Superintendent
W. Brammeier, ISI/IST Inspector
J. Clancy, Manager, Technical Support, Radiation Protection and Chemistry
M. Conroy, Senior Engineer, Engineering Programs
A. Garcia, Senior Engineer, Engineering Programs
J. Hutton, Hope Creek Plant Manager
A. Johnson, Supervisor, Civil Design Engineering
L. Koberlein, Nuclear Shift Supervisor
P. Lindsay, Engineering Supervisor, Design Engineering
T. Macewen, Nuclear Shift Supervisor
R. Montgomery, Senior Engineer (FAC Program)
J. Morrison, Engineering Supervisor
L. Rajkowski, Manager, Hope Creek System Engineering
T. Roberts, Supervisor, Engineering Programs
B. Sebastian, Technical Superintendent, Hope Creek Radiation Protection
G. Sosson, Hope Creek Operations Manager
H. Swartz, Simulator Support Group Supervisor
B. Thomas, Senior Licensing Engineer
W. Treston, ISI/IST Supervisor
R. Villar, Senior Licensing Engineer
J. Williams, Manager, Hope Creek System Engineering
LIST OF ITEMS OPENED, CLOSED AND DISCUSSED
Opened
05000354/2004013-04 FIN Failure to Adequately Evaluate and Correct a
Failed Open Level Control Valve in the Moisture
Separator Drain System (Section 3.3)
Opened and Closed
05000354/2004013-01 NCV Failure to Properly Set Limit and Torque Switches
on HPCI Valve in Accordance with Procedures
(Section 2.2.1)
Attachment
A-2
05000354/2004013-02 NCV Failure to Incorporate Operating Experience for
Low Flow Operations of RCIC Into Operating
Procedures and Operator Training (Section 2.2.2)05000354/2004013-03 FIN Failure to Effectively Implement Preventive
Maintenance for the HPCI Barometric Condenser
Vacuum Pump (Section 2.2.3)
LIST OF DOCUMENTS REVIEWED
Procedures
HC.OP-AB.CONT-0002(Q) Primary Containment, Rev. 2
HC.OP-AP.ZZ-0000(Q) Reactor Scram, Rev. 3
HC.OP-AR.ZZ-0014(Q) Overhead Annunciator Window Box D3, Rev. 17
HC.OP-DG.ZZ-0101 Hope Creek Post-Trip Data Collection Guidelines, Rev. 6
HC.OP-EO.ZZ-0101(Q) Reactor Pressure Vessel Control, Rev. 10
HC.OP-EO.ZZ-0102(Q) Containment Control, Rev. 11
HC.OP-FT.AC-0005(Q) Turbine Overspeed Protection System Operability Test, Rev. 4
HC.OP-SO.AF-0001(Z) Extraction Steam, Heater Vents and Drains System Operation,
Rev. 23
HC.OP-SO.BD-0001(Q) RCIC System Operation, Rev. 27
HC.OP-SO.BJ-0001(Q) HPCI System Operation, Rev. 26
HC.OP-SO.SM-0001(Q) Isolation Systems Operation, Rev. 13
NC.CA-DG.ZZ-0102 Operational and Technical Decision Making Process Desk Guide,
Rev. 0
NC.ER-DG.ZZ-0011(Z) System Walkdown Guideline, Rev. 1
NC.WM-AP.ZZ-0002(Q) Corrective Action Process, Rev. 8
SE.MR.HC.01 Maintenance Rule System Function and Risk Significance Guide,
Rev. 10
SH.OP-AP.ZZ-0108(Q) Operability Assessment and Equipment Control Program, Rev. 15
SH-MD-EU-ZZ-0011(Q) VOTES Data Acquisition for Motor Operated Valves, Rev. 8
Drawings
01-1 Main Steam, Rev. 0
02-1 Extraction Steam, Rev. 24
03-1 Vents and Drains Heaters 1 and 2, Rev. 19
04-1 Vents and Drains Heaters 3, 4, 5 and 6, Rev. 19
1-P-AC-02 System Isometric/Turbine Bldg. MS A Drain to Feedwater Heater 5A, 5B & 5C
and Condenser A, Rev. 17
1-P-AF-05 System Isometric/Turbine Bldg. Feedwater Heater Drains, Heaters 3A, 3B, 3C,
4A, 4B, 4C & 5A, 5B, 5C, Rev. 14
1-P-AF-06 System Isometric/Turbine Bldg. Feedwater Heater Dumps Heaters 3A, B & C,
Rev. 18
1-P-CA-05 System Isometric/Steam Seal Evaporator Relief Valve Piping, Rev. 14
Attachment
A-3
10855-P-0500 Piping Class Sheet - Class GAD, Rev. 5
PM4-0037(01) Turbine Condenser Connection Listings (General Arrangement), Rev. 6
PM4-0037(02) Turbine Condenser Connection Listings, Rev. 20
PM4-0037(03) Turbine Condenser Connection Listings (Encapsulation Detail), Rev. 5
PSEG Control Valve Data Sheet (1ACLV-1039A and 1ACLV-1039B), Rev. 0
Calculations
4M-Z-02503 Change to Stress Calc C-1063 Steam Seal Evaporator Relief Valve Line
C-1045 Turbine Building MS A Drain to Feedwater Heater 5A, 5B and 5C - Pipe
Stress Calculation, Rev. 12
H-1-ZZ-MDC-1932 Moisture Separator Drain Downstream of 1039A, RFO 11, FAC Exam
H-1-ZZ-MDC-1792 Moisture Separator Drain Downstream of 1039A, RFO 8, FAC Exam
H-1-ZZ-MDC-1754 Moisture Separator Drain Downstream of 1039A, RFO 7, FAC Exam
SC-0263 R0 Moisture Separator Drain (Pipe) of 1039A, RFO 11, FAC Exam
Modifications (Design Change Packages - DCP)
4HM-0494 Encapsulate Vacuum Leak at Condenser A, Penetration #56, Rev. 0
4M-Z-02503 Addition of Encapsulation on Condenser Nozzle #29
80051144 Install Quick Exhaust Valves on MS Dump Valves, Rev. 0 and Rev. 1
80075423 Restore HC Main Condenser Nozzle 56 to Original Design, Rev. 0
Work Orders
30087552 60025008 60048663
30087588 60029413 60048697
30087640 60048662 70020654
Notifications
20080033 20206631 20206821 20206946
20081426 20206632 20206848 20206978
20084783 20206633 20206849 20207019
20103635 20206634 20206851 20207038
20140081 20206635 20206880 20207049
20180818 20206665 20206885 20207055
20186175 20206668 20206888 20207067
20203784 20206669 20206889 20207288
20204256 20206766 20206908 20207691
20206604 20206772 20206921 20211633
20206606 20206783 20206926 20212885
20206626 20206801 20206931 20214655
20206627 20206806 20206943 20214768
Attachment
A-4
Examination Test Reports
20206851 RFO12 UT Thickness Examination Record, 1039B downstream area
20206851 RFO12 MT Exam of Encapsulation and two upstream welds to valve 1039B
50045145 RFO11 UT Thickness Examination Record, 1-AC-032-S12-V1
60048662 RFO12 MT Exam of FW 56P, 65
60048697 RFO12 Surface Examination Record of HC-1-AF-135-FW9-R1
60048698 RFO12 MT Exam of nozzle weld downstream of 1505C
60048699 RFO12 MT Exam of nozzle weld downstream of 1531B
60048700 RFO12 MT Exam of nozzle weld downstream of 1531C
60048725 RFO12 MT Exam of nozzle weld downstream of 1521C
60048726 RFO12 MT Exam of nozzle weld downstream of 1521A
60048727 RFO12 MT Exam of nozzle weld downstream of 1521B
60048728 RFO12 MT Exam of nozzle weld downstream of 1513A
60048729 RFO12 MT Exam of nozzle weld downstream of 1513B
60048730 RFO12 MT Exam of nozzle weld downstream of 1513C
60048731 RFO12 MT Exam of nozzle weld downstream of 1505A
60048732 RFO12 MT Exam of nozzle weld downstream of 1505B
60048744 RFO12 MT Exam of nozzle weld downstream of 1994E
980501020 RFO7 UT Thickness Examination Record, 1-AC-032-S12-V1
991109003 RFO8 UT Thickness Examination Record, 1-AC-032-S12-V1
Miscellaneous
Root Cause Analysis Report, Hope Creek Moisture Separator Drain Line Failure (70041898)
Root Cause Investigation Report, Technical Specification and LCO Management (70041900)
Root Cause Investigation Report, Reactor Vessel Water Level Control Difficulties (70041930)
GEK 37949A MS/Reheater Drain Systems, GE Industrial/Power Systems, Rev. A, June 1977
OE17818 Operating Experience (Steam Leak on Drain to Condenser)
Gaseous Radioactive Waste Release Permits: 200788.017.063.G and 200784.017.060.G
Radiation Protection Shift Log, October 10 & 11, 2004
Station Operation Review Minutes, Meeting No. H2004-023, October 15, 2004
System Health Report, HPCI Sytem, 3rd Quarter, 2004
System Health Report, RCIC Sytem, 3rd Quarter, 2004
System Health Report, Main Turbine and Auxiliary Systems, 3rd Quarter, 2004
Licensee Event Report 50-354/90-001-00
Licensee Event Report 50-354/90-028-01
Licensee Event Report 50-354/98-008-00
Licensee Event Report 50-354/02-004-00
Attachment
A-5
NOH0100ELERS-00 Nuclear Training Center Lesson Plan, Operating Experience, 7/1/02
NOH051004TSJ-00 Nuclear Training Center Lesson Plan, Just-in-Time Training, 11/24/04
SG-600, Simulator Scenario Guide, 10/10/04 Steam Leak Demonstration, 12/8/04
LIST OF ACRONYMS
ANSI American National Standards Institute
ASTM American Society for Testing and Materials
BOP Balance of Plant
BWR Boiling Water Reactor
CCDP Conditional Core Damage Probability
CDF Core Damage Frequency
CIV Combined Intermediate Valve
CRD Control Rod Drive
CRIDS Control Room Indication Display System
CRS Control Room Supervisor
CST Condensate Storage Tank
CDF Delta Core Damage Frequency
LERF Delta Large Early Release Frequency
ECG Emergency Classification Guide
EDG Emergency Diesel Generator
EOP Emergency Operating Procedure
FAC Flow Accelerated Corrosion
FIN Finding
GPM Gallons per Minute
HPCI High Pressure Coolant Injection
IE Initiating Event
ISI/IST In-service Testing/In-service Inspection
IMC Inspection Manual Chapter
LCO Limiting Condition of Operation
LER Licensee Event Report
LERF Large Early Release Frequency
LPCI Low Pressure Injection System
MR Maintenance Rule
MS Moisture Separator
MSIV Main Steam Isolation Valve
MT Magnetic Particle Test
MWe MegaWatt - Electric
NCV Non-Cited Violation
NRC Nuclear Regulatory Commission
ODCM Offsite Dose Calculation Manual
OE Operating Experience
OTDM Operational and Technical Decision Making
PRA Probabilistic Risk Assessment
PSEG Public Service Enterprise Group
RCIC Reactor Core Isolation Cooling
Attachment
A-6
RFA Request for Follow-up Assessment
RFO Refueling Outage
RFP Reactor Feed Pump
RG Regulatory Guide
SDP Significance Determination Process
SM Shift Manager
SPAR Standardized Plant Analysis Risk
SPDS Safety Parameter Display System
SRA Senior Reactor Analyst
SRO Senior Reactor Operator
TPCS Transient - Power Conversion System
TS Technical Specification
TSAS Technical Specification Action Statement
UT Ultrasonic Test
VOTES Valve Operation Test and Evaluation System
UE Unusual Event
UFSAR Updated Final Safety Analysis Report
Attachment
B-1
ATTACHMENT B
SPECIAL INSPECTION TEAM CHARTER
October 13, 2004
MEMORANDUM TO: Raymond K. Lorson, Manager
Special Team Inspection
Stephen M. Pindale, Leader
Special Team Inspection
FROM: Wayne D. Lanning, Director /RA/
Division of Reactor Safety
SUBJECT: SPECIAL TEAM INSPECTION CHARTER -
HOPE CREEK NUCLEAR GENERATING STATION
A special inspection has been established to inspect and assess an event that occurred on
October 10, 2004, at the Hope Creek Nuclear Generating Station. At 6:14 p.m., the plant was
manually scrammed due to the failure of an 8-inch moisture separator drain line. Following the
shutdown, there were a number of equipment and operator performance issues. The special
inspection will commence on October 14, 2004, and will include:
Manager: Raymond K. Lorson, Chief, Performance Evaluation Branch
Leader: Stephen M. Pindale, Senior Reactor Inspector
Full Time Members: Steven Dennis, Senior Reactor Engineer
Thomas F. Burns, Reactor Inspector
Joel S. Wiebe, Reactor Inspector
Edward C. Knutson, Resident Inspector
Part Time Members: Nancy T. McNamara, EP Specialist
Wayne L. Schmidt, Senior Reactor Analyst
Robert Davis, Materials Engineer, NRR
This special team inspection was initiated in accordance with NRC Management Directive 8.3,
NRC Incident Investigation Program. The decision to perform this special team inspection
was based on deterministic criteria in Management Directive 8.3 and the initial risk assessment.
Specifically, the condition involved possible adverse generic implications, and involved
questions or concerns pertaining to licensee operational performance. The initial risk
Attachment
B-2
assessment characterized the conditional core damage probability to be approximately 1 in
200,000, which is in the range for a special inspection.
The inspection will be performed in accordance with the guidance of NRC Inspection Procedure
93812, Special Inspection, and the inspection report will be issued within 45 days following the
exit meeting for the inspection. If you have any questions regarding the objectives of the
attached charter, please contact Ray K. Lorson at 610-337-5282.
Attachment: Special Inspection Charter
Distribution:
E. Cobey, DRP
M. Gray, DRP
S. Barber, DRP
W. Schmidt, DRS
W. Lanning, DRS
R. Crlenjak, DRS
R. Blough, DRP
B. Holian, DRP
D. Screnci, ORA
Attachment
B-3
Special Inspection Charter
Hope Creek Nuclear Generating Station
Steam Leak Due to a Rupture of the Hope Creek A Moisture Separator (MS) Drain Line
Preliminary information regarding the event: On October 10, 2004, at approximately 6:00 p.m.,
operators lowered reactor power in response to an offgas flow increase and a turbine building
ventilation exhaust radiation monitor alarm due to a reported steam leak in the turbine building.
At 6:14 p.m., because offgas flow continued to increase and steam was noted in other areas of
the turbine building, operators initiated a manual reactor scram to isolate the steam leak, and
began a cool down and depressurization to stabilize plant conditions. At 6:22 p.m., in
anticipation of a loss of normal heat sink due to slowly degrading main condenser vacuum,
operators placed the reactor core isolation cooling (RCIC) and high pressure coolant injection
(HPCI) systems in operation for reactor level and pressure control. At 6:28 p.m., operators
closed all of the main steam isolation valves (MSIVs) prior to their automatic closure on low
condenser vacuum. During the cooldown, reactor water level cycled high and low repeatedly in
response to various plant conditions including operators initiation of RCIC and HPCI, closure of
MSIVs, and opening safety relief valves for pressure control. The cooldown and
depressurization continued and operators stabilized the plant in Hot Shutdown at 10:11 p.m. on
October 10. The plant was placed in cold shutdown at 5:09 a.m., on October 12.
The source of the steam leak was a rupture of the 8-inch A MS drain line to the main
condenser. This rupture also caused a decrease in condenser vacuum which complicated post
trip reactor water level and pressure control. There were no injuries associated with this event.
There was a minor radiation release from the plant that was below federally approved operating
limits. The release was monitored by the turbine building exhaust and south plant ventilation
stack radiation monitors.
Objectives of the Special Inspection: The objectives of the special inspection are to evaluate
the circumstances associated with the event described above. Specifically the inspection
should accomplish the following.
a. Develop a detailed event chronology, including key transition (e.g., MSIV closure, RCIC
& HPCI initiation, Mode changes, etc.) and operator decision points.
b. Independently evaluate the equipment and human performance issues that complicated
the response to this event to assess the adequacy of Hope Creeks investigation and
root cause evaluation with respect to the identification of performance deficiencies,
extent of condition review, assessment of potential common mode failures, root
cause(s), and corrective actions.
C Assess operator control of the plant during the event including abnormal and
emergency operating procedure usage, the bases for decisions made, and
actions taken. Evaluate the causes for, and the significance of, various level
changes that occurred during the event.
Attachment
B-4
C Assess whether Hope Creeks investigation appropriately considered operator
training issues and effectiveness.
C Assess the adequacy of Hope Creeks plans for corrective actions for the
equipment and human performance issues.
c. Evaluate the adequacy of Hope Creeks analysis of the cause(s) for the steam system
piping failure, extent of condition and actions to prevent recurrence.
d. Verify that radiological releases were monitored and did not exceed regulatory
requirements.
e. Determine whether prior opportunities were available to identify and correct the
conditions that led to the steam pipe failure.
f. Review operator compliance with Technical Specifications, Emergency Action Levels,
and the Emergency Plan.
g. Independently determine the risk significance of the event.
h. Document the inspection findings and conclusions in a special inspection report in
accordance with Inspection Procedure 93812 within 45 days of the exit meeting for the
inspection.
Attachment
C-1
ATTACHMENT C
SEQUENCE OF EVENTS
Entries that appear in italics are notes or observations made by the NRC inspection team. All
other entries were obtained from various licensee sources, and the specific sources are noted
in parentheses, ( ). Refer to the end of this attachment for the source codes.
Initial Plant Conditions (Pre-Event) - 100% Reactor Power
Time Event
[October 10, 2004]
17:39 Received annunciator Offgas Recombiner Panel 00C327" in the control room.
This was the operators first indication of the developing steam leak from the A
moisture separator (MS) drain line. (1)
17:41 Turbine Building Exhaust Radiation Monitor in alarm at Alert level and rising.
This was due to steam in the turbine building. (1)
17:50 Operator in the plant reports that steam is evident on the 137' turbine building
elevation (turbine deck) in the vicinity of the front standard of the main turbine.
(1)
17:51 Radwaste operator investigating the offgas recombiner panel alarm at 17:39
reports that the alarm is due to elevated offgas flow at 63 cfm (normal offgas
flow is about 25 cfm) and rising. B steam jet air ejector backpressure is 5 psig
and rising. (1)
17:53 Received annunciator Fire Protection Panel 10C671 associated with room
1704/1705 (elevation 171' turbine building). This was also due to steam in the
turbine building. (1)
17:59 Operators commenced power reduction with recirc flow. (1)
Offgas flow greater than 115 cfm, B steam jet air ejector backpressure is
pegged high. (1)
Main condenser vacuum has yet to be significantly affected and remains steady
at 3.2" Hg. (1), (2)
18:04 Power has been reduced to just under 80 percent with recirc flow, main
condenser vacuum has improved to 2.6" Hg due to the power reduction. (1)
Attachment
C-2
18:05 Operator in the plant observes that steam leakage from a previously existing
leak on the 6A feedwater heater extraction steam line appears to have gotten
worse. Operators initially believed this was the source of the current steam leak
in the turbine building. (1)
18:06 Operators closed the 6A feedwater heater extraction steam isolation valve in an
attempt to isolate the steam leak. (1)
18:10 Operator in the plant reports that steam leakage in the turbine building is
continuing to worsen. (1)
Operators resumed power reduction with recirc flow. (1)
18:12:35 Based on the inability to isolate the steam leak and worsening conditions in the
turbine building, the control room supervisor directs a manual reactor scram.
Operators insert the scram (from about 69% power) by placing the reactor mode
switch in Shutdown.
18:12:40 Received reactor low water level scram due to level shrink following the scram
(expected condition). (4)
18:12:45 Operators secured the B reactor feedwater pump (expected operator action to
secure one of the operating reactor feedwater pumps following a scram to
prevent overfeeding the reactor vessel. A and C feedwater pumps remain in
service). (4)
18:12:57 Received scram discharge volume level high scram alarm (expected condition).
(4)
18:13:15 Reactor vessel water level reaches a post-scram transient low point of -14
inches. (6)
18:13:30 Operators trip the main turbine (expected operator action following a scram),
received turbine stop and control valve scrams (expected condition). (4)
Using the rod worth minimizer and SPDS, operators verified that all control rods
are fully inserted. (1), (3)
18:15 Condenser backpressure beginning to rise, indicates 6" Hg. Operators initiate
reactor pressure reduction to less than 700 psig using the bypass valves, to be
within the capability of the secondary condensate pumps, and to reduce the
driving force on the steam leak. (1), (3)
18:16 The operating RWCU system pump (B) trips. Operators attribute this to the
rapid reactor pressure reduction (an expected possibility based on plant
operating experience, but not due to a system design function). (1), (3)
Attachment
C-3
Note that, as a result, the RWCU system is no longer available for vessel
inventory reduction. Although this capability is desirable, its loss did not result in
any significant operational consequences during the remainder of the event.
18:17:12 A and C reactor feedwater pumps trip automatically due to low condenser
vacuum (turbine driven pumps), reactor pressure is 705 psig (still beyond the
capability of the secondary condensate pumps). (1), (4)
Note that CRD (a relatively low volume source) is the only input to reactor vessel
water inventory.
18:17:43 Bypass valves peak open at about 47 percent. (6)
18:17:55 Reactor vessel water level has risen (due to swell caused by the bypass valves
opening) to a high point of 30 inches. (6)
18:18 Based on rates of pressure and level change, the shift technical advisor
determined that there will not be enough vessel water inventory to depressurize
to within the capability of the secondary condensate pumps before reaching
Level 2 (-38 inches). The control room supervisor directed closing the bypass
valves to maintain inventory (four bypass valves are currently open), and
directed startup of HPCI and RCIC. (1), (3)
Note that bypass valve closure is not instantaneous, but rather occurs
sequentially.
Reactor vessel water level begins to drop rapidly due to shrink from closing the
bypass valves and inventory depletion. (6)
18:19:14 Operators initiate RCIC using the manual initiation switch. This initiates the
system startup sequence (the pump is not yet injecting). (4)
18:19:27 Reactor vessel water level reaches a transient low point of -38 inches (Level 2).
(6)
The level 2 condition was caused by low initial vessel water inventory, loss of the
reactor feedwater pumps, continuing inventory use while the bypass valves were
closing, and level shrink due to closure of the bypass valves. (1)
18:19:39 RCIC injection peaks at about 650 gpm, then stabilizes at about 600 gpm. (6)
HPCI injection peaks at about 6000 gpm. Operators take action to secure HPCI
flow to the reactor, to prevent vessel overfill (since the Level 2 condition was, in
part, due to shrink, level would recover to some extent on its own following the
transient; and, RCIC was already providing adequate makeup). (6), (10)
18:19:43 All bypass valves are closed (they were sequentially closing since 18:18). (4)
Attachment
C-4
18:19:50 Water level has risen to -20 inches and then begins to recover more slowly. (6)
18:20:12 HPCI injection flow to reactor vessel secured (system in idle mode - minimum
recirculation flow). (6)
18:20 - 18:30 Operators are switching HPCI to pressure control mode. In this mode of
operation, the HPCI pump recirculates water to the condensate storage tank (no
injection to the reactor vessel) and the amount of flow determines the amount of
steam that is drawn from the reactor to power the pump. This provides greater
control of steam demand than use of the SRVs, and reduces the energy content
of the steam being discharged to the torus. (1), (3)
When operators attempted to reposition the HPCI full flow test line control valve
(F008), the valve did not open. The valve is interlocked with two other system
valves in the injection flow path, to prevent inadvertent diversion of flow from the
reactor. After the two other valves were given an additional closed signal (they
were already closed), operators were able to open F008.
18:22:34 Recognizing that MSIV closure due to degrading main condenser vacuum was
imminent, operators start to reopen the bypass valves (to reject as much energy
to the main condenser as possible). (4), (10)
18:24:35 Bypass valves peak at about 14 percent, then begin to close. (6)
18:25:10 Main condenser vacuum has dropped to 20" Hg, operators closed the inboard
and outboard MSIVs. (1), (4)
18:27 Operators placed the RHR system in torus cooling mode using A and B RHR
pumps. (1), (4)
18:30:23 Operators commence operation of HPCI in pressure control mode, recirculation
flow to the CST is increasing slowly. (6)
18:32:07 Reactor vessel water level is now 12.5 inches (Level 3) by wide range indication
and increasing. (6)
18:33:41 Operators decrease RCIC flow to stabilize reactor vessel water level. (6)
18:35:06 At about 350 gpm, RCIC flow begins to oscillate (dropping to nearly zero and
then recovering). This was due to system instability when using the automatic
flow controller at low flow. This condition is known to the vendor, who
recommends use of the manual flow controller at low flow conditions, but was
not known to the operators, nor was it discussed in the system operating
procedure. As a result, operators continued to operate the system in automatic,
and believed that there may be a problem with the RCIC system. (6)
18:36:45 Operators secured RCIC injection. (6)
Attachment
C-5
18:38 Reactor vessel water level stable at about 30 inches. (6)
18:41 Water level begins to lower slowly. (6)
18:43 HPCI recirculation flow to the CST has slowly been increased to 5500 gpm and
is now stable. (6)
18:45 Operators commence RCIC injection, with flow oscillations occurring at low flow
(five oscillations over the next three minutes). (6)
18:45:29 Operators consider that plant conditions have adequately stabilized and reset the
scram. (4)
Note that resetting the scram is desirable because it stops CRD system flow to
the reactor vessel and reduces thermal fatigue to the CRD vessel nozzles;
resetting the scram prior to achieving stable plant conditions is not desirable
because transient conditions could produce another automatic scram.
18:46:53 Received reactor low water level scram. The low level condition was due to
RCIC injection not commencing at the onset of the HPCI control mode transition
due to concern for the unexpected RCIC flow oscillations, greater than
anticipated vessel water inventory use in transitioning HPCI to pressure control
mode, and loss of the vessel water inventory contribution from the CRD system
due to resetting the scram. (1), (3), (4)
18:48 RCIC flow oscillations stop, operators gradually increase flow to 600 gpm. (6)
18:53:05 Reactor low water level scram signal clear. (4)
18:57 Commenced torus level reduction to radwaste. (1)
This is noted here because the ongoing discharge of HPCI/RCIC turbine exhaust
steam (and later, steam from the SRVs) to the torus introduces short-lived (half-
life on the order of hours) radionuclides (in the form of dissolved gases) into the
torus water. After discharge to the radwaste system, these gases are released
to the atmosphere and result in a slight (but measurable) increase in the activity
being released from the plant through the plant ventilation system. This is an
expected and previously analyzed condition.
18:58:14 Operators secured injecting with RCIC and secured the system. Reactor vessel
level control is maintained with the secondary condensate pumps. (7), (10)
19:00 For the next two hours, the plant is in a relatively stable slow cooldown,
maintaining vessel level at about 30 inches with the secondary condensate
pumps, and with HPCI operating in pressure control mode at 4000-5000 gpm
flow. Pressure decreases from about 650 psig to about 550 psig. (7)
Attachment
C-6
19:06 Operators reset the reactor scram. (1)
19:10 An RP technician noted water condensing and falling outside of the
Administration Building and Turbine Building. The apparent source was steam
that had escaped from the turbine building. This was due to pressure in the
turbine building exceeding outside atmospheric pressure for a short period, as a
result of the steam leak, which caused steam to leak through building
penetrations (doors, construction material joints, etc.). (5)
Note that this constitutes an unmonitored radioactive release to the environment,
the magnitude of which cannot be precisely quantified. However, based on
actual radiation monitoring system measurements and the decrease in plant
water inventory over the course of the event, the release is conservatively
estimated to have been less than one percent of the regulatory limit. Note also
that the design basis steam line failure in the turbine building (far more severe
than this event) has been previously analyzed and shown to result in an
environmental release that would be within the limits of 10 CFR 100.
20:15 Secured torus level reduction to radwaste. (10)
21:05 HPCI alarm received in the control room, Overload/Power Fail. (1)
Operators placed RCIC in service in pressure control mode in anticipation of
HPCI being secured. (1), (7), (10)
21:12 Cause of the HPCI alarm determined to be that the thermal overloads for the
HPCI barometric condenser vacuum pump had tripped. Operators reset the
thermal overloads and restarted the vacuum pump. (1), (10)
21:17 Operators removed RCIC from service. (1), (8)
21:27 HPCI barometric condenser vacuum pump thermal overloads tripped again. (1)
Operators again placed RCIC in service in pressure control mode. (1), (8)
21:31 Operators removed HPCI from service. (1), (8)
HPCI remains operable and available for use, if necessary; operators elect to
secure it because continued operation without the barometric condenser vacuum
pump would result in steam leakage from the turbine which, in turn, would result
in radioactive contamination of the HPCI pump room.
21:31 - 21:38 Decay heat is beyond the capacity of RCIC in pressure control mode. As a
result, reactor vessel water level is increasing due to inventory heatup. (10)
21:38 Wide range reactor water level reached 54" (level 8) and continuing to rise, RCIC
automatically tripped. (1), (3), (8)
Attachment
C-7
21:48 Commenced use of SRVs (one at a time) to augment RCIC pressure control.
Opening an SRV initially produces a rapid increase in vessel water level due to
swell, then a rapid decrease due to inventory reduction, and finally, a further
decrease due to shrink upon closure. (1), (8)
Note that, due to the change in water density as temperature is lowered, the
reactor vessel wide and narrow range water level instruments diverge at less
than normal operating pressure. For example, at the current reactor pressure of
450-500 psig, a narrow range level of 48 inches corresponds to a wide range
level of about 60 inches. The narrow range level instrument provides the low
water level scram signal at 12.5 inches (Level 3), and the wide range level
instrument provides the high level HPCI/RCIC automatic shutdown signal at 54
inches (Level 8). As a result, controlling water level between these two trip
setpoints presents an operational challenge.
21:54 Wide range reactor water level is at 54 inches and decreasing. (8)
21:57 Operators closed the first SRV at about 25 inches (narrow range); subsequent
shrink caused reactor water level to decrease below 12.5" (Level 3), received
reactor low water level scram. (1), (3), (8)
Operators returned RCIC to service in pressure control mode. (1), (8), (10)
22:04 RCIC automatically tripped due to reactor water level again reaching level 8. (1)
Note that RCIC is not operated again for the remainder of the plant cooldown.
As a result, vessel level exceeding 54 inches (wide range) no longer presents an
operational limitation. Pressure control is by the SRVs.
October 11, 2004
00:31 Completed the last of 10 SRV manual cyclings that had commenced at 21:48.
(9)
02:33 Manually opened one SRV; it will remain open for about 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. (9)
14:52 Operators placed RWCU in service. (1)
October 12, 2004
01:13 Commenced operation of RHR in shutdown cooling mode. (1)
02:11 Closed the SRV that had been opened about 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> earlier. (9)
05:09 The reactor is in Cold Shutdown. (1)
Attachment
C-8
19:45 After-the-Fact report made to the NRC for not meeting TS 3.6.2.3 Action b. to
have plant in Cold Shutdown within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of reaching Hot Shutdown.
Source Codes
1. Control Room Narrative Log. Note: The time of the scram in the log is 18:14, and by
the sequence of events printout is 18:12:35. Subsequent events recorded in both do not
indicate a consistent offset. Therefore, the team used sequence of events printout
times beginning with the scram, and control room narrative log times may have been
changed to be consistent with the sequence of events.
2. Simulator Instructor interviews
3. Operator interviews
4. Sequence of Events printout or plant computer printout
5. Notification 20207055
6. 45 minute plant computer graphs (6:05 - 6:50 p.m.) of the following:
a. Reactor water level
b. HPCI/RCIC pump discharge flow
c. Main turbine bypass valve position
d. Indication for reactor scram, MSIV not full open scram, and RFP Turbine low
control oil pressure
7. Five hour plant computer graphs (6:00 - 11:00 p.m.) of the following:
a. Reactor pressure
b. Reactor water level
c. HPCI/RCIC pump discharge flow
8. 1.5 hour5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> plant computer graphs (9:10 - 10:40 p.m.) of the following:
a. Reactor narrow range level channel A
b. Reactor wide range level channel B
c. HPCI/RCIC pump discharge flow
d. Indication for the first five SRV openings
9. PSEGs Post Reactor Scram Review
10. Root Cause Evaluation, Reactor Vessel Water Level Control Difficulties, and interview
with a senior reactor operator.
Attachment
D-1
ATTACHMENT D
MOISTURE SEPARATOR / DRAIN SYSTEM CHRONOLOGY
Information notes and relevant NRC team assessments notes are shown in italics.
Date Event
10/13/1988 Sometime prior to October 13, 1988: LV-1039A failed open. Within two days, a
vacuum leak due to a pipe crack developed on the downstream piping (line AC-
8-GAD-032) near penetration 56 into the condenser. The in-leakage was
sufficient to cause concerns about continued operation of the plant. Design
modification (DCR) 4-HM-0494 was initiated to correct the vacuum leakage by
installing (welding) a mechanical encapsulation at the circumferential crack. LV-
1039A was repaired. {Source: DCR 4-HM-0494} This was a missed opportunity
to identify consequences of operating with valve LV-1039A in the failed open
position.
01/06/1990 MS High Level, Turbine Trip, Reactor Scram: During main turbine combined
intermediate valve (CIV) testing, the A MS experienced a high level condition.
Dump Valve LV-1039A opened, but not in time to prevent a turbine trip on high
level. The reported cause was a combination of equipment failure and personnel
errors (failure to wait the required time for the turbine control system and MS
levels to stabilize). The A and B MS instrumentation loops were tuned at 25%
power during restart. {Source: LER 90-001-00}
11/17/1990 MS High Level, Turbine Trip, Reactor Scram: During main turbine CIV testing,
the A MS experienced a high level condition. Dump valve LV-1039A began to
open, but level continued to rise until it reached the turbine trip setpoint. The
cause of the high level was reported as a broken bushing on the hinge pin of the
check valve in the normal drain line. This is postulated to have allowed backflow
from the #5 feedwater heater, which put additional water in the MS and
decreased the effectiveness of LV-1039A to reduce level. A contributing cause
was reported to be sluggish operation of LV-1039A since it did not begin to
stroke open until 22 seconds after the high level alarm was received in the
control room. The check valve was repaired and LV-1039A was disassembled
and inspected, and the valve operator stroke time was adjusted.
{Source: LER 90-028-01}
11/15/1998 MS High Level, Turbine Trip, Reactor Scram. Operators were in the process of
isolating instrument air to a steam seal evaporator level control valve in
preparation for a system outage. The instrument air to the normal level control
valves for the A and B MSs were inadvertently isolated as a result of an error in
the piping and instrumentation diagram. Dump valve LV-1039A failed to open to
maintain level. The cause was reported to be sticking from an iron oxide buildup
in the plug/seat area. Corrective action was a modification to install appropriate
Attachment
D-2
valve trim/plug material to prevent iron oxide buildup. {Source: LER 98-008-01}
10/26/2001 LV-1039A positioner not responding. Replaced positioner. {Source: Notification
20081426/ Work Order 70020654}
11/29/2001 LV-1039A Seat Leakage (Identified by thermal performance assessment -
reduced efficiency): Valve rebuilt in May 2003 during refueling outage. {Work
Order 60025008}
06/22/2002 MS High Level, Turbine Trip, Reactor Scram: Cause was B secondary
condensate pump trip and resulted in recirculation pump runback. During the
runback, MS level increased and LV-1039A failed to stop the level increase.
{Source: Notification 20103635/ Work Order 60029413/ LER 2002-004-00}
06/23/2002 LV-1039A Troubleshooting: The probable cause of the failure of LV-1039A to
stop the level increase during the June 22, 2002, transient was the transient
occurred in such a short period of time that the pneumatic controls did not react
fast enough for the dump valves to respond. LV-1039A stroked fully open in 55
seconds compared to 35 seconds for the LV-1039B valve. The LV-1039A valve
proportional band was adjusted to be the same as the B side. {Source:
Notification 20103635/ Work Order 60029413}
05/07/2003 Installed modification DCP 80051144, intended to reduce the opening time of MS
dump valves LV-1039A and 1039B to less than 10 seconds by installing quick
exhaust valves for the valve operators. {Source: Notification 20143359/ DCP
80051144}
04/16/2004 LV-1039A Appears to be Leaking: Work Order 60040509 canceled after
Thermal Performance Group confirmed on June 9, 2004, that LV-1039A was not
leaking. {Source: Notification 20145639/Order 60040509}
09/16/2004 A MS Dump Valve Failed Open: {Source: Notification 20203784} Thermal
performance/efficiency graph provided by system engineering showed that LV-
1039A was dumping significant heat (equivalent to 9 MWe) to the main
condenser. The MS A low level alarm came in at same time. A review of
computer point data shows that computer point D2602 (LV-1039A Not Closed
Limit Switch) changed state. Post event investigation shows that hanger H25
connection rod was disconnected and was rubbing on the LV-1039A positioner
air line. At this point, it appeared that a large enough hole in the airline had been
made so that the valve came off its open seat (it requires air pressure to
maintain the valve shut). This was a missed opportunity to identify the cause of
the open valve and the consequences of operating with it open.
09/20/2004 Operability assessment for operating with LV-1039A failed 10% open.
Operability considered OE regarding damage to condenser while operating on
MS dump valves to condenser. The operability assessment reasonably
concluded that the condenser will not be damaged and that the condenser is
Attachment
D-3
operable. {Source: 20203784} However, no consideration of two-phase flow in
the line from the MS to LV-1039A was documented. Two-phase flow was
apparent from the low-level in the MS and from the significant loss of MWe. This
was a missed opportunity to recognize the effects of operating with LV-1039A
open.
An operator, not satisfied with the September 20, initiated another notification
(No. 20204256) that same day, stating that the prior notification did not appear
address the potential impact to the condenser penetration which had cracked on
an earlier occasion, when this same dump valve had failed open for an extended
period of time (resulting in elevated offgas flow due to increased inleakage
through the crack at the penetration to the condenser). {Source: Notification
20204256} Again, a formal engineering response, completed on September 22,
did not address the part of the concern related to the prior experience with the
dump valve open. This was another missed opportunity to recognize the effects
of operating with LV-1039A open.
10/4/2004 A review of computer point data shows that computer point D2603 (LV-1039A
Open Limit Switch) changed state several times in 20 minutes. A likely scenario
is that as the hole in the air line became larger because of the hanger rubbing,
the valve opened toward its full open position such that the open limit switch
actuated. Since the air pressure was low and two-phase flow was causing pipe
and valve movement, the limit switch was bumped several times before air
pressure bled off enough to keep the limit switch actuated. This was a missed
opportunity to recognize failure was degrading, investigating cause of
degradation, and identifying cause of failure.
10/10/2004 MS Dump Line Piping Failed at Condenser Penetration {Source: Notification
20206626}
Attachment
E-1
ATTACHMENT E
REACTOR VESSEL LEVEL INSTRUMENT
DEFINITIONS / RANGES
Reactor vessel level is measured through different ranges with diverse and independent
instrumentation. The level ranges and definitions are listed below:
Level Setpoint Definitions
Level 8 +54 inches: High Level Trip Setpoint for HPCI and RCIC
Level 7 +39 inches: High Level Alarm Setpoint
Level 4: +30 inches: Low Level Alarm Setpoint
Level 3: +12.5 inches: Low Level Scram Setpoint Signal to the RPS
Level 2: -38 inches: HPCI, RCIC Initiation Setpoint, Some Containment Valve
Isolations, Reactor Recirculation Pump Trip Setpoint
Level 1: -129 inches: ECCS Initiation Setpoint (Core Spray, RHR, EDG start signals),
Additional Containment Isolations
Reference Values
C Normal Reactor Vessel Level (routine, full power operation): +35 inches
C Top of Active Fuel: -161 inches (which is about 10 feet below Level 2)
Level Instrument Ranges
Narrow Range Instrumentation: 0 inches to +60 inches: Provides Input to RPS Scram and
Main and Reactor Feed Pump Turbine High Level Trip
Wide Range Instrumentation: -150 inches to +60 inches: Provides Trip and Initiation
Inputs to HPCI and RCIC, Auto Start to EDGs, and Signal
for Containment Isolations.
Note: During a plant cooldown, the density of the water is affected; it becomes more dense
(heavier). This change in density affects the water level transmitters, which measure
the weight of water in the reactor vessel. During a cooldown, the water that becomes
more dense would cause a higher than actual level indication. Both the narrow and
wide range transmitters are affected by this phenomenon; and, in fact, because the wide
range instruments have a much larger span than the narrow range (210 inches vs. 60
inches), the effect is roughly tripled on the wide range instruments. This becomes a
factor during a plant cooldown, as operators control reactor vessel level between Level 3
and Level 8. Each range of the instruments causes different actuations, so operators
control reactor vessel level to the equivalent of a more narrow control band (as the
indicated wide range instruments approach the Level 8 setpoint).
Attachment