IR 05000277/2003013

IR 05000277-03-013, 05000278-03-013; 09/24/2003 - 11/18/2003; Peach Bottom Atomic Power Station, Units 2 and 3; Augmented Inspection Team

ML033530016
Person / Time
Site:	Peach Bottom
Issue date:	12/18/2003
From:	Crlenjak R Division of Reactor Safety I
To:	Skolds J Exelon Generation Co, Exelon Nuclear
References
EA-03-224 IR-03-013
Download: ML033530016 (73)

Text

ber 18, 2003

SUBJECT:

NRC AUGMENTED INSPECTION TEAM (AIT) 05000277/2003013 AND 05000278/2003013, AND PRELIMINARY WHITE FINDING - PEACH BOTTOM ATOMIC POWER STATION

Dear Mr. Skolds:

On November 18, 2003, the US Nuclear Regulatory Commission (NRC) completed an Augmented Inspection at the Peach Bottom Atomic Power Station, Units 2 and 3. The enclosed report documents the inspection findings which were discussed with you and other members of your staff during an exit meeting on November 18, 2003.

This inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

The team reviewed selected procedures and records, observed activities, and interviewed personnel. In particular, the inspection reviewed event evaluations (including technical analyses), root cause investigations, relevant performance history, and extent of condition to assess the significance and potential consequences of issues related to the September 15, 2003, dual unit scram.

The team concluded that the overall response of Exelon to the dual unit scrams on September 15, 2003, was adequate in that the plants were safely taken to a cold shutdown condition. Nevertheless, the operators were challenged by equipment and procedural problems. Prior to the event, there were lapses in equipment performance monitoring, such as for the offsite power protection circuits and the emergency diesel generators. Additionally, some degraded conditions and incomplete resolution of equipment problems have been tolerated. During the event, some operator actions, such as lowering the torus water level, were delayed due to procedure problems. Exelons corrective actions regarding equipment and procedures for this event have been appropriate.

Mr. John This report discusses a finding that appears to have low to moderate safety significance. As described in Section 2.2 of this report, this finding involves issues related to the E2 emergency diesel generator (EDG), one of four EDGs that constitute the highly risk important standby emergency AC power system. The finding includes a deficient maintenance procedure for installation of cylinder liner adapter gaskets on the EDG and inadequate corrective actions for low jacket water pressure conditions observed on the E2 EDG in March and April 2003. Some of the adapter gaskets leaked, resulting in combustion gas entering the jacket water cooling system. This ultimately led to a low jacket water pressure condition and an automatic trip of the EDG.

This finding was assessed using the reactor safety Significance Determination Process (SDP)

as a potentially safety significant finding that was preliminarily determined to be White for Unit 2 (i.e., a finding with some increased importance to safety, which may require additional NRC inspection) and Green for Unit 3. The difference in risk significance between the units is due to differences in electrical bus loads. The finding appears to have low to moderate safety significance, because an EDG became inoperable during an actual loss of offsite power to the emergency buses. The loss of the EDG did not present an immediate safety concern because operators were able to restore power to the affected safety buses from offsite sources.

Following identification of the degraded gaskets, you implemented appropriate corrective actions by replacing all of the adapter gaskets on the E2 EDG and testing for gas inleakage on that EDG and the other three EDGs. Long term corrective actions include procedure changes and enhancing the monitoring for combustion gas inleakage on all four EDGs.

The finding also involves two apparent violations of NRC requirements which are being considered for escalated enforcement action in accordance with the General Statement of Policy and Procedure for NRC Enforcement Actions (Enforcement Policy), NUREG-1600. The current Enforcement Policy is included on the NRCs Website at http://www.nrc.gov/what-we-do/regulatory/enforcement.html.

We believe that we have sufficient information to make our final risk determination for the performance issue regarding the E2 EDG. However, before the NRC makes a final decision on this matter, we are providing you an opportunity to either submit a written response or to request a Regulatory Conference where you would be able to provide your perspectives on the significance of the finding, the bases for your position, and whether you agree with the apparent violations. If you choose to request a Regulatory Conference, we encourage you to submit your evaluation and any differences with the NRC evaluation at least one week prior to the conference in an effort to make the conference more efficient and effective. If a Regulatory Conference is held, it will be open for public observation. The NRC will also issue a press release to announce the Regulatory Conference.

Please contact Dr. Mohamed Shanbaky at (610) 337-5209 within 10 business days of the date of this letter to notify the NRC of your intentions. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision and you will be advised by separate correspondence of the results of our deliberations on this matter.

Mr. John Because the NRC has not made a final determination in this matter, no Notice of Violation is being issued for this inspection finding at this time. In addition, please be advised that the number and characterization of the apparent violations described in the enclosed inspection report may change as a result of further NRC review.

Additionally, based on the results of this inspection, the team identified three findings of very low safety significance (Green). Two of these issues were determined to involve violations of NRC requirements. One of the findings involved inadequate prior corrective actions, and the other two findings involved procedural issues. However, because of their very low safety significance, and because they have been entered into your corrective action program, the NRC is treating these issues as non-cited violations, in accordance with Section VI.A.1 of the NRCs Enforcement Policy. If you deny the non-cited violations noted in this report, you should provide a response with the basis for your denial, within 30 days of the date of this inspection report, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001; with copies to the Regional Administrator, Region I; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, D.C. 20555-0001; and the NRC Resident Inspector at the Peach Bottom facility.

In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter and its enclosures will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

If you have any questions, please contact Dr. Shanbaky at (610) 337-5209.

Sincerely,

/RA/

Richard V. Crlenjak, Deputy Director Division of Reactor Safety Docket Nos: 50-277, 50-278 License Nos: DPR-44, DPR-56 Enclosure: Inspection Report 05000277/2003013 and 05000278/2003013 w/Attachment: Supplemental Information Attachments:

A. Supplemental Information B. Augmented Inspection Team Charter C. Sequence of Events D. System Figures

Mr. John

SUMMARY OF FINDINGS

IR 05000277/2003-013, 05000278/2003-013; 09/24/2003 - 11/18/2003; Peach Bottom Atomic

Power Station, Units 2 and 3; Augmented Inspection Team.

The inspection was conducted by four regional inspectors, one resident inspector, one headquarters inspector, and one regional senior reactor analyst. One finding, assessed as Preliminary White on Unit 2 and Green on Unit 3, and 3 other Green findings were identified.

The significance of most findings is indicated by their color (Green, White, Yellow, Red) using IMC 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Cornerstone: Mitigating Systems

• Preliminary White for Unit 2 and Green for Unit 3. A self-revealing finding was identified for the failure to adequately maintain the E2 emergency diesel generator (EDG) between July 1992 and September 2003. This finding involved two apparent violations. An apparent violation of Technical Specifications was identified for the failure to maintain the maintenance procedure for installation of EDG adapter gaskets. The procedure did not incorporate certain vendor recommendations intended to provide proper sealing of the gaskets, leading to relaxation over several years that allowed combustion gases to enter the jacket coolant system. An apparent violation of 10 CFR 50 Appendix B, Criterion XVI, Corrective Actions was identified because Exelon did not correct a condition adverse to quality following two instances of low jacket water pressure observed on the E2 emergency diesel generator (EDG) in March and April 2003. Subsequently, the EDG failed due to a low jacket water pressure condition.

This finding is more than minor because it is associated with the equipment performance attribute of the Mitigating Systems cornerstone and adversely affects the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events. The finding was assessed using a Phase 3 evaluation.

The finding is of low to moderate safety significance (WHITE) at Unit 2 based on delta core damage frequency ( CDF) and delta large early release frequency ( LERF). The finding is of very low safety significance (GREEN) at Unit 3 based on CDF and LERF.

The difference between the two units is attributable to differences in electrical bus loads.

(Section 2.2)

Green.

A self-revealing non-cited violation of Technical Specification (TS) 5.4.1,

"Administrative Controls - Procedures," was identified. The existing emergency operating support procedures did not have adequate instructions to be used when Group II / III isolation signals were present. This resulted in delaying restoration of torus level and reducing containment pressure for approximately 14 hours1.62037e-4 days <br />0.00389 hours <br />2.314815e-5 weeks <br />5.327e-6 months <br /> while a procedure was developed.

iii

Summary of Findings (contd)

This finding is more than minor because it is associated with the Procedure Quality attribute of the Mitigating Systems cornerstone and adversely affects the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. It delayed the operators in placing Unit 3 into a cold shutdown condition. This finding is of very low safety significance (Green) using Phase 1 of the Significance Determination Process for reactor inspection findings for At-Power reactor situations. The finding is of very low safety significance because the finding is not a design or qualification deficiency, does not result in a loss of safety function, and is not potentially risk significant due to seismic, flood, fire, or weather related initiating event. (Section 3.1)

Green.

A self-revealing finding was identified because Exelon did not correct a previously known equipment deficiency with the Unit 2 "B" condenser hotwell level instrument as required by the corrective action program. The equipment deficiency resulted in draining the condensate storage tank (CST) to the condenser hotwell and automatically transferring the high pressure coolant injection and reactor core isolating cooling systems suction from the CST to the torus. The automatic transfer of the suction to the torus was unexpected at this point during the event and therefore resulted in an added operational burden for the operators.

This finding is more than minor because it is associated with the Equipment Performance attribute of the Mitigating Systems cornerstone and adversely affects the cornerstone objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences. This finding is of very low safety significance (Green) using Phase 1 of the Significance Determination Process for reactor inspection findings for At-Power reactor situations. The finding is of very low safety significance because the finding is not a design or qualification deficiency, does not represent an actual loss of safety function, and does not screen as potentially risk significant due to a seismic, fire, flooding, or severe weather initiating event. (Section 3.3)

Green.

A non-cited violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions,

Procedures, and Drawings," was identified because maintenance procedures for the installation of the packing on the safety relief valve (SRV) air operator assembly were inadequate. The packing installation procedures did not assure that the packing was properly installed (packing nuts adequately tightened, etc.) on RV-3-02-071G resulting in hot steam leaking past the packing and damaging the air operator diaphragm during the September 15, 2003, event.

This finding is more than minor because it is associated with the Procedure Quality attribute of the Mitigation Systems cornerstone and adversely affects the cornerstone objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences. The finding was evaluated in accordance with IMC 0609, Appendix A, "Significance Determination of Reactor Inspection Findings for At-Power Situations," using a Phase 3 significance determination process (SDP) analysis. This issue is of very low safety significance, based on the iv

Summary of Findings (contd)

Phase 3 analysis results, assuming that RV-3-02-071G would not have opened from the control room for one year. The Phase 3 evaluation, using the 3.01 SPAR model for Peach Bottom, indicated a negligible increase in risk of not being able to manually depressurize with the 10 remaining valves. (Section 2.3)v

Report Details 1.0 Description of Events and Chronology 1.1 Event Summary SUMMARY OF

PLANT STATUS

At approximately 1:32 a.m. on September 15, 2003, Unit 2 and Unit 3 experienced a brief loss of offsite power to the emergency buses. The loss of offsite power resulted in the loss of power to the reactor protection system (RPS) motor generator sets which automatically shut down Unit 2 & 3 and automatically initiated Primary Containment Isolation System (PCIS) Group I, II, III isolations. All four emergency diesel generators automatically started and the standby gas treatment system automatically started. The High Pressure Coolant Injection (HPCI) System and Reactor Core Isolation Cooling (RCIC) System were manually started to maintain reactor water level. Reactor pressure was controlled using the Main Steam Safety Relief Valves (SRV) on both units. Exelon determined that the loss of offsite power was the result of a lightning strike approximately 35 miles northeast of the plants.

Conditions Prior to the Event Unit 2 was in Mode 1, and operating at 100 percent rated thermal power when the event occurred. There were no activities in progress related to the loss of offsite power. All plant parameters were normal and all systems were operating as expected at the time of the event.

Unit 3 was in Mode 1 and operating in coastdown at 91.5 percent rated thermal power when the event occurred. There were no activities in progress related to the loss of offsite power. All the plant parameters were normal and plant systems were operating as expected at the time of the event.

Unit 2 Following the lightning strike, the four 4 kV emergency buses lost their normal supply of offsite power. When all power was lost to the emergency buses the emergency diesel generators (EDGs) received a start signal. All four EDGs automatically started and supplied power to the 4 kV emergency buses within the required 10 seconds. The reactor automatically shut down due to the loss of power to the reactor protection system (RPS) motor generator (MG) sets. The loss of power to the RPS MG sets also resulted in Group I, II, III isolations, which closed the main steam isolation valves (MSIVs) and isolated the reactor from its normal heat sink.

Isolating the reactor from the normal heat sink resulted in the reactor decay heat going into containment (torus) through the safety relief valves (SRVs), rather than to the normal heat sink (main condenser). The SRVs opened and closed automatically, on their pressure setpoints, while the operators assessed the status of the various systems.

The high pressure coolant injection (HPCI) and reactor core isolation cooling (RCIC)systems were manually started when reactor vessel water level was at 150 inches above the top of active fuel to restore level. Approximately 10 minutes after HPCI and RCIC were started their normal suction from the condensate storage tank (CST)automatically transferred to the torus suction on low CST level because the B hotwell level transmitter failed at a low level, which resulted in draining the CST to the hotwell.

The operators placed the A hotwell level transmitter in control and the hotwell water was pumped back to the CST. The CST level was restored to normal about an hour later.

At approximately 2:00 a.m. the control room was notified that offsite power was available. However, restoring offsite power was not a priority at this time because all emergency buses were being powered by the EDGs. Operators were focused on coordinating the operation of Unit 2 torus cooling with Unit 3. This coordination was required because the residual heat removal (RHR) system logic does not allow the same division RHR pump to operate on both units simultaneously (Unit 2 A pump and Unit 3 A pump cannot be run simultaneously). Therefore, the operators were coordinating the operation of torus cooling using only two of four RHR pumps on each unit. At 2:05 a.m. the operators entered off normal procedures for increasing drywell pressure and temperature, due to the decay heat going to the torus. The B RHR pump was started in preparation for establishing torus cooling. While throttling open the torus cooling valve (MO-34) the E2 EDG automatically shut down, tripping the B RHR pump. Power was lost to Unit 2 emergency bus E22 and it remained de-energized because offsite power was not restored to transformer 2 SU after it was available for use. The Unit 3 emergency bus E23 automatically transferred to offsite power because transformer 343 SU was automatically repowered shortly after the initial loss of offsite power.

At 2:48 a.m. torus temperature reached 120E F and the A RHR pump was placed in torus cooling. The Unit 3 C RHR pump was shut down to allow starting of the Unit 2 C RHR pump for torus cooling. At 2:59 a.m. torus cooling was in service using the A loop heat exchanger and the A and C RHR pumps. RCIC and HPCI suctions were manually transferred back to the CST and offsite power was restored to 4 kV bus E22.

However, drywell pressure continued to increase due to compressing the nitrogen in containment as the torus water level increased from the condensing steam from the SRVs, HPCI and RCIC. At approximately 4:00 a.m. drywell pressure increased to 2 psig due to this torus level increase. At 6:45 a.m. the Group I isolation was reset in preparation for reestablishing the condenser as the main heat sink. The condenser was reestablished as a heat sink and at 1:30 p.m., and the torus level was lowered to restore drywell pressure and torus level to normal. Shutdown cooling was placed in service at 2:00 a.m. on September 16, 2003.

After the reactor automatically shut down, the operators entered the applicable emergency operating procedures (EOPs) to control reactor power, pressure and level.

The primary containment EOP was also entered due to the SRVs, HPCI and RCIC steam addition to the torus thereby increasing water level.

The operators were challenged by a high number of equipment deficiencies that complicated their ability to stabilize the plant in a shutdown condition. The most notable complications were:

• the E2 emergency diesel generator tripped on low jacket water coolant pressure stopping the B RHR pump and draining the B torus cooling loop, which resulted in reducing the availability of torus cooling on Unit 2

• the CST water was drained to the hotwell, which automatically transferred the suction for the HPCI and RCIC pumps to the torus.

In addition, to the equipment challenges noted above, the following degraded equipment, equipment failures and procedural issues complicated the operators response and recovery from this event.

• Erratic indication on the C SRV at low pressures (approximately 600 psig)resulted in operators not using the C SRV. Similar problems were noted on December 22, 2002, at approximately 200 psig. The cause was determined to be an out of specification acoustic sensor. Condition Report (CR) 137762 and 176081 were initiated to develop and implement corrective actions concerning indication problems.

• Erratic indication on the H SRV at low pressures (approximately 450 psig)resulted in operators not using the H SRV. Similar problems were noted on December 22, 2002, at approximately 450 psig. Troubleshooting in the drywell found no failures. The acoustic sensor and cable were replaced, and CR 137762 and 176081 were initiated to develop and implement corrective actions concerning indication problems.

• A cable fault alarm on the K SRV resulted in the operators not using the K SRV. The cause was determined to be an out of specification acoustic sensor.

CR 176897 was initiated to develop and implement corrective actions concerning indication problems.

• Vacuum Relief Valve POS-8096J had a split indication. This was due to a bent plastic barrier that splits the indication lights from each other. This issue was previously identified on 7/28/03.

• The HPCI room had a burning oil smell which was due to oil residue on piping from a prior maintenance activity. No hydraulic control or lube oil leaks were identified during a system walkdown.

Unit 3 Following the lightning strike the initial Unit 3 response was similar to Unit 2. The four 4 kV emergency buses lost offsite power and the EDGs automatically started and supplied power to the emergency buses. The reactor automatically shut down and experienced a Group I, II, and III isolation from the loss of power to the RPS MG sets. However, Unit 3 experienced several other equipment issues which complicated the operators response during the event. During the Group I isolation the outboard D MSIV failed to immediately close. The redundant inboard D MSIV closed as designed and isolated the reactor from the normal heat sink (condenser). The D SRV opened and initially failed to close, depressurizing the reactor to 369 psig before it closed. The rapid depressurization allowed the condensate system to inject at low pressure and had to be manually shut down. These unexpected equipment failures challenged the operators ability to maintain vessel water level and reactor vessel pressure during the first fifteen minutes of the event.

Shortly after identifying that the D MSIV was not closed, plant operators were dispatched to manually close the open MSIV using AO-1A.2-3, Closing a Stuck Open Inboard or Outboard Main Steam Isolation Valve (section 3.4, below). Five of the SRVs, including the D SRV automatically opened to control reactor pressure and discharge steam to the torus. Four of the SRVs closed, but the D SRV remained open reducing reactor pressure over the next fifteen minutes. The operators entered procedure OT-114, Stuck Open SRV, and dispatched operators to attempt to close the SRV. At 1:38 a.m. and 1:40 a.m. RCIC and HPCI were manually started at 132 inches above the top of active fuel. This resulted in further reducing reactor pressure allowing the condensate pumps to inject into the reactor vessel at low pressure. At 1:43 a.m. the reactor vessel high level was experienced due the injection from the condensate pumps.

The condensate pumps were tripped in accordance with OT-110, Reactor High Level -

Procedure, and T-102, Primary Containment Control, was reentered due to increasing torus level. At 1:47 a.m. the D SRV closed without operator action at a reactor pressure of 369 psig thereby stopping the depressurization.

The C and D RHR pumps were placed in torus cooling to reduce torus water temperature. At 2:34 a.m. the E2 EDG tripped, and the E-23 4 kV bus automatically transferred to offsite power. At 2:48 a.m. the outboard D MSIV closed without any operator action. The Unit 3 C RHR pump was removed from torus cooling to allow the Unit 2 C RHR pump to be used. The B RHR pump was placed into torus cooling on Unit 3. At 4:51 a.m. the RPS MG set was reenergized and the isolations were reset.

After resetting the isolations the torus filter pump was started, water was transferred from the torus to radwaste and the crew began to restore the normal heat sink. At 5:38 a.m. reactor water level was inadvertently lowered due to a communication error and a Group II, III isolation occurred, isolating the torus drain path. Before the drain path could be reestablished, the rising torus level increased drywell pressure to 2 psig, preventing the isolation from being reset. The drywell pressure continued to increase due to the increasing torus level from the SRVs, HPCI, and RCIC by compressing the nitrogen in the drywell and torus. This resulted in preventing the torus level from being reduced until a procedure was developed to drain the torus with an isolation signal present. At 8:40 p.m. a procedure was developed and the torus level reduction began.

The torus pumpdown was completed at 8:02 a.m. on September 16, 2003, and the unit was placed in shutdown cooling at 9:39 a.m. and reached cold shutdown at 1:21 p.m.

In general the operating crew performed in accordance with station procedures. After the reactor automatically shut down, the operators entered the applicable emergency operating procedures (EOPs) to control reactor power, pressure, and level. The primary containment EOP was also entered due to the SRVs, HPCI, and RCIC steam addition to the torus thereby increasing the water level.

The operators were challenged by a high number of equipment deficiencies that complicated their ability to stabilize the plant in a shutdown condition. The most notable complications were:

• the trip of the E2 emergency diesel generator (EDG) due to low jacket coolant pressure approximately one hour after automatic starting

• the failure of the D SRV to close until pressure reached approximately 369 psig which contributed to the high drywell pressure and high torus temperature during the event

• the failure of the G SRV to open upon manual initiation; the valve lifted as expected on the initial transient

• the failure of the D outboard MSIV to close upon receipt of a Group I isolation signal; the valve closed 76 minutes later with no operator action; the associated inboard MSIV closed as expected.

In addition, to the equipment challenges noted above, the following degraded equipment, equipment failures and procedural issues complicated the operators response and recovery from this event.

• Containment could not be vented because the 3A CAC/CAD analyzer would not change sample points, and the 3B CAC/CAD analyzer had failed. An analyzer reset was performed.

• Input power spikes damaged the power supply for the TR-3-02-103 SRV tailpipe temperature recorder. Initial troubleshooting identified a blown fuse. The fuse was replaced and the power supply was malfunctioning, so the entire power supply was replaced.

• The Plant Monitoring System computer did not indicate that SRV F was open.

This was an indication problem only.

• There were packing leaks on HPCI HV-3-23C-33428, MO-3024 C, MO-3024 F, and MO-3163 B. The valves were backseated prior to the event to limit previously identified packing leakage. The plant transient stroked all three valves off of the backseat with notable increase in packing leakage.

• The control room chiller breaker (0AK19) failed to close initially when demanded.

• The E3 EDG Trouble alarm actuated, which was a previously identified equipment problem. Starting of the 3C HPSW pump was delayed until the cause of this alarm was determined.

1.2 Preliminary Risk Significance of Event A preliminary risk assessment was performed to determine that an Augmented Inspection Team (AIT) was the appropriate NRC inspection response to the event. The preliminary risk assessment was conducted for Unit 3 using the Peach Bottom Unit 2 &

3 SPAR model revision 3.02, dated January 2003, with no test and maintenance unavailability included. The assessment assumed a Unit 3 Plant Centered loss of offsite power (LOOP), MSIV closure, a single stuck open relief valve and E2 EDG failure to run. This resulted in an initial conditional core damage probability (CCDP) of low E-3.

The Unit 2 CCDP for a Plant Centered LOOP, MSIV closure, and E2 EDG failure to run was low E-4. Based on these results and using the guidance in Management Directive 8.3, the NRC determined that an AIT should be performed. A final more detailed risk assessment is described in section 6.0 of this report.

2.0 Equipment Failures and Root Causes 2.1 Offsite Power Issues

a. Inspection Scope

The team reviewed the response of the offsite grid to a lightning strike which struck a transmission tower approximately 35 miles from the Peach Bottom Atomic Power Station. This lightning strike resulted in the loss of two of the three offsite power supplies to both Peach Bottom units. The team examined documents, interviewed personnel, and reviewed Exelon's root cause investigation.

2.1.1 Electrical Grid Transient A lightning strike on a transmission tower resulted in the loss of two of the three offsite power supplies to both Peach Bottom units and caused both of them to automatically shut down. The plant lost two of the three offsite power sources and the power source for station blackout for a brief period.

2.1.2 Electrical Fault and Cascading Effect of Electrical Outage At 1:32 a.m. on September 15, 2003, lightning struck a transmission tower, between Bradford and Planebrook Electric substation (See Attachment D), located approximately 35 miles away from the Peach Bottom Nuclear Stations. The lightning struck the C phase insulator of line 220-31 between Bradford and Planebrook substation. Visual inspection of the insulator assembly confirmed a flashover on the insulator. A flashover occurs when lightning hits the static wire or tower and produces a voltage on the tower top. The voltage was of sufficient magnitude to cause a breakdown in the air column along the insulator string, resulting in an electrical arc between the C phase and the tower.

Once the arc occurs the air is ionized and becomes conductive, allowing fault current to flow to ground until interrupted. The faulted condition appears to have lasted almost 2.5 seconds based on review of the protective relay operational records. The insulator was damaged by the thermal stresses from the extended fault clearing time. The two independent protection schemes for isolating the faulted condition failed to function.

The primary and backup protection from the directional ground fault relay used fault current and locally generated polarizing voltage to determine fault condition. The primary relay utilized a signal from the bus potential device and the backup relay utilized a signal from the line potential device. The primary protection circuit was found to have a mechanically failed fuse and the backup protection circuit had a loose connection on a screw terminal block.

Because the primary and backup protection in the faulted zone did not isolate the fault, the faulted condition was sensed at a greater distance and the automatic isolation expanded to a larger zone. The system outage spread through several other substations for periods up to 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> and 43 minutes. The Nottingham substation did not isolate the spread at that substation because the directional ground relay protection was not activated when the new protective relay system was put into service.

2.1

.3 Offsite Power Recovery

The electrical fault from the lightning continued to arc to the ground for more than 2.4 seconds and was evidenced through reopening of the breaker when it automatically reclosed. The fault condition cleared after the lightning induced arcing diminished.

According to the distribution companys records another automatic reclosure at 16.6 seconds, from the time of the initial fault, re-energized the line. Even though the insulator on the tower had some damage, it retained essential operational integrity and was capable of remaining in service until it was replaced. Following the energization of the faulted line, the 343 startup transformer, one of the offsite power sources, was energized through closing of breaker 175, at 18.3 seconds. In 27 seconds from the event, line 220-08 was automatically energized. The breaker SU-25 for #2 startup transformer was closed by the Peach Bottom Operator at 6:01 a.m.

The Station Blackout (SBO) power supply was recovered when line 220-03 was recovered along with line 220-08 by the automatic closing of Goshen 125 breaker at 16.1 seconds.

2.1.4 Offsite Power System Impact on Peach Bottom Nuclear Stations A lightning strike at a transmission line tower, approximately 35 miles away from the stations resulted in a two unit reactor trip and loss of two of the three offsite power sources and the SBO power source.

Peach Bottom is equipped with two offsite power sources from the 230 kV transmission system with each aligned to two of the four safety related 4160 V buses for each unit.

The buses are designed to make one automatic transfer to the alternate source when undervoltage or dead bus conditions are experienced. If that transfer attempt is not successful, the respective EDG is commanded to automatically start and accept the loads.

During the event, both offsite power supplies from the 230 kV system were lost; however, the third source of offsite power from the 500 kV system remained available to the 13.2 kV distribution system through an autotransformer and a regulating transformer. This system supports most of the non-safety related loads at the plant.

During the event, the loads successfully transferred to this source. Exelon has the flexibility to manually connect this 13 kV source or the SBO power supply to safety buses when needed.

The only abnormal condition on the SBO power source that existed after 16 seconds into the event was a breaker (CB 1005) had opened due to the undervoltage condition.

Procedures were available to align the SBO power source as needed. An additional source coupled to the SBO source through Conowingo Hydro station was connected after 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> from the event.

2.1.5 Root Causes and Corrective Actions for the Offsite Power System Exelon chartered a multi-discipline team of managers and technical experts, combining several divisions of the organization to investigate the root causes of the offsite power system failure, and to develop procedural and process changes to prevent any recurrence. During the systematic evaluation process, Exelon analyzed causes, root causes, contributing factors and extraneous conditions adverse to quality, promptly completed the required immediate corrective actions, and assigned managers to develop long term corrective actions, design & process improvements, and institute new/revised management expectations.

The electrical grid transient was caused by a lightning induced fault that was not instantaneously isolated at the first level of the engineered protection. The fault occurred on line 220-31 between the Bradford and Planebrook substations. The relays at the Planebrook end operated properly and opened breaker 315 to isolate one end of the faulted line. However, breaker 290, at the Bradford end of the faulted line did not open.

The root cause team found out that the primary and secondary directional ground fault protection did not actuate. These relays were calibrated in September of 2000 and trip tested in April 2003 as part of the four year maintenance program. The relays had last operated to isolate a fault on August 6, 1999. The investigation identified that some protection circuits were not routinely checked due to standards not rigorously implemented, testing and surveillances on protection circuits were not fully understood, and poor work practices and management expectations existed.

There are two independent directional ground fault relay protection circuits provided at Bradford Substation. The independent voltage signals to the relays were absent on both relays. This condition could not be detected through a monitoring system as the circuit carries no potential except when the system experiences an electrical fault. The primary relay system that failed was found to have a mechanically failed fuse, and the secondary relay, designed as the backup protection, had a loose wire that prevented the voltage signal. In the absence of these voltage signals, these relays cannot detect the direction of a fault and the relay logic sensed that the fault was elsewhere in the system.

The prevailing ground fault current caused the cascading trips that spread the outages to 17 other transmission lines.

The cascading trips could have been limited to one of the offsite power sources for Peach Bottom if Nottingham substation breaker 90 had the directional ground fault feature activated during the system upgrade. The other offsite power could have been sustained if the Newlinville substation had undergone the system upgrade with directional ground fault relays. Exelon has activated the directional ground fault relay on breaker 90 and has scheduled the upgrade of the Newlinville substation. The upgrades are part of the system wide improvements.

Corrective action by Exelons transmission division consisted of prompt replacement of all the fuses in the 230 kV system (about 60 fuses) and replacement of 10 fuses in the 500 kV system that included the serving areas of Limerick Nuclear Station. Exelon examined the western division for similar problems and found that the western transmission system was designed differently without these type of fused circuits. The loose connection was tightened for circuit integrity.

The faulty fuse was sent to a lab to analyze the type of failure. The fuse had a fracture of the solder connection between the fuse element and the ferrule. The mechanism of the failure was unknown, though no indication of electrical, thermal, or mechanical abuse was observed. The lab analysis attributed the failure to mechanical, thermal/electrical cycling, vibration, or a combination thereof. The ferrule cracking likely reduced the clamping action of the ferrule onto the fuse barrel, transferring load to the solder connection that had incomplete fusion.

During this event another dormant relay problem surfaced at Muddyrun Substation tripping breaker 175 that connected the 230 kV grid to the 343 startup transformer. The factory examined the relay that signaled the trip and identified the problem to be hardware related. The relay was promptly replaced and other applications of the same type of relay were examined to rule out any common mode problems.

Based on the teams inquiry, Exelon examined why the automatic breaker reclosing triggered at 16 seconds. Since information that was collected could not explain this timing, Exelon plans to retrieve further data from the relay during a scheduled transmission line outage. Exelon plans to determine the bases and the necessity of the 16 second delay for the breaker, verify accuracy of reclosing time for other breakers in this event, and implement corrective actions as needed.

Exelon identified the following corrective actions to be implemented in the region after additional details are worked out: evaluate the need for using fuses in the open delta directional ground relay circuits given the repeated removal of fuses for testing purposes; develop a fuse handling procedure based on vendor recommendation and in consultation with the test lab; develop and implement a test practice to verify the conductivity of circuits that are not monitored by indicating equipment; evaluate the maintenance alteration log for ensuring adequate restoration to avoid loose terminal connections; and evaluate each microprocessor relay on the 230 kV system to have the protection scheme in accordance with internal standards, PJM standards, and Mid-American Interconnected Network standards and best practices.

b. Findings

The team concluded that the Exelon investigation was of adequate scope and depth.

No findings of significance were identified.

2.2 E2 Emergency Diesel Generator

a. Inspection Scope

The team reviewed the circumstances related to the unexpected automatic shutdown of the E2 emergency diesel generator (EDG) approximately one hour after it began supplying emergency bus loads following the loss of offsite power on September 15, 2003. The EDG shut down on an engine protective trip initiated by low jacket water pressure.

Peach Bottom has four Fairbanks Morse 12 cylinder, opposed piston diesel engines.

The EDG jacket water system provides cooling to the cylinder liner jackets and will initiate an automatic shutdown if jacket water pressure drops too low. Each cylinder contains four adapter ports with copper gaskets that assure a seal between high pressure gases in the cylinder and the jacket water system.

The team examined maintenance, engineering, and operations documents, and interviewed station personnel. The team also performed walkdowns of all four EDGs.

The team reviewed Exelons root cause determination, which concluded that the EDG failed due to inadequate initial pre-load of the copper adapter gaskets combined with time-related stress relaxation of copper. This condition allowed combustion gases to leak into the jacket water system and ultimately degrade the performance of the jacket water pump.

b. Findings

Introduction A self-revealing finding, assessed as Preliminary White on Unit 2 and Green on Unit 3, was identified for the failure to adequately maintain the E2 EDG between July 1992 and September 2003. This finding involved two apparent violations. An apparent violation of Technical Specifications was identified for the failure to maintain the maintenance procedure for installation of EDG adapter gaskets. The procedure did not did not incorporate certain vendor recommendations intended to provide proper sealing of the gaskets, leading to relaxation over several years that allowed combustion gases to enter the jacket coolant system. An apparent violation of 10 CFR 50 Appendix B, Criterion XVI, Corrective Actions was identified because Exelon did not correct a condition adverse to quality following two instances of low jacket water pressure observed on the E2 emergency diesel generator (EDG) in March and April 2003. Subsequently, the EDG failed due to a low jacket water pressure condition.

Description On September 15, 2003, the E2 EDG shut down automatically approximately one hour after both units experienced a loss of offsite power to the emergency buses. After examining a number of potential failure modes, Exelon concluded that the EDG tripped on low jacket water pressure caused by gas binding of the jacket water pump. Exelon determined that combustion gases had leaked by cylinder adapter gaskets and had displaced a portion of the liquid in the jacket water system.

Inadequate Initial Installation of Adapter Gaskets According to Exelons investigation, the most significant contributor to the gasket leakage was inadequate installation in 1992. Some of the gaskets had not been compressed or crushed sufficiently for a proper seal during initial installation. The gaskets began to leak after a period of relaxation, based on laboratory analysis of a sample of the adapter gaskets. The laboratory observed that at least four gaskets exhibited signs of staining, indicative of combustion gas leakage over a period of time.

One gasket that had been installed in cylinder #8 was stained significantly more than the others. Exelon believed that the gaskets had been leaking for a long period of time and the leakage became progressively worse until the leakage rate exceeded the self-venting capability of the jacket water system.

Exelon considered the incomplete crush to be the result of inadequate guidance for the installation of the gaskets. The procedures for installation of the gaskets were based on Fairbanks Morse Service Information Letter A-24, dated May 1, 1990. This letter provided two alternatives for gasket installation to prevent combustion gas leakage, one of which was based on a torque value only, and the other listed a torque value and an angular rotation. Exelon used the torque-only approach in the EDG maintenance procedures. Exelons root cause investigation team concluded that this guidance was not sufficient to ensure adequate crush of the gaskets.

Exelon noted other installation procedure inadequacies during their investigation. First, they found that the maintenance procedure should have included directions to inspect the gaskets prior to installation to assure that an annealing process did not cause gasket warping or irregularities. This precaution was discussed in Fairbanks Morse Service Information letter A-15, dated June 22, 1987, but was not placed in the procedure. Secondly, Exelon identified that there was no direction in the maintenance procedure to verify that the adapter and cylinder liner threads were clean. The importance of this step was illustrated by post-event maintenance activities.

Technicians replaced some of the adapter gaskets initially, and some of the new gaskets leaked due to pieces of the original gaskets remaining on the threads.

Exelon developed corrective actions to address the procedure inadequacies. Exelon determined that the most critical corrective action was to revise the maintenance procedures to include a direct means to measure the final adapter gasket crush and assure that it meets acceptance criteria.

The team noted that most E2 adapter gaskets had not been retained for further examination by the laboratory or other station personnel. Thus, both Exelons root cause investigation team and the AIT were unable to determine whether a significant number of other gaskets exhibited signs of degradation.

Anecdotal information from technicians who removed the E2 adapter gaskets following the event provided further evidence that there were deficiencies during initial installation or greater than typical relaxation of the gaskets. The technicians stated that, in general, the adapters were not as tight as others they had maintained on other Exelon EDGs.

Also, they observed that upon removal of the adapters, the gaskets did not adhere as tightly to the adapters as seen on other EDGs.

Stress Relaxation of Copper Gaskets Stress relaxation of the copper adapter gaskets was a contributing factor that explains the timing of the failure. The gaskets likely did not leak initially following installation, but began to leak due to stress relaxation over time, which is characteristic of copper.

Additionally, Exelons investigation considered external forces that may have contributed to the relaxation of the gaskets that were installed without adequate preload. The expected operational forces on the joints and gaskets included cylinder firing pressure cycles that can reach 1325 psig, normal cylinder operating temperatures, and rapid changes in temperature during quick starts and rapid loading.

Exelons investigation determined that no single start of the EDG or rapid loading of the EDG caused the gasket failures.

The team concluded that the time-dependent relaxation of the gaskets occurred primarily during the periods in which the EDG was starting and operating, when the gaskets were subject to the highest extremes of temperature and differential pressure.

In addition to the normal, expected operational forces that acted on the gaskets and the adapter/cylinder liner joints, there was a period of higher-than-normal vibration that may have contributed to the rate of gasket relaxation.

Untimely Resolution of High Vibration Condition on E2 EDG The team noted that a degraded condition involving high vibration existed on the E2 EDG for an extended period, from 1996 through 2002. According to Exelons root cause investigation team, this was likely a minor contribution to the gasket relaxation, because the vibration may have caused the gasket preload to decrease faster than under normal vibration conditions. During the period of high vibration, on numerous occasions Exelon observed maximum horizontal vibration at the generator that exceeded the vendors maximum allowed in-service level (6 mils) and the shutdown level (10 mils). The team noted that Exelon had not evaluated the impact of the vibration on EDG operability.

Prior Opportunities to Identify Degrading Adapter Gaskets The team identified three potential opportunities for Exelon to have identified degradation of the cylinder adapter gaskets prior to the event. In each of these cases, Exelon identified abnormal equipment conditions on the E2 EDG, but did not conduct thorough review and follow-up activities that would have either identified or ruled out the possibility of combustion gas leakage past the gaskets.

1. Low Jacket Water Pressure - Inadequate Corrective Action

The team determined that Exelon identified but did not correct a condition adverse to quality involving two instances of low jacket water pressure recorded during EDG operation on March 22 and April 5, 2003. Exelon did not perform several steps in the troubleshooting section of the vendor manual for these observed conditions. Additionally, Exelon closed the associated maintenance action request without resolving the low pressure condition, contrary to the guidance in an Exelon maintenance standard.

The team identified one occurrence that was documented during surveillance testing on March 22, 2003, thirty minutes after the EDG was loaded. The operator recorded jacket water pressure at 17 psig, and noted that pressure was swinging between 17 and 20 psig (the acceptance band is 25 to 35 psig). This deficiency was documented in Action Request A1410114. Exelons review team identified a similar occurrence on April 5, in which jacket water pressure was recorded at 16 psig.

Exelon technicians conducted troubleshooting for these conditions in May 2003.

The purpose of the action request was to resolve the low pressure indication on the gauge. Technicians found no problem with the gauge and closed the maintenance action request. Exelon took no further action to determine the cause of the low pressure conditions. The closure of the action request was contrary to Exelon maintenance guidance document MA-MA-716-010-1010, which specifies that the responsible supervisor verify that the work performed satisfies the action request.

In addition to the check of the pressure gauge, there were five other troubleshooting steps in the Fairbanks Morse vendor manual for low jacket water pressure. These steps provided opportunities to identify in-leakage of air or combustion gases, but Exelon did not perform these steps. Exelon also did not evaluate the pressure swinging or fluctuations observed by the operator.

Exelon did not treat the reported swinging as fluctuations. Had the station used the vendor manual troubleshooting guidance for pressure fluctuations, they would have been directed to check for compression leaks into the jacket water system. This check specifically addresses adapter gasket leakage.

Additional insight was available in Fairbanks Morse Service Information Letter A-24 dated May 1, 1990, which stated that combustion gas entering the jacket water system would cause the discharge pressure to fluctuate or drop (emphasis added). Thus, Exelon had vendor information that the observed conditions could be caused by combustion gas entering the jacket water system, but Exelon did not check for this cause.

Following questioning by the team, Exelon reviewed the jacket water temperature data for the March and April tests discussed above. Exelon found a higher than normal temperature differential across the engine, which indicated that the jacket water flow rate was lower than normal. This data provided additional supporting information that there may have been air/gas entrainment, as would be expected for a combustion gas leak into the jacket water system.

The team noted that the first instance of low jacket water pressure occurred following an unplanned, rapid start of the EDG without the normal pre-lubrication.

The September failure of the EDG also occurred following an unplanned, rapid start without pre-lubrication. Exelons investigation did not fully explore whether there was a link between these two occurrences.

2. Multiple Additions to Jacket Water Expansion Tank - Not Investigated

In May through September 2003, operators documented a number of additions to the E2 EDG jacket water expansion tank. The Fairbanks Morse manual states that the jacket water system should be checked for leaks when the expansion tanks require frequent make-up water. The additions may have been explained by pre-existing external jacket water leaks, but Exelon did not conduct a thorough review to rule out the possibility of internal leakage through the adapter gaskets. Also, the external jacket water leakage may have countered a level increase due to combustion gases during surveillance testing in this timeframe.

3. Recommended Actions from 2001 Low Jacket Water Pressure Event - Canceled

by Station Management In October 2001, the E2 EDG tripped on low jacket water pressure because the expansion tank isolation valve was left closed. Although Exelons investigation for this event concluded that the root cause was a human performance error, this team recommended that operators perform additional follow-up and trending of EDG parameters to check for possible air or gas in-leakage to the jacket water system. For instance, the team recommended trending jacket water expansion tank level and heat exchanger venting time. These actions could have identified and quantified any air/gas in-leakage. They were subsequently canceled by station management because they were believed to be unnecessary. However, the actions could have provided an opportunity to identify combustion gas in-leakage.

Extent of Condition Review The team reviewed Exelons assessment of the extent of the leaking gasket condition on other Peach Bottom EDGs and concluded that it was satisfactory. Exelon appropriately recognized that the same procedures, personnel and maintenance practices for installing adapter gaskets were common to all Peach Bottom EDGs. As interim measures, Exelon tested E1, E3, and E4 EDGs using temporary sightglasses installed on the jacket water vent line between the EDG and the expansion tank to look for gas bubbles. Also, they checked jacket water expansion tank levels and sampled the tank air space for combustion gases. These tests showed that there was no detectable combustion gas entering the jacket water system. For long-term corrective action, Exelon intends to install permanent sightglasses, revise the test and maintenance procedures, and replace all adapter gaskets on E1, E3, and E4 during their next 24-month inspections.

Analysis The team identified a performance deficiency for the failure to adequately maintain the E2 EDG between July 1992 and September 2003, with two aspects as described below:

First, the procedure used in July 1992 to install EDG adapter gaskets did not assure that the gaskets provided an adequate seal between the cylinder gases and the jacket water system. The maintenance procedure did not incorporate certain vendor recommendations intended to provide proper sealing of the gaskets. Specifically, the procedure did not include directions to inspect the adapter gaskets prior to installation to ensure that a required annealing process did not cause gasket damage or irregularities.

This recommendation was specified in Fairbanks Morse Service Information letter A-15, dated June 22, 1987. Exelons failure to maintain the maintenance procedure is a performance deficiency. This failure is also an apparent violation of Technical Specification 5.4.1.

Secondly, when opportunities existed in March and April 2003 to identify the problem with the EDG indicated by low, swinging jacket water pressure, actions were not sufficient to correct the condition. Although Exelon initiated a maintenance action request for the observed conditions, they closed the action request without performing several troubleshooting steps recommended by the vendor manual and without resolving the problem. The closure of the action request was also contrary to the guidance in an Exelon maintenance work order standard. Exelons failure to correct a condition adverse to quality is a performance deficiency. This failure is also an apparent violation of 10 CFR 50 Appendix B, Criterion XVI, Corrective Actions.

For the degraded condition, the team determined that the fault exposure time, the time the EDG was unable to perform its safety function, was 35 days. This determination was based on treating the failure as service-related; namely, the gasket degradation occurred with the EDG in service. The EDG failed 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> after starting on September 15, 2003, therefore the EDG was unable to provide power for 23 hours2.662037e-4 days <br />0.00639 hours <br />3.80291e-5 weeks <br />8.7515e-6 months <br /> of its 24-hour mission time. By counting back 23 hours2.662037e-4 days <br />0.00639 hours <br />3.80291e-5 weeks <br />8.7515e-6 months <br /> of service time, the team determined the EDG would have been unable to meet its mission time beginning on August 9, 2003. This approach was based on:

1. The failure mode involved time-related degradation of the gaskets, as

determined by Exelon's laboratory analysis.

2. Gasket relaxation and degradation occurs during starting, operation, and

loading, based on Exelon and Fairbanks Morse information. The gaskets must withstand high differential pressures during both operation and loading.

3. Exelons investigation determined that the gasket failures were not the result of a

single start of the EDG or rapid loading of the EDG.

Exelon estimated the fault exposure period to be 13 days, based on the last successful surveillance test on September 2, 2003. Exelon believed that this test demonstrated that the EDG could run for its mission time. Exelon concluded that the EDG would have failed to meet its mission time following the September 2 run. Exelon performed a preliminary risk significance determination for the EDG failure using the 13 day fault exposure period and concluded that CDF was Green for both units.

The team noted that Exelons position on fault exposure relied on the satisfactory completion of a 2-hour surveillance test that did not include detailed checks of the jacket water expansion tank and the jacket water pressure gauge. These detailed checks may have revealed the presence of gas in the jacket water system that could have led to a failure beyond the 2-hour timeframe. The team also noted that an event-related start and loading of the EDG is more stressful on the engine than a start for a routine surveillance test. The September 15, 2003, event included a rapid start without pre-lube and a rapid increase in load at one hour after the start. By contrast, a routine surveillance test includes pre-lube and gradual loading.

In accordance with Inspection Manual Chapter (IMC) 0612, Appendix B, "Issue Disposition Screening," this finding was more than minor because it was associated with the equipment performance attribute of the Mitigating Systems cornerstone, and it adversely affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events. The finding was evaluated in accordance with IMC 0609, Appendix A, "Significance Determination of Reactor Inspection Findings for At-Power Situations," using Phase 1, Phase 2, and Phase 3 significance determination process (SDP) analysis. The Phase 1 analysis required a Phase 2 evaluation because the finding represented a loss of safety function of a single train, for greater than its allowed outage time. The Technical Specification allowed outage time is 14 days for a single EDG.

The Phase 2 analysis, based on the exposure time of between a year and 30 days for internal events for CDF and LERF, was conducted using the Risk-informed Inspection Notebook for Peach Bottom Units 2 and 3, Revision 1, dated September 15, 2003, and with Draft IMC 0609, Appendix H, Containment Integrity SDP, respectively.

The finding had low-moderate CDF safety significance and substantial LERF safety significance at Unit 2 and very low CDF safety significance and low-moderate LERF safety significance at Unit 3. This Phase 2 evaluation was conservative based on the actual exposure time being about one tenth of a year and because of the assumption that the drywell floor was dry at the time of reactor vessel breach following core damage.

The Region I senior risk analyst conducted a Phase 3 evaluation to address the conservatism, further refine the Phase 2 determination, and to address the impact of external events. The analyst used the 3.01 SPAR model for Peach Bottom Unit 2, information from the licensees updated PRA, and a simplified containment event tree to refine the CDF and LERF estimates. Both CDF and LERF were dominated by the following internal event sequences: 1) a station blackout following a LOOP with successful reactor shutdown and operation of HPCI or RCIC in the short term and no stuck open SRVs; the three remaining EDGs fail and power is not restored through offsite, a recovered EDG, or the Conowingo SBO line within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. 2) a LOOP followed by a successful reactor shutdown and operation of HPCI or RCIC in the short term and no stuck open SRVs, with at least one of the remaining EDGs powering a safety bus and the operator depressurizes the plant successfully; suppression pool cooling, low pressure injection and late injection fail because of power dependencies on the failed EDGs and/or independent failures.

Based on the low relative frequency of a seismic, fire or high wind, floods, and other external events that could cause a LOOP there would not be significant increase in the risk of the finding relative to CDF and LERF. The analysis for internal and external LOOP initiator determined that this finding had low-moderate CDF and LERF safety significance at Unit 2 and very low CDF and LERF safety significance at Unit 3.

As such, the finding represented low to moderate safety significance and was determined to be WHITE at Unit 2 based on CDF and LERF. The finding is GREEN at Unit 3 based on CDF and LERF. The difference between the two units is attributable to differences in electrical bus loads.

Enforcement The team identified two apparent violations:

Technical Specification 5.4.1 states, in part, that written procedures shall be established, implemented, and maintained covering the applicable procedures recommended in Regulatory Guide 1.33, Appendix A, November 1972. These procedures include those for performing maintenance that can affect the performance of safety related equipment.

Contrary to the above, Exelon maintenance procedure M-052-011, Standby Diesel Generator Cylinder Liner Replacement, Revision 1, dated March 24, 1992, did not incorporate certain vendor recommendations intended to provide proper sealing of the gaskets. Specifically, the procedure did not include directions to inspect the adapter gaskets prior to installation to ensure that a required annealing process did not cause gasket damage or irregularities. This recommendation was specified in Fairbanks Morse Service Information letter A-15, dated June 22, 1987.

10 CFR 50 Appendix B, Criterion XVI, Corrective Actions, requires, in part, that measures be established to assure that conditions adverse to quality, such as malfunctions and deficiencies, are promptly identified and corrected. Contrary to the above, when low, swinging jacket water pressure indicators were identified in March and April 2003, actions were not sufficient to correct the condition. Although Exelon initiated a maintenance action request for the observed conditions, they closed the action request without performing several of the troubleshooting steps recommended by the vendor and without resolving the problem.

This finding did not present an immediate safety concern, because operators were able to restore power to the affected safety buses. Exelon has implemented interim measures that included monitoring for combustion gases on all EDGs. Planned corrective actions include procedure changes, replacement of adapter gaskets, and actions to reinforce the need to resolve equipment deficiencies identified in action requests. This issue was entered in the Exelon corrective action program (CAP) as condition report 175881. (AV 05000277/2003013-01, 05000278/2003013-02)2.3 Unit 3 Safety Relief Valves Malfunction of Main Steam Safety Relief Valves (SRV) During Reactor Scram

a. Inspection Scope

As part of the follow-up to this event, the team evaluated the root cause evaluations associated with the malfunctioning SRVs. The team reviewed plant operating logs, computer printouts (i.e., strip charts, sequence of events logs, etc.), system health reports, photos of damaged SRV components, operator training course material for main steam and pressure relief, Plant Operations Review Committee (PORC) meeting minutes, maintenance procedures, maintenance history, calibration check procedure of SRV position switches, work order packages, Target Rock safety/relief valve vendor manual, condition reports, industry operating experience, attended PORC meetings, and interviewed plant engineering personnel.

The team evaluated the technical adequacy associated with Exelons equipment apparent cause evaluation by reviewing Technical Specifications, the Updated Final Safety Analysis Report (UFSAR), and other documentation related to the malfunction of Target Rock safety relief valves on September 15, 2003. A list of documents reviewed by the team is provided in Attachment 1.

Description During the Peach Bottom dual unit reactor scram event on September 15, 2003, several main steam relief valves experienced problems. Specifically, the more significant malfunctions occurred with Unit 3 main steam relief valves RV-3-02-071D and RV-3-02-071G, which are three-stage Target Rock relief valves.

RV-3-02-071D opened on high reactor pressure and failed to close when the control switch was taken to the close position in the control room and remained in the open position for approximately five minutes until the valve closed when reactor pressure decreased to approximately 369 pounds per square inch gauge (psig). Operators entered and completed procedure OT-114, Stuck Open SRV, and pulled the fuses for RV-3-02-071D after the SRV had closed. RV-3-02-071D remained closed throughout the rest of the event and recovery.

RV-3-02-071G, an Automatic Depressurization System (ADS) valve, opened on high reactor pressure and initially closed when control room operators placed the control switch in auto. After initial successful manual openings of RV-3-02-071G during the event on September 15, 2003, the SRV failed to open manually.

Main Steam Relief Valve RV-3-02-071D RV-3-02-071D was removed from Peach Bottom Unit 3 and sent to Wyle Labs for setpoint testing. The setpoint for the valve was 1135 psig and it lifted successfully three times within its acceptable Technical Specification pressure band. However, it was identified prior to testing that the SRV was leaking from the pilot area. The pilot leakage was determined to be approximately 14 lbm/hr. This pilot leakage would delay the second stage closure, which would delay the main valve closure. Exelon experience gained from SRV testing of three-stage Target Rock valves at Limerick, indicates that an SRV will not re-close with leakage between 11 lbm/hr to 15 lbm/hr. Since the leakage from RV-3-02-071D was within this leakage band, Exelon determined this was the reason that RV-3-02-071D remained open.

Exelons initial determination was that the most probable cause of the pilot leakage was improper pilot disc to seat contact. Investigation at Wyle labs identified foreign material in the pilot area. Preliminary identification of the material was iron oxide. Some shinny material was also observed. Examination of the pilot at Wyle Labs indicated that there was no damage to the pilot disc or pilot seat. The lack of damage to the pilot disc and seat would indicate that the pilot leakage was initiated during the September 15, 2003, event. Exelon is working with Wyle labs and Target Rock to further investigate the foreign material and the disc to seat contact to determine the root cause of the pilot leakage. The pilot disc was then sent to Exelon Power Labs where it was examined under a microscope. Although no damage was visible to the eye, under the microscope channeling was identified on one side of the disc that extended through the seating surface. Exelon plans to investigate this further with Target Rock, Wyle Labs and Power Labs.

Main Steam Relief Valve RV-3-02-071G After the Unit 3 drywell became accessible, maintenance technicians identified air blowing from the air operator of RV-3-02-071G. This provided evidence that the air operator diaphragm had failed. RV-3-02-071G was removed from Peach Bottom Unit 3 and sent to Wyle Labs for failure analysis.

The as-found lift setpoint of RV-3-02-071G was 1130 psig, 15 psig below its nominal setpoint of 1145 psig, (-1.3%), which is not within Peach Bottom Technical Specifications specified nominal setpoint of +/- 1%.

During the disassembly of the RV-3-02-071G air operator, technicians noted that the torque on the actuator bolts was low. The eight diaphragm bolts and diaphragm plate looked blue which is indicative of a heating effect. The diaphragm was severely damaged, most likely due to exposure to heat. The air operator had internal steam damage and condensation residue. Further inspection revealed corrosion and condensation residue in the air operator stem and packing area. Areas where the packing had lost contact with the stuffing box were noted. Condensation residue left a clear trail of where the high energy steam leaked by the stem packing and up through the air operator damaging the diaphragm.

Additionally, the packing nuts were identified to be loose. As an extent of condition, examination of the seven other SRV air operators removed in Unit 3 refueling outage 14 (3R14) revealed inconsistent torque of the packing nuts. The packing that is installed in the Target Rock SRV air operator is asbestos Teflon which is no longer manufactured.

Exelon is working with Target Rock to select suitable replacement packing material for the SRV air operator.

Exelon concluded that the failure of RV-3-02-071G was caused by the failure of the air operator diaphragm, preventing the SRV from opening remotely from the control room on control switch demand.

b. Findings

Failure of Main Steam Relief Valve RV-3-02-071G Introduction The team identified a Green non-cited violation (NCV) because maintenance procedures for the installation of the packing on the SRV air operator assembly were inadequate.

The packing installation procedures did not assure that the packing was properly installed (packing nuts adequately tightened, etc.) on RV-3-02-071G resulting in hot steam leaking past the packing and damaging the air operator diaphragm during the September 15, 2003, event.

Description On September 15, 2003, RV-3-02-071G was initially opened and closed manually by operators in the control room, during which time high temperature/pressure steam leaked past the packing into the air operator causing the failure of the diaphragm in the air operator. This resulted in the inability of the SRV to be remotely controlled from the control room during the actual event. The team determined that RV-3-02-071G was vulnerable to this type of failure once the SRV had opened, in the period since maintenance was last performed on the valve, approximately two years earlier.

Target Rock Corporation, Technical Manual M1-JJ-16, Safety/Relief Valve Model 67F contained vague instructions and requirements for packing replacement of the SRV air operator. The procedure required technicians to inspect and replace the packing every 24 months or as required and then assemble the packing in reverse order. The procedure did not contain any specific packing installation instructions or qualitative acceptance criteria, such as leak testing of the packing.

Analysis The team determined that the issue of inadequate maintenance procedures related to SRV packing replacement was a performance deficiency in that 10 CFR 50, Appendix B requires adequate procedures.

This finding is more than minor because it is associated with the Procedure Quality attribute of the Mitigation Systems cornerstone and adversely affects the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. The finding was evaluated in accordance with IMC 0609, Appendix A, "Significance Determination of Reactor Inspection Findings for At-Power Situations," using Phase 1, Phase 2, and Phase 3 significance determination process (SDP) analysis.

This issue is of very low safety significance, based on the Phase 3 analysis results, assuming that RV-3-02-071G would not have opened from the control room for one year. The Phase 1 analysis required a Phase 2 evaluation because the finding represented an actual loss of safety function of a single SRV system. The Phase 2 analysis produced overly conservative results, since there were still10 remaining SRVs that could have functioned as needed. To address the conservatism the Region I senior risk analyst conducted a Phase 3 evaluation, using the 3.01 SPAR model for Peach Bottom, which indicated a negligible increase in risk of not being able to manually depressurize with the 10 remaining valves.

This issue would not have prevented RV-3-02-071G from performing its over-pressure relief (self-actuation) mode function. Exelon entered this issue into their corrective action program as Action Report (AR) 00175894.

Enforcement:

10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings," requires, in part, that activities affecting quality shall be prescribed by documented instructions, procedures or drawings of a type appropriate to the circumstances.

Contrary to the above, activities affecting quality were not prescribed by documented instructions, procedures or drawings of a type appropriate to the circumstances in that effective instructions had not been established for installation of SRV packing to ensure that high energy steam leakage would not leak past the packing. Specifically, Target Rock Corporation, Technical Manual M1-JJ-16, Safety/Relief Valve Model 67F contained vague instructions and requirements for packing replacement of the SRV air operator. The procedure required technicians to inspect and replace the packing every 24 months or as required and then assemble the packing in reverse order. The procedure did not contain any specific packing installation instructions or qualitative acceptance criteria, such as leak testing of the packing. As a result, high energy steam bypassed the packing and significantly damaged the air operator diaphragm of Unit 3 RV-3-02-071G, resulting in the inability to manually operate the SRV during the event on September 15, 2003. Because this violation was of very low safety significance and was entered into the corrective action program, this violation is being treated as a Non-Cited Violation consistent with Section VI.A of the NRC Enforcement Policy: (NCV 05000278/2003013-03).

Corrective Actions Following the event on September 15, 2003, Exelon performed an equipment apparent cause evaluation of the SRV malfunctions. Exelon initiated a number of corrective actions to address the apparent causes of the SRV failures including

evaluate the foreign material and disc to seat contact to determine the cause and duration of the RV-3-02-071D pilot leakage.

evaluate the channeling that was discovered on the RV-3-02-071D pilot disc.

develop improved maintenance, cleaning and inspection procedures and processes.

develop improved packing installation procedure.

The team concluded that the completed and proposed corrective actions were adequate.

2.4 Unit 3 Main Steam Isolation Valve Description The 3D main steam isolation valve (MSIV) is a 26 inch Wye-pattern type valve, manufactured by Atwood & Morrill Co., Inc. The MSIV is actuated with an air cylinder operator (air-to-open and air-and/or spring-to-close) and is an outboard containment isolation valve. In the event of a major leak from a steam line outside of containment, this valve is required to close to maintain primary containment integrity and limit the loss of reactor coolant inventory. Peach Bottom Technical Specifications require a three to five second closure time. The Updated Final Safety Analysis Report (UFSAR) specifies a maximum design isolation time of 10 seconds for a main steam line break outside of containment.

On September 15, 2003, at 1:32 a.m., Unit 3 received a valid Group I isolation signal due to loss of power to both Reactor Protection System (RPS) Motor Generator (MG)sets. This resulted in closure of all inboard and outboard MSIVs, with the exception of the 3D outboard MSIV. At 2:48 a.m., the 3D MSIV closed with no operator action.

Root Cause of Equipment Failures Failure of the 3D MSIV to Close

a. Inspection Scope

The team reviewed the results of Exelons investigation of the failure of the 3D MSIV to close as documented in the respective Equipment Apparent Cause Evaluation to determine if the evaluation was of the proper scope and effectiveness. The team also reviewed documents such as maintenance records and procedures, vendor information, corrective action documents, and lab test results, and interviewed members of the Exelon staff. (A complete list of documents reviewed can be found in Attachment 1).

Licensee Root Cause and Corrective Actions The day following the event, Exelon stroked the 3D MSIV in the as-found condition. The valve stroked smoothly and all stroke times were satisfactory. Inspection of valve position limit switches revealed no deficiencies. Prior to actuator and valve disassembly, Exelon performed diagnostic testing on the valve by measuring stem force and valve position at cold shutdown conditions for both fast and slow closure testing. The resultant traces from these diagnostic tests showed no abnormalities that could explain the cause of the MSIV delayed closure.

Exelon conducted an in-body valve inspection to evaluate for binding or any other anomalies that could have resulted in the delayed closure of the valve. No binding or other potential causes of the failure were found. In-body clearances were measured and this data was transmitted to an Atwood & Morrill representative for review. Based on evaluation of the in-body inspection and consultation with the vendor, Exelon concluded that binding internal to the MSIV was an unlikely failure mode. Therefore, they determined that the most likely cause of the failure was external to the valve body, such as the in actuator and actuator sub-components.

The MSIV air manifold contains a four-way directional control valve, a two-way redundant exhaust valve, AC and DC solenoid valves, and a test valve and associated test solenoid. The purpose of the four-way directional control valve is to direct the flow of operating air to either open or close the associated MSIV. When the AC and DC solenoid valves de-energize and close on a Group I isolation signal, the four-way directional control valve should realign to exhaust the air from the bottom of the air actuator piston, and port air to the top of the piston so that the MSIV will close via air and spring pressure. When the solenoid valves de-energized during this event, it is possible that the four-way valve did not immediately realign and therefore caused delayed closure of the MSIV. The air actuator and manifold were tested by station maintenance using station maintenance procedures. All test results were satisfactory and there was no indication of any delays with any of the components.

The AC and DC solenoid valves were also a potential failure mode in this event. Failure of either the AC or DC solenoid valves to close when de-energized would prevent repositioning of the four-way directional control valve and thus, failure of the associated MSIV to close. All three of these solenoids were removed from the air control manifold and sent to Exelon Power Labs for failure analysis. Testing at temperature revealed no operating deficiencies with any of the solenoids. The solenoids were in good condition, and electrical performance was satisfactory.

Despite extensive troubleshooting, Exelon could not attribute the cause of the delayed MSIV closure to a single element. The solenoids that were removed from the manifold were replaced with new solenoids. Additionally, the air manifold for the 3D MSIV was completely rebuilt to account for any binding that may have occurred during the event.

Following completion of all maintenance, Exelon performed diagnostic testing on the 3D MSIV. The stroke times and results from the diagnostic test were satisfactory. In addition, the other three Unit 3 outboard MSIVs were stroked for the same diagnostic testing, and no abnormalities were found. Exelon currently plans to perform full stroke testing of the 3D MSIV at power, initially on a quarterly basis until there is confidence the valve is operating correctly. Exelon has also installed switch locks that can use a test switch in the control room as a redundant method to slow close the MSIVs.

b. Findings

The team determined that the Exelon investigation into the failure of the 3D MSIV to close was of adequate scope and depth. No findings of significance were identified.

3.0 Human Factors and Procedural Issues 3.1 Inadequate Emergency Operating Support Procedures to Reduce Torus Level

a. Inspection Scope

The team reviewed the plant response and human performance during the September 15, 2003, dual unit automatic shutdown. This review included plant operating logs, the prompt investigation report for the dual unit scram, computer printouts (i.e., strip charts, sequence of events logs, etc.), plant procedures including emergency operating procedures, emergency operating support procedures, abnormal operating procedures and system operating procedures. Licensed senior reactor operators were interviewed as well as operations support staff and operations management personal.

The team evaluated the operators ability to reduce torus level using existing emergency operating procedures and emergency operating support procedures under the conditions during the dual unit automatic shutdown.

b. Findings

Introduction A self revealing non-cited violation of Technical Specification (TS) 5.4.1, "Administrative Controls - Procedures," of very low safety significance (Green) was identified. The existing emergency operating support procedures did not have adequate instructions to reduce torus water level when Group II / III isolation signals were present.

Description Station emergency operating procedure (EOP) support procedures S0 14A.1.A, "Torus Water Filter Pump," AO 10.1, "RHR to Radwaste," and AO 10.2, "RHR to Condenser,"

did not contain adequate steps to reduce torus water level when Group II / III isolation signals were present. Specifically, these support procedures did not contain steps to bypass the isolations and reject torus water to lower torus level for Unit 3 during the event on September 15, 2003.

During the dual unit automatic shutdown on September 15, 2003, the normal heat sink (condenser) was lost due to the Group I isolations which closed the main steam isolation valves. This resulted in the reactor decay heat being discharged into the torus through the safety relief valves (SRVs), HPCI and RCIC. The condensing steam in the torus increased the torus level and reduced the "free air space" in primary containment.

Since the nitrogen was not vented, the reduction in the "free air space" compressed the nitrogen and resulted in an increase in primary containment pressure. Emergency operating procedure T-102, "Primary Containment Control," was entered initially on Unit 3 when torus temperature exceeded 95° F (EOP entry condition). Operator actions to vent the drywell were delayed due to equipment problems encountered during the event, and therefore as the torus level continued to increase the primary containment pressure exceeded 2 psig. With containment pressure above 2 psig the Group II / III isolation signals could not be reset or bypassed using existing procedures, which would could have allowed the RHR system or torus filter pump to be used to reject torus water and reduce torus level and drywell pressure.

The inadequate support procedures resulted in a delayed reduction of torus water level and containment pressure while a procedure was developed. It took 14 hours1.62037e-4 days <br />0.00389 hours <br />2.314815e-5 weeks <br />5.327e-6 months <br /> to develop a procedure to perform the task. The delay in the reduction of torus water level kept drywell pressure above 1.68 psig. With drywell pressure above 1.68 psig, operators did not want to reduce reactor pressure to below 450 psig to avoid the automatic starting of the low pressure emergency core cooling systems that were not needed. This would needlessly further complicate operator actions. Ultimately the inadequate support procedures delayed the operators in establishing cold shutdown conditions on Unit 3.

Analysis EOP support procedures were not adequate in that they did not contain steps to bypass the containment isolation signals and allow implementation of EOP T-102, "Primary Containment Control." Inadequate support procedures is a performance deficiency because TSs require these support procedures to be adequately established.

Traditional enforcement does not apply for this issue because it did not have any safety consequence or the potential for impacting the NRCs regulatory function, and was not the result of any willful violation of NRC regulations.

This finding is more than minor because it is associated with the Procedure Quality attribute of the Mitigating Systems cornerstone and adversely affects the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. It delayed the operators in placing Unit 3 into a cold shutdown condition. This finding is of very low safety significance (Green) using Phase 1 of the Significance Determination Process (SDP) for reactor inspection findings for At-Power reactor situations. The finding is of very low safety significance because the finding is not a design or qualification deficiency, it does not result in a loss of safety function, and is not potentially risk significant due to seismic, flood, fire, or weather related initiating event Exelon entered this issue into their corrective action program as Condition Report (CR) 176420, assignment 13.

Enforcement Technical Specification Section 5.4.1, "Administrative Controls - Procedures," requires that written procedures be established, implemented, and maintained covering

(b) the emergency operating procedures required to implement the requirements of NUREG-0737 and NUREG-0737, Supplement 1. Contrary to this requirement the existing emergency operating support procedures were not adequately established in that they did not contain steps to bypass the containment isolations and allow the reduction of torus level. This violation of Technical Specifications 5.4.1

(b) is being treated as a non-cited violation consistent with section VI.A.1 of the NRC Enforcement Policy. (NCV 05000278/2003013-04)3.2 Component Cyclic or Transient Limit Program Not Correctly Implemented

a. Inspection Scope

The team reviewed the plant response and human performance during the September 15, 2003, dual unit automatic shutdown. This review included plant operating logs, the prompt investigation report for the dual unit scram, computer printouts (i.e., strip charts, sequence of events logs, etc.), plant procedures including emergency operating procedures, emergency operating support procedures, abnormal operating procedures, system operating procedures, surveillance procedures, engineering analysis, and condition reports. Licensed senior reactor operators were interviewed as well as operations support staff and operations management personal.

The team evaluated the operation of the high pressure coolant injection (HPCI) and reactor core isolation cooling (RCIC) systems during the event and evaluated Exelons method of accounting and analyzing the component cycling on the feedwater nozzles due to operation of the HPCI / RCIC systems.

Description Technical Specification 5.5.5, "Administrative Controls - Component Cyclic or Transient Limit," is implemented through station procedure ST-J-080-940-3, "Reactor Pressure Vessel Transient Cycles Record." This program requires tracking cyclic and transient occurrences that are specified in the Updated Final Safety Analysis Report (UFSAR)table 4.2.4, "Reactor Design Cycles (40 Year Life)," to ensure that the reactor vessel stress analysis and load combinations remain bounded by the UFSAR analysis.

Specifically, the number of HPCI and RCIC injections are required to be tracked to ensure that the stress and fatigue analysis remains bounding for the weld connecting the safe end of the reactor vessel feedwater nozzle and the feedwater piping. The UFSAR Table 4.2.4 as well as the procedure listed the number of HPCI and RCIC cycles as 20 over the forty year life of each plant.

During the dual unit automatic shutdown on September 15, 2003, the Unit 3 RCIC system injected into the vessel approximately 46 times, which exceeded the 20 cycles specified in the UFSAR and procedure. However, after reviewing the data from the September 15 event, a review of past events identified that the total number of HPCI and RCIC injections were approximated at 105 injections on Unit 2 and 75 injections on Unit 3. Exelon did not properly count the number of HPCI and RCIC injections because they did not adequately translate analytical assumptions and design basis parameters into procedure ST-J-080-940-3, "Reactor Pressure Vessel Transient Cycles Record."

This resulted in Exelon improperly implementing Technical Specification (TS) 5.5.5, "Administrative Controls - Component Cyclic or Transient Limit," program.

Since the number of HPCI and RCIC injections exceeded the number specified in the UFSAR, Exelon performed an analysis to verify that the reactor vessel stress analysis and load combinations were bounded by the UFSAR analysis. The analysis determined that the design transient profiles for the Hot Standby event (on / off feedwater injections)conservatively bounded the HPCI and RCIC injections. The Hot Standby event was analyzed for a total of 2600 repetitive on / off feedwater injections. Currently each unit has experienced approximately 1250 feedwater on / off injections and because of plant modifications the plant is no longer susceptible to this type of feedwater cycling. Since the feedwater injections bounded the HPCI and RCIC injections, the HPCI and RCIC injections were combined with the feedwater injections. This resulted in a total of approximately 1400 injections on both units, which is less than the 2600 injections considered in the analysis. Therefore, Exelons analysis of the HPCI and RCIC injections confirm that the original reactor vessel stress analysis and load combinations remains bounded by the UFSAR analysis. In addition, since the plants are no longer susceptible to the on / off feedwater injections the total number of HPCI and RCIC injections allowed by the analysis is approximately 1200 cycles. Based on past operation of HPCI and RCIC and the infrequent use of the HPCI and RCIC system for vessel injection it is not expected that the number of injections will exceeded the number in the analysis. Exelon entered this issue into their corrective action program as Condition Report CR 179199.

b. Findings

This finding was considered a minor violation of Technical Specification (TS) 5.5.5, "Administrative Controls - Component Cyclic or Transient Limit." TS Section 5.5.5 requires Exelon to count the number of HPCI / RCIC injections and verify that the injections remain below the injections specified in the Updated Final Safety Analysis Report (UFSAR) table 4.2.4, "Reactor Design Cycles (40 Year Life)," to ensure that the reactor vessel stress analysis and load combinations remain bounded by this analysis.

Contrary to this, Exelon failed to correctly account for all the injections that had occurred during this event and prior to this event. This improper accounting resulted in Exelon exceeding the number of HPCI and RCIC injections specified in the analysis. Exelon performed a reanalysis and determined that reactor vessel stress analysis and load combinations remains bounded and should remain bounding in the future. Therefore, this violation is considered minor because the finding could not have resulted in a more significant event, even if left uncorrected; the finding had a minimal impact on the Barrier Integrity (reactor pressure vessel) objective and it is not associated with a performance indicator.

3.3 Ineffective Corrective Actions Resulted in Draining the Condensate Storage Tank to the Condenser Hotwell

a. Inspection Scope

The team reviewed the plant response and human performance during the September 15, 2003, dual unit automatic shutdown. This review included plant operating logs, the prompt investigation report for the dual unit scram, computer printouts (i.e., strip charts, sequence of events logs, etc.), plant procedures including emergency operating procedures, emergency operating support procedures, abnormal operating procedures, system operating procedures, surveillance procedures, engineering analysis, and condition reports. Licensed senior reactor operators were interviewed as well as operations support staff and operations management personal.

The team evaluated the issues associated with the automatic transfer of the HPCI and RCIC suction from the condensate storage tank (CST) to the torus on Unit2 during the event and compared this event to previous similar events that occurred over the past eight years.

b. Findings

Introduction The team reviewed a self revealing finding of very low safety significance (Green)because Exelon did not correct a previously known equipment deficiency with the Unit 2 "B" condenser hotwell level instrument as required by Exelons corrective action program. The equipment deficiency caused the draining of the condensate storage tank (CST) to the condenser hotwell which resulted in the automatic transfer of the High Pressure Cooling Injection (HPCI) and Reactor Core Isolating Cooling (RCIC) Systems suction from the CST to the torus earlier than expected. The automatic transfer of the suction to the torus was unexpected and therefore resulted in an added operational burden for the operators.

Description During the dual unit automatic shutdown on September 15, 2003, Unit 2 experienced a momentary loss of offsite power. The B hotwell level controller, LT-2085 B was the controlling channel. The momentary loss of power resulted in LT-2085 B failing low and when power was restored the B level channel indication remained low. This resulted in draining the water in the CST to the hotwell. Within approximately 25 minutes the CST level decreased to the low level setpoint, resulting in transfer of the HPCI and RCIC suction automatically to the torus.

Operation of HPCI and RCIC with a suction from the torus and at an elevated torus temperature can lead to pump cavitation due to reduced net positive suction head and can also lead to breakdown of the HPCI and RCIC control / lube oil since the turbine oil coolers are supplied from the pump suction water. This breakdown can lead to turbine control valve problems as well as bearing damage. During this event the maximum torus temperature while HPCI was aligned to the torus was approximately 120°F. If HPCI continued to operate at this torus temperature, the expected control / lube oil temperature would not have reached temperatures that would have exceeded the oil rating. In addition, the operators as well as computer data determined that HPCI and RCIC operated as expected.

The team determined that Exelon had experienced similar problems with LT-2085 B in the past. Specifically, on August 15, 1995, AR0952261 identified hotwell level LT-2085 B, failed to 9 inches after a brief loss of power. The "A" hotwell level LT-2085 A was selected for control and the hotwell level stabilized. A calibration check of LT-2085 B was performed and found to be in calibration.

On November 30, 1998, action request AR1184096 identified that during an electrical transient, with the "B" hotwell level in control, the hotwell level indication failed to 10 inches and did not track actual level. The "A" and "C" indications increased as the hotwell level control valves opened fully to make up level. The "A" hotwell level indication was placed in control and hotwell level was restored to normal. The cause of the level indication error was suspected to be a binding displacer. LT-2085 B was monitored for approximately two days and determined to be functioning correctly. No additional maintenance was performed on this instrument.

Analysis The team determined that Exelon did not correct an LT-2085 B failure mode on momentary loss of power as required by Exelons corrective action program. This is a performance deficiency. Traditional enforcement does not apply for this issue because it did not have any safety consequence or the potential for impacting the NRCs regulatory function, and was not the result of any willful violation of NRC regulations.

This finding was considered more than minor because it was associated with the Equipment Performance attribute of the Mitigating Systems cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability and capability of systems that respond to initiating events to prevent undesirable consequences. This finding was determined to be of very low safety significance (Green) using Phase 1 of the Significance Determination Process (SDP) for reactor inspection findings for At-Power reactor situations. The issue was of very low safety significance because the finding was not a design or qualification deficiency, the finding did not represent an actual loss of safety function, and the finding did not screen as potentially risk significant due to a seismic, fire flooding, or severe weather initiating event. Exelon entered this issue into their corrective action program as Action Report (AR) 00177717. (FIN 05000277/2003013-05)

Enforcement No violation of regulatory requirements occurred. The team determined that the finding did not represent a noncompliance because it occurred on nonsafety-related plant equipment.

3.4 AO 1A.2-2, "Closing a Stuck Open Outboard Main Steam Isolation Valve," Not Properly Implemented

a. Inspection Scope

The team reviewed the plant response and human performance during the September 15, 2003, dual unit automatic shutdown. This review included plant operating logs, the prompt investigation report for the dual unit scram, computer printouts (i.e., strip charts, sequence of events logs, etc.), plant procedures including emergency operating procedures, emergency operating support procedures, abnormal operating procedures, system operating procedures, surveillance procedures, and condition reports. Licensed senior reactor operators were interviewed as well as operations support staff and operations management personal.

The team evaluated the issues associated with implementation of station procedure AO 1A.2-2, "Closing a Stuck Open Outboard Main Steam Isolation Valve."

Description On September 15, 2003, Unit 3, "D" outboard Main Steam Isolation Valve (MSIV) failed to initially close with an isolation signal present. The Unit 3 control room supervisor (CRS) directed two non-licensed operators (NLOs) to close the valve using AO 1A.2-3, "Closing a Stuck Open Outboard Main Steam Isolation Valve," section 4.1, which removed power to the valves AC and DC powered solenoid valves. Removal of power to these solenoid valves should cause the MSIV to close. The operators identified the correct cabinet and fuse block; however, the operators noted that fuse F4 on terminal board BB did not have a standard emergency operating procedure label identifying the specific fuse. Since the fuse did not have the label, they did not remove fuse BB/F4 because they were not certain that it was the correct fuse. The operators reported this condition to the CRS. The CRS had the NLOs stop executing the procedure at this point because the MSIV was now closed.

Exelon identified fuse labeling as a contributing factor to the operators inability to complete this procedure. Specifically, fuses that are used to implement emergency operating procedures are clearly labeled as required. Fuses that are utilized to implement abnormal operating procedures are not required to be labeled in the same manner as the EOP fuses. Exelon subsequently labeled the fuses that are used for AO 1A.2-3 in the same manner as the emergency operating procedure labels. Exelon also revised the procedure to provide more detail on the location of the fuses in the local cabinet. Exelon documented this condition in CR 176420.

b. Findings

This finding is a minor violation of Technical Specification (TS) 5.4.1, "Administrative Controls - Procedures," was identified due to not properly implementing station procedure AO 1A.2-3, "Closing a Stuck Open Outboard Main Steam Isolation Valve."

The failure to properly implement this procedure is minor because use of this procedure was not needed to close the open isolation valve.

3.5 Torus Cooling was not Maximized on Unit 3 After Unit 2 Torus Temperature was Less Than 95° F.

a. Inspection Scope

The team reviewed the plant response and human performance during the September 15, 2003, dual unit automatic shutdown. This review included plant operating logs, the prompt investigation report for the dual unit scram, computer printouts (i.e., strip charts, sequence of events logs, etc.), plant procedures including emergency operating procedures, emergency operating support procedures, abnormal operating procedures, system operating procedures, and condition reports. Licensed senior reactor operators were interviewed as well as operations support staff and operations management personal.

The team evaluated the implementation of station procedure T-102, "Primary Containment Control," to maximize torus cooling.

Description During the dual unit automatic shutdown on September 15, 2003, the normal heat sink was lost and the reactor decay heat was added to the torus using SRVs, HPCI and RCIC. Unit 3 torus water temperature exceeded 95° F at 1:46 a.m. which required plant operators to "maximize" torus cooling in accordance with emergency operating procedure T-102, "Primary Containment Control." Initially establishing torus cooling on Unit 3 was complicated by the status of the plant electrical system and the unexpected trip of the E2 Emergency Diesel Generator (EDG). At 11:26 a.m. the Unit 2 torus water temperature decreased below 95° F; however, the Unit 3 torus water temperature remained at 107° F, and trending down slowly. The operators continued to operate torus cooling on Unit 2, instead of stopping the Unit 2 RHR pumps and starting additional Unit 3 RHR pumps to maximize torus cooling as required by T-102, "Primary Containment Control." Since torus cooling was not maximized on Unit 3, torus water temperatures remained elevated until approximately 4:00 p.m., when the torus water temperature decreased below 95° F.

b. Findings

This finding is considered a minor violation of Technical Specification (TS) 5.4.1, "Administrative Controls - Procedures," due to not correctly implementing emergency operating procedure T-102, "Primary Containment Control." Operations personnel did not maximize torus cooling on Unit 3 after Unit 2 torus water temperature decreased below 95° F. The operators continued to focus on restoration activities for the main condenser, which had a higher priority because it would completely remove the decay heat input into containment, and in addition, the Unit 3 torus temperature was slowly decreasing. This violation is considered minor because the finding could not have resulted in a more significant event, even if left uncorrected; the finding had a minimal impact on the Barrier Integrity (torus) objective; and it is not associated with a performance indicator.

4.0 Emergency Preparedness

a. Inspection Scope

On September 15, 2003, at 2:39 a.m., the operations Shift Manager declared an Unusual Event (UE), based on discretion, after the E2 EDG tripped. The Emergency Response Facilities were activated to support actions to place both plants in stable conditions. At 10:46 a.m., the UE was terminated based on restoration of the electrical distribution system for both units, stabilization of primary containment parameters, and indications of decreasing drywell pressure. The team reviewed the circumstances surrounding the event declaration to determine if it was appropriate and timely, and the event termination to determine if it was appropriate. Documents reviewed are listed in

1.

b. Findings

Offsite electrical power was lost to all 4 kV safeguard buses for both units at 1:32 a.m..

The shift manager evaluated plant conditions and decided not to declare a UE at that time since power was unavailable to the transformers for a very short time (less than one minute). Emergency Action Level MU1, Loss of AC Power, requires that a UE be declared for the loss of all offsite AC power for >15 minutes to all 4 kV safeguard buses.

The team concluded that power was available to the safeguard buses in well under 15 minutes from the initial loss of power, and therefore, the Emergency Action Level was not met.

Following the trip of the E2 EDG at 2:35 a.m., the shift manager declared a UE, based on Emergency Action Level HU6, Discretionary, which requires that a UE be declared if conditions indicate a potential degradation in the level of plant safety. The shift manager concluded that this declaration was appropriate since he did not know the cause for the loss of power to the safeguard buses, which caused a dual unit scram, and the subsequent loss of one EDG. Though not required to be staffed and activated at the UE level, the Technical Support Center, Operations Support Center, and the Emergency Operations Facility were staffed and activated to support control room personnel in troubleshooting activities, restoration of electrical distribution lineups, and stabilization of containment parameters. The team concluded that this was a prudent use of the Emergency Response Organization for assistance during this complicated event.

5.0 Generic Issues During this inspection, no significant issues were identified requiring the issuance of generic communications to the nuclear industry.

6.0 Risk Significance of the Event The initial risk assessment for Unit 3 resulted in a conditional core damage probability (CCDP) of low E-3. The initial Unit 2 CCDP was low E-4. Subsequently, Exelon completed a detailed risk assessment for the event. Using their normal offsite power recovery probability they estimated CCDPs of 1.6E-4 for Unit 2 and 7.3 E-5 for Unit 3.

The Exelon assessment then took credit for 100 percent recovery of offsite power within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> resulting in CCDPs of 9.9 E-6 for Unit 2 and 7.3 E-6 for Unit 3.

The team gathered information concerning plant electrical bus loads and ESW/ECW pump capabilities. This information was factored into a revised SPAR Revision 3.02 model and NUREG/CR 5496 offsite power recovery probabilities given a plant centered LOOP were included. The SPAR model was based on Unit 2 electrical bus loads not on Unit 3 loads. Exelons PRA model indicated a lower overall core damage probability at Unit 3 and that the E2 EDG was less risk important at Unit 3 than at Unit 2. These differences appeared to be due to differences in electrical bus loads between the two plants.

To account for the offsite power circumstances on September 15, 2003, the team modified the Unit 2 SPAR model to account for the ability to re-energize the eight safety buses (four per unit) if the associated EDG failed. The team found that following the initial LOOP, one source of offsite power (343 SU) was available to the eight buses almost immediately. Depending on the specific plant and bus in question, the bus could have transferred automatically to the 343 SU source or the operators would have been able, from the control room using existing procedures, to quickly supply power from 343 SU by closing one circuit breaker. The Unit 2 model was changed to allow for the possibility of re-energizing any of the safety buses either automatically or manually from 343 SU.

Based on the SPAR model changes and incorporation of the probabilities for recovering the safety buses the revised estimate of CCDP was 1.3 E-5 for Unit 2 with only the E2 EDG failing to run. The Unit 2 SPAR model could not directly be used to determine the CCDP for Unit 3. However, the team used the CCDPs for Unit 2 and Unit 3 from Exelons PRA model for similar failure conditions ( E2 EDG failure to run and one SRV stuck open) to establish a ratio. Multiplying the Unit 2 SPAR CCDP result for the E2 EDG failing to run and an SRV sticking open (4.1 E-5) by the Unit 3 CCDP/Unit 2 CCDP (7.3 E-6 / 1.0 E-5 or about 73%) the estimate for CCDP at Unit 3 was approximately 3 E-5.

The following chronology showed the progression of the LOOP and the recovery of offsite power to the 8 safety buses at Peach Bottom on September 15, 2003:

1:32 LOOP 343 SU and 2 SU offsite sources experience degraded grid voltage - all EDGs start.

1:33 343 SU source is reenergized and is available to all 8 safety buses.

Almost immediately the undervoltage condition sensed on the supply side of the eight breakers from this source is cleared. The 2 SU supply breaker to the site SU 25 opens, and 2 SU is not available to the plants.

At this point if an EDG tripped the buses would have behaved as follows based on the offsite power breaker control logic (the normal offsite supply for each bus is in parenthesis):

Unit 2:

E12 (2 SU) E212 tripped on loss of 2 SU. Alternate E312 would have auto closed powering from 343 SU.

E32 (2 SU) E232 tripped on loss of 2 SU. Alternate E332 would have auto closed powering from 343 SU.

E22 (343 SU) E322 tripped on loss of 343 SU, it could be manually closed by the operator. Alternate E222 would not close without 2 SU power.

E42 (343 SU) E342 tripped on loss of 343 SU, it could be manually closed by the operator. Alternate E242 would not close without 2 SU power.

Unit 3:

E23 (2 SU) E223 tripped on loss of 2 SU. Alternate E323 would have auto closed powering from 343 SU.

E43 (2 SU) E243 tripped on loss of 2 SU. Alternate E343 would have auto closed powering from 343 SU.

E13 (343 SU) E313 tripped on loss of 343 SU, it could be manually closed by the operator. Alternate E213 would not close without 2 SU power.

E33 (343 SU) E333 tripped on loss of 343 SU, it could be manually closed by the operator. Alternate E233 would not close without 2 SU power.

2:34 E2 EDG Trips due to low jacket cooling water pressure, resulting in:

Unit 2: E2 EDG output breaker E22 opens and bus E22 was dead as discussed above.

Unit 3 E2 EDG output breaker E23 opens and bus E23 is repowered from 343 SUE when E323 closes, as discussed above.

3:15 Bus E22 is reenergized from 343 SU by closing its normal supply breaker E322.

06:01 SU 25 reclosed aligning 2 SU to the supply breakers of the eight safety buses. At this point all the safety buses that were being carried from the E1, E3 and E4 EDGs would have auto closed to their alternate supply if an EDG tripped. Buses E22 and E23, if degraded voltage was sensed, would have tried to fast transfer to the other source.

7:16 - 8:00 The safety buses were paralleled with offsite power and the EDGs secured as follows:

Unit 2:

E12 7:16 E212 closed// E1 with 2 SU 7:27 E1 output breaker E12 open E32 7:41 E232 closed//E3 with 2 SU 7:46 E3 output breaker E32 open E42 7:54 E342 closed//E4 with 343 SU 7:58 E4 output breaker E42 open Unit 3:

E13 7:41 E313 closed//E1 with 343 SU 7:31 E1 output breaker E13 open E33 7:47 E333closed//E3 with 343 SU 7:48 E3 output breaker E33 open E43 8:00 E243 closed//E4 with 2 SU 8:07 E4 output breaker E43 open 7.0 Overall Adequacy of Licensee Response The team concluded that the overall response of Exelon to the dual unit scrams on September 15, 2003, was adequate in that the plants were safely taken to a cold shutdown condition. The Unusual Event declaration was warranted and appropriate.

However, the operators were challenged by equipment and procedural problems. Prior to the event, there were lapses in equipment performance monitoring, such as for the offsite power protection circuits and the emergency diesel generators. Additionally, some degraded conditions and incomplete resolution of problems has been tolerated.

During the event, some procedures were inadequate and delayed operator actions, such as lowering the torus water level. Exelons corrective actions regarding equipment and procedures for this event have been appropriate. However, this event underscores the need to continue with strong equipment reliability improvements.

8.0

Exit Meeting Summary

The NRC presented the results of this special inspection to Mr. John Skolds and members of Exelon's management on November 18, 2003, at a meeting open for public observation. Exelon management acknowledged the findings presented. No proprietary information was identified.

ATTACHMENT A

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

S. Beck, Emergency Preparedness Manager

W. Coyle, Electrical and I&C Manager, Corporate Engineering

W. Dalton, Operations Shift Supervisor (SRO)

M. Gallagher, Director, Licensing and Regulatory Affairs

J. Gyrath, Senior Staff Engineer, Peach Bottom Station

J. Hallenbeck, Senior Engineer

A. Hegedus, Manager, Plant Engineering

M. Hochreiter, Engineering Systems Manager

A. Javorik, Site Maintenance Director

D. Keene, Manager, Plant Engineering - Electrical

E. Kriner, Technical Specialist

G. Krueger, Senior Staff Engineer

J. Liddy, Senior Engineer, Relay & Protection Services, Exelon Power Delivery

J. McLaughlin, EDG System Manager

D. Warfel, Senior Manager, Engineering

D. Weaver, Manager, Relay & Protection Services, EED

LIST OF ITEMS

Opened

05000277/2003013-01 AV Failure to adequately maintain the E2 Emergency Diesel Generator (Section 2.2)

05000278/2003013-02 AV Failure to adequately maintain the E2 Emergency Diesel Generator (Section 2.2)

Opened and Closed

05000278/2003013-03 NCV Ineffective Instructions for Installation of SRV Packing (Section 2.3)

05000278/2003013-04 NCV EOP Support Procedures Not Adequately Established With Steps to Bypass Containment Isolations (Section 3.1)

Attachment

05000277/2003013-05 FIN Inadequate Corrective Actions to Correct a Hotwell Level Controller (Section 3.3)

Closed

None

Discussed

None