IR 05000247/2003013

IR 05000247-03-013, IR 05000286-03-010; on 08/11/2003 - 10/24/2003, for Indian Point Energy Center, Units 2 and 3; 95001, 93812. Package Accession No. ML03510006

ML033570386
Person / Time
Site:	Indian Point
Issue date:	12/22/2003
From:	Brian Holian Division Reactor Projects I
To:	Dacimo F Entergy Nuclear Northeast
Shared Package
ML03510006	List: IR 05000247/2003013 ML033520105
References
FOIA/PA-2006-0299 IR-03-010, IR-03-013
Download: ML033570386 (47)

Text

ber 22, 2003

SUBJECT:

INDIAN POINT NUCLEAR GENERATING UNITS 2 AND 3 - NRC SPECIAL INSPECTION REPORT 05000247/2003013 AND 05000286/2003010

Dear Mr. Dacimo:

On October 24, 2003, the US Nuclear Regulatory Commission (NRC) completed a special inspection at the Indian Point Nuclear Generating Station. The enclosed inspection report documents the inspection findings, which were discussed on November 7, 2003, with you and other members of your staff.

The purpose of this special inspection was to assess the electrical system disturbances at Unit 2 and Unit 3 over the past 18 months. The team inspected the adequacy of Entergys root cause evaluations, including the adequacy of completed and planned corrective actions. The team independently evaluated equipment and human performance issues that surfaced during disturbances on August 3 and August 14, 2003, which led to reactor plant scrams. Additionally, the team performed the supplemental inspection, called for by the NRC Inspection Manual, for the recent Unit 3 unplanned scrams Performance Indicator threshold change from Green to White. A copy of the Special Inspection Charter is attached to the enclosed inspection report.

Most of the electrical disturbances experienced at Indian Point 2 in the past three years have resulted from protective relay failures in transmission and distribution systems located off-site.

The team observed that, overall, Entergys investigations and associated corrective actions were appropriate. The team noted that the Entergy staff has been working more closely with Consolidated Edison since mid-2003 to address the need for improved 345 kV, 138 kV, and 13.8 kV grid protective relaying reliability and to evaluate means to improve the overall Buchanan switchyard 345 kV ring bus resilience and fault protection scheme. Performance concerns identified during this inspection are similar to those underlying human performance and corrective action issues addressed in our mid-cycle performance assessment letter, dated August 27, 2003. The NRC will continue to monitor Entergys progress on both switchyard enhancements and performance improvements in these cross-cutting areas.

Mr. Fred Dacimo 2 The team also reviewed a self-revealing issue involving the readiness of Entergys Emergency Response Facilities (ERF) which surfaced during the August 14, 2003 event. The Emergency Response Organization (ERO) was challenged by equipment problems that degraded communications and limited automated data acquisition and assessment capabilities during the event. While the ERO dealt effectively with this event through established compensatory measures, and prompt action was taken following the event to fix specific equipment failures, these ERF equipment problems reinforce the need for Entergy to continue its efforts to reduce maintenance backlogs at the site.

The team concluded that Entergys evaluation of the recent Unit 3 unplanned reactor scrams, which contributed to the Performance Indicator threshold change from Green to White, was appropriately self-critical. Entergys corrective actions include several actions to further improve contractor oversight, which is an area that has historically been challenging for the station. The effectiveness of these corrective actions will be monitored via our baseline inspection program.

We consider the Inspection Procedure (IP) 95001 supplemental inspection activities completed for this issue.

Based on the results of this inspection, the inspectors identified six findings of very low safety significance (Green) which did not present an immediate safety concern. One of the findings was determined to be a violation of NRC requirements. However, because this issue is of very low safety significance, and because the issue has been addressed and entered into your corrective action program, the NRC is treating this issue as a non-cited violation, in accordance with Section VI.A.1 of the NRCs Enforcement Policy. If you deny this non-cited violation, you should provide a response with the basis for your denial, within 30 days of the receipt of this letter, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-001; with copies to the Regional Administrator, Region 1; the Director, Office of Enforcement; and the NRC Resident Inspector at the Indian Point 2 facility.

In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). Should you have any questions regarding this report, please contact Mr. David Lew at 610-337-5120.

Sincerely,

/RA/

Brian E. Holian, Deputy Director Division of Reactor Projects Docket No. 50-247 License No. DPR-26

Enclosure:

Inspection Report 05000247/2003013 and 05000286/2003010 w/Attachment: Supplemental Information

REGION I==

Docket No. 50-247 50-286 License No. DPR-26 DPR-64 Report No. 05000247/2003013 05000286/2003010 Licensee: Entergy Nuclear Operations, Inc.

Facility: Indian Point Nuclear Generating Unit 2 Indian Point Nuclear Generating Unit 3 Location: Buchanan, New York 10511 Dates: August 11, 2003 - October 24, 2003 Inspectors: W. Cook, Senior Project Engineer, Team Leader G. Morris, Senior Electrical Inspector M. Cox, Resident Inspector D. Jackson, License Examiner T. Jackson, Project Engineer (part-time)

T. Fish, Senior License Examiner (part-time)

J. DAntonio. License Examiner (part-time)

J. Vora, Senior Electrical Engineer (part-time)

P. Habighorst, Senior Resident Inspector (part-time)

S. Lewis, Reactor Engineer (part-time)

D. Silk, Senior Emergency Preparedness Inspector (part-time)

W. Schmidt, Senior Reactor Analyst (part-time)

Approved by: David C. Lew, Chief Projects Branch 2 Division of Reactor Projects i Enclosure

SUMMARY OF FINDINGS

IR 05000247/2003-13, 05000286/2003-010; 08/11/2003 - 10/24/2003, Indian Point Energy

Center, Units 2 and 3; 95001, 93812.

The report covered several weeks of on site and in-office inspection by the resident, region-based, and headquarters-based inspectors. Six Green findings, of which one was a non-cited violation, were identified. The significance of the findings are indicated by their color (Green,

White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Cornerstone: Initiating Events

The U.S. Nuclear Regulatory Commission (NRC) performed a supplemental inspection to assess the licensees evaluation associated with the Green to White threshold change of the Unplanned Scrams per 7000 Critical Hours Performance Indicator (PI) at Unit 3. The PI threshold was crossed when Unit 3 tripped on June 22, 2003, as a result of the failure of the 345 kV breaker No. 3. During this supplemental inspection, conducted in accordance with Inspection Procedure (IP) 95001, the team determined that Entergy performed a comprehensive review of the specific issues involving the breaker No. 3 failure and the broader issues involving other recent unplanned reactor scrams at Unit 3. Entergy identified, through an apparent cause determination process, that the common theme of a lack of direct contractor oversight and quality control measures, along with the absence of Entergy subject matter experts to independently assess contracted work activities, contributed to the unplanned reactor scrams (a self-revealing Green finding associated with this issue is discussed below). Based upon Entergys acceptable performance in addressing the root and contributing causes for the individual scrams and the common causal factors involving the series of recent scrams which resulted in the PI change, we consider the supplemental inspection activities completed for this issue.

The special inspection review of the numerous grid-related reactor scrams at Unit 2, including the dual unit scram on August 14, 2003, identified a few performance deficiencies, as discussed in the subsequent paragraphs. Overall, the team noted that Entergy was working closely with Consolidated Edison (Transmission and Distribution operator) to address the need for improved 345 kV, 138 kV, and 13.8 kV grid protective relaying reliability and to evaluate means to improve the overall Buchanan switchyard 345 kV ring bus resilience and fault protection scheme.

The team noted that Entergy was cognizant of the potential impact of the plant specific grid-related events on their Individual Plant Examination (IPE) initiating events frequency modeling assumptions. The team learned that the Unit 2 IPE risk model is currently being converted from the large event/small fault tree to small event/large fault tree methodology, to be consistent with the Unit 3 IPE. Coincident with this conversion, Entergy plans to update their model with the latest plant specific and industry operating history event frequency data. The team observed that for the Unit 2 IPE, the contribution from the loss of offsite power (LOOP) events (a station blackout and coincident general transient) to the overall core damage frequency (CDF) is approximately 19 percent (5.926E-6 of 3.13E-5 total CDF). For the Unit 2 grid-related reactor iii

Summary of Findings (contd)trip events reviewed by the team, the station emergency diesel generators operated, as designed, and precluded a station blackout event. The NRC resident inspectors will follow-up the licensees efforts to update the initiating events frequency assumptions, considering the data from the recent grid-related events.

Cornerstone: Initiating Events

Green.

Poor maintenance work practices (failure to follow vendor manual instructions)and insufficient contractor oversight (monitoring, quality verification, and knowledge of work activity) contributed to this self-revealing finding involving the failure of the 345 kV circuit breaker No. 3 on November 15, 2002 and June 22, 2003.

This finding is greater than minor because it is associated with improperly performed maintenance which directly impacted the Initiating Events Cornerstone. The June 22, 2003, breaker failure resulted in the Unplanned Scrams in 7000 Critical Hours Performance Indicator exceeding the Green to White threshold. This finding is of very low safety significance because, even though both breaker failures resulted in reactor trips, the inadequately performed maintenance did not contribute to the likelihood of LOCA initiator; did not contribute to the combination of both a reactor trip and the unavailability of accident mitigation equipment; and did not increase the likelihood of a fire or flood. (Section 02.04)

Green.

The team identified a violation involving the failure of an operating crew to adhere to a continuous action step of Emergency Operating Procedure ES-0.1,

Reactor Trip Response, resulting in an avoidable plant transient. Specifically, in response to the reactor trip and partial loss of offsite power (LOOP) event on August 3, 2003, the Unit 2 operating crew did not correctly implement continuous action step 1 of ES-0.1, which led to the cycling of the pressurizer power-operated relief valves (PORVs)ten times, complicating reactor coolant system (RCS) pressure control.

This finding is greater than minor because it affected the Initiating Events Cornerstone and could reasonably be viewed as a precursor to a more significant event, in that, the failure to implement established procedures could place the reactor outside its design envelope and, for this particular event, the repeated cycling of the PORVs could have resulted in a loss of coolant event had a PORV stuck open. This finding is of very low safety significance because all mitigation systems were available during the event and was treated as a non-cited violation, consistent with Section VI.A.1 of the NRC Enforcement Policy. (Section 05)

Green.

This team-identified finding involves inadequate corrective actions for repeat Unit 2 reactor scrams attributed to grid-related faults and associated protective relaying failures. The lack of thorough evaluations and corrective actions on the part of Entergy, in cooperation with the responsible Transmission and Distribution Operator for the local area electrical grid, have resulted in an increased frequency of plant transients and consequential challenges to Unit 2 safety related systems and licensed operators.

iv

Summary of Findings (contd)

This finding is greater than minor because it affects the Initiating Events Cornerstone and represents an increased likelihood of an event that challenges critical safety functions and operator response. Using the Indian Point Unit 2 Significance Determination Process Phase 2 Transient with Power Conversion System Available worksheet, this finding was determined to be of very low safety significance. (Section 04)

Cornerstone: Emergency Preparedness

Green.

This team-identified finding involves the failure of the Unit 2 TSC back-up diesel generator to function on August 14, 2003. The conditions which caused the diesel generator to fail to function involved electrical loading of the diesel generator in excess of its design capacity. This condition was initially identified in February 2000 and not resolved in a timely manner.

This finding is considered more than minor because a significant amount of TSC/OSC emergency response equipment, necessary to implement the Emergency Plan, was either de-energized by the Entergy staff because of the loss of sufficient air conditioning to ensure emergency response equipment would not be damaged due to overheating, or was without AC power because the diesel was non-functional. This finding is of very low safety significance because key members of the ERO were able to implement established compensatory measures to effectively perform their emergency response functions. (Section 6.0.c.1)

Green.

This team-identified finding involves the failure of the Unit 3 Technical Support Center back-up diesel generators to function on August 14, 2003. The conditions which caused the diesel generators to fail to function were previously identified by Entergy on April 18, 2003, as a result of a failed periodic load test and inadequate retest. This condition was not resolved in a timely manner.

This finding is considered more than minor because a significant amount of the Unit 3 TSC/OSC emergency response equipment was without AC power because the diesel was non-functional. On August 14, Entergy elected to de-energized all of the remaining emergency response equipment and plant information computer systems. The Unit 3 TSC/OSC functions were all transferred to the Unit 2 TSC/OSC under one site Technical Support Center Manager. This finding is of very low safety significance because key members of the Unit 3 ERO were able to implement established compensatory measures to effectively perform their emergency response functions from the Unit 2 TSC/OSC. (Section 6.0.c.2)

Green.

This team-identified finding involves the August 14, 2003, loss of off-site power event which revealed that Entergy did not have a preventive maintenance program in place to ensure the continued functionality of the numerous un-interruptible power supplies in the Emergency Operations Facility (EOF) which provide back-up power to emergency response equipment.

v

Summary of Findings (contd)

This finding is considered greater than minor because a significant amount of the Unit 2 and Unit 3 emergency response organization communications equipment was non-functional on August 14 until off-site power was restored. However, this finding is of very low safety significance because key members of the ERO were able to implement established compensatory measures to effectively perform their emergency response functions from the EOF, TSC/OSC, and Unit 2 and 3 central control rooms, using back-up telephone communications. (Section 6.0.c.3)

B. License-Identified Violation A violation of very low safety significance, which was identified by the licensee has been reviewed by the team. Corrective actions taken or planned by the licensee have been entered into the licensees corrective action program. This violation and corrective actions are listed in Section 4OA7 of this report.

C 10 CFR 50.72 requires that the licensee notify the NRC as soon as practical and in all cases, within four hours of the occurrence of any event or condition that results in actuation of the reactor protection system when the reactor is critical.

Following the Unit 3 reactor trip and entry into a natural circulation cooldown on August 3, 2003, the Shift Manager failed to make the required 10 CFR 50.72, four-hour report, within the specified time period. The 10 CFR 50.72 report was made at 11:10 a.m., approximately two and one-half hours late (reference Emergency Notification System No. 40045). This finding is of very low safety significance because the event was an uncomplicated reactor trip and no outside assistance or emergency response organization activation was warranted.

vi

Report Details 01 SPECIAL

INSPECTION SCOPE

A special inspection team was established to inspect and assess the repeat automatic reactor scrams at Unit 2 and 3, including the reactor trip of Unit 2 on August 3, 2003 and the dual unit trip on August 14, 2003. The inspection was initiated in accordance with NRC Management Directive 8.3, NRC Incident Investigation Program, based upon the deterministic criterion involving repetitive failures (in this case, repeat loss of offsite power events at both units). The associated core damage probability of the August 3 Unit 2 reactor trip was initially estimated at 1.7 E-6, which would allow either resident inspector follow-up or a special inspection (see Section 05). The NRC chose a special inspection due to the above stated deterministic criterion, and because the inspection could be efficiently coordinated with a planned 95001 inspection for Unit 3 (see Sections 02 and 03). Following the August 14, 2003, loss of offsite power dual unit trip, the NRC staff concluded that these plant transients and any associated licensee performance issues could be easily incorporated into the existing team charter (see Attachment D).

The special inspection was conducted in accordance with NRC Inspection Procedure 93812, Special Inspection, and Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, consistent with Manual Chapter 0305, Operating Reactor Assessment Program.

02 EVALUATION OF INSPECTION REQUIREMENTS - No. 3 Breaker Failures, Unit 3 Inspection Scope (95001)

The team assessed Entergys evaluation of the four Unit 3 unplanned reactor shutdowns which contributed to the Unplanned Scrams in 7000 Critical Hours Performance Indicator (PI) changing from Green to White following the June 22, 2003 reactor scram from 100 percent power. The team used Inspection Procedure 95001 to guide the inspection activities and assess the adequacy of Entergys root cause determination and associated corrective actions. The team focused their inspection efforts in two principle areas: the examination of the 345 kV No. 3 breaker failures which occurred on November 15, 2002 and June 22, 2003; and Entergys collective evaluation of the four unplanned reactor scrams which contributed to the PI change.

Background Between November 15, 2002 and June 22, 2003, Unit 3 experienced two automatic turbine and reactor trips due to the failure of 345 kV generator output circuit breaker No. 3 (see Figure 1). This SF6 gas insulated ITE circuit breaker was rated at two cycle, 3000A, and 1300 kV basic insulation level (BIL) (see Figure 2). Breaker No. 3, and the identical parallel generator output breaker (No. 1) are maintained by Entergy.

[The team notes that the Unit 3 generator output circuit breakers were included in the sale of Indian Point Unit 3 from the New York Power Authority, whereas the Unit 2 345 kV generator output circuit breakers remained under the ownership of Consolidated Edison when Unit 2 was purchased by Entergy.]

During the November 15, 2002 event, 345 kV circuit breaker No. 3 failure resulted in the trip of circuit breaker Nos. 1, 3, and 6. This resulted in the isolation of Unit 3, tripping of primary and backup lockout relays, and resultant turbine trip/reactor trip. The breaker No. 3 failure was attributed to contact misalignment during previous maintenance.

Misalignment of stationary and moving contacts caused overheating, burning, pitting and arcing, and carbonization. This resulted in the lowering of the dielectric voltage withstand capability of the gaseous dielectric and a subsequent phase to ground fault.

This failure involved major internal components and sub-components of the circuit breaker.

During the June 22, 2003 event, breaker No. 3 experienced severe internal damage resulting in a trip of both 345 kV generator output breakers (Nos. 1 and 3). Unit 3 was electrically isolated by the tripping of the main generator primary and backup lockout relays. The generator trip, turbine trip, and reactor trip followed. Similar to the November 22, 2002 event, the dielectric voltage withstand capability of the SF6 dielectric was compromised. Entergy concluded that an inadequate level of moisture and contaminants were present inside the circuit breaker and resulted in the lowering of electric field strength of the SF6 dielectric. Failure occurred during an over-voltage switching transient.

During the site visit the week of August 25, the special inspection team toured the Buchanan substation shared by Entergy and Consolidated Edison. The team inspected the old ITE SF6 insulated 345 kV circuit-breaker and a new vintage of GE-Hitachi replacement circuit-breaker. The inspection team also reviewed the internals of the failed ITE circuit breaker, disconnect switches, potential and current transformers, surge arresters, and other switchyard equipment and controls.

02.01 Problem Identification

Determination of who (i.e., licensee, self revealing, or NRC) identified the issues and under what conditions.

In both the November 2002 and June 2002 events, a dielectric breakdown occurred due to switching transients and the fault/failures were self revealed.

Issues were identified due to tripping of the circuit breakers, tripping of relays, generator trip, and turbine trip/reactor trip.

Determination of how long the issue existed, and prior opportunities for identification.

The November 2002 breaker failure was attributed to the misalignment of the stationary and moving contacts during preventive maintenance. The June 2003 event was attributed to the presence of moisture in the SF6 cooling gas and contaminants on the pull rod assembly which compromised the B phase dielectric introduced during corrective maintenance performed on the breaker following the November 15, 2002 failure. For both events, the conditions which led to the eventual breaker failures were attributed to maintenance.

Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue.

The licensees root cause analysis reports (CR-IP3-2002-04550, CA-00007, dated 12/12/2002 and CR-IP3-2003-03809, CA-00023, dated 8/6/03)documented the discussion of Radiological Safety, Environmental Safety, Industrial Safety, Potential Safety Consequences and overall Safety Significance of the events. The licensee concluded that Chapter 14 of the UFSAR bounds both events. The licensee also provided discussion and insight into the generic implications of the events.

02.02 Root Cause and Extent of Condition Evaluation

Evaluation of method(s) used to identify root cause(s) and contributing cause(s).

For No. 3 breaker failures, Entergy conducted a comprehensive evaluation and used a systematic method to identify root and contributing causes. The licensee evaluated maintenance effectiveness, man-machine interface related issues, human performance, and communication and resource management attributes to identify the root cause and contributing causes.

Level of detail of the root cause evaluation.

The root cause evaluations for the No. 3 ITE circuit breaker failures were conducted in sufficient detail and commensurate with the significance of the problem. The June 22, 2003 event root cause evaluation was more comprehensive than the November 15, 2002 event evaluation. Notwithstanding, both evaluations identified root and contributing causes that were appropriate.

Using the Kepner-Tregoe process, Entergy concluded that the November 15 breaker failure was directly caused by the phase to ground fault of the B phase due to overheating caused by high resistance at the contact surfaces. This fault was due to misalignment of the stationary and moving contacts due to poor workmanship on the part of the contracting vendor.

Detailed investigation and evaluation of the June 22 breaker No. 3 failure identified that the B phase dielectric was compromised due to moisture in the SF6 gas and contamination of a pull rod assembly. Root causes identified by Entergy for this maintenance related failure were: inadequate vendor oversight with respect to limited knowledge of the Entergy staff and insufficient supervisory resources to oversee work in the field; and inadequate work practices with respect to failing to follow established vendor instructions.

3. Consideration of prior occurrences of the problem and knowledge of prior

operating experience.

As documented above, Entergy identified the causal factors of inadequate maintenance and poor vendor oversight as contributing to both the November 2002 and June 2003 failures of breaker No. 3. The inspectors note that the two events were approximately seven months apart and that the opportunity was provided, although not exercised, for some time-trend analysis and checks to evaluate the effectiveness of the November 2002 corrective maintenance.

Entergys root cause evaluation identified that the purging and filling of the breaker with SF6 during cold weather conditions likely contributed to the moisture intrusion. The vendor recommends moisture checks within two months and six months to ensure the moisture content is below maximum allowable levels. These checks were not performed.

Consideration of potential common cause(s) and extend of condition of the problem.

Prior to restart from the November 15, 2002 forced outage, Entergy identified that the same contractor performed maintenance on both the No. 3 and No. 1 breakers. The No. 1 breaker was tested and found to have elevated resistance readings on two of three phases. Corrective maintenance was performed prior to placing this breaker in service. Following the June 22, 2003 event, Entergy again tested the No. 1 breaker and performed corrective maintenance to restore the breaker to within vendor specifications. In addition, Entergy worked with the Substation Predictive Maintenance Group to have a contractor conduct a comprehensive review of the Buchanan Substation. Breakers, feeders, disconnects, insulators, current transformers, potential transformers, and surge arresters were examined using a combination of vibration, ultrasonic, infrared, corona, and dissolved gas analysis (DGA) techniques to assess the material condition of these components. The results were analyzed and corrective maintenance performed or planned, as appropriate.

02.03 Corrective Actions

Appropriateness of corrective action(s).

Entergy has taken appropriate corrective actions for the root and contributing causes for the November 2002 and June 2003 circuit breaker failures. Besides the immediate actions to repair and then replace the No. 3 circuit breaker with a new GE-Hitachi type 345 kV breaker, plans are moving forward to replace the No. 1 breaker during the next refuel outage (3R13). The inspection team verified that the licensee had appropriately addressed the 345 kV system performance issues within the Maintenance Rule Program guidance. To address the specific concerns of inadequate oversight of the contractor, Entergy assigned a full-time switchyard coordinator to work with the Consolidated Edison staff and thus freed the component and system engineers to provide direct oversight of the work being performed in the field without having to be burdened with project management responsibilities. In addition, following a meeting between senior Entergy and Consolidated Edison managers, a working meeting was conducted on September 29, 2003, to develop the framework to improve the Buchanan switchyard reliability and protective circuitry which impacts both Indian Point units. This action was one of the first elements to be implemented in Entergys switchyard reliability improvement plan.

2. Prioritization of corrective actions.

The prioritization of corrective actions associated with the June 22, 2003, breaker failure are consistent with plant safety and conform to the guidance of 10 CFR 50.65, Maintenance Rule. The corrective actions associated with the November 15, 2002, failure were limited and contributed to the subsequent June 2003 failure. (See Section 02.04 for the teams characterization of this observation.)

Improvement in the oversight of contractors has been an historical concern at the facility. Actions by Entergy have been initiated to improve site performance in this area. (See Sections 03.02 and 03.03 for additional inspector review and assessment of this issue.)

3. Establishment of schedule for implementing and completing the corrective

actions.

The team believes Entergy has developed and implemented an appropriate work schedule for the completion of the action plan associated with the 345 kV circuit breakers. The No. 1 breaker is planned to be replaced during the next refuel outage. This activity is dependent upon the availability of a replacement GE-Hitachi breaker. The team noted that Entergy did not risk the completion of post-installation testing of the new No. 3 breaker until the unit was off-line. The necessary testing was completed satisfactorily and the new breaker placed in service following restart from the August 14 reactor trip and associated outage.

During this outage, Entergy and its contractor commenced corrective maintenance on the No. 1 breaker; the team observed work being performed on the breaker during the week of August 25. The licensees decision to pursue replacement of both 345 kV breakers with new technology equipment appears reasonable and prudent, based upon the age, limited availability of replacement parts, and few knowledgeable technicians or engineers familiar with the older breaker technology and more challenging maintenance requirements.

4. Establishment of quantitative or qualitative measures of success for determining

the effectiveness of the corrective actions to prevent recurrence.

In addition to the Maintenance Rule Program (See Section 02.05) and related performance monitoring, Entergy implemented a detailed Effectiveness Review Plan per Condition Report No. IP3-2003-03809, Corrective Action 19 (CA-00019). This multi-tasked, multi-disciplined review will monitor various aspects of the 345 kV and 138 kV systems and is targeted to have a formal report on corrective action effectiveness submitted to the Corrective Action Review Board (CARB) after twelve months.

02.04 Findings

Introduction.

A Green finding was identified involving the poor maintenance work practices (failure to follow vendor manual instructions) and insufficient contractor oversight (monitoring, quality verification, and knowledge of work activity) which contributed to the failure of the 345 kV circuit breaker No. 3 on November 15, 2002 and June 22, 2003.

Description.

On November 15, 2002 and June 22, 2003, Unit 3 tripped as a result of the failure of 345 kV generator output breaker No. 3. In both instances, Entergy root cause investigations concluded that the cause of the breaker failures was directly attributed to inadequately performed maintenance. Contributing to these failures was the lack of appropriate contractor oversight of the preventive and corrective maintenance activities performed on breaker No. 3. The November 15 failure was traced to a breaker overhaul performed during the Spring 2001 refueling outage. The misalignment of the breaker contacts did not have an immediate impact, but rather caused a degradation of the contact surfaces over time, due to high resistance overheating. The June 2003 failure was the result of improperly conducted corrective maintenance following the November failure. The B phase dielectric was compromised due to moisture in the SF6 gas and contamination of a pull rod assembly caused by the vendor not adhering to the established repair guidance. [For additional detail, reference sections 02.01 through 02.03, above.]

Analysis.

The inspection team concluded that the performance deficiencies were: poor workmanship and inadequate work practices (failing to follow established vendor instructions) on the part of the contracted vendor; and poor vendor oversight with respect to limited knowledge of the Entergy staff and insufficient supervisory resources to oversee work in the field. Traditional enforcement does not apply because the issue did not have an actual safety consequence or potential for impacting the NRCs regulatory function, and was not the result of a willful violation of NRC requirements or Entergy procedures. This finding is greater than minor because it is associated with the Reactor Safety Initiating Events Cornerstone and adversely impacts the objective of limiting the likelihood of events that upset plant stability and challenge safety functions at power. The equipment performance attribute involving availability and reliability was compromised by improperly performed maintenance on the generator output breakers (a Maintenance Rule plant system).

The inspection team determined that this finding was of very low safety significance (Green). This issue screened to Green (IMC 0609, SDP, Appendix A Phase 1 Screening Worksheet) based upon the finding: not contributing to the likelihood of LOCA initiator; not contributing to the likelihood of a reactor trip and the unavailability of mitigation equipment; and not increasing the likelihood of a fire or flood. (FIN 50-286/2003-010-001)

Enforcement.

No violation of regulatory requirements occurred.

02.05 Maintenance Effectiveness Review

a. Inspection Scope

The 345 kV circuit breaker Nos. 1 and 3 are within the scope of the Maintenance Rule (MR), 10 CFR Part 50.65, Monitoring the Effectiveness of Maintenance at Nuclear Power Plants. Entergys characterization of the 345 kV breakers is consistent with 10 CFR Part 50.65 (b)(2)(iii), which includes non-safety related structures, systems and components whose failure could cause a reactor scram or actuation of a safety-related system. Based upon these requirements and guidelines, the inspection team reviewed two quarterly health reports relevant to the 345 kV breaker performance and maintenance status. Fourth Quarter 2002 and Second Quarter 2003 reports were reviewed. The following observations were made with regard to maintenance effectiveness.

Fourth Quarter 2002 Report:

Overall System Status: The system performance for the fourth quarter was satisfactory until the middle of the quarter when the B phase of breaker No. 3 failed, resulting in a unit trip and a seven-day forced outage.

Breaker No. 1 was tested prior to unit return to service. The contact resistance was found to be out of tolerance. The breaker was removed from service for maintenance, repairs were completed, and the breaker was returned to service.

Breaker No. 3 was completely rebuilt by the vendor prior to its return to service.

Maintenance Rule Status: The 345 kV system remained in (a)(2) status with one identified functional failure in the past two-year monitoring period.

Long Range Plans: There were no long range plans at that time for the November 2002 345 kV breaker failure. Preventive maintenance was continued at the normal six-year interval.

Licensees Regulatory Compliance Review: During the WANO audit, the auditors determined that Entergy was not in compliance with SOER 99-01, recommendation No. 5, due to the failure of breaker No. 3. A corrective action was generated as a result of the root cause investigation of the incident.

Second Quarter 2003 Report:

Actions to Return System Health to Green: Since breaker No. 3 failed for the second time, the breaker was replaced and breaker No. 1 was scheduled for replacement in 3R13. The action plan was to be revised to reflect a new date to return the system to (a)(2) status.

Maintenance Rule Status: The 345 kV system remained in (a)(1) status due to the maintenance preventable functional failure on November 15, 2002 (Entergys root cause evaluation, completed in December 2002, concluded the failure was preventable) and the second functional failure of breaker No. 3 that occurred on June 22, 2003.

Equipment History: The 345 kV breaker failed for a second time within a six-month period. A decision was made to replace the breaker.

Preventive/Predictive Maintenance: After the failure of breaker No. 3 in June 2003, breaker No. 1 was tested and found to have some out of tolerance values.

The breaker was taken out of service and degassed. Repairs were made by the vendor.

Engineering Changes: 345 kV breakers are being replaced with GE-Hitachi breakers.

Short Term Plans: During the 3R12 outage contact resistance checks were made on breaker No. 1. Results were satisfactory and no further work was performed. A work order was issued to test breaker No. 1 prior to its return to service after the June 2003 unplanned outage to ensure its reliability until the breaker is replaced during 3R13. Entergy determined that preventive maintenance will continue at the current six-year frequency for these breakers.

b. Findings

No findings of significance were identified.

03 EVALUATION OF INSPECTION REQUIREMENTS - Collective Evaluation of Four Unplanned Scrams at Unit 3 Inspection Scope and Background (93812)

In addition to the two unplanned automatic scrams involving the failure of breaker No. 3 which contributed to the PI change, two manual trips occurred. On January 13, 2003, operators manually tripped the reactor after a high differential pressure developed between two sections of the plants main condenser (each of the three condenser sections are supplied by two circulating water pumps). The loss of the No. 35 circulating water pump (CWP) in one section while the companion No. 36 CWP was out of service for maintenance caused a loss of vacuum and a high differential pressure between two sections which necessitated a manual trip.

Entergy evaluation of the No. 35 CWP failure identified that insufficient detail in maintenance procedures and poor work control practices during restoration of the pump motor from preventive maintenance led to the pump failure. Improper routing of the DC cables between the pumps motor and exciter rotors allowed the cable to rub on the motors dust hood. Vibration of the cable during normal pump operation eventually caused the cable to break and motor to fail. The preventive maintenance on the No.

35 CWP motor, which included the replacement of the motor upper oil pot cooling coils, was performed by the Entergy maintenance staff. Entergy staff elected to perform the maintenance, rather than the electric motor vendor, in an effort to save cost, time, and resources. This event was previously reviewed by the NRC in inspection report 50-286/2003-002, dated May 13, 2003, and resulted in a Green Finding (FIN 50-286/03-02-03).

The second manual turbine and reactor trip occurred during power ascension on April 29, 2003, from 59% power after operators detected a lubrication oil fire in the insulation surrounding the high pressure turbine. The licensee declared a Notice of Unusual Event (NUE) in accordance with plant procedures when the duration of the fire exceeded 15 minutes. The fire was extinguished after 51 minutes and the NUE was exited.

Entergys investigation of this event identified that the No. 2 turbine high pressure bearing was disassembled for inspection during the refueling outage and not properly reassembled by the responsible contractor. Poor maintenance practices and inadequate work controls contributed to the improper reinstallation of the bearing housing. As a result, when the turbine lubricating oil system was returned to service, oil leaked into the surrounding lagging and eventually ignited due to the high temperature of the steam turbine casing. This event was previously reviewed by the NRC in inspection report 50-286/2003-006, dated August 4, 2003, and resulted in a Green Finding (FIN 50-286/03-06-04).

The team examined Entergys collective evaluation of these two events and the two 345 kV breaker failures previously discussed. Entergys evaluation was documented in Condition Report No. IP3-2003-03866, dated June 25, 2003. The team reviewed the condition reports and root cause evaluations for each event and interviewed responsible Entergy staff. The teams evaluation of Entergys collective review follows:

03.01 Problem Identification

Determination of who (i.e., licensee, self revealing, or NRC) identified the issues and under what conditions.

The four unplanned reactor scrams which contributed to the White PI for Unplanned Scrams in 7000 Critical Hours were self-revealing.

Determination of how long the issue existed, and prior opportunities for identification.

The Green to White PI threshold was crossed on June 22, 2003, with the automatic reactor trip caused by the failure of the 345 kV generator output breaker No. 3.

Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue.

Not applicable. The four events contributing to this PI change have been assessed individually, as previously discussed.

03.02 Root Cause and Extent of Condition Evaluation

Evaluation of method(s) used to identify root cause(s) and contributing cause(s).

Consistent with Entergys Site Management Manual IP-SMM-LI-102, the Condition Report (CR) initiated for this White PI (No. IP3-2003-03866) was assigned a Category B investigation which required an apparent cause determination. Attachment 10.7 of IP-SMM-LI-102 provided specific guidance for conducting the apparent cause assessment. The responsible engineering supervisor for conducting the assessment assembled an eight member team of engineers with experience in electrical systems, maintenance, work planning and project management. Each of the four reactor scrams detailed root cause evaluation (Category A CR) was scrutinized by the Entergy team and common issues were identified.

Level of detail of the root cause evaluation.

The apparent cause determination performed by the Entergy team was appropriate. Three principle apparent causes were identified, common to all three events. These causes were: weaknesses in the subject matter expert arenas; weaknesses in the methods Entergy uses to control vendor interfaces; and lack of experienced staff in the traditional power generation areas.

3. Consideration of prior occurrences of the problem and knowledge of prior

operating experience.

Consistent with Entergys guidance, the apparent cause team evaluated both IPEC and external operating experience. The apparent cause team acknowledged that the control and oversight of contractors on site has been an ongoing concern in recent years at both Units 3 and 2. Entergys apparent cause determination for the PI change was weighed more on the two breaker failures and the turbine lubricating oil fire event. These three events clearly demonstrated that the lack of direct contractor oversight and quality control measures, along with the absence of IPEC subject matter experts to independently assess contracted work activities, contributed to the unplanned transients.

Consideration of potential common cause(s) and extent of condition of the problem.

As mentioned above, the four events which led to the PI change were examined by Entergy to identify common causes and/or contributing factors. Additional examples of poor contractor oversight at Indian Point 2 and 3 were identified via Entergys review of plant operating experience.

03.03 Corrective Actions

Appropriateness of corrective action(s).

Entergy has initiated a number of corrective actions to address the weaknesses in the subject matter expert arena and control of vendor interfaces. The team reviewed these corrective actions and determined them to be appropriate. To address the limitation in the IPEC subject matter expert ranks, Entergy has made available their entire nuclear organization to provide technical and engineering support to the Indian Point plants, as well as all Entergy nuclear facilities. To address the weaknesses in contractor oversight, detailed job descriptions with clearly defined roles and responsibilities will be implemented to ensure that subject matter experts (SMEs), systems engineers, and project leads and managers understand their duties when assigned to oversee contracted services. This action is intended to prevent dual roles and responsibilities for engineering and maintenance staff overseeing vendor activities.

2. Prioritization of corrective actions.

Entergys corrective actions to address the second 345 kV breaker failure and associated contractor oversight deficiencies were timely. A switchyard coordinator with project management responsibilities was promptly assigned and the responsible systems and component engineers were directed to provide day-to-day direct oversight of the contractor activities. The longer term corrective actions to address the subject matter expert concerns and to formalize the duties and responsibilities of SMEs, project leads, and managers with respect to vendor oversight have been appropriately prioritized in the IPEC corrective action system.

Establishment of schedule for implementing and completing the corrective actions.

The inspection team reviewed the schedule for completion of the above stated corrective actions and determined (consistent with the risk significance of the subject equipment) that the schedule for implementation of the proposed administrative changes was appropriate.

Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence.

Consistent with IPEC guidance (IP-SMM-LI-102 and ENN-LI-104), an effectiveness review is planned for six months following implementation of the corrective actions.

03.04 Findings No findings of significance were identified.

REVIEW OF UNIT 2 GRID-INDUCED REACTOR SCRAMS

a. Inspection Scope

(93812)

The inspection team evaluated four specific plant trips attributed to grid disturbances which occurred between 1997 and August 3, 2003. Each of these trips resulted in the plant being placed in a natural circulation condition. The inspectors focused on the interactions between the plant and electric grid, and how this impacted overall plant transient response. The team reviewed licensee event reports, post-transient analyses, condition reports involving the 138 kV and 345 kV systems for the last two years, modifications to the generator protection and fast bus transfer scheme, transmission protective relaying schemes, and a number of other electrical distribution system related events dating back to mid-1997 (see Attachment D). The team also reviewed minutes of the inter-company Buchanan switchyard oversight committee, relay calibration reports, and agreements between Entergy and the Transmission and Distribution (T&D)operator (Consolidated Edison). The team also reviewed Entergys actions with respect to evaluating the potential impact of the recent grid-related events on their IPE initiating events frequency assumptions.

Background Electric power produced by the plant is transmitted to the electrical grid through a 345 kV ring bus. A one-line diagram of this bus is provided in Figure 3. The ring bus has three outgoing feeders. Two of these, Y94 and W93, are each capable of carrying full plant output. Feeder TA5 transforms the voltage down to 138 kV and can only carry approximately 20% of the plant output. Incoming power from the plant flows through two output breakers, 7 and 9, both of which are rated for full plant output. The system is designed so that a fault on either main output feeder will be isolated from the ring bus by opening two of the three ring bus breakers. This would maintain the plant online supplying the grid through the remaining output breaker and feeder. For example, if a fault occurred on W93, breakers 9, 11 and the downstream switchyard breakers would open to isolate the feeder. The plant would still be supplying power to the grid through breaker 7 and feeder Y94. This fault would also be detected by the protective relays on Y94, however, blocking relays, by design, see that the fault is not on that feeder and send a signal to prevent it from being stripped by its associated downstream switchyard breakers. If faults occurred on both W93 and Y94, both plant output breakers (7 and 9)would be tripped to isolate the faults. If the unit were to continue to operate in this condition the turbine speed would rapidly increase causing a concurrent increase in generator frequency. Primary overspeed protection for the turbine is provided by both mechanical and electronic overspeed protection. To prevent overspeed and over-frequency, the opening of both plant output breakers 7 and 9 causes a direct trip of the main generator through generator protection relays.

Internal plant electrical loads are supplied through six 6.9 kV buses. During at power plant operations, two of these buses are supplied by off-site 138 kV power and the remaining four are supplied by the unit generator via the unit auxiliary transformer. The four busses supplied by the unit generator provide power to the reactor coolant pumps.

Upon a unit trip, a fast bus transfer removes power from the unit auxiliary transformer and connects all the buses to the 138 kV off-site power source, thus maintaining continuity of power. This fast transfer is blocked by relay protection if the two sources of power are not synchronized or if a generator trip is caused by an over-frequency condition. If this fast transfer block occurs, power is lost to the reactor coolant pumps placing the plant in a natural circulation condition and all three emergency diesel generators auto start (the emergency diesel generators will not load, by design, unless AC power is lost to buses 5 and 6 and/or a safety injection signal is received). Safety related 480V buses 5A and 6A remain energized via the station auxiliary transformer, but operator action is required to restore power to the 2A and 3A safety related 480V buses via EDG No. 22. The analysis for a loss of reactor coolant flow and loss of off-site power are discussed in the Updated Final Safety Analysis Report (UFSAR) in Chapters 14.1.6 and 14.1.8, respectively, and the design basis for the plant electrical systems is discussed in UFSAR Chapter 8.

Overview of Plant Trips

• July 26, 1997 Trip - Prior to this reactor trip, output feeder W93 had been taken out of service for maintenance. Breaker 9 was maintained in the closed position to maintain the ring bus. A directional current relay on TA5 failed due to water intrusion into a fuse box. This relay failure caused breakers 7 and 11 to open to isolate the feeder. The tripping of these breakers also isolated feeder Y94 which was the only available feeder to handle full load at the time. This resulted in a 100 percent loss of load for the unit. Breaker 9 never received a trip signal since there was no fault with its associated feeder, therefore no direct trip was sent to the main generator. This resulted in an over-frequency condition and the blocking of the fast transfer device since the power sources were not maintained in phase. Power was lost to the four 6.9kv busses supplied by the unit auxiliary transformer, which resulted in the plant being in a natural circulation condition, per design. This event also caused reactor coolant system (RCS) flow limits to be exceeded due to the generator over-frequency (reference inspection reports 50-247/1997-009, dated 9/10/97 and 50-247/1997-010, dated 9/29/97). As a corrective action, a permanent modification had been installed in early 1998 to trip the generator on over frequency to prevent this excessive flow condition.

This modification also inhibits the fast transfer on over frequency.

• December 26, 2001 Trip - A phase A fault on W93 caused Breaker 9 to open per design. Due to the failure of a blocking relay, the downstream switchyard breakers in Ramapo for Y94 opened, isolating that feeder which caused a loss of load. Breaker 7 did not receive a trip signal since there was no fault on its associated feeder; therefore, no direct trip occurred. The main generator began to speed-up and was tripped by the over-frequency relay which also blocked the fast transfer. The plant relied on natural circulation, and the EDGs automatically started, per design.

• April 28, 2003 Trip - A phase A fault on Y94 caused breakers 7, 11 and F7 to open to isolate the fault. A blocking relay downstream of F7 failed due to downstream grid interactions and a relay failure at the Millwood ring bus. This resulted in the loss of the two 138kV feeders required by Technical Specifications (TS). One additional 138 kV feeder (the Peekskill Refuse Burning Generating Station, not credited by TS) remained online to supply plant loads.

Breaker 11 was closed manually during ring bus restoration. A failed blocking relay caused the downstream breakers for W93 in Sprain Brook to open, isolating that feeder from the grid. Breaker 9 remained closed since no fault was seen on its associated feeder, therefore, no direct trip occurred. The main generator began to speed up and was tripped by the over frequency relay which also blocked the fast transfer. The plant relied on natural circulation, and the EDGs automatically started, per design.

• August 3, 2003 Trip - A phase A fault due to a lightning strike on W93 caused Breaker 9 to open, per design. Due to the failure of another blocking relay, the downstream switchyard breakers in Ramapo for Y94 opened, isolating that feeder and causing a loss of load. Breaker 7 did not receive a trip signal since there was no fault on its associated feeder, therefore no direct trip occurred.

The main generator began to speed up and was tripped by the over frequency relay which also blocked the fast transfer. The plant relied on natural circulation, and the EDGs automatically started, per design.

Observations: The NRC does not directly regulate the electric grid. However, under the revised Reactor Oversight Program, the performance of offsite electrical distribution system is a significant factor in the evaluation of initiating events and overall risk. With respect to the Indian Point facilities, the 345kV system is not as risk significant as the 138 kV system, but it does have a credible impact on the initiating event frequency of plant transients. The teams review of the above stated events identified that in the past three years each 345 kV electrical distribution fault that resulted in the loss of a feeder from the Buchanan switchyard was compounded by one or more transmission and distribution protective relay failures. As part of their collective review of the Unit 3 switchyard breaker failures and the above described Unit 2 grid-related plant trips, the team notes that Entergy has been working closely with Con Edison to: evaluate possible enhancements in the maintenance of 345kV protective relays; assess the adequacy of the existing protective relaying scheme; and assess possible design improvements to the outgoing 345kV transmission capacity.

The team also notes that following the August 14, 2003 widespread loss of offsite power event, a joint United States and Canadian Task Force was initiated to evaluate and make recommendations to prevent a recurrence. Prior to this event, transmission grid reliability was becoming a rising concern throughout the industry due to the use of the transmission and distribution (T&D) system in ways not originally considered when the local area T&D systems and existing grid inter-ties were designed. Consequently, the design insights and specific lessons learned from Indian Point and other nuclear and non-nuclear generating stations gained from the August 14, 2003 event will be assessed and communicated to the industry via a separate correspondence. An NRC headquarters sponsored pubic meeting to discuss this initiative was conducted on October 31, 2003.

The team learned that the Unit 2 IPE risk model is currently being converted from the large event/small fault tree to small event/large fault tree methodology, to be consistent with the Unit 3 IPE. Coincident with this conversion, Entergy plans to update their model with the latest plant specific and industry operating history event frequency data. The team observed that for the Unit 2 IPE, the loss of offsite power (LOOP) events (a station blackout and coincident general transient) contribution to the overall core damage frequency (CDF) is 18.98 percent (5.926E-6 of 3.13E-5 total CDF). For the Unit 2 grid-related reactor trip events reviewed by the team, the station emergency diesel generators operated, as designed, and precluded a station blackout event. The NRC resident inspectors will follow-up the licensees efforts to update the initiating events frequency assumptions, considering the data from the recent grid-related events.

b. Findings

Introduction.

A Green finding was identified by the team involving inadequate corrective actions for repeat Unit 2 reactor scrams attributed to grid-related faults and associated protective relaying failures. The lack of thorough evaluations and corrective actions on the part of Entergy, in cooperation with the responsible T&D operator for the local electrical grid, Buchanan Switchyard, and the circuit breakers and relaying at remote switchyards that directly affect Unit 2, have resulted in an increased frequency of risk significant plant transients and consequential challenges to Unit 2 safety related systems and licensed operators.

Description.

The team reviewed the four Unit 2 automatic reactor trips, due to a loss of generator load, that occurred during the period from July 1997 to August 3, 2003.

These loss of generator load events were the result of grid-related faults and the failure of one or more associated protective relays. The primary protection for a loss of generator load is the direct trip from the T&D operators protective scheme. The direct trip provides generator protection and involves the removal of generator excitation and electrical output. The normal at power Unit 2 electrical line-up has the generator output providing 6.9 kV power to house loads (and 480V safeguards buses 2A and 3A)via buses 1 through 4 and the unit auxiliary transformer. On a normal loss of generator load transient, the direct trip is followed by a fast bus transfer (FBT) from the unit auxiliary transformer to the station auxiliary transformer, ensuring continuity of power to the house loads. The particular sequence of grid-related faults and protective relay failures that have occurred in the past 18 months have prevented actuation of the direct trip and FBT. Absent the FBT, the unit loses 6.9 kV power to the reactor coolant pumps and operators ensure the reactor coolant system is maintained in natural circulation cooling until pumping power is restored by operator action.

Two of the 6.9 kV buses (Nos. 5 and 6), that feed two of the three safety-related 480 V buses (Nos. 5A and 6A), remained powered from the 138 kV system making these events of very low safety significance. However, all three emergency diesel generators started, by design, and operator action was required to restore electrical power to the 2A and 3A 480 V safety-related buses. The necessity for operator action to appropriately cope with these partial loss of offsite power transients introduces a greater probability of operator error. As discussed in Section 05 below, the August 3 plant transient was exacerbated by operator error. In that instance, the automatic operation of the auxiliary feedwater system and the subsequent cycling of power-operated relief valves (to maintain RCS pressure within plant design limits under natural circulation conditions)demonstrated the additional challenges to safe plant operations associated with events of this type.

The NRC team believes Entergy has been slow to formally address the grid-related problems impacting Unit 2, given the repeat nature of the protective relaying system failures and the pronounced increase in frequency of occurrences at Indian Point 2 (three grid-related trips in the 18 months between December 2001 and August 2003, not including the August 14, 2003 total loss of off-site power event).

Analysis.

This team-identified finding is greater than minor as it affects the Initiating Events (IE) Cornerstone objective of limiting the likelihood of an event that upsets plant stability and challenges critical safety systems and plant operators. Traditional enforcement does not apply because this finding did not have any actual safety consequence, did not impact the NRCs regulatory function, and was not a willful violation of NRC requirements or Entergy procedures. This finding was evaluated using Phase 2 of the Significance Determination Process (SDP) because the finding did contribute to the likelihood of a reactor trip and the likelihood of mitigation equipment or functions not being available because of the loss of the normal electrical power to half of the engineered safety systems (emergency power was not impacted). This finding was determined to be of very low safety significance (Green) using the Transient with Power Conversion System Available Worksheet; the dominant sequence being the loss of secondary heat removal (AFW), feedwater/condensate system (PCS), and high pressure recirculation (HPR). Assumptions used by the inspectors in this analysis included: zero credit for the initiating event likelihood; full credit for all mitigation systems, except AFW and HPR (because one of the two motor-driven auxiliary feedwater trains and one residual heat removal pump did not have normal electrical power (Bus 3A) due to the block of the fast bus transfer. However, full operator recovery credit was given because procedures are in place to restore the electrical buses and operators have been appropriately trained on these procedures. (FIN 50-247/2003-013-01)

Enforcement.

No violation of regulatory requirements occurred.

REVIEW OF UNIT 2 AUGUST 3, 2003 OPERATOR PERFORMANCE

a. Inspection Scope

and Background (93812)

At 4:30 a.m. on August 3, 2003, Indian Point 2 experienced a partial loss of offsite power (LOOP) which led to a reactor trip. All safety systems functioned as designed in response to the trip. However, control room operator errors led to plant conditions which caused the pressurizer power-operated relief valves (PORVs) to cycle ten times over a twelve minute period. The pressurizer PORVs vent steam from the pressurizer steam space to the pressurizer relief tank in containment to reduce reactor coolant system pressure. The root cause of this human performance error was the failure of the operating crew to take Emergency Operating Procedure (EOP) required actions for controlling auxiliary feedwater flow in the post-trip condition. Specifically, continuous action step 1 in ES-0.1 Reactor Trip Response, requires that if coolant temperature is lowering, then auxiliary feedwater flow should be throttled down to a minimum value such that the reactor coolant system cooldown is mitigated.

The operating crew failed to follow ES-0.1, step 1 which allowed the plant cooldown to continue resulting in an isolation of chemical and volume control system (CVCS)letdown flow. The isolation of CVCS letdown flow caused a mismatch between CVCS charging and letdown such that pressurizer level began to increase. As pressurizer level increased, pressurizer pressure increased to the point at which the PORVs began to lift. The pressurizer PORVs cycled ten times before the control room operators re-established letdown flow and restored reactor coolant system pressure to normal post-shutdown values. Performance deficiencies identified for the August 3, 2003, event are summarized in the following paragraphs.

b. Findings

===.1

Introduction.

A Green finding was identified involving the failure of an operating crew to===

adhere to a continuous action step of EOP ES-0.1 resulting in an avoidable plant transient. Specifically, in response to a reactor trip and partial LOOP event on August 3, 2003, the Unit 2 operating crew failed to correctly implement continuous action step 1 of ES-0.1 Reactor Trip Response, which led to a sequence of events involving the cycling of the pressurizer PORVs ten times before operators re-established proper reactor coolant system (RCS) pressure control.

Description.

The crew responded to the reactor trip by entering EOP E-0, Reactor Trip or Safety Injection. Step 4 of E-0 requires a transition to EOP ES-0.1, Reactor Trip Response. The crew entered ES-0.1 at 4:40 a.m., approximately 10 minutes after the reactor trip. Step 1 of EOP ES-0.1 checks the trend of RCS temperature. If RCS temperature is stable or is trending down to the normal no-load setting (547 degrees F),then the step requirements are met and the response not obtained actions are not taken. Response not obtained actions are taken to mitigate problems identified as plant parameters are monitored during EOPs. When the crew reached this step (at approximately 4:40 a.m.) RCS temperature was trending up to 547 degrees F. The crew appropriately continued to step 2 and beyond without taking actions to affect RCS temperature. Step 1, however, is required to be applied in a continuous fashion while in this procedure. This means that if RCS temperature begins to trend down or is unstable during the conduct of ES-0.1, then the response not obtained actions of Step 1 must be completed. Specifically, auxiliary feedwater flow rate should be lowered in accordance with the procedure to mitigate the cooldown of the RCS. At 4:45 a.m., RCS temperature began to trend lower, and was not mitigated until auxiliary feedwater flow was throttled at approximately 5:07 a.m.. During this 22-minute period, the plant cooldown continued, resulting in the repetitive PORV lifts described above.

The crew failed to apply the required actions of the Step 1 continuous action step. The Control Room Supervisor did not realize that Step 1 was applicable throughout the procedure. He did realize that the RCS cooldown was undesirable, and he knew that a future step in the procedure would allow the throttling down of auxiliary feedwater.

Through interviews with the crew, it is apparent that different crew members had supporting pieces of information that could have been put together such that the EOP would have been correctly followed. However, there was weak supervision and teamwork that left the Control Room Supervisor to direct the EOP actions without significant input from other crew members. Further analysis can be found in paragraph 05.b.2 below.

Analysis.

The Unit 2 operating crew failed to adhere to continuous action step 1 of EOP ES-0.1, Reactor Trip Response. The consequence of this procedural error was an RCS pressure transient which involved the automatic cycling of the pressurizer PORVs ten times before operators restored pressure to the normal band. This human performance error (procedural non-compliance) is greater than minor in that this type of error could be reasonably viewed as a precursor of a more significant event.

This performance deficiency affected the reactor safety Initiating Event Cornerstone and was evaluated using Phase 2 of the SDP because the multiple opening and re-closing of the PORVs increased the probability of a stuck open PORV initiating event which would be a small loss of coolant accident condition. The initiating event frequency for a stuck open relief valve was assumed to increase by one order of magnitude, resulting in the finding having a very low safety significance (Green). The dominate stuck open relief valve core damage sequence included the failure of the block valve to close and the failure of low pressure recirculation. There was no impact on the capability to either close the block valve or to use the residual heat removal or recirculation systems to conduct low pressure recirculation. Therefore, this finding is Green.

Enforcement.

The failure to adhere to action step 1 of ES-0.1 on August 3, 2003, is contrary to the emergency operating procedure and Technical Specification 6.8.1.

Because this failure to implement emergency operating procedure ES-0.1 is of very low safety significance and has been entered into the licensees corrective actions program (CR-IP2-2003-04933), this violation is being treated as a non-cited violation, consistent with Section VI.A of the NRC Enforcement Policy. (NCV 50-247/2003-013-02).

.2 August 3, 2003 Event - Contributing Factors

Based upon the inspection teams review, the following observations were made which the team believes contributed to the continuous action step not being followed:

• The Control Room Supervisor (CRS) implementing the EOPs apparently overlooked the on-going applicability of the continuous action step, and was waiting for future procedural guidance to throttle auxiliary feedwater flow (individual based error).

• The rest of the crew did not adequately challenge the CRS about the details of his EOP implementation strategy (crew based error).

• The Shift Manager and Watch Engineer/ Field Support Supervisor became involved with other activities and did not provide sufficient oversight of EOP implementation. The Shift Manager was involved in completing required notifications of the event. The Watch Engineer/ Field Support Supervisor in the role of Shift Technical Advisor became directly involved in peer-checking and overseeing electrical bus restoration, as well as, connecting a recording device for monitoring an impending reactor coolant pump start.

• The operating crew was not fully sensitive to the impact auxiliary feedwater has on plant cooldown (simulator post-trip decay heat load and auxiliary feedwater injection to the steam generators does not closely model actual plant response).

Simulator training focuses on actions for Loss of Offsite Power events, but because the simulator does not closely replicate the auxiliary feedwater cooling effect, it appears that the applicable EOP steps are not fully exercised in training.

This is an unresolved item and discussed further in Section b.3 below.

• Scheduled lessons-learned training from the April 28, 2003 reactor trip, which involved enhanced operator response to natural circulation cooldown scenarios, had not been received by the on-shift crew (this training was scheduled coincident with the requalification training cycle).

• Two modifications related to feedwater system control (the removal of the post-trip main feedwater bypass valve closing delay, and raising the feedwater isolation signal set-point) had a minor impact on the plant cooldown, but were not specifically addressed in simulator training. The team viewed this observation as a missed opportunity to identify the previously mentioned simulator modeling problem.

.3 Unresolved Item 50-247/2003-013-03: Acceptability of the Unit 2 simulator modeling of

decay heat load and auxiliary feedwater cooldown.

The simulator for Indian Point Unit 2 does not accurately model cooling effects of auxiliary feedwater injection during loss of offsite power post-reactor trip situations. A simulator demonstration conducted for the team identified that the simulator does not exhibit a reactor coolant system cooldown with maximum auxiliary feedwater injection during post-reactor trip plant conditions with a loss of offsite power (LOOP) at any modeled time in core life. The actual plant experiences a significant cooldown until auxiliary feedwater is throttled down in accordance with Emergency Operating Procedures (EOPs). The team compared actual plant performance graphs with the controlled simulator runs and significant differences were observed in how wide range cold leg temperatures responded over a 40-minute period following the reactor trip with a LOOP. It appears that the decay heat load is too high to allow any cooldown from auxiliary feedwater injection. This simulator modeling inconsistency with known plant conditions provides a potential for negative training of the operators. One potential example of this was the inappropriate actions taken by the on-duty crew responding to the August 3, 2003 reactor trip and natural circulation cooldown event.

Additional specialist inspector follow-up is necessary to determine if this simulator modeling discrepancy is the result of a human performance problem or simulator/training program deficiency as it relates to 10 CFR 55.46. The follow-up inspection will also examine the simulator testing program to determine if it was possible to identify this modeling issue during testing of the simulator. The need to further develop this potential compliance issue with specialist assistance and analyze the risk and safety significance of this modeling problem (simulator decay heat load versus auxiliary feedwater cooldown) remains unresolved.

REVIEW OF AUGUST 14, 2003 LOSS OF OFFSITE POWER EVENT RESPONSE -

UNITS 2 AND 3

a. Inspection Scope

In accordance with the Special Inspection Teams revised charter, the inspectors reviewed Entergys performance in response to the August 14, 2003 loss of offsite power event. The inspection team was on site at the time of the event. The team supported the resident inspectors and implemented a 24-hour inspector coverage plan to monitor licensee actions, which included: initial control room operator response and plant stabilization, including the use of emergency operating procedures; operator actions in the plant to minimize the impact of the loss of offsite power to non-essential systems and verification of appropriate safety system response (i.e., auxiliary feedwater system and emergency diesel generator operation); emergency response organization activation coincident with the declaration of an Unusual Event; offsite electrical power restoration and emergency diesel generator shutdown; and Unit 2 restart activities which occurred on August 16. The inspection team used inspection procedures 93802 and 93812 to review Entergy staff response to this dual-unit trip event.

Following restart of Unit 2, the 24-hour inspector shift coverage was secured. Unit 3 restart was delayed because of control rod drive mechanism and individual rod position indication issues. Following corrective maintenance, Unit 3 was restarted on August 20.

b. Background After both the Unit 2 and Unit 3 Shift Managers entered into an Unusual Event (loss of off-site power greater than 15 minutes), the Unit 2 Shift Manager activated the emergency response organization (ERO) as a prudent measure (required at Alert or higher levels) to obtain additional plant staff support to address the numerous balance of plant equipment problems caused by the lack of offsite power. The Unit 2 Technical Support Center and Operations Support Center (TSC/OSC) was staffed early in the event, but not formally activated. Offsite power was restored to the station after about 1.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />. Back-up electrical power to the TSC/OSC, other than the TSC un-interruptible power supplies, was not available because the Unit 2 TSC diesel failed to function. The Unit 2 TSC was also staffed with the Unit 3 TSC/OSC personnel because the Unit 3 TSC diesel also failed to start and run. Station management made a conscious decision to shutdown all Unit 3 TSC/OSC equipment. Based upon the nature of the event, coordination of both units repair teams and support to the control rooms was more easily conducted out of a central location (Unit 2 TSC).

As stated above, both TSC diesel generators failed to operate during the August 14 loss of offsite power event. In addition, a number of un-interruptible power supplies (UPSs)in the EOF (located in the Training Center) did not function. These UPSs are important because the EOF does not have a back-up AC power source, by design. The UPSs provide short-term DC battery back-up power to dedicated ERO communications and data transmission systems. The failure of these back-up diesel generators and UPSs degraded the full design capability of the emergency response facilities (ERF), but did not compromise the functional capability of the ERO and the associated 10 CFR 50.47(b)(8) planning standard. The following ERF equipment was without power for the approximate 1.5 hour5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> duration of the offsite power outage:

TSC/OSC Impact C U-2, all four Unit 1 Control Room Air Conditioning (CRAC) Units - these CRAC units provide the primary cooling of the Unit 1 control room and supplement the cooling of the Unit 2 control room.

C U-2 and U-3, TSC, OSC, and Computer Room Air Conditioning - no other HVAC sources available for these spaces. The team notes that on August 14, 2003, the plant operations support staff elected to minimize room heating loads and minimize the TSC UPS electrical loads by securing all plant computing systems, except Plant Information Computer (PIC) at Unit 2. At Unit 3, the licensee elected to secure all computer systems, including PIC. Computer systems removed from service were: Unit 3 PIC; Unit 3 Critical Function Monitoring System (CFMS); Local Area Network for both units; Unit 2 Safety Assessment System/Emergency Data Display System; and Unit 2 Digital Radiation Monitoring System (DRMS) and Unit 2 Safety Assessment System (SAS).

C All TSC lighting and power outlets (except the outlets powered by the UPSs) -

one wall of the U-2 TSC remained powered by the UPSs which provided one set of PIC displays (three terminals and monitors). The team notes that no facsimile or copy machines were functional and no computer terminals (other than one set of U-2 PICs terminals) were available to communicate data or electronic messaging.

C Back-up diesel auxiliaries and support (instrumentation) power EOF Impact C Unit 2 and Unit 3 Local Government Radios were not functional because the Emergency Offsite Facility (EOF) transmitter UPS did not operate. Commercial telephone back-up was used.

C Unit 2 and Unit 3 Radiological Emergency Communications System (RECS)were not functional due to the failure of the associated EOF UPS. Commercial telephone back-up was used.

C Five-Way and Three-Way Direct telephone lines did not function due to UPS failures. These direct telephone systems are automatic ringing between the TSC/OSC, Control Rooms, EOF/Alternate EOF, JNC, and White Plains office.

C The meteorological tower back-up diesel power supply failed (started and tripped), but its associated UPS did function and meteorological data was available to the EOF.

ERF/ERO Communications Although not directly related to the failure of either the TSC diesels or UPSs, additional communications difficulties were encountered and included:

C Unit 2 and Unit 3 Operations Department Radios were not functional. The power supplies to the base stations are not from vital power sources.

C Personal computer-based MEANS (Modular Emergency Assessment Notification System) was without electrical power in the control room (lap-top computer).

This lap-top driven PC-based software is used to access and enter data for three emergency management forms: NYS Radiological Emergency Data Forms (INForms); Dose Assessment and Protective Action Recommendations (DAPARS); and Emergency Action Level Computerized Information System (EALCIS). Hard-copy forms are used for back-up, but on August 14, 2003, no copy machines were functional in the control room.

C Unit 2 ERDS - PICs data was available to the EOF, but the EOF computer data acquisition and modem was without power because its associated UPS failed.

C Unit 3 ERDS - no data available because the Unit 3 PIC was secured; however, the EOF computer and modem was available (the UPS was functional).

C Several commercial telephone lines were not functional (private branch - PBX),in addition to the communications mentioned above, because the PBX UPS was not functional. However, RECs and Local Government Radios were successfully backed-up by a commercial telephone.

C Dialogic Notification System (DNS) - This system, in conjunction with Entergys pager system (SKYTEL), is designed to notify the ERO staff for ERF activation.

This system did not function properly during the August 14 event. Following the initiation of a coded message, the pager system notifies the ERO staff. In response to the coded message, the ERO staff calls the DNS for a pre-recorded message. Because of the wide-spread power outage, sections of the SKYTEL system were without power or back-up power supplies. Consequently, some members of the ERO were never notified. Fortunately, the event occurred while many plant staff members were still on site and they responded to the Public Address system notifications. In addition, ERO staff with cordless telephones were unable to be contacted due to the electrical power outages.

.1 Unit 2 TSC Diesel Generator Failure - August 14, 2003

Introduction.

A Green finding was identified involving the failure of the Unit 2 TSC back-up diesel generator to function on August 14, 2003. The conditions which caused the diesel generator to fail to function involved electrical loading of the diesel generator in excess of its design capacity. This condition was initially identified in February 2000 and not resolved in a timely manner.

Description.

On August 14, 2003, the Unit 2 TSC diesel did not automatically start and subsequent operator actions to manually start and load the diesel failed. The conditions potentially impacting the operability of the TSC diesel were first identified on February 3, 2000. Condition Report (CR) No. 2000-00705 documented an observation during emergency planning training that there was a potential for the TSC diesel generator to be overloaded. The corrective actions for CR No. 2000-00705 were transferred into CR No. 2000-08332. CR No. 2000-08332, dated October 30, 2000, documented that based upon a review of the electrical power distribution drawings, a potential existed for the TSC back-up diesel generator to be overloaded under some conditions. The licensee reviewed Abnormal Operating Instruction (AOI) 27.1.10, Loss of Power to the TSC and found that the load on the diesel for both TSC electrical bus sections would be about 1,150 amps and 875 KVA. The long-term corrective actions, postponed until September 30, 2003, involved the generation of an electrical calculation to definitively determine if the diesel generator was overloaded; revise drawings, as appropriate; and determine a method to control future TSC diesel electrical loading. This proposed TSC load study was incorporated into the Design Basis Improvement project. More recently, on May 25, 2003, during the annual load test of the TSC diesel, the diesel tripped on reverse power (reference CR Nos. 2003-03200, dated 5/25/03, and 2003-03367, dated 5/31/03).

The Unit 2 TSC/OSC remained without a back-up AC electrical power supply until September 15, 2003, when a temporary alteration was installed and satisfactorily tested, which provided a 1500 kW skid-mounted diesel generating unit (reference Inspection Report 50-247/2003-011, Section 1R23).

Analysis.

The failure of the Unit 2 TSC diesel generator to function on August 14, 2003, is the result of a performance deficiency (failure to take timely and effective corrective action) that adversely impacts Non-Risk Significant Planning Standard 10 CFR 50.47(b)(8), which states that adequate facilities and equipment are maintained to support emergency response. Traditional enforcement does not apply because the issue did not have an actual safety consequence or potential for impacting the NRCs regulatory function, and was not the result of a willful violation of NRC requirements or Entergy procedures. This finding is considered more than minor because the objective of the Emergency Preparedness Cornerstone, to ensure adequate facilities and equipment are capable of protecting the health and safety of the public in the event of a radiological emergency, was impacted. A significant amount of TSC/OSC emergency response equipment, necessary to implement the Emergency Plan, was either de-energized by the Entergy staff because of the loss of sufficient air conditioning to ensure emergency response equipment would not be damaged due to overheating or was without AC power because the diesel was non-functional. This finding is of very low safety significance because key members of the ERO were able to implement established compensatory measures to effectively perform their emergency response functions. This determination was made using Manual Chapter (MC) 0609, Appendix B, Emergency Preparedness Significance Determination Process, Sheet 2. Specifically, the August 14 loss of offsite power event resulted in a Notice of Unusual Event declaration, consistent with established Emergency Action Level guidelines. The diesel failure constitutes a failure to implement a program element vice a failure to comply with or satisfy a Planning Standard function. Therefore, this finding is Green. (FIN 50-247/2003-013-04)

Enforcement.

No violation of regulatory requirements occurred.

.2 Unit 3 TSC Diesel Generator Failure - August 14, 2003

Introduction.

A Green finding was identified involving the failure of the Unit 3 Technical Support Center back-up diesel generators to function on August 14, 2003. The conditions which caused the diesel generators to fail to function were previously identified by Entergy on April 18, 2003, as a result of a failed periodic load test and inadequate retest. This condition was not resolved in a timely manner.

Description.

On April 18, 2003, the TSC diesel was tested under blackout conditions during the Unit 3 refueling outage. The diesel started and then tripped while being loaded. As a result of the diesel trip, the Unit 3 system engineer initiated a work order that included replacement of the overspeed controller. Seven hours after this first test, per the Nuclear Plant Operator (NPO) log entries, the TSC diesel was re-tested and run unloaded. Per the Unit 2 system engineer (backup to the Unit 3 system engineer), this run did not include loading the diesel, however, this diesel run was determined to be SAT per the NPO log entry.

Inspector follow-up identified that on May 5, the Unit 3 system engineer wrote a new Priority 3 work order (WO No. 02609) to replace the TSC diesel generator overspeed trip module. This new WO was written to focus on a single maintenance action, whereas the original WO included additional work. However, the maintenance planning organization revised WO No. 02609 to a priority 4 and scheduled the repairs for November 2003, based upon the diesel problem being determined not an operability concern.

On August 14, the TSC diesel generator started and then tripped while being loaded, and was unavailable for the duration of the blackout event. On August 15, the priority of WO No. 02609 was revised to Priority 2. Subsequent troubleshooting confirmed the overspeed module problem, first identified on April 18, 2003. Following replacement of the overspeed controller and a satisfactory load test, completed on September 16, 2003, the Unit 3 TSC diesel was restored to a functional condition. Accordingly, the team concluded that the TSC diesel generator could not have provided its intended design function from April 18 to September 16, 2003.

Analysis.

The failure of the Unit 3 TSC diesel generator to function on August 14, 2003, is the result of a performance deficiency (failure to take timely and effective corrective action) that impacts Non-Risk Significant Planning Standard 10 CFR 50.47(b)(8), which states that adequate facilities and equipment are maintained to support emergency response. Traditional enforcement does not apply because the issue did not have an actual safety consequence or potential for impacting the NRCs regulatory function, and was not the result of a willful violation of NRC requirements or Entergy procedures. This finding is considered more than minor because the objective of the Emergency Preparedness Cornerstone, to ensure adequate facilities and equipment are capable of protecting the health and safety of the public in the event of a radiological emergency, was adversely impacted. A significant amount of the Unit 3 TSC/OSC emergency response equipment was without AC power because the diesel was non-functional. On August 14, Entergy elected to de-energized all of the remaining emergency response equipment and plant information computer systems. The Unit 3 TSC/OSC functions were all transferred to the Unit 2 TSC/OSC under one site Technical Support Center Manager. This finding is of very low safety significance because key members of the Unit 3 ERO were able to implement established compensatory measures to effectively perform their emergency response functions from the Unit 2 TSC/OSC. This determination was made using Manual Chapter (MC) 0609, Appendix B, Emergency Preparedness Significance Determination Process, Sheet 2. Specifically, the August 14 loss of offsite power event resulted in a Notice of Unusual Event declaration, consistent with established Emergency Action Level guidelines. The diesel failure constitutes a failure to implement a program element vice a failure to comply with or satisfy a Planning Standard function. Therefore, this finding is Green. (FIN 50-286/2003-010-02)

Enforcement.

No violation of regulatory requirements occurred.

.3 Failure of Emergency Operations Facility Un-Interruptible Power Supplies

Introduction.

A Green finding was identified involving the August 14, 2003, loss of off-site power event which revealed that Entergy did not have a preventive maintenance program in place to ensure the continued functionality of the numerous un-interruptible power supplies in the Emergency Operations Facility (EOF) which provide back-up power to emergency response equipment.

Description.

By design, there is no electrical back-up power supply to the EOF.

Instead, the EOF has a number of UPSs which provide short-term DC battery back-up power to dedicated ERF communications and data transmission systems. The team determined that prior to the sale of the Indian Point Units 2 and 3 to Entergy, the EOF UPSs were maintained via a Consolidated Edison service contract with a private vendor.

Following the sale, this service contract was not picked-up by Entergy and periodic testing and replacement of the UPS battery back-ups was not conducted. During the August 14 event, the UPSs failed outright or functioned for only a fraction of their design capacity. The meteorological tower back-up diesel power supply failed (started and tripped), but its associated UPS did function and meteorological data was available to the EOF.

The failure of the EOF UPSs primarily compromised the designed communications capability of the Emergency Response Organization (ERO). The following communications equipment was without power for approximately 1.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />, the duration of the offsite power outage: the Unit 2 and Unit 3 Local Government Radios were not functional (the EOF transmitter UPS did not operate, but commercial telephone back-ups were used); Unit 2 and Unit 3 Radiological Emergency Communications System (RECS) were not functional due to the failure of the associated EOF UPS (commercial telephone back-up was used); and the Five-Way and Three-Way Direct Telephone Systems did not function due to UPS failures. These UPS-powered automatic ringing telephone systems between the TSC/OSC, Control Rooms, EOF/Alternate EOF, JNC, and White Plains office are dedicated telephones typically used by the ERO staff as the primary means of inter-ERO communications and planning.

Analysis.

The failure of the EOF UPSs to function on August 14, 2003, impacts the Emergency Preparedness Cornerstone, Non-Risk Significant Planning Standard 10 CFR 50.47(b)(8), which states that adequate facilities and equipment are maintained to support emergency response. Traditional enforcement does not apply because the issue did not have an actual safety consequence or potential for impacting the NRCs regulatory function, and was not the result of a willful violation of NRC requirements or Entergy procedures. This finding is considered more than minor because the objective of the Emergency Preparedness Cornerstone, to ensure adequate facilities and equipment are capable of protecting the health and safety of the public in the event of a radiological emergency, was adversely impacted. A significant amount of the Unit 2 and Unit 3 emergency response organization communications equipment was non-functional on August 14 until off-site power was restored. However, this finding is of very low safety significance because key members of the ERO were able to implement established compensatory measures to effectively perform their emergency response functions from the EOF, TSC/OSC, and Unit 2 and 3 central control rooms, using back-up telephone and face-to-face communications. This determination was made using Manual Chapter (MC) 0609, Appendix B, Emergency Preparedness Significance Determination Process, Sheet 2. Specifically, the August 14 loss of offsite power event resulted in a Notice of Unusual Event declaration, consistent with established Emergency Action Level guidelines. The EOF UPS failures and consequential dedicated ERO communications circuit failures constitutes a failure to implement a program element vice a failure to comply with or satisfy a Planning Standard function.

Therefore, this finding is Green.

(FIN 50-247/2003-013-05 and 50-286/2003-010-03)

Enforcement.

No violation of regulatory requirements occurred.

4OA2 Problem Identification and Resolution (PI&R)

.1 Annual Sample Review

The 95001 inspection activities (see Sections 02 and 03) encompassed the planned review of a PI&R sample involving the failure of 345 kV breaker No. 3 and licensee identified Buchanan switchyard problems with electrical disconnects (reference Condition Report No. IP3-2002-04550). This PI&R sample inspection activity was planned prior to the June 22, 2003, second failure of breaker No. 3 and the Unplanned Scrams in 7000 Critical Hours Performance Indicator White threshold being crossed.

Accordingly, this PI&R sample inspection is being credited in this report. A Green finding involving the lack of appropriate contractor oversight of maintenance activities was identified (see Section 02.04).

4OA6 Meetings, Including Exit

The inspectors met with Entergy representatives periodically throughout the inspection and at the conclusion of the inspection on November 7, 2003, to review the purpose and scope of the inspection and to discuss the teams preliminary findings. The exit meeting on November 7, also served as the Regulatory Performance Meeting for the one Green finding associated with the 95001 inspection activities. Entergy acknowledged the teams preliminary inspection findings and did not take issue with the findings preliminary characterizations.

The inspector asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was reviewed during this inspection.

4OA7 Licensee-Identified Deficiency

The following violations of very low safety significance (Green) were identified by the licensee and are violations of NRC requirements which meet the criteria of Section VI of the NRC Enforcement Policy, NUREG-1600, for being dispositioned as a Non-Cited Violation.

.1 Following the Unit 2 reactor trip and entry into a natural circulation cooldown on August

3, 2003, the licensee identified that the Shift Manager failed to make the required 10 CFR 50.72, 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> report, within the specified time period. This oversight by the on-duty Shift Manager was identified by his relief during a detailed log review and post-trip assessment. The 50.72 report was made at 11:10 a.m., approximately two and one-half hours late (reference Emergency Notification System No. 40045).

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Entergy:

Vincent Andreozzi

Earl Libby

Chris Schwarz

John McCann

Richard Louie

William Blair

Thomas McCaffrey

Anthony Williams

Lizbeth Lee

Fred Dacimo

Joseph Reynolds

Frank Inzirillo

Charles Braun

Eric Anderson

Joseph Raffaele

Louis Cortopassi

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Open/Closed FIN 50-286/2003-010-01 Poor workmanship, improperly performed corrective maintenance, and inadequate contractor oversight contributed to the failure of 345 kV breaker No. 3 on two separate occasions.

FIN 50-247/2003-013-01 Failure to take appropriate and timely corrective actions to address the repeated grid-related reactor trips of Unit 2.

NCV 50-247/2003-013-02 TS 6.8.1 violation - failure to adhere to Emergency Operating Procedure ES-0.1, continuous action step 1.0 on August 3, 2003.

URI 50-247/2003-013-003 Acceptability of the Unit 2 simulator modeling of decay heat load and auxiliary feedwater cooldown.

FIN 50-247/2003-013-04 Failure of the Unit 2 TSC diesel on August 14, 2003 - failure to implement non-risk significant planning standard program element.

FIN 50-286/2003-010-02 Failure of the Unit 3 TSC diesel on August 14, 2003 - failure to implement non-risk significant planning standard program element.

Attachment

FIN 50-247/2003-013-05 and 50-286/2003-010-03 Failure of the EOF UPSs on August 14, 2003 - failure to implement non-risk significant planning standard program element.