ML042960499
| ML042960499 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 10/18/2004 |
| From: | NRC Region 4 |
| To: | |
| References | |
| FOIA/PA-2004-0307 | |
| Download: ML042960499 (32) | |
Text
Slide 1
.0.0 Good evening. My name is Tony Gody. I led the Nuclear Regulatory Commissions Augmented Inspection Team and coordinated the development of our conclusions which are the subject of today's meeting.
The exit meeting being-held thit-eveninrg is a
-meeting between the NRC and the management of the Palo Verde Nuclear Generating Station.
While we recognize'that many of the focus areas being discussed to'night'affected mrany of'you directly. We would like'you to save your --
questions for the time period reserved following the exit meeting with Palo Verde Management.
In addition, we request, that, when the time comes for public comment, you limit your
questions to the fact-finding aspects of the June 14, 2004 loss of offsite power event.
Before we begin, let me introduce the.NRC staff here this evening.,
To my left is Br'uce. Mallett.. llHe' is the'..
Administrator of NRC RegionI'lV.'
To my right is:'Chuck Paulk. Chuck was a senior engineer on my-team.
In the audiencejis Roger Hanna,' heis the Region 11 Public Affairs Officer.
Mr. Overbeck, would you like to introduce your staff.
Thank you.
Mr. Overbeck, before I begin, I would like to
,thank you and the members of.your staff for their willingness to work with us in obtaining the information we needed to assess the, circumstances surrounding the June 14th loss of offsite power event. In every casei your staff was open to our questions and professional.
This evening-I will-discuss the following'to'pics:
The augmented inspection team charter.
The bases for the augmented inspection team effort.
The'sequence of events.
Our'assessment of plant system perfoirmance.
Our-assessment of plant procedures.
Our assessment of ho'w well facility personnel responded to'the event. '
The apparent cause of certain aspects of the' event.
An assessment of iany maintenance related aspects of the event.'
An assessment of'thefacilities interaction 'with and coordination of off-site activities.'
An assessment of the overall risk posed by the event.
Before I. begin discussing the details of our; inspection, let me discuss the overall conclusion.
The AIT implemented approximately 300 hours0.00347 days <br />0.0833 hours <br />4.960317e-4 weeks <br />1.1415e-4 months <br /> of staff effort reviewing the circumstances surrounding the June 14 event. Overall, we found that an equipment failure in the West Wing substation located approximately.40 miles from the Palo Verde Nuclear Generating Station resulted in a ground-fault not isolating from.the electric power grid. This uninterrupted. ground-fault caused.-the, protective tripping.of a number of other transmission lines ultimately precipitating, into a loss of offsite power. event at the Palo Verde Nuclear Generating Station.
The team found that the Palo Verde Nuclear Generating Station responded appropriately to the event by placing all three units in a safe condition.. Nevertheless, a number of issues occurred which required. both immediate and long-term corrective actions. The team verified that. all immediate corrective actions, required for plant re-start, were.implemented~properly prior.
to re-starting the facilities. Other issues, some of which will be discussed tonight were highlighted for a followup NRC inspection to be held in September.
Slide 2
- ~
~
6 0il..
.¢X, U.~
6.
It is the policy 'of the United -States Nuclear.'
Regulatory Commission to.ensure th'at significant operational events involvin'gr~eacto'r and material facilities license-d by the NRC are' investigated in -a timely,: objective',: Syst ematic, and technically sound manner; that the-factual information pertaini'ng to each event is documenited; and that the ca'us'e or 'causes' of, each event arescrtain-e'd.
On June 14, 2004-, all offsite po~wer supplie's to' the Palo Verde' Nuclea'r Generating Statfioon.were disuptdwit aconcurrent. trip of all1 three'u'nits.
Add itionlallytheUn*it 2Em-ergency Diesel Generator "A" failed. NRC_ Region-IV co'nducted a preliminary assessment of the'eVen'tanid
concluded it met the criteria contained'in NRC.
Management Directive 8.3 for an Augmented Inspection.
Early on June 14th NRC management was informed 'of the losse off offsite' power'event at Palo Verde and.immediately began. identifying team members with' the experience '`necessary to evaluate a complicated transmission system originated. event: Chuck and '-immediately left for the Palo Verde.facility and arrived here in Goodyear that evening. In addiion, other members of the Augmented Inspection Team not already. located here travelled to the.facility..
On the morning of June 15, Chuck and I received updates from the Region IV management team and reported to the Palo Verde facility.
Beyond myself and Chuck whom I have already introduced, the augmented inspection team consisted of the following personnel: (1) Tim McConnell - a reactor engineer temporarily assigned to the Palo Verde facility, (2) Peter Alter, a Senior Resident Inspector from the River Bend facility in Louisiana, (3) Tom Koshy and (4)
Amar Pal, both electrical engineers from the Office of Nuclear Reactor Regulation in
An assessmenft'of-the overall risk posed by the event. -
The sequence of events was developed through a detailed and independent review of alarm printouts, computer records, and interviews of plant personnel.
Our assessment of system performance involved a review of the details of each equipment problem that occurred during and following the event.-In addition, -the team independently.compared the computer data with the expected plant response described-in the plants licensing bases.
Rockville, MD, (5) Joe Tapia, a Senior Reactor Engineer from our regional office in Arlington, Texas, and (6) George Skinner, an electrical contractor. The team represented well over 170 years of nuclear experience.
As discussed earlier, the augmented inspection team charter tasked the team to develop:
The sequence of events.
An assessment of plant system performance.
An assessment of plant procedures.
An assessment of how well facility personnel responded to the event.
The apparent cause of certain aspects of the event.
An assessment of any maintenance related aspects of the event.
An assessment of the facilities interaction with and coordination of off-site activities.
Slide 3'
, ',,..
I "
It
- IV PI"R----P-"W-ft Each equipment problem, 'human performance problem, or difference in plant response from what was expected was reviewed from the perspective of procedural adequacy.
Each problem was critically reviewed in detail and the team assessed how well the facility personnel responded to the event.
Each equipment failure and/or human performance error was critically reviewed in detail and the team compared its' own assessment to the licensee's to ascertain if the licensee's apparent cause of the failure'or error was reasonable. When differences were noted,
the team engaged facility management in a discussion of the apparent cause.
Slide 5 On June 1.4, 2004, at approximately 7:41 a.m.
MST, a ground-fault occurred on Phase."(C" of a 230 kV transmission line in northwest Phoenix,-
Arizona, between the "West Wing" and "Liberty" substations located -approximately 47 miles from the Palo Verde Nuclear Generating Station. A failure in the protective -relaying resulted in the -
ground-fault not isolating from the local grid for.
approximately 38 seconds. This uninterrupted fault cascaded into the protective tripping of a number of 230kV and 525kV transmission lines, a nearly concurrent trip of all three Palo Verde Nuclear Generating Station units -and.the loss of six additional generation.units nearby within approximately 30 seconds of fault initiation. This represented a total loss of nearly 5,500
II Slide 4
)
All potential maintenance related contributions to the event were reviewed and compared to the licensee's assessment.
To ascertain if the licensee's efforts to coordinate activities with off-site organizations, the team reviewed all emergency notifications and communications with the electrical grid operators. Interviews of licensee staff and grid operators were conducted.
Finally, the team worked with the Region IV Senior Reactor Analyst to develop a clear overall risk assessment of the event.
to; #9-;
Slide 6:
t
. m i ".. ..' ?
.
I.- - -1 P-PO -N-"
M L
~
The-Unit 2-Train."A" emergency diesel generator (EDG) started but failed early in the load sequence process-due 'to a diode with less than 70 hours8.101852e-4 days <br />0.0194 hours <br />1.157407e-4 weeks <br />2.6635e-5 months <br /> of run time,.in the, exciter., rectifier circuit' failed, causing a shortlcircuit. This resulted in the Train "A" Engineered Safeguards Features busses deoenergizing, which limited the availability of certain safety equipment for operators. Because of this failure, the licensee elevated the emergency declaration for Unit 2 to an Alert at 7:54 MST. All three units were safely shutdown and stabilized under hot shutdown conditions.
An NRC Augmented Inspection Team was dispatched to the site later that same day.
megawatts-electric (MWe) of local electric generation. Because of the loss of offsite power (LOOP), the licensee declared a Notice of Unusual Event for all three units at approximately 7:50 a.m. MST.
Overall, we found that an equipment failure in the West Wing 'substation located approximately 40 miles from' the Palo Verde Nuclear Generating. Station resulted in a ground-fault not isolating from-the electric power grid.' This uninterrupted ground-fault caused the protective tripping of a number of other transmission lines ultimately precipitating into a loss of offsite power event at the Palo Verde Nuclear Generating Station. Improvements were made to increase both reliability and independence prior to re-starting the facility.
Slide 7
~ ~
~
I]
Eel:._j As Idiscussed earlier,-a ground-fault -occurred.
on Phase ".C",of a,230 kV transmission line in.
northwest Phoenix, Arizona, between the "West Wing" and "Liberty" substations located approximately 47 miles from the-Palo Verde Nuclear Generating Station. A failure in the protective relaying resulted in the ground-fault not isolating from the local grid for approximately 38 seconds. This uninterrupted fault cascaded into the protective tripping of a number of 230kV and 525kV trans mission lines.
Apparent causes:
The ground-fault initiated because of a bird taking off of a tower emitted organic matter at takeoff. The organic matter shorted across the
The team found that the Palo Verde Nuclear Generating Station responded appropriately to the event by placing all three units in a safe condition. Nevertheless, a number of issues occurred which required both immediate and long-term corrective actions. The team verified that all immediate corrective actions, required for plant re-start, were implemented properly prior to restarting the facilities. One of those issues was the repair of the Unit 2 Train "A" EDG.
Other issues, some of which will be discussed tonight were highlighted for a followup NRC inspection to be held in September.
Phase "C" insiulator. Once the short began'and the air 'ionized, the 'fault confinued while' the' insulator failed. High fault "currents in 'the static line above the transmission lines resulted in the static line failing at several points changing the characterization of the fault.
The failure in the protective relaying occurred when a relay in the "WESTWING" substation failed to open a breaker. Once this breaker failed, other protective devices began to gradually isolate other transmission lines as they sensed the fault.
As l discussed earlier, the Palo Verde Nuclear Generating Station response to the event was generally acceptable. 'What this means is all three nuclear units automatically shutdown and initiated actions to ensure the reactor was cooled. Control room operators took appropriate actions as needed to ensure the maximum levels of safety.
The team found that the licensee aggressively pursued the causes of these failures and issues.
Arizona Public Service and Salt River Project implemented corrective actions to improve the reliability of the offsite power sources prior to starting up the Palo Verde Nuclear Generating
Station. The NRC plans on reviewing the implementation of those corrective actions during a followup inspection.
Slide 8',
-., 4M.
As discussed earlier,--the Unit 2 Train"A" emergency diesel, gnrto (E DG) sta'rte"d, b ut failed early in the load sequence processd de to a diode with less than 70 hours8.101852e-4 days <br />0.0194 hours <br />1.157407e-4 weeks <br />2.6635e-5 months <br /> of run time in the exciter rectifier circuit failed, causing a short-circuit. This resulted in the Train "A" Engineered Safeguards Features busses de-energizing.
The Palo Verde Nuclear Generating Station repaired this problem before starting Unit 2 and planned on conducting a root cause analysis of the failed diode. The NRC will review this root cause analysis when it is complete.
During and following the event a number of unnecessary equipment failures became
apparent. Each of these failures had diverse apparent causes and, at most, only moderately affected the event response.
For example:
An atmospheric dump valve (ADV) on Unit 1 drifted closed due to an apparent equipment malfunction which posed a minor operational nuisance to the control room operators during the event. The ADV was repaired prior to placing it back in service. The NRC plans on conducting a followup inspection to review the licensee's root cause and extent of condition review.
Slide 9-While the majority of procedures were implemented without error, some problems were noted. In each case, the licensee took effective actions to mitigate the impact of the procedure issue. For example:
Operators did not anticipate that the Unit 1 letdown system would not automatically isolate because a temporary modification was not fully understood or translated into operating procedures. This resulted in high temperatures in'that system. The high temperatures resulted in fumes being generated as paint heated up which precipitated a fire brigade response. This complicated the Unit 1 event.
The Unit 2, Positive Displacement Charging Pump "E" was temporarily lost due to human performance errors. This resulted in a total loss of Unit 2 charging flow for a short period.
Both of these examples were corrected prior to re-starting the facility. The NRC plans on conducting a followup inspection to verify that the cause determination, extent of condition, and long term corrective actions were appropriate.
Slide 10 ', :
.c
.r As indicated earlier, theteam found tha t Palo Verde Nuclear Generating Station personnel.
generally' respoinded to tAhe event 'in an acceptableomanner. Neverthele s,
-anumbero lessons were learne-d with re'spect to the,-
emergency 'response -organization'. For example:
The ability of licensee personnel to use the notification alert network and to develop protective action recommendations, had they been needed, appeared to have been affected by the loss of power.
i a Communication and coordination issues affected the notification of state and local officials of emergency classifications.
Human performance errors resulted in delays in notifying the emergency response organization.
The team found that the Unit 2 notification of state and local officials was appropriate.
Immediate actions were taken to correct the deficiencies. For example, uninterruptible power was supplied to computers for protective action recommendations. In addition, training was conducted to correct the issues related to human performance. The NRC plans on conducting a followup inspection to assess the licensees cause determination, extent of condition, and effectiveness of long term corrective actions.
Slide 1 1
'I'.
- -,. 4 ,
.1 _7%,,,
I.
. I
- I.
.:
4 3-s-go**
-e S.
S Despite the fact the the teamn found that Arizona Public Service and Salt River Project maintained the switchyard and substation equipment annually, a relay failed causing the loss'of offsite power event.
Several issues were revealed which have -
maintenance related aspects. For example:
The Technical Support'Center (TSC):'EDG failed because a test-switch was not returned to it's:-
proper position following maintenance six days prior to the event. As a result, the emergency response organization assembled in the alternate TSC. This resulted in some confusion
I-and posed some unique challenges to the emergency response organization.
A check-valve leakage problem in the Unit 3 safety injection system resulted in operators having to manually depressurize the low-pressure safety injection system three times during the event. This posed an unnecessary additional distraction for the event.
Two non safety-related Magna-Blast circuit breakers failed to operate during recovery operations in Unit 1 and Unit 3 which delayed electrical system recovery efforts.
The team found that the licensee's immediate corrective actions to repair the TSC EDG, stop the check valve leakage, and troubleshoot the non-safety related breaker issues appropriate.
The NRC plans on conducting a followup inspection to review the licensees cause determination, extent of condition, and effectiveness of corrective actions.
Slide 12 I
The team -concluded that the coordina'tion'w~ith offsite electrical -organizations was,-very good and the remedial measures coordinated between PVNGS, SRP, and APS personnel improved reliability and independence and appropriately minimized the possibility of a similar LOOP event occurring in the PVNGS 500 kV switchyard.
The three transmission lines between the Hassayampa and PVNGS switchyard were designed with negative sequence relaying.-
intended to serve as pole mismatch protection.
This design was implemented in 1999 as. part of extensive modifications to the Hassayampa switchyard intended to accommodate-new co-
-a-generation facilities local to the PVNGS. The negative sequence protection scheme was designed to actuate a complete isolation of all three of the subject transmission lines after a 5Dsecond time delay to avoid spurious tripping due to faults. Although these individual lines are considered as separate sources of offsite power by the NRC, this event demonstrated that the lines were subject to'simultaneous failure (acting as one) because of the protective relaying scheme. Personnel employed by SRP and the licensee stated that the negative sequence relaying was disabled and pole mismatch protection was being implemented by alternate relaying.
Slide 1 3
...1 1.1,
I 1 1
, r.1 "..
I-I I
. i'. I..
-, t
,1:
." 11 -
1, I
r_- -
I I.
11_7 II Despite the number of challenges to the plant operating staff and management, all three units were safely shutdown, placed in a stable condition immediately following the LOOP event, and power restoration efforts began immediately. With the exception of the local 525kV transmission grid surrounding the Palo Verde switchyard, the Arizona, California, and Nevada electrical grid remained relatively stable, only noting the fault through some minor frequency and voltage fluctuations. This was notable considering the amount of generation lost..The total local generation lost during the event included the three Palo Verde units, three coogeneration units at the Red Hawk Generating Station, and three co-generation J
units at the Arlington Generating Station for a total of approximately 5,500 MWe.
Slide 14 This concludes the public exit with the Palo Verde Nuclear Generating Station. We will now transition to the public question and answer period.
Slide 15
)