ML20044G230
| ML20044G230 | |
| Person / Time | |
|---|---|
| Site: | Crystal River  |
| Issue date: | 05/17/1993 |
| From: | Fillion P, Freudenberger, Holmesray P, Landis K NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II) |
| To: | |
| Shared Package | |
| ML20044G221 | List: |
| References | |
| 50-302-93-11, NUDOCS 9306020207 | |
| Download: ML20044G230 (11) | |
See also: IR 05000302/1993011
Text
..
. .. . , _ .
- _ _ _ _ _ _ _ _ - _ _ _ _ . - -
. _ _ _ _ _ _ _
.___
_
,_
_
. [/*"4
UNITED STATES
g
j,,
NUCLEAR REGULATORY COMMisslON
g
o
RE GION ll
$
j$
101 MARIETTA STRE ET N.W.
l
.g
ATL ANTA, GEORGI A 30323
l
%, -
/
l
...,<
,
Report No.: 50-302/93-11
Licensee: Florida Power Corporation
3201 34th Street, South
St. Petersburg, FL 33733
Docket No.:
50-302
License No.: DRP-72
Facility Name: Crystal River 3
Inspection Conducted: April 12 - 17, 1993
Inspect r: I d_
>MA
P. Holmes-R v, Senior Resident Inspector
Datd Signed
SIvd3
Inspec or:
er
pR. Fretidenberger, Resident Inspector
Date Signed
!.
M b _
IMT/93
Inspector
P.~Fillion , actor Inspector, RII
Date Signed
~
Approved by: 4
cot
f75
m
K. landis, Section Chief
Date Signed
Division of Reactor Projects
SUMMARY
Scope:
This special inspection was conducted by two resident inspectors and a region
,
l
based inspector in the area of followup to operational events. The inspectors
addressed questions to determine the circumstances surrounding a three minute
'
!
interruption in the operation of the decay heat removal system due to a loss
I
of the 500kV offsite power source that occurred on April 8, 1993. Offsite
power remained available through the 230kV switchyard during the event.
Results:
,
The loss of 500kV offsite power was the result of inadequacies in plant
procedures which constituted a' violation of NRC. requirements. The licensee's-
short term corrective actions were adequate. A root cause analysis was
underway at the time of the inspection. Long term corrective actions were to
be defined after completion of the root cause analysis. One violation was
cited:
VIO 50-302/93-11-01, Failure to establish an adequate procedure for
operation of the 500kV switchyard DC control power system (paragraph
4.6).
9306020207 930517
ADDCK 05000302
O
m
_
REPORT DETAILS
1.
Persons Contacted
Licensee Employees
- J. Alberdi, Manager, Nuclear Plant Operations
- G. Boldt, Vice President, Nuclear Production
- R. Davis, Manager, Nuclear Plant Maintenance
- M. Fitzgerald, Supervisor, Nuclear Plant Systems Engineering
- E. Froats, Manager, Nuclear Compliance
- G. Halnon, Manager, Nuclear Plant Technical Support
- B. Ilickle, Director, Nuclear Plant Operations
K, Lancaster, Superintendent, Nuclear Maintenance Work Controls
- W. Marshall, Manager, Nuclear Plant Operations
W. Morgan, Field Supervisor, Substation Maintenance
- S. Powell, Manager, Electrical Shop
W. 5tevenson, Supervisor, Shift Technical Assistants
- J. Terry, Manager, Nuclear Plant Systems Engineering
Other licensee employees contacted during this inspection included
engineers, operators and technicians.
NRC Inspectors
- P. Holmes-Ray, Senior Resident Inspector
- R. Freudenberger, Resident Inspector
- P. Fillion, Reactor Inspector, RII
- Attended exit interview
Acronyms and initialisms used throughout this report are listed in the
last paragraph.
2.
Background and Inspection Plan
Crystal River 3 had four reportable events involving degraded offsite
power between March 13 and April 8,1993 (Reference EN 25241, 25273,
25325, and 25377). The most recent event occurred while performing
maintenance on the 500kV DC control power battery. The resident
inspectors, augmented by one Region II based electrical engineer, .
evaluated the root cause of this event and corrective actions.
In
addition, the most recent loss of power event was evaluated in terms of
similarity to current events at other sites and other non-reportable
switchyard events at Crystal River during the maintenance shutdown. The
evaluations were accomplished by addressing the following inspection
elements which were prepared by NRC management:
a.
Independently assess management oversight and effectiveness in
accurately determining the root cause for the loss of 500kV
offsite power event.
b.
Determine the technical adequacy of the short and long term
corrective actions.
<
,
2
c.
Determine the similarity of this event to the recent event
at Vermont Yankee described in NRC Information Notice 91-81,
Switchyard Problems that Contribute to Loss of Offsite Power;
and the event at Oconee described in NRC Inspection Report
!
50-269/92-26 (Augmented Inspection Team).
d.
Did the licensee properly assess the risk factors associated with
the work to be accomplished in the 230kV and 500kV switchyards.
e.
Determine if there are any sufficiently common factors in the
three recent degraded offsite power events for which the
licensee's corrective action should have prevented occurrence of
the most recent event.
f.
Determine if the operational controls (including training) on
switchyard activities are adequate.
!
3.
Description of Event
'
On April 8,1993, the plant was in Mode 5 with a steam bubble in the
,
pressurizer. The train
'A' diesel generator was functionally inoperable
due to maintenance activities.
Train 'B' was being powered from the
500kV bus via backfeed through the generator step up transformer. The
230kV bus was energized and available as a power source via the offsite
power transformer. At about 6:00 p.m., plant personnel were in the
process of replacing a cell in the battery which provides control power
for the 500kV switchyard. The work was authorized through work request
.
308885. The work request did not include instructions for performing
~l
the necessary switching operations to disconnect the battery from the
charger.
Nor was an equipment clearance order utilized. With the
!
battery charger connected to the system, electrical maintenance
personnel opened a switch to disconnect the battery. Within seconds of
disconnecting the battery, 500kV circuit breakers tripped, deenergizing
the safety-related buses.
Since the battery charger was not designed to
operate independent from a battery, the charger produced voltage outside
the specified operating range when the battery was disconnected.
Apparently, the abnormal voltage on the system damaged a solid state
,
!
auxiliary tripping relay, causing it to go into the tripped condition,
tripping the 500kV breakers. This deenergized the 'B' 4160V Engineered
Safeguards electrical bus in service which resulted in an automatic
start of the
'B' Emergency Diesel Generator and a three minute
interruption in the operation of the decay heat removal system until it
was manually restarted.
Preliminary analysis of the event indicated that safety-related systems
-
operated properly in response to the loss of power. The sequence of
events recorder printout showed that power was restored to the
safety-related buses approximately 13 seconds after the 500kV breakers
,
tripped and approximately 8 seconds after the Emergency Diesel Generator
received a start signal. Operators transferred to the 230kV power
source about two hours later.
The 500kV bus was restored to service
about two and one half hours after the initial event.
I
.
l
!
!
'
3
4.
Inspection
The inspection focused on the following inspection elements which are
\\'
addressed individually below.
l
4.1
Inspection Element (a):
Independently assess management oversight
[
and effectiveness in accurately determining the root cause for the
!
loss of 500kV offsite power event.
[
The inspection identified that there were three aspects of the
[
event which had some form of cause analysis ongoing. The
l
technical analysis of equipment performance, plans for an
i
evaluation of the human performance aspects of the event, and
i
plans for a Management Review Team of commonality of switchyard
i
events at the site during the maintenance outage were reviewed by
the inspectors.
,
i
Following the event, all of the relays potentially affected were
tested and replaced, as necessary.
Since the system had been
returned to service prior to the inspection, the technical
analysis of the hardware involved was virtually complete.
j
Nonetheless, the licensee had not determined the magnitude and
.
waveshape of the charger output voltage during the period of time
{
that the battery was disconnected.
Therefore, it could not be
demonstrated that the relay failure mode was as expected or that
the battery charger had been operating properly. A printout from
the switchyard sequence of events recorder had not been obtained.
Actions such as these appear to have been warranted to demonstrate
'
that the equipment involved operated as expected for the
conditions created by the alignment of the system which caused the
event. Management oversight of the technical aspects of the root
cause analysis, although adequate, could have been improved to
>
allow a more thorough understanding of the event and to ensure
comprehensive corrective actions.
The Nuclear Maintenance Work Controls Superintendent and the
i
Nuclear Safety Supervisor were assigned the task of performing the
root cause analysis. They plan to utilize Compliance Procedure
CP-144, Root Cause Analysis, to perform an evaluation of the human
performance aspects of the event. Some of the subsidiary
'
questions that the analysis will attempt to answer are:
What was the sequence of events and personnel actions?
!
-
I
Is the division of responsibility between the plant and
-
switchyard maintenance groups adequately defined?
'
Is there a general problem with maintenance crew shift
i
-
turnovers?
l
-
Is communications among operations, the Outage Shift Manager
and substation maintenance effective?
!
.
.
_.
I
i
!
!
!
4
[
-
Is training related to the switchyard equipment for
I
maintenance and operations personnel adequate?
t
-
What is the detailed description of the event in terms of
'
hardware performance?
.
!
In addition to the evaluation of the human performance aspects of
'
this event, the licensee had established a Management Review Team
to review the switchyard events associated with the maintenance
outage. The Management Review Team has responsibility to evaluate
the adequacy of root cause evaluations, short and long term
'
corrective actions; and to identify generic problems, and cost
effective enhancements which would serve to prevent similar events
in the future.
[
t
Management decisions to perform the evaluation of the human
l
performance aspects and to establish the Management Review Team
i
were sufficient to establish the basis for the identification of
!
the significant contributing causes of the April 8 interruption of
shutdown cooling and commonality to other switchyard problems
j
during the maintenance outage.
.
4.2
Inspection Element (b):
Determine the technical adequacy of the
f
short and long term corrective actions.
!
'
The primary short term corrective action to prevent recurrence was
an operations Standing Order issued .on April 9, which prohibited
maintenance activities in the switchyard (230kV or 500kV) that was
supplying power to the safety-related buses, and therefore
a
'
shutdown cooling, while in Modes 5 and 6.
The licensee plans to
,
incorporate this prohibition into applicable shutdown operations
!-
procedures. This action alone is sufficient to prevent work in
the switchyard from affecting decay heat removal.
f
The licensee plans to define long term corrective actions when the
root cause analysis and Management Review Team activities
described above are complete, therefore, long term corrective
,
actions were not evaluated as part of this inspection. However,
,
'
the inspectors noted that the plans for evaluation of the event
did not include a review of the adequacy of existing procedures
-
associated with operation of systems in the switchyards.
4.3
Inspection Element (c): Determine the similarity of this event to
i
the recent event at Vermont Yankee described in NRC Information
i
Notice 91-81, Switchyard Problems that Contribute to Loss of
Offsite Power; and the event at Oconee described in NRC Inspection
Report 50-269/92-26 (Augmented Inspection Team).
On October 19, 1992, the Oconee plant experienced a loss of. power
which was initiated when a DC System bus tie breaker was opened,
l
1 aving a portion of the system powered only by a battery charger
without a battery. Since the charger could not maintain a proper
I
l
,
h
-
,
-
!
5
I
voltage in this configuration, the resultant voltage excursion
'
caused zener diodes in the breaker failure relays to conduct which
caused breaker failure relay operations. This in turn set in
motion circuitry which caused loss of power to plant auxiliaries.
Post-event testing on the charger identified that the charger was
not functioning properly. This was not discernable by observation
of the voltmeters.
,
i
NRC Information Notice 91-81, Switchyard Problems that Contribute
to Loss of Offsite Power, describes an event that took place on
i
April 23, 1991, at the Vermont Yankee Plant very similar to the
Oconee event. The Vermont Yankee event began when the battery was
i
disconnected from the DC System. The charger output became
'
unstable which caused breaker failure relay operations which
caused the loss of power event.
Common elements in these two events are the fact that battery
'
chargers may not be able to maintain voltage within acceptable
limits when the battery is not connected to the system, and a
certain style of breaker failure relay is particularly vulnerable
to overvoltages.
Also in both these events the battery chargers
were not functioning properly.
The Crystal River event began when the battery was disconnected
from the bus leaving only a charger to supply the loads. The
,
charger was not designed for this mode of operation. This aspect
of the event was exactly like the Oconee and Vermont Yankee
events.
Relays were damaged during the Crystal River event, but
breaker failure relays were not involved.
In this respect the
!
events were similar but not exactly alike. The licensee had
previously addressed the breaker failure relay issue and
determined that their breaker failure relays did not have the
vulnerability described in the notice and report.
With respect
'
to the possibility that the battery charger was not functioning
properly thus aggravating the voltage transient, the licensee has
not tested the charger as part of the event analysis and therefore
the answer is indeterminate at the time of this inspection. The
licensee stated that there was no periodic preventive maintenance
program for the chargers in question.
Consequently, the analysis
of the available information was limited in scope.
The licensee was aware of the Oconee and Vermont Yankee problems
associated with isolated charger mode of operation. However,
since the analysis of the information available was narrowly
focused, it precluded the identification of the similar
vulnerability at Crystal River.
4.4
Inspection Element (d): Did the licensee properly assess the risk
factors associated with the work to be accomplished in the 230kV
and 500kV switchyards?
1
'
At the time this inspection element was developed, it was believed
l
l
i
-
L
i
6
that work had been ongoing in both the 230kV and 500kV switchyards
at the time of the event. The inspection revealed that this was
not true. Work was authorized on equipment in the 230kV
switchyard associated with the functional testing of the newly
installed Backup Engineered Safeguards Transformer. This work was
on hold while the cause was being determined for an inadvertent
deenergization of the 'B' 230kV bus during day shift on April 8.
Although work was not being performed in both the 230kV and the
500kV switchyards simultaneously, the inspectors determined that
'
the risk of interrupting power to the shutdown cooling systems
'
while performing the 500kV switchyard battery cell replacement was
not adequately evaluated by the licensee. This observation was
,
supported by the fact that the Offsite Power Transformer was
operable and available to supply the shutdown cooling loads during
the replacement of the 500kV switchyard battery cell; however, the
control room operators did not recognize a need nor were they
advised to transfer to the alternate power supply. As noted in
l
paragraph 4.2, above, a Standing Order was issued on April 9,
'
which prohibited maintenance activities in the switchyard (230kV
or 500kV) that was supplying power to shutdown cooling systems.
4.5
Inspection Element (e): Determine if there are any sufficiently
common factors in the recent three degraded offsite power events
for which the licensee's corrective action should have prevented
occurrence of the most recent event.
The three events referred to by this inspection element were the
three previous reportable events which occurred on March 13,
March 17, and March 29. Additionally, three non-reportable events
which occurred on March II, March 25, and April 8, were also
evaluated. They are briefly described below.
On March 11, during maintenance activities on 230kV breaker
-
No. 1157 (Unit 1 tie breaker - fossil unit) by the
substation construction crew, 230kV breaker No. 1692 was
inadvertently tripped.
Breakers 1691 and 1692 are the tie
,
breakers for the 230kV Unit 3 startup transformer. Breaker
No. 1691 remained closed so the offsite power transformer
remained energized.
Safety-related buses were energized
'
from the 500kV scurce at the time. Miscommunication and
failure to follow procedure by the substation crew played a
role in the event. There was no power loss to the unit.
(reference Problem Report 93-59)
[
-
On March 13, severe weather caused a degradation of the
,
offsite power supplies. The Startup transformer tripped and
l
one of two 500kV backfeed breakers opened. There was no
power loss to the unit.
(reference Problem Report 93-69 and
!
,
1
.
.
t
!
!
7
>
1
-
On March 17, light rain combined with salt, that had
!
deposited in the switchyard due to the March 13 severe
!
weather, to form electrolyte. This resulted in electrical
degradation and loss of 230kV switchyard.
500kV backfeed
!
remained available.
(reference Problem Report 93-95 and EN 25273)
-
On March 25, a substation relay technician placed his test
instrument leads on the wrong terminals which caused a bus
-
differential relay operation which caused deenergizing the
'B' 230kV bus.
(reference Problem Report 93-87)
l
'
-
On March 29, a control power cable in the 500kV switchyard
failed causing a loss of the 500kV backfeed and interruption
of shutdown cooling.
(reference Problem Report 93-95 and EN
l
25325)
-
On April 8, while performing functional testing on the new
backup emergency safeguards transformer, a plant technician
i
,
placed a jumper wire on the wrong terminals which caused
.
operation of a lockout relay which deenergized the 'B'
230kV
!
'
bus. The technician was performing trouble shooting at the
time. A typographical error in the procedure and a drawing
i
error played a role in this event.
The safety-related buses
were being powered from the 500kV bus at the time of this
event.
(reference Problem Report 93-112)
The non-reportable events and the event which is the main subject
of this inspection have one common factor; all involved a problem
with procedures (i.e., failure to follow procedure or inadequate
procedure). However, the inspectors determined there was not
!
,
sufficient commonality among the events. Consequently, the latest
event was not a result of inadequate corrective action for the
previous events.
4.6
Inspection Element (f):
Determine if the operational controls
(including training) on switchyard activities are adequate.
i
Controls on activities in the switchyards at the Crystal River
site were delineated in the CR-3/0thers Interface Matrix Manual
and Administrative Instruction AI-1300, Crystal River Coal Plants,
Substation, ECC, and Relay Departments Interface with Crystal
i
River Unit 3.
Inspector review of these documents identified that
,
these documents provided direction for control of activities in
the switchyards such as; notification of the Unit 3 SS00 for
t
routine entries, and notification of the Unit 3 Senior Nuclear
i
'
Scheduling Coordinator for preplanning of maintenance and/or
modification activities. Additional controls had been implemented
i
following the loss of offsite power event at plant Vogtle,
including limiting access to a main gate, requiring flagmen for
backing of vehicles in the switchyards, and restricting the use of
hazardous materials in the switchyards.
t
-
_ _ _ _ _ _ _ _ _ _ _ .
8
9
There were indications of less than rigorous coordination of work
activity in the switchyard. For example, when there was a half
.
trip of the 230kV switchyard on March 11, personnel in the Unit 3
- {
control room were not aware there was testing ongoing although the
work had been coordinated through the Senior Nuclear Scheduling
~
Coordinator. Also, on April 8, the electricians who performed the
battery cell replacement contacted the control room from the relay
house in the 500kV switchyard without having previously receiving
permission to enter the switchyard as discussed in AI-}300.
The CR-3/0thers Interface Matrix Manual and Administrative
Instruction AI-1300, Crystal River Coal Plants, Substation, ECC,
and Relay Departments Interface with Crystal River Unit 3,
_
specifically address the performance of surveillance testing on
the Unit 1/2 batteries which supply control power to the 230kV
-
switchyard breakers. The surveillance testing of the Unit 1/2
batteries was required because both normal offsite power circuits
to Unit 3 are supplied through the 230kV switchyard. The 500kV
]
switchyard is only used as an offsite power source when the plant
is shut down and backfeed has been aligned.
In order to ensure
compliance with TS, tne licensee applied the TS surveillance
requirements which applied to the Unit 1/2 batteries to the 500kV
switchyard batteries when utilizing the 500kV backfeed as an
offsite power source. The CR-3/0thers Interface Matrix Manual and
l
AI-1300 did not address the performance of these surveillance
l
tests. They were performed by Unit 3 electricians. As described
l
in NRC Inspection Report 50-302/93-06, paragraph 4.a, difficulties
1
had been experier.ced maintaining the specific gravity of cu cell
of the battery within tolerance since the testing had been
initiated prior to the beginning of the outage.
It was decided to
replace the cell. The work was authorized through work request 308885.
The Unit 3 electricians were inadequately trained on
operation of the system and the work request did not include
instructions for performing the necessary switching operations to
disconnect the battery from the charger. The Unit 3 electric shop
did not routinely work on, nor were they trained on the operation
of the switchyard battery systems. The day shift electrical
supervisor who reviewed the work request prior to implementation,
(
recognized that further information was necessary for performance
of the work. He interfaced with systems engineering and
'
operations to develop an acceptable method to perform the work.
j
This information was not incorporated into the work request, nor
E
passed on to the night shift electric shop supervision. The night
shift electrical supervisor did not recognize that further
information was necessary for performance of the work and the work
was assigned to the shop crew. With the battery charger connected
to the system, Unit 3 electrical maintenance personnel opened a
switch to disconnect the battery. Within seconds of disconnecting
the battery, 500kV circuit breakers tripped, deenergizing the
safety-related buses.
Since the battery charger was not designed
to supply the load independent from a battery, the charger
produced voltage outside the specified operating range when the
,
-
_ _ .___._._.________________.
_
_
-
9
4
battery was disconnected. Apparently, the abnormal voltage on the
system damaged a solid state auxiliary tripping relay, causing it
to go into the tripped condition, tripping the 500kV breakers.
This deenergized the 'B' 4160V Engineered Safeguards electrical
bus in service which resulted in an automatic start of the 'B'
Emergency Diesel Generator and a three minute interruption in the
operation of the decay heat removal system.
The failure to
establish an adequate procedure for the operation of the 500kV
,
switchyard DC control power system was a violation and is
l
Identified as VIO 50-302/93-11-01.
,
5.
Conclusions
Prior to the interruption of decay heat removal on April 8, licensee
-
personnel were aware of the problems associated with isolated charger
mode of operation at other sites; however, tb i analysis of the
information available was narrowly focused, p.ecluding the
identification of the similar vulnerability which existed at Crystal
River (paragraph 4.3).
The risk of interrupting power to the shutdown cooling systems while
performing the 500kV switchyard battery cell replacement was not
'
adequately evaluated by the licensee (paragraph 4.4).
Personnel assigned to replace the 500kV switchyard battery. were
inadequately trained on the operation of the system and were supplied
,
inadequate procedures to do the work (paragraph 4.6).
Short term corrective actions, particularly the Operations Standing
Order to prohibit work in the switchyard powering the shutdown cooling
systems, were very good (paragraph 4.2).
'
Management decisions to perform the evaluation of the human performance
aspects and to establish the Management Review Team were sufficient to
establish the basis for the identification of the significant
contributing causes of the April 8 interruption of shutdown cooling and
commonality to other switchyard problems during the maintenance outage.
i
Management oversight of the technical aspects of the root cause
,
analysis, although adequate, could have been improved to allow a more
thorough understanding of the event and to ensure comprehensive
[
corrective actions (paragraph 4.1).
!
The latest event was not a result of inadequate corrective action from
.
!
previous events at Crystal River (paragraph 4.5).
!
6.
Exit Interview
The inspection scope and findings were summarized on April 16, 1993,
with those persons indicated in paragraph 1.
The-inspectors described
the areas inspected and discussed in detail the inspection findings and
the violation listed below.
Proprietary information is not contained in
this report.
Disse eing comments were not received from the licensee.
<
c
.
.
t
.
t
-
10
!
'
'
,
>
Item Number
Description and Reference
'
50-302/93-11-01
Violation - Failure to establish an adequate
procedure for the operation of the 500kV
switchyard DC control power system (paragraph
'
4.6).
7.
Acronyms and Abbreviations
>
'
AI
- Administrative Instruction
CFR
- Code of Federal Regulations
- Compliance Procedure
-
CR
- Crystal River Unit 3
,
- Direct Current
- Energy Control Center
EN
- Event Notification
i
kV
- kilovolt
NRC
- Nuclear Regulatory Commission
l
SSOD - Shift Supervisor of Duty
i
TS
- Technical Specification
V
- volt
- Violation
1
!
!
,
i
I
!
'
!
.
'!
'!
l
.
-
. , -
,