ML20056C976
| ML20056C976 | |
| Person / Time | |
|---|---|
| Site: | South Texas  |
| Issue date: | 07/23/1993 |
| From: | Johnson W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML20056C974 | List: |
| References | |
| 50-498-93-22, 50-499-93-22, NUDOCS 9307300175 | |
| Download: ML20056C976 (10) | |
See also: IR 05000498/1993022
Text
. ..
.
__
___
_ _ _ _ _ _ _ _ _ _ _
. _ _ _ _ .
__
APPENDIX B
.U.S.
NUCLEAR REGULATORY COMMISSION
REGION IV
NRC Inspection Report:
50-498/93-22
50-499/93-22
Operating Licenses:
r,PF-80
Licensee:
Houston Lighting & Power Company
P.O. Box 1700
Houston, Texas 77251
Facility Name:
South Texas Project Electric Generating Station,
l
Units 1 and 2
l
Inspection At: Matagorda County, Texas
l
Inspection Conducted:
June 28 through July 2, 1993
Inspectors:
Mark A. Satorius, Project Engineer
Approved:
/b I
bI
7/C /93
W. D. JohnsonT Chief, Project Section A
Dafe
l
Inspection Summ3ry
Areas inspected:
A special inspection was conducted to determine the
circumstances surrounding the loss of spent fuel pool (SFP) cooling on
June 13-14, 1993.
(
Results:
!
Two examples of operators failing to follow station procedures were
identified:
An oncoming operations shift failed to conduct an adequate review
o
of plant status prior to assuming the shift (Section 3.2.1).
A reactor plant operator failed to note that the noise level in
o
the area of the SFP pumps and heat exchangers was significantly
reduced following the isolation of component cooling water to the
SFP heat exchanger (Section 3.2.2).
These two examples of failing to follow procedures resulted in the
13-hour duration of a loss of SFP cooling event,
,
9307300175 930723
P"
ADDCK 05000498
5f-
J'
G
.
..-
.
-.
.
- - . .
_-
. . -
-
.
-
4
-2-
-i
,
I
The failure of both the off-going and oncoming shifts to identify the
loss of SFP cooling during routine control board walkdowns were
considered to be significant operator performance weaknesses.
The failure to initiate a station problem report and to take adequate
corrective action following deficiencies identified during a previous
,
safety-related Class IE distribution panel power supply transfer was a
violation of the requirements of 10 CFR Part 50, Appendix B,
Criterion XVI (Section 3.2.3).
t
Summar_y of Inspectinn Findings:
Violation 499/9322-01 was opened (Sections 3.2.1 and 3.2.2).
L
Violation 499/9322-02 was opened (Section 3.2.3).
Attachments and/or Enclosures:
I
Attachment 1 - Persons Contacted and Exit Meeting
l
i
i
F
1
i
1
_
)
.-
.
..
--
-
.=.
_
. -
- _
.
-
-
.
'l
,
-3-
P
DETAILS
!
i
1 BACKGROUND
.
t
On June 13,1993, Unit 2 was in Mode 6, with all fuel off-loaded from the core
and stored in the SFP.
The SFP cooling pump and Heat Exchanger 2A was in
i
service cooling the SFP and was being supplied cooling water by component
'
cooling water Train A.
SFP temperature was 99oF, which was 41oF less than the
Final Safety Analysis Report limit of 140 F, and had bcan steady for several
weeks. Operating crews were on an outage rotation that consisted of 12-hour
shif ts, with shift changes taking place at 7 a.m. and 7 p.m.
'
2 EVENT DESCRIPTION
The inspector reviewed the operational aspects of the events surrounding the
loss of SFP cooling.
f
2.1
Electrical Power Supply Shifts
A corrective maintenance activity scheduled for the day shift on June 13,
1993, consisted of shifting the power supply for 120 volt AC class lE
Distribution Panel DP001, from its normal _ safety-related
'
Inverter / Rectifier EIV001, to the standby Regulated Transformer ERV001A.
Because of the configuration of Distribution Panel _DP001 and its two sources
of power, the transfer involved a break-before-make operation that left
Distribution Panel DP001 de-energized for a short period before power could be
i
restored.
This activity was planned following the earlier identification-by
i
licensee personnel of a hot terminal lug on Inverter EIV001 through a routine
!
f
thermography inspection.
i
Prior to the power supply shift taking place, operators reviewed
!
Procedures OPOP02-AE-0004, "120 VAC ESF Vital Distribution Power Supplies,
Revision 0, OPOPO4-EE-0002, " Loss of 120 VAC Class-lE Vital Bus," Revision 0,
.
and SE542EL8000 " Unit 2 Electrical Load List," Revision 0, in order to ensure
{
that any unexpected electrical perturbations resulting from the break-before--
l
make transfer would be identified.
In addition to reviewing these procedures,
the shift supervisor also reviewed two uncontrolled database information
t
systems available to operators on- personal computers:
(1) unit outage notes
that utilize the database in the licensee's equipment clearance order-program-
3
to permit operators to search previous equipment clearance orders to determine
'
if past problems have been identified and documented when performing
maintenance, and (2) computer database load lists that document plant
-
'
distribution panels and loads. After conducting the review, operators
determined that the only equipment that would be affected was reactor
}
containment ventilation, the fuel handling building heating and air
,
conditioning system, and Train C of the control room heating and air
l
conditioning system. Operators took actions to ensure that this equipment was
l
properly aligned to prevent either an inadvertent actuation or isolation.
i
i
i
i
,
,
,
_ _
- _ -
.
.
-
-
__
_
_
_.
l
.
.
.
,
-4-
L
At 2:15 p.m on June 13, 1993, Distribution Panel DP001 was transferred to its
l
J
standby power source by the mechanical / electrical auxiliary building reactor
'
plant operator. Distribution Panel DP001 was de-energized for approximately
-
5 seconds while the feeder breaker from Inverter EIV001 opened and the feeder
breaker from Transformer ERV001A was shut, re-energizing Distribution Panel
DP001.
The control room operators acknowledged all the annunciator alarms as
received and did not note any abnormal alarms or conditions.
!
Following successful repairs to the terminal lug on Inverter EIV001, it was
-
re-energized and Distribution Panel DP001 was transferred back to its normal
i
source of power at 4:35 p.m. on the same day. At approximately 5:15 p.m., the
mechanical / electrical auxiliary building reactor plant operator _ entered the
fuel handling building to conduct a routine 8-hour tour that consisted of
l
'
recording SFP temperature and level and observing satisfactory operation of
the operating SFP pump.
During that tour, the reactor plant operator did not
note any abnormalities.
Shift turnover was conducted between 6 p.m. and
7 p.m. on June 13, 1993.
During that turnover, all off-going control room
l
operators discussed with their reliefs the earlier power supply shift, the
.
alarms received, and actions taken to repair Inverter EIV001.
In addition,
all oncoming operators conducted a control board walkdown with the off-going
operators. The operators noted no abnormalities.
2.2 SFP Cooling Loss Identification
!
At approximately 3 a.m. on June 14, 1993, while performing a logtaking tour,
the mechanical /e"ectrical auxiliary building reactor plant operator noted that
the in-service Component Cooling Water Pump 2A discharge pressure was reading
abnormally high.
The expected pressure was 110-115 psi; however, the
discharge pressure was approximately 136 psi. The reactor plant operator
,
informed the control room of the abnormal condition.
After being informed of
the local component cooling water pump discharge pressure, operators checked
-
the control room indication of Train A component cooling water flow and
discovered that flow was approximately 5500 gallons per minute (gpm) when the
expected flow was 12,000 gpm. At this same time, control room operators
observed that the component cooling water Train A common isolation supply and
return valves, CC-M0V-0316 and CC-MOV-0052, were indicating shut. With these
,
valves shut, component cooling water flow was isolated from the SFP heat
!
excharger. The operators immediately opened Valves CC-MOV-0316 and
'
CC-MOV-0052, to restore SFP cooling and directed the mechanical / electrical
'
auxiliary building reactor plant operator to proceed to the fuel handling
building and report the SFP temperature. The reactor plant operator entered
'
!
the fuel handling building and SFP temperature was 117 F.
c
Based on a review of the logs and'a component cooling water flow trace
j
retrieved from the plant computer, the inspector determined that component
cooling water had been isolated from the SFP from June 13 at approximately
!
2:15 p.m. until June-14, 1993, at approximately 3:15 a.m., a period of
!
13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br />.
This would equate to a linear heat-up rate of about 1 1/2 F per
hour.
I
h
i
_- ,
_
_
_
.
.
_
_
_
__
.
'
4
.
-5-
3 FOLLOWUP (92701)
The inspector conductad a review of the events that lead to the loss of SFP
cooling and the licensee's immediate and longer term corrective action.
3.1
Repositioning of Valves CC-MOV-0316 and CC-M0V-0052
The inspector reviewed the events associated with the repositioning of
Valves CC-MOV-.0316 and CC-MOV-0052.
These valves are located in the component
,
cooling water system in order to isolate non-Technical Specification governed
equipment from components governed by the Technical Specifications.
Non-Technical Specification equipment supplied by the component cooling water
system included both SFP heat exchangers, the letdown and seal water heat'
exchangers, the boron thermal regenerative system chiller, and the boron
recycle evaporator package. Valves CC-MOV-0316 and CC-MOV-0052 received an
automatic closure signal on a low level in the component cooling water surge
tank.
Following the automatic closure, the valves received an open permissive
signal that permits operators to reopen the valves after the component cooling
water surge tank low level condition clears.
The basis for this signal. was to
isolate these nonessential loads from the essential Technical Specification
j
loads in the event of a low component cooling water surge tank level that
would result from a break in the component cooling water piping. - If.the
postulated break occurred in the nonessential portion of the component cooling
water system, the closure signal would prevent component cooling water
inventory loss and the potential of loss of the component cooling water train.
l
When the operators initially de-energized Distribution Panel DP001 from
!
Inverter EIV001, the component cooling water surge tank low level alarm was
received and the circuit logic was completed to shut Valves CC-MOV-0316 and
CC-MOV-0052; however, the valves did not shut because-ie-energizing
Distribution Panel DP001 removed control power to-operate these valves. When
Distribution Panel DP001 was re-energized from Transformer ERV001A, a relay
race occurred such that control power to operate Valves CC-MOV-0316 and
CC-MOV-0052 was restored and the valve's control. power circuit hold-in contact
'
closed and remained closed after the component cooling water surge tank low
level circuit cleared. This relay race resulted in Valves CC-MOV-0316 and
.
CC-MOV-0052 shutting and isolating flow to the SFP heat exchangers. The
l
inspector determined that numerous ' annunciators alerting operators to the
condition would have illu.ninated during the 5-second period that Distribution.
.
Panel DP001 was de-energized; however, following re-energizing the panel,
these alarms cleared.
After the repairs were completed on Inverter EIV001, Distribution Panel DP001
was transferred back to the primary power source at 4:35 p.m. on June 13,
1993.
During this transfer, which was also a break-before-make evolution,
similar alarms to the those received on the earlier transfer at 2:15 p.m.
l
,
annunciated in the control' room. During this transfer, since
' Valves CC-MOV-0316 and CC-MOV-0052 were already closed and required operator.
-
action to re-open, the component cooling water surge tank low level did not -
,
cause any adverse plant conditions.
'
,
I
l
~
.
-
- ,
,
w
._.
.
.
.
.
_
-
_
_
_ . _
,
'
.
-
.
.
i
-6-
,
3.2 Operator Performance issues
The inspector reviewed operator logs, security logs that recorded entry and
egress into Unit 2 areas, and operator statements that the licensee gathered
following the event.
In addition, the inspector interviewed selected members
of the operations staff who were present during the period that the SFP was
isolated on June 13-14, 1993.
3.2.1
Control Room Performance
.
l
The inspector determined that there were three separate indications available
to control room operators to indicate that component cooling water had been
isolated from the SFP.
These included the valve position indications for
Valves CC-M0V-0316 and CC-MOV-0052, which are normally-open and were shut
following re-energizing Distribution Panel DP001; Component Cooling Water 2A
heat exchanger flow, which normally indicates approximately 12,000 gpm and was
approximately 5500 gpm following the component cooling water isolation; and
,
Component Cooling Water 2A heat exchanger outlet pressure, which normally wa
in the range of 100-105 psi and was indicating approximately 115 psi during
the event.
None of these parameters were required to be logged by control
room operators.
'
,
During the period that component cooling water was isolated to the SFP, a
shift turnover occurred at 7 p.m. on June 13, 1993. During that turnover, the
multiple indications of the loss of SFP cooling went unnoticed by both the
off-going and oncoming shift.
Technical Specification 6.8.1.a states that
written procedures shall be established, implemented, and maintained covering
the activities referenced in Appendix A of RG 1.33, Revision 2, February 1978.
Paragraph 1.9 of RG 1.33, Appendix A, recommends that administrative
procedures addressing shift and relief turnover should be implemented.
,
Paragraph 3.2.5 of Procedure OPOP01-ZQ-0022, " Plant Operations Shift
Routines," Revision 2, states, in part, that each individual relieving a
watchstation SHALL be fully aware of plant status prior to assuming the watch.
j
The failure of the oncoming shift to be fully aware of plant status, in that
component cooling water was isolated from the SFP, was the first example of a
,
violation of Technical Specification 6.8.1.a (50-499/9322-01).
During interviews conducted with the control room operators, the inspector
determined that a perception of a relatively low safety risk existed in the
control room. This operator perception was based on the plant condition of
being 'defueled, with no decay heat removal requirements to remove residual
j
heat from the core. This perception by the operators and the failure of both
the off-going and oncoming shifts to identify the loss of SFP cooling during
routine control board walkdowns were considered to be significant operator
performance weaknesses.
3.2.2
Reactor Plant Operator Performance
The inspector conducted a tour of the fuel handling building, tracing the path
that the mechanical / electrical auxiliary building reactor plant operators
_
. _ .
..
.
_.
. . .
. .
.
..
.
.
.
.
-7-
.
follow during their normal tours.
In addition to the normal fuel handling
building logs that the reactor plant cperator was required to take each shift,
plant operations had invoked an additional temporary logsheet that required
recording local observation of the running SFP pump and SFP level- and
temperature.
Although the normal fuel handling building logs required
readings to be taken each shift (every 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />), this temporary logsheet
required reading to be recorded every 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />, which necessitated a second
fuel handling building entry per shift by one reactor plant operator during
each 24-hour period.
In the area of the SFP pumps and heat exchangers, hearing protection was
required, due to the high noise level from component cooling water flow.
Based on this observation, the inspector concluded =tnat the noise levels would
be expected to be significantly lower if the component cooling water flow were
isolated, as was the case after Valves CC-M0V-0316 and CC-M0V-0052 were
inadvertently closed.
The mechanical / electrical auxiliary building reactor plant operator entered
the fuel handling building at approximately 2 p.m. on June 13, 1993, to
conduct his routine tour shift of the building and to record the second 8-hour
temporary logsheet readings on the SFP parameters.
During this tour,
component cooling water supply to the SFP heat exchangers was not isolatea.
SFP temperature and level were normal, with temperature recorded at 99 F and
level at 66-feet,1-inch.
SFP Pump 2A was in service, with no pump
deficiencies observed.
The same reactor plant operator entered the fuel handling building again at-
approximately 5:15 p.m., to record the third 8-hour temporary logsheet
readings on the SFP parameters. During this tour, component cooling water
supply to the SFP heat exchangers had been isolated for approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />.
SFP temperature and level were recorded at 99oF and level at 66-feet,1-inch.
SFP Pump 2A was in service and with no pump deficiencies noted.
The inspector questioned the licensee concerning the 5:15 p.m. recorded SFP
temperature.
Assuming a linear heat-up rate, the expected SFP temperature
would be approximately 103 F, and not 99 F as recorded by the reactor plant
.
operator.
The licensee responded that the heatup rate would not be expected
to be linear.
Following component cooling water isolation to the SFP heat
exchanger, the shell side of the SFP heat exchanger would contain a large
volume of component cooling water. This volume of water would require a
period of time to reach an equilibrium temperature prior to-the SFP
temperature increasing and that a reading of 99 F taken at 5:15 p.m. would not
be considered unrealistic.
Although based on a deterministic evaluation, the
inspector concluded that this argument was not unreasonable.
During the interview with the reactor plant o'perator that recorded the
2:15 p.m. and 5:15'p.m. fuel handling building parameters, the inspector
questioned the individual concerning the level of component cooling water flow
,
noise that was present in the area of the SFP pumps and heat exchangers. The
'
reactor plant operator acknowledged that there was a high level of flow noise
,
?
I
, - - .
.
-
..
,-
-
.
.
.
.
.
-
--
- .
..
.
-
.
.
\\
-8-
.
in that area and he acknowledged that the component cooling water flow noise
would be expected to be significantly reduced after component cooling water
isolation. However, the reactor plant operator stated that he did not recall.
whether there was a reduction of flow noise during the 5:15 p.m. fuel handling
building tour. Technical Specification 6.8.1.a states that written procedures
shall be established, implemented, and maintained covering the activities
f
referenced in Appendix A of RG 1.33, Revision 2, February 1978.
Paragraph 1 9
of RG 1.33, Appendix A, recommends that administrative procedures addressing
shift and relief turnover should be implemented.
Paragraph 5.3.1.11 of
i
Procedure OPOP01-ZQ-0022, " Plant Operations Shift Routines," Revision 2,
i
states, in part, that while conducting local operator rounds, individuals
,
should inspect all areas and equipment, ensuring all noise and vibration
levels are normal.
The failure of the reactor plant operator to note that the
'
noise level in the area of the SFP pumps and heat exchar.gers was significantly
.
reduced following the isolation of component cooling water the SFP was the
!
second example of a violation of Technical Specification 6.8.1.a
~
(499/9322-01).
3.2.3
Operations Staff Support
l
The inspector reviewed the systems in place to provide information to
operators when equipment was required to be de-energized for corrective or
,
'
preventive maintenance. As mentioned in Paragraph 2.1, the control room
operators, prior to shifting Distribution Panel DP001 from its normal power
7
supply to its alternate power supply, reviewed Procedures OPOP02-AE-0004,
OPOP04-EE-0002, SE542EL8000 and the uncontrolled unit outage notes and
!
computer database load lists. These systems appeared to be unwieldy and
difficult for the operator to utilize in order to extract useful information
l
to assist them in operating plant equioment.
During an interview with a shift
supervisor, the inspector was informed that the formal, controlled sources of
'
information were less useful than the uncontrolled load list and outage notes.
i
i
During interviews with other control room operators, the inspector identified
a similar event that occurred during the Unit 2 second refueling outage.
In
f
order to perform maintenance on safety-related Class IE Distribution
Panel DP002, the licensee de-energized the panel in a manner similar to the
i
Distribution Panel DP001 event.
When Distribution Panel. DP002 was
!
re-energized, a relay race occurred which resulted in several valves changing
!
position unexpectedly.
This event was not documented with a station problem
l
report.
The inspector determined that the condition concerning the unexpected valve
!
re-positioning during the transfer of Distribution Panel DP002 constituted a
l'
condition adverse to quality and that the licensee failed to take prompt and
effective actions to ensure that the cause of this identified deficiency was
j
identified, corrected, and steps taken to preclude recurrence. This was a
'
violation of the requirements of 10 CFR Part 50, Appendix B,
Criterion XVI (499/9322-02).
I
'
..
.
.
.
.
. . - . .
.
-
.
_ _
_
-
.
.
_
.
.
.
.
.
-9-
,
'4
LICENSEE CORRECTIVE ACTION
i
As a result of this event, the licensee had taken several corrective actions.
Station Problem Report 932030 was generated to investigate the hardware
deficiencies.
The scope of this station problem report was increased to
address the nonhardware issues, such as procedural weaknesses and personnel
performance inadequacies that were identified.
Although licensee personnel had not completed their review of the event at the
- I
end of the inspection, they had identified a number of causes and had
developed a corrective action plan to address these deficiencies.
l
The licensee has revised Procedure OPOP02-AE-0004, "120 VAC ESF Vital
,
Distribution Power Supplies," to alert station personnel to the possibility of
inadvertent valve repositioning due to circuit relay races or other electronic
!
perturbations when shifting power supplies using a break-before-make transfer.
The Operations Manager has conducted crew briefings with all shifts (licensed
and nonlicensed operators), and operations personnel involved in the event
-
have received discipline in accordance with the Houston Lighting & Power
Company Constructive Discipline Program.
The licensee's training department will implement improved simulator scenarios
for use in operator requalification training. These will include
t
demonstrations of operator awareness of mispositioned/ misaligned components
during shift turnover and normal control board walkdowns.
In addition, the
,
training department will revise initial licensed operator training and
operator requalification training to increase operator awareness that the loss
of SFP cooling when the core is defueled to the SFP constitutes a loss of
These actions were scheduled for completion by September
15, 1993.
.
'
The licensee's design engineering department has initiated a modification to
add an annunciated alarm in the control room to alert operators when component
l
cooling water flow to the SFP cooling heat exchangers or SFP cooling flow to
the SFP has degraded.
In addition, the design engineering department is
evaluating a reduction of the' current.SFP temperature control room annunciated
.
alarm from 154af to a lower temperature that would provide operators a more
timely indication of SFP temperature abnormalities.
These actions were.
!
scheduled for completion by-October- 31,1993. A longer term corrective action
i
assigned to the design engineering department was validation and
implementation of formal control of the computer databases and other reference
material currently utilized by operations personnel to identify electrical.
loads. This task was scheduled for completion by March 31, 1994.
!
7
7
.
i
+
, -
-
--
. - - .
.
.--
. -
--.
.
.
3
..
.
.
ATTACHMENT 1
.
1 PERSONS CONTACTED
1.1 Licensee Personnel
H. Bergendahl, Manager, Technical Services
J. Calloway, Consultant, Participant Services
,
M. Chakravorty, Executive Director, Nuclear Safety Review Board
K. Christian, Manager, Plant Operations
W. Cottle, Group Vice President
J. Groth, Vice President, Nuclear Generation
S. Head, Deputy General Manager, Nuclear Licensing
.
T. Jordan, General Manager, Nuclear Engineering
D. Leazar, Manager, Plant Engineering
i
L. Ledgerwood, Consulting Engineering Specialist
M. Ludwig, Manager, Nuclear Training
,
G. Parkey, Plant Manager
,
P. Parrish, Senior Specialist, Licensing
K. Poling, Assistant to Manager, Nuclear Training
D. Ruthven, Shift Supervisor
S. Walker, Manager Public Affairs
!
G. Weldon, Manager, Operations Training
!
M. Wigginton, Associate Engineer, Nuclear Training
1.2
NRC Personnel
R. Evans, Resident Inspector
J. Keeton, Resident Inspector
l
M. Satorius, Project Engineer
i
The personnal listed above attended the exit meeting.
In addition to the
i
personnel listed above, the inspectors contacted other personnel during this
inspection period.
2 EXIT MEETING
'
An exit meeting was conducted on July 2, 1993. During this meeting, the
inspector reviewed the scope and findings of the report. The licensee did.not
identify as proprietary any information provided to, or reviewed by, the
inspector.
. .
.