ML17331A953
| ML17331A953 | |
| Person / Time | |
|---|---|
| Site: | Cook  |
| Issue date: | 08/17/1993 |
| From: | Jorgensen B NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML17331A951 | List: |
| References | |
| 50-315-93-16, 50-316-93-16, NUDOCS 9308250296 | |
| Download: ML17331A953 (24) | |
See also: IR 05000315/1993016
Text
U. S.
NUCLEAR REGULATORY COHHISSION
REGION I I I
Report
Nos.
50-315/93016(ORP);
50-316/93016(ORP)
Docket Nos. 50-315;
50-316
Licensee:
Indiana Hichigan Power
Company
1 Riverside
Plaza
Columbus,
OH
43216
License
Nos.
OPR-58;
Facility Name:
Donald
C.
Cook Nuclear
Power Plant, Units
1 and
2
Inspection At:
Donald
C.
Cook Site,
Bridgman,
HI
Inspection
Conducted:
June
16,
1993 through July 27,
1993
Inspectors:
J.
A.
Isom
D. J. Hartland
- T. Tella
Approved By:
. L.
g
sen,
Chief
Reactor
rojects Section
2A
ate
(7
Ins ection
Summar
- Inspection
from June
16,
1993,
through July 27,
1993.
(Report Nos.
50-315/93016(DRP);
50-316/93016(ORP) )
Areas
Ins ected:
Routine,
unannounced
inspection
by the resident
and region-
based
inspectors of:
plant operations;
maintenance
and surveillance;
engineering
and technical
support;
actions
on previously identified items;
reportable
events;
and, safety assessment/quality
verification.
Results:
Two Severity Level
IV violations were identified.
0
11,
p f
i
ti
g
d.
Il
,
p
did not address
compliance to technical specifications for unidentified
leakage during
a plant evolution until prompted
by the inspector.
In
addition, the licensee
declared
the time of inoperability of the Unit 2
"M"
'entrifugal charging
pump to be the time when
pump performance
was observed
to
be deteriorating,
only after the inspector
discussed
the issue with plant
management.
Haintenance:
Overall, the quality of work observed
by the inspector during
this period was good.
However, the inspector
noted that the planning for the
repair work on valve 1-IRV-300 was inadequate,
which resulted
in the plant
being in an undesirable
line-up for
an extended
period of time.
En ineerin
and Technical
Su
ort:
The inspection disclosed
a weakness
in the
licensee's
handling of a "minor" HDAFW. pump packing leak.
This resulted
in
the
pump outer bearing
becoming seriously degraded,
such that
pump capability
to perform its desigb function was questionable.
. 9308250296
9308f7
(5
ADOCK 0500031'5
"8
PDRg
Safet
Assessment
ualit
Verification:
The inspection disclosed
a weakness
in the licensee's
failure to perform
a formal safety evaluation prior to de-
energizing
an
EDG room ventilation damper in the
open position.
This
ultimately contributed to the inoperability of the
EDG.
In addition, the
licensee's
investigation into the event failed to identify this as
a root
cause.
DETAILS
Persons
Contacted
A.
- K.
L.
- J
B.
T.
p.
D.
.L.
T.
- S
p.
- J
L.
G.
D.
- M.
A. Blind, Plant Manager
R. Baker, Assistant Plant Manager-Production
S. Gibson, Assistant
Plant Manager-Projects
E. Rutkowski, Assistant
Plant Manager-Technical
Support
A. Svensson,
Executive Staff Assistant
P. Beilman, Maintenance
Superintendent
F. Carteaux,
Training Superintendent
L. Noble, Radiation Protection Superintendent
.J. Matthias, Administrative Superintendent
K. Postlewait,
Design
Changes
Superintendent
A. Richardson,
Operations
Superintendent
G. Schoepf,
Project Engineering
Superintendent
S. Wiebe, Safety
5 Assessment
Superintendent
H. Vanginhoven, Site Design Superintendent
A. Weber,
Plant Engineering
Superintendent
C..Loope,Chemistry
Superintendent
L. Horvath, guality Assurance
Supervisor
The inspector also contacted
a number of other licensee
and contract
employees
and informally interviewed operations,
maintenance,
and
technical
personnel.
- Denotes
some of the personnel
attending the Management
Interview on
August 4,
1993.
Plant
0 erations
71707
71710
42700
The inspector
observed routine facility operating activities
as
conducted
in the plant and from the main control
rooms.
The inspector
monitored the performance of licensed
Reactor Operators
and Senior
Reactor Operators,
and of Auxiliary
Equipment Operators
including procedure
use
and adherence,
records
and
logs,
communications,
and the degree of professionalism of control
room
activities.
The inspector
reviewed the licensee's
evaluation of corrective action
and response
to off-normal conditions.
This included compliance with
any reporting requirements.
The inspector
noted the following with regard to the operation of Units
1 and
2 during this reporting period:
a.
Unit
1 Status:
The licensee
operated
the unit at full power throughout the
inspection period, with no significant operational
problems noted.
-3
b.
Unit 2 Status:
The licensee
operated
the unit at
70 percent
power during the
period until July 10,
1993,
when power was raised to 91 percent to
support
system grid demand.
The licensee
reduced
power back to 70
percent
on July 17,
and intended to operate
the unit at that power
level for the remainder of the cycle in order to separate
the two
units'cheduled
1994 refueling outages.
The
NRC granted
the licensee
a Notice of Enforcement Discretion
(NOED) on July 9,
1993, to extend the
72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action statement for
Technical Specifications
(TS) 3. 1.2.4
and 3.5.2
due to the
inoperability of the Unit 2
"W" centrifugal charging
pump
(CCP).
The
NOED allowed for an additional
60 hour6.944444e-4 days <br />0.0167 hours <br />9.920635e-5 weeks <br />2.283e-5 months <br /> s to repair the
pump
without the requirement to initiate
a plant shutdown.
At about
10:37 p.m.,
on July 6, the licensee
switched from the
"W"
to the "E" CCP after experiencing difficulty in maintaining
pressurizer
level with the
"W" CCP in service.
The licensee
did
not declare
the
pump inoperable at that time because
the
TS
requirement for pump discharge
pressure
was satisfied at the time
the
pump was shut
down.
The licensee
declared
the
pump inoperable
at 10:32 a.m. the following day after it exhibited high vibrations
during
a troubleshooting
run.
On July 8, after discussions
with
the inspector,
the licensee
moved back the time of inoperability
to 10:37 p.m.
on July 6.
--The licensee
replaced
the -internal rotating assembly
and declared
the
pump operable after
a successful
surveillance
run at
ll:45 p.m.
on July 10.
The inspector
observed
portions of the
pump repair
and surveillance
run and verified licensee
compliance
'to the compensatory
measures
documented
in the
NOED.
The
inspector did not identify any deficiencies.
The licensee
had not
yet
determined
the root cause of the
pump failure, but had
shipped
the internal
assembly to the vendor for disassembly
and
inspection.
The inspector will review the licensee's
determination of the root cause of the
pump failure and the
corrective action taken,
as appropriate.
No violations, deviations,
unresolved,
or inspector followup items were
identified.
Haintenance
Surveillance
62703
61726
42700
The inspector
reviewed maintenance
activities
as detailed
below.
The
focus of the inspection
was to assure
the maintenance activities were
conducted
in accordance
with approved
procedures,
regulatory guides
and
industry codes or standards,
and in conformance with Technical
Specifications.
The following items were considered
during this review:
the Limiting Conditions for Operation
were met while components
or
systems
were removed from service;
approvals
were obtained prior to
initiating the work; activities were accomplished
using approved
procedures;
and post maintenance
testing
was performed
as applicable.
The following activities were inspected:
a ~
Solid State Protection
S stem
SSPS
Test Circuit Card
Re lacement
b.
The inspector
observed
licensee
actions
taken in response
to
problems
encountered
during performance of SSPS surveillance
testing.
The inspector determined that the licensee
response
was
satisfactory
and that the
ILC personnel
involved appeared
to be
knowledgeable.
On June
16,
1993, while checking that the test circuitry was
functional per step 7.3. 12 of "Reactor Trip SSPS
Logic and Reactor
Trip Breaker Train "A" Surveillance
Test (Monthly)," lIHP4030
STP.410,
the licensee
did not receive the required
response
from
the testing
lamps.
At that time, the licensee
was in a
2 hour2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />
Limiting Condition for Operation
(LCO) per
TS No. 3.3. 1. 1 due to
the bypassing of the Train "A" reactor trip breaker/automatic trip
logic during the surveillance testing.
Upon review and
concurrence
from supervision,
the Instrument
and Control
(I&C)
technicians
determined that
a test circuit card
had probably
failed; however,
the test circuit did not affect operability of
the
SSPS trip logic.
Therefore,
the licensee carefully backed out
of the surveillance
and exited the
LCO.
The following day,
I&C replaced
the card per Action Request
(AR)
A0045723,
which provided generic instructions to troubleshoot
and
repair the problem while performing the surveillance.
The
inspector
observed that
I&C successfully
completed
the
surveillance after replacing the card.
Re air of Valves 1-IRV-300 and 1- RV-303
The inspector
observed activities associated
with the repair of
chemical
and volume control
system
(CVCS) letdown valves
1-IRV-300
and
I-HARV-303.
The inspector
observed that, overall, the
evolution was well-executed
and that the quality of the
maintenance
was good.
However, the licensee
did not address
TS
compliance to unidentified leakage during
a preliminary evolution
until prompted
by the inspector.
In addition, the licensee
was
delayed
in completing the repairs
on
1-IRV-300, which resulted
in the plant being in a undesirable
line-up for an extended
period of time.
On June
16,
1993, the licensee
removed
normal
letdown from service
and established
excess
letdown in an attempt to quantify leakage
past letdown isolation valves
HARV 160,
161,
& 162
and to observe
5
and monitor conditions while letdown was isolated.
The licensee
performed the evolution in preparation for the scheduled
repair of
1-IRV-300 and I-QRV-303.
The licensee
performed
system
(RCS) leakrate
calculation during the evolution
and determined that about
1.3
GPH
was leaking past the orifice isolation valves.
At that point, the
inspector queried the operations
crew as to whether this leakage
was considered
to be identified or unidentified.
The licensee
was
required to enter
TS
LCO No. 3.4.6.2
in
excess of
1
GPH.
Only after being prompted
by the inspector did
the licensee
address
the issue.
The operations
crew determined
after
some debate that since the leakage
source
had not been
specifically identified, nor was it measured
to
a closed
system,
that they were in the
TS
LCO.
The licensee
exited the
LCO after
returning normal letdown to service later that day.
On July 7,
1993, the licensee initiated repairs
on the two letdown
valves.
In order to isolate the valves,
the licensee
again
was
required to isolate letdown
and establish
excess
letdown.
The
licensee
directed the 1.3
GPH leakage
from the orifice isolation
valves to a floor drain.
The licensee
valved in
a leak-off bottle
periodically to monitor the leak rate.
The licensee
also purged
the piping connected
to 1-QRV-303 with nitrogen to remove
any
and waste
gas from the piping,
and then pressurized
the
piping downstream of an isolation check valve to prevent
any
potential
in-leakage
from the waste
gas
system.
The inspector
attended
briefings
and observed
portions of the
valve isolation evolution and repair activities.
The inspector
observed that the evolution was well-planned
and coordinated.
The
quality of the maintenance
work was also good.
However,
the
licensee
experienced
a delay in completing the repairs to 1-IRV-
300 because
improperly-sized
were staged for the job.
As
a result,
the licensee
maintained
excess
letdown in service for 24
more hours than expected
in order to obtain the gaskets
from the
manufacturer.
The inspector
noted that being
on excess
letdown was not
a
desirable
condition because,
in the event of a safety injection
(SI) actuation,
operator action would have
been required to
reestablish
CCP recirc flow from the volume control tank
(VCT) to
the suction of the
CCP.
The realignment
would have
been necessary
to prevent the
VCT from overpressurizing,
which would have lifted
a safety valve and diverted
some SI flow from the core to the
hold-up tanks.
This scenario
would have
been likely in the event
of a small-break loss-of-coolant-accident
(LOCA), as the recirc
valves automatically would have
gone closed
on
a SI actuation
but
reopened until
RCS pressure
was reduced to 2000 psig.
In order
for the valve realignment to take place,
an operator
would have
had to have
been dispatched
by the control
room, dressed
out in
full anti-c's,
and entered
the
VCT hallway to manipulate the
6
valves.
This process
would have easily taken several
minutes to
complete.
During follow-up discussions,
the inspector discovered that
maintenance
personnel
had identified improperly-sized
as
a
potential root cause for the problem with 1-IRV-300, but that the
that were staged for the job were not checked for correct
sizing until shortly before the work started.
The inspector also
reviewed the job order (¹ 0268)
and noted that it was generic in
nature
and did not require that the gaskets
be inspected.
The
inspector
concluded that planning for this high profile job was
inadequate
due to
a lack of coordination
between
the planner
and
the maintenance
supervisor,
which resulted
in the plant being in
the undesirable
line-up for an extended
period of time.
The inspector also observed
the following activities
and did not
note
any deficiencies:
Surveillances
"Turbine Driven Auxiliary Feed
Pump Trip and Throttle Valve
Operability Test,"
lOHP4030.STP.017TV
Stop Valve Dump Valve Surveillance Test,"
10HP4030.STP.018
Stop Valve Partial
Closure Surveillance Test,"
10HP4030.STP.019P
Maintenance
JO ¹ C14939,
Repair of Unit
1
CD
EDG 1-LLA-115, Hi Lube Oil Level
Alarm
JO ¹ C16640, As-Left Static Testing of 1-NMO-753
JO ¹ C16924,
Replace
2-FRV-240 Master Controller
No violations, deviations,
unresolved,
or inspector followup items were
identified.
En ineerin
and Technical
Su
ort
37828
The inspector monitored engineering
and technical
support activities at
the site and,
on occasion,
as provided to the site from the corporate
office.
The purpose of this monitoring was to assess
the adequacy of
these functions in contributing properly to other functions
such
as
operations,
maintenance,
testing, training, fire protection,
and
configuration management.
7
Unit
2 East Motor-Driven Auxiliar
Pum
Unit 2 East Motor-Driven Auxiliar
HDAFW
Pum
Bearin
Dama e:
2)
3)
The inspector
concluded his investigation into the failure
of the Unit 2
HDAFW pump bearing
(50-315/93011-03(DRP);
50-316/93011-03(DRP))
through review of various licensee
documents
and interviews with system
and maintenance
engineers.
The inspector
concluded that Unit 2
HDAFW pump
outboard thrust bearings
had sustained
severe
damage to the
point where
pump operability was questionable
when the water
in the oil was discovered
on April 20,
1993.
Additionally,
the inspector
concluded that under the best of
circumstances,
the Unit 2 East
HDAFW pump had
been in this
condition from the
end of its last run, which concluded at
10:54 a.m.,
on March 5,
1993.
Therefore,
the Unit 2
HDAFW
pump was potentially inoperable
from about
March 5,
1993, to
April 20,
1993,
a period of 45 days.
Water Found in Unit 2
um
outer bearin
oil:
On April 20,
1993, while performing
a routine oil change
on
the Unit 2 East
HDAFW pump, the mechanics
discovered
water
present
in the oil from the outer bearing
housing.
The
licensee,
as part of the plant's preventive maintenance
program,
performs
an oil change
on the
HDAFW pump every 48
-weeks.
Because of the water found in the bearing oil, the
licensee
decided to disassemble
and inspect the outboard
bearing
assembly.
The licensee
found the brass retaining
rings were in several
pieces
and
some of the ball bearings
were badly deformed.
Subsequent
review of the Unit 2 East
HDAFW pump bearings
by the bearing manufacturer
confirmed
that the cage
(brass retaining ring) had broken into pieces,
the inner ring turned
and
smeared
to the shaft,
and
some of
the ball bearings
were distorted.
Cause of the Bearin
Failure:
The licensee
determined that the cause of the bearing
failure on the Unit 2 East
HDAFW pump was water intrusion
into the bearing
housing from an excessive
outboard
packing
leak.
The licensee
found about
20 percent
water in the
lubricating oil.
The presence
of water in the oil reduced
the oil viscosity and accelerated
bearing
cage
wear
and
fracture.
The
HDAFW pump outer bearings
are angular contact ball
bearings.
There are two bearing units which are mounted
onto the shaft in the "back-to-back" configuration.
The
bearing unit consists of the inner race,
the ball bearings
(which are
housed
in the cage
assembly
sometimes
referred
8
to as the brass retaining ring),
and the outer race.
The
cage retains the ball bearings
in their proper positions in
the annular
area
formed by the inner and outer races.
The
inner and outer races
are designed
not to rotate with the
shaft.
The inspector
examined
the
damaged
bearings
and
made the
following observations:
the surface of the inner thrust bearing inner race
was
grooved,
indicating the shaft
had turned against
the
inner race.
the surface of the outer thrust bearing inner race
appeared
to be smeared,
indicating that the inner race
had
been slipping against
the shaft.
The maintenance
and system engineers
informed the inspector
that the shaft surface
was found to be blue.
The licensee
concluded that the color of the shaft in the bearing
area
was from intense
heat
due to the friction from the condition
of the inner races for the inner and outer thrust bearings.
After inspecting the
damaged
bearings,
reviewing various
documents,
reports,
and interviewing numerous
licensee
personnel,
the inspector
agreed with the licensee's
root
cause
analysis that the cause of the severe
damage to the
Unit .2 East
HDAFW pump thrust bearing discovered
on April
20,
1993,
was water intrusion into the oil.
4)
Unit 2 East
HDAFW Pum
0 erabilit
Determination:
After examinin
the condition of the outboard
um
bearin
s
the ins ector concluded that with the outboard thrust
bearin
s in the condition found
on
A ril 20
1993
the Unit
2 East
HDAFW
um
was at that time
robabl
not ca able of
erformin
its full safet
functions.
In addition
the
ins ector concluded that the
um
had
been in that condition
for a certain
eriod before
A ril 20
1993.
To determine
at about what time period the Unit 2 East
HDAFW
pump bearing
became
damaged,
the inspector relied
on the
following facts:
the outboard
pump packing
had
been leaking since
October 9,
1991,
as
documented
on Action Request
A0005591.
On August 9,
1992, the Unit 2 East
HDAFW pump outboard
bearing temperature
was
105 degrees
Fahrenheit
9
the previous routine oil change
on the Unit 2 East
HDAFW pump was performed
on October 7,
1992.
The licensee's
recent interviews with the mechanics
who performed the oil change
on October 7,
1992,
indicated that they
saw no abnormalities with the
drained oil.
Date:
The Unit 2 East
HDAFW run history from October 7,
1992, to April 20,
1993:
Run Hours:
Oct.
20,
1992
Nov.
5,
1992
Nov.
10,
1992
Nov.
11,
1992-
Nov. 30,
1992
Dec.
6,
1992-
Dec.
8,
1992
Dec.
9,
1992-
Dec.
10,
1992
Dec.
11,
1992-
Dec.
18,
1992
Har.
5,
1993
Total Hours:
2.2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />
9.9
hours'3.7hours
451.8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />
48.9 hours1.041667e-4 days <br />0.0025 hours <br />1.488095e-5 weeks <br />3.4245e-6 months <br />
23.4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />
161.1
hours
2.1 hours1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />
723. 1 hours1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />
Based
on the
above facts,
the inspector
made the following
observations:
The outboard bearings
were in a satisfactory condition
to support continued
and prolonged
pump run on
August 9,
1992,
as evidenced
by the bearing
temperature
taken
on this date.
The packing leak and the water intrusion into the oil
reservoir worsened
sometime after the October
1992 oil
change.
At the very latest,
the
damage to the bearings
existed
by the time the
pump last operated
(for about
7
minutes
from 10:47 a.m. to 10:54 a.m.
on Harch 5,
1993) before
damage discovery
on April 20,
1993.
Based
on the observations
made
above,
the inspector
concluded that bearing
damage
from presence
of water in the
oil must have occurred
sometime after the October 7,
1992,
oil change
and the most recent
pump run on Harch 5,
1993.
The damage to the East
HDAFW pump bearings
must
have existed
by Harch 5,
1993,
because
the type of bearing
damage
10
observed
could only be sustained
during
pump operation.
With the
pump at rest,
the heating of the
pump shaft
as
a
result of the frictional forces
caused
by the condition of
the inner
and outboard
races
could not have occurred.
Such
heating of the shaft from bearing failure could only occur
when two moving elements
touched
one another with little or
no lubrication.
The inspector
concluded that,
as of April 20,
1993, the
Unit 2 East
HDAFW pump was in
a condition such that it could
not be relied upon to mitigate the consequences
of accidents
for which it was designed.
Specifically, the ability of the
pump to sustain
prolonged periods of operation,
as designed,
was very questionable.
Although the Unit 2 East
HDAFW pump was probably inoperable
for a period greater
than the time period allowed by the TS,
this condition was not absolutely certain,
and
no Notice of
Violation is in order to address
pump operability.
Further,
the inspector
concluded that the licensee
did not know
before April 20,
1993, that the
pump was potentially
nor did they have information which would have
led them to question its operability.
Corrective Action For an
A ril 1991
Event:
The inspector
reviewed the licensee's
corrective action for
a similar bearing failure which had occurred
on April 23,
1991.
The cause of the lubrication breakdown in 1991 could
not be firmly established.
Therefore,
the root causes
for
the lubrication breakdown in 1991
and
1993 could
be
dissimilar.
Nonetheless,
the inspector
concluded that the
~ licensee's
corrective action taken in 1991
was weak,
because
the recommended
preventive action from the
1991
investigation resulted
in no new information being taken to
monitor for possible
bearing degradation.
The previously similar condition in 1991
was documented
in
Problem Report 91-0532.
Although it was initially assigned
to the maintenance
department for investigation,
PR 91-0532
was subsequently
transferred
to the corporate
problem
assessment
group for resolution.
On April 23,
1991, while the mechanics
were investigating
a
concern raised
by the operators that the slinger ring for
the Unit 2 East
NDAFW pump appeared
to be not throwing oil
to the top of the bearing
assembly,
they found that the
thrust bearing
brass retainer ring was broken into two
pieces
and that there were wear particles present
in the
oil.
The licensee's
analysis of the discolored oil detected
brass material.
Also, the bearing
and
pump manufacturer
concurred that the most likely cause of the bearing failure
11
was due to lubrication breakdown.
At the time of the
bearing failure, it was determined that these
bearings
had,
only about
8 to
10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br /> of operation
from the pump's
monthly surveillance tests.
The licensee
could not identify the cause for the
lubrication breakdown.
They reviewed
such potential
causes
as the use of proper oil, proper
balance
drum setting
(which
would minimize the thrust
on the bearings),
and the proper
oil level.
None could
be determined
as the root cause for
this accelerated
bearing
wear
and failure.
As a preventive action,
the licensee stipulated
performance
of a bearing temperature
test whenever maintenance
of the
pump rotor or bearings is performed,
This requirement
was
incorporated
into the auxiliary feed pumps'aintenance
procedure with the test to be performed per the requirements
of 4030 STP.17.
The inspector determined that the licensee's
preventive
action
was weak because
acquisition of the bearing
temperatures
was contingent
upon the
use of the auxiliary
pump maintenance
procedure,
"Motor Driven
Auxiliary Feed
Pump Maintenance,"
- 12HHP5021.056.001,
Revision 5, March 12,
1993.
This procedure
may not be used
between refueling outages.
This procedure
provides
mechanics with the instructions
and documentation for
disassembly,
inspection or repair,
and assembly of the motor
driven auxiliary feedwater
pump.
During the period from
August of 1992 to April of 1993, the type of corrective
maintenance
activities performed
on the
pumps during reactor
power operations
did not utilize this maintenance
procedure.
As noted
above,
during the investigation into the April 1991
bearing failure, the licensee
determined that the brass
retaining ring on the Unit 2 East
MDAFW pump had failed
after 8 to
10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br /> of operation.
However, at the time of
the bearing failure in April of 1993, the inspector
calculated that the
pump had accumulated
about
999 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.801195e-4 months <br /> of
operation
since the last bearing temperature
measurement
was
taken
on August 9,
1992.
The inspector
concluded that although the corporate
engineers
had stated their concern for potential
pump
inoperability if the bearing condition was not promptly
detected
(September
20,
1991,
memorandum),
the licensee
took
weak preventive actions to monitor and detect for bearing
wear
and damage.
It is not possible to absolutely
conclude
that stronger corrective action for the
1991 bearing
problem
would have prevented
the
1993 problem.
Therefore,
a Notice
of Violation regarding
inadequate
corrective action is not
in order.
12
Leakin
Packin
Gland Job Order:
The inspector also reviewed the causes
behind the protracted
repair-of the leaking packing= gland,
which was first
identified as
a maintenance
deficiency
on October 9,
1991,
(A/R Number A0005591)
by the operations staff.
The
inspector
had
been
concerned for some time on the number of
backlogged
maintenance
job orders
which may affect system
performance.
The inspector
had previously reviewed the
licensee's
computer printout to determine
whether
any Action
Requests
had
a potential to impose
an operability impact
with any of the safety-related
systems.
As discussed
in
paragraph
3.c of Inspection
Report 50-315/93011(DRP);
50-316/93011(DRP),
the inspector did not identify any
which could potentially impact the operability of safety-
related
equipment at that time.
Because
the failure to repair the leaking packing gland
was
the cause for the water found in the bearing oil, the
inspector contacted
the integrated
scheduling
group to
determine
the reasons
why it took so long to make the
repair.
The inspector's
discussion
with the plant
integrated
scheduling
group found that the packing leak
repair,
a priority 30 job,
was never identified for work
during the Unit 2 outage in 1992.
In addition,
although it
was scheduled
to be done during the week of Harch
1,
1993,
it was deferred
because
of resource
constraints.
The repair
was then rescheduled
to be done during Hay 1993;
however,
before identification of the bearing failure on April 20,
1993, created
an opportunity to correct the leak.
The
licensee
repaired
the packing leak on the
pump packing
during the
same period it was out of service to replace the
pump shaft
and the outboard bearings.
The prolonged failure to perform corrective maintenance,
over
a period of 18 months,
allowed
a condition which was
originally considered
"minor" to develop
consequences
which
were not minor,
as described
above.
As such,
the original
condition adverse to quality, which involved an equipmen't or
material defect,
was not promptly corrected.
Title 10 of
the
Code of Federal
Regulations,
Part 50, Appendix B,
Criterion XVI, requires that conditions
adverse
to quality,
including equipment or material defects,
shall
be promptly
corrected.
Failure to correct the
known seal
leakage
on
Unit 2 East
NDAFP,
a safety-related
component,
over
a period
of 18 months, is considered
a violation of Appendix B,
Criterion XVI (Viol ati on 50-316/93016-01(DRP)
) .
The inspector's
discussion with the head of the integrated
scheduling
group found that work activities classified
as
priority 30 are currently repaired within 6 months.
13
7)
Oil Chan
e Pro ram:
The inspector
reviewed the licensee's
practice of oil
changes
performed
on all auxiliary feedwater
pumps
and found
that it was inconsistent with the frequency
recommended
by
the vendor manual.
Although the inspector
concluded that
the licensee's
oil change periodicity performed
on the
pump
may not necessarily
cause
any damage to the bearings,
he did
conclude that the licensee
may want to consider additional
oil changes
based
on
pump run hours.
Currently, the
licensee's
program does not require oil change
based
on the
number of hours of pump operation.
The licensee's
vendor manual
made the following
recommendations:-
"Remember that oil requires
frequent replenishment
at
normal temperatures
and very frequent replenishment
at
high temperatures.
Oil is always subject to gradual
deterioration
from use
and contamination
from dirt and
moisture.
In time the accumulated
sludge will be
harmful to the bearings
and cause
premature
wear.
For
this reason,
draining
and flushing are necessary
at
regular intervals (at least
every
3 months)."
"It is
a good practice to change
the oil every 600
hours of operation."
The licensee
performed oil changes
on the auxiliary
pumps every 48 weeks.
The inspector's
review of
the oil changes
on the Unit 2 East
HDAFW pump using the
completed job order packages
found that the licensee
was
. performing the scheduled oil changes
on the Unit 2 East
HDAFW pump.
At the time of discovery of the Unit 2 East
HDAFW pump bearing failure, the
pump had accumulated
about
723 hours0.00837 days <br />0.201 hours <br />0.0012 weeks <br />2.751015e-4 months <br /> of operation
since the last oil change.
The inspector also conducted
discussions
with some of the
engineers
and found that although the licensee
does
have oil
analysis
programs for the turbine
and the diesel
generator
systems,
they did not have it implemented for the other
safety related
pumps at the time of the April 1993
HDAFW
bearing failure.
Consequently,
some of the oil samples
taken from the
HDAFW pumps were never analyzed.
Currently,
the licensee
is in the process of implementing
an oil
analysis
program for other safety-related
pumps to determine
what type of impurities are being found in the oil.
AFW Pum
Seal
Packin
Leaka
e
The inspector
reviewed licensee
action in response
to concerns
related to packing leakage
on the
AFW pump outboard seals.
The
inspector
noted that the licensee
took appropriate
preventive
action after being prompted
by the inspector.
The licensee
had identified the root cause of the failure of the
Unit 2 "E" HDAFP outboard thrust bearing,
as discussed
above
and
in earlier inspection reports,
to be water intrusion into the
bearing
housing from an excessive
pump outboard
seal
During a subsequent
routine plant walkdown, the inspector
observed
that leakage
from the Unit 2 turbine-driven auxiliary feedwater
pump
(TDAFP) outboard
seal
was excessive
in relation to the other
pumps.
The licensee
determined that the leak was not
an immediate
operability concern;
however,
the inspector noted that the
licensee
did not take
immediate action to monitor seal
leakage,
including providing guidance to plant personnel
as to what was
acceptable
leakage.
In response
to the inspector's
concern,
the licensee initiated
¹ 17327 to monitor seal
leakage
during the Unit 2 TDAFP routine
surveillance
run on July 22,
1993.
The inspector
observed
the
activity and noted that there
was minimal leakage after the
pump
was started
and seal
temperatures
remained
acceptable;
therefore,
only a minimal packing adjustment
was required.
As preventive action,
the licensee
proposed
to provide guidance
in
auxiliary equipment operator shift tours
and appropriate
maintenance
procedures
to monitor seal
leakage
to verify no water
migration from the seals to the bearing housings.
The inspector
will continue to monitor licensee
actions
in response
to this
issue.
One violation,
and
no deviations,
unresolved,
or inspector followup
items were identified.
Actions on Previousl
Identified Items
92701
92702
a ~
(Closed)
Open Item 315/88007-02:
Non-Class
IE Motor Installed in
a Class
lE Valve Motor 0 erator
as
a
Tem orar
Modification
Replacement
of a class
1E motor with a non-class
lE motor for
valve
I-WMO-754 in Temporary Modification (TM) No.
152;
and delay
in closeout of this TM.
The vendor qualified the installed motor for class
lE
applications,
and this
TM was closed out.
The inspector
reviewed
the licensee's
TH status report dated
June
28,
1993.
There were
only 10 non-outage
related
TMs open for both units.
The
TMs were
prioritized based
on the importance of these modifications.
There
were only two items
on this status
report which were
open for over
18 months.
These
items were not considered
to be safety
significant.
This item is considered
closed.
15
b.
(Closed)
Unresolved
Item 316/88018-01:
Re
1 acement
Pum
Failed to Meet Performance
Curves
Low developed
head for Unit 2 West Essential
Service
Water
(ESW)
pump did not meet
UFSAR statements.
A UFSAR revision, clarifying
the
ESW flow requirements
was submitted for approval
in July 1993.
The differences of
ESW requirements
in Technical Specifications
for Units
1 and
2 have
been reconciled
by revisions to these
documents.
Table 9.8-5 of the proposed
UFSAR revision states
that
an
ESW flow of 6890
gpm is required for LOCA injection.
Each of
the
ESW pumps were being tested
at
a flow of 7000
gpm for a
minimum differential pressure
of 67.8 psid.
The inspector
reviewed the
pump performance
data trended for all
the
ESW pumps since January
1991,
and concluded that these
pumps
currently meet the minimum requirements
set for these
pumps.
This
item is considered
closed.
c.
(Closed)
Unresolved
Item 315/88023-02:
Potential Violation of
E
Re uirements
For Reactor
Vessel
And Pressurizer
Head Vents
d.
Discrepancies
in configuration of junction boxes
and flex conduits
for=reactor
head
and pressurizer
vent solenoids resulting in
environmental qualifications
(Eg) concerns.
The licensee
issued
an
LER (315/88010)
and
a Problem Report
(PR-88-732) to address
this issue.
The configuration
discrepancies
were corrected.
Several
procedure
enhancements
were
made to preclude similar discrepancies.
The licensee
started
a
comprehensive
program to compile
Eg design data
and to confirm the
as built configurations
by walkdowns.
The licensee
provided
Eg
training to several
plant personnel
in Engineering,
Operations,
Maintenance,
and
gA/gC Groups.
This item is considered
closed.
(Closed)
Unresolved
Items
315 93011-03
316 93011-03:
Root
Cause
s
of the Unit 2 East
MDAFW Pum
Bearin
Failure
Unit 2 East
MDAFW pump thrust bearings
were found to have failed
catastrophically
on April 20,
1993.
The licensee
conducted
an
investigation into the causes
of failure.
The inspector
reviewed the licensee's
investigation into the
causes
of the Unit 2 East
MDAFW pump bearing failure and agreed
that the primary cause
was the water found in the oil which caused
the lubricating properties of the oil to degrade.
A detailed
discussion of the root causes
and other aspects
of this issue is
discussed
in paragraph
4.a. of this inspection report.
No violations, deviations,
unresolved,
or inspector followup items were
identified.
16
I
6.
Re ortable
Events
92700
92720
The inspector
reviewed the following Licensee
Event Reports
(LERs) by
means of direct observation,
discussions
with licensee
personnel,
and
review of records.
The review addressed
compliance to reporting
requirements
and,
as applicable,
that immediate corrective action
and
appropriate
action to prevent recurrence
had
been
accomplished.
a 4
(Closed)
LER 316/92005-LL:
The Flan
e Seals
on the Containment
E ui ment Hatch
and Personnel
Airlock Were Not T
e-B Tested
Since
Initial Start-u
Due to Pi in
Errors
This
LER was closed
based
on adequate
root cause
determination
and
corrective action.
On April 15,
1992, the licensee
discovered that piping intended to
facilitate Type
B leak rate testing of the Unit 2 equipment
hatch
and personnel
airlock flange seals
was not connected
to the flange
seal test ports,
but to other holes only partially penetrating
the
As a result,
the licensee
determined that it had not
performed the Type
B testing
on the flanges since initial plant
start-up.
The licensee
determined
the root cause
to be
an error in the
original plant piping configuration drawing.
As corrective
action,
the licensee verified the correct piping configuration
on
Unit
1 and revised the appropriate
plant procedure
to properly
test the Unit 2 seals.
The licensee
successfully
tested
the seals
on Unit 2 on April 17,
1992.
In addition, the licensee initiated
a design
change to remove the piping connected
to the wrong flange
This event involved
a violation of TS 3.6. 1.2
and
Appendix J, which required
a Type
B leak rate test
be performed
every
24 months.
However, the event
had minimal safety
significance
because
the flange seals
were successfully
challenged
during previous
Type
A integrated
leak rate tests
(ILRTs).
In
addition, the licensee
properly reported the event
and took
appropriate corrective action.
Therefore,
pursuant to the
NRC
enforcement policy, (10 CFR 2, Appendix C), the
NRC is exercising
enforcement discretion for this matter,
and
will be issued.
b.
(Closed)
LER 315/92002-LL:
Ino erable
Emer enc
Diesel
Generator
Caused
B
Low Governor Oil Tem erature
The inspector determined that the licensee
performed
an adequate
root cause
evaluation
and took appropriate corrective action to
prevent recurrence
of the event.
However, the inspector
determined that the licensee failed to perform an adequate
safety
17
evaluation of the condition which resulted
in the event,
as
required
by plant procedures,
and their investigation failed to
identify this
as
a contributing cause.
On February
10,
1992,
the Unit
1
on overspeed
during
a routine surveillance test.
The licensee
determined that
the
EDG could have
been
since
February
6,
1992,
when
I-HV-DGS-DAB, Unit
1
malfunctioned
and
was de-energized
in the open position to ensure
adequate
cooling
to the
EDG.
The open
damper allowed outside air, with
temperatures
between
15 and
36 degrees
F, to cool the
EDG governor
warming line which was located adjacent
to the damper outlet.
This resulted
in a low governor oil temperature
and, ultimately,
sluggish governor operation
which caused
the overspeed trip.
As immediate corrective action,
the licensee
repaired
the damper,
returned the warming line to normal operating temperature,
and
restored
the
status
on February
10,
1992.
As
long-term corrective action,
the licensee verified adequate
flow
in the governor warming lines
on all four EDGs and placed
insulation
on these lines.
In addition, the licensee initiated
a
design
change to improve the reliability of the damper.
As follow-up, the inspector determined that the licensee failed to
perform
a documented
safety evaluation prior to failing the damper
open.
The purpose of the damper
was to prevent cold outdoor air
from entering the
EDG room during the winter whenever the
EDG was
not operating.
The damper
was normally maintained
in the closed
position
when both the supply
and exhaust
fans were not running.
The damper
was interlocked to open if either fan started.
The
fans were designed
to start automatically
when the
EDG started or
when the
room temperature
reached
90 degrees
F,
and stop
when room
temperature
decreased
to 70 degrees
F.
Paragraph
4.1 of licensee
procedure
"Temporary Hodifications,"
PHP
5040 HOD.001, defined
"any short term
configuration that exists
on plant systems,
components,
or
structures ... which does not conform to approved plant drawings
...
and is being used to maintain operation of the plant."
The
definition also provided "blocked open valves"
as
an example of a
In the case
described
above,
the licensee
failed to implement
as required
by
PHP
5040 HOD.001, for failing the Unit
1 AB EDG ventilation damper in
the open position.
This is considered
a violation (Violation 50-
316/93016-02(DRS)).
One violation,
one non-cited violation,
and
no deviations,
unresolved
or
inspector followup items were identified.
18
Safet
Assessment
ualit
Verification
37701
38702
40704
92720
The effectiveness
of management
controls, verification,
and oversight
activities, in the conduct of jobs observed
during this inspection,
was
evaluated.
The inspector frequently attended
management
and supervisory meetings
involving plant status
and plans
and focusing
on proper co-ordination
among departments.
Corrective Action to Reactivit
Excursion Transient
The inspector
reviewed the licensee's
investigation into
a reactivity
excursion transient
and determined that the root cause
evaluation
and
corrective action were adequate.
On April 11,
1993, shortly after placing the Unit
1 "S"
CVCS mixed bed
demineralizer
in service,
the Reactor Operator
(RO) observed
an
unplanned
increase
in Tave.
In response,
the
RO attempted
to borate
using
l-HARV-400,
CCP suction
from the "N" boric acid blender,
and then
l-HARV-451, "N" boric acid blender to the
VCT.
Both of these
normal
boration flow paths failed to provide any boric acid injection.
The
then took manual
control of the control rods to maintain Tave
and
commenced
emergency boration
when the
TS limit for axial flux
distribution
(AFD) was approached
to stabilize the reactivity transient.
The licensee
observed
a maximum power spike at
102 percent for a brief
period
and accumulated
a total of 6 penalty minutes outside their AFD
target
band during the transient.
The licensee
determined that the root cause of the temperature
excursion
was the incomplete boration of the demineralizer
which had
been
performed
on April 6,
1993, after the resin
was replaced.
The
incomplete boration
was due to inadequate
coordination
between
the
reactor
operators
and the Chemistry technician
who was taking the
RCS,
samples.
As corrective action,
the appropriate
operations
procedure
was
enhanced
to provide more specific guidance
on borating
a new
demineralizer
bed.
In addition,
a
memo
was distributed to all
operations
personnel
to stress
the importance of proper coordination
with Chemistry technicians
when
RCS samples
are being taken.
With regard to the failure of the parallel
normal boration flow paths to
provide boric acid, the licensee
determined that the failure I-HARV-400
to operate
was
due to severe galling to the stem
and bushings.
The
licensee
could not determine
the root cause of the failure of the flow
path through
I-HARV-451.
Since the valve opened
as expected
during the
event,
the fault appeared
to be intermittent blockage in the line
upstream of the blender.
Boric acid flow was attained
through this path
after the transient
was over.
The System Engineer walked
down the
affected piping and did not detect
any indication of faulty heat tape.
The System
Engineer will continue.to monitor system
performance
as part
of his normal responsibilities.
19
No violations, deviations,
unresolved,
or inspector followup items were
identified.
Mana ement Interview
The inspectors
met with licensee
representatives
denoted
in paragraph
1
on August 4,
1993, to discuss
the scope
and findings of the inspection.
In addition, the inspector also discussed
the likely informational
content of the inspection report with regard to documents
or processes
reviewed
by the inspector during the inspection.
The licensee
did not
identify any such
documents
or processes
as proprietary.
The licensee
disagreed
with the inspector that the Unit 2 East
pump was inoperable with the failed thrust bearings
on April 20,
1993.
They believed that the
MDAFW pump would have performed satisfactorily to
fulfillits design functions.
20
k'