ML17058A989
| ML17058A989 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 07/01/1992 |
| From: | Ruland W, Woodard C NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML17058A988 | List: |
| References | |
| 50-410-92-15, NUDOCS 9207170074 | |
| Download: ML17058A989 (20) | |
See also: IR 05000410/1992015
Text
U.S. NUCLEAR REGULATORY COMMISSION
REGION'1
DOCKET NOS.
REPORT NOS.
LICENSE NOS.
LICENSEE:
FACILITYNAME:
92-15
Niagara Mohawk Power Corporation
301 Plainfield Road
Syracuse,
13212
Nine Mile Point Nuclear Power Station, Unit 2
INSPECTION AT:
Scriba, New York
INSPECTION DATES:
May 12-14 and May 26-29, 1992
INSPECTORS:
C. H. Woodard, Reactor Engineer, DRS
C. Bennett, Reactor Engineer, DRS
3. Rajan, Mechanical Engineer, NRR/EMEB
E. Murphy, Consultant Inspector, American Systems Corp.
INSPECTOR:
C. H. Woodard, Reactor Engineer,
Electrical Section, EB, DRS
ate
APPROVED BY:
. Ruland, Acting Chief, Electrical Section,
Engineering Branch, DRS
a
s
y
Date
9207170074
920702
ADOCK 05000410
EXECUTIVESUMMARY
This inspection was in response
to the licensee's inspection findings which identified engine
cylinder liner and piston distress in their Cooper KSV-16 emergency diesel generator (EDG) ~
This distress was identified as similar to that found in 1989 at Susquehanna
in their Cooper
KSV-16 engines
as precursors which had led to several crankcase'explosion
events.
The inspectors found that the Cooper EDG users group had developed
a draft inspection
manual in April 1992 to be used as a guide in inspectirig the Cooper KSV engine cylinder
liners and pistons for distress.
This manual included input from the Susquehanna
root cause
evaluations which were concluded late in 1991.
It included color photographs of distressed
and damaged KSV-16 pistons, cylinder liners wrist pins, and wrist pin bearings from
Susquehanna
and told the users how to look, what to look for, and what constitutes the
distress indications which could lead to EDG failure. It also covered corrective actions which
included the removal of piston wrist pin end cap seals and the piston bottom of skirt oil
scraper ring to improve cylinder lubrication.
The inspectors found that the licensee had completed the inspection and changes
to the
Division 11 EDG.
Inspection procedures,
methods,
results and changes were reviewed for
this unit. The inspectors witnessed
the inspection of the Division I unit.
As a consequence
of the distress indications found, the licensee has replaced six of the pistons and liners on the
Division I EDG and nine of the pistons and liners on the Division IIEDG. With Cooper
concurrence,
all of the wrist pin end caps and bottom piston skirt oil scraper rings were
removed in both EDG units in order to improve piston to liner lubrication.
The licensee
f'ound no damage to wrist pins,,wrist pin bearings, crankshaft journals, and connecting rod
bearings.
The inspectors concl'uded that the licensee had aggressively pursued the EDG inspection with
a dedicated
system engineer aided by onsite Cooper representatives
in identifying the
distressed
components,
replacing them and in making engine modifications to overcome the
potential root causes of the problems.
This work was actively supported by upper
management
with much day-to-day direct involvement in the EDG rooms.
Modifications
were made in accordance with approved procedures
and included appropriate QA involvement
in all areas.
Licensee root cause analyses with followup enhanced
inspections are planned to
provide additional confirniation that the basic problems (root causes)
have been corrected.
1.0
BACKGROUND
Niagara Mohawk (NM) Unit 2 Nuclear Power Plant utilizes a Cooper KSV-16 emergency
diesel generator (EDG) unit in each of the Division I and Division II safety-related electrical
power trains.
Cooper KSV-16 EDG units at the Pennsylvania Power and Light (PP&1.)
Susquehanna
Nuclear Power Plants underwent a series of crankcase explosions the last of
which occurred in 1989.
Distress and damage found in the failed Susquehanna
EDG units
included piston and cylinder liner distress in which piston tin coating (bearing material) was
wiped (galled/scraped) off onto the liners.
Piston wrist pins and wrist pin bearings were
damaged
and press-fit piston wrist pin end cap seals had moved out and galled cylinder
chrome-plated liners.
As a consequence of these events, PP&L took corrective actions and made modifications
based
upon suspected
root causes.
These included the replacement of all distressed cylinder
liners, pistons, wrist pins and bearings;
the removal of the piston wrist pin end cap seals and
the piston skirt bottom oil scraper rings to provide for better piston/liner lubrication; the
elimination of cold, dry and unnecessary
fast starts; and modifications to provide for warm
pre-lubricated engines.
PP&L has not reported any evidence of cylinder distress since these
changes were made in 1989.
Since 1989, PP&L conducted extensive root cause evaluations
including extensive proof testing of a similar Cooper KSV engine which was heavily
instrumented and operated
under controlled conditions to confirm the root causes of the
cylinder distress and engine failures.
These evaluations were concluded during late 1991.
Based upon the evaluations, PP&L along with other nuclear utilities which use the Cooper
KSV EDG units prepared
a draft inspection manual.
This inspection manual provided the
details needed for inspection of the EDG units.
It included vivid detailed color photographs
of normal and distressed
pistons and cylinder liners. It provided information as to what to
look for, how to look for it and what constitutes distress which could lead to EDG failure.
The draft inspection manual was received by NM Unit 2 on April 30, 1992.
At this time the
Division I EDG had con>pleted its outage maintenance,
surveillance, testing and was
considered operable.
The Division II EDG was near completion of the same.
Although there
was no requirement for NM to conduct the draft manual inspections, both the Division II and
l EDG units were sequentially declared inoperable and inspected per the draft manual by NM
personnel with the direct assistance of Cooper field service personnel.
(The EDG units each
had started approximately 350 times and had operated approximately 750 hours0.00868 days <br />0.208 hours <br />0.00124 weeks <br />2.85375e-4 months <br />.)
2.0
INSPECTIOÃ DETAILS
2.1
The inspectors reviewed the "Draft Inspection Manual for Cooper Bessemer (C-B) Model
, KSV Diesel Engine Cylinder Liners, Pistons and Bearings," dated April 9, 1992.
The
manual was prepared by and for the use of the utilities which comprise the C-B KSV EDG
users group for the 16 nuclear power stations which use this EDG unit. The stated purpose
of this manual is to provide guidance for the evaluation of degradation of cylinder liners,
pistons, main and connecting rod bearings and camshafts.
These components were selected
for inclusion due to the potential for a significant reduction in reliability ifthe degradation
is
not identified, or because
the mode of degradation is relatively unique to C-B KSV diesel
engines.
The intent was to provide general industry guidelines for identifying the various
modes of component degradation.
The manual was incomplete in that it did not yet include
guidance for the evaluation of bearings and camshafts.
However, the inspectors found the
draft manual to be adequate for inspection for the tin smear degradation of the pistons and
cylinder liners.
The information presented
relating to the piston-to-liner tin smearing found
in the inspection
n>anual represents
information not previously available from the EDG
inanufacturer, other users or the NRC.
Al tire time of this inspection, the licensee had completed the inspection and refurbishment of
the Division II EDG by the replacement of nine of sixteen (16) pistons hnd cylinder liners
and by the removal of all 16 cylinders piston wrist pin seal end caps and the piston skirt
bottom oil scraper rings.
Cooper concurred by letter to the licensee in the modification,
including the removal of end cap seals and bottom piston skirt oil scraper rings.
By
conducting a review of the licensee's documentation of the inspection of the Division II
EDG, including color photographs of each of the cylinders and liners, the inspectors
concluded that the licensee had conservatively identified and replaced all distressed
pistons
and cylinder liners.
ln addition, the inspection of the piston wrist pins and piston bronze
insert bearings mei the inspection'riteria.
No indications of damage were evident.
The inspectors witnessed portions of the licensee's
inspections,
replacements
and modification
of the Division I EDG unit including the removal of three of the six pistons and liners that
were replaced.
The inspection and modifications were performed in accordance with the
licensee's
approved NMP 2 simple design change (SDC) Procedure No. SDC 2-0136-92 and
the user's Group Inspection Manual.
Initial inspection of each of the 16 cylinders was
conducted by observing, the upper cylinder liner and piston crown through the fuel injection
port, by means ot'
and then by inspecting the lower portion of the cylinder liner
and piston skirt via the crankcase
access cover openings.
With the limited visibilityprovided
during this initial inspection, the licensee indicated that as many as 10 cylinders liners and
pistons were suspect.
By the sequential removal of cylinder heads, pistons and liners from
the engine for all 16 cylinders, the licensee was able to thoroughly inspect each cylinder.
As
a consequence,
the licensee found that six of the pistons and liners required replacement.
The licensee found no indications of overheating or damage to any of the wrist pins and wrist
pin bearings.
Since the connecting rod caps were removed to pull the pistons, the licensee
was able to inspect all connecting rod caps and bearings and the crank shaft journals.
There
was no evidence of daniage.
The inspectors found active quality assurance
and management
involvement evident in all of the procedures
and in the EDG rooms on a routine daily basis.
The inspectors also found that the licensee's EDG system engineer had "changed hats" to
become a maintenance
mechanic in order to be able to participate with "hands-on" direct
involvement.
This direct hands-on involvement was considered
to be a positive factor in
conducting this major effort.
Overall, the inspectors concluded that the inspection,
modifications, and replacements
were conducted in a professional manner, by
experienced/trained
professionals who used approved procedures,
and whose efforts were
monitored effectively on an on-going basis by quality assurance
and plant management.
The inspectors tried to determine when some of the tin removal from the pistons occurred.
Since some ot'he iin reiiioved trom the piston will be washed down into the crankcase,
engine oil, analyses were examined for preceding years for tin content.
The licensee
performed no detailed analyses for tin prior to 1990.
These monthly analysis did not disclose
any significant variations in tin content of the lube oil. Therefore, the inspectors could not
identify a specific time during the preceding two years in which significant tin transfer
occurred.
This does not rule out tin transfer during this time.
Flakes of tin transferred from
the piston to the engine crankcase will tend to float in the oil and much of it could be filtered
out by the engine oil tilters.
The licensee had made no periodic analyses of engine oil filters
for collected elements.
Oil samples
had been taken from shutdown units crankcase without
regard to time after shutdown and crankcase location.
The licensee agreed to establish
a
meaningful oil sampling procedure that will permit comparing and trending subsequent
~ samples on known reference basis.
The inspectors found that "pop testing" of fuel.injectors was not routinely performed to verify
proper performance.
(It was not included in the vendors routine periodic maintenance
instructions.)
The "pop test" verifies that the injector will open at the proper fuel oil pressure
and that the injector produces
a conical spray mist of fuel.
The proper operating pressure
and
spray characteristics
are essential
to a balanced combustion engine which produces its peak
operating pressure
in each cylinder at the proper (approximately the same) crank shaft angle.
lnjectors that weep or dribble or pop early or late can be compensated
for by adjusting the
fuel rack for each cylinder; however, this does not resolve the basic engine combustion
problem.
The inspectors found up to eight degrees difference in the fuel pump rack setting
between cylinders.
The licensee is currently addressing
this potential problem.
The inspectors questioned
the licensee's continuing approval as satisfactory the EDG lube oil
based
upon satisfactory viscosity when the monthly analysis received (since February
1991)
show fuel oil dilution in the engine oil ranging between
5 and 6%.
This represents
approximately 20 gallons of fuel contamination in the oil. Fuel dilution in engine oil above
2% is generally considered
to be a problem.
A source of raw fuel oil is from fuel injectors
down past the pistons into the crankcase.
Any liquid fuel oil degrades piston to liner
lubrication as it moves from injectors to the crankcase
and also reduces the lubricity of the oil
in the crankcase.
During the inspection, the licensee obtained information which indicates
that there was an error in the analysis as a consequence of the contract laboratory using an
improper oil standard.
Since the inspection, the inspectors were advised that new analyses
made utilizing the proper standard indicated that the fuel oil dilution was less than 2%.
The
licensee has now established
acceptance
limits on the permissible levels of fuel oil in the
engine oil (the C-8 EDG manual did not provide the licensee with limits). Although this
item is now resolved, the inspectors considered it a weakness in that the high dilution had not
been properly addressed.
2.2
EDG Fuel Oil Receipts
The inspector found that the licensee's procedures permit up to 30.days to accomplish the full
analysis of the EDG t'uel oil after it is added to the EDG fuel tanks.
This procedure is
generally in accordance
with the guidance provided to the licensee in'the NRC Standard
Technical Specifications and in the NRC Regulatory Guide 1.137.
However, the licensee had
not recognized tliat by adding tliis unanalyzed
class
1E material to the class 1E EDG units,
that the operability status ot each of the EDG is in question until the confirming analysis is
received.
In fact, by topping off all of the EDG -fuel tanks from a single tanker (the common
practice), there exists a means for the simultaneous common mode failures of all EDG units.
Prior to the conclusion of this inspection, the licensee had confirmed the analysis of the fuel
oil in each of the EDG tanks and had implemented temporary procedural changes to address
this problein.
Draft procedure N2-PM-015, Diesel Fuel Off Load, was reviewed.
It includes
provisions for holdin< the fuel oil in the tanker until full analysis is made prior to adding it to
the tanks.
According to the licensee, previous procedure N2-DDI-5.19 is to be superseded
by the new procedure et'fective June
I', 1992.
Implementation of the new procedure will
resolve this problem.
2.3
Fast Start and Load
Fast starting and fast loading of cold, dry (not pre-lubed) EDG units is considered to
contribute to the unreliability of the units.
NRC Generic Letter 84-15 requested
that
licensee's
address
this issue by making changes
in their surveillance and testing procedures
to
minimize or eliminate this type of starting and loading.
The inspectors determined that NM
had addressed
this issue except for a 184-day test when the units are dry-fast started and fast
loaded into the grid.
The dry start and fast loading during the six-month test are required by
licensee Technical Specification 4.8.1.1.2.a.5
and are conducted in accordance with EDG test
procedure N2-OSP-EGS-M001.
The inspectors questioned
the need for the six-month dry
start, fast load test.
The licensee plans
to review the need.and, ifnot required, to request a
Technical Specification amendment
to delete this requirement.
3.0
CONCLUSION
The licensee acted responsively to the EDG Draft Inspection Manual and proceeded with
further inspection in which precursors of potential EDG failure were identified and corrected
prior to plant startup.
The inspection was made by the licensee with good management
sup'port, active QA involvement, with the effective utilization of a knowledgeable systems
engineer using approved procedures
and with the support of the EDG manufacturer.
Although the license's high fuel oil concentration in the EDG lube oil proved to be an
analysis error, they failed to properly question the high concentration.
The Cooper-Bessemer
(CB) KSV engine is susceptible to piston-to-liner tin smear degradation in the nuclear power
industry which can lead to EDG failure.
APPENDIX A
PERSONS CONTACTED
Nia ara M hawk Power Co
ra ion
"Q. Brownell, Site Licensing Supervisor
- G. Thompson, System Engineering Supervisor
"'M. McCormick, Plant Manager
T. Fiorenza, Diesel Generators
Systems Engineer
K. Murray, Engineer
D. Kazyaka, Engineer
K. Coates,
Maintenance Manager
R. Dean, Lead Systems Engineer
J. Blasiack, Plant Chemistry Supervisor
J. Savoca,
Maintenance Support Manager
J. Doherty, Procurement Engineer
.onsultant
E. Murphy, Diesel Generator Specialist, American System Engineering Corporation
Coo er-Bessemer
Co
oration
~
g
J. Horn, Manager, Engineering
.S. Nuclear Re ulator
Commission
E. Tomlinson, Diesel Generator Specialist, NRR
J. Rajan, Supervisor, Mechanical Engineering Branch, NRR
":W. Mattingily, Resident Inspector
"C. Bennett, Inspector
W. Schmidt, Senior Resident Inspector
Asterisk (":) indicates those present at the exit meeting.
J
'
APPENDIX B
DOCUMENTS REVIEWED
1.
Inspection Manual for Cooper-Bessemer
Model KSV Diesel'Engine Cylinder Liners,
Pistons and Bearings.
Draft Issue, dated April 8, 1992.
U.S. NRC Safety Evaluation of the Emergency Diesel Generator (EDG) Crankcase
Overpressurization
Events, Susquehann'a
Steam Electric Station, Units
1 and 2 (TAC
Nos. M81347 and M81348) dated May 11, 1992.
Cooper-Bessemer
instruction Manual ¹93080, KSV Turbo Charged Diesel Generating
Unit, Nuclear Power Plant-Emergency
Standby.
- 200xx,Rev 9/81
~
Power and Light Company (PP&L) Root Cause Investigation.
Piston
Tests at SMLP and Distortion/Contact Patch Analysis, dated September
1991.
5.
Susquehanna
Steam Electric Station EDG Root Cause Investigation Baseline Engine
Test Report Cooper-Bessemer
KSV-12-GT Diesel Engine at Sumner Municipal Light
Plant, MPR-1218, Revision A, dated December
1990.
~
6.
U.S. NRC Region I Inspection Report of Cooper Diesel Generator Failures'at PP&L
Susquehanna
Power Station, Reports 50-387/89-30 and 50-388/89-27.
U.S. NRC Memorandum for Hehl, Region I from Varga, NRR Task Interface
Agreement Regarding Emergency Diesel Generator Failures at Susquehanna
Steam
Electric Station, Units l and 2.
8.
NMPC ¹2 Technical Specification 3/4.8, Electrical Power Sources.
NMPC ¹2 Division I and
11 Diesel Generator Operability Test Procedure N2-OSP-
EGS-M001, Rev. 3, dated 12/22/90.
10.
U.S. NRC Generic Letter 84-15, Proposed Staff Actions to Improve and Maintain
Diesel Generator Reliability.
1 l.
NMPC ¹2 Divisions l,2 and 3 EDG Lube Oil Analysis Summaries (2/17/91-5/24/92).
12.
NMPC ¹2 Chemistry Procedure N2-CSP-17V Hydraulic and Lubrication Oil
Surveillance,
Rev. 4/23/91.
13.
NMPC ¹2 Division l and 2 EDG Operating Logs (Operation through 4/15/92).
14.
NMPC ¹2 Division I and II Diesel Generator Operating Cycle 24-hour Run and Load
Rejection Test Procedure N2-OSP-EGS-R002,
Rev. 2, dated 8/6/90,
Appendix 8
15.
NMPC ¹2 Division I and II Diesel Generator Operating Cycle Simulated Loss of
Offsite Power with ECCS, Test Procedure N2-OSP-EGS-R004,
Rev. 3, dated
8/22/90.
16.
NMPC ¹2 Division I and II Diesel Generator Operating Cycle Simulated Loss of
Offsite Power with No ECCS, Test Procedure N2-OSP-EGS-R003,
Rev. 1, dated
2/7/91.
17.
NMPC ¹2 Division I and II Diesel Generator ECCS Start, Test Procedure
.
N2-OSP-EGS-R001,
Rev.
1, dated 10/26/92.