IR 05000530/1994016
| ML17310B327 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 05/27/1994 |
| From: | Wong H NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML17310B326 | List: |
| References | |
| 50-530-94-16, NUDOCS 9406060294 | |
| Download: ML17310B327 (60) | |
Text
APPENDIX U. S.
NUCLEAR REGULATORY COMMISSION
REGION IV
Inspection Report:
50-530/94-16 License:
NPF-74 Licensee:
Arizona Public Service Company P. 0.
Box 53999, Station 9082 Phoenix, AZ 85072-3999 Facility Name:
Palo Verde Nuclear Generating Station, Unit 3 Inspection At:
Maricopa County, Arizona Inspection Conducted:
April 15. through April 22, 1994 Inspectors:
K. Johnston, Senior Resident Inspector H. Freeman, Resident Inspector ong ie
,
roJec Branc a e Ins ection Summar Areas Ins ected; Special, announced, resident inspection of the licensee's discovery of damage to Unit 3 Emergency Diesel Generator (EDG) B.
During the inspection, Inspection Procedures 62703 and 92701 were used.
Results:
The inspectors concluded that the damage to Unit 3 EDG B, identified on March 23, 1994, represented a significant degradation to the EDG.
The damage resulted when intake and exhaust rocker arms failed in July 1993.
The damage included a bent articulated rod and damaged articulated rod pin, bushing, and wrist pin for Cylinder 6-left (6L).
The inspectors concluded that the damage to the diesel generator components was enough that assurance could not be provided that the diesel generator would have met its design function and, therefore, concluded that the EDG had been inoperabl since the rocker arm failures in July 1993 (Section 2).
The inspectors concluded that the licensee had missed several opportunities to identify and correct the damage to the EDG.
These opportunities were missed as a result of the following weaknesses:
9406060294
'F40527 PDR ADOCK 05000530
'
i I
l
~
Weak Work Instructions and Postmaintenance Testing The inspectors identified examples where work instructions for the inspection of the Unit 3 EDG B following the July 1993 rocker arm failures were not thorough in that they did not provide adequate inspection criteria (Section 4.2).
In addition, the postmaintenance testing specified in the work order was not adequate in that it did not specify a compres'sion test (Section 5).
The inspector concluded that these problems resulted from a weak work order development process for significant emergent work.
The process did not include formal Engineering and guality Assurance (gA) Department reviews of the technical content of the work order (Se~tion 4.3).
Inadequate Technical Review The inspectors found several instances where inadequate technical reviews were performed in the disposition of Unit 3 EDG B deficiencies.
These included:
Opportunities were missed to evaluate the hydraulic forces required to displace approximately 74 gallons of fuel oil past the piston rings on Unit 3 EDG B.
These hydraulic forces resulted in the damage to the articulated rod of Cylinder 6L of Unit 3 EDG B and associated pin (Sections 4. 1 and 6.2.3).
Licensee engineers performed an inadequate review when determining the likely bend direction of the articulated rod which had been subjected to cylinder hydraulic lock forces (Section 6.2. 1).
~
Licensee engineers failed to adequately apply industry information and vendor manual information to the analysis of the possibility of a bent articulated rod.
Industry information did not support their conclusions regarding bend direction (Section 6.2. 1)
and the vendor manual cautioned against continued operation with a bent rod (Section 6.2.2).
~
Following an inspection performed on Cylinder 6L during the midcycle outage in December 1993, engineers failed to adequately consider several factors which would have indicated that the inspection performed did not conclusively determine the cause of low combustion pressures in the cylinder (Section 8.3).
Weak management and guality Organization Oversight The inspectors identified that weak management and quality organization oversight contributed to the licensee's failure to identify the damage to Unit 3 EDG B.
Specifically:
~
Licensee management was not adequately involved in the oversight of the inspection and testing of Unit 3 EDG B following the Unit 3 EDG B rocker arm failures (Sections 4.2 and 5).
I I
l
j
~
The QA and Quality Control (QC) organizations were not involved in the oversight and review of Unit 3 EDG 8 for collateral damage following the rocker arm failures (Section 4.2)
Licensee management was slow to recognize and address a significant weakness in their process for the review of deficiencies which challenged system operability (Section 7. 1)
Engineering management failed to critically assess the technical evaluations performed by the system engineers to support continued operation of Unit 3 EDG B with a significant deficiency (Section 7.2).
Engineering management failed to provide adequate support of the system engineers and facilitate the resolution of the Unit 3 EDG B deficiencies (Section 7.3).
~
The QA and QC organizations had missed opportunities to critically assess the evaluations performed by engineering (Section 7.4).
Summar of A arent Violations:
Based on the significant damage observed in Unit 3 EDG B, the inspectors concluded that the diesel generator has been inoperable since July 1993.
Contrary to Technical Specifications requirements, the licensee did not restore the deisel generator to an operable condition within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> and did not shut down the reactor and proceed to cold shutdown within the time limits specified (530/9416-01).
The work order providing instructions for the repair of the rocker arms did not prescribe instructions appropriate for the circumstances.
The instructions did not include sufficient guidance for inspection of potentially affected diesel engine components (530/9416-02).
Appropriate testing to demonstrate that the EDG would perform satisfactorily in service was not identified following the repairs performed on Unit 3 EDG 8 in July 1993.
Compression testing was not specified to be performed even though fuel oil had passed from the cylinder to the lube oil sump and the cylinder head had been removed (530/9416-03).
The licensee failed to promptly identify and correct a bent articulated rod and a seized articulated rod bushing, conditions adverse to quality, in July 1993 when the rocker arms failed and in October 1993 when conditions (low peak pressure)
indicated that there were significant deficiencies withjthe EDG cylinder (530/9416-04),
I'
f I
Attachments:
Persons Contacted and Exit Meeting Figure
Figure
~
1
'
I r
DETAILS
BACKGROUND 1. 1
~Summar While performing outage maintenance on the Unit 3 EDG 8 on Harch 23, 1994, the licensee discovered damage to the Cylinder 6L connecting rod (articulated),
the Cylinder 6R connecting rod (master)
and the articulated rod pin.
The licensee discovered that the articulated connecting rod had bent, that the Cylinder 6L articulated rod pin had seized in its bushing, that the Cylinder 6R master connecting rod showed signs of heat stress, and the Cylinder 6L wrist pin was deformed.
The damage to Unit 3 EDG 8 was apparently due to a the failure of the intake and exhaust rocker arms oF Cylinder 6L during a surveillance test in July 1993.
During the test, Unit 3 EDG B ran for approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> with both the intake and e'xhaust valves in the closed position.
The fuel injector continued to inject fuel oil and subsequently filled the cylinder.
The resultant hydraulic forces were transferred to the articulated rod and its bearing.
The inspectors determined that, between July 1993 and December 1993, the licensee had several opportunities to identify the damage to the articulated rod.
However, due to inadequate engineering review, inspections, postmaintenance testing, and management/quality oversight, the damage to Unit 3 EDG B was not identified until the refueling outage in Harch 1994.
Based on the observed damage,
.the inspectors concluded that the EDG had been inoperable from July 1993 to Harch 1994.
1.2 Descri tion of Emer enc Diesel Generator Connectin Rod Confi uration Unit 3 EDG 8 is a vee-20 (20 cylinders),
turbocharged, 4-stroke diesel engine (KSV-20T) built by Cooper-Bessemer.
The connecting rod assemblies, consisting of an articulated rod, master rod, and rod cap, are forged from carbon steel.
The master rod (right cylinder bank) is connected to the crankshaft via a
journal bearing and rod cap.
The articulated rod (left cylinder bank) is attached to the master rod via a pin and bushing assembly (see Figure 1).
This arrangement allows the articulated rod to pivot relative to the master rod as the crankshaft rotates.
1.3 EDG Desi n Basis The purposes of onsite emergency electric power systems are:
(1) to provide power promptly to engineered safety features during a concurrent loss of offsite power and an accident, and (2) to provide power to equipment needed to maintain the plant in a safe condition during an extended loss of offsite power.
Additionally, the licensee's design basis manual stated that the diesel generator shall be capable of supplying power for the longer of the
following:
(1)
7 days, or (2) the time required to restore offsite power following a limiting design basis event.
DESCRIPTION OF DAHAGE TO UNIT 3 EDG B
2. 1 Descri tion and A
arent Cause of Dama e
On July 28, 1993, the licensee performed a Technical Specification required rapid load test of Unit 3 EDG B.
While investigating abnormal exhaust temperatures, the licensee identified that intake and exhaust rocker arms for Cylinder 6L had failed.
Following the rocker arm failures, the engine continued to operate for approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />.
Fuel Injector 6L continued to operate normally and injected fuel into the cylinder; however, because the intake rocker arm had failed, a fresh air charge was not introduced into the cylinder and the fuel, consequently, did not ignite.
Because the exhaust rocker arm had failed, the fuel mixture did not leave the cylinder through the exhaust system.
The fuel injector continued to inject fuel.
At some-point following the rocker arm failures, the fuel had displaced enough volume in the cylinder that excessive pressures developed during the piston's compression stroke.
These pressures forced fuel past the four piston compression rings and into the lube oil sump.
The licensee estimated that approximately
gallons of fuel oil had entered the lube oil sump.
Based on the observed damage to components and the potential overstress condition that occurred in July 1993, the licensee in Harch 1994 decided to replace the cylinder liner, the cylinder he'ad, and cylinder head studs, in addition to the obviously damaged components.
Hetallurgical analysis concluded that the overstress condition following the rocker arm failures could have caused undetected flaws to develop on the water jacket side of the cylinder liner and head.
I 2.2 0 erabilit Determination After the EDG damage was identified in Harch 1994, the licensee initiated a
review of the operability of Unit 3 EDG B to determine whether the EDG would be able to perform its safety function for the required length of time.
The inspectors noted that Unit 3 EDG 8 was run during surveillance testing for approximately 60 hours6.944444e-4 days <br />0.0167 hours <br />9.920635e-5 weeks <br />2.283e-5 months <br /> in the damaged condition between July 1993 and Harch 1994, with approximately 53 hours6.134259e-4 days <br />0.0147 hours <br />8.763227e-5 weeks <br />2.01665e-5 months <br /> in a loaded condition.
Unit 3 EDG B passed all surveillance testing acceptan'ce criteria during this period.
k By the end of the inspection period, the licensee had not completed its evaluation of the EDG's endurance capability.
However, the inspectors noted the following as-found conditions of Unit 3 EDG B, which would significantly affect the diesel generator's capability of meeting its design function:
~
Articulated rod bushing:
The articulated rod bushing, which should be affixed to the master rod by set screws, had seized to the articulated
I I
I
[
i
rod pin.
This caused the set screws to shear and the bushing to rotate inside the master rod.
The bushing was observed to have several cracks.
Had the bushing failed completely, there may have been a significant loss of lube oil to the Cylinder 6L piston because the piston received oil through the master rod and the bushing.
~
Articulated rod:
The articulated rod was bent such that the overall length was approximately 3/8 inch shorter than normal.
The inspectors measured the centroid of the rod and found the centroid to be shifted approximately 1 1/4 inch from the center of force.
It was not apparent whether the bending had occurred during the rocker arm failures in July 1993 or had continued to bend during subsequent EDG surveillance tests.
~
Naster connecting rod:
The master connecting r'od, in the area around the articulated rod pin, was subjected to substantial heat due to friction caused by a lack of lubrication as indicated by a blue discoloration of the metal.
The heat may have affected the material strength.
Additionally, the inspectors questioned whether the heat required to create these indications could have created an ignition source for the lube oil in the crankcase'
EDG operation with a degraded cylinder:
During engine analysis performed in October 1993, the licensee observed that Cylinder 6L had a
peak pressure substantially lower than average.
The licensee performed an evaluation at that time and concluded that Unit 3 EDG B could perform its design function even if one of the cylinders was not firing.
The inspectors questioned whether this analysis also considered the ability of Unit 3 EDG B to start within 10 seconds as required.
~
Other components:
On April 12, 1994, the licensee removed the Cylinder 6L head, liner, and piston.
Plant engineering recommended this be done after discussing the observed damage with an in-house metallurgist, The metallurgist was concerned that the head, a cast iron structure with jacket water cooling passages, could have experienced excessive tension stress which could result -in undetected cracking.
The licensee planned to evaluate the hydraulic forces which occurred in the cylinder.
Based on this evaluation, the licensee will evaluate the need for destructive examination of the cylinder head.
2.3 A
arent Violation Based on the above considerations and the extent of damage. evident in the diesel generator, the inspectors concluded that Unit 3 EDG B was inoperable from July 1993 to Narch 1994 in that the EDG would not have been able to perform its safety function.
During this period, when the unit was in Nodes 1,
2, 3, or 4, the licensee failed to comply with Technical
l
Specifications Action Statement 3.8. l. l.b in that the licensee did not restore the EDG to an operable status within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />, was not in hot standby within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, and was not in cold shutdown within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />.
The licensee was performing its own evaluation of the capability of the EDG to perform its safety function.
This is an apparent violation (530/9416-01).
CHRONOLOGY OF EVENTS li k
d 1 <<k fold C lid I f Unit 3 EDG 8 while conducting a semi-annual, rapid-load, surveillance test (required by Technical Specifications)
concurrent with a midcycle engine analysis.
The unit was in Node 1 at the time'f the test.
The engine continued to run for approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> until the rocker arm Failures were detected by engineers performing the engine analysis.
The licensee determined that the failure was caused by inadequate strength of the rocker arms.
The licensee inspected and replaced damaged components in the head; however, the licensee failed to inspect for damaged components in the power train (below the piston crown).
The licensee performed a monthly surveillance test of the EDG and declared the EDG operable within the 72-hour Limiting Condition for Operation (LCO) window.
~Au ust
Unit 3 EDG 5 uas started for a monthly surveillance test.
Engineering personnel attempted to complete the engine analysis that was interrupted by the rocker arm failure.
Because the engine analysis equipment did not operate reliably and failed while testing Cylinder 6L, a complete analysis was not conducted.
[Note - A review of,. this partial data following an October 20, 1993, engine analysis showed that Cylinder 6L had a low peak pressure during the August 25 test.]
~b>>
Ill
EOG 8 <<
t d f tly ill t;
however, scheduling problems prevented Engineering personnel from performing the engine analysis.
October
Unit 3 EDG B was started for a monthly surveillance test.
Engineering personnel completed the midcycle engine analysis that was started in July and determined that the peak pressure in Cylinder 6L was approximately 315 pounds lower than the average for the EDG.
Engineering personnel repeated the test on the cylinder, checked the test connection, and tested adjacent cylinders to confirm the results.
The diesel was run for approximately 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />.
The licensee initiated an evaluation to investigate the low peak pressures on
.
the cylinder.
The engineers determined several possible causes, including poor fuel delivery, poor piston ring sealing (blowby), or a distorted articulated rod.
The licensee performed an evaluation of the possibilities and concluded that poor fuel delivery or poor piston sealing would not significantly degrade the performance of the EDG in that there was sufficient design margin of the EDG to operate with one cylinder not firing.
The licensee also concluded that potential connecting rod damage was not significant because there were no significant abnormal vibration or ultrasonic
l
J0 l
I
indications from the EDG.
After consulting with the diesel vendor and briefing plant management, Engineering personnel concluded that the diesel was operable.
The Engineering Department concluded that further investigations were needed to determine the, actual cause of the low peak pressure and drafted an action plan.
November
Unit 3 EDG B was declared inoperable due to annunciator alarms that had no apparent cause.
Unit 3-Maintenance personnel performed troubleshooting and replaced a failed relay, but were not tasked to inspect for the cause of the low peak pressure while the diesel was inoperable.
Unit 3 Operations personnel performed a monthly surveillance test and ran the diesel for approximately 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> and then declared the diesel operable.
November
Unit 3 was shut down in midcycle to perform steam generator tube eddy current testing.
Eddy current testing required the licensee to enter midloop to install steam generator nozzle dams while the core still had a
substantial amount of decay heat.
Because of the plant condition, the diesel system engineers had difficulty scheduling an inspection of Unit 3 EDG B.
December
In a meeting regarding the operability of Unit 3 EDG B, a system engineer announced'hat he had a safety concern with regard to the continued operability of Unit 3 EDG B.
He raised this issue in the meeting in an effort to schedule an inspection while, the unit was shut down.
Unit 3 management responded to the engineer's concern and developed a plan for diesel inspections while in a reduced inventory condition.
December
Engineering personnel reperformed the engine analysis on Cylinder 6L to confirm the previous indications and obtained similar results.
Engineering also performed a compression check on the cylinder (by disengaging the fuel rack)
and determined that the compression pressure was approximately 175 psi lower than normal.
Based on 'the compression pressure, the licensee concluded that fuel delivery problems were not the cause of the low peak pressure.
December
Unit 3 EDG B was declared inoperable and removed from service to perform visual inspections of Cylinder 6L.
A video probe was inserted into the cylinder through the, injector port to inspect the cylinder liner and the piston crown area.
The licensee observed that the liner was in an acceptable condition and that there were some signs of piston ring "blowby" which would indicate exhaust gases blowing by the piston rings.
Engineering personnel concluded that the low peak pressure was caused by poor piston ring sealing and, after discussions with unit management, Maintenance, gC, and Planning, concluded that the cause could be resolved at the next refueling outage.
The licensee did not visually inspect the power train components.
March
1994 Unit 3 was shut down for a refueling outage.
Unit 3 EDG B was removed from service for maintenance and to resolve the cause of the low peak pressure, On March 23, the licensee discovered that the articulated rod in Cylinder 6L was bent and that the articulated rod pin had seized with its bushing.
The licensee replaced the damaged components and initiated a root
0,
)
-10-cause investigation.
The licensee concluded that the bent articulated rod and seized bushing appeared to be a result of the rocker arm failures that occurred on July 28, 1993.
INVESTIGATION FOLLOWING ROCKER ARK FAILURES The inspectors reviewed the actions the licensee took following the rocker arm failure event in July 28, 1993, to determine if there had been opportunities to identify the damage to the Cylinder 6L articulated rod and its bearing.
The inspectors concluded that Engineering had not assessed the impact of hydraulic forces resulting from the fuel oil in Cylinder 6L.
Additionally, the inspectors concluded that inspection guidance was not provided in the work order, resulting in the system engineer and work planner working directly with Haintenance personnel to determine the possible components that had been damaged during the failed rocker arm event.
4. 1 Review of H draulic Forces Followin the Rocker Arm Failures The inspectors reviewed the July 1993 troubleshooting and repair of the EDG and determined that the licensee had focused on the rocker arm failures and not on consequential damage.
The system engineers provided two reasons why they did not believe that significant hydraulic forces had been-established:
(1) the fuel oil was atomized as it was injected into the cylinder and would behave as a compressible gas, and (2) past experience at Palo Verde led them to believe that the most susceptible component in the engine to a hydraulic lock was the cylinder liner.
The inspectors concluded that these assumptions were not valid.
Immediately after the rocker arm failures in July 1993, the licensee took samples of the EDG lube oil which showed that when the EDG was operated for 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> with the cylinder intake and exhaust rocker arms failed, approximately 74 gallons of fuel oil went into the EDG sump.
The inspectors concluded that the most likely source of the fuel oil was from Cylinder 6L.
Sufficient pressure was built up for the fuel oil to be forced past the piston rings and into the lube oil sump.
This was caused by the injection of fuel without combustion due to the air intake valve being closed and, with the exhaust
.
valve closed, there was no other place for the fuel oil to go.
The licensee believed that an indicator of a hydraulic lock in the cylinder would be a cracked cylinder liner.
This was based on a past event at Palo Verde in which a crack in a cyl.inder head had allowed jacket cooling water to fill the cylinder.
When the licensee had attempted to start the engine, the cylinder liner split longitudinally.
The inspectors reviewed this event and the licensee's root cause of failure evaluation.
The cylinder which had the cracked head was adjacent to a cylinder in which a connecting rod had broken and was ejected through the side of the cylinder 5 months earlier.
During that failure, the diesel continued to run for more than 40 minutes while the licensee attempted to shut it down.
The cylinder was subjected to severe thermal stresses due to a lack of jacket water cooling caused by the ejected rod.
The licensee's root cause of failure analysis for the cylinder liner
-11-concluded that the cylinder may have cracked due to higher than normal thermal stresses.
The inspectors concluded that this failure did not support the conclusion that the liner was the most susceptible component to a hydraulic lock.
After identification of the damaged components in March 1994, the inspectors determined that the licensee had not,evaluated the amount of force that had acted upon these components when the rocker arms failed.
The licensee later estimated that the event developed pressures between 5000 and 8000 pounds per square inch (psi) in Cylinder 6L.
The excessive pressures in the cylinder generated an estimated 700,000 to 1. 1 million pounds of force, which were transferred from the piston, to the wrist pin, articulated rod, articulated rod pin and bushing, and finally to the master rod and crankshaft.
These components normally see approximately 240,000 pounds of force (1,650 psi average peak pressure x 143 in'iston surface area).
The excessive forces apparently caused deformation of the piston wrist pin, compressive and bending deformation of the articulated rod, and the articulated rod pin to seize in its bushings The force apparently exceeded the carrying capacity of oil film upon which the pin normally rides.
As the articulated rod continued to pivot, the pin/bushing assembly sheared the set screws that normally hold the bushing in place.
Finally, a lack of lubrication between the pin and bushing and/or the bushing and master rod caused heat stress (bluing) to occur on the master rod bail.
The inspectors concluded that, at the time of the initial event, Engineering did not adequately assess the impact of the potential hydraulic lock caused by the rocker arm failures.
In fact, the root cause of failure did not address a
hydraulic lock as a potential concern.
The inspectors concluded that the licensee should have assessed the forces imposed on the articulated rod and its bushing as the fuel oil was forced past the piston rings or should have performed a visual inspection of these components.
Finally, the inspectors concluded that the licensee did not have an adequate"'industry experience base upon which to conclude that a hydraulic lock did not occur.
4.2 Work Order Ins ection Guidance Following the rocker arm failures, the licensee developed a work order (WO 621749) to "Inspect, repair and replace parts as required to restore the 6-L rocker arms and associated equipment to service."
The inspectors reviewed the work order and noted that it lacked specific guidance on inspection for potentially damaged components.
Step 4.9 of the instructions required the mechanics to "perform a thorough examination of all areas for damage and cause of failure
.
~
.."
The work order planner explained to the inspectors that the work order relied on the skill and experience of the diesel engine mechanics to determine the extent of required inspection activities.
The inspectors discussed the inspection activities with the planning supervisor, the mechanics, and the diesel engineers.
Through these discussions, the inspectors determined that 'the inspection activities were directed from the field.
This direction came in
l
-12-the form of discussions between the mechanics performing the work, their supervisors, the planner, and the engineers.
The inspectors concluded that, because the inspection plans were not formally documented nor meetings conducted to discuss the plans for the EDG inspection, appropriate management was not involved in, the work order to assure that there was proper consideration all potentially affected EDG components.
Because of the unusual conditions in which the EDG was operated (both intake and exhaust rocker arms failed),
an added degree of planning was needed.
The inspectors reviewed the inspection activities documented in the work order.
The licensee'ad documented an inspection of the turbo charger, the cam lobes, the cylinder liner, the camshaft drive gear, and the push rods.
The licensee did not inspect any components below the piston crown.
The turbo charger was inspected based on a previous failure that the licensee had sustained.
The inspectors concluded that the work order was weak and should have provided detailed guidance on the needed inspection activities.
The inspection plan should have been based on an engineering analysis of potential damage, industry experience of similar failures, maintenance experience, and management guidance.
The inspectors determined that the EDG vendor had been involved in the followup activities related to the rocker arm failures but, apparently, not involved in the actions to restore the EDG to an operable condition.
Because the inspection activities were field-directed, the inspectors concluded that the licensee did not gain the benefit of a group evaluation and that licensee management (in Maintenance, Operations, and Engineering)
had been bypassed in the process.
4,3 Work Order Develo ment Inade uacies The inspectors reviewed the licensee procedures for maintenance instruction writing and work order development (30DP-OAP01 and 30DP-9WP02, respectively)
and discussed work order development with the planning supervisor, the planner, and the gC manager.
The inspectors concluded that the process may not provide assurances that the work order was adequate.
In the work order development process, the work control planner determines the quality classification of the component, the quality classification of the activity, the required retest, and whether a technical review is required.
If the planner determines that the activity is quality related, the process requires an independent review by another work control planner.
Additionally, for quality related activities, the work order requires gC review and
.
approval.
The gC review and approval signifies review of the work order for nonconforming conditions, maintenance inspection activities (such as cleanliness, torquing, etc,),
and the inclusion of appropriate gC hold points, The gC review does not provide a review for technical adequacy.
The inspectors discussed the inclusion of industry and plant experience into the work order.
The planning supervisor stated that, if the planner was experienced on the equipment, then the planner would probably recall previous
-
j
-13-onsite experiences for consideration in the work order process.
If the planner was not experienced on the equipment, then the supervisor expected the planner to conduct some type of previous onsite experience research for consideration.
The planner was not specifically required to factor in industry experience into the planning.
For example, until informed by the inspectors, the planner was unfamiliar with Zion Station's bent connecting rod or with Byron Station's decision to conduct cylinder compression checks following a nearly identical rocker arm failure.
Additionally, Engineering did not routinely provide input into the development of a work order.
The inspectors considered that the planning process could provide an acceptable work package for, preplanned work activities and simple emergent work.
However, in the case of the Unit 3 EDG B rocker arm failure, which involved a significant amount of work on a complex system in a short period, the planning process was not effective.
The inspectors concluded that industry experience (both on-site and off-site) were not being effectively used, that plant engineering and gC were not being effectively used, and that a technical review conducted by the planning department did not provide an independent review for adequacy.
4.4 A
arent Violation The inspectors concluded that the work order providing instructions for the repair of the rocker arm did not prescribe instructions appropriate for the circumstances'he instructions did not include either quantitative or qualitative acceptance criteria for determining that the EDG inspection had been satisfactorily accomplished.
This is an apparent violation (530/9416-02).
TESTING FOLLOWING ROCKER ARM FAILURES The inspectors reviewed the postmaintenance testing performed on Unit 3 EDG B
following the rocker arm failures.
Additionally, the inspectors reviewed several attempts between July and October 1993 where the licensee had attempted, and was not successful, in obtaining engine analysis data.
The inspectors concluded that the postmaintenance testing required by the work order was inadequate based on the failure mechanism and the extent of the repairs.
Additionally, the inspectors concluded that, because obtaining engine analysis data was not a requirement, there was little emphasis placed on obtaining this data.
5.1 Postmaintenance Testin The inspectors reviewed the maintenance testing specified by the rocker arm failure troubleshooting and repair work order (WO 621749).
The inspectors found that the work order required the operators to perform a surveillance test in accordance with Procedure 43ST-3DG02.
This surveillance procedure was intended to demonstrate operability in accordance with Technical Specifications 3.8. 1. 1 and 3.8. 1.2.
The portion of the procedure used as the postmaintenance test verified that the diesel would start and would attain
l
~ (
-
l
-14-rated voltage and frequency within 10 seconds.
The inspectors concluded that this was an insufficient retest for the failure mode and the extent of the corrective maintenance history.
The inspectors reviewed the licensee's procedure for postmaintenance retest development (30DP-9WP04).
The procedure described that the postmaintenance test should provide reasonable assurance that:
(1) the original problem was corrected, (2) the original problem was not masking another problem, (3) no new problems were created by the maintenance, and (4) plant equipment configuration was not altered.
I The inspectors reviewed the work order to asses~
the extent of the corrective maintenance.
The work order included extensive maintenance,to Cylinder 6L, including removal and resetting of the cylinder head, replacement of the intake and exhaust rocker arms (with one of a different design)
along with
, their bushings, replacement of a push rod, replacement of a crosshead, replacement of the tappet adjusting screws, and valve train adjustment.
The inspectors concluded that the extent of the cylinder head disassembly and reinstallation could have introduced maintenance problems, such as a
misalignment or a cylinder head gasket leak.
Additionally, the replacement and adjustment of valve train components with new components of a different design could have caused valve sealing or other unforeseen problems.
Based on the extent of the maintenance, the inspectors concluded that the retest specified by the work order did not provide reasonable assurance that no new problems were created by the maintenance.
The. inspectors reviewed the facts pertinent to the retest development known to the licensee at the time of the emergency repairs in July 1993.
The diesel had operated for several hours after the rocker arms had failed.
The Cylinder 6L fuel injector continued to pump fuel into the cylinder and approximately 74 gallons ended up in the lube oil sump.
The fuel oil must have been forced past the piston rings into the sump.
Because the fuel oil flow could have damaged the piston compression rings, the inspectors concluded that the licensee should have conducted a compression check of the cylinder.
5.2 A
arent Violation Based on the extent of the maintenance and the failure mechanism, the inspectors concluded that the licensee should have specified a more thorough postmaintenance test and acceptance criteria to include a peak pressure or compression pressure check.
The inspectors concluded that adequate testing to demonstrate that the EDG would perform satisfactorily in service was not identified in that a compression test of Cylinder 6L was not specified following the repairs performed on the EDG in July 1993.
This is an apparent violation (530/9416-03),
5,3 Hissed Testin
ortunities Following the repairs to Cylinder 6L, the licensee missed several opportunities to discover low peak pressure.
These missed opportunities can
I I
-15-be partially attributed to the low priority placed on obtaining engine analysis data.
The first opportunity occurred during the actual repair and testing of the diesel following the rocker arm failures.
A diesel engineer obtained vibration and ultrasonic data during the retest; however, the engineer did not obtain the pressure vs. time information that could have revealed whether there were any compression problems with the cylinder.
Additionally, the inspectors noted that neither the work instructions nor the postmaintenance test required any type of engine analysis.
The second opportunity occurred during the August 1993 monthly surveillance test.
Although the engine analysis equipment operated erratically and stopped performing while taking data on Cylinder 6L, a review of the data in October 1993 indicated that the cylinder peak pressure was significantly lower than normal.
The third opportunity occurred during September 1993 when scheduling conflicts prevented Engineering from performing engine analysis during the monthly operability surveillance test.
The inspectors concluded that, because engine analysis data was not required by the postmaintenance retest nor by a midcycle requirement, the licensee placed little emphasis on obtaining the data and subsequently delayed the identification of a condition adverse to quality.
REVIEW OF LOW PEAK PRESSURE HEASURENENTS On October 20, 1993, system engineers completed the engine analysis testing on Unit 3 EDG B.
During this testing, they observed a low peak firing pressure in Cylinder 6L.
The inspectors concluded that while the immediate evaluation demonstrated a reasonable expectation that Unit 3 EDG B was operable, the followup engineering evaluation to confirm the initial operability determination was not thorough and the EDG operability evaluation was based on a faulty assumption.
6. 1 Immediate Evaluation of Low Peak Pressure The licensee initiated a Condition Report/Disposition Request (CRDR 3-3-0430)
when low peak pressure was discovered in Cylinder 6L on October 20, 1993.
The vendor manual acceptance criteria regarding peak pressure required all cylinders to be within 160 psig of the average of all cylinders.
The measured peak pressure for Cylinder 6L was 315 psig lower than the average.
The svstem engineers performed an "immediate evaluation" of diesel generator operability based on the 1ow peak pressure.
After contacting the vendor, Engineering personnel assessed the possible causes of the low peak pressure and its affect on system reliability and concluded that the condition in the cylinder was not expected to degrade rapidly.
The evaluation concluded that the condition of Cylinder 6L did not represent an immediate operability concer i
'
!
-16-The inspectors reviewed the evaluation submitted on October 20.
The evaluation concluded that there were four possible causes of the low peak pres'sure.
These were:
(1) fuel delivery problems, (2) improperly seated valves, (3) poor piston ring sealing, and (4)
a distorted articulated rod.
The licensee rejected fuel delivery problems as the cause because of the consistent cylinder performance.
The engineers recognized that improperly seated valves and poor piston ring sealing (blowby) could cause low peak pressure.
They determined that either of these conditions could cause additional wear and could continue to, degrade to the point where the cylinder might not fire.
Their evaluation referenced an evaluation from Cooper Energy Services which concluded that an EDG could perform its safety-related function with one cylinder not firing.
Therefore, they concluded that these conditions did not represent an immediate operability concern.
The system engineers recognized in their evaluation that the fourth possibility, a distorted articulated rod, could have been caused by the July rocker arm failures.
They postulated that, because the engine ran with fuel in the combustion space for approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />, the articulated rod was subject to increased stress from having a noncompressible fluid in the combustion space.
The vendor alerted the engineers to a Zion Station experience where a connecting rod bent due to water from a cracked head in the combustion space.
The Zion Station diesel operated for several years with the bent rod prior to its identification.
The vendor concluded that a distorted articulated rod was the most likely cause of the low peak pressure.
Based on a lack of significant vibration and a lack of ultrasonic indications in Cylinder 6L and with Zion's experience, the licensee concluded that a bent articulated rod did not represent an immediate operability concern.
The inspectors concluded that the 1'icensee's initial evaluation was appropriate at that time in that it provided a reasonable expectation that the EDG was operable.
However, the licensee failed to aggressively pursue the root cause of the low compression reading.
6.2 Followu Review of Low Peak Pressure The inspectors reviewed the analysis performed by the system engineers following the discovery of the low peak pressure.
The analysis was performed on a continuing basis from October 20 to December 21, 1993, but, was not documented or reviewed by licensee management until December 21.
The inspectors found several weaknesses in the technical review.
6'. 1 Bend Direction of the Articulated Rod Shortly after October 20, 1993; the licensee estimated the amount of articulated rod bending required to produce the low peak pressure.
The licensee estimated that the Cylinder 6L articulated rod would have to be shortened by approximately 1/2 inch to reduce peak pressure by the amount observed.
In Harch 1994, the articulated rod was measured to be approximately 3/8 inch shorter than normal, which was close to the October 1993 estimat o sf i
I l
-17-The licensee performed an assessment of the possibility that the articulated rod could have bent 1/2 inch and the effects of the distortion.
The articulated rod was constructed from carbon steel with a cross section that somewhat resembled a rounded I-beam (see Figure 2).
Because a classic I-beam is more resistant to bending in a plane containing the web ("strong" direction) than in a plane perpendicular to the web ("weak" direction), the engineers assumed that the articulated rod would bend in the "weak" direction.
Based on this assumption, the engineers concluded that, if the articulated rod had been distorted in the
"weak" direction, the piston would have become misaligned in the cylinder, resulting in cylinder damage which would have been apparent in the July cylinder inspection and would have resulted in excessive noise.
Based on this conclusion, the engineers determined that having a
distorted articulated rod was less likely.
The engineers stated that the vendor 'concurred with this conclusion.
There was no inspection performed to verify whether the articulated rod was straight or distorted.
The inspectors determined that a visual inspection of the articulated rod required the removal. of an inspection panel on the side of the engine and could be performed in approximately 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.
The inspectors noted that the engineers did not consider that, the articulated rod had been distorted in the "strong" direction.
The inspectors calculated that a 1/2-inch reduction in articulated rod length due entirely to bending would shift the centroid more than 2 inches from its original position, Since the centroid displacement was actually less than 2 inches, the inspectors concluded that the distortion of the articulated rod was due to a combination of bending and compression.
When the engineers predicted the bend direction, they failed to recognize that the piston and the master rod would resist articulated rod bending in the
"weak" direction.
Bending forces in the "weak" direction would be resisted by the piston and cylinder liner because the bending would tend to cock the piston relative to the liner.
Additionally, bending forces in the
"weak" direction would be resisted at the bottom of the rod because the bending would tend to distort the master rod bail and articulated rod pin.
To bend in the
, rod's
"weak" direction, the articulated rod would have had to distort at three points.
In light of the above, it would be reasonable to expect that a more likely bend direction would be in the "strong" direction.
In this direction, bending would not cock the piston (the piston wrist pin wou~d compensate for the bend)
nor would it distort the master rod bail and pin.
As a result, the articulated rod would have to distort at only one point.
The inspectors identified that the engineers did not base their assumption regarding the direction of rod bending on calculations, estimates, or operating experience.
The inspectors concluded that, had either calculations or operating experience been applied, the licensee would have determined that a bent articulated rod was likel [
,
I I
-18-Additionally, the engineers failed to follow up on industry information.
Although they were aware of bent connecting rods at the Zion Station which occurred in March 1988 and November 1989, they did not know which direction the rods had bent.
The inspector contacted the licensee's diesel generator system engineer and was informed that the Zion connecting rod had bent in the
"strong" direction - contrary to the Palo Verde engineers'ssumption.
Additionally, the engineers did not identify any industry events, either Cooper-Hessemer or other vendors, which would support their bending prediction.
The inspectors concluded that the engineers made a poor assumption on the bending direction, did not have any estimates or calculations upon which to base this assumption, and did not effectively use known industry information.
6.2.2 Vendor Technical Manual While discussing the potential problems of continued operation with a bent rod, Unit 3 maintenance personnel identified to the inspectors that the vendor's technical manual addressed a bent rod.
In a section describing inspection of the connecting rod, the technical manual stated that "if bending is suspected, remove the rod from service since bending may continue at an accelerated rate."
The inspectors discussed the section with plant Engineering personnel.
The Engineering personnel were not familiar with the precaution and had not factored the statement into the evaluations performed following the identification of ilow peak pressure.
In response to the discussions with the inspector, plant Engineering personnel contacted the vendor for clarification.
The vendor stated that the technical manual statement was a generic warning so that inexperienced personnel would not reinstall a bent rod.
The vendor also provided analytical data that compared the offset of the rod centroid and the cylinder pressure required to continue bending.
The data concluded that, with the centroid offset by 2 inches, a cylinder pressure of 1670 psi (near normal peak cylinder pressure)
could cause continued bending 'of the connecting rod.
The inspector'oted that, in October 1993, Engineering personnel had calculated that the articulated rod may have been as much as I/2 inch shorter in length.
The inspector calculated this length reduction would correspond to a centroid shift by more than 2 inches (see Section 6.2. 1).
Thus, it appears that, under normal EDG operating conditions and with the amount of distortion of the articulated rod which had been calculated by licensee engineers in October 1993, the licensee engineers should have been concerned that continued operation of the EDG could cause further bending and possibly failure of the EDG.
The inspectors concluded that the vendor manual information regarding a bent connecting rod should haye been evaluated by the system engineers following low peak pressure identification.
Had an evaluation, been performed, the continued operability of Unit 3 EDG 8 should have been questioned and a visual inspection of the articulated rod performe f
]
I
-19-6.2.3 Estimate of Hydraulic Forces On October 20, 1993, the system engineers recognized the possibility that a
hydraulic lock event had occurred in Cylinder 6L following the rocker arm failures.
Subsequently they determined that the low peak pressure correlated to an articulated rod approximately 1/2 inch shorter.
However, during their subsequent review, no calculations or estimates were performed to determine the amount of force required to bend or compress the articulated rod.
In addition, no calculations or estimates were made of the force required to drive the fuel oil past the piston rings.
The licensee did not assess the effects of these forces on load bearing surfaces, such as the articulated rod pin and bushing.
As discussed previously, an 'estimate of the forces and a
comparison to the relative strength of, the articulated rod would have resulted in an expectation that the articulated rod could be distorted.
6.3 A
arent Violation The inspectors concluded that the bent articulated rod and the seized articulated rod bushing represented conditions adverse to quality that were not promptly identified and corrected when conditions indicating potential damage were evident.
The apparent causes of this were an improper assumption on the bending direction of the articulated rod, failure to use available industry information, failure to evaluate a vendor manual caution on the use of a bent connecting rod, the lack of formal calculations to support assumptions made, and the failure to,perform inspections when conditions indicated this need:
This is an apparent violation (530/9416-04).
FAILURE TO PERFORH A TIMELY INSPECTION As discussed in Section 6.2, the system engineers had developed an inspection plan to determine the cause of the low peak pressure in Cylinder 6L.
Some inspections were performed during the Unit 3 midcycle outage starting December 9,
1993.
The inspectors discovered that the inspections were only performed after one of the diesel generator system engineers raised a safety concern regarding the continued operability of the EDG in a meeting to discuss the scheduling of the EDG inspections.
The inspectors reviewed the evaluations and communications done between October 20 and December 9 to determine the circumstances surrounding the licensee's failure to perform a timely inspection.
The inspectors found that the licensee had not established a
process for dispositioning operability concerns, that, engineering management did not critically assess the evaluation of the engineers, that engineering management had failed to adequately communicate the necessity of an inspection with plant management, and that the oversight organizations were not involved in the review.
7.1 0 erabilit Evaluation Process Weaknesses The operability evaluation discussed in Section 6. 1 was documented in CRDR 3-3-0430.
The evaluation was performed by the system engineers to support the operations shift supervisor's review of system operability.
The
-20-evaluation was not specifically required by the CRDR process, nor was 'any followup review.
In the case of the low peak pressure on Cylinder 6L, the CRDR did not establish any immediate actions or reviews.
The CRDR did require that an equipment root cause of failure analysis be performed within 90 days.
The CRDR disposition was signed by the Director of Site Technical Support.
The inspectors found that, while no formal expectations had been established for a followup review of the low peak pressure, the system engineers intended to perform an investigation.
The system engineers conducted a review and drafted an inspection plan.
They apprised engineering management of their inspection plan and received Engineering management's concurrence that the inspection plan should be implemented during an EDG outage prior to the unit's midcycle outage in December 1993.
However, Engineering management did not communicate their desire to inspect the EDG to Unit 3 plant management.
Additionally, during this period, Unit 3 plant management had no concerns regarding the operability of Unit 3 EDG B.
As a result, no action was taken for a month and a half.
The inspectors concluded that the lack of a formal process for performing reviews of degraded conditions to determine operability was a significant contributing factor in the failure to perform a more timely inspection of the EDG.
NRC Generic, Letter 91-18 provides guidance to licensees regarding the performance of operability determinations.
This guidance allows licensees to make an initial assessment of operability based on engineering judgement and allows licensees to conclude a system is operable based on a reasonable expectation that the subsequent determination process will support the expectation.
The guidance also states that initial determinations regarding operability should be revised, as appropriate, as new or additional information becomes available.
The inspectors have noted in previous NRC Inspection Reports (50-528/94-02; 50-529/94-02; 50-530/94-02 and 50-528/93-12; 50-529/94-12; 50-530/94-12) that the licensee had not implemented the guidance provided in Generic Letter 91-18 into plant procedures.
The licensee did not have a formal process which defines the expectations for initial operability decisions based on engineering judgement.
Additionally, the licensee did not have a process to follow up the initial operability assessments with an in-depth review which conclusively establishes operability in a timely manner.
7.2 Lack of Mana ement Assessment of En ineerin Conclusions Mechanical engineering support for the diesel generators was provided by two engineers.
Both individuals had several years of diesel experience and were considered by the Engineering and unit management to be their'iesel
"experts."
One of the diesel engineers was the chairman of the Cooper-Bessemer owners group technical committee.
The other engineer previously instructed students in diesel engines, The inspectors concluded that both engineers are well qualified.
- In addition, the inspectors found that the
'
f I
I I
-21-engineers displayed a high degree of involvement in the routine operation and testing of the EDGs.
During interviews with Engineering and Unit 3 management, the inspectors found that all individuals expressed a high degree of respect for the performance of the system engineers.
However, the inspectors also concluded that the licensee's willingness to rely on these engineers as "experts" contributed to the failure to identify the damage.
The inspectors found that there was little evidence that management,had performed any critical assessment of the technical conclusions reached by the engineers.
7.3 Lack of Mana ement Oversi ht and Su ort On December 3,
1993, the system engineer and immediate supervisor set up a
meeting with the Unit 3 outage and maintenance managers to discuss the performance of testing on Unit 3 EDG B.
At the time, Unit 3 was in the process of draining the reactor coolant system to enter midloop operations.
The licensee had recognized this as a critical period of plant operations, necessitating the availability of critical safety systems, including both EDGs.
The engineering supervisor presented the inspection plan to the outage manager stating that the work would be done to support a root cause investigation.
The outage manager challenged the necessity of the inspection during the midcycle outage.
The system engineer, recognizing that the inspection was being challenged, stated that he had a safety concern regarding the continued operability of the EDG.
Specifically, he stated that the low peak pressure could be a result of a bent articulated rod and that it could impact the long term operability of the diesel.
The outage manager and the maintenance manager recognized the significance of.the system engineer's statements and provided appropriate support to the system engineer to establish an inspection during the midcycle outage.
The inspectors interviewed all of the participants in the December 3 meeting and reviewed, the circumstances which led to the system engineer raising the issue.
The inspectors found that Engineering management had not provided adequate oversight and support of the system engineer's efforts.
As previously discussed, the system engineers had proposed an inspection plan in early November 1993 and had discussed having the inspection performed while Unit 3 was operating during an EDG LCO outage.
The system engineer received concurrence from his immediate supervisor and the department manager that the inspection was necessary.
Engineering management needed to obtain Unit 3 management support to perform the inspection.
The inspectors found that the Engineering supervisors and managers with oversight of the EDGs had not communicated this issue to Unit 3 plant management.
During this period, the two immediate supervisors were intermittently involved in the Unit 3 EDG B
issue, One supervisor was involved in periodic Station'Technical Advisor training and the other was absent due to illness.
In early December 1993, one of the supervisors responsible for the EDGs'as reassigned and a replacement was selected.
The system engineer expressed his
'
,
e
-22-frustration to the new supervisor regarding the failure to establish an EDG LCO outage and his desire to include the inspection in the midcycle outage.
The subsequent followup resulted in the December 3 meeting.
I The inspectors concluded that, by the time the December 3 meeting was held, despite repeated attempts by the system engineer, little support had been provided by his management regarding the resolution of a significant EDG deficiency.
This lack of management support and a formal process to resolve the operability concern essentially left the system engineer as the only proponent of further review of the operability of Unit 3 EDG B.
This placed the system engineer in a position where he felt the only means of issue resolution was to characterize the issue in a meeting as a safety concern.
The inspector concluded that this represented a significant breakdown in management oversight and in the process for reviewing and tracking significant issues.
7.4 Involvement of the ualit Oversi ht Grou s'he inspectors determined that the QA organization was not involved in the review of the initial rocker arm failures and later in the low peak pressure in the Cylinder 6L.
The inspectors noted three missed opportunities.
The first opportunity occurred when the quality organization was neither involved in the inspection planning nor in the postmaintenance testing activities following the rocker arm failures in July 1993 (details in Section 4.3)
The next missed opportunity involved the Unit 3 QC group, which reports to the QA Director.
QC was involved in,the discussions on October 20, 1993, which concluded that Unit 3 EDG B was operable.
However, they were not involved in the subsequent followup of the deficiency nor did they ensure that the issue was reviewed by QA or Engineering.
t The third missed opportunity involved QA's review of the Unit 3 EDG B CRDR.
While the QA organization routinely performs a review of all new CRDRs initiated to determine the extent of their involvement and did review CRDR 3-3-0430, they did not select the CRDR for further review.
The QA organization concluded that the CRDR problem description did not indicate a
significant problem.
The problem description stated:
"A mid-cycle engine analysis was performed on the Unit 3 "B" erne'rgency diesel engine on 10/20/93.
The engine analysis was performed by plant engineering.
All engine operating parameters were normal with the exception of lower than expected peak pressures in the 86 left cylinder."
The inspectors agreed that the CRDR description did not portray a significant concern.
The CRDR description did not reflect that the difference between the Cylinder 6L peak pressure and the average peak pressure exceeded vendor tolerances.
In addition, the description did not identify that this was the
~ I
,
I
-23-same cylinder which had experienced the rocker arm failure.
However, the inspectors noted that the immediate evaluation of the CROR, discussed in Section 6. 1, was included in the initial distribution which would have been sent to gA.
The immediate evaluation provided adequate indication of a significant problem.
INSPECTIONS DURING BIOCYCLE OUTAGE On December 9, the licensee implemented an inspection plan for Unit 3 EDG B.
The inspectors reviewed the inspections performed and'he conclusions made
'ased on the inspection.
The inspectors concluded that the inspections,did not conclusively support the licensee's subsequent determination that the cause of the low peak pressure was poor piston ring sealing.
Additionally, the inspectors concluded that the licensee relied too heavily on indirect inspection and'ngineering analysis when a direct inspection could have been performed.
'. l Ins ection Plan Im lementation The first inspection plan was developed in early November 1993.
The inspection to measure for a bent articulated rod was the second step in the plan.
During the meeting on December 3, the inspection plan was substantially rewritten.
The licensee developed an incremental plan to inspect for the cause of low'peak pressure.
The inspections were conducted while Unit 3 was in reduced reactor coolant system inventory with steam generator nozzle dams installed.
Considering this, the licensee developed an inspection plan which would minimize the amount of time that the diesel was unavailable.
The first step of the inspection plan was to verify the low peak pressure indications.
On December 9, the licensee removed the test cock assembly to verify that the indicated pressure was accurate and then reperformed the engine analysis on Cylinder 6L.
Engine analysis indicated that the peak pressure was still approximately 300 psi lower than the other cylinders and that the compression pressure was approximately 175 psi low.
The licensee concluded that the low peak pressure was caused by the low compression pressure.
The licensee met to discuss the next phase of the inspection.
Some of the maintenance personnel suggested removing an inspection port on the side of the crankcase and visually inspecting the articulated rod assembly.
However, the licensee decided to verify what it thought to be the most likely cause of the low peak pressure poor piston ring sealing by performing a visual inspection of the cylinder liner area with a video probe.
Because the licensee had incorrectly assumed an incorrect bending direction, it assumed that a bent articulated rod would show signs of cylinder liner degradation.
The licensee assumed that a lack of cylinder degradation would support a
conclusion that the rod was not bent.
On December 12, the licensee inserted a video probe into the cylinder through the injector nozzle.
The probe revealed some chatter indications in the upper
areas of the cylinder.
Additionally, the licensee observed some indications of blowby" in the upper ring reversal area.
The ring reversal area is indicated by circular carbon bands that develop where the piston compression rings reverse direction.
Finally, the licensee observed a glossy sheen of unburned fuel on the cylinder which was caused by poor combustion.
The engineers concluded that the low peak pressure was caused by poor piston ring sealing and that the condition was not an operability concern.
Based on this determination, the licensee did not perform the next step in the inspection plan, which involved removing an inspection port to view the articulated rod.
The inspectors determined that such an inspection could have been performed within a 1-hour period and, with proper planning, could have positively identified the distorted articulated rod.
8.2 Ins ector's Review of Ins ections The inspectors reviewed the analysis performed by the system engineers of the cylinder condition.
In addition the inspectors observed some photographs of the indications observed by the licensee.
The inspectors determined that the analysis performed by the system engineers did not conclusively indicate that poor piston ring sealing was the only cause of the reduced compression pressure.
The inspectors determined that the licensee had not considered that the pre-existing ring reversal indications may have been substantially washed out by fuel oil during the July rocker arm failure.
Following the rocker arm failure, a substantial amount of fuel oil was forced past the piston rings.
Fuel oil, acting as a solvent, could have cleaned the carbon bands which make up the ring reversal marks, resulting in the
"washed out" indication that would mimic blowby indications.
Additionally, the system engineers should have considered that a bent articulated rod could have substantially masked old ring reversal marks. and would create new ones.
Since the bent rod was shorter, the piston rings reversed direction at a new location on the cylinder liner.
Because the EDG had a short operating history since the rocker arm failures, the ring reversal indications may not have become well established.
This could have been confused with the
"washed out" ring reversal marks caused by piston ring blowby.
Finally, the system engineers did not consider that a combination of effects could have occurred.
When the piston was inspected in Harch 1994, the licensee found that the upper compression ring was stuck.
This would have allowed expansion gasses past the first, and most prominent, ring reversal mark and provided the
"washed out" appearance.
However, it is not uncommon for a cylinder to achieve full compression with one or two stuck rings.
Additionally, carbon build-up due to the lower peak pressure and incomplete combustion (engineers noted that the cylinder was "knocking" during the
l
,
I
)
-25-October 1993 run) could have caused a ring to stick.
In effect, the stuck compression ring and, therefore, the blowby indications observed in December 1993, may have been the direct result of the bent articulated rod.
8.3 Evaluation Followin Midc cle Outa e
On December 21, 1993, engineering completed an "interim evaluation" of the apparent cause of the low peak pressure observed in Cylinder 6L.
The inspector found the evaluation was exceptionally detailed.
The evaluation concluded that the apparent cause of the low compression pressure was poor sealing of one or more piston rings.
I, The evaluation supported the conclusion of poor sealing primarily with the visual inspection of the cylinder liner, which indicated that there was circumferential scuffing in the ring reversal area and evidence of blowby past the rings.
Engineering discounted a bent articulated rod primarily from the absence of cylinder liner damage which they would have expected to see.
Based on this review, the evaluation concluded that poor piston ring sealing was the apparent cause but that additional inspections would be necessary to confirm the apparent cause.
The planned inspections included a cylinder pressure drop test, a measurement of the articulated rod length, an inspection of the intake and exhaust valves, and an inspection of the cylinder.
The evaluation concluded that the inspections could be delayed until the refueling outage.
The evaluation provided the following reasons to delay the inspections to the refueling outage:
(1) the "economic expense" associated with the potential extension of the midcycle outage, and (2) the EDG unavailability associated with the additional inspections.
The evaluation concluded that the EDG was operable based on the successful completion of surveillance testing, the evaluation that the EDG could operate
'ith one dead cylinder, and a determination that further degradation would not challenge the operability of the EDG.
The inspectors concluded that the licensee's evaluation relied heavily on engineering analysis and indirect inspections.
As discussed previously, the engineering evaluations included inaccurate assumptions.
The licensee did not adequately consider the benefits of a direct inspection of the articulated rod.
A direct inspection could have been performed i i a timely manner without a significant impact on plant schedules and would have provided conclusive evidence regarding the condition of the articulated rod.
SUMMARY The inspectors concluded that the licensee failed to determine that Unit 3 EDG B was inoperable from July 1993 to March 1994 and missed several opportunities to identify and correct the EDG damage which occurred after Unit 3 EDG B was run for 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> after intake and exhaust rocker arms for a
I'J f
l
l l
]
-26-cylinder failed in July 1993.
These opportunities occurred immediately after the July 1993 rocker arm failures, after repairs were made to the EDG, after low compression readings were identified in October 1993, and after the system engineer expressed his concerns in a meeting in December 1993.
Specifically:
~
The initial inspection of the EDG immediately after the July 1993 rocker arm failures was not thorough and was not performed with formal guidance.
The postmaintenance testing of the EDG was not complete in that the testing did not include any compression checks of the affected cylinder even though fuel oil was forced past the piston rings and the cylinder head was removed and reset.
~
Routine engine analysis during monthly surveillance testing was not relied on or performed until October 1993.
In October 1993, when low cylinder compression was identified, a faulty engineering evaluation determined that a bent articulated rod was not likely.
A formal engineering evaluation was not performed which could have identified the significant forces on the connecting rod, and bearings would have occurred when the rocker arms failed.
In December 1993, based on an inability to get the EDG inspections scheduled, the system engineer felt compelled to voice his concerns over EDG operability in a meeting.
The apparent reasons for the difficulties in getting the EDG problem addressed were:
Failure to perform a formal evaluation of the potential components that were possibly damaged when the rocker arms failed.
Lack of Engineering involvement in specifying posttesting requirements and acceptance criteria.
Low priority given to the followup of the EDG rocker arm failures and abnormal EDG readings.
No formal process existed for performing reviews of degraded conditions for operability considerations.
Faulty engineering evaluation of low compression readings, without adequate followup of industry information.
Ineffective vendor, gA, and management oversight of the technical resolution of the EDG issues.
There was an apparent over-reliance on the system engineers to provide evaluations without a critical review or questionin t I
C l
-27-
~
Simple inspections were not adequately considered and excessive reliance was placed on the engineering'valuation as the basis for EDG operability.
~
There appeared to be an over-emphasis to maintain the EDGs in an operable status over considerations to promptly resolve the root cause of the EDG abnormal indications.
~
The licensee's nonconformance process did not assure that the concerns were adequately addresse I
ATTACHNENT 1
Persons Contacted F 1 Arizona Public Service Com an APS R. Adney, Plant Manager, Unit 3
- J, Bailey, Assistant Vice President, Nuclear Engineering Administration H. Berryman, Supervisor, Unit 3 Mechanical Maintenance D. Brunson, Senior Mechanical Planner, Unit 3
- W. Conway, Executive Vice-President, Nuclear T. Engbring, Principle Discipline Engineer J.
Evans, Supervisor, Unit 3 Planning
- A. Fakhar, Manager, Mechanical Group, Site Technical Support
- D. Garchow, Director, Site Technical Support D. Hettick, Supervisor, Station Operating Experience Department S.
Hofmann, Engineer, Balance of Plant
- J. Irwin, Senior Engineer, Nuclear Regulatory Affairs
- R. Kerwin, Manager, Unit 3 Haintenance
- A. Krainik, Hanager, Nuclear Regulatory Affairs
- J.
Levine, Vice President, Nuclear Production
- D. Hauldin, Director, Site Maintenance and Modifications P. Haynard, Supervisor, Balance of Plant G. Overbeck, Assistant to the Vice President, Nuclear Production R. Prabhakar, Manager, Independent Safety and guality Engineering T. Radtke, Supervisor, Unit 3 Operations
- H. Riley, Supervisor, Balance of Plant
- C. Russo, Manager, guality Control C.
Seaman, Director, guality Assurance and Control
- E. Simpson, Vice President, Nuclear Support
- N. Thibodaux, Performance Engineer W. Winn, Foreman, Unit 3 Mechanical Maintenance 1.2 Others
- J.
Draper, Site Representative, Southern California Edison
- F. Gowers, Site Representative, El Paso Electric 1.3 NRC Personnel
- H. Wong, Chief, Project Branch F
- J.
Ganiere, Intern, Nuclear Reactor Regulation
- Denotes personnel in attendance at the exit meeting held with Messrs.
Johnston and Freeman on April 22, 1994.
EXIT MEETING An exit meeting was conducted on April 22, 1994.
During this meeting, the inspectors summarized the scope and findings of the report.
.The licensee acknowledged the inspection findings documented in this report.
The licensee did not identify as proprietary any information provided to, or reviewed by, the inspector O~
)AGISTER ROO CAP
1.
2.
3.
5.
6.
7.
8.
9.
10.
11.
12.
Pi,ston Pi.n Oil Passage Dowel (3)
Washer (4)
Bolt Lock (4)
Pin Bolt (4)
Drake Nut (4)
Bolt Lock (2)
Washer (2)
Oil Passage Bushing Dowel Pi.n (4)
13.
Bearing Shell, Top 14.
Dowel (2)
15.
Rod Cap 16.
Locknut (4)
17.
Stud (4)
18.
Bearing Shell Bottom 19.
Art. Rod Pin 20.
Rod Pin Bolt 21.
Bearing Cap (Hut Tightening Sequence)
22.
Washer Figure I
Articulated Rod / ~iaster Rod Assembly
I (
Figure 2 - Cross Sectional View of the Articulated Rod (Approximate)
f l
l