IR 05000528/1987045
| ML17303B008 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 03/22/1988 |
| From: | Paul Prescott, Stone J Office of Nuclear Reactor Regulation |
| To: | |
| Shared Package | |
| ML17303B006 | List: |
| References | |
| 50-528-87-45, 50-529-87-44, 50-530-87-43, NUDOCS 8804130098 | |
| Download: ML17303B008 (15) | |
Text
UNITED STATES NUCLEAR REGULATORY COMMISSION REGION V
8804i30098 880405 PDR ADOCK 05000528
PDR Report No.:
License No.:
Licensee:
Facility Name:
Inspection Of:
50-528/87-45, 50-529/87-44 and 50-530/87-43 NPF-51 CPPR-143 Arizona Nuclear Power Project Post Office Box 52034 Phoenix, Arizona 85072-2034 Palo Verde Nuclear Generating Station - Unit 3 Palo Verde Site, Mintersburg, AZ Inspected By:
P. Prescott, Team Leader, Program Development and Date Reactive Inspection Section (PDRIS)
J.
Conway, PDRIS E, Tomlinson, NRR Approved By:
> P3 gg-J. Stone, Acting Chief, Vendor Inspection Branch ate Summary:
Ins ection on November 9-13 1987 Re ort No. 50-529 87-45 50-529 87-44 and 50-530/87-43 A~Id:
A d
p ii p
d P
P d
Nuclear Generating Station (PVNGS) to assess the implementation of the Arizona Public Service quality assurance program particularly in the areas of 10 CFR 21 reportability requirements, control of purchased materials and services, nonconformances, and corrective actions.
The major inspection effort was focused on the Palo Verde Unit 3 3B emergency diesel generator (EDG), which suffered a catastrophic failure on December 23, 198 I
1.
PALO VERDE UNIT 3 1.1 3B EDG Piston Pin Back round On June 30, 1987, during testing, the 3B EDG experienced a crank case over-pressurization.
The cause of the overpressurization was ignition of combustible products in the crank case which were ignited by excessive temperatures in the
¹9-L cylinder.
Following the over pressurization, the ¹9-L cylinder was disassembled to determine the cause of the excessive temperatures.
Upon inspection, it was determined that the piston pin had overheated and expanded, causing the piston to expand and scuff the cylinder liner.
The piston scuffing caused the cylinder liner to score and overheat, thereby providing the ignition source.
The piston pin overheated initially because of inadequate lubrication between the pin and its mating surfaces in the piston.
The lack of lubrication to the piston pin has been attributed to inadequate clearances between the pin and the piston, improper surface finish on the pin, or a combination of the two. It is possible that either or both of the contributing factors were manufacturing defects.
This is discussed further below.
Discussion During the week of October 5, 1987, the NRC inspectors audited the manufacturing records pertaining to the 3B EDG, at the Cooper Industries (CI) facility in Grove City, PA.
The audit included a tour of CI's manufacturing facility with special emphasis on the processing of piston pins, from their machining and finishing through final inspection.
Piston pins are inspected for conformance to both dimensional and finish specifications.
The finish on the piston pins for KSV engines (the 3B EDG is a model KSV 20T) is inspected electronically and visually.
As a result of the tour, it appeared that all piston pins receive careful handling, and are coated following machining and inspection to protect the finish from physical damage and/or corrosion.
At the conclusion of the audit, the NRC inspectors had not found any evidence that the piston pin that failed in the ¹9-L cylinder had not been handled and inspected in the same way as all other piston pins, nor could the NRC inspectors substantiate that a defective piston pin had been shipped from CI to Palo Verde.
A more logical explanation is that the piston pin was improperly inserted into the piston during reassembly of the 3B EDG at Palo Verde.
There also appears to be a lack of verification by CI that proper assembly procedures were followed on the ¹9-L piston pin.
This could have resulted in improper piston-to-piston-pin clearances.
Another possibility is that the piston pin finish was altered at some point between the time it was shipped from the factory, stored, and installed in the 3B EDG.
During the inspection at Palo Verde, the NRC inspectors reviewed several quality control (gC) installation records associated with the 3B EDG work control package used for the rebuilding of the EDG.
The NRC inspectors also requested all of the gC inspection records in regard to the installation of the master and articulating rods for the ¹9 cylinder and all other cylinders that required new piston/bushing/pin combinations following the December 1986 failure.
During the review of these records, the NRC inspectors noted that there was no reference to a blueing check for the ¹9-L piston pin and bushin Failure of technicians to perform the blueing check could lead to lack of required contact between the piston pin and its bushing, thus causing high localized loading and eventual failure of the components.
CI identified this as a possible root cause of the 89-L cylinder failure in Report gCG-3630 dated July 3, 1987.
During a review of the records, the NRC inspectors also noted that the blueing step in the installation procedures for the lower pin and bushing for the 82 articulating rod was also missing, thus putting the installation and functioning of the associated components in question.
However, in a conference call held on November 17, 1987 between the NRC, CI, and the licensee, CI stated that the articulating rod bushing and pin receive generic blueing checks prior to shipment which would adequately assure their performance.
GI also stated that the blueing check performed in the field is a recommendation, not a
requirement, in the CI Technical Manual.
CI also provided the manufacturing records (gCIP MSV-4-2A81) to the NRC staff for review.
These records reflected the generic blueing process, and no discrepancies were noted.
1.2 3B EDG Hain Bearin
Back round and Discussion During qualification testing of the 3B EDG, following the repair of damage resulting from the December 23, 1986 master rod failure, there were several incidents where the EDG tripped because of high temperature at the 82 main bearing.
Following corrective action, the 82 main bearing was operated successfully at a somewhat higher temperature than that of an adjacent main bearing.
Successful operation was demonstrated by (a) operation in excess of 100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> without tripping and (b)
a temperature profile of the bearing during operation which showed stable temperatures.
To obtain the temperature profile, a thermal trip device (one of two) in the 82 main bearing was removed and replaced with a thermocouple.
The thermocouple, in turn, was connected to a
chart recorder which made an analog record of the bearing temperature as a
function of time.
In a meeting on July 10, 1987 between the licensee, CI, and NRC, it was determined that monitoring the 82 main bearing temperature should be continued in order to establish a trend and thereby be better able to assess the potential for long-term reliability of the 82 main bearing.
Continued monitoring will provide the necessary data to establish a trend for the k'2 main bearing.
As the trend evolves, it will be possible to assess the potential for long-term reliability of the bearing.
To monitor bearing temperature, it is necessary to remove one of the two trip devices installed in the bearing cap and replace it with a thermocouple.
CI was opposed to removing a trip device on a permanent basis.
However, the licensee proposed an alternative solution; that is, install a thermocouple instead of a vendor-supplied trip device and connect the thermocouple output in such a way that it becomes a part of the engine trip system as well as provides temperature data.
Under the licensee's proposal, the EDG would trip on high bearing temperature the same as if the vendor-supplied trip device was installed.
This proposal was discussed with the EDG vendor.
CI subsequently agreed that the alternative high temperature trip proposal would be acceptable in lieu of the vendor supplied trip devic o
Conclusion The staff has reviewed the licensee's proposed thermocouple installation and concluded that it will provide the necessary temperature data without compromising the safety function of engine trip devices.
At the time of this inspection, the licensee had completed the engineering for the installation, but was unable to proceed with the actual work because not all components had been received.
The licensee committed to install the thermocouple/trip at the first outage of Unit 3 for any purpose.
The licensee should document the design, actual installation, and testing of the thermocouple installation.
This document package should be made available on site for staff review.
1.3 3B EDG C linder Head
~Bk d
In late Hay 1987, reassembly of the 3B EDG, following extensive repairs, was completed.
On June 2, 1987, the 3B EDG was operated for about 15 minutes with apparent success.
Following this operation, the EDG was shut down for approximately 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />, at which time an attempt was made to restart the EDG, which was unsuccessful.
It was subsequently determined that the 3B 'EDG had experienced a "hydraulic lock" on the 88-R cylinder which prevented the engine from rotating.
The cause of the hydraulic lock was coolant water entering and filling the 88-R cylinder via a through crack between the cooling water passages and the fire face of the k'8-R cylinder head.
As a consequence of excessive forces that had developed during the hydraulic lock, the 88-R cylinder liner also cracked.
To date, all involved parties (the licensee, CI, and NRC) agree that there is/
was a crack in the 88-R cylinder head, that the water in the 88-R cylinder was the cause of the EDG failure to start, and that the 88-R cracked cylinder liner was a direct result of trying to start 3B EDG with water in the 88-R cylinder.
To determine the root cause of the crack in the 88-R cylinder head, the licensee elected to have the vendor (CI) examine the outboard side of the fractured surface and contracted with Scanning Electron Analysis Laboratories, Inc.
(SEAL) to analyze -the inboard side of the fractured surface.
Each party performed their own separate and independent analysis for the fractured surface furnished by the licensee to determine the initial cause, crack propagation and crack origin.
However, on the basis of the analyses of secondary cracking, which is a strong indicator of where the crack originated and the direction in which the crack propagates, the two parties could not agree on the same conclusion.
In their original report, SEAL concluded that the through crack in the 88-R cylinder head initiated at an inclusion in a radius in the cooling water passages and progressed as a function of fatigue to the fire side of the cylinder hea CI, however, concluded that the crack in the 88-R cylinder head originated at the fire face and propagated through to the cooling water passage, and that the cause of the crack was directly related to the December 23, 1986 failure of the 89 master rod.
As a result, CI and Palo Verde agreed that the fracture surface from the outboard side of the cylinder (used by Cooper Bessemer to arrive at its conclusion regarding the head failure), which was the mating half of the fracture SEAL examined, would be sent to SEAL for further examination.
On November 4, 1987, a conference call was held between CI and SEAL and a
consensus was reached.
A description of the failure was hypothesized for the initiation and progression of the crack star ting on the water cooling face.
This description was consistent with all of the findings to date, and the parties participating in the conference call agreed that, most probably, the sequence of the failure was as follows:
When the rod failure in the 89 cylinder occurred on December 23, 1986, water was draining through the area during the time the engine kept running.
Because of the damage to the intake valves in the 89 cylinder, it is believed that the 88 cylinder was providing most of the power that kept the engine turning, so that the fire face of the 88 cylinder remained in compression.
It is hypothesized that there was a lack of water followed by sufficient water in the area between the exhaust valve ports of the 4'8 cylinder head, so that a tensile force developed on the water cooling side.
A thermal stress crack initiating on the water cooling side occurred which arrested.
It is hypothesized that the crack did not break through the fire face at that time or any time before the start of the 18-minute test run on June 2, 1987.
It is believed that the crack broke through the fire face in a location approximately at the midpoint between the two valve ports sometime during that 18-minute test run and then the crack proceeded outward and upward away from the fire face as the run proceeded.
The engine continued to run forcing carbon up through the crack and covering the fracture surface until the run was terminated.
Following this 18-minute run, water seeped into and filled the combustion chamber, resulting in the failure, which occurred at the initiation of the run performed
hours later.
This hypothesis was tested against all of the observations, including the location and orientation of secondary cracking, and no contradictory evidence was found.
Therefore, it was agreed among all of the participants in the conference call of November 4, 1987 that the crack started on the water cooling side of the 88 head during the December 23 incident, but did not break thr'ough the fire face at that time.
During the 18-minute run on June 2, 1987, the crack broke through the fire face near the midpoint between the two valve ports, and continued outward and upward along the valve insert ports.
The NRC inspectors discussed this hypothesis with the licensee and the EDG vendor during various inspections and conference calls, and it was determined that the background information, the knowledge of the individuals involved, and the additional sample all provided the necessary clues needed to arrive at an agreement regarding this failure hypothesi.
PALO VERDE UNITS
2 AND 3 2.1 Cracked In'ector Nozzles Back round and Discussion On August 26, 1987, Houston Light 8 Power Company issued a
CFR 50.55(e)
report concerning the standby diesel generator fuel injection nozzles at the South Texas Project (STP).
To date, there have been two failures of fuel injection nozzles at South Texas.
In both cases, the nozzles failed in such a
manner that significant amounts of diesel fuel leaked from the nozzles into the EDG lube oil sump where the fuel raised the sump level, diluted the lube oil, and eventually caused a drop in lube oil pressure.
Both injection nozzle failures had the potential for causing EDG failure due to inadequate lubrication because of the dilution of the of lube oil.
CI contacted the supplier of the injection nozzles (Bendix Corp.) with respect to the failed injection nozzles at STP.
Based on discussions with Bendix, CI reported that the injection nozzle failures are most likely the result of manufacturing defects (machining and/or heat treating).
CI further stated that the manufacturing defects appear to be limited to one lot of injection nozzles, and that Bendix will replace all suspect injection nozzles.
The licensee has reviewed available records and determined that there are
injection nozzles from the suspect lot at Palo Verde.
The exact location of the suspect injection nozzles is not known.
It is assumed, however, that most of the nozzles are installed in the six EDGs at Palo Verde Units I, 2, and 3.
To date, there have been no injection nozzle failures at Palo Verde.
However, the licensee has agreed to locate and replace all suspect injection nozzles at the earliest opportunity.
In the interim, the licensee has agreed to implement additional EDG monitoring until the suspect injection nozzles have been replaced.
The licensee will monitor EDG operation for specific indications of nozzle failure such as decreased cylinder exhaust temperature, changes in exhaust appearance, engine lube oil level and pressure, and contamination of lube oil with fuel oil.
The preceding are all conditions associated with a failed injection nozzle.
Conclusion Because there have been no injection nozzle failures at Palo Verde and because of the increased EDG monitoring the licensee has agreed to implement, the NRC staff has concluded that EDG operability at Palo Verde is not impaired.
However, the licensee is encouraged to locate and replace all suspect injection nozzles as expeditiously as possible.
2.2 JAAR1 Back round During required surveillance testing of the Unit 2 "A" emergency diesel generator (EDG), it was noted that the EDG did not reach the full rated speed of 600 rpm (equal to 60 Hz) when operated with the governor in isochronous mode.
As a result, the licensee issued a
CFR 21 report on July 17, 1987,
which stated that the actual frequency was about 59.5 Hz, or approximately 595 rpm.
The cause of the problem was determined to be an oxide buildup on the contacts of an Agastat relay that is used to energize the EDG governor isochronous circuit under emergency conditions (loss-of-coolant accident, loss of offsite power).
Under emergency conditions, the Agastat relay is energized and a contact is closed to place a fixed resistance in the governor control circuit.
This fixed resistance circuitry provides a reference signal against which the engire speed signal is compared.
When the two signals (reference and engine speed)
are mismatched, the engine speed will increase (or decrease)
until the signals are equal.
The oxidation on the relay contacts represents additional resistance in the circuit.
This additional resistance causes the fixed reference signal to be reduced with the result that the engine speed necessary to generate a matching signal is less than the design speed of 600 rpm.
Consequently, the steady-state engine speed and associated frequency are lowered (595 rpm and 59.5 Hz, respectively).
Discussion It was determined that the root cause of the problem was the use of the wrong relay for this governor circuit application.
The Agastat relay used is designed to carry substantially more current than it does in this application.
With more current, the small arc that would be drawn each time the relay was operated would be adequate to remove oxidation from the contact surfaces.
The current available in the governor circuit, however, is much too low to effect any contact surface cleaning on relay operation.
Consequently, the oxidation builds up and adds resistance to the circuit.
The licensee has proposed to replace the Agastat relay in question with a different type of relay from another relay vendor.
The replacement relay will be an Allen Bradley Type R (700 DC-R220-Zl) relay.
This replacement relay has bifurcated contacts and is intended for applications such as the EDG governor control circuit, Bifurcated contacts have a "wiping" action and are self cleaning with each actuation.
Consequently, oxidation buildup will not be a
problem.
The proposed replacement Allen Bradley relays are qualified for the intended application.
The licensee's proposed modification was discussed during telephone conversa-tions involving the licensee, the EDG vendor (CI), and the NRC staff on November 17, 1987.
As a result of this telephone conversation, the vendor verbally accepted the licensee's proposed modification.
Conclusion On the basis of discussions with the licensee and the EDG vendor, the staff has concluded that the licensee's proposal to replace an existing Agastat relay in the EDG governor control circuit with an Allen Bradley Type R relay, when implemented, should eliminate the problem of oxidized relay contact.
The licensee also proposes to replace two additional Agastat relays in similar applications with the Allen Bradley Type R relay.
Fastener s - Procurement Some NRC procurement inspections over the past year have included the collection and testing of a small sample of fasteners (i.e., bolts, studs, nuts).
Of 14 bolts obtained from Palo Verde, 5 did not meet the specification requirements for chemical properties.
The five nonconforming bolts were all marked as Society of Automotive Engineers (SAE) Grade 8, but were actually found to be SAE Grade 8.2.
In addition, two bolts obtained from Rancho Seco failed to meet the requirements for mechanical and chemical properties.
The two bolts had a grade marking of B7 and a manufacturer's marking of CB.
The manufacturer of these two bolts has been identified as Custom Bolt (CB), which is located in Phoenix, Arizona.
The purpose of this inspection was to determine if any fasteners for the Palo Verde nuclear site were purchased from CB and also to review Arizona Public Service's (APS's) overall procurement activities relating to fasteners.
The structure for the procurement activities in the quality assurance (QA)
organization has a Manager, Procurement QA reporting directly to the Director QA/QC.
Three department supervisors report to the Manager, Procurement QA.
The departments are Receiving, Procurement Engineering, and Vendors.
APS utilizes five procurement levels for purchasing four categories of fastener material.
The levels are designated as follows:
I - specification written by APS, II - original equipment manufacturer (OEM), III - verification of item, IV - catalog item, and V - commercial grade.
For levels I, II, and IV, the vendor is on the APS approved vendor list (AVL), but the vendor is not on the AVL for levels III and V.
A visual inspection is performed on items ordered at any of the five levels, and hardness testing is performed on SAE fasteners greater than 1/2 inch in diameter in accordance with the requirements of the material specification.
APS recently purchased an alloy analyzer to perform checks to verify compliance with chemical requirements, but to date, the unit has not been placed in operation.
Specification ECE-ZZ-A098, Revision 1, dated September 10, 1987, was written to establish consistent bolting procurement guidelines.
American Society of Mechanical Engineers (ASME) material fasteners are purchased to the requirements of Section III of the ASME Boiler and Pressure Vessel Code, which requi res that the supplier/manufacturer have a
QA program that meets the requirements of Section NCA 3800.
In addition, a hardness test (sampling plan per MIL-STD-105)
is performed on material greater than 1 inch in diameter and ordered to American Society of Testing Materials (ASTM) Specifications A325, A354, A490, and A540 for bolts and studs, and A540 and A563 for nuts.
The requirements of 10 CFR 21 are imposed on suppliers/manufacturers of both ASME and ASTM material fasteners.
SAE fastener material is procured as "quality related - commercial grade" from the manufacturer/supplier on the basis of specifications set forth in the manufacturer's catalog or industry specification (e.g.,
SAE J429).
This material is dedicated and undergoes a physical configuration check (visual and threaded pitch gage),
a hardness test (bolts greater than 1/2 inch in diameter),
and a plating check (visual).
Specialty fastener material (i.e., identified by a vendor part number or a drawing) for safety-related applications is ordered from the OEM who is on the APS AVL.
Fasteners for important-to-safety applications are ordered from the
OEM as "quality related - commercial grade."
The inspector reviewed the AVL, Revision 87-11, dated October 29, 1987, and noted that APS had audited five suppliers/manufacturers of fasteners since June 1986.
The vendors included AKG Engineering, Hardware Specialty, Texas Bolt, Allied Nut 5 Bolt, and Ingersoll-Rand (an OEti).
A manufacturer known as Custom Bolt (CB) was not on the AVL, and the inspector was told in discussions with APS QA management personnel that CB was never on the APS AVL or the Bechtel evaluated supplier list.
The qualification records fo} the three persons who performed the audits also were reviewed.
All three individuals were qualified in accordance with the require-ments of American National Standards Institute ANSI N45.2.23.
Purchase Order (PO)
No. 60137434, dated January 19, 1987, to Texas Bolt for ASME Section III fasteners was reviewed.
The PO contained an attachment entitled
"MMIS Quality Assurance Clause,"
which documented a number of QA requirements.
The major areas included certificate of conformance, QA/QC documentation, subtier vendor QA program requirements,
CFR 21 reportability, packaging/handling/shipping, certified material test report, vendor access, and nonconformance reporting.
The PO was signed and dated by a QA representative.
APS reviewed its QA record files for any procurement activities at Palo Verde with CB from the start of construction to the present and determined that it had not done any business with CB from 1984 to the present.
Four POs (two in 1977 and two in 1978) from APS to CB specified quality class
"S" fasteners and referenced Section 2, "General Conditions for the Palo Verde Facility," which specified that high-strength bolts were to be ASTM A325 and all other bolts were to be ASTM A307.
A review of approximately 20 field material requisitions (FMRs) indicated that Bechtel purchased a large number of fasteners from CB from 1976 through 1982.
Several of the FMRs required the fasteners to be galvanized in accordance with ASTM A-123.
QA requirements were not identified on the FMRs, and the fasteners were ordered to ASTM specifications (e.g.,
A193, A307, A325, and A540).
On the basis of discussions with APS procurement QA personnel and a review of the documents made available to the inspector, it would appear that all the fasteners that CB supplied to the Palo Verde site from 1976 through 1982 were commercial grade and not safety-relate tI
,