IR 05000528/2017008

NRC Special Inspection Report 05000528/2017008, 05000529/2017008 and 05000530/2017008

ML17100A130
Person / Time
Site:	Palo Verde
Issue date:	04/10/2017
From:	Geoffrey Miller NRC/RGN-IV/DRP/RPB-D
To:	Bement R Arizona Public Service Co
Miller G
References
ML16358A676, ML17004A020 IR 2017008
Download: ML17100A130 (30)

Text

UNITED STATES ril 10, 2017

SUBJECT:

PALO VERDE NUCLEAR GENERATING STATION - NRC SPECIAL INSPECTION REPORT 05000528/2017008, 05000529/2017008 AND 05000530/2017008

Dear Mr. Bement:

On February 10, 2017, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Palo Verde Nuclear Generating Station. The enclosed report documents the inspection findings, which were discussed on February 10, 2017, with Ms. M. Lacal, Senior Vice President, Nuclear Regulatory & Oversight, and other members of your staff.

On December 15, 2016, during a scheduled surveillance test run of the Unit 3 train B emergency diesel generator, the diesel engine experienced a catastrophic failure resulting in large quantities of oil and metal debris being expelled from the diesel engine. During a teleconference with the NRC on December 20, 2016, the licensee discussed their intent to identify the root cause of the failure, and evaluate the extent of condition as it related to the remaining emergency diesel generators at Palo Verde. On December 23, 2016, the NRC issued an emergency license amendment (Agencywide Documents Access and Management System (ADAMS) Accession No. ML16358A676) to extend the allowed outage time for the emergency diesel generator to a total of 21 days, based on compensatory measures implemented by the licensee. Subsequently, on January 4, 2017, the NRC issued a second emergency license amendment (ADAMS Accession No. ML17004A020) to extend the allowed outage time to 62 days based on the calculated overall plant risk with compensatory measures in place. On February 6, 2017, the special inspection team was dispatched to better understand the cause of the emergency diesel generator failure, extent of condition, potential generic implications, and the corrective actions proposed and taken by the licensee. The resident inspection staff at Palo Verde conducted inspections during the amendment requests and provided additional support for this inspection.

The NRC inspectors did not identify any finding or violation of more than minor significance. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Geoffrey B. Miller, Branch Chief Project Branch D Division of Reactor Projects Docket Nos: 05000528, 529, 530 License Nos: NPF-41, NPF-51, NPF-74

Enclosure:

Inspection Report 05000528/2017008, 05000529/2017008, and 05000530/2017008 w/ Attachments:

1. Supplemental Information 2. Memorandum to Ron Kopriva dated January 27, 2017 3. Diagram of Connecting Rod

REGION IV==

Dockets: 50-528; 50-529; 50-530 Licenses: NPF-41, NPF-51, NPF-74 Report No.: 05000528/2017008; 05000529/201708; 05000530/2017008 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station Location: 5801 South Wintersburg Road Tonopah, AZ 85354 Dates: February 6, 2017 through February 10, 2017 Inspectors: R. Kopriva, Senior Reactor Inspector (Team Leader)

D. Reinert, Ph.D., Resident Inspector Approved By: Geoffrey B. Miller Chief, Project Branch D Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000528/2017008; 05000529/2017008; 05000530/2017008 02/06/2017 - 02/10/2017;

PALO VERDE NUCLEAR GENERATING STATION Special Inspection Report.

This report covers a special inspection that reviewed the failure of Unit 3 Emergency Diesel Generator B and assessed the licensees response to the failure. The inspection team was composed of a resident inspector and one region-based engineering inspector. No findings were identified. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, dated July 2016.

Summary of Event and Inspection Results On December 15, 2016, Palo Verde Nuclear Generating Station (Palo Verde) Unit 3 was in Mode 1 at 100 percent power with the reactor coolant system at normal operating temperature and pressure. No major plant equipment was out of service. Unit 3 Emergency Diesel Generator B (3B) was running for a planned performance of procedure 40ST-9DG02, Diesel Generator B Test. The emergency diesel generator had been running since 3:02 a.m.

Mountain Standard Time (MST) and at 3:46 a.m. was loaded to 2.7 Megawatts (MW). At 3:56 a.m. MST, the Low Lube Oil Pressure trip was received in the Unit 3 Control Room. The Area Operator reported a large amount of smoke and oil on the floor of the diesel engine room which indicated a catastrophic failure of the number 9 Right (9R) piston on Emergency Diesel Generator 3B. The failure caused damage to the crankcase and expulsion of some engine materials onto the floor of the room. Control room operators declared Emergency Diesel Generator 3B inoperable at 3:56 a.m. MST and entered the applicable Limiting Condition of Operations.

After an evaluation of the emergency action levels in procedure EP-0901, Emergency Classification, operators declared an Alert at 4:10 a.m. based on criterion HA2.1: fire or explosion resulting in visible damage to any power block structure, or Control Room indication of degraded performance of safety systems. The event did not escalate beyond the Alert classification level. The licensee completed notifications to state and county agencies at 4:19 a.m. and notified the NRC at 4:55 a.m. All required emergency response facilities were staffed and activated within two hours as required by the Palo Verde Emergency Plan and were returned to a state of readiness following event termination at 6:36 a.m.

The team determined that operator response was appropriate and neither caused nor contributed to the failure. Good communications and response by the operators helped to characterize the immediate problem.

The team determined the licensees programs for maintenance, testing, and performance monitoring of the emergency diesel generators were appropriate, met applicable regulatory requirements, and did not contribute to this failure. The team determined that the licensees root cause assessment reached appropriate conclusions based on factual data. The direct cause of the Emergency Diesel Generator 3B failure was a high cycle fatigue crack in the master rod ligament between the crankpin bore and articulated rod pin bore (see Attachment 3). The crack initiated as fretting in areas between the master rod and the backside of the crankpin bearing and then propagated as a fatigue crack. The fretting in the master rod ligament was caused by a crankshaft bore that was out of alignment. At the time of the inspection, the most probable cause for the high cycle fatigue failure of the cylinder 9 master rod in Emergency Diesel

Generator 3B on December 15, 2016, was centerframe main bore misalignment from a 1986 master connecting rod failure event. This condition was not detectable by normal manufacturing or in-service inspections and was not reasonably within the licensees ability to foresee and prevent.

The licensee developed a repair strategy for Emergency Diesel Generator 3B incorporating vendor and engineering companies to assist in inspection, identification of the root causes of the failure, and rebuilding and testing of the emergency diesel generator. A new crankshaft, bearings, master and articulated connecting rods, and other parts were installed, along with crankcase repairs to restore Emergency Diesel Generator 3B to an operable status. The licensee reviewed the past history and operating experience of the other five diesel engines at the station. The licensee did not identify any common mode failure concerns.

The team did not identify any performance deficiencies or violations of NRC requirements as a result of this special inspection.

NRC-Identified and Self Revealing Findings

None.

REPORT DETAILS

1.0 SPECIAL

INSPECTION SCOPE

The NRC conducted this special inspection to better understand the facts and circumstances surrounding the failure of Palo Verde Unit 3 Emergency Diesel Generator B (Emergency Diesel Generator 3B) on December 15, 2016. This inspection reviewed the cause of the diesel failure, the extent of the condition, the potential generic implications, and the corrective actions proposed by the licensee. The team used NRC Inspection Procedure 93812, Special Inspection Procedure, and Inspection Procedure 71153, Follow-up of Events and Notices of Enforcement Discretion, to conduct the inspection. The special inspection team reviewed procedures, corrective action documents, and design and maintenance records for the equipment of concern.

The team interviewed key station personnel regarding the event, the root-cause analysis, and corrective actions.

A list of specific documents reviewed is provided in Attachment 1. The charter for the special inspection is provided as Attachment 2. A diagram of a connecting rod assembly is provided as Attachment 3.

2.0 SYSTEM AND EVENT DESCRIPTION 2.1 System Description Palo Verde Emergency Diesel Generator 3B is a Cooper Bessemer 20-cylinder, V-type turbo-charged engine, Model KSV-20, supplied by Cooper Energy Services. The engine operates at 600 rpm and has a rated output of 5500 kilowatts. The engine was manufactured in 1981. The connecting rod assembly consists of two connecting rods: a master connecting rod which drives the right cylinder and an articulated connecting rod which drives the left cylinder as shown in the diagram in Attachment 3. The twin cylinders operate on opposite portions of the 4-stroke cycle. The failed component was forged from AISI #1050 steel.

The failure occurred in the ligament between the articulated rod bushing and the main crankshaft throw journal bearing. The primary failure caused the articulated bearing straps to subsequently fail due to overload.

Each of the three units at the Palo Verde Nuclear Generating Station has two emergency diesel generators to provide emergency power to its respective train of safety-related equipment in the event that normal power is not available from offsite.

The diesel generators are normally in a standby condition, although they are run monthly for routine testing, as well as infrequently for other purposes.

2.2 Event Description On December 15, 2016, during a scheduled surveillance test run of Emergency Diesel Generator 3B, with the diesel generator loaded to approximately 50 percent, the control room received a low lube oil emergency trip annunciator. The Area Operator reported a catastrophic failure of the 9R piston on Emergency Diesel Generator 3B with a large amount of smoke and oil on the floor of the diesel engine room. The failure caused damage to the crankcase and expulsion of some engine materials onto the floor of the room. The licensee determined that there had been a catastrophic failure of a connecting rod for the number 9 cylinders, including crankcase damage and engine internal parts ejected from the crankcase. The licensee declared an Alert at 4:10 a.m.

MST based on an explosion resulting in visible damage to a safety system required for safe shutdown (HA2.1). The Palo Verde Fire Department responded and no fire was observed. The Alert was terminated at 6:36 a.m. MST. No other safety functions were impacted. No personnel injuries occurred.

2.3 Preliminary Significance of Events The NRC staff considered both deterministic and probabilistic criteria, established in NRC Management Directive 8.3, NRC Incident Investigation Program, to determine whether a special inspection would be performed. The NRC staff determined that the following two deterministic criteria were met:

(1) the diesel generator failure involved possible adverse generic implications, and

(2) the failure appeared to involve repetitive failures of safety-related equipment.

An NRC senior reactor analyst performed a preliminary risk assessment using the NRC's Standard Plant Analysis Risk Model, Revision 8.24, and SAPHIRE Version 8.1.4. The risk assessment assumed that the diesel generator failure was the result of a common cause failure mechanism. The exposure time of the condition was unknown, but was estimated as the time since the machine was last loaded (30 days).

Applying these assumptions resulted in an incremental conditional core damage probability of 3.7E-6. This result would increase if the exposure time included the time that had passed since the accrued run time exceeded the diesel mission time, and the result would diminish if modifications (e.g., placing, testing, and connecting two 2-Megawatt portable diesel generators to the Unit 3 FLEX connections, staging of the diesel-driven FLEX steam generator make-up pump, and suspension of discretionary maintenance on electrical and safety systems) that the licensee had implemented within the ten days following the failure were credited in the analysis.

This incremental conditional core damage probability was within the band for a special inspection. Based on meeting the deterministic criteria and the estimated incremental conditional core damage probability value, the NRC determined that a special inspection was warranted to further examine the circumstances surrounding the failure.

3.0 Special Inspection Areas 3.1 Appropriateness of Inspection Response (Charter Item 1)

a. Inspection Scope

The team reviewed available information and documentation from December 15, 2016, through January 30, 2017, on the failure of Emergency Diesel Generator 3B to ascertain whether the special inspection should be upgraded to an augmented inspection team. The team discussed the event with plant personnel, including operators, fire brigade members, emergency response organization members, management, the root cause team, and others.

b. Findings and Observations

The NRC concluded that a special inspection was the appropriate level of agency response.

No findings were identified.

3.2 Sequence of Events and Operator Response (Charter Items 2 and 3)

a. Inspection Scope

The team developed a sequence of events and evaluated operator response to the failure. The team reviewed the plant procedures for engine operation and surveillance testing and the alarm response procedures for emergency diesel generators. The team reviewed plant computer data and operator logs. The team discussed the sequence of events with the plant operators who were conducting the testing on December 15, 2016. The team also interviewed the acting fire captain who led the fire departments response to the event.

b. Findings and Observations

Operator Response The shift manager dispatched a reactor operator to the Emergency Diesel Generator 3B engine room to assess the damage. Other area operators and engineers who heard the radio conversations also responded to the room. All personnel stated that the engine appeared to have shut down almost immediately after the failure.

The team noted that the sudden failure of the emergency diesel generator was not an entry criteria for any of the licensees abnormal or emergency operating procedures.

The control room alarm response procedure directed the operators to confirm that the diesel generator trip had occurred and investigate the cause of the trip. The operators, however, quickly recognized that the failure had ruptured both the 9R and 9L piston cylinder liners and that engine jacket water was draining from the jacket water system into the lubricating oil system. Operators stopped the Emergency Diesel Generator 3B lube oil pump and heaters and the jacket water pump and heaters using system operating procedure, 40OP-9DG02, Emergency Diesel Generator B.

Because initial reports had indicated smoke in the engine room, control room operators notified the Palo Verde on-site fire department which arrived at Unit 3 at approximately 4:06 a.m. The fire department found no indications of a fire. The fire department assisted in the containment and cleanup of oil in the engine room and remained in the area for approximately 90 minutes following the event.

The Unit 3 operations shift manager evaluated the emergency action levels described in station procedure EP-0901, Emergency Classification. At 4:10 a.m., the shift manager declared an Alert based on an explosion affecting the operability of plant safety systems required to establish or maintain safe shutdown. Notifications were completed to the state and county agencies at 4:19 a.m. The NRC headquarters operations officer was notified of the event at 4:55 a.m. The event did not escalate beyond the Alert classification level, and the event was declared terminated at 6:36 a.m.

The team determined that the operator response was appropriate.

No findings were identified.

3.3 Cause Determination and Potential Common Failure Modes (Charter Items 4 and 5)

a. Inspection Scope

The team reviewed the scope and processes used by the licensee to identify the root cause of the failure of Emergency Diesel Generator 3B. The team held numerous discussions with the root cause team and the companies that were assisting the licensee to identify the root causes of the failure. The team evaluated the licensees ongoing efforts to review industry operating experience, including previous diesel generator failures for applicability to the failure of Emergency Diesel Generator 3B.

b. Findings and Observations

Cause Determination The team evaluated the results of the licensees document Extent of Condition and Extent of Cause, February 8, 2017, for the Emergency Diesel Generator 3B cylinder 9 master connecting rod failure. The report identified that the cylinder 9 master connecting rod was the first engine subcomponent to fail based on a review of subcomponent functions, inspection of the subcomponents, and visual characterization of the failure surfaces. The licensee identified a fatigue crack on the cylinder 9R master connecting rod, which had initiated at a ligament between the articulated pin bore and the crankshaft pin bore. As this fatigue crack grew, the ligament weakened. The associated cylinder 9L articulated connecting rod became detached after the master connecting rod failed.

A fatigue crack initiates and propagates through applied cyclic loads or stresses on a material flaw. Fatigue cracks leave indications of cyclic stresses as visible striations called beach marks on the fracture surface. Beach marks (also called clamshell marks) are features marking an interruption in the fatigue cracking progress. Beach marks were visible and striations confirmed by examination of the fracture surface.

Each beach mark on the failed surface can be associated to an actual engine run demand. Using this information, the licensee estimated the crack in Emergency Diesel Generator 3B had initiated a few months before November 16, 2013, since that was the oldest visible beach mark on the fracture surface. Close examination of the surface indicates the fatigue crack had been initiated a few engine load demands prior to the oldest visible beach mark.

From the examination, the licensee estimated the crack to have a cross-sectional area of 0.0658 inches based on the size of the fracture pattern. The licensee had routinely been using phased array ultrasonic testing on the master connecting rods as a result of previous engine failures in the industry. The licensee last performed phased array ultrasonic testing on the cylinder 9R master connecting rod in October 2013. The calibration block used for calibrating the phased array ultrasonic testing equipment had a notch size of 0.075 inch, which is the smallest notch that could be detected using the current technology available. At the time the phased array ultrasonic testing was performed the crack size was smaller than this calibration standard, so the crack could not have been detected with the available technology.

Further metallurgical evaluation of the failed connecting rods revealed that the Emergency Diesel Generator 3B cylinder 9R master connecting rod fatigue crack began as fretting fatigue. Fretting occurs between two surfaces in contact with each other when there is small, repetitive relative motion between them on the order of 5 to 100 microns. This occurs between components that are intended to be fixed in place relative to one another. Fretting is not uncommon in connecting rods at the interference fit between the connecting rod and the backside of the crankpin bearing shell. There is an oscillatory load placed on these components during the four stroke engine cycle as the cylinders fire and the crankshaft turns. The different geometries and materials of the connecting rod and bearing result in varying stresses in the components under the cyclic load. These differences can create relative micro-scale movement between the surfaces. The amount of damage caused by the fretting depends on the amount of relative motion (slip) between the surfaces, the shear stress at the surfaces, and the tensile stresses at or just below the surfaces. Fretting was identified by visual observation of a roughened surface of several master rods. The backside of the upper crankpin bearing shell had similar roughened surfaces in the same locations as the master rod saddle. These fretted areas were similar in appearance to photographs of fretting shown in diagnostic information provided by the bearing manufacturer. Fretting fatigue crack initiation involves the formation of surface cracks, and whether or not cracks form depends on the tensile stress at or near the surface.

In an internal-combustion engine, piston reciprocating motion is converted to rotational motion through a crankshaft, which becomes the power output. The crankshaft transmits energy from combustion occurring in all cylinders. During the disassembly of Emergency Diesel Generator 3B, the crankshaft bore was found to be out of alignment.

The misalignment was identified by measuring the crankshaft main bearing bore alignment within the centerframe once the crankshaft was removed. During the repairs and restoration of Emergency Diesel Generator 3B from the 1986 failure, the licensee did not have reason to separate the diesel engine crankcase, and therefore had not removed the crankshaft to inspect the centerframe main bore. The crankshaft must be removed to be able to directly measure centerframe main bore alignment. There was no known industry information pertaining to centerframe main bore misalignments, and as such, the licensee did not have reason to suspect that the centerframe main bore had become misaligned from the 1986 diesel engine failure. Fretting fatigue cracking in the cylinder 9R master connecting rod ligament was caused by a centerframe main bore that had not been held in design specified alignment. The most probable cause for the high cycle fatigue failure of cylinder 9R master connecting rod in Emergency Diesel Generator 3B on December 15, 2016, was centerframe main bore misalignment from the 1986 master connecting rod failure event. This condition was not detectable by normal in-service inspections and was not reasonably within the licensees ability to foresee and prevent.

Level of Detail Commensurate with the Safety Significance of the Event The team discussed the event with the licensees management and root cause team.

The licensee partnered with multiple companies and industry experts to determine the causes of the failure and assist with disassembly, repairs, reassembly, and testing of Emergency Diesel Generator 3B to restore the diesel generator to operable status.

The team concluded that the licensee appropriately incorporated assistance from a variety of industry experts to identify the cause of the failure of Emergency Diesel Generator 3B and perform state of the art repairs and testing of the engine. The licensee also refurbished the generator, though the failure of the diesel engine had not been associated with any faults or failure of the generator. The team concluded that the licensees level of effort was commensurate with the safety significance of the event.

Industry Operating Experience and Previous Diesel Generator Failures Since the Emergency Diesel Generator 3B failure was attributed to a connecting rod failure, the licensees extent of condition review focused on emergency diesel generators with Cooper-Bessemer engines based on the following:

• Operating experience involving five emergency diesel generator load failure events directly attributed to connecting rod failures within the diesel engine. All five events occurred on diesel engines manufactured by Cooper-Bessemer.

• Diesel engines manufactured by other manufacturers did not have an actual metallurgical failure of a connecting rod.

• Connecting rods are not interchangeable parts that can be installed in engines built by different manufacturers.

Cooper-Bessemer model KSV engines are designed with two types of connecting rods, a master connecting rod driving the right cylinder and an articulated connecting rod driving the left cylinder. All five connecting rod failure events began with a failure of the master connecting rod. Articulated rods have been damaged as a consequence of master rod failures, but have never initiated an event. Therefore, the licensee narrowed the extent of sub-components to master connecting rods and excluded articulated connecting rods. The five Cooper-Bessemer diesel engine failure events with master connecting rod failures are:

• Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod in 1986.

• South Texas Project Emergency Diesel Generator 22 cylinder 4R master connecting rod in 1989.

• Braidwood Emergency Diesel Generator 2B cylinder 1R master connecting rod in 1994.

• South Texas Project Emergency Diesel Generator 22 cylinder 9R master connecting rod in 2003.

• Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod in 2016.

All five master connecting rod failures listed above were caused by a fatigue crack.

Three of these failures were caused by obvious manufacturing defects and occurred before the engines had run long enough to put the number of operating cycles on the master connecting rods beyond the fatigue limit, which is the number of cycles and magnitude of applied cycle stresses that are necessary to initiate and propagate a fatigue crack in a specific material. For connecting rods, the fatigue limit is approximately 200-1000 hours of operation. After this period of time in operation, the residual stresses in the connecting rods are reduced, and fatigue cracking is unlikely to occur. The three failures were:

• Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod failure in 1986 was attributed to a manufacturing defect. Iron plating had inappropriately been used as a manufacturing repair for an over machined articulated connecting rod pin bore. This master connecting rod failure occurred with 100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> of engine operation.

• South Texas Project Emergency Diesel Generator 22 cylinder 4R master connecting rod failure in 1989 was attributed to an isolated manufacturing defect of an over-drilled oil passage. This master connecting rod failure occurred with 634 hours0.00734 days <br />0.176 hours <br />0.00105 weeks <br />2.41237e-4 months <br /> of engine operation.

• Braidwood Station Emergency Diesel Generator 2B cylinder 1R master connecting rod failure in 1994 was attributed to a manufacturing defect on the iron plating repair of the articulated connecting rod pin bore. This rod failure in the master connecting rod ligament occurred with approximately 900 hours0.0104 days <br />0.25 hours <br />0.00149 weeks <br />3.4245e-4 months <br /> of engine operation. Cooper-Bessemer issued a Part 21 report in February 1995 stating that all master cylinder rods with iron-plating repairs had been removed from service (Agencywide Document Access and Administration System (ADAMS) Accession No. ML9502170084).

The other two failures occurred after the engines had run long enough to put the number of operating cycles on the master connecting rods beyond the fatigue limit. No obvious manufacturing defects were associated with these two events:

• South Texas Project Emergency Diesel Generator 22 cylinder 9R master connecting rod failure in 2003 occurred with 2116 hours0.0245 days <br />0.588 hours <br />0.0035 weeks <br />8.05138e-4 months <br /> of engine operation.

• Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod failure in 2016 occurred with 3200 hours0.037 days <br />0.889 hours <br />0.00529 weeks <br />0.00122 months <br /> of engine operation.

These two failures both involved enough hours of operation on the engines to have exceeded the material fatigue limit of the master connecting rods such that fatigue cracking should not occur. During the disassembly of the diesel engine, the licensee discovered that the as-found centerframe main bearing bore alignment measurements were out of specification. From a document review of South Texas Project diesel failure, the team noted that the centerframe main bearing bore alignment measurements were out of specification in the South Texas emergency diesel generator that failed in 2003. Cooper-Bessemers centerframe alignment tolerances are as follows:

• Vertical alignment must be within 0.004 inch overall

• Horizontal alignment must be within 0.006 inch overall

• Horizontal alignment must be within 0.002 inch between adjacent bores The Palo Verde Emergency Diesel Generator 3B alignment measurements were:

• Eight of twelve bearings were outside of the 0.004 inch overall vertical specification.

• Three of twelve bearings were outside of the 0.006 inch overall horizontal specification.

• Four of eleven bearing pairs were outside of the 0.002 inch adjacent bearing, bore-to-bore, specification.

The South Texas Project Emergency Diesel Generator 22 measurements were:

• Six of twelve bearings were outside of the 0.004 inch overall vertical specification.

• Four of twelve bearings were outside of the 0.006 inch overall horizontal specification.

• Six of eleven bearing pairs were outside of the 0.002 inch adjacent bearing, bore-to-bore, specification.

From the review of the historic documentation, the licensee identified that the vertical and horizontal alignment measurements were not within Cooper-Bessemer design specifications on both engines, which likely occurred following the earlier catastrophic master connecting rod failure events (Palo Verde in 1986 and South Texas Project in 1989). Neither Palo Verde nor South Texas Project recognized that the centerframe main bore was out of alignment prior to the most recent diesel engine failures.

Following the South Texas Project 2003 emergency diesel generator failure, the South Texas Project licensee did not identify the centerframe main bore misalignment as the root cause of the failure. The licensee attributed the root cause of the failure to high cycle, low stress fatigue made possible by pre-existing microcracking below the surface at a location of stress concentration. This conclusion was factually supported with scientific evidence. However, the mechanism that induced the microcracking and the presumption that it occurred at the time of manufacturing were a best-estimate based on available information at the time. It was only after the Palo Verde licensee discovered centerframe main bore misalignment in February 2017 as the root cause of the 2016 failure that a review of the data from the South Texas Project diesel engine failure concluded a similar cause. Both engines have since had corrective actions implemented to restore the centerframe bore alignment to Cooper-Bessemer specifications by re-boring the centerframe.

Generic Concerns The team found that the licensees root cause conclusion at the time of the inspection was that the Emergency Diesel Generator 3B failure was the result of centerframe main bore misalignment from the 1986 master rod failure event. The misaligned centerframe main bore caused high cycle fretting fatigue in the master connecting rod at the interference fit between the connecting rod and the backside of the crankpin bearing shell. The result of the fretting was a fatigue crack on the cylinder 9R master connecting rod, which initiated at a ligament between the articulated pin bore and the crankshaft pin bore.

Thirty-six of the 55 Cooper-Bessemer KSV engines manufactured were used in nuclear emergency diesel generator applications. As of January 2016, there were 21 Model KSV 20-cylinder engines and 10 Model KSV 16-cylinder engines still in service in nuclear applications.

The team concluded that high cycle fatigue and fretting could occur in other Cooper-Bessemer KSV engines in the nuclear industry based on the general construction of the engines (connecting rods, bearings, bearing shells, et cetera). However, other engines in use at nuclear facilities have run hours that are longer than the run hours the Palo Verde Emergency Diesel Generator 3B had when it failed. All of these engines are susceptible to high cycle and fretting fatigue; however, none of these other Cooper-Bessemer KSV engines that have experienced an event that could cause a centerframe main bore misalignment. In the master connecting rod failure at Braidwood Station, a section of the rod bale area struck the high temperature trip lever and immediately tripped the engine, resulting in considerably lower stresses than in the failures at South Texas Project and Palo Verde. Absent growth from a centerframe bore misalignment, any developing fatigue cracks would be detectable with the use of the vendor recommended Phased Array Ultrasonic Testing prior to failure. The licensee shared information regarding the connecting rod failure with other nuclear facilities using KSV engines through the Cooper-Bessemer owners group.

No findings were identified.

3.5 Monitoring and Maintenance of the Emergency Diesel Generators (Charter Items 6 and 7)

a. Inspection Scope

The team reviewed the licensees program for performance monitoring and preventive maintenance of the emergency diesel generators, including inspection and assessment techniques, scope, periodicity, trending, and the results of past inspections. The team reviewed records of the licensees implementation of the Cooper-Bessemer Owners Group recommendations, including changes made to the vendors maintenance manual.

b. Findings and Observations

The team reviewed results from the licensees diesel generator performance monitoring program. The performance monitoring program includes trending of combustion parameters, exhaust temperatures, lubricating oil analysis, vibration analysis, jacket water temperatures, start times, and system engineer walkdown results. The team interviewed licensee lubrication engineers and vibration monitoring technicians. The team concluded that there were no indications of an impending failure of Emergency Diesel Generator 3B in the monitoring program results.

The team also reviewed maintenance history records for all six emergency diesel generators at Palo Verde, including the previous connecting rod failure event in Emergency Diesel Generator 3B that occurred in 1986. The team determined that there were no failure trends, or unusual or repetitive failures. The team also confirmed that the maintenance practices were in agreement with Section 8.3.1.1.4.12 in the Updated Final Safety Analysis Report.

The team reviewed changes that the licensee had made to the vendors maintenance recommendations to see if these changes played a role in the failure. The team also reviewed changes made to the emergency diesel generator maintenance and monitoring program in response to industry operating experience. The team reviewed results of inspections and corrective maintenance performed in response to NRC Information Notice 92-78, Piston to Cylinder Liner Tin Smearing on Cooper-Bessemer KSV Diesel Engines. The team also reviewed the licensees actions following the 2003 master connecting rod failure at South Texas Project Emergency Diesel Generator 22.

Following the South Texas Project failure, Palo Verde performed phased array ultrasonic examinations of all master connecting rods on all six emergency diesel generators. The team concluded that changes to the maintenance strategy were appropriate and made in accordance with vendor or industry recommendations.

No findings were identified.

3.6 Extent of Condition (Charter Item 8)

a. Inspection Scope

The team reviewed the licensees preliminary root cause evaluation for applicability to the other emergency diesel generators at the Palo Verde site. The team obtained documents pertaining to the licensees emergency diesel generator maintenance programs, surveillances, and non-destructive examination of selected components of the diesel generator. The team focused on the licensees procedures developed for the nondestructive examination on the emergency diesel generator connecting rods. The team also reviewed the metallurgical analysis report for the failed connecting rod.

b. Findings and Observations

The team reviewed the licensees Extent of Condition and Extent of Cause for the Palo Verde Emergency Diesel Generator 3B Cylinder #9 Master Connecting Rod Failure Event on December 15, 2016. The licensee had not completed their evaluation at the time of the inspection, but had identified centerframe main bore misalignment from the 1986 master connecting rod failure event as the most probable cause of the failure.

The team confirmed that none of the other emergency diesel generators at Palo Verde have experienced a master connecting rod failure. Absent the failure of a master connecting rod, there is no known mechanism that could cause a centerframe main bore misalignment after installation of the emergency diesel generator.

High cycle fatigue and fretting are a possibility due to the construction of the diesel engines (connecting rods, bearings, bearing shells, et cetera). Micro movement of these components could initiate a conditions to where fretting could be established.

Also, reworked or re-machined components (e.g., connecting rods), could become susceptible to failure. The Cooper Bessemer Owners Group issued a notice to KSV emergency diesel generator owners to periodically perform phased array ultrasonic testing of the master connecting rods to detect degradation. The team reviewed the licensees ultrasonic testing records for all of the emergency diesel generators at Palo Verde. There were no indications or flaws identified.

Metallurgical Examination The team reviewed the metallurgical analysis for the failed connecting rod. The team observed analytical samples from the failed rod under high magnification in both visual and electron microscopes. These samples clearly displayed classic striations indicative of cyclic fatigue. The fatigue cracks from the South Texas Project Emergency Diesel Generator 22 failure of 2003, and the Palo Verde Emergency Diesel Generator 3B failure of 2016, were nearly identical. Based on examination of the in-service connecting rods and the failure analysis of the failed connecting rod, the licensee concluded that the fatigue failure was an isolated occurrence and the remaining in-service rods were free of cracks. This eliminated the concern that there was a potential for common mode failure of standby diesel generators from the failure mechanism. The team concluded that the licensees examinations were thorough and that the conclusions drawn were reasonable.

No findings were identified.

3.7 Prompt and Long-Term Corrective Actions (Charter Item 9)

a. Inspection Scope

The team reviewed the scope and processes used by the licensee to identify the prompt and long-term corrective actions. The team evaluated the licensees ongoing efforts to assess common mode failure potential and extent of condition for the other emergency diesel generators. The team evaluated the results of nondestructive examination for applicability to the root cause of this event. The team reviewed sequence of events, operator performance, operating experience in the industry, as well as the mechanical and metallurgical issues associated with this event.

b. Findings and Observations

Prompt Corrective Actions and Review of Regulatory Relief Requests The team discussed the sites prompt corrective actions with the licensee and reviewed License Amendment No. 199, Renewed Facility Operating License No. NPF-74 for the Palo Verde Nuclear Generating Station, Unit 3 and License Amendment No. 200, Revision To Technical Specification 3.8.1, "AC [Alternating Current] Sources -

Operating." Amendment No. 199 revised the technical specifications for an extension of the emergency diesel generator completion time described in Technical Specification 3.8.1.B.4 from 10 days to 21 days for the purpose of collecting and analyzing data associated with the failure of Emergency Diesel Generator 3B and begin repair. Amendment No. 200 was a risk-informed amendment that further extended the required action completion time from 21 days to 62 days for the purpose of completing repairs and testing to re-establish operability of Emergency Diesel Generator 3B. The licensee evaluated the defense-in-depth and compensatory measures and requested the license amendments to extend the completion time based upon the guidance of NUREG-0800, Standard Review Plan, Branch Technical Position 8-8, "Onsite (Emergency Diesel Generators) and Offsite Power Sources Allowed Outage Time Extensions."

The team reviewed the compensatory measures the licensee had established during extended completion time, which included:

• Three, 2-MW portable diesel generators staged, tested and connected to Unit 3 FLEX (the diverse and flexible coping strategies, or FLEX,) 4.16 kV connections.

• Diesel-driven FLEX steam generator make-up pump staged in Unit 3.

• Suspension of discretionary maintenance on station black-out generators, switchyard, and safety systems.

• Establish protected equipment controls for Train A equipment, station black-out generators, and other portable equipment.

• Additional measures to reduce fire risk put into place to stringently control transient combustibles and limit performance of hot work.

• Additional personnel on-shift dedicated to compensatory measures.

Long Term Corrective Actions The team reviewed the long-term corrective actions to be taken by the licensee. At the time of the inspection, the licensee was working through their processes for determining the root cause and extent of condition and engaged several industry experts and engineering organizations for assistance in the analysis of failed mechanical parts. The team concluded the licensees actions were appropriate and commensurate with the safety significance of the event.

No findings were identified.

3.8 Corrective Actions for Past Similar Failures (Charter Item 10)

a. Inspection Scope

The team reviewed industry operating experience to identify whether there was a history of similar failures or existing preventive action recommendations for this failure mechanism. The team also reviewed the 1989 failure of a master connecting rod in the South Texas Project Emergency Diesel Generator 22 for similarities in root cause, common failure modes, and the scope and effectiveness of corrective actions.

b. Findings and Observations

The Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod failure began as a fretting fatigue. A fretting fatigue crack in the master rod ligament was caused by a crankshaft that was not held in design specified alignment by the crankshaft main bearings in the centerframe. The most probable cause for the high cycle fatigue failure of the Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod on December 15, 2016, was centerframe main bore misalignment from the 1986 master connecting rod failure. In 1986, there was no known industry information pertaining to centerframe main bore misalignments, and as such, the licensee did not have reason to verify centerframe main bore alignment by removing the crankshaft. The measured as-found centerframe main bore alignment during the Emergency Diesel Generator 3B repairs following the 2016 master connecting rod failure confirmed the centerframe main bore alignment was outside of the Cooper-Bessemer design specification. The inspectors determined the licensees preliminary conclusion that the misalignment had occurred following the 1986 failure was reasonable.

No findings were identified.

3.9 Post-Maintenance Testing (Charter Item 11)

a. Inspection Scope

The team evaluated the licensee's planned post-maintenance testing to demonstrate the operability and reliability of the repaired emergency diesel generator. The team compared the scope of the testing with vendor manual recommendations, technical specifications, Regulatory Guide 1.9, Institute of Electrical and Electronics Engineers Standard 387(IEEE 387), and the post-repair testing performed following the 1986 event for demonstrating operability and reliability of the emergency diesel generator.

The team reviewed the scope of the work packages performed to assess whether the post-maintenance testing scope was also appropriate for the work performed.

b. Findings and Observations

The licensee planned an extensive testing program prior to returning Emergency Diesel Generator 3B to service. The team concluded that the planned testing was in conformance with vendor recommendations for testing new engines. It also complied with IEEE 387 and Regulatory Guide 1.9 for testing and establishing adequate reliability of new engines. Additionally, the team concluded that the licensee conducted all applicable surveillance tests and inspections.

The team concluded that the tests were adequate and consistent with the post-maintenance testing program. The team also reviewed the scope of the planned performance monitoring and concluded that it was appropriate and consistent with the licensees past practices.

No findings were identified.

4OA6 Meetings, Including Exit

On February 10, 2017, the team presented the inspection results to Ms. M. Lacal, Senior Vice President, Nuclear Regulatory & Oversight, and other members of the licensee staff who acknowledged the findings. The team confirmed that all proprietary information reviewed during this inspection was returned to the licensee.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee personnel

G. Andrews, Director, Nuclear Regulatory Affairs

D. Bence, Assistant Plant Manager, Operations

B. Berles, Director, Nuclear Fuels

P. Bury, Director, Nuclear Training

S. Dornseif, Compliance Consultant, Nuclear Regulatory Affairs

D. Elkinton, Acting Section Leader, Nuclear Regulatory Affairs

J. Fearn, Manager, Emergency Preparedness

J. Glass, Department Lead, Maintenance Fix It Now Team

K. Graham, Director, Nuclear Engineering

M. Hooshmand, Department Leader, Nuclear Assurance

T. Horton, Director, Operations

C. Kharrl, Vice President Site Operations, General Plant Manager

M. Kura, Compliance Department Leader, Nuclear Regulatory Affairs

M. Lacal, Vice President, Nuclear Regulatory & Oversight

M. McGhee, Department Leader, Nuclear Regulatory Affairs

M. Radspinner, Department Leader, Nuclear Engineering

B. Rash, Vice President, Nuclear Engineering

C. Schingeck, Manager, Operations

J. Schrock, Engineer III, System Engineering, Balance of Plant

L. Weaver, Senior Engineer, Nuclear Regulatory Affairs

Contractors

R. Cryer, Director of Internal Combustion Engines, MPR

A. Killinger, Senior Associate, MPR

D. Zink, Diesel Generator System Engineering, South Texas Project

NRC personnel

C. Peabody, Senior Resident Inspector

D. You, Resident Inspector