IR 05000528/2017008

NRC Special Inspection Report 05000528/2017008, 05000529/2017008 and 05000530/2017008

ML17100A130
Person / Time
Site:	Palo Verde
Issue date:	04/10/2017
From:	Miller G B NRC/RGN-IV/DRP/RPB-D
To:	Bement R S Arizona Public Service Co
Miller G B
References
ML16358A676, ML17004A020 IR 2017008
Download: ML17100A130 (30)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION REGION IV 1600 E. LAMAR BLVD.

ARLINGTON, TX 76011-4511 April 10, 2017 Mr. Robert Executive Vice President Nuclear/

Chief Nuclear Officer Arizona Public Service Company P.O. Box 52034, MS 7602

Phoenix, AZ 85072

-2034

SUBJECT: PALO VERDE NUCLEAR GENERATING STATION

- NRC SPECIAL INSPECTION REPORT 05000528/2017008, 05000529/201 7008 AND 05000530/201 7008

Dear Mr. Bement:

On February 10, 2017, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Palo Verde Nuclear Generating Station.

The enclosed report documents the inspection findings, which were discussed on February 10, 2017, with Ms. M. Lacal, Senior Vice President, Nuclear Regulatory & Oversight, and other members of your staff.

On December 15, 2016, during a scheduled surveillance test run of the Unit 3 train B emergency diesel generator, the diesel engine experience d a catastrophic failure resulting in large quantities of oil and metal debris being expelled from the diesel engine. During a teleconference with the NRC on December 20, 2016, the licensee discussed their intent to identify the root cause of the failure, and evaluate the extent of condition as it related to the remaining emergency diesel generators at Palo Verde. On December 23, 2016, the NRC issued an emergency license amendment (Agencywide Documents Access and Management System (ADAMS) Accession No. ML16358A676) to extend the allowed outage time for the emergency diesel generator to a total of 21 days, based on compensatory measures implemented by the licensee. Subsequently, on January 4, 2017, the NRC issued a second emergency license amendment (ADAMS Accession No. ML17004A020) to extend the allowed outage time to 62 days based on the calculated overall plant risk with compensatory measures in place. On February 6, 2017, the special inspection team was dispatched to better understand the cause of the emergency diesel generator failure, extent of condition, potential generic implications, and the corrective actions proposed and taken by the licensee. The resident inspection staff at Palo Verde conducted inspection s during the amendment requests and provided additional support for this inspection

.

The NRC inspectors did not identify any finding or violation of more than minor significance. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading

-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/ Geoffrey B. Miller, Branch Chief Project Branch D Division of Reactor Projects Docket Nos:

05000528, 529, 530 License Nos

NPF-41, NPF-51, NPF-74 Enclosure:

Inspection Report 05000528/2017008, 05000529/2017008, and 05000530/2017008 w/ Attachments:

1. Supplemental Information 2. Memorandum to Ron Kopriva dated January 27, 2017 3. Diagram of Connecting Rod cc: Electronic Distribution

Enclosure U.S. NUCLEAR REGULATORY COMMISSION REGION IV Dockets: 50-528; 50-529; 50-530 Licenses: NPF-41, NPF-51, NPF-74 Report No.:

05000 528/20 17 00 8; 05000 529/201708; 05000530/2017008 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station Location: 5801 South Wintersburg Road Tonopah, AZ 85354 Dates: February 6, 2017 through February 10, 2017 Inspectors:

R. Kopriva, Senior Reactor Inspector (Team Leader)

D. Reinert, Ph

.D., Resident Inspector

Approved By:

Geoffrey B. Miller Chief, Project Branch D Division of Reactor Projects

2

SUMMARY OF FINDINGS

IR 05000 528/20 17008; 05000 529/20 17008; 05000530/2017008 02/06/2017

- 02/10/2017

PALO VERDE NUCLEAR GENERATING STATION Special Inspection Report.

This report covers a special inspection that reviewed the failure of Unit 3 Emergency Diesel Generator B and assessed the licensee's response to the failure. The i nspection team was composed of a resident inspector and one region-based engineering inspector. No findings were identified. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG

-1649, "Reactor Oversight Process," dated July 2016. Summary of Event and Inspection Results On December 15, 2016, Palo Verde Nuclear Generating Station (Palo Verde) Unit 3 was in Mode 1 at 100 percent power with the reactor coolant system at normal operating temperature and pressure. No major plant equipment was out of service. Unit 3 Emergency Diesel Generator B (3B) was running for a planned performance of procedure 40ST

-9DG02, "Diesel Generator 'B' Test." The emergency diesel generator had been running since 3:02 a.m. Mountain Standard Time (MST) and at 3:46 a.m. was loaded to 2.7 Megawatts (MW). At 3:56 a.m. MST , the Low Lube Oil Pressure trip was received in the Unit 3 Control Room. The Area Operator reported a large amount of smoke and oil on the floor of the diesel engine room which indicated a catastrophic failure of the number 9 Right (9 R) piston on Emergency Diesel Generator 3B. The failure caused damage to the crankcase and expulsion of some engine materials onto the floor of the room. Control room operators declared Emergency Diesel Generator 3B inoperable at 3:56 a.m. MST and entered the applicable Limiting Condition of Operations.

After a n evaluation of the emergency action levels in procedure EP

-0901, "Emergency Classification," operators declared an Alert at 4:10 a.m

. based on criterion HA2.1: fire or explosion resulting in visible damage to any power block structure, or Control Room indication of degraded performance of safety systems. The event did not escalate beyond the Alert classification level. The licensee completed notifications to state and county agencies at 4:19 a.m. and notified the NRC at 4:55 a.m. All required emergency response facilities were staffed and activated within two hours as required by the Palo Verde Emergency Plan and wer e returned to a state of readiness following event termination at 6:36 a.m

. The team determined that operator response was appropriate and neither caused nor contributed to the failure. Good communications and response by the operators helped to characterize the immediate problem.

The team determined the licensee's programs for maintenance, testing, and performance monitoring of the emergency diesel generators were appropriate, met applicable regulatory requirements, and did not contribute to this failure. The team determined that the licensee's root cause assessment reached appropriate conclusions based on factual data. The direct cause of the Emergency Diesel Generator 3B failure was a high cycle fatigue crack in the master rod ligament between the crankpin bore and articulated rod pin bore (see Attachment 3). The crack initiated as fretting in areas between the master rod and the backside of the crankpin bearing and then propagated as a fatigue crack. The fretting in the master rod ligament was caused by a crankshaft bore that was out of alignment. At the time of the inspection, the most probable cause for the high cycle fatigue failure of the cylinder 9 master rod in Emergency Diesel 3 Generator 3B on December 15, 2016 , was centerframe main bore misalignment from a 1986 master connecting rod failure event. This condition was not detectable by normal manufacturing or in-service inspections and was not reasonably within the licensee's ability to foresee and prevent. The l icensee developed a repair strategy for Emergency Diesel Generator 3B incorporating vendor and engineering companies to assist in inspection, identification of the root cause s of the failure, and rebuilding and testing of the emergency diesel generator. A new crankshaft, bearings, master and articulated connecting rods, and other parts were installed, along with crankcase repairs to restore Emergency Diesel Generator 3B to an operable status. The licensee reviewed the past history and operating experience of the other five diesel engines at the station. The licensee did not identify any common mode failure concerns. The team did not identify any performance deficiencies or violations of NRC requirements as a result of this special inspection.

NRC-Identified and Self Revealing Findings

No ne.

4

REPORT DETAILS

1.0 SPECIAL

INSPECTION SCOPE

The NRC conducted this special inspection to better understand the facts and circumstances surrounding the failure of Palo Verde Unit 3 Emergency Diesel Generator B (Emergency Diesel Generator 3B) on December 15, 2016. This inspection reviewed the cause of the diesel failure, the extent of the condition, the potential generic implications, and the corrective actions proposed by the licensee. The team used NRC Inspection Procedure 93812, "Special Inspection Procedure," and Inspection Procedure 71153, "Follow

-up of Events and Notices of Enforcement Discretion," to conduct the inspection. The special inspection team reviewed procedures, corrective action documents, and design and maintenance records for the equipment of concern. The team interviewed key station personnel regarding the event, the root

-cause analysis, and corrective actions.

A list of specific documents reviewed is provided in Attachment 1. The charter for the special inspection is provided as Attachment 2.

A diagram of a connecting rod assembly is provided as Attachment 3.

2.0 SYSTEM AND EVENT DESCRIPTION 2.1 System Description Palo Verde Emergency Diesel Generator 3B is a Cooper Bessemer 20

-cylinder, V

-type turbo-charged engine, Model KSV

-20, supplied by Cooper Energy Services. The engine operates at 600 rpm and has a rated output of 5500 kilowatts. The engine was manufactured in 1981. The connecting rod assembly consists of two connecting rods: a master connecting rod which drives the right cylinder and an articulated connecting rod which drives the left cylinder as shown in the diagram in Attachment 3.

The twin cylinders operate on opposite portions of the 4-stroke cycle. The failed component was forged from AISI #1050 steel.

The failure occurred in the ligament between the articulated rod bushing and the main crankshaft throw journal bearing. The primary failure caused the articulated bearing straps to subsequently fail due to overload.

Each of the three units at the Palo Verde Nuclear Generating Station has two emergency diesel generators to provide emergency power to its respective train of safety-related equipment in the event that normal power is not available from offsite. The diesel generators are normally in a standby condition, although they are run monthly for routine testing, as well as infrequently for other purposes.

2.2 Event Description On December 15, 2016, during a scheduled surveillance test run of Emergency Diesel Generator 3B, with the diesel generator loaded to approximately 50 percent, the control room received a low lube oil emergency trip annunciator. The Area Operator reported a catastrophic failure of the 9R piston on Emergency Diesel Generator 3B with a large amount of smoke and oil on the floor of the diesel engine room. The failure caused damage to the crankcase and expulsion of some engine materials onto the floor of the room. The licensee determined that there had been a catastrophic failure of a connecting rod for the number 9 cylinders, including crankcase damage and engine internal parts ejected from the crankcase.

The licensee declared an Alert at 4:10 a.m. MST based on an explosion resulting in visible damage to a safety system required for safe shutdown (HA2.1). The Palo Verde Fire Department responded and no fire was observed. The Alert was terminated at 6:36 a.m. MST. No other safety functions were impacted. No personnel injuries occurred.

2.3 Preliminary Significance of Events The NRC staff considered both deterministic and probabilistic criteria, established in NRC Management Directive 8.3, "NRC Incident Investigation Program," to determine whether a special inspection would be performed. The NRC staff determined that the following two deterministic criteria were met:

(1) the diesel generator failure involved possible adverse generic implications, and

(2) the failure appeared to involve repetitive failures of safety

-related equipment.

An NRC senior reactor analyst performed a preliminary risk assessment using the NRC's Standard Plant Analysis Risk Model, Revision 8.24, and SAPHIRE Version 8.1.4. The risk assessment assumed that the diesel generator failure was the result of a common cause failure mechanism. The exposure time of the condition was

unknown, but was estimated as the time since the machine was last loaded (30 days). Applying these assumptions resulted in an incremental conditional core damage probability of 3.7E

-6. This result would increase if the exposure time included the time that had passed since the accrued run time exceeded the diesel mission time, and the result would diminish if modifications (e.g., placing, testing, and connecting two 2

-Megawatt portable diesel generators to the Unit 3 FLEX connections, staging of the diesel-driven FLEX steam generator make

-up pump, and suspension of discretionary maintenance on electrical and safety systems) that the licensee had implemented within the ten days following the failure were credited in the analysis.

This incremental conditional core damage probability was within the band for a special inspection. Based on meeting the deterministic criteria and the estimated incremental conditional core damage probability value, the NRC determined that a special inspection was warranted to further examine the circumstances surrounding the failure

. 3.0 Special Inspection Areas 3.1 Appropriateness of Inspection Response (Charter Item 1)

a. Inspection Scope

The team reviewed available information and documentation from December 15, 2016

, through January 30, 2017

, on the failure of Emergency Diesel Generator 3B to ascertain whether the special inspection should be upgraded to an augmented inspection team. The team discussed the event with plant personnel, including operators, fire brigade members, emergency response organization members , management, the root cause team, and others.

b. Findings and Observations

The NRC concluded that a special inspection was the appropriate level of agency response.

No findings were identified.

3.2 Sequence of Events and Operator Response (Charter Items 2 and 3)

a. Inspection Scope

The team developed a sequence of events and evaluated operator response to the failure. The team reviewed the plant procedures for engine operation and surveillance testing and the alarm response procedures for emergency diesel generators. The team reviewed plant computer data and operator logs. The team discussed the sequence of events with the plant operators who were conducting the testing on December 15, 2016. The team also interviewed the acting fire captain who led the fire department's response to the event.

b. Findings and Observations

Operator Response The shift manager dispatched a reactor operator to the Emergency Diesel Generator 3B engine room to assess the damage. Other area operators and engineers who heard the radio conversations also responded to the room. All personnel stated that the engine appeared to have shut down almost immediately after the failure.

The team noted that the sudden failure of the emergency diesel generator was not an entry criteria for any of the licensee's abnormal or emergency operating procedures.

The control room alarm response procedure directed the operators to confirm that the diesel generator trip had occurred and investigate the cause of the trip. The operators, however, quickly recognize d that the failure had ruptured both the 9R and 9L piston cylinder liners and that engine jacket water was draining from the jacket water system into the lubricating oil system. Operators stopped the Emergency Diesel Generator 3B lube oil pump and heaters and the jacket water pump and heaters using system operating procedure, 40O P-9DG02, "Emergency Diesel Generator B."

Because initial reports had indicated smoke in the engine room, control room operators notified the Palo Verde on

-site fire department which arrived at Unit 3 at approximately 4:06 a.m. The fire department found no indications of a fire. The fire department assisted in the containment and cleanup of oil in the engine room and remained in the area for approximately 90 minutes following the event.

The Unit 3 operations shift manager evaluated the emergency action levels described in station procedure EP

-0901, "Emergency Classification." At 4:10 a.m., the shift manager declared an Alert based on an explosion affecting the operability of plant safety systems required to establish or maintain safe shutdown.

Notifications were completed to the state and county agencies at 4:19 a.m.

The NRC headquarters operations officer was notified of the event at 4:55 a.m. The event did not escalate beyond the Alert classification level, and the event was declared terminated at 6:36 a.m.

The team determined that the operator response was appropriate.

No findings were identified.

3.3 Cause Determination and Potential Common Failure Modes (Charter Items 4 and 5)

a. Inspection Scope

The team reviewed the scope and processes used by the licensee to identify the root cause of the failure of Emergency Diesel Generator 3B. The team held numerous discussions with the root cause team and the companies that were assisting the licensee to identify the root causes of the failure. The team evaluated the licensee's ongoing efforts to review industry operating experience, including previous diesel generator failures for applicability to the failure of Emergency Diesel Generator 3B.

b. Findings and Observations

Cause Determination The team evaluated the results of the licensee's document "Extent of Condition and Extent of Cause, February 8, 2017," for the Emergency Diesel Generator 3B cylinder 9 master connecting rod failure. The report identified that the cylinder 9 master connecting rod was the first engine subcomponent to fail based on a review of subcomponent functions, inspection of the subcomponents

, and visual characterization of the failure surfaces. The licensee identified a fatigue crack on the cylinder 9R master connecting rod, which had initiated at a ligament between the articulated pin bore and the crankshaft pin bore. As this fatigue crack grew, the ligament weakened. The associated cylinder 9L articulated connecting rod became detached after the master connecting rod failed.

A fatigue crack initiates and propagates through applied cyclic loads or stresses on a material flaw. Fatigue cracks leave indications of cyclic stresses as visible striations called "beach marks" on the fracture surface. Beach marks (also called "clamshell marks") are features marking an interruption in the fatigue cracking progress. Beach marks were visible and striations confirmed by examination of the fracture surface. Each beach mark on the failed surface can be associated to an actual engine run demand. Using this information, the licensee estimated the crack in Emergency Diesel Generator 3B had initiated a few months before November 16, 2013, since that was the oldest visible beach mark on the fracture surface. Close examination of the surface indicates the fatigue crack had been initiated a few engine load demands prior to the oldest visible beach mark.

From the examination, the licensee estimated the crack to have a cross

-sectional area of 0.0658 inch es based on the size of the fracture pattern. The licensee had routinely been using phased array ultrasonic testing on the master connecting rods as a result of previous engine failures in the industry. The licensee last performed phased array ultrasonic testing on the cylinder 9R master connecting rod in October 2013.

T he calibration block used for calibrating the phased array ultrasonic testing equipment ha d

a notch size of 0.075 inch, which is the smallest notch that could be detected using the current technology available. At the time the phased array ultrasonic testing was performed the crack size was smaller than this calibration standard

, so the crack could not have been detected with the available technology.

Further metallurgical evaluation of the failed connecting rods revealed that the Emergency Diesel Generator 3B cylinder 9R master connecting rod fatigue crack began as fretting fatigue. Fretting occurs between two surfaces in contact with each other when there is small, repetitive relative motion between them on the order of 5 to 100 microns. This occurs between components that are intended to be fixed in place relative to one another. Fretting is not uncommon in connecting rods at the interference fit between the connecting rod and the backside of the crankpin bearing shell. There is an oscillatory load placed on these components during the four stroke engine cycle as the cylinders fire and the crankshaft turns. The different geometries and materials of the connecting rod and bearing result in varying stresses in the components under the cyclic load. These difference s can create relative micro

-scale movement between the surfaces. The amount of damage caused by the fretting depends on the amount of relative motion (slip) between the surfaces, the shear stress at the surfaces, and the tensile stresses at or just below the surfaces. Fretting was identified by visual observation of a roughened surface of several master rods. The backside of the upper crankpin bearing shell had similar roughened surfaces in the same locations as the master rod saddle. These fretted areas were similar in appearance to photographs of fretting shown in diagnostic information provided by the bearing manufacturer. Fretting fatigue crack initiation involves the formation of surface cracks, and whether or not cracks form depends on the tensile stress at or near the surface

. In an internal

-combustion engine, piston reciprocating motion is converted to rotational motion through a crankshaft, which becomes the power output. The crankshaft transmits energy from combustion occurring in all cylinders. During the disassembly of Emergency Diesel Generator 3B, the crankshaft bore was found to be out of alignment. The misalignment was identified by measuring the crankshaft main bearing bore alignment within the centerframe once the crankshaft was removed. During the repairs and restoration of Emergency Diesel Generator 3B from the 1986 failure, the licensee did not have reason to separate the diesel engine crankcase, and therefore had not removed the crankshaft to inspect the centerframe main bore. The crankshaft must be removed to be able to directly measure centerframe main bore alignment. There was no known industry information pertaining to centerframe main bore misalignments, and as such, the licensee did not have reason to suspect that the centerframe main bore had become misaligned from the 1986 diesel engine failure. Fretting fatigue cracking in the cylinder 9R master connecting rod ligament was caused by a centerframe main bore that had not been held in design specified alignment. The most probable cause for the high cycle fatigue failure of cylinder 9R master connecting rod in Emergency Diesel Generator 3B on December 15, 2016

, was centerframe main bore misalignment from the 1986 master connecting rod failure event.

This condition was not detectable by normal in

-service inspections and was not reasonably within the licensee's ability to foresee and prevent.

Level of Detail Commensurate with the Safety Significance of the Event The team discussed the event with the licensee's management and root cause team. The licensee partnered with multiple companies and industry experts to determine the cause s of the failure and assist with disassembly, repairs, reassembly

, and testing of Emergency Diesel Generator 3B to restore the diesel generator to operable status.

The team concluded that the licensee appropriately incorporated assistance from a variety of industry experts to identify the cause of the failure of Emergency Diesel Generator 3B and perform state of the art repairs and testing of the engine. The licensee also refurbished the generator, though the failure of the diesel engine had not been associated with any faults or failure of the generator. The team concluded that the licensee's level of effort was commensurate with the safety significance of the event.

Industry Operating Experience and Previous Diesel Generator Failures Since the Emergency Diesel Generator 3B failure was attributed to a connecting rod failure, the licensee's extent of condition review focused on emergency diesel generators with Cooper

-Bessemer engines based on the following:

Operating experience involving five emergency diesel generator load failure events directly attributed to connecting rod failures within the diesel engine. All five events occurred on diesel engines manufactured by Cooper

-Bessemer. Diesel engines manufactured by other manufacturers did not have an actual metallurgical failure of a connecting rod.

Connecting rods are not interchangeable parts that can be installed in engines built by different manufacturers.

Cooper-Bessemer model KSV engines are designed with two types of connecting rods, a master connecting rod driving the right cylinder and an articulated connecting rod driving the left cylinder. All five connecting rod failure events began with a failure of the master connecting rod. Articulated rods have been damaged as a consequence of master rod failures, but have never initiated an event. Therefore, the licensee narrowed the extent of sub

-components to master connecting rods and excluded articulated connecting rods. The five Cooper

-Bessemer diesel engine failure events with master connecting rod failures are:

Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod in 1986. South Texas Project Emergency Diesel Generator 22 cylinder 4R master connecting rod in 1989.

Braidwood Emergency Diesel Generator 2B cylinder 1R master connecting rod in 1994. South Texas Project Emergency Diesel Generator 22 cylinder 9R master connecting rod in 2003.

Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod in 2016. All five master connecting rod failures listed above were caused by a fatigue crack. Three of these failures were caused by obvious manufacturing defects and occurred before the engines had run long enough to put the number of operating cycles on the master connecting rods beyond the fatigue limit

, which is the number of cycles and magnitude of applied cycle stresses that are necessary to initiate and propagate a fatigue crack in a specific material.

For connecting rods, the fatigue limit is approximately 200

-1000 hours of operation. After this period of time in operation, the residual stresses in the connecting rods are reduced, and fatigue cracking is unlikely to occur. The three failures were

Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod failure in 1986 was attributed to a manufacturing defect. Iron plating had inappropriately been used as a manufacturing repair for an over machined articulated connecting rod pin bore. This master connecting rod failure occurred with 100 hours0.00116 days <br />0.0278 hours <br />1.653439e-4 weeks <br />3.805e-5 months <br /> of engine operation.

South Texas Project Emergency Diesel Generator 22 cylinder 4R master connecting rod failure in 1989 was attributed to an isolated manufacturing defect of an over-drilled oil passage. This master connecting rod failure occurred with 634 hours0.00734 days <br />0.176 hours <br />0.00105 weeks <br />2.41237e-4 months <br /> of engine operation.

Braidwood Station Emergency Diesel Generator 2B cylinder 1R master connecting rod failure in 1994 was attributed to a manufacturing defect on the iron plating repair of the articulated connecting rod pin bore. This rod failure in the master connecting rod ligament occurred with approximately 900 hours0.0104 days <br />0.25 hours <br />0.00149 weeks <br />3.4245e-4 months <br /> of engine operation.

Cooper-Bessemer issued a Part 21 report in February 1995 stating that all master cylinder rods with iron

-plating repairs had been removed from service (Agencywide Document Access and Administration System (ADAMS) Accession No. ML9502170084).

The other two failures occurred after the engines had run long enough to put the number of operating cycles on the master connecting rods beyond the fatigue limit. No obvious manufacturing defects were associated with these two events: South Texas Project Emergency Diesel Generator 22 cylinder 9R master connecting rod failure in 2003 occurred with 2116 hours0.0245 days <br />0.588 hours <br />0.0035 weeks <br />8.05138e-4 months <br /> of engine operation.

Palo Verde Emergency Diesel Generator 3B cylinder 9 R master connecting rod failure in 2016 occurred with 3200 hours0.037 days <br />0.889 hours <br />0.00529 weeks <br />0.00122 months <br /> of engine operation.

T he se two failures both involved enough hours of operation on the engines to have exceeded the material fatigue limit of the master connecting rods such that fatigue cracking should not occur. During the disassembly of the diesel engine, the licensee discovered that the as

-found centerframe main bearing bore alignment measurements were out of specification. From a document review of South Texas Project diesel failure, the team noted that the centerframe main bearing bore alignment measurements were out of specification in the South Texas emergency diesel generator that failed in 2003. C ooper-Bessemer's centerframe alignment tolerances are as follows:

Vertical alignment must be within 0.004 inch overall Horizontal alignment must be within 0.006 inch overall Horizontal alignment must be within 0.002 inch between adjacent bores The Palo Verde Emergency Diesel Generator 3B alignment measurements were

Eight of twelve bearings were outside of the 0.004 inch overall vertical specification.

Three of twelve bearings were outside of the 0.006 inch overall horizontal specification.

Four of eleven bearing pairs were outside of the 0.002 inch adjacent bearing, bore-to-bore, specification.

The South Texas Project Emergency Diesel Generator 22 measurements were

Six of twelve bearings were outside of the 0.004 inch overall vertical specification.

Four of twelve bearings were outside of the 0.006 inch overall horizontal specification.

Six of eleven bearing pairs were outside of the 0.002 inch adjacent bearing, bore-to-bore, specification.

From the review of the historic documentation, the licensee identified that the vertical and horizontal alignment measurements were not within Cooper

-Bessemer design specifications on both engines, which likely occurred following the earlier catastrophic master connecting rod failure events (Palo Verde in 1986 and South Texas Project in 1989). Neither Palo Verde nor South Texas Project recognized that the centerframe main bore was out of alignment prior to the most recent diesel engine failures.

Following the South Texas Project 2003 emergency diesel generator failure, the South Texas Project licensee did not identify the centerframe main bore misalignment as the root cause of the failure. The licensee attributed the root cause of the failure to high cycle, low stress fatigue made possible by pre

-existing microcracking below the surface at a location of stress concentration. This conclusion was factually supported with scientific evidence. However, the mechanism that induced the microcracking and the presumption that it occurred at the time of manufacturing were a best

-estimate based on available information at the time. It was only after the Palo Verde licensee discovered centerframe main bore misalignment in February 2017 as the root cause of the 2016 failure that a review of the data from the South Texas Project diesel engine failure concluded a similar cause. Both engines have since had corrective actions implemented to restore the centerframe bore alignment to Cooper

-Bessemer specifications by re-boring the centerframe.

Generic Concerns The team found that the licensee's root cause conclusion at the time of the inspection was that the Emergency Diesel Generator 3B failure was the result of centerframe main bore misalignment from the 1986 master rod failure event. The misaligned centerframe main bore caused high cycle fretting fatigue in the master connecting rod at the interference fit between the connecting rod and the backside of the crankpin bearing shell. The result of the fretting wa s a fatigue crack on the cylinder 9R master connecting rod, which initiated at a ligament between the articulated pin bore and the crankshaft pin bore.

Thirty-six of the 55 Cooper

-Bessemer KSV engines manufactured were used in nuclear emergency diesel generator applications. As of January 2016, there were 21 Model KSV 20-cylinder engines and 10 Model KSV 16-cylinder engines still in service in nuclear applications.

The team concluded that high cycle fatigue and fretting could occur in other Cooper

-Bessemer KSV engines in the nuclear industry based on the general construction of the engines (connecting rods, bearings, bearing shells, et cetera). However, other engines in use at nuclear facilities have run hours that are longer than the run hours the Palo Verde Emergency Diesel Generator 3B had when it failed. All of these engines are susceptible to high cycle and fretting fatigue; however , none of these other Cooper

-Bessemer KSV engines that have experienced an event that could cause a centerframe main bore misalignment. In the master connecting rod failure at Braidwood Station

, a section of the rod bale area struck the high temperature trip lever and immediately tripped the engine, resulting in considerably lower stresses than in the failures at South Texas Project and Palo Verde.

Absent growth from a centerframe bore misalignment, any developing fatigue cracks would be detectable with the use of the vendor recommended Phased Array Ultrasonic Testing prior to failure. The licensee shared information regarding the connecting rod failure with other nuclear facilities using KSV engines through the Cooper

-Bessemer owners group.

No findings were identified.

3.5 Monitoring and Maintenance of the Emergency Diesel Generators (Charter Items 6 and 7)

a. Inspection Scope

The team reviewed the licensee's program for performance monitoring and preventive maintenance of the emergency diesel generators, including inspection and assessment techniques, scope, periodicity, trending, and the results of past inspections. The team reviewed records of the licensee's implementation of the Cooper

-Bessemer Owners Group recommendations

, including changes made to the vendor's maintenance manual.

b. Findings and Observations

The team reviewed results from the licensee's diesel generator performance monitoring program. The performance monitoring program includes trending of combustion parameters, exhaust temperatures, lubricating oil analysis, vibration analysis, jacket water temperatures, start times, and system engineer walkdown results. The team interviewed licensee lubrication engineers and vibration monitoring technicians. The team concluded that there were no indications of an impending failure of Emergency Diesel Generator 3B in the monitoring program results.

The team also reviewed maintenance history records for all six emergency diesel generators at Palo Verde, including the previous connecting rod failure event in Emergency Diesel Generator 3B that occurred in 1986. The team determined that there were no failure trends, or unusual or repetitive failures. The team also confirmed that the maintenance practices were in agreement with Section 8.3.1.1.4.12 in the Updated Final Safety Analysis Report.

The team reviewed changes that the licensee had made to the vendor's maintenance recommendations to see if these changes played a role in the failure. The team also reviewed changes made to the emergency diesel generator maintenance and monitoring program in response to industry operating experience. The team reviewed results of inspections and corrective maintenance performed in response to NRC Information Notice 92

-78, "Piston to Cylinder Liner Tin Smearing on Cooper

-Bessemer KSV Diesel Engines." The team also reviewed the licensee's actions following the 2003 master connecting rod failure at South Texas Project Emergency Diesel Generator 22.

Following the South Texas Project failure, Palo Verde performed phased array ultrasonic examinations of all master connecting rods on all six emergency diesel generators. The team concluded that changes to the maintenance strategy were appropriate and made in accordance with vendor or industry recommendations.

No findings were identified.

3.6 Extent of Condition (Charter Item 8)

a. Inspection Scope

The team reviewed the licensee's preliminary root cause evaluation for applicability to the other emergency diesel generator s at the Palo Verde site. The team obtained documents pertaining to the licensee's emergency diesel generator maintenance programs, surveillances, and non

-destructive examination of selected components of the diesel generator. The team focused on the licensee's procedures developed for the nondestructive examination on the emergency diesel generator connecting rods. The team also reviewed the metallurgical analysis report for the failed connecting rod.

b. Findings and Observations

The team review ed the licensee's "Extent of Condition and Extent of Cause for the Palo Verde Emergency Diesel Generator 3B Cylinder #9 Master Connecting Rod Failure Event on December 15, 2016." The licensee had not completed their evaluation at the time of the inspection, but had identified centerframe main bore misalignment from the 1986 master connecting rod failure event as the most probable cause of the failure

. The team confirmed that none of the other emergency diesel generators at Palo Verde have experienced a master connecting rod failure. Absent the failure of a master connecting rod, there is no known mechanism that could cause a centerframe main bore misalignment after installation of the emergency diesel generator

. High cycle fatigue and fretting are a possibility due to the construction of the diesel engines (connecting rods, bearings, bearing shells, et cetera). Micro movement of these components could initiate a conditions to where fretting could be established.

Also, reworked or re-machined components (e.g., connecting rods), could become susceptible to failure.

The Cooper Bessemer Owners Group issued a notice to KSV emergency diesel generator owners to periodically perform phased array ultrasonic testing of the master connecting rods to detect degradation. The team reviewed the licensee's ultrasonic testing records for all of the emergency diesel generators at Palo Verde. There were no indications or flaws identified

. Metallurgical Examination The team reviewed the metallurgical analysis for the failed connecting rod. The team observed analytical samples from the failed rod under high magnification in both visual and electron microscopes. These samples clearly displayed classic striations indicative of cyclic fatigue. The fatigue cracks from the South Texas Project Emergency Diesel Generator 22 failure of 2003, and the Palo Verde Emergency Diesel Generator 3B failure of 2016, were nearly identical. Based on examination of the in-service connecting rods and the failure analysis of the failed connecting rod, the licensee concluded that the fatigue failure was an isolated occurrence and the remaining in

-

service rods were free of cracks. This eliminated the concern that there was a potential for common mode failure of standby diesel generators from the failure mechanism. The team concluded that the licensee's examinations were thorough and that the conclusions drawn were reasonable.

No findings were identified.

3.7 Prompt and Long

-Term Corrective Actions (Charter Item 9)

a. Inspection Scope

The team reviewed the scope and processes used by the licensee to identify the prompt and long

-term corrective actions. The team evaluated the licensee's ongoing efforts to assess common mode failure potential and extent of condition for the other emergency diesel generators. The team evaluated the results of nondestructive examination for applicability to the root cause of this event. The team reviewed sequence of events, operator performance, operating experience in the industry, as well as the mechanical and metallurgical issues associated with this event.

b. Findings and Observations

Prompt Corrective Actions and Review of Regulatory Relief Requests The team discussed the site's prompt corrective actions with the licensee and reviewed License Amendment No. 199 , "Renewed Facility Operating License No. NPF

-74 for the Palo Verde Nuclear Generating Station, Unit 3" and License Amendment No. 200, Revision To Technical Specification 3.8.1, "AC [Alternating Current] Sources

- Operating." Amendment No. 199 revised the technical specifications for a n extension of the emergency diesel generator completion time described in Technical Specification 3.8.1.B.4 from 10 days to 21 days for the purpose of collecting and analyzing data associated with the failure of Emergency Diesel Generator 3B and begin repair. Amendment No. 200 was a risk

-informed amendment that further extended the required action completion time from 21 days to 62 days for the purpose of completing repairs and testing to re

-establish operability of Emergency Diesel Generator 3B. The licensee evaluated the defense

-in-depth and compensatory measures and requested the license amendment s to extend the completion time based upon the guidance of NUREG-0800, Standard Review Plan, Branch Technical Position 8

-8, "Onsite (Emergency Diesel Generators) and Offsite Power Sources Allowed Outage Time Extensions."

The team reviewed the compensatory measures the licensee had established during extended completion time, which included:

Three, 2-MW portable diesel generators staged, tested and connected to Unit 3 FLEX (the diverse and flexible coping strategies, or "FLEX,") 4.16 kV connections.

Diesel-driven FLEX steam generator make

-up pump staged in Unit 3.

Suspension of discretionary maintenance on station black

-out generators, switchyard, and safety systems. Establish protected equipment controls for Train A equipment, station black

-out generators, and other portable equipment.

Additional measures to reduce fire risk put into place to stringently control transient combustibles and limit performance of hot work. Additional personnel on

-shift dedicated to compensatory measures.

Long Term Corrective Actions The team reviewed the long

-term corrective actions to be taken by the licensee. At the time of the inspection, the licensee was working through their process es for determining the root cause and extent of condition and engaged several industry experts and engineering organizations for assistance in the analysis of failed mechanical part s. The team concluded the licensee's actions were appropriate and commensurate with the safety significance of the event.

No findings were identified.

3.8 Corrective Actions for Past Similar Failures (Charter Item 10)

a. Inspection Scope

The team reviewed industry operating experience to identify whether there was a history of similar failures or existing preventive action recommendations for this failure mechanism. The team also reviewed the 1989 failure of a master connecting rod in the South Texas Project Emergency Diesel Generator 22 for similarities in root cause, common failure modes, and the scope and effectiveness of corrective actions.

b. Findings and Observations

The Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod failure began as a fretting fatigue. A fretting fatigue crack in the master rod ligament was caused by a crankshaft that was not held in design specified alignment by the crankshaft main bearings in the centerframe. The most probable cause for the high cycle fatigue failure of the Palo Verde Emergency Diesel Generator 3B cylinder 9R master connecting rod on December 15, 2016, was centerframe main bore misalignment from the 1986 master connecting rod failure.

In 1986, there was no known industry information pertaining to centerframe main bore misalignments, and as such, the licensee did not have reason to verify centerframe main bore alignment by removing the crankshaft. The measured as

-found centerframe main bore alignment during the Emergency Diesel Generator 3B repairs following the 2016 master connecting rod failure confirmed the centerframe main bore alignment was outside of the Cooper

-Bessemer design specification. The inspectors determined the licensee's preliminary conclusion that the misalignment had occurred following the 1986 failure was reasonable.

No findings were identified.

3.9 Post-Maintenance Testing (Charter Item 11)

a. Inspection Scope

The team evaluated the licensee's planned post

-maintenance testing to demonstrate the operability and reliability of the repaired emergency diesel generator. The team compared the scope of the testing with vendor manual recommendations, technical specifications, Regulatory Guide 1.9, Institute of Electrical and Electronics Engineers Standard 387(IEEE 387), and the post

-repair testing performed following the 1986 event for demonstrating operability and reliability of the emergency diesel generator.

The team reviewed the scope of the work packages performed to assess whether the post-maintenance testing scope was also appropriate for the work performed.

b. Findings and Observations

The licensee planned an extensive testing program prior to returning Emergency Diesel Generator 3B to service. The team concluded that the planned testing was in conformance with vendor recommendations for testing new engines. It also complied with IEEE 387 and Regulatory Guide 1.9 for testing and establishing adequate reliability of new engines. Additionally, the team concluded that the licensee conducted all applicable surveillance tests and inspections

. The team concluded that the tests were adequate and consistent with the post

-maintenance testing program. The team also reviewed the scope of the planned performance monitoring and concluded that it was appropriate and consistent with the licensee's past practices.

No findings were identified.

4OA6 Meetings, Including Exit

On February 10, 2017, the team presented the inspection results to Ms. M. Lacal, Senior Vice President, Nuclear Regulatory & Oversight, and other members of the licensee staff who acknowledged the findings. The team confirmed that all proprietary information reviewed during this inspection was returned to the licensee.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee personnel

G. Andrews, Director, Nuclear Regulatory Affairs

D. Bence, Assistant Plant Manager, Operations

B. Berles, Director, Nuclear Fuels

P. Bury, Director, Nuclear Training

S. Dornseif, Compliance Consultant, Nuclear Regulatory Affairs

D. Elkinton, Acting Section Leader, Nuclear Regulatory Affairs

J. Fearn, Manager, Emergency Preparedness

J. Glass, Department Lead, Maintenance Fix It Now Team

K. Graham, Director, Nuclear Engineering

M. Hooshmand, Department Leader, Nuclear Assurance

T. Horton, Director, Operations

C. Kharrl, Vice President Site Operations, General Plant Manager

M. Kura, Compliance Department Leader, Nuclear Regulatory Affairs

M. Lacal, Vice President, Nuclear Regulatory & Oversight

M. McGhee, Department Leader, Nuclear Regulatory Affairs

M. Radspinner, Department Leader, Nuclear Engineering

B. Rash, Vice President, Nuclear Engineering

C. Schingeck, Manager, Operations

J. Schrock, Engineer III, System Engineering, Balance of Plant

L. Weaver, Senior Engineer, Nuclear Regulatory Affairs

Contractors

R. Cryer, Director of Internal Combustion Engines, MPR

A. Killinger, Senior Associate, MPR

D. Zink, Diesel Generator System Engineering, South Texas Project

NRC personnel

C. Peabody, Senior Resident Inspector

D. You, Resident Inspector

LIST OF DOCUMENTS REVIEWED

Condition Reports

16-04152 16-19864 16-20302 17-01284 17-01577 17-01578 17-01793

Palo Verde Action Requests (PVARs)

3861204

4435983

4633702

4634167

4635553

Condition Report

/ Disposition Requests (CRDRs)

2669379

Procedures

Number Title Revision/Date

16DP-OEP35 Actual Event Investigation

40AL-9DG02 Diesel Generator B Alarm Panel Responses

40DP-90P08 PVNGS Nuclear Administrative and Technical Manual

- Diesel Generator Test Record

40DP-9DG02 Emergency Diesel Generator B

40ST-9ZZ37 Inoperable Power Sources Action Statement

43TI-3DG02 Emergency Diesel Generator B Operations for Post Maintenance Test

73ST-9DG08 Class 1E Diesel Generator Load Reje ction and 24 Hour Rated Load and Hot Start Test Train B

GSM-WI-60-07 Goltens:

Instruction for Engine Run

-In Procedures

ODP-31 Operations Department Practices.

Operations Emergency Plan Implementation Engineering Reports Number Title Revision/Date

16-15545-027 Engineering Evaluation

- Perform the risk insight assessment required to meet SR 3.8.1.9 Note and Bases to perform a single largest load rejection test for Unit 3 Diesel Generator "B" in Mode 1.

February 3, 2017

16-19864-018 Engineering Evaluation

- Unit 3 'B' Emergency diesel Generator Flush Plan.

16-19864-037 Engineering Evaluation

- Unit 3 B Emergency Diesel Generator Failure.

December 30, 201 6 16-19864-043 Engineering Evaluation

- Unit 3 B Emergency Diesel Generator Failure.

Request for Additional Information for support for Second License Amendment Request.

January 1, 2017

16-19864-057 Engineering Evaluation

- Unit 3B experienced a catastrophic engine failure during monthly surveillance testing per 40ST

-9DG02. A retest plan and basis following the extensive maintenance needs to be developed.

January 23, 2017

17-01578-006 Engineering Evaluation

- Assess Unit 3, Train B, Emergency Diesel Generator (EDG 3B) rapid over pressurization of the crankcase to determine if the other EDGs are susceptible to the same condition.

February 5, 2017

2-DG-028 Engineering Evaluation Request.

Emergency Diesel Generator.

Cooper

-Bessemer revised the

KSV Operation and Maintenance Manual, Section 15.

Evaluate the impact on the plant.

July 8, 1992

Miscellaneous Documents

Number Title Revision/Date

Operating Experience

- External Review February 3, 2017

Operating Experience

- Internal Review February 3, 2017

Unit 3 Operations Log December 15, 2016

KSV Operating and Maintenance Manual Section 15, Version 15RR1

-6/92 July 2, 1992

Emergency Diesel Generator 1A Work History with Actions Taken January 31, 2017

Emergency Diesel Generator 1B Work History with Actions Taken January 31, 2017

Emergency Diesel Generator 2A Work History with Actions Taken January 31, 2017

Emergency Diesel Generator 2B Work History with Actions Taken January 31, 2017

Emergency Diesel Generator 3A Work History with Actions Taken January 31, 2017

Emergency Diesel Generator 3B Work History with Actions Taken February 1, 2017

Miscellaneous Documents

Number Title Revision/Date

A

-3A and 3B Corrective Maintenance Timeline for Emergency Diesel Generator 3A and 3B

February 3, 2017

EDG PV3B #9 Master Rod Failure Event on 12/15/16

CR 16-19864,

Extent of Condition & Extent of Cause February 8, 2017

Preliminary Assessment of Transportability and Functionality of Palo Verde

DGs to South Texas Project's (STPs) Thrown Rod Event February 3, 2004

Unit 3 Control Room Operations Log December 15, 2016

Arizona Nuclear Power Project Metallurgical Investigation Report PVNGS Unit 3 Diesel Engine Failure February 4, 1987

A listing of Palo Verde Action Requests for the Unit 3B EDG from January 1, 2014

through June , 2015

February 3, 2017

A listing of Palo Verde Action Requests for the Unit 3B EDG from December 16, 2016 through February 2, 2017. February 2, 2017

3MDGBH01

Unit 3 B EDG Engine Oil Sample 53647

November 29, 2016

3MDGBH01

Unit 3 B EDG Engine Oil Sample 54024

February 7, 2017

3MDGBH01

Unit 3 B Emergency Diesel Generator and Auxiliaries Vibe Levels December 15, 2016

Inspection and Parts lists for Unit 3 B EDG Rebuild.

February 7, 2017

Station Response to Unit 3 Train B Diesel Generator Failure - Palo Verde Nuclear Generating Station January 26, 2017

Unit 3 "B" EDG Stitch Repairs and Locations (Summarized from DEC

-1069) February 6, 2017

04-095 Unit 1B Diesel Connecting Rod Ultrasonic Examination Report April 9, 2004

04-175 Unit 1A Diesel Connecting Rod Ultrasonic Examination Report April 19, 20

4 04-271 Unit 3A Diesel Connecting Rod Ultrasonic Examination Report June 30, 2004

04-289 Unit 3B Diesel Connecting Rod Ultrasonic Examination Report August 11, 2004

04-290 Unit 2B Diesel Connecting Rod Ultrasonic Examination Report August 18, 2004

04-350 Unit 2A Diesel Connecting Rod Ultrasonic Examination Report September 22, 2014

06-186 Unit 3A Diesel Connecting Rod Ultrasonic April 20, 2006

Miscellaneous Documents

Number Title Revision/Date Examination Report

07-289 Unit 1A Diesel Connecting Rod Ultrasonic Examination Report June 13, 2007

08-128 Unit 2A Diesel Connecting Rod Ultrasonic Examination Report April 3, 2008

09-143 Unit 3B Diesel Connecting Rod Ultrasonic Examination Report April 9, 2009

09-532 Unit 2B Diesel Connecting Rod Ultrasonic Examination Report October 8, 2009

10-205 Unit 1B Diesel Connecting Rod Ultrasonic Examination Report April 8, 2010

11-537 Unit 3A Diesel Connecting Rod Ultrasonic Examination Report July 18, 2011

11-732 Unit 1A Diesel Connecting Rod Ultrasonic Examination Report October 16, 2011

2-614 Unit 2A Diesel Connecting Rod Ultrasonic Examination Report September 11, 2012

13-841 Unit 3B Diesel Connecting Rod Ultrasonic Examination Report October 29, 2013

14-216 Unit 2B Diesel Connecting Rod Ultrasonic Examination Report March 28, 2014

14-880 Unit 1B Diesel Connecting Rod Ultrasonic Examination Report September 16, 2014

2MDGAH01 Comparison of Cylinder Block Vibration Readings With Cylinder #9.

December 29, 2016

476-00429-MM/WA Letter from Mike Melton, ISI to Distribution.

Diesel Connecting Rods NDE Report January 27, 2005

87-2 Palo Verde Nuclear Generating Station

- Magnetic Particle Examination Report January 13, 1987

ANPP-40058 Final Report

- RER QSE 86

-47

ANPP-40058-JGH/DJW/DRL

-92.11 Letter - A rizona Nuclear Power Project to U.S. NRC.

Final Report

-

RER-QSE 86-47.

A 50.55(e) and 10CFR21 Reportable Condition Relating to Diesel Generator Engine Failure.

File:

87

-006-216 February 9, 1987

B/ANPP-M-158450

MIC 262091 Letter - Bechtel Power Corporation to Arizona Nuclear Power Project.

Arizona Nuclear Power Project, Bechtel Job 10407, 10DCFR50.55(e) Reports:

Deficiency Evaluation Report (DER)

No. 86

-33, File D.4.33 March 12, 1987

Miscellaneous Documents

Number Title Revision/Date

CTC-17-001 DP Engineering - Independent Review of Failure Recovery Retesting for PVNGS Emergency Diesel Generator 3B

January 26, 2017

EER 92-DG-028 Engineering Evaluation Request for revised KSV Operating and Maintenance Manual, Section 15

July 8, 1992

ICES # 207741

OE 17536 - Standby Diesel Generator Failure During a Surveillance Run December 9, 2003

IEEE 387-1972 IEEE Trial

-Use Standard: Criteria for Diesel

-Generator Units Applied as Standby Power Supplies for Nuclear Power Generating Stations

1972 IEEE 387-1977 IEEE Standard Criteria for Diesel

-Generator Units Applied as Standby Power Supplies for Nuclear Power Generating Stations

1977 IEEE 387-1984 IEEE Standard Criteria for Diesel

-Generator Units Applied as Standby Power Supplies for Nuclear Power Generating Stations. 1984

LTT-0115-1005-1 Letter: Mr. A. Killinger to Mr. B. Rash.

Independent Technical Assessment of EDG 3B as Compared to

EDG 3A at Palo Verde Nuclear Generating Station.

December 29, 2016

PNL-6287 Battelle Technical Evaluation Report:

Study Group Review of Nuclear Service Diesel Generator Testing and Aging Mitigation.

March, 1988

Regulatory Guide 1.9 NRC Regulatory Guide 1.9.

Selection, Design, Qualification, and Testing of Emergency Diesel Generator Units used as Class 1E Onsite Electric Power Systems at nuclear Power Plants.

VTD-C628-0069 Cooper Energy Services Service News KSV Engines, Commercial and Nuclear Piston Rings (Bub. #752)

September 14, 1992

Vendor Manuals Number Title Revision/Date

15RR1-6/92 Cooper-Bessemer KSV Operation and Maintenance Manual, Section 15

July 2, 1992

VTD-C628-00051 Cooper Energy Services Instruction Manual for KSV Turbocharged Diesel Generating Unit for Nuclear Power Plant Emergency Stand

-By Service [Pub.# 010997]. 15

Work Orders

2927

211410

869599

2668903

047602

577549

873052

2668904

071034

577549

899853

2668908

082533

632333

901934

2668909

083213

632334

907716

2668910

083214

668989

907717

3485615

083215

662391

924074

4178217

083216

662442

924086

4851757

083217

668989

1190720

48446831

084842

684636

2341038 16-15545-027

089838

692381

2392385 16-19864

090634

692442

2668898 17-01578-006

143110

824086

January 27, 2017

MEMORANDUM TO:

Ron Kopriva, Senior Reactor Inspector Engineering Branch 1

Division of Reactor Safety

FROM:

Kriss M. Kennedy, Regional Administrator /RA/

SUBJECT: CHARTER FOR THE NRC SPECIAL INSPECTION TEAM AT PALO

VERDE NUCLEAR GENERATING STATION

-

REVIEW OF LICENSEE ACTIONS RELATED TO THE FAILURE OF THE UNIT 3 TRAIN B EMERGENCY DIESEL GENERATOR

On December 15, 2016, the Unit 3 train B emergency diesel generator experienced a significant mechanical failure during testing that resulted in ejection of a connecting rod.

That same emergency diesel generator previously experienced a significant mechanical failure on December 23, 1986, that also resulted in ejection of a connecting rod from the same cylinder.

The cause of the failures may or may not be similar.

Because of the potential generic implications, the potential that the failures may be repetitive, and the risk significance of the emergency diesel generator failure, a special inspection team is being chartered.

You are hereby designated as the team leader.

A. Basis

On December 15, 2016, during a scheduled surveillance test run of the Unit 3 train B emergency diesel generator, with the emergency diesel generator loaded to approximately 50 percent of full load, the control room received a low lube oil emergency trip annunciator.

The area operator reported a large amount of smoke and oil, and large metal debris that had been expelled from the engine.

The licensee declared an ALERT at 0410 [MST] based on an explosion resulting in visible damage to a safety system required for safe shutdown (HA2.1).

Palo Verde Fire Department responded and no fire was observed.

The Alert was terminated at 0636 [MST].

No other safety functions were impacted.

No personnel injuries occurred.

The licensee determined there had been a catastrophic failure of a connecting rod for cylinder 9 causing crankcase damage and engine internal parts to be ejected from the crankcase.

This same emergency diesel generator experienced a catastrophic failure of the same cylinder (9) in 1986.

Connecting rod 9R failed during the startup testing program in 1986, and the current failure is also being attributed to connecting rod 9R.

This engine

UNITED STATES

NUCLEAR REGULATORY COMMISSION

REGION

IV 1600 E. LAMAR BLVD

ARLINGTON, TX 76011

-4511

R. Kopriva

2 has approximately 3500 run time hours since the repair in 1986.

Palo Verde emergency diesel generators are Cooper

-Bessemer Model KSV

-20.

Two other similar Coop er-Bessemer diesel generator failures have occurred, both at South Texas Project.

In 1989, South Texas Project emergency diesel generator 22 had a connecting rod for cylinder 4 fail, and in 2003 emergency diesel generator 22 had a connecting rod for cylinder 9 fail.

The Palo Verde licensee last performed ultrasonic testing of the connecting rods for Unit 3 train B emergency diesel generator in October 2013, and did not identify any deficiencies.

During a December 20, 2016, tele

-conference with the Palo Verde licensee, NRR, and Region IV, the licensee stated their intent to identify the root cause of the failure, and to evaluate the extent to which the condition could affect the remaining emergency diesel generators at Palo Verde.

On December 23, 2016, the NRC issued an emergency license amendment (ADAMS ML16358A676) to extend the allowed outage time for the Unit 3 train B emergency diesel generator to a total of 21 days based on compensatory measures implemented by the licensee.

Subsequently, on January 4, 2017, the NRC issued a second emergency license amendment (ADAMS ML17004A020) to extend the allowed outage time for the emergency diesel generator to a total of 62 days based on the calculated overall plant risk with compensatory measures in place for the period of time needed to complete repairs to the engine.

A special inspection team will be dispatched to better understand the cause of the emergency diesel generator failure, the extent of condition, the potential generic implications, and the corrective actions proposed and taken by the licensee.

The resident inspection staff at Palo Verde conducted considerable inspection effort during the review of the amendment requests and will provide additional inspection support as needed.

A preliminary risk analysis performed by a Senior Reactor Analyst resulted in an estimated Incremental Conditional Core Damage Probability greater than 1 E

-06.

B. Scope The team is expected to perform data gathering and fact

-finding in order to address the following:

1. Provide a recommendation to Region IV management as to whether the inspection should be upgraded to an augmented inspection team response.

This recommendation should be provided by the end of the first day on site.

2. Develop a chronology of the emergency diesel generator failure and operator response.

3. Review and assess the adequacy of operator response to the emergency diesel generator failure.

4. Review the current status of the licensee's cause determination and determine whether it is being conducted at a level of detail commensurate with the safety significance of the event, including review of relevant plant

-specific and industry (foreign and domestic) operating experience and previous diesel generator failures.

R. Kopriva

3

5. Review the circumstances associated with the emergency diesel generator failure to identify potential common failure modes and generic safety concerns.

6. Review records associated with the maintenance history for the emergency diesel generators at Palo Verde, including previous mechanical failures.

7. Review the licensee's program for periodic monitoring and maintenance of the emergency diesel generators, including inspection and assessment techniques and scope, periodicity, and the results of past inspections.

8. Review and assess the adequacy of the licensee's ongoing evaluation of extent

-of-condition as it relates to the other emergency diesel generators in Units 1, 2, and 3.

9. Review and assess the licensee's prompt and long

-term corrective actions to address the root and probable causes of the condition.

Assess the adequacy of repair activities and independently verify information submitted in support of NRC review of any regulatory relief requests.

10. If applicable, review and assess the corrective actions for past similar failures, including vendor recommended actions to prevent such failures.

11. Review and assess the licensee's testing program to confirm the operability of the Unit 3 train B emergency diesel generator following repair activities.

C. Team Members Ron Kopriva, Team Leader Dustin Reinert, Team Member

D. Guidance

This memorandum designates you as the special inspection team leader.

Your duties will be as described in Inspection Procedure 93812, "Special Inspection."

The team composition has been discussed with you directly.

During performance of the special inspection activities assigned to them, designated team members are separated from their normal duties and report directly to you.

The team is to emphasize fact finding in its review of the circumstances surrounding the event, and it is not the responsibility of the team to examine the regulatory process.

Safety concerns identified that are not directly related to the event should be reported to the Region IV office for appropriate action.

You should notify the licensee and the team should begin inspection activities on or before February 6, 2017, based on the licensee's schedule of activities.

You should conduct an entrance meeting with the licensee at the appropriate time at the site.

A report documenting the results of the inspection, including findings and conclusions, should be issued within 45 days of the exit meeting conducted at the completion of the inspection.

While the team is active, you will provide periodic status briefings to Region IV management.

This Charter may be modified should the team develop significant new information that warrants review.

Should you have any questions concerning this Charter, contact Geoffrey Miller, Chief, DRP Branch D, at 817

-200-1173.

Diagram of Connecting Rod Articulated Rod Master Rod Site of crack initiation (on the master rod under the bearing shell)

ML17100A130

SUNSI Review

ADAMS:

Non-Publicly Available Non-Sensitive Keyword:

By:JLD/RDR

Yes

No

Publicly Available Sensitive

NRC-002 OFFICE DRP/PBD:RI

DRS/EB1 DRS:SRA DRP/PBD:C

NAME DReinert RKopriva RDeese GMiler

SIGNATURE /RA-E/ /RA/ /RA/ /RA/

DATE 3/21/17 3/21/17 3/22/17 4/10/17