05000369/LER-2014-001

LER-2014-001, Condition Prohibited by Technical Specifications (TS) due to Emer ency Diesel Generator 1B Failure.

Mcguire Nuclear Station, Unit 1
Event date:	08-21-2014
Report date:	12-17-2014
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
Initial Reporting
3692014001R01 - NRC Website

Reported lessons learned are incorporated into the licensing process and fed back to industry.

Send comments regarding burden estimate to the FOIA, Privacy and Information Collections Branch (T-5 F53), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by intemet e-mail to infocollects Resource@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NE0E3-10202, (3150-0104), Office of Management and Budget, Washington, DC 20503. If a means used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

BACKGROUND:

The following information is provided to assist readers in understanding the event described in this LER.

Applicable Energy Industry Identification [EIIS] system and component codes are enclosed within brackets.

McGuire's unique system and component identifiers are contained within parentheses.

For McGuire, the onsite standby power source for each 4160 volt Engineered Safety Feature (ESF) bus is a dedicated Emergency Diesel Generator [DG] (EDG). An EDG starts automatically on a safety injection (SI) signal (i.e., low pressurizer pressure or high containment pressure signals) or on an ESF bus degraded voltage or undervoltage signal. After the EDG has started, it will automatically tie to its respective bus after offsite power is tripped as a consequence of ESF bus undervoltage or degraded voltage, independent of or coincident with an SI signal. The EDGs will also start and operate in the standby mode without tying to the ESF bus on an SI signal alone. Following the trip of offsite power, a sequencer strips loads from the ESF bus. When the EDG is tied to the ESF bus, loads are then sequentially connected to its respective ESF bus by the automatic load sequencer.

The sequencing logic controls the permissive and starting signals to motor breakers to prevent overloading the EDG by automatic load application.

Each EDG unit is rated for continuous operation at 4000 kilowatts (kW) with added capacity to operate between 4200 — 4400 kW for a period of two hours out of every twenty-four hours of operation without adversely affecting the life of the unit. The design basis accident load level for each of the redundant systems does not exceed the 4000 kW continuous rating of the EDG unit assigned to each system.

The prime movers for the McGuire EDGs are Nordberg diesel engines. The EDGs have 16 cylinders, each of which has one inlet valve and one exhaust valve. These valves are retained in guides pressed into the cylinder head. The inlet and exhaust valves are closed by spring force exerted on the valve stems through spring retainers and split rings.

The inlet valve is made of two parts, the head and the stem, welded together and then chrome plated. There is a groove at the top of the stem for the split ring, also known as a keeper, to capture the stem and secure the spring retainer to the stem. The groove of the valve stem allows two keepers to fit tightly around the valve, and therefore transfer the spring force for closure to the stem. When installed, the keepers hold the spring retainer to the valve stem securely. The two large valve springs maintain pressure on the spring retainer to close the valve.

The Nordberg engine has variable inlet valve timing. The inlet valves are equipped with hydraulic lash adjusters which support the variable timing of the inlet valve. As load increases, the lash adjuster expands to maintain solid contact. During engine operation the hydraulic valve lash adjuster cycles approximately 4.28 times per second. The internal fit-up, spring forces, oil viscosity, temperature, and degree of link shaft advance all affect the function of the lash adjuster. If the lash adjuster is unable to maintain its expansion under the compressive force in the valve train (i.e., it collapses), the associated tappet will impact the seat on the valve stem with a higher force.

APPROVED BY OMB: NO. 3150-0104 EXPIRES: 01/31/2017 Reported lessons learned are incorporated into the licensing process and fed back to industry.

Send comments regarding burden estimate to the FOIA, Privacy and Information Collections Branch (T-5 F53), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by intemet e-mail to infocollects Resource@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0104), Office of Management and Budget, Washington, DC 20503. If a means used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

Engine Signature Analysis (ESA) is a means to monitor diesel engines in service and allows for condition based maintenance rather than time based maintenance. McGuire collects this data every six months (via PM program) for engineering and maintenance information. ESA consists of the collection and evaluation of cylinder firing pressure, vibration, and ultrasonic signatures. The signatures are used to assess, compare, and trend performance of the cylinders, which allows both the fuel injection system and the mechanical condition of each cylinder to be evaluated.

TS Limiting Condition of Operation (LCO) 3.8.1, Required Action B.4 governs AC Sources - Operating for Modes 1, 2, 3, and 4. LCO 3.8.1 requires in part that two EDGs be operable. Required Action B.4 states that with one EDG inoperable, the EDG must be restored to operable status within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. Condition G states that with the Required Actions and associated Completion Times of Condition B not met, the unit must be in Mode 3 within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and in Mode 5 within 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />.

SR 3.8.1.14 verifies that every 18 months each EDG, when connected to its bus in parallel with offsite power and operating with maximum Kilovolt-ampere (kVAR) loading that offsite power conditions permit, operates for >/= 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />s:

a. For >/= 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> loaded >1= 4200 kW and b. For the remaining hours of the test loaded >1= 3600 kW and At the time of the event, repair and testing of the EDG 1B was anticipated to not be completed within the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> completion time mandated by TS 3.8.1, Required Action B.4. Therefore, Duke Energy requested the Completion Time of the Required Action be extended from the current 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> by an additional 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br />, for a total of 120 hours0.00139 days <br />0.0333 hours <br />1.984127e-4 weeks <br />4.566e-5 months <br />, to allow McGuire Unit 1 to remain in Mode 1 (Power Operation) until repairs could be completed and testing to demonstrate operability of EDG 1B was completed. The NOED was subsequently granted by the NRC on the same day for an additional 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> of unit operation. EDG 1B was restored to operable status approximately 34 hours3.935185e-4 days <br />0.00944 hours <br />5.621693e-5 weeks <br />1.2937e-5 months <br /> into the 48 hour5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> period allowed by the NOED.

This event is being reported under 10 CFR 50.73(a)(2)(i)(B), any operation or condition which was operated in Mode 1 at 100% power. No significant structures, systems or components were out of service at the time of discovery such that they contributed to the event.

EVENT DESCRIPTION:

On August 18, 2014, at 1718, approximately 14 hours1.62037e-4 days <br />0.00389 hours <br />2.314815e-5 weeks <br />5.327e-6 months <br /> into a 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> surveillance test of EDG 1B, cylinder 5L exhaust temperature dropped from a steady 741 degrees Fahrenheit (F) down to 611 degrees F. Control room and local power indications were observed to be fluctuating. Operations directed a procedural shutdown (stepping down load) of EDG 1B. An operator in the field reported an unusual sound from the engine when load was reduced. Operations immediately unloaded and stopped EDG 1B and subsequently declared EDG 1B inoperable on August 18, 2014, at 1729 hours0.02 days <br />0.48 hours <br />0.00286 weeks <br />6.578845e-4 months <br />. Maintenance activities commenced to inspect and repair EDG 1B.

APPROVED BY OMB: NO. 3150-0104 EXPIRES: 01/31/2017 Reported lessons learned are incorporated into the licensing process and fed back to industry.

Send comments regarding burden estimate to the FOIA, Privacy and Information Collections Branch (T-5 F53), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by internet e-mail to infocollects Resource@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0104), Office of Management and Budget, Washington, DC 20503. If a means used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

During the initial disassembly of EDG 1B cylinder 5L for repair, the inlet valve spring was discovered in a relaxed state (not compressed) indicating that the inlet valve and/or stem was broken and was no longer captured by the valve stem keeper. Further examination concluded the inlet valve failed at the bottom of the keeper groove on the valve stem. This caused the valve to repeatedly contact the valve seat insert which broke the insert into small pieces. The insert pieces fell into the cylinder damaging the piston, head, exhaust valve, and cylinder liner. The damage was confined to EDG 1B cylinder 5L with the exception of some broken pieces of valve seat that entered the engine exhaust header. The pieces were swept down the exhaust header and stopped by the turbocharger inlet protection screen.

On August 21, 2014, at 1300, McGuire requested a NOED for an additional 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> to TS LCO 3.8.1, Required Action B.4 to allow for repair of EDG 1B. The NOED was subsequently granted by the NRC for an additional 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> of unit operation. The enforcement discretion period began on August 21 at 1529.

On August 23, 2014, at 0335, the operability test was completed satisfactorily for EDG 1B.

On August 24, 2014, at 1041, the 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> Surveillance test was completed satisfactorily for EDG 1B.

EDG 1B cylinder 5L parts were disassembled and taken to the Duke Energy metallurgical laboratory for examination and testing with the support of an industry expert.

CAUSAL FACTORS:

The cause of the EDG 1B cylinder 5L failure is high cycle fatigue of the inlet valve due to the combined effects of three inlet valve train parts which were manufactured outside of original equipment manufacturer (OEM) specified tolerances.

The following factors caused the 5L inlet valve failure:

• Inlet valve stem hardness/strength below specification

• Hydraulic lash adjuster internal inadequate clearance

• Inlet valve keeper inadequate upper ID radius and thickness Interaction of the above factors subjected the cylinder 5L inlet valve to forces beyond the design and endurance limit. Evidence of inadequate manufacturing was confirmed by inspection, measurement, and testing of the EDG 1B cylinder 5L inlet valve train components. No single factor on its own can cause an inlet valve failure. A combination of the above factors created the conditions necessary for a failure of the inlet valve from high cycle fatigue over a relatively long time period. EDG 1B operated for approximately 13 years with the out of tolerance parts. All of the above parts issues were created during manufacturing by inadequate adherence to original Nordberg specifications for materials and tolerances by a third party supplier that is no longer in operation.

Reported lessons learned are incorporated into the licensing process and fed back to industry.

Send comments regarding burden estimate to the FOIA, Privacy and Information Collections Branch (T-5 F53), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by internet e-mail to infocollects Resource@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0104), Office of Management and Budget, Washington, DC 20503. If a means used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

CORRECTIVE ACTIONS:

Immediate:

1. Replaced the affected parts for EDG 1B Cylinder 5L.

2. Performed series of EDG 1B break-in runs following replacement of parts.

3. Performed EDG 1B Operability test.

4. Performed EDG 1B 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> Surveillance test.

Subsequent:

1. Duke Energy's Metallurgy Laboratory analyzed the affected cylinder components.

2. Determined by ESA historical review that the EDG 1B cylinder 5L signature was atypical with respect to some of the other cylinder signatures. Further review determined the EDG 1B cylinders 4L, 6L, and 3R also had similar ESA responses (see note below).

3. Confirmed no other unit 1 or unit 2 EDG cylinders have atypical ESA responses.

4. Determined EDG 1B cylinder 5L hydraulic lash adjuster was outside manufacturer's tolerance. Similar hydraulic lash adjusters were discovered on cylinders 4L, 6L, 3R, and in warehouse stock (removed).

5. The hardness (tensile strength) of EDG 1B 5L, 4L, 6L, and 3R inlet valves was analyzed.

6. Replaced EDG 1B cylinder heads 4L, 6L, and 3R. Replaced hydraulic lash adjusters for 4L, 6L and 3R with new parts pre-tested to function properly.

7. Disassembled turbocharger and inspected for further damage.

8. Performed ESA on EDG 1B after maintenance. Results were satisfactory.

9. Placed a hold on purchase of the following EDG parts pending revision of procurement specifications: inlet valve, exhaust valve, hydraulic lash adjuster, and valve keepers.

Planned:

1. Revise the procurement specification to add additional vendor verification of the following parts to meet Nordberg specification: inlet valve, exhaust valve, hydraulic lash adjuster, and valve keepers.

2. Identify and replace other suspect lash adjuster, valve stem, and valve keeper parts on EDGs 1A, 1B, 2A, and 2B regardless of ESA signature acceptability. Inspection complete on EDG 1A.

Note: McGuire attempted to address the elevated ESA responses in the 1990s by replacing the lash adjusters and having a third party supplier test those removed. The test performed by the third party supplier indicated there were no issues with the lash adjusters. The lash adjusters were replaced with similar out of tolerance parts made by the third party supplier. The cause evaluation determined that the failure occurred because of the combined effects of inadequate adherence to Nordberg lash adjuster, valve stem, and valve keeper specifications. The latter could only have been determined after the Duke performed independent inspections and tests of each part during the cause evaluation. The collection of ESA data is done by some members of the industry but not others. There was previously no original equipment manufacturer (OEM) guidance or industry best practice relative to ESA data evaluation. McGuire previously used this data to qualitatively compare the relative performance of properly operating cylinders. Since the occurrence of the failure of the EDG 1B 5L cylinder inlet valve, the review of ESA data looks for specific indications of lash adjusters which are not operating correctly when compared to lash adjusters of properly operating cylinders.

Reported lessons learned are incorporated into the licensing process and fed back to industry.

Send comments regarding burden estimate to the FOIA, Privacy and Information Collections Branch (T-5 F53), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by internet e-mail to infocollects Resource@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0104), Office of Management and Budget.

Washington, DC 20503. If a means used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

SAFETY ANALYSIS:

The nuclear safety significance of this event was that the site operated for a period of time outside the technical specification limits with the EDG 1B inoperable. The associated risk, including the Probabilistic Risk Assessment analysis, of this condition was evaluated as part of the NOED and was found to be acceptable.

The redundant EDG 1A was operable and available and could have performed its intended safety function at all times while the EDG 1B was tagged out to support maintenance and testing.

ESA data for EDG 1B cylinder 5L indicated an atypical response as compared to other cylinders. Subsequent analysis indicated an out of tolerance hydraulic lash adjuster caused the atypical ESA response. When EDG 1B cylinder 5L inlet valve was replaced and a new hydraulic lash adjuster installed, testing confirmed the atypical ESA response was eliminated. Laboratory analysis also indicated the hardness of the inlet valve stem was less than that specified by the vendor. The hydraulic lash adjuster and inlet valve were replaced on cylinder 5L as part of the overall cylinder head replacement.

EDG 1B cylinders 4L, 6L, and 3R also exhibited atypical ESA responses. The hydraulic lash adjusters for these cylinders were tested during the 2014 fall unit 1 outage. Results indicated these cylinders also had out of tolerance hydraulic lash adjusters; however, the as-found performance of these hydraulic lash adjusters was better than that of 5L during testing. Further evaluation indicated these hydraulic lash adjusters were from the same lot number as EDG 1B cylinder 5L's hydraulic lash adjuster.

The hardness of EDG 1B cylinder 4L, 6L, and 3R inlet valves were also evaluated. Cylinder 4L inlet valve hardness was found below specification; however, 4L had a better performing hydraulic lash adjuster than 5L during testing. Although the 4L, 6L, and 3R inlet valves operated without issue during the August 23, 2014, 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> test, the hydraulic lash adjusters and cylinder heads were conservatively replaced during the 2014 fall unit 1 outage.

Evaluation determined no operability concerns exist with installed inadequate inlet valve keepers and weaker inlet valve stem material. EDG 1B operated for approximately 13 years with the out of tolerance parts. The inadequate keepers and weaker inlet valve stems have to be combined with the inadequate hydraulic lash adjuster internal clearances to cause an inlet valve failure. ESA has confirmed there are currently no installed inadequate valve lash adjusters on the EDGs.

The extent of condition review determined that the EDGs 1A, 2A, and 2B were not susceptible to the same failure mode as EDG 1B based on the ESA of the engines. EDGs 1A, 2A, and 2B cylinders do not exhibit ESA atypical responses similar to that of EDG 1B cylinders 5L, 4L, 6L, and 3R. The scope of this issue was confirmed to be limited to the four cylinders with the lash adjusters from the same lot only installed on EDG 1B.

The lash adjusters of the identified lot have been confirmed to not be installed in the 1A, 2A, and 2B EDGs.

Given the above, this event was determined to be of no significance to the health and safety of the public.

Reported lessons learned are incorporated into the licensing process and fed back to industry.

Send comments regarding burden estimate to the FOIA, Privacy and Information Collections Branch (T-5 F53), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by internet e-mail to infocollects Resource@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0104), Office of Management and Budget, Washington, DC 20503. If a means used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

ADDITIONAL INFORMATION:

A review of the McGuire corrective action program was conducted to determine whether this was a recurring event (i.e., similar event with the same cause code). No other EDG failures associated with high cycle fatigue have been documented within the past five years. Therefore, this is not considered a recurring event.

10 CFR Part 21 Applicability This event was evaluated for 10 CFR Part 21 applicability in accordance with Duke Energy administrative procedures. This event was determined to be not reportable in accordance with 10 CFR Part 21. The details of this evaluation are documented in the McGuire corrective action program.