IR 05000416/2022050

From kanterella
Jump to navigation Jump to search
Special Inspection Reactive Report 05000416/2022050
ML22291A065
Person / Time
Site: Grand Gulf Entergy icon.png
Issue date: 10/20/2022
From: Jeffrey Josey
NRC/RGN-IV/DORS/PBC
To: Kapellas B
Entergy Operations
Schaup W
References
IR 2022050
Download: ML22291A065 (29)


Text

October 20, 2022

SUBJECT:

GRAND GULF NUCLEAR STATION - SPECIAL INSPECTION REACTIVE REPORT 05000416/2022050

Dear Mr. Kapellas:

On June 3, 2022, the U.S. Nuclear Regulatory Commission (NRC) completed its initial assessment of the failure of the Division 1 standby diesel generator, which occurred on May 16, 2022, at Grand Gulf Nuclear Station. Based on this initial assessment, the NRC sent an inspection team to your site on June 13, 2022.

On September 2, 2022, the NRC completed its special inspection. On September 8, 2022, the NRC inspection team discussed the results of this inspection with you and other members of your staff. The results of this inspection are documented in the enclosed report.

Two findings of very low safety significance (Green) are documented in this report. Both findings involved violations of NRC requirements. We are treating these violations as non-cited violations (NCVs) consistent with Section 2.3.2 of the Enforcement Policy.

If you contest the violations or the significance or severity of the violations documented in this inspection report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN:

Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region IV; the Director, Office of Enforcement; and the NRC Resident Inspector at Grand Gulf Nuclear Station.

If you disagree with a cross-cutting aspect assignment in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region IV; and the NRC Resident Inspector at Grand Gulf Nuclear Station. This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with Title 10 of the Code of Federal Regulations 2.390, Public Inspections, Exemptions, Requests for Withholding.

Sincerely, Signed by Josey, Jeffrey on 10/20/22 Jeffrey E. Josey, Chief Projects Branch C Division of Operating Reactor Safety Docket No. 05000416 License No. NPF-29

Enclosure:

Inspection Report 05000416/20220050 w/Attachment: Special Inspection Charter, dated June 14, 2021

Inspection Report

Docket Number: 05000416 License Number: NPF-29 Report Number: 05000416/2022050 Enterprise Identifier: I-2022-050-0000 Licensee: Entergy Operations, Inc.

Facility: Grand Gulf Nuclear Station Location: Port Gibson, MS Inspection Dates: June 13 to June 17, 2022 Inspectors: E. Lantz, Resident Inspector C. Speer, Reactor Systems Engineer, Team Lead H. Strittmatter, Emergency Preparedness Inspector Approved By: Jeffrey E. Josey, Chief Projects Branch C Division of Operating Reactor Safety Enclosure

SUMMARY

The U.S. Nuclear Regulatory Commission (NRC) continued monitoring the licensees performance by conducting a special inspection at Grand Gulf Nuclear Station, in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information.

List of Findings and Violations

Failure to Establish Measures to Ensure the Division 1 Replacement Temperature Control Valve was Suitable for the Application Cornerstone Significance Cross-Cutting Report Aspect Section Mitigating Green None (NPP) 93812 Systems NCV 05000416/2022050-01 Open/Closed The inspectors identified a Green, non-cited violation of 10 CFR Part 50, appendix B, criterion III because the licensee did not establish for the selection and review for suitability of application of materials, parts, equipment, and processes that are essential to the safety-related functions of the structures, systems, and components. Specifically, the licensee did not establish the suitability of the division 1 standby diesel generator jacket water cooling temperature control valve installed in 2007 to replace one that failed in service. The licensee did not review the suitability of the replacement valve in that the licensee failed to ensure it did not contain the same manufacturing defect that caused the original valves failure.

Failure to Consider Out of Tolerance Measurements in Operability Determinations Cornerstone Significance Cross-Cutting Report Aspect Section Mitigating Green [H.7] - 93812 Systems NCV 05000416/2022050-02 Documentation Open/Closed The inspectors identified three examples of a Green non-cited violation of 10 CFR Part 50, appendix B, criterion V, Instructions, Procedures, and Drawings, which occurred because the licensee did not perform an operability determination in accordance with procedures.

Specifically, the licensees operability determination, when applied to the division 1 and division 2 standby diesel generator jacket water cooling temperature control valves, did not address the potential impact of out-of-tolerance measurements as required by Procedure EN-OP-104, Operability Determination Process, revision 17.

Additional Tracking Items

None.

INSPECTION SCOPES

Inspections were conducted using the appropriate portions of the inspection procedures (IPs) in effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with their attached revision histories are located on the public website at http://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared complete when the IP requirements most appropriate to the inspection activity were met consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection Program - Operations Phase. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel to assess licensee performance and compliance with Commission rules and regulations, license conditions, site procedures, and standards.

OTHER ACTIVITIES

- TEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL

93812 - Special Inspection

In accordance with the attached Special Inspection Team (SIT) Charter, the inspection team conducted a detailed review of the division 1 standby diesel generator (SDG) jacket water cooling temperature control valve (TCV) event which occurred on May 16, 2022, at the Grand Gulf Nuclear Station (hereafter referred to as "the licensee"). Additionally, the inspectors reviewed conditions surrounding both the division 1 and division 2 SDG TCVs from 1999 through 2022 that were potentially related to the May 16, 2022, event.

As detailed in the SIT Charter, the following items were reviewed:

1. Description of Event and Reactive Inspection Basis

On May 16, 2022, the licensee shut down the division 1 SDG during a surveillance test because the associated jacket water cooling TCV did not properly control temperature. This failure caused jacket water and lube oil temperatures to rise at about 1 degree Fahrenheit (F)/minute past the normal expected operating band. Based on this, operations secured the SDG before any temperature limits were reached.

The licensee's subsequent troubleshooting identified a swollen and extruded O-ring and a cracked internal valve component. The licensee determined that the suspect O-ring was not an original equipment manufacturer (OEM) part but had been procured through their commercial grade dedication process and installed approximately 3 years prior (2019). The licensee also determined that a similar O-ring, also commercial grade dedicated, was installed in the division 2 SDG approximately 1 year ago (2021).

Following the event, the resident inspectors reviewed the applicable alarm response instructions and performed walkdowns with station operators to demonstrate taking local control of the TCV.

Based on these walkdowns, the resident inspectors raised concerns that the licensee had insufficient procedural guidance for this issue and did not have all necessary equipment staged to allow personnel to take manual control of the TCV. This called into question the licensees ability to take manual actions before damage occurred to the diesel.

The NRC used Management Directive 8.3, NRC Incident Investigation Program, to evaluate the event and determine the appropriate NRC response. Based on the deterministic criteria and risk insights, the NRC determined the event warranted follow-up via a special inspection. This SIT was chartered to identify the circumstances surrounding this event and review the licensees actions to address the causes of the event.

Upon arrival for the on-site inspection, the SIT determined that the inspection did not warrant an upgrade to an augmented inspection team response.

2. Develop a complete sequence of events and understanding related to the emergency diesel

generator temperature control valve issue(s). The chronology should include the status of the equipment and licensee actions to address the condition(s) up and including the failures beginning in 1999 Inspectors reviewed major work and other events associated on both division 1 and division 2 of the SDG jacket water cooling system since 1999 through the May 16, 2022, event. The inspectors noted three previous events similar in nature to the May 16 event:

July 27, 1999: The division 1 SDG experienced elevated jacket water and lube oil temperatures during a monthly surveillance, including the associated alarms for high temperatures. Operators secured the division 1 SDG and declared it inoperable. The licensees apparent cause evaluation concluded that faulty thermal elements caused the failure of the division 1 SDG TCV and replaced the thermal elements. The NRC later determined that this cause evaluation was inadequate as documented in Special Inspection Report 05000416/2007006 (Agencywide Documents Access and Management System (ADAMS) Accession No. ML071380478).

June 22, 2004: The division 1 SDG experienced elevated jacket water and lube oil temperatures during a monthly surveillance, including the associated alarms for high temperatures. Operators secured the division 1 SDG and declared it inoperable. The licensees root cause evaluation concluded that faulty thermal elements caused failure of the associated TCV and replaced the thermal elements. The NRC determined that this cause evaluation was inadequate as documented in Special Inspection Report 05000416/2007006 (ML071380478).

January 30, 2007: The division 1 SDG experienced elevated jacket water and lube oil temperatures during a monthly surveillance, including the associated alarms for high temperatures. Operators secured the division 1 SDG and declared it inoperable. In response, the NRC launched a special inspection to review the repetitive failures of the division 1 SDG, documented in Special Inspection Report 05000416/2007006 (ML071380478). In response to the failure of the division 1 SDG and the conclusions of the NRC SIT, the licensee performed a root cause evaluation which identified a manufacturing defect associated with the division 1 SDG TCV valve cover, which resulted in binding of the valve internals and its ultimate failure. The licensee replaced the division 1 SDG TCV.

May 16, 2022: The division 1 SDG experienced elevated jacket water and lube oil temperatures during a monthly surveillance, including the associated alarms for high temperatures. Operators secured the division 1 SDG and declared it inoperable. The licensees apparent cause evaluation concluded that an O-ring internal to the division 1 SDG TCV was swollen and extruded, which resulted in binding of the valve internals and its ultimate failure. The licensee noted that the failed O-ring was a commercial grade dedicated part, rather than an OEM part. The licensee substituted the commercial grade dedicated O-rings in place of the OEM O-rings in 2019 on division 1 and 2021 for division 2. On June 1, 2022, the licensee rebuilt the division 1 SDG TCV using new OEM O-rings and a new element tube. On June 2, 2022, the licensee rebuilt the division 2 SDG TCV using new OEM O-rings and a new element tube.

The inspectors developed the following detailed timeline of events for the May 16, 2022, event:

1:05 a.m.: Division 1 SDG is started for a surveillance run.

1:55 a.m.: SDG load is raised to approximately 5.7 MW.

2:05 a.m. (approximately): The LOW TEMP JACKET WATER IN alarm is received.

Operators observed lower temperature than expected and trending down. Operators began working through the alarm response instruction (ARI) for the alarm.

2:11 a.m. (approximately): The HIGH TEMP JACKET WATER IN alarm is received. The setpoint for this alarm is 175 degrees F.

2:16 a.m. (approximately): The HIGH TEMP LUBE OIL OUT alarm is received. The setpoint for this alarm is 190 degrees F. The highest value seen was 192 degrees F and trending up at approximately 1 degree F/minute. The temperature of the jacket water to the heat exchanger is 183 degrees F and rising.

2:19 a.m. (approximately): Operators begin unloading the SDG.

2:23 a.m.: The SDG is fully unloaded.

2:25 a.m. (approximately): Operators hear a loud bang. Following this sound, the operators observed the temperature of the jacket water leaving the heat exchanger rising rapidly toward maximum indication of 130 degrees F and lowering temperature on the jacket water outlet temperature. The licensee attributes this to the TCV becoming un-bound by the O-ring and beginning to function.

2:27 a.m. (approximately): Jacket water outlet temperature indicates 148 degrees F and trending down.

3. Review and assess the adequacy of operator response to the emergency diesel

generator failure that occurred on May 15, 2022 The inspection team reviewed data for plant operating parameters from the event, reviewed station logs and procedures, and interviewed operations and engineering personnel regarding actions that were taken in response to the conditions encountered during the event. The team determined that the operating crew responded appropriately to the failure of the division 1 SDG TCV. The operators in the field responded to the received local alarm for low temperature jacket water in and began to take the required actions from the stations alarm response instruction. Operators in the control room received jacket water and lube oil inlet and outlet temperatures from operators in the field. Operators unloaded and secured the diesel in accordance with the alarm response procedure. Peak temperature at the lube oil outlet and jacket water outlet were reported at 192 degrees F and 182 degrees F respectively, both lower than the automatic trip setpoint for the diesel of 200 degrees F.

The inspectors reviewed licensee procedures and performed interviews and equipment walkdowns with operations and engineering personnel to assess the adequacy of operator response in case of a standby situation that would require the diesel to continue running during a high jacket cooling water or lube oil condition. The inspectors found that, in response to a similar issue in 2007, the licensee updated their procedures to address several issues noted around the response to a high temperature alarm for the diesel generator jacket water and lube oil. Enhancements made by the licensee included pre-staging the tools needed to perform the required actions in a timely manner. A review of these procedures and corrective actions were found to adequately address the concerns from the 2007 event.

After the event on May 16, 2022, the licensee further updated their ARI procedures to improve the guidance to operators for responding to high temperature lube oil and jacket water alarms. Walkdowns of the division 1 and division 2 SDGs, including a walkthrough of the most recent update of the ARI response with an operator, were performed with no concerns noted regarding the operators ability to respond to the situation and do so in a timely manner.

The inspectors documented a finding in the Inspection Results report section below related to the licensee's operability evaluations performed when unexpected conditions were identified as part of the troubleshooting following the event.

4. Review the current status of the licensees cause determination to determine whether it is

being conducted at a level of detail commensurate with the safety significance of the event and includes reviews of relevant plant-specific and industry (foreign and domestic) operating experience and previous diesel generator failures The inspectors concluded that the licensee's cause determination was being performed as an apparent cause evaluation as allowed per the licensee's cause evaluation procedures.

The inspectors determined that the licensee's apparent cause evaluation was being conducted at a level of detail commensurate with the safety significance of the problem. The licensee tentatively identified that the apparent cause of the failure of the installed division 1 SDG TCV was a swollen O-ring internal to the valve that caused binding of the valves internal mechanisms. The licensee sent the failed O-ring as well as similar commercial grade and OEM O-rings to a third-party vendor for failure analysis. The third-party analysis could not determine a conclusive cause for the failure of the non-OEM O-ring. However, it was noted that the non-OEM O-rings were of slightly different composition than the OEM O-rings. This material difference is further discussed in Section 5 of this report.

The inspectors determined through interviews and reviews of draft documents that the cause evaluation appropriately reviewed plant-specific, domestic, and foreign industry operating experience, such as the prior failure of the division 1 SDG TCV at a similar failure at Catawba Nuclear Station in 2008. The inspectors further confirmed that the personnel performing the cause evaluation were appropriately qualified in accordance with the licensees procedural requirements.

5. Review the licensees extent of condition evaluation to determine if the licensee has

adequately considered degradation of similar equipment (i.e., other TCVs or O-rings)

The inspectors determined through interviews and reviews of the licensees draft documents that the licensee appropriately considered degradation of similar TCVs, including both TCVs of the same brand and TCVs of similar design as the division 1 SDG TCV. The licensees extent of condition has not incorporated the degradation of other similar O-rings at the site.

At the time of the inspectors review, the work was pending a third-party vendor evaluation to help determine the possible causes of the May 16, 2022, division 1 SDG TCV O-ring failure, which would inform any additional actions taken by the licensee. The inspectors noted that all of the non-OEM O-rings purchased in the same batch as the O-ring suspected of causing the May 16, 2022, failure have been replaced on the licensees equipment.

Additionally, the remaining O-rings from the batch in storage have been quarantined in the licensees warehouse to ensure they are not installed on plant equipment.

The third-party evaluation found that, although all O-rings were comprised of Buna-N rubber as specified by the valve vendor, the non-OEM O-rings were of a slightly different composition than the original OEM O-rings. The currently available information indicates that this composition difference contributed to the failure of the division 1 SDG TCV O-ring.

However, the inspectors determined that this slight composition difference was not reasonably within the licensee's ability to foresee. The technical manual and procurement documents for the valve only specified Buna-N as the O-ring material and did not make note of a particular composition blend. Additionally, the licensee's dedication process for the commercial grade O-rings was performed in accordance with generally accepted industry practices as discussed in section 8 below.

6. Review and assess the licensees prompt and long-term corrective actions to address the

root and contributing causes Because the licensee performed their analysis as an apparent cause evaluation, root causes for the May 16, 2022, event have not been identified. Through their evaluation, the licensee identified swelling of an installed non-OEM O-ring as the apparent cause of the May 16, 2022, failure of the division 1 SDG TCV. The licensee has sent the failed O-ring, as well as another non-OEM O-ring purchased in the same batch as the failed O-ring, to a third party for failure analysis to assist in determining why it failed. The licensee has also sent a new, never-installed OEM O-ring for comparison.

The licensee replaced the O-rings used in both the division 1 and division 2 SDG TCVs with OEM O-rings and quarantined the remaining non-OEM O-rings purchased in the same batch stored in their warehouse to ensure they are not installed in the future.

The inspectors reviewed the licensees prompt and long-term corrective actions and determined them to be adequate to address the causes the licensee identified.

7. Review records associated with the O-ring maintenance history for the last 15 years for

division 1 and 2 emergency diesel generator temperature control valves at Grand Gulf, including previous failures. Based on this review, assess the licensees maintenance program for periodic monitoring and maintenance of the emergency diesel generators The inspectors reviewed the stations preventive maintenance procedure for replacing the thermal element in the division 1 and division 2 SDG TCV, which also replaces the valve's O-rings. It also requires inspection/cleaning of all internal valve soft parts and surfaces. The inspectors reviewed the completed work orders documenting this preventive maintenance (PM) for the past 15 years for both division 1 and 2. The inspector's review of the work order history and associated deficiencies discovered during the work did not provide insight into the May 16, 2022, event.

8. Review the adequacy of the commercial grade dedication process used by the licensee

associated with the O-ring that apparently caused the TCV to fail on May 15, 2022 The inspection team reviewed data for the commercial grade dedication of O-rings used in the division 1 and 2 SDG TCVs to replace the OEM O-rings, including receipt inspections, quality control reports, engineering evaluations and station procedures, and interviewed engineering personnel.

In 2019, the licensee performed an evaluation of an OEM O-ring used in the division 1 and 2 SDG TCVs to determine the necessary critical characteristics. This was to support ordering 10 commercial grade O-rings to be dedicated in accordance with station procedures and the associated industry guidance. This evaluation resulted in the identification of critical characteristics which were used in lieu of part numbers for the purchase order and identification of the O-rings to be commercially dedicated. These critical characteristics identified were configuration, dimensions, material, and hardness.

The licensee developed a sampling plan in accordance with their procedures for each of the critical characteristics and documented the plan and reasoning in the associated justification document. The inspectors found the identification of critical characteristics, and their sampling plan to ensure conformance with the requirements of those characteristics, to be in accordance with industry guidance and licensee procedures.

9. Evaluate completed and proposed corrective actions from the root cause analysis report

documented in condition report CR-GGN-2007-00378, for the 2007 failure at Grand Gulf, and determine whether the licensee implemented appropriate corrective actions and effectiveness reviews The inspection team reviewed the root cause evaluation documented in condition report CR-GGN-2007-00378 and identified shortcomings in the licensees corrective actions and effectiveness reviews. In 2007 the licensee identified two root causes that led to the January 30, 2007, failure of the division 1 SDG TCV to control jacket water temperatures.

The first root cause (RC1) was inadequate manufacturing of the valve. Two conditions, excessive and non-uniform offset of both the valve cover and seat as well as seat height below specification, could cause degradation of the valves internal moving parts. The second identified root cause (RC2) was degraded valve internal components. The O-ring sleeve and sliding valve internal to the TCV were found degraded and that degradation caused the resulting valve failure. The licensee identified both conditions as root causes because it could not conclusively determine if the degraded condition of the internals was caused directly and solely by the manufacturing defects identified in RC1 or if it occurred in combination with other potential sources of damage such as inadequate maintenance practices.

The licensees corrective action to prevent recurrence of the condition was to replace the division 1 SDG TCV with one verified to meet the manufacturers specifications. The inspectors noted that the defects identified in RC1 were not included in the procurement specifications used by the licensee, and the licensee did not measure the cover offset, seat offset, or seat height prior to installing the replacement valve to ensure that it was not subject to the same manufacturing defect as the failed valve it was replacing. In fact, the licensee did not measure the division 1 SDG TCV installed in 2007 to ensure it did not contain the same manufacturing defect until June 3, 2022, as part of their follow-up to the May 16, 2022, failure. Similarly, the licensee did not take measurements to detect the defect on the division 2 SDG TCV until June 1, 2022, as part of their extent of condition review for the May 16, 2022, failure. However, when measured, both the installed division 1 and division 2 SDG TCVs were found to be free of the defect identified in 2007.

To assure the adequacy of their corrective action of replacing the division 1 SDG TCV and to prevent recurrence of the issue, the licensee reviewed two subsequent preventive maintenance inspection results to observe and measure the O-ring and sliding valve surface to assure they were not degraded. The inspectors concluded that this effectiveness review for the root causes should not have been relied on to show the same defects did not exist in the replacement valve. In the documentation for the two preventive maintenance tasks prior to the 2007 failure, the licensee did not note any similar degradation to the O-ring sleeve or sliding valve surfaces that were observed following the failure. Therefore, there was no basis to assume that the condition would be notable in the two subsequent preventive maintenance tasks after replacing the valve.

The inspectors documented a finding in the Inspection Results report section below related to the licensee's inadequate corrective actions related to the 2007 valve failure.

10. Evaluate the potential for generic implications associated with issues identified during the inspection, and work with the Office of Nuclear Reactor Regulation to gather information to inform any potential generic issues The inspectors determined that there were no potential generic implications associated with the 2007 division 1 SDG TVC failure.

INSPECTION RESULTS

Failure to Establish Measures to Ensure the Division 1 Replacement Temperature Control Valve was Suitable for the Application Cornerstone Significance Cross-Cutting Report Aspect Section Mitigating Green None (NPP) 93812 Systems NCV 05000416/2022050-01 Open/Closed The inspectors identified a Green, non-cited violation of 10 CFR Part 50, appendix B, criterion III because the licensee did not establish for the selection and review for suitability of application of materials, parts, equipment, and processes that are essential to the safety-related functions of the structures, systems, and components. Specifically, the licensee did not establish the suitability of the division 1 standby diesel generator jacket water cooling temperature control valve installed in 2007 to replace one that failed in service. The licensee did not review the suitability of the replacement valve in that the licensee failed to ensure it did not contain the same manufacturing defect that caused the original valves failure.

Description:

On January 30, 2007, the licensee discovered elevated temperatures in the jacket water system of the division 1 standby diesel generator (SDG) during a monthly test run.

Approximately 5 minutes after increasing the diesel power load to 4400 kW, the jacket water heat exchanger outlet high temperature annunciator alarmed at 175 degrees F. In response, operators reduced load and shut down the SDG.

The licensee performed a root cause evaluation for the temperature control valve's (TCV's)failure to control jacket water temperature and documented their findings in condition report CR-GGN-2007-00378. In that evaluation, the licensee identified inadequate manufacturing of the valve as a root cause. Two conditions, excessive and non-uniform offset of the valve cover and seat as well as the seat height below specification, could cause degradation of the valves internal moving parts.

The licensee replaced the division 1 SDG TCV to correct the issue. The replacement valve was declared operable and placed into service on February 1, 2007. However, the inspectors noted that 1) the manufacturing defect identified in their root cause was not explicitly covered by the procurement specifications used by the licensee, 2) the licensee did not measure the cover offset, seat offset, or seat height of the replacement valve prior to installation to ensure that it was not subject to the same manufacturing defect as the failed valve, and 3) the preventative maintenance actions the licensee implemented to detect the manufacturing defect prior to the valves failure could not be relied upon because significant degradation was not seen in the preventive maintenance tasks immediately prior to the original valves failure.

The licensee did not ensure that the replacement division 1 SDG TCV did not contain the same manufacturing defect as the failed valve it replaced until June 3, 2022, following inspector questioning as part of their follow-up to the valves failure on May 16, 2022.

Similarly, the licensee did not take measurements to detect the same potential defect on the division 2 SDG TCV until June 1, 2022. However, when the licensee took direct measurements, both the installed division 1 and division 2 SDG TCVs were shown to not contain the same manufacturing defect identified in 2007.

Corrective Actions: Following the failure of the division 1 SDG TCV on May 16, 2022, the licensee took measurements of both the installed division 1 and division 2 SDG TCVs and, in consultation with the valve vendor, verified that they were not subject to the same manufacturing defect identified in 2007.

Corrective Action References: condition report CR-GGN-2022-06316

Performance Assessment:

Performance Deficiency: The failure to establish measures for the selection and review for suitability of application of materials, parts, equipment, and processes that are essential to the safety-related functions of the structures, systems, and components as required by 10 CFR Part 50, appendix B, criterion III was a performance deficiency. Specifically, the licensee did not review the replacement TCV for suitability of application following the failure of the original TCV and subsequent identification of a manufacturing defect.

Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to establish measures to ensure that the division 1 SDG TCV installed to replace the one that failed in service did not contain the same manufacturing defect as the failed valve impacted its capability, availability, and reliability.

Significance: The inspectors assessed the significance of the finding using IMC 0609 appendix A, The Significance Determination Process (SDP) for Findings At-Power. The inspectors determined that the finding was of very low significance (Green) because the finding did not cause the valve to lose operability. Specifically, although not confirmed prior to installing the replacement valve, when the licensee took applicable measurements as part of troubleshooting the valves May 16, 2022, failure, the valve was shown to not contain the same manufacturing defect as the original valve.

Cross-Cutting Aspect: Not Present Performance. No cross-cutting aspect was assigned to this finding because the inspectors determined the finding did not reflect present licensee performance. Specifically, the performance deficiency occurred in 2007 when the licensee replaced the original, failed division 1 SDG TCV.

Enforcement:

Violation: Title 10 CFR Part 50, appendix B, criterion III, requires, in part, that measures be established for the selection and review for suitability of application of materials, parts, equipment, and processes that are essential to the safety-related functions of the structures, systems and components.

Contrary to the above, from February 1, 2007, until June 3, 2022, the licensee failed to review for suitability of application of a part essential to the safety-related functions of the division 1 SDG. Specifically, the licensee failed to determine if the same manufacturing defects that caused the TCV to fail in 2007 did not exist in the valve that was used to replace it.

Enforcement Action: This violation is being treated as a non-cited violation, consistent with section 2.3.2 of the Enforcement Policy.

Failure to Consider Out of Tolerance Measurements in Operability Determinations Cornerstone Significance Cross-Cutting Report Aspect Section Mitigating Green [H.7] - 93812 Systems NCV 05000416/2022050-02 Documentation Open/Closed The inspectors identified three examples of a Green non-cited violation of 10 CFR Part 50, appendix B, criterion V, Instructions, Procedures, and Drawings, which occurred because the licensee did not perform an operability determination in accordance with procedures.

Specifically, the licensees operability determination, when applied to the division 1 and division 2 standby diesel generator jacket water cooling temperature control valves, did not address the potential impact of out-of-tolerance measurements as required by Procedure EN-OP-104, Operability Determination Process, revision 17.

Description:

Following the failure of the division 1 SDG TCV on May 16, 2022, the licensee took action to measure various aspects of the TCVs as part of their troubleshooting. On both the division 1 and division 2 SDG TCVs, the licensee found several dimensions, including the cover flange face thickness, outside of the expected values.

The licensee performed operability determinations to address the operability of the SDGs given the out-of-tolerance measurements. The inspectors found that while many of the other out-of-tolerance measurements were addressed by the licensee, the operability determinations did not address the out-of-tolerance cover flange face thickness measurements. This failure occurred on three occasions:

On June 2, 2022, in condition report CR-GGN-2022-06042, for the division 2 SDG TCV On June 3, 2022, in condition report CR-GGN-2022-06091, for the division 1 SDG TCV On June 7, 2022, in condition report CR-GGN-2022-06042, for the division 2 SDG TCV when the original operability determination was revised In step 8.2 of Procedure EN-OP-104, Operability Determination Process, revision 17, the licensee is required to determine the impact that such degraded or non-conforming conditions may have.

The inspectors noted that on the data sheet included with the condition reports, the cover flange face thickness measurements were marked SAT despite being out of the expected range and with no accompanying explanation for why the out-of-tolerance measurements would be acceptable.

Given the potential for the out-of-tolerance flange face thicknesses to cause the valve cover to seat incorrectly and cause binding of the valves internal components, the inspectors determined the failure to address these measurements called into question the validity of the licensees operability determinations. Further, on June 3, 2022, the licensee received input from the valve vendor that these out-of-tolerance measurements could impact that alignment of the cover and impact the function of the valve.

Corrective Actions: Due to the inspectors questioning, the licensee performed new operability determinations to assess the impact of the cover flange face thickness measurements for both the division 1 and division 2. Given additional engineering input, the licensee ultimately concluded the diesel generators remained operable.

Corrective Action References: condition reports CR-GGN-2022-06316 and CR-GGN-2022-06424

Performance Assessment:

Performance Deficiency: The failure to consider the impact of the out-of-tolerance cover flange face thickness measurements when assessing the operability of the division 1 and division 2 SDG TCVs in accordance with Procedure EN-OP-104, Operability Determination Process, was a performance deficiency. Specifically, the licensee did not determine the impact of out-of-tolerance measurements of the division 1 or division 2 SDG TCVs as required by step 8.2 of the procedure.

Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the licensee did not consider the impact the out-of-tolerance measurements could have on the ability of the cover to seat properly and the potential resulting impacts on the division 1 or division 2 SDG TCV internals.

Significance: The inspectors assessed the significance of the finding using IMC 0609, appendix A, The Significance Determination Process (SDP) for Findings At-Power. The inspectors determined the finding to be of very low risk significance (Green) because the structures, systems, and components ultimately maintained operability once the impact of the out-of-tolerance measurements was assessed.

Cross-Cutting Aspect: H.7 - Documentation: The organization creates and maintains complete, accurate and up-to-date documentation. Specifically, maintenance personnel provided operations staff with misleading documentation for performing their operability determination. The measurements discovered out of their expected range were marked SAT on the maintenance forms with no accompanying explanation of why the measurements were acceptable despite being outside the expected range.

Enforcement:

Violation: Title 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, requires, in part, that activities affecting quality be prescribed by documented procedures of a type appropriate to the circumstances and be accomplished in accordance with these procedures. The licensee established EN-OP-104, Operability Determination Process, revision 17 as the implementing procedure for performing operability determinations, an activity affecting quality.

Procedure EN-OP-104, step 8.2.1.d states, determine the impact of the degraded or non-conforming condition on the technical specification structure, system, or component or the specified safety function.

Contrary to the above, on June 2, June 3, and June 7, 2022, the licensee failed to accomplish step 8.2.1.d of procedure EN-OP-104. Specifically, the licensee did not determine the impact that the identified out-of-tolerance measurements could have on the SDG TCVs.

Enforcement Action: This violation is being treated as a non-cited violation, consistent with section 2.3.2 of the Enforcement Policy.

EXIT MEETINGS AND DEBRIEFS

The inspectors verified no proprietary information was retained or documented in this report.

On September 8, 2022, the inspectors presented the special inspection results to Mr. B. Kapellas, Site Vice President, and other members of the licensee staff.

DOCUMENTS REVIEWED

Inspection Type Designation Description or Title Revision or

Procedure Date

93812 Corrective Action CR-GGN- 1999-00768, 2004-00868, 2004-02575, 2004-02620, 2004-

Documents 03275, 2004-03353, 2006-01022, 2006-01461, 2007-00378,

20-01738, 2022-05559, 2022-05706, 2022-06042, 2022-

06091

93812 Corrective Action CR-GGN- 2022-06304, 2022-06316, 2022-06424, 2022-06426, 2022-

Documents 06474, 2022-08690

Resulting from

Inspection

93812 Miscellaneous Quality Assurance Program Manual 42

93812 Miscellaneous Amot Model 8D Thermostatic Valve

93812 Miscellaneous SEP-GGNS-IST-1 GGNS Inservice Testing Basis Document 8

93812 Procedures 04-1-01-P75-1 Standby Diesel Generator System 117

93812 Procedures 04-1-01-P75-1 Standby Diesel Generator System 118

93812 Procedures 04-1-02-1H22- Panel No. 1H22-P400 119

P400

93812 Procedures 04-1-02-1H22- Panel No. 1H22-P400 120

P400

93812 Procedures 04-1-02-1H22- Panel No. 1H22-P400 121

P400

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 0

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 1

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 2

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 3

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 4

Inspection Type Designation Description or Title Revision or

Procedure Date

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 5

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 6

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures 04-S-24-P75- Preventive Maintenance Instruction 7

F501-1 Jacket Water Thermostatic Valve

Thermal Element Replacement

93812 Procedures EN-DC-115 Engineering Change Process 32

93812 Procedures EN-DC-115-01 Industry Standard Design Process (IP-ENG-001) 1

93812 Procedures EN-DC-115-02 Industry Standard Digital Engineering Process (NISP-EN-04) 0

93812 Procedures EN-DC-306 Acceptance of Commercial Grade Items/Services in Safety- 9

Related Applications

93812 Procedures EN-DC-313 Procurement Engineering Process 19

93812 Procedures EN-DC-313-02 Procurement Engineering - Technical Evaluation of 6

Replacement Parts

93812 Procedures EN-DC-324 Preventive Maintenance Program 7

93812 Procedures EN-DC-349 Critical Component Failure Determination 3

93812 Procedures EN-LI-102 Corrective Action Program 47

93812 Procedures EN-LI-108 Event Notification and Reporting 21

93812 Procedures EN-LI-108-01 10 CFR 21 Evaluations and Reporting 13

93812 Procedures EN-LI-118 Casual Analysis Process 35

93812 Procedures EN-MP-111 Inventory Control 17

93812 Procedures EN-MP-117 Standardized Purchasing Process 13

93812 Procedures EN-OE-100 Operating Experience Program 35

93812 Procedures EN-OE-100-01 IER Level 1 and Level 2 Evaluation and Effectiveness 6

Reviews

93812 Procedures EN-OP-104 Operability Determination Process 17

93812 Procedures EN-QV-102 Quality Control Inspection Program 9

93812 Procedures EN-QV-102 Quality Control Inspection Program 9

Inspection Type Designation Description or Title Revision or

Procedure Date

93812 Procedures EN-QV-109 Audit Process 40

93812 Procedures EN-WM-100 Work Request Generation, Screening and Classification 17

93812 Work Orders WO-GGN- 81761, 125214, 181070, 273001, 390293, 525343,

2756396, 52925293, 52970468,

June 14, 2022

MEMORANDUM TO: Chris

A. Speer, Reactor Systems Engineer

Generic Communications and Operating Experience

Office of Nuclear Reactor Regulation

FROM: Ryan

E. Lantz, Director

Division of Operating Reactor Safety Ryan E. Lantz Digitally signed by Ryan E. Lantz

Date: 2022.06.14 14:14:07 -05'00'

SUBJECT: SPECIAL INSPECTION CHARTER TO EVALUATE DIESEL

GENERATOR TEMPERATURE CONTROL VALVE FAILURE AT

GRAND GULF NUCLEAR STATION - REACTIVE INSPECTION

REPORT 05000416/2022050

In response to a failure of the Division I emergency diesel generator temperature control valve

on May 16, 2022, at Grand Gulf, a Special Inspection will be performed. While performing a

surveillance test, operators manually tripped the emergency diesel generator when the jacket

water and lube oil temperatures began rising at about degree 1 F/min and approached the

alarm response setpoints prior to being secured by operators. You are hereby designated as

the Special Inspection Team leader. The following members are assigned to your team:

Eric Lantz, Resident Inspector, Palo Verde, Project Branch D

Henry Strittmatter, Emergency Preparedness Inspector, Response Coordination Branch

A. Basis

Grand Gulf Nuclear Station Division I and II diesel generators are Transamerica Delaval,

Incorporated engines rated at 5740 kw. The engines are DSRV-4 series (16-cylinder, 4-stroke,

turbocharged, 45-degree V-type) and are designed to operate at 450 revolutions per minute.

The Grand Gulf Transamerica Delaval, Incorporated engines use an independent cooling water

system called the jacket water system to provide cooling water to the diesel engine, the

governor oil cooler, the lube oil cooler, and the turbocharger aftercoolers. The jacket water

system is a closed loop system that rejects heat to the standby service water system through

the jacket water heat exchanger.

CONTACT: Jeffrey Josey, DORS/Project Branch C

817-200-1148

A three-way thermostatic control valve, manufactured by Amot Controls, directs cooling water

to the jacket water heat exchanger to maintain temperature between the operating range of 160

C. Speer 2

F to 175

F. During operation, the temperature control valve directs approximately 200-300 gpm

to the heat exchanger using thermal elements that modulate the valve to maintain cooling water

within design temperature limits.

On May 15, 2022, the Division I diesel was shut down during its surveillance test when the

temperature control valve failed to control temperature. This failure caused jacket water and

lube oil temperatures to rise at about 1 degree F/min. Based on this, operations secured the

EDG before any temperature limits were reached.

Licensee troubleshooting of the thermostatic control valve identified a swollen and extruded

O-ring along with a cracked internal component (element tube). The licensee determined that

the O-ring was not an original equipment manufacturer (OEM) part but had been procured

through commercial grade dedication and installed approximately three years ago (2019). The

licensee also determined that a similar O-ring, also commercial grade dedicated, had been

installed in the Division II EDG approximately one year ago.

The vendor specifies a safety-related O-ring that is a proprietary blend of BUNA-N. Through a

review of the licensees dedication paperwork, it was determined that the licensee did not

specify any composition requirements other than simply BUNA-N in their critical characteristics

for the commercial grade specifications.

The residents reviewed the current alarm response instructions and performed walkdowns with

station operators. Based on these walkdowns it was determined that there was insufficient

procedural guidance for manually controlling the thermostatic control valve. For example, the

licensee did not have all necessary equipment staged to allow personnel to take manual control

of the temperature control valve. This called into question the licensees ability to take manual

actions before damage occurred to the diesel.

In 2007, the NRC completed a special inspection at Grand Gulf that also examined activities

associated with the Division I emergency diesel generator high temperature event that occurred

on January 30, 2007. Inspection Report 0500416/2007006 (ML071380478) documented the

results of the inspection and established a pattern of equipment issues associated with the

EDG TCVs that extended back to 1999.

B. Scope

The inspection is expected to perform data gathering and fact-finding to address the

following:

1. Provide a recommendation to Region IV management as to whether the

inspection should be upgraded to an augmented inspection team response. This

recommendation should be made following the first day of inspection.

2. Develop a complete sequence of events and understanding related to the

emergency diesel generator temperature control valve issue(s). The chronology

should include the status of the equipment and licensee actions to address the

condition(s) up and including the failures beginning in 1999

3. Review and assess the adequacy of operator response to the emergency diesel

generator failure that occurred on May 15, 2022.

C. Speer 3

4. Review the current status of the licensees cause determination to determine

whether it is being conducted at a level of detail commensurate with the safety

significance of the event and includes reviews of relevant plant-specific and

industry (foreign and domestic) operating experience and previous diesel

generator failures.

5. Review the licensees extent of condition evaluation to determine if the licensee

has adequately considered degradation of similar equipment (i.e. other TCVs or

O-rings).

6. Review and assess the licensees prompt and long-term corrective actions to

address the root and contributing causes.

7. Review records associated with the O-ring maintenance history for the last 15

years for Division I and II emergency diesel generator temperature control valves

at Grand Gulf, including previous failures. Based on this review assess the

licensees maintenance program for periodic monitoring and maintenance of the

emergency diesel generators.

8. Review the adequacy of the commercial grade dedication process used by the

licensee associated with the O-ring that apparently caused the TCV to fail on

May 15, 2022.

9. Evaluate completed and proposed corrective actions from the root cause

analysis report documented in CR-GGN-2007-00378, for the 2007 failure at

Grand Gulf, and determine whether the licensee implemented appropriate

corrective actions and effectiveness reviews.

10. Evaluate the potential for generic implications associated with issues identified

during the inspection, and work with the Office of Nuclear Reactor Regulation to

gather information to inform any potential generic issues.

11. Collect data necessary to support completion of the significance determination

process, if applicable.

C. Guidance

Inspection Procedure 93812, Special Inspection, provides additional guidance to

be used by the Special Inspection Team. Your duties will be as described in

Inspection Procedure 93812. The inspection should emphasize fact-finding in its review

of the circumstances surrounding the event. It is not the responsibility of the team to

examine the regulatory process. Safety concerns identified that are not directly related

to the event should be reported to the Region IV office for appropriate action.

You will formally begin the Special Inspection with an entrance meeting to be conducted

no later than June 13, 2022. You should provide a daily briefing to Region IV

management during the course of your inspection and prior to your exit meeting. The

results of the inspection will be documented in reactive inspection

C. Speer 4

report 05000416/2022050 and should be issued within 45 days of the completion of the

inspection.

This Charter may be modified should you develop significant new information that

warrants review.

Docket No. 50-416

License No. NPF-29

ML22165A251

SUNSI Review ADAMS: Non-Publicly Available Non-Sensitive Keyword:

By: JEJ Yes No Publicly Available Sensitive MD 3.4/B.1

OFFICE RIV/DORS:BC RI/DORS:Dir

NAME JJosey RLantz

SIGNATURE /RA/E /RA/E

DATE 6/6/2022 6/14/2022