ML19070A308

OpESS 2019-01 Commercial Grade Dedication Procurement and Design Control

ML19070A308
Person / Time
Issue date:	07/30/2019
From:	Robles J NRC/NRR/DIRS/IOEB
To:
Robles J
Shared Package
ML19070A306:ML19211C614	List: ML19070A308
References
CN 19-025, DC 19-007
Download: ML19070A308 (8)

Text

NRC INSPECTION MANUAL IOEB OPERATING EXPERIENCE SMART SAMPLE (OpESS) 2019/01 COMMERCIAL GRADE DEDICATION, PROCUREMENT, AND DESIGN CONTROL CORNERSTONE: BARRIER INTEGRITY MITIGATING SYSTEMS INITIATING EVENTS APPLICABILITY:

• This OpESS applies to all licensed operating commercial nuclear reactors.

• Supports informing sample selection for Inspection Procedure (IP) 71111.12, 71111.18, 71111.21M, and 71111.21N.

• Performance of this OpESS is voluntary.

OpESS 2019/01-01 OBJECTIVES 01.01 Provide support to baseline inspection activities in the area of procurement and commercial grade dedication activities.

01.02 Provide examples where deficiencies were introduced during the design modification or procurement processes that resulted in equipment failures in order to inform the inspection of design, modification, and maintenance activities.

OpESS 2019/01-02 BACKGROUND 02.01 Commercial Grade Dedication Inspection Pilot In 2014, in response to several violations issued to vendors for issues involving commercial grade dedication (CGD), the Office of New Reactors (NRO) conducted pilot inspections of selected licensees in an effort to determine if they were adequately dedicating commercial grade items for safety-related use. Inspection results for the three sites involved in the pilot are available at Agency-wide Document Access and Management System (ADAMS) Accession Numbers ML15127A549, ML15037A046, and ML14132A203. The staff noted the following issues with the CGD programs of the facilities that were inspected:

• Failure to establish adequate controls for performing technical evaluations of items or services to be dedicated, including the review of materials, parts, equipment, and processes for suitability of application as established in Criterion III of Appendix B;

• Failure to verify that the dedicating entity has properly developed and implemented a plan for CGD;

• Failure to verify that there are adequate controls for the acceptance of items procured from a third party dedicating entity; and, Issue Date: 07/30/19 1 2019/01

• Failure to appropriately identify critical characteristics (e.g., seismic and environmental qualification).

02.02 Issues Introduced Through Procurement and Design Modifications Operating experience staff recently identified several examples of failures attributed to deficiencies introduced through procurement and dedication activities over the past five years.

Many of these deficiencies were introduced as part of design modification activities and resulted in violations of Appendix B to 10 CFR Part 50, Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants and Technical Specifications. These events show that many of the issues that were identified during the CGD inspection pilot described above continue to occur. Many of the events involved inadequate assumptions that introduced new failure modes. These events are described in more detail in sections 04.03, Generic Communications and 04.04 Examples of previous inspection findings below.

Some of the issues operating experience staff identified from analysis of these events include:

• Failure to verify the suitability of replacement parts o failure to perform seismic testing or provide a justification for continued qualification, such as a similarity analysis, when replacing obsolete breakers o unexpected failure or behavior of equipment as a result of unused features (e.g.,

thermal overloads, jumpers, pump ports, grounding straps)

• Failure to identify or properly document design requirements o failure to identify critical characteristics o lack of appropriate definitions or guidance for technical evaluations

• Inadequate qualification/testing o testing does not mimic actual configuration o differences between testing configuration and actual configuration are not analyzed to ensure testing configuration bounds expected conditions

• Inadequate oversight/overreliance on outside organizations contracted to perform design, manufacturing, testing, qualification, reverse engineering, etc.

o incorrect components installed o inadequate verification that services were performed as specified o equivalency evaluations with inadequate technical justifications for differences from original equipment o inadequate assumptions made to designate a component as a like-for-like replacement

• Failure to properly dedicate commercial grade items o improper documentation or control of components design and manufacturing processes

• Inadequate inspection/testing to verify items conform to specifications o Failure to inspect test or fabrication records o inadequate inspection or checks to identify design changes in procured items o incorrect parts used o failure to perform testing or verify adequacy of testing procedures o inadequate receipt inspections

• Failure to perform and document evaluation of non-conforming conditions

• Inadequate segregation of items identified as non-conforming (e.g. exceeding shelf life, failing to conform to testing requirements etc.)

• Failure to properly classify procured items commensurate with safety impact Issue Date: 07/30/19 2 2019/01

• Failure to take appropriate corrective actions for identified deficiencies

• Inadequate extent of condition evaluation when deficiencies are identified OpESS 2019/01-03 INSPECTION GUIDANCE The following inspection guidance may be applied as appropriate to support baseline inspection activities. Inspector judgment should be used when determining the extent to which this OpESS should be used to inform inspection activities under the applicable baseline IPs. Additional information can be found in the references. IP 43004, Inspection of Commercial-Grade Dedication Programs provides further detail on inspecting CGD.

03.01 Maintenance Effectiveness The recommended inspection activity described below supports IP 71111.12, Maintenance Effectiveness.

Review a maintenance activity where the licensee is replacing electrical or electronic components such as relays, breakers, or circuit cards with a like-for-like or equivalent component. [Supports IP 71111.12, Section 02.02a]

1. Verify that there are adequate measures for the acceptance of procured items, including receipt inspections.

2. Verify that there are adequate measures for ensuring that the replacement component can perform the same functions as the original component.

• Relying on the model or part number or external physical appearance may not be enough to ensure that functionality remains unchanged, or a different mode of failure was not inadvertently introduced. Like-for-like replacement is described in further detail in IP 43004, Inspection of Commercial Grade Dedication Programs, Section 03.01b. In general, like-for-like may be applicable when the replacement item was purchased from the same vendor (successor companies may be accepted), provided all design, materials, and manufacturing processes are kept the same, or the replacement item was purchased at the same time and from the same vendor as the item it is replacing.

3. For items that are not like-for-like replacements, review the licensees equivalency evaluation. This evaluation should evaluate any changes in design, material, manufacturing process, safety, form, fit, function or interchangeability that could impact the alternate replacement items ability to function under all design conditions.

• Note that equivalency evaluations should not be used as the sole basis to accept a commercial grade item for safety-related use. The item must still undergo dedication.

4. Review maintenance documentation accounts for configuration control and wiring, as well as unused features such as jumpers, thermal overloads, and grounding straps.

Issue Date: 07/30/19 3 2019/01

03.02 Plant Modifications The recommended inspection activity described below supports IPs 71111.18, Plant Modifications, 71111.21M, Design Bases Assurance Inspection (Teams), and 71111.21N, Design Bases Assurance Inspection (Programs).

Review a plant modification involving electrical or electronic components such as relays, breakers, or circuit cards that are to be dedicated or reverse engineered. Consider reviewing plant modifications involving replacement of obsolete components with dedicated or reverse engineered items. Note that the dedication may be done by a licensee, a third party, or a combination of the two. [Supports IP 71111.18, Section 03.02a, IP 71111.21M Section 02.03, and IP 71111.21N, Section 03.02]

1. Verify that the dedicating entity has properly developed and implemented a plan for the CGD.

2. Verify that there are adequate controls for the acceptance of items procured from a commercial supplier.

3. For items procured from third party dedicating entities, review the related procurement documents to:

• Verify that the licensee invoked 10 CFR Part 50 Appendix B and 10 CFR Part 21 requirements on the supplier.

• Verify that the supplier is on the Approved Suppliers list with no limitations impacting the required work to dedicate the item in question.

• Verify that the justification for adding the supplier to the Approved Suppliers List includes a valid audit that covers the scope of the dedication activities.

4. Verify that there are adequate controls for the acceptance of procured items that were dedicated by a third party:

• Verify that the procurement documents specify adequate controls and acceptance methods for the dedicating entity.

• Verify that the proper critical characteristics are identified.

• Verify that receipt inspections adequately verify that the acceptance criteria are met for the dedicated item.

• Verify that any restrictions to the use of the dedicated item are clearly documented upon receipt so that the item is only used in an application that is prescribed in the procurement documents.

OpESS 2019/01-04 REFERENCES These references may include pre-decisional information contained on NRC internal websites.

Once the agency has formally evaluated an OpE issue and has determined that it meets the criteria for agency action, the NRC communicates the issue to the public and the industry through one or more appropriate methods (e.g., generic communication, rulemaking public comment periods, etc.).

Issue Date: 07/30/19 4 2019/01

04.01 Applicable Regulations 10 CFR Part 21, Reporting of Defects and Noncompliance 10 CFR Part 50, Appendix B, Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants 04.02 Inspection Manual Chapters and Procedures IP 43004, Inspection of Commercial-Grade Dedication Programs IP 71111.12, Maintenance Effectiveness IP 71111.18, Plant Modifications IP 71111.21M, Design Basis Assurance Inspection (Teams)

IP 71111.21N, Design Basis Assurance Inspection (Program) 04.03 NRC Generic Communications Generic Letter 1989-02, Actions to Improve the Detection of Counterfeit and Fraudulently Marketed Products Generic Letter 1991-05, Licensee Commercial-Grade Procurement and Dedication Programs Information Notice 2016-01, Recent Issues Related to the Commercial Grade Dedication of Allen Bradley 700-RTC Relays Information Notice 2016-09, Recent Issues Identified When Using Reverse Engineering Techniques in the Procurement of Safety-Related Components Information Notice 2017-05, Revision 1, Potential Binding of Schneider Electric/Square-D Masterpact NT and NW 480-VAC Circuit Breaker Anti-Pump Feature Information Notice 2011-01, Commercial-Grade Dedication Issues Identified During NRC Inspections Information Notice 1996-40, Deficiencies in Material Dedication and Procurement Practices and in Audits of Vendors, and Supplement 1 Information Notice 2014-11, Recent Issues Related to the Qualification and Commercial Grade Dedication of Safety-Related Components Information Notice 1991-25, Commercial-Grade Structural Framing Components Supplied as Nuclear Safety-Related Equipment Information Notice 1987-66, Inappropriate Application of Commercial-Grade Components Information Notice 1992-68, Potentially Substandard Slip-on, Welding Neck, and Blind Flanges, and Supplement 1 Issue Date: 07/30/19 5 2019/01

04.04 Examples of previous inspection findings Clinton - In March 2017, the licensee discovered that a time delay relay in the Division 1 emergency diesel generator (EDG) room vent fan was cycling, which would have prevented the fan from starting if required. The licensee had implemented a design change in 2008 to replace obsolete relays but did not consider differences in the drop out voltages of the replacement relays. The design change that replaced the relays used an equivalency evaluation to determine the acceptability of the new relays but failed to consider the suitability of the relays for load shed and re-sequencing of the EDG Room vent fan. In the equivalency evaluation, the licensee justified using the new relay simply because it had been used to replace the obsolete relay in another application. This resulted in a White finding and associated violation of Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Appendix B, Criterion III, Design Control.

Additional information is available in NRC Integrated Inspection Report 05000461/2017009, available at ADAMS Accession Number ML17226A321.

Waterford - In July 2017 during a rain and lightning event, arcing was observed on a main transformer isophase bus duct, prompting operators to manually trip the reactor. The generator trip breakers automatically opened, but the breakers for the startup transformer failed to close, resulting in the de-energization of all safety and non-safety AC buses. Both EDGs started as designed and reenergized the safety buses. Troubleshooting revealed that the failure of the automatic fast bus transfer was caused by digital relays timing out in approximately one-tenth of the expected time. These relays had been replaced as part of a design modification, but the design change procedure did not account for the relays susceptibility to inductive kick. This resulted in a White finding and associated violation of Technical Specification 3.8.1.1, A.C.

Sources. Additional information is available in NRC Integrated Inspection Report 05000382/2017011, available at ADAMS Accession Number ML17354A690.

Brunswick - In the Spring 2015 refueling outage during post-maintenance testing at Brunswick, the EDG output breaker failed to remain closed to power its emergency bus. Subsequent troubleshooting revealed that the relays installed in the circuit contained an internal digital component that was susceptible to inductive kick. The relays were the same model and same manufacturer as the relays that had previously been installed in the circuit, but the original equipment manufacturer had changed the internal relay design. Multiple dedicating entities failed to identify the design change. The issue had been identified during a Component Design Basis Inspection by NRC inspectors at Vogtle, but the relays were never installed there and the licensee at Brunswick was not aware of the earlier issue. Several of these entities subsequently submitted 10 CFR Part 21 reports, and the NRC issued Information Notice 2016-01. This event resulted in a violation of Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Appendix B, Criterion III, Design Control. Additional information is available in NRC Integrated Inspection Report 05000324/2015007, available at ADAMS Accession Number ML15210A725.

Callaway - While responding to a reactor trip, operators noted that the motor-driven auxiliary feedwater flow control valve did not respond to the open demand signals from the control room. Troubleshooting identified a failed rectifier diode bridge on a Modutronics circuit card. The card had been purchased as a reverse engineered component to replace an obsolete card, but the vendor that reverse engineered the card incorporated a rectifier with half the current carrying capacity of the original circuit card. The licensee failed to provide adequate oversight of the vendor. This event was included in NRC Information Notice 2016-09 and resulted in a violation of Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Appendix B, Criterion III, Design Control. Additional information is available in NRC Integrated Inspection Report 05000483/2015009, available at ADAMS Accession Number ML16013A021.

Issue Date: 07/30/19 6 2019/01

River Bend - In March 2015, during division I emergency core cooling system/loss of coolant accident (ECCS/LOCA) testing, a control building chiller failed to restart and sequence onto the emergency diesel generator. Operators attempted to start chillers on division II, but those failed to start because the air handling units were not operating due to a breaker switch linkage rod that was out of alignment. The failure of the division I chiller to sequence on the EDG was the result of a design issue with the associated Masterpact breaker. These Masterpact breakers were part of a design change to replace obsolete GE HK breakers, but the licensee failed to identify a failure mode that exists in certain scenarios where the breaker receives simultaneous open and close signals. The breaker vendor issued guidance to modify the circuitry in the breaker to preclude this failure mode. This event is described in NRC Information Notice 2017-05 and resulted in a violation of Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Appendix B, Criterion XVI, Corrective Action. Additional information is available in NRC Integrated Inspection Report 05000458/2015010, available at ADAMS Accession Number ML16047A268.

St Lucie 1 - In August 2016 during restart following an outage, an unexpected actuation of the Main Generator Inadvertent Energization Lockout relay caused the main generator to trip and a subsequent reactor trip. The generator lockout prevented an automatic transfer to the startup transformer and de-energized the safety buses. The EDGs started and powered the safety buses. A design modification in 2013 resulted in a wire being incorrectly removed, which resulted in the relay being energized and undetected. This event resulted in a White finding for the licensees failure to maintain configuration control of the inadvertent energization lockout relay manual synchronization circuitry as required by licensee procedures. Additional information is available in NRC Integrated Inspection Report 05000389/2016012, available at ADAMS Accession Number ML17179A461.

OpESS 2019/01-05 REPORTING RESULTS/TIME CHARGES/ADDITIONAL ISSUES If information from this OpESS is used to inform a baseline inspection sample, reference the OpESS number in the scope section of the report.

In addition, if any findings or violations are identified in conjunction with this OpESS, include a statement similar to the following in the description section of the finding write-up:

This finding was identified in connection with a review of Operating Experience Smart Sample (OpESS) 2019/XX.

Inspection time for this OpESS is to be charged to the normal baseline procedure under which it is being used and the level of effort is expected to be within normal baseline inspection sample resource estimates.

OpESS 2019/01-06 CONTACTS For technical support regarding the performance of this OpESS and emergent issues, contact Aaron Armstrong (NRR/DIRS/IQVB) at 301-415-8396 or Aaron.Armstrong@nrc.gov, Alfred Issa (NRR/DIRS/IOEB) at 301-415-5342 or Alfred.Issa@nrc.gov or Rebecca Sigmon (NRR/DIRS/IOEB) at 301-415-0895 or Rebecca.Sigmon@nrc.gov.

Issue Date: 07/30/19 7 2019/01

Revision History for OpESS 2019/01 Commitment Accession Description of Change Description of Comment Resolution and Tracking Number Training Required Closed Feedback Form Number Issue Date and Completion Date Accession Number Change Notice (Pre-Decisional, Non-Public Information)

N/A ML19070A308 Initial issuance to provide support for initial N/A ML19071A004 07/30/19 baseline inspection activities in the area of CN 19-025 Maintenance Effectiveness, Plant Modifications, and Design Basis Assurance Inspections.

Issue Date: 07/30/19 Att1-1 2019/01