ML23024A117

IMC 0307 Appendix C, Reactor Oversight Process (ROP) Self-Assessment ROP Implementation Audit

ML23024A117
Person / Time
Issue date:	03/24/2023
From:	Nicole Fields NRC/NRR/DRO/IRAB
To:
References
DC 23-002, CN 23-009
Download: ML23024A117 (1)

Text

NRC INSPECTION MANUAL IRAB INSPECTION MANUAL CHAPTER 0307 APPENDIX C REACTOR OVERSIGHT PROCESS (ROP) SELF-ASSESSMENT ROP IMPLEMENTATION AUDIT Effective Date: 03/24/2023 0307C-01 PURPOSE The Reactor Oversight Process (ROP) self-assessment program evaluates the overall effectiveness of the ROP in meeting its pre-established goals and intended outcomes. The Office of Nuclear Reactor Regulation (NRR) is responsible for providing guidance and implementation direction to the regions on reactor inspection and oversight, and also appraises regional program performance in terms of effectiveness and uniformity pursuant to 10 CFR 1.43. This procedure establishes the process for an independent, NRR-led ROP implementation audit to satisfy, in part, this regulatory requirement. The ROP implementation audit is part of Element 1 of the ROP self-assessment program as described in Inspection Manual Chapter (IMC) 0307, ROP Self-Assessment Program.

0307C-02 OBJECTIVES 02.01 Appraise regional program performance in terms of effectiveness and uniformity of ROP implementation pursuant to 10 CFR 1.43(e).

02.02 Ensure transparent, objective, predictable, scrutable, reliable, and uniform ROP implementation across all NRC regions and in accordance with NRR program office policies, programs, and procedures.

02.03 Determine whether revisions to ROP policies, programs, or procedures are warranted to promote effective and uniform regional ROP implementation.

0307C-03 RESPONSIBILITIES AND AUTHORITIES 03.01 Director, Division of Reactor Oversight (DRO), Office of Nuclear Reactor Regulation (NRR)

a. Oversees implementation of the annual ROP implementation audits, such as attending the ROP implementation audit entrance and/or exit meeting, and/or observing portions of the onsite audit.

b. Reviews and concurs on the final report containing results of the annual ROP implementation audit.

c. Reviews and concurs on the charter for the annual ROP implementation audit.

Issue Date: 03/24/23 1 0307 App C

03.02 Regional Directors, Branch Chiefs, and Staff, Division of Reactor Safety, Division of Reactor Projects, Division of Operating Reactor Safety, and Division of Radiological Safety and Security

a. Ensure applicable regional data is collected and submitted to facilitate the ROP implementation audit.

b. Help coordinate the ROP implementation audit teams office visit and provide office and meeting space for the audit team.

c. Provide input for selection of the ROP implementation audit focus areas.

d. Review the ROP implementation audit results for regional applicability and document the conclusions and any resulting regional actions in a response memorandum.

e. Provide staff for the ROP implementation audit team, as requested by DRO.

f. Reviews and concurs on the final report containing results of the annual ROP implementation audit for their region.

03.03 Chief, Reactor Inspection Branch (IRIB)

Provides staff for the ROP implementation audit team.

03.04 Chief, Reactor Assessment Branch (IRAB)

a. Monitors the effectiveness of corrective actions and improvements to the ROP that are developed in response to the ROP implementation audits.

b. Provides initial coordination for ROP implementation audit team and focus area selection.

c. Develops and issues the charter for each annual ROP implementation audit, choosing audit focus areas in consultation with IRIB and the regions.

d. Provides staff for the ROP implementation audit team.

e. Reviews and concurs on the final report containing results of the annual ROP implementation audit.

03.05 Audit Team Leader

a. Coordinates with the IRAB chief to select team members.

b. Leads the development of the charter, working with the IRAB chief, region, and other team members.

c. Develops the plan to execute a successful audit prep week and onsite week including the entrance and exit meeting, other daily meetings with regional staff, and a method to capture team notes regarding the standardized audit item and focus areas.

d. Leads the disposition of comments on the draft audit report before obtaining concurrences.

Issue Date: 03/24/23 2 0307 App C

e. Communicates with the ROP self-assessment lead as needed to ensure requirements of the audit, documentation, and responses are met.

0307C-04 REQUIREMENTS AND GUIDANCE The ROP self-assessment program will include ROP implementation audits of regional offices to appraise regional program performance in terms of effectiveness and uniformity of ROP implementation. This annual audit ensures transparent, objective, predictable, scrutable, reliable, and uniform ROP implementation across all regions. This appendix provides the implementing directions for activities described in IMC 0307, Section 06.01.d.

04.01 Audit Periodicity and Schedule The NRR-led ROP implementation audits will be conducted annually at one NRC region on a rotating basis. In lieu of an ROP implementation audit in the fifth year, program office staff, with regional participation, will conduct a comprehensive review of the baseline inspection program (see IMC 0307, Appendix B). As such, over a 5-year period, each region will receive one ROP implementation audit and the overall baseline inspection program will be subjected to a comprehensive review.

The office visit for the ROP implementation audit will typically be scheduled between May (after the Agency Action Review Meeting) and early September of a given year.

Consideration should be given to schedule the office visit to minimize the impact on regional operations, maximize the effectiveness and efficiency of the audit team activities, and to accommodate NRR program office management participation in the entrance or exit briefing (at a minimum). For a complete timeline of the ROP implementation audit process, see exhibit 1.

04.02 Audit Scope The ROP implementation audit measures each regions program performance in effectively and uniformly implementing the ROP. The audit has two parts: a data-driven, standardized implementation audit covering all four ROP program areas (assessment, Significance Determination Process (SDP), inspection, and performance indicators (PIs)), and pre-selected audit focus areas where the audit team conducts a deep-dive review. The standardized implementation audit is completed using attachment 1 and uses standardized program performance ratings (meets/does not meet/not applicable).

The audit focus areas (nominally two focus areas, minimum one focus area, maximum three focus areas) will be selected based on recent areas of management interest, results of ROP metrics or data trending analysis, IMC/IP lead data-driven analysis, and regional input. The focus areas selected for the audit will be included in the audit charter and all four regions will be notified of the selected focus areas in advance of the audit.

Note: Since the focus areas are unique for each years ROP Implementation Audit, each region that is not the subject of a specific years audit will need to do their own assessment of those focus areas. The regions shall report the results of their assessments in the regional response letters that are discussed in section 04.04.b below.

Issue Date: 03/24/23 3 0307 App C

04.03 Audit Procedure

a. Assembling the Audit Team: The ROP implementation audit team will consist of three (maximum four) staff from IRIB and IRAB (one of which will be designated as the audit team lead), and one regional staff member from a region not being audited.

Consideration should be given to having the regional team member be from the region scheduled for the next ROP implementation audit. Collectively, the audit team should have staff with experience across multiple ROP program areas to be able to adequately evaluate the standardized audit items and the audit focus areas. Staff from other regions may choose to be onsite to observe the audit for benchmarking purposes but are not part of the formal audit team.

b. Developing and Issuing the Audit Charter: Once the team lead is identified and the team is formed, the team will develop and issue, with the DRO division directors approval, a charter for the ROP implementation audit that outlines the audit schedule (see exhibit 1 of this appendix) and establishes the focus areas that will be assessed. The draft charter will be shared with the region that is being audited for a 2-week feedback period before the charter is finalized. The IRAB branch chief will issue the finalized charter no less than 30 days before the scheduled audit start date.

c. Completing the Standardized Audit Items: Insofar as the necessary data is available at NRC headquarters or can be compiled with remote support from the region being audited, the audit team should complete most of the standardized implementation audit items using attachment 1 prior to being onsite in the audited region. The audit team will assess all audit items from each ROP program area in attachment 1.

d. Audit (Onsite) Week: The audit team leader will conduct an entrance meeting with applicable regional personnel upon arrival at the region to help facilitate the review. The team leader will coordinate with the region being audited for any support needed to facilitate the onsite week efficiently and effectively, such as space, access, personnel availability, and connectivity requirements. Additional visits to the regional office may be scheduled if necessary to meet the objectives of the ROP implementation audit, but efforts should be made to limit the onsite portion of the audit to one week.

e. Objective Evaluation Standards: The audit team will evaluate the standardized implementation audit items in terms of objective performance ratings (meets/does not meet current requirements in ROP governance documents), reflecting the effectiveness of delegated ROP functions within the region under audit. When practical, the regional audit team member will provide amplifying information on whether the audited regions approach to ROP implementation in each area is uniform with other regions. During the audit, should any clarification be required from the NRR program office regarding ROP implementation or ROP governance documents, it will be noted and included in the final report.

f. Exit Meeting: An exit meeting will be scheduled at the end of the ROP implementation audit onsite week to discuss the results. Throughout the onsite week, the team leader will discuss the teams findings with the appropriate regional management.

Issue Date: 03/24/23 4 0307 App C

04.04 Documentation Required

a. Audit Report A draft report will be prepared by the team leader, with inputs from all team members, within 30 days of the exit meeting. This report will summarize the results of the standardized implementation audit items and the results of the deep-dive review of the audit focus areas, as well as any areas where the NRR program office is requested to provide additional clarification or guidance. The completed standardized implementation audit worksheet (attachment 1) will be included as an enclosure to the report.

DRO will send the preliminary (draft) audit report to the audited regional office for a 2-week review and comment period (the other regions will receive the draft report at the same time for informational purposes). DRO will consider the audited regional offices comments for incorporation in the final report.

The final report should be issued within 60 days of the exit meeting and will include the results of both parts of the ROP implementation audit (the standardized implementation audit and the audit focus areas). The final report will summarize where the audited region met/did not meet the requirements put forth by ROP governance documents for implementing the ROP functions, and where the audited regions implementation of the ROP is/is not uniform with other regions. The final report will also include any program recommendations for updates or clarifications to the ROP program or ROP implementation. The report will be from the audit team leader to the director of NRR thru the DRO division director. Copies will be distributed to the regions and the deputy executive director for Reactor and Preparedness Programs (DEDR). A summary of the final report will be included in the annual ROP self-assessment SECY paper, which is publicly available.

b. Regional Response Within 45 days of issuance of the final audit report, the audited region and the other regional offices will review and evaluate the audit results, compare to their own ROP program implementation, and provide a response memo detailing any actions taken to address any identified issues. The response memo from each non-audited region will also specifically include the regions conclusions from their evaluation of the selected audit focus areas. The purpose of this review is to determine for any of the noted areas where the audited region did not meet ROP governance document requirements and for the results of the selected audit focus areas, if they are unique to the audited region or whether they are common across the regions. The response memo should be addressed to the DEDR with a copy to the director of NRR and the DRO division director.

c. Follow-up As discussed above, the results of the ROP implementation audit will be referenced in the annual ROP self-assessment SECY paper. The results will also be presented to senior NRC management at the Agency Action Review Meeting (AARM) and the associated Commission briefing on the results of the AARM. Any program recommendations that are included in the final audit report will be entered into the ROP Issue Date: 03/24/23 5 0307 App C

lessons learned tracker. Any regional actions in response to regional recommendations will also be entered into the ROP lessons learned tracker.

END List of Exhibits:

Exhibit 1: Sample Timeline for Regional ROP Implementation Audit List of Attachments:

Attachment 1: Worksheet for Standardized Implementation Audit Attachment 2: Revision History for IMC 0307, Appendix C Issue Date: 03/24/23 6 0307 App C

Exhibit 1: Sample Timeline for Regional ROP Implementation Audit Activity Month Lead Select audit team lead January Chief, IRAB Chief, IRAB and Audit Select audit team members January team lead Coordinate with region on Audit team lead February optimal audit (onsite) week Develop draft audit charter Audit month minus Audit team (including focus areas) 3 months Provide draft audit charter to Audit month minus Audit team lead audited region 2 months (allow 2-weeks for review)

Issue final audit charter Audit month minus Chief, IRAB (including focus areas) and 6 weeks (at least 30 days distribute to all regions before audit start)

Kick off meeting ahead of prep Audit team Week before prep week week Prep week Audit month (but no later Audit team than September)

Onsite week Audit month Audit team Finalize draft audit report Audit month plus one Audit team month Provide draft audit report to Audit month plus one DRO regions for review month (allow 2-weeks review)

No later than 60 days after Director, DRO the onsite Audit week (allow at least 45 days for Issue final audit report regional responses prior to the end of the calendar year)

Respond to audit 45 days after the audit Regions recommendations report is issued (section 03.02)

Provide input for annual ROP Mid-January Audit team lead self-assessment SECY Issue Date: 03/24/23 Ex1-1 0307 App C

Attachment 1: Worksheet for Standardized Implementation Audit AUDIT OF REGIONAL EFFECTIVENESS AND UNIFORMITY IN IMPLEMENTING THE ROP STANDARDIZED IMPLEMENTATION AUDIT

SUMMARY

SHEET DEFINITIONS OF AUDIT RATINGS:

Meets ROP Governance Documents (M)

• goals and requirements in ROP program governance documents are consistently met or exceeded

• schedules or timeliness goals as described in ROP program governance documents are consistently met Does Not Meet ROP Governance Documents (DNM)

• goals and requirements in ROP program governance documents are consistently not met

• schedules as described in ROP program governance documents are frequently not met

• management attention is warranted to address potential area of weakness Not Applicable (NA)

Not applicable or not evaluated. Any use of this rating must be justified.

AUDIT NOTES:

Samples: Generally, as used in this IMC, a sample is defined as about 10 - 20 instances to review; however, an audit team member has discretion regarding sample size to adequately assess the audit item.

Measuring Current Performance: Unless otherwise stated, samples for audit items should cover the current in-progress calendar year and the previous 2 calendar years to measure current performance. If the audit team needs to review samples greater than 2 years old to achieve clarity on an audit issue, it will be noted in the audit report.

1.0 Assessment Program Area M / DNM 1.1 Determine whether end-of-cycle assessment meeting agendas and plant performance summaries for all plants were entered into ADAMS. (IMC 0305, section 07.03.b(1))

(M / DNM / NA)

Comments / Regional Uniformity:

Issue Date: 03/24/23 Att1-1 0307 App C

1.2 Determine whether staff reviewed the Plant Issues Matrix (PIM) findings to determine if there are any programmatic trends for consideration during the end-of-cycle assessment meeting. (IMC 0305, section 07.03.b(2))

(M / DNM / NA)

Comments / Regional Uniformity:

1.3 Determine whether annual assessment public meetings were conducted and documented in accordance with IMC 0305, section 09.01.

(M / DNM / NA)

Comments / Regional Uniformity:

1.4 Review a sample of RPS data to determine whether the PIM date for inspection findings is correctly entered as the last day of the quarter for findings documented in quarterly integrated inspection reports, or the final exit date of the inspection in which the issue was documented as an AV, FIN, NOV, or NCV for all other inspection reports. For green findings, RPS automatically populates the PIM date, though it is user editable, so this review should be primarily focused on potentially greater-than-green (GTG) findings. (IMC 0305, section 11.01.b)

(M / DNM / NA)

Comments / Regional Uniformity:

1.5 Determine whether, for traditional enforcement violations, regional staff determined if the licensee met the criteria for a follow-up inspection under IP 92702, IP 92723, or IP 92722. If the criteria were met, review whether the decision to conduct or not to conduct a follow-up inspection, and the basis for the decision, was documented in the cover letter. (IMC 0305, section 13.02.b)

(M / DNM / NA)

Comments / Regional Uniformity:

2.0 Significance Determination Process Program Area M / DNM 2.1 Evaluate whether the region is implementing the Inspection Finding Resolution Management (IFRM) process in accordance with the guidance in IMC 0609, Attachment 5. This could be accomplished by reviewing Inspection Finding Review Board (IFRB) forms and considering the following:

• Is the IFRB form in ADAMS and profiled correctly?

• Was the most up-to-date version of the form used?

• Does the form indicate timeliness dates and NRC management POC?

• For those findings that did not hold an IFRB, was that appropriate?

(M / DNM / NA)

Comments / Regional Uniformity:

Issue Date: 03/24/23 Att1-2 0307 App C

2.2 Evaluate whether the region is consistent in planning and conducting Significance and Enforcement Review Panels (SERPs). This could be accomplished by reviewing SERP forms and considering the following:

• Is the SERP form in ADAMS and profiled correctly?

• Was the most up-to-date version of the form used?

• Did the finding meet the threshold for a planning SERP and if so, was it conducted?

• Does the form indicate timeliness dates and a discussion of any timeliness challenges?

• Does the form include adequate detail regarding uncertainty and qualitative risk considerations? Supplemental material, such as a PowerPoint can fulfil this requirement if referenced in the SERP form and available in ADAMS.

(M / DNM / NA)

Comments / Regional Uniformity:

2.3 Evaluate whether publicly available reports are consistent and follow the guidance in IMC 0609 and/or other documents. Sample inspection reports, preliminary determination letters, and final determination letters and consider the following:

• Are timelines provided for submitting additional information and supporting a Regulatory Conference?

• If requested by the licensee, are Regulatory Conferences completed within 40 days of issuance of the preliminary determination letter?

• For those preliminary GTG, is there a clear statement identifying the type of information that is needed to improve the fidelity of the significance characterization?

• For security-related issues, is there a publicly available cover letter and does it characterize the finding as GTG if the official use only enclosure is white, yellow, or red?

• Is the EA number listed and is the docket correct?

• Verify no SDP worksheets or portions of the SERP package are included.

(M / DNM / NA)

Comments / Regional Uniformity:

3.0 Inspection Program Area M / DNM 3.1 Evaluate a sample of inspection reports from ADAMS across different sites for conformance with IMC 0611. This evaluation should not focus on the capabilities and limitations of the auto report generator in RPS-Inspections but should focus on the conformance with IMC 0611 of the report details and results as input into RPS.

(M / DNM / NA)

Comments / Regional Uniformity:

Issue Date: 03/24/23 Att1-3 0307 App C

3.2 Evaluate a sample of RPS data entered for findings, violations, observations, unresolved items, and very low safety significance issue resolutions, as compared to the issued inspection reports, covering both resident and regional generated inspection findings, for conformance with IMC 0306, with an emphasis on:

• double-counted or missing items

• item details in RPS are consistent with the issued inspection report

• items have the correct PIM date (as applicable), closure date, and issue date

• items in RPS have the appropriate status (Open/Closed) and are approved (M / DNM / NA)

Comments / Regional Uniformity (Are there regional inconsistencies in RPS data entry methods/procedures/accepted practices?):

3.3 Are Licensee Event Report (LER) reviews being conducted in accordance with IP 71153 Follow-up of Events and Notices of Enforcement Discretion? Are LER reviews being documented in accordance with IMC 0611? Evaluate a sample of LER reviews and consider the following:

• whether findings and violations identified during the LER review, including violations which are minor, are dispositioned in accordance with IMC 0611

• when no performance deficiency nor violation was identified during the LER review, whether the report clearly indicates why no violation or performance deficiency exists to allow an informed, independent reader to understand the NRCs conclusion

• whether references are provided to any previously documented LER inspection reviews related to the updated submittals or LER inspection reviews documented outside of report section IP 71153

• whether LER submittals, including revised submittals, are either inspected and documented or are being tracked for future inspection in RPS (M / DNM / NA)

Comments / Regional Uniformity:

Issue Date: 03/24/23 Att1-4 0307 App C

3.4 Evaluate a sample of inspection findings and violations for adherence to IMC 0612, Issue Screening and IMC 0611, Power Reactor Inspection Reports and consider the following:

• whether multiple examples of the same performance deficiency that share the same cause and require the same corrective actions are documented as a single finding

• whether the proper IMC 0612, Appendix B path(s) are used, and whether findings and violations are appropriately characterized

• whether or not multiple minor performance deficiencies are being aggregated to a more-than-minor finding

• whether present performance is appropriately justified for any assigned cross-cutting aspects (M / DNM / NA)

Comments / Regional Uniformity:

3.5 Verify that inspection sampling is conducted and documented following the guidance in IMC 2515, section 08.04 and section 09, and IMC 0306, section 06.08. If fewer than the minimum number of samples are performed because the inspection samples are not available or more than the maximum number of samples are performed because of licensee performance, inspectors shall obtain management approval. The basis for the approval shall be documented in RPS for the applicable procedure in the Note on the site All Procedures page.

(M / DNM / NA)

Comments / Regional Uniformity:

3.6 Review all GTG findings from the region. Verify that GTG finding planned corrective actions to prevent recurrence (CAPRs) by the licensee at the conclusion of the supplemental inspection are being tracked by the region.

Review any closed CAPRs that have been inspected by the region. Refer to OIG-19-A-19 Audit of NRCs Oversight of Supplemental Inspection Corrective Actions (ML19256A776), and note that since this audit, there have been changes that allow CAPRs to be entered into RPS, as well as related changes to IMC 0611, Appendix C and IMC 2515, Appendix B.

(M / DNM / NA)

Comments / Regional Uniformity:

3.7 Evaluate items documented in accordance with IMC 0611, section 0611-12 or section 0611-06, and all items closed per the very low safety significance issue resolution (VLSSIR) process and documented in accordance with IMC 0611, section 0611-12

• Verify that any items documented are in accordance with IMC 0611-12 or 0611-06, as applicable, such as minor deficiencies, minor violations, observations, assessments, VLSSIR, and unresolved items (URIs).

Issue Date: 03/24/23 Att1-5 0307 App C

• Verify that any URIs closed to the VLSSIR process are documented in accordance with IMC 0611 section 12.03.

• Review all open URIs to determine where the URI is in the inspection process and if the URI should be closed to the VLSSIR process.

(M / DNM / NA)

Comments / Regional Uniformity:

3.8 Evaluate the regions decision-making process in inspection items that require regional divisional management time to resolve. Since some of these issues may not be documented, the auditor should also interview a sampling of ROP regional branch chiefs to discuss these inspection issues. The purpose of this audit area is to contribute to meeting objective 02.03 of this appendix. Some areas to focus on are:

• IMC 0612 more-than-minor criteria

• issues that could have been closed to the VLSSIR process

• Green findings that had a significant amount of input from and discussion with the licensee.

(M / DNM / NA)

Comments / Regional Uniformity:

4.0 Performance Indicator Program Area M / DNM 4.1 Verify that IP 71151 was accomplished annually for all sites within the region as part of the baseline inspection program. This can be easily accomplished if the region being audited reported 100% baseline inspection program accomplishment. If there were any missed samples, the missed samples should be evaluated to see if an IP 71151 sample was missed.

Data sources: RPS-Inspections and ADAMS (regional baseline inspection completion memos, referenced in the annual ROP self-assessment SECY)

(M / DNM / NA)

Comments / Regional Uniformity:

Issue Date: 03/24/23 Att1-6 0307 App C

Attachment 2: Revision History for IMC 0307 Appendix C Commitment Accession Description of Change Description of Training Comment Resolution and Tracking Number Required and Closed Feedback Form Number Issue Date Completion Date Accession Number (Pre-Change Notice Decisional, Non-Public Information)

ML16147A455 Initial issuance. Created to address self- None ML16148A045 07/15/16 assessment process changes. Researched CN 16-016 commitments for the last four years and found none.

ML19274C225 Complete reissuance (major rewrite, satisfies None ML19274C541 05/29/20 periodic/review update requirement) to reflect CN 20-025 change from regional peer review to ROP implementation audit as a result of 2019 holistic review of ROP self-assessment program.

ML23024A117 Routine revision to clarify and improve the None ML23024A272 03/24/23 regional implementation audit process based CN 23-009 on program experience performing these audits. These revisions address Program Recommendation 1 from the calendar year 2022 ROP implementation audit of Region I (ML22285A231). Minor revisions to update organizational titles and document formatting.

Changed number of focus areas from a range of one to two to a nominal two, with a minimum of one, and a maximum of three.

Adjusted standardized audit items and required that all of these items be completed.

Added sample timeline for regional audit process.

Issue Date: 03/24/23 Att2-1 0307 App C