ML21341B399

IMC 0307 Reactor Oversight Process Self-Assessment Program

ML21341B399
Person / Time
Issue date:	05/03/2022
From:	Nicole Fields NRC/NRR/DRO/IRAB
To:
Nicole Fields, NRR/DRO/IRAB 630-829-9570
Shared Package
ML22117A125, ML21355A431, ML21334B356	List:
References
CN 22-009, DC 21-024
Download: ML21341B399 (21)

Text

Issue Date: 05/03/22 1

0307 NRC INSPECTION MANUAL IRAB INSPECTION MANUAL CHAPTER 0307 REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT PROGRAM Effective Date: 05/03/2022 0307-01 PURPOSE The Reactor Oversight Process (ROP) self-assessment program evaluates whether the ROP meets its pre-established goals and intended outcomes by appraising the uniformity and effectiveness of regional and program office ROP implementation, evaluating effectiveness of significant ROP changes, and performing comprehensive reviews of selected ROP program areas to verify adherence to ROP program governance documents. The program uses a mix of data-driven monitoring efforts and traditional team-based retrospective reviews to accomplish the above purpose. This self-assessment approach ensures that the staff maintains multiple avenues for internal and external stakeholder feedback for continuous ROP improvement.

0307-02 OBJECTIVES 02.01 To establish the processes for collecting information and data to support the ROP self-assessment program.

02.02 To establish a process for objectively evaluating the effectiveness of the ROP in adhering to the Principles of Good Regulation and achieving the ROP program goals and intended outcomes.

02.03 To provide timely, objective information to inform program planning and to develop recommended improvements to the ROP.

02.04 To leverage ROP program execution data in all aspects of ROP self-assessment to support data-driven decision-making.

02.05 To appraise NRC regional program performance and program office performance in terms of effectiveness and uniformity in implementing the ROP.

02.06 To assess the effectiveness of significant changes to the ROP.

02.07 To ensure that complex ROP feedback from supplemental and reactive inspections is considered as appropriate for improvements to the ROP.

02.08 To perform focused in-depth assessments of specific program areas of interest.

02.09 To monitor trends in the implementation of the baseline inspection program and to conduct periodic baseline inspection program comprehensive reviews.

02.10 To provide mechanisms to solicit and assess feedback from both internal and external stakeholders.

Issue Date: 05/03/22 2

0307 02.11 To support the agencys strategic goals as described in the current NRC Strategic Plan, NUREG-1614.

02.12 To inform the Commission, NRC senior management, and the public of the results of the ROP self-assessment program, including any conclusions and resulting ROP enhancements.

0307-03 APPLICABILITY The self-assessment process described in this Inspection Manual Chapter (IMC) is designed to assess the uniformity and effectiveness of the implementation of the ROP, which outlines the oversight process for operating reactors. This self-assessment applies to all seven cornerstones of the ROP, and all processes and procedures in place that are utilized to implement the ROP.

Reactors that are under construction or are in decommissioning are not within the scope of this process, nor are materials licensees or small modular or research and test reactors.

0307-04 RESPONSIBILITIES AND AUTHORITIES 04.01 Director, Division of Reactor Oversight (DRO), Office of Nuclear Reactor Regulation (NRR)

a. Oversees the implementation of the ROP self-assessment program.

b. Develops policies and procedures for the ROP self-assessment program.

c. Reviews, approves, and ensures issuance of the annual ROP self-assessment SECY and other supporting reports.

d. Identifies, with assistance from regional division directors, the topics for focused assessments.

e. Confirms, as indicated by IMC 0040 Document Issuing Forms, any significant ROP changes that require effectiveness reviews.

f. Ensures ROP program area leads and ROP baseline inspection procedure (IP) leads are assigned appropriately.

04.02 Directors, Division of Security Operations (DSO) and Division for Preparedness and Response (DPR), Office of Nuclear Security and Incident Response (NSIR)

a. Assist in identifying topics for focused assessments.

b. Support NSIR office-level review and concurrence on the annual ROP self-assessment SECY.

c. Support the ROP implementation audits, baseline inspection program reviews, and focused assessments, as applicable.

d. Ensure data is collected and submitted to facilitate analysis of the ROP performance metrics.

Issue Date: 05/03/22 3

0307 04.03 Regional Directors, Divisions of Reactor Safety, Reactor Projects, Operating Reactor Safety, Radiological Safety and Security, and Nuclear Materials Safety (as applicable)

a. Assist in identifying topics for focused assessments.

b. Review the annual ROP self-assessment SECY, as coordinated by the regional Technical Support and Assessment Team Lead or Branch Chief.

c. Support the ROP implementation audits, baseline inspection program reviews, and focused assessments, as applicable.

d. Ensure data is collected and submitted to facilitate analysis of the ROP performance metrics.

04.04 Chief, Reactor Assessment Branch (IRAB)

a. Develops and updates governance documents for the ROP self-assessment program.

b. Ensures data from all sources are collected and consolidated to facilitate analysis of the ROP performance metrics.

c. Recommends and implements improvements to the ROP self-assessment program.

d. Ensures coordination and execution of planned ROP self-assessment activities.

e. Monitors the effectiveness of corrective actions and improvements to the ROP that are developed in response to recommendations from self-assessment activities.

f. Develops the annual ROP self-assessment SECY.

04.05 Chief, Reactor Inspection Branch (IRIB)

a. Ensures data are collected and consolidated to facilitate inspection program analyses.

b. Reviews results of baseline IP monitoring, effectiveness reviews, and other analysis conducted by IP leads and other IRIB staff.

c. Provides support or staffing support to the baseline inspection program review, ROP implementation audits, or focused assessments, as applicable.

04.06 ROP Program Area Leads

a. Monitor implementation and effectiveness of assigned program areas of the performance indicator (PI) program, the inspection program, the significance determination process (SDP), and the assessment program.

b. Collect and analyze self-assessment data for the previous year and develop the annual program evaluation for assigned program area for input into the annual ROP SECY.

Issue Date: 05/03/22 4

0307 04.07 ROP Baseline Inspection Procedure Leads

a. Monitor ROP program execution data as it relates to all assigned baseline IPs as described in IMC 0307, Appendix B, Reactor Oversight Process Self-Assessment Baseline Inspection Program Monitoring and Comprehensive Review, and act as appropriate to address any identified issues.

b. Participate and/or provide support or subject matter expertise and input to the baseline inspection program review every 5th year.

04.08 ROP Inspection Manual Chapter/Inspection Procedure Leads1

a. Perform the actions as described in Section 06.02 of this IMC regarding the effectiveness reviews of significant ROP changes.

b. Monitor ROP program execution data as it relates to all assigned IMCs and IPs and act as appropriate to address any identified issues, in accordance with the memorandum Staff Expectations for Inspection Procedure and Inspection Manual Leads of Reactor Oversight Process Governance Documents, (ML19219A225, non-public) and in accordance with IMC 0040, IMC 0308, IMC 0801, and Management Directive (MD) 8.13.

0307-05 SELF-ASSESSMENT BACKGROUND AND BASIS The ROP is the NRC's primary means of ensuring that commercial nuclear power plants are operated safely, securely, and in accordance with applicable regulations. It is important that the ROP be periodically evaluated and improved when necessary to ensure continued achievement of its specified goals and intended outcomes. Additional information on the history and basis of the ROP self-assessment program can be found in Section 05.07 of IMC 0308.

The ROP self-assessment process has been an integral part of the staffs implementation of the ROP since its inception in April 2000. The ROP development model presented in SECY-99-007, Recommendations for Reactor Oversight Process Improvements, dated January 8, 1999, included a steady-state process evaluation, or self-assessment process, that utilize measured objectives and predetermined success criteria to monitor the performance of the ROP. The results of these initial self-assessment efforts helped to determine the effectiveness of the new ROP, and in SECY-00-0049, Results of the Revised Reactor Oversight Process Pilot Program, dated February 24, 2000, the staff stated that it would continue to perform ROP self-assessments to collect additional lessons learned and gain insights from the new oversight process. In SECY-01-0114, Results of the Initial Implementation of the New Reactor Oversight Process, dated June 25, 2001, the staff reaffirmed that the ROP will continue to require close scrutiny and oversight and established a self-assessment program that will identify additional areas for improvement. Additionally, in the same SECY, the staff committed to, as part of the Agency Action Review process continue to report to the Commission on an annual basis the results of its self-assessment and any significant changes to the ROP. These staff commitments form the purpose and the reporting requirements of the ROP self-assessment program. In the SRMs to SECY-00-0049 (Part 1 and Part 2), dated March 28, 2000, and May 17, 2000, the Commission approved the implementation of the new ROP program, including implementation of the associated self-assessment program.

1 This includes ROP Baseline IP leads.

Issue Date: 05/03/22 5

0307 05.01 ROP Goals As noted in IMC 0308, Reactor Oversight Process Basis Document, the goals of the ROP include the four specific program goals of being objective, risk-informed, understandable, and predictable. Each of these ROP goals support the NRCs mission and characterize the manner in which the agency intends to achieve its strategic goals of safety and security: to ensure adequate protection of public health and safety and the environment, and to ensure adequate protection in the secure use and management of radioactive materials.

The four ROP goals are summarized below:

Objective - Decisions are based on factual information and uninfluenced by emotion, surmise, or personal prejudice.

Risk-informed - Risk insights are considered along with other factors (such as engineering judgment, safety limits, redundancy, and diversity) to better focus licensee and regulatory attention on issues commensurate with their importance to health and safety.

Understandable - The process and its results are clear and written in plain language.

Predictable - More than one individual can follow the same defined process and arrive at the same conclusion in a consistent manner (i.e., repeatable).

05.02 Principles of Good Regulation As noted in the Strategic Plan, the NRC maintains its regulatory competence, conveys that competence to stakeholders, and promotes trust in the agency by adhering to the longstanding Principles of Good Regulation and its organizational values. These principles focus on ensuring safety and security while appropriately balancing the interests of the NRC's stakeholders, including the public and licensees. The five Principles of Good Regulation are summarized below:

Independence - Ethical performance and professionalism should influence regulation. Final decisions must be based on objective, unbiased assessments of all information, and must be documented with reasons explicitly stated.

Openness - The public must be informed about and have the opportunity to participate in the regulatory processes as required by law.

Efficiency - Regulatory decisions should be made without undue delay. Regulatory activities should be consistent with the degree of risk reduction they achieve.

Clarity - Regulations should be coherent, logical, and practical. Agency positions should be readily understood and easily applied.

Reliability - Regulatory actions should always be fully consistent with written regulations and should be promptly, fairly, and decisively administered.

The goals of the ROP are consistent with the Principles of Good Regulation.

Issue Date: 05/03/22 6

0307 05.03 ROP Intended Outcomes The ROP self-assessment process utilizes program evaluations and performance metrics to determine its success in meeting the goals and intended outcomes of the ROP. The intended outcomes of the ROP, which support its basis and are incorporated into various ROP processes, include the following:

Monitor and assess licensee performance Identify performance issues through NRC inspection and licensee PIs Determine the significance of identified performance issues Adjust resources to focus on significant performance issues Evaluate the adequacy of corrective actions for performance issues Take necessary regulatory actions for significant performance issues Communicate inspection and assessment results to stakeholders Make program improvements based on evaluation of stakeholder feedback and lessons learned Ensure reliable and predictable program implementation 05.04 Relation to NRC Strategic Plan, NUREG-1614 The ROP self-assessment program fulfills the ROP planned program review described in Appendix C of the current NRC Strategic Plan, Fiscal Years 2018-2022. In that way, the ROP self-assessment program is one aspect of ensuring the agency meets its goals as identified in the Strategic Plan.

05.05 Self-Assessment Approach The ROP self-assessment approach has evolved since the staff conducted the first ROP self-assessment in 2001 after the first year of ROP implementation. In 2015, the self-assessment program was redesigned as a 3-element program to measure effectiveness of and adherence to the ROP, to evaluate effectiveness of recent ROP changes, and to perform in-depth reviews of specific areas of interest. The 3-element program also addresses and aligns with Recommendation 8 from the Commission-directed independent assessment, Reactor Oversight Process Independent Assessment 2013 (ADAMS Accession No. ML14035A571), to revise the ROP self-assessment process to better solicit and assess both tactical and strategic feedback.

In 2020, the staff improved the self-assessment process, maintaining the 3-element approach while realigning the periodicity, scope, and type of reviews with a mature ROP (SECY-20-0039, Revisions to the ROP Self-Assessment Program, dated April 30, 2020). The enhanced self-assessment approach ensures that the ROP is being implemented reliably (consistently and as designed) across all regional and headquarters offices. Additionally, the new approach ensures that the staff appropriately invests resources to streamlined reviews and assessments that reveal high-value

Issue Date: 05/03/22 7

0307 improvements in ROP program efficiency and effectiveness. Finally, the revised program ensures that to the maximum extent possible, self-assessment activities leverage ROP program data monitoring and analytics to evaluate ROP effectiveness.

The self-assessment approach consists of three distinct elements as described in this manual chapter:

1. Measure regional and headquarters program effectiveness and uniformity in implementing the ROP,

2. Assess effectiveness of recent ROP changes and evaluate the NRCs response to significant licensee events or declining licensee performance, and

3. Perform focused assessments of specific ROP program areas, including the baseline inspection program.

0307-06 FORMAT AND STRUCTURE FOR SELF-ASSESSMENT The self-assessment program consists of three distinct elements as noted in section 05.05 above. This section provides additional detail for each aspect of the ROP self-assessment program, organized by the three program elements.

06.01 Element 1 - Measure Regional and Headquarters Program Effectiveness and Uniformity in Implementing the ROP

a. Objective Performance Metrics A set of performance metrics will be monitored and assessed as an integral part of each annual ROP self-assessment. These performance metrics align with the Principles of Good Regulation and are consistent with the goals and intended outcomes of the ROP.

The performance metrics are designed to be objective and measurable based on readily available ROP program execution data, and to maximize the use of existing databases.

Metric data is collected and analyzed by region and agency-wide for comparison purposes and to ensure reliable and uniform program implementation. A detailed description of these performance metrics is contained in Appendix A.

Each metric in Appendix A includes a definition and basis, the criteria to determine whether it is met, the organization responsible for gathering the data, program area(s) affected, and cross-references to the principle(s) of good regulation, ROP goals, and ROP intended outcomes that each metric is intended to support. A graded approach is used for performance criteria:

1. Green - A metric is considered Green if it meets or exceeds the specified criterion that represents expected performance and does not warrant further evaluation;

2. Yellow - A metric is considered Yellow if it falls within the specified range that warrants further evaluation and potential staff action to correct before the acceptance criterion has been exceeded; and

3. Red - A metric is considered Red if it meets the criterion that represents unexpected performance and necessitates further evaluation and likely staff action to address the cause(s) for the failed metric.

Issue Date: 05/03/22 8

0307 Separate and distinct from the ROP performance metrics, the NRC utilizes a planning, budgeting, and performance management process to ensure that the performance goals of the NRC Strategic Plan are properly assessed, and that key performance measures and program goals are met. The Agency performance management framework, which includes performance indicators and targets in the Annual Performance Plan and Congressional budget justification, are the primary means of determining whether the strategic performance goals are being met.

The ROP self-assessment program is not intended to replicate or replace this activity; however, many of the ROP self-assessment program metrics are the same as or similar to performance indicators found in the operating reactor business line annual performance plan. When applicable the same data should be provided, and the same guidance should be followed for both the ROP self-assessment program metrics and the performance indicators that are part of the annual performance plan to simplify data collection and to ensure clear and consistent data reporting.

The ROP performance metrics will be reviewed as part of the annual ROP self-assessment process to evaluate their effectiveness in providing a useful assessment of the ROP. Metrics will be added, deleted, or modified as necessary to provide a meaningful assessment of ROP implementation and effectiveness.

b. ROP Data Trending Focus Areas The NRR/DRO staff monitors ROP program execution data related to an established set of data trending focus areas on a monthly basis. The purpose of the ROP data trending program is to supplement the ROP performance metrics in identifying positive or negative trends (as compared to historical averages or expected trends) in selected focus areas preferably spanning all four ROP program areas (inspection, PIs, SDP, and assessment). Early identification of these trends will guide staff toward areas where weakness or strength may exist, and/or where staff action may be required to improve program performance. Appendix A includes a list of the data trending focus areas, including a description, program area(s) affected, and the principles of good regulation each data trending focus area is intended to support. To the maximum extent possible, this data trending will be accomplished via automated data analysis and visualization software in a dashboard-style display.

Should any significant trends or insights be identified, the data will be provided to the appropriate program area lead for further analysis and action, including input to the annual metrics report and ROP self-assessment paper. Additionally, significant observations from the data trending focus areas will inform future ROP self-assessment efforts, including topics for change effectiveness reviews, focused assessments, and regional ROP implementation audits.

The ROP data trending focus areas will be reviewed as part of the annual ROP self-assessment process to evaluate their effectiveness as a tool in evaluating effectiveness and uniformity in ROP implementation. Data trending focus areas will be added, deleted, or modified as necessary to provide a meaningful assessment of ROP implementation and effectiveness.

c. ROP Program Area Evaluations

Issue Date: 05/03/22 9

0307 ROP program area leads evaluate each of the four ROP program areas (the PI program, the inspection program, the SDP, and the assessment program) for their effectiveness, potential improvements, and potential future focus areas on an annual basis.

Specifically, the ROP program area leads conduct their respective program area evaluations based on ROP performance metrics data and analysis, ROP data trending insights, other ROP program execution data, internal (e.g., regional staff and resident inspectors) and external (e.g., ROP monthly public meeting discussions with Nuclear Energy Institute (NEI) staff or members of the public, Regulatory Information Conference discussions, or insights from participation in other industry events) feedback, and other relevant information. The evaluations for each ROP program area also include summaries of significant changes over the past year. The ROP program area evaluations align directly with and fulfill the intent and scope of the ROP planned program review as stipulated in Appendix C to the current volume of the NRCs Strategic Plan, Fiscal Years 2018-2022, NUREG-1614.

d. ROP Implementation Audits NRR will lead an annual ROP implementation audit of one region each year on a rotating basis. The objective of the audit is to appraise regional program performance in terms of effectiveness and uniformity of implementation of the ROP. The audit will be conducted in two parts: (1) a set of standardized, data-driven audit items covering the four ROP program areas that will objectively evaluate the regions implementation of definitive program requirements (meets/does not meet), and (2) one or two focus areas where the audit team will perform a deeper-dive review. IMC 0307, Appendix C, Reactor Oversight Process Self-Assessment ROP Implementation Audit, contains the guidance for this self-assessment element. Should there be any programmatic recommendations resulting from an ROP implementation audit, these recommendations will be entered into the ROP Lessons Learned Tracker for disposition.

06.02 Element 2 - Assess Effectiveness of Recent ROP Changes and Evaluate the NRCs Response to Significant Licensee Events or Declining Licensee Performance

a. Assess Effectiveness of Recent, Significant ROP Changes Significant changes to the ROP that require Commission approval prior to implementation, consistent with MD 8.13, Reactor Oversight Process, will undergo effectiveness reviews. Additionally, an effectiveness review shall be performed of the status of all of the ROP recommendations from any of the activities described in Section 06.02.b. within 4 years of the report issuance. NRR/DRO management has discretion to approve any additional ROP changes that may warrant effectiveness reviews, as proposed by NSIR, regional offices, and NRR/DRO staff. As part of the IMC/IP revision process for these significant ROP changes, the responsible IMC/IP lead will identify on the Document Issuing Form (see IMC 0040) the specific data streams that they will monitor, as well as the duration of monitoring needed to determine change effectiveness and any unintended consequences. To the maximum extent possible, IMC/IP leads will use existing, automated ROP program execution data streams. As necessary, the effectiveness review lead will engage regional representatives or other ROP program or IP/IMC leads to shed light on the effectiveness review or any effects of the change on their programs.

Once the monitoring period is complete, the resulting data analysis, along with any additional insights, will form the basis for the staffs assessment of change effectiveness.

Issue Date: 05/03/22 10 0307 At a minimum, each effectiveness review will include a review of the basis of the change, verification of the intended outcomes of the specific change, an assessment of unintended consequences of both individual and collective changes (e.g., cumulative impacts) made to the ROP during the period of time under consideration, an explanation of how effectiveness was measured, and verification that the changes remain consistent with the ROP goals, Principles of Good Regulation, and ROP intended outcomes as specified in Sections 05.01 through 05.03 of this manual chapter. If the review concludes that a change has been ineffective or warrants further improvement, adjustments will be considered as needed to more fully address the issue or concern.

An effectiveness review report memo template is available for use at ML19274B459 (non-public) for the formatting and minimum requirements of the effectiveness review.

As appropriate, recommendations from the effectiveness review requiring revisions to ROP program documents shall be tracked through the ROP feedback form process governed by IMC 0801, Inspection Program Feedback Process. The staff will include a summary of the results of the effectiveness review(s) completed each year in the annual ROP self-assessment SECY.

b. Evaluate NRC Response to Significant Licensee Events and Declining Licensee Performance If an incident investigation team (IIT) is warranted in response to a significant licensee event (see MD 8.3), part of the scope of the investigation will consider whether NRC activities preceding and during the event were timely and adequate. If a licensees performance has declined such that the licensee is in Column 4 of the ROP Action Matrix, and a supplemental IP 95003 is performed, a portion of that inspection (see IP 95003, Sections 02.11 and 03.11) includes a review of the NRCs assessment and inspection process (including the PI program, the inspection program, the SDP, and the assessment program) at the subject facility. If a licensees performance has declined such that the licensee has entered the IMC 0350 process, the IMC 0350 oversight panel will collect lessons learned from the IMC 0350 process and any subsequent recommended changes to the ROP and provide them to NRR/DRO (see IMC 0350, Sections 0350-07.01.f. and 0350-09.n.).

The overall purpose of these assessments is to determine the effectiveness of NRC response to significant licensee events and declining licensee performance. These inward-looking evaluations are an important aspect of ROP self-assessment, in that they can provide complex feedback regarding NRC oversight and incident response capabilities, guidance, and execution. Additionally, in the case of supplemental inspections, the evaluations provide a direct assessment of whether the ROP (as implemented) adequately detected declining licensee performance, and where the ROP might be improved in that regard.

As described below in Section 06.02.c., the complex ROP feedback from the response to significant licensee events and declining licensee performance is tracked to completion using the ROP lessons learned tracker. As the feedback from one of these activities (IIT, IP 95003, IMC 0350) is being incorporated into the ROP, an effectiveness review shall be performed of the status of all of the recommendations from that effort, in accordance with Section 06.02.a. of this IMC.

c. ROP Lessons Learned Tracker

Issue Date: 05/03/22 11 0307 NRR/DRO has the overall responsibility to gather feedback, lessons learned, and recommendations for ROP program improvement and to ensure they are adequately addressed. The staff maintains a non-public SharePoint database (https://usnrc.sharepoint.com/teams/NRR-ROP-Lessons-Learned-Tracker/Lists/ROP%20Tracker/) of the lessons learned from the evaluations as described in Section 06.02.b. to track the status of complex, longer-term program changes, as they often involve multiple internal and external stakeholders to evaluate and resolve and may require Commission approval to revise the policy and implement the changes, as required by MD 8.13. This lessons learned tracker is also used to track other complex ROP feedback as appropriate, such as complex ROP action items from Government Accountability Office or the NRC Office of the Inspector General audits or any other complex ROP feedback from other ROP self-assessment activities. As applicable, the staff will enter less complex ROP recommendations pertaining to only one IMC or IP into the ROP feedback form process governed by IMC 0801. The ROP feedback form process and ROP lessons learned database ensure that ROP recommendations are gathered, assessed, and tracked to completion. The staff will compile and provide status updates of ROP changes periodically to NRC management and will provide an overall summary of the staffs efforts in this area in the annual ROP self-assessment SECY.

06.03 Element 3 - Perform Focused Assessments of Specific ROP Program Areas, Including the Baseline Inspection Program

a. Focused Assessments Every 3 years, NRR/DRO, in consultation with the regions and NSIR, will designate one topic for a focused assessment that delves more deeply into a programmatic area of the ROP that shows indications of potential weakness or areas for improvement. The topic may be prompted by any of the data sources described in section 07.01, or any other insights, and consideration should be given to ensure that focused assessments are performed across a number of the four ROP program areas (i.e., PI program, inspection program, SDP, and assessment program) at least once within a ten-year period. For the inspection program, the baseline inspection program review described in Section 06.03.b below is considered to be a focused assessment and may be counted as such if it aligns with the focused assessment periodicity. The selected focused assessment topic will be presented to senior agency management at the Agency Action Review Meeting (AARM). The Office Director of NRR will make the final determination of the selected focused assessment topic.

NRR/DRO will assemble a working group and develop a charter that tailors the review to the subject area, specifically defines the scope of the focused assessment, identifies the attributes to be verified and relevant standards or guidance, and intended outcomes.

The assessments can verify consistency of ROP program area implementation among the regions, verify an aspect of the program over all four regions, or focus on specific areas within one or two regions. The assessments will utilize ROP program execution data to the maximum extent possible in analyzing program performance or gaps, while also typically including focused surveys and/or interviews to gather feedback and perspectives from affected stakeholders. The assessments are generally led by NRR/DRO staff, with working group members including staff with relevant experience from NRR, stakeholders from other program offices, and the regions.

Issue Date: 05/03/22 12 0307 The three-year periodicity of the focused assessments is set such that the team/working group has sufficient time to complete the assessment, develop recommendations, communicate appropriately, and implement the resulting program changes, if any.

Because of the comprehensive nature of the focused assessments, the recommendations resulting from these efforts often require Commission communication in the form of Information or Notation Vote SECY papers, or Commissioners Assistant Notes, as appropriate. The focused assessment teams/working groups should consider these requirements when developing their project plans. Changes resulting from these focused assessments will also frequently meet the threshold for conducting an effectiveness review as described in Section 06.02.a above, and special attention shall be paid to monitor appropriate data streams for both effectiveness and unintended consequences.

b. Baseline Inspection Program Comprehensive Review and Routine Monitoring Every fifth year after all regions have received an ROP implementation audit, the staff will conduct a baseline inspection program review, with engagement from NRR/DRO and regional Division management, and participation from a cross-section of baseline IP leads and regional representatives.

The lead for each baseline IP will conduct data-driven monitoring of assigned baseline IPs. Based on the results of the monitoring, baseline IP leads will act as necessary to maintain the effectiveness of each baseline IP. The data-driven approach described in this aspect of the self-assessment process does not preclude each IP leads responsibility to maintain relationships and avenues for feedback from resident inspectors and regional staff, including periodically observing or participating in inspections to verify baseline IP adequacy and clarity firsthand.

Guidance for the baseline inspection program comprehensive review and baseline IP monitoring is provided in Appendix B to this manual chapter.

The results of evaluations conducted under the three elements of the ROP self-assessment program will be documented or referenced in the annual ROP self-assessment report. The results will also be presented to senior NRC management at the AARM. The staff will enter lessons learned and recommended program improvements from ROP self-assessment activities into the ROP feedback form process or ROP lessons-learned tracker, as appropriate.

0307-07 DATA COLLECTION AND STAKEHOLDER FEEDBACK 07.01 Data Sources and Collection NRR/DRO has the overall responsibility for data collection, with support from the regional offices, NSIR, and other NRC organizations, as necessary. A variety of methods and sources are used to collect data regarding the performance of the ROP. These methods include compiling data and information from the Reactor Program System (RPS) or other existing databases, internal and external stakeholder surveys and interviews, independent audits, responses to Federal Register notices, industry-level indicators and operating experience, agency performance indicator databases, program evaluations and document reviews, and other stakeholder interactions.

Issue Date: 05/03/22 13 0307 To the extent possible, data collection is from agency databases and the need for ad hoc, manually developed data is minimized, particularly for the ROP performance metrics, ROP data trending focus areas, data-driven portions of the ROP implementation audit, and baseline IP monitoring efforts. Since the self-assessment program heavily relies on the quality of the data contained in the RPS database, it is imperative that the regions and other internal stakeholders ensure the accuracy and timeliness of the RPS data. NRR/DRO staff will periodically evaluate the need to modify existing systems such as RPS (or add new automated systems) to ensure efficiency and consistency in obtaining necessary ROP program execution data.

Data to support the ROP performance metrics is typically collected quarterly. Data collection and reporting is typically completed within 45 calendar days of the end of the quarter under review. In cases where the NRC performance plan performance indicators match the ROP performance metrics, the regions are not required to double-report that data on a quarterly basis as it can be pulled from on the agencys performance management SharePoint site. However, all regions and NSIR will compile and submit ROP performance metrics data at the end of each calendar year. Due to the short timeframe required to develop the annual ROP self-assessment SECY, this end-of-calendar-year reporting is requested to be completed within 15 calendar days of the end of the calendar year. Offices and regions shall take care to ensure that data is collected and reported accurately.

07.02 Soliciting and Evaluating Stakeholder Feedback The staff emphasizes stakeholder involvement and open communication regarding the ROP. The staff uses a variety of communication methods to ensure that all stakeholders can access ROP information and have an opportunity to participate in the process and provide feedback. The staff actively seeks feedback and implements improvements to the ROP based on evaluation of feedback and insights from all stakeholders.

a. External Stakeholder Feedback Opportunities

1. The staff conducts periodic public working-level meetings with NEI, the industry, and interested stakeholders to discuss the status of ongoing refinements to the ROP. The staff provides a meeting notice with an agenda so that interested stakeholders can determine beforehand whether the NRC plans to discuss topics of interest. The staff also offers the opportunity for public comment at the end of each topical area to enhance the public's ability to engage relevant staff members on topics discussed during these meetings. Additional topic-specific public meetings are held, as warranted.

During these public meetings, the staff provides and collects completed public meeting feedback forms, which are used to evaluate the effectiveness of public messaging and use of plain language. These public meeting feedback forms will be used to inform external communication initiatives to ensure that external content and publications relative to the ROP are clear and understandable.

2. The staff conducts public meetings or other engagement activities in the vicinity of each operating reactor site to discuss the results of the NRCs assessment of the licensees performance. These annual engagements provide an opportunity for interested stakeholders to engage NRC staff on its role in ensuring safe and secure plant operations, either by asking questions during the meeting or engaging NRC

Issue Date: 05/03/22 14 0307 staff. Public meeting feedback forms (described above) are collected during these meetings as well.

3. The staff maintains numerous external Web pages to communicate current ROP-related information and results. The staff monitors the ROP Web pages to ensure information accuracy.

Additionally, the staff maintains an ROP Contact Us form on the external NRC Website. External stakeholders can use this web-based form to provide feedback or ask questions regarding ROP implementation. The staff will provide an initial response acknowledging receipt of the feedback generally within 5 working days and will provide a more complete technical response generally within 45 days. Feedback received through this mechanism that is outside the scope of the ROP will be forwarded to the Office of Public Affairs and/or other appropriate organizations for their consideration. All feedback received through with the ROP Contact Us form will be maintained per the public ADAMS records guidelines and requirements.

4. The staff typically sponsors a breakout session at the annual Regulatory Information Conference focused on ROP-related issues and topics of interest, or sponsors an ROP poster session, so that interested stakeholders can gather information, ask questions, or provide feedback about the ROP. Additional ROP topics may be discussed during the regional breakout sessions.

5. The staff conducts external stakeholder surveys and/or interviews as needed to perform focused assessments per Element 3 of the ROP self-assessment process.

6. Independent assessments of ROP-related program areas from external agencies such as the Government Accountability Office provides valuable external feedback on the efficacy and consistency of the ROP.

b. Internal Stakeholder Feedback Opportunities

1. NRR and NSIR staff and management conduct periodic conference calls with regional management and staff to discuss current issues associated with the ROP.

The staff also meets periodically with regional managers to discuss more complex ROP topics and issues.

2. NRR and NSIR staff participate in each regions inspector counterpart meeting and/or end-of-cycle meetings so that regional staff and management can provide feedback on ROP implementation.

3. The ROP feedback process described in IMC 0801 allows NRC staff to identify concerns or issues and recommend improvements related to ROP governance or guidance documents. Responsible staff will respond to and address the feedback in accordance with the requirements and expectations of IMC 0801.

4. As described under Element 2 of the self-assessment program, the staff performs lessons-learned evaluations and recommends ROP program improvements upon completion of significant NRC reactive or supplemental inspection activities.

5. The staff maintains an internal Contact Us Form on the ROP Digital City Website.

Internal stakeholders can provide feedback or ask questions regarding ROP

Issue Date: 05/03/22 15 0307 implementation across a wide array of topic areas, including but not limited to, resident inspector support concerns. Similar to the external ROP Contact Us form, responsible staff will provide an initial response acknowledging receipt of the feedback generally within 5 days and will provide a more complete response generally within 45 days. Feedback received through this mechanism that is more technical in nature will follow the formal ROP feedback process described in IMC 0801.

6. The staff conducts focused surveys and/or interviews as needed to perform the focused assessments per Element 3 of the ROP self-assessment process.

7. While no longer an explicit aspect of the ROP self-assessment program, the staff has previously completed internal independent evaluations of the ROP, such as the Reactor Oversight Process Independent Assessment 2013 (ADAMS Accession No. ML14035A571). The staff maintains the ability to perform independent assessments of the ROP and may choose to do so when deemed appropriate.

The staff notes that there are a number of ROP communications efforts that promote regional uniformity and effectiveness in implementing the ROP, but do not necessarily provide an avenue for explicit feedback. For instance, NRR staff issues the inspector newsletter on a quarterly basis to share value-added findings, best practices, inspection guidance, and regulatory issues that are timely and have wide application and interest to inspectors and staff implementing the ROP. NRR and each of the four regions has a newsletter editorial board member who identifies potential topics and authors for newsletter articles and provides technical review of the content. Other efforts that promote uniformity and effectiveness include operating experience communications and the NRR staffs maintenance and frequent updates to the ROP dynamic website.

0307-08 ROP SELF-ASSESSMENT DATA ANALYSIS NRR/DRO has the overall responsibility for setting appropriate thresholds for evaluating ROP implementation and effectiveness, analyzing ROP program execution data and other self-assessment results, developing plans to address any potential areas of weakness, and implementing appropriate ROP improvements. NRR/DRO staff, with regional input, determine the performance thresholds for acceptable ROP implementation and effectiveness, which are provided for each ROP performance metric in Appendix A. There are no performance thresholds for the data trending focus areas listed in Appendix A, however, NRR/DRO staff use data analytics and graphics tools to determine any significant positive or negative trends in these areas.

Based on the staffs program-wide analysis of each calendar years self-assessment results, a positive overall favorable comparison of results to performance criteria would indicate that the ROP met its program goals and objectives for that calendar year. However, for any instance where an aspect of the ROP program may exhibit signs of weakness in terms of performance (e.g., a Red performance metric), the staff, as noted above, will develop a plan to further analyze and address the issue, determine causal factors, and develop recommended process improvements. A situation such as a Red performance metric may result in programmatic improvements and/or other action (such as clarifying program documents or training) to improve performance in the measured area. If no changes or other actions are being proposed or planned based on a Red metric, the staff will justify the reason(s) for the anomaly and why the anomaly is not indicative of poor performance or a program issue.

Issue Date: 05/03/22 16 0307 Due to their direct experience with the inspection and oversight programs gained through their implementation of the procedures, the regions and NSIR will be consulted during the data analysis and recommendation development process to ensure the regional insights are incorporated into the evaluation and change process.

0307-09 ROP SELF-ASSESSMENT REPORTING There are several periodic ROP self-assessment reports that serve different purposes as described below:

09.01 Annual ROP Performance Metric Report NRR/DRO develops an annual ROP performance metric report after the conclusion of each calendar year. The overall summary report must discuss any metrics that did not meet their pre-established criteria, the staffs analysis of the reasons for not meeting the criteria, and any actions taken or planned to change the program or improve its implementation. The report will briefly discuss any other significant lessons from the analyses of the metrics, even if the lesson is related to a metric that did meet its criteria.

The report will also identify any metrics not counted during the previous year with appropriate justification. Lastly, the report will also include any significant positive or negative trends or insights gained from the staffs monitoring and analysis of the ROP data trending focus areas, including any staff actions to address identified issues. The annual ROP performance metric report may be issued separately from or as an enclosure to the annual Commission paper discussed in Section 09.02.

09.02 Annual Commission Paper The staff presents the results of its annual self-assessment of the ROP in a SECY Information paper. The annual ROP self-assessment SECY also supports the AARM and the Commission briefing on the results of the AARM. This paper typically includes the results of each element of the ROP self-assessment program for the preceding calendar year including lessons learned, planned ROP self-assessment activities for the next calendar year, and any other significant ROP changes or updates.

The paper also presents the staffs overall conclusions as to whether the ROP has been successful in meeting the goals and intended outcomes of the ROP and the Principles of Good Regulation. The paper typically includes the ROP program area evaluations (Element 1 of the ROP self-assessment program) as an enclosure.

09.03 Reports Documenting ROP Self-Assessment Activities The results of the staffs effectiveness reviews, ROP implementation audits, and focused assessments may be documented in individual reports or may be incorporated into the annual ROP Commission paper discussed above. Recommendations made by internal and external stakeholders resulting from these self-assessment activities will be tracked and addressed under Element 2 of the ROP self-assessment program.

Issue Date: 05/03/22 17 0307 0307-10 REFERENCES This list of references encompasses the entire ROP self-assessment process, including the Appendices to this Chapter.

IMC 0040, Preparing, Revising and Issuing Documents for the NRC Inspection Manual IMC 0102, Oversight and Objectivity of Inspectors and Examiners at Reactor Facilities IMC 0305, Operating Reactor Assessment Program IMC 0306, Planning, Scheduling, Tracking, and Reporting of the Reactor Oversight Process (ROP)

IMC 0308, Reactor Oversight Process Basis Document IMC 0350, Oversight of Reactor Facilities in a Shutdown Condition Due to Significant Performance and/or Operational Concerns IMC 0608, Performance Indicator Program IMC 0609, Significance Determination Process IMC 0611, Power Reactor Inspection Reports IMC 0801, Inspection Program Feedback Process IMC 1245, Qualification Program for Reactor Inspectors IMC 2515, Light-Water Reactor Inspection Program Operations Phase IMC 2523, NRC Application of the Reactor Operating Experience Program in NRC Oversight Processes IP 95003, Supplemental Inspection for Repetitive Degraded Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs or One Red Input MD 3.5, Attendance at NRC Staff-Sponsored Meetings MD 8.13, Reactor Oversight Process MD 8.3, NRC Incident Investigation Program NEI 99-02, Regulatory Assessment Performance Indicator Guideline NRR Office Instruction OVRST-102, NRR Procedures for Processing Inspection Manual Documents (nonpublic)

NUREG-1614, NRC Strategic Plan SECY-99-007, Recommendations for Reactor Oversight Process Improvements, dated January 8, 1999

Issue Date: 05/03/22 18 0307 SECY-00-0049, Results of the Revised Reactor Oversight Process Pilot Program, dated February 24, 2000 SECY-01-0114, Results of the Initial Implementation of the New Reactor Oversight Process, dated June 25, 2001 SECY-20-0039, Revisions to the ROP Self-Assessment Program, dated April 30, 2020 SRMs for SECY-00-0049 Results of the Revised Reactor Oversight Process Pilot Program (Part 1 and Part 2)," March 28, 2000, and May 17, 2000 Staff report, Process Improvement Review of the Significance Determination Process, dated November 17, 2014 Staff report, Davis-Besse Reactor Vessel Head Degradation Lessons-Learned Task Force Report, dated September 30, 2002 END Appendices:

Appendix A, Reactor Oversight Process Self-Assessment Metrics and Data Trending Appendix B, Reactor Oversight Process Self-Assessment Baseline Inspection Program Monitoring and Comprehensive Reviews Appendix C, Reactor Oversight Process (ROP) Self-Assessment ROP Implementation Audit Appendix D, Power Reactor Resident Inspector Retention and Recruitment Program Monitoring and Assessment

Attachment:

, Revision History for IMC 0307

Issue Date: 05/03/22 Att1-1 0307 : Revision History for IMC 0307 Commitment Tracking Number Accession Number Issue Date Change Notice Description of Change Description of Training Required and Completion Date Comment Resolution and Closed Feedback Form Accession Number (Pre-Decisional, Non-Public Information)

N/A ML023650446 12/12/02 CN 02-045 Revised significantly to include a more detailed discussion of the role of inspectable and program area leads, the annual review of the baseline inspection program, and other aspects of the self-assessment program. The specific metrics for these roles were added to Appendix A.

N/A N/A N/A ML033640661 12/12/03 CN 03-039 Revised to provide greater detail for documenting the results of the annual inspection procedures reviews, and some metrics in Appendix A were modified to better align with the operating plan metrics and other program commitments.

N/A N/A N/A ML040150392 01/14/04 CN 04-001 Based on a decision at the DRP/DRS counterpart meeting held on December 17-18, 2003, metric IP-5 was revised to change the inspection report timeliness to 45 calendar days for all inspection reports, with exception of reactive inspection reports, which will stay at 30 days.

N/A N/A N/A ML060110214 02/20/06 CN 06-004 Revised to support the new safety performance measures of the NRCs Strategic Plan, to better define the ROP goals and intended outcomes, and to consolidate and clarify several of the performance metrics. Completed 4 year historical CN search.

N/A ML060110235 N/A ML063050572 11/28/06 CN 06-034 Revised to measure the effectiveness of the safety culture enhancements to the ROP, to clarify expectations regarding the resident demographics and staffing metrics, and to include a discussion of the consolidated response to external survey questions.

N/A

Issue Date: 05/03/22 Att1-2 0307 Commitment Tracking Number Accession Number Issue Date Change Notice Description of Change Description of Training Required and Completion Date Comment Resolution and Closed Feedback Form Accession Number (Pre-Decisional, Non-Public Information)

N/A ML073520141 01/10/08 CN 08-002 Revised to eliminate and consolidate several metrics, to separate Appendix A from the base IMC to serve as a stand-alone document, and to summarize and link to Appendix B on the ROP realignment process.

N/A ML073510410 W200800299 ML090300565 03/23/09 CN 09-010 Revised to address the Commission SRM dated June 30, 2008, to reflect the recently issued Strategic Plan for FY 2008 - 2013, and to reincorporate the security cornerstone in the ROP self-assessment process, and some metrics were revised for clarification purposes while others were removed to eliminate redundancy or unnecessary burden.

N/A ML090300620 N/A ML15216A347 11/23/15 CN 15-025 Significantly revised the process using a three-element approach designed to assess the effectiveness of a mature program.

N/A ML15225A105 N/A ML19274B865 05/29/20 CN 20-025 Complete reissuance (major rewrite, satisfies periodic/review update requirement) due to significant process changes to streamline the ROP self-assessment process, refresh the ROP metrics, and better leverage ROP program execution data in ROP self-assessment activities. This is the product of the 2019 holistic review of the ROP self-assessment program (reference SECY-19-0037 and SECY-20-0039).

N/A ML19274C587

Issue Date: 05/03/22 Att1-3 0307 Commitment Tracking Number Accession Number Issue Date Change Notice Description of Change Description of Training Required and Completion Date Comment Resolution and Closed Feedback Form Accession Number (Pre-Decisional, Non-Public Information)

N/A ML21341B399 05/03/22 CN 22-009 Revised to clarify that the same data and guidance should be used where appropriate for comparable ROP metrics and performance plan metrics Revised to align with conforming changes to the baseline inspection monitoring program as described in IMC 0307 Appendix B Revised to conform with the current (FY 2018-2022) and draft (FY 2022-2026) NRC Strategic Plans Revised to clarify the role of the ROP lessons learned tracker, and the ROP self-assessment portions of incident investigation teams, IP 95003, and IMC 0350 Revised to require an effectiveness review after the incorporation of ROP feedback from significant licensee events or declining licensee performance (Fort Calhoun Station Lessons Learned Report Recommendation 11, ML14128A376)

Revised to clarify that the decision maker for focused assessment topics is the Director of NRR Revised basis section for clarification and a reference to IMC 0308 N/A ML21341B400