SECY-20-0039, Revisions to the Reactor Oversight Process Self-Assessment Program
| ML19218A133 | |
| Person / Time | |
|---|---|
| Issue date: | 04/30/2020 |
| From: | Margaret Doane NRC/EDO |
| To: | Commissioners NRC/OCM |
| Mayer A, 415-1081, NRR/DIRS | |
| References | |
| 201100134 SECY-20-0039 | |
| Download: ML19218A133 (10) | |
Text
April 30, 2020 SECY-20-0039 FOR:
The Commissioners FROM:
Margaret M. Doane Executive Director for Operations
SUBJECT:
REVISIONS TO THE REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT PROGRAM PURPOSE:
The purpose of this paper is to notify the Commission of the U.S. Nuclear Regulatory Commission (NRC) staffs planned revisions to the Reactor Oversight Process (ROP) self-assessment program pursuant to Commission direction as described in Management Directive 8.13, Reactor Oversight Process, dated January 16, 2018 (Agencywide Documents and Access Management System (ADAMS) Accession No. ML17347B670). This paper does not address any new commitments.
SUMMARY
In SECY-19-0037, Reactor Oversight Process Self-Assessment for Calendar Year 2018, dated April 12, 2019 (ADAMS Accession No. ML19042A100), the staff shared its intent to conduct a limited ROP self-assessment in calendar year (CY) 2019 in order to conduct a holistic review of the ROP self-assessment program. As a result of its review, the staff identified program revisions to take a more modern, risk-informed approach to ROP self-assessment activities by more fully leveraging ROP program execution data and optimizing the program to focus on evaluating the effectiveness of the most significant ROP changes. The planned revisions to the self-assessment program were evaluated to ensure that the program remains effective in assessing whether the ROP provides objective, predictable, risk-informed, and understandable CONTACT: Andrea G. Mayer, NRR/DRO 301-415-1081
The Commissioners oversight to ensure safe and secure operation of nuclear power plants. The staff retained the existing three-element program structure but adjusted the review scope and periodicity of some elements and eliminated others that did not contribute to ROP self-assessment program effectiveness. The staff also updated ROP performance metrics to include data trending to more meaningfully measure NRC regional and program office ROP implementation effectiveness and uniformity. The staff plans to implement the revised ROP self-assessment program for CY 2020.
BACKGROUND:
Inspection Manual Chapter (IMC) 0307, Reactor Oversight Process Self-Assessment Program, dated November 23, 2015 (ADAMS Accession No. ML15216A347), and its appendices establish the ROP self-assessment program. The NRC staff revised the ROP self-assessment program in 2015 and notified the Commission of the revisions to the program in SECY-15-0156, Improvements to the Reactor Oversight Process Self-Assessment Program, dated December 11, 2015 (ADAMS Accession No. ML15310A086). The 2015 revision established three self-assessment program elements. The first element measured the effectiveness and implementation of the ROP with objective performance metrics and program area evaluations. Under the second element, the staff provided ROP status updates, monitored long-term revisions, and performed effectiveness reviews of recent changes. Under the third element, the staff performed focused assessments of selected aspects of the ROP, including baseline inspection procedure (IP) assessments, and formal peer reviews of regional offices.
DISCUSSION:
The staff solicited program feedback from ROP IP and IMC leads in the Office of Nuclear Reactor Regulation (NRR), the Office of Nuclear Security and Incident Response, and all four NRC regions. The staff also reviewed historical ROP self-assessments and ROP metrics iterations, international guidance on nuclear regulatory effectiveness evaluation, previous Government Accountability Office and NRC Office of the Inspector General audit reports pertaining to the ROP, NRC performance management measures, and other relevant reports and data sources that could provide insight.
The staff concluded that the 2015 changes to the ROP self-assessment program were effective and provided a robust ROP self-assessment framework. However, the staff also concluded that the number, scope, and frequency of self-assessment activities are not commensurate with the maturity of the ROP, given that the ROP self-assessment program has been a part of the ROP since its inception in 2000. This imbalance resulted in annual-to-biennial extensions in frequency of the regional peer review, baseline IP assessment, and focused assessment program elements between 2016 and 2017. The staff determined that there were opportunities to modernize the ROP self-assessment program by taking a more data-driven approach, including more timely monitoring of ROP program execution data, and other data-driven approaches to self-assessment activities. Lastly, the staff determined that the 2015 changes to the program relegated some routine IP/IMC lead responsibilities to periodic reviews, instead of maintaining ownership of routine IP/IMC data analysis and monitoring with the respective IP/IMC leads.
The staff re-affirmed that the ROP self-assessment program should retain the existing three-element approach, and plans to revise the program structure by introducing data-driven routine monitoring of certain ROP focus areas, such as data related to recent significant ROP changes, completion of the baseline inspection program, and trends in inspection findings, as
The Commissioners well as a data-driven approach to appraising regional ROP implementation effectiveness and uniformity. The staff also plans to provide revised guidance on the expected process and scope for ROP self-assessment activities. The next sections detail the changes to the ROP self-assessment program, organized by program element.
Element 1: Measure Regional and Program Office Effectiveness and Uniformity in Implementing the ROP The revised Element 1 measures regional and NRC headquarters effectiveness and uniformity in implementing the ROP, fulfilling NRR responsibilities described in paragraph (e) of Title 10 of the Code of Federal Regulations (10 CFR) Section 1.43, Office of Nuclear Reactor Regulation.
As described further below, this element includes streamlined ROP performance metrics, new ROP program data trending focus areas, and a new ROP implementation audit. The staff made no changes to the frequency or scope of the ROP program area evaluations.
ROP Performance Metrics The staff uses objective ROP performance metrics aligned to the NRCs Principles of Good Regulation to measure the effectiveness of and adherence to the ROP. As described in IMC 0307, Appendix A, Reactor Oversight Process Self-Assessment Metrics, the staff will continue to employ a graded approach to measure performance with Green (expected performance), Yellow (performance requiring further evaluation), and Red (unexpected performance requiring staff action to correct) criteria for each metric. The staff evaluated the existing 26 ROP performance metrics for effectiveness, redundancy, and potential for improvement in measuring whether the ROP is implemented uniformly across all regions and offices, and in accordance with governance documents. Although the staff found the ROP performance metrics to be effective overall, the staff determined that some individual metrics were ineffective at measuring their intended programmatic aspects, some were redundant to other internal controls, and some revisions were needed to optimize the remaining metrics.
As a result of its evaluation, the staff eliminated six metrics, combined four metrics into two, and transitioned one metric to the ROP data trending program discussed in the ROP Data Trending section of this paper. The staff added one new metric and revised several of the remaining metrics to ensure that the ROP metrics remain an effective measurement of ROP implementation. Revisions to metrics included adjustments to Green/Yellow/Red threshold criteria based on historical metrics data to ensure the thresholds were appropriately characterizing performance, revisions to conform to updated governance document requirements, and revisions to clarify metric measurement and scope. Overall, the new revision of IMC 0307, Appendix A, includes 18 (compared to 26 previously) ROP performance metrics aligned to each of the NRCs Principles of Good Regulation.
The staff eliminated metrics found to be redundant to other internal controls or ineffective measures of their associated Principles of Good Regulation in areas such as: resident inspector (RI) and senior resident inspector (SRI) maximum tour lengths, staff responsiveness to rarely used web-based forms, timeliness of reactive inspection entrance meetings, traceability of greater-than-Green inspection findings in internal databases and public-facing ROP web pages, performance of lessons learned evaluations of certain supplemental and reactive inspection activities, and consideration of operating experience insights during end-of-cycle assessment meetings.
The Commissioners The staff combined two different sets of metrics that measured (1) the staffs efficiency in completing two aspects of the significance determination process (SDP), and (2) the staffs review and correction of ROP-related public web pages (former clarity metrics C-1 and C-2, now combined into a single metric). Former ROP efficiency metrics E-4, Completion of Performance Deficiency Determinations and E-5, Completion of Final Significance Determinations, measured the staffs timeliness in completing each respective aspect of the SDP. The staff combined these two metrics into one broader metric that measures the staffs ability to complete the entire SDP for all inspection findings identified as potentially greater-than-Green (GTG) in 255 days or less.1 This change incorporates a recommendation from a CY 2018 effectiveness review documented in a memo titled, Results of the Calendar Year 2018 Reactor Oversight Process Self-Assessment Effectiveness Reviews on the Inspection Finding Resolution Management Pilot Program and Inspection Manual Chapter 0307 Pilot Metric E-4, dated May 9, 2018 (ADAMS Accession No. ML18123A319). While the staff will no longer report metric data on components of SDP timeliness, the staff will continue to monitor the timeliness of each component of the SDP. Finally, because the revised metric includes the entire SDP population of potentially GTG findings, instead of only those that have a final significance of GTG, it is a more accurate measure of SDP efficiency.
The staff determined that one metric measuring the staffs efficiency in dispositioning internal ROP feedback would be best evaluated via the new ROP data trending program discussed in the ROP Data Trending section of this paper. As the staff discussed in SECY-19-0037, the previous metric E-6, Responsiveness to ROP Feedback Forms, was ineffective, because it only measured the timeliness of current-year feedback forms and did not account for the backlog of ROP feedback forms awaiting disposition. The staff has eliminated this metric and, instead, will monitor the timeliness and inventory of ROP feedback forms in the ROP data trending program. Additionally, the staff has revised IMC 0801, Inspection Program Feedback Process, dated March 17, 2020 (ADAMS Accession No. ML19343A777) to incorporate an updated ROP feedback form process that aims to ensure that the highest priority ROP feedback is received and resolved in a timely manner.
The staff included one new metric in the reissued IMC 0307, Appendix A. The new metric, aligned with the reliability Principle of Good Regulation, measures whether inspection findings are adequately supported and documented such that when they are contested by licensees, the number of overturned violations should be infrequent. While this metric closely mirrors an agency-level metric tracked by the Office of Enforcement, the staff found value in evaluating supportability of inspection findings across the regions and program offices that implement the ROP.
The staff retained the continuity of reactor site coverage metric in the streamlined set of ROP performance metrics consistent with the staffs position in COMSECY-15-0014, Proposed Elimination of Annual Reporting Requirements for Specific Evaluations within the Reactor Oversight Process Self-Assessment Process, dated May 7, 2015 (ADAMS Accession No. ML15072A202). Additionally, the staff continues to recognize the importance of recruitment and retention of the agencys RIs and SRIs as the NRCs front line of regulatory oversight.
Accordingly, the staff is piloting IMC 0307, Appendix D, Power Reactor Resident Inspector Retention and Recruitment Program Monitoring and Assessment, dated May 21, 2019 (ADAMS Accession No. ML19045A287), to standardize collection and monitoring of indicators reflecting the NRCs ability to recruit and retain qualified resident inspection staff. This pilot addresses the 1 The staff updated the Congressional Budget Justification metric regarding SDP timeliness to reflect the same changes noted in this paragraph, effective fiscal year 2021.
The Commissioners staffs commitments in COMSECY-15-0014 to explore ways to better measure and provide insights into RI and SRI staffing and demographics, and to provide a means by which the staff collects and analyzes demographic analysis of the RI program for reporting to the Commission on a triennial basis. The staff provided the CY 2017 RI demographics analysis as an enclosure to SECY-19-0002, Implementation of Changes to the Resident Inspector Program, dated January 4, 2019 (non-publicly available), and the next RI demographic analysis that the staff will provide to the Commission will be for CY 2020.
ROP Data Trending The staff will take a more modern approach to ROP performance monitoring by incorporating routine ROP data trending efforts as a complement to the traditional ROP performance metrics.
Specifically, the staff will trend and analyze ROP program execution data in a number of focus areas throughout the calendar year to quickly identify and respond to unanticipated or adverse trends. The focus areas include inspection hours charged by site, baseline inspection completion progress, inspection findings per IP and per region, supplemental inspection hours, GTG findings overall and per region, open unresolved issues, ROP feedback form inventory and timeliness, and licensee event reports. Additional focus areas will be added as appropriate.
The data trending program will not have the Green/Yellow/Red thresholds. Instead, the staff will discuss significant trends or insights (as compared to historical averages or expected trends) revealed by the ROP data trending efforts in the annual ROP metrics report and consider them for inclusion in the annual ROP self-assessment SECY paper. The staff will also use the data trending results to inform other ROP self-assessment activities, including focused assessment topics and focus areas for ROP implementation audits (discussed below). The staff is currently revising IMC 0307, Appendix A, to incorporate the new ROP data trending self-assessment activity and the refreshed ROP performance metrics.
ROP Implementation Audit Element 3 of the ROP self-assessment program previously had a biennial regional peer review led by a regional staff member with a team comprised of regional and NRC headquarters staff.
The revised program will include an annual, NRR-led audit with a team comprised of NRC headquarters staff of 3 to 4 personnel and a representative from another region. One region each year, on a rotating basis (under Element 1 of the ROP self-assessment program),2 will be audited. The report of the audit results will be provided to the Regional Administrator with copies to the other regions and the Deputy Executive Director for Reactor and Preparedness Programs (DEDR). The regions will address the report including any actions taken to address audit findings to the DEDR with a copy to the Director of NRR. This revised approach ensures that NRR fulfills its regulatory responsibility to appraise regional ROP program performance in terms of effectiveness and uniformity under 10 CFR 1.43(e), while ensuring that the EDO responsibility to supervise the activities of NRR and the regions under 10 CFR 1.32(b) is fulfilled by the DEDR pursuant to the authorities delegated in Management Directive 9.17, Organization and Functions, Office of the Executive Director for Operations, dated May 26, 2015 (ADAMS Accession No. ML100680449).
2 Every fifth year, once all regions have completed the ROP implementation audit, the staff will conduct a comprehensive review of the baseline inspection program. The ROP implementation audit cycle will then begin again the next year with Region I. The baseline inspection program review is described in the Baseline Inspection Program Review section of this paper under Element 3.
The Commissioners The ROP implementation audit will consist of two parts: (1) a standardized, data-driven assessment of the regions implementation of the four ROP program areas (inspection, SDP, PIs, and assessment) using existing data sources (e.g., Replacement Reactor Program System
- Inspections); and (2) a deep-dive review of one or two focus areas. The selection of focus areas will incorporate regional management recommendations, DRO management focus, ROP trending data insights, and ROP performance metrics results. The audit will provide an objective assessment of whether the regions implementation of the program comports with ROP governance document requirements and the degree of uniformity between the audited regions implementation of the ROP and the other three regions. Specifically, the staff assesses regional uniformity through a feedback loop process where the regions not audited that year respond to the final report, detailing whether any findings from the audit are also applicable to their regional programs, including planned corrective actions. Additionally, since the first part of the audit is standardized, over a four-year rolling period, the staff can discern the degree to which regions are uniformly implementing the major program areas of the ROP. This revision should require the same level of staff effort as the previously conducted regional peer reviews.
The staff plans to issue a revision to IMC 0307, Appendix C, Reactor Oversight Process Self-Assessment Regional Peer Reviews, dated July 15, 2016 (ADAMS Accession No. ML16147A455), to incorporate guidance for the new ROP implementation audit.
Element 2: Assess Effectiveness of Recent ROP Changes and Evaluate NRC Response to Significant Licensee Events or Declining Licensee Performance The revised Element 2 uses a more modern, risk-informed approach to evaluating ROP changes. ROP status updates will no longer be part of ROP self-assessment program since they do not perform an assessment function, but the staff will continue to include relevant ROP status updates in the annual ROP self-assessment SECY. The staff will continue to track complex ROP feedback resulting from NRC staff evaluations of supplemental and reactive inspections.
Assess Effectiveness of Recent ROP Changes The first activity under Element 2 is the evaluation of the effectiveness of recent ROP program changes. Under the former Element 2, DRO management, in consultation with regional management, selected ROP changes each year to undergo effectiveness reviews. Under the revised Element 2, only significant3 ROP changes will undergo effectiveness reviews. This standardized threshold represents a risk-informed approach because it focuses the staffs review efforts on those changes with the highest potential impact to the ROP.
Additionally, the staff will use a more modern, data-driven approach to evaluate effectiveness of the significant ROP changes. Under the former Element 2, after an ROP change had been in effect for about 1 to 2 years, the staff generally convened a working group to complete the effectiveness review. Under the revised Element 2, as part of the formal document revision process, the responsible IP/IMC lead will identify specific ROP program execution data that they will monitor at a routine interval to determine change effectiveness. As a result, the staff will monitor significant ROP changes for effectiveness upon issuance of the change instead of waiting up to 2 years to do a backwards-looking review.
3 A significant ROP change is one that requires Commission approval before implementation, consistent with Commission direction in SRM-COMSECY-16-0022. The DRO Director may direct additional effectiveness reviews as appropriate.
The Commissioners For example, for a significant ROP change that involved major changes to an inspection program, the IP/IMC lead might choose to monitor the severity and number of inspection findings that result from implementation of that revised procedure to determine the effectiveness of the change and whether it has brought about any unintended consequences. This ensures that IMC/IP leads retain ownership of their programs and procedures and that they are empowered to routinely evaluate and maintain IMC/IP effectiveness. The real-time monitoring of significant ROP changes is also intended to identify adverse consequences early so that they can be addressed in a timely manner.
The staffs assessment of the effectiveness of changes will consider the results of data monitoring, data analysis, and any other additional insights, and will be included in the annual ROP self-assessment SECY. The goal of the standardized threshold is to ensure that effectiveness reviews are performed on the most significant changes; while the goal of the data-driven review approach is to reach conclusions on effectiveness reviews more efficiently.
The staff plans to revise IMC 0307 to incorporate this program revision.
Evaluate NRC Response to Significant Licensee Events or Declining Licensee Performance Under the former Element 2, the staff was directed to monitor ROP revisions and assess effectiveness of recent program changes. Monitoring ROP revisions included tracking recommended ROP changes resulting from complex ROP feedback from completed supplementary and reactive inspections. Revised Element 2, now directs the staff to assess effectiveness of recent ROP changes and evaluate NRC response to significant licensee events or declining licensee performance. This revised title and guidance better emphasizes the importance of the staffs inward-looking evaluations of certain supplemental and reactive inspection activities as an aspect of ROP self-assessment. These evaluations assess the NRCs response to significant licensee events or declining licensee performance, often resulting in valuable, complex, lessons learned for the ROP. The requirements for these evaluations are already established in NRC Management Directives and ROP governance documents, and so this is not a programmatic change. The staff continues to track resolution of this complex, longer-term ROP feedback in the ROP Lessons Learned Tracker, so that, in conjunction with the revised ROP feedback form process, all recommended changes to the ROP are gathered, assessed, and resolved in a timely manner.
Element 3: Perform Focused Assessments Revised Element 3 of the ROP self-assessment program retains the focused assessments and reviews of the baseline inspection program, as these are important aspects of ensuring that the basic building blocks of the ROP continue to remain effective. However, acknowledging the maturity of the ROP and the staffs continuous monitoring of ROP program execution data, the staff increased the periodicity between focused assessments of selected ROP program areas and the periodicity between comprehensive reviews of the baseline inspection program.
Focused Assessments Under revised Element 3, the staff will continue to perform focused assessments of selected ROP program areas but will perform them on a triennial, versus biennial, basis. This allows the staff to more effectively complete the assessment, communicate the results as necessary (including preparing Commission correspondence), and implement the recommended actions before commencing another focused assessment.
The Commissioners Baseline Inspection Program Monitoring and Comprehensive Reviews Under former Element 3, the staff assessed each baseline IP on a biennial basis, as described in IMC 0307, Appendix B, Reactor Oversight Process Baseline Inspection Procedure Assessments and Reviews, dated August 25, 2017 (ADAMS Accession No. ML17165A508).
Overall, the staff determined that this approach removed responsibilities from the baseline IP leads for routine monitoring of their respective IPs. Further, the staff determined that while this approach did look at each baseline IP individually, it did not include a holistic review of the baseline inspection program. Accordingly, the staff revised Element 3 to include (1) routine monitoring of individual baseline IPs by the respective lead, and (2) a comprehensive baseline inspection program review every fifth year, after all regions complete a cycle of ROP implementation audits.
Under revised Element 3, the baseline IP leads will conduct routine monitoring of their respective baseline IPs by analyzing ROP program execution data related to the implementation of their assigned baseline IPs (e.g., inspection hours charged, number and severity of findings, inspection sample trends, etc.). Based on the routine monitoring results, baseline IP leads will act as necessary to maintain the effectiveness of each assigned baseline IP.
Additionally, the staff will conduct a comprehensive baseline inspection program review every fifth year after all regions have completed the ROP implementation audit with NRR headquarters and regional division director oversight. This review will holistically evaluate the entire baseline inspection program for necessary program-wide adjustments in inspection hours or samples, opportunities for increased use of risk information in the program, any additions or deletions of baseline IPs, and any other revisions. The staff will leverage ROP program execution data to the maximum extent possible, including in the review process and in data-driven decision-making in dispositioning recommendations from the review. The staff is currently revising Appendix B to IMC 0307 to implement these changes and will reissue it in its entirety.
CONCLUSION:
The revised ROP self-assessment program aligns the type, scope, and frequency of self-assessment activities with the maturity of the program; aligns NRR responsibilities with program attributes and applicable regulations; maximizes the use of ROP program execution data in the conduct of self-assessment activities; and refreshes the ROP metrics with the inclusion of data trending and a more focused set of objective, formal metrics. The revisions to the ROP self-assessment program described in this paper increase the efficiency and effectiveness of the program while maintaining a robust evaluation of the ROP each year. The changes to the program also improve the staffs ability to evaluate and assess the effectiveness of larger, more complex changes to the ROP, because the effectiveness review process is repeatable, data-driven, and limited in scope to only those changes that require Commission approval before implementation. The annual ROP self-assessment will continue to fulfill the requirements described in Appendix C to the NRC Strategic Plan.
The Commissioners COORDINATION:
The Office of the General Counsel has reviewed this Commission paper and has no legal objection. The Office of the Chief Financial Officer has reviewed this Commission paper for resource implications and has no objections.
Margaret M. Doane Executive Director for Operations Margaret M. Doane Digitally signed by Margaret M.
Doane Date: 2020.04.30 16:49:11 -04'00'
The Commissioners
SUBJECT:
REVISIONS TO THE REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT PROGRAM DATE APRIL 30, 2020 ADAMS Accession Number: ML19218A133
- via email SECY-012 OFFICE NRR/DRO/IRSB Tech Editor* NRR/DRO/IRSB/LA NRR/DRO/IRSB* NRR/DRO/IRAB*
NAME AMayer JDougherty IBetts PMcKenna AMasters DATE 8/9/19 8/9/19 08/12/19 8/28/19 9/10/19 OFFICE NRR/DRO/IRIB* RI*
RII*
RIII*
RIV*
NAME THipschman DLew (RLorson for)
LDudes (JMunday for)
DRoberts SMorris (TInverso for)
DATE 9/10/19 9/27/19 9/25/19 10/3/19 10/02/19 OFFICE NSIR:D*
OCFO*
NRR/DRO:D OGC*
NRR:D NAME BHolian (MLombard for)
MSampson CMiller TCampbell HNieh (EBenner for)
DATE 9/30/19 10/3/19 10/16/19 10/23/19 11/20/19 OFFICE EDO NAME MDoane DATE 4/30/20 OFFICIAL RECORD COPY