NUREG-2250 Volume 1, Annual Evaluation Plan Fiscal Year 2023
| ML22066B059 | |
| Person / Time | |
|---|---|
| Issue date: | 04/30/2022 |
| From: | Office of Administration |
| To: | |
| Malone, Tina, Meyer M | |
| Shared Package | |
| ML22066B051 | List:
|
| References | |
| NUREG-2250 V1 | |
| Download: ML22066B059 (16) | |
Text
U.S. NUCLEAR REGULATORY COMMISSION Annual Evaluation Plan Fiscal Year 2023 NUREG-2250, Vol. 1 Photo courtesy of Dominion Energy
AVAILABILITY OF REFERENCE MATERIALS IN NRC PUBLICATIONS NRC Reference Material As of November 1999, you may electronically access NUREG-series publications and other NRC records at the NRCs Library at www.nrc.gov/reading-rm.html. Publicly released records include, to name a few, NUREG-series publications; Federal Register notices; applicant, licensee, and vendor documents and correspondence; NRC correspondence and internal memoranda; bulletins and information notices; inspection and investigative reports; licensee event reports; and Commission papers and their attachments.
NRC publications in the NUREG series, NRC regulations, and Title 10, Energy, in the Code of Federal Regulations may also be purchased from one of these two sources:
1. The Superintendent of Documents U.S. Government Publishing Office Washington, DC 20402-0001 Internet: www.bookstore.gpo.gov Telephone: (202) 512-1800 Fax: (202) 512-2104 2. The National Technical Information Service 5301 Shawnee Road Alexandria, VA 22312-0002 Internet: www.ntis.gov 1-800-553-6847 or, locally, (703) 605-6000 A single copy of each NRC draft report for comment is available free, to the extent of supply, upon written request as follows:
Address: U.S. Nuclear Regulatory Commission Office of Administration Digital Communications and Administrative Services Branch Washington, DC 20555-0001 E-mail: Reproduction.Resource@nrc.gov Facsimile: (301) 415-2289 Some publications in the NUREG series that are posted at the NRCs Web site address www.nrc.gov/reading-rm/
doc-collections/nuregs are updated periodically and may differ from the last printed version. Although references to material found on a Web site bear the date the material was accessed, the material available on the date cited may subsequently be removed from the site.
Non-NRC Reference Material Documents available from public and special technical libraries include all open literature items, such as books, journal articles, transactions, Federal Register notices, Federal and State legislation, and congressional re ports. Such documents as theses, dissertations, foreign reports and translations, and non-NRC conference proceedings may be purchased from their sponsoring organization.
Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are maintained at The NRC Technical Library Two White Flint North 11545 Rockville Pike Rockville, MD 20852-2738 These standards are available in the library for refer ence use by the public. Codes and standards are usually copyrighted and may be purchased from the originating organization or, if they are American National Standards, from American National Standards Institute 11 West 42nd Street New York, NY 10036-8002 Internet: www.ansi.org (212) 642-4900 Legally binding regulatory requirements are stated only in laws; NRC regulations; licenses, including technical speci fications; or orders, not in NUREG-series publications. The views expressed in contractor prepared publications in this series are not necessarily those of the NRC.
The NUREG series comprises (1) technical and adminis trative reports and books prepared by the staff (NUREG-XXXX) or agency contractors (NUREG/CR-XXXX), (2) proceedings of conferences (NUREG/CP-XXXX), (3) reports resulting from international agreements (NUREG/
IA-XXXX), (4) brochures (NUREG/BR-XXXX), and (5) compilations of legal decisions and orders of the Com mission and the Atomic and Safety Licensing Boards and of Directors decisions under Section 2.206 of the NRCs regulations (NUREG-0750).
DISCLAIMER: This report was prepared as an account of work sponsored by an agency of the U.S. Government.
Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third partys use, or the results of such use, of any information, apparatus, product, or process disclosed in this publication, or represents that its use by such third party would not infringe privately owned rights.
iii Abstract The U.S. Nuclear Regulatory Commission (NRC or the agency) is an independent agency established by the Energy Reorganization Act of 1974, which began operations in 1975 as a successor to the Atomic Energy Commission. The NRC is required by the Foundations for Evidence-Based Policymaking Act of 2018 to develop an annual evaluation plan. The Annual Evaluation Plan provides summary information on evaluations being initiated in fiscal year 2023. The Evidence Act defines an evaluation as an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency. The evaluations being conducted will assist in answering priority questions established in the Evidence-Building Plan or other evaluations determined to be significant, such as those required by statute or those of high value to the agency.
This evaluation plan contains two evaluations to be initiated in FY 2023.
Photo courtesy of Idaho National Laboratory
iv
v Table of Contents Abstract.................................................................................................................iii Foundations for Evidence-Based Policymaking Act of 2018............................1 About the NRC....................................................................................................... 1 Purpose of the Annual Evaluation Plan............................................................. 1 Requirements........................................................................................................ 2 Significant Evaluation Factors............................................................................. 2 Significant Evaluations......................................................................................... 3 Evaluation of the Strategic Workforce Planning Process.............................3 To what extent are NRCs workforce planning processes adequately accommodating potential workload fluctuations?........................................................3 Evaluation of the NRCs Licensing Actions..................................................... 5 To what extent are licensing actions performed by the NRC becoming more or less resource intensive over time and have there been any changes in work product quality?.................................................................................................................................5
1 Foundations for Evidence-Based Policymaking Act of 2018 The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act),1 signed into law January 14, 2019, emphasizes collaboration and coordination to advance data and evidence-building functions in the Federal Government. The Evidence Act statutorily mandates Federal evidence-building activities, open Government data, confidential information protection, and statistical efficiency. Evidence includes fact finding, performance measurement, policy analysis, and program evaluation used to make critical decisions about program operations, policy, and regulations, and to gain visibility into the impact of resource allocation on achieving program objectives. The Evidence Act builds on longstanding principles underlying Federal policies and data infrastructure investments supporting information quality, access protection, and evidence building and use.2 The Evidence Act requires the U.S. Nuclear Regulatory Commission (NRC), as a Chief Financial Officers Act agency, to develop an annual evaluation plan. This report is the NRCs Fiscal Year (FY) 2023 Annual Evaluation Plan and identifies significant evaluations to be initiated between October 1, 2022, and September 30, 2023.
About the NRC Congress created the NRC as an independent agency in 1974. Its mission is to license and regulate the Nations civilian use of radioactive materials, to provide reasonable assurance of adequate protection of public health and safety, to promote the common defense and security, and to protect the environment.
The NRC regulates commercial nuclear power plants, nuclear fuel cycle facilities, decommissioning of licensed facilities and sites, nuclear waste, and other uses of nuclear materials, such as the medical use of radioactive materials, through licensing, inspection, and enforcement of its requirements.
Purpose of the Annual Evaluation Plan This report fulfills the NRCs requirement to complete an Annual Evaluation Plan as established by Section 101(a)(2) of the Evidence Act.3 The Annual Evaluation Plan provides summary information on evaluations being initiated in FY 2023. The Evidence Act defines an evaluation as an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.4 Generally, evaluations are performed for organizational learning and improvement purposes and to enhance the agency mission. The evaluations being conducted will assist in answering priority questions established in the Evidence-Building Plan5 or other 1
Pub. L. No. 115-435, 132 Stat. 5529 (2019).
2 Office of Management and Budget (OMB) Memorandum M-19-23, Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning Guidance, pp. 1-2, July 10, 2019.
3
5 U.S.C. § 312(b).
4
5 U.S.C. § 311(3).
5 OMB Memorandum M-20-12, Phase 4 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018:
Program Evaluation Standards and Practice, March 10, 2020.
Annual Evaluation Plan Fiscal Year 2023
2 evaluations determined to be significant, such as those required by statute or those of high value to the agency. The evaluation plans are subject to change and will continue to be refined as new information or insights are identified.
The NRC is committed to meeting the intent of the Evidence Act by evaluating the efficacy and efficiency of its programs to help the agency achieve its mission. Evaluations and other evidence-building activities conducted by the NRC are expected to adhere to the standards discussed in the NRCs Evidence-Building and Evaluation Policy Statement (Agencywide Documents Access and Management System (ADAMS) Accession No. ML21124A234).
Requirements The Evidence Act requires the following information to be included in the Annual Evaluation Plan:
1) a description of key questions for each significant evaluation study that the agency plans to begin in the next FY; 2) a description of key information collections or acquisitions the agency plans to begin in the next FY; and 3) any other information included in guidance issued by the Director of OMB andadditional requirements for the Annual Evaluation Plan in OMB Memoranda M-19-23 and M-21-27.6 Significant Evaluation Factors The NRC uses several factors to identify significant evaluations. Generally, significant evaluations have the following characteristics:
- They have the potential for broad impacts, meaning evaluation results could be widely applicable and provide valuable information to a varied set of stakeholders, including Congress, the public, other Federal agencies and organizations as well as informing enhancements to existing NRC programs.
- They support NRC mission-related regulatory programs and activities and are likely to yield actionable and useful evidence to support agency decisionmaking on priority actions in a timely manner.
- They yield opportunities for significant change or improvement to NRC programs, policies, or organization.
- They strengthen agency risk management by identifying new or expounding on known programmatic risk areas.
- They retain broad support by agency leadership and are prioritized in response to legislative requirements or evolving external factors that have the potential to affect strategic priorities and objectives.
6 OMB Memorandum M-21-27, Evidence-Based Policymaking: Learning Agendas and Annual Evaluation Plans, June 30, 2021.
3 Sources for significant evaluations may come from various activities and programs across the agency such as the Reactor Safety, Nuclear Materials and Waste Safety, and Corporate Support Programs; priority questions from the Evidence-Building Plan; research; financial management; information technology; statutory requirements; and audit recommendations from the Government Accountability Office and the Office of the Inspector General.
Significant Evaluations The evaluations discussed below summarize the NRCs significant evaluation activities. All publicly available documents can be accessed through ADAMS.
Evaluation of the Strategic Workforce Planning Process To what extent are NRCs workforce planning processes adequately accommodating potential workload fluctuations?
Summary The goal of Strategic Workforce Planning (SWP) is to formulate strategies and action plans that enable the NRC to recruit, retain, and develop the workforce required to address emerging needs and workload fluctuations. The SWP process supports agency efforts to better forecast the amount and type of work now and in the future, and the workforce needed to perform this work. The SWP process also helps staff understand the future direction of the agencys work and empowers staff to plan their professional career development. The NRC will perform an evaluation that assesses the effectiveness and efficiency of the current SWP processes and will compare estimated workloads and staffing projections against actual results. The NRC will engage with internal stakeholders using the SWP process and benchmark against other Federal agencies.
Evaluation Objective The objective is to evaluate whether the NRCs approach to workforce planning, including associated processes and procedures, is effective in meeting its intended goals and whether it is being implemented efficiently.
Key Questions Key questions, as identified below, are designed to identify areas of strength as well as opportunities for improvement within the NRCs current approach to SWP.
- Can comparisons be derived from workload and staffing projections resulting from the SWP process for past years and the agencys formulated and executed budgets?
- To what extent is the relationship between inputs and outputs for the SWP process timely, cost-effective, and producing worthwhile results?
- To what extent is the SWP process producing worthwhile results (outputs, outcomes) and meeting agency objectives?
- Are the workload projections used to support workforce planning reliable to support the short and long-term planning efforts?
4
- What indicators or metrics can be established to assess the long-term success and benefits of the SWP process?
- To what extent has the SWP process identified or mitigated challenges across agency programs?
- How can the SWP process be made more efficient, and are the expended resources commensurate with the benefits?
Data Needs and Sources Data needs will require a combination of qualitative and quantitative data that draw from prior evaluations and assessments performed by the NRC and external organizations, interviews and focus groups, and NRC administrative data systems. The strategies for analyzing data will rely on statistical tools where necessary, but also incorporate visual or graphic representations of findings.
Evaluation Methods The NRC will conduct an evaluation for the SWP process to assess its effectiveness and efficiency, as well as identify areas for improvement, if any, to maximize the agencys efforts. Conducting a formative evaluation of the NRCs SWP process will require the agency to comprehensively assess multiple aspects of the overall approach to workforce planning as implemented on an agencywide basis. The formative evaluation will use a combination of methods that include an implementation assessment and a needs assessment. This evaluation will inform a subsequent outcome-focused evaluation to assess the cost-effectiveness of the SWP process. These methods are further discussed below.
Implementation AssessmentAn implementation assessment will be used to determine if the SWP process has been fully implemented as intended and would involve looking at each activity, assessing the way that it had been implemented, identifying and describing any bottlenecks in the processes, and assessing whether the outputs have been produced as intended.
Needs AssessmentA needs assessment will be used to assess whether the SWP process is meeting the needs of the agency and where gaps remain. This assessment may suggest ways of improving the existing process, including refocusing the process to better meet agency needs.
Cost-Effectiveness AnalysisA cost-effectiveness analysis will be used to determine whether the cost of conducting the SWP process is effectively balanced against the desired outcomes. The analysis will also compare the cost-effectiveness of two or more alternative scenarios for conducting the SWP process based on the results of the needs assessment.
Stakeholder Engagement In conducting this formative evaluation, the NRC staff will engage and seek input from the agencys management and staff involved in SWP efforts and associated activities. Outreach efforts will occur with Federal agencies to benchmark the NRC SWP process.
Challenges and Mitigating Strategies Resources expended by staff and management to support the SWP process may be aggregated and tracked with other generic administrative activities. This challenge may be mitigated by estimating resources based on discussions with staff and management to determine their level of effort.
5 Use and Dissemination Findings from this evaluation will be shared with agency staff and management to inform decisions that may influence the SWP process to ensure that the NRC is building and maintaining a workforce that is of appropriate size and makeup, and provides the necessary flexibility to adjust for various factors. The evaluation findings will be made publicly available, as appropriate, in a report or posted on the NRCs Web site.
Evaluation of the NRCs Licensing Actions To what extent are licensing actions performed by the NRC becoming more or less resource intensive over time and have there been any changes in work product quality?
Summary The NRCs regulatory process includes five main components: (1) developing regulations and guidance for applicants and licensees, (2) licensing or certifying applicants to use nuclear materials or operate nuclear facilities until license termination, (3) overseeing licensee operations and facilities to ensure that licensees comply with safety requirements, (4) evaluating operational experience at licensed facilities or at locations where licensed activities are performed, and (5) conducting research, holding hearings at the request of parties that may be affected by agency decisions, and obtaining independent reviews to support the agencys regulatory decisions. To receive a license or certification, or to amend, renew, or transfer an existing license, an entity or individual, must submit an application to the NRC. The NRC reviews applications to ensure that the application meets the relevant regulatory requirements and that the proposed activities will be conducted safely and in accordance with the common defense and security. License reviews use evidence, such as analyses, to support decisions that ensure the NRC is accomplishing its mission.
The NRC will perform an evaluation that analyzes licensing actions associated with licensing programs for which the agency has developed generic milestone schedules, as required by the Nuclear Energy Innovation and Modernization Act (NEIMA).7 The evaluation will determine if licensing actions performed by the NRC are becoming more or less resource intensive over time and whether there have been any changes in work product quality. This evaluation will provide a better understanding of the effectiveness of the licensing programs based on expended resources and quality of the work products for similar licensing actions. The evaluation may provide key insights to further risk inform the agencys licensing programs.
7 The NRC established generic milestone schedules for different types of licensing actions for requested activities of the Commission that involve the issuance of a final safety evaluation as required by Section 102(c) of NEIMA (Pub. L. No. 115-439, tit I, § 102(c),132 Stat. 5570 (2019)). The NRCs generic milestone schedules can be found on the Web site at https://www.nrc.gov/
about-nrc/generic-schedules.html.
6 Evaluation Objective The objective is to ensure that the NRCs licensing review and certification process is data-driven, evidence based, applies a risk informed approach, and reflects an appropriate and reasonable expenditure of resources to complete, based on the requested activity.
Key Questions Key questions, as noted below, are designed to identify areas of strength and opportunities for improving the effectiveness and efficiency of the NRCs licensing reviews.
- Do resource variances (e.g., full-time equivalent, contract funding) exist between similar types of licensing actions and if so, what relationship do they have to the quality (e.g., rigor, timeliness) of the documented analysis or external factors?
- Has the level of resources expended on each type of licensing action review changed over time?
If so, how and why?
- Has the timeliness of completing reviews of similar types of licensing actions, changed over time?
If so, how and why?
- Have changes to timeliness and resource expenditure affected the overall quality of the licensing review?
- Does the generic milestone schedule applicable to each type of licensing action review reasonably reflect the time actually needed to complete the licensing review? If not, why?
- Is the level of rigor applied to each type of licensing action appropriate and consistently demonstrated through activities that support independent and objective analysis by the NRC staff?
Data Needs and Sources An evaluation of the NRCs licensing review process will consider prior related evaluation and audit reports by the NRC and other Government organizations. Additionally, the evaluation will include a review of data pertaining to license fees billed by the NRC under Title 10 of the Code of Federal Regulations Part 170, Fees for Facilities, Materials, Import and Export Licenses and Other Regulatory Services under the Atomic Energy Act of 1954, as Amended; quality assessments of individual licensing action reviews (for each type of licensing action); results from product quality surveys completed by stakeholders; and other forms of feedback from licensees pertaining to the NRCs licensing action review process.
Evaluation Methods The NRC will perform a process evaluation to comprehensively assess the agencys licensing program to determine if licensing actions are being performed effectively and efficiently. The evaluation will (1) determine if similar licensing actions have become more or less resource intensive over time, (2) identify resource variances between similar licensing actions, (3) identify the factors contributing to the increase, decrease, and variance of resources for each type of licensing action, and (4) determine if there were any changes to the quality of the work products. Identified resource variances will be analyzed to better understand the factors leading to the variance. Potential factors may include varying levels of
7 complexity between similar licensing actions, varying analysis methods used by the NRC staff, and the quality of the applications submitted. The summative evaluation will use a combination of methods that include a comparative analysis, trend analysis, quality assessment, and a needs assessment. These methods are further discussed below.
Comparative AnalysisA comparative analysis will use statistical methods to quantitatively identify variances and distributions of the resources expended to perform licensing reviews within each type of generic milestone schedule. A multimodal distribution of the expended resource data could indicate a distinction between similar types of licensing actions. If variances are identified, a qualitative comparative analysis will be performed to determine the potential cause of the variance and provide a better distinction between similar licensing actions.
Trend AnalysisA trend analysis will be performed on the expended resource data for past licensing reviews within each type of generic milestone schedule. The trend analysis will quantify and explain trends over time to determine if similar licensing actions have become more or less resource intensive.
Quality AssessmentA quality assessment will be performed on the work products (e.g., safety evaluation reports, technical evaluation reports) to determine if quality has changed over time. The assessment will not assess the quality of the NRCs past decisions, but will focus on key elements of the products such as clarity, readability, and accessibility to supporting evidence and data.
Needs AssessmentA needs assessment will be used to evaluate the effectiveness of the licensing program. The needs assessment determines if the licensing program is achieving its intended outcomes and whether the needs of the agency are being met. This assessment may suggest ways of improving the existing process, including further risk informing licensing reviews to better meet agency needs.
Stakeholder Engagement To inform this process evaluation, the NRC staff will engage and seek input from the NRCs management and staff who are involved in or have working knowledge of and experience with various aspects of the licensing programs.
Challenges and Mitigating Strategies There are potential challenges that would affect the NRCs ability to conduct a comprehensive and meaningful evaluation of its licensing program. First, sufficient historical data may not be readily available for all types of licensing actions, which could limit the scope of the evaluation. This challenge may be mitigated by using advanced technologies such as machine learning to automate data extraction from documents.
Second, establishing consistent and collectively agreed upon criteria or metrics for determining the quality of the NRCs work products could pose a significant challenge. At the start of the evaluation, the evaluation team will collaborate across the agency to establish appropriate quality criteria or metrics.
Use and Dissemination Findings from this evaluation will be shared with agency staff and management to inform decisions that may influence the licensing program. The evaluation findings will be made publicly available, as appropriate, within a report or posted on the NRCs Web site.
NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION (12-2010)
NRCMD 3.7 BIBLIOGRAPHIC DATA SHEET (See instructions on the reverse)
- 1. REPORT NUMBER (Assigned by NRC, Add Vol., Supp., Rev.,
and Addendum Numbers, if any.)
NUREG 2250, Volume 1
- 2. TITLE AND SUBTITLE Annual Evaluation Plan Fiscal Year 2023
- 3. DATE REPORT PUBLISHED MONTH April YEAR 2022
- 5. AUTHOR(S)
Matthew Meyer, Jacki Storch, et. al.
- 6. TYPE OF REPORT Evaluation Plan
- 7. PERIOD COVERED (Inclusive Dates)
Annual
- 8. PERFORMING ORGANIZATION - NAME AND ADDRESS (If NRC, provide Division, Office or Region, U. S. Nuclear Regulatory Commission, and mailing address; if contractor, provide name and mailing address.)
Office of the Executive Director for Operations U.S. Nuclear Regulatory Commission Washington, DC 20555-001
- 9. SPONSORING ORGANIZATION - NAME AND ADDRESS (If NRC, type "Same as above", if contractor, provide NRC Division, Office or Region, U. S. Nuclear Regulatory Commission, and mailing address.)
Same as above
- 10. SUPPLEMENTARY NOTES
- 11. ABSTRACT (200 words or less)
The Annual Evaluation Plan provides summary information on evaluations being initiated in fiscal year 2023. The Evidence Act defines an evaluation as an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency. The evaluations being conducted will assist in answering priority questions established in the Evidence-Building Plan or other evaluations determined to be significant, such as those required by statute or those of high value to the agency. This evaluation plan contains two evaluations to be initiated in FY 2023.
- 12. KEY WORDS/DESCRIPTORS (List words or phrases that will assist researchers in locating the report.)
- 13. AVAILABILITY STATEMENT unlimited
- 14. SECURITY CLASSIFICATION (This Page) unclassified (This Report) unclassified
- 15. NUMBER OF PAGES
- 16. PRICE NRC FORM 335 (12-2010)
U.S. Nuclear Regulatory Commission NUREG-2250, Vol. 1 April 2022 www.nrc.gov STAY CONNECTED