NUREG-2250 Volume 2, Annual Evaluation Plan Fiscal Year 2024

NUREG-2250, Vol. 2, Annual Evaluation Plan Fiscal Year 2024

ML23073A062
Person / Time
Issue date:	03/31/2023
From:	NRC/EDO
To:
References
NUREG-2250 V2
Download: ML23073A062 (1)

Text

NUREG-2250, Vol. 2

Evaluation Annual Plan Fiscal Year 2024 AVAILABILITY OF REFERENCE MATERIALS IN NRC PUBLICATIONS

NRC Reference Material Non-NRC Reference Material

As of November 1999, you may electronically access Documents available from public and special technical NUREG-series publications and other NRC records at the libraries include all open literature items, such as books, NRCs Library at www.nrc.gov/reading-rm.html. Publicly journal articles, transactions, Federal Register notices, released records include, to name a few, NUREG-Federal and State legislation, and congressional re-series publications; Federal Register notices; applicant, ports. Such documents as theses, dissertations, foreign licensee, and vendor documents and correspondence; reports and translations, and non-NRC conference NRC correspondence and internal memoranda; bulletins proceedings may be purchased from their sponsoring and information notices; inspection and investigative organization.

reports; licensee event reports; and Commission papers and their attachments. Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are NRC publications in the NUREG series, NRC regulations, maintained at and Title 10, Energy, in the Code of Federal Regulations The NRC Technical Library may also be purchased from one of these two sources: Two White Flint North 11545 Rockville Pike

1. The Superintendent of Documents Rockville, MD 20852-2738

U.S.GovernmentPublishingOffice Washington, DC 20402-0001These standards are available in the library for refer-Internet: www.bookstore.gpo.gov ence use by the public. Codes and standards are usually Telephone: (202) 512-1800copyrighted and may be purchased from the originating Fax: (202) 512-2104 organization or, if they are American National Standards, from

2. The National Technical Information ServiceAmerican National Standards Institute 5301 Shawnee Road11 West 42nd Street Alexandria, VA 22312-0002New York, NY 10036-8002 Internet: www.ntis.gov Internet: www.ansi.org 1-800-553-6847 or, locally, (703) 605-6000(212) 642-4900

A single copy of each NRC draft report for comment Legally binding regulatory requirements are stated only in is available free, to the extent of supply, upon written laws; NRC regulations; licenses, including technical speci-request as follows: fications;ororders,notinNUREG-seriespublications. The views expressed in contractor prepared publications in this Address: U.S. Nuclear Regulatory Commission series are not necessarily those of the NRC.

Officeof AdministrationThe NUREG series comprises (1) technical and adminis-Digital Communications and Administrative trativereportsandbookspreparedbythestaff(NUREG-Services Branch XXXX)oragencycontractors(NUREG/CR-XXXX),(2)

Washington, DC 20555-0001 proceedingsofconferences(NUREG/CP-XXXX),(3)

E-mail: Reproduction.Resource@nrc.gov reports resulting from international agreements (NUREG/

Facsimile: (301) 415-2289 IA-XXXX),(4)brochures(NUREG/BR-XXXX),and(5) compilations of legal decisions and orders of the Com-Some publications in the NUREG series that are posted mission and the Atomic and Safety Licensing Boards and at the NRCs Web site address www.nrc.gov/reading-rm/ of Directors decisions under Section 2.206 of the NRCs doc-collections/nuregs are updated periodically and may regulations(NUREG-0750).

differfromthelastprintedversion. Althoughreferences DISCLAIMER: This report was prepared as an account to material found on a Web site bear the date the material of work sponsored by an agency of the U.S. Government.

was accessed, the material available on the date cited Neither the U.S. Government nor any agency thereof, nor may subsequently be removed from the site. any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third partys use, or the results of such use, of any information, apparatus, product, or process disclosed in this publication, or represents that its use by such third party would not infringe privately owned rights.

Abstract

The U.S. Nuclear Regulatory Commission (NRC or the agency) is an independent agency established by the Energy Reorganization Act of 1974, which began operations in 1975 as a successor to the Atomic Energy Commission. The NRC is required by the Foundations for Evidence-Based Policymaking Act of 2018 to develop an annual evaluation plan. The Annual Evaluation Plan provides summary information on evaluations being initiated in fiscal year 2024 and the status of evaluations initiated in previous fiscal years. The Evidence Act defines an evaluation as an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency. The evaluations being conducted will assist in answering priority questions established in the Evidence-Building Plan or other evaluations determined to be significant, such as those required by statute or those of high value to the agency. This evaluation plan contains two evaluations to be initiated in FY 2024: NRCs University Nuclear Leadership Program and NRCs Regulatory Analysis Process for Rulemaking.

iii iv Table of Contents

Abstract................................................................................................................iii

Foundations for Evidence-Based Policymaking Act of 2018........................... 1

About the NRC......................................................................................................1

Purpose of the Annual Evaluation Plan............................................................1

Requirements.......................................................................................................2

Evaluation Factors...............................................................................................2

Significant Evaluations........................................................................................3 Evaluation of the NRCs University Nuclear Leadership Program............. 3

How can the NRC better leverage research conducted through NRC-sponsored university research and development grants?..............................................................3 Evaluation of the NRCs Regulatory Analysis Process for Rulemaking...... 5

Status Update of Evaluation Activities.............................................................7 Evaluation of the NRCs Strategic Workforce Planning Process................. 8

To what extent are NRCs workforce planning processes adequately accommodating potential workload fluctuations?.......................................................8 Evaluation of the NRCs Knowledge Management Program....................... 9 Evaluation of the NRCs Licensing Actions.................................................. 11 To what extent are licensing actions performed by the NRC becoming more or less resource intensive over time and have there been any changes in work product quality?..............................................................................................................................11

v

Annual Evaluation Plan Fiscal Year 2024

Foundations for Evidence-Based Policymaking Act of 2018 The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act)1, signed into law January 14, 2019, emphasizes collaboration and coordination to advance data and evidence-building functions in the Federal Government. The Evidence Act statutorily mandates Federal evidence-building activities, open Government data, confidential information protection, and statistical efficiency. Evidence includes fact finding, performance measurement, policy analysis, and program evaluation used to make critical decisions about program operations, policy, and regulations, and to gain visibility into the impact of resource allocation on achieving program objectives. The Evidence Act builds on longstanding principles underlying Federal policies and data infrastructure investments supporting information quality, access protection, and evidence building and use.2 The Evidence Act requires all Chief Financial Officers Act agencies, which includes the NRC, to develop an annual evaluation plan. This report is the NRCs Fiscal Year (FY) 2024 Annual Evaluation Plan and identifies significant evaluations to be initiated between October 1, 2023, and September 30, 2024.

About the NRC

Congress created the NRC as an independent agency in 1974. Its mission is to license and regulate the Nations civilian use of radioactive materials, to provide reasonable assurance of adequate protection of public health and safety, to promote the common defense and security, and to protect the environment.

The NRC regulates commercial nuclear power plants, nuclear fuel cycle facilities, decommissioning of licensed facilities and sites, nuclear waste, and other uses of nuclear materials, such as the medical use of radioactive materials, through licensing, inspection, and enforcement of its requirements.

Purpose of the Annual Evaluation Plan

This report fulfills the NRCs requirement to complete an Annual Evaluation Plan as established by Section 101(a)(2) of the Evidence Act.3 The Annual Evaluation Plan provides summary information on evaluations being initiated in FY 2023. The Evidence Act defines an evaluation as an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.4 Generally, evaluations are performed for organizational learning and improvement purposes and to enhance the agency mission. The evaluations being conducted will assist in answering priority questions

1 Pub. L. No. 115-435, 132 Stat. 5529 (2019).

2 Office of Management and Budget (OMB) Memorandum M-19-23, Phase 1 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Learning Agendas, Personnel, and Planning Guidance, pp. 1-2, July 10, 2019.

3 5 U.S.C. § 312(b).

4 5 U.S.C. § 311(3).

1 established in the Evidence-Building Plan (a component of the NRCs Strategic Plan) or other evaluations determined to be significant, such as those required by statute or those of high value to the agency.5 6 The evaluation plans are subject to change and will continue to be refined as new information or insights are identified.

The NRC is committed to meeting the intent of the Evidence Act by evaluating the efficacy and efficiency of its programs to help the agency achieve its mission. Evaluations and other evidence-building activities conducted by the NRC are expected to adhere to the standards discussed in the NRCs Evidence-Building and Evaluation Policy Statement.7 Requirements

The Evidence Act requires the following information to be included in the Annual Evaluation Plan:

1) a description of key questions for each significant evaluation study that the agency plans to begin in the next FY;

2) a description of key information collections or acquisitions the agency plans to begin in the next FY; and

3) any other information included in guidance issued by the Director of OMB.

Evaluation Factors

The NRC uses several factors to identify significant evaluations. Generally, significant evaluations have the following characteristics:

• They have the potential for broad impacts, meaning evaluation results could be widely applicable and provide valuable information to a varied set of stakeholders, including Congress, the public, other Federal agencies and organizations as well as informing enhancements to existing NRC programs.

• They support NRC mission-related regulatory programs and activities and are likely to yield actionable and useful evidence to support agency decision-making on priority actions in a timely manner.

• They yield opportunities for significant change or improvement to NRC programs, policies, or organization.

5 OMB Memorandum M-20-12, Phase 4 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Program Evaluation Standards and Practice, March 10, 2020.

6 The NRCs Strategic Plan for Fiscal Years 20222026 is available at https://www.nrc.gov/about-nrc/

plans-performance/strategic-planning.html 7 86 Fed. Reg. 29,683 (June 3, 2021).

2

• They strengthen agency risk management by identifying new or expounding on known programmatic risk areas.

• They retain broad support by agency leadership and are prioritized in response to legislative requirements or evolving external factors that have the potential to affect strategic priorities and objectives.

Sources for significant evaluations may come from various activities and programs across the agency such as the Reactor Safety, Nuclear Materials and Waste Safety, and Corporate Support Programs; priority questions from the Evidence-Building Plan; research; financial management; information technology; statutory requirements; and audit recommendations from the Government Accountability Office and the Office of the Inspector General.

Significant Evaluations

The evaluations discussed below summarize the NRCs significant evaluation activities. All publicly available documents can be accessed through the Agencywide Documents Access and Management System.

Evaluation of the NRCs University Nuclear Leadership Program

FY 2022 Evidence-Building Plan Priority Question: How can the NRC better leverage research conducted through NRC-sponsored university research and development grants?

Summary

The NRCs University Nuclear Leadership Program awards funding to universities for research and development, fellowships, scholarships, and faculty development grants. The program is intended to develop a workforce capable of supporting the design, construction, operation, and regulation of nuclear facilities and the safe handling of nuclear materials. The NRC will evaluate the University Nuclear Leadership Program to identify opportunities to leverage university grants to support NRC research needs as well as the capabilities of the nuclear workforce and the nuclear industry. The evaluation will include activities such as internal and external stakeholder engagement, process reviews, and benchmarking with other Federal agencies. Ideally, the evaluation will reveal strategies for more effective use of research funding in the future.

Evaluation Objective

The objective is to evaluate the effectiveness of the NRCs University Nuclear Leadership Program to ensure it is meeting its intended goals and identifying program improvements.

3 Key Questions

Key questions, as identified below, are intended to guide the initial direction of the evaluation.

Addressing the key questions can result in the identification of areas of strength as well as opportunities for improving the NRCs University Nuclear Leadership Program.

• To what extent are the NRC-funded grants aligning with Congressional program goals and/or requirements?

• To what extent have the NRC funded grant projects aligned with agency priorities?

• To what extent do the grant limitations, such as limited funding, affect the potential benefit of the proposed university grants?

• To what extent has the University Nuclear Leadership Program enhanced the capabilities of and demonstrated tangible benefits to the nuclear workforce and the nuclear technology?

Data Needs and Sources

This evaluation will require a combination of qualitative and quantitative data associated with benefits to universities and the nuclear industry programs (e.g., job creation, academic interest, outputs, developments) through contributions from the University Nuclear Leadership Program. The NRC will review grant program periodic and final reports, data on outputs and developments resulting from grants, and qualitative data on university experience with the grant program. Additionally, the NRC will need information from other Federal agencies on their grant programs, award criteria, and agency benefits, to support a comparative benchmarking analysis. The strategies for analyzing data will rely on statistical tools where necessary, but also incorporate visual or graphic representations of findings.

Evaluation Methods

The NRC will conduct a formative evaluation to assess effectiveness and efficiency, as well as identify areas for improvement, if any, to maximize the agencys efforts. The evaluation will determine the extent to which:

a) The University Nuclear Leadership Program is achieving the intended benefits for the NRCs mission and the nuclear workforce development and research needs.

b) Current processes for awarding funding are effective and efficient in meeting the program objectives.

c) Improvements to the program can be made to better leverage the NRC-sponsored grants.

Additionally, the NRC will conduct an analysis that identifies the University Nuclear Leadership Programs effect on universities and the nuclear industry.

Stakeholder Engagement

In conducting this formative evaluation, the NRC will engage and seek input from universities, internal NRC staff, other Federal agencies, and the nuclear industry.

4 Challenges and Mitigating Strategies

The NRC may be challenged with drawing a cohesive connection between the awarded funding and the full impact and benefits that resulted from the grant. To mitigate this challenge, the NRC will identify the stakeholder groups that benefit from the grant and conduct interviews and research to better interpret the connection between the end product and the beneficial impacts.

Use and Dissemination

Findings from this evaluation will be shared with agency staff and management to inform decisions that may influence the University Nuclear Leadership Program. The evaluation findings will be made publicly available, as appropriate, in a report or posted on the NRCs website.

Evaluation of the NRCs Regulatory Analysis Process for Rulemaking

The NRCs FY 2022 Capacity Assessment identified the need for a formative evaluation that will inform decision-making about the use of retrospective reviews of past rulemaking implementation to improve the effectiveness of the NRCs regulatory analysis process.8

Summary

As part of its rulemaking process, the NRC uses regulatory analyses to consider preferred alternatives from the potential courses of action studied. A regulatory analysis contains estimates of benefits and costs with a conclusion as to whether the proposed regulatory action is cost beneficial and documents the analysis in an organized and understandable format. The evaluation of the NRCs regulatory analysis process will systematically assess how agencies that perform retrospective reviews of past rulemaking implementation undertake that work, including methods, data needs, costs, as well as how agencies apply the results of these reviews to subsequent rulemaking. This information will inform NRCs approach to testing retrospective reviews with the aim of improving the effectiveness of the NRCs regulatory analysis process. If the initial information gathering indicates that the benefits of performing the retrospective review would outweigh the resource costs, then NRC may pilot this approach to better understand its benefits for the organization. Such a review would focus on whether the rulemaking implementation was reasonably estimated within the regulatory analysis, including the costs and benefits to the NRC and affected entities to support agency decision-making.

Evaluation Objective

The objective of the evaluation is to determine the extent to which retrospective reviews of past rulemakings and their implementation could enhance NRCs regulatory analysis process.

Key Questions

Key questions, as identified below, are designed to identify focus areas for this formative and descriptive evaluation.

8 The NRCs FY 2022 Capacity Assessment is available at https://www.nrc.gov/about-nrc/plans-perfor-mance/evidence-building-and-evaluation/capacity-assessment.html

5

• What does input from external stakeholders regarding the accuracy of regulatory analyses and identifying opportunities for improvement indicate about the potential benefits of retrospective reviews?

• What methods do agencies use to conduct retrospective reviews, what are the costs associated with these activities, and how do agencies analyze and appmly the results?

• To what extent, governmentwide, are procedures modified following retrospective reviews and do the modifications result in measurable improvements? How do agencies quantify those improvements?

• What processes do agencies that conduct retrospective reviews integrate in their regulatory analysis development process to facilitate conducting future retrospective reviews efficiently, and what changes would be needed in NRCs processes to similarly facilitate retrospective reviews?

• To what extent do agencies that conduct retrospective reviews consider the benefits of performing a retrospective review to outweigh the cost of conducting a retrospective review? What methods or measures are used to understand or account for the benefits of this approach?

Data Needs and Sources

This evaluation will require a combination of qualitative and quantitative data associated with the cost, impact to licensees, regulators, and stakeholders, and the benefits to public health and safety to establish a baseline of accuracy of recently completed regulatory analyses. Additionally, this evaluation will require qualitative and quantitative data from other Federal agencies that are conducting retrospective reviews, to understand their methods and processes, as well as their assessment of the benefits of the approach and the factors that facilitate its successful application. This evaluation will also require quantitative data associated with the costs to agencies to conduct retrospective reviews. The strategies for analyzing data will rely on statistical tools where necessary, but also incorporate visual or graphic representations of findings.

Evaluation Methods

The NRC will conduct a needs assessment to determine whether retrospective reviews of rulemakings could enhance the regulatory analysis process. The NRC will estimate the cost of conducting a retrospective review for a cross-section of rulemaking based on information gathered from Federal agencies as well as qualitative and quantitative input from licensees, regulators, and stakeholders.

Additionally, the NRC will conduct benchmarking with other Federal agencies that have a retrospective review process in place, as well as those agencies that do not conduct retrospective reviews. The purpose of benchmarking both cases is to develop an understanding of the evidence supporting the determination to implement the retrospective review process or not. This evidence will be used to inform an analysis to determine the extent to which the benefits of performing a retrospective review outweigh the cost.

Stakeholder Engagement

In conducting this needs assessment and cost benefit analysis, the NRC staff will engage and seek input from external stakeholders, NRC staff, and other Federal Agencies.

6 Challenges and Mitigating Strategies

The NRC may be challenged in obtaining sufficient internal and external data to determine the actual cost and level of effort to implement regulations. To mitigate this challenge, the staff will reach out to Federal partners and coordinate with OMB Clearance Officers to facilitate a data collection from external stakeholders if needed.

Use and Dissemination

Findings from this evaluation will be shared with agency staff and management to inform decisions that may influence the regulatory analysis program. The evaluation findings will be made publicly available, as appropriate, in a report or posted on the NRCs website.

Status Update of Evaluation Activities

In support of Title I of the Evidence Act, the NRC is building organizational capacity to perform evidence building activities and evaluations by establishing a dedicated team in the Office of the Executive Director for Operations. This dedicated team will provide a sustainable capacity to implement the requirements of the Evidence Act and will be an agencywide resource for evidence-building and evaluation; strategic planning; innovation and continuous learning; business analytics and solutions; and collaboration and communications on activities across the agency to avoid silos and duplication of efforts. The NRC is in the process of hiring program evaluators and has awarded a multi-year contract for designing and conducting evaluations and assessments. These staff and contractors will conduct evidence-building and evaluation activities, such as addressing priority questions identified in the NRCs FY 2022 Evidence Building Plan and findings in the NRCs FY 2022 Capacity Assessment. This team will provide a focused effort that will ensure the agency is building the capacity necessary to develop evidence to make evidence-based decisions.

In FY 2023, the NRC initiated two evaluations for Strategic Workforce Planning and Knowledge Management. The NRC has contracted with Pacific Research and Evaluation LLC (PRE) to conduct evidence building and evaluation work over a 5-year period. PRE uses a four phased approach to conduct evaluations with each phase resulting in development of a final product. Phase one has been completed for both evaluations and has resulted in the development of a logic model that will be used to inform the design of the evaluation.

7 Evaluation of the NRCs Strategic Workforce Planning Process

FY 2022 Evidence-Building Plan Priority Question: To what extent are NRCs workforce planning processes adequately accommodating potential workload fluctuations?

Summary

The goal of Strategic Workforce Planning (SWP) is to formulate strategies and action plans that enable the NRC to recruit, retain, and develop the workforce required to address emerging needs and workload fluctuations. The SWP process supports agency efforts to better forecast the amount and type of work now and in the future, and the workforce needed to perform this work. The SWP process also helps staff understand the future direction of the agencys work and empowers staff to plan their professional career development. The NRC will perform an evaluation that assesses the effectiveness and efficiency of the current SWP processes and will compare estimated workloads and staffing projections against actual results. The NRC will engage with internal stakeholders using the SWP process and benchmark against other Federal agencies.

Evaluation Objective

The objective is to evaluate whether the NRCs approach to workforce planning, including associated processes and procedures, is effective in meeting its intended goals and whether it is being implemented efficiently.

Evaluation Status

In October 2022, the NRC evaluation team, together with its contractor consultant PRE, held a kick-off meeting with internal stakeholders to begin Phase 1 of the SWP evaluation. At this meeting, the leads for the SWP process provided an overview which included background information on the development of the process, data collected, timeframes for each of the process phases, and the overall objective.

Following the kick-off meeting, informational interviews were scheduled with key stakeholders that have critical roles in the SWP process from initial development to implementation and analysis, and background documentation was reviewed.

The informational interviews and background documentation review provided the basis for the development of a logic model, which will be used to design the evaluation in Phase 2 of the evaluation process. The Evidence Building Activities page on the NRC public website will have additional information on this evaluation as it progresses through the next three phases.

8 Evaluation of the NRCs Knowledge Management Program Summary

The Nuclear Regulatory Commission (NRC) defines Knowledge Management (KM) as a continuous, disciplined, and timely process of identifying, collecting, and using information to better accomplish the job. The NRC practiced the capturing, preservation, sharing, and use of organizational knowledge long before the term Knowledge Management came into common use. In 2006 with the establishment of a formal KM Program, the NRC initiated a more structured and systematic approach to KM. Since then, NRC has established over 37 Communities of Practice that serve to create opportunities for learning, building capability, and sharing knowledge. The agency has also established valuable KM tools like Nuclepedia, the KM Toolkit for Supervisors, other Office of the Chief Human Capital Officer Toolkits, the Career Mentoring Program, Weekly KM sessions for inspectors, KNOWvember, and Design Your KM Approach.

In 2017, the KM Programs focus shifted to address how the agency can better integrate KM practices and approaches into daily operations and do so systematically. NRC developed a Knowledge Management Strategy covering 2018-2022, which included a KM Program goal to Maintain NRCs regulatory and technical excellence by ensuring critical knowledge is captured, available and used by all NRC staff, present and future. The KM Strategy was updated for 2023-2027 (ML22259A022) and defined the following expected outcomes or desired state of the KM Program for 2027:

• Knowledge Management becomes routine and is integrated into daily work activities and work processes by employees who have a shared understanding of what it means and how it applies to their jobs.

• Knowledge is captured at all levels of the organization which results in improved productivity, allowing people to do their jobs more effectively

• Knowledge is consistently managed and readily accessible and used, improving the organizations ability to execute operations more efficiently.

• Knowledge Management progress is effectively measured, applied, and evaluated.

In April 2022, NRC issued NUREG-1614, Vol. 8, Strategic Plan, Fiscal Years 2022-2026 ( ML22067A170).

The Strategic Plan defines NRCs strategic goals, objectives, and key activities that will be used to achieve the agencys mission. Strategic Objective 2.2 states, Enable the workforce to carry out the agencys mission by leveraging modern technology, innovation, and knowledge management to support data-driven decisions in an evolving regulatory landscape. Additionally, Strategy 2.3.4 states, Improve knowledge management by identifying and capturing critical information and leveraging the agencys investment in modern information management and technology to enhance information accessibility and searchability. This further emphasizes how the KM Program is vital for the NRC to accomplish its mission.

9 Additionally in April 2022, NRC issued NUREG-2251, Vol. 1, Capacity Assessment for Statistics, Research, Evaluation, and Other Analysis, Fiscal Year 2022 (ML22066B054). One area it assessed was the KM Program. It included the following finding and mitigation strategy for KM in section 5.7.2.1:

Finding: KM tools are not utilized to their fullest extent to ensure successful capture and transfer of knowledge to staff. Survey results for each of the key agency functions show that approximately half of surveyed staff and management usually use knowledge management resources and processes (internal wiki site, videos, publications, etc.) to capture best practices. Knowledge management will influence agency performance over the next 5 years, given that approximately 26 percent of the NRCs workforce is currently eligible to retire and approximately 44 percent will be eligible to retire within the next 5 years. High attrition over the next 5 years could negatively impact some positions identified in this assessment and will leave a critical knowledge gap.

Mitigating Strategy: The NRC should evaluate the NRCs KM program to better align the efforts with expected outcomes. The evaluation should explore ways to elevate the priority and urgency of capturing critical knowledge and best practices. Attention should be focused on the positions with highest projected attrition as identified through the NRCs SWP process (ML17109A315). The evaluation should consider methods to increase KM engagement with the NRCs senior level staff. The evaluation should include a cost-effectiveness analysis to better understand the cost compared to the expected outcomes. To measure effectiveness, performance indicators should be established as a result of the evaluation. In addition, usage data for Nuclepedia should be thoroughly tracked and analyzed to find how to maximize the usefulness of this resource for the NRC.

As a result of this finding, and as specified in the mitigating strategy, the NRC is conducting an evaluation of the KM program.

Evaluation Objective

The objective is to evaluate whether the NRCs approach to capturing and transferring knowledge is effective in meeting its intended goals and whether it is being implemented efficiently.

Evaluation Status

In October 2022, the NRC evaluation team, together with its contractor consultant PRE, held a kick-off meeting with internal stakeholders to begin Phase 1 of the KM evaluation. At this meeting, the leads for the KM program provided an overview of the KM activities conducted throughout the agency, the tools in place for capturing and sharing agency knowledge, and the overall program objective.

Following the kick-off meeting, informational interviews were scheduled with key stakeholders with critical roles in the process from initial development to implementation and analysis, and background documentation was reviewed.

The informational interviews and background documentation review provided the basis for the development of a logic model, which will be used to design the evaluation in Phase 2 of the evaluation process.

The Evidence Building Activities page on the NRC public website will have additional information on this evaluation as it progresses through the next three phases.

10 Evaluation of the NRCs Licensing Actions

FY 2022 Evidence-Building Plan Priority Question: To what extent are licensing actions performed by the NRC becoming more or less resource intensive over time and have there been any changes in work product quality?

Summary

The NRCs regulatory process includes five main components: (1) developing regulations and guidance for applicants and licensees, (2) licensing or certifying applicants to use nuclear materials or operate nuclear facilities until license termination, (3) overseeing licensee operations and facilities to ensure that licensees comply with safety requirements, (4) evaluating operational experience at licensed facilities or at locations where licensed activities are performed, and (5) conducting research, holding hearings at the request of parties that may be affected by agency decisions, and obtaining independent reviews to support the agencys regulatory decisions. To receive a license or certification, or to amend, renew, or transfer an existing license, an entity or individual, must submit an application to the NRC. The NRC reviews applications to ensure that the application meets the relevant regulatory requirements and that the proposed activities will be conducted safely and in accordance with the common defense and security. License reviews use evidence, such as analyses, to support decisions that ensure the NRC is accomplishing its mission.

The NRC will perform an evaluation that analyzes licensing actions associated with licensing programs for which the agency has developed generic milestone schedules, as required by the Nuclear Energy Innovation and Modernization Act. The evaluation will determine if licensing actions performed by the NRC are becoming more or less resource intensive over time and whether there have been any changes in work product quality. This evaluation will provide a better understanding of the effectiveness of the licensing programs based on expended resources and quality of the work products for similar licensing actions. The evaluation may provide key insights to further risk inform the agencys licensing programs.

Evaluation Objective

The objective is to ensure that the NRCs licensing review and certification process is data driven, evidence based, applies a risk informed approach, and reflects an appropriate and reasonable expenditure of resources to complete, based on the requested activity.

Evaluation Status

The NRC evaluation team, partnered with evaluation contractors PRE, plan to initiate this evaluation in Quarter 4 of FY 2023.

11

NRC FORM 335 U.S. NUCLEAR REGULATORY COMMISSION 1. REPORT NUMBER (12-2010) (Assigned by NRC, Add Vol., Supp., Rev.,

NRCMD 3.7 and Addendum Numbers, if any.)

BIBLIOGRAPHIC DATA SHEET NUREG 2250, Volume 2

(See instructions on the reverse)

2. TITLE AND SUBTITLE 3. DATE REPORT PUBLISHED Annual Evaluation Plan Fiscal Year 2024 MONTH YEAR March 2 02 3

4. FIN OR GRANT NUMBER N/A

5. AUTHOR(S) 6. TYPE OF REPORT Angela Randall, Matthew Meyer, et. al. Evaluation Plan

7. PERIOD COVERED (Inclusive Dates)

Annual

8. PERFORMING ORGANIZATION - NAME AND ADDRESS (If NRC, provide Division, Office or Region, U. S. Nuclear Regulatory Commission, and mailing address; if contractor, provide name and mailing address.)

Office of the Executive Director for Operations U.S. Nuclear Regulatory Commission Washington, DC 20555-001

9. SPONSORING ORGANIZATION - NAME AND ADDRESS (If NRC, type "Same as above", if contractor, provide NRC Division, Office or Region, U. S. Nuclear Regulatory Commission, and mailing address.)

Same as above

10. SUPPLEMENTARY NOTES

11. ABSTRACT (200 words or less)

The Annual Evaluation Plan provides summary information on evaluations being initiated in fiscal year 2024. The Evidence Act defines a n evaluation as a n assessment us ing systematic d ata collection and analysis of one or m ore pr ograms, policies, a nd organizations i ntended to assess t heir e ffectiveness a nd efficiency. The e valuations be ing conducted will a ssist in answering priority questions established in the Eviden ce-B uilding Plan or other evaluations determined to be significan t, su ch as th ose req uired by statute or those of high value to the agency. This ev aluation p lan contains two evaluations to be initiated in FY 2024.

12. KEY WORDS/DESCRIPTORS (List words or phrases that will assist researchers in locating the report.) 13. AVAILABILITY STATEMENT unlimited

14. SECURITY CLASSIFICATION (This Page) unclassified

(This Report) unclassified

15. NUMBER OF PAGES

16. PRICE

NRC FORM 335 (12 - 2010)

U.S. Nuclear Regulatory Commission NUREG-2250, Vol. 2 March 2023

www.nrc.gov STAY CONNECTED