ML25077A175

Consolidated Federal Agency Staff Responses to Advance Act Survey

ML25077A175
Person / Time
Issue date:	03/17/2025
From:	Daniel Merzke, Lauren Nist NRC/NRR/DRO/IRAB
To:
References
Download: ML25077A175 (17)

Text

Federal Agency Staff Responses to ADVANCE Act Section 507 Survey For other federal agencies: The Advance Act requires the NRC to report to Congress what actions the NRC may implement to maximize the efficiency of such programs through, where appropriate, the use of risk-informed, performance-based procedures, expanded incorporation of information technologies, and staff training. Your agency has an oversight and inspection role.

Staff Responses from the Federal Emergency Management Agency

1. Please briefly describe your oversight and/or inspection program.

For the purpose of the provided responses, FEMA is defining inspection/oversight as its review and approval of plans/procedures, evaluation of exercises and drills, staff assistance visits, technical assistance, Annual Letter of Certification (ALC) and Preliminary Capability Assessments/Disaster Initiated Reviews. Details of the FEMA inspection/oversight program are found in the 2023 Radiological Emergency Preparedness (REP) Program Manual (RPM).

To ensure their radiological emergency plans and preparedness are adequate to protect public health and safety in the event of a radiological incident, FEMA works with OROs to assess their capability to implement various aspects of their emergency plans. FEMA uses the RPM to support their review of ORO radiological emergency plans, assess offsite preparedness, and provide technical assistance to OROs, including providing clarification on planning guidance found within the RPM. FEMA uses the RPM in preparing, conducting, evaluating, and/or assessing REP activities.

2. How do you routinely assess your oversight program?

The 2023 RPM provides details on routine assessments. Exhibit III-1 on pages 205-208 provides details on objectives and capability targets assessed by FEMA, including frequency of those assessments. In addition, other activities assessed by FEMA or through reporting in the ALC are identified in Exhibits IV-2 through IV-4 pages 282 - 284. Exercise planning milestones are identified in Exhibit IV-1 on pages 280-281. All assessment activities are captured in a Biennial Preparedness Report FEMA. This report consists of a summary of FEMAs reasonable assurance determination in terms of the core capabilities assessed through evaluations of the objectives/capability targets throughout the biennial assessment period. The results of all the Evaluation Reports, as well as the work plan and other supporting documentation produced during the biennial assessment period are appended to, or referenced in, the Biennial Preparedness Report.

3. Do you take into account past licensee/contractor/ grantee performance in your oversight program (e.g. a graded approach to oversight)? If so, how is this done?

Exercises and drills include improvement and corrective action components in the after-action reports. These components primarily provide offsite response organizations with a roadmap to improve performance and correct deficiencies, but are also used to maintain FEMAs focus on areas where additional technical assistance, training, or other support may be required as well as to identify where continuing further oversight is required. FEMAs programmatic methodology does allow offsite response organizations to receive credit for actual incident responses (to radiological and non-radiological incidents), other non-program exercises and assessments, and like activity where the alignment to or correspondence with REPP requirements can be documented. In these cases, FEMA may elect not to separately evaluate all or a portion of a given capability if the credited activity adequately demonstrates it.

4. To what extent do you use information technologies or other technologies (other than in-person observation) to perform oversight and inspection activities (e.g.,

do you perform oversight or inspection activities remotely using electronic reading rooms, or review livestream videos, etc.), including planning and preparing for these activities, and how do you choose what activities or events are inspected or overseen using these methods?

FEMA performs some preparatory activities using virtual platform meetings, and document review activities may be conducted asynchronously/remotely. Evaluation activities such as functional and full-scale exercises are conducted on site and in person. FEMA did use remote observation of exercises and drills for assessments during the COVID-19 pandemic; these methodologies are considered sub-optimal and are no longer in place, with remote meetings currently being used for exercise planning and coordination only where deemed efficient on a case-by-case basis. Post-disaster, Preliminary Capability Assessments (PCA) are generally conducted through virtual meetings; Disaster Initiated Reviews (DIR) are generally conducted in person and on site but may include the use of virtual meetings to discuss or clarify information. DIRs may include remote sensing where FEMA REP can include reasonable assurance validation requirements in agency/partner remote sensing mission sets or draw information from agency/partner remote sensing data for disaster-affected areas.

5. Is your inspection or oversight program updated over time using any risk-informed or performance-based assessments or information?

FEMA REPP has implemented a data analytics program to identify trends in exercise and drill results to include areas where consistent findings occur across regions or across the national program. At this point results have been used primarily to focus technical assistance and training; modification of assessment activities may be an outcome as the data analytics effort matures and collects sufficient data over time.

6. Are travel resources ever a constraint for oversight/inspection activities, and if so, how do you manage or prioritize use of the travel resources for oversight/inspection activities?

FEMA REP is authorized to bill user fee payers (nuclear power plant licensees) in advance for 100% of estimated expenses, including estimated travel costs to perform the missions required by law, regulation, or agency MOU. Fees collected may not be used for other than program purposes. In recent (2024) disasters, REP essential activities (PCA/DIR, exercises, training) have been considered mission essential activities and have not been constrained by agency policies restricting non-disaster travel.

7. Does your agency/program include an assessment process to identify areas of duplication or unnecessary activities, or areas for improving efficiency? If so, what criteria are used to determine what changes will be implemented?

As mandated in FEMAs annual appropriation, the REP Program is required to collect at least 100% of its anticipated operational budget. The Program operates from the assessment and collection of user fees. These user fees do not expire and, therefore, are available until expended. The Program established a $2 million operational reserve to augment costs associated with response and recovery activities performed by REPP-funded employees during a real-world, catastrophic accident at a commercial nuclear power plant in the United States. FEMAs cost model utilizes historical cost data to inform future costs and identify opportunities for efficiencies.

FEMA REPP data analytics efforts have as a primary goal the building and analysis of expenditure data sets that can be used to identify areas for efficiency and highlight areas of inconsistency between like program activities for investigation and adjustment.

Consistency and efficiency targets and metrics are included in supervisory personnel performance management goals where appropriate.

Economic impact significance determination for proposed regulations is governed by a separate process at the Agency level and is not under program control.

8. Does your agency/program modify or adjust its inspection program (e.g., type of inspection done, frequency of an inspection, resources used for inspection) based on the safety significance of the inspection area or an issue, and if so, how?

In addition to the notes on question #3, a Level 1 finding (defined as An observed or identified inadequacy of organizational performance during an assessment activity that could cause a determination that offsite emergency preparedness is not adequate to provide reasonable assurance that appropriate protective measures can be taken in the event of a radiological emergency to protect the health and safety of the public living in the vicinity of an NPP) will require additional assessment activity, with associated resource commitments, to re-evaluate and ensure correction. Activities associated with Level 1 findings may also include custom training design and delivery, again with additional resource requirements. Level 1 finding documentation, communication, and correction follow specific timelines and procedures; see the 2023 RPM Part III: REP Program Assessment Policies and Guidance for further information.

A disaster impacting a REPP community may require a Preliminary Capability Assessment (PCA) and, if deemed warranted, a deeper Disaster Initiated Review (DIR) to determine if a community adjacent to a commercial nuclear power plant still has the capability to respond to a potential radiological incident. A PCA or DIR is, by definition, an unscheduled assessment conducted in challenging conditions which will require the immediate virtual or physical deployment of staff resources to perform the assessments (with physical deployment probable for DIR) as well as tasking of agency remote sensing and other resources where available in the context of broader agency response to disasters. See the April 2024 Standard Operating Guide:

Assessment of Offsite Emergency Preparedness Infrastructure and Capabilities Following an Incident in the Vicinity of a U.S. Nuclear Regulatory Commission Licensed Nuclear Power Plant for further information on the PCA/DIR process and scope.

9. Does your agency use AI or machine learning to risk-inform oversight and inspection decisions or outcomes?

FEMA does not currently use AI/machine learning for these purposes but is investigating its use.

10. Is your agency/program preparing to regulate or oversee new technologies (including digital technologies) for use by industry? If so, how are you doing that to ensure your agencys/programs decisions are made in time for industry to determine whether to adopt the new technologies?

If this question refers to SMRs/other reactor technologies, FEMA does not have a role in any licensing that does not include an off-site EPZ. Facilities licensed by NRC with an off-site EPZ will be managed within the current framework. If this question refers to new technologies within the existing scope of practice (e.g. use of cellular mobile alerting versus or in addition to traditional sirens for alert and notification, or the use of digital technologies in land mobile radios used for emergency worker communications),

FEMAs evaluation structure is based on capabilities (as described in the applicable planning standards) and the offsite response organizations ability to effectively execute their plans to implement those capabilities; this is largely technology-agnostic. FEMAs staffing model for REP does include use of employee and contract SMEs to remain abreast of developments in applicable emergency management doctrine and technology.

11. Does your organization use risk-insights to resolve differences of opinion or conflict? If so, how?

FEMA does not use a risk insights tool in difference resolution. For How does your organization resolve differences of opinion or conflict see #12.

12. How does your organization ensure that differing views on organizational decisions are resolved promptly, without compromising safety or causing unnecessary delays to project timelines?

While FEMA does not have a formal Differing Views process, the REPP internal programmatic structure is designed to promote frequent interaction at staff and leadership levels and a culture where open discussion is encouraged and welcomed.

Specific avenues include quarterly issue-focused meetings between region and headquarters leadership; active peer-to-peer relationships across communities of practice (health physics, training, policy, et al.); frequent 1-on-1 and group opportunities for staff to interact with leadership at multiple levels; like opportunities for leaders to interact and discuss issues on a peer-to-peer basis; and a document review process that includes those with equity early in the document process. FEMA REPP also has an embedded attorney at the program level to ensure that legal sufficiency and scope reviews and discussions can occur early in decision processes.

13. How does your organization address and resolve mission-related issues that are not considered risk or safety significant but are still important to the staff?

See #12. We do not currently differentiate issues on this basis.

14. Do you have any other suggestions on how the NRC can update our Differing Views Processes to ensure that any impacts on agency decisions and schedules are commensurate with the safety significance of the differing opinion?

FEMA currently has limited familiarity with NRCs Differing Views processes and has no input at this time. FEMA REPP is interested in the NRC Differing Views processes and would like to learn more to determine if the processes could be applied in the FEMA REPP paradigm. Should there be any comment once there is an opportunity to review NRCs processes, we will reach out.

Staff Responses from the Federal Aviation Administration The Advance Act requires the NRC to report to Congress what actions the NRC may implement to maximize the efficiency of such programs through, where appropriate, the use of risk-informed, performance-based procedures, expanded incorporation of information technologies, and staff training. Your agency has an oversight and inspection role. Can we discuss the following areas / questions:

1. Please briefly describe the FAAs aviation safety oversight program for entities under the purview of the Flight Standards Service.

The FAAs authority for oversight of civil aviation in the U.S. is codified in Chapter 49 of the U.S. Code. From that authority, FAA licenses (more accurately, certificates) entities to operate in the National Airspace System. The FAA issues certificates to airmen (pilots, mechanics, repairmen, etc.) and to entities (airlines, manufacturers, maintenance facilities, etc.). The FAA uses risk-based systems to inform our staff (including but not limited to our field inspectors (aviation safety inspectors and aviation safety engineers) on oversight activities.

The FAA uses an oversight tool called the Safety Assurance System (SAS). SAS is used to perform certification, surveillance, and Continued Operational Safety (COS). SAS was also developed to satisfy the Safety Assurance component of the FAA. SAS is not a separate safety standard. SAS includes policy, processes, and associated software the FAA Flight Standards Service (FS) uses to capture data when conducting oversight. SAS helps the FAA with the following functions:

Standardizes the work being accomplished across FS, improves consistency and collaboration between FAA and industry, helps FAA Aviation Safety Inspectors (ASIs) determine risk-based, data-supported oversight decisions, helps determine hazard identification and risk assessment strategies to formulate surveillance plans and where to focus FAA resources, and provides the standardized protocols to evaluate whether Certificate Holder operations are in compliance with regulations.

2. How do you routinely assess your oversight program?

Within the Flight Standards Service, we have a division (the Safety Analysis and Promotion Division, or AFS-900) within the Office of Safety Standards that marintains our oversight database, the Safety Assurance System (SAS), on a continuous basis. SAS provides a structured means of SRM to ensure key decision making, increased confidence in risk controls through structured SA process, and a safety promotion framework to support sound safety culture.

The Principal Inspector (PI) also has the capability to identify areas of risk they want observed by any aviation safety inspector (ASI) to conduct enhanced surveillance, en route inspections, or ramp inspections on a certificate holder.

Annual and quarterly planning meetings are conducted with PI and office personnel to review and evaluate the certificate holders system and operating environment for risks, including the results from coordinated surveillance.

3. Do you take into account past licensee [certificate holder] performance in your oversight program? If so, how is this done?

Yes. Certificate holder performance is considered when weighing future oversight tasks.

Surveillance priority, criticality, and cadence can be individually adjusted based on prior performance of a certificate holder. Additionally, we have tools that assist us in assessing the operators risk that takes into account recent performance and activities that can adjust the oversight program.

The FAAs Certificate Holders Evaluation Program (CHEP) allows for an in-depth look at the certificate holders system and has three primary goals:

1. Verify the certificate holder complies with applicable regulations;

2. Evaluate whether the certificate holder is effectively managing safety; and

3. Identify hazards, assess risk, and provide documentation for the Certificate Management Team (CMT) to mitigate associated risks.

A CHEP is launched when there are specific conditions present to trigger this process such as substantial changes, labor disputes, or rapid expansion or growth.

(8900.1V10C8S1)

Also, see response to question number 6.

4. To what extent do you use information technologies or other technologies (other than in-person observation) to perform oversight and inspection activities (e.g., do you perform oversight or inspection activities remotely using electronic reading rooms, or review livestream videos, etc.), including planning and preparing for these activities, and how do you choose what activities or events are inspected or overseen using these methods?

Flight Standards system for oversight of certificate holders is heavily augmented by information technologies.

The onset of the pandemic accelerated the need for a remote technology (RT) standard.

Stakeholders who wish to use RT in collaboration with FS must provide their own RT equipment and trained personnel familiar with its use. RT data may be live streamed or recorded. Data recorded by the stakeholder remains the property of the stakeholder and may be viewed by FS via a platform provided by the stakeholder but not retained by FS.

Government-furnished equipment (GFE) must be used by FS employees to view live streaming video or recordings, unless the RT equipment is provided by the stakeholder. FS will not record any stakeholder activities with GFE, nor will the FS employee(s) retain any stakeholder recordings. (8900.1V1C3S9)

5. Is your inspection or oversight program updated over time using any risk-informed or performance-based assessments or information?

Yes, using the FAA Safety Assurance System (SAS) Flight Standards (FS) work can be planned, tracked, and resourced based on the principles of data-supported risk-based decision making and risk-based resource targeting (RBRT). The RBRT process employs a standardized, data-supported, risk-based methodology to assist in establishing work priorities and allocating resources. The objective of SAS is to transform FS and the aviation industry to a national standard of system safety based on SMS principles. Safety Analysis &

Promotion Division l Federal Aviation Administration (faa.gov) under SASO history.

A Risk Management Process (RMP) is a process used when the certificate holder knowingly or unknowingly accepts, or generates, an undesirable level of risk, and the PI takes action to ensure that the certificate holder is effectively managing these risks in their operation. An RMP is tracked within SAS.

6. Are travel resources ever a constraint for oversight/inspection activities, and if so, how do you manage or prioritize use of the travel resources for oversight/inspection activities?

Yes, travel budgets are a challenge. Supervisors at the Flight Standards Service work against limited budgeted funds for employee travel. Generally, travel for oversight is weighted for approval before travel for training and conference attendance.

7. Does your agency/program include an assessment process to identify areas of duplication or unnecessary activities, or areas for improving efficiency? If so, what criteria are used to determine what changes will be implemented?

Yes, please see our description of SAS within question number 1.

In addition, Post-Implementation Performance Reviews (PIR) are performed after SAS automation tools' deployment. PIRs contain four broad categories (1) Business Results; (2)

Performance; (3) Strategic Initiatives and Service Objectives, and (4) Benefits.

Further, the Safety Analysis and Promotion Division (AFS-900) continues to enhance the accuracy of the SAS Risk Profile over time using currently implemented periodic reviews of the risk model. When this occurs, the metrics within SAS, and the priority/resource order of assessments will change.

8. Does your agency/program modify or adjust its inspection program (e.g., type of inspection done, frequency of an inspection, resources used for inspection) based on the safety significance of the inspection area or an issue, and if so, how?

Yes, please see our description of SAS within question number 1.

When a new hazard is identified, PIs may request national level hazard analysis. We define a new hazard as one where current directives do not adequately control the associated risk, or risk controls do not exist to effectively mitigate risk, such as new or emerging technology that did not previously exist in the NAS. These processes enable a Principal Inspector (PI) to request national-level support to address a safety issue.

9. Does your agency use AI or machine learning to risk-inform oversight and inspection decisions or outcomes?

Our SAS tool incorporates limited AI capability. The Risk Profile Assessment Tool (RPAT) in SAS, identifies safety related issues for the Principal Inspector (PI) and Certificate Management Team (CMT) of their respectively assigned certificates. In addition, Office Managers (OM) can evaluate the RPAT scores for the certificates assigned to their office so they may make risk-based resource allocation decisions based on the RPAT. Further, information outside of the SAS automation, such as the Interim Certificate Priority Index (ICPI) and Safety Performance Analysis System (SPAS) profiles, can be obtained through SAS Resources function. This outside information is designed to be used to give CMTs more information as needed to augment their decisions.

To be more descriptive, the RPAT CHI model applies a holistic approach in calculating its component value whereby underlying risk factors are measured and valued in each component, and those values are added together to arrive at the overall component level value.

Recently, the FAA Roadmap for Artificial Intelligence Safety Assurance roadmap was recently published. Within this roadmap, the FAA charts a path forward to incorporate AI into FAA functions with the focus on how AI will improve safety.

10. Is your agency/program preparing to regulate or oversee new technologies (including digital technologies) for use by industry? If so, how are you doing that to ensure your agencys/programs decisions are made in time for industry to determine whether to adopt the new technologies?

The FAA adapted to the influx of new aircraft by creating organizational groups intended to facilitate the integration of new aircraft. One example is the Office of Unmanned Aircraft Systems Integration, under the Associate Administrator for Aviation Safety. The FAA has devoted extensive resources to anticipate new technologies, but we have to start with rulemaking, which takes approximately 5 years. In general, we are unable to justify (including but not limited to cost/benefit analysis) new rules unless the technology exists and the aircraft or device can be analyzed and its activity accurately predicted.

Another example is Automation. Several companies, especially in the area of Advanced Air Mobility, have expressed interest in developing new aircraft with significantly higher levels of automation. In response, the FAA established an Agency-wide Autonomy Working Group to collaborate with industry stakeholders on how to safely integrate these technologies into the National Airspace System.

11. Does your organization use risk-insights to resolve differences of opinion or conflict? If so, how?

The Risk Profile Assessment Tool (RPAT) in SAS, identifies safety related issues for the Principal Inspector (PI) and Certificate Management Team (CMT) of their respective assigned certificates. In addition, Office Managers (OM) can evaluate the RPAT scores for the certificates assigned to their office so they may make risk-based resource allocation decisions based on the RPAT. Further, information outside of the SAS automation, such as the Interim Certificate Priority Index (ICPI) and Safety Performance Analysis System (SPAS) profiles, can be obtained through SAS Resources function. This outside information is designed to be used to give CMTs more information as needed to augment their decisions.

Furthermore, the SAS RPAT currently provides a high resolution of the risk related data at the Certificate Holder (CH) level. The RPAT analyzes multiple data points over a broad spectrum captured under the Components of Safety Performance History, Observed Risk, Organizational/Operational Factors, Uncertainty, and Flight Exposure multiplier. This high resolution of actual risk data points can be scoped and increased in resolution to enable the Certificate Management Team (CMT) to make effective risk-based decisions. Therefore, SAS has the capability to readily identify safety related trends across the National Aerospace System (NAS).

12. How does your organization ensure that differing views on organizational decisions are resolved promptly, without compromising safety or causing unnecessary delays to project timelines?

Our FAA Order 8900.1, Flight Standards Management System (FSMS), directs the activities of aviation safety inspectors (ASI) responsible for the certification, technical administration, and surveillance of air carriers, certain other air operators conducting operations in accordance with the appropriate part of Title 14 of the Code of Federal Regulations (14 CFR), certificated airmen, and other aviation activities. This order also provides direction for tasks related to aircraft accidents and incidents, investigations and compliance, the Aviation Safety Action Program (ASAP), administrative areas, and miscellaneous tasks not related to a specific regulation.

There is a provision contained within FSMS to apply for a deviation that do not apply to a specific case. (8900.1V1C1S1)

In the event of confection with FAA orders or directives, the order/directive with the most recent date is used. (8900.1V1C1S1)

The Safety Analysis and Promotion Division (AFS-900) developed an automated analytical model known as the Interim Certificate Holder Priority Index (ICPI). The ICPI supplements SAS data and is another tool used to compare risk with SAS. The ICPI is designed to identify certificate holders who have an increased risk of a failure that could lead to an accident. The ICPI evaluates certificate holder (CH) safety performance levels and risk factors and generates a numerical index that can be used to evaluate, analyze, compare and prioritize 14 CFR parts 121, 135, and 145 CHs for oversight planning and resource allocation purposes.

13. How does your organization address and resolve mission-related issues that are not considered risk or safety significant but are still important to the staff?

Currently, SAS can identify risk-based surveillance priority at the PI, Office, and national levels via SAS Standard Reports across peer groups and CFR parts, at both the certificate and assessment levels. Several factors can be assessed within SAS including Resources Not Available (RNA) captured as Deferred Surveillance in the Uncertainty Component, Safety Performance history (accidents, incidents, enforcements, etc.), previous and current assessment results and emerging risk from the SAS Certificate Holder Assessment Tool (CHAT) are data points used in the RPAT to assess the relative health of peer groups and CHs.

14. Do you have any other suggestions on how the NRC can update our Differing Views Processes to ensure that any impacts on agency decisions and schedules are commensurate with the safety significance of the differing opinion?

Without a comprehensive understanding of the NRCs processes, this is a difficult question to answer. Can the NRC reciprocate answers with these questions?

Staff Responses from the Defense Nuclear Facilities Safety Board The Advance Act requires the NRC to report to Congress what actions the NRC may implement to maximize the efficiency of such programs through, where appropriate, the use of risk-informed, performance-based procedures, expanded incorporation of information technologies, and staff training. Your agency has an oversight and inspection role. Can we discuss the following areas / questions:

1. Please briefly describe the DNFSB oversight program for Defense Nuclear Facilities.

The DNFSB is an independent organization within the executive branch of the United States Government, chartered with the responsibility of providing recommendations and advice to the President and the Secretary of Energy regarding public health and safety issues at Department of Energy defense nuclear facilities. The Board reviews and evaluates the content and implementation of the standards relating to the design, construction, operation, and decommissioning of defense nuclear facilities of the Department of Energy (including all applicable Department of Energy orders, regulations, and requirements) at each Department of Energy defense nuclear facility.

There are approximately 120 people in the agency that include 14 resident inspectors at 7 facilities and about 65 technical staff. The technical staff supports the Board by providing expertise in the fields relating to the design, construction, operation, and eventual decommissioning of defense nuclear facilities.

The resident inspectors are tasked with providing information gathered through field oversight of select Department of Energy (DOE) defense nuclear facilities. Issues are frequently resolved by providing informal and actionable feedback to site personnel to improve nuclear safety. Resident inspectors also perform coordinated focus area reviews to monitor safety conditions within the defense nuclear complex, which are briefed to the Board. The resident inspectors share both improvement opportunities and good practices with DOE field leadership to resolve problems and promulgate ideas to foster overall safety improvement. Resident Inspectors produce a Weekly Report that summarizes the pertinent activities and events that occurred during that week at their Defense Nuclear Facility.

2. How do you routinely assess your oversight program?

Yes, as part of the internal controls program.

3. Do you take into account past licensee [certificate holder] performance in your oversight program? If so, how is this done?

Yes, in the development of the annual oversight plan.

4. To what extent do you use information technologies or other technologies (other than in-person observation) to perform oversight and inspection activities (e.g., do you perform oversight or inspection activities remotely using electronic reading rooms, or review livestream videos, etc.), including planning and preparing for these activities, and how do you choose what activities or events are inspected or overseen using these methods?

No, all of the oversight and inspection is performed on site due to security requirements.

5. Is your inspection or oversight program updated over time using any risk-informed or performance-based assessments or information?

There is an annual oversight plan developed by the Technical Director at the beginning of the annual planning cycle.

The Technical Director issues a memorandum that provides guidance that the technical staff should use when preparing the draft annual work plan for the upcoming year. While the oversight plan owner determines the approach used to gather input, a suggested method is to facilitate a planning meeting that includes key individuals responsible for coordinating and accomplishing the oversight defined in the oversight plan. In most cases, this will include the cognizant associate technical director, the plan owner, and applicable resident inspectors, site cognizant engineers and topical cognizant engineers. For programmatic review oversight plans, appropriate subject matter experts should be involved in the planning meeting. Regardless of approach, the oversight plan owner should:

(a) Apply the priorities and strategic direction communicated by the technical director.

(b) Ensure that oversight is comprehensive and provides adequate coverage across the defined scope of the oversight plan and supports accomplishment of the performance goals contained in the Boards annual performance plan.

(c) Avoid prematurely narrowing the scope of oversight plans to preclude new or infrequently reviewed programs that merit consideration for oversight attention.

(d) Look at all defense nuclear sites, facilities, and programs that fall within the Boards jurisdiction and are within the scope of the Boards strategic plan elements covered by the oversight plan. Plans should be constrained based on the guidance provided by the technical directors work/oversight plan guidance memorandum and priorities and strategic direction.

(e) Specifically assess cases where a facility or program has never been reviewed by the staff, or where it has been longer than five years since the staff has reviewed

6. Are travel resources ever a constraint for oversight/inspection activities, and if so, how do you manage or prioritize use of the travel resources for oversight/inspection activities?

There is enough funding to complete the oversight plan, but sometimes travel resources are limited for non-priority travel such as meetings and conferences.

7. Does your agency/program include an assessment process to identify areas of duplication or unnecessary activities, or areas for improving efficiency? If so, what criteria are used to determine what changes will be implemented?

The annual oversight plan takes care of this.

8. Does your agency/program modify or adjust its inspection program (e.g., type of inspection done, frequency of an inspection, resources used for inspection) based on the safety significance of the inspection area or an issue, and if so, how?

The oversight plan takes this into account.

9. Does your agency use AI or machine learning to risk-inform oversight and inspection decisions or outcomes?

No.

10. Is your agency/program preparing to regulate or oversee new technologies (including digital technologies) for use by industry? If so, how are you doing that to ensure your agencys/programs decisions are made in time for industry to determine whether to adopt the new technologies?

The annual oversight plan would take this into account.

Staff Responses from the Department of Energy, Office of Enterprise Assessment Douglas Bollock and Aixa Belen interviewed Kevin Kilp, Director of the Office of Environment, Safety and Health Assessments and Joseph Demers, Nuclear Safety Enforcement Officer.

1. Please briefly describe your oversight and/or inspection program.

Enforcement Office - The Enforcement Office is divided into 3 Offices. Office of Nuclear Safety Enforcement, Office of Worker Safety & Health Enforcement, and the Office of Security Enforcement. The Office of Nuclear Safety investigates the areas of nuclear safety, quality assurance, and radiation protection and currently have four employees.

All the Enforcement Offices evaluates events reportable to DOE and evaluate which events require further investigations. The Office of Enforcement perform fact findings and/or investigations. The results of the investigations are Notice of Violation, Consent Order/Settlement Agreement, Compliance Order, Special Report Order, Enforcement Letter and/or Advisory Note. The Notice of Violations and Orders include either civil penalties or reduction of contract budget depending on the severity of the violations. There are no metrics for the number of investigations to perform per year. It all depends on the significance of the events reported. This process is reactive.

Enforcement Process Overview - January 2021_0.pdf Environment, Safety and Health Assessment Office - This Office focuses on the performance and continuous improvement of the DOE sites. The reports provide unbiased recommendations in any area under DOE oversight. The assessments provide recommendations on how to improve performance and a summary of the overall health of the program assessed.

The Environment, Safety and Health Assessment Offices includes 4 offices: Office of Nuclear Safety & Environmental Assessments, Office of Worker Safety & Health Assessments, Office of Emergency Management Assessments and the Office of Nuclear Engineering & Safety Basis Assessments. In these Offices, the employees are assigned to specific site. These offices are supported by contractors (20-30 subject matter experts).

They performed around 40 targeted assessments a year. They use the resources to the best of their ability.

2. How do you routinely assess your oversight program?

The Assessment group perform biannual discussions with internal and stakeholders to determine what should be assess and what can be improved. The DOE sites offices provide input of what should be assess in the next year. Assessments planning is performed per request or are directly assigned from Sr Management.

The Office of Enterprise Assessment cannot direct the site to do anything. They must work through the Site office to do the assessments.

Enforcement assesses, track and trend occurrences data to determine what drives them and where are the weaknesses. It is based on past performance and the process may trigger a fact finding and/or investigations. Most of the fact findings goes back to investigation.

3. Do you consider past licensee/contractor/ grantee performance in your oversight program (e.g. a graded approach to oversight)? If so, how is this done?

Yes. Both offices evaluate events reported, DFNB findings, GAO findings, IG, Site offices findings and previous enforcement results when making the decisions to perform an investigations or assessment. The issue of overlap is important with other external stakeholders and coordination among all are considered in the planning.

Site leads are the eyes and ears to operational awareness and maintain a one pager with all the information about the site. They perform briefings 3 times a year per site.

4. To what extent do you use information technologies or other technologies (other than in-person observation) to perform oversight and inspection activities (e.g., do you perform oversight or inspection activities remotely using electronic reading rooms, or review livestream videos, etc.), including planning and preparing for these activities, and how do you choose what activities or events are inspected or overseen using these methods?

In general, investigations are onsite to interphase directly in people. Leverage of technology is used in the preparation phase. Interviews are performed for 4 days. Physical arrangement is very enlightening and important to determine the health of the program being evaluated.

In-person provides some advantages to gather information from employees without the screen of the management.

Not using AI or anything outside the box.

5. Is your inspection or oversight program updated over time using any risk-informed or performance-based assessments or information?

None of the Offices incorporate a specific Risk-inform analysis. They gathered information from different stakeholders. Program offices and the sites have different approaches for risk informed.

6. Are travel resources ever a constraint for oversight/inspection activities, and if so, how do you manage or prioritize use of the travel resources for oversight/inspection activities?

No. Enforcement and Assessment work is considered mission related. Consequently, they have no travel constraints.

7. Does your agency/program include an assessment process to identify areas of duplication or unnecessary activities, or areas for improving efficiency? If so, what criteria are used to determine what changes will be implemented?

See question #3.

8. Does your agency/program modify or adjust its inspection program (e.g., type of inspection done, frequency of an inspection, resources used for inspection) based on the safety significance of the inspection area or an issue, and if so, how?

Enforcement is reactive to the events happening. Assessment plans and coordinate with other offices.

Highly technical inspections with larger teams are conducted less frequently (every 4-5 years) are made up of higher paid specialists (EN 4/5) from a specific group within the oversight organization.

9. Does your agency use AI or machine learning to risk-inform oversight and inspection decisions or outcomes?

No

10. Is your agency/program preparing to regulate or oversee new technologies (including digital technologies) for use by industry? If so, how are you doing that to ensure your agencys/programs decisions are made in time for industry to determine whether to adopt the new technologies?

No information

11. Does your organization use risk-insights to resolve differences of opinion or conflict? If so, how?

DOE has a DPO program. None of the offices had a need to use it because their documents are focused on facts and not opinion. There has been a different of opinion for a technical issue.

12. How does your organization ensure that differing views on organizational decisions are resolved promptly, without compromising safety or causing unnecessary delays to project timelines?

13. How does your organization address and resolve mission-related issues that are not considered risk or safety significant but are still important to the staff?

14. Do you have any other suggestions on how the NRC can update our Differing Views Processes to ensure that any impacts on agency decisions and schedules are commensurate with the safety significance of the differing opinion?