ML24257A090
| ML24257A090 | |
| Person / Time | |
|---|---|
| Issue date: | 09/17/2024 |
| From: | Dennis M, Pensado O Office of Nuclear Regulatory Research, Southwest Research Institute |
| To: | |
| References | |
| Download: ML24257A090 (1) | |
Text
Data Science and AI Regulatory Applications Public Workshop
AI Regulatory Gap Analysis Update September 17, 2024
Matt Dennis U.S. Nuclear Regulatory Commission Office of Nuclear Regulatory Research Dr. Osvaldo Pensado Staff Scientist Southwest Research Institute NRC AI Strategic Plan
- NUREG-2261 issued May 2023
- AI Project Plan (AIPP) issued September 2023
- The AI Project Plan describes how the agency will execute the five strategic goals from the AI Strategic Plan
- Provides estimated timelines for various task completions within each Strategic Goal
- Communicates NRC priorities to internal and external stakeholders
- AI research activities
- AI safety and security test, evaluation, verification and validation (AIPP, Task 1.3)
- US-Canada-UK Trilateral Collaboration on AI ( ML24241A252)
- AI standards engagement (IEC, ANS, ASME) ML23132A305
2 AI Regulatory Framework Applicability Assessment
- Project objectives
- Assess the applicability of the existing regulatory framework in considering the unique aspects of AI applications Potential AI Technical Considerations
- Determine where regulatory gaps may exist which could for Regulatory Decision-Making require updating existing or developing new regulations, Explainability Trustworthiness Bias guidance, or procedures to evaluate and review AI uses in Robustness Ethics Security NRC-regulated activities Test, Evaluation, Assurance
- Project phases Risk Analysis Verification and Processes Validation
- AI-specific technology considerations Model Domain
- Literature review of regulatory requirements and guidance Maintenance Adaptation Data Drift Fielded Data Quality,
- Regulatory applicability assessment Performance Life Cycle Quantity,
- AI-specific regulatory considerations Degradation Management Applicability, and Un cer tai nty
- Review of AI standards
- Project supports AI Project Plan, Task 1.1
3 Regulatory Framework Applicability Assessment of Artificial Intelligence in Nuclear Applications (AIRGA: AI Regulatory Gap Analysis)
O. Pensado, P. LaPlante, M. Hartnett, K. Holladay
Data Science and AI Regulatory Applications WAI Regulatory Framework Applicability Considerationsorkshop:
September 17, 2024
4 Acknowledgments
This project benefitted from discussions with M. Dennis, L. Betancourt, A. Hathaway, N. Tehrani, A. Valiaveedu, and S. Haq of the NRC Office of Nuclear Regulatory Research This project was sponsored by the NRC Office of Nuclear Regulatory Research, Division of Systems Analysis The work is an independent product of SwR I and it does not necessarily reflect the views of the NRC
5 Disclaimer
This project was prepared as an account of work sponsored by an agency of the U.S. Government. Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use, or the results of such use, of any information, apparatus, product, or process disclosed in this publication, or represents that its use by such third party complies with applicable law.
6 Project Objective Support NRC s readiness to evaluate uses of AI technologies in NRC-regulated activities Main task: conduct an AI regulatory gap analysis (AIRGA)
- Identify types of AI technologies to be potentially used in the nuclear industry
- Identify potential AI uses in NRC-regulated activities
- Examine whether the existing regulatory framework is appropriate for AI technologies
7 Project Scope Regulatory framework considered
- NRC's regulations, Title 10, Chapter I, of the Code of Federal Regulations, Parts 1 - 171
- 517 regulatory guides (RGs)
Out of scope guidance documents:
- NUREGs, Interim Staff Guidance (ISG), Standard Review Plan (SRP), Inspection Procedures, and standards cited in regulations
- Technical areas without specific RGs, such as human factors engineering (ISG 2023-03, NUREG-0711, NUREG-0700), may need to be examined in the future
8 Main Project Tasks
Task 3: Analyze Task 4: Examine AI Task 1: Identify Task 2: Analyze regulations standards by examples of AI RGs to identify applicable to the professional uses potential gaps subset of RGs with communities potential gaps
Considered examples of AI uses based on known applications and R&D activities Examples discussed in past Data Science and AI Regulatory Applications Workshops Examples in NUREG/CR-7294
9 Analysis Approach for Regulatory Guides
Q0: Could AI technologies be used within RG is excluded from further analysis.
the scope of the RG? NONo gap in the RG
Y ES
Q1: Is the regulatory guide flexible to allow use of AI? Answer to Q1 The RG has a potential gap: detailed notes Q2: Does the regulatory guide provide or Q2 is NO are recorded in report appendices adequate guidance to evaluate the use of AI?
Y ES Answer to both Q1 and Q2 is YES
No gap in the RG
10 Regulatory Guides: 372 active RGs
Division
Division 9, Antitrust and Financial Review, has no active RGs
11 Regulatory Guides with Potential Gaps
Division
The RGs of divisions missing in the chart do not have potential gaps
71 RGs with potential gaps after applying the process from slide 7
- Questions Q0, Q1, and Q2
12 Categories of Potential Gaps Gap 1: Implied Manual Actions Gap 2: Special Computations Gap 3: Preoperational and Initial Testing Programs May Omit AI Gap 4: Habitability Conditions under Autonomous Operations Gap 5: Periodic Testing, Monitoring, and Reporting Gap 6: Software for Safety Critical Applications Gap 7: Radiation Safety Support Gap 8: Miscellaneous: Training, Human Factors Engineering, and AI Introduced as Changes
13 Gap 1: Guides call for human manual actions; AI systems may offer different alternatives to execute those actions Ta b l e 3-1. Regulatory guides related to Gap 1: Implied Manual Actions 1.7 Control of Combustible Gas Concentrations in Containment 1.114 Guidance to Operators at the Controls and to Senior Operators in the Control Room of a Nuclear Power Unit 1.141 Containment Isolation Provisions for Fluid Systems 1.147 Inser vice Inspection Code Case Acceptability, ASME Section XI, Division 1 1.149 Nuclear Power Plant Simulation Facilities for Use in Operator Training, License Examinations, and Applicant Experience Requirements 1.189 Fire Protection for Nuclear Power Plants 1.205 Risk-Informed, Performance-Based Fire Protection for Existing Light -Water Nuclear Power Plants 5.7 Entry/Exit Control for Protected Areas, Vital Areas, and Material Access Areas 5.44 Perimeter Intrusion Alarm Systems
14 Gap 2: AI techniques may be used in special computations; guidance may be needed on documentation and verification Ta b l e 3-2. Regulatory guides related to Gap 2: Special Computations 1.59 Design Basis Floods for Nuclear Power Plants 1.60 Design Response Spectra for Seismic Design of Nuclear Power Plants 1.76 Design-Basis Tornado and Tornado Missiles for Nuclear Power Plants 1.157 B e st-Estimate Calculations of Emergency Core Cooling System Performance 1.198 Procedures and Criteria for Assessing Seismic Soil Liquefaction at Nuclear Power Plant Sites 1.200 Acceptability of Probabilistic Risk Assessment Results for Risk-Informed Activities 1.203 Transient and Accident Analysis Methods 1.245 Preparing Probabilistic Fracture Mechanics (PFM) Submittals 1.247 TRIAL - Acceptability of Probabilistic Risk Assessment Results for Non-Light Water Reactor Risk-Informed Activities 3.27 Nondestructive Examination of Welds in the Liners of Concrete Barriers in Fuel Reprocessing Plants 3.76 Implementation of Aging Management Requirements for Spent Fuel Storage Renewals 5.11 Nondestructive Assay of Special Nuclear Material Contained in Scrap and Waste 5.21 Nondestructive Uranium-235 Enrichment Assay by Gamma Ray Spectrometry 5.23 In Situ Assay of Plutonium Residual Holdup 5.37 In Situ Assay of Enriched Uranium Residual Holdup 5.38 Nondestructive Assay of High-Enrichment Uranium Fuel Plates by Gamma Ray Spectrometry 10.4 Guide for the Preparation of Applications for Licenses to Process Source Material
15 Gap 3: Critical AI systems may need to be explicitly included in preoperational and initial testing programs
Table 3-3. Regulatory guides related to Gap 3: Preoperational and Initial Testing Programs May Omit AI 1.68 Initial Test Programs for Water-Cooled Nuclear Power Plants 1.68.2 Initial Startup Test Program to Demonstrate Remote Shutdown Capability for Water-Cooled Nuclear Power Plants 1.79 Preoperational Testing of Emergency Core Cooling Systems for Pressurized Water Reactors 1.79.1 Initial Test Program of Emergency Core Cooling Systems for New Boiling-Wa t e r Reactors
16 Gap 4: Habitability conditions under autonomous operations; variable role of operators
Ta b l e 3-4. Regulatory guides related to Gap 4: Habitability Conditions Under Autonomous Operations
1.78 Evaluating the Habitability of a Nuclear Power Plant Control Room During a Postulated Hazardous Chemical Release
1.189 Fire Protection for Nuclear Power Plants 1.196 Control Room Habitability at Light -Water Nuclear Power Reactors
17 Gap 5: Periodic testing, monitoring, sur veillance, and reporting; AI systems may offer different strategies for those activities
RGs related to testing and monitoring deemed with potential gaps:
1.7 1.205 5. 115.718.228.38 1.9 1.246 5.21 8.8 8.25 10.2 1.21 3.27 5.23 8.1 8.26 10.3 1.9 3.76 5.27 8. 118.3110.4 1.118 4.1 5.37 8.15 8.32 1.129 4.14 5.38 8.18 8.34 1.141 4.16 5.41 8.19 8.36 1.147 5.7 5.44 8.2 8.37 RG 5.71 Cyber Security Programs for Nuclear Power Plants: AI may be used as monitoring tool to detect anomalies as indicators of cyber attacks
18 Gap 6: Software guides may need to be updated to address special features and risks of AI systems
Ta b l e 3-6. Regulatory guides related to Gap 6: Software for Critical Applications 1.168 Verification, Validation, Reviews, and Audits for Digital Computer Software Used in Safety Systems of Nuclear Power Plants 1.169 Configuration Management Plans for Digital Computer Software Used in Safety Systems of Nuclear Power Plants 1.171 Software Unit Testing for Digital Computer Software Used in Safety Systems of Nuclear Power Plants 1.172 Software Requirement Specifications for Digital Computer Software and Complex Electronics Used in Safety Systems of Nuclear Power Plants
1.173 Developing Software Life Cycle Processes for Digital Computer Software Used in Safety Systems of Nuclear Power Plants
1.231 Acceptance of Commercial -Grade Design and Analysis Computer Programs Used in Safety -Related Applications for Nuclear Power Plants
19 Gap 7: Radiation safety support; AI may be used for tasks and functions of radiation safety professionals
Table 3-7. Regulatory guides related to Gap 7: Radiation Safety Support
8.8 Information Relevant to Ensuring that Occupational Radiation Exposures at Nuclear Power Stations Will Be as Low as Is Reasona bly Achievable 8.10 Operating Philosophy for Maintaining Occupational Radiation Exposures as Low as Is Reasonably Achievable 8.11 Applications of Bioassay for Uranium 8.15 Acceptable Programs for Respiratory Protection 8.18 Information Relevant to Ensuring that Occupational Radiation Exposures at Medical Institutions Will Be as Low as Reasonably Achievable 8.20 Applications of Bioassay for Radioiodine 8.22 Bioassay at Uranium Mills 8.25 Air Sampling in the Workplace 8.26 Applications of Bioassay for Fission and Activation Products 8.31 Information Relevant to Ensuring that Occupational Radiation Exposures at Uranium Recovery Facilities Will Be as Low as Is Reasonably Achievable 8.32 Criteria for Establishing a Tritium Bioassay Program 8.34 Monitoring Criteria and Methods to Calculate Occupational Radiation Doses 8.35 Planned Special Exposures 8.36 Radiation Dose to the Embryo/Fetus 8.38 Control of Access to High and Very High Radiation Areas of Nuclear Plants 10.4 Guide for the Preparation of Applications for Licenses to Process Source Material
20 Gap 8: Miscellaneous: Training, Human Factors Engineering, and AI Introduced as Changes
Ta b l e 3-8. Regulatory guides related to Gap 8: Miscellaneous: Training, Human Factors Engineering,
and AI Introduced as Changes 1.149 Nuclear Power Plant Simulation Facilities for Use in Operator Training, License Examinations, and Applicant Experience Requirements 1.206 Applications for Nuclear Power Plants 5.74 Managing the Safety/Security Interface
21 The analysis identified only few gaps in applicable regulations Regulations applicable to the RGs deemed with potential gaps were examined
- Not all the regulations were examined in detail In general, the applicable regulations (10 CFR 1 to 171) were high level and adequate for AI technologies, with a few exceptions The exceptions are related to regulatory statements calling for actions by humans when those actions could also be executed by AI systems
- Sur veillance using computer vision
- Searches of people and vehicles
- Escorting people in facilities Some regulations should be examined in light of questions related to autonomous operation and control room habitability
- Role of operators, protection of equipment in the control room, constraints on autonomous operation
22 Recommendations
Develop few general guides addressing cross-cutting issues associated with potential gaps, such as software development with AI systems and use of AI in special computations
- More efficient to develop few guides than inserting explicit AI considerations in many RGs Existing AI standards by professional societies do not readily address the identified potential gaps
23 Potential cross-cutting guides
- 1. Data quality for machine learning (ML) purposes, including aspects of accuracy, context, data management, data variety, and data quantity.
- 2. Type of systematic testing and documentation needed to enhance confidence in responses by AI systems, with emphasis on rare and extreme inputs.
- 3. Systematic fail-safe design, including active detection of inputs different than in the ML database, active detection of anomalous responses by AI systems, and mitigation of errors by AI systems.
- 4. Types of testing and documentation needed to enhance confidence in computations and predictions that use AI techniques.
24 Backup Slides
25 Abbreviations and acronyms AI Artificial intelligence A I RG A AI regulatory gap analysis ASME American Society of Mechanical Engineers CFR Code of Federal Regulations FA A U.S. Federal Aviation Administration F DA U.S. Food and Drug Administration HFE Human factors engineering IEEE Institute of Electical and Electronics Engineers ISG Interim Staff Guidance LLM Large language model ML machine learning N RC U.S. Nuclear Regulatory Commission PFM Probabilistic fracture mechanics R&D Research and development RG Regulatory Guide SRP Standard Review Plan Sw RI Southwest Research Institute
26 Terminology
Artificial AI includes a range of technologies Intelligence Deep neural networks and Machine machine learning methods are Learning notable because of their broad Deep Natural range of applicability Learning Language Processing
Computer Generative AI & Vision Large Language Models
27 Potential future work
Examine work by the FDA and FAA to regulate AI, to yield insights useful in developing the guidance described in recommendations 1 through 4.
Deploy a pilot program aimed at evaluating computations by an existing licensee using AI technologies, to yield insights relevant to recommendation 4 (use of AI for special computations).
28