ML26021A177

Human Readiness Levels NRC Presentation

ML26021A177
Person / Time
Issue date:	01/21/2026
From:	See J Office of Nuclear Reactor Regulation, Sandia, US Dept of Energy, National Nuclear Security Admin
To:
References
SAND2026-I6206PE
Download: ML26021A177 (18)

Text

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energys National Nuclear Security Administration under contract DE-NA0003525.

1 Judi E. See, Ph.D., CPE Sandia National Laboratories/New Mexico, USA jesee@sandia.gov, 505-844-4567 Applying Human Readiness Levels at the Nuclear Regulatory Commission SAND2026-16206PE Nuclear Regulatory Commission Human and Organizational Factors Seminar 21 January 2026

Biography 2

• Affiliation:

o Sandia National Laboratories (SNL), Albuquerque, New Mexico

• Education:

o Doctorate and Masters in Experimental Psychology (Human Factors) o Master of Engineering in Systems Engineering o BCPE certification in human factors and ergonomics

• Work Experience:

o Current: 15 years at SNL working in human factors, systems analysis, and nuclear explosive safety o Previous: 16 years as DOD contractor, primarily supporting Air Force rapid acquisition

• Published Research:

o Vigilance o Signal detection theory o Target detection o Visual inspection o Workload o Human readiness levels BCPE = Board of Certification in Professional Ergonomics; DOD = Department of Defense Dr. Judi See gave permission for this image to be released publicly for this briefing.

BLUF 3

• Human Readiness Level (HRL) scale was developed over a period of 12 years

• HRL scale can be applied to any technology, including automation, autonomy, and AI

• Formal HRL technical standard was completed in September 2021 and adopted by the DOD in May 2025

• HRLs can be applied to support a variety of NRC mission and goals BLUF = Bottom Line Up Front; DOD = Department of Defense; HSI = Human Systems Integration; TRL = Technology Readiness Level The HRL scale promotes a consistent HSI approach across diverse organizations and technologies throughout the lifecycle.

Human Readiness Level (HRL) Scale 4

• HRL scale is a simple nine-level scale to evaluate, track, and communicate readiness of a system for human use o Complements and supplements existing TRL scale during technology development o Provides a familiar systematic and consistent approach o Focuses on readiness of a technology for human use o Fully incorporates human element throughout lifecycle

• TRLs are routinely used in DOD, DOE, industry, and academia HRLs do for humans what TRLs do for technology Provide Assurance That:

TRL Technology will function as intended HRL Human is able to use the technology as intended DOD = Department of Defense; DOE = Department of Energy; TRL = Technology Readiness Level Novel leading-edge technique to enable decisive action promoting technology usability and suitability HRLs do NOT address physical or mental readiness of operators or teams to perform a mission.

Nine Levels of the HRL Scale 5

Level Technology Readiness Level Human Readiness Level Production / Deployment 9

Operational use of deliverable System successfully used in operations across the operational envelope with systematic monitoring of human-system performance 8

Actual deliverable qualified through test and demonstration Total human-system performance fully tested, validated, and approved in mission operations, using completed system hardware and software and representative users 7

Final development version of the deliverable demonstrated in operational environment Human systems design fully tested and verified in operational environment with system hardware and software and representative users Technology Demonstration 6

Representative of the deliverable demonstrated in relevant environments Human systems design fully matured as influenced by human performance analyses, metrics, prototyping, and high-fidelity simulations 5

Key elements demonstrated in relevant environments Human-centered evaluation of prototypes in mission-relevant part-task simulations completed to inform design 4

Key elements demonstrated in laboratory environment Modeling, part-task testing, and trade studies of human systems design concepts and applications completed Research &

Development 3

Concepts demonstrated analytically or experimentally Human-centered requirements to support human performance and human-technology interactions established 2

Concept and application formulated Human-centered concepts, applications, and guidelines defined 1

Basic principles observed and reported Basic principles for human characteristics, performance, and behavior observed and reported Increasing Maturity HRL scale is designed to map 1:1 with TRL scale

Benefits of the HRL Scale HRL scale emphasizes both progress and performance

• Human systems evaluations progress from basic conceptual design phases through prototype demonstrations and final qualification and fielding

• Human performance must be deemed satisfactory by qualified experts before advancement to next HRL level can occur 6

Benefits of HRL scale mirror those found in TRL scale TRL Scale Demonstrating high maturity for new technologies increases chances of program success HRL Scale Demonstrating human readiness for new technologies increases chances of program success Provides common language across diverse programs and organizations Promotes testing and verification Gauges progress to plan future level of effort Manages schedule and cost risks Provides proactive cradle-to-grave planning framework Provides assurance that technology functions and can be used as intended

Key Events in HRL Scale Evolution and Maturation 2010: Dr. Hector Acosta proposed HRL concept (Acosta, 2010) 2010: Naval Postgraduate School (NPS) masters thesis fleshed out concept into first nine-level HRL scale (Phillips, 2010) 2014: NPS masters thesis proposed a framework to standardize HSI throughout development (ONeil, 2014) 2015: Chief Scientist of the Air Force advocated requirements to augment TRL scale with HRL scale (Endsley, 2015) 2015: DOD HSI working group refined HRL scale with detailed descriptions (Phillips, 2015) 2019: SNL, Old Dominion University, and NPS chaired a second working group to mature HRL scale and assess utility (Salazar et al, 2021) 2020: SNL chaired writing committee to develop HFES 400 technical standard and complete peer reviews 2021: American National Standards Institute (ANSI) reviewed and approved HFES 400-2021 (ANSI/HFES, 2021) 2025: U.S. DOD formally adopted ANSI/HFES 400-2021 7

HRL scale was matured over a period of 12 years

ANSI/HFES 400-2021 Writing Committee Diverse writing committee Diverse set of peer reviewers Consensus committee (N = 16) o Formally voted whether to approve draft standard o

Represented multiple entities in DOD, industry, and academia Public reviewers (N = 7) o Volunteered in response to announcements in ANSI Standards Action weekly publication and HFES Web site o

Provided feedback but did not formally vote 8

HRL Writing Committee Organizations

1. Federal Aviation Administration

2. General Motors Company

3. Human Factors and Ergonomics Society

4. Johns Hopkins University Applied Physics Laboratory

5. Navy Expeditionary Combat Command

6. Northrop Grumman

7. Old Dominion University

8. SA Technologies

9. Sandia National Laboratories (Chair)

Producer 31%

User 44%

General Interest 25%

HRL standard underwent a rigorous development and review process ANSI/HFES 400-2021 was formally approved and released in September 2021.

ANSI/HFES 400-2021 Layout and Contents Main body o

1.0 Purpose and background o

2.0 Scope o

3.0 Related standards and documents o

4.0 Human readiness level scale o

5.0 Relationship between HRL and TRL scales o

6.0 Mapping the HRL scale to acquisition frameworks o

7.0 Applying the human readiness level scale o

8.0 References Appendices o

Appendix A:

DOD budget activities o

Appendix B:

HRL-AAF mapping o

Appendix C:

HRL guidance o

Appendix D:

Application examples o

Appendix E:

Process considerations 9

https://www.hfes.org/Publications/Technical-Standards Purpose Define nine levels of HRL scale Provide guidance to apply HRL scale Main body provides high-level overview, with details in appendices AAF = Adaptive Acquisition Framework

ANSI/HFES 400-2021: Appendix C HRL Guidance 10 1.

Have key human behaviors, capabilities, and limitations been identified?

2.

Have preliminary usage scenarios for potential users been identified?

3.

Have potential key human performance issues and risks been identified and concomitant basic research conducted?

4.

Has basic human research relevant to a developing concept or application been conducted?

HRL Level 1 Basic principles for human characteristics, performance, and behavior observed and reported Additional Description for Each Question At this very early conceptual stage, key human-centered issues requiring further investment in research and development may be identified. Exploring potential human-centered issues and risks provides an opportunity to consider each HSI domain with respect to possible implications for technologies, systems, operations, concepts, and support. The intent is to highlight areas that may warrant in-depth attention from a human systems perspective and begin planning approaches to mitigate or prevent those issues and risks.

HRL scale questions serve as triggers to consider applicability of multiple HSI topics throughout design and development.

Appendix C is the main section for human systems practitioners to apply HRL scale Guidance and considerations are provided to further clarify each HRL level Exit criteria and supporting evidence o

Document key human characteristics, performance, and behaviors o

Document potential technology or concept usage scenarios at a high level o

List potential key human performance issues and risks o

Document basic human research findings

HRL Scale Supports Multiple Human Systems Evaluations Evaluation Activity HRL 3 HRL 4 HRL 5 HRL 6 HRL 7 HRL 8 Usage scenarios

Human performance metrics

Human-machine allocations

Human factors engineering

Safety and occupational health

Manpower, personnel, training

Environment

Other relevant HSI domains

Maintenance and sustainment

Strategies for human use

Conformance to guidelines and principles

User procedures and other manuals

Issue tracking system

11 HRL scale supports iterative evaluation of all HSI domains and other core human-centered topics as technical maturity progresses.

HRL scale provides multiple opportunities to evaluate key human systems topics

Application of ANSI/HFES 400-2021 Qualified human systems experts provide HRL ratings to support decision making at key milestones

• Future program direction

• Investment of time and resources 12 2

2 6

1 7

7 Technology concept and application formulated Human-centered concepts, applications, and guidelines defined System/subsystem model or prototype demonstration in a relevant environment Basic principles for human characteristics, performance, and behavior observed and reported System prototype demonstration in an operational environment Human systems design fully tested and verified in operational environment with system hardware and software and representative users

• Very low level of maturity

• TRL and HRL activities are well aligned

• Technical maturity has advanced

• HRL lags behind TRL by 5 levels

• High level of maturity

• TRL and HRL activities are well aligned TRL Rating HRL Rating HRL scale consolidates multiple HSI metrics to provide a single human readiness number

Applying the HRL Scale at the NRC (1 of 2) 13

• NRC regulations and licensing to protect the public and nuclear power plant workers

• Incorporate ANSI/HFES 400-2021 into Code of Federal Regulations

• Establish compliance as a legally binding requirement for licensees

• Equipment and processes to maintain security of facilities, materials, and information

• Evaluate human usability of intrusion detection systems and perimeter patrols

• Evaluate human roles in cyber security to protect against sabotage or malicious use

• Inspections to identify weaknesses in design, operations, procedures, or equipment

• Communicate human-system issues with HRL language

• Guide and evaluate upgrades

• Evaluate human readiness of maintenance procedures and processes The need to assure human readiness permeates NRCs mission HRLs can be used to support NRCs mission to protect public health and safety, the environment, and national security.

• Are new/existing NRC Operations Center interfaces and processes human ready?

Applying the HRL Scale at the NRC (2 of 2) 14

• Increased interest in nuclear power to support U.S. energy needs

• Meet power demands for AI data centers

• Provide zero-emission energy supply that helps stabilize the grid

• Changing nature of nuclear reactor technologies and operations

• Fission batteries plus micro and small modular reactors

• Mobile and temporary deployments

• Deployment in remote areas with limited support

• Possible configurations consisting of one control room for multiple reactors

• Design of automated control rooms to reduce staffing requirements Recent developments in the nuclear power arena introduce changes in human roles that can be addressed via HRLs Continued or increased use of nuclear power depends on presence of highly reliable, safe systems designed and operated in alignment with human operator and maintainer capabilities and limitations.

Small Secure Transportable Autonomous Reactor (SSTAR)

Applying the HRL Scale in the Age of Automation and AI 15

• Automation, autonomy, and AI change human roles but do not eliminate people altogether

• Application of HRL scale can capture and mitigate human systems issues early in design for any technology

• Multiple questions throughout HRL scale are relevant to human-machine teaming scenarios (Yalim & Handley, 2025)

HRL scale can be applied to any technology 1.

Have key human behaviors, capabilities, and limitations been identified?

2.

Have preliminary usage scenarios for potential users been identified?

3.

Have potential key human performance issues and risks been identified and concomitant basic research conducted?

4.

Has basic human research relevant to a developing concept or application been conducted?

Basic principles for human characteristics, performance, and behavior observed and reported HRL Level 1 Pacific Northwest National Laboratory Example Incorporated human readiness evaluations of early-TRL AI-based machine learning algorithm o Provides contingency recommendations for power grid control room operators to maintain grid stability and prevent blackouts o Algorithm development to date had focused on accuracy of recommendations Low operator trust and high workload ratings indicated lack of human readiness and path to increase HRL Wenskovitch et al., 2022 Optimizing the AI algorithm alone does not necessarily optimize performance of the joint human-AI system.

Morey et al., 2025

Adoption of ANSI/HFES 400-2021 16 DOD officially adopted the HRL standard

• DOD adopted HRL standard at Tier 1 on May 1, 2025

• Featured in FedGovToday interview on August 24, 2025

• Implementation guidance will be included in revised DOD Human Systems Integration Guidebook (2022)

DOD Adoption of Non-Government Standards Tier 1 Standards Essential for DOD system effectiveness, safety, and interoperability Recognized as critical to mission success Re-evaluated for adoption with each revision Ensure changes do not affect performance or mission accomplishment Tier 2 Standards Considered important but not as critical for safety or interoperability May support specific applications or systems Adoption is a one-time event No requirement for re-evaluation with each revision Per DODM 4120.24 Defense Standardization Program (DSP) Procedures What does this mean for the NRC?

Conclusions and Future Activities 17

• HRL scale and standard have been available more than four years

• HRL scale can be applied to any technology to evaluate, track, and communicate human readiness

• HRLs can be applied to support NRC mission and goals

• ANSI/HFES 400-2021 standard must be reaffirmed or revised every five years o First renewal due by September 7, 2026 o E-mail jesee@sandia.gov if interested in contributing Questions and Discussion For additional information or questions, contact Judi See at jesee@sandia.gov or 505-844-4567.

References Acosta, H. (2010, May 9-13). Human readiness levels: Implementing HSI - Connecting some dots [Panel discussion]. 81st Annual Scientific Meeting of the Aerospace Medical Association, Phoenix, Arizona, USA.

American National Standards Institute/Human Factors and Ergonomics Society (2021). Human readiness level scale in the system development process (ANSI/HFES 400-2021). Human Factors and Ergonomics Society. https://my.hfes.org/online-store/publications Endsley, M. (2015, February 10-11). Human readiness levels: Linking S&T to acquisition [Plenary address]. 2015 National Defense Industrial Association Human Systems Conference, Alexandria, Virginia, USA. https://ndiastorage.blob.core.usgovcloudapi.net/ndia/2015/ human/WedENDSLEY.pdf Morey, D. A., Rayo, M. F., & Woods, D. D. (2025). Empirically derived evaluation requirements for responsible deployments of AI in safety-critical settings.

npj Digital Medicine, 8, 374-384. https://doi.org/10.1038/s41746-025-01784-y ONeil, M. P. (2014). Development of a human systems integration framework for Coast Guard acquisition (Unpublished masters thesis). Naval Postgraduate School. https://apps.dtic.mil/dtic/tr/fulltext/u2/a608012.pdf Phillips, E. L. (2010). The development and initial evaluation of the human readiness level framework (Unpublished masters thesis). Naval Postgraduate School. https://apps.dtic.mil/sti/pdfs/ADA525365.pdf Phillips, H. (2015, May 4-8). DOD HFE TAG: Human readiness level (HRL) working group [Paper presentation]. DOD HFE TAG 69th meeting, Orlando, Florida, USA.

Salazar, G., See, J. E., Handley, H. A. H., & Craft, R. (2021). Understanding human readiness levels. Proceedings of the 2020 Human Factors and Ergonomics Society 64th Annual Meeting, 64(1), 1765-1769. https://doi.org/10.1177/1071181320641427 Wenskovitch, J., Jefferson, B., Anderson, A., Baweja, J., Ciesielski, D., & Fallon, C. (2022). A methodology for evaluating operator usage of machine learning recommendations for power grid contingency analysis. Frontiers in Big Data, 5(897295). https://doi.org/10.3389/fdata.2022.897295 Yalim, C., & Handley, H. (2025). Application of human readiness levels to AI intensive systems. MODSIM World 2025. Paper No. 0018.

https://modsimworld.org/papers/2025/MODSIM_2025_paper_18.pdf 18