ML23279A096

2023 Ehprg Paper Failure Modes in Human-Automation Integration

ML23279A096
Person / Time
Issue date:	09/25/2023
From:	Niav Hughes, Jing Xing NRC/RES/DRA/HFRB
To:
References
Download: ML23279A096 (1)

Text

Failure Modes In Human-Automation Integration Jing Xing, Niav Hughes Green U.S. Nuclear Regulatory Commission Jing.xing@nrc.gov Niav.Hughes@nrc.gov Abstract The overall level of automation in advanced nuclear power plants (NPPs) is expected to be much higher than in currently operating plants in the United States. Similarly, more automation technologies supported by digital instrument and controls (DI&C) are emergent for control room modernization of operating plants. It is important for designers, utilities, and regulators to be cognizant of current practices and trends in the use of automation and understand the influences of automation on control room design, human performance, and conduct of operations. Specific questions for consideration include: How reliable is the automated system and how does this reliability impact the operators use of the system? What other applications have used the system that would support determination of its reliability and exemplify issues with its reliability? How does automation affect operators during normal, abnormal, and emergency operations? The knowledge and data for such an understanding constitute the technical basis for design and regulatory review guidance. This paper organizes the knowledge and data about automation failures and deficiencies in human-automation interaction into a structured framework and relates those to the elements of human factors engineering (HFE) review in NUREG-0711. This structure can enhance the consideration of potential automation failures in the design evaluation process and facilitate integration of DI&C and HFE in the regulatory review process of automation technologies.

1. Introduction Advanced nuclear reactor technologies present new opportunities and new challenges for both the U.S. Nuclear Regulatory Commission (NRC) staff and the nuclear industry. High level automation is expected to be prevalent in advanced nuclear power plants (NPPs) and modernization of NPP control rooms. Automation has been used for several purposes in NPPs worldwide. The NRC staff have approved the design of technologies proposing higher levels of automation including the Westinghouse AP1000 [1] and NuScale [2]. Two AP1000 units are authorized for operation by the NRC [3, 4]; each unit is operated from a nearly fully digital control room. Vogtle Unit 3 is currently operating, and Vogtle Unit 4 is in the final stages of construction and testing [5]. The NRC is engaged in pre-application activities with vendors proposing new and advanced designs that will likely utilize fully digital control rooms with high levels of automation. Meanwhile, modernization activities also engender new implementations of control room automation. For example, modifications to traditional plants in the US seek to employ digital instrumentation and controls (DI&C) on safety systems. It is important that the NRC staff is cognizant of current practices and trends of automation applications in NPP control rooms and understands the influences of automation on control room design, human performance, and conduct of operations. Knowledge about the implications of different implementations of automation on operator performance will enhance the technical basis for NRC staff performing human factors engineering (HFE) review in regulation and licensing.

The NRC has taken steps towards the preparation and readiness for licensing activities associated with modernized and advanced reactors. These activities necessarily include consideration of DI&C and HFE. Regarding modernization of operating plants, DI&C-ISG-06((6] contains the interim staff guidance (ISG) to review license amendment requests associated with safety-related DI&C equipment modifications in operating and new plants once they become operational. DI&C-ISG-06 indicates that, for modifications that may involve HFE considerations, an HFE safety evaluation should be performed in accordance with NUREG-0711 [7], Human Factors Engineering Program Review Model; and NUREG-1764 [8], Guidance for the Review of Changes to Human Actions, with close coordination with the DI&C evaluation under SRP Chapter 7, Instrumentation and Controls. Modernization efforts involve replacing analog instrumentation and control with DI&C as well as automating what once may have been manually performed activities.

Regarding the regulatory landscape for advanced reactors, in 2023, the NRC issued SECY-23-0021 (Proposed Rule: Risk-Informed, Technology-Inclusive Regulatory Framework for Advanced Reactors) [9]. When approved, this proposed rule will be codified in Title 10 of the Code of Federal Regulations (10 CFR) Part 53. Guidance in several key technical areas supporting the proposed rule were identified to accompany the 10 CFR 53 rule package. Accompanying the proposed 10 CFR Part 53 rule language is draft ISG Development of Scalable Human Factors Engineering Review Plans (DRO-ISG-2023-03) [10]. This ISG provides a risk-informed and performance-based process to screen and target the most safety-significant systems, systems with likely human factors challenges, and novel elements of the design (ref Scalable ISG). Novel elements of the design will likely include more advanced implementations of automation.

Automation in NPP control rooms includes those controlling reactor systems, such as an automatic feedwater system, and human-system interface (HSI) automation such as a computerized procedure system. Given that reactor control is a closed-loop control system, essentially every NPP function involves automation. While such automatic controls do not directly interact with operators, they can impact operator performance in that the operator needs to understand how the automation works and needs to be able to intervene in situations where automation is not reliable. On the other hand, HSI automation directly interacts with operators. Specific research questions that the staff consider in the safety review of an automated system include the following:

- How reliable is the automated system, and how does this reliability impact the operators use of the system?

- What other applications have used the system that would support determination of automation reliability and exemplify issues with its reliability?

- What are the automation failure modes that impact operator performance?

- How does automation affect operators during normal, abnormal, and emergency operations?

Research has been conducted on human-automation systems for NPP control rooms. For example, the Halden Reactor Project consolidated its two decades long research results studying human-automation interaction in NPP simulators [11], Several organizations have reported human performance studies of computerized procedures with automation functioned embedded [12].

Operational experience review has documented many DI&C and automation events in NPPs. Most DI&C events involve issues in human factors engineering (HFE) considerations. Moreover, vast operational experience and research on automation related events or accidents are available in non-nuclear domains, especially in aviation. Taken together, these different sources of information can provide an understanding of automation reliability and how that reliability impacts the operators use of automation. Skraaning and Jamieson, in their paper The Failure to Grasp Automation Failure [13],

reviewed recent aviation accidents involving automation failures and proposed an initial taxonomy of

automation failure and automation-related human performance challenges. They point out the utility of the taxonomy for advancing human-automation interaction research.

In 2022, an interdisciplinary team of NRC staff working in DI&C, HFE, and risk analysis systematically evaluated the findings from investigative reports of BOEING 737 crashes 149].The team examined the recommendations in these investigative reports for their potential implementation in the NRCs DI&C regulatory process. The NRC team recommended focusing on the following areas to continue to improve DI&C licensing and regulatory oversight:

- The NRC should continue to improve integration and communication among DI&C technical reviews, HFE reviews, and subsequent inspection oversight for new or significantly different applications from conception to installation.

- The NRC should continue to improve its oversight programs for DI&C modifications that are implemented through 10 CFR 50.59, Changes, tests and experiments, that do not require prior NRC approval.

- The NRC should develop guidance for assessing systems engineering approaches for the DI&C design and human factors life-cycle evaluation, which are important for ensuring that approved DI&C designs are appropriately integrated to maintain safety functionality.

- The NRC should explore potential avenues for increasing the collection and communication of DI&C operating experience to enable application in a quantitative assessment of DI&C systems in the licensing and oversight processes.

Two of the four leading recommendations address HFE in DI&C reviews and they underscore the importance of effective communication and integration among HFE and DI&C for the review of advanced reactor technologies. For this paper, the authors reviewed the literature on automation reliability and operational events and accidents related to DI&C to gather information of automation failures, causes, impacts on human performance, and HFE considerations. Moreover, we relate the information to the NRCs HFE review process described in NUREG-0711. This paper presents a summary of the results. Echoing the recommendations made by the NRC team on integration of DI&C, HFE, and risk analysis, the results of this study serve to document the regulatory basis and enhance the integration of the NRCs regulatory and licensing activities in HFE and DI&C.

2. Approaches First, the literature and event reports related to automation reliability in nuclear systems as well as non-nuclear applications that share commonalities with NPP control room operations were reviewed.

Then, four applications were selected for the literature and operational experience review:

- Automation and DI&C systems in nuclear power plants

- Aircraft (NRSB database and literature)

- Locomotive (NTSB database and literature)

- Process control systems (literature)

These applications were selected for the sources of information because they shared many critical functional aspects in common with NPP systems:

1) They are complex control systems

2) Operators are in control of the systems, although the systems can run at high or full automation modes.

3) Operators tasks involved monitoring, situational assessment, decision-making / planning, manipulation / control, and teamwork;

4) Operator actions procedure-based.

Also, differences between the operations of the non-nuclear automation systems and NPP operations were considered. The major differences identified include:

- Time scale: Automation operations and interactions with human in aircraft occur much faster than those in NPP in emergency situations.

- Procedure-following: Aircraft operations are less procedural than NPPs and more skill-dependent.

For each piece of literature or event reports reviewed, the reported automation or DI&C failures, deficiencies or causals to the failure, and the resulting human failures of performing tasks or recovering system failures were documented. Then, the identified information was consolidated into failures in various DI&C elements and human cognitive failure modes. Also, the causes were consolidated into a list of human-automation integration deficiencies and structured the deficiencies into categories corresponding to the elements in the NRCs HFE review framework in NUREG-0711.

Note that the authors performed the majority of operational experience review during 2009 - 2011.

The work presented in this paper has very limited review and updates of the more recent information from the past decade.

3. Results In the section, a framework for integrating DI&C and HFE, followed by automation failure modes, human-automation integration deficiencies, and human failure modes is presented.

3.1 A Framework for Integrating DI&C and Human Factor Engineering The authors consolidated the review findings into a framework to facilitate regulatory and licensing activities in integrating DI&C and human factors engineering. This framework represents how ideally human-automation integration works: i) The DI&C elements together achieve the functions of automation systems; the behaviors of the DI&C elements impact the elements of human-automation interaction; ii) The human-automation interaction elements are designed to ensure that the automation functions work without leading to human errors and that failures of DI&C elements do not propagate to human failures, iii) The human cognition elements should ensure that human operators are capable of identifying and recovering from automation failures to make a successful recovery.

Figure 1. A framework for integrating DI&C systems and HFE Figure 1 shows the framework that integrates DI&C systems and HFE. In the diagram, the box on the left represents DI&C elements that constitute an automation system. Each element incurs some known types of automation failure modes. The box in the middle of the diagram represents HFE elements that support reliable human interaction with the automation. Each element incurs some known types of human-automation integration deficiencies. The box on the right represents the elements of human cognitive task performance. Each element incurs known types of human cognitive failure modes.

The framework also represents how human-automation integration may fail. In the events we reviewed, failures of human-automation integration led to undesired failure events occurring in the following ways:

1) The automation system worked as expected but the deficiencies in human-automation interaction led to human failures.

2) The automation failures led to, or aggravated, human-automation integration deficiencies and then led to human failing to identify or recover automation failures.

3) The DI&C element had unusual behaviors or deviated from operators understanding, thus leading to or aggravating human-automation integration deficiencies and then leading to human failures.

This framework can be used to analyze the success and failures of human-automation integration.

Since this paper focuses on failures, the results section next will present the failure modes and deficiencies analyzed from the review.

3.2 DI&C Failure Modes There have been many studies on DI&C failure modes from the perspective of I&C and software reliability. We do not intend to propose a new or comprehensive taxonomy of DI&C failure modes.

This section documents the types of DI&C element failures from our limited review. These failures are the initiators to human failures in the reviewed events.

Table 1: Failure Modes for DI&C System Elements DI&C element Failure modes

• Sensor failure Sensors / signals

• Unreliable signals

• Boundary conditions not clearly defined

• Problems in control strategies or algorithms

• Failure of automation disengagement

• Unclear or unexpected change in automation modes Control logic

• Dependency between systems or subsystems

• Errors or inappropriate use of the database

• Poor Software standardization, e.g., similarity of terms and units Software may be superficial, software versions may proliferate

• Unclear or unidentified software failures Network

• Network congested

• Failed execution Soft control

• Unauthorized execution 3.3 Human-Automation Integration Deficiencies NUREG-0711 includes twelve HFE elements. The human-automation integration deficiencies were synthetized into the following elements:

- Functional Requirements Analysis and Function Allocation

- Staffing and Qualifications

- Task Analysis

- Treatment of Important Human Actions

- Human-System Interface Design

- Procedure Development

- Training Program Development

- Human Factors Verification and Validation Functional Requirements Analysis and Function Allocation NUREG-0711 states The purpose of this element is to verify that the applicant defined those functions that must be carried out to satisfy the plants safety goals and that the assignment of responsibilities for those functions (function allocation) to personnel and automation in a way that takes advantage of human strengths and avoids human limitations.

Below are the human-automation integration deficiencies related to this element:

• Authority - Authority describes the ability of the automated system to override or block human input, and vice-versa:

o operators have responsibility but lack authority o mode transitions may be uncommanded o communication between automation and other systems is unsupervised o control authority may be diffused o automation puts operators out of loop

• Autonomy - Autonomy refers to the capability of automated systems to operate for long periods of time with minimal operator input:

o automation behavior may not be apparent o automation behavior may be unexpected and unexplained

• Complexity - structural and functional complexity of automation. As automated systems grow more powerful, they are also more complex in the number of automated components and the calculations and logic required to produce system behavior:

o too many layers of automation loops o complex automation may have overly simplistic interface o complexity of automation results in high workload, e.g., high working memory demands task management may be difficult., data entry and programming may be difficult and time consuming.

• Information requirements - This refers to the information provided by the automation to operators:

o information presented is too complex o information provided by automation systems overloads operators o information is not timely o information is incomplete, e.g., inadequate information for decision-making o information is not reliable o incorrect information for situational assessment due to model (control logic) errors o improper or biased information for decision-making

• Functionality requirements - This refers the requirements for defining how automation functions work:

o automation may not work as expected under unusual conditions, e.g., automation does not function as designed due to exceeding the design limits o automation requirements may conflict among themselves o the boundary conditions for automation performance may be limited o automation may use control strategies that are different from operators expect

• Coupling or dependency - Coupling refers to interconnections between systems.

o unknown dependency exists o system coupling may lead to human failure dependency, i.e., failure of human action in one system leads to failure with other systems o systems become dependent through human actions

• Observability or feedback o automated systems provide inadequate feedback regarding their actions.

Functional Allocation

• Automation level decisions may be inappropriate

• Automaton function (e.g., protections) may be lost though operators continue to rely on them

• Manual operation may be difficult after transition from automated control

• Automation use may slow operator responses

• Inadvertent automation engagement or disengagement

• Operators in low vigilance due to long-duration continuous monitoring tasks

• Task Analysis NUREG-0711 states, The functions allocated to plant personnel define the roles and responsibilities that they then accomplish via human actions (HAs). HAs can be divided into tasks, a group of related activities with a common objective or goal. The objective of this review is to verify that the applicant undertook analyses identifying the specific tasks needed to accomplish personnel functions, and also the alarms, information, control- and task-support required to complete those duties.

Below are the human-automation integration deficiencies related to this element:

• Using automation causes operators multitasking

• Monitoring requirements may be excessive

• Automation may demand sustained attention (Keyhole effect)

• Use of automation diverted crews attention from their primary tasks

• Data entry and interface management of automation systems may be difficult and time consuming

• Cognitively demanding tasks that require operators to understand the structure logic of automation.

• Crew coordination and communication are weak due to the use of automation

• Non-automated tasks may not be integrated Treatment of Important Human Actions NUREG-0711 states, The objective of this element of an HFE program is to identify those HAs (human actions) most important to safety for a particular plant design; this is accomplished through a combination of probabilistic and deterministic analyses.

Below are the human-automation integration deficiencies related to this element.

• Inadequate identification of system failure modes - Automation failure modes may be unanticipated by operators (and designers)

• Lack of understanding automation failure - the nature and consequences of automation failures and how operators respond to and cope with them

• Lack of understanding and considerations of human cognitive capacity limits and failure modes

• Operators failing to understand automation failures and how to recover from failures

• Inadequate considerations of possible scenarios of transition from automation to manual operation

• Lack of consideration of dependencies between important human actions Human-System Interface Design NUREG-0711 states, The objective of this review element is to evaluate the process used by applicants to translate the functional- and task-requirements to HSI design requirements, and to the detailed design of alarms, displays, controls, and other aspects of the HSI. A structured methodology should guide designers in identifying and selecting candidate HSI approaches, defining the detailed design, and performing HSI tests and evaluations. The review also addresses the formulation and employment of HFE guidelines tailored to the unique aspects of the applicants design, e.g., a style guide to define the design-specific conventions.

This element of NUREG-0711 is supported by the detailed design guidelines in the NRCs human-system-interface review guidance, NUREG-0700. Section 9 of Revision 3 of NUREG-0700 [15]

contains additional guidelines specifically for reviewing automation in control room human-system interfaces. The guidelines consider the following aspects of automation: Automation Displays, Alerts, Notifications, and Status Indications, Interaction and Control, Automation Modes, Automation Levels, Shared Control, Operation by Consent, Operation by Exception, Adaptive Automation, Computerized Operator Support Systems (COSS), HSI Integration.

Below are the human-automation integration deficiencies related to this element:

• Human-centered design - the degree to which human characteristics, capabilities, limitations, and preferences are taken into account in automation design:

o operational knowledge may be lacking in design process o cultural differences may not be considered

• Automation displays and controls:

o Cognitively-demanding information integration may be required to use the automation o data access may be difficult o critical data may not be directly visible to operators o controls of automation may be poorly designed o lack of data entry verification o flaw in automation state indication o automation reset variables are not known to operators o inadequate feedback from system to operators.

o lack of confirmation of actions execution Training Program Development NUREG-0711 states, Training plant personnel is important in ensuring the safe, reliable operation of nuclear power plants. Training programs aid in offering reasonable assurance that plant personnel have the knowledge, skills, and abilities needed to perform their roles and responsibilities. The objective of the training program review is to verify that the applicant has employed a systems approach for developing personnel training.

Below are the human-automation integration deficiencies related to this element:

• Operator Skills of using automation:

o skills specific to automation may not be acquired o manual skills may be lost o monitoring / action patterns may change

• Operators inadequate knowledge of automation system - how well operators understand the structure and function of automation.

• Operators inadequate mental model due to not understanding the system - the mode and behavior of the automation, including its current and projected state, i.e., what it is doing now and what it will do in the future.

• Automation behaviors misunderstood.

• Inadequate training on communication and coordination using automation systems - how automation affects crew interaction:

o over reliance on automation o cross checking may be difficult or ignored o inter-team communication may be reduced o communication between computers may be unsupervised or unaware to operators o crew coordination problems may occur

Procedure Development Procedures are essential to plant safety because they support and guide personnel interactions with plant systems and personnel responses to plant-related events. In the nuclear industry, procedure development is the responsibility of individual utilities. The objective of the NRC procedure review is to confirm that the applicant's procedure development program incorporates HFE principles and criteria, along with all other design requirements, to develop procedures that are technically accurate, comprehensive, explicit, easy to utilize, validated, and conform to the requirements in 10 CFR 50.34(f)(2)(ii). Potential failures from not following this guidance include:

• Automation information in manuals may be inadequate

• Procedure does not match specific scenarios

• Procedure is not readily useful, e.g.,

o Difficulty making timely determination of the right procedure for unusual situations o Operators need to infrequently use multiple-step procedures from memory Human Factors Verification and Validation NUREG-0711 states, Verification and validation (V&V) evaluations comprehensively determine that the final HFE design conforms to accepted design principles and enables personnel to successfully and safely perform their tasks to achieve operational goals. This element involves three evaluations, with the following objectives:

• HSI Task Support Verification - the applicant verified that the HSI provides the alarms, information, controls, and task support defined by tasks analysis needed for personnel to perform their tasks.

• HFE Design Verification - the applicant verified that the design of the HSIs conform to HFE guidelines (such as the applicants style guide).

• Integrated System Validation - the applicant validated, using performance-based tests, that the integrated system design (i.e., hardware, software, procedures and personnel elements) supports safe operation of the plant.

The current proposed 10 CFR Part 53 rule § 53.440 (n)(4) states, A functional requirements analysis and function allocation must be used to ensure that plant design features address how safety functions and functional safety criteria are satisfied, and how the safety functions will be assigned to appropriate combinations of human action, automation, active safety features, passive safety features, or inherent safety characteristics.

The authors did not systematically identify human-automation integration deficiencies related to this element. Most studies and event reports reviewed by the authors did not provide information on how V&V was performed or why V&V did not reveal the problems. Several nuclear power plant DI&C event reports made brief notes on the need for improvements to human factors validation. For example, one DI&C event report noted that the human error resulted from installing a DI&C automation system could have been captured during the system validation process. Echoing Skraaning and Jamiesons recent work The Failure to Grasp Automation Failure, future research should be performed on capturing potential human-automation failures in the V&V process.

Staffing and Qualifications NUREG-0711 states, The objective of reviewing staffing and qualification analyses is to verify that the applicant has systematically analyzed the requirements for the number of personnel and their

qualifications that includes gaining a thorough understanding of the task and regulatory requirements.

The authors did not notice reported human-automation integration deficiencies related to this element through our review of literature and event reports. Automation can affect staffing. In fact, often the main purpose of using automation is to reduce staffing requirements. Human factors analysis and validation are needed for staff modifications due to the use of automation. For example, the Boeing company performed validation testing when changing the required flight deck crew number from three to two with a newly designed cockpit. Likewise, NuScale performed a series of testing and evaluation for small modular reactor staffing resulting in an exemption from the current federal requirements of nuclear power plant control room staffing. Lessons learned from staffing evaluation and testing should be documented to enhance HFE in this element.

3.4 Human Cognitive Failure Modes Resulted from Human-Automation Integration Deficiencies The NRCs new human reliability methodology, the Integrated Human Event Analysis System (IDHEAS), has a set of cognitive failure modes that models human failures in task performance. The cognitive failure modes represent failures of macrocognitive functions, which are the basic cognitive elements to achieve tasks. The cognitive failure modes are human-centered and technology neutral, thus they can be used to model human failures in DI&C environments and with any automation systems. We were able to represent the types of human failures in the events reviewed with IDHEAS cognitive failure modes.

IDHEAS cognitive failure modes consist of the failures of the following five macrocognitive functions:

• Detection (D) is noticing cues or gathering information in the work environment.

• Understanding (U) is the integration of pieces of information with a persons mental model to make sense of the scenario or situation.

• Decisionmaking (DM) includes selecting strategies, planning, adapting plans, evaluating options, and making judgments on qualitative information or quantitative parameters.

• Action execution (E) is the implementation of the decision or plan to change some physical component or system.

• Interteam coordination (T) focuses on how various teams interact and collaborate on an action.

Each macrocognitive function is achieved through a set of basic cognitive processors. IDHEAS uses the failure of the processors as detailed cognitive failure modes. These detailed failure modes are more specific when being used to model types of human failures in human-automation integration.

Below are the detailed cognitive failure modes:

• Failure of Detection (D)

D1. Fail to establish the correct mental model or to initiate detection D2. Fail to select, identify, or attend to sources of information D3. Incorrectly perceive or classify information D4. Fail to verify perceived information D5. Fail to retain, record, or communicate the acquired information

• Failure of Understanding (U)

U1. Fail to assess/select data U2. Fail to select/adapt/develop the mental model U3. Fail to integrate data with the mental model to generate the outcome of understanding (situational awareness, diagnosis, resolving conflicts)

U4. Fail to verify and revise the outcome through iteration of U1, U2, and U3 U5. Fail to export the outcome

• Failure of Decisionmaking (DM)

DM1. Fail to Adapt the infrastructure of decisionmaking DM2. Fail to Manage the goals and decision criteria DM3. Fail to Acquire and select data for decisionmaking DM4. Fail to Make decision (judgment, strategies, plans)

DM5. Fail to Simulate or evaluate the decision or plan DM6. Fail to Communicate and authorize the decision

• Failure of Action Execution (E)

E1. Fail to Assess action plan and criteria E2. Fail to Develop or modify action scripts E3. Fail to Prepare or adapt infrastructure for action implementation E4. Fail to Implement action scripts E5. Fail to Verify and adjust execution outcomes

• Failure of Interteam Coordination (T)

T1. Establish or adapt teamwork infrastructure T2. Fail to Manage information T3. Fail to Maintain shared situational awareness T4. Fail to Manage resources T5. Fail to Plan interteam collaborative activities T6. Fail to Implement decisions and commands T7. Fail to Verify, modify, and control the implementation

4. Insights and Concluding Remarks

1) Through review and analysis, the authors noticed that most human-automation failure events involved one or more deficiencies in the element of functional requirement analysis. This element occurs in the very beginning of the design process of an automation system. The identified deficiencies are directly tied to the design specifications of DI&C elements and systems. Results suggest that integration of DI&C and HFE should put great consideration in the NUREG-0711 HFE element, functional requirement analysis and function allocation (FRA/FA).

DRO-ISG-2023-03 indicates that there are three HFE activities that should be addressed in all review plans for applications for a license under 10 CFR Part 53. They include FRA/FA, a staffing plan, and identification of important human actions. For applications under Framework A or Framework B, an FRA/FA would be required by 10 CFR 53.730(d). These analyses would be fundamental to understanding the role of plant personnel in accomplishing plant safety and emergency response functions.

2) The literature review was fairly limited in the analysis of the deficiencies in the system verification and validation element. This was due to limited information on system validation in the source documents of the literature and operational events. Several event reports indicated that the event could have been avoided had the deficiency been captured during the human factors validation. In principle, every human-automation accident or event is an indication that the system validation failed to grasp the automation failure.

One challenge to HFE validation during DI&C reviews is that the means for validating HFE of significant control room modifications is via NUREG-0711, Integrated System Validation (ISV)

activity. Given design development and testing schedules for significant control room modifications, the NRC staff may not have sufficient time to consider the results of ISV. One solution identified by the NRC staff to this scheduling challenge is the use of early-stage Multi-Stage Validation (MSV).

MSV incorporates successive, coordinated validation efforts performed at multiple points/periods during the development of design or modification [16, 17,18], Applications should discuss NUREG-0711 criteria for ISV testing that are applicable to the MSV program being used.

Future research is needed to study human-automation integration deficiencies in system validation and understand how an MSV approach that incorporates early HFE evaluation can support identification of automation failure modes.

3) The proposed framework with its taxonomy of DI&C failures, human cognitive failure modes, and human-automation integration deficiencies can enhance HFE process in the design and evaluation of DI&C and automation systems. The current HFE process described in NUREG-0711 focuses on ensuring the state-of-art human factors principles are incorporated into the design and implementation. In contrast, grasping automation failures requires risk-focused considerations of failure modes and failure causes.

4) Finally, we acknowledge that the work presented in this paper is preliminary and the scope of event review was limited. Over the last decade, DI&C systems have come into operation in many nuclear power plants, and many DI&C events have been documented. The availability of the information should yield more comprehensive and updated event analysis to enrich our understanding of DI&C and automation failures as well as the benefits of integrating of HFE and risk insights into the review and evaluation of new technologies with advanced automation capabilities.

5. References

[1] NRC, Final Safety Evaluation Report Related to Certification of the AP1000 Standard Design NUREG-1793, Supplement 2. U.S. Nuclear Regulatory Commission, Washington, DC, 2011.

[2] NRC, 2020. Standard Design Approval for the Nuscale Power Plant Based on the Nuscale Standard Plant Design Certification Application, ML20247J564, U.S. Nuclear Regulatory Commission, Washington DC, 2020.

[3] NRC News: NRC Authorizes Vogtle Unit 3 Fuel Loading and Operation. ML22215A210. U.S.

Nuclear Regulatory Commission, Washington, DC, August 3, 2022.

[4] NRC News: NRC Authorizes Vogtle Unit 4 Fuel Loading and Operation, U.S. Nuclear Regulatory Commission, Washington, DC, July 28, 2023.

[5] Georgia Power, https://www.georgiapower.com/company/news-center/2023-articles/vogtle-unit-3-goes-into-operation.html. July 31, 2023.

[6] NRC, Digital Instrumentation and Controls Licensing Process Interim Staff Guidance, DI&C-ISG-06, Revision 2. U.S. Nuclear Regulatory Commission. Washington, DC, 2020.

[7] NRC, Human Factors Engineering Program Review Model, NUREG-0711, Rev. 3, US Nuclear Regulatory Commission, Washington, DC, 2012.

[8] NRC, Guidance for the Review of Changes to Human Actions, NUREG-1764, Revision 1, US Nuclear Regulatory Commission, Washington, DC, 2007

[9] NRC, Proposed Rule: Risk-Informed, Technology-Inclusive Regulatory Framework for Advanced Reactors (RIN 3150-AK31). SECY-23-0021: ML21162A093, U.S. Nuclear Regulatory Commission. Washington, DC, 2023.

[10] NRC, Predecisional - Documents to Support Part 53 - Development of Scalable Human Factors Engineering Review Plans: Draft Interim Staff Guidance. DRO-ISG-2023-03. (2022). 10/18-19/2022 ACRS Public Meeting, ML22272A051, U.S. Nuclear Regulatory Commission. Washington, DC, 2022.

[11] G. Skraaning Jr., G. Jamieson, D. Armando, Q. Guanoluisa. Future Challenges in Human-Automation Interaction: Technology Trends and Operational Experiences from Other Industries., Halden Working Report HWR-1308. OECD Halden Reactor Project, Halden, Norway, 2020.

[12] Claire Taylor, Michael Hildebrandt, Robert McDonald, Niav Hughes. Operator Response to Failures of a Computerised Procedure System: Results from a Training Simulator Study, Halden Working Report HWR-1198, OECD Halden Reactor Project, Halden, Norway, 2017.

[13] G. Skraaning Jr, G.,Jamieson, The Failure to Grasp Automation Failure. Journal of Cognitive Engineering and Decision Making. https://doi.org/10.1177/15553434231189375, 2023.

[14[ NRC, Boeing 737 Crashes: Lessons Learned for NRC Digital Instrumentation and Controls Evaluation Process. U.S. Nuclear Regulatory Commission, Washington, DC (September 22, 2022.

[15] NRC, Human-System Interface Design Review Guidelines, NUREG-0700, Rev., U.S. Nuclear Regulatory Commission, Washington, DC, 2020.

[16] Nuclear Energy Agency, Multi-Stage Validation of Control Room Designs and Modifications, OECD Publishing, Paris, 2019.

[17] Institute of Electrical and Electronics Engineers. IEEE Guide for Human Factors Engineering for the Validation of System Designs and Integrated Systems Operations at Nuclear Facilities, IEEE Std. 2411-2021. Institute of Electrical and Electronics Engineers, Inc.

https://standards.ieee.org/ieee/2411/10357/ 2411-2021

[18] J.. Vazquez, B. Green, B. D. Desaulniers, Regulatory Considerations for the Potential Use of a Multi-Stage Validation Testing Approach to Support Human Factors Engineering Technical Reviews for Proposed Nuclear Power Plant Control Room Design Modifications.

In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 66, No.

1, pp. 1381-1385. Sage CA: Los Angeles, CA: SAGE Publications, 2022.