ML23292A047

Perceived Digital Control Room Usability and the Role of Individual Affinity for Technology

ML23292A047
Person / Time
Issue date:	09/25/2023
From:	Dickerson K, Grasso J, Natalee Green, Hildebrandt M, Hughes H, Wang I, Watkins H NRC/RES/DRA/HFRB
To:
References
Download: ML23292A047 (1)

Text

Perceived Digital Control Room Usability and the Role of Individual Affinity for Technology Kelly Dickerson1, Michael Hildebrandt2, Heather Watkins1, John Grasso1, Isaac Wang1, and Niav Hughes Green1 1

U.S. Nuclear Regulatory Commission, 2Institute for Energy Technology Halden Human Technology Organization Kelly.Dickerson@nrc.gov, Michael.Hildebrandt@ife.no, Heather.Watkins@nrc.gov, John.Grasso@nrc.gov, Isaac.Wang@nrc.gov, Niav.Hughes@nrc.gov Abstract Many nuclear power plants worldwide are operated from control rooms outfitted primarily with analog instrumentation and controls. Recently, there has been increasing focus on digital modernization, including the addition of digital instrumentation, sensors, and control systems. The U.S. Nuclear Regulatory Commission (NRC), and international regulators, are responsible for reviewing the safety of control room designs, which includes an assessment of how changes to control room human system interfaces (HSIs) are assessed for their usability and support of safe operations. Recent research has found that users with different levels of technology affinity perform differently on usability tests. The present study evaluates the potential of the affinity for technology interaction (ATI) survey on human factors research in the nuclear domain and if technology affinity is associated with subjective ratings of system usability. First, ATI data was collected from students and a group of control room operators, to establish a baseline. Then, ATI and system usability survey (SUS) data was collected from operators to determine if an association was present. The results address how regulators may benefit from new tools aimed at assessing how operators will cope with digital systems in the control room. Systematic probing of the construct of usability will aid regulators who are reviewing digital modernization applications but lack tools validated in a nuclear control room context.

1. Introduction The U.S. Nuclear Regulatory Commission (NRC) regulates the commercial use of nuclear power in the United States. Part of that regulatory role is to review and oversee the design, construction, operation, and decommissioning of commercial nuclear power plants. The review and oversight of commercial nuclear power plants includes examining the human factors engineering (HFE) processes used by licensees during the design and testing of their control rooms. The way these reviews are conducted is described in the NRC Standard Review Plan (NUREG-0800) [1], which provides an overview of all the guidance staff may use during the review process. Chapter 18 of [1]

specifically addresses HFE and references two primary guidance documents that are used by NRC staff and licensees that focus on HFE: 1) NUREG-0700, Human-System Interface (HSI) Design Review Guidelines[2], which contains control room design guidance and helps NRC staff determine if the HSI is designed in accordance with HFE principles 2) NUREG-0711, Human Factors Engineering Program Review Model,[3] which focuses on the design process, and contains the HFE criteria that helps NRC staff determine if an applicants HFE Program is likely to produce a good human factors design. [3] is also commonly used by licensees as they develop their HFE plans. [3]

also recommends that licensees have a plan for the assessment of the HFE of their control room designs and that these assessment techniques include consideration of operator workload and performance. The guidance includes information for the review of workstations containing analog or

computer based HSIs which might include reviewing an assessment usability of hand-held devices, for example.

Currently, the nuclear industry is pursuing modernization activities that involve transitioning control room instrumentation and controls (I&Cs) from fully analog to hybrid analog-digital, and with the advent of advanced reactors, future control rooms are expected to be partially or even fully digital.

There are several drivers for this kind of modernization, including availability and cost of analog components, the desire for improved efficiencies in operator performance, and the perception that digital displays will lead to better operator performance. However, digital control rooms can induce just as much cognitive demand as analog control rooms. [4] found that when fully analog and fully digital control room simulators were directly compared, global measures of workload indicated equivalent load between the two types of control rooms, but when reviewing workload subscales, it appeared that the sources of demand shifted depending on the type of control room. While observing shifting sources of demand, useful instruments like the NASA-TLX [5,6] are not able to provide an explanatory foundation for why this effect occurred. In other words, it could be something induced by the changes to the display of system information, or it could be related to operator attitudes about technology having a differential influence on perceived workload (along with many other individual-and system-related factors). The implications of some of these findings may need to be considered in the development of guidance for modern control room design and review.

1.1 Measuring Attitudes Towards Technology: The Affinity for Technology Interactions (ATI) Scale The ATI scale is a 9-item survey that was designed to measure a persons tendency to engage in intensive interactions with technology or avoid these kinds of interactions [7,8]. ATI represents a stable personality trait that reflects one's intrinsic motivation to actively engage in intensive technology interaction, which may manifest as a tendency to explore and understand systems. This trait is rooted in the construct of "need for cognition," indicating an individual's inherent drive to engage in cognitive activities that require thinking. High need for cognition suggests a propensity to seek, evaluate, and integrate multiple sources of information to make sense of and solve problems in one's surroundings. Similar to the assessment for need for cognition, the ATI scale evaluates individual differences in the inclination to approach situations and activities. While the need for cognition assessment examines differences in engaging in cognitively demanding tasks, the ATI scale captures variations in engaging in intensive technology interaction.

These individual attributional influences can be situated within the broader Technology Acceptance Model (TAM). The original TAM predicts the behavioral intention of adoption of a technological system or component based on perceived ease of use and usefulness [9]. Since its development, many variants and extensions of TAM have been developed. Including a recent extension for the nuclear domain, which focuses in part on the acceptance of a high degree of automation [10]. The nuclear power plant related extensions of TAM also posit that usefulness and ease are mediated by task-technology compatibility, trust, and perceived risk. Further, these mediators are potentially influenced by experience with similar technology [10]. However, [11] suggest that TAM is limited voluntary use environments. Where the use a given technology is mandatory, such as a nuclear power plant, technology acceptance may be more accurately reflected in metrics that reveal the extent of use. The acceptance of new technology in mandatory use environments has been more strongly associated with perceived usefulness [11,12,13,14]; a factor potentially influenced by an individuals general affinity for technology.

Research suggests that the ATI personality dimensions associated with more effective coping strategies for technology, including problem-solving and process learning [8]. Understanding this user attribute is important as it can skew study results, particularly those that have smaller participant samples. For example, users who score high in technology affinity may also tend to be people who enjoy actively engaging to understand technical systems and therefore are better able to cope with usability issues [15]. This means that a high affinity user could perceive a system to be more usable than a user with low technology affinity assessing the same system. There is also a suggestion that high affinity individuals are better able to adapt to changes in highly technical systems, meaning they require less training to reach proficiency and make fewer errors. [16] validated these perspectives empirically by looking at the potential link between the ATI and the system usability survey (SUS).

Their study measured performance, ATI, and SUS data from participants using four different types of interactive websites (web-based newspaper, a route planner, a shopping site, and a discussion forum). Meaningful correlations were found between the ATI and SUS for the online route planning and the shopping website, but not for the less interactive sites (newspaper and discussion forum),

suggesting that for interactive systems with high navigational demands the ATI may uncover users who are likely to provide high usability ratings. Furthermore, attitudes towards technology have also been found to influence the perceived usability of several other types of digital devices and tools, such as activity trackers [17], traceable AI systems [18], educational apps [19] and smart car interfaces [20].

Despite this awareness of the influence of attitudes towards technology in the consumer technology domain, there are no examples in the literature of the role that affinity for technology has on perceived system usability and operator performance for safety-related systems or systems with a safety-related components like nuclear power plants. However, there is broad research support for the notion that other individual differences in attitudes towards technology can predict performance outcomes. For example, researchers have found that dispositional trust, which refers to expectations of trustworthiness and is often measured as automation complacency potential, can significantly impact outcomes in human-automation interactions [21]. Findings demonstrate that detection of automation failures is significantly lower when individuals score higher on measures of complacency

[22,23,24]. Other research demonstrates that positive beliefs about automations performance, as indicated by the Perfect Automation Schema, are associated with increased sensitivity to automation errors [21]. Individuals with higher expectations of technology experience greater trust breakdowns when system errors violate their mental model of the technology. These situations can lead to unanticipated system usage behaviors (e.g., over-trust leading to misuse or under-trust leading to disuse), which become prominent in instances where uncertainty is high. Furthering our understanding of attitudes towards technology is necessary, as they can predict performance outcomes, which are crucial for safe operations.

1.2 The Impact of Examining New and Emerging Assessment Techniques

[3] lays out guidance for NRC staff to make a determination as to whether licensees have established a successful HFE program, including some information about assessments that should be conducted when examining the efficacy of control room designs for supporting safe operations. HFE programs usually include the following elements: task analysis and function allocation, documenting the treatment of important human actions, review of the HSI, procedure development, operating experience review, validation and verification, as well as training. For conventional large light water reactors with primarily analog control rooms, there is substantial operating experience that licensees can use in developing their HFE programs. Advanced reactors and SMRs do not similarly benefit from vast operational experience and therefore may need to rely on other methods for documenting

the safety of their designs. Given this, there is potential that new assessment methods may be needed to ensure that emerging technologies are at least as safe as the previous generation.

1.3 The Present Study The purpose of this study was to assess the usefulness of the ATI for application in the nuclear domain in the context of hybrid or fully digital control room environments. To that end, this study compares samples of university students to nuclear power plant control room operators. A comparison between students and control room operators was made because previous work on the ATI has focused only on consumer technology and general population users and there is a need to understand if such a lightweight measure of technology affinity could be used in a specialized group, such as nuclear power plant control room operators. One general concern about the ATI is that, for individuals who work in highly technical fields, affinity can be strongly (and potentially differently) influenced by personal and professional interactions with technology. This study examines that potential by using the original, unmodified English language ATI scale and a modified version of the scale which focuses only on questions that seem more relevant to professional interactions with technology.

2. Methods The ATI is quick to administer and well-suited to integrating into larger studies aimed at answering other types of technology related questions [18,25]. The ATI data for the present study were collected during two larger studies.

The first ATI data collection was conducted with university students who took part in a larger, survey-based, assessment aimed at understanding the factors that contribute to digital media overuse. The second ATI data collection was conducted during two separate simulation-based studies aimed at understanding performance, workload, and situation awareness of nuclear power plant operators.

The focus of this paper is on the use of the ATI for the nuclear domain and the students are used as a baseline comparison group. As such, the methods will focus only on ATI-related methodology and will not address the larger study contexts.

2.1 Participants Participants were recruited from two populations: university students and nuclear power plant operators. The operator sample was collected from three distinct locations, creating three subgroups of 12 operators each. These subgroups did not differ from one another in any of the measures taken and will not be discussed further. Five students and three operators were removed from the final sample due to failure to complete one or more sections of one or more survey. The final sample size was N = 72 (36 students and 36 operators).

2.2 Survey Administration and Modifications ATI data was collected twice for each participant within each participant group (students and control room operators). One administration was the unmodified 9-item English language ATI survey. The other administration was a modified version of the ATI which asked only questions that appeared highly relevant to control room operations. Table 1 provides the ATI question. The unmodified ATI was presented to participants with an instruction that set a personal use context. The modified ATI

was presented to participants with an instruction which established a professional/occupational use context (university, power plant control room).

A subset of 12 operator participants completed the SUS. The SUS was administered in its original form and was given to participants at the end of their testing session. An additional 12 operators completed a slightly modified version of the SUS, which was customized to better fit the sample, which had extensive experience with the control room design that was tested.

Table 1. Survey questions. Note that bolded ATI questions are the subset of items that were repeated for the professional use context (modified) ATI.

ATI Questions SUS Questions Modified SUS I enjoy tinkering with technical I think that I would like to use I enjoy using the system.

systems. this system frequently.

I enjoy testing the functions of I found the system I find the system new technical systems. unnecessarily complex. unnecessarily complex.

I primarily deal with technical I thought the system was I think the system is easy systems because I have to. easy to use. to use.

When I have a new technical I think that I would need the system in front of me, I try it out support of a technical person extensively. to be able to use this system.

I enjoy spending time I found the various functions I think the various functions becoming acquainted with new in this system were well of the system are well technical systems. integrated. integrated.

It is enough for me that a I thought there was too much I think there is too much technical system works; I dont inconsistency in this system. inconsistency in the care how or why. system.

I try to understand how a I would imagine that most I would imagine that most technical system works people would learn to use this people would learn to use exactly. system very quickly. the system very quickly.

It is enough for me to know the I found the system very I find the system very basic functions of a technical cumbersome to use. cumbersome to use.

system.

I try to make full use of the I felt confident using the I feel confident using the capabilities of a technical system. system. system.

I needed to learn a lot of I needed to learn a lot of things before I could get things before I could get going with this system. going with this system.

2.3 Design and Analysis There were three hypotheses this study aimed to examine. 1) Students will have greater affinity for technology than operators. 2) The context of the affinity survey (personal vs. occupational use) will influence the affinity ratings of both participant groups. 3) For the subset of operators who completed the system usability survey (SUS) and the modified SUS, there will be a positive correlation between affinity and perceived system usability, such that higher ATI scores are associated with higher SUS scores. Participants responded to each ATI question using a 6-point Likert scale with the following anchors: 1 = completely disagree, 2 = largely disagree, 3 = slightly disagree, 4 = slightly agree, 5 =

largely agree, and 6 = completely agree. Responses to the three negatively worded items (3, 6, and

8) were reverse coded.

Participant SUS scores were calculated using the approach reported in the literature [26,27]. For odd numbered items, 1 was subtracted from the raw score. For even numbered items, the raw score is subtracted from 5. The sum of these scores was then multiplied by 2.5 to produce the final SUS score. Generally, a score of greater than 68 is considered average usability across all types of digital systems.

Modified SUS scores were calculated using the same approach described by Brooke but with a minor adjustment applied to account for the removal of one of the items. Removal of one item would result in a roughly 10% reduction in the overall SUS score on average. To account for this and to generate a SUS score that could be compared to other studies, the sum of the individual items was increased by 10 percent. For example, if the total score is 28, the adjusted total is 30.80 and the final modified SUS score 77.

3. Results and Discussion Since this was the first (to our knowledge) use of the ATI in the nuclear domain, the primary interest was to examine if the sample of control room operators differed from a sample of college students since college students were one of the groups used by the ATI creators to validate the survey.

3.1 ATI Results There was a significant difference in technology affinity between participant types (students, operators) for both the personal (F(1,70) = 15.43, p < .001) and professional (F(1,70) = 44.76, p <

.001) use contexts. However, the difference between personal and professional technology affinity exhibited different patterns. Students rated their affinity for technology lower when given the classroom as the framing context when compared to personal use of technology (see table 2).

Conversely, operators rated their affinity for technology as lower when the framing context was their personal use, compared to their affinity for technology in the context of a nuclear power plant control room (table 2). The difference between use contexts was also smaller for operators than students, potentially suggesting that their attitude towards technology is not use-context dependent.

Table 2. Average (SE) scores for the Standard and modified ATI surveys.

Participant Type ATI Personal Use Context ATI Professional Use Context Operators 4.32 (.14) 4.41 (.13)

Students 3.61 (.12) 3.24 (.12) 3.2 SUS Results 3.2.1 Standard SUS Twelve nuclear power plant operators provided SUS ratings following their participation in a series of scenarios related to control room operations. The average SUS score was 83.96 (SE = 2.80). The range of SUS scores was 79.50-97.50. These scores are well above the reported literature average SUS score of 68 [27,28]. Pearsons correlation tests were used to determine if there was a correlation between operators SUS ratings and their affinity for technology. Neither correlation (Personal Use [r2

= .-059], Professional Use [r2 = -.156]) was significant p > .05, suggesting that for this sample of operators there was no association between technology affinity and perceived system usability as rated by the SUS.

3.2.2 Modified SUS Twelve nuclear power plant operators provided SUS ratings following their participation in a series of scenarios related to control room operations. The average adjusted SUS score was 68.52 (SE =

3.68). The range of SUS scores was 49.50-90.75. These scores are consistent with the reported literature average SUS score of 68 [27,28]. Pearsons correlation tests were used to determine if there was a correlation between the modified SUS scores and individual affinity for technology.

Neither correlation (Personal Use [r2 = .321], Professional Use [r2 = .136]) was significant p > .05, suggesting that for this sample of operators, there was no association between technology affinity and perceived system usability as rated by the modified SUS. This was also true for the unadjusted scores. The adjustment applied to account for the missing question was only one of several possible adjustment strategies. To ensure the correlation results were not influenced by the specific adjustment applied to the modified SUS scores (i.e., the 10% increase), comparison between the ATI and an unadjusted score, as well as an adjusted score which incorporated an average for the removed item from the unmodified SUS group, were examined. None of these additional comparisons were significant.

4. Conclusions This study is the first to our knowledge to use the ATI in the context of nuclear control room operations. The technologies used in a nuclear control room are distinct from those used in daily life in that they are complex and represent a range of digital and analog displays and tools. Given the distinct characteristics between the technology landscape of daily life and that of the control room, there was reason to believe that the standard ATI questions would be perceived as less applicable.

To address this, all participants completed two versions of the ATI, an unmodified version, where the instructions provided personal day-to-day use of technology as the context, and a modified version, where the professional environment (either school-related, or control room-related) was the framing context. Interestingly, students rated their professional affinity for technology lower than their personal affinity, and control room operators rated their professional affinity higher than their personal affinity for technology. These results suggest that additional research into other individual factors (e.g., age, gender, educational background) as well as technology interaction-related factors may be needed to determine what if any impact affinity for technology has in a modern control room environment.

4.1 Limitations and Next Steps Despite the findings reported in the literature, this preliminary study of the applicability of the ATI to nuclear control room environments did not replicate the association between the ATI and the SUS.

This could have been due to many different factors, including some related to practical limitations of the present study and some related to broader theoretical and methodological challenges.

At a practical level this study was limited by small sample sizes. The student and operator groups were roughly 30% of the smallest sample published by the ATI authors. Follow-up data collections are planned to increase the sample size, which also will aid in refining the statistical approach.

Studies with larger samples and a greater range of ATI scores will be needed to determine if affinity

for technology is a factor driving perceived usability. The operator sampling also presented a limitation. The operator sample was generated based on three different data collection activities, two occurred in real-world plant simulators and one in a laboratory setting at HAMMLAB. As a result, each operator subgroup had a distinctly different plant experience in general and had a different simulator experience during the study. While the three groups did not differ significantly from one another in terms of their overall ATI scores, it added variability and potentially limited the potential to detect an association between the ATI and the SUS (or modified SUS).

The finding that operators had a higher overall level of affinity compared to college students was somewhat surprising, but makes sense given the training and experiences of control room operators.

[8] reported that affinity for technology is grounded in the need for cognition. The need for cognition is defined as the seeking, evaluating, and integration of multiple sources of information for the purpose of making sense of the environment. Much of the operators job is using the HSIs to seek and integrate multiple sources of information while using their knowledge and problem-solving abilities to make sense of the indications and perform actions accurately and safely. Therefore, it is possible that higher affinity operators would be better able to cope with modernized control rooms than lower affinity operators. However, follow up studies that include a performance element would be needed to observe any potential differences between levels of affinity. Additionally, while the SUS did not reveal differences in perceived usability as a function of ATI score, this does not rule out that operators with high affinity may be biased towards rating a difficult to use system as usable because they are better able to cope with difficult or cumbersome technology. Performance and workload measures would be needed to further evaluate the impact of affinity driven bias on system evaluations that rely on operator inputs.

Ongoing modernization activities in the nuclear industry are changing the control room environment substantially. Increasing the amount and complexity of technology in the control room will necessarily increase the need for easy-to-use tools for assessing the human performance and safety implications of modernization activities. Research into the applicability and validity of novel assessment techniques from a regulators point of view will be critical in establishing good practices for evaluation of the safety of systems that are novel and have limited operational history to establish precedent for the applicability of existing approaches. Promising approaches identified by foundational research can be further examined to inform future updates to guidance, such as [3], thereby enhancing regulatory preparedness for the review of modern technologies in future licensing activities.

5. References

[1] U.S. Nuclear Regulatory Commission. Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants: LWR EditionHuman Factors Engineering, Chapter 18, NUREG-0800, Rev. 3, 2016, Washington, DC: US Nuclear Regulatory Commission.

[2] U.S. Nuclear Regulatory Commission, Human-System Interface Design Review Guidelines NUREG-0700, Rev.3, 2020. Washington, D.C.: U.S. Nuclear Regulatory Commission.

[3] U.S. Nuclear Regulatory Commission, Human Factors Engineering Program Review Model, NUREG-0711, Rev. 3, 2012, Washington, DC: US Nuclear Regulatory Commission.

[4] N. Hughes, L. Reinerman-Jones, J. Lin, G. Matthews, D. Barber, and K. Dickerson, Human Performance Test Facility (HPTF) Volume 2 - Comparing Operator Workload and

Performance Between Digitized and Anlog Simulated Environments, RIL 2022-11, 2023, U.S.

Nuclear Regulatory Commission.

[5] S.G. Hart and L. E. Staveland, Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, 1988, Advances in Psychology, Vol. 52, pp. 139-183, 1988.

[6] S.G. Hart, NASA-Task Load Index (NASA-TLX); 20 Years Later, Proceedings of the Human Factors and Ergonomics Society Annual Meeting Vol. 50, pp. 904-908, 2006.

[7] V. Heilala, R. Kelly, M. Saarela, P. Jskel, and T. Krkkinen, The Finnish Version of the Affinity for Technology Interaction (ATI) Scale: Psychometric Properties and an Examination of Gender Differences, International Journal of Human-Computer Interaction, 39, pp. 874-892, 2023.

[8] T. Franke, C. Attig, and D. Wessel, A Personal Resource for Technology Interaction:

Development and Validation of the Affinity for Technology Interaction (ATI) Scale, International Journal of Human-Computer Interaction, 35, pp. 456-467, 2019.

[9] D. Davis, Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly, 13, pp. 319-339, 1989.

[10] C. Kovesdi, Examining the Use of the Technology Acceptance Model for Adoption of Advanced Digital Technologies in Nuclear Power Plants Advances in Artificial Intelligence, Software and Systems Engineering, pp. 502-509, 2021.

[11] M. Ghazizadeh, J. D. Lee, and L. N. Boyle, Extending the Technology Acceptance Model to Assess Automation Cognition, Technology and Work, 14, pp. 39-49, 2012.

[12] V. Venkatesh, and F. D. Davis, A Theoretical Extension of the Technology Acceptance Model:

Four Longitudinal Field Studies, Management Science, 46, pp. 186-204, 2000.

[13] D. Gefen, E. Karahanna, and D. W. Straub, Inexperience and Experience with Online Stores:

The Importance of TAM and Trust, IEEE Transactions on Engineering Management, 50, pp.

307-321, 2003.

[14] E. Karahanna, R. Agarwal, and C. M. Angst, Reconceptualizing Compatibility Beliefs in Technology Acceptance Research, MIS quarterly, pp. 781-804, 2006.

[15] C. Attig, D. Wessel, and T. Franke, Assessing Personality Differences in Human-Technology Interaction: An Overview of Key Self-Report Scales to Predict Successful Interaction HCI International 2017 Posters' Extended Abstracts, Proceedings, Part I 19, pp. 19-29, 2017.

[16] C. Attig, D. Wessel, and T. Franke, Human-Technology Interaction from a Personal Resource Perspective: The Affinity for Technology Interaction (ATI) Scale Poster presentation at 51.

Conference DGPs-Kongress At: Frankfurt am Main, 2018.

[17] L. M. Hofbauer, and F. S. Rodriguez, How is the Usability of Commercial Activity Monitors Perceived by Older Adults and by Researchers? A Cross-Sectional Evaluation of Community-Living Individuals, BMJ open, 12, pp. e063135, 2022.

[18] T. Schrills, and T. Franke, How do Users Experience Traceability of AI Systems? Examining Subjective Information Processing Awareness in Automated Insulin Delivery (AID) Systems, ACM Transactions on Interactive Intelligent Systems, 2022.

[19] T. Winkler, D. Akyildiz, M. Herczeg, SpelLit: A Collaborative Body-and Space-Related Interactive Learning Game for School Children to Acquire Reading and Writing Skills EdMedia+ Innovate Learning, pp. 63-71, 2019.

[20] M. Avramidis, C. Kellner, J. Staudt, V. Zimmermann, and N. Gerber, Evaluating a Smart Car Interface in Terms of Usability, User Experience and User Acceptance, Mensch und Computer, 2021.

[21] S. M. Merritt, J. L. Unnerstall, D. Lee, and K. Huber, Measuring Individual Differences in the Perfect Automation Schema, Human Factors, 57, pp. 740-753, 2015.

[22] B. M. Muir, Trust Between Humans and Machines, and the Design of Decision Aids, International Journal of Man-Machine Studies, 27, pp. 527-539, 1987.

[23] R. Parasuraman, R. Molloy, and I. L. Singh, Performance Consequences of Automation-Induced 'Complacency', The International Journal of Aviation Psychology, 3, pp. 1-23, 1993.

[24] R. Parasuraman, and V. Riley, Humans and Automation: Use, Misuse, Disuse, Abuse, Human factors, 39, pp. 230-253, 1997.

[25] M. Babamiri, R. Heidarimoghadam, F. Ghasemi, L. Tapak, and A. Mortezapour, A. Insights into the Relationship Between Usability and Willingness to Use a Robot in the Future Workplaces: Studying the Mediating Role of Trust and the Moderating Roles of Age and STARA, PloS one, 17, pp. e0268942, 2022.

[26] J. Brook, SUS: A "Quick and Dirty" Usability Scale, Usability Evaluation in Industry, 1996.

[27] J. Brooke, SUS: A Retrospective, Journal of Usability Studies, 8, pp. 29-40, 2003.

[28] A. Bangor, P. Kortum, and J. Miller, Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale, Journal of Usability Studies, 4, pp. 114-123, 2009.