ML22070A156

Paper - PSA 2021 IDHEAS Suite

ML22070A156
Person / Time
Issue date:	03/11/2022
From:	Chang Y, Segarra J, Jing Xing NRC/RES/DRA/HFRB
To:
Xing, Jing - 301 415 2410
References
Download: ML22070A156 (8)

Text

IDHEAS SUITE FOR HUMAN RELIABILITY ANALYSIS Y. James Chang, Jing Xing, Jonathan DeJesus Segarra U.S. Nuclear Regulatory Commission Washington, DC 20555-0001, USA james.chang@nrc.gov, jing.xing@nrc.gov, jonathan.dejesus-segarra@nrc.gov

[Digital Object Identifier (DOI) placeholder]

ABSTRACT The U.S. Nuclear Regulatory Commission (NRC) developed the suite of Integrated Human Events Analysis System (IDHEAS) products. The IDHEAS suite aims to improve the technical basis and quality of human reliability analysis (HRA), including the estimates of human error probabilities in the NRC's risk-informed applications. The activities include conducting extensive literature analysis to provide the cognitive basis and data basis to support IDHEAS development, developing a general HRA methodology, and developing methods and software tools to assess human reliability for specific applications and domains.

This paper provides an overview of the NRCs IDHEAS development activities.

Key Words: Human reliability analysis, Cognitive model 1 INTRODUCTION The U.S. Nuclear Regulatory Commissions (NRCs) Reactor Oversight Process is used to monitor the operating commercial nuclear power plants' performance through inspections and risk assessments of events. The risk assessment results provide crucial input for the NRC to make regulatory decisions. The NRC's Significance Determination Process is the process to assess how the inspection findings (i.e.,

component and human performance deficiencies) affect the risk of a nuclear plant accident, either as a cause of the accident or the ability of plant safety systems or personnel to respond to the accident. This type of risk assessment is called event and condition assessment (ECA). Assessing human reliability in ECA has been challenging because the analysis incorporates the specific component and operator behaviors observed during inspections and applies the specifics to probabilistic risk assessment (PRA) to calculate the associated risk. In some cases, the NRC's SPAR-H [1, 2] human reliability analysis (HRA) method may lack the needed modeling details to address the specific human reliability considerations. In such situations, the options provided by SPAR-H can be either too optimistic or too conservative. The two options could lead to different regulatory decisions.

In 2006, the Commission issued a Staff Requirements Memorandum (SRM) [3] directing the NRC's Advisory Committee on Reactor Safeguards (ACRS) to work with the NRC staff and external stakeholders to evaluate the different human reliability models to propose either a single model for the NRC to use or guidance on which model(s) should be used in specific circumstances. The International HRA Benchmark Study results [4-7] and the Fukushima Daiichi accident in 2011 motivated the NRC to develop new HRA methods to address the Commission's SRM, which led to the development of the Integrated Human Event Analysis System (IDHEAS) for HRA.

2 IDHEAS 2.1 Overview of IDHEAS IDHEAS consists of a suite of products to provide the technical basis, methods, and tools for NRC's HRA applications. The NRC staff started with a large-scale literature review (NUREG-2114 [8]) to synthesize human cognition research into a cognitive basis framework for HRA. NUREG-2114 established a technological foundation to use five macrocognitive functions (MCFs) to represent human tasks. These MCFs are detecting and noticing, understanding and sensemaking, decisionmaking, action, and teamwork.

Assessing human performance requires assessing the five MCFs. The IDHEAS General Methodology (IDHEAS-G) [9] is an enhancement of NUREG-2114 and is designed to be a master methodology to provide all technical components needed to develop HRA methods for specific applications. The purposes of IDHEAS-G are to improve analyst-to-analyst consistency and HRA technical and data basis, as well as provide a common framework for developing HRA methods or tools for specific HRA applications. The Fukushima Daiichi accident occurred at early in IDHEAS-G development and drove IDHEAS-G to model human performance in severe operating conditions.

IDHEAS-ECA [10] is a method developed from IDHEAS-G for the NRC's ECA. The IDHEAS-At Power Method [11] was developed for at-power internal events applications. IDHEAS-DATA [12]

generalizes and documents human error data from various sources. IDHEAS-G presents a comprehensive set of the performance influencing factors (PIF) and their attributes (PIF Attributes). The MCFs, PIFs, and PIF Attributes (PIFAs) in IDHEAS-G provide a convenient structure to generalize human error data. The data provide a basis for the effects of PIFAs on human error probability (HEP).

2.2 IDHEAS-G IDHEAS-G aims to be a one-stop-shop, providing all technical components of an HRA process to develop application-specific HRA methods. Its cognitive model provides a structure to generalize and integrate human error data from various sources for HEP calculation. The IDHEAS-G report [9] has 13 appendixes. Each appendix provides detailed guidance for performing a technical component, such as scenario analysis, identification of important human actions, and task analysis. The following sections highlight some elements of IDHEAS-G.

2.2.1 Cognitive Model The IDHEAS-G HRA process is consistent with the HRA Good Practices [13], ATHEANA [14, 15]

and SHARP1 [16, 17]. The HRA process contains four stages of activities, including scenario analysis, modeling important human actions, estimating HEPs, and integrative analysis (e.g., dependency analysis and uncertainty analysis). IDHEAS-G uses the five MCFs to assess human reliability, including Detection, Understanding, Decisionmaking, Action Execution, and Interteam coordination. Each MCF is modeled by a three-level structure that, from top to bottom, includes MCF, cognitive processors, and cognitive mechanisms. For example, Detection is noticing cues or gathering information in the work environment.

Emphasized in this MCF are the processes that allow humans to perceive copious amounts of information and selectively focus on those pieces of information that are pertinent to the task being performed. Each MCF requires successfully completing a series of microcognitive processes (i.e., cognitive processors) to achieve the MCF's objectives. The reliability of the MCF is determined by the context of the MCF being performed. The context, in turn, is represented by the PIFs and PIF Attribures.

For example, Detection starts with a mental model of the object to be detected and the detection success criteria, followed by attending to the information sources, and perceiving and recognizing the information, etc. Each of these activities is a cognitive processor and is enabled by cognitive mechanisms. Attention, memory, and vigilance are examples of cognitive mechanisms. PIFAs specify the specific conditions that affect the reliability of cognitive mechanisms. The reliability propagates upward affecting the reliability of the cognitive processors then the reliability of MCFs. Therefore, the statuses of IDHEAS-G's PIFAs affect the performance of MCFs.

2.2.2 Human Error Probability and Performance Influencing Factors PRA uses human failure events (HFEs) as the basic units to represent important human actions that affect system reliability. IDHEAS-G provides guidance to systematically perform scenario analysis, identify important human actions, and perform task analysis to define the HFEs and analyze their context.

The results of these analyses establish the basis to calculate HEPs. IDHEAS-G recommends three approaches to estimate HEPs: data-driven calculation, expert judgment, and models. IDHEAS-G also develops its own HEP quantification model.

The IDHEAS-G HEP quantification model calculates an HEP in two parts: Pt and Pc. Pt is the error probability attributing to inadequate time available for completing the human action. Pc is the error probability attributing to failures of the MCFs. The overall HEP is the probabilistic sum of Pt and Pc. Pt is the convolution of the time required and time available distributions and represents the probability that the time required exceeds the time available. The time required and the time available are estimated for the HFE. IDHEAS-G calculates Pc as the probabilistic sum of the error probabilities of the actions critical task(s). The error probability of a critical task is the probabilistic sum of the failure probability of its MCFs, which, in turn, is a function of the PIFs and their aattributes. Therefore, each PIF aattribute affects the overall HEP. IDHEAS uses the term Macrocognitive Failure Mode (CFM) to indicate MCF failures. The probability of a CFM (PCFM) is the failure ptrobability of a MCF. The PCFM is calculated based on the statuses of the PIFs and their attributes defined in IDHEAS-G.

IDHEAS-G has 20 PIFs. These PIFs are in four context categories: environment and situation, system, personnel, and task. Each PIF has a set of attributes (PIFAs) to represent the PIFs' specific conditions affecting human reliability. The PIFs and PIFAs aim to be orthogonal in definition. Together, they provide a comprehensive set of conditions affecting human reliability. Each MCF has its own set of PIFs. Even under the same PIF, the same PIFA could have different effects on the PCFM of different CFMs. The following are examples of the Task Complexity's attributes that are unique to specific CFMs:

• Detection: detection demands great attention

• Understanding: ambiguity associated with assessing the situation

• Decisionmaking: conflicting considearations to prioritize the goals

• Action execution: no immediacy to initiate execution

• Inter-team coordination: complex or ambiguous authorization chain Each PIFA represents a specific performance condition that has a negative impact on human reliability.

The condition can either be measured or reasonably inferred. This property improves consistency between analysts. To apply IDHEAS, the analysts need to apply all the PIFAs in the task context category to calculate HEPs.

Equation 1 is the IDHEAS-Gs HEP quantification to a PCFM. IDHEAS-G distinguishes two different types of PIFs based on the types of their effects on human reliability: base PIFs and modification PIFs. The base PIFs include scenario familiarity, information completeness and reliability, and task complexity. The other 17 are modifier PIFs. Each PIF, regardless of base PIFs and modifier PIFs, has a few PIFAs to represent specific conditions affecting PCFM. The base PIFs determine the HEP anchor point, and the modifier PIFs shift the HEP away from the anchor point. For example, the PIF Task Complexity attribute "performing a straightforward procedure with many steps" has a base HEP of 1E-3. The attribute of the PIF Staffing "lack of backup and peer check" has a modification value of 110%, which is the Wj in Equation 1, and would increase the HEP by 10%.

The first two brackets of Equation 1 show how the base PIFAs and modifier PIFAs affecting HEPs are different from SPAR-H method in two ways. First, SPAR-H assigns a base HEP to each of its two cognitive types, diagnosis and action, while IDHEAS-Gs base HEP for a CFM is determined by the base PIFAs applied to the MCF. The PIFAs applied to a MCF depend on the scenario of the analysis. When more than one base PIFA applies to the MCF, the base HEP of the MCF is the probabilistic sum of the base HEPs of the base PIFAs. Second, SPAR-H uses multiplication to calculate the combined effects of its performance shaping factors, while IDHEAS-G uses linear addition for the combined effects of the modification PIFAs.

The technical basis of IDHEAS-G in modeling the combined effects of multiple PIFAs is summarized in

[18] and discussed in detail in the Appeindix D of IDHEAS-G report [9] .

1

= 1 (1 ) 1 + 1 [ ], > 0 (1)

=1 =1

Where, N is the total number of base PIFAs.

M is the total number of the modification PIFAs.

Wj is the weight of the PIFA-j, e.g., a W of 110% increases the HEP by 10%.

Re is a factor that accounts for the potential recovery from failure of a task, and it is set to one unless there are empirical data suggesting otherwise.

C is a factor that accounts for the interaction between PIFs, and it is set to one for linear combination of PIF impacts unless there are data suggesting otherwise.

If N equals zero, the base HEP is the lowest base HEP specified by IDHEAS-G. IDHEAS-G defines that the minimum base HEPs are performance in optimal teamwork conditions, and the performance time window is sufficient but not excessive.

2.2.3 Dependency The conventional dependency models [1, 19] use a set of factors to determine the dependency level to calculate the dependency effects on human reliability. The common factors are whether the tasks are performed by the same people, at the same location, close in time, and relying on the same cues. These factors provide a reasonable starting point to assess dependency levels. However, these factors provide little information on the reasons for dependency that can be tackled to improve performance. Therefore, there is little that the operating plants can do to reduce the dependence to improve human reliability.

IDHEAS-G has a new dependency model. The dependency model addresses dependency between consecutive HFEs. Reviewing operational experience and literature, IDHEAS-G developers identified three types of dependency:

• Consequential dependency: The outcome of an HFE directly affects subsequent HFEs.

• Resource-sharing dependency: Two HFEs share the same resource.

• Cognitive dependency: The outcome of an HFE introduces biased mindset or expectation on the performance of subsequent HFEs.

The NRC is working on guidance to assess these depdendency effects quantitatively.

2.2.4 Error Recovery and Uncertainty Treatment IDHEAS specifies the MCFs' base HEPs based on performing the tasks in a teamwork environment.

This provides additional opportunities to identify and correct errors before the errors' effects are irreversible compared to just relying on self-check. Therefore, IDHEAS has the PIFA "Lack of backup or lack of peer check or cross-checking" that reduces human reliability. On the basis of teamwork, the additional error recovery credits in IDHEAS-G have to be specific. An example is that IDHEAS-DATA includes nuclear power plant operating crew performance data in full-scope simulator training and experiments, e.g.,

SACADA [20] and HRA empirical studies [4, 5, 21]. These simulated scenarios generally last no longer than two hours. The operator could respond to a few system malfunctions in a scenario. The actual events with longer time windows could provide recovery opportunities that are not included in IDHEAS-DATA.

Another example is that the empirical maintenance data used in IDHEAS-DATA (e.g., [22, 23]) already includes the error recovery opportunities for routine maintenance work, such as peer-check and management task sign-off. Additional credits for error recovery need to be scenario-specific and are not in typical teamwork activities. The NRC plans to develop guidance on how to credit error recovery in IDHEAS.

IDHEAS uses the uncertainty framework [24] that classifies three types of uncertainty, including model uncertainty, parameter uncertainty, and completeness uncertainty, to discuss uncertainty in the HRA process. IDHEAS treats time uncertainty explicitly. It requires the analysts to explicitly specify the uncertainty distributions of the time required and the time available. Analysts need to be aware of the sources of uncertainty and address the uncertainties to provide more informed risk assessment results.

2.3 IDHEAS-DATA IDHEAS-DATA is an extensive human reliability data collection effort for the IDHEAS program.

IDHEAS-DATA documented data from more than 300 sources (documents) after filtering out a much larger set of potential sources. The data sources include nuclear and non-nuclear, empirical data, experimental data, expert elicitation results, and literature documenting performance statistics and performance ranking, etc. The data do not include HEPs generated by the other HRA methods. Operator performance data in full-scope nuclear power plant simulators, such as the HRA benchmark studies [4, 5, 21], HuREX [25, 26], and SACADA [20, 27], and nuclear power plant maintenance, e.g., [22, 23] are part of IDHEAS-DATA. Each datapoint is classified according to IDHEAS-G MCFs and PIFAs. A datapoint includes the information of the task of which the errors were measured, the MCFs of the task, the context or manipulations under which human error rates were measured, the correspond PIF, and the uncertainties in the reported human error measures.

The data in IDHEAS-DATA were used to develop IDHEAS-ECAs quantification technique for HEP estiation. An independent review on IDHEAS-DATA has been performed to verify the literature's human performance information is classified and applied correctly. The NRC continues to collect human reliability data for IDHEAS.

2.4 IDHEAS-ECA IDHEAS-ECA is an HRA method developed according to IDHEAS-G for performing ECA. In light of the U.S. nuclear industrys desire to credit the Diverse and Flexible Coping Strategies (FLEX) [28]

equipment in risk-informed decisionmaking [29], a specific requirement on IDHEAS-ECA development was to be able to assess FLEX actions' reliabilities.

IDHEAS-ECA uses the same qualitative analysis guidance (i.e., scenario analysis, identification of important human actions, and task analysis) as IDHEAS-G. In the quantitative portion, IDHEAS-ECA has the following: (1) uses the same five MCFs as IDHEAS-G (i.e., detection, understanding, decisionmaking, action execution, and inter-team coordination), (2) maintains the 20 PIFs but merging some PIFAs, (3) uses the IDHEAS-G HEP quantification model (discussed in section 2.2.2) to calculate HEPs, with the numeric values of the quation integrated from the human error data in IDHEAS-DATA. The NRC staff developed the IDHEAS-ECA Software Tool for calculating HEPs using the HEP quantification model and the human error data. The Software makes it easy to calculate HEPs, while HRA analysts still need to perform qualitative analysis as specified in IDHEAS-ECA before using the Software for HEP calculation. The IDHEAS-ECA method and software were tested in an NRC and Electric Power Research Institute joint workshop [30] and an NRC ECA [31].

3 CONCLUSIONS IDHEAS' contributions to HRA are stated well by the ACRS in a letter to the NRC Commission to complete the Commission's request "to evaluate the different human reliability models to propose either a

single model for the NRC to use or guidance on which model(s) should be used in specific circumstances." [32] The ACRS states:

The staff has completed a herculean task in assembling the cognitive basis for HRA and IDHEAS-G. IDHEAS-G has advanced the science and art of HRA in a number of ways: its cognition model is tied to and synthesized from the current cognitive and behavioral science literature; its HRA process has been adopted and expanded from the best aspects of previous methods tempered by the results of the international and U.S. empirical studies. IDHEAS-G and its derivative applications satisfy the goals of the 2006 Commission SRM.

There are remaining IDHEAS technical components (dependency and error recovery) to be developed. The NRC is finishing the dependency model development, planning to develop guidance on crediting error recovery, and improving the IDHEAS methods and software tools through data collection and user feedback.

4 REFERENCES

1. Gertman, D., H. Blackman, J. Marble, J. Byers, and C. Smith, The SPAR-H Human Reliability Analysis Method, 2005, NUREG/CR-6883 (ADAMS Accession No. ML051950061), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML0519/ML051950061.pdf.

2. Whaley, A.M., D.L. Kelly, R.L. Boring, and W.J. Galyean, SPAR-H Step-by-Step Guidance, 2011, INL/EXT-10-18533, Rev. 2 (ADAMS Accession No. ML112060305), Idaho National Laboratory, https://www.nrc.gov/docs/ML1120/ML112060305.pdf.

3. U.S. Nuclear Regulatory Commission, Staff Requirements - Meeting with Advisory Committee on Reactor Safeguards, 2:30 p.m., Friday, October 20, 2006, Commissioners Conference Room, One White Flint North, Rockville, Maryland (Open to Public Attendance), 2006, SRM M061020, U.S.

Nuclear Regulatory Commission, ADAMS Accession No. ML063120582 https://www.nrc.gov/reading-rm/doc-collections/commission/srm/meet/2006/m20061020.pdf.

4. Bye, A., E. Lois, V.N. Dang, G. Parry, J. Forester, S. Massaiu, R. Boring, P.Ø. Braarud, H. Broberg, J. Julius, I. Mnnist, and P. Nelson, International HRA Empirical Study - Phase 2 Report: Results from Comparing HRA Method Predictions to Simulator Data from SGTR Scenarios, 2011, NUREG/IA-0216, Vol. 2 (ADAMS Accession No. ML11250A010), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML1125/ML11250A010.pdf.

5. Dang, V.N., J. Forester, R. Boring, H. Broberg, S. Massaiu, J. Julius, I. Mnnist, H. Liao, P.

Nelson, E. Lois, and A. Bye, International HRA Empirical Study - Phase 3 Report: Results from Comparing HRA Methods Predictions to HAMMLAB Simulator Data on LOFW Scenarios, 2014, NUREG/IA-0216, Vol. 3 (ADAMS Accession No. ML14358A254), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML1435/ML14358A254.pdf.

6. Forester, J., V.N. Dang, A. Bye, E. Lois, S. Massaiu, H. Broberg, P.Ø. Braarud, R. Boring, I.

Mnnist, H. Liao, J. Julius, G. Parry, and P. Nelson, The International HRA Empirical Study:

Lessons Learned from Comparing HRA Methods Predictions to HAMMLAB Simulator Data, 2014, NUREG-2127 (ADAMS Accession No. ML14227A197), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML1422/ML14227A197.pdf.

7. Lois, E., V.N. Dang, J. Forester, H. Broberg, S. Massaiu, M. Hildebrandt, P.Ø. Braarud, G. Parry, J. Julius, R. Boring, I. Mnnist, and A. Bye, International HRA Empirical Study - Phase 1 Report:

Description of Overall Approach and Pilot Phase Results from Comparing HRA Methods to Simulator Performance Data, 2009, NUREG/IA-0216, Vol. 1 (ADAMS Accession No. ML093380283), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML0933/ML093380283.pdf.

8. Whaley, A.M., J. Xing, R.L. Boring, S.M.L. Hendrickson, J.C. Joe, K.L. Le Blanc, and S.L.

Morrow, Cognitive Basis for Human Reliability Analysis, 2016, NUREG-2114 (ADAMS Accession No. ML16014A045), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML1601/ML16014A045.pdf.

9. Xing, J., Y.J. Chang, and J. DeJesus, The General Methodology of an Integrated Human Event Analysis System (IDHEAS-G), 2021, NUREG-2198 U.S. Nuclear Regulatory Commission, ADAMS Accession No. ML21127A272.

10. Xing, J., Y.J. Chang, and J. DeJesus, Integrated Human Event Analysis System for Event and Condition Assessment (IDHEAS-ECA), 2020, RIL-2020-02, U.S. Nuclear Regulatory Commission, ADAMS Accession No. ML20016A481 https://www.nrc.gov/docs/ML2001/ML20016A481.pdf.

11. Xing, J., G. Parry, M. Presley, J. Forester, S. Hendrickson, and V. Dang, An Integrated Human Event Analysis System (IDHEAS) for Nuclear Power Plant Internal Events At-Power Application, Volume 1, 2017, NUREG-2199, U.S. Nuclear Regulatory Commission; Electric Power Research Institute, ADAMS Accession No. ML17073A041.

12. Xing, J., Y.J. Chang, and J. DeJesus, Draft - Integrated Human Event Analysis System for Human Reliability Data (IDHEAS-DATA), 2020, RIL-2021-XX U.S. Nuclear Regulatory Commission, ADAMS Accession No. ML20238B982.

13. Kolaczkowski, A., J. Forester, E. Lois, and S. Cooper, Good Practices for Implementing Human Reliability Analysis (HRA), 2005, NUREG-1792; ADAMS Accession No. ML051160213, U.S.

Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML0511/ML051160213.pdf.

14. Barriere, M., D. Bley, S. Cooper, J. Forester, A. Kolaczkowski, W. Luckas, G. Parry, A. Ramey-Smith, C. Thompson, D. Whitehead, and J. Wreathall, Technical Basis and Implementation Guidelines for A Technique for Human Event Analysis (ATHEANA), 2000, NUREG-1624, Rev. 1 (ADAMS Accession No. ML003736288), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML0037/ML003736288.html.

15. Forester, J., A. Kolaczkowski, S. Cooper, D. Bley, and E. Lois, ATHEANA User's Guide, 2007, NUREG-1880 (ADAMS Accession No. ML072130359), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML0721/ML072130359.pdf.

16. Wakefield, D.J., G.W. Parry, G.W. Hannaman, and A.J. Spurgin, SHARP1-A Revised Systematic Human Action Reliability Procedure, 1992, TR-101711, Tier 1, Electric Power Research Institute, https://www.epri.com/#/pages/product/TR-101711-T1/?lang=en-US.

17. Wakefield, D.J., G.W. Parry, G.W. Hannaman, A.J. Spurgin, and P. Moieni, SHARP1-A Revised Systematic Human Action Reliability Procedure, 1992, TR-101711, Tier 2, Electric Power Research Institute, https://www.epri.com/#/pages/product/TR-101711-T2/?lang=en-US.

18. Xing, J., Y.J. Chang, and N. Siu. Insights on human error probability from cognitive experiment literature. in 2015 International Topical Meeting on Probabilistic Safety Assessment and Analysis (PSA 2015). 2015.2015/04/26/30.

19. Lewis, S., S. Cooper, K. Hill, J. Julius, J. Grobbelaar, K. Kohlhepp, J. Forester, S. Hendrickson, E.

Collins, B. Hannaman, and M. Presley, EPRI/NRC-RES Fire Human Reliability Analysis Guidelines, 2012, EPRI 1023001/NUREG-1921 (ADAMS Accession No. ML12216A104),

Electric Power Research Institute and U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML1221/ML12216A104.pdf.

20. Chang, Y.J., D. Bley, L. Criscione, B. Kirwan, A. Mosleh, T. Madary, R. Nowell, R. Richards, E.M. Roth, S. Sieben, and A. Zoulis, The SACADA database for human reliability and human performance. Reliability Engineering & System Safety, 2014. 125: p. 117-133.https://doi.org/10.1016/j.ress.2013.07.014.

21. Forester, J., H. Liao, V.N. Dang, A. Bye, E. Lois, M. Presley, J. Marble, R. Nowell, H. Broberg, M. Hildebrandt, B. Halbert, and T. Morgan, The U.S. HRA Empirical Study - Assessment of HRA Method Predictions against Operating Crew Performance on a U.S. Nuclear Power Plant Simulator, 2016, NUREG-2156 (ADAMS Accession No. ML16179A124), U.S. Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML1617/ML16179A124.pdf.

22. Preischl, W. and M. Hellmich, Human error probabilities from operational experience of German nuclear power plants. Reliability Engineering & System Safety, 2013. 109: p. 150-159.https://doi.org/10.1016/j.ress.2012.08.004.

23. Preischl, W. and M. Hellmich, Human error probabilities from operational experience of German nuclear power plants, Part II. Reliability Engineering & System Safety, 2016. 148: p. 44-56.https://doi.org/10.1016/j.ress.2015.11.011.

24. Drouin, M., A. Gilbertson, G. Parry, J. Lehner, G. Martinez-Guridi, J. LaChance, and T. Wheeler, Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decisionmaking, 2017, NUREG-1855, Rev. 1 (ADAMS Accession No. ML17062A466), U.S.

Nuclear Regulatory Commission, https://www.nrc.gov/docs/ML1706/ML17062A466.pdf.

25. Jung, W., J. Park, Y. Kim, S.Y. Choi, and S. Kim, HuREX - A framework of HRA data collection from simulators in nuclear power plants. Reliability Engineering & System Safety, 2020. 194: p.

106235.http://www.sciencedirect.com/science/article/pii/S0951832017309857.

26. Kim, Y., P. Jinkyun, and M. Presley. Preliminary Findings from APR1400 HuREXdata collection and analysis. in HRA Data Workshop. 2020. Rockville Maryland, U.S. Nuclear Regulatory Commission.ADAMS Accession No. ML20083J306.

27. Chang, Y.J. Preliminary Analysis Results of SACADA Data for HEPs. in HRA Data Workshop.

2020. Rockville Maryland, March 12-13, 2020.ADAMS Accession No. ML20083J313.

28. Nuclear Energy Institute, Diverse and Flexible Coping Strategies (FLEX) Implementation Guide, 2015, NEI 12-06, Rev. 1, ADAMS Accession No. ML15244B006.

29. Nuclear Energy Institute, Crediting Mitigating Strategies in Risk-Informed Decision Making, 2016, NEI 16-06 Rev. 0, ADAMS Accession No. ML16286A297.

30. Cooper, S. and C. Franklin, Applying HRA to FLEX - Using IDHEAS-ECA, Volume 2, 2020, RIL 2020-13, U.S. Nuclear Regulatory Commission, ADAMS Assession No. ML21032A119.

31. U. S. Nuclear Regulatory Commission, Turkey Point Units 3 and 4 - Special Inspection Report 05000250/2020050 and 05000251/2020050 Dated December 9, 2020, 2020, ADAMS Accession No. ML20344A126.

32. Advisory Committee on Reactor Safeguards of Nuclear Regulatory Commission, NRC Human Reliability Methods, 2021, ADAMS Accession No. ML21076A421 https://www.nrc.gov/docs/ML2107/ML21076A421.pdf.