ML21076A421

NRC Human Reliability Methods

ML21076A421
Person / Time
Issue date:	03/30/2021
From:	Matthew Sunseri Advisory Committee on Reactor Safeguards
To:	Christopher Hanson NRC/Chairman
Lui, C, ACRS
Shared Package
ML21095A032	List: ML21076A421 ML21116A155
References
Download: ML21076A421 (9)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION ADVISORY COMMITTEE ON REACTOR SAFEGUARDS WASHINGTON, DC 20555 - 0001 March 30, 2021 The Honorable Christopher T. Hanson Chairman U.S. Nuclear Regulatory Commission Washington, DC 20555-0001

SUBJECT:

NRC HUMAN RELIABILITY METHODS

Dear Chairman Hanson:

In a November 8, 2006 Staff Requirements Memorandum (SRM), resulting from the October 20, 2006 meeting with the Advisory Committee on Reactor Safeguards (ACRS), the Commission directed us to "work with the staff and external stakeholders to evaluate the different Human Reliability models in an effort to propose either a single model for the agency to use or guidance on which model(s) should be used in specific circumstances."

In our letter of the following year, April 23, 2007, we stated that the staff and the Electric Power Research Institute (EPRI) were making progress in developing a plan to evaluate several human reliability analysis (HRA) models. By the time of our letter of May 14, 2014, much more information was available, and we reported that substantial progress had been made toward meeting the SRM objectives. The staff had decided that development of a single methodology, an Integrated Human Event Analysis System (IDHEAS), would be the best approach. However, work remained to refine the proposed methods and models into a form that could be used for practical HRA. We provided a summary of significant research accomplishments and detailed recommendations that should be addressed before the methodology is endorsed for use.

Over the intervening years, we have held a series of subcommittee meetings on draft and final reports covering the cognitive basis for HRA, a method for HRA of nuclear power plants at-power internal events, and early drafts of the IDHEAS general methodology, IDHEAS-G.

Specifically, we held open discussions with the staff and representatives of external stakeholders on April 24, 2015, and received internal status updates on the progress of IDHEAS-G on August 21, 2015, October 19, 2015, March 22, 2016, August 15, 2016, and May 1, 2017 as the staff completed various draft sections of the document. On September 18, 2019, our Reliability and Probabilistic Risk Assessment (PRA) Subcommittee held open discussions with the staff on the completed draft IDHEAS-G report. Our PRA Subcommittee further reviewed this matter during a meeting on September 23, 2020.

C. Hanson During the 682nd meeting of the ACRS, February 3-5, 2021, we reviewed three final reports:

• NUREG-2198, The General Methodology of an Integrated Human Event Analysis System (IDHEAS-G): a broad unified methodology that has been and will continue to be specialized (simplified and optimized) for specific applications;

• RIL-2020-13, Volume 1, Applying HRA to FLEXExpert Elicitation and Volume 2, Applying HRA to FLEXUsing IDHEAS-ECA: one of the applications derived from IDHEAS-G; and

• RIL-2020-02, Integrated Human Event Analysis System for Event and Condition Assessment (IDHEAS-ECA): a computer-assisted derived application.

We also reviewed one additional report that is not yet in final form:

• Draft RIL-2021-XX, Integrated Human Event Analysis System for Human Reliability Data (IDHEAS-DATA): it will provide the bases for important findings included in IDHEAS-G and the data for specific derived applications.

In addition, we had the benefit of the referenced documents.

CONCLUSIONS AND RECOMMENDATIONS

1. IDHEAS-G meets the primary intent of the 2006 Commission SRM, as a single HRA model for the agency to use.

2. The derived detailed application methods are expected to meet the intent of the Commission direction in the SRM for guidance on which model(s) should be used in specific circumstances.

3. IDHEAS-G should be issued after further editing to clarify presentation problems identified in the discussion that follows.

4. IDHEAS-FLEX Volumes 1 and 2 are not yet ready to be re-issued as a set of NUREG reports.

5. IDHEAS-DATA will be important to the general model but requires further work and peer review.

6. IDHEAS-ECA provides a specific derived application. It should be updated periodically to reflect user feedback and to synchronize with model and guidance refinements. Peer review is needed.

BACKGROUND In our letter of April 23, 2007, we summarized the history of HRA and the status of several NRC and EPRI methods and discussed the importance of operator actions to plant risk. We also mentioned an early HRA benchmark study and the HRA empirical studies that were intended to be part of a broader effort to collect evidence regarding the validity of HRA models.

C. Hanson By the time of our letter of May 14, 2014, the staff program to address the SRM had followed in the direction laid out in our 2007 letter and gone substantially further. After extended interactions with HRA experts and analysts and careful review of the HRA empirical studies, the staff had decided that development of a single methodology would be the best approach. The plan was to draw the best aspects of existing methods together, supplemented by extensive guidance on qualitative analysis, underpinned by a solid connection to the literature of psychology. We provided a summary of significant research accomplishments, citing NUREG-2114 on the psychological foundation of HRA as providing valuable information to improve understanding of the theoretical basis for human cognitive performance, the causes of human error, and a structured framework to assess the contribution to errors in the context of an evolving event scenario. It was later revised extensively and published under the title, Cognitive Basis for Human Reliability Analysis, in January 2016 (Cognitive Basis Document).

Our specific recommendations emphasized the need to develop detailed operational narratives and set expectations for other details of the methodology. Discussions during later subcommittee meetings emphasized the need for a generalized methodology, under which detailed and specialized methods could be developed. It would provide the appropriate linkage to the cognitive basis of HRA and lay out key elements of a quality analysis that should be maintained for all applications.

DISCUSSION IDHEAS-G AND THE SRM The 2006 Commission SRM includes two aspects: we should work with the staff and external stakeholders 1) to evaluate the different human reliability models in an effort to propose either a single model for the agency to use, or 2) to provide guidance on which models should be used in specific circumstances. We have followed staff efforts to develop a single model for the agency since 2006 and supported their plan to develop a single hybrid methodology. A number of complications became apparent, as the development continued and as the evolving method was applied to specific situations by multiple test teams. The result of this work, including a number of internal and external peer reviews, is a single general methodology called IDHEAS-G. We agree with the staff that IDHEAS-G meets the SRM direction for a single human reliability methodology.

The Fukushima event in 2011 and growing focus on new technologies forced the staff to think more broadly about HRA. They needed to consider events beyond nuclear power plant internal events at power, e.g., ex-control room actions, spent fuel pool activities, fuel storage and transportation, long-term waste disposal, mining operations, and medical applications.

IDHEAS-G is a human-centered (not technology-centered) general methodology and can support a wide range of human activities. The two major elements of IDHEAS-G are a cognition model of human performance and an HRA process that implements the cognition model. While, in principle, it could be used directly, the consideration of a very broad range of cognitive mechanisms and performance influencing factors (PIFs) in the cognition model would be burdensome. For specific applications, the cognition model can usually be simplified, and specializations of the quantification approach are possible. Therefore, the staff recommends using IDHEAS-G to develop application-specific HRA methods. The staff has developed two application-specific methods, IDHEAS for Nuclear Power Plant Internal Events at Power and IDHEAS-ECA. The first was released as a NUREG after being tested and evaluated; the second was released as a Research Information Letter (RIL) to demonstrate to the staff that it was possible to develop a computer-assisted method from IDHEAS-G to support risk-informed

C. Hanson decisionmaking. It was applied to HRA of Diverse and Flexible Coping Strategies (FLEX) activities in RIL-2020-13. Together, these application-specific methods show that the second criterion of the SRMto provide guidance on which models should be used in specific circumstancescan be realized for many different circumstances. Therefore, we consider that the goals of the SRM have been met, although some further refinement of the IDHEAS-G report is necessary, and the associated supporting and derivative reports and their methods need more extensive work.

IDHEAS-G IDHEAS-G presents the overall general framework that has been needed for providing a scientific underpinning to HRA models, a general structure that can be applied to any human activity, and an understanding of how to organize a coherent description and analysis of human actions in complex environments. It fills an important gap in the current state of HRA. It enhances the ability to analyze scenarios from the perspective of the cognitive and collaborative challenges they pose. The staff developed important links to the cognitive and behavioral literature in their Cognitive Basis Document and in IDHEAS-DATA. They received substantial help from a number of reviewers to integrate knowledge of the collaborative performance of teams and PRA-related ideas, such as modeling of available time, modeling dependency between human failure events, and treatment of uncertainty. It appears that the authors addressed most of the concerns raised by reviewers, trial users, and ourselves. With a few exceptions, the current version of the report is well-structured and organized to force analysts to consider the important issues in human performance.

The discussion of the cognition model in Chapters 2 and 3 is comprehensive and is supported by the Cognitive Basis Document. It is important information to understand, before attempting to use the HRA process of Chapter 4. The HRA process is straightforward, derived from the best of previous methods and includes four stages: scenario analysis, modeling of important human actions, human error probability quantification, and integrative analysis. The text thoroughly explains what is to be done in each stage. The emphasis on various types of context is an expansion and clarification of previous methods. The focus on qualitative analysis with the associated narratives is an improvement directed by the results of the international and U.S.

empirical studies performed to support the SRM-directed objectives. The report contains thirteen appendices that provide useful background information, guidance on selected issues, and illustrative examples. The staff discussed with us three sources of variability in HRA results: uncertainties in the scenario, analysts practices, and HRA methods. IDHEAS-G provides strong guidance to control the last two sources of variability. It also provides an approach to help control variability caused by uncertainties in the scenario, which can have major impact on the analysis. This comes in the way of searching for and identifying deviation scenariosboth changes in the physics assumptions of the PRA that can affect human performance (through a mismatch of procedures and training against the actual scenario) and changes in the progression as a result of particular human actions.

The supplemental report, IDHEAS-DATA, is expected to support many of the assumptions asserted in IDHEAS-G and will provide a source of data and information to assist in the quantification of IDHEAS-G derived models.

REMAINING ISSUES IN THE IDHEAS-G REPORT There is an issue of presentation in Chapter 2 Cognition ModelCognitive Basis Structure of the report that should be correctedeither by changing the text or by improving the

C. Hanson accompanying figures. We prefer the latter. In many places, the text claims that the associated figures show how elements (such as cognitive activities, processors, and cognitive mechanisms) are linked. The figures do not show such links. It appears that someone simplified the figures, but not the language. The Cognitive Basis Document actually does illustrate such linkages.

The report makes two strong assumptions: 1) there are three base PIFs that affect human error probabilities significantly more than other modification PIFs, and 2) a linear combination of PIF effects is appropriate for multiple PIFs. The basis for these two assumptions is lacking and must be provided, either in this report or in IDHEAS-DATA.

ISSUES IN THE SUPPORTING AND DERIVATIVE REPORTS The supporting and derivative reports have not yet been subjected to the rigorous peer reviews that were done for IDHEAS-G. Before they are issued as NUREG reports, peer review is essential.

IDHEAS-DATA promises to be an important element supporting IDHEAS-G and the derivative methods. However, it remains in rough condition, is difficult to parse, and does not yet provide the needed justification for the two strong assumptions discussed above. It brings together data developed from an extensive review of the literature and nuclear plant training data from the SACADA program and other sources. The data and report are currently under review by a national laboratory.

IDHEAS-ECA and IDHEAS-FLEX were both issued as RILs, as the staff explained, to get the methods out to the staff and industry trial users. Neither has been subjected to peer review, but licensee and owners group representatives participated in the FLEX trials. ECA has had favorable reviews by users. While issuing the reports quickly as RILs was a reasonable approach, they will have lasting value only if peer reviewed and revised to be consistent with other reports in the series.

IDHEAS-ECA. We have not performed a review of this report. After it is updated based on comments from trial users, we would like to review it. However, from initial discussions with the staff and demonstrations provided by them, we have some concerns. When an analysis that requires detailed understanding of the scenarios to be analyzed is automated, computer prompts should be provided to ensure the analyst is consciously making informed decisions consistent with the principles of IDHEAS-G. We are not sure how thoroughly such prompts have been implemented. One of the more creative tasks is the search to identify possible deviation scenarios that can change the PRA model and lead to more error-prone situations.

This and similar elements of the general methodology may have been short-circuited. Also, it appears that some data are embedded in the code; this needs to be examined to ensure it accounts for scenario-specific aspects of human performance. The data embedded in the computer code need to be justified and may require adjustments, after the data report is revised. Simplifications of the cognition model and new rules to help analysts move quickly need to be justified and reviewed.

IDHEAS-FLEX. There is no direct connection between Volumes 1 and 2 of the FLEX reports.

Rather than a comparison of using expert elicitation (Volume 1) and ECA (Volume 2) to model and quantify FLEX actions, the reports document work performed years apart and do not address the same situations. No direct comparisons are possible. There was an important

C. Hanson lesson learned from the analyses. It had been hoped that a limited set of crew failure modes and PIFs would be found for FLEX actions. That was not the case. The PIFs were found to be scenario-specific, rather than FLEX-specific.

The expert elicitation work in Volume 1 has a number of flaws. First, the method used to combine probability distributions is wrong. The report cites a staff White Paper on using expert elicitation. That paper is very good, but its recommendations are ignored. Substantial changes will be required if the report is to be reissued as a NUREG document. The RIL suggests that one either uses data or uses expert judgment. Actually, it is best to use both. First, there is no objectively appropriate data, some tempering with judgment is always needed. Second, if judgment is used in the absence of data, Bayesian updating is appropriate as data accumulate.

Again, we use both judgment and data, rather than either/or. There are methods to ensure that overly restrictive initial probability distributions (priors) are not over-riding the accumulating data.

There also seems to be a disconnect between Volume 1 and IDHEAS-G in that the cognition model language of IDHEAS-G (macrocognitive functions names) is not used. That also could be an issue of timing, but should be updated, if the reports are reissued. Many of the geometric mean calculations in Appendix D are incorrect, but rather than correcting them, the approach for combining distributions should be revised.

SUMMARY

The staff has completed a herculean task in assembling the cognitive basis for HRA and IDHEAS-G. IDHEAS-G has advanced the science and art of HRA in a number of ways: its cognition model is tied to and synthesized from the current cognitive and behavioral science literature; its HRA process has been adopted and expanded from the best aspects of previous methods tempered by the results of the international and U.S. empirical studies. IDHEAS-G and its derivative applications satisfy the goals of the 2006 Commission SRM.

With minor improvement, IDHEAS-G is ready for publication. IDHEAS-DATA, IDHEAS-ECA, and IDHEAS-FLEX require substantial improvement and peer review before final publication.

We look forward to continued interactions with the staff on those reports.

Sincerely, Matthew W. Sunseri Digitally signed by Matthew W. Sunseri Date: 2021.03.30 23:09:26 -04'00' Matthew W. Sunseri Chairman REFERENCES

1. U.S. Nuclear Regulatory Commission, M061020, Staff Requirements - Meeting with Advisory Committee on Reactor Safeguards, 2:30 p.m., Friday, October 20, 2006, Commissioners Conference Room, One White Flint North, Rockville, Maryland (Open to Public Attendance), November 8, 2006 (ML063120582).

2. Advisory Committee on Reactor Safeguards, Human Reliability Analysis Models, April 23, 2007 (ML071140297).

C. Hanson 3. Advisory Committee on Reactor Safeguards, Human Reliability Analysis Models, May 14, 2014 (ML14134A328).

4. U.S. Nuclear Regulatory Commission, NUREG/IA-0216, Volume 1, International HRA Empirical Study - Phase 1 Report: Description of Overall Approach and Pilot Phase Results from Comparing HRA Methods to Simulator Performance Data, November 2009 (ML093380283).

5. U.S. Nuclear Regulatory Commission, NUREG/IA-0216, Volume 2, International HRA Empirical Study - Phase 2 Report: Results from Comparing HRA Method Predictions to Simulator Data from SGTR Scenarios, August 2011 (ML11250A010).

6. U.S. Nuclear Regulatory Commission, NUREG/IA-0216, Volume 3, International HRA Empirical Study - Phase 3 Report: Results from Comparing HRA Methods Predictions to HAMMLAB Simulator Data on LOFW Scenarios, December 2014 (ML14358A254).

7. U.S. Nuclear Regulatory Commission, NUREG-2127, The International HRA Empirical Study: Lessons Learned from Comparing HRA Methods Predictions to HAMMLAB Simulator Data, August 2014 (ML14227A197).

8. U.S. Nuclear Regulatory Commission, NUREG-2156, The U.S. HRA Empirical Study:

Assessment of HRA Method Predictions against Operating Crew Performance on a U.S.

Nuclear Power Plant Simulator, June 2016 (ML16179A124).

9. U.S. Nuclear Regulatory Commission, NUREG-2114, Cognitive Basis for Human Reliability Analysis, January 2016 (ML16014A045).

10. U.S. Nuclear Regulatory Commission, White Paper: Practical Insights and Lessons Learned on Implementing Expert Elicitation, October 13, 2016 (ML16287A734).

11. U.S. Nuclear Regulatory Commission, NUREG-2199, Volume 1, An Integrated Human Event Analysis System (IDHEAS) for Nuclear Power Plant Internal Events at-Power Application, March 2017 (ML17073A041).

12. Liao, H., Morrow, S., Parry, G., Bley, D., Criscione, L., and Presley, M., Final Conclusions and Lessons Learned from Testing the Integrated Human Event Analysis System for Nuclear Power Plant Internal at-Power Application, 2017 International Topical Meeting on Probabilistic Safety Assessment and Analysis (PSA 2017), Pittsburgh, Pennsylvania, U.S.A., September 24-28, 2017 (ML21061A094).

13. U. S. Nuclear Regulatory Commission, NRCs High-Value Datasets: Human Reliability Analysis Data - Scenario Authoring, Characterization and Debriefing Application (SACADA), April 2019 (https://www.nrc.gov/data/).

14. Chang, Y.J., Preliminary Analysis Results of SACADA Data for HEPs, HRA Data Workshop, Rockville, Maryland, March 12-13, 2020 (ML20083J313).

15. U.S. Nuclear Regulatory Commission, Draft NUREG-2198, The General Methodology of an Integrated Human Event Analysis System (IDHEAS-G), August 2019 (ML19235A161).

C. Hanson 16. U.S. Nuclear Regulatory Commission, Draft NUREG-2198, The General Methodology of an Integrated Human Event Analysis System (IDHEAS-G), June 2020 (ML20238B988).

17. U.S. Nuclear Regulatory Commission, Draft Final NUREG-2198, The General Methodology of an Integrated Human Event Analysis System (IDHEAS-G),

November 2020 (ML20329A428).

18. U.S. Nuclear Regulatory Commission, RIL-2020-02, Research Information Letter - Integrated Human Event Analysis System for Event and Condition Assessment (IDHEAS-ECA), February 2020 (ML20016A481).

19. U.S. Nuclear Regulatory Commission, Draft Research Information Letter - Utilization of Expert Judgment to Support Human Reliability Analysis of Flexible Coping Strategies (FLEX), September 2020 (ML20245E458).

20. U.S. Nuclear Regulatory Commission, Draft Research Information Letter - Flexible Coping Strategies (FLEX) HRA Using IDHEAS-ECA, September 2020 (ML20245E459).

21. U.S. Nuclear Regulatory Commission, RIL-2020-13, Volume 1, Research Information Letter - Applying HRA to FLEX: Expert Elicitation, December 2020 (ML20345A318).

22. U.S. Nuclear Regulatory Commission, RIL-2020-13, Volume 2, Research Information Letter - Applying HRA to FLEX: Using IDHEAS-ECA, December 2020 (ML20345A319).

23. U.S. Nuclear Regulatory Commission, Draft RIL-2021-XX, Draft Research Information Letter - Integrated Human Event Analysis System for Human Reliability Data (IDHEAS-DATA), August 2020 (ML20238B982).

C. Hanson March 30, 2021

SUBJECT:

NRC HUMAN RELIABILITY METHODS Accession No: ML21076A421 Publicly Available (Y/N): _Y___ Sensitive (Y/N): N If Sensitive, which category?

Viewing Rights: NRC Users or ACRS only or See restricted distribution OFFICE ACRS SUNSI Review ACRS ACRS ACRS NAME CLui CLui LBurkhart SMoore (SWM) MSunseri DATE 3/17/21 3/17/21 3/17/21 3/29/21 3/29/21 OFFICIAL RECORD COPY