ML21116A155
| ML21116A155 | |
| Person / Time | |
|---|---|
| Issue date: | 04/29/2021 |
| From: | Raymond Furstenau Office of Nuclear Regulatory Research |
| To: | Matthew Sunseri Advisory Committee on Reactor Safeguards |
| Xing, J - 301 415 2410 | |
| Shared Package | |
| ML21095A032 | List: |
| References | |
| LTR-21-0109-1 | |
| Download: ML21116A155 (5) | |
Text
April 29, 2021 Mr. Matthew Sunseri, Chairman Advisory Committee on Reactor Safeguards U.S. Nuclear Regulatory Commission Washington, DC 20555-0001
SUBJECT:
U.S. NUCLEAR REGULATORY COMMISSION HUMAN RELIABILITY METHODS
Dear Mr. Sunseri:
On behalf of the U.S. Nuclear Regulatory Commission (NRC) staff, I am responding to your letter to Chairman Hanson dated March 30, 2021, Agency Document and Management System (ADAMS) Accession Number (ML21076A421). The letter provides your conclusions and recommendations on the staffs development of human reliability analysis (HRA) methods, referred to as Integrated Human Event Analysis System (IDHEAS), for the NRCs use in risk-informed applications. We value the Advisory Committee on Reactor Safeguardss review, evaluation, and useful perspectives and insights on the IDHEAS products.
The staff has taken into consideration your recommendations and has revised the document referenced in your letter, The General Methodology of Integrated Human Event Analysis System (IDHEAS-G), which is in publication as NUREG-2198 (ML20329A428). The staff also plans to incorporate your remaining recommendations in our other products.
The enclosure to this letter lists the staffs responses to your recommendations on the IDHEAS products.
Sincerely, Digitally signed by Raymond V. Raymond V. Furstenau Date: 2021.04.29 Furstenau 11:32:36 -04'00' Raymond V. Furstenau, Director Office of Nuclear Regulatory Research cc: Chairman Hanson Commissioner Baran Commissioner Caputo Commissioner Wright SECY
Mr. Sunseri 2 NRC STAFF RESPONSE TO THE ACRS RECOMMENDATIONS ON THE IDHEAS PROGRAM
- 1. REMAINING ISSUES IN THE IDHEAS-G REPORT
- 1) ACRS recommendation: There is an issue of presentation in Chapter 2 Cognition Model
- Cognitive Basis Structure of the report that should be corrected either by changing the text or by improving the accompanying figures. We prefer the latter. In many places, the text claims that the associated figures show how elements (such as cognitive activities, processors, and cognitive mechanisms) are linked. The figures do not show such links. It appears that someone simplified the figures but not the language. The Cognitive Basis Document actually does illustrate such linkages.
Staff Response The NRC staff simplified the figures in Chapter 2 because the number of links among the elements of each macrocognitive function was impractical to show in a figure. In response to this comment, the NRC staff revised the text in Sections 2.2, 2.3.1, 2.3.2, 2.3.3, 2.3.4, and 2.3.5 of the IDHEAS-G report.
- 2) The report makes two strong assumptions: (1) three base PIFs [Performance Influencing Factors] affect human error probabilities significantly more than other modification PIFs and (2) a linear combination of PIF effects is appropriate for multiple PIFs. The basis for these two assumptions is lacking and must be provided either in this report or in IDHEAS-DATA.
Staff Response The three base PIFs that significantly affect human performance are based on the signal detection theory. The NRC staff added a description of the signal detection theory to Section 4.4.3.2 of the IDHEAS-G report. The description connects the three main components that determine human responses to the base PIFs: information availability and reliability, scenario familiarity, and task complexity.
The basis for the linear combination of PIF effects will be described in the IDHEAS-DATA report.
- 2. IDHEAS-ECA RIL-2020-02 We have not performed a review of this report. After it is updated based on comments from trial users, we would like to review it. However, from initial discussions with the staff and demonstrations provided by them, we have some concerns. When an analysis that requires detailed understanding of the scenarios to be analyzed is automated, computer prompts should be provided to ensure the analyst is consciously making informed decisions consistent with the principles of IDHEAS-G.
Mr. Sunseri 3 Staff Response The IDHEAS-ECA RIL is out for public comment until July 31, 2021. The staff will subsequently update the RIL to a NUREG by incorporating public comments and will share the draft NUREG with the ACRS for review. Further, the staff will update the IDHEAS-ECA Software Tool and will add computer prompts, as appropriate, that integrate scenario and task analysis into the software.
- 3. IDHEAS-DATA IDHEAS-DATA promises to be an important element supporting IDHEAS-G and the derivative methods. However, it remains in rough condition, is difficult to parse, and does not yet provide the needed justification for the two strong assumptions discussed above. It brings together data developed from an extensive review of the literature and nuclear plant training data from the SACADA program and other sources. The data and report are currently under review by a national laboratory.
Staff Response The Pacific Northwest National Laboratory has completed an in-depth review of the data included in the draft IDHEAS-DATA report and has provided detailed critiques and recommendations. The NRC staff plans to update the data generalized in IDHEAS-DATA and revise the draft report based on these recommendations. This update will provide an enhanced justification for the two strong PIF effect assumptions mentioned by the ACRS.
The staff plans to solicit comments from the public on the revised report and share this revision with the ACRS for further review.
- 4. IDHEAS-FLEX
- 1) The method used to combine probability distributions is wrong. The report cites a staff White Paper on using expert elicitation. That paper is very good, but its recommendations are ignored. Substantial changes will be required if the report is to be reissued as a NUREG document. Many of the geometric mean calculations in Appendix D are incorrect, but rather than correcting them, the approach for combining distributions should be revised.
Staff Response The staff did not ignore the recommendations in the White Paper. Instead, the staff found them impossible to implement, because the experts were unable or unwilling to estimate the full probability distribution. They only estimated the most likely probabilities and the lower and upper bounds. Therefore, it was not possible to perform a combination of probability distributions. When probability distributions are not elicited, the experts estimates should be presented individually instead of combined. The staff plans to update and reissue the document to address the ACRS comments about the geometric mean calculations. At this time, the staff does not intend to reissue this document as a NUREG.
- 2) If judgment is used in the absence of data, Bayesian updating is appropriate as data accumulate. Again, we use both judgment and data rather than either/or. There are methods to ensure that overly restrictive initial probability distributions (priors) are not
Mr. Sunseri 4 over-riding the accumulating data.
Staff Response The staff agrees and will update the language in the report accordingly. Moreover, this comment is applicable to the staffs Expert Elicitation Guidance White Paper. The staff will address the comment when reissuing the white paper as a RIL or NUREG.
- 3) There also seems to be a disconnect between Volume 1 and IDHEAS-G in that the cognition model language of IDHEAS-G (macrocognitive functions names) is not used.
Staff Response The language in Volume 1 was purposely adapted to make it easier for the experts to understand. The report update will include a note on the connections between the terms used in Volume 1 and those used in IDHEAS-G.
ML21116A155 LTR-21-0109-1 OFFICE RES/DRA RES/DRA Tech Editor D:DRA D:RES NAME J. Xing S. Peters J. Zabel M. Thaggard R. Furstenau DATE 04/26/2021 04/26/2021 04/29/2021 04/26/2021 04/29/2021