ML20245E458

Volume 1: FLEX HRA Expert Elicitation

ML20245E458
Person / Time
Issue date:	09/01/2020
From:	Matthew Humberstone, Michelle Kichline, Jing Xing Office of Nuclear Reactor Regulation, Office of Nuclear Regulatory Research
To:
Carmen Franklin, Susan Cooper
Shared Package
ML20245E456	List: ML20245E457 ML20245E458 ML20245E459
References
Download: ML20245E458 (125)

Text

Research Information Letter (Draft)

DRAFT - Utilization of Expert Judgment to Support Human Reliability Analysis of Flexible Coping Strategies (FLEX)

VOLUME 1 Manuscript Completed:

Date Published: TBD Prepared by:

Jing Xing, Michelle Kichline, John Hughey, and Mathew Humberstone U.S. Nuclear Regulatory Commission

Disclaimer Legally binding regulatory requirements are stated only in laws; NRC regulations; licenses, including technical specifications; or orders. Although the NRC staff may suggest a course or action in a RIL, these suggestions are not legally binding, and the regulated community may use other approaches to satisfy regulatory requirements. Only unclassified information is published in this series.

ii

ABSTRACT Implementation of flexible coping strategies (FLEX) following the accident at Fukushima Dai-ichi resulted in the purchase of portable equipment (including diesel generators and diesel-driven pumps) specifically intended to support plant shutdown after extreme external events. However, much of the equipment may also be used as added defense in depth to mitigate the consequences of non-FLEX-designed accident scenarios (involving anticipated internal initiating events) in which installed plant equipment fails. Many nuclear power plants have considered using FLEX equipment during non-FLEX-designed accident scenarios and are taking credit for the additional equipment and mitigation strategies in their probabilistic risk analyses. The U.S. Nuclear Regulatory Commission (NRC) needs a method of human reliability analysis (HRA) capable of quantifying human error probabilities (HEPs) of FLEX types of actions (such as transportation, placement, connection, or local control of portable equipment) to support risk-informed license amendment requests, evaluations of notices of enforcement discretion, event evaluations, and significance determination process evaluations.

The NRC staff performed a formal expert elicitation with the purpose of using expert judgment to support the development of IDHEAS-ECA that can be used to quantify the HEPs associated with the use of FLEX equipment. The objectives of the expert elicitation were to (1) quantify the HEPs associated with a few typical strategies for using FLEX equipment for added defense in depth during non-FLEX-designed accident scenarios and during FLEX-designed accident scenarios, (2) develop a unique set of factors associated with FLEX strategies that influence performance, and (3) quantify the contribution of those performance-influencing factors on the HEPs. The expert elicitation used a structured process following the NRCs guidance document (Agency-wide Documents Access and Management System (ADAMS) Accession No. ML16287A734).

The panel consisted of six experts with expertise in HRA, implementation of FLEX strategies, and typical maintenance practices in nuclear power plants. This report presents the technical problems, the expert elicitation process, and the resulting expert judgments.

iii

iv EXECUTIVE

SUMMARY

Following the Fukushima Dai-ichi accident, the NRC issued orders requiring all U.S. nuclear power plants to implement mitigation strategies for coping without permanent electrical power sources for an indefinite amount of time. These mitigation strategies use a combination of currently installed equipment, additional portable equipment that is stored on-site, and equipment that can be transported from support centers. Implementation of flexible coping strategies (FLEX) resulted in the purchase of portable equipment (including diesel generators and diesel-driven pumps) specifically intended to support plant response after extreme external events. Yet, much of the equipment can also be used as added defense in depth to mitigate the consequences of non-FLEX-designed accident scenarios involving anticipated internal initiating events. Many nuclear power plants have considered using FLEX equipment during non-FLEX-designed accident scenarios and are taking credit for the additional equipment and mitigation strategies in their probabilistic risk analyses (PRAs).

The NRC maintains PRA models for each operating reactor in order provide independent assessment capabilities for various risk-informed activities, including license amendment requests, notice of enforcement discretion requests, event evaluations, and significance determination process (SDP) evaluations. Implementation of FLEX strategies largely involve human manual actions and decision-making. However, the current state of knowledge in human reliability analysis (HRA) does not fully support the use of FLEX equipment in PRA. Current HRA methods have limitations in quantifying human error probabilities (HEPs) for actions performed outside the control room such as the transportation, placement, connection, or local control of portable pumps and generators. Current HRA methods such as the Standardized Plant Analysis Risk-Human Reliability Analysis method (SPAR-H) used by the U.S. Nuclear Regulatory Commission (NRC) assume that these types of actions (which can only be generically outlined in procedures) are not feasible and cannot be credited in the PRA. The NRC needs an HRA method capable of quantifying the HEPs of the human actions associated with FLEX strategies in PRA models.

The NRCs approach is to develop a new HRA method that can quantify the HEPs associated with the use of portable FLEX equipment. The method is based on the NRCs HRA methodology, the General Methodology of an Integrated Human Event Analysis System (IDHEAS-G). The new method is referred to as the Integrated Human Event Analysis System for Event and Condition Assessment (IDHEAS-ECA). IDHEAS-ECA uses human error data of various sources for HEP quantification. At present, human performance data on FLEX human actions are rare, and there are limited, sparse data on human errors of actions similar to those in FLEX strategies. Thus, the NRC sponsored an expert elicitation project to use an expert panel to estimate benchmarking HEPs for a representative set of FLEX actions and identify the factors impacting the HEPs. The ultimate purposes of the project are to gain an understanding of human performance in implementing FLEX strategies and to use the expert judgment of the HEPs to benchmark IDHEAS-ECA for FLEX HRA applications..

An NRC project team consisting of staff from the Office of Nuclear Regulatory Research and Office of Nuclear Reactor Regulation managed the project. The expert elicitation used the NRCs expert elicitation guidance documented in an NRC white paper, Practical Insights and Lessons Learned on Implementing Expert Elicitation, 2016, ML16287A734. The NRC staff developed this guidance in response to the Commissions request for the use of formal expert judgments in risk-informed decision-making. The guidance has been used in several NRC expert elicitation activities for PRA. It emphasizes that expert judgment is the distribution (central tendency and v

the uncertainty range) of the overall technical communitys beliefs about the state of knowledge of the problem. To capture the state of knowledge, the project team disseminated extensive datasets relevant to the technical problems for the expert panel to evaluate and formulate the judgment. The expert panel consisted of three NRC staff members and three industry experts who are knowledgeable in PRA/HRA, implementation and audits of FLEX strategies, and human performance at nuclear power plants. The experts interacted through five teleconference meetings and one face-to-face workshop. This report describes the technical problems, the expert elicitation process, and the resulting expert judgment.

The objectives of the expert elicitation were to (1) estimate the HEPs associated with a few typical strategies for using FLEX equipment for added defense-in-depth during non-FLEX-designed accident scenarios and during FLEX-designed accident scenarios, (2) identify a unique set of factors associated with FLEX strategies that influence performance, and (3) assess the impact of those factors on HEPs. The NRC project team and the expert panel together developed two scenarios representing the two ways of using FLEX equipment. Scenario 1 is a non-FLEX-designed scenario in which the plant first loses offsite power then reaches a station blackout (SBO) without external hazards. It evolves in two parts. In the first part, one diesel generator (DG) is out of service, a loss of offsite power occurs, and there is a good chance that the second DG may fail. The plant may choose to use FLEX portable equipment without declaring an extended loss of AC power (ELAP). In the second part, the scenario progresses to the point that the plant loses the second DG and decides to declare an ELAP. Scenario 2 is a FLEX-designed scenario in which some external hazards cause a loss of offsite power and a loss of both diesel generators, resulting in an SBO and declaration of an ELAP. The contexts of both scenarios were designed to be very challenging for human actions. Time was not evaluated as a influencing factor in the expert elicitation. Both scenarios assumed that there was adequate time available to perform the required actions.

The expert panel quantified HEPs of the following five actions for both scenarios:

Action 1: transportation, connection, and local control of portable pumps Action 2: transportation, connection, and local control of portable generators Action 3: refilling water storage tanks using alternate water sources Action 4: ELAP declaration Action 5: deep load shed The project team and the expert panel developed specifications (e.g., the size of portable generator, the source of water for refilling) for these actions in the FLEX context. The expert panel estimated the HEP distributions of the actions for the given scenario context and action specifications. Along with the HEP estimates, the experts discussed their experience and understanding of the uncertainties, variation, and challenges in performing the FLEX actions. One lesson learned from the expert elicitation was that the HEPs are very sensitive to changes in scenario context or action specifications. Therefore, the HEPs estimated in the expert elicitation are only applicable to the actions and scenarios defined in this project.

Due to the challenging scenario context, the estimated HEPs of the FLEX actions appear high when compared to most HEPs for control room actions under normal emergency operating conditions. The experts defended their judgment using their understanding of the current status (circa 2018) of FLEX implementation. Although FLEX equipment had been purchased, strategies had been proceduralized, and implementation was audited and validated, there were still several factors that impact the likelihood of success of the strategies. Among those driving the HEPs were FLEX training requirements, scenario unfamiliarity due to lack of hands-on experience, and long-term management and maintenance of FLEX equipment and tools. In particular, the expert vi

panel had concern over FLEX training requirements. FLEX training, at that time, was not in the Systematic Approach Training (SAT) program so the training intervals were not well defined. The effectiveness of training varies as well. Personnel performing FLEX actions may or may not have hands-on training. The expert panel believed that they would reduce their HEP estimation if FLEX strategies are included in the SAT program. This effect was seen in the evaluation performed for Volume 2 of this report. In addition, there was a high level of uncertainty associated with the time required to implement FLEX strategies. Changes in the amount of time available to perform the actions can impact the associated HEPs significantly.

While the expert panel was not optimistic about the likelihood of success of FLEX strategies due to the challenging context, the panel believed that using FLEX equipment in non-FLEX-designed scenarios would help improve the training and scenario familiarity aspects associated with FLEX actions, therefore, reducing the associated HEPs. In addition, using FLEX equipment in non-FLEX-designed scenarios would help confirm the equipment is well-maintained and effective.

Finally, the experts indicated that reduction of core damage frequency in PRA models may not be the only measure of the benefit of FLEX strategies. Having FLEX strategies available enhances defense-in-depth and benefits plant safety regardless of whether it reduces core damage frequency. In an extreme hazard situation, having additional coping strategies available onsite improves the ability of the plant to respond to a wider range of events.

vii

TABLE OF CONTENTS Executive Summary ....................................................................................... v Chapter 1. Introduction .............................................................................. 1 1.1 Background ........................................................................................................................... 1 1.2 Purpose and Objectives ........................................................................................................ 1 1.3 Outcomes and Applicability ................................................................................................... 2 1.4 Overview of the expert elicitation process ............................................................................. 3 1.5 Related work ......................................................................................................................... 3 Chapter 2. Methodexpert elicitation process ....................................... 4 2.1 Basis for using expert judgment ............................................................................................ 4 2.2 Introduction to the NRCs White Paper Guidance for expert elicitation ................................ 5 2.3 Overview of the FLEX-HRA expert elicitation project ............................................................ 6 2.3.1 Problem statement.......................................................................................................... 6 2.3.2 Expected results of the study and how the results are to be used ................................. 6 2.3.3 Deliverables .................................................................................................................... 7 2.4 Expert elicitation process ...................................................................................................... 7 2.5 Summary .............................................................................................................................. 21 Chapter 3. Results .................................................................................... 22 3.1 Results of Objective 1: Estimation of HEPs of the FLEX actions ....................................... 22 3.1.1 Scenario definition ........................................................................................................ 22 3.1.2 Overview of the process of HEP estimation ................................................................. 31 3.1.3 Estimated HEPs and justifications ................................................................................ 31 3.2 Results of Objective 2 and Objective 3: Expert judgments of FLEX-specific PIFs ............. 60 3.3 Objective 3: Quantification of PIF effects on HEPs ............................................................ 63 Chapter 4. Discussion and lessons learned .......................................... 81 4.1 Insights on human reliability with FLEX strategies .............................................................. 81 4.2 Sanity check on the estimated HEPs .................................................................................. 83 4.3 Expert-to-expert variability in HEP judgments ..................................................................... 83 4.4 Topics regarding the expert elicitation process ................................................................... 84 4.5 Lessons-learned on implementing the White Paper Guidance ........................................... 86 4.6 Concluding remarks ............................................................................................................ 87 References ................................................................................................. 88 Appendix A. Project Plan for FLEX-HRA expert elicitation .................. 91 Appendix B. Annotation of example references used in the data packages .................................................................................................... 99 viii

Appendix C: workshop procedure ......................................................... 102 Appendix D. The estimated HEPs ......................................................... 105 ix

1 INTRODUCTION

1.1 Background

Implementation of flexible coping strategies (FLEX) following the accident at Fukushima Dai-ichi resulted in the purchase of portable equipment (including diesel generators and diesel-driven pumps) specifically intended to support plant shutdown after extreme external events [1,2].

However, much of the equipment can also be used as added defense-in-depth to mitigate the consequences of non-FLEX-designed accident scenarios (involving anticipated internal initiating events) in which installed plant equipment fails. Many nuclear power plants have considered using FLEX equipment during non-FLEX-designed accident scenarios and are taking credit for the additional equipment and mitigation strategies in their probabilistic risk analyses (PRAs).

However, in 2018, the state of knowledge in human reliability analysis (HRA) did not fully support the use of FLEX equipment in PRA.

The HRA methods were not developed to quantify the human error probabilities (HEPs) associated with the transportation, placement, connection, or local control of portable equipment.

HRA methods such as the Standardized Plant Analysis Risk-Human Reliability Analysis method (SPAR-H) [3] used by the U.S. Nuclear Regulatory Commission (NRC) assume that these types of actions (which can only be generically outlined in procedures) are not feasible and cannot be credited in the PRA. Research by the NRC staff and international nuclear community indicates limitations of existing HRA methods in addressing ex-control room human actions such as those associated with FLEX strategies ((4,5]. To support risk-informed license amendment requests, evaluations of notices of enforcement discretion, event evaluations, and significance determination process (SDP) evaluations, the NRC needed an HRA method capable of quantifying these types of HEPs in its SPAR models.

1.2 Purpose and Objectives The NRC sponsored an expert elicitation project for FLEX HRA. The purpose of the project was to gain the state of knowledge about human performance with FLEX so that the NRC can use the Integrated Human Event Analysis System for Event and Condition Assessment (IDHEAS-ECA) to quantify the HEPs associated with the use of portable FLEX equipment. IDHEAS-ECA is based on the NRCs new HRA methodology, the Integrated Human Event Analysis System - General Methodology (IDHEAS-G) [6]. IDHEAS-G defines a generic set of cognitive failure modes and performance influencing factors (PIFs), each characterized with many attributes. The FLEX-HRA expert elicitation project used an expert panel to estimate benchmarking HEPs for a representative set of FLEX actions under given scenarios and to describe the panels understanding of PIFs that are pertinent to FLEX strategies.

The project team determined the following three objectives of the expert elicitation project:

(1) Objective 1: Quantify the HEPs associated with a few typical human actions for using FLEX equipment for added defense in depth during a non-FLEX-designed scenario and a FLEX-designed scenario.

(2) Objective 2: Identify the unique PIF attributes associated with the use of FLEX equipment.

(3) Objective 3: Quantify the impact of the PIFs on HEPs.

1

Table 1-1 shows the scenarios and FLEX actions forming the basis for the expert panels estimation of the HEPs.

Table 1-1. FLEX human actions for HEP estimation Two scenarios used for HEP estimation:

Scenario 1This is a non-FLEX-designed scenario in which the plant reaches a station blackout (SBO) and loses important safety functions without external hazards. It evolves in two parts. In the first part (Scenario 1.1), one diesel generator (DG) is out of service, a loss of offsite power occurs, and there is a good chance that the second DG may fail. The plant may choose to use FLEX portable equipment without declaring an Extended Loss of AC Power (ELAP). In the second part (Scenario 1.2), the scenario progresses to the point that the plant loses the second DG and decides to declare an ELAP.

Scenario 2This is the FLEX-designed scenario in which some external hazards cause a loss of offsite power (LOOP) and loss of both DGs and, therefore, leads to SBO. The plant is likely to use FLEX strategies.

FLEX actions for HEP estimation:

The following seven actions were chosen for HEP estimates. Of these, the expert panel analyzed only the factors affecting the HEPs of Action 6 and Action 7 without estimating the HEP values.

Action 1: transportation, placement, connection, and local control of portable pumps Action 2: transportation, placement, connection, and local control of portable generators Action 3: refilling water storage tanks using alternate water sources Action 4: ELAP declaration Action 5: deep load shed Action 6: restoration of equipment from direct current (dc) load shedding Action 7: removal of debris 1.3 Outcomes and Applicability The outcomes of the expert elicitation included the following:

the definitions of a non-FLEX-designed scenario and a FLEX-designed scenario HEP distributions (center and bounds) of Actions 1-5 along with the experts justifications for their judgments experts analysis of Actions 6 and 7 a set of FLEX-specific PIFs along with the attributes quantitative effects of the PIFs on HEPs The HEP values apply to the scenarios defined and actions specified in this study. They may not be applicable to scenarios in which the assumptions and context are different from the ones defined for the scenarios in this study. Yet, the justifications associated with the HEPs help in understanding how the HEPs may change in different scenarios.

2

PIFs and attributes are a subset of IDHEAS-G PIFs selected by the project team. Yet, additional PIFs or attributes may be needed for event-specific scenarios involving FLEX strategies and for applications other than FLEX strategies.

1.4 Overview of the expert elicitation process This project used the NRCs expert elicitation guidance documented in an NRC white paper, Practical Insights and Lessons Learned on Implementing Expert Elicitation, 2016, ML16287A734

[7], referred to as White Paper Guidance in the rest of this report. The NRC staff developed this guidance in response to the Commissions request for the use of formal expert judgment in risk-informed decisionmaking [8]. The guidance has been piloted in four expert elicitation activities for PRA. It emphasizes that expert judgment is the distribution (central tendency and the uncertainty range) of the overall technical communitys beliefs about the state of knowledge of the problem.

To capture the state of knowledge, the project team disseminated extensive datasets relevant to the technical problems for the expert panel to evaluate and formulate the judgment. The expert panel consisted of three NRC staff members and three industry experts who are knowledgeable in PRA/HRA, implementation and audits of FLEX strategies, and maintenance practices at nuclear power plants. The experts interacted through five teleconferences and one face-to-face workshop.

1.5 Related work The NRC has developed IDHEAS-G methodology for HRA in all nuclear applications. IDHEAS-G provides a structure to quantify HEPs, but it does not supply application-specific numeric values for off-the-shelf HEP calculation. The outcomes of the expert elicitation support the use of IDHEAS-ECA for FLEX strategies. The estimated HEPs of the FLEX actions serve as anchoring HEP values, and the FLEX-specific PIFs informed the development of IDHEAS-ECA from IDHEAS-G.

3

2. MethodExpert Elicitation Process 2.1 Basis for using expert judgment Expert judgment is not a substitute for dataif it is possible to collect data to inform decisionmaking, then collecting data is the preferred option. Expert judgment should be considered only in the absence of sufficient experiential or experimental data from which to develop acceptable models.

Determining whether an expert elicitation is needed typically involves a choice among alternatives. The White Paper Guidance suggests these general criteria for deciding to perform an elicitation:

• There is a decision to be made or a decision support model to be developed.

Licensing review of the use of FLEX portable equipment involves modeling FLEX equipment and human actions. While data can support the reliability of FLEX equipment, systematic data are not collected on the reliability of human actions using FLEX equipment. Expert judgment is the only way to fill the gap.

• The currently available information base is sparse or uncertain.

FLEX strategies vary greatly from plant to plant. Information about human actions in FLEX strategies is available, but views range widely as to its applicability to human reliability. These differences in views can lead to a difference in results. Also, uncertainties in FLEX human actions are potentially significant, and more than one conceptual model can explain them.

• It is advantageous to perform an expert elicitation.

An alternative approach to FLEX HRA is to systematically collect operator performance data in FLEX training. However, FLEX training is administered every several years. It would take years to collect adequate data, and there is no established infrastructure for collecting such information. Thus, using expert judgment is quicker and costs less.

• There are domain experts in the technical areas of the problem, and knowledge about the problem is enough for experts to make technical judgments.

Since 2016, U.S. plants have been implementing, validating, and training personnel on FLEX strategies. There are experts among the plant operators, trainers, industrial specialists, and NRC staff members who audit or review FLEX implementation. Moreover, some senior plant operators and trainers have many years of experience with operator performance of actions outside the control room (ex-CR) that share common features with many FLEX actions.

While direct data on human errors in FLEX actions are sparse, indirectly relevant data are available in the following areas:

• ex-CR manual actions in plant equipment installation, maintenance, and repair from plants CR reports, which the NRC may be able to access

• literature on human error data in other fields (e.g., railroad, offshore oil, aviation)

• literature about the effects of various PIFs (e.g., environmental factors) on human error rates

• models of human error probabilities in nonnuclear domain manual actions 4

With a collection of indirect data and models, the expert elicitation will evaluate the available data, estimate a set of anchoring HEPs, and provide a basis for using PIFs in HRA methods.

2.2 Introduction to the NRCs White Paper Guidance for expert elicitation This expert elicitation followed the White Paper Guidance developed by the NRC staff in 2016.

The staff developed the paper for the agencys use in applying expert judgment in risk-informed decisionmaking. The guidance integrates the NRCs previous guidance documents on expert elicitation [9,10,11] and incorporates up-to-date research. The NRC has piloted it and used it in several PRA research projects.

The ultimate objective of conducting an expert elicitation is to appropriately represent the distribution (center, body, and range) of the overall technical communitys understanding of the problem. When expert judgment is used to support decisionmaking, the elicitation should ensure confidence in the results. For this reason, expert elicitation should conform to the following principles, regardless of the scale, level of effort, and method or procedures employed in the process:

• Representation of technical communityThe purpose of expert elicitation is not to create new knowledge, but rather to find the center, body, and range of the views of the technical community on the state of knowledge. While it is impractical to engage an entire technical community in the elicitation process, the expert panel should (1) be an adequate sample of the overall technical community, (2) have sufficiently broad knowledge that it can evaluate the available data, and (3) include leaders in the technical field who can capture the communitys degree of consensus and diversity. The resulting expert judgment should represent the overall communitys views and beliefs about the state of knowledge for the technical problem.

• Independent intellectual ownershipWhile the project sponsors have legal ownership of the project deliverables, the expert panel collectively has intellectual ownership of the results. Intellectual ownership means that the expert panel takes responsibility for the robustness and defensibility of the results. To ensure intellectual ownership, all inputs to the elicitation should be shared with every expert. To maintain the independence of intellectual ownership, expert judgment must be based on the experts knowledge and expertise, not the positions of the project sponsors or organizations with which the experts are associated. The panel members must clearly understand that they are not representing their employer or organization on the panel, but are serving in their own right as recognized leaders in their respective fields. To avoid (or mitigate) the risk of a group-think bias, each expert should also maintain independence from the other experts on the team.

• Avoidance of conflicts of interestTo minimize bias in the elicitation, the sponsors should carefully consider potential conflicts of interest before selecting the panel experts.

The experts should be free from direct and potential conflicts of interest to the extent practical. In all cases, experts should disclose potential conflicts of interest or even the appearance of conflicts of interest up front.

• Breadth of state of knowledgeTo obtain the range of knowledge and interpretations about the technical issue, the expert panel should evaluate a range of data and models that are representative of the overall technical community.

• Interaction and integrationTo represent the knowledge and interpretations of the technical community, experts should interact with each other as they accumulate and evaluate existing knowledge and make interpretations. The expert panel cannot simply collect and evaluate inputs from the literature or elicit the judgment of one or more 5

experts. Instead, individual experts should make their interpretations based on the integration of their own knowledge and inputs from other experts. The final results should be the integration of the individual judgments to represent the center, body, and range of knowledge about the technical issue.

• Structured processAn expert elicitation should employ a structured process to facilitate interaction and integration and reduce biases in the outcomes.

• TransparencyOften the results of an expert elicitation serve a range of users with different needs. To ensure that the results are used appropriately, the information generated must be documented in a transparent way. Transparency includes the input data, the models considered, the process used, the results obtained, and the caveats and limitations of the inputs, process, and results. Transparency also helps to demonstrate the stability and integrity of the results as a whole.

When formally eliciting expert judgment, a structured and systematic process should be used to encompass all of the basic principles described above. The White Paper Guidance recommends a systematic expert elicitation process, which is discussed in Section 2.4.

2.3 Overview of the FLEX-HRA expert elicitation project 2.3.1 Problem statement Providing quantitative credit to FLEX strategies requires knowledge of failure probabilities for equipment and human errors when operators perform manual actions outside CRs. The NRC needs an HRA tool for crediting operator manual actions in using portable FLEX equipment in FLEX-designed scenarios and non-FLEX-designed scenarios. IDHEAS-G has a quantification framework with a basic set of cognitive failure modes and a PIF model, yet the failure modes and PIFs are not specific to FLEX actions. Also, IDHEAS-G does not provide HEPs for the failure modes for combinations of PIF states and the quantitative effects of PIFs on HEPs. Therefore, the NRC wants to use elicit expert judgment on (1) a set of benchmarking HEPs for key FLEX actions in representative scenarios, (2) a set of PIFs that are specific for FLEX actions, and (3) the quantitative effects of the PIFs on HEPs. The agency then can use the expert judgment to support the development of IDHEAS-ECA for FLEX applications.

2.3.2 Expected results of the study and how the results are to be used The expected results of the study include the following:

• estimation of the HEPs of representative FLEX actions under a non-FLEX-designed and a FLEX-designed scenario

• a set of PIFs specific for FLEX actions across various scenarios and the guidance on how to assess the PIFs IDHEAS-G has a generic set of PIFs and the associated attributes that can negatively impact human performance; a PIF is assessed through the attributes. The expert panel will evaluate the PIF attributes for their relevance and importance to FLEX actions. The project team will then integrate the experts inputs into a FLEX-specific set of PIF attributes and define the PIF normal and poor states based on the significance of the attributes.

• changes to HEPs when PIFs change from nominal to poor states These results can be used in reviewing risk-informed licensing applications. The HEPs of the representative FLEX actions can serve as benchmarks for evaluating the HEPs of other FLEX 6

actions; the FLEX-specific PIF guidance and quantification allow HRA analysts to assess the HEPs of FLEX actions in the context of different scenarios. Overall, the results are to be used as input to the development of IDHEAS-ECA.

2.3.3 Deliverables The deliverables of this project include the following:

• a final report that documents the materials, process, and results of the expert elicitation, with results that include a set of benchmarking HEPs of the selected FLEX actions in the representative scenarios and the quantification of the PIFs relevant to FLEX scenarios

• the datasets, including the compiled data, models, and examples of human errors in tasks surrogating to FLEX actions 2.4 Expert elicitation process The White Paper Guidance recommends a 10-step expert elicitation process. The guidelines for each step implement the basic principles described in Section 2.2. Figure 2-1 is a diagram of the expert elicitation process.

Figure 2-1. Expert elicitation process A formal, structured expert elicitation should include all ten steps shown in Figure 2-1. Each of the steps of the expert elicitation process is briefly described below.

Step 1 Define the expert elicitationSufficiently define the technical problems to address the regulatory application of interest.

Step 2 Form the expert panelForm a panel that has adequate expertise to address the technical problems.

Step 3 Develop the project planDevelop a plan to ensure that the elicitation process is appropriate and the experts receive necessary information before the actual elicitation.

Step 4 Assemble and disseminate the datasetCompile the dataset to represent the overall state of knowledge about the technical problems.

Step 5 Become familiar with and refine the technical issues.

7

Step 6 Conduct training and pilotingEnsure that all of the team members understand the project, the technical problems, the individuals role and responsibilities, and the theories of probabilities and uncertainties.

Step 7 Conduct evaluation and elicitationThe expert panel interacts to evaluate the data and models, make interpretations, and form initial judgments through workshops.

Step 8 Integrate expert judgmentsIntegrate the judgments to represent the distribution of the views of the technical community.

Step 9 Documentation - Document the process and results and have the expert panel and the sponsor organization review the documentation.

Step 10 Conduct participatory peer review. This is not a separate phase but is intended to ensure that the entire expert elicitation process is conducted with participatory peer review in all steps.

The following describes each step in detail.

Step 1. Define the Expert Elicitation In defining the expert elicitation, the sponsors and the project team identify the objectives and scope of the elicitation process. Inputs from the expert panel refine the definitions. This step includes the following activities:

• Form the project team.

• Define the objectives of the expert elicitation.

• Define boundary conditions.

• Define the level of effort.

• Identify and define technical issues.

Form the project team. The project team is responsible for constructing and managing the entire expert elicitation process. The team should consist of project managers who direct the entire expert elicitation process and represent the project sponsors. The sponsors are the organizations that have legal ownership of the results. Sponsor representatives are typically the domain experts who will apply the results to their decisionmaking or models.

The project team for this elicitation consisted of one NRC research staff member, who is experienced in developing HRA methods and conducting expert elicitations, and three representatives of the project sponsor (the NRC Office of Nuclear Reactor Regulation). Several additional NRC staff members were frequently consulted.

Define the objectives of the expert elicitation. The objectives of the elicitation come from analysis or models that support decisionmaking. The objectives should be defined explicitly and reflect a clear understanding of how the judgments obtained will be used. The definition should then guide the entire elicitation process. Defining the objectives should include inputs from technical staff who are familiar with the specific information needs of the analysis or models that the technical problem addresses and the intended uses of the expert judgments.

The purpose of this expert elicitation is to provide a basis for performing FLEX HRA.

Specifically, the outcomes of the expert elicitation will allow the research staff to implement the IDHEAS method in FLEX applications, including the use of FLEX equipment in both non-FLEX-designed scenarios and FLEX-designed scenarios. The project team defined three objectives: (1) evaluating the PIFs from IDHEAS-G for their relevance and significance to FLEX strategies, (2) assessing the quantification effects of the PIFs on FLEX actions, and (3) estimating the HEPs of several FLEX actions in a range of scenarios that represent the two potential ways of using FLEX equipment.

8

Define boundary conditions. The boundary conditions for the expert judgment are the basic assumptions underlying the judgments about the technical issues of interest. The boundary conditions guide the identification of technical problems, the scope of data to be evaluated, and the expert panels understanding of the technical problems. The final report should clearly document boundary conditions to inform users of the assumptions underlying the judgments made.

The project team defined four boundary conditions:

(1) FLEX actions are feasibleThe FLEX strategies have been implemented and validated according to the FLEX implementation orders. Therefore, all the FLEX actions are feasible within the range of FLEX-designed scenarios.

(2) FLEX equipment is stored on site at the plantsEquipment may be pre-staged and need transport to the designated locations.

(3) Unless specified otherwise, all the actions are performed outside the main control rooms (MCRs). They may be performed inside or outside a shelter.

(4) The HEPs are estimated under the condition that there is adequate time margin to perform the actionsTime is an important factor affecting the human reliability of FLEX actions.

The utilities are required to demonstrate that all the FLEX actions can be performed within the available time. The IDHEAS method models two parts of HEPs for a human action:

(1) the error probability Pt attributed to uncertainties in time margin (i.e., the difference between the time available and the time needed to perform the action) and (2) the error probability Pc attributed to cognitive failures. Because the IDHEAS method already provides Pt calculation, this expert elicitation deals only with Pc estimation. That is, all the HEPs in this expert elicitation are estimated under the assumption that there is adequate time margin to perform the actions.

Define the level of effort. A formal expert elicitation can be expensive in cost, project management effort, and project duration. The determination of the level of effort is the balance between the technical demands and the available resources (funding, time, information overload, and project management burden). The most important factor to consider regard technical demands is the safety significance of the objectives and technical issues with respect to the intended use.

In determining the level of effort, the project team should recognize the pros and cons of different options; a bigger panel and more input is not always better. Aside from the limits on funding and time, the experts can become overloaded with the inputs from a large panel, and the interactive workshops can become unmanageable. Table 2-1 summarizes the level of effort in this expert elicitation.

9

Table 2-1. Level of effort indicators for designing the expert elicitation Components Level of effort Identification of technical

• Multiple meetings by the project team and problems sponsors Development of datasets

• Project team performs extensive literature study and develops data packages Key personnel

• 6 panel experts

• 2 resource experts

• 2 technical integrators

• 2 participatory peer reviewers Expert panel familiarization,

• Performed through electronic/remote tutorial training, and piloting sessions Workshops

• 5 telecom meetings

• 1 face-to-face, structured, interactive workshop Peer review

• Participatory peer review of the full elicitation process Identify and define technical issues. Based on the boundary conditions, the project team initially decomposed the broad objectives of the elicitation by clearly and precisely specifying more focused technical issues. The description of a technical issue includes the problem to be answered, the expected format of the answers, and any boundary conditions or assumptions about the problem.

The initial identification of the technical issues guides the selection of experts. The selected experts in turn work with the project team to refine the issues to ensure that they are unambiguously defined, directly support the objectives, and can be objectively assessed by the expert panel. The technical issues are further refined. Finally, the project team and experts should agree on a common set of technical issues. Still, some technical issues needed to be revised during this elicitations interactive workshop.

To use the experts most efficiently, it is important for the project team to narrow and refine the technical problem and boundary conditions as they break the objectives into technical issues.

Often, the panel cannot (and should not try to) address all potentially relevant issues. The project team should ensure that the experts focus on those issues really needing their judgment.

Ideally, the process would identify the HEPs for all the FLEX actions in a wide range of scenarios. However, the expert panel can handle only a limited number of technical problems.

The panel should be given adequate time at the workshop for in-depth interaction on each technical problem and a thorough exploration of the uncertainties in the problems. The project team determined three technical topics: (1) estimating the HEPs of seven FLEX actions in two scenarios (a non-FLEX-designed scenario and a FLEX-designed scenario), (2) evaluating 12 PIFs in the IDHEAS method for their relevance and significance to FLEX strategies, and 10

(3) assessing the quantification effects of the PIFs on the HEPs of FLEX actions. Table 2-2 summarizes the technical issues.

Table 2-2. Summary of the technical issues for expert elicitation Objective 1: HEP estimation The scenarios:

Scenario 1This is a non-FLEX-designed scenario in which the plant reaches a station blackout (SBO) and loses important safety functions without external hazards. It evolves in two parts. In the first part (Scenario 1.1), one diesel generator (DG) is out of service, a loss of offsite power occurs, and there is a good chance that the second DG may fail. The plant may choose to use FLEX portable equipment without declaring an Extended Loss of AC Power (ELAP). In the second part (Scenario 1.2), the scenario progresses to the point that the plant loses the second DG and decides to declare an ELAP.

Scenario 2This is the FLEX-designed scenario in which some external hazards cause a loss of offsite power (LOOP) loss of both DGs and, therefore, leads to SBO. The plant is likely to use FLEX strategies.

The seven actions for HEP estimation:

Action 1: transportation, placement, connection, and local control of portable pumps Action 2: transportation, placement, connection, and local control of portable generators Action 3: refilling water storage tanks using alternate water sources Action 4: ELAP declaration Action 5: deep load shed Action 6: restoration of equipment from dc load shedding (tentative)

Action 7: removal of debris Objectives 2 and 3: PIF evaluation and quantification The 12 PIFs for evaluation and quantification:

Environmental Factors (Note: this is the combination of five PIFs in IDHEAS-G: Work-Place Accessibility and Habitability, Visibility, Noise, Cold/Heat/Humidity, Resistance to Movement.)

Human-System Interface (HSI) (Indicators and Controls)

Tools and Parts Procedures, Guidance, and Instructions Training and Experience Teamwork and Organizational Factors Information Availability and Reliability Scenario Familiarity Multitasking, Interruption, and Distraction Task Complexity Mental Fatigue and Stress (Note: this is the combination of two PIFs in IDHEAS-G:

Mental Fatigue, Time Pressure and Stress.)

Physical Demands (Note: The above covers 17 of the 20 PIFs in IDHEAS-G. The remaining three PIFs, System and Instrument and Control Transparency, Staffing, and Work Process, are assumed to be nominal in the scenarios evaluated.

11

Step 2. Form the expert panel Experts. The experts are individuals who are at the forefront of a specialty relevant to the topic and are recognized by their peers as authorities because of their sustained and significant expertise on the topic. They serve as the primary subject matter experts who evaluate the data and make judgments. The responsibility of an expert is to make judgments about the technical issues of interest.

An expert typically has extensive knowledge and experience in more than one key technical area involved in the topic. The expert panel, as a whole, should be balanced across all of the key technical areas needed to address the technical issue of interest.

The expert panel for this project consisted of six experts with knowledge and experience in one or more of the following areas:

• PRA and HRA

• FLEX strategies

• operation of portable equipment

• nuclear power plant operation

• operator training and performance Resource specialists. Resource experts offer their technical knowledge to the expert panel but do not make judgments. Resource specialists should have a deep and broad knowledge of one or more key areas relevant to the technical issue of interest, as evidenced by years of experience working on the topic and recognition in the technical community as a subject matter expert. Their main responsibility is to share technical knowledge in an impartial way with the expert panel. For this elicitation, two resource specialists, who are on the NRC staff, shared with the panel their extensive knowledge of FLEX strategies and the NRCs risk-informed applications.

Technical integrators (TIs). These individuals lead the entire elicitation process. In particular, they are responsible for integrating the experts judgments to form the final results (i.e., the center, body, and range of the judgments on the technical issues) and resolving technical disagreements. TIs also have responsibility for challenging the technical basis of judgments made by the experts. TIs identify existing data, models, and methods, as well as alternative technical interpretations, and evaluate these in terms of their general quality and reliability and their specific applicability to the technical issue of interest. The TIs for this project have broad knowledge and experience in expert elicitation and HRA.

Participatory peer reviewers. They review the elicitation process and results and evaluate whether the project has met the objectives of the expert elicitation. Peer reviewers are denoted as participatory because they are expected to be involved throughout the entire project and interact with the project team and the experts at many stages. Their review includes determining whether the project is consistent with the basic principles of expert elicitation, whether it follows a formal elicitation process, and whether the technical assessment has been adequately defended and documented. Participatory peer reviewers help identify problems early so they can be corrected before the project ends. This project had two peer reviewers.

Step 3. Develop the project plan A sound project plan is needed because elicitations are often complex, resource intensive, and subject to strong schedule pressure because of their role in supporting decisions. The project team develops a project plan that describes the project objectives and all of the programmatic and technical activities in implementing the elicitation process, with clearly defined roles and tasks for all of the project participants. A project plan is the fundamental tool for documenting and communicating the specific elements and details of the study among the participants. It is also 12

used for the proper management and monitoring of a study to ensure that all procedural steps are followed. The project plan for this project includes the following:

• introduction and context of the study

• objectives of the study (i.e., a clear definition of the problem statement)

• project organization

• key tasks

• workshops

• deliverables

• risk identification and mitigation strategies

• the need for checkpoints throughout the process Appendix A presents the project plan and a one-page description of the project for quick communication with stakeholders.

Step 4. Assemble and disseminate datasets The goal of this step is to provide the expert panel with the most complete and up-to-date information that adequately represents available data on the technical issues. Identifying and compiling data are critical for experts to develop the judgments representing the range of views of the technical community. The project team initially compiles the data. As the elicitation process proceeds, the expert panel may recommend additional sources of information. The dataset is augmented based on data needs identified during the workshops. Where appropriate, data should be organized in formats that facilitate the experts use of the data to make judgments. The sources of data, the conditions under which the data were originally collected, and the caveats in the data should be documented. The process of developing the dataset should be clearly documented and avoid biases in the selection of data. The compiled dataset should be checked against the following criteria:

• RepresentativenessThe dataset should include the most important (e.g., high risk significance) and most recent data.

• BalanceThe dataset should balance the needs of the experts in different technical areas involved in the study.

• UsabilityThe dataset should be readily accessible and searchable by the experts.

Before the project, the NRC research staff assembled human error data from tasks that surrogate ex-CR actions. The data sources included (1) quantification of unsatisfactory task performance in nuclear power plant operator simulator training (as collected in the NRCs Scenario Authoring, Characterization, and Debriefing Applications (SACADA) database [12]), (2) human error rates in nuclear power plant operational tasks, as well as tasks in other domains (such as aviation, assembly industry, offshore operation), (3) human error rates of cognitive tasks in controlled experiments, and (4) quantitative effects of PIFs in the literature. These data play different roles in estimating HEPs.

Baseline HEPs or HEPs with known states of PIFs Some sources of data present statistical human error rates of certain types of tasks in various contexts and scenarios. Such data can inform the baseline HEPs for the cognitive failure modes (CFMs) applicable to the tasks. Below are two examples:

(1) Quantification of unsatisfactory task performance in nuclear power plant operator simulator training, as collected in the SACADA database. The SACADA database was built with the same macrocognitive model as that in IDHEAS-G. SACADA collects operator unsatisfactory task performance in different types of failures in various contexts.

13

The types of failures can be mapped to the detailed level CFMs in IDHEAS-G, and the context can be mapped to IDHEAS-G PIF attributes. Thus, the SACADA database can inform baseline HEPs of IDHEAS-G CFMs and the quantitative effects of some PIF attributes.

(2) The analysis of human errors in maintenance operations of German nuclear power plants.

Preischl and Hellmich [13,14] studied human error rates of various basic tasks in maintenance operations. Below are some example human error rates they reported:

• 1/490 for operating a circuit breaker in a switchgear cabinet under normal conditions

• 1/33 for connecting a cable between an external test facility and a control cabinet

• 1/36 for reassembly of component elements

• 1/7 for transporting fuel assemblies These error rates can inform base HEPs of the CFMs for action execution.

Quantification of PIF effects Some data sources present the changes in human error rates when varying one or more PIFs from a nominal to a poor state. Such data can inform the quantification of PIF effects on CFMs.

Two examples follow:

(1) NUREG/CR-5572, An Evaluation of the Effects of Local Control Station Design Configurations on Human Performance and Nuclear Power Plant Risk, issued September 1990 [15], estimated that the HEP = 2E-2 for ideal conditions and the HEP = 0.57 for challenging conditions with poor HSIs and distributed work locations.

(2) Prinzo et al. [16] analyzed aircraft pilot communication errors and found that the error rate increased nonlinearly with the complexity of the message communicated. The error rate was around 4 percent for the information complexity index of 4 (i.e., the number of messages transmitted per communication), 30 percent for the index of 12, and greater than 50 percent for indices greater than 20.

The significance of PIFs for certain types of tasks Studies in human error analysis and root causal analysis typically classify and rank the frequencies of various PIFs in reported human events. Some studies correlate PIFs with various types of human errors. Those studies only analyze the relative human error data without reporting how many times personnel performed the kind of tasks. The data from such studies cannot directly inform HEPs, but they can inform which PIFs or attributes are more relevant to the CFMs of the reported human errors. Below are two examples:

(1) Virovac et al. [17] analyzed human errors in airplane maintenance and found that the prevalent factors with frequent occurrence in human errors are communication (16 percent), equipment and tools (12 percent), work environment (12 percent), and complexity (6.5 percent).

(2) Kyriakidis et al. [18] analyzed railway accidents in the United Kingdom caused by human errors and calculated proportions of PIFs in the accidents. The authors reported that the most frequent PIFs in the accidents were safety culture (19 percent),

familiarity (15 percent), and distraction (13 percent).

The preceding examples are just a few in a large body of human error data the project team had documented. To avoid overloading the experts, the project team members disseminated the 14

dataset and compiled two data packages for the expert panel. One data package contained the HEPs or human error rates from nuclear power plants and surrogates, such as the offshore oil industry, assembly industries, and nuclear waste process. The other package included data on the quantitative effects of PIFs on human error rates, classified by the 12 PIFs. In both packages, the sources of data, the conditions under which the data were originally collected, and the caveats in the data were documented. Appendix B lists the reference sources of the data included in the packages.

Step 5. Become familiar with and refine technical issues It is essential for the experts to have a clear, precise, and thorough understanding of the technical issues and boundary conditions. The complexity and uncertainties in the questions being studied may lead the experts to different understandings of the issues. The project team and TIs should ensure that all of the experts share the same understanding of the technical issues, including the following:

• the problems asked

• the intended use of the results

• the assumptions and boundary conditions of the issue The project team and TIs interacted with the experts to achieve a common understanding of the technical issues through a series of tele-meetings and e-mail communications. They used strategies such as probing or feedback to ensure the common understanding. The experts were encouraged to ask what-if questions on the technical issues. The experts also challenged the information provided and asked for clarification or more specification of the issues. These interactions among the project team and the experts on the panel led to refinements in the definitions of the scenarios and FLEX actions.

A major caveat in this step was inadequate familiarization with the issues because of the limitations of telecommunication and the great variation and uncertainty in the technical issues.

During the face-to-face workshop, the project team and the experts made an extra effort to clarify and specify the assumptions and boundary conditions of the technical issues. The project team also asked the experts to document their additional assumptions on their final worksheets.

Step 6. Conduct training and piloting The project team conducted multiple training sessions during the five teleconferences and at the beginning of the workshop. The training was to familiarize the experts and provide them with practice in the following areas:

• the subject matter and the technical problems being asked, including the necessary background information on why the elicitation is being performed and how the results will be used

• the basic principles of elicitation, the elicitation process, beliefs and subjective probabilities, and calibration of uncertainties

• possible biases that could influence the judgments and introductions to de-biasing strategies

• workshop purposes, procedures, worksheets, and good practices for workshops

• practices in formally articulating judgments, as well as explicitly identifying associated assumptions, rationale, and factors contributing to uncertainties The White Paper Guidance also recommends piloting following the training. Piloting is a test before introducing the project in its full scope. It is a small-scale preliminary study conducted to 15

evaluate feasibility, time, cost, and potentially adverse events and improve the studys design before performance of a full-scale project. The piloting could reveal areas that need further training. It could also lead the project team to refine the technical issues, elicitation procedure, or the elicitation worksheet. Yet, because of the time limitation, the project team planned but did not actually conduct the piloting other than a brief exercise during a teleconference.

Step 7. Conduct evaluation and elicitation Workshops To elicit expert judgments, workshops are the critical platform for interaction among the expert panel members. The White Paper Guidance recommends three stages of workshops:

(1) evaluation workshop for evaluating the available data and models (2) elicitation workshop for eliciting individual judgments (3) integration workshop for integrating the expert judgments (see Step 8)

Each workshop serves a specific function in the elicitation process, and the interactions among the experts evolve as the process moves from evaluation to elicitation and, finally, to integration.

Evaluation workshop. This elicitation process used two teleconferences in lieu of the evaluation workshop. During the teleconferences, the project team introduced the two data packages to the experts. The expert panel evaluated the datasets relevant to the technical issue. The outputs of the evaluation workshop included the experts understanding and interpretations of the available data and identification of data gaps and significant issues for which the dataset should be expanded. However, the teleconferences did not give the experts enough time to thoroughly discuss the uncertainties, limitations, and caveats in the data. The experts mostly evaluated the data packages individually.

Elicitation workshop. A face-to-face workshop was held at the NRC Headquarters for experts to interact on their judgments of the technical issues. The workshop followed the process outlined below:

(1) Before the workshop, the experts used their worksheets to make an initial assessment of the technical issues.

(2) During the workshop, the experts discussed each technical issue and their judgments, along with assumptions and justifications, with other panel members.

(3) After the workshop, the experts reviewed their worksheets and workshop notes, documented additional assumptions and justifications, and modified their judgments as needed on their final worksheets.

Worksheets The project team developed worksheets for the experts to use in assessing the technical issues.

The team developed three sets of worksheets, one for each objective. The team first developed draft worksheets and then discussed them with the expert panel. The final worksheets were the result of several iterations.

Worksheet 1: Estimation of HEPs. This worksheet was for the experts to use in estimating the HEPs of the FLEX actions. The team asked the experts to estimate the HEPs of five FLEX actions in the non-FLEX-designed and FLEX-designed scenarios. The experts were also asked to discuss two actions, restoration of equipment from ELAP in the non-FLEX-designed scenario and debris removal in the FLEX-designed scenario, without estimating their HEPs. Worksheet 1 has 12 tables, one for each action in a given scenario. Each table contains the action definition, specifications, and task description. The table provides spaces for the experts to document the HEPs and justifications. The experts were asked to do the following:

16

• Estimate HEP distribution at the 1st, 99th, and 50th percentiles.

• Annotate the justifications or basis for the estimate.

• Note additional assumptions and specifications on the scenario and action used for the estimate.

Worksheet 2: Evaluation of PIFs. This worksheet was for the experts to use in evaluating PIFs relevant to FLEX strategies. A table represents each PIF and contains the definition of the PIF and its attributes. The experts were asked to do the following:

• Evaluate the applicability and significance of the impact of every attribute on FLEX actions.

• Propose additional attributes, if any, that can significantly affect FLEX actions.

• Make suggestions for revising, merging, or eliminating PIFs and their attributes.

Worksheet 3. Quantification of PIF effects. This worksheet was for the experts to use in quantifying PIF effects on HEPs, including the definition of PIF states and quantitative changes to HEPs when a PIF degrades from its nominal state to a poor state.

Based on the experts inputs from Worksheet 1 and the data package on PIF effects, the project team generalized four states for each PIF: nominal, low, moderate, and high. The definition of a poor PIF state is a group of PIF attributes based on their impacts on HEPs. Worksheet 3 consists of 12 tables, one for each PIF. Each PIF table contains the PIF description and definitions of the nominal and the attributes of the poor states. The experts were asked to do the following:

• Verify the definitions of PIF states, modify the definitions as needed, and recommend operational examples of the PIF states.

• Evaluate the data package of PIF effects and estimate the HEP range that a PIF alone can impact.

Workshop procedure The face-to-face workshop followed the procedure outlined below:

(1) The workshop began with a clear definition of its goals, an explanation of the process to be followed, a statement of the experts non-conflict-of-interest, and a definition of the roles of all those attending.

(2) The TIs conducted onsite training and led an exercise on representing judgments in probabilities and calibrating uncertainty bounds.

(3) The workshop proceeded to HEP estimation (Worksheet 1) and PIF quantification (Worksheet 3). The panel first discussed and refined the two scenarios under which the HEPs of the FLEX actions were to be estimated. The panel then worked on individual technical issues:

• The TIs presented the issue.

• The expert panel and resource specialists discussed their understanding of the issue, refined the definition as needed, and proposed and agreed on additional assumptions and specifications.

• The experts took turns presenting and defending their estimations and justifications, while the TIs, other experts, and resource specialists discussed and 17

challenged the presenters judgment. The TIs facilitated the discussion and queried the experts on uncertainties in the technical issue.

• Every expert had a chance to discuss his or her views without the pressure of reaching consensus with other experts judgments.

• During and after each experts presentation and discussion of a technical issue, TIs asked whether the expert would like to revise or clarify his or her judgments based on the discussion; the TIs stressed that the rationale for revisions should be carefully documented.

• The two participatory peer reviewers mainly interacted with the technical integrators on their concerns and suggestions during the meeting breaks and at the end of the day. They also expressed their suggestions during the workshop sessions when they observed things that needed to be corrected immediately.

(4) The project team determined the ground rules of interaction as a part of the workshop procedures. The general ground rules for the workshop included the following:

• All the experts should present and be queried uniformly and be asked to provide specific answers to questions about the technical issues and the reasoning behind their responses.

• The workshop should focus on elicitation of experts judgments and should not move into integration of judgments. Each expert should have the opportunity to discuss his or her views without the pressure of reaching consensus with other experts judgments.

• All key definitions and assumptions about the technical issues should be reviewed.

• The importance of active listening and having the experts defend other points of view was emphasized throughout the workshop.

The experts were informed with a written workshop procedure (Appendix C) before the meeting, and the technical integrator briefed and discussed the procedure at the beginning of the meeting to ensure that all the participants understood their roles and responsibilities.

Key roles of workshop participants

• The TIs presented the workshop objectives and technical issues, facilitated consideration and discussion of uncertainties in the technical issues, fostered experts challenges to others judgments, and resolved technical disagreement. The TIs also ensured that the workshop procedures were followed, the experts used the worksheets as intended, and consistently enforced the ground rules of interaction. While the TIs frequently queried the experts and sought confirmation of the results from the experts, they guarded the workshop against attempting to force consensus or influence the outcome.

• The experts presented and defended their initial judgments and revised their judgments based on discussions. They also challenged other experts judgments. The experts also made specific contributions in refining the two scenarios and further specifying FLEX actions.

• One resource specialist participated in the whole workshop. He provided information to the expert panel in several areas of expertise such as the FLEX implementation orders and observations of plants FLEX drills. The resource specialist also helped in refining the 18

scenarios.

Step 8. Integrate expert judgments Individual judgments, collectively, produce insights about the topic being studied. Yet, the use of expert elicitation results for decisionmaking requires the integration of multiple expert judgments into a single metric. Integration may involve simply processing individual judgments through mathematical means such as smoothing, interpolation, or aggregation. This expert elicitation required aggregation of the HEP distributions and quantification of PIF effects, as well as consolidation of the experts justifications for each technical issue. Because of the variation in FLEX implementation and uncertainties in the technical issues, the justifications are critical for the proper use of the HEP and PIF quantification.

The White Paper Guidance recommends that integration be accomplished in two parts:

(1) TIs develop the distribution (center, body, and range) of the views that represent the informed technical community.

(2) The project team conducts an integration workshop to evaluate the preliminary results and uses expert feedback to finalize the results.

This elicitation used teleconferences with individual experts and e-mail communications to achieve the purpose of the integration workshop.

TI integration TIs integrated the experts judgments through the following activities:

• They deliberated and determined ways of integrating the judgments of the expert panel based on the nature of the judgments and the intended use.

• They resolved challenging issues in integration, such as treating alternative or conflicting viewpoints or incorporating uncertainties.

• They performed the integration to generate the distribution.

• They performed a sanity check by applying the preliminary integration of results to their intended use for decisionmaking. This check provided insights into the process used for integration and demonstrated the effect that disparate views would have on the final result.

The integration process employed the following practices:

• All the experts inputs were equally weighted, and none of the inputs were treated as an outlier.

• Since the HEP estimation was made only to the center and the bounds of the distribution, the TIs did not attempt to fit the estimation to any probability distribution.

• The TIs used both arithmetic mean and geometric mean to aggregate the experts HEP estimates. The arithmetic means are reported as the final results. The geometric means are documented in Appendix D for potential future reference.

• The individual experts inputs on the scenario context, HEP justifications and uncertainties, and PIF effects were preserved and documented, particularly the disparate views among the experts and the impact of disparate views on the final consolidated judgments.

19

Integration Workshop Through teleconferences and group e-mail discussions (in lieu of a face-to-face workshop),

the TIs, experts, project team, and peer reviewers evaluated the preliminarily integrated results and used the feedback to finalize the judgments. This stage of integration included the following activities:

(1) The TIs e-mailed the preliminary results to the experts with particular emphasis on the way in which alternative viewpoints and uncertainties were incorporated.

(2) The expert panel questioned and probed aspects of the preliminary results to understand the way in which the views of the larger technical community were considered and the range of technically defensible interpretations included.

(3) The expert panel discussed significant contributors to uncertainty, potentially significant biases in the results, and constraints on the results.

The feedback generated helped to ensure that no significant issues were overlooked and allowed for a comprehensive understanding of the implications of the results and uncertainties. The TI team should use the feedback to finalize the results, including the integrated judgments, constraints on use, and implications that disparate views would have for the final results.

Step 9. Document the process and results The project team documented its work throughout the project. The final documentation of a formal expert elicitation should indicate what was done, why, and by whom. The final report includes the following:

• The elicitation process usedThe project report describes in detail how the process was conducted and explains selections when there were multiple alternatives (e.g., the justification for the level of effort, combination or omission of some steps, the approach for combining individual judgments). The report documents the activities, workshops, participants, schedules, and organizational structure used to achieve the project deliverables.

• The dataset usedIt is important for the reader of a project report to fully understand what data were considered at the time of the study and how the expert panel used those data in its deliberations. Chapter 2 presents an overview of the project datasets and references. The full data packages used in the expert elicitation are documented in the NRCs Agencywide Documents Access and Management System (ADAMS).

• Key assumptions used in the elicitation.

• The technical issues and the resulting judgments, along with the reasoning supporting these judgments.

• A discussion of how the results will be used for decisionmaking and how the process used provides confidence that the objectives of the elicitation were met.

The report also includes the factors that might have introduced some biases to the results or imposed some limitations on the use of the results, the caveats in the elicitation process, and lessons learned from the process.

Step 10. Conduct participatory peer review Step 10, the participatory peer review, is not actually a last step. Instead, the peer review runs throughout the entire elicitation process. Its purpose is to provide full, frequent, and 20

independent input into the process, so that concerns can be addressed and corrections made before the project is complete.

The peer reviewers evaluate the draft report and provide comments. These final comments should present the reviewers final evaluation of whether the TI team has considered the overall technical communitys viewpoints and made a concerted attempt to capture the center, body, and range of technically defensible views. The reviewers comments should also address their final assessment of the elicitation process used in the project and whether that process meets the objectives of the expert elicitation. The final project report should include the reviewers assessment.

For this elicitation, the project team selected two peer reviewers before the first teleconference.

They provided timely review and feedback at each phase of the process or periodically following key project milestones. Their input included verbal and written comments following the teleconferences and the workshop.

2.5 Summary In summary, the FLEX-HRA expert elicitation implemented the basic principles and the process described in the White Paper Guidance. Following the guidance provides regulatory assurance to the project outcomes. This expert elicitation also adopted the flexibility of the White Paper Guidance to balance the level of effort and the resources available, such as combining individuals roles and using teleconferences instead of face-to-face workshops. Regardless of the level of the effort, the expert elicitation complied with the central objective of using expert judgment in risk-informed decisionmaking: obtaining the distribution of the technical communitys beliefs about the state of knowledge concerning the technical problems.

21

3 RESULTS This chapter summarizes the integrated results of the expert elicitation. The chapter first presents the two scenarios developed by the experts and used for HEP estimation. The results for Objective 1 include the definition and specifications of the FLEX actions, the estimated HEPs, and the experts justifications for their estimation. Appendix D documents the detailed information on individual experts judgment of the HEPs. The results for Objective 2 include the PIF definitions, the PIF attributes that are relevant to FLEX actions, and the assessment of the significance of the attributes and their relevance to FLEX actions. The outcomes of Objective 2 were served as the input to Objective 3, the results of which are the experts opinions on how the PIFs and attributes may impact HEPs.

3.1 Results of Objective 1: Estimation of HEPs of the FLEX actions The experts received the following documents:

• two data packages, one consolidating data on human error rates and the other consolidating data on PIF effects

• definitions of two scenarios and the associated context

• Worksheet 1 with 12 tables (one for each FLEX action in one of the two scenarios), with each table containing the definition, tasks, and specifications of the action for experts to estimate the HEPs of the tasks The experts were asked to perform the following for each worksheet:

• Estimate the HEP distribution (1st, 50th, and 99th percentiles) of every task.

• Annotate the justification or basis of the estimation.

• Document additional assumptions and specifications on the actions used for estimation.

3.1.1 Scenario definition The expert panel used two scenarios for HEP estimation:

Scenario 1This is a non-FLEX-designed scenario in which the plant loses important safety functions without external hazards. It evolves in two parts. In the first part (Scenario 1.1), a plant is in SBO, one DG is down, and there is a good chance that the second DG may go down too. The plant may choose to use FLEX portable equipment without declaring ELAP. In the second part (Scenario 1.2), the scenario progresses to the point that the plant loses the second DG and may decide to declare ELAP.

Scenario 2This is the FLEX-designed scenario in which some external hazards lead to SBO and loss of both DGs. The plant is likely to use FLEX strategies.

Table 3-1 and 3-2 describe in detail the scenarios and the associated context characterized with the various states of PIFs. Definitions of the four PIF states are as follows:

• Nominal means that the attribute is not significant enough to cause human errors.

• Low impact means that the attribute alone may not lead to human errors but may increase the chance of human errors when some other PIFs are in poor states.

• Moderate impact means that the attribute alone may or may not lead to human errors but definitely increases the chance of human errors when some other PIFs are in poor states.

22

• High impact means that the attribute has significant impact on FLEX actions, and just this one attribute alone (assuming that all other PIFs are in nominal states) can lead to a high chance of human errors.

Table 3-1. Definition and context of Scenario 1 (non-FLEX-designed scenario)

Initiating event This would be consistent with having an emergency diesel generator (EDG) in a maintenance outage (out of service conditionEDG taken apart in many pieces) and then losing offsite power because of grid oscillations, severe weather or storms, sabotage, an accident in the switchyard (e.g., a vehicle backs into a transformer), or other event.

Initiating conditions

• LOOP

• loss of one safety bus (one EDG lost and not coming back)

• possible loss of the second EDG at any time Operational narrative In the first part of the scenario (Scenario 1.1), one EDG is out of service and the second EDG is running but may go down at any time. The Technical Support Center (TSC) decides to use the portable FLEX DG to power the bus associated with the out-of-service EDG and use the portable FLEX pump to provide RCS injection. In the second part of the scenario (Scenario 1.2), the second EDG is lost and may not come back soon, leading to the decision to declare ELAP and shed the load. After ELAP and load shed, offsite power returns, and the plant has the option of restoring power from the load shed.

Scenario 1.1 also has one or more potential common mode failures leading to the use of the FLEX pump. Below is the description of potential failure scenarios developed by one of the experts:

Blockage of intake by seaweed/jellyfish ingestion, silt, physical damage to screens, frazile ice, unusually low tide or low river water level, dam break downstream dropping water level quickly, or barge collision, etc. Loss of Turbine Lube Oil CoolerMust get Turbine shut down, loss of H2 Seal Oil System CoolingMust vent Main Generator, loss of Condenser Vacuum Pumps, loss of Circulating Watervacuum lossno Steam Generator (PWR

[pressurized-water reactor]) or Reactor (BWR [boiling-water reactor]) Feed Pumpsno steam dumps (smaller atmospheric dumpsPWR) or turbine bypass valves (BWR), loss of EDG Cooling water, loss of closed cycle cooling water systems (Service Water, Emergency Service Water or RHR [residual heat removal] Service Water), Instrument Air Systemno interstage cooling and no Air Dryer cooling For PWRs:

Letdown Heat Exchangersmust isolate Letdown (will wipe out Resin Beds if the temperature is over 140Fmust bypass but still put HOT water to VCT

[volume control tank]or divert to Liquid Radwaste which will steam out Aux Building. Must commence Cooldown to keep PZR [pressurizer] level under control WHILE charging.) Sampling System Coolersno boron verification, eventually could be a bigger concern verifying SDM [shutdown margin].

Control Room HVAC [heating, ventilation, and air conditioning]causes erroneous instrument readings and strange alarms. Switchgear Room 23

Ventilationneed to open doors as in SBO if in summer conditions.

ISOPHASE Bus Duct Cooling Systemnot immediate but could cause problems, not a big issue for now. Hot Penetration Room Cooling. Loss of motor cooling to most secondary pumpsneed to close MSIVs [Main Steam Isolation Valves] and Feedwater Isolations (only AFW [auxiliary feedwater] in useeventually only the Motor Driven AFW Pumps when SGs [steam generators] cooldownBUT you cannot cooldowninability to go on Shutdown Cooling. Therefore, must maintain Hot Standby BUT no cooling to the RCP [reactor coolant pump] thermal barrier so must keep Charging running to inject into seal. BUT as stated above, Letdown securedwhere is VCT getting makeup? Shift Suction to RWST [refueling water storage tank]. This is good for a while to keep the RCP Seals cool. Loss of Motor Cooling to RCPsMotor Winding Temp alarmsmust securego to natural circulation, loss of Motor Cooling to Shutdown Cooling Pumps (if installed)Necessary to go on Shutdown Cooling. So, you cannot go on Shutdown Cooling. Must simmer away feeding SGs with AFW Tanks and alternate sources of inventory (that needs to be replenished). Must keep RCP Seals cool by charging into Seal Injection and letting down to Radwaste.

Need to get FLEX pump setup to put water either into CST [condensate storage tank] (if motor-driven AFW pump available) or directly into the steam generators to keep RCS [reactor coolant system] cool.

For BWRs:

Controlling reactor pressure on safety relief valves (SRVs), Reactor Core Isolation Cooling (RCIC) starts on low reactor level and injects water into the reactor pressure vessel (RPV) to maintain water level. Depressurize with SRVs and use RCIC to put water into the RPV from the CST. Either torus (or suppression pool) level and temperature increases (RPV pressure control and reduction via SRVs)no cooling water to cool torus. RCIC loses barometric condenser. Torus (suppression pool) will heat up and may start steaming may need to have torus venting, need to get FLEX pump setup to put water either into CST (if RCIC is available) or directly into the RPV.

Human actions in the scenario Action 1 in Scenario 1.1Use portable FLEX DG to power the bus associated with the out-of-service EDG.

Action 2 in Scenario 1.1Use portable FLEX pump.

Action 3 in either Scenario 1.1 or 1.2Refill CST.

Action 4 in Scenario 1.2Declare ELAP.

Action 5 in Scenario 1.2Deep load shed.

Action 6 in Scenario 1.2Restore power from ELAP.

Assumptions for HEP estimation System and environment

• There is a single unit (unless specified otherwise).

• It is a sunny day, with no adverse weather factors, and no debris.

24

• Equipment is available (feasible but may not be reliable).

• Information is available (feasible but may not be reliable).

• The non-FLEX-strategy scenario includes no damage to equipment.

• The plant has only two EDGs (some plants have an SBO generator, but Scenario 1 assumes that they do not).

• RCS leakage is nominal (low leakage seal package).

• There is no alternative alternating current (ac).

• The FLEX generator and pump are not pre-staged and are brought from the FLEX building outside the fence (although some plants do have the diesels pre-staged).

Personnel

• Minimal and sufficient personnel are on site to perform all the actions needed; it is assumed that there is no second checker for any of these actions. While there might be a second check, which would be better, it cannot be assumed that there is one.

• Decisionmakers are in the TSC emergency response organization.

• Personnel are trained on FLEX strategies every 2-4 years.

Human actions

• The time available for performing all the human actions is adequate (4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> coping time).

• All the human actions are within the design basis and are feasible.

Human performance challenges:

• FLEX actions are inherent to the complications of the situation that leads to the use of FLEX.

• FLEX complies with feasibility, while the design basis is for feasibility and reliability. So, FLEX equipment may not be reliable even though the equipment has been demonstrated working.

• FLEX implementation was validated with the freshly trained A team. The performance may not be the same several years out when it is not as emphasized.

Scenario context The context of the scenario is characterized by various states of the PIFs.

PIF PIF state (A nominal PIF state means that the conditions of a PIF are normally designed for and do not impact performance.)

Information Information is nominalaccurate, complete, reliable, and timely.

Avaliability and Reliability Environmental Nominal factors

• available offsite resources

• heat/coldnessmay be cold (frazil ice concerns)

• noisenominal

• radiationnominal 25

• waterpossible loss of downstream dam or high water may bring debris into contact with plant (intake screens can be clogged with debrisseaweed, jellyfish, water grass) or cause physical damage to screens

• windmoderately strong winds that would focus debris on the intake structure

• visibilitynominal (lighting in some working areas may be less than adequate but personnel can manage)

• worksite accessibility and habitabilitynominal Human-system Nominal - The FLEX implementation guidance addresses clear consistent interface (HSI) labeling, color coding, and other human-system interface features. (from Nuclear Energy Institute (NEI) 12-06, Diverse and Flexible Coping Strategies (FLEX) Implementation Guide, Rev.4, 2016)

Parts and tools NominalAll the needed parts and tools are available and easy to access.

needed Procedures, Low impact Guidelines,

• Guidelines allow flexibility for adaptation.

Instructions

• Some parts of the guidelines can be nonspecific, lack details, or lack adaptability (e.g., the guidelines for connecting something may not specify the interfaces, labels, or the sequence or order of the connected parts).

Training and Moderate experience

• Training is infrequent (e.g., training for SBO may be every 2 years).

• Training may not offer hands-on practice.

• Transitioning from Phase 1 to Phase 2 of FLEX coping strategies requires much more plant knowledge for personnel.

• Phase 1 equipment is not used as designed (e.g., the way of using portable pumps may be different from routinely swapping service water pumps).

• Personnel may rush to set up equipment and lack the experience to thoroughly evaluate the specific situation for the type of the scenario.

Teamwork and

• Teamwork is low impact.

Organizational

• Communication is available and nominal.

Factors

• Coordination is low impact; actions and decisionmaking take place in multiple (more than three) locations outside the MCR.

• Command and Control (C&C) is low impact

- Three to four things are going on at once.

- CR staff handles emergency operating procedures (EOPs); FLEX Support Guidelines (FSGs) are not EOPs, so the CR does not tell the field what to do.

- It is unclear when the TSC gets involved, who gives orders, and who reports to whom.

Scenario Moderately unfamiliar with the scenario familiarity

• Personnel may not be familiar with the type of scenario even though they are familiar with or trained on the actions needed.

• Personnel may not know the situation.

• The scenario needs more troubleshooting.

Task Low to moderate impact Complexity

• Multiple influences affect scenario evolution.

26

• There are multiple potential outcomes of situational assessment or decisionmaking.

• Action execution requires monitoring of outcomes and adjusting actions accordingly over a long period of time.

• Actions are long lasting and not continuous.

Multitasking, Low impact Interruption,

• Personnel are frequently distracted by other ongoing activities.

Distraction

• Personnel concurrently make decisions or plans that may be intermingled.

Mental Fatigue Nominal and Stress (Note: Personnel may have high mental fatigue (given that the scenario needs to use FLEX) because of complicated, stressful, and long-lasting scenarios; mental fatigue can affect personnel performance. This project assumes it to be nominal in this scenario because it is difficult to assess.)

Physical Low impact Demands

• Personnel need to carry heavy equipment and tools.

• Connecting and assembling equipment or tools may require fine motor skills.

Table 3-2: Definition and context of Scenario 2 (FLEX-designed scenario)

Initiating event An external hazard caused flooding in a single-unit nuclear power plant and led to damage.

Initiating conditions

• SBO and LOOP

• some plant systems, equipment, structures damaged

• indications available

• debris removal needed

• personnel working in partially flooded areas Assumptions for HEP estimation

• System and environment

- There is a single unit (unless otherwise specified).

- There are adverse environmental factors (water and debris in working areas, cold, moderately strong wind, and dark).

- Minimum FLEX credited instrumentation is available (feasible but may not be reliable).

- Information is available (feasible but may not be reliable).

- RCS leakage is nominal (low-leakage seal package).

- There is no alternative alternating current (ac).

- FLEX generator and pump are not preinstalled, and they are brought from the FLEX building outside the fence.

- FLEX equipment is operated outside unless specified otherwise.

• Personnel

- Minimal and sufficient personnel are on site to perform all the actions needed; there is no second checker for any of these actions.

- Personnel are not going to be relieved for 8-12 hours.

- Decisionmakers are in the TSC emergency response organization.

- It has been 2-3 years since plant personnel were trained on FLEX strategies.

27

• Human actions

- The plant is probably pursuing multiple success paths.

- The time available for performing all the human actions is adequate (4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> coping time).

- All the human actions are within the design basis and are feasible.

Operational narrative

• ELAP should be declared within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.

• FLEX actions are performed following the declaration of ELAP, including load shed, debris removal, use of portable generator, use of portable pumps, and refilling of CST.

• Personnel transition from Phase 1 to Phase 2 FLEX strategies.

FLEX actions for HEP estimation The following actions were chosen for HEP estimates. Actions 1-5 are the same as in Scenario 1.

Action 1: transportation, placement, connection, and local control of portable pumps Action 2: transportation, placement, connection, and local control of portable generators Action 3: refilling of water storage tanks using alternate water sources Action 4: ELAP declaration Action 5: deep load shed Action 7: removal of debris Human performance challenges The hazard scenario includes damage to some paths, so personnel need to assess site damage and identify the deployment pathway.

• Plant validation exercises may provide limited insight into a complete, integrated response.

• In a complex FLEX strategy, it is difficult to predict how personnel error in one part of the strategy may affect other individual areas and the possible impact on the overall strategy.

• Operator delay in one task may hold up the completion of a subsequent task that affects multiple other operators (cascading failures).

• Resource management can be very challenging:

- Prioritize items, such as actions and resource allocations, between damage assessment, system restoration, and FLEX deployment.

- Assess and prioritize resources to perform the action in the given time.

• C&C between parties in different locations can be challenging.

Scenario context Scenario context is characterized by the PIF states below.

PIF PIF states Information Availability Low impact and Reliability

• Some primary sources of information are not available, and secondary sources of information might not be reliable.

Personnel do not know how to trust and verify secondary sources of information.

• Information is not timely.

• There is a lack of adequate elaboration in transmitting information and instructions.

Environmental Factors Moderately poor

• VisibilityWorkplace has moderately poor lighting (e.g., because of darkness, fog, smoke, dust).

28

• Water level is moderately poor; some workplaces or travel paths are in water (e.g., 1-3 feet deep).

• Worksite is moderately cold.

• Worksite accessibility and habitability are low impact.

HSI NominalIndications and controls are well designed and not damaged.

Parts and tools Low impact

• One or more connecting pieces may be missing or hard to find.

• Parts are difficult to use (e.g., connectors in different sizes, hoses in different configurations).

• Special tools may not be readily available (they may not be kept where they should be).

• More tools may be needed or may not be adequate because they are a shared resource.

Procedures, Low impact Guidelines, and

• not specific Instructions

• lacking details

• needing judgment

• needing adaptation (e.g., guidelines do not fully match the scenario)

Training and Moderately poor experience

• In this more complicated scenario, personnel found it hard to put their training and experience into action.

• Need to use flexibilitynot the preferred path, uncertainties in situations.

• Less training (whole scenario drills are infrequent).

• Lack of or less previous experienceit was difficult to operate by analogy, as no one had ever experienced a similar situation.

Teamwork and Low impact Organizational

• Team coordination is low impact. Whenever the actions were not Factors directly within the MCR, the shift team experienced three types of difficulties:

(1) interdependence with other stakeholders (e.g., evacuation of the surrounding populations and working with firefighters)

(2) difficulty in anticipating events for unexpected action sequences (e.g., no one had anticipated that an air compressor would be needed to open the venting valve remotely, and sourcing one significantly delayed the venting)

(3) the complexity of the system meant that an action could affect another action sequence

• C&C is low impact.

- Close coordination of activities is needed - activities are interdependent such that the action of one person cannot be achieved until the action of another is achieved and/or the action of one person can complicate or block the action of another.

29

- Coordination between site personnel and decisionmakers is inadequate to adapt or modify planned actions based on site situation.

- Personnel are unable to verify the plan because of inadequate communication (of the goals, negative impacts, deviations) with decisionmakers.

- Supervision in monitoring actions and questioning current mission is inadequate.

Scenario familiarity Highly unfamiliar

• The situation does not match prior training or experience.

• There is no existing mental model for the situation.

• The scenario is not recognized based on procedures or guidance; personnel must rely on knowledge to develop a mental model.

Multitasking, Low impact Interruption and

• Personnel are concurrently executing intermingled or Distraction interdependent action plans.

• The decisionmaker has multiple issues to address in parallel. (In Fukushima, the tsunami warnings affected the site superintendents planning of accident management because he was concerned that the tsunami might damage seawater pumps.)

• Personnels work was often interrupted by things such as aftershocks; personnel were also distracted by other ongoing activities.

Task Complexity Moderately complex

• Decisionmaking is complex because of competing goals, competing resources, dynamically changing situations, and confusing decision chains.

• Decision criteria are ambiguous and subject to different interpretations.

• The action sequences are long lasting and noncontinuous.

- Actions demand prospective memory (i.e., long lapse time before starting a follow-up activity).

- An action sequence includes a disconnected activity in the future for which there is no strong memory cue. Performing the action sequence requires personnel to memorize past status over a prolonged period (longer than several hours).

Mental Fatigue and Nominal Stress (Note: This study considered mental fatigue as nominal for HEP assessment in this particular scenario. In real FLEX strategy scenarios, personnel would have substantially high mental fatigue because of managing an accident involving multiple reactors at a site, lasting for extended durations, and/or involving stranded plant conditions, sleep deprivation, and worrying about family members safety.)

Physical Demands Low impact

• Personnel need to carry heavy things.

• Connecting and assembling some equipment or tools may require fine motor skills.

30

3.1.2 Overview of the process of HEP estimation HEP estimation One important assumption in the experts HEP estimation is that there is adequate time margin for all the actions. IDHEAS-G quantifies the HEP of a human action in two parts: (1) human errors attributed to cognition failure (i.e., failures of detection, diagnosis, decisionmaking, action execution, and C&C), and (2) errors attributed to inadequate time. Because the time available and time needed for an action vary greatly with plant configuration and implementation of FLEX strategies, the scope of this project is limited to estimating the HEPs attributed to cognition failures with the assumption of adequate time.

Even with adequate time margin, personnel may still experience time pressure for time-critical actions. The experts accounted for the time pressure in the PIF mental fatigue and stress.

The experts judgment of the HEPs was elicited through the following activities:

• After teleconferences for topic familiarization and training on estimating probability distributions, the experts made their initial HEP estimates and documented their justification and basis.

• At the workshop, the expert panel discussed each FLEX action, including assumptions, human performance challenges, and potential failures. The experts then made their estimates and took turns presenting and defending their judgments.

• After the workshop, the experts reviewed their notes, finalized their HEP estimates and justifications, and submitted their final worksheets.

Integration of the experts judgment The TIs performed integration through the following activities:

• review and consolidation of workshop notes (from two integrators and two observers)

• verification of the consolidated workshop notes with the experts final worksheets

• discussion with individual experts through e-mails or phone calls to clarify and verify experts assumptions and justifications

• consolidation of the experts views of the actions and justifications of their HEP estimates, including all the documented views and representing the diversities of the views as the problem uncertainties

• calculation of the average mean of the estimated HEPs, with all the experts judgments aggregated equally 3.1.3 Estimated HEPs and justifications This section presents the main results of Objective 3 in Table 3-3 which has 12 subtables (one for each action under Scenarios 1 and 2). In each subtable, the first portion is the action description and specifications that were given to the experts. The HEP estimation was made for the action described and specified under the defined scenario. The middle portion of each subtable presents the integrated HEP and the experts justifications. Note that the integrated justifications represent the range of the experts beliefs, not the consensus. In general, the justifications are arranged in the order of the basis for the lower HEP, the main drivers for the most likely HEP, and the basis for the higher HEP. Two experts used the SPAR-H method to justify their estimates. The justifications also include the experts opinions on uncertainties in the HEP. The bottom portion of a subtable documents the experts discussion of human 31

performance of the action. The discussion may not be directly relevant to HEP estimates, but it provides an understanding of the factors that affect the human performance of the action.

The justifications are listed in bullets. However, the content of a bullet is integrated from the inputs of the expert panel, so it represents the judgment of the expert panel as a technical community. To capture individual experts beliefs, Appendix D documents every experts HEP estimate along with his or her specific justification that is different from others.

Table 3-3 Action 1 in Scenario 1.1: Use of portable generator Definition: This action assumes that the other EDG is running and that the plant would set up the FLEX DG to lower core damage frequency (CDF). Examples of the action are aligning FLEX generators for SG level instruments (and control level), 480-volt loads and supporting the battery charger, instrumentation, and AFW. The tasks in Action 1 include identifying the need for and deciding to use the portable generator and transporting, staging, and operating the generator.

Specifications:

• specific tools: FLEX generators and associated cabling, tow vehicles, forklift, and tooling maintained on site

• guidelines: FSGs

• FLEX generator480-megavolt generator, about a queen-sized bed wide and 6-feet tall, sits on trailer, hauled by truck.

• use of the same connector points as in the FLEX-designed scenario Task 1.1 Decide to use portable generator Activities:

• Assess plant situation and decide to use FLEX generators.

• Plan and develop work orders and personnel assignment (development of work instructions is generally not needed because they are typically already in the FSGs).

Average estimated HEP: [0.016, 0.052, 0.10] (1st, 50th, and 99th percentile, respectively)

Justifications:

• This is for coping plants, not for alternate ac plants. The plant would probably deploy the FLEX generator but would likely not connect to the bus unless the credited safety function was lost.

• In this scenario, the plant is described as having a fully functioning EDG. Even after the event starts, this functional EDG appears to operate properly. With a functional EDG, there is significant time available to operate the FLEX DG.

• Assuming that the time to make the decision is longer and based on a plan to lower core damage frequency, the HEP is estimated for hurried setup.

• Resource management should be considered in decisionmaking.

• The task is performed in parallel with many other activities.

• In SPAR-H analysis, the nominal diagnostic HEP is 1E-2. This is modified by recognizing the additional time supplied by the functional EDG, which would work to 32

decrease the HEP. As a diagnostic HEP, the off-nominal conditions described in the outside environment do not affect the decisionmaking in the MCR or the TSC.

• Lower bound of the HEP: If the plan to use the FLEX generator is just to stage it every time that the plant goes to an EDG maintenance outage, then the action is a success, meaning that the HEP would be very low.

• Upper bound of the HEP: An alternative path (not using the FLEX generator) exists.

MCR is focused on maintaining stability of the plant. TSC works on managing activities, giving orders to operators, and restoring the DGs; it needs to set priorities.

The TSC is spending all its time trying to get the diesels going. Being in SBO does not mean going to FLEX. Plant operators are in the EOPs, and the EOPs are trying to restore the diesels and offsite power.

Task 1.2 Transport and stage portable generator (and cables)

Activities:

• Conduct pre-job briefing and review work instructions and guidelines.

• Hook up portable generator.

• Transport portable generator (by vehicle).

• Transport cable from storage building and deploy.

• Unhook the generator at the correct location.

Average estimated HEP: [0.023 0.057 0.27] (1st, 50th, and 99th percentile, respectively)

Justifications:

• The task is moving partsvehicle and placement of generator with no external event.

• Coordination is required (either in person or by handheld radio) to move cable from storage and deploy in parallel with the generator being transported.

• The task is performed in parallel with many other activities; interface with other actions should be considered.

• This is an action HEP. The base HEP is 1E-3 according the HRA methods, SPAR-H and THERP. The functional EDG supplies significant time to perform this action, which would lower the action HEP. However, the scenario context describes negative environmental effects, possibly including cold, water on site, and moderately strong winds. These increase the HEP. Also described is the teamwork issue with low-impact coordination and C&C. The scenario description also indicates low-impact guidelines and procedures. Finally, the scenario description of familiarity states that personnel may not be familiar with the scenarios, personnel may not know the situation, and the scenario may need more troubleshooting. The combination of significant additional time (decreasing the probability) with poor environmental conditions, poor teamwork, low impact procedures, and an unfamiliar scenario leads to uncertainties represented by the lower and upper bounds.

Task 1.3 Connect portable generator Activities:

• Terminate cable connections from generator to connection points.

• Manipulate circuit breakerensure phase rotation.

• Start generator using a local control panel and ensure that it runs properly.

33

• Ensure phase rotation.

• Validate charger output voltage and current, which are available locally.

Average estimated HEP: [0.027 0.088 0.31] (1st, 50th, and 99th percentile, respectively)

Justifications:

• Use of FLEX equipment, in SBO or not, indicates a really bad situation. Personnel must stabilize electrical power. Personnel are, in general, unfamiliar with the portable diesel because it is not used in the same way as permanently installed ones. Many other activities are going on at the same time.

• To connect to the portable generator, personnel need to disconnect loads and open many breakers (proceduralized, but the procedures are not often looked at and never physically tested). Personnel may fail to open the breakers. If not, everything is disconnected, the generator will be tripped. In addition, the cables can run into problems, they may not go straight to the designated areas.

• The task involves skill-of-craft activities such as phase rotation.

• High stress, low training, procedures not as good (not detailed at the same level) as in the EOPs (e.g., if personnel run into problems, the procedure tells you to get the supplemental box, but does not walk you through fixing the problems.

• On the lower bound, connecting and operating the generator with good guidance procedures can be as easy as in the EOPs. Connectors are color coded so plugging and using them should be easy. Phase rotation should have been verified via surveillance.

• Procedures may or may not be adequate and training may have been long ago, raising the upper bound.

1.4 Operate the generator Activities:

• Sequentially load generator.

• Monitor generator indications for any abnormalities.

• Monitor generator temperatures, voltages, current, and fuel levels.

Average estimated HEP: [0.024 0.052 0.22] (1st, 50th, and 99th percentile, respectively)

Justifications:

• Local manipulations are required, and requirements are communicated from the MCR.

• The action is long term (low decay heat, long response time available).

• Portable generators can be a problem if not carefully operated. There can be problems changing load (if the load is variable, generators trip easily). Personnel with expertise may not be present. The plant may not have the right people to perform the task.

• The licensee may need to go outside of existing procedures (declare Title 10 of the Code of Federal Regulations (10 CFR) 50.54(x) because these are normal LOOP scenarios and presumably not an ELAP caused by a large external event.

• Training and familiarization drive the uncertainty. Personnel can perform a task with high reliability if they have just been trained on it. However, the training will be 34

conducted every 6 or 7 years, which means personnel might be familiar with the task now but may not continue to be familiar with the task.

Discussion:

Scenario 1.1 specifies that one EDG is running; thus, all the system functions and instrument conditions are nominal. If both EDGs are out, less time would be available for personnel to perform the actions, which would impose time pressure on the personnel.

Table 3-4. Action 1 in Scenario 2: Use of portable generator Task 1.1 Decide to use portable generator ELAP declaration and the deployment of FLEX equipment are the same decision, so this task is a part of ELAP declaration.

Task 1.2 Transport and stage portable generator (and cables)

• Conduct pre-job brief and review work instructions and guidelines.

• Hook up portable generator.

• Transport portable generator (by vehicle).

• Transport cable from storage building and deploy.

• Unhook the generator at the correct location.

Average estimated HEP: [0.038 0.14 0.52 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• The biggest difference between the two scenarios for this action is weather. Weather affects operating the generator while outside (I talked to operators at every plant I went to and the bottom line was that if it is bad outside, it sucks, said one panel member).

• Debris removal may not be complete, or weather may still be bad.

• There is no additional staff for the first 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />. Minimum staffing is a big issue. For example, four of the field operators are taken for other duties (AFW and steam dump manual operations); one or two crew members are looking to get a diesel back and get information on the diesel. So, time is constrained and personnel are stressed.

• The lower bound is for the situation in which the effects of a bad environment are the same as those of a sunny day.

• The upper bound is driven by factors such as the information on plant situations may not be available timely, poor visibility can be an issue, and the weather can be very bad (e.g., the site experiences major flooding, the whole site becomes an island under 4-5 feet of water and personnel cannot even get to the work location).

• This is an action HEP. The base SPAR-H and THERP action HEP is 1E-3. With no functional EDG, significant stress exists for those performing this action. The negative environmental effects include moderately poor visibility with poor lighting and moderately poor water level in some workplaces or travel paths. Workplace accessibility and habitability are low impact. Parts and tools needed are described as low impact, with one or more connecting pieces missing or difficult to find and connectors and hoses that are difficult to use. Guidelines and procedures are described as low impact as they are nonspecific, lacking in details, and requiring 35

judgment and adaption. Training is described as moderately poor or complicated, and it is hard to put training and experience into action. Teamwork and C&C are low impact. The scenario is highly unfamiliar as it does not match prior training and experience. The action requires multitasking with low-impact conditions and moderately complex actions that are physically demanding. In short, the operators are set up for failure as they contend with these extremely difficult influences.

Task 1.3 Connect portable generator Activities:

• Terminate cable connections from generator to connection points.

• Manipulate circuit breakerensure phase rotation.

• Start generator using a local control panel and ensure that it runs properly.

• Ensure phase rotation.

• Validate charger output voltage and current, which are available locally.

Average estimated HEP: [0.043 0.16 0.41 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• Procedural guidance is not well defined; personnel have much information but do not know what to do.

• While connecting the generator could be simpler than transporting it, the conditions remain very poor and demanding.

• Lower boundThe generators are indoors so the task is similar to that on a sunny day because there are no additional hazards indoors.

• Upper boundIt is extremely difficult to connect cables because of weather conditions, multiple connections, and propping open doors with wind blowing and rain.

Task 1.4 Operate the generator Activities:

• Sequentially load generator.

• Monitor generator indications for any abnormalities.

• Monitor generator temperatures, voltages, current, and fuel levels.

Note: Refueling is not included in this task definition. If it were included, it would be affected by weather in a way similar to that described for Task 1.2.

Average estimated HEP: [0.036 0.12 0.44 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• Personnel are under high stress operating FLEX equipment. They are not familiar with this kind of operation, and their mindset for operating a generator in normal operation and EOPs is different from operating a FLEX generator.

• Weather and generator location drive the upper bound. Personnel could be outside operating the generator between turbines with wind upwards of 30 miles per hour (mph); different plants may have different capabilities for handling bad weather. On the other hand, if the generator is indoors, then the situation is the same or similar to a sunny day because there are no additional hazards indoors.

36

• Operating the generator should be simpler than transporting or connecting it. The conditions remain very poor and demanding. The HEP is somewhat lower than in the previous two tasks.

Table 3-4. Action 2: Use of portable pump in Scenario 1.1 Definition: This action uses the FLEX pump to inject clean water for Scenario 1 (for PWRs, the study does not consider injecting borated water, which needs a much longer time). One or several potential common failures described in Scenario 1.1 lead to use of the FLEX pump in the non-FLEX-designed scenario. For example, this pump is replacing the RCS heat removal. In this scenario, the plant is described as having a fully functional EDG. This action runs a FLEX pump injecting into the RPV (BWR) or an SG (PWR). The action includes the following tasks: identify the loss injection and decide to use a portable pump; transport, stage, and connect the portable pump; start the pump; align the flow path from the pump to the intended target (e.g., the RPV or an SG); and control the target water level. Failure of the action can result from a failure of any of these tasks. This action is performed because the permanently installed equipment fails.

Specifications:

• SBO and everything else is nominal. The core is covered, there is no wild surge of pressure, and RCS inventory is under control.

• The plant reaches the conditions relying on FLEX. The intake structure is lost but still has power. Personnel need to find water.

• Discharge pressureBWR and PWR are roughly the same pressure (depressurize injection around 250 pounds per square inch).

• Deployment of the pump includes choosing an alignment location or flowpath. The non-FLEX-designed scenario has no damage to the deployment path or equipment.

• Personnel manually control valves during SBO.

• The action is long term (low decay heat, long response time available). The event progression behaves slowly, so time is not critical.

• Many other activities are occurring. Resource management should be considered.

• Action is performed in parallel with many other activities; interface with other actions should be considered.

• Staff is aware that the need for injection is pending (planned action)

- Exact time may not be known.

- Loss of running system (RCIC) is obvious.

- EOPs and FSGs govern level control and prompt shift managers direction to initiate pump via FLEX guidelines.

- Local manipulations are required for flow control. Level/flow requirements are generally communicated from the MCR.

• Specific tools: FLEX pumps and associated suction/discharge piping, suction strainers, tow vehicles, forklift (if needed), and tooling are maintained on site.

37

• Instrumentation: RPV level, RCIC status (pump trip, flow).

• Training: The operators are trained to maintain RCIC for as long as possible, then switch to the FLEX pump because they understand that the other injection options are not available because of the SBO or other conditions.

• Procedures: EOP governs level control but may be augmented by ELAP procedure and FSGs. No procedures are specifically designed for the use of FLEX pumps in non-FLEX-designed scenarios.

• Pump typeBig ones, sit-on-wheels, BWR and PWR pumps are similar.

Task 2.1 Decide to use portable pump Activities:

• Assess plant situation and decide to use portable pumps.

• Choose the correct procedures and guidelines, develop or adapt work instructions, and assign personnel as needed.

• Decide (when) to implement the action.

Average estimated HEP: [0.034 0.055 0.1 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• The decision can fail if personnel decide to restore power and fix the trouble instead of using the FLEX pump.

• This task entails the same hazards and same general rigging as for portable generators.

• Personnel need to understand how water sources would be prioritized in a specific scenario.

• This is a diagnostic-only HEP. The anchor point from SPAR-H for a nominal diagnostic HEP is 1E-2. The scenario description discusses problems at the pump house leading to loss of service water to both safety-related and non-safety-related loads. The decision is not affected by poor conditions as the MCR is generally not affected (other than the significant loss of lighting and HVAC). The scenario context describes challenges in the guidelines being low impact, teamwork being low impact, and the scenario being moderately unfamiliar. These factors drive the HEP higher than the SPAR-H nominal value.

Task 2.2 Transport and stage the pump Activities:

• Hook up portable equipment.

• Transport portable equipment.

• Transport hoses from storage building and deploy.

• Stage the pump.

Average estimated HEP: [0.016 0.058 0.33 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• The numbers and justifications are the same as for Task 1.1.

38

• Although not time critical, personnel need to transport and stage the pump very quickly.

• The pump location and the configuration of its staging drive the uncertainty.

• This is an action HEP. An anchoring HEP is the base SPAR-H and THERP action HEP of 1E-3. Failure of the EDGs places considerable time pressure on performance of this action, which lowers the action HEP. However, the event analysis describes negative environmental effects including possible cold, high water on site, and moderately strong winds. These increase the HEP. Also described is a teamwork issue with low-impact coordination and C&C. The scenario description also indicates low-impact guidelines and procedures. Finally, the scenario familiarity description states that personnel may be unfamiliar with the scenarios or may not know the situation, and the scenario may need more troubleshooting. The combination of time pressure with poor environmental conditions, poor teamwork, low-impact procedures, and an unfamiliar scenario moves the HEP higher than the base HEP of 1E-3.

Task 2.3 Connect the pump Activities:

• Establish building and water source access (may need to cut security fencing) and/or breach secondary containment.

• Connect portable pump suction hoses and pipes

• Connect portable pump discharge piping to charging line.

Average estimated HEP: [0.019 0.078 0.27 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• Difficulty in reaching the connectors that are far away (100s of feet).

• Sometimes personnel have to run the hoses through protected routes (so hoses are not damaged by vehicles driving over them). Challenges include suction/discharge floating/snaking around; opening isolation valves, difficult areas, hose running over hose, making connections in water, and putting long hoses through water.

• Working in the dark drives the upper bound.

Task 2.4: Start and operate the pump Activities:

• Start portable pump using a local control panel and establish the flow.

• Valve alignmentsLocally open manual valves for injection (not many valves are manually operated).

• Operate pumpWater level control with portable pump is an on-off switch using valves to have flow going, swapping back and forth; on-off can be a problem causing overfill.

• Monitor water source.

• Monitor portable pump suction screens.

Average estimated HEP: [0.017 0.05 0.21 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

39

• This task would be easier if there were no external event.

• Once the pump is staged and connected, operating the pump should be comparatively straightforward compared to transporting and connecting it. However, the event analysis describes negative environmental effects including cold, water on site, and moderately strong winds. These effects increase the HEP.

• After the pump starts, there are many connections; keeping them all going is somewhat difficult.

• The task also includes monitoring and cleaning the strainers.

• The task needs to be tightly coordinated with the CR.

• Uncertainty is high because the circumstance for operating the pump is not easily knowable; therefore, the HEP bounds spread; the high end is for the worst possible day, which is barely feasible.

Table 3-4. Action 2: Use of portable pump in Scenario 2 Definition: In this scenario the plant is described as having no functional EDG. This action runs a FLEX pump, injecting into the RPV (BWR) or an SG (PWR) after ELAP is declared in a FLEX-designed scenario caused by external hazards. The action includes the following tasks:

transport, stage, and connect the portable pump; start the pump; align the flowpath from the pump to the intended target (e.g., the RPV or an SG); and control the target water level.

Failure of any of these tasks can lead to failure of the action.

Specifications:

• ELAP, external hazards, and multiple activities are going on. The normal water source is gone. Inventory safety functions are working. The RCS inventory runs out of water.

Personnel need to use the pump to inject bay water.

• Pump is diesel driven so power is not needed, and it is not necessary to start generators first.

• Minimum design equipment is available, but the event hinders connection and execution.

• Babcock & Wilcox (B&W) plants have an emergency feedwater steam-driven pump, so the timeline is comparable. Otherwise, the timeline for a B&W plant is very different (minutes versus hours) for secondary heat removal.

• Discharge pressure This is roughly the same for BWR and PWR; depressurized injection is around 250 pounds per square inch.

• The core is covered, there is no extreme surge of pressure, and the scenario proceeds slowly.

• The RCS inventory is under control.

• The hazard has damaged some deployment paths, so personnel need to assess site damage and to identify the deployment pathway.

• This is a long-term action (low decay heat, long response time available).

• Everything is outside, no wall and no ceiling; bay water is used; personnel need to roll hoses and make connectors.

40

• The portable pump will be hooked to an alternate source such as the Ultimate Heat Sink which has a different water quality.

• Resource management should be considered.

• The task is performed in parallel with many other activities; interface with other actions should be considered.

• Personnel need to understand how water sources would be prioritized in a specific scenario.

• Staff is aware that the need for injection is pending (planned action):

- Exact time may not be known.

- Loss of running system (RCIC) will be obvious.

- EOPs and FSGs govern level control and prompt shift managers direction to initiate pump via FLEX guidelines.

- Local manipulations are required for flow control. Level/flow requirements are generally communicated from the MCR.

• Specific tools: FLEX pumps and associated suction/discharge piping, suction strainers, tow vehicles, forklift (if needed), and tooling are maintained on site.

• Instrumentation: RPV level and/or RCIC status (pump trip, flow).

• Training: The operators are trained to maintain RCIC for as long as possible, then switch to the FLEX pump.

• Procedures: The EOP governs level control but may be augmented by ELAP procedure and FSGs.

• Pump type: Pumps are the big ones sitting on trailers; BWR and PWR pumps are similar.

Task 2.1 Decide to use portable pump This task is a part of the action declaration of ELAP. Deciding to enter ELAP automatically leads to use of the portable pump.

The scenario stipulates that some deployment paths are damaged, so assessment of the site damage and an identification of the deployment pathway are needed. This scenario requires injection of river water by the portable pump; injecting river water (which is far from clear and clean) causes significant material degradation to the plant. Personnel could be very reluctant to take this highly detrimental step; however, the severity of the scenario should overcome some, if not all, of the reluctance. Poor conditions do not influence the decision, as the MCR is generally not affected (other than the significant loss of lighting and HVAC).

Task 2.2 Transport and stage the pump Activities:

• Hook up portable equipment.

• Transport portable equipment.

• Transport hoses from storage building and deploy.

• Stage the pump.

41

Average estimated HEP: [0.023 0.12 0.47 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• The hazards and general rigging are the same as for portable generators.

• There are placards clearly indicting where the pumps and hoses go. Strategies are designed so they are not physically strenuous.

• There could be a mismatch between the trailer and hitch. Personnel have tool kits that allow them to adapt.

• FLEX strategy implementation drives the lower bound; the connections and tools should be correct. If the connections are right and personnel have their tools, then the HEP goes down. If personnel have to improvise on the fly, then there is a high chance of failure.

• Weatherhurricane, dealing with much debris, location is cramped and basically a big wind tunneldrives the upper bound.

• This is an action HEP. The scenario stipulates that some deployment paths are damaged so assessment of the site damage and identification of the deployment pathway are necessary. The scenario also describes negative environmental effects including moderately poor visibility with poor lighting, moderately poor water level in some workplaces or travel paths, and limited workplace accessibility and low-impact habitability. Parts and tools needed are described as low impact, with one or more connecting pieces missing or hard to find and difficult-to-use connectors and hoses.

Guidelines and procedures are described as low impact as they are nonspecific, lack detail, and require judgment and adaption. Training is described as moderately poor, because it is complicated, and it is hard to put training and experience into action.

Teamwork and C&C are low impact. The scenario is highly unfamiliar as it does not match prior training and experience. The action requires multitasking in low-impact conditions, involves moderately complex actions, and is physically demanding. In short, the operators are set up for failure with these extremely difficult influences.

Task 2.3 Connect the pump Activities:

• Establish building and water source access (may need to cut security fencing) and/or breach secondary containment.

• Connect portable pump suction hoses or pipes.

• Connect portable pump discharge piping to charging line.

Average estimated HEP: [0.036 0.13 0.45] (1st, 50th, and 99th percentile, respectively)

Justifications:

• There is no charging line. This could be a long hose run, potentially including many stairs. Multiple connection points are available.

• One expert observed of the testing: It took the personnel hours to get it up and running -

guys out there thinking, Is it priming or not self-priming?. You get the right mechanics there and it works.

42

• There could be many challenges. Hoses are heavywhere do you put them?

Discharge to where? Did you close all the isolation valves? There is an absence of indications. How do you know your water is going in? How do you have confidence that it is working?

• RCS needs depressurization so pump size matters.

• There are steps for cooldown and stabilization. The action lasts about 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />.

• There could be problems with equipment, procedures, training, and indications.

• There are not many places where the pump can be connected incorrectly. Plants have different options, but the connections are fairly specific, and the procedures locate the connections.

• The HEP should be higher than that for connecting the generator because of extra considerations (e.g., where to put the water, clogging).

• Indoor/protected versus outside/nonprotected connection points and equipment can shift the HEP an order of magnitude. Being outside in bad weather, it is difficult to access connection points and may require hanging off the rafters to reach them. If that area has been damaged such that the pumps are damaged too, there is a very high probability that it will be difficult to access the connection points.

Task 2.4: Start and operate the pump Activities:

• Start the portable pump using a local control panel and establish the flow.

• Align valvesLocally open manual valves for injection.

• Operate pumpWater level control with portable pump entails monitoring gauges, with the operator watching and changing flow speed, using the valve to keep flow going, and swapping back and forth; on-off can be a problem causing overfill.

• Monitor portable pump suction screens.

Note: This task does not consider refueling.

Average estimated HEP: [0.043 0.14 0.44 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• Lack of transparencyHow do you know the water is going to the right place? You can hook up the lines and start the water, but these are big lines and the flow is relatively small, and it is difficult to know whether there are isolation valves to direct the flow to the right place (dispositioning in the flow-path). On the other hand, it is not an issue if you continue to use the path you were using.

• This is a continuous action that requires monitoring over a long period.

• The minimum design equipment is available, but the event could cause a need to adapt (e.g., a missile dents the vent and personnel need to act).

• This task is more difficult than operating the generator because personnel have to watch and clean the strainer since the suction source is a river or ocean.

43

• These activities are not complicated to operate. The complicating factors would be more in line with diagnosing pump problems while the pump is operating.

• The lower bound is on the best basis: no standing water, debris cleared out, and indoors/protected.

Discussion:

• The panel members believed Actions 1 and 2 were similar enough that the HEP values should be identical; it would not matter to the success of the action whether the trailer being hauled from the FLEX Phase 2 building on site has an electrical generator or a high-pressure pump.

• The HEP estimation is based on the specification that only one diesel is running in Scenario 1.1. If both diesels are down, the HEP would increase by a factor of about 2-5.

• In one experts interpretation of the Scenario 1.1 context in SPAR-H, the time available was approximately equal to the time necessary, or at least there was not much margin.

The stress level would be high but probably not extreme, and the training PIF was rated as low because of the infrequency of the training that operators receive on moving and connecting this equipment.

• Refueling would make the situation worse and increase the HEP by a factor of about 2-4.

Table 3-4. Action 3: Refill CST with alternative water source in Scenarios 1 and 2 Definition: In Scenario 1.1, the CST normal fill system is not available because of electrical or breaker problems related to the fill pumps or because the normal fill pumps and piping are damaged or out of service (under maintenance). RCIC normally takes suction from the CST.

The CST level is low, so personnel need to use an alternate water supply to refill the CST.

This action uses an alternate water source to supply the CST to keep the RCIC in operation, taking suction from the CST versus the suppression pool (torus).

In this scenario, the plant has a fully functioning EDG. This action uses FLEX to refill the CST from an alternative water source. This implies that the FLEX generator and pump have been staged, connected, and placed in operation as necessary, or that FLEX was not needed for these measures. In either case, core cooling has been established, and for whatever reason, refilling of the CST cannot be established via permanently installed equipment, so FLEX is being used to perform this task.

Specifications:

• The source of water is clear water in Scenario 1 and bay water in Scenario 2.

• The action is to pump water directly to the tank; personnel must remove the manhole cover from the top of the CST to insert hoses.

• Decisionmakers may be reluctant to use the alternative water source.

• It is necessary to prioritize water sources and resources needed (vehicles and personnel).

44

• This is a long-term action, lasting a day; both scenarios have plenty of time for this action.

• No staff is dedicated for the action, but personnel are available.

• Guidelines or procedures are available but are not specific or detailed.

• The CST is outside.

45

Task 3.1 Decide to use alternative water source Activities:

• Recognize the need to use an alternative water source (e.g., tank is getting low).

• Decide and authorize the use of an alternative water source.

• Decide which source of water is most available or cleanest for the application.

• Coordinate and manage resources for using the alternative water source.

Average estimated HEP: [0.034 0.057 0.11 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• Pump is in low pressure and not as large. The hose may kink and prevent flow. The probability of actually needing this source would be very low. The decision to use this source of water would be made only when all other sources are almost exhausted or unavailable.

• The decisionmakers may be reluctant to use an alternative water source. However, the long timeframe involved and the recognition that core injection also has been operating for a long period should override this reluctance.

• Personnel must decide to use this in a non-FLEX situation. They may be reluctant to use water that is not ideal. Using dirty water means the CST will be damaged.

• Making the decision means swapping water sources. Decisionmakers must consider and evaluate other water sources. They have a list of tanks and will prioritize them, moving down the list as they seek the easiest and cleanest source. If good sources are exhausted or unavailable, there will be little reluctance to use dirty water. The cues are straightforward.

• The action is not time critical. With establishment of core cooling, the CST will not be required for many hours before depletion.

• HEP is low because there is plenty of time and an obvious diagnosis. There is no external event and no distraction. This is a high-priority action, monitored by a dedicated operator, with good procedures.

• This is a diagnostic-only HEP. The anchor point from SPAR-H for a nominal diagnostic HEP is 1E-2. This is not time critical for the reasons discussed above. In addition, with the passage of hours, it becomes obvious that the CST will need to be refilled. The scenario description discusses problems at the pump house leading to loss of service water to both safety-related and non-safety-related loads. The decision is not influenced by poor conditions as the MCR and TSC are generally not affected (other than a significant loss of lighting and HVAC). The general description of Scenarios 1 and 2 states that challenges in guidelines are low impact, teamwork is low impact, and unfamiliarity with the scenario is moderate. However, the additional time and obvious diagnosis dominate the analysis.

Task 3.2 Use alternative water source to fill water tank Activities:

• Connect the FLEX pump to the CST (remove the manhole cover from the top of the CST and insert the hose in the top of the tank).

46

• Open or connect the flow-path to the alternative water source.

• Start filling the water tank.

Note: This task does not include transporting and staging the pump.

Average estimated HEP: [0.01 0.046 0.28 ] (1st, 50th, and 99th percentile, respectively)

Justifications:

• PRA already has modeled this as a very simple operation with a very low HEP. The HEP is in the range or E-3 to E-2 given high stress, low training, procedures available, and ample time (so that staffing would not be an issue).

• A relatively optimistic value for this HEP is assigned largely because refilling the CST from any number of sources is performed often, trained on, and (it is believed) with formal job performance measurements for operator qualifications. The nominal HEP for an action (1E-3) was increased slightly because of the somewhat elevated stress that would likely occur, and because the procedures may not address every eventuality of refilling the CST.

• If an event is bad enough that it tore up the CST and pipes, it may be necessary to use the pump to suck water from the ocean. The damage to working paths, location, and structure are uncertain; personnel climbing up to the manhole can be in a difficult situation.

• The hose can get a kink that would choke off flow. There is a long flow-path and a tight turning radius to drop into the tank.

• This is an action HEP. An anchoring point is the base SPAR-H and THERP action HEP of 1E-3. Successfully establishing core cooling (thus supplying additional time to perform this action) lowered the action HEP. However, the event analysis describes negative environmental effects including possible cold, high water on site, and moderately strong winds. These increase the HEP. Also described is a teamwork issue with low-impact coordination and C&C. The scenario description also indicates low-impact guidelines and procedures. The scenario familiarity description states that personnel may be unfamiliar with the scenarios or may not know the situation, and the scenario may need more troubleshooting. The combination of time pressure with poor environmental conditions, poor teamwork, low-impact procedures and an unfamiliar scenario drives the HEP higher.

Table 3-4. Action 3: Refill CST with alternative water source in Scenario 2 The action definition and specifications are the same as for Scenario 1, except that the water source in Scenario 2 is river water.

Task 3.1 Decide to use alternative water source Activities:

• Recognize the need to use an alternative water source (e.g., tank getting low).

• Decide and authorize the use of an alternative water source.

47

Average estimated HEP: [0.03 0.06 0.091] (1st, 50th, and 99th percentile, respectively)

Justifications:

• When in a FLEX scenario, this decision would be more straightforward.

• Decisionmakers may not decide in time because of many distractions and other high-priority activities; they may underestimate debris removal time.

• Coordinating and managing resources for using an alternative water source can be challenging.

Task 3.2 Use alternative water source to fill water tank Activities:

• Transport and stage the pump (this is referred to in Task 2.2 and is not included in the estimation here).

• Connect the pump to the water tank.

• Open or connect the flowpath to the alternative water source.

• Start filling the water tank.

Average estimated HEP: [0.072 0.14 0.36] (1st, 50th, and 99th percentile, respectively)

Justifications:

The FLEX-designed scenario is more challenging as personnel may be unsure of the damage, not know connections, have difficulty getting there, have difficulty getting hoses inside, have less time than on a sunny day because of other activities and distractions, and cues and equipment are not ideal.

• Debris can cause problems.

• C&C faces challenges in assigning priorities and resource management. There is little training in C&C in FLEX.

• The CR uses indicators but does not know what others are doing and does not know if indicators are reliable, does not have optimal indications, and lacks communications with other parties.

• One panel member assigned a lower value (i.e., more optimistic) to this HEP based on the net effects of two factors. The first factor is that this action is frequently performed by operators while the plant is at power, and operators are usually trained in a number of different methods as demonstrated in their Job Performance Measurements. This factor swamped the negative effect of higher stress and difficulty in performing this action in an adverse environment.

• Upper bound is driven by bad weather (e.g., extreme wind and rain). One panel member said: I didnt see this being done very often and wouldnt want to stand on a ladder doing this in a hurricane. Or if it is icy on top of the tank, good luck getting the manhole cover offmaybe have to torch it or something). Some plants have blank flanges on the bottom, so it is easier.

48

Table 3-4. Action 4: Declare ELAP in Scenario 1.2 and Scenario 2 Definition in Scenario 1.2: SBO (e.g., power grid disturbance) leads to the declaration of ELAP. It is not certain when offsite power will come back. The success of the action is to decide to go to ELAP because the battery lasts only 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />. The HEP is for failing to declare ELAP within the required time (60 minutes from the SBO).

This scenario is described as a SBO with uncertainty as to when offsite power will be restored. The station has 4-hour batteries, and the criterion for declaring ELAP is that the power is not restored in 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and is not expected to be back in 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />.

Definition in Scenario 2: A FLEX-designed external hazard caused LOOP and SBO, leading the station to declare ELAP and shed load. The success of the action is to decide to go to ELAP because the battery lasts only 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />. The HEP is for failing to declare ELAP within the required time.

Declaration of ELAP includes these tasks:

• Identify the conditions corresponding to an ELAP event.

• Declare ELAP.

• Plan the various ELAP-related activities.

• Prioritize resources for recovering the EDG or performing load shed.

Downstream actions no longer require decisions. It is just a matter of implementing the procedures, which requires more resource management, particularly if the station is at minimal staffing and will not have anyone arrive from off site within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />.

Specifications (applicable to both scenarios):

• This applies to plants with a 4-hour coping time.

• The timeline for declaring ELAP is very short. If ac power is not restored to the emergency 4-kilovolt busses within 60 minutes of the loss of all ac power and is not expected back within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />, then the station should declare ELAP.

• The knowledge of emergency ac power availability is critical, but information about when ac power will be restored is uncertain (e.g., there has been no contact from offsite staff concerning offsite power restoration, and the extent of the EDG failures has not yet been diagnosed).

• Parties include the CR crew, field operators, and maintenance personnel. The Control Room Supervisor (CRS) is the key decisionmaker, the Shift Manager (SM) declares the emergency, and the offsite emergency response officer (ERO) interacts with the CRS.

• Information is not promptly available.

• Training (drill) occurs every 2 years. Personnel know the consequences of ELAP.

• Guidelines and procedures exist but require judgment and vary widely from plant to plant.

Average estimated HEP in Scenario 1: [0.046 0.31 0.66 ] (1st, 50th, and 99th percentile, respectively)

Justifications for ELAP declaration in Scenario 1:

49

• ELAP is to be declared in an SBO. Once the plant is in SBO, regardless of the cause, the situation is bad and very challenging to human performance.

• The ELAP decision will be based on the physical design. NEI documents on ELAP strategies are flexible on training and decisions, resulting in many variables from plant to plant, which can lead to gaps and holes in the implementation of the strategies.

The equipment associated with ELAP is not purchased, controlled, or trained on at the same level as permanently installed equipment.

• If return of power is imminent, crews will hesitate. Personnel in non-hazard situations may hesitate more than when there is a hazard because the situation seems fine, and they may assume they have more time. Battery life is a factor. Personnel know their plant, and they monitor their battery to know how long it lasts.

• The action cannot separate detection, assessment, and decision; they are all combined. Detecting cues (e.g., opening a door to see the situation outside) is a part of the decisionmaking. Making the decision dominates the HEP. Other activities are easy; many are driven by procedures.

• With regard to the decision, the simplicity of the choice when no offsite or onsite power is available is swamped by the difficulty in declaring ELAP for a condition not being driven by external events (which is exactly what FLEX was designed for). Put another way, the operator would need to declare an ELAP condition for a situation he or she was not trained for and not a true FLEX situation. Stress would likely be high and the procedures could be viewed as incomplete for this event or condition.

• Potential negative factors affecting the HEPs include more interaction, resource management, resource constraints, delayed actions, and limited personnel available.

• The HEP is bound by the content and quality of the information and uncertainty as to generator status, decisionmakers are reluctant to go to ELAP because of the consequences. In particular, the content and quality of the information drive the bounds. If the information says the DG is trashed, that requires a different decision than I dont know what is wrong, which is similar to knowing that ac power is or is not coming back soon. Different types of LOOPs, from the hazard or grid center, add uncertainty.

• The HEP could be even worse than 5E-1 because personnel would not want to declare ELAP in this situation.

• This is a diagnostic-only HEP. The anchor point from SPAR-H for a nominal diagnostic HEP is 1E-2. The scenario description states that it is unknown when offsite power will be returned to service. The critical input indicates that restoration timing is ambiguous. That is, it is not obvious that power cannot be restored nor is it obvious that power can be restored. This is the worst of all situations, leaving the decision to the operators judgment with only 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> to decide. The decisionmaker will probably use all of the available hour, delaying the decision until the end of the hour, hoping that power will be restored or that additional information will become available to help make the decision.

• Input into the declaration includes (1) identifying that the existing conditions correspond to an ELAP, (2) entering into the required procedure within the required time constraints, and (3) prioritizing resources in recovering the EDG versus performing the required 50

battery load shedding. One panel member assumed that performing the battery load shedding would negatively impact the ability to recover an EDG by removing dc power from the EDG and/or the safety bus that the EDG supplies. Thus, there will be significant caution and reluctance to take this step, which will be difficult, if not nearly impossible, to reverse if the EDG problem is identified and is correctable. Also, there is a very strong preference for permanently installed and safety-related equipment, with which the operators have extensive familiarity, versus portable equipment with which the operators are significantly less familiar and comfortable.

Average estimated HEP in Scenario 2: [0.089 0.19 0.35] (1st, 50th, and 99th percentile, respectively)

Justifications specific to Scenario 2:

• The transition from Phase 1 to Phase 2 creates complications. Phase 1 to 2 transition can be in the range of 4-8 hours. Some plants may have only 15 minutes while others may have 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. The timelines vary greatly.

• In Scenario 2, the decisionmaker has information on the hazard and might have one field person or none on the DG. Personnel might have higher confidence that the offsite power is not back in Scenario 2 compared to Scenario 1, but they wouldnt necessarily have better information on diesels.

• In the FLEX scenario, even if the information is unknown, personnel still know that they are in a hazard event. For example, in the FLEX scenario, one can open the door and see the conditions outside. Less information makes it easier to make a timely decision, though not necessarily the right decision, to go to the known projected success path. If the diesel is lost and not likely to come back soon, then the decision is made to go to ELAP. The HEP of declaring the ELAP slowly in Scenario 1 should be worse than in the FLEX-designed Scenario 2. If it is obvious that the generator is fixable, the station is not likely to go to ELAP.

• On the other hand, C&C factors can drive the HEP in Scenario 2 higher than in Scenario 1. More C&C is needed, and many things work in parallel. Personnel may interact poorly in the plant. Communication comes in four or more channels (e.g., CR, TSC, field, load dispatcher off site), and offsite people are not trained for three-way communication. Much information is coming in, and misunderstanding is very possible. Personnel interpret messages differently and make decisions based on their perceptions, not what other parties actually meant. For example, they are getting there in an hour can be interpreted as the task is completed in an hour, while it actually meant that they are sending people there in an hour.

• Data or information can be wrongly interpreted. Data may be interpreted in the wrong order or backwards, or the time that the data were collected may not be clear to decisionmakers so they may use out-of-date information.

• Too much information can be overwhelming in a chaotic situation like the Fukushima event, which makes missing ELAP very likely.

• To declare ELAP, it is necessary to have people performing ELAP-associated actions.

Minimum staffing can be challenging in severe accidents like the Fukushima event (e.g., water hits the site or a tornado knocks out power, causing major destruction and 51

loss of diesels). Obviously, the plant is having a LOOP and an SBO. Multiple programs are going on (e.g., B.5.b - Station Blackout and Advanced Accident Mitigation Orders, Severe Accident Management Guidelines). Assessing damage, evacuating people if the site is on fire or in water, losing operators, or taking care of other things, knowing what is workingall take a long time and use up resources.

Even worse, offsite personnel cannot get to the site because the road is flooded. After all, the ELAP staffing plan may not reliably ensure that there are adequate people performing ELAP actions in beyond-design-basis scenarios.

• When no offsite or onsite power is available, the decision to declare ELAP for a condition (which is exactly what FLEX was designed for) is relatively straightforward; put another way, the operator is declaring an ELAP condition in exactly the situation for which he or she was trained. However, in the panel members opinion, the influence of this factor would be overwhelmed by the difficulty of declaring ELAP knowing the potential impact of implementing FLEX (e.g., nonrecovery of offsite power because of load shedding and choosing to use portable equipment instead of the installed, tested, safety-related plant equipment). Stress would likely be extreme.

• Training drives the HEP high. ELAP is outside the Standard Approach Training (SAT) program. There is no legal requirement for training of ELAP actions. Passing the training does not mean that personnel can reliably perform the actions. It cannot be said that they are fully trained or capable of making the ELAP decision. Site leadership development can be quite different from plant to plant.

• This is a diagnostic-only HEP. The anchor point from SPAR-H for a nominal diagnostic HEP is 1E-2. The scenario description states that there is no contact with offsite staff concerning offsite power restoration and that the extent of EDG failures has not been diagnosed. This critical input indicates that knowledge of the restoration is ambiguous. Conditions on site, however, are extremely poor, and the inability to communicate with offsite parties could make the ELAP decision easier. The conditions under which the decisionmaker is operating are extremely poor with low-impact information, poor environmental conditions, low-impact guidelines or procedures, moderately poor training and experience, low-impact teamwork, and multitasking requiring the decisionmaker to address multiple issues in parallel.

• Communication and the severity of flooding drive the high bound of the HEP.

Communication is vital but can easily fail. It is the biggest unknown. The design can be perfect, but communication can still fail. No matter how well trained, personnel cannot perform ELAP-associated actions in 10 feet of water.

Discussion:

• Operation in SBO In nuclear power plants, EOPs govern the plants while SAMGs, B.5.b, or ELAP procedures do not. Regulations require personnel to follow the basis of operation, including design basis and licensing basis. Thus, EOPs drive personnels decisions.

What leads personnel to declare ELAP in an SBO? Following the declaration of the emergency, the TSC is established in 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. The senior reactor operator keeps working on EOPs, letting the TSC determine ELAP. While waiting for power to be restored, the MCR crew pulls the ELAP procedure and waits for all the information to arrive. Amid many duties and distractions, the shift manager watches everything, checking the technical data, figuring out where the crew is in the technical procedures, asking each persons opinion, keeping others informed of what is happening in the 52

plant. If the shift manager believes that the plant is not going to get power back, he or she would declare ELAP.

Going into ELAP, the plant needs to shed loads. Personnel need to stabilize the plant. FSGs may have detailed guidelines on what is needed. Personnel need to verify the procedures in the FSGs and make sure that the onsite staff is adequate to perform everything.

• Challenges in declaring ELAP

- uncertainty as to when to declare (many considerations in declaring ELAP)

- AC power from different sources

- operators use of various types of information to make decisions depending on situation

- unknowns or uncertainty in plant status (e.g., how to be sure that all ac power is lost)

- lack of cues giving the personnel confidence that ac power is or is not coming back

- making sure that conditions are stabilized

- unclear how long it takes to do something

- personnel trying to restore power in parallel to doing ELAP

• Uncertainties in decisionmaking

- Declaration of ELAP means performing deep load shed. The biggest loads are the first on the list. These biggest loads are equipment like lube oil pumps, so some equipment may be ruined by going to ELAP. This could affect the economic viability of the plant. There would be an economic consequence with that decision, but the situation would not be unrecoverable as in the case of putting dirty water in the core. Because of the consequential measures (i.e., deep load shed) that go along with declaring ELAP, personnel may be reluctant to make that decision.

- Some plants may have limited batteries; the system might be lower on the decay heat curve and that might gain some time, but overall the battery time is uncertain.

- Some hazards (fire, tornado) give better cues or higher confidence for clearly knowing that power will not come back.

- Viable alternative solutions exist, such as the choice to work on diesel recovery rather than abandoning diesel and declaring ELAP.

- Personnel try to restore power in parallel with doing ELAP. If DGs are obviously fixable, the plant is not likely to go to ELAP. If the plant loses the DGs and is not likely to get them back soon, then the plant is more likely to go to ELAP. Personnel need to consider the chances of recovering from EDG failure when deciding which path to take.

- Personnel are still trying to get the diesels back, even after ELAP is declared.

They want 4,160-volt power back to regain full control, but that will not be the focus of the MCR (the TSC will focus on that). Some plants have 4,160-volt 53

FLEX equipment. Those plants would be less reluctant to go to ELAP than plants that have only 480 volts.

- The decision also depends on what the failure isif a valve is out of place, it might be possible to get the DG back, but if the crank shaft is sheared, that is another matter. Personnel are trained to restore equipment and they are quite skilled at this, so unless there is a clear deciding point to go to FLEX and stop trying to restore the DG when resources are limited, ELAP declaration is not likely to be successful.

- The importance of this action and not delaying even when there are economic consequences has been discussed. This is a difficult psychological decision to make, but it is critical that crews make it. All crews do this on the simulator in time, but that does not mean they will be able to do it in real life. For example, in one plants fire event, the personnel knew that they should abandon under those conditions, but the crews reported that none of them would abandon the plant unless they could not physically stay there.

- In Scenario 1, depending on the problems, the ability to make this decision could be easier or harder. If one EDG was out for repairs, that would simplify the decision. The failure mode of the second EDG would influence the decision. Offsite power loss would also make the decision easier.

- The non-hazard scenario would entail more hesitation than the hazard scenario because it is slower moving. The crew might have more time and might receive some field reports on the first DG that failed.

• Decisionmakers reluctance to declare ELAP compared to other reluctances

- Self-induced SBOPlants have training on SBO every year. Self-induced SBO is similar. The plant is putting itself into a worse state that is hard to reverse. However, self-induced SBO has better information on power restoration.

- Feed & bleed (F&B)F&B involves less uncertainty or reluctance. It has been around for a long time, the criteria are clearer, and personnel are very confident in using F&B. The industry has long used F&B, and personnel have experience with it. Personnel would have hesitated to use it when it was first in place, but now there is a line in the sand after many years of practice.

- Preemptive external flooding actions based on forecasting are similar to ELAP with respect to reluctance. However, flooding actions have clearer criteria.

- CR abandonment in fireELAP is an unprecedented pathway. The reluctance to declare ELAP may be similar to feelings about control room abandonment during fire; personnel tend to try to back out (I get time, I can recover it).

• The work process for ELAP decisionmaking Once ELAP is declared, the rest is easy. During this time, many procedures are being implemented and much information is coming in, but decisionmakers are focused on this decision, and they are trained to deal with this workload.

- DecisionmakerScenario 1 is a slower event, in which the plant loses one DG and then later loses the second, so the ERO will have time to be informed.

The control room supervisor (CRS) is responsible for making the decision. He will follow his EOPs first and then call the ERO.

54

- Interaction between CRS and ERO managementData on unusual events are abundant and show that the ERO management generally stays away from the CRS and does not distract him or her during events. Management may criticize the CRS after but not during the event. Yet, some tension between the CRS and the ERO was observed in emergency drills. For example, the ERO asked questions such as, Why havent you done this yet? and the CRS replied that they did not have the information they needed. There is an element of tension in that decision. That is the function of the ERO. Typically, it is not the CRS on the phone but the supervisor or a designated person.

• ELAP training Training on ELAP guidelines occurs every 2 years, yet there is no requirement for this.

Plants do not train on ELAP annually, but they do train on SBO every year, so personnel come across the ELAP decision every year. Plants also train for extreme stress, and they are critiqued quite harshly. In real events, some people hesitate, but the rest of the team does notthey perform as a team and that compensates for some individual hesitation. On the other hand, the training is in a simulator, and the psychological burden in a real event could be high. Also, the simulator training may not well address group-thinking, as in the Robinson uncontrolled cooling event.

• Individual characteristics of decisionmakers While many research studies show that individual characteristics of decisionmakers can lead to great diversity in decisions, it may not be the case with nuclear power plant CRSs who are the most trusted SROs on site. For a given SBO situation, CRSs from the same plant should all come to the same answer. CRSs have worked together for many years and they have great similarity in their mental models. Yet, CRS mental models for decisionmaking may vary across plants.

Table 3-4. Action 5: Deep load shed in Scenario 1 This is the deep load shed for ELAP, not the normal load shed for SBO.

Definition: Deep load shed starts after ELAP is declared. This action includes the tasks of opening all of the designated dc breakers for the analyzed unit in time to ensure that station batteries will support required plant loads until the plant staff is able to align the 480-volt FLEX generator to the station battery chargers. Failure of the action is defined as missing one or more breakers. The HEP does not include opening a wrong breaker (i.e., a failure caused by inappropriately opening a breaker that should not be opened, such as dc power to required instrumentation or a turbine-driven pumps controls).

The action typically must be initiated early in the scenario to effectively extend battery life.

The action initiation is often tied to the declaration of ELAP. Declaration of the ELAP includes entry into the proper procedure(s), which direct the subsequent actions. One of those actions is to perform the deep load shed. It is one of the most time-constrained actions in an ELAP scenario. Failure of the action has a high consequence.

Specifications:

• There are 18 breaker manipulations in two locations.

55

• Missing any of the 18 breakers means failure of the action.

• Personnel have flashlights, and battery backup is available.

• There is no FLEX-specific labeling (although some plants do have this).

• One person does the work in series using self-check. It is not assumed that the load is verified after execution, and there is no feedback warning the worker of opening a wrong breaker or missing a breaker.

56

Deep Load Shed The action includes the following activities:

Assess situations and adapt guidelines as needed.

Access the locations.

Select and open the specified circuit breakers.

Perform ELAP electrical alignment and resupply critical loads using temporary cables.

Average estimated HEP in Scenario 1: [0.011 0.057 0.22 ] (1st, 50th, and 99th percentile, respectively)

Justifications for the HEP in Scenario 1:

• The equipment to be manipulated in this action includes only permanently installed breakers in two different plant locations. Eighteen dc breakers in the two locations need to be opened. The scenario description defines failure as missing any one of the 18 breakers. (This failure criterion seems wrong; if 17 of 18 breakers are correctly opened, then a significant portion of the load shedding will have been successfully accomplished). The scenario description stipulates that there is no FLEX-specific labeling of breakers.

• Skill of craft, training: Operators are trained to open breakers within the timeline and are accustomed to manipulating breakers frequently.

• If procedures are good and personnel are well trained, the chance of success is very high because these are similar to the actions in EOPs.

• The consequences of failing in the action are high, but individually, the consequence of failing one part is not high, so overall the action should succeed.

• This is a typical action that operators perform in an outage but not in an extreme environment (e.g., poor lighting). The action is somewhat time constrained and performed under high stress and in poor lighting, but otherwise fairly routine.

• The time margin is ample, but the person may have self-imposed stress. Time pressure could affect performance. The work order typically says, complete by X hours/mins. The worker may not know when the allotted time begins.

• The way the procedure is written may cause failure of this action. The ELAP procedure just directs the personnel to do load shed, but the EOP gives a list of equipment to be shed. Things to be shed may be different from plant to plant.

• No confirmation or peer checking: If the person does not do it right, there is no indication of this failure. There is indication in the CR, but the feedback is not timely.

• In general, the context for this action is characterized by the following: stress is moderately high (not extreme), experience and training are low (skill of craft but deep load shed is never done), and complexity may vary between nominal and moderately high because of multiple locations. However, operators use procedures to open and close breakers all the time, and it is a very basic action. Therefore, the HEP is relatively low compared to other FLEX actions.

• The HEPs of FLEX actions are highly dependent on plant specifics (design, layout, and other factors), especially for the load shed action. Most plants must take load 57

shed actions within the first hour. Influences deemed to be significant are lighting, layout, labeling, and stress.

• Lack of labeling is the big driver. The dc circuit breakers are all molded-case and close to each other. Many breakers have no specific labeling, but the electricians should know where things are, and they should be able to identify them. Overall, the FLEX labeling helps greatly; it is much easier with reflective labeling special for FLEX. This could reduce the HEP by a factor of 2 to 10.

• Sources of stress are the number of breakers, the limited time available (personnel may rush even if the time available is adequate), and the lack of a second check.

Average estimated HEP in Scenario 2: [0.025 0.08 0.31 ] (1st, 50th, and 99th percentile, respectively)

Justifications specific for the HEP in Scenario 2:

• The justifications for the non-FLEX-designed scenario also apply to the FLEX-designed scenario.

• Locations are typically specified with the breaker (some even have drawings). Non-licensed operators have to memorize the locations.

• Outside environmental conditions could make access to the appropriate buildings and rooms somewhat more difficult but should not impact the HEP. Even with flooding, the locations should still be accessible. If doors are locked, personnel should have keys.

• One possible difference between the scenarios is that in Scenario 2, many more actions need to be taken in an adverse environment, so personnel are likely to be more distracted and have higher stress. This difference would contribute to a higher upper bound of the HEP.

Discussion:

• Number of breakers: In reality, there could be many breakers. The deep load shed in some plants would likely involve more than 70 breakers. For the action specified here, the HEP is for 18 breakers. A higher number of breakers would contribute to the upper bound. The experts discussed this questiondoes doubling the number of breakers or having 50-100 breakers increase the failure probability?

- This depends on the success criteria; if the procedure is front loaded so that the biggest breakers are up front, then the probability would not change, but if success is really defined as opening every breaker, then the failure probability would increase.

- It would not change much as long as there is adequate time. It is not important to open all breakers. Each one is less consequential when there are many breakers.

• Time constraint: Some plants say, complete action in 2 hrs, raising the question as to when to start the action. Thus, many procedures changed to start this action by <>

so it can be complete in 2 hrs to ensure that the validated time is met so personnel do not need to make that decision on-the-fly.

58

• Training: Currently, training for deep load shed occurs every 2 years, but it is possible that it will be extended to every 4 years. There would be a pre-job brief that this is an important action. The brief typically flags what systems and equipment are going to be lost. Thus, training is generally adequate.

Table 3-4. Action 6: Restore equipment from dc load shed in Scenario 1 Definition: Some time (e.g., 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />) in deep load shed, offsite power comes back. The plant may want to restore equipment from deep load shed and return to normal operation. This action is to restore the equipment from deep load shed performed for FLEX strategies.

The action includes two tasks:

Task 6.1 Decide to restore equipment from dc load shed Task 6.2 Restore equipment, including the following activities:

• Develop work orders for restoration.

• Assess the operating status of the equipment to be restored.

• Manipulate the equipment for restoration.

Estimated HEP and justifications:

The expert panel believed that under most cases, the crew would not stop or reverse deep load shed immediately if offsite power is restored. Personnel would finish load shed because of all kinds of uncertainties. Because of the lack of knowledge of what could happen in a restoration, the expert panel advocated a uniform distribution between 0 and 1 for the HEP.

Discussion:

• Uncertainties in the action

- The plant systems can be in various states of operation.

- System behaviors can be unforeseen or unpredictable.

- No routine procedure or training can cover all situations; personnel cannot rely on a single procedure.

- The TSC, not the CR, will work on restoration guidelines at that point.

• Challenges to the personnel performing the action

- not knowing what is running

- not clear what is happening at the time

- reclose a breakernot knowing what is happening

- not knowing the consequence of restoration actions (when normal strategies were not working)

- not knowing which things automatically start when power comes back

- possible bypassing trips

• Decision to stay in ELAP or go to restorationWhat is a safe, stable state? Is staying on FLEX considered safe and stable? Or should restoration be attempted?

- The plants first choice would be going forward. You do not have people back out; the time to get the diesel back can be 2, 3, 4, or 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />. One panel member commented: I would very rarely recall or interrupt a task once it has begun because then it is hard to know and communicate the plant configuration mid-task. When I asked plants if they would stop FLEX and try 59

to restore. They said no, they have a success path. Even if they restore the DG, they would be worried that the fault might happen again. They are not going to jeopardize their success path until there are lots of people on site.

- There are no recovery procedures, and they cannot be written generically to cover all kinds of situations.

- It is not just a matter of turning everything back on. If not done in the correct sequence, major damage can result.

- It is not just an electrical issue. With FLEX pumps, there are other interaction issues, and a hole is possible.

Table 3-4. Action 7: Remove debris in Scenario 2 Definition: External hazards cause damage and may result in debris in work locations and travel routes. Debris removal is needed to clean the worksites and paths. In particular, debris removal is necessary for successful deployment of FLEX equipment. The failure of the action is defined as not removing the debris within the time margin.

HEP estimation Same as Action 6. Because of the high uncertainties, the expert panel advocates a uniform distribution of the HEP between 0 and 1.

Discussion:

• Important factors affecting the success or failure of the action include the number of path options, the tools needed, and whether the sites capabilities match the level of debris expected.

• Debris might persist after removal (especially with water, more debris may collect along the site).

• Environmental conditions greatly affect the time to complete the action.

• The personnel available can vary and can make a big difference. With minimal staffing, the plant may not have adequate qualified people (e.g., those who know how to operate the tools) to perform this action.

3.2 Results of Objective 2 and Objective 3: Expert judgments of FLEX-specific PIFs The HEPs estimated are for the specific contexts defined in the two scenarios. An HRA for FLEX actions needs to be able to quantify HEPs for various contexts. IDHEAS-G characterizes scenario context through the states of PIFs. IDHEAS-G provides a generic, application-independent set of PIFs, and each PIF is associated with a set of attributes.

Table 3-5 shows the 12 PIFs in IDHEAS-G.

60

Table 3-5. PIFs for evaluation PIF Nominal state

1. Environmental Personnel task performance is not adversely impacted by factors environmental conditions such as heat/cold, visibility, noise, radiation, water, debris, and wind.

2. Information Information is needed for personnel to perform tasks. Information is expected to be complete, reliable, and unambiguous and presented to personnel in a timely manner.

3. Human-system The HSI refers to indications (displays, indicators, labels) and interface (HSI) controls for personnel to execute action systems. HSI is expected to support human performance as designed.

4. Tools/equipment/ The tools/equipment/parts assessed in an event include everything parts availability needed to support personnel actions. They should be available and and usability readily usable.

5. Procedures These include the existence and usefulness of operating procedures, guidance, instructions, and protocols. Normally, procedures are expected to be available and to facilitate human performance.

6. Training Included in this consideration are personnels work-related experience and whether they have been trained on the types of actions, the amount of time passed since training, and training on the specific systems involved in the actions. It is expected that professional staff have adequate training requirements.

7. Teamwork factors Teamwork factors, such as work planning, refer to everything affecting team communication, coordination, and cooperation.

8. Scenario familiarity The scenario is familiar to personnel, with predictable event progression and system dynamics, and does not bias personnel in their understanding of what is happening.

9. Multitasking, The event does not require personnel to perform intermingled distraction, and parallel actions; distractions and interruptions are manageable.

interruption

10. Complexity This includes complexity in information detection, situation understanding, decisionmaking, action execution, or teamwork that demands personnels cognitive resources. Normal complexity refers to a level of complexity that does not overwhelm personnel.

11. Mental fatigue The normal status of mental fatigue refers to a situation in which personnel do not experience decrement of vigilance and ability to perform complex cognitive tasks.

12. Physical demands These are the aspects of a task that require a person to use physical capabilities, such as twisting, reaching, dexterity, or force.

Objective 2 is to evaluate the generic PIFs in IDHEAS-G, identify the PIF attributes that are pertinent to FLEX strategies, and evaluate the impacts of the relevant PIF attributes on HEPs.

61

The experts were given Worksheet 2, which consists of 12 tables, one for each PIF. Each table contains the definition of the PIF and a list of the attributes. The instructions for the experts follow:

(1) Evaluate the relevance of every attribute to FLEX actions.

(2) For every relevant attribute, assess its impact on HEPs of FLEX actions in both non-FLEX-designed and FLEX-designed scenarios; the experts rate the impact in four PIF states:

- Nominal means that the attribute is not significant enough to affect HEPs, although the attribute may increase the difficulty of performing FLEX actions.

- Low impact means that the attribute alone may not lead to human errors but may increase the chance of human errors when some other PIFs are in poor states.

- Moderate impact means that the attribute alone may or may not lead to human errors but definitely increases the chance of human errors when some other PIFs are in poor states.

- High impact means that the attribute has significant impact on FLEX actions, and just this one attribute (assuming that all other PIFs are in nominal states) can lead to a high chance of human errors.

(3) Comments on PIFs and suggested additional attributes that are missing from the list.

The experts made two assumptions in evaluating the PIFs:

• The panel evaluated each PIF state independently. That is, it did not consider possible interactions or dependency between PIFs at this point, although some interaction between PIFs is possible. (The panel left this issue for the HRA method to handle.)

• Some PIF attributes may appear to be double-counted as they are in more than one PIF. The panel treated each PIF attribute independently; the HRA method will address double-counting.

All but one of the experts completed the worksheets. The TIs consolidated the experts inputs, along with the information on PIF attributes from the literature. Overall, the experts assessments of the PIF impacts reflected the same trends as reported in the literature. Yet, the literature provides finer scales to some attributes. For example, an expert may rate several attributes as high impact, while the data in the literature show that those attributes affect human errors differently. In such cases, TIs used their judgment to determine the scale of the impact. The final outcome of this part of the elicitation is the combination of the expert judgment, data in the literature, and the TIs analysis.

To avoid missing important PIFs and attributes, the criteria for keeping a PIF or attribute in the FLEX-specific PIF model are (1) a PIF/attribute is included in the model unless all the experts assessed it as not relevant, (2) a PIF/attribute is included as long as it is considered relevant in either non-FLEX-designed scenarios or FLEX-designed scenarios, and (3) a PIF/attribute is added to the model as long as one or more experts suggested it. The outcome of Objective 2 is a FLEX-specific PIF model, consisting of 12 PIFs, each having four states (nominal, low, moderate, and high impact) characterized by a group of attributes.

62

3.3 Objective 3: Quantification of PIF effects on HEPs Objective 3 is to assess the impacts of the PIFs and attributes on HEPs. The project team used the integrated results from the experts Worksheet 2 to generate Worksheet 3, which was intended for the experts to use in quantifying the PIF effects. For every PIF, Worksheet 3 presents the definition of the PIF nominal state and the attribute characterization for low-,

moderate-, and high-impact states. The experts were asked to use Worksheet 3 to quantify HEP changes of the low-, moderate-, or high-impact state from the nominal state.

Performance of Objective 3 requires the experts to thoroughly study the large volumes of data on PIF effects compiled by the project team. Only two experts provided limited inputs on quantification of the PIF effects. More importantly, the experts expressed the view that it was difficult to estimate the change in the HEP of an action caused by a single PIF because the condition was artificial and unlikely to happen in real operations. The expert panel contended that such estimation should come from research, not from expert judgment. In particular, several experts pointed out that research should aggregate the data on PIF effects to quantify HEP changes caused by PIFs. The consolidated expert opinions and data from the literature on PIF effects should be considered together to inform HEP quantification of FLEX actions for any given scenario.

Before the workshop, the experts examined Worksheet 3 for verification. At the workshop, instead of having the experts quantify the PIF effects, the technical integrators fostered the panel to qualitatively assess how the PIF attributes could affect HEPs. The expert panel members, together, reviewed the PIF state characterization, developed examples for the attributes, and discussed how the various attributes might affect operator performance. The workshop did not seek the expert panels consensus on the impacts of individual attributes or elimination of some PIFs or some attributes. Rather, the technical integrators focused the workshop on eliciting the experts suggestions and opinions.

Following the workshop, the experts were asked to use Worksheet 3 to document their comments on PIFs and the attributes. Four experts turned in Worksheet 3. The technical integrators consolidated the experts inputs from the workshop and the experts responses in Worksheet 3. The consolidation was intended to preserve all the experts opinions even if the experts held opposite opinions. For example, one expert suggested eliminating several PIFs because they could be represented by other PIFs, while another expert objected to this suggestion because it would make the PIFs less differentiable. The integrators documented both opinions without eliminating the PIFs.

Table 3-5 documents the results of Objective 2 and 3. It has 12 sub-tables, one for each PIF.

The upper portion of the table contains the PIF definition; the middle portion shows the results of Objective 2 (i.e., the four states of the PIF, each characterized with a group of attributes); the lower portion consolidates the experts comments and discussion of the PIF.

Table 3-5 Table 3-5.1. Environmental Factors Environmental factors refer to conditions that negatively impact personnel performance, such as temperature (heat/cold), visibility, noise, radiation, water, wind, seismic aftershocks, etc. Hazards such as steam, fire, toxic gas, seismic events, or flooding can introduce environmental factors. Environmental factors either directly affect human performance, or they may aggravate the poor states of other PIFs. For example, flooding can directly impact 63

personnels execution of an action because it makes physical movement difficult; it can also aggravate some PIFs such as accessibility to a physical structure or work location or availability of tools.

Environmental factors can adversely impact human performance as in the following examples:

• Noise, smoke, and precipitation affect information detection and inter-team collaboration.

• Harsh environmental conditions, such as extreme heat or cold, may lead to early termination of situation assessment because personnel are unwilling to seek additional data to reconcile conflicts in the information.

• Harsh environmental conditions adversely impact decisionmaking (e.g., by reducing decisionmakers ability and effort in evaluating available strategies, thoroughly deliberating decisions, or mentally simulating action plans).

• Environmental conditions on travel paths and at worksites restrict personnels motor movement, reduce personnels motor skills, or limit the time that personnel can steadily perform motor activities. Example conditions include the need to wear heavy protective clothing, high water on travel paths, high winds, extreme heat or cold, earthquake aftershocks, and chemical or other toxic contamination.

Assumption for the evaluation: The impact of environmental factors is evaluated for actions outside the CRs.

PIF Attributes of the PIF state state N Personnel task performance is not adversely impacted by environmental conditions such as heat/cold, visibility, noise, radiation, water, debris, wind.

L

• low visibility (fog, smoke, precipitation)

• heat (33-41 degrees Celsius) or cold (5-10 degrees Celsius)

• wind (20-40 mph)

• wearing protective clothing

• noise

• reduced workplace accessibility M (Not specified)

H

• very low visibility

• extreme heat or cold (habitable with low sustainability)

• very strong wind ((habitable with low sustainability)

• flooded or under (still or running) water

• highly limited worksite accessibility or habitability with congested objects, debris, icing, radiation, etc.

Discussion:

• Representative cases for each scale

- NThis would be a perfect day during the week with plenty of support available.

- LThis is more representative of reality because of its higher probability of occurrence.

64

- HThis would represent extreme conditions such as a hurricane.

• Limited accessibilityThis is something included in the FLEX scenario; however, if something were to go further wrong, accessibility could be a problem.

• Worksite is flooded or underwaterSites plan for this; however, some sites have multiple locations with varying flooding levels, which could cause confusion.

• Wearing protective gearThe effect should be nominal for FLEX-designed events but may apply to other applications.

• Temperature rangeThe effect accounts for personnel having weather-appropriate gear.

• ColdOperators use their gloves to manipulate equipment all the time, but the FLEX interface is touch screen, so they have to take off their gloves. Extreme cold can affect use of the touch screen outside.

• WindPersonnel cannot use a crane (or cherry picker) in winds greater than 20 mph unless the equipment manufacturer so specifies; at 70 mph, operational personnel shelter in a hardened structure and are not allowed outside.

Table 3-5.2. Information Availability and Reliability Information is the input needed for personnel to perform tasks. Information is available to personnel via instrumentation, indicators, displays, alarms, written documents, or oral communication, among other means. Information is expected to be complete, reliable, and presented to personnel promptly and in a user-friendly manner. Information in event scenarios may be incomplete, unreliable, not presented timely, or even incorrect or misleading.

PIF state Attributes N Information needed for a task is complete, reliable, unambiguous, and presented to personnel promptly.

L

• not timely

• not organized (e.g., updated differently)

• secondary sources (primary source not available)

M

• incomplete

• conflicting

• from an unfamiliar or unclear source H

• overriding or masked

• misleading

• unreliable Discussion:

• Examples of the PIF states

- NominalPersonnel are trained on nominal states. FLEX requires that the minimum set of information be preserved. This minimum set is highly unlikely to be nominal because the SBO will result in the loss of some instruments and controls.

65

- LowThis state is anything below nominal, such as missing something with respect to what is expected based on training or getting the information from less preferred sources. This is more representative of reality and has a higher probability of occurrence.

- ModerateIncomplete Information on diesel generators and unfamiliar or unclear sources of information are probably the biggest contributors to declarations of ELAP.

- High (misleading information)Tank instrumentation if the line is frozen is a good example of a misleading instrument that might be applicable.

• For most PRA scenarios, the information should be nominal, or the human actions could otherwise be infeasible. Low impact comes into play primarily in SDP events.

For example, in ELAP, information leads to decisionmaking; information is mixed, and C&C is important for getting the information. Manual operation may require radio communication with the CR. If a scenario reaches the point where FLEX equipment is needed, radio traffic will be intense and can lead to untimely, incomplete, or unreliable information.

• One expert recommended eliminating this PIF because it is entirely redundant to several other PIFs.

• Assessment of the impact:

- ModerateGiven minimal training and lack of SAT control, the probability of failure increases.

- HighThis leads to high probability given the emphasis on FLEX.

Table 3-5.3. Human-System Interface (indications and controls)

HSIs refer to indications (e.g., displays, indicators, alarms, labels) that present information for personnel and controls for personnel to physically manipulate the systems to execute actions. HSIs are expected to support human performance. For example, advanced alarm displays in nuclear power plant CRs organize alarms according to their urgency to help operators focus on what is most important. However, HSIs may be designed poorly. HSIs may also become unavailable or unreliable in event scenarios.

PIF state Description or attributes of the PIF state Nominal The HSI refers to indications (displays, indicators, labels) and controls for personnel to execute action systems. The HSI is expected to support human performance as designed.

Low

• indication of low salience Impact

• unintuitive indications or controls

• mismatched formats or requiring conversions Moderate

• ambiguous or confusing formats, labels, alarms

• indications or controls not localized by functions (distributed spatially or temporally)

• controls lack of status indication or feedback

• related information is spatially distributed or unsynchronized 66

High

• lack of key indications

• unreliable indications

• difficult to maneuver controls

• unreliable controls

• transition in control states unknown (e.g., controls reset following trips or spurious actions)

• Examples of HSI indication attributes:

- NominalUnlikely in FLEX scenarios because some indications are lost in an SBO. Operators would focus more intently on available indicators.

- Low or Moderate o Personnel cannot readily perceive the location of indications such as alarms from certain work locations such as at the back panel.

o The scale of parameters can be confusing (e.g., gauges may not be identical).

o Making sense of indications can be challenging because of unfamiliar FLEX scenarios.

o The procedure and indications may be mismatched.

- HighThe indication is unreliable (e.g., field reporters may not effectively or reliably report uncertainty in diagnosis, indications, instruments, or C&C).

• Examples of HSI control attributes that can have significant impact:

- Feedback on controlFeedback from system is unclear or not timely.

Location of feedback is not in personnels focus of view.

- Manually open some things without confirmation (e.g., an operator starts something, and the only confirmation is the sound).

- Reset mode (system status due to reenergizing in fire events) is not obvious to personnel.

- The logic of a system changes, and personnel are unaware of it.

- Controls are in unusual states compared to what personnel are usually trained on.

- Controls are unreliableThere is a 20 percent chance that they are not working properly, if you have to push and hold a button for a while to start something.

Table 3-5.4. Tools, Parts, and Instrumentation availability and usability In event scenarios, special tools, parts, or instrumentation may be needed. Examples are portable radios, portable generators, torque devices to turn wheels or open flanges, flashlights, ladders to reach high places, and electrical breaker rack-out tools. The tools assessed in an event include everything needed to support personnel actions. For example, use of a portable diesel pump would include the vehicle towing the pump to its staging location, the water source, pipes, hoses, junctions, and fittings (e.g., to connect to fire hydrants).

67

PIF state Description or attributes of the PIF state Nominal The tools/equipment/parts assessed in an event include everything needed to support personnel actions. They should be available and readily usable.

Low

• Tools are difficult to reach or use.

• Tools are unfamiliar (i.e., personnel do not know how to calibrate or use the tools).

• Failure modes or operational conditions of the tools are not clearly presented (e.g., in ranges, limitations, and requirements).

• Tools/parts/instrumentation lack clear labeling.

• Instructions for using equipment/tools do not state what to do if equipment/tool is operating outside of the specified range.

• Critical tool is not working properly because of aging, lack of maintenance, lack of power supply, incompatibility, improper calibration, or some other reason.

Moderate Not specified.

High

• Critical tools are not available.

• Parts are missing.

Discussion:

• Tools are unfamiliar, difficult to reach or to useThis should not have an impact because it is skill of the craft. However, personnel may be unfamiliar with some tools such as those for debris removal (front loaders, chainsaws). Under minimum staffing, people who do not normally use this kind of equipment may be using these tools (e.g., security personnel may need to use a chainsaw).

• Tool failure modes are not applicable because these are simple tools.

• Tools or parts are not availablePlants have configuration and control of tools.

Tools are generally well maintained. There are N+1 tools on site. They should be available because of FLEX design, surveillances, and audits. All plants are required to inventory the boxes. Most tamper seal the box, and people generally dont break the tamper seal. Some sites have a special tool for access that they must store in the FLEX area. Tools are not likely to be missing from the box.

• The occurrence of this attribute has a low probability but could be relevant. The lack of requirements in technical specifications for surveillances of FLEX equipment and the lack of SAT controls on training make this a highly vulnerable area for failure.

- Special tools to manually backup power are security guarded with electrical protection equipment. Yet, they could become inaccessible during an SBO due to problems with locks or unavailability of authorized personnel.

- Some environmental conditions can affect the reliable use of tools:

o Electronics are sensitive to radiation.

o A chainsaw could die in a heavy downpour.

o Electric-powered tools are not resistant to rain.

• One expert recommended eliminating this PIF. The impact on the FLEX strategy of tools, parts, and instrumentation is either 1.0 or 0.0. In other words, either the parts, 68

tools, and equipment will work or they will not, and this PIF does not add substantial value.

• Feasibility versus reliability If a worker needs to get a socket wrench and it is not in the bag, that does not mean the action not feasible. There are wrenches on site, though it might take some time to get another at the warehouse. So, the action is deemed to be feasible. However, getting the wrench from the warehouse would affect reliability. For example, if the time margin is 30 minutes and it is a 10-minute walk to the warehouse, this uses up some of the margin. The worker can go to the warehouse to get a wrench, but reliability is reduced; in tornado or flood conditions, the warehouse may not be accessible.

Table 3-5.5. Procedures, Guidelines, and Instructions This refers to the existence and usefulness of operating procedures, guidance, instructions, or protocols. Normally, procedures are expected to be available and to facilitate human performance. However, there are situations where procedures give incorrect or inadequate guidance for human actions. Procedures may not apply to the scenario. Other common problems with procedures include ambiguity in their steps, lack of adequate detail, or conflict with the situation.

PIF state Description or attributes of the PIF state Nominal Operating procedures, guidance, instructions, and protocols exist and are useful. Normally, procedures are expected to be available and to facilitate human performance.

Low

• Procedure format or logic is difficult to use.

Impact - inadequate procedure formats (e.g., ambiguity, lack of consistency, unclear foldout instructions)

- multiple guidance documents to be referenced at the same time or frequent transitions between procedures

- requires complex calculations or logic reasoning (e.g., sequential presentation of a procedure requires the crew to go through several loops before finding the correct indications to diagnose the plant status)

Moderate

• Not specific or lacking details Impact - not specific in searching for additional information when the primary cues are not available or not reliable

- does not warn of all the conditions that should be avoided during procedure performance

- insufficient provision of contingency steps

- unclear logic such that the operators are likely to have trouble identifying a way to advance through the procedure

- no warnings about the pitfalls related to the decision High

• confusing or requires judgment

• unclear logic such that the operators are likely to have trouble identifying a way to advance through the procedure

• inconsistent procedures used for the same human action

• conflicts with existing policies, requirements, or other documents 69

• available procedure does not fit the situation (e.g., needs deviation or adaptation)

• procedure unavailable thus personnel have to find ways to perform the task based on their knowledge

• misleading Discussion:

• FSGs versus plant procedures (EOPs, ARPs, AOPs):

- FSG development is a part of implementing the FLEX program. Thus, FSGs should be adequate. They are prescriptive enough. The FSGs have OR logic, but that is not much different from the EOPs. The FSGs have been validated. The validations demonstrate FSGs on sunny days, add variations, and have margin.

- Both FSGs and EOPs have conditional steps and some flexibility in what can be done in parallel. From a procedural perspective, FSGs are adequate for the purposes for which they are designed.

- If EOPs/ARPs/AOPs are the bar for nominal, then FSGs are generally not as nominal as EOPs because of their lack of details. This requires more judgment by personnel, who have less experience in improvising FSGs than they have with EOPs.

• Judgment in procedures:

- FSGs are clear, but their details need judgment. Because FLEX events are not known, FSGs are far more flexible and less specific than EOPs/ARPs/AOPs. FSGs typically have two or more options. Personnel may pick the wrong option compared to following the single success path most often presented in EOPs.

- On the other hand, procedures that require judgment are not necessarily bad. For example, there are three routing options, and personnel pick the appropriate one based on the event and the location of the debris.

FSGs should not be criticized for demanding this kind of judgment.

• Multiple procedures: Following ELAP declaration, about 15 procedures are in use. Yet, having the different FSGs is much like pulling out many AOPs at once.

Typically, multiple people are doing multiple procedures; it is not a case of one person doing multiple procedures. Multiple procedures affect performance only when they are intermingled (e.g., executing one procedure depends on how other procedures are doing).

• Procedure details and level of quality: FSGs are more generic so they can be flexible. They cannot be as prescriptive as EOPs because there are unknowns and unknown unknowns in scenario details. An FSG for the same action can vary largely in the level of details and quality from plant to plant. For example, deciding to remove debris is not a hard decision because it is obvious. However, in a flood scenario, personnel may not be able to use the primary location, but the procedures may not describe what secondary locations should be used; there is a chance that either the specified or unspecified location might be underwater.

70

• Configuration and control of FLEX procedures: FSGs can change over time, yet administrative and configuration control may not update the changes. This is a vulnerable area. There are no required surveillances in the technical specifications, which means that the FSGs can be revised at the will of the licensee. Also, the lack of SAT means that operators could be trained on the first revision of a procedure, but the second or third revision might be in effect when they need to use the procedure.

• Procedure is ambiguous: The entry condition to an FSG may not be clear (for example, the cues are ambiguous for going to load shed or for waiting). There are no judgment criteria on how to dispatch because each FSG is referenced in ECA 0.0 and is driven by the EOP.

• Procedure is available but does not fit the situation: This would not be expected because of the FLEX design, but it is possible and could affect HEPs.

Procedures do not fit in many SDP examples, such as hurricanes. For example, a procedure directs an action, but there is water in the area. The FSGs have more flexibility to adapt to unfit situations. If the flood comes in differently than expected, the primary path may not work; other paths are not designated but will work. In that case, personnel are going to deviate from the FSG. However, the possible deviations are not tested or validated as EOPs. In general, the FSG directed actions are less reliable.

• Procedure is misleading: This should be rare but can have significant impacts if it does happen. Examples are turning an instrument the wrong direction, operating the wrong train, or placing the wrong phase of three-phase generator.

Table 3-5.6. Training and Experience This PIF refers to training that personnel receive to perform their tasks. Included in this consideration are personnels work-related experience and whether they have been trained on the type of event, the amount of time since training, and training on the specific systems involved in the event. It is expected that adequate training is required for professional staff.

Yet, training may not address all possible event scenarios. For example, nuclear power plant operator training focuses on use of normal and emergency operating procedures; the training may not adequately emphasize how operators need to develop novel strategies to handle unusual accident or hazard situations.

PIF state Nominal Included in this consideration are personnels work-related experience and whether they have been trained on the type of actions, the amount of time since training, and training on the specific systems involved in the actions. It is expected that adequate training is required for professional staff.

Low

• Training frequency is low (greater than 6 months).

Impact

• Training duration is inadequate.

Moderate

• Inadequate specificity for the following:

Impact - urgency and the criticality of essential information such as key alarms

- procedure adaptation

- system failure modes 71

- system design to the level of detail needed for responding to the situation

• Inadequate training on procedure adaptation: Training focuses on following procedures without adequately preparing personnel to evaluate all available information, seek alternative interpretations, or evaluate the pros and cons of procedural action plans.

High

• The training lacks practicality:

Impact - lack of hands-on training on action execution (e.g., virtual training, classroom training, or demos only without hands-on practice)

- lack of experience or training on procedures, guidelines, or instructions for the type of event (e.g., using nonoperators to perform some ex-CR actions)

• Action context is an infrequent part of training (more than 6 months between sessions on this topic), or personnel rarely perform the actions in a specific context.

• Personnel are not trained for the type of actions.

Discussion:

• Training frequency: At present, FLEX training may occur once every 2 to 4 years; it has been proposed to give FLEX training every 6 to 8 years. After 4 years, proficiency will decline steeply, especially in the operation of big equipment like a fire truck.

• Adequacy and duration of training: Training should be a part of implementing the FLEX program, so it should be adequate. However, NEI 12-06, Section 11.6.2, states, Operator training for beyond design basis event accident mitigation should not be given undue weight in comparison with other training requirements. This statement gives a green light to the licensee to minimize FLEX training. There are no EOP-like criteria to assess the trainings adequacy. For example, at one plant, it took half a day for the personnel to start the pumps, yet the personnel might still be considered having adequate training.

• Specificity of training: Personnel are trained rarely or not at all for the type of actions.

However, there are no specifications for the content of the training, personnel who should receive the training (e.g., mechanical technicians, operators, or security staff),

or the ways in which the training is administered (e.g., is it like B5B where trainees just walk down the procedures, or do the trainees actually perform them?). Some plants try to involve personnel who would be doing the FLEX work as much as possible.

• Practicality of training:

- There is a lack of practicality. For example, at a U.S. plant, trained personnel tried to start the gas turbine (installed equipment for decades), and they could not do so without help from the vendor representative.

- It is difficult to train personnel to a high level of practicality in decisionmaking such as ELAP declaration. The problem in ELAP declaration is the uncertainty in information, but trainees must receive adequate information in training.

- The NRC credits minimum staffing for FLEX strategies. Yet, most places may not have minimum staffing all the time, so the plant uses onsite extra 72

personnel such as security staff. The non-licensed staff could be trained rarely or not at all on the types of the actions. They may also not have the same level of dedication (or reliability) as the operating staff.

Table 3-5.7. Teamwork and Organizational Factors Teamwork factors refer to everything affecting inter-team communication, coordination, and cooperation. Teamwork activities include how human actions are planned, communicated, and executed across individuals, teams, and organizations. Examples of teamwork problems seen in event analysis are problems resulting from information not being communicated during shift turnover, and loss of command and control (C&C) between the operational center team and field maintenance personnel.

PIF state Description or attributes of the PIF state Nominal Teamwork factors, such as work planning, refer to everything affecting team communication, coordination, and cooperation.

Low

• Inadequate teams Impact - inadequate teamwork resources (not enough personnel, knowledge gaps, improper team information management)

- distributed or dynamic operational teams

- inadequate team management

- lack of team verification or peer checking

- improper or ambiguous roles and responsibility assignment

- unfamiliar (e.g., newly formed) teams or lack of drills or experience together Moderate

• Inadequate C&C infrastructure Impact - inadequate coordination between site personnel and decisionmakers to adapt or modify planned actions based on site situation

- inadequate verification of the plan with decisionmakers

- inadequate supervision in monitoring actions and questioning current mission

- inadequate communication infrastructure

- inadequate decisionmaking structure High

• Poor or ineffective C&C infrastructure Impact - lack of coordination between site personnel and decisionmakers to adapt or modify planned actions based on site situation

- inability to verify the plan with decisionmakers

- lack of supervision in monitoring actions and questioning current mission

- ineffective communication infrastructure Discussion:

• If the scenario is degraded to the point where FLEX equipment is needed, it is obviously a complex accident mitigation process. C&C should be called out.

• Nuclear power plant crews are constantly drilled in maintaining cohesion. In the best case, a crew works together for an extended time. However, because of illness, 73

vacation, new hires, and other factors, the crew may become more mixed, which increases the probability of failure.

• Inadequate team information management should have a nominal or low impact because personnel make visual displays and write down information, just as they do with EOPs.

• Inadequate teamwork resources are unlikely because resource analysis is required for the FLEX event. Inadequate resources could affect the HEP.

• Unfamiliar, distributed, or dynamic (e.g., newly formed) operational teams should respond similarly to the normal EOP response, but more field operations are needed to ensure success.

• The team decisionmaking infrastructure may be inadequate: Normal decisionmaking C&C can be used, but there may be more use of 10 CFR 50.54(x).

• Team coordination difficulty has a low probability but could have an impact. FLEX implementation requires timeline and resource analysis.

• C&C infrastructure may be poor or ineffective. Given that a FLEX scenario is beyond design basis, C&C becomes a crucial element. One should consider the central characters: the shift manager, emergency coordinator, and engineering coordinator.

If these key individuals are the A-Team, all is well, but if they happen to be whoever showed up, then the story changes dramatically.

• Challenges to C&C in FLEX strategy:

- Challenges are not much different than those in normal operations, except there will be no peer checking and personnel may be without a readily available way to communicate with the CR.

- The directing person may experience information overload.

- Personnel will need a common understanding of language (e.g., flow of pumps, pump flow versus flow in vessel). In normal operations, there is a set language for understanding the state of equipment. However, FLEX strategies have a different type of equipment, posing a challenge for communication. For example, the person might say we are at 50 percent flow, and the CR operator was expecting to hear a gallon per minute flow rate. In another example, a CR operator is looking at a procedure that asks for the flow to the vessel. The operator asks, What is the flow? and the field person responds with a value, but that value is the flow of the pump, some of which may be diverted so the value is not the same.

- Reporting may be slow, or there may be no reports from the field.

- Team cohesion is not assured.

- The quality of information (e.g., status, accuracy) could be poor compared to that in EOP scenarios.

- There is a lack of verification of how well information is received.

- Workers may be unaware of the status of other parties because of multiple locations.

74

Table 3-5.8. Scenario Familiarity Unfamiliar scenarios typically pose challenges to personnel in understanding the situation and making decisions. In addition, responses could entail greater uncertainty for unfamiliar scenarios compared to familiar scenarios. In unfamiliar scenarios, personnel are more likely to perform situation-specific actions not identified in the procedures.

PIF state Description or attributes of the PIF state Nominal The scenario is familiar to personnel, with predictable event progression and system dynamics, and does not bias personnel in their understanding of events.

Low

• unfamiliar with sites of manual actions Impact

• need to unlearn familiar practices or skills

• unexpected malfunctions

• dynamic work environment Moderate

• unpredictable dynamics:

Impact - The event evolution and system responses are unpredictable.

- Feedback information is not available in time to correct a wrong decision or adjust the strategy implementation.

- The decision has unintended side effects that are hard to predict in advance.

- Personnel are unable to effectively evaluate the strategies pros and cons.

- Dynamic decisionmaking is requiredcomplex system dynamics require constantly collecting information to adjust the decision.

- Because of shifting objectives, the tasks originally given to personnel change over time. This requires a revision in personnels mental models and a plan to meet the original goals and intent.

High

• Rarely performed tasks are involved. The type of the scenario is not Impact entirely unfamiliar (knowable), but the specific event is different from personnels mental model (e.g., a non-FLEX scenario uses FLEX equipment).

Extremely

• The scenario is unfamiliar (unknowable), so personnel have no existing high mental model; innovative strategies will be needed.

75

Discussion:

• The use of FLEX is obviously unfamiliar. By definition, a FLEX scenario is beyond design basis and, therefore, beyond EOP expected response. There is a high probability for technical support staff to influence operators into unpredictable responses.

• Unfamiliarity with system failure modes is very unlikely because there are no new failure modes.

• The inability to effectively evaluate the strategies pros and cons has a low probability of occurrence since those are already defined for FLEX strategies, but it could have an impact.

• One expert disagrees with this PIF: If we are trying to describe something unknowable, i.e., black swan events, then there is no way to anticipate what we dont know for a particular HEP we are evaluating. If we are trying to describe something knowable such as lack of experience or not having exercised a particular scenario before, or unease, etc. then this is addressed in other PIFs such as training or stress etc.recommend eliminating this PIF; it is entirely redundant to several other PIFs that are included here. Additionally, this PIF would be very difficult if not impossible to quantify because it deals with black swans. An analyst would not be [able to] quantify the impact of this PIFbecause we dont know what we dont know.

Table 3-5.9. Task Complexity Task Complexity refers to a tasks demands for cognitive or physical resources (e.g., working memory, attention, mental computation, executive control). Humans tend to make errors when the demands approach or exceed their cognitive or physical capacities.

Complexity includes the following aspects:

• Information complexity: the quantity, variety, and relation of information items that personnel need to process to do the task. Information complexity demands working memory and attention.

• Process complexity: mental computation, reasoning, integration of multiple sources of information, actively seeking for cues to start a task, producing new strategies needed to perform a task.

• Criterion complexity: multiple states of task outcomes, multiple or ambiguous criteria for successfully performing a task.

• C&C complexity: the need for C&C (issuing command, executing command, reporting back) to achieve a task.

PIF state Description or attributes of the PIF state Nominal Nominal level of complexity refers to the level of complexity that demands cognitive resources within human cognitive capacities and does not overwhelm personnel.

76

Low

• Personnel must detect 7-11 pieces of information.

Impact

• Detection requires sustained attention (e.g., determining a parameter trend, monitoring a slow-response system behavior).

• There are multiple causes for situation assessment.

• Relations of systems involved in a diagnosis are complicated.

• Decisionmaking requires integration of a variety of types of information with complex logic.

• Long-lasting, non-continuous action sequences require memorization.

• Initiation of the action requires monitoring of certain parameters over a period of time or waiting for a period of time.

• Controlled actions require monitoring of action outcomes and adjusting actions accordingly.

• Multiple personnel at different locations require close coordination.

• C&C is required.

Moderate There is no definition of a moderate complexity state because complexity is Impact a continuous variable.

High

• Information is overwhelming and approaches personnels capacity Impact (e.g., detecting more than 11 pieces of information and retaining them in working memory).

• Actions are knowledge based, requiring judgment, integration, and innovation.

• Personnel are actively seeking cues for starting the action.

• Criteria for successfully performing the task are complicated (several intermingled criteria), ambiguous, or unexpected.

• There are multiple output states (not a yes/no result for a few single values).

Discussion:

The expert panel did not discuss this PIF.

Table 3-5.10: Multitasking, Interruption, and Distraction Multitasking refers to performing concurrent and intermingled actions. Personnel must frequently switch between these actions during multitasking. Switching is error prone. An example of multitasking is concurrently implementing multiple procedurespersonnel may skip procedure steps when switching between procedures. An example of extreme multitasking is a situation where personnel handle several operational systems that are in different critical states, and information about different systems may be mixed or transposed.

Distraction and interruption mean that personnel are distracted or interrupted from performing their critical tasks. Examples include answering phone calls, being asked for information, and giving ways to other activities.

PIF state Definition Nominal Personnel can focus on their primary task and do not perform intermingled parallel actions; distractions and interruptions are manageable and do not impact performance.

77

Low

• persistent or frequent distraction Impact

• frequent interruption

• relatively independent tasks performed by switching between them Moderate

• Secondary task interference: Personnel perform one main task and a Impact secondary task that has low cognitive demand (e.g., keep an eye on something).

• There are prolonged (greater than 2-minute) interruptions.

High

• Personnel engage in parallel, intermingled multitasking:

Impact - concurrently detecting (monitoring or searching) multiple sets of parameters where the parameters in different sets may be related

- concurrently diagnosing more than one complex event that requires continuously seeking additional data to understand the events

- concurrently making decisions or plans that may be intermingled (e.g., addressing multiple issues at the same time)

- concurrently executing intermingled or interdependent action plans Discussion:

• If the scenario is degraded to the point where FLEX equipment is needed, this PIF is obviously applicable.

• In an SBO scenario, crews will be attempting to restore the EDGs, and those assigned to FLEX preparation should be able to focus on their tasks.

• Decisions are already prioritized for a FLEX event, but the pre-prioritization may not match the situation; it is possible that decisionmakers will have to perform parallel, intermingled tasks.

Table 3-5.11. Mental Fatigue and Stress Mental fatigue can be caused by performing a task for an extended period of time, nonroutine tasks, and cognitively demanding tasks. Mental fatigue leads to loss of vigilance, difficulty in maintaining attention, and reduced working memory capacity. People tend to use heuristics (shortcuts) in understanding and decisionmaking when experiencing mental fatigue. Stress includes anxiety, time pressure of completing tasks, and worries about the hazardous situation.

PIF state Description or attributes of the PIF state Nominal Nominal status of mental fatigue refers to a situation where personnel do not experience a decrement of vigilance and abilities to perform complex cognitive tasks.

Low

• Cognitive activities are sustained and highly demanding (e.g., procedure-Impact situation mismatches demand constant problem-solving and decisionmaking).

• Personnel switch to a low cognitive workload after a period (more than 30 minutes) of high workload, or switch to high workload after a long period (more than 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />) of very low workload

• Working hours are long with cognitively demanding tasks.

• Personnel experience sleep deprivation, exposure to noise, disturbed dark and light rhythms, and air pollution.

78

Moderate Personnel are reluctant to execute an action plan because of potential Impact negative impacts (e.g., adverse economic impact, or personal injury).

High Not specified.

Impact Discussion:

• Personnel may be reluctant to execute an action plan because of potential negative impacts (e.g., adverse economic impact, or personal injury). This should not happen for trained crews, as specified in 10 CFR 50.54(x) and (y).

• Concern for family safety in a severe accident can contribute to stress.

• Security personnel may not have the same level of discipline, so they are more susceptible to stress.

• One expert suggests removing this PIF or always setting it to nominal because, though it may be a valid PIF, an analyst has very little capability to assess this factor.

There is no realistic, repeatable way that anyone can assess this PIF. Unless there is explicit information to assess the PIF states and their impacts, this PIF should be set to nominal. If an analyst has all the information about this PIF, such as all of the timing information, the crew complement, knowledge of whether site access had been lost or not, then the analyst might be able to make a quantified guess at fatigue.

Otherwise, any assessment of this PIF is nothing but a guess.

Table 3-5.12. Physical Demands Physical demands describe the aspects of a task that require a person to use physical capabilities, such as twisting, reaching, dexterity, or force.

PIF state Description or attributes of the PIF state Nominal Physical demands are within personnels physical limits and do not impact the correct execution of an action.

Low Conditions involve resistance to motor movement (e.g., wearing heavy Impact clothes, lifting heavy materials, opening or closing rusted or stuck valves, executing actions in water or high wind, extreme cold or heat, or unstable ground).

Moderate Actions are physically strenuous or involve an unusual or unbalanced load Impact (e.g., lifting heavy objects, opening or closing rusted or stuck valves, moving a heavy load in water or high wind).

High Action execution requires high accuracy with fine motor skills, fine motor Impact coordination, or skills-of-craft (e.g., installing or connecting delicate parts).

Discussion:

• Examples of high physical demands:

- carrying heavy things (hose and cables can be heavy)

- backing up a truck when driver cannot see obstacles such as hedges

- backing off a trailer

- removing debris (e.g., operating a chainsaw) 79

- moving heavy things in a flood

• Other PIFs, such as weather or stress, may cover some attributes of this PIF. When using this PIF, analysts should be careful not to double-count the impact on HEPs.

• One expert recommended eliminating this PIF because it is redundant to several other PIFs (complexity, stress, and time needed).

In summary, this chapter documents the expert judgments of the HEPs of the selected FLEX actions and the evaluation of FLEX-relevant PIFs along with the experts justifications and discussion. The judgments reflect the integration of the experts beliefs and thus represent the belief of the expert panel rather than that of the individual panel members. Readers should note that the HEP estimation is made for the given assumptions, scenario context, and action specifications.

80

4 DISCUSSION AND LESSONS LEARNED 4.1 Insights on human reliability with FLEX strategies The expert elicitation estimated the HEPs of five FLEX actions for the non-FLEX-designed and FLEX-designed scenarios. The outcomes of the elicitation are not only the HEP distributions, but also the experts justifications of the HEPs. The justifications provided rich insights on understanding how FLEX equipment works, uncertainties and challenges in performing FLEX actions, and factors that could lead to failure of the actions.

While the project team developed the initial outlines of the two scenarios, it was the expert panel that developed the details of the scenarios to represent the two ways of using FLEX equipment.

Both scenarios are challenging to human performance. Even though the non-FLEX-designed scenario is assumed to be non-hazardous, the experts believed that the involved FLEX actions inherited the challenges in the situation that led to the use of FLEX equipment. Thus, both scenarios have the context in which several PIFs have moderate to high impact on HEPs, such as scenario unfamiliarity and inadequate training. The expert panel believed that such context represented the current status of FLEX implementation. As a result, the estimated HEPs for FLEX actions are about an order of magnitude higher than most HEPs for well trained, proceduralized EOP actions in main control rooms. Table 4-1 summarized the HEPs of the five actions for both scenarios.

Table 4-1. Summary of the estimated HEPs Action Task Non-FLEX HEP FLEX-scenario HEP (1th, 50th, and 99th) (1th, 50th, and 99th)

Generator Decide 0.016 0.052 0.101 Transport 0.023 0.057 0.27 0.038 0.14 0.52 Connect and start 0.027 0.088 0.31 0.043 0.16 0.41 Operate 0.024 0.052 0.22 0.036 0.12 0.44 Pump Decide 0.034 0.055 0.1 Transport 0.016 0.058 0.23 0.023 0.12 0.47 Connect and start 0.019 0.078 0.27 0.036 0.13 0.45 Operate 0.017 0.05 0.21 0.043 0.14 0.44 Refill Decide 0.034 0.057 0.11 Refilling 0.01 0.046 0.28 0.072 0.14 0.36 ELAP Decide 0.046 0.31 0.66 0.089 0.19 0.35 81

Load Shed Open 18 (assumed) 0.011 0.057 0.22 0.025 0.08 0.31 breakers With the HEPs in Table 4-1, the likelihood of failure for the overall FLEX strategy in the assumed scenarios is high. Implementation of the FLEX strategies can fail due to the failure of any of the four key actions: Declaration of ELAP, Load Shed, Use of Portable Generator, and Use of Portable Pump. We asked the experts to estimate the overall HEP for failing FLEX strategies, i.e., to what extent they believed that FLEX strategy would succeed or fail in a real event. The experts estimation for the failure of implementing FLEX strategies ranged from 0.3~0.6, with an average of 0.51. While this number appeared high, the experts defended their judgment with their understanding of the current status of FLEX implementation (i.e., circa 2018). Although FLEX equipment has been purchased and their implementation was audited and validated, there were still several main drivers that potentially impede the success of the strategies. Among those drivers were FLEX training requirement, unfamiliarity due to lack of real event experience and variability in implementing FLEX strategies, and long-term management and maintenance of FLEX equipment.

At that time, FLEX training was not in the Systematic Approach Training (SAT) program and it varied across plants and over the time. The training intervals may have been two to four years or even longer. The practicality of training was also in question because there was no guarantee that personnel performing FLEX actions had hands-on training. Some experts expressed that no credit should be granted for FLEX actions unless they were in the SAT; on the other hand, some experts indicated that they would reduce their HEP estimation about an order of magnitude lower if the FLEX strategies were included in SAT.

Scenario familiarity refers to the extent that personnel understand the overall event (causes, potential consequences, system conditions, etc) and are able to predict or foresee event evolution. An unfamiliar scenario usually means big surprise, i.e., the event involves an unusual combination of the things that personnel are not familiar with and not well trained on.

Unfamiliarity with an event scenario is a strong factor contributing to high HEPs. The experts considered that both non-FLEX-designed and FLEX-designed scenarios were unfamiliar to personnel. The expert panels general remark on the success of FLEX strategies is that the strategies have not been really used yet, thus we do not have high confidence of its success.

Moreover, the experts commented that FLEX was intended to add defense-in-depth and it did not necessarily reduce risk.

While the experts were conservative on the likelihood of success of FLEX strategies, they all believed that using FLEX equipment in non-FLEX-designed scenarios helps to improve training of FLEX actions and scenario familiarity; therefore, in the long run, it helps to reduce HEPs of FLEX actions. In addition, some experts also expressed concern on maintenance of FLEX equipment and tools. The FLEX implementation orders require maintenance and administrative controls of FLEX equipment and tools, and plants have demonstrated their compliance with the orders through validation and audits. Yet, things may change some years after FLEX has been implemented and the equipment has sat unused. Frequently using FLEX equipment in non-FLEX-designed scenarios could help test the equipments wellness and validate its effectiveness in the long run.

At last, the experts indicated that the reduction of core damage frequency in PRA models may not be the proper measure of the benefit of FLEX strategies. Having FLEX strategies available 82

enhances defense-in-depth and benefits plant safety. In an extreme hazard situation, people would use FLEX equipment if they have to regardless of the high likelihood of failure.

4.2 Sanity check on the estimated HEPs The project team performed several sanity checks on the estimated HEPs:

• The HEP for the overall FLEX strategy The overall FLEX strategy includes at least four core actions: declaration of ELAP, deep load shed, use of portable pumps, and use of portable generators. The combined HEP for these actions is 0.55. In the workshop, the experts were asked to estimate the overall HEP for implementing the FLEX strategy. The mean value of the experts estimates of the overall HEP is 0.51, close to the combined HEP of the four individual actions. The experts considered that the high HEP value was reasonable because the strategy has not been performed before and unexpected factors could hinder its success. Thus, there is a high chance that the FLEX strategy will not be successful.

However, if plants reach this situation, they must respond. Even with the high chance of failure, the strategy still has a 50/50 chance of succeeding. In principle, FLEX is an additional layer of defense for safety, and it does not necessarily give much extra credit in risk assessment.

• Declaration of ELAP contributing to the high HEP of the overall FLEX strategy The HEP for declaration of ELAP is 0.31 for Scenario 1 and 0.19 for Scenario 2, dominating the HEP of the overall FLEX strategy. The high HEP is the result of incomplete information and reluctance in deciding to go to ELAP. Because of the high economic cost of ELAP, personnel at the plant may tend to choose the other preferred paths such as fixing the diesel generator. The experts compared the decisionmaking in similar actions for which the crew might show reluctance (e.g., going to feed and bleed in a loss of coolant accident event). The panel advocated a higher HEP for declaration of ELAP than for feed and bleed.

• Comparison to load shed in existing PRA models Existing plant PRAs have modeled the action load shed similarly to the FLEX action deep load shed in a different context. We identified two example HEPs for load shed in existing PRA models. The HEPs are 0.1 and 0.12. They are comparable to the mean HEPs of Deep Load Shed estimated for Scenarios 1 and 2 (0.06 and 0.08). The HEPs are within the same order of magnitude.

4.3 Expert-to-expert variability in HEP judgments The HEPs estimated by the experts for the same action can vary up to one order of magnitude.

The in-depth discussion of the scenario context and action specifications established the common ground among the experts. The variability resulted from the experts different perspectives on the technical issues, which can be traced to their experience and expertise.

For example, one expert believed that high stress affects operator performance and increases the chance of errors, while another expert believed that stress has a negligible effect on HEPs for experienced crews. Throughout the elicitation, the team also observed that the experts often anchored their judgment to experience in their home plants. Finally, the experts used different mental models to convert their beliefs into probability distributions. Two of the experts used SPAR-H to reference and anchor their estimates, while the other four experts relied more on their past experience and the information in the data packages.

83

Appendix C documents the experts individual estimates of the HEPs and annotates the specific justification that reflects the variability. While the outcome of a formal expert elicitation is the integrated distribution of the experts beliefs, the differences in the experts beliefs reveal the spectrum of variability and uncertainties in the technical issues.

4.4 Topics regarding the expert elicitation process

1) Treatment of time uncertainties One general assumption in HEP estimation is that there is adequate time margin for all the actions; thus, the time available for the actions does not affect the estimated HEPs in this report.

This is based on IDHEAS-G in which the HEP of an action consists of two parts: Pc as the failure probability of macrocognitive functions under the condition that the time effect on the HEP is neutral, and Pt as the error probability attributed to the uncertainties in time available versus time needed. This expert elicitation deals only with the first part. The effect of time on HEPs needs to be considered separately.

FLEX strategy implementation requires demonstration that the time available for FLEX actions is adequate under minimum staffing. The NRCs audits of utilities FLEX implementation paid specific attention to ensure that the time available for the actions allows for sufficient margin to the time needed to perform the actions. Thus, the effect of time on the HEP should be neutral for FLEX implementation. Yet, in real events, both the time available and the time needed for a FLEX action can vary from what was designed. Therefore, a FLEX HRA should assess Pt using the IDHEAS-G time uncertainty model.

This expert panel estimated the HEPs of the FLEX actions with the assumption of adequate time margin. Their estimation of Pc considered time criticality. If an action is time critical, personnel are under high stress regardless of how big the time margin is. Also, some experts believe that the effect of the PIF scenario familiarity has a great impact on the HEP for time-critical actions.

2) Scenario development The expert elicitation needed scenarios that were generic yet represented real FLEX applications and had enough detail for the experts to estimate the HEPs with a common basis.

Developing such scenarios was very challenging given the variations in FLEX strategy implementation. Before the workshop, the project team outlined the main requirements for the two scenarios and then worked with the expert panel to refine the two scenarios through several rounds of teleconferences and e-mail communications. At the workshop, the expert panel spent much of the first day in discussing and further specifying the scenarios. The panel did not consider some scenario details until the FLEX actions were discussed at the workshop. The final description and context definition of the two scenarios are valuable byproducts of the expert elicitation.

From this experience, the project team learned that it should have planned a separate face-to-face workshop on scenario development. Because FLEX strategies are new and their implementation varies widely from plant to plant, it is hardly possible for any two experts to have an identical perception of the scenario details without face-to-face interaction.

3) Aggregation of experts HEP estimates To obtain the outcomes that represent the expert panels beliefs about the technical issues, the project team needed to integrate the experts HEP estimates. The White Paper Guidance discusses various methods and techniques for mathematical aggregation, as well as the pros 84

and cons of the techniques. Probably the most commonly used algorithmic combination method is the linear arithmetic combination (the average of the experts distributions) and geometric combination. Hora et. al. [19] evaluated the performance of various combination rules and compared arithmetic and geometric combinations of experts distributions. He found that geometric combination performs best when experts are independent and well calibrated, but its performance deteriorates compared with the arithmetic combination as dependence increases or expert calibration decreases. He also found that geometric combinations tend to have lower variance than the arithmetic combination. In general, if the numbers vary by one or more orders of magnitude, arithmetic combination tends to emphasize higher probabilities and under-weigh lower probabilities, while geometric combination has the opposite effect.

This project used arithmetic combination to generate the HEP distributions. The team chose this method for two main reasons: (1) the variability of the experts HEPs for most of the estimated actions is within one order of magnitude, and (2) the experts estimates were not well calibrated. Because of the high uncertainties of the technical issues and lack of direct data for calibration, the project team asked the experts to estimate the central tendency (the 50th percentile) and the bounds (1st and 99th percentile). Yet, the team also performed geometric combination for comparison. The HEP tables in Appendix D show the outcomes of both techniques.

To develop results that could be easily used by current PRA software, one popular technique is to first fit each individual judgment into a probabilistic function, then aggregate the fitted functions. PRA models often use a standard probability function (the beta distribution) to approximate the probability estimates. In some cases, the fitting process was straightforward.

In others, because of the limited flexibility of the beta distribution, choices had to be made as to which portions of the experts distributions should be emphasized in the fitting process. In a few cases, the use of the beta distribution may have masked situations where multimodal distributions (representing the possibility of distinct, competing models of the world) accurately represent the current state of knowledge. Overall, it is not clear whether the use of the beta distribution has a positive or negative effect when representing the range of uncertainty of the technical community. The literature also shows controversy over fitting versus no-fitting.

Because the experts on the panel estimated only the central tendency and the bounds of the HEPs, fitting the estimates into any probability function (e.g., beta, normal, or log normal function) can potentially mask or introduce deviations from the truth. Therefore, the project team simply applied the arithmetic combination to the central tendency and bounds of the individual estimates without using any fitting.

4) Use of PIFs in SPAR-H method Two experts used the SPAR-H process to calibrate their HEP estimates. The scenario context used in this expert elicitation is characterized with the states of 12 PIFs from IDHEAS-G; only half of these correspond to the PIFs in SPAR-H. The experts were able to map the scenario context to the PIFs in SPAR-H. Because SPAR-H PIFs are defined at a high level and leave ample space for HRA analysts interpretation, it is possible to use the SPAR-H PIFs to represent a variety of factors. However, the mapping is subjective; different analysts may devise different ways of mapping, which can lead to variability in HEP estimation.

85

4.5 Lessons-learned on implementing the White Paper Guidance The expert elicitation process followed the principles and guidelines in the White Paper Guidance. Yet, because of resource limitations, the project team compromised and simplified the implementation in some areas. We observed some subsequent downsides.

1) Thorough evaluation of available data The White Paper Guidance recommends having an Evaluation workshop for the expert panel to thoroughly evaluate the available data. We used a 2-hour teleconference in lieu of a face-to-face workshop. The short teleconference was not enough for the panel to thoroughly discuss all the data in the two data packages. Following the teleconference, the experts reviewed the data packages individually. The understanding of the data and the extent to which the data were used in the judgment varied among the experts.

2) Definition and Familiarization of the technical issues The definition of the technical issues included the description of the two scenarios used, the context of the scenarios, and the specifications of the selected FLEX actions. The project team initially outlined the definitions and the experts filled in the details and refined them.

Familiarization of the technical issues is for the expert panel and the project team members to review the issues together to ensure that the experts have the same understanding of the issues. The refinement and familiarization were carried out through three teleconferences.

Yet, during the face-to-face workshop, we found that the experts still needed to fill in some details of the definitions. For example, some experts assumed that the portable generators and pumps were operated inside a shelter while others considered it as the plant-to-plant variability.

Another example is the lack of specification on the travel path for refilling water tanks because different pathways could affect the human error probability of connecting hoses to the tank. The expert panel expressed that it would have been better to clearly specify all the details upfront.

On the other hand, the experts also acknowledged that it would be impossible or unreasonable to pre-specify every detail for such complex scenarios. Because FLEX strategies are implemented differently from plant to plant, the knowledge and understanding of FLEX implementation also varies among the experts. It would be difficult for all the experts to agree on all the details without a thorough face-to-face discussion. Ideally, a separate face-to-face workshop should be devoted to develop and refine the scenarios, review the assumptions, and determine the level of detail of scenario description and action specifications.

3) Lack of piloting The White Paper Guidance recommends piloting the elicitation workshops with a small set of subject matter experts. The piloting could lead the project team to refine the technical issues, elicitation procedure, and the elicitation worksheet. It also can identify areas that need further training or things that need attention during the workshops. This project skipped piloting because the project team could not recruit subject matter experts to pilot prior to the face-to-face workshop. One shortcoming from the lack of piloting is the time management at the workshop. On the last day of the workshop, there was not enough time for the expert panel to evaluate all twelve PIFs as planned. Therefore, the project team had to remove several PIFs from evaluation.

86

4) Workshop process Throughout the workshop, the technical integrators fostered the experts to talk about the basis for their judgment and challenged them with what-if questions. One observer commented that the integrators should have put more emphasis on uncertainties - pressing the experts to think and talk about what drove the lower and upper bounds of the HEPs. Because of the limitation of time available for the experts to discuss each technical issue, the technical integrators did not press the experts to think of potential failure modes of the FLEX actions and have the experts elaborate in-depth on how the various factors may increase or decrease HEPs of the particular failure modes. Lastly, the integrators did not make the experts clearly state what data they used and how they used the data provided to them in the data packages, other than that some experts discussed their use of data when presenting their justifications. We could have allocated additional time and experts effort on these areas to obtain more information.

However, the technical integrators had to carefully manage the experts mental fatigue.

Processing a large volume of information during the workshop placed a very high cognitive demand on the experts. It is a good practice for the project team to prioritize the areas of focus during the workshop and guide the workshop to get the high priority work done without exhausting the experts.

4.6 Concluding remarks The expert judgments documented in this report were obtained through a formal, structured process; they capture the distribution of the technical communitys beliefs about the state-of-knowledge. The experts estimated the HEPs of five typical FLEX actions in a non-FLEX-designed and a FLEX-designed scenario along with the justification on how the actions may fail and what could lead to the failures. The estimated HEPs can serve as benchmarks for FLEX HRA; they are valid only for the assumptions, scenario context, and action specifications made for the expert elicitation. The experts justifications for the HEPs and their evaluation of the FLEX-specific performance influencing factors should be used to inform the use of the HEPs for scenarios outside the assumptions, contexts, and specifications.

87

REFERENCES

1. U.S. Nuclear Regulatory Commission, Issuance Of Order To Modify Licenses With Regard To Requirements For Mitigation Strategies For Beyond-Design-Basis External Events, NRC EA-12-049, March 12, 2012

2. Nuclear Energy Institute, Diverse and Flexible Coping Strategies (FLEX)

Implementation Guide, NEI 12-06, Rev. B1, Nuclear Energy Institute, Washington, DC, 2012.

3. D. Gertman, H. Blackman, J. Marble, J. Byers, C. Smith, The SPAR-H Human Reliability Analysis Method, NUREG/CR-6883, U.S. Nuclear Regulatory Commission, Washington, DC, 2005.

4. S. Cooper, J. Xing, Y. Chang, What HRA needs to support site-wide, multi-hazard level 2 PRA, International Topical Meeting on Probabilistic Safety Assessment and Analysis, 2013, pp. 1686-1696

5. L. Raganelli, B. Kirwan, Can we quantify human reliability in Level 2 PSA?,

Proceedings of the 12th Probabilistic Safety Assessment and Management Conference, PSAM-308, 2014, pp. 1-12.

6. U.S. Nuclear Regulatory Commission, The general methodology of an Integrated Human Event Analysis System (IDHEAS), NRC, NUREG-2198, 2019

7. U.S. Nuclear Regulatory Commission, Practical Insights and Lessons Learned on Implementing Expert Elicitation, NRC White Paper, 2016, ML16287A734, accessed via https://www.nrc.gov/docs/ML1628/ML16287A734.pdf, (2016)

8. U.S. Nuclear Regulatory Commission, Staff RequirementsCOMGEA-11-0001, Utilization of Expert Judgment in Regulatory Decision Making, SRM-COMGEA-11-0001, March 15, 2011.

9. U.S. Nuclear Regulatory Commission, Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program, NUREG-1563, November 1996.

10. U.S. Nuclear Regulatory Commission, Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts, NUREG/CR- 6372, Vols. 1 and 2, April 1997.

11. U.S. Nuclear Regulatory Commission, Practical Implementation Guidelines for SSHAC Level 3 and 4 Hazard Studies, NUREG-2117, Rev. 1, April 2012.

12. Chang Y. J., et. al., The SACADA database for human reliability and human performance, Reliability Engineering and System Safety 125:117-133. (2014)

13. Preischl W, Hellmich M, Human error probabilities from operational experience of German nuclear power plants, Part II, Reliability Engineering and System Safety 109: 150-159, (2013)

14. Preischl W, Hellmich M, Human error probabilities from operational experience of German nuclear power plants, Part II, Reliability Engineering and System Safety 148: 44-56, 2016.

15. U.S. Nuclear Regulatory Commission, An Evaluation of the Effects of Local Control Station Design Configurations on Human Performance and Nuclear Power Plant Risk, NUREG/CR-5572, 1990 88

16. O.V. Prinzo, A. M. Hendrix, and R.Hendrix. Outcome of ATC (Air Traffic Control)

Message Complexity on Pilot Readback Performance. Technical report by Federal Aviation Administration, Washington, DC. Office Of Aerospace Medicine, (2007)

17. D. Virovac, A. Domitrovi, E. Bazijanac, The influence of human factor in aircraft maintenance, Promet - Traffic&Transportation, 29: 257-266 , (2017)

18. Kyriakidis M, Pak, KT, Majumdar A. Railway Accidents Caused by Human Error -

Historic Analysis of UK Railways 1945 to 2012 Transportation Research Record Journal of the Transportation Research Board 2476:126-136, (2015)

19. Hora, S.C., An Analytic Method for Evaluating the Performance of Aggregation Rules for Probability Densities, Operations Research, 58(5):1440-1449, 2010.

89

90 APPENDIX A PROJECT PLAN FOR FLEX-HRA EXPERT ELICITATION A.1 Introduction Project Sponsor: NRC (RES and NRR)

Past related studies and applicability NEI White Paper (Crediting Mitigating Strategies in Risk-Informed Decision Making, Nuclear Energy Institute, Washington DC: August 2016. NEI 16-06, Rev. 4) proposed a 3-tier approach to HRA in crediting FLEX mitigating strategies (MS). Industries are incorporating FLEX equipment to their PRA. The NRC and EPRI research (Incorporating Flexible Mitigation Strategies into PRA Models: Phase 1, Gap Analysis and Early Lessons Learned, EPRI, Palo Alto, CA: 2014. 3002003151) all indicated that there are gaps in existing HRA methods for addressing FLEX-related human actions. The NRC needs a technical basis for reviewing HRA of using FLEX equipment in licensing applications.

Justification or new developments leading to the need to conduct the study There are ex-CR human action data available in the literature and in plants routine maintenance / installation / repair work. The NRC also recently developed the new HRA method, IDHEAS-G, that allows to systematically analyze human failure modes and relevant PIFs. IDHEAS-G has the capability to address HRA in ex-CR actions. The data and literature should allow for a formal expert judgment of the technical basis for FLEX HRA.

A.2 Objectives of the study A2.1 Problem statement Providing quantitative credit to FLEX strategies requires failure probabilities for equipment and human errors (HEPs) for operator manual actions performed outside control rooms. HEPs will likely dominate equipment failure rates. We need data and enhanced methods for crediting operator manual actions outside control rooms.

Data directly relevant to failure rate of FLEX manual actions are not available as of now.

Indirectly relevant data are available in the following areas:

• Ex-CR manual actions in plant equipment installation, maintenance, and repair - plants have CR reports and it may be possible for the NRC to get access to such data.

• Literature on human error data in other field (railroad, offshore oil, aviation, etc)

• Literature about the effects of various PIFs (e.g., environmental factors) on human error rates

• Models of human error probabilities in non-nuclear domain manual actions With a collection of indirect data and models, the expert elicitation will evaluate the available data, estimate a set of anchoring HEPs, and provide a basis for using performance influencing factors in HRA methods.

A2.2 Objectives and expected results of the study The project has three objectives.

91

1) Objective 1 - Estimate the HEPs of a selected set of FLEX actions under given context /

scenarios.

The actions selected will be the most critical actions in FLEX implementation such as declaration of ELAP and use of portable FLEX generators. The scenarios should represent the range of non-FLEX-designed use of the portable equipment and FLEX-designed purposes. The expected output of Objective 1 includes the HEP distributions of the selected actions along with the justifications for the estimation. The experts should indicate how the actions may fail, what may lead to the failures, and what uncertainties are associated with the HEPs.

2) Objective 2 - Evaluation of performance influencing factors (PIFs)

The expert panel will evaluate a set of PIFs for their relevance and importance to FLEX actions. For each PIF, the technical committee provides a set of attributes that can negatively impact human performance. The expert panel will perform the following:

i) evaluate the attributes and are encouraged to propose additional important attributes, ii) consider how to use the PIF (e.g., what normal vs poor status of a PIF means),

and iii) if possible, provide examples where the PIF can significantly impact human performance of some FLEX actions

3) Objective 3 - Quantification of PIF effects The experts will estimate how an individual PIF changes the HEP when its status varies from nominal to Poor.

Based on the experts evaluation of the PIFs, the project team generalizes and selects a set of PIFs / attributes and define their nominal and Poor status. The experts estimate the effects on HEPs.

A2.3 How the results will be used (including regulatory framework)

1) The HEPs of the representative human actions/scenarios can be used in risk-based licensing applications;

2) The HEPs of the representative human actions/scenarios are used as benchmarks for the IDHEAS-ECA method;

3) Experts evaluation of performance influencing factors will be served as guidance on how to use the PIFs in review of licensing applications;

4) Quantification of PIF effects can be used to estimate HEPs of human actions for various contexts (where PIF status changes).

A.3 Project organization A3.1 Level of effort Table A-1 summarizes the planned level of effort for the various components of the expert elicitation.

92

Table A-1. Level of Effort Indicators for Designing the Expert Elicitation Components Level of effort Project team o A small team with individuals taking more than one role o One or more individuals fulfill each role Identification of o Via informal group meetings technical issues Develop datasets o By the project team members Composition of the o 3-5 resource experts expert panel o 1-2 combined technical integrators and lead integrator Expert panel o Limited electronic/remote tutorial sessions familiarization, training, and piloting Workshops o 1 face-to-face, structured, interactive workshop Peer review o Participatory peer review of the full elicitation process A3.2 Project schedule Table A-2 shows the project schedule. The dates are tentative and subject to change.

Table A Tentative project schedule Activity Dates Notes Project beginning Oct 1 Required deliverable date March 31 Technical team Preparation Oct-Dec (determine technical issues, form the panel, and prepare dataset)

Training and piloting Jan 2018 Workshop Feb 2018 Integrate the results Feb-March 2018 Deliver a draft report (white paper) March 31 A3.3 Team structure Table A Team structure Member Roles and participating steps Project team manager Run the project, conduct training and piloting, prepare information package for the workshop Project team members Team decision-making and oversight (technical sponsors) 93

Data specialist Organize data searching and compile data Expert panel Serve the experts Lead integrator Lead the technical decisions, facilitate the workshop, and integrate the judgments Peer reviewer Thoroughly review the technical and process aspects of the elicitation A3.4 Infrastructure for communicating and reporting the project

• Expert panel reports to project manager

• Project manager reports to technical integrator and project team

• Peer reviewer reports to project manager and technical integrator

• Communications among the expert panel go through the project manager A.4. Key tasks of expert elicitation and resource planning Table A-4 presents the typical tasks in a formal expert elicitation. This table will be broken into several task resource estimation tables for each personnel category with roles and responsibilities, deliverables, estimated levels of effort, and travel/lodging needs.

Table A-4. Description of key tasks Step Tasks Description (Who, what, when, and resources needed)

Step 1 1.1. Form project team: Project Manager, technical sponsors, data Define the expert specialists elicitation 1.2. Define project 1.3. Determine the level of effort 1.4. Identify technical issues /Conduct a PIRT Step 2 2.1. Form expert team: Resource experts, proponent experts, TIs, Form the expert and participatory peer reviewers panel 2.2. Determine lead TI 2.3. Select the experts 2.4. Set up contracts or paperwork with experts (e.g.,

consideration of conflict of interest)

Step 3 3.1. Develop the initial project plan Develop the project 3.2. Modify the plan based on inputs from the TI, peer reviewer, plan and other project participants 3.3. Refine workshop procedures and elicitation worksheets 94

Step 4 4.1. Compile and analyze available data Assemble and 4.2. Collect new data if applicable disseminate the 4.3. Disseminate the dataset to project participants dataset 4.4. Compile and disseminate additional data in late phases of the project Step 5 5.1. Conduct meetings to develop a common understanding of the Familiarize the technical issues technical issues 5.2. Refine the issues as needed 5.3. Modify workshop procedures and worksheets as needed Step 6 6.1. Prepare training materials Conduct training 6.2. Perform initial general training and piloting 6.3. Perform specific training on issues revealed from the initial training 6.4. Perform piloting 6.5. Modify workshop procedures and worksheets as needed based on piloting Step 7 7.1. Before workshop : Prepare workshop agent and logistics, Elicit judgment compile and disseminate the information package for the experts 7.2. At workshop : Evaluate the available data and models at the workshop 7.3. At workshop : Elicit individual judgments 7.4. After workshop : Project team compiles the results of workshop and communicates with the expert panel for verification and possible modification.

Step 8 8.1. Conduct TI meetings Integrate 8.2. Perform calculation for combining individual judgments and judgments sensitivity analysis 8.3. Communicate the integrated results with the experts for their verification and feedback Step 9 9.1. Develop the report, all of the experts review the report, and Document the incorporate the feedback from the expert panel process and results 9.2. The peer reviewers and sponsors review the report 9.3. Revise and finalize the report Step 10 10.1. Describes all planned activities for the peer review including Conduct how the reviewers will observe and review key project participatory peer activities during the course of the project review 10.2. Perform participatory review and provide timely feedback 10.3. Prepare written comments or a peer reviewed report.

A.5 Workshop The White Paper Guidance recommends having three workshops. The project plan should include the following items:

95

• Workshop outlines, including the time, duration, focus, and expected output.

• Workshop procedures, including the activities before, during, and after each workshop, how the activities are to be implemented, key personnels roles and responsibilities, lines of communications among the experts, and the ground rules of interaction.

• Experts worksheets, including the worksheets for resource experts, proponent experts, and technical integrators to work with and documents their assessment, judgments, reasoning, and notes for other experts and TIs to pay attention.

A.6 Deliverables This section describes the expectations or requirements for the final deliverables. Project deliverables should be described in sufficient detail to provide confidence that the project will meet the project objectives and realistic cost and schedule estimates can be developed. This description will also provide a basis for users of the results of the study to understand exactly what they can expect the project to deliver.

• Evaluation results of IDEHAS-FLEX model

• A set of PIFs that can significantly change the HEPs of FLEX manual actions and the quantitative estimation of their effects on HEPs

• A set of anchoring HEPs of FLEX manual actions in a range of representative scenarios

• Compiled data set - data, models, and examples of human errors in tasks similar to FLEX actions.

A.7 Project Risk Identification The plan should document the projects known risks in the expert elicitation process such as yielding biased results and suffering from mental fatigue (depending on the duration of the project) along with viable and executable mitigation strategies.

1) Getting the right expert panel. Who are the experts that can technically judge the error probabilities in FLEX manual actions? In particular, who can serve the technical integrator and who possesses outstanding knowledge and experience in the field?

2) Manage the amount of technical issues - The experts can only work on a limited set of technical problems in a 2-3 day workshop. The project needs to have sufficient anchoring points in order to stretch them to cover the whole spectrum of crediting FLEX strategies.

3) Feasibility of Objective 3 - The objective asks experts to estimate changes in HEPs caused by the change of a single PIF. This may not be feasible because the experts rarely see events in which only one PIF is poor. In other words, they do not have the expertise to judge the technical issues in Objective 3.

A.8 The need for checkpoints throughout the process Teams may be at risk of working on an expert elicitation process from beginning to end with no checkpoints for briefing key stakeholders on the achieved process only to find out, at the end, that they fell short of the expectations. Therefore, having checkpoints (perhaps via deliverables and face-to-face meetings with key stakeholders) throughout the process (particularly for long projects) would be beneficial to avoid/minimize the risk of rework at the end.

96

1) Identify key stakeholders

2) Three checkpoints for briefing key stakeholders:

1st briefing - Brief the project plan, with a focus on the technical issues, expected results, and how to use the results 2nd briefing - After piloting and before the Workshop for stakeholders to review the piloting results and assess whether the workshop can achieve the expected results 3rd briefing - Brief the preliminary results of the workshop A.9 Biography of the experts All the experts made the declaration of no Conflict-of-Interest with the project John David Hanna, Senior Reactor Analyst, USNRC, Region II Mr. Hanna is currently a Senior Reactor Analyst for the United States Nuclear Regulatory Commission in the Region III Office (Chicago) and has been in that role since 2017 and was an SRA in the Region II Office (Atlanta) from 2009. John has held numerous positions within the NRC of increasing responsibility. He has been a Senior Resident Inspector (Fort Calhoun, Turkey Point, River Bend) and Resident Inspector (Callaway and Arkansas Nuclear One). John also has experience as Acting Branch Chief and Senior Project Engineer/Project Engineer.

From 1990 to 1997, John worked for the United States Navy as a Shift Test Engineer evaluating both fast attack subs and cruisers following maintenance prior to them being returned to the fleet. John graduated from Georgia Institute of Technology in 1990 with a Bachelors Degree in Mechanical Engineering while specializing in Industrial/Organizational Psychology.

Jeffery Mitman, Senior reliability and risk analyst, NRC/NRR Mr. Mitman is degreed nuclear engineer (University of Michigan BSE-NE) with over 35 years of nuclear experience. Early career highlights included: Ten years with GE in construction, operations (shift technical advisor), startup testing (SRO certified) and outage management. Mid-career endeavors involved project management at EPRI in the areas of outage risk management, risk informed in service inspection, fire risk, spent fuel cask PRA and software development. For the last 12 years, he has worked in the NRC Office of Nuclear Reactor Regulation (NRR) as a senior reliability and risk analyst conducting evaluations of at-power and shutdown events and conditions; flooding issues; and spent fuel pool risk evaluations. He has also contributed to the development of human reliability analysis and common cause failure methods, implementing procedures and PRA standards.

Joshua Miller, Reactor Systems Engineer, NRC/NRR Mr. Miller has been with the NRC for 10 years. Started in the Division of System Safety, Reactor Systems and is now in the Beyond Design Basis Engineering Branch. He was onsite for 50 of the mitigating strategies audits.

97

Susan Sallade, Senior Reactor Operator, Three Mile Island Nuclear Generating Station Ms. Sallade is employed by Exelon at the Three Mile Island Nuclear Generating Station. She has 33 years of experience in Operations, Training, and Engineering. Her responsibilities have included positions in the Main Control Room, Operations Management, and Operations Training Management. She maintains qualifications as a Senior Reactor Operator, and a Licensed Operations Training Simulator and Classroom Instructor. Previous qualifications include Reactor Operator, Shift Technical Advisor, Core Reload Engineer, Station Qualified Reviewer for Engineering and Operations, Emergency Medical Technician, and Rescue Worker. She has owned and maintained the sites Emergency Operating Procedures since 2006. She participated in the development of the PWROG Severe Accident Management Guidelines, the FLEX Support Guidelines, EOP Maintenance, Time Critical Action Program Standard, and the Transient Response Procedure Usage Standard.

John Bowen, Consultant, Mega-Tech Services, LLC Mr. Bowen has over thirty-four years of experience in the nuclear industry with a focus on reactor operations (Senior Reactor Operator), project management and engineering, reactor systems, reactor physics, reactor fuel, electrical systems, mechanical systems, licensing, regulatory and nuclear security reviews/safety evaluation development, and new power plant systems preop/startup and testing. Sixteen years of Mr. Bowens operating experience was direct plant experience at various sites. His SRO certification was obtained at the Callaway Station in Fulton, Mo., which is a Westinghouse PWR. Mr. Bowen was also certified as a Station Nuclear Engineer for BWRs. Mr. Bowen provided technical expert assistance to support the NRC staff in the evaluation of final/overall integrated plans, develop safety evaluations, support on-site audits and verified strategies and guidance developed pursuant to Orders EA-12-049 (mitigating strategies) and EA-12-051 (spent fuel pool level instrumentation) subject to on-site inspections for compliance verification. He also provided technical support and assistance during on-site inspections/assessments of the licensees implementation of their B.5.b mitigation strategies. This support involved the review of procedures, training, engineering bases, maintenance, and plant modifications as well as documenting the results in inspection reports.

George Tullidge, former Senior Reactor Operator Mr. Tullidge has spent almost 40 years in commercial nuclear power plants. He was at Three Mile Island and involved in the recovery of the damaged Unit 2 Reactor for 5 years. He has a Senior Reactor Operators License and a degree in Physics. He has working experience with multiple Reactor SCRAMS and entries into the Emergency Plan. He has been a qualified Operations Shift Manager of a multi-unit nuclear power station and has several years of experience in Probabilistic Risk Assessment.

98

APPENDIX B ANNOTATION OF EXAMPLE REFERENCES USED IN THE DATAPACKAGES The project team compiled two data packages for the expert panel. One package consolidates data on human error rates or human error probabilities (HEPs) to inform the experts HEP estimation, the other package consolidates human error data measured or estimated at different states of PIFs to inform the experts assessment of PIF effects for FLEX HRA. The full packages are lengthy and documented separately in the NRCs ADAMS. This appendix shows some key references used in the HEP data package and the project teams annotation to the references.

Reference ID:

1. Preischl W, Hellmich M, Human error probabilities from operational experience of German nuclear power plants, Part I, Reliability Engineering and System Safety 109: 150-159, (2013)

2. Preischl W, Hellmich M, Human error probabilities from operational experience of German nuclear power plants, Part II, Reliability Engineering and System Safety 148: 44-56, 2016.

3. BENHARDT, H.C., ET. AL., SAVANNAH RIVER SITE HUMAN ERROR DATA BASE DEVELOPMENT FOR NONREACTOR NUCLEAR FACILITIES, U.S. OFFICE OF SCIENCE, TECHNOLOGY, AND INFORMATION, TECHNICAL REPORT, NO. WSRC-TR--93-581, 1994

4. U.S. Nuclear Regulatory Commission, An Evaluation of the Effects of Local Control Station Design Configurations on Human Performance and Nuclear Power Plant Risk, NUREG/CR-5572, 1990

5. US NUCLEAR REGULATORY COMMISSION, (2016) AN INTEGRATED HUMAN EVENT ANALYSIS SYSTEM (IDHEAS) FOR NUCLEAR POWER PLANT INTERNAL EVENTS AT-POWER APPLICATION, NUREG-2199, VOL. 1

6. Chandler, F., Heard, A., Presley, M., Burg, A., Midden, E., Mongan, P. (2010). NASA Human Error Analysis. NASA Office of Safety and Mission Assurance.

7. Basra G. and Kirwan B. (1998) Collection of Offshore Human Error Probability Data, Reliability Engineering and System Safety 61: 77-93

8. Thommesen, J., & Andersen, H. B. (2012). Human Error Probabilities (HEPs) for generic tasks and Performance Influencing Factors (PIFs) selected for railway operations. Department of Management Engineering, Technical University of Denmark. (DTU Management Engineering Report; No. 3.2012)

9. Williams, J. C., and J. Willey (1985), Quantification of Human Error in Maintenance for Process Plant Probabilistic Risk Assessment, in The Assessment and Control of Major Hazards, Institution of Chemical Engineer, Rugby, P.

353.

10. J Kubicek and J Holy, Simulator Data Collection in Czech Republic, presented to SACADA HRA data workshop, Washington DC, USA, March 2018

11. Wondea Jung, HuREX-Human Reliability data Extraction, Korea Atomic Energy Research Institute, presented to SACADA HRA data workshop, Washington DC, USA, March 2018

12. HIGH-ALERT MEDICATION MODELING AND ERROR-REDUCTION SCORECARDS (HAMMERS) FOR COMMUNITY PHARMACIES. HTTPS://PSNET.AHRQ.GOV/RESOURCES/RESOURCE/26381/HIGH-ALERT-MEDICATION-MODELING-AND-ERROR-REDUCTION-SCORECARDS-HAMMERS-FOR-COMMUNITY-PHARMACIES 99

Annotation of the references:

Ref. Context Example HEPs ID 1 Germany NPP maintenance human error data Transporting fuel assemblies with the fuel handling machine: 1/7 37 HEPs for a wide variety of tasks together with information about relevant performance HEP = [0.26,4.4] E-1 influencing factors.

Removing a ground connection from a 20+ HEPs for commission errors with switchgear cabinet: 1/48, information about relevant performance

[0.37,7.9] 2E-2 influencing factors.

2 More Germany NPP maintenance human error Plugging connectors to jacks in control data cabinets: 1/112 48 HEPs for a variety of tasks, many about ex- [0.1; 3.5] 1E-2 CR actions, a number of them concern Monitoring the main steam pressure over memory related or cognitive errors.

time during stretch out operation: 1/7

[0.3, 4.4] 1E-1 Connect an electrical module to an external signal = 1/1 Connect a cable = 1/33 Reassemble of component elements = 1/888 3 Savannah River Site human error database for Error in selecting controls outside CR non-reactor nuclear facilities 1E-2 for good layout 35 HEPs for human actions in nuclear waste 5E-2 for poor layout processing including maintenance and testing, facility operations, mobile equipment operation, and accident response. Error factor is estimated for each HEP when PIFs deviate from nominal.

4 Expert judgment (based on data from Failure of the operators to initiate HPI locally inspection reports) of HEPs for 9 human after a fire induced actions in NPP local control stations; The HEPs SRV LOCA: nominal HEP = 2E-1 were estimated for 2 PIFs: Functional Centralization, Panel Design.

HEPs of the 9 actions in Oconee PRA were provided.

5 Expert judgment HEPs in IDHEAS internal at- Data misleading:

power proceduralized CR actions (Good crew, Best PIF HEP = 4E-2 procedure, training, etc); HEPs for 14 crew failure modes for combinations of most Worst PIF HEP= 3E-1 relevant PIFs.

100

6 NASA space shuttle operation HEPs estimated Shuttle switch throw: 1.90x103 from three databases ISSs command error: 5.30x104; MER command error: 1.05x104 7 Human error data and HEPs for lifeboat Fail to position wheel to clear installation =

evacuation. HEPs are for controlled (vanilla) 7E-3 for vanilla and 3.6E-2 for severe and severe scenarios.

Incorrectly operate brake cable = 2E-2 for both Fail to release hooks properly = 3.7E-2 8 HEPs estimated by 19 experts for six generic Complicated routing task with stress = 0.3 tasks and four PIFs targeted at railway HEP (non trivial, familiar task ) = 1.6E-2, operations interval: 0.012-0.028 Communication, routine = 6E-3 Communication non-routine = 3E-2 Emergency scenario = 1E-1 Emergency scenarios - unknown = 3E-1 9 Human error data and HEPs for HUMAN Electrical Installation:

ERROR IN MAINTENANCE FOR PROCESS Deforming mechanical protection = 0.1 PLANT. Data have been collected from an Apprentice Training School and organizations Failure to select the correct units was about engaged in industrial maintenance to 2E-2 characterize skill acquisition in panel wiring, Failure to locate units in the correct position electrical installation, welding, milling and was about 6E-3 design draughting tasks.

Cutting failure was about 3E-2 10 Human error data collected from nuclear Many HEPs power plant operator simulation for HRA purpose.

11 Human performance data are collected from Many HEPs nuclear power plant operator training simulations. Many HEPs of control room actions are estimated from the data collected.

12 A collection of reported human error rates in Unfamiliar task performed at speed with no pharmacy idea of likely consequences = 0.5 Set a switch in wrong position = 1E-2 101

APPENDIX C WORKSHOP PROCEDURE The following procedure was provided to all participants of the face-to-face Workshop.

Procedure for 2018 Workshop on FLEX-HRA Expert Elicitation Workshop Attendees

• Two Technical Integrators (TIs)

• Six experts

• One resource specialist

• One data specialist (who also has the role of Technical Integrator)

• Two Peer reviewers

• Observers Objectives of the Workshop

1) Complete HEP estimation in Worksheet 1

2) Complete PIF evaluation in Worksheet 3

3) (Tentatively) work on error factor estimation in Worksheet 3.

Elicitation process For each HEP worksheet a) TIs briefly review the problem statement (i.e., the scenario, context, action) b) TIs facilitate expert discussion of state-of-knowledge - Panel experts talk about their understanding of the problem statements, what may drive human failure, what data source are particularly useful, uncertainties, etc.; Resource experts provide additional knowledge as needed for the topic.

c) The data specialist provides/explains/recommends relevant data in the data package upon experts request d) Expert reviews or modifies his/her homework estimation e) Evaluation and deliberation o Each expert presents his/her HEP estimates and justification/arguments tied to data set and additional experience/information; other experts challenge the presenter; the presenter may defend his/her original position or make modifications, based on inputs from other experts.

o Experts presentation should try to capture the following aspects as much as possible:

- judgment,

- evidence/examples/data supporting the judgment,

- boundary conditions within which the judgment is valid,

- exceptions and what-if consideration where the judgment does not apply,

- uncertainties as well as assumptions made for the uncertainties.

o If the expert wishes to significantly modify the estimate, he/she should briefly annotate the justification on the worksheet.

f) TIs flag areas of significant disagreement for more discussion or leave it as a parking-lot issue.

102

Workshop Structure - Roles and Responsibilities Technical integrator and facilitator (TI)

• Lead the meeting and ensure that the meeting stays on track and discussions focus on what is needed

• Facilitate elicitation and considerations of uncertainties

• Make technical decisions and resolve technical issues

• Monitor time and make sure that we have lunch breaks Panel experts

• Make and present estimation of the HEPs along with justification/arguments

• Question/challenge other experts the estimation

• Modify estimation based on the inputs as needed Resource specialists

• Participate in discussion of each problem statement

• Provide specific knowledge of their expertise at the request of TIs or experts Data specialist

• Provide data/documents upon TI and experts request

• Explain data upon TI and experts request Peer reviewers

• Observe and review the elicitation process

• Comment on the process at the end of the day

• Give peer opinions/suggestions to the TIs during breaks for quick incorporation Observers

• Observe the elicitation process

• Stay quiet - the current expert panel is balanced and additional inputs from observers disrupt the process

• Talk with Jing and Michelle during breaks for suggestions and questions

• Take notes as much as you can - Jing will compile a consolidated set of notes from all the observers and reviewers.

Workshop ground rules

1) We want the range of opinions, not consensus The ultimate objective of conducting an expert elicitation is to appropriately represent the center, body, and range of the technical communitys views about a technical problem.

Experts challenge each other and provide your viewpoint to other experts in order to elicit the state-of-knowledge about the problem. However, we do not need to debate.

We are dealing with problems with large uncertainties so every opinion has its value.

103

2) You are representing the informed technical community, not your organization While the project sponsors have legal ownership of the project deliverables, the expert panel collectively has intellectual ownership of the results. Intellectual ownership means that the expert panel takes responsibility for the robustness and defensibility of the results. Thus, the expert panel members must remember that you are not representing your employer or organization on the panel, but are serving in your own right as a recognized leader in your respective field. Each expert should also maintain independence from the other experts in the team in order to avoid (or mitigate) a groupthink bias risk.

3) Interaction and integration To represent the knowledge and interpretations of the technical community, experts should interact with each other as you accumulate and evaluate existing knowledge and make interpretations/judgments. Individual experts should make judgments based on the integration of your own knowledge and inputs from other experts. Be patient and be a good listener when others are presenting.

4) No outliers, no weighting your inputs above others Every expert is a recognized technical leader in your field. Your inputs are as important as everyone elses, even if your judgment appears to be an outlier from the rest of the panels. Being an outlier often means that you are filling a knowledge gap.

104

APPENDIX D THE ESTIMATED HEPS Table D presents individual experts HEP estimates and the aggregated HEPs. The White Paper Guidance discusses various methods and techniques for mathematical aggregation, as well as the pros and cons of the techniques. The most commonly used algorithmic combination method is the linear arithmetic combination (the average of the experts distributions) and geometric combination. In general, if the numbers vary by one or more orders of magnitude, arithmetic combination tends to emphasize higher probabilities and underweight lower probabilities, while geometric combination has the opposite effect. In addition, geometric combinations tend to have lower variance than the arithmetic combination.

We first aggregated the experts HEP estimates using both arithmetic and geometric combination. The results are shown in Table D. Overall, the geometric combination results in slightly lower HEPs for 50th percentile than the arithmetic combination. However, the geometric combination results in much narrower HEP distributions, and even produces distorted distributions. The project team decided to use arithmetic combination for the aggregation. The team chose this method for two main reasons: (1) the variability of the experts HEPs for most of the estimated actions is within one order of magnitude, and (2) the experts estimates were not well calibrated. Because of the high uncertainties of the technical issues and lack of direct data for calibration, the project team asked the experts to estimate the central tendency (the 50th percentile) and the bounds (1st and 99th percentile). We present the results of both methods in Table D for documentation.

Table D-1. Action 1.1 Decide to use FLEX generator in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 0.04 0.1 The team agrees the action is needed and utilizes the correct procedure for pump operation.

B 0.06 0.17 0.28 The HEPs are for hurried setup. If the plan to do it is just to stage it every time that they go to an EDG maintenance outage then it is a success.

C 0.01 0.04 0.1 If you are in the situation of using FLEX generator, it is a VERY bad day, so people will be under extreme stress. This is a TSC decision and needs communicating to other locations; other things are going on such as managing stuff.

D 0.01 0.05 0.1 FLEX procedures are pretty generic, on purpose left open.

E 1E-4 0.001 0.01 In SPAR-H analysis, the nominal diagnostic HEP is 1E-2, which is modified here by recognizing the additional time available supplied by the functional EDG.

F 3.5E-3 0.01 0.02 The licensee may need to go outside of existing procedures (declare 50.54(x)) because these are normal LOOP scenarios and presumably not an ELAP caused by a large external event.

Arithmetic 0.016 0.052 0.101 Mean 105

Geometric 0.005 0.023 0.06 mean Table D-2. Action 1.2 Transport generator in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 0.04 0.1 Assume that they have lost a critical function and there are procedures telling you what to do. Sunny day, no impediments.

B 0.05 0.07 0.2 Using the HEP for hurried setup. If the plan to do it is just to stage it every time that they go to an EDG maintenance outage, then it is a success.

C 0.02 0.07 0.5 D 0.05 0.1 0.3 Lots of tasks going on and lots of communication.

E 0.005 0.05 0.5 Anchored this against the base SPAR-H and THERP nominal action HEP of 1E-3, modified by the PIFs of the scenario.

F 3.5E-3 0.01 0.02 Arithmetic 0.023 0.057 0.27 Mean Geometric 0.014 0.046 0.17 mean Table D-3. Action 1.2 Action Transport generator in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.05 0.2 0.4 Assume that debris removal is complete, but may not be completely successful, or weather may remain adverse.

B 0.05 0.09 0.22 .

C 0.01 0.1 0.8 Getting out of the way for transporting, getting our backhoe, and the trailer may not fit the truck (2 v. 2.5 ball). A lot of cables cross in the protected area. Security will be involved, and things can be moved to the wrong place. If there is an outage, there will be a lot of equipment out there; there may be a problem with security opening the door; personnel get lots of interference.

D 0.1 0.2 0.5 E 0.01 0.1 0.9 F 0.011 0.17 0.3 Arithmetic 0.038 0.14 0.52 Mean Geometric 0.025 0.13 0.46 mean 106

Table D-4. Action 1.3 Connect the generator in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.005 0.01 0.05. Same as in EOPs. Connecting and operating the generator is easy with guidance, just plugging and playing connectors that are color coded. Our generators are pretty basic.

B 0.02 0.06 0.18 This is a similar situation to the FLEX scenario and has more time.

C 0.03 0.2 0.8 You have to stabilize electrical power; portable diesel is not the same, unfamiliar. Getting connected to a load center, you need to disconnect things; open a lot of breakers (proceduralized, but procedures that do not get looked at often and never physically tested) to disconnect the load center from the main control room.

You may fail to open the breakers. If you do not disconnect everything, that will trip the generator; cable can run into problems, not straight to the areas.

D 0.1 0.2 0.3 Lots of activities, phase rotation, etc.

E 5E-3 5E-2 0.5 F 3.5E-3 E-2 2E-2 Recommended value for pre-staged equipment = 1E-3 Arithmetic 0.027 0.088 0.31 mean Geometric 0.013 0.048 0.17 mean Table D-5. Action 1.3 Connect the generator in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.005 0.01 0.05. The same as non-external event. Assume that generators are indoors.

B 0.02 0.06 0.20 Assume that the connection points are indoors.

C 0.08 0.3 0.9 Extremely difficult connecting cables, multiple connections, wind blowing, rain, propping open doors.

D 0.15 0.4 0 .5 Procedural guidance not well defined; lots of information and not knowing what to do.

E 0.01 0.1 0.5 F 1.1E-2 0.17 0.3 The more pessimistic HEP values because this action would be taken in a very potentially adverse environment in terms of debris.

Arithmetic 0.0 0.16 0.41 mean Geometric 0.048 0.15 0.30 mean 107

Table D-6. Action 1.4 Operate the generator in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.005 0.01 0.05. Not much different than operating a generator in EOPs B 0.03 0.08 0.25 Assumed that the generator is sitting on a pad to use, operating outside. And the operator has to go outside to check on it.

C 0.01 0.07 0.7 Portable generators can be a big problem if not careful on how you operate it. Changing load can cause problems - If the load is variable, they trip easily. People with expertise doing these things may not be there. You may not have the right people doing it.

D 0.08 0.1 0.2 Seeing some of the training issues in FLEX audits; the training will be every 6 or 7 years. That means operators might be familiar with FLEX equipment now, but does not mean that they will continue to be familiar.

E 1.3E-2 4E-2 8E-2 F 3.5E-3 E-2 2E-2 Arithmetic 0.024 0.052 0.22 mean Geometric 0.013 0.036 0.12 mean Table D-7. Action 1.4 Operate the generator in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.005 0.01 0.05 The same as non-external event, no additional hazards at location.

B 0.03 0.09 0.27 C 0.01 0.1 0.8 You are outside operating the generator between turbines with wind upwards of 30mph.

D 0.15 0.25 0.4 Stress, mindset, unfamiliar.

E 0.01 0.1 0.8 F 1.1E-2 0.17 0.3 Arithmetic 0.036 0.12 0.44 mean Geometric 0.017 0.085 0.32 mean Table D-8. Action 2.1 Decide to use FLEX pump in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 0.04 0.1 Pumps are in a sheltered location.

108

B 0.17 0.22 0.27 In BWR scenarios, if RCIC tripped on high level and was not available, it would most likely end in failure (assuming no HPCI).

The scenario would need to also involve a path to remove water from the Torus if injection from FLEX pump. For both BWR and PWR, connecting to the RCS or the SG lines would be inappropriate with the EDG working.

C 0.01 0.04 0.1 D 0.01 0.03 0.1 Decision failure: decide to get power back and fix the trouble instead.

E 1E-4 1E-3 1E-2 F 3.5E-3 E-2 2E-2 Arithmetic 0.034 0.055 0.1 mean Geometric 0.0062 0.047 0.061 mean Table D-9. Action 2.2 Transport FLEX pump in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 0.04 0.1 Pumps are in a sheltered location.

B 0.05 0.1 0.15 C 0.01 0.07 0.5 D 0.02 0.08 0.1 E 0.01 0.1 0.9 F 3.5E-3 E-2 2E-2 Arithmetic 0.016 0.06 0.33 mean Geometric 0.011 0.047 0.14 mean Table D-10. Action 2.2 Transport FLEX pump in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.05 0.2 0.4 B 0.05 0.09 0.22 C 0.01 0.1 0.8 Hurricane, dealing with a lot of debris, location is cramped and basically a big wind tunnel.

D 0.02 0.08 0.2 In an initial test, it took personnel hours to get it up and running.

You get the right mechanics there and it works. The HEP is a bit higher than that for FLEX generator because of extra 109

considerations like where are you putting the water, clogging, hoses are heavy.

E 0.01 0.1 0.9 F 1.1E-2 0.17 0.3 Arithmetic 0.023 0.12 0.47 mean Geometric 0.042 0.12 0.4 mean Table D-11. Action 2.3 Connect FLEX pump in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.005 0.01 0.05. Pumps are in a sheltered location, so it is much like in EOPs.

B 0.02 0.06 0.18 C 0.03 0.2 0.8 Failure of connecting here is difficulty in reaching the connectors and they are far away (100s of feet). Sometime operators have to run the hoses through protected routes. There are suction/discharge, floating/snaking around for opening isolation valves, difficult areas, hose running over hose, making connections in water, putting long hoses through water.

D 0.05 0.15 0.3 Need to be depressurized, pump size matters. Took many hours to get it running; hoses are heavy; where you put it, discharge to where? Did you close all the isolation valves? Absence of indications.

E 0.01 0.1 0.9 While connecting the pump could be simpler than transporting, I do not have sufficient experience to meaningfully quantify these potential differences. Therefore, the analysis for HEP 1.2 was used for HEP 1.3.

F 3.5E-3 E-2 2E-2 Arithmetic 0.019 0.078 0.27 mean Geometric 0.011 0.098 0.27 mean Table D-12. Action 2.3 Connect FLEX pump in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.003 0.01 0.02 Staged in area protected from weather. If not protected, the HEP would be worse by a factor of 10.

B 0.01 0.07 0.18 No charging line, a long hose run, lots of stairs potentially, multiple connection points available. There are not many places you can connect the pump incorrectly. They have different 110

options, but the connections are pretty specific and the procedures locate the connections.

C 0.08 0.3 0.9 Difficult to access connection points, hanging off the rafters to get there, if that area has been damaged there is a very high probability that you will have difficulty accessing the connection points.

D 0.06 0.2 0.3 E 5E-3 5E-2 0.5 F 1.1E-2 0.17 0.3 Arithmetic 0.036 0.13 0.45 mean Geometric 0.019 0.089 0.28 mean Table D-13. Action 2.4 Operate FLEX pump in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.005 0.01 0.05.

B 0.03 0.08 0.25 C 0.01 0.07 0.7 Has to be tightly coordinated with control room D 0.05 0.1 0.2 You start it, and you can have a lot of connections so keeping it all going has some difficultymonitoring the strainers and cleaning them, etc.

E 1E-3 1E-2 5E-1 Once the pump is staged and connected, operating the pump should be comparatively straight forward in comparison to transporting and connecting.

F 3.5E-3 E-2 2E-2 Arithmetic 0.017 0.05 0.21 mean Geometric 0.026 0.078 0.20 mean Table D-14. Action 2.4 Operate FLEX pump in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 0.08 0.1 Cleaning of strainer causes this to be a higher HEP. Refueling would make worse by a factor of 4.

B 0.01 0.08 0.2 These are not as complicated to operate. The complicating factors would be more in line with diagnosing things.

C 0.01 0.1 0.8 Best base is no standing water, debris cleared out. This is a continuous action - How to do water level control? Monitoring gauges, operator watches and changes floor speed.

111

D 0.15 0.3 0.5 Many steps for cooldown and stabilization, the action is about 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />.

E 0.01 0.1 0.8 F 1.1E-2 0.17 0.3 This action would be taken in a very potentially adverse environment in terms of debris, hence the more pessimistic HEP values.

Arithmetic 0.043 0.14 0.44 mean Geometric 0.022 0.14 0.40 mean Table D-15. Action 3.1 Decide to refill CST in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.005 0.01 0.05 No external event, no distractions. Source of secondary water is a very high priority.

B 0.18 0.25 0.4 Have to decide to use this in a non-flex situation.

C 0.01 0.05 0.1 Assume fire water not available. Do not know how much damage there is, but you are in a difficult mess if you have to climb up to the manhole. Fire pump should be the choice.

D 0.01 0.03 0.08 Putting non-ideal water may cause reluctance.

E 1E-4 1E-3 1E2 The anchor point from SPAR-H for a nominal diagnostic HEP is 1E-2. This is not time critical, the additional time and obvious diagnoses dominate the HEP.

F 3.5E-3 E-2 2E-2 Arithmetic 0.034 0.057 0.11 mean Geometric 0.006 0.039 0.056 mean Table D-16. Action 3.2 Refill CST in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 0.05 0.1 Pump is low pressure, not as large. Transport is easier.

B .06 .2 .4 Must do it in the right amount of time, not a normal way of filling the CST.

C 0.01 .1 .8 Use of FLEX pump can be more challenging than use of fire pump in EOPs - not sure about damage, do not know connections ,difficult getting there and getting hoses inside.

Challenges in command and control - prioritize things, resource management, barely any training on command and control in FLEX. Control room operators use indicators, do not know what others are doing, do not know if indications are reliable, do not have optimal indications, lack of communications.

112

D 0.02 0.04 0.1 E 1E-3 0.01 0.1 Assumed the core cooling had been successfully established, thus supplying additional time to perform this action - this lowered the action HEP.

F 2E-3 5E-3 9E-3 A relatively optimistic value to this HEP largely due to the fact that refilling the CST from any number of sources is often performed, trained on and (it is believed) a formal JPM for operator qualifications.

Arithmetic 0.01 0.046 0.28 mean Geometric 0.031 0.08 0.25 mean Table D-17. Action 3.2 Refill CST in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.05 0.2 0.4 More distractions. May not make determination in time, may underestimate debris removal time.

B 0.08 0.15 0.5 Crazy wind and rain and whatever else.

C 0.08 0.3 0.9 You are not having your regular crew do this in ELAP. The people are not trained, unfamiliar with the ways to the top of the CST, especially in bad weather.

D 0.02 0.05 0.15 E 1E-3 0.05 0.5 F 2E-2 0.05 0.09 A lower value (i.e., more optimistic) to this HEP. This action is frequently performed by operators while at power and there are usually a number of different methods, and they are trained on it (JPMs). Higher stress and difficulty in performing this action in an adverse environment.

Arithmetic 0.072 0.14 0.36 mean Geometric 0.033 0.078 0.25 mean Table D-18. Action 4 Declaration of ELAP in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.06 0.2 0.4 If return of power is imminent, crews will hesitate. Non-hazard may have more hesitation than hazard because things seem fine, and you may assume you have more time.

B 0.005 0.25 0.5 Failure modes of second EDG and offsite power loss would influence ease of decision. If one EDG was out for repairs, that would help the decision.

C 0.05 0.1 0.9 - Difficult communication (CR, TSC, field, load dispatcher off-site, and others), at least four channels. (off-site people not trained for 3-way communication) 113

- Lots of information coming, very easy for misunderstanding.

- Misinterpretation of the data, e.g., order or backwards.

Overall, FLEX equipment is not purchased, controlled, trained at the same level as permanently installed equipment. It should not be used in EOPs.

D 0.1 0.3 0.5 Declaring the ELAP slowly is worse than the FLEX-designed scenario, and the driver was the hesitation.

E 0.05 0.5 0.9 There will be a significant caution and reluctance to make this action, if not nearly impossible, there is a very strong preference for permanently installed and safety related equipment, which the operators have extensive familiarity with, versus portable equipment that the operators are significantly less familiar and comfortable with.

F 0.01 0.5 0.8 Difficulty in declaring ELAP for a condition not being driven by external events (which is exactly what FLEX was designed for);

the operator would need to declare an ELAP condition for a situation he/she was not trained for and not a true FLEX situation; stress would likely be high and the procedures could be viewed as incomplete for this event/condition.

Arithmetic 0.046 0.31 0.66 mean Geometric 0.03 0.27 0.6 mean Table D-19. Action 4 Declaration of ELAP in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.05 0.1 0.3 You have information on the hazard rolling around. Less information is easier to make a timely decision, not necessarily the right decision, but to decide to go to the known projected success path.

B 0.001 0.01 0.1 The failure on this is mainly going to be based on the information that they get back from operators working on the EDGs. This should be an easy call going through the EOPs. The complicating factors would be whether or not operators would think that they could get EDGs started.

C 0.3 0.5 0.9 Lots of things going on, assessing hazard status and damage, evacuation and loss of operators; operators take care of other things, figuring out what is working - those take a long time. It uses up all your resources. Information is overwhelming but uncertain or incomplete.

ELAP can directly damage the core. ELAP is outside SAT. No legal driver. You cannot say that personnel are fully trained or capable to make ELAP.

All of these factors make it very likely to miss ELAP. Without a design basis requirement, it may not happen. Unless they have it in the SAT requirement, no credit should be granted.

D 0.08 0.25 0.3 In the FLEX scenario you can open the door and see the conditions outside and you do not have the TSC/ERO - in this 114

case less information is easier to make the decision (not necessarily better).

E 0.1 0.8 1.0 This is a diagnostic only HEP. The anchor point from SPAR-H for a nominal diagnostic HEP is 1E-2. There is no contact with offsite staff for offsite power restoration and the extent of EDG failures has not been diagnosed. These critical inputs indicate that knowledge regarding restoration is ambiguous. Conditions on site are extremely poor.

F 5E-3 0.1 0.2 The operator would be declaring an ELAP condition for exactly the situation he/she was trained for; difficulty of declaring ELAP knowing what the potential impact of implementing FLEX would be. Stress would likely be extreme.

Arithmetic 0.089 0.19 0.35 mean Geometric 0.029 0.14 0.34 mean Table D-20. Action 5 Deep load shed in Scenario 1 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 .06 0.1 Operators are trained to open breakers within timeline. Plenty of time margin but could be self-imposed high stress.

B 0.009 0.02 0.15 In reality, manipulating breakers is something that the operators do often. There could be a lot of breakers.

C 0.02 0.09 0.3 DC are all molded-case circuit breakers, close to each other.

Humans make errors in picking the wrong one under high stress.

D 0.01 0.1 0.3 The main driver to the HEP is no labeling, but the electricians should know where things are and they should be able to go do it. Also, the instructions can be ambiguous: complete by 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> - How do I know when I start, starting time depends on ELAP XYZ.

E 0.001 0.01 0.1 The equipment to be manipulated in this action includes only permanently installed equipment into two different plant locations. Poor lighting and stress.

F 0.01 0.1 0.3 All of the HEPs are highly dependent on plant specifics (design, layout, etc.) but for the load shed action it is even more so; however a majority of plants must take load shed actions within the first hour and the HEP recommended here was based on that assumption; Ergonomics was set to poor based on the multiple actions that would need to be taken in an adverse environment.

Stress was set to high.

Arithmetic 0.011 0.07 0.22 mean Geometric 0.011 0.062 0.21 mean 115

Table D-21. Action 5 Deep load shed in Scenario 2 Expert Estimated HEP Note - Specific basis the individual expert used for his/her estimation 1th 50th 99th A 0.01 .06 0.1 Same as non-external event. Assume no load verification B 0.009 0.02 0.15 This would not change much from one scenario to another.

C 0.02 0.09 0.3 D 0.05 0.2 0.5 No FLEX-specific labeling is the big driver. It should not be difficult access but has poor lighting. It would be much easier with reflective labeling special for FLEX, maybe a factor of 10.

E 0.005 0.05 0.5 F 0.01 0.1 0.3 Same as in Scenario 1 Arithmetic 0.025 0.08 0.31 mean Geometric 0.018 0.087 0.26 mean 116