ML19092A463

T1 Generalization and Integration of Human Error Data to Inform HEP Estimation

ML19092A463
Person / Time
Issue date:	03/14/2019
From:	James Chang, Jing Xing NRC/RES/DRA/HFRB
To:
J. Chang 415-2378
Shared Package
ML19092A461	List: ML19092A463 ML19092A464 ML19092A470 ML19092A471 ML19092A472 ML19092A473 ML19092A474 ML19092A475
References
Download: ML19092A463 (30)

Text

Generalization and Integration of Human Error Data with IDHEAS-G to Inform HEP Estimation Jing Xing, Y. James Chang US Nuclear Regulatory Commission Presentation to 2019 NRC HRA Data Workshop 1

Whats next in human reliability analysis

- DATA-based human error probability estimation

1. of errors (failure mode)

HEP (failure mode under specific context) =

1. of Occurrence (under the context)

• Ideal world:

- The same task for a failure mode is repeated thousands of times with the same people under the identical context;

- Do this for all possible contexts Failure modes # Occurrence Context Variety Well-defined Known, sufficient Context clearly Sufficient data for all failure modes number of task defined and failure modes and occurrences repeated contexts 2 3

Human error data: The ideal world and reality

1. of errors (failure mode)

HEP (failure mode under specific context) =

1. of Occurrence (under the context)

• Reality:

Good news - There are human error data in human error analysis, operational databases, cognitive and human factors experiments.

Not so good news X Failure modes not analyzed and in great variety X Context undocumented and/or unrepeated X Limited coverage - limited failure mode / context tested X Not talking to each other 2

Human error data - generalization and integration

• Human error data exists

- From various fields, in different formats, varying context and levels of details

• Data generalization and integration for human reliability analysis

- The General Methodology of Integrated Human Event Analysis System (IDHEAS-G) has an inherent structure for generalizing and integrating human error data 4

Outline

• Human error modeling and HEP estimation in IDHEAS-G

• Generalization of human error data with IDHEAS-G

• Integration of generalized human error data to inform HEPs 5

IDHEAS-G Modeling human failures and estimating HEPs Human failure event Modeling HEP estimation Cognitive Human action / failure modes tasks (CFMs)

HEP(CFM) =

f(PIF1, PIF2, PIF3)

Performance Context influencing factors (PIFs)

IDHEAS-G Modeling human errors and estimating HEPs Human failure event Modeling HEP estimation Cognitive Human action / failure modes tasks (CFMs)

HEP(CFM) =

f(PIF1, PIF2, PIF3)

Performance Context influencing factors (PIFs)

Demonstration of IDHEAS-G cognitive failure modes Failure of Failures of processors Behaviorally observable macrocognitive failure modes function D1- Incorrect mental D3-1 Failure to use Failure of Detection model for Detection secondary information D2 - Fail to attend to D3-2 Key alarm not Failure of sources of information attended to Understanding D Fail to perceive D3-3 Key information Failure of the information Decisionmaking not perceived or misperceived D4- Fail to verify and Failure of Action modify detection D3-4 Parameters Execution incorrectly monitored D5- Fail to retain or Failure of Teamwork communicate Information 8 10

Demonstration of IDHEAS-G PIF structure Personnel / team / Task /

Context Environment System situation organization

- Accessibility - System and - Information

- Procedures and habitability I/C opacity completeness and

- Training reliability

- Visibility - Tools and - Work process PIF - Scenario familiarity

- Noise parts - Staffing

- Multitasking,

- Heat/coldness - HSI - Team and Interruption, and organization distraction

- Resistance to factors physical - Task complexity movement PIF - Stress and time

- Alarm not salient pressure attributes

- Mode confusion - Mental fatigue

- Information - Physical demands masking

- Ambiguity of Indicators

IDHEAS-G HEP quantification model Assumption 1: Two types of PIFs: Base PIFs (Information, Scenario familiarity, and Task complexity) and modification PIFs.

PIF2 PIF1 1

E-1 HEP E-2 E-3 E-4 State of Base PIFs Assumption 2: The combined effect of multiple PIFs on the HEP generally can be modeled with a simple linear sum of individual effects, where the PIF effect is calculated as the weight on the HEP, i.e.,

W=(HEP at a poor PIF state - HEP at the base state) / (HEP at the base state)

IDHEAS-G HEP quantification model Calculating the HEP of a CFM for a given set of PIF states IDHEAS-G uses the following equation to calculate the HEP of a CFM for any given set of PIF states provided that all the PIF weights and base HEPs are known:

HEPCFM = BPCFM x (1+W1+W2+W3, ) x C x Re, Where BPCFM is the base HEP of a CFM for the given states of the three base PIFs, Wi is the PIF weight for the given state of modification PIFs; C is a factor denoted to interaction between PIFs, and it is set to 1 for linear combination of PIF impacts unless there is data suggesting otherwise; Re is a factor denoting to potential recovery from failure of a task; it is set to 1 unless there is empirical data suggesting otherwise.

Outline

• Modeling human errors and estimating HEPs in IDHEAS-G

• Generalization of human error data with IDHEAS-G

• Integration of generalized human error data to inform HEPs 12

Generalizing human error data to inform human error probability estimation HEP = f(states of performance influencing factors)

Data source 1 Data source 2 Tasks Context Tasks Context Failure PIFs Failure PIFs modes modes A generic, adaptable set of failure modes and PIFs 13

Generalizing human error data with IDHEAS-G CFMs and PIFs Generalize data: Information Task complexity

1. Evaluate data source Cognitive function Training HSI

2. Interpret and - Cognitive failure modes

• CFM1 represent

• CFM 2 data

• CFM3

3. Consolidate Procedures Multitasking data 14

Data generalization in three IDHEAS-G Human Error Tables

1. Evaluate 2. Interpret and 3. Consolidate data source represent data data Cognitive failure HEP Table - BP Human action modes (CFMs)

/ tasks PIF Impact Table - Wi Performance Context influencing factors (PIFs) PIF Interaction Table - C HEPCFM = BPCFM x (1+W1+W2+W3, ) x C x Re

IDHEAS-G HEP Table

• The HEP Table consolidates data of human error rates and HEPs for every cognitive failure modes.

• Cognitive activities involved in the tasks of a data source are mapped to corresponding IDHEAS-G CFMs of different levels.

• Human error rates or HEPs for the CFMs are documented along with their associated PIF states.

The HEP Table documents the following dimensions of information for every data point:

CFMs Human error rate or HEP PIF states or PIF attributes Time information, (with or without time-constraint, adequate/inadequate time).

Brief narrative of the task or types of human failure in the data source Uncertainties in the data source and in the mapping

Example 1: The effect of message complexity on pilot read-back errors REF: Message Complexity on Pilot Readback Performance (Prinzo et a., 2006)

Task: Pilots listen to and read back messages from air traffic controllers CFMs: Information misperceived; Information not retained or miscommunicated PIF: Detection complexity - Message complexity indicated by the number of information items that pilots have to retain in their working memory Results: Readback errors increase with message complexity Data:

Complexity % errors 35 1 <1 5 3.6

% errors 10 6.1 11 10.8 5

12 12 13 19 16 37 Message Complexity 17

Example 2: Human error rates in an experimental study The effect of incomplete information on decision-making in simulated pilot de-icing Task: Make decision on de-icing in flight simulation under icing weather Context: Providing pilots additional accurate vs. inaccurate information for handling of icing encounters, time-critical tasks with inadequate time, no teamwork.

Failure mode: Incorrectly select or use information for decision-making PIFs: Incomplete or unreliable information (30%), inadequate time Accurate and Accurate and Inaccurate

% error additional incomplete additional information information information

% Stall 18.1 30 89

% recovery 26.7 63.8 75

- ERDM (adequate information + inadequate time + no teamwork) = 0.18

- ERDM (inadequate information + inadequate time + no teamwork) = 0.3

- ERDM (inadequate info + unreliable info + inadeq. Time + no teamwork) = 0.89 18

IDHEAS-G PIF Impact Table

• The PIF Impact Table has many sub-tables, one for each PIF.

• The PIF Impact Table documents the data points of which the human error rates or HEPs of a task are measured for two or more states of a PIF.

• The weight of a PIF between a poor and base state can be calculated.

A PIF sub-table contains the following dimensions of information:

PIF attribute or state The PIF description in the original data Error rates or HEPs for the states or attributes PIF weight, calculated as the error rate at a poor state divided by the error rate at the base or low-impact state.

CFMs associated with the data point Brief description of the task and context Uncertainties in the data and mapping to PIFs

Example 3: The effect of long working-hours (mental fatigue)

REF: Effects of sleep loss on team situation assessment (JV Baranski, 2015)

Task: Team makes judgment of threat on a military surveillance task (situation assessment)

CFMs: Incorrect situation assessment PIF: Long working-hour; feedback information, supervision and peer-checking Results: Sleep loss affects assessment accuracy and time needed.

Data: Assessment error rate (%)

No sleep loss Sleep loss Full feedback 4.2 5.5 No feedback 4.5 6 Solo 6 8 Team 4.5 5.5

- Wu (mental fatigue) = (5.5-4.2)/4.2=0.31

- Wu (no team) = (6-4.5)/4.5= 0.33

- Wu (no feedback) = (4.5-4.2)/4.2= 0.08 20

IDHEAS-G PIF Interaction Table

• The PIF Interaction Table documents data points in which there are human error rates measured as two or more PIFs vary independently and jointly.

• A data point in the table consists of human error rates in a 2x2 or larger matrix for individual or combined PIFs.

• The combined effect of multiple PIFs can be inferred from the data point.

PIF 1 Low impact High impact PIF weight PIF2 Low impact R11 R12 W1 =R12/R11 High impact R21 R22 PIF weight W2 =R21/R11 W3 =R22/R11 No interaction (linear sum): W3=W1 + W2 Multiplicative interaction: W3=W1 x W2

Mapping SACADA taxonomy to IDHEAS-G

• IDHEAS-G and SACADA taxonomies share the same cognitive framework so their elements can be mapped to each other (not necessarily a one-to-one mapping).

• The error mode statistics of SACADA is generalized to IDHEAS-G HEP Tables; data regarding SACADA context factors is generalized into the PIF Impact Table and PIF Interaction Table.

• The scopes of the functions in SACADA are specifically for NPP control room actions performed by licensed crews, so SACADA error modes and context factors are mapped to only a subset of IDHEAS-G CFMs and PIF attributes.

Outline

• Modeling human errors and estimating HEPs in IDHEAS-G

• Generalization of human error data with IDHEAS-G

• Integration of generalized human error data to inform HEPs 23

Example - HEP Table for Task Complexity CFMs Attributes in the original data HEP Other PIFs Ref PIF attribute Attribute states ID D D4-Failure to Number of compelling Few 3E-3 None 02 respond to signals Several 1E-2 alarms Many 1E-1 D4 - Failure Number of criteria for Simple 0.273 Time constrained, 30 of getting the cue Low -Complex 0.253 Multitasking, Low-information training, No peer checking D4 or D6 - Number of messages 1, 0.005 Mixed levels of stress 31 Failure of communicated 5, 0.036 multitasking getting 8, 0.05 No peer-checking information 11, 0.11 15, 0.23 17, 0.32

>20 >0.5 U comprehension and skill high level 0.15 Unknown 21 cognitive complexity Typical 3E-3 Unknown Moderate 0.03 Unknown High 0.05 Unknown U4 and U5 # of aviation topics in 1 0.038 No peer checking 31 one communication 2 0.060

Discussion :Integrate generalized data to inform base HEPs Option A: Look up table of PIF attribute - HEP Attributes of Task complexity - Base HEP of Failure of Detection Attribute Attribute state HEP distribution

1. of items to detect <4 4 to 7

>7 Variety of items Simple Mixed formats Interrelated criteria Straightforward Interrelated Issues:

1) Inconsistency in HEP distributions due to uncertainties in original data and mapping (e.g. unknown PIFs)

2) Gaps in data - Some attributes do not yet have human error data.

IDHEAS-G HEP quantification model Option B: Integrated discrete PIF states 1

1) Define base HEP states N: No impact, HEP <E-4 E-1 L: Low impact, E-4 < HEP < E-3 HEP M: Moderate impact, E-3 < HEP < E-2 E-2 H: High impact, E-2 < HEP < E-1 E-3 Extreme High, HEP > E-1 E-4 N L M H E States of Base PIFs

2) Group PIF attributes to the corresponding states based on the HEP Table High-impact attributes of Task complexity on Detection

• # of items to be detect: 9~13

• # of unrelated detection criteria: 3~4

• Detection criteria are interrelated

Example - PIF Impact Table on Multitasking/Interruption/Distraction IDHEAS-G Tasks and macro- PIF in the original HEP (% of PIF PIF attribute functions data incorrect) weight Multitasking - D - missing cue Single vs. dual task 2.8% vs 21% 7.5 intermingled D - missing Single vs. dual task 5% vs 20% 4 changes U - Wrong Single vs. dual task 1% vs 37% 37 diagnosis Excessively E- sequence task 0, 3s, 30s 2%, 4%, 16% 2, 8 frequent or long interruption E - nonsequence 0, 3s, 30s 2%, 2%, 2% 1 task Interruption by No interruption vs 4% vs. 8% 2 same task modality interruption Interruption by D No interruption vs 1 different task U weak interruption 0.9 modality Distraction - D - Monitoring Without vs with 2.5% vs. 7% 2.8 irrelevant to the target distraction task

Integrating the data to inform PIF weight Example PIF - Multitasking, interruption, and distraction Detection Understanding (diagnosis)

PIF weight PIF weight PIF attributes PIF attributes 15

Summary

• IDHEAS-G performs HEP estimation with a basic set of CFMs, a PIF structure, and a HEP Quantification Model

• Human error data of various sources can be generalized into three IDHEAS-G Human Error Data Tables: HEP Table, PIF Impact Table, and PIF Interaction Table

• The generalized data needs to be integrated to inform HEP estimation.

• Data generalization is an on-going, continuous effort; Data integration should be periodically updated.

29

Thank you!

30