ML19092A463
| ML19092A463 | |
| Person / Time | |
|---|---|
| Issue date: | 03/14/2019 |
| From: | James Chang, Jing Xing NRC/RES/DRA/HFRB |
| To: | |
| J. Chang 415-2378 | |
| Shared Package | |
| ML19092A461 | List: |
| References | |
| Download: ML19092A463 (30) | |
Text
Generalization and Integration of Human Error Data with IDHEAS-G to Inform HEP Estimation Jing Xing, Y. James Chang US Nuclear Regulatory Commission 1
Presentation to 2019 NRC HRA Data Workshop
2
- Ideal world:
- The same task for a failure mode is repeated thousands of times with the same people under the identical context;
- Do this for all possible contexts Failure modes
- Occurrence Context Variety
Well-defined failure modes
Known, sufficient number of task occurrences
Context clearly defined and repeated
Sufficient data for all failure modes and contexts HEP (failure mode under specific context) =
- of errors (failure mode)
- of Occurrence (under the context) 3 Whats next in human reliability analysis
- DATA-based human error probability estimation
2 Human error data: The ideal world and reality
- Reality:
Good news - There are human error data in human error analysis, operational databases, cognitive and human factors experiments.
Not so good news X Failure modes not analyzed and in great variety X Context undocumented and/or unrepeated X Limited coverage - limited failure mode / context tested X Not talking to each other HEP (failure mode under specific context) =
- of errors (failure mode)
- of Occurrence (under the context)
Human error data - generalization and integration
- Human error data exists
- From various fields, in different formats, varying context and levels of details
- Data generalization and integration for human reliability analysis
- The General Methodology of Integrated Human Event Analysis System (IDHEAS-G) has an inherent structure for generalizing and integrating human error data 4
Outline
- Human error modeling and HEP estimation in IDHEAS-G
- Generalization of human error data with IDHEAS-G
- Integration of generalized human error data to inform HEPs 5
Human failure event Modeling HEP estimation Human action /
f(PIF1, PIF2, PIF3)
IDHEAS-G Modeling human failures and estimating HEPs Cognitive failure modes (CFMs)
Performance influencing factors (PIFs)
Human failure event Modeling HEP estimation Human action /
f(PIF1, PIF2, PIF3)
IDHEAS-G Modeling human errors and estimating HEPs Cognitive failure modes (CFMs)
Performance influencing factors (PIFs)
Failure of macrocognitive function Failures of processors Behaviorally observable failure modes 8
Failure of Detection D1-Incorrect mental model for Detection D2 - Fail to attend to sources of information D Fail to perceive the information D4-Fail to verify and modify detection D5-Fail to retain or communicate Information D3-1 Failure to use secondary information D3-2 Key alarm not attended to D3-3 Key information not perceived or misperceived D3-4 Parameters incorrectly monitored Demonstration of IDHEAS-G cognitive failure modes Failure of Understanding Failure of Decisionmaking Failure of Action Execution Failure of Teamwork 10
Environment Personnel / team /
organization Task /
situation System and I/C opacity Tools and parts HSI Procedures Training Work process Staffing Team and organization factors
- Information completeness and reliability
- Scenario familiarity
- Multitasking, Interruption, and distraction
- Task complexity
- Stress and time pressure
- Mental fatigue
- Physical demands PIF PIF attributes Alarm not salient Mode confusion Information masking Ambiguity of Indicators Context Demonstration of IDHEAS-G PIF structure 11 Accessibility and habitability Visibility Noise Heat/coldness Resistance to physical movement System
State of Base PIFs HEP E-4 E-3 E-2 E-1 1
PIF1 PIF2 IDHEAS-G HEP quantification model Assumption 1: Two types of PIFs: Base PIFs (Information, Scenario familiarity, and Task complexity) and modification PIFs.
Assumption 2: The combined effect of multiple PIFs on the HEP generally can be modeled with a simple linear sum of individual effects, where the PIF effect is calculated as the weight on the HEP, i.e.,
W=(HEP at a poor PIF state - HEP at the base state) / (HEP at the base state)
IDHEAS-G HEP quantification model Calculating the HEP of a CFM for a given set of PIF states IDHEAS-G uses the following equation to calculate the HEP of a CFM for any given set of PIF states provided that all the PIF weights and base HEPs are known:
HEPCFM = BPCFM x (1+W1+W2+W3, ) x C x Re, Where BPCFM is the base HEP of a CFM for the given states of the three base PIFs, Wi is the PIF weight for the given state of modification PIFs; C is a factor denoted to interaction between PIFs, and it is set to 1 for linear combination of PIF impacts unless there is data suggesting otherwise; Re is a factor denoting to potential recovery from failure of a task; it is set to 1 unless there is empirical data suggesting otherwise.
Outline
- Modeling human errors and estimating HEPs in IDHEAS-G
- Generalization of human error data with IDHEAS-G
- Integration of generalized human error data to inform HEPs 12
Generalizing human error data to inform human error probability estimation Data source 1 Tasks A generic, adaptable set of failure modes and PIFs Context Failure modes PIFs Data source 2 Tasks Context Failure modes PIFs HEP = f(states of performance influencing factors) 13
Generalizing human error data with IDHEAS-G CFMs and PIFs Cognitive function
- Cognitive failure modes
- CFM1
- CFM 2
- CFM3 Information Task complexity Training Multitasking HSI Procedures 14 Generalize data:
- 1. Evaluate data source
- 2. Interpret and represent data
- 3. Consolidate data
Human action
/ tasks Context Data generalization in three IDHEAS-G Human Error Tables Cognitive failure modes (CFMs)
Performance influencing factors (PIFs)
HEP Table - BP PIF Impact Table - Wi PIF Interaction Table - C HEPCFM = BPCFM x (1+W1+W2+W3, ) x C x Re
- 1. Evaluate data source
- 2. Interpret and represent data
- 3. Consolidate data
IDHEAS-G HEP Table The HEP Table consolidates data of human error rates and HEPs for every cognitive failure modes.
Cognitive activities involved in the tasks of a data source are mapped to corresponding IDHEAS-G CFMs of different levels.
Human error rates or HEPs for the CFMs are documented along with their associated PIF states.
The HEP Table documents the following dimensions of information for every data point:
Human error rate or HEP
Time information, (with or without time-constraint, adequate/inadequate time).
Brief narrative of the task or types of human failure in the data source
Uncertainties in the data source and in the mapping
17 Example 1: The effect of message complexity on pilot read-back errors REF: Message Complexity on Pilot Readback Performance (Prinzo et a., 2006)
Task: Pilots listen to and read back messages from air traffic controllers CFMs: Information misperceived; Information not retained or miscommunicated PIF: Detection complexity - Message complexity indicated by the number of information items that pilots have to retain in their working memory Results: Readback errors increase with message complexity Data:
Message Complexity
% errors 35 5
Complexity
% errors 1
< 1 5
3.6 10 6.1 11 10.8 12 12 13 19 16 37
18 The effect of incomplete information on decision-making in simulated pilot de-icing Task: Make decision on de-icing in flight simulation under icing weather Context: Providing pilots additional accurate vs. inaccurate information for handling of icing encounters, time-critical tasks with inadequate time, no teamwork.
Failure mode: Incorrectly select or use information for decision-making PIFs: Incomplete or unreliable information (30%), inadequate time
% error Accurate and additional information Accurate and incomplete information Inaccurate additional information
% Stall 18.1 30 89
% recovery 26.7 63.8 75 Example 2: Human error rates in an experimental study ERDM (adequate information + inadequate time + no teamwork) = 0.18 ERDM (inadequate information + inadequate time + no teamwork) = 0.3 ERDM (inadequate info + unreliable info + inadeq. Time + no teamwork) = 0.89
IDHEAS-G PIF Impact Table
- The PIF Impact Table documents the data points of which the human error rates or HEPs of a task are measured for two or more states of a PIF.
- The weight of a PIF between a poor and base state can be calculated.
A PIF sub-table contains the following dimensions of information:
PIF attribute or state
The PIF description in the original data
Error rates or HEPs for the states or attributes
PIF weight, calculated as the error rate at a poor state divided by the error rate at the base or low-impact state.
CFMs associated with the data point
Brief description of the task and context
Uncertainties in the data and mapping to PIFs
20 Example 3: The effect of long working-hours (mental fatigue)
No sleep loss Sleep loss Full feedback 4.2 5.5 No feedback 4.5 6
Solo 6
8 Team 4.5 5.5 REF: Effects of sleep loss on team situation assessment (JV Baranski, 2015)
Task: Team makes judgment of threat on a military surveillance task (situation assessment)
CFMs: Incorrect situation assessment PIF: Long working-hour; feedback information, supervision and peer-checking Results: Sleep loss affects assessment accuracy and time needed.
Data: Assessment error rate (%)
Wu (mental fatigue) = (5.5-4.2)/4.2=0.31 Wu (no team) = (6-4.5)/4.5= 0.33 Wu (no feedback) = (4.5-4.2)/4.2= 0.08
IDHEAS-G PIF Interaction Table The PIF Interaction Table documents data points in which there are human error rates measured as two or more PIFs vary independently and jointly.
A data point in the table consists of human error rates in a 2x2 or larger matrix for individual or combined PIFs.
The combined effect of multiple PIFs can be inferred from the data point.
PIF 1 PIF2 Low impact High impact PIF weight Low impact R11 R12 W1 =R12/R11 High impact R21 R22 PIF weight W2 =R21/R11 W3 =R22/R11 No interaction (linear sum): W3=W1 + W2 Multiplicative interaction: W3=W1 x W2
Mapping SACADA taxonomy to IDHEAS-G
- IDHEAS-G and SACADA taxonomies share the same cognitive framework so their elements can be mapped to each other (not necessarily a one-to-one mapping).
- The error mode statistics of SACADA is generalized to IDHEAS-G HEP Tables; data regarding SACADA context factors is generalized into the PIF Impact Table and PIF Interaction Table.
- The scopes of the functions in SACADA are specifically for NPP control room actions performed by licensed crews, so SACADA error modes and context factors are mapped to only a subset of IDHEAS-G CFMs and PIF attributes.
Outline
- Modeling human errors and estimating HEPs in IDHEAS-G
- Generalization of human error data with IDHEAS-G
- Integration of generalized human error data to inform HEPs 23
Example - HEP Table for Task Complexity CFMs Attributes in the original data HEP Other PIFs Ref ID PIF attribute Attribute states D
D4-Failure to respond to alarms Number of compelling signals Few Several Many 3E-3 1E-2 1E-1 None 02 D4 - Failure of getting information Number of criteria for the cue Simple 0.273 Time constrained, Multitasking, Low-training, No peer checking 30 Low -Complex 0.253 D4 or D6 -
Failure of getting information Number of messages communicated 1,
5, 8,
11, 15, 17,
>20 0.005 0.036 0.05 0.11 0.23 0.32
>0.5 Mixed levels of stress multitasking No peer-checking 31 U
comprehension and skill high level 0.15 Unknown 21 cognitive complexity Typical Moderate High 3E-3 0.03 0.05 Unknown Unknown Unknown U4 and U5
- of aviation topics in one communication 1
2 0.038 0.060 No peer checking 31
Discussion :Integrate generalized data to inform base HEPs Option A: Look up table of PIF attribute - HEP Issues:
1)
Inconsistency in HEP distributions due to uncertainties in original data and mapping (e.g. unknown PIFs) 2)
Gaps in data - Some attributes do not yet have human error data.
Attributes of Task complexity - Base HEP of Failure of Detection Attribute Attribute state HEP distribution
- of items to detect
<4 4 to 7
>7 Variety of items Simple Mixed formats Interrelated criteria Straightforward Interrelated
N L M H E States of Base PIFs HEP E-4 E-3 E-2 E-1 1
IDHEAS-G HEP quantification model Option B: Integrated discrete PIF states N: No impact, HEP <E-4 L: Low impact, E-4 < HEP < E-3 M: Moderate impact, E-3 < HEP < E-2 H: High impact, E-2 < HEP < E-1 Extreme High, HEP > E-1
- 1) Define base HEP states
- 2) Group PIF attributes to the corresponding states based on the HEP Table High-impact attributes of Task complexity on Detection
- of items to be detect: 9~13
- of unrelated detection criteria: 3~4 Detection criteria are interrelated
Example - PIF Impact Table on Multitasking/Interruption/Distraction IDHEAS-G PIF attribute Tasks and macro-functions PIF in the original data HEP (% of incorrect)
PIF weight Multitasking -
intermingled D - missing cue Single vs. dual task 2.8% vs 21%
7.5 D - missing changes Single vs. dual task 5% vs 20%
4 U - Wrong diagnosis Single vs. dual task 1% vs 37%
37 Excessively frequent or long interruption E-sequence task 0, 3s, 30s 2%, 4%, 16%
2, 8 E - nonsequence task 0, 3s, 30s 2%, 2%, 2%
1 Interruption by same task modality No interruption vs interruption 4% vs. 8%
2 Interruption by different task modality D
No interruption vs weak interruption 1
U 0.9 Distraction -
irrelevant to the task D - Monitoring target Without vs with distraction 2.5% vs. 7%
2.8
Example PIF - Multitasking, interruption, and distraction 15 Integrating the data to inform PIF weight PIF attributes PIF weight PIF weight Detection Understanding (diagnosis)
PIF attributes
29 Summary IDHEAS-G performs HEP estimation with a basic set of CFMs, a PIF structure, and a HEP Quantification Model Human error data of various sources can be generalized into three IDHEAS-G Human Error Data Tables: HEP Table, PIF Impact Table, and PIF Interaction Table The generalized data needs to be integrated to inform HEP estimation.
Data generalization is an on-going, continuous effort; Data integration should be periodically updated.
30 Thank you!