ML19092A463
| ML19092A463 | |
| Person / Time | |
|---|---|
| Issue date: | 03/14/2019 |
| From: | James Chang, Jing Xing NRC/RES/DRA/HFRB |
| To: | |
| J. Chang 415-2378 | |
| Shared Package | |
| ML19092A461 | List: |
| References | |
| Download: ML19092A463 (30) | |
Text
Generalization and Integration of Human Error Data with IDHEAS-G to Inform HEP Estimation Jing Xing, Y. James Chang US Nuclear Regulatory Commission Presentation to 2019 NRC HRA Data Workshop 1
Whats next in human reliability analysis
- DATA-based human error probability estimation
- of errors (failure mode)
HEP (failure mode under specific context) =
- of Occurrence (under the context)
- Ideal world:
- The same task for a failure mode is repeated thousands of times with the same people under the identical context;
- Do this for all possible contexts Failure modes # Occurrence Context Variety
Well-defined Known, sufficient Context clearly Sufficient data for all failure modes number of task defined and failure modes and occurrences repeated contexts 2 3
Human error data: The ideal world and reality
- of errors (failure mode)
HEP (failure mode under specific context) =
- of Occurrence (under the context)
- Reality:
Good news - There are human error data in human error analysis, operational databases, cognitive and human factors experiments.
Not so good news X Failure modes not analyzed and in great variety X Context undocumented and/or unrepeated X Limited coverage - limited failure mode / context tested X Not talking to each other 2
Human error data - generalization and integration
- Human error data exists
- From various fields, in different formats, varying context and levels of details
- Data generalization and integration for human reliability analysis
- The General Methodology of Integrated Human Event Analysis System (IDHEAS-G) has an inherent structure for generalizing and integrating human error data 4
Outline
- Human error modeling and HEP estimation in IDHEAS-G
- Generalization of human error data with IDHEAS-G
- Integration of generalized human error data to inform HEPs 5
IDHEAS-G Modeling human failures and estimating HEPs Human failure event Modeling HEP estimation Cognitive Human action / failure modes tasks (CFMs)
f(PIF1, PIF2, PIF3)
Performance Context influencing factors (PIFs)
IDHEAS-G Modeling human errors and estimating HEPs Human failure event Modeling HEP estimation Cognitive Human action / failure modes tasks (CFMs)
f(PIF1, PIF2, PIF3)
Performance Context influencing factors (PIFs)
Demonstration of IDHEAS-G cognitive failure modes Failure of Failures of processors Behaviorally observable macrocognitive failure modes function D1- Incorrect mental D3-1 Failure to use Failure of Detection model for Detection secondary information D2 - Fail to attend to D3-2 Key alarm not Failure of sources of information attended to Understanding D Fail to perceive D3-3 Key information Failure of the information Decisionmaking not perceived or misperceived D4- Fail to verify and Failure of Action modify detection D3-4 Parameters Execution incorrectly monitored D5- Fail to retain or Failure of Teamwork communicate Information 8 10
Demonstration of IDHEAS-G PIF structure Personnel / team / Task /
Context Environment System situation organization
- Accessibility - System and - Information
- Procedures and habitability I/C opacity completeness and
- Training reliability
- Visibility - Tools and - Work process PIF - Scenario familiarity
- Noise parts - Staffing
- Multitasking,
- Heat/coldness - HSI - Team and Interruption, and organization distraction
- Resistance to factors physical - Task complexity movement PIF - Stress and time
- Alarm not salient pressure attributes
- Mode confusion - Mental fatigue
- Information - Physical demands masking
- Ambiguity of Indicators
IDHEAS-G HEP quantification model Assumption 1: Two types of PIFs: Base PIFs (Information, Scenario familiarity, and Task complexity) and modification PIFs.
PIF2 PIF1 1
E-1 HEP E-2 E-3 E-4 State of Base PIFs Assumption 2: The combined effect of multiple PIFs on the HEP generally can be modeled with a simple linear sum of individual effects, where the PIF effect is calculated as the weight on the HEP, i.e.,
W=(HEP at a poor PIF state - HEP at the base state) / (HEP at the base state)
IDHEAS-G HEP quantification model Calculating the HEP of a CFM for a given set of PIF states IDHEAS-G uses the following equation to calculate the HEP of a CFM for any given set of PIF states provided that all the PIF weights and base HEPs are known:
HEPCFM = BPCFM x (1+W1+W2+W3, ) x C x Re, Where BPCFM is the base HEP of a CFM for the given states of the three base PIFs, Wi is the PIF weight for the given state of modification PIFs; C is a factor denoted to interaction between PIFs, and it is set to 1 for linear combination of PIF impacts unless there is data suggesting otherwise; Re is a factor denoting to potential recovery from failure of a task; it is set to 1 unless there is empirical data suggesting otherwise.
Outline
- Modeling human errors and estimating HEPs in IDHEAS-G
- Generalization of human error data with IDHEAS-G
- Integration of generalized human error data to inform HEPs 12
Generalizing human error data to inform human error probability estimation HEP = f(states of performance influencing factors)
Data source 1 Data source 2 Tasks Context Tasks Context Failure PIFs Failure PIFs modes modes A generic, adaptable set of failure modes and PIFs 13
Generalizing human error data with IDHEAS-G CFMs and PIFs Generalize data: Information Task complexity
- 1. Evaluate data source Cognitive function Training HSI
- 2. Interpret and - Cognitive failure modes
- CFM1 represent
- CFM 2 data
- CFM3
- 3. Consolidate Procedures Multitasking data 14
Data generalization in three IDHEAS-G Human Error Tables
- 1. Evaluate 2. Interpret and 3. Consolidate data source represent data data Cognitive failure HEP Table - BP Human action modes (CFMs)
/ tasks PIF Impact Table - Wi Performance Context influencing factors (PIFs) PIF Interaction Table - C HEPCFM = BPCFM x (1+W1+W2+W3, ) x C x Re
IDHEAS-G HEP Table
- Cognitive activities involved in the tasks of a data source are mapped to corresponding IDHEAS-G CFMs of different levels.
The HEP Table documents the following dimensions of information for every data point:
CFMs Human error rate or HEP PIF states or PIF attributes Time information, (with or without time-constraint, adequate/inadequate time).
Brief narrative of the task or types of human failure in the data source Uncertainties in the data source and in the mapping
Example 1: The effect of message complexity on pilot read-back errors REF: Message Complexity on Pilot Readback Performance (Prinzo et a., 2006)
Task: Pilots listen to and read back messages from air traffic controllers CFMs: Information misperceived; Information not retained or miscommunicated PIF: Detection complexity - Message complexity indicated by the number of information items that pilots have to retain in their working memory Results: Readback errors increase with message complexity Data:
Complexity % errors 35 1 <1 5 3.6
% errors 10 6.1 11 10.8 5
12 12 13 19 16 37 Message Complexity 17
Example 2: Human error rates in an experimental study The effect of incomplete information on decision-making in simulated pilot de-icing Task: Make decision on de-icing in flight simulation under icing weather Context: Providing pilots additional accurate vs. inaccurate information for handling of icing encounters, time-critical tasks with inadequate time, no teamwork.
Failure mode: Incorrectly select or use information for decision-making PIFs: Incomplete or unreliable information (30%), inadequate time Accurate and Accurate and Inaccurate
% error additional incomplete additional information information information
% Stall 18.1 30 89
% recovery 26.7 63.8 75
- ERDM (adequate information + inadequate time + no teamwork) = 0.18
- ERDM (inadequate information + inadequate time + no teamwork) = 0.3
- ERDM (inadequate info + unreliable info + inadeq. Time + no teamwork) = 0.89 18
IDHEAS-G PIF Impact Table
- The PIF Impact Table documents the data points of which the human error rates or HEPs of a task are measured for two or more states of a PIF.
- The weight of a PIF between a poor and base state can be calculated.
A PIF sub-table contains the following dimensions of information:
PIF attribute or state The PIF description in the original data Error rates or HEPs for the states or attributes PIF weight, calculated as the error rate at a poor state divided by the error rate at the base or low-impact state.
CFMs associated with the data point Brief description of the task and context Uncertainties in the data and mapping to PIFs
Example 3: The effect of long working-hours (mental fatigue)
REF: Effects of sleep loss on team situation assessment (JV Baranski, 2015)
Task: Team makes judgment of threat on a military surveillance task (situation assessment)
CFMs: Incorrect situation assessment PIF: Long working-hour; feedback information, supervision and peer-checking Results: Sleep loss affects assessment accuracy and time needed.
Data: Assessment error rate (%)
No sleep loss Sleep loss Full feedback 4.2 5.5 No feedback 4.5 6 Solo 6 8 Team 4.5 5.5
- Wu (mental fatigue) = (5.5-4.2)/4.2=0.31
- Wu (no team) = (6-4.5)/4.5= 0.33
- Wu (no feedback) = (4.5-4.2)/4.2= 0.08 20
IDHEAS-G PIF Interaction Table
- The PIF Interaction Table documents data points in which there are human error rates measured as two or more PIFs vary independently and jointly.
- A data point in the table consists of human error rates in a 2x2 or larger matrix for individual or combined PIFs.
- The combined effect of multiple PIFs can be inferred from the data point.
PIF 1 Low impact High impact PIF weight PIF2 Low impact R11 R12 W1 =R12/R11 High impact R21 R22 PIF weight W2 =R21/R11 W3 =R22/R11 No interaction (linear sum): W3=W1 + W2 Multiplicative interaction: W3=W1 x W2
Mapping SACADA taxonomy to IDHEAS-G
- IDHEAS-G and SACADA taxonomies share the same cognitive framework so their elements can be mapped to each other (not necessarily a one-to-one mapping).
- The error mode statistics of SACADA is generalized to IDHEAS-G HEP Tables; data regarding SACADA context factors is generalized into the PIF Impact Table and PIF Interaction Table.
- The scopes of the functions in SACADA are specifically for NPP control room actions performed by licensed crews, so SACADA error modes and context factors are mapped to only a subset of IDHEAS-G CFMs and PIF attributes.
Outline
- Modeling human errors and estimating HEPs in IDHEAS-G
- Generalization of human error data with IDHEAS-G
- Integration of generalized human error data to inform HEPs 23
Example - HEP Table for Task Complexity CFMs Attributes in the original data HEP Other PIFs Ref PIF attribute Attribute states ID D D4-Failure to Number of compelling Few 3E-3 None 02 respond to signals Several 1E-2 alarms Many 1E-1 D4 - Failure Number of criteria for Simple 0.273 Time constrained, 30 of getting the cue Low -Complex 0.253 Multitasking, Low-information training, No peer checking D4 or D6 - Number of messages 1, 0.005 Mixed levels of stress 31 Failure of communicated 5, 0.036 multitasking getting 8, 0.05 No peer-checking information 11, 0.11 15, 0.23 17, 0.32
>20 >0.5 U comprehension and skill high level 0.15 Unknown 21 cognitive complexity Typical 3E-3 Unknown Moderate 0.03 Unknown High 0.05 Unknown U4 and U5 # of aviation topics in 1 0.038 No peer checking 31 one communication 2 0.060
Discussion :Integrate generalized data to inform base HEPs Option A: Look up table of PIF attribute - HEP Attributes of Task complexity - Base HEP of Failure of Detection Attribute Attribute state HEP distribution
- of items to detect <4 4 to 7
>7 Variety of items Simple Mixed formats Interrelated criteria Straightforward Interrelated Issues:
- 1) Inconsistency in HEP distributions due to uncertainties in original data and mapping (e.g. unknown PIFs)
- 2) Gaps in data - Some attributes do not yet have human error data.
IDHEAS-G HEP quantification model Option B: Integrated discrete PIF states 1
- 1) Define base HEP states N: No impact, HEP <E-4 E-1 L: Low impact, E-4 < HEP < E-3 HEP M: Moderate impact, E-3 < HEP < E-2 E-2 H: High impact, E-2 < HEP < E-1 E-3 Extreme High, HEP > E-1 E-4 N L M H E States of Base PIFs
- 2) Group PIF attributes to the corresponding states based on the HEP Table High-impact attributes of Task complexity on Detection
- # of items to be detect: 9~13
- # of unrelated detection criteria: 3~4
- Detection criteria are interrelated
Example - PIF Impact Table on Multitasking/Interruption/Distraction IDHEAS-G Tasks and macro- PIF in the original HEP (% of PIF PIF attribute functions data incorrect) weight Multitasking - D - missing cue Single vs. dual task 2.8% vs 21% 7.5 intermingled D - missing Single vs. dual task 5% vs 20% 4 changes U - Wrong Single vs. dual task 1% vs 37% 37 diagnosis Excessively E- sequence task 0, 3s, 30s 2%, 4%, 16% 2, 8 frequent or long interruption E - nonsequence 0, 3s, 30s 2%, 2%, 2% 1 task Interruption by No interruption vs 4% vs. 8% 2 same task modality interruption Interruption by D No interruption vs 1 different task U weak interruption 0.9 modality Distraction - D - Monitoring Without vs with 2.5% vs. 7% 2.8 irrelevant to the target distraction task
Integrating the data to inform PIF weight Example PIF - Multitasking, interruption, and distraction Detection Understanding (diagnosis)
PIF weight PIF weight PIF attributes PIF attributes 15
Summary
- IDHEAS-G performs HEP estimation with a basic set of CFMs, a PIF structure, and a HEP Quantification Model
- Human error data of various sources can be generalized into three IDHEAS-G Human Error Data Tables: HEP Table, PIF Impact Table, and PIF Interaction Table
- The generalized data needs to be integrated to inform HEP estimation.
- Data generalization is an on-going, continuous effort; Data integration should be periodically updated.
29
Thank you!
30