ML20267A196
| ML20267A196 | |
| Person / Time | |
|---|---|
| Issue date: | 09/23/2020 |
| From: | Chang Y, Segarra J, Jing Xing Office of Nuclear Regulatory Research |
| To: | |
| Sean Peters, Carmen Franklin | |
| Shared Package | |
| ML20267A193 | List: |
| References | |
| Download: ML20267A196 (37) | |
Text
IDHEAS-DATA - Human Error Data generalized in IDHEAS-G framework Jing Xing, Y. James Chang, Jonathan DeJesus Segarra, U.S. Nuclear Regulatory Commission Presented by Jing Xing to ACRS subcommittee 2020-9-23
Development of IDHEAS
- An Integrated Human Event Analysis System Cognitive Basis for HRA (NUREG-2114) 2 IDHEAS General Methodology (IDHEAS-G) (NUREG-2198)
IDHEAS Internal At-power Application (NUREG-2199)
Scientific Literature
- Research, operation experience IDHEAS-DATA IDHEAS-ECA (RIL-2020-02)
HRA applications HRA applications SACADA and all data sources
Outline I.
Approach of using human error data for HRA II.
Data source evaluation III. Data generalization (IDTABLEs)
IV. The story of PIF combination 3
I. Approach of using human error data for HRA Evaluation of human error data sources Human error data exist from various domains, in different formats, varying context and levels of details.
Data generalization The General Methodology of Integrated Human Event Analysis System (IDHEAS-G) has an inherent structure for generalizing human error data:
Five macrocognitive functions represent failure of human actions.
20 PIFs represent the context that affects human performance of an action.
Data integration for human error probability (HEP) estimation Generalized human error data can be integrated to inform HEP estimation for specific HRA methods and applications.
4
Generalizing human error data to inform HEPs Data source 1 Tasks A generic, adaptable set of failure modes and PIFs Context Failure modes PIFs Data source 2 Tasks Context Failure modes PIFs HEP = f(states of performance influencing factors) 5
Context and task Variables and Measurements Uncertainties Use human error data to inform HEPs Human tasks ->
Cognitive failure modes (CFMs)
Context ->
Performance influencing factors (PIFs)
Error rates - Base HEPs Change of error rates -
PIF weights (Wi)
Others (e.g., PIF Interaction, time distribution, dependency)
- 1. Evaluation -
Assess data source
- 2. Generalization -
Represent source data with the CFMs and PIFs in IDHEAS-DATA
- 3. Integration -
Integrate the data in IDHEAS-DATA for HEP calculation 6
II. Data sources A. Nuclear simulator data and operational data (e.g., SACADA, HuREX, German NPP maintenance database analysis)
B. Operation performance data from other domains (e.g., transportation, off-shore oil, military operations, manufacture)
C.Experimental studies in the literature (e.g., cognitive and behavior science, human factors, neuroscience)
D.Expert judgment of human reliability in the nuclear domain E. Unspecific context (e.g., statistical data, ranking, frequencies of errors or causal analysis) 7
Data source evaluation
- Participants - Normal adults, trained for the tasks, good sample size
- Measurements - Human error rate preferred, task performance measures related to human error rates
- Uncertainties - Controlled, known, or traceable
- Breath of representation - Repetitive and representative 8
Outline I.
Approach of using human error data for HRA II.
Data source evaluation III. Human error data generalization (IDTABLEs)
IV. The story of PIF combination 9
IDHEAS-DATA Structure 10 IDHEAS-DATA has 27 tables (IDTABLEs) documenting generalized human error data and empirical evidence Human error data are generalized to IDHEAS-G CFMs and PIF attributes IDHEAS-DATA IDTABLE IDTABLE 1-3 Base HEPs IDTABLE-1 Scenario Familiarity IDTABLE-2 Information IDTABLE-3 Task Complexity IDTABLE 4--20 PIF Weights IDTABLE 4-8 Environment PIFs IDTABLE 9-11 System PIFs IDTABLE 11-16 Personnel PIFs IDTABLE 17-20 Task PIFs IDTABLE-21 Lowest HEPs of CFMs IDTABLE-22 PIF Interaction IDTABLE-23 Distribution of Task Needed IDTABLE-24 Modification to Time Needed IDTABLE-25 Dependency of Human Actions IDTABLE-26 Recovery of Human Actions IDTABLE-27 Main drivers to human events 10
11 Data generalization process Generalizing a data source is the same as performing an HRA using IDHEAS-G
- Analyze the data source to understand the context and determine the human error data for generalization
- Analyze the tasks and identify the applicable CFMs
- Map the context to relevant PIF attributes
- Identify other PIF attributes present in the study
- Analyze uncertainties
- Document the reported human error data in IDTABLE
12 Example 1: a datapoint for base HEP PIF CFM Error rates Task (and error measure)
PIF measure Other PIFs (and Uncertainty)
REF SF3.1 U
1.2E-1 (8/69)
NPP operators diagnose in simulator training Anomaly scenario (Other PIFs may exist)
[26]
SF3.1 DM 1.1E-2 (1/92)
NPP operators decisionmaking in simulator training Anomaly scenario (Other PIFs may exist)
[26]
The NRCs SACADA database collects NPP operators task performance data in simulator training for requalification examination. The rates of unsatisfactory performance (UNSAT) for training objective tasks were calculated from the SACADA data available before April 2019.
The UNSAT rates are generalized in IDTABLE-1, -2, and -3 for the three base PIFs.
For example, SACADA characterizes Scenario Familiarity as three options:
Standard, Novel, and Anomaly. The generalized datapoints are shown in the following:
13 Example 2: a datapoint for PIF weight PIF CFM Error rates Task (and error measure)
PIF measure Other PIFs (and Uncertainty)
REF VIS1 D
Luminance Reading error Military operators dial reading (incorrect reading)
Luminance (L/m2)
No peer-
- checking, maybe HSI VIS-9 0.15 0.16 1.5 0.1
>15 0.08 Braunstein and White measured human errors in reading dials as the luminance on the dials was varied from 0.015 to 150 L/m2.
The error rate decreased with luminance. When the luminance was greater than 15 L/m2, the error rate was low and remained the same.
Many other studies reported similar relation between luminance and error rates.
The following is the datapoint generalized in IDHEAS-DATA IDTABLE-5 for Visibility:
14 Data sources Limited use of nuclear operation/simulation data (SACADA, HuREX, Halden studies)
~300+ literature generalized; another 200+ evaluated and selected for generalization
- 300~400 literature on task completion time to be generalized in 2021 Overview of IDHEAS-DATA in 2020
15 Overview of IDHEAS-DATA in 2020 IDTABLEs The data in IDTABLE-1 through -21 (base HEPs, PIF weights, and lowest HEPs) were integrated for IDHEAS-ECA.
IDTABLE-23 and -24 (Task Completion Time) are on the way.
IDTABLE-25 (dependency), -26 (recovery) and -27 (main drivers) are in piloting.
Areas lacking human error data CFMs: Interteam Coordination PIFs: Work Process, Team and Organizational Factors
Outline I.
Approach of using human error data for HRA II.
Data source evaluation III. Human error data generalization (IDTABLEs)
IV. The story of PIF combination 16
A story of PIF combination An operators HEP is 0.01 in nominal conditions, 0.05 in loud burst noise environment, and 0.1 under poor visibility. What is his HEP when working under loud noise and poor visibility?
Answer 1: Additive 0.05 + 0.10 = 0.15 Answer 2: Multiplicative 0.01 x 5 x 10 = 0.5 17
Whats in data Good Poor Error rate PIF2 Good PIF2 poor Additive Multiplicative Subtractive (or interactive)
Good Poor Good Poor PIF1 18
Most Whats in data Good Poor Error rate PIF2 Good PIF2 poor Additive Multiplicative Subtractive (or interactive)
Good Poor Good Poor Some Rare PIF1 IDHEAS-DATA observation from 100+ studies evaluated with human error data under individual and PIF combination:
Most datapoints are roughly additive Some datapoints show multiplicative 19
20 Why and when PIF combination is more than Additive?
- If both PIFs demand the same cognitive resource, and the demand of a single PIF already approaches to the capacity limit, then the combined effect can be more than the Additive effects; This reflects the catastrophic effect of exceeding the capacity limit.
2 3
4 5
6 7
1 2
3 List Words recalled High Span Low Span
% errors 35 5
- of working memory items Working memory capacity and interfere (Kane & Engle, 2000)
Working memory capacity (Prinzo et al, 2006)
Proactive interfere list Words recalled 20
21 Meta-analysis on PIF combination PIFs
- of studies Findings Ref.
- Noise, temperature, sleep loss 51 reports Combined effect is no more than the added single effects and can be predicted from single effects.
Grether 1970 Noise and heat 20~30 reports The majority of evidence indicates that noise and heat do not interact significantly within the ranges experienced commonly in the industrial setting.
Hancock 2010 Distraction, experience, HSI, others 23 data-points Additive fits better than Multiplicative; Additive over estimates for large PIF weights Xing 2015 Cognitive ability and motivation on performance 51 reports Additive accounted for ~ 91% of job performance data; Multiplicative accounted for only about 9% of the explained variance.
Iddeking e
2017
22 Perspective of IDTABLE-21: PIF Interaction Solid evidence that most PIF combinations are additive.
IDTABLE-21 should focus on PIF interaction:
Interaction between a base PIF and modification PIFs More-than-additive interaction Red flag PIF combinations
23 Summary of IDHEAS-DATA Human error data of various sources are generalized into IDHEAS-DATA with IDHEAS cognitive failure modes (CFMs) and PIF attributes Data generalization is generic with IDHEAS CFMs and PIF attributes; Data integration is specific to the HRA method or application that uses the data.
Data generalization is an on-going, continuous effort; Data integration should be periodically updated.
Backup slides Integration of human error data for IDHEAS-ECA 24
IDHEAS-ECA uses the HEP Quantification Model
25 HEP from Base PIFs PIF weight factors from Modification PIFs PIF interaction factor; set to 1 with linear combination Recovery factor; set to 1 unless data suggest otherwise IDHEAS-ECA needs:
Lowest HEPs for the 5 CFMs Base HEPs of every CFM at every associated attribute of the 3 base PIFs PIF weights of every CFM at every associated attribute of the 17 modification PIFs
26 Data integration process The process of integrating human error data is described as follows:
1)
Use single-component data to make initial estimation of the base HEPs and PIF weights; 2)
Use the initial estimation to detach multi-component data into single-component ones; 3)
Integrate all the single-component and detached multi-component datapoints to estimate the range and mean of a base HEP or PIF weight; 4)
Use the unspecific datapoints to calibrate the estimated HEPs and PIF weights; 5)
Iterate the process 2), 3), and 4) until the obtained values represent the breath of the available data.
27 Approaches used in integration process The confidentiality in integrating a set of data to generate a single representative value or probabilistic distribution depends on the sample size and quality of the data set.
The following approaches were used in the integration for IDHEAS-ECA:
(See notes)
- 2) No single-component data exclusive for a base HEP or PIF weight, but there were multi-component datapoints on the combined effects of several CFMs and/or PIF attributes
- 3) No datapoint for a PIF weight
- 4) Consistency checking and adjustment with benchmark values
Example - IDHEAS-DATA IDTABLE-21 Lowest HEPs for Failure of Detection Error rate Task Criteria for lowest HEPs:
TA - Time adequacy SelfV - Self verification TeamV - Team verification Rec - Recovery O - other factors (Y-Yes, N - No, M-Mixed Un-Unknown)
Uncertainty REF 1
2.1E-3 (4/1872)
NPP operators alarm detection in simulator training. Alarms are self-revealing TA-Yes, SelfV-Y, TeamV-Y, R-Unknown O - Y (unspecified)
(Other PIFs may exist)
[26]
2 3.4E-3 (3/870)
NPP operators check indicators in simulator training, procedure directed checking.
TA-Yes, SelfV-Yes, TeamV-yes, Rec - Unknown O - Y (unspecified)
(Other PIFs may exist)
[26]
3 5E-4 Military operators read meters, Alphanumeric reading, Detection straight-forward TA-Y, SelfV-Y, TeamV-No, Rec-No (Maybe time constraint, 10K+ source data trials)
[109]
4 E-4 Estimated lowest probity of human failure events TA-Yes, SelfV-Yes, TeamV-yes, Rec - Unknown (Engineering judgment)
[110]
5 E-4 Simplest possible tasks TA-Yes, SelfV-Yes, TeamV-Unknown, Rec - Unknown (Engineering judgment)
[111]
6 E-3 Routine simple tasks TA-Yes, SelfV-Yes, TeamV-Unknown, Rec - Unknown O - Maybe weak complexity (Engineering judgment)
[111]
7 5E-3 Line-oriented text editor. Error rate per word TA-Yes, SelfV-Yes, TeamV-No, Rec - No No apparent uncertainty
[112]
8 5E-3 Reading a gauge incorrectly. Per read TA-Yes, SelfV-Yes, TeamV-No, Rec - Unknown O - HSI No apparent uncertainty
[113]
9 E-3 Interpreting indicator on an indicator lamp.
Per interpretation TA-Yes, SelfV-Yes, TeamV-Unknown, Rec - Unknown O-complexity in interpreting indicator (Engineering judgment)
[109]
10 9E-4 NPP operator simulator runs TA - Y, Selv-V - Y TeamV - Y, R - Unknown O - Mixed complexity No apparent uncertainty
[114, 115]
11 5.3E-4 Gather information and evaluate parameters TA - Y, Selv-V - Y TeamV - Y, R - Yes No apparent uncertainty
[116]
12 9E-3 Collision avoidance and target monitoring in simulated ship control Fixed situation TA - Y, Selv-V - Yes TeamV No R Yes Dual task
[27]
28
Detaching multi-component human error data The critical step in the process is detaching multi-component datapoints. The following rules are derived from initial estimates of base HEPs of task complexity and PIF attribute weights. They are used for detaching:
- 1) If SelfV=NO or TeamV=NO, the detached error rate is the original error rate divided by a factor of 5; If both are NO, the detached error rate is the original error rate divided by a factor of 10.
- 2) If Recovery = YES, the detached error rate is the original error rate multiplied by a factor range of 2 to 10.
- 3) If there are other PIFs, the detached error rate is the original error rate divided by multiplication of a factor range of (5 to 10 for complexity) and the sum of the weights of other PIF attributes. The weights of the PIF attributes are from the initiation estimation of the single-component data in IDHEAS-DATA.
29
CFM Error rate Criteria for lowest HEPs Detached error rate Notes 1
2.1E-3 (4/1872)
TA-Yes, SelfV-Y, TeamV-Y, R-Unknown O - Y (unspecified) 2.1E-3 / (5 to 10) = 2.1E-4 to 4E-4 A factor of 5 to 10 represents the combined effect of possible other PIFs 2
3.4E-3 (3/870)
TA-Yes, SelfV-Yes, TeamV-yes, Rec - Unknown O - Y (unspecified) 3.4E-3 / (5 to 10) = 3.4E-4 to 7E-4 A factor of 5 to 10 represents the combined effect of possible other PIFs 3
5E-4 TA-Y, SelfV-Y, TeamV-No, Rec-No 5E-4 / 5 = 1E-4 Divided by 5 for no team verification 4
E-4 TA-Yes, SelfV-Yes, TeamV-yes, Rec - Unknown E-4 No change 5
E-4 TA-Yes, SelfV-Yes, TeamV-Unknown, Rec - Unknown E-4 No change 6
E-3 TA-Yes, SelfV-Yes, TeamV-Unknown, Rec - Unknown O - Maybe weak complexity E-3 / 5 = 2E-4 Divided by 5 for weak complexity 7
5E-3 TA-Yes, SelfV-Yes, TeamV-No, Rec - No 5E-3 / 10 = 2E-4 Divided by (5+5) for lack of self and team verification 8
5E-3 TA-Yes, SelfV-Yes, TeamV-No, Rec - Unknown O - Maybe HSI 5E-3 / (5+2) = 7E-4 Divided by (5+2) for lack of self verification and possible HSI attributes 9
E-3 TA-Yes, SelfV-Yes, TeamV-Unknown, Rec - Unknown E-3 / 5 = 2E-4 Divided by 5 for no team verification.
10 9E-4 TA - Y, Selv-V - Y TeamV - Y, R - Unknown O - Mixed complexity 9E-4 / (5 t o10) =
9E-5 to 4.8E-4 Divided by (5 to 10) for mixed complexity 11 5.3E-4 TA - Y, Selv-V - Y TeamV - Y, R - Yes O - Mixed complexity 5.3E-4 x 2 / (5-10)
1.06E-4 to 2.12E-4 Multiplied by 2 for existence of recovery 12 9E-3 TA - Y, Selv-V - Yes TeamV - No, R - Yes O - Dual task, and maybe mixed complexity 9E-3 / (5 to 10) x (5-10)
9E-5 to 3.6E-4 Divided by (5 to 10) for mixed complexity and divided by (5 to 10) for dual task.
Table 3-8: Detached human error rates for the lowest HEP of Failure of Detection 30
Table 3-9. Single-component and detached multi-component human error rates for the lowest HEP of Failure of Detection Single-component Multi-component detachable Bounding A - Nuclear operation 2.1E-4 to 4E-4, 3.4E-4 to 7E-4, 9E-5 to 4.8E-4 B - Other operation 1.06E-4 to 2.12E-4 1E-4, 2E-4 7E-4 C - Controlled experiment E-4, 2E-4 9E-5 to 3.6E-4 D - Expert judgment E-4 2E-4 E - Unspecific 31
Table 3-9. Single-component and detached multi-component human error rates for the lowest HEP of Failure of Detection Figure 3-1. The human error rates for the lowest HEP of Failure of Detection Category A datapoints: [ 1.8, 3.6, 5.3]E-4 for lower bound, mean, and upper bound; Category B datapoints: [ 1.06, 2.8, 2.1]E-4 Category C datapoints: [ 0.9, 1.7, 3.6]E-4 Category A, B, C datapoints: [1.4, 1.8, 4.4 ]E-4 Based on the data, the value 1E-4 is taken as the lowest HEP for Failure of Detection.
32
A story of two type of PIFs (Backup slides)
A story of two type of PIFs Mental model Signal / Noise Cognitive processing Signal-noise ratio - Information Availability and Reliability Mental model - Scenario Familiarity Demands for cognitive processing - Task Complexity Mental representation (outcome of macrocognitive functions)
Base PIFs Modification PIFs - modify the base PIFs
- 1. What s in the cognitive basis
Attention & Working Memory for integration External World Detect/Notice How human achieves Understanding (NUREG-2114)
New Info Data Percept Prior Info Frame LTM Knowledge Expertise Experience Goals Work process Subconscious Desires Workload Interfaces Procedures Fatigue Motivation 35
- 2. Whats in data about PIF effects on HEPs Information Availability and Reliability can vary HEP from nearly 0 to 1; Scenario Familiarity can vary HEP from nearly 0 to 1; Task Complexity can vary HEP from nearly 0 to 1; Base PIFs Modification PIFs -
A single modification PIF attribute typically varies HEP in the range of 1.1 to 10 times, with a few exception high up to 30 times for feasible tasks.
% errors 3.5E-1 E-3 Base PIF - Task complexity A story of two type of PIFs
- example data from German NPP maintenance performance database Memorized task step not remembered in carrying outa sequence of tasks Scenario Familiarity (frequently to extreme rarely performed tasks) varied the error rate from 7.78E-5 to 3.52E-1