ML20267A196
| ML20267A196 | |
| Person / Time | |
|---|---|
| Issue date: | 09/23/2020 |
| From: | Chang Y, Segarra J, Jing Xing Office of Nuclear Regulatory Research |
| To: | |
| Sean Peters, Carmen Franklin | |
| Shared Package | |
| ML20267A193 | List: |
| References | |
| Download: ML20267A196 (37) | |
Text
IDHEAS-DATA - Human Error Data generalized in IDHEAS-G framework Jing Xing, Y. James Chang, Jonathan DeJesus Segarra, U.S. Nuclear Regulatory Commission Presented by Jing Xing to ACRS subcommittee 2020-9-23
Development of IDHEAS
- An Integrated Human Event Analysis System Scientific Cognitive Basis for HRA Literature (NUREG-2114)
SACADA and all data sources
- Research, IDHEAS General Methodology IDHEAS-operation experience (IDHEAS-G) (NUREG-2198) DATA IDHEAS Internal At- IDHEAS-ECA (RIL-2020-02) power Application (NUREG-2199)
HRA applications HRA applications 2
Outline I. Approach of using human error data for HRA II. Data source evaluation III. Data generalization (IDTABLEs)
IV. The story of PIF combination 3
I. Approach of using human error data for HRA
- Evaluation of human error data sources Human error data exist from various domains, in different formats, varying context and levels of details.
- Data generalization The General Methodology of Integrated Human Event Analysis System (IDHEAS-G) has an inherent structure for generalizing human error data:
- Five macrocognitive functions represent failure of human actions.
- 20 PIFs represent the context that affects human performance of an action.
- Data integration for human error probability (HEP) estimation Generalized human error data can be integrated to inform HEP estimation for specific HRA methods and applications.
4
Generalizing human error data to inform HEPs HEP = f(states of performance influencing factors)
Data source 1 Data source 2 Tasks Context Tasks Context Failure PIFs Failure PIFs modes modes A generic, adaptable set of failure modes and PIFs 5
Use human error data to inform HEPs
- 1. Evaluation - 2. Generalization - 3. Integration -
Assess data Represent source data Integrate the data in source with the CFMs and PIFs IDHEAS-DATA for in IDHEAS-DATA HEP calculation
- Context and Human tasks -> Error rates - Base HEPs task Cognitive failure modes (CFMs) Change of error rates -
- Variables and PIF weights (Wi)
Measurements Context ->
Performance Others (e.g., PIF
- Uncertainties influencing Interaction, time factors (PIFs) distribution, dependency) 6
II. Data sources A. Nuclear simulator data and operational data (e.g., SACADA, HuREX, German NPP maintenance database analysis)
B. Operation performance data from other domains (e.g., transportation, off-shore oil, military operations, manufacture)
C. Experimental studies in the literature (e.g., cognitive and behavior science, human factors, neuroscience)
D. Expert judgment of human reliability in the nuclear domain E. Unspecific context (e.g., statistical data, ranking, frequencies of errors or causal analysis) 7
Data source evaluation
- Participants - Normal adults, trained for the tasks, good sample size
- Measurements - Human error rate preferred, task performance measures related to human error rates
- Uncertainties - Controlled, known, or traceable
- Breath of representation - Repetitive and representative 8
Outline I. Approach of using human error data for HRA II. Data source evaluation III. Human error data generalization (IDTABLEs)
IV. The story of PIF combination 9
IDHEAS-DATA Structure
- IDHEAS-DATA has 27 tables (IDTABLEs) documenting generalized human error data and empirical evidence
- Human error data are generalized to IDHEAS-G CFMs and PIF attributes IDHEAS-DATA IDTABLE IDTABLE 1-3 Base HEPs IDTABLE-21 Lowest HEPs of CFMs IDTABLE-1 Scenario Familiarity IDTABLE-22 PIF Interaction IDTABLE-2 Information IDTABLE-23 Distribution of Task Needed IDTABLE-3 Task Complexity IDTABLE-24 Modification to Time Needed IDTABLE 4--20 PIF Weights IDTABLE-25 Dependency of Human IDTABLE 4-8 Environment PIFs Actions IDTABLE 9-11 System PIFs IDTABLE-26 Recovery of Human Actions IDTABLE 11-16 Personnel PIFs IDTABLE-27 Main drivers to human events IDTABLE 17-20 Task PIFs 10 10
Data generalization process Generalizing a data source is the same as performing an HRA using IDHEAS-G
- Analyze the data source to understand the context and determine the human error data for generalization
- Analyze the tasks and identify the applicable CFMs
- Map the context to relevant PIF attributes
- Identify other PIF attributes present in the study
- Analyze uncertainties
- Document the reported human error data in IDTABLE 11
Example 1: a datapoint for base HEP
- The NRCs SACADA database collects NPP operators task performance data in simulator training for requalification examination. The rates of unsatisfactory performance (UNSAT) for training objective tasks were calculated from the SACADA data available before April 2019.
- The UNSAT rates are generalized in IDTABLE-1, -2, and -3 for the three base PIFs.
- For example, SACADA characterizes Scenario Familiarity as three options:
Standard, Novel, and Anomaly. The generalized datapoints are shown in the following:
Other PIFs Error Task (and error PIF CFM PIF measure (and REF rates measure)
Uncertainty)
SF3.1 U 1.2E-1 NPP operators Anomaly (Other PIFs [26]
(8/69) diagnose in simulator scenario may exist) training SF3.1 DM 1.1E-2 NPP operators Anomaly (Other PIFs [26]
(1/92) decisionmaking in scenario may exist) simulator training 12
Example 2: a datapoint for PIF weight
- Braunstein and White measured human errors in reading dials as the luminance on the dials was varied from 0.015 to 150 L/m2.
- The error rate decreased with luminance. When the luminance was greater than 15 L/m2, the error rate was low and remained the same.
- Many other studies reported similar relation between luminance and error rates.
- The following is the datapoint generalized in IDHEAS-DATA IDTABLE-5 for Visibility:
PIF CFM Error rates Task (and error PIF measure Other PIFs REF measure) (and Uncertainty)
VIS1 D Luminance Reading error Military Luminance No peer- VIS-0.15 0.16 operators dial (L/m2) checking, 9 1.5 0.1 reading maybe HSI
>15 0.08 (incorrect reading) 13
Overview of IDHEAS-DATA in 2020
- Data sources
- Limited use of nuclear operation/simulation data (SACADA, HuREX, Halden studies)
- ~300+ literature generalized; another 200+ evaluated and selected for generalization
- 300~400 literature on task completion time to be generalized in 2021 14
Overview of IDHEAS-DATA in 2020
- IDTABLEs
- The data in IDTABLE-1 through -21 (base HEPs, PIF weights, and lowest HEPs) were integrated for IDHEAS-ECA.
- IDTABLE-23 and -24 (Task Completion Time) are on the way.
- IDTABLE-25 (dependency), -26 (recovery) and -27 (main drivers) are in piloting.
- Areas lacking human error data
- CFMs: Interteam Coordination
- PIFs: Work Process, Team and Organizational Factors 15
Outline I. Approach of using human error data for HRA II. Data source evaluation III. Human error data generalization (IDTABLEs)
IV. The story of PIF combination 16
A story of PIF combination
- An operators HEP is 0.01 in nominal conditions, 0.05 in loud burst noise environment, and 0.1 under poor visibility. What is his HEP when working under loud noise and poor visibility?
- Answer 1: Additive 0.05 + 0.10 = 0.15
- Answer 2: Multiplicative 0.01 x 5 x 10 = 0.5 17
Whats in data Additive Multiplicative Subtractive (or interactive)
Error rate PIF2 poor PIF2 Good PIF1 Good Poor Good Poor Good Poor 18
Whats in data Additive Multiplicative Subtractive (or interactive)
Error rate PIF2 poor PIF2 Good PIF1 Good Poor Good Poor Good Poor Most Some Rare IDHEAS-DATA observation from 100+ studies evaluated with human error data under individual and PIF combination:
- Most datapoints are roughly additive
- Some datapoints show multiplicative 19
Why and when PIF combination is more than Additive?
- If both PIFs demand the same cognitive resource, and the demand of a single PIF already approaches to the capacity limit, then
- the combined effect can be more than the Additive effects;
- This reflects the catastrophic effect of exceeding the capacity limit.
Working memory capacity 7 Working memory capacity and interfere (Prinzo et al, 2006) (Kane & Engle, 2000) 35 6
Words recalled 5
% errors Words recalled 4
5 3 High Span Low Span 2
- of working memory items 1 2 3 Proactive Listinterfere list 20 20
Meta-analysis on PIF combination PIFs # of Findings Ref.
studies Noise, 51 Combined effect is no more than the added Grether temperature, reports single effects and can be predicted from single 1970 sleep loss effects.
Noise and 20~30 The majority of evidence indicates that noise Hancock heat reports and heat do not interact significantly within 2010 the ranges experienced commonly in the industrial setting.
Distraction, 23 data- Additive fits better than Multiplicative; Xing experience, points Additive over estimates for large PIF weights 2015 HSI, others Cognitive 51 Additive accounted for ~ 91% of job Iddeking ability and reports performance data; Multiplicative accounted for e motivation on only about 9% of the explained variance. 2017 performance 21
Perspective of IDTABLE-21: PIF Interaction
- Solid evidence that most PIF combinations are additive.
- IDTABLE-21 should focus on PIF interaction:
- Interaction between a base PIF and modification PIFs
- More-than-additive interaction
- Red flag PIF combinations 22
Summary of IDHEAS-DATA
- Human error data of various sources are generalized into IDHEAS-DATA with IDHEAS cognitive failure modes (CFMs) and PIF attributes
- Data generalization is generic with IDHEAS CFMs and PIF attributes; Data integration is specific to the HRA method or application that uses the data.
- Data generalization is an on-going, continuous effort; Data integration should be periodically updated.
23
Backup slides Integration of human error data for IDHEAS-ECA 24
IDHEAS-ECA uses the HEP Quantification Model Recovery factor; set HEP from Base PIFs PIF weight factors from to 1 unless data Modification PIFs suggest otherwise PIF interaction factor; set to 1 with linear combination IDHEAS-ECA needs:
Data integration process The process of integrating human error data is described as follows:
- 2) Use the initial estimation to detach multi-component data into single-component ones;
- 3) Integrate all the single-component and detached multi-component datapoints to estimate the range and mean of a base HEP or PIF weight;
- 5) Iterate the process 2), 3), and 4) until the obtained values represent the breath of the available data.
26
Approaches used in integration process The confidentiality in integrating a set of data to generate a single representative value or probabilistic distribution depends on the sample size and quality of the data set.
The following approaches were used in the integration for IDHEAS-ECA:
(See notes)
- 2) No single-component data exclusive for a base HEP or PIF weight, but there were multi-component datapoints on the combined effects of several CFMs and/or PIF attributes
- 3) No datapoint for a PIF weight
- 4) Consistency checking and adjustment with benchmark values 27
Example - IDHEAS-DATA IDTABLE-21 Lowest HEPs for Failure of Detection Criteria for lowest HEPs:
Error Task TA - Time adequacy Uncertainty REF rate SelfV - Self verification TeamV - Team verification Rec - Recovery O - other factors (Y-Yes, N - No, M-Mixed Un-Unknown) 1 2.1E-3 NPP operators alarm detection in simulator TA-Yes, SelfV-Y, (Other PIFs may exist) [26]
(4/1872) training. Alarms are self-revealing TeamV-Y, R-Unknown O - Y (unspecified) 2 3.4E-3 NPP operators check indicators in simulator TA-Yes, SelfV-Yes, (Other PIFs may exist) [26]
(3/870) training, procedure directed checking. TeamV-yes, Rec - Unknown O - Y (unspecified) 3 5E-4 Military operators read meters, Alphanumeric TA-Y, SelfV-Y, (Maybe time [109]
reading, Detection straight-forward TeamV-No, Rec-No constraint, 10K+ source data trials) 4 E-4 Estimated lowest probity of human failure TA-Yes, SelfV-Yes, (Engineering judgment) [110]
events TeamV-yes, Rec - Unknown 5 E-4 Simplest possible tasks TA-Yes, SelfV-Yes, (Engineering judgment) [111]
TeamV-Unknown, Rec - Unknown 6 E-3 Routine simple tasks TA-Yes, SelfV-Yes, (Engineering judgment) [111]
TeamV-Unknown, Rec - Unknown O - Maybe weak complexity 7 5E-3 Line-oriented text editor. Error rate per word TA-Yes, SelfV-Yes, No apparent [112]
TeamV-No, Rec - No uncertainty 8 5E-3 Reading a gauge incorrectly. Per read TA-Yes, SelfV-Yes, No apparent [113]
TeamV-No, Rec - Unknown uncertainty O - HSI 9 E-3 Interpreting indicator on an indicator lamp. TA-Yes, SelfV-Yes, (Engineering judgment) [109]
Per interpretation TeamV-Unknown, Rec - Unknown O- complexity in interpreting indicator 10 9E-4 NPP operator simulator runs TA - Y, Selv-V - Y No apparent [114, 115]
TeamV - Y, R - Unknown uncertainty O - Mixed complexity 11 5.3E-4 Gather information and evaluate parameters TA - Y, Selv-V - Y 28 No apparent [116]
TeamV - Y, R - Yes uncertainty 12 9E-3 Collision avoidance and target monitoring in TA - Y, Selv-V - Yes Dual task [27]
simulated ship control Fixed situation TeamV No R Yes
Detaching multi-component human error data The critical step in the process is detaching multi-component datapoints. The following rules are derived from initial estimates of base HEPs of task complexity and PIF attribute weights. They are used for detaching:
- 1) If SelfV=NO or TeamV=NO, the detached error rate is the original error rate divided by a factor of 5; If both are NO, the detached error rate is the original error rate divided by a factor of 10.
- 2) If Recovery = YES, the detached error rate is the original error rate multiplied by a factor range of 2 to 10.
- 3) If there are other PIFs, the detached error rate is the original error rate divided by multiplication of a factor range of (5 to 10 for complexity) and the sum of the weights of other PIF attributes. The weights of the PIF attributes are from the initiation estimation of the single-component data in IDHEAS-DATA.
29
Table 3-8: Detached human error rates for the lowest HEP of Failure of Detection CFM Error rate Criteria for lowest HEPs Detached error rate Notes 1 2.1E-3 TA-Yes, SelfV-Y, 2.1E-3 / (5 to 10) = 2.1E-4 A factor of 5 to 10 represents the combined (4/1872) TeamV-Y, R-Unknown to 4E-4 effect of possible other PIFs O - Y (unspecified) 2 3.4E-3 TA-Yes, SelfV-Yes, 3.4E-3 / (5 to 10) = 3.4E-4 A factor of 5 to 10 represents the combined (3/870) TeamV-yes, Rec - Unknown to 7E-4 effect of possible other PIFs O - Y (unspecified) 3 5E-4 TA-Y, SelfV-Y, 5E-4 / 5 = 1E-4 Divided by 5 for no team verification TeamV-No, Rec-No 4 E-4 TA-Yes, SelfV-Yes, E-4 No change TeamV-yes, Rec - Unknown 5 E-4 TA-Yes, SelfV-Yes, E-4 No change TeamV-Unknown, Rec - Unknown 6 E-3 TA-Yes, SelfV-Yes, E-3 / 5 = 2E-4 Divided by 5 for weak complexity TeamV-Unknown, Rec - Unknown O - Maybe weak complexity 7 5E-3 TA-Yes, SelfV-Yes, 5E-3 / 10 = 2E-4 Divided by (5+5) for lack of self and team TeamV-No, Rec - No verification 8 5E-3 TA-Yes, SelfV-Yes, 5E-3 / (5+2) = 7E-4 Divided by (5+2) for lack of self verification and TeamV-No, Rec - Unknown possible HSI attributes O - Maybe HSI 9 E-3 TA-Yes, SelfV-Yes, E-3 / 5 = 2E-4 Divided by 5 for no team verification.
TeamV-Unknown, Rec - Unknown 10 9E-4 TA - Y, Selv-V - Y 9E-4 / (5 t o10) = Divided by (5 to 10) for mixed complexity TeamV - Y, R - Unknown 9E-5 to 4.8E-4 O - Mixed complexity 11 5.3E-4 TA - Y, Selv-V - Y 5.3E-4 x 2 / (5-10) Multiplied by 2 for existence of recovery TeamV - Y, R - Yes = 1.06E-4 to 2.12E-4 O - Mixed complexity 12 9E-3 TA - Y, Selv-V - Yes 9E-3 / (5 to 10) x (5-10) = Divided by (5 to 10) for mixed complexity and TeamV - No, R - Yes 9E-5 to 3.6E-4 divided by (5 to 10) for dual task.
O - Dual task, and maybe mixed complexity 30
Table 3-9. Single-component and detached multi-component human error rates for the lowest HEP of Failure of Detection Single- Multi- component Bounding component detachable A - Nuclear operation 2.1E-4 to 4E-4, 3.4E-4 to 7E-4, 9E-5 to 4.8E-4 B - Other operation 1.06E-4 to 1E-4, 2.12E-4 2E-4 7E-4 C - Controlled experiment E-4, 2E-4 9E-5 to 3.6E-4 D - Expert judgment E-4 2E-4 E - Unspecific 31
Table 3-9. Single-component and detached multi-component human error rates for the lowest HEP of Failure of Detection Figure 3-1. The human error rates for the lowest HEP of Failure of Detection Category A datapoints: [ 1.8, 3.6, 5.3]E-4 for lower bound, mean, and upper bound; Based on the data, the value 1E-4 is taken as the Category B datapoints: [ 1.06, 2.8, 2.1]E-4 lowest HEP for Failure of Category C datapoints: [ 0.9, 1.7, 3.6]E-4 Detection.
Category A, B, C datapoints: [1.4, 1.8, 4.4 ]E-4 32
A story of two type of PIFs (Backup slides)
A story of two type of PIFs
- 1. What s in the cognitive basis Mental model Mental representation Signal / Noise Cognitive (outcome of processing macrocognitive functions)
Signal-noise ratio - Information Availability and Reliability Base PIFs Mental model - Scenario Familiarity Demands for cognitive processing - Task Complexity Modification PIFs - modify the base PIFs
How human achieves Understanding (NUREG-2114)
Attention & Working Memory for integration Knowledge Expertise Experience External World Detect/Notice New Info Prior Info Data Frame Percept LTM Goals Work process Workload Subconscious Desires Interfaces Procedures Fatigue 35 Motivation
A story of two type of PIFs
- 2. Whats in data about PIF effects on HEPs 3.5E-1 Information Availability and Reliability Base can vary HEP from nearly 0 to 1; Scenario Familiarity can vary HEP
% errors PIFs from nearly 0 to 1; Task Complexity can vary HEP from nearly 0 to 1; E-3 Base PIF - Task complexity Modification PIFs -
A single modification PIF attribute typically varies HEP in the range of 1.1 to 10 times, with a few exception high up to 30 times for feasible tasks.
- example data from German NPP maintenance performance database Memorized task step not remembered in carrying outa sequence of tasks Scenario Familiarity (frequently to extreme rarely performed tasks) varied the error rate from 7.78E-5 to 3.52E-1