ML20083J313

From kanterella
Revision as of 22:47, 18 April 2020 by StriderTol (talk | contribs) (StriderTol Bot insert)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
P5 Preliminary Analysis Results of SACADA for HEPs
ML20083J313
Person / Time
Issue date: 03/12/2020
From: Chang Y
Office of Nuclear Regulatory Research
To:
J. Chang
Shared Package
ML20083J304 List:
References
Download: ML20083J313 (25)
v • d • e


Text

Preliminary Analysis Results of SACADA Data for HEPs Y. James Chang, Ph.D.

U.S. Nuclear Regulatory Commission James.Chang@nrc.gov Presented at HRA Data Workshop at US Nuclear Regulatory Commission March 12-13, 2020 1

SACADA Development Background

  • Aim to collect operator performance information in simulator training to improve human error probability (HEP) estimates

- Developed software for nuclear plants to use in operator simulator training

- Plant-collected information shared with the NRC

  • Strength Collect large amount of data points to provide statistical evidence of the relation between context and cognitive performance
  • Weakness Data taxonomy needs to accommodate constraints in operator simulator training
  • Strength & Weakness Information entered by operation instructors and crews, not researchers
  • Strength: (1) Enable a long-term, continuous data collection; (2) Data are accurate because entered by the instructors and crew in the simulator training and immediate immediately after the training
  • Weakness: data are entered in a time constraint environment SACADA: Scenario Authoring, Characterization, and Debriefing Application 2

Data of Interest

  • Basic human error probability (HEPs) of various macrocognitive functions (MCF)

- MCFs: Detection (Alarms vs. Indicators),

Understanding/Diagnosis, Deciding, and Action

  • Effects of individual performance influencing factors (PIFs) and combinations of PIFs
  • MCF-dependent effects
  • PIF combinations having cliff effect
  • How does the PIF combined effect work (multiplication or addition or something else)?
  • Effects not captured in the current data taxonomy 3

Data used in this analysis are partially available to the public in https://www.nrc.gov/data/

4

Performance Influencing Factors of Diagnosis (Partial List)

  1. PIF PIF Attribute o Skill-based 1 Diagnosis Basis o Procedure-based o Knowledge-based o Standard 2 Familiarity o Novel o Anomaly Timing of information 3 Information Integration Ambiguous information Integration required o Specific 4 Information specificity o Not specific Missing Information 5 Information quality Misleading Information Conflicting Information o Normal 6 Workload o Concurrent Demandes o Multiple Concurrent Demandes o Expansive Time Available 7 Time Criticality o Nominal Time Available o Barely Adequate Time Available 5

Data Unit: Training Objective Elements Example of Malfunction of SGTR in 1B SG

  1. Training Objective Element MCF UNSAT Ratio 1 Diagnose SGTR in B SG Diagnosis 0/13 2 Direct a reactor trip and safety injection based on increasing RCS leakage. Diagnosis 0/13 Performs Immediate actions of 0POP05-EO-EO00, including RNO actions for Throttle Valve 3 Deciding 0/13 stuck open.

4 -7 0/13 8 Transitions to EO30 SGTR Deciding 0/13 9 Completes isolation of ruptured S/G: (4 isolation tasks) Action 0/13 Properly select and maintain target temperature for cooldown based on the chart provided 10 Diagnosis 0/13 in EO30.

11 Directs/initiates RCS cooldown Action 0/13 12 Directs/stops RCS cooldown and maintains < target temperature Action 0/13 Depressurize RCS to meet SI termination criteria before either of the following occur: (1) SG 13 Action 0/13 PORV or Safety Valve opens (2) SG Narrow Range level goes Off scale high Terminate SI and control RCS pressure and makeup flow so that RCS pressure is at SG 14 Deciding 0/13 Pressure and stable before the end of the scenario 15 Declares an Alert Based on SGTR greater than the capacity of one CCP (FA1). Diagnosis 0/13 16 Other items to discuss None 0/13 6

Overview of the Data (As of 4/2019, available in https://www.nrc.gov/data/)

Cognitive Type Data Points # of Unique Context # of Possible Available Unique Context*

Alarm detection 2099 61 ~ 60,000 Indicator detection 2198 95 ~ 80,000 Diagnosis/

2613 116 ~ 3,000,000 Understanding Deciding 6019 147 ~ 60,000 Action 5167 159 ~ 3,000,000 TOTAL 18096 578

Overview of Operator Performance UNSAT Ratio Distributions 1

0.9 0.8 0.7 Portion of Data Points 0.6 0.5 0.4 0.3 0.2 0.1 0

< 0.01 0.01 - 0.1 0.1-0.2 0.2-0.3 0.3-0.4 0.4-0.5 UNSAT Ratio Calculated based on the Training Objective Elements with the same context Detect Alarm Detect Indicator Diagnosis Deciding Action 8

Preliminary Analysis Results (Grouped Based on Same Context, in https://www.nrc.gov/data/)

UNSAT Ratio Average Highest Value* Highest Value/

MCF (All Context) (A Specific Context) Average Detection (Alarm) 6.2E-3 0.19 31 (13/2099) (5/26)

Detection (Indicator) 7.7E-3 0.21 27 (17/2198) (3/14)

Diagnosis 1.1E-2 0.5 45 (28/2613) (3/6)

Decisionmaking 5.3E-3 0.13 25 (32/8019) (2/15)

Action 1.1E-2 0.33 30 (59/5167) (4/12)

  • Calculate based on the crew performance of a Training Objective Element of a specific scenario 9

Single PIF Effects

  • Based on available data, the effects were calculated by changing the status of one-and-only-one PIF (before and after)
  • The effects may or may not only caused by difference of the PIFs status PIF because the data populations before and after are exclusive to each other.
  • Many context groups have zero UNSAT which are not discussed in this analysis.
  • The directions of effects are generally as expected.

Skill- Rule- KB-Based Based Based 10

Single PIF Effects - Alarm Detection (Change a PIFs status only)

Status 1 Status 2 Status 3 PIF UNSAT Ratio 1 (2)/(1) (3)/(1)

Alarm Board Dark Busy Overloaded 2.1E-3 1 5.0E-3 2.4 3.9E-2 18.4 Status (2/953) (5/991) (6/155)

Self Awareness Detection Revealing /Inspect 1 23.8 Mode 2.1E-3 5.1E-2 (4/1872) (9/177)

Note: the other PIFs statuses are likely different in the above data points.

The effects (UNSAT ratio 2/UNSAT Ratio 1) may include the other factors effects.

11

Single PIF Effects - Indicator Detection (Change a PIFs status only)

Status 1 Status 2 UNSAT Ratio 2/

PIF UNSAT Ratio 1 UNSAT Ratio 2 UNSAT Ratio 1 Procedure Awareness/

KB-Driven Directed Inspection Monitoring Detection Mode Check 1 6.4E-3 1.9 1.9E-2 5.4 3.4E-3 (5/782)

(8/432)

(3/870)

Distinct Slight Change Change 4.4E-3 Change Degree 1.0E-2 1 (4/907) 0.44 (13/1291)

Meter Computer Meter Type 2.4E-2 1 2.9E-3 0.1 (11/463) (3/1019)

Note: the other PIFs statuses are likely different in the above data points.

The effects (UNSAT ratio 2/UNSAT Ratio 1) may include the other factors effects. 12

Single PIF Effects - Diagnosis Detection (1/2)

(Change a PIFs status only)

Status 1 Status 2 Status 3 PIF UNSAT Ratio 1 (2)/(1) (3)/(1)

Diagnosis Standard Novel Anomaly 7.6E-3 1 8.8E-3 1.2 1.2E-1 15.3 Familiarity (13/1718) (7/800) (8/69)

Poor Not Exist Exist Information 9.5E-3 1 4.5E-2 4.7 Timing (24/2524) (4/89)

Ambiguous Not Exist Exist Information 8.1E-3 1 3.4E-2 4.2 (19/2350) (9/263)

Note: the other PIFs statuses are likely different in the above data points.

The effects (UNSAT ratio 2/UNSAT Ratio 1) may include the other factors effects.

13

Single PIF Effects - Diagnosis Detection (2/2)

(Change a PIFs status only)

Status 1 Status 2 PIF UNSAT Ratio 1 UNSAT Ratio 2 Integration Not Exist Exist 9.5E-3 1 2.7E-2 2.9 Required (23/2429) (5/184)

Information Specific Not Specific 7.7E-3 1 1.5E-2 1.9 Specificity (10/1293) (16/1077)

Note: the other PIFs statuses are likely different in the above data points.

The effects (UNSAT ratio 2/UNSAT Ratio 1) may include the other factors effects. 14

Single PIF Effects - Decisionmaking (Change a PIFs status only)

Status 1 Status 2 Status 3 PIF UNSAT Ratio 1 (2)/(1) (3)/(1)

Adaption Decision Standard Required Anomaly 5.1E-3 1 1.3 1.1E-2 2.1 Familiarity (24/4691) 6.6E-3 (1/92)

(7/1062)

Competing Decision Clear Uncertain 1 1.3 Priority 1.8 Uncertainty 4.9E-3 6.3E-3 (22/4460) (4/634) 8.7E-3 (6/693)

Note: the other PIFs statuses are likely different in the above data points.

The effects (UNSAT ratio 2/UNSAT Ratio 1) may include the other factors effects. 15

Single PIF Effects - Action (Change a PIFs status only)

Status 1 Status 2 Status 3 PIF UNSAT Ratio 1 (2)/(1) (3)/(1)

Simple and Action Distinct Order Maintaining 1 1.1E-2 1.1 1.5E-2 1.5 Type 1.0E-2 (21/1876) (12/793)

(22/2151)

Procedure Skill-of-the- STAR(Faulted Action -Based Craft Hardware) 1 1.3 3.2 Guidance 1.1E-2 8.9E-3 1.3E-2 (55/4786) (2/224) (2/157)

Additional Not Exist Exist Mental Effort 1.0E-2 1 3.4E-2 3.2 Required (52/4960) (7/207)

Note: the other PIFs statuses are likely different in the above data points.

16 The effects (UNSAT ratio 2/UNSAT Ratio 1) may include the other factors effects.

Workload Effects in Various MCFs Normal Concurrent Demands Multiple (1) (2) Concurrent Demands (3)

Alarm detection 1.2E-3 1 9.6E-3 7.7 0 --

(1/801) (12/1255) (0/29)

Indicator 2.8E-3 7.8E-3 2.5E-2 1 2.8 9.0 Detection (2/711) (10/1289) (5/198)

Diagnosis 1.3E-2 1 8.5E-3 0.6 2.9E-2 2.2 (9/674) (15/1759) (4/138)

Deciding 3.7E-3 1 5.9E-3 1.6 5.2E-3 1.4 (5/1342) (23/3899) (4/762)

Action 9.2E-3 1 9.4E-3 1.0 2.0E-2 2.2 (9/983) (30/3181) (20/987)

PIF effects may be MCF-dependent. Need detailed analyses for confirmation.

Note: the other PIFs statuses are likely different in the above data points.

The effects (UNSAT ratio 2/UNSAT Ratio 1) may include the other factors effects. 17

PIF Combined Effects -

Alarm Detection PIF Base PIF Status PIF Status 1 PIF Status 2 Dark Busy Overloaded Alarm Board Status 2.1E-3 1 5.0E-3 2.4 3.9E-2 18.4 (2/953) (5/991) (6/155)

Multiple Concurrent Normal Concurrent Workload - Alarm Demand Detection 1.2E-3 1 9.3E-2 7.7 Demands --

(1/801) 0 (12/1255)

(0/29)

Workload Alarm Board Status Dark Busy Overloaded Normal 1.5E-3 0 0 1 -- --

(1/687) (0/111) (0/3)

Concurrent 4.0E-3 2.7/ 5.7E-3 3.9/ 4.9E-2 33.5/

Demand (1/252) 1 (5/880) 15.8 (6/123) 12.3 Multiple 0 0 0 Concurrent (0/0) (0/0) (0/29) 18

PIF Combined Effects -

Indicator Detection PIF Base PIF Status PIF Status 1 PIF Status 2 Procedure- Awareness/ KB-Driven Directed Inspection Monitoring Detection Mode Check 1 6.4E-3E-2 1.9 1.9E-2 5.4 3.4E-3 (3/87) (5/782) (8/432)

Multiple Concurrent Normal Concurrent Workload - Indicator Demand Detection 2.8E-3 1 7.8E-3 2.8 Demands 9.0 (2/711) 2.5E-2 (10/1298)

(5/198)

Workload Indicator Detection Mode Procedure directed Check Awareness/ KB-Driven Inspection Monitoring Normal 0 4.3E-3 0

-- 1.6 --

(0/324) (1/235) (0/50)

Concurrent Demand 2.7E-3 7.5E-3 1.4E-2 1 2.8 5.1 (1/376) (4/533) (5/366)

Multiple Concurrent 1.2E-2 0 0.21 4.4 -- 80.6 (2/170) (0/14) (3/14) 19

Are PIFs Effects Context Independent?

PIF Poor Info Ambiguous Require Info Not Timing Info Integration Specific PIF Effect 4.7 4.2 2.9 1.9 Poor Infor Timing No Yes Ambiguous Info No Yes No Yes 1 -- 31 --

No Require Integration (3/1116) (0/24) (3/36) (0/0)

No 2.7 16 -- --

Info Not Yes (6/812) (9/209) (0/13) (0/0)

Specific 12.7 -- -- --

No Yes (4/117) (0/0) (0/0) (0/0)

Yes (0/0) (0/0) (0/0) (0/0) 20

Single and Combined PIFs Effects

  • A PIFs effects should be evaluated in the condition when the other factors statuses remain identical; otherwise, the results may be misleading.
  • Based on currently available for analysis, it does not conclude whether the PIFs combinational effect is addition or multiplication.

21

Support IDHEAS-ECA Method

  • Ideally, data in optimal operational conditions should be used to inform for base HEPs. Indirect indications may be able to inform base HEPs with limited data.

UNSAT Ratio IDHEAS-ECA (2)

Average Base HEP MCF (1)/(2)

(1) (2)

Detection (Alarm) 6.2E-3 1E-4 62 Detection (Indicator) 7.7E-3 1E-4 77 Diagnosis 1.1E-2 1E-3 11 Decisionmaking 5.3E-3 1E-4 53 Action 1.1E-2 1E-3 11 22

Does task Importance Affect Human Performance

  • SACADA data does not support operator performance is affected by task importance
  • The table shows statistics of SACADAs TOEs classified in four importance levels

- Item (1) the scenario-specific TOEs met

  • UNSAT ratio greater than 10%; and
  • >= 2 crews dispositioned as UNSATs

- The higher (1)/(2) value in critical tasks (the most important tasks) is attributed to scenario design

  • Critical tasks are typically appear in the end of scenarios whether the situation is generally more complex).

Importance Level # of Items (1) Total # of Items (2) (1)/(2) 1 (Critical Task) 2 71 2.8%

2 (Safety related) 11 822 1.3%

3 (professional tasks) 20 1619 1.2%

4 (General) 8 559 1.4%

23

Qualitative Context Related to TOEs with high UNSAT Ratios General suspects with scenario-system specific information

  • Unfamiliarity + Information masking

- Fail to trip RCPS because tripping criteria were determined not met

  • HHPI is running and indicating flow rate
  • The local valve was closed so the flow did not reach to the targeted location
  • Not a trivia to identify the flow did not reach to the target area
  • Multiple concurrent tasks

- Fail to trip ESF DG manually before its automatic trip

  • Monitor ESF DGs parameters and detect the lube oil temperature was increasing abnormally
  • The crew was tie up to responding a previous malfunction 24

Ongoing SACADA Work

  • Recently completed SACADA-2 tool development (simulator data portion)
  • Welcome collaborations on using SACADA to collect operator performance data
  • Analyzing SACADA data to improve IDHEAS-ECA
  • Welcome collaborations on analyzing SACADA data

Contact:

James.Chang@nrc.gov 25

Retrieved from "https://kanterella.com/w/index.php?title=ML20083J313&oldid=2746705"