ML18081A086

APR1400-E-I-NR-14008-NP, Rev. 2, Human Factors Verification and Validation Implementation Plan

ML18081A086
Person / Time
Site:	05200046
Issue date:	01/31/2018
From:	Korea Hydro & Nuclear Power Co, Ltd
To:	NRC/NRO/DNRL
Shared Package
ML18081A107	List: ML18081A082 ML18081A084 ML18081A086 ML18081A088 ML18081A090 ML18081A091 ML18081A093 ML18081A096 ML18081A098 ML18081A099 ML18081A101 ML18081A103
References
MKD/NW-18-0037L APR1400-E-I-NR-14008-NP, Rev. 2
Download: ML18081A086 (104)

Text

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 2 Human Factors Verification and Validation Implementation Plan Revision 2 Non-Proprietary January 2018 Copyright 2018 Korea Electric Power Corporation &

Korea Hydro & Nuclear Power Co., Ltd All Rights Reserved KEPCO & KHNP

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 REVISION HISTORY Revision Date Page Description December 0 All First Issue 2014 xii Terminology changed. (SAGAT -> SART)

(Acronyms R_352-8205(74) and Abbreviation) 4 Description for DIHA added.

(3) R_358-8449(89) 8 Description that DIHA credited in TAA and D3CA (4.1.1.2) added.

R_358-8449(89) 10 Description for error tolerance and recovery added.

(4.1.1.3) R_352-8205(62) 14 Editorial correction.

(Table 4-3) R_352-8205(61) 16 Description that DIHA credited in TAA and D3CA (4.1.3) added.

R_358-8449(89) 1 March 2017 22 Description for HEDs information added.

(4.3.3) R_352-8205(64) 25 Design verification process modified.

(Figure 4-2) R_352-8205(65) 26 Description for design verification process modified.

(4.4.2) R_352-8205(65) 28 Description for independent teams responsibility (4.5) added.

R_352-8205(66) 30 Description for independent team added.

(4.5.1) R_352-8205(66) 31 Description for error tolerance added.

(4.5.2) R_352-8205(62) 33 Description for MCR composition added.

(4.5.4) R_352-8205(67), R_352-8205(67R)

KEPCO & KHNP ii

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Revision Date Page Description 35 Description for pass/fail measure criteria added.

(4.5.5) R_352-8205(69) 35 Description for evaluation of convergence between (4.5.5) measures added.

R_352-8205(80) 36 Editorial correction.

(4.5.5) R_352-8205(73) 36 Terminology changed. (SAGAT -> SART)

(4.5.5) R_352-8205(74) 37 Description for performance measure during scenario (4.5.5.1) added.

R_352-8205(69) 37 Terminology changed. (SAGAT -> SART)

(4.5.5.1) R_352-8205(74) 38 Description for measurement characteristic added.

(4.5.5.1) R_352-8205(73) 39 Editorial correction.

(4.5.5.1) R_352-8205(71) 40 Description for measurement characteristic modified.

(4.5.5.1) R_352-8205(73R) 40 Editorial correction (4.5.5.1) S_MES/KP-160186M 41 Description for SAGAT deleted and SART added.

(4.5.5.1) R_352-8205(74) 42 Description for observer questionnaire added.

(4.5.5.2) R_352-8205(71) 44 Description for performance measure criteria added.

(4.5.5.2) R_352-8205(74) 45 Description for performance evaluation measure (4.5.5.2) modified.

R_352-8205(75) 46 Description for secondary task measure questionnaire (4.5.5.2) added.

R_352-8205(70) 47 Description for basis for performance criteria added.

(Table 4-6) R_352-8205(74) 47 Description for performance evaluation measure KEPCO & KHNP iii

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Revision Date Page Description (Table 4-6) modified.

R_352-8205(75) 50 Description how test procedure minimize bias added.

(4.5.6.2) R_352-8205(77) 50 Description that test procedure minimize bias deleted.

(4.5.6.2.1) R_352-8205(77) 51 Description for validation team added.

(4.5.6.3) R_352-8205(66) 53 Description for degree of freedom for scenario (4.5.7) evaluation added.

R_352-8205(78) 53 Description for quantitative and qualitative method (4.5.7) added.

R_352-8205(78R) 54 Description for scenario evaluation method added.

(4.5.7.1) R_352-8205(79) 58 Description for time margin of DIHA added.

(4.5.7.2) R_358-8449(89) 60 Description for margin of error added.

(4.5.7.3) R_352-8205(81) 60 Description for margin of error modified.

(4.5.7.3) R_352-8205(81R) 62 Description for qualitative method added.

(4.5.8) R_352-8205(78R) 63-64 Description for HEDs selection modified.

(4.5.9.2) R_298-8356(41) 63-64 Definition of HED modified.

(4.5.9.2) R_352-8205(82) xi, xii, 12, 20, Editorial corrections (typos, acronyms, references) 22, 23, 28, 36, 68 (Overall) x Figure for ITTAAC Closure Schedule added (Figure 3-1) R_553-9084(134R) 2 January 2018 4-5 Description and figure for ITTAC Closure Schedule (3) added.

R_553-9084(134R)

KEPCO & KHNP iv

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Revision Date Page Description 26 Editorial correction (Underlining, text line arrangement, (4.4.2) deletion of unnecessary sentence)

CR-17-J220 37 Editorial correction (typo)

(4.5.5.1) CR-17-J220 48 Editorial correction (typo)

(4.5.6) CR-17-J220 60 Description for pilot test deleted.

(4.5.7.4) R_352-8205(78R)

CR-17-J220 61 Editorial corrections (text line arrangement, numbering)

(4.5.7.4) CR-17-J220 61 Description for margin of error deleted.

(4.5.7.4) R_352-8205(78R)

CR-17-J220 68 Revision No. and Issue Year/Month updated (7.2,7.3,7.4, 7.6,7.7)

KEPCO & KHNP v

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 This document was prepared for the design certification application to the U.S. Nuclear Regulatory Commission and contains technological information that constitutes intellectual property of Korea Hydro & Nuclear Power Co., Ltd.

Copying, using, or distributing the information in this document in whole or in part is permitted only to the U.S.

Nuclear Regulatory Commission and its contractors for the purpose of reviewing design certification application materials. Other uses are strictly prohibited without the written permission of Korea Electric Power Corporation and Korea Hydro & Nuclear Power Co., Ltd.

KEPCO & KHNP vi

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 ABSTRACT This document provides the implementation plan (IP) for the Human Factors Engineering (HFE)

Verification and Validation (V&V) program element, which is one of twelve program elements within the Advanced Power Reactor 1400 (APR1400) HFE Program. This IP governs the technical activities conducted within the verification and validation (V&V) program element by defining the scope and methodology of the V&V, including the output products generated and the qualifications of the personnel who will generate those products.

The main purposes of the V&V is to comprehensively determine that the human system interface (HSI) design conforms to HFE design principles and that it enables plant personnel to successfully perform their tasks to assure plant safety and operational goals.

The scope of the V&V program includes the main control room (MCR), remote shutdown room (RSR),

and voice communications when it can influence the MCR crews performance, between the MCR and the technical support center (TSC), emergency operations facility (EOF) and other off site emergency entities.

The V&V program also includes local control stations (LCS) associated with important human actions.

This report contains the V&V IP describing five major V&V activities that are in full compliance with NUREG-0711, Human Factors Engineering Program Review Model:

1. Sampling of operational conditions (SOC)

2. Design Verification

3. Integrated System Validation (ISV)

4. Human Engineering Discrepancy (HED) Resolution

5. Documentation of the results of the V&V program in a results summary report (ReSR)

It is impractical and unnecessary to review the entire HSI; therefore a SOC process is applied to support the selection of HSIs to include in the V&V. The resulting HSIs are applied as guidance for the design verification and are incorporated into realistic scenarios that are used in the ISV.

The design verification is composed of two types of evaluations, task support verification and human factors engineering (HFE) design verification. Task support verification will assess the HSI design to ensure that it supports the tasks identified in the task analysis (TA). The HFE design verification confirms that the HSI meets the HFE requirements of the Style Guide (Reference 2). The design verification is completed early in the V&V program so that any HSI modifications identified during the design verification and determined to require a change to the HSI design are completed before the start of the ISV.

The ISV confirms the acceptable performance of the integrated system through comprehensive performance based, human in the loop, testing on a verified dynamic MCR simulator using a set of scenarios based on the SOC. As needed, part task simulation and interface mockup are used to augment the application of the simulator. The use of these alternative test platforms is limited to cases where the use of the MCR simulator is not reasonable, e.g., local control stations identified by the treatment of important human actions (TIHA) and the TSC.

KEPCO & KHNP vii

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Acceptable performance of the HSI is based on measures of plant performance, human task performance, situation awareness, cognitive and physical workload, and anthropometric and physiological factors.

Measures are divided into two types, pass/fail and diagnostic.

The human engineering discrepancies process is used to track, through closure, all discrepant findings resulting from the V&V program.

The V&V program uses results from the operating experience review (OER), TA, TIHAs, staffing and qualifications (S&Q) and human system interface design (HD). The V&V program also applies the results of the procedure and training development programs.

The V&V program confirms that the HSI design conforms to HFE design principals and that plant staff are able to perform their tasks to achieve plant safety and plant operational goals. The V&V results are documented in the ReSR. The ReSR confirms that the V&V program was conducted in accordance with this implementation plan and the HSI design is acceptable.

IP has the following contents:

• Section 1 of this IP presents the purpose of the V&V program and of this IP.

• Section 2 presents the scope of the V&V program.

• Section 3 is an overview of the V&V methodology and documentation program and discusses the link between the V&V program and other elements of the HFE program.

• Section 4 describes the details of the methodology criteria and documentation of the V&V program.

• Section 5 describes the makeup of the V&V teams personnel along with their professional qualifications.

• Section 6 describes the content of the ReSR

• Section 7 references

• Section 8 definitions

• Appendix A contains a matrix of how this IP meets the criteria from NUREG-0711 rev 3

• Appendix B contains the Behavioral Anchored Rating Scales (BARS) used to analyze ISV data

• Appendix C contains the NASA-TLX questionnaire to be used in the ISV.

KEPCO & KHNP viii

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TABLE OF CONTENTS

1. PURPOSE ............................................................................................................ 1

2. SCOPE ................................................................................................................. 3

3. METHODOLOGY OVERVIEW ............................................................................... 4 3.1. V&V Interfaces in the HFE Program ............................................................................................... 5

4. IMPLEMENTATION .............................................................................................. 6 4.1. Sampling of Operational Conditions ............................................................................................... 6 4.1.1. Sampling Dimensions ..................................................................................................................... 6 4.1.2. Identification of Scenarios ............................................................................................................ 15 4.1.3. Scenario Design Requirements .................................................................................................... 15 4.1.4. Scenario Definition........................................................................................................................ 16 4.1.5. Scenario Components .................................................................................................................. 17 4.2. Design Verification ........................................................................................................................ 18 4.2.1. Human-system interface Inventory and Characterization ............................................................ 18 4.3. Task Support Verification .............................................................................................................. 20 4.3.1. Verification Criteria........................................................................................................................ 20 4.3.2. General Methodology ................................................................................................................... 20 4.3.3. HED Identification ......................................................................................................................... 22 4.4. HFE Design Verification ................................................................................................................ 23 4.4.1. Verification Criteria........................................................................................................................ 23 4.4.2. General Methodology ................................................................................................................... 25 4.4.3. HED Identification ......................................................................................................................... 27 4.5. Integrated System Validation ........................................................................................................ 28 4.5.1. Validation Team............................................................................................................................. 30 4.5.2. Test Objectives ............................................................................................................................. 31 4.5.3. Validation Testbeds ....................................................................................................................... 32 4.5.4. Plant Personnel (Participants) ...................................................................................................... 32 4.5.5. Performance Measurement .......................................................................................................... 35 4.5.6. Test Design ................................................................................................................................... 48 4.5.7. Data Analysis and HED Identification ........................................................................................... 53 4.5.8. Validation Conclusions .................................................................................................................. 61 4.5.9. Human Engineering Discrepancy Resolution Review Criteria ..................................................... 63

5. IMPLEMENTATION TEAM .................................................................................. 66

6. RESULTS

SUMMARY

REPORT .......................................................................... 67

7. REFERENCES .................................................................................................... 68 KEPCO & KHNP ix

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

8. DEFINITIONS..................................................................................................... 69 APPENDIX A NUREG-0711 REVIEW CRITERIA CONFORMANCE TABLE ...................... A1 APPENDIX B BARS QUESTIONNAIRE ........................................................................... B1 APPENDIX C NASA-TLX QUESTIONNAIRE ................................................................... C1 KEPCO & KHNP x

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 LIST OF TABLES Table 4-1 SOC Dimension Matrix Example (Plant Condition) ................................................................... 11 Table 4-2 SOC Dimension Matrix Example (Types of Personnel Tasks) .................................................. 12 Table 4-3 SOC Dimension Matrix Example (Situational Factors or Error forcing Contexts) ..................... 14 Table 4-4 Operator Experience Record .................................................................................................... 34 Table 4-5 Basis for Performance Criteria .................................................................................................. 43 Table 4-6 Summary of Performance Measures ........................................................................................ 47 Table 4-7 ISV Scenario Sequence Example ............................................................................................. 49 Table 4-8 Example of BARS T-test............................................................................................................ 55 Table 4-9 Example of Expert Correlation Analysis .................................................................................... 55 Table 4-10 Example of T-test Result Form for the Secondary Task .......................................................... 56 Table 4-11 Example of T-test Result Form for the Situation Awareness ................................................... 56 Table 4-12 Example of T-test Result Form for the Workload .................................................................... 57 Table 4-13 Example of T-test for Anthropometric and Physiological Factors ........................................... 58 Table 4-14 Example of Acceptance Criteria for an Individual Scenario including IHA .............................. 59 Table 4-15 Example of Acceptance Criteria for ISV .................................................................................. 62 Table 5-1 Implementation Team Expertise ................................................................................................ 66 LIST OF FIGURES Figure 3-1 ITAAC Closure Schedule ................................................................................................... 5 Figure 4-1 Task Support Verification Process ................................................................................... 21 Figure 4-2 Design Verification Process ............................................................................................. 25 Figure 4-3 Integrated System Validation Process ............................................................................. 29 KEPCO & KHNP xi

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 ACRONYMS AND ABBREVIATIONS ANS American Nuclear Society ANSI American National Standards Institute APR1400 Advanced Power Reactor 1400 BARS behaviorally anchored rating scale BTP Branch Technical Position C&ID control and instrumentation diagram CCW component cooling water CFR Code of Federal Regulations CLD control logic diagram DBA design basis accident DCD Design Control Document DI design implementation DIHA deterministically important human action D3 diversity and defense-in-depth EO electrical operator EOF emergency operation facility EOP emergency operating procedure FPD flat panel display FRA/FA functional requirements analysis and function allocation HA human action HD HSI design HED human engineering discrepancy HFE human factors engineering HFEPP human factors engineering program plan HSI human-system interface HVAC heating, ventilation, and air conditioning I&C instrumentation and control IAEA International Atomic Energy Agency IHA important human action IP implementation plan ISV integrated system validation ITS issue tracking system KEPCO Korea Electric Power Corporation KHNP Korea Hydro & Nuclear Power Co., Ltd.

LCS local control station KEPCO & KHNP xii

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 LDP large display panel LOCA loss of coolant accident MCR main control room NRC Nuclear Regulatory Commission NASA National Aeronautics and Space Administration OER operating experience review P&ID piping and instrumentation diagram PRA probabilistic risk assessment RCS reactor coolant system ReSR results summary report RIHA risk-important human action RG Regulatory Guide RO reactor operator RSC remote shutdown console RSR remote shutdown room SART situation awareness rating technique S&Q staffing and qualifications SG steam generator SI safety injection SIAS safety injection actuation signal SME subject matter expert SOC sampling of operational conditions SOP system operating procedure SRO senior reactor operator SS shift supervisor STA shift technical advisor TA task analysis TAA transient and accident analysis TIHA treatment of important human actions TLX task load index TO turbine operator TS trade secret TSC technical support center V&V verification and validation VDU visual display unit KEPCO & KHNP xiii

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Page intentionally blank KEPCO & KHNP xiv

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

1. PURPOSE The purpose of the human factors verification and validation (V&V) is to comprehensively determine, through evaluations and performance tests, that the integrated system design of the Advanced Power Reactor 1400 (APR1400) conforms to accepted human factors engineering (HFE) design principles, and that the design ensures that plant personnel are able to successfully perform their tasks to achieve plant safety and power production goals.

The V&V ensures that the integrated system design:

• Consistent with the APR1400 design

• Remains within acceptable performance conditions under a broad set of operating modes and conditions

• Provides the alarms, displays and controls required to support personnel tasks as defined by the task analysis (TA) and

• Meet the guidance in the Style Guide (Reference 2)

The goals of the V&V implementation plan (IP) are to:

• Identify a sample of operating conditions that represent a broad range of events that could be encountered during operation, reflect the characteristics expected to contribute to variations in the APR1400 performance, and consider the safety significance of the HSI

• Ensure that the human-system interface (HSI) inventory and its characteristics accurately represent the HSI system per the sampling of operational conditions (SOC)

• Verify that the HSI contains the required alarms, information, controls and task support as defined by the task analysis (TA)

• Validate that the HSI design (HD) supports the safe operation of the plant through an integrated system validation (ISV)

• Identify and evaluate human engineering discrepancies (HEDs) that result from the V&V

• To achieve its purpose and goals, the V&V uses the output from the:

• Operating experience review (OER)

• TA

• Treatment of important human actions (TIHA)

• Staffing and qualification (S&Q)

• HD

• Operating procedures development and training programs The V&V IP provides a description of the methods, measures, and acceptance criteria that are required to be performed to achieve an effective V&V evaluation that fully meets NUREG-0711, Human Factors KEPCO & KHNP 1

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Engineering Program Review Model.

The results from the V&V are documented in the V&V results summary report (ReSR), which serves to demonstrate that the V&V was performed in accordance with this IP and its implementation procedures and that the HD is acceptable.

KEPCO & KHNP 2

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

2. SCOPE The HF V&V Implementation Plan applies to all HSIs in the main control room (MCR), remote shutdown room (RSR), and voice communications when HSI design can influence the MCR crews performance, between the MCR and the technical support center (TSC), emergency operation facility (EOF) and other off site emergency entities. This IP also applies the V&V process to the HSIs at the local control stations (LCSs) associated with important human actions (IHAs).

The V&V process consists of five activities:

1. Sampling of operational conditions (SOC):

• Identify conditions that are representative of the range of events that could be encountered during plant operation,

• Reflect the characteristics expected to contribute to variations in the operation performance and consider the safety significance of HSIs

2. HFE design verification:

• Determine the HSI inventory and characterizing of the HSI displays, controls, and related equipment that are within the scope defined by the selected sample of operational conditions.

• Verify through task support verification that the HSI provides the needed alarms, information, controls, and task support defined by the TA for personnel performing their tasks.

• Verify through HFE design verification that the HD conforms to the Style Guide

3. Integrated system validation (ISV):

• Validate, through human in the loop performance tests, that the integrated system design (e.g.,

hardware, software, procedures, personnel elements) supports the safe operation of the plant.

4. HED resolution:

• Evaluate HEDs to determine whether they require corrections

• Identify design solutions to address HEDs that must be corrected

• Verify the completed implementation of the HED design solutions and documents the HED in the issue tracking system (ITS)

5. ReSR

• Documents the results of the V&V KEPCO & KHNP 3

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

3. METHODOLOGY OVERVIEW TS KEPCO & KHNP 4

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Figure 3-1 ITAAC Closure Schedule 3.1. V&V Interfaces in the HFE Program TS KEPCO & KHNP 5

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

4. IMPLEMENTATION 4.1. Sampling of Operational Conditions TS 4.1.1. Sampling Dimensions TS KEPCO & KHNP 6

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.1.1.1. Plant Conditions TS KEPCO & KHNP 7

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.1.1.2. Personnel Tasks TS KEPCO & KHNP 8

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.1.1.3. Situational Factors that are known to Challenge Human Performance TS KEPCO & KHNP 9

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 10

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Table 4-1 SOC Dimension Matrix Example (Plant Condition)

KEPCO & KHNP 11

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Table 4-2 SOC Dimension Matrix Example (Types of Personnel Tasks)

TS KEPCO & KHNP 12

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 13

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Table 4-3 SOC Dimension Matrix Example (Situational Factors or Error forcing Contexts) TS KEPCO & KHNP 14

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 4.1.2. Identification of Scenarios TS 4.1.3. Scenario Design Requirements TS KEPCO & KHNP 15

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.1.4. Scenario Definition TS KEPCO & KHNP 16

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.1.5. Scenario Components TS KEPCO & KHNP 17

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.2. Design Verification TS 4.2.1. Human-system interface Inventory and Characterization TS 4.2.1.1. Scope TS 4.2.1.2. HSI Characterization TS KEPCO & KHNP 18

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 4.2.1.3. Inventory Verification TS KEPCO & KHNP 19

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.3. Task Support Verification TS 4.3.1. Verification Criteria TS 4.3.2. General Methodology TS KEPCO & KHNP 20

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Figure 4-1 Task Support Verification Process KEPCO & KHNP 21

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.3.3. HED Identification TS KEPCO & KHNP 22

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.4. HFE Design Verification TS 4.4.1. Verification Criteria TS KEPCO & KHNP 23

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 24

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 4.4.2. General Methodology TS Figure 4-2 Design Verification Process KEPCO & KHNP 25

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 26

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.4.3. HED Identification TS KEPCO & KHNP 27

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5. Integrated System Validation TS KEPCO & KHNP 28

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Figure 4-3 Integrated System Validation Process KEPCO & KHNP 29

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 4.5.1. Validation Team TS KEPCO & KHNP 30

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.2. Test Objectives TS KEPCO & KHNP 31

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 4.5.3. Validation Testbeds TS 4.5.4. Plant Personnel (Participants)

TS KEPCO & KHNP 32

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 33

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Table 4-4 Operator Experience Record TS KEPCO & KHNP 34

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 4.5.5. Performance Measurement TS KEPCO & KHNP 35

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 36

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.5.1. Types of Performance Measures Used TS KEPCO & KHNP 37

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 38

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 39

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 40

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 41

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.5.2. Performance Measure Information and Validation Criteria TS KEPCO & KHNP 42

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Table 4-5 Basis for Performance Criteria TS TS KEPCO & KHNP 43

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 44

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 45

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 46

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Table 4-6 Summary of Performance Measures TS KEPCO & KHNP 47

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.6. Test Design TS 4.5.6.1. Scenario Sequencing TS KEPCO & KHNP 48

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Table 4-7 ISV Scenario Sequence Example TS KEPCO & KHNP 49

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 4.5.6.2. Test Procedures TS 4.5.6.2.1. Test Procedure Requirements TS KEPCO & KHNP 50

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.6.3. Test Personnel Training TS KEPCO & KHNP 51

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.6.4. Participant Training TS 4.5.6.5. Pilot Testing TS KEPCO & KHNP 52

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.7. Data Analysis and HED Identification TS 4.5.7.1. Individual Performance Measures TS KEPCO & KHNP 53

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 54

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Table 4-8 Example of BARS T-test TS TS Table 4-9 Example of Expert Correlation Analysis TS TS KEPCO & KHNP 55

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Table 4-10 Example of T-test Result Form for the Secondary Task TS TS Table 4-11 Example of T-test Result Form for the Situation Awareness TS KEPCO & KHNP 56

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS TS Table 4-12 Example of T-test Result Form for the Workload TS KEPCO & KHNP 57

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Table 4-13 Example of T-test for Anthropometric and Physiological Factors TS 4.5.7.2. Individual Scenarios and Assumptions of IHAs TS KEPCO & KHNP 58

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Table 4-14 Example of Acceptance Criteria for an Individual Scenario including IHA TS TS KEPCO & KHNP 59

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.7.3. Extent of the Issue Determination TS 4.5.7.4. Margin of Error TS KEPCO & KHNP 60

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.8. Validation Conclusions TS KEPCO & KHNP 61

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS Table 4-15 Example of Acceptance Criteria for ISV TS TS KEPCO & KHNP 62

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.9. Human Engineering Discrepancy Resolution Review Criteria TS 4.5.9.1. Human Engineering Discrepancy Analysis TS 4.5.9.2. Selection of HEDs to Correct TS KEPCO & KHNP 63

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS 4.5.9.3. HED Analysis and Development of Design Solutions TS 4.5.9.4. Design Solution Evaluation TS 4.5.9.5. HED Evaluation Documentation TS KEPCO & KHNP 64

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 TS KEPCO & KHNP 65

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

5. IMPLEMENTATION TEAM TS Table 5-1 Implementation Team Expertise TS KEPCO & KHNP 66

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

6. RESULTS

SUMMARY

REPORT TS KEPCO & KHNP 67

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

7. REFERENCES

1. NUREG-0711, Human Factors Engineering Program Review Model," Rev. 3, U.S. Nuclear Regulatory Commission, November 2012.

2. APR1400-E-I-NR-14012-P, Style Guide, Rev.2, KHNP, January 2018.

3. APR1400-E-I-NR-14001-P, Human Factors Engineering Program Plan, Rev.2, KHNP, January 2018.

4. APR1400-E-I-NR-14010-P, Human Factors Verification and Validation Scenarios, Rev.2, KHNP, January 2018.

5. NUREG-0800, Rev. 2, "Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants: LWR Edition, 2007.

6. APR1400-E-I-NR-14006-P, Treatment of Important Human Actions Implementation Plan, Rev.2, KHNP, January 2018.

7. APR1400-E-I-NR-14002-P, Operating Experience Review Implementation Plan, Rev.2, KHNP, January 2018.

8. Regulatory Guide 1.33, Quality Assurance Program Requirements (Operation), Rev. 2, U.S.

Nuclear Regulatory Commission, 1978.

9. 10 CFR 50.34(f), "Additional TMI-related requirements," U.S. Nuclear Regulatory Commission.

10. NUREG-1342, "A Status Report Regarding Industry Implementation of Safety Parameter Display Systems," U.S. Nuclear Regulatory Commission, 1989.

11. Regulatory Guide 1.97, "Criteria for Accident Monitoring Instrumentation for Nuclear Power Plants," Rev. 4, U.S. Nuclear Regulatory Commission, 2006.

12. NUREG-0700, Rev. 2, Human-System Interface Design Review Guidelines, U.S. Nuclear Regulatory Commission, May 2002.

13. ANSI/ANS-3.5-2009, "Nuclear Power Plant Simulators for Use in Operator Training and Examination, ANSI and ANS.

14. Endsley, M.R., Towards a Theory of Situation Awareness in Dynamic Systems, Human Factors, Vol. 37, pp. 32-64, 1995a.

15. Paul Salmon, Neville Stanton, Guy Walker & Damian Green, Situation Awareness Measurement: A review of applicability for C4i environments. Journal of Applied Ergonomics, 37, 2, pp. 225-238, 2007.

16. Sandra G. Hart, Lowell E. Staveland, Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, NASA/CR-1997-205754, 1988.

17. Hill, S.G., Iavecchia, H.P., Byers, J.C., Bittier, A.C., Zaklad, A.L., & Christ, R.E., Comparison of Four Subjective Workload Rating Scales. Human Factors, 34, pp.429-440, 1992.

18. Sebok, A., Team Performance in Process Control: Influences of Interface Design and Staffing Level, Ergonomics, Vol. 43, pp. 1210-1236, 1988.

19. Endsley, M.R., Measurement of Situation Awareness in Dynamic Systems, Human Factors, Vol.

37, pp. 65-84, 1995b.

20. Nygren, T.E. (1991), Psychometric Properties of Subjective Workload Measurement Techniques:

Implication For Their Use in The Assessment Of Perceived Mental Workload, Human Factors, Vol. 33, pp 17-33.

21. APR1400-K-I-NR-14009-P Design Implementation Plan, Rev. 1, KHNP, March 2017.

KEPCO & KHNP 68

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

8. DEFINITIONS

1. Bias Bias is an aspect of an evaluation methodology that systematically modifies performance or its interpretation.

2. Important Human Actions (IHAs)

Important human actions consist of those actions that meet either risk or deterministic criteria in terms of plant safety.

• Risk-important human actions: Actions defined by risk criteria that plant personnel use to assure plant safety. There are absolute and relative criteria for defining risk important human actions.

For absolute criteria, a risk-important human action (RIHAs) is any action whose successful performance is needed to reasonably assure that predefined risk criteria are met. For relative criteria, the risk-important actions are defined as those with the greatest risk compared to all human actions. Risk-important human actions based on relative criteria can be identified quantitatively from risk analyses, and qualitatively from various criteria, such as an evaluation of the task performance based on considering performance-shaping factors.

• Deterministically identified important human actions: Deterministic engineering analyses typically are completed and described as part of the APR1400 DCD. Such analyses include instrumentation and controls (I&C) as described in Chapter 7 and transient and accident analyses, as described in Chapter 15. These deterministic analyses also credit human actions.

3. Human-System Interface (HSI)

A human system interface (HSI) is that part of the system through which personnel interact to perform their functions and tasks. Major HSIs include alarms, information displays, controls, and procedures. HSI use can be influenced directly by factors such as (1) the organization of HSIs into workstations (e.g., consoles and panels); (2) the arrangement of workstations and supporting equipment into facilities, such as a main control room, remote shutdown station, local control stations, the technical support center (TSC), and the emergency operation facility (EOF);

and (3) the environmental conditions in which the HSIs are used, including temperature, humidity, ventilation, illumination, and noise. The use of HSIs also can be affected indirectly by other aspects of plant design and operation, such as personnel training, shift schedules, work practices, and management and organizational factors, such as the safety culture of the plant staff.

4. Plant The operating unit of a nuclear power station, including the nuclear steam supply system, the turbine, electrical generator, and all associated systems and components (commonly referred to as the balance of plant). For a multi-unit plant, the term plant refers to all systems and processes associated with a specific units ability to produce electrical power, even though other units might share some systems or portions of systems.

5. Primary Tasks Primary Tasks are those tasks performed by personnel to supervise the plant (e.g., monitoring, detection, situation assessment, response planning, and response implementation).

KEPCO & KHNP 69

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

6. Operating Procedures Written instructions providing guidance to plant personnel for operating and maintaining the plant, and for handling disturbances and emergency conditions.

7. Secondary Tasks Secondary tasks are those tasks personnel must complete when interfacing with the HSI, such as navigation through computer screens to find a needed display and HSI configuration.

Complicated secondary tasks often have negative effects on the performance of primary tasks.

8. Simulator A facility that physically represents the HSI configuration and that dynamically represents the operating characteristics and responses of the plant in real time.

KEPCO & KHNP 70

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 APPENDIX A NUREG-0711 REVIEW CRITERIA CONFORMANCE TABLE NUREG-0711 Rev. 3 Criteria IP Section(s) 11.4.1 Sampling of Operational Conditions 4.1 Sampling of As stated in Section 11.2, the objective of the Sampling of Operational Operational Conditions review is to verify that the applicant identified a sample of Conditions operational conditions that (1) includes conditions representative of the range of events that could be Table 4-1, 4-2, 4-3 encountered during the plants operation, (2) reflects the characteristics expected to contribute to variations in the systems performance, and (3) considers the safety significance of HSIs.

These sample characteristics are best identified by using a multidimensional sampling strategy to reasonably assure that V&V evaluations include variation along important dimensions.

The sampling methodology will identify a range of operational conditions to guide Task Support Verification, HFE Design Verification, and ISV. The NRCs review of this activity considers the dimensions to be used to identify and select conditions, and their integration into scenarios.

11.4.1.1 Sampling Dimensions 4.1.1 Sampling The following sampling dimensions are addressed below: Plant conditions, Dimensions personnel tasks, and situational factors known to challenge personnel performance.

(1) The applicant should include the following plant conditions:

• normal operational events including plant startup, shutdown or refueling, and significant changes in operating power

• I&C and HSI failures and degraded conditions that encompass:

- The I&C system, including the sensor, monitoring, automation and control, and communications subsystems; [e.g., safety-related system logic and control unit, fault tolerant controller, local "field unit" for multiplexer (MUX) system, MUX controller, and a break in MUX line] - common cause failure of the I&C system during a design basis accident (as defined by BTP 7-19)

- HSIs including, loss of processing or display capabilities for alarms, displays, controls, and computer-based procedures

• transients and accidents, such as:

- transients (e.g., turbine trip, loss of off-site power, station blackout, loss of all feedwater, loss of service water, loss of power to selected buses or MCR power supplies, and safety and relief valve transients)

- accidents (e.g., main-steam-line break, positive reactivity addition, control rod insertion at power, anticipated transient without scram, and various-sized loss-of coolant accidents)

- reactor shutdown and cooldown using the remote shutdown system

- reasonable, risk-significant, beyond-design-basis events that should be determined from the plant-specific PRA (2) The applicant should include the following types of personnel tasks:

• Important HAs, Systems, and Accident Sequences - The sample 4.1.1(B) IHAs, should include all important HAs, as determined in Section 7. Additional Systems and factors that contribute highly to risk, as defined by the PRA, also should Accident be sampled: Sequences

- dominant accident sequences

- dominant systems (selected through PRA importance measures, such as Risk Achievement Worth or Risk Reduction Worth

• Manual Initiation of Protective Actions - The sample should include KEPCO & KHNP A1

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) manual system level actuation of critical safety functions.

• Automatic System Monitoring - The sample should include situations in which humans must monitor a risk-important automatic system.

• OER-Identified Problematic Tasks - The sample should include all personnel tasks identified as problematic during the applicant's review of operating experience.

• Range of Procedure Guided Tasks -The sample should include tasks that are well defined by procedures. Personnel should be able to understand and execute the specified steps as part of their rule-based decision-making. Regulatory Guide 1.33, Appendix A, contains several categories of "typical safety-related activities that should be covered by written procedures." The sample should include appropriate procedures in each category:

- administrative procedures

- general plant operating procedures

- procedures for startup, operation, and shutdown of safety-related systems

- procedures for abnormal, off-normal, and alarm conditions

- procedures for combating emergencies and other significant events (e.g., reactor accidents, and declaration of emergency-action levels)

- procedures for controlling radioactivity

- procedures for controlling measuring and test equipment and for surveillance tests, procedures, and calibration

- procedures for performing maintenance

- chemistry and radiochemical control procedures

• Range of Knowledge-Based Tasks - The sample should include tasks that are not well defined by detailed procedures.

• Range of Human Cognitive Activities - The sample should include the range of cognitive activities that personnel perform, including:

- detecting and monitoring (e.g., of critical safety-function threats)

- situation assessment (e.g., interpreting alarms and displays to diagnose faults in plant processes and in automated control and safety systems)

- planning responses (e.g., evaluating alternatives to recover from plant failures)

- response implementation (e.g., in-the-loop control of plant systems, assuming manual control from automatic control systems, and carrying out complicated control actions)

- obtaining feedback (e.g., feedback of the success of actions taken)

• Range of Human Interactions - The sample should include the range of interactions among plant personnel, including tasks performed independently by individual crew members, and those undertaken by a team of crew members. These interactions among plant personnel should include interactions between:

- main control room operators (e.g., operations, shift turnover walkdowns)

- main control room operators with auxiliary operators and other plant personnel performing tasks locally (e.g., maintenance or I&C technicians, chemistry technicians)

- main control room operators and the TSC and the EOF

- main control room operators with plant management, the NRC, and other outside organizations (3) The applicant should include the following situational factors or error-KEPCO & KHNP A2

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) forcing contexts known to challenge human performance. It also should include situations specifically designed to create human errors to assess the systems error tolerance, and the ability of personnel to recover from any errors, should these occur, for example:

• High-Workload Situations - The sample should include situations where variations in human performance due to high workload and multitasking situations can be assessed.

• Varying-Workload Situations - The sample should include situations wherein variations in human performance due to workload transitions can be determined.

These include conditions where there is (1) a sudden increase in the number of signals that must be detected and processed after a period in which signals were infrequent, and (2) a rapid reduction in the need for detecting signals and processing demands following a time of high sustained task-demand.

• Fatigue Situations - To the extent possible, the sample should include situations that may be associated with fatigue, such as work on backshifts and tasks performed frequently with repetitive actions, such as repeated inputs to a touch screen during plant operations or pulling rods.

• Environmental Factors - To the extent possible, the sample should include environmental conditions that may cause human performance to vary, e.g., poor lighting, extreme temperatures, high noise, and simulated radiological contamination.

11.4.1.2 Identification of Scenarios 4.1.2 Identification (1) The applicant should combine the results of the sampling to identify a set of Scenarios of V&V scenarios to guide subsequent analyses. Table 4-1, 4-2, 4-3 (2) The applicant should not bias the scenarios by overly representing the following: 4.1.2 Identification

• scenarios for which only positive outcomes are expected of Scenarios,

• scenarios that, for ISV, are relatively easy to conduct (e.g., scenarios Scenario Design should not be avoided simply because they are demanding to set up Requirements and run on a simulator)

• scenarios that, for ISV, are familiar and well structured (e.g., which address familiar systems and failure modes that are highly compatible with plant procedures, such as textbook design-basis accidents) 11.4.1.3 Scenario Definition 4.1.3 Scenario (1) The applicant should identify operational conditions and scenarios to be Definition used for HSI Task Support Verification, Design Verification, and ISV. The applicant should develop detailed scenarios suitable for use on a full- Reference 4 scope simulator. The level of detail should be comparable to what one would include in a test plan. For each one, the following information should 4.1.4 Scenario be defined to reasonably assure that important dimensions of performance Components are addressed, and to allow the scenarios to be accurately and consistently presented for repeated trials:

• a description of the scenario and any pertinent prior history necessary for personnel to understand the state of the plant at the start-up of the scenario

• specific initial conditions (a precise definition of the plants functions, processes, systems, component conditions, and performance parameters, e.g., similar to that at shift turnover)

• events (e.g., failures) that will occur during the scenario and their initiating conditions, e.g., based on time, or a value of a specific KEPCO & KHNP A3

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) parameter

• precise definition of workplace factors, (e.g., environmental conditions, such as low levels of illumination)

• needs for task support (e.g., procedures and technical specifications)

• staffing level

• details of communication content between control room personnel and remote personnel (e.g., load dispatcher via telephone)

• scripted responses for test personnel who will act as plant personnel in the test scenarios

• the precise specification of what, when, and how data are to be collected and stored (including videotaping, questionnaires, and rating-scale administrations)

• precise specifications on simulator set up

• specific criteria for terminating the scenario (2) The applicants scenarios should realistically replicate operator tasks in the tests; then, the findings from the test can be generalized to the plants actual operations.

(3) When the applicants scenarios include work associated with operations remote from the main control room, the effects on personnel performance due to potentially harsh environments (e.g., high radiation) should be realistically simulated (e.g., additional time to don protective clothing, and access radiologically controlled areas).

11.4.2 Design Verification Review Criteria 4.2 Design 11.4.2.1 HSI Inventory and Characterization Verification As stated in Section 11.2, the objective of the review is to verify that the 4.2.1 HSI Inventory applicant's HSI inventory and characterization accurately describes all HSI and displays, controls, and related equipment lying within the scope defined by the Characterization sampling of operational conditions.

Applicants may document their HSI inventory in different ways. They should describe the means by which this is done, and provide it to the NRCs staff for review using the criteria in this section.

(1) Scope - The applicant should develop an inventory of all HSIs that 4.2.1.1 Scope personnel require to complete the tasks covered in the validation scenarios that were identified by the applicants Sampling of Operational Conditions.

The inventory should include aspects of the HSI used for managing the interface, such as navigation and retrieving displays, as well as those that control the plant.

(2) HSI Characterization - The applicants inventory should describe the 4.2.1.2 HSI characteristics of each HSI within the scope of the verification. The Characterization following is a minimal set of information for this characterization:

• a unique identification code number or name

• associated plant system and subsystem

• associated personnel functions and tasks

• type of HSI, e.g.,

- computer-based control (e.g., touch screen or cursor-operated button and keyboard input)

- hardwired control (e.g., J-handle controller, button, and automatic controller)

- computer-based display (e.g., digital value and analog representation)

- hardwired display (e.g., dial, gauge, and strip-chart recorder)

• display characteristics and functionality [e.g., plant variables/parameters, units of measure, accuracy of variable/

KEPCO & KHNP A4

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) parameter, precision of display, dynamic response, and display format (e.g., bar chart or trend plot)]

• control characteristics and functionality [e.g., continuous versus discrete settings, number and type of control modes, accuracy, precision, dynamic response, and control format (method of input)]

• user-system interaction and dialog types (e.g., navigation aids and menus)

• location in data-management system (e.g., identification code for information display screen)

• physical location in the HSI (e.g., control panel section), if applicable The applicant should include photographs, copies of display screens, or similar samples of HSIs in the HSI inventory and characterization.

(3) Inventory Verification - The applicant should verify the inventory description of HSIs to ensure that it accurately reflects their current state.

11.4.2.2 HSI Task Support Verification 4.2.1.3 Inventory HSI Task Support Verification addresses the availability of items needed to Verification support task requirements. As stated in Section 11.2, the objective of the HSI Task Support Verification review is to ensure that the applicant verified that the 4.3 Task Support HSI provides the needed alarms, information, controls, and task support for Verification personnel to perform their tasks, defined by the task analysis.

(1) Verification Criteria - The applicant should base the HSI task support 4.3.1 Verification criteria on the alarms, controls, displays, and task support needed by Criteria personnel to complete their tasks as identified by the applicants task analysis.

(2) General Methodology - The applicant should compare the HSIs and their 4.3.2 General characteristics (as defined in the HSI inventory and characterization) to the Methodology Steps needs of personnel identified in the task analysis for the defined sampling 1, 2, 3, 4, 5 of operational conditions, noted in Section 11.4.1.

(3) HED Identification - The applicant should identify and document an HED 4.3.3 HED when: Identification

• An HSI needed for task performance (e.g., a necessary control or 4.5.9 HED display) is unavailable. Resolution Review

• HSI characteristics do not match the requirements of the personnel task Criteria (e.g., a display may show the needed plant parameter but not within the range or precision needed for the task).

• HSIs are available that are not needed for any task.

(4) HED Documentation - The applicant should document HEDs to identify 4.5.9.5 HED the HSI, the tasks affected, and the basis for the deficiency (what aspect of Evaluation the HSI was identified as not meeting task requirements). Documentation 11.4.2.3 HFE Design Verification 4.4 HFE Design HFE Design Verification addresses the suitability of the HSI with regard to Verification human capabilities and limitations. As stated in Section 11.2, the objective of the HFE Design Verification review is to evaluate the applicants verification that the design of the HSIs conforms to HFE guidelines.

(1) Verification Criteria - The applicant should base the criteria used for HFE 4.4.1 Verification Design Verification on HFE guidelines. Criteria (2) General Methodology - The applicants HFE Design Verification 4.4.2 General methodology should include the following: Methodology Steps

• Procedures for comparing the characteristics of the HSIs with HFE 1, 2, 3, 4 5 guidelines for:

(1) the defined sampling of operational conditions, as noted in Section 11.4.1, and (2) the general environment in which HSIs are sited, including workstations, KEPCO & KHNP A5

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) control rooms, and environmental characteristics (e.g., lighting and noise).

• Procedures for determining for each guideline whether the HSI is "acceptable" or "discrepant." If discrepant, it should be designated as an HED, tracked, and evaluated (see Sections 2.4.4 and 11.4.4).

• Procedures for evaluating whether an HED is a potential indicator of additional issues.

(3) HED Identification - The applicant should identify an HED when a 4.4.3 HED characteristic of the HSI is "discrepant" from a guideline. Identification (4) HED Documentation - The applicant should document HEDs in terms of 4.5.9.5 HED the HSI involved, and how its characteristics depart from a particular Evaluation guideline. Documentation (5) Additional Considerations for Reviewing the HFE Aspects of Plant Modifications - In addition to any of the criteria above that relate to the modification being reviewed, the applicant should address the following considerations:

• The scope of HFE design verification may be restricted to the modified HSIs and their interactions with the rest of the HSIs.

• When both old and new versions of similar HSIs are available, this verification should offer reasonable assurance that their means of presentation and methods of operation are compatible, such that personnel performance will not be impaired when alternating the use of each one.

• HEDs should be identified for the following:

- failure to meet "personnel-identified" functionality in addition to that specified by system designers. When a digital system replaces an existing system, it is important to ensure that all operational uses of the former system were addressed, even those that were not intended in the original design. The replacement system's design should consider the ways in which personnel actually used the former system

- poor integration with the rest of the HSI

- poor integration with procedures and training

• Temporary configurations of the HSIs and plant systems that operations and maintenance personnel may use when the plant is not shutdown, should be reviewed to verify that their design is consistent with the principles of good HFE design, including consistency with the rest of the HSIs.

11.4.3 Integrated System Validation 4.5 Integrated As stated in Section 11.2, the objective of the ISV review is to verify that the System Validation applicant validated, using performance-based tests, that the integrated system design (e.g., hardware, software, procedures and personnel elements) supports the safe operation of the plant.

The scenarios for ISV should be performed using a simulator, or other suitable representation of the system, to determine the complete designs adequacy to support safe operations.

Validation should be performed after the resolution of all significant HEDs identified in verification reviews.

Applicants submitting an ISV IP for staff review should follow the general guidance in Section 1.2.2. The IP should describe the methodology of the tests that will be performed. It should identify the specific scenarios to be used, and detail them at a level that will support the staffs review, using the criteria stated in this section. The level of scenario detail should be comparable to that in a test plan. For each scenario, the applicant should specify the specific KEPCO & KHNP A6

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) performance measures used for pass/fail along with the criteria for diagnostic evaluations to be used in assessing the results. The NRC will not accept submittals that merely provide a plan for developing the detailed ISV methodology.

11.4.3.1 Validation Team (1) The applicant should describe how the team performing the validation has independence from the personnel responsible for the actual design.

11.4.3.2 Test Objectives (1) The applicant should develop detailed test objectives to provide evidence that the integrated system adequately supports plant personnel in safely operating the plant, to include the following considerations:

• Validate the acceptability of the shift staffing level(s), the assignment of tasks to crew members, and crew coordination within the control room, between the control room and local control stations and support centers, and with individuals performing tasks locally. This should encompass validating minimum shift staffing levels, nominal levels, maximum levels, and shift turnover (see Section 6 for definitions).

• Validate that the design has adequate capability for alerting, informing controlling, and feedback such that personnel tasks are successfully completed during normal plant evolutions, transients, design-basis accidents, and also under selected, risk significant events beyond-design basis, as defined by sampling operational conditions.

• Validate that specific personnel tasks can be accomplished within the time and performance criteria, with effective situational awareness, and acceptable workload levels that balance vigilance and personnel burden.

• Validate that the HSIs minimize personnel error and assure error detection and recovery capability when errors occur.

• Validate the assumptions about performance on important HAs.

• Validate that the personnel can effectively transition between the HSIs and procedures in accomplishing their tasks, and that interface management tasks, such as display configuration and navigation, are not a distraction or an undue burden.

11.4.3.3 Validation Testbeds 4.5.3 Validation A testbed is the HSI representation used to perform validation evaluations. One Testbed approach an applicant can use to acceptably meet criteria 1 through 7 in this 3 Methodology section is to use a testbed that is compliant with "Nuclear Power Plant Overview Simulators for Use in Operator Training" (ANS, 2009).

(1) Interface Completeness - The applicants testbed should represent completely the integrated system. It should include HSIs and procedures 4.1 Sampling of not specifically required in the test scenarios. Operational (2) Interface Physical Fidelity - The testbeds HSIs and procedures should be Conditions represented with high physical fidelity to the reference design, including the presentation of alarms, displays, controls, job aids, procedures, communications equipment, interface management tools, layout, and spatial relationships.

(3) Interface Functional Fidelity - The testbeds HSI and procedure 4.5.1 Validation functionality should be represented with high fidelity to the reference Team design. All HSI functions should be available.

(4) Environmental Fidelity - The testbeds environmental fidelity should be 4.5.2 Test represented with high physical fidelity to the reference design, including Objectives the expected levels of lighting, noise, temperature, and humidity. Thus, for 1 Purpose example, the noise contributed by equipment, such as air-handling units, KEPCO & KHNP A7

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) computers, and communications equipment should be represented in validation tests.

(5) Data Completeness Fidelity - Information and data provided to personnel should completely represent the plants systems they monitor and control.

(6) Data Content Fidelity - The testbeds data content fidelity should be 4.5.3 Validation represented with high physical fidelity to the reference design. The Testbeds presentation of information and controls should rest on an underlying model accurately mirroring the reference plant. The model should provide input to the HSI such that the information accurately matches that which is presented during operations.

(7) Data Dynamics Fidelity - The testbeds data dynamics fidelity should be represented with high fidelity to the reference design. The process model should be able to provide input to the HSI so that information flow and control responses occur accurately and within the correct response time; e.g., information should be sent to personnel with the same delays as occur in the plant.

(8) For important HAs at complex HSIs remote from the main control room (e.

g., a remote shutdown facility), where timely, precise actions are essential, the use of a simulator or mockup should be considered to verify that the requirements for human performance can be met. (For less important HAs, or for non complex HSIs, human performance may be assessed on analysis, such as task analysis, rather than on simulations.)

(9) The applicant should verify the conformance of the testbed to the testbed-required characteristics before validation tests are conducted.

11.4.3.4 Plant Personnel 4.5.4 Plant (1) Participants in the applicants validation tests should be representative of Personnel plant personnel who will interact with the HSI (e.g., licensed operators, (Participants) rather than training personnel or engineers).

(2) To properly account for human variability, the applicant should use a sample of participants that reflects the characteristics of the population from which it is drawn. Those characteristics expected to contribute to variations in system performance should be specifically identified; the sampling process should reasonably assure that the validation encompasses variation along that dimension. Determining Table 4-4 representativeness should include considering the participants license type and qualifications, skill/experience, age, and general demographics.

(3) In selecting personnel for participating in the tests, the applicant should consider the minimum shift staffing levels, nominal levels, and maximum levels, including shift supervisors, reactor operators, shift technical advisors, etc.

(4) The applicant should prevent bias in the sample of participants by avoiding the use of participants who:

• are members of the design organization

• participated in prior evaluations

• were selected for some specific characteristic, such as crews identified as good performers or more experienced 11.4.3.5 Performance Measurement 4.5.5 Performance ISV employs a hierarchal set of performance measures including measures of Measures plant performance, personnel task performance, situation awareness, cognitive workload, and anthropometric/physiological factors. Errors of omission and commission also are identified. A hierarchal set of measures provides sufficient information to validate the integrated system design and affords a basis to KEPCO & KHNP A8

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) evaluate deficiencies in performance and thereby identify needed improvements. Pass/fail measures are those used to determine whether the design is or is not validated. Diagnostic measures are used to better understand personnel performance and to facilitate the analyses of errors and HEDs.

11.4.3.5.1 Types of Performance Measures 4.5.5.1 Types of (1) The applicant should identify the specific plant performance measures Performance applicable to each ISV scenario. Measures used (2) The applicant should identify the primary task measures applicable to each ISV scenario.

• For each scenario, the applicant should identify the primary tasks operators must perform to accomplish scenario goals, so that such measures can be developed.

• The measures chosen to evaluate personnel task performance should reflect those aspects of the task that are important to system performance, such as:

- time

- accuracy

- frequency

- amount achieved or accomplished

- consumption or quantity used

- subjective reports of participants

- behavior categorization by observers

• The analysis of primary tasks will support the identification of errors of omission (primary tasks not performed). Also, any actions and tasks that operators actually perform that deviate from the primary tasks should be identified and noted. These actions should be used to identify errors of commission.

(3) The applicant should identify the secondary task measures applicable to each scenario.

(4) The applicant should identify the measures of situation awareness applicable to each scenario (5) The applicant should identify the workload measures obtained for each scenario.

(6) The applicant should identify the anthropometric and physiological measures obtained for each scenario.

11.4.3.5.2 Performance Measure Information and Validation Criteria 4.5.5.2 (1) The applicant should describe the methods by which these measures are Performance obtained, e.g., by simulator data recording, participant questionnaires, or Measure observation by subject-matter experts. Information and (2) The applicant should specify when each measure is obtained (recorded), Validation Criteria such as continuously, at specific points during the scenario, or after the scenario ends.

(3) The applicant should describe the characteristics (see Table 11-1) of the performance measures.

Table 11-1 Characteristics of Performance Measures Table 4-6 Construct Validity A measure should represent accurately the aspect of performance it is intended to measure.

Reliability A measure should be repeatable; e.g., same behavior measured in exactly the same way under identical circumstances should yield the same results.

Sensitivity A measure's range (scale) and its frequency (how often data are collected) should be appropriate to that aspect of performance being assessed.

KEPCO & KHNP A9

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s)

Unobtrusiveness A measure should minimally alter the psychological or physical processes that are being investigated.

Objectivity A measure should be based on easily observed phenomena.

(4) The applicant should identify the specific criterion for each measure used to judge the acceptability of performance and describe its basis.

Table 11-2 Basis for Performance Criteria Requirement The observed performance of the integrated system is compared with a quantified performance requirement; e.g., the requirements for the performance of systems, subsystems, and personnel are defined through engineering analyses.

Benchmark The observed performance of the integrated system is compared with a criterion established using a benchmark system, e.g., a current system is predefined as acceptable.

Norm The observed performance of the integrated system is compared with a criterion using many predecessor systems (rather than a single benchmark system).

Expert Judgment The observed performance of the integrated system is compared with a criterion established by subject-matter experts.

(5) The applicant should identify whether each measure is a pass/fail one or a diagnostic one.

11.4.3.6 Test Design 4.5.6 Test Design The criteria in this section are divided into the following subsections:

11.4.3.6.1 Scenario Sequencing 4.5.6.1 Scenario (1) The applicant should balance scenarios across crews to provide each Sequencing crew with a similar, representative range of scenarios. Table 4-7 (2) The applicant should balance the order of presentation of scenarios to crews to provide reasonable assurance that the scenarios are not always presented in the same sequence (e.g., the easy scenario is not always used first).

11.4.3.6.2 Test Procedures 4.5.6.2 Test (1) The applicant should use detailed, unambiguous procedures to govern the Procedures conduct of the tests. These procedures should include the following: 4.5.6.2.1 Test

• the identification of which crews receive which scenarios, and the order Procedure in which they should be presented Requirements

• detailed and standardized instructions for briefing the participants

• specific directions for the testing personnel on conducting the test scenarios, as elaborated in Scenario Definition (Section 11.4.1.3)

• guidance on when and how to interact with participants when difficulties occur in simulation or testing Appendix B

• instructions on when and how to collect and store data. These Appendix C instructions should stipulate which data are to be recorded by:

- simulator computers 4.5.6.2.1 Test

- special-purpose instruments and devices for collecting data (such as Procedure situation awareness-and workload-questionnaires, or physiological Requirements measures)

- video recorders (locations and views) 4.5.6.2 Test

- test personnel and subject-matter experts (such as via observational Procedures checklists)

• procedures for documentation:

- identifying and maintaining files of test records including details of the crew and scenarios See Reference 4

- data collected

- logs created by those who conducted the tests KEPCO & KHNP A10

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s)

The procedures should detail the types of information that should be 5 Implementation logged (e.g., when the tests were performed, deviations from the test Team procedures and why they occurred, and any unusual events that may be important to understanding how a test was run or for interpreting the findings from it). The procedure also should state when the types of information should be recorded.

(2) The applicants test procedures should minimize the opportunity for bias in the test personnels expectations and in the participants responses.

11.4.3.6.3 Training Test Personnel 4.5.6.3 Test (1) The applicant should train test personnel (those who conduct or administer Personnel Training the validation tests) on the following:

• the use and importance of test procedures

• bias and errors that test personnel may introduce into the data through failures to follow test procedures accurately or to interact with participants properly

• the importance of accurately documenting problems arising during testing, even if they were due to an oversight or error of those conducting the test 11.4.3.6.4 Training Participants (1) The applicants training of participants should be very similar to the training plant personnel receive. It should reasonably assure that the participants knowledge of the plants design, and operations, and the use of the HSIs and procedures represents that of experienced plant personnel.

Participants should not be trained specifically to carry out the selected validation scenarios.

(2) To assure that the participants performance is representative of plant personnel, the applicants training of participants should result in near asymptotic performance (e.g., stable, not significantly changing from trial to trial) and should be tested for such before conducting the validation.

11.4.3.6.5 Pilot Testing 4.5.6.5 Pilot (1) The applicant should conduct a pilot study before the validation tests begin Testing to offer an opportunity for the applicant to assess the adequacy of the test design, performance measures, and data-collection methods.

(2) The applicant should not use participants in the pilot testing who will then be participants in the validation tests.

11.4.3.7 Data Analysis and HED Identification 4.5.7 Data Analysis (1) The applicant should use a combination of quantitative and qualitative and HED methods to analyze data. The analysis should reveal the relationship Identification between the observed performance and the established performance criteria.

(2) The applicant should discuss the method by which data is analyzed across trials, and include the criteria used to determine successful performance for a given scenario.

(3) The applicant should evaluate the degree of convergence between related measures (e.g., consistency between measures expected to assess the same aspect of performance).

(4) When interpreting test results, the applicant should allow a margin of error to reflect the fact that actual performance may be slightly more variable than observed validation-test performance.

(5) The applicant should verify the correctness of the analyses of the data.

This verification should be done by individuals or groups other than those who performed the original analysis, but may be from the same organization.

KEPCO & KHNP A11

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s)

(6) The applicant should identify HEDs when the observed performance does not meet the performance criteria.

(7) The applicant should resolve HEDs identified by pass/fail measures before the design is accepted.

11.4.3.8 Validation Conclusions (1) The applicant should document the statistical and logical bases for determining that performance of the integrated system is, and will be acceptable.

(2) The applicant should document the limitations in the validation tests, their possible effects on the conclusions of the validation, and their impact on implementing the design.

11.4.4 Human Engineering Discrepancy Resolution Review Criteria 4.5.9 HED HEDs are identified in the V&V process during: Resolution Review

• Task Support Verification (Section 11.4.2.2, criterion 3) Criteria

• HFE Design Verification (Section 11.4.2.3, criterion 3 and criterion 5 for plant modifications)

• ISV (Section 11.4.3.5.2, criterion 4)

As stated in Section 11.2, the objectives of the NRC staffs review is to verify that the applicant has (1) evaluated HEDs to determine if they require correction, (2) identified design solutions to address HEDs that must be corrected, and (3) verified the completed implementation of these HED design solutions. The applicants resolution of HEDs is reviewed by the NRC staff using the guidance is this section. HED Resolution can be performed iteratively throughout V&V. Thus, issues identified during one V&V activity can be addressed and resolved before starting another.

(1) HED Analysis - The applicants HED analyses should include the 4.5.9.1 HED following: Analysis

• Personnel Tasks and Functions - The impact of HEDs on personnel tasks and the functions supported by those tasks.

• Plant Systems - The impact of HEDs on plant systems, considering the safety significance of that system(s), their effect on accident analyses, and their relationship to risk-significant sequences in the plants PRA.

• Cumulative Effects of HEDs - The analysis of HEDs should identify the 4.5.7.3 Extent of cumulative effects that multiple HEDs may have on plant safety and the Issue personnel performance. Determination

• HEDs as Indications of Broader Issues - As well as addressing specific HEDs, the applicants analysis should determine whether the HEDs point to potentially broader problems.

(2) Selection of HEDs to Correct - The applicant should conduct an evaluation 4.5.9.2 Selection of to identify which HEDs to correct. HEDs to Correct The evaluation should identify those HEDs that are acceptable as is (The Additional Information below provides examples).

The remaining discrepancies should be denoted as HEDs to be addressed by the HED-resolution process.

HEDs the applicant should correct are those with direct safety consequences, namely, those that could adversely impact personnel 4.5.9.2 Priority 1 performance such that the margin of plant safety may be reduced below 4.5.9.3 Priority 1 an acceptable level. Unacceptability is indicated by such conditions as violations of Technical Specification safety limits, operating limits, or limiting conditions for operations, or failing an ISV pass/fail criterion.

HEDs with potential safety impact, not as severe as those described above, also should be corrected unless the applicant justifies leaving the KEPCO & KHNP A12

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 NUREG-0711 Rev. 3 Criteria IP Section(s) condition as is.

The applicant should correct HEDs that may adversely impact personnel performance in a way that has potential consequences to plant performance or SSC operability, and personnel performance or efficiency.

This may include failing to meet personnel information needs or violating HFE guidelines for tasks associated with plant productivity, availability, and protecting investment.

(3) Development of Design Solutions - The applicant should identify design solutions to correct HEDs. As part of the design solution, the application should evaluate the interrelationships of individual HEDs.

(4) Design Solution Evaluation - The applicant should evaluate design solutions to demonstrate the resolution of that HED and to ensure that new HEDs are not introduced. Generally, the evaluation should use the V&V method that originally detected the HED.

(5) HED Evaluation Documentation - The applicant should document each 4.5.9.5 HED HED, including: Evaluation

• the basis for not correcting an HED Documentation

• related personnel tasks and functions

• related plant systems

• cumulative effects of HEDs

• HEDs as indications of broader issues KEPCO & KHNP A13

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 APPENDIX B BARS QUESTIONNAIRE Date:

Rater:

During the last set of exercises that you observed, did you observe any items listed below that significantly affected the work outcome? Place an X on the team members who were involved. If the incident took place more than once, please indicate the number of occurrences.

1. Communication Communication SS RO TO EO STA Sufficient amount of communication - information that needed to be stated was stated.

Information was informative, not chatter.

Information was relevant to the situation.

Information was correct.

Information had the desired effect - the targeted person responded appropriately.

Information was given at the appropriate time in the scenario.

Did you observe any examples of extremely effective or extremely poor communication?

Please place an X over the rating which you feel best describes the crews performance on this dimension.

1 4 7 1: Failed to communicate or communicated misleading information 4: Communicated the important issues, but neglected other issues 7: Optimal communication - communication was informative, effective and timely KEPCO & KHNP B1

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

2. Team spirit Team spirit SS RO TO EO STA Joking in a positive manner.

Appropriate degree of informal discussion.

Silence due to hostility (negative).

Gestures of hostility (negative).

Arguing over trivial issues (negative).

Information was given at the appropriate time in the scenario Did you observe any examples of extremely effective or extremely poor team spirit?

Please place an X over the rating which you feel best describes the crews performance on this dimension.

1 4 7 1: Were openly negative, hostile 4: Were neutral 7: Demonstrated encouraging, positive and reinforcing behaviors KEPCO & KHNP B2

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

3. Openness Openness SS RO TO EO STA Helpful in giving information to crew members.

Asking for/providing assistance when needed.

Seeking confirmation when appropriate.

Did you observe any examples of extremely open or extremely closed team openness?

Please place an X over the rating which you feel best describes the crews performance on this dimension.

1 4 7 1: Were not willing to give assistance or ask for assistance 4: Moderate openness - provided/received some assistances, but more would have been better 7: Optimal openness - provided/received assistances as often as needed KEPCO & KHNP B3

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

4. Coordination as a crew Coordination as a crew SS RO TO EO STA Good ability to work together.

Able to shift roles when needed.

Work together to find facts needed for decision making.

Appropriate degree of delegation of task Good utilization of technical support Did you observe any examples of extremely good or extremely poor crew coordination?

Please place an X over the rating which you feel best describes the crews performance on this dimension.

1 4 7 1: Unable to work together 4: Moderately effective as a crew 7: Delegated wisely, shifted roles as needed - the team work together highly effectively KEPCO & KHNP B4

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1

5. Task focus/Decision making Task focus/Decision making SS RO TO EO STA Considering alternative actions Staying with chosen primary task (remaining focused on main task)

Prioritizing tasks Solving the correct problem Decisions appropriate to diagnosis Did you observe any examples of extremely effective or extremely poor task focus/decision making?

Please place an X over the rating which you feel best describes the crews performance on this dimension.

1 4 7 1: Unfocused, poor decision making 4: Moderately effective decision making 7: Optimal degree of consideration of alternatives and focus on task KEPCO & KHNP B5

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 APPENDIX C NASA-TLX QUESTIONNAIRE Title Endpoints Description How much mental and perceptual activity was Mental Demand Low / High required (e.g., thinking, deciding, calculating, remembering, etc.)?

How much physical activity was required (e.g.

pushing, pulling, turning, controlling, activating, Physical Demand Low / High etc.)? Was the task slow or brisk, slack or strenuous, restful or laborious?

How much time pressure did you feel due to the rate Temporal Demand Low / High or pace at which the task elements occurred? Was the pace slow and leisurely or rapid and frantic?

How successful do you think you were in accomplishing the goals of the task set by the Performance Good / Poor experimenter (or yourself)? How satisfied were you with your performance in accomplishing these goals?

How hard did you have to work (mentally and Effort Low / High physically) to accomplish your level of performance?

How insecure, discouraged, irritated, stressed and Frustration Low / High annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

KEPCO & KHNP C1

Non-Proprietary HF V&V Implementation Plan APR1400-E-I-NR-14008-NP, Rev. 1 Crew Number: Scenario:

Date: Evaluator:

1. Metal Demand 0 1 2 3 4 5 6 7 8 9 10 Low Medium High

2. Physical Demand 0 1 2 3 4 5 6 7 8 9 10 Low Medium High

3. Temporal Demand 0 1 2 3 4 5 6 7 8 9 10 Low Medium High

4. Effort 0 1 2 3 4 5 6 7 8 9 10 Low Medium High

5. Performance 0 1 2 3 4 5 6 7 8 9 10 Low Medium High

6. Frustration 0 1 2 3 4 5 6 7 8 9 10 Low Medium High KEPCO & KHNP C2