Text

LLC Revision 1 to Standard Design Approval Application, Part 2, Chapter 18, TR-130415-NP, Human Factors Engineering Verification and Validation Implementation Plan
	ML23304A384
Person / Time
Site:	05200050
Issue date:	12/31/2022
From:	; NuScale
To:	; Office of Nuclear Reactor Regulation
Shared Package
ML23306A033	List: ML23304A316; ML23304A317; ML23304A318; ML23304A320; ML23304A321; ML23304A323; ML23304A325; ML23304A327; ML23304A329; ML23304A331; ML23304A333; ML23304A335; ML23304A336; ML23304A338; ML23304A339; ML23304A341; ML23304A343; ML23304A345; ML23304A346; ML23304A348; ML23304A349; ML23304A354; ML23304A355; ML23304A357; ML23304A358; ML23304A359; ML23304A361; ML23304A363; ML23304A364; ML23304A365; ML23304A366; ML23304A368; ML23304A370; ML23304A371; ML23304A372; ML23304A373; ML23304A374; ML23304A375; ML23304A377; ML23304A378; ML23304A380; ML23304A382; ML23304A383; ML23304A384; ML23304A385; ML23304A387; ML23304A388; ML23304A389; ML23304A390; ML23304A391; ... further results
References
	LO-151262 TR-130415-NP
	Download: ML23304A384 (1)
	v • d • e

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Licensing Technical Report Human Factors Engineering Verification and Validation Implementation Plan December 2022 Revision 0 Docket: 52-050 NuScale Power, LLC 1100 NE Circle Blvd., Suite 200 Corvallis, Oregon 97330 www.nuscalepower.com

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Licensing Technical Report COPYRIGHT NOTICE This document bears a NuScale Power, LLC, copyright notice. No right to disclose, use, or copy any of the information in this document, other than by the U.S. Nuclear Regulatory Commission (NRC), is authorized without the express, written permission of NuScale Power, LLC.

The NRC is permitted to make the number of copies of the information contained in these reports needed for its internal use in connection with generic and plant-specific reviews and approvals, as well as the issuance, denial, amendment, transfer, renewal, modification, suspension, revocation, or violation of a license, permit, order, or regulation subject to the requirements of 10 CFR 2.390 regarding restrictions on public disclosure to the extent such information has been identified as proprietary by NuScale Power, LLC, copyright protection notwithstanding.

Regarding nonproprietary versions of these reports, the NRC is permitted to make the number of additional copies necessary to provide copies for public viewing in appropriate docket files in public document rooms in Washington, DC, and elsewhere as may be required by NRC regulations. Copies made by the NRC must include this copyright notice in all instances and the proprietary notice if the original was identified as proprietary.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Licensing Technical Report Department of Energy Acknowledgement and Disclaimer This material is based upon work supported by the Department of Energy under Award Number DE-NE0008928.

This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights.

Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table of Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Abbreviations and Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.0 Sampling of Operational Conditions and Scenario Development. . . . . . . . . . . . . . . 8 2.1 Sampling Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Identification of Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 Scenario Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Scenario Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.0 Design Verification Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1 Human-System Interface Inventory and Characterization . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.1 Human-System Interface Inventory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.2 Human-System Interface Characterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.3 Inventory Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.2 Human Factors Engineering Design Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.2.1 Verification Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2.2 Design Verification Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.3 Human-System Interface Task Support Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.3.1 Human-System Interface Task Support Verification Criteria . . . . . . . . . . . . . . . 13 3.3.2 Human-System Interface Task Support Evaluation Methodology . . . . . . . . . . . 14 4.0 Integrated System Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.1 Validation Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.2 Test Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.3 Validation Test Bed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.3.1 Interface Completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3.2 Interface Physical Fidelity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3.3 Interface Functional Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3.4 Environmental Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3.5 Data Completeness Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.3.6 Data Content Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table of Contents 4.3.7 Data Dynamics Fidelity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.3.8 Remote Human-System Interfaces Containing Important Human Actions . . . . 18 4.3.9 Test Bed Conformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.4 Integrated System Validation Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.5 Performance Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.5.1 Types of Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.5.2 Performance Measure Information and Validation Criteria . . . . . . . . . . . . . . . . 23 4.6 Test Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.6.1 Scenario Sequencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.6.2 Test Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.6.3 Training Test Personnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.6.4 Training Integrated System Validation Participants . . . . . . . . . . . . . . . . . . . . . . 28 4.6.5 Pilot Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.7 Data Analysis and Human Engineering Discrepancy Identification . . . . . . . . . . . . . . . . 28 4.8 Validation Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.0 Human Engineering Discrepancy Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 5.1 Human Engineering Discrepancies Design Solution Implementation . . . . . . . . . . . . . . 31 5.2 Human Engineering Discrepancy Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.0 Verification and Validation Results Summary Report . . . . . . . . . . . . . . . . . . . . . . . 35 7.0 NUREG-0711 Conformance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 8.0 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 8.1 Source Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 8.2 Referenced Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 List of Tables Table 1-1 Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Table 1-2 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Table 4-1 Characteristics of Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Table 4-2 Basis for Performance Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Table 7-1 Conformance with NUREG-0711 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Abstract Human factors verification and validation is a critical element of the Human Factors Engineering (HFE) Program that performs evaluations to verify that the HFE design conforms to HFE design principles and that it enables plant personnel to successfully and reliably perform their tasks to ensure plant safety and operational goals. The verification and validation HFE element consists of four major activities:

sampling of operational conditions (SOC) design verification integrated system validation human engineering discrepancy resolution NuScale employs an SOC strategy to guide the selection of human-system interface (HSI) to evaluate. This strategy is important because it is impractical to review all HSI due to the large number of tasks and combinations of HSI that are possible. Scenarios are created that sample tasks involving normal, abnormal, and emergency conditions.

Design verification involves HFE design verification, HSI inventory and characterization, and HSI task support verification. The goal of this element is to verify that the HSIs are designed in accordance with HFE design guidance. Human-system interface inventory and characterization describes all HSI displays, controls, and related equipment within the scope defined by the SOC.

Human-system interface task support verification confirms that the HSI supports task performance as defined by the task analysis.

Integrated system validation is an evaluation, using performance-based tests, to determine whether an integrated system design (e.g., hardware, software, and personnel elements) meets performance requirements and supports the plants safe operation.

Human engineering discrepancies are identified if HSI displays are discrepant when compared against design guidance or integrated system validation performance criteria are not met.

Verification and validation evaluations determine that the HFE design conforms to HFE design principles and that they enable plant personnel to perform required tasks to ensure plant safety and operational goals.

This implementation plan describes the methodology for conducting the evaluations and identifying and resolving human engineering discrepancies. The methodology described is consistent with the applicable provisions of Section 11 of NUREG-0711, Revision 3.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Executive Summary The human factors verification and validation (V&V) element of the Human Factors Engineering (HFE) Program consists of the following four major activities:

sampling of operational conditions design verification integrated system validation (ISV) identifying and resolving human engineering discrepancies Sampling of operational conditions identifies the conditions that are representative of the events that may be encountered during plant operation, conditions that reflect the characteristics that may contribute to variations in system performance, and conditions that consider the safety significance of the human-system interfaces (HSIs). These identified operational conditions are used in HSI inventory and characterization, HSI task support verification, HFE design verification, and ISV.

The HSI inventory and characterization describes HSI displays, controls, and related equipment lying within the scope defined by the sampling of operational conditions. The HSI task support verification confirms that the HSIs provide the alarms, information, controls, and support needed for personnel to perform their tasks as defined by the task analysis. Human Factors Engineering design verification confirms that the design of the HSIs conforms to HFE guidelines. Integrated system validation verifies, using performance-based tests, that the integrated system design (e.g., hardware, software, procedures, and personnel elements) supports the safe operation of the plant.

Human engineering discrepancies are identified during the V&V process. Human engineering discrepancy resolution may be performed iteratively. That is, the identified human engineering discrepancies are evaluated and resolved appropriately during one V&V activity before conducting other V&V activities. The preferred order of the process is HSI inventory and characterization, HSI task support verification, HFE design verification, and ISV, although iteration may be needed.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 1.0 Introduction 1.1 Purpose This document provides the human factors verification and validation (V&V) implementation plan (IP) for the NuScale Power, LLC (NuScale) plant human-system interface (HSI) design. The HSI design includes the hardware, software, and personnel elements used to operate a NuScale Power Plant.

The NuScale human factors V&V program confirms that the HSI design conforms to the specified design.

conforms to appropriate design criteria.

performs within acceptable limits under analyzed operating modes and conditions.

provides the complete set of alarms, controls, indications, and procedures needed to support the personnel tasks as identified in the task analysis (TA).

adequately supports plant personnel in the safe and reliable operation of the plant.

1.2 Scope This IP describes the methodology for conducting the four major activities of the human factors V&V element (sampling of operational conditions (SOC), design verification, integrated system validation (ISV), and human engineering discrepancy resolution),

including identification of sampling dimensions and scenarios used for validation of the HSI.

human-system interface inventory and characterization.

the criteria used for task support verification and Human Factors Engineering (HFE) design verification.

selection and training of the Validation Team.

determination of validation test objectives.

use of the main control room (MCR) test bed for validation.

selection and training of personnel used as operating crews (i.e., ISV participants).

scenario selection and definition for the validation.

performance measures to be used in the validation.

design of testing.

data analysis methods applied to validation data.

validation of procedures.

guidance for initiation and evaluation of human engineering discrepancies (HEDs).

This IP provides a description of the methodology for the identification of scenarios for the ISV. The V&V results summary report (RSR) provides the information as discussed in

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Section 6.0. A detailed ISV test report will be developed that supports the findings documented in the verification & validation results summary report.

1.3 Abbreviations and Definitions Table 1-1 Abbreviations Abbreviation Definition HED human engineering discrepancy HFE Human Factors Engineering HFEITS human factors engineering issue tracking system HSI human-system interface I&C instrumentation & control IHA important human action IP implementation plan ISV integrated system validation MCR main control room RSR results summary report SA situational awareness SME subject matter expert SOC sampling of operational conditions TA task analysis V&V verification & validation

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 1-2 Definitions Term Definition Embedded procedure An electronic procedure that is part of the NuScale HSI system that allows the operators to safely monitor and control the plant. An embedded procedure has bidirectional connection to the control networks.

Human Factors Engineering Generic term for the Plant Operations organization that consists of Design Team operators, human factor engineers, and simulator developers. The HFE Design Team does not include ISV participants. The HFE Design Team is responsible for the HFE associated with the NuScale design. Also referred to as the design team.

Human System Interface The human-system interface is that part of the system through which personnel interact to perform their functions and tasks. In this document, "system" refers to a nuclear power plant. Major HSIs include alarms, information displays, controls, and procedures. Use of HSIs can be influenced directly by factors such as

the organization of HSIs into workstations (e.g., consoles and panels)

the arrangement of workstations and supporting equipment into facilities such as an MCR, local control station, Technical Support Center, and Emergency Operations Facility and

the environmental conditions in which the HSIs are used, including temperature, humidity, ventilation, illumination, and noise. The HSI use can also be affected indirectly by other aspects of plant design and operation such as crew training, shift schedules, work practices, and management and organizational factors.

ISV participants Operating crew members participating in the ISV. Participants are not part of the HFE Design Team or Validation Team.

Simulator operator Person responsible for running the simulator during design, training, and testing. During training and testing, simulator operators keep track of directions given to non-licensed operators and other personnel simulated outside the control room. Simulator operators role play as personnel outside the control room and only provide data that are allowed per the applicable scenario or training guide. Simulator operators answer questions asked by the crew but do not lead them to the correct answer or diagnosis. Simulator operators are also referred to as booth operators.

Simulator Review Board The Simulator Review Board reviews the results of simulator testing and compares them to analysis and engineering calculations to certify that the simulator reflects the plant design. The review is focused on realism to the operator and model validity.

Unit A NuScale unit consists of the components necessary to generate electricity. This includes a primary side containing a reactor power module and its specific supporting systems, and a secondary side containing a turbine generator and its specific supporting systems.

Validation Team The Validation Team is responsible for administering the ISV tests. The Validation Team consists of test administrators, Operations observers, HFE observers, and simulator operators. The Validation Team administers the ISV and collects data via questionnaires, post-scenario debriefing, personal observations, and simulator-archived data. The Validation Team is also referred to as the ISV Test Team or test team.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 2.0 Sampling of Operational Conditions and Scenario Development The purpose of SOC is to identify a broad and representative range of operating conditions to be sampled during the HSI inventory and characterization (Section 3.1),

HFE design verification (Section 3.2), task support verification (Section 3.3), and ISV testing (Section 4.0). The sample is deemed representative if the samples safety significance, risk-significance, and challenges to the operating crew are considered to be within the range of events that the operators could encounter during the plant life cycle.

Subject matter experts (SMEs) select a combination of HSI features, test scenarios, plant process equipment degradations and failures, number and qualifications of operating staff members, and operating procedures to be employed, exercised, and stressed with each HSI validation test scenario.

2.1 Sampling Dimensions A range of plant conditions, personnel tasks, and situational factors is considered within the sampling dimensions included in Section 11.4.1 of Human Factors Engineering Program Review Model, NUREG-0711, Rev. 3 (Reference 8.1.1) as applicable to the NuScale design.

In the NuScale Power Plant US460 standard design, up to six units are operated from a single control room that uses a digital control system and relies heavily on automation and computer-based procedures.The sampling dimensions include normal operational events, transients, and accidents. Due to the increased use of digital technology in the NuScale control room, scenarios must specifically provide an emphasis on instrumentation and control (I&C) and HSI failures as well as degraded conditions.

Scenario development goals are written to ensure the scenarios are comprehensive, and when taken together, cover aspects of all sampling dimensions relevant to the NuScale design.

2.2 Identification of Scenarios Members of the NuScale HFE Design Team develop the ISV scenarios using multiple sampling dimensions to accomplish the goals and set the conditions to be included in each scenario based on the SOC.

Biases for individual dimensions are possible, but collectively, the scenarios avoid bias by representing scenarios that have both positive and negative outcomes.

require varying degrees of administrative burden to the ISV participants.

minimize the use of well-known and well-structured sequences (e.g., textbook design-basis accident mitigation).

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 The ISV scenarios are reviewed by the appropriate SMEs and approved by Operations management.

2.2.1 Scenario Security The following scenario security steps are maintained throughout the ISV entire development and testing process.

The scenario descriptions and collection of tasks are stored in a separate work area with access only granted to the scenario and testing developers.

The selected operating crew member participants (ISV participants) are not allowed to review documents associated with the completed scenarios (i.e.,

scenario guides).

Printed copies of scenario information are destroyed or placed in a secure location when not in use.

2.3 Scenario Definition The scenarios used for ISV testing are selected during the SOC and scenario development process. Scenarios are run in the test bed to validate performance of the integrated system (e.g., hardware, software, and personnel elements) and ensure the design is consistent with the objective. The defined scenarios are designed to involve major plant evolutions or transients, reinforce team concepts, and identify the role each individual plays within the team. Tasks performed by operators outside the MCR are modeled in the ISV scenarios to realistically simulate effects on personnel performance due to potentially harsh environments. Effects such as additional time to don protective clothing, set up of radiological access control areas, and employment of damage control, emergency, or temporary equipment are described in scenarios by use of time constraints or additions.

The NuScale ISV scenarios are developed in a systematic manner and include the following applicable test attributes:

a synopsis objectives initial conditions of the plant specific initial conditions pertinent to commencement of the scenario a timeline of events to be run including initiating conditions where appropriate critical tasks to be conducted workplace factors (e.g., environmental conditions) material or knowledge-based needs to support the task to be tested staffing level

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 where specific types of communications are necessary (e.g., an event notification to regulators via dedicated telephone line) details of that expected communication content scripted responses for test personnel (both in and out of the MCR) data to be collected by observers and instructors (rating scales for administrators are included where appropriate) pass or fail criteria for the scenario initial test bed set up criteria for terminating the scenario The ISV scenarios are developed to be representative of the range of events that could be encountered during the plants operation, determined by SOC as described in Section 2.1. The HFE Design Team provides variation in the scenarios developed to avoid scenarios leading to a well-structured and positive outcome. Scenarios are selected to confront the operating crew with challenging normal conditions and abnormal events containing multiple and unanticipated failures.

Test objectives are discussed in Section 4.2. An individual scenario cannot address all test objectives, but the aggregate ISV includes testing of all objectives. Each scenario tests some portion of the HSI for primary actions (control and verification via the plant response) and secondary actions (navigating the HSI for monitoring of other plant parameters); communication equipment is also verified during scenarios.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 3.0 Design Verification Methodology The design verification activity consists of the following three categories of tasks:

Inventory and Characterization - Compare the inventory of indication and controls included in the design to those presented in the HSI Design Verification - Compare the HSI with the requirements in the HSI Style Guide Task Support Verification - Evaluate a sampling of tasks identified in Section 2.0 to verify tasks can be performed successfully Discrepancies noted during the design verification are captured as an HED.

This implementation plan provides high level criteria and methods. These criteria are intended to be included in specific test plans that are used to test and obtain results. The use of consistent plans and procedures helps remove testing bias by providing clear evaluation criteria for each test.

Integrated system validation testing can involve hundreds or thousands of individual HSIs, and it is impractical and unnecessary to review all of them. Therefore, NuScale employs a sampling strategy to guide the selection of HSIs to review.

3.1 Human-System Interface Inventory and Characterization The objective of the HSI inventory and characterization is to accurately describe the set of selected HSI displays, controls, and related equipment within the scope defined by the SOC. This check ensures that the instrumentation and controls listed in design documents are reflected in the HSI implementation.

3.1.1 Human-System Interface Inventory The list of HSI inventory is generated from the plant design. During the TA, if an alarm, control, or indication is not available, the applicable design process will be followed to ensure the indication is added.

3.1.2 Human-System Interface Characterization Characterization defines the functionality of each HSI selected for verification.

Human-system interface design documents such as equipment lists, design specifications, and input and output lists are produced during HSI design.

Characteristics of each HSI component are included in the associated design document that includes the minimum set of information:

a unique equipment identification code that links the HSI component to the associated plant system or subsystem associated personnel functions and sub-functions type of HSI (e.g., indication, control, alarm, procedure, hard-wired, screen-based)

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 HSI characteristics and functionality (e.g., unit of measure, accuracy of variable or parameter, format, continuous or discrete (if a control), system response time)

HSI control characteristics and functionality (modes, accuracy, precision, format) method of use and associated user-aids physical or virtual (i.e., on a screen) location of HSI 3.1.3 Inventory Verification Inventory verification confirms the visual aspects (alarms, controls, indications, and the means of navigation among elements) of the HSI, including conformance to the NuScale Human System Interface Style Guide during HFE design verification. This process also includes verification of other HSI characteristics such as tag number, location, piping, and instrument diagram or logic diagram implementation.

NuScale HSI navigation and notifications are part of the spatially dedicated continuously visible main navigation bar. These elements do not need to be verified for every system HSI developed. These global elements are verified once during this verification phase for all selected HSIs following the process used during staffing plan validation.

3.2 Human Factors Engineering Design Verification The HFE design verification is conducted to confirm that HSI characteristics conform to HFE guidelines as represented in NUREG-0700 (Reference 8.1.3) and the NuScale Human System Interface Style Guide.

Procedures describing HFE design verification include checklists and guidelines for comparison of the HFE design criteria (HSI Style Guide) to HSI components (e.g., alarms, controls, indications, procedures, navigation aids).

a description of the means of comparing HFE design criteria to HSI components in the context of the various environmental conditions or locations of those HSIs (e.g.,

noise, lighting, ambient temperature and humidity).

guidelines for determining whether the HSI is acceptable or discrepant based on the associated HFE design criteria.

methods for preparation and review of the HFE design verification as well as a course of action when reviewers do not agree on the results.

design verification HEDs are generated for HSIs that do not meet the HFE design criteria.

3.2.1 Verification Criteria The criteria for HFE design verification is provided by the HSI Style Guide. The style guide includes guidance for determining appropriate design criteria. Deviations from

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 the HSI Style Guide may be taken and documented when appropriate with consent of the design team, but normally should conform to the guidance provided.

During HFE design verification, it is not required to evaluate all guidelines applicable to the HSI in total; a characteristic (e.g., alarm prioritization) of an individual HSI component (e.g., alarm management screens) may be selected for evaluation against a guideline or a subset of a guideline as appropriate.

3.2.2 Design Verification Evaluation Methodology The design verification evaluation is performed by the following method:

1. Select an SME to perform design verification that has sufficient knowledge of design criteria and HSI.

2. Select the sample of HSI displays or individual components to test.

3. Compare the HSI design to the design guide (e.g., HSI Style Guide or other authoritative document).

4. Document discrepancies noted during verification.

The design verification phase for all selected HSIs follows a process that provides a retest step if necessary.

3.3 Human-System Interface Task Support Verification The purpose of HSI task support verification is to verify the HSIs support the task requirements on the selected HSI. The assessment verifies that HSIs provide the alarms, controls, indications, and task support for personnel to perform their tasks as defined by the SOC. For HSI task support verification related to performance (e.g., accuracy and dynamic response), the validation test bed is used.

3.3.1 Human-System Interface Task Support Verification Criteria The task support verification checks that the HSI design supports the tasks within the sampling of conditions and the tasks are able to be performed correctly.

3.3.2 Human-System Interface Task Support Evaluation Methodology The HSI task support evaluation is performed by the following method.

1. Select an SME to perform task verification that has sufficient knowledge of the plant design, HSI, and task intent.

2. Ensure the appropriate task procedure or standard is ready for performance.

3. Perform the selected task using the HSI.

4. Confirm the HSI supports all control room licensed operator steps necessary to complete task.

5. Document discrepancies noted during performance of the task.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 4.0 Integrated System Validation The ISV is the process by which an integrated system design (e.g., hardware, software, and personnel elements) is evaluated using performance-based tests to determine whether it acceptably supports safe operation of the plant. In most cases, the ISV is undertaken after HEDs that were identified in the upstream process, including design verification, have been resolved and the resulting changes implemented. Resolution typically includes implementing the documented changes but in cases with no safety significance and no expected impact on ISV results, exceptions are allowed provided a strong justification is documented.

Objective performance measures and success criteria are developed as part of the methodology.

This section describes the minimum requirements needed to perform an ISV activity such that there is high confidence the HSI design demonstrates it is safe for operation.

4.1 Validation Team Validation Team members can be selected from the HFE Design Team. The Validation Team members are trained and qualified to conduct the ISV in an objective and unbiased manner. The conduct of the ISV is scheduled such that all portions are available for audit.

A detailed ISV test report is developed that supports the findings documented in the verification and validation results summary report; both documents are submitted to the NRC. The HFE Design Team developing and conducting the ISV is analogous to a commercial nuclear plants Training Department developing and conducting an NRC license exam or annual requalification exam.

The Validation Team consists of the following:

test lead plant operations experts HFE experts one lead test bed engineer (simulator operator) test bed support staff (simulator operator and communicator)

The observers (test lead, plant operations experts, and HFE experts) collect data via questionnaires, post-scenario debriefing, personal observations, review of video, and from HSI computer system logging. The observers are trained and qualified using the NuScale Training Program.

The administrators (test lead, test bed engineer, and test bed support staff) manage the ISV, control each scenario in accordance with the test procedure, maintain and set up the test bed, and collect the test bed archived data following each scenario. The Validation Team personnel act as simulated plant personnel as necessary within each scenario. The administrators are trained and qualified using the approved training program. Bias is

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 further reduced by obtaining results by consensus of the Validation Team rather than individual observations.

4.2 Test Objectives The objectives of the ISV are to validate the acceptability of the shift staffing, the assignment of tasks to operating crew members, and crew coordination within the control room, among the control room and local control stations and support centers, and with individuals performing tasks locally. This objective encompasses validating minimum shift staffing levels, nominal levels, higher levels, and shift turnover.

the design has adequate capability for alerting, informing, controlling, and feedback such that personnel tasks are successfully completed during normal plant evolutions, transients, design-basis accidents, and also during selected risk-significant events beyond-design-basis, as defined by the SOC.

specific personnel tasks can be accomplished within the time and performance criteria, with effective situational awareness (SA), and acceptable workload levels that balance vigilance and personnel burden.

the HSIs minimize personnel error and ensure error detection and recovery capability when errors occur.

the assumptions about performance on important human actions (IHAs).

4.3 Validation Test Bed The principal validation test bed for the ISV is the control room simulator. The fidelity of the validation test beds models and HSI are verified to represent the current, as-designed NuScale Power Plant before use for the validation.

The test bed model consists of multiple modeling software packages, all based from current NuScale designs. Together, they provide a high level of fluid and reactivity modeling. Precisely modeling the predicted behavior of the reactor core, thermodynamic performance, balance-of-plant, and electrical system design is desired as NuScale does not have a comparison reference plant. Up to six units are simultaneously and independently modeled, but they also share common systems that provide input for multiple units.

The test bed is validated against the seven criteria described in Section 11.4.3.3 of Reference 8.1.1: interface completeness, interface physical fidelity, interface functional fidelity, environment fidelity, data completeness fidelity, data content fidelity, and data dynamics fidelity. These criteria are further discussed in Section 4.3.1 through Section 4.3.7 below.

The validation test bed attempts to accurately simulate a NuScale Power Plant MCR environment. When simulation is not achievable by the test bed (e.g. room temperature rise and backup lighting during a loss of all AC power), a list of discrepancies is documented in the verification and validation results summary report discussed in

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Section 6.0. If necessary, changes are made to the ISV test procedure to reflect the alternate test bed configuration. In some limited cases, the Validation Team may consider the test bed discrepancies to affect specific aspects of the validation results. If so, an HED is generated to document the discrepancy and the concern. The HED is resolved in accordance with the HED resolution process described in Section 5.0.

4.3.1 Interface Completeness The test bed represents a complete and integrated system. The HSI and procedures provide a holistic representation of the MCR and support the sample of conditions listed in the test scenarios. The test bed further represents interfaces with other control stations as appropriate (e.g., communications) to provide an integrated system.

4.3.2 Interface Physical Fidelity High physical fidelity in the HSI and procedures is represented, including presentation of alarms, displays, controls, procedures, automation, job aids, communications, interface management tools, layout, and spatial relationships. The test bed is a replica in form, appearance, and layout of the NuScale MCR design.

4.3.3 Interface Functional Fidelity High functional fidelity in the HSI, procedures, and automation is represented so that the HSI functions are available and the HSI component modes of operation, types of feedback, and dynamic response characteristics operate in the same way as the physical plant.

4.3.4 Environmental Fidelity The test bed is representative of the physical NuScale Power Plant with regard to environmental features such as lighting, noise, temperature, humidity, and ventilation characteristics. In cases where the test bed cannot accurately simulate the environment, the ISV captures Human Factors Engineering issue tracking system (HFEITS) entries for evaluation and resolution.

4.3.5 Data Completeness Fidelity In the test bed, information and data provided to personnel represent the complete set of plant systems monitored and controlled from the corresponding facility.

4.3.6 Data Content Fidelity The test bed represents a high degree of data content fidelity. The alarms, controls, indications, procedures, and automation presented are based on an underlying plant model that accurately reflects the engineering design of the NuScale Power Plant.

The model also provides input to the HSI, such that the information matches what is presented during operations.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 4.3.7 Data Dynamics Fidelity The test bed represents a high degree of data dynamic fidelity. The plant model provides input to the HSI in a manner such that information flow and control responses occur accurately and in the correct response time. Information is provided to personnel with the same anticipated delays as would occur in the plant.

4.3.8 Remote Human-System Interfaces Containing Important Human Actions NuScale identified no IHAs that are conducted outside of the MCR. In the event that a remote IHA is determined in a later design stage, the test bed will use mockups to verify human performance requirements for IHAs conducted at HSIs remote from the MCR. The simulation or mockup considers, for example, transit times, use of personal protective equipment, and delays associated with the need for operator precision (e.g., self-checking).

4.3.9 Test Bed Conformance The test bed is verified to conform to required characteristics before validation tests are conducted.

4.4 Integrated System Validation Participants Operators participating in the ISV are previously licensed reactor or senior reactor operators, operators with Navy nuclear experience, design engineering staff members familiar with the NuScale Power Plant design, previously non-licensed operators at a nuclear plant, or personnel with a technical degree. The personnel participating in ISV are trained, qualified, and are assigned to roles commensurate with their experience, skill, and knowledge level.

Participants who constitute the ISV operating crews are not part of the Human Factors Engineering Validation Team or HFE Design Team. Operating crew makeup is not intentionally varied from scenario to scenario and remains generally consistent throughout the validation (i.e., crew members are not rotated among operating crews).

To control crew bias, individual crew members are distributed across crews with consideration for age distribution.

gender distribution.

education level distribution.

experience distribution.

Operating crew size for the validation tests includes a range of expected sizes to ensure that the HSI supports operations and event management. This range includes the minimum operating crew, nominal levels, and higher levels as defined during the staffing and qualifications program element NuScale Human Factors Engineering Staffing and

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Qualifications Results Summary Report (Reference 8.2.2) for a range of plant operating modes. The crew size for each scenario is identified in the ISV test procedure, and scenarios are not repeated with different crew sizes.

The ISV includes at least one scenario with more than minimum crew staffing defined in Reference 8.2.2 (e.g., additional licensed operators to complete a complex evolution) to simulate times of high control room traffic distractions, and high environmental loading.

The roles of the additional personnel and their interaction with the operating crew are determined by the scenario developers based on meeting the test objectives and goals and by applying the SOC criteria.

4.5 Performance Measurement Performance measures for ISV include measures of plant performance, personnel task performance, SA, cognitive and physical workload, and anthropometric or physiological factors. Test acceptance criteria is associated with clear and object dispositive measures whereas diagnostic measures are associated with supporting details or providing additional insight into observations and conclusions.

4.5.1 Types of Performance Measures 4.5.1.1 Plant Performance Measures Plant performance resulting from operator action or inaction includes plant process data (e.g., temperature, pressure) and component status (e.g., on or off; open or closed) as a function of time from multiple simulated locations. These data are obtained from various plant equipment including nuclear, fluid, structural, and electrical components. Components that provide plant process data or component status in the plant are simulated with appropriate fidelity. The test bed has the ability to record complete plant process data and component status (including state changes) for the full length of the ISV scenarios.

4.5.1.2 Personnel Task Performance Measures For each scenario, tasks that personnel are required to perform are identified and assessed. Primary and secondary personnel tasks are evaluated.

Primary tasks are those involved with function and task completion including detection, assessment, planning, and response. The level of detail to which primary tasks are measured and performance measures selected are assessed based on the complexity of the task. Time and accuracy are measured for lower level rule-based tasks to recognize and respond, while tasks that are knowledge-based (e.g., detection, seeking additional data, making decisions, or taking actions) entail the use of more detailed performance measures.

Secondary task performance measures reflect the workload associated with HSI manipulations associated with maintaining the overall plant. Test personnel

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 evaluate secondary tasks in conjunction with primary tasks to observe effects on overall performance and workload both at individual and operating crew level.

Personnel task performance measurements are selected to reflect those aspects of the task important to system performance and used depending on the particular scenario such as time.

accuracy.

frequency.

amount achieved or accomplished.

consumption or quantity used.

subjective report of participants.

behavior categorization by observers.

For knowledge-based tasks, more detailed data are collected in order to assess the complexity of the crew actions such as number of attempts.

number of navigational steps.

accuracy of actions.

frequency with which a specific action must be taken (repetitive actions).

number and severity of errors of omission or commission.

frequency with which plant parameters reach a limit before action is taken.

feelings and observations of scenario operating crews used in the ISV.

observations of test administrators.

plant performance.

Objective measures of individual or crew and system performance are also collected during validation scenarios and are used for documenting the performance and future use. They include video recordings of operator performance.

alarm history log.

operator control interactions.

plant variable control interactions (resulting from operator controls).

component status changes.

HSI use log (display screen request history and operational history).

The capturing of data using cameras enables NuScale to document the operators actions as they are performed. With the information archived, it is available for the

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 life of the design for tracking purposes. The comparison between actual and expected actions is an important test criterion when identifying errors of omission and commission. NuScale performs this comparison during the V&V testing process and maintains a retrievable video library, as a contingency, for instances where observations conflict or actions come into question.

4.5.1.3 Situational Awareness Performance Measures To measure SA, ISV applies a combination of objective measures along with subjective post-scenario questionnaire methods.

Performance measures for SA are obtained using non-intrusive human performance measures. Specific failure, malfunction, or out-of-alignment events are included in test bed scenarios to determine if the HSI adequately supports operators in detecting the event from indications and alarms and identifying the source of the problem. Observation of operator response provides an objective indication of SA. The data for this type of measurement can be collected while the scenario is dynamically running without interfering with the natural cognitive and collaborative operator crew processes.

subjective questionnaires. Post scenario self-assessments by test participants provide a complementary measure of SA that evaluates operator confidence in the HSI to provide adequate information to support SA.

4.5.1.4 Cognitive and Physical Workload Performance Measures To measure cognitive workload, the ISV employs the following methods.

Test subjects fill out a questionnaire after each scenario. This questionnaire asks for an assessment of each subjects mental workload including mental stress and effort required. Sample questions include

- How would you rate your ability to maintain awareness of plant conditions (i.e., SA)?

- How would you rate your overall mental workload during the scenario?

- Mental workload could be measured by perceived mental and perceptual activity (e.g., remembering, thinking, searching, calculating, and deciding).

- How would you rate your physical workload overall during the scenario?

- Physical workload for purposes of evaluating the HSI performance is a subset of mental workload and measures actions (e.g., pushing, pulling, turning, controlling, activating)

The operators ability to gather specific plant information, including the amount of time it takes to gather the information, and the ability to feed it correctly back to the crew while managing the scenario are collected to measure the level of cognitive workload and reserve capacity at a given time.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Observations of crew performance during scenarios and through post-scenario verbal debriefs and interviews are documented following each scenario to provide background for interpreting results.

4.5.1.5 Anthropometric and Physiological Factor Performance Measures The primary purpose of anthropometric and physiological performance measures during ISV is to assess those aspects of the design that cannot be evaluated during design verification. Anthropometric and physiological performance measures evaluate how well the HSI supports plant personnel in monitoring and control of the plant. Many of these design aspects are assessed as part of verifying the HFE design. Therefore, the focus is on those areas of the design that can only be addressed by testing the integrated system (e.g., the ability of personnel to effectively use the various controls, displays, workstations, or consoles while performing their tasks). Anthropometric challenges are collected during the scenarios or during review of video recordings. Observations that indicate a need for anthropometric or physiological changes to the HSI design include such concerns as visibility of displays.

accessibility of control devices.

ease of manipulating the control device.

4.5.2 Performance Measure Information and Validation Criteria 4.5.2.1 Collection Methods Subjective assessments of the HSI and its impact on performance, including self-ratings of workload, SA, and teamwork, are conducted by ISV participant operating crews. Operator feedback on the HSI is collected via post-scenario debriefs and questionnaires, which include scale rating questions and open feedback (long answer) questions.

Objective data (e.g., video recording, administrator observations) collected during test scenarios are analyzed to assess impacts of operator actions on plant processes and equipment states. The analysis compares the performance derived from parameters and times collected by the test bed to the evaluation criteria for operator actions and for overall plant process behavior developed for each scenario.

Test observers and administrators document individual assessments of crew performance on a post-scenario observer form after the scenario. The form documents observed performance problems in the following categories:

monitoring and detecting problems errors of omission critical action delays

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 errors of commission procedure deviations teamwork SA workload other problems In addition to HSI performance problems, observers and administrators rate technical and teamwork performance on the post-scenario observer form. Crew size sufficiency is rated, and potential or noticeable HEDs are identified.

Test subjects also document their feedback on a post-scenario test subject form following the scenario. The test subject form is similar to that of the observer and administrator with observations of HSI performance problems, technical and teamwork performance observations, crew size sufficiency ratings, and potential or noticeable HEDs.

The data collected from subjective and objective sources are analyzed by the HFE Design Team to determine the sufficiency of the HFE design.

4.5.2.2 Performance Measure Characteristics and Bases Performance measures observed during ISV contain the characteristics described in Table 4-1.

Table 4-1 Characteristics of Performance Measures Characteristic Meaning Construct Validity A measure should represent accurately the aspect of performance it is intended to measure.

Reliability A measure should be repeatable (i.e., same behavior measured in exactly the same way under identical circumstances should yield the same results).

Sensitivity A measure's range (scale) and its frequency (how often data are collected) should be appropriate to that aspect of performance being assessed.

Unobtrusiveness A measure should minimally alter the psychological or physical processes that are being investigated.

Objectivity A measure should be based on easily observed phenomena.

The basis for inclusion of a performance criterion in the ISV (or a particular scenario within ISV) used to judge acceptability of that criterion is determined during the development of the scenario. Bases for performance criteria are described in Table 4-2.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 4-2 Basis for Performance Criteria Criteria Basis Meaning Requirement The observed performance of the integrated system is compared with a quantified performance requirement (i.e., the requirements for the performance of systems, subsystems, and personnel are defined through engineering analyses).

Benchmark The observed performance of the integrated system is compared with a criterion established using a benchmark system (e.g., a current system is predefined as acceptable).

Norm The observed performance of the integrated system is compared with a criterion using many predecessor systems (rather than a single benchmark system).

Expert Judgment The observed performance of the integrated system is compared with a criterion established by SMEs.

Performance measures are designated as pass, fail or diagnostic. Diagnostic is measureable and the criteria include both range and unit of measure.

4.6 Test Design Test design refers to the process of developing scenarios, developing test plans, and conducting ISV based on the integrated HSI as described in the preceding sections. The goal of test design is to permit the observation of integrated system performance while minimizing bias.

Once the ISV test plan and scenarios are developed, they are reviewed by the appropriate SMEs and approved by operations management.

This section describes characteristics of the test design important to supporting ISV validity.

4.6.1 Scenario Sequencing Integrated System Validation: Methodology and Review Criteria, NUREG/CR-6393 (Reference 8.1.2), is employed as the standard for selection of crew or scenario order as follows.

A minimum of two operating crews perform each scenario.

Crews perform a grouping of scenarios in a different order than other crews.

When running individual scenarios across multiple crews, the order of the crews is varied when the scenario is changed.

Integrated system validation scenarios also contain variable normal operation time before introducing events to ensure that operating crews are not pre-tuned to immediate events and actions at the beginning of each scenario or at the same time during each scenario.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 4.6.2 Test Procedures Before ISV, detailed test procedures are prepared to manage tests, ensure consistency, control test bias, support repeatable results, and focus the test on the specific scenario objectives. The test observers and administrators use the test procedures to set up each scenario, manage the scenario, and analyze the test results. Scenario developers use test procedures to build the scenario set.

Integrated system validation test procedures are designed to minimize the introduction of bias by observers, administrators, and operating crews. A standardized scenario template is part of the test procedure. Test procedures include scenario order for each crew and order of crews when running a single scenario multiple times.

detailed and standardized instructions for briefing the test participants before each scenario.

specific instructions and criteria for observers and administrators on conduct of scenarios.

scripted questions and responses for administrators acting as plant staff during the scenario.

guidance on when and how to interact with the operating crew when the test bed encounters difficulties.

specification of unique data to be collected and stored (including what, when, and how) (Section 4.5).

guidance for documenting

- operating crews and scenario details.

- deviations from the test procedure, test difficulties, and significant unusual events.

- plant data.

- observer and administrator notes.

- post-scenario and final debriefing notes.

- crew questionnaires.

- observer and administrator questionnaires.

- observer and administrator consensus notes.

- video and audio recordings.

- HEDs.

post-testing instructions for each operating crew to not discuss the scenarios and HSI with others.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 4.6.3 Training Test Personnel Before starting ISV, observer and administrators are trained and qualified on NuScale Power Plant systems, the HSI, and ISV test procedures. Training consists of both classroom and test bed time. Training goals include ensuring familiarity with test procedures and scenarios.

reduction of potential bias and errors introduced by the observers and administrators due to test-based learning, failure to follow the test procedure, or incorrect interaction with the operating crew.

use of the test procedure.

documentation needs for each test, including

- where the test did not follow the scenario.

- problems that occur during testing, even if they were due to an oversight or error of those conducting the test.

the necessity of limiting observer and administrator interaction with test personnel to what is in the scenario description.

how to conduct post-scenario debriefings.

familiarity with HFE data collection tools and techniques.

familiarity with observation techniques, goals, and responsibilities specific to each observers role.

4.6.4 Training Integrated System Validation Participants Test participants undergo training similar to plant operators including conduct of operations, plant systems, HSI, plant events, and operating procedures. Test participants are not trained specifically on the scenarios in which they participate.

To ensure near-asymptotic performance and a consistent level of proficiency among individuals comprising the operating crews, only participants who have successfully completed the training program and have reached an acceptable level of proficiency are considered qualified for operating crew assignment.

4.6.5 Pilot Testing A test operating crew, which does not participate in ISV, conducts a pilot test (i.e., a pre-validation test) to assess the adequacy of test design, performance measures, and data collection methods.

give the observers and administrators experience in running the test.

ensure that the ISV runs smoothly and correctly.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 4.7 Data Analysis and Human Engineering Discrepancy Identification Test data are analyzed using both quantitative and qualitative methods. The analysis identifies the relationship between the observed and measured performance and the established acceptance criteria described in Section 4.5.2. Data are analyzed for each scenario across multiple trials. The method of analysis, consistency of measure assessing performance, and criteria used to determine successful performance for a given scenario is determined by the HFE Design Team.

Data is collected from multiple sources including crew debriefs, observer debriefs, National Aeronautics and Space Administration Task Load Index questionnaires, SA questionnaires, and management observations. The data are collected and reviewed by HFE and Operations SMEs to assess performance results to identify significant adverse issues and trends. This analysis compares and contrasts data sources, data across crews, data across trials, and data across scenarios. The HFE and Operations SMEs then collaborate on trending results and HED identification. Data management is accomplished by organizing issues into related categories in a manner that can be analyzed easily (e.g., using a database with search and filter capability).

These general categories provide the structure to conduct subsequent data analysis.

Specific issues are identified within each general category. The SMEs use experience and judgment to identify these issues. In some cases a single data point is identified as an issue. An example would be failure to meet a pass or fail performance criterion. In more typical cases, multiple data points are collected to support a collective root cause.

Data from different sources, crews and scenarios (convergent measures) are used to provide reinforcement of the significance, validity and extent of the issue.

The following rules are applied to this trending process.

At least two individuals work independently to identify trends. Each person then reviews the trends identified by the other(s).

Performance data can be placed in multiple general categories, and support multiple issues.

Performance data are not excluded from continuing review just because it has been included in a trend.

Trends address all aspects of the ISV process including procedures, training, HSI design, simulator performance, and crew performance.

Positive observations are also recorded in areas where a negative trend has been identified to provide perspective on the extent of the condition.

Human engineering discrepancy identification and resolution details are discussed in Section 5.0.

4.8 Validation Conclusions Integrated system validation conclusions are based on

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 a comprehensive testing program performed by an independent ISV team using test procedures covering the scope described above.

a high-fidelity test platform representative of the actual system, model, and HSI in aspects important to the integrated systems performance; variable aspects of the integrated system are adequately sampled.

acceptance criteria are measurable, reflect proper operational practices, and are representative of important aspects of performance.

test design minimizes bias or confounding effects so as not to affect the validity of the results.

statistical conclusions, where possible, are based on convergence of multiple measures.

specific pass and fail performance criteria documented as HEDs also identify the extent of the issue.

Integrated system validation conclusions documented in the V&V results summary report include the statistical and logical bases for determining that performance of the integrated system is acceptable.

the limitations in identifying possible effects on validation conclusions and that the impact on the design integration HFE Program element is considered, including

- aspects of the tests that are not well controlled.

- potential differences between the test situation and actual operations such as the absence of productivity-safety conflicts.

- differences between test platform design and the as-built NuScale Power Plant.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 5.0 Human Engineering Discrepancy Resolution Human engineering discrepancies are identified, documented, and resolved throughout the V&V process.

Human engineering discrepancies are not always resolved; HEDs may be found acceptable after an evaluation in the context of the integrated design. The basis for a decision that accepts an HED without change in the integrated design is documented, and is based on accepted HFE practices, current published HFE literature, trade-off studies, tests, or engineering evaluations. Human engineering discrepancies are identified in the V&V process during HFE design verification (Section 3.2).

task support verification (Section 3.3).

ISV (Section 4.0).

Human Factors Engineering issues and HEDs are identified and tracked in the HFEITS database. The HFEITS database is available to members of the HFE Design Team and identification of issues is part of the NuScale safety-conscious work environment. The HFEITS database is maintained until fuel load.

A sampling of HEDs found during the V&V process is discussed in the verification and validation results summary report human engineering discrepancy evaluation documentation section and includes information on the potential cumulative effects of HEDs observed and samples of HEDs that show an indication of broader issues seen during testing.

5.1 Human Engineering Discrepancies Design Solution Implementation During ISV testing, HEDs are analyzed for priority selection and design category placement (e.g., HSI or simulator). Once the HED has been received, a discrepancy entry is created in the HFEITS database and the HED is prioritized as priority 1, priority 2, or priority 3 HEDs according to their importance as follows.

Priority 1 HEDs have a potential direct or indirect impact on plant safety and are resolved before ISV testing is considered complete. Human engineering discrepancies initiated as a result of a performance measure not being met (pass or fail performance measures) are priority 1 HEDs. Cross-cutting issues determined through HED analysis or performance measure analysis are also priority 1 HEDs due to their global impact on the HSI design performance.

Priority 2 HEDs have a direct or indirect impact on plant performance and operability and are resolved before the plant design is completed.

Priority 3 HEDs are those that do classify as priority 1 or priority 2. Priority 3 HEDs do not have to be resolved. If resolution of priority 3 HEDs is determined to be needed, they are resolved during design implementation.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 The HED is then routed to the appropriate group for resolution. Human engineering discrepancies related to the HSI are sent to the HFE Design Team, and HEDs related to simulator modeling are sent to the Simulator Review Board. It is possible for HEDs to be routed to both groups.

The HED is then resolved, and the discrepancy entry closed. The HED resolution is reviewed for final closure in the HFEITS database by an HFE review committee.

5.2 Human Engineering Discrepancy Analysis Human Factors Engineering verification and validation human engineering discrepancies are categorized based on their principal impact on personnel tasks and functions.

plant systems.

human-system interface features.

individual HSI components.

operating procedures.

Extent of condition and causal effect across the various HSI design features and functions are assessed as part of the HED process. Extent of condition determination considers cumulative or combined effects of multiple HEDs and human engineering discrepancies that represent a broader issue.

Extent of condition evaluation includes questionnaires and debriefings that include explicit questions for test participants about issues that appear to represent larger underlying problems with the HSI design.

administrators and observers that review each HED against other HEDs to determine relationships and overlaps among issues.

the HFE Design Team independently reviews each HED to determine relationships and overlaps among issues.

The broad-reaching testing and number of evaluated performance measures limit the ability to perform statistical analyses. Testing of multiple scenarios with multiple crews (generally, each crew will develop a different strategy) makes it impractical to make conclusions based on performance of the population or deviations from a norm.

Therefore, observer and administrators, test participants, and the Validation Team evaluate instances where a performance measure is not met to determine causal factors.

Design-related deficiencies determined for alarms, controls, indications, and procedures are documented in an HED. Previous HFE Program elements may need to be evaluated to resolve the deficiency. The HSI design is not considered validated until an HED initiated by pass or fail measures as a result of ISV is resolved. Test-related deficiencies are documented in the HFEITS and may result in changes to the test procedure or scenario definition.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Assessments attained by different means that are intended to measure the same or similar performance measures are compared. When differing conclusions are reached, more detailed cause analysis is performed, including the review of test bed logs and video and audio tapes, if necessary. Measuring convergence may be necessary for a single team and single scenario or for multiple teams and across several scenarios depending on the performance measure. In cases where multiple types of data (debriefs, questionnaires, video recording, and administrator observations) are collected and none of the measures reveal a problem, the fact that multiple measures were used to probe for a problem increases confidence that there is no deficiency.

Additional assessment to determine extent of condition and identify patterns is conducted when two or more test participant crews exhibit general performance problems in at least one of the following categories:

SA control of the plant procedure adherence error tolerance mental workload physical workload team work supervision of automated systems Expert HFE Design Team judgment is employed to infer a margin of error from the observed performance or data analysis. This judgement takes into account that actual performance may be slightly more variable than ISV test results.

Data and data-analysis tools (e.g., equations, measures, spreadsheets, expert opinions, resulting HEDs) are documented for subsequent audit and application during design integration and human performance monitoring HFE Program elements. Individual HFEITS items are maintained as auditable records in the HFEITS database.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 6.0 Verification and Validation Results Summary Report Following completion of verification and validation activities, the results are compiled in an RSR. The RSR contains a description of Human Factors Engineering and Validation Team participants and roles.

Human Factors Engineering V&V results overview and principal findings from design verification.

a list of priority 1 HEDs generated from the V&V, the analyses associated with these HEDs, and their resolutions.

Human Factors Engineering V&V execution results.

- verification:

a description of the application of the verification program verification results based on TA verification results based on the HSI Style Guide discussion of HEDs that resulted from the verification, extent of condition, resolution, and subsequent HSI design changes made before validation verification test procedures verification procedure and analysis tools used to draw conclusions and provide assurance that selected scenarios are representative of expected operational conditions (tools may include tables or checklists)

- validation:

a description of the application of the validation program validation test procedures ISV procedure, including scenarios a detailed description of the specific scenario sets used in testing including test instructions, data collection instruments, SOC versus scenario comparison table, and scenario identification summary table data analysis results and validation conclusions, as compared to the minimum set of test objectives a discussion of pass and fail HEDs that resulted from the validation, extent of condition, resolution, and subsequent HSI design changes, analyses, or retest a discussion of performance improvement measures a discussion of validation results and conclusions that pass or fail criteria have been met

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 7.0 NUREG-0711 Conformance Evaluation Table 7-1 indicates where each NUREG-0711, Revision 3 criterion is addressed in this IP.

Table 7-1 Conformance with NUREG-0711 HFE V&V IP Section No.

Review Criteria and paragraph 11.4.1.1 Sampling Dimensions Section 2.1 The following sampling dimensions are addressed below: Plant conditions, personnel tasks, and situational factors known to challenge personnel performance.

(1) The applicant should include the following plant conditions:

normal operational events including plant startup, shutdown or refueling, and significant changes in operating power

I&C and HSI failures and degraded conditions that encompass:

- The I&C system, including the sensor, monitoring, automation and control, and communications subsystems; [e.g., safety-related system logic and control unit, fault tolerant controller, local "field unit" for multiplexer (MUX) system, MUX controller, and a break in MUX line]

- common cause failure of the I&C system during a design basis accident (as defined by BTP 7-19)

- HSIs including, loss of processing or display capabilities for alarms, displays, controls, and computer-based procedures

transients and accidents, such as:

- transients (e.g., turbine trip, loss of off-site power, station blackout, loss of all feedwater, loss of service water, loss of power to selected buses or MCR power supplies, and safety and relief valve transients)

- accidents (e.g., main-steam-line break, positive reactivity addition, control rod insertion at power, anticipated transient without scram, and various-sized loss-of coolant accidents)

- reactor shutdown and cooldown using the remote shutdown system

- reasonable, risk-significant, beyond-design-basis events that should be determined from the plant-specific PRA

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (2) The applicant should include the following types of personnel tasks: Section 2.1

Important HAs, Systems, and Accident Sequences - The sample should include all important HAs, as determined in Section 7. Additional factors that contribute highly to risk, as defined by the PRA, also should be sampled:

- dominant accident sequences

- dominant systems (selected through PRA importance measures, such as Risk Achievement Worth or Risk Reduction Worth)

Manual Initiation of Protective Actions - The sample should include manual system level actuation of critical safety functions.

Automatic System Monitoring - The sample should include situations in which humans must monitor a risk-important automatic system.

OER-Identified Problematic Tasks - The sample should include all personnel tasks identified as problematic during the applicant's review of operating experience.

Range of Procedure Guided Tasks -The sample should include tasks that are well defined by procedures. Personnel should be able to understand and execute the specified steps as part of their rule-based decision-making. Regulatory Guide 1.33, Appendix A, contains several categories of "typical safety-related activities that should be covered by written procedures." The sample should include appropriate procedures in each category:

- administrative procedures

- general plant operating procedures

- procedures for startup, operation, and shutdown of safety-related systems

- procedures for abnormal, off-normal, and alarm conditions

- procedures for combating emergencies and other significant events (e.g., reactor accidents, and declaration of emergency-action levels)

- procedures for controlling radioactivity

- procedures for controlling measuring and test equipment and for surveillance tests, procedures, and calibration

- procedures for performing maintenance

- chemistry and radiochemical control procedures

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph

Range of Knowledge-Based Tasks - The sample should include tasks that Section 2.1 are not well defined by detailed procedures.

Additional Information: A situation may demand knowledge-based decision-making if the procedural rules do not fully address the problem, or when the selection of an appropriate rule is unclear. An example in a pressurized water reactor plant may be the difficulty in diagnosing a steam generator tube rupture (SGTR) with a failure of radiation monitors on the plants secondary side. This happens because (1) there is no main indication of the rupture (the presence of radiation in secondary side), and (2) the other effects of the rupture (i.e., slight changes in pressures and levels on the primary and secondary sides) may be attributed to other causes. While the operators may use procedures to treat the symptoms of the event, the determination that the cause is a SGTR may call for a situational assessment based on an understanding of the plant\'s design and the possible combinations of failures that entail the observed symptoms. Errors in rule-based decision-making result from selecting the wrong rule, or incorrectly applying a rule. Errors in knowledge-based decision-making result from mistakes in higher-level cognitive functions, such as judgment, planning, and analysis. The latter are more likely to occur in complex failure events wherein the symptoms do not resemble the typical case, and thus, are not amenable to pre-established rules.

Range of Human Cognitive Activities - The sample should include the range of cognitive activities that personnel perform, including:

- detecting and monitoring (e.g., of critical safety-function threats)

- situation assessment (e.g., interpreting alarms and displays to diagnose faults in plant processes and in automated control and safety systems)

- planning responses (e.g., evaluating alternatives to recover from plant failures) response implementation (e.g., in-the-loop control of plant systems, assuming manual control from automatic control systems, and carrying out complicated control actions)

- obtaining feedback (e.g., feedback of the success of actions taken)

Range of Human Interactions - The sample should include the range of interactions among plant personnel, including tasks performed independently by individual crew members, and those undertaken by a team of crew members. These interactions among plant personnel should include interactions between:

- main control room operators (e.g., operations, shift turnover walkdowns)

- main control room operators with auxiliary operators and other plant personnel performing tasks locally (e.g., maintenance or I&C technicians, chemistry technicians)

- main control room operators and the TSC and the EOF

- main control room operators with plant management, the NRC, and other outside organizations

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (3) The applicant should include the following situational factors or error-forcing Section 2.1 contexts known to challenge human performance. It also should include situations specifically designed to create human errors to assess the systems error tolerance, and the ability of personnel to recover from any errors, should these occur, for example:

High-Workload Situations - The sample should include situations where variations in human performance due to high workload and multitasking situations can be assessed.

Varying-Workload Situations - The sample should include situations wherein variations in human performance due to workload transitions can be determined. These include conditions where there is (1) a sudden increase in the number of signals that must be detected and processed after a period in which signals were infrequent, and (2) a rapid reduction in the need for detecting signals and processing demands following a time of high sustained task-demand.

Fatigue Situations - To the extent possible, the sample should include situations that may be associated with fatigue, such as work on backshifts and tasks performed frequently with repetitive actions, such as repeated inputs to a touch screen during plant operations or pulling rods.

Environmental Factors - To the extent possible, the sample should include environmental conditions that may cause human performance to vary, e.g., poor lighting, extreme temperatures, high noise, and simulated radiological contamination.

11.4.1.2 Identification of Scenarios Section 2.2, all (1) The applicant should combine the results of the sampling to identify a set of V&V scenarios to guide subsequent analyses.

Additional Information: A given scenario may combine many of the characteristics identified by sampling of operational conditions.

(2) The applicant should not bias the scenarios by overly representing the Section 2.2, all following:

scenarios for which only positive outcomes are expected

scenarios that, for ISV, are relatively easy to conduct (i.e., scenarios should not be avoided simply because they are demanding to set up and run on a simulator)

scenarios that, for ISV, are familiar and well structured (e.g., which address familiar systems and failure modes that are highly compatible with plant procedures, such as textbook design-basis accidents)

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.1.3 Scenario Definition Section 2.3, all (1) The applicant should identify operational conditions and scenarios to be used for HSI Task Support Verification, Design Verification, and ISV. The applicant should develop detailed scenarios suitable for use on a full-scope simulator. The level of detail should be comparable to what one would include in a test plan. For each one, the following information should be defined to reasonably assure that important dimensions of performance are addressed, and to allow the scenarios to be accurately and consistently presented for repeated trials:

a description of the scenario and any pertinent prior history necessary for personnel to understand the state of the plant at the start-up of the scenario

specific initial conditions (a precise definition of the plants functions, processes, systems, component conditions, and performance parameters, e.g., similar to that at shift turnover)

events (e.g., failures) that will occur during the scenario and their initiating conditions, e.g., based on time, or a value of a specific parameter

precise definition of workplace factors, (e.g., environmental conditions, such as low levels of illumination)

needs for task support (e.g., procedures and technical specifications)

staffing level

details of communication content between control room personnel and remote personnel (e.g., load dispatcher via telephone)

scripted responses for test personnel who will act as plant personnel in the test scenarios Additional Information: Test personnel act as surrogates for personnel outside the control room. To the greatest extent possible, prepare responses to questions that may be asked by operators communicating with the personnel outside the control room. There are limits to the ability to preplan communications because personnel may ask unanticipated questions or make unforeseen requests. However, efforts should be made to detail what information personnel outside the control room can provide, and script the responses to likely questions.

the precise specification of what, when, and how data are to be collected and stored (including videotaping, questionnaires, and rating-scale administrations)

precise specifications on simulator set up

specific criteria for terminating the scenario

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (2) The applicants scenarios should realistically replicate operator tasks in the Section 2.3, all tests; then, the findings from the test can be generalized to the plants actual operations.

(3) When the applicants scenarios include work associated with operations remote from the main control room, the effects on personnel performance due to potentially harsh environments (e.g., high radiation) should be realistically simulated (e.g., additional time to don protective clothing, and access radiologically controlled areas).

11.4.1.4 Additional Considerations for Reviewing the HFE Aspects of N/A Plant Modifications In addition to any of the criteria above that relate to the modification being reviewed, the applicant should address the following considerations.

(1) The applicants operational conditions should reflect tasks that involve a modification, rather than the entire range of topics discussed in Section 11.4.1.

(2) For ISV, the applicants operational conditions should encompass the transfer of learning effects on personnel performance when modifying an old HSI or procedure.

Additional Information: Negative transfer of learning may occur when the new and old components are different and impose different demands on personnel.

(3) For ISV, when both old and new versions of the same HSIs are permanently present in the HSI but with different means of presentation and methods of operation, then the applicants evaluations should reasonably assure that personnel can alternate their use of these HSIs without degrading performance.

(4) Where old HSIs are to be deactivated but left in place in the HSI, the applicant should identify conditions for an ISV that would test the potential for their interfering with tasks.

Additional Information: For example, the presence of deactivated HSIs may cause visual clutter that interferes with the ability of personnel to locate and use other HSIs.

11.4.2 Design Verification Review Criteria Section 3.1.1, all 11.4.2.1 HSI Inventory and Characterization (1) Scope - The applicant should develop an inventory of all HSIs that personnel require to complete the tasks covered in the validation scenarios that were identified by the applicants Sampling of Operational Conditions.

The inventory should include aspects of the HSI used for managing the interface, such as navigation and retrieving displays, as well as those that control the plant.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (2) HSI Characterization - The applicants inventory should describe the Section 3.1.2, all characteristics of each HSI within the scope of the verification. The following is a minimal set of information for this characterization:

a unique identification code number or name

associated plant system and subsystem

associated personnel functions and tasks

type of HSI, e.g.,

- computer-based control (e.g., touch screen or cursor-operated button and keyboard input)

- hardwired control (e.g., J-handle controller, button, and automatic controller)

- computer-based display (e.g., digital value and analog representation)

- hardwired display (e.g., dial, gauge, and strip-chart recorder)

display characteristics and functionality [e.g., plant variables/parameters, units of measure, accuracy of variable/parameter, precision of display, dynamic response, and display format (e.g., bar chart or trend plot)]

control characteristics and functionality [e.g., continuous versus discrete settings, number and type of control modes, accuracy, precision, dynamic response, and control format (method of input)]

user-system interaction and dialog types (e.g., navigation aids and menus)

location in data-management system (e.g., identification code for information display screen)

physical location in the HSI (e.g., control panel section), if applicable The applicant should include photographs, copies of display screens, or similar samples of HSIs in the HSI inventory and characterization.

(3) Inventory Verification - The applicant should verify the inventory description Section 3.1.3, all of HSIs to ensure that it accurately reflects their current state.

11.4.2.2 HSI Task Support Verification Section 3.2 HSI Task Support Verification addresses the availability of items needed to Section 3.2.1, all support task requirements. As stated in Section 11.2, the objective of the HSI Task Support Verification review is to ensure that the applicant verified that the HSI provides the needed alarms, information, controls, and task support for personnel to perform their tasks, defined by the task analysis.

(1) Verification Criteria - The applicant should base the HSI task support criteria on the alarms, controls, displays, and task support needed by personnel to complete their tasks as identified by the applicants task analysis.

(2) General Methodology - The applicant should compare the HSIs and their Section 3.2.2, all characteristics (as defined in the HSI inventory and characterization) to the needs of personnel identified in the task analysis for the defined sampling of operational conditions, noted in Section 11.4.1.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (3) HED Identification - The applicant should identify and document an HED Section 5.1 all when:

An HSI needed for task performance (e.g., a necessary control or display) is unavailable.

HSI characteristics do not match the requirements of the personnel task (e.g., a display may show the needed plant parameter but not within the range or precision needed for the task).

HSIs are available that are not needed for any task.

Additional Information: Unnecessary HSIs introduce clutter, and can distract personnel from selecting the appropriate ones. It is important to verify that the HSI is unnecessary. Appropriate ones may not appear to be needed with personnel tasks for the following reasons:

The HSI is essential for a task that the task analysis did not address (i.e.,

it was not within the scope of the design review).

The task analysis was incomplete, overlooking the need for the HSI.

The HSI only partially meets the established requirements for the personnel task.

(4) HED Documentation - The applicant should document HEDs to identify the Section 5.2, all HSI, the tasks affected, and the basis for the deficiency (what aspect of the HSI was identified as not meeting task requirements).

Additional Information: The analysis and correction of HEDs is detailed in Section 11.4.4, Human Engineering Discrepancy Resolution Review Criteria.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (5) Additional Methodology Considerations for Plant Modifications - In addition N/A to any of the criteria above that relate to the modification being reviewed, the applicant should address the following considerations:

HSI Task Support Verification should address all aspects of HSIs described above related to the modification. For modifications to plant systems that do not include modifications of the HSIs, verification of task support should highlight any new demands for monitoring and control, and assess whether the existing HSI design adequately addresses them.

HSI Task Support Verification should cover configurations in the modification in which old HSIs are deactivated permanently, but not removed (e.g., abandoned in place). Criterion 4 in this subsection states that the HSIs should not contain any information, displays, or controls that do not support personnel tasks. This verification should identify deactivated HSIs that might negatively affect personnel performance, such as obstructing the view of important information or adding visual clutter that could interfere with monitoring. The applicant should identify deactivated HSIs requiring further evaluation through HFE design verification or ISV.

HSI Task Support Verification should address the temporary configurations of the HSIs and plant systems that may be created when establishing the modification, and so used by operations and maintenance personnel when the plant is not shutdown. These configurations may include:

- the use of HSIs that differ from the intended final design

- combinations of HSIs and system configurations that differ from both the original design and the intended final one For each temporary HSI configuration, the task requirements of personnel should be identified and compared to the information and control capabilities available.

Additional Information: For example, if a temporary configuration of plant systems introduces special monitoring requirements, the HSIs should provide the necessary information.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.2.3 HFE Design Verification Section 3.2 Section 3.2.1, all HFE Design Verification addresses the suitability of the HSI with regard to human capabilities and limitations. As stated in Section 11.2, the objective of the HFE Design Verification review is to evaluate the applicants verification that the design of the HSIs conforms to HFE guidelines.

(1) Verification Criteria - The applicant should base the criteria used for HFE Design Verification on HFE guidelines.

Additional Information: The choice of guidelines used in this verification depends upon whether the applicant developed a design-specific style guide. The acceptability of the style guide used by the applicant should be reviewed by the NRC staff using the review guidance in Section 8.4.3, HFE Design Guidance for HSIs. Using an NRC-reviewed style guide affords the criteria for verifying the HFE design. When no style guide is available, the guidelines in NUREG-0700 can be used by the applicant for this purpose.

However, because not all of the guidelines therein will be applicable to each review, the applicant should select those based on the characteristics of the HSIs being evaluated. Applicants should identify a subset of guidelines appropriate to a specific design based on the HSI characterization.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (2) General Methodology - The applicants HFE Design Verification Section 3.2.2, all methodology should include the following:

Procedures for comparing the characteristics of the HSIs with HFE guidelines for (1) the defined sampling of operational conditions, as noted in Section 11.4.1, and (2) the general environment in which HSIs are sited, including workstations, control rooms, and environmental characteristics (e.g., lighting and noise).

Additional Information: A single guideline may apply to many HSIs. By verifying all HSIs within the scenarios defined in Section 11.4.1, the consistency of applying a guideline across multiple HSIs can be assessed.

Procedures for determining for each guideline whether the HSI is "acceptable" or "discrepant." If discrepant, it should be designated as an HED, tracked, and evaluated (see Sections 2.4.4 and 11.4.4).

Additional Information: A judgment that an HSI is acceptable should be made only if compliance is total, i.e., only if every instance of the item is fully consistent with the criteria established by the HFE guidelines. If there is any noncompliance, full or partial, then an evaluation of discrepant should be given, and a notation made as to where it occurs.

Procedures for evaluating whether an HED is a potential indicator of additional issues.

Additional Information: For example, identifying an inappropriate format for presenting data on an individual display should be considered a potential sign that other display formats might be used incorrectly, or that the observed format is employed inappropriately elsewhere. Then, the sampling strategy should be modified to encompass other display formats. In some cases, discovering these discrepancies will warrant further review in the identified areas of concern.

(3) HED Identification - The applicant should identify an HED when a Section 5.0 characteristic of the HSI is "discrepant" from a guideline.

(4) HED Documentation - The applicant should document HEDs in terms of the HSI involved, and how its characteristics depart from a particular guideline.

Additional Information: The analysis and correction of HEDs is addressed in Section 11.4.4, Human Engineering Discrepancy Resolution Review Criteria.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (5) Additional Considerations for Reviewing the HFE Aspects of Plant N/A Modifications - In addition to any of the criteria above that relate to the modification being reviewed, the applicant should address the following considerations:

The scope of HFE design verification may be restricted to the modified HSIs and their interactions with the rest of the HSIs.

When both old and new versions of similar HSIs are available, this verification should offer reasonable assurance that their means of presentation and methods of operation are compatible, such that personnel performance will not be impaired when alternating the use of each one.

HEDs should be identified for the following:

- failure to meet "personnel-identified" functionality in addition to that specified by system designers. When a digital system replaces an existing system, it is important to ensure that all operational uses of the former system were addressed, even those that were not intended in the original design. The replacement system\'s design should consider the ways in which personnel actually used the former system

- poor integration with the rest of the HSI

- poor integration with procedures and training

Temporary configurations of the HSIs and plant systems that operations and maintenance personnel may use when the plant is not shutdown, should be reviewed to verify that their design is consistent with the principles of good HFE design, including consistency with the rest of the HSIs.

11.4.3 Integrated System Validation Section 4.0 11.4.3.1 Validation Team Section 4.1, all (1)The applicant should describe how the team performing the validation has independence from the personnel responsible for the actual design.

Additional Information: The members of the Validation Team should have no responsibility for the design; i.e., they should never have been part of the design team. While they may work for the same organization, their responsibilities must not include contributions to the design, other than validating it.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.3.2 Test Objectives Section 4.2, all (1) The applicant should develop detailed test objectives to provide evidence that the integrated system adequately supports plant personnel in safely operating the plant, to include the following considerations:

Validate the acceptability of the shift staffing level(s), the assignment of tasks to crew members, and crew coordination within the control room, between the control room and local control stations and support centers, and with individuals performing tasks locally. This should encompass validating minimum shift staffing levels, nominal levels, maximum levels, and shift turnover (see Section 6 for definitions).

Validate that the design has adequate capability for alerting, informing controlling, and feedback such that personnel tasks are successfully completed during normal plant evolutions, transients, design-basis accidents, and also under selected, risk significant events beyond-design basis, as defined by sampling operational conditions.

Validate that specific personnel tasks can be accomplished within the time and performance criteria, with effective situational awareness, and acceptable workload levels that balance vigilance and personnel burden.

Validate that the HSIs minimize personnel error and assure error detection and recovery capability when errors occur.

Validate the assumptions about performance on important HAs.

Additional Information: For example, the HRA within the plant PRA contains several assumptions regarding the performance of risk-important HAs. These assumptions should be validated for dominant sequences, such as decision-making and diagnosis strategies, and also for the human actions. This process should be completed before the final quantification stage of the PRA.

Validate that the personnel can effectively transition between the HSIs and procedures in accomplishing their tasks, and that interface management tasks, such as display configuration and navigation, are not a distraction or an undue burden.

(2) Additional Considerations for Reviewing the HFE Aspects of Plant N/A Modifications - In addition to any of the criteria above that relate to the modification being reviewed, the tests objectives and scenarios should be developed to encompass aspects of performance affected by the modified design (even when the HSIs are not modified), including personnel tasks.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.3.3 Validation Test beds Section 4.3 A test bed is the HSI representation used to perform validation evaluations. Section 4.3.1, all One approach an applicant can use to acceptably meet criteria 1 through 7 in this section is to use a test bed that is compliant with "Nuclear Power Plant Simulators for Use in Operator Training" (ANS, 2009).

(1) Interface Completeness - The applicants test bed should represent completely the integrated system. It should include HSIs and procedures not specifically required in the test scenarios.

Additional Information: Adjacent controls and displays may affect the ways in which personnel use those addressed by a particular validation scenario.

(2) Interface Physical Fidelity - The test beds HSIs and procedures should be Section 4.3.2, all represented with high physical fidelity to the reference design, including the presentation of alarms, displays, controls, job aids, procedures, communications equipment, interface management tools, layout, and spatial relationships.

(3) Interface Functional Fidelity - The test beds HSI and procedure Section 4.3.3, all functionality should be represented with high fidelity to the reference design. All HSI functions should be available.

Additional Information: High fidelity covers the HSI modes of operation (i.e.,

the changes in functionality that can be invoked by personnel selecting them), or changes in plant states.

(4) Environmental Fidelity - The test beds environmental fidelity should be Section 4.3.4, all represented with high physical fidelity to the reference design, including the expected levels of lighting, noise, temperature, and humidity. Thus, for example, the noise contributed by equipment, such as air-handling units, computers, and communications equipment should be represented in validation tests.

(5) Data Completeness Fidelity - Information and data provided to personnel Section 4.3.5, all should completely represent the plants systems they monitor and control.

(6)Data Content Fidelity - The test beds data content fidelity should be Section 4.3.6, all represented with high physical fidelity to the reference design. The presentation of information and controls should rest on an underlying model accurately mirroring the reference plant. The model should provide input to the HSI such that the information accurately matches that which is presented during operations.

(7) Data Dynamics Fidelity - The test beds data dynamics fidelity should be Section 4.3.7, all represented with high fidelity to the reference design. The process model should be able to provide input to the HSI so that information flow and control responses occur accurately and within the correct response time; e.g., information should be sent to personnel with the same delays as occur in the plant.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (8) For important HAs at complex HSIs remote from the main control room (e. Section 4.3.8, all g., a remote shutdown facility), where timely, precise actions are essential, the use of a simulator or mockup should be considered to verify that the requirements for human performance can be met. (For less important HAs, or for non-complex HSIs, human performance may be assessed on analysis, such as task analysis, rather than on simulations.)

(9) The applicant should verify the conformance of the test bed to the test Section 4.3.9, all bed-required characteristics before validation tests are conducted.

11.4.3.4 Plant Personnel Section 4.4, all (1) Participants in the applicants validation tests should be representative of plant personnel who will interact with the HSI (e.g., licensed operators, rather than training personnel or engineers).

(2) To properly account for human variability, the applicant should use a sample of participants that reflects the characteristics of the population from which it is drawn. Those characteristics expected to contribute to variations in system performance should be specifically identified; the sampling process should reasonably assure that the validation encompasses variation along that dimension. Determining representativeness should include considering the participants license type and qualifications, skill/experience, age, and general demographics.

(3) In selecting personnel for participating in the tests, the applicant should consider the minimum shift staffing levels, nominal levels, and maximum levels, including shift supervisors, reactor operators, shift technical advisors, etc.

(4) The applicant should prevent bias in the sample of participants by avoiding the use of participants who:

are members of the design organization

participated in prior evaluations

were selected for some specific characteristic, such as crews identified as good performers or more experienced 11.4.3.5 Performance Measurement Section 4.5, all ISV employs a hierarchal set of performance measures including measures of plant performance, personnel task performance, situation awareness, cognitive workload, and anthropometric/physiological factors. Errors of omission and commission also are identified. A hierarchal set of measures provides sufficient information to validate the integrated system design and affords a basis to evaluate deficiencies in performance and thereby identify needed improvements. Pass/fail measures are those used to determine whether the design is or is not validated. Diagnostic measures are used to better understand personnel performance and to facilitate the analyses of errors and HEDs.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.3.5.1 Types of Performance Measures Section 4.5.1.1, all (1) The applicant should identify the specific plant performance measures applicable to each ISV scenario.

Additional Information: They may address the performance of functions, systems, or component.

(2) The applicant should identify the primary task measures applicable to each Section 4.5.1.2, all ISV scenario.

For each scenario, the applicant should identify the primary tasks operators must perform to accomplish scenario goals, so that such measures can be developed.

Additional Information: The primary tasks are those involved in carrying out the functional role of the operator in supervising the plant; i.e.,

monitoring, detection, situation assessment, response planning, and response implementation. Primary tasks should be assessed at a level of detail appropriate to the tasks demands. For example, for some simple scenarios, measuring the time to complete a task may suffice. For complicated tasks, especially those described as knowledge-based, it may be appropriate to undertake a fine-grained analysis, such as identifying the tasks components, viz., seeking specific data, making decisions, taking actions, and obtaining feedback.

The measures chosen to evaluate personnel task performance should reflect those aspects of the task that are important to system performance, such as:

- time

- accuracy

- frequency

- amount achieved or accomplished

- consumption or quantity used

- subjective reports of participants

- behavior categorization by observers

The analysis of primary tasks will support the identification of errors of omission (primary tasks not performed). Also, any actions and tasks that operators actually perform that deviate from the primary tasks should be identified and noted. These actions should be used to identify errors of commission.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (3) The applicant should identify the secondary task measures applicable to Section 4.5.1.2, each scenario. paragraph 3 Additional Information: Secondary tasks are those personnel must perform when interfacing with the HSI, such as navigating through computer screens to find a needed display and to configure HSIs. The measurement of secondary task performance should reflect the demands of the detailed HSI implementation, e.g., time to configure a workstation, navigate between displays, and manipulate them (e.g., changing display type and scale settings).

(4) The applicant should identify the measures of situation awareness Section 4.5.1.3, all applicable to each scenario.

Additional Information: Situation awareness is the degree to which personnels perception of plant parameters and understanding of the plant's condition corresponds to its actual condition at any given time and influences predictions about future states.

(5) The applicant should identify the workload measures obtained for each Section 4.5.1, all; scenario. Section 4.5.1.4 Additional Information: Workload is comprised of the physical, cognitive, and other demands that tasks place on plant personnel. The impact of one or many of these aspects of workload should be considered in the performance measures.

(6) The applicant should identify the anthropometric and physiological Section 4.5.1.5, all measures obtained for each scenario.

Additional Information: Anthropometric and physiological factors include such concerns as visibility of displays, accessibility of control devices, and ease of manipulating the control device. Many of these design aspects are assessed as part of verifying the HFEs design. Therefore, attention should focus on those areas of the design that only can be addressed by testing the integrated system, e.g., the ability of personnel effectively to use the various controls, displays, workstations, or consoles while performing their tasks.

11.4.3.5.2 Performance Measure Information and Validation Criteria Section 4.5.2, all; Section 4.5.2.1 (1) The applicant should describe the methods by which these measures are obtained, e.g., by simulator data recording, participant questionnaires, or observation by subject-matter experts.

(2) The applicant should specify when each measure is obtained (recorded), Section 4.5.2.2 such as continuously, at specific points during the scenario, or after the scenario ends.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (3) The applicant should describe the characteristics (see Table 11-1) of the Section 4.5.2.2, performance measures. Table 4-1 Table 11-1 Characteristics of Performance Measures Characteristic Meaning Construct Validity A measure should represent accurately the aspect of performance it is intended to measure.

Reliability A measure should be repeatable; i.e., same behavior measured in exactly the same way under identical circumstances should yield the same results.

Sensitivity A measure's range (scale) and its frequency (how often data are collected) should be appropriate to that aspect of performance being assessed.

Unobtrusiveness A measure should minimally alter the psychological or physical processes that are being investigated.

Objectivity A measure should be based on easily observed phenomena.

(4) The applicant should identify the specific criterion for each measure used to Section 4.5.2.2, judge the acceptability of performance and describe its basis. Table 4-2 Additional Information: Table 11-2 describes the different bases for performance criteria.

Table 11-2 Basis for Performance Criteria Criteria Basis Meaning Requirement The observed performance of the integrated system is compared with a quantified performance requirement; i.e., the requirements for the performance of systems, subsystems, and personnel are defined through engineering analyses.

Benchmark The observed performance of the integrated system is compared with a criterion established using a benchmark system, e.g., a current system is predefined as acceptable.

Norm The observed performance of the integrated system is compared with a criterion using many predecessor systems (rather than a single benchmark system).

Expert Judgment The observed performance of the integrated system is compared with a criterion established by subject-matter experts.

(5) The applicant should identify whether each measure is a pass/fail one or a Section 4.5.2.2, final diagnostic one. paragraph

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.3.6 Test Design Section 4.6 11.4.3.6.1 Scenario Sequencing Section 4.6.1, all (1) The applicant should balance scenarios across crews to provide each crew with a similar, representative range of scenarios.

Additional Information: Random assignment of scenarios to crews for ISV is undesirable. The value of using random assignment to control bias is effective only when the number of crews is quite large.

(2) The applicant should balance the order of presentation of scenarios to Section 4.6.2, bullet 1 crews to provide reasonable assurance that the scenarios are not always presented in the same sequence (e.g., the easy scenario is not always used first).

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.3.6.2 Test Procedures Section 4.6.2, all (1) The applicant should use detailed, unambiguous procedures to govern the conduct of the tests. These procedures should include the following:

the identification of which crews receive which scenarios, and the order in which they should be presented

detailed and standardized instructions for briefing the participants Additional Information: The type of instructions given to participants can affect their performance on a task. This source of bias is minimized by developing standard instructions.

specific directions for the testing personnel on conducting the test scenarios, as elaborated in Scenario Definition (Section 11.4.1.3)

guidance on when and how to interact with participants when difficulties occur in simulation or testing Additional Information: Even when a high-fidelity simulator is used, the participants may encounter artifacts of the test environment that detract from their performance of the tasks that are the focus of the evaluation.

Guidance should be available to the test conductors to help resolve such conditions.

instructions on when and how to collect and store data. These instructions should stipulate which data are to be recorded by:

- simulator computers

- special-purpose instruments and devices for collecting data (such as situation awareness- and workload-questionnaires, or physiological measures)

- video recorders (locations and views)

- test personnel and subject-matter experts (such as via observational checklists)

procedures for documentation:

- identifying and maintaining files of test records including details of the crew and scenarios

- data collected

- logs created by those who conducted the tests

The procedures should detail the types of information that should be logged (e.g., when the tests were performed, deviations from the test procedures and why they occurred, and any unusual events that may be important to understanding how a test was run or for interpreting the findings from it). The procedure also should state when the types of information should be recorded.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (2) The applicants test procedures should minimize the opportunity for bias in Section 4.6.2, final the test personnels expectations and in the participants responses. paragraph Additional Information: The expectancies of test personnel may introduce a bias if the expectations of the testers systematically influence the collection of data. Expectancies can influence performance in many ways (e.g., test personnel may, by giving subtle cues or communications, provide direction to participants, or they may tend to evaluate the performance of participants in ways that reflect more favorably upon the design than would an objective observer). Participant response bias means that the design of the test itself affects the data obtained from participants. It is not necessarily implied that a response bias represents any deliberate attempt by the participants to be untruthful. The test environment can influence participants in ways that have little to do with the tests objectives. Response bias can occur in four ways. First, participants may wish to influence outcomes and so be biased toward producing data consistent with their desired result. Second, participants may want to provide data that they think the test personnel want to obtain. Third, participants may try to figure out how performance should vary under different conditions, and then influence data to be consistent with such differences. Fourth, participants may want to excel because they know that they are being observed. See NUREG/CR 6393 (O'Hara et al., 1997) for additional information.

11.4.3.6.3 Training Test Personnel Section 4.6.3, all (1) The applicant should train test personnel (those who conduct or administer the validation tests) on the following:

the use and importance of test procedures

bias and errors that test personnel may introduce into the data through failures to follow test procedures accurately or to interact with participants properly

the importance of accurately documenting problems arising during testing, even if they were due to an oversight or error of those conducting the test 11.4.3.6.4 Training Participants Section 4.6.4, paragraph 1

(1) The applicants training of participants should be very similar to the training plant personnel receive. It should reasonably assure that the participants knowledge of the plants design, and operations, and the use of the HSIs and procedures represent that of experienced plant personnel. Participants should not be trained specifically to carry out the selected validation scenarios.

(2) To assure that the participants performance is representative of plant Section 4.6.4, paragraph personnel, the applicants training of participants should result in near 2 asymptotic performance (i.e., stable, not significantly changing from trial to trial) and should be tested for such before conducting the validation.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.3.6.5 Pilot Testing Section 4.6.5, all (1) The applicant should conduct a pilot study before the validation tests begin to offer an opportunity for the applicant to assess the adequacy of the test design, performance measures, and data-collection methods.

(2) The applicant should not use participants in the pilot testing who will then be participants in the validation tests.

11.4.3.7 Data Analysis and HED Identification Section 5.0 (1) The applicant should use a combination of quantitative and qualitative methods to analyze data. The analysis should reveal the relationship between the observed performance and the established performance criteria.

(2) The applicant should discuss the method by which data is analyzed across Section 5.0 trials, and include the criteria used to determine successful performance for a given scenario.

(3) The applicant should evaluate the degree of convergence between related Section 5.0 measures (i.e., consistency between measures expected to assess the same aspect of performance).

Additional Information: For example, if situation assessment is measured by both a participant questionnaire, and an observer rating scale, the results should be consistent with each other. If they do not converge, the reason for this should be identified.

(4) When interpreting test results, the applicant should allow a margin of error Section 5.0 to reflect the fact that actual performance may be slightly more variable than observed validation-test performance.

(5) The applicant should verify the correctness of the analyses of the data. This Section 5.0 verification should be done by individuals or groups other than those who performed the original analysis, but may be from the same organization.

(6) The applicant should identify HEDs when the observed performance does Section 5.0 not meet the performance criteria.

Additional Information: The analysis and correction of HEDs is addressed in Section 11.4.4, Human Engineering Discrepancy Resolution Review Criteria.

(7) The applicant should resolve HEDs identified by pass/fail measures before Section 5.0 the design is accepted.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.3.8 Validation Conclusions Section 5.0 (1) The applicant should document the statistical and logical bases for determining that performance of the integrated system is, and will be acceptable.

(2) The applicant should document the limitations in the validation tests, their possible effects on the conclusions of the validation, and their impact on implementing the design.

Additional Information: Examples of possible limitations include:

aspects of the tests that were not well controlled

potential differences between the test situation and actual operations, such as the absence of productivity-safety conflicts

potential differences between the validated design and the as-built plant or system (if validation is directed to a plant under construction where such information is available, or to a new design using the validation findings from a predecessor)

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph 11.4.4 Human Engineering Discrepancy Resolution Review Criteria Section 5.0 (1) HED Analysis The applicants HED analyses should include the following:

Personnel Tasks and Functions - The impact of HEDs on personnel tasks and the functions supported by those tasks.

Additional Information: The potential effects of HEDs is determined, in part, by the importance of the personnel function to plant safety (e.g.,

consequences of failure), and their cumulative effect on personnel performance (e.g., degree of impairment and types of potential errors).

Plant Systems - The impact of HEDs on plant systems, considering the safety significance of that system(s), their effect on accident analyses, and their relationship to risk-significant sequences in the plants PRA.

Additional Information: The potential effects of these HEDs on the plants safety and personnel performance are determined, in part, by the safety significance of the plant system(s) related to the particular component.

Cumulative Effects of HEDs - The analysis of HEDs should identify the cumulative effects that multiple HEDs may have on plant safety and personnel performance.

Additional Information: Although an individual HED might not be considered sufficiently severe to warrant correction, the combined effect of several of them on a single aspect of the design could significantly degrade plant safety, and therefore, necessitate corrective action.

Likewise, when a single plant system with multiple associated HEDs affects several HSIs, then their possible combined effect on the operation of that plant system should be considered.

HEDs as Indications of Broader Issues - As well as addressing specific HEDs, the applicants analysis should determine whether the HEDs point to potentially broader problems.

Additional Information: For example, identifying multiple HEDs associated with one particular aspect of the HSI design, such as the remote shutdown panel, also might suggest other problems with that aspect of the design, such as inconsistent use of design procedures and style guides. In some cases, findings from evaluating HEDs could warrant further review in the identified areas of concern, e.g., when multiple cases of mislabeling are found, the reviewers may wish to do a more complete examination of labeling.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (2) Selection of HEDs to Correct Section 5.0, final paragraph The applicant should conduct an evaluation to identify which HEDs to correct. The evaluation should identify those HEDs that are acceptable as is (The Additional Information below provides examples). The remaining discrepancies should be denoted as HEDs to be addressed by the HED-resolution process.

HEDs the applicant should correct are those with direct safety consequences, namely, those that could adversely impact personnel performance such that the margin of plant safety may be reduced below an acceptable level. Unacceptability is indicated by such conditions as violations of Technical Specification safety limits, operating limits, or limiting conditions for operations, or failing an ISV pass/fail criterion.

HEDs with potential safety impact, not as severe as those described above, also should be corrected unless the applicant justifies leaving the condition as is.

The applicant should correct HEDs that may adversely impact personnel performance in a way that has potential consequences to plant performance or SSC operability, and personnel performance or efficiency.

This may include failing to meet personnel information needs or violating HFE guidelines for tasks associated with plant productivity, availability, and protecting investment.

Additional Information: HEDs could be acceptable within the context of the fully integrated design. The technical basis for such a determination could include an analysis of recent research literature, current practices, tradeoff studies, or design engineering evaluations.

(3) Development of Design Solutions Section 5.2, all Also described in The applicant should identify design solutions to correct HEDs. As part of Reference 8.2.1.

the design solution, the application should evaluate the interrelationships of individual HEDs.

Additional Information: HEDs should not be considered in isolation and to the extent possible, their potential interactions should be considered when developing and implementing solutions. For example, if the HSI for a single plant system is associated with many HEDs, then the set of design solutions should be coordinated to enhance overall performance and avoid incompatibilities between individual solutions. Similarly, if a single plant system is associated with multiple HSIs associated with HEDs, then the design of individual solutions should be harmonized so that the outcome enhances rather than detracts from that system's operation. Approaches that develop design solutions to some HEDs before all are identified in a particular V&V activity are acceptable provided that the potential interactions between HEDs are specifically considered before implementing the design solutions.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 Table 7-1 Conformance with NUREG-0711 (Continued)

HFE V&V IP Section No.

Review Criteria and paragraph (4) Design Solution Evaluation Section 5.2, all Also described in The applicant should evaluate design solutions to demonstrate the Reference 8.2.1.

resolution of that HED and to ensure that new HEDs are not introduced.

Generally, the evaluation should use the V&V method that originally detected the HED.

Additional Information: For example, if the HED was identified using HFE Design Verification, then that verification should be employed to evaluate the solution. However, there may be reasons for documenting a satisfactory resolution using other methods. For example, if an aspect of the HSI was significantly changed from the resolution of multiple HEDs, the final HSI design may be validated to ensure that the net effect of all the changes is acceptable.

(5) HED Evaluation Documentation Section 5.2 all The applicant should document each HED, including:

the basis for not correcting an HED

related personnel tasks and functions

related plant systems

cumulative effects of HEDs

HEDs as indications of broader issues Additional Information: Some, or all, of this documentation may be included in the issues tracking system (Section 2.4.4). Other information, such as cumulative effects or indications of broader issues, may be documented separately.

Human Factors Engineering Verification and Validation Implementation Plan TR-130415-NP Revision 0 8.0 References 8.1 Source Documents 8.1.1. U.S. Nuclear Regulatory Commission, Human Factors Engineering Program Review Model NUREG-0711, Rev. 3, November 2012.

8.1.2. U.S. Nuclear Regulatory Commission, Integrated System Validation:

Methodology and Review Criteria NUREG/CR-6393, January 1997.

8.1.3. U.S. Nuclear Regulatory Commission, Human-System Interface Design Review Guidelines NUREG-0700, Rev. 3, July 2020.

8.2 Referenced Documents 8.2.1. NuScale Human Factors Engineering Program Management Plan, TR-130414, Revision 0.

8.2.2. NuScale Human Factors Engineering Staffing and Qualifications Results Summary Report, TR-130412, Revision 0.

ML23304A384

Text

Navigation menu

Search