ML18201A351

LLC - Transmittal of Response to NRC Request for Additional Information No. 482on Design Certification Application

ML18201A351
Person / Time
Site:	NuScale
Issue date:	07/20/2018
From:	Rad Z NuScale
To:	Document Control Desk, Office of New Reactors
Shared Package
ML18201A350	List: ML18201A351
References
AF-0718-60925, RAIO-0718-60922
Download: ML18201A351 (33)

Text

RAIO-0718-60922 July 20, 2018 Docket No.52-048 U.S. Nuclear Regulatory Commission ATTN: Document Control Desk One White Flint North 11555 Rockville Pike Rockville, MD 20852-2738

SUBJECT:

NuScale Power, LLC Response to NRC Request for Additional Information No.

482 (eRAI No. 9395) on the NuScale Design Certification Application

REFERENCE:

U.S. Nuclear Regulatory Commission, "Request for Additional Information No.

482 (eRAI No. 9395)," dated May 24, 2018 The purpose of this letter is to provide the NuScale Power, LLC (NuScale) response to the referenced NRC Request for Additional Information (RAI).

The Enclosures to this letter contain NuScale's response to the following RAI Question from NRC eRAI No. 9395:

18-47 is the proprietary version of the NuScale Response to NRC RAI No. 482 (eRAI No.

9395). NuScale requests that the proprietary version be withheld from public disclosure in accordance with the requirements of 10 CFR § 2.390. The enclosed affidavit (Enclosure 3) supports this request. Enclosure 2 is the nonproprietary version of the NuScale response.

This letter and the enclosed responses make no new regulatory commitments and no revisions to any existing regulatory commitments.

If you have any questions on this response, please contact Carrie Fosaaen at 541-452-7126 or at cfosaaen@nuscalepower.com.

Sincerely, Zackary W. Rad Director, Regulatory Affairs NuScale Power, LLC Distribution: Gregory Cranston, NRC, OWFN-8G9A Samuel Lee, NRC, OWFN-8G9A Prosanta Chowdhury NRC, OWFN-8G9A Enclosure 1: NuScale Response to NRC Request for Additional Information eRAI No. 9395, proprietary NuScale Power, LLC 1100 NE Circle Blvd., Suite 200 Corvalis, Oregon 97330, Office: 541.360.0500, Fax: 541.207.3928 www.nuscalepower.com

RAIO-0718-60922 : NuScale Response to NRC Request for Additional Information eRAI No. 9395, nonproprietary : Affidavit of Zackary W. Rad, AF-0718-60925 NuScale Power, LLC 1100 NE Circle Blvd., Suite 200 Corvalis, Oregon 97330, Office: 541.360.0500, Fax: 541.207.3928 www.nuscalepower.com

RAIO-0718-60922 :

NuScale Response to NRC Request for Additional Information eRAI No. 9395, proprietary NuScale Power, LLC 1100 NE Circle Blvd., Suite 200 Corvalis, Oregon 97330, Office: 541.360.0500, Fax: 541.207.3928 www.nuscalepower.com

RAIO-0718-60922 :

NuScale Response to NRC Request for Additional Information eRAI No. 9395, nonproprietary NuScale Power, LLC 1100 NE Circle Blvd., Suite 200 Corvalis, Oregon 97330, Office: 541.360.0500, Fax: 541.207.3928 www.nuscalepower.com

Response to Request for Additional Information Docket No.52-048 eRAI No.: 9395 Date of RAI Issue: 05/24/2018 NRC Question No.: 18-47 Regulatory Basis Title 10 of the Code of Federal Regulations (10CFR) Section 52.47(a)(8) requires an applicant for a design certification to provide a final safety analysis report (FSAR) that must include the information necessary to demonstrate compliance with any technically relevant portions of the Three Mile Island requirements set forth in 10 CFR 50.34(f), except paragraphs (f)(1)(xii), (f)(2)

(ix), and (f)(3)(v). Section 10 CFR 50.34(f)(2)(iii) requires an applicant to "Provide, for Commission review, a control room design that reflects state-of-the-art human factor principles prior to committing to fabrication or revision of fabricated control room panels and layouts."

Chapter 18, "Human Factors Engineering," of NUREG-0800, "Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants: LWR Edition," and NUREG-0711, "Human Factors Engineering Program Review Model," identify criteria the staff uses to evaluate whether an applicant meets the regulation. The applicant stated in the FSAR, Tier 2, Section 18.0, "Human Factors Engineering - Overview," that its human factors engineering (HFE) program incorporates accepted HFE standards and guidelines including the applicable guidance provided in NUREG- 0711, Revision 3.

Background Information By letter dated December 31, 2016, NuScale submitted a design certification application (DCA) for review (Agencywide Documents Access and Management System (ADAMS) Accession No. ML17013A229). The DCA includes Design Control Document (DCD) Tier 2, Chapter 18, "Human Factors Engineering," which describes the human factors engineering (HFE) program for the NuScale Power Plant (ADAMS Accession No. ML17013A289). As part of DCD Chapter 18, NuScale submitted an implementation plan (IP) for the HFE verification and validation (V&V) element. In a letter dated April 8, 2016, from Mr. Thomas Bergman, NuScale, Vice-President, Regulatory Affairs; to Mr. Frank Akstulewicz, Director, Division of New Reactor Licensing, NRC (ADAMS Accession No. ML16099A270); NuScale stated that NuScale would submit a V&V results summary report (RSR) prior to Phase 4 of the NRC's review of the NuScale DCA.

Based on NuScale's response to RAI 9123 dated January 10, 2018 (ADAMS Accession No. ML18002A554), the staff understands NuScale plans to commence integrated systems validation (ISV) testing, a significant V&V activity, mid-year.

NuScale Nonproprietary

NUREG-0711, Section 1.2.2, "Review Elements," says, "An IP review gives the applicant the opportunity to obtain an NRC staff review of, and concurrence with the methodology before the applicant conducts the work associated with the element. This type of review is desirable from the NRC staff's perspective because it offers the staff an opportunity to identify issues with the methodology and provide input early in the analysis or design process when the applicant more easily can address staff concerns than when the element is completed." The response to RAI 8747, Question 18-11 provided by NuScale on January 10, 2018 (ADAMS Accession No. ML17354A845), says, "NuScale submitted the V&V IP in order that NRC staff better understand the methodology used by NuScale during performance of V&V activities described in NUREG-0711. NuScale will not include a detailed description of the V&V methodology in the V&V RSR unless the methods actually used during V&V activities differ from those described in the V&V IP."

Consistent with NUREG-0711, Section 1.2.2, the staff commenced a review of the V&V IP and observed in some cases, the document did not contain enough information for the staff to assess how the V&V methodology conforms to the staff's guidance in NUREG-0711, Section 11.4, "Review Criteria." The staff conducted an audit of non- docketed information that contained more detail on NuScale's proposed V&V methods, including the "ISV Test Plan," as described in the audit plan dated July 25, 2017 (ADAMS Accession No. ML17205A465). As stated in the audit plan, the purpose of the audit was to review non-docketed information related to V&V methods to evaluate conformance with regulatory guidance and identify information that will require docketing to support the basis of the regulatory decision.

The following questions resulted from this review. The questions are categorized into two types.

The first category identifies instances where the V&V IP either does not provide any information or enough detail for the staff to evaluate conformance with regulatory guidance, and the information was not found in the non-docketed material reviewed during the audit. The second category includes questions where the V&V IP and the non-docketed material are not consistent, and therefore it is not clear to the staff what method(s) NuScale intends to use. In order to provide the greatest amount of regulatory certainty for NuScale on the proposed ISV methodology, the staff is requesting the information listed below to complete the review of the methodology prior to NuScale commencing ISV testing.

The questions to follow relate to the staff's review criteria cited here:

Criteria 11.4.3.5.1 (1-6) of NUREG 0711, states applicants should identify, 1) plant performance measures 2) primary task measures, 3) secondary task measures, 4) situation awareness measures, 5) workload measures and 6) anthropometric/physiological measures for each ISV scenario.

Criteria 11.4.3.5.2 (1-5) of NUREG-0711 states that the applicant should expand on these measures and the validation criteria associated with them.

NuScale Nonproprietary

Questions Category 1: Lack of Information

1. In the Final Safety Analysis Report (FSAR), Section 18.10.2.3.5.1, the applicant identified "non-intrusive human performance measures" as one way that situation awareness will be measured. The staff referenced NUREG/CR- 7190, "Workload, Situation Awareness, and Teamwork" in evaluating the method. NUREG/CR-7190 cites three types of implicit metrics, each which has advantages and limitations to consider as part of ISV test design. It is unclear to staff what type of implicit measure will be used, is being used and thus staff cannot evaluate its conformance to regulatory guidance.

a. Please clarify whether the non-intrusive measure will be part of the operator's primary task or unrelated to the primary task (e.g., an alarm sounds in the main control room that is unrelated to the current task).

b. If it is part of the primary task, please explain how the relationship between task performance and situation awareness is determined as performance can be caused by factors other than situation awareness (e.g. procedure problems, teamwork, etc.).

2. In Section 4.5.1.3, bullet 2 of the V&V IP, the applicant identifies a measure of situation awareness.

a. Please explain how the information collected relates to operator situational awareness.

b. Please explain why this is a valid measure of situation awareness.

3. Criteria 11.4.3.5.2 (3) states, "The applicant should describe the characteristics (see Table 11-1 of NUREG- 0711) of the performance measures" (construct validity, reliability, sensitivity, unobtrusiveness, objectivity). In Section 4.5.2.2 of the V&V IP, Rev 4, the applicant states, "Performance measures to be observed during ISV contain the characteristics described." This statement is followed by Table 11-1 of NUREG-0711 with no further information provided. Please provide a description of all characteristics identified in Table 11-1 of NUREG 0711 for each performance measure.

4. Criterion 11.4.3.5.2 (5) states that, "The applicant should identify whether each measure is a pass/fail one or a diagnostic one." In Section 4.5 of the V&V IP, Rev 4, the applicant states "Performance measures for ISV are hierarchical and include measures of plant performance, personnel task performance, SA, cognitive and physical workload, and anthropometric or physiological factors. Both pass or fail and diagnostic measures are applied." For each performance measure identified, please specify whether the measure will be pass/fail, diagnostic or both.

5. Criterion 11.4.3.5.2 (4) states, "The applicant should identify the specific criterion for each measure used to judge the acceptability of performance and describe its basis (Requirement, benchmark, norm, expert judgment)." While the staff understands that some specific details of NuScale Nonproprietary

certain criterion will not be available until the Integrated System validation scenarios are finalized (e.g. numeric performance thresholds, time requirements), the staff requests that NuScale provide additional information to clarify the measurement approach in order to assess the adequacy of the V&V IP.

a. For those measures that will be obtained via observation please identify the types of behaviors that observers will be looking for.

b. For those measures that will be obtained via observation, identify any measurement tools they will use (e.g. stopwatch for timing a behavior, checklists, etc.)

c. For those measures that will be obtained via observation, clarify how expert observers will know what to look for (e.g. use of briefing booklets)

d. Section 4.5.2.2 of the V&V IP states, "The basis for inclusion of a performance criterion in the ISV (or a particular scenario within ISV) used to judge acceptability of that criterion is determined during the development of the scenario." While the staff understands that precise thresholds for acceptability for many performance measures will not be available until Integrates System Validation scenarios have been finalized, the basis for those thresholds is needed to understand the measurement approach. In Section 4.5.2.1 of the V&V IP, Rev 4, the applicant provides some generic information regarding basis such as, "Objective data (e.g., video recording, administrator observations) collected during test scenarios are analyzed to assess impacts of operator actions on plant processes and equipment states. The analysis compares the performance derived from parameters and times collected by the test bed to the evaluation criteria for operator actions and for overall plant process behavior developed for each scenario."

However, it is not clear which specific measures are being referred to, nor whether "evaluation criteria for operator actions and for overall plant process behavior" is based on a requirement (e.g. engineering analysis), a norm, expert judgment or a benchmark.

For each performance measure, please describe the basis that will be used (e.g.

requirement, benchmark, norm, expert judgment) to establish the specific criterion used to judge the acceptability of performance.

Category 2: Information in the V&V IP and ISV Test Plan are different

6. The staff reviewed information regarding performance measures via an audit of documents in the NuScale electronic reading room (the audit plan is available as ADAMS Accession No. ML17205A465), including the "Integrated Systems Validation Test Plan." Section 11.0 of this document discusses a measure of situation awareness that is not discussed in the FSAR or the V&V IP, Rev 4.

Please clarify whether the measure of situation awareness cited in Section 11 of the "Integrated Systems Validation Test Plan" will be used during integrated system validation testing and, if so, update the application accordingly.

NuScale Nonproprietary

7. In the FSAR, Tier 2, Section 18.10.2.3.5.1, the applicant states, "To measure cognitive workload, the ISV employs questionnaires and observations of operators' ability to gather specific plant information, and crew performance." Based on a staff audit of documents in the NuScale electronic reading room (the audit plan is available as ADAMS Accession No. ML17205A465), specifically Section 11 of "Integrated Systems Validation Test Plan," in addition to audit discussions with the applicant, the staff understood that the applicant intends to use a widely-accepted industry standard questionnaire to assess workload. However, this workload measure was not identified by name in either the Tier 2 information or in the V&V IP.

a. Please clarify whether the named questionnaire in section 11 of the Integrated Systems Validation Test Plan is the questionnaire being used to assess workload during Integrated Systems Validation scenarios and update the application accordingly.

b. If an alternate measure is being used, please explain why it is a valid measure of workload.

8. Criteria 11.4.3.5.2 (1) states, "The applicant should specify when each measure is obtained (recorded), such as continuously, at specific points during the scenario, or after the scenario ends." In section 4.5.1.4 of the V&V IP, Rev 4, bullet 1, the applicant describes when the identified measure of workload will be obtained. However, in Section 11 of the "Integrated Systems Validation Test Plan," there is conflicting information about when the measure is obtained.

a. Please clarify when the measure will be obtained and update the application accordingly.

b. If the measure is being obtained in the manner identified in Section 4.5.1.4 of the V&V IP, Rev 4, please describe the method used to determine the timing of the data collection.

NuScale Response:

Response to Question 1:

The integrated system validation (ISV) scenarios introduce primary tasks, external tasks unrelated to the primary task, and embedded tasks to measure situational awareness. These tasks may be introduced on a single unit or distributed across several units. Subject matter expert (SME) observations are used to assess the human-system interface (HSI) effectiveness in supporting the operator's situational awareness during task performance. The following example illustrates how the three task types are used in ISV scenarios:

RO1 is responsible for oversight of the 12 reactor modules. His or her recognition of changing conditions (primary task) is a key situational awareness observation. If he or NuScale Nonproprietary

she does not recognize a changing condition, an observation would be documented.

Alarms are introduced on a single unit or on multiple units (external task). RO1 announces the alarm(s), opens the appropriate alarm procedure, and depending on workload initiates the first action (embedded task). SME observations record the effectiveness of the HSI in supporting the operator in performing the secondary tasks in parallel with the primary task. If a performance issue is identified, the cause is evaluated separate from and subsequent to the observation. As this question points out, inadequate task performance can be due to one of several causes or a combination of those causes. The cause evaluation uses the observation, operator feedback, and any other pertinent information to determine cause(s) and corrective action(s).

Generally, the ISV scenarios introduce groupings of dependent and independent tasks such as continued monitoring of alarms and notifications, surveillance management, maintenance tagouts, plant announcements, screen navigation and administrative functions. If a normal operations scenario is in progress, implementing a a surveillance requirement may contain a primary task and would be accompanied by other tasks providing the external and embedded tasks around which situational awareness observations are focused. When an event is initiated, the primary task changes. Observations then focus on the task prioritization. Again, performance challenges are documented. Subsequent cause analysis collects pertinent information to determine the cause and corrective actions.

Response to Question 2:

There are two questionnaires used to collect information from the crew. The questionnaire described in Section 4.5.1.3, bullet 2 of RP-0914-8543, Human Factors Verification and Validation Implementation Plan (V&V IP), collects information on how the HSI design supported operational activities. One question on this form asks how easy it was to maintain situational awareness of plant conditions. This question is not intended to measure situational awareness.

A second questionnaire is specifically designed to measure situational awareness. It is administered at freeze points strategically placed within scenarios to minimize intrusiveness.

V&V IP Section 4.5.1.3, bullet 2 has been revised to describe the correct form, summarize how the form is used, and to state that the questionnaire is administered during and after the scenario.

Response to Question 3:

Appendix A has been added to the V&V IP describing the characteristics for each type of performance measure.

Response to Question 4:

Appendix A has been added to the V&V IP specifying whether the measure is pass/fail, diagnostic or both.

NuScale Nonproprietary

Response to Question 5:

Appendix A has been added to the V&V IP describing the acceptance criteria bases and providing examples of acceptance criteria used for primary and secondary task performance.

Response to Question 5a:

Observers will be looking for the following general behaviors:

hesitation confusion distraction frustration These behaviors have been documented in Appendix A of the V&V IP. Each of these behaviors may be recognized by a variety of physical responses such as facial expression, voice change, task shedding, task avoidance, changes in frequency of communication, and off-script actions.

These physical responses will be noted by the SMEs who will be performing observations and have been discussed as part of observer training.

Response to Question 5b:

Tools: ((2(a),(c) No other measurement tools are needed. Checklists: Specific checklists are not used. The intent is to place full attention on the observation of the behaviors listed above. Prior to the scenario, the observer pre-job brief familiarizes the observers with the scenario and expected operator actions. A designated individual records time required for completion of tasks with time related acceptance criteria but otherwise the significance of observations is determined after the scenario is complete when individual observations can be correlated. The audio and video recordings are used to address uncertainties in the observations. Backup tools: Audio and video recordings are made to assist in verifying observations, and reconcile any perceived observer bias. Response to Question 5c: The observers have been selected based on their understanding of important aspects to observe during the ISV and how they relate to the overall goal of the ISV. They have demonstrated satisfactory application of this knowledge through observations made during the NuScale Nonproprietary

staffing plan validation, pilot testing and simulator training for the operating crew participants. Formal classroom training for observers has been completed. Its completion will be documented within the NuScale company training program. The pre-job brief is used to standardize specific knowledge about the scenario to be executed. Each scenario is described in detail in a scenario guideline which is also available to the observers. Response to Question 5d: Appendix A has been added to the V&V IP describing the acceptance criteria bases and providing examples of acceptance criteria used for primary and secondary task performance. Response to Question 6: The V&V IP is intended to describe the same situational awareness measures as the ISV Test Plan. The V&V IP Section 4.5.1.3, bullet 2 has been updated to reflect the situational awareness measures described in section 11.0 of the ISV Test Plan. Response to Question 7: An electronic version of the questionnaire illustrated in the ISV test plan is used to assess work load during the ISV. V&V IP, Section 4.5.1.4 has been revised to include the Task Load Index (TLX) as the method used to measure workload. Response to Question 8: RP-0914-8543, Section 4.5.1.4 correctly states when the identified measure of workload will be obtained. ((

                                                 }} 2(a),(c)

The conflicting information in Section 11 of the ISV test plan will be revised to be consistent with the V&V IP. Impact on DCA: RP-0914-8543, HFE Verification and Validation Implementation Plan, has been revised as described in the response above and as shown in the markup provided with this response. NuScale Nonproprietary

RP-0914-8543 Draft Revision 54 Page 2 of 75 Revision History Rev Date Description November 18, 0 Original Issue 2015 September 16, Revised to incorporate NRC comments, on Revision 0, to provide 1 2016 clarification and detail to Human Factors V&V element scope Revised to incorporate updates to the Design Verification 2 December 2, 2016 Methodology (Section 3), ISV (Section 4), and the HED process (Section 5), also added redaction markings Revised in response to eRAI 8758. Validation Team (Section 4.1) updated to include additional description on how to mitigate 3 July 24, 2017 potential test bias. Scenario Sequencing (Section 4.6.1) updated to include a basis for use of a minimum of two test crews. November 30, Revised section 4.1 to state two independent observers will be 4 2017See approval used and not one page Added Sections 3.2.3 - 3.2.5 per RAI 9398 18-21 and Section 5.3 per RAI 8758 18-2S2. Revised Section 4.2 per RAI 9414 18-23 See approval 5 and Section 4.1 per RAI 8758 18-2S2. Update to section 5.2 for page eRAI 9394 Question 18-18. RAI 9395 revised section 4.5 and added Appendix A. CP-0603-8892-F01-R5

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 4.3.4 Environmental Fidelity The test bed is representative of the actual NuScale plant with regard to environmental features such as lighting, noise, temperature, humidity, and ventilation characteristics. In cases where the test bed cannot accurately simulate the environment, the ISV captures human factors engineering issue tracking system (HFEITS) entries for evaluation and resolution. 4.3.5 Data Completeness Fidelity In the test bed, information and data provided to personnel represent the complete set of plant systems monitored and controlled from that facility. 4.3.6 Data Content Fidelity The test bed represents a high degree of data content fidelity. The alarms, controls, indications, procedures, and automation presented are based on an underlying plant model that accurately reflects the engineering design of the NuScale plant. The model also accurately provides input to the HSI, such that the information matches what is presented during operations. 4.3.7 Data Dynamics Fidelity The test bed represents a high degree of data dynamic fidelity. The plant model provides input to the HSI in a manner such that information flow and control responses occur accurately and in a correct response time. Information is provided to personnel with the same anticipated delays as would occur in the plant. 4.3.8 Remote Human-System Interfaces Containing Important Human Actions NuScale has no IHAs that are conducted outside of the MCR. In the event that a remote IHA is determined in a later design stage, the test bed uses mockups to verify human performance requirements for IHAs conducted at HSIs remote from the MCR. The simulation or mockup considers, for example, transit times, use of personal protective equipment, and delays associated with the need for operator precision (self-checking). 4.3.9 Test Bed Conformance The test bed is verified by performance testing to conform to required characteristics before validation tests are conducted. 4.3.10 ISV Simulator Performance Testing The purpose of ISV Simulator performance testing is to ensure simulator performance is sufficiently complete and accurate to meet the requirements recommended in NUREG-0711, Human Factors Engineering Program Review Model as it pertains to simulators used during ISV activities. NUREG-0711 recommends that the simulator used for ISV should have fidelity and functionality compliant with industry standard ANSI/ANS-3.5-2009, Nuclear Power Plant Simulators for Use in Operator Training and © Copyright 20187 by NuScale Power, LLC 19

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 Examination. ANSI/ANS-3.5 is intended to provide standards used to train licensed operators at an operating facility and not to conduct ISV testing. Therefore, the ISV Simulator Performance Testing uses selected criteria in a similar manner to the concept already used within the ANSI/ANS-3.5 Appendix D standard to establish criteria for part-task and limited-scope simulators. The following criteria are used to evaluate ISV Simulator performance:

• real time and repeatability testing
• limits of simulation testing
• normal evolution testing
• malfunction testing
• steady state testing Prior to the start of ISV, the ISV simulator will have completed ISV Simulator performance testing to validate overall performance. The ISV Simulator performance testing provides a comprehensive evaluation of overall simulator performance, while Scenario-Based Testing provides a detailed review of the simulator response to the individual ISV scenarios.

4.3.11 Scenario-Based Testing The testing is conducted by determining a set of key parameters to be evaluated and ensuring those parameters behave as expected for the developed ISV scenarios. ANSI/ANS-3.5-2009 was referenced for a draft list to select steady state and transient parameters. The scenarios are then conducted in real time, to ensure the completion of the objectives and termination point is reached. The procedures are executed as described in the current task analysis. The freeze feature may only be used during testing to obtain additional data and shall have no effect on the simulator parameters or resuming of the scenario for the test to be considered valid. The following criteria are used to evaluate the simulator performance while running the ISV scenarios:

        *    ((
                                                                               }}2(a),(c)

© Copyright 20187 by NuScale Power, LLC 20

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54

• experience distribution; generally industry operators have a minimum of one year of experience, while engineers have a minimum of two years experience in addition to NuScale plant systems training Operating crew size for the validation tests includes a range of expected sizes to ensure that the HSI supports operations and event management. This range includes the minimum operating crew, nominal levels, and higher levels as defined during the staffing and qualifications program element NuScale Human Factors Engineering Staffing and Qualifications Results Summary Report (Reference 8.2.3) for a range of plant operating modes. The crew size for each scenario is identified in the ISV test procedure, and scenarios are not repeated with different crew sizes.

The ISV includes at least one scenario with more than minimum crew staffing defined in Reference 8.2.3 (e.g., additional licensed operators to complete a complex evolution) to simulate times of high control-room traffic and distractions and high environmental loading. The roles of the additional personnel and their interaction with the operating crew are determined by the scenario developers based on meeting all the test objectives and goals and by applying the SOC criteria. 4.5 Performance Measurement Performance measures for ISV are hierarchical and include measures of plant performance, personnel task performance, situation awareness (SA), cognitive and physical workload, and anthropometric or physiological factors. Both pass or fail and diagnostic measures are applied. 4.5.1 Types of Performance Measures 4.5.1.1 Plant Performance Measures Plant performance resulting from operator action or inaction includes plant process data (e.g., temperature, pressure) and component status (e.g., on/off; open/closed) as a function of time at as many locations in the plant simulation as is possible. These data are obtained from the entire plant: nuclear, fluid, structural, and electrical components. Any component that provides plant process data or component status in the plant is simulated with appropriate fidelity. The test bed has the ability to record all plant process data and component status (including state changes) for the full length of any ISV scenario. 4.5.1.2 Personnel Task Performance Measures For each scenario, tasks that personnel are required to perform are identified and assessed. Primary and secondary personnel tasks are evaluated. Primary tasks are those involved with function and task completion including detection, assessment, planning, and response. The level of detail to which primary tasks are measured and performance measures selected are assessed based on the complexity of the task. It may only be necessary to measure time and accuracy for a lower level © Copyright 20187 by NuScale Power, LLC 22

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 rule-based task to recognize and respond, while tasks that are knowledge-based (e.g., detection, seeking additional data, making decisions, or taking actions) may entail the use of more detailed performance measures. Secondary task performance measures reflect the workload associated with HSI manipulations associated with maintaining the overall plant. Test personnel evaluate secondary tasks in conjunction with primary tasks to observe effects on overall performance and workload both at individual and operating crew level. Personnel task performance measurements are selected to reflect those aspects of the task that are important to system performance and used depending on the particular scenario such as

• time
• accuracy
• frequency
• amount achieved or accomplished
• consumption or quantity used
• subjective report of participants
• behavior categorization by observers

((

                                      }}2(a),(c)

Objective measures of individual or crew and system performance are also collected during validation scenarios and are used for documenting the performance and future use. They include

• video recordings of operator performance
• alarm history log

© Copyright 20187 by NuScale Power, LLC 23

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54

• operator control interactions
• plant variable control interactions (resulting from operator controls)
• component status change
• HSI use log (display screen request history and operational history)

The capturing of data using cameras enables NuScale to document the operators actions as they are performed. With the information archived, it is then available for the life of the design for tracking purposes. The comparison between actual and expected actions is an important test criterion when trying to identify errors of omission and commission. NuScale performs this comparison during the V&V testing process and will maintain a retrievable video library, as a contingency, for instances where observations conflict or actions come into question. 4.5.1.3 Situational Awareness Performance Measures To measure SA, ISV applies a combination of objective measures along with subjective post-scenario questionnaire methods. ((

                                                                                                }}2(a),(c)

((

             }}2(a),(c)

© Copyright 20187 by NuScale Power, LLC 24

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 ((

                          }}2(a),(c) 4.5.1.4 Cognitive and Physical Workload Performance Measures To measure cognitive workload, the ISV employs the NASA Task Load Index using the following methods:
        *    ((
                                             }}2(a),(c)

© Copyright 20187 by NuScale Power, LLC 25

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 Criteria Basis Meaning established using a benchmark system, e.g., a current system is predefined as acceptable. The observed performance of the integrated system is compared with a criterion Norm using many predecessor systems (rather than a single benchmark system). The observed performance of the integrated system is compared with a criterion Expert Judgment established by subject-matter experts. Performance measures are designated as pass, fail or diagnostic. Diagnostic is measureable and the criteria include both range and unit of measure. Appendix A provides specific attributes for the measures being used. With respect to each measure, the associated characteristics are rated as Hi, Med, or Low to provide a qualitative rating of how well the characteristic is achieved. An explanation for the rating is also provided. 4.6 Test Design Test design refers to the process of developing scenarios, test plans, and conducting ISV based on the integrated HSI as described in the preceding sections. The goal of test design is to permit the observation of integrated system performance while minimizing bias. Once the ISV test plan and scenarios are developed they will be reviewed by the appropriate SMEs and approved by operations management. Upon approval, the ISV scenarios and test plan will be available for review or audit by the NRC sufficiently before the conduct of ISV so that comments or concerns can be adequately addressed prior to commencing ISV. This section describes characteristics of the test design important to supporting ISV validity. 4.6.1 Scenario Sequencing Integrated System Validation: Methodology and Review Criteria, NUREG/CR-6393 (Reference 8.1.2), is employed as the standard for selection of crew and scenario order. ISV scenarios contain variations of normal operation and abnormal events and are sequenced to ensure that operating crews are not expecting events and actions at the same time during each scenario. The scenario performance sequence is developed using the following guidance:

• Equalize the opportunity for testing among all participants.
• Vary the types of scenarios within the sequence; such that all are not easy at first and then progress to hard.
• A minimum of two operating crews perform each scenario.

© Copyright 20187 by NuScale Power, LLC 28

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 HFE V&V IP Section Review Criteria No. and paragraph license type and qualifications, skill/experience, age, and general demographics. (3) In selecting personnel for participating in the tests, the applicant should consider the minimum shift staffing levels, nominal levels, and maximum levels, including shift supervisors, reactor operators, shift technical advisors, etc. (4) The applicant should prevent bias in the sample of participants by avoiding the use of participants who:

• are members of the design organization
• participated in prior evaluations
• were selected for some specific characteristic, such as crews identified as good performers or more experienced 11.4.3.5 Performance Measurement Section 4.5, all ISV employs a hierarchal set of performance measures including measures of plant performance, personnel task performance, situation awareness, cognitive workload, and anthropometric/physiological factors. Errors of omission and commission also are identified. A hierarchal set of measures provides sufficient information to validate the integrated system design and affords a basis to evaluate deficiencies in performance and thereby identify needed improvements. Pass/fail measures are those used to determine whether the design is or is not validated. Diagnostic measures are used to better understand personnel performance and to facilitate the analyses of errors and HEDs.

11.4.3.5.1 Types of Performance Measures Section 4.5.1.1, all; Appendix A (1) The applicant should identify the specific plant performance measures applicable to each ISV scenario.

• Additional Information: They may address the performance of functions, systems, or component.

© Copyright 20187 by NuScale Power, LLC 51

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 HFE V&V IP Section Review Criteria No. and paragraph (2) The applicant should identify the primary task measures applicable Section 4.5.1.2, all; to each ISV scenario. Appendix A

• For each scenario, the applicant should identify the primary tasks operators must perform to accomplish scenario goals, so that such measures can be developed.

Additional Information: The primary tasks are those involved in carrying out the functional role of the operator in supervising the plant; i.e., monitoring, detection, situation assessment, response planning, and response implementation. Primary tasks should be assessed at a level of detail appropriate to the tasks demands. For example, for some simple scenarios, measuring the time to complete a task may suffice. For complicated tasks, especially those described as knowledge-based, it may be appropriate to undertake a fine-grained analysis, such as identifying the tasks components, viz., seeking specific data, making decisions, taking actions, and obtaining feedback.

• The measures chosen to evaluate personnel task performance should reflect those aspects of the task that are important to system performance, such as:

         - time
         - accuracy
         - frequency
         - amount achieved or accomplished
         - consumption or quantity used
         - subjective reports of participants
         - behavior categorization by observers

• The analysis of primary tasks will support the identification of errors of omission (primary tasks not performed). Also, any actions and tasks that operators actually perform that deviate from the primary tasks should be identified and noted. These actions should be used to identify errors of commission.

(3) The applicant should identify the secondary task measures Section 4.5.1.2, applicable to each scenario. paragraph 3; Additional Information: Secondary tasks are those personnel must Appendix A perform when interfacing with the HSI, such as navigating through computer screens to find a needed display and to configure HSIs. The measurement of secondary task performance should reflect the demands of the detailed HSI implementation, e.g., time to configure a workstation, navigate between displays, and manipulate them (e.g., changing display type and scale settings). (4) The applicant should identify the measures of situation awareness Section 4.5.1.3, all; applicable to each scenario. Appendix A Additional Information: Situation awareness is the degree to which personnels perception of plant parameters and understanding of the plant's condition corresponds to its actual condition at any given time and influences predictions about future states. (5) The applicant should identify the workload measures obtained for Section 4.5.1, all; each scenario. Section 4.5.1.4; Additional Information: Workload is comprised of the physical, Appendix A cognitive, and other demands that tasks place on plant personnel. © Copyright 20187 by NuScale Power, LLC 52

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 HFE V&V IP Section Review Criteria No. and paragraph The impact of one or many of these aspects of workload should be considered in the performance measures. (6) The applicant should identify the anthropometric and physiological Section 4.5.1.5, all; measures obtained for each scenario. Appendix A Additional Information: Anthropometric and physiological factors include such concerns as visibility of displays, accessibility of control devices, and ease of manipulating the control device. Many of these design aspects are assessed as part of verifying the HFEs design. Therefore, attention should focus on those areas of the design that only can be addressed by testing the integrated system, e.g., the ability of personnel effectively to use the various controls, displays, workstations, or consoles while performing their tasks. 11.4.3.5.2 Performance Measure Information and Validation Section 4.5.2, all; Criteria Section 4.5.2.1; (1) The applicant should describe the methods by which these Appendix A measures are obtained, e.g., by simulator data recording, participant questionnaires, or observation by subject-matter experts. (2) The applicant should specify when each measure is obtained Section 4.5.2.2, (recorded), such as continuously, at specific points during the paragraph 3 scenario, or after the scenario ends. (3) The applicant should describe the characteristics (see Table 11-1) of Section 4.5.2.2, Table the performance measures. 4-1; Appendix A Table 11-1 Characteristics of Performance Measures Characteristic Meaning Construct Validity A measure should represent accurately the aspect of performance it is intended to measure. Reliability A measure should be repeatable; i.e., same behavior measured in exactly the same way under identical circumstances should yield the same results. Sensitivity A measure's range (scale) and its frequency (how often data are collected) should be appropriate to that aspect of performance being assessed. Unobtrusiveness A measure should minimally alter the psychological or physical processes that are being investigated. Objectivity A measure should be based on easily observed phenomena. (4) The applicant should identify the specific criterion for each measure Section 4.5.2.2, Table used to judge the acceptability of performance and describe its 4-2; Appendix A basis. Additional Information: Table 11-2 describes the different bases for performance criteria. Table 11-2 Basis for Performance Criteria Criteria Basis Meaning Requirement The observed performance of the integrated system is compared with a quantified performance requirement; i.e., the requirements for the performance of systems, subsystems, and personnel are defined through engineering analyses. © Copyright 20187 by NuScale Power, LLC 53

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 HFE V&V IP Section Review Criteria No. and paragraph Benchmark The observed performance of the integrated system is compared with a criterion established using a benchmark system, e.g., a current system is predefined as acceptable. Norm The observed performance of the integrated system is compared with a criterion using many predecessor systems (rather than a single benchmark system). Expert The observed performance of the integrated system is Judgment compared with a criterion established by subject-matter experts. (5) The applicant should identify whether each measure is a pass/fail Section 4.5.2.2, final one or a diagnostic one. paragraph; Appendix A 11.4.3.6 Test Design Section 4.6 11.4.3.6.1 Scenario Sequencing Section 4.6.1, all (1) The applicant should balance scenarios across crews to provide each crew with a similar, representative range of scenarios. Additional Information: Random assignment of scenarios to crews for ISV is undesirable. The value of using random assignment to control bias is effective only when the number of crews is quite large. (2) The applicant should balance the order of presentation of scenarios Section 4.6.2, bullet 1 to crews to provide reasonable assurance that the scenarios are not always presented in the same sequence (e.g., the easy scenario is not always used first). © Copyright 20187 by NuScale Power, LLC 54

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 ((

                                                                                                 }}2(a),(c)
  © Copyright 20187 by NuScale Power, LLC 63

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 ((

                                                                                               }}2(a),(c)
  © Copyright 20187 by NuScale Power, LLC 64

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 ((

                                                                                              }}2(a),(c)
  © Copyright 20187 by NuScale Power, LLC 65

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 ((

                                                                                               }}2(a),(c)
  © Copyright 20187 by NuScale Power, LLC 66

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 ((

                                                                                              }}2(a),(c)
  © Copyright 20187 by NuScale Power, LLC 67

Human Factors Verification and Validation Implementation Plan RP-0914-8543-NP Draft Rev. 54 ((

                                                                                             }}2(a),(c)
  © Copyright 20187 by NuScale Power, LLC 68

RAIO-0718-60922 : Affidavit of Zackary W. Rad, AF-0718-60925 NuScale Power, LLC 1100 NE Circle Blvd., Suite 200 Corvalis, Oregon 97330, Office: 541.360.0500, Fax: 541.207.3928 www.nuscalepower.com

NuScale Power, LLC AFFIDAVIT of Zackary W. Rad I, Zackary W. Rad, state as follows:

1. I am the Director, Regulatory Affairs of NuScale Power, LLC (NuScale), and as such, I have been specifically delegated the function of reviewing the information described in this Affidavit that NuScale seeks to have withheld from public disclosure, and am authorized to apply for its withholding on behalf of NuScale.

2. I am knowledgeable of the criteria and procedures used by NuScale in designating information as a trade secret, privileged, or as confidential commercial or financial information. This request to withhold information from public disclosure is driven by one or more of the following:

a. The information requested to be withheld reveals distinguishing aspects of a process (or component, structure, tool, method, etc.) whose use by NuScale competitors, without a license from NuScale, would constitute a competitive economic disadvantage to NuScale.

b. The information requested to be withheld consists of supporting data, including test data, relative to a process (or component, structure, tool, method, etc.), and the application of the data secures a competitive economic advantage, as described more fully in paragraph 3 of this Affidavit.

c. Use by a competitor of the information requested to be withheld would reduce the competitor's expenditure of resources, or improve its competitive position, in the design, manufacture, shipment, installation, assurance of quality, or licensing of a similar product.

d. The information requested to be withheld reveals cost or price information, production capabilities, budget levels, or commercial strategies of NuScale.

e. The information requested to be withheld consists of patentable ideas.

3. Public disclosure of the information sought to be withheld is likely to cause substantial harm to NuScale's competitive position and foreclose or reduce the availability of profit-making opportunities. The accompanying Request for Additional Information response reveals distinguishing aspects about the method by which NuScale develops its human factors engineering.

NuScale has performed significant research and evaluation to develop a basis for this method and has invested significant resources, including the expenditure of a considerable sum of money. The precise financial value of the information is difficult to quantify, but it is a key element of the design basis for a NuScale plant and, therefore, has substantial value to NuScale. If the information were disclosed to the public, NuScale's competitors would have access to the information without purchasing the right to use it or having been required to undertake a similar expenditure of resources. Such disclosure would constitute a misappropriation of NuScale's intellectual property, and would deprive NuScale of the opportunity to exercise its competitive advantage to seek an adequate return on its investment. AF-0718-60925

4. The information sought to be withheld is in the enclosed response to NRC Request for Additional Information No. 482, eRAI 9395. The enclosure contains the designation "Proprietary" at the top of each page containing proprietary information. The information considered by NuScale to be proprietary is identified within double braces, "(( }}" in the document.

5. The basis for proposing that the information be withheld is that NuScale treats the information as a trade secret, privileged, or as confidential commercial or financial information. NuScale relies upon the exemption from disclosure set forth in the Freedom of Information Act ("FOIA"), 5 USC § 552(b)(4), as well as exemptions applicable to the NRC under 10 CFR §§ 2.390(a)(4) and 9.17(a)(4).

6. Pursuant to the provisions set forth in 10 CFR § 2.390(b)(4), the following is provided for consideration by the Commission in determining whether the information sought to be withheld from public disclosure should be withheld:

a. The information sought to be withheld is owned and has been held in confidence by NuScale.

b. The information is of a sort customarily held in confidence by NuScale and, to the best of my knowledge and belief, consistently has been held in confidence by NuScale.

The procedure for approval of external release of such information typically requires review by the staff manager, project manager, chief technology officer or other equivalent authority, or the manager of the cognizant marketing function (or his delegate), for technical content, competitive effect, and determination of the accuracy of the proprietary designation. Disclosures outside NuScale are limited to regulatory bodies, customers and potential customers and their agents, suppliers, licensees, and others with a legitimate need for the information, and then only in accordance with appropriate regulatory provisions or contractual agreements to maintain confidentiality.

c. The information is being transmitted to and received by the NRC in confidence.

d. No public disclosure of the information has been made, and it is not available in public sources. All disclosures to third parties, including any required transmittals to NRC, have been made, or must be made, pursuant to regulatory provisions or contractual agreements that provide for maintenance of the information in confidence.

e. Public disclosure of the information is likely to cause substantial harm to the competitive position of NuScale, taking into account the value of the information to NuScale, the amount of effort and money expended by NuScale in developing the information, and the difficulty others would have in acquiring or duplicating the information. The information sought to be withheld is part of NuScale's technology that provides NuScale with a competitive advantage over other firms in the industry.

NuScale has invested significant human and financial capital in developing this technology and NuScale believes it would be difficult for others to duplicate the technology without access to the information sought to be withheld. I declare under penalty of perjury that the foregoing is true and correct. Executed on July 20, 2018. Zackary W. Rad AF-0718-60925}}