ML20107C742

Non-proprietary Man-In-The-Loop Test Plan Description

ML20107C742
Person / Time
Site:	05200003
Issue date:	04/08/1996
From:	Kerch S WESTINGHOUSE ELECTRIC COMPANY, DIV OF CBS CORP.
To:
Shared Package
ML20107C738	List: ML20107C740 ML20107C742
References
WCAP-14396, WCAP-14396-R, WCAP-14396-R00, NUDOCS 9604170518
Download: ML20107C742 (36)

Text

WESTINGHOUSE PaorsuETADY CLASS 3 l

WCAP.14396 Revision 0 l

l l

MAN-IN-THE-LOOP TEST PLAN DESCRIPTION April 1996 S. Kerch E. Roth R.Mumaw AP600 Document No. OCS-T5-001 WESTINGHOUSE ELECTRIC CORPORATION Energy Systems Business Urit Nuclear Technology Division P.O. Box 355 Pittsburgh, Pennsylvania 15230-0355 i 01996 Westinghouse Electric Corporation l All Rights Reserved i

i

' 9604170518 960412 PDR ADOCK 05200003 l A PDR u:\2158w.wp;1b-0409%

. . - . . . - . . . . - - - ~ . . . - - -. .-. ~ ~ . - - - . - - --. . - - . . . .

AP600 DOCUMENT COVER SHEET TDC: IDS: 1 S Form 58202G(5/94)[t\xxxx.wpt:1x)

AP600 CENTRAL FILE USE ONLY:

0058.FRM RFS#: RFS ITEM #:

AP600 DOCUMENT NO. REVISION NO. ASSIGNED TO AP600 Doc No. OCS TS-001 C Page 1 of.3L. Enter Assigned To ALTERNATE DOCUMENT NUMGER: WCAP 14396, Rev. O WORK BREAKDOWN #: WBS 3.3.2.4.15 DESIGN AGENT ORGANIZATION: Westinghouse Electric TITLE: Man-in-The-Loop Test Plan Description ATTACHMENTS: DCP #/REV. INCORPORATED IN THIS DOCUMENT  !

REVISION:

CALCULATION / ANALYSIS

REFERENCE:

ELECTRONIC FILENAME ELECTRONIC FILE FORMAT ELECTRONIC FILE DESCRIPTION l u:\2158w.wpf:1b-040496 Word Perfect l

l l

l (C) WESTINGHOUSE ELECTRIC CORPORATION 1996 O WESTINGHOUSE PROPRIETARY CLASS 2 This document contains informahon propnetary to Westinghouse Electric Corporation: it is submrlted in confidence and is to be used solely for the purpose for which it is fumished and retumed upon request. This document and such information is not to be reproduced, transmitted, disclosed l or used otherwise in whole or in part without prior written authorization of Westnghouse Electric Corporation, Energy Systems Business Unit, l' sutyct to the legends contained hereof.

O WESTINGHOUSE PROPRIETARY CLASS 20 . i This document is the property of and contains Proprietary information owned by Westinghouse Electric Corporation and/or its subcontractors and  !

suppliers, it is transmitted to you in confidence and trust, and you agree to treat this document in strict accordance with me terms and condebors of the agreement under which it was provided to you.

@ WESTINGHOUSE CLASS 3 (NON PROPRIETARY)

! COMPLETE 1 IF WORK PERFORMED UNDER DESIGN CERTIFICATION OR, COMPLETE 2 IF WORK PERFORMED UNDER FOAKE. j 1 @ DOE DESIGN CERTIFICATION PROGRAM - GOVERNMENT LIMITED RIGHTS STATEMENT [See page 2)

Copyright statement A license is reserved to the U.S. Govemment under contract DE-ACO3-90SF18495.

l' l

DOE CONTRACT DELIVERABLES (DELIVERED DATA)

@ Subject to specified excephons, disclosure of this data is restricted until September 30,1995 or Design 90SF18495, whichever is later.

. EPRI CONFIDENTIAL: NOTICE: 1 3 20 3 4 5 CATEGORY: A 3 B C D E F0 1 2 0 ARC FOAKE PROGRAM - ARC LIMITED RIGHTS STATEMENT [See page 2)

Copyright statement A license is reserved to the U.S. Govemment under contract DE-FCO2-NE34267 and subcontract ARC-93-3-SC-001.

0 ARC CONTRACT DELIVERABLES (CONTRACT DATA)

Subject to t,pecified exceptions, disclosure of this data is restricted under ARC Subcontract ARC-93-3-SC-001.

l i ORIGINATOR SIGNATU E/DA

.DJ i

AP600 RESPONSIBLE M#. NAGER SIG '

APPROV DA E 0'

• Y f

• Approval of the responsible manager sigrufies that document is complete, all requered reviews are complete, electronic file is attached and document is j released for use.

i

AP600 DOCUMENT COVER SHEET Page 2 Form 58202G(5/94) LIMITED RIGHTS STATEMENTS DOE GOVERNMENT UANTED RIGHTS STATEMENT (A) These data re sutmtted with lirnited rights under government contract No. DE-AC03-90SF18495. These data may be re and ,

used by the govemment with the express limitaNon that they will not, without written permission of the contractor, be for purposes of manufacturer nor esclosed outside the gcucJ a, except that the govemment may esciose these data outside the government for the fonowing purposes, if any, provided that the govemment makes such disclosure subject to prohitWtion against further use and declosure.

(l) This

• Proprietary Data

• may be disclosed for evaluabon purposes under the restrictions above. *

(II) The ' Proprietary Data

• may be rear

• wad to the Electne Power Research Ins #tute (EPRI), electric utility representaNves and their direct consultants, excluding direct commercial competitors, and the DOE National Laboratories under the prohitHtions and I restrictons above. l (B) This notice shaN be marked on any reproduchon of these data, in whole or in part.

l 1

ARC UINTED RIGHTS STATERIENT:

This propnetary data, fumiehed under Subcontract Number ARC-93-3-SC-001 with ARC may be dupbcated and used by the govemment and  ;

ARC, subsect to the limitatons of Arbcie H-17.F. of 91st subcontract, with the express limitations that the propr6etary data may not be ' Ear 4amad  ;

outside the govemment or ARC, or ARC's Class 1 & 3 members or EPRI or be used for purposes of manufacture without prior permenion of  !

the Subcontractor, except that further dieciosure or use may be made solely for :he foNowing purposes This W ry data may be,dieciosed to other than commercial compettors of Subcontractor for evaluation purposes of this subcontract under the restricton that the propnetary data be retained in conRdence and not be further declosed, and subject to the terms of a non-disclosure agreement between the Subcontractor and that organization, excluding DOE and its contractors DEFINITIONS CONTRACT /DEUVERED DATA - Consists of documents (e.g. specifications, drawings, reports) which are e generated under the DOE or ARC contracts which contain no background proprietary data.

EPRI CONFIDENTIALITY / OBLIGATION NOTICES NOTICE 1: The data in this document is subject to no conRdenhality obhgebons NOTICE 2: The data in this document is proprietary and confidental to Westmghouse Electric Corporaton and/or its Contractors. It is forwarded to recipient under an obligabon of Confidence and Trust for limited purposes only. Any use, #6ar4aaive to unauthorized reons, or copying of this document or parts thereof is prohibited except as agreed to in advance by the Electric Power Research institute (E 1 and W ,-

Electric Corporabon Recipient of this data has a duty to inquire of EPRI and/or Westmghouse as to the uses of tie informat)on herein that are permitted-NOTICE 3: The data in this document is proprietary and conndental to Weennghouse Elecinc tion and/or its Contractors. It is forwarded to recipent under an obligabon of ConRdence and Trust for use only wt evaluston tasks authorized the Electric Power Research -

insutute (EPRI). Any use, daar*=me to unauthorized persons, or copying this document or parts thereof is xcept as agreed to in advance by EPRI and Westinghouse Electnc Corporanon. Recipient of tus data has a duty to inquire of EP 1 and/or Wesenghouse as to the uses of the information contained herein that are permitted. This document and copies or excerpts thereof that may have been generated are to be retumed to Wesunghouse, dirocoy or through EPRI, when requested to so.

NOTICE 4: The data in this document is proprietary and confidental to Weebnghouse Electric Corporation and/or its Contractors. It is being revealed in confidence and trust only to Emplo Any use, disclosure to unauthorized persons,yees or copying ofofthis EPRI and toorcertain document contractors parts thereof of EPRI is prohibited. Thisfor limited evaluston Document and any copies tasksor authorized b excerpts thereof that may have been generated are to be retumed to Westnghouse, direcoy or through EPRI, when requested to do so.

NOTICE 5: The data in this document is proprietary and confidential to Westmghouse Electric Corporation and/or its Contractors. Access to this data is g to unauthonzed iven in Confidence persons, or copying and of Trust only at Westmghouse this document or parts thereoffacNites for hmited is prohitMted Neitherevaluation tasks this document norassigned by EPRI.

any excerpts Any use, therefrom are todisclosure be removed from Westinghouse facihtes EPRI CONFIDENTIALITY / OBLIGATION CATEGORIES CATEGORY "A"-(See Delivered Data) Consists of CONTRACTOR Foreground Data that is contained in an issued reported. .'

CATEGORY "B" - (See Delivered Data) Consists of CONTRACTOR Foreground Data that is not contained in an issued report, except for computer programs.

CATEGORY "C"- Consists of CONTRACTOR Background Data except for computer programs.

CATEGORY "D"- Consets of computer programs developed in the course of performing the Work.

CATEGORY *E"- Consists of computer programs developed prior to the Effective Date or after the Effeceve Date but outside the scope of the Work. I CATEGORY "F"- Consists of administrative plans and administrative reports.

)

l

)

_ , _ . . ~ _ . _ _ _ _

TABLE OF CONTENTS

, Section Title P_ age l

1.0 SCOPE AND PURPOSE OF DOCUMENT l-1 1 2.0 CONCEPT TESTING AS PART OF HUMAN FACTORS ENGINEERING / 2-1 M-MIS DESIGN PROCESS I 2.1 Role of Concept Testing in M-MIS Design Process 2-1 2.2 Overview of Concept Tests to be Performed 2-1 2.3 Testbeds for Concept Testing 2-2 2.4 Test Participants 2-3 2.5 Number of Participants 2-3 3.0 FORMAL V&V OF FINAL M-MIS DESIGN 3-1 3.1 Crew Sampling 3-2 3.2 Scenario Sampling 3-2 4.0 MAN-IN-THE-LOOP CONCEPT TEST DESCRIPTIONS 4-1 i 4.1 Soft Controls 4-1 Concept Test 1. Simple control action execution and feedback j

- Preliminary design concepts 4-1 i Concept Test 2. Simple control action execution and feedback

- AP600 Functional Design 4-3 ,

Concept Test 3. Keeping pace with plant dynamics 4-4 4.2 Workstation Displays 4-5 Concept Test 4. Ability to navigate displays to find information 45 Concept Test 5. Coordination of physical and functional displays 4-6 4.3 Computerized Procedures 4-7 Concept Test 6. Usability of computerized procedures 4-7 Concept Test 7. Coordination of computerized procedures wdh workstation displays and soft controls 4-8 4.4 Wall Panel Information System 4-10 Concept Test 8. WPIS support for situation assessment 4-10 Concept Test 9. Ability of WPIS to support multiple crew member situation awareness 4-11 4.5 Alann System 4-12 Concept Test 10. Alarm system organization and prioritization scheme 4-12

5.0 REFERENCES

5-1 l

l l uA2158w.wpf:Iba04% iii

LIST OF TABLES Table Title h ,  !

l 2-1 List of Concept Tests 2-5 2-2 Mapping of Concept Tests to SSAR Evaluation Issues 2-6 4-1 AP600 Concept Test Schedule 4 34 l

l t

i 1

4 l

uA2158w.wpf:lb-0404% jy

i l

LIST OF FIGURES

• EM .ILG1 g 2-1 The Role of Concept Testing in the M-MIS Design Process 2-7 1

I l

l l

I I

i l

l l

l l

t f

a:\2158w.wpf:IM)404% y l -

i l

l 1.0 SCOPE AND PURPOSE OF DOCUMENT

, This document describes the Man-in-the-Loop tests that are planned as part of the AP600 man-machine interface system (M-MIS) design process. It is inter ded as an adjunct to the M-MIS Design Verification and Validation (V&V) Process provided in C1 apter 18 (Section 18.8.2.3) of the l Standard Safety Analysis Report (SSAR). l This docunent provides information supporting preparation for and scheduling of the Man-in-the-Loop tests to be conducted as part of the M-MIS design process. The document:

• Specifies the set of the Man-in-the-Loop tests that are to be conducted as part of the AP600 M-MIS design process

• Provides methodological details supporting preparation for conducting tests, including:

- Number and type of participants that will be required for different tests  !

- Hardware / software requirements for prototyping the M-MIS resources to be tested, and collecting human performance data

- M-MIS resources that will need to be designed, and specific displays that will need to be l

created to support testing.

In addition, this document clarifies which of the set of 17 Evaluation Issues specified in Chapter 18 of the SSAR are to be addressed as part of the M-MIS design process and which are to be conducted as part of the formal human factors V&V to be performed on the final AP600 M-MIS design.

This document does not provide detailed test procedures for the Man-in-the-Loop tests. At this stage in the AP600 M-MIS design process, it is not possible to develop complete and detailed specifications for all tests to be performed, specifically because the AP600 M-MIS has not been fully designed and test details will depend on the M-MIS design and the operational philosophy developed for how the M-MIS resources are to be used. Detailed test procedures will be prepared for each test when the test is to be conducted.

This is intended to be a living document. Details of the Man-in-the-Loop tests are expected to be updated as the design evolves and feedback from Man-in-the-Loop tests are obtained. l Revision 0 provides a revised description of the concept tests planned as part of the AP600 FOAKE l i program. The planned tests have been revised based on additional information about two testbeds i anticipated to be available for implementing and testing AP600 M-MIS design concepts: (1)a

low-fidelity testbed that is being implemented in the AP600 Main Control Room Test Facility at the l Energy Center, and (2) a higher-fidelity, dynamic testbed at the Waltz Mill site.

uA2158w.wpf:Ib-o404% l-1

The AP600 Main Control Room Test Facility established at the Westinghouse Energy Center was used to conduct the Concept Test 1 (Soft Control test). Refer to " Effects of Control Leg and Interaction Mode on Operator's Use of Soft Controls," OCS-J1-008, for the report on the results of this test. ,

Several M-MIS resources included in the AP600 control room are relatively matum designs that are in the process of being implemented as retrofits in conventional pressurized water reactor (PWR) plants.

Examples of mature M-MIS resources include computerized procedures, the alarm system, and workstation display system. These M-MIS resources are being installed in a high-fidelity training simulator at the Westinghouse Waltz Mill site under a separate program funded by a utility customer for acceptance testing of the M-MIS resources, and requalification training for their operator crews.

The M-MIS resources should be implemented in a high-fidelity training simulator at Waltz Mill by June 1996. We propose to use this facility to conduct Man-in-the-Loop tests of the M-MIS resources under dynamic plant simulation-driven conditions.

l l

uT.158w.wpf:!b-0404% l-2

2.0 CONCEPT TESTING AS PART OF HUMAN FACTORS ENGINEERING /M-MIS DESIGN PROCESS 2.1 Role of Concept Testing in M MIS Design Process Figure 2-1 shows the phases in the M-MIS design process and the role that concept testing plays in the design process. Concept testing is performed as part of the functional design phase of the M-MIS design process. It is during the functional design phase that the core conceptual design for an M-MIS resource and corresponding functional requirements are developed. An integral part of this phase is rapid prototyping and testing of design concepts.

Concept testing during the functional design phase serves two purposes:

1) It provides input to aid designers in resolving design issues for which there is no well-established human factors guidance. An example might be assessing whether a system response time is adequate to support operator performance on a time-critical task. A second example might be aiding a designer in selecting between altemative design concepts that have different strengths and weaknesses. An example might be determining whether it is preferable to include soft controls in information displays or dedicate a separate video display unit (VDU) for control stations.

2) It establishes the adequacy of the design concept andfunctional requirements that are produced in thefunctional design stage. A main objective of concept testing is to establish that the conceptual design resulting from the functional design stage is adequate to support operator performance in the range of situations anticipated to arise. A major element in designing a concept test is to specify test scenarios representative of the range of situations and task complexities that can arise in actual events, ensuring that the M-MIS conceptual design is adequate to support operator performance in those complex cases.

2.2 Overview of Concept Tests to be Performed The starting point for the concept tests to be performed is the Human Factors V&V process description l

provided in Chapter 18 of the SSAR where 17 Evaluation Issues are defined. The first 15 Evaluation Issues are organized around particular human performance issues. Issues 16 and 17 correspond to formal verification and validation of the final M-MIS design, respectively.

The concept tests described as part of Evaluation Issues 1-15 in the SSAR constitute the basis for the Man-in-the-Loop tests that are proposed as part of the M-MIS design process. These tests are consistent with rapid prototyping and testing activities described under Element 7 of the proposed NRC Human Factors Engineering Program Review Model for Advanced Nuclear Power Plants l (O'Hara, Higgins & Stubler,1994). In Chapter 18 of the SSAR these concept tests are organized i around human performance issues ensuring that all major human performance issues are covered. The present document takes the same performance issues to be evaluated, and reorganizes them around u \2158w.wpf:ll>O404% 2-1

specific M-MIS resources. This was done to enable concept testing to begin early in the M-MIS design process before all M MIS resources have been fully designed and integrated.

• \

The M-MIS resources addressed in these Man-in-the-Loop tests are: l

• Soft controls

• Workstation displays

• Computerized procedures

• Wall panel information system (WPIS)

• Alarm system Human performance issues addressed in Evaluation Issues 1-15 of SSAR Chapter 18 were reviewed and reorganized around the specific M-MIS elements associated with those issues. Based on this analysis, Man-in-the-Loop concept tests were defined that could be conducted to examine human performance issues associated with individual M-MIS resources, and Man-in-the-Loop concept tests that examined the coordination of two or more M-MIS resources. The goal was to de5ne Man-in-the-Loop tests that were narrowly focused, enabling early testing of M-MIS design concepts.

Table 2-1 lists the 10 concept tests proposed as part of the AP600 FOAKE program. Table 2-2 shows the correspondence between the concept tests and evaluation issues presented in Chapter 18 of the SSAR. The majority of evaluation issues identified in Chapter 18 of the SSAR are addressed in one or more of the 10 concept tests proposed. The exceptions are Evaluation Issues 7-10 and 15. These evaluation issues examine the impact of all the M-MIS resources on crew performance using a high-fidelity dynamic plant simulation. These issues are beyond the scope of this revision to the test plan,

• but will be included in a future revision. ,

2.3 Testbeds for Concept Testing The M-MIS resources included in the AP600 control room vary in their design maturity levels. De alarm system, computerized procedures, and workstation displays, and much of the functional design activity has already been completed as part of earlier Westinghouse projects. Concrete examples of these M-MIS resources exist for conventional pressurized water reactor plants. Other M-MIS i resources to be included in the AP600, specifically the WPIS and soft controls, have not been previously implemented in Westinghouse Nuclear Power Plant (NPP) control room designs. Dese M-MIS resources are in an earlier stage of the functional design process. Early concept testing will focus on the less mature M-MIS resources, since this is the area where input from concept testing can ,

1 be most beneficial.

An AP600 Main Control Room (MCR) Test Facility, established at the Westinghouse Energy Center, included capabilities for rapid prototyping of M-MIS design concepts. This facility enabled execution of the first soft control test (Concept Test 1) under low-fidelity conditions. It is currently planned to move the AP600 MCR Test Facility to the Waltz Mill site (April 1996) to take advantage of the high-fidelity SNUPPS/Beznau phnt simulator located at that site.

u:\2158w.wpf:lb-o404% 2-2

Versions of the advanced alarm system and workstation display system are being installed in the high-fidelity SNUPPS/Beznau training simulator at Waltz Mill. This facility is being developed under a

, separate program funded by a utility customer for the purposes of acceptance testing of the M-MIS resources, and requalification training for their operator crews. Current plans are to have the M-MIS resources implemented in the simulator at Waltz Mill by early June 1996. This high-fidelity simulation facility will enable Westinghouse to conduct M-MIS msource tests under dynamic plant simulation-driven conditions.

While the application being implemented at Waltz Mill is a conventional PWR plant rather than a passive plant design, key elements important to human performance (e.g., characteristics of the M-MIS, plant dynamics, operational philosophy) are sufficiently similar in the two types of plants that the results of concept tests performed at the Waltz Mill simulator facility will be valid for the AP600 plant.

2.4 Test Participants In Human Factors literature, a distinction between two types of evaluations is made. There are:

1. Heuristic Evaluations - where experts in M-MIS design, human factors, and operations review prototypes that illustrate conceptual design

2. User Tests - whem individuals from the target user population (i.e., actual NPP control room -

1 operators) are observed as they try to use a prototype under controlled conditions Several human factors studies have been conducted examining the relative value of Heuristic Evaluations and User Tests (Jeffries. Miller, Wharton, & Uyeda,1991; Jeffries & Desurvire,1992). i Generally these techniques are complementary, and a test plan combining both evaluations are advocated. Heuristic Evaluations identify serious problems early. User Tests are more definitive in l establishing design adequacy.

l In the AP600 design, the distinction between Heuristic Evaluation and User Tests is obscured because no one has operating experience on AP600 plants. There is a proposal to use Westinghouse personnel from the training, procedures development, and M-MIS design groups who have NPP operational experience as test participants in the early concept tests. In later concept tests, operators from existing NPP will be used, since they bring a complementary experience base and perspective.

2.5 Number of Participants An issue surrounding the number of participants that should be included in each concept test has arisen. Recently, research has explicitly examined the proportion of usability problems detected as a function of number of users tested or heuristic evaluators employed (Virzi,1992; Nielson & Landauer, 1993). A general finding is that the relationship between the number of test participants and proportion of usability problems identified is well modeled as a Poisson process.

uA2158w.wpf:Ib-0404% 2-3

These studies suggest that five participants in an evaluation or test typically is sufficient to detect most (70 to 90 percent) of the serious M-MIS problems (Virzi,1992). This is particularly true with an iterative evaluation approach where repeated evaluations are conducted until the number of problems identified is very low.  !

As an initial estimate, there will be at least five participants in each concept test. In later Man-in-the-Loop evaluations, the number of participants in a study may be adjusted to achieve a specific level of thoroughness in detecting problems that may exist.

The math model parameters for the best-fit Poisson curve specific to AP600 Man-in-the-Loop testing will be better estimated when proceeding with the evaluations. Once these parameters are estimated, it is possible to specify a priori the number of participants that need to be included in a Man-in-the-l Loop test to identify a soecific percentage of usability problems that exist in an M-MIS design (Nielson & Landauer,1993). For example, the number of participants that need to be included in the ,

Man-in-the-Loop test in order to catch 95 percent of the usability problems could be specified.

I 1

l l

l I

i u:\2158w.wpf;1b-0404% 2-4

1 l TABLE 2-1 l' LIST OF CONCEPT TESTS 1

Soft Controls '

l t

Concept Test 1: Simple control action execution and feedback (Preliminary design concepts)  ;

Completed September 1994 Concept Test 2: Simple control action execution and feedback (Prototype based on functional l requirements document) l l Concept Test 3: Keeping pace with plant dynamics Workstation DisDlaVS l 1

Concept Test 4: Navigating displays to find information '

Concept Test 5: Coordinating physical and functional displays  !

l l t Computerized Procedures '

l Concept Test 6: Usability of computerized procedures 1

l Concept Test 7: Coordinating procedures, displays, and soft controls '

l l

Wall Panel Information System 1

Concept Test 8: Support for situation assessment (single person)

Concept Test 9: Support for multiple crew member performance Alarm System l Concept Test 10: Alarm organization and prioritization scheme l

[

l i

I l

l uA2155w.wpf:Ib&O4% 2-5 1

l l

TABLE 2-2 MAPPING OF CONCEPT TESTS TO SSAR EVALUATION ISSUES Concept Test l Soft Workstation Computerized Alarm -

l l SSAR Evaluation Issues Controls Displays Procedures WPIS System i

1. Passive Monitoring Test 8

2. Directed Search Test 8

(WPIS)

3. Directed Search Test 4 (Request)

4. Group Situation Test 9 l

l Assessmeat l

5. Alarm System Test 10

6. Interpretation and Test 5 Planning

7. Response to Single-Fault Events

8. Response to Multiple-l l Fault Events l

9. Interpretation and Planning - Crew

10. Interpretation and l

Planning - with TSC i

11. Simple, Operator- Tests 1,2 Tests 6,7 l Paced Control

12. Conditional Operator- Test 2 Tests 6,7 Paced Control i

l

13. Multiple Procedures Tests 6,7

14. Event-Placed Control Test ', Tests 6,7

15. Control Tasks with Crew Coordination 5

mi2158w.wpf.lb 0404% 2-6

t C

=

5 4

C l

l

• 3 s.!

l

El M-MIS M-MIS

" Functional I='de-tation  %;

M-MIS Function-Based +

l y a Design + Task Analysis Design + (Hardware Software Integra$on Basis Integration) o z

o ir Ik o

• Resolve design issues Design concep s 3

o Establish adequacy of

$

• Functional design concep and li! requirements y functional requirements s g E.

E ir F Concept Test x Man-in-the-loop test of concrete example g of functional design. ,

(;; t

• Rapid pretygm

.E.

• Actual M-MIS for similar plant h

i .

s l

_< _ - _ ~

3.0 FORMAL V&V OF FINAL M MIS DESIGN Evaluations 16 and 17 in Chapter 18 of the SSAR represent the formal HFE/M MIS design V&V.

They.will be performed when an AP600 plant has been purchased. The tests will be conducted using an AP600 dynamic, high-fidelity, training simulator.

The formal V&V evaluations are not covered in this document; however, a high-level description of how the Man-in-the-Loop Validation Test (Evaluation Issue 17 in the SSAR) will be conducted clarifying the proposed approach is provided.

l As the evaluations proceed, a better estimate will be made regarding the math model parameters for )

the best-fit Poisson curve specific to AP600 Man-in-the-Loop testing. Once these parameters are l estimated it is possible to specify a priori to the number of participants needed in a Man-in-the-Loop test to identify a specific percentage of usability problems that exist in an M-MIS design (Nielson &

Imnhr,1993). For example, one could specify the number of participants that need to be included in the Man-in-the-Loop Test to catch 95 percent of the usability problems.

The purpose of the final V&V test is to establish that crews can achieve mission goals using the AP600 M-MIS, and that no serious human factor deficiencies remain. 'Ihe methodological approach proposed is analogous to the methodology currently used to validate emergency operating procedures (EOPs) (cf. NUREG-0899; NUREG-1358; WCAPI1638; WCAP10599; INPO 83-006).

The current process for validating EOPs employs a formal process for identifying, tracking, and resolving potential problems. An interdisciplinary observation team, which includes specialists in plant design, operations, procedures, and human factors, is brought together to observe operator crews go through pre-dermed scenarios on a high-fidelity training simulator. The scenarios are selected to exercise different procedure branches and include a variety of complicated situations. The observation team documents any problems identified on discrepancy forms. These discrepancy forms are then reviewed, and recommendations for resolving the problem are made.

Westinghouse proposes to use an analogous approach for final validation of the AP600 M-MIS.

Operator crews will be observed as they go through a pre-defined set of scenarios on an AP600-specific, high-fidelity training simulator. An interdisciplinary observation team will observe  ;

crew performance and document any problems observed. Since the AP600 M-MIS design will have undergone a series of tests prior to final validation, many problems should not be detected in the fmal

. validation test. Any problems that are detected will be evaluated for severity and recommendations for resolving the problem will be made.

i 1

u:\2158w.wpf:Ib-0404% 3-1

. - . . - . - . . - ..- - - . - _. . . . . - - . ~ - - - .-

l l

l To ensure a thorough evaluation, an emphasis will be placed on:

l

[

• A representative sample of crews ,

l A representative sample of scenarios that reflects events known to be important to maintain l '

plant safety, and events known to be demanding from a human factors / cognitive perspective l

• Use of redundant and convergent measures l

l More details on the performance measures included in the final validations are provided in Chapter 18 of the SSAR.

l l 3.1 Crew Sampling 1

l l Participants in the final validation test will be the control room operators and related staff from the first AP600 plant to be built. All individuals undergoing training to operate the AP600 plant will I' participate in the validation study. This will include individuals who will be training to be plant control room operators and the management and administrative staff who will be receiving training for l licensing purposes. Inclusion of all personnel licensed to run the AP600 plant in the validation test i should ensure a representative sample that includes a wide range of experience and skill levels.

l Final validation will be conducted when personnel will be undergoing training on the AP600 plant and its operation.

! 3.2 Scenario Sampling A multi-dimensional set of criteria will be used to define the set of scenarios included in the final l validation ensuring a wide sampling of situations. The dimensions considered include:

1

• A range of operational modes including normal and emergency operations

! = Design-basis events specific to AP600

- AP600-specific design features

= Scenarios including human performance actions identified as critical to plant safety a Situations that produce cognitive challenges identified in the V&V process description in the SSAR. including AP600-specific scenarios that may:

'.' - Complicate situation assessment by providing degraded or conflicting plant state information i

i a:\2158w.wpf:lb-04o4% 3-2

.____ ___.m . - . _ _ _ _ _ . _ . . - ~ _ _ _ . __ _._ _ _ _ _ . _ . - _ _

- Complicate response (e.g., require balancing of multiple goals; require manual takeover of automatic systems)

- Complicate performance by increasing personnel communication / coordination requirements

- Introduce high physical or mental workload While it is premature to define the set of scenarios that will be included in the final AP600 validation, a preliminary list of the types of scenarios that need to be considered has been developed based on  ;

discussions with engineers and training instructors familiar with the APb00 design:

1) Major plant transitions - Major transitions, such as significant heatup or cooldown transitions, cover multiple plant modes and require transitions for several types of equipment. These are i typically high-workload operations that also have significant surveillance requirements. l l

l l 2) Reactor startup - This component of plant startup is one of the most difficult and sensitive

. operations. The AP600 needs to be able to demonstrate its effectiveness here.

3) Power production system changes - During power operations, a critical role for the control room is to make transitions between power levels (e.g., transition to full power). In addition to the traditional reasons why this maneuver is important, the AP600 design introduces gray rods for certain power transitions.

4) Mode I high-power operations - In Mode 1, an important maneuver is the load-follow evolutions.

The AP600 relies on gray rods and boron control for this set of operations.

5) Maintenance support at full power (tag-out) - Tag-out is an important element of operation.

AP600 validation needs to show that testing and maintenance are well supported, and that there are adequate ties to the Technical Specification document.

6) Shutdown operations - The control room has a number of important duties during shutdown. One j element of operations important in the AP600 is the transition from RCS intact to RCS open (mid-loop).

7) Accidents and emergency operations - Emergency operations are always an important part of V&V. Certainty within a range of events is important. AP600 design issues indicate that the l

following events may be appropriate to include in the final validation:

. Loss of main feedwater where PRHR is required

• Sequence involving Core Makeup Tank (CMT) actuation j = Steam Generator Tube Rupture (SGTR)

!

• Very small LOCA with CMT recirculation l I&C failures of PMS (crossover to DAS/ DIS)

• Shutdown events -

uA2158w.wpf;tb4404% 3-3

i The actual set of scenarios included in the final validation will be defined by an interdisciplinary team, j including input from AP600 design engineers, EOP developers, training instmetors, M-MIS designers, human factors specialists, and human reliability analysis /PRA analysts, ,

l l

i i

1 I

l 1

l l

i I

l l

l l

l t

I l

l l I

i l

l u:\2158w.wpf:lb-0404% 3-4

l l

l 4.0 MAN IN-THE-LOOP CONCEPT TEST DESCRIPTIONS {

, I k, Below are descriptions of concept tests to be conducted as part of the M-MIS design process. The descriptions are organized around specific M-MIS resources. l 1

~

Each concept test includes a description of the issue (s) to be evaluated, the test method to be employed, testbed characteristics required to run the test, type of materials required, performance i f measures to be recorded, and expected test outcome (i.e., types of conclusions and recommendations expected).

ll 1

! The evaluation issue (s) specified ir3 the SSAR that are addressed by the concept test are presented in l parentheses under the issues heading. j

\

These test descriptions are not intended to replace detailed test procedures. A detailed test procedure 1 l

should be prepared for each test close to the time that the test is to be conducted.

5 The current schedule for performing the tests, and the testbed planned to be used for each test is i j presented in Table 4-1.

1 1

i 4.1 Son Controls l Concent Test 1: Simple control action execution and feedback - Preliminary design concepts. This test was completed September 1994.

j Issue: 'Ihe ability for the soft controls provided in the workstation to adequately support the operator j in execution of simple, operator-paced, control actions. (Evaluation Issue 11)

The initial test evaluated preliminary soft control design concepts that draw on current soft-control designs and soft control capabilities of the Process Control Division (PCD) grrzgl.iu scftware.

The concept test assessed the ability for the soft controls to support operators in selecting and bringing up a control station, taking the control action and receiving feedback with respect to whether the control action was actuated as intended, whether the control action had the desired effect on plant state, and where the control was not actuated as intended, the reason why.

Specific issues include:

I

= Ability of the soft control design to support a range of control activities that:

- Arise in discrete and continuous control actions

- Require simultaneous control of several controls

- Require sequential execution of controls

- Require monitoring automated systems and taking manual control as necessary.

uA2158w.wpf:1b-o404% 4-1

• Ability of operator to:

- Evaluate current control state, target, and limit values ,

- Determine possible and desired control action

- Enable effective control when there are time lags

- Determine when action cannot be performed because desired conditions are not satisfied (such as interlocks or limits on rates)

- Obtain and interpret feedback on actuation of control action

- Identify and recover from errors Test Method: The participant was asked to perform simple, operator-paced control actions. For example, the participant was asked to open a valve, increase flow to some target value, perform a simple proceduralized sequence, and inonitor the activity of an automated system and take manual control if necessary. The participant was required to bring up the appropriate control (s), take the control action (s), indicate whether the control actions were actuated, and whether the desired plant state was achieved.

Testbed Characteristics: This test was performed in the AP600 Main Control Room Test Facility, located in the Westinghouse Energy Center. Conceptual designs were developed as workstation-based prototypes. The set of displays and controls were dynamic in functionality (activating soft controls resulted in appropriate changes in the displays) but were not tied to a high-fidelity plant simulation.

Rudimentary math models that enabled changes to occur in displayed plant parameters based on control actions were used to drive the displays and soft controls. The math model used to drive the display allond specification of different functional relationships between control actions and their eLet on plant parameters (linear relation, logarithmic relation) and different lags to mimic different types of controls.

Type of Materials: A set of control scenarios were defined that exemplified types of control activities expected to arise in the AP600 plant, (such as discrete control tasks, and continuous control tasks).

Performance Measures: To assess that performance issues' response times were recorded, sequences of actions were documented, and subjective ratings were obtained. Subjective ratings gave subjective feedback on design adequacy.

Outcome: Test results provided data to support development of the functional requirements for soft controls for AP600 o A2158w.wpf:1b-0404% 4-2

s Concept Test 2: Simple control action execution and feedback - AP600 Functional Design his test is similar to Concept Test 1, but is performed later in the functional design process. The test will be conducted using prototype displays that are representative of the AP600 soft controls functional design. This test establishes the adequacy of the AP600 soft control functional design and functional requirements for simple self-paced control tasks.

Issue: The types of issues to be addressed will be similar to the issues addressed in Concept Test 1.

The specific issues addressed will depend on the results of Concept Test 1, and the set of design issues that arise during the AP600 soft control functional design process for which Man-in-the-Loop concept  :

test data is desired. ne test will address the issues described in Evaluation Issues 11 and 12.

Test Method: The participant will be asked to perform simple, operator-paced control actions. The ,

participant will be required to bring up appropriate control (s), take the control action (s), indicate whether the control actions were actuated and whether the desired plant state was achieved, and, in cases where the control action failed, indicate why it failed.

Testbed Characteristics: This test will be performed in the AP600 Main Control Room Test Facility.

He set of displays and controls will be dynamic in functionality (i.e., activating soft controls will result in appropriate changes in the displays) but need not be tied to a high-fidelity plant simulation.

Type of Materials: A set of control scenarios will be defined tnat exemplify the different types of control activities that are expected to arise in the AP600 plant (such as discrete control tasks, continuous control tasks, and tasks that involve simultaneous control of multiple controls). These j scenarios will include cases where the control action is blocked (due to interlocks, limits on rates, and equipment malfunction).

Performance Measures: To assess performance issues identified above, response times will be recorded, sequences of actions will be documented, and sut>jective ratings obtained. Subjective ratings allow for subject feedback on design adequacy.

Outcome: Test results will establish the adequacy of the functional requirements for soft controls for AP600 simple control tasks.

Concept Test 3: Keeping pace with plant dynamics l 1

Issues: Soft controls supporting the operator in executing control actions and evaluating feedback in pace with the event. (Evaluation Issue 14)

Workstation Physical and Functional Displays should support the operator in locating relevant displays and executing soft controls at a rate that allows operators to keep pace with the event. (Evaluation Issue 14) i uA2158w.wpf:lb r404% 4-3 i

This evaluation determines whether the soft control design that comes out of the first evaluation supports performance in a more realistic setting where operators are required to keep pace with a plant evolution. ,

Test Method: The participants will be asked to perfonn several control evolutions involving both manual and supervisory control of automated systems. Examples will include cases where the plant dynamics are rapid and cases where the plant dynamics are slow. It will include cases where the control maneuvers go as planned, cases where components fail, and cases where automated systems fail and manual takeover is required.

Testbed Characteristics: This test will be performed in the high-fidelity simulator at Waltz Mill. A workstation-based prototype that is tied to a plant simulation will be used, allowing simulation of several common plant evolutions. For example a plant staitup or a change in power level.

Type of Materials: The prototype will present process and control displays built around the plant evolution scenarios identified. .

Performance Measures: To assess each user's ability in soft controls efficiency, without error, and maintaining pace with the simulated event, the sequence of control actions and the time required for each control action will be recorded. To ensure that the participant was able to control the plant evolution, key plant simulation parameters will also be recorded (for example, key plant parameters stayed within target range and limits were avoided). Performance will be videotaped. In addition, subjective judgments will be elicited to identify and rate the severity of problems and to assess subjective workload.

Outcome: This test will assess whether the AP600 soft-control functional design is adequate to support operators in keeping pace with dynamic plant evolutions.

4.2 Workstation Displays Concept Test 4: Ability to navigate displays, finding information Issues:

• Operator accuracy and efficiency, when given a specific request in locating and selecting the correct workstation display where that information is located (Evaluation Issue 3) .

• Effective operator use of zoom and pan capabilities to locate information

• The coding scheme should indicate active display points where more information is available (e.g., through zoom and pan, or through transitions to another display)

• The coding scheme should be adequate to locate information within a display n:\2158w.wpf:4-0404% 4-4

l Test Method: 'lhe participant will be asked to locate a particular piece of information requiring navigation through the displays. Specific information requests will be defined to exercise various available navigation mechanisms.

Testbed Characteristics: This test will be performed in the high-fidelity simulator at Waltz Mill.

Type of Materials: The test will be performed using physical displays and possibly functional displays for the plant simulated in the high-fidelity simulator at Waltz Mill. The displays, while not AP600-specific, will be representative of the AP600 Functional Requirements for workstation displays.

Performance Measures:

To assess the performance issues, the following will be recorded:

• Whether the information is located

. Response times

• The navigation path taken to get to the target display The navigation path length to

• target information will be compared to the optimal navigation path identified a priori by the desper.

In addition, subjective judgments will be elicited from the participants regarding the adequacy of display coding conventions and navigation mechanisms.

Outcome: This test will assess the adequacy of the AP600 Functional Requirements for display coding and navigation.

Concept Test 5: Coordination of physical and functional displays Issue: This test addresses the use and coordination of physical and functional displays supporting situation awareness and response planning (Evaluation Issue 6), including the types of information expected to be drawn from physical and functional displays, when these displays should be accessed, and how physical and functional displays are to be coordinated.

Specific issues include whether the workstation physical and functional displays support the operator

. in:

• Distinguishing situations where physical displays should be examined from situations where functional displays should be examined

. Understanding interrelationships among systems and processes

• Assessing whether currently active processes are performing correctly l uA2158w.wpf:ltWo4% 45

• Assessing whether automated systems are performing correctly ,

l

= Assessing goal satisfaction i

• Identifying implications of plant state for plant goals

. Assessing the availability of alternative processes for achieving a given goal

• Making choices among alternative means for achieving a given goal (recovery path)

• Assessing the effect of the selected recovery path on other plant goals (side effects)

Test Method: Participants will be presented with several scenarios that require forming a situation j assessment and determining a course of action based on examination of physical and functional displays. They will be asked to assess plant state, determine implications for plant goals, determine a j course of action, and assess a selected course of action on other plant goals (detennine whether there l are any side effects).

Testbed Characteristics: This test will be performed in the high-fidelity simulator at Waltz Mill.

Type of Materials: The test will be performed using physical and functional displays for the plant simulated in the high-fidelity simulator at Waltz Mill. The displays, while not AP600-specific, will be representative of the AP600 Functional Requirements for workstation displays.

Performance Measures: The main measure of performance will be accuracy of response to questions related to situation assessment and response determination and evaluation. The displays that are accessed and the order in which they are accessed will also be recorded and examined to detennine whether the participants accessed the physical and functional displays that contained relevant  ;

information. Subjective judgments will also be elicited from participants regarding the adequacy of the physical and functional displays for supporting plant-state interpretation and planning.

Outcome: This test will assess the adequacy of the AP600 Functional Requirements for the use and coordination of physical and functional displays.

4.3 Computerized Procedures Concept Test 6: Usability of computerized procedures Issue: The computerized procedure design should be adequate to support operators in performing tasks that involve following procedures.

uA2158w.wpf:Ib-o4o4% 4-6

Current Westinghouse emergency operating procedures (EOPs) are paper-based and employ a two-column format. Operating experience with current procedures has revealed a number of situations that are difficult for operators to handle and can lead to error, including:

Complex logical statements and branching found in the response not obtained (RNO) column  ;

l

• Pending procedure steps Nested procedures, in which a separate procedure must be completed before returning to complete the current procedure a Multiple, independent procedures that must be executed in parallel (for example, when a procedure attachment is handed to one control room operator to perform, while the rest of the crew continues with the main procedure)

It is important to determine how effectively compaterized procedures handle these difficult situations.

It is also important to determine whether computerized procedures introduce new difficulties not found in current paper-based procedures. 1 The current Westinghouse M-MIS design for computer-based procedures offers a new medium with new support features, including support for procedure steps and foldout pages that require continuous  ;

monitoring, place-keeping aids, and automatic assessment of whether plant-state requirements specified .

in the procedure step are met. These support features should facilitate operator performance and I reduce potential for error. However, user tests are required, insuring that no new problems are ,

introduced by the change in technology. 'Ihe test will determine the impact of having a narrower field of view, the isnpact of display navigation requirements on ability to look ahead and look back in the procedure, and the fact that the computer automatically determines whether a procedure step is met, ,

resulting in a more passive review role for the operator.

This study addresses the types of issues raised in Evaluation Issues 11,12,13, and 14. These issues address:

• How the M-MIS features support the operator in performing control tasks that require assessment of preconditions, side-effects, and post-conditions (Evaluation Issue 12)

• How the design of the procedure display interfaces prevent operators from getting lost in nested procedures (Evaluation Issue 13)

• How the design of procedure display interfaces supports the concurrent use of multiple, independent (not nested) procedures (such as, specific alignment procedures that appear as EOP attachments and are normally handed to the Reactor Operator or Balance of Plant Operator to accomplish) (Evaluation Issue 13) u A2'.58w.wpf:lt>0404% 4-7

How operators are able to perform the next steps (suspend a step) and return to complete the pending step at the appropriate time, in a case where event dynamics are slow (Evaluation Issue 14)

Test Method: Participants will be observed as they respond to several simulated emergency events j using computerized procedures. The test objectives will be met by observing utility crews using

• computerized procedures in the high-fidelity simulator at Waltz Mill as part of acceptance testing of the computerized procedure or as part of crew requalification training.

Testbed Characteristics: This test will be performed in the high-fidelity simulator at Waltz Mill. j Type of Materials: The test will be performed using computerized procedures for the plant simulated ,

in the high-fidelity simulator at Waltz Mill. The computerized procedure, while not AP600-specific, will be representative of the AP600 Functional Requirements for computerized procedures. I Performance Measures: How operators follow procedure steps and keep up with the simulated event

)

will be assessed.

Outcome: The test will assess the adequacy of the AP600 Functional Requirements for computerized procedures.  ;

I Concept Test 7: Coordination of computerized procedures with workstation displays and soft controls Issues: ,

l The workstation displays and soft controls should be coordinated with procedures to allow efficient location and ecution of control actions. (Evaluation Issues 11 and 13)

The M-MIS features should support the operator in performing control tasks that require assessment of preconditions, side-effects, and post-conditions. (Evaluation Issue 12)  ;

• Procedures should be coordinated with workstation physical and functional displays, allowing the operator to keep pace with the event. (Evaluation Issue 14)

• Workstation physical and functional displays and procedures should allow the operator to perform the next steps and return to complete the pending step at the appropriate time in cases where event dynamics are slow. (Evaluation Issue 14)

• The soft controls should support the operator in executing control actions and evaluating feedback in pace with the event. (Evaluation Issue 14) u A2158w.wpf:Ib-o4N% 4-8

i Test Method: Participants will be asked to perform several simulated tasks using computerized procedures in combination with functional and physical displays and soft control mechanisms. They will be required to take control actions using soft controls.

Testbed Characteristics: his test will be performed in the high-fidelity simulator at Waltz Mill.

The plant simulation will be modified so that control actions can be executed using soft controls.

Type of Materials: The test will be performed t' sing displays and computerized procedures for the plant simulated in the high-fidelity simulator at Waltz Mill. The displays and compu'erized procedures, while not AP600-specific, will be representative of the AP600 Functional Requirements for workstation displays, soft controls, and computerized procedures. Several scenarios will be defined that exemplify the types of system monitoring and control executions that are performed using these procedures. The scenarios should include cases where plant dynamics are rapid and extensive manual action is required, cases where plant dynamics are slow, and cases where supervisory control of j automated systems and manual takeover from automated systems is required. Ideally, there should be I overlap between the scenarios included in Concept Tests 3 and 7.

Performance Measures:

To assess performance issues identified above, the following will be recorded:

. Whether the right display is brought up, correct information located, and correct control action taken

• Response times

• Ability to detect and correct execution errors l

In addition to these objective measures of performance, subjective judgments will be obtained from participants regarding M-MIS design adequacy.

Outcome: His test will assess the adequacy of coordination between workstation displays and soft i control mechanisms and procedures.

i 4.4 Wall Panel Information System (WPIS) l Concept Test 8: WPIS support for situation assessment Issue: Completeness

• He WPIS display should present sufficient information for the operator to maintain awareness of the plant state during diffemnt modes of operation. (Evaluation Issue 1) ruissw.wpr:ib.040496 4-9

i

• The WPIS should suppon the operator in getting more detail about plant status and system l

! availability by directed search of the workstation functional and physical displays. (Evaluation Issue 2) .

• Initial concern will be with the completeness of the information presented on the WPIS. Also, this test will determine if the WPIS modes are appropriate. -l l

Test Method: Participants will view a WPIS display and be asked to make a series of judgments or decisions regarding current plant state, system availability, and on the conducting of a shift turnover.

Testbed Characteristics: Conceptual WPIS designs will be developed as workstation-based j prototypes. Testing actual hardware is not a concern at this point. This test may be conducted either in the low-fidelity simulation in the AP600 Main Control Room Test Facility or the high-fidelity simulator at Waltz Mill.

Type of Materials: WPIS displays will be developed to show plant state for the full range of plant modes to be addressed by the WPIS For each mode represented, a range of plant states will be presented.

Performance Measures: The ability of participants to accurately determine plant state based on i information in the WPIS will be assessed. In addition to objective measures of performance, subjective judgments will also be elicited from participants regarding the adequacy of the WPIS design concept. ,

Outcome: 'Ihis test will assess the adequacy of the AP600 WPIS functional design, supporting single individuals in maintaining plant situation awareness. f Concept Test 9: Ability of WPIS to support multiple crew member situation awareness issue: The WPIS should suppon multiple member crew awareness of plant conditions (Evaluation  ;

Issue 4).

Specifically:

.c

• The M MIS should support the crew in maintaining awareness of plant conditions and those implications. .

• The M-MIS should support the crew in maintaining awareness of each others' actions, intents, and information needs.

• The M-MIS should support effective and efficient shift turnover.

u:\2158w.wpf:Ib-0404% 4-10

• The M-MIS should support new personnel entering the control room to develop an awareness of plant conditions and their implications.

1 It is critical that the WPIS supports operators in maintaining awareness of the plant state as they operate the plant. It is also critical to show that the WPIS supports operators in maintaining awareness of each other's activities. Evidence of changes in operator behavior motivated by the information presented on the WPIS should be noted. These behavior changes will indicate that the WPIS is communicating effectively in that critical information is salient and timely.

Test Method: The WPIS prototype will be located where it affords easy access for a multi-person crew. For each scenario, the WPIS will indicate an important event that should alter crew operations.

The ability of the crew to assess plant state and perform maneuvers that require shared plant knowledge will be assessed.

Testbed Characteristics: This test will be performed in the high-fidelity simulator at Waltz Mill.

The simulator will be modified so that the WPIS will be driven by dynamic, simulated plant data I Type of Materials: The test will be performed using a WPIS for the plant simulated in the high- i fidelity simulator at Waltz Mill. While the scenarios employed will not be AP600-specific, the WPIS will be representative of the AP600 Functional Requirements for a WPIS.

Performance Measures: Objective measures of the ability of crews to assess plant state and to perform maneuvers that require shared knowledge of plant state will be obtained, in addition, subjective judgments will be obtained from participants regarding the adequacy of the WPIS functional design.

Outcome: This test will assess the adequacy of the AP600 WPIS functional design, supporting group situation awareness.

4.5 Alarm System Concept Test 10: Alann System organization and prioritization scheme Issue: The Alarm System should organize alarm messages so that it facilitates operator understanding of the alarmed state and its implications for the plant operational goals. (Evaluation Issue 5)

Specific issues include:

I

• Alarm System presentation format should enable rapid detection and interpretation of alarm j messages. (Evaluation Issue 5)

• Alarm System prioritization scheme should facilitate operator understanding of the relative j importance of alarm conditions. (Evaluation Issue 5) j e

u:\2158w.wpf:1b 04o496 4-1I

l

• - Alarm System prioritization scheme, including the management of redundant or low importance alarm messages should facilitate operator understanding of the relative importance i of alarm conditions. (72 valuation Issue 5) ,

j

• Alarm System should enable operators to identify and interpret the implications of lower l priority alarms. (Evaluation Issue 5)  !

l Test Method: Participants will be observed as they respond to the Alarm System during simulated emergencies. The participant will be asked to indicate the alarm messages presented and their priority for response from most to least important. The participant will also be asked to describe the implications of the alarms to plant safety and productivity goals and any causal interrelationship among alarms. In addition, the participant will be asked to identify other alarm conditions that may have existed but were not displayed on the main alarm panel. Following this, the display of additional alarms will be presented, and the subject will be asked to describe the implications of these alarms. ,

Testbed Characteristics: This test will be performed in the high-fidelity simulator at Waltz Mill.

Type of Materials: The test will be performed using scenarios for the plant simulated in the high- j fidelity simulator at Waltz Mill. The Alarm System, while not AP600-specific, will be representative  ;

of the AP600 Alarm System Functional Requirements. A set of plant upset scenarios will be defined that vary in severity from single malfunction events to multiple failure events. Alarm messages will  ;

vary in number and level of abstraction. Upsets will include cases where a single fault leads to a cascade of alarms (where the objective is to determine whether the participant can correctly assess the interrelation between the original fault and the consequent disturbances), multiple fault cases (where the objective is to determine the subject's ability to identify, prioritize and track the implications of multiple functionally unrelated alarms), and where additional alarm queues of varying types (redundant versus low priority alarms) and number exist.

Performance Measures: The objective of this test is to identify if operators are able to correctly interpret and assess the implications of alarm messages. This is assessed through objective performance measures and the participant's subjective assessments.

I Objective dependent measures will be collected, including the:

• Number of alarms correctly identified

• Participant's assessment of alarm priorities compared to the priorities assigned during the development of the test scenario e Extent to which the alarm implications for present and future plant state are correctly assessed 1 i

• Extent to which the causal interrelationship among alarms is recognized l

i u:\2158w.wpf;1b-oeo4% 4 12 l

1

Ability to infer alarms not displayed on the main alarm panel

• Ability to interpret alarms In addition, subjective judgments will be elicited from participants regarding the adequacy of the Alarm System Functional Design.

Outcome: This test will assess the adequacy of the AP600 Alarm System Functional Design.

l l

J l

l

. l u:\2158w.wpf.Ib-0404% 4 13 1

I

TABLE 4-1 AP600 CONCEIT TEST SCHEDULE Schedule Concept Test 1996 1997 1998 Test Bed Soft Control Test 1: Preliminary design concept - AP600 Main Control Room Test simple control tasks Facility (Completed 9/94)

Test 2: Prototype based on x AP600 Main Control Room Test functional requirements Facility l Test 3: Keeping pace with plant x High-Fidelity Simulator at Waltz dynamics Mill (WM)

Workstation Displays  :

Test 4: Navigating displays to find x High-Fidelity Simulator at WM information j Test 5: Coord. of physical & x High-Fidelity Simulator at WM tunctional displays Computerized Procedures

• Test 6: Usability of computerized x High-Fidelity Simulator at WM procedures Test 7: Coord. of procedures, x High-Fidelity Simulator at WM displays & soft controls l Wall Panel Information System (WPIS) x AP600 Main Control Room Test Test 8: Support for situation Facility awareness (single person)

Test 9: Support for multiple crew x High-Fidelity Simulator at WM member performance Alarm System

• Test 10: Alarm organization & x High-Fidelity Simulator at WM prioritization scheme  ;

i

• Objectives of this test will be met by observing utility crews utilizing the M-MIS resource during utility acceptance testing and/or crew requalification training currently scheduled for late 1995.

l uA2158w.wpf:lb-0404% 4 14

1 I

l i

i

5.0 REFERENCES

i Ekctric Power Research Institute, Chapter 10 " Man-Machine Interface Systems," Advanced Light Water Reactor Utility Requirements Document. Volume III ALWR Passive Plant, Palo Alto, Ca., Revision 4, Dec.1992.

Hoecker, D. G. & Roth, E. M., " Effects of Control Leg on Operator's Use of Soft Controls," l OCS-J1-008, September 23,1994.

Jeffries, R., Miller, J., Whanon, C. & Ulyeda, K., " User Interface Evaluation in the Real World: A Comparison of Four Techniques," Proceedings of CHI,1991 (New Orleans, Louisiana, April 28-May 2,1991) ACM, New York,1991. pp.119-124.

Jeffries, R. & Desurvire, H. " Usability Testing Versus Heuristic Evaluation: Was There A Contest?,"

SIGCHI Bulletin, 24(4), October 1992, 39-41.

Kantowitz, B. H. " Selecting Measures for Human Factors Research" Human Factors, 34 (4),

1992,387-398.

Nielsen, J. & Landauer, T. "A Mathematicai Model of the Finding of Usability Problems," INTER CHI

'93 Conference Proceedings (Amsterdam, The Netherlands, April 1993), ACM,1993.

pp. 206-213.

NUREG-0899 Guidelines for the Preparation of Emergency Operating Procedures, Division of Human Factors Safety, Office of Nuclear Reactor Regulation, US Nuclear Regulator Commission, August 1982.

l NUREG- 1358 Lessons Learnedfrom the SpecialInspection Program for Emergency Operating Procedures, Office of Nuclear Reactor Regulation, US Nuclear Regulatory Commission,  !

April,1989.

O'Hara, J. M., Higgins, J. C., & Stubler, W. F. Human Factors Engineering Program Review Model for Advanced Nuclear Power Plants, draft technical report prepared for the US Nuclear Regulatory Commission, January 25,1994.

Westinghouse, AP600 Standard Safety Analysis Report, " Human Factors Engineering," Chapter 18.

DE-AC03-90SF18495, June,1992.

Virzi, R A. " Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?,"

Human Factors, 34(4), 1992,457-468.

u:\2158w.wpf.lb-(Mo4% 5-1 l