NUREG/CR-5411, Forwards Draft NUREG/CR-5411, Elicitation & Use of Expert Judgement in Performance Assessment for High Level Waste Repositories, for Review by 891107

Forwards Draft NUREG/CR-5411, Elicitation & Use of Expert Judgement in Performance Assessment for High Level Waste Repositories, for Review by 891107

ML19324A988
Person / Time
Issue date:	10/30/1989
From:	Ballard R NRC OFFICE OF NUCLEAR MATERIAL SAFETY & SAFEGUARDS (NMSS)
To:	Mckenna E Office of Nuclear Reactor Regulation
References
CON-FIN-A-1165, RTR-NUREG-CR-5411 NUDOCS 8910300118
Download: ML19324A988 (91)

Text

D.

~

DCT 3 01999 k

PBR00KS/10/24/89 1

t l

MEMORANDUM FOR:

Eileen McKenna, Acting Branch Chief Policy Development and Technical Support Branch Program Management Policy Development andAnalysisStaff,NRR FROM:

Ronald L. Ballard, Chief Geosciences & Systems Performance Branch Division of High. Level Waste Management, NHSS

SUBJECT:

REQUEST FOR REVIEW OF SANDIA REPORT ON EXPERT JUDGMENT

{g Sandia National Laboratories has submitted for review by NRC staff a-draft report entitled "E11 citation and Use of Expert Judgment in Performance i

l Assessment for High. Level Waste Repositories." prepared under contract FIN A.1165. Since there has been similar work done in relation to MUREG-1150, especially by Leon Reiter, we are requesting review by HRR staff as well as our own staff.

A. copy of the report is enclosed.

Review comments at'e needed by November 7, 1989. We appreciate your cooperation in this review.

Ronald L. Ballard, Chief Geosciences & Systems Performance Branch Division'of High. Level Waste Management, HMSS As stated cc: Leon Reiter, NRR DISTRIBUTION:

Central Files REBrowning, DHLWM BJYoungblood, DHLWM RLBallard, HLGP 00 Bunting, HLEN JLinehan, HLPM HLGP r/f HMSS r/f PDR & LPDR CNWRA W

ACNW PBrooks, HLGP SCoplan, HLGP 9910300118 091030 PDR WMRES EXISANL A-1165 FDC UIC : HLGP HLGP HL Pd /

' 1///./

j

.Hu 1 NAME :PBrooks/

SC

RLBa1$rd

NH-1 t' i

.....:............:............:............:............:............:...l;.,!!b!,

u.../ f.I 10/2%89

%W5 g p [> g g

/

10/37/89 DATE : 10/aff/89

[w 6..

j 1

l NUREGCR 5411 SAND 891821 i

EUCITAT10N AND USE OF EXPERT JUDGMENT IN PERFORMANCE ASSESSMENT j

FOR HIGH LEVEL RADIOACnVE WASTE REPOSITORIES l

Evaristo J. Bonano i

Stephen C. Horal RaIph L Keene 2 Detlof von Winterfeldt 2

J July 1989 l

i 3

Sandia National Laboratories i

Albuquerque,NM 87185 Operated by Sandia Corporation

. for the I

U.S. Department of Energy Prepared for Division of High 12 vel Waste Management i

Office of Nuclear Material Safety and Safeguards U.S. Nuclear Regulatory Commission 1

i Washington,DC 20555 NRC FIN A1165 l

i 1 University of Hawaii, Hilo, MI 2 University of Southern Califomia,I.os Angeles, CA 1

~

1 1

~

}

i ABSTRACT This report presents the concept of formalizing the clicitation and use of expert judgment in the performance assessment of high level radioactive waste repositories

}

i m deep geologic formations. The report outlines aspects of wrformance assessment in which the clicitation and use of expert judgment should x. formalized, discusses existing techniques for formalizing the ebcitation and use of expert judgment, and presents guidelines for applying these techniques in performance assessment.

l t

p t

1 1

e i

4 l

i r-iy-

CO -

EA&R 1

1.

INTRODUCTION 2

1.1 Objective of this Report 1.2 Expert Judgment in Performance Aucument of HLW 2

Repositories 1.3 Characteristics of a Formalized Expert Judgment Process 2

1.4 Previous Formal Uses of Expert Judgments in HLW Program 4

1.5 When to Use Expert Judgment 7

1.6 Relationship of Formal Use of Expert Judgment to Informal 7

Use, Modelms, and Data Collection 2.

AREAS IN NEED OF FORMAL EXPERT JUDGMENT IN 9

HLW DISPOSAL 9

2.1 Scenario Development and Screening 2.1.1 Identification of Events and Procenes 10 2.1.2 Claulfication of Events end Procenes 10 2.1.3 Screening of Events and Processes 10 2.1.4 Formulation of Scenarios 11 2.1.5 Screenin of Scenarios 11 2.1.6 Probabil of Occurrence il 12 2.2 Model Development 2.2.1 Data Selection and Interpretation 12 12 2.2.2 Development of Conceptual Models 13 2.2.3 Confidence Building 14 2.3 Parameter Estimation 2.3.1 Identification of Important Parameters 14 2.3.2 Quantificati:m of Uncertainty in Parameters 15 i

16 2.4 Information Gathering 16 t

2.5 Strategic Repository Decisions 3.

EUC!TATION, USE, AND COMMUNICAT10N OF 18 EXPERT JUDGMENTS 18 3.1 Definitions 20 3.2 The Process of Eliciting Expert Judgments

'iii-

+

1 t

CON'ITNT3 (Continued)

I i

EAST j

3.2.1 Identification of Issues and Infonnation Needs 20 3.2.2 Selection of Experts 21 j

3.2.2.1 Selection of Generalists 22 3.2.2.2 Selection of Spcialists 22 j

3.2.2.3 Selection of b ormative Experts 24

]

3.2.3 Training 24 I

28 3.2.4 Conducting Elicitation Sessions 3.2.4.1 Basic Elicitation Arrangements 29

{

3.2.4.2 Structure of a Standard Elicitation Session 30 3.2.43 Post Elicitation Activities 31 33 Techniques for Expert Judgment Elicitation 31

]

33.1 Identification Techniques 32

]

3.3.1.1 Techniques for Event and Scenario Identification 32 l

33.1.2 Identification of Conceptual Models 34

{

3.3.2 ScreerJngTechniques -

34

_l t

33.2.1 Setting Target 1.svels or Constraints 35 3.3.2.2 Selection 36 l

333 Decomposition Techniques 36 j

333.1 Decomposition of Factual Problems 36 3.33.2 Decomposition of Value Problems 38 33.3.3 Variants of Decomposition 39 l

l 3.3.3.4 Benefits and Costs of Decompositions 40 l

f 33.4 Techniques for Quantifying Probability Judgments 40 33.4.1 Ma itude Judgments about Discrete Events 42 3.3.4.2 Ma nitude Judgments about Continuous Uncertain Quantities 42 1

t 33.43 Fractile Technique 43 33.4.4 Interval Technique 44 t

i l

33.4.5 Indifference Judgments Between Gambles l.

with Discrete Events 44 l

.iv.

[

i i

CONTENT!i(Continued)

M l

3.3.4.6 Indifference Judaments among Gambles with Continuous' Uncertain Quantities 45 j

3.3.5 Techniques for Quantifying Value Judgments 45 l

t 3.3.5.1 Simple Multiattribute Rating Technique 46 3.3.5.2 Indifference Technique for Measurable i

Value Functions 47 33.5.3 Aggregation Steps 48 3.4 Combining Expert Judgments 48 3.4.1 Combinin Lists 48 i

3.4.2 Combinin Probabili Judgments 49 3.43 Combin Value J ments 49 l

3.4.4 Behavior vs. Analyti i Combination 49 i

3.5 Communicating Expert Judgments 51 l

3.5.1 Documentation

$1 I

3.5.2 Presentation of Results 53 3.6 Interpretation,Use, and Misuse'of Expert Judgments 54 i

4.

SUGGESTIONS FOR THE USE OF EXPERT JUDGMENT IN HLW DISPOSAL 56 l

4.1 Scenario Development and Screening 56 4.1.1 Identification and Classification of Events and Processes 57 i

4.1.2 Screening of Events and Processes 57 4.13 Generation of Scenarios 58 58 4.1.4 Screening of Scenarios 4.1.5 Probability of Scenarios 59

-i 59 4.2 Model Development 4.2.1 Data Selection and Interpretation 59 4.2.2 -

Development of Conceptual Models 60 4.23 Confidence Building 61 43 Parameter Estimation 63

.y.

_ _. _.., _ _ _ _ _ _, _ _. _ _ - _ ~ _ _ _ _ _ _ _ _ _ _ _ _. _ _ _ _ _. _. _ _ _ _ _ _ _ _ _. _ _ _ _ _ _

1 CON' TENTS (Concluded)

=

4J.1 Identification of Parameters 63 4.3.1.1 Guidelines for Parameter Identification 63 4.3.2 Quantification of Parameters 64 4.3.2.1 Guidelines for Quantifying Parameters 64 4.4 Information Gathering 66 4.4.1 Informational Drilling 66 4.4.2 Selecting Models to Develop 68 4.4.3 1.aboratory and Field Expenments 69 4.5 Strategic Repository Decisions 71 4.5.1 Specifying and Structuring Objectives 71 4.5.2 Identification of Alternatives 72 4.5.3 Imlmets of Alternatives 72 4.5.4 Va ue Judgments 72 4.5.5 Analysis of the. Alternatives

.73 i

4.5.6 Documentation of Analysis 73

~

REFERENCES 75 9

+

vi.

.m

-4

+

LIST OF TABLES I

l t

Iatst b

i 3.1 Taxonomy of Probability Elicitation Techniques 41 l

[

i 9

I i

i r

i f

f

+

e r

I e

I a

vil-i

p.,

01AF(

FOREWORD 1

This report presents the concept of formalizing the clicitation and use of expert jud,; ment in the performance assessment of high level radioactive waste repositories m c eep geologic formations. The report outlines aspects of wrformance assessment in which the clicitation and use of expert judgment should x formalized, discusses existing techni,qttes for formalizing the elicitation and use of expert judgment, and presents guidelmes for applying these techniques in performance assessment.

t i

viii-

~

,y1 i

l

1. INTRODUCTION l

The use of expert judgment permeates all scientific inquiry and decision making.

l De choice is not whether to use expert judgment, but whether to use it in an explicit and disciplined manner or in an ad hoc manner. For significant technical, environmental, and socioeconomic problems, it is often useful to formalize the clicitation and use of expert jud ment. One such problem is the long term disposal of high level radioactive waste HLW) in repositories mined into deep geologic formations. In siting and desi ning a safe, environmentally sound, and legally acceptable repository, many of the ana yses must use expert judgment.

l De Environmental Protection Agency (EPA) has mandated quantitative analyses in L

'its Standard 40 CFR Pan 191 for the disposal of spent nuclear fuel and high level and transuranic radioactive wastes. In particular, the EPA requires a so-called i

"pnformance assessment" in the containment requirement of this standard. (The other requirements are individual and groundwater protection requirements that L

concern only the undisturbed behavior of the repository system.) Performance i

assessment refers to "guantitative analyses that (1) identify the processes and events that might affect the disposal system; (2) examine the effects of these processes and L

events on the performance of the disposal systemt and (3) estimate the cumulative l

releases of radionuclides, considering the associated uncertainties, caused by all L

significant processes and events" (EPA,1985). EPA further requires that performance assessment estimates be represented by an overall probability distribution of cumulative releases. Furthermore, these probability distributions are to be used to determine whether the release standards in 40 CFR Part 191 are met.

L The Nuclear Regulatory Commission (NRC) has been charged with implementing this standard and examines the quality of a performance assessment when evaluating l

a license submitted by the Department of Energy (DOE) to construct and operate an HLW repository.

i Obviously expert judgment is extensively used in any responsible analysis of potential health im3 acts from a repository and particularly in performance assessments.

Expert juc gment is required in identifying and screening events and scenarios, in developmg and selecting models that characterize the geo ogy and hydrology of the repository system, in assessing model parameters, in collectmg data, and in making strategic decisions about the repository that could affect its performance. While it is desirable to use data and modeling extensively in performance assessment, it is nevertheless clear that these data and models can never substitute for the many crucial expert judgments in the assessment.

The quality of a performance assessment rests on its foundation of expert judgments.

to demonstrate that an HLW repository meets regulatory Consequently,l significant expert judgments should be documented and suppo requirements, al with sound logic and the best information. This is particularly important because of the need for multiple scientific disciplines to address the long term dis >osal of HLW and because of the intense scrutiny that all decisions will licely receive.

Responsibility and accountability can be enhanced by a formal elicitation and use of judgment, which is a well-documented, systematic process whereby experts make mferences or evaluations about a problem using available information as well as accepted scientific methods. This allows for traceability of the procedures, 1

techniques, and methods, assumptions, and physical principles relied on in any inferences or evaluations.

j 9

1.1 Objective of this Report l

This report discusses the formal elicitation and use of expert judgment in l

>erformance assessment of HLW disposal systems. More specifically, professional i

Inowledge about the analysis of HLW disposal systems and about the clicitation and use of expert judgment is combined to develop insights on the formalization of expert judgments applicable to HLW repositories, ne report (1), discusses the role of expert judgment in performance assessment of HLW repositories, (2) identifies areas L

- needing formal expert judgment in HLW disposal, (3) describes the formal clicitation and communication of e rt judgment, and (4) provides suggestions for the use of l

expert judgment in HLW isposal 1.2 Exped Judgment la Performance Assessment of NLW Repositories Experts are used to design and implement activities to understand present site conditions and predict the schavior of the disposal system. Expert judgment will be used in (1)(3) determining the level of resources fo(2) designing site data co setting priorities for data collection, activities, r reduction of uncertainties, (4)

L quantifying the uncertainty in numerical values for key parameters, (5) develop (ing scenarios and assigning corresponding probabilities of occurrence, and 6) formulating approacies for validating conceptual and mathematical models as well as verifying computer codes. These important tasks need to be addressed before using models and computer codes to predict behavior of the disposal system. Expert judgment is also used with the models and codes to estimate the system's performance for comparison with the numerical criteria in the regulations. For example, expert judgment is required to' screen insignificant scenarios, select methods for propagating uncertainty through the models and codes, quantify uncertainty in the predictions, and interpret results.

1.3 Characteristics of a Formalized Exnert Judoment Process A formal expert judgment process has a predetermined structure for the collection, processin3, and documentation of experts' knowledge. As discussed in Chapter 3, this includes professionally designed procedures to select problem areas and experts and to train experts for the elicitation of their judpnents. The actual clicitations of judgments should involve the expert and a professionally trained person to assist the expert in expressing judgments. De elicited judgments and their rationales should be carefully documented.

Dere are advantages and drawbacks in using such a process. The advantages include the following:

Improved Accuracy of Expert Judgmems. ne methods in a formal expert clicitation process im prove the accuracy and reliability of the resultin g information over less structured methods (Lichtenstein, Fischhoff, and Phi: lips,1977 and 1982;

. This is so because psychological Lichtenstein and Fischhoff,1980; Fischhoff,1982)d and communication is improved biases are openly dealt with, problems are define (Merkhofer,1987), issues are systematically analyzed, and rationales and results are 2-

.c.

!~. ;

9.e r

g:

a

.A i

- documented. The level of expertise may also be improved over less structured

-i methods since a formal process encourages a broadening of the tante of expertise.

i

~ Experts are carefully selected in a formal process rather than in a hapiazard manner j

for reasons of convenience.

Well Thought 7hrough Design for Elicitation. %e procedures that will be used in a ne design relies on the knowledge concerning er>crt opinion, problem b g faced, formal expe:t judgment process are designed specially for the previous studes that have used formal expert judgment, and knowlec ge of the problem domain to be studied. Careful planning of the procen can substantially reduce the likelihood of critical mistakes that will render information suspect or biased. Mistakes such as including xperts with motivational biases, failing to document rationales, inadverten mfluencing the experts' responses, failing to check for consistency, and i

allowingin duals to dominate group interactions can be avoided.

l Consistency of Procedures. A formal expert judgment process enhances consistency i

and comparability of procedures throughout a study and across related studies i

because participants follow the same procedures. On the other hand, informal processes are often subject to the whims and desires of panicipants.

i Scrutu'ity. A formal process requires the establishment and dissemination of rules and procedures for clicitation and use of expert judgment. A normal pan of a formal i

expert judgment >rocess is the documentation of procedures and assessments, which helps to ensure tsat varienu reviewers and users of the findings can understand and j

evaluate the methods and insights of the study. Since the methodology and its implementation are transparent, there is accountability.

j Communication. Establishing a' formal' process' helps to provide for reference

- 1 documents useful in communication and external review. A formal process also encourages communication and understanding among expens and analysts about the problems studied and the values assessed.

Less Delay. Projects have been delayed because critical judgments were not carefully obtained or documented, and a formal expert judgment process had to be designed and conducted before the project movec forward (DOE,1986). A well executed l

formal process would have avoided costly delays.

1 Dere are also drawbacks to the formal expert judgment process:

i Resources. There are costs in designing and implementing a formal process.

Documentation is often more extensive with a formal process, and more resources are thus required.

Time. He time to establish and implement a formal process may be significantly greater than that required for an informal process. Scheduling of participants from external organizations adds a layer to the effort that is not present in an internal, informal process.

Reduced Flexibility. Formalization of the process may reduce flexibility and make on-going changes to the study more difficult. If it is necessary to redo pan of a study, reenacting the expen-judgment process may be cumbersome and expensive.

t I

1 1

d l.

Vulnerability to Criticism. The transpatency of a formal process and the l

documentation of procedures and findings ppen it to inspection and criticism. Expert iudgment is an area in which misun, erstanding of the methods and aims still exists, i

d

?

L but a carefully designed and implemented process may thwart such criticisms, j

1 While a formal process often requires more resources and time than an informal 1-process initially requires, a faulty process that fails to withstand criticism or must be redone because of mappropriate design or improper execution may end up failin satisfy the project's objectives and cost more in both time and resources bto e

'~

potential for further costs in an informal study should be considered when evaluating the need for a formal process.

Formalizing the clicitation of expert judgments can clearly be expensive and time consuming. For this reason, the areas m which the process should be used should be i

carefully selected, it is neither practical nor reasonable to formalize the use of expert judgments in all aspects of HLW repository performance as,essment.

1.4 Preslous Formal Uses of Ernert.ludgments in HLW Program Several studies involved the formal elicitation and use of expert judgment on important problems facing the HLW program. Recent studies re,evant to

$,erformance assessment analysis of HLW repositories are outlined here. In L

scenario development and screen, pert judgments in HLW disposal are de five areas in need of formal ex l

l information gathering (e.g., data collection and experiments), and strategic L

repository decisions. Collectively, the analyses outlined here address problems in all i

five areas.

The Draft Environmental Assessment for the Hanford site in Washingten State (DOE,1984), reports an analysis that screened candidate horizons and identified a

-)

preferred horizon. A multidisciplinary team developed a set of eight measures to l

rank the horizons. These measures involved repository performance, construction ease, and costs. Deterministic and probabilistic descriptions of the candidate j

were probability distributions based on analytical models,probabilistic de horizons were developed using the eight measures. The 1

available scientific data, and explicit assessment of expert judgments. Because none of the candidate horizons dominated the others, a utility function was also assessed, using value judgments of the interdisciplinary team to combine the measures. The horizon descriptions were i

then evaluated using the utility function to rank the candidate horizons.

At the Hanford site, the formal clicitation and cuantification of expert udgment l

i-helped in designing an underground test facility (dolder and Associates,1986). To estimate groundwater and methane gas flow into the proposed test facility, estimates I

of site-specific geologic, hydrologic, and dissolved gas parameters were obtained.

l Specifically, pro) ability distributions were assessed for 41 parameters pertaining to flow path length, timing of encounters with geologic features, and transmissivity and storativity of the geologic surroundings near the test facility. The entire clicitation exercise included developing an influence diagram to help identify parameters to be assessed, identifying a panel of experts to be assessed, and conducting training sess, ions on probability elicitation for the panel of experts before the clicitation sessions.

I 1

4

.,_.,__,.__..,,,._,,_.y,

V Formal clicitation of expert j,udgm'ent was extensively used in a multiattribute I

decision analysis comparing honzontal and vertical emplacement modes for casks of spent nuclear fuel in a salt repository (Fluor Technology, Inc.,1988). First,10 attributes covering health and safety, cost, and environmental concerns were

'l selected. An influence diagram related several variables to these attributes. Expert judgment was elicited to provide probability distributions for both emplacement l

modes for some of the vanables. Deterministic estimates were obtained for others.

These estimates were input into a simulation model to describe the emplacement modes in terms of the attributes. A utility function was then assessed using the value judgments of a Fluor employee to evaluate alternatives.

The Department of Energy, following a recommendation of the Board on Radioactive Waste Management of the National Academy of Sciences, chose multiattribute utility analysis (MUA) as the methodology to rank five p(otentia provided part of the information to reduce the number of possible host sites to three.

l In the MUA. two different types of ex erts were used.- One type was senior I

managers of DOE who provided value ' dgr.1ents about risk attitudes and value

(

tradeoffs among the objectives of the stud. The second type were specialists in one or more of the technical areas needed to assess repository performance. These-i technical experts were divided into six panels addressing economic costs, 1

environmental impacts, social impacts, transportation of waste, repository l

l construction, and postclosure considerations. The technical experts were asked to l

develop measures of repository performance for both the preclosure and postclosure phases of HLW disposal; formulate scenarios for the postclosure phase; screen the L

scenarios to eliminate tinose that did not apply to particular sites; quantify the l

ilkelihood of each scenario occurring during the first 10,000 years after repository closure; estimate radionuclide discharge to the accessible environment in 10,000 years for each scenario; and finally, decide on the performance of each potential site for each of the performance measures (Merkhofer and Keeney,1987).

The Board on Radioactive Waste Management reviewed the methods used in the l

multiattribute utility analysis of potential repository sites. As part of its review, the Board stated (Appendix H, DOE,1986):

l.

While recognizing that there is no single, generally accepted procedure for I

integrating technical, economic, environmental, socioeconomic, and health and safety issues for ranking sites, the Board believes that the t

multiattribute utility methoc used by DOE is a satisfactory and appropriate decision aiding tool. The multiattribute utility method is a useful approach for statin g clearly and systematically the assumptions, l

judgments, preferences, anc tradeoffs that must go into a siting decision.

In addition, the expert judgments and methods in this report were publicly scrutinized by peer review (Gregory and Lichtenstein,1987).

l A subsequent analysis was based on the same expert judgments elicited for the multiattribute utility siting study. Because the Nuclear Waste Policy Act of 1982 i

stated that three sites should be characterized, Keeney (1987) analyzed portfolios of three sites for simultaneous characterization and strategies for sequential i

1.

e

.u characterization. Based on 1986 characterization costs estimated to be $1 billion per site, sequential characterization strategies were identified that could save $1.7 to 52.0 billion compared with simultaneous characterization of the three sites chosen by the DOE. This portfolio analysis and the multiattribute utility siting analysis provided lasights used by Congress m designing the Nuclear Waste Policy Act Amendments Act of 1987 that eliminated the simultaneous characterization of three sites and chose Yucca Mountain, Nevada, as the planned repository site.

Merkhofer and Runchal (1989) summarized a study to quantify judgmental uncertainty in values of hydrologic parameters at a repository site. S secifically, effective experts obtained cumulative density functions (cdfs) for the values of (1) ford site.

porosity, (2) average effective porosity, and (3) anisotropy ratio at the Han Two different groups of technical experts were used in the study. One group was five well-known hydrologists not directly involved with the site investigations at Hanford but, nevertheless, familiar with waste-disposal issues. The second group was three hydrologists involved in the characterization of the site. The probability clicitation process utilized structured interviews between a trained interviewer and each of the motivating, structuring, experts. The interviews consisted of five phases:

encoding, and verifying (Stael von Holstein and Matheson,1979). To conditioning, differences in judgments between the experts, all the results of the

)

reduce the original assessments were anonymously exchanged, as suggested by the original Delphi method (Dalkey and Helmer,1963). The revised probabilities showed at i

most only minor revisions; even though there was a considerable diversity of opinion.

i The experts indicated that any substantial changes would occur only after the exchange of logic and data by the experts.

HLW repository operation requires the transport of waste from nuclear power plants to the repository. A study by Westinghouse Electric Corporation developed a set of objectives for evaluating scent nuclear fuel transport ex > licitly using the judgments of experts (Westinghouse Electric Corporation,1986). "o establish a comprehensive set of objectives, three panels with individuals in the nuclear industry, state governments, and public mterest organizations were guided through sessions to create and structure objectives. Structured objectives of the three pancis were combined into one hierarchy for review. These objectives concerned health and safety and economic, environmental, political, social, and equity considerations as well as scheduling and Sexibility. The results were a basis for further analysis and communication among interested parties. The process of eliciting the objectives and the results is found in Keeney (1988b).

These studies clearly indicate that experts have been and will be used in a variety of ways to address critical issues relevant to the long term disposal of HLW in repositories mined in deep geologic formations. In some cases, the experts provided quantitative assessments (e.g., quantification of the uncertainty about a parameter, or ttie likelihood of a senario occurring); in other cases, they addressed qualitative identification and screening problems (e.g., selection of appropriate measures of i

repository performance, formulation and screening of postclosure scenarios); and in still other cases, they provided value judgments (e.g., attitudes toward risk and value tradeoffs). The funtamental concepts in the formal clicitation and use of expert

udgment are generic and indep

ndent of the type of issue the experts address.

Mowever, the choice of specific techniques during the clicitation process and the way

)

the judgments are used to address a problem should be issue-specific.

1

+

i

1 1.5 Wh= to Use Fraart Indement Formal methods should be used whenever the benefits are greater than the costs.

Indicators of when the clicitation of expert judgments should be formalized are as follows:

Lack ofData. When extensive, noncontroversial data directly relevant to a problem is lackmg, existing data must be supplemented with expert judgments, and it may be worthwhile to obtain them using a formal clicitation process.

rtance of the Issues. Formal methods are most appropriate when the expert ju ents wilI have a major impact on the study and improvements in the quality of th judgments are then most worthwhile. Important issues also draw the most scrutm, y. A formal methodology promotes documentation and communication and should be employed when the issue studied is apt to receive extensive review and criticism or when the findings will be widely disseminated.

Co of the Issues. When a problem is complex, or when several experts are either redundantly or as a team, formal methods are appro >riate. These r

em oye ods can provide the structure so that all participants understanc' the methods me used and apply procedures consistently.

l Level of Drumentation Required. Formal methods are a vehicle to obtain complete and consistent documentation of the methods and the findings. Informal methods.

often produce documentation that is incomplete with regard to the assumptions and procedures used. The critical reviews that the study will undergo, the variety and types of users, and the uses of the.information may also suggest whether a formal udgments may be process should be instituted. In some studies, the expert,Lormal methods are important fmdin'gs and, perhaps, used in subsequent studies, so needed.

Errent of the Use of E.rpert Opinion. When expert judgments are used extensively in a study, formalization of the collection and processmg of that information is apt to bc consistently, and efficiently using formal methods. Costs that done most accurately,f the size of the effort, such as creation of forms, training, etc.,

are fixed regardless o may be spread over many assessments. Also, when similar assessments are to be l

made by various experts, formalization of the procedures is necessary for consistency.

1.6 Relationshin of Formal Use of Ernert.fudament to Informal Use. Modelinn.

and Data Collection

~

~

~

As stated in the introduction, expert judgment enters performance assessments in many places. The question is therefore not whether to use expert judgment, but

)

whether to use it formally or informally, and how to use it with other sources of l

I information like basic physical principles, models, and data.

Informal use of expert judj; ment means implicit and undocumented use. Given the l

cost of formal expert juc gment, this may be reasonable in many instances in performance assessment. In some cases, " semi formal" uses may be advocated, such as brainstorming and/or taped group discussions about the issues. In such cases, it is 1

7-

~ _..

i important to identify carefully the objectives of the use of expert judgment and to be sure that its benefits outweigh its costs. Documentation is still important in semi-formal uses of expert judgments, because complex interactions may be involved.

Peer review should not be confused with the formal use of expert judgment. Both peer review and formal expert judgment are explicit and documented processes to meresse the likelihood that a resolution of an issue is of highest quality. However, the formal use of expert judgment attempts to bring out the available information that bears on the problem as part of its solution, while peer review evaluates and criticizes a given approach and solution to a oroblem. It should be noted that formal use of expert judgment can, and often shoul6 be subject to peer review. nus, these processes are compatible.

When formal expert judgment is used, a question arises about how it relates to other activities such as collectmg data or modeling phenomena,and processes. A simple answer is that any of these means of obtaining and quantifymg information should be used in a cost effective mix that solve.s the particular problem. In addition, formal l

expert judgment can often be beneGeial m integrating diverse sets of data and modeling activities and results. Thus, expert judgment and data collection and modeling activities should never be seen as substitutes, but as complements.

To contrast formel expert judgment to data collection and modeling, consider its favorable and unfavorable properties (Einhorn,1972). Expert judgment is a flexible and general source of information. A formal expert judgment process is unique in-that it can readily incorporate many disparate pieces of information into a coherent evaluation. Formal expert judgment, though, does not possess some properties of well behaved experimental / statistical data. For example, increasing the number of experts whose judgments are collected does not ensure that the " average" jud,gment will somehow converge to the true value. Nor can the usual assumption of independence and the assumption of convenient underlying distributions be called forth for use in expert judgment processes as they often are in the analysis of experimental data. It should be noted, however, that in most complex problems experimental / statistical data are not well behaved in this respect, either.

Formal expert judgments will not be as precise and clear as computer or mathematical models. However, these models build on expert judgment and may also suffer from the same limitations. Models that do not account for unforeseen factors or ignore potentially important variables fail in the same way that expert judgment fails when an expert or group of experts do not properly recognDe or account for all important factors.

l I

i l -.

x -.

l

'f

2. AREAS IN NEED OF FORMAL EXPERT JUDGMENT IN HLW DISPOS i

Expert judgment has been used and will be used in many aspects of performance I

asticssment as well as in other analyses, evaluations, and decisions related to HLW disposal. However, it may not be useful to formalize all expert judgments. As discussed in Section 1.3, there are many advantages and drawbacks to formal expert pudgment, and consequently, the decision of when to use it has to carefully consider

>enefits against costs.

In this chapter five areas of performance assessment in HWL repositories are discussed for which the benefits of formal expert judgment may outwei h its costs.

These five areas are (1) scenario development and screening, model development, (3) parameter estimation, (4) data collection and experimentat (5) strategic repository decisions. This chapter does not describe these areas in a comprehensive manner, but rather highlights those aspects in which expert judgment is likely to be formalized. It should be noted that some of these areas are described in detail elsewhere (Cranwell et al.,1989; Bonano and Cranwell,1988).

2.1 Scenario Develon=aat and Screenine To carry out a comprehensive performance assessment of the possible releases of radionuclides to the environment and to obtain >robabilistic assessments of these releases and the resulting health effects, an ana,ysis should consider the possible future states of the repository as influenced, for example, by climatic, geologic, and h drologic changes in the natural repository environment as well as by changes in the ysical and chemical characteristics of the man made repository system.

ecognizing this need to consider the repository system and its changes comprehensively, both the NRC (1983) and the EPA (1985) require that all ohysically plausible events and processes be considered in a performance assessment.

ln this context, events are discrete changes in the evolving states of the repository system, while processes are continuous and coherent!y linked changes.

Cranwell et al. (1989) describe a methodology developed by Sandia National for the selection and screenm of scenarios. This methodology

. Laboratories (SNL)he NRC and is currently used was developed for t a number of countries in their nuclear waste disposal programs. (Scenario W rking Group, Nuclear Energy Agency, Organization for Economic Cooperation and Development, Paris, France.)

Although other approaches with a slightly different focus are being developed (Thompson et al.,1988), DOE is also expected to use the scenario approach in performance assessment analysis of an HLW repository at Yucca Mountain.

initial identification of plausible events Scenario selection and screening involves (1)d processes, (3) initial screening o and processes, (2) classification of events an unimportant events and processes, (4) combining of important events and processes into scenarios, and (5) screening of scenarios to arrive at a final set for consequence analysis. Both for screening and for subsec uent analysis, each scenario is assigned a probability of occurrence during the regu atory period (i.e.,10,000 years). Expert jud ment is used in all steps of scenario selection and screening and in the estimation of obability of occurrence of scenarios as summarized below.

(

j

)

2.1.1 Identl5 cation of Events and Processes he initial listing of physically plausible events and processes is a creative task that depends almost exclusively on expert judgment. Dere is no widely acce >ted method that al olentially for arriving at this list, and there is no method for ensuring significant events and processes are included in the initial list category like "none of the above" and thereby ensuring completeness.

ormalizmg -

expert judgment is one means of decreasing the likelihood that important events and processes have been omitted. Formalized expert judgment is likely to be more useful than ad hoc methods because it draws on a varictv of experts, and because it is documented it can be scrutinized by many indiviiusis and groups interested in including events and processes that they consider significant.

2.1.2 Classification of Events and Processes For completeness and organizational purposes, events and processes are often.

human induced, and repository mduced. Often, the classified as naturally occurring,ified as affecting either the release of radionuclides events and processes are class from the repository to the geosphere or affecting the migration of radionuclides through the geosphere. Expert judgment combined with 3rmeiples of groundwater flow and transport phenomena is used to classify events anc processes.

2.1.3 Screening of Events and Processes The initial list of events and processes is often generic. Thus, the list should, in principle, be shortened on a site specific basis. That is, events and processes must be

. screened for each site. The NRC (NRC, 1983,1988) suggests to classify the events and processes into Anticipated Events and Processes Natural geological events and processes i

presently occurring or known to have occurred during the Quaternary Period (1.8 I

rnillion years ago to the present). In addition, one may want to consider natural events and processes that are not presently taking place but may be anticipated sometime in the future.

Unanticipated Events and Processes Natural and human-induced events and processes that are not likely during the 10,000 year regulatory period but are sufficiently credible that they cannot )e ignored.

Not Credible Events and Processes - Events and processes outside the other two categories.

Anticipated events and )rocesses and unanticipated events and processes, according to the NRC (NRC,1986), must be considered in the development of scenarios for a performance assessment to demonstrate comp (liance with the containment requirement of EPA Standard 40 CFR Part 191.13 EPA,1985) and the NRC Rule 10 CFR Part 60.113 (NRC,1983). Events and processes that are not credible can be eliminated from further consideration. Classiting events and processes into these categories depends on the experts' interpretation of historical records, site-characterization information, and conceptuahzations of the future of the repository and even of human behavior. his interpretation will, in turn, depend on a given

• 10-

r h

expert's technical background and may depend on the information base an approacn to the problem. Some aspects of the classification can be highly speculative because the meaning and interpretation of information depend on how an expert visualized the evolution of the system. In addition, the screening process depends on the expert's definition of " credible,"

2.1.4 Formulation of Scenarios Scenarios are formulated from all possible combinations of events and arocesses remaining after screening. Typically, an event tree is used to generate at: possible d

i ihf d if the initial

. combinations of events and processes. He proce ure s stra g t orwar list of events and processes is fairly complete and potentially significant events and processes have not been screened out. While this can, in principle, be done mechanically, expert judgment is needed to prune first-cut event trees and to check their consistency and completeness. De formulation of scenarios can also be done using fault trees by working backwards from potentially important future state (s) of the dis msal system and relating these outcomes to possible causes. Expert judgment is necc ed in identifying the states and in deriving common causes of sets of events.

In most cases, both event trees and fault trees should be used.

2.1J Screening of Scenarios An initial screening of scenarios is based on (1) physical reasonableness, which eliminates physically impossible or implausible combmations of events and processes, (2) the consequence of scenarios, which eliminates those with little or no impact on i

repository performance, and (3) likelihood of occurrence. In this manner, the number of scenarior can be reduced. Expert judgments play an important role in this preliminary screening by developing criteria for screening and applying them.

2.1.6 Probability of Occurrence i

Probabilities need to be assigned to scenarios for two reasons: to disregard from l

further consideration scenarios less likely than the screening criterion and to quantify the likelihoods of remaining scenarios to estimate cumulative radionuclide releases and health effects.

l scenarios.gmentplays a simificant role in estimating probabilities of occurrence for i

Expert jud L

Ideally, some iistorical data exist for a given site on climatic chan ges, I

j seismic activity, volcanic activity, human intrusion, etc., that can be used to formu ate models and pr5ide input used to predict the evolution of the site (a similar approach to the global modeling advocated by Thompson et al.,1988). Expert judgment is used to interpret the data, estimate the numerical values of model parameters, and, finally, to interpret the results of simulations and arrive at probability estimates. More realistically, data are likely to be scarce. Data for some phenomena (e.g., human intrusion) may not exist or models may be nonexistent or madequate. Expert judgment is then the main basis for estimating probability.

The probability of occurrence of the scenario is a combination of the probabilities of its individual events and processes. Expert judgment plays a major role not only in determining the probability of the events and processes, but also in the way these probabilities are combined to arrive at the probability of the scenario. For example,,

~

v experts are likely to be used to decide whether a scenario's events and processes occur in a sequence and, if this is so, to determine tue sequence.

2.2 Model Development In a performance assessment, assumptions and simplifications are made about the behavior of the repository system that can be incorporated into a "cor.ceptual model" for mathematical simulation of system behavior.

l Conceptual modeling of an HLW disposal site is based on a combination of the application of physical principles and data interpretation. Once the models have l

been developed using wlutever information or data are available (e.g., from small-scale, short term experiments), confidence must be built that the models are adequate to predict the behavior of the system over much larger spatial and temporal scales. Both the development of conceptual models and confidence building are L

creative and interpretative activities that are largely founded on expert judgment.

2.2.1 Data Selection and laterpretation Model development is based on limited, site specific information about the system

eometry, past and active processes, and potential disrupting processes and events.

. ittle or no data will be available to determine all of these factors at the proposed repository location. Therefore, experts select and interpret data from similar sites and relate them to the repository site. Interpretations of scant geologic data are used i

to define the system geometry. Experts must infer such things as the geologic continuity between drill holes, the extent and thickness of units, and the extent and l

character of geologic discontinuities such as faults. De geometry defined by these I

experts is based not only on interpolation and extrapolation of the site-specific data, but on data from similar geologic environments. Many processes are active in the geosphere (i.e., water flow, vapor flow, heat flow, etc.). Experts select and interpret data to decide which processes to consider in assessing the performance of a repository system. Not only do the experts have to decide the current dominant I

processes, but they must predict future processes that could adversely affect the l

repository system. His later assessment requires the experts to identify and interpret data from sunilar systems (i.e., analogs to the future states of the repository). Direct L

measurements of system performance (i.e., integrated discharge over 10,000 years) will never be available, so inferences about the possible system behavior and the accuracy of system models are from indirect site measurements and from information I

about sunilar systems.

2.2.2 Development of Conceptual Models i

Data cannot be collected over the temporal and spatial scales of interest in performance assessments of HLW repositories, so considerable data interpretation is required to formulate conceptual models. Because the conceptual model is the foundation of the mathematical models, computer codes, and data collection supporting performance assessment and because its development relies so heavily on expert juc gment, formalized expert judgment could be most beneficial in modeling.

A conceptual model includes simplifications and assumptions about (1) the geometry of the system, (2) the current or future physiochemical processes, (3) the boundary and initial conditions, and (4) the parameters governing these processes.

12-

Mig The most common approach to conceptual modeling begins with a rough sketch of 4

the model and continues to refine that sketch based on whatever experimental data l

and other information are available until an adequate first-cut model is produced.

Typically, this is done by using one expert's judgment and interpretations of experimental data and other inTormation. To make conceptual modeling more comprehensive and to encourage considerations of alternative models as well as scrutability of the experts' reasoning, Bonano and Cranwell (1988) suggest an approach for formalizing the use of expert judgment with multiple experts well versed on the groundwater flow and transport models. The approach forces the experts to articulate all assumptions, and to look for interpretations that challenge their conventional wisdom and are consistent with available data. De second point could lead to alternative conceptual models. Finally, the approach could include procedures for allowing the experts to identify bounding analyses and experimental investigations aimed at distinguishing between alternate conceptualizations and eventually reducing their number.

2.2.3 Confidence Building After conceptual models for the disposal system have been assembled, appropriate mathematical models and computer codes must be developed to simulate the behavior of the system over the spatial and temporal scales prescribed by the l

regulations (5 km and 10,000 years).

l Experts are an integral part of limited scope activities to build confidence in models l

and codes. For example, international groups have been formed such as INTRACOIN, HYDROCOIN, and INTRAVAL to select problems of common interest to the radioactive waste management community. These are simulated by interested parties, and the results are compared. These groups attempt to find discrepancies among the results from different experts and their causes. One important result is that the group may implicitly or explicitly agree that, given the current state of the art, existmg models and codes are as good as they can be. To date, these groups (specifically, INTRACOIN and HYDROCOIN) have focused on benchmarkmg activities that are an aspect of " code verification."*

He recently started INTRAVAL program goes one step further in that it aims at " validating' conceptual models, mathematical models, and computer codes."

l l

Validation means comparing the predictions of the models to experimental results.

l Because the models' predictive capabilities cannot be fully tested, "true" validation can never be achieved. The alternative is to build confidence in the models and codes through a synthesis of experiments and calculations. Experiments are likely to include laboratory and controlled field investigations as well as natural analogs.

Calculations could consist of bounding analyses and preliminary overall system

• Verification is defined by the NRC as the " process of obtaining assurance that a given computer code implements the solution of a given mathematical model."

"NRC defines validation as the " process by which assurance is obtained that a model as embodied in a computer code is an accurate representation of the process or system for which the model is intended."

13-

YW case, experts (1) design experiments and performance assessments. In any'ty and limitations of these experiments and establish the validi calculations, (2) define app (ropriate measures to ascerta i

calculations, (3 of the models )and codes, 4)(5)bility of the

)

ascertain the validity of important couplings in the models that cannot be tested, interpret the results of model runs agamst existing and new data, and (6) judge the a

]

and spatial scales, j

2.3 Parameter Estimation Performance assessment predictions depend on the numerical values of the mrameters used by their models and codes. Selecting appropriate numerical values for parameters and quantifying the uncertainty about them is a difficult but important aspect of performance assessment. First, im mrtant parameters must be identified, and then uncertainty in their values quantificc. Expert judgment is important in both of these aspects, as discussed below.

1 It might be worthwhile to defm' e the terms " parameter" and " data." Parameters are coefficients or constants of models and processes that describe or control the j

behavior of a model. Coefficients refer to the proportionality constants such as hydraulic conductivity and diffusivity needed in rate equations such as Darcy's law i

and Fick's law, respectively, and to the mean and standard deviation of a probability distribution. Data are values taken from experiments, observations of physical processes, or other sources, as well as functions (parameters such as the mean or variance) calculated from them.

1 2.3.1 Identification ofImportant Parameters Conceptual models enhance the quality of a performance assessment (e.g., improving

~

l the description of uncertainties about cumulative radionuclide releases and their effects on humans). Therefore, parameters should be identified to enhance the i

likelihood that their quantification leads to improved performance assessment.

Initially, the identification and selection of important parameters requires substantial judgment by the experts who decide how a given parameter may affect the descriptions of uncertainty for repository performance.

Once parameters are identified, their relative importance can often be ascertained by sensitivity analyses (i.e., by varying the value of the parameter and determining the l

l overall variation in the probability distribution of radionuclide emissions or some l

other intermediate performance measures) (Cranwell et al.,1987; Bonano et al.,

1989). For example, Bonano et al. (1989), m their analysis of a hypothetical HLW l

repository in basalt formations, show that the hydraulic conductivities of some l

geologic layers were important, while those of other layers did not influence the total radionucifde discharge in 10,000 years. These results indicate that to reduce uncertainty about the containment requirement (40 CFR Part 191.13), research should focus on reducing the uncertainty in the value of the hydraulic conductivity for i

the important layers and not the others. Intuitively, one could have stated a priori that hydraulic conductivity in general is a relatively important parameter. However, for stratified repository sites, it is important to distinguish among the different strata and identify the most important, which can be achieved only with a preliminary L

performance assessment.

14-l

7 j

g BWT Dere are various approaches for sensitivity analysis, but unfortunately, there can be large inconsistencies in the results from different approaches (Iman and Helton, i

1985), Iman ud Helton also show that different interpretations of the results from a given sensitivity analysis approach can lead to a different ranking of important l

variables. The problem is further complicated because not all sensitivity analysis approaches are appropriate under all circumstances.

Thus, expert judgment clearly plays an important role in the identification of parameters, in the selection of sensitivity analyses, and in the assessment of the unportance of parameters.

2J.2 Quantification of Uncertalaty la Parameters To assess the uncertainty in performance predictions for HLW disposal systems, it is necessary to quantify the uncertainty in the input parameters of the models and codes

(

used. De uncertainty in parameters can be expressed in a variety of ways. One way is to estimate a mean value and the variance about the mean. Another way is to determine the range of possible parametric values and to assess a probability density function (pdf) covering that range. The latter method is conventionally used in

- performance assessment analyses for HLW repositories (Cranwell et al.,1987; Bonano et al.,1989) because it provides a complete description of uncertainty and facilitates the generation of multiple samples o: the values of input parameters for carrying out M onte Carlo simulations. For these reasons, the examples below focus on the assessment of pdfs for input parameters.

r In principle, estimation of the possible range of values and pdfs of input parameters should rely on a very large sample of field data. However, such a large sample is not Gely to be collected at a candidate re sository site. Expert judgments are required to Uctermine what samples to take and how to interpret the results and to assess a probability distribution on the basis of the sample. Using Bayes' theorem, expert j_udgments can also be combined with data to arrive at a revised pdf for a parameter.

s Techniques for the clicitation and use of expert judgment can also be applied to to form quantify expert knowledge on a given parameter (e.g., hydraulic conductivity)ing sit a " prior pdf for that parameter. If n observations are obtained dur characterization, a joint distribution of the n observations can be constructed. This i

{cint distribution from collected data is used to modify the prior pdf to arrive at a posterior" pdf.

Given that experts have to decide on what to sample and given that financial and other practical considerations are likely to prevent the collection oflarge amounts of data, it is imperative that expert judgments su lement sampling with documented and traceable procedures. He study described Merkhofer anc Runchal(1989)in Section 1.4 is an example of the use of expert ju gments to quantify the uncertainty in the value of key parameters.

Another area in which expert judgment may ay a considerable role is in the quantification of the spatial variability of h rologic parameters. Although geostatistical techniques (such as kriging) exist f these purposes, they require input mformation, such as the mathematical form of the covariance function (describmg i.

tial correlation), which is likely to be determined using expert judgment (see sgnano and Cranwell,1988) 2.4 Infbreation Gathering Expert judgments are used with other sources of information to improve behavior predictions for the repository system. De current state of knowledge serves as a basis to decide what type of mformation should be collected and how it should bc collected to predict the future behavior of the repository with less uncertainty.

Additional information can be gathered in a variety of ways: collection of site-specific data, collection of related off site data, laboratory experiments, and analysis with model systems. Expert jud,gment is important in selecting among the alternatives to obtain more information.

The activities to obtain new information are likely to depend heavily on expert jud gments, if field data are to be collected at a proposed disposal site, experts must adc ress issues such as the test to be conducted; the number, location, and depth of drilled boreholes; and interpretation of collected data; etc. In laboratory experiments, experts deal with issues such as how representative the experiments are I

of field conditions; under what conditions the experiments are likely to be invalid; how the laboratory data are to be used with field data; etc. Finally, if analyses use existing models to supplement experimental information, experts need to address issues such as how the adequacy of the models was established; what key assumptions are in the models that cannot be tested; and how to select the parameter values in the model(s) so that they represent the current state of knowledge about the disposal H

system, etc.

When contemplating any of these que.stions, one should consider the prior knowledge about the repository and its performance, the possible chances that could be produced by'new information, the likelihood of these chang,es, and tle cost of the

(

information against its benefits. Clearly, any of these considerations requires a l

substantial amount of expert judgment, both about uncertainties (e.g., the prior uncertainty about a parameter) and about values (e.g., whether a million dollar l

experiment to decrease the uncertainty about a parameter is worth the cost).

2.5 Stratente Renository Decialons The four areas of performance assessment discussed in Sections 2.1 to 2.4 pertain to the need for formal expert judgment within a repository designed, constructed, and i

operated according to a given set of specifications. Hence, the performance assessment largely depends on decisions about the design, construction, and operation of a repository, which will affect the postclosure behavior of the repository.

For example, repository induced events and processes must be considered in the development of scenarios (Section 2.1). All these decisions must rely heavily on expert judgment.

Many design decisions are critical. For c:: ample, the exact depth and size of the repository needs to be determined, ne angle of the shaft to deliver the canisters to the repository needs to be decided, here are important decisions concerning the or horizontafly or exact placement of the canisters. Should they be placed vertically? These decisions at some other angle? And how near to each other should they be

\\

l !

\\

' c' BRIR 1

o could impact postclosure regulatory requirements such as canister lifetime and j

release rate from the engineered barrier system, which, in turn, could affect radionuclide transport through the ge phere and release to the biosphere. Clearly,

-1 these decisions require both factua gments (e.g., the lifetime of a canister), and

)f value judgments (e.g., the worth of a ing engineered barrier systems rom experts.

For each of the design decisions, there are complementary construction decisions.

There may be different alternatives to sink and enlarge the shaft to reach the both in repository. Different alternatives may be useful for excavating the repository,ials may terms of the techniques used and the timing of the activity. Different mater be used to insulate the shafts, and different '.:ngineerin g solutions may be found for constructing the repository Doors and walls. All these decisions affect the repository performance and involve crucial expert j'adgments that weigh performance against the costs and proclosure benefits.

Repository operation during the preclosure period also influences postclosure

' performance. For example, the management of the placement of canisters affects the degree of compliance with the design concepts of engineered barriers. Some decision problems may be necessitated by design or construction errors. Others will necessarily need to account for the possibility of such errors. In a similar vein, decisions about removing si,ightly damaged canisters or leaving them in the repository will affect long term repository performance. Any of these decisions requires both factual and val ue-laden expert judgments.

The general point here is that one connot examine expert judgment in (postelosure) performance assessment in isolation from the preclosure decisions and the numerous expert judgments involved in them. Simply put, postclosure expert judgments are only as good as the preclosure assumptions and judgments on which they are based.

.g

<17-L

s BRUT

~

3. ' ELICITATION, USE, AND COMlWUNICATION OF EXPERT JUDGMENTS In Chapter 2, five critical areas in need of expert judgment in performance assessment of HLW repositories were identified. This chapter describes the available formal approaches to elicit, use, analyze, and communicate expert judgment.

Section 3.1, defines the main terms used in formal expert judgment processes. While vary from situation to situation,plicable techniques for eliciting, expert judg,m the specific problems and the ap identifying the clicitation issues, selecting the experts, training the experts and carrying out the clicitation sessions (Section 3.2). Within this process several techniques are useful, dependir.g on the specific task at hand. These include techniques (e.g., selecting (e.g., generating scenarios or conceptual models) identification techniques scenarios), quantification techniques for probabilities (e.g.,

, and quantification techniques for values quantifing uncertainties about a parameter)dels). Man variants of these techniques (e.g., evaluating alternative conceptual mo are described in Section 3.3. Once individual expert ments are elicited, they can be analyzed and used in a variety of ways. Section, describes the issues and procedures for combining expert judgments. There are several apnoaches to communicating expert judpments. These include the specific form of c ocumenung expert judgments and ot presenting the results of expert elicitations. These approaches are described in Section 3.5. Finally, Section 3.6 discusses the interpretation, use, and misuse of expert judgments.

3.1 Definitions

~

v This section defines some technical terms used in this report such as issue, fudgment, expert, and probability, andfactual, value, quantitative, explicit, andformaljudgranents.

A repository issue is a question about the present state of a repository, its future state, or events and processes that may lead it from one state to another. Issues may concern assumptions about the repository and the re. lated natural and human systems. Issues may also concern the method of analysis for performance assessment. Issues are questions that shuuld be addressed to carry out a performance assessment.

l A fudgment is an inference or an evaluation based on an assessment of data, assumptions, criteria, and models. There are two basic types of judgments:

Judgments about facts and judgments about values. Judgments about facts are usually called beliefk or oppuons. People express their belicIs or opinions regarding propositions about facts or events whose truth or falsity can, at least in principle, be proven. For example, a person may believe that a nuclear waste repository will cost m excess of $20 bil.lon m 1988 currency. Or a person can have the opmion that there will be no radionuclide discharges to the accessible environment from a nuclear l

waste re itory within the first thousand years following closure. Although it would take 1 years to determine the truth about whether such discharge occurred, this is in principle [ossible.

1

i Judgments involving the use of criteria, priorities, and tradeoffs are usually called valuejudgments. Tliere is no possibility of proving a value judgment true or false as can be done with factual judg,ments. For example, when comparing the value of the health benefits for workers with the health benefits for members of the public, some

'>eo>le might conclude that a worker fatality avoided is as important as a public lata ity averted..Other >copie might conclude that a ublic fatality averted is more im >ortant because wor (ers take the risks voluntari. Such differences in value jug gments are quite legitimate expressions of dif erent social philosophies or i

. priorities.

Many judgments mix factual and value elements. For example, beliefs about the costs of a nuclear waste repository, coupled with a value judg, ment about the socially could lead to the desirable tradeoff between costs and benefits of the repository, beliefs about the conclusion that the repository is "too axpensive." Similarly, bout the relative predictive ability of a model, coupled with a value judgment a importance of predictive ability vs. simplicity, could lead to the conclusion that the model is " adequate."

An experr. has or is alleged to have superior knowledge about data, models, and rules in a specific area or field. Expertise is characterized by easy access to relevant information and by the ability to process that information and to use it effectively.

Shanteau (1987) observed other characteristics that define experts: the ability to simplify complex problems and to identify and react to exceptions; a strong sense of responsibility; confidence in their own udgmerit; and adaptability related to their c

knowledge domain. The domain of an expert can be a factual demain (e.g., a scientific dats base) or a value domain (e.g., the area of policy tradeoffs). Factual and value domains are often mixed, however, and one of the characteristics of expertise is the ability to separate factual and value components of judgments. For example, experts decide what data are relevant, what models should be used, how to interpret data to make recommendations, etc. Any of these decisions involve both l

l value and factual judgments.

Expert judgments can be implicit or crplicit. An explicit expert judgment is stated and documented for others to appraise. For example, when a particular conceptual model for a repository is chosen, the reasoning behind that choice can be made explicit in writmg. Or when a numerical estimate of a parameter value is chosen, evidence can justify that choice. In contrast, implicit expert judgments are supporting;le for appraisal and need to be inferred from actions and statements that not availa >

are available for appraisal. For example, when screening, scenarios, certain screening criteria may have been applied, but these criteria or ticir rationale may not have been explicit.

An explicit expert judgment can be quantitative or qualitative. A quantitative judgment expresses opmions or evaluations in numerical terms. Examples are the estimation of a parameter or the judgment of a probability of an event. Another example is the statement that public fatalities are four times as important as worker fatalities when evaluating health impacts from the repository. Exp" licit qualitative judgments are often expressed as verbal statements like " acceptable, "high chance has been or virtually im possible." The decision that " reasonable assurance I

provided that a:1 regulatory requirements will be met is an explicit qualitative l

1 L

, _ _..... - _., _.... _. _. _. _ _., _ _ _ ~.. _.. _. _ _. _ _ _ _ _ _ _.. _ _ -

..__.._,___..d_

L MW L

judgment. Many qualitative judgments enter scenario screening and conceptual-model selection and may be used to make the judgments explicit.

Quantitative expert judgments about facts can be expressed as probabilitics.

Probability is a degree of belief in an unverified proposition (DeFinetti,1937; Ramsey,1931; Savage,1954). Probabilities record the state of knowledge that an expert has about a specific proposition. These propositions can be about uncertain events (e.g., "there will be an earthquake of magnitude 7 or higher on the San Andreas fault within the next 30 years") or about uncertain quantities (e.g., "th:

l sverage travel time of radionuclides in medium A"). Uncertain quantities are also callet madom variables. Probabilities are numbers between 0 and I (inclusively), and they obey the laws of probability theory. Nonprobabilistic quantitative judg"ments include ranges of parameters or point estimates such as the "best guess of a parameter value.

Quantitative judgments about values can be expressed as utilities. Utilities express the tradeoffs among attributes of the alternatives to which the value judgments are relevant (Keeney and Raiffa,1976). For example, in selecting experiments for testing a given performance assessment model, a tradeoff is made between the information to be gained and the cost of the alternative experiments. Possible tradeoffs may be between the costs and benefits of laboratory experiments vs. field L

tests.

Decision analysis is a systematic procedure to assist experts and decision-makers in making judgments and choices in the presence of uncertainties, risks, and multiple conDicting objectives. Decision analysis comprises a philosophy for problem solving, formal axioms and models for inference, evaluation, and decision making, and a set of techniques for their implementation.. Decision analysis includes techniques for decomposmg issues and problems, quantifying expert opmions and value judgments, analyzmg and dsing these judgments, and recombmmg the decomposed problem.

3.2 The Process of Elicitine Exnert Iudements j

3.2.1 IdentlGcation ofIssues and Information Needs In the previous section, issues were defined as questions about the present state of a repository, its future state, and events and processes that may lead it from one state to another. Resolution of issues improves the quality of decisions about the repository and, as a special part of such decisions, the quality of performance assessments.

Issues range from general to fairly specific and from extremely complex to simple.

For example, a general, complex question may be, "Which conceptual model provides an adequate description of the past, present, and future states of the repository?" A fairly sMcific and somewhat simpler question may be, "Within a given conceptual model, what is the appropriate numerical value of a parameter o

describing hydraulic conductivity?" Issue identification may involve identification of the geologic and hydrologic features of the repository, identification of all major failure modes and pathways to the accessible environment, and identification of possible conceptual models and scenarios for analyzing failures.

l 1 l

u

Y s

Early in issue identification, emphasis should be on broadening the range of issues rather than narrowing it. It is often useful to invite persons outside the analysis staff to participate in this earl tage. For example, public interest groups may be asked to express their concerns, ectives, and potential scenarios regarding failure modes in the re >otitory. External r iew can aid in achieving completeness of the analysis and curtail criticism for failing to examine some issues. Examining and discarding an issue will be more ecceptable than justifying, after the fact, why the issue was not considered at all.

Oi.a a complete list of candidate issues has been created, it should be screened to 2

identify those most relevant to repository performance. Relevance includes both udgments of the likelihood that an issue mfluences the overall probability of a lailure at a repository as well as the extent of the possible consequences of failures.

Screening should employ both criteria.

After reducing the set of issues, information needs should be identified. In making decisions about the acquisition of information, consideration should be given to the relative accuracy, cost, and availability of alternative sources of information. The result, again, is not a final list, since the issue under consideration will be further analyzed and reviewed as issue descriptions are formulated and decomposed into

)

subissues.

1

)

Clearly laying out the issues for the experts is crucial. If five experts are asked to write down their understanding of an issue, one is apt to get five somewhat different j

1 l

descriptions. Critical differences can arise in the assumptions that experts make.

L The understanding of the initial conditions may vary greatly. If these assumptions or initial conditions are not explicitly defined, there can be an ensuing confusion during L

subsequent clicitations regardmg the issue..

3.2.2 Selection of Experts Performance assessment for HLW repositories requires several types of experts:

generalists, specialists, and normative experts. The generalists should be knowledgeable about various overall aspects of the repository, performance assessment. They typically have substantive knowledge m one discipline (e.g.,

geology, hydrology, transport phenomena) and a general understancling of the technical aspects of the problem. However, they are not necessarily at the forefront of any specialty within their main discipline. The specialists, on the other hand, are at the forefront of one specialty relevant to the performance of the repository,ise but they often do not have the generalist's knowledge about how their expert contributes to the overall >erformance assessment. Normative experts tpica,1y have psychology, and decision analysis. fhey assist training in probability tacory,ith substantive knowledge in articulating their

)

generalists and specialists w i

professional judgments and thought processes so that they can be meaningful,1y used l

m the performance assessment. A high quality performance assessment requires the teamwork of all three types of experts.

Each expert to be used in a performance assessment should be carefully selected to achieve a high quality performance assessment. Operationally, this means that the performance assessment team should address all the complex technical aspects of the problem and do this in a logically sound, practical manner that is open to evaluation j. - -

u

and peer review. De assessment should be politically acceptable, compatible with existing scientific and governmental institutions, and condtscive to learning (Fischhoff et al.,1981).

L 3.2.2.1 Selection of Generalists Generalists oversee completion of the performance assessment and provide c uality control for the performance assessment models and resulting analyses. I ence, generalists are usually selected from among the professionals within the organization responsible for the wrformance assessment. In selecting these generalists, project management shoulc. consider technical skills, organizational skills, and personal interaction skills. The generalists must have an understanding of the technical l

aspects of the overall performance assessment at a level where they can substantively communicate with specialists and normative experts. They should have L'

' organizational skills to schedule appropriately the gathering of information for the performance assessment. Generalists also need personal interaction skills to interact effectively with the numerous project personnel, specialists, and normative experts involved in the performance assessment.

3.2.2.2. Selection of Specialists L

There are three alternatives to consider in selecting s xcialists: (1) a single specialist to provide the set of judgments required, (2) a pane of more than one specialist in which each provides the set of judgments required, and (3) an expert team of specialists with the synergistic knowledge to provide a single set of judgments in situations rec uiring broader substantive Knowledge than is typically possessed.by an individual. The following addresses the identification and selection of individual specialists, panels of specialists, and expert teams.

The process of selecting specialists must be considered reasonable. Whether selecting individuals, panels, or teams, the first step is to identify specialists whose judgments might be appropriate for the performance assessment. The performance-assessment staff may have a number of suggestions for possible specialists. Others may come to mind from reviews of the published scientific literature addressing specific topics of interest. Parties interested in HLW disposal, such as utility companies and environmental groups, may have suggestions for appropriate specialists. Indeed, an open solicitation of nominations for specialists, including self-nominations, is one way to instill public confidence in the process. On important problems like HLW disposal, a formal solicitation of exnerts m the form of a request for expertise (much like a request for proposal) could M: very useful to identify the full range of expertise available and to ensure that an adequate search for expertise has occurred. Once a list of candidate specialists for use on a specific aspect of performance assessment is identified, a selection process must occur.

In the selecting a specialist, there are a number of important considerations.

Foremost, it is critical to ensure that the specialist has the expertise necessary. This should be verified by reviewing the individual's vita, by discussion with. peers in the field of specialty, and, most importantly, by discussions directly with that expert. It is also important that selected specialists be perceived as having that expertise by peers and others in related fields. If these criteria are met, then the potential specialists need to be both willing and available to participate. Another key consideration is l

l l

222-l'

i L

to have their name attached to their expert judgments in the whether they are willing(Section 3.5.1) Naming experts may enhance the quality of l

project documentation the expressed judgments, but more sijnificantly it increases the ability to evaluate the process and raises its credibility. Tie criteria used for selection should be explicit v

L and ). ' documented.

~

It is very important to avoid any potential conDict of interest between the specialists and the results of the performance assessment. A common issue is whether the prospective specialists derive their employment or any income from organizations l

charged with conducting the overall performance assessment or with constructing the repository. Those avsi able specialists with no conflicts should be chosen based on their expertise.

i fli

. Individuals with a perceived or real conflict of interest may not allow th s con ct to inDuence their professional judgments. Furthermore, we would not like to exclude crucial information from the performance assessment simply because a Therefore, it is knowledgeable individual had a potential conflict of interest.

i important to design the explicit clicitation and use of expert judgment such that the knowledge and reasoning of experts with potential conflicts can be made known to I

selected specialists in a timely manner. His communication process may include l

l distribution of written publications and analyses, as well as oral presentations.

u L

' When a sanel of specialists is to be selected, each specialist should, of course, have a high professional stature. However, additional issues are important. One of these is how many specialists are appropriate. Evidence sugests that three to five experts are usually sufficient to tap most of the expertise (C emen and Winider,1985). It is l

desirable to have the full range of legitimate opinions on a particular scientific topic available on any panel of. specialists and this implies that the specialists on a panel should be as independent as possible. Diversity is achieved when the specialists' sources of information and their reasoning, processes are different, and their approaches (e.g., theoretical models vs. experimentation) and >rofessional training Of course, to some degree, all experts wou d likely be at least are different.

somewhat familiar with the work of other experts in their fields. In addition, they would base their judgments on common scientific and engineering principles and knowledge. Thus, specialists cannot be completely indepetident, but this goal is important because it provides a more complete picture of the state of scientific knowledge as well as lending credibility to the performance assessment by representmg a broader viewpoint.

A quality performance assessment requires the expert judgments based on knowledge and experience in many disciplines. 'Ihese expert judgments will need to be logically integrated, along with all other relevant information and data, into models. No expert teams are necessary if the results of expert judg,ments from at other times the natural package of informationl; rate into the ana individuals or panels are naturally packaged to inte based on experts' judgments can only be acquired from an expert team comprised of specialists in related but syner gistic disciplines. An example is a study involving seismicity on the east coast of the L nited States. Each expert team was comprised of at least one seismologist, one geologist, and one geophysicist (see Electric Power Research Institute,1986)..

~

lf[

Each specialist on an erwrt team should meet all of the qualifications of individual experts stated above. Tie disciplines whose knowledge is essential to the scientific i

problem under investigation must be represented as part of each expen : cam. The performance assessment staff and then the expert team itself must ensure that all relevant disci lines are included. The performance assessment staff originally selects the specialis for the expert team based on project needs and the required scientific judgments. The expert team and performance assessment staff should initially review the task knd outline procedures to combine logically the udgments of various team members to provide the required overall udgments. I specific expertise is identified as lacking from the team at this stage, t e team should be augmented with additional specialists possessing the required knowledge.

3.2.2.3 Selection of Normative T.xperts The criteria for selecting normative experts are essentially the same as those that guide selection of individual specialists. Both the process of selection and its results are important because both mfluence the quality and the perceived quality of the ensuing elicitations of expert j gments. Normative experts require a sound theoretical and conceptual know dge of probability and techniques for eliciting judgments, and they need to be knowledgeable about the psychological processes i

occurring in the specialists' minds as they are processing mformation to produce requested results. Normative experts should also have significant skill and i

experience in working with technical professionals to make them feel comfortable in normative expreuing their judgments and in explaining their reasoning. Finally,bstantively experts should possess the communication skills necessary to mteract su with pr ect generalists and specialists and to document thoroughly the results of expert e citations.

i As with specialists, the qualifications of normative experts can be verified by j

appraising the individual's vita, discussion with peers experienced in clicitation and q

with specialists whose knowled e has been elicited by the individual in question, and by discussion with the indivi ual. Unlike the case with specialists, prospective normative experts can be asked to demonstrate their skills in actual clicitations using individuals on the performance assessment t,taff as specialists.

3.2.3 Training l

The professional literature on expert judgment clearly stresses the importance of trainmg experts in various aspects of the task facing them (

tzler and Stael von L

Holstem,1975; Merkhofer,1987; von Winterfeldt and Edwar 1986; Mosich, Bier, and Apostolakis,1988). Training consists of the following tasks:

familiarizing e rts with the expert judgment process and motivating them to provide formal dgments, giving rts practice in expressing their

• gments formal t

educa,n the experts about the possible eases in expert j dgment and applying I

debiasmg techniques.

l To accomplish these tasks, it is desirable to convene the experts individuali or as a group before the actual clicitation for at least a day. 'The training session s ould be L

j l

led by a normative exped with an in depth knowledge and experience in the art and r,cience of formal expert judgment processes, j

ne remainder of this section provides some general guidelines and ideas about how j

to accomplish these three tasks.

i Familiarizing the experts with the Judgment process and motivating them to pmvide formalfudgments. In most expert clicitations, the experts are specialists with J

substantial knowledge in a fairly restricted domain who have developed their own styles of communication and expectations about types of questions they can or cannot answer. They are usually very cautious regarding conclusions and judgments that may appear to be beyond the direct implications of data and experimental findings, scientific reasonmg, or models.

may even be threatening.gments is usually unfamiliar to expert Providing formal expert jud They may feel that they will be asked unreasonable questions. In particular, they may worry that they will be asked to provide more precise answers than their current knowledge justifies. In addition, they may not understand why they should express their juc gment at all, or if so, why in terms of numerical judgments such as probabilities or utilities. Furthermore, they may consider the expression of judgment based on incom plete knowled e to be inferior to J

the scientific work that would improve their knowled ge base. Fina ly, they may worry that their judgments may be misused or misrepresented.

It is therefore important that the training session address these concerns explicitly.

First, the normative expert, with technical input from generalists, should provide an -

overview of the performance assessment and indicate where the specific expert judgments will be used. De normative expert should point out that the experts were L

chosen to accomplish an important task and explain why they are among the more l

suitable for this task. Second, the need for formal expert judgment should be stressed. In performance assessment for HLW repositories, this need clearly arises because there are large uncertainties about scenarios, models, and parameters, and data are scarce. In addition, many decisions involve tradeoffs, as in betwe.en development cost and predictive accuracy in a conceptual model. Third, the normative expert should stress that there are no right or wrong answers to questions about expert judgments and that the purpose of the clicitations is to assess both what the experts know and what they do not know. Fourth, the normative expert should clearly explain that the process of eliciting expert judgments is not a substitute for further work in the expert's fields, but is, rather; a tool to summarize their current information. Formal clicitation of expert judgment often identifies very clearly l'

where sufficient knowledge exists, and where more research is needed. Finally, the L

way in which judgments will be used should be explained carefully. If, for example, judgments are averaged across experts, this should be explicitly stated and discussed.

i The normative expert should present a number of examples to illustrate various forms of expert jud gments. Dese include implicit and explicit judgments, qualitative and quantitative ludgments, and probability and utility judgments. De examples should preferably W drawn from the substantive knowledge domain of the specialists, such as geology or hydrology.

1 25 a..

Most experts know that they use judgment in their work all the time, but the specific forms of judgments in expert clicitations, especially probability and utility judgments, are likely to be unfamiliar to them. It is thereIore useful to explain the basic concepts as well as the main properties of probcbilities and utilities, Experts should be shown many examples of probability distributions and utility functions from within and outside of their field.

An important issue in any expert elicitation is the definition of the variable or event for which the judgment is to be ex >ressed. The normative expert should present many examples of well defined anc ill defined events and variables and illustrate them with the pitfalls of poor definitions: misunderstandings, miscommunication, and inappropriate assumptions.

Even after a thorough training session, some apprehension and concern may remain.

Most of these remammg concerns can be addressed only in performance of the tasks and it is therefore more useful to,give the experts some practice in clicitation of expert judgments rather than discussmg the issues abstractly.

Giving expertspmctice in crpressing theirjudgments erplicitly. There are several aspects of expert judgments that require practice:

making implicit judgments explicit, decomposing problems, and e

providmg numerical judgments, especially probabilities and utilities.

To show how implicit judgments can be made explicit, the normative expert should present the experts with several simple tasks involving judgments and afterwards point out that the answers require judgment and many answers include implicit assumptions. For example, when asked whether a canister in a repository will leak l

within the first thousand years, an expert may say that this is extremely unlikely, Implicit in this judgment are assumptions about the repository condition and canister o

corrosion. De normative expert sbould elicit these assumptions and point out their role in the judgments made.

Most expert judgments can be aided by decomposing the problem. For example, when estimatmg groundwater travel time through a layered medium, an expert may decompose his j,udj;ments by defining several layers and estimating groundwater travel time separately for each layer. Judgment of the relative controution of each layer can then be combined with the conditional estimates of groundwater travel time to arrive at an expected groundwater travel time.

Dere are several modes of decomposition. For factual judgments, event trees, fault trees, and functional decompositions are helpful (McCormick,1981; Raiffa,1968),

and for value judgments, value trees and objectives hierarchies are used (Keeney and Raiffa,1976). Smce any of these may be useful for representing and decomposing expert k,nowledge in a specific problem, it is useful to provide experts with some trammg m each mode.

The third area of practice is the actual clicitation of numerical values, especially probabilities and utilities. This can be done by carrying out some example clicitations interactively with the group. De literature on cognitive illusions and l

l

' 26-l

j i

pro'vability biases (Hogarth,1980; Kahnemann, Slovic, and Tversky,1982; von Wir,arfeldt and Edwards,1986) has many useful examples.

All tasks that are likely to occur in the clicitation sessions should be practiced. At a minimum, the experts should learn to respond to questions both outside their field and within their field, to factual and value problems, to questions about discrete i

events and continuous uncertain variables, and to difficult and easy questions. It is best to begin with easy questions on discrete events outside the experts' field and to I

end with difficult questions on continuous uncertain variables in their field. This I

sequence allows the experts to develop a degree of comfort eith answering questions before the challenging and presumably more uncomfortable ;pestions are posed.

Educatin experts about biases and applying deblasin technt uts. Cognitive psycholo sts have identified man biases m expert ju ments ogarth,1980; Kahnemann, Slovic, and Tversky,l biases can occur because the expert has a st 1

and cognitive biases. Motivationa the issue considered that may lead to conscious or unconscious distortions of his l

udgments. For exam ple, s bridge engineer is motivated to claim that a bridge that i

he just hel d to build is absolutely safe (i.e., the probability of it collapsing is zero).

Cognitiv biases occur when ex >crts fail to appropriately the available data anc information. process, aggregate, or mtegrate l

Most experimental research is on cognitive, rather than motivational biases, yet it is important in the training sessions l

to discuss and elaborate on both.

Research on cognitive biases has concentrated on probability, cognitive biases, and this section focuses on them. However, cognitive biases occur m utility judgments as l

well. Some recent experiments Weber et al.,1988 indicate, for example, that objectives presented in more dcta tend to be weighte more heavily. Furthermore,

~

cognitive biases can occur when structuring and framing the task at hand. Two comm'on structural biases are incomplete specification of alternatives and incom lete

~

statement of the assumptions underlying

• gments. Fischhoff et al. (1978, for L

exa e, showed that car mechanics and er subjects often fail to recognize all l

possi le failure modes of a car defect (e.g., failure to start). Experts often make estimates based on " normal" conditions or assumptions, but fail to make these conditions or assumptions explicit.

Most cognitive biases related to probabilityjudgments include Overconfidence Giving probrnbility judgments that express less uncertainty than the experts' knowledge would justify (i.e., too tight or too steep probability distributions);

Anchoring Adjusting judgments insufficiently after anchoring on an mitial estimate (e.g., a mean or median);

l Availability Overestimating probabilities of events that are easily imaginable or recalled; I

i <,

~

l i

Ignoring base rates Focusing on concrete evidence and data as a main l

nource.of probability judj;ments and ignorin g more abstract information lice base rates anc prior probabilitiest l

Nonregressive prediction Ignoring the unreliability of the relationship between

~

variables and therefore making predictions as if the j

relationship were reliabic.

Training should focus on the more likely biases in the particular performance assessment. In scenario construction and selection, for example, likely biases are incomplete events and assumptions, availability, and overconfidence, in the identification, appraisal, and selection of conceptual models, anchoring and availability, are most likely. In the assessment of uncertainty for parameters of models, overconfidence, anchoring, and nonregressive prediction are likely.

Debiasing techniques have only recently been developed (Kahnemann and 'INersky, 1979; Fischhoff,1982). For motivational biases, awareness of motivational factors t

both by the expert and by the clicitor is important. Sometimes it hel to present the l

question in the form of a hypothetical gamble (Section 3.3.4 to counteract motivational biases. For example, an expert may state that it is abso tely impossible that a nuclear reactor containment fails at 3ressures below 120 psig. In that case, one j

l might ask him, if he is willing to accept a >et awaroing him $10 in the event that no U.S. reactor containment will fail below 120 psig in the next 10 years vs. the loss of all i

of his possessions if one such accident occurs. Experts should be trained in such I

questions and be made aware in the training that the elicitator might attempt to deblas them this way when they suspect motivational biases.

l

\\

For cognitive biases, familiarity with the taisk, awareness of the bias, feedback, and personal experience with the bias help to reduce it. A useful training exercise is to j

provide experts with a catalogue of probability questions that are similar to those esed in the bias experiments and to let them ex xrience the bias themselves. While this does not assure self correction, it at least a erts them to the problem in a more vivid way. Since overconfidence, ancWing. availability, and nonregressiveness seem to be the main problems that m.o influence a performance assessment, a questionnaire that induces these four 6tases would make excellent training material.

f 3.2.4 Conducting Elicitation Sessions The clicitation of expert judgments should be based on a well-defined set of issues (Section 3.2.1). However, since the issues are identified before the selection of the i

experts, the experts may have suggestions for redefining details of the issue they are supposed to address. Before bcl;mning the clicitation, it is therefore important to discuss the issues, the possible pro>lem decompositions, the events and vanables, and I.

the questions that will be asked. In the clicitation of probability judgments, it is I

especially important that the events and variables are well defined. In the clicitation of utilities, it is important that the objectives and scales for measuriag them are well I

defined. For quahtatively described events this means, among other things, that the events are mutually exclusive and collectively exhaustive and that all conditioning L

events are defined. For quantitative variables, this means, among other things, that I

the meaning, dimension, and unit of the variable are well defmed. If events or i

i 28-

i

~

d l

variables are DI defined, various implicit judgments may enter the elicitation to fill the ' definition gap." Different experts may make different assumptions, and the slicitators and analysts may apply other assumptions in analyzing the responses, leading to confusion, miscommunication, and poor performance anaIyses.

If expert judgments provide specific inputs into a performance assessment, it is imporant that they match the requirements of the overall analysis. Thus, there also i

i should be preelicitation discussion of the nature and amount of expert judgment required by the overall performance assessment.

Alternative problem decompositions should be discussed, but some discretion should be left to the ex wrts in matching the individual decom >osition to their thought processes. In add ition, there often are alternative means o f expressing the clicitation events or variables through probabilistically related events or through functionally related variables. Again, each expert should feel free to choose among the alternatives that best accommodate his or her thinking, as long as the resulting responses can be related functionally or probabilistically to the clieitation events or variables.,

encralists or It hel ps for the staff involved in the clicitation and one or two ft. Guidelines j

specia ists to think through the whole clicitation process and practice for the clicitation should be drawn up, and materials (forms, graphs, etc.) should be designed for the actual elicitation.

An elicitation is an interaction between at least two people: the specialist and the i

normative expert. De specialist provides,udgments, for example, in the form of l

probabilities or utilities, as well as all re evant technical reasoning concerning judgments and conclusions. In addition to verbal statements, the specialist should i

l provide written _ materials documenting the reasoning as well as any background material used in preparing for the clicitation.

De normative expert is knowledgeable in the art and practice of expert clicitation, with special knowledge in probability and utility elicitation. The normative expert

)

asks the specialist to provide specific answers to questions regarding the : vents or variables considered, assists the specialists in explicating their reasoning, ensures that the required information is obtained, checks the consistency of the specialist's i

judgments especially with the laws of probability, and documents the numerical i

results for later processing.

In some clicitations, it is useful to request the participation of a cencralist for expertise in the requirements of the overall >roject and expertise in the specialist's area. The generalist ensures the technical vs.idity and consistency of the specialist's judgment, clarifies technical issues, documents the specialist's technical reasoning, and provides technical data and assumptions when needed.

12.4.1 Basle Elicitation Arrsagements The elicitation should take place in an undisturbed environment, preferably a separate room without telephone interruptions, visitors, or disturbing noise. The desk arrangement should be comfortable, encourage interaction among the f

l l

O

.o BRWf i

indMduals involved in the clicitation, and have work s documentation materials, forms, and re~ cording devices, pace and sufficient sj There are several ways of documenting an ongoing elicitation: tape recording, written notes by the normative expert, written notes by the generalist, and notes or i

documents that the specialkt brings into the session. Tape recordings provide a o

complete voice record. During taped clicitation sessions, it is important to refer i

explicitly to the materials and documents, figures, and tables used in the discussion to faclitate transcription and cross referencing in the written documentation. While tape recordings may provide more detail than necessary, they can be important for i

accountability, and for verification and clarification during written documentation.

l 1-Notes taken during the elicitation session by the normative expert and the generalist i

have different focuses. De normative expert focuses on writing down judgments and making lists, tables, and figures summarizing and relating these judgments for i

communication 2nd feedback. In case of probability clicitation, for example, the i

elicitator should write down the probabilities as tables, distributions, or functions that allow quick consistency checks and calculations for feedback. While most documentation of the normative expert is numerical,it is useful to note on the tables and plots the the s xclalist's rationale for certain judgments. De generalist should record the s pecia ist's reasoning in su sport of the judgments as well as cross-referencing, t to the specialist's own Cocumentation. It is important that the documentation schemes of the normative expert and the generalist are similar so that i

they can be cross referenced when documentation is consolidated.

3.1.4.1 Structure of a Standard Elicitation Session i

I A standard clicitation session begins with easing the' specialist into the situation and mapping out the task. De normative expert should ask the specialist to provide a brief overview of his or her approach to the problem and, in particular, the problem j

structure and decomposition used. After this exchange, the normative expert should i

define a road map for the remainder of the clicitation to determine the amount of work ahead.

l Next the definition of the events or variables to be elicited should be reconfirmed.

l i

l The normative expert should define the events and variables carefully, check the various meanings with the specialist and the generalist and write down the i

i I

dimensions and units on the forms prelated for the clicitation. Assumptions, especially about conditioning events, should be discussed and documented.

In the case of a decomposed event or variable, the normative expert should first map out a rough decomposition to clearly describe the logic used and simplify the i

judgmental tasks. Next the normative expert uses any combination of specific techniques (Section 3.3) to elicit expert judgment. These techniques rang;c from largely qualitative for identifying scenarios, models, or events to mixed qua itative-quantitative for screening, to largely quantitative for probability and utility Judgments.

Cunsistency checks by the normative expert are important to assure the internal logic of the expert judgments and to assist in identifymg sources of inconsistencies and j

resolving them. Consistency checks should be used to stimulate the specialist's r

do-

l i

thought processes. In probability elicitation, for example, it is useful to ask the same I

queshon by eliciting the desired probabilities directly or by eliciting probabilities for related variables or events. At a minimum, decomposed jud gments should be reaggregated to arrive at a calculated judgment about the clicitatc< event or variable,

-l and this calculated judgment should be compared with the specialist's intuition.

3.2.4J Post 486eltation AetMtles i

The specialists should be given quick feedback on the results of the elicitation. In l

distributions.y should be shown the numerical information in the form of ta particular, the Changes required by the specialist upon such feedback should be l'

adopted and reasons for them should be carefully documented.

in some cases, it is desirable to organise a group meeting of specialists, generalists, and normative experts after the individual sessions to discuss agreements and l

disagreements and whether it is pouible or desirable to reach consensus. There are j

several ways to organize such an interaction (See Section 3.4.4 and Seaver,1978). In some instances, it may even be desirable to reclicit some individuals after this group l

i session.

Sometimes specialists may want to change thelt clicitations after a significant time has passed. Such change requests should be probed carefully but accommodated if 4

feasible within the framework of the overall project. Reelicitation may be necessary, I

gnd the documentation should reflect the revisions and the reasons for them.

'!he basic design also requires eliciting one specialist at a time. It is conceivable to clicit several specialists simultaneously, for example, in groups or classroom sessions.

While this method is preferable to a pure questionnaire format,it suffers from some of the.same drawbacks. In particular, classroom settings require more conformity on

-l case structure and decompositions, allow less flexibliity in mdividual responses, and may suppress expressions of alternative views.

l There are, of course, many variants to the postelicitation activities. An important issue is whether the clicitation to achieve group consensus, to aggregate different judgments, or simply to report the results from different specialists (Section 3.4.).

3J Techniones for Emnert _Iudement Elleltatlan An expert engages in three fundamental cognitive processes when making judgments:

(1)idenaficanon of options or events to be judged; (2) screening of the options and events; and (3) quanaficadon of comparative judgments about the options and events.

/dena]Icadon consists of recall, search, and creation. Recall identifies easily available alternatives, search systematically lists existing alternatives, and creation generates previously unknown or inaccessible alternatives. Screening consists of selecting l

screening attributes, setting screening constraints, and selecting alternatives based on the attributes and constraints. Quanafication consists of assignmg numbers to factual or value judgments about alternatives. Factual judgments about events or random variables are usually quantified by probability distributions. Value judgments (e.g.,

i about the advantages or disadvantages of alternative conceptual moclels) are usuaDy

(

quantified by utility and tradeoff judgments.

31-

....._.-_.,__.,___._..m.._.,,,,,....-,__....._,

l l

De literature on identification techniques is fairly small. Dere are a few techniques i

for creative option and event generation (Pearl,1978; Pitz, Sachs, and Heerbroth, 1980; Gettys, Fisher, and Mehle,1978; Keeney,1988a). Most screening techniques l

consist of utting numerical cutoffs on selected screening attributes and searching,for the subset of " survivors." Keeney (1980) describes the basic idea for screening m a value judgment context, and several reports discuss the use of " cutoff probabilities" l

for screemng undesirable events (Department of Ene.'gy,1986; Okrent,1980; Wilson, j

1964).

j In contrast to the small literature on identification and screening techniques, there is f

a rich literature on quantification technic ues that draws mainly on psychophysics l

Poulton,1979t Ekman and Sjoberg,1965; Zinnes,1969) and decision analysis Raiffa,1968; Brown, Kahr, and Peterson,1974; Keeney and Raiffa,1976; von f

interfeldt and Edwards,1986). The decision analysis literature typically

('

emphastaes quantification of probabilities (Sytzler and von Holstein,1975; Selvidge, 4

1975; Seaver,1978; Keene 1980; Stillwel;, Seaner, Schwartz,1981; Wallsten and i

Budescu,1983; Merkhofer,y,1987) and utilities (Keeney and Raiffa,1976; Keeney, i

1980; Edwards and Newman,1982),

i l

The following three sections summarire this literature and make recommendations about techniques for identification, screening, and quantification.

{

3.3.1 Identification Techniques Identification techniques primarily assist experts in identifying scenarios and conceptual models for performance assessment. In scenario identification, the emphasis is on stretching the experts' imagination and on creative processes of event generation. Conceptual model identification, emphasizes generating desirable model alternatives.

l h

3.3.1.1 Techniques fbr Event and Scenario identification l

Recall and search are fairly trivial tasks in event and scenario identification. in the recall mode, one simply asks the experts to list all the events and scenarios that they recall that are relevant for the normal performance of the repository or for scenarios that could adversely impact that performance. In the search mode, experts survey the literature for relevant events or scenarios. It helps to enrich the set of events and I

scenarios by asking nonexperts and those with a stake in the decision (e.g.,

environmental groups, residents living near the repository). De emphasis at this stage should be on completeness and comprehensiveness, not on logic, reasonableness, or likelihood of occurrence.

Event and scenario creation is the most interesting and innovative as l

Dere are three cognitive techniques to creative scenario generation:

forward and backward inductior.;

e value-driven event and scenario generation; and e

analogy-or antinomy-driven event and scenario generation.

l e

Forward and backward induction builds on the notion that scenarios are logical I

sequences of events linked through processes. It begins with listing all possible and j

32-l l

^

5 gM6JeVF i

l conceivable events that could occur related to a repository. In the forward induction mode, events are linked to create an event tree that fans out from initiatin g events to events that may occur in thousands of years. Provided that the events anc processes j

this event tree can, in principle, be constructed are defined sequentially,ing to a very large tree representmg with thousands of mechanically, typically lead scenarios. His tree should be pruned to e iminate branches that are impos ible, extremely unlikely, or redundant. In the backward induction mode, the final states of l

the repository are the starting point of the process. A possible final state may be defined as " major releases to the accessible environment occur in the year 3000."

Backward induction defines the possible causes of this final event and thus works back to the initial conditions, events, and processes that make it possible.

i Forward induction typically creates too many scenarios, while. backward induction may create too few. By applying both processes and reconciling the results, it should be possible to identify a subset of scenarios that spans the range of scenarios relevant j

to t'he performance of the repository.

)

i The second technique begins with the question: What are the performance l

objectives for a repository and how can they be achieved? Presumably the main objective is to protect public health and safety, but other objectives like cost and l

long term environmenta protection may be important as well. After identifying a set of objectives, events and scenarios are developed that would lead to extremely poor, average, and extremely good performance on each objective (Keeney,1988at l

Edwards et al.,1987). For example,in the case of health and safety, an " undisturbed-i performance" or " base case" scenario without major geological events or human mtrusions would presumably lead to average performance. Adding favorable assumptions about the behavior of the canister materials and the rock medium may lead to extremely good performance. Combining major magnetic and seismological events with poor geology and excessive corrosion may lead to very poor i

performance. While this technique tends to look at the worst case in terms of health l

and safety, it is very instructive to look at other cases and other objectives as well, s

I Event and scenario creation by analogy or antinomy attempts to stimulate the thought processes of the experts (Jungermann and Ducring,1987). In an analogy, l

one would take the events and scenarios out of the context of an HLW repository and ask experts to instead think of the repository, for example, as a coal mine i

containij: lethal gases. De question would be: What could go wrong in this coal 3

n mine? Tie follow up question would be: Do any of these coal mine events and scenarios apply to the real repository case? In an antinomy one could ask experts to think of the repository, for example, as containing the most precious human i

i possession that required protection from attempted theft. De question might be:

iow an thieves enter the repository, and how can theft be prevented? Agam, the answers would be checked for their relevance to repository performance.

Any of these three techniques can be combined with various forms of interactions among experts. Dese incdude Delphi tyx techniques (Linstone and Turoff,1975; 1969), the Nominal Group Tec mique (Celbecq et al.,1975), and several Dalkey,f brainstorming. Furthermore, they can be substantially enhanced by forms o I

involving individuals with very different perspectives regarding the repository (e.g.,

local residents, environmentahsts, and nuclear engineers). Since '.he purpose at this point is to assure comprehensiveness, any inputs that are novel and creative should 33

s g

^

I be appreciated. Peer review is another useful mechanism to identify events and j

scenarios that have been overlooked..

It is very important that the activities during event and scenario identification and the l

results are carefully documented. In particular, reasons for eliminating certain events and scenarios should be carefully recorded.

3.3.1.2 Identification of Crgsl Models As in scenario identification, recall and search are fairly straightforward activitics to identify conceptual models. De main techrique for the innovative creation of l

conceptual models is similar to the value driven technique described above (Pitz and l

1988a). The technic us be gins with a listing of the Sachs,1984; Pearl,1978; Keeney,for a conceptual mooel. Next the experts develop l

desired properties or objectives features of conceptual models that would serve one objective well. After completing i

this task with the first objective, it is repeated for the second, the third, and so on.

i l

Features developed from subsets of objectives are combined to characterize one I

possible conceptual model. Repeating this process suggests many different conceptual models.

Having generated a large number of conceptual models, the next task is to narrow this set down to a reasonable size. His task includes examining all conceptual models on all c'bjectives simultaneously and eliminating those that are clearly unacce ptable on one or more objectives. Since this task involves screemng, many of the techniques discussed in the next section will be applicable.

3.3.2 Screening Techniques The first step in screening scenarios or conceptual models is to identify the attributes with which to screen alternatives. This step is followed by setting target levels or l

constraints on the attributes. Alternatives are then screened out that do not meet the target levels and constraints. Typically, this process is iterative: when too many i

alternatives survive, more strinl;ent target levels or constraints should be applied.

l When too few survive, target leve s or constraints should be relaxed.

l t

Identification of Attsbutes. Scenarios should be physically consistent sequences of events. It is therefore important to screen out those that are logically flawed. For i

example, if one event is the coming of another ice age combined with the migration i

of the earth's population to the southern hemisphere, it is log ~ically inconsistent to couple this event with large numbers of human exposures because of radioactive I

leakage. Given another ice age, it is improbable, although not logically inconsistent, that there would be exploratory drilling for minerals other than the radioactive t

materials themselves.

Before eliminating a particular scenario because of a physically illogical se uence of j

events, it is instructive to ask several experts to explam the presumabl illogical sequence. In the above example, some experts may find the combination o icing and l

exploratory drilling illogical. But others may speculate that the exploratory drilling for some yet unvalued mineral would go on all over the world even in unfriendly climates, just as it is going on in the polar regions today.

34 n

Scenarios can also be screened on potential consequences, eliminating scenarios with j

relatively insignificant impacts, and probability. Probability criteria can be dermed l

on the whole scenario, on individual events, and on part of the sequences of events.

In addition, probability criteria can be set differently, depending on the conseqtsences of a scenarlo. It is useful to spell out different sets of probability criteria and j

investigate their use before fixing target levels and constraints.

Attributes for screening conceptual models can be very diverse. Examples include scientific acceptance, predictive ability, ability to estimate the parameters, simplicity, j

and cost. Techniques for identifying and structuring such attributes are described m i

Section 3.3.3.

3.3.2.1 Setting Target Imeis or Constroints a main issue is the selection of probabilities to screen out In scenario screening,lity events or scenarios, eliminating those that most peopic extremely low probabi would consider " incredible," " implausible," " virtually impossible," or even

" unbelievable" or " inconceivable." *!Tiese target probabilities can pertain to an event i

in a scenario or to the total scenario. These probabilities are linked, as the J

probability of any event in a scenario must be larger than the probability of the scenario, in other words, if a single event in a scenario has probability p, then the scenario has to have a smaller probability pq, where q is the conditional >robability of all the other event elements of the scenario given the event under consic cration, j

r When setting event or scenario screening probabilities, one should consider the possible consequences. A common technique is to define smaller screening l

probabilities on overall scenarios if the possible consequences are more significant.

For nuclear power plant accidents, for example, a screening probability for a core i

meltdown may be 104, but the screening pr'obability for a core melt with containment j

failure may be set as low as 10 9. A more explicit approach is to set a target level on the probability (distribution or, alternatively, on the complemen t

density function NRC,1975)d in Wilson (1984).

l potential benefits as describe f

Screening conceptual models is more complicated, since there are more attributes to consider. Keeney (1980) discusses this issue in the context of screening alternative sites for energy facilities. He points out that screening is a simplified selection l

process and as such requires value tradeoffs among the screening attributes.

To illustrate this point, consider two screening attributes of conceptual models:cost of computer run time and empirical validity. One could set target levels on both attributes. For example, one could say that to be selected, a model run should not cost more than $10,000 and the expected error in predicting radionuclide travel time should be less than 100 years.

P Alternatively, one could set the target levels at a model run not costing more than

$10,000 but an expected error in predicting radionuclide travel time of.less than 50 j

years.

Notice that the second set of target levels is more restrictive on the empirical validity attribute. Thus, in effect, by using the second set of target levels, we assign more 35-t

d weight to the attribute of predictive validity. This is a general feature of setting target levels and constraints: setting these levels by itself involves crucial value tradeoffs among attributes.

makes these tradeoffs Multiattribute utility analysis (Keeney and Ralffa,1976)ls. While a fuli fledged explicit and could be used to set constraints and target leve multiettribute utility analysis may be too costly for the purpose of screeninJ, it is important to be cognizant of the tradeoffs made when setting target leve.s and constraints. As a practical rule, it helps to set target levels and constraints interactively, startina with very lenient levels and examining the set of surviving concep.ual models afIer each setting of target levels.

3.3.2.2 Selection Once attributes and target levels or constraints are defined, the selection is essentially mechanical. It is useful, however, to reiterate and go through a number of changes in setting target levels and constraints to investigate their implications for the selected subset. it is also useful to explain the logic of the process to a broad range of interested parties and to let them critique both the process and the result.

3.3.3 Decomposition Techniques Problem decomposition is widely used in scientific study to simplify a complex problem into components that are more manageable and more easily solved.

Problem decomposition has also been recognized as an important tool m expert udgment clicitation (Raiffa,1968; Brown, Kahr, and Peterson,1974; Armstrong.

,Denniston, and Gordon,1975).

Problem decomposition in elicitation refers to breaking down issues to provide for easier and less com lex assessments that can be recombined into a probability i

distribution or utilit function for the quantity of interest. The recombination is usually accomplishe through a mathematical model that expresses the quantity of interest as a mathematical function of component quantities. The techniques l

decom msition depend on whether the problem is a factual or value problem. Event trees, Lault trees, and functional decompositions are used for factual issues, and objectives hierarchies are used for value issues.

3.3J.1 Decomposition of Factual Problems Several types of decompositions facilitate expert judgment about facts and probabilities. A familiar type of decomposition is the fault tree (McCormick,1981),

i which focuses on a possible failure of a system and traces back the possible j

component causes of this failure. Fault trees are common:!y represented as circuit 1

i diagrams that display the relations among system components and the failure of a system. In fault tree analysis, the components are assigned probabilities of failure, l

from which overall failure probability of the system can be found. Usually failures of j

various components are treated as inde wndent events, although sometimes common i

i causes lead to related component fai utes. Fault trees serve as a vehicle for the decomposition of expert judgments when the compnent events are dichotomous (0 to 1), inde wndent, and the overall failure event is,ogically related to the component events..iowever, when decomposing, care must be taken to ensure that 36-n

's-completeness is not lost. When finer detail about the causes of failure of some event in a fault tree is sought, experience suggests that incompleteness can easily occur (Fischhoff, Storic, and LJehtenstein,1978).

While fault trees end in a single failure event and trace its possible causes, event trees begin with an initiating event and draw out its x>ssible consequences. The event tree lays out the sequence such that the probabil! ties of successive events are conditional on their predecessors. The branching in an event tree leads to a proliferation of paths, each path having a terminus as ociated with a system state or consequence. Event trees are a natural means of representation when phenomena have discrete outcomes. When the outcomes are continuous, however, the use of event trees rsquires that the continuous outcomes be approximated by a discrete categorization of ranges of the outcome variables.

A related type of decomposition uses the conditioning of possible events on known or hypothensed events (Bunn,1984). De events can be laid out as an event tree i

where predecessor events are the conditions for the event in question. For instance, l

De assessment task then requires the probabilities of A,ypothetica the probability of event A may be conditioned on the h j

given various combinations of B and C and their complements. Further, the probabilities of B and its complement, given C and its complement, must be assessed as well as the probabilities of C and the complement of C. Denoting the complement of an event l

E by E, the probability of the event A becomes l

l P(A) = P(AlB,C)P(BlC)P(C) + P(AlB',C)P(B'lC)P(C) +

l P(AlB,C')P(BlC)P(C) t P(AlB',C')P(B'lC')P(C').

I Barclay et al. (1977) demonstrate the use of this style of decomposition to ascertain the likelihood that a nation will have the capability of producing nuclear wea >ons within a given time frame. An analysis and discussion of theoretical aspects o' the probability decomposition are provided by Ravinder, Kleinmuntz, and Dyer (1988).

l l

A tree structure related to the event tree is the decision tree (Ralffa,1968; Holloway, 1979). In addition to possible events, decision trees incorporate choices or decisions I

that partially determine the path followed. Decision trees are particularly valuable in the evaluation of alternatives. Decision trees should be he pful in the analysis of information gathering activities associated with the potential repository and in evaluating design and construction options for the repository.

Decomposition may also use physical models of the phenomena bein g analyzed. The physical relationship between the quantity of interest and several constituent or determined quantities is expressed through a mathematical function such as T =

f(X,Y,Z). This type of decomposition is called algorithmic decomposition by MacGre or Licluenstein, and Slovic (1988). Rather than assessirig a single ity distributions for X, principle of decomposition leads to t distribution for T, the probabil of prob distribution for T. If the expert is better able to ex 3ress knowledge about the constituent quantities than about the original quantity, tie issue is a good candidate for decomposition. This strategy has been used in the reactor risk reference G7-

.>l n

bh t

1 document (Wheeler, Hora, and Cramond,1989), and the EPRI study of seismicity

(

!)

l (Electric Power Research Institute,1986).

If the expert possesses knowledge about X, Y, and Z and, further, knows the functional re ationshik f then the expert should be able to give equivalent assessments either in Lerms of T or in terms of X, Y, and Z. However, the combination of X, Y, and Iis likely to be too complex for the human mind to do without substantial assistance. Decomposition, then, can serve as an aid to human i

thought processes in that the mind is relieved of tasks that it is ill equipped to l

i perform (Einhorn,1975).

l 3J.3.1 Decomposition of Value Problems i

De best known technique for decomposint value problems is structuring so called objectives hierarchies. Objectives hierarciles structure the expert's general valac concerns, intermediate objectives, and specific value relevant attributes m a tree like hierarchy in which the lower levels define what is meant by the upper levels (Keeney r

and Raiffs,1976; von Winterfeldt and Edwards,1986). Object ves hierarchies are (tructured by either the top down or the bottom up approach. Both approaches are i

i

..nplemented in interviews with experts knowledgeable about the value domain considered. They are illustrated below with an example of evaluating alternative conceptual models.

I The top down approach begins with general value concerns like costs, scientific validity, etc., and subsequently specifies the meaning of these general terms at increasing levels of detail. For example,scienr@c validity could be broken down into empirical validity, and ariomatic validity. Empirical validity could be face validity, ken down into experimental validation at the repository and empin' cal t

further 6to validition at other sites, When considering a hierarchy of concerns, objectives, and

[

attributes, it is important to pursue and to eliminate means objectives.

I f

The bottom up approach begins with listing the features that differentiate the options. From this list, features are eliminated that are not relevant for comparative evaluation. Amony conceptual models, for example, average run time is value relevant because o cost and delay of feedback. On the other hand, place of l

i development may not be value relevant. Having screened for value relevance, the next step is to eliminate means and pursue ends. Finally, the remaining features are clustered and organized into a logical hierarchy, l

he results of the top-down and bottom up approaches should be similar hierarchies with general value concerns at the top and specific attributes at the bottom. Once a first-cut hierarchy is built, the following checks can be used to examine and revise it:

t Are any concerns, objectives, or attributes redundant 7 e

Is the set of concerns, objectives, and attributes exhaustive?

Are the concerns, obj ectives, and attributes independent?

e Is the tree manageable for further analysis?

e Are the lowest level attributes operational; that is, can one measure and compare, e

for example, conceptual models on them?

38-

O N.W l

ddd f

Checking and revising often involves returning the initial hierarchies to the experts for reexamination.

ne previously described decompositions of factual and value problems are fairly j

formal in that they express the results es trees or functions.

3JJJ Verlaats of Decompoaltion Decomposition an also be used less formally. De goal of a less formal procedure might be to promote deeper insight into the rationsic for judgments and to enhance the interchange of beliefs and assumptions about the likely causes of studied events without formally encoding the decomposition. The decompositions might be in terms of casual or mitigating factors that are loosely related to the event or quantity of interest. In this form, decomposition enhances the experts' introspection and communication.

A key aspect of decomposition relates to the source of the model or models used as a i

I decomposing framework. De models can be imposed upon the experts from an

[

external source, or they can be generated by the experts. Individual experts may be allowed to choose their own decompositions, or a consensus decomposition may be j

j used.

Using a single decompsition has several advantages. First, the costs of recombining i

the judgments may be substantially reduced. Experience with NUREG 1150 indicated that the effort to process elicitations from multiple experts who used J

unique decompositions was much greater than expected (Wheeler, Hora, and l

Cramond,1989).

-l Another potential advantage of using a single decomposition is that comparisons can be made among elicitations for component quantities and events. Combining j

l assessments at the component level and then recomposing is also feasible when a single model is employed. Neither comparison at the component level nor aggregation at a subissue hevel is feasible with multiple decompositions.

I A single decomposition by multiple experts also has important drawbacks. First, there needs to be significant discussion to ensure that all experts understand and j

accept the chosen decompositions, which is often difficult to achieve. Second, the l

influence that a decomposition has on the ultimate result is considerable. Requiring experts to abide by a single model may force their judgments to appear to be in i

agreement and thus understate their underlying differences as to the appropriate processes and assumptions. And if the decomposition itself is somewhat faulty, the i

i results can be misleading. It is important to recognize that the decomposition itself embodies much information.

l f

De advantage of multiple decompositions is that a wider variety of approaches to the problem are permitted. Single decompositions may understate the true T

uncertainty about an issue because the experts are forced to conform to a single view.

Research has shown that the method of analysis, or decomposition, is important in forming judgments (Booker and Meyer,12_). Multiple decompositions also provide a vehicle for discussion and documentation of alternative viewpoints-an important by-product of the expert judgment process.

I

' 39-l

8 i

i When an issue requires the expertise of several experts, decompositions are particularly useful. Teams of experts who collectively possess the requisite j

knowledge may be formed to address the issue. Each team member must embrace bis or her portion of a collectively acceptable model so that the team's judgments are coherent and based upon the same conditions and assumptions. In such a utting, the decomposition separates the issue into components that can be addressed by i

members of the team having the relevant expertise. De decomposition also is the basis for integrating the assessments of the team members. A team format where teams had the Dexibility to modify their models was und in a wismicity study of the Eastern United States (Electric Power Research Institute,1986).

3JJ.4 Benefits and Costs of Decompositions Decomposition beyond a point may detract from the quality of the information i

obtained. Decomposition should be done until a balance exists between the difficulty of the assessments, the complexity of the decomposition, and the inherent number of

)

assessments that must be made. In some instances, no decomposition may be desirabic.

Problem decomposition is beneficial in two ways. One is that the expert judgments obtained through decomposition may better represent the true state of knowledge i

about the problem. This is because simpler assessments can be made more i

accurately by the experts because their answers will be better calibrated.

Psycholog! cal biases such as overconfidence and the base rate >henomena are

{

thought to be less pronounced for easy tasks than more difJicult tasks, so decomposing into easier tasks may lessen the impact of these biases (Merkhofer, 1987; Lichtenstein and Fischhoff,1980). Mathematical recomposition of assessments relieves the expert of a difficult integration or aggregation task.

l t

he ucond type of benefit from decomposition is the stimulation of alternative views i

and the documentation of reasoning that follows naturally from a decomposition.

De use of multiple decompositions also helps explain why experts differ in their l

rationales.

Cost may be relevant when considering decomposition, he number of assessments may increase substantially because many questions may be required for a single issue.

Beyond this expenu, an additional requirement is that com!) uter programs or other methods be constructed to perform the recomposition. Tie dhersity of potential decompositions often precludes the use of existing software. Significant analyst i

effort is usually required to recompou an issue. Decomposition may also produce the false impression of objectivity and sometimes may introduce bias by systematically omitting an important component.

l 3.3.4 Techniques for Quantifying Probability Judgments Probability elicitation techniques are described in several references (e.g., Spetzler and von Holstein,1975 Selvidge,1975; Seaver,1978;'Keeney,1980; Stillwell, Seaver, and Schwartz,1981; Wallsten and Budescu,1983; von Winterfeldt and Edwards, 1986; Merkhofer,1987). In addition, several reviews of experimental validation of l

these techniques exist (Peterson and Beach,1967; Goodman,1972; Lichtenstein, I

+

y 8

Fischhoff, and Phillips,1977,1982; Slovic, Fischhoff and Lichtenstein,1977; Pitz and Sachs,1984). Drawmg on this literature, there appear to be four distinct classes of j

proceduua, depending on the nature of the uncertain quantity (discrete events vs.

j continuous sendom variables) and the nature of the questions asked (magnitude i

judgments about events vs. Indifference judgments about gambles). The resulting taxonomy is shown in Table 3.1.

j De eight techniques listed in this taxonomy are the most commonly used ones in the quantification of probabilityjudgments. Before describing these techniques in detail, l

r it is useful to spell out some general guidelines for probability clicitation that are applicable to all eight techniques.

I Table 3.1 Taxonomy of Probability Elicitation Techniques i

i Judgment VariaMe l

Magnitude judgments indifference judgments t

about events about gambles I

l l

l Discrete Direct probability Reference gambles (discrete) l Events Direct odds Certainty equivalent (discrete) i Continuous

' Fractile technique Reference gambles (continuous) i Quantities Interval technique Certainty equivalent (continuous) l l

(

i le when comparin the First,it is important to begin with easy questions. For examkav,e no feeling for the probabilities of two rare events, an expert may initially absolute magnitude of probabilities, but it may be fairly easy to establish a rank order of the relative likelihood of the events. Second, it is preferable to select observable quantities for eliciting probabilities. As an specific case, one observes failures of l

equi > ment rather than failure rates. Assessing the cumulative probability for the num >er of failures with 100 units originally operating for a fixed tiene period in extreme conditions may be easier than assessmg the probability for the likelihood I

(i.e., a parameter that an individual unit will fail in that time period with those conditions. Third,)it is useful to ask the same question in different ways and to use i

the results for consistency checks. These consistency checks should not be presented as a challenge to the expert, but rather as a means to stimulate thought and to improve judgments. Fourth, it helps to have computer support for decompositions, reaggregation, consistency checks, and displays.

I t

j 41 l

L

l 3J.4.1 Magnitude Judgments about Discrete Events j

i ne techni<jues described in this subsection involve two or any finite number of mutually canaustive and exclusive events to which probabilities have to be assigned I

by makmg direct numerical magnitude judgments. nese probabilities should add to j

one by vinue of the addition law of probability. For two events, one need elicit only

)

one of the probabilities, but it is good practice to check on the other one as well. For multiple events (e.g.,10 or more), it is usually worthwhile to reconsider the event space, either by clustering events or by identifying the continuous quantity that l

corresponds to the events. Frequently, with a continuous quantity, it is easier to i

construct probabilities for many events, since one can exploit monotonicity, single i

peakedness, and other properties of the probability distribution, i

Dirret hobability, his is perhaps the simplest technique. De elicitator asks the expert, "What do you think the probability is that this event occurs and why?" Often l

it is useful first to obtain a rank order of the probabilities of the events considered.

i in the case of two events, the first question may be which is more likely and why, l

followed by a judpnent of the magnitude of the robability for the more likely event, i

and finished by tic judgment of the probabili of the less likely event. Assuming i

that the two events are mutually exclusive an collectively exhaustive, these two probability judgments would, of course, have to add to one.

l For more than two events, there are two variants of this procedure: one can either I

ask the expert to assign arobabilities to each event separately without the constraint of adding to 1.0 or to < o so with that constraint. When time permits, it may be desirable to ask the questions without constraints and check the sum. His sum will often be larger than 1.0, since experts tend to overestimate probabilities, esycially 1

when they are small. Adjustments.will then be necessary so that the revised sum is 1.0.

Direct Odds. Sometimes the probabilities of events are hard to judge abstractly, but I

easier to judge in comparison. In this case, the normative expert can ask the j

substantive expert to state the relative odds of one event in favor of the other for i

l selected pairs of events. If there are two mutually exclusive and exhaustive events A and B, the expert would need only to state the odds O(A), in favor of A over B.

l From O(A) the probability of A can be calculated as P(A) = O(A)/(1 + O(A)},

from which the probability of B follows. Similarly,be calculated. However, as in thefor assign n 1 odds, and the resulting, probabilities can direct probability procedure,it might be useful to elicit n or more odds, point out the inconsistencies, discuss them and resolve them, 3.3.4.2 Magnitude Judgments about Continuous Uncertain Quantitles The uncertain variable in this category is a continuous numerical quantity,d has he technic ues described in this subsection also apply if the variable is dense an interva' quali. De two magnitude judgment techniques are mirror images of each l

other, in the etite technique the normative expert provioes the substantive expert with a proba lity and asks for a magnitude of the uncertain quantity such that the l

I _

s y

probability of the true value falling below it is equal to that probability, in the fired j

point technique, the normative expert provides the substantive expert with a set of fixed pints of the uncertain quantity and asks for the probability corresponding to thew 1xed points or for intervals in between them.

3J.4J Fractile Technique The fractile technique is the most widely used probability clicitation technique for continuous uncertain quantities, it is used to construct the cumulative density function of the uncertain q uantity that describes the expert's current state of l

of the uncertain quantity x such that knowledge. A a fractile is tsat magnitude x de falls below x and a 12 probability there is a probability of a that the true magnitu that k falls above it. De lower bound therefore should be the 0.0 fractile and the l

upper bound should be the 1.0 fractile. The cumulative density function simply plots the fractiles against the probabilities that the actual magnitude falls below it.

After carefully defining the uncertain quantity, the substantive expert is asked to state its upper and lower bounds. In other words, he or she should define two magnitudes i

such that there is absolute certainty that the true magnitude would fall in between i

these extremes. In practice, because a continous variable may have no obvious lower I

or upper bound, assessments may focus ora the 0.01 and 0.99 and/or on the 0.05 and i

0.95 factiles as relative extremes. After the initial extremes are defined, it is often i

useful :o ask probing questions. De substantive expert is asked to consider a i

hypothetical event in which the actual magnitude of the variable considered was if i

found to lie outside the range of extremes. Can this event be explained? Clearly,le any credible explanation exists, the extremes were not 0.0 and 1.0 fractiles. Credib explanations also provide a basis for estimating the probabilities of being outside the extremes. Such considerations can Jesd to tevisions of the initial extremes.

I After having obtained the extremes, the normative ex wrt pically moves to the i

middle range of the uncertain quantity and attempts to ic ent the magnitude of the uncertain quantity such that the substantive expert thinks the e ances are about 50 50 that the actual magnitude would fall above or below that value. This point is called i

the median or the 0.5 fractile of the cumulative density function. The answer should be probed, especially if it falls exactly in the middle of the range between the extremes (since this suggests arithmetic averaging) or if it is very close to one extreme (smce this suggests p)oor definition of extremes or a poor select scale and unit of measurement.

Having obtained three points of the cumulative density function (the extremes and the 0.5 fractile, the remaining tasks are to clicit between two and four additional l

fractiles. If th have not been determined in setting extremes, it is often useful to elicit the 0.05 the 0.95 or the 0.01 and 0.99 fractiles next. To obtain the 0.05-fractile, the normative rt asks the substantive expert to state that magnitude of the uncertain quantity that the probability of the true magnitude falling below it is 0.05. Finally, the 0.25 and 0.75 fractiles are commonly assessed.

Usually knowing the extremes and five fractiles is sufficient to sketch a cumulative density function. The normative expert should smooth a graph of this function and discuss its shape with the expert. In addition, it is very helpful to show the plot of the 43

_(

.l OV?@

.\\

f correspondin!he cumulative density function more clearly, probability dej asynunetries o(

l 3.3.4.4 IntervalTechnique a

la the interval technique the normative expert preselects points of the uncertain quantity and asks the substantive expert to assign Them >robabilities. Dere are two l

versions of this method. In the open interval version, tse substantive expert assigns l

penbabilities that the actual magmtude falls into the open intervals below and above i

r endi selected point. In the closed interval version, the substantive expert states the probabilities that the true magnitude falls between the preselected points.

Both versions of the interval technique begin with extremes, preferably bounds or the

/

0.01 and 0.99 fractiles, just as in the fracti e technique, in the open interval version, the normative expert then chooses three to seven points between and asks, for each point, what the probability is that the actual magmtude of the uncertain quantity is

(

above or below that pomt. Having obtained these probability judgments, the normative expert can then smooth a cumulative density function and proceed as with l

the fractile procedure.

in the closed interval version, the normative expert again lays out three to seven i

i points, possibly equally spaced, but this time asks the substantive expert to assign l

probabilities that the true magnitude falls in each of the intervals. De result can bc plotted both as a cumulative density function or as a probability distribution, it is i

useful to begin by rank ordering the probabilities of the intervals before assigning J

actual probabilities.

Both versions can be used in consistency checks, in addition, the fractile method can be mixed with the fixed point method it is quite easy, for example, to infer fractile-j I

type questions from interval clicitations and to construct interval type questions from i

fractile type results. For example, after constructing the 0.25,0.5,'and 0.75 fractile, the substantive expert should consider the intervals selow the 0.25 fractile, between the 0.25 and the 0.5 fractile, between the 0.5 and 0.75 fractile, and above the 0.75 l

fractile to be equally likely.

l 3.3.4J Indifference Judgments Between Gambles with Discrete Events l

De techniques discussed in this subsection derive probabilities from comparisons among gambles with discrete events and (usually hypothetical) monetary outcomes.

i i

Jteference Gamble Technique. To illustrate the reference gamble technique, the ex >ert is asked to select one of two gambles. he first gamble involves the event "It wil rain tomorrow" with unknown probability. If it rams, the expert will receive a I

stated priret if it does not, he will receive nothing. Alternatively, he can choose the

'i gamble in which he receives the prize with known probability p or otherwise nothing with probability 1 p. If the expert bets on rain, the probability p is reduced until the expert is indifferent between the two gambles. If indifference occurs when the probability is pi, this probability is assigned to the likelihood of the event because the expert should be indifferent when there are equal chances of winning the prize with both gambles.

{

i I

'44 l-

Cerssinty Egulveient Techs we. De certainty equivalent technic.ue is somewhat simpler in that it asks ont for comparisons between one gamb c and one sure amount rather than between two gambles. However, in order to use it, one must or assume) that the substantive is an expected value maximizer. To illustrate verify (hnique, consider again the g'stantive e amble for $10 if it rains vs. nothing if it does not.

the tec The normative ex wrt askEs the sub rt to state a certain amount of money at which he woulc be indifferent between i ng the gamble or takin less as a gift.

)

about this question, e normative expert could gin by asking To facilitate thinkinl;ive expert would prefer a certain amount of $1 over play' the j

whether the substan atica says that he would prefer to the ange t e certain amount to, say,59.p gamble. If the substantive expert em this gamble, the normative expert could l

point the substantive expert may consider the certain amount to be much more attractive. De normative expert then continues to vary the certain amount until the substantive expert is indifferent between the choices. At this point, the certam amount is said to be the certainty equivalent of the gamble.

Assume, for example, that the certainty equivalent in this case is $7. Den, by the assumption of the expected value principle, l

$7 = p(Rain)$10 + p(No Rain)$0 i

p(Rain) =.70.

or

. Similar schemes can be devised with multiple event gambles.

3J.4.6 Indifference Judgments among Gambles with Continuous Uncertain j

Quantities nis report will not describe indifference techniques for continuous variables as they

-l are direct extensions of the techniques for discrete events, ne main idea in applying i

these techniques to continuous quantities is to discretire these variables usin g ranges of' values and to apply the indifference techniques to the discretized events J

t (Matheson and Wink lct,1976).

i r

3.3J Techniques for Quantifying Value Judgments gments related to the performance of an HLW repository will Many expert gments, especially in screening scenarios and selecting conceptual include value ways important to make these value judgments explicit and document i

models. it is value ments l

them carefully, la some cases, it also may be important to quanti

aiffa,

von with multiattribute utility elicitation techniques (Keeney and Winterfeldt and Edwards,1986). These techniques range from simple rating functions.

l techniques to sophisticated indifference techniques to multisttribute utili This section describes two techniques with diffeient degrees o technical i

the sophistication that are applicable to the task of evaluating conceptual models:

simple multiattribute ratm technique (Edwards,1977) and an indifference technique r

to elicit a measurable mu lattribute value function (Dyer and Sarin,1979). These techniques are fairly similar in the basic task structure, but differ in the procedure of l

the clicitation.

45-t

~

here are seven steps in an evaluation-1.

Define the objectives for evaluation.

2.

Develop attributes and scales for ' measuring the objectives.

l 3.

Estimate the performance of the alternatives with respect to each attribute.

l 4.

Develop singe attributu value functions.

.l Develop weights for the attributes.

l 5.

l 6.

Convert the performance estimates of step 3 into single attribute values using sit) 4.

7.

Ca culate an overall value for the alternative, typically by a weighted average i

j using the weights in step 5.

ne simple multiettribute rating technique and the measurable multiattribute value function technique differ primarily in steps 4 and 5. In the rating technique, both single attribute value functions and weights are elicited using direct numerical rating jud gments. In the indifference technique, both elements are clicited using tradeoffs anc indifference judgments. Before detsiling these techniques, we will briefly discuss j

steps 1 to 3.

ne objectives hierarchy provides a logical structure of the objectives for evaluating the alternatives (i.e., conceptual models). We discussed some principles for constructing an objectives hierarchy in Section 3.3.3 on decomposition techniques for value problems.

Developing attributes and scales that measure the objectives in the objectives hierarchy is still an art. There are two types of attribute scales: natural and j

constructed. Natural attribute scales are numerical scales commonly used. For example, run time of a conceptual model may be def' ed in terms of seconds of CPU m

time. A constructed scale is needed when no natural scale is available or convenient.

l An example is scientific acceptability of a conceptual model. In this case a scale can i

~

be constructed that defines qualitatively (perhaps a paragraph or more) several distinct achievement levels. For example, the worst level could be defined as "a l

conceptual model that has virtually no scientific acceptability, only a few supporters, and very little published evidence su pporting it." ne best level could be delined as j

"a conceptual model that has very high scientific acceptability, many supporters of high scientific status, and significant published support." Similarly, intermediate i

levels could be definei De next sie > (step 3) estimates the performance or achievement of each alternative on each of the attributes. His is a nonprobabilistic version of an expert clicitation.

In the assessment of conceptual models, a group of experts may be convened who estimate attributes such as run time, scientific acceptability, cost, etc. If the l

uncertainty about these estimates is significant and if it a important to quantify this j

uncertainty,3.3J.plete probability distn

>utions should be elicited using the techniques com in Section With uncertainty, a multiattribute utility function, rather than a value funct% will be necessary to compare alternatives.

3.3.5.1 Sirapie Multiattribute Rating Technique To construct single attribute value functions with this technique, the worst and the best levels of the attribute scale are identified and arbitrarily assigned a ulue of 0 i

-46 i

n

o

/

and 100, respectively. For natural scales, several values between the wont and the best level are then selected and rated on the O to 100 scale. De resulting points are 1

plotted, and a single attribute value curve is fitted. For constructed scales, each l'

constructed level is rated on the 0 to 100 Scale. The same process is followed for all 34LF9putes.

l To obtain weights for the attributes, two hymthetical alternatives are constructed, one representmg all the worst attribute sea e levels, one representing all the best.

De expert is then asked to imagine being stuck with the worst alternative. Which i

attribute would he or the like to change most from its worst to its best level? Which is second, etc.? Dis ranks the value differences for attribute ranges between worst i

and best levels of the attnbutes.

}

i Nest, the att.ribute range that was ranked highest (i.e., which the expert would like to is assigned 100 importance points and an attribute range (not change the most) list) that is utterly unimportant is assigned O. All othet attribute i

necessarily in the ranges are rated between, according to their relative importance. De resulting raw l

range weights are normalized to add to one.

3.3J.2 indiffbrence Techalque Ibr Measurable Value Functions To obtain single attribute value functions, an indifference technique called bisection is used. De expert is again presented with the worst and the best levels of an t

attribute. Next, he or she is asked to identify a mid level of the attribute (not necessarily the numerical mid point) such that the increase in value obtained by stepping from the worst level to the mid level is ec ual to the increare in the value epping from the mid level to the best evel, his mid level is the value obtained

int, artdtrarily assigning a value of 0 to the wont level and a value of 100 to mid 51 lev, the value mid >oint has a calculated value of 50. By further bisecting the the range between the worst evel and the value midpoint, the value midpoint and the i

l best level, etc., a value function can be defined to any reasonably achievable detail.

t For attributes with natural scales, the results can be plotted as a value function. His process is repeated for all attributes.

I To elicit the weights, the expert is presented with two hypothetical alternatives that t

vary only on two attributes, while all other attributes are held constant at some level.

ne first alternative has the worst level of attribute A and the best of attribute B.

he second alternative has the best level of attribute A and the worst of attribute B.

i The expert is asked to state a preference for one of the alternatives. If the preference is for the first alternative, he or she is asked to worsen the level of attribute B in the first alternative until both alternatives are indifferent. if the preference is for the second alternative, the expert worsens the level of attribute A in f

the second alternative until both alternatives are indifferent. In either case, the clicitator assists the expert by providing easy comparisons along the way to 1

indifference.

f Once the indifference is established, the rehtive wei,ghts for attribute A vs. attribute B can be calculated assuming an additive value mocel. Let (a.b',c,d....) be the first o

alternative with the worst level of attribute A and the best level of attribute B, and let l

(s',bo,c d...) be the second alternative with the best level of attribute A and the worst of attribute B. Both have identicat levels e, d, etc., of attributes C, D, etc. If the first 1

s !

-,-r

-e

o alternative is preferred, then attribute B should be worsened to, say, level b' to achieve indifference, he indifference means that the overall values, denoted by v of the alternatives are now equal so v(a.b,c,d,...) = v(a',bo,c,d....),

d o

Using the additivity assumption, we can write wAvA(a ) + wa e(b ) + wcyc(c) + WDvp(d) +.. =

j v

o WAvA(s') + wavg(be) + wcvC(C) + wDVD(d) + '

• f i

and shnce, by definition, vA(4) = vs(be) = 0 and VA(s') = vs(b') = 100,

)

wA ws = va(b )/100.

/

Obtaininfdes the solution for the weights in this procedure.n 1 suc3 one prov 3.3JJ Aggregation Steps Step 6 is identical for both techniques it consists of a mechanical conversion of the performance measures obtained in step 3 into single attribute values using the results of either the rating or indifference technique. Step 7 also identical for both techniques aggregates single attribute values and weights to a wei ted sum. Having completed aTull cycle usmg these techniques for making'value gments, it is good practice to compare the calculated results with the experts intui non and to iterate.

3.4 Cannhinine Fwnert Judesments When using a panel of experts, there are three basic reasons to combine the iudgments of individual experts. De first is to provide a base case, or more than one Sase case, for analysis and sensitivity analysis m the performance assessment. The second is to gain msichts from the analysis for decision making. De third is to simplify analyses and, t ierefore, to save time and effort in acquiring these insights.

De sending on the types of judgments, combining expert judgment takes somewhat different forms. In the qualitative expert judgment tasks (identification and

, the combination consists of generstmg a joint list of things such as initial screening)d processes or screened scenarios. In probability udgment, individual events an probabilities or probability distributions are combined.

n value judgments,

ndividual functions or weights are combined.

3.4.1 CombiningIJsts De simplest approach to create a joint list is to take the union of the individual lists.

Often, the creation of the joint list involves some restructuring and some relabeling.

Such changes should be communicated to the ex wrts that created the individual lists, and care should be taken to assure that their in<ividual concerns are reflected in the 48 n

D u..$m 1

'oint list. Beyond these suggestions, however, there is little technical advice about W to combine qualitative mformation.

I 3.4.2 Combining Probability Judgments A key issue in combining probability judgments concerns what should be combined.

i The answer in almost ail cases is that the overall probability jud ments of the ments are j

individual experts or expert teams should be combined. These overall variables.

typically, a jomt probability distribution function over the set of tech Combining at this level recognizes that the fundamental unit in expert assessment is l

the state of knowledge of the expert. By combining across the complete re resentation of experts' knowledge, different experts can use different models, l

lo e, data, and p'rocesses to develop and represent their overall judgment.

l C

bining experts judgments at component levels in the process (e.g., combining j

marginal 3robability distributions) would put severe restrictions on the assessments i

of the ind vidual expert. Each of the experts would essentially have to go through the same reasoning processes and provide the same intermediate representations of

}

knowledge, in addition,if experts are in disagreement on their judgments and if the judgments are combined at component levels, you can develop situations in which I

the overall judgments of each expert would lead to a preference of an alternative A l

to an alternative B, but where alternative B would be preferable using the combined judgments (Raiffa,1968).

i 3.4J Combining Value Judgments I

As with probability judgments, the appropriate level of aggregation is at the level of overall utility functions, not at the level of single attribute utilities or value tradeoffs.

Dere are, however, additional problems.with aggregating utilities (Arrow,1951; Keeney and Raiffs,1976). These problems are a result of the difficulty of making l

impersonal comparisons of utility). As a practical solution to this comparab prob:em, Keeney and Raffia (1976 propose the concept of a supra decision maker that is to incorporate the valuejudgments of each individual decision maker. Using i

the supra decision maker model and making certain reg,ularity assumptions, it is

[

reasonable to aggregate individual (overall) utilities as a weighted average.

With value judgments, a fair amount of agreement usually exists about the general f

nature of the single attribute utility functions (see Section 3.3.4). In particular, agreement is likely to be found about the direction and the monotonicity of the utility function. If the utility functions have very different shapes, the underlying attribute may not have been clearly defined. On the other hand, weights are very personal expressions of value judgments and value tradeoffs. It is impossible to speak of "better" or " correct" weights. Experience has shown that in many controversial problems, the differences in value judgments appear as legitimate differences in weights (Edwards and von Winterfeldt,1987).

3.4.4 Behavioral vs. Analytical Combinatios The two general a >proaches to combining expert iudgments are referred to as the behaviora approaci and the analytical approach. With the behavioral approach, the experts on a panel are brought together to discuss and combine their judgments. In this process, the thinking, logic, and information of the different experts are l

l 49-l

nnm did j

f exchanged. This may brir5g about some reconciliation of differences and result in a single representation of the state of knowledge, or it may minimize the differences among experts. De behavioral approach seems particularly useful when the experts r

i have basic differences in fundamental assumptions upon which their judgments are based. In this situation, the interaction among experts promotes deep thinking about l

the problem that can lead to more thorough understanding and documentation. A possible serious disadvantage is that some exserts may be dominated or " forced" to l

i suppress their ideas to maintain harmony on tic expert panel.

t l

Analytical combination procedures are comprised of a logic and formulas consistent with that logic developed by the analysts (e.g., the normative experts) for combining individual judgments (Fischer,1981; Genest and Zidek,1986). The complete set of analytical combinations of expert judgmenti, that seem reasonable for consideration is the convex combination of the mdividual expert judgments. In other words, it is l

the set of additive weightings of the various expert's judgments such that the sum of the weights is one. One of these combinations is the average of the various experts' ments. Other combinations, in which the weight on one expert is one and ju we his on all the others are zero, are simply an expression of the state of knowledge e individual rated one. De obvious advantages to analytical combination of I

procedures is that they are easy to use, it is easy to do extensive sensitivity analyses t

around any base case combination, and individual experts have no influence on the judgments of other experts after the clicitation.

The most common analytical combination procedure is the average, in which all experts receive an equal weighting. A substantial amount of evidence suggests that this average weightmg often produces a reasonable base case for analysis (Seaver,.

i 1978; von Winterfeldt and Edwards,1986). However, some experience suggests that l

differential weighting techniques'to account for the relative expertise of mdividual experts result in a better combined representation of knowledge (Ashton and Ashton, 1985. One useful property of weighting techniques that positively weighs all indiv)idual assessments is that the ful range of the variable under consideration is included in the combined representation, in other words, the weighting does not i

eliminate the rang,e of diversity among different experts (Merkhofer,1987). This property of combinmg judgments is of particular concern in risk analysis.

r l

A combination of behavioral and analytical procedures can be used for combining individual experts' judgments. In this case, behavioral methods are first used. Here, the individuals exchange all their reasoning and data and assumptions upon which their judgments are based. If this process results in any changes of judgments by individual experts, the implications of these changes are included in updated representations of the individual expert's state of knowledge. If this process happens to lead to a commonly held tepresentation of the state of knowledge, then that representation of each mdividual should also be the representation for the group, if, l

after behavioral aggregation approaches, there are still residual differencer, between the individual experts, these can be combined by an analytical procedure as outlined above.

Regardless of how expert judgments are combined, the resulting uses of the experts' judgments should recognize tiree important items. First, any report should include more than one possible combination. This should facilitate hard thinking about the implications of different combinations and inform readers that there is no absolutely 50-

---,y-,,-.._,-e,..

-.m----.

,-----.----------,----.--,w'e--'

-4

-,-,,,w.-,,-,yy,-,

.,r.

--v-.

or

,w,---

E 0.w.j i

i correct way to do the combination. Second, different procedures for combinations may provide different insights from the analysis. For instance,if the combination is chosen that takes the "most conservative" estimate on any variable, the result should

[

be a theoretical bound on the "most conservattve" possible overall judgment based j

on the individual expert's judgments. If th: analysis indicates, for instance, acceptable implications with these conservative (,i.e., high) probabilities of failure, i

then perhaps no funhet analysis is necessary. Third, in all situations, the reported i

results should not be only combinations of the individual judgments. It is essential i

that the individual expert s judgments are also thoroughly reponed and documented l

as discussed in Section 3.S.I.

l

),g f%=municallne Funert _ludemments 3J.1 Documentation i

The reasons for documentin the use of expert judgment on technic I

specified by the following le xer review and appraisal, (4) to recognize and avoid communication,(3) to fac biases in expert judgments, (S) to indicate unambiguously the current state of r

knowledge about important technical and scientific matters, and (6) to provide a l

basis for updating that knowledge.

Complete documentation of the use of expert judgment would include both the interaction with the experts and the results (i.e., expert judgments) of that interaction. Thus, documentation would describe the selection of experts, the t

decision on whether to have expert teams, and whether to have panels of specialists.

i Documentation would include the selection of the specific issues to be add ressed by the specialists and how these were chosen. It would include the normative training I

about the methods used to elicit' expert judgments from the specialists and the

~!

preparation process to provide any necessary or requested substantive information to i

the specialists. Finally,) documentation would certainly inclul

(

reasoning to support them.

The fundamental unit of information of explicit expert judgments is the information i

provided by each expert. Hence, in any documentation, it is crucial to clearly

[

distinguish between the information provided directly by each expert and any interpolation, extrapolation, processing of that information, such as smoothing, drawing ofinferences from the combining of the judgments of different specialists, or judgments of experts. Maintaining, as part of the documentation, the individual expert judgments, potentially provides more information for decision making than if the information were aggregated (Clemen,1987).

The documentation of an individual's expert judgments should indicate what was done, why it was done, how it was done, who the mdividuals involved were and what their roles were, what the resulting judgments were, and what reasoning was used to suppon these judgments. The documentation should begin with a clear definition of the specific issue being addressed and should contain unambiguous definitions of all j

l the specific terms used in the clicitation. All assumptions about conditions that l

prevailed or would prevail that relate to the expen judgment should be stated. For mstance, if one is assessing judgments about ground water travel times, assumptions l

i 251-

,,,--+ -

+----,..v,,--..

.n.,_,.--,----,n-

~ - - -

1 about the particular rock types, the. amount of fracturing in the rock, and the tortuousity of the rock might be assumed by a given expert. IT so, these assumptions should be stated. De judgments as they are stated by the exped should be provided in the documentation. To support these judgments, the logic anC data on which they are based should be completely specified. Any calculatiors that the expert considered important in determming his judgments or mode's used should be indicated. Allliterature,whether public or restricted, should be specified.

It is also important to document the approach by which the exper, judgments were elicited. Some of this documentation may appear as a general sectit n ahead of many elicitations since the procedure used for many expert auessments would be similar.

However, the documentation would include both a description of the procedures and an exalanation of why they were used, as well as examples of their use. In some specific problems, it is important to document what was not done. If some professionals are likely to question the procen because of what was not explicitly done, clarification about why this was so may contribute to many objectives of documentation stated above.

The documentation should also indicate the types of consistency checks performed in the assessment of an individual's expert judgments. Invariably with complex expert assessments, such inconsistencies occur and are identified by these consistency checks. Dat is, in fact, one reason for going through a careful process to elicit expert judgments. Identification of the inconsistencies allows experts to understand their j

source and to adjust appropriately their judgments to account for this increased i

l understanding. The final, consistent set of expert judgments are those utilized in the performance auessment and this set requires the documentation just described.

L When a panel of experts is used for a problem, additional documentation is

_1 necessary. It is important to document how individual expert judgments are i

combined. The discussion in Section 3.4 indicates many guidelines for selecting a combination procedure. It is important to document the mdividual expert judgments in a common format and in the same format as the combination of expert judgments.

l The documentation should clearly indicate agreements and disagreements among the l

experts and the reasoning for any disagreements.

i Documentation can take significant time and effort. Hence, it is very important to begin with a system for documentation and a standard form to be used in documenting all experts. Because the specific issues addressed by different experts may vary, this form must be general enough to handle a wide range of specific problems, he responsibility falls upon a normative expert to document the results I

of any clicitation of expert judgment and upon the generalists and specialists to document the technical and scientific reasoning that led to those results. However,

)

once the documentation of an individual specialist's judgments is completed, it is important,that the specialist review, making any necessary adjustments and then approvmg it as accurate.

l Man factors need to be considered when selecting a dc-r - -Man $proach. Part of the documentation can include audio taping or video taping e clicitation sessions. With either, it is cuential to provide written documentation in addition. In situations where there are many separate individual clicitations,it would probably be better to have the documentation of some clicitations more complete and polished

\\

\\

l

52-

)

l

\\

ji than others. For example, with 100 elicitation sessions, each involving a specialist, a generalist, and a normative expert, it might be appropriate to have five of them carefully documented with a quality of writing appropriate for publication in peer-reviewed technical journals. The other expert clicitations should be documented i

with the same quality of logic, but not necessarily with the same thoroughness and style in writing appropriate for journal publication his would save a great deal of time in documentation, and yet provide the essential information for achieving the objectives of documentation stated above, j

The final issue about documentation concerns whether the experts should be anonymously treated or whether their names should be clearly assigned to their expert judgments. De main arg,ument to maintain anonymity is that some experts might feel a pressure to take the ' party line" of their organization if their name were associated with their judgments. With anonymity, they presumably could state what l

they really think. On the other hand, with the names ot ex xrts cIcarly stated along with their judgments, there is an additional motivation for t te expert to be clear and thorough and consistent. Naming experts greatly enhances the perceived quality of the analysis and the ability of others to appraise and utilize the expert judgments.

Indeed, ex >erts typically possess a strong sense of responsibility for their juc gments and a conEidence about them. In other words, experts are willing to stand behind their judgments and have these represented as such (Shanteau,1987). In the recent l

clicitation of expert judgments from approximately 50 experts in numerous disciplines for the NUREG 1150 project on the safety of nuclear power plants, only one indicated that he would prefer not to have his name attached to his judgments.

Because of the importance to the overall study of attaching the experts names to their judgments, one criterion in selecting experts should be the willmgness to have his or her name associated with the judgments.

l 3.5.2 Presentation of Results f

~'

The presentation of results of expert clicitations discusses and appraises the insip' hts from the expert judgments and their implications for decis'on making.

he l

objectives of this presentation are to inform decision makers and others about these implications and to have a constructive influence on decision making. The presentation of results of expert clicitations is distinct from the documentation of the elicitations. Documentation simply states the results of the expert clicitations, but 4

presentation uses the judgments of the analysts to appraise the relevance of the l

expert judgments to the decision faced.

It is important to recognize that the presentation of results is itself a decision

>roblem for which there are many alternatives (Keeney and von Winterfeldt,1986).

i

How deep the presentation is, whether illustrative examples are used to indicate insights, and whether the insights are expressed mainly in qualitative or also in quantitative fashion are alternatives for that decision problem. These alternatives involve factors such as how and how much to use cumulative distribution functions or probability density functions (Ibrekk and Morgan,1987), tables, diagrams, and decomposed probability trees. Alternatives also concern the degree to which there is comparability among the assessments of different experts. The presentation section may also contain decision analysis about the value of obtaining additional information regarding various uncertain phenomena investigated using expert 1

r l

53-

% nrv i

dMjb t

judgment. Key considerations in deciding on a presentation alternative include for I

whom and for what specific decision making purposes the presentation is prepared.

l For an HLW repository, the performance assessment provides insights for technical and licensing decisions and for communication to government officials and the public. Presentation of the results of the expert judgments should indicate how these judgments relate to whether the repository can be safely operated and meet legal standards. The presentation should indicate ci:arly which of these judgments are i

crucial to decisions on whether the repository can perform safely and legally. It should also indicate what changes in these judgments might lead to different implications and the bases that could lead to those changes in judgments. The presentation of results should clearly indicate which disagreements between experts are relevant to whether the repository can be safely and fegally o wrated, and which are important. Particularly for those that are important, it wou d be significant to indicate how one might resolve the disagreements among experts. His resolution might be possible simply with additional interaction among the experts, with additional experts, or only through additional gathering of data and scientific experiments.

3.6 lpterpretation. Una, and Misuse of Expert.Tudgments l

Expert judgments are crucist in the wrformance assessment of an MLW repository.

However, as is the case with al scientific work, expert judgments can be misinterpreted, misrepresented, and misused. To enhance the likelihood that this does not occur, it is important to interpret and use expert judgment in perf,ormance assessment appropnately, j

The formal use of expert judgment in performance assessment is a complement, j

rather than a substitute, for other source's of scie'ntific and technical information.

such as data collection and experimentation. Expert judgments should not be considered equivalent to technical calculations based on universally accepted scientific laws or to the availability of extensive data on recisely the quantities of interest. Expert judgrnents are perhaps most useful when are made explicit for problems in which site data are lacking, since they express what the experts r

know and do not know.

Expert judgments are a snapshot of the state of knowledge of the individual expert about the stated item of interest. As new data, calculations, or scientific understanding become available, these should be systematically incorporated within the existin g state of knowledge, nis learning process, which is a natural part of science anc knowledge, will result in changes in the expert's judgments.

Since different experts may have different information or different interpretations of l

information, there is no logical reason why various experts should have the same state of knowledge. For new and complex problems, a diversity of opinions might be expected. If such differences exist, these would clearly be identified in expert assessments. For a problem as important as the design and construction or an HLW l

repository, it is useful to know the range of expert interpretations.

Numerous expensive and lengthy projects have been suggested to investigate the physical conditions at a potential HLW repository site and the phenomena that affect l

1 34-m

M g.

+

J$

i those conditions. With the explicit use of expert ludgment, the value of the j

information derived from such projects can be calcuisted. His provides a sound l

basis for selecting projects that should be pursued. When one recognizes that the combined cost of proposed projects is several billion dollars, the significance of systematically appraising proposed projects becomes obvious, q

De main misuses of explicit expert judgments stem from misrepresentation or over-reliance on them. Expert judgments often have significant uncensinties, and it is f

critical to include these in the documentation. For example, just reporting, an sverage without a range or a, probability distribution for a quantity of interest gives i

the il usion of too much precision and objectivity. Expert judgments are sometimes inappropriately used to avoid gathering additional management or scientific j

information. These judgments should complement information that should be i

gathered, not substituts for it. Sometimes decision makers with a predisposed desire whose views support or justify their position.given design alternative s to prove the HLW site is safe or to select a His is clearly a misuse of expert judgments. However, it is worth noting that with formal expert judgments, it is easier i

to identify weaknesses in the reasoning behind a decision.

l In conclusion, it is worthwhile to remark on circumstances that should be considered successes or failures resulting from expert assessments. Science and knowledge are constantly changing. Dus, it is natural that as the knowledge of an individual

(

changes, his or her expert judgments will likely change. De representation of expert judgments as probabilities and utilities facilitates adjustments to account for new i

L information. Even after the completion of a given assessment, an expert may recognize that he failed to account for some important information. De assessment process is designed to enhance the likelihood that such omissions are recognized.

Then it is easy to update the overall expert judgment to account for the omission.

j l

De ability to change and the need to change expert assessments are not failures of the experts, the~ assessments, or the assessment process. Rather, they are natural and desired features to deal with the reality of science and knowledge for a complex problem such as an MLW repository.

After the explication of expert judgment, someone or some organization may wish to f

demonstrate that some of the assessments are not correct. For exa,mple,if rome j

organization felt that the groundwater flow parameters near the repository site were incorrect, they might begin additional experimentation or search for additional l

information that would support their point. If this led to a process that eventually improved the overall state of knowledge, that would not be a failure of the assessment process. Rather, it would be one of the desired products of explicitly eliciting expert ju ments. Because the overall intent of the expert judgment assessments and of riormance assessment is a safe and legally operated repository.

The formal use of expert judgment in the performance assessment of an HLW repository contributes to understanding, learning, communicating, and decision makin g. In the final appraisal, the significance of the explicit use of expert judgment

~

shoulc be evaluated by the overall value it adds to the performance assessment.

Naturally, this is the same criterion applied to any of the mputs for or aspects of a performance assessment.

l c55-

i 4.

SUGGESTIONS FOR THE USE OF EXPERT JUDGMENT IN HLW j

DISPOSAL t

This chapter specifies how to apply the techniques for eliciting and using expert judgment discussed in Chapter 3 to the five problem areas of HLW disposal outimed m Chapter 2: scenario development and screening, model development, parameter and strategic repository decisions. Some of the i

estimation, information gathering,ive areas, and others are relevant only to single techniques apply to each of the f areas. For each of the five areas, experts must be selected and trained for the clicitation process, an appropriate clicitation process must be designed, and results must be thoroughly documented and presented.

For scenario development and screening, identification and screening techniques are directly a f

hi h b

ilities are then Assessed. pplicable to produce the set of scenarios or w c pro a)

For model development, the identification and screening techniques are initially most relevant to se ect the variables to use in the conceptual models. Techniques for quantifying values may also be relevant to evaluate alternative models. Then mathematical models are developed to quantify the conceptual models. In this process, information gathering techniques are utilized as well as parameter estimation, both of which are addressed in the descriptions of the two problem areas that follow.

The main techniques in parameter estimation are screening to select the key parameters and quantification of the uncertainties in the form of probability distributions for those parameters.

Information gathering provides better in' formation for the other areas of scenario l

and 'model development and parameter estimation. Information gathering uses L

techniques for identifying and screening information gathering strategies and for quantifying probabilities and values.

t Strategic repository decision making can use all the techniques described in Section l

3. First there is the task of generating alternatives for the construction and operation of the repository, which can use identification and screening techniques. Decision and event trees are next used to decompose the alternatives and events in a logical sequence. Objectives hierarchies are used to decompose the oojectives that are relevant to evaluate the outcomes of decision and event sequences. Probability quantification techniques are used to assign probabilities to events in the decision tree, and utility quantification techniques are used to assign utilities to outcomes.

l Then decision analysis can be used to develop insights for decision making.

l l

4.1 Sc==-lo Develonment and Screenine SNLA's methodology for development and screening of scenarios that hypothesize the possible future states of the disposal system was described in Section 2.1. The methodology consists of the following: (1) identification and classification of events and processes, (2 screening of events and processes, (3) formulation of scenarios, and (4) screening o)f scenarios. In addition, we discussed earlier the need to es the likelihood of occurrence of eac scenario to demonstrat:: compliance with the 1

L....

. Below we containment requirement in the EPA Standard (40 CFR Part 191.13)3 to each of present guidelines for the applying techniques described in Chapter these areas.

4.1.1 Identification and Classification of Events and Processes e

The main objective of these tasks is to arrive at a comprehensive list of events and processes from which the scenarios are formulated. A secondary objective is to classify the events and processes to increase the likelihood that the list is indeed

)

comprehensive. His classification should also be useful for organizational purposes.

The group of experts that prepares the list of events and processes needs to be interdisciplinary. The experts should be specialists that have substantive knowledge in at least the following disciplines: general cology, seismicity, volcanology, h rology, and mining and/or rock tectonics, resource exploration, climatology,b havior (e.g., human intrusion) can mechanics. In addition, since future human strongly influence, and indeed create, future scenarios, the experts should also include historians, sociologists, and psychologists knowledgeable about issues of technological change. It should be noted that these s >ecialists should not be required tc, have in-depth knowledge of nuclear waste disposa issues; the s secialists should be complemented by generalists (i.e., experts with general knowiec ge in performance assessment). Generalists show the specialists how their judgments contribute to the perforraance assessment.

Section 3.2.3). De De experts should be sensitized to biases, primarily availability (f the experts to re bias of availability in this context refers to a possible tendency o too heavily on existing records that do not nqcessarily represent the future adequately. De expert,s may not allow for adjustments to the existing information and may need some training from the generalists on performance assessment and how their judgments will be used.

The particular clicitation techniques applied in the identification and classification of events and processes were described primarily in Section 3.3.1. forward and backward induction, value driven identification, and analogy / antimony driven identification. We believe that more than one clicitation technique should te used to enhance the likelihood that the sets of events and processes are comprehensive.

The approach should be documented so that interested individuals may clearly discern the ratior.sle of the clicitation process and the results. Intermediate lists as well as the final list of events and processes should be presented and should also include the steps to go from or e list to another if multiple lists preceded the final one. An additional advantage of distributing the sets of events and processes is that any omitted examples may be identified and then, of course, added to the list.

4.1.2 Screening of Events and Processes The basic problem is to screen out insignificant events and processes from the list generated m the previous step. While the list of events and processes should be generated generically as well as specifically for each site, the screening out of events and processes by necessity must be site specific. To screen out events and processes, screening criteria must first be formulated and applied to arrive at a " final" list of 57-

('..

t events and processes to be used in formulatin,g scenarios. De importance of both steps cannot be overemphs. sized. If the screenmg criteria are developed poorly, then the likelihood increases of eliminating potentiall significant events and processes and/or of including insignificant ones. If the cri cria themselves are not applied correctly,is defeated.the same consequences are possible. In either case, t screening The s ecialists selected for identi events and processes can also be used for.

identil i,ng screening criteria ney s obid be trained specifically to overcome such overconfidence" and " availability" (Section 3.2.3).

The clicitation techniq ues for screening events and processes are discussed in Section 3.3.2. The first part of the clicitation exercise should concentrate on developing the screening critena based on physical reasonableness, potential consequences, and i

likelihood of occurrence, ne second aspect of the clicitation exercise should focus on setting reasonable constraints for the screening criteria. For example, in dealing with the likelihood of occurrence of a given initiating; event or process, what probability of occurrence is too low? The last part of tie exercise should be the application of the screening criteria. Multiattribute utility ana is (Section 3.3.5) is an approach for explicitly making tradeoffs between the dif erent criteria. it is important to point out that iterating through the target levels and constraints in the criteria is recommended as a mechanism for determming the impact that these may have on the final list of events and processes.

The documentation and presentation of results mainly explains clearly the logic of the approach used in sufficiently general terms that it can be followed and critically i

reviewed by a wide range of interested parties. De documentation should allow not only critique of the approach, but of the results as well. The result should be a final list'of events and processes that will be combined to form scenarios.

4.1.3 Generation of Scenarios Once unimportant events and processes have been eliminated from further consideration, the surviving ones are combined to form scenarios. His step can be conducted by generalists knowledgeable about the application of event trees. The forward and backward induction techniques described in Section 3.3.1 and techniques for combining may be useful.

4.1.4 Screening of Scenarios ne guidelines for using expert judgment in this step are identical to those described in Section 4.1.2 for the screening of events and processes, ne problem is to reduce the number of scenarios for the performance assessment to a tractable and representative set. His is accomplished by aggregating scenarios and by developfmg and applying screening criteria as in the screening of events and processes. The sical reasonableness, potential screening criteria should again stress p consequences, and likelinood of occurrence.

e selection and training of experts, the clicitation techniques, and the documentation and presentation of results should be identical to that in Section 4.1.2.

S8-

Q s

4.1J Probability of Scenarios ne problem to be addressed by the experts in this step is tw estimatin and i

,i combining these probabilities to arrive at the probability of the scenario. To estimate the proba sility on the individual events and processes, the experts need to identify the i

initiating event or process and decide whether the occurrence of the other events and l

processes in the scenario are conditional on the occurrence of the initiating one.

Dis step requires a multidisciplinary team of specialists with substantive knowledge seismicity, tectonics, volcanology, climatology, hydrology, rock in general geolo6y,ing, etc. Generalists with knowledge of performance assessmen mechanics and mm can provide insights on what type of scenarios are hkely to be more si nificant.

t Finally, normative experts with experience in probability clicitation are needed to train the other groups of experts as well as to serve as the clicitators.

1 The specialists should be trained in overcoming probability biases (mainly and availability), decomposing, expressing judgments overconfidence, anchoring,ing, and assessing conditional probabilities. Tic specific explicitly, probability encodclicitation techniques applicable to this step are the uantification techniques described in Section 3.3.4. The techniques for estimating iie 3robability l

of discrete events such as the direct probability technique or the d, rect odds technique may be particularly useful.

4.2 Model Development ne development of models for performance assessment includes the development of conceptual models, mathematical models, and associated computer codes. His effort involves the selection and interpretation of available data and other sources of information, the formulation of relevant assumptions, and confidence building in the models and codes developed. Each requires expert judgment.

4.2.1 Data Selection and Interpretation His task mainly provides the basis for the formulation of conceptual model(s) of the disposal system. Expens r, elect and interpret data and other mformation that will lead to the establishment of the system's geometry; boundary,and initial conditions; and past, p(resent, and future events and processes that m the system Section 2.2.1).

It is expected that specialists, generalists, and normative experts will be required to 1

carry out this task. Specialists primarily should concentrate in the fields of geology I

and hydrology; however, some specialists involved in the identification and classification of events and processes in the scenario development (Section 4.1.1) should also be used here. Generalists who have participated in earlier or preliminary performance assessments of HLW disposal sites should be used in this task.

Generalists should be able to provide insights regarding the relative importance of different types of data and information based on their past experiences. Normative experts should assist the specialists in searching and cataloging different sources of mformation.

.59-

BRAR The clicitation exercise is likely to be in three phases. In the first phase, the specialists and encralists ident both site specific and eneric sources of data and other informat on. For this ase, the experts shou be trained to overcome

" availability" bias (Section 3.

. The specific clicitation techniques relevant to the identification task are presented in Section 3.3.1.

In the second phase of the clicitation, the experts must screen out unimportant sources of information and select the most relevant ones. To achieve this goal, criteria must be developed to accomplish the screening step, and then these criteria need to be applied to arrive at the most relatively important sources of data and information. This phase of the clicitation is similar to that discussed in Sections 4.1.2 and 4.1.4 (Screenmg of Events and Processes, and Screening of Scenarios), ne training and clicitation techniques are similar to those suggested in Section 4.1.2 and are presented in Sections 3.2.3 and 3.3.2.

The third phase involves the interpretation of the selected information. In this phase, the experts make inferences based on this information that will form the basis for the development of models. The ex1erts should be trained to overcome biases ignoring >ase rates, and nonregressive predictions associated with availability, fers here to the tendency to follow a conventionalline (Se.: tion 3.2.3). Availability re of reasoning when interpreting the available information without considering evidence that may challenge this convention. ignoring base rates as applied to data J

interpretation refers to ignoring soft or abstract information while focusing only on concrete evidence and data. Noreregressive prediction is the tendency to make established for tie system in question.plicability and validity of which h inferences using; relationships the ap 4.2.2 Development of Conceptual Mo$els Constructing cor.ceptual models 'uses inferences based on the selection and I

interpretation of data to formulate assumptions for the behavior of the disposal system. These assumptions, in turn, are the cornerstone for the assembly of mathematical models and their computer codes used in the quantitative analyses.

Modeling most likely will result in a multitude of alternative conceptual models because of the lack of data during the early stages of a site investigation. As more information becomes available, it could be possible to disting,uish among the different conceptual models and possibly reduce their number. Fmelly, it wou d be feasible, if a number of conceptual models survive screenin, to quantify a relative likelihood for each conceptual model that it adequatel describes the "true" groundwater flow and transport processes, for instance.

i Again, specialists, generalists, and normative experts will probably be needed. The i

specialists should be in the area of drology_and should include both modelers and experimentalists as will be discussed low. Generalists should be used to assure that the specialists render ju ments within the context of performance assessment.

Normative experts should used to assist the specialists in making value judgments.

(

Some of the experts used in data r, election and interpretation should be mvolved in this task to provide continuity. Multiple teams of experts may be appropriate.

l-The first phase of the clicitation is the development of meaningful criteria for the formulation of assumptions and the construction of conceptual models. These i V

u

'd

-.., a criteria include beliefs regarding the importance of model attributes such as the ability to simulate specific events and processes, groundwater flow geometry, levant parameters, complexity, etc. De selection of these criteria is likely regime, re tok based on value judgments and will require all three types of experts. While the specialists should be expected to play the biggest role in this shase, generalists should provide the basis for acceptable tradeoffs tiat can be mace in light of regulations that need to be addressed in the performance assessment. Normative experts are

'likely to be clicitators. Techniques for express.ng value judgments are described in Section 3.3.5.

De second phase is to develop a procedure for distinguishing among the alternative conceptual models and, if possible, screening some out. His should be accomplished by attem sting to identify the salient features of each conceptual model, formulating and conc uctmg specific analyses and experiments that could test the validity and/or im,portance of these features, and setting screening criteria and applying them. In this phase, both specialists in model development and experimental studies are needed because a synthesis of analyses and experiments will likely be necessary.

Screening techniques described in Section 3.3.2 should be useful in this phase.

The third phase consists of an attempt to quantify the likelihood that each conceptual model that survives screening is the best of the available models. Specialists and normative experts will be needed in this phase, and probability clicitation tools such as sequential conditional probability assessment and others presented in Section 3.3.4 are applicable to this phase. Ap ropriate training to overcome such biases as

(

overconfidence, anchoring,before the licitation., and ignoring base rates, d l

availabil

{

3.2.3, should be conducted A po:tfolio of conceptual models should be cl$osen that, at the very least, represents extreme sets of conditions for a performance assessment and that, at the same time, can be tested during site-characterization investigations. Situations in which two or more conceptual models are very similar should be avoided. Refinement of the final t

portfolio of conceptual models can be done using decision analysis and, in particular, preposterior analysis (Winkler,1972). These techniques increase the likelihood that the set of conceptual models selected is adequate for conducting a performance assessment, the results of which will allow making regulatory decisions with confidence.

1 4.2J Confidence Building Following the development of conceptual models, mathematical models will be formulated that cast the models in terms of mathematical equations (i.e., algebraic, partial, and/or integral equations. In setting up these equations, assumptions are made, the validity of which needs to be established. Typically, because of the i

com >lexity of the equations in even the simplest models to simulate the behavior of an F LW disposal system, the solution to these equations is implemented in computer codes. Dependin g on the nature of the equations (linear vs. nonlinear, partial vs.

anc the coupling between two or more equations, these can be solved algebraic, etc.)ically or numerically. In any case, the implementation of neither E

either analyt analytical solutions nor numerical solutions is exact. For example, if an analytical solution involves an infinite series, this series needs to be truncated after a finite number of terms, or if it includes a complex integral, this integral is often evaluated I

l l.

1-

4 4

\\

!~

numerically. Numerical solutions inherently are approximations to the "true" solution of the equation (s). In whatev' r form (either analytical or numerical), errors e

are introduced when solving the equation (s) in a mathematical model. Since the j

L validity of these mathematical models and computer codes cannot be established l

over the temporal and spatial scales of interest in HLW disposal (Section 2.2.3),

d validation cannot be achieved in the truest sense. Nevertheless, confidence must be built to the extent that, given the present state of the art, these models and codes are deemed adequate for the job at hand: predicting the behavior of the disposal system over several kilometers and 1:ns of thousands of years. To build confidence m the will play a major role in designing and conducting these activities, pe models and codes, limited scope activities will be carried out, and ex as well as in

. interpreting the results, Experts are likely to be used in selecting important features in the models to be p

tested and the type of testing. For example, there may not be a need to test the p

expression for radioactive decay in the radionuclide transport equation because this is a well established and accepted expression. On the other hand, the use of a l

Fickian model for diffusion to represent dispersion or the use of a linear sorption-L equilibrium based retardation factor are both models that are the subject of much L

criticism and should be tested. De question then becomes what tests to conduct, for example, laboratory vs. field tests. Experts will also be involved in the selection of appropriate criteria to establish the measures of goodness of the models. These are competing measures, and experts should select those criteria that are most l

meaningful to the regulatory requirements to be addressed. De experts must also set the limits and constraints in these criteria. Experts will also be needed to assess -

l-the ability of the models to extrapolate from the temporal and s patial scales at which l.

they.were tested to the scales of interest in HLW disposal. Finally, there are likely to be some couplings in the models that are so complex it is impractical to test their validity. In this case, expert judgment assesses the adequacy of the modeling of these coupimgs.-

The experts required include primarily specialists and generalists; however, it may be appropriate to include normative experts, but this may not be necessary. It is sugg,ested that multiple teams of experts be used, each team consisting of both specialists and generafists, and modelers and experimentalists.

tradeoffs regardingwhat aspects of models need ne experts make value judgments (Section 3).3.5 should be useful. In addition, they to be tested, and the techniques in develop criteria for establishing the validity of given models. Therefore, the techniques for setting criteria, limits, and constraints to the criteria, and the applications of the criteria in Section 3.3.2 should be employed. As the " ultimate" va:idation test at an HLW disposal site cannot be performed and because of the complexity of the model, perhaps one of the biggest tasks to be faced by the experts l

of the overall system model into requires the decomposition (Section 3.3.3)d that there are likely to be couplings L

meaningful pieces. While it has been recognize i

that cannot be tested, extreme care must be taken to assure that the decomposition

~

of the problem does not eliminate significant couplings. For example, in testing for the validity of the linear-sorption-equilibrium model as the dominant radionuclide retardation, the problem should not decompose such that a test is conducted that does not include flow-field effects because evidence exists that they have a significant impact on sorption.

L l-

'62-

~

l

~,

4J Parameter Estimation 4.3.1 Identification of Parameters As stated in Section 2.3.1, parameters are embedded in conceptual models that predict the performance of'the repository in terms of radionuclido emissions and their potential health effects. Therefore, the importance of parameters is closely related to the variation in the amount of radionuclide emissions relative to variations in the parameters. The main method for identifying and selecting parameters is 4

sensitivity analysis. In such analysis, parameters of conceptual models are systematically varied (both individually and in sets) to determine which parameter or combination of parameters has the strongest impact on radionuclide emissions.

Sensitivity analysis is currently more a craft than a science. It is therefore especiall,y important that the expert judgments that select and interpret the sensitivity analysis for parameter identification are made explicitly.

4.3.1.1 Guidelines fbr ParameterIdentification l

l At this sta,ge of the analysis of the HLW disposal problem, the issues for parameter identification are typically fairly clear cut: Given a chosen conceptual model, what l'

are its parameters hat should be quantified for further analysis. There may be two I

complications with this problem definition that may require resolution before identifVing important parameters. First, there may be several conceptual models, and second, there may be different ways to categorize parameters. If these complications occur, it is useful to convene an ex pert panel to address these issues before the actual parameter identification process. Guidelin~es for issue identification and selection of experts for this part of the study should be followed (Sections 3.2.1 and 3.2.2). In particular, a diverse set of experts and examination of a diverse set of conceptual models and sets and subsets of parameters should be considered.

Once a conceptual model and the possible parameters and their subsets are agreed I

upon, identifying "important" parameters is more technical and better defined.

Three types of experts are necessary identifying important parameters: Substantive experts with knowledge of geology and hydrology, among others; generalists with expertise in the conceptual models; and experts m sensitivity anafysis. An effort should be made to obtain the best expertise in these areas, as well as to maintain some diversity of opinion. This diversity is especially important for the experts concerning the conceptual model, as they are likely to disagree a priori about what constitutes important parameters of the model. Less emphasis on diversity is needed in selecting experts in hydrology and geology, and even less in selecting experts in sensitivity analyses.

Training in clicitation techniques is not required in this area. However, both the substantive experts and the sensitivity analysts need to learn about the nature of the conceptual model, its assumptions, its behavior and some of its preconceptions about sensitivities. For the substantive experts, this may provide guidance for reformulating parameters (e.g., by dividing hydraulic conductivity into separate strata). 'Ihis type oT training alerts sensitivity analysts to possible interactions among parameters, as well i

o n.

7 as to possible problems and opportunities in carrying out sensitivity analyses. This l

training should consist of two parts: presentation and familianzation with the L,

conceptual models and some of their predictions and extensive question and answer pen'ods regarding the use of the conceptual models.

Because sensitivity analysis plays a key role in identifying important parameters and l

because the clicitation centers around a conceptual model, the clicitation session should be structured somewhat differently from the standard session described in Section 3.2.4.~ In particular, display and discussion of sensitivity analysis results of l

running parts or the complete conceptual model should be emphasized.

Comparatively less time should be spent in mdividual clicitations, and the amount of I

i actual numencal clicitation should be fairly small at this stage.

There are two sul;gestions for structuring an clicitation session in this context, depending on whetier sensitivity analyses can be done on-line. If they can be done on-line, it is highly desirable to structure the clicitations as an interactive exercise in which the experts formulate hypotheses about sensitivity and importance and test them in real time. Some structure should be provided to make sure that the more prominent hy@ theses are tested and that all parameters are examined. Beyond that, the experts smould be able to develop their own plan for carrying out sensitivity analyses and judging their outcomes.

If sensitivity analysis cannot be done on line, the experts should convene at least twice 'The first meeting determines which sensitivity analyses should be carried out.

The second meeting discusses the results of the sensitivity analyses and makes judgments about which parameters are important enough for further quantification of uncertainties. If certain parts of sensitivity analyses can be done on-line, this should be done to liven up the exercise. However, care should be taken that the on-

-line sensitivity-analyses do not gain more prominence by making the respective parameters more available to the experts (Section 3.2.3).

1 In both cases (dgments about the parameter:on-line vs. prepared sensi making three ju 1.

Sensitivity related to selected performance measures; 2.

CNerallimportance; 1

3.

Need for further quantification or data collection.

4.3.2 Quantification of Parameters A fairly large amount of research and applied work exists for quantifying expert judgments about uncertainties in parameters with probability distributions. The recommendations that follow are therefore grounded in significant amounts of experience (Section 3.3.4).

4.3.2.1 Guidelines for Quantifying Parameters After the conceptual model and its important parameters are identified, the issue is to quantify the knowledge of substantive experts in hydrology and geology about the i

1 64-

=

^ '*

parameters at, probability distributions. Price to any assessments, is useful to identify c

current or near future data collection efforts, to put the actual ex>crt clicitation of uncertainties before this data collection into perspective. In ac dition, it is very important that the parameters be unambiguously defm' ed.

1 Parameter quantification addresses specific issues such as th estimation of hydraulic conductivity parameters in specific strata of the repository. Experts should be selected on a parameter-by parameter basis. Depth of knowledge is crucial, breadth and diversity are secondary m this case. Motivational biases should be considered.

For example, a hydrologist on record as stating that Yucca Mountain is an absolutely safe site for the repository might give estimates of hydraulic conductivity that are too low. It is useful to counterbalance such potential biases through expert selection.

i Training should focus on constructing (usually continuous) probability density functions (pdfs) or cumulative density functions (cdfs) over parameters. The main recommendations in Section 3.2.3 apply with full force here. In particular, experts o

should be familiar with the probabi'ity clicitations task, and they should ;;et ample practice using many examples of the ty>cs of clicitation that they are like y to face.

Anchoring and adlustments, overconfidence, and motivational biases should be demonstrated, and debiasing procedures should be explained.

All experts must agree on the precise definition of the parameter to be elicited, For i'

example, when hydraulic conductivity is discussed, it must be absolutely clear which strata of the repository is referred to, whether one wants to assess mean or maximum what maximum may mean, etc. It is useful to structure the hydraulic conductivity, involve a "generalist" knowledgeable about the conceptual clicitation session to model and the interpretation of the parameter within that model.

l A variety of decomposition techniques may be useful, depending on the specific parameter or, the expert (see Section 3.3.3). If functional decompositions are utilized, direct probability assessments should be used as consistency checks for probabi!ities calculated based on decomposed assessments. For example, when assessing h draulic conductivity in four different strata and subsequently assessing average h raulic conductivity, the results can be checked for consistency with the average h raulic conductivity.

Parameters should usually be represented as continuous random variables.

Therefore, our suggestions for applying elicitation techniques are very straightforward: use the fractile techmque described in Section 3.3.4 and check it with the interval technique and perhaps a few {; amble questions. Pay particular attention to the extremes and prose them careful y, possibly by considermg physical impossibilities and extreme gambles. For example, when considering hydraulic e

conductivity, the clicitator may ask for the expected minimum and maximum areas of conductivity in the repository, for the minimum and maximum in comparable formations, and for the minimum and maximum in a variety of substances and materials. An appropriate range should then be selected. By broadening the notion of minima and maxima, the expert may be induced to consider the full range of possibilities for the case at hand as well.

Having obtained a first cut range, the normative expert should ask the specialist to explain a set of hypothetical data that indicates events outside the range. One may 65-

pp%

t El k also ask the expert, whether he or she.would be willing to bet a large sum of money t

that all possible experiments would lead to the conclusion that the parameter is in the range stated. Both techniques are useful for debiasing.

4.4 Infhr==ation Catherine i

To better design, construct, and operate a nuclear repository, numerous important decisions must be made, many of which will affect repository performance. To improve the quality of these decisions and to improve performance assessment, numerous efforts must be carried out to gather information. Collectively, this information will be very costly. In terms of dollars, the cost will be in the billions; in terms of human resources, the cost will be in the thousands of person-years of professional time; and in terms of the environmental and social disruption of the testing to information gather, there will be significant effects. Thus, decisions about information gathering should be made carefully and thoughtfully. Information scenario development, model gathering cuts across the three areas discussed earfier:

development, and parameter estimation. In all of these cases, the information is intended to im nove the quality of the scenarios, the usefulness of the models, and the estimates of the parameters.

Information gathering is also different from the first three areas in that it concerns decisions. Some important decisions concerns how many,f decisions concerns whathow drill test holes into the repository media. Another class o computer codes should be developed and what conceptual models should be fleshe out mto analytical models. For example, should the groundwater flow models be in' two dimensions or in three dimensions, and what variables should they include?

Regarding parameters in these models, how can we best estimate a variable such as porosity to reasonably balance the' insight gained about the variable against the cost and effort necessary to gain that insight?

Ralffa and The use of the concept of expected value of sample information (information

)

allows appraisal of the various alternatives for gathering Schlaifer,1961)f the one that is best given expectations about what information might

)

and selection o be obtained from the various alternatives and about the economic cost, time required, and damage caused by that alternative. Value judgments must balance the advantages and disadvantages of gathering the information and take into account the l

overall goal of creating a safe, legal repository.

In the rest of this section, three special classes of problems concerning information l

gathering are discussed. These problems concern informational drilling, J

development of models, and conducting laboratory or field experiments other than drilling.

4.4.1 Informational Drilling The informational drilling program is one of the major activities in the characterization of the repository. It should be carried out only after careful appraisal of the alternatives. To do this, there are several distinct activities that should be completed that rely partially on the use of expert judgment.

I

'.66-i

• p 1

i To characterize the informational drilling problem, the objectives of the drilling program and reasonable alternatives first need to be identified. Then the r

alternatives should be screened to specify the competitive options. For each of these competitive options, estimates are necessary for the information that will possibly bc

'l learned and for the time, cost, and damage caused. Using value judgments to balance these and the concept of the exsected value of sample infurmation, an i

l analysis can indicate the relative desirabiity of the options under a wide range of l

assumptions. Esch of these tasks are elaborated below.

The first and driving task for the informational drilling program is to specify its i

objectives. It is important to be very explicit about the relative desirability of different information that might be learned from the pro ram. In this step, i

s ecialists need to be selected to assist in specifying the ob ctives because the l

ectives of the drilling program willlikely be technical. It is al important that the dr lling objectives be logically related to the fundamental obj'ective of better designmg, constructing, and operating a safe and legal repository. Diis relationship may be best specified by generalists with a broader understanding of the repository program. The techniques for structuring objectives hierarchies are useful in this task (Section 333), and careful documentation and review of the objectives hierarchy is appropriate before completing the additional tasks below.

The second task is to identify a large number of reasonable alternatives for gathering information via drilling. To develop these alternatives, specialists and generalists should again be used. At this stage, the alternatives need not be carefully refined (e.g., the exact location of each hole), but they should be specific enough to distinguish them from other alternatives.

The next task is to screen the large number of alternatives to identify those that are comp'etitive. The relationship of the objectives of the drilling program to the fundamental objectives of the repository should be a basis for this screening. The screening criteria should at first be specified by generalists using techniques discussed in Section 33.2 and then be used to eliminate many noncompetitive alternatives. At a later stage in the analysis of information drilling options, when the relative desirability of alternatives that passed the screening are known, the screening criteria should be reexamined to determine whether more related screening criteria might have yielded better alternatives. The way screening criteria can be verified with information that comes later in the analysis is out ined in Keeney (1980). If the appropriateness of the screening criteria is to be verified, the original use of expert

+

Expert judgment is judgment to set the criteria for screening is not so significant.

crucial not only to screening but in setting up the relationships of objectives of the various drillmg options m the next task.plications of what might drilling program and in specifying im The fourth task is to defm' e better the competitive options that make it through the screening. There are two aspects to this definition. ' Die first is to specify exactly what drilling will occur, and the other is to predict the possible information learned from the drilling and its time, cost, and resulting damage. This task relies heavily on Some of it will be from specialists, specifically information expert judgment.

re1 erring to details learned about the hydrology and geology at the site. Other information will necessarily come from genera ists about the time and cost of the drilling options. For each of these circumstances, experts need to be carefully

L t

selected and trained. The assessments should indicate the implications of the alternatives in terms of a probability distribution function as discussed in Section 3.3.4. An important subtask in the estimation of the impact of information is assessing the conditional probability distributions with the information that can be obtained from the alternative drilling activities and assessing the probability L

distribution of the information from the drilling. In particular, the probability distribution for cumulative radionuclide releases and health effects will strongly J

depend on the information obtained. Dese probability distributions are a major ingredient for carrying out a value of information analysis.

De next task is to quantify the value judgments (Section 3.3.5) necessary to integrate 1

all the objectives of the informational drilling program. Because of the uncertamties

)

about what will be learned by the various drilling options, a multiattribute utility function should be used to integrate these objectives (Keeney and Raiffa,1976).

Expert jud ment will be necessary to specify the value judgments for the utility f

function. T tese ludgments are of a policy nature because they relate to the c uality of information available for key decisions regarding the repository, and they siould be provided by individuals with policy positions in the repository program and stakeholders with a legitimate voice in that program. Examples of this in the repository program are discussed in Section 1.4. To assist the policy makers in quantifying their judgments, it is important to have the assistance of a normative expert with substantial experience in quantifying value judgments.

With the tasks above completed, it remains to analyze the options and identify those that provide the most information for the time and effort. At this stage, it is critical to gain the insights about why the better options are better and about why they are that much better. His inter >retation is the link that provides useful information to the decision making process from the explicit use of expert judgment in the appraisal ofinformational drillmg options.

4.4.2 Selecting Models to Develop With any information f;athering, problem, the key is to specify the objectives to be achieved. In this case, t ie objectives to be achieved by developmg models need to be carefully specified. Furthermore, these objectives need to be related to the and operating a repository. In this fundamental objectives of designing, constructing, development are the same as regard, the fundamental ob ectives for mode; fundamental objectives for informational drilling. What is different in this case is the means objectives by which those fundamental objectives are achieved. To specify the relationship between the means objectives and the fundamental objectives, expert judgments of both specialists and generalists are needed. Essentially, these relationships answer the questions about how model development will contribute to better understanding and better decision making regarding the repository.

After the experts are selected, they need to be trained to distinguish between fundamental and means objectives and :; understand conce pts such.as influence diagrams and objectives hierarchies for relating them. Then the clicitation process needs to be carefully documented. This documentation can be reviewed by a large number of peers for completeness and reasonableness, and the revised results should provide a basis for the additional tasks in selecting appropriate models for development.

4 i

j w

a

+

The next task is to select general types of alternative models that may be worthwhile to develop. Some of these may be analytical models, and others may be simulation models tepresented by codes. Other factors defining the alternatives concern the j

number ot variables in the models and exactly which variables they should be. A

.j l

combination of generalists and specialists should be a ppropriate for defining a large number of alternative models. Identification techniques for expert clicitation discussed in Section 33.1 will be uced extensively in this task.

I The next task is to screen the alternatives to focus on those that seem most u provide information for the repository. In this phase, the scree combination of judgments from saecialists and generalists. The exact screenmg criteria are not too unportant as ticir appropriateness should be verified after the models have jonc through various stages of development. In general, if the models selected for c evelopment are not providing the insights expected, either because of lack of available data or field data indicates that they are inappropriate, then the models can be revised or new models selected for development.

The fourth task is essentially model development as discussed in Section 4.2. Details are found in that section, so only a brief overview is included here. The task is essentially to specify the variables appropriate for each of the'models selected for development and to identify data sources to provide information about those variables. Also, using; any available physical relationships, it is necessary to relate the variables to each ot1er to provide the structure for the model. At this stage, it is essentially the judgments of specialists that are important. Normative experts should l

assist these experts in expressing their judgments about the relationships of the l

l variables. -

There are a number of input variables to a large model and one or more output i

variables of interest. Probability distributions quantify the current state of knowledge about the input variables and are used in the model to derive implications for the output variables. How this is carried out is described in Section 43. It relies heavily on the techniques for quantifying probability judgments discussed in Section 33.4.

The last task is to run the models many times and gain the insights available from l

them. A team of generalists and specialists will likely be most aapropriate to interpret the results of the analyses. Based on these insights, it wi'l probably be appropriate to repeat various runs of the model to gain ad implications. At this stage, the team of experts should also variables, the relationships between variables, and their quantification.

4.43 Laboratory and Field Experiments be done before final design and construction of the reposito of these situations is to specify the objectives to be acnieved by the experiment proposed. As with the problems discussed above, the task is to provide information that results in a legal and environmentally sound repository through better design L

j l

'69-

C' W

Ji and construction. His task requires balancing of the impacts of the experimentation in terms of cost and effort against the.value oT the information learnec. For each of the proposed experiments, different objectives contribute to information obtained.

He kind of information expected needs to be specified using expert judgments of 7

generalists and specialists and the assistance of a normative expert to exp icate that Once these ob ectives are clarified, we have a basis for evaluating judgment.

different alternatives for thelaboratory and field experiments.

For any proposed experiment, the next task is to identify alternatives for conducting vary in the sophistication of testmg equi > ment used.pth or breadth, ney also may that experiment. Dese may vary in cost, time, or de At this stage, the judgment of generalists with some assistance of specialists should be appropriate for characterizing the alternatives.

He next task is to screen the various alternatives to identify the types that seem more appropriate. The screening criteria should be set by the generalists usinJ concepts described in Section 3.3.2, since the information is relevant to the overa 1 repository program. However, at later stages in the analysis, the appropriateness of the screening criterion should be validated. If it turns out the information sought from the experiments is not being provided, the analysis should be repeated to determine which experiments should be conducted and whether they are worth the information. Experiments that at one time were thought not to be appropriate because of the expectation that certain information would become available have i

become appropriate when it is known that that information is not available, in l

simpler terms, if some field experiments are not successful, the relative desirability of others may merease, I

f For the alternatives that have made it ttirough the screening, one should more carefully specify details of the experiment to be conducted. As part of this, there j-should be probabilistic estimates of the amount of information obtained by each of the experiments as well as estimates of their cost, time, and any damage from the experimentation. As in the task of informational drilling, two sets of quantitative i

estimatrs are especially important: the conditional probability distribution over l

radionuclide emission for different experimental outcomes and the probability distribution over those outcomes. The judgment of generalists will likely be necessary for some of the cost and time information, although this judgment might be J

augmented by some specialists, whereas the judgment of specialists will mainly bc used to judge the information expected from each experiment.

De objectives in the first task above need to be integrated into an overall utility function. These value judgments should be in accordance with the techniques discussed in Section 3.3.5 and should use the judgments of generalists on the repository team. However, these value judgments should be carefully related to the policy value judgements made about the fundamental value tradeoffs of the mformation gathering process. In other words, since the obiectives of the I

experiments are means to achieve the objectives of designing and constructing a l

repository, the specific value judgments dealing with tradeoffs among the objectives of experiments must relate to the value tradeoffs that concern the policy objectives, nis relationship should be carefully documented.

I r ^ t b

u

yw.,

d

)

'The final task is to analyze the various laboratory and field experiments using the value-of information techniques and.to select those that seem appropriate. In all cases, one of the alternatives that definitely should be cont.idered is not conducting the experiment. In some sense, one of the more useful pieces of information gathered from such an analysis is whether specific experiments, given their quality, cost, and time, are worth the effort. In some cases, it may be cheaper simply to design the repository assuming; that a certain situation exists, rather than verifying it.

In other situations, although tie information desired might be very,important, if the experiments are unlikely to provide that information, they simply might not be worth the time, effort, and cost.

4J Straemale Renonitorv Decialons Strategic repository designs are those that directly concern the design, construction, and operation of the repository. As pointed out in Section 2.5, many of these -

decisions will affect the performance of a repository and therefore should be considered when developing and screening scenarios, developing model, estimating parameters, and gathering mformation. In a sense, any performance assessment is conditi:nal on these strategic decisions.

For discussion it is useful to think of the analysis of those strategic decisions in terms of six components. De first two components, which identify the strategic problem, are specification of the objectives and identification of the alternatives. The degree to which the objectives are achieved by the various alternatives is quantified la the third component. The fourth component integrates the different objectives u ing value judgments concerning risk attitudes and the relative importance of different objectives. All the information is integrated and analyzed in component five to provide insight for decision making. Component six is documentation of the process and results.

~

The main techniques in these components are described in Section 3.3.3 (structuring objectives), Section 3.3.4 (probability quantification), Section 3.3.5 (value quantification), and decision analysis (Raiffa,1968; Howard,1968; Keeney and Raiffa,1976; and von Winterfeldt and Edwards,1986).

4.5.1 Specifying and Structuring Objectives ne overall objectives for constructing and operating the repository should guide the development of specific objectives for constructing or operating the repository. The techniques for constructing objectives hierarchies are useful for this step (Section

. A group of experts te > resenting all interested parties should be selected to 3.3.3)fy the overall objectives for constructing and operating the repository. At this speci l

stage, it is important to have a broad diversity of opinions providmg objectives for l

the repository, as these objectives should provide the foundation for future strategic l

decisions (Keeney,1988a,b). De training for these experts need not be extensive, but it should clearly indicate how the stated objectives will be used and methods that l

may facilitate broad thinking about their objectives, ne clicitation process itself needs to be done by normative experts trained to elicit objectives in an operational manner for further analysis. The objectives should then be structured by the normative analysts, with the assistance of project members, and then carefully reviewed by peers and others interested in the repository program. Modifications are 71-

.. 1 welcomed, as the intent is develop an appropriate fundamental set of objectives for the repository. Finally, these objectives should be documented.

With a given specific strategic decision, the repository objectives need to be related to specific objectives influenced by the strategic decision. That linking can likely be

.done by generalists with the assistance of normative experts. In essence, it is a deductive process that relates the overall objectives to a given decision problem. As always, the resulting objectives should be carefully documented after review by peers contributed to the overall objectives. y program, meluding all memlM who in and others interested m the repositor 4.5.2 Identification of Altarnatives For a cific strategic dec:sion, the alternatives need to be identified. Thus, the identi ca on techniques Section 3.3.1 are relevant. The experts involved in specifying alternatives should have substantial knowledge about details of the specific decision to be addressed. Normative experts should assist them in defining generic alternatives (e.g., sets of alternatives that differ in terms of parameters). After a wide range of alternatives has been identified, it may be worthwhile to screen the alternatives using the screening techniques in Section 3.3.2. Appropriate screening i

criteria should be set by generalists to facilitate focusing on alternatives that are presumed to be better. After the analysis, the reasona)leness of the screening; criteria should be reexamined considering the quality of the screened alternatives. If

- it is likely that alternatives screened out would in fact be better than some of those L

retained, the analysis should be revised and repeated.

4.5.3 Impacts of Alt.ernatives Once the objectives and alternatives in a' specific strategic decision problem are articulated, they effectively define a matrix in which objectives relate to the individual columns of the matrix and alternatives to the individual rows. To specify the impacts of the alternatives, one wants to fill in each cell in the matrix, indicating i

the degree to which the alternative impacts the corresponding objective. This process utilizes scientific and engineerin g knowledge and necessarily relies on the techniques and procedures models, data, and expert judgments. For tiis step,ing, model development, and 1.

outlined for scenario development and screen I

parameter estimation are repeatedly used. Since these are detailed in Sections 4.1 through 4.3, there is no need to elaborate on them here. It is simply worth noting l

that expertise from a variety of fields that includes the behavioral sciences, economics, and medical sciences will likely be required. Most impacts will be I

l uncertain. In those cases, the techniques for probability quantification (Section 3.3.4) 1 1

will be useful.

l

\\

4.5.4 ValueJudgments At this stage, it is critical to aggregate the various com nent impacts for each of the alternatives. Because of the uncertainties regardin those impacts, some of these value jud pnents must address risk attitudes concerne with those uncertainties, and value tradeoffs among objectives addressing environmental,gments concern because t 1ere are multiple objectives, some of these value jud social, economic, and health and safety impacts. The value judgments should be made as follows. -_ _. ~ _ _.. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. _ _,

a

i Lt u,_

n u

D 'n

,6 i

I i

First, the original group who s >ecified the overall objectives to the repository should 4

specify quantitative value juc gments regarding risk attitudes and value tradeoffs

-]

among those objectives using the value quantification techniques described in Section 3.3.5. Each of the individuals in that group should provide individual value' I

judgments, and each of these sets of values should be carefully appraised for l

consistency. Also, individuals should be allowed to hear the logic of other people's

>oints of view regarding the values and reiterate their judgments. However, it would unreasonable to force a consensus (Section 3.4)y the same values, so it would

>e unlikely that everybody would have precisel

. Each individual value should be carefully documented, and collectively they should provide a range for the values used in the problem.

4.5J Analysis of the Alternatives

%e analysis of alternatives should integrate all the information from the preceding four components for the given strategic decision using decision analysis.

Operationally, it may be reasonable to take an " average" set of the value j,udgments as a base case and do sensitivity analysis from this to incorporate all the different i

newpoints. The intent is to identify alternatives that clearly are not competitors and l

identify circumstances under which each of the remaining alternatives are the best and how much better they are than the alternatives. Because of the uncertainty about quantitative parameters relating to the impacts, sensitivity analysis of some of these may also be appropriate. The experts working on this part of the problem should be analysts. It is unlikely that their use of expert judgments needs to be made explicit, but they certainly use expert judgment in deciding what sensitivity analyses to pursue. The degree of sensitivity analysis should be guided by the insights provided and the need for careful documentation.

l 4.5.6 Documentation of Analysis

~

l The documentation of the analysis and its insights (or decidon making is essentially a

~

col!cetion of the documentation of each of the com >onents o! the ana ysis. However, it is worth recognizing that documenting the overal decision process does have some requirements different from documenting the components. This comes about because the overall process is of interest to different ty, pes of individuals, some of i

l whom may not be concerned about details. Documentation of technical information relevant to impacts is likely of concern mainly to peers and individuals with a technical know edge about those aspects of the repository. Documentation of the decisions made may be of concern to a large number of lay people as well as to numerous individuals concerned with or entangled by the )olitics of the repository problem. Documentation of the overall decision need not focus on detailed aspects of the problem that turn out not to be crucial. The documentation should very carefully, explain what the alternatives are, what the objectives are for evaluating the alternatives, and the logic of why a given alternative was chosen. References can naturally be made to more detailed documentation elsewhere.

Documentation of any strategic decision should be considered itself a decision problem. One should carefully think of the objectives of the documentation and who the documentation is meant to inform because the communication alternatives have pros and cons. These need to be balanced appropriately in documenting the overall -

0 90 7,..

buun 0 decision. The analysis of the documentation decision need not be made explicitly, l

but consideration of the appropriate componer.ts will likely result in better.

t documentation,

.i

)

i 4

9 e

a 0

D 54 b

i l

j.

i l

1 74

,% w j; q

g REFERENCES Armstrong, J. S., W. B. Denniston, and M. M. Gordon, "The Use of the Decomposition Principle in Making Judgments," Ornanizational Behavior and Human Performance 14, 257-63 (1975).

Arrow, K., bial choice and Individual Value<- (New York: John Wiley & Sons, 1951).

Ashton, A. A., and R. H. Ashton, " Aggregating Subjective Forecasts: Some Empirical Results," Management Science. 31,14991508 (1985).

Barclay,je., et al., Hanhook for Decision Analvsis, Defense Ad S

Pro cts Agency (1977).

Bonano, E. J., and R. M. Cranwell, " Treatment of Uncertainties in the Performance Assessment of Geologic High Level Radioactive Waste Repositories," J. Mathematical Geoloov. 20,543 565 (1988).

Bonano, E. J., P. A. Davis, L R. Shipers, K. F. Brinster, W. E. Beyeler, C. D.

Updegraff, E. R. Shepherd, L M. Tilton, and K. K. Wahl, Demonstration of a Performance Ae<**cment Methodolosv for Hinh level Radioactive Waste Dissnsal in Bacalt Formations, SandiaNationallaboratories, Albuquerque, NIM, SAb D 86 2325, NU REG /CR-4759 (1989).

Brown, R. V., A. S. Kahr, and E. Peterson, Decision Analysis for the Mananer (New York: Holt, Rinehart, and Winst,on,1974).

~

~

l Bunn, D.W., Abolled Decision Analysis (New York: McGraw Hill,1984).

Clemen, R. T., and R. L Winkler, " Limits for the Precision and Value of Information from Dependent Sources," Operations Research. 33,427-442 (1985).

Clemen, R. T.,' " Combining Overlapping Information," Manneement Science 13, 373 380 (1987).

Cranwell, R. M., R. V. Guzowski, J. E. Campbell, and N. R. Ortiz, Risk Methodolony for Geolonic Disposal of Radioactive Waste: Scenario Selection Procedure, Sandia National Laboratories, Albuc ucrque, NM,

~

SAND 801429, NUREG/CR 1667, draft report, copy available in NRC Public Document Room.

Cranwell, R. M., J. E. Campbell, J. C. Helton, R. L Iman, D. E. Longsine, N. R.

Ortiz, G. E. Runkle, and M. J. Shortencarrier, Risk Methodoloav for

~

Geolonic Disposal of Radioactive Waste: Final Reoort. Sandia. National Laboratories,' Albuquerque, NM, SAND 81-2573, NUREG/CR-2452 (1987).

Dalkey, N., and O. Helmer, "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science. 9,458-467 (1963).

I 1,

y

.j Nf[

An Fmerimental Study of Grou 3 Ooinion.

Dalkey, N. C., The Delohl Method:

'Ihe Rand Corporation, Santa Monica, CA, Report No RM 58880R (1969).

I' Institute Henri Poincare. 7,168 (gique, Sources Subjectives," A de Finetti, B., "La Prevision: Ses Lo 1937).

,l DeGroot, M. H., Ootimal Statistical Decisions (New York: McGraw Hill,1970).

Delbecq, A., A. Van de Ven, and D. Gustafson, Groun Techniaues for Proeram Elanning (Glenview,IL: Scott Foresman,1975).

~

~

A MuliinttriNr e Utilatv Analvsir of Sites Nomir sted for Department of Energy,for the Erst Characte rization b lioactie Wasie Tenository - A heision

~

)

Aidine W ethodolonv. Office ot civilian Radioactive Waste Management, Washington, D.C., DOE /RQ 0074 (1986).

Department of Energy, Draft Environmental Aesatement. Reference Renositorv

~

wation. Hanford Site. Office of Civilian Radioactive Waste Management,

1. ashington, D.C., DOE /RW 0017 (1984).

Dyer, J. S., and R. A. Sarin, " Measurable Multiattribute Value Functions,"

Onerations Research. 22, 810 822 (1979).

l Edwards, W., "How to Use Multiattribute Utility Measurement for Social IEEE Transactions of Systems. Man and Cybernetics Decision Making,"977).

SMc-7. 326 340 (1 i

Edwards, W., and J. R. Newman, Multiattribute Evaluation, (Beverly Hills: Sage, 1982).

Edwards, W., T. Eppel, A. Hybel, D. Mefford, and D. von Winterfeldt, Deiign Princiales for an Analyst's Work Station. Social Science Research Institute, Unive'rsity of Southern California, Los Angeles, Research Report No. 871 (1987).

Edwards, W., and D. von Winterfeldt, "Public Values in Risk Debates," Risk Analvsis. 7,141-154 (1987).

Einhorn, H.,' and R. Hogarth, " Unit Weighting Schemes for Decision Making,"

Oronnizational Behavior and Human erformance. 13, 171-192 (1975).

Einflorn, H. J., " Expert Measurement and Mechanical Combination,"

Oiar.ization Behavior and Human Performance. 7,86106 (1972).

Ekman, G., and L Sjoberg, " Scaling," Annual Review of Psychotoev. 16, 451 474 (1965).

Electric Power Research Institute, Seismic Havard Methodolony for the Central and Eastern United States. Vol,1. Methodolony. NP-4/26 (1986).

l l

.76-t L

E

, g. : (. t Isb )

EPA,40 CFR Part 191, " Environmental Standard for the Mana[ement and Disposal of Spent Nuclear Fdel, High Level Waste, and 1 ransuranic Radioactive Waste; Final Rule, U.S. Environmental Protection Agency,"

1 Federal Register. Vol. 50, Washington, D.C. (1985)

Fischer, G. W., "When Oracles Fail: A Comparison of Four Procedures for Aggregating Subjective Probability Forecasts," Ornanizational Behavior and Human Performance. 28,96110 (1981).

~

Fischhoff, B., "Deblasing," in Judoment Under Uncertaintv: Heuristics and Blatas, D. Kahneman, P. Sl6vic, and A. Tversky (eds.) (Cambridge:

Cambridge University Press,1982),422 444.

Fischhoff, B., S. I.ichtenstein, P. Slovic, S. L Derby, and R. L Keeney, Acceptable Risk (Cambridge: Cambridge University Press,1981).

Fischhoff, B., P. Slovic, and S. Lichtenstein, " Fault Trees: Sensitivity of u

Estimated Failure Probabilities to Problem Representation," Journal of l

Fw wrimental Psvcholonv: Human Percention and Performance. 4, 330 344 (1178).

~

~~

~

l Inc., Salt Renository Emniacement Mode Evaluation and Fluor Technolka,l Report. Advanced Techn' ology Division, Irvine, CA (March Selection.

1988).-

Genest, C., and J. V. Zidek, " Combining Probability Distributions: A Critique and an Annotated Bibliography," Statistical Science.1,114148 (1986).

Gettys, C. F., S D. Fisher, and T. Mehle, Hvoothesis Generation and Plausibility

~

Assessments, University of Oklahoma, Decision Process Laboratory, Norman, Oklahoma, Technical Report No.15 10 78 (1978).

Golder Associates, Renort to Rockwell Hanford Onerations. Probability Encodino of Vi riab'les Affectina Inflow into the Exnloratory Shaft Test

)

l Facility at the 3asalt Waste Isolation Project. Ridneland. Washinnton.

l Redmond, Washington (June 1986).

~

j i

I i

Goodman, B., " Action Selection and Likelihood Estimation by Individuals and u

(1972).

~

7, 121-141 Groups," Ornanizational Behavior and Human Performance 1

Gregory, R., and S. Lichtenstein, "A Review of the High level Nuclear Waste Repository Siting Analysis," Risk Analysis 7, No. 2,219-223 (1987).

Hogarth, R., Judrtment and Choice: The Psycholoav of Decisions. (Chichester, England: Wf.cy,1980).

C. A., Decision Makine Under Uncertaintv (Englewood Cliffs, NJ:

Holloway, ice Hall,1979).

~

Prent -

f l-Howard, R. A., "The Foundations of Decision Analysis." IEFF Transactions on L

Systems Science and Ovbernett ESC:1,211219 (1968).

Ibrekk, H., and M. G. Morgan, "Gra shical Communication of Uncertain

- Quantities of Nontechnical People," Ria Analvsis 7,519 529 (1987).

r Iman, R. L, and J. C. Helton, A Comnarison of Uncertainty and Sensitivity Analysis Techniaues for Comnuter Models. Sandia National Laboratories,

~

Albuquerque, NM, SAND 84 1461, NUREG/CR-3904 (1985).

~

Jungermann, H., and M. Thuering, "The Use of Casual Knowled ge of Inferential Reasoning," in Krnert Judament and Ernert Systems J. L N umpower, L D.

i Phillips, O. Renn,~and V. R. R. Uppuluri (eds.) (New York: Springer,1987) 131 146.

Kahnemann, D., and A. Tversky,ience

" Intuitive Prediction: Biases and Corrective Procedures," Management Sc 12,313 327 (1979).

Kahnemann, D., P. Slovic, and A. Tversky,idge Oniversity Press,1982).

~

Judoment Under Uncertaintv:

Heuristics and Biases. (New York: Cambr Keeney, R. L, Siting Enerav Facilities (New York: Academic Press,1980).

Keeney, R. L, "An Analysis of the Portfolio of Sites to Characterize for Selecting a Nuclear Repository," Risk Analysis 7, No. 2,195 218 (1987).

Keeney, R. L, "Value Focussed Thinking and the Study of Values," in Decision Analysis Descriotive. Normative and Prescriotive Asnects. D. E. Bell, H.

~

Ralffa, and A. Tversk Press,1988a),654-690.y (eds.)-(Cambridge, England: Cambridge University Keeney, R. L., " Structuring Objectives for Problems of Public Interest,"

Operations Research. 36, No. 3,396-405 (1988b).

Keeney, Value Tradeoffs (New York:

~

~

R. L, and H. Raiffa, Decisions with Multiole Obiectives: Preferences and Wiley,1976).

Keeney, lysis. 6,417-424 (1986).R. L, and D. von Winterfeldt, " Improving Risk C Ana Lichtenstein, S., and B. Fischhoff, " Training for Calibration," Ornanizational Behavior and Human Performance. 26,149-171 (1980).

~

Lichtenstein, S., B. Fischhoff, and L D. Phillips, " Calibration of Probabilities:

The State of the Art," in Decision Makinn and Chan ze in Human Affairs. H.

Jungermann and G. de Zeeuw (eds ), (D'ordrecht, Eolland: Reidell,1977),

275-324. u -.

c

[;o s ;

l,h il q

Lichtenstein, S., B. Fischhoff, and L. D. Phillips, " Calibration of Probabilities:

The State of the Art to 1980," in Judoment Under Uncertaintv:' Heuristics -

and Biases, D. Kahneman, P. Slovii, and A. Tversky (eds.) (New York:

~

Cambridge University Press,1982),305 334.

, 'nie Delohl Method: Technlaues and Linstone, H. A., and M. Turoff (eds.)ison Wesefey,1975).

L Applications:' (Reading, MA: Add Matheson, T. E., and R. L Winkler, " Scoping Values for Continuous Probability Distribution," Mananement Science. 22,10874090 (1976).

McCormick, N. J., Reliability and Rick Analvsis (New York: Academic Press,

'1981).

~

MacGregor, S. Lichtenstein, and P. Slovic, " Structuring" Knowledge Retrieval: An Analysis of Decomposing Quantitative Judgments, Ornanizational Behavior and Human Decision Processes. 42,303 323 (1988).

~

Merkhofer, M. W., "Quantifyi,nr Judgmental Uncertainty: Methodology, Experience, and Insights, EEE Transactions on Systems. Man and cvbe rneties. SMC-17. 741 752 (1987).

"A Multiattribute Utility Analysis of Merkhofer, M. W., and R. L. Keeney,f Nuclear Waste," Risk Analysis 5,173-l Alternative Sites for the Disposal o l

194 (1987).

l Quantifying l

Merkhofer, M.'W., and A. K. Runchal, " Probability Encoding: Basalt," hs Judgmental Uncertainty over Hydrologic Parameters for Conference on Geostatistical. Sensitivity. and Uncertainty Methods for l

i Ground Water Flow and Radionuclide Transoor: Modelind. B. E. Buxton (ed.) (Columbus, OH: Battelle Press,1989) 629-648.

~

l

~

l Meyer, M. A., and J. M. Booker, Sources of Correlation Between Exnerts:

l Imoirical Results from Two Extremes. Los Alamos National 1.aboratory, I

os Alamos, NM, NUREG/CR-4814 (1988).

~

4 Mosleh, A., V. M. Bier, and G. Apostolakis, "A Critique of Current Practice for i

the Use of Expert Opinions m Probabilistic Risk Assessment," Reliability Ennineerina and Svstem Safety. 20,63-85 (1988).

t NRC,10 CFR Part 60, "Dispsal of High Level Radioactive Wastes in Geologic 48,positories, U. S. Nue:

car Regulatory Commission," Federal Renister. Vol.

Re Washington, D.C. (1983).

l NRC, Guidance for Determination of Anticinated Processes and Events and Unanticinated Processes and Events, Diaft Generic Technical Position, l

Division of High-1.evel Waste Management, Office of Nuclear Material

~

and Safeguards, U.S. Nuclear Regulatory Commission, Washington, Safety (1988).

D.C.

b m

1 v,2a NRC, Es.Rgior Safety Studv; An Assessment of Accident Risks in U.S.

Commercini Nucl~ ear Po' er Pla'nts. Washington, DC, Nuclear Regulatory w

. Commission, NUREG 75414 (1975).

L Okrent, D., " Comment on Societal Risk," Science 208,372 375 (1980).

PSAC, IEvel S. Soecification for a Sensitivity Analysis Exercise Based on 12 vel E Test C=*. P. C. Robinson, A. Saltelli, ~A. Alonson, and E. J. Bonano (eds.),

Probabilistic Safety Assessment Code User's Group, Nuclear Energy Agency.

Organization for Economic Cooperation and Developmer6t, Paris, France (1989)'

Pearl, J. A., A Goal Directed Annroach to Structurine Decision Problems, University of California, Los Angeles, Report No. UCLA ENG 7811 (1978).

~

Peterson, C. R., and L R. Beach, " Man as an Intuitive Statistician," Psychological Bulletin 68,29-46 (1%7).

Pitz, G. F., " Subjective Probability Distributions for Imp (ed.) (New York:erfectly K Quantities," in Knowledne and Cognition. L W. Gregg John Wiley & Sons,1974).

4 Pitz, G. F., and N. Sachs, " Judgment and Decision: Theory and Application,"

Annual Review of Psveholoav,35,139163 (1984).

Pitz, G. F., N. Sachs, and J. Heerbroth, " Procedures for Eliciting Choices in the Analysis of Individual Decisions," Ornanizational Behavior and Human

~

Performanca. 26,3%-408 (1980).

~

Poulton, E. C., "Models of Biases in Judging Sensory Magnitude," Psycholonical l

, Bulletin. 86,777 803 (1979).

L Raiffs, H., Decision Analvsis (Reading, MA: Addison Weseley,1%8).

L Raiffs, H., and R. L Schlalfer, Anolied Statistical Decision Theory. (Cambridge, L

MA, Harvard University Pres's,' 1%1).

~

1 Ramsey, F. P., " Truth and Probability," in The Foundations of Mathematics and l

L Other I.ngical Essays. R. B. Brainwaithe (ed.), (New York: Harcourt Brace, L

1931).

Ravinder, H. V., D. N. Kleinmuntz, and J. S. Dyer, "The Reliability of Subjective No. 2,186-99 (February 1988)gh Decomposition," Manneement Science. 3 Probabilities Obtained Throu j

Savage, L J., The Foundations of Statistigg (New York: Wiley,1954).

Shachter, R. D., "ProbaHlistic Inference and Influence Diagrams," Operations Research. 36,589-604 (1987). l L

~ -.

.. -. ~

yo, u ;,..

id

.1

[

Seaver, D. A., #=***i's Piubability with Multiole Individuals: Groun Interaction

)

Versus Mechanial Anerenation, Social Science Research Institute, University of Southern California, Los Angeles, CA, Research Report No.

78 3 (1978).

a Selvidge, J., Fanerimental Comnarlson of Different Meth 3ds for Assessinn J

F2tremes of Probasility Distributions by the Fractile Mgthod, Technica'.

~

l Report, Graduate School of Business,~ University of Colorado, Boulder (1975).

Shanteau, J., " Psychological Characteristics of Expert Decision Makers," m Runert JuAement and svaert Svstems. J. Mumpower et al. (eds.), (New York:

Springer,1987) 289 304. ~

Slovic, P., B. Fischhoff, and S. Lichtenstein, " Behavioral Decision Theory,"

Annu=I Review of Psvchaloov 28,139 (1977).

Spetzler, C. S., and C. A. Stael von Holstein, " Probability Encoding in Decision Analysis," Management Science 22,340-352 (1975).

f L

Stael von Holstein,' C. A., S., and J. E. Matheson, A Manual for Encodinn

~

Probability Distributions. SRI International, Menlo Park, CA, SRI Project 770778 (1979).-

Siillwell,' W. S., D. A. Seaver, and J. P. Schwartz, Fanert Estimates 3f Human irror Probabilities in Nuclear Power Plant Ooerations: AIteview of 3robability Assessment and Scalinn, Nuclear Regulatory Commission:

l' Washington, D.C., NUREG CR 2255 (1981).

l Thomyson, B. U. J., C. J. Frizelle, D. B. Nichols, and M. C. Thorne, A Method of Overcominn Limitations of Conventional Scenario. Rated Risk Assessment by Usinn Monte-Carlo Simulations of Possible Future Environmental Changes transmitted to Scenario Working Grou Nuclear Energy A ency,

~

ization for Economic Cooperation and velopment, Paris, rance L

r r

von Winterfeldt, D., and W. Edwards, Decision Analysis and Behavioral Research (New York: Cambridge University Press,1986).~

Wallsten, T. S., and D. V. Budescu, " Encoding Subjective Probabilities: A logical and Psychometric Review," Management Science. 29, 151-73 s

L Weber, M. O., F. Eisenfuehr, and D. von Winterfeldt, "The Effects of Splitting I '

Attributes on Weights in Multiattribute Utility Measurement," Management Science. 34,43145 (1988).

rack Systems for Westinghouse Electric Corporation, Phase I Stud of Metallic Soent Fuel Manneement from Reactor to Renository. Westinghouse Waste Technology Services Division, Madison, PA, Report WTSD TME 085 (1986).

l 81-u

[ =:

//

,4s, DRAF Wheeler, T., S. C. Hora, and W. Cra'nond, Analysis of Core Damane Freauency:

Ernert Elicitation on Internal Events, Vol. 2 U.S. Nucleir Regstatory Co~mmission, NJREG/CR4550 (1989).

Wheeler, T., S. C. Hora, and W. Cramond, Front End Exoert Oninion Elicitation:

a DInft.Rapon Sandia National Laboratories (1988). '

~

Wilson, R,'" Commentary: Risks and Their Acceptability," Science Technoloav -

l and Human Values 9,122 (1984).

Winkler, R. L, Introduction to Baveslan Inference and Decision (New York:

Holt, Rinehart, and Winston, Inc.,1972).

4 Zinnes, P., " Scaling," Annual Review of Psycholonv 20,447-488 (1%9).

i w

9 9

h 1

i l

l l.

-B2-1

.