ML19327B294
| ML19327B294 | |
| Person / Time | |
|---|---|
| Issue date: | 10/30/1989 |
| From: | Ballard R NRC OFFICE OF NUCLEAR MATERIAL SAFETY & SAFEGUARDS (NMSS) |
| To: | Cunningham M NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES) |
| References | |
| CON-FIN-A-1165, RTR-NUREG-CR-5411 NUDOCS 8910300113 | |
| Download: ML19327B294 (91) | |
Text
x i
J sL i
Ll3 00T 3 019B3 -
MEM0/ BROOKS l
1-o
.c?
MEMORANDUM TOR:
Mark A. Cunningham, Branch Chief j
Probabilistic Risk Analysis Branch i
Division of Systems Researrc., RES FROM:
Ronald L. Ballard, Chief Geosciences & Systems Performance Branch Division of High. Level Waste Management, NMSS l
SUBJECT:
REQUEST FOR REVIEW OF SANDIA REPORT ON EXPERT JUDGMENT I have enclosed a copy of the draft report submitted by Sandic National Laboratories entitled ' Elicitation and Use of Expert Judgment in Performance Assessment for liigh. Level Waste Repositories.' prepared under contract f!N A.1165. Although Lee Abramson has already provided review comments, P.K. Niyogi has also been following the devetloprnent of this report. Any ad(itionai review comments are needed by November 7, 1989. We appreciate your cooperation in this matter.
Ronald L. Ballard, Chief Geosciences & Systems Perfo:mance Branch D Oision of High. Level Waste Management, HMSS
Enclosure:
As stated
.cc: L. Abramson, RES P.K. Niyooi. RES 8910300113 891030 N16$MRESEXJ$t$
DISTRIBUTION:
r Central files REBrowning, DhlkM BJYoungblood, DHLUM RLBallard, HLGP J0 Bunting, HLEN JLinehan, HLPM HLGP r/f HMSS r/f PDR & LPDR CNWRA 5
ACNW PBrooks, HIGP SCoplan, HLGP hlY'.
(j/
0 l
0FC : HLGP HL HL t[
J
- ............:..........:......hy.:..-
j l
NAME :PBrooks ej I:
- S an
- RLB llh l
.....:............:............:............:............:....3......:...........:
DATE :10/J N89 10/Q/u9 :
10/17/89
- [I. Ll0
/ b' k h') ""{,,
h.-
j
y
., e,:e D:,f.,i,*'72 C'
s NUREG/CR 5411 SAND 891821 EUCITAT10N AND USE OF EXPERT JUDGMENT IN PERFORMANCE ASSESSMENT FOR HIGH LEVEL RADIOACf1VE WASTE REPOSITORIES Evaristo J. Bonano Ste hen C. Horst h L Keeney2 Detlof von Winterfeldt2
)
I q
July 1989 l
F Sandia National Laboratories Albug rque,NM 87185 I
Operated Sandia Corporation j
for the U.S. Department of Energy Pre ared for Division of High-vel Waste Management Office of Nuclear Material Safety and Safeguards U.S. Nuclear Regulatory Commission Washington,DC 20555 NRC FIN A1165 l
i s
t 1 University of Hawaii, Hilo, HI 2 University of Southern California, Los Angeles, CA t
l'
.tl
~
ABSTRACT o
This report presents the toq cpi o'f formalizing the clicitation and use of expert jud ment in the performance asseumsnt of high level radioactive waste repositories J
m c ecp geol ic formations The report cuttines aspects of priormance assessment in which the licitation and use of expert gment should se formalized, discusses existing techniques for formalizing the el stion and use of expert judgment, and presents guidelines for applying these techniques in performance assessment.
4 e
l l
iV-l 1
4
\\
?!
i CONTENTS 1
]
1.
INTRODUCTION 2
1.1 Objective of this Report i
1.2 Expert Judgment in Performance Assessment of HLW Repositories 1.3 Characteristics of a Formalized Expert Judgment Process 2
4 1.4 Previous Formal Uses of Expert Judgments in MLW Program 7
i 13 When to Use Expert Judgment 1.6 Relationshi of Formal Use of Expert Judgment to Informal 7
Use, Model ng, and Data Collection 2.
AREAS IN NEED OF FORMAL EXPERT JUDGMENT IN 9
HLW DISPOSAL 9
2.1 Scenario Development and Screening 2.1.1 Identification of Events and Processes 10 2.1.2 Classification of Events and Processes 10 2.1.3 Screening of Events and Processes 10 2.1.4 Formulation of Scenarios 11 2.1.5 Screenin of Scenarios 11 2.1.6 Probabil of Occurrence 11 12
-l 2.2 Model Development 2.2.1 Data Selection and Interpretation 12 2.2.2 Development of Conceptual Models 12 2.2.3 Confidence Building 13 t
14 2.3 Parameter Estimation 2.3.1 Identification of Important Parameters 14 2.3.2 Quantificatiori of Uncertainty in Parameters 15 16 2.4 Information Gathering 16
,l 2.5 Strategic Repository Decisions 3.
EUCITATION, USE, AND COMMUNICATION OF 18 l
EXPERT JUDGMEN7S 18 3.1 Definitions i
3.2 The Process of Eliciting Expert Judgmeats 20 i
k
'lii-
.. ~.. -. -
~
~
^
t
~.t j
i CONTENT 5 (Continue:1)
U
.c 3.2.1 Identification ofissues r ' Information Needs 20 l
3.2.2 Selection of Experts 21 1
3.2.2.1 Selection of Genera'ists 22 I
3.2.2.2 Selection of Specialists 22 3.2.2.3 Selection of N ormative Experts 24 3.2.3 Training 24 28
{
3.2.4 Conducting Elicitation Sessions 3.2.4.1 Basic Elicitation Arrangements 29 3.2.4.2 Structure of a Standard Elicitation Session 30 3.2.4.3 Post Elicitation Activities 31 3.3 Techniques for Expert Judgment Elicitation 31 1
3.3.1 Identification Techniques 32 3.3.1.1 Techniques for Event and Scenario Identification 32 33.1.2 Identification of Conceptual Models 34 33.2 Screening Techniques -
34 33.2.1 Setting Target Levels or Constraints 35 33.2.2 Selection 36 33.3 Decomposition Techniques 36 333.1 Decomposition of Factual Problems 36 l
333.2 Decomposition of Value Problems 38 i
333.3 Varia.its of Decomposition 39 Bere its and Costs of Decompositions 40 r
3.'. 3.4 33.4 Techniques for Quantifying Probability Judgments 40 33.4.1 Magnitude Judgments about Discrete Events 42 i
3.3.4.2 Magnitude Judgments about Continuous 33.43 Fractile Technique 43 Uncertain Quantities 42 33.4.4 Interval Technique 44 i
3.3.4.5 Indifference Jodgments Between Gambles with Discrete Events 44 t
i
.jy.
e
'v.. ;
i i
CONTEhrI3 (Continued)
Pags 33.4.6 Indifference Judgments among Gambles with Cominuous Uncertain Quantities 45 l
3.33 Techniques for Quantifyir.g Vclue Judgments 45 p
Techni ue 46 Sim le Multiattribute Ratinbeasurahle 33.5.1 Indifference Technique for 3.3.5.2 Value Functions 47 3333 Aggregation Steps 48 48 3.4 Combining Expert Judgments l
3.4.1 Combinia Usts 48 3.4.2 Combinin Probabili Judgments 49 i
3.43 Combin Value J gments 49
?s4.4 Behavio vs. Analytical Combination 49 j
\\
3.5 Communicating Expert Judgments 51 3.5.1 Documentation 51 3.5.2
. Presentation of Eesults 53 3.6 Interpretation,Use, and Misuse'of Expert Judgments 54 4.
SUGGE!nONS FOR THE USE OF EXPERT JUDGMENT IN HLW l'<ISPOSAL 56 j
4.1 Scenario 1 evelopment and Seruning 56 J
4.1.1 Identification and Classification of Events
)
and Processes 57 4.1.2 Screenin,g of Events and Processes 57 4.13 Generation of Scenarios 58 58 l
4.1.4 Screening of Scenarios 4.1.5 Probability cf Scenarios 59 j
l 59 i
4.2 Model Development 4.2.1 Data Selection and Interpretation 59 4.2.2 Development of Conceptual Models 60 4.23 Confidence Building 61 l
43 Parameter Estimation 63
~
.y.
o.
CONTENTS (Concluded)
=
4.3.1 Identification of Parameters 63 4.3.1.1 Guidelines for Parameter Identification 63 4.3.2 '
Quantification of Parameters 64 4.3.2.1
' Guide!ines for Quantifying Parameters 64 4.4 Information Gathering 66 4.4.1 Informational Drilling 66 4,4.2 Selecting Models to Develop 68 4.4.3 Laboratory and ; ield Expenmer.ts 69 4.5 Strategic Repository Decisions 71 4.5.1 Specifying and Structuring Objectives 71 4.5.2 Identification of Alternatives 72 4.5.3 Imtsacts of Alternatives 72 4.5.4 Value Judgments 72 4.5.5 Analysis of the Alternatives 73 4.5.6 Documentation of Analysis 73 REFERENCES
5 vi-
~
~ -.
- o.'.
I i
t-i LIST OF TABLES i
i j
Iatst E481 I
3.1 Taxonoiay of Probabil;;f Elicitation Techniques 41 l
ti
)
6 d
f k.
I I
-p 3
I l
I.
t l
l C
l a
l l
l t
.yji.
,-e a
-, -, -,.., - -,., - -, ~
n.,--
si i
FOREWORD c
This report presents the concept of formalizing the e!icitation and use of expert
,l judgment in the performance assessment of liigh level radioactive waste repositories m deep geologic formations. The report outlines aspects of p:rformance assessment l
in which the clicitation and use of expert judgment should oc formalized, discusses existing techniques for formalizing the ehcitation and use of expert judgment, and presents guidelines for applying ti.ese tcchniques in performance assessment.
l f
i i
i
~i l
?
t l'
j
+
i i
I; l
l l
i P
viii-
- 1. INTRODUCTION De use of expert judgment permeates all scientific inquiry and decision making, ne choice is not whether to use expert judgment, but whether to use it in an explicit and disciplined manner or in an ad hoc manner. For significant technical, environmental, and socioeconomic problems, it is often useful to formalize the clicitation and use of expert jud ment. One such problem is the long term disposal of high level radioactive waste HLW) in repositories mined into deep geologic formations. In siting and desi ning a safe, environmentally sound, and legally acceptable repository, many of the ana yses must use expert judgment.
De Environmental Protection Agency (EPA) has mandated quantitative analyses in its Standard.4 CFR Part 191 for the disposal of spent nuclear fuel and high level and transuranic radioactive wastes. In particular, the EPA requires a so called l
" performance assessment" in the containment requirement of this standard. (ne other requirements are individual and groundwater protection re concern only the undisturbed behavior of the repository system.)quirements 1
Performance I
assessment refers to " quantitative analyses that (1) sdentify the processes and events l
l that might affect the disposal system; (2) examine the effects of these processes and l
events on the performance of the disposal system; and (3) estimate the cumulative l
l releases of radionuclides, consWring the associated uncertainties, caused by all i
. : performance assessment estimates be represented by an overall probability l
distribution of cumulative releases. Furthermore, these probability distributions are to be used to determine whether the release standards in 40 CFR Pa'rt 191 are met.
The Nuclear Regulatory Commission (f a performance assessment when evalua NRC) has been charged with implementing this standard and examines the quality o l
a license submitted by the Department of Energy (DOE) to construct and operate an HLW repository.
Obviously expert judgment is extensively used in any responsible analysis of potential health im > acts from a repository a id particularly in performance assessments.
l Expert juc gment is required in identifying and screening events and scenarios, in l
developmg and selecting models that characterize the geo ogy and hy6 ology of the l
repository system, in assessing model parameters, in collectmg data, and in makin,g strategic decisions about the repository that could affect its performance. While it is I
desirable to use data and modeling extensively in performance assessment, it is I
nevertheless clear that these data and models can never substitute for the maay I
crucial expert judgments in the assessment.
De quality of a performance assessment rests on its foundation of expert judgments.
Consequently, to demonstrate that an HLW repository meets regulatory j
requirements, au significant expert judgments should be documented and supported with sound logic and the best information. His is particularly important because of l
the need for multiple scientific disciplines to address the long term dis >osal of HLW l
and because of the intense scrutiny that all decisions will licely receive.
Respons'bility and accountability can be enhanced by a formal elicitation and use of I
judgment, which is a well decumented, systematic process whereby, experts make mferences or evaluations about a problem using available information as well as accepted scientific methods. This allows for traceability of tne procedures, l
1 l
techniques, and methods, assumptions, and physical principles telled on in any inferences or evaluations.
i 1.1 Oblactive of this Report This report discusses the formal elicitation and use of expert judgment in
>erformance assessment of HLW disposal systems. More specifically, professional
.cnowledge about the analysis of HLW disposal systems and about the clicitation and use of expert judgment is combined to develop insights on the formalization of expert discusses the role of judgments applicable to HLW repositories. De report (1)ies,(2) identifies areas expert judgment in performance assessment of HLW repositor needing formal arpert judgment in HLW disposal, (3) describes the formal clicitation and communication of expert judgment, and (4) provides suggestions for the use of expert judgment in HLW disposal.
1.2 F=4 Judement in Perfbr=arce Assessment of HLW Renealtarias Experts are used to design and implement activities to understand present site i
conditions and predict the >ehavior of the disposal sy) stem. Expert judgment will be setting priorities for data collection,(2 designmg site data-collection used in (1)3) determining the level of resources for reduction of uncertainties, (4) activities, (
quantifying the uncertainty in numerical values for key parameters, (3) develop (mg scenarios and assigning corresponding probabilities of occurrence, and 6) formulating approacies for validating conceptual and mathematical models as well as verifying computer codes. These important tasks need to be addressed before using models and computer codes to predict behavior of the disposal system. Expert judgment is also used with the models and codes to estimate the system's performance for comparison with the numerical criteria in the regulations. For i
example, expert judgment is reqtilted to' screen insignificant scenarios, select
- i methods for propagating uncertainty through the models and codes, quantify uncertainty in the predictions, and interpret results.
1J Characteristics of a Formalized Exnert Judoment Praesen j
A formal expert judgment process has a predetermined structure for the collection, i
processing, and documentation of experts' knowledge. As discussed in Chapter 3, I
this incluc es professionally designed procedures to select problem areas and experts and to train experts for the clicitation of their judpnents. The actual clicitations of i
judgments should involve the expert and a professionally trained person to assist the expert in expressing judgments. De elicited judgments and their rationales should be carefully documented.
Dere are advantages and drawbacks in using such a process. The advantages include the following:
hrpoved Accumcy of Dperr Judgments. De methods in a formal expert elicitation process improve the accuracy and reliability of the resulting information over less structured methods (Lichtenstein, Fischhoff, and Phillips,1977 and 1982;
. His is so because psychological Lichter.r,tein and Fischhoff,1980; Fischhoff,1982)d and communication is improved
)
biases are openly dealt with, problems are define l
(Merkhofer,1987), issuas are systematically analyzed, and rationales and results are 2
o
M e, e
...'%l a l i
documented. The level of expertise may also be improved over less structured methods since a formal process encourages a broadening of the rante of expertise.
Experts are carefully selected in a formal process rather t 1an in a hapiazard manner
{
for reasons of convenience, j
i Well 7hought 7hrough Design)br Elicitation, he procedures that will be used in a formal expert pudgment process are designed specially for the De design rel es on the knowledge concerning expert opinion, problem being fa previous studies that have used formal expert judgment, and knowledge of'the problem domain to be studied. Careful planning of the proceu can substantially reduce the likelihood of j
critical mistakes that will render information suspect or biased. Mistakes such as including perts with motivational biases, failing to document rationales, inadverten influencin g the experts' responses, failing to check for consistency, and allowing i:.
duals to c ominate group interactions can be avoided.
Consistency of hocedurer. A formal expert judgment process enhances consistency and comparability cf procedures throughout a study and across related swdles because participants follow the same procedures. On the other hand, informal l
processes are often subject to the whims and desires of participants.
I Senstability. A formal process reouires the establishment and dissemination of rules
- acd procedures for clicitation and use of expert judgment. A normal part of a formal expert judgment process is the documentation of procedures and assessments, which l
helps to ensure that various reviewers and users of the findings can understand and i
t evaluate the methods and insights of the study. Since the methodology and its implementation are transparent, there is accountability.
Communication. Estr.blishing s' formal' process' helps to provide for reference documents useful in communication and external review. A formal process also encourages communication and understanding among experts and analysts about the problems studied and the values assessed.
Less Delay. Projects have been delayed because critical judgments were.iot carefully obtained or documented, and a formal ex?crt judgment process had to be designed and conducted before the project movec, forward (DOE,1986). A well executed formal process would have avoided costly delays.
here are also drawbacks to the formal expert judgment process:
Resources. There are costs in designing and implementing a formal process.
Documentation is often more extensive with a formal process, and more resources are thus required.
7!me. De time to establish and impl: ment a formal process may be significantly j
greater than that required for an informal process. Scheduling of participants from external organizations adds a layer to the effort that is not present in an internal, informal process.
Reduced Flcribility. Formalliation of the process may reduce flexibility and make on-going changes to the study more difficult. If it is necessary to redo part of a study, reenecting the expert judgment process may be cumbersome and expensive.
3
)
Vulnerability to Criticism. The transparency of a formal process and the i
documentation of procedures and findings ppen it to inspection and criticism. Expert l
pudgment is an area in which misun,derstanding of the methods and aims still exists,
(
>ut a carefully designed and implemented process may thwart such criticisms, i
While a formal process often requires more resources and time than an informal j
l process initially requires, a faulty process that fails to withstand criticism or must bc redone because of mappropriate design or ;mproper execution may end up failing to i
satisfy the project's objectives and cost more m both time and resources. The i
potential for further costs in an informal study should be considered when evaluating the need for a formal process.
Formalizing the clicitation of expert judgments can clearly be expensive and time consuming. For this reason, the areas m which the process should be used should be i
carefully selected. it is neither practical nor reasonable to formalize the use of expert judgments in all aspects of HLW repository performance assessment.
l 1.4 Previous Formal Uses of Ernert Judgments in HLW Program Several stuuies involved the formal clicitation and use of expert judgment on I
importaru problems facing the HLW program. Recent studies re evant to l
peitormt.nce assessment analysis of HLW repositories are outlined here. In Chapter 2, five areas in need of formal expert judgments in HLW disposal are described:
scenario development and screenmg, model development, parameter estimation, j
information gathering (e.g., data collection and experiments), and strategic repository decisions. Collectively, the analyses outlined here address problems in all five areas.
l The Draft Environmental Assessment for the Hanford site in Washington State (DOE,1984), reports an analysis that screened candidate horizons and identified a preferred hor,izon. A multidisciplinary team developed a set of eight measures to rank the horizons. These measures mvolved repository performance, construction case, and costs. Deterministic and probabilistic descriptions of the candidate were probability distributions )ased on analytical models,probabilistic descriptions
~
horizons were developed using the eight measures. T..e available scientific data, and explicit assessment of expert judgments. Because none of the candidate horizons dominated the others, a utility function was also assessed, using value judgments of the interdisciplinary team to combine the measures. The horizon descriptions werc then evaluated using the utility function to rank the candidate horizons.
At the Hanford site, the formal clicitation and quantification of expert udgment helped in designing an underground test facility (Golder and Associates,1986). To i
estimate groundwater and methane gas flow into the proposed test facility, estimates of site specific peologic, hydrologic, and dissolved gas parameters were obtained.
Specifically, pro > ability distributions were assessed for 41 parameters pertaining to flow path length, timing of encounters with geologic features, and transmissivity and
~
storativity of the geologic surroundings near the test facility. The entire clicitation exercise meluded developing an influence diagram to heb identify parameters to be assesred, identifying a panel of experts to se assessed, and conducting training sess,'.ms on probabi;ity clicitation for the panel of experts before the clicitation i
s%5lons.
i r
4
~.
-v- -. - - - -. - -- -, _,.
w._.-..,
ym,-
.o
-~-,-- - - -,
DRM Formal clicitation of expert j,udgment was extensively used in a multiattribute decision analysis comparing horizontal and vertical emplacement modes for casks of spent nuclear fuel in a salt repository (Fluor Technology, Inc.,1988). First,10 attributes covering health and safety, cost, and environmental concerns were selected. An influence diagram related several variables to these attributes. Expert judgment was elicited to provide probability distributions for both emplacement modes for some of the vanables. Deterministic estimates were obtained for others.
These estimates were input into a simulation model to describe the emplacement modes in terms of the attributes. A utility function was then assessed using the value judgments of a Fluor employee to evaluate alternatives.
The Department of Energy, following a recommendation of the Board on Radioactive Waste Management of the National Academy of Sciences, chose multiattribute utility analysis (MUA) as the methodology to rank five potential host sites for an HLW repository in the United States. The analysis (DOE,1986) 3rovided part of the information to reduce the number of possible host sites to three.
Ln the MUA, two different types of experts were used. One type was senior managers of DOE who provided value ludgments about risk attitudes and value I
i tradeoffs among the objectives of the study. 'Ihe second type were specialists in onc
-or more of the technical areas needed to assess repository performance. These t technical experts were divided into six panels addressing economic costs,
. environmental impacts, social impacts, transportation of waste, repository i
-c. "ruction, and postclosure considerations. 'The technical experts were asked to phases of HLW disposal; formulate scenarios for the postclosure phase;p de 9p measures of repository performance for both the preclosure and l
screen the I
scenarios to climinate those that did not apply to parti:ular sitest quantify the likelihood of each scenario occurring during the first 10,000 years after repository closure; estimate radionuclide discharge to the accessible environment in 10,000 years for each scenario; and finally, decide on the pe:formance of each potential site I
for each of the performance measures (Merkhofer and Keeney,1987).
t l
The Board on Radioactive Waste Management reviewed the methods used in the multiattribute utility analysis of potential repository sites. As part of its review, the Board stated (Appendix R, DOE,1986):
While recognizing that there is no single, generally accepted procedure for integrating techmcal, economic, environmental, socioeconomic, and health and safety issues for ranking sites, the Board believes that the I
multiattribute utility methoc used by DOE is a satisfactory and appropriate decision aiding tool. The multiattribute utility method is a i
useful approach for statin g clearly and systematically the assumptions, judgments, preferences, and tradeoffs that must go into a siting decision.
In addition, the expert judgments and methods in this report were publicly scrutinized by peer review (Gregory and Lichtenstein,1987).
A subsequent analysis was based on the same expert judgments elicited for the multiattribute utility siting study. Because the Nuclear Waste Policy Act of 1982 stated that three sites should be characterized, Keeney (1987) analyzed portfolios of thice sites for simultaneous characterization and strategies for sequential
b5.
i characterization. Based on 1986 characterization costs estimated to be $1 billion per sise, sequential characterization strate gies were identified that could save $1.7 to 52.0 i
billion compared with simultaneous e saracterization of the three sites chosen by the i
DOE. This portfolio analysis and the multiattribute vtility siting analysis provided i
i insights used by Congress a designing the Nuclear Waste Policy Act Amendments i
Act of 1987 that eliminated the simultaneous characterization of three sites and chose Yucca Mountain, Nevada, as the planned repository site.
i udgmental l
Merkhofer and Runchal (1989) summarized a study to quantify !ipecifically, uncertainty in values of hydrologic parameters at a repository site.
o effective experts obtained cumulative density functions (cdfs) for the values of (t) ford site, j
ity,, (2) average effective porosity, and (3) anisotropy ratio at the Han ditterent oups of technical experts were used in the study. One group was five well known h ologists not directly involved with the site investigations at Hanford ess, familiar with waste disposal issues. The second group was three but, nevert hydrologists involved in the characterization of the site. De probability clicitation process utilized structured interviews between a trained intervie ver and each of the experts. The intervle'vs consisted of five phases: motivating, structuring, i
conditioning, differences in judgments between the experts, all the res reduce the l
original assessments were anonvmously exchanged, as suggested by the original l
Delphi method (Dalkey and Helmer,1963). The revised probabilities showed at id bl diversity of opinion.
. most only minor revisions; even though there was a cons era e The experts indicated that any substantial changes would occur only after the i
exchalige of logic and data by the experts.
HLW repository operation requires the transport of waste from nuclear power plants to the repository. A study by Westinghouse Electric Corporation developed a set of objectives for evaluating spent nuclear fuel transport ex slicitly using the judgments of experts (Westinghouse Electric Corporation,1986). "o establirh a comprehensive set of objectives, three panels with individuals in the nuclear industry, state governments, and public interest organizations were guided through sessions to create and structure objectives. Structured objectives of the three panels were combined into one hierarchy for review. These objectives concemed health and safety and economic, environmental, political, social, and equity considerations as well as scheduling and Dexibility. De results were a basis for further analysis and l
communication amonginterestedgrties. De process of eliciting the objectivesj the results is found in eeney (19
).
Dest; studies clearly indicate that cxperts have been and will be used in a variety of ways to address critical issues relevant to the long term disposal of HLW in repositories mined in deep geologic formations. In some cases, the experts provided q uantitative assessments (e.g., quantification of the uncertair.ty about a parameter, or the likelihood of a scenario occurring)(; in other cases, they Mdresse repository performance, formulation and screening of postclosure scenarios); and in still other cases, they provided value judgments (e.g., attitades toward risk and value tradeoffs). De fundamental concepts in the formal elicitation and use of expert r
udgment are g,eneric and independent of the type of issue the experts address.
- However, the choice of specific techniques during the clicitation pro ess and the way the judgments are used to address a problem should be issue-specific.
%4 1,3 p ta Une F9 -tladasnant Formal methods should be used whenever the benefits are greater than the costs.
Indicators of when the elicitation of expert judgments should be formalized are as follows:
Lack o/ Data. When extensive, noncontroversial dets directly relevant to a problem is lackmg, existing data must be supplemented with expert judgments, and it may be worthwhile to obtain them using a formal elicitation process, nee of the Issues. Formal methods are most appropriate when the expert ments wilI have a major impact on the study and improvements in the quality of ju th judgments are then most worthwhile. Important issues also draw the most scrutiny. A formal methodology promotes documentation and communication and i
l should be employed when the issue studied is apt to receive extensive review and criticism or when the findings will be widely disseminated.
Co of the Issues. When a problem is complex, or when several experts are i
em oyed either redundantly or as a team, formal methods are appro>riate. These met ods can provide the structure so that all participants understanc the methods used and apply procedures consistently.
Level ofDocumentation Required. Formal methods are a vehicle to obtain complete and consistent documentation of the methods and the findings. Informal methods j
often produce documentation that is incomplete with regard to the assumptions and l
procedures used. The critical reviews that the study will undergo, the variety and types of users, and the uses of the.information may also suggest whether a formal udgments may be process should be instituted. In some studies, the expert,Lormal methods are important findin~gs and, perhaps, used in subsequent studies, so needed.
Errent of the Use of Erpert Opinion. When expert judgments are used extensively in a study, formalization of the collection and processing of that information is apt to be consistently, and efficiently using formal methods. Costs that done most accurately,f the size of the effort, such as creation of forms, training, etc.,
are fixed regardless o i
may be spread over many assessments. Also, when similar assessments are to bc made by various experts, formalization of the procedures is necessary for consistency.
1.6 Relationshin of Fr..
=I Une of Ernert Judement to Informal Una. Modeline.
and Data Collection As stated in the introduction, expert judgment enters performance assessments in many places. The question is therefore tiot whether to use expert judgment, but l
whether to use it formally or informally, and how to use it witti other sources of i
information like basic physwal principles, models, and data.
informal use of expert ju ent means implicit and undocumented use. Given the cost of formal expert j ment, this may be reasonable in many instances in n some cases, " semi formal" uses may be advocated, such performance assessment.
at. brainstorming and/or taped group discussions about the issues. In such cases, it is I
l 1
- 7-
)
f f
u i
important to identify carefully the objectives of the use of expert j, dgment and to be sure that its benefits outweigh its costs. Documentation is still important in serm-i formal uses of expert judgments, because complex interactions may be involved.
Peer review should not be confused with the formal use of expert judgment. Both i
peer review and formal expert judgment are explicit and documentea processes to i
meresse the likelihood that a resolution of an issue is of highest quality. However, the formal use of expert judgment attempts to bring out the available information that bears on the problem as part of its solution, while peer review evaluates and criticizes a given approach and solution to a aroblem. It should be noted that formal use of expert judgment can, and often shoule, be subject to peer review. Thus, these processes are compatible.
l When formal expert judgment is used, a question arises about how it relates to other l
activities such as collectmg data or modeling phenomena and processes. A simple i
answer is that any of these means of obtaining and quantifying information should be l
used in a cost effective mix that solves the particufar problem. In addition, formal expert judgment can often be beneficial m integrating diverse sets of data and modeling activities and results. Thus, expert judgment and data collection and
(
l modeling activities should never be seen as substitutes, but as complements.
To contrast formal expert judgment to data collection and modeling, consider its favorable and unfavorable properties (Einhorn,1972). Expert judgment is a flexible and general source of information. A formal expert judgment process is unique in i
that it can readily incorporate many disparate pieces of information into a coherent evaluation. Formal expert judgment, though, does not possess some properties of well behaved experimental / statistical data. For example, mereasing the number of i
i experts whose judgments are collected does not ensure that the average" judgment will somehow converge to the true valse. Nor can the usual assumption of independence and the assumption of convenient underlying distributions be called forth for use in expert. judgment processes as they often are in the analysis of t
experimental data. It should be noted, however, thatexperiment in most com Formal expert judgments will not be as precise and clear as computer or l
mathematical models. However, these models build on expert judgment and may also suffer from the same limitations. Models that do not account for unforeseen i
factors or ignore potentially important variables fail in the came way that expert judgment fails when an expert or group of experts do not properly recognize or I
account for all important factors.
l l-i
!- l
\\
- 2. AREAS IN NEED OF FORMAL EXPERT JUDGMENT IN HLW DISPOS Expert judgment has been used and will be used in many aspects of performance nuessment as well as in other analyses, evaluations, and decisions related to HLW disposal. However, it may not be useful to formalize all expert judgments. As discuued in Section 1.3, there are m advantages and drawbacks to formal expert 1
pudgment, and consequently, the decision of when to use it has to carefully consider
>cnefits against costs.
i In this chapter five areas of performance assessment in HWL repositories are j
discussed for which the benefits of formal expert judgment may outweigh its costs.
i model These five areas are (1) scenario development and screening, (2) ion,and development, (3) parameter estimation, (4) data collection and experimentat (5) strategic repository decisions. This chapter does not describe these areas in a comprehensive manner, but rather highlights those aspects in which expert judgment is likely to be formalized. It should be noted that some of these areas are described in detail elsewhere (Cranwell et al.,1989; Bonano and Cranwell,1988).
2.1 Scenario Deselopment and Screening To carry out a comprehensive performance assessment of the possible releases of radionuclides to the environment and to obtain probabilistic auessments of these i
releases and the resulting health effects, an ana:ysis should consider the possib!c future states of the repository as influenced, for example, by climatic, geologic, and h drologic changes in the natural repository environment as well as by changes in the ysical and chemical characteristics of the man made repository system.
f ecognizing this need to consider the repository system and its changes comprehensively,, both the NRC (1983) and the EPA (1985) require that all abysically plausible events and processes be considered in a performance assessment.
_1 i
En this context, events are discrete changes in the evolving states of the repository l
system, while procenes are continuous and coherently linked changes.
Cranwell et al. (1989) describe a methodology developed by Sandia National I.aboratories (SNL for the selection and screenmc of scenarios. This methodology was developed for)the NRC and is currently used
>y a number of countries in their nuclear waste disposal programs. (Scenario Working Group, Nuclear Energy Agency, Organization for Economic Cooperation and Development, Paris, France.)
A though other approaches with a slightly different focus are being developed j
L (Thompson et al.,1988), DOE is also expected to use the scenario approach in performance assessment analysis of an HLW repository at Yucca Mountain.
initial identification of plausible events Scenario selection and screening involves (1)d processes, (3) initial screening o and proceses, (2) classification of events an unimportant esents and processes (4) combining of important events and processes into scenarios, and (5) screening of scenarios to arrive at a final set for consequence aa.alysis. Both for screening and for subsec vent analysis, each scenario is assigned a probability of occurrence during the retu atory period (i.e.,10,000 years). Exp,ert judgment is used in all steps of scenario selection and screening and in the estimation of probability of occurrence of scenarios as summarized below.
l
. 9-
4' e
I 2.1.1 Identification of Events and Processes ne initial listing of physically plausible events and processes is a creative task that depends almost exclusively on expert judgment. Dere is no widely acce >ted method for arriving at this list, and the*: as no method for ensuring that al otentially defining a significant events and processes are included in the initial list (excep)t category like "none of the above" and thereby ensuring completeness.
ormalizmg l
expert jud gment is one means of decreasing the likelihood that important events and L
processes aave been omitted. Formalized expert judgment is likely to be more useful than ad hoc methods because it draws on a varactv of emerts, and because it is documented it can be scrutinized by many indivisuals nd groups interested in e
including events and processes that they consider significm.
H 2.1.2 Classification of Events and Processe*
i L
For completeness and organizational purposes, events and processes are often human induced, and repository 6duced. Often, the classified as naturally occurring,ified as affecting either the release of radionuclides events and processes are class frc'n the repository to the geosphere or affecting the mig, ration of radionuclides through the geosphere. Expert judgment combined with prmeiples of groundwater flow and transpen phenomena is used to classify events and processes.
- 2.1.3 Screening of Events and Processes The initial list of events and processes is often generic. Thus, the list should, in principle, be shortened on a site specific basis. That is, events and processes must be screened for each W. The NRC (NRC, 1983,1988) suggests to classify the events and processes into Anticipated Events and Proc" Natural geological events and processes e
presently occurring or known -
c:urred during the Quaternary Period (1.8 anillion years ago to the prese:.
addition, one may want to consider natural i
events and processes that are not presently taking place but may be anticipated sometiae in the future.
Unanticipated Events and Processes Natural and human induced events and processes that arc not likely during the 10,000 year regulatory period but are sufficiendy credible that they cannot t>e ignored.
Not Credible Events and Processes - Events and processes outside the othe' two e
categories.
Anticipated events and processes and unanticipated events and processes, according to the NRC (NRC,1988), must be considered in the development of scuarios foi a performance assessment to demonstrate comp (!!ance with the conta 10 CFR Part 60.113 (NRC,1983). Events and proces.i s that are not credible can be eliminated from further consideration. Classi)ing events and promses into these categories depends on the exp:rts' interpretation of historica! record:, site-characterization information, and conceptualizations of the future of the repository and even of human behavior. This interpretation will, in turn, depend on a given 10-q l
l n
u.
OUW expert's technical background and may depend on the information base and approach to the problem. Some aspects of the classification can be highly speculative i
because the meaning and interpretation of information depend on how an expert i
visualized the evolution of the system. In addition, the screening process depends on the expert's definition of " credible."
2.1.4 Formulation of Scenarios 1
Scenarios are formulated from all possible combinations of ever.ts and processes remaining after scree ning. Typically, an event tree is used to generate at: possible combinations of events arid processes. The procedere is straightforward if the initial list of events and processes is fairly complete and potentially significant events and processes have not been screened cut. While this can, in principle, be donc j
mechanically, expert judgment is needed to prune first-cut event trees and to check their consistency and completeness. The formulation of scenarios can also be done i
using fault trees by working backvards from potentially important future state (s) of the dis x> sal system and relating there outcomes to possible causes. Expert judgment j
is necc ed in identifying the states and in deriving common causes of sets of events.
In most cases, both event trees and fault trees should be used.
j i
2.1.5 Screening of Scenarios An initial screening of scenariot is based on (1) physical reasonableness, which eliminates physically impossible or implausible combinations of eve.its and processes, (2) the consequence of spenarios, which eliminates those with little or no impact on likelihood of occurrence. In this manner, the
)
repository performance, and (3)d. Expert judgments play an important role in this number of scenarios can be reduce preliminary screening by developing criteria for screening and applymg them.
2.1.6 Probability of Occurrence Probabilities need to be assigned to scenarios for two reasons: to disregard from fur'her consideration scenarios less likely than the screening criterion and to quantify the likelihoods of remaining scenarios to estimate cumulative radionuclide releases and health effects.
Expert judgment plays a significant role in estimating probabilities of occurrence for scenarios. Ideally, some historical data exist for a given site on climatic chan ges, human intrusion, etc., that can be used to formu ate seismic activity, volcanic activity,d to predict the evcntion of th: site (a similar models and provide input use approach to the global modeling advocated by Thompson et al.,1988). Expert jnagmeat is used to interpret the data, estimate the numerical values of model parameters, and, finally, to interpret the results of dmulations and arrive at
{
probability estimates. More realistically, data are likely to be scerce. Data for some phenomena (e.g., human it trusion) may not exist or models may be nonexistent or madequate. Expert judgment is then the main basis for estimating probability.
1 The probat>ility of occurrence of the scenario is a combinatica of the pro'babilities of its individual events and processes. Expert judgment plays a major role not only in determining the probability of the events and processes, but also in the way these probabilities are combined to arrive at the probability of the scenario. For example, i
.p.
x..
y-experts are likely to be used to decide whether a scenario's events and processes occur in a sequence and, if this is so, to determine the sequence.
1 2.2 Model Development
,l In a performance assessment, assumptions and simplifications are made about the i
behavior of the repository system that can be incorporated into a " conceptual model" for mathematical sunulation of system behavior.
i Conceptual modeling of an HLW disposal site is based on a combination of the application of physical principles and data interpretation. Once the models have been developed using wliatever information or data are available (e.g., from small-scale, short term experiments), confidence must be built that the models are adequate to predict the behavior of the system over much larger spatial and temporal
. scales. Both the development of conceptual models and confidence building are creative and interpretative activities that are largely founded on expert judgment.
2.2a Data Selection and Interpretation
.Model development is based on limited, site-specific information about the system geometry, past and active processes, and potential disrupting proce.sses and events.
. ittle or no data will be available to determine all of these factors at the p,roposed repository location. Therefore, experts select and interpret data from similar sites i
and relate them to the repositon site. Interpretations of scant geologic data are used to ddine the system geometry. Experts must infer such things as the geologic continuity between drill holes, the extent and thickness of units, and the extent and character of geologic discontinuities such as faults. The geometry defm' ed b these experts is based not only on interpolation and extrapolation of the site specif data, but on data from similar geologic eraironments. Many processes are ective in the j
geosphere i.e., water flow, vapor flow, heat flow, etc.). Experts select and interpret data to de ide which processes to consider in assessing the performance of a j
repository system. Not only do the exptts have to decide the current dominant processes, but they must predict future processes that could adversely affect the repository system. This later assessment requires the experts in i:lentify and imerpret data from sunilar systems (i.e., analogs to the future states of the reposito Direct measurements of system performance (i.e., integrated discharge over 10, years) will never be available, so inferences about the possible system behavior and the accuracy of system models are from indirect site measurements and from information j
about sunilar systems.
2.2.2 Development of Conceptual MWels Data cannot be collected over the temporal and spatial scales of interest in i
performance assessments of HLW repositories, so considerable data interpretation is required to formulate conceptual models. Because the conceptual model is the foundation of the mathematical models, com,puter codes, and data collection supporting performance assessment and because its development relies so heavily on excert judgment, formalized expert udgment could be most beneficial in modeling.
A' conceptual model inclut% simpli cations and assumptions about )the geomety of the system,
) the curwnt or future physiochemical processes,( ) the mundary and initial con ions, and (4) the parameters governing these processes.
2-12 f
~ -
~
t J
i De most common approach to conceptual modeling begins with a rough sketch of j
the model and continues to refine that sketch based on waatever experimental data and other information are available until an adequate first cut model is produced.
Typically, this is done by using one expert's judgment and interpretations of experimental data and other inTormation. To mace conceptual modeling more comprehensive and to encourage considerations of alternative models as well as scrutability of the experts' reasoning, Bonano and Cranwell (1988) suggest an approach for formalizing the use of expert judgment with multiple experts well versed on the, groundwater flow and transport models. The approach forces the i
experts to articulate all assumptions, and to look for interpretations that challenge their conventional wisdom and are consistent with available data. De second point could lead to alternative conceptual models. Finally, the approach could include procedures for allowing the experts to identify bounding analyses and experimental i
investigations aimed at distinguishing between alternate conceptualizations and eventually reducing their number.
1 2.2J Confidence Building 1
After conceptual models for the disposal system have been assembled, appropriate mathematict' models and computer codes must be developed to simulate the behavior of the systent over the spatial and temporal scales prescribed by the regulations (5 km and 10,000 years).
Experts are an integral part of limited scope activities to build confidetice in models and codes. For example, international groups have been formed such as INTRACOIN, HYDROCOIN, and INTRAVAL to select problems of common interest to the radioactive waste management community. These are simulated by interested parties, and the results are compared. The'se groups attempt to find discrepancies among the results from different experts and their causes. One important result is that the group may implicitly or explicitly agree that, given the current state of the art, existmg models and codes are as good as they can be. To date, these poups (specifically, INTRACOIN and HYDROCOIN) have focused on benchmarking activities that are an aspect of " code verification."* De recentiv started INTRAVAL program goes one step further in that it aims at " validating
(
conceptual models, mathematical models, and computer codes."
Validation means comparing the predictions of the models to experimental results.
Because the models' predictive capabilities cannot be fully tested, "true""alidation can never be achieved. De alternative is to build confidence in the mt 'els and L
codes through a synthesis of experimen:s and calculations. Experiments are ilkely to include laboratory and controlled field investigations as well as natural analogs.
Calculations could consist of bounding analyses and preliminary overall system e
' Verification is defined by the NRC as the " process of obtaining assurance that a given computer code implements the solution of a given mathematical model."
"NRC defines validation as the " process by which assurance is obtained that a model as 2mbodied in a com}, uter code is an accurate representation of the process or system for which the model is intended.".
l performance assessments. In any case, experts (1) design experiments and calculations, (2) establish the validity and limitations of these experiments and calculations, (3) define appropriate measures to ascertain the predictive capabilities of the models and codes, (4 ascertain the validity of important couplings in the models that casmot be tested,)(5)bility of the models to e interpret the results of snodel runs agamst existing i
and new data, and (6) judge the a and spatial scales.
2.3 Parameter Estimation Performance assessment predictions depend on the numerical values of the
- mrameters used by their models and codes. Selecting appropriate numerical values Jor parameters and quantifying the uncertainty about t iem is a difficult but important aspect of performance assessment. First, im mrtant parameters must be identified, and then uncertainty!scussed below.quantificc
. Fxpert judgmer> is important in both in their values of thest aspects, as d It might be worthwhile to define th: terms "pararrieter" and " data." Parameters are i
coefficients or constants of models and processes that describe or control the
. behavior of a model. Coefficients refer to the proporticnality constants such as i
- hydraulic conductivity and diffusivity needed in rate equations such at Darcy's law and to the mean and standard deviation of a probability and Fick's law, respectively, lues taken from experiments, observations of physical distribution. Data are va processes, or other sources, as well as functions (parameters such as the mean or variance) calculated from them.
2.3.7 Identification ofimportant Parameters Conceptual models enhance the quality of a performance assessment (e.g., improving the description of uncertainties about cumulative radionuclide releases and their effects on humans). Therefore, parameters should be identified to enhance the likelihood that their quantification leads to improved performance assessment.
Initially, the identification and selection of important parameters requires substantial judgment t.y the experts who decide how a given parameter may affect the l
descriptions of uncertainty for repository performance.
g Once parameters are identified, their relative importance can often be ascertained by sensitivity analyses (i.e., by varying the value of the pararreter and determinir.g the overall variation in the probability distribution of radicaulide omissions or some l
other intermediate performance measures) (Cranwell et al.,1487; Bonano et al.,
1989). For example, Bonano et al. (1989), m their analysis of a hypothetical HLW repository in basalt formations, show that the hydraulic conductivities of some 1
geologic layers were important, while thosc of other layers did not influence the total radionuclide discharge in 10,000 years. These results indicate that to reduce uncertainty about the containment requirement (40 CFR Part 191.13), research should focus on reducing the uncertainty in the value of the hydraulic conductivity for the important layers and not the others. Intuitively, one could have stated a priori that hydraulic conductivity in general is a relatively important parameter. However, for stratified repository sites, it is important to distinguish among the different strata and identify the most important, which can be achieved only with a preliminary l
performance assessment.
l l
l 1
' L
s.
nere are various approaches for sensitivity analysis, but unfortunately, there can be large inconsistencies in the results from different approaches (Iman and Helton, 1985). Iman cad Helton also show that different interpretations of the results from a given sensitivity analysis approach can lead to a different ranking of important variables. The problem is further complicated because not all sensitivity analysis approaches are appropriate under all circumstances.
Thus, expert judgment clearly plays an important role in the identification of parameters, in the selection of sensitivity analyses, and in the assessment of the i
unportance of parameters.
2J.2 Quantification of Uncertainty in Parameters To assess the uncertainty in performance predictions for M,W disposal systems, it is necessary to quantify the uncertainty in the input parameters o'ithe models and codes l
used. %e uncertainty in parameters can be expressed in a variety of ways. One way i
l is to estimate a mean value and the variance about the mean. Another way is to j
determine the range of possible parametric values and to assess a probability density function (pdf) covering that range. The latter method is conventionally used in i
l y berformance assessment analyses for HLW repositories (Cranwell et al.,19i I
L nano et al.,1989) because it provides a com plete description of uncertainty and s facilitates the generation of multiple samples of the values of input parameters for i
carrying out Monte Carlo simulations. For these reasons, the examples below focus on the assessment of pdfs for input parameters.
a l
In principle, estimation of the possible range of values and pdfs of inout parameters should rely on a very large sample of field data. However, such a larg'e sample is not j
likely to be collected at s candidate re msitory site. Expert judgments are requ' red to determine what samples to take aitc how to interpret the results and to assesr. a probability distribution on the basis of the sample. Using " ayes' theorem, expert judgments can also be combined with data to arrive at a revised pdf for a parameter.
Techniques for the clicitation and use of expert judgment can also be applied to
. quantify expert knowledge on a given parameter (e.g., hydraulic conductivityj to form a " prior pdf for that parameter. If n observations are obtained during site characterization, a joint distribution of the n observations can l>e constructed. This joint distribution from collected data is used to modify the prior pdf to arrive at a l
posterior" pdf.
4 Given the, experts have to decide on what to sampic and given that financial and 1
l L
other practical considerstions are likely to prevent the collection of targe amounts of with documented 2
lement samplin ;Runchal (1989) in data, it is imperative that expert judgments su Merkhofer anc and traceable procedures. 'ITie study described Section 1.4 is an example of the use of expert ju gments to quantify the uncertainty in the value of key parameters.
Another area in which expert judgment may lay a considerable role is in the quantificat.on of the spatial varisbility of h drologic parameters. Although 1
geostatistical techniques (such as kriping) exist these purposes, they require input information, such as the mathematid form of the covariance function (describing L l 1
-an-.
.w-.. -, -, -,,, -,,,, - -,, -, -,., - -..,, -
,.,n-n,.www--
I s >atial correlation), which is likely to be determined using expert judgment (see E onano and Cranwell,1988).
2.4 Information Gatherine Expert judgments are used with other sources of information to improve behavior predictions for the repository system. The current state of knowledge serves as a basis to decide what type of mformaJon should be collected and how it should be collected to predict the future behavior of the repository with less uncertainty.
collection of site-Additional information can be gathered in a variety of ways:
specific data, collection of related off-site data, laboratory experiments, and analysis with model systems. Expert judgment is important in selecting among the alternatives to obtam more mformation.
The activities to obtain new information are likely to depend heavily on expert jud gments. If field data are to be collected at a proposed disposal site, experts must adc ress issues such as the test to be conducted; the number, location, and depth of drilled boreholes; and interpretation of collected data; etc. In laboratory experiments, experts deal with issues such as how representative the experiments are of field conditions; under what conditions the experiments are likely to be invalid; how the laboratory data crc to be used with field data; etc. Finally, if analyses use existing models to supplement experimental information, experts need to address issues such as how the adequacy of the models was established; what key assumptions are in the models that carinot be tested; and how to select the parameter values in the model(s) so that they represent the current state of knowledge about the disposal system, etc.
When contemplating any of these questions, one should consider the prior knowledge about the repository and its performance, the possible changes that could be produced by~new information, the likelihood of these changes, and 11e cost of the information against its benefits. Clearly, any of these considerations requires a substantial amount of expert judgment, both about uncettainties (e.g., the prior uncertainty about a parameter) and about values (e.g., whether a million dollar experiment to decrease the uncertainty about a parameter is worth the cost).
2.5 Stratenic Renository Decisions The four areas of performance assessment discussed in Sections 2.1 to 2.4 pertain to the need for formal expert judgment within a repository designed, constructed, and operated according to a given set of specifications. Hence, the performance assessment largely depends on decisions about the design, construction, and operation of a repository,which will affect the postclosure behavior of the repository.
For example, repository induced events and processes must be considered in the development of scenarios (Section 2.1). All these decisions must rely heavily on expert judgment.
Many design decisions are critical. For example, the exact depth and size of the repository needs to be determined. The angS of the shaft to deliver the canisters to the repository needs to be decided. There are important decisions concernin; the 1. '. '
or horizonta ly or exact placement of the canisterr. Should they be placed vertically? These decisions g%
at some other ar.gle? And how near to each other should they be 4,p1 1
@]
could impact postclosure regulatory requirements such as canister lifetime and release rate from the engineered barrier system, which, in turn, could affect radionuclide transport through the g sphere and release to the biosphere. Clearly, L
these decisions require both factual dgments (e.g., the lifetime of a canister), and value judgments (e.g., the worth of a ing engineered barrier systems) from experts.
For each of the design decisions, there are complementary construction decisions.
There may be different alternatives to sink and enlarge the shaft to reach the both in repository. Different alternatives may be useful for excavating the repository,ials may terms of the techniques used and the timing of the activity. Different mater l
be used to insulate the shafts, and different engineerin g solutions naay be found for constructing the repository Doors and walls. All these decisions affect the reposi'ory performance and involve crucial expert judgments that weigh performance against the costs and preclosure benefits.
Rt.pository operation during the preclosure period also influences postclosure
' performance. For example, the management of the placement of canisters affects the degree of compliance with the design concepts of engineered barriers. Some decision problems may be necessitated by design or construction errors. Others will necessarily need to account for the possibility of such errors. In a similar vein, decisions about removing slightly damaged canisters or leaving them in the repository 1will affect long-term repository performance. Any of these decisions requires both l
' factual and value-laden expert judgments.
l l
The general point here is that one cannot examine expert judgment in (postclosure) performance assessment in isolation from the preclosure decisions and the nbmerous l
I expert judgments involved in them. Simply put, postclosure expert judgments are l
only as good as the pr: closure assumptions.and judgments on which they are based.
- 1 1
1 l
l
)
1 9
i 47-8
.9 5
3.
ELICITATION, USE, AND COMMUhlCATION OF EXPERT JUDGMENTS in Chapter 2, five critical areas in need of expert judgment in performance assessment of HLW repositories were identified. This chapter describes the available formal approaches to elicit, use, analyze, and communicate expert
. Judgmeat.
Section 3.1, defines the main terms used in formal expert judgment processes. While the specific problems and the, applicable techniques for eliciting, expert judg,ments sary from situation to situation, the overall process is generic. It consists of identifying the clicitation issues, selecting the experts, training the experts and carrying out the clicitation sessions (Section 3.2). W! thin this process several techmques are useful, depending on the specific task at hand. These include techniques (e.g., selecting (e.g., generating scenarios or conceptual model identification techniques l
scenarios), quantification techniques for probabilities (e.g.,
quantirmg uncertainties about a parameter), and quantification techniques for values
.(e.g., evaluating alternative conceptual models). Man variants of these techniqu:s i
are described in Section 3.3. Once individur.1 expert j gments are elicited, they can
. be analyzed and used in a variety of ways. Section
.4 describes the issues and procedures for combining expert judgments. Dere are several aporoaches to
- communicating expert judgments. These include the specific form of docamenting expert judgments and of presenting the results of expert elicitations. These approaches are descriited in Section 3.5. Finally, Section 3.6 discusses the interpretation. use, and miruse of expert judgments.
3.1 hef1nitions his section defines some technical terms used in this report such as frsue, Judgment, expert, and probability, andfactual, value, quantitative, explicit, andformaljudgments.
A repository issue is a question about the present state of a repository, its future state, or events and processes that may lead it from one state to another. Issues may con:ern assumptions about the repository and the related natural and human systems. Issues may also concern the method of analysis for performance assessment. Issues are questions that should be addressed to carry out a performance assessuent.
A fudgment is an inference or an evaluation based on an assessment of data, assumptions, criteria, and models. There are two basic types of judgments:
Judgn.ents about facts and juogments about values. Judgments about facts are usually called beliefJ or opuuons. People express their beliefs or opinions regarding propositions about facts or events whose truth or falsity can, at least in principle, be proven. For examole, a person may believe that a nuclear waste repository will cost m excess of $20 bilion in 1988 currency. Or a person can have the opmion that there will be no radionuclide discharges to the accessible environment from a nuclear waste repository within the first thousand years following closure. Although it would take 1003 years to determine the truth about whether such discharge occurred, this is in principle possible.
l L
. L
i Judgments involving the use of criteria, priorities, and tradeoffs are usually called 1
valuejudgments. There is no possibility of proving a value judgment true or false as i
she value of the j
can be done with factual judgments. For example, when comparin;the public, some health benefits for workers with the health benefits for members oL
>cosle might conclude that a worker fatality avoided is as important as a public fata iry averted. Other ;>eople might conclude that a ublic fatality averted is more im >ortant because wor cers take the risks voluntaril. Such differences in value j
juc gments are quite legitimate expressions of di erent social philosophies or pnonties.
Many judgments mix factual end va:ue elements. For excmple, beliefs about the costs of a nuclear waste repository, coupled with a value judg, ment about the socially could lead to the desirable tradeoff between costs and benefits of the repository,be!iefs about the conclusion that the repository is "too expensive." Similarly, bout the relative predictive ability of a mom, coupled with a value judgment a
[
importance of predictive ability vs. simplicity, could lead to the conclusion that the model is " adequate."
An superf. has or is alleged to have superior knowledge about data, models, and rules i
in a specific area or field. Expertise is characterized by easy access to relevant information and by the s.bility to process that information and to use it effectively.
Shanteau (1987) observed other characteristics that define experts: the ability to simplify complex problems and to identify and react to exceptions; a strong sense of responsibility; coafidence in their own judgment; and adaptability related to their imowledge domain. The domain of an expert can be a factual domain (e.g., a scientific data base) or a value domain (e.g., the area of poliev tradeoffs). Factual and value domains are often mixed, however, and one of the characteristics of expertise is the ability to separate factual and value components of judgments. For example, experts decide what data are relevant, what models should be used, how to Interpret data to make recommendations, etc. Any of these decisions involve both value and factualjudgments.
Expert judgments can be impIlcit or explicit. An explicit expert judgment is stated and documented for others to appraise. For example, when a particular conceptual model for a repository is chosen, the reasoning behind that choice can be made explicit in writmg. Or when a numerical estimate of a parameter value is chosen, supporting evidence can justify that choice. In contrast, implicit expert judgments are not availa >le for appraisal and need to be inferred from actions and statements that are available for appiaisal. For example, when screening scenarios, certain screening criteria may have been applied, but these criteria or ticir rationale may not have been explicit.
An explicit expert judgment can be quantitative or qualitative. A quantitative judgment exresses opmions or evaluations in numerical terms. Examples are the estimation of a parameter or the judgment of a probability of an event. Another example is the statement that public fatalities are four times as important as worker fatalities when evaluating health impacts from the repository. Explicit qualitative judgrnents are often expressed t.s verbal statements like "acceptabie," "high chance,"
virtually im >ossible." The decision that "reasonab!c assurance has been or provided that aL1 regulatory requirements will be met is an explicit qualitative I '.
o
...s BRW nt. Many ( m.litative judgments enter scenario screening and conceptual-I selection and may be used to make the judgments explicit.
Quantitative expert judgments about facts' enn be expressed as probabilities.
Probability is a degree of belief in an unverified proposition (DeFinetti,1937; Rarasey,1931; Swage,1954). Probabilities record the state of knowledge that an expert has about a specific proposition. 'These propositions can be about uncertain events (e.g., "there will be an earthquake of magnitude 7 or higher on the San or about uncertain quantities (e.g., "the Andreas fault within the next 30 years") dium A"). Uncertain quantities are also sversge travel time of radionuclides in me called random variabler. Probabilities are numbers between 0 and I (inclusively), and they obey the laws of probability theory. Nonprobabilistic quantitative judg"ments include ranges of parameters or point estimates such as the "best guess of a parameter value.
Quantitative judgments about values can be expressed as utilities. Utilitics express the tradeoffs among attributes of the alternatives to which the value judgments are relevant (Keeney and Raiffs,1976). For example, in selecting expenments for testing a given performance assessment model, a tradeoff is made between the information to be gained and the cost of the alternative experiments. Possible tradeoffs may be between the costs and benefits of laboratory experiments vs. field tests.
Decision analysis is a systematic, procedure to assist experts and decision makers in making, judgments and choices m the presence of uncertainties, risks, and multiple conflictmg objectives. Decision analysis comprises a philosop,hy for problem solving, formal axioms and models for inference, evaluation. and decision making, and a set of techniques for their implementation. Decision analysis includes techniq'aes for decomposmg issues and problems, quantifying expert opmions and salue judgments, analyzmg and using these judgments, and recombining the decomposed problem.
3.2 The Process of Elicitine Ernert Judoments 3.2.1 Identification ofIssues and Information Needs In the previous section, issues were defined as questions about the present stm of a repository, its future state, and events and processes that may lead it from or,e state to another. Resolution of ismes improves the quality of decisions about the i
repository and, as a special part of such decisions, the quality of performance assessments.
Issues range from gencial to fairly specific and from extremely complex to simple.
l For example, a general, complex question may be, "Which conceptual model provides an adequate description of the past, present, and future states of the repository?" A fairly specific and somewhat simpler question may be, "Within a l
given conceptual rnodel, what is the appropdate numerical va'ue of a parameter describing hydraulic conductivity?" Issue identification may involve identUication of the 3cc!ogic and hydrologic features of the repository, identification of all major failure modes and pathways to the accessible environment, and identification of l
l possible conceptual models ar'd scenarios for analyzing failures.
, (
s.
Early in issue identification, emphasis should be on broadening the range of issues rather than narrowing it. It is often useful to invite persons outside the analysis staff to participate in this early stage. For example, public interest groups may be asked to 1
express their concerns, objectives, and potential scenarios regarding failure modes in the repository. External review can aid in achieving completeness of the analysis and curtai, criticism for failing to examine some issues. Examining and discarding an L
issue will be more acceptable than justifying, after the fact, why the issue was not considered at all.
Once a comp:ete !ist of candidate issues has been created, it.should be screened to identify those most relevant to repository performance. Relevance includes both i
%dgments of the likelihood that an issue influences the overall probability of a lailure at a repository as well as the extent of the possible consequences of failures.
Screening should carpioy both criteria.
After reducing the set of issues, information needs should be identified. In making decisions about the acquisition of information, consideration should be given to the relative accuracy, cost, and availibility of alternative sources of informrtion. The result, again, is not a final list, since the issue under consideration will be further analyzed and reviewed as issue descriptions are formulated and decomposed into subissues.
' Clearly laying out the issues for the experts is crucial. If five experts are asked to
).
. write down their understanding of an issue, one is apt to get five somewhat different descriptions. Critical differences ccn arise in the assumptions that experts make.
The understanding of the initial conditions may vary greatly. If these assumptions or initial conditions are not explicitly defined, there can be an ensuing confusion during subsequent clicitations regarding the issue..
1 p
3.2.2 Selection of Experts Performance assessment for HLF repositories requires several types of experts:
generalists, specialists, nrd aormative experts. The generalists should be knowledgeable-about at c.u: overall aspects of the repository, performance assessment. They typie: ally have substantive knowledge m one discipline (e.g.,
hydrology, tre nsport geology,l aspects of the 1roblem. phenomena) and a general under technica However, they are not necessarily at the sorefront i'
of raiy specialty within their main discipline. The specialists, on the other hand, are at the forefront of one specialty relevant to the performance of the repository,isc but they often do not have the generalist's knowledge about how their expert j
contributes to the overall >erformance assessment. Normative experts tpically have l
psychology, and decision analysis, fhey assist training in probability tacory,,th substantive knowledge in articulating their generalists and specialists wi l
piofessional judgments and thought processes so that they can be rneaningful,1y used m the performance assesenent. A high quality performance assessment requires the j
l teamwork of all three types of experts.
l Each expert to be used in a performance assessment should be carefully selected to achieve, a high-quality performance assessment. Operationally, this means that the perfornance assessment team should address all the complex technical aspects of the l
problem and do this in a logically sound, practical manner that is open to evaluation y
l L
21-L
F J
and peer review. The assessment should be politically acceptable, compatible with existmg scientific and governmental institutions, and conducive to learning (Fischhoff et al.,1981).
3.2.2.1 Selection of Generalists L
Generalists oversee completion of the performance ast.essment and provides cuality control for the performance assessment models and resulting analyses. F ence, i
generalists are usually selected from among the professionals wit'nin the organization responsible for the wrformance assessment. In selecting these generalists, project management shoulc, consider technical skills, organizational skills, and personal i
interaction skills. The gene.alists must have an understanding of the technical aspects of the overall performance assessment at a level where they can substan:ively communicate with specialists and normative experts. They should have organizational skills to schedule appropriately the gathering of information for the performance assessment. Generalists also need personal interaction skills to interact eficctively with the numerous project personnel, specialists, and normative experts
. involved in the performance assessment.
3.2.2.2 Selection of Specialists There are three alternatives to consider in selecting s xcialists: (1) a single specialist to provite the set of judgments required, (2) a pone of more than one specialist in which each provides the set of judgn:ents required, and (3) an expert team of specialists with the synergistic knowledge to provide a single set of judgments in situation" rec uiring broader substantise knowledge than is typically, possessed by an individual. The following addresses the identification anc; selection of individual 1
specialists, panels of specialists, and expert teams.
The process of selecting specialists must be considered reasonable. Whether selecting individuals, panels, or teams, the first step is to identify specialists whose judgments might be appropriate for the performance assessment. The performance-l assa.ssment staff may have a number of suggestions for possible specialists. Others may come to mind from reviews of the published scientific literature addressing specific topics of interest. Parties interested in MLW disposal, such as utility companies and environmental groups, may have suggestions for appropriate specialists. Indeed, an open solicitation of nominations for specialists, includmg self-nominations, is one way to instill public confidence in the process. On important problems like HLW disposai, a formal solicitation of experts m the form of a request for expertiae (much like a request for proposal) could be very useful to identify the full range of creertise availabic ar.d to ensure that an adequate search for expertise has occurred. ~Once a list of candidate specialists for use on a specific aspect o' perfonnance assessment is identified, a selection process must occur.
In the selecting a specialist, there are a number of important considerations.
Foremost, it is critical to ensure that the specialist has the expertise necessary. 'Ihis should be verified by reviewing the individual's vita, by discussion witit peers in the field of specialty, and, most importantly, by discussions directly with that expert. It is also important that selected specialists be perceived as having that expertise by peers and others in related fields. If these critena are met, then the potential specialists need to be both willing and available to participate. Ariother key consideration is l
222-t c
r~ _ -
.w+--
-u-+e
-m
.m,-.
. m m.a e
(
)
[.
to have their name attached to their expert judgments in the whether they are willing(Section 3.5.1)ignificanty it increa project documentation
. Naming experts may enhance the quality of the expressed judgments, but more s I
the process and raues its credibility. T1e criteria used for selection shou'd be explicit i
and well documented.
It is very important to avoid any potential conDict of interest between the specialists A common issue is whether the and the results of the performance assessment.
L
' prospective specialists derive their employment or any income from organir.ations charged with conducting the overall peiformance assessment or with constructing'the repasitory. Those available speciahsts with no conflicts should be chosen bas ; on l
their expertise.
Individuals with a perceived or real conDict of interest muy not allow this con 0ict to inDuence their professional judgments. Furthermore, we would not like to exclude crucial information from the performance assessment simply because a knowledgeable individual had a potential conflict of interest. Therefore, it is 1
important to design the oplicit clicitation and un of eyert judgment such that t! :
knowledge and reasons.,, >f experts with potential conflicts can be made known to L
selected specialists in a timely manner. 'Ihis communication process may includ:
distribution of written publications and analyses, as well as oral presentations.
' When a >anel ci spedalists is to be selected, each specialist should, of course, have a high pro'essional stature. However, additional issues are important. One of these is L
~
how many specialists are appropriate. Evidence suigests that three to five experts a
are. usually sufficient to tap most of the expertise (L emen and Winkler,1985). It is desirable to have the full range of legitimate opinions on a particular scientific topic
}
available on any panel of. specialists and this implies that the specialists cn a panel should be as independent as possible. Diversity is achieved when the specialists' i
j sourdes of information and their reasoning, processes are different, and ; heir L
approaches (e.g., theoretical models vs. experimentation) and >rofessional training l
are different. Of course, to some degree, all experts wou d likely be at least l
somewhat famille; with the work of other experts in their fields. In cddition, they L
would base their judgments on common scientific and engineering principles and knowledge. 'Ihus, specialists cannot be completely independent, but this goal is I
important because it provides a nore complete picture of the state of :cientific I
knowledge as well as lending credibility to the perform mce assessment by representmg a broader viewpoint.
A quality performance assessment requires the expert judgments based on knowledge and experience in muy disciplines. These expert judgments will need to be logically integrated, along withill other relevant information and data, into models. No expert teanis are necessary if the results of expert judgments from at other times the natural package of information based on experts'y individuals or panels are naturally pack iged to integrate into the anal.
judgments can only be acquired from an expert team comprised of speci? lists in related but synergistic disciplines. An example is a study involving seismicity on the cast coast of the United States. Each expert team was comprised of at twt one seismologist, one geologist, and one geophysicist (see Electric Power Research R.stitute,1986).
23-2
V Each specialist on sin ex wrt team should meet all of the qualifications of individual experts stated above. Tie disciplines whose knowledge is essential to the scientific probleir. under investigation must be represented as part of :ach expert team, ne perforaiance assessment staff and then the expert team itself must ensure that al!
relevant disciplines are included. De performance assessment staff originally selects
. the specialists for the expert team based on project needs and the required scientific judgments. The expert team.nd grformance assessment staff should initially review the tark and outline procedures to combine logically the judgments of various team members to provide'the required overall judgments. if specific expertise is l
identified as lacking from th: team at this stage, the team should be augmented with i
additional specialists possessing the required knowledge.
3.2.2.3 Selection of Normative Experts The criteria for selecting normative experts are essentially the same as those that guide selection of individual specialists. Both the process of selection and its results are important because both mfluence the quality and the perceived quality of the l-ensuing clicitations of expert jud-ments. Normative experts require a sound theoretical and conceptual knowleIge of probability and techniques for eliciting judgments, and they need to be knowledgeable about the psychological processes occurring in the specialists' minds as they are processing iniormation to produce requested results. Normative experts should also have significant skill and experience in working with technical professionals to make them feel comfortable in y
normative expres;ing their judgments and in explaining their reasoning. Finally,bstantively i
experts should possess the communication skills necessary to mteract su i
with project generalists and specialists and to do:cment thoroughly the results of axpert clicitatrons.
l
~
As with specialists, the qualifications of normative experts can be verified by eppraising the individual's vite, discussion with peers experienced in clicitation and with specialists whose knowledge has been elicited by the individual in question, and by discussion with the individual. Unlike the case with specialists, prospective normative experts can be asked to demonstrate their skills in actua' elicitations using L
individuals on the perforniance assessment staff as specialists.
I<
3.2.3 Training The professional literatt'.re on expert judgment clearly stresses the importance of traimng experts in various aspects of the task facing them (Spetzler and Stact von i
Holstem,1975; Merkhofer,19B7; von Winterfeldt and Edwards,1986; Mosleh, Bier, and Apostolakis,1988). Training consists of the following tasks.
familiarizing experts with the expert judgment proccss and motivating them to provide formal judg.ments, giving experts pmetice in expressing their judgments forrially, educatmg the experts about the possible biases in expert judgment and applying debiasing techniques.
To accomplish these tasks, it is desirable to convene the experts individually or as a group before the actual elicitation for at least a day. De training session should be 1
24-
t led by a normative expert with an in depth knowledge and experience in the art and science of formal expert judgment processes.
l De remainder of this section provides some general guidelines and ideas about how i
to accomplish these three tasks.
,i i
Familiarizing J,e experts with the judgment process and motivating them to provide formalfudgn,.nts. In most expert clicitations, the experts are specialists with substantial knowledge in a fairly restricted domain who have developed their own styles of communication and expectations about types of questions they can or cannot answer. They are usually very cautious regardmg conclusions and judgments that may pooear to be beyond the direct implications of data and experimental findings, scientuse reasoning, or models.
Providir,g formal expert judgments is usually unfamiliar to experts, and sometimes it 4
may even be thicatening. They may feel that they will be asked unreasonable questions. In particular, they may worry that they will be asked to provide more precise answers thsa their current knowledge justifies. In addition, they may not understand why they should express their juc gment at :ll, or if so, why in terms of numerical judgments such as probabilities or utilities. Furthermore, they may
- consider the expression of judgment based on incom plete knowledy to be inferior to ithe scientific work that would improve their knowled ge base. Fina ly, they may worry
.that their judgments may be misused or misrepresented.
It is therefore important that the training session address these concerns explicitly.
First, the normative expert, with technical input frora generalists, should provide an overview of the performance assessment and indicate where the specific expert judgments will be used, ne normative expert should point out that the experts werc chosen to accomplish an important task and explain why they are among the more
-l i
suitable for this task. Second, the need for formal expert judgment should be stressed. In performance assessment for HLW repositories, this need clearly arises because there are large uncertainties about scenarios, models, and parameters, and
. data are scarce. In addition, many decisions involve tradeoffs, as in between deselopment cost and predictive accuracy in a conceptual model. Third, the normative expert should stress that there are r.o right or wrong answers tc questions i
about expert judgments and that the purpose of the clicitations is to assess both what t' e experts know and what they do not'know. Fourth, the norniative expert should n
clearly explain that the process of eliciting expert judgments is not a substitute for further work in the expert's fields, but is, rather,' a tool to summarize their current l
information. Formal elicitation of expert judgment often identifies very clearly where sufficient knowledge exists, and where more researd is needed. Finally, the way in which judgments will be used should be explained carefully. If, for example, I.
judgmems are averaged across experts, this should be explicitly stated and discussed p
The normative expert should present a number of examples to illustrate various forms of expert jud gments, nese include implicit and c.plicit judgments, qualitative and quantitalve judgments, and probability and utility judgments. He examples should preferat,iy be drawn from the substantive knowledge domain of the specialists, such as geology or hydrology.
1:
25-
p
- ~
~
~
m s
)
Qs Most experts know that they use judgment in their work all the time, but the specific e
forms of judgments in expert clicitations, especially probability and utility judgments, j
are likely to be unfam list to them. It is therefore usef'il to explam the basic concepts as well as the main properties of probabilities and oJ; ties. Experts should be shown (nany examples of probability distributions and utility functions from within 4
an.d outside of their field.
An important issue in t.ny expert clicitation is the definition of the variable or event for which the judgment is to be expressed. De normative expert should present j
many examples of well defined and f defined events and variables and illustrate J
them with the, pitfalls of poer definnions: misunderstandings, miscommunication, and mappropnate assumptions.
j Even after a thorough training session, some apprehension and wncern may remain.
]
Most of these resnaming concerns can be addressed only in performance of the tasks and it is therefore itore useful to give the experts some practice in clicitation of expert jut:gments rather than discussing the issues abstractly.
Giving experts pmetice in sping theirjudgmenu explicitly. %ere are several asnects J
of expert judgments that require practice:
making implicit judgments explicit, decomposmg probleias, and e
providing numerical judgments, especially probabilities and utilities.
To show how implicit judgments can be made explicit, the normative expert should present the experts with several simple tasks involving judgments and afterwards point out that the answers require judgment and many answers include implicit
- )
assumptions. For example, when asked whether a canister in a repository will leak within the first thousand years, an expert n'ay say that this is extremely unlikely.
Implicit in this judgment are assum ptions about the repository condition and canister j
corrosion. De normative expert should elicit these assumpuons and point out their j
role in the judgments made Most expert judgments can be aided by decomposing the problem. For example, when estimatmg groundwater travel time through a layered medium, an expert may decompose his judgments by defining several layers and estimating groundwater i
travel time separately for each icyer. Judgment of the relative controution of each layer can then be combined with the conditional estimates of groundwater travel time to arrive at an expected groundwater travel time.
l There are several modes of decomposition. For factual jud pents, event trees, fault i
trees, and functional decompositions are helpful (McCormick,1981; Raiffa,1968),
and for value judgments, value trees and objectives hierarchies are used (Keeney and Raiffa,1976). Smce may of these may be useful for representing and decomposing expert knowledge in a spccific problem, it is useful to provide experts with some trammg m each mode.
l The third area of practice is the actual elicitation of numerical values, especially probabilities and utilities. This can be done by carrying out some example clicitations interactively with the group. De literature on cognitive illusions and
.26-
g.a..
^ ' ~ ~ ~ ~ ~,
~
i probability biases (Hogarth,1980; Kahnemann, Slovic, and Tversky,1982; von i
Winterfeldt and Edwards,1986) has many useful examples.
All tasks that are likely to occur in the clicitation sessions should be practiced. At a minimum, the experts should learn to respoad to questions both outside their field and within their field, to factual and value problems, to questions about discrete events and continuous uncertain variab:es, and to difficult and easy questions. It is best to begin with easy questions on discrete events outside the experts' field and to
. end with difficult questions on continuous uncertain variables in their field. This sequence allows the experts to develop a degree of comfort with answering questions before the challenging and presumably mors uncomfortable questions are posed.
Educating experts about biases and applying deblasing techniques. Cogaitive asychologists have identified many biases m expert judgments (Hogarth,1980; Xahnemann, Slovic, and Tversky,1982) Two general classes are motivational biases and cognidve biases. Motivational biases can occur because the expert has a stake in the issue considered that may lead to conscious or unconscious distortions of his udgments. For exam ple, a bridge engineer is motivcted to claim that a bridge that 4
.he just helped to built is absolutely safe (i.e., the probability of it collapsing is zero).
]
Cognitive biases occur when ex serts fail to process, aggregate, or integrate l
appropriately the available data anc information. Most experimental research is on
- cognitive, rather than motivational biases, yet it is important in the training sessions
- to discuss and elaborate on both.
Research on cognitive biases has concemrated on probability, cognitive biases, and this section foc ases on them. However, cognitive biases occur m utility judgments as well. Some recent experiments (Weber et al.,1988) indicate, for example, that objectives presented in more detail tend to be weighted more heavily. Furthermore, cognitive biases can occur when structuring and framing the task at hand. Two comm'on structural biases are incomplete specification of alternatives and incomplete
~
statement of the assumptions underlying judgments. Fischhoff et al. (1978), for example, showed that car mechanics and other subjects often fail to recognize all possible failure modes of a car defect (e.g., failure to start). Experts often make estimates based on " normal" conditions or assumption's, btit fail to make these conditions or assumptions explicit.
Most cognitive biases related to probability judgments include
- Overconfidence Giving probability judgments that express less uncertainty than the experts' knowledge would justify (i.e., too tight or too steep probability distributions);
Ancho-ing Adjusting judgments insufficiently after anchoring on an mitial estimate (e.g., a mean or median);
Availability Overestimating probabilities of events that are easily imaginable or recalled;.
. - ~ -.
~.
1.
Ignoring base rates Focusing on concrete evidence and data as a main source.of, probability judi;ments and ignorina'prios more abstract information ifce base rates and probabilities; Nonregressive prediction Ignoring the unreliability of the relationship between variables and therefore making predictions as if the relationship were reliable.
Training should focus on the more likely biases in the particular performance 3s assessment. In scenario construction and selection, for example, likely 'iases are incomplete esects and assumptions, availability, and overconfidence. In the identification, appraisal, and selection of conceptual models, anchoring and availability, are most likely. In the assessment of uncertainty for parameters of snodels, everconfidence, anchoring, and nonregresive prediction are likely.
Debiasing techniques have only recently been developed (Kahnemann and Tversky, 1979; Fischhoff,1982). For motivational biases, awareness of motivational factors both by the expert and by the clicitor is important. Sometimes it hel s to present the question in the form of a hypothetics1 gamble (Section 3.3.4 to counteract motivational biases. For example, an expert may state that it is abso tely impossible that a nuclear reactor containment fails at aressures below 120 psig. In that case, one
. might ask him, if he is willing to accept a set awarding him $10 in the event that no U.S. reacter containm:nt will fail below 120 psig in the next 10 years vs. the loss of all of his possessions if one such accident occurs. Exper:s should be trained in such stions and be made aware in the training that the clicitator might attempt to
' ias them this way when they suspect motivational biases.
For cognitive biases, familiarity with the taisk, awareness of the bias, feedback, and personal experience with the bias help to reduce it. A useful training exercise is to provide experts with a catalogue of probability questions that are similar to those med in the bias experiments and to let them ex serience the bias themselves. While this does not assure self correction, it at least a erts them to the problem in a more availability, and nonregressiveness seem vivid way. Since overecnfidence, anchoring, fluence a performance assessment, a to 'oe the main problems that might in apcstionnaire that induces these four biases woald make excellent training material, l
l 3.2.4 Conducting Elicitation Sessions L
Tbc elicitation of expert judgments should be br. sed on a well-defined set of issues L
(Section 3.2.1). However, smce the issues are identified before the selection of the experts, the experts may have suggestions for redefining details of the issue they are supposed to address. Before bep,lem decompositions, the events and variable discuss the issues, the possible pros the questions that will be asked. In the clicitation of probability judgments, it is especially important that the events and variables are well defined. In the clicitation eftlities, it is important that the objectives and scales for measuring them are weil defined. For qualitatively described events this means, among other things, that the events are mutually exclusive and collectively exhaustive and that all conditioning events cre def' ed. For quantitative variables, this means, emong other things, that m
l the meaning, dimension, and unit of the varidic are well defined. If events or - -
'. e
.&j variables are ill derined, various implicit judgments may enter the clicitation to fill J
the " definition gap." Different experts may make different assumptions, and the I
elicitators and enalysts may apply other assumptions in analyr.ing the responses, leading to confusion, miscommunication, and poor performance analyses.
If expert judgments provide specific inputs into a performance assessment, it is
'important that they match the requirements of the overall aralysis. Thus, there also should be preclicitation discussion of the nature and amount of expert judgment required y the overall performance assessment.
j Alternative problem decompositions should be discussed, but some discretion should be left to the ex >erts in matching the individual decomposition to their thought processes. In adc ition, there often are alternative means of expressing the clicitation events or variables through probabilistically related events or through functionally related variables. Again, each expert should feel free to choose among the alternatives that best accommodate his or her thinking, as long as the resulting responses can be related functionally or probabilistically to the clicitation events or variables.
It hel?s for the staff involved in the clicitation and one or two generalists or specia ists to think through the whole clicitation process and practice it. Guidelines for the clicitation should be drawn up, and materials (forms, graphs, etc.) should be
' designed for the actual elicitation.
An clicitation is an interaction between at least two people: the specialist and the normative expert. The specialist provides fudgments, for example, in the form of probabilities or utilities, as well as all re evant technical reasoning concerning judgments and conclusions. In addition to verbal statements, the specialist should provide written. materials documenting the reasoning as well as any background material used in preparing for the clicitation.
The normative exnert is knowledgeab!c in the art and practice of expert clicitation, with special knowledge in probability and utility clicitation. 'Ihe noimative expert asks the specialist to provide specific answers to questions regarding the events or variables considered, assists the specialists in explicating their reasoning, ensures that the required information is obtained, checks the consistency of the specialist's judgments especially with the laws of probability, and documents the numerical rer :lts for later processing.
In some clicitations, it is useful to request the participation of a (;eneralist for expertise in the requirements of the overall project and expertise in tie specialist's area. The generalist ensures the technical va idi y and consistency of the specialist's t
judgment, clarifies technical issues, documents the specialist's technical reasoning, and provides technical data and assumptions when needed.
3.2.4.1 Basic Elicitation Arrangements The clicitation should take place in an undisturbed environment, preferably a separate room without telephone interruptions, visitors, or distur6>ing noise. The desk arrangement should be comfortable, encourage interaction among the 29-
^
individuals involved in the elicitation, and have work space and sufficient space for documentation materials, forms, and re' cording devices.
r There are several ways of documenting an ongoing elicitation: tape recording, I
written notes by the normative expert, written notes by the generalist, and notes or documents that the specialist brings into the session. Tape recordings provide a complete voice record. During taped clicitation sessions, it is important to refer explicitly to the materials and documents, figures, and tables used in the d4cussion to facilitate transcription and cross referencing in the written documentation. While tape recordings may provide more detail than necessary, they can be important for accountability, and for verification and clarification during written documentation.
Notes taken during the clicitation session by the normative expert and the generalist have different focuses. The normative expert focuses on wriHng down judgmeats and making lists, tables, and figures summarizing and rela; ag these judgments for communication and feedback. In case of probability clicitation, for example, the clicitator should write down 'he probabilities as tables, distributions, or functions that allow quick consistency checks and calculations for feedback. While most documentation of the normative expert is numerical,it is useful to note on the tables and plots the the s >c.cialist's rationale for certain judgments. De generalist should record the specia ist's reasoning in support of the judgments as well as cross-referencingt ut to the specialist's own documentation. It is important thct the documentation schemes of the normative expert and the generalist are similar so that they can be cross referenced when documentation is consolidated.
p 3.2.4.1 Structure of a Standard Elicitation Session A standard clicitation session begins with easing the~ specialist into *.he situation and mapping out the task. De normative expert should ask the specialist to provide a brief overview of his or her approach to the problem and, in particular, the problem structure and decomposition used. After this exchange, the normative expert should define a road map for the remainder of the clicitation to determine the amount of work ahead.
Next the definition of the events or variables to be elicited should be reconfirmed.
The normative expert should define the events and variables carefully, check the various meanings with the specialist and the generalist and write down the dimensions and units on the forms pre sared for the clicitation. Assumptions, especially about conditioning events, should be discussed and documented.
In the case of a decomposed event or variable, the normative expert should first map out a.ough decomposition to clearly describe the logic used and simplify the Judgmental tasks. Next the normative expert uses any combination of specific Section 3.3) to elicit expert judgment. These techniques range f 3m l
techni ves (itative for identifying scenarios, models, or events to mixed qualitative-large qual l
quant tative for screening, to largely quantitative for probability and utility judgments.
Consistency checks by the normative expert are important to assure the internal logic of the expert judgments and to assist in identifym[, sources of inconsistencies and resolving them. Consistency checks should be used to stimulate the specialist's dO-
thought processes. 'In probability clicitation, for example, it is useful to ask the same i
question by eliciting the desired probabilities directly or by eliciting probabilities for related variables or events. At a minimum, decomposed jud gments should be rea ggregated to arrive at a calculated judgment about the clicitatec event er variable, 4
and this calculated judgment should be compared with the specialist's intuition.
3.2.4J Post Elicitation Activities The specialists should be given quick feedback on the results of the clicitation. In particular, they should be shown the numerical information in the form of tables and distributions. Changes required b the specialist upon such feedback should be adopted and reasons for them shoul be carefully documented.
In some cases, it is desirable to organize a group meeting of specialitts, generahsts, and normative experts after the individual sessions to discuss agreements and disagrecraents and whether it is possible or desirable to reach consensus. There are
~
several ways to organize such an interaction (See Section 3.4.4 and Seaver,1978). In some instances, it may.cVen be desirable to reelicit some individuals after this group i
session.
1 iSometimes specialists may want to change their clicitations after a significant time l
has passed. Such change requests should be probed carefull but accommodated if feasible within the frr mework of the overall project. Reelicit tion may be necessary,
' and the documentation should reflect 'he revisions and the reasons for them.
The basic design also requires clicitin ene special!st at a time. It is conceivable to 4
elicit several specialists simultaneousi, for example, in groups or classroom sessions.
While this method is preferable to a pure questic.maire format, it suffers from some of the.same drawbacks. In particular, cisssroom settin s require more conformity on case structure and decompositions, allow less flexibili in mdividual responses, and l
may suppress expressions of alternative views.
I There are, of course, man variants to the postelicitation activities. An imherent rtant issue is whether the clicitation to achieve group consensus, to aggregate d 4
l l
judgments, on simply to report the results from different specialists (Section 3.4.).
l 3.3 Technlaues for Snert Judement Elicitation An expert engages in three fundamental cognitive processes when making judgments:
(1) siennpcanon of o ions or events to be j,u ed; (2) screening of the options and eventst and (3)qua ation of comparative ju gments about the options and events.
Identipcation consists of recall, search, and creation. Recall identifies easily available alternatives, search systematically lists existing alternatives, and creation generates I
previously unknown or inaccessible alternatives. Screening consists of selecting i
screening attributes, setting screening constrr.ints, and selecting alternatives based on the attri~outes and cor.straints. QuannAcation consists of assig mg numbers to factual i
or value judgments about alternatives. Factual ud pnents a ut events' or random
~
variables are usually quantified by probability di troutions. Value u ments (e.g.,
about the advantages or disadvantages of alternative conceptual mo e s are usually I.
quantified by utility and tradeoff judgments.
l l-
, 1
De literature on identification techniques is fairly small. Dere are a few techniques for creative option and event generation (Pearl,1978; Pitz, Sachs, and Heerbroth, 1980; Gettys, Fish :r, and Mehle,1978; Keeney,1988a). Most screening techniques consist of setting numerical cutoffs on 5 elected screening attributes and searching for the subset of " survivors." ' Keeney (1980 describes the basic idea for screenin g in a value judgment context, and several repo)rts discuss the use of " cutoff pro ities" for screenmg undesirable events (Department of Energy,1986; Okrent,1980; Wilson, 1984).
In contrast to the small literature on identification and screening techniques, there is a rich literature on quantification techniques that draws mainly en psychophysics Poulton,1979; Ekman and Sjoberg,1965; Zinnes,1%9) and decision analysis Raiffs,1968; Brown, Kahr, and Peterson,1974; Keeney and Raiffa,1976; von
)
interfeldt and Edwards,1986). The decision analysis literature typically i
emphasites quantification of probabilities (Spetzler and von Holstein,1975; Selvidge, 1975; Seaver,1978; Keene 1980; Stillwell, Seaner, Schwartz,1981; Wallsten and Budescu,1983; Merkhofer,y,1987) and utilities (Keeney and Raiffa,1976; Keeney, 1980; Edwards and Newman,1982).
(
De following three sections summarize this literature and make recommendationi i
kbout techniques for identification, screening, and quantification.
3.3.1 Identification Techniques identification techniques primarily assist experts in identifyi,ng s:enarios and conceptual models for performance assessment. In scenario identification, the emphasis is on stretching the experts' imagination and on creative processes of event generati,on. Conceptual model identificatiorr, emphasizes generstmg desirable model alternatives.
3.3.1.1 Techniques for Event and Scenario identification Recall and search are fairly trivial ta ks in event and scenario identification. In the
. recall mode, one simply asks the experts to list all the events and scenarios that they recall that are relevant for the normal performance of the repository or for scenarios 1
that could adversely impact that performance. In the search mode, experts survey the literature for relevant events or scenarios. It helps to enrich the set of events and scenarios by asking nonexperts and those with a stake in the decision (e.g.,
environmental groups, residents living near the repository). De emphasi. at this stage should be on completeness and comprehensiveness, not on logic, i
reasonableness, or likelihood of occurrence.
l Event and sce ario creation is the most interesting and innovative aspect of this task.
Here are three cognitive techniques to creative scenario generaten:
forward and backward induction; I
value-driven event and scenario generation; and L
analogy-or entinomy-driven event and scenario generation.
l Forward and backwstd induction builds on the notion that scenarios are logical sequences of events linked through processes. It begms with listing all possible and l
32-a
' O..
0 conceivable events that could occur related to a repository. In the forward induction mode, events are linked to create an event tree that fans out from initiatinJ events to events that may occur in thousands of years. Provided that the events anc processes this event tree can, in principle, be constructed are defined sequentially,ing to a very large tree representmg with thousands of typically lead mechanically, is tree should be pruned to eliminate branches that are impossible, scenarios. H extremely unlikely, or redundant. In the backward induction mode, the final states of the repository are the starting point of the process. A possible final state may be defined as " major releases to the accessible environment occur in the year 3000."
Backward induction defines the possible causes of this final event and thus works back to the initial conditions, events, and processes that make it possible.
Forward induction cally creates too many scenarios, while backward induction may create too few, app'ying both processes and reconciling the results, it should be possible to identify a subset of scenarios that spans the range of scenarios relevant to the performance of the repository.
The second technique begins with the question: What are the performance objectives for a repository and how can they be achieved? Presumably the main objective is to protect public health and safety, but other objectives like cost and long term environmenta; protection may be important as well. After identifying a set of oojectives, events and scenarios are developed that would lead to extremely poor,
, taverage, and extremely good performance on each objective (Keeney,1988a; Edwards et al.,1987). For example, in the case of health and safety, an " undisturbed-performance" or " base case" scenario without major geological events or human mtrusions would presumably lead to average performance. Adding favorable assumptions about the behavior of the canister materials and the rock medium may lead to extremely good performance. Combining major magnetic and seismological events with poor geology and excessive corrosion may lead to very poor performance. While this tecnnique tends to look at the worst case in terms of health and safety, it is very instructive to look rnt other cases and other objectives as well.
analogy or antinomy attempts to stimukte the Event and scenario creation by(Jungermann and Thuering, thought processes of the experts
. In an analogy, one would take the events and scernrios out of the context of an and ask experts to instead think of the repository, for example, as a coal mine containinJ ethal gases. The question would be: What could go wrong in this coal l
mine? Tie follow up question would be: Do any of these coal mine events and scenarios apply to the real repository case? In an antinomy one could ask experts to think of the repository, for example, as containing the most precious human oossession that required protection from attempted tieft. He question might be:
Mow can thieves enter the repository, and how can theft be prevented? Agam, the answers would be checked for their relevance to repository performance.
Any of these three techniques can be combined with various forms of interactions among experts. These include Delphi-type techniques (I.instone and Turoff,1975:
1969), the Nominal Group Technique (Delbecq et al.,1975), and several Dalkey,f brainstorming. Furthermore, they can be substantially enhanced by forms o involving individuals with very different perspectives regarding the repository (e.g.,
local residents, environmentahsts, and nuclear engineers). Since the purpose at this peint is to assure comprehensiveness, any inputs that c.re novel and creative should - -., -,. _
be appreciated. Peer review is another useful mechanism to identify events and scenarios that have been overlooked.,
i It is very important that the activities during event and scenario identification and the results are carefully documented. In particular, reasons for eliminating certain j
events and scenarios should be carefully recorded.
3.3.1.2 identification of Conceptual Models As in scenario identification, recall and search are fairly straightforward activities to identify conceptual models. The main technique for the mnovative creation of Sachs,ptual models is similar to the value-driven technique describe conce 1984; Pearl,1978; Keeney,1988a), he technique be gins with a listing of the desired properties or objectives for a conceptual model. Isext the experts develop features of conceptual models that would serve one objective well. After completing this task with the first objective, it is repeated for the second, the third, and so on.
Features developed from subsets of objectives are combined to characterize one possible conceptual model. Repeating this process suggests many different conceptual models.
Having generated a large number of conceptual models, the next task is to narrow this ser down to a reasonable size. His task includes examining all conceptual
- models on all objectives simultaneously and eliminating those that are clearly unacce ptable on one or more objectives. Since this task involves screening, many of the techniques discussed in the next section will be applicabi,e.
3.3.2 ScreeningTechniques De first step in screening scenarios or conceptual models is to identify the attributes with which to screen alternatives, his step is followed by setting target levels or constraints on the attributes. Alternatives are then screened out that do not meet ue target levels and constraints. Typically, this process is iterative: when too many alternatives survive, more stringent target levels or constraints should be applied.
When too few survive, target levels or constraints should be relaxed.
Identification of Attributes. Scenarios should be physically consistent sequences of events. It is therefore important to screen out those that are logically flawed. For example, if one event is the coming of another ice age combined with the migration i
of the carth's population to the southern hemisphere, it is logically inconsistent to couple this event with large numbers of human exposures because of radioactive leakare. Given another ice age, it is improbable, although not logically inconsistent, that t sere would be exploratory drilling for minerals other than the radioactive materials themselves.
Before eliminating a,particular scenario because of a physically illogical se uence of events, it is instructive to ask several experts to explain the presumabi illogica' sequence. In the above example, some experts may find the combination o icing and exploratory drilling illogical. But others may speculate that the explorator'j drilling for some yet unvalued mineral would go on all over the world even in unfriendly climates, just as it is going on in the polar regions today.
34-i
-v.-.
...__, -, - - - - - - - - -. - - -. _ _ - -. -. _ = _ _ _ _. - - -.
_--.---m- - - - -,. -... - - -
---.e
s# f#
E b
k IMAGE EVALUATION
((j/
$ '4, 4pff;v@gy,,>
s,
<f(,p g
- f restranGET 313) yf
+yp
+>
1.0 lf 82 EL4 5!NE l-l t u lilM t:
=
1.25 ll 1.4 1.6 4
150mm O
4 6"
=
b d *'%
A_
s <g4%
+gy% 77777 7 s,
y u
(M
+ 9)
E
<:#** %k M
////p g[f,4 IMAGE EVALUATION 5[.f f/
TEST TARGET (MT-3)
///p / p 4,g%
4 S>+
'2M d
l.0 llj DE i,l i m HM J1 1.25 1.4
'l 1.6 h__
4 150mm 6"
[*'%
+ gl y &,,,,(, =
/g!(b 4%
oI#L
- k*
m
. qa l
o g>%++@>;,
b 777//,($ '#4 IMAGE EVALUArlON a
+
l.0 lt M M
'ljDE I,l l m lll!E 1.8 l.25 1.4_
l.6
+
150mm 6"
d*
/J4%
+yy>>>%,
- 4%
3 <sp+%
7777 m - :
7 op
- e. ee n
...c a
1 MW Scenarios can also be screened on potential consequences, eliminating scenarios with relatively insignificant impacts, and probability. Probability criteria can be defined on the whole scenario, on individual events, and on part of the sequences of events, in addition, probability criteria can be set differently, depeading on the consequences of a scenario. It is useful to spell out different sets of probability criteria and
]
investigate their use before fixing target levels and constraints.
Attributes for screening conceptual models can be very diverse. Examples include scientific acceptance, predictive ability, ability to estimate the parameters, simplicity, and cost. Techniques for identifying and structuring such attrit>utes are described in Section 3.3.3.
3.3.2.1 Setting Target levels or Constraints a main issue is the selection of probabilities to screen out In scenario screening,ility events or scenarios, eliminating those that most peopic extremely low probab would consider " incredible," " implausible," " virtually impossible," or even
" unbelievable" or " inconceivable." These target probabilities can pertain to an event L
in a scenario or to the total scenario. These probabilities are linked, as the probability of at:y event in a scenario must be larger than the probability of the i
scenario. In other words, if a single event in a scenario has probability p, then the
- scenano has to have a smaller probability pq, where q is the conditional >robability 3
x of all the other event elements of the scenano given the event under consic eration.
o When setting event or scenario screening probabilities, one should consider the L'
possible consequences. A common technique is to define smaller screening
~a probabilities on overall scenarios if the possible consequences are more significant.
For nuclear power plant accidents, for example a screening probability for a core meltdown may be 10 6, but the screening probability for a core melt with containment failure may be set as low as 10 9. A more explicit approach is to set a target level on the probability (distribution or, alternatively, on the complementa
. Yet another approach is to combine target levels with density function NRC,1975)d in Wilson (1984).
potential benefits as describe s
Screening conceptual models is more complicated, since there are more attributes to consider. Keeney (1980) discusses this issue in the context of screening alternative sites for energy facilities. He points out that screening is a simplified selection l
process and as such requires value tradeoffs among the screening attributes.
l To illustrate this point, consider two screening attributes of conceptual models:cost of computer run time and empirical validity. One could set target levels on both attributes. For example, one could say that to be selected, a model run should not cost more than $10,000 and the expected error in predicting radionuclide travel time should be less than 100 years.
Alternatively, one could set the target levels at a model run not costing more than
$10,000 but an expected error in predicting radionuclide travel time of. less than 50 years.
Notice that the second set of target levels is more restrictive on the empirical validity attribute. Thus, in effect, by using the second set of target levels, we assign more 35-
~
.....~. u
9 9
weight to the attribute of predictive validity. This is a general feature of setting target levels and constraints: setting these levels by itself involves crucial value tradeoffs among attributes.
Multiattribute utility analysis (Keeney and Raiffa,1976) makes these tradeoffs explicit and could be used to set constraints and target levels. While a full fledged multiattribute utility analysis may be too costly for the purpose of screeninJ, it is important to be cognizant of the tradeoffs made when setting target leve s and constraints. As a practical rule, it helps to set target leve s and constraints-interactively, starting with very lenient levels and examining the set of surviving conceptual models after each setting of target levels.
3J.2.2 Selection Once attributes and target levels or constraints are defined, the selection is essentially mechanical. It is useful, however, to reiterate and go through a number of changes m setting target levels and constraints to investigate their implications for the selected subset. It is also useful to explain the logic of the process to a broad range of interested parties and to let them critique both the process and the result.
3.3.3 Decomposition Techniques I
Problem decomposition is widely used in scientific study to simplify a complex u
problem into components that are more manageable and more easily solved.
Problem dedomposition has also been recognized as an important tool m expert udgment clicitation (Raiffa,1968; Brown, Kahr, and Peterson,1974; Armstrong.
r
- Denniston, and Gordon,1975).
Problem decom. position in clicitation refers to breaking down issues to provide for
~
casier and less com lex ascessments that can be recombined into a probability distribution or utilit function for the quantity of interest. The recombination is usually accomplishe through a mathematical model that expresses the quantity of L
interest as a mathematical function of component quantities. The techniques decom >osition depend on whether the problem is a factual or value problem. Event tress, Cault trees, and functional decompositions are used for factual issues, and objectives hierarchies are used for value issues.
i.
l 3.3.3.1 Decomposition of Factual Problems
\\
Several t,ypes of decompositions facilitate expert judgment about facts and probabilities. A familiar type of decomposition is the fault tree (McCormick,1981),
which focuses on a possible failure of a system and traces back the possible component causes of this failure. Fault trees are commonly represented as circuit diagrams that display the relations among system components and the failure of a system. In fault tree analysis, the components are assigned probabilities of failure, from which overall failure probability of the system can be found. Usually failures of various components are treated as inde >endent events, although sometimes common causes lead to related component fai ures. Fault trees serve as a vehicle for the decomposition of expert judgments when the component events are dichotomous (0 to 1), inde sendent, and the overall failure event is logically related to the component events. Mowever, when decomposing, care must be taken to ensure that 36-
?..
4
?
completeness is not lost. When finer detail about the causes of failure of some event in a fault tree is so g experience suggests that incompleteness can easily occur (Fischhoff, Storic, an tenstein,1978).
'While fault trees end'in a single failure event and trace its possible causes, event trees begin with an initiating event and draw out its possible conseguences. The event tree lays out the sequence such that the probabilities of ruccessive events are conditional on their predecessors. The branching in an event tree leads to a proliferation of paths, each path having a terminus associated with a system state or cc nsequence. Event trees are a natural means of representation when phenomena j
have discrete outcomes. When the outcomes are continuous, however, the use of event trees requires that the continuous outcomes be approximated by a discrete i
categorization of ranges of the outcome variables.
A related type of decomposition uses the conditioning of possible events on known or hypothesized events (Bunn,1984). The events can be laid out as an event tree j
' where predecessor events are the conditions for the event in question. For instance, the probability of event A may be conditioned on the hypothetical events B and C.
- Ihe assessment task then requires the probabilities of A, given various combinations of B and C and their complements. Further, the probabilities of B and its
, complement, given C and its complement, must be assessed as well as the probabilities of C and the complement of C. Denoting the complement of an event C E by E, the probability of the event A becomes P(A) = P(AlB,C)P(BlC)P(C) + P(AlB',C)P(B'jC)P(C) + -
P(AjB,C')P(BlC)P(C) + P(AlB',C)P(B'lC')P(C').
Barclay et al. (1977) demonstrate the use of this style of decomposition to ascertain
.-l the likelihood that a nation will have the capability of producmg nuclear wea3ons within a given time frame. An analysis and discussion of theoretical aspects of the probability decomposition are provided by Ravinder, Kleinmuntz, and Dyer (1988).
A tree structure related to the event tree is the decision tree (Ralffa,1%8; Holloway, 1979). In addition to possible events, decision trees incorporate choices or decisions i
that partially determine the path followed. Decision trees are particularly valuable in i
the evaluation of alternatives. Decision trees should be he pful in the analysis of information-gathering activities associated with the potential repository and in evaluating design and construction options for the repository.
Decomposition may also use physical models of the phenomena being analyzed. The physical relationship between the quantity of interest and several constituent or determined quantities is expressed through a mathematical function such as T =
f(X,Y,Z). This type of decompositioni is called algorithmic decomposition by l
MacGre or, Lichtenstein, and Slovic (1988). Rat ser than assessing a single ity distributions for X, principle of decomposition leads to t distribution for T, the L
probabili of prob I
distribution for T. If the expert is better able to ex3ress knowledge about the constituent quantities than about the original quantity, tie issue is a good candidate I
l for decomposition. This strategy has been used in the reactor risk reference j
l G7-j 1
{
document (Wheeler, Hora, and Cramond,1989), and the EPRI study of seismicity (Electric Power Research Institute,1986).
^
If the expert >ossesses knowledge about X, Y, and Z and, further, knows the functional re.ationship f, then the expert should be able to give equivalent assessments either in terms of T or in terms of X, Y, and Z. However, the-I L
combination of X, Y, and Z is likely to be too complex for the human mind to 'do without substantial assistance. Decomposition, then, can serve as an aid to human fl L
thought processes in that the mind is relieved of tasks that it is ill equipped to L
perform (Einhorn,1975).
i.
3J.3.2 Decomposition of Value Problems a
value problems is structuring so-called The best-known technique for decomposing;ics structure the expert's g,eneral value objectives hierarchies. Obje,ctives hierarct l-concerns, intermediate objectives, and specific value relevant attributes m a tree like hierarchy in which the lower levels define what is meant by th,c upper levels (Keeney and Raiffa,1976; von Winterfeldt and Edwards,1986). Objectives hierarchies are structured by either the top-down or the bottom up approach. Both approaches are
.nplemented in interviews with experts knowledgeable about the value domain considered. They are illustrated below with an example of evaluating alternative L
. conceptual models.
The top down approach begins with general value concerns like costs, scientific validity, etc., and subsequently specifies the meaning of these general terms at increasing levels of detail. For example, scientific validity could be broken down into face validity, empirical validity, and axiomatic validity. Empirical validity could be further broken down into experimental v'alidation at the repository and empirical
_l validition at other sites. When considering a hierarchy of concerns, objectives, and attributes,it is important to pursue and to eliminate means objectives.
The bottom up approach begins with listing the features that differentiate the options. From this list, features are eliminated that are not relevant for comparative 1
evaluation. Among conceptual models, for example, average run time is value L
relevant because of cost and delay of feedback. On the other hand, place of l
development may not be value relevant. Having screened for value relevance, the n a step is to eluninate means and pursue ends. Finally, the remaining features are mered and organized into a logical hierarchy.
l l
th results of the top-down and bottom up approaches should be similar hierarchies with general value concerns at the top and specific attributes at the bottom. Once a L
first-cut hierarchy is built, the following checks can be used to examine and revise it:
Are any concerns, objectives, or attributes redundant?
Is the set of concerns, objectives, and attributes exhaustive?
Are the concerns, objectives, and attributes independent?
Is the tree manageable for further analysis?
Are the lowest level attributes operational; that is, can one measure and compare, for example, conceptual models on them?
L
\\
138-i 1
.,... -... _ _ _ _ _ _ _ _ - _.. _. - - - _. _ _... _ - _ _ _ _ _. _.. _.. ~.
2 '.
a Checking and revising often involves returning the initial hierarchies to the experts for reexamination.
The previously described decompositions of factual and value problems are fairly formal in that they express the results as trees or functions.
)
3 3.3JJ Variants of Decomposition
)
i Decomposition can also be used less formally. De goal of a less formal procedure 1
might be to promote deeper insight into the rationale for judgments and to enhance the interchange of beliefs and assumptions about the likely causes of studied events without formally encoding the decomposition. The decompositions might be in terms of casual or mitigating factors that are loosely related to the event or quantity of interest. In this form, decomposition enhances the experts' introspection and communication.
A key aspect of decomposition relates to the source of the model or models used as a decomposing framework. De models can be imposed upon the experts from an external source, or they can be generated by the experts. Individual experts may be allowed to choose their own decompositions, or a consensus decomposition may be
- used.
' Using a single decomposition has several advantages. First, the costs of recombining the judgments may be substantially reduced. Experience with NUREG 1150 t
le experts who used indicated that the effort to process clicitations from multip(Wheeler, Hora, and j
unique decompositions was much greater than expected t
i i
Cramond,1989).
Another potential advantage of using a single decomposition is that comparisons can be made among elicitations for component quantities and events. Combining assessments at the component level and then recomposing is also feasible when a single model is employed. Neither comparison at the component level nor aggregation at a subissue level is feasible with multiple decompositions.
A single decomposition by multiple experts also has important drawbacks. First, there needs to be significant discussion to ensure that all experts understand and accept the chosen decompositions, which is often difficult to achieve. Second, the influence that a decomposition has on the ultimate result is considerable. Requiring experts to abide by a smgle model may force their judgments to appear to be in agreement and thus understate their underlying differences as to the appropriate processes and assumptions. And if the decomposition itself is s embodies much information.
The advantage of multiple decompositions is that a wider variety of approaches to the problem are permitted. Single decompositions may understate the true uncertainty about an issue because the experts are forced to conform to a single view.
Research has shown that the method of analysis, or decomposition, is important in forming judgments (Booker and Meyer,12_). Multiple decompositions also provide l
a vehicle for discussion and documentation of alternative viewpoints-an important by-product of the expert-judgment process.
l
.39 l
h l
When an issue requires the expertise of several experts, decompositions are
>articularly useful Teams of experts who collectively possess the requisite cnowledge may be formed to address the issue. Each team member must embrace
.l his or her portion of a collectively acceptable model so that the team's judgments are coherent and based upon the same conditions and assumptions. In such a setting, the decomposition separates the issue into components that can be addressed by members of the team having the relevant expertise. The decomposition also is the basis for integrating the assessments of the team members. A team format where teams had the flexibility to modify their models was used in a seismicity study of the Eastern United States (Electric Power Research Institute,1986).
3.3.3.4 Benefits and Costs of Decompositions Decomposition beyond a point may detract from the quality of the information i
obtained. Decomposition should be done until a balance exists between the difficulty i
of the assessments, the complexity of the decomposition, and the inherent number of I
assessments that must be made. In some instances, no decomposition may bc desirable.
Problem decomposition is beneficial in two ways. One is that the expert judgments obtained through decomposition may better represent the true state of knowledge about the problem. This is because simpler assessments can be made more accurately by the experts because their answers will be better calibrated.
Psychological biases such as overconfidence and the base rate >henomena 'are thought to be less pronounced for easy tasks than more difficult tasks, so decomposing into easier tasks may lessen the impact of these biases (Merkhofer, 1987; Lichtenstein and Fischhcff,1980). Mathematical recomposition of assessments relieves the expert of a difficult integration or aggregation task.
The.second type of benefit from decomposition is the stimulation of alternative views and the documentation of reasoning that follows naturally from a decomposition.
The use of multiple decompositions also helps explain why experts differ in their rationales.
Cost may be relevant when considering decomposition. The number of assessments e
f may increase substantially because many, questions may be required for a single issue.
Beyond this expense, an additional requirement is that com > uter prog, rams or other methods be constructed to perform the recomposition. T1e diversity of potential decom, positions often precludes the use of existing software. Significant analyst effort is usually required to recompose an issue. Decomposition may also produce the false impression of objectivity and sometimes may introduce bias by systematically omitting an important component.
3.3.4 Techniques for Quantifying Pmbability Judgments Probability clicitation techniques are described in several references (e.g., Spetz!er and von Holstein,1975; Selvidge,1975; Seaver,1978; Keeney,1980; Stillwell, Seaver, and Schwartz,1981; Wallsten and Budescu,1983; von Winterfeldt and Edwards, 1986; Merkhofer,1987). In addition, several reviews of experimental validation of these techniques exist (Peterson and Beach,1967; Goodman,1972; Lichtenstein, -
b Fischhoff, and Phillips,1977,1982; Slovic, Fischhoff and Lichtenstein,1977; Pitz and Sachs,1984). Drawmg on this literature, there appear to be four distinct classes of procedures, depending on the nature of the uncertain quantity (discrete events vs.
i continuous random variables) and the nature of the questions asked (magnitude judgments about events vs. indifference judgments about gambles). The resulting
.i taxonomy is shown in Table 3.1.
l l
The eight techniques listed in this taxonomy are the most commonly used ones in the quantification of probability judgments. Before describing these techniques in detail, l
it is useful to spell out some general guidelines for probability clicitation that are applicable to all eight techniques, j
i i
Table 3.1 i
Taxonomy of Probability Elicitation Techniques I
Judgment
- Variable 1
Magnitude judgments Indifference judgments about events about gambles Discrete Direct probability Reference gambles (discrete)
Events Direct odds Certainty equivalent (discrete)
Contihuous
~ Fractile technique Reference gambles (continuous)
Quantities Interval technique Certainty equivalent (continuous)
J l
i l
First, it is important to begin with easy questions. For example, when comparing the probabilities of two rare events, an expert may initially have no feeling for the L
absolute magnitude of probabilities, but it may be fairly easy to establish a rank order of the relative likelihood of the events. Second, it is preferable to select observable quantities for eliciting probabilities. As an specific case, one observes failures of equipment rather than failure rates. Assessing the cumulative probability for the number of failures with 100 units originally operating for a fixed time period in extreme conditions may be casier than assessmg the probability for the likelihood (i.e., a parameter) that an individual unit will fail in that time period with those conditions. Third, it is useful to ask the same question in different ways and to use the results for consistency checks. These consistency checks should not be presented as a challenge to the expert, but rather as a means to stimulate thought and to improve judgments. Fourth, it helps to have computer support for decompositions, reaggregation, consistency checks, and displays.
l 1
41-1
9
[
3.3.4.1 Magnitude Judgments about Discrete Events ne techniques described in this subsection involve two or any finite number of mutually exhaustive and exclusive events to which probabilities have to be assigned by making direct numerical magnitude judgments. Dese probabilities should add to one by virtue of the addition law of probability. For two events, one need clicit only l
one of the probabilities, but it is good practice to check on the other one as well. For multiple events (e.g.,10 or more), it is usually worthwhile to reconsider the event h
space, either by clustering events or by identifying the continuous quantity that corresponds to the events. Frequently, with a continuous quantity, it is easier to construct probabilities for many events, since one can exploit monotonicity, single peakedness, and other properties of the probability distribution.
Direct Probability. This is perhaps the simplest technique. De clicitator asks the expert, "What do you think the probabilitv is that this event occurs and why?" Often it is useful first to obtain a rank order of'the probabilities of the events considered.
In the case of two events, the first questiott may be which is more likely and why, followed by a judpnent of the magnitude of the probability for the more likely event, and finished by tie judgment of the probability of 'he less likely event. Assuming that the two events are mutually exclusive and collectively exhaustive, these two
. probability judgments would, of course, have to add to one.
i For more than two events, there are two variants of this procedure: one can either
. ask the expert to assign probabilities to each event separately without the constraint j
of adding to 1.0 or to oo so with that constraint. When time permits, it may be i
desirable to ask the questions without constraints and check the sum. This sum will often be larger than 1.0, since experts tend to overestimate probabilities, especiall,y when they are small. Adjustments.will then be necessary so that the reviscu sum is
~
1.0.
Direct Odds. Sometimes the probabilities of events are hard to judge abstractly, but easier to judge in comparison. In this case, the normative expert can ask the substantive expert to state the relative odds of one event in favor of the other for selected pairs of events. If there are two mutually exclusive and exhaustive events A and B, the expert would need only to state the odds O(A), in favor of A over B.
From O(A) the probability of A can be calculated as p(A) = O(A)/(1 + O(A)},
i from which the probability of B follows. Similarly, for n events, the expert needs to assign n-1 odds, and the resulting, probabilities can be calculated. However, as in the l
direct probability, procedure, it might be useful to elicit n or more odds, point out the inconsistencies, discuss them and resolve them.
3.3.4.2 Magnitude Judgments about Continuous Uncertain Quantitles ne uncertain variable in this category is a continuous numerical quantity. The technic ues described in this subsection also apply if the variable is dense and has interva: quali. The two magnitude judgment techniques are mirror images of each other. In the ctile technique the normative expert provides the substantive expert with a proba lity and asks for a magnitude of the uncertain quantity such that the.. _.. _
i 1
probability of the true value failing below it is equal to that probability. In the fired
. point technique, the normative expert provides the substantive expert with a set of fixed points of the uncertain quantity and asks for the probability corresponding to these fixed points or for intervals in between them.
3.3.4.3 Fractile Technique The fractile technique is the most widely used probability clicitation technique for continuous uncertain quantities. It is used to construct the cumulative density function of the uncertain quantity that describes the expert's current state of knowledge. A z fractile is t4at magnitude x of the uncertain quantity x such that there is a probability of z that the true magnitude falls below x: and a 1 z probability that it falls above it. The lower bound therefore should be the 0.0-fractile and the upper bound should be the 1.0 fractile. The cumulative density function simply plots the fractiles against the probabilities that the actual magnitude falls below it.
After carefully defining the uncertain quantity, the substantive expert is asked to state its upper and lower bounds. In other words, he or she should define two magnitudes such that there is absolute certainty that the true magnitude would fall in between these extremes. In practice, because a continous variable may have no obvious lower or upper bound, assessments may focus on the 0.01 and 0.99 and/or on the 0.05 and
.^ 0.95 fractiles as relative extremes. After the initial extremes are defined, it is often useful to ask probing questions. The substantive expert is asked to consider a hypothetical event in which the actual magnitude of the variable considered was found to lie outside the range of extremes. Can this event be explained? Clearly,le if any credible explanation exists, the extremes were not 0.0 and 1.0-fractiles. Credib explanations also provide a basis for estimating the probabilities of being outside the extremes. Such considerations canlead to tevisions of the initial extremes.
After having obtained the extremes, the normative ex >crt pically moves to the middle range of the uncertain quantity and attempts to icenti the magnitude of the uncertain quantity such that the substantive expert thinks the e ances are about 50 50 that the actual magnitude would fali above or below that value. This point is called the median or the 0.5 fractile of the cumulative density function. The answer should be probed, especially if it falls exactly in the middle of the range between the extremes (since this suggests arithmetic averaging) or if it is very close to one scale and(unit of measurement). extreme since this suggests poor definition Having obtained three points of the cumulative density function (the extremes and the 0.5 fractile), the remaining tasks are to elicit between two and four additional fractiles. If they have not been determined in setting extremes, it is often useful to elicit the 0.05 and the 0.95 or the 0.01 and 0.99 fractiles next. To obtain the 0.05-fractile, the normative expert asks the substantive expert to state that magnitude of the uncertain quantity such that the probability of the true magnitude falling below it is 0.05. Finally, the 0.25 and 0.75 fractiles are commonly assessed.
i Usually knowing the extremes and five fractiles is sufficient to sketch a cumulative density function. The normative expert should smooth a graph of this function and discuss its shape with the expert. In addition, it is very helpful to show the plot of the I
h 43-
corresponding probability density function, which shows the symmetry or asymmetries of the cumulative density function more clearly.
3.3.4.4 Interval Technique In the interval technique the normative expert preselects points of the uncertain quantity and asks the substantive expert to assign them probabilities. There are two versions of this method. In the open interval version, the substantive expert assigns probabilities that the actual magmtude falls into the open intervals below and above 3
cach selected point. In the closed interval version, the substantive expert states the probabilities that the true magnitude falls between the preselected points.
Both versions of the interval technique begin with extremes, preferably bounds or the 0.01 and 0.99 fractiles, just as in the fractile technique. In the open interval version, the normative expert then chooses three to seven points between and asks, for each point, what the probability is that the actual magmtude of the unce:tain quantity is above or below that pomt. Having obtained these probability judgments, the i
normative expert can then smooth a cumulative density function and proceed as with i
the fractile procedure, In the closed interval version, the normative expert again lays out three to seven points, possibly equally spaced, but this time asks the substantive expert to assign probabilities that the tnic magnitude falls in each of the intervals. The result can be plotted both as a cumulative density function or as a probability distribution. It is useful to begin by rank ordering the probabilities of the intervals before assigning actual probabilities.
Both versions can be used in consistency checks. In addition, the fractile method can be mixed with the fixed-point method. It is quite easy, for example, to infer fractile-type questions from interval clicitations and to construct interval type questions from fractile type results. For example, after constructing the 0.25,0.5,~and 0.75 fractile, the substantive exxrt should consider the intervals below the 0.25 fractile, between the 0.25 and the 1.5 fractile, between the 0.5 and 0.75 fractile, and above the 0.75 fractile to be equally likely.
3.3.4.5 Indifference Judgments Between Gambles with Discrete Events i
The techniques discussed in this subsection derive probabilities from comparisons among gambles with discrete events and (usually hypothetical) monetary outcomes.
Reference Gamble Technique. To illustrate the reference gamble technique, the ex >crt is asked to select one of two gambles. The first gamble involves the event "It wii rain tomorrow" with unknown probability. If it rains, the expert will receive a stated prizet if it does not, he will receive nothing. Alternatively, he can choose the gamble in which he receives the prize with known probability p or otherwise nothing with probability 1 p. If the expert bets on rain, the probability p is reduced until the expert is indifferent between the two gambles, if indifference occurs when the probability is pi, this probability is assigned to the likelihood of the event because the expert should be indifferent when there are equal chances of winning the prize with both gambles, i
1 544-w
,_me, I..
Certainty Equivalent Technique. The certainty equivalent technique is somewhat simpler in that it asks only for comparisons between one gamb.e and one sure amount rather than between two gambles. However, in order to use it, one must verify (hnique, consider again the gamble for $pected value m or assume) that the substantive is an ex10 if it rains vs. nothing if it does not.
l the tec De normative exoert asks the substantive expert to state a certain amount of money at which he woulo be indifferent between playing the gamble or taking less as a gift.
To facilitate thinking, about this question, the normative expert could begin by asking whether the substantive expert would prefer a certain amount of $1 over play'n the the r
gamble. If the substantive expert emphatically says that he would prefer to p this gamble, the normative expert could change the certain amount to, say, $9.
point the substantive expert may consider the certain amount to be much more attractive. De normative expert then continues to vary the certain amount until the substantive expert is indifferent between the choices. At this point, the certain amount is said to be the certainty equivalent of the gamble.
Assume, for example, that the certainty equivalent in this case is $7. Then, by the assumption of the expected value principle,
$7 = p(Rain)$10 + p(No Rain)$0 p(Rain) =.70.
or I
Similar schemes can be devised with multiple event gambles.
3.3.4.6 Indifference Judgments among Gambles with Continuous Uncertain Quantities This. report will not describe indifference techniques for continuous variables as they are direct extensions of the techniques for discrete events. The main idea in applying these techniques to continuous quantities is to discretize these variables usin g ranges l
l of' values and to apply the indifference techniques to the discretized events (Matheson and Winkier,1976).
3.3J Techniques for Quantifying Value Judgments Many expert judgments related to the performance of an HLW repository will include value judgments, especially in screening scenarios and selecting conceptual 1
models. It is always important to make these value judgments explicit and document them carefully. In some cases, it also may be important to quantify value judgments with multiattribute utility clicitation techniques (Keeney and Raiffa,1976; von Winterfeldt and Edwards,1986). These techniques range from simple rating techniques to sophisticated indifference techniques to multiattribute utility functions.
This section describes two techniques with different degrees of technical the sophistication that are applicable to the task of evaluating conceptual models:
and an indifference technique simple multiattribute ratmg technique (Edwards,197 er and Sarin,1979). These to elicit a measurable multiattribute value function techniques are fairly similar in the basic task structure, ut differ in the procedure of the clicitation.,
b, L
- l).
~
There are seven steps in an evaluation:
1.
Define the objectives for evaluation.
2.
Develop attributes and scales for' measuring the objectives.
3.
Estimate the performance of the alternatives with respect to each attribute.
4.
Develop singfe attribute value functions.
Develop weights for the attributes.
l S.
6.
Convert the performance estimates of step 3 into single attribute values using ste ) 4.
l 7.
Ca culate an overall value for the alternative, typically by a weighted average g
L using the weights in step 5.
l The simple multiattribute rating technique and the measurable multiattribute value function technique differ primarily in steps 4 and 5. In the rating technique, both single attribute value functions and weights are elicited using direct numerical ratir g jud gments. In the indifference technique, both elements are elicited using tradeoffs anc indifference judgments. Before detailing these techniques we will briefly discuss L
steps 1 to 3.
The objectives hierarchy provides a logical structure of the objectives for evaluating the alternatives (i.e., conceptual models). We discussed some principles for constructing an objectives hierarchy in Section 333 on decomposition techniques for value problems.
L Developing attributes and scales that measure the objectives in the objectives There are two types of attribute scales: natural and hierarchy is still an art.
I constructed. Natural attribute scales are numerical scales commonly used. For example, run time of a conceptual model may be defined in terms of seconds of CPU time. A constructed scale is needed when no natural scale is available or convenient.
An example is scientific acceptability of a conceptual model. In this case a scale can
~
be constructed that defines qualitatively (perhaps a paragraph or more) several distinct achievement levels. For example, the worst level could be defined as "a conceptual model that has virtually no scientific acceptability, only a few sup porters, and ven little published evidence supporting it." The best level could be defined as l
"a conceptual model that has very lugh scientific acceptability, many s,upporters of l
Similarly, mtermediate l
l high scientific status, and significant published support."
levels could be defined.
l The next ste > (step 3) estimates the performance or achievement of each alternative on each of tie attributes. This is a nonprobabilistic version of an expert clicitation.
I In the assessment of conceptual models, a group of experts may be convened who l
estimate attributes such as run time, scientific acceptability, cost, etc. If the i.
uncertainty about these estimates is significant and if it is important to quantify this l
uncertainty,3.3.3.plete probability distri
>utions should be elicited using the techniques com l
in Section With uncertainty, a multiattribute utility function, rather than a value function, will be necessary to compare alternatives.
3.33.1 Simple Multiattribute Rating Technique To construct single attribute value functions with this technique, the worst and the best levels of the attribute scale are identified and arbitrarily assigned a value of 0 L
- kd*
1 m
l i
9 and 100, respectively. For natural scales, several values between the worst and the best level are then selected and rated on the 0 to 100 scale. De resulting points are plotted, and a single attribute value curve is fitted. For constructed scales, each constructed level is rated on the 0 to 100 scale. The same process is followed for all attributes.
To obtain weights for the attributes, two hypothetical attematives are constructed, t
one representmg all the worst attribute sete levels, one representing all the best, s
The expert is then asked to imagine being stuck with the worst alternative. Which attribute would he or she like to change most from its worst to its best level? Which is second, etc.7 his ranks the value differences for attribute ranges between worst and best levels of the attributes.
Next, the attribute range that was ranked highest (i.e., which the expert would like to is assigned 100 importance points and an attribute range (not
. change the most) list) that is utterly unimportant is assigned O. All other attribute i
necessarily in the ranges are rated between, according to their relative importance, ne resulting raw range weights are normalized to add to one.
L L
3.3.5.2 Indifference Technique for Measurable Value Functions l
- To obtain single attribute value functions, an indifference technique called bisection
.is used. The expert is again presented with the worst and the best levels of an attribute. Next, he or she is asked to identify a mid level of the attribute (not (s
necessarily the numerical mid point) such that the increase in value obtained by k
stepping from the worst 16 vel to the mid level is ec ual to the increase in the value obtamed by stepping from the mid level to the best evel. This mid level is the value l
midpoint. By arbitrarily assigning a value of 0 to the worst level and a value of 100 to the best level, the value mic >omt has a calculated value of 50. By further bisecting the range between the worst : evel and the value midpoint, the value midpoint and the y
best level, etc., a value function can be defined to any reasonably achievable detail.
For attributes with natural scales, the results can be plotted as a value function. His process is repeated for all attributes.
To clicit the weights, the expert is presented with two hypothetical alternatives that vary only on two attributes, while all other attributes are held constant at some level.
The first alternative has the worst level of attribute A and the best of attribute B.
The second alternative has the best level of attribute A and the worst of attribute B.
The expert is asked to state a preference for one of the alternatives. If the preference is for the first alternative, he or she is asked to worsen the level of attribute B in the first alternative until both alternatives are indifferent. If the preference is for the second alternative, the expert worsens the level of attribute A in the second alternative until both alternatives are indifferent. In either case, the clicitator assists the expert by providing easy comparisons along the way to indifference.
Once the indifference is established, the relative weights for attribute A vs. attribute B can be calculated assuming an additive value mocel. I.et (a,b',c,d,...) be the first o
alternative with the worst level of attribute A and the best level of attribute B, and let (a',bo,c,d,...) be the second alternative with the best level of attribute A and the worst of attribute B. Both have identical levels e, d, etc., of attributes C, D, etc. If the first.
i l
______._..__..______._.___.____,.._._______.a_.
j t
L pg
~ ;
1-alternative is preferred, then attribute B should be worsened to, say, level b' to achieve indifference. The indifference means that the overall values, denoted by v, of
-l 1
the alternatives are now equal so v(a b,c,d,. ) = v(a',bo,c,d,...).
o Using the additivity assumption, we can write W VA(a ) + W VB(D ) + WCVC(c) + wovp(d) +... =
A o
B WA A(a') + wB B(Do) + WCVC(C) + WDVD(d) +
V V
p 1
L and sir,ce, by definition,
\\
VA(a ) = VB(bo) = 0 and VA(a*) = vB(b') = 100,
l o
f wA/WB
- VB(b )n00.
Obtaining n-1 such equatior.s and using the convention that ;ne weights shouid add to i
one provides the solution for the weights in this procedure, i
'3JJ.3 Aggregation Steps
- Step 6 is identical for both techniques. It consists of a mechanical conversion of the performance measures obtained in step 3 into single attribute values using the results g
of either the rating or indifference technique. Step 7, also identical for both e
techniques aggregates single attribute values and weights to a weighted sum. Having l
completed aTull cycle usmg these techniques for making value judgments, it is good practice to compare the cal lated results with the experts' intui non and to iterate.
3.4 Combinine Ernert Judements When using a panel of experts, there are three basic reasons to combine the ludgments of individual experts. The first is to provide a base case, or more than one j
sase case, for analysis and sensitivity analysis m the performance assessment. The second is to gain msights from the analysis for decision making. The third is to simplify analyses and, t ierefore, to save time and effort in acquiring these insights.
i De xnding on the types of judgments, combining expert judgment takes somewhat different forms. In the qualitative expert judgment tasks (identification and j
, the combination consists of generatmg a joint list of things such as initial screening)d processes or screened scenarios. In probability udgment, individual events an probabilities or probability distributions are combined. Ln value judgments, mdividual functions or weights are combined.
3.4.1 Combining Lists
'Ihe simplest approach to create a joint list is to take the union of the individual lists.
Often, the creation of the joint list involves some restructuring and some relabeling.
Such changes should be communicated to the ex >crts that created the individual lists, and care should be taken to assure that their incividual concerns are reflected in the L
wEl.
4 oint list. Beyond these suggestions, however, there is little technical advice about how to combine qualitative mformation.
3.4.2 Comblaing Probability Judgments A key issue in combining probability judgments concerns what should be combined.
The answer in almost all cases is that the overall probability judgments of the individual experts or expert teams should be combined. These overall jud gments are typically,a jomt probability distribution function over the set of technica variables.
Combinmg at this level recognizes that the fundamental unit in expert assessment is j
the state of knowledge of the expert. By combining across the complete re resentation of experts' knowledge, different experts can use different models, lo ic, data, and p'rocesses to develop and represent their overall judgment.
C bining experts judgments at component levels in the process (e.g., combining marginal probability distributions) would put severe restricuons on the assessments of the individual expert. Each of the experts would essentially have to go through the same reasoning processes and provide the same intermediate representations of knowledge. In addition, if experts are in disa greement on their judgments and if the judgments are combined at component levels, you can develop situations in which the overall judgments of each expert would lead to a preference of an alternative A to an alternative B, but where alternative B would be preferable using the combined
. Judgments (Raiffa,1968).
L 3.4.3 Combining Value Judgments As with probability judgments, the appropriate level of aggregation is at the level of overall utility functions, not at the level of single attribute utilities or value tradeoffs.
There are, however, additional problems.with aggregating utilities (Arrow,1951; i
Keeney and Raiffa,1976). These problems are a result of the difficulty of making impersonal comparisons of utility. As a practical solution to this comparability problem, Keeney and Raffia (1976) propose the concept of a supra decision maker that is to incorporate the value judgments of each individual decision maker. Using the supra decision maker model and making certain regularity assumptions, it is o
I reasonable to aggregate individual (overall) utilities as a weighted average.
With value judgments, a fair amount of agreement usually exists about the general nature of the smgle attribute utility functions (see Section 3.3.4). In particular, agreement is likely to be found about the direction and the monotonicity of the utility l
function. If the utility functions have very different shapes, the underlying attribute may not have been clearly defined. On the other hand, weights are very personal expressions of value judgments and value tradeoffs. It is impossible to speak of 1
"better" or " correct" weights. Experience has shown that in many controversial problems, the differences in value judgments appear es legitimate differences in l.
weights (Edwards and von Winterfeld t,1987).
3.4.4 Behavioral vs. Analytical Combination The two general approaches to combining expert !udgments are referred to as the behavioral approach and the analytical approach. With the behavioral approach, the experts on a panel are brought together to discuss P.rni combine their judgments. In this process, the thinking, logic, and information of the different experts are 49-
o I
exchanged. Thi: may brink about some reconciliation of differ among experts. The behavioral approach seems particularly useful when the experts have basic differences in fundamental assumptions upon which their judgments are based. In this situation, the interaction among experts promotes deep thinking about the problem that can lead to more thorough understanding and documentation. A q
possible serious disadvantage is that some ex xrts may be dominated or " forced" to suppress their ideas to maintain harmony on tie expert panel.
Analytical combination procedures are comprised of a logic and formulas consistent j
with that logic developed by the analysts (e.g., the normative experts) for combining individual judgments (Fischer,1981; Genest and Zidek,1986). The complete set of.
j analytical combinations of expert judgments that seem reasonable for consideration is the convex combination of the mdividual expert judgments. In other words, it is the set of additive weightings of the various expert's judgments such that the sum of the weights is one. ' One of these combinations is the average of the various experts' Jud ments. Other combinations, in which the weight on one expert is one and wei hts on all the others are zero, are simply an expression of the state of knowledge of e individual rated one. The obvious advantages to analytical combination procedures is that they are easy to use, it is easy to do extensive sensitivity analyses around any base case combinat on, and individual experts have no influence on the i
judgments of other experts after the clicitation.
The most common analytical combination procedure is the average, in which all experts receive an equal weighting;. A substantial amount of evidence suggests that L
this averare wei ;hting often proc uces a reasonable base case for analysis (Seaver, I
l 1978; von Winter'eldt and Edwards,1986). However, some experience suggests that differential weighting techniques to account for the relative expertise of individual experts result in a better combined representation of knowledge (Ashton and Ashton, 1985. One useful property of weighting techniques that positively weighs all indiv)idual assessments is that the ful range of the variable under consideration is included in the combined representation. In other words, the weighting does not eliminate the rang,e of diversity among different experts (Merkhofer,1987). This property of combinmgjudgments is of particular concern in risk analysis.
A combination of behavioral and analytical procedures can be used for combining individual experts' judgments. In this case, behavioral methods are first used. Here, the individuals exchange all their reasoning and data and assumptions upon which their judgments are based. If this process results in any changes of judgments by individual experts, the implications of these changes are included m updated representations of the individual expert's state of knowledge. If this process happens to lead to a commonly held representation of the state of knowledge, then that l
representation of each mdividua) should also be the representation for the group. If, l
L after behavioral aggregation approaches, there are still residual differences between the individual experts, these can be combined by an analytical procedure as outlined above.
Regardless of how expert judgments are combined, the resulting uses of the experts' judgments should recognize t vec important items. First, any report should include l
more than one possible combination. This should facilitate hard thinking about the implications of different combinations and inform readers that there is no absolutely l
l -
i' correct way to do the combination. Second, different procedures for combinations may provide different insights from the analysis. For instance, if the combination is chosen that takes the "most conservative" estimate on any variable, the result should
- be a theoretical bound on the "most conservative" possible overall judgment based L
l on the individual expert's judgments. If the analysis indicates, for instance, acceptable implications with these conservative (i.e., high) probabilities of failure.
then perhaps no further analysis is necessary. Third, in all situations, the reported results should not be only combinations of the individual judgments. It is essential that the individual expert s judgments are also thoroughly reported and documented j
as discussed in Section 3.5.1.
3J Communtentine Ernert.fudoments 3J.1 Documentation
)
i The reasons for documentin the use of expert judgment on technical problems are etives: (1) to improve decision making, (2) to enhance specified by the following communication,(3) to fac te peer review and appraisal,(4) to recogmze and avoid l
biases in expert judgments, (5) to indicate unambiguously the current state of
)
knowledge about important technical and scientific matters, and (6) to provide a
]
i l
. basis for updating that knowledge.
- Complete documentation of the use of expert judgment would include both the
' interaction with the experts and the results (i.e., expert judgments) of that
' interaction. Thus, documentation would describe the selection of experts, the decision on whether to have expert teams, and whether to have panels of specialists.
r.
i Documentation would include the selection of the specific issues to be addressed by i
the specialists and how these were chosen. It would include the normative training 1
about the methods used to elicit' expertjudgments from the specialists and the preparation process to rovide any necessary or requested substantive information to the specialists. Finall, documentation would certainly include the results (e.g.,
probability distributio s) from any elicitation of expert judgment, as well as the reasoning to suppon them.
The fundamental unit of information of explicit expert judgments is the information provided by each expert. Hence, in any documentation, it is crucial to clearly distinguish between the information provided directly by each expert and any processing of that information, such as smoothing, interpolation, extrapolation, combining of the judgments of different specialists, or drawing of inferences from the judgments of experts. Maintaining, as pait of the documentation, the individual expert judgments, potentially provic es more information for decision making than if the information were aggregated (Clemen,1987).
7 The documentation of an individual's expert judgments should indicate what was done, why it was done, how it was done, who the individuals involved were and what their roles were, what the resulting judgments were, and what reasoning was used to support these jud ;ments. The documentation should begin with a clear definition of t
the specific issue >eing addressed and should contain unambiguous definitions of all the specific terms used in the clicitation. All assumptions about conditions that prevailed or would prevail that relate to the expert judgment should be stated. For instance, if one is assessing judgments about ground water travel times, assumptions
- S 1-
h
.f
-., e j
,l 2
about the particular rock types, the, amount of fracturing in the rock, and the tortuousity of the rock might be assumed by a given expert. If so, these assumptions should be rotated. De judgments as they are stated by the expert should be provided in the documentation. To support these judgments, the logic and data on which they l
are based should be completely specified. Any calculations that the expert l
i considered important in determmmg his judgments or models used should be i
indicated. All hterature,whether public or restricted, should be specified.
It is also important to document the approach by which the expert judgments were i
elicited. Some of this documentation may appear as a general section ahead of many 1
clicitations since the procedure used for many expert assessments would be similar.
However, the documentation would include both a description of the procedures and an ex91anation of why they were used, as well as examples of their use. In some specific problems, it is important to document what was not done. If sorne professionals are likely to question the process because of what was not explicitly done, clarification about why this was so may contribute to many objectives of documentation stated above.
ne documentation should also indicate the types of consistency checks performed in the assessment of an individual's expert judgments. Invariably with complex expert assessments, such inconsistencies occur and are identified by these ccasistency checks. Dat is, in fact, one reason for going through a careful process to elicit expert l
judgments. Identification of the inconsistencies allows experts to understand tielt source and to adjust appropriately their judgments to account for this increased i
I J
understanding. He final, consistent set of expert judgments are those utilized in'the performance assessment and this set requires the documentation just described.
When a panel of experts is used for a problem' additional documentation is
_l necessary. It is important to document how individual expert judgments are l
combined. The discussion in Section 3.4 indicates many guidelines for selecting a 1
i combination procedure. It is important to document the mdividual expert judgments in a common format and in tne same format as the combination of expert judgments, i
The documentation should clearly indicate agreements and disagreements among the experts and the reasoning for any disagree.ments.
l Documentation can take significant time and effort. Hence,it is very important to t
l begin with a system for documentation and a standard form to be used in documenting all experts. Because the specific issues addressed by different experts may vary, this form must be general enough to handle a wide range of specific 1
proslems. The responsibility falls upon a normative expert to document the results l
of any clicitation of expert judgment and upon the generalists and specialists to l
document the technical and scientific reasoning that led to those results. However, once the documentation of an individual specialist's judgments is completed, it is l
I important that the specialist revicw, making any necessary adjustments and then approving it as accurate.
Many factors need to be considered when selecting a documanention approach. Part l
of the documentation can include audio tapi,ng or video taping the clicitation sessions. With either, it is essential to provide written documentation in addition. In situations where there are many separate individual clicitations, it would probably be better to have the documentation of some clicitations more complete and polished
)
1 2-5 m
than others. For example, with 100 clicitation sessions, each involving a specialist, a
(
generalist, and a normative expert, it might be appropriate to have five of them carefully documented with a quality of wnting appropnate for publication in peer-reviewed technical journals. The other expert clicitations should be documented with the same quality of logic, but not necessarily with the same thoroughness and style in writing appropriate Tor journal publication. This would save a great deal of time in documentation, and yet provide the essential information for achieving the
. objectives of documentation stated above.
The final issue about documentation concerns whether the experts should be anonymously treated or whether their names should be clearly assigned to their expert judgments. De main arg"ument to maintain anonymity is that some experts j
might feel a pressure to take the party line" of their organization if their name were associated with their judgments, With anonymity, they presumably could state what they really think. On the other hand, with the names of experts clearly stated along with their judgments,'there is an additional motivation for the expert to be clear and thorough and consistent. Naming experts greatly enhances the perceived quality of the analysis and the ability of others to appraise and utilize the expert judgments.
Indeed, ex3erts typically possess a strong sense of responsibility for their jucgments and a con'idence about them. In other words, experts are willing to stand behind their judgments and have these represented as such (Shanteau,1987). In the recent clicitation of expert judgments from approximately 50 experts in numerous disciplines for the NUREG 1150 project on the safety of nuclear power plants, only e one indicated that he would prefer not to have his name attached to his judgments.
- Because of the importance to the overall study of attaching the experts names to their judgments, one criterion in selecting experts should be the willmgness to have his or her name associated with the judgments.
3.5.2 Presentation of Results
~
L The presentation of results of expert clicitations discusses and appraises the insights from the expert judgments and their implications for decision making. The l
objectives of this presentation are to inform decision makers and others about these i
implications and to have a constructive influence on decision making. The l
presentation of results of expert clicitations is distinct from the documentation of the l
clicitations. Documentation simply states the results of the expert clicitations, but presentation uses the judgments of the analysts to appraise the relevance of the expert judgments to the decision faced.
It is important to recognize that the presentation of results is itself a decision problem for which there are many alternatives (Keeney and von Winterfeldt,1986).
How deep the >resentation is, whether illustrative examples are used to indicate j
insights, and w1 ether the insights are expressed mainly m qualitative or also in quantitative fashion are alternatives for that decision problem. Dese alternatives involve factors such as how and how much to use cumulative distribution functions or probability density functions (Ibrekk and Morgan,1987), tables, diagrams, and decomposed probability trees. Alternatives also concern the degree to which there is comparability among the assessments of different experts. The presentation section may also contain decision analysis about the value of obtaining additional l
information regarding various uncertain phenomena investigated using expert.
.~ -.--.
Q i='y MndQ Judgment Key considerations in deciding on a presentation alternative include for whom and for what specific decision making purposes the presentation is prepared.
For an HLW repository, the performance assessment provides insights for technical and licensing decisions and for communication to government officials and the public. Presentation of the results of the expert judgments should indicate how these judgments relate to whether the repository can be safely operated and meet legal standards. The presentation should indicate clearly which of these judgments are crucial to decisions on whether the repository can perform safely and legally. It should also indicate what changes in these judgments might lead to cifferent implications and the bases that could lead to those changes in judgments. The presentation of results should clearly indicate which disagreements bereen experts are relevant to whether the repository can be safely and legally o >erated, and which are important. Particularly for those that are important, it wou d be significant to indicate how one might resolve the disagreements among experts. This resolution
- might be possible simply with additional interaction among the experts, with additional experts, or only through additional gathering of data and scientific experiments.
3.6 Internretation. Use. and Misuse of Ernert Judements
. However,gments are crucial in the '>erformance assessment of an HLW repo Expert jud as is the case with al scientific work, expert judgments can be misinterpreted, misrepresented, and misused. To enhance the likelihood that this does not occur, it is, important to interpret and use expert judgment in performance assessment appropnately.
The formal use of expert judgment in performance assessment is a complement, r
rather than a substitute, for other source's of scientific and technical information, such as data collection and experimentation. Expert judgments should not be considered equivalent to technical calculations based on universally accepted scientific laws or to the availability of extensive data on precisely the quantities of interest. Expert judgments are perhaps most useful when they are made explicit for stoblems in which site data are lacking, since they express both what the experts tnow and do not know.
Expert judgments are a snapshot of the state of knowledge of the individual expert about the stated item of interest. As new data, calculations, or scientific understanding become available, these should be systematically incorporated within the existin g state of knowledge. This learning process, which is a natural part of science anc knowledge, will result in changes in the expert's judgments.
Since different experts may have different information or different interpretations of 1
information, there is no logical reason why various experts should have the same state of knowledge. For new and complex problems, a diversity of opinions might be expected. If such differences exist, these would clearly be identified in expert assessments. For a problem as important as the design and construction of an HLW j
repository, it is useful to know the range of expert interpretations.
Numerous expensive and lengthy projects have been suggested to investigate the physical conditions at a potential HLW repository site and the phenomena that affect
- S4-
M m q q.
n
.ii
. those conditions.' With the explicit use of expert judgment, the value of the information derived from such projects can be calculated. His provides a sound basis for selecting projects that should be pursued. When one recognizes that the combined cost of proposed projects is several billion dollars, the significance of i
systematically appraismg proposed projects becomes obvious.
ne main misuses of explicit, expert judgments stem from misrepresentation or over-l reliance on them. Expert judgments often have significant uncertainties, and it is critical to include these in the documentation. For example, just reporting, an average without a range or a, probability distribution for a quantity of interest gives the illusion of too much precision and objectivity. Expert judgments are sometimes inappropriately used to avoid gathering additional management or scientific j
information. These judgments should complement information that should be gathered, not substitute for it. Sometimes decision makers with a predisposed desire whose views support or justify their position.given design alternative s to prove the HLW site is safe or to select a This is clearly a misuse of expert judgments. However, it is worth noting that with formal expert judgments, it is easier to identify weaknesses in the reasoning behind a decision.
In conclusion, it is worthwhile to remark on circumstances that should be considered
. successes or failures resulting from expert assessments. Science and knowledge are
, constantly changing. Thus, it is natural that as the knowledge of an individual t changes, his or her expert judgments will likely change. De representation of expert judgments as probabilities and utilities facilitates adjustments to account for new mformation. Even after the completion of a given assessment, an expert may recognize that he failed to account for some important information. De assessment process is designed to enhance the likelihood that such omissions are recognized.
Den it is easy to update the overall expert judgment to account for the omission.
The ability to change and the need to change expert assessments are not failures of the experts, the~ assessments, or the assessment process. Rather, they are natural and desired features to deal with the reality of science and knowledge for a complex problem such as an HLW repository.
After the explication of expert judgment, someone or some organization may wish to demonstrate that some of the assessments are not correct. For example, if some organization felt that the groundwater flow parameters near the repository site were incorrect, they might begin additional experimentation or search for additional information that would support their point. If this led to a process that eventually improved the overall state of knowledge, that would not be a failure of the assessment process. Rather, it would be one of the desired products of explicitly eliciting expert judgments. Because the overall intent of the expert judgment assessments and of performance assessment is a safe and legally operated repository.
The formal use of expert judgment in the performance assessment of an HLW repository contributes to understanding, learning, communicating, and decision making. In the final appraisal, the significance of the explicit use of expert judgment Thoulc be evaluated by the overall value it adds to the performance assessment.
Naturally, this is the same criterion applied to any of the mputs for or aspects of a performance assessment.
55-
4.
SUGGESTIONS FOR THE USE OF EXPERT JUDGMENT IN HLW DISPOSAL l
This chapter specifies how to apply the techniques for eliciting and using expert judgment discussed in Chapter 3 to the five problem areas of HLW disposal outimed c
m Chapter 2: scenario development and screening, model development, parameter estimation, information gathering, and strategic repository decisions. Some of the techniques apply to each of the five areas, and others are relevant only to single areas. For each of the five areas, experts must be selected and trained for the elicitation process, an appropriate clicitation process must be designed, and results must be thoroughly documented and presented.
identification and screening techniques are For scenario development and screening,f scenarios for which probabilities are then directly applicable to produce the set o e
arelessed.
i For model develoament, the identification and screening techniques are initially most relevant to se ect the variables to use in the conceptual models. Techniques for quantifying values may also be relevant to evaluate alternative models. Then l
mathematical models are developed to quantify the conceptual models. In this process, information gathering techniques are utilized as well as parameter estimation, both of which are addressed in the descriptions of the two problem areas j
' that foIIow.
The main techniques in parameter estimation are screening to select the key parameters and quantification of the uncertainties in the form of probability l
I distributions for those parameters.
1 Information gathering provides better information for the other areas of scenario l
and model development and parameter estimation. Information gathering uses l
techniques for identifying and screening information gathering strategies and for l
quantifying probabilities and values.
Strategic repository decision making can use all the techniques described in Section
- 3. First there is the task of generating alternatives for the construction and operation of the repository, which can use identification and screening techniques. Decision and event trees are next used to decompose the alternatives and events in a logical L
sequence. Objectives hierarchies are used to decompose the objectives that are relevant to evaluate the outcomes of decision and event sequences. Probability quantification techniques are used to assign probabilities to events in the decision tree, and utility quantification techniques are used to assign utilities to outcomes.
Then decision analysis can be used to develop insights for decision making.
4.1 Scenario Develonment and Screeninn t
SNLA's methodology for development and screening of scenarios that hypothesize the possible future states of the disposal system was desenbcd in Section 2.1. The
(
methodology consists of the following: (1) identification and classification of events screening of events and processes, (3) formulation of scenarios, ocesses, (2)f scenarios. In addition, we discussed earlier the need to estimate I.
and screening o and e
the li elihood of occurrence of each scenario to demonstrate compliance with the y
I f
56-1.
l'.1..
BRMi containment requirement in the EPA Standard (40 CFP. Part 191.13)3 to each of
. Below we present guidelines for the applying techniques described in Chapter these areas.
4.1.1 Identifiestion and Class 10 cation of Events and Processes
'(
The main objective of these tasks is to arrive at a comprehensive list of events and processes from which the scenarios are formulated. A secondary objective is to classify the events and processes to increase the likelihood that the hst is indeed comprehensive. This classification should also be useful for organizational purposes.
The group of experts that prepares the list of events and processes needs to be interdisciplinary. The experts should be specialists that have substantive knowledge in at least the following disciplines: general cology, seismicity, volcanology, h rology, and mining and/or rock tectonics, resource exploration, climatology,b havior (e.g., human mtrusion) can mechanics. In addition, since future human strongly influence, and indeed create, future scenarios, the experts should also include historians, sociologists, and psychologists knowledgeable about issues of
('
technological change. It should be noted that these s >ecialists should not be required l
, to have in-depth knowledge of nuclear waste disposal issues; the s xcialists should be 1 complemented by generalists (i.e., experts with general knowiec ge in performance i assessment). Generalists show the specialists how their judgments contribute to the performance assessment.
(
Section'3.2.3). The The experts should be sensitized to biases, primarily availability (f the experts to re l
bias of availability in.this context :cfers to a possible tendency o L
too heavily on existing records that do not necessarily represent the future adequately. The experts may not allow fo'r ad,ustments to the existing information and may need some training from the generalists on performance assessment and how their judgments wil' be used.
The particular clicitation techniques applied in the identification and classification of events and processes were described primarily in Section 3.3.1: forward and backward induction, value driven identification, and analogy / antimony driven identification. We believe that more than one clicitation technique should be used to enhance the likelihood that the sets of events and processes are comprehensive.
l The approach should be documented so that interested individuals may clearly discern the rationale of the clicitation process and the results. Intermediate lists as well as the final list of events and processes should be presented and should also include the steps to go from one list to another if multipic lists preceded the final one. An additional advantage of distributing the sets of events and processes is that any omitted examples may be identified and then, of course, added to the list.
l 4.1.2 Screening of Events and Processes The basic problem is to screen out insignificant events and processes from the list L
generated in the previous step. While the list of events and processes should be generated generically as well as specifically for each site, the screening out of events and processes by necessity must be site specific. To screen out events and processes, l
)
screening criteria must first be formulated and applied to arrive at a " final" list of 1
l 1 ~
l'
N events and processes to be used in formulating scenarios. De importance of both steps cannot be overemphasized. If the screening criteria are developed poorly, then the likelihood increases of eliminating potentiali significant events and processes and/or of including insignificant ones. If the cri cria themselves are not applied correctly, the same consequences are possible. In either case, the purpose of l
l-screening is defeated.
The s ecialists selected for identifying events and processes can also be used for identil i,ng screening criteria ney should be trained specifically to overcome biases such
, overconfidence" and " availability" (Section 3.23).
De clicitation techniques for screening events and processes are discussed in Section 33.2. The first part of the clicitation exercise should concentrate on developing the screening criteria based on physical reasonableness, potential consequences, and likelihood of occurrence, ne second aspect of the clicitation exercise should focus on setting reasonat>1e constraints for the screening enteria. For example, in dealing with the likelihood ef occurrence of a given initiating; event or process, what probabilit/ of occurrence is too low? The last part of tie exercise should be the I
r application of the screening criteria. Multiattribute utility analysis (Section 33.5) is an approach for explicitly making tradeoffs between the different criteria. It is important to point out that iterating through the target levels at:d constraints in the criteria is recommended as a mechanism for determining the impact that these may
" have on the finallist of events and processes.
The documentation and presentation of results mainly explains clearly the log,ic of the approach used in sufficiently general terms that it can'be followed and critically reviewed by a wide range of interested parties. The documentation should allow not only critique of the approach, but of the results as well. The result should be a final list of events and processes that will be combined to form scenarios.
4.1.3 Generation of Scenarios Once unimportant events and processes have been eliminated from further consideration, the surviving ones are combined to form scenarios. This step can be conducted by generalists knowledgeable about the application of event trees. The forward and backward induction techniques described in Section 33.1 and techniques for combining may be useful.
4.1.4 Screening of Scenarios De guidelines for using expert judgment in this step are identical to those described l
in Section 4.1.2 for the screening o events and processes. He problem is to reduce l
the number of scenarios for the performance assessment to a tractable and representative set. This is accomplished by aggregatin scenarios and by develo g
and applying screening criteria as in the screenmg o events and processes.
e screening criteria should again stress p sical reasonableness, potential consequences, and likelihood OL occurrence.
e selection and training of experts, the clicitation techniques, and the documentation and presentation of results should be identical to that in Section 4.1.2.
I
- 58-l-
)
t 1
4.1J Probability of Scenarios De problem to be addressed by the experts in this step is estimatin 1
and combining these probabilities to arrive at the probability of the scenario. To estimate the proba >ility of the individual events and processes, the experts need to identify the initiating event or process and decide whether the occurrence of the other events and processes in the scenario are conditional on the occurrence of the initiating one.
Dis step requires a multidisciplinary team of specialists with substantive knowledge scismicity, tectonics, volcanology, climatology, hydrology, rock in general geology,ing, etc. Generalists with knowledge of performance assessment mechanics and mm can provide insights on what type of scenarios are fakely to be more significant.
Finally, normative experts with experience in probability clicitation are needed to train the other groups of experts as well as to serve as the clicitators.
The specialists should be trained in overcoming probability biases (mainly and availability), decomposing, expressing Ludgments L
overconfidence, anchoring,ing, and assessing conditional probabilities %e specific I
explicitly, probability encod i
. elicitation techniques applicable to this step are the probability c.uantification
- techniques described in Section 3.3.4. The techniques for estimating tae probability
? of discrete events such as the direct probability technique or the direct odds technique may be particularly useful.
4.2 Model Development De development of models for performance assessment includes the development of conceptual models, mathematical models, and associated computer codes. This effort involves the selection and interpretation of available data and other sources of information, the formulation of relevant assumptions, and confidence building in the models and codes developed. Each requires expert judgment.
4.2.1 Data Selection and Interpretation This task mainly provides the basis for the formulation of conceptual model(s) of the disposal system. Experts select and interpret data and other information that will lead to the establishment of the system's geometry; boundary and initial conditions; and past, p(resent, and future events and processes that ma the system Section 2.2.1).
It is expected that specialists, generalists, and normative experts will be required to i
carry out this task. Specialists primarily should concentrate in the fields of geology and hydrologyt however, some specialists involved in the identification and I~
classification of events and processes in the scenario development (Section 4.1.1)
L should also be used here. Generalists who have participated in earlier or preliminary l'
performance assessments of HLW disposal sites should be used in this task.
Generalists should be able to provide insights regarding the relative importance of different types of data and information based on their past experiences. Normative e,xperts should assist the specialists in searching and catalogmg different sour talormation.
l
.59-n
The clicitation exercise is likely to be in three phases. In the first phase, the specialists and generalists identify both site-specific and generic sources of data and other information. For this phase,'the experts should be trained to overcome e
" availability" bias (Section 3.2.3). The specific clicitation techniques relevant to the j
identification task are presented in Section 3.3.1.
In the second phase of the elicitation, the experts must screen out unimportant 1
l sources of information and select the most relevant ones. To achieve this goal, criteria must be developed to accomplish the screening step, and then these criteria need to be applied to arrive at the most relatively important sources of data and informaticn. 'This phase of the clicitation is similar to that discussed in Sections 4.1.2 -
Screenmg of Events and Processes, and Screening of Scenarios). The j
and 4.1.4 (d clicitation techniques are similar to those suggested in Section 4.1.2 and training an are presented in Sections 3.2.3 and 3.3.2.
The third phase involves the interpretation of the selected information. In thi,s
)
l
'>hase, the experts make inferences based on this information that will form the basis Bor the development of models. The ex >erts should be tramed to overcome biases ignoring sase rates, and nonregressive predictions associated with availability, fers here to the tendency to follow a conventional lme (Section 3.2.3). Availability re of reasoning when interpreting the available information without considermg
- evidence that may challenge this convention. Ignoring ba,se rates as applied to data Linterpietation refers to ignoring soft or abstract information while focusmg only on l
concrete evidence and data. Fonregressive prediction is the tendency to make inferences using; relationships the, app icability and validity of which have not been j
established for tie system in question.
f 4.2.2 Development of Conceptual Models -
~l i
1 Constructing conceptual models uses inferences based on the selection and interpretation of data to formulate assumptions for the behavior of the disposal J
system. These assumptions, in turn, are the cornerstone for the assembly of J
mathematical models and their computer codes used in the quantitative analyses.
Modeling most likely will result in a multitude of alternative conceptual models because of the lack of data during the early stages of a site investigation. As more information becomes available, it could be possible to distinguish among the it would be different conceptual models and possibly reduce their number. Fmally,fy a relative to quanti feasibic, if a number of conceptual models survive screening, describes the "true" likelihood for each conceptual model that it adequately groundwater flow and transport processes, for instance.
Again, specialists, generalists, and normative experts will probably be needed. The specialists should be in the area of hydrology and should include both modelers and experimentalists as will be discussed below. Generalists should be used to assure that the specialists render judgments within the context of performance assessment.
Normative experts should be used to assist the specialists in making value judgments.
Some of the experts used in data selection and interpretation should be mvolved in this task to provide continuity. Multiple teams of experts may be appropriate.
The first phase of the clicitation is the development of meaningful criteria for the l
l formulation of assumptions and the construction of conceptual models. These o
,. - -,. + -. - -.. -... -. - - -, -,.. -....,.. -, - - -.. - - - - - _. -..
,--.--,-.-.-....-.~..~,-,,..--..,..---..--__.,.,~,n.-----
.e crateria include beliefs regarding the importance of model attributes such as the ability to simulate specific events and processes, groundwater flow geometry, levant parameters,* complexity, etc. De selection of these criteria is likely regime, re to be based on value judgments and will require all three types of experts. While the specialists should be expected to play the biggest role in this phase, generalists should provide the basis for acceptable tradeoffs tsat can be mace in light of regulations that need to be addressed in the performance assessment. Normative experts are likely to be clicitators. Techniques for expressing value judgments are described in Section 3.3.5.
ne second phase is to develop a procedure for distinguishing among the alternative I
conceptual models and,if possible, screening some out. His should be accomplished by attem > ting to identify the salient features of each conceptual model, formulating and conc uctmg specific analyses and experiments that could test the validity and/or importance of these features, and setting screening criteria and applying them. In j
this phase, both specialists in model development and experimental studies are needed because a synthesis of analyses and experiments will likely be necessary.
Screening techniques described in Section 3.3.2 should be useful in this phase.
l The third phase consists of an attempt to quantify the likelihood that each concep! cal f
i model that survives screening is the best of the available models. Specialists and
)
- normative experts will be needed in this phase, and probability clicitation tools such s as sequential conditional probability assessment and others presented in Section 3.3.4 priate training to overcome such biases as are applicable to this phase. Ap overconfidence, anchoring, availabil, and ignoring base rates, discussed in Section 3.2.3, should be conducted before the licitation.
A portfolio of conceptual models should be chosen that, at the very least, represents extreme sets of conditions for a performance assessment and that, at the same time, can be tested during site-characterization investigations. Situations in which two or L
more conceptual models are very similar should be avoided. Refinement of the final portfolio of conceptual models can be done using decision analysis and, in particular, i
preposterior analysis (Winkler,1972). These techniques increase the likelihood that the set of conceptual models selected is adequate for conducting a performance i
assessment, the results of which will allow making regulatory decisions with j
l confidence.
4.2.3 Confidence Building Following the development of conceptual models, mathematical models will be formulated that cast the models in terms of mathematical equations (i.e., al,gebraic, partial, and/or integral equations. In setting up these equations, assumptions are made, the validity of which needs to be established. Typically, because of the com >lexity of the equations in even the simplest models to simulate the behavior of an ELW disposal system, the solution to these eguations is implemented in computer codes. Dependin g on the nature of the equations (linear vs. nonlinear, partial vs.
anc the coupling between two or more equations, these can be solved algebraic, etc.)cally or numerically. In any case, the implementation of neither either analyti analytical solutions nor numerical solutions is exact. For example, if an analytical solution involves an infinite series, this series needs to be truncated after a finite number of terms, or if it includes a complex integral, this integral is often evaluated O
3.
~
numerically Numerical solutions inherently are approximations to the "true" solution of the equation (s). In whatev'er form (either analytical or numerical), errors are introduced when solving the equation (s) in a mathematical model. Since the validity of these mathematical mcdels and computer codes cannot be established l
over the temporal and spatial scales of interest in HLW disposal (Section 2.2.3),
+
validation cannot be achieved in the truest sense. Nevertheless, confidence must be built to the extent that, given the present state of the art, these models and codes are deemed adequate for the job at hand: predicting the behavior of the disposal system j
over several kilometers and tens of thousands of years. To build confidence m the will play a major role in designing and conducting these activities, pe models and codes, limited-scope activities will be carried out, and ex as well as in interpreting the results.
Experts are likely to be used in selecting important features in the models to tic tested and the type of testing. For example, there may not be a need to test t!'e expression for radioactive decay in the radionuclide transport equation because this l
is a well-established and accepted expression. On the other hand, the use of a Fickian model for diffusion to represent dispersion or the use of a linear-sorption-i equilibrium based retardation factor are both models that are the subject of much e
criticism and should be tested. The question then becomes what tests to conduct, for
' example, laboratory vs. field tests. Experts will also be involved in the selection of l
appropriate criteria to establish the measures of goodness of the models. These are competing measures,' and experts should select those criteria that are mooi meaningful to the regulatory, requirements to be addresse4 The experts must also l
set t.he imits and constraints m these criteria. Exp'erts will also be needed to assess the ability of the models to extrapolate from the temporal and s patial scales at which they were tested to the scales of mterest in.HLW disposal. Fina:ly, there are likely to be some couplings in the'models that are so complex it is impractical to test their validity. In this case, expert judgment assesses the adequacy of the modeling of these coupimgs.
The expe,rts required include primarily specialists and generalists; however, it may be appropriate to include normative experts, but this may not be necessary. It is sugg,ested that multiple teams of experts be used, each team consisting of both specialists and generalists, and modelers and experimentalists.
5 regarding what aspects of models need The experts make value judgments (tradeoffs)3.5 should be useful. In addition, they to be tested, and the techmq,ues in Section 3.
develop criteria for establishing the validity of given models. Therefore, the techniques for setting criteria, limits, and constraints to the criteria and the applications of the criteria in Section 3.3.2 should be employed. As the "s ultimate" validation test at an HLW disposal site cannot be performed and because of the complexity of the model, perhaps one of the biggest tasks to be faced by the expens of the overall system model into requires the decomposition (Section 3.33)d that there are likely to be couplings meaningful pieces. While it has been recognize that cannot be tested, extreme care must be taken to assure that the decomposition of the problem does not eliminate significant couplings. For example, in testing for the validity of the linear-sorption-equilibrium model as the dominant radionuclide retardation, the problem should not decompose such that a test is conducted that does not include now-field effects because evidence exists that they have a significant impact on sorption.
'62-i i
1 i.-
- _. - -- -- ~... _
y '.. a.. '
h~~ a CT u
l^
4.3 Parammatar Estisaatiop 4.3.1 Identifiestion of Parameters
'1 i
f.
As stated in Section 2.3.1, parameters are embedded in conceptual models that predict the performance of'the repository in terms of rad;onuclide emissions and l.
their potential health effects. Derefore, the importance of parameters is closely related to the variation in the amount of radionuclide emissions relative to variations in the parameters. The main method for identifying and selecting parameters is l
L sensitivity analysis. In such analysis, parameters of conceptual models are systematically varied (both individually and in sets) to determine which parameter or I
combination of parameters has the strongest impact on radionuclide emissions.
i Sensitivity analysis is currently more a craft than a science. It is therefore especially important that the expert judgments that select and interpret the sensitivity analysis I
for parameter identification are made exphcitly.
l 4.3.1.1 Guidelines fbr Parameter Identification l
$ At this sta,ge of the analysis of the HLW disposal problem, the issues for parameter i
l identification are typically fairly clear cut: Given a chosen conceptual model, what are its parameters that should be quantified for further analysis. There may be two complications with this problem definition that may require resolution before I
identifying important pararacters. First, there may be several concerval models, and second, there may be different ways to categorize parameters. If these mmplications occur, it is useful to convene an expert panel to address these issues before the actual parameter identification process. Guidelin'es for issue identification and selection of j
experts for this part of the study should be followed (Sections 3.2.1 and 3.2.2). In i
particular, a diverse set of experts and examination of a diverse set of conceptual models and sets and subsets of parameters should be considered.
Once a conceptual model and the possible parameters and their subsets are agreed upon,identifymg "important" parameters is more technical and better defined.
Three types of experts are necessary identifying important parameters: Substantive 3
experts with knowledge of geology and hydrology, among others; generalists with i
expertise in the conceptual models; and experts in sensitivity analysis. An effort should be made to obtain the beu expertise in these areas, as well as to maintain j
some diversity of opinion. This diversity is especially important for the experts t
concerning the conceptual model, as they are likely to disagree a pn'ori about what constitutes important, parameters of the model. Less emphasis on diversity is needed in selecting experts m hydrology and geology, and even less in selecting experts in sensitivity analyses.
Training in clicitation techniques is not required in this area. However, both the substantive experts and the sens'tivity analysts r;eed to learn about the nature of the I
i conceptual model, its assumptions, its behavior and some of its preconceptions about sensitivities. For the substantive experts, this may provide guidance for reformulating parameters (e.g., by dividing hydraulic conductivity into separate strata). This type oT training alerts sensitivity analysts to possible interactions among parameters, as well M3-g r-w,,.,.-_...-w..
.,,_m-
,_w_
___,,...,_,....____.--.___.._,-,,._.______._________...-,_-._m,
rN7 n.~
as to possible roblems and opportunities in carIion and familiarization with theing o training shou d consist of two parts: presenta conceptual models and some of their predictions and extensive question and answer periods regarding the use of the conceptual models.
Because sensitivity analysis plays a key role in identifying important parameters and because the clicitation centers around a conceptual model, the clicitation session should be structured somewhat differentif from the standard session described in i
Section 3.2.4. In particular, display and discussion of sensitivity analysis results of running parts or the complete conceptual model should be emphasized.
Comparatively less time should be spent in mdividual elicitations, and the amount of actual numencal clicitation should be falf.y small at this stage.
There are two suggestions for structuring an clicitation session in this context, depending on whether sensitivity analyses can be done on line. If they can be done on line, it is highly desirable to structure the clicitations as an interactive exercise in which the experts formulate hypotheses about sensitivity and importance and test them in real time. Some structure should be provided to make sure that the more prominent hy% theses are tested and that all parameters are examined. Beyond that, the experts siould be able to develop their own plan for carrying out sensitivity analyses and judging their outcomes.
If sensitivity analysis cannot be done on line, the experts should convene at least twice. De first meeting determines which sensitivity analyses should be carried out.
The second meeting discusses the results of the sensitivity analyses and makes judgments about which parameters are important enough for further quantification of uncertainties. If certain parts of sensitivity analyses can be done on line, this should be done to liven up the exercise. However, care should be taken that the on-line sensitivity-analyses do not gain more prominence by making the respective
)
parameters more ailable to the experts (Section 3.2.3).
In both cases (on line vs. prepared sensitivity analyses) the experts should aim at making three judgments about t e parameter:
1.
Sensitivity related to selected performance measures; 2.
Overall importance;
)
3.
Need for further quantification or data collection.
J 4.3.2 Quantification of Parameters A fairly large amount of research and applied work exists for quantifying expert judgments about uncertainties in parameters with probability distributions. The recommendations that follow are therefore grounded in significant amounts of experience (Section 3.3.4).
4.3.2.1 Guidelines for Quantifying Parameters After the conceptual model and its important parameters are identified, the issue is to quantify the knowledge of substantive experts in hydrology and geology about the
- 64-
p a.-
Mb l
t I
parameters as probability distributions. Price to any assessments, is useful to identify current or near future data collection efforts, to put the actual ex>ert clicitation of uncertainties before this data collection into pers ective. In ac dition, it is very important that the parameters be unambiguous! de med.
Parameter quantification addresses specific issues such as the estimation of hydraulic conductivity parameters in specific strata of the reposito. Experts should be i
selected on a parameter by parameter basis. Depth of knowl e is crucial, breadth and diversity are secondary m this case. Motivational biases s uld be considered.
I For example, a hydrologist on record as stating that Yucca Mountain is an absolutely safe site for the repository might give estimates of hydraulic conductivity that are too i
Iow. It is useful to counterbalance such potential biases through expert selection.
Training should focus on constructing (usually continuous) probability density l
pdfs) or cumulative density functions l
functions (dations in Section 3.2.3 apply with full (cdfs) over parameters. Th; recommen should be familiar with the probabiity clicitations task, and they should get ample i
practice using many examples of the ty>es of clicitation that they are like y to face.
Anchoring and ad ustments, overconfidence, and motivational biases should be j
demonstrated, and icblasing procedures should be explained.
r All experts must agree on the precise definition of the parameter to be elicited. For I
example, when hydraulic conductivity is discussed, it must be absolutely clear which strata of the re pository is referred to, whether one wants to assess mean or maximum i
what maximum may mean, etc. It is useful to structure the hydraulic conductivity, involve a "generalist" knowledgeable about the conceptua l
clicitation session to l
model and the interpretation of the parameter within that model.
l t
A variety of decomposition techniques may be useful, depending on the specific
_I parameter or, the expert (see Section 3.3.3). If functional decompositions are i
utilized, direct probability assessments should be used as consistency checks for probabilities calculated based on decomposed assessments. For example, when assessin ydraulic conductivity in four different strata and subsequently assessing l
average
'draulic conductivity, the results can be checked for consistency with the average h raulic conductivity.
3 Parameters should usually be represented as continuous random variables.
l Therefore, our suggestions for applying elicitation techniques are very straightforward: use the fractile technique described in Section 3.3.4 and check it with tlic interval technique and >crhaps a few J; amble questions. Pay particular l
attention to the extremes and prose them careful y,le, when considering hydraulic j
possibly by considering physical impessibilities and extreme gambles. For examp conductivity, the clicitator may ask for the expected minimum and maximum areas of t
conductivity in the repository, for the minimum and maximum in comparable formations, and for the minimum and maximum in a variety of substances and materials. An appropriete range should then be selected. By broadening the notion l
of minima and maxima, the expert may be induced to consider the full range of l
possibilities for the case at hand as well.
Having obtained a first cut range, the normative expert should ask the specialist to I
explain a set of hypothetical data that indicates events outside the range. One may l
l 1
I 15 A m-j
)
also ask the expert, whether he or she.would be willing to bet a large sum of money i
that all possible experiments would lead to the conclusion that the parameter is in the range stated. Both techniques are useful for deblasing.
4.4lah==*6 caebeian j
To better design, construct, and operate a nuclear repository, numerous important j
decisions must be made, many of which will affect repository performance. To i
improve the quality of these decisions and to improve performance assessment, numerous efforts must be carried out to gather information. Collectively, this information will be very costly. In terms of dollars, the cost will be in the billions; in j
f terms of human resources, the cost will be in the thousands of person years of professional time; and in terms of the environmental and s i
l information gathering should be made carefully and thoughtfully Information scenario development, model gathering cuts across the three areas discussed earlier: developm intended to im > rove the quality of the scenarios, the usefulness of the models, and l
i the estimates o' the parameters.
I
'Information gathering is also different from the first three areas in that it concerns how deep, and where to
[
decisions. Some important decisions concerns how many,f decisions concerns what r
drill test holes into the repository media. Another class o computer codes should be developed and what conceptual two dimensions or in three dimensions, and what variables should they include?
i Regarding parameters in these models, how can we best estimate a variable such as porosity to reasonably balance the insight gained about the variable against the cost
~
1 i
and effort necessary to gain that insight?
The use of the concept of expected value of sample information (Raiffa and l
allows ap staisal of the various alternatives for gathering information Schlaifer,1%1)f the one that is best given expectations about what information might and selection o i
be obtained from the various alternatives and about the economic cost, time required, and damage caused by that alternative. Value judgments must balance the l
advantages and disadvantages of gathering the hformation and take into a: count the
~
overall goal of creating a safe, legal repository.
In the rest of this section, three special classes of problems conceming information gathering are discussed. These problems concern informational drilling, development of models, and conducting laboratory or field experiments other than drilling.
1 4.4.1 Informational Drilling The informational drilling program is one of the major activities in the characterir.ation of the repository. It should be carried out only after careful appraisal of the alternatives. To do this, there are several distinct activities that should be completed that rely partially on the use of expert judgment.
'.g.
Y V.
- 3#M 1
To characterize the informational drilling problem, the objectives of the drilling program and reasonable alternatives first need to be identified. Then the allematives should be screened to specify the competitive options. For each of these competitive options, estimates are necessary for the information that will possibly be learned and for the time, cost, and damage caused. Using value judgments to balance these and the concept of the exsected value of samp!c information, an i
analysis can indicate the relative desirabiity of the options under a wide range of i
assumptions. Each of these tasks are elaborated below.
]
ne first and driving task for the informational drilling program is to specify its j
objectives. It is important to be very explicit about the relative desirability of different information that might be learned from the pro ram. In this step, s ecialists need to be selected to assist in specifying the ob ctives because the ectives of the drilling p ram willlikely be technical, it is al important that the i
liy related to the fundamental objective of better dr ling objectives be lo designmg, constructing, an operating a safe and legal repository. This relationship i
i may be best specified by generalists with a broader understanding of the repository l
program. De techniques for structuring objectives hierarchies are useful in this task (Section 3.3.3), and careful documentation and review of the objectives hierarchy is appropriate before completing the additional tasks below.
' De second task is to identify a large number of reasonable alternatives for gathering informat>on via drilling. To develop these alternatives, specialists and generalists Should again be used. At this stage, the alternatives need not be carefully refined (e.g., the exact location of each hole), but they should be specific enough to l
i distinguish them from other alternatives.
The next task is to screen the large number of alternatives to identify those that are competitive. The relationship of the objectives of the drilling program to the l
fundamental objectives of the repository should be a basis for this screening. The I
screening criteria should at first be specified by generalists using techniques discussed in Section 3.3.2 and then be used to eliminate many noncompetitive alternatives. At l
a later stage in the analysis of information drilling options, when the relative desirability of alternatives that passed the screening are known, the screening crit:ria
[
should be reexamined to determine whether more related screening criteria might r
have yielded better alternatives. The way screening criteria can be verified with information that comes later in the analysis is outlined in Keeney (1980). If the appropriateness of the screening criteria is to be verified, the original use of expert Expert judgment is judgment to set the criteria for screening is not so significant.
crucial not only to screening but in setting up the relationships of objectives of the various < rilling options in the next task.plications of what might be le drilling ?rogram and in specifying im l
The fourth task is to define better the competitive options that make it through the screening. There are two aspects to this definition. The first is to specify exactly i
what drilling will occur, and the other is to predict the possible information learned from the drilling and its time, cost, and resulting damage. This task relies heavily on Some of it will be from specialists, specifically information i
expert judgment.
rejerring to details learned about the hydrolog and geology at the site. Other l
information will necessarily come from genera ists about the time and cost of the drilling options. For each of these circumstances, experts need to be carefully
- e. -
l e
selected and trained. The assessments should indicate the implications of the alternatives in terms of a probability distribution function as discussed in Section 3.3.4. An important subtask in the estimation of the impact of information is assessing the conditional probability distributions with the information that can be i
obtained from the alternative drilling activities and assessing the probability 1
distribution of the information from the drilling. In particular, the probability distribution for cumulative radionuclide releases and health effects wal strongly depend on the information obtained. Dese probability distributions are a major ingredient for carrying out a value of information analysis, j
De next task is to quantify the value judgments (Section 3.3.5) necessary to integrate all the objectives of the informational drilling program. Because of the uncertamties about what will be learned by the various drillmg options, a multiattribute utility function should be used to integrate these objectives (Keeney and Raiffs,1976).
j Expert jud_tment will be necessary to specify the value iudgments for the utility a
function. Tsese judgments are of a policy nature because they relate to the c uality of
)
and they siould be i
Information aval.able for key decisions regarding the repository,itory program and provided by individuals with policy positions in the repos stakeholders with a legitimate voice m that program. Examples of this in the i
repository program are discussed in Section 1.4. To assist the policy makers in quantifying their jud gments, it is important to have the assistance of a normative expert with substantia experience in quantifying value judgments.
With the tasks above completed, it remains to analyze the options and identify those that provide the most information for the time and effort. At this stage,it is critical to gam the insights about why the better options are better and ioout why they are that much better. Dis inter 9tetation is the link that provides useful information to the decision making process from the explicit use of expert judgment in the appraisal
_l l
ofinformational drillmg options.
i 4.4.2 Selecting Models to Develop
{
With any information$sthering, problem, the key is to specify the objectives to be achieved. In this case, t se objectives to be achieved by developmg models need to be carefully specified. Furthermore, these objectives need to be related to the s
fundamental objectives of designing, constructing, development are t be same as theand operating a repository. I regard, the fundamental ob ectives for model i
i fundamental objectives for iniormational drillin. What is different in this case is the means objectives by which those fundamental ectives are achieved. To specify the relationship between the means objectives an the fundamental objectives, expert t
judgments of both specialists and generalists are needed. Essentially,bute to these relationships answer the questions about how model development will contri better understanding and better decision making regarding the repository.
Mter the experts are selected, they need to be trained to distinguish between fundamental and means objectives and to understand conce pts such as influence diagrams and obectives hierarchies for relating them. Den the clicitation process needs to be carctily documented. Dis documentation can be reviewed by a large number of peers for completeness and reasonableness, and the revised results should provide a basis for the additional tasks in selecting appropriate models for development.
e 4
u De next task is to select general types of alternative models that may be worthwhile l
i to develop. Some of these may be analytical models, and others may be simulation models represented by codes. Other factors defining the alternatives concern the l
number of variables in the models and exactly which variables they should be. A l
combination of generalists and specialists should be appropriate for defining a large number of alternative models. Identification techniques for expert clicitation discussed in Section 3.3.1 will be used extensively in this task.
j De next task is to screen the alternatives to focus on those tilat seem most useful to arovide information for the repository. In this phase, the screening models outlined i
a Section 3.3.2 will be utilized, ne criteria for screening should be set using a l
l combination of judg,ments from saccialists and generalists. De exact screenmg l
criteria are not too important as tseir appropriateness should be verified after the models have Jone through various stages of development. In general,if the models j
selected for c evelopment are not providing the insig; hts expected, either because of i
lack of available data or field data indicates that they are inappropriate, then the models can be revised or new models selected for development.
De fourth task is essentially model development as discussed in Section 4.2 Details
- are found in that section, so only a brief overview is included here. De task is l
l
. essentially to specify the variables appropriate for each of the models selected for development and to identify data sources to provide information about those i
l variables. Also, using any available physical relationships, it is necessary to relate the variables to each otser to provide the structure for the model. At this. stage,it is essentially the judgments of specialists that are important. Normative experts should assist these experts in expressing their judgments about the relationships of the variables.
There are a nt$mber of input variables to a large model and one or more output i
variables of interest. Probasility distributions quantify the current state of knowledge about the input variables and are used in the model to derive implications for the output variables. How this is carried out,is described in Section 4.3. It relies heavily l
on the techniques for quantifying probability judgments discussed in Section 3.31 4.
The last task is to run the models many times and gain the insights available from them. A team of generalists and specialists will likely be most a apropriate to interpret the results of the analyses. Based on these insights, it wil probably be appropriate to repeat various runs of the model to g sin additional insights about the I
sensitivity of parameter values for different variables with respect to the model's l
implications. At this stage, the team of experts should also verify any assumstions l
made in selecting models to develop. Dese assumptions pertain to the numxt of 1
variables, the relationships between variables, and their quantification.
c 4.4.3 Laboratory and Field Experiments I
Many laboratory and field experiments, exclusive of informational drilling, will likely be done before final design and construction of the repository. De first task in each of these situations is to specify the objectives to be achieved by the experiment I
proposed. As with the problems discussed above, the task is to provide inf ormation that results in a legal and environmentally sound repository through better design N69-
i) and construction his task requires balancing of the impacts of the ex xrimentation I
in terms of cost and effort agamst the.value oT the information learned. For each of the proposed experiments, different objectives contribute to information obtained.
De kind of information expected needs to be specified using expert jud gments of generalists and specialists and the assistance of a normative expert to exp icate that Once these ob ectives are clarified, we have a basis for evaluating judgment.
different alternatives for the;laboratory and field experiments.
For any proposed experiment, the next task is to identify alternatives for conducting I
vary in the sophistication of testmg equi > ment used.pth or breadth. They also mayAt that experiment. These may vary in cost, time, or de i
generalists with some assistance of specialists should be appropriate for l
characteriting the alternatives.
l l
The next task is to screen the various alternatives to identify the types that seem i
more appropriate. De screening criteria should be set by the generalists usin; concepts described in Section 3.3.2, since the information is relevant to the overa i repository program. However, at later stages in the analysis, the appropriateness of the screening criterion should be validated. If it turns out the information sought from the ex scriments is not bein g provided, the analysis should be repeated to
- determine which experiments shou d be conducted and whether they are worth the information. Experiments that at one time were thought not to be appropriate i
because of the expectation that certain information would become available have l
become appropriate when it is known that that information is not availabic. In simpler terms, if some field experiments are not successful, the relative desirability of l
others may mcrease.
j For the alternatives that have made it through.the screening, one should more j
carefully specify details of the experiment to be conducted. As part of this, there 1
i should be probabilistic estimates of the amount of information obtained by each of the experiments as well as estimates of their cost, time, and any damage from th:
experimrntation. As in the task of informational drilling, two sets of quantitative estimates are especially important: the conditional probability distribution over i
radionuclide emission for different experimental outcomes and the probability i
distribution over those outcomes. The judgment of generalists will likely be necessary for some of the cost and time information, although this judgment might be j
augmented by some specialists, whereas the judgment of speciahsts will mainly be used to judge the. information expected from each experiment.
he objectives in the first task above need to be integrated into an overall utility function. These value iudgments should be in accordance with the techniques discussed in Section 3.1.5 and should use the judgments of generalists on the repository team. However, these value judgments should be carefully related to the policy value judgements made about the fundamental value tradeoffs of the information gathering process. In other words, since the objectives of the experiments are means to achieve the objectives of designing and constructing a repository, the specific value judgments dealing with tradeoffs among the objectives of experiments must relate to the value tradeois that concern the policy objectives.
l nis relationship should be carefully documented.
1 l
l i
70-lL.-.
^
l
- ,, w..
ll De final task is to analyse the various laboratory and field experiments using the value-of information techniques and.to select those that seem appropriate. In all cases, one of the altematives that definitely should be considered is not conducting l
the experiment. In some sense, one of the more useful pieces of information i
gathered from such an analysis is whether specific experiments, given their quality, cost, and time, are worth the effort. In some cases, it may be cheaper simply to design the repository assuming that a certain situation exists, rather than verifymg it.
In other situations, although tie information desired might be very important, if the experiments are unlikely to provide that information, they simply might not be worth the time, effort, and cost.
{
4J Strategic Repoaltory Decisions l
.i Strategic repository designs are those that directly concern the design, construction, i
and operation of the repository. As pointed out in Section 2.5, many of these decisions will effect the performance of a repository and therefore should be e
considered when developing and screening scenarios, developing model, estimating parameters, and gathering mformation, in a sense, any performance assessment is conditional on these strategic decisions.
l For discussion it is useful to think of the analysis of those strategic decisions in terms l
l of six components. De first two components, which identify the strategic problem, are specification of the objectives and identification of the alternatives. The degree l
to which the objectives are achieved by the various alternatives is quantified in the l
third component. The fourth component integrates the different objectives using I,
l value judgments concerning risk attitudes and the relative importance of different objectives. All the information is integrated and analyzed in component five to provide insight for decision making. Component six is documentation of the process
-i and results.
De main techniques in these components are described in Section 3.3.3 (structuring objectives) ion), and decision (probability quantification), Section 3.3.5
, Section 3.3.4 quantificat Raiffa,1976; and von Winterfeldt and Edwards,1986).
4J.1 Speciffing and Structuring Objectives De overall objectives for constructing and operating the repository should guide the development of specific objectives for constructing or operating the repository, ne techniques for constructing objectives hierarchies are useful Tor this step (Section
. A group of experts te > resenting all interested parties should be selected to 3.3.3)fy the overall objectives for constructing and o rating the repository. At this speci l
stage, it is important to have a broad diversity of inions providmg objectives for i
I the repository, as these objectives should provide th foundation for future strategic decisions (Keeney,1988a,b). De training for these experts need not be extensive, but it should clearly indicate how the stated o ectives will be used and methods that may facilitate broad thinking about their ob ctives. De clicitation process itself needs to be done by normative experts traine to elicit objectives in an operational manner for further analysis. The objectives should then be structured by the l
normative analysts, with the assistance of project members, and then carefully i
reviewed by peers and others interested in the repository program. Modifications are 71 l
(
i welcomed, as the intent is develop an appropriate fundamental set of objectives for j
the repository. Finally, these objectives shoufd be documented, r
With a given specific strategic decision, the repository objectives need to be related i
to specific objectives influenced by the strategic decision. That linking can likely be done by generalists with the assistance of normative experts. In essence, it is a deductive process that relates the overall objectives to a given decision problem. As always, the resulting objectives should be carefully documented after review by peers i
and others interested m the repository program, meluding all members who mitially contributed to the overall objectives.
4JJ Identification of Alternatives For any specific strategic decision, the alternatives need to be identified. Thus, the identification techniques Section 3.3.1 are relevant. The experts involved in specifying alternatives should have substantial knowledge about details of the specific decision to be addressed. Normative experts should assist them in defining generic alternatives (e.g., sets of alternatives that differ in terms of parameters). After a wide range of alternatives has been identified, it may be worthwhile to screen the i
alternatives using the screening techniques in Section 3.3.2. Appropriate screening criteria should be set by generalists to facilitate focusing on alternatives that are presumed to be better. After the analysis, the reasona >leness of the screening l
criteria should be reexamined considering the quality of the screened alternatives. If retained,y that alternatives screened out would in fact be better than some of t it is likel the analysis should be revised and repeated, j
4J.3 Impacts of Altematives Once the objectives and alternatives in a' specific strategic decision problem are articulated, they effectively define a matrix in which objectives relate to the individual columns of the matrix and alternatives to the individual rows. To specify the impacts of the alternatives, one wants to fill in each cell in the matrix, indicating L
the degree to which the alternative impacts the corresponding objective. This process utilizes scientific and engineering knowledge and necessarily relics on the techniques and procedures i
models, data, and expert judgments. For tais step,ing, model development, and outlined for scenario development and screen l
parameter estimation are repeatedly used. Since these are detailed in Sections 4.1 through 4.3, there is no need to elaborate on them here, it is simply worth noting i
that expertise from a variety of fields that includes the behavioral sciences, i
l economics, and medical sciences will likely be required. Most impacts will be uncertain. In those cases, the techniques for probability quantification (Section 3.3.4) j will be useful.
4J.4 Value Judgments At this stage,it is critical to aggregate the various component impacts for each of the alternatives. Because of the uncertainties regarding those impacts, some of these value jud gments must address risk attitudes concerned with those uncertainties, and value tradeoffs among objectives addressing environmental,hments conce because t 1ere are multiple objectives, some of these value jud social, economic, and health and safety impacts. De value judgments should be made as follows. l
DNT 4
First, the original group who s >ecified the overall objectives to the repository should j
~
specify quantitative value juc gments regarding risk attitudes and value tradeoffs among those objectives using the value quantification techniques described in l
Section 3.3.5. Each of the individuals in that group should provide individual value i
judgments, and each of these sets of values should be carefully appraised for i
consistency. Also, individuals should be allowed to hear the logic of other people's x>ints of view regarding the values and reiterate their judgments. However,it would i
>e unlikely that evenbody would have precisel unreasonable to force a consensus (Section 3.4)y the same values, so it would b
. Each individual value should bc s
carefully documented, and collectively they should provide a range for the values i
used in the problem.
)
4JJ Analysis of the Alternatives I
ne analysis of alternatives should integrate all the information from the preceding four components for the given strategic decision using decision analysis, j
Operationally, it may be reasonable to take an " average" set of the value judgments as a base case and do sensitivity analysis from this to incorporate all the different l
viewpoints. De intent is to identify alternatives that clearly are not competitors and Identify circumstances under which each of the remaining alternatives are the best
- and how much better they are than the alternatives. Because of the uncertainty about quantitative parameters relating to the impacts, sensitivity analysis of some of these may also be appropriate. The experts working on this part of the problem should be analysts, it is unlikely that their use of expert judgments needs to be made j
explicit, but they certainly use expert judgment in deciding what sensitivity analyses to r
pursue. The degree of sensitivity analysis should be guided by the insights provided and the need for careful documentation.
4.5.6 Documentation of Analysis
~
The documentation of the analysis and its insights for decision making is essentially a collection of the documentation of each of the com >onents of the analysis. However, i
it is worth recognizing that documenting the overal decision process does have some requirements different from documenting the components. This comes about because the overall process is of interest to different types of individuals, some of whom may not be concerned about details. Documentation of technicalinformation relevant to imaacts is likely of concern mainly to peers and individuals with a technical know edge about those aspects of the repositon. Documentation of the decisions made may be of concern to a large number of lay people as well as to numerous individuals concerned with or entangled by the solitics of the repositoy problem. Documentation of the overall decision need not focus on detailed aspects of the problem that turn out not to be crucial. The documentation should vey carefully, explain what the alternatives are, what the objectives are for evaluating the alternatives, and the logic of why a given alternative was chosen. References can naturally be made to more detailed documentation elsewhere.
Documentation of any strategic decision should be considered itself a decision problem. One should carefully think of the objectives of the documentation and who the documentation is meant to inform because the communication alternatives have pros and cons. These need to be balanced appropriately in documenting the overall 73-
QDOP ~
U,u g <3 ilan s ef4 i
decision. The analysis of the documentation decision need not be made explicitly, but consideration of the appropriate components will likely result in better i
documentation.
i
?
P P
i i
i I
t t
9 9
i i
O g
't 4
' I I
p i
h 74
s.
\\
M Tl l
mod REFERENCES Armstrong, J. S., W. B. Denniston, and M. M. Gordon, "The Use of the Decomposition Principle in Makin Jud ents," Ornanizational Behavior i
and Human Performance 14, 257 63 1975.
~
Arrow, K., Social choice and Individual Values. (New York: John Wiley & Sons, 1951).
Some Ashton, A. A., and R. H. Ashton, " Aggregating Subjective Forecasts:
Empirical Results," Management Scienca,31,14991508 (1985).
Barclay,je., et al., Handbook for Decision Analvsh, Defense S
~
Pro cts Agency (1977).
Bonano, E. J., and R. M. Cranwell, " Treatment of Uncertainties in the Performance Assessment of Geologic High Level Radioactive Waste l
Repositories," 1 Mathematical Geolonv. 20,543 565 (1988).
l Bonano, E. J., P. A. Davis, L. R. Shipers, K. F. Brinster, W. E. Beyeler, C. D.
Upde graff, E. R. Shepherd, L. M. Tilton, and K. K. Wahl, Demonstration of i
ormance Anessment Methodolovv for Hieh 12 vel Radioactive Waste a Per1 D:ssosal in Batnit Formations, SandiaNationalbboratories, Albuquerque, NM, SAND 86 2325, NUREG/CR-4759 (1989).
Brown, R. V., A. S. Kahr, and E. Peterson, Decision Analysis for the Manacer (New York: Holt, Rinehart, and Winston,1974).
~
Bunn, D.W., Applied Decision Analysis (New York: McGraw Hill,1984).
Clemen, R. T., and R. L. Winkler, " Limits for the Precision and Value of i
Information from Dependent Sources," Ooerations Research. 33, 427 442 (1985).
Clemen, R. T., " Combining Overlapping Information," Management Science 13, l
373 380 (1987).
Cranwell, R. M., R. V. Guzowski, J. E. Campbell, and N. R. Ortiz, Ruk Methodoloav for Geolonic Disoosal of Radioactive Waster Scenario Selection PFocedurs, Sandia National Laboratories, Albuquerque, NM, SAND 801429, NUREG/CR 1667, draft report, copy availab.'e in NRC Public Document Room.
Cranwell, R. M., J. E. Campbell, J. C. Helton, R. L Iman, D. E. I.ongsine, N. R.
Ortiz, G. E. Runkle, and M. J. Shortencarrier, Risk Methodoloav for
~
Geoloale Disontal of Radioactive Waste: Final Reoort. Sandia. National Laboratories,~ Albuquerque, NM, SAND 812573, NUREG/CR 2452 (1987).
Dalkey, N., and O. Helmer, "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science. 9,458-467 (1%3).
75-
,.----.,,e._.,--
,..~,.,--m.m
,. _ _. -mr,-..--__
.s
--m.
O----w
,. )
i t
An Exnerimental Study of Grou 3 Ooinion, Dalkey, N. C., The Deinhi Method:
ne Rand Corporation, Santa Monica, CA, Repon No. RM 58880R (1969).
j de Finetti, B., "La Prevision: Ses I.ogique, Sources Subjectives," Annales de l' Institute Henri Poincare. 7,1-68 (1937).
DeGroot, M. H., Optimal Statistical Decisiom (New York: McGraw Hill,1970).
Elanning(Glenview,IL: Scott Foresman,1975). ~
~
Delbecq, A., A. Van de Ven, and D. Gustafson, Groun Techniaues for Pronram A Multiattribute Utility Analysis of Sites Nomir ated for Department of Energy,for the First Radioactiv'e Waste tenository A 3ecision Characterization
~
Aidinn Methodolonv. Office of Civilian Radioactive Waste' Management, Washington, D.C., DOE /RQ 0074 (1986).
Department of Energy, Draft Environmental Act,ument. Reference Renositorv
~
L2 ration. Hasford Site. Office of Civilian Radioactive Waste Management, Washington, D.C., DOE /RW 0017 (1984).
1 Dyer, J. S., and R. A. Sarin, " Measurable Multiattribute Value Functions,"
]
l Onerations Research. 22,810 822 (1979).
Edwards, W., "How to Use Multiattribute Utility Measurement for Social Decision Making," IEEE Transactions of Systems. Man and Ovbernetics j
SMC-7. 326 340 (1977).
~
i i
Edwards, W., and J. R. Newman, Multiattribute Evaluation. (Beverly Hills: Sage, i
1982).
Edwards, W., T. Eppel, A. Hybel, D. Mefford, and D. von Winterfeldt, Dnign Erincioles for an Analvst's Work Station. Social Science Research Institute, l
i Unive'rsity of Southern California, l.os Angeles, Research Report No. 871 i
(1987),
Edwards, W., and D. von Winterfeldt, "Public Values in Risk Debates," Riik Analysis. 7,141 154 (1987).
Einhorn, H., and R. Hoganh, " Unit Weighting Schemes for Decision Making,"
Oreanintional Behavior and Human Performance.13,171 192 (1975).
Einhorn, H. J., " Expert Measurement and Mechanical Combination,"
i Oreanintion Behavior and Human Performance. 7,86106 (1972).
Ekman, G., and L Sjoberg, " Scaling," Annual Review of Pnvcholonv. 16, 451-474 (1%5).
Electric Power Research Institute, Seismic Harmed Methadalmv for the Central and httern United States. Vol.1. Methodoloev. NP 4/26 (386).
(
76-
- .,. /
MMT EPA,40 CFR Part 191, " Environmental Standard for the Mana[ement and Dis posal of Spent Nuclear Fuel, High Level Waste, and 1 ransuranic j
Radioactive Waste; Final Rule U.S. Environmental Protection Agency,"
Federal Register. Vol. 50, Washington, D.C. (1985)
,j Fischer, G. W., "When Oracles Fail: A Comparison of Four Procedures for Aggregating Subjective Probability Forecasts," Ornanizational Behavior and Human Performance. 28,96110 (1981).
~
l Fischhoff, B., "Deblasing," in Judament Under Uncertaintv: Heuristics and l
RIA315, D. Kahneman, P. S16vic, and A. Tversky (c'ds.) (Cambridge:
l Cambridge University Press,1982),422 444.
Fischhoff, B., S. Lichtenstein, P. Slovic, S. L Derby, and R. L Keeney, Acceptable Risk (Cambridge: Cambridge University Press,1981).
Fischhoff, B., P. Slovic, and S. Lichtenstein, " Fault Trees: Sensitivity of Estimated Failure Probabilities to Problem Representation," Journal of l
(1578).
~
~
4, 330 344 Er wrimental Psycholorv: Human Percention and Performance.
Inc., Smit Renository Emniacement Mode Evaluation and
- Fluor Technology,l Report, Ady'anced Techn' ology Division, Irvine, CA (March Selection. Fina 1988).
Genest, C., and J. V. Zidek, " Combining Probability Distributions: A Critique and an Annotated Bibliography," Statistical Science.1,114148 (1986).
~
Get
, C. F., S D. Fisher, and T. Mehle, Hvoothesis Generation and Plausibility Assessments, University of Oklaho'ma, Decision Process Laboratory, Norman, Oklahoma, Technical Report No. 15 10 78 (1978).
Golder Associates, Renort to Rockwell Hanford Operations. Probability ineodine of Vtriab'les Affectinn Inflow into the Eroloratory Shaft Tes't Tacility at the Basalt Waste Isolation Project. Rid' eland. Washinnton, n
Redmond, Washington (Jur.c 1986).
~
~
~
Goodman, B., " Action Selection and Likelihood Estimation by Individuals and l
Grou s," Ornanizational Behavior and Human Performance 7, 121 141 (1972.
~
Gregory, R., and S. Lichtenstein, "A Review of the High Level Nucleat Waste Repository Siting Analysis," Risk Analyag 7, No. 2,219 2D (1987).
Hogarth, R., Judgment and Choice: The Psycholorv of Decisions, (Chichester, England: Wi.ey,1980).
C. A., Decision Makine Under Uncertainty (Englewood Cliffs, NJ:
Holloway, ice Hall,1979).
Prent 77-
.~
7 i
k t
Howard, R. A., "he Foundations of Decision Analysis," MFFF Transactions on c
Svstems Science and C6ternetiet SSC:3,211219 (19tW ).
Ibrekk, H., and M. G. Mo'rgan, "Gra shical Communication of Uncertain i
Quantities of Nontechnical Peopic," Rhk Analysis 7,519 529 (1987).
l Iman, R. L, and J. C. Helton, A Comnarlson of Uncertainty and Sensitivity Analysis Techniaues for,Cgnggter'Modelt Sandia National Laboratories, l
~
~
Albuquerque, NM, SANDS 4 1461, NUREG/CR 3904 (1985).
Jungermann, H., and M. Thueting,7hc Use of Casual Knowled e of Inferential t
Reasoning," in Eroen Judement and Exnert Systems, J. L N umpower, L D.
Phillips, O. Renn,'and V. R. R. Uppulur'i (eds.) (New York: Springer,1987) 131-146.
i Kahnemann, D., and A. Tversky, *Inteitive Prediction: Blases and Corrective Procedures," ManagemenLSt.itnsa 12,.313 327 (1979).
Kahnemann, D., P. Slovic, and A. Tversky,idge University Press,1982).
iv.dyment Under Uncertainty; Heuristics and Biases. (New York: Cambr Keeney, R. L, Siting Enerev.Easiluigi (New York: Academic Press,1980).
Keeney, R. L, "An Analysis of the Fortfolio of Sites to Characterize for Selecting a Nuclear Repository," Risk Analysis 7 No,2,195-218 (1987).
l Keeney, R. L, "Value Focused Thinking and the Study of Values," in Decision Analysis: Descrip1]FfJ.DirnaliE.JNihq3cdpliys_Asoeets D. E. Bell, H.
Raiffa, and A. Tversky Press,1988a),654 690, (;ds.)'(Cambridge, England: Cambridge University l
e Keeney, R. L. " Structuring Objectives for Problems of Public Interest,"
Qptptions Resmph,36, No. 3, 396 405 (19285).
l Keeney, Wlue T adf.q((i(New Yort.:R. L, and H. Raiffa, Qtqitinn13.i3h Multiole l
l.
and l
Wiley,1976).
\\
r L
Keet:cy, R. L, and D. von Winterfeldt, "Irnproving Risk Communication," Hisk l.
Riitlysi'i,6,417 424 (1986).
Lichtenstein, S.. and B. Fischhoff, " Training for Calibration," Orennizational l
Echinictmt.lintrJAMqtmatica, 26,149 171 (1980).
~
Lichtenstein, S., B. Fischhoff, and L D. Phillips, " Calibration of Probabilities:
De State of the Art," in Quition Making and Chan te in Human Affairs. H.
Jungermann and G. de 'Isecuw (eds.), (D'ordrecht, Eolland: Reidell,1977),
275-324.
1 78-l-
I%*'
i t
Uchtenstein, S., B. Fischhoff, and L D. Phillips, " Calibration of Probabilities:
The State of the Art to 1980," in Judament Une er Uncertaintv: Heuristics
, D. Kahneman, P. Slovif, and A. "versky (eds.) (New York:
~
bri ge University Press,1982),305 334.
Linstone, H. A., and M. Turoff (eds.), The Deinhl Method: Technlaues and Aprilentinas: (Reading, MA: Addnon Wesci~ey,1975).
~
Matheson, T. E., and R. L Winkler, "Sco >ing Values for Continuous Probability Distribution," Manmoement Scienee.12,10871090 (1976).
McCormick, N. J., Reliability and Risk Analvsis (New York: Academic Press, 1981).
~
MacGregor, S. Uchtenstein, and P. Slovic, " Structuring" Knowledge Retrieval: An Analysis of Decomposing Quantitative Jud_ gments, Ornanizational Behavior and Human Decision Proceues. 42,303 323 (1988).
~
Merkhofer, M. W., "Quantifyi,n, r Judgmental Uncertainty: Methodology, Experience, and insights, EEE Transactionc on Systems. Man and I
l cvhernetica. SMC 17. 741752 (1987).
~
A Multiattribute Utility Analysis of Merkhofer, M, W., and R. L. Keeney, " Nuclear Waste," Risk Analysis 5,173-Alternative Sites for the Disposal of 194 (1987).
l Quantifyirig Merkhofer, M. W., and A. K. Runchal, " Probability Encoding: Basalt," Ems, l
Judgmental Uncertainty over Hydrologic Parameters for l
Conference on Geostatistical Sensitivi:v. and Uncertainty Methods for Ground Water Flow and Radionuclide Transoor: Modelini. B. E. Buxton (ed.)(Columbus, OH: Battelle Press,1989) 629'643.
~
Meyer, M. A., and J. M. Booker, sources of Ccrrelation Between Exoerts:
~
imeirical Results from Two Extremes. l.os Alamos National Laboratory,
.os Alamos, NM, NUREG/CR 4814 (1988).
~
Mosleh, A., V. M. Bier, and G. Apostolakis, "A Critique of Current Practice for the Use of Expert Opinions in Probabilistic Risk Assessment," Reliability Ennineerina and System Caferv. 20,63 85 (1988).
NRC,10 CFR Part 60, "Disxmal of High Level Radioactive Wastes in Geologic 48,positories, U. S. Nuc car Regulatory Commission," Federal Reaister, Vol.
Re Washington, D.C. (1983).
NRC, Guidance for Determination of Anticinated Processes and Events and Unanticinated Procenes and Events, Draft Generic Technical Position, 3ivision'of High Level Waste Management, Office of Nuclear Material and Safeguards, U.S. Nuclear Regulatory Commission, Washington, Safety (1988).
D.C.
6 79-s
. - ~ ~.
m
.~.1 b
e NRC, Rgneint Rafety Studvr An Assessment of Accident Risks in U.S.
cammercial Nuernar Po'wer Plantt Washington, DC, Nuclear Regulatory Commission, NUREG 75414 (1975).
j Okrent, D., " Comment on Societal Risk," Science 208,372 375 (1980).
PSAC,12 vel S. Soecification for a Sensitivity Analysis Exercise Based on 1.svel E i
Test c=>- P~. C. Robinson, A. Saltelli,'A. Alonson, and E. J. Bonano (cds.),
Probabilistic Safety Assessment Code User's Group, Nuclear Energy Agency, Organization for Economic Cooperation and Development, Pans, France (1989).
Pearl, J. A., A Goal Directed Annroach to Structurine Deelsion Problems, University of California, Los Angeles, Report No. L CLA ENG 7811 (1978).
)
~
Peterson, C. R., and L R. Beach, " Man as an intuitive Statistician," Psveholocical kllsiin 68,29-46 (1967).
Pitz, G. F., " Subjective Probability Distributions for Imp (ed.) (New York:erfectly K Quantities," in Knowledne and Connition, L W. Gregg j
John Wiley & Sons,1974).
Pitz, G. F., and N. Sachs, " Judgment and Decision: Theory and Application,"
i Annual Review of Psvcholorv,35,139163 (1984).
Pitz, G. F., N. Sachs, and J. Heerbroth, " Procedures for Eliciting Choices in the Analysis of Individual Decisions," Ornanizational Behavior and Human l
~
Performance. 26,396-408 (1980).
Poulton, E. C., "Models of Biases in Judging Sensory Magnitude," Psycholocical
,kljstin,86,777 803 (1979).
Raiffa, H., Decision Analysis (Reading, MA: Addison Weseley,1968).
Raiffs H., and R. L Schlaifer, Anelled Statistical Deeielan heolv. (Cambridge, MA, Harvard University Press' 1%1).
~
l 5
R F. P., " Truth and Probability," in The Foundations of Mathematics and 131).
~
Harcourt Brace, her Imaical havt R. B. Brainwaithe (ed.), (New York:
Ravinder, H. V., D. N. Kleinmuntz, and J. S. Dyer, "The Reliability of Subjective l
No. 2,186-99 (February 1988)gh Decomposition," Man==ement Science.
Probabilities Obtained Throu Savage, L J, he Foundations of Statistics (New York: Wiley,1954).
Shachter, R. D., "Probabills:ic Inference and Influence Diagrams," Operations EsArmsh. 36,589-604 (1987).
i 1
80-i
d c>, '
g YT
_ad i Server, D. A., A**-8"* MeSllity with Multiole Individuals: Groun Interaction Varama Mechanical Amarenation, Social Science Research Institute, Unhpersity of Southem Califo~ rnia, Los Angeles, CA, Research Report No.
78-3 (1978).
Selvidge, J., Ernerimentti Comnarlson of Different Methods for Assessinn f
Extremes of Prahnhi tev Distributions by the Fractile Method, Technica'
~
Report, Graduate School of Business,~ University of Colorado, Boulder (1975).
Shanteau, J., " Psychological Characteristics of Expert Decision Makers," in W Judement and Freert Systems, J. Mumpower et al. (eds.), (New York:
Springer.1987) 289 304.'
~
Slovic, P., B. Fischhoff, and S. Lichtenstein, " Behavioral Decision Theory,"
Annual Review of Pavchaloav 28,139 (1977).
Spetzler, C. S., and C. A. Stael von Holstein, " Probability Encoding in Decision l
i Analysis," Manmoement trieace 22,340 352 (1975).
'Stact von Holstein, C. A., S., and J. E. Matheson, A Manual for Encodine Probability Distributlant-SRI International, Menlo Park, CA, SRI Projed j
l 770778 (1979).
Stillwell, W. S., D. A. Seaver, and J. P. Schwartz, Ernert Estimates 3f Human
- Error Probabilities in Nuclear Power Plant Onerations: A Review of Drobability Assestmitnt and Scalinn, Nuclear Regulatory Commission:
- ashington, D.C., NLREG CR 2255 (1981).
r
'!hompson, B. d. J., C. J. Friselle, D. B. Nichols, and M. C. Thome, A Method of Overcomine Limitations of Conventional Scenario Based Risk Assessment l
by Usinn Monte Carlo Simulations of Possible Future Environmental Nuclear Ener A ency, l
ChangeL' tiansmitted to Scenario Working Grou velopment, Pag, rance l
r ization for Economic Cooperation and ll von Winterfeldt. D., and W. Edwards, Decision Anah' sis and Behavioral Research (New York: Cambridge University Press,1986).~
Wallsten T. S., and D. V. Budescu, " Encoding Subjective Probabilities: A ogical and Psychometric Review," Management Science. 29, 151 73 Weber, M. O., F. Eisenfuehr, and D. von Winterfeldt, "The Effects of Splitting Attributes on Weights in Multiattribute Utility Measurement," Manmoement Sciangs,34,43145 (1988).
Westinghouse Electric Corporation, Phat* I Stud of Metallic Stk Systems for S_ent Fuel Manmoement from Reactor to Renository. Westinghouse Waste
~
r n
Technology Serv' ices Division, Madison, PA, Report WTSD TME 085 (1986).
4 0
81-
- o op j
Wheeler, T., S. C. Hora, and W. Cramond, Analvtis of Core Damane Freauenev:
Ernert Elielta" lor on Internal Events. Vol. 2. U.S. Nucleir Regulatory CommistJon, N JR EG/CR4550 (1989).
- 1 Wheeler, T., S. C. Hora, and W. Cramond, Front.End Ernert Oninion Elici tion
Draft Report. Sandia National Laboratories (1988) ~
Wilson, R. " Commentary: Risks and Their Acceptability," Science Technology and Human Values 9,122 (1984).
)
l Winkler, R. L, Introduction to Bavesian Inference and Decision (New York:
Holt, Rinehart, and Winston, Inc.,1972).
J
)
Zinnes, P., " Scaling," Annual Review of Psycholonv 20,447 488 (1959).
j Y
i N
l
~
i k
l I
i I
l
.B2-
-