ML20153D210

Approaches to Uncertainty Analysis in Probabilistic Risk Assessment

ML20153D210
Person / Time
Site:	LaSalle
Issue date:	01/31/1988
From:	Bohn M, Parry G, Tyrus Wheeler SANDIA NATIONAL LABORATORIES
To:	NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES)
References
CON-FIN-A-1384 NUREG-CR-4836, SAND87-0871, SAND87-871, NUDOCS 8805090083
Download: ML20153D210 (108)

Text

.

NUREG/CR-4836 S AND87 -0871 RX Prir.ied January 1988 l

Approaches to Uncertainty Analysis in Probabilistic Risk Assessment Michael P. Bohn, Timothy A. Wheeler, Gareth W. Parry

n. n,ea e, Land a N3 tons Labc'a!Orms A!tv)Rf que hea S %a00 $7 f $$ 3M Mmof e CaMarna 94550 tot t'w Uruted States Der a tmert of Er frgy urd, t Contra:t Cf. ACC4-76DP00783 i

l

~

~- ~

8805090083 DR ADOCK G8O'13'1 '

05000373 PDR 1 l Prepared for U. S. NUCLEAR REGULATORY COMMISSION )

. . , 'n j

s. -'

i

! l I

NOTICE l This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, or any of their em-ployees makes any warranty, expressed or implied, or auumes any legal liability or responsibility for any third party's use, or the results of such use, et any information, apparatus product or proccw disclosed in this report, or represents that its uw by such third party would not ininnge privately owned rights.

Available from Superintendent of Documents U.S Government Printing Office Pmt Of fice Ikn 37082 Washington. D C. 20011 7082 and National Technical Information Service l Springfield, VA 22161

)

NUREG/CR-4836 SAND 87-0871 RX APPROACHES TO UNCERTAINTY ANALYSIS IN PROBABILISTIC RISK ASSESSMENT Michael P. Bohn Timothy A. Wheeler Gareth W. Parry

• January 1988 Sandia National Laboratories Albuquerque, NM 87185 Operated by Sandia Corporation for the J U.S. Department of Energy Prepared for Division af Reactor and Plant Systems Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555 Under Memorandum of Understanding DOE 40-550-75 NRC FIN A1384

• NUS Corporation, 910 Clopper Road, Gaithersburg. Maryland, 20878

ABSTRACT An integral part of any probabilistic risk assessment (PRA) is the performance of an uncertainty analysis to quantify the un-certainty in the point estimates of the risk measures consid-ered. While a variety of classical methods of uncertainty analysis exist, application of these methods and developing new techniques consistent with existing PRA data bases and the need for expert (subjective) input has been an area of considerable interest since the pioneering Reactor Safety Study (WASH-1400) in 1975. This report presents the results of a critical review of existing methods for performing uncertainty analyses for PRAs, with special emphasis on identifying data base limita-tions on the various methods. Both classical and Baysian approaches have been examined. This work was funded by the U.S. Nuclear Regulatory Commission in support of its ongoing full-scope PRA of the LaSalle nuclear power station. Thus in addition to the review, this report contains recommendations for a suitable uncertainty analysis methodology for the LaSalle PRA.

4 4

I lil/iv

CONTENTS ,

Pace I. INTRODUCTION....................................... 1 ,

l II. UNCERTAINTY METHODS FOR PROBABILISTIC RISK ASSESSMENT...................................<..... 3 l 11.1 A Perspective on Uncertainty Analysis....... 3 l

11.1.1 Uncertainty Analysis Goals......... 3 1I.1.2 Uncertainty and Physical Variability........................ 4 II.1.3 The Representation of Uncertainty........................ 5 11.1.4 Data Availability Constraints...... 9 II.2 Methods for Propagating Component Uncertainty Through PRA Models.............. 13 11.2.1 Maximus/ Bounding Approach.......... 13 II.2.2 Method of Moments.................. 16

. II.2.3 Method of Moments Using Tchebyschev's Inequality........... 17 II.2.4 Propagation of Discrete Probability Distributions.......... 17 II.2.5 Monte Carlo Method................. 17 II.2.6 Latin Hypercube Sampling........... 18 II.3 Method Recommended for RMIEP................ 19 III. ELEMENTS OF RECOMMENDED APPROACH FOR UNCERTAINTY PROPAGATION........................................ 20 l III.1 Propagating Uncertainty Through RMIEP Models...................................... 20 111.1.1 Latin Hypercube Sampling........... 20 111.1.2 Quantification of Top Event Attributes......................... 20 l

11I.1.3 Uncertainty Correlation in Sampling 21 1 j 1I1.1.4 Sensitivity and Importance l Measures Computed in TEMAC......... 22 i 111.1.5 Inclusion of Modeling Uncertainty........................ 25 IV. DATA AND DATA ANALYSIS............................. 29 IV.1 The Parameter Value Estimation Process...... 29 IV.l.1 Component Definition and Classi-fication of Failures............... 29 l

V

e CONTENTS

-(Continued)

^r P a c,e_

IV.1.2 Screening'Quantification of Component Failures................. 29 IV.l.3 Finalizing of Parameter Estimates and Uncertainty.................... 30 IV.2 Data Needs and Data Types................... 31 IV.3 Data Sources................................ 36 IV.4 Data Analysis Methods Available............. 38 IV.4.1 Bootstrapping Method............... 38 IV.4.2 Use of Plant-Specific Data in a Bayesian Analysis.................. 41 IV.4.2.1 Use of Non-Informative Priors................... 41 IV.4.2.2 Bayesian Updating of Generic Data............. 41 IV.4.3 Expert Opinion..................... 42 IV.5 Elements of the Bayesian Updating of Basic Event Parameter Attributes.................. 42 IV 5.1 Bayes' Theorem..................... 43 IV.5.2 Constant Failure Rate Model........ 44 IV.S.2.1 Bayes' Theorem with Conjugate Priors......... 44 IV.S.2.2 Bayes' Theorem with Non-Informative Priors... 45 IV.S.2.3 Bayes' Theorem with Lognormal Priors......... 46 IV.5.3 Constant Failure Probability on Demand Mode 1.................... 46 IV.S.3.1 Bayes' Theorem with Conjugate Priors......... 47

, IV.5.3.2 Bayes' Theorem with Non-Informative Priors... 47 IV.5.3.3 Bayes' Theorem with Lognormal Priors......... 48 IV.5.4 Development.of Prior Distributions from Subjective Opinion............ 48 V. CONCLUSIONS........................................ 49 VI. REFERENCES......................................... 50 vi

r CONTENTS (Concluded)

Pace APPENDIX A. INTERFACE BETWEEN CORE DAMAGE FREQUENCY ANALYSIS (RMIEP), THE ACCIDENT PROGRESSION e

AND CONTAINMENT FAILURE ANALYSIS (PRUEP)

AND THE HEALTH AND ECONOMIC CONSEQUENCE ANALYSIS.................................... A-1 i

APPENDIX B. LATIN HYPERCUBE SAMPLE DEMONSTRATION ....... B-1 APPENDIX C. ELICITATION OF EXPERT OPINION--SOME GUIDELINES.....................,............ C-1

)

vil

LIST OF FIGURES Fiqure Pace 1 Chart Providing 95 Percent Confidence Limits For p in Binominal Sampling. Given a Sample Fraction x/n..................................... 7 2 Lognormal Distribution--Valves................... 12 3 Illustration of Maximum / Bounding Intervals....... 15 LIST OF TABLES Table Page 1 LaSalle Unit 2 Initiating Event Categories....... 32 2 Collections and Summaries of Actual Failure {

Events........................................... 37 3 Statistical Analycis of Data..................... 39 4 Generic Failure Rate Data Bases.................. 40 '

vili

ACKNOWLEDGMENT This work was supported by the Division of Reactor cnd Plant  :

Systems of the U.S. Nuclear Regulatory Commission. Valuable l review and comments were provided by D. M. Rasmuson and M. A.

Cunningham of the NRC staff. Within Sandia. R. G. Easterling.

F. L. Spencer, and R. L. Iman provided generously of their time in providing background papers, discussion and comment based on their current research activities in uncertainty analysis techniques. We are especially in debt to Jane Booker and Mary Meyer of Los Alamos Laboratory (LASL) who prepared a summary of the LASL research on the use of expert opinior and guidelines l for use of such information. This contribution is included as Appendix C. Finally, special thanks are due to Emily L.

Preston and Bonnie Haynes for their patience and perseverance in preparation of this manuscript.

4 i

e i

e e

1 IX

__ _ m - - - _ _ _ ._ . _ _ _ . _

I. INTRODUCTION This report presents the results of a critical review of methods for performing uncertainty analyses for Level 1 probabilistic risk assessments (PRAs). This work was performed in suppott of the Risk Methods Integration and Evaluation Program (RMIEP)1 as part of the first full-scope probabilistic risk assessment being performed under the auspicee f the US Nuclear Regulatory Commission since the Reactor Safet, Study. This work was per-formed as part of two programs supporting RMIEP: the Dependent Failure Methodology Program and the PRA Methods Development Program, both of which are sponsored by the Division of Reactor System Safety. U.S. Nuclear Regulatory Commission. These pro-grams have the goal of developing new risk assessment methods and integrating both new and existing methods into a uniform procedure for performing an in-depth PRA with consistent levels of analysis for external, internal and dependent failure scenarios.

As part of RMIEP, a PRA of the LaSalle Nuclear Generating Sta-tion is being performed. LhSalle is a Mark II BWR/6 commercial power station. The PRA will include an analysis of the uncer-tainty in the risk estimates calculated for LaSalle. This

, report presents recommendations for methods to be used in per-forming the RMIEP uncertainty analysis for LaSalle.

There are three major facets of an uncertainty analysis

(1) identification and characterization of the uncertainties in the inputs, (2) propagation of these uncertainties through the systems models to arrive at a measure of the uncertainty in the output, and (3) identification of the significant sources of uncertainty with respect to the result. In general, we are j interested in the quantification of system fault trees and accident sequence equations. The inputs for which uncertain-ties are generally provided are the parameters which are used to derive estimates of basic event probabilities. The outputs are system unavailabilities, accident sequence frequencies, and core damage frequency.

Another type of input uncertainty that may be addressed is modeling uncertainty. Where it is not clear, for example, I which assumptions about success criteria, or the effects on  !

certain components of a particular environmental condition, are I appropria te, alternative hypotheses can be explored by making l different assumptions, j l

This report presents a summary of uncertainty analysis methods l applicable to risk analyses. In the context of this repott, risk refers to estimates of core damage frequency. The methods reviewed will be pr^sented in two parts: methods for propagat- ,

ing the uncertainty in basic event model parameters (and also 1 I uncertalaty due to alternative modeling hypotheses) through system and accident sequence models, and methods for character-izing the uncertainty in the basic event model parameters and modeling assumptions.

Input uncertainty characterization methods reviewed are:

Plant-Specif.'c Data Analysis Bootstrapping Method Bayesian Update Approach Use of Expert Opinion The uncertainty p:opagation methods reviewed are:

Method of Moments Method of Moments Using Tchebyschev's Inequality Propagation of Discrete Probability Distributions Maximus/ Bounding Approach j Monte Carlo Methods Latin Hypercube Sampling Methods The uncertainty analysis method recommended for RMIEP is the 1 Latin hypercube sampling technique which uses stratified sampling of the basic event probabilities, initiating event frequencies, and alternative modeling hypotheses for quanti-fication of the top event uncertainty. Uncertainty in basic event probabilities is characterized using probability distribu-tions derived in three different ways: when sufficient plant-specific data on individual components or component types exist, a Bayesian analysis using non-informative priors may be used; Bayesian updating of generic parameter estimates may be used for components for which only limited plant-specific data exist; expert opinion is used to characterize parameter uncertainty when data are limited or nonexistent. Expert opinion is, of necessity, the only method available for the quantification of uncertainty with respect to alternative modeling hypothesis. Expert opinion is used generally to develop uncertainty representations from literature cutveys.

Delphi surveys, and non-nuclear experience.

The approach recommended Cor RMIMP and discussed in this report is intended to cover all aspects of the uncertainty analysis of accident sequence frequencies and core damage frequency. Very similar (but somewhat generalized) techniques will be used for the uncertainty analysis of containment failures, source term, and health and consequence modeling calculations. These aspects are being considered in the NRC-sponsored Probabilistic Risk and Uncertainty Evaluation Program (PRUEP).2 Appendix A presents a discussion of the interface between these two sets of calculations, and it is shown there that the data and model-ing techniques used in the accident sequence / core melt uncer-tainty analysis are directly compatible with the accident progression and health and economic modeling, process.

l l

i l

II. UNCERTAINTY METHODS FOR PROBABILISTIC RISK ASSESSMENT l In this section, methods used in PRA analyses for uncertainty l are briefly reviewed. Top event uncertainty methods are treat 6d  !

first, then component parameter uncertainty specification tech-niques will be considered.

i 11.1 A Perspective on Uncertainty Analysis Before discussing methods for performing uncertainty analyses, it is worthwhile to examine the goals of performing such an analysis, and to consider the sources of uncertainty we ' are trying to characterize. Clearly, the goal is to assess un-certainty in the estimate of the frequency and sensitivity measures of some top event given a characterization of the uncertainty in the input (independent) variables or assumptions. .

In the context of a PRA, the analyst is interested in assessing the uncertainty in computed risk measures (such as accident sequence frequency, core melt frequency, and plant damage state frequency) as a function of-uncertaintles in the interpretation -

of data and the parameter estimation process as well as uncer-tainties inherent in the modeling process-itself. Tr.e ultimate goal, however, is to be able to make meaningful comparisons between measures of risk so that robust decisions regarding the margin of safety and allocation of resources to increase safety can be made. A crucial aspect of selecting any method of uncertainty analysis is to identify the comparisons to be made, i

and to assure that tne selected uncertainty analysis method

] will provide the information needed to make those comparisons.

11.1.1 Uncertainty Analysis Goals In the context of a PRA for a nuclear power plant, we ulti-mately wish to make the following comparisons and assessments after the PRA is completed: ,

l  !

1. Compare one plant's risk measures with those of other ,

, plants to assess relative level of safety. '

i

2. Compare relative risk contributions due to the various ,

sources of risk modeled in the PRA (random initiating events, fires, earthquakes, etc.) to identify any particular vulnerability which may exist at the plant in question.

3. Compare health and economic consequences of hypo-thesized accidents at the plant with cimilar conse-.

] quence measures due to other sources of risk (e.g.,

aircraft crashes, chemical releases, etc.) to which the i public is unavoidably subjected.

I  ;

)

]  :

f

4. Compute both abest estimate" and "worst case" measures of risk and consequence to allow informed societal decisions as to "acceptable" level of risk to be made.

5. Compare the contributions of different component failures to the plant risk measures to determine those components for which an increase in reliability would result in the greatest reduction in plant risk and, thus, provide a basis for an effective allocation of resources.

6. Compare the "best estimate" accident-induced loads (e.g., pressures, thermal forces, earthquake ground accelerations, floor spectra, etc.) with the "conserva-tive" deterministic loads used in the design of the plant to assess the "margin of safety" of the various components, structures, and safety systems and to identify any areas where changes in design procedures

, could result in a more uniform level of "margin of .

safety" throughout the plant. I

Thus the goal of performing an uncertainty analysis is to qual-ify the conclusions made as a result of point estimate evalua-tions, so as to identify those conclusions which are robust when considering uncertainties in the analysis. It should also be used to identify where improving the state of knowledge can lead '

to maximum benefit with respect to an accurate assessment of 1 J risk.

4 In particular, an uncertainty analysis should identify those ,

) components or models whose uncertainty is "driving" the overall uncertainty in the plant risk measures, so as to identify any components or models whose uncertainty is preventing any desired comparisons (as discussed above) to be made, and hence identify those components or models for which a reduction in uncertainty

, (even without a change in our perception of their point esti-mate) would result in upgrading our decision-making capability.

11.1.2 Uncertainty and Physical Variability L

it is important to distinguish between the concepts of uncer-

tainty and variability. The nature of the events considered in a PRA (such as initiating events, component failures, operator i actions, etc.) is such that they are treated as being random processes, and modeled through the une of probabilistic models. i lt is this use of probability which gives the PRA its name.  !

l Sources of random variability are incorporated directly in the i PHA models. As examples, the random nature of component 4

failure times is incorporated by assuming a constant failure

cate model for those components. The randomness of operator I recovery times is incorporated by establishing a relationship l between time and the probability of recovery. The randomness (

in the magnitudes of earthquakes in a given seismic region is l

L d.

l incorporated through the use of a relationship expressing i frequene" of occurrence as a function of magnitude. These sources of variability then directly affect the evaluation of the frequency of the events of interest, such as core melt frequency.

However, due to lack of data, cc a lack of detailed under-standing of the physical phenomena being modeled, the relation-ships that are used to de tsc r ibe the variability are not known precisely. This can be reflected in a lack of precision in the value of a component failure rate, or in the provision of alternative mathematical formalisms. This lack of knowledge is the uncertainty that is of interest here, and which leads to the lack of precision in the predictions of the PRA. An increased level of knowledge will not change the fact that a -

PRA is a probabilistic model, but it will give greater confidence in the predictions of that model.

II.1.3 The Representation of Uncertainty

) There are two basic types of uncertainty to be addressed: the first is that associated with the values of the parameters of the input models, the second is that associated with the possibility of alternate modeling hypotheses.

In the case of parameter values that are estimated on the basis of data, there are two basic statistical approaches; the so-called classical approach, and the Bayesian approach. Given j sufficient data, they both produce numerically compatible results, but due to differences of interpretation, propagation of uncertainty measures in the Bayesian approach is easier than in the classical approach.3 (a) Classical Statistical Measures of Uncertainty l

l For parameters that are assumed to have the same constant value for all members of the population being sampled, we can compute a statistical confidence interval, which is an interval about the computed parametet value that is "reasonable" considering the data from which the parameter estimate was derived. For example, consider a component with a history of x failures in n l demands. Assuming that the demand failure cate is constant and that the Binomial distribution is an appropriate model for the occurrence of such failures, the point estimate of the demand failure rate is given by j i

p,Xn (1) i i

1 n

The cumulative binomial distribution iunction is given by

• 1 n x F(x1,n p) = (x)p(y,p)n-x (2) x=o l

l which gives the probability of X < x1 for a known value of p.

From the component failure history, we know the number of failures x and number of demands n, _ so we can solve equation (2) f or values of p such that the observed value of x lies in the central region of the~ distribution, i.e., would be likely ,

to have been observed. Hence, if we sought values of p which l would imply the measured value of (x.n) to lie in the central l 95 percent of the distribution, we would solve the pair of l equations )

x l x

(n)p gy,p )n-x = 0.025 o

(3) x t

~

(n)P 2 (1-p2) = 0.975 i o and obtain two values, pi, p2, between which we would -

(loosely speaking) expect the true value of p to lie with 95 percent confidence.*

j Figure 1 gives the solution to this pair of equations for 4

various values of x and n. The number attached to each curve is also n, the number of demands. The ordinate gives the 95 percent confidence intervals for any value of x/n. Similar curves could be constructed for any other desired level of confidance.

J r

• More precisely, a (1-a) percent confidence interval is defined by (Ref. 4), "If, in a series of very many repeated '

experiments an interval such as the one calculated were l

obtained, we would in the long run be correct (1-a) 100 ,

percent of the time in claiming that p is located in the i interval."

l i

s

) i

I

~s

,,............................n..................../n- ..,

}

, - y, .

.n

/p // en

1. N,/z'D/ ..

/ / / /,

, /- -

,/

-y y - -

.. /

, l 9 /,.'3 l/

-+/  ;# ..

• • ~

7 f /

~7 l /g

,}y} /f-/ A , f}Y

,, -f - .-

$l'.7 . yA*

- /

7 d

.. ! -- l' _

/r b- ' -

en

, y' -' - ,- -.' -.

en

++ l' gh -; ., '

r f. ..

i: h ( M 9 '- [.

sin

.........u...I......n.........

1 Figure 1. Chart Providing 95 Percent Confidence Limits for p in Binomial Sampling, Given a Sample Fraction x/n.

I Different curves represent different sample sizes n.

Values on ordinate represent 95 percent confidence 9 bounds on p. (Reference 4).

It is important to note from this figure that as n becomes large, the confidence limits approach the point estimate p = x/n. Thus, by accumulating more and more data, we can know the value of p as closely as desired (with 95 percent confi-dence). Thus, the confidence interval is a measure of the sampling uncertainty in determining a constant parameter from data.

Note also that pi and P2 are not points on any distribu-tion, for the failure rate is, by definition, a constant and not a random variable, and thus no distribution can be attached to it. In particular, it is not valid to assume pt, p2 to be the 5th and 95th percentiles of some (unknown) distribution for p.

I Even though, as discussed earlier, the effect of variability arising from the inherent randomness of the processes modeled  ;

is generally incorporated directly into the frequency evalua- 1 tion, it is useful to discuss here the representation of variability because it is a concept raised later in the consideration of pooling of data from different populations.

In dealing with a quantity which is a random variable with a known distribution, we state the variability in terms of a probability or tolerance interval taken directly off the prob ability distribution for the quantity. Thus, we can ' identif y the 5th and 95th percentiles of the probability distribution which characterizes the random variability and state that there is a 90 percent probability that any randomly selected value lies between these two limits. When the parameters of the distribution are not known exactly, but are inferred from a set of n observations (such as the sample mean and sample standard deviation for a normally distributed random variable) we can compute a _ statistical tolerance interval, which is an interval "within which we can state with a given confidence of being correct that a prespecified portion of values from a sampled distribution are located" (Reference 4). The probability of being correct, of course, increases with increasing number of observations.

(b) Bayesian and Subiectivist Measures of Uncertainty The subjectivist interpretation of probability as a measure of an analyst's degree of belief in some proposition allows the following approach to the characterization of a parameter value uncertainty. Suppose a probability density function f(K) exists, such that the integral A

[c f(K)dk = c l

i i

represents the analyst's degree of belief that the true value of the parameter is less than Ke, then (0, Kc) is a i Bayesian probability interval. While it has the same intent as the classical confidence interval, i.e., to represent the analyst's uncertainty in the value of the parameter, the ,

mathematical formalism adopted does allow the combination of ,

measures of uncertainty in different parameters in a straight- ,

l forward manner. This is discussed in several references (see, for example, the review paper by Parry and Winter, Reference -

3), and will not be elaborated on here.

There are often several parameters in a PRA model for which i data do not exist from which to make estimates, and engineering judgment is used. If uncertainties on parameter values are to be characterized in a manner consistent with that for these parameters for which data do exist, the Bayesian framework provides a mechanism for doing so.

By contrast, classical statistics does not provide a means of providing such estimates or uncertainty measures.

In addition, the subjectivist approach allows a probabilistic characterization of modeling uncertainty, whereac there is no equivalent classical statistical method. The value of perform-ing such a characterization is a function of the purpose for which the PRA is being performed. Since the assignment of consistent probabilities is not trivial and is analyst-

• dependent, some have found the Bayesian approach to probability i unacceptable. ,

II.1.4 Data Availability Constraints l The availability of data as to type, quantity, and applicability is the second crucial aspect in choosing an uncertainty analysis

method for any given application. In the context of a PRA of a ,

l nuclear power plant, we are (in most cases) dealing with rare j i events. For example, random failures of components at any i j given plarat are quite infrequent, and the analyst is usually l i forced to merge the experience of a number of plants to obtain l a data base which covers the full spectrum of components needed l l in a PRA. Similarly, the occurrence of earthquakes is a rare i event anywhere, while the occurrence of an earthquake in the J

vicinity of a plant (close enough to at least trip the plant) is even more rare. A similar situation exists with respect to I pipe break frequencies (for LOCA initiators) and the occurrence

of significant in-plant fires.

2 Given the sparseness of the data, the analyst is usually forced to merge the data from several plants together and to group 1 similar (but not identical) components into generic classes so 1 i as to pool their data. Engineering judgment is required to l 1 determine the applicability of the data and to perform the i i aggregation of the different sources of data into the generic  !

l groupings, i

J 1

J

Even in the case of the simplest type of data--component failure rate in time or per demand--true "data" in the sense of a set of measurements of the quantity in question does not exist.

Typically, in the literature, one has a record of a number of components failing in a given span of years. From this the failure rate may be estimated by computing the ratio Number of Component Failures J

• Hours of Exposure or Number of Demands

• The data are often taken at a number of different plants and on components in different systems having different operating environments, test intervals and demand histories. Typically, ,

the denomination is not known precisely and in many data sources I

engineering judgment has been used to determine reasonable average exposure times, demand histories, etc.

Thus, in addition to a component type's inherent (irreducible) 1 variability in failure history

• due to randomness in materials '

used in construction, tolerances in part sizes, etc., we also have variability which is due to data source differences:

• Plant-to-plant differences (type, location, climate, mode of plant operation, age of plant, vendor, etc.).

t

• In-plant differences (component manufacturer, age of I component, location in plant, mode of usage during ,

operation).  !

e Generic grouping, j i

e Mode of failure (electrical vs. binding vs. leakage, i etc.)

, This type of variability is often termed systematic, because we l 1 could (in theory) reduce or eliminate this variability given j enough time and money for testing or monitoring the particular  !

component of interest. However, this variability becomes a source of uncertainty when data from several sources are applied to the analysis of a particular plant, as is the case with RMIEP.

In practice, for failure data on nuclear power plant components, 3 one must use the existing data with its systematic variability, l and the questions become: '

i

• It should be remembered that this variability is accommodated l J by the treatment of failures as random occurrences. The adop-  !

l

' tion of a constant failure rate or constant failure probability  ;

is equivalent to making an assumption about the appropriate l probabilistic model to describe the failure history. f t

I

i r

s

a. How to characterize the uncertai.nty in order to reflect  :

l the systematic variability,

b. How to reconcile generic data (from many sources) with

, more limited plant-specific data (if available).

The specific choices one makes in answering these questions will play an important and often limiting role in determining a '

suitable method for performing an uncertainty analysis. Some of the considerations involved are discussed in more detail in Mosleh5 and Parry.6 One resolution, which has been adopted in several PRAs to date, c is to define a generic group of components for which the failure rate with time or on demand are assumed to be a fixed value.

Then, data from other plants are used to derive a probability ,

distribution that reflects the plant-to-plant variation in the value of the parameter for the equivalent group of components. ,

This probability distribution, which represents a measure of physical (systematic) variability can be regarded as a con-fidence interval measure for the parameter value for a particular plant of interest, under the assumption that the plant is one of the general population that has been sampled.  ;

Thus, for example, in the WASH-1400 study,7 the data on valve  ;

failure rates from 10 different sources were plotted in l i

cumulative distribution form to obtain the lognormal t distribution shown in Figure 2.

l ,

When no plant-specific data are available, the generic prob-ability distribution can be used unmodified. When plant- i specific data are available, the generic distributions can, in -

the Bayesian method, be modified, using it as a prior distribu-tion, and utilizing the plant-specific data to specify the c This is discussed, for example, in likelihood (see later).

Apostolakis et al.8 and Kaplan.9 This use of generic data .

has, important implications for the propagation of uncertainty measures as discussed in Section 111.1.3 - Incorporation of i Correlation, j It may also be desirable to include other sources of variability  :

in deriving the probability distribution. For example, the distribution may be chosen to reflect both plant-to-plant and  !

system-to-system or component-to-component variability. This i distribution should, however, be used to represent uncertainty I in the component's failure characteristics only if there is no l consistent system-to-system variability at all plants. If there is a significant consistent system-to-system variability, the components from different systems should not be grouped into the same population, but rather each system should be treated sepa-rately by constructing a plant-to-plant distribution for com-ponents of that system.

. _ _ . _ _ ~_ --_______ _ _ , - _ - .__ .__ ___..__...__ _ ___-__- . - - -

s l

l l

i

\

l 8

I I l I i i

I I I I I l l l

"- l I

• l =

l I I YJs',' '

I i l

l

.- l i l l l l l l

.- l l l l l J j

[ l

.- 1 o I ai a- l  !

t i 3

l I l 1 l l 1 i l l > 1 I

l 1 i_ l i 1 1 I I l i i i illi i i i iliti i i i iiilt ii4 s ait i t i I

l l

Figure 2. Lognormal Distribution--Valves (from Reference 7)

II.2 Methods for Propagatino Component Uncertainty _Through PRA Models Several methods have been proposed for the propagation of uncertainty on parameter values through the PRA analysis, to provide uncertainty measures on the PRA results. Of the methods discussed below, the first is directed toward the classical statistical approach, the others are appropriate where a proba-bility distribution is used to characterize uncertainty and thus can only be used in conjunction with a Bayesian approach.

II.2.1 Maximus/ Bounding Approach The Maximus/ Bounding method by Spencer and Easterling (Refer-ence 10) has two components to its propagation of parameter uncertainty through fault tree models: statistical confidence interval estimation (as discussed in Section 11.1.2), and a bounding analysis. Data on component failures are divided into two categories. Data derived from designed experiments are classified as obiective data. The contribution to top event uncertainty due to parameters based on objective data is modeled by a confidence interval estimate calculated using the Maximus technique. Parameter estimates that do not fit the objective category (e.g., expert opinion) are classified as subiective. Uncertainty due to subjective parameter estimates is exhibited by a bounding analysis and a seven point pictorial format.

The bounding analysis is accomplished by selecting a range of values for each subjectively defined parameter. The accident sequences are then quantified for the following three cases--all subjective parametets are fixed at their low estimate, nominal estimate, and high estimate. The uncertainty of the objectively estimated parameters is propagated through the sequence model (in each case) using the Maximus method for each set of sub-jective parameters.

The Maximus method of confidence interval calculation is an application of the tinstrom-Madden theoremll and applies to failure data that art binomially distributed. In essence, for each series or parallel group of components, rules are pre-scribed from which "equivalent" binomial data (n-failures in a tests) can be derived. From this equivalent data, approximate binomial confidence limits for the accident sequence are derived in the usual way, using equations (3).

Note that, in application, a reliability block diagram of the system must be generated, as the method cannot be applied directly to fault trees or cutsets. Further, the algorithm applied to serial components is exact, while the algorithm applied to parallel components is heuristic, but has been tested over a wide variety of situations. Finally, it should

be noted that the results of the Maximuc calculation depend on the set of values which have been selected for the subjective parameters.

After the accident sequence confidence intervals have been determined for each set of subjective parameters, the Maximus/

Bounding approach provides three different perspectives of the

top event uncertainty; subjective uncertainty, statistical uncertainty, and overall uncertainty. The subjective uncer-tainty le calculated by f ix) ng all objective parameter values

at their nominal estimates based on the data. The range of the top event value due to uncertainty of the subjective parameters is calculated by varying the subjective parameter values from their low values to their high values. The statistical confi-dence limits of the top event are calculated by fixing all subjective parameter values at their nominal estimates. A Maximus calculation is then performed using the objective param-eter data to calculate a confidence interval. The overall uncertainty is estimated by setting the subjective parameter l values fixed at two different values, their low value and their I high value. At each set of flxed subjective values, a Maximus  !

calculation is performed on the objective parameters. Recall from earlier discussion that the Maximus solution will vary for different values of the subjective parameters. An example of the three types-of uncertainty intervals estimated by Maximus/

Bounding is shown in Figure 3.

Comparisons of the Maximus/ Bounding approach with a Bayes'/

Monte Carlo approach (using assumed distributions) for a typical pRA systems model have shown12 that the Maximus approach leads to wider confidence bounds (as expected) but that the upper bounds of both methods are in reasonable agreement.

The advantages of the Maximus/ Bounding approach are:

1. No assumptions need be made concerning the distri-butional form of the basic event probabilities, since failure rates are assumed to be constant parameters, j 2. Once the systems models are in series / parallel block diagram form, the method is straightforward and simple to apply.

Disadvantages of the method are:

1. The typical pRA systems model in fault tree format must

be converted to a block diagram format.

1

2. A certain degree of arbitrariness exists when generating "equivalent" data for groups of like components.

3. The Maximus "rules" for parallel configurations cannot l be proven with mathematical rigor, although extensive demonstration calculations substantiate their heuristic foundations. ,

Point Estimato l

l l

1 i

< z Subjective Statistical 4 m--

Overall Uncertainty i

l 1

l Figure 3. Illustration of Maximus/ Bounding Intervals 11.2.2 Method of Moments J

The Method of Moments propagates the basic event uncertainty through PHA models by generating lower order moments, such as the mean and variance, of the distribution of the top event i from the moments of the basic event distributions. In essence, one is using a Taylor series expansion of the variation of the top event value with respect to variations in the independent

] variables (the basic events). The coefficients of the Taylor j'

series are related to the moments of the distributions of the basic events, a

in theory, if sufficient terms of the expansion are generated, the distribution of the top event can De determined completely.

In practice, only a few lower ordered terms are generated. For example, if one has estimates of only the means and variances of the basic events, one can approximate the mean and variance of the distribution of the top event. However, with the Method of Moments, once the top event attributes (mean and variance) have been calculated, it is necessary to make an assumption about the form of its distribution so that other attributes, such as 95th and 5th quantiles, can be estimated. A more detailed treatment of this method is covered in a comparison study of various uncertainty analysis methods.13 Techniques >

also exist that include the use of higher order moments as i

well.14 Advantageous of the Method of Moments are:  !

1. The mean and variance of the top event can be approxi-mately determined even if only means and variances of

, the basic events are known, rather than their complete  !

I distributions.

2. For simple top event functions with derivatives that [

can be determined analytically, the Method of Moments  ;

is simple to apply.

Disadvantages are:

1. For functional relationships between the top event and the basic events which are significantly nonlinear, the use of only lower-ordered terms in the Taylor series expansion can lead to significant inaccuracies.

2. For complex functional relationships (Or which analyti-cal derivatives cannot be easily obt'ined, some form of experimental design is required to nt irically determine

, the coefficients of the Taylor-series expansion.

l

3. For large systems models of the type usually encountered t

, in PHAs, this approach has been found to be unwieldy in ,

application. I 1

16 I

11.2.3 Method of Moments Using Tchebyschev's Inequality In this technique, the mean and variance of the top event dis-tribution are calculated from the basic event moments as in Section 11.2.2. However, to arrive at uncertainty intervals.

Tchebyschev's inequality is used instead of assuming a form for 1 l the top event's distribution. Tchebyschev's inequality is a l theorem in probability theory which states, "for ADI distribu-tion with finite mean and variance, at least (1-k-2) . loo percent of the probability is in the range of ie k around the 1 mean," (Reference 4). Thus, for example, this inequality implies that at least 89.9 percent of the probability lies in i the interval of u 1 3o. Such tolerance intervals calculated

! with Tchebyschev's inequality tend to be very conservative. i

)

j II.2.4 Propagation of Discrete Probability Distributions Propagation of Discrete Probability Distributions (DPD) is a I

' technique in which the basic event distributions are discre-tized, and a discrete analog of the top event distribution is calculated. Each distribution is divided into some number of intervals, say n. A value for each basic event probability is chosen for each interval (usually the mean of the basic event probability in th,t interval), and the probability that the basic event probability occurs in each interval is calculated.

! Thus, the distribution of each basic event probability is ,

2 discretized into n values, each value with a corresponding

probability. The top event is evaluated nk times where k is l the number of independent variables. The result of each evalua- i j tion has an associated probability equal to tne product of the  !

l probabilities of the independent variables. Thus, a discrete ,

probability distribution can be constructed from these values.  !

J l The DPD technique is, in theory, a valid method for functions l of many variables. However, the technique can become quite l cumbersome with respect to computer storage, and the propagation i l must be performed in stages with intermediate re-discretization, j Incorporation of correlation is then quite awkward. More dis-

! cussion of this method can be found in Reference 15.

1  !

II.2.5 Monte Carlo Method i ,

l The Monte Carlo approach is the most fundamental approach to q uncertainty analysis. Monte Carlo simulation consists of making j repeated quantifications of the top event value. For each

quantification or Monte Carlo run, each random variable basic j

event is sampled using a random number generator to select a value from the basic event's distribution. This procedure is '

s repeated numerous times, and the various top event outcomes are  !

! sorted to obtain empirical estimates of the desired top event  !

j attributes such as mean, median, 95th, and 5th quantiles. l I

I I

! t

As more and more Monte Carlo samples are made for a top event equation, the precision of the empirical quantile estimates of the top event distribution improves. Ilowever, the rate of convergence to the true distribution tends to decrease as the number of samples increases.13 Of course, the validity of the result is only as good as that of the input.

i l

Advantages of the Monte Carlo method are:

1. Complete flexibility in the selection of basic event distributions, i 2. Any specified precision of the top event attributes can be achieved (limited only by cost and cour. doff-error).

3. The method is reasonably easy to implement.

Disadvantages of the Monte Carlo method are-  !

l

1. Probability distributions must be defined for all )

parameters.

2. Computer costs can become significant for large problems.

3. Sensitivity measures of the basic event uncertainties contribution to the tot event uncertainty are not directly obtained.

11.2.6 Latin liypercube Sampling Latin hypercube sampling (Liis) was developed to improve upon the accuracy and precision of Monte Carlo simulations in estimating functions of multiple random variables.16 L}{s is a stratified sampling technique, where n different values are selected for each basic event. The values are selected by dividing each basic event's probability distribution into n intervals, each of equal probability. Within each interval, one value of each basic event's probability is randomly selected. The n values of a particular random variable are then combined (in a manner which keeps the pair-wise correla-tions near zero) with the n values of the other basic events.

The result is an n x k matrix (k is the number of sampled basic events) where the ith row of the matrix contains specific values of each random variable to be used in the ith run or calculation of the top event value. The top event's distribu-tion and quantiles are empirically estimated from the n runs.

Lits has the same advantages as associated with Monte Carlo methods. Additional advantages include:

I l

1. The accuracy in modeling the top event uncertainty is improved with respect to direct Monte Carlo methods for the same number of samples.

2. Use of Liis ensures that thu tails of the basic event distributions are included in the propagation of uncer- I tainty.

Disadvantages of the Lits method may be:

1. Possible bias in result, particularly for small sample  !

sizes.

2. 12 not suitably automated, this procedure can be cumbersome and time-consuming.

3. Being a relatively echnique, all of its properties may not be f ully un: <

' sod.

11.3 Method Recommended for 9MIEP l Following a review of the various methods for uncertainty propagation, and in view of the recommendation that uncertainty in parameter values should be represented by di .e;butions (as discussed in 11.1.3), an approach based on U. in hypercube ,

sampling (Liis ) and Monte Carlo simulation is to. aendad for '

RMIEP. This approach was chosen for the following reasons:

1. This approach can conveniently accept any form of distribution for each basic event, and thus the effect of different distributional assumptions could be assessed.

2. This approach can be applied directly to the output of typical fault tree analysis codes (e.g., SETS, FTAP,  !

etc.), i

3. This approach allows a direct assessment of the effects of correlation between uncertainties for components in l generic categories. l

4. The approach has been found to be relatively simple and inexpensive for large systems models.

The choice of Lits as the method for propagation of uncertainty will allow flexibility in characterizing basic event uncer-tainty. Any method of basic event uncertainty characterization which establishes a distribution for the failure probability j can be used in conjunction with Liis. '

i 1

1 l

Ill. ELEMENTS OF RECOMMENDED' APPROACH FOR UNCERTAINTY PROPAGATION The Latin hypercube/ Monte Carlo approach recommended for HMIEP will be implemented using the Top Event Matrix Analysis Code (TEMAC).17 This computer code . can accept the samples gen-erated from either an LHS routine or a Monte Carlo simulation.

III.l. Propagating Uncertainty Through RMIEP Models The following sections describe the codes which are recommended for use in the Latin Hypercube-Gampling procedure for propaga-tion of uncertainties through the RMIEP systems models. Code output capabilities and sensitivity measures computed are also described.

I11.1.1 Latin Hypercube Sampling The TEMAC code requires a set of sample vectors chosen in accordance with the stratified sampling approach. A Latin Hypercube Sampling (LHS) routine 16 is used to generate the input sample vectors by sampling basic event probabilities from their distr'butions. The output of the LHS will be read directly into TEMAC, which is designed to take simulations (either pure Monte Carlo or Latin hypercube samples) of paramete' values and quantify a top event equation. The LHS and TEMAC codes permit flexibility in uncertainty modeling by allowing for different probability distributions to be used in the same computer run. This eliminates the necessity of assuming all random variables have the same type of distribution.

An introduction to the LHS technique, with a brief discussion of the theory of LHS and a simple example demonstration is included in Appendix B. The example and discussion are taken directly from Reference 16, the LHS user's guide.

111.1.2 Quantification of Top Event Attributes Oace the Latin hypercube sample matrix has been generated for the basic event probabilities, it is read into T F,M A C , along '

with the top event equation. TEMAC evaluates the equation for each sample vector in the matrix.

TEMAC generates a variety of attributes of the top event's distribution as a result of the propagation of basic event parameter uncertainty through the model. These include esti-mates of the top event's mean, median, standard deviation, 5th and 95th quantiles and a nominal estimate of the top event's value calculated with the nominal estimates of the base events.

Four importance measures of basic event contributions to the top event's value are calculated--partial derivative of the top event equation with respect to each basic event, risk reduction, L

risk increase, and a measure of an event's uncertainty contribu-tion to the variance of the top event distribution called uncer-tainty importance. For each measure, the basic events are ranked in order of contribution. These measures are briefly described in Section I11.1.4.

111.1.3 Uncertainty Correlation in Sampling It is recommended that uncertainty correlation be included in the LHS samples. The need for this is discussed below. When components are grouped together into a single generic category, the assumption is made that each component in that group has the same failure rate or probability. Thus suppose that an accident sequence ACC is given in terms of basic event proba-bilities as ACC = ala2 + alb 2 + a2b1+bbi2 e (4) where at is the failure unavailability of component i of the group identified by the letter ai and similarly for bi.

Since each component within the group has the same unavaila-bility, ACC can be rewritten as ACC = a2 + 2ab + b2 , (5) where a represents the unavailability of a component in generic group a and similarly for b.

In performing an uncertainty analysis, it is equation (5), must be used to evaluate the probability distribution on ACC. This gives different results from using equation (4), as can be seen from taking the expected value. In using the first expression, the ai are treated as independent so that E(ACC) =E2(a) + 2E(a)E(b) +E2(b) , (6) whereas in the second it is E(ACC) = E(a2) + 2E(a)E(b) + E(b2) , (7)

I where E(x) is the expected value of x. l The necessity of including the so-called uncertainty correlation was discussed, in a Bayesian context, by Apostolakis and Kaplan l l

4

in Reference 18. They pointed out that the values of the al and a2 (and bl and b2) must be regarded as being the same since the analysts state of knowledge about them is identical.

Thus in a Monte Carlo analysis, when in one sample a random valua from the probability distribution on al is chosen, the same value must be used for the value of ag, and so on. It

-is in this sense that the values are correlated.

Consideration of uncertainty correlation can be important when screening is performed. The fact that the expected value of ACC using equation (5) is greater than that using equation (4) means that extra care has to be taken when using point estimates directly in the equation for ACC as a screening method for identifying dominant cut sets. This is particularly important when there are more than two components from a single group in a cut set and/or the uncertainty on the value of the component availability is large, when E(an) >> En(a) .

l It may thus be desirable, unless it is easy to identify in a general way the possibility of such cut sets, to accept some sort of second screening where, for example an is replaced by a whenever the former appears in a cut set, to guarantee saving the potentially troublesome cut sets. This would mean, in the current example, calculating the unavailability (for screening purposes) as ACC = a + 2ab + b .

Any cut sets that reviewed this second screening would also be included in a full uncertainty analysis, (expressed of course in their original form).

For the final quantification and uncertainty analysis, the TEMAC code can incorporate this uncertainty correlation directly, since the Latin hypercube sampling program permits the specification of full correlation between arbitrary numbers of variables in t.h e top event Boolean equation. (In fact, these codes permit the spccification of any degree of correlation less than full correlation also.) It is recommended, however, that full correlation be used for failure rates of components in the same generic category (and same failure mode).

111.1.4 Sensitivity and Importance Measures Computed in TEMAC Th. TEMAC code calculates various measures of the sensitivity of the top event estimates to changes in the estimates of the basic events. These measures enable the basic events to be ranked aa _to their contribution to the top event values and enhancr. intarpretation and comparison of the PRA results.

In TEMAC, the mathematical representation of a Boolean equation consisting of m cut sets, each of which consists of k events, is written as m a a a 5 5 g(X) = E X 1

X 2

• k , (8) j=1 ith event, contained where ai) = 1 if the Xi, is in the jth cut set, otherwise aij = 0.

The basic event Xi may be either a component failure rate de-scribed in probabilistic terms, or an initiating event repre-senting a frequency of occurrence and expressed as a certain number of occurrences during some time period. Thus, g(X) expresses the frequency of the top event.

The sensitivity and uncertainty measures available are computed (for each simulation) from this equation as described below.

Partial Derivative The partial derivative is a measure of the sensitivity of a function to changes in a parameter's value at a specific point. The partial derivative of the top event g(X) given with respect to Xi is m a a ag(X)/8X X 5 E X ***

g = j=1 1 2 (9) a g) X)ij~ ... X k)

• k  !

Note that the partial derivative expressed above will never contain the variable Xi since a value of ai3 =

0 eliminates the term from the partial derivative and a value of aij = 1 generates an exponent of zero for Xi.

___-__-._________________m

Risk Reduction

• Risk reduction is a measure of the maximum potential reduction in the top event value if a basic event failure

). rate is reduced to zero. Risk reduction identifies com-ponents for which improved reliability could yield signi-ficant improvement in risk. Improved concepts in system-designs and operator procedures are types of insights possibly gained here.

The measure is calculated by setting the nominal value of an event, Xij, equal to zero in g(X) and subtracting the resulting value from the nominal evaluation of g(X):

Risk Reduction for Xj = g(X) - g(XlXj = 0) . (10)

Risk Increase

• Risk increase is a measure of the maximum potential in-crease in top event frequency if a basic event's proba-bility increases to one. Risk increase yields insights of critical components and procedures which, if allowed to become less reliable or effective, could result in a worsening of the risk profilu at a plant. The importance of improvements in preventative measures, such as maintenance and testing, are types of insights derived from increase measures. The measure is calculated by:

Risk Increase for Xj = g(XIX) = 1) - g(X) . (11)

Uncertainty Importance Uncertainty importance, a new measure, is a measure of the significance of a basic event's uncertainty to the uncer-tainty of the top event. This measure gives insight into which basic event probabilities, due to their own uncer-tainty, are driving the uncertainty of the top event estimate. Areas of need for improved data analysis or increased data gathering can be identified by this measure. Uncertainty importance, UIj, for event Xj, is calculated as:

UIj = [ Var (X3)]l/2(ag(X)/8Xj) . (12)

• The risk increase and risk reduction measures in TEMAC are in-terval estimates. Ratio estimates can also be defined and are useful in certain applications. See Reference 19.

Note that each of these sensitivity and uncertainty measures are computed for each simulation, so that sample distributions are also available for each of the sensitivity measures as well as for the Boolean equation being evaluated.

The risk measures calculated by TEMAC provide a set of measures which enhance resolution of safety issues from various regula-tory perspectives. Insights to design improvements, operational improvements, and critical factors for reactor integrity and accident consequences can be gained from the TEMAC output.

1I1.1.5 Inclusion of Modeling Uncertainty As discussed in Section I.1, modeling uncertainty is that which arises because of the existence of plausible alternative models or hypotheses concerning the basic events being analyzed.

Thus, for example, one might wish to include the uncertainty due to the fact that two alternative models for a reactor coolant pump seal failure exist, and it is not possible to exclude either model due to limited experience with the phenomena.

Incorporation of such modeling uncertainty can be directly ac-complished with the Latin hypercube sampling routine and the TEMAC code, provided one can assign a relative likelihood to each alternative. Thus, for example, if a modeling alternative consists of two hypotheses A and B, and if hypothesis A is assigned a 10 percent likelihood and hypothesis B is assigned a 90 percent likelihood, then in choosing N vectors of samples for the TEMAC uncertainty analysis, the LHS routine will select 0.lN of the vectors utilizing hypothesis A, and 0.9N of the vectors utilizing hypothesis B. More generally, the LHS routine has the capability of considering multiple (more than two) alternatives for a given basic failure event, provided relative likelihoods are prescribed for each alternative (and, of course, these relative weightings sum to unity). By this method, the uncertainty in the top event will include both uncertainty in the basic event parameters as well as uncer-tainty due to alternative modeling parameters. In addition, for this simple case, regression modeling can be used to rank the various modeling issues with respect to their impact on the top event, and to rank the alternatives within each issue. An example as iication of this approach is given in Reference 20.

In the more general case, the LHS sampling code and the TEMAC code can, in principle, be used to incorporate modeling uncertainty into the distribution of the accident sequence frequency (or other top event of interest). To illustrate, consider a simple accident sequence involving these basic events:

ACC =CCC3 I 2 . (13)

Each of the basic events has an associated uncertainty dis-tribution. Modeling uncertainty can enter in if there is significant uncertainty with respect to the parameters of the associated distribution, say the mean. Thus, for example, due to lack of knowledge of the physics of the situation, the mean may be assured to take on one of two discrete values correspond-ing to two different failure modes. Further, based on experi-ence on expert judgment we may associate a discrete probability with each of the two alternatives. The existence of two (or more) plausible alternatives (with associated probabilities) will result in greater uncertainty in the sequence frequency.

A second source of modeling uncertainty results when there is uncertainty in the Boolean expression for the accident sequence, due to uncertainty in the success criteria for safety systems.

Thus a plausible alternate to the sequence might be ACC =C1+CC3 2 (14) and again, we must associate some probability (degree of belief) with the validity of each of the plausible Boolean expressions.

Again, this uncertainty in modeling the success criteria will increase uncertainty in the accident sequence.

These two types of modeling uncertainty can, in principle, be modeled using the LHS sampling code and the TEMAC code. Thus we can form the "super" sequence ACC* =ACCC3t 2 + (1-A) [C1+ CC3] 2 (15) and assert that the new variable A takes on the value 1.0 a specified percentage of the time, and a value of zero for the remainder of the time. In effect, the variable A .becomes a switch characterized by two deterministic values with associated probabilities of occurrence.

To incorporate the modeling uncertainty associated with the variables C1, C2 and C3 one must make multiple LHS sample code runs to generate the sample vectors for input to the TEMAC code. Since there are (in this case) two levels associated with each of three variables, there are 23 = 8 distinct combina-tions to be considered. Each combination has an associated probability determined from the probability of occurrence of the basic event "*tevels" themselves. Thus if Prob (Cf") =P ty

Prob (Chigh) ,p Prob (Chigh) ,p Then, assuming independence of the levels, high Prob and C and C high) ,p p p (CfU" if, however, the levels of the variables are not independent, then the degree of correlation must be specified. Thus, for example, if the "high levels" of variables C2 and C3 always occur together, then they are fully correlated (p=1) and high and Chigh) = Min (P g. Pg)

Prob (C rather than equal to P22 P32 as in the independent case.

Any degree of correlation can, in principle, be considered.

In any case, the probability of each distinct combination of levels can be computed, and the total number of LHS sample runs is subdivided in properties to the computed probabilities of occurrence of the varicus combinations.

As an example, consider each of the two accident sequences to be equally likely and the level of basic event C1 to be independent of C2 and C3, but the events C2 and C3 to be fully (positively) correlated. Further, assume the levels of each of the three events to be equally likely. Thus Prob (Cf") = 0.5 Prob (C ) = 0.5 U"

Prob (C and C U") = 0.5 Prob (C and C ) = 0.5

" and Chigh) = 0.0 Prob (C

and hence we.have four distinct combinations

" U" Prob.(C and C and C U") 4 0.25 high Prob and C and C ) = 0.25 (CfU" high Prob (C high . and C and C high) = 0.25 9 and-C 2 Prob (C and C 3

) = 0.25 Thus, if we intended to generate 1000 LIIS samples, we would  !

generate samples of the bas.ic events C1, C2, C3 according j to the relative proportive given in Equation 2. That is, 250  ;;

sample vectors with random variable Ct sampled from its distribution with the low mean, but variables C2 and' C3 sampled from their corresponding . means set at their high mean value, etc.

Further, in this set of 250 sample vectors, the switch A would be treated as a variable, and set to 1.0 in 50 percent of the vectors and to zero in the temaining 50 percent of the vectors.

Finally, the four sets of 250 sample vectors would be input to the TEMAC code, which would evaluate the "super-sequence" in Equation 1 for each sample vector,. and generate an empirical dis *.ribution for the accident sequence frequency.

The above example illustrates how modeling uncertainty can be incorporated in the core damage frequency evaluations. In application, the problem is one of scale, for the number of "lovel" combinations into which the' Lits samples must be parti-tioned is m n where m is the number of levels associated with each variable and n is the number of variables. Clearly, a number of sensitivity runs should be potCormed prior to evalu-ating the empirical distribution of the top event to eliminate as many variables as possible, llowever, the generation of LilS samples for core damage accident sequences is relatively effi- t cient and inexpensive, so that incorporation of modeling uncertainties using the LilS and TEMAC codes is quite feasible.

3 i

i

-28 o

, , . , . . - . _ , . .w - .,__c,.,n_.m,_v.._. ,mw., sy._m_,._,,,_u__,___m w_y__.. _m.- ~-,-#m_.

I IV. DATA AND DATA ANALYSIS .

1 As has been discussed earlier, the methods for performing uncer- i tainty analyses are inextricably linked to the data available in the following, the data needed in performing a Level 1 PRA are described, and current sources of data are outlined.

IV.1 The Parameter Value Estimation Process The development of generic basic event probability estimates and uncertainty is a vital component in performing any uncertainty analysis. Limited operating experience forces the analyst to rely on information compiled from other plants. The data de-velopment process is usually interative, with the initial accident sequence evaluations performed using a set of conser-vative, screening input data. The screening values are then reviewed and adjusted as appropriate for the development of a finalized set of parameter values. The process of developing the screening values and adjusting them to final values is l

discussed in this section.

IV.1.1 Component Definition and Classification of Failures A vital link in developing parameter estimates is the coordina-tion of the available information with the needs of the plant models. At the start of any PRA project, the analyst has a general idea of the level of detail and type of basic events he will incorporate into his system models. As the systems analysis progresses, the nature of the basic events--actual components modeled, failure modes, human errors of interest, recovery actions--become better defined and understood.

However, the available data and generic estimates do not always correspond to the models an analyst may develop. The actual definition of component boundaries and the groupino ef 1' f orma-tion into failure categories can vary between oc a sources.

Component boundary and failure mode definitions are not always appropriate for the data analyst's needs. Problems in finding useful information on certain component failure modes can arise. While many of these component failure modes will be matched with well-documented, well-founded generic parameter estimates, usually several component failure modes will arice 4

that represent areas where little or no data collection has occurred.

IV.l.2 Screening Quantification of Component Failures Once the component boundaries and failure modes are defined, a set of conservatively estimated values are developed for the screening analysis based on a review of published data. A nominal estimate for each component failure mode is made. A conservative or high estimate is also made. In most cases, any ,

difference between nominal and conservative values reflects I differences in published values. In some cases, though, it

might reflect analyst judgment of uncertainty. The conserva-tive values may become a foundation for modeling parameter uncertainty in the final basic event probability models. An initial screening analysis of the sequences is enformed using these conservative screening point estimates. For the screening analysis, no attempt is usually made to determine or select distributions for the parameters. This is undertaken during the finalizing of the parameter value estimates. However, care must be taken so that the screening estimates are sufficiently conservative that components which might contribute to the se-quence' uncertainty analysis are not screened out of the analysis due to their point estimate value. To check this, the analyst should examine those cutsets in the dominant accident sequences whose numerical value is close to the truncation value used in the screening process.

Incorporation of common cause failure events and associated screening values requires additional consideration in that an appropriate conservative model of common cause failure must be chosen. A rather complete treatment of these issues and recom-mendations for screening procedures is given in Reference 21.

IV.l.3 Finalizing of Parameter Estimates and Uncertainty After the initial screening of the accident sequences has been completed, additional data is sought for the remaining basic e event failure or unavailability frequencies, and probability distributions on these frequencies must be developed. The goal is to provide nominal values (usually means), which are best-estimate to the extent possible, and to characterize the uncer-tainty in these nominal values to reflect the full range of sampling, plant-to-plant and modeling assumption uncertainties inherent in the estimates.

Both site-specific and generic data are sought. The generic data sources are reexamined, and the generic failure frequency estimates are refined to the extent possible, based on a closer examination of the failure events in the data sources. The FRAC data attribute analysis code 22 developed at Los Alamos National Laboratory is a tool that can be used to develop com-ponent failure rates applicable to specific reactor types, spe-cific component subgroups or specific failure modes--limited only by the degree of detail in the failure event reporting.

Site-specific data are sought to augment the generic data bases.

In some cases, it may be possible to base a components failure or unavailability rate on site-specific data above. The more common situation is that the site-specific data is utilized in conjunction with the generic failure rate estimates. For failure modes for which little or no data exist, recourse to expert judgment must be made. In any case, probability distri-butions muet be developed for the failure rates.

l l

In the following sections, data needs, data sources and methods i for developing uncertainty distributions from various combina- I tions of site specific data, generic data and expert judgment 1 are described.

l IV.2 Data Needs and Data Types ,

1 For evaluation of accident sequences in the internal event analysis, point estimates and uncertainties are needed for the following:

a. Initiating Events

b. Component Unavailabilities

c. Recovery Terms

d. Common Mode Failure Model Parameters Each of these are discussed below.

a. Initiating Events l

l A variety of LOCA and transient-inducing initiating events are identified for the p' ant under considera-tion. For example, a total of 30 !aitiating events are being treated for the LaSalle plant in RMIEP, as shown in Table 1. The choice of initiating events is based on a review of plant-specific occurrences plus an evaluation of initiating events occurring at similar plants which may be applicable to the site in question, and a detailed examination of plant support systems.

b. Component Unavailabilities Components may be unavailable due to a wide variety of causes. In general, the component unavailability may '

be computed from (some or all) of the following expres- l sions as appropriate:

Qp Demand Failure Unavailability j T I Ks-f Standby Failure KR T m Failure to Run XM TR Unavailable Due to Unscheduled Maintenance i Unavailable Due to Failure to kg P Restore After Unscheduled pg[2 t

Maintenance

)

Table 1 LaSalle Unit 2 Initiating Event Categories Transients (General)

1. Turbine trip with turbine bypass available

2. Turbine trip with turbine bypass unavailable

3. Total main steam isolation valve closure

4. Loss of normal condenser vacuum

5. Total loss of feedwater

6. Trip of one feedwater or condensate pump

7. Inadvertent opening of a safety-relief valve (stuck)

8. Loss of offsite power Transients (Special)

9. Loss of 125 Vdc bus

10. Loss of 4160 Vac bus

11. Loss of instrument air

12. Loss of drywell pneumatic

13. Loss of 100 drywell pneumatic

14. Complete loss of reactor vessel narrow range

15. Loss of Channels A and C of reactor vessel narrow range level instrumentation (false high level indications)

LOCA (Inside Containment)

16. Small LJCA inside containment (s 0.005 square 1 ft. for liquid, 1 0.1 square ft. for steam)

17. Medium LOCA inside containment (0.005 to 0.3 square ft. for liquid, 0.1 to 0.3 ft, for steam)

I Table 1 LaSalle Unit 2 Initiating Event Categories (Concluded)

18. Large LOCA inside containment (1 0.3 square ft.)

19. Reactor vessel rupture LOCA (Outside Containment)

20. Steamline LOCA outside containment LOCA (Interfacing System, Outside Containment)

21. RHR or LPCS LOCA outside containment without isolation

22. CRD LOCA outside drywell without isolation ATWS

23. Turbine trip with turbine bypass available combined with reactor protection system failure

24. Turbine trip with turbine bypass unavailable combined with reactor protection system failure

25. Total main steam isolation valve closure combined with reactor protection system failure

26. Loss of normal condenser vacuum combined with reactor protection system failure

27. Total loss of feedwater combined with reactor protection system failure

28. Trip of one feedwater or condensate pump combined with reactor protection system failure

29. Inadvertent opening of a safety-relief valve (stuck) combined with reactor protection system failure -

30. Loss of offsite power combined with reactor protection system failure

K T gg Unavailable Due to Test t

Outage K P g7 T Unavailable Due to Failure to Restore After Test

/or any specific component, not all sources of unavail-ability are important, but this must be determined for-each component. The parameters are defined as:

Qu = Demand failure probability Kg = Standby failure rate Tt = Average time between tests that would detect the standby failure K r ' Running failure rate Tm " Mission time following demand that component must perform satisfactorily Km

• Frequency of unscheduled maintenance Tr = verage repair time for components due to un-scheduled maintenances P f m ' Probability of failure to restore component to service following unscheduled maintenance Tt = Average time between ter.ts which would detect the failure to restore error K t = Frequency of scheduled testing T t o

• Average test outage time P"ft = Probability of failure to restore component to service following testing T" = Average time between tests which would detect the failure to restore error In general, we need both point estimates and uncertain-ties for each term above.

The demand, standby and running failure rates are de-termined using plant-specific component failure records and/or generic data. The process of combining plant-specific and generic data is described explicitly in Sections IV.4.2 and IV.4.3.

The fcoquency of scheduled testing is obtained directly from the plant test and maintenance procedures. No un-certainty is usually associated with this parameter.

The frequency of unscheduled maintenance is determined by examining plant documents such the control room log and maintenance request records or data from similar plants. These data will define a distribution on the frequency, again often taken in a lognormal form.

The mission time is determined for each accident sequence by identifying the time required for the component to function (following its demand) in order to mitigate the off-normal situation. Often, this is defined explicitly by success criteria for the safety system of which the component is a part. It is usually treated as a parameter with no uncertainty.

The average time to repair and average test outage times are determined from plant records (control room,.

log, etc.). These times are used to define distribu-tions (often lognormal) from which average values and uncertainty measures can be taken.

The terms Tt, Tt, T" define the times be-tween scheduled testing or routine evaluation which would detect the components unavailability due to failure of restoration of the component. These times may or may not be the same. They are, however, ider.t if ied from the plant test and maintenance procedures.

Finally, the probabilities of "failure to restore" are determined both from plant-specific histories of such occurrences, generic models and theoretical models.

These human error probabilities are themselves treated as random variables, with uncertainty distributions assigned to each.

c. Recovery Terms Recovery terms are appended to the (dominant) accident sequence cut sets to reflect the probability that the plant operators will either not correctly diagnose the accident condition and appropriate mitigating response, or will not effectively perform the actions required to l effect the recovery. Models involve simulator data, generic grouping of recovery actions and expert judgment in choosing the "most likely" recovery action that the operator will select for a given accident scenario.

Point estimates and uncertainties are usually provided by the model selected, hence no separate data gathering is involved. For the BMIEP PRA of the LaSalle plant, a new simulator-based recovery model has been developed (Reference 23). Thus the recovery terms used in this PRA will be quite plant specific, although indications are that the new model has wide generic applicability.

d. Common Cause Failure Model Parameters Often, there are important cut sets that involve failure of like components in parallel service condi-tions, i.e., failure of both diesel generators or both emergency station battery banks. The common cause failure of such important components is often expli-citly modeled in the fault tree as a single basic event. The probabilities of these common mode failure basic events are computed using one of several empirically-based models, such as

1. The beta factor model24

2. The binomial failure rate model25

3. The multiple greek letter model26 Each of these models require data, usually in the form of the fraction of multiple component failures as a percentage of the total number of failures.

IV.3 Data Sources There is a vide variety of component failure rate "data" exist-ing in the literature, although the amount of actual raw data is quite limited. Available nuclear component reliability sources may be categorized as folicws.

1. Collections of actual failure events

2. Statistical analyses of data

3. Generic failure rate data bases The analyst must be aware that many sources of component failure characteristics are, in fact, just reanalyses of existing data and thus are not new or independent data sources. This situa-tion must always be kept in mind when reviewing the literature for appropriate "generic" failure rates for use in a PRA. In this section, we will categorize and summarize the more impor-tant sources of reliability data for nuclear power plant components.

As mentioned above, the number of actual data collections (category 1) is relatively small. Historically, the most important in the United States are the Licensee Event Reports (LERs). Summaries of these reports and associated statistics for different component types are contained in reports generated at the Idaho National Engineering Laboratory. More recently, the In-Plant Reliability Data System Program at the Oak Ridge National Laboratory has been collecting and summarizing failure data from U.S. plants in a systematized format. A third source of data is the Nuclear Plant Reliability Data System (NPRDS) produced by the Institute for Nuclear Power Opetations (INPO).

In addition, there have been a number of special Wrpose data collections related to loss of offsite power, anticipated tran-sients witnout scram and diesel generator reliability. The sources of actual failure event compilations are shown in Table 2.

The second category of reliability data sources consist of reports that have analyzed failure event data from one or more of the above sources, and produced data-based estimates of the

Table 2 I

Collections and Summaries of Actual Failure Events Title Source Reference

1. Licensee Event Reports USNRC

2. Licensee Event Report Summaries Idaho National Engineering Laboratory Pumps NUREG/CR-1205 Valves NUREG/CR-1363 Electrical Power NUREC/CR-1362 Circuit Dreakers. Protective NUREG/CR-4126 Relays Initiating Events NUREG/CR-3862 Selected I&C Components NUREG/CR-1740 Control Rods and Drive Mechanisms NUREG/CR-1331

3. In-Plant Reliability Data System Oak Ridge National Laboratory i

Pumps NUREG/CR-2886 Valves NUREG/CR-3154

Electrical Power (Diesel Batteries. NUREG/CR-1362 Chargers and Inverters)

I

4. Nuclear Plant Reliability Institute for Nuclear Quarterly Reports Data System Power Operations 1

I- 5. Reactor Safety Study Section USNRC NASil-1400 l III - LER Data for 1972-1973 i

6. ATNS: A Reappraisal Electric Power Research EPRI NP-2330

Institute l

7. Loss of Offsite Power at Electric Power Research EPRI NP-2301

. Nuclear Power Plants Institute 3,

) 8. Diesel Generator Reliability Electric Power Research EPRI NP-2433 at Nuclear Power Plants Institute l

t

9. Clasolfication and Analysis of Electric Power Research EPHI NP-3967 Reactor Operating Experience Institute Involving Dependent Events

10. PORV Failure Reduction Methods Combustion Engineering CEN-145 i

failure or unavailability rates for different components.

Reports often differ as to assumptions with respect to number of demands, plant down-time or method of statistical analysis, and thus different reports can arrive at different failure rates using the same data base of failure events. Table 3 lists a number of such studies which have been found useful.

They differ from the reports in category 1 in that sufficient information for reanalysis under different assumptions is not available.

The final category consists of compilations of "generic" com-ponent failure rates and associated estimates of uncertainty.

These generic values are usually obtained by review of two or more category 1 or 2 sources, and may also include expert opinion on component failure rates or probabilities derived from other (non-nuclear) industrial experience. Table 4 lists the more important generic data bases in use today. The user is cautioned that these various generic data bases should never be construed as being independent, as in no case is this true.

Further, considerable expert opinion has usually been used in  !

choosing appropriate generic values. l IV.4 Data Analysis Methods Available The techniques discussed earlier are methods of propagating the uncertainty of basic events through ,a system or accident sequence model to estimate the uncertainty of the top event.

Methods of characterizing the uncertainty of the individual basic events of these models in terms of probability distribu-tions are discussed here.

IV.4.1 Bootstrapping ML hod s

This is an approach to generating a sampling distribution for one or more (constant) parameters in a failure model when only the results of a single sample are available. To illustrate this method, consider a binomial failure rate model, with data of x observed failures in n demands. Let p be the observed failure rate (x/n).

Random samples of the failure rate p are generated by randomly choosing a probability value P and solving for a value of y i from the binomial probability distribution equation j P= (p)Y(1-p)n-y .

(16)

Thus, we are using the observed values of n and p to generate random sample values of number of failures y, and thus random samples of the failure rate p* = y/n. From these, a distribu-tion on p can be generated.

l l

1

~ ._

l 1

l Table 3 Statistical Analyses of Data i Title Source Reference

1. Probabilistic Safety Analysis USNRC NUREG-0666' of DC Power Requirements for Nuclear Power Plants i

2. Reliability Data Book Swedish Nuclear RKS85-25 l Power Inspectorate s

w l

' 3. Statistical Analysis of Nuclear Los Alamos NUREG/CR-3650 i Power Plant Pump Failure National Laboratory l Rate Variability-Preliminary l Results i

l In addition, items 2, 3, 5, 7, 8, 9, and 10 of Table 2 present analyses of reported i data.

i l

I I

(

Table 4 Generic Failure Rate Data Bases Title Source Reference Reactor Safety Study USNRC WASil- 14 00 Interim Reliability and Evaluation Sandia National NUREG/CR-2778 Program (IREP) Procedures Guide Laboratories Accident Sequence Evaluation Program Sandia National NUREG/CR-4550 Laboratories i

a Reliability Data Book Swedish Nuclear Power RKS85-25

? Inspectorate Station Blackout Accident Analyses USNRC NUREG/CR-3226

- TAD A-44 I

i 1

This bootstrapping process does yield a sampling distribution on the failure rate parameter which can be used in an uncer-tainty propagation procedure requiring parameter distributions as input. It has been found to lead to optimistic (conserva-tive) results when dealing with components having few (or zero) failures in the data base used. The interpretation of the bootstrap-derived distributions may vary greatly depending on the sources of the data used in constructing the distributions.

The method does not allow the impact of plant-specific data to be segregated and understood as is the case of the Bayesian approach and hence the latter is to be performed in those cases.

IV.4.2 Use of Plant-Specific Data in a Bayesian Analysis In many older plants, data obtained from control room logs, maintenance request and event reports may be used to develop plant-specific estimates of failure rates, either for specific components or for a generic component category. Conversion of these point estimates into probability distributions can be accomplished utilizing Bayesian analysis in two ways depending on the extent of additional (industry wide) data available.

IV.4.2.1 Use of Non-Informative Priors Ideally, sufficient data would exist so that statistical analysis of a specific component's actual failure record could be made so as to yield an experience-based estimate of a component's failure rate. A Bayesian analysis using a noninformative prior distribution (see IV.4.2.2) yields a characterization of the uncertainty in terms of a probability distribution. Such an analysis could be conducted on individual components, or on generic families of components, such as all check valves in a plant. All components or events of the same family would be modeled with the same distribution. Unfor-tunately, it is rare in nuclear power experience that plant-specific data exist in sufficient quantity to produce accurate parameter estimates i.e., narrow uncertainty intervals.

However, the Bayesian technique should be employed by RMIEP whenever data is available in appropriate quantity.

IV.4.2.2 Bayesian Updating of Generic Data Bayusian data analysis allows for incorporation of limited plant-specific data into past experience or knowledge as contained in generic data bases to yield improved understanding or updated distributions of basic event probabilities. Limited plant-specific data implies that some information on a specific component or family of components is available, but not enough for a full plant-specific data analysis as discussed in Section 11.2.1. Bayesian updating utilizes Bayes' theorem which is discussed in Section IV.5.1.

In using Bayes' theorem, a basic event's probability may be modeled by a certain distribution based on past knowledge, perhaps from . previous PRAs or other experience. This is referred to as the prior distribution of the parameter in question. Suppose also that there is some plant-specific data available for the basic event at hand. Bayes' theorem gives us a probabilistic relationship between the basic events prior distribution, the current data, and a new, or updated, distri- e bution. Thi. updated distribution is called the posterior distribution tor the parameter, and represents a revision of the prior distribution based on the available plant-specific data. The posterior distribution is then used in the PRA models.

IV.4.3 Expert Opinion Use of expert opinion or judgment is an approach to estimating failure characteristics based on tapping into unquantified l experience. For example, one might seek failure rate data from personnel in related non-nuclear areas where accurate failure or repair records are not kept. Or one might seek to infer failure characteristics for one component (for which little or no actual data are available) from data on a component of ,

similar design by appealing to the judgment of experts involved in the design and application of the two types of components.

Or, in the case of simple mechanical failures, one might calculate a failure rate based on fatigue limit data for sub-pieces of the component in question.  !!e r e again, expert judgment is used to choose the most appropriate material data and calculational scheme.

The process of assembling expert opinion usually involves literature searches of past relate' experience and formal surveys of "experts" with applicable experience. Often, the results of formal surveys are combined, summarized and returned to the experts surveyed so that clarification or revisions are possible (as in a formal Delphi process). Generic lists of component failure probabilities or failure rates are a typical result of this method, such as printed in the Interim Reliability Evaluation Proqtam (IREP) Procedures Guide 27 or

the NUREG/CR-4550 document.z8 While expert opinion can yield '

good estimates of means and extreme values, assumptions must often be made regarding the shape of distributions. However, the assumptions on distribution shapes are frequently guided by expert opinion. Appendix C contains a summary paper prepared by Los Alamos National Laboratory which describes the results of an NRC-sponsored research program on the quantification of I expert opinion, and presents recommendations for soliciting '

such information.

IV.5 Elements of the Bayesian Updating of Base Event Parameter Attributes 3 The Bayesian method allows for prior knowledge of a component's behavior to be revised with a limited amount of plant-specific data. Experience from other power plants or data sources can be updated with the latest contribution to the data pool from the current component of interest. Use of Bayes' theorem to achieve this update ensures that the rules of probability theory are observed.

Bayes' theorem can be used to generate a distribution of a parameter's value for two different cases: (1) prior knowledge exists, and (2) no prior knowledge exists, or the analyst chooses not to use prior knowledge so as to avoid biasing the analysis. In the first case, a distribution of a parameter's value which reflects the analyst's beliefs about the parameter's possible values prior to collecting the current information is chosen. This distribution is called the prior distribution.

Its form is not theoretically prescribed, but it is possible to select a prior distribution so that the mathematical calcula-tions involved in the updating are not too complex. In the second case, the analyst chooses a "non-informative" prior distribution. In both cases, the prior distribution is used along with the current information on the parameter in Bayes' theorem to derive a new, updated distribution of the parameter's value. This new distribution is called the posterior distribu-tion. The posterior distribution is used in whatever sampling technique is chosen for propagating component uncertainty through the system or sequence models.

IV.5.1 Bayes' Theorem Bayes' theorem provides a medium by which some prior knowledge i

about a random variable can be updated using new data in a manner such that the rules of probability are observed. A prior probability distribution is defined for the parameter value in question. The parameter could be a demand failure rate for standby components, a time failure rate for standby components, or a time failure rate for components which must function for a certain period of time. The prior distribution will reflect the analysts opinion of the components behavior based on published analyses of similar failures at other plants. Licensee Event Reports (LERs), previous PRAs, and generic failure models are examples of sources of information used to define prior distributions. Any plant specific information is then used to update prior distributions.

Bayes' theorem can be expressed as f(Klg) .

b(X b) f (M (17)

JL(xlK)f(K)dk where L(xlk) = likelihood that, given a parameter value K, a set of observations on some quantity x would result in the values given by a vector x.

1 l f(Klx) = posterior distribution on the values of the l parameter K, given that one has observed the values x.

i f(K) = prior distribution on the value of the parameter K.

The form of the likelihood function is determined by the nature of the data, binomial (constant failure on demand model) or poisson (constant failure rate model). The form of the prior distribution is not necessarily prescribed. However, it is often advantageous to select a prior distribution which will simplify the mathematics of using Bayes' theorem. One such approach is to use conjugate priors, as discussed in Section II.1. Examples of applying Bayes' theorem to constant-failure-rate and constant failure on demand models are shown below.

IV.5.2 Constant Failure Rate Model Many component failures are well described by a model that estimates the frequency at which failures will occur. The time period over which a component's failure probability is of interest may be short--the length of time needed to mitigate an accident condition, or long--the time between testing of standby components such as normally open valves. Furthermore, the rate  ;

at which a component will fail can be thought of as having a constant value. Unfortunately, the knowledge associated with the component's behavior is usually incomplete, and the exact value for the failure rate is not known. Data on failures of similar components in other systems and other nuclear plants may yield a probabilistic range for the value of the failure rate. This probability distribution is derived from information on other components of similar design, but different locations, environments, and maintenance histories. Even so, it serves as a basis to model the failure rate of a particular component.  ;

Bayes' theorem provides a mechanism by which limited plant-specific data on a component can be incorporated into a distri-bution characterized by a large body of data pooled from other plants.

IV.5.2.1 Bayes' Theorem with Conjugate Priors Suppose that current experience has shown that a particular component type at a nuclear power plant has failed r times in a total time period of T. The likelihood function for this type of data is:

L(rlT,K) = RT) r exp(-KT) (18) rl .

The prior distribution for the component failure rate, K, '

when combined with this likelihood function in Bayes' theorem, will yield a posterior distribution of X of . similar form as '

the prior is We gamma distribution. Suppose that past experience at other plants shows a failures in a total time period 8. The prior distribution would be a gamma distribution with parameters a and 8:

f(K) =p X"~1exp.(-8K) ,

(19) l where P(a) is the gamma function. The resulting posterior distribution is also a gamma distribution with' parameters a+T and 8+T:

f(Klx) =

+ ) (a4T-1) e xp [- ( 8+ T)K] (20) p .

t The initial model of the failure rate distribution has thus been adjusted to account for the new information at hand.

l While conjugate priors are attractive from a mathematical view- f point, they are only useful when the existing data justifies i the use of a distribution of conjugate form. Mathematical simplicity should not be the primary criterion of selection.

IV.5.2.2 Bayes' Theorem with Non-Informative Priors If no additional information other than limited plant-specific data exists, or if the analyst does not wish to bias the analysis with outside information, a non-informative prior distribution can be chosen. A non-informative prior is merely a mechanism by which plant-specific data can be converted into i

a distributional form, and does not necessarily represent any engineering judgment or experience. As with other priors. l 7

mathematical simplicity can be enhanced with conjugate dis-  !

tributions.

A gamma prior distribution with parameters a = 8 = 0 can be  !

used as a conjugate non-informative prior

• for the constant failure rate model, incorporating r failures in the time period l i T, t

• 1t should be noted that the form of the appropriate non- '

informed prior does depend on the censoring scheme for sampling data as detailed in Reference 29. l r

1 I

f(K) = 1/K .

(21)

The resulting posterior distribution is also a gamma distribu-tion with parameters r and T.

IV.S.2.3 Bayes' Theorem with Lognormal Priors In practice, mathematically convenient priors which are conju-gates of the likelihood functions of the data are not neces-sarily an adequate characterization of past experience. More often, the lognormal distribution is used to represent uncer-tainty. This distribution is not a conjugate with Poisson data, but is very useful for many components. Although the mathematics of Bayes' theorem with lognormal priors is not as simple as with the conjugate priors, simple numerical integra-tion techniques can be used to generate an empirical posterior density function, f(klr.T) from Bayes' theorem:

gr-1 e

-AT exp 1 " ~"

f(klr,T) -2 s (22)

=

x#~1e -xT exp

-f ",*~" dx o

where u = mean of the natural logarithms of the prior data o= the standard deviation of the natural logarithms of the prior data  ;

r = number of failures observed T = component exposure time This expression is easily evaluated. Thus, it is seen that prior distributions need not fit conveniently into conjugate forms in order to benefit from applications of Bayes' theorem.

1 i IV.5.3 Constant-Failure-Probability on Demand Model Many component failure modes are characterized by a constant probability of failure given that the component is demanded to function in the appropriate mode. Examples of such component failure modes are pumps failing to start upon receiving initia- '

tion signals and valves which must change state for system

, operation. Just as with constant failure rate models, the knowledge of a component's failure probability on demand is often incomplete. Limited plant-specific data may not be

.! I I

sufficient to estimate a component's behavior with a high degree of confidence. Information pooled from other sources can be used to derive a probability distribution indicative of the behavior of similar component failures. This probability dis-tribution does not represent exactly the possible range of a particular component's failure probability, since plant-to-plant differences are incorporated into it. However, it does provide a foundation for generating a distribution using the limited plant-specific data.

IV.S.3.1 Bayes' Theorem with Conjugate Priors Suppose the current information on a component's behavior is in the form of n failures in N demands. This is binomial data, and the likelihood function used is:

I L(nlN,p) = p (1-p)"~ (23) n!( n)I l

The conjugate prior for the failure probability, p, for this likelihood function is the beta distribution. If past experi-ence has shown 1 failures in m demands, the prior distribution ,

is: i i

• ~

f(p) =

g)p( p (1-p) (24)

The resulting posterior distribution is also a beta distribu-tion, with parameters 1+n and m+N-n.

IV.5.3.2 Bayes' Theorem with Non-Inf ormative Priors As with constant failure rate models, limited plant-specific data of constant probability on demand can be converted to a  ;

distribution even without the use or existence of other infor- '

mation. Mathematical simplicity can bo enhanced by selecting a ,

non- inf o rmed conjugate prior distribution for the availab'e. I date.

The beta distribution with parameters O and 0 is a commonly used non-informative prior:

f(p) =

(25) pg _p) .

The resulting posterlot distribution is also a beta distri-

bution with the parametets n and N-n.

l

] _ - - _ - _ - - _ _ _ _ - _ _ - _ - - _ _ _ _ _ _ _ _ _ - _ _ _ _ - _ - _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ -

IV.5.3.3 Bayes' Theorem with Lognormal Priors Just as with constant failure rate models, conotant demand probability information often can not justify the use of conjugate priors. The lognormal distribution has been used frequently to represent uncertainty .in practice. Numerical integration techniques can be used to solve for an emp,tical posterior density function, f(p), from Bayes' theorem:

p*~1(1-p)n-x gxp l ,.

_g '

~

f(plx,n) = 1- ~_pl- (26) y*-1 ( 1- y) n- x gxp , _

g dy where j u = mean of the natural logarithms of the prior data o = standard deviation of the natural logarithms of the data x = number of failures observed .

q n = number of demands Again, this expression is easily evaluated. Thus, again it is 1

seen that there ic no need to sacrifice accuracy of models for mathematical simplicity.

IV.S.4 Development of Prior Distributions from Subjective Opinion In the above two examples, the parameters of the prior distribu-tions were assumed to be from past information on component failures. However, methods exist for incorporating subjective opinion of component behavior in lieu of data.30 For example, suppose our prior knowledge of a component's demand rate, p, is such that we believe there is only a five percent chance that the value of p will exceed some value, p*, A mean value of p is also assumed to be known. This pair of quantile estimates of p can be used to generate the parameters of an assumed prob-ability distribution. Tables exist which relate these quantile estimates to appropriate values of parameters for distributions  :

such as the beta and gamma.31 Thus, an analyst's subjective opinion of an event's probability attributes such as mean and 95th quantile can be used to generate a prior distribution for the event's failure probability.

i e

J i

1 l

i V. CONCLUSIONS This report has presented the results of a brief review of available uncertainty analysis methods and deteribed the Latin Hypercube sampling approach (as embodied in the TEMAC code) that is recommended for use in the RMIEP probabilistic risk '

analysis of the LaSalle plant. One important aspect of this approach, which was an important . consideration in its selection, is its comprehensive sensitivity measure option. It should be remembered that uncertainty propagation is, of itself, of limited interest since it is essentially an integration function. The ability to perform sensitivity studies through the use of the importance measures and the l

ability to identify the parameter estimates whose uncertainties ,

l contribute most to the uncertainty in the end result are a key l feature of the approach adopted here. In addition the method treats modeling uncertainty in a manner compatible with that >

proposed for the level 2 and 3 parts of the analysis. Thus the approach to be adopted has the potential for enhancing the understanding and analysis of the results ot the Level I PRA both as a stand-alone analysis and as input to a Level 2 and 3 PRA.

The TEMAC code has recently been successf1llly applied in an uncertainty analysis of the Peach Bottom plant as part of the Accident Sequence Evaluation Program.32 This application  ;

showed that this approach is both effective and relatively inexpensive with respect to computer cost. Based on the above considerations, the use of the TEMAC code and Latin hypercube sampling was deemed the optimal analysis tool for use in RMIEP.

l l

i I

-49 j

VI. REFERENCES

1. Kolb, G. J. "Overview and Status of the LaSalle ' Nuclear Plant Probabilistic Risk Assessment," Proceedinos of the 8th International Structural Mechanics in Reactor Technolooy Conference, August 16-25, Brussels, Belgium, 1985,

2. Helton, J. C., et al., "Detegration of the NUREG-1750 Analysis: Mlculation of Risk and Propagation of Uncertainties,' paper prescribed at 14th Water Reactor Safety Information- Meeting, October 26-30, 198/,

Gaithersburg, MD.

3. Party, G. W. and Winter, P. W., "Characterization and Evaluation of Uncerta!nty in Probabilistic Risk Analysis,"

Nuclear Safety, Vol. 22, No. 1, 1981.

4. Hahn, G. J. and Shapiro. S. S., Statistical Models in Encineerino, (New York, NY: John Wiley & Sons, Inc.,

1967).

5. Mosleh, A., "Hidden Sources of Uncertainty: Judgement in Collection and Analysis of Failure Data," Nuclear Encineering and Desion, 9 3_ , pp. 187.

6. Parry, G. W., "Incompleteness in Data Bases: Impact-on Parameter Estimation Uncertainty " Presented at 1984 Annual Meeting at the Socloty for Risk Analysis, Knoxville, TN; September 30-October 3, 1984.

7. U.S. Nuclear Regulatory Commission, "Reactor Safety Study:

An Assessment of Accident Risks in US Nuclear Power Plants," WASH-1400 (NUREG 75/014) 1975.

8. Apostolakis, G., Kaplan, S., Garrick, B. J., and Duphily, R. J., "Data Specialization for Plant Specific Risk Studies," Nuclear Engineering and Desion, ).6,, 1980.

9. Kaplan, S., "On a 'Two-Stage' Bayesian Procedure for Determining Failure Rates from Experimental Data," IEEE Transactions on Power Apparatus and Systems, Vol. PAS-102, No. 1 January 1983.

10. Spencer, F. W., and Easterling, R. G., Use of the Maximus Methodoloav for Confidence Bound Calculations in Fault Trees --

Trial Problem, Sandia National Laboratories, Albuquerque, NM, draft, 1984.

11. Lloyd, D. K., and Liporo, M., Italia bility: Manacement.

Methods and Mathematics, (NJ: Prentice llall, 1962).

12. Hora. S. C., Iman, R. L., A Comparison of Maximus/

Doundino and Bayes / Monte Carlo for Fault Tree Uncertainty Analysis, SAND 85-2939, Sandia National Laboratories, Albuquerque, NM, March, 1986.

13. Martz, H. F. et al, A Comparison of Methods for Un-certainty Analysis of Nuclear Power Plant Safety System Fault Tree Models, NUREG/CR-3263, Los Alamos National Laboratory, Los Alamos NM, April 1983.

14. Jackson, P. S., Hockenbury, R. W., and Yeater, M. L.,

"Uncertainty Analysis of system Reliability and Availability Assessment," Nuclear Encineerino Desion, 8 ,

j_8_

May 29, 1981.

15. Kaplan, S., "On the Method of Discrete Probability Distributions in Risk and Reliability Calculation -

Application to Seismic Risk Assessment," Risk Analysis 1, 189-196, 1981.

16. Iman, R. L. Shortencarier, M. J. A FORTRAN 77 Program and User's Guide for the Generation of Latin Hypercube and Random samples for Use with Computer Models, NUREG/CR-3624, SAND 83-2365, Sandia National Laboratories, Albuquerque, NM, March 1984.

17. Iman, R. L., Shortencatier, M. J., A User's Guide for the Top Event Matrix Analysis Code (TEMAC1, NUREG/CR-4598 SAND 86-0960, Sandia National Laboratories, Albuquerque, NM , August 1986.

18. Apostolakis, G. and Kaplan, S., "Pitfalls in Risk Calcula-tions," Reliability Engineerino, 2, pp. 135-145, 1981.

19. Veseley, W., Measures of Risk Importance and Their Appli-cation. Idaho National Engineering Laboratory, Idaho Falls, ID, NUREG/CR-3385

20. Amos, C. N., et al, Evaluation of Severe Accident Risks and Potential for Risk Reduction: Peach Bottom. Unit 2, NUREG/CR-4551, Vol. 3, Sandia National Laboratories, Albuquerque, NM, February 1987,

21. Mosleh, A., et al., Procedures for Treating Common Cause Failures in Safety and Reliability Studies, U.S. Nuclear Regulatory Commission, Washington, DC, NUREG 4780, 1987.

, 22. Martz, H. F., Beckman, R. J. and McInteer, C. R., 1982, FRAC (Failure Rate Analysis Code): A Computer Program for, Analysis of Variance of Failure Rates, Los Alamos National Laboratory, Los Alamos, NM, NUREG/CR-2433.

i

23. Whitehead. D. W., Weston, L. M., and Graves, N. L.,

Recovery Actions in PRA for the Risk Methods Integration and Evaluation Program (RMIEP): Vol 1: Development of the Data-Based Method Vol 2: Application of _ _ the Data-Based Method, Sandia National Laboratories. Albuquerque. NM.

NUREG/CR-4834, 1987.

24. Fleming, K. N., A Reliability Model for Common Mode Failures in Redundant Safety Systems. General Atomics Report, GA-A13284, 1975.

25. Atwood, C. L., Estimators for the Binomial Failure Rate Common cause Model. Idaho National Engineering Laboratory, Idaho Falls, ID., NUREG/CR-1401, 1980.

l

26. Fleming. K. N. and Kalinowski, A. M., An Extension of the

, Beta Factor Method to Systems with High_ Level of Redun- _

dancy. Pickard, Lowe and Garrick Report. PLG-0289, 1983. ,

l

27. Carlson, D. D., et al., Interim Reliability Evaluation  !

P_rogram Procedures Guide. NUREG/CR-2728, SAND 82-Il00, Sandia National Laboratories, Albuquerque, NM, January 1983.

28. Drouin, M. T., et al., Ana ly_s_i_s_ o f Core Damage Frequency.

from Internal Events: Grand Gulf; Unit l_ . NUREG/CR-4550 S ANDS 6 -,2 08 4 , Volume 5 Sandia National Laboratories, Albuquerque, NM. February 1987.

29. Martz, !! . F., Walker, R. A., Bayesian Reliability Analysis.

(New York. NY: Wiley & Sons. 1982).

I

30. Cook. I. and Unuin. S. D., Controllino Principles for Prior P_robability Assessments in Nuclear Risk Assessments, Sandia National Laboratories, Albuquerque. NM, NUREG/CR-4514.

1986.

31. Abramowitz. M. and I. Stegan, flandbook of Mathematical Functions. Dover, 1965.

32. Kolaczkowski, A. M., et al., s _ of Core. Da tna_g e_ ,F_r_e -

Analysis quency from Internal Events: Peach Bottoma Unit 2 NUREG/

CR-4550 SAND 86-2084, Vol. 4 Sandia National I,a bo r -

atories. Albuquerque, NM. October 1986.

l

-52

~

t 4

l I

e i

Appendix A Interface Between Core Damage Frequency Analysis (RMIEP),

2 the Accident Progression and Containment Failure 4 Analysis (PRUEP) and the Health and Economic Consequence Analysis

• i i i

i I

J i I l e

1 I

l l

l l

4 l

~

A-1/A-2

0 As has been indicated, the uncertainty and sensitivity analyses for the accident sequence and core damage frequency calculations will be performed using the TEMAC code and limited Latin hyper-cube sampling techniques. The analysis of accident sequence frequencies and core damage frequencies is often referred to as the "fcont end"' analysis.

To complete the full scope PRA, it is necessary to characterize the accident progression following core damage, and to compute fission product radionuclide release, and estimate the health and economic consequences of each accident scenario. These calculations are typically referred to as the "back end" analysis, and an uncertainty and sensitivity analysis will also be performed here. Uncertainty analysis methods to be used in RMIEP for the LaSalle back end analysis are still evolving, and work is continuing into 1988.

It is currently planned that the RMIEP front end analysis will be performed and reported as an entity in itself. Then the back end analysis for RMIEP will be performed and reported as a separate entity. Given the planned sequential nature of the front and back end analyses, it was necessary to examine the interfaces between the front and back analyses to assure that no incompatibilities would be present, and that the data needed for the back end uncertainty calculations would be available in the proper form. This section discusses the information flow and compatibility between the front and back end analyses, and demonstrates that the uncertainty analysis data and methods to be used in the front end analysis are both compatible and complementary with those being used for the back end analysis.

Areas are highlighted where careful attention to the interface is required.

Figure A-1 shows a block diagram representation of the four main steps in performing a full scope (Level III) PRA. The first block (accident sequence and core damage analysis) ,

represents the front end calculations (the RMIEP analysis for l LaSalle). The remaining three blocks together constitute the back end analysis (as will be analyzed in the PRUEP program for LaSalle). Each of these blocks of analysis and their interface requirements is discussed below.

Figure A-2 shows the input and output of the front end acci-dent sequence analysis. The inputs consist of the plant system descriptions and operating procedures, plus the failure rate data which was discussed previously in Section IV. The output I consists of the dominant cut sets for each of the accident I sequences defined by the event trees, plus the frequencies and l uncertainties of the accident sequences themselves. Although '

there are a variety of point estimate results, uncertainty analyses, and sensitivity studies which result from the front end analysis, there are really only two pieces of output which are input to the back end calculations, namely.

A-3

~

j ACCIDENT ACCIDENT . HEALTH'AND-SEQUENCE PROGRESSION SOURCE TERM ECONOMIC.

l ANALYSIS ANALYSIS CALCULATIONS CONSEQUENCES 1

i i

I

?

Figure A-1. Major Steps in a Level III PRA Analysis of Reactor l Accidents  :

I h

i

?

t

. A-4 i

ACCIDENT SEQUENCE ANALYSIS: OUTPUTS:

• EVENT TREES

• CORE DAMAGE CUT SETS ,

• SYSTEM FAULT TREES

• ACCIDENT SEQUENCE OR I e llUMAN RELIABILITY ANALYSIS  : CUT SET FREQUENCIES ,

• PLANT STATUS I L INFORMATION i i

CODES: SETS, TEMAC o

i

INPUT DATA & ASSUMPTIONS

4 i

• PLANT SYSTEM CONFIGURATIONS CORE VULNERABLE

• PLANT OPERATING PROCEDURES SEQUENCE TREE i e INITIATING EVENT FREQUENCIES

• EQUIPMENT FAILURE PROBABILITIES

• COMMON CAUSE FAILURE RATES

• ITUMAN ERROR PROBABILITIES

• RECOVERY ACTION PROBABILITIES l

i I

1 i

I i I Figure A-2. Accident Sequence Analysis - Input and Output 1

l A-5

• Core damage cut sets

• Plant ufety systems status information implied by each cut set

]

of course, all the component failure and test and maintenance outage data plus their associated uncertainty characterizations must also be available to the back end analysts. But the only front end result which is passed to the back end is a listing

, of the dominant core damage cut sets. Working together, the front end and the back end analysts will regroup these cut sets into plant damage states.

There is one other aspect of this interface Waich requires further description, and that has to do with those front end accident sequences which are denoted as "core vulnerable" 4 rather than "core damage" states. In these accident sequences, l j the question of whether or not the core is damaged depends on how the accident progression affects the functioning of safety systems operating to prevent core damage. To determine which cut sets in the core vulnerable accident sequencos lead to core 1 damage, an additional small event tree is appinded to these  !

This tree contains phenomenologicil alternatives sequences.

(i.e., does the containment fail, etc.). _his troe and associated split fractions (probabilities) associated with each option will be provided by the back end analysis team prior to

final accident sequence cut set determination by the front end  :

team. .

Figure A-3 shows the inputs and outputs to the accident i '

progression calculations. The accident sequence cut sets provided by the front end analysis are first resorted into '

plant damage states. All cut sets assigned to a given plant I damage state present the same initial and boundary conditions  ;

to the accident progression analysis, and thus are propagated [

together through the accident prooression event tree. Cut sets s from a given accident sequence may be assigned to different l damage state bins. The definitions of the various damage states are determined by the first few (ten to 20) questions on the i accident progression event tree (which is developed for each  ;

i plant) and to some degree, with consideration of the important accident sequences identified in the front end analysis. An example of the questions which define the damage states (taken 4 from the draft Grand Gulf PRA NUREG-1150) is given in Table 1 A-1. Taken together, it was found on the basis of these questions that eight damage states could be defined which would J cover all the dominant internal event cut sets. Definition of these plant damage states requires a significant amount of interaction with the internal event (front end) analysts to determine the plant systems status for each cut set.

l

)

i

ACCIDENT PROGRESSION ANALYSIS: OUTPUTS:

o PLANT DAMAGE STATES DEFINITION ACCIDENT o CONSTRUCT ACCIDENT PROGRESSION o ACCIDENT PRO-SEQUENC E ---*= EVENT TREE ---*- GRESSION BINS CUT SETS o ACCIDENT PRO-GRESSION RIN FREQUENCIES CODE: EVNTRE o

INPUT DATA & ASSUMPTIONS:

o PLANT DESIGN DETAILS o EQUIPMENT SURVIVABILITY o SEVERE ACCIDENT PHENOMENA o EXPERT OPINION o

ACCIDENT PROGRESSION PHENOMENOLOGY CODE RESULTS FROM:

o SOURCE TERM CODE PACKAGE (STCP) o MELCOR o RELAP-SCOAP o TRAC-MELPROG o CONTAIN o HECTR q

figure A *s. Accident Progression Analysis - Input and Output A-7

d' -

Table A-1 Initial Questions on the Grand Gulf Accident Progression Event Tree Used to Define Plant Damage States Questions Prior Question Dependencies

'l. What is the ~ initiating event? None

2. What is the level of preexisting None leakage or isolation failure?

3. What is the level of preexisting None suppression pool bypass?

4. Is there a station blackout (diesel None generators fall)?

5. Is DC power not available? None

6. For TC accident sequence, does None SLC fail to inject?

7. Does HPCS fail to inject? None  ;

C. Does RCIC fail to inject? None  ;

9. Does the CRD hydraulic system inject? None

10. Does the condensate system fall? None ,

11. Do the LPC and LPCI systems fall? 1,4,7,8

12. Does RHR fail (heat exchangers not 1,4,11 available)?

13. Does the service watet cross-tie to 1,4,11 LPCI fail?

14. Are the containment (wetwell) sprays 1,4,11  ;

failed?

15. Is ADS inhibited or failed? None li. Does the RPV remain pressurized? 1.5,7,8,13

17. What type of sequence is this (sum- 1,7,8,16  :

mary of plant damage)?

L i

A-8

The remainder of the accident progression analysis makes use of the full accident progression event tree (APET), which has on the order of 50 to 100 brancn points at which different possi-bilities of accident progression are evaluated as a function of the plant systems status and the various phenomenological possi-bilities and their associated probabilities (split fractions).

Although a logic tree having 50 to 100 branches has potentially millions of separate branches, these outcomes are grouped into accident progression bins. As with the plant damage states, each accident bin presents one unique set of initial and boundary conditions for the source term calculations of Block

3. Table A-2 is a (partial) list of typical BWR source term i attributes. Each branch of the containment event tree is

! numerically evaluated, and each path through the tree cor-responds to a particular set of values for the source term attributes. A vast majority of these accident paths are deleted from further consideration based on a) negligible frequency of occurrence, and b) negligible consequence of the resulting source terms.

(At least one low consequence bin is always retained to ,

represent those accident sequences with no containment failure, and hence, low consequence.) The remaining non-negligible accident paths are then grouped according to the source - term attributes, and these groupings constitute the accident sequence bins. Typically, 50 to 200 individual accident bins result from these groupings for each plant damage state.

The accident progression event tree is a logical enumeration of all the initial and boundary conditions for the accident pro-gression as well as all the potential accident paths identified by results of calculations using more detailed computer code calculations. The codes used in the phenomenological calcula-tions are shown as an input on Figure A-3. The containment event tree is the vehicle for incorporating all the insights  ;

gained from these detailed and scmetimes costly code analyses ,

of accident paths into a model which can be efficiently evaluated for use in uncertainty and sensicivity studies.

Numerical evaluation of the accident progression event tree is performed using the computer code EVNTRE. Three important aspects of this code are that

] a) Multiple branches are allowed at each node of the tree, l b) Branch probabilities may depend on paths taken through 2 the tree.

1

) c) Proper correlation between dependent probabilities can i be incorporated.

, A- 9 i

._-r -. _ _ . , _ _ . _-__.--_,_..m_ _ . - . - --

, . _ _ , - _ . , , , _ - - - *

• r m->- - - - - - - - - - - - - ~ - - - - -

j Table A-2 Typical BWR_ Source Term Attributes'Used' to Define Accident Progression Bins Initiating Event Type Time to Core Cooling Failure -~<2 hr., <10 hr., >10 hr.

RpV Pressure at Vessel Breach -- High.-Intermediate, Low j Containment Failure Time --

Before Core Degradation During Core Degradation At Vessel Breach

] Late l 1

'l Containment Leak Level -- Four sizes considered Pool Bypass Level -- Three levels considered  ;

i Pool Temperature -- Subcooled. Saturated  :

Reactor Building Bypass -- No, Yes Core Concrete Interaction -- No. Yes with Water, Yes, but dry j

Containment Sprays -- Always, Never. Early Not Late. Late Not l i Early  ;

l I

l i.

f

A -lO t

The output of the accident progression analysis consist of frequencies of the accident progression bins.

Figure A-4 shows the input and output of the source term analysis block of the back end analysis. In this analysis, the flow of fission products released from the fuel is modeled and analyzed. The numerical analysis will be performed using the RELTRAC code which utilizes a set of time-dependent rate equations to model transport, suspension and deposition of fission products from the fuel, through the reactor vessel and primary piping, the containment and subsequent release to the environment. These rato equations are based on the results of core damage and fission product release calculations and experiments. The rates used in RELTRAC reflect insight and l results from calculations using the more time consuming and detailed codes such as MELCOR. The use of the simpler rate equation formulation in RELTRAC results in a code which can be efficiently used in a sampling mode.

The RELTRAC code is run for each of the accident progression bins. The result (for each bin) is a vector of released radionuclide mass as a function of time for each of the nine radionuclide groups shown on Table A-3. These nine time histories (resulting from each accident progression bin) are then further collapsed to a vector of three numbers:

}EEERl IEER kLR where EEER = equivalent mass of 12 released in the first two hours after warning IEER = equivalent mass of 1 2 released following two hours after warning LR = equivalent maes of Cs released The condensation from the vector of nine time histories of mass release to three constant values of release utilizes studies reported in NUREG/CR-4467 in which the relative importance of the various fission product elements and their isotopos were studied with respect to their radiobiological effect, and equivalences were established in terms of equivalent masses of iodine (early biological effect) and cesium (late biological effect).

A-11

i SOURCE TERM ANALYSIS OUTPUTS:

o PARAMETRIC SOURCE TERM MODELS o SOURCE TERMS ACCIDENT GIVEN ACCIDENT 4

PROGRESSION % # PROG. BINS BINS AND o CLUSTERED FREQUENCIES SOURCE TERMS FREQUENCIES CODE: RELTRAC, CLUSTER a

INPU1 DATA & ASSUMPTIONS:

o RADIONUCLIDE RELEASE EXPERIMENTS o RADIONUCLIOE CHEMISTRY o AEROSOL PHYSICS l o EXPERT OPINIONS d

l 4

DETAILEO SOURCE TERM CODE CALCULATIONS:

o SOURCE TERM CODE PACKAGE (STCP) o MELCOR o RELAP-SCDAP o TRAC-MELPROG I

Figure A-4 L A-12  !

Table A-3 Radionuclide Groups 1

Group Elements 1 Xe, Kr 2 I, Br 3 Cs, Rb 4 Te, Sb, Se 5 Sr 6 Ru, Rh, Pd, Mo, Tc 7 La, Zr, Nd, Eu, Nb, Pm, Pr 2 Ce, Pu, Np 9 Ba A-13

Thus the source term analysis results in- a release effect vector (EEER, IEER, LE) of three numbers for each accident progression bin identified as important in the accident

, progression analysis. When uncertainties in parameters, models and timing are included, a number of different vectors are obtained for each accident progression bin. These different vectors may be viewed as different observations for a Monte Carlo analysis. Thus, if we have 50 accident progression bins and 100 observations (i.e., resulting vectors) for each bin, then 5000 different release effect vectors result. Each one of these vectors could form the input to a single health / economic consequence calculation (using the MACCS code or equivalent).

However, to reduce the number of such calculations, the release effect vectors are grouped into clusters, and a single health / economic consequence calculation is performed for each cluster. The cluster analysis is performed by considering each release effect vector as a point in three dimensional space with coordinates (EEER, EELR, LE}, and using the CLUSTER code to identify groups of points which are closely spatially related. Then, a single vector of fission products which characterizes the cluster is used as input to the consequence Calculations, t

The health and econonic consequence calculation block is shown in Figure A-5. The input is the cource term vector described above, as well as the site and environment data shown on the figure. The code to be used for the RMIEP application is the MELCOR Accident Consequence Code System (MACCS), which calculates estimatos of population dose and health effects as well as associated costs. Repeating the MACCS calculations for each input source term vector yields distributions on the t

output values which characterize the uncertainty in the back .

l end calculation of core damage accident consequences. l 1

t r

4 l

)

l A-14

P I

L CONSEQUENCE ANALYSIS: OUTPUTSt MELCOR ACCIDENT CONSEQUENCE POPULATION DOSE  !

CODE SYSTEM HEALTH EFFECTS RELEASE OFFSITE COSTS E F F ECT ---4>  : REPLACEMENT  !

VECTOR POWER COSTS r

CODE: MACCS ,

i a

?

INPUT DATA & ASSUMPTIONS:

f o SITE METEOROLOGY 4

o SITE DEMOGRAPHY l

' o 0051 METRY l

a o RADIATION INDUCED HEALTH EFFECTS '

o ENVIRONMENTAL TRANSPORT o EMERGENCY RESPONSE l

i l

2 4

l l l

t figure A-5. Consequence Analysis - Input and Output ,

a ,

A-15 i ,

i Appendix B Latin Hypercube Sample Demonstration E

I l

)

I B-1/B-2

Latin Hypercube Sample Demonstration The following discussion and demonstration of Latin hypercube sampling are taken directly from the LHS users guide.

Iman, R. L., Shortencarier, M. J., 1984, A FORTRAN 77 Proccam and User's Guide for the Generation of Latin Hypercube and ]igndom Samples for Use with Computer Models. NUREG/CR-3624, SAND 83-2365, Sandia National Laboratories, Albuquerque, NM.

This appendix describes the latest version of a computer program for the generation of multivariate samples either completely at random or by a constrained randomization termed Latin hypercube sampling (LHS). This program has been developed at Sandia National Laboratories and replaces the previous program described in Iman, Davenport, and Zeigler (1980). Every attempt has been made to make the present program portable and user-friendly while, at the same time, expanding the capability of the program to include additional ,

sampling distributions.

The situation addressed by the computer program is the follow-ing. There is a variable of interest. Y, that is a function of othat variables X,1 X2' ** > XK. This function may be l quite complicated, for example, a computer model. A question to be investigated is: How does Y vary when X's vary according to some assumed joint probability distribution? Related questions are: What is the expected value of Y? What is the 99th percentile of Y? etc.

A conventional approach to these questions is Monte Carlo. By sampling repeatedly from the assumed joint probability density function of the X's and evaluating Y for each sample, the distribution of Y, its mean, percentiles, etc., can be estimated. This is one option provided by the program for generating the X's. The program output, say for n Monte Carlo repetitions, is a set of k-dimensional vectors of input variables.

An alternative approach, which can yield more precise estimates, is to use a constrained sampling scheme. One such scheme, de-veloped by McKay, Conover, and Beckman (1979), is Latin hyper-cube sampling (LHS). LHS selects n different values from each of k variables X,1 .... Xg in the following manner. The range of each variable is divided into n nonoverlapping inter-vals on the basis of equal probability density in the interval.

The n values thus obtained for X are paired in a random man-ner (equally likely combinations)1 with n values of X2 These n pairs are combined in a random manner with the n values of X3 to form n triplets, and so on, until n k-tuplets are formed. This is the Latin hypercube sample. It is convenient B-3

to think of the Liis , or a random sample of size n, as forming an n x k .aatrix of input where the in row contains specific values of each of the k input variables to be used on the in run of the computer model.

The Lits technique has been applied to many different computer models since 1975. The results of an application of LilS to a large computer model can be found in Steck, Iman, and Dahlgren (1976). A more detailed description of LilS with application to sensitivity analysis techniques can be found in Iman, lle l t o n ,

and Campbell (1981, 1981b). A tutorial on LilS may be found in Iman and Conover (1982b). A comparison of LilS with other techniques is given in Iman and llelton (1983).

To help clarify how intervals are determined in the LilS ,

consider a simple example where it is desired to generate a Lits of a size n = 5 with two input variables. Let us assume that the first random variable Xi has a normal distribution concentrated on the range from A to B. In this program, the following interpretations (not subject to change by the user l without modifying the code) are given to A and B for both the I normal and lognormal distributions, namely P(X1 s A) =

.001 and P(X1 s B) = .001, where P(E) denotes the probability of event E. That is, A is defined as the .001 quantile and B is defined as the .999 quantile of the distribution of Xt. Thus, P(A g X1 5 B) =

.998, so both the normal and lognormal distributions are truncated slightly in the program. That is, the sampling procedure excludes values outside the interval [A, B). These definitions of A and B imply that the mean of the normal distribution is given by u . (A 4 B)/2 and since for a standardized normal variable Z, P(Z s -3,09) = .001, it follows that the standard deviation of the desired truncated normal distribution is given (to a close approximation) by o- (B - u)/3.09 = (B - A)/6.18.

With the parameters u and a thus defined, the endpoints of the intervals are easily determined. The intervals for n = 5 are illustrated in Figure B-1 in terms of both the density function and the more easily used cumulative distribution function (cdf). If the distribution were not truncated, then the intervals in Figure B-1 would satisfy P(A 1 X1 s C) = P(F s X1 s B) = .198

=

P(C 5 X3 s D) = P(D 5 X) s E) =

P(E s X t i F) = .2.

B-4

To account for truncation requires dividing these probabilities by .998. Thus, for all practical purposes, the five intervals correspond to 20% probability.

We shall assure in thin example that the second random variable, X2, has a uniform distribution on the interval from G to 11 . The corresponding intervals used in the LilS for X2 are given in Figure B2 in terms of both the density function and the cdf.

The next step in obtaining the Ll!S is to pick specific values of X1 and X2 in each of their five respectivo intervals.

This selection should be done in a random manner with respect to the density in each interval, that is, the selection should reflect the height of the density across the interval. For example, in the (A,C) interval for X1, values close to C will have a higher probability of selection than will those values close to A. Next, the selected values of X1 and X2 are paired to form the required five input vectors. In the orig-inal concept of Lils as outlined in McKay, Conover, and Beckman f(sl >

b  : ;

L  : 1 1

. l l

; l l l  ?

,-  : l l l T .: j ,e j . { . ! .:

!  ! l  !

k C o a y a

,g. , .... ........ .. .................. ..

.e ... .-...............  ;  ;

,4 ...................., f f

.: L. ........ ..... ll  !  ! I

,! I  !  ! l A c o a e e J

\

l Figu r e B- 1. Intervals Used with a I.lis o f Size n = 5 in Terms of the Density Function and Cumulative Distribution l

Function for a Normal Handom Variable l

H$

l l

l l

1 l

• b .8 .8 .8 .3 .3 e i J x t M F(x) .

.e ................ i e

e

.s .......

l l l i i . i

.a ...

l i e

.  !  !  ! l l e i J x t H i Figure B-2. Intervals Used with a Lits of She n 5 in Terms t of the Density Function and Cumulative Distribution l Function for a Uniform Handom Variable '

l i

(1979), the pairing was done by associating a random permu- i tation of the first n integors with oach input variable. For purposes of illustration, in the procont example consider two random permutations of the intogors (1, 2, 3, 4, 5) as follows:

Permutation Set No. 1: (3 1, 5, 2, 4) f Permutation Set No. 2: (2, 4, 1, 3, 5)

By using the respective position within thoso permutation sets as interval numbers for Xi (Set 1) and X2 (Sot 2), the .

following pairing of intervals would be formed.

Interval No. Interval No. f Computer Run No. Used for X1 Used for X2 j 1 3 2 2 1 4  !

3 5 1 4 2 3 5 4 5 j i

I B-6 r l

I r

Thus, on computer run number 1 the input vector is formed by selecting the specific value of X1 from interval number 3 (D to E) and pairing this value with the specific value of X2 selected from interval number 2 (1 to J), etc. Once the specific values of each variable are obtained to form the five input vectors, a two-dimensional representation of the LilS can be made such au given in Figure B-3.

Note in Figure B-3 that all of the intervals for Xi have been sampled, 'and the same is true of X2 In general, a set of n 1.11 S points in k-dimensional Euclidean space contains one point in each of the intervals for each of the k variables, i

l

\

~

H e

l l

( > =. -

GE X e i

• K - -

O uJ

O

• Z J . _ _ _ _ ._

i E I . _ . _ _ . . _

i g

• A C D E F B i

R /.N G G O F X i Figure B-3. A Two-Dimensional Representation of One l Possible i.liS of Size 5 Utilizing Xt and X2 B-7

i U"I "

N(5.2.418)

. l *'

I I

0 1

, i l l I l e i t .

' i  ! i

. i .

l n

l l ,I 1

, i i i

I e t i

, t 8

' $ j i

e

> ' s, i i

. i a i ' .

1 i  ; i A=0 * * " e B = 10

• * ?

• n

• e e F(x) -

p, ., g g 4 , - - - - - - . _ _ _ _ _ _ _ _ _ _

p, .,y g y r.___-._____..

i i  !

- I -

1 Pe = . 8 0 5 r- - - - - - - - - - - - - ,!

[ t Pe =.S i t ' . - . - - - - - - - - - - - - - 8

, i I

e l I

. e 1 I I l l i i i Pi =. 014 - - - - 1 J ' I e n - e e n e N e

• i a o e a a s 4 4 i A Figure B-4. Interval Endpoints Used with a Lits of Size 5 (top) and Specific Values of X Selected Through the Inverse of the Distribution Function (bottom)

B-8 f

6 To illustrate how the specific values of a variable are obtained in a Lits, consider the following example. Suppose it in desired to obtain a Liis of size n = 5 from a normal distribution on the range from 0.0 to 10.0. Recall that these two limits are taken to represent the lower and upper .001 quantiles, respectively. Therefore, the random variable has a mean of five and a variance of 2.618 as indicated in Figure 4.

These points together with the density characteristics of the normal distribution allow for the definition of the equal probability interval endpoints. These endpoints are shown in Figure 4 in terms of a density function. The next step is to randomly select an observation within each of the intervals.

This colection is not done uniformly within the intervals shown in Figure 4, but rather it is done relative to the distribution This being sampled (in this case, the normal distribution).  ;

means that the sampling is done uniformly _o_n the vertical axis <

o,f_the cdf as shown in Figure B.4.

Therefore, to get the specific values, n 4 5 randomly selected uniform (0, 1) numbers (Um, m = 1, 2, 3, 4, 5) are obtained to serve as probability levels. These probabilities are then scaled by Pm

• Um(.2) + (m - 1)(.2) m=1, 2, 3, 4, 5 Thic ensures that exactly one probability, Pm, will fall ,

within each of the five intervals (0, .2), (.2, .4), (.4, .6),

(.6, .8) and (.8, 1). The values Pm are used with the inverse normal distribution function to produce the specific values to be used in the LilS. Note that exactly one observation is taken from each interval shown in Figure B-4. The entire process is shown in Table B.1. Figure B-4 makes it clear that when obtaining a Liis , i t. ic casier to work with the cdf for each -

variable. This is the approach used in the computer program, rather than defining the ondpoints of the intervals on the x axis.

The above illustration shows how one input variable having a normal distribution is sampled with LilS . This proceduto is  ;

repeated for each input variable, each time working with the corresponding cumulativo distribution function. If a random sample is desired, then it is not necessary to divide the vertical axis into n intervals of equal width. Rather, n random numbers between 0 and 1 are obtained and each is mapped through the inverse distribution function to obtain the specific values. The final step in the sampling process involves pairing the selected values, it should be noted that even though two variables are sampled ,

independently and paired randomly, the sample correlation l coefficient of the n pairs of variables in either a random sample or a Lits will, in general, not equal zero, just due to sampling fluctuations. In order to obtain a sample in which the sample correlations more nearly match the assumed, or B- 9

_ = _ _ _ _ _ _ . _ _ _ _ _ _ _ -_ _ _ - - _ . __ _ _ = __ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ . -.

Table B.1 One Possible Selection of Values for a Liis of Size 5 from a Normal Distribution on the Interval (0,10)

Scaled Corresponding Corresponding Probabilities Standard Normal N(5,2.618)

Interval Uniform (0,1) Within the Interval Value (z-score) Observation Number Random No. Pm = Um(.2)+ From the Inverse Within the m U_m (m-1)(.2) Distribution Intervals 1 .080 .016 -2.144 1.529 5 2 .610 .322 -0.462 4.252 3 .525 .505 0.013 5.021 4 .935 .787 0.796 6.288 5 .620 .924 1.433 7.319 l

intended, correlations, Iman and Conover (1982a) proposed a method for restricting the way in which the variables are paired. The effect of this restriction on the statistical properties of the estimated distribution of Y, its mean and percentiles, is not known, but is felt to be small. . The pairing of variables in the program can be done either randomly or by the restriction procedure through use of an input parameter which is explained in the next section.

Additionally, the restricted pairing procedure of Iman and Conover can be used to induce a user-specified correlation among selected input variables through use of another input parameter explained in the next section. However, it should be pointed out- that such induced correlations are based on the non-parametric technique known as rank correlation. Such a measure is used since it remains meaningful in the presence of non- normal distributions on the input variables.

As a final note if a correlation structure is pg specified by the user, then the program computes a measure for detecting I large pairwise correlations. This measure is known as the l variance inflation factor (VIF) and is defined as the largest element on the diagonal of the inverse of the correlation i4trix. As the VIF gets larger th.tn 1, there may be some uidesirably large pairwise correla ans present. Marquardt and Snee (1975) deal with some very Arge VIPS (> 2 x 106) and provide a very readable explanation on reasonable sizes of VIPS. Marquardt (1970) indicates that there can be serious collinearity (i.e., large pairwise correlations present) for VIF > 10. Thus, there is certainly no problem as long as the VIF is close to 1. The VIF appears as part of the computer printout when the user requests the correlation matrix to be printed, given that no correlation structure has been specified by the user.

l l

l L-ll

References Iman, R. L. and Conover, W. J. (1982a). "A Distribution-Free Approach to Inducing Rank Correlation Among Input Variables," Communications in Statistics, Bil(3), 311-334.

Iman, R. L. and Conover, W. J. (1982b). Sensitivity Analysis Techniques: Self-Teachino Cu r r i cu lu_m , NUREG/CR-2350, SAND 81-1978, Sandia National Laboratories, Albuquerque, New Mexico 87185.

Iman, R. L. and Davenport, J. M. (1982). An Interative Alco-rithm to Produce a Positive Definite Correlation Matrix from an "Approximate Correlation Matrix (With a Program User's Guide), SAND 81-1376, Sandia National Laboratories, Albuquerque, New Mexico 87185.

Iman, R. L., Davenport, J. M., and Zeigler, D. K. (1980).

Latin Hypercube Samplino (Procram User's Guide), SAND 79-1473, Sandia National Laboratories, Albuquerque, New Mexico 87185.

1 Iman, R. L. and Helton, J. C. (1983). A Comparison of Uncer-tainty and Sensitivity Analysis Techniques for Computer Models, NUREG/CR-3904, SAND 84-1401, Sandia National Laboratories, Albuquerque, New Mexico, 87175.

Iman, R. L., Helton, J. C., and Campbell, J. E. (1981a). "An Approach to Sensitivity Analysis of Computer Models, Part 1.

Introduction, Input Variable Selection and Preliminary Variable Assessment," Journal of Ouality Technolocy, 13(3),

174-183.

Iman, R. L., Helton, J. C., and Campbell, J. E. (1981b). "An Approach to Sensitivity Analysis of Computer Models, Part 2.

Ranking of Input Variables, Response Surface Validation, Distribution Effect and Technique Synopsis," Journal of Quality Technology, 13(4). 232-240.

Marquardt, D. W. and Snee, R. D. (1975). "Ridge Regression in Practice," The American Statistician, 29(1), 3-20.

Marquardt, D. W. (1970). "Generalized Inverses, Ridge Regres-sion, Biased Linear Estimation, and Nonlinear Estimatior,"

Technometrics, 12(3), 591-612.

McKay, M. D., Conover, W. J., and Beckman. R. J. (1979). "A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code,"

Technometrics, 21, 239-245.

B-12 4 -

Steck.-G.'P., Iman, R. L., and Dahlgren. D. A. (1976). Proba-bilistic Analysis of LOCA. Annual ReDort for FY1976 SAND 76-0535 (NUREG76-6513), Sandia National Laboratories, Albuquerque, New Mexico.

I I

i B-13

. _ _ _ . _. . . . . _ _ . _ . . _ _ _ . _ _ _ _ _ _ . _ _ _ _ , , . _ _ _ _ _ . . _ . _ . . _ , , . _ . _ _ - , _ _ , _ , . , . _ _ _ _ . . _ _ , - . , . _ _ _ _ , _ . . . _ _ _ . - _ , . ,i

Appendix C Elicitation of Expert Opinion--Some Guidelines C-1/C-2

ELICITATION OF EXPERT OPINION--SOME GUIDELINES J. M. Booker and M. A. Meyer Los Alamos National Laboratory This appendix outlines some guidelines and issues of importance to eliciting expert opinion data. It is intended to provide a brief overview.

However, it offers references for those interested in more detailed 1 information.  :

Some of the concepts discussed below are 1) the nature of expert opinion data; 2) means of selecting and motivating experts; 3) basic methods for l eliciting expert opinion and tradeoffs in selecting one; 4) selection .of a l response mode and means of measuring dispersion; 5) how much data to elicit and record; and 6) reducing or countering the occurrence of bias in i elicitation methods, response modes, or requests for problem-solving data; and '

7) analyzing the information.

EXPERT OPINION DATA Expert opinion data consists of expert's _ subj ective judgments.

Typically, it has been gathered where experimental data is sparse or unavailable such as in risk analyses or reliability assessments. The use of expert opinion data is wide spread in Probabilistic Risk Assessments (PRA) applications and ranges from those formally elicited to those which are done as part of the PRA process. Most people think of the formal use of expert opinion such as in the requesting of probabilities of rare events. However, expert opinion also includes the less formally elicited judgments that are made by engineers and analysts at various stages of the PRA. Some examples are which systems vill be included; what vill be used as a lover bound on screening the cut sets; vhat features vill be included in the accident sequence or fault tree; who vill serve on the review board of the PRA; who vill be on the PRA team; and what data vill be used in the PRA7.

Until recently, many scientific disciplines viewed expert opinion, as being lover in quality than data gathered by observations or measurements.

This view had been most common in fields, such as engineering, where even measured data that was probabilistic in nature ("fuzzy" or uncertain) was not readily accepted. In these environments, .information that -vas not deterministic or totally quantitative in nature tended to be regarded as other than '; al data" or as unanalyzable. This philosophy has been changing and nov even the data purists seem to be more amenable to utilizing all the information available, including that from human beings.

The current trend is one of acceptance of "fuzzy" or probabilistic data, whether from measurements or observations, and of concentration on better, more consistent ways of handling it. Expert opinion fits into this viewpoint because it is observational data, can be both quantitative and qualitative, and has a probabilistic structure. Like any other type of data, care is needed in its elictation, analysis, and interpretation. The "care" that is needed is in the use of consistent and valid procedures for handling this data. This appendix focuses on the use of such procedures in the elicitation process.

Although expert opinion ranges from those formally to informally elicited, none of these judgments need be gathered in an ad-hoc fashion. All C-3

varieties of expert judgment can benefit from careful elicitation. Adherrance to simple guidelines on eliciting and recording is usually sufficient to providing vell-documented and consistent procedures for handling expert data in PRAs. Some of these suggestions are given below.

SELECTING AND MOTIVATING EXPERTS An expert is anyone especially knowledgeable in the field and at the level of knowledge being elicited. For example, one level of knowledge could be core damage frequency for a group of plants ; and a more detailed lovel could be the risk of direct containment heating at the Surry Plant. Expertise can be of two types: normative and substantive. Substantive expertise is having knowledge in the field in question such as on nuclear reactors or seismology. Normative expertise is based on having knovledge related to the response mode (i.e., the form in which the experts are asked to encode their judgments). Substantive expertise is no guarantee of normative expertise.

For example, because the experts are asked frequently about the probabilities of events, an expert could have substantive knowledge of the field at the necessary level of detail required but not normative knowledge of the rules of probabilities.

In selecting experts, normative expertise, substantive expertise, and knowledge at the needed level of detail are always important. In addition, such factors as the reputations and diversity of the experts can beco'ne Selecting experts who are vell known and respected among important. their peers or the public can lend greater credibility to the study as vell as provide motivation for other experts to participate. Diversity among experts is desirable when trying to counter anticipated correlation or dependency among experts (Seaver, 1976).

Motivating experts can be done by communicating the intrinsic benefits of participation or by offering renumeration. The authors recommend motivating through communication because paying the experts is costly, may attract one type of participant (Gordon, 1980:118), or may detrimentally affect their viev of the study (Baron and Byrne, 1974:122). Some aspects of the study which could be presented as motivating to the experts are: the chance for recognition; the opportunity to contribute to the development or improvement of a process; and the experiencing of something new or, at least, different from their usual vork routine.

In general, how the aspects of the study are communicated to the expert vill affect his desire to participate. Usually, more individuals vill respond positively to a request delivered in person than by mail. For this reason, it is recommended that the experts be contacted, initially, in person or via the telephone. Guidelines abstracted from communications theory (Stroud 1981; Gorden, 1980) and the authors' interviewing experiences suggest that particular items of information be communicated and in the following order: 1) why the experts are being contacted (the purpose of the study and why they are needed); 2) who is conducting or sponsoring the study 3) how much time and effort their participation vill involve; 4) how they were selected; 5) What tasks vill be required; 6) how confidentiality vill be handled; 7) vhether participation is voluntary or reauired; 7) vhat the product of the study vill be and whether they vill have access to the results (Heyer and Booker,1987b).

ELICITATION METHODS There are about four methods commonly used for elicitation the staticized (also called the "nominal") group, the Delphi, the structured interactive group, and the interactive group. These vary in terms of the degree of interaction that they allov between the experts and how the expert's C-4

l estimates are combined. For example, the staticized group method allows no interaction either by mail or in person between the experts. In this case, estimates tend to be mathematically aggregated. The Delphi method involves limited interactions between the experts via the mail. The experts' estimates and reasoning are made anonymous and then circulated to the experts to allow them to consider this data in revising their estimates. This process can be continued until consensus, if it is desired, is achieved. The structured interactive method permits the face-to-face meeting of experts but structures their interactions. For example, the experts may be allowed to speak to one another only at specific times during their meetings. Their estimates may be aggregated mathemathically or through achieving consensus during the elicitation sessions. The interactive group method consists of a group of experts who are interacting in a spontaneous fashion in a face-to-face situation. This method employs few controls and, thus, resembles the mee' tings that commonly take place in vork settings. Estimates obtained from this method can be combined by either method, although mathematical aggregation is recommended. In particular, simple aggregation schemes, such as medians, are suggested (Martz et al., 1985). (For a more complete description of these elicitation methods, see Meyer et al., 1982).

At the basis of these four methods lie two views of interaction between experts. One view holds that interactions are to be avoided or limited i because of their af fect on the judgment process. For example, the staticized I group method avoids any inter-expert interaction, while the Delphi method permits only limited and controlled interaction. The Delphi sethod, developed by Rand Corporation, was designed to prevent bias arising from group dynamics in a face-to-face setting. One common problem arising from group dynamics is the group think phenomenon, also called the follov-the-leader effect (Janis, 1972). To avoid having experts consciously or unconsciously adjust their judgments to agree with those of the leader (s), the Delphi keeps experts physically separated. Their judgments are sent to a central location where any distinguishing features are removed. The newly anonymous judgments are then recirculated to all the experts, who are allowed to change their previous estimates.

At the other end of the spectrum, are approaches like the interactive group methods that encourage group interaction in the belief that the best quality ansvers come from the interaction of knowledgeable persons in a face-to-face setting (Seaver, 1976). Some interactive methods, like the structured interactive group, employ structure to minimize the effects of group dynamics.

A few examples of this structuring would be: having everyone state their viev, one at a time, while no one else is allowed to comment; having the natural group leaders give their thoughts last; and having the individuals record their judgments as a means of encouraging them to anchor to their own judgments. The interactive group method can be structured to any degree as time permits.

Elicitation methods are usually selected on the basis of the analyst's belief as to whether interaction between the experts is desirable or not. The analyst frequently makes this decision because the literature has not been conclusive as to which method produces the best, the least biased, results. i I

The authors favor a structured interactive group method because of the synergistic effect of discussion on the judgment process. Techniques, such as i

mentioned in Reducing or Countering Bias, can be used to mitigate the biases that occur in an interactive group setting.

Any of the basic elicitation methods can be structured to different I

degrees. For example, the interviewing of the experts in the staticized group I can be done informally or in a structured manner. The structured manner could

)

c-5 i l

- . _ _ . _ _ , _ . -- __ __ _- ____J

involve presenting each expert with the assumptions or definitions that they are to use in solving the problem and then monitoring their use of these. As a general rule, the more structure imposed on the method, the more time required.

Generally, it is best to gather expert opinion in a structured and controlled manner for several reasons: 1) to knov vhat considerations have entered into the experts' estimates, 2) to defend the gathered data, and 3) to update or modify this data (Heyer, 1984).

Some considerations that enter into the experts' estimates are their definition of the problem; their interpretation of directions; their assumptions or boundary conditions; and the calculations or heuristics used to solve the problem.

Expert data may require oefense against critics of the elicitation and/or the conclusions. The method of elictation needs to be considered valid by current theory in communications, cognition, and psychology. Often, if the elicitation is properly conducted, the experts vill feel positively enough about the study to defend it.

Frequently, there is a need to update or modify the expert data (e.g., if circumstances have changed). If the elictation method has been structured properly, there is a greater chance of knowing which factors relating to the estimate have been affected by the change cnd therefore, how to modify the data acordingly. It is recognized that most expert opinion is biased, one way or another. The challenge is 'o knov which vay so that a counter-bias can be employed (Payne, 1951; Meyer, 1984).

RESPONSE MODE AND DISPERSION MEASURES Response mode: The response mode is the form in which the expert is requested to give his judgment. Probabilities, odds, logs, intervals, paired comparisons, and ratings (Seaver and Stillvell 1983; comer et al.,1984) are a fev of the commonly used response modes. Selection of a response mode involves two issues: obtaining the experts' judgments in a form that can be analyzed or used in a model, and avoiding bias. For example, the analyst may prefer probabilities and request the expert's judgments in this form. If the data is to serve as input to a model, the experts' judgments will need to be given in a compatible form or to be converted later. Associated with these considerations is the question of whether the expert can encode his thoughts into the requested response mode.

Typically, the focus in expert opinion has been on the practical issue of obtaining the experts' judgments in the needed form rather than on the experts' ability to encode their thoughts into that form. The later involves the problem of bias. Bias can be defined as occurring when: 1) the expert's underlying judgments or reporting of these is altered by the elicitation process; and 2) when the expert's estimates do not follow normative statistical or logical rules (Heyer, 1987). The response mode can lead to either type of bias. For example, the expert may not be able to express his exact thinking in the requested response mode -- In the translation from his judgment to an acceptable response, his judgment may be misrepresented. Then too, experts can give estimates that do not follow the mathematical or logical rules governing their use. For example, probabilities for mutually exlusive, exhaustive events should sum to 1.0 and the expert's probabilities may not.

Some of the bias may stem from the expert's unfamiliarity with the response mode. Generally, substantive experts (those with expertise in their field) are not accustomed to couching their judgments in the form of logarithms, intervals, percentiles, or even probabilities. Their substantive expertise does not guarantee normative expertise (expertise in the use of the C-6

response mode). Another contributor to bias may be the way in which the human mind works.

Hogarth (1980) has attributed many biases to our "limited, sequential, information processing." The human mind is not a miniature calculator.

Because of its limited memory space for information processing, the mind resorts to short cuts, heuristics, to solve problems. These heuristics can result in a skeving, biasing, of the answer from what it should be mathematically. For example, in solving a complex problem an individual may form an initial impression of what the answer should be and then consider components of the problem one at a time. In solving each part of the problem, the individual may mentally adjust up or down from his last impression. This heuristic, anchoring and adjustment, tends to produce a final estimate which is biased tovards the individual's initial impression. (See Hogarth 1980 for other biases stemming from the way in which we think.)

It is advisable to avoid the use of response modes with which percentiles and individuals are reputed to have difficulty, such as probabilities, in favor of simple linear scales. Difficult responsethat modes are do not defined here to mean those that lead frequently to estimates follov normative statistical or logical rules. A continuous linear scale is a number line with defined endpoints. These endpoints should represent extreme l values and be labeled with text or numbers. The scale also includes tick marks or intermediate values. The expert is instructed to mark his answer at or between any of the delineations on the scale.

The experts find these scales convenient to use and the results of their use are easily interpreted for analysis. Some advantages of this scale are:

1) it requires very little definition from the interviewer; 2) it does not restrict the range of response possibilities like a caltiple choice response mode; 3) it provides the opportunity for responses in the extremes values; 4) it relies on linear thinking, a mode that most humans use naturally (Kahneman and Tversky, 1982); and 5) it is easily converted to numerical, continuous variables for analysis.

Dispersion Measures: In addition to the single estimate (a central probability measure), experts are often asked to provide a dispersion measure such as percentiles, variances, ranges, or error bars. These dispersion measures are prone to bias (Kahneman and Tversky, 1982) because the experts are not able to provide the measures that match the prescribed defintions.

For example, the expert may be asked to provide the 5th and 95th percentiles of his distribution for the probability of an event. He vill provide two estimates that he thinks correspond to the 5th and 95th percentiles; however, l those two estimates vill more likely correspond to approximately the 33rd and l 67th percentiles. Humans, in general, tend to underestimate uncertainties.

Even in studies where experts vere trained or vere knowledgeable of this bias, l the results were still only marginally improved and uncertainty remained l underestimated (Lichtenstein et al., 1982). In addition, humans also tend to i be poor at estimating variances or standard deviations (Kahneman and Tversky, l 1982). Humans also have difficulties in estimating extreme values such as the absolute maximum or minimum values (Martz et al., 1985).

Therefore, asking experts for specifically defined measures of dispdon vill yield estimates that cannot be accurately interpreted. The only .

interpretation that can be made is : hat they vill underestimate the true l uncertainty. This bias also occurs if the experts are asked to estimate the dispersion of a distribution of multiple experts' estimates. This occurs I because each expert is likely to perceive others as having the same values as he does.

l C-7

I I

One solution is to elicit some specified measure (e.g., the 5th and 95th percentiles), and then to double or triple the range of values knowing that the provided ranges vill underestimate the true uncertainty. Then the question becomes how much adjustment is needed. There is no clear answer to this question because there is no vay of knowing what is actually represented by the ranges given.

A second solution is to elicit only the central tendancies and to use multiple experts to form a single distribution. The problems with this are twofold: 1) a large number of expert estimates is required; 2) the resulting distribution is usually multimodal which leads to the question of how to interpret the mixture of distributions.

A third solution is to elicit the central tendancies and to allow the expert to volunteer an undefined range of values around it. This undefined range could also be requested; however, in such a case, the expert vould need a definition of the range being requested and the analyst is faced again with the problems in solution one. The volunteered dispersion also lacks a specific interpretation; however, it can be interpreted as defining a set of values that the expert felt like specifying. This set of values could then be used as multiple estimates from the same expert allowing the expansion of a usually sparse data set, or it could be interpreted as some middle quantiles (e.g., 40th and 60th percentiles) for formulating expert prior or likelihood distributions.

Iae specified and volunteered dispersion measures both have the same interpretation problem when using them to formulate expert distributions. The one advantage of the volunteered measure over the specified one is that the volunteered measure is free from any biases imposed by the interviewer in requesting and defining it. The volunteered estimates fit into the elictation philosophy of attempting to minimize bias.

H0V HUCH DATA TO ELICIT AND RECORD A variety of data can be elicited and recorded the estimate with a dispersion measure, if elicited; data on how the expert arrived at his answer (e.g., information considered and heuristics used in solving the problem); and some of the characteristics of the expert's professional background (e.g.,

education, vork experiences, and experience with similar types of problems).

The question of how much data to gather arises when the researcher veighs the possible benefits of gathering more data than the estimate against the additional time that this vould take. Many researchers do not know how an expert's reasoning could be used or how this qualitative data could be analyzed by itself or in combination with the expert's estimates. For these reasons. data on how the experts arrived at an estimate has generally not been gathered in the field of expert opinion. The fev exceptions to this rule can be found in studies where the expert has asked that one of his considerations, such as a caveat, be recorded to clarify his answer (Benjamin et al.,1986).

In addition, data on the experts' background is either not elicited or not printed to protect the experts' identities. For example, in an earlier seismic study, not only were the expert's names not linked to their judgments but their names were not identified (Baecher, 1979).

The authors suggest that data on how the experts solve the problem be elicited and recorded. Each expert should be observed and asked to name any information that they are using in solving the problem. For example, the expert may say that he is considering the conditions of state or the definitions given in the background to the problem. In addition, the expert should be asked to explain how he is solving the problem. His description C-8 1

might include some equations, enculations, or more general heuristics, such as mentioned in the Response Mode section.

Recording the problem-solving data offers such benefits as providing guidance and justification for aggregation and for the eliminationand of outliers. Aggregations are frequently performed toAcombine bimodal distribution of multiple, often differing estimates, into one estimate.

- estimates is common but if the analyst does not know why the experts are split into two camps, he is handicapped in his decision on how to aggregate (Booker and Meyer, 1985). An aggregation scheme can be selected on the basis of the problem-solving information rather than guesswork, if problem-solving data to is elicited. In a Bayesian approach to aggregation, the decision maker has decide how to combine the expert's estimates with each other and with his judgments. Knowledge of the expert's problem-solving methods vill allow the decison maker to judge which estimates he wishes to veight most highly (Meyer and Booker, 1987a).

In the treatment of outliers, the "aberrant observations should be recorded and the reason for deleting them explained" (Tietjen, 1986). The problem-solving data can shov vhether the experts who supplied the suspected outliers solved the problem in a different manner and whether their estimates can be eliminated. If the suspected outlier appears to be assumptions legitimate data (e.g., the expert did not appear to use unvarranted In this or case, unacceptable definitions), then this evidence can be stated.

the outliers could be accomodated by means of a robust estimator.

In addition, it may be a good idea to record some background information such as, the experts' years of experience, formal ede tion, previous work experiences related to the problem area, years since t+2 expert vorked on a similar problem, and colleagues with whom the expert has wen associated.

The recording of this magnitude of data can be made easier and more thorough by the use of a note-taking observer or a tape recorder. The means of recording can be chosen by asking which vill b2 the most convenient, unobtrusive, and reliable in the particular setting. Whatever means is used should have as its goal the recording of data in as exact manaer as possible.

Filtering, reducing and modeling of the information gathered can be done following the interview; however, the results from such screening should always be traceable back to the original data.

REDUCING OR COUNTERING BIAS IN ELICITATION As a general rule, bias in expert opinio*, can be bandled by anticipating where it is likely to occur, designing the means to counteract it, and then monitoring the elicitation process. This bias countering plan can be applied to the different parts of elicitation, auch as the elicitation method, the response mode / dispersion measure, and the problem-solving interviev. As mentioned earlier, bias can occur when: 1) the subject's underlying thoughts or reporting of these has been altered by some aspect of the elicitations or

2) when thi subject's thinking does not follow normative statistical or logical rules.

For example, in selecting an elicitation method, one could anticipate the occurrence of both types of bias. If, for instance, an interactive group method was being considered for use, one could expect the possibility of group think. Vith the group think phenomena, the expert's thinking is altered as he consciously or unconsciously acquieses to what he believes to be the group opinion. Then too, in an interactive group method, the experts' failure to follov logical or statistical rules could surface. For example, if the experts vere meeting for several sessions, they vould be prone to inconsistency in their thinking. In particular, the definitions or conditions C-9

that the group has agried to use are likely to under go subtle changes and reinterpretation in the minds of each expert. Because the expert's judgments are based on such info *21 tion, these changes can affect their ansvers.

The elf:itation method can be designed to counter the anticipated biases.

For example, in the case of the interactive group method, group think could be countered in several ways. One way would be to have the experts record their thoughts and tiien, one at a time, present these to the group. (Other vays are given in the section, Elicitation Methods, or in Meyer, 1984.) To counter inconsistency, in part, one could conduct a review of the definitions or conditions at the beginning of every session.

In attempting to counter bias in the use of response modes and dispersion measures, selection is the key. As mentioned in the Response Mode section, people do not use the modes and measures, statistically or logically, as they should. Humans not natural Bayesians nor are they good assessors of encertainty. Ve recommend the use of simple response modes, such as linear scales, which are not as prone to bias. <

If the expert is to be intervieved on how he solved the problem, the phrasing and timing of these questions is critical in avoiding bias. The major bias to occur in this form of elicitation is the influencing of the expert's thoughts or his descriptions or these. To avoid introducing this source of bias, the authors recommend several techniques: verbal protocal from educational psychology (Belkin et al., 1986), the ethnographic method from cultural anthropology (Spradley, 1979), and a type of verbal probe from psychology (Ericsson and Simon, 1980). These techniques work best in individual interviews but they can be adapted to interactive group situations.

Vith the verbal protocal method, the expert is instructed to think aloud as he progresses through the problem. The verbal protocal is believed to deliver the data that the subject currently possesses in his short-term memory (Ericsson and Simon, 1980). Because the interviewer does not ask questions but simply records the subject's verbalizations, there is little chance of his affecting the subject's thoughts. The method of verbal protocal need not be used, if sufficient information can be obtained using the ethnographic method and serbal probe, described below. A disadvantage of the verbal protocal method is that it can slow the expert's progress on solving the problem. Then too, it does not vork on very complex problems because the expert stops thinking aloud to concentrate.

The ethnographic method involves restating the expert's words into questions (Heyer and Booker, 1987a). Thus, it does not insert the interviever's thoughts and bias the expert's account. The ethnographic method can used to pursue, in greater depth, elements that the expert mentioned in his thinking aloud.

The verbal probe is used immediately after the expert has given his solution to gather additional data on how the expert solved the problem. This probe is asked immediately because the needed information is only available in an individual's short-term memory for a brief period of time. Thereafter, the individual is likely to be able to retrieve only portions of what he had in his short-term memory and to have to fill in the gaps by guessing (Ericsson and Simon, 1980). A simple phrasing of the probe is used, such as "Vhy did you give this ansver," as opposed to a check-list phrasing. An example of a check-list phrasing would be, "Did you consider factors X, Y, and Z?" The simple phrasing is preferrable because it is less prone to bias. Vith the check-list phrasing, the subject is more likely to say that he considered these factors when he did not and to believe that he should consider these in his future problems. The authors recommend that the verbal probe be used to obtain data on the expert's problem solving, even if no other method is used.

C-10

It takes very little time and can be used easily in group settings. Asking the expert to explain his ansver immediately after he has given it vill provide enough data for the receiving the benefits mentioned in Elication Methods. If the explanation is unclear or too general, the ethnographic method can be used to gather additional information.

In addition to designing or structuring elicitation methods to counter bias, it is advisable to monitor the sessions. For example, during an interactive group session, one could vatch for any indication of a group think phenomena. The lack of any dissenting views could be an indication of a group think situation. By monitoring the elicitation sessions, one can gauge the presence of bias and employ further countermeasures, if necessary.

ANALYZING THE DATA There are several analytical problem areas vorthy of mention. These are discussed in Meyer and Booker (1987c) and include recommendations on: 1) how to handle the large amounts of data gathered; 2) how to handle the qualitative / quantitative data mixture; 3) hov to formulate models at the desired level of generality; 4) how to deal with the correlation among

experts, if any exists; and 5) how to aggregate estimates or distributions from multiple experts. Vhether the approach taken is a Bayesian or classical one, these problems are likely to be present. Each is discussed briefly below.

Many multivariate statistical techniques exist for handling the large amounts of data. Cluster and correlation analyses can provide means for identifying relationships between the variables and the response data.

Non-parametric and rank procedures are suggested for handling the qualitative / quantitative data mixture to avoid the making of assumptions about data distributions (which are not usually normal or multivariate normal).

analyses.

The level of generality is an important assumption made in these experiments.

All analysts deal with this problem in designing and analyzing Usually, the experiments are designed to gather the specific information required to answer the research questions. In expert opinion, the information gathered cannot alvays be done in such a controlled environment.

This is why the authors recomnend that the analyst gather all possible information first and screen, filter, and model what is needed later. It is at this point that the level of generality must be decided. Hov finely vill the information be screened? What granularity vill be assumed? It should be noted that the conclusions drawn only apply to the level of generality used in the modeling process. Conclusions can be different (Heyer and Booker, 1987c). for different levels An example of the importance of the level of generality can be found in the issue of correlation among experts. At a general level of problem j solving, correlation among experts vas found (Heyer and Booker, 1987a). l However, at a more detailed level of analysis, this source of correlation was  ;

not evident (Heyer and Booker, 1987c). Inter-expert correlation seems to i exist only at certain levels of modeling the problem-solving processes. Thus, the correlation issue may not be as prevasive or troublesome as previously I I

thought.

If correlation among experts is not a problem, then combining expert  !

estimates or distributions together becomes an easier process. Veighting factors or distributions can be assumed and used in many aggregation schemes (see Martz et al., 1985). The aggregation can be done in a Bayesian framework using prior and likelihood distributions for data and for experts, or by the formulation of a single distribution from the multiple experts. Decision i

P C-11 t _

l l

analysis techniques can be used in either framevork to incorporate the effects of the decision maker (Booker and Bryson, 1982).

SUMMARY

The benefits of using the suggested techniques for elictation outveigh the efforts required to implement them. 1) A consistent and structured methodology for elicitation can be established and used. 2) As situations change, old information can be updated if it has been properly documented. 3)

Information can be compared from different studies. 4) Defense against critics can be given with confidence. 5) Futher avenues of research and development can be discovered and implemented thus, adding to the field of knowledge and improving the methodology.

REFERENCES Most of these references are available, upon request, from the authors:

J. M. Booker or M. A. Meyer at Los Alamos National Laboratory, P.O. Box 1663, MS F600, Los Alamos. NM, 87545, USA.

Baecher, G.B. (1979), "Correlations Among Experts' Opinions," Unpublished manuscript, Massachusettes Institute of Technology, Boston, MA.

Baron, R.A. and Bryne, D. (1974), Social Psychology: Understanding Human Interaction, Allyn and Bacon Inc., Boston, MA.

Belkin, N.J., Brooks, H.M., and Daniels, P.J. (1986), "Knowledge Elicitation Using Discourse Analysis," Proceedings from the Knowledge Acquisition for Knowledge-Based Systems Vorkshop, Banff, Canada, pp. 3,0-3,16.

Benjamin, A.S., Kunsman, D.M., Villiams, D.C., Boyd, G.J., and Murfin, V.B.

(1986), "Evaluation of Severe Accident Risks and the Potential for Risk Reduction: Surry Power Station, Unit 1," SAND 86-1309, Sandia National Laboratory, Albuquerque, NM.

Booker, J.M. and Bryson, M.C. (1985), "Decision Analysis with Applications to Proj ec t Management: State-of-the-Art Survey and Bibliography," American Journal of Mathematical and Management Sciences, 5, pp. 1-62.

Booker, J.M. and Meyer, M.A. (1985), "Sources and Effects of Correlation of Expert Opinion," LA-UR-85-1879, Los Alamos National Laboratory, Los Alamos, NH.

Cleaves, D.A. (1986), "Cognitive Biases and Corrective Techniques: Proposals for Improving Elicitation Procedures for Knowledge-Based Systems," Proceedings from Knowledge Acquisition for Knowledge-Based Systems Vorkshop, Banff, Canada, pp. 9,0-9,11.

Comer, M.K, Seaver, D.A., Stillvell, V.G. and Gaddy, C.D. (1984), "Generating Human Reliability Estimates Using Expert Judgment," NUREG/CR-3688, SAND 84-7115, Sandia National Laboratories, Albuquerque, NH.

Ericsson, K.A. and Simon, H.A. (1980), "Verbal Reports as Data," Psychological Reviev, 87, 3, pp. 215-250.

Gorden, R.L. (1980), Intarvieving: Strategy, Techniques, and Tactics, Dorsey Press, Homevood, ILL.

C-12

l Hogarth, R. (1980), Judgment and Choice: The Psychology of Decisions, Viley-Interscience, Chicago, IL.

Janis, I.L. (1972), Victims of Group Think: A Psychological Study of Foreign Policy Decisions and Fiascos, Houghton Mifflin, Boston, MA.

Kahneman, D. and Tversky, A. (1982), "Subjective Probability: A Judgment of Representativeness," D. Kahneman, P. Slovic, and A. Tversky (eds.), Judgment Under Uncertainty: Heuristics and Biases, Cambridge University Press, NY., pp.

32-47.

Lichtenstein, S., Fischhoff, B. and Phillips, L.D. (1982), "Calibration of Probabilities: The State of the Art to 1980", D. Kahneman, P. Slovic, and A.

Tversky (eds.), Judgment Under Uncertainty: Heuristics and Biases, Cambridge University Press, NY., pp. 306-334.

Martz, H.F., Bryson, M.C. and Valler, R.A. (1985), "Eliciting and Aggregating Subjective Judgments--Some Experimental Results," Proceedings for the Tenth Annual SAS Users Group International Conference, SAS Institute Inc., Cary, NC.

Heyer, M.A. (1984), "Human Factors Affecting Subjective Judgments," LA-UR 3176, Los Alamos National Laboratory, Los Alamos, NM.

Meyer, M.A. and Booker, J.H. (1987a), "Sources of Correlation Betvten Experts:

Empirical Results f rom Two Extremes," LA-10918-MS, NUREG/CR-4814, Los Alamos National Laboratory, Los Alamos, NM.

Meyer, M.A. and Booker, J.M. (1987b), "Selecting and Motivating Experts," A Guide for the Elicitation and Analysis of Expert Opinion, draft for book, Los Alamos National Laboratory, Los Alamos, NM.

Meyer, M.A. and Booker, J.M. (1987c), "Problems with Expert Data in Studies of Inter-Expert Correlation," Invited paper to be presented at the Joint Statistical Meetings, Aug 17-20, 1987, San Francisco, CA.

Meyer, M.A. (1987) "Handling Human Bias in Acquiring Knowledge for Knowledge-Based Systems," LA-UR-87-1221, Los Alamos National Laboratory, Los Alamos, NH.

Heyer, M.A., Peaslee, A.T. and Booker, J.M. (1982), "Group Consensus Method and Results," LA-9584-MS, Los Alamos National Laboratory, Los Alamos, NH.

Payne, S. (1951), The Art of Asking Questions, Princeton University Press, Princeton, NJ.

Seaver, D.A. (1976), "Assessments of Group Preferences and Group Uncertainty for Decision Making," Social Science Research Institute, University of Southern California, Los Angeles, CA. ,

l Seaver, D.A. and Stillvell V.G. (1983), "Procedures for Using Expert Judgment to Estimate Human Error Probabilities in Nuclear Power Plant Operations," ,

NUREG/CR-2743, SAND 82-7052, Sandia National Laboratories, Albuquerque, NM. I Spradley, J.P., (1979), The Ethnographic Interviev, Holt, Rinehart and Vinston, New York, NY.

C-13  !

l l

Stroud, H. (1981), "Communication Strategies: Strategies to Improve the Transfer of Knowledge," Communication Strategies for Advanced Technologies, Albuquerque, NH.

Tietjen, G.L. (1986), A Topical Dictionary of Statistics, Chapman and Hall, New York, NY.

C-14

l Distribution l US Government Printing Office Receiving Branch (Attn: NRC Stock) 8610 Cherry Lane Laurel, MD 20707 320 copies for RX James Abel (10)

Commonwealth Edison Co.

35 1st National West Chicago, Illinois 60690 B. B. Agrawal US Nuclear Regulatory Commission Office of Nuclear Regulatory Research 5650 Nicholson Lane Rockville, MD 20852 H. Banon EQE Inc.

3300 Irvine Ave., Suite 345 Newport Beach, CA 92660 Patrick Baranowsky US Nuclear Regulatory Commission Office of Nuclear Reactor Regulation Washington, DC 20555 i

Robert A. Bari Brookhaven National Laboratories Building 130 Upton, NY 11973 Richard J. Barrett US Nuclear Regulatory Commission Office of Nuclear Reactor Regulation Division of Radiation Protection and Emergency Preparedness Washington, DC 20555 Doyle Batt NRC Risk Analysis Unit EG&G Idaho, Inc.

P. O. Box 1625 Idaho Falls, ID 83415 Barbara Bell Battelle Columbus Laboratories 505 King Avenue Columbus, OH 43201 Dist-1

R. C. Bertucio Energy Incorporated 1851 So. Central Place, Suite 201 Kent, WA 98031 Dennis Bley Pickard, Lowe & Garrick 2260 University Drive Newport Beach, CA 92660 Gary Boyd Safety & Reliability Optimization Services 624 Glen Willow Dr.

Knoxville, TN 37922 Robert J. Budnitz Future Resources Associates 734 Alameda Berkeley, CA 94707 Gary Burdick Division of Reactor and Plant Systems US Nuclear Regulatory Commission 5650 Nicholson Lane Rockville, MD 20852 N. G. Cathey Idaho National Engineering Lab.

P. O. Box 1625 Idaho Falls, ID 83415 Frank Coffman US Nuclear Regulatory Commission Division of Reactor and Plant Systems Office of Nuclear Regulatory Research Washington, DC 20555 George Crane 1570 E. Hobble Creek Dr.

Springville, Utah 84663 Garth Cummings Lawrence Livermore Laboratory L-91, Box 808 Livermore, CA 94526 Mark A. Cunningham US Nuclear Regulatory Commission Office of Nuclear Regulatory Research Washington, DC 20555 Dist-2 l

Peter Cybulskis Battelle Columbus Divisien 505 King Avenue Columbus, OH 43201 Richard S. Denning Battelle Columbus Division 505 King Avenue Columbus, OH 43201 Ed Dougherty Science Applications International Corp.

P. O. Box 2501 Oak Ridge, TN 37831 Adel El-Bassioni US Nuclear Regulatory Commission Office of Nuclear Reactor Regulation Division of Radiation Protection and Emergency Preparedness Washington, DC 20555 Karl Fleming Pickard, Lowe & Garrick 2260 University Drive Newport Beach, CA 92660 J. B. Fussell JBF Associates, Inc.

1630 Downtown West Boulevard Knoxville, TN 37919 Norman L. Graves 2114 Sparrow Ct.

West Richland, WA 99352 Lon N. Haney Human Factors Organization EGtsG Idaho, Inc.

P. O. Box 1625 Idaho Falls, ID 83415 G. William Hannaman NUS Corporation 16835 West Bernardo Drive Suite 202 San Diego, CA 92127 '

Steven Hodge Oak Ridge National Laboratories P. O. Box Y Oak Ridge, TN 37831 l

Dist-3

I Peter Humphreys US Atomic Energy Authority Wigshaw Lane, Culcheth Warrington, Cheshire United Kingdom, WA3 4NE Brian Ives UNC Nuclear Industries '

P. O. Box 490 Richland, WA 99352 Alan Kolaczkowski Science Applications Int. Corp.

2109 Air Park Rd. SE Albuquerque, NM 87106 Jim Kolanowski Commonwealth Edison Co.

35 1st National West Chicago, Illinois 60690 George Klopp Commonwealth Edison Co.

P. O. Box 767, Room 35W Chicago, Illinois 60690 Dave Lappa Lawrence Livermore National Labs 7000 E Avenue Livermore, CA 94550 Josette Larchier-Boulanger i Electricte de France Direction des Etudes Et Recherches 30, Rue de Conde 65006 Paris, France Peter Lohnberg Expresswork International, Inc.

1740 Technology Drive San Jose, CA 95110 .

Herbert Massin Commonwealth Edison Co.

35 1st National West Chicago, Illinois 60690 Andrew S. McClymont IT-Delian Corporation 1340 Saratoga-Sunnyvale Rd.

Suite 206 San Jose, CA 95129 Dist-4

Joseph A. Murphy Division of Reactor Accident Analysis US Nuclear Regulatory Commission 5650 Nicholson Lane Rockville, MD 20852 Gareth Parry NUS Corporation 910 Clopper Rd.

Gaithersburg, MD 20878 Blake Putney Science Applications Int. Corp.

5150 El Camino Real, Suite C31 Los Altos, CA 94022 David Pyatt US Nuclear Regulatory Commission office of Nuclear Regulatory Research Washington, DC 20555 d

Dale Rasmuson (10)

Division of Reactor and Plant Systems US Nuclear Regulatory Commission 5650 Nicholson Lane, NL-005 Rockville, MD 20852 M. K. Ravindra EQE Inc.

3300 Irvine Ave., Suite 345 Newport Beach, CA 92660 Tom Ryan Division of Reactor and Plant Systems US Nuclear Regulatory Commission 5650 Nicholson Lane Rockville, MD 20852 1

Paul Shemanski US Nuclear Regulatory Commission office of Nuclear Reactor Regulation Washington, DC 20555 Stephen Sholly MHB Technical Associates 1723 Hamilton Ave.

Suite K San Jose, CA 95125 Desmond Stack Los Alamos National Laboratory Group Q-6, Mail Stop K556 Los Alamos, NM 87545 Dist-5 r _ _ _ _ _ _ _ _ _ -

Alan D. Swain 712 Sundown Pl. SE Albuquerque, NM 87108 Steve Topp EI DuPont DeNemours & Co. Inc.

Savannah River Laboratory Aiken, SC 29808-0001 Stephen D. Unwin Department of Nuclear Energy Division of Safety and Risk Evaluation Brookhaven National Laboratory Building 130, Brookhaven Lane Upton, NY 11973 William E. Vesely Science Applications Int. Corp.

2929 Kenny Rd., Suite 245 Columbus, Ohio 43221 Jim E. Wells Lawrence Livermore National Labs 7000 E Avenue Livermore, CA 94550 3141 S. A. Landenberger (5) 3151 W. L. Garner 6400 D. J. McCloskey 6410 N. R. Ortiz 6412 A. L. Camp 6412 W. R. Cramond 6412 S. L. Daniel 6412 D. M. Kunsman 6412 K. J. Maloney 6412 L. A. Miller 6412 A. C. Payne, Jr.

6412 T. T. Sype 6412 T. A. Wheeler 6412 D. W. Whitehead 6413 E. Gorham-Bergeron 6413 F. T. Harper 6415 J. C. Helton 6415 R. L. Iman 6440 D. D. Dahlgren 6447 M. P. Bohn (25) 6513 D. D. Carlson 8024 P. W. Dean Dist-6

U $. NUCL La ktGVLITomT Coeses,SS,oq t *t'o* Y 49 88" 8dat*** 8* I'oc see vs 4s . ,r ears 4.C70RW336 NUREG/CR-4836

, BUOGRAPH!C DATA SHEET in .% 1.uct,o~,o,t .sv .s SAND 87-0871

.......o..... . ..... ...

Approaches to Un rtainty Analysis in Probabilistic Risk ssessment

.o.n. c.,ce p ieo

.o%v.

y u..

i s.ui-oms. December g 1987 Michael P. Bohn, Timo y A. Wheeler, Se.n p ar,uvio ~~

Gareth W. Parry wo*'= "aa

,f l Decembe# 1987 __

,n..o....oo.:.wa.i.os. . .~o...t.so.oo s ow i. c , . .oncv ,.s. e . vvi w .. .

IV Sandia National Laborator s ,,,,0,c, 4 w ...

Albuquerque, New Mexico 87 5 ff F A1384

,, , c o o. so o.s.e ,.1,o , ... .so.,.4.so .co.isi m c- "

Y Division of Reactor and Plant ystems k.n"o'"'o*'

Office of Nuclear Regulatory R earch [ ? Final U.S. Nuclear Regulatory Commiss.t n * " "

• c o " " 8 o "*""'"

Washington, DC 20555 o

N[/

! ,a. 1.. . ~o r u f

f

. , . s , . e , ,a . , .,

y An integral part of any probabilistic sk assessment (PRA) is the performance of an uncertainty analysi to quantify the uncertainty in the point estimates of the risk meagpr considered. While a variety of classical methods of uncertaintypana sis exist, application of these methods and developing new chnigt s consistent with existing PRA data bases and the need for port (s bjective) input has been an area of considerable interest sj ce the pi eering Reactor Safety Study (WASH-1400) in 1974. This rep t presents he results of a critical review of existing methods fo performing u ertainty analyses for identifying d ta base limitations on  ;

PRAs, the various with methods.

special emphasis Both . . assical and Bay an approaches have been  !

examined. This work was f ded by the U.S. Nt lear Regulatory Commission in support of its ongoinc ull-scope PRA of th LaSalle nuclear power station. Thus, in addi n to the review, thi report contains recommendations for a e itable uncertainty anal, is methodology for the LaSalle PRA.

i is omoe gt .%..vs t . . alv mo.og  ;.'7o85

't *g'q '8 Unlimited

'tlic. 3'C6.ll8 C. tac %

aws

. .o %ta ias on= i o a.......

U _

U

.,w...o...cu

' t # 4.G .

1It U S Gort RNutNt e.,%Tswo ospect 13es-sm.s sto)s

- - ~ ~

h 1 1ANIRX 12055'5078877 US NRC-0 ARM-ADM OIV 0F PUB SVCS POLICY & PUB MGT BR-POR NUREG W-537 DC 20555 WASHINGTON