ML20151H768

Forwards Statisticians View of NRC Statistics, Presented to American Stds Assoc Advisory Committee on Nuclear Research

ML20151H768
Person / Time
Site:	Indian Point
Issue date:	02/04/1981
From:	Rubenstein D NRC OFFICE OF MANAGEMENT AND PROGRAM ANALYSIS (MPA)
To:	Abell T, Abramson E, Bassett H NRC
Shared Package
ML20151H308	List: ML20151H306 ML20151H487 ML20151H768 ML20151H775
References
NUDOCS 8305050287
Download: ML20151H768 (8)

Text

. .

,fas: .

1. - ,'t, UNITED STATES "

[ 3.3,f j _ NUCLEAR REGULATORY COMMISSION _

. ?.n..am - v.t.smiNoTot. o. c.2:sss t,.,i' .- '. a;/e i -

1. ewe

• H.ii O 4 T1- .

,, . ~ .

MEMORANDUM FOR: Those on Attached List FROM: David Rubinstein Applied Statistics. Br.anch

. Office 'of Management and Program Analysis

SUBJECT:

A STATISTICMN'S VIEW OF NRC STATISTICS - A PRESENTATION TO THE- ASA ADVISORY COMMITTEE ON NUCLEAR RESEARCH'

~

The encle' sed transcript of'my' talk to the ASA Ad Hoc Cc=mittee,on Nuclear Research might be of interest to you.

- - n *

~-

1 eld.d. ,,, h As.h5A -

David Rubinstein Applied Statistics Branch Office of Ma'nagement and Program Analys.is' En:lesure:

~

As stated . -

cc: See attached list '

~

? ~

. 030505028 8300u4~'

DR ADDCK 05000247 -

PDR -

ENCLOSURE 4

Di r ri: . -4:r.

.. Arra.s:r-H. ~asse:- .

T. Abell R. Sernero W. Bivins .

J. Burns .

M. Cu.1lingford S. Conver . ,

W. Dooly F. Goldstein ,

S. Hanauer J. Griesmeyer (ACRS)

J. Johnson -

R. Hartfield J . Ke r.t L. Lancaster - - ~

I. Kirk

'M. - Messinger '

W. Minner -

- --i S.'Me:1 ewer

r. v.oisome A. Thtdani -

J. Telford .

L. D. Ong . .

n. t1.sassion1

.. W. YeselY .

H. Orenstein .

R. Easterling T. Fine R. Me . sing . .

R. M::re O

e we.*

O

• eD

. 9e e

A STATZSTICIAN'S VIEW OF'NRC STATISTICS * .

MR. RUBINSTEIN: Since Carl started to talk about where I fit in, let me say that I belong to the Applied Statistics Branch whic,h is the central consulting group on statistical problems, serving all of NRC. We are not part of the risk assessment group, a,nd we have been relatively little involved in risk

. assessment.. Sometimes we get involved either because we push our noses into it and occasionally because we are asked.to. ,

As I was listening to the various speakers. I wanted to change ny speech, b'ut.I' sort of gave up'. I may ' repeat certain things which other speakers l ,

h' ave said. Please forgive me for that.

I should like to start.on an upbeat note. There have been improvements in NRC uses 'of statistics. I seem to sen,se a refre'shing quickening 'of pace; ,

at this meeting and at a meeting last monTth on risk assessment, I noticed -

r .

mdch more concern for the subleties of statistical problems and much more recognition and ackn'owledgment that the past performance has been less, than perfect. This quickening of pace, might be called acceleration ~- I should like to call it "a jerk." In 'ommon c speech a jerk is a sudden

= change in force or acceleration. Engineers us,e the word " jerk", or " jerk .

function" to denote the derivative of acceleration. One might speculate -

that the current large value of the jerk function is caused by anticipation of the judement which you are going to pass. Regardless of the merit of this speculation, the very fact that NRC has invited you to look over its shoulder is an extremely good onen. I am confident that your advice and t

guidance will help to keep the jerk, and perhaps the jerk of the jerk, positive for considerable time to come.

(l.aughter)

A review of statistics at NRC is an ambitious undertaking I cannot do justiceio. It covers a considerable time span; it covers many practitioners -

The latter include. groups at' NRC as,well as practitioners outside of NRC.

working under contract to NRC, as well as employees of vendors and licensees f Often' who are required to demonstrate some aspect of performance or safety.

NRC licensing does rely on analyses . .

perfomed by outsiders. .

I D

'An edited transcript of a talk given by David Rubinstein on Nov. 7,1980 in Washington, D.C., to the ASA (American Statistical Association) Ad Hoc Cormittee

f. -

_ on__N uclear Reculatory Research. _ _ _ _

Despite the introductory and sincere upbeat note,'here t is still much room for improvement in NRC,'s statistical applications. I shall deal in a broad brush

. fashion with some of the troublesome issues. I want to emphasize the word broad brush; NRC statistics is not a simple monolith. I indicated already evolution over time and that statistics is practiced in one fom or another by many individuals within and outside NRC. Obviously the various individuals and groups do not perfom uniformly well, nor does each perform unifomly well in all instances. ,

Despite diversity of application and quality, some deficiencies can be found rather frequently in statistical applications in the nuclear field. First I shall speculate on why there are rather Trequent shortcomings, and then discuss some of the specific issues, and finally end with some more or less philosophical

' questions. .

~

I believe that the penetration of the AEC and NRC by statisticians ha,s been minimal and rather late, The full subtlety and' complexity of statistical

~

~

problems in the nuclear field ha's not been appreciated by many in the nuclear field, and this includes managerial personnel. There. is widespread belief f- that physical scientis.ts with some acquaintance of stetistical methods can handle statistical problems adequately. This point relates to my next observation. .

- A technological, or perhaps even technocratic attitude, seems rather prevalent in the nuclear field. If there is a problem, there is a technological fix.

- Associated with this attitude or philosophy is an action-oriented approach that is not overly concerned with intellectual considerations; pragmatic considerations will co. Oftentimes, I am concerned whether the methods are even good pragmatism.

The acti,vist approach, whether pragmatically sound or not, is reinforced by pressure to provide answers and to provide them quickly.

l

\ . . .

l r

Before I turn to some of the specific issues, I would like to point out that statistics is at least a moderately successful scien.ce because of somewhat precise concepts and somewhat rigorous methodology. Unfortunately, in the nuclear. field, statistical concepts and methods get often blurred. It is now

.. well known.that th' Lewis e Committee called WASH 1400 inscrutable. ' Leaving WASH 1400 aside, I find much of the statistics in the nuclear field inscrutable.

. or vague. In fact, I -- and I believe other statisticians will -- find some analyses inscrutable, vague, questionable or wrong that might not have been regarded so by the Lewis Comittee.

- ~ .

~

Yesterday, we already noted the confusion of rates, probabilities, and expected values. There were some . incisive coments made about choosing distri-butions for maximum floods, and I would like to note 'that it was an NRC non-i statistician who pointed out that we are frequently concerned with mixtures of populations. .

- Among other items of concern I find: .

1. Confusion between ra'ndom variables and parameters is common even when no Bayesian approach is intended. -

2. Best estimate is a term that is extremely vague and frefluently used. It could ccme from ' data or from subjective belie'f. It could be a mode, a mean, a median a 50 percent confidence limit -- usually an upper confidence limit -- or what strikes somebody as best without clear elucidation of what is best. It may only be a matter of coincidence that the best estimate is the minimum variance estimate in a particular class of estimates.

3. You. Sill often hear the word " uncertainty"; a term that nuclear people seem to be particulary fond of. The first major technical report I reviewed in NRC used the word " uncertainty" where I think the .

2

- following terms might have been more appropriate:

e e

4-a) Random yariable, or perhaps a variation thereof such as random error or measurement error;

.b) Standard deviation; -

c) Confidence Ifmit; d) Error or bias; and perhaps here one could even become more specific whether this was an error in a parameter value or an error in the mathematical model; e) It was also used in that report for the density or distribution

~

function.

At . times I just did "not know what the intended meaning of the word was, and I am not sure that the authors always knew what they were talking about. Other uses and misuses include the following: .

a) The word " uncertainty" is also used as an equivalent to l what is called an upper bound, and this term is not well-defined.

j l

It seems to denote a large or very large value in a not-clearly-specified set of values. In nuclear jargon, " upper bound" is not a literal upper bound. ,

b) Finally, " uncertainty" is used as sort of a catchall phrase for what one might call Bayesian' uncertainties.

c) And, lo and behold, sometimes " uncertainty" is used as the condition of oeing in doubt. This particular usage of the word

T prefer.

d) R[gardless of the varied nature of the uncertainties, they often are f l ,

sum-root-squared to yield " total uncertainty." k'hile there is frequent l

recognition in the nuclear field of the diversity of uncertainties ~,

~

) ,

much sloppiness and con' fusion still exist.

Now let me turn to some methodological problems. Bay'esian statistics of one

~

sort or another is widely used. The material that 'was distributed to you contains some examples and critiques. I do not wish to elaborate on these in detail. However, even at the . risk of repetition, I should like to point out j that often th'e Bayesian framework is not clearly fonnulated and there is con .

siderable sliding between Bayesian statistics and frequentist statistics, and

, it can go both ways perhaps thiough several cycles in a particular analysis.

Often there is no explicit-mention or an indication of an a priori distribution, and even when a report starts with an a priori distribution, it may not end with

~

an' explicit posteriori distribution or probability. The probabilities seem to h' ave become absolute. _- -

j. .-

Another technical problem of NRC is' that'5f compon6nts or variance. 'Often we

-] erk'irri.RC with generic values which are evaluated and applied over presumably

'similar classes. Pla'nt to plant variabilities may.be ignored, or differences '

between d'ifferent components may be ignored. The ignored variation may be the dominant contributor to variability arid Bill Vesely in his talk clearly '

recognized this. There has been some progress in dealing with.the complex random

~

structure of the things NRC has worked with, b.ut more systematic exploration, clarification, and proper analysis of the random structure is indicated. .

Model II, or mixed models of the analysis of variance are not well recognized in' NRC. For that matter, Model I may be ignored.

As Bill Vesely pointed out, human factors and common cause problems are important, or perhaps even dominant contributors, to risk. It is extremely difficult to model these convincingly and to find appropriate data for ' estimation of parameters.

' ~

'In NRC terminology, there are great uncertainties with respect to these areas.

This ' leads to some philosophical questions.

In view of large and not necessarily well understood uncertainties, can one properly qua,ntify uncertainties? How should numerical analyses be used? How

- should they'be comunicated? The subject of comunication is a large one in / ,

itself. . - -

l ._ ._. . . -. . -

A related 'ouestion is what should be the proper role of subjective judgment in go'vernmental policy and regulation and how should one deal with subjective judgments and how should they be presented to the public and the political' representatives? ~

I was going to stop here, but Mike Cullingford stimulated me to say that perhaps we.shouldn't argue about whether we want Bayesian statistics or classical statistics. Perhaps we should ask the question, what is good scientific inference? -

Thank you.

W 4

es l

t'-

I i  !

e me

, s -

^ O *

. e _ _._.._,.