ML20151H306
| ML20151H306 | |
| Person / Time | |
|---|---|
| Site: | Indian Point  |
| Issue date: | 10/26/1981 |
| From: | Rubenstein D Office of Nuclear Reactor Regulation |
| To: | |
| Shared Package | |
| ML20151H308 | List: |
| References | |
| NUDOCS 8301120436 | |
| Download: ML20151H306 (12) | |
Text
. . ..
DRAFT October 26, 1981 Random Thoughts on Uncertainties, Risk Analysis, and Nuclear Regulation David Rubinstein Note:~
This paper reflects the views of the author. It should not be construed as-a rolicy statement of the Nuclear Regulatory Comission.
( -
l l- .
M Copy His Been Sent to PDR g
1 bg[u 33o1U0 ggg -
enctosuas s
DRAFT
- Random Thoughts on Uncertainties, Risk Analysis, and Nuclear Regulation David Rubinstein _
- l. Introduction Okrent's testimony [Ref 1] stimulated me to write some of my thoughts on uncer-tainty, risk analysis, and judgment in nuclear regulation. These thoughts re-present my personal view. Except as noted below, I do not attempt to review other people's work in this area. I shall use Okrent's testimony and a paper
] [2] cited therein as a point of departure for developing my views.
I shall offer my reaction to the issues raised by Okrent and develop my own perspective on those issues which I think NRC should consider. This report has
" been written primarily for NRC insiders. However., to accomodate likely outside
- readers, I may explain some matters familiar to, NRC personnel. Here I convey the t,opic of Okrent's tes'timony by citing its opening sentence.
"My understanding is that the general focus of discussion for my appearance before the Nuclear Safety Oversight Committee today
- is to be the matter of how the Nuclear Regulatory Commission (NRC) makes decisions concerning the public health and safety in the I presence of very considerable technical uncertainty."
Before getting on with the subject matter I should like to acknowledge that I have had little direct responsibility for probabilistic risk assessment at NRC. I have' been an observer more or less on the fringes l .
of'NRC risk assessment. While this may indicate shortcomings, it may also provide a detached perspective.
In Section 2, I shall comment on Okrent's testimony and on how it fits into the evolution of'probabilistic risk analysis at NRC. I discuss my personaI1 view of the role of uncertainty in nuclear regulation in Section 3. Section 4 will elaborate on some shortcomings of analyses
~
other than large uncertainties; it also deals briefly with the role of judgment "in regulation and Okr'ent's' call for criteria for judgment.
- e
e
- 2. A Partial Review of Okrent's Testimony and of the Evolution of Risk Analysis at,NRC First I should state that I greatly appreciate Okrent's testimony.
It comes closer. t.o my own philosophy than any other NRC document I o have read on this subject matter. On basic issues of NRC decision making, Okrent and I may be cousins, but not necessarily kissing cousins. I look upon Okrent's testimony as a stage in an evolution of how AEC/NRC deals with risk. I shall offer a grossly over-simplified outline of this evolution. This outline may do injustice to those who, in NRC jargon, have remained determinists or who have, contrary to the prevailing tendency (or just lip service), drifted toward " determinism". (In NRG the " determinists" place relatively
,little reliance on probabilistic' risk analysis for regulatory pur-poses; they rely primarily on " judgment".)
The evolution has the following stages:
A. "All-is-safe" stage: The conception that nuclear power is safe with relatively little formal analysis as backup.
. B. " Wash-1400" stage: Development and frequent use of " scientific" ,
or " technical" approach primarily by "probabilistic risk assess .
ment" with loads and loads of fault trees. Still marked dis-agreement on the relative merits of judgment and probabilistic ,
l risk assessment.
C. " Post-Lewis-Committee" stage: Greater reliance on probabilistic l risk assessment (and peer review) with strong admonition for evaluation and statement of uncertaintiet - even more fault trees.
I shall discuss below how Okrent is at least in an advanced part of the Post-Lewis-Committee stage. .
ed e
g 9 O
e e .
e
,n . -
In his introduction Okrent notes that, there is profound uncertainty in many regulatory &ctivities. In the body of his testimony he makes
, the additional points related to specific cases of NRC regulatory a'ctivity, or the lack thereof. Among these are:
- 1) Advocacy for more plant specific analysis and decisions.
- 2) Tough questioning of action criteria related to probabil-ities of severe core damage, and of permitting plants to be operated under some stated circumstances.
- 3) Critique of imprecise terminology used in risk assessment.
- 4) Critique of incons.istencies in regulatory prescriptions.
Observations such as these have been made before. What gives Okrent's l
testimony special force is the cohesiveness and toughness of the entire testimony. It is in contrast to more typical (and lenient) attitudes and standards under the Post-Lewis-Committee stage. The final portion 1
of Okrent's concluding comments also seem to indicate a change. To quote:
. "...Despite these potentially serious difficulties with probabilis-tic analysis, it appears that an effort to quantify the risks, or the increment in risk, associated with a particular safety issue
, ' is a worthwhile part of the process leading to decision.. The assumptions must be clearly stated. The uncertainties should be defined, as possible. Criteria for judoment should be developed and independent peer review should be used.
"Ah yes, how should we expect the NRC to make decisions on matters like hydrogen and non-seismically qualified auxiliary feedwater systems, which involve an atmosphere of technical uncertainty?
"With difficulty." [ Emphasis added.]
~
i The words I underlined seem to convey something less than an absolute faith in probabilistic risk assessment. First, I shall briefly discuss the reference to judgment. Even in the Post Lewis tradition, judgment has been called for with varying emphasis ranging frum the notion that probaSTlistic analysis should categorically supplant ' judgment to actual I reliance _on probabilistic analysis only if it confirms one's prior judg-ment. SIJch extreme positions may be rarely stated, but I believe they come close to some persons' outlook or behavior. Undoubtedly, Okrent calls for judgment and he calls for the development of criteria for judg-ment. One plausible implication of that recommendation is that prob-
= I- ..~+, --. - - = , -- -- ,
4 abilistic risk assessment is not always trustworthy; therefore we must
.use judgment and attempt to rationalize and formalize the judgment process.
The call for the development of criteria for judgment again seems to go beyond Post Lewis stage. I shall return later to this aspect._
I also wish to coment on the final phrase "With difficulty." I have heard speakers in the Post Lewis era give eloquent and penetrating description of the difficulties with probabilistic risk assessment and yet conclude with an optimistic prognosis of its use. What Okrent finds is difficulty from beginning to end. However, in the context of his testimony the phrase "With difficulty" seems to imply a hope or expectation of success. Because of this implication, I may part company with Okrent. In any case the phrase is not precise. Nor is the sentence "...to make decisions on...non-seismically qualified auxiliary
.feedwater systems." Is it just a matter making decisions or is a matter of
~
making good decisions-with difficulty? (Presumably, Okrent meant good decisions.)
What does difficulty mean? Can NRC solve the problem in one year, or in ten years. How many man-years or man-millenia are required? Would it require giant shake tables on which critical configurations o.f piping could be stressed with , ~
. simulated earthquakes? Would it require eight full scale experimental power plants -- each with an additional outer containment building and machinery to I
absorb radioactive iodine? Perhaps we should opt for nine experimental
- power plants. With eight nuclear power plants we can run a full factorial of three fac crs, each at two levels. .With nine we could accommodate a Graeco-Latin sqtare with three levels for four factors; however, we could not estimate interactions. Neither design would provide a clean estimate for error; there-fore should we double these numbers to achieve replication in each cell?
The last few sentences have been deliberately couched in statistical
~
jargon and are facetious in this context. However, they are valid teasers.
What constitutes plausible evidence (never mind scientific or compelling evidence) as a basis for regulatory action? How much should a probabil-istic risk assessor know about statistics and the principles of design of experimep s-even if he does not conduct experiments? -
l l
7 .
5
- 3. A Personal View on Uncertainty in Nuclear Regulation The Post Lewis idea that we will substantially advance nuclear regu-lation if we just quantify the uncertainties (and have peer review) is o in my opinion more a matter of illusion than substance. The tautology "If one does not know,' one does not know" has obviously a much firmer basis, and, .in my opinion, has more relevance to regulation. I do not-believe that we can quantify uncertainties in a meaningful way over the whole range of regulatory problems. I find the following propo:ition difficult to accept as a general rule. We may not understand a phenomenon very well and are therefore uncertain about it; yet at the same time we understand it and o'ur process of thinking about it so well, that we know the nature of the error in our thinking and therefore can quantify the uncertainty. I find empirical confirmation of my somewhat philosophical probing in an occasional gesture by some NRC engineers.
The arm is raised with an open hand; the arm is pulled down and the hand is closed - the value of interest and its uncertainty was pulled out of the air.
. I have not undertaken a review of studies on. uncertainty. In his testimony, Okrent cites a study of his [2] which at least in some respects is similar to what I read or heard elsewhere. In his testimony,. Okrent summarises that "...
seven [ respondents, i.e.] seismologists and geologists making their judgments independently, usually differed by a factor of 10-3 to 10-4 in their estimates of return frequencies for wide range of earthquakes. at eleven different re-actor sites." Refe7e~nce [2] also deals briefly and ra5her vaguely with respondents' assessment of their uncertainty. While it is not clear what they understood by uncertainty (standard deviations (of what randem variable or population), maximum error, or whateveh, from a pragmatic point of view their estimates of the occurrence rat' es and their assess-ments of the " uncertainties" in their estimates are not consistent. Of two respondents one "... generally estimated an uncertainty of 10-20%",
and another, "...of a factor of two in the probabilities per year." .
However, in several estimated return frequencies they differed by as much as a factor of 1000. Other respondents' (vaguely stated) estimates of uncertainty also do appear too small in terms of the spread of the
- Are they personal uncertainties, or are they in some sense objective?
Are they dependent on specific theories which.in turn may not be firmly established? Were such theories shared by several or all assessors?
1[ .
6-estimated probabilities among all respondents.
While the case study of Okrent may be a rather extreme example, I believe that on many NRC problems o a) 'the uncertainty is large - one, two, or perhaps even three or
. more orders of magnitude:
b) the nature of the uncertainty is vague c) whatever the conceptual basis of uncertainties they are mis-estimated from a common sense point of view and often under-estimated or grossly underestimated. '
i Often these large and not well understood uncertainties need to be
. combined and propagated into a " final" uncertainty with no compelling prescription for combining and propagating uncertainties. Frequently
- one would expect that in some sense the uncertainty of the final re-sult is larger than that of any of its inputs and therefore very large and that points (b) and (c) above are also amplified. Let me combine all these aspects of uncertainty under the label uncertainty complex.
~
There are many more or less specific facets of uncertainty that merit l ' consideration. Later on I shall deal with some psychological a.spects of analysis which re) ate to uncertainty. For sake of brevity I shall deal with only one more aspect under the label of futurology. The primary purpose of risk analysis is to assess future risk. While it is indeed reasonable to project from the past and'present such pro-jections are not error free. Reliability growth is a very plausible l ,
effect and most likely the dominant one. However, there are also potential adverse changes the likelihood of which is speculative.
- 1) Bathtub curve: most of the reliability data is from commercial
. reactors less than 20 ylars old. There is a possibility that some failure rates of vital components or systems could rise
~
sharply after 20 years.
- 2) State of emergency plans 5 to 10 years hence. ,
- 3) 'l presumed safety feature backfires.
- 4) Sabotage from dissatisfied labor or terrorists, t 5) Several years of successful operation bringing about
. complac ency.
- 6) Economic conditions promoting shortcuts.
0 ^ ^ '^
_ . , , - - . , , .._ .1 -D - - , - - - - -
v.--- p
This list could probably go on; quite'possibly if a very detrimental change were to come it might not be even: thought of now.
There are various ways of viewing the uncertainty issue in nuclear regulation. Ok're'nt in his first sentence sayi "...(NRC) makes de-cisions concerning the public health and safety in the presence of
. very considerable technical uncertainty." My reaction is that the NRC decision process is beset with an overwhelming uncertainty complex.
It brings to mind the emperor without clothes. This must be an un-comfortable position for NRC as it would be for any regulating agency.
- However, it is not a circumstance about 'which NRC needs to be apolo-getic. We are in a new domain with many phenomena abo,ut which de-pendable knowledge has not been obtained. As Okrent points out in his introduction many, regulatory agencies are regulating under similar circumstances. In fact, uncertainty complexes beset our lives as in-dividuals as well as collectively as a nation. They range from dif- ,
ficulties in raising children to problems of national defense. The ,
latter may affect the likelihood of nuclear war which in comparison would make any nuclear ' power plant catastrophy look puny.
If my assessment of the NRC~ uncertainty complexes is correct, then NRC has three broad choices. -
a) It can take the current type of risk analysis at face value, make regulations in accordance with their results, and bluff on the validity of its decisions.
b) It can start or continue with vigorou's efforts to make the probabilistic risk analysis more rigorous and convincing.
c) It can explicitly acknowledge profound uncertainty and re-gulate with recognition of this limitation.
These, options are not mutually exclusive. One can use various shadings of these options, and the shadings may vary with the circumstances as indeed h the case now. Superficially option (b) appears attractive. ,
However, I believe that in terms of " reasonable" precision many of our problems are intractable and will continue to be so. I would expect'
- ~
/ .
e b 0 O - - - - -- - - .- - , -
only a slow nibbling away at a problem here or there. Option (a) does - -
have the advantage that in principle it maintains a stronger degree of authoritativeness than option (c). Undoubtedly it is tough to regulate without an air of aethoritativeness. However, option (a) may lead to
. bad decisions and may not be viable in the democratic process-in.which NRC must. function. This brings us to facing up to option (c). In a
} fashion option (c) was or is operative for persons who favor the judg-mental approach to regulation. Option (c) in my view does not necessarily call for judgmental approach. The intended thrust of option (c) is that
, both judgment and probabilistic risk assessment are very limited tools
- for assessing. nuclear risks or for optimizing benefits against penalties with respect to nuclear energy.
- 4. Some Additional Discussion of Analysis and Judgment I do -not attempt here to resolve what role a highly fragile risk analysis should play in reculation, and in particular its relationship to or in competition with judgment.* This subject matter is outside my area of competence. I can only present some ideas related to it and state what my inclinations are. Before doing that I sh,all summarize some points _
that weaken the case for probabilistic risk analysis.
Particularly in difficult problems, mathematical analysis provides its own .
straightjacket. The analyst will only use methods he or she knows and that do not require an inordinate amount of effort. Thus certain types of failures will be treated as independent, constant failure rates will be postualted, or generic values will be applied to differing members of a class. Even if the analyst is sophisticated enough to use one of the few models for dependence, he or she is still limited to the few known models,, all of which have i
l'mitations. The phenomena of nuclear power plants are very complex; many cannot be dealt with realistically with workable models. The thought proces,ses of the analyst may in part be dictated or influenced by the
' medium of his choice; i.e., the mathematics that is practically available to him. -This applies to the best analyst as well as the poorest; even -
though the former can deal more deftly with limitations and will generally I . .
- C. Bennett and M. Ernst pointed out that it is the relationship or interaction between analysis and judgment that is paramount; I agree.
Nevertheless, there are differing views on the relative reliance to, bt 91 aced on each. - - m
.g.
have more awareness of the limitations. The straightjacket idea is illustrated by the rather frequent and sometimes unjustified criticism directed at the mathematical analyst: "You have a beautiful s31ution
, to the wrong problem."
Analysts differ in their awareness of the limitations of their analyses and in their efforts to report that awareness. Some become so engrossed with their achievements that they do not see the shortcomings. Others while perhaps recognizing the shortcomings may not report them; in fact some may present their ana' lysis with puffery. Perhaps the majority will state briefly and rather inconspicuously some of the assumptions and limitations of the analysis. It is my impress. ion that only a minority of analysts at NRC drive home with force and proper elucidation the limita-
, tions of their analysis. The analyses are provided directly or indirectly -
to " users" which may be colleagues, supervisory personnel, the com-missioners, or ultimately the public. Ev'en if limitations of the analysis 4
are serious and stated forcefully, the user has a strong tendency to take
, the results at face value - particularly if 'they confirm his predilection, or' seemingly help him to get out of a dilemma. Analysis no matter under what label (mathematical, statistical, risk) has a ring of authority and authenticity. It is often unquest'ioningly accepted by the, lay analyst.
In fact even capable analysts are affected by the halo effect of " analysis".
Unless a) they give other persons' analysis careful scrutiny
, b) had experiences with the type of analysis under review, they might accept the results with less reservations than they deserve. .
Besides the wrong psychological impact, analyses often have unjustified staying power. Early analyses become the basis of later analyses, thus relieving the later analyst of having to deal with the tough issues of not well* understood phenomena. It is much easier to cite than to inves- '
tigate and think through difficult problems. . Individuals and institutions will defend their analyses and . insist they are valid, discounting evidence e .
_O.
- ~ ~ ~ '
10.
to the contrary. While I sense in NRC lessening rigidity in the defense and use of past analyses, the problem of unjustified staying power is still with us.
If formal analysis has so many limitations-I do not claim a complete
- catalogue of them - are the results of judgment to be trusted more than those of formal analysis? As I indicated earlier I lack competence to answer this question. For the sake of discussion let me speculate. It is conceivable that a knowledgeable person with a subtle mind might pro-vide better answers for the following reasons. He or she:
a) is not bound by the fonnalism of analysis -
b) brings to bear conscious and unconscious knowledge and wisdom; and c) has a broader perspective on the problem than the formal analyst.
On an intellectual level I am not convinced by such speculation, and even if true in some or most cases, how do we decide which are these cases? Also, how do we decide who is the most' knowledgeable, wise, and subtle person to provide this ' judgment? Yet'on an emotional level, I tend toward judgment over analysis in nuclear regulation for the fol-lowing reasons-
- 1) Too much credibility is given to analysis.
l
- 2) The staying power of analysis is too strong.
- 3) The judgment and its limitations are often more readily under-
! stood than the analysis.
While some of the same cause for points (1) and (2) also functi,a ir judgment, I believe that they function less intensively for judgment. I bel.ieve that judgment will in general be accorded less unjustified authority and less staying power than analysis.
.9
'Okrent states " Criteria for judgment should be developed...". NRC has difficulty developing criteria for good analysis. I believe it is even ~
more difficult to develop criteria for good judgment. Despite the bleak outlook, such an effort may be worthwhile. I believe it should be com-bined with a review of the philosophy of regulation, in particular with i optibn (c), namely that NRC explicitly acknowledge profound uncertainty.
- __ , = _ - _ - ._
_11 and regulate.with recognition of this limitation. The review group should have profound thinkers and good pragmatists (not necessarily mutually exclusive). It could conceivably be supported by several regulatory agencies having common problems.
- 5. Postscript
, It might be inferred from the te$k above that I believe that probabilistic risk analysis is useless, or nearly so, for regulatory purposes. This is not my point at all. I expressed my concern with various limitations of analysis and its misuse particularly with respect to uncertainties. I do believe that probabilistic risk analysis does have a vital role to play in NRC regulations. The why and how of this role I prefer not to tackle in this-document. Some issues of risk abalysis in NRC are raised in my talk to the ASA Ad Hoc Advisory Committee on Nuclear Research (3).
- 6. Acknowledgments Dr. Carl Bennett, Mr. Malcolm Ernst, Mr. William Maher, Mrs. Esther Rubinstein, Mr. Joel Rubinstein, and Mr. Norman Wagner provided constructive criticism. I thank them for their help. *
- 7. References
- 1. David Okrent, Testimony to the Nuclear Safety Oversight Conr.ittee,
' January 20, 1981, Santa Barbara.
- 2. A Survey of Expert Opinion on Low Probability Earthouakes, Annals -
of_ Nuclear Energy, Volume 2, pp 601-614,1975.
- 3. David Rubinstein, 'A Statistician's View of HRC Statistics, edited transcript of talk to the ASA (American Statistical Association) -
Ad Ho'c Advisory Committee on Nuclear Research, November 7,1980,
- Washington, D. C.
e
, .