ML20235B003

Comments on Code Scaling,Applicability & Uncertainty Methodology for Determination of Uncertainty Associated W/ Use of Realistic ECCS Evaluation Models.Addl Comments by ACRS Member Hw Lewis Also Stated

ML20235B003
Person / Time
Issue date:	09/16/1987
From:	Kerr W Advisory Committee on Reactor Safeguards
To:	Zech L NRC COMMISSION (OCM)
References
FRN-52FR6334, RULE-PR-50 AC44-2-03, AC44-2-3, ACRS-R-1274, NUDOCS 8709230491
Download: ML20235B003 (4)

Text

(Jefs2-/a74

@ Rtcy Jf' o

UNITED STATES

,3cf(g NUCLEAR REGULATORY COMMISSION g

//. j ADVISORY COMMITTEE ON REACTOR SAFEGUARDS WASHINGTON, D. C. 20555 o,

&g

%/

....+

Septerrber 16, 1987 The Honorable Lando W. Zech, Jr.

Chairman U.S. Nuclear Regulatory Commission Washington, D.C.

20555

SUBJECT:

ACRS COMMENTS ON COE SCALING, APPLICABILITY AND UNCERTAINTY METHODOLOGY FOR DETERMINATION OF UNCERTAINTY ASSOCIATED WITH THE USE OF REALISTIC ECCS EVALUATION MODELS During the 329th meeting of the Advisory Committee on Reactor Safe-l guards, September 10-12, 1987, we reviewed the methodology developed by I

the NRC Office of Nuclear Regulatory Research for determination of the overall uncertainty associated with the use of realistic models, includ-ing related computer codes, for the calculation of thermal-hydraulic phenomena associated with loss of coolant accidents (LOCAs).

In our I

review, we had the benefit of discussions with representatives of the l

Office of Nuclear Regulatory Research (RES) and the Office of Nuclear l

Reactor Regulation (NRP,).

Subcommittee meetings during which this topic l

was discussed were held on April 29-30, 1986, August 28, 1986, April 29-30, 1987, and August 4, 1987.

We also had the benefit of the docu-ments referenced.

A recently proposed revision to the ECCS Rule (10 CFR 50.46 and Appendix K) will permit use of realistic or "best estimate" methods in demon-l strating that a peak cladding temperature (PCT) of 2200 F will not be i

exceeded during a LOCA.

This is in contrast to the original version of the rule which insisted on the use of a number of conservative assump-tions which were believed to provide an overestimate of PCT large enough to account for uncertainties. With the new rule change, a licensee may l

demonstrate that the calculated PCT, when adjusted with an appropriate allowance for overall uncertainty, has an estimated 95% probability of not exceeding 2200"F.

In our Septembei 16, 1986 letter to you comment-ing on the proposed ECCS Rule, we noted the following:

"The acceptability of realistic evaluation models rests on the development of satisfactory methodology for determination of the i

overall uncertainty.

Most of the development work needed here is either ongoing or planned by the Office of Nuclear Regulatory Research.

We recommend that the methodology used to evaluate uncertainty be subjected to peer review. We also wish to review this work."

00/3o 41 yer 1

l

1 t

U The Honorable Lando W. Zech,-Jr. September 16, 1987

)

RES'has developed a method for quantifying uncertainty in PCT which it refers to as the Code Scaling, Applicability and Uncertainty (CSAU)

-)

evaluation methodology.

CSAU is designed ' to address ~ uncertainties in j

the capability of a code to extrapolate small-scale test data to' full i

scale, to correctly assess a' particular sequence of events, and to 1

account for variability in important parameters.

The focus. of CSAU is on the important thermal-hydraulic processes with detailed attention

-i given only to those processes which contribute importantly to overall i

uncertainty.

The end product of the CSAU method is 'an estimate of the total uncertainty associated with the calculation of a key parameter j

(e.g., PCT) by a given realistic thermal-hydraulic code for a particular plant and a particular accident transient.

It must be recognized that absent an abundance of full-scale LWR plant-

)

transient data, it is necessary to rely substantially on engineering

.i judgement in lieu of a rigorous statistical. analysis.

The CSAU meth-odology systematizes the application of this judgment for the derivation of a quantitative allowance for uncertainty.

We believe that the CSAV method proposed by RES offers an acceptable means to estimate uncertainty associated with the use of realistic l

codes.

However, we wish to note the following:

• The CSAV methodology has not yet been tested over a wide range of I

applications.

Currently, RES is in the process of demonstrating

.i the applicability of the method by using it to determine the uncer-tainties resulting from a large break-LOCA. calculation using the l

TRAC PF1/ MOD-1 code.

While it appears that CSAV will be success-l fully applied to TRAC, we recommend that RES complete an adequate l

c evaluation before the methodology is judged acceptable for use in regulatory actions.

Before CSAU can be applied to a given code, complete documentation (e.g., code manual, model and correlation quality assurance docu-ment, and assessment reports) is necessary.

In the past, such thorough documentation has not always been available for licensing codes.

We recommend that steps be taken to ensure that future development of codes for licensing activities be performed in a manner that ensures completion and availability of needed docu-mentation before the code is released.

• The codes used to analyze thermal-hydraulic behavior are very large and complex.

Validity of calculated results is dependent on the competence of the code user and the way in which the code is used.

For CSAU to be effective, the code developers, assessors and users must use the code consistently.

We recomend the-NRC Staff take the necessary steps to ensure that proper controls are established.

I f-d The Honorable Lando W. Zech,'Jr. September 16, 1987 i

• In order to ensure the ultimate success 'of the method, we believe it is necessary for RES to direct its -experimental thermal-hydraulic programs appropriately to the needs of CSAU.

These experimental programs include the MIST, 2D/3D, and ROSA-IV coopera-tive efforts.

• We wish to caution that use of the CSAU method for. regulatory

)

applications will require the maintenance of an ongoing high level of competence and experience on the part of the NRC Staff members.

1 We suggest that the NRR call upon RES for such support as neces-sary.

'.3 We are encouraged by the move toward the use of realistic calculations for ECCS/LOCA phenomena.

We intend to follow the progress of this 1

effort closely, and we wish to be kept informed..

i Additional comments by ACRS Member Harold W. Lewis are presented below.

i Sincerely, 1

J 1

William Kerr-l Chairman l

l l

Additional Comments by ACRS Member Harold W. Lewis I support the Committee's -letter, but do wish.to add 'some cautionary notes about the misuse of some familiar words,. which can lead to po-tential misuse of the CSAU (so-called) methodology.

To begin with, I support the move to " realistic" evaluations, since I believe that all evaluations should be made as honestly and realistical-ly as possible, after which regulatory conservatism can-be applied cleanly and openly.

That is the thrust of this effort, and is fine.

l Unfortunately, however, the words "best estimate" are often used inter-changeably with " realistic" to describe calculational techniques, and that is an error.'

To a statistician, a best estimate is an estimate taken from the top of a probability distribution, and that'is simply a different idea.

This is not sophistry, since the misunderstanding of words that have established technical meanings can lead to incorrect.

I calculations. To call an apple an orange does not make it one.

We were also briefed about a set of calculations in which parameters and assumptions were varied to provide. a feel for the sensitivity of the' l

l E

1 I

The Honorable Lando W.' Zech, Jr. September 16, 1987 results to the specific assumptions made.

That is a reasonable way to learn about the sensitivity, but is not a way to learn about the "uncer-tainty" in the result, as any statistician.would understand the word uncertainty.

Statistical uncertainty in its simplest form is based on the concept of random sampling from a population. of known characteris-tics but unknown parameters'.

In that case, one can learn the uncer--

tainty in an estimate of a parameter by studying the variance in a set' of measurements, but that is not the situation here, where the variance in the results bears no relation whatever to.any uncertainty, in any credible statistical sense.

The only reason for saying this is that in the familiar' case of a normal distribution of sample measurements, one can estimate the uncertainty from the variance, and thereby estimate the probability that the mean of the measurements differs from the true value by any ratio. One can also estimate " confidence levels," but that is another saga.

None of that is true here, and this is again not sophistry.

In particu-lar, the draft Regulatory Guide supporting the proposed rule has state-ments about the "95% probability limit," " confidence. level," and such things, and even states that the "use of two standard deviations for evaluating the 95% probability level is acceptable."

None of this is possible within the framework described, and simply reflects confusion on the part of the Staff about fundamental statistical concepts.

I still support the letter and the' program, since it is a major step forward, but repeat a recommendation I have-made many times:

the NRC would benefit greatly by hiring a few good statisticians. One cannot do competent safety analysis in the presence of uncertainty (popular use of the word) without doing the statistics carefully.

References:

1.

CNuclear Regulatory Commission, Proposed. Rule, " Emergency Core Cooling Systems, Revisions to Acceptance Criteria," February 26,

1987, 2.

U.S. Nuclear Regulatory Commission, " Request for Comments on Draft Regulatory Guide, 'Best Estimate Calculations of Emergency Core Cooling System Performance,'" March 1987.

3.

U.S. Nuclear Regulatory Commission, NUREG-1230, " Compendium of ECCS Research for Realistic LOCA Analysis," April 1987.

-