ML18178A431

Technical Letter Report - Important Aspects of Probabilistic Fracture Mechanics Analyses - Final

ML18178A431
Person / Time
Issue date:	09/14/2018
From:	Michael Benson, Matthew Homiack, Matthew Kirk, Patrick Raynaud NRC/RES/DE/CIB
To:
Raynaud P
References
TRL-RES/DE/CIB-2018-01
Download: ML18178A431 (48)

Text

Technical Letter Report TLR-RES/DE/CIB-2018-01 Important Aspects of Probabilistic Fracture Mechanics Analyses Date:

September 14, 2018 Prepared in response to Task 1 in User Need Request NRR-2016-004, by:

Patrick Raynaud Senior Materials Engineer Component Integrity Branch Mark Kirk Senior Materials Engineer Component Integrity Branch Michael Benson Materials Engineer Component Integrity Branch Matthew Homiack Materials Engineer Component Integrity Branch Division of Engineering Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555-0001

DISCLAIMER This report was prepared as an account of work sponsored by an agency of the U.S. Government.

Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use, or the results of such use, of any information, apparatus, product, or process disclosed in this publication, or represents that its use by such third party complies with applicable law.

This report does not contain or imply legally binding requirements. Nor does this report establish or modify any regulatory guidance or positions of the U.S. Nuclear Regulatory Commission and is not binding on the Commission.

Executive Summary In response to increasing interest in probabilistic fracture mechanics (PFM) analyses to support regulatory decisionmaking, this report develops the concept of a PFM analysis methodology and outlines important considerations for a high-quality and high-confidence PFM analysis. Fracture mechanics analyses have historically been performed deterministically; however, in response to the need to quantify risk in regulatory decisionmaking, the use of probabilistic approaches has increased.

While both deterministic and probabilistic methods are only as good as the models developed to represent reality, they differ in the way that they treat uncertainty. Each approach has its advantages, but increased emphasis on cost-benefit analyses and risk-based metrics may offer more opportunities for using probabilistic methods.

This report presents a general framework to describe, perform, and evaluate a PFM analysis. The important pieces of a PFM analysis that should be considered include models, inputs, uncertainty characterization, probabilistic framework, and PFM outputs.

• Models can be categorized into different types, but in all cases, model verification, validation, and uncertainty quantification are key steps in gaining confidence in the adequacy of the models used.

• Treatment of random inputs may consist of constructing probability distributions; determining input bounds if applicable; and quantifying any assumptions, conservatisms, or dependencies between inputs.

• Uncertainty characterization and treatment are at the core of a PFM analysis. In many PFM analyses, it may be useful to separate epistemic and aleatory uncertainty. Uncertainty identification, quantification, and propagation are essential elements in describing a PFM methodology or analysis. The proper choice of sampling techniques is also an important step that needs justification. This report discusses concepts and methods to verify and validate a probabilistic framework.

• Ways to demonstrate PFM convergence include varying sample size and sampling strategy, as well as performing stability analysis. Output uncertainty analysis can take various forms depending on the problem being analyzed. Sensitivity analyses can help to identify the drivers of uncertainty for a given problem or output. Sensitivity studies are useful in understanding which parameters drive the issue being investigated and in showing that some expected trends are indeed reflected in the analysis results. This report presents methods to perform such studies.

Following this report, the next step in developing guidance to perform and evaluate PFM analyses is the preparation of a draft regulatory guide, accompanied by a more detailed technical basis NUREG document. In parallel, a pilot study will be defined and executed to test the draft guidance produced and to identify deficiencies and areas that need improvement. The lessons learned from this process will be incorporated in a final guidance document, and the technical basis NUREG will be revised accordingly.

i

Creating a PFM analysis by following the regulatory guidance that will be developed in this project may be only one part of the acceptance of analyses in a risk-informed submittal. The U.S. Nuclear Regulatory Commissions process of risk-informed decisionmaking is rooted in several tenets that are aimed at ensuring that known and unknown uncertainties are accounted for. Therefore, PFM (or probabilistic risk assessment) analyses should allow for the direct consideration of the uncertainties that can be quantified. The other aspects, such as defense in depth or condition monitoring, all ensure that phenomena and occurrences that are not accounted for are addressed. Regulatory Guide 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, describes the requirements for risk-informed design-basis changes based on assessing their impacts on core damage frequency. However, assessing the impacts of relief requests based on changes in core damage frequency alone may not be appropriate for events of very low probability.

ii

Contents Executive Summary....................................................................................................................................... 2 Contents ....................................................................................................................................................... iii Definitions ..................................................................................................................................................... v 1 Introduction .......................................................................................................................................... 1 1.1 Motivation for Increasing Confidence in Probabilistic Fracture Analyses................................... 1 1.2 Objective...................................................................................................................................... 2 1.3 Report Structure .......................................................................................................................... 2 2 Definition of a PFM Analysis ................................................................................................................. 4 2.1 The Framework for Structural Integrity Analysis Provided by Fracture Mechanics .................... 4 2.2 Similarities and Differences between Probabilistic and Deterministic Fracture Mechanics Analysis Methodologies........................................................................................................... 6 2.3 Considerations in the Use of PFM ............................................................................................... 9 3 Models for PFM Analyses.................................................................................................................... 12 3.1 Model Selection ......................................................................................................................... 12 3.1.1 General Concepts .................................................................................................................. 12 3.1.2 Theoretical Models ................................................................................................................ 13 3.1.3 Empirical Models ................................................................................................................... 13 3.1.4 Computational Models .......................................................................................................... 14 3.1.5 Model Validity ....................................................................................................................... 14 3.2 Gaining Confidence in Models ................................................................................................... 15 3.2.1 Model Definition, Verification, and Validation ...................................................................... 15 3.2.2 Documenting Model Uncertainty .......................................................................................... 16 3.2.3 Assessment of Alternative Models ........................................................................................ 17 4 Inputs for PFM Analyses ..................................................................................................................... 18 4.1 Input Types ................................................................................................................................ 18 4.2 Construction of Probability Distributions .................................................................................. 18 4.3 Input Bounds ............................................................................................................................. 20 4.4 Quantification of Input Assumptions ........................................................................................ 20 4.5 Conservatisms............................................................................................................................ 21 4.6 Dependencies ............................................................................................................................ 21 5 Uncertainty Characterization and Framework for PFM Analyses....................................................... 23 5.1 Epistemic and Aleatory Uncertainty .......................................................................................... 23 5.2 PFM Analysis Uncertainty Identification and Context ............................................................... 26 iii

5.3 Probabilistic Framework, Uncertainty Propagation, and Sampling Techniques ....................... 26 5.4 Verification and Validation ........................................................................................................ 28 6 Analysis of PFM Outputs and Results ................................................................................................. 31 6.1 Convergence of Results ............................................................................................................. 31 6.2 Output Uncertainty Quantification (Uncertainty Analysis) ....................................................... 32 6.3 Sensitivity Analysis (Input Importance Quantification) ............................................................. 33 6.4 Sensitivity Studies ...................................................................................................................... 33 6.5 Analysis of Problem Drivers and Confidence Demonstration ................................................... 34 7 Conclusions ......................................................................................................................................... 35 7.1 Summary.................................................................................................................................... 35 7.2 Future Work............................................................................................................................... 36 8 References .......................................................................................................................................... 37 iv

Definitions This section lists and defines important concepts and terms mentioned throughout this technical letter report to promote a clear common understanding of the ideas and discussions to follow.

Accuracy and Precision:

Accuracy is the degree to which the result of a measurement, calculation, or specification conforms to the correct value (i.e., reality or a standard accepted value). Precision is a description of random errors and a measure of statistical variability for a given quantity. In other words, accuracy is the proximity of measurement results to the true value; precision is the repeatability or reproducibility of the measurement.

Figure 1: Illustration of the concepts of accuracy and precision Aleatory Uncertainty:

Uncertainty presumed to stem from the intrinsic randomness of a phenomenon. Aleatory uncertainty is also known as statistical uncertainty.

Epistemic Uncertainty:

Uncertainty presumed to be caused by lack of knowledge (or data). Epistemic uncertainty is also known as systematic uncertainty.

Probabilistic Fracture Mechanics (PFM):

Any application of fracture mechanics that considers some of the inputs to be random. A prime example of a random input is initial crack size, which is seldom accurately known and usually has a strong influence on lifetime. In nuclear power plants, PFM may be used to calculate probabilities or frequencies of failure for components of interest. Among many possible uses, these probabilities or frequencies of failure may be used as inputs in probabilistic risk assessment calculations.

Probabilistic Risk Assessment (PRA):

Sometimes called probabilistic safety assessment, an engineering analysis process that uses current knowledge to answer three fundamental questions [1]:

(1) What can go wrong?

(2) How likely is it?

(3) What are the consequences?

v

A fourth question often related to PRA is how confident are we in the answers to the first three questions?

Within the regulatory context, PRA can be used as an engineering analysis process that models a nuclear power plant (NPP) as a system. By answering the three fundamental questions, it provides both qualitative and quantitative information. It builds on and represents the current state of knowledge regarding (1) what design, operation, and environmental (physical and regulatory) aspects of the plant are important to risk, what elements are not important, and why, and (2) what is the magnitude of the risk associated with the overall plant.

A PRA model for an NPP will, for some plant operational modes (e.g., at-power operation, low-power conditions, shutdown conditions), typically delineate and analyze many accident scenarios, starting with an initiating event (i.e., a disturbance to plant normal operations such as loss of coolant), an internal hazard (e.g., a fire or flood starting within the plant boundaries), or an external hazard (e.g., an earthquake or external flood), and ending with some predicted level of consequence.

Regarding consequences, the definition of PRA is sufficiently broad to encompass a wide variety of analysis endpoints. Thus, depending on the needs of the problem at hand, a PRA can assess consequences in terms of, for example, core condition, radioactive material release from the plant, or offsite effects.

Realization:

In probability and statistics, a realization, observation, or observed value of a random variable is the value that is actually observed (what actually happened) in a given single simulation. Within the context of probabilistic analyses, a realization often refers to a single run (or simulation) of the model.

Sensitivity Analysis:

The process of characterizing the impact of variability in the model inputs on variability in the model output. That is, it is the study of uncertainty in an output of interest (e.g., probability of rupture) caused by the uncertainty in the inputs (and model) of the system under consideration. Therefore, sensitivity analysis should be used throughout a PFM analysis. During stability analysis, sensitivity analysis is used to help determine which sampling options must be adjusted to achieve a converged solution. Once a converged solution is found, sensitivity analysis techniques can be used to determine the input variables that have the most impact on the uncertainty in the results. Section 6.3 further discusses sensitivity analysis.

Sensitivity Study:

Studies used to determine the effect of different models and assumptions on the quantities of interest.

In this sense, sensitivity studies are what if analyses performed for individual sub- or full-model assumptions to determine how results might change under different, yet relevant, assumptions. Ideally, results would be insensitive to a collection of assumptions in the sense that regulatory decisions would not change across this collection. However, a well-planned sensitivity study can elucidate the assumptions that could change regulatory decisions. Subject matter experts can then further evaluate the plausibility of these assumptions. Section 6.4 further discusses sensitivity study.

Stability Analysis:

vi

Analysis that compares results from several probabilistic simulations. The goal is to find the simulation options that lead to converged, stable solutions for the specific application. Section 6.1 further discusses stability analysis.

Uncertainty Analysis:

The process of characterizing the uncertainty of the analysis output. This may include calculation of means and percentiles for quantities of interest, as well as determining the confidence intervals on quantities of interest for outputs of interest.

Uncertainty Propagation:

The uncertainty in the input is propagated through the model using sampling-based methods. The available means to propagate uncertainty ultimately depend on the computational constraints as well as the nature of the inputs and outputs under consideration.

Validation:

The process of determining the degree to which a model is an accurate representation of the physical phenomenon considered from the perspective of the intended uses of the model.

Verification:

The process of determining that a model implementation (1) accurately represents the conceptual description of the model and (2) correctly implements the chosen coding solution for the model. In particular for software, this process implies ensuring that the code being verified meets the software requirements.

vii

1 Introduction 1.1 Motivation for Increasing Confidence in Probabilistic Fracture Analyses In the past, the U.S. Nuclear Regulatory Commission (NRC) has typically regulated the use of nuclear structural materials on a deterministic basis. Safety factors, margins, and conservatisms were used to account for model and input uncertainties. However, in the mid-1990s, the NRC issued a policy statement [2] that encouraged the use of probabilistic risk assessments (PRAs) to improve safety decisionmaking and regulatory efficiency. Since then, the NRC has made progress in incorporating risk-informed and performance-based approaches in its regulation and continues to revisit and update the approaches regularly.

A major component of the overall safety of nuclear structures is the fracture behavior of the materials.

Consensus codes and standards often used for the design and analysis of nuclear structures, such as the American Society of Mechanical Engineers (ASME) Boiler and Pressure Vessel Code [3], typically rely on conservative fracture models with applied safety factors and conservative bounding inputs to account for the many uncertainties that may be present. Improving the efficiency of such models by truly understanding the impacts of the assumptions and uncertainties becomes difficult because of the conservative nature of the models and inputs and the inadequate documentation of the basis for safety factors. Probabilistic fracture mechanics (PFM) is an analysis technique that can be used to complement deterministic fracture mechanics analyses to gain greater insight into the structural integrity of components. Two notable NRC efforts in the area of PFM include the FAVOR (Fracture Analysis of VesselsOak Ridge) [4, 5] and xLPR (eXtremely Low Probability of Rupture) projects [6].

PFM allows the direct representation of uncertainties through the use of best-estimate models and distributed inputs (i.e., inputs for which the likelihood of being equal to a given value is represented by a probabilistic distribution). The term best estimate refers here to models that attempt to fit data or phenomena as well as possiblethat is, models that do not intentionally bound or bias data for a given phenomenon. The PFM analysis methodology permits determination of the direct impact of uncertainties on the results, which gives the user the ability to determine and possibly refine the specific drivers of the problem. However, PFM analyses can be more complicated and difficult to conduct than deterministic analyses. Determining validated best-estimate models, developing input distributions with limited experimental data, characterizing and propagating input and model uncertainties, and understanding the impacts of problem assumptions on the adequacy of the results can complicate the development and approval of PFM analyses in a regulatory application. Moreover, the underlying deterministic models that drive PFM are sometimes simplistic for a variety of reasons, such as to permit reasonable solution times, or because of the state of knowledge. The uncertainty introduced in a PFM by overly simplified deterministic models can be difficult to quantify. However, rigorously assigning uncertainty can be challenging whether the model is simple or complex.

The Electric Power Research Institute has submitted several Materials Reliability Program (MRP) reports containing PFM analyses to the NRC, either for information or for review and approval. Such efforts include:

• Materials Reliability Program: Probabilistic Fracture Mechanics Analysis of PWR Reactor Pressure Vessel Top Head Nozzle Cracking (MRP-105) [7]

1

• Materials Reliability Program: Alloy 82/182 Pipe Butt Weld Safety Assessment for U.S. PWR Plant Designs (MRP-113) [8]

• Materials Reliability Program: Probabilistic Risk Assessment of Alloy 82/182 Piping Butt Welds (MRP-116) [9]

• Materials Reliability Program: Inspection and Evaluation Guidelines for Reactor Vessel Bottom-Mounted Nozzles in U.S. PWR Plants (MRP-206) [10]

• Materials Reliability Program: Topical Report for Primary Water Stress Corrosion Cracking Mitigation by Surface Stress Improvement (MRP-335 Revision 3) [11]

• Materials Reliability Program: Reevaluation of Technical Basis for Inspection of Alloy 600 PWR Reactor Vessel Top Head Nozzles (MRP-395) [12]

In general, NRC staff reviews of PFM submittals have been difficult and lengthy, although there have been successful applications of PFM (e.g., risk-informed inservice inspection and the elimination of vessel weld inspections [13, 14]). The reasons for these difficulties are complex and include, but are not limited to, lack of proper input choice justifications, choice of inputs that does not reflect observations from operational experience, insufficient description of the PFM tools used in the analyses, insufficient verification and validation (V&V) of the tools used for PFM analyses, and insufficient analysis of results sensitivity to choice of inputs. The NRC staffs overall impression is that defining quality assurance (QA) requirements and acceptable methodologies to perform a PFM analysis would be very beneficial for the development and use of PFM analyses in regulatory applications.

This technical letter report presents some ideas on how to improve confidence in structural analyses performed using PFM, by focusing on topics such as problem definition, PFM model development, input definition, uncertainty analyses, probabilistic framework development, and output analysis including uncertainty analysis, sensitivity analyses (to determine the impact of uncertainties on the results), and sensitivity studies (to determine the impact of mean values on the results). By determining the main drivers of the probabilistic results and investigating the impacts of the assumptions made to develop those drivers, the confidence in the overall results can be greatly improved. The report also discusses some preliminary thoughts on regulatory review of PFM analyses.

1.2 Objective The aim of this work is to develop a methodology that is general enough to guide the development or critique of any PFM analysis and that also effectively and logically manages the level of detail and breadth of scope that PFM analyses can assume. The project is expected to produce not only reports documenting such a methodology but also regulatory guidance describing its implementation.

1.3 Report Structure Chapter 2 proposes a definition of a fracture mechanics analysis of component structural integrity. The chapter also discusses the commonalities and differences between deterministic and probabilistic approaches to these analyses and reflects on differences in situations that may favor each methodology.

Chapters 3, 4, and 5 discuss some of the major building blocks of a PFM analysis:

2

• Chapter 3 discusses models for PFM analyses (e.g., a model of fracture toughness and how it varies with temperature).

• Chapter 4 discusses representation of inputs within a PFM analysis (e.g., a value of reactor coolant temperature might be an input).

• Chapter 5 discusses how uncertainties in both models and inputs are represented and addresses the so-called framework that binds the models, the inputs, and their uncertainty characterization into the overall mathematical representation of the system. Key here is the critical need to evaluate uncertainty propagation within not only the individual pieces of the model but also within the framework itself.

• Chapter 6 discusses the PFM outputs and includes such topics as convergence and sensitivity studies and code V&V.

3

2 Definition of a PFM Analysis 2.1 The Framework for Structural Integrity Analysis Provided by Fracture Mechanics Any analyses involving fracture mechanics, whether they are deterministic (DFM) or probabilistic (PFM),

involve the same key elements, as explained by way of example using the following equations inspired by the classical work of George Irwin and colleagues [15]:

Is or  ? (1)

= x x (2)

Eq. (1) represents an example of a simple fracture mechanics-based failure criterion. Within Irwins so-called linear-elastic fracture mechanics 1 (LEFM), KAPPLIED represents the applied driving force to fracture produced by the structural loading, while KRESISTANCE represents the resistance to this driving force provided by the materials of construction. These two quantities are compared to assess the likelihood of structural failure: if KAPPLIED remains below KRESISTANCE, then failure is unlikely. Conversely, conditions for which KAPPLIED exceeds KRESISTANCE have a greater probability of failure.

Either of these K values can be calculated using Eq. (2), where represents the applied stress, a represents the crack size, and F is a nondimensional factor that accounts for the geometry of a structure or of a fracture toughness test specimen To determine KRESISTANCE, any one of several standard test methods developed by the American Society of Testing and Materials (ASTM) can be used [16, 17, 18]; these will give the form of Eq. (2) that is appropriate for the specimen geometry being tested and the conditions of the test.

In determining KAPPLIED for a structure (e.g., a pipe, a pressure vessel, or a baffle bolt), the following apply:

• The stress () is typically determined by either finite-element analysis of the actual structure or by a closed-form analysis of a simplified representation of the structure.

• The crack size (a) may be determined by nondestructive evaluation of an indication found to exist in the structure, might represent the size of a flaw that nondestructive evaluation could miss, or might represent a nominal flaw size agreed to as appropriate for certain types of assessments.

• The nondimensional geometry factor (F) is typically determined by either finite-element analysis of the actual structure or by a closed-form analysis of a simplified representation of the structure.

1 LEFM applies to situations where the effects of plasticity are highly localized to the crack tip and do not interact with any structural boundaries. Elastic plastic fracture mechanics (EPFM) extend LEFM concepts to higher loading levels.

However, the key elements of both LEFM and EPFM are identical. High-temperature fracture mechanics, which may involve the active degradation mechanism of time-dependent creep crack growth, is not important for current U.S.

nuclear power plants but will be important for future advanced plants being considered.

4

While the techniques and technologies needed to determine these variables (KRESISTANCE, KAPPLIED, , a, F) can be complex, their ultimate use in a fracture mechanics assessment of structural integrity involves only simple algebra.

Fracture toughness, applied stress, flaw size, and component geometry are common to any fracture mechanics analysis. Only two additional variables, environment and time, need be added to allow complete understanding, characterization, and evaluation of virtually any fracture mechanics analysis.

All of these variables may or may not evolve with time and spatial location within a component or structure. Furthermore, as a whole, the structure is evolving toward some end state that is characterized using a quantitative metric (e.g., failure occurrence).

Regardless of the details of a particular problem, these six categories of variables (i.e., material properties, stress, flaw size, environment, spatial extent, and time) constitute the most important aspects of any fracture mechanics analysis, whether it be DFM or PFM. Human interaction, such as inspection, is considered as a measure to confirm the validity of the fracture mechanics model, and repair or mitigation is considered as a measure to change the key, if not all, variables.

The following are two more aspects of real-world problems that fracture mechanics analyses should reflect:

• Dependencies between variable categories: These six variable categories may be interdependent; their accurate representation in the DFM or PFM model is key to the development of a model that reasonably approximates reality. As one example, the environment may change with time, which may produce changes in other variables such as the material properties (e.g., material degradation or time-dependent aging) and the flaw size (environmentally assisted crack growth).

• Multiple time and spatial scales: Multiple time scales such as the following may need to be modeled:

o In a reactor pressure vessel, loading that may give rise to fatigue damage occurs over the months and years the plant operates. Superimposed over this are possible accident loadings from unanticipated or faulted conditions; this would occur over the time scale of minutes to hours. Superimposed over this is the possibility, albeit remote, that cleavage fracture could occur because of one of these accidents (e.g., a pressurized thermal shock transient). Were this to happen, the fracture would make its way through the vessel wall in fractional seconds.

o In piping systems, inspections can result in subsequent weld repair (including weld overlay or inlay) or mechanical stress improvement of found cracks. Both actions (repair or mechanical stress improvement) could be viewed as altering the material properties, stress pattern, or flaw geometry and resetting the position on the time axis for the repaired component.

5

2.2 Similarities and Differences between Probabilistic and Deterministic Fracture Mechanics Analysis Methodologies Historically, most assessments of structural integrity using fracture mechanics are performed deterministically. That is, a single-value estimate of the magnitude of the driving force for structural failure (KAPPLIED) produced by loading events is compared to a single-value estimate of the resistance to that driving force (KRESISTANCE) of the materials from which the structure is fabricated. The use of single values is an approximation. As an example, the KAPPLIED produced by loading cannot always be the same because applied pressures and temperatures, even when controlled, vary somewhat. Likewise, the size of cracks (that is, the variable a in Eq. (2)) is uncertain. Similarly, KRESISTANCE is a distributed quantity because real engineering materials are not homogeneous and isotropic, which is but one reason why fracture toughness data show a range of values even under tightly controlled conditions. Thus, KAPPLIED and KRESISTANCE are both inherently distributed quantities (that is, they are best represented by a range of values and not by a single value).

Figure 2-1 depicts the actual situation just described. The bell curves located on the left of each subfigure represent the distribution of the driving force in the structure (KAPPLIED), while the curves on the right of each subfigure are the toughness distribution (KRESISTANCE). If these distributions are widely separated (top row, left-most column), then there is either no probability of structural failure, or it is exceedingly small. Other degrees of distribution overlap are of course possible; these correspond to different probabilities of structural failure. The DFM and PFM modeling approaches approximate the actual situation depicted by the left-most column differently, as follows:

• DFM analysis (the middle column): Deterministic models represent the uncertainties in KAPPLIED and KRESISTANCE by a number of techniques (e.g., conservative bounding values, conservative submodels, margins). All of these techniques permit bounding of the distributed KAPPLIED and KRESISTANCE quantities using single values, usually representative of some high or low percentile bound. To ensure a safe, or conservative, outcome (i.e., failures are overestimated), features that contribute to the driving force are systematically overestimated (e.g., higher loads, bigger flaw sizes), while features that contribute to material resistance (i.e., fracture toughness) are systematically underestimated. The use of such conservative modeling constitutes implicit conservatism. Furthermore, as shown in the first row, a very unlikely occurrence of events would be considered as a zero probability of occurrence without an actual estimate of what the real probability of occurrence is. Without characterization of the conservatism involved, a zero deterministic probability of occurrence is not informative. A similar comment applies to the second row, where the separation between failure and no-failure, which has serious consequence, is decided by the level of conservatism. This level needs to be documented.

Imposing a safety factor (e.g., 3 as specified in ASME Code,Section XI) on top of the conservatisms in overestimating driving force and underestimating material resistance can mitigate these DFM issues. The safety factor represents explicit conservatism, as opposed to the implicit conservatism applied in the choice of models described earlier in this paragraph.

• PFM analysis (the right-most column): Probabilistic models represent the uncertainties in KAPPLIED and KRESISTANCE in one of two ways:

(1) As distributed quantities that seek to represent the distributions in the component parts as accurately as possible. The estimates of the distributed quantities are shown as darker bell curves superimposed over the actual distributions shown by the lighter bell 6

curves. In principle, the estimated variability is different than the actual variability, if only by a little, because all factors affecting variability can never be fully identified, and the exact variability of each factor is not always precisely known. This approach is typically used for the model components thought to be most important or significant to predicting the outcome.

(2) In some situations, it is necessary to represent a probability distribution with a discrete input or model, which produces a discrete output. This approach is typically used for model components that are thought not to contribute greatly to the predicted outcome or about which little is known. In these instances, best-estimate values or models should be employed. As in purely probabilistic assessments, conservatisms should be applied in the final step of the analysis (e.g., in the acceptance criteria) and not in the intermediate steps. This embrace of certain DFM aspects within a PFM should be the exception rather than the rule and should be done only after consideration, documentation, and elimination of approaches that represent the uncertainties as probability distributions.

(3) As illustrated previously, the overlap in the driving force and resistance distributions provides a means to estimate the probability of structural failure.

DFM and PFM approaches are thus fundamentally similar: both are mathematical abstractions used to approximate reality. Moreover, both share the common goal of representing the uncertainties in a mathematical form that enables problem solution, although they achieve this by different means.

7

Figure 2-1. Illustration of how deterministic and probabilistic modeling approaches represent the actual distributions of driving force (KAPPLIED) and structural resistance (KRESISTANCE) and how these representations influence the estimate of structural performance that is the outcome of the analysis Some view the fact that a failure probability is one possible numeric outcome of a PFM analysis as an impediment to the acceptance of PFM techniques. The following are possible reasons for this view and arguments against it:

• Statements regarding a probability of failure, however remote, explicitly acknowledge this possible outcome. In contrast, DFM obscures this possibility in its lexicon; DFM results are often expressed in terms of factors of safety or margins against failure. Nevertheless, it must be acknowledged that the language that engineers use to describe DFM analysis results do not alter outcomes: these structures can, and sometimes do, fail. In this respect, PFM language is seen as more accurate and transparent than that used to express DFM results.

• Because engineers have been trained to think in terms of factors of safety or margins against failure, some of them may find the idea of a failure probability too abstract to grasp. However, the terms factor of safety, margin against failure, and probability of failure are all numbers reflecting levels of conservatism. Recognizing this similarity promotes an easier acceptance of PFM as an alternative to DFM.

• Procedures for reviewing PFM assessments and probabilistic acceptable criteria are, in general, less well established and less formalized than for DFM assessments. Indeed, the point of this project is to directly address this issue.

8

To summarize, the key points of this section are as follows:

• Both DFM and PFM analyses are models used to represent reality. Although use of DFM has met its objectives, as reflected in various engineering support for different purposes throughout the nuclear industry, there is more room for using PFM as an alternative to better quantify implicit and explicit conservatism in nuclear components.

• Both DFM and PFM analyses treat uncertainties mathematically but in different ways.

Furthermore, it may not be possible to properly account for every uncertainty.

• PFM analyses will represent most, but not necessarily all, uncertainties identified so far as important distributed quantities.

• Both PFM and DFM use specific metrics to quantify results.

• Use of DFM or PFM methods does not represent an either-or choice. In many circumstances, DFM and PFM methods are complementary parts of a safety case.

2.3 Considerations in the Use of PFM Most engineering structures operating today were originally designed on a deterministic basis. This is also true of many NRC regulations, which were developed using prescriptive, deterministic requirements based on a combination of experience, test results, and expert judgment. In developing those requirements, the NRC staff considered engineering margins (both implicit as associated with the use of conservative models, and explicit as associated with safety factors), as well as the principle of defense in depth (also called redundancy). It was assumed that undesirable events, however improbable, might occur, and so the plant designs included safety systems capable of preventing or mitigating the consequences of those events.

Over time, the reliance on a deterministic basis for engineering designs and regulations has given way to an increased use of probabilistic techniques for such purposes. Many factors support and motivate this evolution, including the following:

• Public policy decisions: In the early 1990s, Congress passed the Government Performance and Results Act [19]. One objective of that law is to improve Federal program effectiveness and public accountability by promoting a new focus on results, service quality, and customer satisfaction. In response to this law, the NRC committed to moving toward risk-informed, performance-based regulation and issued a policy statement that encouraged the use of PRAs to improve both safety decisionmaking and regulatory efficiency. This policy statement formalized the Commissions commitment to the expanded use of PRA, stating in part that the use of PRA technology should be increased in all regulatory matters to the extent supported by the state-of-the-art in PRA methods and data and in a manner that complements the NRCs deterministic approach and supports the NRCs traditional defense-in-depth philosophy. Therefore, use of PFM for passive pressure boundary components, which is the topic of this report and this research effort, aligns with the Commissions statement.

• Factors unanticipated in the design phase or not addressed by codes and standards: There is a fundamental difference between how deficiencies, or potential deficiencies, are addressed when they are discovered during the design and construction of a structure versus when they are revealed later, often after many years or decades of safe service. During design and 9

construction, deficiencies that do not meet specifications are often addressed by repair, replacement, or reconstruction, because the effort to demonstrate the acceptability of the deficiency often exceeds the effort associated with correcting the deficiency. However, once operation begins, repairs that were considered feasible during construction can become cost prohibitive (cost in terms of dollars, time, or dose). While the NRCs primary mission is safety, it is nevertheless obligated (see Title 10 of the Code of Federal Regulations (10 CFR) 50.109(c)(5) and (7) [20]) to assess whether safety benefits justify the attendant cost.

PFM assessments are ideally suited to such situations, because PFM metrics relate directly and clearly to systems that can challenge safety (i.e., probability of structural failure). Indeed, the Backfit Rule (see 10 CFR 50.109(c)(3) [20]) explicitly requires an assessment of risk. Conversely, DFM analyses are not well suited for risk quantification, and therefore the safety impact is quantified differently. Typically in DFM, safety is assessed based on the concept of safety margin: a larger margin implies added safety. In contrast, PFM deems an element safe if acceptance criteria are met: being farther from the acceptance criteria (as long as they are met with a high degree of confidence) implies greater safety.

PFM also provides more flexible methods to account for factors that occur during service (e.g., new damage mechanisms, unanticipated loadings, and aging) that were not considered during design. Especially when such factors are encountered for the first time, the performance of deterministic analyses following the guidelines of codes, standards, and regulations can be difficult because these established procedures may not account for the new factors.

Historically, unanticipated material degradation mechanisms have arisen regularly in NPPs [21].

The latest example is the primary water stress-corrosion cracking aging issue in Alloy 600 and 182/82 welds in pressurized-water reactors, which led in part to the development of the xLPR code. Attempts to force-fit the new phenomena into an existing deterministic codified framework are complicated because the original deterministic framework typically includes many judgments that result in poorly characterized conservatism. Consequently, the resulting construct is not particularly amenable to treating a new phenomenon and rarely is the result transparent or quantifiable. In contrast, the adaptability inherent in PFM fosters the clear treatment of such situations. However, unanticipated degradation mechanisms may require the implementation of new deterministic models in the PFM code to capture the new degradation mechanisms. Because of this, both DFM and PFM codes must continually be updated.

• Use of structures beyond their design, or licensed, lifetimes: One currently important case of a factor unanticipated during the initial construction of NPPs is operation for more than the original licensed life. As plants consider additional 20-year license extensions, imposing a safety factor of 3 in the DFM method becomes impractical. Thus, PFM methods could be used to better quantify the risks associated with operation of NPPs beyond their initial licensed life.

Beyond these specific benefits, PFM also offers some intangible benefits that DFM cannot. In particular, PFM may force analysts to look more deeply at the data used to develop models or inputs and to be more critical of these data. This is because it is often easier to conservatively bound data than to attempt a best-fit model with quantified error bounds. As a result of this process, analysts are likely to better understand the data they use in their inputs and models.

In two recent cases where the NRC has pursued a PFM assessment (i.e., for reactor pressure vessels in the Alternate Pressurized Thermal Shock (PTS) Rule and for piping systems in the xLPR project), the rigor 10

of the process needed to build the models revealed deficiencies in them that may have been otherwise overlooked. For example, in developing the FAVOR PFM model to support the Alternate PTS Rule, analysts discovered that the initial sampling scheme used for the embrittlement trend curve double-counted uncertainties and that crack-face pressure for internal surface breaking cracks had not been modeled. Similar improvements are expected as the NRC staff adopts a uniform process for reviewing and assessing PFM analyses submitted by licensees.

11

3 Models for PFM Analyses While the probabilistic framework manages input sampling and compilation of results, the individual models use the sampled inputs to approximate a physical system and generate the outputs. Most individual models, as part of the overall PFM analysis, are deterministic by nature. Multiple models may be interdependent within the same analysis, with the probabilistic framework passing variables from model to model before the final result is generated. This section discusses various aspects of selecting and substantiating models for a PFM analysis.

3.1 Model Selection 3.1.1 General Concepts The goal of any engineering assessment methodology is to determine the response of a system to a variety of inputs. The assessment of the system in question should be designed and developed using a set of models that best represents the physical behavior of the system of interest. 2 At times, more than one model may be available for predicting the response of the system to a particular phenomenon.

Expert engineering judgment may be used in selecting which model to employ, but including several available models allows for sensitivity studies and determining the impacts of model predictions on the response of interest.

Model selection may involve balancing the accuracy and practicality of various mathematical approaches. In weighing these concerns, the analyst must consider the desired level of convergence of the PFM analysis (i.e., the number of realizations required to achieve confidence in the result). The other factor here is computational resources. While a particular approach may be considered best estimate, it may not be practical for a PFM analysis given the time and resource constraints imposed on the analyst. The occasional need to choose a model that has less fidelity but is easy to solve for because of the solution speed requirements of PFM, may affect results. In such cases, biases or uncertainties relative to the best-estimate model should be quantified and accounted for by propagating the associated uncertainty through the probabilistic model. The model choice can further be complicated because PFM generally favors the use of the most accurate deterministic models rather than conservative models, but even the most accurate models are not always a very good representation of the data, resulting in known model biases and uncertainties. A simple example of this is the surface crack stability models in the xLPR code. Currently in xLPR, surface crack stability is based on limit load solutions that are believed to be conservative. This is because the more accurate surface crack J-estimation schemes are not fully vetted, and currently the only available J-Resistance curves are for compact tension specimens, which produce conservative lower bound resistance curves. If a surface crack J-estimation scheme were used in conjunction with compact tension J-Resistance curves, the predictions would be overly conservative. In this case, methods to limit load surface crack stability will be nonconservative for lower toughness materials.

In all cases, the rationale behind model selection should be thoroughly documented. The documentation should include the following:

2 The chosen model must be simple, fast, and practical for the many realizations required by PFM.

12

• a discussion of alternative approaches that were rejected, with justification for the rejection

• a discussion of how the selected model compares to other potential models that better represent reality, or to reality itself

• a qualitative or quantitative discussion of the modeling uncertainties for the selected models 3.1.2 Theoretical Models The mathematical models used in PFM codes to represent physical processes or phenomena are based on evidence or understanding from a variety of information sources. Models chosen should represent the actual physical system as realistically as possible. Some models have a greater theoretical underpinning, but even theoretical models usually have simplifying assumptions (e.g., isotropy, homogeneity). For example, models used to describe the stress in a pipe wall or the stress intensity factor for a crack rely on a solution of the equilibrium and compatibility equations of basic solid mechanics [22]. Thus, these models generally differ from the physical reality only inasmuch as the boundary conditions and initial conditions differ from that reality. Even so, approximations are sometimes introduced in these models. In the case of the stress intensity factor calculation, a complicated loading condition on the crack faces may be approximated by a polynomial fit [23], but the use of a general weight function method for calculating K for complex stresses may yield more accurate results [24]. The analyst should identify and account for the impacts of this type of approximation.

Theoretical models may present themselves in terms of closed-form equations, which are ideal computationally. Numerical discretization approaches (e.g., finite-difference methods and nonlinear equation solving) in an individual model may have several drawbacks in a PFM framework, including high computational cost and possible spatial and temporal nonconvergence. Interpolation of tabulated coefficients may be necessary in many physical models. Interpolation itself may be considered computationally burdensome. Fitting closed-form equations to tabulated data may lead to greater efficiency, but the analysis should account for uncertainties inherent in uncertainty modeling (i.e., confidence bounds) or those resulting from fitting errors.

3.1.3 Empirical Models Models developed using test data are sometimes called empirical models. Data underlying these models often exhibit considerable scatter (i.e., uncertainty, usually indicative of unidentified and typically uncontrolled, or unaccounted for, important variables). Nevertheless, these data reflect the underlying physical processes under the specific conditions in which the data were gathered and within the range in which the data were gathered. As a result, if the data source is actual operational experience from components in service, the data may better represent the real situation being modeled than data from a controlled laboratory environment that simulates inservice conditions, though this may not always be the case. When incorporating empirical models into a PFM analysis, the analyst should consider several aspects.

First, the analyst should have knowledge of and confidence in the dataset that gave rise to the empirical fit. This dataset should be well documented in terms of experimental procedures and data processing.

Relevant information on the dataset may include the following:

13

• experimental design, statistical design of experiments (were the experiments completely random or was a certain range targeted for optimization? what was the degree of statistical replication?), or data-generating mechanism

• consensus standards referenced while conducting experiments and processing data

• location or locations within a test specimen where the data were gathered

• data screening procedures that may have led to exclusion of some data points from the dataset

• ranges of parameters that were varied in the experimental work

• measurement uncertainty information The analyst should also understand the statistical model used to fit the data, including any potential theoretical underpinnings of the statistical model. The model may have applicability limitations that originate from the mathematical form of the equation or the range of conditions covered by the underlying dataset, or both. These limitations should be understood and incorporated in the computer coding of the model.

Finally, the analyst should assess uncertainties related to goodness-of-fit and scatter about the best-estimate curve. For example, the assessment should identify which estimator constitutes the best estimate and whether the model incorporated into the PFM framework uses the best estimate or a conservative bound. Using conservative bounds without scrutiny may lead to biased PFM analysis results. In both cases (conservative bound or best estimate), the analyst should assess the sensitivities of the approach on the final results of the PFM calculation.

3.1.4 Computational Models As computational resources and solution speeds improve in the future, finite-element or finite-difference models could also directly be part of some PFM models. However, presently limited by computational resources, these self-contained PFM models are for the most part not practical and are not further discussed here.

3.1.5 Model Validity Model applicability limits should be considered for both theoretical and empirical models. Applicability limits for empirical models will most likely come from the underlying dataset (e.g., the model is valid only for the range of temperatures considered in the testing program). The applicability limits of a theoretical model may not be as straightforward to determine. To arrive at appropriate limits, the analyst may have to consider the physical system, the mathematical derivation of equations, relevant test data, and expert judgment. Extrapolation (the use of models beyond their strict applicability limits) should be avoided when possible, regardless of the type of model (theoretical or empirical). For example, an alternative option to extrapolation could be to define the input probability distributions so that such values are not sampled, by truncating the distributions to avoid regions where no information exists for the model validation. However, sometimes the joint collection of sampled inputs may produce a result outside the range of applicability, and these distributions are not easy to specify a priori because of the complex input-output relationship.

Since inputs are sampled randomly from distributions, it is important to define the bounds of applicability and validity of any model (including the input distribution) used within PFM analyses. It is 14

equally important to ensure that the data used to construct distributions for model parameters are relevant for the problem being modeled, which may mean using only the subset of a large database that is most relevant to the problem at hand. Ways to find the bounds of model applicability include the following:

• determination from the limits of the data from which the model was developed

• validation testing or expert judgment

• examination of the physical limits of the system In any case, the analyst should explicitly document the bounds of model applicability, along with how the code handles those situations in which the sampled inputs may be outside these bounds. An extreme example of these bounds might be to ensure that yield stress sampled is always positive.

When an applicability limit is violated, the coded model will have to take appropriate action. Exiting the PFM analysis completely upon violation may not be an ideal approach, especially when long run times are required. Warnings explaining to the user which input was in violation and the action the code took may be an acceptable alternative. However, the user should be able to make an informed judgment about the uncertainties introduced by violated limits.

3.2 Gaining Confidence in Models Establishing confidence in the individual models is an important step in substantiating the PFM analysis as a whole. Confidence in models relies heavily on careful documentation of the models limitations and of the V&V activities supporting the technical basis of the software. After identifying these limitations, the analyst should understand their impact on results. This section addresses these aspects of PFM analysis.

An important part of establishing confidence may involve running the code deterministically and initially focusing primarily on the physics. As for any complex system with many inputs, confidence in the code and its models is obtained not only via mathematical verification, but also by gaining experience in exercising the code and analyzing the trends of the code outputs.

3.2.1 Model Definition, Verification, and Validation As introduced in Section 3.1, models will likely have an input domain with limited validity specified by the PFM code developer. The analyst should document the rationale behind applications outside this validity domain. The actions taken by the coded model when the domain is violated should also be documented. This documentation should include guidance to the user on how to interpret results when the domain is violated and the action that the code took to remedy the situation. In addition, the analyst should mathematically assess the likelihood of being outside of the model bounds, including the physical reality of such states and the downstream consequence of such situations. For example, NRC analysts have encountered a case where K solutions were limited to a radius-to-thickness ratio smaller than 2, and thus could not be used for a thick-walled pressurizer surge nozzle that needed to be analyzed. In this case, the NRC analysts were forced to artificially increase the pipe diameter, while keeping the thickness constant, to obtain a result from the code.

Within the bounds of the validity domain, the model will have uncertainties. Potential sources of uncertainties include data-fitting errors, scatter in the dataset used to define an empirical model, and 15

simplifying assumptions in deriving equations. These sources of uncertainties should be documented for each model in a PFM analysis. The documentation should identify the relevant sources of uncertainty for each model and assess their impacts on final and intermediate results. An intermediate result may be fed to another model in the PFM analysis. In this case, the logic tying each model to the final result should be documented.

Two important analyses used to gain confidence in models are often referred to as model verification and model validation. Only brief descriptions are given here, but good starting points for the vast amount of literature on the subject include [25] and [26]. Model verification is a QA process by which the coded model is verified to meet the defined software requirements (i.e., the mathematics of the model are correctly coded). Someone independent of the model development should review all software requirements, define and execute relevant software testing, and report on any anomalies.

Potential testing activities include comparing coded equations to documented equations, comparing code results to independent spreadsheet calculations, and any other activities to verify that the code requirements are met.

Model validation scrutinizes the degree to which the chosen model represents the physical system. This validation may take different forms. For example, in the case of a closed-form solution for arriving at mechanical stress, a finite-element solution could be used to validate the range of applicability of the closed-form solution. Conversely, both theoretical and empirical models can be validated against experimental data not used in their original development. The owner of the PFM code should document the steps used to validate all models.

3.2.2 Documenting Model Uncertainty As Section 3.1 suggests, model uncertainty has various sources such as the following:

• simplifying assumptions in model derivation

• scatter in the dataset used to create an empirical model

• lack of experimental data justifying theoretical models

• the need to use a simple model instead of a more accurate but computationally intense model All known sources of uncertainty should be identified and documented for each model in the PFM analysis. The documentation should explain the impact of the uncertainty on intermediate and final results. Given this known uncertainty, the user should take care in interpreting the results, in particular when assessing whether the final answer is conservative or nonconservative for the application.

Caution must be used when determining the impacts of a given input range on the final output of a code. In fact, some inputs may affect different models in different ways: an input that may lead to conservatism in one submodel may have the opposite effect in a different submodel. Any relevant bases that justify the positions described on modeling uncertainty should also be documented.

Quantifying uncertainties is useful but not always practical. If a qualitative description of model uncertainties and their impacts on results is chosen, then the analyst should explicitly state the reasons that a quantitative approach is not feasible. Expert engineering judgment may be adequate in some cases to account for qualitative uncertainty in the results.

Careful documentation of sources of model uncertainty has several benefits. First, it provides assurance that the owner of the PFM code recognizes and accounts for the limitations of the software. This can be 16

helpful for users and decisionmakers in appropriately interpreting the results. As an additional benefit, this process can identify potential areas of research for improving the PFM analysis.

3.2.3 Assessment of Alternative Models Consideration of alternative models, when available, can provide an additional layer of confidence in the quality of the PFM analysis. If the technical community considers the model to be best estimate, then results from this model should fall between those of a known conservative model and a known nonconservative model. However, most models are not consistently conservative or nonconservative, and many models will have a different bias depending on the input used. This type of analysis can bolster the technical communitys consensus opinion. It may also be useful at the model selection stage.

Diversity of modeling approaches is an important concept here, as well as in other topics addressed in this report. A numerical approach can be compared to a closed-form solution or vice versa. A theoretical model can be compared to an empirical model and the associated dataset. Such comparisons are part of an overall sensitivity study discussed above. Careful use of a high-fidelity DFM model may help in this assessment.

17

4 Inputs for PFM Analyses In PFM analyses, the common practice is to treat the inputs and associated uncertainties by representing them through probability distributions. Estimating the functional forms of these distributions and applying potential simplifications or corrections (e.g., by bounding the distribution to avoid unphysical inputs) are essential to building confidence in the end results. This section addresses these issues and related considerations.

4.1 Input Types Inputs to a PFM analysis can be represented in two ways: constant and random. The approach to choosing the type of representation for each input may focus on available data and may rely on expert judgment. Distribution fitting remains an important and ever-growing subject of interest in statistics (see Chapter 1 of [27] for instance). The two most classical techniques are moments matching and maximum likelihood (an example of such techniques can be found in [28]). Distribution fitting tools can be found in both dedicated software and generic mathematical and statistical software (for example Matlab©, R, or JMP). When the available data are relatively small or qualitative only, a broader approach relaxing the probabilistic framework can be considered [29].

Constant inputs characterize an input variable by a single point value. No matter the number of realizations used for the analysis, the input value is always the same. In contrast, random inputs can take on a range of potential values. Uncertainty in these inputs is typically represented by a probability distribution on the input space, and for this reason, uncertain inputs are sometimes also referred to as distributed inputs. Uncertain inputs are the core of a PFM analysis.

Many different probability distributions can be used to represent data [30]. These distributions can be classified as either continuous or discrete. Some common parametric forms of continuous distributions include uniform, normal, log-normal, and Weibull. The binomial distribution is a common discrete distribution. To the analyst, the parametric probability distributions are characterized using several inputs. The analyst would first specify the type of distribution, then he or she should enter the necessary parameters to define its shape. For example, values for the mean and standard deviation would be entered to define a normal distribution. Joint parametric distributions, incorporating dependencies within a group of input variables, can also be specified. After the distributions are selected, a sampling algorithm is usually used to randomly select values from the corresponding probability distribution for use in the PFM analysis. In some less common cases, methods that are not sampling based may be used, such as First/Second Order Reliability Methods (FORM/SORM), factorial design, or discretization of input values (for smaller problems). There are also other ways besides applying common parametric forms to specify probability distributions, such as nonparametric or empirical density estimation. Regardless, the framework is similar in that choices must be made regarding the type of input distribution. The next section discusses this decision process.

4.2 Construction of Probability Distributions Proper development of a representative probability distribution for an uncertain input requires a detailed knowledge of the available data, as well as qualitative judgments. In reality, however, collection of available data is always limited by cost and time, and developing a robust uncertainty distribution for every input may be impossible because of these limitations. The data itself also contain 18

biases through measurement uncertainties, omissions, and other factors. Ultimately, the analyst should develop the uncertainty distributions with relevant data and be able to defend the chosen distribution.

The amount and pedigree of the source data, as well as the influence of the particular input parameter on the results of the overall PFM analysis, are important considerations when justifying a distribution.

The typical approach is to gather a set of empirical data and then choose one of the parametric probability distribution forms as the best representation of that data. Many goodness-of-fit statistical tests exist and may also be used to assess how well a given distribution is suited to a dataset. Such tests include Kolmogorov-Smirnov [31, 32], Anderson-Darling [33], Shapiro-Wilk [34], and many others. In practice, engineering judgment frequently drives the selection of the probability distribution form.

Large data sample sizes are typically needed to accurately determine the form of the underlying probability distribution, though the minimum number of datum needed for a suitable fit may be subjective and context specific. However, the sample size is directly linked to the resolution of the desired probability distribution. Thus, fewer data can be acceptable when a coarse resolution is sufficient. Uncertainty analyses can be used to gauge the level of resolution that is appropriate for the analysis.

When choosing a probability distribution, several important parameters must be considered. First, to ensure that the correct support is selected, it is important to consider the minimum and maximum of a distribution. Next, all that is known about the input being considered should be listed. Last but not least, the shape of the distribution should also be considered. Common summaries of distribution shape include skewness and kurtosis. Skewness is a measure of the lack of symmetry of a distribution.

Kurtosis is a measure of how heavy the tails of the distribution are and is typically quantified relative to the normal distribution. Estimating higher order moments of statistical distributions, such as skewness and kurtosis, typically requires large sample sizes; subsequently, expert judgment, coupled with data, is often used to inform distributional shape. Given that the tails of distributions often drive low-probability events such as structural failures, it is important to investigate how much confidence there is in the underlying probability distributional form and whether the specified distribution fits the underlying data well in the tails. Similar but not identical distributions may sometimes lead to the same answer for a given problem (e.g., in the case of a beta distribution matching a truncated lognormal distribution or in the case of a triangular distribution that is close to the truncated normal distribution).

Once the form of the probability distribution has been specified, statistical estimation techniques can be applied to estimate the parameters of the distribution. Examples of such statistical methods include maximum likelihood estimation, moment-matching, and Bayesian inference. The analyst may need to assess how confident he or she is with the estimates of the distributions parameters. Even small changes to the standard deviation can produce markedly different behavior in the tails of a distribution, so understanding and accounting for uncertainty in distributional parameter estimates are important.

This can be achieved by performing sensitivity studies on input distribution parameters or by studying the impact if changing distributions for important inputs.

In PFM, it is sometimes necessary to estimate a probability distribution when sparse or no data are available. Literature and expert opinion are often sought in these cases. Key considerations are how the literature and expert information were developed and the extent to which they apply to the system under study. Different opinions from the literature and experts should also be sought and considered as applicable. The NRC has developed formal procedures for the use of expert judgment in other technical areas [35]; these procedures and the more general literature on this subject can be used effectively in such situations. When compared to probability distributions developed from data, it can be challenging 19

to justify distributions developed from limited data; however, use of formalized procedures for treatment of expert opinion, coupled with the physical constraints inherent in the distributed quantity, can add robustness to the justification. Sensitivity studies aimed at quantifying the impact of the choice of distribution type and distribution parameters may be used as a means to increase confidence in the ultimate outcome of the analysis.

4.3 Input Bounds Many standard probability distributions have infinite support (i.e., they can take on an infinite range of possible values). In practical engineering applications, inputs are typically bounded, and probability distributions need to be truncated to accommodate the finite range of the inputs. Input bounds are the upper and lower truncation points applied to input probability distributions. These bounds are typically used to prevent the sampling algorithm from selecting input values that are undesirable or nonphysical.

For example, an analyst might select a certain bound to constrain the sampled inputs so that they remain within a models specified range of applicability. Such an approach is often used on the front end of PFM analyses to prevent calculation errors, but limiting the input range because of model deficiencies should be properly documented and justified. The analyst may also wish to impose bounds to prevent a nonphysical response from the system. Negative material strength is such an example.

Selecting appropriate input bounds can be just as important as selecting the distribution itself because the bounds determine where the tails of the distribution are cut off and thus can have a large impact on uncertainty in the outputs. If bounds are to be applied to an input, the sensitivity studies discussed in the previous section could assess their effect.

The selection of input bounds can range from straightforward to complex. Choosing bounds to constrain a model within its range of applicability can be straightforward if the bounds are derived directly from the technical basis for the model, but they should nonetheless be accounted for. It is important to exercise care if that same input is also used in another model with a different range of applicability. For example, it is easy to picture a PFM code that can model crack growth as a function of applied stress intensity factor K for both fatigue and primary water stress-corrosion cracking. In such a case, if one of the crack growth models is defined over a wider range for K values than the other, restricting K values to the smaller of the two ranges of applicability may not be an adequate choice. In fact, choosing to limit input values within a models domain of applicability may artificially suppress uncertainty, as well as some outcomes in the model predictions, particularly if the domain of applicability of the model covers only a small fraction of what is physically possible. Accordingly, it is important for the analyst to understand how the various inputs may interact within a model of a complex system. The selection of physical bounds can be more complex. Sometimes, the true physical limit may not be known, even if the distribution is a based on a large amount of high-quality data.

Expert opinion is thus often necessary to establish physical limits. The technical basis supporting these opinions should be documented along with explanations for alternate opinions that were excluded.

Additionally, although it is critical to set an appropriate bound for a distribution tail that is known to drive failures, it may be just as important to set a bound on the noncritical part. This is because unrealistic values sampled from this region could still alter the overall results when a number of realizations are analyzed in aggregate.

4.4 Quantification of Input Assumptions A PFM analysis focuses on those inputs that have the most influence on the output of interest.

Uncertainty analyses are used to quantify the contributions of individual uncertain inputs to the 20

uncertainty in the results. Once the input sensitivities are known, the analyst can focus on those inputs that have the most impact. If the input uncertainty has very little impact on the output uncertainty, a strong technical basis for the input distribution may not be necessary. If low-probability events are the desired outcome, it may be appropriate to scrutinize the appropriateness of the tails of the input distributions that are driving the output uncertainty. In this case, sensitivity analyses and sensitivity studies that vary the input distribution type or parameters can be conducted to determine the impact of the chosen distribution type and its representation of the tails. If these results indicate a large impact, more data, more refined statistical techniques, or additional expert elicitation may be needed to further refine the inputs probability distribution. In this way, the development of inputs for PFM analysis can be an iterative process.

Section 6 discusses the many different methods for determining input sensitivity [36].

4.5 Conservatisms Conservative treatments are the norm in deterministic analyses; however, they may not be appropriate in a PFM analysis where the objective is often to generate a best-estimate distribution for the probability of occurrence of an event, and use a specific percentile (which will likely depend on the application) of that distribution as the final answer to the PFM analysis. Of principal concern is the bias that conservative inputs can impart to the PFM analysis results. In fact, in complex models, use of conservative assumptions may not always lead to conservative results. For example, choosing a high through-thickness crack growth rate may increase the likelihood of a leak instead of failure, because less time will elapse before a leak, leaving less time for a long part-wall circumferential crack to develop and become unstable. On the other hand, some conservative assumptions can mitigate complex and uncertain inputs (e.g., by assuming that a crack has initiated as opposed to using a problematic crack initiation model). Therefore, it is important to document all inputs considered to be conservative rather than best estimate. In some cases, it may be acceptable to have multiple inputs that are either conservative or nonconservative, if they are self-correcting in nature. The analyst should quantify the influence the conservative inputs have on the results so that the end impact of all conservatisms (and potential nonconservatisms) is understood. If the influence is minor, then the conservatisms may be sufficient. However, if the inputs have a strong influence, additional work may be needed to refine the conservatisms.

Two other risks are associated with the use of conservatism. First is the risk of having too many variables set to conservative values, leading to a case that is extremely unlikely (for instance, 10 variables set at their 90th percentile limits the analysis to an area in the input space with a probability of 10-10 or 1 chance in 10 billion). Second is the risk of erroneously considering a parameter as conservative because of its role in a submodel, while its overall role in the whole system becomes nonconservative (e.g., high crack growth rate assumption leading to an increased leak rate may be considered conservative but may result in events being suppressed because of leak rate detection). In general, for a PFM analysis, it is appropriate to have all random variables set at their 50th percentile, or mean values. However, it may be appropriate to have constant variables set at a higher percentile, depending on the level of uncertainty in these variables and the specific context of the analysis.

4.6 Dependencies In a PFM analysis, most distributed inputs are assumed to be statistically independent from each other and are sampled as such. However, a subset of input variables is often statistically dependent. The 21

association between variables is often needed to ensure a physically possible input set, or to preserve the physical laws that drive the problem being modeled. For example, the yield stress of a material is usually smaller than its ultimate tensile stress, so a relationship between these variables may be imposed to ensure physicality of the inputs. Dependence between variables may or may not be causal.

If two or more variables should be dependent according to, for example, the laws of physics, the input set should be checked for consistency and physicality. For example, this is the case for the yield strength of a material, which is always lower than its ultimate tensile strength.

Known variable dependencies can be represented by correlating two or more variables via mathematical relationships. This is desirable to avoid underestimating the influence of correlated variables. For example, if two variables are highly correlated with an output variable, it is likely that they are also correlated with each other, possibly even more strongly than they are correlated with the output of interest. If so, putting both of them in the model will result in some multicollinearity, risking both of them being declared insignificant, and skewing the correlation coefficient estimates.

22

5 Uncertainty Characterization and Framework for PFM Analyses This section defines epistemic and aleatory uncertainty types and describes the process for identifying and classifying uncertain parameters as epistemic or aleatory. The section also discusses a probabilistic framework for uncertainty propagation in a PFM analysis and suggests ways to verify and validate such a probabilistic framework.

5.1 Epistemic and Aleatory Uncertainty The purpose of safety analyses is usually to assess the risk of some unwanted outcome, which can be interpreted discretely as a pair (ci, pi), where represents the consequence (or severity) of a particular outcome i, and pi its likelihood expressed as probability (which can also be expressed as a frequency).

PFM analyses can be used to estimate this probability pi. To do so, Monte Carlo techniques essentially regroup all uncertainties into a single integral, reducing the dimensionality of the problem to a more manageable size. The tradeoff is that it is not possible to directly reconstruct the multidimensional solution. In other words, statistical techniques or the use of response surfaces only approximate the full extent of how uncertainty affects the output.

While it would be impractical to consider the multidimensional problem with all its aspects (not only with respect to the unrealistic number of runs that would be required but also because the interpretation would rapidly become too complex to comprehend), the risk community has for decades made the distinction between the inherent risk of each problem and the uncertainty resulting from lack of knowledge.

The purpose of this distinction is to acknowledge that there will always be a risk given a specific situation. The risk can be the result of unwanted (and sometimes unforeseen) events without control (e.g., natural disasters such as seismic events), human interaction (risk of error), or physical limitation (e.g., strength of the material with respect to the condition of use or flaws that are undetected because of their size or nature). While this risk can be mitigated somewhat (thus changing the problem considered), it is inherent and present in each problem studied and ultimately is the output that needs to be quantified and estimated for decisionmaking. In addition to the risk itself, there is the uncertainty caused by lack of knowledge. This may stem from simplified models, lack of data to express a poorly known input, indecision as to which model or input best represents the problem at hand, or any combination of the above.

While both risk and lack of knowledge need to be acknowledged to have a good understanding of what may happen and what the consequences would be, their interpretation in terms of decisionmaking differ:

• Risk would require a change in the problem considered to be affected (such as mitigation or change in inspection schedule). This type of uncertainty is called aleatory uncertainty.

• Lack of knowledge may (in theory) be reduced by more research on a particular model or input and does not necessarily require a physical change in the problem being considered. This is called epistemic uncertainty.

23

Daneshkhah [37] provides the following definitions for aleatory and epistemic uncertainty:

• Aleatory Uncertainty: This uncertainty arises because of (perceived) natural, unpredictable variation in the performance of the system under study over which there is little control (the variability in fracture toughness), or the inherent randomness in the future (for example, adverse events such as natural disasters or possible human error). The knowledge of experts cannot be expected to reduce aleatory uncertainty although their knowledge may be useful in quantifying the uncertainty. Thus, this type of uncertainty is sometimes referred to as irreducible uncertainty. This implies cases where there is no way to know for sure what will happen or what the value is or will be, but based on past experiment or observation, there is a sense of what may happen. It can also represent error in experiments used to gather data (accuracy of the material used, inherent randomness from the experiment).

• Epistemic Uncertainty: This type of uncertainty stems from a lack of knowledge about the behavior of the system that is conceptually resolvable. That is, it could potentially be reduced by further research, model development, model choice, or testing. It may also result from the choice of a model that is necessary for PFM because of solution time constraints. This uncertainty reflects the level of confidence that people have about a possible value or outcome (considered as unique but poorly known); this confidence may be increased by gathering more information.

Most inputs contain both aleatory and epistemic uncertainties but are nonetheless usually categorized as either one or the other, not both. However, these two uncertainties are often combined, since information often exists that may reduce uncertainty up to a certain point. This is why the expression perceived randomness is used to define aleatory uncertainty.

The approach sometimes used by the risk community to separate aleatory and epistemic uncertainties is to split the unique Monte Carlo integral into two: one to represent aleatory uncertainty and the other epistemic uncertainty. The usual numerical implementation consists of a nested sample (estimated with a double loop). The outer sample is traditionally reserved for the epistemic uncertainty, while the inner sample is used for aleatory uncertainty. For each outer loop, a single sample of the epistemic parameters is selected and held constant, while within the inner loop, aleatory parameters are sampled the desired number of times (NA). This is repeated for the total number of epistemic realizations (NE).

Thus, each epistemic outer loop has NA number of possible outcomes and (NE x NA) represents the total number of possible outcomes generated in the model simulation. The group of aleatory samples under a single epistemic set of parameters is a sample from the conditional distribution of the aleatory variables given the epistemic variables (thus supposing that there is no lack of knowledge). One possible way to summarize this conditional distribution is to estimate a quantity of interest (QoI) from each conditional distribution (e.g., an average, a percentile, or in the specific case of PFM, the probability of failure for a given realization of the outer epistemic loop). Results stemming from the epistemic loop represent the level of knowledge with respect to the probability of an event (in which the probability is represented by the aleatory distribution function, usually represented as a complementary cumulative distribution function).

The cost of separating between aleatory and epistemic uncertainty is evident in the double loop design. For example, a 100 x 100 sample size would require 10,000 calls from the code while giving only 100 different values for each epistemic parameter. If 10,000 epistemic realizations are required in this example, separating aleatory and epistemic uncertainty would imply performing 1 million calls to the 24

code (10,000 x 100). Therefore, the cost is increased tremendously. However, the gain is, as described below, more insight into the problem. Therefore, the choice of separating between those two categories in a PFM depends on the nature of the problem and the outcome. It is important to point out that the uncertainty classification does not affect the mean for a given quantity. That is, using the previous example, the mean over all 10,000 aleatory realizations of the simulation is the same as the mean over the 100 epistemic realizations, since each epistemic output value is actually the mean of the 100 aleatory realizations that make up each epistemic realization.

If an analyst can answer yes to the following questions, then a separation would be appropriate:

• Can the probability be estimated accurately enough with the size of each sample? (If the probability of the event is driven by epistemic uncertainty and one cannot afford an epistemic sample large enough to generate enough events, then the answer is no.)

• Can one distinguish between aleatory and epistemic uncertainty for the most influential parameters? (If epistemic and aleatory uncertainty are mixed together without possibility of distinction, then the answer is no.)

• Does the separation help to make a stronger, more comprehensive case, or help to understand what needs to be done to improve the accuracy of the answer? (If no mitigation or no improvement of the model can be made, or if the separation confuses the decisionmaker, then the answer is no.)

The appropriate separation of aleatory and epistemic uncertainty can be an important component of the design and computational implementation of an analysis of a complex system, as well as the decisions that are made on the basis of that analysis. For instance, knowledge uncertainty, which is akin to epistemic uncertainty, must be differentiated from spatial and temporal variability, which may be more aleatory in nature. Quantification of the aleatory uncertainties may answer the question, How likely is the event to happen, while quantification of the epistemic uncertainties may answer the question, How confident are we in the first answer. Understanding and properly defining these uncertainty types allows for a quantitative categorization of the uncertainty of PFM results.

Characterization of the uncertainty as being aleatory or epistemic is not absolute; it depends on the context and granularity of the problem. The most common approach to categorizing uncertainty is expert elicitation. This does not mean that the process is arbitrary, but that a careful effort should be made in describing each uncertain parameter, including a rationale for its characterization. The interpretation of results depends on this characterization. The more important part is thus to clearly explain the rationale behind each decision. As an example, in a conventional LEFM model, the uncertainty in the linear elastic fracture toughness (KIc) may be regarded as aleatory (irreducible).

Conversely, in a micromechanics model that accounts for features such as grain size, inclusions, dislocations, and so on (that is, the factors that create the uncertainty in KIc), this uncertainty may be regarded as epistemic. Mixed situations (part aleatory, part epistemic) are also possible.

Usually, if the uncertainty of a particular parameter, variable, or model cannot be clearly defined as aleatory or epistemic, an analyst may want to treat that uncertainty as epistemic within a sensitivity study. This allows the user to rank the epistemic uncertainty (for each uncertain parameter) according to its contribution to the output uncertainty, which may give the user an understanding of the importance of that epistemic uncertainty to the output uncertainty. For example, the weld residual stress may be characterized as epistemic. The analyst can then look at the output uncertainty with that 25

input uncertainty separated and see, for example, that the output 95th and 5th quantiles are 5 orders of magnitude apart, suggesting that this uncertainty is overwhelming the results. Once the major contributors to the uncertainty in the output are known, it is customary (and recommended) to revisit the uncertainty associated with these inputs, both in terms of representation (distribution type, parameters) and characterization (epistemic, aleatory).

5.2 PFM Analysis Uncertainty Identification and Context A key aspect in the development of PFM models is the determination of random variables and their treatment throughout the analyses. When random variables have been selected, their influence on the performance metrics needs to be evaluated through quantitative techniques such as analysis of variance (ANOVA), as well as qualitative techniques such as scatterplots (see for instance [38, 39]). Constant (deterministic) variables may also need to be evaluated, usually via one-at-a-time sensitivity studies, as the choice of a constant value may affect the results. As most PFM codes include many inputs, it is often the responsibility of experts and analysts to select which constant inputs will be studied with respect to their impact on the performance metric (based on the equations implemented). Once the important variables are identified, technical justification is needed to (1) confirm that the variable representations appropriately reflect major sources of uncertainty, (2) ensure that model parameter representations (i.e., probability distributions) are reasonable and have a defensible technical basis, and (3) classify the random variables as epistemic or aleatory for the purpose of the PFM analysis.

The context of the analysis may have a direct impact on the uncertainty representation and type classification. The more specific an analysis, the narrower the uncertainty ranges may be. For example, if an analysis is specific to a given pipe in a given plant, the geometry and other characteristics of the system are likely to be precisely defined. As a result, the uncertainty range may be relatively small, and the associated distributions relatively narrow, potentially warranting the classification of geometric variables as constants. Furthermore, the geometry and other characteristics of the system actually correspond to a physical quantity that can be measured; thus, any uncertainty about the dimensions could be categorized as epistemic if the geometric variables are considered random instead of constant.

In contrast, for an analysis meant to represent a series of welds or generic configurations across the U.S. reactor fleet, the variability in geometry, operating conditions, materials, and possible flaw mitigation is likely to be larger and thus result in wider distributions for uncertain variables.

Furthermore, since the exact geometry and other characteristics of the system vary from plant to plant and do not correspond to a single value that could physically be measured, the uncertainty may in this case be classified as aleatory instead of epistemic. In turn, this is likely to have an impact on the variability of the results of the PFM analysis.

5.3 Probabilistic Framework, Uncertainty Propagation, and Sampling Techniques To determine the probabilistic distribution of a predicted response, the analyst propagates the input uncertainties through models (which may be themselves uncertain) to estimate uncertainties in the results. To achieve this goal in the context of PFM, it is useful to create a probabilistic framework that can adequately represent input uncertainties, propagate them through a fracture mechanics model, and adequately process the output uncertainty. The most effective way to propagate uncertainty ultimately depends on the computational constraints, as well as the nature of the inputs and outputs under consideration.

26

Various methods exist for propagating uncertainty using a probabilistic framework. For those situations where the response surface is regular and continuous, methods such as First and Second Order Reliability Methods (FORM and SORM) [40] constitute reliable computational methods for structural reliability assessment, based on the joint probability density of all the factors influencing failure or nonfailure. However, for many complex problems where the response surface is difficult to analytically describe or highly discontinuous, probabilistic sampling-based methods may be used. In this case, the probabilistic framework should be capable of sampling the input space using a predefined sampling technique and simulating the response of the system for each sample generated. Each simulation is referred to as a realization of the system, and uncertainty propagation is achieved through the use of the probabilistic framework as follows:

(1) For each realization, the probabilistic framework samples all of the uncertain parameters and variables. If aleatory and epistemic uncertainties have been identified and separated, two nested sampling loops may exist in the framework: an outer loop for the epistemic variables and an inner loop for the aleatory variables. Section 5.1 presents potential reasons for separating aleatory and epistemic uncertainty.

(2) The system is then simulated (given the particular set of input parameters) such that the response of the system can be estimated. This results in many separate and independent results, each representing one possible future state of the system.

(3) The results of the independent system realizations are assembled into probability distributions of possible outcomes and may be analyzed by the framework or by using other postprocessing tools to study the probabilistic response of the system.

Because an important role of the probabilistic framework used to propagate uncertainty through a system is input space sampling, this section briefly discusses sampling techniques commonly used in PFM analyses to propagate uncertainty. Monte Carlo sampling techniques [41] are perhaps the most commonly used, and in this case, the uncertain parameters and variables should be sampled from their distributions. There are many techniques for sampling, all of which intend to represent the distribution for the particular analyses desired. For instance, simple random sampling (SRS) can adequately cover the input space, but accurate modeling of low-likelihood events requires many samples which can be prohibitive. Latin hypercube sampling (LHS) is one method that may be used to get more uniform coverage of the sample space [42]. LHS divides the input space into equal-probability strata based on input distribution quantiles, and a fixed number of samples (usually one) is obtained from each stratum.

LHS covers the input space more uniformly than random sampling for the same sample size and thus typically increases precision in characterizing the output distribution. Discrete probability distribution (DPD) is another method of increasing the stratification by discretizing each probability and reducing the monodimensional coverage of the input space in the benefit of multidimensional aspect [43]. If low-probability events are being assessed, techniques such as importance sampling, adaptive importance sampling, or stratified sampling [44] may be used. These techniques allow the analyst to specify increased weighting for the part of the distribution that has the most effect on the response (based on the results of sensitivity studies on an initial analysis) so that low-probability events can be found with limited realizations. However, the weighting should be consistent and properly propagated throughout the analyses. Furthermore, different sampling techniques may require different levels of convergence testing to ensure converged solutions. For instance, LHS and importance sampling have been built into the xLPR code.

27

5.4 Verification and Validation High confidence in a PFM analysis requires that any PFM software developed for the analysis undergo V&V. Verification establishes the correspondence between the PFM computer code and its specifications, while validation establishes whether the PFM code is fit for its intended purpose. The primary purpose of the PFM software is usually to estimate the failure probability of a component in a problem driven by fracture mechanics to the best extent possible based on the best available input distributions. However, examining other outputs of the PFM model can be as important in meeting the performance requirements for the system being analyzed. Furthermore, as seen in the development of the xLPR code for example, small probabilities are hard to predict with high accuracy and difficult to validate, and thus other outputs may be more informative of risk.

All plant-specific activities affecting the safety-related function of NPP structures, systems, and components must comply with the QA requirement of 10 CFR Part 50, Domestic Licensing of Production and Utilization Facilities, Appendix B, Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants [20]. For software applied to plant-specific issues by the licensee, this requirement implies that a documented QA program should exist that contains the requirements for the PFM software, including development, procurement, maintenance, testing, and configuration management. The purpose of the QA program is to ensure software verification. Furthermore, the QA plan should follow well-defined criteria, such as those specified in ASME NQA-1, Quality Assurance Requirements for Nuclear Facility Applications [45] or NUREG/BR-0167, Software Quality Assurance Program and Guidelines, issued February 1993 [46].

In addition to following a strict QA process for the development of a PFM code, it is important to validate both the individual deterministic models that compose the PFM code and the integrated PFM code. Ideally, all outputs from individual deterministic fracture mechanics models can be validated against analytical solutions that have been shown to accurately represent reality, field (or relevant experimental) data, advanced finite-element solutions, or all three, if applicable. Once the individual models have been validated, the relevant outputs from the integrated PFM code can also be validated for each specific application of the PFM code. For a given application, it is desirable to validate all important outputs (those that represent the QOIs for the problem being analyzed) over the entire range of possible inputs. Validation of a given output of interest may not ensure validation of all outputs, nor does it ensure validation of that specific output for a range of inputs outside of those used during the validation process. For example, in the case of primary coolant system piping in a nuclear reactor, the validation of crack growth rate and through-wall crack opening displacement predictions does not ensure the validation of a leak rate prediction. Furthermore, still using the same example, the predictions of crack growth and crack opening displacement may be validated only for the temperature, pressure, and chemical environment applicable to steady-state operation in a light-water reactor; thus, they may not be applicable to other reactor types that use different operating conditions or a different type of coolant. 3 Consequently, the validation of an integrated PFM code is application specific and should be tailored to the application. It is possible that validation for one application is sufficient to demonstrate validation for some other applications, but this should be verified on a case-by-case basis.

3 Extensive deterministic model changes would be necessary for a PFM code that was developed for light-water reactors compared to a high-temperature air or sodium-cooled reactor since crack growth mechanisms, damage development, and stability will be different.

28

Establishing and meeting the acceptance criteria for the validation of an integrated PFM code over a broad range of conditions can be complicated by limited availability of field data and by the probabilistic nature of the code. As a result, the software could be validated using a graded approach, in which the extent of testing applied to different individual models of the software is commensurate with the importance of the individual model (for the problem being analyzed) and the risk that the software element does not function properly at the integral level. For instance, if a piping PFM code is used to calculate leakage and rupture, and the purpose of the analysis is to demonstrate the impact of inspection on leakage times, the rupture portions of the code (i.e., through-wall flaw stability) may not need the same V&V rigor as the leak rate calculations. Nonetheless, when possible, the software results should be validated to the extent practical against available field data and hypothetical test cases (e.g., impact of piping rupture probability on varying inspection intervals or probability of detection) that have been developed based on engineering judgment. For instance, the xLPR PFM code was partially validated by comparing predictions of crack depth versus time to crack occurrences that occurred in operating plants. The level of testing should be chosen to provide reasonable assurance that the PFM code makes sense in the context of the intended use of the software.

The extent of the V&V and other aspects of QA, such as documentation of the technical basis for the PFM code, could vary depending on factors such as the following:

• The application of the code, for example whether the code is used to support an inspection interval, license extension, or a permanent change to the plant configuration (e.g., remove pipe whip restraints). The more permanent a change, the higher the requirements should be for V&V.

• Scale and complexity of the code and models. A more complex code would require more extensive V&V.

• Whether the code is preverified software or one-time-use software (in which case the calculation results are verified and validated on an individual-run basis, thus requiring less V&V up front), though many single uses of one-time-use software may result in a high degree of verification.

• Whether submodels are applied that have consensus support as standard modeling approaches (similar to standard deterministic approaches that are accepted without specific validation documentation).

The events that a PFM code is designed to model and predict are often of very low probability, (e.g., the rupture of a pipe in the primary reactor coolant system). As a result, little or no field data may be available to validate such low-probability events. In some cases, precursor events may be used to validate the PFM code, such as incidences of cracking in reactor coolant system piping, which could lead to piping rupture if not detected and repaired. In other cases, experimental data applicable to the phenomenon being studied might be available. Nonetheless, validation based on field or experimental data can be challenging and limited. In such cases, other studies can be performed, such as sensitivity or model behavior studies and benchmarking against other comparable codes. Sensitivity studies can be used to demonstrate that the overall behavior of the PFM code is consistent with expert understanding of the expected system behavior, including demonstrating expected trends and correlations between inputs and outputs of interest. Benchmarking against other comparable codes may be used to increase confidence in the PFM code by demonstrating that the results produced by the PFM code are 29

reasonable and can be predicted by similar codes [47]. Although benchmarking alone is not sufficient to validate a PFM code, it can significantly contribute to ensuring the accuracy of a PFM code.

In general, peer review by one or more independent parties, detailed documentation of the theory and structure, and independent effort to confirm or reproduce results (either with the same PFM code or another one with similar capabilities) would also increase confidence in the PFM code or analysis.

30

6 Analysis of PFM Outputs and Results Producing reliable, realistic results from PFM analyses requires more than a single probabilistic analysis with a given PFM code (referred to as a run of the code). The user should conduct multiple runs to demonstrate that the solution is converged, the uncertainties are properly represented, and the parameters driving the problem have been sufficiently characterized. This chapter discusses how convergence of a PFM analysis can be demonstrated, how to study output uncertainty distributions, how to extract information from a PFM analysis via uncertainty analyses, and finally how to identify problem drivers (i.e., important contributors to the output) and gain confidence in the PFM analysis via sensitivity studies.

6.1 Convergence of Results The concept of convergence is intrinsically linked to the decision that must be made after reviewing the PFM results. Convergence is often a matter of perspective and really comes back to whether one has enough confidence in the results to use them in making a decision. As such, before addressing the convergence of an output, it is necessary to consider a threshold value (or range) for each output considered in final decisionmaking. This threshold constitutes the pass/fail criterion for the problem considered. The second point of consideration is how much confidence is required in the final answer or decision, often linked to the seriousness or magnitude of the consequence. With relatively small safety consequences, having a (theoretical) higher 95th or 99th percentile close or at the threshold may be considered acceptable. With more serious consequences, the 95th or 99th percentile may be required to be at least 1 order of magnitude from the threshold for increased safety.

Once these two criteria are defined, a solution is considered converged enough if the uncertainty is sufficiently low that it does not change the conclusion that could be drawn from the analysis. As a result, output statistics far from the threshold can afford large uncertainties, while output statistics close to the threshold will require tighter confidence bounds.

To demonstrate convergence, one should consider temporal (if the problem is time based), spatial (if the problem is location based), and statistical solution convergence. Temporal or spatial convergence can be demonstrated by simply running the code with different (either increasing or decreasing) time steps or spatial discretization until the solution diverges. However, the solution convergence for any given output does not guarantee convergence for all outputs. As a hypothetical example, convergence could be achieved for crack length for a coarse azimuthal spatial discretization, while convergence on crack azimuthal position may require a much finer discretization. Similar convergence tests may be needed for each unique desired output.

Demonstrating solution, or statistical, convergence can be a bit more complicated and may be achieved via stability analysis. Simulation options to explore include (but are not limited to) the following:

• sample size

• sampling strategy (SRS, LHS, DPD) [48], importance sampling)

• number of strata to use for DPD or LHS

• degree of spatial or temporal discretization for the problem being analyzed, as applicable 31

The sampling options chosen for consideration are application specific but, without further guidance, should include at a minimum the sample size and sampling strategy. Plots of the QoI as driven by acceptance criteria (e.g., mean probability of crack depth, leakage, and rupture) under the different simulation options should be provided and interpreted. These plots will help with decisions on the convergence of the solutions. When considering the number of strata to use for DPD, the number of segments for spatially varying parameters, or time step size, results from more granular simulations should be sufficiently close to results with finer resolution. Note that convergence results depend on the output of interest and the purpose of the application. For example, acceptable convergence for a mean probability will likely be achieved sooner than convergence for a large quantile of the distribution of probabilities.

Sampling techniques are numerical procedures and thus may result in varying degrees of accuracy. The accuracy depends on the method and the sample size used. As discussed in Section 5.3, several types of sampling techniques can be used in the development of a probabilistic output. For Monte Carlo type analyses, conducting multiple runs with replicated samples using a different random seed is the most straightforward solution. Another approach is to increase sample size until the desired result converges.

The advantage of the first approach is that if SRS or LHS is used, it is possible to increase the sample size while testing replicates. However, with computational time limitations or advanced sampling techniques (importance sampling, etc.), these convergence runs may not be practical. Statistical uncertainty quantification methods can often be applied to characterize uncertainty in estimates of a QoI caused by sampling uncertainty, but these methods must account for the sampling scheme (e.g., importance sampling, LHS). By either repeated convergence runs or statistical uncertainty quantification, demonstrating solution convergence adds greatly to the confidence in the probabilistic output.

6.2 Output Uncertainty Quantification (Uncertainty Analysis)

Once convergence of a PFM analysis has been shown, one can perform uncertainty analyses on the outputs of interest (e.g., to compare the likelihood of an event to a safety metric). Uncertainty analyses are the calculations required to develop the uncertainty distribution of output from distributed inputs.

Output uncertainty analysis can take different forms depending on the nature of the output or answer being sought. Temporal outputs may consist of creating distributions for the probability of an event or the value of a parameter of interest over time, while spatial outputs may consist of distributions of the location of a given event if it happens. To represent the uncertainty of the output, the analyst may choose to show the temporal or spatial output for each and every realization of the problem or to represent the result via one or more statistical quantities.

The relevant QoIs that can be defined as a result of uncertainty analysis depend on the acceptance criteria for a given problem and should be justified. Whatever the choice, the analyst should explain why the statistical quantities being chosen are a relevant representation of the output uncertainty. For example, for the overall probability of leakage as a function of time, the mean of the probability of leakage over all epistemic runs is probably the most representative statistical quantity and can be termed the expected leakage probability. For conservatism, a large quantile for the probability of leakage, such as the 95th percentile, may be preferred as the primary QoI in a safety analysis. In this particular example, the probability of leakage is known for each individual realization and based on an indicator function: it is set to 0 if the system is not leaking, and 1 if the system is leaking. In another example, a relevant way to represent the uncertainty of a spatial output may be via a probability density function (PDF) or cumulative distribution function (CDF) as a function of location (i.e., a CDF and PDF for 32

each location, resulting in a set of PDFs and CDFs at different locations), or a PDF or CDF representing the location variation. However, such risk numbers are not absolute but depend on the deterministic models chosen to characterize the PFM code. Introduction of different deterministic models will change these numbers.

6.3 Sensitivity Analysis (Input Importance Quantification)

An important aspect of PFM analyses consists of understanding the relationship between problem input and output uncertainties. In particular, it is useful and important to identify the drivers of uncertainty for any given problem, and this goal can be achieved by performing sensitivity analyses. Sensitivity analyses are also important in helping to develop sampling options that provide converged results.

Sensitivity analyses can be performed using a variety of methods which can be grouped into two main categories: graphical qualitative methods and quantitative variance-explained methods. A common graphical qualitative method looks at the relationship between uncertain inputs and outputs through the use of scatterplots of the output variables against individual input variables. The advantage of this approach is that it can deal with any set of arbitrarily distributed input and output variables and give a direct qualitative visual indication of sensitivity. Scatterplots can also be used to help guide decisions about which inputs may be good candidates for importance sampling. For large problems with many uncertain inputs and outputs, it is not practical to create scatterplots for all variables, and data reduction may be needed.

Quantitative metrics should also be used to quantify and rank the importance of the relationship between inputs and outputs. The proportion of variance in the output driven by each input can be calculated using a large number of model runs; when this is not possible, regression analysis methods are often applied as a computationally efficient means of quantifying and ranking the importance of the inputs. Many methods exist for regression analysis, and in general, the NRCs experience with running sensitivity analyses [49, 50, 51] dictates that three or more methods should be combined to ensure that false positives for correlation are eliminated, as well as to increase confidence that the correlations found are significant. The specific number of regression methods is not as important as showing that the final results are comprehensive, well supported, and well documented for the problem under analysis. Some common regression analysis methods include linear regression, rank regression, quadratic regression, Gaussian regression, regression trees, and multivariate adaptive regression splines, among others.

6.4 Sensitivity Studies In addition to uncertainty analysis and sensitivity analyses, it is expected that any PFM analysis submitted to the NRC for review include sensitivity studies to explore the specific influence of important parameters and analyze specific scenarios of interest. Sensitivity studies are useful to better understand the physics of a problem, to show that some expected trends are indeed reflected in the analysis results.

Sensitivity studies are also used to perform what if analyses, to revisit some assumptions (constant values or distributions changed), and to investigate alternative scenarios in support of defense in depth [52].

Deterministic sensitivity studies consider one (or a set of) input(s) of interest and change it in the deterministic reference to estimate how much it impacts a specific output. Experts change the parameters that could influence the output (based on the previous sensitivity analysis or their own 33

expert judgment) one at a time. Such analyses are also performed to assess the impact of some alternative scenarios (such as extreme condition or mitigation). Deterministic sensitivity studies usually focus on the physics of the system and build confidence in the theoretical aspect of the analysis.

Probabilistic sensitivity studies are the probabilistic equivalent of the deterministic sensitivity studies.

They allow the comparison of the effect of changing one input, not only from a median reference but from different points of reference in the input space and globally. The change in input can be as for the deterministic case (with a shift) but also as a change in uncertainty (spread) or point of interest (skewness). If epistemic and aleatory uncertainties have been separated as part of a PFM analysis, one may also change the uncertainty classification for some variables to investigate the effects of such changes. These studies include the probabilistic aspect which focuses more on the changes in probabilities of occurrence of an event of interest associated with each change.

6.5 Analysis of Problem Drivers and Confidence Demonstration Once the main uncertainty drivers have been identified for all outputs of interest, the uncertainty of the inputs that drive the problem should be revisited to gain confidence in the PFM analysis conclusions. It is important to ensure that the distributions characterizing the uncertainty of the problem drivers are representative, and it is even more important to study the consequences if these distributions are not representative. For example, if the uncertainty of the problem drivers is so large that the 95th percentile of the PFM results does not meet acceptance criteria for the problem being analyzed, the analyst may conclude that more research is needed to reduce this uncertainty. Such a reduction may require additional analyses or additional data collection. Similarly, if the distributions of the problem drivers are found to not be good representations of actual data, the analyst should investigate the consequences of changing the distributions of the problem drivers. If the change in output is minimal, it may still be possible to reach the same conclusion. Varying the distribution of an input of interest does not necessarily require rerunning the entire analysis as long as the support of the original distribution (i.e., the domain of the distribution containing those elements that are not mapped to zero) contains the support of the new distribution. Instead, the distribution may be changed by altering the weights of each realization according to the relative change in probability of occurrence when changing the distribution for the input of interest.

To further increase confidence in the conclusions, the areas in the distribution that could change the conclusion should be identified and sufficiently sampled for each significant driver of the problem. If these areas are in the tails of the distributions, which is often the case, importance sampling may be used to get more results for those areas of the distributions. When confidence cannot be gained in the chosen distribution for a given parameter that drives the problem, several alternatives can be explored.

One such alternative relies on expert judgment to narrow the input distribution or to qualitatively assess the confidence in the mean values of the input that drive the problem. When such an alternative is chosen, sensitivity studies should be performed to determine whether the input mean selection is driving the solution to the problem, and if so, steps should be taken to increase the confidence in the mean values of key inputs. This typically requires the collection of additional data on those inputs and can be resource intensive.

34

7 Conclusions 7.1 Summary In response to increasing interest in PFM analyses to support safety demonstrations and regulatory relief requests from NRC licensees, this report defined the concept of a PFM analysis and outlined important concepts that should be considered when producing a high-quality, high-confidence PFM analysis. Fracture mechanics analyses have historically been performed deterministically, although the use of probabilistic approaches has increased in response to the need to quantify risk in safety decisionmaking. Deterministic and probabilistic approaches are only as good as the models developed to represent reality; they differ primarily in the way they treat uncertainty. Each approach has its advantages, though policy has resulted in a greater emphasis on cost-benefit analyses and risk-based metrics, which favor an increased use of probabilistic methods.

This report presented a general framework to describe, perform, and evaluate a PFM analysis. The important pieces of a PFM analysis that should be considered with great care include models, inputs, uncertainty characterization, probabilistic framework, and PFM outputs.

• Models can be categorized as different types, but in all cases, model verification, validation, and model uncertainty quantification are key steps in gaining confidence in the adequacy of the models used. Often models used in PFM are simplified representations of the reality because of constraints on solution time. The possible errors introduced by this must be understood. It is also important to verify and validate the system as a whole and not only each individual model, since a complex system is more than a simple assembly of independent models.

• Treatment of uncertain inputs may consist of constructing probability distributions, determining input bounds if applicable, and quantifying any assumptions, conservatisms, or dependencies between inputs, which ensure that the input space is limited to and includes all of the domain of physical realm.

• Uncertainty characterization and treatment are at the core of a PFM analysis. In many PFM analyses, it may be useful to separate epistemic and aleatory uncertainty to be able to assess the confidence levels for probabilities of interest. Uncertainty identification, quantification, and propagation are essential elements in describing a PFM methodology or analysis. The proper choice of sampling techniques (when sampling-based methods are considered) is also an important step that needs justification. The report discussed concepts and methods to verify and validate a probabilistic framework.

• Ways to demonstrate PFM convergence when sampling-based methods are used include varying sample size, use of replicated samples, and sampling strategy, as well as performing stability analysis. Output uncertainty analysis can take various forms depending on the problem being analyzed. Sensitivity analyses can help to identify the drivers of uncertainty for a given problem.

Sensitivity studies are useful to better understand the physics of a problem, to show that some expected trends are indeed reflected in the analysis results, and to perform what if analyses and revisit some assumptions (constant values or distributions changed), as well as to investigate alternative scenarios in support of defense in depth. The report presented methods to perform such studies.

35

7.2 Future Work Following this report, the next step in preparing guidance to perform and evaluate PFM analyses is the development of a more detailed technical basis NUREG document. The NUREG will dive deeper into the concepts described in this report and will more precisely define PFM quality metrics and criteria, or a process to define such metrics and criteria. From this NUREG document, the NRC staff will produce a more succinct draft regulatory guide highlighting the steps required to produce a high-confidence PFM analysis and the ways to evaluate such a PFM. In parallel, the staff will define and execute a pilot study to test the draft guidance produced and to identify deficiencies and areas that need improvement. The staff will incorporate the lessons learned from this process in a final guidance document and revise the technical basis NUREG accordingly.

36

8 References

[1] S. Kaplan and B. J. Garrick, "On The Quantitative Definition of Risk," Risk Analysis, vol. 1, no. 1, pp.

11-27, 1981.

[2] "60 FR 42622 - Use of Probabilistic Risk Assessment Methods in Nuclear Regulatory Activities; Final Policy Statement," Federal Register, vol. 60, no. 158, August 16, 1995.

[3] American Society of Mechanical Engineers, ASME Boiler & Pressure Vessel Code, New York, NY, USA: ASME, 2017.

[4] P. T. Williams, T. L. Dickson, B. R. Bass and H. B. Klasky, "ORNL/LTR-2016/309: Fracture Analysis of Vessels - Oak Ridge - FAVOR v16.1, Computer Code: Theory and Implementation of Algorithms, Methods, and Correlations," Oak Ridge National Laboratory, 2016.

[5] T. L. Dickson, P. T. Williams, B. R. Bass and H. B. Klasky, "ORNL/LTR-2016/310: Fracture Analysis of Vessels - Oak Ridge - FAVOR, v16.1, Computer Code:Users Guide," Oak Ridge National Laboratory, 2016.

[6] D. L. Rudland and C. Harrington, "NUREG 2110: xLPR Pilot Study Report," U.S. Nuclear Regulatory Commission, Washington, DC, 2012.

[7] Electric Power Research Institute, "Materials Reliability Program: Probabilistic Fracture Mechanics Analysis of PWR Reactor Pressure Vessel Top Head Nozzle Cracking (MRP-105), Report 1007834,"

EPRI, Palo Alto, CA, USA, 2004.

[8] Electric Power Research Institute, "Materials Reliability Program: Alloy 82/182 Pipe Butt Weld Safety Assessment for U.S. PWR Plant Designs (MRP-113), Report 1009549," EPRI, Palo Alto, CA, USA, 2006.

[9] Electric Power Research Institute, "Materials Reliability Program: Probabilistic Risk Assessment of Alloy 82/182 Piping Butt Welds (MRP-116), Report 1009806," EPRI, Palo Alto, CA, USA, 2004.

[10] Electric Power Research Institute, "Materials Reliability Program: Inspection and Evaluation Guidelines for Reactor Vessel Bottom-Mounted Nozzles in U.S. PWR Plants (MRP-206), Report 1016594," EPRI, Palo Alto, CA, USA, 2009.

[11] Electric Power Research Institute, "Materials Reliability Program: Topical Report for Primary Water Stress Corrosion Cracking Mitigation by Surface Stress Improvement (MRP-335 Revision 3), Report 3002007392," EPRI, Palo Alto, CA, USA, 2016.

[12] Electric Power Research Institute, "Materials Reliability Program: Reevaluation of Technical Basis for Inspection of Alloy 600 PWR Reactor Vessel Top Head Nozzles (MRP-395), Report 3002003099," EPRI, Palo Alto, CA, USA, 2014.

[13] EPRI, "BWR Reactor Pressure Vessel Shell Weld Inspection Requirements," EPRI-TR-105697, BWRVIP-05, September 1995.

[14] Westinghouse Electric Company, "Risk-Informed Extension of the Reactor Vessel In-Service Inspection Interval," WCAP-16168-NP, Revison 2, Pittsburgh, PA, 2006.

[15] G. R. Irwin, "Linear fracture mechanics, fracture transition, and fracture control," Engineering Fracture Mechanics, vol. 1, no. 2, pp. 241-257, 1968.

[16] ASTM E399-12e3, Standard Test Method for Linear-Elastic Plane-Strain Fracture Toughness KIc of Metallic Materials, West Conshohocken, PA: ASTM International, www.astm.org, 2012.

[17] ASTM E1820-15ae1, Standard Test Method for Measurement of Fracture Toughness, West Conshohocken, PA: ASTM International, www.astm.org, 2015.

37

[18] ASTM E1921-16, Standard Test Method for Determination of Reference Temperature, To, for Ferritic Steels in the Transition Range, West Conshohocken, PA: ASTM International, www.astm.org, 2016.

[19] U.S. Congress, Government Performance and Results Act of 1993, Washington, DC: 103rd Congress, Congressional Record, 1993.

[20] U.S. Nuclear Regulatory Commission, "Chapter I - Nuclear Rgulatory Commission, Part 50 -

Domestic Licencing of Production and Utilization Facilities," in Code of Federal Regulations, Title 10

- Energy, Washington, DC, U.S. Government Publishing Office, 2015.

[21] G. Wilkowski, R. Tregoning, P. Scott and D. Rudland, "Status of Efforts to Evaluate LOCA Frequency Estimates Using Combined PRA and PFM Approaches," in Proceedings of 28th MPA Seminar, Stuttgart, Germany, October 2002.

[22] T. L. Anderson, Fracture Mechanics: Fundamentals and Applications, Third Edition, CRC Press, 2005.

[23] API 579-1/ASME FFS-1: Fitness-for-Service, Washington, DC: ASME, 2007.

[24] S. X. Xu, D. A. Scarth and R. C. Cipolla, "Technical Basis for Explicit Inclusion of Weight Function Method for Calculation of Stress Intensity Factor for Surface Flaws in ASME Section XI Appendix A,"

in Proceedings of the 2011 ASME Pressure Vessels and Piping Conference, Baltimore, MD, USA, July 17-21, 2011.

[25] W. L. Oberkampf, T. G. Trucano and C. Hirsch, "Verification, validation, and predictive capability in computational engineering and physics," Applied Mechanics Reviews, vol. 57, no. 5, pp. 345-384, 2004.

[26] T. G. Trucano, L. P. Swiler, T. Igusa, W. L. Oberkampf and M. Pilch, "Calibration, validation, and sensitivity analysis: What's what," Reliability Engineering & System Safety, vol. 91, no. 10, pp.

1331-1357, 2006.

[27] Z. A. Karian and E. J. Dudewicz, Fitting Statistical Distributions: The Generalized Lambda Distribution and Generalized Bootstrap Methods, CRC Press, 2000.

[28] M. Kirk, C. Sallaberry, M. Erickson and R. Kurth, "Draft report "A User's guide to the xLPR Uncertainty Characterization Spreadsheet"," U.S. Nuclear Regulatory Commission, Washington, DC, USA, 2012.

[29] S. Ferson, J. O'Rawe and M. Balch, "Computing with confidence: imprecise posteriors and predictive distributions," in International Conference on Vulnerability and Risk Analysis and Management and International Symposium on Uncertainty Modeling and Analyis, Liverpool, UK, 2014.

[30] [Online]. Available: https://en.wikipedia.org/wiki/List_of_probability_distributions.

[31] A. Kolmogorov, "Sulla determinazione empirica di una legge di distribuzione," Giornale dell Istituto Italiano degli Attuari, vol. 4, pp. 83-91, 1933.

[32] N. Smirnov, "Table for estimating the goodness of fit of empirical distributions," Annals of Mathematical Statistics, vol. 19, pp. 279-281, 1948.

[33] T. W. Anderson and D. A. Darling, "A Test of Goodness-of-Fit," Journal of the American Statistical Association, vol. 49, pp. 765-769, 1954.

[34] S. S. Shapiro and M. B. Wilk, "An analysis of variance test for normality (complete samples),"

Biometrika, vol. 52, no. 3-4, pp. 591-611, 1965.

38

[35] J. P. Kotra, M. P. Lee, N. A. Eisenberg and A. R. DeWispelare, "NUREG-1563: Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program," U.S.

Nuclear Regulatory Commission, Washington, DC, 1996.

[36] M. Ionescu-Bujor and D. G. Cacuci, "A Comparative Review of Sensitivity and Uncertainty Analysis of Large-Scale Sytems: Part I and II," Nucliear Science and Engineering, vol. 147, no. 3, pp. 189-217, 2004.

[37] A. R. Daneshkhah, "Uncertainty in Probabilistic Risk Assessment: A Review,"

http://www.shef.ac.uk/content/1/c6/03/09/33/risk.pdf, August 2004.

[38] A. Saltelli, K. Chan and E. M. Scott, Sensitivity Analysis, Wiley series in probability and statistics, 2000.

[39] C. B. Storlie and J. C. Helton, "Multiple predictor smoothing methods for sensitivity analysis:

Description of Techniques," Reliability Engineering and System Safety, vol. 93, no. 1, pp. 28-54, 2008.

[40] B. Fiessler B, H. J. Neumann and R. Rackwitz, "Quadratic limit states in structural reliability,"

Journal of Engineering Mechanics, ASCE, vol. 105, no. 4, pp. 661-676, 1979.

[41] N. Metropolis and S. Ulam, "The Monte Carlo Method," Journal of the American Statistical Association, vol. 44, p. 335, 1949.

[42] R. Iman, J. Helton and J. Campbell, "An approach to sensitivity analysis of computer models, Part 1.

Introduction, input variable selection and preliminary variable assessment," Journal of Quality Technology, vol. 13, no. 3, pp. 174-183, 1981.

[43] R. E. Kurth and D. C. Cox, "Discrete Probability Distributions for Probabilistic Fracture Mechanics,"

Risk Analysis, vol. 5, no. 3, pp. 235-240, 1985.

[44] C.-E. Srndal and e. al., "Stratified Sampling," in Model Assisted Survey Sampling, New York, Springer, 2003, pp. 100-109.

[45] ASME NQA-1-2015: Quality Assurance Requirements for Nuclear Facility Applications, ASME International, 2015.

[46] NUREG/BR-0167: Software Quality Assurance Program and Guidelines, Washington, DC: U.S.

Nuclear Regulatory Commission, 1993.

[47] B. Brickstad, T. Schimpfke, O. J. Chapman, P. Dillstrom, C. Cueto-Felgueroso and C. D. Bell, "Project NURBIM (Nuclear RI-ISI Methodology for Passive Components): Benchmarking of Structural Reliability Models and Assocaited Software," in ASME/JSME 2004 Pressure Vessel and Piping Conference, San Diego, 2004.

[48] "Discrete distribution," Encyclopedia of Mathematics, 7 February 2011. [Online]. Available:

http://www.encyclopediaofmath.org/index.php?title=Discrete_distribution&oldid=16276.

[Accessed 24 February 2017].

[49] P. Mattie, R. Gauntt, K. Ross, N. Bixler, D. Osborn, C. Sallaberry, J. Jones and T. Ghosh, "NUREG/CR-7155 / SAND2012-10702P: State-of-the-Art Reactor Consequence Analyses Project: Uncertainty Analysis of the Unmitigated Long-Term Station Blackout of the Peach Bottom Atomic Power Station," U.S. Nuclear Regulatory Commission (ML16133A461), Washington, DC, USA, May 2016.

[50] Sandia National Laboratories, Severe Accident Analysis Department, "Draft Report: State-of-the-Art Reactor Consequence Analysis (SOARCA) Project: Sequoyah Integrated Deterministic and Uncertainty Analyses," U.S. Nuclear Regulatory Commission (ML17156A270), Washington, DC, USA.

39

[51] K. Ross, N. Bixler, S. Weber, C. Sallaberry and J. Jones, "Draft Report: State-of-the-Art Reactor Consequence Analysis Project: Uncertainty Analysis of the Unmitigated Short-Term Station Blackout of the Surry Power Station," U.S. Nuclear Regulatory Commission (ML15224A001),

Washington, DC, USA.

[52] C. J. Sallaberry, R. E. Kurth, F. W. Brust and E. A. Kurth, "Proposed Approach of Scenario Analysis Using a Probabilistic Code," in Proceedings of the ASME 2017 Pressure Vessels and Piping Conference, Waikoloa Village, Hawaii, USA, July 16-20, 2017.

40