ML14072A211

STP Sensitivity Report and Presentation for Proposed Meeting

ML14072A211
Person / Time
Site:	South Texas
Issue date:	03/06/2014
From:	Harrison A W South Texas
To:	Singal B K Division of Operating Reactor Licensing
References
MF2400, MF2401
Download: ML14072A211 (110)

Text

1 NRR-PMDAPEm Resource From: Harrison Albon [awharrison@STPEGS.COM]

Sent: Thursday, March 06, 2014 2:18 PM To: Singal, Balwant

Subject:

STP Sensitivity Report and Pres entation for Proposed Meeting Attachments:

Sensitivity_Analysis_February_10_2014.pdf; Sensitivity Analysis Framework.pdfBalwant, Per our discussion yesterday, here is the background information for a meeting on the STP risk-informed GSI-191 sensitivity analysis. We think that dialog with the staff regarding how the sensitivity was done and the results would be beneficial before we make a formal submittal. We are prepared to meet with the staff at their earliest convenience.

Please call Ernie Kee or me if you have any questions.

Regards, Wayne Harrison

Hearing Identifier: NRR_PMDA Email Number: 1165 Mail Envelope Properties (8C918BCF8596FB49BD20A610FA5920CF01F60589)

Subject:

STP Sensitivity Report and Presentation for Proposed Meeting Sent Date: 3/6/2014 2:17:43 PM Received Date: 3/6/2014 2:19:44 PM From: Harrison Albon Created By: awharrison@STPEGS.COM Recipients: "Singal, Balwant" <Balwant.Singal@nrc.gov>

Tracking Status: None

Post Office: CEXMBX02.CORP.STPEGS.NET Files Size Date & Time MESSAGE 459 3/6/2014 2:19:44 PM Sensitivity_Analysis_February_10_2014.pdf 2146859 Sensitivity Analysis Framework.pdf 511958

Options Priority: Standard Return Notification: No Reply Requested: No Sensitivity: Normal Expiration Date: Recipients Received:

A Practical Guide to Sensitivity Analysis of a Large-scale Computer Simulation Model

+,

& ~,

'22 APracticalGuidetoSensitivityAnalysisofaLarge-scaleComputerSimulationModelDavidMorton,JeremyTejada,andAlexanderZolanTheUniversityofTexasatAustin AbstractWedescribea10-stepsensitivityanalysisprocedurethatappliestoalarge-scalecomputersim-ulationmodel.Weproposeusingtornadodiagramsastheinitialtoolforidentifyingtheinput parameterstowhichthesimulationsoutputsaremostsensitive.Sensitivityplotsandspider plotscomplementtornadodiagramsbycapturingnonlinearresponsesinoutputstochangesin inputs.Regressionmetamodels,andassociatedexperimentaldesign,helpunderstandsensitivi-tiesto,andinteractionsbetween,inputparameters.OurmotivatingmodelfromGSI-191hasa numberofdistinguishingfeatures:(i)Themodelislargeinscaleinthatithasahigh-dimensional vectorofinputs;(ii)Someofthemodelsinputsaregovernedbyprobabilitydistributions;(iii)

Akeyoutputofthemodelistheprobabilityofsystemfailurearareevent;(iv)Themodels outputsrequireestimationbyMonteCarlosampling,includingtheuseofvariancereduction techniquesassociatedwithrare-eventsimulation;(v)Itiscomputationallyexpensivetoob-tainpreciseestimatesofthefailureprobability;(vi)Weseektopropagatekeyuncertainties onmodelinputstoobtaindistributionalcharacteristicsofthemodelsoutputs;and,(vii)The overallmodelinvolvesaloosecouplingbetweenaphysics-basedstochasticsimulationsub-model andalogic-basedPRAsub-modelviamultipleinitiatingeventsinthelattersub-model.We reviewasubsetofamuchlargerliterature,guidedbytheneedtohaveapracticalapproachto sensitivityanalysisforacomputersimulationmodelwiththesecharacteristics.Weillustrate ourproposed10-stepprocedureonasimpleexampleofasimulationmodelforsystemreliabil-ity.Importantthemesrepeatthroughoutourrecommendations,includingtheuseofcommon randomnumberstoreducevariabilityandsmoothoutputanalysis,afocusonassessingencesbetweentwomodelcon"gurations,andpropercharacterizationofbothsamplingerroranduncertaintiesoninputparameters.InAppendixAweassessthesensitivityofcoredamage frequency(CDF)estimatestochangesininputparametersfortheSouthTexasProjectElectric GeneratingStationGSI-191risk-informedresolutionproject.Inparticular,weuseoutputfrom theCASAGrandesimulationmodeltoconstructatornadodiagramtoassesswhichparameters, fromalistofcandidateparameters,CDFappearsmostsensitive,andwefurtherconstructaone-waysensitivityplotforoneofthemostsensitiveparameters.1Background,Purpose,andLexiconParaphrasingKleijnen[9],Sensitivityanalysis,ofacomputersimulationmodel,estimateschangesinthemodelsoutputswithrespecttochangesinthemodelsinputs.Inthisreport,wereviewapproachestosensitivityanalysisofacomputersimulationmodel,fo-cusingonspeci"capproachesthatweseeaspracticallyviableinthecontextofresolvingGSI-191 througharisk-informedapproach.And,weproposea10-stepsensitivityanalysisprocedure.Before proceeding,someremarksonourcharacterizationofsensitivityanalysis,adoptedfromKleijnen, areinorder.

1 1.Wedeliberatelyusetheverb estimatesbecausethetruevalueofourmodelsoutputs(e.g.,probabilityofsystemfailure)cannotbecomputed.Ratherthisoutputmustbeestimated usingMonteCarlosamplinginourimplementationviaacomputersimulationmodel.2.Forthemoment,wearepurposefullyvagueabouthowtomakechangesinthemodelsinputsbecausethisisakeydistinguisheroftapproachestosensitivityanalysis.Weexplore thisissueinsomedetailinthisreport.3.Anotherkeydistinguisherofapproachestosensitivityanalysisisthemannerinwhichwequantitativelyandqualitativelysummarizechangesinourestimatesofthemodelsoutputs.

Wediscusssomeimportantandcomplementaryapproaches.4.Weusethepluralmodelsinputsbecauseofourinterestinbeingabletohandleahigh-dimensionalvectorofinputparameters.5.Weagainusemodelsoutputsrecognizingthatmultipleperformancemeasuresareofsimulta-neousinterest.Forexample,wemaybeinterestedinboththecoredamagefrequency(CDF) andthechangeincoredamagefrequencyrelativetoabasemodelwherethelatter modeldoesnotaccountforfailuremodesassociatedwithGSI-191.Or,wemaybeinterested inthefour-tuple(CDF,LERF,whereLERFdenoteslargeearlyrelease frequency.6.Thereisnoredundancyinthetermcomputersimulationmodel.Physicalsimulationmodelsfrommodelsimplementedonacomputer.Thenotionsoftomodel andtosimulatebothinvolvemimickingareal-worldsystem.However,inourcontextmodelingmeanstheactofabstractingthereal-worldsystemintoasetofmathematicalequationsand/orlogic constitutinganabstractmodel.Whileasimulationmodelisamathematicalmodel,itisusuallytakenasdistinctfromotherclassesofmathematicalmodelsthatyieldananalytical

solution.7.Whileasimulationmodelmaybedeterministicorstochastic,wefocusonstochasticsimula-tionmodels.Theoutputofastochasticsimulationmodelmaybedeterministicorrandom.

ConsidertheprobabilityofsystemfailureinthecontextofGSI-191.Ifweconditionthe inputonarandominitiatingfrequency,theoutputofthemodelisaconditionalprobability ofsystemfailure;i.e.,theoutputisarandomvariable.Ontheotherhand,ifweintegrate withrespecttothedistributiongoverningtherandominitiatingfrequency,theprobabilityof failureisadeterministicoutputparameter.Weconsiderbothalternatives.8.Wecannotcomputeevenadeterministicoutputmeasureexactly.Rather,wemustestimatetheoutputusingMonteCarlosampling.Wehavesampling-basederrorsassociatedwith MonteCarlomethods,buttheseerrorscanbequanti"ed.Theerrorscanalsobereduced byincreasingthesamplesize,andtheycanbereducedbyusingso-calledvariancereduction 2

techniques.Thelattermethodsareparticularlyimportantbecauseofourinterestinrare-eventsimulationinwhichfailureprobabilitiesareverysmall.9.Itisimportanttodistinguishthreesourcesoferror:First,wehavesampling-basederrorsassociatedwithMonteCarlomethodsdiscussedabove.Second,wehaveerrorsduetouncer-taintiesonsomemodelinputs.Third,wehaveerrorsduetoalackof"delityofthemodel itself.Wecanattempttoreducethesecondtypeoferrorbygatheringmoredataoreliciting (more)informationfromexperts.Wecanalsousedataandelicitationtoreducethethird typeoferrorifwehavecompetinghypothesizedmodels,orsub-models.Inallthreecases, sensitivityanalysiscanhelpguidewheresuchshouldfocus.Asecondkeynotionregardingsensitivityanalysisinthecontextofdecisionproblemsinvolvesunderstandingwhichininputsmakeainthedecisionratherthansimplyinmodeloutputs.Inthiscontext,ClemenandReilly[3]characterizesensitivityanalysis bysaying:Sensitivityanalysisanswersthequestion,Whatmakesaenceinthisdecision?...Determiningwhatmattersandwhatdoesnotrequiresincorporatingsensitivityanal-ysisthroughoutthemodelingprocess.Asimilarsentimentisre"ectedinEschenbach[6]:Thissensitivityanalysismaybeused(1)tomakebetterdecisions,(2)todecidewhichdataestimatesshouldbere"nedbeforemakingadecision,or(3)tofocusmanagerial attentiononthemostcriticalelementsduringimplementation.Shouldsomepipesthatcurrentlyhave"berglassinsulationberetro"ttedtoinsteadhavere"ec-tiveinsulationtomitigateriskassociatedwithapossibleGSI-191LOCAevent?Suchanaction wouldincursigni"cantcostandsigni"cantradiationexposuretoworkers.Whatininput parametersofamodelleadtoayesversusnoanswertothisquestion?Asurrogatequestion involvesthenotionofRegionsI,II,andIIIfromRegulatoryGuide1.174[4].Supposenominal valuesofourmodelsinputparametersleadtoanassessmentthattheplantisinRegionIII.Then, wemayask:Whatchangesintheinputparameterswouldleadustoconcludethattheplantwould insteadbeinRegionIorRegionII?Itisnotthepurposeofthisreporttodirectlyanswertheabovepairofquestions.Rather,weprovideaframeworkthat,whenproperlyapplied,cananswerthesequestions.Forconcreteness, weapplyoursensitivityanalysisframeworktoasimpleexampleofaparallel-seriessystemwith fourcomponentsillustratedinFigure1.Here,ouranalogousquestionwillbe,Shouldweperform preventivemaintenanceoncomponent3todecreasetheprobabilitythesystemfailspriortoa pre-speci"edtime, t 0?WedescribethisillustrativeexampleinmoredetailinSection2.

3 1 2 4 3 1/2 These are critical components. If either fails, the system fails.

Component that may have preventive maintenance Figure1:The"guredepictsaseries-parallelsysteminwhichthe"rsttwocomponents(1and2)areinserieswithasubsystemthathastwocomponents(3and4)inparallel.Oneofthetwoparallel componentsmustbeupforthesystemtobeup,alongwithbothcomponents1and2.2IllustrativeExampleFigure1depictsasimplemodelofsystemreliabilityweusetoillustratesensitivityanalysisinthisreport.Ouremphasishereisonhavingaconcretemodelthatisrichenoughtoservethis purposeasopposedtohavingahigh"delitymodelofanactualsystem.Theblockdiagramin the"guredepictsfourcomponentswithindependentrandomfailuretimes T 1 , T 2 , T 3,and T 4.Ifacomponentfails,itwillnotberepaired.Weseektounderstandthefailuretimeofthesystem,given

by T=min{T 1 ,T 2 , max{T 3 ,T 4.While Tisarandomvariable,weusetwodeterministicoutput measures P{T>t 0}and E[T],wheretheformeroutputisourprimaryperformancemeasureofsystemreliability;i.e.,theprobabilitythesystemfailsafterapre-speci"edtime, t 0.Asecondaryoutputistheexpectedtimeuntilthesystemfails.Wehaveorientedthemeasuressothatweprefer largervalues.Theparametersofthefourrandomvariables, T 1 ,...,T 4,areinputstoourmodelofsystemreliability.Weassumethefourrandomvariableshaveexponentialdistributions,andsowehave asmodelinputsthefailureratesofeachofthecomponents,12 ,3,and4(whichhaveunitsoffailuresperunittime),alongwiththetimethresholdforwhichwedesirethesystemtosurvive, whichwedenote t 0.Usually,wesuppressthedependencyof T 1 ,...,T 4ontheirratesbutsometimeswewriteexpressionssuchas T 3 (3)toemphasizetherateassociatedwith T 3.Theinverseofthefailurerateisthemeantimeuntilacomponentfails,andoftenitismorenaturaltothinkintermsof thesemeans:1 11 2 ,1 3,and1 4,whichhaveunitsoftime.Inwhatfollowsweinterchangeablyspeakoffailureratesormeantimestofailure,dependingthecontext.Inourexample,wehaveadecisiontomakeforthissystem.WecanoperatethesystemasdepictedinFigure1withfailurerates14.Wecallthisthebaseoption.Or,attime0wecantakecomponent3andperformpreventivemaintenanceonthatcomponent. Component3wouldthencomebackon-lineattime tt.Importantly,thesystemoperatesevenwhencomponent3isoforpreventivemaintenance.Sothesystemwouldoperateasthree 4 componentsinserieswithfailurerates12,and4duringthetimeinterval(0 ,t)andasthesystemdepictedinFigure1withfailurerates12 ,3/k,and4,ontimeintervalt,t 0).Here,performingpreventivemaintenanceoncomponent3attime0reducesitsfailureratefrom3 to3/k for k1.ForbrevitywecallthislatteroptionthePMoption,althoughitarguablyamountstoanupgradeofcomponent3giventhememorylesspropertyoftheexponentialrandomvariable. ForthePMoption,modelinputsinclude12 ,3 ,4 , t 0t,and k.Systemreliabilityunderthebaseoptionisgivenby: P{min{T 1 ,T 2 , max{T 3 (3),T 4}}>t 0}.(1)SystemreliabilityunderthePMoptionis: P{min{T 1 ,T 2 ,T 4}>t}*P{min{T 1 ,T 2 , max{T 3 (3/k),T 4}}>t 0t}.(2)Weareusingthememorylesspropertyofexponentialrandomvariablesinequation(2),bywritingtheproductandbywritingthereliabilityofthesystemovertimeintervalt,t 0]asthesecondterm.Also,theratesassociatedwith T 1 ,T 2,and T 4donotchangeunderthebaseandPMoptionsandhencewedonotmakethemexplicitinequations(1)and(2).Thatsaid,weinvestigatebelow changesintheseratesinthecourseofoursensitivityanalysis.Wemaytreatinputparameters,suchas1,asdeterministicbutvarytheparameterforthepurposeofunderstandingthesensitivityofsystemreliabilitytochangesin1.Or,wemaytreat1asarandomvariablegoverned,e.g.,byagammadistributionorbyaboundedJohnsondistribution.Ineithercase,wemaycompute,orestimate,theconditional output P{T>t 0l1}underthebaseoption,where T=min{T 1 ,T 2 , max{T 3 ,T 4}}.Inthelattercase,because P{T>t 0l1}isarandomvariable,wemaycompute,orestimate,thepercentiles(e.g.,the5th,50th,and95thpercentiles)of

P{T>t 0l1}knowingthecorrespondingpercentilesof1.Alternatively,wemayintegratewithrespectto1sdistributionandobtain P{T>t 0}.Finally,if1isgoverned,e.g.,byaboundedJohnsondistribution,wecouldseektounderstandthesensitivityof P{T>t 0}totheparametersoftheJohnsondistribution.InthecontextofGSI-191,exampleinputparametersincludemarginsgoverningvariousfailuremodessuchasthenetpositivesuctionheadmarginforpumps,thestructuralmarginforpump strainers,airintrusionlimitsforpumps,in-vessellimitsondebrispenetration,andsolubilitylimits onboronconcentrationinthecore.Otherkeyinputsincludethetemperature,pH,andwater volumeofthepool,parametersgoverningdebrisgenerationanddebristransport,parametersgov-erningstrainercharacteristics,andsoon.Someparameterssuchaspooltemperaturechangeover timeaccordingtoaspeci"edinputpro"le.Inastochasticsimulationmodel,randomvariablesplayakeyroleandtheirinputscanbecharacterizedinoneoftwokeyways,andwetaketheinitiatingLOCAfrequenciesasanexample. Wemodelaprobabilitydistributionasgoverningthefrequencyofbreaksofvarioussizes.We caneithertakeasinput:(i)theparametersofthatprobabilitydistribution,whichinthecaseof STPsGSI-191analysisaretheparametersoftheJohnsondistributionsgoverningthebreaksizes 5 forthesixNUREG-1829break-sizecategoriesor(ii)wecantakeasinputapercentile(e.g.,themedian)associatedwiththatdistribution.Thischoicehowwecharacterizemodeloutput. Similarchoicescanbemadeforrandomvariablesgoverning,e.g.,thestrainersafetymargins.The discussionaboveregardingthetreatmentof P{T>t 0l1}isouranalogfortheillustrativeexample.IntermsofmodeloutputsforGSI-191,wemaybeinterestedinboththecoredamagefrequency(CDF)andthechangeincoredamagefrequencyrelativetoabase-casemodelOr,we maybeinterestedinthefour-tuple(CDF,LERF,Adetailedphysics-based simulationmodel,suchasCASAGrande,canhelpcharacterizetheriskassociatedwithaspeci"c failuremode.However,properassessmentofoverallriskrequirespropagatingsuchfailuresthrough acoupledPRAmodel,andhenceproperassessmentofsensitivitiestochangesinunderlyinginput parametersrequiresasimilarpropagation.IntheremainderofthisreportwedonotdiscussaGSI-191exampleindetail,eventhoughwearemotivatedbyrisk-informedGSI-191analyses.Rather,werestrictattentiontotheexample discussedinthissectiontoillustrateideas.Thisstreamlinesthediscussionandallowsustoprovide simpleandtransparentinsightsontherelativemeritsofvariousapproachestosensitivityanalysis.3APracticalStep-by-StepGuidetoSensitivityAnalysisStep1:De"netheModelWelet f: R nR mdenoteouridealizedmodelofthesystem.Here,ournotationmeansthatthemodeltakesasinputthevaluesof nparametersandgivesasoutput mperformancemeasures.Thevectorofinputsisdenoted x=(x 1 ,x 2 ,...,x n)andthevectorofoutputsisdenoted y=f (x 1 ,x 2 ,...,x n),where y=(y 1 ,y 2 ,...,y m).Wecallthemodelidealizedbecauseweassume,forthemoment,thattheoutputsareknownexactlygiventhevaluesoftheestimates;i.e.,forthe momentweassumewedonotneedtoperformaMonteCarlosimulationinordertoestimatethe valuesoftheoutputs.OurillustrativeexamplehastwomodelsrootedinthebaseandPMoptions.Thebase-optionmodelhasthefollowingprimitives:Fourindependentexponentialrandomvariables, T 1 ,...,T 4 ,governthefailuretimesoffourcomponentswithrespectivefailurerates14,andthefailuretimeofthesystemisgivenby T=min{T 1 ,T 2 , max{T 3 ,T 4}}.Withtheseconstructsand m=2outputswehavethemodel fde"nedby f (14 ,t 0)=(P{T>t 0}, E[T]),wheretheequationsfor P{T>t 0}and E[T]couldbefurtherdetailedusingfour-dimensionalintegrals.(Wedonotdosohereasitdoesnotfurtherourdiscussionofsensitivityanalysis.)An analogousidealizedmodelcanbewrittenunderthePMoptionusingequation(2). 6 Step2:SelectOutputsofInterestOurmodel f has moutputs(y 1 ,...,y m).Instep2oftheproposedprocess,wecanrestrictattentiontoasubsetoftheseoutputs.Thereareanumberofpossibilitiesforourillustrativeexample.We mayhave m=1with(y 1)=(P{Tt 0})or(y 1)=(E[T])asthesingleoutputofinterest.Or,wehavetwooutputsofinterest: m=2and(y 1 ,y 2)=(P{Tt 0}, E[T]).Wemayhave m=3outputs:(y 1 ,y 2 ,y 3)=(P{T>t 0}, E[T], P{T>t 0l1}),andthiscanbeextendedtoincludeadditionaloutputssuchas P{T>t 0li}forall i=1 , 2 ,..., 4.Thenotionof attributionistiedtoouroutputsofinterest.Considerthebaseoptioninourexample.Giventhatoursystemfailedpriortotime t 0,wecanassesswhetherthisisduetoafailureofcomponent1,component2,orduetothefailureoftheparallelsubsystemofcomponents3-4. Thuswecancompute P{T 1=TlT<t 0}, P{T 2=TlT<t 0},and P{max{T 3 ,T 4}=TlT<t 0}.Whenfocusingonattribution,wecouldmodel m=3outputparameters: (y 1 ,y 2 ,y 3)=(P{T 1=TlT<t 0}, P{T 2=TlT<t 0}, P{max{T 3 ,T 4}=TlT<t 0}).Ofcourse,wecouldfurtherassesswhethercomponent3or4causedthefailureratherthantakingtheirpairedsubsystemvia P{T 3=TlT<t 0}and P{T 4=TlT<t 0}.Step3:SelectInputsofInterestOurmodel f has ninputs(x 1 ,x 2 ,...,x n).Instep2oftheprocess,wehavealreadyrestrictedattentiontoasubsetofthemodeloutputs.Itmayseemcounterintuitivetochoosetheoutputs beforechoosingtheinputs,butthisorderispurposeful.Ourchoiceofinputshingesbothonwhat theanalystseesasimportantandontheoutputsofinterestthattheanalystselectedinstep2.The notionofimportanthereisdrivenbymultipleconsiderations.Inourexample,theanalystmay believeaninputparametermaynotchangewhetherthebaseoptionversusPMoptionchoice leadstohighersystemreliabilityuntiltheparameterchangestosomerelativelyextremevalue,and theanalystmayseektounderstandthemagnitudeofthatextreme.Or,theanalystmaybelievean outputdependscruciallyonaninputparameter,andtheanalystseekstounderstandthedirection andmagnitudeofchangeintheoutputwithrespecttochangesintheinput.Forourexamplesbaseoption,ifwehaveselected m=1with(y 1)=(E[T])thenwemaychooseastheinputvector(x 1 ,...,x 4)=(14)anddropthetimethreshold t 0becausethisisnotrelevantwhenestimating E[T].Iftheanalystbelievesthatcomponents1and2areidenticalandcomponents3and4areidenticalthenitmaytohavethesmallerdimensionalinputvector

(x 1 ,x 2)=(13),becausechangesin1=2applytobothcomponents1and2andchangesin3=4applytobothcomponents3and4. If P{T>t 0}isoneofouroutputsofinterestthenwemayseektounderstandthesensitivityofthefailureprobabilitytochoicesof t 0,andhenceinclude t 0asaninputparameter.However,even if P{T>t 0}isoneofouroutputsofinterestwemaynotseektounderstanditssensitivitywithrespectto t 0,ifchangesin t 0arehighlyunlikelyorthevalueof t 0is"xedbymandate. 7 Step4:ChooseNominalValuesandRangesforInputsInthepreviousstep,wehaveselected ninputparametersofinterest,(x 1 ,x 2 ,...,x n).Instep4,weselectnominalvaluesfortheseparametersandlowerandupperboundsforeachoftheseinput parameters.Wedenotethenominalvaluesby(x 0 1 ,x 0 2 ,...,x 0 n),thelowerboundsby(x 0 1 ,x 0 2 ,...,x 0 n),andtheupperboundsby(- x 0 1 ,-x 0 2 ,...,-x 0 n).Thenominalvalueforaninputparameteristypicallybasedontheanalystsbestpointestimateforthatinput.Thatsaid,therearesometimesreasonsforselectinganappropriatelyconservativenominalvalue.Considerourillustrativeexample.Thethresholdtime, t 0,maydenotethelifetimeforwhichwerequirethesystemtosurvive,butwemaynotknowthevalueof t 0withcertainty.Wecouldselectaconservative(i.e.,largebutreasonable)valueof t 0,andif P{T>t 0}istlycloseto1,wemaybesatis"edthatoursystemisoftlyhighreliability.Sensitivityanalysis exploresthisnotioninarichermanner,seekingnotjusttounderstandthefailureprobabilityat asingle,perhapsconservative,valueof t 0,butrathertounderstandthefailureprobabilityoverarangeofvaluesof t 0.Table1givestheinputparametersassociatedwithoursystemreliabilityexample.Lowerandupperboundsarespeci"edbywhattheanalystseesashowloworhightheseparametersmightbe, inanabsolutesense.Moretypically,rangesarespeci"edsothattheintervalcontainsvaluesthat arebothreasonableandlikely(e.g.,wemightexcludevaluesthathavelessthana10%chanceof occurring).AllsevenparametersinTable1haveabsolutelowerboundsof0andupperboundsof .However,wehavenointentionofexploringthisentireinterval.EvenifPMmightconceivablydegradeacomponent,wewillnotexplore k<1.Similarlywewillnotexplore t>t 0becauseundersuchalargevalueforthePMtime,itisclearlynotworthwhiletopursuePM.Itisimportanttochooserangesfortheinputparametersthattheanalystseesasreasonableandcommensurate.Thistaskcanbeandwedonotmeantominimizethaty. Thatsaid,suchchoicesarecontinuallymadeduringtheprocessofmodelingasystem,andwesee thisyasimplicitintheintimateconnectionbetweenmodelingandsensitivityanalysis.Step5:EstimatingModelOutputsunderNominalValuesofInputParametersSofarwehavereferredtotheidealizedmodel, f (x).So,with m=1modeloutput, P{T>t 0},wecandiscussthevalueofsystemreliabilityunderthenominalvaluesoftheinputparameters, x=x 0 ,giveninTable1.However,forthelarge-scalestochasticmodelsinwhichwehaveinterest,wecannot

compute f (x 0)exactly.Rather,wemustestimate f (x 0)usingMonteCarlosampling.Formallythismeansthatwehaveanothermodel,whichwedenote, f N (x 0),wherethismodelisparameterizedbyasamplesize, N.Wecancompute f N (x 0),butbecauseitsinputsinvolvesamplingrandomvariablessuchasthefailuretimesofthefourcomponentsinFigure1themodeloutput, f N (x 0),isalsoarandomvariable.Therandomsamplingerrorassociatedwith f N (x 0)canbequanti"ed,providedthesamplesize, N,istlylarge,using f N (x 0)sstandarddeviationviathecentrallimittheorem. 8 Table1:Nominalvalues,lowerbounds,andupperboundsfortheinputparametersinoursystemreliabilityexample:1 11 4denotemeantimeuntilfailureforthefourcomponentsdepictedinFigure1; t 0speci"esthedesiredlifetimeofthesystem; tisthetimerequiredtoperformPMoncomponent3;and,ifPMisperformedcomponent3sfailureratedropsto3/k where k1.Input NominalLowerUpperparameter(x)value(x 0)bound(x)bound(-x)1 1(months)200 150 2501 2(months)200 150 2501 3(months)50 25 751 4(months)50 25 75 t 0(months)18 12 24t(months)1 0.5 3 k (unitless) 2 1 5UsingthenominalvaluesfortheinputparametersfromTable1,estimatesofthesystemreli-ability,expectedtimetofailure,andfailureattributionprobabilitiesarereportedinTable2.The tablecontainspointestimates,andestimatesofsamplingerrorintheformof95%con"dencein-tervalhalfwidths.Forexample,thepointestimateofsystemreliabilityis0 .7520underthebaseoptionand0 .7850underthePMoption,asreportedinthe P{T>t 0}rowofTable2.ThissuggeststhatthePMoptionleadstohighersystemreliability,however,wecannotignoresampling errorincomingtothisconclusion.WealsoseefromTable2thatthemeanlifetimeofthesystem appearstobelongerunderthePMoption.And,wegetasenseofhowtheattributionprobabilities changeunderthebaseandPMoptions,withtheprobabilitythattheparallelsubsystem3-4is thecauseofsystemfailuredroppingunderthePMoption.Thethreeprobabilitiesinrows3-5of Table2sumtoonebecausetheyareconditionalonthesystemfailing.UnderthePMoptionthe attributiontoparallelsubsystem3-4drops,andhencethelikelihoodofthefailurebeingattributed tocomponents1and2necessarilygrows.BasedonTable2,weare95%con"dentthatthetruevalueforsystemreliabilityunderthebaseoptionliesintheinterval(0 .7252 , 0.7788),andwearesimilarlycon"dentthatthetruevalueforsystemreliabilityunderthePMoptionliesintheinterval(0 .7595 , 0.8105).Wemaybetemptedtousethetwocon"denceintervals(0 .7252 , 0.7788)and(0 .7595 , 0.8105)toinferthattheisnotstatisticallysigni"cant(becausethecon"denceintervalsoverlap),butthisisnottheproper waytoanalyzethisWedescribetheapproachwerecommendshortly. 9 Table2:Estimatesofoutputperformancemeasuresforthebase-optionmodelandthePM-optionmodelunderthenominalvaluesoftheinputparametersfromTable1.Forexample,thepoint estimatefor P{T>t 0}underthebaseoptionis0 .7520anda95%con"denceintervalhalfwidthis 0.0268.Allestimatesinthetablearebasedonasamplesizeof N=1000.Output Base-option PM-option MeasureModelModel P{T>t 0}0.7520+/-0.0268 0.7850+/-0.0255 E[T]44.0174+/-2.3008 54.9770+/-2.9976 P{T 1=TlT<t 0}0.2150+/-0.0255 0.2520+/-0.0270 P{T 2=TlT<t 0}0.2350+/-0.0263 0.2940+/-0.0283 P{max{T 3 ,T 4}=TlT<t 0}0.5500+/-0.0309 0.4540+/-0.0309Itisoftensigni"cantlyeasiertoestimatetheofanoutputmeasureundertwosystemcon"gurationsthanitistoestimatetheabsolutevaluesofthatsameoutputmeasure.When estimatingwecantakeadvantageofthesimulationtechniquecalledcommonrandomnumbersinwhichsimilarcomponentsinthetwosystemsseesimilarinputs.Weillustratethisby estimatingP{T>t 0P{T PM>t 0}P{T base>t 0}usingbothcommonrandomnumbersandindependentrandomnumberswiththesamesamplesize N=1000,andwepresenttheresultsinTable3.Thetablerightlysuggeststhatwecanreducethevarianceofestimatesofsuchbyusingcommonrandomnumbers.Table3:Estimatesoftheinoutputperformancemeasuresbetweenthebase-optionmodelandthePM-optionmodelunderthenominalvaluesoftheinputparametersfromTa-ble1usingbothcommonandindependentrandomnumbers.Forexample,thepointestimate for P{T>t 0}underthebaseoptionis0 .0330anda95%con"denceintervalhalfwidthis0 .0149,butthehalfwidthforthesameestimateusingindependentrandomnumbersislargerbyafactor of2.5at0.0365.in CommonIndependentOutputMeasuresRandomNumbersRandomNumbersP{T>t 0}0.0330+/-0.0149 0.0330+/-0.0365E[T]10.9597+/-1.3985 13.1036+/-3.8276P{T 1=TlT<t 0}0.0370+/-0.0124 0.0590+/-0.0375P{T 2=TlT<t 0}0.0590+/-0.0159 0.0640+/-0.0381P{max{T 3 ,T 4}=TlT<t 0}0.0960+/-0.01970.1230+/-0.0424 10 Ourpointestimateof P{T>t 0}is0.0330andthesamplingerroris0 .0149whenusingcommonrandomnumbers.ThepointestimateindicatesthatthePMoptionappearstohave higherreliabilitythanthebaseoption,andthefactthat0isnotincludedintherange0 .0330+/-0.0149=(0.0181 , 0.0479)indicatesthatthisisstatisticallysigni"cantatacon"dencelevelof95%.Notethatourpointestimateof P{T>t 0}inTable3isidenticaltotheofthepointestimatesinTable2.However,thekeyisthatthesamplingerrorof0 .0149undercommonrandomnumbersissigni"cantlysmallerthanthecorrespondingsamplingerrorsreported inTable3forindependentrandomnumbers,andsigni"cantlysmallerthanthesamplingerror forthecorrespondingabsoluteperformancemeasuresinTable2.WithouthypothesizingaprioriwhetherthebaseorPMoptionleadstohigherreliability,thequestionofstatisticalsigni"cance hingesonwhetherthe95%con"denceintervalfor P{T>t 0}includes0.Ifitdoesnot,theresultisstatisticallysigni"cantwiththesignofthepointestimateof P{T>t 0}determiningwhetherthebaseorPMoptionleadstoamorereliablesystem.Inthiscase,apositiveindicates thePMoptionispreferred.Step6:One-WaySensitivityAnalysis:SensitivityPlotsandTornadoDiagramsFromsteps1-4,wehavespeci"edamodel,restrictedattentiontokeymodeloutputsandinputs,andspeci"ednominalvaluesandrangesforthemodelinputs.Fromstep5,wehavepointestimates, andestimatesofsamplingerror,associatedwiththemodelsoutputsunderthemodelsnominal inputparameters.Sensitivityplotsrestrictattentiontooneortwomodeloutputsatatimeandconsiderasingleinputparameter.Tornadodiagramsrestrictattentiontoonemodeloutputatatimeandconsidermultipleinputs.WefollowClemenandReilly[3]inreferringtosensitivityplotsandtornado diagramsasone-waysensitivityanalysisbecausewevaryoneinputparameteratatime,holding allotherinputsattheirnominalvalues.Figure2isasensitivityplot,showinghowsystemreliability P{T>t 0}changesforthebase-optionmodel(withoutPM)as t 0varies.As t 0growsthesystemreliabilitydrops.The"guredepictspointestimatesalongwith95%con"denceintervalson P{T>t 0}.Panel(a)ofthe"gureshowstheresultswhenusingcommonrandomnumbers,andpanel(b)showsthesameresultswhenusingindependentrandomnumbersforestimatingsystemreliability.Theimportanceofusing commonrandomnumbersisevidentfromthesmoothnessoftheresultsinpanel(a)versusthelack ofsmoothnessinpanel(b).Figure3isasimilarsensitivityplotbutfor P{T>t 0},wherepositivevaluesindicatethatthePMoptionispreferable.Theplotindicatesanotionofdominance.That is,whenotherparametersareheldattheirnominalvalues,systemreliabilityforthePMoption exceedsthatofthebaseoptionforall t 0valuesofinterest.Figure4againshowsasensitivityplotfor P{T>t 0},butnowasafunctionofthereductioninthefailurerate k.Here,weseethatas kgrowsthePMoptionbecomesincreasinglypreferable.Weknowthatat k=1thebaseoptionshouldbepreferred,butfromoursimulationresultsusingasamplesizeof N=1000,wecannotmakethisconclusionwithstatisticalsigni"cance,asthe"gure 11 indicates.(Thiswouldchangeunderalargersamplesize.)Figures5and6concernattribution.Here,wesuppressthecon"denceintervalstoavoidclutter,butTables2and3provideasense oftherespective95%con"denceintervalhalfwidthsundercommonrandomnumbers.Thesetwo "guresquantifyhowtheattributionprobabilityoftheparallelsubsystem3-4dropsas kgrows.Notethatsamplingerroraccountsfortheinattributiontocomponents1and2because thesecomponentsplayidenticalrolesandhaveidenticalfailurerates. 12 5675 5678 5695 5698 56:5 56:8 56;5<=<><?<8<7<9<:<;=5=<===> =?P[T> t 0]t 0 (months)Sensitivity@PlotA@Common@Random@Numbers Mean(a)CommonRandomNumbers 5675 5678 5695 5698 56:5 56:8 56;5<=<><?<8<7<9<:<;=5=<===> =?P[T> t 0]t 0 (months)Sensitivity@PlotA@Independent@Random@Numbers Mean(b)IndependentRandomNumbersFigure2:The y-axisinthe"gureissystemreliability, P{T>t 0},andthe x-axisisthevalueof t 0inmonthsforthebase-optionmodel.The"guredepictsquantitativelyhowsystemreliabilitydropsastherequiredlifetimeofthesystemgrowsfromitslowerboundtoitsupperbound.Thenominal valueof t 0is18months,anditslowerandupperboundsare12and24months,respectively,asindicatedinthe"gure.Inadditiontopointestimatesof P{T>t 0},95%con"denceintervalsateachvalueof t 0aredisplayed.Panel(a)versuspanel(b)ofthe"guredistinguishestheresultswhenusingcommonrandomnumbers(recommended)versusthemorena¨veapproachofusingindependentrandomnumbersattvaluesof t 0.Allestimatesarebasedonasamplesizeof N=1000.13 B565<5655 565<565=565>565?5658 5657 5659 565:<=<><?<8<7<9<:<;=5=<===> =?[T> t 0]t 0 (months)Sensitivity@PlotA@Differences@BReliability@E t o F MeanFigure3:The"gureistobereadinthesamemannerasFigure2exceptthat P{T>t 0}replaces P{T>t 0}onthe y-axis.B565=B565<5655 565<565=565>565?5658 5657 5659 565:<65<68=65=68>65>68?65?68 865P[T> t 0]kSensitivity@PlotA@Differences@BReliability@E k F MeanFigure4:The"gureistobereadinthesamemannerasFigure2exceptthat P{T>t 0}replaces P{T>t 0}onthe y-axis,and k replaces t 0onthe x-axis.14 56=5 56=8 56>5 56>8 56?5 56?8 5685 5688 5675<65<68=65=68>65>68?65?68 865 P[T i= T l T< t 0]kSensitivity@PlotA@PM@Option@BGttribution@E k F PM@Gtt@<PM@Gtt@=PM@Gtt@>H?Figure5:The"gureistobereadinthesamemannerFigure2exceptthatweareplottingtheattributionprobabilitiessuchas P{T 1=TlT<t 0}.Notethatatanyverticallinedrawnthroughthethreeseries,thesumoftheattributionprobabilitiesisone. B56>5 B56=8 B56=5 B56<8 B56<5 B5658 5655 5658 56<5 56<8 56=5<65<68=65=68>65>68?65?68 865[T i= T l T< t 0]kSensitivity@PlotA@Differences@BGttribution@E k F J@Gtt@<J@Gtt@=J@Gtt@>H?Figure6:The"gureistobereadinthesamemannerFigure5exceptthatinattributionprobabilitiessuchas P{T 1=TlT<t 0}replace P{T 1=TlT<t 0}onthe y-axis.Inthiscase,atanyverticallinedrawnthroughthethreeseriesthesumoftheintheattribution probabilitiesiszero. 15 Atornadodiagramcomparestheofcontinuouslydecreasingeachinputparameterfromitsnominalvaluedowntoitslowerboundandincreasingtheparameteruptoitsupperbound,and seeingtheonthemodelsoutput.Oftenoutputmeasureschangemonotonicallywithrespect totheinputs.Forexample,withallotherinputparametersheldconstant,increasingthemean timetofailureofcomponent1,1 1,willincreasesystemreliability(P{T>t 0}),increaseexpectedsystemlifetime(E[T]),anddecreasetheprobabilitythatafailureisattributabletocomponent 1(P{T 1=TlT<t 0}).Decreasesin1 1willhavetheoppositeHowever,inothercasesmonotonicityofoutputisnotensured,andhenceweshouldexercisecarethatweobtaincorrect minimumandmaximumvaluesoftheoutputaswevaryaninputoveritsrange.Onthe x-axisofatornadodiagramweplottheoutputmeasureofinterest,inthiscase,systemreliability, P{T>t 0},expectedtimetosystemfailure, E[T],ortheir P{T>t 0}andE[T].The y-axisstacksbarswiththerangeoftheseoutputsforeachinputparameterofinterest.Theoutputunderthenominalvaluesoftheinputsishighlighted.Thehorizontalbarsfortheinput parametersareorderedbysensitivity,withthelongestbar,i.e.,mostsensitiveinputparameterson top.Notetheimportanceofhavingselectedcommensuraterangesfortheinputvariablesinstep4, asthesenowwhichparametersareseenasmostimportant.Panels(a)and(b)ofFigure7 displaytornadodiagramsforthebase-optionmodelfor P{T>t 0}and E[T].Panels(a)and(b)ofFigure8areanalogousbutforthePMoption.Panels(a)and(b)ofFigure9display P{T>t 0}and E[T],wherepositivevaluesindicatethePMoptionhashigherreliabilityandlongerexpectedsystemlifetime.Panel(a)ofFigure7indicatesthatsystemreliabilityunderthebaseoptionismostsensitivetothevalueof t 0,followedbythemeanfailuretimesofcomponents3and4andthenthemeantimesforcomponents1and2.Systemreliabilityunderthebaseoptionisnotbyinput

parameters kand t.For E[T],panel(b)ofFigure7indicatesthemostsensitiveparametersarethemeanfailuretimesofcomponents3and4followedbythesametimesforcomponents1and 2.Output E[T]isnotbyinputparameters kt,or t 0.TheresultsforthePMoptioninFigure8aresimilarexceptthatweseetheimportanceofparameter k.Interestingly,theresultsarerelativelyinsensitivetothedurationofthePMinterval, t,althoughthischangesslightlyinFigure9whenexaminingtheresultsfor P{T>t 0}.Forthisreason,wedonotshowsensitivityplotswithrespectto there,althoughwerevisitsensitivityto tinstep8(inspiderplots)andstep9(inatwo-waysensitivityanalysiswith k).Samplingerrorsaremoreeasilydisplayedonsensitivityplotsthanontornadodiagrams.Still,inFigures7-9wedisplayhorizontalerrorbarsatthetwoextremes.Sensitivityplotsalsohave theadvantagethatmultipleoutputscanbeplottedsimultaneously.Thatsaid,tornadodiagrams candisplaymanyinputparameterssimultaneouslyandarewidelyusedtoassesstowhichinput parametersanoutputismostsensitive. 16 (a)P{T>t 0}(b)E[T]Figure7:The"guredepictstornadodiagramsfor P{T>t 0}and E[T]forthebase-optionmodel.Pointestimatesalongwith95%con"denceintervalsaredisplayed.The"gureindicatesthatsystem reliabilityismostsensitivetothevalueof t 0,followedbythemeanfailuretimesof1 3 and1 4 andthenbythemeantimes1 1 and1 2.Expectedsystemlifetimeismostsensitiveto1 3 and1 4andthen1 1 and1 2.Notethatthecolorshadingindicateswhetherahighorlowvalueoftheinputparametercorrespondstothechangeintheoutput.Thus,highervaluesof t 0leadtolowervaluesofsystemreliability,buthighercomponentmeanlifetimesleadtohighersystemreliability. Again,estimatesanderrorbarsarecalculatedbasedonasamplesizeof N=1000.17 5677567:5695569=569?5697569:56:556:=56:?56:756:: t5 k<HK><HK=<HK?<HK<JtTornado@DiagramA@PM@Option@BP[T >t 5]LowQigh (a)P{T>t 0}?5?=???7?:858=8?878:757=7?777:959=9?979::5 k<HK><HK?<HK=<HK<Jt t5Tornado@DiagramA@PM@Option@BU[ T]LowQigh (b)E[T]Figure8:The"guredepictstornadodiagramsfor P{T>t 0}and E[T]forthePMoption,andistobereadinthesamemannerasFigure7. 18 B565=B565<5655565<565=565>565?565856575659565: k t5<HK?<HK>Jt<HK<<HK=Tornado@DiagramA@Difference@BJP[T >t 5]LowQigh (a)P{T>t 0}B=5=?7:<5<=<?<7<:=5===?=7=:>5>= k<HK><HK=<HK?<HK<Jt t5Tornado@DiagramA@Difference@BJU[ T]LowQigh (b)E[T]Figure9:The"guredepictstornadodiagramsfor P{T>t 0}and E[T]wherepositivevaluesfavorthePMoption. 19 Step7:UncertaintyQuanti"cationPlotsAnimportantpartofuncertaintyquanti"cation(UQ),andapartthatdistinguishesitfromroutinesensitivityanalysis,concernspropagatingaprobabilitydistributionplacedononeormoreinput parametersthroughthenonlinearfunctionrepresentedbyasimulationmodelandcharacterizing theresultingprobabilitydistributiononanoutputmeasure.Weemphasizethattheprobability distributionwespeakofhereisaprobabilitymodelthatweplaceoninputparametersandnotthe MonteCarlosampling-basederrorwereferenceabove.(Thatsaid,aselsewhere,wealsocapture sampling-basederrorhere,too.)Wecallagraphicalplotoftheresultingprobabilitydistribution

aUQplot,regardlessofwhetheritisexpressedasacumulativedistributionfunction(cdf)oraprobabilitydensityfunction(pdf).Thisideaiscloselyrelatedtothesensitivityplotsweformin step6,exceptthatwenowembedinformationassociatedwiththeprobabilitydistributionplaced ontheinputparameters.Webeginbyfocusingonthecasewhenaunivariatedistributionisplaced onasingleinputparameter,andwethenturntoUQplotswhenmultivariatedistributionsare placedoninputparameters.Whenconstructingasensitivityplot,the y-axisistheoutputparameter,andthe x-axisistheinputparameter.Forasensitivityplotwetypicallyformauniformgridovertherangeoftheinput parametervalues,e.g.,overtheinputrangesthatTable1speci"es.Acdf-basedUQplotisaplot ofthecumulativedistributionfunctionoftheoutputmeasure.Wealsoformpdf-basedUQplots. Inbothcasesweformestimatesofthesefunctionbasedonsampling,wherethesamplingisdone inamannerwemakeprecisebelow.ForacdfUQplot,the x-axiscontainslevelsoftheoutputmeasure,the y-axiscontainsprobabilities,andtheprobabilitydistributionontheinputparameterisimplicitlyencodedintheresult.Weagainuseourexampletomakethisideaconcrete.Supposethattheimprovementfactor, k,hasacontinuousuniformrandomvariableontheinterval(1,5)speci"edinTable1.Figure10containsUQplotsof E[Tlk]and P{T>t 0lk}forthePM-optionmodel.Thetwopanelsofthe"gurecontainestimatesofthecdf-basedUQplotsforboththesetwooutputs.As kgrowstheprobabilitythatasystemfailureisduetoafailureoftheparallelsubsystemofcomponents3-4 drops.Asaresult,weseebothcdfsgrowquicklytowardsoneforlargevaluesof E[Tlk]and P{T>t 0lk}becausethereisalargeprobabilitymassfor kassociatedwithlittleimprovementinthesevalues.Figure11issimilar,exceptthatwenowshowcdfsfor E[Tlk]and P{T>t 0lk}ratherthan E[Tlk]and P{T>t 0lk}.Finally,Figure12showsthepdfsfor E[Tlk]and E[Tlk].Wedonotshowanalogouspdfsfor P{T>t 0lk}and P{T>t 0lk}becausetheestimateshaveexcessivesamplingerror.Developinggoodpdfestimatesfor P{T>t 0lk}and P{T>t 0lk}wouldrequirealargersamplesize. 20 5655 56<5 56=5 56>5 56?5 5685 5675 5695 56:5 56;5<655?5?8858875789598

5 CDF of E[Tlk]E[Tlk]VW@PlotA@PM@Option@BCDX@U[

Tlk]Mean VL LL (a)E[Tlk]cdf 5655 56<5 56=5 56>5 56?5 5685 5675 5695 56:5 56;5<655569<569=569>569?569856975699569:569;56:556:<56:=56:> 56:?CDF of P[T> t 0lk]P[T> t 0lk]VW@PlotA@PM@Option@BP[T > t 5lk]Mean VL LL (b)P{T>t 0lk}cdfFigure10:The"guredepictsUQplotswhichconsistofestimatesofthecdfofthecorrespondingoutputmeasureswhentheimprovementfactor, k,isauniformrandomvariableontheinterval(1,5).Pointestimatesaswellasa95%con"denceenvelopeareplotted.As kgrowstheprobabilitythatasystemfailureisduetotheparallelsubsystemofcomponents3-4shrinks.Asaresult,we seebothcdfsgrowquicklytowardsoneforlargevaluesof E[Tlk]and P{T>t 0lk}.21 5655 56<5 56=5 56>5 56?5 5685 5675 5695 56:5 56;5<655B858<5<8=5=8>5 >8 Tlk]Tlk]VW@PlotA@Difference@BCDX@JU[ Tlk]Mean VL LL(a)E[Tlk]cdf 5655 56<5 56=5 56>5 56?5 5685 5675 5695 56:5 56;5<655B565=B565<5655565<565=565>565?565856575659 565: T> t 0lk]T> t 0lk]VW@PlotA@Difference@BJP[ T > t 5lk]Mean VL LL(b)P{T>t 0lk}cdfFigure11:The"gureistobereadasFigure10exceptthatwenowshowcdfsfor E[Tlk]andP{T>t 0lk}ratherthan E[Tlk]and P{T>t 0lk}.22 565<565=565> 565?5658 5657 5659 565:?=???7?:858=8?878:757=7?777:959= 9?PDF of E[Tlk]E[Tlk]VW@PlotA@PM@Option@BPDX@U[ Tlk]Mean VL LL (a)E[Tlk]pdf 565<565=565>565?5658 5657 5659 565:B858<5<8=5=8 >5 Tlk]Tlk]VW@PlotA@Difference@BPDX@JU[ Tlk]Mean VL LL(b)E[Tlk]pdfFigure12:The"guredepictsUQplotswhichconsistofestimatesofthepdfof E[Tlk]and E[Tlk]whentheimprovementfactor, k,isauniformrandomvariableontheinterval(1,5). 23 Inourexample,whenformingUQplotsof E[Tlk]and P{T>t 0lk}weregardtheothersixinputparameters,14t,and t 0,asdeterministicparameters,andthefailuretimesofthefourcomponents, T 1 ,...,T 4asrandomvariables.Oursamplingconsistsofdrawing N=1000independentandidenticallydistributed(i.i.d.)observationsofthefour-tuple(T 1 ,...,T 4).Inthisone-dimensionalsettingweformthe1%,2%,3%,...,99%percentilesofthedistributionof k ,usingitsdistribution,andwethenuseour N=1000i.i.d.observationsof(T 1 ,...,T 4)toestimate E[Tlk=k],for=0.01 , 0.02 ,..., 0.99,where kdenotesthesepercentiles.Althoughwedescribeconditioningonevenly-spacedquantilesevenlyspacedintermsofprobabilityitmaybedesirable tohavea"nergridinregionswherethefunctionchangesmostrapidly.Again,weemphasizethe importanceofusingcommonrandomnumbersinformingUQplotssuchasthoseinFigures10-12.Obviousalternativestowhatwehavejustsketchedarealsopossible,appropriate,andevennecessary.(Italsoimportanttorecognizewhatisinappropriateandwepointtothat,too,below.) Forexample,wecouldregardtheothersixparametersasrandomvariablesinsteadof"xingthem attheirnominalvaluesandsampletheminthesamewaywesample(T 1 ,...,T 4),whilestillconditioningon k=ktoformestimatesof E[Tlk=k].Clarityinexpositionshouldindicatewhatpreciselytheexpected-valueoperatorisaveragingover.Inanotheralternative,wecouldalsosamplefrom ksdistributioninsteadofconditioningonitsquantilesinordertoformaUQplot.Whensampling k,itisimportanttodistinguishthissamplingfromthatfor(T 1 ,...,T 4).Speci"cally,wecoulduseonesamplesize N uq for k and form k i , i=1 ,...,N uq from ksdistribution.Foreachofthesesampleswethencompute,orrather estimate E[Tlk=k i], i=1 ,...,N uq,whereeachestimateaveragesoverthe Nsampledrealizationsof(T 1 ,...,T 4).Wethenusetheestimatesof E[Tlk=k i], i=1 ,...,N uq,toformthetypesofplotsinFigure10.Wedidnotusethissampling-basedmethodinformingtheUQplotsofFigure10becauseitismoretintheone-dimensionalsettingtoconditiononthequantilesaswedescribe above.However,thissampling-basedapproachisnecessarytoformaUQplotwhenbivariate,or higherdimensionalmultivariate,distributionsareplacedoninputparameters.Forexample,ifwe placeabivariatedistributiononthePMtime-reductionfactorpair,t,k),thensuchabivariatedistributionhasnonotionofquantiles,andsotheone-dimensionalproceduredoesnothavea bivariateanalog.Wemustsample.Importantly,wedonothavetosample(T 1 ,...,T 4)fromitsunderlyingdistribution.Ifwehaveavariancereductionschemethatforms,e.g.,unbiasedestimates

of E[Tlk=k i]thenthatsamplingschemecanbeused.However,itisimportanttorecognizethatwemustsamplefromthetrueunderlyingdistributionoft,k).Ifweuseadistribution-alteringvariancereductionschemetosamplefromt,k)whenaveragingoutthoseparameters,thatschemecannotbeusedwhenformingaUQplot.Suchalteredschemesaredesignedtoreducevarianceand hencewouldyieldmisleadingplots,indicating,e.g.,apdfthatistoonarrowaboutthemeanvalue oftheoutputmeasure. 24 Step8:One-WaySensitivityAnalysis:SpiderPlotsAspiderplotisan x-ygraph,inwhichthe x-axisdepictschangesintheinputparametersandthe y-axiscapturescorrespondingchangesinthemodelsoutputmeasure.Likeatornadodiagram,aspiderplotinvolvesmultipleinputparametersandasingleoutputvariable.Theoutputvariable istypicallyexpressedinitsnaturalunits.Forexample,weexpresschangesin E[T]inmonthsandweexpresschangesin P{T>t 0}asaunitlessvaluebetween0and1,wherethechangesarerelativetoestimatesunderthenominalvalueoftheparameters.Inordertoallowthe x-axistosimultaneouslyrepresentmultipleinputparameters,whichareontscaleswitht units,therearetwopossibilities.Onepossibilityistoexpresspercentagechangesintheinput parametersfromtheirnominalvalues.Thesecondpossibilityistoexpresschangesasmultiples ofthestandarddeviationoftheinputparameters,whenthoseinputparametersaregovernedby probabilitydistributions.Ineithercase,themagnitudewevarytheparametersisdeterminedby thereasonableandcommensuraterangeswehavespeci"edinstep4oftheanalysis,e.g.,inTable1. Intheformercase,ifthenominalvalueoftheinputparameteriszero,thenasecond x-axismustbeadded.Atornadodiagramcanincludealargernumberofinputvariablesthanaspiderplot.Aspiderplotallowsdisplayingaboutseveninputparametersbeforeitbecomescluttered.Ifthe outputvariableismonotonic(increasingordecreasing)inaninputparameterthenweonlyneed toestimatethemodelsoutputatthelowerbound,nominalvalue,andupperboundoftheinput parameter.Aspiderplotrequiresestimatingthemodelsoutputatenoughvaluesofeachinput parameterthataseeminglycontinuousplotof(x,y)pairscanbeformed.Aspiderplotcontainsmoreinformationthanatornadodiagram.Thetornadodiagramsendpointsdenotetheendpoints ofthespiderplot,butthespiderplotalsospeci"eschangesintheoutputatintermediatevalues,as doesasensitivityplot.Againlikeasensitivityplot,wecanassesswhetherchangesintheoutputare linearornonlinearwithrespecttochangesintheinput.Spiderplotscancontainpointestimates or95%,say,con"denceintervalsonthosechanges.(Inthelattercase,wemayneedtoreducethe numberofinputparameterssimultaneouslydisplayed.)Figure13displaystwospiderplotsforourexamplefor E[T]and P{T>t 0}(y-axis)forthebase-optionmodelasafunctionofpercentagechangesintheinputparameters(x-axis).Panel(a)showsaspiderplotfor E[T]whilepanel(b)showsthespiderplotfor P{T>t 0}.Figure14displaysspiderplotsforthePMoptionforourexample,andFigure15displaysspiderplotsfor E[T]andP{T>t 0}.Notethatwechoosenottoincludecon"dencelimitsforeachparameterdisplayedinthegraphbecauseofthecluttertheyinduce.Alsonotethattherangeofthe x-axisisdeterminedbythenominalvaluesandthelowerandupperboundsspeci"edinTable1.Importantly,the inputparametersare noteachvariedthesamepercentage.Rather,thelimitsarethosespeci"edinTable1.Qualitatively,the"guresaresimilartothetornadodiagramsastothemostsensitive inputparameters.However,Figures13-15aremoreinsightfulastotherateofchangeandany associatednonlinearitiesinthechange. 25 B: B7 B?B=5 = ?7B85B?5B>5B=5B<55<5=5>5?585Change@in@U[ T]Y@Change@in@Input@ParametersSpider@PlotA@Zase@Option@BU[ T]<HK<<HK=<HK><HK?(a)E[T]B56<8 B56<5 B5658 5655 5658 56<5B85B?5B>5B=5B<55<5=5>5?585Change@in@P[ T\@t 5]Y@Change@in@Input@ParametersSpider@PlotA@Zase@Option@BP[ T\@t 5]<HK<<HK=<HK><HK?t5 (b)P{T>t 0}Figure13:The"guredepictstwospiderplotsforthebaseoptioninourexample.Theplotinpanel(a)shows E[T](y-axis)asafunctionofpercentagechangesintheinputparameters(x-axis).Theplotinpanel(b)isidenticaltotheoneinpanel(a)exceptthatthe y-axisis P{T>t 0}.26 B<8 B<5 B8 5 8<5 <8 =5B85B=85=88598<55<=8<85Change@in@U[ T]Y@Change@in@Input@ParametersSpider@PlotA@PM@Option@BU[ T]<HK<<HK=<HK><HK?k Jt (a)E[T]B56<5 B565: B5657 B565?B565=5655 565= 565?5657 565:B85B=85=88598<55<=8<85Change@in@P[ T\@t 5]Y@Change@in@Input@ParametersSpider@PlotA@PM@Option@BP[ T\@t 5]<HK<<HK=<HK><HK?k Jt t5 (b)P{T>t 0}Figure14:The"gurereadsasFigure13exceptthatitisforthePMoptioninourexample. 27 B<8 B<5 B8 5 8<5<8=5B85B=85=88598<55<=8<85Change@in@JU[ T]Y@Change@in@Input@ParametersSpider@PlotA@Difference@BJU[ T]<HK<<HK=<HK><HK?k Jt(a)E[T]B565?B565>B565=B565<5655 565< 565= 565>B85B=85=88598<55<=8<85Change@in@JP[ T\@t 5]Y@Change@in@Input@ParametersSpider@PlotA@Difference@BJP[ T\@t 5]<HK<<HK=<HK><HK?k Jt t5(b)P{T>t 0}Figure15:The"gurereadsasFigure13exceptthatitisfortheinperformanceforour example.28 Step9:Two-waySensitivityAnalysisTwo-waysensitivitygraphsallowforvisualizingtheinteractionoftwoormoreinputvariables.Whilesuchanalysiscanbemoretoperform,itcanprovidevaluableinsight.Forour example,Figure16depictstheofsimultaneouschangesinthedurationofPM t)andthefactorbywhichthePMreducesthefailurerateofcomponent3(k)o P{T>t 0}.Panel(a)containsonlythepointestimate,andpanel(b)containsboththepointestimateandthecon"dence limits,allowingustoseethezonewhereneitheroptionisstatisticallybetterthanthe other.Figure17issimilartoFigure16,excepttheparametersofinterestare t 0 and k.Two-waysensitivityanalysescanbeextendedtoincludemorethantwodecisionalternatives,andinsuchcasestheplotstypicallypartitionthespaceintothreeormoreregionsinwhicheach alternativeispreferred.Itisalsopossibletoformathree-dimensionalplotofanoutputvariable (e.g.,theinsystemreliability P{T>t 0})asafunctionoftwoinputvariables(e.g., t and k),althoughwedonotpursuethathere. 29 5<=>?8<=>?8 J t kTwoB^ay@Sensitivity@PlotA@Difference@BJP[ T\@t 5]Mean PM@Option@ Preferred Zase@Option@ Preferred(a)PointEstimate 5<=>?8<=>?8 J t kTwoB^ay@Sensitivity@PlotA@Difference@BJP[ T\@t 5]Mean LL VL PM@Option@ Preferred Zase@Option@ Preferred(b)PointEstimateandCon"denceLimitsFigure16:The"guredepictsatwo-waysensitivityplotfortheinputparametersgoverningthedurationofPM, t,andthefactorbywhichthefailurerateofcomponent3isreduced, k.Whentissmalland kislarge,thePMoptionispreferred,andwhen tislargeand kissmallthebaseoptionispreferred.The"gurequanti"esthisnotionwiththetwo-dimensionalanalogofa thresholdanalysis. 30 <=<?<7<: =5 == =?<=>?8 t 5 kTwoB^ay@Sensitivity@PlotA@Difference@BJP[ T\@t 5]Mean PM@Option@ Preferred Zase@Option@ Preferred(a)PointEstimate <=<?<7<: =5 == =?<=>?8 t 5 kTwoB^ay@Sensitivity@PlotA@Difference@BJP[ T\@t 5]Mean VL LL PM@Option@ Preferred Zase@Option@ Preferred(b)PointEstimateandCon"denceLimitsFigure17:The"guredepictsatwo-waysensitivityplotfortheinputparametersgoverningtheminimumtimethesystemmustbeoperational, t 0,andthefactorbywhichthefailurerateofcomponent3isreduced, k.When t 0issmalland kissmall,thebaseoptionispreferred,andwhen t 0islargeand kislargethePMoptionispreferred. 31 Step10:Metamodels&DesignofExperimentsThepossiblepitfallsofchangingonlyoneinputparameteratatimearewelldocumentedintheliterature(see,e.g.,[8,10])andincludethefactthatinteractionbetweeninputparameters arelost.Graphicalsensitivityanalysesbecomemorewhenmovingpastaone-ortwo-dimensionalanalysis,butwecanformametamodel(whichisalsocalledaresponsesurface,an emulator,orasurrogatemodel)andcarryoutanexperimentaldesignto"tthatmetamodel.Recallfromstep1thatweuse ytodenotethesimulationmodelsoutputandweuse x=(x 1 ,...,x n)todenotethesimulationmodelsinput.Forsimplicitywefocusonasingleoutputmeasure.Amongthesimplestmetamodelstypicallypostulatedarepolynomialregressionmodels oflowdegree;e.g., y=0+nk=1k x k+nk=1 nk=k+1k,kx k x k+(3)To"ttheparameters0 ,1n,and1 , 21 ,nn1 ,n,weuseanexperimentaldesigntospecify M,say,inputparametervectors(x i 1 ,...,x i n), i=1 ,...,M,coupledwiththecorrespondingestimatedsimulationoutput y i.Thecorrespondingerrorterms,i,areassumedtobeindependentandnormallydistributedwithmeanzero,atleastinthesimplestapproach.Sofar,wehaveem-phasizedtheimportanceofusingcommonrandomnumbersincarryingoutouranalyses.However, theassumptionthattheerrortermsiareindependentrequiresthatwedrawindependentMonteCarlosamplesateachdesignpoint.Whenestimatingawestillusecommonrandom numbersforthebaseandPMoptionsatthatdesignpoint.Wecouldaddhigher-ordercrosstermstoequation(3).Thiscanimprovethequalityofthe"t,butweneedtotakecarethatwehaveanadequatenumberofobservationsrelativetothelarger numberofmodelparameterssothemodelisnotover-"t.Perhapsmoreimportantly,athorough understandingofahigher-degreepolynomialregressionmodelcanbechallenging.Table1speci"esseveninputparametersforourexample,butonly"veofthoseinputparametersmatterforthebaseoption.(ThePMparametersof t and karenotrelevantforthebaseoption.)Inthiscase,wecanseektoexplainsystemreliability y=P{T>t 0}asarelativelysimplefunction of1 1 ,1 2 ,1 3 ,1 4,and t 0.Afullfactorialdesignwithtwolevelsinthiscasewouldinvolve 2 5=32combinationsofthesevaluesplacedattheirlowerandupperboundsfromTable1.Whenwealsoincludethenominalcaseforthevaluesoftheparameterswehave3 5=243combinations.IfweinsteadhaveseveninputparametersforthePMoption,thesevaluesbecome2 7=128and 3 7=2187.Notethattheprecedingvaluesrepresentthenumberofdesignpointsinaspeci"cexperimentaldesign.Inordertoestimatehigher-orderandinteractionterms,multiple replicationsaretypicallyrunateachdesignpoint.Soforourexample,ateachdesignpoint,we couldestimate y=P{T>t 0}usingasamplesizeof N=1000,andwecouldreplicatethatpoint estimate N r=3times.Forourexample,wepresenttheresultsofaregressionmetamodel,wheretheperformancemeasureofinterestistheinreliability, P{T>t 0},whereagain,positivevaluesfavor 32 thePMoption.Weusea3 7fullfactorialdesign,wherethethreelevelsaretheminimum,nominal,andmaximumvaluesinTable1.Ateachdesignpoint,weperform N r=3independentreplicationsofthesimulationusingasamplesizeof N=1000foreachreplicate.Table4:Experimentaldesignparametersforthereplicated3 7factorialdesignandoverallmodel"tstatistics.The F-Statisticmeasureswhetheranyoftheinputvariablesinanycombinationwithoneanotherhaveastatisticallysigni"cantontheresponse P{T>t 0}).Theadjusted R 2valueassessestheoverallgoodness-of-"tofthemodeltothesimulationoutput.#ofDesignPoints 2,187#ofReplicationsPerDesignPoint: 3TotalSampleSize: 6,561ResidualStandardError:

0.0 1295ResidualDegreesofFreedom

6,532 Adjusted R 2Value: 0.8895 F-Statistic: 1,886DegreesofFreedom1: 28DegreesofFreedom2: 6,532 p-value:<2.20E-16FromtheinformationinTable4,wecanseethatbasedontheadjusted R 2valueof0.8895,wehaveagood(notexcellent)"ttothesimulationoutput.Anexcellent"tiscategorizedby anadjusted R 2valueofatleast0 .90.Asweindicateabove,ina3 7fullfactorialexperimentaldesign,thereare2,187designpoints;i.e.,2,187uniquecombinationsofinputparameters.We replicatethesedesignpointsthreetimes,foratotalsizeof6,561.Eachofthese6,561estimatesof P{T>t 0}isbasedonasamplesizeof N=1000.The F-Statisticand p-valueareusedtotestthehypothesisthatallofthecotsk andk,k,withtheexceptionoftheintercept0,arezero.Ifthisisthecase,noneoftheinputparameters(i.e.,factorsintheterminologyofregression andexperimentaldesign)orinteractionsamongtheseparametersaresigni"cantinestimatingthe responsevariable, P{T>t 0}.The p-valuemeasuresthestatisticalsigni"canceoftheresult.For the F-Test,weseethatthe p-valueislessthan2 .2x1016,whichismuchlessthanthestandardsigni"cancelevelof0 .05.Soforthis F-Testwecanconcludethatatleastonecotinourregressionmodelissigni"cantlytfromzero,andhenceisasigni"cantfactorinestimating P{T>t 0}.This,coupledwithareasonablygoodadjusted R 2valueof0.8895suggeststhatwehavearelativelygood"ttothesimulationoutputandthe"tparameters,otherthantheintercept, playasigni"cantroleinestimating P{T>t 0}.Thenextstepintheanalysisistoexaminewhichregressioncots,k andk,k,arestatis-ticallysigni"cantinourmodel.Thisisanotionofattributioninthatweareattemptingtoattribute achangeintheresponsevariabletochangesintheinputvariableandcharacterizethestrengthof therelationship.Table5presentstheestimatesforthecotsintheregressionmodel(3)as wellastheirstandarderrors,andtheassociated p-valuesfordeterminingthesigni"cancelevelsof 33 individualparameters.Table5:Resultsof"ttinglinearregressionmodel(3)for P{T>t 0}.Weexcludealltermshigherthansecondordertermsasequation(3)indicates.Usingthe p-valuewecandeterminewhichandtwo-wayinteractionshaveasigni"cantinestimating P{T>t 0}.CotsParametersCotStandardt-valuep-valueSigni"cantEstimatesErrorat=0.050Intercept-4.374E-028.909E-03-4.9109.34E-07Yes11 1-4.420E-063.132E-05-0.1418.88E-01No21 2-4.166E-053.132E-05-1.3301.84E-01No31 32.448E-048.680E-052.8204.82E-03Yes41 44.089E-048.680E-054.7112.52E-06Yes5 k1.480E-029.005E-0416.4342.00E-16Yes6t-2.160E-021.801E-03-11.9932.00E-16Yes7 t 02.959E-032.816E-0410.5072.00E-16Yes1 , 21 1*1 27.682E-099.591E-080.0809.36E-01No1 , 31 1*1 3-3.916E-072.398E-07-1.6331.02E-01No1 , 41 1*1 4-3.837E-072.398E-07-1.6001.10E-01No1 , 51 1*k1.009E-052.398E-064.2062.63E-05Yes1 , 61 1*t-6.564E-064.796E-06-1.3691.71E-01No1 , 71 1*t 01.901E-067.993E-072.3781.74E-02Yes2 , 31 2*1 3-4.407E-072.398E-07-1.8386.61E-02No2 , 41 2*1 4-1.680E-082.398E-07-0.0709.44E-01No2 , 51 2*k1.061E-052.398E-064.4279.73E-06Yes2 , 61 2*t-5.576E-064.796E-06-1.1632.45E-01No2 , 71 2*t 03.070E-067.993E-073.8421.23E-04Yes3 , 41 3*1 41.319E-055.994E-0721.9982.00E-16Yes3 , 51 3*k-2.303E-045.994E-06-38.4162.00E-16Yes3 , 61 3*t-8.428E-051.199E-05-7.0302.28E-12Yes3 , 71 3*t 0-3.768E-051.998E-06-18.8582.00E-16Yes4 , 51 4*k-2.378E-045.994E-06-39.6752.00E-16Yes4 , 61 4*t2.797E-041.199E-0523.3292.00E-16Yes4 , 71 4*t 0-5.333E-051.998E-06-26.6912.00E-16Yes5 , 6 k*t-1.725E-031.199E-04-14.3902.00E-16Yes5 , 7 k*t 01.183E-031.998E-0559.2002.00E-16Yes6 , 7t*t 03.536E-043.996E-058.8472.00E-16YesTable5providesseveralinsights,andcanbeviewedastypicalregressionoutput.Inthe"rsttwocolumnsofthetable,welistallofthecotsinthemodel,theircorresponding inputparameters,andinteractionsamongtheinputparameters.Forexample,0isthecotrepresentingtheintercept,1representsthecotof1 1,and5 , 7representstheinteractionbetween k and t 0.Thethirdcolumnpresentspointestimatesofeachoftheregressioncotsandthefourthcolumnpresentsthestandarderrorassociatedwiththesecots.Theratioof 34 theestimatetothestandarderroristhe t-valueforthe t-test,whichteststhehypothesisthateachindividualparametertakesvaluezerosothatrejectingthenullhypothesisindicatestheparameter issigni"cant.The p-valuesforthe t-testsforeachoftheseterms(column6)canbeinterpretedinsamemannerasthe p-valueforthe F-testwedescribeabove,andcanbeusedtodeterminethestatisticalsigni"canceofeachtermintheregressionmodelindividually,whereasthe F-testdeterminesthesigni"canceoftheparametersasagroup.Thepositivesignsonthecotsfor k and t 0areconsistentwiththesensitivityplotsinFigures3and4andthespiderplotinFigure15and,ofcourse,withintuition.Similarly,thenegative cotfor tisconsistentwithintuitionandFigure15:AsthetimerequiredtocarryoutPMgrows,thebene"tofthePMoptiondrops.Thesignsandmagnitudeofothercotsaremore subtle.Thepositivecotfor1 3iscounterintuitive:Asthereliabilityofcomponent3grows,thebene"tfromPMshouldshrink,notgrow.However,examiningthesignandrelativemagnitude ofthecotsfor1 3 and1 3*kandknowingthenominalvalueof kis2weseethatas1 3grows,theregressionestimateof P{T>t 0}indeedshrinks.Thisholdsexceptforvaluesof k1,ontheboundaryoftheexperimentaldesign,wherethequalityoftheregression"tislikelypoorer.WecanseefromtheinformationinTable5,thatseveralofthetwo-factorinteractionstermsaresigni"cant,includingallinteractiontermsinvolving k and t 0,indicatingthattheseparametershaveasigni"cantontheresponsevariable P{T>t 0}.ThisisagainconsistentwithwhatwelearnedfromthetornadodiagraminFigure9andthespiderplotofFigure15.Inaddition,we seethattheinteractionterm k*t 0hasthegreatest t-statisticamongallterms,andwecanconcludethatitisoneofthemostsigni"cantcontributorstotheestimateof P{T>t 0}.Thefactthatthisnonlinearityisimportant,andthatthesignof k*t 0scotispositive,isnotsurprisinggiventhetwo-waysensitivityplotinFigure17for k and t 0.Toillustratethevalueofthisregressionoutput,wedemonstratehowtousethisregressionequationtopredictthevalueof P{T>t 0},withouttheneedtorerunthesimulationmodel.Thisisespeciallyimportantforlarge-scalestochasticsimulationmodelsforwhichitmaytakeseveral daysorevenweekstorunadesignedexperimentandcollectthetypeofoutputwecollectedfor thisillustrativeexample.Ifthestatisticalexperimentisplannedwell,theregressionmetamodel canbeusedtopredictperformancemeasureswithouttheneedforrerunningthesimulationmodel. However,wenotethatusingthisapproach,weshouldnotusevaluesoftheinputparameters outsidetheboundsofthoseusedintheexperimentaldesign,inourcasetheminimumparameter valuesfromTable1.And,aswehavealreadyseen,neartheboundaryofthedesign,theregression "tmaydegrade.Inaddition,weshouldexaminethegoodness-of-"tstatisticswedescribeabove toensurethestatisticalmodelisadequatebeforerelyingonitspredictivepower.Table6applies ourregressionmetamodeltothenominalvaluesforallparameters,andformsanestimateof P{T>t 0}withouttheneedtorunthesimulationmodel.Table7presentstheresultsofsimplyusingthesimulationmodeltoestimate P{T>t 0}anditsassociatederror,andwecancomparetheseestimatestothevalueobtainedbyusingtheregressionmetamodeltodeterminethevalidity 35 ofmetamodel.Table6:Anexampleofapplyingtheresultsoftheregressionmodeltothenominalvaluesoftheinputparametersinordertopredict P{T>t 0}.Notethatusingthismethod,wenolongerneedtorunthesimulationmodeltopredict P{T>t 0}asafunctionofanycombinationofinputparameters.Theadjusted R 2valuefromTable4is0.8895,andinpracticethisvalueisacceptable,although0.90istypicallyconsideredthethresholdforaexcellent"t.CotsCotParametersParameterContribution EstimatesValues0-4.374E-02InterceptNone -0.04371-4.420E-061 1 200 None2-4.166E-051 2 200 None3 2.448E-041 3 50 0.01224 4.089E-041 4 50 0.02045 1.480E-02 k 2 0.02966-2.160E-02t 1-0.02167 2.959E-03 t 0 18 0.05331 , 2 7.682E-091 1*1 2 40000 None1 , 3-3.916E-071 1*1 3 10000 None1 , 4-3.837E-071 1*1 4 10000 None1 , 5 1.009E-051 1*k 400 0.00401 , 6-6.564E-061 1*t 200 None1 , 7 1.901E-061 1*t 0 3600 0.00682 , 3-4.407E-071 2*1 3 10000 None2 , 4-1.680E-081 2*1 4 10000 None2 , 5 1.061E-051 2*k 400 0.00422 , 6-5.576E-061 2*t 200 None2 , 7 3.070E-061 2*t 0 3600 0.01113 , 4 1.319E-051 3*1 4 2500 0.03303 , 5-2.303E-041 3*k 100-0.02303 , 6-8.428E-051 3*t 50-0.00423 , 7-3.768E-051 3*t 0 900-0.03394 , 5-2.378E-041 4*k 100-0.02384 , 6 2.797E-041 4*t 50 0.01404 , 7-5.333E-051 4*t 0 900-0.04805 , 6-1.725E-03 k*t 2-0.00355 , 7 1.183E-03 k*t 0 36 0.04266 , 7 3.536E-04t*t 0 18 0.0064EstimateofP{T>t 0}0.03591 36 Table7:Resultsfor P{T>t 0}fromrunningthesimulationmodelwithallinputparametersatnominallevelswithsamplesize N=1000.P{T>t 0}0.033095%CIHalfwidth 0.014995%CILowerLimit 0.018195%CIUpperLimit 0.0479InTable6weagainprovidetheestimatesofthecotsandtheparametersandinteractionstheyrepresent.Thefourthcolumnofthistablecontainstheparametervaluesforeachofthe associatedcotsforthenominalcase,withtheexceptionoftheinterceptwhichisnotdirectly linkedtoanyparameterorinteraction.The"rstsevenvaluesinthiscolumnmatchthenominal parametervaluesgiveninTable1.Theinteractiontermsaresimplytheproductsoftheseinput parametervalues.Forexample,thenominalvalueof t=1,andthenominalvalueof k=2,andthustheparametervaluefortheinteraction t*k=2,asshowninthetable.Asshowninregressionmodel(3),anestimateof P{T>t 0}canbeformedbycomputingtheproductoftheparameter(orinteractionterm)valuesandthecot,andsummingthosevaluesalongwiththevalueof theintercept.The"nalcolumninTable6istheproductofthecotandparametervalue. Summingallthevaluesinthiscolumnprovidesuswithanestimateof P{T>t 0}=0.03591whenallparametersareattheirnominallevels.Itisusefultocomparethisestimatewithapointestimateandassociated95%con"dencelimitsof P{T>t 0}usingthesimulationmodel.Table7presentsresultsfromthesimulationmodel,andweseethattheestimateof P{T>t 0}is0.0330+/-0.0149=(0.0181 , 0.0479).Wecanseethatourregressionmetamodelestimateof0.0359iswithinthe95%con"dencelimits,andist fromthesimulationpointestimateof0.0330bylessthan10%.Thissuggeststhatourregression metamodelcanbeusedinlieuofrunningthesimulationunderappropriatecircumstances.Inadditiontopredictingthevaluesofperformancemeasures,wecanalsousetheresultsofthedesignedexperimenttoconstructtwo-wayinteractionplotsthatdescribehowagivenresponse changesasafunctionoftwoinputparameters.Figures18-21showtwo-wayinteractionplotsfor pairsofinputparameters,wheretheresponsevariableisagain P{T>t 0}.Forexample,inFigure18,weseehow P{T>t 0}changeswhenallinputparametersareheldconstantexcept for k and1 3.First,weseethatwhen k=1,usingthepointestimates,wepreferthebaseoptionregardlessofthevalueof1 3.However,forvaluesof kgreaterthanone(namely3and5),wepreferthePMoptionregardlessofthevalueof1 3.When1 3isatitsminimumvalueof30,changesintherepairfactor khaveamoresigni"canton P{T>t 0}thanwhen1 3isatthenominalormaximumlevels.Figure19showssimilarresultsfor k and t 0.Thefactthat P{T>t 0}grows with k and t 0isasexpected,asistheampli"cationoftheofgrowing kforlargervaluesof t 0.Figures20and21depictanalogousresultsfortherespectivepairs(k,t)and(t 0 ,t).37 Figure18:The"gureisatwo-wayinteractionplotfor k and1 3,wheretheresponsevariableisP{T>t 0}.Figure19:The"gureisatwo-wayinteractionplotfor k and t 0,wheretheresponsevariableisP{T>t 0}.38 Figure20:The"gureisatwo-wayinteractionplotfor kand t,wheretheresponsevariableisP{T>t 0}.Figure21:The"gureisatwo-wayinteractionplotfor t 0and t,wheretheresponsevariableisP{T>t 0}.39 Runningfullfactorialdesigns,asthenumberofinputparametersgrowslarge,andtheunderly-ingsimulationmodeliscomputationallyexpensive,quicklybecomesintractable.Thewayforward istoemployfractionalexperimentaldesignsand/ortoattempttoreducethedimensionofthe inputvector.Fractionalfactorialdesignsusefewerdesignpoints,andliketheanalysiswegive above,disregardhigherorderinteractiontermsinfavorofestimatingmainandtwo-way interactions.Thereisalargeliteratureonthistopic,anditisnotourgoalheretoreviewthisin detail.See,forexample,thesurveyin[9]andreferencescitedtherein.4FurtherDiscussionImplicitinmuchofourdiscussioninsteps5-9isthenotionofthresholdanalysis.Forourexample,thePMoptionispreferredunderthenominalvaluesoftheinputparameters.Understanding howmuchaninputparameterwouldneedtochangeinordertochangethatassessmenthasbeen importantinourdiscussionsofone-andtwo-parametersensitivityplots,tornadodiagrams,UQ plots,andspiderplots.Thisnotionisevenembeddedinourde"nitionoftheoutputperformance measure P{T>t 0P{T PM>t 0}P{T base>t 0}.Strictlyspeaking,thenotionofathresholdvalueapplieswhenestimatesoftheoutputvariablesareprecise.Whentheycontainsamplingerror, ortheyareuncertainbecauseofuncertaintiesintheinputparameters,amorenuancedanalysisis needed.Forexample,inasensitivityplotfor P{T>t 0}asafunctionof k,weexaminewhether95%con"denceintervalsincludezerotounderstandwhetherthePMoptionispreferred,thebase optionispreferred,orwhether,basedonsamplingerror,wecannotassesswhichispreferable.This thirdcharacterizationismadeifthecon"denceintervalcontainszero.Inthatcase,weareinan encezoneinwhichwecannotassesswhetherthePMorbaseoptionispreferred.Whenweplaceapriordistributionon k,wecanobtainaprobabilitydistributiongoverningtherandomvariables P{T>t 0lk},and E[Tlk],andwecanassesstheprobabilitythatoneoftheseoutputsispositive(favoringthePMoption)ornegative(favoringthebaseoption).Insomecases,aswerangeaninputparameter,thepreferenceforoneoptionoveranalternativedoesnotchange.Inthiscasewehaveadominancerelationship.AsFigure3illustrates,wepreferthePMoptionoverthebaseoptionforallvaluesoftheinputparameter t 0ofinterest.Itisimportanttonotethatwemayhaveconsiderablevariabilityinanoutputmeasure,butifwehavea dominancerelationshipthenthisvariabilityisofsecondaryinterest.Ofprimaryinterestiswhether thedecisionwewouldmakechanges.Perhapstheforemostcaveatwhenperformingaone-waysensitivityanalysisofatrustedmodelistoassesswhetheritmakessensetovarytheinputparametersoneatatime.Iftwoormore inputparametersdependonanunstatedauxiliaryfactor,thismaynotbevalid.Forexample,it maybethatcomponents1and2inourexampleareidenticalbutweareunsureoftheassociated failurerate.Inthiscase,weshouldhave1=2,i.e.,weshouldreplacethetwoinputparameters1 and2withasingleparameter.Inothercases,thedependencybetweentwoparametersisnotsosimple.Forexample,if1 and2arerandomvariablestheymayhaveadependentjoint 40 probabilitydistributioninwhichthecorrelationispositive(butnotperfect).Inthiscase,wecouldviewthatcorrelationcotbetween1 and2asaninputparametertobevariedorwecouldformaUQplotandtocharacterize P{T>t 0l(12)}.Anothercaveatinourone-at-a-timesensitivityanalysisconcernsthenotionthatallinputparameterstaketheirnominalvaluesexceptforone.Thiscanmasktheofcrossterms. Thelevelofoneinputvariabledepartingfromitsnominalvaluemayamplifytheofchanges inanotherinputvariable.Thepurposeofthemetamodelanalysisinstep10istounmaskprecisely suchinteractions.Whetherdonebyaformalexpertelicitationoraninformalscheme,thenominalvaluesoftheparametersandtheirrangesaretypicallybasedonexpertopinionandhencesubjecttowell-known biasesrelatedtoanchoring,overcon"dence,etc.ParticularlyrelevantforGSI-191analysisare inassessingrare-eventprobabilities.Forexample,seethediscussionsinTverskyand Kahneman[21]andOHaganetal.[15],andalsoseeKynn[11].Modeluncertaintyisanoftenneglectedpartofsensitivityanalysis.Asimpleformofmodeluncertaintyforourexampleconcernsthedistributionalassumptiononthefour-tupleoffailure times(T 1 ,...,T 4).Wehaveassumedthesefourfailuretimestobeindependentandtohaveexponentialdistributions.tdistributionalassumptions,e.g.,adependentjointdistribution inwhicheachcomponentisaWeibullrandomvariablemightprovideahigher"delitymodel.These distinctionscanarisebecauseofimportantdinassumptionsmadeontheunderlying physicalmodelgoverningcomponentfailure.Hypothesizingcompetingmodelsforanunderlying phenomenonandunderstandingthedomainofapplicabilityofsuchmodelsis,ofcourse,centralto scienti"cinvestigation.5ConclusionsandSomeEmergingToolsforSensitivityAnalysisInthisreport,wehaveproposeda10-stepsensitivityanalysisprocedurethatweseeaspractical forlarge-scalestochasticcomputersimulationmodels.And,wehaveillustratedtheseideasona simpleexampleofasimulationmodelforsystemreliability.Noneofthestepsweproposearenew. Rather,werelyontheliteraturesfromdecisionanalysis,econometrics,statistics,andsimulation toguidewhatwehaveproposed.Tornadodiagramsprovideasimplemeansforvisualizingthein"uenceofasigni"cantnumberofinputparametersonanoutputvariableandforunderstandingtowhichinputvariablesthe outputvariableismostsensitive.euseoftornadodiagramsrequirestheanalysttospecify reasonableandcommensuraterangesforacollectionofinputparameters.Bothsensitivityplots andspiderplotscomplementtornadodiagramsinthattheymoreeasily:(i)depictthenonlinear responseofanoutputvariabletochangesinaninputparameter,and(ii)depictthesampling-basederrorassociatedwithestimatinganoutputmeasure.Wehaveadvocatedcarefuldistinction ofMonteCarlosampling-basederrorsfromuncertaintiesininputparameters.Wemodelthelatter typeofuncertaintybyplacinga(possiblyjoint)probabilitydistributionontheinputparameters, 41 andinthissettingweseektounderstandtheresultingprobabilitydistributionontheoutputbypropagatingtheuncertaintythroughthenonlinearfunctionrepresentedbythesimulationmodel. Wediscusshowtopropagateuncertaintyforunivariateandmultivariatedistributionsoninput parameters.Finally,wedescriberegressionmetamodels,andassociatedexperimentaldesign,for understandingsensitivitiesandinteractionsbetweeninputparameters.Anumberofimportantthemesrepeatthroughoutourrecommendations.Thesehaveincludedtheuseofcommonrandomnumbersinordertoreducevariabilityandsmoothoutputanalysis.We havealsofocusedonassessingininputparametersthatmakeaindecisions orqualitativecharacterizationsofthesystemathand.Therelevantideaswehavediscussedin thisregardincludethresholdanalyses,zones,andestablishingdominancerelations. Sensitivityanalysissimpli"essigni"cantlywhenusingdeterministicsimulationmodels.However, oursimulationsarestochasticandhencepropercharacterizationofbothsamplingerrorandun-certaintiesoninputparametershasbeenapervasivethemeinourpresentation.Ourdiscussionisbynomeanscomprehensiveforsensitivityanalysisofcomputersimulationmodels.Alternativesincludecomputingderivatives,whichisoftentermedlocalsensitivityanalysis

see,e.g.,Sobol[19].Speci"cally,with f (x 1 ,x 2 ,...,x n)denotingasingleoutputmeasurefromoursimulationmodel,wecouldestimatethegradientx f (x 1 ,...,x n)= 1 ,..., n.Localsensitivityanalysisisofparticularinterestwhenattemptingtooptimize f,butoptimizationovertheinputsisnotourfocushere.Moreimportantly,ifwesimplyreportestimatesofthepartial derivatives i , i=1 ,...,n,thiscanmisleadwithrespecttowhatinputparametersaremostimportantbecause

(i)aperunitchangeinthetemperatureofwateratasumppumpmaynotbe commensuratewithapergram/fuel-assemblychangeindebrismasshavingpenetratedthestrainer; and,(ii)evenifwecompute ix iforcommensuratevaluesof x i,alinearapproximationmaybepoorovertherangeofparametersofinterest.Itisforthereasonsjustdiscussedthatweadvocatetheglobalsensitivityanalysisthatwehaveproposedinsteps4-10.Here,thenotionofglobalisspeci"edbytheanalystviarangesassociated withtheinputparameters(aswehavedoneinTable1)ratherthanratesofchangeatasingle point.Theserangesplayacentralroleinsensitivityplots,tornadodiagrams,spiderplots,andthe experimentaldesignsassociatedwithregressionmetamodels.(Eventhoughwetermtheseglobal, itisimportanttorecognizethatallbuttheregressionmetamodelinvolvechangingoneparameter atatime.)Suchanalysesneednotassociateaprobabilitydistributionwiththeinputparameters.

However,inourviewitispreferablewhensuchprobabilitydistributionscanbespeci"edbecause wecanmakeuseoftheminUQplotsaswellasourothersensitivityanalyses.Insuchcasesthe speci"cendpointsoftherangesbecomelessimportant.Wehaverecommendedusingtornadodiagramsasaninitialtoolforassessingthemostimportantinputparameters,andusingsensitivityplots,UQplots,spiderplots,andmetamodelstoenable aricherexplorationofmodelsensitivity.Itispossibletoemploymoresophisticatedstatistical 42 schemesforscreeningfactorsusingmetamodels[25],includingsequentialbifurcationscreening[24].However,evenwhensuchschemesareadvocated,itisacknowledgedthatsuchapproacheshaveyet toseesigni"cantapplicationinpractice[9].Originallydevelopedforinterpolationingeostatistical andspatialsampling,Kriging(see,e.g.,[5,17,18])hasseenwidespreadsuccessfulapplicationin thecontextofdeterministicsimulationmodelsasanalternativetoregressionmetamodels.More recently,Krigingmetamodelshavebeguntoseeapplicationtostochasticsimulationmodels;see, e.g.,[1,7,2].Whiletheseapproachesareapromisingalternativetoregressionmetamodels,there areanumberofoutstandingresearchissuesthatremaintobesolved[9]. 43 References[1]Ankenman,B.,B.L.Nelson,andJ.Staum(2008).StochasticKrigingforsimulationmetamod-eling.InS.J.Mason,R.R.Hill,L.Moench,andO.Rose(Eds.),Proceedingsofthe2008WinterSimulationConference,pp.362-370.[2]Beers,W.V.andJ.P.C.Kleijnen(2003).Krigingforinterpolationinrandomsimulation.JournaloftheOperationalResearchSociety54,255-262.[3]Clemen,R.T.andR.Reilly(2001).MakingHardDecisionswithDecisionTools.Duxbury,Paci"cGrove,CA.[4]Commission,N.R.(2011).RegulatoryGuide1.174:AnApproachforUsingProbabilisticRiskAssessmentInRisk-InformedDecisionsOnPlant-Speci"cChangestotheLicensingBasis, Revision2,NuclearRegulatoryCommission,Washington,DC.[5]Cressie,N.A.C.(1993).StatisticsforSpatialData.Wiley,NewYork.[6]Eschenbach,T.G.(1992).Spiderplotsversustornadodiagramsforsensitivityanalysis. Inter-faces22,40-46.[7]Gramacy,R.B.andH.K.H.Lee(2008).BayesiantreedGaussianprocessmodelswithanapplicationtocomputermodeling.JournaloftheAmericanStatisticalAssociation103,1119-1130.[8]Kleijnen,J.P.C.(1995).Veri"cationandvalidationofsimulationmodels.EuropeanJournalofOperationalResearch82,145-162.[9]Kleijnen,J.P.C.(2011).Sensitivityanalysisofsimulationmodels.InWileyEncyclopediaofOperationsResearchandManagementScience.Wiley,NewYork.[10]Kleijnen,J.P.C.andW.V.Groenendaal(1992).Simulation:AStatisticalPerspective.Wiley,Chichester.[11]Kynn,M.(2008).Theheuristicsandbiasesbiasinexpertelicitation.JournaloftheRoyalStatisticalSociety:SeriesA171,239-264.[12]Letellier,B.,T.Sande,andG.Zigler(2013,November).SouthTexasProjectRisk-InformedGSI-191Evaluation,Volume3,CASAGrandeAnalysis.Technicalreport,STP-RIGSI191-V03, Revision2.[13]Morton,D.P.,J.Tejada,andA.Zolan(2013,September).SouthTexasProjectRisk-InformedGSI-191Evaluation:Strati"edSamplinginMonteCarloSimulation:Motivation,Design,and SamplingError.Technicalreport,STP-RIGSI191-ARAI.03,TheUniversityofTexasatAustin. 44 [14]Ogden,N.,D.P.Morton,andJ.Tejada(2013,June).SouthTexasProjectRisk-InformedGSI-191Evaluation:FiltrationasaFunctionofDebrisMassontheStrainer:FittingaParametric Physics-BasedModel.Technicalreport,SSTP-RIGSI191-V03.06,TheUniversityofTexasat

Austin.[15]OHagan,A.,C.E.Buck,A.Daneshkhah,J.Eiser,P.Garthwaite,D.Jenkinson,J.Oakley,andT.Rakow(2006).UncertainJudgements:ElicitingExpertProbabilities.Wiley,Chichester.[16]Pan,Y.-A.,E.Popova,andD.P.Morton(2013,January).SouthTexasProjectRisk-InformedGSI-191Evaluation,Volume3,ModelingandSamplingLOCAFrequencyandBreakSize.Tech-nicalreport,STP-RIGSI191-V03.02,Revision4,TheUniversityofTexasatAustin.[17]Sacks,J.,W.J.Welch,T.Mitchell,andH.Wynn(1989).Designandanalysisofcomputerexperiments.StatisticalScience4,409-435.[18]Santner,T.J.,B.J.Williams,andW.I.Notz(2003).TheDesignandAnalysisofComputerExperiments.Springer-Verlag,NewYork.[19]Sobol,I.M.(2001).GlobalsensitivityindicesfornonlinearmathematicalmodelsandtheirMonteCarloestimates.MathematicsandComputersinSimulation55,271-280.[20]Tregoning,R.,P.Scott,andA.Csontos(2008,April).EstimatingLoss-of-CoolantAccident(LOCA)FrequenciesThroughtheElicitationProcess:MainReport(NUREG-1829).NUREG 1829,NRC,Washington,DC.[21]Tversky,A.andD.Kahneman(1981).Rationalchoiceandtheframingofdecisions. Sci-ence211,453-458.[22]Wake"eld,D.andD.Johnson(2013,January).SouthTexasProjectRisk-InformedGSI-191Evaluation,Volume2,ProbabilisticRiskAnalysis:DeterminationofChangeinCoreDamage FrequencyandLargeEarlyReleaseFrequencyDuetoGSI-191Issues.Technicalreport,STP-RIGSI191-VO2,Revision0,ABSGConsultingInc.[23]WCAP-16793-NP(2011,January).EvaluationofLong-TermCoolingConsideringParticulate,FibrousandChemicalDebrisintheRecirculatingFluid,Revision2.[24]Xu,J.,F.Yang,andH.Wan(2007).Controlledsequentialbifurcationforsoftwarereliabilitystudy.InS.G.Henderson,B.Biller,M.-H.Hsieh,J.Shortle,J.D.Tew,andR.R.Barton(Eds.), Proceedingsofthe2007WinterSimulationConference,pp.281-288.[25]Yu,H.-F.(2007).DesigningascreeningexperimentwithareciprocalWeibulldegradation rate.ComputersandIndustrialEngineering52,175-191.45 AAppendix:ASensitivityAnalysisforSTPGSI-191InthisappendixweapplytheproposedframeworktoanalyzethesensitivityofestimatesofariskmeasuretochangesininputparametersoftheCASAGrandesimulationmodel,usingSTPdata. TheCASAGrandesimulationmodelhasthecharacteristicswedescribeintheabstractandin Section1,andtheframeworkwedescribeinthisreportwasdesignedwithlarge-scalestochastic simulationmodelslikeCASAGrandeinmind.Inwhatfollows,webrie"yreviewthesamplingschemewithinCASAGrande,andwedescribetheloosecouplingbetweenCASAGrandeandthePRAmodel.Wedescribehowweestimaterisk intermsofthecontributiontocoredamagefrequency(CDF)fromGSI-191,inunitsofevents percalendaryear(CY),usingestimatesoftheconditionalfailureprobabilitiesthatCASAGrande provides.Thatis,ourriskmeasureisthechangeincoredamagefrequencyrelative toabaseCDFduetonon-GSI-191issues.Wepresenttheresultsof22scenarios,whereeachscenariospeci"esthevaluesoftheinputparameterstoCASAGrandeandwhereoneofthese scenarioscontainsnominalvaluesfortheparameters.Thispresentationincludesatornadodiagram representingchangesinallparametersweconsider,andfurtherincludesasensitivityplotforone ofthekeyinputs.A.1Step1:De"netheModelWerefertoVolume3[12]foradiscussionoftheCASAGrandesimulationmodel.Oneimportantaspectofthissimulationforthepurposeofouranalysishereisthefactthatastrati"edsampling estimatorisemployed,inwhichthestrati"cationisontheinitiatingfrequency.Theprobability distributiongoverningtheinitiatingfrequencyisconsistentwithpercentilesfromNUREG-1829 [20]aswedescribein[16].Thisstrati"edsamplingestimatorcanbethoughtofasanouterloop ofreplicationswhenrunningthesimulationmodel,whichwerefertoasfrequencyreplications. Thisouterloopfacilitatespreservationoftheprobabilitydistributionforinitiatingfrequencyin thesenseoftheuncertaintyquanti"cationplotsdescribedinStep7,andthestrati"edestimator furtherreducesvarianceversusana¨veMonteCarloestimator.Withineachfrequencyreplication,i.i.d.replicationsareperformedinordertoestimatecon-ditionalfailureprobabilitiesforeachmodeoffailure(sumpandboron"berlimit)andbreaksize (small,medium,andlarge),conditionedonthepumpstateaswellastheinitiatingfrequency.A strati"cationwith15cellsisusedforthestrati"edestimatorwithrespecttotheinitiatingfrequency, andimportantly,thesamplingindistinctcellsofthestrataisdoneindependently,unlessspeci"ed otherwise.Asamplesizewithineachcellofthestrati"cationisselected,aswellastheboundaries ofeachcellofthestrata,asindicatedinTable8.See[13]forbackgroundonthestrati"edsampling estimator.Theright-mostcolumninTable8isbasedonoptimizationmodel(10)in[13]. 46 Table8:Strati"cationofinitiatingfrequencyintermsofquantilesofitsdistributionfunction F.Theprobabilitymassforeachcellisindicated,asisthesamplesizedevotedtoi.i.d.replications withineachcell.FrequencyCellCellProbabilityNumberofReplicationLowerLimitUpperLimitMassStat.Replications 1 F1 (0.000)F1 (0.045)0.04511 2 F1 (0.045)F1 (0.115)0.07012 3 F1 (0.115)F1 (0.195)0.08011 4 F1 (0.195)F1 (0.260)0.0659 5 F1 (0.260)F1 (0.295)0.0357 6 F1 (0.295)F1 (0.365)0.07011 7 F1 (0.365)F1 (0.435)0.0708 8 F1 (0.435)F1 (0.510)0.07523 9 F1 (0.510)F1 (0.620)0.11045 10 F1 (0.620)F1 (0.685)0.06518 11 F1 (0.685)F1 (0.720)0.03513 12 F1 (0.720)F1 (0.830)0.11051 13 F1 (0.830)F1 (0.955)0.12550 14 F1 (0.955)F1 (0.990)0.03528 15 F1 (0.990)F1 (1.000)0.01011A.2Step2:SelectOutputsofInterestThechangeincoredamagefrequencywhenaccountingforGSI-191processes,isselectedastheoutputofinterestforthissensitivitystudy.A.2.1CoreDamageFrequencyThemethodforestimatingcombinesestimatesfromtheCASAGrandesimulationmodelwithcots,asweexplainhere,fromSTPsPRA[22].TheCASAGrandesimulationmodel isusedtoestimatetheconditionalprobabilitiesofasumpfailureandaboron"berlimitfailureat variousbreaksizes(small,medium,andlarge),whenweconditionontheinitiatingfrequencyof aLOCAandthepumpstate.FromthePRA,wecanuseabaseCDFfromnon-GSI-191events, aswellasthecoredamagefrequenciesassociatedwithasumporboron"berlimitfailure,further conditionedoneachpermutationofpumpstateandbreaksize.Thisisformalizedasfollows. 47 IndicesandSets: i=1 ,...,Findexforcellsstratifyingfrequencyreplications j=1 ,...,M iindexforstatisticalreplications k=1 ,...,Nindexforsetofpumpstates Events: SLsmallLOCA MLmediumLOCA LLlargeLOCA PS kpumpsinstate k F iinitiatingfrequencyincell i Ssumpfailure Bboron"berlimit CDcoredamageParameters: f SLfrequency(events/CY)ofaSmallLOCA f MLfrequency(events/CY)ofaMediumLOCA f LLfrequency(events/CY)ofaLargeLOCA P (PS k)probabilitymassof PS k P (F i)probabilitymassof F iP (SlLOCA, F i,PS k)estimateofprobabilityof SgivenLOCA=SL,ML,orLL , F i , PS kP (BlLOCA, F i,PS k)estimateofprobabilityof BgivenLOCA=SL,ML,orLL , F i , PS k RBASEnon-GSI-191coredamagefrequency(events/CY) R CDestimateofcoredamagefrequency(events/CY)Thethreefrequencies, f SL , f ML,and f LL,aretakenfromtheright-mostcolumnofTable4-1fromVolume2[22].Inthesensitivityanalysiscomputationsthatfollow,weuse F=15frequencyreplications,andforthe i-thfrequencyreplicationcell,weuse M istatisticalreplicationswiththevaluesfor M igivenintheright-mostcolumnofTable8.AsdiscussedinVolumes2and3[22,12],ingeneralwewouldconsideratotalof N=64pumpstates,althoughareducednumberofboundingpumpstatesareusedinactualcomputation.These64pumpstatesincludethemostlylikelyPump State1,whichhasallpumpsonallthreetrainsavailable.Intheresultswepresenthere,we onlyconsiderPumpState1,whichhasaprobabilitymassof0.935whenweconsiderall64pumpstates. Intermsofournotationthismeanswehave N=1and P (PS 1)=1,elyeliminatingthesumover k.Ifweweretocompute P (PS k)moregenerally,wewoulddosobynormalizingthepumpstatefrequenciesintheleft-handcolumnofTable9-1inVolume2[22]orequivalentlythe samefrequenciesinTable2.2.11inVolume3[12].Inourcomputationswealsotake RBASE=0sothatR CDestimatesThatsaid,wedeveloptheformulasthatfollowforgeneralvaluesof F , M i , N,and RBASE.48 EstimatingCDFTheestimateforCDFiscalculatedbysummingtheprobabilityofeachpumpstate-initiatingfrequencypair,andmultiplyingeachtermbythecorrespondingPRAfrequenciescoupledwiththe conditionalfailureprobabilitiesestimatedbyCASAGrande.Theformulafortheestimatorisas follows:R CD=RBASE+Fi=1 Nk=1 P (F i)P (PS k)*(4a)f SL*P (SlSL, F i,PS k)+f SL*P (BlSL, F i,PS k)(4b)+f ML*P (SlML, F i,PS k)+f ML*P (BlML, F i,PS k)(4c)+f LL*P (SlLL, F i,PS k)+f LL*P (BlLL, F i,PS k).(4d)Eachofthesixprobabilityestimates,e.g.,P (SlLL, F i,PS k),isformedviaasamplemeanofi.i.d.observations,e.g.,P j (SlLL, F i,PS k), j=1 ,...,M i,withintheCASAGrandesimulation.That is,P (SlLL, F i,PS k)=1 M i M ij=1P j (SlLL, F i,PS k).(5)Inthisway,equation(5),andits"veanalogs(e.g.,for P (SlML, F i,PS k)),aresubstitutedforthecorrespondingtermsinequation(4)toformtheestimator R CD.EstimatingtheVarianceoftheCDFEstimatorWemustestimatethevarianceoftheestimator R CDinordertoquantifyitssamplingerror.Thereareatotalof6

• F*N+1termsinequation(4)de"ning R CD,andinordertoestimatethevariance,wemustclarifywhichofthesepairsoftermsareindependentandwhicharedependent.

First,weassumetheterms P (F i)and P (PS k),aswellasthefrequenciesfromthePRAsuchas RBASE and f LL,aredeterministic.Dependency,orthelackthereof,betweenpairsofestimatorslike P (SlSL, F i,PS k)andP (BlSL, F i,PS k)dependonhowthesimulationisperformed.ThetermsacrossdistinctfrequencyreplicationsandpumpstatesareindependentbecauseindependentMonte Carlosamplesaredrawnwithineachpumpstate-initiatingfrequencypair;however,thesixterms withineachpumpstate-initiatingfrequencypairaredependent.Thus,with V (*)andCOV (*)denotingthesamplevarianceandsamplecovarianceoperators,wehavethefollowingequation

for V (R CD),whichhas15samplecovariancetermsbecausewehavesixpumpstate-initiatingfrequencypairs: 49 VR CD=Fi=1 Nk=1[P (PS k)]2*[P (F i)]2*(6a)f 2 SL*VP (SlSL, F i,PS k)+f 2 SL*VP (BlSL, F i,PS k)(6b)+f 2 ML*VP (SlML, F i,PS k)+f 2 ML*VP (BlML, F i,PS k)(6c)+f 2 LL*VP (SlLL, F i,PS k)+f 2 LL*VP (BlLL, F i,PS k)(6d)+2*f SL*f ML*COVP (SlSL, F i,PS k),P (SlML, F i,PS k)(6e)+2*f SL*f LL*COVP (SlSL, F i,PS k),P (SlLL, F i,PS k)(6f)+2*f SL*f SL*COVP (SlSL, F i,PS k),P (BlSL, F i,PS k)(6g)+2*f SL*f ML*COVP (SlSL, F i,PS k),P (BlML, F i,PS k)(6h)+2*f SL*f LL*COVP (SlSL, F i,PS k),P (BlLL, F i,PS k)(6i)+2*f ML*f LL*COVP (SlML, F i,PS k),P (SlLL, F i,PS k)(6j)+2*f ML*f SL*COVP (SlML, F i,PS k),P (BlSL, F i,PS k)(6k)+2*f ML*f ML*COVP (SlML, F i,PS k),P (BlML, F i,PS k)(6l)+2*f ML*f LL*COVP (SlML, F i,PS k),P (BlLL, F i,PS k)(6m)+2*f LL*f SL*COVP (SlLL, F i,PS k),P (BlSL, F i,PS k)(6n)+2*f LL*f ML*COVP (SlLL, F i,PS k),P (BlML, F i,PS k)(6o)+2*f LL*f LL*COVP (SlLL, F i,PS k),P (BlLL, F i,PS k)(6p)+2*f SL*f ML*COVP (BlSL, F i,PS k),P (BlML, F i,PS k)(6q)+2*f SL*f LL*COVP (BlSL, F i,PS k),P (BlLL, F i,PS k)(6r)+2*f ML*f LL*COVP (BlML, F i,PS k),P (BlLL, F i,PS k).(6s)Toillustratecomputationofthesamplevarianceterms, V (*),andsamplecovarianceterms,COV (*),inequation(6),wegivetheformulasforthe"rstsamplevariancetermfrom(6b)andthe"rstsamplecovariancetermfrom(6e): VP (SlSL, F i,PS k)=1 M i 1 M i1 M ij=1P j (SlSL, F i,PS k)P (SlSL, F i,PS k)2 (7)50 andCOV (P (SlSL, F i,PS k),P (SlML, F i,PS k)=(8a)1 M i 1 M i1 M i X j=1 hP j (SlSL, F i,PS k)P (SlSL, F i,PS k)i*hP j (SlSL, F i,PS k)P (SlML,F i,PS k)i.(8b)FurtherRemarksonEstimatingCDFForthissensitivityanalysisstudy,weestimate R CDconditionalonbeinginthepumpstateinwhichallthreepumpsareoperatingonallthreetrains;i.e.,themostlikelypumpstatecalled PumpState1.Thiseliminatesthesumacrosspumpstatesintheestimatorsgivenaboveand, becausewearecomputingaconditionalprobability,hastheofsettingtheprobabilitymass associatedwithPumpState1equalto1.Inadditionwetake RBASE=0sothatR CD estimatesratherthanCDF.Inwhatfollows,weareinterestedinin R CDunderpairsofscenarios,i.e.,undertwosetsofinputparameterstoCASAGrande.Speci"cally,wehave R 0 CDunderthenominalsettingsoftheparameters,RCDunderperturbedsettingsoftheparameters,andourinterestliesintheestimatorR CD=RCDR 0 CD.Whencomputingthisweusecommonrandomnumbersacrossthetwoscenariostoreducethevarianceof R CD,whichiscomputedusingastraightforwardvariantofequation(6)inwhichthesamplevarianceandsamplecovariancetermshavetermslike P j (SlSL, F i,PS k)andP (SlSL, F i,PS k)replacedby P j (SlSL, F i,PS k)andP (SlSL, F i,PS k),respectively.Sometimesitisconvenienttoreportanddisplaytheratioofthethesefrequenciesestimates,ratherthantheiri.e.,wereport R 1 CDR 0 CD , (9)andtermthistheratioofrisk.Weagainusecommonrandomnumberswhencomputingthisestimator,andthesamplevarianceofthisisestimatedvia VR 1 CDR 0 CD=1[R 0 CD]2V (R 1 CD)2R 1 CDR 0 CDCOV (R 1 CD ,R 0 CD)+R 1 CDR 0 CD2 V (R 0 CD).(10)A.3Step3:SelectInputsofInterestThefollowinginputparametersofinterestwereselected,viamultiplediscussionsoftheSTPTech-nicalTeam,fromalargercollectionofcandidateparameters.Theselectionwasbasedon:(i)the uncertaintyassociatedwithestimatesofthevaluesoftheparameters,and(ii)theperceivedlikeli-hoodthatbiaswould,basedontheteamsdeliberations,havethelargestontheestimates ofrisk,eitherintheincreasingordecreasingdirection. 51 1.AmountofLatentFiberinthePool;4levels2.BoronFiberLimitintheCore;4levels 3.DebrisTransportFractionsInsidetheZoneofIn"uence;3levels 4.ChemicalPrecipitationTemperature;2levels 5.TotalFailureFractionsforDebrisOutsidetheZoneofIn"uence(Unquali"edCoatings);2levels6.ChemicalBump-UpFactor;2levels 7.FiberPenetrationFunction;2levels 8.SizeofZoneofIn"uence;2levels 9.TimetoTurnOneSprayPump;2levels10.TimetoHotLegInjection;2levels 11.StrainerBucklingLimit;2levels 12.WaterVolumeinthePool;3levels 13.DebrisDensities;2levels 14.Time-DependentTemperaturePro"les;2levels 15.SprayTransportFractionsforDebrisOutsidetheZoneofIn"uence(Unquali"edCoatings);2levelsAshortdescriptionoftheparametersofinterestisprovidedinthefollowing.SeealsoSections2and5ofVolume3[12]forfurtherdiscussion.AmountofLatentFiberinPoolThereisanamountofexistingdustanddirtinthecontain-ment,whichisbasedonplantmeasurement.Thelatent"berisassumedtobeinthepoolat thestartofrecirculation.Thislatentdebrisisthereforeavailableimmediatelyuponstartof recirculation,uniformlymixedinthecontainmentpool.During"llup,thislatentdebrisis alsoavailabletopenetratethesumpscreen.BoronFiberLimitTheboron"berlimitreferstotheassumedsuccesscriterion,orthresholdofconcernwhereboronprecipitationwouldbeassumedtooccurforcoldlegbreaks.The"ber limitcomesfromthetestingperformedbythevendorthatshowsnopressuredropoccurs withfullchemicalTheassumptionisthatall"berthatpenetratesthroughthesump screenisdepositeduniformlyonthecore. 52 DebrisTransportFractionsinZOIThisreferstothethree-zoneZOIdebrissizedistribution.EachttypeofinsulationhasacharacteristicZOIwhichisdividedinthreesections totakeintoaccountthetypeofdamage(debrissizedistribution)withineachzone.ChemicalPrecipitationTemperatureCASAGrandeassumesthat,onceathinbedof"berisformedonthestrainer,thechemicalprecipitationbumpupfactorsareappliedwhenthe pooltemperaturereachestheprecipitationtemperature,de"nedininput.TotalFailureFractionforDebrisOutsidetheZOICASAGrandeusesatableoftotalfail-urefractionsthatareappliedtothetransportlogictrees.Thefractionofeachtype("ber, paintandcoatings,andsoforth)thatpassesthroughareastothepoolareusedtounder-standwhatisinthepoolasafunctionoftimeduringrecirculation.Thetotalfailurefraction multipliesthetotalinventoryofunquali"edcoatings.ChemicalBumpUpFactorThechemicalbumpupfactorisusedasamultiplieronthecon-ventionalheadlosscalculatedinCASAGrande.Themultiplierisappliedifathinbedis formedandthepooltemperatureisatorbelowtheprecipitationtemperature.FiberPenetrationFunctionTheamountof"berthatbypassestheECCSsumpscreen(asafraction)iscorrelatedtothearrivaltimeandtheamountof"beronthescreen.The cotsofthecorrelationde"nethefractionalpenetrationamounts.SizeofZOIThezoneofin"uence(ZOI)isde"nedasadirectfunction(multiplier)ofbreaksize(andnominalpipediameter).Forexample,forNUKON"ber,theZOI(forSTP)is17times thebreakdiameter.TheZOIisassumedtobesphericalunlessitisassociatedwithlessthan afull(double-ended)breakinwhichcaseitishemispherical.Otherwise,itistruncatedby anyconcretewallswithintheZOI.TimetoTurnOneSprayPumpIfthreespraypumpsstart,thenbyprocedureoneisse-cured.Thetimetosecurethepumpisgovernedbytheoperatoractingontheconditional actionstepintheprocedure.TimetoHotLegInjectionSimilartothespraypumpturntime,thetimetoswitchoneormoretrainstohotleginjectionoperationisgovernedbyprocedure.StrainerBucklingLimitThestrainerbucklinglimitisthetialpressureacrosstheECCSstraineratwhichthestrainerisassumedtofailmechanically.Thislimitisbasedonengi-neeringcalculationsthatincorporateafactorofsafety.WaterVolumeinthePoolDependingonthebreaksize,theamountofwaterthatisinthepool,asopposedtoheldupintheRCSandotherareasincontainment,isvariable.Smaller breakstendtoresultinlesspoolvolumethanlargerbreaks. 53 DebrisDensitiesThedebrisdensitydependsontheamountandtypeofdebristhatarrivesinthepool.Thesedensitiesareusedinheadlosscorrelationstocalculate,forexample,debris volume.TimeDependentTemperaturePro"lesThetemperatureofthewaterinthesumpairreleaseandvaporizationduringrecirculation.Thetime-dependenttemperaturepro"lecomes fromthecoupledRELAP5-3DandMELCORsimulationsdependingonbreaksize.SprayTransportFractionsforDebrisOutsidetheZOICASAGrandeusesatableoffail-urefractionsthatareappliedtothetransportlogictrees.Thefractionofeachtype("ber, paintandcoatings,andsoforth)thatpassesthroughareastothepoolareusedtounderstand whatisinthepoolasafunctionoftimeduringrecirculation.Thespraytransportfraction isthefractionoffailedcoatingsthatwashtothepoolduringsprayoperation.The"rsttwoinputs,latent"berandboron"berlimit,havefourlevels,includingthenominalcase.Items3and12,debristransportwithintheZOIandwatervolume,havethreelevels.And, allotheroftheotherinputparametershavetwolevels,thenominalcaseandaperturbationina singledirection.Thus,thenumberofrunsneededtoconductone-at-a-timesensitivityanalysisis:

2*(41)+2*(31)+11*(21)+1=22.Thenumberofrunsneededtoconductasinglereplicatefullfactorialdesignwouldbe4 2*3 2*2 11=294 ,912.Weconducttheformer,butwedonotattemptthelatterhere.A.4Step4:ChooseNominalValuesandRangesforInputsThenominalvalueforaninputparameterissometimesbasedontheSTPTechnicalTeamsbestpointestimateforthatinput.However,wesometimesinsteadselectanappropriatelyconservative nominalvalue,asisthecaseforthestrainerbucklinglimit,aswementionabove.Whenselecting rangesfortheparameters,sometimeswebothincreaseanddecreaseaparameterfromitsnominal value,butothertimesweonlychangetheparameterinasingledirection.Wecanmakethelatter choicebecausewewishtolimitchangestodirectionsthatweknowwillincreaseriskorbecause thenominalvalueisalreadyseenasbeingconservative.Here,welistthe22scenariosweusefor sensitivityanalysis.Scenario0:Nominal-valueCaseAllinputshaveanominalvalue.Thosenominalvaluesareasfollows:1.Theamountoflatent"berinthepoolis12.5ft 3.2.Theboron"berlimitinthecoreis7.5g/FA.3.Thedebristransportfractionsfordebrisgeneratedinsidethezoneofin"uencearegiveninTable9.54 Table9:Nominaldebristransportfractionsfordebrisgeneratedinsidethezoneofin"uence.DebrisTransportModel/LDFGLDFGLDFGMicrothermQualCoatCrudDebrisTypeFinesSmallLargeFinesFinesFinesBlowdownUpper0.700.600.220.700.700.70BlowdownLower0.300.250.000.300.300.30Washdown Inside0.530.270.000.530.530.53WashdownAnnulus0.470.190.000.470.470.47WashdownBCInside0.000.270.000.000.000.00WashdownBCAnnulus0.000.000.000.000.000.00PoolFill Sump0.020.000.000.020.020.02PoolFillInactive0.050.000.000.050.050.05 RecirculationLower1.000.640.001.001.001.00 Recirculation Inside1.000.640.001.001.001.00 RecirculationAnnulus1.000.580.001.001.001.00 ErosionSpray0.000.010.010.000.000.00 ErosionPool0.000.070.070.000.000.004.Thechemicalprecipitationtemperatureis140F.5.Thetotalfailurefractionsfordebrisgeneratedoutsidethezoneofin"uence(unquali"edcoatings)aregiveninrow1ofTable10.Table10:Nominaldebrisfailurefractionsfordebrisgeneratedoutsidethezoneofin"uence(un-quali"edcoatings).DebrisTransportModel/EpoxyEpoxyEpoxyEpoxyEpoxyAlkydBakedIOZDebrisTypeFinesFineSmallLargeCurlsEnamelFinesChipsChipsChipsTotalFailureFraction1.001.001.001.001.001.001.001.00UpperContainment0.150.150.150.150.150.540.000.83LowerContainment0.020.020.020.020.020.461.000.17ReactorCavity0.830.830.830.830.830.000.000.00PriortoSecuringSprays0.060.060.060.060.060.060.000.06RecirculationLowerContainment1.000.410.000.001.001.001.001.00RecirculationReactorCavity0.000.000.000.000.000.000.000.006.Thechemicalbump-upfactorsasafunctionofbreaksizeare:SmallBreaks:TruncatedExponentialDistributionMean=1.25Minimum=1 Maximum=15 .3MediumBreaks:TruncatedExponentialDistributionMean=1.50Minimum=1 Maximum=18 .2LargeBreaks:TruncatedExponentialDistribution 55 Mean=2.00Minimum=1 Maximum=24 .07.The"berpenetrationfunctionparametersare:FractionofSheddableDebris:UniformDistributionMin=0.00956Max=0.02720SheddingRate(1/min):UniformDistributionMin=0.008236Max=0.054600PerGramofDebris:UniformDistributionMin=0.000339Max=0.003723FitCutPoint(g):UniformDistributionMin=790 Max=880InitialUniformDistributionMin=0.656Max=0.706ExponentialRateConstant(1/g):ContinuousEmpiricalDistributionParameters: 0.0011254;0.10 0.0013078;0.45 0.0317870;0.108.Thesizeofzonesofin"uence(R/D)asafunctionofbreaksizeare:NUKON:17.0NUKON2:17.0Microtherm:28 .6RMI:1.0Lead:1.0ThermalWrap:17 .0IOZ:1.0Alkyd:1.09.Thetimestoturnonespraypump(minutes)asafunctionofbreaksizeare:SmallBreaks:0.0MediumBreaks:NormalDistribution 56 Mean=20.0StandardDeviation=5 .0LargeBreaks:NormalDistributionMean=20.0StandardDeviation=5 .010.Thetimestohotleginjection(minutes)asafunctionofbreaksizeare:SmallBreaks:UniformDistributionMin=345 Max=360MediumBreaks:UniformDistributionMin=345 Max=360LargeBreaks:UniformDistributionMin=345 Max=36011.Thestrainerbucklinglimitis9.35 ft H 2 O.12.Thewatervolumesinthepool(ft 3)asafunctionofbreaksizeare:SmallBreaks:UniformDistributionMin=43 , 464Max=61 , 993MediumBreaks:UniformDistributionMin=39 , 533Max=69 , 444LargeBreaks:UniformDistributionMin=45 , 201Max=69 , 26313.Thedebrisdensities(lb m/ft 3)are:LDFGFines:2.4LDFGSmall:2.4 LDFGLarge:2.4 MicrothermFilaments:2.414.Thetemperaturepro"les(F)asafunctionofbreaksizearegiveninFigure22. 57 98<55 <=8<85<98 =555<55=55>55?55855755955Temperature@E°XFTime@Gfter@Zreak@EQoursFTemperature@Profiles Small@_@Medium@ZreaksLarge@Zreaks(a)Full700HourPro"le 98<55 <=8<85<98 =5558<5<8=5=8>5>8Temperature@E°XFTime@Gfter@Zreak@EQoursFTemperature@Profiles Small@_@Medium@ZreaksLarge@Zreaks(b)36HourPro"leSeenByCASAGrandeFigure22:Temperaturepro"les(F)forsmallandlargebreaks.Panel(a)showsthefull700hourtemperaturepro"le,whilepanel(b)showsonlythe"rst36hours,whichisthestandardrunlength forCASAGrandescenarios.15.Thespraytransportfractionsfordebrisgeneratedoutsidethezoneofin"uence(unquali"edcoatings)aregiveninthe"fthrowofTable10.Weuseanupperlimitof15g/FAfortheboron"berlimitinScenario5becauseWCAP-16793[23]establishedaboron"berlimitof15g/FAasthemaximumvaluefortheSTPfueldesignbasedoncooling.Thereasonwedecreasebelowthatvalueinthenominalcase,andfurtherinScenario 4,istounderstandthesensitivitytothisvaluesinceitisnotthevalueforcorecoolingbutrather thevalueweuseforboricacidprecipitationfailure.At15g/FA,thereiselynoresistance 58 to"owasmeasuredinWCAP-16793.Incontrasttothesituationfortheboron"berlimit,forthestrainerbucklinglimit,thenominalvalueisconservative,containingasigni"cantsafetymargin. Hence,wedonotmakethislimitevenweakerinoursensitivityanalysis,andweonlyconsideran increaseinthelimitinScenario16.Scenario1:DecreaseLatentFiberTheamountoflatent"berinthepoolischangedto6.25ft 3,whereasthenominalvalueis12.5ft 3.Scenario2:IncreaseLatentFiberITheamountoflatent"berinthepoolischangedto25.0ft 3,whereasthenominalvalueis12.5ft 3.Scenario3:IncreaseLatentFiberIITheamountoflatent"berinthepoolischangedto50.0ft 3,whereasthenominalvalueis12.5ft 3.Scenario4:DecreaseBoronFiberLimitTheboron"berlimitinthecoreischangedto4.0g/FA,whereasthenominalvalueis7.5g/FA.Scenario5:IncreaseBoronFiberLimitITheboron"berlimitinthecoreischangedto15.0g/FA,whereasthenominalvalueis7.5g/FA.Scenario6:IncreaseBoronFiberLimitIITheboron"berlimitinthecoreischangedto50.0g/FA,whereasthenominalvalueis7.5g/FA.Scenario7:IncreaseDebrisTransportInsidetheZoneofIn"uenceThedebristransportfractionsforthisscenarioaregiveninTable11.NotethattheonlyincreaseddebristransportfractionsaretheblowdownandwashdowntransportfractionsforLDFG"nesand

small.59 Table11:Increaseddebristransportfractionsfordebrisgeneratedinsidethezoneofin"uence.DebrisTransportModel/LDFGLDFGLDFGMicrothermQualCoatCrudDebrisTypeFinesSmallLargeFinesFinesFinesBlowdownUpper1.001.000.220.700.700.70BlowdownLower1.001.000.000.300.300.30Washdown Inside1.001.000.000.530.530.53WashdownAnnulus1.001.000.000.470.470.47WashdownBCInside1.001.000.000.000.000.00WashdownBCAnnulus0.000.000.000.000.000.00PoolFill Sump0.020.000.000.020.020.02PoolFillInactive0.050.000.000.050.050.05 RecirculationLower1.000.640.001.001.001.00 Recirculation Inside1.000.640.001.001.001.00 RecirculationAnnulus1.000.580.001.001.001.00 ErosionSpray0.000.010.010.000.000.00 ErosionPool0.000.070.070.000.000.00Scenario8:DecreaseDebrisTransportInsidetheZoneofIn"uenceThedebristransportfractionsforthisscenarioaregiveninTable12.NotethattheonlydecreaseddebristransportfractionsaretheblowdownandwashdowntransportfractionforLDFG"nesand

small.Table12:Decreaseddebristransportfractionsfordebrisgeneratedinsidethezoneofin"uence.DebrisTransportModel/LDFGLDFGLDFGMicrothermQualCoatCrudDebrisTypeFinesSmallLargeFinesFinesFinesBlowdownUpper0.600.510.220.700.700.70BlowdownLower0.260.210.000.300.300.30Washdown Inside0.450.220.000.530.530.53WashdownAnnulus0.400.160.000.470.470.47WashdownBCInside0.000.230.000.000.000.00WashdownBCAnnulus0.000.000.000.000.000.00PoolFill Sump0.020.000.000.020.020.02PoolFillInactive0.050.000.000.050.050.05 RecirculationLower1.000.640.001.001.001.00 Recirculation Inside1.000.640.001.001.001.00 RecirculationAnnulus1.000.580.001.001.001.00 ErosionSpray0.000.010.010.000.000.00 ErosionPool0.000.070.070.000.000.00Scenario9:IncreaseChemicalPrecipitationTemperatureThechemicalprecipitationtemperatureischangedto160F,whereasthenominalvalueis140F.60 Scenario10:DecreaseTotalFailureFractionsforDebrisOutsidetheZoneofIn"uence(Unquali"edCoatings)Table13:Decreasedtotalfailurefractionsfordebrisgeneratedoutsidethezoneofin"uence(un-quali"edcoatings).DebrisTransportModel/EpoxyEpoxyEpoxyEpoxyEpoxyAlkydBakedIOZDebrisTypeFinesFineSmallLargeCurlsEnamelFinesChipsChipsChipsTotalFailureFraction0.800.800.800.800.800.430.430.92UpperContainment0.150.150.150.150.150.540.000.83LowerContainment0.020.020.020.020.020.461.000.17ReactorCavity0.830.830.830.830.830.000.000.00PriortoSecuringSprays0.060.060.060.060.060.060.000.06RecirculationLowerContainment1.000.410.000.001.001.001.001.00RecirculationReactorCavity0.000.000.000.000.000.000.000.00Scenario11:IncreaseChemicalBump-UpFactorThechemicalbump-upfactorsasafunctionofbreaksizearechangedasfollows:SmallBreaks:TruncatedExponentialDistributionMean=1.875Minimum=1 Maximum=30 .0MediumBreaks:TruncatedExponentialDistributionMean=2.25Minimum=1 Maximum=30 .0LargeBreaks:TruncatedExponentialDistributionMean=3.00Minimum=1 Maximum=30 .0Scenario12:IncreaseFiberPenetration(LowerEnvelope)The"berpenetrationfunctionparametersarenolongersampledfromdistributions.Instead,theyarenowconstantswiththefollowingvalues:FractionofSheddableDebris:0 .0196SheddingRate(1/min):0 .0538PerGramofDebris:0 .0003391FitCutPoint(g):880 Initial0.656 61 ExponentialRateConstant(1/g):0 .0013Scenario13:DecreaseSizeofZoneofIn"uence(ZOI)Thesizeofzonesofin"uence(R/D)asafunctionofbreaksizearechangedasfollows:NUKON:12.75NUKON2:12.75Microtherm:21 .45RMI:0.75Lead:0.75ThermalWrap:12 .75IOZ:0.75Alkyd:0.75Scenario14:IncreaseTimetoTurnOneSprayPumpThetimestoturnonespraypump(minutes)asafunctionofbreaksizearechangedasfollows:SmallBreaks:0.0MediumBreaks:NormalDistributionMean=1440.0StandardDeviation=5 .0LargeBreaks:NormalDistributionMean=1440.0StandardDeviation=5 .0Scenario15:IncreaseTimetoHotLegInjectionThetimestohotleginjection(minutes)asafunctionofbreaksizearechangedasfollows:SmallBreaks:450MediumBreaks:450 LargeBreaks:450Scenario16:IncreaseStrainerBucklingLimitThestrainerbucklinglimitischangedto10.30 ft H 2O,whereasthenominalvalueis9.35 ft H 2 O.62 Scenario17:DecreaseWaterVolumeThewatervolumesinthepool(ft 3)asafunctionofbreaksizearechangedasfollows:SmallBreaks:UniformDistributionMin=39 , 191Max=56 , 720MediumBreaks:UniformDistributionMin=34 , 084Max=63 , 995LargeBreaks:UniformDistributionMin=39 , 478Max=63 , 540Scenario18:IncreaseWaterVolumeThewatervolumesinthepool(ft 3)asafunctionofbreaksizearechangedasfollows:SmallBreaks:UniformDistributionMin=48 , 737Max=67 , 266MediumBreaks:UniformDistributionMin=44 , 982Max=74 , 893LargeBreaks:UniformDistributionMin=50 , 924Max=74 , 986Scenario19:IncreaseDebrisDensitiesThedebrisdensities(lb m/ft 3)are:LDFGFines:3.0LDFGSmall:3.0 LDFGLarge:3.0 MicrothermFilaments:3.0Scenario20:DecreaseTime-DependentTemperaturePro"lesThemodi"edtemperaturepro"les(F)asafunctionofbreaksizearegiveninFigure23. 63 98<55 <=8<85<98 =555<55=55>55?55855755955Temperature@E°XFTime@Gfter@Zreak@EQoursFTemperature@Profiles Small@_@Medium@ZreaksLarge@ZreaksSmall@_@Medium@Zreaks@ELowFLarge@Zreaks@ELowF(a)Full700HourPro"le 98<55 <=8<85<98 =5558<5<8=5=8>5>8Temperature@E°XFTime@Gfter@Zreak@EQoursFTemperature@Profiles Small@_@Medium@ZreaksLarge@Zreaks Small@_@Medium@Zreaks@ELowFLarge@Zreaks@ELowF(b)36HourPro"leSeenByCASAGrandeFigure23:Temperaturepro"les(F)forsmallandlargebreaks.Panel(a)showsthefull700hourtemperaturepro"le,whilepanel(b)showsonlythe"rst36hours,whichisthestandardrunlength forCASAGrandescenarios. 64 Scenario21:IncreaseSprayTransportFractionforDebrisOutsidetheZoneofIn"u-ence(Unquali"edCoatings)Table14:Increasedspraytransportfractionsfordebrisgeneratedoutsidethezoneofin"uence(unquali"edcoatings).DebrisTransportModel/EpoxyEpoxyEpoxyEpoxyEpoxyAlkydBakedIOZDebrisTypeFinesFineSmallLargeCurlsEnamelFinesChipsChipsChipsTotalFailureFraction1.001.001.001.001.001.001.001.00UpperContainment0.150.150.150.150.150.540.000.83LowerContainment0.020.020.020.020.020.461.000.17ReactorCavity0.830.830.830.830.830.000.000.00PriortoSecuringSprays0.120.120.120.120.120.120.000.12RecirculationLowerContainment1.000.410.000.001.001.001.001.00RecirculationReactorCavity0.000.000.000.000.000.000.000.00A.5Step5:EstimatingModelOutputsunderNominalValuesofInputParametersTable15presentstheresultsfromrunningCASAGrandewithallparameterssetattheirnominalvalues(Scenario0),usingtheformulafor R CDfromequation(4),alongwithestimatesofsamplingerror.Weseethepointestimateofriskintermsof(events/CY)is1.817E-08,withlower andupper95%con"dencelimitsof1.626E-08and2.009E-08,respectively.Ifwetake1.00E-06 asathresholdofinterestthenourestimateof R CDisnotclosetothisthresholdwhenallinputparametersaresettotheirnominalvalues.Table15:ResultsforScenario0withallinputparameterssettotheirnominalvalues.The"rstcolumnisR CDinunitsofevents/CYwhiletheremainingcolumnscharacterizethesamplingerror.Mean95%CI95%CI95%CICIHW%RiskHalf-WidthLowerLimitUpperLimitofMean1.817E-081.914E-091.626E-082.009E-0810.53%A.6Step6:One-WaySensitivityAnalysis:SensitivityPlotsandTornadoDiagramsInthisstep,wepresentresultsassociatedwithrunningall22scenariosbothintabularformandusingatornadodiagram,andwefurtherpresentaone-waysensitivityplot.Webeginwith numericalresultsintabularformforboththeabsoluterisk(again,conditionalonbeinginthe statewithallpumpsworkinginallthreetrains)andfortheinwithrespecttoa perturbationoftheinputparametersrelativetothenominalparametervalues.Then,wepresentatornadodiagramcorrespondingtochangingthe15inputsaswedetailinSectionA.4.Finally, wepresentaone-waysensitivityplotfortheboron"berlimit,which,afterexaminingthetornado plotandtablesofresults,appearstobetheinputparametertowhichourestimateofis mostsensitive. 65 TablesofResultsTable16presentsthenumericalresultsforeachofthe22scenarios.Inthistable,wepresentourestimateofthe(MeanRisk)associatedwitheachscenario,andweprovide95%con"dence interval(CI)limitsassociatedwiththisestimate.Wealsopresenttheratioofthe95%CIhalf-width tothemeanasapercentage.Table16:estimates,andsamplingerror,forallscenarios.The"rsttwocolumnsprovidethescenarionumberandtheparameterbeingchanged;seeSectionA.4.Thethirdcolumnreportshowweanticipatethewillchange,andtheremainingcolumnsreadinthesamemannerasinTable15.

1. SensitivityExpectedMean95%CI95%CI95%CICIHW%MeasureDirectionRiskHalf-WidthLLULofMean 0BaselineNone1.817E-081.914E-091.626E-082.009E-0810.53%

1LatentFiberLow(6.25ft 3)Decrease1.905E-081.928E-091.712E-082.098E-0810.12% 2LatentFiberHigh(25ft 3)Increase1.669E-081.770E-091.492E-081.846E-0810.61% 3LatentFiberVeryHigh(50ft 3)Increase3.394E-081.447E-081.947E-084.840E-0842.63% 4BoronFuelLimit(4.0g/FA)Increase1.690E-061.146E-065.445E-072.836E-0667.79% 5BoronFuelLimit(50g/FA)Decrease1.308E-081.412E-091.167E-081.449E-0810.80% 6BoronFuelLimit(15g/FA)Decrease1.329E-081.415E-091.188E-081.471E-0810.65% 7DebrisTransportInsideZOIHighIncrease7.896E-082.250E-085.645E-081.015E-0728.50% 8DebrisTransportInsideZOILowDecrease1.241E-081.493E-091.092E-081.390E-0812.03% 9ChemicalTempHighIncrease1.905E-081.937E-091.712E-082.099E-0810.17% 10TotalFailure%OutsideZOILow(80%)Decrease1.770E-081.878E-091.582E-081.958E-0810.61% 11BumpFactorHighIncrease2.287E-082.024E-092.085E-082.490E-088.85% 12PenetrationLowEnvelopeIncrease1.552E-071.696E-081.382E-071.721E-0710.93% 13ZOISizeSmallDecrease6.795E-098.275E-105.967E-097.622E-0912.18% 14Turn1SprayLongerDecrease1.569E-081.763E-091.393E-081.745E-0811.23% 15HotLegInjectionLongerIncrease1.962E-081.954E-091.766E-082.157E-089.96% 16StrainerLimitHigherDecrease1.639E-081.801E-091.459E-081.819E-0810.99% 17WaterVolumeLowIncrease2.001E-082.027E-091.798E-082.203E-0810.13% 18WaterVolumeHighDecrease1.655E-081.776E-091.477E-081.833E-0810.73% 19DebrisDensityHighIncrease2.567E-082.353E-092.331E-082.802E-089.17% 20TemperaturePro"lesLowIncrease1.963E-081.991E-091.764E-082.162E-0810.14% 21SprayTransport%OutsideZOIHigh(12%)Increase1.798E-081.914E-091.606E-081.989E-0810.65%Table17focusesontheandtheratiosoftheunderthenominalparametervaluesandundertheperturbedparametervalues;i.e.,wereport RCD/R 0 CDinthethirdcolumn(Ratio)and R CD=RCDR CDinthefourthcolumn(Mean),where R 0 CDisthepointestimateofunderthenominalscenarioand RCDisthatundertheperturbationscenarios. 66 Table17:estimatesforallscenariosincomparisonwiththenominalcase.ThiscomparisonisperformedviatheratioandtheTheCIstatementsarefortheandthe "nalcolumnindicateswhethertheisstatisticallysigni"cantata95%con"dencelevel.

1. SensitivityRatioMean95%CI95%CI95%CISigMeasureHalf-WidthLLUL?

0Baseline1.000.00E+000.00E+000.00E+000.00E+00No 1LatentFiberLow(6.25ft 3)1.058.76E-105.37E-103.39E-101.41E-09Yes 2LatentFiberHigh(25ft 3)0.92-1.49E-098.00E-10-2.29E-09-6.87E-10Yes 3LatentFiberVeryHigh(50ft 3)1.871.58E-081.45E-081.27E-093.02E-08Yes 4BoronFuelLimit(4.0g/FA)93.011.67E-061.15E-065.24E-072.82E-06Yes 5BoronFuelLimit(50g/FA)0.72-5.10E-091.09E-09-6.18E-09-4.01E-09Yes 6BoronFuelLimit(15g/FA)0.73-4.88E-091.08E-09-5.96E-09-3.80E-09Yes 7DebrisTransportInsideZOIHigh4.346.08E-082.22E-083.86E-088.30E-08Yes 8DebrisTransportInsideZOILow0.68-5.76E-091.05E-09-6.81E-09-4.72E-09Yes 9ChemicalTempHigh1.058.78E-104.22E-104.56E-101.30E-09Yes 10TotalFailure%OutsideZOILow(80%)0.97-4.73E-103.07E-10-7.80E-10-1.66E-10Yes 11BumpFactorHigh1.264.70E-099.52E-103.75E-095.65E-09Yes 12PenetrationLowEnvelope8.541.37E-071.69E-081.20E-071.54E-07Yes 13ZOISizeSmall0.37-1.14E-081.52E-09-1.29E-08-9.86E-09Yes 14Turn1SprayLonger0.86-2.49E-097.02E-10-3.19E-09-1.78E-09Yes 15HotLegInjectionLonger1.081.44E-092.46E-09-1.02E-093.90E-09No 16StrainerLimitHigher0.90-1.78E-096.57E-10-2.44E-09-1.13E-09Yes 17WaterVolumeLow1.101.83E-095.45E-101.29E-092.38E-09Yes 18WaterVolumeHigh0.91-1.62E-096.87E-10-2.31E-09-9.38E-10Yes 19DebrisDensityHigh1.417.49E-091.36E-096.14E-098.85E-09Yes 20TemperaturePro"lesLow1.081.46E-095.51E-109.05E-102.01E-09Yes 21SprayTransport%OutsideZOIHigh(12%)0.99-1.96E-102.08E-10-4.04E-101.25E-11NoTornadoDiagramandAnalysisBecausetheCDfrequenciesweestimatearesosmall,itisusefultopresentatornadodiagramandone-waysensitivityplotfortheratiosonalogarithmicscale;seealsoourdiscussionsurroundingTable17.Figure24isatornadodiagramforthe15inputparameterswevariedforthissensitivity study.Becausewereportresultsfortheratio RCD/R 0 CD,iftheratiohasvalue10,itmeansthatthepointestimateoftheundertheperturbedscenariois10timesgreaterthanthatunder thenominalscenario.TheCIboundsfortheriskratiosarecalculatedusingequation(10).In whatfollowsweexaminesixfactorstowhichtheestimateofseemstohavethegreatest sensitivity.Changesintheotherinputparametersleadtomoremodestchangesinthe estimate,withpercentageoflessthan30%. 67 56<5<655<5655<55655 <`555655Zoron@Xuel@Limit@E?65@gHXG@B@85@gHXGF Penetration@Low@UnvelopeDebris@Transport@Inside@jOI jOI@Size@Small Latent@Xiber@E76=8@ftq>@B@85@ftq>F Debris@Density@Qigh Zump@Xactor@Qigh^ater@xolumeTurn@Off@<@Spray@LongerStrainer@Limit@Qigher Temperature@Profiles@LowQot@Leg@In{ection@LongerChemical@Temp@QighTotal@Xailure@Y@Outside@jOI@Low@E:5YFSpray@Transport@Y@Outside@jOI@Qigh@E<=YFRatio@of@Risk@under@the@Scenarios@to@Risk@under@Nominal@Parameter@xaluesScenario@DescriptionsTornado@DiagramA@RiskDecreased@Parameter@xaluesIncreased@Parameter@xaluesIncreasing RiskDecreasing RiskFigure24:Tornadodiagraminlogspaceforratiosoftheestimates(risk);i.e.,weplot RCD/R 0 CD,alongwiththecorrespondingcon"denceintervalsforeachendpointofthehorizontalbars.Valuesoftheratiothatexceedonecorrespondtoanincreaseinriskrelativetothenom-inalcase.Thescenarionumbersinthe"rstcolumnofTables16and17mapthebriefscenario descriptionsheretothericherscenariodescriptionsinSectionA.4.WeseefromthetornadodiagraminFigure24thattheboron"berlimitappearstobethefactortowhichourestimateofismostsensitive.Increasingthelimitfromitsnominal valueof7.5g/FAshoulddecreasetheanddecreasingthelimitshouldhavetheoppositeThisholdsbecauseunderalargerlimit,more"bercanpenetratethestrainerwithoutthe simulationmodeldeclaringafailure.Increasingthelimitfrom7.5g/FAto15.0g/FA decreasesbyabout27%.Increasingthevaluefurtherto50.0g/FAleadstolittlefurtherdecreaseinHowever,decreasingthevaluefrom7.5g/FAto4.0g/FAincreasesthepointestimateofto1.69E-06,largerthanthenominalpointestimatebyafactorof93.Thetornadodiagramsuggeststhenextperturbationtowhichtheestimateofismostsensitiveinvolveschangingthe"ltrationfunctiontoitslowerenvelopeusingtheestimatesprovided in[14]andreportedinSectionA.4.The"ltrationfunctionhowmuch"berpenetratesthe strainer.Modifyingthefunctionsothatlessmassis"lteredmeansthatmoremasspenetrates,and hence,weanticipatethiswillincreaseAtthislowerenvelope,theestimateincreases byafactorof8.5to1.55E-07.ThetransportmatrixfordebrisinsidetheZOIgovernstheamountofeachtypeofdebris 68 transportedtothesump.Whenthismatrixhassmallertransportfractions,wewouldexpecttheestimatedtodecreasebecauselessdebrisreachesthestrainer,andlargertransportfractions shouldhavetheoppositeUndertheperturbationtosmallertransportfractionsspeci"edin SectionA.4,thepointestimateofdecreasesbyabout32%to1.24E-08,andwithmore debristransportedtothestrainertheestimategrowsbyafactorof4.34to7.9E-08.Next,weexaminetheofthesizeoftheZOI.WeregardournominalestimatesoftheZOIsizeasconservative(largerZOIthanexpected),andsoweexaminetheresultofreducingthesizeof theZOI.AsmallerZOImeanslessdebriswillbegenerated,whichshouldreduceReducing thesizeoftheZOIasspeci"edinSectionA.4,decreasesbyabout63%to6.8E-09.Wenowexaminetheoflatent"berinthesump.Weanticipatethatincreasingtheamountoflatent"berinthesumpwillincreaseasmore"berreachesthestrainerandcanpenetrate tothecore.AsthetornadodiagramandTable17indicate,increasingtheamountoflatent"ber from12.5ft 3to50.0ft 3leadstoan87%increaseinHowever,moremodestchangesinlatent"berproducecounterintuitiveresults,asshowninTable17.Adecreaseintheamountoflatent "berfrom12.5ft 3to6.25ft 3leadstoa5%increaseintheestimate,andanincreaseintheamountoflatent"berfrom12.5ft 3to25.0ft 3leadstoa8%reductionintheestimate.Thereasonforthesecounterintuitiveresultsisasfollows.CASAGrandeassumesthatsomefractionof latent"berisdepositedonthescreenwhenthesimulationmodelisinitialized.Thislatent"beris

noteligibletopenetratethescreen,althoughitiseligibletopenetratebyshedding.Ofcourse,inrealitysomeofthis"berwillpenetratethescreen,andhencetheSTPTechnicalTeammaysuggest aminormodi"cationtoCASAGrandeinthisvein.Increasingthedebrisdensitiesforlow-density "berglassandMicrotherm"lamentsleadstoa41%increaseintheestimatetoavalueof

2.6E-08.Eventhoughthechangeissmallinmagnitude,weclosethissectionbydiscussingacounter-intuitiveresult.IncreasingthespraytransportfractionfordebrisoutsidetheZOI(unquali"ed coatingsfailure)counterintuitivelyresultsinelynochangeoraslightreductionindebris bedheadlossandconsequentlyverylittlechangetoThisresultiscausedbyareduction intheoverall(composite)surface-area-to-volumeratio(S v)ofthedebrisbedwhenspraytransportappliedtofailedcoatingsisincreasedasasingleparameterinScenario21.Theiscausedby competingratiosofconstituentsintheweightedaverage S v.Speci"cally,enamelcoatingshavealargeinventoryandasmalldiameter,sotheconstituent S vforenamelislarge,butenamelcoatingswereassignedaspraytransportfractionofzero.Asquantitiesofotherunquali"edcoatingtypes increasewithincreasingspraytransportfraction,therelativeproportionofenameldecreases,re-sultinginaloweraggregate S v.Theisobservedregardlessoftheweightingschemechosen for S v,andboththemagnitudeanddirectionofthedependsontherelativeinventoriesandparticlediametersthatarespeci"edforthecoatingsdebristypes.Intreatmentofheadlossthrough porousmedia,theparameter S vrepresentsthetotalsurfaceareainsideofthebedthatcaninducedragontheinternal"ow.Additionofanymaterialinsideofthesamebedthicknessshouldboth

increase S vanddecreaseporosity.Theunusualdependenceofthecomposite S vonrelativedebris 69 quantitiesandcharacteristicssuggeststhattraditionaluseofaparticulate-weightedbedpropertyisnotappropriate.Ifacompositeparameterisneeded,totalavailabledragareashouldbeaveraged overthespatialdimensionsofthebedandnotoveraggregatepropertiesofthedebriselements themselves.One-WaySensitivityPlotandAnalysisGiventhesigni"cantoftheboron"berlimitontheestimate,weexplorethissensitivityfurtherviaaone-waysensitivityplot.Table18containstheresultsofrunningthesimulationwith boron"berlimitvaluesof4.0,4.5,5.0,5.5,6.0,6.5,7.5,and8.5(g/FA).Figure25isaone-waysensitivityplotoftheestimateversusboron"berlimitoverthisrange.Aswiththetornado diagram,wepresenttheratiooftheestimateforeachofthesevaluestotheestimate atthenominalvalueof7.5g/FA,andwepresenttheseratiosonalogscale.TheCIboundsfortheriskratiosarecalculatedusingequation(10).Weemploycommonrandomnumbersinthe simulationrunsacrosstheset"berlimitvalues.Table18:(MeanRisk)asafunctionoftheboron"berlimit.BoronFiberLimit(g/FA)MeanRisk95%CIHWRatio 4.01.690E-061.146E-0693.01 4.55.860E-078.359E-0732.24 5.01.059E-075.789E-085.83 5.56.242E-083.961E-083.43 6.03.699E-082.038E-082.04 6.52.050E-081.999E-091.13 7.01.931E-081.950E-091.06 7.51.817E-081.914E-091.00 8.01.700E-081.790E-090.94 8.51.658E-081.729E-090.91 70 56<<65<565<5565<55565>65>68?65?68865868765768965968:65:68;65 Risk Ratios Zoron@Xuel@Limit@E g/FA FSensitivity@PlotA@Common@Random@Numbers Mean@Risk Risk@}@<65UB7Increasing RiskDecreasing RiskFigure25:Sensitivityplotforboron"berlimit(g/FA).FromFigure25weseethereislittlechangeintheestimateasthe"berlimitrangesfrom6.5g/FAto8.5g/FA.However,theestimategrowsquicklyaswedecreasethe"berlimitfrom6.5g/FA.AnalysisofAmountofFiberPenetrationUsingPerturbedFiltrationFunctionInadditiontoexaminingtheestimateassociatedwitheachofthescenariosdescribedabove,anotherperformancemeasureofinterestistheamountof"berpenetratingthecore(g/FA)fortscenarios.Inparticular,itisinterestingtocomparetheaverageamountof"berpenetrating thecoreusingthenominal"ltrationfunction(scenario0),andusingthelowerenvelopeofthe "ltrationfunction(scenario12).InTable19,wepresenttheresultsofastatisticalanalysison"ber penetrationforthesetwocases.Whenusingthenominal"ltrationfunction,theaverageamountof "berpenetratingthestrainerinCASAGrandeis0.318g/FA,andwhenusingthelowerenvelopeofthe"ltrationfunction,thissamevalueis0.648g/FA.Weseethatapproximatelytwiceasmuch"berpenetrateswhenusingthelowerenvelope"ltrationfunction.ThefarrightcolumnofTable19 presentsstatisticalinformationaboutthebetweenthesetwoscenariosintermsof"ber penetration.Wecanseethea95%con"denceintervalonthemeandoesnotinclude zero,whichindicatestheisstatisticallysigni"cant,andofcourse,twiceasmuch"ber penetratingthestrainerisalsosigni"cantfromapracticalperspective. 71 Table19:Statisticalanalysisof"berpenetrationunderthenominalsettingsoftheparametersandwhenweusethelowerenvelopeofthe"ltrationfunction. Measure/CaseNominalLowerEnvelope Mean0.3180.6480.330Variance0.0630.2310.289StandardDeviation0.2510.4810.538NumberofObservations308308308Con"denceLevel0.950.950.95CIHalf-Width0.0280.0540.060CILowerLimit0.2900.5950.270CIUpperLimit0.3460.7020.390p-value--3.673E-23Signi"cant--Yes SummaryThisappendixhasfocusedonidentifyingtheinputparameterstowhichtheperformancemeasureof(changeincoredamagefrequencyduetoGSI-191issues)ismostsensitive.Ingeneral,we estimatebycouplingconditionalfailureprobabilities,asestimatedbytheCASAGrande simulationmodel,with:(i)thefrequencyofsmall,medium,andlargeLOCAevents,and(ii) theprobabilitymassfunctiongoverningtheplanthavingaccesstoasetofECCSpumps.Inthe analysiswepresentedhere,wehaveassumedthatthemostlikelypump-statecase,inwhichthe planthasaccesstoallpumps,occurswithprobabilityone.Forthissensitivityanalysis,theSTPTechnicalTeamselectedatotalof15inputparameterstotheCASAGrandesimulationmodel.(Wenotethatsomeparametersactuallycorrespond toacollectionofparameters;e.g.,wesimultaneouslychangeasetofdebristransportfractions.) NominalvaluesfortheseparameterscorrespondtotheanalysisperformedinVolumes2and3 [22,12].Alongwiththisnominalscenario,21furtherscenarioscorrespondedtochangingthe valuesofthese15parameters,oneatatime.Someparameterswerechangedinonlyonedirection, andotherparameterswerebothincreasedanddecreased.Withtherangesfortheseparameters inhand,weconstructedatornadodiagramcharacterizingthesensitivityoftochangesin theinputparameters.Keytoouranalysisisthattheperturbationstothese15inputparameters arecommensurate,meaningthattheyrepresentchangestothenominalcasethathavecomparable likelihood,asjudgedbytheSTPTechnicalTeam.Ourestimateismostsensitivetothreeparametersthatconcern:(i)howmuchdebrisisrequiredtotriggeranin-vesselfailure(boron"berlimit),(ii)thefractionofdebristhatpenetrates thesumpstrainer("berpenetrationfunction),and(iii)thefractionofdebrisofttypesthat istransportedfromtlocationsduringtoperationalphases(debristransportfractions inZOI).Theoftheboron"berlimitexceedsthatofthenextmostsensitiveparameterbyan orderofmagnitude,andsoweexaminedversustheboron"berlimitinfurtherdetailviaa 72 one-waysensitivityplot.Thegrowthinismodestaswedecreasetheboron"berlimitfrom7.5gramsperfuelassembly(g/FA)to6.5g/FA,butthenweseesharpgrowthinaswefurtherdecreasethislimit.Theappendixofthisreportappliestheinitialstepsofthesensitivityanalysisprocedurewepropose.Additionalanalysiswillbecarriedout.Wewillseektounderstand,conditionalupona sumporboron"berlimitfailureoccurring,whichweldlocationsaremostlikelytohaveexperienced abreak.Wewillformaspiderplot(step8)forthethreeorfourmostsensitiveparameters.We willconstructameta-modelofthetypeindicatedinstep10ofourframework.And,weintendto includefurtherpumpstates,beyondthemostlikelystateconsideredhere. 73 A Practical Guide to Sensitivity Analysis of a Large-Scale Computer Simulation Model David Morton Jeremy Tejada Alex Zolan The University of Texas at Austin February 2014

Contents *Background on applying sensitivity analysis to CASA Grande simulation model

• Practical step-by-step guide to sensitivity analysis: 10-step process
• Illustrative example
• Results from CASA Grande Background
• Distinguish three sources of error:

-sampling-based errors due to Monte Carlo simulation -errors due to uncertainty in model input -errors due to a lack of fidelity of the model

• Include sampling errors via error bars
• Use UQ plots to characterize second uncertainty
• We do not address model uncertainty here
• Use common random numbers (CRNs) to reduce the variance when comparing performance measures 10-Step Sensitivity Analysis Process
• Step 1: Define the Model
• Step 2: Select Outputs of Interest
• Step 3: Select Inputs of Interest
• Step 4: Choose Nominal Values and Ranges for Inputs
• Step 5: Estimate Model Outputs under Nominal Input Values
• Step 6: One-Way Sensitivity Analysis: Sensitivity Plots & Tornado Diagrams *Step 7: One-Way Sensitivity Analysis: UQ Plots
• Step 8: One-Way Sensitivity Analysis: Spider Plots
• Step 9: Two-way Sensitivity Analysis: Two-way Sensitivity Plots
• Step 10: Metamodels & Design of Experiments Step 1: Define the Model
• 1,2,34 : failure rates
• t 0 : desired lifetime of system
• : time required to perform PM on component 3
• k 3/k Step 2: Select Outputs of Interest
• Step 3: Select Inputs of Interest
• BASE Option:

-1,2,34 : failure rates -t 0 : desired system lifetime

*PM Option: 

-Base option parameters plus:

- : time required to perform PM on component 3 

-k : PM benefit factor

• Component 3 3 /k , where Step 4: Choose Nominal Values and Ranges for Inputs $&(

Step 5: Estimate Outputs under Nominal Values of Input Parameters Step 6: 1-Way Sensitivity Analysis: Tornado Diagrams Step 6: 1-Way Sensitivity Plots Step 6: 1-Way Sensitivity Plots Step 6: 1-Way Sensitivity Analysis: Tornado Diagrams Step 7: 1-Way Sensitivity Analysis: Uncertainty Quantification Plots Step 8: 1-Way Sensitivity Analysis: Spider Plots Step 8: 1-Way Sensitivity Analysis: Spider Plots Step 9: Two-way Sensitivity Analysis: Two-way Sensitivity Plots Step 10: Metamodels & Design of Experiments

• So far, change one or two input parameters at a time
• Metamodel (aka, response surface or surrogate model) built on an experimental design, better captures interaction effects
• A parsimonious metamodel is a polynomial regression model of low degree

Summary *Proposed a 10-step sensitivity analysis procedure and illustrated ideas on a simple example

• Recommended using tornado diagrams as initial tool for assessing the input parameters to which output is most sensitive
• Recommended using sensitivity plots, UQ plots, spider plots, and metamodels for a richer exploration of model sensitivity Appendix Sensitivity Analysis for STP GSI-191 Step 1: Define the Model
• We wont detail CASA Grande here (Volume 3)
• Use CASA Grande to estimate probability of sump failure and boron fiber limit failure, conditional on small, medium & large breaks
• Estimate change in core damage frequency (CDF) in events/year due to GSI-191 issues using these failure probabilities and link to PRA
• All results are conditional on all pumps working Step 2: Select Outputs of Interest
• Change in core damage frequency (CDF) *Sometimes, we report ratio of CDF estimate for a scenario to CDF estimate for baseline and call this the risk ratio
• Use stratified sampling on initiating frequency
• Use IID replications within each cell of stratification
• Use common random numbers across scenarios; i.e., use CRNs across specified changes in input parameters Step 2: Outputs: Estimating CDF IndicesandSets:

i=1 ,...,Findexforcellsstratifyingfrequencyreplications k=1 ,...,Nindexforsetofpumpstates Events:SL,ML,LLsmall,medium,largeLOCA PS kpumpsinstate k F iinitiatingfrequencyincell i Ssumpfailure Bboron"berlimitfailure CDcoredamageParameters: f SL ,f ML ,f LLfrequency(events/CY)ofasmall,medium,largeLOCA P (PS k)probabilitymassof PS k P (F i)probabilitymassof F iP (SlLOCA, F i,PS k)estimateofprobabilityof SgivenLOCA=SL,ML,orLL , F i , PS kP (BlLOCA, F i,PS k)estimateofprobabilityof BgivenLOCA=SL,ML,orLL , F i , PS k RBASEnon-GSI-191coredamagefrequency(events/CY) R CDestimateofcoredamagefrequency(events/CY) Step 2: Outputs: Estimating CDF

• We report results with:

-fSL , fML , fLL from Volume 2s Table 4-1 -P(all pumps working)= 1 -P(F i ): Bounded Johnson fit to NUREG-1829

• We form a variance estimate for the above estimator =R CDRBASE=Fi=1 Nk=1 P (F i)P (PS k)*f SL*P (SlSL, F i,PS k)+f SL*P (BlSL, F i,PS k)+f ML*P (SlML, F i,PS k)+f ML*P (BlML, F i,PS k)+f LL*P (SlLL, F i,PS k)+f LL*P (BlLL, F i,PS k)

Step 3: Select Inputs of Interest

• Amount of Latent Fiber in Pool:

Existing dust/dirt in containment, based on plant measurement. Assumed to be in the pool at start of recirculation, uniformly mixed. During fill up, latent debris available to penetrate sump

screen. *Boron Fiber Limit: Refers to threshold where boron precipitation occurs for cold leg breaks. Fiber limit comes from vendor testing that shows no pressure drop occurs with full chemical effects. Assume all fiber that penetrates sump screen deposits uniformly on core.

*Debris Transport Fractions in ZOI:

Refers to three-zone ZOI debris size distribution. Each insulation type has characteristic ZOI divided in three sections to account for type of damage within each zone.

Step 3: Select Inputs of Interest

• Chemical Precipitation Temperature: CASA Grande assumes that, once a thin bed of fiber forms on strainer, chemical precipitation bump up

factors apply when pool temperature reaches precipitation temperature.

*Total Failure Fraction for Debris Outside the ZOI: CASA Grande uses table of total failure fractions applied to transport logic trees. Fraction of each type (fiber, paint and coatings, etc.) that passes to the pool are used 

to understand what is in the pool as a function of time during recirculation. Total failure fraction multiplies total inventory of unqualified coatings.

• Chemical Bump Up Factor:

Used as a multiplier on conventional head loss calculated in CASA Grande. Multiplier is applied if thin bed is formed and pool temperature is at or below precipitation temperature.

Step 3: Select Inputs of Interest

• Fiber Penetration Function:

Fraction of fiber that bypasses the ECCS sump screen as a function of the amount of fiber on the screen.

*Size of ZOI:

ZOI defined as direct function (multiplier) of break size and nominal pipe diameter; e.g., for NUKON fiber, ZOI is 17 times break diameter. ZOI is spherical unless break is not DEGB, in which case it is hemispherical. Truncated by any concrete walls within the ZOI.

*Time to Turn Off One Spray Pump:

If three spray pumps start, then by procedure one is secured. Time to secure the pump is governed by operator acting on the conditional action step in procedure.

Step 3: Select Inputs of Interest

• Time to Hot Leg Injection: Similar to the spray pump turn off time, the time to switch one or more trains to hot leg injection operation is governed

by procedure.

*Strainer Buckling Limit: Limit is the differential pressure across ECCS strainer at which strainer is assumed to fail mechanically. This limit is based on engineering calculations that incorporate safety factor. 

• Water Volume in the Pool:

Depending on break size, amount of water in pool, as opposed to amount in RCS and other areas in containment, varies. Smaller breaks tend to result in less pool volume than larger

breaks. Step 3: Select Inputs of Interest

• Debris Densities:

Depends on amount and type of debris that arrives in pool. These densities are used in head loss correlations to calculate, for

example, debris volume.

*Time Dependent Temperature Profiles: Temperature of water in sump affects air release and vaporization during recirculation. Time-dependent temperature profile comes from coupled RELAP5-3D and MELCOR 

simulations depending on break size.

*Spray Transport Fractions for Debris Outside ZOI: CASA Grande uses a table of failure fractions applied to transport logic trees. Fractions of each type of debris that passes to pool are used to understand what is in pool as function of time during recirculation. The spray failure fraction is 

fraction of failed coatings that wash to pool during spray operation. Step 4: Nominal Values and Ranges for Inputs &'( .

@

$30&@ 1 Step 5: Estimate Outputs Under Nominal Values of Inputs $ Step 6: One-Way Sensitivity Analysis &&&&7

7'

75 7$ 7 ,

$

$

7 Step 6: One-Way Sensitivity Analysis: Sensitivity Plots & Tornado Diagrams 1/4J897.J~1<;A~~!LJ!J"7 Step 6: One-Way Sensitivity Analysis: Sensitivity Plots & Tornado Diagrams (}}