ML18173A253

OEDO-18-00198: Response to the Office of the Inspector General'S Evaluation of Proposed NRC Modifications to the Probabilistic Risk Assessment Process (OIG-17-A-26)

ML18173A253
Person / Time
Issue date:	06/29/2018
From:	Brian Holian Office of Nuclear Reactor Regulation
To:	Baker B NRC/OIG/AIGA
Montecalvo M, NRR/DRA, 415-1678
Shared Package
ML18115A132	List: ML18173A253
References
OEDO-18-00198, OIG-17-A-26
Download: ML18173A253 (33)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 June 29, 2018 MEMORANDUM TO: Dr. Brett M. Baker Assistant Inspector General for Audits Office of the Inspector General FROM: Brian E. Holian, Acting Director /RA/

Office of Nuclear Reactor Regulation

SUBJECT:

RESPONSE TO THE OFFICE OF THE INSPECTOR GENERALS EVALUATION OF PROPOSED NRC MODIFICATIONS TO THE PROBABILISTIC RISK ASSESSMENT PROCESS (OIG-17-A-26)

The staff has received the Office of the Inspector Generals (OIGs) report OIG-17-A-26, Evaluation of Proposed NRC Modifications to the Probabilistic Risk Assessment Process, dated September 21, 2017 (Agencywide Documents Access and Management System Accession No. ML17264A298). In this report, OIG recommends formally documenting evaluation results that will establish the agency position on the U.S. Nuclear Regulatory Commissions (NRCs) use of licensees probabilistic risk assessment (PRA) models, to include reliable, verifiable cost data.

The staff appreciates OIGs audit of the evaluation of the proposed NRC modifications to the PRA process. The enclosure provides the recommended report documenting the evaluation results. If you have any questions, please contact me at 301-415-1270. You may also contact Mr. Michael Franovich or Mr. Christopher Fong at the numbers listed below.

Enclosure:

NRC Staff Evaluation

cc: Chairman Svinicki Commissioner Baran Commissioner Burns Commissioner Caputo Commissioner Wright SECY CONTACTS: Michael X. Franovich, NRR/DRA 301-415-2884 Christopher J. Fong, NRR/DRA 301-415-8474

ML18173A253 *via email NRR-106 OFFICE NRO/DSRA* RES/DRA* NRR/DRA* OGC* NRR NAME KCoyne JNakoski MFranovich AGhosh BHolian DATE 5/16/18 5/11/18 06/12/18 6/21/18 6/29/2018 MAY 1, 2018 U.S. NUCLEAR REGULATORY STAFF EVALUATION OF USING LICENSEE PROBABILISTIC RISK ASSESSMENT MODELS TO REPLACE THE STANDARDIZED PLANT ANALYSIS RISK MODELS FOR REGULATORY DECISIONMAKING EXECUTIVE

SUMMARY

Internal and external stakeholders have raised questions on the continued maintenance and upgrades to the U.S. Nuclear Regulatory Commissions (NRCs) standardized plant analysis risk (SPAR) models. This discussion usually leads to questions on transitioning to a structure in which the NRC would rely on licensees probabilistic risk assessment (PRA) models instead of maintaining an NRC-specific SPAR model for each plant. This action has been proposed several times in the past, and the NRC has completed evaluations of different aspects of the proposals.

This report summarizes the NRCs evaluation of proposed options to modify certain regulatory processes that use PRA results. Specifically, it evaluates the option to transition from the use of the SPAR Model Program to reliance on licensee-developed PRA models to support regulatory decisions. This report provides background on the history of the NRCs use of PRA models; the NRCs development of the plant-specific SPAR models; industry PRA model development, maturity, and use; the NRCs use of licensee PRA models; and other Government agencies use of the risk and reliability assessment software platform (the Systems Analysis Programs for Hands-on Integrated Reliability Evaluations (SAPHIRE) computer code). The report also summarizes the results from an NRC effort in the 2006/2007 timeframe to evaluate the use of licensees PRA models for the significance determination process (SDP) in the Reactor Oversight Process (ROP). The most recent 2015/2016 effort by the NRC is also provided, along with a discussion and summary of a cost-benefit analysis (CBA) based on reliable, cost-verifiable data.

As noted in this report, the NRC relies on results from licensees PRA models for certain risk-informed regulatory applications (e.g., licensing) and uses the SPAR models for other regulatory applications (e.g., the significance determination process). Maintaining two models may, on the surface, seem like an inefficiency that could easily be eliminated. However, shifting the regulatory structure to use one model, especially a model that is developed and maintained by a licensee, presents many challenges. Ultimately, the Risk-Informed Steering Committee (RISC) decided to end the staffs evaluation of this option for the following reasons:

• lack of broad industry support for sharing licensees PRA models

• CBA results showing significant transition costs with some scenarios showing increased overall ongoing costs

• potential efficiency gains not achievable without considerable risk to the agency Collectively, these factors make it clear that, at this time, it is not appropriate to transition from NRCs SPAR models to the use of licensees PRA models. Based on the history, this topic is likely to surface again, and at a minimum each of these factors, in addition to the issues previously identified during the 2006/2007 effort, would need to be addressed to support a Enclosure

change in the use of NRC SPAR models for risk-informed decisionmaking. The purpose of this report is to review past efforts and provide a basis for the NRCs decision to continue to maintain independent plant-specific SPAR models at this time.

INTRODUCTION This report summarizes the NRCs evaluation of proposed options to modify regulatory processes that use PRA results-specifically, the evaluation of transitioning regulatory decisions from support by the NRCs SPAR Model Program to support by licensee-developed PRA models. This report provides background on the history of the NRCs use of PRA models; the NRCs development of the plant-specific SPAR models; industry PRA model development, maturity, and use; the NRCs use of licensee PRA models; and other Government agencies use of the risk and reliability assessment software platform (the SAPHIRE computer code). The report also summarizes the results from the NRC effort in 2006/2007 to evaluate the use of licensees PRA models for the SDP in the ROP. The most recent 2015/2016 effort by the NRC is also discussed, along with the reasons why the NRCs RISC decided to end the evaluation of this proposed change. The report also includes a discussion and summary of a CBA based on reliable, cost-verifiable data. Following is a brief discussion of a congressional inquiry on this topic and the Office of Inspector General (OIG) review. For the interested reader, the report includes a list of references and a bibliography with additional documentation related to the subject.

BACKGROUND PRA Use in Regulation Before establishing the policy on Use of Probabilistic Risk Assessment Methods in Nuclear Regulatory Activities in the Federal Register (Ref. 3), the NRC generally regulated the use of nuclear material based on deterministic approaches. Deterministic approaches to regulation consider a set of challenges to safety and determine how those challenges should be mitigated.

A probabilistic approach to regulation enhances and extends this traditional, deterministic approach by (1) allowing consideration of a broader set of potential challenges to safety, (2) providing a logical means for prioritizing these challenges based on risk significance, and (3) allowing consideration of a broader set of resources to defend against these challenges.1 The Federal Register notice establishing the Policy Statement states:

the Commission believes that an overall policy on the use of PRA in nuclear regulatory activities should be established so that the many potential applications of PRA can be implemented in a consistent and predictable manner that promotes regulatory stability and efficiency. This policy statement sets forth the Commissions intention to encourage the use of PRA and to expand the scope of PRA applications in all nuclear regulatory matters to the extent supported by the state-of-the-art in terms of methods and data. Implementation of the policy statement will improve the regulatory process in three areas: Foremost, through safety decision making enhanced by the use of PRA insights; through more efficient use of agency resources; and through a reduction in unnecessary burdens on licensees. Therefore, the Commission adopts the following policy 1 SECY-98-144, White Paper on Risk-Informed and Performance-Based Regulation, dated January 22, 1998, (Ref. 25) defined the terms and Commission expectations for risk-informed and performance-based regulation.

statement regarding the expanded NRC use of PRA: (1) The use of PRA technology should be increased in all regulatory matters to the extent supported by the state-of-the art in PRA methods and data and in a manner that complements the NRCs deterministic approach and supports the NRCs traditional defense-in-depth philosophy. (2) PRA and associated analyses (e.g., sensitivity studies, uncertainty analyses, and importance measures) should be used in regulatory matters, where practical within the bounds of the state-of-the-art, to reduce unnecessary conservatism associated with current regulatory requirements, regulatory guides, license commitments, and staff practices. Where appropriate, PRA should be used to support the proposal for additional regulatory requirements in accordance with 10 CFR 50.109 (Backfit Rule). Appropriate procedures for including PRA in the process for changing regulatory requirements should be developed and followed. It is, of course, understood that the intent of this policy is that existing rules and regulations shall be complied with unless these rules and regulations are revised. (3) PRA evaluations in support of regulatory decisions should be as realistic as practicable and appropriate supporting data should be publicly available for review. (4) The Commissions safety goals for nuclear power plants and subsidiary numerical objectives are to be used with appropriate consideration of uncertainties in making regulatory judgments on the need for proposing and backfitting new generic requirements on nuclear power plant licensees.

Additionally, on the subject of implications of the new Policy Statement, the notice stated:

This policy statement affirms the Commissions belief that PRA methods can be used to derive valuable insights, perspective, and general conclusions as a result of an integrated and comprehensive examination of the design of nuclear facilities, facility response to initiating events, the expected interactions among facility structures, systems, and components, and between the facility and its operating staff. The Commission also recognizes, and encourages, continuation of industry initiatives to improve PRA methods, applications and data collection to support increased use of PRA techniques in regulatory activities.

The NRC has used PRA methods to address complex safety issues and make risk-informed decisions, such as those involved in rules on station blackout, anticipated transients without scram, and pressurized thermal shock; to formulate the Backfit Rule; to set the priorities for addressing generic safety issues; and to prepare and evaluate responses to generic letters. In July 1998, the NRC issued the Standard Review Plan Section 19.0, Use of Probabilistic Risk Assessment in Plant-Specific, Risk-Informed Decisionmaking: General Guidance, (Ref. 10) and the associated Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, (Ref. 8) documenting general guidance on risk-informed decisionmaking for changes to the plant-specific licensing basis using PRA methods. Since then, the NRC has issued application-specific, risk-informed guidance documents for other programs (e.g., inservice testing, inservice inspection, technical specifications (TS)).

PRA methods contribute to risk-informed decisionmaking approaches in a variety of NRC processes including the ROP. The ROP was implemented in 2000 to provide an objective, risk-informed, performance-based, understandable, and predictable approach to the oversight of commercial nuclear power plant (NPP) performance. The ROP supports the NRCs strategic goals for safety and security and demonstrates the agencys values of openness and

effectiveness. Later sections of this report present more information on the ROP and use of PRA methods.

The NRC has also used PRA methods and risk-informed decisionmaking approaches to modify the regulations in Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Domestic Licensing of Production and Utilization Facilities. (Ref. 1) The intent of these modifications was to add, modify, or delete requirements so that the regulatory burden imposed by individual regulations is commensurate with the regulations importance to protecting public health and safety. One example of such a modification to existing regulations was the Commission adoption, in November 2004, of a new section (10 CFR 50.69, Risk-Informed Categorization and Treatment of Structures, Systems, and Components for Nuclear Power Reactors)

(Ref. 37). This section permits power reactor licensees and license applicants to implement an alternative regulatory framework with respect to special treatment, which refers to requirements that provide increased assurance beyond normal industrial practices that structures, systems, and components (SSCs) will perform their design-basis functions. In this framework, licensees using a risk-informed process for categorizing SSCs according to their safety significance can remove SSCs of low safety significance from the scope of certain identified special treatment requirements.

Regulatory applications involving modifications to the current licensing basis for an operating reactor licensed under 10 CFR Part 50 do not require the adoption of a risk-informed approach, or development of a PRA in support of the application. However, if a licensee chooses to adopt a risk-informed approach for changes to its licensing basis, decisions made by the NRC on the acceptability of those changes are strongly tied to the use of PRA methods as discussed, for example, in RG 1.174. One condition for using PRA results in a risk-informed regulatory application is that the PRA be of sufficient technical adequacy to support the specific decision.

The NRC presents its expectations and the staffs position on one acceptable approach for determining the technical adequacy of a PRA in RG 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities.

(Ref. 6) When used in support of an application, RG 1.200 obviates the need for an in-depth review of the base PRA by NRC reviewers. This contributes to a more efficient review process by allowing reviewers to focus on key assumptions and areas identified by peer reviewers as being of concern and relevant to the application. Specifically, RG 1.200 provides guidance on a definition of a technically acceptable PRA, the NRCs position on PRA consensus standards and industry peer-review program documents, demonstration that the base PRA used in support of regulatory applications is of sufficient technical adequacy, and documentation to support a regulatory submittal.

NRC Development and Use of SPAR Models The objective of the NRCs SPAR Model Program is to develop standardized risk analysis models and tools to support various regulatory activities, including the Accident Sequence Precursor (ASP) Program; the Incident Investigation Program described in Management Directive (MD) 8.3 (Ref. 26), and the SDP. The SPAR models have evolved from two sets of simplified event trees initially used to perform precursor analyses in the early 1980s into PRA models that more fully reflect, on a plant specific basis, the as-built and operated facility.

Plant-specific SPAR models are built with a standard modeling approach, using consistent modeling conventions that enable staff to easily use the models across a variety of U.S. NPP designs.

Software Platform and Scope of Models SPAR models are developed, modified, and quantified on a single probabilistic risk and reliability assessment software platform, the SAPHIREs computer code. Idaho National Laboratory (INL) developed the system for the NRC, which funds it. INLs primary role in the project is software development and quality assurance2 (QA). Although the NRC constitutes the main user of SAPHIRE, INL also plays an important role in supporting and training SAPHIRE users (e.g.,

other Federal Agencies, national laboratories, the private sector, foreign countries). INL maintains the NRC SPAR models on a secure Web site accessible to NRC users in various regulatory programs. Through a help desk function, INL frequently supports NRC users in modifying the models required for evaluating the risk significance of plant conditions or events.

Development of SAPHIRE began in the mid-1980s when the NRC began exploring two notions:

(1) that PRA information could be displayed and manipulated using the emerging microcomputer technology of the day and (2) the rapid advances in PRA technology required a relatively inexpensive and readily available platform for teaching PRA concepts to students.

SAPHIRE can be used to model a complex systems response to initiating events and to quantify associated consequential outcome frequencies or probabilities. Specifically, for NPP applications, SAPHIRE can identify important contributors to core damage (Level 1 PRA) and containment failure during a severe accident that leads to releases (Level 2 PRA). It can be used for a PRA where the reactor is at full power, low power, or shutdown conditions.

Furthermore, it can be used to analyze both internal and external initiating events and has special features for managing models such as flooding and fire. It can also be used in a limited manner to quantify risk, using PRA techniques, in terms of release consequences to the public and environment (Level 3 PRA). The SAPHIRE code was specifically designed to provide risk results in forms that are easy to use in NRC regulatory applications associated with reactor plant oversight (e.g., delta core damage frequency (CDF), conditional core damage probability).

SAPHIRE also includes a number of user friendly workspaces that are tailored to the needs of specific users, ranging from a simplified basic interface to new users to a more complex interface for experienced users.

The scope of every SPAR model includes logic modeling covering internal initiating events at power through core damage. The NRC is continuing to develop new SPAR modules for assessing plant risk from internal fires, external hazards (e.g., high wind and seismic events),

and for assessing accident progression following severe core damage. Todays SPAR models for internal events are far more comprehensive than their predecessors. For example, the revised SPAR models include improved loss-of-offsite power (LOOP) and station blackout models, an improved reactor coolant pump seal failure model, new support system initiating event models, state of the art common cause failure modeling, and updated estimates of accident initiator frequencies and equipment reliability based on recent operating experience data. The NRCs SPAR models are generally focused on at-power operations; however, low-power/shutdown (LPSD) models have been developed and are being used to support regulatory programs.

The SPAR models consist of a standardized set of plant-specific risk models that use the event-tree and fault-tree linking methodology. Although the SPAR models are plant-specific models, as stated above, they rely on a set of standardized modeling conventions (e.g., standard 2 The SAPHIRE quality assurance program is consistent with guidance contained in NUREG/BR-0167, Software Quality Assurance Program and Guidelines (Ref. 29)

naming conventions, modeling approaches, and logic structure). They employ a standard approach for event-tree development, as well as a standard approach for initiating event frequencies, equipment performance parameters, and human performance data. These input data can be modified to be more plant and event specific, when necessary. SPAR standardization is needed to allow agency risk analysts to efficiently use SPAR models for a wide variety of NPPs without having to relearn modeling conventions and basic assumptions.

Additionally, the standardization conventions used in SPAR support efficient updating of model information and allow risk assessments to be conducted across a range of plant models in an automated manner. Although the system fault trees contained in the SPAR models may not be as detailed as those in some licensee PRAs, in some cases, SPAR models may contain more sophisticated modeling, such as that for common-cause failures, support systems, and losses of offsite power. The staff maintains 76 SPAR models representing all 99 operating commercial NPPs.

Model Improvements, Quality Assurance, and Peer Review The staff initiated the Risk Assessment Standardization Project (RASP) in 2004. A primary focus of RASP was to standardize risk analyses performed in SDP, in ASP, and in the NRC Incident Investigation Program completed under MD 8.3. The project enhanced SPAR models to be more plant specific and documented consistent methods and guidelines for risk assessments of internal events during power operations; internal fires and floods; external hazards (e.g., seismic events and tornadoes); and internal events during LPSD operations. One outcome of the RASP was the development of the Risk Assessment of Operational Events Handbook (Ref. 24)

(commonly referred to as the RASP Handbook) and better alignment between the event assessment processes. The RASP Handbook is a living document that undergoes periodic revision based on guidance needs when new issues involving PRA techniques are identified.

The staff implemented a QA plan for the SPAR models in 2006. The main objective of the plan is to ensure that the SPAR models continue to represent the as-built, as-operated NPPs and continue to be adequate for assessing operational events in support of the staffs risk-informed activities. In addition to model development, the QA plan provides mechanisms for internal and external peer review, validation and verification, and configuration control of the SPAR models.

The staff has processes in place to verify, validate, and benchmark these models according to the guidelines and standards established by the SPAR Model Program. As part of this process, the staff reviews the SPAR models and results against the licensee PRA models, when applicable. The QA plan also provides a feedback process from the model users for error reporting, tracking, and resolution.

In 2010, the staff (with the cooperation of industry experts) performed a peer review of SPAR models for a representative boiling-water reactor and a representative pressurized-water reactor in accordance with American Society of Mechanical Engineers/American Nuclear Society RA-S-2008, Standard for Level-1/Large Early Release Frequency Probabilistic Risk Assessment for Nuclear Power Plant Applications, (Ref. 5) and RG 1.200. The peer review teams noted a number of strengths for the SPAR models:

• The SPAR model structure is robust and well developed.

• The SPAR model fault trees are streamlined with an appropriate level of detail for the models intended uses.

• The SPAR model structure and the SAPHIRE software are state of the technology.

• The SPAR models are an efficient method to develop qualitative and quantitative insights for risk-informed applications, SDP evaluations, inspections, event assessments, and model evaluations.

The peer review teams also noted enhancements that could be made to the SPAR models. The staff reviewed and prioritized the peer review comments to identify potential improvements to the SPAR models. Enhancements that improved the usability, capabilities, and technical adequacy of the models in a cost-effective manner received high priority, and the staff initiated projects to address these comments. Specific enhancements completed include structuring the SPAR model documentation to more closely align with the structure of the PRA standard, incorporating improved LOOP modeling, developing new support system initiating event models, and expanding the SAPHIRE Web site to better log and track model change requests. The staff completed all high-priority SPAR peer review enhancements for boiling-water and pressurized-water reactors by August 2015.

The SPAR models are generally used to categorize and prioritize operational events and conditions, including licensee noncompliance with existing regulations. Although the SPAR models are not maintained under an RG 1.200 program, the SPAR QA program and other process controls (such as internal and external reviews) ensure that SPAR-based analyses appropriately reflect the as-built, as-operated NPP.

Model Updates and Continuing Development Existing SPAR models for operating plants need to be updated regularly as a result of any significant plant changes that may affect the risk profile of the plant. In general, the staff goal is to perform significant updates to approximately 6 to 10 SPAR models per year. As SPAR models are updated, their documentation (i.e., the model report and the plant risk information eBook summary reports) is also updated to represent the latest PRA information included in each SPAR model. The staff also compares the SPAR model baseline results and licensee model results (when voluntarily submitted by the licensee). These comparisons include baseline CDF, conditional core damage probability for each initiator type, top cutsets, and importance measures. These comparisons help ensure that SPAR models and associated risk assessments that support the SDP are of high quality and reflect the as-built, as-operated plants.

To properly reflect and credit an important risk-informed change at several plants, the Office of Nuclear Regulatory Research (RES) recently updated all 15 SPAR models identified as having the new Westinghouse Generation III SHIELD reactor coolant pump shutdown seals.

Integration of this plant modification into SPAR models will enable the staff to appropriately consider the potential significant decrease in CDF associated with the installation of these seals as part of licensing application reviews and as part of the ROP.

SPAR models are routinely updated to support specific SDP or ASP activities. These more limited SPAR model updates are made by agency risk analysts, with assistance from INL when required. These updates are normally required to better model specific features of an operational event that are not normally captured in a base PRA or to reflect an enhanced understanding of the as-built, as-operated plant as a result of event followup activities.

The NRC is pursuing development of SPAR All-Hazard (SPAR-AHZ) models, which contain accident scenarios from all hazard categories (including seismic, high wind, and internal fire) applicable at a given site. This initiative will allow the NRC to better understand the complete risk profile of the plants to ensure efficient regulatory decisions. Currently, 22 of the 76 SPAR

models, representing 28 NPPs, include internal fire and external hazard groups. Further, a project has begun to credit equipment procured to comply with post-Fukushima orders (FLEX) in NRCs SPAR models.

Before new plant operation, the staff may perform risk assessments to inform potential risk-informed applications for combined licenses, focus construction inspection scope, or assess the significance of construction inspection findings. Once the plants begin operation, the staff will make independent assessments using SPAR models for the evaluation of operational findings and events similar to the assessments performed for current operating reactors.

There are currently five new reactor internal hazard SPAR models. These include one model for the Advanced Passive (AP) 1000, two advanced boiling-water reactor (ABWR) models (one for the Toshiba design and one for the General Electric-Hitachi design), one model for the U.S. Advanced Pressurized-Water Reactor, and one model for the U.S. Evolutionary Power Reactor. In addition to these internal events models, the Toshiba ABWR model contains a LPSD model. The AP1000 SPAR-AHZ model includes seismic, internal flooding, and internal fire models. The staff recently completed a LPSD model and developed a new severe accident model (Level-2 PRA model) for the AP1000 reactor design. The staff plans to continue developing new reactor SPAR models, including AHZ and LPSD models, as needed, to support licensing and oversight activities.

The staff continues to perform MELCOR analyses to investigate success criteria associated with specific Level 1 PRA sequences. In some cases, these analyses confirm the existing technical basis, and in other cases, they support modifications to increase the realism of the agencys SPAR models. The results of these studies are published as NUREGs and used to confirm specific success criteria for the corresponding suite of plants of similar design. They are also used to support application-specific consultation on the use of the SPAR models. These efforts directly support the agencys goal of using state-of-the-art tools that promote effectiveness and realism. The NRC communicates the project plans and results to internal and external stakeholders through mechanisms such as the Regulatory Information Conference and the industrys Modular Accident Analysis Program Users Group.

Industry Model Development, Maturity, and Use Licensees developed most U.S. PRAs in response to Generic Letter (GL) 88-20, Individual Plant Examination for Severe Accident Vulnerabilities10 CFR 50.54(f). (Ref. 11) GL 88-20 requested that licensees perform an individual plant examination (IPE) for severe accident vulnerabilities associated with internal hazards (including internal flooding hazards but not internal fire hazards). Supplement 4 to GL 88-20 (Ref. 12) asked licensees to perform an individual plant examination of external events (IPEEE) for severe accident vulnerabilities associated with seismic events, internal fires, high winds and tornadoes, external floods, transportation, and nearby facility accidents.

The use of industry PRA models for regulatory purposes expanded with the development and use of the Mitigating Systems Performance Index (MSPI) as an element of the ROP. One of the conditions agreed to by industry and the NRC before adoption of the index was that all plants should participate. Since development and use of the index requires a plant-specific PRA, in effect, this agreement required all licensees to have a PRA model. Before the implementation of the MSPI, the licensees had to demonstrate that their PRA models were of sufficient technical adequacy to support the MSPI application.

Licensees use a number of software platforms to support the development and maintenance of their PRA models. Most licensees use the Computer Aided Fault Tree Analysis System (CAFTA),3 but several options are available (e.g., one licensee uses SAPHIRE). Since the development of initial risk reviews by licensees in support of the IPE and IPEEE, licensees site-specific PRAs have grown more sophisticated. There have been advances in mathematical methods and greater detail in the modeling of the as-built, as-operated plant. NRC standards for adequacy in using the plant-specific PRAs for regulatory activities have also evolved. As discussed earlier in this document, licensee PRA models developed to support licensing-basis changes must display appropriate technical adequacy. One acceptable method of demonstrating technical adequacy is to meet the guidelines of RG 1.200. Figure 1, compiled for the May 11, 2017, Public Commission Meeting on Risk, shows the PRA modeling and RG 1.200 peer reviews completed as of December 2013. For context, not all licensees have internal fire, seismic, or external flooding PRAs; the figure represents those licensees who have the specified model and whether it is peer reviewed.

Figure 1. Percentage of industry PRA models that are peer reviewed (from the Public Commission Meeting on Risk, May 11, 2017)

All U.S. NPPs have Level 1 internal event PRAs and some have Level 2 assessments. Most of the current Level 2 assessments are limited in scope, as they focus on large early release frequency. Each holder of a combined license, under 10 CFR Part 52, Licenses, Certifications, and Approvals for Nuclear Power Plants, (Ref. 2) is required to develop a Level 1 and a Level 2 PRA before initial loading of the fuel. The PRA must cover those initiating events and modes for which NRC-endorsed consensus standards on PRA exist 1 year before the scheduled date for initial loading of fuel.

Some U.S. plant PRAs address seismic initiating events, while others use simplified approaches (e.g., seismic margins analyses) aimed at identifying vulnerabilities to satisfy the requirements of the IPEEE program without providing quantitative estimates of risk. In 2014, as part of its implementation of lessons learned from the 2011 events at the Fukushima Daiichi NPP, licensees submitted updated seismic hazard information in response to an NRC request for information pursuant to 10 CFR 50.54(f). Based on the findings of its review of the updated 3 CAFTA and RISKMAN are standard industry-used PRA computer software programs utilized by many licensees to develop their PRA models.

seismic hazard, the NRC has requested that 20 sites submit the results of a plant-specific seismic PRA and the insights related to updated earthquake risks.4 To support the adoption of programs under National Fire Protection Agency (NFPA)-805, Performance-Based Fire Protection for Light Water Reactor Electric Generating Plants, (Ref. 27) under 10 CFR 50.48(c),

28 sites (44 units) have developed fire PRA models. Only a few plants have performed PRAs for other external hazards (e.g., high winds, external flooding, accidental aircraft crashes). As discussed above, new reactors are required to address external hazards since the NRC staff has endorsed the latest combined American Society of Mechanical Engineers/American Nuclear Society standard on PRA.

Most of the current licensee PRAs consider only events occurring during full-power operation; just a few address events occurring during LPSD operation. Although the NRC staff has not yet endorsed the NRC consensus standards on LPSD operation, most new reactor designs have addressed these events in the PRAs using current best practices.

Since the inception of these techniques, NPP licensees have dramatically increased their use of PRA technologies to inform decisions made daily at their plants. Licensees use the results of their PRAs to inform their scheduling for maintenance and refueling outages, online workweeks, functional equipment group maintenance windows, emergent work, and plant modification approval lists, among other things. In the scheduling process for online and outage work, the plant staff bundles groups of maintenance activities together that make sense from a train or system perspective, while looking at the risk insights provided by the plant-specific PRA. When approving emergent work, licensees generally use an online risk tool (e.g., equipment out of service, Phoenix) to evaluate the effect of removing a piece of equipment from service on the overall risk profile of the plant. These tools provide a user-friendly interface that works with the background PRA software (e.g., SAPHIRE, CAFTA) so that a non-PRA professional, usually a senior reactor operator, can decide on approving or deferring work until a better time based on the risk profile. The ultimate decision on equipment maintenance lies with the approving individual, but the online risk tool provides valuable information to inform the level of risk management actions needed. These actions can vary from protecting other trains of equipment to designating a higher level of management oversight and approval of the maintenance configuration. The plant PRA is also used to inform decisions on which modifications will be approved at the plant. The plant PRA staff can perform analyses to evaluate the potential risk reductions provided by various proposed changes to the plant. This gives licensee management another input to its decisionmaking that is directly related to nuclear safety.

Use of Licensee PRA Models by the NRC Despite the fact that there is no regulatory requirement for 10 CFR Part 50 licensees to have or maintain a PRA, there are a number of processes that rely on the results from licensee PRA models so that the NRC can provide appropriate regulatory oversight. Among them are portions of the ROP, the Maintenance Rule, and licensing activities.

4 By letters dated May 9, 2014 (Ref. 21), and May 13, 2015 (Ref. 22), the NRC informed licensees of the initial screening and prioritization results, indicating that 33 power reactor sites would perform seismic PRAs (SPRA) because their reevaluated hazard exceeds their current design basis. The NRC staff has completed an assessment of reevaluated hazard and available seismic risk information and has concluded that 13 sites with low to moderate reevaluated seismic hazard exceedance above their current design basis are no longer expected to perform an SPRA. For the remaining 20 sites, SPRAs continue to be the appropriate analyses to assess the total plant response to the reevaluated hazard, so as to inform the NRCs regulatory decisions on the adequacy of these sites current seismic design basis.

https://www.nrc.gov/docs/ML1519/ML15194A015.pdf (Ref. 23)

The ROP measures nuclear plant performance by monitoring objective performance indicators (PIs) and by conducting the NRC inspection program. Monitoring and inspection closely focus on those plant activities having the greatest impact on safety and overall risk. The ROP PIs use objective data to monitor performance within each cornerstone area. The utilities generate the data that make up the PIs and submit these data to the NRC quarterly. Most PIs are based on discriminate data associated with plant operations (e.g., unplanned power changes per 7,000 critical hours); however, a subset of the mitigating systems cornerstone PIs (i.e., MSPI) is developed using results from the licensees PRA. Each PI is measured against established thresholds related to the PIs effect on safety. While PIs can lend insight into plant performance in selected areas, the NRCs inspection program provides a greater depth and breadth of information for monitoring and assessing plant performance. The inspection program is designed to verify the accuracy of PI information and to assess performance that is not directly measured by the PI data.

The Maintenance Rule (10 CFR 50.65, Requirements for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants) requires that power reactor licensees monitor the performance or condition of SSCs against licensee-established goals in a way that provides reasonable assurance that the SSCs can fulfill their intended functions. The NRC determined that a maintenance rule was needed because proper maintenance is essential to plant safety.

As discussed in the Statements of Consideration for this rule (Ref. 4), there is a clear link between effective maintenance and safety related to factors such as the number of transients and challenges to safety systems and the associated need for operability, availability, and reliability of safety equipment. In addition, effective maintenance is also important to minimize the failure of SSCs not related to safety that could initiate or adversely affect a transient or accident. Minimizing challenges to safety systems is consistent with the NRCs defense-in-depth philosophy. Maintenance is also important to ensure that design assumptions and margins in the original design basis are maintained and are not unacceptably degraded.

Therefore, NPP maintenance is important to protecting public health and safety. The extent of monitoring may vary from system to system depending on the particular systems importance to safety. Some monitoring at the component level may be necessary; however, the staff envisions that most of the monitoring can be done at the plant, system, or train level.

RG 1.160, Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, (Ref. 7) endorses Nuclear Management and Resources Council (NUMARC) 93-01, Industry Guideline for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, (Ref. 9) which provides methods acceptable to the NRC staff for complying with the provisions of 10 CFR 50.65. NUMARC 93-01 contains multiple references to the use and application of a PRA in a licensees implementation of the Maintenance Rule. The NRC staff endorses the use and application of these risk analyses as described in NUMARC 93-01. Like other types of engineering analyses used to support the regulatory process, risk analyses must be sound and technically defensible to increase confidence and consistency in decisionmaking. When a licensee uses a PRA in implementing the Maintenance Rule, the technical adequacy of the base PRA should be sufficient to provide the needed confidence in the results being used in the decision.

A licensee can take advantage of several risk-informed initiatives (e.g., 10 CFR 50.69, NFPA-805, risk-informed completion times) to gain greater operational flexibility and to improve safety. Each of these programs relies on the results of the licensees PRA model. Depending on the risk-informed application, if the licensee has more flexibility, or there is greater reliance on the PRA results, the NRC performs a more comprehensive review before issuing the safety evaluation. RG 1.174 provides guidance on the use of PRA findings and risk insights to support

licensee requests for changes to a plants licensing basis, as in requests for license amendments and TS changes under 10 CFR 50.90, Application for Amendment of License, Construction Permit, or Early Site Permit, through 10 CFR 50.92, Issuance of Amendment.

This guidance describes an acceptable method for the licensee and the NRC staff to use in assessing the nature and impact of licensing-basis change that may substantially increase risk.

The guidance in RG 1.174 provides a method for integrated decisionmaking that meets the following set of key principles:

• The proposed change meets the current regulation unless it is explicitly related to a requested exemption (i.e., under 10 CFR 50.12, Specific Exemptions).

• The proposed change is consistent with a defense-in-depth philosophy.

• The proposed change maintains sufficient safety margins.

• When proposed changes result in an increase in CDF or risk, the increases should be small and consistent with the intent of the Commissions Safety Goal Policy Statement (Ref. 32).

• The impacts of the proposed change should be monitored using performance measurement strategies.

Improved safety and greater operational flexibility have led many licensees to apply for and receive approval for these risk-informed initiatives. Many of these initiatives have a self-approval component that allows licensees to make changes based on their PRA results once the NRC has approved the risk-informed initiative. Focusing on risk significance can result in a more streamlined and efficient licensing review, but as analytical improvements support more advanced risk-informed initiatives, the licensees PRAs will continue to grow in importance.

2006/2007 EFFORTS TO EVALUATE USE OF LICENSEE PRA MODELS FOR THE SIGNIFICANCE DETERMINATION PROCESS In September 2006, the PRA Steering Committee, a predecessor to the current RISC, instructed the NRC staff (Ref. 15) to form a team to evaluate how licensee PRA models that meet the requirements of RG 1.200 could be used in the SDP. The committee held a kickoff meeting in December 2006 (Ref. 16) to identify issues and concerns with the current process and major issues that needed to be resolved before moving forward with alternatives. During the meeting, the team identified 10 items for further clarification and study. These items would form the major elements of the working group charter:

(1) What are the reasons for the SPAR model?

(2) How would a change to the use of the SPAR models in the SDP impact the other uses of SPAR models?

(3) What are the regulatory, logistical, and legal issues associated with the staff obtaining and using licensees PRA models?

(4) What does it mean to conform to RG 1.200 (regarding the technical adequacy of PRA for risk-informed initiatives) in the context of SDP?

(5) What recent improvements have been made to the SDP, and what others are under consideration?

(6) How are external events, LPSD, and large early release frequency treated under the SDP?

(7) What are the implications for the industry of having several different PRA software platforms?

(8) What are the sensitive unclassified non-Safeguards Information concerns with making the PRA models available for staff use?

(9) What about taking advantage of user-friendly configuration management tools (e.g., online risk monitoring)?

(10) What about SPAR model adequacy and standards?

At the conclusion of this meeting, the industry and NRC staff committed to continue working together to develop the working group charter and address the challenges identified.

The next meeting of the group, held in February 2007 (Ref. 17), focused on industry concerns, the status of the SPAR models, and options to the current process and criteria for evaluating them. The draft objectives of the group were also presented:

(1) Identify concerns with SPAR models in specific SDP evaluations. Determine if the issues are the result of deficiencies in SPAR model representation of plant design and operation, and not engineering inputs such as SSC performance degradation and duration.

(2) Review the status of the SPAR model updates. As one alternative, assess further enhancements to the SPAR model including a voluntary process by which significant plant changes and major updates to licensee PRAs are provided to the NRC staff and its contractors. Assess the advantages, disadvantages, and practical logistical aspects of this option.

(3) Identify the regulatory, logistical, legal, and resource issues, including staff training associated with the staff having all updated licensee PRA models in-house for use in the SDP.

(4) Identify and assess other practical alternatives including the use of licensee PRA models for SDP. Identify the standards, guidelines (e.g., RG 1.200), and regulatory conditions that would need to be met. Assess how and to what extent PRA model changes would be made to address the inspection findings. Identify key NRC staff and licensee interactions.

The group identified 11 criteria for the evaluation of the various options to be presented:

A. impact on licensee resources B. improved timeliness C. impact on NRC staff efficiency D. impact on licensee efficiency E. impact on NRC resources (e.g., training, model infrastructure)

F. ability to extend to external events and LPSD PRA modeling G. impact on ability of NRC staff to perform independent confirmation H. incentive to industry to adopt RG 1.200 promptly I. improved completeness and fidelity of models J. scrutability of models and results K. degree of implementation by industry

The following options, including the pros and cons of using the criteria above, were discussed and included. (Note: Criteria not listed for an option do not work strongly in favor or against the option.)

• Option 1 - Status quo: The NRC will continue to improve SPAR models. For specific findings, both the licensee and NRC make adjustments to their individual models. There is usually good agreement between licensee and NRC staff results. The staff relies on licensee PRA models for some regulatory applications. As licensees comply with RG 1.200, the NRC staff has more confidence in the results. For external events, the staff uses input from licensee external events PRA and/or bounding analyses.

CON - F, H PRO - G

• Option 2 - The licensee PRA model meets RG 1.200 and the licensee uses its PRA model to make a determination of significance.

CON - E, G, J PRO - A, B, C, D, F, H, I, K (Note: For Options 2 - 4, the evaluation is made in comparison to Option 1.)

• Option 3 - The licensee PRA model meets RG 1.200, and the model and all supporting documentation are provided to the NRC staff to run. The option assumes there is a translator for basic event and other modeling information.

CON - A, E, G, K PRO - B, D, F, H, I, J To be determined (not evident if a PRO or CON at this time - C

• Option 4 - The NRC fully updates the SPAR models to reflect licensees RG 1.200 conforming model. SPAR models all use the same methods and general modeling assumptions (e.g., human reliability analysis (HRA), though performance shaping factors may differ, as would vendor-specific reactor coolant pump seal failure model). The standard level of detail in the models should be specified and credit restricted for use of nonstandard systems for which full training (including hands-on walkthrough) is not in use. The models would use plant-specific data, though the data analysis would be prescriptive.

CON - A, E, F, H, K PRO - B, D, I, J To be determined (not evident if a PRO or CON at this time) - C

• Option 5 (applies to replacement of phase 2 screening worksheets) - The NRC staff uses licensees configuration risk management model. This model can be used by the resident inspector or a visiting inspector. Its results provide risk insights, not just a risk number, and configuration risk management meets regulatory requirements.

CON - A, E, G PRO - B, C, F, H, I, K Note: Criteria not listed above appear to be neutral in impact compared to the status quo for phase 2 screening described in Option 6.

• Option 6 (applies to replacement of phase 2 screening worksheets) - The NRC staff uses a SPAR model with SAPHIRE 8. This option is the status quo for phase 2 screening in that the staff appears to be headed in this direction, and the option is therefore neutral in impact.

• Option 7 - Replace licensee PRA models with Super-SPAR: NRC resources required to upgrade 70+ models to the level of detail in all licensee PRA models, including external events, would be in the many tens of millions of dollars. It is doubtful that industry would discard its PRA models, so this option was dismissed as impractical.

No determination regarding the preferred option(s) was made at this meeting. After the meeting, the industry and staff working groups were to comment on the objectives, better define the options (including the consideration of hybrid or mixed options), and begin to consider weights for the criteria identified.

Discussions between the NRC staff and industry were held at the third and final team meeting in May 2007 (Ref. 18). The team discussed the options identified during the previous public meeting. The industry made a presentation on the status of the portability of PRA models between different software platforms, which would make some aspects of Option 3 more feasible. The NRC staff expressed skepticism that all aspects of model logic would easily be converted from one software platform to another, given the wide variety of models in the industry. The staff shared that while the approach holds promise for the future, logistical and institutional issues would need to be resolved before full implementation could be realized.

The industry stated its belief that Option 2, in which licensees PRA models were used as the primary tool for the significance determination, was most viable. The industry felt that use of the licensee models would not necessarily compromise the ability of the staff to perform independent review. They noted that licensee models are used in many risk-informed applications that appear to be as important as the SDP, including TS initiative 4b (configuration risk management) and license amendment requests for extension of allowed outage time. The industry further stated that meeting the requirements of RG 1.200 is a major effort and that allowing the use of licensee models that meet RG 1.200 for significance determination would help the business case for licensees to undertake the major upgrade effort. The industry wanted to initiate a pilot to test the feasibility of its proposed approach.

The NRC staff expressed several concerns with Option 2. In particular, allowing licensees to determine the significance of inspection findings would infringe on a fundamental function of the regulator. The staff also expressed its concern about the public perception of adopting such an approach. The staff noted the value of having senior reactor analysts perform risk assessments and engage the industry on proposed findings. Moreover, the staff believed there was a lack of justification for a change in process when only 10 to 12 greater-than-green findings are processed per year for the entire fleet of operating reactors. The staff reiterated that most of the issues with the current process are not related to SPAR modeling but concern how the degraded condition is represented in the risk assessment. The staff discussed the merits of addressing PRA technical issues jointly with industry as a preferred approach. Finally, the staff

noted that there are fundamental differences in risk-informed applications under RG 1.174 and enforcement actions being evaluated under the ROP.

No final positions were taken during the meeting, and the NRC staff stated its intention to deliberate the merits of the industrys proposal and respond in the near future. In an October 2007 letter to the Nuclear Energy Institute (NEI) (Ref. 19), the staff presented its final position and rationale regarding the topic areas discussed during the public meetings:

(1) Maintaining the independence of the NRC and licensees models.

The NRC staff has concluded that, because the NRCs Reactor Oversight Process (ROP) is intended to provide an independent regulatory assessment of licensee performance, it would be inappropriate for licensee risk analysts to take the lead in assessing the significance of performance deficiencies at their site.

Such an arrangement would also minimize the NRC staffs ability to ensure that issues are assessed in a timely manner. Maintaining the NRCs independent oversight of licensee performance is critical for effective NRC oversight and is an important aspect of upholding public confidence in the process.

(2) The nature of the differences in the SDP outcomes using NRC versus licensee assessment methodologies.

Many insights were shared and conclusions reached by these meetings. In summary, it was noted that differences in SDP outcomes between the NRC and the licensee are driven by factors other than the baseline PRA model used for the analysis; in fact, the PRA models are often in close agreement. The differences, however, are seen in the way engineering assumptions, human reliability analysis, and recovery are handled within the analysis. We recognize that licensees may have unique perspectives on the event or condition under agency review. Therefore, the SDP allows for input from licensees regarding such risk insights and we intend to encourage further engagement with the licensees on SDP findings.

(3) Standardization of modeling and assessment techniques used.

At present, the industry has not uniformly implemented a standardized approach to performing risk analysis that would ensure uniform application across the spectrum of industry PRA models. In this regard, the NRCs use of the SPAR models, together with the ongoing development of guidance on conducting Phase 3 risk assessments, commonly referred to as the risk assessment standardization project (RASP), ensures greater uniformity in the agencys regulatory assessments. To aid licensees, we intend to make the RASP manual publicly available in the near future.

(4) Potential use by the NRC staff of licensees PRA models.

We also considered an alternative to the current NRC staff use of SPAR models where the staff would be provided with the licensee PRA models. Under this option, the staff would perform the assessment of risk significance using the licensee model. We have concluded that the logistical and resource needs to maintain the 70-plus industry PRA models on some four software platforms would require the diversion of NRC staff efforts and the addition of risk analysts.

These NRC resources would be more effectively used for other tasks. At present, this alternative is not a viable option unless the industry implemented a single RG 1.200 compliant modeling approach on one analysis platform facilitating efficient use of NRC resources.

We believe that continued improvement to the standardization of PRA modeling methods in SPAR and industry PRA models is the most effective use of resources, commensurate with the need for the staff to maintain its own methods for confirmatory and independent analysis.

This letter outlined the staffs view, and the effort to evaluate the use of licensee PRA models ended at that time. The staff continued to collaborate with the NEI on various aspects of the use of SPAR models to evaluate the risk of findings associated with the ROP. That collaboration continues to this day with the staff reviewing materials submitted by the NEI and holding public meetings on various topics including common-cause factor modeling, minimum joint human error probability floor values, and credit for equipment procured to comply with post-Fukushima orders. As mentioned in the letter, the RASP continued, and the RASP handbook was made publicly available and remains so to this day.

2015/2016 EFFORT TO EVALUATE USE OF LICENSEE PRA MODELS IN ALL REGULATORY APPLICATIONS WHERE SPAR MODELS ARE CURRENTLY USED Rationale for Evaluation Late in 2015, the RISC chair asked for a team to be formed to research the use of licensee PRA models in regulatory applications. Discussion of the rationale for forming such a team centered on the perceived inefficiency of maintaining two models (licensee PRA models and NRC SPAR models) and the potential for cost savings to the NRC by using a licensee-developed PRA to make regulatory decisions. Stakeholders generally agree that SPAR and the licensees base PRA models closely align, and the issues encountered in regulatory processes when there is disagreement are with the modeling of specific factors and boundary conditions in the PRA (e.g., HRA, common cause). The NRC staff reminded RISC members that the NRC completed a comprehensive review of the use of licensee PRA models for the SDP in 2006/2007 and that the NRC staff ultimately decided to maintain the status quo and continue use of SPAR models.

In reviewing the findings from the NRC staff in 2007, the RISC members stressed the need to focus specifically on the progress licensees have made toward standardization (including one software platform and similar assumptions and nomenclature) and potentially overcoming the other obstacles identified during that assessment. The RISC members also contended that this review should not be limited only to the SDP (as the 2006/2007 review was) and should evaluate the use of licensee PRA models in all processes where SPAR models are currently used with the intention of eliminating the NRC dependence on the SPAR models for regulatory decisions.

Defining the Scope In January 2016, the NRC staff made its first presentation to the RISC (Ref. 30), in which it defined the scope and goals of the project. The staffs presentation identified three different

options regarding the use of licensee PRA models in regulatory applications, including the advantages and disadvantages of each option:

(1) Option 1 - Status quo: The NRC will continue to update 6 -10 SPAR models per year and develop new All Hazard (including fire and seismic hazards) SPAR models. The advantages and disadvantages of this approach included the following:

- This approach provides a means to maintain an assessment tool independent from that of the licensees.

- Public confidence is enhanced since the regulator will arrive at a conclusion based on a diverse path from that of the licensee.

- Continuing current practice enables the staff to assess risk significance of various plants on a common basis.

- This option enables staff in the Office of Nuclear Reactor Regulation (NRR) to support management efforts to risk-inform regulatory decisions on a generic or plant-specific basis.

- The staff will continue to expend resources to update and develop models.

(2) Option 2 - Results will be provided to the NRC staff for use in making regulatory decisions. The advantages and disadvantages of this approach included the following:

- Staff will not need to expend additional resources to update the models.

- Allowing licensees to use models compliant with RG 1.200 for SDPs may motivate them to invest resources to improve their models.

- A certified PRA would allow the licensees PRA to be the PRA of record for all regulatory actionsincluding the SDP.

- The option may inhibit staffs ability to provide timely, independent inputs to time-critical NRC decisions in support of notices of enforcement discretion and MD 8.3 actions.

- The option may inhibit the ability of the NRR to deal effectively with future challenges in aggregation of results and other integrated risk-informed decision making related challenges since staffs ability to enhance external event risk assessments will be affected.

- The staffs ability to evaluate generic issues affecting the nuclear fleet may be inhibited.

- The staffs ability to provide input to regulatory basis documents for potential rulemaking efforts may be inhibited.

- The option may negatively affect public perception of the independence of the NRCs regulatory processes.

(3) Option 3 - The NRC staff will be given access and training to run licensees PRA models in support of various regulatory actions.

- Option 3a-Licensees are required to give the NRC their model of record at some established point in the regulatory process. The NRC staff would then run the analysis using the licensee PRA model instead of the SPAR model.

- Option 3b-The NRC maintains a model of record for each plant with the requirement that, if the licensee updates the model, it must send the NRC the new model within a certain timeframe.

- The advantages and disadvantages of this approach included the following:

The option will not require staff to expend additional resources to update SPAR models.

The most up-to-date model will always be used.

The option may motivate licensees to develop and maintain PRA models that comply with RG 1.200.

Although the NRC is using the licensee model, independence is maintained.

The NRC will keep its ability to assess generic and other fleetwide issues if the agency maintains a model of record.

The NRC will have to expend significant resources to train multiple staff members (in NRR, the Office of New Reactors, and the regions) to run licensees models.

Each PRA model contains important switches or house events that require familiarity on the part of the analyst (i.e., the training burden can be significant).

The NRC will need to maintain commercial software licenses for multiple PRA codes (e.g., CAFTA, RISKMAN).

It is not likely that all licensees will grant the NRC staff access to their PRA models.

Infrastructure similar to that for SPAR models must be established to maintain a model of record for each plant that would be accessible to risk analysts.

The RISC members immediately dismissed Option 2 which would allow the licensees to generate results and provide them to the NRC. Based on this management direction, the team did not analyze this option. The rest of the conversation revolved around the use of licensee PRA models by NRC risk analysts and some of the challenges they would encounter. RES also presented a CBA that had previously been developed and presented through RES management. This CBA formed the starting point for the work by NRR cost analysts to address

the challenges and to evaluate the comprehensive cost of maintaining SPAR models, including a full evaluation of lifetime costs and needed impending upgrades (e.g., seismic).

The RISC also directed the team to provide a plan to pilot the use of licensee PRA models and to develop criteria to objectively evaluate a pilot. In parallel with developing the plan for a pilot, the team members were to engage with the industry to evaluate licensee willingness to participate in such a program (i.e., providing the NRC with access to use licensee PRA models for all applications currently using SPAR models) and to evaluate the level of standardization among industry licensees in areas such as software platform, modeling assumptions, and nomenclature.

Approach and Evaluation One of the teams initial activities was to develop a list of challenges to be overcome or evaluated during the project. There was some debate on whether a pilot would be beneficial and cost effective to the project. A pilot would allow the NRC staff to evaluate some, but not all, of the potential challenges. The team determined that the list of challenges could be logically separated into three distinct categories: (1) challenges that could be evaluated before the pilot, (2) challenges that could be evaluated during the pilot, and (3) challenges that could not be evaluated with the pilot. The identified challenges were broken down as follows:

• Challenges the NRC can evaluate before the pilot:

- the need to maintain commercial software licenses for multiple PRA codes (e.g., CAFTA, RISKMAN)

- legal issues involved in making regulatory decisions using models generated and maintained by licensees

- training costs associated with new software (e.g., CAFTA)

• Challenges the NRC can evaluate during the pilot:

- For processes like the SDP, the SPAR models are often changed at the analysts request. INL currently makes these changes. The licensee would have to agree to any change that the NRC would make in the licensee PRA, or the NRC version of the model would deviate from the licensees. The NRC would need to create a process for handling changes to the models.

- An infrastructure similar to that for the SPAR models must be established to maintain a model of record for each plant that would be accessible to risk analysts when needed.

- Base SPAR model and licensee PRA models are often in close agreement.

Analyst decisions made on modeling assumptions, HRA assumptions, application of common cause, and other specific elements are usually the cause of differing results. Using the licensee models may not alleviate these issues.

- A procedure must be identified to determine how questions about the PRA models will be handled and resolved during actual events.

- The level of training needed to run models must be evaluated.

- The level of effort needed to develop and agree on a memorandum of understanding (MOU) must be decided.

- The timeliness of decisionmaking for MD 8.3 reviews and notices of enforcement discretion must be determined.

- The pilot could evaluate false positive and false negative results (e.g., obtaining a green result when in reality the risk is white) by simultaneously running CAFTA and SAPHIRE. However, if SPAR models were eventually eliminated, the NRC would have little or no ability to run a second check.

- The pilot could evaluate the stability of the model of record. Experience has shown that many licensees change their models almost continuously (e.g., Duke and the Oconee model). Holding a model static and not allowing further changes without agreement from both the licensee and the NRC (for example, during a greater-than-green SDP issue) would be a good stress test during the pilot program.

• Challenges the NRC cannot evaluate with the pilot:

- All licensees may not agree to grant access to their models.

- It is unknown how many licensees must participate for the change to be worthwhile.

- Industry lacks standardization in nomenclature, modeling conventions, post-processing rules, and other areas.

- Given its limited regulatory authority over licensee PRAs, the NRC must determine how to handle inadequate PRA models on a case-by-case basis.

- Even within existing fleets, each PRA model contains important switches or house events that require familiarity on the part of the analyst (i.e., the training burden can be significant).

- The NRCs approach to performing generic assessments will be complicated if SPAR models are eliminated.

- The NRC must determine an update process for licensee models and find a way to control the model of record.

- Different plants or fleets may require different MOU. Licensees probably will not all agree to the same terms of use. Resolving this challenge will likely require NEI involvement.

- Increased variability in the NRCs results (if any) would be difficult to assess in the pilot program because of differences in modeling approaches taken by different licensees. For example, an NFPA-805 plant would have very good fire modeling, yet another similarly designed plant that did not transition to NFPA-805

might have good internal flooding modeling. The failure of a specific component, such as an emergency diesel generator, might yield one result at the first plant and a very different result at the second, yet this would not be discernible during the pilot.

- The NRC would have to determine the possibility of any perceived lack of independence.

The pilot effort presented to the RISC had an aggressive schedule lasting a total of 12 months (3 months of an initiation phase, 6 months of implementation, and 3 months of evaluation). The initiation phase would allow the setup of an MOU with the pilot plant, development of a training plan (for NRC staff not familiar with CAFTA), and determination of which representative processes (e.g., SDP) would be used in the pilot. The 3 months allowed for the initiation phase was based on an efficient development and agreement process with the volunteer licensee.

Based on past experiences with accessing plant models, this could take far longer than the targeted 3 months. During the implementation phase, the NRC staff would use the licensees PRA model to perform tabletop exercises representing actual events or conditions that the NRC has evaluated using SPAR models. Diversity in each area of interest was stressed to maximize the information obtained from the pilot. Finally, the documentation phase would focus on developing a final pilot report and getting the appropriate approvals before presenting the report to the internal RISC. The team recognized that the targeted 12-month duration for the pilot would change if unforeseen challenges or resource limitations delayed the staff in obtaining the data necessary to evaluate the feasibility of shifting the regulatory structure. Before deciding on the pilot, the RISC wanted more clarity on the CBA and overall willingness of licensees to participate.

Cost-Benefit Analysis One of the important components of the analysis and evaluation of a transition to using licensee PRA models was the CBA developed with NRR cost analysts. RISC members requested this information to make an informed decision on whether implementing this approach would result in efficiencies. The RISC asked that the CBA consider the additional information listed below, in addition to what was considered in the RES analysis:

• training costs for both current practice and a shift to the use of licensee PRA models

• software required including costs of information technology support associated with new software (e.g., network testing and acceptance)

• costs associated with complexity and the unfamiliarity of current practitioners with models based on actual agency use

• future efficiencies resulting from the shift to licensee PRA model use (mostly from eliminating the need for continued updates to SPAR models)

• estimated headcount changes anticipated by the shift to licensee models

• potential changes required to the regulatory structure to accommodate the use of licensee PRA models

• realistic costs through the transition period (3 years) and beyond

• an assessment of the required level of review of licensee models (no review to full-scope NRC staff review)

The NRR staff on the project, including NRR cost analysts, started the CBA by first evaluating the previous RES cost analysis. The CBA focused on two scenarios. The first was the current costs of maintaining and using the NRC SPAR models, and the second was the cost of transitioning and using licensees PRA models in lieu of the SPAR models. The analysts considered additional factors along with adjusting certain assumptions to develop their CBA.

The cost data were analyzed in five areas, four quantitative and one qualitative. The four quantitative areas analyzed were the following:

(1) contract expenses (2) user expenses (3) training (4) other costs The fifth area, qualitative costs, was included to describe some of the positive and negative aspects of the two options that were not represented in the quantified analysis but should be considered when making a decision.

The staff performing the CBA engaged NRC risk analysts in other offices to estimate how their workload would be affected by using a licensees model instead of the current SPAR models. In all cases, analysts predicted that the complexity and lack of standardization of industry models would increase the resource requirements to perform comparable assessments. However, contract expenses associated with current SPAR model maintenance and development would be significantly reduced or could be eliminated altogether. As requested by the RISC, the CBA displayed an estimated range of future costs from a low estimate to a high estimate. Figure 2 shows the current SPAR model expenses and a calculated average (low estimate + high estimate / 2) of the cost estimate of transitioning based on the in-progress CBA (Ref. 31).

Ultimately, the CBA showed a significant cost to transition to licensee PRA models. There was a potential for longer term cost savings once full transition was complete based on the low estimate, but the average estimated long-term costs were higher than the costs associated with maintaining the current SPAR model, with the potential for the higher costs to be significant.

These results represent a best estimate without further refinement from data available at the time. If the project had continued, the staff would have completed a more thorough and complete analysis.

Current SPAR Average NRC Cost Estimates To Use Model Expenses Licensees PRA Models Initial Costs Ongoing Costs Contract Expenses $ 2,757,627.00 $ 1,056,600.00 $ 1,056,600.00 User Expenses $ 2,505,344.00 $ 3,758,016.00 $ 3,873,216.00 Training $ 216,000.00 $ 850,080.00 $ 182,160.00 Other Costs $ - $ 3,283,000.00 $ 1,100,472.00 TOTAL $ 5,478,971.00 $ 8,947,696.00 $ 6,212,448.00 Figure 2: Comparison of average cost estimates for transitioning to licensees PRA models

Decision to Stop Evaluation An essential consideration in deciding whether to transition from NRC-developed SPAR models to use of licensee PRA models in regulatory applications was that licensees would need to voluntarily provide their models to the NRC for use. This challenge was first presented (Ref. 33) and discussed with the industry at a public RISC meeting in February 2016. As the project developed, the NRC staff determined that the success of a new approach using licensee PRA models would be heavily based on the industry willingness to participate. A presentation (Ref.

35) requesting industry assistance in evaluating licensee willingness was made at a public meeting in May 2016 (Ref. 34). While some individual licensees expressed willingness to provide access to their models, overall the NRC staff found the industry, as a whole, unwilling to participate in such a program at this time. The licensees and the NEI did not believe there was sufficient benefit to the licensees to justify participation in this voluntary program (Ref. 36).

To experience any cost savings, the NRC would need to stop maintaining the SPAR models and eliminate any further enhancements or maintenance of the models. This would also include the elimination of contracts with INL for support and maintenance of the models. A major concern was that this would be a voluntary program from which licensees could potentially withdraw their support in the future. This would open the agency to significant risk once the transition to the use of licensee-developed PRA models was complete. Since the contracts and support from INL would be terminated to realize the cost savings from eliminating the program, the NRC would have to expend considerable cost and effort if it needed to reestablish the program.

The insights from the CBA showed a significant cost to transition to licensee PRA models.

There was a potential for longer term cost savings once the full transition was complete, but the average estimated long-term costs were higher than maintaining the current SPAR model.

Therefore, significant costs to the agency could continue despite the transition to the use of licensee PRA models. Based on the results of the CBA, and considering other factors involved (i.e., licensee willingness to participate), the NRC staff recommended that the NRC should continue to rely on SPAR models in implementing its risk-informed regulatory activities. Based on the staffs recommendation, the RISC made the decision to no longer continue the evaluation and continue to use SPAR models for operating reactor oversight programs.

Other Government Agencies Use of SAPHIRE Any analysis of the elimination of a program should account for all parties affected or potentially affected. In this case, there are a number of other government agencies that have adopted the use of SAPHIRE. The National Aeronautics and Space Administration (NASA) has adopted the use of SAPHIRE for risk analysis in its operations and could be detrimentally affected by the NRCs elimination of support for the program. In the past, NASA relied on worst-case failure modes and effects analysis (FMEA) for safety assessments. This approach has some notable problems, including its qualitative nature and its inability to aggregate risk at a system or mission level. These aspects are key factors for NASA to consider when attempting to evaluate overall risk and to inform its decisions concerning mission or crew.

In its 1986 report (Ref. 13) on the investigation of the Challenger accident, the U.S. House Committee on Science and Technology criticized NASA for not using tools beyond FMEA for a more complete safety assessment:

The Committee finds the FMEA to be an appropriate method for identifying the Critical 1 and 1R elements of the National Space Transportation System (NSTS);

however, not all the elements so identified pose an equal threat. Without some means of estimating the probability of failure of the various elements it is not clear how NASA can focus its attention and resources as effectively as possible on the most critical systems.

Further, in January 1988, the National Resource Council post-Challenger investigation (Ref. 14) recommended that probabilistic risk assessment approaches be applied to the Shuttle risk management program at the earliest possible date.

Consequently, NASA now embraces probabilistic methods and has used or is using SAPHIRE software as the primary probabilistic risk analysis tool for the International Space Station, the Space Shuttle, studies in support of nuclear missions, conceptual designs (e.g., Constellation),

and the Mars Exploration Rover. Although the staff did not do a complete analysis of the impact of withdrawing NRC support for SAPHIRE, it remains a potential challenge and impact as defined by the team. If the NRC staff had recommended a reduction or elimination of support for SAPHIRE, the staff would have completed more analysis, including interactions with other governmental agencies. In addition to NASA, the US Air Force, the Naval Postgraduate School, the US Department of Defense Missile Defense Agency, and the Bureau of Safety and Environmental Enforcement are active members of the SAPHIRE users group. There are many tools and software platforms that could potentially be used in place of SAPHIRE and research into potential options (including other government agencies funding SAPHIRE) would be included in a more thorough analysis and should be pursued if the issue of elimination of the SPAR program is revisited.

CONGRESSIONAL INQUIRY In a request for specific information (Ref. 28) on the SDP, Congress posed the following question:

Licensees maintain detailed, plant-specific Probabilistic Risk Assessment (PRA) models that are accessible to the NRC. Please describe the potential to utilize these more detailed models in lieu of the NRCs Standardized Plant Analysis Risk (SPAR) model as a means of reaching quantitative regulatory decisions that are more efficient and timely. Please also describe the actions taken and/or planned to address this opportunity.

The staff responded in a letter dated February 2017 (Ref. 28), which explained the development and use of plant-specific SPAR models for regulatory applications. The response briefly described the effort to evaluate the use of licensee PRA models in lieu of SPAR models in regulatory applications, including the decision that was made by the NRC RISC to continue to use SPAR models for operating reactor oversight programs at this time.

OFFICE OF INSPECTOR GENERAL REPORT The Office of the Inspector General (OIG) reviewed the NRCs assessment of alternative risk modeling techniques from March 2017 to June 2017 (Ref. 20). The objective of this evaluation was to assess the NRCs process for piloting alternative risk modeling techniques and analyzing costs, benefits, and feasibility of these alternatives. The evaluation focused on the NRCs process of assessing alternative risk modeling techniques and found the following:

improved coordination and documentation of staff assessments would better support NRCs efforts to evaluate the costs, benefits, and feasibility of alternatives to its current risk modeling program (SPAR). Although preliminary staff assessments show credible cost and feasibility limitations to adopting industry risk models, NRC has yet to document the results of this work and use it as the basis for a formal policy position. These actions are particularly important in the current regulatory climate, which emphasizes risk-informed decision-making. Moreover, better process management can help NRC more efficiently revisit SPAR alternatives if new cost data and feasibility solutions become available.

OIG recommended that the Executive Director for Operations formally document evaluation results that will establish the agency position on the NRCs use of licensee PRA models, to include reliable, verifiable cost data.

The NRC staff provided formal comments on the report (Ref. 20), and agreed to formally document evaluation results that will establish the agency position on NRCs use of licensees PRA models, to include reliable verifiable cost data. In accordance with the OIGs recommendation, the staff developed this report to document its evaluation results regarding the NRCs decision to continue to maintain independent plant-specific SPAR models.

In reviewing the OIG report, the staff also noted that the report stresses the importance of the cost analysis in the overall RISC decision. The RISC decision to terminate the staff evaluation of the use of licensee PRA models in lieu of SPAR models was based principally on the lack of broad support for sharing licensee PRA models; further refinement of the cost data to support a decision became unnecessary based on the lack of an industry commitment to sharing licensee PRA models. The staff further noted that the general descriptions in the OIG report are not necessarily reflective of the staffs view regarding the importance of the cost analysis to the overall RISC decision to terminate the staff evaluation.

CONCLUSION As discussed in this report, the NRC relies on results from licensees PRA models for certain risk-informed regulatory applications (e.g., licensing) and uses the SPAR models for other regulatory applications (e.g., the SDP).

Maintaining two models may, on the surface, seem like an inefficiency that could easily be eliminated. However, shifting the regulatory structure to use one model, especially a model that is developed and maintained by a licensee, poses many challenges.

This report summarizes the NRCs effortsspecifically, the effort to evaluate transitioning regulatory decisions from SPAR models to the licensees PRA models. Ultimately, the RISC decision to terminate the staff evaluation at this time was based principally on the lack of broad industry support for sharing licensees PRA models, combined with the CBA results showing that transition costs were significant and possible efficiency gains would not be achieved without considerable risk to the agency. Collectively, these factors make it clear that, at this time, it is not appropriate to transition from NRCs SPAR models to the use of licensees PRA models.

Both internal and external stakeholders have shown interest in exploring a transition to relying exclusively on licensees PRA models instead of maintaining an NRC-specific SPAR model for each plant. As described above, this transition has been proposed several times in the past, and the NRC staff has completed evaluations of different aspects of the proposals. Based on the

history of this topic, the staff anticipates that this issue is likely to surface again. At a minimum, each of the factors evaluated in the staffs 2015/2016 effort, in addition to the issues previously identified during the 2006/2007 effort, would need to be addressed before the NRC staff could support a change in the use of NRC SPAR models for risk-informed decisionmaking.

REFERENCES

1. U.S. Code of Federal Regulations (CFR), Domestic Licensing of Production and Utilization Facilities, Part 50, Chapter 1, Title 10, Energy.

2. CFR, Licenses, Certifications, and Approvals for Nuclear Power Plants, Part 52, Chapter 1, Title 10, Energy.

3. NRC, Use of Probabilistic Risk Assessment Methods in Nuclear Regulatory Activities:

Final Policy Statement, Federal Register, Vol. 60, No. 158, p. 42622, August 16, 1995 (60 FR 42622), ADAMS Accession No. ML021980535.

4. NRC, Statements of Consideration, Final Rule Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, Federal Register, Volume 56, Number 132, p.

31306, July 10, 1991 (56 FR 31306).

5. ASME/ANS RA-Sa-2009, Standard for Level 1/Large Early Release Frequency Probabilistic Risk Assessment for Nuclear Power Plant Applications, Addendum A to RA-S-2008, ASME, New York, NY, American Nuclear Society, La Grange Park, Illinois, February 2009, February 2, 2009, ADAMS Accession No. ML15322A099.

6. NRC, Regulatory Guide 1.200, Rev. 2, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities, March 2009, ADAMS Accession No. ML090410014.

7. NRC, Regulatory Guide 1.160, Rev. 3, Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, May 2012, ADAMS Accession No. ML113610098.

8. NRC, Regulatory Guide 1.174, Rev. 3, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, January 2018, ADAMS Accession No. ML17317A256.

9. NUMARC 93-01, Industry Guideline for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, Revision 4A, April 2011, ADAMS Accession No. ML11116A198.

10. NRC, NUREG-0800, Standard Review Plan for the Review of the Safety Analysis Reports for Nuclear Power Plants: LWR Edition, Section 19, Probabilistic Risk Assessment and Severe Accident Evaluation for New Reactors, Rev. 3, December 2015, ADAMS Accession No. ML15089A068.

11. NRC, Generic Letter 88-20, Individual Plant Examination for Severe Accident Vulnerabilities 10 CFR 50.54(f), November 23, 1988, ADAMS Accession No. ML031150465.

12. NRC, Generic Letter 88-20, Supplement 4, Individual Plant Examination of External Events (IPEEE) for Severe Accident Vulnerabilities10 CFR 50.54(f), June 28, 1991, ADAMS Accession No. ML031150485.

13. U.S. House Committee on Science and Technology, Investigation of the Challenger Accident; Report of the Committee on Science and Technology, House of Representatives, U.S. Government Printing Office, October 1986.

14. National Research Council, Post-Challenger Evaluation of Space Shuttle Risk Assessment and Management, January 1988, Washington, DC: The National Academies Press, https://doi.org/10.17226/10616.

15. NRC, Summary of September 28, 2006, Probabilistic Risk Assessment Steering Committee Public Meeting, October 12, 2006, ADAMS Accession No. ML062840400 (Package).

16. NRC, Summary of December 13, 2006, Public Meeting Regarding Use of Standardized Plant Analysis Risk Models and Licensee Probabilistic Risk Assessment Models in the Reactor Oversight Process, December 19, 2006, ADAMS Accession No. ML063530303.

17. NRC, Summary of February 22, 2007, Public Meeting Regarding Use of Standardized Plant Analysis Risk Models and Licensee Probabilistic Risk Assessment Models in the Reactor Oversight Process, February 28, 2007, ADAMS Accession No. ML070640582 (Package).

18. NRC, Summary of May 15, 2007, Public Meeting Regarding Use of Standardized Plant Analysis Risk Models and Licensee Probabilistic Risk Assessment Models in the Significance Determination Process, May 25, 2007, ADAMS Accession No. ML071490160 (Package).

19. Reyes, Luis A., NRC, letter to Marvin S. Fertel, NEI, regarding NRC response to NEI letter on August 2 Commission Briefing, October 15, 2007, ADAMS Accession No. ML072490540.

20. Office of Inspector General, Evaluation of Proposed NRC Modifications to the Probabilistic Risk Assessment Process, Audit Report OIG-17-A-26, September 21, 2017, ADAMS Accession No. ML17264A298.

21. NRC, Final Determination Regarding Seismic Probabilistic Risk Assessments, October 27, 2015, ADAMS Accession No. ML15194A015.

22. NRC, Seismic Screening and Prioritization Results, May 9, 2014, ADAMS Accession No. ML14111A147.

23. NRC, Seismic Screening and Prioritization for Western United States Sites, May 13, 2015, ADAMS Accession No. ML15113B344.

24. Risk Assessment of Operational Events (RASP) Handbook, Volumes 1 - 4, https://www.nrc.gov/reactors/operating/oversight/program-documents.html.

25. NRC, SECY 98-144, White Paper on Risk-Informed and Performance Based Regulation, January 22, 1998, https://www.nrc.gov/reading-rm/doc-collections/commission/secys/1998/secy1998-144/1998-144scy.pdf.

26. NRC, Management Directive 8.3, NRC Incident Investigation Program, June 25, 2014, ADAMS Accession No. ML13175A294.

27. NFPA 805, Performance-Based Standard for Fire Protection for Light Water Reactor Electric Generating Plants, 2001 Edition, ADAMS Accession No. ML010800360.

28. NRC, Correspondence to Senators John A. Barrasso and Shelley Moore Capito, Status Report on the Licensing Activities and Regulatory Duties of the U.S. Nuclear Regulatory Commission, February 2017, ADAMS Accession No. ML17025A395.

29. NRC, NUREG/BR-0167, Software Quality Assurance Program and Guidelines, February 1993, ADAMS Accession No. ML012750471.

30. NRC, Presentation for Internal RISC Meeting, Use of Licensee PRA Models Options, January 2016, ADAMS Accession No. ML18135A243.

31. NRC, Cost Benefit Analysis (Draft, Revision 17), July 2016, ADAMS Accession No. ML18131A308.

32. NRC, Safety Goals for the Operations of Nuclear Power Plants; Policy Statement, Federal Register, Volume 51, p. 30028, August 4, 1986, republished with corrections, Volume 51, No. 169, pp. 30028-30023, August 21, 1986, (51 FR 30028) ADAMS Accession No. ML051580404.

33. NRC, Use of Licensee PRA Models in Regulatory Applications, Presentation at RISC Public Meeting, February 9, 2016, ADAMS Accession No. ML16048A605.

34. NRC, Summary of May 10, 2016, Public Meeting with Risk-Informed Steering Committees to Engage the Industry and Public in Discussions on Risk-Informed Topics of Interest, June 9, 2016, ADAMS Accession No. ML16159A027.

35. NRC, Use of Licensee PRA Models in Regulatory Applications, Presentation at May 10, 2016, RISC Public Meeting, ADAMS Accession No. ML16140A077.

36. Anderson, Victoria, Senior Project Manager - Risk Assessment NEI, Email to Montecalvo, Michael, U.S. NRC, Question, July 2016, ADAMS Accession No. ML18177A273.

37. NRC, Statements of Consideration, Final Rule, Risk-Informed Categorization and Treatment of Structures, Systems and Components for Nuclear Power Reactors, Federal Register, Volume 69, Number 224, p. 68008, November 22, 2004 (69 FR 68008).

BIBLIOGRAPHY

1. NRC, Backgrounder on Probabilistic Risk Assessment, February 2016, Agencywide Documents Access and Management System (ADAMS) Accession No. ML032200337.

2. NRC, NUREG-0800, Standard Review Plan for the Review of the Safety Analysis Reports for Nuclear Power Plants: LWR Edition, Section 19.1, Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities, Rev. 3, September 2012, ADAMS Accession No. ML12193A107.

3. NRC, NUREG/BR-0058, Regulatory Analysis Guidelines of the U.S. Nuclear Regulatory Commission, Rev. 5, April 2017, ADAMS Accession No. ML17100A480.

4. NRC, NUREG/CR-7039, Volume 1, Systems Analysis Programs for Hands-on Integrated Reliability Evaluations (SAPHIRE) Version 8: Overview and Summary, June 2011, ADAMS Accession No. ML11195A298.

5. NRC, Advisory Committee on Reactor Safeguards, Regulatory Effectiveness of Unresolved Safety Issue A-45, Shutdown Decay Heat Removal Requirements, Letter Report, November 18, 2003, ADAMS Accession No. ML033230548.

6. Travers, William D., U.S. NRC, letter to Mario V. Bonaca, ACRS Chairman, Regulatory Effectiveness of Unresolved Safety Issue A-45, Shutdown Decay Heat Removal Requirements," December 24, 2003, ADAMS Accession No. ML033510551.

7. Office of the Inspector General, Evaluation of NRCs Use of Probabilistic Risk Assessment (PRA) in Regulating the Commercial Nuclear Power Industry, OIG-06-A-24, September 29, 2006, ADAMS Accession No. ML062720275.

8. Dyer, J.E., NRC, letter to Anthony Pietrangelo, NEI, regarding how licensee PRA models that are updated to meet Regulatory Guide 1.200 can be factored into the NRC significance determination process, July 26, 2007, ADAMS Accession No. ML071990509.

9. NRC, Transcript of Commission Meeting on Risk-Informed Performance Based Regulation, August 2, 2007, ADAMS Accession No. ML072190233.

10. Fertel, Marvin S., NEI, letter to Chairman Dale Klein, U.S. Nuclear Regulatory Commission, regarding August 2 Commission Briefing on Risk-Informed Regulation, August 15, 2007, ADAMS Accession No. ML072290362.

11. Pietrangelo, Anthony, NEI, letter to Chairman Allison Macfarlane, NRC, Industry Support and Use of PRA and Risk-Informed Regulation, December 19, 2013, ADAMS Accession No. ML13354B997.

12. NRC, Proposal for Use of Licensee PRA Models in the Significance Determination Process, Enclosure 2, Handouts Discussed during the April 2, 2014 ROP WG Public Meeting, April 21, 2014, ADAMS Accession No. ML14106A571.

13. NRC, Summary of May 19, 2014, Public Meeting to Discuss Specific Issues Related to Risk Assessment Methods Used in the NRC Significance Determination Process, May 28, 2014, ADAMS Accession No. ML14148A455.

14. NRC, Public meeting to discuss specific issues related to risk assessment methods used in the NRC Significance Determination Process, Presentation, May 19, 2014, ADAMS Accession No. ML14141A361.

15. NRC, Perspectives on the NRCs Standardized Plant Analysis Risk (SPAR) Models, Presentation by Kevin Coyne at the RASP Handbook Public Meeting, May 19, 2014, ADAMS Accession No. ML14141A419.

16. Nuclear Energy Institute, Use of Licensee PRA Models in SDP Assessments, Presentation at the RASP Handbook Public Meeting, May 19, 2014, ADAMS Accession No. ML14149A064.

17. NRC, SECY-15-0124, Status of the Accident Sequence Precursor Program and the Standardized Plant Analysis Risk Models, Enclosure 2, Status of the Standardized Plant Analysis Risk Models, October 5, 2015, ML15188A106.

18. NRC, Briefing to RES Office Director and Deputy Director on Leveraging the Use of Licensees PRA Models in Lieu of SPAR Models, January 2016, ADAMS Accession No. ML16232A422.

19. NRC, Summary of February 9, 2016, Public Meeting to Continue Discussions Between the NRC and Industry Risk-Informed Steering Committees, ADAMS Accession No. ML16082A527.

20. Fewell, J. Bradley, Exelon, letter to William Dean, NRC, re: Dresden, Unit 3, Request for Improvements in NRC SPAR Model and RASP Handbook Use, January 12, 2017, ADAMS Accession No. ML17013A161.

21. Dean, William, NRC, letter to J.B. Fewell, Exelon, regarding SDP/SPAR models, March 21, 2017, ADAMS Accession Nos. ML17080A308, ML17066A245.

22. NRC, Staff Response to the Office of Inspector Generals Evaluation of Proposed NRC Modifications to the Probabilistic Risk Assessment Process (OIG-17-A-26),

October 18, 2017, ADAMS Accession No. ML17272A099.

23. NRC, Status Report on the Licensing Activities and Regulatory Duties of the U.S. Nuclear Regulatory Commission, February 2017, ADAMS Accession No. ML17025A395.