ML18169A205

Findings and Recommendations Report Regarding Office of Nuclear Reactor Regulation Risk-Informed Decision-Making Action Plan Tasks 1, 2, 3, and 4 (CAC No. A11008)

ML18169A205
Person / Time
Issue date:	06/26/2018
From:	Jason Paige Plant Licensing Branch II
To:	Laura Dudes Office of Nuclear Reactor Regulation
Paige J, 415-1474
References
CAC A11008
Download: ML18169A205 (127)

Text

June 26, 2018 MEMORANDUM TO: Laura A. Dudes, Acting Deputy Director for Engineering Office of Nuclear Reactor Regulation THRU: Kathryn M. Brock, Deputy Director /RA/

Division of Operating Reactor Licensing Office of Nuclear Reactor Regulation Russell N. Felts, Deputy Director /RA/

Division of Risk Assessment Office of Nuclear Reactor Regulation Eric J. Benner, Director /RA/

Division of Engineering Office of Nuclear Reactor Regulation Shana R. Helton, Acting Director /RA/

Division of Engineering and Infrastructure Office of New Reactors FROM: Jason C. Paige, Project Manager /RA/

Plant Licensing Branch II-1 Division of Operating Reactor Licensing Office of Nuclear Reactor Regulation

SUBJECT:

FINDINGS AND RECOMMENDATIONS REPORT REGARDING OFFICE OF NUCLEAR REACTOR REGULATION RISK-INFORMED DECISION-MAKING ACTION PLAN TASKS 1, 2, 3, AND 4 (CAC NO. A11008)

By memorandum dated June 29, 2017 (Agencywide Documents Access and Management System (ADAMS) Accession No. ML17180A061), the Director of the Office of Nuclear Reactor Regulation (NRR) provided direction related to the creation of a risk-informed decision-making (RIDM) project. The memorandum also included the projects mission statement, supporting objectives and tasks that the NRR staff should complete to enhance the integration of risk into our decision-making procedures and processes.

By revision dated May 4, 2018 (Agencywide Documents Access and Management System (ADAMS) Accession No. ML18116A023), the NRR staff issued the NRR RIDM action plan defining the NRR RIDM tasks and projects process, including the use of a systematic or

L. Dudes two-phased approach to successfully complete the project. Below is a summary of the eight NRR RIDM tasks that are being evaluated by the NRR staff.

NRR RIDM Task 1: Expand use of license review teams, including evaluating the technical reviewer, risk analyst, and project manager roles/responsibilities.

NRR RIDM Task 2: Broaden the definition of risk more transparently such that all of the technical staff can see how their work embodies risk considerations beyond core damage frequency and large early release frequency (LERF).

NRR RIDM Task 3: Develop a graded approach for using risk information more broadly in licensing reviews. Tasks 1 and 2 are related to NRR Task 3.

NRR RIDM Task 4: Review of Branch Technical Position (BTP) 8-8.

NRR RIDM Task 5: Evaluate Differing Professional Opinion (DPO) Recommendations 5 and 6, including evaluating the possibility to leverage the risk-informed notebooks as job aids for the staff.

NRR RIDM Task 6: Evaluate the guidance in the four pertinent documents (Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, RG 1.177, An Approach for Plant-Specific, Risk-Informed Decision-Making: Technical Specifications, RG 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities, and NUREG-1855, Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decision-Making) discussed in DPO Panel Recommendation 7 to determine if better harmonization is appropriate, and provide recommendations on a path forward, if appropriate.

NRR RIDM Task 7: With the impending revision to RG 1.174, determine whether there are any other pertinent documents or procedures that should be updated (e.g., RG 1.177).

NRR RIDM Task 8: Evaluate the use of LERF in various regulatory applications and determine if different approaches are supported: permanent license amendment versus one-time allowed outage time extension; significance determination process analysis (truncates LERF after 8 days);

consistency among regions when processing Notices of Enforcement Discretion.

As stated in the NRR RIDM action plan, Phase 1 is defined as the evaluation and analysis phase to support the tasks working groups findings and recommendations reports. Phase 1 of each task is considered complete with the completion of the reports. Enclosed are the findings and recommendations reports for NRR RIDM Tasks 1 and 3,1 2, and 4. NRR RIDM Tasks 5, 6, 7, and 8 were evaluated before the implementation of the NRR RIDM action plan; therefore, the 1

The findings and recommendations for NRR RIDM Tasks 1 and 3 are combined due to their interrelationships.

L. Dudes completion of these tasks are documented in their own separate memoranda. Enclosure 1 provides a summary of the NRR RIDM Tasks 1-4 recommendations and references to the closeout memoranda for NRR RIDM Tasks 5-8.

With the issuance of this memorandum, Phase 1 of the NRR RIDM action plan is complete and staff will proceed to Phase 2. Staff anticipates the first part of Phase 2 will be a scoping plan to integrate the recommendations in the Phase 1 report, and will prioritize and resource the next steps accordingly.

If you have any questions, please feel free to contact me.

Enclosures:

1. Summary of NRR RIDM Tasks 1-4 Recommendations

2. Findings and Recommendations Report Related to NRR RIDM Tasks 1 and 3

3. Findings and Recommendations Report Related to NRR RIDM Task 2

4. Findings and Recommendations Report Related to NRR RIDM Task 4

ML18169A205, Attachment to Enclosure 4 ML18169A214

• via e-mail OFFICE NRR/DORL/LPL2-1/PM NRR/DORL/LSPB/LA NRR/DORL/DD NRR/DRA/DD NAME JPaige JBurkhardt KBrock* RFelts*

DATE 6/15/18 6/20/18 6/22/18 6/21/18 OFFICE NRO/DEI/D(A) NRR/DE/D NRR/DORL/LPL2-1-PM NAME SHelton* EBenner* JPaige DATE 6/21/18 6/21/18 6/26/18 ENCLOSURE 1 Summary of NRR RIDM Tasks 1-4 Recommendations and References to Closeout Memoranda for NRR RIDM Tasks 5-8 Page 1 of 3

SUMMARY

OF NRR RIDM TASKS 1-4 RECOMMENDATIONS AND REFERENCES TO CLOSEOUT MEMORANDA FOR NRR RIDM TASKS 5-8 Office of Nuclear Reactor Regulation (NRR) Risk-Informed Decision-Making (RIDM) Tasks 1 and 3 Recommendations For the purposes of implementing an integrated review team approach, including processing licensing actions and tracking various metrics for Recommendation 1 those actions in the Replacement Reactor Program System (RRPS),

Type 1, 2, and 3 applications should be redefined for the purposes of the RIDM tasking.

Implement the integrated review team roles and responsibilities Recommendation 2 described in Attachment 1, Section 2.0 [Enclosure 1] on a trial basis from July through October 2018.

Implement the integrated review team processes described in Recommendation 3 Attachment 1, Sections 3.0, 4.0, and 5.0 [Enclosure 1] on a trial basis from July through October 2018.

After a trial implementation period from July through October 2018, staff will assess feedback and adjust the roles, responsibilities, and processes described in Attachment 1 [Enclosure 1] prior to updating Recommendation 4 office instructions, guidance, and training discussed in Attachment 3, which will be accomplished via the NRR ticketing process. The ticketing to update guidance, office instructions, and training will be created by December 31, 2018.

After a trial implementation period of the process described in Attachment 1, implement permanent RRPS changes described in Recommendation 5 Attachment 1, Section 6.0 [Enclosure 1], as modified based on feedback during the trial implementation period, via ticketing. The ticketing to update RRPS will be created by December 31, 2018.

By November 30, 2018, NRR divisions should determine whether their processes that were not covered within the scope of actions covered by Attachment 1 [Enclosure 1] could be enhanced using the integrated review team approach described in Attachment 1 or a similar process Recommendation 6 and develop separate plans for updating those processes, if applicable. If NRR divisions determine to enhance their processes, NRR will create tickets by December 31, 2018, to update those processes.

NRR RIDM Task 2 Recommendations Establish NUREG-2122, Glossary of Risk-Related Terms in Support Recommendation 1 of Risk- Informed Decision-making, as the main glossary for risk-related terms so that there is one authoritative source.

Page 2 of 3 Recognizing that the five Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Recommendation 2 Plant-Specific Changes to the Licensing Basis, principles are well known, develop guidance for use of other principles that are important to RIDM.

Develop a desktop glossary of terms and make available as a quick Recommendation 3 reference guide for more commonly used RIDM terms.

Enhance the current review guidance and office procedures to place Recommendation 4 greater emphasis on using risk insights.

Develop guidance (e.g., management directive) that identifies the hierarchy of RIDM documents, provides a roadmap on which RIDM Recommendation 5 documents to use, and complements current regulatory approaches, as appropriate.

Develop training for staff to facilitate using risk insights for reviews that Recommendation 6 dont require probabilistic risk assessment.

Augment position descriptions and performance appraisal elements Recommendation 7 and standards to include the use of risk information and risk insights to enhance decision-making.

NRR RIDM Task 4 Recommendations Provide training to staff on the expectation memorandum, Current Expectations for Using Existing Guidance for Reviewing License Recommendation 1 Amendment Requests for Diesel Generator Technical Specification Completion Time Extensions.

Implement the revisions to NUREG-0800, Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants Recommendation 2 (SRP) Branch Technical Position (BTP) 8-8, Onsite (Emergency Diesel Generators) and Offsite Power Sources AOT Extensions.

Revise SRP Section 16.1, Risk-Informed Decision-making: Technical Specifications, to include key points of the expectation memorandum, Recommendation 3 and to reference the current version of RG 1.177, An Approach for Plant-Specific, Risk-Informed Decision-making: Technical Specifications.

Revise RG 1.177, An Approach for Plant-Specific, Risk-Informed Decision-making: Technical Specifications, to address Recommendation 4 (1) long-duration completion times for one-time extensions and (2) backstop completion times.

All NRR staff should have risk-informed application training, training on RG 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities, RG 1.174, An Approach for Using Probabilistic Risk Assessment in Recommendation 5 Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, and RG 1.177, An Approach for Plant-Specific, Risk-Informed Decision-making: Technical Specifications, and associated refresher training on integrated reviews every 24 months.

All NRR staff should have training on how to review/disposition license Recommendation 6 amendment requests with risk insights.

Page 3 of 3 NRR RIDM Tasks 5-8 Closeout Memoranda Closure Memorandum, Risk Informed Decision Making Action Plan Tasks 5, 6, and 7 Recommendations, issued on March 15, 2018 NRR RIDM Task 5 (ADAMS Accession No. ML18009A219).

Memorandum, Pilot Program Regarding Lessons Learned, issued on June 11, 2018 (ADAMS Accession No. ML18116A623).

Closure Memorandum, Risk Informed Decision Making Action Plan NRR RIDM Task 6 Tasks 5, 6, and 7 Recommendations, issued on March 15, 2018 (ADAMS Accession No. ML18009A219).

NRR RIDM Task 7 Evaluating under NRR RIDM Tasks 1-4.

Memorandum, Consideration of Large Early Release Frequency in NRC Staff Evaluation of Licensee Requests for Enforcement NRR RIDM Task 8 Discretion, issued on June 26, 2018 (ADAMS Accession No. ML17348A758).

ENCLOSURE 2 Findings and Recommendations Report Related to NRR RIDM Action Plan Tasks 1 and 3 Page 1 of 7 FINDINGS AND RECOMMENDATIONS REPORT RELATED TO NRR RIDM ACTION PLAN TASKS 1 AND 3

1.0 INTRODUCTION

By revision dated May 4, 2018 (Agencywide Documents Access and Management System (ADAMS) Accession No. ML18116A023), the Office of Nuclear Reactor Regulation (NRR)

Risk-Informed Decision-Making (RIDM) action plan defines the NRR RIDM tasks and projects process, including the use of a systematic or two-phased approach to successfully complete the project. Phase 1 is defined as the evaluation and analysis phase to support the completion of the working groups findings and recommendations reports. This enclosure documents the findings and recommendations associated with NRR RIDM Task 1, Expand Use of License Review Teams, and Task 3, Develop a Graded Approach for Using Risk Information More Broadly in Licensing Reviews. The findings and recommendations for NRR RIDM Tasks 1 and 3 are combined in this report due to their interrelationships.

The working groups data collection consisted of identifying: (1) the scope of NRR licensing actions and major milestones associated with those actions that could be affected by Tasks 1 and 3, (2) licensing processes, procedures, and training that could be affected by Tasks 1 and 3, (3) Replacement Reactor Program System (RRPS) capabilities that could be affected by Tasks 1 and 3, and (4) lessons-learned from recent licensing actions related to Tasks 1 and 3.

The staff interviewed a subset of technical reviewers and project managers that worked on risk-informed submittals. The working groups reviewed examples of licensing actions with risk-informed elements, consulted with subject matter experts and peer reviewers, and reviewed draft products for ongoing licensing actions, as available. This report summarizes the insights from the data collection efforts and captures the working groups recommendations for consideration as the RIDM effort proceeds to Phase 2. The recommendations support Strategy 1 (evaluate and update guidance) and Strategy 2 (develop a graded approach) from SECY-17-0112, Plans for Increasing Staff Capabilities to Use Risk Information in Decision-Making Activities, dated November 13, 2017 (ADAMS Accession No. ML17270A192).

2.0 BACKGROUND

The NRR RIDM action plans description of Task 1 states, Expand use of license review teams that would team up risk analysts with technical staff vice relying on sequential or independent reviews. In addition, evaluate the roles and responsibilities for the technical reviewer, risk analyst, and project manager. The NRR RIDM action plans description of Task 3 states, Develop a graded approach for using risk information more broadly in licensing reviews.

Task 1 is related to project management structure and processes; whereas Task 3 is related to a review methodology.

In his memorandum dated June 29, 2017 (ADAMS Accession No. ML17180A061), the NRR Office Director tasked the staff to expand the use of license review teams consisting of technical staff and risk analysts. The desired outcome of this task is the development of a review team framework with supporting guidance and training for project management and technical staff for integrating risk-informed and deterministic reviews. The summary of the NRR executive and leadership team strategy meeting held on June 12, 2017, stated that regarding this task, the staff should inculcate more formally an approach to licensing reviews that would team up risk analysts with technical staff vice relying on sequential or independent reviews. This would Page 2 of 7 include joint conclusions and a safety evaluation (SE) developed by the team. The Task 1 working group determined that to accomplish Task 1, it would develop workload management tools for reviewing applications processed in NRR that will enable:

• Early and periodic review team communications to understand the purpose of the application, each reviewers scope and interdependencies among reviewers scopes, how the review will be accomplished, what conclusions and decisions need to be made and by whom, whether risk and traditional engineering insights can be used together and complement one another, and team member functional responsibilities.

• Development of a consolidated SE early in the process that outlines the integration of risk and traditional engineering insights.

• Early identification of issues (ideally during the acceptance review), management engagement, and path for resolution.

Successful implementation of the Task 1 recommendations will rely on NRR fostering a common understanding of concepts such as risk, probabilistic risk analysis, defense in-depth, safety margins, and compliance (see Enclosure 3, NRR RIDM Task 2 Findings and Recommendations Report), which will enable review teams to work from a common language and understanding to ease SE and conclusion development. Implementation of Task 1 recommendations also relies on the Task 3 tools for considering risk and traditional engineering insights to complete the technical evaluation of licensing reviews (i.e., how to identify when risk and technical insights can complement one another, and how to develop conclusions using risk and technical insights). The working groups for Tasks 1 and 3 combined their products and recommendations into a single report because of the tasks interdependencies.

The Task 3 working group used information and direction from the memorandum dated June 29, 2017, to develop a graded approach for using risk information more broadly in licensing reviews by establishing a repeatable framework for integrating technical and risk analyst licensing reviews. The Task 3 working group reviewed numerous licensing action examples that were an outcome from current practices, staff guidance, and generic communications. The Task 3 working group coordinated with the RIDM action plan working groups for NRR RIDM Tasks 1, 2, and 7 by consulting subject matter experts, obtaining peer reviews for products, and having diverse working group membership. The Task 3 working group concluded that challenges to implementing a repeatable and efficient review process could be addressed by developing the following products: a checklist for determining whether an integrated review team (IRT) should be used based on the type of application and level of risk information provided; a checklist for risk analysts and technical reviewers to use to determine available risk tools and insights that they can develop; and template SE language to assist with the development of consolidated SEs.

The working groups identified the scope of NRR licensing actions and major milestones associated with those actions that could be affected by the tasks; licensing processes, procedures, and training that could be affected by the tasks; RRPS capabilities that could be affected by the tasks; and lessons-learned from recent licensing actions related to the tasks.

The working groups interviewed a subset of technical reviewers and project managers (PMs) that worked on risk-informed submittals. The working groups reviewed examples of licensing actions with risk-informed elements, consulted with subject matter experts and peer reviewers to review the tasks products, and reviewed draft products for ongoing licensing actions, as available.

Page 3 of 7 describes a process for implementing an IRT approach to licensing actions, including changes to RRPS, based on the data collection results and findings. contains the checklists and SE guidelines produced by the Task 3 working group. contains the results from the working groups data collection efforts.

3.0 DISCUSSION The Task 1 and 3 working groups used the scope provided in the NRR Office Directors letter dated June 29, 2017, and developed processes to expand the use of license review teams and apply a graded approach to using risk in regulatory reviews in a consistent manner. A flowchart of these processes and how they are integrated is shown in Figure 1 below. Figure 1 represents the IRT and graded review approach framework, including key decision points for forming an IRT and the types of conclusions that the staff can develop based on the level of risk information provided in submittals or developed by the staff.

Figure 1. Integrated Review Team Process Flowchart Page 4 of 7 The process discussed below and in the attachments provides guidance for using licensee or U.S. Nuclear Regulatory Commission (NRC)-developed risk insights in licensing action reviews, even for applications that are not submitted in accordance with Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis. The NRR staff may use risk insights to inform and scope its review of proposed changes. For example, the NRR staff may want to discuss in its SE a safety context of the proposed change in regards to its contribution to core damage frequency (CDF), the subject components interactions during certain events, or the likelihood of the subject components failure. The NRR staff may also use risk information to adjust its focus on a certain aspect of a review.

Upon receipt of a licensee request (Box 1), the PM will review the request (i.e., an application or submittal) to identify the level of probabilistic risk assessment (PRA)-related information in the submittal (Decision 2). Based on the level of PRA information, the PM will categorize the submittal as being Type 1, 2, or 3. Recommendation 1 relates to defining the Type 1, 2, and 3 categories based on the level of PRA information in the licensees submittals. If the submittal includes any PRA information, such as CDF or large early release frequency (LERF) values, the PM will initially assign technical reviewer and risk analyst staff. If the submittal does not contain PRA information, then the PM will initially assign technical reviewers. The staff assigned initially will meet to determine if risk analyst support is needed or if the risk analysts need technical support. If the review team consists of both technical reviewers and risk analysts, then the NRR staff will use an IRT approach. If the staff uses an IRT approach, then the team will develop consolidated requests for additional information, audits, and SEs.

Recommendations 2 and 3 relate to the use of integrated review teams, including roles and responsibilities, and consolidated products.

The extent of risk information used in the staffs review will determine how the staff will form and document its regulatory decision. The working groups developed example SE language that the NRR staff can use as a guideline for documenting regulatory decisions informed by risk based on the level of risk information used. If the licensee submits a risk-informed application or submittal meeting the guidance of RG 1.174, then the result of the staffs review will be a regulatory decision that is based, in part, on PRA information. If submittals contain PRA information but are not risk-informed (i.e., not meeting the guidance of RG 1.174), the NRR staff may develop PRA or risk insights using its own risk models or tools. Generally, if PRA information or risk analysis is used for submittals that are not risk-informed, the NRR staff will document whether the PRA information or risk analysis is consistent with the regulatory decision. If the staff uses risk information that is not part of a full PRA to support the decision, the NRR staff will document whether the likelihood (probabilistic) information supports the regulatory decision. This language will not be a requirement for regulatory decision language; rather, it is intended to serve as guidance. The IRT will be responsible for crafting the SE language.

The NRR staff will have an opportunity to develop consistent SE language as it acquires experience from using the IRT approach. The NRR staff will use lessons learned from near-term implementation of the processes in this report prior to incorporation into formal procedures and training. Recommendation 4 relates to the use of a trial period prior to making permanent changes to procedures and training. The NRR staff will also employ continual learning and improvement after it updates procedures and training. The working groups also developed conceptual changes to RRPS for tracking metrics associated with the various submittal types and use of IRTs. Recommendation 5 relates to the use of a trial period prior to making permanent changes to RRPS.

Page 5 of 7 The working groups focused on a subset of licensing actions processed through NRRs Division of Operating Reactor Licensing (DORL) because of the limited timeframe to complete this project. Recommendation 6 relates to reviewing other NRR processes that can use from the IRT approach.

4.0 RECOMMENDATIONS Recommendation 1 For the purposes of implementing an IRT approach, including processing licensing actions and tracking various metrics for those actions in RRPS, Type 1, 2, and 3 applications should be defined as follows for the purposes of the RIDM tasking:

Type 1: Applications that do not contain PRA information.

Type 2: Applications that contain PRA information but are not submitted as risk-informed applications meeting the guidance of RG 1.174.

Type 3: Applications that are submitted as risk-informed and intended to meet the guidance of RG 1.174.

BASIS:

Qualitative risk insights may be present in applications in addition to quantitative risk insights. A licensing PM might have difficulty with identifying qualitative risk insights in an application if that PM does not have expertise in a specific technical area. For example, applying the type descriptions in SECY 17-0112, Plans for Increasing Staff Capabilities to Use Risk Information in Decision-Making Activities, dated November 13, 2017 (ADAMS Accession No. ML17270A197), would result in more technical and risk staff resources spent on assisting the PMs with identifying the application type because PMs may not know what qualitative risk insights will look like in various applications. However, a PM should be able to easily identify quantitative risk information (e.g., frequencies, probabilities, references to PRAs, etc.).

Therefore, defining the types of applications as described above for the purposes of the RIDM tasking will minimize resources and the burden on the PM on identifying and tracking the various types of applications and assignment of review teams.

Recommendation 2 Implement the IRT roles and responsibilities described in Attachment 1, Section 2.0 on a trial basis from July through October 2018.

BASIS:

The RIDM Task 1 description and supporting bases included defining roles and responsibilities for implementing a team structure that integrates the reviews of technical reviewers and risk analysts. Section 3.0 of Attachment 1 relies on the staff having the roles and responsibilities described in Section 2.0 of Attachment 1. A trial implementation period will provide for feedback and enhancements to the roles and responsibilities prior to incorporating into formal guidance or instructions.

Page 6 of 7 Recommendation 3 Implement the IRT processes described in Attachment 1, Sections 3.0, 4.0, and 5.0 on a trial basis from July through October 2018.

BASIS:

The process in Sections 3.0, 4.0, and 5.0 of Attachment 1 describes the project management and implementation of a team structure that will integrate risk and traditional engineering reviews and develop consolidated products. A trial implementation period will provide for feedback and enhancements to the process prior to incorporating into formal guidance or instructions.

Recommendation 4 After a trial implementation period from July through October 2018, NRR staff will assess feedback and adjust the roles, responsibilities, and processes described in Attachment 1 prior to updating office instructions, guidance, and training discussed in Attachment 3, which will be accomplished via the NRR ticketing process. The ticketing to update guidance, office instructions, and training will be created by December 31, 2018.

BASIS:

A trial implementation period will allow time to communicate the process to internal and external stakeholders, obtain feedback on the feasibility of the process, and evaluate the effectiveness and efficiency of the process. Feedback should result in process improvements prior to formal implementation into NRR office instructions and guidance.

Recommendation 5 After a trial implementation period of the process described in Attachment 1, implement permanent RRPS changes described in Attachment 1, Section 6.0, as modified, based on feedback during the trial implementation period, via ticketing. The ticketing to update RRPS will be created by December 31, 2018.

BASIS:

The RRPS changes require contractor assistance; therefore, the trial implementation period should provide some useful feedback regarding the feasibility of the proposed changes prior to implementing permanent changes and hiring contractors. If the process is formally adopted, then RRPS changes will be needed if management wishes to continue tracking the efficiency of reviews of Type 1, 2, and 3 submittals and those reviews with an IRT.

Recommendation 6 By November 30, 2018, NRR divisions should determine whether their processes that were not covered within the scope of actions covered by Attachment 1 could be enhanced using the IRT approach described in Attachment 1 or a similar process. If so, develop separate plans for updating those processes, if applicable. If NRR divisions determine to enhance their processes, NRR will create tickets by December 31, 2018, to update those processes.

Page 7 of 7 BASIS:

The Task 1 working group focused on a subset of licensing actions processed by DORL because of the limited timeframe given to accomplish the tasking. Processes and projects not considered include power uprates; improved standard technical specifications conversions; security-related, emergency plan, and quality assurance plan amendments; environmental reviews; license renewals and associated commitment reviews; orders; backfits; Technical Specification Task Force Travelers; and topical reports. Such processes might be able to use the IRT approach, if allowed by their regulatory structures.

Attachments:

1. Integrated Review Team Process

2. Review Tools for Applying a Graded Approach

3. Data Collection Findings and Results

, Attachment 1 Page 1 of 14 INTEGRATED REVIEW TEAM PROCESS NRR RIDM ACTION PLAN TASK 1

1.0 INTRODUCTION

Based on its review of the data collected, the Task 1 working group developed project management processes for the following licensing actions:

• routine unique plant-specific licensing actions that are requested via Type 1, 2, or 3 submittals,

• emergent licensing actions (i.e., emergency and exigent amendments and verbal relief requests), and

• multiple (large volume of, or industry-wide) licensing actions requested via Type 3 submittals (e.g., license amendment requests (LARs) to adopt risk-informed Technical Specification Task Force (TSTF) travelers or Section 50.69 of Title 10 of the Code of Federal Regulations (10 CFR)).

The process described in this attachment does not constitute a review methodology for integrating risk and traditional insights to complete a technical evaluation; rather, it describes a framework for forming the teams that will need to integrate the review products from technical reviewers and risk analysts. The use of the terms technical branches, technical reviewers, or traditional engineering refers to staff that are not qualified probabilistic risk assessment (PRA) analysts or reliability experts. The use of the terms risk analyst or PRA analysts refers to qualified PRA analysts or reliability experts, who would typically be assigned to PRA Licensing Branch A or B (APLA or APLB) in the Division of Risk Assessment (DRA) of the Office of Nuclear Reactor Regulation (NRR).

The use of the term, integrated review team refers to a review team consisting of at least one project manager (PM), a technical reviewer, and a risk analyst, whereby the team members will be responsible for developing requests for additional information (RAIs), audit plans, and safety evaluations (SEs). If a branch has a concurrence-only role, an integrated review team (IRT) might not be necessary. An IRT can be created to review Type 1, 2, or 3 submittals.

2.0 TEAM MEMBER ROLES AND RESPONSIBILITIES The following roles, responsibilities, knowledge, skills, and abilities shall be incorporated into applicable position descriptions, guidance documents, qualification plans, and training classes for IRTs.

2.1 Division of Operating Reactor Licensing Plant PM The Division of Operating Reactor Licensing (DORL) Plant PM will facilitate team formation, meetings, written products, and meeting milestones. The Plant PM will have a leadership role in consolidated products (e.g., RAIs and SEs) by ensuring that the products are consistent with office instructions and training courses involving the fundamentals of reactor licensing. The Plant PM will ensure that the consolidated SE conforms to the guidance in LIC-101, License Amendment Review Procedures, Appendix B, Attachment 2 or LIC-102, Relief Request

, Attachment 1 Page 2 of 14 Reviews. The PM will ascertain the teams abilities with using available information technology (IT) tools, such as ADAMS, SharePoint, Office 365, Skype, and Go-To-Meetings to determine which platform would best suit the team for consolidated products and accommodating remote workers. The Plant PMs will be familiar with various IT tools, such as ADAMS, SharePoint, Office 365, Skype, and Go-To-Meetings. Plant PMs will be familiar with using risk insights in licensing reviews, various risk-informed licensing initiatives (e.g., TSTFs and 10 CFR 50.69 LARs), and the U.S. Nuclear Regulatory Commission (NRC) Regulatory Guides (RGs) for risk-informed reviews (e.g., RG 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, RG 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities, etc.) to better inform team formation and review needs. The Plant PMs will facilitate issue resolution and escalation to management. The Plant PMs will be familiar with the Type 1, 2, and 3 categorizations. Plant PMs will communicate the status of and any major issues arising from integrated reviews to the DORL Review Facilitator (see below).

2.2 DORL Review Facilitator The DORL Review Facilitator (RF) will provide an advisory role for routine licensing actions as a collateral duty until management determines that any cultural and process challenges with integrating risk and traditional engineering reviews are addressed. The DORL RF would not be assigned to every project; rather, the DORL RF would serve an advisory role, as needed. The DORL RF will provide a leadership role in coordinating the reviews for a concurrent large volume of similar risk-informed licensing requests (e.g., TSTF LARs, 10 CFR 50.69 LARs, etc.).

The DORL RF will also communicate the status of and major issues arising from integrated reviews to senior management. The DORL RF will be familiar with the planned submittals list and precedents involving integrated review teams (e.g., knowledge of lessons-learned). The DORL RF will provide assistance, guidance, and recommendations, as needed, to Plant PMs on how to document integrated risk and traditional engineering insights. The DORL RF will need to be familiar with the Type 1, 2, and 3 categorizations. It is not expected that the DORL RF have PRA expertise.

2.3 Technical Reviewers Technical reviewers will be familiar with the Type 1, 2, and 3 categorizations. Technical reviewers will be familiar with the application of risk in licensing reviews to assist Plant PMs with team formation, integrate risk and traditional engineering reviews, and enable early identification of the need to involve risk analysts with the reviews. Technical reviewers will understand the various SE formats and guidance in LIC-101 and LIC-102. Technical reviewers will serve a technical coordination role in the integration of risk and traditional engineering insights.

Technical reviewers will be familiar with various IT tools, such as ADAMS, SharePoint, Office 365, Skype, and Go-To-Meetings.

2.4 Risk Analysts Risk analysts will be experts in identifying the Type 1, 2, and 3 categorizations. Risk analysts will assist Plant PMs with team formation and the technical reviewers with integrating the risk and traditional engineering reviews. Risk analysts will understand the various SE formats and guidance in LIC-101 and LIC-102 to assist with the consolidated SE format. Risk analysts will serve a technical coordination role in the integration of PRA and traditional engineering insights.

Risk analysts will be familiar with various IT tools, such as ADAMS, SharePoint, Office 365,

, Attachment 1 Page 3 of 14 Skype, and Go-to-Meetings. With Plant PMs assistance, risk analysts will be able to identify which technical branch or discipline will be needed to review RG 1.174 applications.

2.5 Management The NRR management will maintain awareness of industry initiatives that will result in a large volume of similar risk-informed applications and allot the resources needed to efficiently address those applications. Management will ensure that the PMs, technical reviewers, and risk analysts are aware of such situations and will communicate with management peers through workload management meetings. Branch chiefs will review the interim or draft consolidated SEs to ensure the adequacy of the review methodology and documentation approach and adjust resources or redirect the staff as needed. Management will ensure the staff is sufficiently trained in the integrated review approach. Management will assist, as needed, in resolving differing views that may arise during the integration of risk and traditional engineering insights.

3.0 ROUTINE UNIQUE PLANT-SPECIFIC LICENSING ACTIONS The following process description is for establishing IRTs for routine unique plant-specific licensing actions submitted as Type 1, 2, or 3 applications from licensees. The process description covers submittal type designation, staffing assignments, determination of whether to use an IRT, and consolidated RAI and SE development. The processes described in established and applicable NRR office instructions apply if not addressed by the following description. The term routine means that the application is not an emergency or exigent (i.e.,

emergent) request or a verbal relief request.

3.1 Submittal Type Determination The following process for submittal type determination (see the green block in Figure 1 of Section 6.0) applies when the submittal is received to when the PM creates the project in RRPS and makes initial branch assignments.

A. Within the typical timeliness for adding a project to Replacement Reactor Program System (RRPS), when an application is received, the Plant PM will review the application to determine whether it is a Type 1, 2, or 3 application based on the level of quantitative PRA information in the application (e.g., core damage frequencies (CDFs),

large early release frequencies (LERFs), initiating event frequencies, references to PRAs, etc.). For large applications, doing a Find for PRA terms or asking the licensee during routine discussions can be helpful. If the Plant PM is uncertain how to determine the type of application, the Plant PM can consult with the DORL RF and/or DRA. Until or unless RRPS is modified to designate a Project as Type 1, 2 or 3, the Plant PM will add Type 1, Type 2, or Type 3, as applicable, in RRPS, in the Other Considerations field (which is within the Project Details tab and then the Project Attributes section) when opening a project. For Type 3 applications, the Plant PM should consult with the DORL RF, as needed, to determine which branches have reviewed similar applications in the past. For the purposes of implementing this step, Type 1, 2, and 3 applications are defined as follows:

Type 1: Applications that do not contain PRA information.

Type 2: Applications that contain PRA information but are not submitted as risk-informed applications meeting the guidance of RG 1.174.

, Attachment 1 Page 4 of 14 Type 3: Applications that are submitted as risk-informed and intended to meet the guidance of RG 1.174.

B. The Plant PM will assign branches in RRPS as follows:

• Routine Type 1: Assign technical branches but not the PRA branches. The need for risk analysts will be determined after meeting with the technical reviewer(s) or after the PM determines that it needs assistance with the no significant hazards consideration (NSHC) determination regarding frequency of accidents. The Plant PM may assign DRA at this stage if it is already known that the technical staff will want a consideration of risk insights (e.g., if this was discussed after a presubmittal meeting).

• Routine Type 2 or Emergent: Assign both PRA and technical branches.

• Routine Type 3: Assign PRA and technical branches. If technical branch assistance is not needed, the reviewer can be removed later from the project. Assigning the technical branches at the start of the project will provide the technical branches an early opportunity to raise concerns with risk-informed applications.

3.2 Integrated Review Team Determination The following process for team formation and project scoping is applicable from the time the initial set of reviewers are assigned in RRPS to when the acceptance review is issued. The major differences between this process and the current expectations and guidance include explicit instructions to hold team meetings prior to issuance of the acceptance review to verify team composition, scopes of review, and any issues, regardless of the hours estimate, and the possibility that more than 25 days may be needed to do an acceptance review for an IRT approach.

A. The Plant PM will hold an initial Staffing Meeting with the assigned reviewers, regardless of the estimated hours, within 10 working days of the application being added to ADAMS to determine if additional branches should be added to or consulted about the review.

The Plant PM and assigned reviewers should use the checklists in Attachment 2 for assistance. Assigned reviewers should use RRPS to identify other branches needed and start recommending scopes of review. However, the PM will confirm with the reviewers via a meeting to understand the bases for their recommendations. The Plant PM should consider inviting the DORL RF to the Staffing Meeting to assist, if needed.

For Type 1 applications, the Staffing Meeting will be used to determine if risk analysts or any other technical branches should be assigned to the review using the checklists in Attachment 2 for assistance. If risk analysts will not be used (i.e., the review team agrees that it does not want risk insights from DRA), then staff should follow the acceptance review processes established via applicable NRR office instructions. For Type 2 applications, the Staffing Meeting will be used to confirm that assigned branches will remain on the review and whether to assign other branches using the checklists in Attachment 2 for assistance. For Type 3 applications, the Staffing Meeting will be used to confirm whether the technical branches will remain on the review and whether to assign other branches. If technical staff will not be assigned to Type 3 reviews, then staff should follow the acceptance review processes established via applicable NRR office instructions.

, Attachment 1 Page 5 of 14 B. After the Staffing Meeting, the Plant PM will use RRPS to assign other branches to the review, as needed. It is not expected that, at this stage, the extent of each reviewers scope is known; therefore, it is possible that at a later time, some reviewers may be still be added, denoted as having a concurrence-only scope, or no longer needed.

However, the meeting provides assurance that the reviewers have read the application and have considered whether risk insights will be used.

C. After the review team is staffed and assigned in RRPS, the Plant PM will hold a Scoping Meeting, regardless of the estimated hours, within 15 working days of the application being added to ADAMS. The following will occur during the Scoping Meeting:

• The Plant PM will discuss the application, including all of the proposed changes that need evaluated; plant-specific issues or background information (e.g., licensing and design basis, stakeholder interests, time-charging expectations, or generic RAI and SE expectations). The PM will also identify if there are other ongoing reviews for applications that may overlap with the current application (e.g., multiple risk-informed applications from the same plant that use the same PRA information).

• Team members will discuss their expertise as it pertains to the review.

• The Plant PM will introduce any scoping recommendations with input from the technical reviewers and risk analysts. The Plant PM should also identify whether it will need the teams help with the NSHC determination (e.g., whether the change could affect the frequency of accidents).

• For Type 1 and 2 applications, the team will decide if DRA is to be on the review using the checklists in Attachment 2. The DRA division must be on the review if the staff intends to document qualitative risk insights in the SE.

• For each change identified in the submittal, the review team will confirm each change proposed by the licensee, which team member is reviewing each of the change(s),

and the scope of each team members review. The Plant PM will ensure that the entire application (i.e., every proposed change) has an assigned reviewer. The team members will identify what information, evaluations, and conclusions the reviewers will need from each other to help inform the hours estimates. This action may take more than one meeting because it requires the staff to understand how risk and traditional engineering insights can be integrated.

• With assistance from the review team, the Plant PM will document the proposed changes and reviewer assignments and note the scoping assignments in an RRPS comment. At this time, the PM will denote in RRPS whether an IRT approach will be used. Until or unless RRPS is modified to designate that an IRT is being used, the Plant PM will type Integrated Review Team if applicable, in RRPS, in the Other Considerations field (which is within the Project Details tab and then the Project Attributes section) after the Type designation (e.g., Type 2, Integrated Review Team). If it is determined that both risk analysts and technical reviewers are providing input to the review, or if the risk analysts are assisting with documenting qualitative risk insights, then an IRT approach will be used. If either the risk analysts or technical reviewers are on concurrence-only, then an integrated team approach is not necessary; however, the staff can and should consult with each other throughout the review to ensure any issues are identified and resolved early in the process.

, Attachment 1 Page 6 of 14 D. Within 20 working days2 after the application is added to ADAMS, the Plant PM will hold an Acceptance Review Meeting. This meeting will cover the following topics:

• Hours estimates and the need for a metric exclusion memorandum. With the IRT approach, the average hours per review may increase; therefore, the hours estimates per reviewer in the expectations memorandum (ADAMS Accession No. ML16202A029) may no longer be accurate, and the total hours estimate for the review may no longer be a sufficient gauge to determine whether the review may be complex.

• Acceptance review issues (e.g., information insufficiencies).

• Challenges to meeting the acceptance review metric. This metric is currently set at 25 working days; however, additional time may be needed to understand how risk insights will be used in the review for Type 1 and 2 applications and the scope of technical staff reviews for Type 3 applications.

• The IT tools (e.g., e-mail, ADAMS, SharePoint, or Office 365) to be used to collectively develop the consolidated RAIs and SE. The team will also determine the need for Skype or Go-To-Meetings for teleworking staff.

E. Prior to issuing the acceptance review results to the licensee, the PM may hold additional team meetings or discussions to finalize any acceptance review issues, and to confirm hour estimates, review scopes, and milestone dates. Reviews with an IRT might exceed the 25-day metric for acceptance reviews because the review team will need to understand how it plans to use risk insights or how technical staff will contribute to Type 3 reviews. In these instances, the staff should consider applying the guidance in NRR Office Instruction LIC-109, Acceptance Review Procedures, which states, in part, If there are factors that would justify a longer review period for the acceptance review, the PM or [reactor engineer (RE)] and technical staff must obtain approval of their Division management and respective technical Division Director (DD). During a trial period (July through October 2018), RRPS will not be modified, nor will the acceptance review metric be changed. However, the PM should document in the comments section the basis for why more than 25 days is needed in RRPS, if applicable. This issue will be revisited after a trial period, if sufficient experience can be acquired; otherwise, management may determine to revisit this issue as a result of continuous learning and improvement initiatives.

3.3 Consolidated SE and RAI Process The following process is applicable from the time the acceptance review is issued to when the project is complete. The following process steps are for the development of the Consolidated SE and RAIs; however, the process does not describe a specific review methodology for how to apply risk insights. The major differences between this process and the current process are:

(1) additional team meetings to develop the consolidated products, (2) increased Plant PM resources to coordinate the overall review, (3) increased technical reviewer and risk analyst 2

The technical staffs acceptance review is due to the PM within 20 days. The PM may hold the meeting prior to this date if efficiencies can be gained (e.g., if previous meetings or discussions can be held sooner).

, Attachment 1 Page 7 of 14 resources to develop an integrated review methodology, (4) branch chief review of draft consolidated SE outlines, and (5) the RAI and SE concurrence process. The consolidated SE process will involve elimination of branch-specific SE inputs; instead, the PM will send one SE package for concurrence that will have the technical reviewers, risk analysts, licensing assistant, branch chiefs, Office of the General Counsel (OGC), and DORL concurrence. The RAIs will continue to be created per branch; however, the RAIs will be reviewed for duplication or overlap and consolidated into one document without the branch name in the RAI identifier.

A. Within 30 working days of the acceptance review being sent to the licensee, the review team will hold a Consolidated SE Development Meeting to confirm the draft SE outline and SE development approach. Given that the scopes of review were determined during the acceptance review process, the review team should already have a conceptual understanding of the SE outline. The review team can use the SE language guidance in Attachment 2 to assist. The goal of the meeting is to get alignment on the contents and conclusions that need to go into the Technical Evaluation section of the SE and how risk will be integrated with traditional engineering evaluations.

In order to ensure that the technical evaluation will encompass all proposed changes, the Introduction and Regulatory Evaluation sections of the SE (or the exemption criteria) should be developed first. Therefore, prior to this meeting, the following should occur:

i. The technical reviewers and risk analysts will meet to (1) coordinate and develop the review methodology and (2) nominate an overall technical coordinator (if not the Plant PM) for the risk and traditional engineering SE inputs. The review team can use the Attachment 2 guidance to assist. If the application is a Type 3 application, the risk analyst will likely be the technical coordinator; whereas for Type 1 and 2 applications, the technical reviewer will likely be the technical coordinator. The technical coordinator will be responsible for (1) ensuring the review inputs are aligned and done in accordance with the agreed upon review methodology, (2) addressing any repetitive or missing information in the consolidated SE and RAIs, and (3) developing audit plans to submit to the Plant PM. The Plant PM and DORL RF may be used for assistance.

ii. The technical reviewers and risk analysts will prepare their exemption input outline or their SE input outline for the subsection of the Regulatory Evaluation (for amendments and relief requests) that lists the applicable regulations, design and licensing basis information, and guidance documents used for this review (e.g., Section 2.3 for license amendments). The technical reviewers and risk analysts should also develop an outline of the Technical Evaluation SE section (e.g., Section 3.0 for amendment SEs) that describes the method(s) of their reviews, which changes are being reviewed, what conclusions have to be made about the change, and which review standards will be used. The first paragraph in the Technical Evaluation section will generically describe the method of review. The technical reviewers and risk analysts will share their consolidated SE or exemption outline with the Plant PM 10 working days prior to the first Consolidated SE Development Meeting. It is not expected that the write-ups be in final draft quality at this stage.

iii. While the technical reviewers and risk analysts are assembling their outlines and methods of review, the Plant PM will maintain control of the overall final consolidated SE to ensure that it meets the LIC-101, Appendix B, Attachment 2

, Attachment 1 Page 8 of 14 (or LIC-102) guidance for format, organization, and contents of an SE, or LIC-103, Exemptions from NRC Regulations, for format, organization, and contents of an exemption package. The Plant PM will develop a draft amendment, exemption, or relief request package that includes, as needed, the cover letter, amendment pages, any clean operating license and technical specification (TS) pages if already provided in the application (or use the mark-ups as a placeholder), a draft SE outline that follows the format of the applicable office instruction, and Federal Register notices, as applicable. The Plant PM will start to fill in the SE Introduction and the parts of the Regulatory Evaluation that cover the background information about the application subject (i.e., the structures, systems, and components (SSCs) or the TSTF traveler involved, current requirements, and a complete description of proposed changes to ensure every change is covered in the SE). Most of this information should already be in the licensees applications. The Plant PM should assist with developing the list of regulatory requirements to the extent practicable (e.g., the Plant PM can identify whether the plant was licensed in accordance with the general design criteria (GDC) of Appendix A, General Design Criteria for Nuclear Power Plants, to 10 CFR Part 50, or if the plant is a pre-GDC plant, then the Plant PM can describe where the plants GDC are located in the updated final safety analysis report (UFSAR)). The Plant PM will also add a placeholder for the Technical Evaluation section, a Commitments section, if applicable, a State Consultation section with placeholder language, if applicable, and a Conclusion section. Relief requests and exemptions will have different sections -

essentially, the Plant PM will describe what is in the application so that the review team can focus on doing the technical evaluation rather than writing up background; however, the team should ensure these sections are adequate to verify understanding of the application and that every proposed change will be reviewed.

iv. After the technical coordinator sends the Plant PM the technical evaluation outline, the Plant PM will consolidate this and any other reviewers draft inputs and outlines into the amendment or relief request SE package that the PM started. The Plant PM will share the consolidated draft with the review team 5 working days prior to the meeting. It is not expected that the draft SE be complete at this stage; rather, that the major components of the licensing action package and SE subsection outlines be developed to facilitate the Consolidated SE Development Meeting discussion. The PM should use ADAMS, SharePoint, or Office 365 to share the consolidated SE to facilitate the meetings and/or writing sessions and so that the team can be aware of the current status of the consolidated SE. The PM should also schedule a meeting where a laptop and projector can be used to view and edit the draft consolidated SE (or use Go-To-Meeting, Webinar, or Skype for remote meetings).

B. At the first Consolidated SE Development Meeting:

i. The review team will establish the ground rules for document version control when consolidating SE inputs. For example, the technical reviewers and risk analysts may want to do their work on their P-drives until ready to send a version to the technical coordinator or Plant PM.

, Attachment 1 Page 9 of 14 ii. The review team will ensure that the Regulatory Evaluation section information is adequate and captures all of the licensees proposed changes. The Plant PM may need assistance from the review team to complete this section after the meeting.

iii. The review team will discuss the content and format of the Technical Evaluation section, with focus on the method of review (e.g., using RG 1.174 methodology or some other methodology and establishing acceptance criteria and the conclusions that have to be made), and a high level discussion of how risk insights are used in the review. This information should be documented in the beginning of the Technical Evaluation section (e.g., 3.1, Method of Review).

iv. The review team will also discuss the order of topics in the Technical Evaluation section and what content needs to go into those subsections.

v. Steps i through iv may take several meetings. After the SE outline and method of review are developed, the Plant PM will share the draft consolidated SE with the branch chiefs, who will confirm with the Plant PM (verbally or e-mail) that the review methodology and draft SE outline is adequate. The purpose of this step is to ensure that management supports the review approach early in the process to reduce the potential for any late-breaking major changes in direction or staffing.

The review team should highlight which SE sections are applicable to which branch to streamline the branch chief review; however, the branch chiefs can review the entire document, if they wish.

C. After the draft consolidated SE or exemption outline is developed, the review team should meet every 2 to 4 weeks, or as needed, to have a draft consolidated SE or exemption developed within 4 calendar months of the acceptance review being issued.

The technical reviewers and risk analysts can meet more often without the Plant PM, as needed. While the draft consolidated SE is being developed, the Plant PM with assistance from the technical coordinator will keep a consolidated list of draft RAIs being developed. These meetings are intended to accomplish the following:

i. Determine the need for an audit.

ii. Develop a consolidated list of RAIs. Each branchs RAIs will be consolidated into one set of RAIs. The technical coordinator will determine which RAIs may need revising to address repetition and will advise the team about which RAIs can be consolidated. With assistance from the technical coordinator, the Plant PM will track which branches and reviewers the RAIs came from to enable faster concurrences from the branch chiefs and traceability. The branch chiefs will concur on the final set of consolidated RAIs, but should focus their concurrence review on the RAIs their branch developed. Therefore, the version being sent to the branch chiefs for concurrence should have branch-specific input highlighted for that branch chief. After branch chief concurrences, the Plant PM will develop the formal set of RAIs from the consolidated list to send to the licensee. This consolidated version should not identify specific NRC review branches.

iii. Discuss the status of the SE development and any new issues or concerns.

, Attachment 1 Page 10 of 14 iv. Determine if management involvement is needed (e.g., if the team is having difficulty developing a conclusion about acceptability based on the integration of traditional engineering and risk insights).

v. Determine if obtaining DORL RF comments (not concurrence) on the draft SE is beneficial.

D. Within 10 calendar days of the draft consolidated SE/exemption and RAI development, the Plant PM will obtain concurrences from the branch chiefs on the draft RAIs and issue the draft RAIs to the licensee. Within LIC-101s RAI timelines, the Plant PM will hold any clarification calls and issue the final set of RAIs to the licensee and obtain RAI responses from the licensee.

E. Within 10 calendar days after the RAI response is received, the review team will meet to discuss the RAI responses and any other review issues and continue developing the consolidated SE.

F. Within 8 calendar months, the consolidated SE/exemption will be finalized by the review team, and the Plant PM will obtain technical and risk branch chief concurrences preferably by e-mail, in addition to the DORL licensing assistant review. The Plant PM should highlight within the SE (e.g., using comment bubbles) which SE portions pertain to which branch; however, the branch chiefs can review the package in its entirety.

Once the consolidated SE is developed and concurred on by the reviewers and branch chiefs, the Plant PM will process the package through remaining legal and management reviews. This results in one product with concurrences from all staff involved rather than separate SE inputs from specific branches. Only branch chiefs will be listed in the concurrence block for branches designated as concurrence only in RRPS.

The following timelines in Figure 1 compare the current milestones, goals, and 1-year Congressional budget justification metric for timeliness of routine licensing actions and the IRT timeline. Additional time may be needed during the acceptance review stage for IRTs, at least upon initial implementation. DORL management increases monitoring of those actions older than 9 months; therefore, the schedules shown in Figure 1 show milestones that would support a 9-month completion goal. If an audit or subsequent rounds of RAIs are needed or if significant or complex review issues arise, the schedule may change, and the review may go past 9 months, as is the case with the current review process. The 9-month goal is not intended to be a requirement.

, Attachment 1 Page 11 of 14 Figure 1. Review Timelines for Traditional and Integrated Review Teams 4.0 EMERGENT LICENSING ACTIONS Given the time constraints to process an emergent action (i.e., an emergency or exigent amendment request or a verbal relief request), NRR staff will not be able to complete the steps in Section 3 of this attachment for an IRT approach within the timeframes provided for a routine licensing action. The guidance in Section 7 of LIC-101, Revision 5 may discourage the staff from considering risk in emergent actions; therefore, the LIC-101 guidance will be revised to not discourage such reviews. For example, a review team can consider the RG 1.174 guidance regarding adjusting the emphasis on PRA scope and other attributes for limited applications for emergent licensing actions. Another approach may involve using recently completed reviews of risk-informed licensing actions for that licensee. The Plant PM should consider early inclusion of DRA staff. The Task 3 products described in Attachment 3 will assist with integrating risk more efficiently into the review of emergent actions.

The NRR staff will maintain the current processes and practices for emergent actions but apply the IRT approach to the extent practicable. For an emergent licensing action, the Plant PM should involve the technical and risk branches, DORL RF, branch chiefs, and senior executive service (SES) managers in an initial and subsequent staffing and scoping meetings as soon as practical to understand the methodology for the review and who will be reviewing which aspects of the proposed changes. The review team should decide whether there is sufficient time to develop a consolidated SE at the start of the review and if individual branch SE inputs are needed or preferred. The Plant PM or a technical coordinator will need to consolidate the SE

, Attachment 1 Page 12 of 14 inputs into a LIC-101, Appendix B or LIC-102 SE format; therefore, the Plant PM with assistance from the review team will begin developing the consolidated SE outline at the start of the review.

The review team can use the Attachment 2 SE guidelines to assist.

5.0 MULTIPLE LICENSING ACTIONS - TYPE 3 The following process applies for project managing multiple Type 3 applications of the same subject (e.g., a specific TSTF Traveler or to adopt 10 CFR 50.69). Reviews for these actions follow the process described in Section 3.0, with the following additions to the project management structure.

A. DORL management, with assistance from the DORL RF and the technical assistant(s),

will periodically evaluate the planned submittals list maintained on the DORL SharePoint site and maintain communication with the nuclear power industry to ascertain when licensees intend to submit a large volume of similar Type 3 submittals over a limited time frame, as this will challenge NRR resources. This information will be communicated to the technical and risk divisions and the DORL Plant PMs.

B. DORL management will assign a PM (likely the DORL RF) to project manage the group of submittals. DRA and technical staff management will assign technical coordinator(s) that will be responsible for reviewing the applications to establish consistency among the reviews and help coordinate the technical evaluation. The DORL PM will work with the risk and technical reviewers to develop the following:

• Communications strategy that covers: informing DORL Plant PMs, technical reviewers, and risk analysts about the submittal topic and schedules; management and staff briefings and updates; and a review schedule (possible negotiation with industry to stagger submittals).

• Review team staffing (ideally the same set of risk analysts and technical reviewers would be assigned).

• Review methodology that outlines how risk will be used with traditional engineering insights and what conclusions need to be made.

• Model SE (staff will need to determine whether to treat the first submittal as a pilot for this initiative prior to the application, allow the first submittal to take the brunt of the resources to develop the model SE, or develop a non-fee billable EPID for this effort).

C. Upon receipt of the first application, the Plant PM will process the application in RRPS and verify with the DORL RF that the application may be the first of many or one of a group of submittals. The DORL RF will contact the technical branches to determine their involvement in the review. The Plant PM should be involved in the scoping process given that the Plant PM can offer insights regarding site-specific variations from the TSTF traveler or generic application. Subsequent applications will take advantage of the technical branch staffing assignments on the first application; the DORL RF will make the reviewer assignment information available to the division.

D. The review team will follow the process in Section 3.0 of this attachment for a routine Type 3 application; however, the DORL RF will be involved. The Plant PM will be

, Attachment 1 Page 13 of 14 responsible for communications with the licensee and processing documents (e.g.,

issuing RAIs and the SEs).

• For the first application, using the Task 3 SE guidance (Attachment 2), the DORL RF, technical reviewers, and risk analysts will develop a consolidated SE that can be used as a model for other submittals. However, the Plant PM will control the development of plant-specific SEs with DORL RF assistance and review. The use of a technical coordinator can be used, similar to the National Fire Protection Association (NFPA) Standard 805 reviews.

• The DORL RF and technical coordinators will maintain awareness of RAIs to determine if any will have generic applicability.

• The DORL RF will project manage generic issues with the TSTF or applications.

The DORL RF will also track the volume of applications and coordinate the overall review schedules and priorities for multiple LARs. The DORL RF will also establish routine meetings for the review teams and management to go over review status and generic issues.

6.0 RRPS CHANGES The following RRPS changes will assist with project managing and monitoring the efficiency of the IRT approach.

A. The RRPS will enable the assignment of applications as Type 1, 2, or 3 so that the review team understands what level of risk information is provided in submittals and resources can be tracked for each type of submittal. For Type 3 submittals, RRPS will provide another drop-down menu to select a category of Type 3 submittals (e.g., specific TSTF travelers, 10 CFR 50.69 LARs, etc.).

B. The RRPS will enable the Plant PM to designate whether an integrated team approach is being used. This designator would be assigned after project creation but before the acceptance review is issued. This designator (e.g., Integrated Review Team) can replace the Risk-informed Project Attribute in RRPS because risk-informed applications will be Type 3.

C. The RRPS will allow the Plant PM to assign new milestones, if desired, if an IRT is used.

These new milestones would include those such as: Staffing Meeting, Scoping Meeting, Acceptance Review Meeting, First Consolidated SE Development Meeting, Consolidated RAI Issued, Consolidated SE Issued, in addition to others needed by the team.

D. The RRPS will enable reports to be ran on metrics associated with each Type of review and on the use of IRTs.

Figure 2 is a conceptual representation of the proposed RRPS changes, as described above.

, Attachment 1 Page 14 of 14 Figure 2. RRPS Changes for Integrated Review Teams

, Attachment 2 Page 1 of 11 REVIEW TOOLS FOR APPLYING A GRADED APPROACH NRR RIDM TASK 3

1.0 INTRODUCTION

The Task 3 working group concluded that a graded approach for using risk information more broadly in licensing reviews could be implemented in a repeatable and efficient review process by developing the following products: a checklist for determining whether an integrated review team (IRT) should be used; a checklist for risk analysts and technical reviewers to use to determine available risk tools and insights that they can develop; and template safety evaluation (SE) language to assist with the development of consolidated SEs.

2.0 INTEGRATED REVIEW TEAM CHECKLIST As discussed in Attachment 1, when the U.S. Nuclear Regulatory Commission (NRC) receives an application, the Plant PM will determine the type of submittal based on the level of probabilistic risk assessment (PRA) information and which branches to initially assign to the review. The team then meets to determine if additional reviewers are needed and if an IRT should be used. The Task 3 working group developed a checklist that will assist the teams with determining whether to apply an IRT. This checklist (i.e., Table 1 below) can be used for Type 1, 2, and 3 submittals, including license amendments, relief requests (or proposed alternatives), and exemption requests. The categories in the checklist correspond to the decision to form an IRT and do not necessarily correspond to Type 1, 2, and 3 applications (e.g., Category 1 can apply to Type 3 applications).

Table 1, Integrated Review Team Checklist Category Description of Review or Submittal Topic I. Integrated A. Risk-informed reviews in accordance with Regulatory Guide Review Team (RG) 1.174, An Approach for Using Probabilistic Risk Assessment Recommended in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis B. Requests to adopt Risk-informed Technical Specifications Task Force (TSTF) travelers (e.g., Consolidated Line Item Improvement Process, or CLIIPs), unless risk considerations were taken into account during the TSTF approval process C. Submittals that include risk information, but not in accordance with RG 1.174

, Attachment 2 Page 2 of 11 Table 1, Integrated Review Team Checklist Category Description of Review or Submittal Topic II. Integrated A. Reviews that technical divisions have historically found to be Review Team challenging, specifically where PRA insights may help in completing Considered the review more effectively B. Reviews where it isnt feasible to complete a portion of the deterministic justification in an appropriate time frame C. Changes to Technical Specifications where single failure and other conservatisms in Updated Final Safety Analysis Report (UFSAR)

Chapter 15 are no longer considered as part of the assumption D. Reviews that involve multiple system integration E. Reviews that are related to structures, systems, and components (SSCs) that are important to safety, but risk information is not provided III. Integrated A. Administrative changes Review Team Not Necessary B. Reactor risk neutral activities, such as offsite sirens, domestic water, etc.

C. Other reviews where integrating the review with risk analysts will have no significant use, such as those where risk methods are not available, or reviews where including risk information is not needed to ensure an efficient and effective review

• NOTE: If NRC SEs will include some discussion of risk insights, then the Division of Risk Assessment (DRA) should be involved.

How to Use the IRT Checklist Each of the Table 1 categories (i.e., Integration Recommended, Integration Considered, and Integration Not Necessary), which are described in more detail below, relate to when the Office of Nuclear Reactor Regulation (NRR) Division of Risk Assessment (DRA) risk analysts should be involved in technical reviews. There may be exceptions or first of a kind reviews that dont fit into any of these categories. In cases of uncertainty, technical reviewers can reach out to DRA risk analysts for insights.

I. IRT Recommended I.A Risk-informed reviews in accordance with RG 1.174 These reviews, which are Type 3 reviews per Attachment 1, require the use of risk analysts from DRA. These reviews should be integrated as soon as practicable to ensure that the technical reviewers and risk analysts understand the scope of their respective portions of the review, the justification for the bases for acceptance in each of the technical and risk areas, and technique for formatting an integrated technical evaluation.

, Attachment 2 Page 3 of 11 Technical reviewers should become familiar with risk-informed reviews in accordance with NRC RG 1.174, with focus on the principles of risk-informed decision-making (RIDM). The NRC offers the Risk-Thinking Workshop periodically (see available references in Module 1 of iLearn Course 322150, Inspector Training on Risk-Informed Completion Times). The Technical Specification Branch Specific Training, Section B contains a primer about risk. Technical reviewers may also reach out to risk analysts for information about the RIDM process or precedents involving risk reviews.

Risk analysts that are new to the topic area may find technical information in Technical Training Center series training, or reach out to the technical branches for references.

I.B Risk-informed TSTF traveler CLIIPs The CLIIP is intended to make the review of technical specification changes more efficient.

Reviews that use the CLIIP are called CLIIPs. Even though CLIIPs are highly streamlined, CLIIPs that rely on risk information typically involve the review by a risk analyst from DRA.

Revision 5 of LIC-101 states, in part, Only the NRR Division of Safety Systems (DSS)

Technical Specifications Branch (STSB) and the Plant PM in DORL typically need to review a CLIIP license amendment request, unless it is a risk-informed CLIIP license amendment request, which would also need to be reviewed by DRA/APLA. In some cases, risk considerations are taken into account during the initial CLIIP review process. In these cases, the risk review of plant-specific applications of the CLIIP may be unnecessary.

I.C Submittals that include risk information, but not in accordance with RG 1.174 For these reviews, the licensee is presenting what it believes is risk information that supports its safety case. This information is not sufficient to form the bases of a potential NRC acceptance of the submittal because it does not meet RG 1.174. However, the proposed change may lend itself to risk evaluations, and DRA may be able to provide some technical, rather than quantitative, insights to the review. The DRA risk analysts should be engaged early in these reviews to ensure that the provided risk information is used appropriately and within the correct context.

II. Integrated Review Team Considered For these reviews, there may be usefulness in engaging with DRA risk analysts to provide supporting information or other insights to the staff reviews. In general, these are listed below in the order of the most likely to involve risk analyst involvement to the least likely.

II.A Reviews that technical divisions have historically found challenging and where PRA insights may help complete the review more effectively For complex reviews that are technically challenging but without a clear challenge to plant safety, the staff should consider integrating risk insights. Without a fully risk-informed submittal, risk insights may not be used as the bases for acceptance of a request; however, the risk insights would be expected to assist the technical reviewers by supporting their safety conclusions faster and enhancing the technical reviewers confidence in their technical evaluations.

, Attachment 2 Page 4 of 11 II.B Emergency and exigent reviews Emergency and exigent reviews may have the potential for significant impact on plant safety because they typically involve quickly changing a deterministic requirement. It is prudent to involve DRA risk analysts in these reviews early. The role of the use of risk insights would be to assist the technical reviewers in supporting their safety conclusions faster and enhancing the technical reviewers confidence in their technical evaluations.

II.C Changes to technical specifications where single failure, and other conservatisms in UFSAR Chapter 15, are no longer considered as part of the assumptions Safety-related SSCs and analyses often have numerous layers of protection, such as assuming a single failure or worst case scenarios. Where these numerous layers of protection are in place, there may not be any use in applying risk insights developed by risk tools because the risk tools do not make the same assumptions (for example, regarding single failure). In cases where a safety-related analysis no longer relies on these prescriptive assumptions, it is both useful and prudent to look to risk analyses for insights. This is often the case when a licensee is already in a technical specification completion time because single failure does not apply in that case, or for a submittal where the licensee is requesting that they not consider single failure for a period of time (for example, when maintenance is planned).

II.D Reviews that involve multiple system integration Reviews of submittals that involve multiple system integration that may not have an obvious risk connection, should be considered to be discussed with risk analysts.

II.E Reviews that are related to SSCs that are important to safety, but risk information is not provided Structures, systems and components that are important to safety are typically modeled in the PRA. For SSCs that are important to safety, there are typically more tools (e.g., standardized plant analysis risk (SPAR), notebooks, etc.) available to the risk analysts to provide insights to technical reviewers regarding the risk insights related to an SSC that is important to safety.

Discussion with risk analysts may be prudent to ensure all the roles of that equipment are considered in the deterministic review. In this case, the risk analyst may not provide risk insights but may provide connections to other review organizations that are not obviously related to the review topic.

III. Integrated Review Team Not Necessary For these reviews, there is currently not an identified need to engage with DRA risk analysts.

Either there is not a risk aspect to be considered, or the technical review process is mature such that adding risk resources would not improve the efficiency or effectiveness of the review.

III.A Administrative changes If administrative changes do not impact the operation or maintenance of the plant, then there is no need to engage with DRA risk analysts for reviewing these changes. Requests for NRC review of operational, testing, or maintenance procedure related changes could potentially

, Attachment 2 Page 5 of 11 impact plant risk and shouldnt be excluded from possible integration by categorizing these types of changes as administrative changes.

III.B Reactor risk neutral activities This includes reviews of topics that are not related to components that could have an impact on the safety of the reactor, such as the use of offsite sirens or domestic water. If the technical reviewers for such changes are new to the reviews, discussions with risk analysts may be prudent for reviews that are new to the reviewers organizations.

III.C Other reviews that have been agreed with DRA that integrating the review with risk analysts will have no significant use, such as those where risk methods are not available, or reviews where including risk-information is not needed to ensure an efficient and effective review For a large number of reviews there currently is no integrated review. This is likely the most efficient and effective way to manage these reviews. For example, where single failure is required to be considered, there is little value in layering in a risk review that uses different criteria. Other large reviews, such as power uprates and license renewals, have well established processes - and there is no need for project managers (PMs) and technical reviewers to ask if risk analyst involvement is necessary, unless a new or unique situation dictates that risk may play a larger factor.

3.0 DRA CHECKLIST During the scoping meetings, the Plant PMs and reviewers will need to determine whether to use risk insights and thus use an integrated team approach. The Task 3 working group developed a second checklist to identify available risk tools to provide insights for reviews of applications that were not submitted in accordance with RG 1.174. These risk-insights may be developed based on quantitative information, but will be characterized as qualitative risk insights when included in SEs. The staff may use risk tools such as SPAR models, the risk triplet, or event frequencies to develop these qualitative risk insights. The RG 1.174 quantitative risk thresholds are not expected to be used as the basis for approving changes that were not submitted in accordance with RG 1.174.

The questions in the checklist below provide a framework for developing risk insights that may provide additional confidence in conclusions reached by technical reviewers using traditional defense-in-depth and deterministic evaluations. The DRA staff will be responsible for facilitating the use of this checklist with the review team when considering whether PRA risk insights should be included in licensing action decision-making. Quantitative or qualitative PRA risk insights are considered in this checklist. The DRA staff should assist the review team in differentiating probabilistic information from PRA risk insights. The staff will use this checklist to determine whether and in what form PRA risk insights should be included as part of an SE. The DRA risk analysts should be considered as the risk insight specialists and will be responsible for reviewing risk-related information contained in SEs. The DRA staff will use NRC SPAR models or publicly available information provided by a licensee as part of risk-informed reviews to develop these risk insights, when possible.

The review team, with DRAs lead, should compile answers for all questions in the checklist when considering whether the review may use PRA risk insights. If the risk analysts response to Questions A-E is No (N), the proposed change is not considered to be risk significant. If the

, Attachment 2 Page 6 of 11 answer to any of those questions is Yes (Y) or not available (NA), the risk analysis may perform additional analysis in response to Question F or develop risk insights using SPAR models or other tools.

Description of the Proposed Change:

Y N NA A. PREVENTION: Does the proposed change affect the likelihood of events that perturb operation of the plant and lead to an undesired plant condition?

1. Does the proposed change affect the likelihood of perturbations or add new perturbations from internal plant causes (e.g., hardware faults, floods, or fires)?

2. Does the proposed change affect the likelihood of perturbations or add new perturbations from external plant causes (e.g., earthquakes or high winds)?

Y N N/A B. PROTECTION: Does the proposed change adversely affect common cause failures (e.g., an operable SSC may be subject to the same failure mode as an inoperable SSC)?

1. Does the proposed change increase the likelihood of a cause or event that could cause simultaneous multiple component failures?

2. What is the risk significance of common cause failures? Is a related SSC that would be subject to common cause failure highly relied upon by the proposed change?

Y N N/A C. MITIGATION: Does the proposed change affect the likelihood of successful plant response to plant perturbations (events that would cause the plant to implement abnormal operating procedures (AOPs) and emergency operating procedures (EOPs))?

1. Is the likelihood of the affected SSCs to successfully perform its required function(s) (during a specified period) affected by the proposed change?

2. What is the likelihood of the affected SSCs not being capable to supporting their functions due to being unavailable for test or maintenance? Is this unavailability of an SSC due to Test and Maintenance (T&M) affected by the proposed change?

, Attachment 2 Page 7 of 11

3. Does the proposed change affect the likelihood of restoring a function due to failure of affected SSCs?

i. Does the affected function rely on diverse/redundant SSCs? What is the recovery likelihood of the affected SSCs or affected function?

ii. Do the proposed compensatory measures manage risk-significant configurations?

Y N N/A D. DEFENSE-IN-DEPTH CONSIDERATIONS

1. Does the proposed change significantly increase the likelihood of an event or introduce a new event that could simultaneously challenge multiple barriers (e.g., interfacing systems loss-of-coolant accident (ISLOCA) and steam generator tube rupture)?

2. Does the balance among the layers of defense remain appropriate?

Y N N/A E. HUMAN ACTIONS: Does the proposed change increase failures or unavailability of SSCs (or function) caused by human inaction or inappropriate actions?

1. Does the proposed change create new human actions?

2. What is the likelihood of errors associated with new or affected human actions?

3. Are new human actions important to preserving layers of defense?

4. What are the absolute or relative contributions of new or affected human actions to overall risk?

F. RISK INFORMATION:

1. If risk information is known, what are some generic risk insights for the plant?

a. What is the overall risk of the plant (i.e., base line risk)?

b. What are the drivers of risk (or change in risk) (e.g., at the initiating events, accident sequences, and cut sets levels)?

c. Has the NRC reviewed applications that proposed similar changes? What conclusions were made in those reviews by the NRC?

, Attachment 2 Page 8 of 11

2. What are the absolute and relative contributions of the affected SSCs, collectively and individually, to overall risk?

a. What is the increase in risk if the affected SSC (or a collection of SSCs) was assumed to be failed or unavailable?

b. What is the relative contribution of the affected SSC (or a collection of SSCs) to the calculated risk?

c. What is the sensitivity of risk to the performance of the affected SSC?

Overall

Conclusions:

4.0 SAFETY EVALUATION LANGUAGE In order to integrate the reviews of submittals that have varying levels of risk information, the Task 3 working group developed SE template language. The DRA and technical branches are actively developing SEs that integrate traditional reviews and risk-informed reviews (e.g.,

tornado missile related applications), which served as the starting points for these templates.

Describing PRA Information in Regulatory Decision-Making As part of the risk-informed decision-making (RIDM) activities, the NRR staff has made an effort to better integrate PRA information into regulatory decisions. These regulatory decisions include license amendments or other licensing actions, such as exemptions or relief requests.

The Tasks 1 and 3 working groups identified a need to use consistent and commonly understood language when integrating risk information into regulatory decision-making. The common definition of the term risk may have been informally used at the NRC in different contexts - for example, from simply the consideration of likelihood, to likelihood and consequences, and to a full PRA calculation. To better communicate the use of risk in regulatory decision-making, three categories of SE language are recommended. Category 1 would be used when PRA results are used as one of the factors that form the basis for a regulatory decision. Category 2 would be used for when a PRA is used to support a deterministically supported position. Category 3 would be used for when pieces of probabilistic information - not PRA results - are used to support a deterministic regulatory decision.

Category 1: Basis Using the results of a PRA as a part of the basis of a regulatory decision involves using the five principles of the RIDM framework from RG 1.174. These features are: (1) the change meets the regulations, (2) is consistent with defense-in-depth, (3) maintains safety margins, (4) results

, Attachment 2 Page 9 of 11 in small increase in risk using PRA information, and (5) is monitored. For PRA information to be used in this RIDM framework, the PRA should be acceptable to support the application as described in RG 1.174.

The language used to describe PRA results as of the elements of a regulatory decision are documented in RG 1.174, and other RIDM regulatory guidance. The language should focus on PRA. Language such as risk-insights, or PRA insights, may be used, consistent with the guidance in RG 1.174 or RG 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities. Therefore, the Task 3 working group is developing language for basing regulatory decisions using PRA by using existing precedents and guidance from RG 1.174.

Category 2: PRA Used to Support Regulatory Decisions Using PRA insights or results to help support or verify regulatory decisions made using traditional engineering evaluations is a primary outcome of the graded approach in RIDM. The output of PRAs often includes numbers, such as core damage frequency (CDF) or large early release frequency (LERF). Probabilistic risk assessments are also used to gain insights about a facilitys response to initiating events and accident progression, including the expected interactions among facility SSCs, and between the facility and its operating staff. A risk assessment is a systematic method for addressing these questions as they relate to understanding issues such as important hazards and initiators, important accident sequences and their associated SSC failures and human errors, system interactions, vulnerable plant areas, likely outcomes, sensitivities, and areas of uncertainty. Risk insights can be obtained via both quantitative and qualitative investigations. Quantitative risk results from PRA calculations are typically the most useful and complete characterization of risk, but they are generally supplemented by qualitative risk insights and traditional engineering analysis. Qualitative risk insights include generic results - i.e., results that have been learned from numerous PRAs that have been performed in the past, and from operational experience, and that are applicable to a group of similar plants. These qualitative or quantitative risk insights also need not be necessarily developed by PRA models that meet the guidance of RG 1.200 and may be developed using SPAR models or other PRA tools.

The language used in SEs would describe the information from a PRA but would be clear that the information was not used as the basis in NRCs RIDM framework. For example, verification might be described as, PRA information was determined to be consistent with this conclusion, or Nothing in this deterministic evaluation was found to be inconsistent with the PRA.

Using PRA tools such as SPAR models, human reliability analysis (HRA) worksheets, or Plant Risk Information e-Book (PRIB) handbooks, which do not generally meet the acceptability needed for RG 1.174 applications, can also be described as PRA insights or information used to verify a regulatory decision, as described above.

When describing the use of PRA to support regulatory decisions, language such as the following can be used as a template in the following manner:

In the subject application, the licensee stated that the basis for the proposed []

is based upon []. The licensee submitted a risk evaluation that was not consistent with risk-informed decision-making.

, Attachment 2 Page 10 of 11 Because this is not a risk-informed application, the PRA models used to derive risk insights in [] were not reviewed by the staff to determine their technical acceptability as a basis to support this application. As a result, the staff did not rely on the numerical results provided by the licensee. However, the staff considered the licensee-provided risk insights to aid in the deterministic review of the proposed change. The staff also performed an independent assessment using [] to evaluate the risk contribution. The licensee-provided risk insights and the risk insights developed by staff both supported the engineering conclusions []. The currently available risk insights and results did not challenge the engineering conclusions that the proposed change maintains defense-in-depth.

Category 3: Probabilistic Information Used to Support Regulatory Decisions Supporting information can be probabilities of failures of equipment, the frequency of initiating events, or other information that may be used as inputs to PRAs. This information can also be used to support regulatory decisions. In a sense, this is the facts of the matter that may or may not end up in a PRA calculation. Although these facts of the matter may not end up in a PRA, they may be suitable for direct use in a deterministic evaluation. Use of PRA information such as CDF or LERF would indicate that PRA was used in the regulatory decision and, therefore, this category would not apply. Care is needed in this area to avoid calling this supporting information PRA. This information does not have to be used within a risk analysis to help support a deterministic evaluation. Perhaps a better name for these would be risk inputs, or the factors that could be used in a risk analysis.

The language that would be used for this supporting information could be called, likelihood information, probabilistic analyses, or other quasi-qualitative terms. However, this information should not be called PRA, results, or input. Care should be taken to ensure that users of this probabilistic information do not mistakenly assume probabilistic information is PRA results or insights just because such information has the potential to be used in PRA or risk evaluation that is PRA.

The DRA risk analysts may assist the staff in crafting an SE that uses deterministic information to support a regulatory decision. When discussing probabilistic information or risk inputs, technical staff have significant flexibility within the templates provided in office instructions such as LIC-101 to describe the technical basis for their regulatory decision. The limitation is that when describing probabilistic or risk input, the term PRA or use of quantitative CDF or LERF should not be used. Use of these terms would indicate PRA as a basis of the decision or used to support the decision; therefore, it would be in a different category.

, Attachment 2 Page 11 of 11 Categories Compared Table 2 below compares the categories.

Table 2, Categories of Regulatory Decision Language Regulatory Decision Category Quality Language to Use Language to Avoid

1. Basis RG 1.174 PRA (and RIDM) is part of the [any language can be used basis for regulatory decision if used correctly within the RIDM framework]

2. PRA or risk Some PRA PRA is consistent, or is not . . . is acceptable based on analysis or risk inconsistent, with deterministic PRA . . .

supports analysis conclusion

3. Probabilistic Verifiable Probabilistic insights are Probabilistic Risk information facts consistent, or the events are Assessment, PRA, risk supports deemed unlikely because . . . analysis, CDF, or LERF Conclusion The first lesson for properly describing the use of risk-information in regulatory decision-making is ensuring that the term probabilistic risk assessment, or PRA, is only used when there is a PRA calculation used as part of the basis or to support the decision. Secondly, PRA can only be part of the basis for a decision when the PRA is acceptable for the application; that is, the PRA acceptability is addressed in accordance with RG 1.174. If PRA acceptability is not addressed in accordance with RG 1.174, then the PRA results or risk insights could be used only to verify the deterministic conclusion. Lastly, if a PRA or risk analysis was not used, even if there is quantitative information or calculations, the term probabilistic risk assessment or PRA should not be used.

, Attachment 3 Page 1 of 15 DATA COLLECTION RESULTS AND FINDINGS NRR RIDM ACTION PLAN TASKS 1 AND 3 The working groups data collection consisted of identifying: (1) the scope of Office of Nuclear Reactor Regulation (NRR) licensing actions and major milestones associated with those actions that could be affected by Tasks 1 and 3, (2) licensing processes, procedures, and training that could be affected by Tasks 1 and 3, (3) Replacement Reactor Program System (RRPS) capabilities that could be affected by Tasks 1 and 3, and (4) lessons-learned from recent licensing actions related to Tasks 1 and 3. The staff interviewed a subset of technical reviewers and project managers that worked on risk-informed submittals. The working groups reviewed examples of licensing actions with risk-informed elements, consulted with subject matter experts and peer reviewers, and reviewed draft products for ongoing licensing actions, as available.

1.0 Scope of NRR Licensing Actions and Milestones Affected by Tasks 1 and 3 The NRR processes the following types of licensing activities requiring staff evaluations that could involve consideration of risk:

• License amendments o Emergency/exigent o Routine o Power uprates o Improved Standard Technical Specifications conversions o Security, emergency preparedness (EP), quality assurance (QA) plans o National Fire Protection Association (NFPA) Standard 805 Transitions

• Relief requests and proposed alternatives

• License exemptions

• Environmental reviews (environmental assessments and impact statements)

• License renewals and associated commitment reviews

• Orders/backfits

• Technical Specification Task Force (TSTF) Travelers

• Topical reports From this list, the working groups decided to focus the scope of these tasks to relief requests, exemptions, and a subset of license amendments (emergency, exigent, and routine) because of the limited timeframe given to complete the tasks and because the other processes can borrow from these actions working groups formation and evaluation activities as appropriate or may need more extensive review because of regulatory implications or division-specific processes.

The working groups focus on amendments, relief requests, and exemptions does not mean that other actions would be excluded from the tasks; rather, the divisions or branches owning those processes can decide whether to apply the recommendations from this report on a case-by-case basis.

In SECY-17-0112, Plans for Increasing Staff Capabilities to Use Risk Information in Decision-Making Activities, dated November 13, 2017 (ADAMS Accession No. ML17270A197),

the U.S. Nuclear Regulatory Commission (NRC) staff identified three types of applications from the nuclear industry. Type 1 includes traditionally deterministic requests that demonstrate regulatory compliance largely through the use of NRC-approved prescriptive analyses. In these

, Attachment 3 Page 2 of 15 cases, quantitative probabilistic risk assessment (PRA) results have rarely been included.

Type 2 includes licensing submittals that contain quantitative or qualitative risk information but are not formally submitted using the guidance in Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, and RG 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities. Lastly, Type 3 applications are formal risk-informed changes that use quantitative risk information derived from a RG 1.200-compliant PRA and are subject to the review guidelines in RG 1.174. During an executive team/leadership team (ET/LT) meeting, NRR senior management expressed the desire for NRC to consider risk when evaluating any of the three types of applications.

The Task 1 and 3 working groups determined that the concept of qualitative risk insights mentioned in SECY-17-0112 may not be consistently understood or easily identifiable in licensee applications, and that qualitative risk insights might be present in each of the SECY-17-0112 Type 1, 2, and 3 applications rather than in just Type 2 applications. In addition, the staff could develop qualitative risk insights in the absence of licensee-provided insights.

However, Plant project managers (PMs) and technical reviewers should be able to easily identify PRA information and terminology (e.g., RG 1.174, core damage frequencies (CDF),

large early release frequencies (LERFs), cut sets, initiating event frequencies, or licensee references to their PRAs). Therefore, the working groups decided that for the purposes of processing licensing actions and tracking various metrics for those actions in RRPS, the SECY-17-0112 Type 1, 2, and 3 applications should be redefined for the purposes of the NRR risk-informed decision-making (RIDM) tasking as follows:

Type 1: Applications that do not contain PRA information.

Type 2: Applications that contain PRA information but are not submitted as risk-informed applications meeting the guidance of RG 1.174.

Type 3: Applications that are submitted as risk-informed and intended to meet the guidance of RG 1.174.

The working groups also considered the volume of applications (e.g., whether many licensees are simultaneously requesting the same or a similar change) and determined that a large volume of simultaneous similar applications necessitates additional project management activities to promote consistency and efficiency (e.g., developing a model safety evaluation (SE)). A process for this scenario is not explicitly discussed in NRR office instructions for licensing.

The following table shows the current major milestones for processing amendments, relief requests, or exemptions (not all milestones are applicable to all processes). The working group highlighted those milestones via italicized text that could be affected by the RIDM tasking, such

, Attachment 3 Page 3 of 15 as increased hours and review scope associated with the milestones, by the expansion of teams to include risk analysts and technical reviewers.

Milestones for NRR Amendments, Relief Requests, or Responsible Staff Exemptions NRC Receives Application (ADAMS add date) Plant PM PM opens new project in RRPS (gets Project ID #) Plant PM PM chooses milestones, due dates, priority Plant PM PM chooses tech branches and scope of review Plant PM PM identifies if sensitive unclassified non-safeguards Plant PM information (SUNSI)

Tech branches identify scope, estimated hours, milestones Technical Branches dates, reviewers, other branches needed for the review Acceptance Review to PM Technical Branches Acceptance Review Results issued to Licensee Plant PM Proprietary Info Letter if applicable Plant PM/Tech Branches Federal Register Notice (FRN) - no significant hazards Plant PM consideration (NSHC) (frequency of an accident)

Environmental Assessment or Impact Statement Plant PM/Tech Branches Audit Plant PM/Tech Branches Request for Additional Information (RAI) to PM Technical Branches RAI to Licensee Plant PM SE to PM Technical Branches Notification of State Contact Plant PM SE to Licensee/FRN/Concurrences Plant PM Exemption package (FRN, Cover Letter, Exemption Basis) Plant PM/Tech Branches 2.0 Licensing Processes, Procedures, and Training Affected by Tasks 1 and 3 The working groups activities primarily involve the NRR Division of Operating Reactor Licensing (DORL) project management activities and technical reviewer and risk analyst support of these activities. Therefore, Task 1 primarily involves DORL licensing processes and training tools, which are contained in NRR office instructions (e.g., LIC-101, License Amendment Review Procedures, LIC-102, Relief Request Reviews, LIC-103, Exemption from NRC Regulations, and LIC-109, Acceptance Review Procedures), the DORL PM Handbook, the project management qualification program in ADM-504, and training classes. NRR technical and PRA staff also have branch-specific procedures or training that may be affected; therefore, as part of Phase 2 of the RIDM Action Plan, NRR divisions may need to perform a gap analysis to determine how their processes need to be enhanced. The following list identifies the guidance in NRR office instructions and training related to considering risk in licensing reviews that will need enhanced in Phase 2 of the RIDM Action Plan.

LIC-101, License Amendment Review Procedures LIC-101 is mainly procedural and is process-focused. Therefore, technical guidance on the extent to consider risk information during a review is not included. This office instruction provides a reference to review guidance (i.e., NUREG-0800, Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants (SRP), Chapter 19.2, Review of Risk Information Used to Support Permanent Plant-Specific Changes to the Licensing Basis:

General Guidance) for a risk analyst, but does not provide a reference for other reviews

, Attachment 3 Page 4 of 15 concerning the use of risk information. Appendix B, Section 7 (pages 33 to 36) of LIC-101 contains guidance on and unspecific direction to consider risk significance in considering RAIs.

The guidance in Appendix B focuses on identifying when a branch with qualified risk analysts should be included on a project. The inclusion of a risk analyst on a deterministic review is limited to special circumstances or when the licensee provides risk information (i.e., a SECY-17-0112 Type 2 application). The office instruction establishes an expectation of coordination between risk analysts and other technical reviewers. The direction to consider risk significance is overly broad and does not provide actionable direction or reference to other actionable guidance. Appendix B, Section 7 will need revised as part of Phase 2 of the NRR RIDM Action Plan.

Regarding emergency amendments, LIC-101, Appendix B, Section 7.2.1 states:

If a risk-informed emergency amendment request is submitted, the PM should contact the [DRA Probabilistic Risk Assessment Licensing Branch (APLA) branch chief (BC)] as soon as possible. As discussed in Section 7.2.3, a risk-informed amendment request must address the five principles of risk-informed regulation.

APLA staff ensure these principles are met, in part, by confirming the technical acceptability of the licensees probabilistic risk assessment (PRA). Evaluating the scope, level of detail, and technical adequacy (e.g., compliance with RG 1.200) of a PRA is a resource-intensive process that cannot generally be completed under the time constraints of an emergency amendment request

[emphasis added]. Therefore, the PM should prepare, in coordination with DORL and DRA management, for a possible discussion with the licensee regarding whether the review can be completed in time to support the request.

The working group determined that the emphasized text (i.e., italicized) could discourage licensees from submitting risk-informed emergency amendment requests. As discussed in Section 3.0 of the enclosure, this guidance will need to be revised as part of Phase 2 of the NRR RIDM Action Plan to provide conditions under which the staff may be able to complete reviews of risk-informed emergency amendments within the specified time constraints of an emergency amendment request.

LIC-102, Relief Request Reviews (Draft Version)

There is currently no guidance in LIC-102 related to risk. Therefore, as part of Phase 2 of the NRR RIDM action plan, the following sections will need to be evaluated for conformance to the Phase 1 recommendations:

• Sections 4.2.1 through 4.2.5: These sections provide a high-level overview of authorizing alternatives, granting relief requests, approving the use of Code Cases, and the duration of proposed alternatives and relief requests.

• Section 4.4: This section discusses the contents of an SE.

• Appendix A: This appendix discusses the acceptance review and work planning process.

• Appendix B: This appendix discusses the technical review process, and SE and verbal script templates.

, Attachment 3 Page 5 of 15 LIC-103, Exemption from NRC Regulations As part of Phase 2 of the NRR RIDM action plan, the following sections will need to be evaluated for conformance to the Phase 1 recommendations:

• Section 4.2:

o The licensee may address, as appropriate, the guidelines contained in Regulatory Guide (RG) 1.174, An Approach for using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, on the risk impact of the proposed exemption.

o The licensees safety analysis that supports the proposed exemption should include technical information in sufficient detail, including risk assessment impacts (as appropriate), to enable the NRC staff to make an independent assessment regarding the acceptability of the proposal.

• Section 4.9.2:

o The staffs evaluation may include, as appropriate, a discussion of the risk impact of the proposed exemption with regard to RG 1.174 guidelines.

• Attachment 1:

o No Undue Risk to Public Health and Safety. . . . . Based on the above, no new accident precursors are created by [state what the exemption would do/allow], thus, the probability of postulated accidents is not increased. Also, based on the above [or give some other reason], the consequences of postulated accidents are not increased. Therefore, there is no undue risk [since risk is probability x consequences] to public health and safety.

LIC-109, Acceptance Review Procedures There is currently no guidance in LIC-109 related to risk. Given that LIC-109 focuses on the process to determine acceptability of a requested licensing action, it is not clear that there needs to be a specific discussion pertaining to risk. However, LIC-109 does have some discussion related to team formation and coordination. Therefore, as part of Phase 2 of the NRR RIDM action plan, the following sections will need to be evaluated for conformance to the Phase 1 recommendations:

• Section 5.A: This section discusses Plant PM responsibilities.

• Section 5.C: This section discusses technical staff responsibilities.

• Section 5.D: This section discusses branch chief responsibilities.

• Section 5.E: This section discusses NRR division management responsibilities.

• Appendix B, Section 2.0: This section discusses the distribution of the requested licensing action and identification and involvement of the technical branches.

, Attachment 3 Page 6 of 15 Training As part of Phase 2 of the NRR RIDM action plan, the following training on applying risk information in a regulatory review will need to be evaluated for conformance to the Phase 1 recommendations:

• PRA Basics for Regulatory Application

• PRA Technology and Regulatory Perspectives

• Risk-Informed Applications in Reactor Programs

• Risk-Informed Regulation for Technical Staff

• Risk-Informed Thinking Workshop

• ADM-504 Qualification Program o General Study Activities o Operating Reactor Project Manager o Reactor Regulation Project Manager o Reactor Oversight Process Engineer o Reliability and Risk Analyst

• Fundamental of Operating Reactor Licensing for Project Managers

• Fundamental of Operating Reactor Licensing for Technical Reviewers

• PM handbook on DORLs SharePoint site.

3.0 RRPS Capabilities Affected by Task 1 One of the goals of Task 1 includes enabling NRR staff to develop efficient teams to communicate, resolve issues, and complete licensing reviews in a timely manner. Measuring efficiency involves tracking the duration of and hours spent on a project and comparing those metrics in past and future reviews. The RRPS capabilities include project descriptions, importance measures, staff assignments, milestone selections, metric selections (e.g., hours and duration), and reports for various metrics. However, RRPS currently does not track whether applications are Type 1, 2, or 3 (there is an option to select whether the application is Risk-Informed). Therefore, it would be difficult and resource-intensive to do such a comparison. One option is to compare the average hours and duration spent on past reviews to future reviews after the working groups recommendations are implemented, regardless of whether the applications are Type 1, 2, or 3. However, this method might not capture anomalies and unique challenges among the various types of applications. If the agency wishes to track hours and duration of Type 1, 2, or 3 applications, RRPS will need modifying so that it can classify an application as Type 1, 2, or 3. The RRPS will also need to identify if an integrated review team (IRT) was assigned so that reports can be run efficiently on review hours and timeliness for each type of application. Expanding team membership to include risk analysts and traditional engineering reviewers regardless of the application type will also increase initial estimates presented to licensees in the acceptance review and may affect the expectations memorandum (ADAMS Accession No. ML16202A029) hours estimates and considerations for when to consider a project complex.

The Task 1 working group analyzed recent complex reviews and also notes that projects such as the Oconee proposed alternative on cable separation involved significant resources because,

, Attachment 3 Page 7 of 15 in part, review methodology guidelines for integrating risk and traditional engineering insights were and are not available, and a common understanding of the concepts of risk, defense-in-depth, safety margins, etc. has not yet been fostered as a result of the NRR RIDM Task 2 activities. Therefore, efficiencies will also depend on staff acquiring experience with the integration and the effectiveness of the other RIDM tasks in developing a common understanding of how risk and traditional engineering insights complement one another and how the staff is to integrate the insights in SEs. Risk-informed reviews take more agency resources (and likely licensee resources), but they enable justification for and acceptance of licensing actions that may not otherwise not be possible, which the agency should consider when evaluating the efficiency of IRTs.

The working group determined that RRPS already has sufficient capabilities for selecting staff needed to do the review. The working groups recommendations will affect the amount of time needed to do acceptance reviews; therefore, reports and metrics tracked by RRPS regarding acceptance reviews will need to be modified. The working groups recommendations include additional routine practices for which new generic RRPS milestones or a milestone template would be beneficial, and some new milestones that will replace existing milestones if an IRT is used. Attachment 1 further discusses the working groups recommendations for RRPS.

4.0 Lessons Learned from Recent Licensing Actions To obtain insights from recent experience that would inform the recommendations for these tasks, the working groups members consolidated input from their discussions with NRR staff involved in recent licensing actions and their experiences and challenges with recent licensing actions. The working groups considered lessons-learned from reviews of risk-informed technical specification initiatives, 10 CFR 50.69 license amendment requests (LARs),

emergency and exigent LARs, recent proposed alternatives that were submitted as Type 1 and 2 applications, a Type 3 application that is applying an IRT approach, and a Generic Safety Issue (GSI)-191 review. Staff also provided generic feedback, opinions, and concerns. Insights from NRR staff and these reviews are documented as follows.

Risk-Informed Technical Specification Initiatives (TSTF-425, Relocate Surveillance Frequencies to Licensee Control-Risk Informed Technical Specification Task Force (RITSTF) Initiative 5b, and TSTF-505, Provide Risk-Informed Extended Completion Times - RITSTF Initiative 4b)

• These are Type 3 submittals (per SECY-17-0112, Plans for Increasing Staff Capabilities to Use Risk Information in Decision-Making Activities, dated November 13, 2017 (ADAMS Accession No. ML17270A197)).

• During the development of the model SEs for these TSTF travelers, the travelers and draft model SEs were sent to technical branches but not all reviewers within those branches, resulting in a missed opportunity to acquire additional views. The model SEs focused more on the RG 1.174 PRA quality documentation than the defense-in-depth and safety margin analysis or how traditional engineering insights may have factored into the PRA quality.

• Many of the applications took longer than a year to process.

, Attachment 3 Page 8 of 15

• DORL Plant PMs did not receive any periodic or refresher training about the TSTFs, which could have been provided during division or town hall meetings. Specific training on TSTF-505 has now been developed and is currently available in iLearn.

• Because the model SEs were done by DRA, the DORL Plant PMs typically assigned the LARs to just DRA/APLA during the project scoping. If plant-specific variations from the TSTFs were involved, DORL Plant PMs involved the technical branches, which then resulted in more technical staff involvement and questioning.

• Technical branch concerns arose late in the LAR review process, after most TSTF-425 amendments were issued, and mid-way through the initial set of TSTF-505 reviews.

• It was not clear to Plant PMs, risk analysts, and technical reviewers how the PRA and technical reviews needed to be integrated.

• Routine meetings with PRA and technical reviewers helped facilitate an understanding of each reviewers roles and responsibilities; however, this was resource-intensive.

• A separate PM that provided project management of generic issues with the TSTF and complicated plant reviews helped. This PM tracked a large volume of applications and coordinated the overall review schedules and priorities for the multiple LARs. This PM also established routine meetings for the review teams and management.

• Branch chiefs started assigning the same reviewers, which helped with consistency and efficiency.

10 CFR 50.69 LARs

• These are Type 3 submittals (per SECY-17-0112).

• The nuclear power industry developed a draft template for the application, which overall was a good effort, but there were plant-specific deviations that diverted staff resources and delayed reviews (e.g., inclusion of Class 1 piping).

• The pilot plants SE that was developed a few years ago did not follow the SE template in LIC-101, Appendix B, Attachment 2, which made the presentation, format, and order of information confusing (e.g., the pilot plant SE did not mention that a new license condition was being added until the end of the SE).

• DORL assigned a special projects PM as an overall PM for these LARs, and he vetted the recent LAR with technical branches in an attempt to identify any generic concerns and identify review scope. There was not significant technical branch interest during the acceptance review stage.

• DORL is working with APLA to develop a model SE that follows LIC-101, Appendix B Attachment 2. Technical staff was amenable to most of the template; however, developing the technical evaluation section was difficult because of scoping questions, figuring out what was needed to make a conclusion, and the draft Regulatory Guide (RG) format not leading to a well-organized technical review. The draft RG does not clearly delineate between what is needed to get a license amendment approved and how to implement the amendment once approved.

, Attachment 3 Page 9 of 15

• Having the special projects PM serve as the primary point of contact for generic issues and having plant PMs processing the RAIs, acceptance reviews, and SEs is working well.

Emergency and Exigent LARs

• LIC-101, Revision 5 states that evaluating a risk-informed LAR is a resource-intensive process that cannot generally be completed under the time constraints of an emergency amendment request. However, the LIC does not preclude the staff from reviewing a risk-informed emergency LAR. Additional guidance may be needed, for example, to address situations where the staff recently reviewed the licensees PRA in another licensing action.

• Performing an RG 1.174 review for exigent LARs would be challenging but possible.

• Performing an RG 1.174 review for emergency LARs is not feasible to be completed in 3 to 4 days. A past example required 24/7 shift work among reviewers.

Recent Type 2 Proposed Alternative

• The licensee submitted its proposed alternative as a Type 2 (per SECY-17-0112). It provided risk insights but stated that its justification was not based on risk.

• NRR technical reviewers spent 1,500 hours0.00579 days <br />0.139 hours <br />8.267196e-4 weeks <br />1.9025e-4 months <br /> on the action, which was completed just within 2 years. This does not reflect management hours.

• About 300 Plant PM hours were spent earlier in the review during meetings to establish how risk would be considered in the review. This may have resulted from not having a review standard (e.g., regulatory guide, standard review plant, office instruction, etc.)

explaining how to consider risk in Type 1 and Type 2 submittals (e.g., what degree of SPAR model information can be cited). Some management said that risk can be used in engineering judgment in a qualitative fashion; however, the staff did not understand how to implement this notion - whether qualitative insights can only be derived from quantitative data or else be arbitrary and inconsistent.

• Technical reviewers changed very late in the review (in the last 3 months). The reviewers had differing technical views about the acceptability of the submittal and had different experiences with recent risk-informed submittals.

• The technical branch did not have much experience with reviewing proposed alternatives per 10 CFR 50.55a, and some reviewers did not understand the regulatory process under which the licensee submitted its application, as evidenced by early SE inputs that documented the action as a license amendment. The reviewers were not familiar with the SE format of LIC-102 for relief requests. The DORL Plant PM can better address this early on in the review by developing a consolidated SE outline and explaining the applicable regulatory process to the staff.

• Multiple risk analysts worked on the review and consulted with the regional senior risk analyst. The primary DRA reviewer had worked at the site prior to the NRC and had technical expertise in the area of the submittal subject, so his technical expertise and understanding of the site was beneficial. The Regional senior reactor analyst (SRA) did

, Attachment 3 Page 10 of 15 not charge time to the NRR project; however, the region had a concurrent inspection issue to which it could charge time. Additional guidelines would need to be developed for the use of regional staff in NRR reviews, if NRR management wants to pursue this.

• NRR staff (technical reviewers, risk analyst, and licensing Plant PM) was unsure how to consider the risk insights because the submittal was not provided per RG 1.174.

• Staff thought that the deterministic acceptability trumped the risk insights because of Principle 2 (defense-in-depth) of RG 1.174.

• The initial set of reviewers still worked in silos in some regard. Meetings where the risk was discussed eventually helped the first technical reviewer understand that there was not much safety benefit from a risk perspective in asking some of his draft RAIs (the licensee would have likely needed to spend significant resources to address his concerns). The risk discussions helped focus what was needed for safety; however, the technical reviewer believed the licensee needed to resubmit the application under RG 1.174 in order for the NRC to process the application as such.

• The consolidated SE format reflected the staffs confusion over the basis for and applicability of NRCs risk assessment. The risk-related conclusions were high level and qualitative and did not provide the detail to support statements that risk was low; therefore, DORL requested that DRA provide a separate SE input that contained the quantitative risk information that supported the qualitative conclusions. The DORL staff should have taken an earlier lead with developing the consolidated SE. Even after doing so, the reviewers submitted their own individual branch SE inputs with their preferred format instead of following the DORL SE. The DORL Plant PM still had to rearrange the technical branch SE inputs into the consolidated SE.

• The technical branch chief didnt realize until seeing the first draft consolidated SE and draft RAIs late in the review that a significant adjustment was needed regarding the method of review and the format of his branchs input. Earlier technical branch chief engagement was needed (this was a new branch chief that had experience drafting SEs for relief requests). Earlier versions of the draft consolidated SE showed that the first assigned reviewers were treating the request as an amendment request rather than as a proposed alternative.

• Once the consolidated SE was developed, the DORL Plant PM asked other DRA reviewers not assigned to the project to review the treatment of risk from an NRR RIDM project perspective. The risk analysts expressed concern with the draft SEs use of SPAR values given the limitations of SPAR models.

• The technical branch chief indicated that the second electrical reviewer would be able to find the submittal acceptable without the use of risk and questioned whether we needed to discuss risk insights in the SE at all.

• The submittal involved a few specific areas of the plant. The risk analysis was applicable to all of those areas. The risk write-up in a draft of the consolidated SE was repetitive because it was included in each area of review. For the sake of brevity, the staff put the risk insights in a separate section that also conveyed that risk was not the primary basis for the conclusions. This created the perception that the consideration of risk was siloed from the engineering evaluation. Later in the review, the second

, Attachment 3 Page 11 of 15 electrical reviewer used risk insights within his write-up when discussing the likelihood of faults; thus tying risk more into his review.

Recent Type 1 Proposed Alternative

• The licensee submitted the application as a Type 1 application (no explicit discussion of risk).

• The technical reviewer provided an SE input that discussed risk insights that did not involve any quantitative or PRA information. DRA was not used or consulted on the review. The technical reviewer derived his own risk insights.

• The Plant PM requested that the technical branch revise the SE input to describe what type of risk insights were being considered (i.e., qualitative or quantitative), and the source of the risk insights (i.e., licensee-provided or NRC-derived).

• The DORL branch chief requested that the SE clarify that the risk insights were used a function of engineering judgment. The Plant PM did not understand what was meant by this terminology (e.g., what is meant by engineering judgment and how is this form of engineering judgment different than any other form that the NRC uses).

• The Plant PM was not comfortable with the SE discussing likelihood of certain scenarios without a quantitative basis.

Type 3 LAR

• This is a Type 3 application. There was significant traditional engineering involvement.

• Kick-off meetings (during the acceptance review phase) raised questions about how to do the review.

• Because this was a heavily PRA-based application, DRA had the lead and identified the places in the draft SE regarding which group needed to provide their input.

• Standard Review Plan Chapter 19.2 guidance (for risk-informed licensing basis reviews) was used to determine the findings that the staff needed to make in the SE. The appropriate review responsibility was then assigned based on the type of finding and the expertise necessary to make it. The assignment was made initially by DRA staff and discussed with the other reviewers and BCs to achieve alignment.

• The DRA staff struggled with a lack of familiarity among non-DRA staff with the RIDM process and principles. It was a challenge to demonstrate how the technical reviewers input was relevant to the application, the scope of their review in the context of RIDM, and the type of findings that needed to be made for the SE.

• The DRA staff took the lead to develop a consolidated SE early in the process, which helped narrow the scope of review. The DORL PMs should be heavily involved in consolidated SE development to ensure conformance with LIC-101, Appendix B, Attachment 2, which helps with getting concurrences and Office of the General Counsel (OGC) reviews done quickly. The consolidated SE followed the RG 1.174 key principles discussion.

, Attachment 3 Page 12 of 15

• The expectation of having an SE with holes available at the time of RAI issuance is not uniformly enforced as it should be. Therefore, the integrated SE can have significant input from some reviewers and none from others.

• A technical reviewer was added to the review 6 months into the project, after the acceptance review was issued to the licensee; however, the DRA reviewer understood the technical scope from the start of the project, and hours were estimated for this (a fragility study) and provided in the acceptance review, although the technical reviewer doing the review was not assigned until after the acceptance review.

• A technical reviewer increased his hours by almost 200 percent after the acceptance review was issued. The total number of hours used when waiting for an RAI response was about 80 percent of the initial estimate. The PM exceeded his hours estimate in part because of the high level of interest in the task from internal stakeholders and the need for an audit and additional management briefings. The Plant PM believes there is a high potential that the hours estimated in the acceptance review will be exceeded but that the team will not exceed 125 percent of the estimated hours. Staff from other offices and divisions assisted during the initial meetings but they were not part of the project. A senior level service employee is also assisting.

• The RAIs were not labeled by branch so as to appear to be more integrated per management direction. The PMs may need to develop a system to track RAIs by branch if management wants to continue this practice.

• The staff did an audit at the site, which helped with overall understanding by the team of the application. A public teleconference prior to the audit helped resolve some of the details regarding a technical issue.

• The periodic meetings helped identify issues and concerns early in the review, and the team plans to continue having these as more RAI responses are received and the SE is completed.

Type 3 LAR

• This was a Type 3 submittal (RG 1.174 risk-informed licensing action). This was a years-long pilot review, so the reviewers had the opportunity to build rapport. Many reviewers also worked on associated rulemaking and guidance, which facilitated team building and technical understanding.

• It is beneficial for the major risk and deterministic reviewers to understand their counterparts technical areas and how they relate to the review. All reviewers should get the big picture - how it all fits together. This can be picked up during the review, and takes more time. Once reviewers have an understanding, future similar reviews should be easier.

• The team spent significant time together in audits, which facilitated team building and helped reviewers understand the big picture.

• The team used a SharePoint site that had all the required submittals, correspondence, and guidance required for the review. The site is still used for current/future GSI-191 reviews.

, Attachment 3 Page 13 of 15

• The Plant PMs and the team members have to commit to keeping everyone involved, communicating, coordinating the review, and putting the SE together. This requires people signing up to do things and help out where they usually have not had to in the past.

• The NRCs review team had excellent contractor support that understood both the deterministic and risk parts of the review. The contractor provided good visual aids to help reviewers understand, provided a good technical evaluation report to support the SE, and was part of the team and kept involved

• The team worked well because all members worked well together and were committed to making the review team concept work. Without this kind of commitment to the team and the review the process would have been significantly more difficult. The way in which the team worked together was a cultural change for all involved. This may be difficult for some reviewers.

• The team used Go-To Meeting extensively to share and review documents, which helped for team members that worked remotely.

• The team aspect of the review requires better IT infrastructure because some of the team work remotely. Enabling teams to meet in a conference room and share information with remote team members is important.

o Conference rooms need high quality phones that work without any issues and can be heard/allow hearing of people in the entire room.

o Conference rooms need projectors and computers that are available for all meetings and can be started and used in a timely manner.

• Early in the review, the team worked to understand the requirements of RG 1.174 as they applied to the review. Significant time was spent developing a common understanding of what defense-in-depth and safety margins meant with respect to the review. Considerable discussion about and evolution of these concepts occurred during the review.

• The team found it advantageous to use risk information to focus the deterministic part of the review. If the deterministic issue did not impact risk, it was accepted that reduced review was needed. For areas important to risk, the team applied more focus.

General Feedback and Insights

• If a submittal is not a RG 1.174 submittal, DORL Plant PMs will need to be specific about what APLA assistance is needed (e.g., reviewing the frequency of an accident aspect of the no significant hazards consideration) so that APLA will not reject DORL requests for staffing. This might not be apparent until further in the review, which means the acceptance reviews might not reflect a good estimate of hours.

• One method for using SPAR is to consider what component interactions result in high risk configurations and to base a decision on that component interaction rather than overall risk.

, Attachment 3 Page 14 of 15

• Plant PMs should consider taking the technical reviewer training on risk so that the Plant PM has a better understanding of what technical reviewers do in the review and can better project manage the review.

• Plant PMs, technical reviewers, and risk analysts have varying levels of skill with writing and assembling SE inputs. The ability to develop a well-constructed consolidated SE early in the process will depend on team membership and capabilities. Additional training and awareness needs to be provided to the staff on the SE formats of LIC-101 and LIC-102 and perhaps on writing in general (developing legally sound and well-explained conclusions).

• NRRs division meetings should involve more lower-level training or discussions that directly affect the staffs job functions instead of higher-level topics that may or may not directly help the staff process applications.

• Plant PMs need to be sensitized to issues that may need additional attention (e.g.,

notifying management or the special projects PM when a certain type of application is received).

• The integrated team approach will take management leadership and modeling to help get the teams functioning well and to adapt to a cultural change.

• Successful attributes will involve: committed teams and frequent meetings; a full-time PM; the use of a SharePoint (or similar) site, high quality IT capability, and Go-To-Meetings for remote workers; and knowledgeable contractor of risk and technical subjects (NRR may want to consider staffing technical branches with a risk expert).

• Clarification may be needed as to whether the Plant PM should charge to the project Enterprise Project Identifier (EPID) or to the Plant PM EPID when briefing management on the status of the project.

• The agency will need to communicate to the industry that an integrated team approach will not result in faster or less resource-intensive reviews; rather, the resources are expected to increase. However, a normalized process will be developed that may allow licensees to request changes that they may have been reluctant or unable to request before this process.

• Review teams may not have enough familiarity with the submittals to fully understand the scope and depth sufficiently to develop an accurate hours estimate within the current 25-working-day acceptance review metric. This will be compounded by the decision to integrate risk and traditional engineering insights. When kick-off meetings are held prior the acceptance review being issued (2-3 weeks into the ADAMS add), it became apparent that some team members had not read the application by that time.

• Type 2 applications can contain quantitative or qualitative risk information per the SECY.

Plant PMs will not likely be knowledgeable without training or guidance defining what constitutes qualitative risk information. The Plant PMs are more likely to identify CDF and LERF values as risk information.

, Attachment 3 Page 15 of 15

• The NRC staff will need to document the source of the risk insights (i.e., whether the insights were obtained from the licensees application or developed by the staff from other sources).

• It is not clear as to whether risk insights documented in SEs for Type 1 and 2 applications will constitute design basis information, will need to go into the UFSAR, whether the licensing actions would remain valid if the risk insights change after the action is approved, and whether the risk insights documented in SEs could be used as a basis for future enforcement action if they are not met.

• The NRC risk models are not made for licensing decisions. SPAR models do not receive peer reviews or F&Os (facts and observations), are not subject to a formal change control process, and are not compliant with American National Standards Instituter (ANSI) PRA standards. The NRC would have to evaluate whether updating the models would result in any cost savings in reviews. The NRC had to issue a correction to an SE that relied on SPAR results for a decision after the licensee pointed out that the SPAR model contained an error. The licensee questioned whether the amendment was still valid and approved given that the SE relied on the SPAR model error in its decision-making.

• Kick-off meetings for emergency LAR reviews would be needed to establish ground rules, roles and responsibilities, review scopes, and an SES champion.

• The number of team meetings and iterations will depend on the number of branches involved.

• More team-oriented RAIs will result in more resources expended.

ENCLOSURE 3 Findings and Recommendations Report Related to NRR RIDM Action Plan Task 2 Page 1 of 16 FINDINGS AND RECOMMENDATIONS REPORT RELATED TO NRR RIDM ACTION PLAN TASK 2

1.0 INTRODUCTION

By revision dated May 4, 2018 (Agencywide Documents Access and Management System (ADAMS) Accession No. ML18116A023), the Office of Nuclear Reactor Regulation (NRR) risk-informed decision-making (RIDM) action plan defines the NRR RIDM tasks and projects process, including the use of a systematic or two-phased approach to successfully complete the project. Phase 1 is defined as the evaluation and analysis phase to support the completion of the working groups findings and recommendations reports. This enclosure documents the findings and recommendations associated with NRR RIDM Task 2, Broaden the definition of risk more transparently such that all of the technical staff can see how their work embodies risk considerations - beyond CDF [core damage frequency] and LERF [large early release frequency].

The NRR staff has varying degrees of awareness and knowledge of, and support for, RIDM processes and applications. There also exists a very large amount of reference material on the topic of RIDM, but this reference material has not been fully incorporated into staff guidance that is routinely used for licensing decisions (such as NUREG-0800, Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants (SRP)). Therefore, the NRR RIDM Task 2 working group deemed it appropriate to look at how risk is understood within NRR.

This understanding served to establish a starting point for the definition of risk, from which the team made recommendations for ensuring that technical staff can see how their work embodies risk considerations.

The team explored how the concepts of risk and risk insights are currently understood and used in NRR through a variety of data collection methods. The team conducted document reviews, facilitated cafés with staff and management, and online surveys. This report summarizes the insights from the data collection efforts and captures the working groups recommendations for consideration as the RIDM effort proceeds to Phase 2. The recommendations support Strategy 1 (evaluate and update guidance) and Strategy 3 (enhance mandatory training) from SECY-17-0112, Plans for Increasing Staff Capabilities to Use Risk Information in Decision-Making Activities, dated November 13, 2017 (ADAMS Accession No. ML17270A192).

The missions of NRR RIDM Tasks 1, 3 and 4 are primarily to develop guidance for staff which clarifies how to incorporate risk information in licensing reviews. Since the NRR RIDM Task 2 mission includes the fundamental basis for risk as defined, NRR RIDM Tasks 1, 3, and 4 will be directly affected in how they implement risk processes and guidance. Also, some of the Task 2 findings and recommendations will overlap into the scopes of the other Tasks, as indicated in this report.

2.0 BACKGROUND

The U.S. Nuclear Regulatory Commission (NRC) staff is facing increased challenges regarding the integration of risk insights and risk information into our decision-making procedures and processes for areas of review that do not involve the use of PRA. The NRC has well-established and well understood guidance that is used by staff and industry for risk-informed reviews under Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Page 2 of 16 Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis.

The RG 1.174 describes an approach that is acceptable to the staff for developing risk-informed applications for a licensing basis change that considers engineering issues and applies risk insights. It provides general guidance concerning analysis of the risk associated with proposed changes in plant design and operation. In addition to the guidance, many NRC staff have completed a rigorous qualification program to become trained and qualified as risk and reliability analyst. However, no such guidance and training exists for integration of risk insights and risk information into our decision-making procedures and processes for areas of review that do not involve the use of probabilistic risk assessment (PRA). Consequently, there is not a clear and consistent understanding amongst staff of how to increase the use of risk insights and risk information in their reviews. The NRR RIDM Task 2 is necessary to facilitate broadening the definition of risk more transparently such that all of the staff can see how their work embodies risk considerations beyond PRA.

3.0 DISCUSSION The Task 2 working group developed a plan and process to document the currently understood and used concepts of risk and risk insights by:

• Reviewing available documentation

• Conducting RIDM cafés with staff and managers

• Conducting an online survey

• Reviewing a practical example of RIDM challenges The data from staff and managers collected via cafés and an online questionnaire was compared with the agencys current policy and guidance documents on risk. This data was used to provide recommendations which will: 1) re-emphasize the definition of risk to ensure awareness and common understanding; 2) clarify the concepts of risk and risk insights in regulatory applications; and 3) develop or enhance current training tools to broaden the staffs understanding of risk concepts.

To meet the need, the Task 2 working group separated into a Paper Team, leading the document review and a People Team, leading the café and survey efforts. Regarding the practical example, a working group member was involved in a semi-pilot study using risk insights to reach a risk-informed decision regarding the technical basis acceptance of a licensees proposed alternative. The working group member provided insights on RIDM challenges for the NRR RIDM Task 2 working group to evaluate and identify items for which clarity in a process document would be useful. A summary of the data collection and findings of these four efforts is provided in the next section.

3.1 Document Review 3.1.1 Document Review and Data Collection The Task 2 working group chose a series of documents, through recommendation from technical staff, PRA staff and management for detailed review. The purpose of the document review was to identify how the concepts of risk and risk insights are currently understood and used in NRR. These documents, listed in Table 1, consist of policy documents, SECY Page 3 of 16 documents, office instructions, regulatory guides, NUREG documents and approved topical reports all pertaining to the use of risk.

In order to facilitate the review, a series of questions were developed to help the Task 2 working group reviewer focus their review on the definitions relative to risk. The five principles of RIDM in RG 1.174, Revision 3 were the basis for the terms searched in the documents, and are as follows:

• Principle 1: The proposed licensing basis change meets the current regulations unless it is explicitly related to a requested exemption.

• Principle 2: The proposed licensing basis change is consistent with the defense-in-depth philosophy.

• Principle 3: The proposed licensing basis change maintains sufficient safety margins.

• Principle 4: When proposed licensing basis changes result in an increase in risk, the increases should be small and consistent with the intent of the Commissions policy statement on safety goals for the operations of nuclear power plants.

• Principle 5: The impact of the proposed licensing basis change should be monitored using performance measurement strategies.

Each reviewer was asked to determine how the document defines the following RIDM concepts:

defense-in-depth, safety margins, performance monitoring, and risk/risk insights. RIDM Principle 1 was not included because it is a straightforward, ubiquitous concept. Results of the document review were captured in an EXCEL file on the RIDM SharePoint site. Below is the process the Task 2 working group used to review each document.

Search Terms

• The Task 2 working group reviewers were not expected to read every word of the document. They searched using appropriate words or abbreviations and then determined how they were defined in the document.

How is it defined?

• The Task 2 working group reviewers described how the document defines each RIDM term. They included the document section or page number where the definition was located. They were asked to use as much description as needed to describe how the term is defined for use in each document. If the document pointed to another document for the definition, the reference document was recorded.

How is it used?

• The Task 2 working group reviewers described how each RIDM term was used within the document. In addition, they were asked:

o Is it part of a process/procedure?

o Is it mentioned but not directly utilized?

o Is it used as part of a screening criteria, etc.?

Page 4 of 16

• The Task 2 working group reviewers were asked to be as descriptive as possible so that a reader could understand the importance of the term to the outcome of the report.

Comments

• The Task 2 working group reviewers entered any additional information they felt was necessary to understand the term in the context of the report. Also, they were to include any additional references or information that may have been relevant to the report.

3.1.2 Document Review Data Analysis and Findings Once the document reviews were completed, the data was analyzed for common terms and consistency. First, the data developed within the How is it defined fields was sorted and analyzed using a word cloud. A word cloud displays an image composed of words used in a particular text or subject, in which the size of each word indicates its frequency. Essentially, the larger a word in the cloud appears indicates a greater number of occurrences in the text, while the smaller fonts indicate words that appear less frequently.

Figures 1-4 in Attachment 1 of this enclosure illustrate the word clouds for each of the RIDM terms considered. Several distinct observations can be drawn from these illustrations:

• For the defense-in-depth, safety margins, and condition monitoring terms, most documents either did not define those terms directly, or pointed to RG 1.174 for the definition.

• For the risk terms, PRA, CDF and consequences were the most prevalent in the definitions, however, risk was not explicitly defined in many of the document. In addition, there appears to be a wide variety of terms used in the definition of risk.

In addition to the word clouds, all review data generated was compared to determine if there were similarities in definition and/or use occurred between documents. Several of the documents reviewed defined or used terms in a similar manner, pointed to the same document for definition, or did not define a particular term. A summary of these definitions is illustrated in Table 2.

As illustrated in Table 2, many times, the RIDM terms were not defined, or the document pointed to a different document for the definition. For instance, many documents pointed to RG 1.174 for RIDM term definitions. In the cases where the RIDM terms were defined, a common definition was used to describe the term. In fact, both NUREG-2122 and RG 1.174 contained the common definitions for most terms. Each of the common definitions for the RIDM terms is shown in Table 3.

In some cases, the definitions given in a document were similar to the common definitions, but differed slightly. Those are labeled by Diff-x (where x is the difference number) in Table 2.

Those differences are described in Table 4.

Page 5 of 16 Table 1: Documents Reviewed ADAMS Document Title Rev/Year Accession No.

RITW Training Risk-Informed Training Workshop 2017 ML17265A846 Slides Recommendations for Reactor Oversight NRC Website SECY-99-007 Process Improvements Rev 0, 1999 Document Collection Recommendations on Issues Related to SECY-15-0168 Implementation of Risk Management Rev 0, 2015 ML15302A135 Regulatory Framework Use of Probabilistic Risk Assessment NRC Website Commission PRA Methods in Nuclear Regulatory Activities Rev 0, 1995 Document Policy Statement Collection Integrated Risk-Informed LIC-504 Decision-Making Process for Emergent Rev. 4, 2014 ML14035A143 Issues LIC-101 License Amendment Review Procedures Rev. 5, 2017 ML16061A451 Monitoring the Effectiveness of RG 1.160 Rev. 3, 2012 ML113610098 Maintenance at Nuclear Power Plants An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions RG 1.174 Rev. 3, 2018 ML17317A256 on Plant-Specific Changes to the Licensing Basis An Approach for Plant-Specific, RG 1.177 Risk-Informed Decision-Making: Rev. 1, 2011 ML100910008 Technical Specifications An Approach for Plant-Specific RG 1.178 Risk-Informed Decision-Making for Rev. 1, 2003 ML032510128 In-service Inspection of Piping Effective Risk Communication: The Nuclear Regulatory Commission's NUREG/BR-0318 2003 ML050960339 Guidelines for Internal Risk Communication Guidance on the Treatment of NUREG-1855 Uncertainties Associated with PRAs in Rev. 1, 2017 ML090970525 Risk-Informed Decision-Making Revision to RCP Flywheel Inspection TR WCAP-15666 2001 ML012420149 Program Westinghouse Owners Group Rev 1, Application of Risk-informed Methods to TR WCAP-14572 Supplement 1, ML042610375 Piping Inservice Inspection Topical 1999 Report BWR Vessel and internals Weld Inspection Recommendations Project Legacy Library TR BWRVIP-05 1995 BWR Reactor Pressure Vessel Shell 9808040037 (BWRVIP-05)

Page 6 of 16 Table 1: Documents Reviewed ADAMS Document Title Rev/Year Accession No.

Glossary of Risk-Related Terms in NUREG-2122 Support of Risk-Informed Decision 2013 ML13311A353 making Table 2: Definitions Found in Regulatory Documents Defense in Safety Document Monitoring Risk Depth Margins RG 1.174, SECY-98-144, RITW Training SECY-99-100, RG 1.174 RG 1.174 RG 1.174 Slides SECY-11-0024, NUREG-0800, and LIC-504.

Common RG 1.174, Common SECY-99-007 RG 1.174 Not defined definition Definition SECY-15-0168 Not defined Not defined LIC-504 NUREG-2150 Commission PRA Common Not defined Not defined Not defined Policy Statement definition LIC-504 RG 1.174 RG 1.174 RG 1.174 RG 1.174, see Diff-2 LIC-101 Not defined Not defined Not defined RG 1.174, see Diff-3 RG 1.160 Not defined Not defined Not defined Not defined Common Common Common RG 1.174 Common definition definition definition definition Common RG 1.177 definition, Not defined Not defined RG 1.174 see Diff-6 RG 1.178 RG 1.174 RG 1.174 RG 1.174 RG 1.174 NUREG/BR-0318 Not defined Not defined Not defined Common definition Risk-Informed and RG 1.174, Performance-Based NUREG-1855 RG 1.174 RG 1.174 see Diff-1 Regulation [NRC, 1999]

TR WCAP-15666 RG 1.174 RG 1.174 RG 1.174 RG 1.174, see Diff-4 TR WCAP-14572 RG 1.174 RG 1.174 RG 1.174 RG 1.174 TR BWRVIP-05 Not defined Not defined Not defined Not defined Common Common Common definition, NUREG-2122 Not defined Definition Definition see Diff-5 Page 7 of 16 Table 3: RIDM Terms - Common Definitions An approach to designing and operating nuclear facilities that prevents and mitigates accidents that release radiation or hazardous materials. The key is creating multiple independent and redundant layers of defense to compensate Defense-in-for potential human and mechanical failures so that no single layer, no matter Depth how robust, is exclusively relied upon. Defense-in-depth includes the use of access controls, physical barriers, redundant and diverse key safety functions, and emergency response measures.

The extra capacity factored into the design of a structure, system, or component Safety so that it can cope with conditions beyond the expected to compensate for Margins uncertainty.

Performance monitoring are methods used to verify that the action taken to Monitoring address the issue actually has the intended result and that no adverse, unintended consequences arise.

Risk-informed: A philosophy whereby risk insights are considered together with other factors to establish requirements that better focus licensee and regulatory attention on design and operational issues commensurate with their importance Risk to health and safety. Risk: In the context of a nuclear power plant, risk is commonly expressed as the risk triplet. 1) What can go wrong; 2) How likely is it; 3) What are the consequences.

Table 4: Differences from the Common Definitions Deterministic analyses that are performed in risk-informed licensing applications Diff-1 also contain uncertainties; however, they are addressed via defense-in-depth and safety margin.

The information provided to the decision-maker may be qualitative in many cases; Diff-2 engineering judgment should be employed to properly characterize the risk insights developed.

Risk-informed licensing action is defined as any licensing action that uses quantitative or qualitative risk assessment insights or techniques to provide a key Diff-3 component of the basis for the acceptability or unacceptability of the proposed action. The mere mention of quantitative or qualitative risk insights does not in itself make a licensing action risk-informed.

The NRC definition of risk-informed regulation is, insights derived from probabilistic risk assessments are used in combination with deterministic system and engineering Diff-4 analysis to focus licensee and regulatory attention on issues commensurate with their importance to safety.

The terms risk-informed approach, risk-informed decision-making, and risk-informed Diff-5 regulation are often used interchangeably and somewhat correctly to describe the same concept; therefore, these terms are grouped under the same definition.

The defense-in-depth philosophy has traditionally been applied in reactor design Diff-6 and operation to provide multiple means to accomplish safety functions and prevent the release of radioactive material.

Page 8 of 16 3.1.3 Document Review Key Messages The following key observations came from the document review task:

• A collection of 18 documents were reviewed and for the majority of the cases, the RIDM terms were not defined. [Recommendations 1 and 4]

• In many cases, the documents pointed to RG 1.174 for term definition.

[Recommendation 2]

• A common definition was found among several documents (e.g., RG 1.174 and NUREG-2122). However, some deviations from these definitions occurred and illustrate the uncertainty in the definition among the documents reviewed. [Recommendations 1 and 4]

• More consistency and clarity are needed in the definition of RIDM terms, if the staff are to understand and use them properly. [Recommendations 1, 2, 4, 5]

3.2 RIDM Cafés 3.2.1 RIDM Cafés Data Collection In order to identify and gain insights on how the concepts of risk and risk insights are currently understood and being used by staff and managers, a series of three café sessions were hosted.

To encourage free sharing of ideas, care was taken to run separate sessions for the staff and managers the first and third sessions were for technical staff and project managers, and the second session was for senior executive service, branch chiefs, and senior technical advisors.

For each session, the participants were split into tables of four to six people, where each table had a working group assigned scribe who recorded participant responses on a flipchart.

Scribes were asked not to interfere with discussions nor to add their own opinions. However, scribes were encouraged to ask questions where needed to clarify and better understand the opinions being relayed by the participants. Approximately 75 total staff and managers participated in the café sessions.

The questions that were discussed at the café sessions were split into three topics. Topic 1, Definitions, focused on gaining a better understanding the breadth of differing definitions that staff use to represent concepts related to risk and risk insights. Topic 2, Attitudes Toward Risk, targeted understanding how staff and managers feel about the current use of risk in their work and by the agency in general. Topic 3, Looking Forward, took aim at understanding how the agency can better support the needs of the staff to better leverage risk in future work. The thematically related questions within each topic were intentionally designed to overlap with one another, to initiate and fuel discussions at the table, and to elicit a range of viewpoints. The questions were arranged to start with broad questions, followed by more detailed questions that dug deeper into a variety of related topics. The list of questions used in each of the three café discussion topics can be found in Attachment 2.

3.2.2 RIDM Cafés Data Analysis And Findings After the café sessions were conducted, the results were collated and analyzed by the Task 2 working group to identify common threads and consistent themes. A complete list of responses from the cafés is included in Attachment 4.

Page 9 of 16 3.2.3 RIDM Cafés Key Messages and Observations (by Topic)

Topic 1: Definitions Key Message:

• There is inconsistency in the way staff interpret various terms related to risk, which subsequently affects how staff applies the terms in our work. [Recommendations 1 and 3]

Observations:

• There is broad agreement that the term risk is viewed as a quantifiable number and PRA results in number. Risk insights and risk-informed are in the eye of the beholder.

• There is little agreement on how to apply:

• Risk insights or risk-information in deterministic reviews. [Recommendations 2, 4 and 5]

• The concept of RIDM outside of a numerical value derived from PRA informed by RG 1.174. [Recommendations 1 and 3]

Topic 2: Attitudes Toward Risk Key Message:

• There is general agreement among the staff that applying risk or risk insights is useful in reaching decisions.

Observations:

• The staff feels that while there is high-level policy, it lacks the tools and guidance to apply risk or risk insights at the working level. [Recommendations 4 and 5]

• There is a gap between management and staff in the expectations of the use of risk or risk insightsmanagement believes there are plenty of policy documents, while staff believes it needs more tools to apply it consistently. [Recommendations 2, 4, and 5]

Topic 3: Looking Forward Key Message:

• A culture change is needed to make use of risk information and risk insights more in how we do our work on a daily basis. [Recommendations 2, 4, and 7]

Observations:

• There is a disparity between staff and management on the level of comfort associated with broadening the use of risk and risk insights. For staff this is due, in part, on the lack Page 10 of 16 of common understanding associated with risk related concepts, lack of trust in PRA, and status quo for technical reviews. [Recommendations 1, 2, 5, 6, and 7]

• Staff believes more working level guidance, training and tools are needed. (This observation also ties to efforts in RIDM Tasks 1, 3, and 4.) [Recommendations 5 and 6]

• Staff believes there should be high-level policy on RIDM. [Recommendations 4 and 5]

3.3 Online Survey 3.3.1 Online Survey Data Collection In order to supplement the results of the café sessions, as well as to give people who missed the café sessions a chance to participate, a ten question online survey was developed and administered using the Survey Monkey website. The first two questions were demographic questions and asked about the participants job function at the agency, and the types of risk-informed work they have performed. The next five questions inquired about the participants attitudes toward a variety of statements about risk and were scored with a sliding scale from Strongly Disagree to Strongly Agree. The last three questions addressed the participants views looking forward. Two of these questions addressed the sufficiency of current guidance and whether there is a gap between management expectations and staff use of risk or risk insights, and were scored with a sliding scale from Not at all to Absolutely. The last question was a free response field asking participants to describe barriers to incorporating risk and risk-insights in their daily work.

A total of 107 respondents participated in the online survey; 58 technical reviewers, 27 project managers, 17 branch chiefs, three senior executives, and two inspectors. The questions presented in the online survey can be found in Attachment 3 of this enclosure.

3.3.2 Online Survey Data Analysis And Findings The data from the online survey was analyzed using a variety of methods. Questions that were scored using a sliding scale were analyzed by looking at the statistical mean for all respondents, as well as comparing that for respondents each job function. Figures showing the results of each of those responses is shown in Attachment 5 of this enclosure. Due to the low number of inspector and senior executive respondents, two and three, respectively, comparisons that utilized those groups in isolation were deemed not statistically significant.

For instance, with Question 7, I have concerns with expanding the use of risk or risk insights by the agency (1 = Strongly Disagree, 5 = Neither Agree nor Disagree, 9 = Strongly Agree), the statistical mean response was close to 5, meaning that participants on average neither agreed nor disagreed with the statement. As seen in Figure 1 below, the average response for inspectors strongly agreed with the statement, while the average response for senior executives strongly disagreed with the statement, but this result is statistically inconclusive due to the low number of participants in either of those two categories.

Page 11 of 16 Figure 1. Responses for Q7 - I have concerns with expanding the use of risk or risk insights by the agency. (1 - Strongly Disagree, 5 = Neither Agree nor Disagree, 9 = Strongly Agree)

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9 The free response question was analyzed by collating all of the responses, and then grouping similar responses, and then tallying up the number of responses for each category. A complete list of all responses is provided in Attachment 5 of this enclosure.

3.3.3 Online Survey Key Messages and Observations The data trends indicate that on average:

• The staff views the value of risk and risk insights positively.

• The staff feels that probabilistic and deterministic methodologies can coexist at the NRC.

• The staff is neutral about expanding the use of risk and risk insights by the agency.

• The staff perceives a gap between management expectations and staff use of risk.

The strongest perceived barriers toward incorporating risk into their daily work are:

• A lack of consistent, detailed technical guidance on how to incorporate risk into specific types of reviews, and to determine a consistent threshold of acceptability.

[Recommendation 4]

• A lack of training and shared understanding of how to use risk in technical work, which results in inconsistent decisions made at the discretion of individuals.

[Recommendations 5 and 6]

• Prescriptive/deterministic regulations or policy that are incompatible with risk informed methodologies. [Recommendations 2 and 5]

Page 12 of 16

• A fear that dishonest or poor quality PRA has been/can be used to justify questionable practices. [Recommendations 5 and 6]

3.4 Practical Example 3.4.1 Practical example of RIDM Challenges Faced by the Staff A technical reviewer, as a member of NRR RIDM Task 2, performed a semi-pilot study using risk insights to reach a risk-informed decision regarding the technical basis acceptance of a licensees proposed alternative. Through the license amendment review process, there were differences of opinion between the technical review, technical management, project manager and project management, on the use of terms and how risk insights should be defined in their use and development in the safety evaluation (SE). Additionally, the pilot SE was shared with NRR RIDM Task groups 1, 2, and 3 for their comments. The following paragraphs discuss the resolution of those issues and the success path agreed on by all parties in the review process.

3.4.2 Practical Example Analysis and Findings From a deterministic perspective, the licensee would have had difficulty with the proposed alternative, as it was not consistent with past precedence for similar reviews. Nor did the licensee submit a request for review under RG 1.174, using a quantitative probabilistic risk assessment approach. As a result, staff discussed either changes to the licensees proposed alternative timeline or processing a denial of the request. However, in accordance with the managements directive to emphasize the use of risk-informed in decision making, the technical staff assessed if there was a qualitative risk insight approach that, along with the deterministic analysis, could allow the NRC staff to reach a finding of reasonable assurance of safety.

As part of the final approved SE, the staff used the following format to document the review:

1. The NRC staff considered the following risk insights to evaluate the results of the NRC staff calculations and review: . . and listed the criteria analyzed

2. Discussed each item, From a safety perspective

3. Discussed conservative assumptions

4. Discussed defense-in-depth actions or inspections

5. Concluded with a statement, Therefore, the NRC staff finds, through an analysis using risk insights and engineering judgment, that the licensee's proposed alternative has a minimal, if any, impact on safety.

Below is the final wording documented in the SE:

The NRC staff considered qualitative risk insights and engineering judgment to assist in reaching the overall NRC staff final conclusions.

Finally, the NRC staff considered the following risk insights to evaluate the results of the NRC staff calculations and review: licensee-provided inspection history of the subject welds, including previous finding of no reportable PWSCC

[primary water stress-corrosion cracking] [indications; NRC staff conclusions regarding PWSCC propagation and axial flaw growth; and conservative Page 13 of 16 assumptions used in the NRC staff's flaw evaluations that account for analysis uncertainties. From a safety perspective, the NRC staff notes that the degradation mechanism of concern is PWSCC. As noted previously, only circumferential PWSCC flaws can challenge the structural integrity of the DMBWs [dissimilar metal butt welds]. While the NRC staff analysis found that axial PWSCC flaws could cause leakage during the period of extended inspection interval, these axial flaws would be limited in size by the width of the weld. Beyond the weld, the base material of the pipe and nozzle are not susceptible to PWSCC. Furthermore, PWSCC flaws are very tight intergranular flaws that have limited leak rates such that no concerns for a loss-of-coolant accident could occur simply because of an axial PWSCC flaw in a DMBW.

Additionally, the NRC staff notes that there are several conservative assumptions in the flaw analyses performed both by the licensee and the NRC staff. The primary conservatism among both evaluations is that an axial flaw of 10 percent depth has already initiated and is growing immediately after the last volumetric inspection. While NRC staff recognizes that the hypothetical axial flaw could cause leakage during the proposed inspection interval, the NRC staff finds that this consequence is unlikely based on the conservative assumption that an axial flaw has already initiated of a precise size to be missed by a qualified volumetric exam and in a tensile stress state that would allow continued growth.

Furthermore, the NRC staff notes that volumetric inspections are not the only method used by licensees to assess structural integrity of the subject welds.

Plant walkdowns and plant leakage monitoring systems also provide defense-in-depth measures to assess the leak tight integrity of the subject welds.

Therefore, the NRC staff finds, through an analysis using risk insights and engineering judgment, that the licensee's proposed alternative has a minimal, if any, impact on safety.

3.4.3 Practical Example Key Messages and Observations From the above example, the NRR RIDM Task 2 working group identified items for which clarity in a process document would be useful, which are listed below:

1. Does the term risk insights need to be qualified with a Quantitative or Qualitative term? [Recommendation 2]

2. Is the definition of risk insights sufficient, or should it be qualified with as a function of engineering judgement or as a function of sensitivity calculations, etc.? [Recommendations 1, 5, and 6]

3. As risk is a function of probability and consequences, when we are not using a PRA analysis, how do we best describe the probability using informed engineering judgement? The agreed upon path forward was consequence is unlikely; however, this is a weak term versus a probability statement. [Recommendations 5 and 6]

Page 14 of 16 4.0 RECOMMENDATIONS Recommendation 1 Establish NUREG-2122 as the main glossary for risk-related terms so that there is one authoritative source.

BASIS:

The findings from the Task 2 working groups document review indicate a lack of clarity and consistency in the current guidance of how risk-related terms are defined. This is reflected in RIDM cafés and survey (e.g., the staff as a whole does not have a common understanding of risk-related terms).

There are three considerations regarding implementation of Recommendation 1:

• Make no changes to the NUREG-2122 definitions.

• Enhance the NUREG-2122 definitions with clarifications, and/or

• Add definitions as needed so that all terms are defined in one glossary.

The first consideration requires the least amount of effort and time. The second and third considerations may require approval from the Commission, according to the preamble of NUREG-2122.

Recommendation 2 Recognizing that the five RG 1.174 principles are well known, develop guidance for use of other principles that are important to RIDM.

BASIS:

The teams efforts initially focused on terms related to four of the five RG 1.174 RIDM principles:

defense-in-depth, safety margins, performance monitoring, risk, and risk insights. During their efforts, NRR RIDM Tasks 1 and 3 working groups identified some additional terms that should be considered when establishing a common dialog around risk:

• Quantitative

• Qualitative

• Prescriptive (probably the same as deterministic - but prescriptive is a bit more clear)

• Performance-based

• PRA

• Risk-informed

• Risk-informed decision-making

• Risk insight vs. PRA risk insight (or PRA insight)

• Risk assessment In addition to these terms, the context where it would be appropriate should be identified. For example, identify where the risk-related terms would be appropriate for safety evaluations for Type 1, 2, and 3 applications.

Page 15 of 16 Note: RIDM Task 2 Recommendation 2 overlaps with the efforts of NRR RIDM Tasks 1 and 3.

Implementation should include ensuring consistency with the other working groups.

Recommendation 3 Develop a desktop glossary of terms and make available as a quick reference guide for more commonly used RIDM terms.

BASIS:

Given that NUREG-2122 is a large document, a quick reference guide would be beneficial.

Recommendation 4 Enhance the current review guidance and office procedures to place greater emphasis on using risk insights. In addition, update current guidance documents with consistent defined terms and reference documents (including Recommendation 5).

BASIS:

The results from the document review, RIDM cafés, and survey indicate inconsistent understanding and a lack of clarity in guidance documents.

Implementation may include:

• Short-term changes: Developing interim guidance, branch technical positions, desk guides, and NRR office instructions (LIC-504).

• Long-term changes: Updates to management directives, regulatory guides, and the standard review plan (SRP Section 19.2, Review of Risk Information Used to Support Permanent Plant-Specific Changes to the Licensing Basis: General Guidance.)

• Defining risk-related terms in documents where they are used but not defined, or refer to risk glossary, NUREG-2122.

Recommendation 5 Develop guidance (e.g., management directive) that identifies the hierarchy of RIDM documents, provides a roadmap on which RIDM documents to use, and complements current regulatory approaches, as appropriate. It could be an overarching document that guides the staff on the framework on how to use risk, with NUREG-2122 as the glossary reference. For usefulness, the roadmap needs to be broadly defined and used - a holistic how-to. Without changing them, clarify definitions as needed in the risk guide. The following are examples of how Recommendation 5 could be implemented: create a desk reference for reviewers Mini Guide to Risk, and a flowchart or pyramid that shows the hierarchy of NRCs policy and guidance documents.

Note: NRR RIDM Task 2 Recommendation 5 overlaps with NRR RIDM Task 3 efforts.

Implementation should include ensuring consistency with NRR RIDM Task 3.

Page 16 of 16 BASIS:

The data from the document review, RIDM cafés, and online survey indicate (1) an inconsistent understanding, (2) a lack of clarity in guidance documents, (3) requests for additional guidance, and (4) enhancements to current guidance documents.

Recommendation 6 Develop training for staff to facilitate using risk insights for reviews that dont require PRA.

BASIS:

The data results from the RIDM cafés and survey indicate that training is needed for staff to facilitate using risk insights for reviews that dont require PRA and to ensure consistency and common understanding throughout the agency.

Recommendation 7 Augment position descriptions and performance appraisal elements and standards to include the use of risk information and risk insights to enhance decision-making.

BASIS:

In order to affect change in the culture of the agency regarding risk, we need to increase awareness, emphasize importance, provide guidance and training, and hold staff accountable.

Attachments:

1. Document Review Results

2. RIDM Café Questions

3. Online Survey Questions

4. RIDM Café Results

5. Online Survey Results

, Attachment 1 Page 1 of 4 DOCUMENT REVIEW RESULTS AND FINDINGS NRR RIDM ACTION PLAN TASK 2 The following are word clouds for definitions of risk-related terms, as indicated. In the definitions, the words more commonly used have a larger font size, while words less commonly used have a smaller font size. See Enclosure 3, RIDM Task 2 Findings and Recommendations Report, Section 3.1 for the list of policy and guidance documents that were reviewed.

Figure 1. Defense-in-Depth

, Attachment 1 Page 2 of 4 Figure 2. Safety Margins

, Attachment 1 Page 3 of 4 Figure 3. Performance Monitoring

, Attachment 1 Page 4 of 4 Figure 4. Risk

, Attachment 2 Page 1 of 2 RIDM CAFÉ QUESTIONS NRR RIDM ACTION PLAN TASK 2 Topic 1: Definitions How would you define risk or risk insights?

Do you feel there is a relationship between risk assessment and engineering judgment? If so, how would you characterize it?

How do you interpret the use of defense in depth? And how does that relate with risk or risk insights?

How would you interpret the use of safety margins? And how does that relate with risk or risk insights?

How would you interpret the results from risk or risk insights in helping you make a decision?

Can you define what risk-informed means to you?

Can a reviewer only use risk or risk insights if the licensee requests a risk informed review under RG 1.177, An Approach for Plant-Specific, Risk-Informed Decision-Making: Technical Specifications?

Do you feel that risk assessment should only be performed only by probabilistic risk assessment (PRA) specialists?

Do you feel there is a difference between risk and risk insights? If so, how would you characterize it?

Have you heard any misconceptions of the use of risk or risk insights in our work from (unnamed) colleagues?

Topic 2: Attitudes toward Risk Do you feel the current use of risk or risk insights by the agency is appropriate? Why or why not?

Do you feel it is appropriate to use risk or risk insights in the NRCs current regulatory framework? Please explain.

Do you feel that using risk or risk insights impacts how you would make a finding of reasonable assurance either positively or negatively? Please explain.

Do you feel that using risk or risk insights could be considered within the realm of engineering judgement or not? Please explain.

Do you feel that the use of PRA methods or conclusions results in an unwarranted decrease in safety margins? Can you elaborate on how?

, Attachment 2 Page 2 of 2 Do you feel that probabilistic and deterministic methodologies can coexist at the NRC?

Do you feel that there is a connection between engineering calculations and performing risk analysis? If so, what is it?

Can you cite specific examples where you felt the use of risk or risk insights was particularly appropriate or inappropriate?

Do you have concerns with expanding the use of risk or risk insights by the agency?

Topic 3: Looking Forward Can you describe what you would consider barriers to incorporating the use of risk or risk insights in your daily work?

Do you think the current guidance is sufficient for you to feel comfortable using risk or risk insights in your decision-making? Please explain.

Do you think there is a gap between management expectations and staff use of risk or risk insights? Please explain. If so, what should the NRC do to close that gap?

What additional guidance, tools or training do you think would increase your comfort with using risk or risk insights in your decision-making?

What do you think about using risk or risk-insights to refocus or prioritize the level of effort expended on different parts of reviewing a license amendment request?

What changes do you think are needed for you, or your colleagues, to feel comfortable using risk or risk insights in your daily work?

Do you foresee additional leveraging of risk in your future work? If so, how?

, Attachment 3 Page 1 of 1 ONLINE SURVEY QUESTIONS NRR RIDM ACTION PLAN TASK 2 Q1: What is your job function at the agency?

o Technical Reviewer o Inspector o Project Manager o Branch Chief o Technical Assistant o Senior Level Advisor o Senior Executive o Other (please specify)

Q2: Which of the following have you had review experience with (check all that apply)?

o RG 1.174 o RG 1.175 o RG 1.177 o 50.65 o 50.67 o 50.69 o TSTF-425 o TSTF-505 o NOED Questions Q3 - Q7 were scored on a sliding scale from Strongly Disagree to Strongly Agree.

Q3: The current use of risk or risk insights by the agency is appropriate.

Q4: The use of risk or risk insights result in an unwarranted decrease in safety margins.

Q5: Risk or risk insights are a form of engineering judgment.

Q6: Probabilistic and deterministic methodologies cannot coexist at the NRC.

Q7: I have concerns with expanding the use of risk or risk insights by the agency.

Questions Q8 - Q9 were scored on a sliding scale from Not at all to Absolutely.

Q8: Do you think the current guidance is sufficient for you to feel comfortable using risk or risk insights in your decision-making?

Q9: Do you think there is a gap between management expectations and staff use of risk or risk insights?

Question Q10 was a text field for free answer.

Q10: Can you briefly describe what you would consider barriers to incorporating the use of risk or risk insights in your daily work?

, Attachment 4 Page 1 of 26 RIDM CAFÉ RESULTS NRR RIDM ACTION PLAN TASK 2 These are the raw responses from the cafés merged. Responses are from technical reviewers and project managers from the Office of Nuclear Reactor Regulation (NRR), Office of New Reactors (NRO), and Office of Research (RES). Responses from senior executive service (SES), branch chiefs (BCs), and senior levels (SLs) are as indicated.

Topic 1: Definitions How would you define risk or risk insights?

Technical Reviewer and Project Manager Responses:

• Risk chance or probability of something going wrong. Numerical calculation.

• Risk insight vulnerability, something that you use for or learn from risk assessment, what you learn from insight use of raw data.

• Risk = Probability

• consequence high probably and low consequence or vice-versa

• Risk-insights can be either an output from a tool (PRA, etc.) or judgement.

• Risk assessment always uses RG 1.174 five principles.

• As a consideration of possibility vs. severity, or consequence vs. frequency.

• Risk insight can be ambiguous, and can be a black hole with no getting out.

• Consequences and probabilities; actions with immediate consequence; actions that cause latent effects; related to the plant that is within the context (plant-specific); risk results may not be identical all the times; related to components safety functions.

• Risk triplet. Hazard and how to mitigate.

• Risk is probability x consequence.

• Risk is the threat or danger posed by something

• Risk insight is using the risk equation to determine whether something should be done.

• Risk insight is the information about whats important, important drivers, the good and bad of something.

• Risk is the probability of something bad happening. Risk triplet - 1) what can go wrong, 2) likelihood, 3) consequences.

• Risk insight using risk to inform decisions. The revelation of new information you are not aware of through use of risk (e.g. PRA).

• Risk: Consequences x Probability.

• Risk insight: Conclusions you can draw from risk to inform decisions, or the input used to evaluate risk.

• Risk triplet - what can go wrong, how bad, how likely.

• Chances that a bad event could occur; what is the severity level or impact of an event, consequences.

• Risk Insights: Using risk to make a decision.

, Attachment 4 Page 2 of 26 SES, SL, and BC Responses:

SES

• Risk: Risk triplet - what can happen, what are the consequences, how likely; focus on consequence vs. focus on likelihood of occurrence; not necessarily quantitative.

• Risk Insights: Qualitative consideration; what is driving the risk outcome? What are the influential assumptions?

SL

• Risk = Risk Triplet. Quantitative preferred, Qualitative allowed BC

• Probability x Consequence

• Risk triplet

• Using probability to assess safety significance; probability x consequence; insights can come from op experience, expert panel, inspections Do you feel there is a relationship between risk assessment and engineering judgment? If so, how would you characterize it?

Technical Reviewer and Project Manager Responses:

• There is a correlation between Engineering judgment and Risk assessment.

• Engineering judgment is first then risk assessment is integrated.

• Risk assessment provides insight into engineering judgment. Is a result of engineering judgment.

• Engineering judgment is based on experience. Leverages technology and reasoning.

• Risk assessment always uses RG 1.174 five principles.

• Engineering judgment is a feeling, based on experiences, can be backed up with calculations, but can also have more hand waving. Based on deterministic methods.

• Risk assessment is based on the results of a PRA, based on calculations, and operational experience. A risk assessment can help to choose the objectives of an engineering assessment, and can influence the course of engineering judgment.

• Engineering judgments must be justified (in the context of safety); engineering judgments are quantitative and more in-depth; risk assessment can be quantitative or qualitative; risk assessment is less in-depth.

• Yes. Risk assessment requires engineering judgement to assess inputs and determine if they are appropriate

• The terms are not mutually exclusive - there is a relationship between them.

• Engineering judgment is a term that could be broadly defined and mean many things.

• Risk assessment is theoretical, engineering judgment is what has occurred/what youve learned.

• Risk assessment is a more formal determination than engineering judgment.

• There is a relationship between Engineering judgment and Risk assessment.

• Risk Informed - Risk assessment and Engineering judgment and other relevant information.

• Risk assessment provides insight into engineering judgment and engineering judgment provides insight into Risk assessment.

, Attachment 4 Page 3 of 26

• There always has been. Engineering judgment uses experience to see how likely something is. In a sense engineering judgment is a qualitative form of risk assessment.

• Risk assessment is more quantitative, while engineering judgement comes from experience and knowledge.

• Risk assessment is a structured evaluation (of your feeling).

• Risk assessment is probable; engineering judgement is practical and common sense.

SES, SL, and BC Responses:

SES

• Yes. They are (should be) intertwined. Unconsciously.

• They are largely synonymous. One is more structured.

• Engineering judgement inherently considers risk.

• Engineering judgement would inform the selection assumptions of risk values used in the PRA.

SL

• Risk assessment is more structured, while engineering judgement is a catchall term for poorly defined assessments. Both are different ways of quantifying an issue.

BC

• Yes.

• Risk assessment is a way to quantify engineering judgement.

• Risk assessment was always part of engineering judgement.

• Engineering judgement is qualitative risk assessment.

• Engineering judgement can screen out qualitatively initiating events.

• Engineering judgement -> consequences.

• Risk assessment -> probabilities.

• Yes, they are roughly the same.

How do you interpret the use of defense in depth? And how does that relate with risk or risk insights?

Technical Reviewer and Project Manager Responses:

• Defense in depth can reduce/decrease risk.

• Defense in depth is backup systems, redundancy, EP.

• Defense in depth and risk of any one fission boundary failure are inter-related.

• Defense in depth is beyond the design objectives. It goes back to 100%

probability. Is based on deterministic logic, so defense in depth and risk almost dont touch. The use of defense in depth can reduce risk.

• If a layer of defense is taken away, there is less margin; if a layer of defense is removed, one must make sure actions are taken to replenish the removed layer of defense; diversity, redundancy, independence; if a layer of defense is removed, risk increases; based on historical data or operating experience, risk may increase or decrease when defense in depth is altered.

, Attachment 4 Page 4 of 26

• Defense in depth is multiple diverse mechanisms protecting something. It covers remote possibilities.

• Defense in depth is the mitigation of uncertainties.

• Risk assessment determines the consequences that must be protected against and how much defense in depth is therefore needed.

• Defense in depth is a strategy to lessen risk. Defense in depth is a backup to risk.

• There is a PRA policy statement, but there is NO risk informed policy statement.

• Defense in depth reduces risk. It exists to handle uncertainties that exist in improbable stuff. It can affect either the consequence or the probability, so either side of the risk equation.

• It is separate from risk.

• It is added safety features, layers of protection. It can provide warm fuzzy feelings.

They are barriers in case previous barriers fail.

• Defense in depth is the second principle (RG 1.174) - redundancy, independence, a way to mitigate risk.

• This is done with physical barriers, and administrative barriers plus compensatory measures.

• More defense in depth in one system is not always better because it could cause problems in another area (such as fire mitigation).

SES, SL, and BC Responses:

SES

• Defense in depth is more important when the likelihood of an event or failure is not known.

• Defense in depth protect both the system level or fission product barriers.

• Defense in depth should be proportional to risk.

• In an ideal world, defense in depth is captured in the PRA model.

SL

• Defense in depth is both for known and unknown uncertainties. It involves multiple layers and redundancies.

BC

• What if we are wrong -> DID

• Layers -> Training, Records, Procedures

• Detect, Prevention, Mitigate

• Amount or quality of layers of defense in depth relate to risk

• Human error

• More layers of defense in depth are needed for greater uncertainty or risk

• The more defense in depth you have the less risk insights you need.

• Defense in depth minimizes risk; risk insights tell how much defense in depth is needed.

, Attachment 4 Page 5 of 26 How would you interpret the use of safety margins? And how does that relate with risk or risk insights?

Technical Reviewer and Project Manager Responses:

• Decrease safety margin, then increase risk. Safety margin provides information on risk.

• Safety margin and defense in depth are bottlenecks to risk. The more safety margin and defense in depth you have the more risk you can tolerate.

• Very similar to #3, safety margins provide a built-in cushion in calculation and acceptance criteria to account for beyond anticipated events. Increasing safety margins can lessen risk, but beyond that, because it is based on deterministic logic, the two almost dont touch.

• Bounds that are not to be exceeded; Safety limits; If the limit is crossed, risk increase

• It is the buffer in design specifications. Distance in analysis from limits. As safety margin goes down, probability of failure goes up.

• Safety margin is related to defense in depth - similar concepts - both are intended to deal with uncertainties.

• Safety margin - The measure of amount of risk you are willing to accept. The difference between failure/bad consequences and normal operational state.

• Safety margins are deterministic, and relate to the design basis. This makes it inherently different from risk. Safety margins do not incorporate probability or consequence.

• Another opinion is that safety margins do include risk, in that it incorporates engineering judgment, which has a connection (discussed above) to risk.

• A straight definition of safety margins is a margin between design and failure.

• Safety margins is the third principle (RG 1.174). Safety margin is cushion between what is known and unknown to address uncertainty.

SES, SL, and BC Responses:

SES

• Largely defined at the system or engineering level.

• Safety margins defend from both unknowns and known risks.

• Lack of consensus on safety margins in use.

• Safety margins prevent, mitigate, or barrier.

SL

• Safety margin is confidence in overlapping barriers. Confidence that design will meet needs BC

• How do you know successes are really successes?

• Initial design value plus margin for uncertainty.

• The more uncertainty there is the more margin you need.

• Safety margin provides credibility to risk insights.

• Safety margin provides credibility to PRA output.

• Safety margin is set assuming an event will happen with probability = 1; risk insights inform the amount of margin needed.

, Attachment 4 Page 6 of 26 How would you interpret the results from risk or risk insights in helping you make a decision?

Technical Reviewer and Project Manager Responses:

• Risk insights will help identify vulnerabilities. Risk insight will help determine the process you will be in.

• The more safety margin and defense in depth you have the more risk you can tolerate.

• It is something to consider, but is not used to make the decision itself. In the end of the day, deterministic findings are controlling.

• If the consequence is significant, one need to pay more attention; CDF, LERF; depend on how risky the action is.

• It shows the measure of increase in risk associated with the change to the licensing basis or the deviation from deterministic standards.

• Risk is numbers/probability.

• Risk insights are gained from the numbers and may influence or alter a decision.

• Additional layer of information; marginal acceptance criteria; risk Insights will help drive decisions.

• It helps you understand what is important. It helps you to prioritize, so you can decide whether to accept the applicants statement, or dig deeper or ask for more in cases that have higher risk significance. It can also help to provide determine reasonable assurance.

• The results from risk or risk insights complement, but do not override the rules and regulations.

• Risk results provide confidence that the right approach is right. The more risk is understood, the better informed the decision will be.

SES, SL, and BC Responses:

SES

• (Question skipped - short on time)

SL

• It allows you to look at the range of risk/consequences BC

• Informs the decision, doesnt make the decision.

• Would use for temporary changes but not for the DB itself [Scribe note: Group contrasted RIDM for operating plant situations (ok) with eliminating the large break LOCA (not ok)]

Can you define what risk-informed means to you?

Technical Reviewer and Project Manager Responses:

• Using risk insights into decision-making.

• RG 1.174 five principles.

, Attachment 4 Page 7 of 26

• Same as with above, it informs the decision-making, but is not controlling on its own.

• A good idea because one does not have unlimited resources; numerical assessment is better than engineering judgement, which could be qualitative; saving money at the cost of safety.

• Its a decision made with the information from risk or risk insights.

• Risk Informed - Applying the 5 step criteria. Base decision on risk in addition to defense in depth. Quantitative and Qualitative.

• Continue with above - risk-informing means to use results from risk to complement, but not override the rules and regulations.

• Dont make decisions based entirely on a risk number, since that would be risk-based.

• The line can get fuzzy when you get to smaller chunks, because at what granularity does that small chunk become risk-based?

• This is subject to broad interpretations, and is not well defined.

• The fear is that licensees can use this to get what they want based on their needs.

In total, this is not well structured for rigorous or consistent use.

• Risk informed is how you quantify risk by calculation.

• It is a tool used in making the final safety decision.

SES, SL, and BC Responses:

SES

• RG 1.174 the integrated decision-making diagram.

• Risk informed equals intelligent decision-making.

SL

• Following a risk assessment, you have information to help you look at performance in more detail.

BC

• Risk informed is qualitative and quantitative together.

• Not just PRA.

• Yes - understanding risk in making a decision, including how much effort to expend on an issue.

Can a reviewer only use risk or risk insights if the licensee requests a risk informed review under RG 1.174?

Technical Reviewer and Project Manager Responses:

• Perform PRA, but discuss with other staff members.

• Yes, but no guidance exists.

• A reviewer can use this as one of the factors, but not as the only factor. In short, yes they can.

• RG 1.174 has multiple deterministic elements that must also be considered.

• No.

• No.

• No.

, Attachment 4 Page 8 of 26 SES, SL, and BC Responses:

SES

• (Question skipped - short on time).

SL

• Yes BC

• Loaded question.

• Risk insights can be used for other reviews SRP 19.2 Appendix D.

• If you know, it is high risk Appendix D allows you to ask more questions.

• No.

Do you feel that risk assessment should only be performed only by PRA specialists?

Technical Reviewer and Project Manager Responses:

• Yes.

• No, but no guidance exists.

• That is too limiting. First, logistically we dont have enough PRA specialists, so others should be able to and will need to. Anyone on the technical staff can do a reasonable risk assessment, with appropriate training. In the end this is a team decision, and the information and findings should be shared across the disciplines.

• Risk is a defined concept while insights are results from this analysis.

• You need both disciplines because there are risk and deterministic components.

• PRA and risk are not the same - PRA is one way to complete a risk assessment.

• To the extent the risk assessment will rely on PRA/numbers, it should be done by a PRA specialist.

• No and Yes.

• Everyone should be aware of how it plays in the big pictures. Everyone from technical reviewers to project managers should be versed in the basics of the tool.

For instance, like how technical reviewers should know the basics of being an inspector too. At a high level basic point, no. For more detailed and complex PRA calculations, maybe yes.

• We can help the PRA analysts with understanding structural mechanics.

• We need to be careful about saying PRA risk reviews can only be done by experts because it could discourage the rest of the staff who need to be contributors to the reviews.

SES, SL, and BC Responses:

SES

• No, but the staff should be sufficiently trained.

SL

• No, the decision is linked and must be systematic. The engineering judgement and risk assessment are linked.

, Attachment 4 Page 9 of 26 BC

• PRIB book use, inspectors.

• No.

Do you feel there is a difference between risk and risk insights? If so, how would you characterize it?

Technical Reviewer and Project Manager Responses:

• Yes.

• Risk is more academic. It is characterized by equations, and mathematics.

• Risk insight is more hand-wavy, can be the answer gotten from a risk assessment, but like with risk informed, it applies soft judgment and logic and can be manipulated.

Can have a negative connotation, sort of like with engineering judgment.

• Risk is a defined concept while insights are results from this analysis

• Yes, there is a difference. Risk is an inherent or calculated value (even though it can be subjective as well).

• Risk insight is what is used to make decisions based on the above calculated or subjective values. The results are more debatable.

• Risk insight provides associated risk information.

SES, SL, and BC Responses:

SES

• (Question skipped - short on time).

SL

• They are complementary. Risk feeds into risk assessment.

BC

• Risk equals numbers; risk insights means knowledge of risk.

• Yes. Risk insights are plain English

• AP-1000 has plain language discussion, no colors or numbers

• Risk can be a number or many things.

Have you heard any misconceptions of the use of risk or risk insights in our work from (unnamed) colleagues?

Technical Reviewer and Project Manager Responses:

• Yes.

• Cant use risk outside of RG 1.174.

• There are a number of misconceptions, especially coming from the licensees, who act as if this is a universal get out of jail free card, and can be used to get out of something they otherwise cannot meet. This is still in its infancy, and we are working with licensees to formulate the details on the fly. So we have yet to know what in our belief set is a misconception, because it hasnt solidified.

• Yes.

• There is a lack of understanding of what risk-informed means.

, Attachment 4 Page 10 of 26

• Some folks believe a decision should be based only on PRA - this is wrong.

• It is best to have some PRA insights as one input out of many in a decision, but other considerations need to be included.

• Yes. If someone says that something is not risk significant - it automatically seems to shut the book on the discussion, without understanding the limits of the risk technology. Low risk issues can still be a compliance issue, and can still be important. Technical reviewers and project managers cannot discount those.

• Risk/risk insights can only be used if the licensing action contains a full-blown RG 1.200 PRA.

SES, SL, and BC Responses:

SES

• (Question skipped - short on time).

SL

• People think it is a hoax. No credibility to risk. Answer can be whatever you want.

Will trust when it is not subject to manipulation.

BC

• Yes. Terms like magic; you can get any answer you want using risk.

• Yes.

Topic 2: Attitudes toward Risk Do you feel the current use of risk or risk insights by the agency is appropriate?

Why or why not?

Technical Reviewer and Project Manager Responses:

• Not enough. Inadequate training, communication, and not enough/appropriate guidance.

• Group was asked in who has actually read RG 1.174. Only about 30% have.

• Group thought that there is no clear guidance on the use of risk. They thought the use was very subjective and there was too much variation in the thinking.

• Some said that use of risk in the maintenance rule would be much different than how its used in licensing space.

• Perhaps this is a misconception, but some feel that risk can be used to push troublesome stuff through. We can see that the licensees want to use this so badly -

why is that? Just look at how excited licensees are toward TSTF 505 and 50.69.

Dubious.

• In the end, deterministic still rules the day. We must convince staff about the rigors of risk, and the current messaging does not seem to do that. For instance, one staffer related method discussed in the P-105 class, The Jeffreys Prior, which shows that with a lack of information, the system is modeled as 50:50. And in our work, information can often be a bit thin, so in many cases we are relying on little more than a fancy coin toss.

• One major problem is the lack of training. Risk isnt explained well enough for appropriate use at this agency.

, Attachment 4 Page 11 of 26

• It is good but could be better; No cross-discipline interactions; necessary in the past but there is not much impact now (compliance issue vs. using insight);

beyond-design-basis is a positive example of using risk insight.

• Yes. Some areas are better than others. Guidance could be expanded and clarified.

• In general, yes, but only in that we follow existing processes/procedures. Those processes/procedures need improvement.

• No, because it is not clearly defined. For example, significance determination process is risk based vs risk informed. Some people think that we are giving away the farm.

• It is sometimes inappropriate, for instance with a certain risk-based screening tool.

• That said, it is usually appropriate.

• We spend too much time and effort focusing on assessment. With a relative lack of information we are led to make reductive decisions.

• No, its not appropriate because there is a lack of understanding and lack of training.

• Yes, when used appropriately.

SES, SL, and BC Responses:

SES

• It is too rigid when only focused on the PRA perspective.

• Not appropriate, because we are attempting to shift the paradigm without the infrastructure in place.

SL

• Yes, tools are mature and would be inappropriate to ignore. Operating experience is important and feeds into risk.

BC

• Sometimes too much, sometimes not enough.

• Yes. Heading in the right direction.

• Insufficient guidance.

• Technical reviewers.

• HQ and Region.

• Have not document the lessons learned.

• Knowledge management is not done well.

Do you feel it is appropriate to use risk or risk insights in the NRCs current regulatory framework? Please explain.

Technical Reviewer and Project Manager Responses:

• Yes. Helping the agency understand the difference between risk, risk insights, and PRA.

• The PRA branch is currently overwhelmed -- so clearly it is appropriate, since they are doing lots of stuff. Perhaps this also highlights a staffing need. Are we ready for this? See answer to #1.

• Yes; No; Compliance issue does not need to use risk; whole regulatory framework maybe good to use risk but not individual compliance issue.

• Yes.

, Attachment 4 Page 12 of 26

• Yes, as articulated in WASH-1400 and the Commission Policy Statement on PRA.

However, we do not have a Commission Policy Statement on risk-informed decisions. This is different than PRA since PRA is just one aspect of risk.

• Use of risk is sometimes controversial since it carries baggage. Do you use PRA, deterministic factors, how much should each factor into the decision? The reason is it tends to be controversial; people believe (whether they support it or not) that risk or risk insights = PRA and this is not true.

• Yes. Provides a more integrated decision-making. Need better guidance.

Regulations/framework is not conducive to risk informed approach, and more toward deterministic.

• Yes, absolutely. There are limits though, for instance some of the regulations are limiting since they require specific things, which are not compatible with risk.

• Yes, for maintenance it improves efficiency.

• As for risk informed reviews, they are inefficient because the inclusion of risk information makes the reviews more difficult and take longer to complete.

• PRA and technical branches need to talk (work together).

• In the current regulatory framework, the process is not perfected.

SES, SL, and BC Responses:

SES

• A culture change is needed to fix the attitude that compliance = safety. No checklist mentality.

SL

• Yes. Need to be competent at risk to include it. Need revision to standard review plan and regulatory guides to include. Shouldnt include until framework is in place BC

• Current regulatory framework generally supports, but not everywhere.

• Yes, PRA policy statement.

• Risk triplet with or without a PRA.

• Qualitative information is risk triplet.

Do you feel that using risk or risk insights impacts how you would make a finding of reasonable assurance either positively or negatively? Please explain.

Technical Reviewer and Project Manager Responses:

• Yes. Risk and risk insights impacts decisions positively and negatively.

• Risk is baked into reasonable assurance determinations. You cannot make these determination without thinking about risk.

• In general there is a positive influence. It is another factory to weigh in, so it always adds a data point. It will make an impact in our findings. We just have to stay vigilant about making sure it adds data and does not take any away.

• For findings, yes; for compliance issues such as 50.59, no; yes, if one is making engineering judgement, it is good to have data to support it; some impact.

• Yes, it could be positive or negative depending on how risk is used.

• It would lead to a more informed decision.

, Attachment 4 Page 13 of 26

• Risk informs decisions. Risk is a double edged sword in that it can go either way (positively or negatively).

• Usually results in a positive impact. There are disagreements between deterministic and risk, and it can conflict with the regulations. The impact can also be negative.

With some requirements, it is inappropriate to use risk.

• If applied properly then the finding would impact the finding positively.

• Reasonable assurance equals 10-6 (CDF/LERF).

SES, SL, and BC Responses:

SES

• Risk/risk insights have a positive impact.

• Reasonable assurance is not a defined term.

• Risk insights are tools that could be used to gage reasonable assurance.

• The antonym of reasonable assurance is absolute assurance.

SL

• Once framework is there, it would have a positive impact BC

• Yes, but there may be regulatory prohibitions.

• Yes, but it is not clear how to do it. No process: if in RG 1.174 or significance determination process but murky outside of established processes.

Do you feel that using risk or risk insights could be considered within the realm of engineering judgement or not? Please explain.

Technical Reviewer and Project Manager Responses:

• Yes. Engineering judgment includes risk. Risk is used to determine priority.

• It is, but there is not guidance or consistency in its use.

• It already is considered. Its just not codified. Engineering judgment uses information about how likely (or unlikely) something is to happen all the time to determine focus or findings. We just arent doing it mathematically in engineering judgment, whereas we are in a risk assessment.

• Risk helps to determine engineering judgement; risk can be used to determine plant shutdown; risk can be used to extent the completion time; risk-informed gives us some numbers; if not risk-informed, that is judgement only. Risk information and input has been changed by the management, not necessarily by tech reviewers.

• Yes.

• It dependsmuch discussion, but not alignment, on what the scope of engineering judgment is.

• Yes. Risk is part of engineering judgment.

• Yes, it is not fully contained within engineering judgment, and it also can exceed the realm of engineering judgment.

• Yes. With engineering judgement, use experience to know what to do when making a decision. Risk helps quantify your gut call.

• How do you get the risk numbers?

, Attachment 4 Page 14 of 26 SES, SL, and BC Responses:

SES

• Yes.

SL

• Yes.

BC

• Yes.

• Yes. It has always been part of it.

• Uncertainty in PRA plus engineering judgment.

• Some risk insights some qualitative.

Do you feel that the use of PRA methods or conclusions results in an unwarranted decrease in safety margins? Can you elaborate on how?

Technical Reviewer and Project Manager Responses:

• PRA could potentially decrease safety-margin in systems which may be non-critical.

• Mixed comments. Many said its a culture issue. Deterministic reviewer do not always think qualitatively about risk.

• Not really, not in our current use of it. It helps to provide a better understanding of where the real safety margins are. We end up better informed, so reductions shouldnt be unwarranted. The caution though is that this may open the door to future uses that guide us down the slippery slope of relying too heavily on risk to provide unwarranted decreases in safety margins. This is a concern if we get too comfortable with PRA. If you plot how much a reviewer trusts risk, vs. the amount of experience the reviewer has, we think that trust levels will decrease with experience.

• Depends; No, safety culture is not risk related. No, compliance issue is not risk-related; If 10 CFR Part 50 Appendix B can be risk-informed, then PRA would not work because Appendix B provided a foundation; maybe not for overall use but ok to be used for individual analysis.

• Yes, if risk-based; no, if risk-informed.

• It would be difficult to erode safety margin in an unwarranted manner because safety margins and regulatory requirements should factor into the PRA. Further, we dont engage in risk-based decision-making - we dont (shall not) rely on just PRA.

• It can/may, but not always (PRA and safety-margin).

• Yes, risk technology (or practitioners) are not perfect. In some cases, the results you get from a risk analysis are only limited by your imagination. Not many rules or structure. You can come up with nearly any conclusion you want based on how you choose to run your analysis, or with even small finesses of probabilities that are selected as a given value, and not a rigorously calculated value. If used honestly, it can provide good information, but people can intentionally misuse it to get the results they want without provable ramifications. There are too many unknown unknowns to assure that it is being used right, or to show that it is intentionally being misused.

• You can eat into the safety margin, but it is not unwarranted decrease because there is still margin remaining.

, Attachment 4 Page 15 of 26 SES, SL, and BC Responses:

SES

• Safety margins are a deterministic concept.

• Yes. PRA = realistic assessment. Safety margins = conservatisms stacked between design and realistic values.

SL

• It depends on quality. Garbage in = Garbage out & Quality in = Quality out.

BC

• Depends on the size of the margin and the quality of the data.

• Sometimes - can cause a change in focus. If PRA says something is not risk important it may be inappropriately ignored - hydrogen 50.44.

• PRA could result in decrease, but this is not inherent in PRA. But PRA could be gamed.

Do you feel that probabilistic and deterministic methodologies can coexist at the NRC?

Technical Reviewer and Project Manager Responses:

• Yes.

• They can co-exist but we need increased guidance.

• Dont they? Yes, they do coexist, but not equally. But then again, the goal should never be to have them coexist equally.

• Yes, it is happening now; really depends on areas; for compliance, design, and, Appendix B, PRA and deterministic cannot be co-existing.

• Yes.

• Yes they do, yes they can, and yes they should.

• Yes.

• They already do. But it is an uneasy relationship. Limitations need to be understood and considered. In a Venn diagram, probabilistic logic and deterministic logic dont overlap much.

• And as it stands now, deterministic is the law, while probabilistic isnt. Sometimes risk is determined with a badly undersized database, which doesnt help.

• The two should communicate and support each other - in that case they can coexist.

• Sensitivity studies should generally be used to help bridge the gap between probabilistic and deterministic studies so you know which you can really trust, or if both are good.

• Yes, but we have to weigh deterministic higher than PRA to not make risk-based decisions.

SES, SL, and BC Responses:

SES

• Yes. They already do.

• Yes. Design basis accident selected and anticipated operational occurrences.

, Attachment 4 Page 16 of 26 SL

• Yes.

BC

• Yes, but there are cultural barriers.

• Yes.

Do you feel that there is a connection between engineering calculations and performing risk analysis? If so, what is it?

Technical Reviewer and Project Manager Responses:

• Yes, one can impact the other.

• Yes, deterministic is single failure, where risk is common cause failure. One can inform the other.

• Yes, there are connections, but they are not close-knit.

• Engineering calculations provide physical representations of components, systems, etc. and use them in decision-making.

• Risk analysis IS an engineering calculation, but it has a different goal. So there is not a direct connection. PRA gives you a range of inputs. And in PRA you solve the engineering equation multiple times, instead of just once. It is iterative. In the end, risk is a subset of engineering calculations.

• Yes. Engineering calculations provide inputs for risk analysis. Risk analysis is essentially an engineering calculation itself.

• Yes. PRA are engineering calculations.

• No.

• Yes, isnt risk analysis an engineering calculation anyways?

• In a flood hydrology example from one participant, the two do connect, but there were troubles in resolving differences between the two.

• One exists (engineering calculations), and one doesnt (risk analysis).

• Yes. Engineering calculations use fudge factors. Operating experience includes safety factors to reduce probability of equipment failures.

• Why use risk to reduce safety margins if risk was used to increase the safety factor?

SES, SL, and BC Responses:

SES

• Risk analysis relies on engineering calculations.

SL

• Yes, engineering calculations are a part of performing a risk-analysis.

BC

• Yes, training/education/expectations.

• Yes, PRA relies on engineering calculations, they are completely intertwined.

• PRA success criteria are developed through the engineering calculations.

, Attachment 4 Page 17 of 26 Can you cite specific examples where you felt the use of risk or risk insights was particularly appropriate or inappropriate?

Technical Reviewer and Project Manager Responses:

• Appropriate - risk informed completion times. (( )) TSTF-505.

• Inappropriate - when using PRA risk only.

• Risk in technical specification extensions was positive.

• A participant recalled an issue at (( )) where there was a red finding some number of years ago that would be a green finding today. But otherwise, no specific examples.

• Appropriate = TSTFs 505 and 425. Inappropriate = Power uprates

• Appropriate - (( )), (( )) GSI-191, (( )) TSTF-505, SDP risk based process, risk informed quality assurance.

• See #1. (( )) was planning to open secondary containment for 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> due to a risk assessment, which we dont feel is very appropriate. There can be circular logic feeding this.

• Also, they keep trying to take Large Break LOCA out based on a risk argument. We dont think thats appropriate either.

• Not appropriate (in general) because it causes safety reviews to be inefficient and difficult.

• Appropriate use - (( )) exigent amendment for EDGs used risk insights.

SES, SL, and BC Responses:

SES

• Appropriate use: filter vent analysis, (( )) EDG AOT extension, TS risk initiatives 4b and 5b

• Inappropriate use: (( )) buried cable issue didnt include risk assessment upfront, it was introduced later.

• In general, focusing on compliance = safety.

• We are not legally empowered to make a distinction between safety significant non-compliance and safety irrelevant noncompliance. A less restrictive change or more restrictive change, if both are compliant vs. backfit.

SL

• Yes, SBO rule, (( )) 50.69 high winds and flooding PRA.

BC

• Appropriate in (( )) EDG outage time LAR; would have been appropriate to use more for Fukushima orders.

• Hydrogen 50.44, maybe.

• (( )) EDG allowed outage time extension - appropriate.

• Westinghouse RCP seals Generation 1 and 2, did not work but allowed to be credited - inappropriate.

• Tornado missile risk evaluator - ultimately may be appropriate.

, Attachment 4 Page 18 of 26 Do you have concerns with expanding the use of risk or risk insights by the agency?

Technical Reviewer and Project Manager Responses:

• Yes. Need for appropriate education, training, and communication.

• PRA is just a model, you can have two analysts and get two different answers. Can you verify the results? How do we know if appropriate models are used? What is the accuracy and uncertainty?

• Guidance is limited for LAR with risk.

• In short, there are two ways to expand the use of risk at the NRC. 1) To expand to perform more of the same types of actions we currently do. This needs more risk people, and/or to retrain existing people. This can cost money and slow us down, because in many ways we end up having to do everything twice, once in deterministic, and once in risk. 2) To expand the scope of what we allow to be justified using risk. This could move us toward being more risk-based, which can be a big concern.

• Yes. Avoid risk-based. There is a broad misunderstanding of PRA and what it can do. Need more education and training.

• Yes. Only as good as PRA. Staff understanding of risk-informed decision-making.

Need a Risk-Informed policy statement.

• Yes. See all of the above reasons.

• We are afraid this will be overly politicized, which could drive the findings.

• We are also afraid that the technical limitations of the technology wont be understood. If this is done right, its okay. But there are lots of misconceptions and misuses. One major goal seems to be to use this to increase efficiency, which on its own is good, but at what cost? Are we focusing on efficiency too much to see the big picture?

• Yes, but dont expand it yet.

• Yes, it could be done, but we are not there yet.

SES, SL, and BC Responses:

SES

• No, as long as done appropriately.

• Risk is already used in the resident inspection program.

SL

• Not if we proceed slowly.

BC

• Yes, because training is needed; may not understand limitations; resources lacking to become risk-informed. Need a comprehensive RG 1.174 - type guidance document for risk informed decision-making that doesnt use delta CDF or LERF explicitly.

• Yes we are implementing risk and risk-informed without a sufficient process.

• 50.69, staff unsure what they are to put in their SE.

• What is the process?

• Pilots are not pilots, the information from pilots is not used.

, Attachment 4 Page 19 of 26

• Danger in seeing risk as a silver bullet. Senior Management may not understand and push staff to misapply risk.

• Risk reviews are more expensive - should be licensees choice to have risk reviews done.

• Rushing to solution.

• Edict to risk inform.

Topic 3: Looking Forward Can you describe what you would consider barriers to incorporating the use of risk or risk insights in your daily work?

Technical Reviewer and Project Manager Responses:

• Major barriers - staff not understanding the different terms and using terms incorrectly. Different comfort levels in applying risk/risk insights.

• Major barriers include subjective decision making, dealing with good or bad precedence, lack of experience in RIDM, etc.

• Major barriers are knowledge and training, a lack of communicated expectations, staff, resources, and licensees coming in fast. The participants agreed that they didnt realize that they were really ALLOWED to incorporate risk into their daily work without going through the PRA gate (in order to use risk, you must go through the risk PM). They have seen residents have backlash for using risk in ways that were not historically done.

• Barriers - ASME Codes, XI Codes, O&M Codes, or other items that are adopted by 50.55a. For these issues, the staff must follow current guidance; PRA does not include everything.

• Lack of definition of risk insights. Consistency of use of terms. There is similar inconsistency and lack of clarity in the industry too.

• There are some hardline mindsets that dont want to use riskskeptics. This is particularly notable in some managers. I dont care what the risk people say.

• There are organizational barriers in that risk staff and deterministic staff do not interact or engage in cross-functional dialogue early on and during the assessment of something. It is typically up to the issue owner (for example, a DORL PM) to piece together separate inputs from DRA and DSS/DE.

• Consider embedding some risk staff into technical divisions and vice-versa.

• Lack of knowledge. Certainty of deterministic approach. Some deterministic thinkers. New process/changes. Staff who are uncomfortable reducing safety-margin. Deterministic folks think they are going to be phased out.

• Deterministic safety limits. They exist and are in the regulations.

• There is no guidance.

• Politics.

• Training.

• Lack of understanding and applicability.

• Ability and skill of the staff.

• PRA and traditional engineering reviewers need to work together better.

• How do you know the impact of risk-informed amendments on plants in the long run?

• Good for prioritization but not safety decisions.

, Attachment 4 Page 20 of 26 SES, SL, and BC Responses:

SES

• We are not legally empowered to use risk in compliance issues.

• Get staff more familiar.

• Overreliance on a small set of experts. Even strong technical staff shy away.

• The NRC definition of safety culture is an obstacle. It is not necessarily commensurate with significance. It assigns the same significance to all safety issues.

SL

• Regulatory guides and standard review plans dont cover. Lack of formal guidance.

Discomfort with level of personal judgment required versus deterministic (e.g.,

requirement is x, licensee did x, therefore, its good).

BC

• Education and resources needed; even then, some people cant change. Need to check to see if prohibited anywhere by a regulation.

• Regulations and guidance are deterministic.

• Deterministic input and deterministic output.

• Standards committees are not risk-informed.

• Knowledge and expertise of staff.

• Only a handful of people have regulatory decision-making background.

• Staff reluctance due to lack of knowledge, experience, guidance, organization, and training.

• Fiefdom, it says risk . . . send to DRA.

• Traditional engineering education lacks risk.

Do you think the current guidance is sufficient for you to feel comfortable using risk or risk insights in your decision-making? Please explain.

Technical Reviewer and Project Manager Responses:

• Help staff understand what tools are available for risk insights other than PRA.

Tools - Saphire (SPAR models).

• No, no, no. There are many gaps in guidance. If we have a particular problem, we may have guidance, but for the use of risk in general, the guidance is lacking.

• What guidance? It is definitely insufficient. Are we even allowed to use risk in our decision-making given current guidance? Not only does new guidance have to be written, but old guidance has to be modified to explicitly point to where it is appropriate.

• More training; current guidance does not cover everything; subjectivity in PRA. It is not exact science; PRA does not include everything.

• Need to address Topic 3 Q1.

• At the highest level, there is no Commission Policy on making risk-informed decisions. Expectations change as the makeup of the Commission changes over time, but Policy Statements stand the test of time and ensure consistent expectations.

, Attachment 4 Page 21 of 26

• There is no overriding agency guidance on making risk-informed decisions. There is guidance for certain specific tasks but nothing holistic.

• Office of Research might be a good resource to help develop agency-wide guidance.

• Dont know of the current guidance. Need for training on guidance. RG 1.174 may be considered more limiting to restrict for licensing actions. Need more integrated guidance in one place.

• Nope. We just started developments. There is no room to use it in our daily work.

The fundamental calculations governing what we review are deterministic. We have to meet them. So until there is explicit guidance that tells me how to meet these requirements using risk, and what those regulations and requirements are, I cant.

• More guidance is needed.

• Licensees need to understand the NRCs expectations.

• Is RG 1.200 sufficient guidance?

SES, SL, and BC Responses:

SES

• There is a myriad of Commission decisions that have evolved over time.

• There is a need for increased awareness and one stop shopping for efficient and convenient reviews.

• There is an absence of a broader body of guidance.

• Licensing lacks the structure that the reactor oversight process has.

SL

• Regulatory guides and standard review plans. Also, how does backfitting work with risk?

BC

• Most of the time, no.

• Some risk guidance in 1.174 is okay, but beyond that, there is limited good guidance, process, and organization.

• Staff needs to be trained.

• Integration of risk training in qualifications, at the beginning of qualifications.

Do you think there is a gap between management expectations and staff use of risk or risk insights? Please explain. If so, what should the NRC do to close that gap?

Technical Reviewer and Project Manager Responses:

• Yes. Staff doesnt know expectations, lack of consistency with management understanding of risk and risk insights.

• Management and staff training. Set and implement consistent expectations. Look at other agencies for training and application of risk and risk insights.

• The feel was the staff and management alignment is mixed across the agency and in some cases they are unsure of the management expectation.

• We dont know what the management expectations even are, especially since it differs from manager to manager. In short, we see no gap, because there cannot be a gap if it is all ambiguous. The answer is consistently go talk to DRA. In order to fix this, management must come to a decision of what they want. Where it is

, Attachment 4 Page 22 of 26 appropriate to use, and how, and where its not. Staff must be trained to become specialists. Boundaries must be set to explicitly advise PMs, technical staff, LAs, etc. to know what is and is not expected of them, and what is allowed, like in current standard review plans and other guidance. Agency-wide policy has be changed to codify this. This policy change will then create a gap that management can work to close.

• Yes, that is why we have this meeting/café; Yes, from EDO announcement (read between the lines); Yes, this is top-down, not bottom-up; change the guidance; clear alignment; clear management expectation.

• Yes. There is a gap between staff and management and manager to manager.

• Not so much at lower levels (BC and division management level). But at higher levels, managers may hear of great ideas during drop-in meetings and send the staff off to do good work. The staff is then pulled in many directions by the inconsistent use of risk-informed initiatives by the industry, which makes it challenging to try to meet executive level expectations.

• Staff recognizes the desire by management to risk inform decision-making but in many cases lacks any guidance on how to do so appropriately.

• More of a need for training for both management and staff. Same lack of understanding for both management and staff.

• Agencywide guidance for everyone to be on the same page.

• Depends how high you go. The expectations come from up high, where there is little understanding on how I do my job down low.

• There is a gap in how much we understand, and what we can do.

• There is also a gap on how much risk we are to use, versus what is practical. We spend a lot of time assessing how safety significant something is.

• To close the gap, see #1.

• Management needs to be familiar with what we do and how we do it in order to not demand us to do things we cant feasibly do. The gap needs to be closed in both directions - we can continue to work at this, but management needs to ask the right questions and understand what we are doing and what we cant do before making requests, and management needs to meet us part way on some of these.

• Yes. This is why we are here (at the café).

• Current management is encouraging more use of risk.

• We need to reconcile with industry as we adapt our policies.

SES, SL, and BC Responses:

SES

• Yes. The Cafés, RIDM action plan, and other things are working to close the gap.

• A voluntary approach for licensees to use PRA vs. a Part 50 PRA requirement is a workaround.

SL

• Yes. Message received to use risk, but no vision on how to do so. Seems to be fashion of the day.

BC

• Yes, education/resources/management direction.

• Yes. RISC committee needs to be aligned with the staff.

, Attachment 4 Page 23 of 26

• RISC sets expectations.

• Senior managers sometimes equate gut feeling of low safety to low risk.

• However, staff cannot write an SE based on senior manager gut feeling.

What additional guidance, tools or training do you think would increase your comfort with using risk or risk insights in your decision-making?

Technical Reviewer and Project Manager Responses:

• Hybrid training between risk insight thinking certification and PRA Qualifications.

• Guidance - integration between guidance and industry initiatives.

• More guidance and more training. Understanding what we are approving in risk applications. When/how to use RIDM. Need increase in communication across agency in risk.

• More training. A course prior to P-105 that is more accessible to people with technical but no PRA training. P-105 was no practical help without appropriate knowledge going into it. Bring P-111 to headquarters, but maybe make it shorter (6 days instead of 10?). All the guidance documents we use will need to be revised to be explicit about addressing risk. And we should have annual training that shows us what appropriate use of risk looks like, just like we do for INFOSEC or COOP.

• Need clear management expectation; need real example about the use of risk insight; need to have better guidance; historically, nuclear power plants are designed and built using engineering, not PRA; CDF/LERF, what do those mean? IF CDF/LERF is still within the bound, what does that mean?

• Need practical, hands-on, exercises.

• Commission Policy is needed to ensure consistent expectations over time and to know what agency guidance should be written to.

• Need a collective assessment of existing tools, training, and guidance to identify gaps and needs.

• There is existing training in iLearn discussing uncertainties in PRA analyses. This training appears to be underutilized.

• There is a tendency to rely on a single part of the organization to solve problemsthis needs to be a widespread collective effort.

• Other staff need training on PRA, beyond risk analysts.

• We operate in silos - need a team integrated approach.

• We need more detailed guidance. Focused for my specific type of technical reviewer. Like revisions to all standard review plans sections that we want to bring risk into, or introduce a new document structured like the standard review plan that describes risk use at that level.

• We need an understanding of the inputs for risk assessment, which can sometimes be based on operating experience, and sometimes based on other reasoning, but is often vague. Where does it come from? Engineers inherently question what they dont understand the full basis of.

• Write PRA guidance with simpler language.

• Develop or use a functional PRA model pilot in industry that works.

• Additional training - No tests!

• Rotational assignments.

• We need a visual example/explanation (we are engineers and many of us learn best this way).

, Attachment 4 Page 24 of 26 SES, SL, and BC Responses:

SES

• There is sufficient guidance. What is needed is a reference guide or pointer.

• SRP chapter 19 intro (re: risk informed reviews) should be added to other sections.

SL

• Practical training and examples of internal collaboration across disciplines.

BC

• More specific expectations on when and how, e.g., for deciding level of effort on an issue [Scribe note - Group often brought up the need to have the ability to not waste time on issues that are not risk significant]; training on SPAR models and where numbers come from [data quality].

• Clearly defined process that says for risk-based evaluation. Standard review plan for risk-based. How staff will evaluate the risk.

• Training and qualification program shave not been updated to teach risk.

• P-105 should be required for all.

• New training curriculum.

What do you think about using risk or risk-insights to refocus or prioritize the level of effort expended on different parts of reviewing a LAR?

Technical Reviewer and Project Manager Responses:

• Could be more efficient, but other review areas need to be incorporated/coordinated.

• Sounds good guidance?

• This is important, and has been shown to be a good use of risk. And its something that management should do. This too needs to be codified so that it is consistently applied and never arbitrary.

• It is fine because of the decreasing budget environment; for (( )), it starts with a small risk, so using risk alone could be an area of concern.

• Not practical. Some already do this (e.g., APLA review of PRA is tailored to intended use).

• It should not take away from the deterministic review.

• In some cases it can take a significant effort to identify that something is very low risk.

• Quality of industry submittals continues to vary greatly.

• Yes. Being sidelined by OGC.

• We have to be careful where it interferes with the 10 CFR and other requirements.

Deterministic requirements need to be met. Period.

• Yes - violent agreement.

SES, SL, and BC Responses:

SES

• It is a wonderful idea.

• The risk informed level of effort process is great.

, Attachment 4 Page 25 of 26

• A cultural change is needed.

SL

• Good thing if done properly. Need to make sure we maintain separation between safety significance and risk significance (e.g., Class 1 piping in 50.69).

BC

• Yes.

• Waste of time.

• We have tried. Risk is good for prioritizing, but we dont know how to use that information within the deterministic framework.

• SEs for risk informed LARs can be changed a lot.

• 50.69 (( )) - what is a technical reviewer to do?

• Risk-triplet.

What changes do you think are needed for you, or your colleagues, to feel comfortable using risk or risk insights in your daily work?

Technical Reviewer and Project Manager Responses:

• More training and clearer expectations.

• There should be a qualification program for us to do. Sort of like the Grow your own PRA but lighter. And all technical staff should become qualified to do it. Do senior risk analysts currently have this? Maybe can leverage some of that.

• Management expectation, more training, more guidance, not speculating.

• Need to address Topic 3, Qs 1, 3, and 4.

• Understanding what risk is and what it is not. Some are uncomfortable with risk (essential to understanding limitations with PRA). What level of risk informed actions can we take if the licensees do not have a good PRA model?

• Standard review plan changes (see above) to include new specific criteria. We need to quantify risk thresholds to determine reasonable assurance and adequate protection, and reduce fuzzy logic.

• We need to change the regulations to explicitly bring in risk. Right now we do it by interpretation, not formally.

• There needs to be transparency for the process and logic used. This necessitates improved communication and openness between deterministic and risk folks.

• We always need to check ourselves to make sure we are not becoming risk-based.

OGC has been clear on this.

• RG 1.174 is a good start, but is one document to generically speak to everything.

We need a lot more in the way of specifics.

• Practical risk modeling

• On-the-job training or practical experience, such as a licensing action - No exams.

SES, SL, and BC Responses:

SES

• There is aversion to less conservative action.

• Management needs to clarify agency decisions.

• Training or practical experience, such as a licensing action - No exams.

, Attachment 4 Page 26 of 26 SL

• Stability of staff and management. Cant keep changing out mangers and their vision along with. Need consistency.

BC

• Risk is not a zone of discomfort.

• Risk is not integrated in the review process.

• No clear path what to do with the risk information.

• What does it mean to scale?

• See Response to #4: More specific expectations on when and how, e.g. for deciding level of effort on an issue [Scribe note - Group often brought up the need to have the ability to not waste time on issues that are not risk significant]; training on SPAR models and where numbers come from [data quality].

Do you foresee additional leveraging of risk in your future work? If so, how?

Technical Reviewer and Project Manager Responses:

• Yes. Discipline specific.

• Yes. As the guidance and training increases the use will increase.

• Yes. There of course will be more, certainly for a while. The writing is already on the wall. Technical reviewers will lead the way, and PMs will follow. We may eventually have to pull back on the use of risk in our work. It will ebb and flow with political tides like anything else in our work.

• Yes, if you address identified issues. The rest of the world is going this direction.

• Yes. Commission direction. Industry is trying to cut costs.

• Yes. But we dont know how.

• Management has told us to do this, so Im sure we will, but no one has told us how.

• Yes. Management and industry are headed toward more risk initiatives.

• Its a long journey - no exams.

SES, SL, and BC Responses:

SES

• Yes SL

• Yes BC

• Yes, training/education/expectations

• Yes, How? Changes to inspection program driven by risk

• Customer (industry) preference

, Attachment 5 Page 1 of 9 ONLINE SURVEY RESULTS NRR RIDM ACTION PLAN TASK 2 Question 1 requested the job function of the respondents. The responses for Questions 2 through 9 are averaged within each job function. For Question 10, the raw responses are in bulleted format. (Minor typing errors were corrected.)

Survey Monkey Q2 through Q9 Q2: Which of the following have you had review experience with (check all that apply):

(This is a plot of the number of different types of reviews with which respondents have had experience) 5 4.5 4

3.5 3

2.5 2

1.5 1

0.5 0

All Technical Project Branch Chief Senior Inspector Reviewer Manager Executive Q3: The current use of risk or risk insights by the agency is appropriate.

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9

, Attachment 5 Page 2 of 9 Q4: The use of risk or risk insights result in an unwarranted decrease in safety margins.

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9 Q5: Risk or risk insights are a form of engineering judgment.

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9

, Attachment 5 Page 3 of 9 Q6: Probabilistic and deterministic methodologies cannot coexist at the NRC.

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9 Q7: I have concerns with expanding the use of risk or risk insights by the agency.

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9

, Attachment 5 Page 4 of 9 Questions Q8 - Q9 were scored on a sliding scale from Not at all to Absolutely.

Q8: Do you think the current guidance is sufficient for you to feel comfortable using risk or risk insights in your decision-making?

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9 Q9: Do you think there is a gap between management expectations and staff use of risk or risk insights?

Inspector Senior Executive Branch Chief Project Manager Technical Reviewer All 1 2 3 4 5 6 7 8 9

, Attachment 5 Page 5 of 9 Survey Monkey Q10 Can you briefly describe what you would consider barriers to incorporating the use of risk or risk insights in your daily work?

• Trusting the numbers that come out of PRA.

• Many.

• My area does not affect CDF or LERF which makes it hard to use the normal PRA practices established.

• Consistent use of risk concepts. Misunderstanding between technical reviewers and management as to the appropriate ways to use risk. This is not due to inadequate guidance, but due to a long history, resulting in a culture that does not understand how risk should be used.

• On paper, the use of risk/risk insights seems like a reasonable approach, but in practice it has been transmogrified into a game of give and take, with decisions being based on politics and risk being manipulated technically to support the politics. Industry is disingenuous with what they advocate as risk insights, hiding behind a facade of realism to foist continued non-conservatism on the NRC. The NRC, with complete collusion from upper management and even some within the technical staff, caves into the industry's (and Commission's) political agenda to give industry what it wants under the guise of risk-informed regulation. As a PRA practitioner for over 40 years, including college, it is disgraceful for me to see how risk assessment went from being an honest endeavor when first pursued, WASH-1400 and the early studies, into a political pawn mishandled by NRC upper management (including Commission) and the industry deceivers in poorly-veiled attempts to avoid making improvements to plant safety.

• Lack of regulatory requirements and oversight of PRA analyses. Regulations, facility design, and facility license requirements are based on deterministic principles. These need to be revised if continued expansion of risk insights is to be pursued.

• NRC employees who only want to use deterministic methods.

• RG 1.177 guidance is long and does not lend itself to a quick-read or deciphering its clear uses.

• No guidance and regulations

• Time to properly do the evaluation and analysis.

• Knowledge and procedural guidance.

• Lack of understanding of the limits of PRA, Politics, Push to be more risk based, making risk information more important in decisions than deterministic requirements

• Attitudes about what it means to be in compliance or to meet deterministic criteria.

• None

• There is no NRC policy/process on how to incorporate the use of risk insights into staff reviews. If it is done it is done because management wanted it, but then it is done at the discretion of the reviewer.

, Attachment 5 Page 6 of 9

• The use of risk is a concern that we are shifting too far away from tried and true deterministic methods. Looking at LARs and the exhaustive questions related to risk that we ask sometimes are excessive and getting further and further away from common sense.

• Because risk-informed decision-making is inherently a subjective process it is extremely difficult to establish or maintain consistent application of the process. More training is needed because many NRC staff do not really understand the process or how they are expected to apply it.

• Poor training, lack of clarity and consistent interpretation and implementation - especially when balancing sometimes conflicting risk perspectives and compliance concerns.

• Appropriate guidance, especially for risk insights.

• Branch chiefs who take a strong position on technical issues.

• Other staff's reluctance to accept risk informed thinking. They are stuck in the 1970s deterministic approach. Management is scared that deterministic reviewers will file DPO.

• Unaligned management.

• People tend to resist change. As new people enter the agency this will become less of a problem.

• The directives and NRC management guidance and formal technical reviewer training to use standard review plans for the review of applications and amendments, when there is no written guidance or training regarding the circumstances under which it may be allowable to deviate from the review guidance.

• Not fully understanding it.

• No guidance, no qualifications, no defined tasks, no standards to measure against, no training, no meaningful intersection between risk (prob

• consequence) and deterministic reasonable assurance.

• Need current requirements to risk-informed such as GDCs, 50.46, 50.55a(h), industry standards, regulatory guides for systems design, testing, and operations.

• Have not had a need so far so I am not sure how it will work.

• Absent focused PRA type work, there does not appear to be sufficient guidance, causing staff to have to figure it out in the course of doing technical reviews. As an example, many staff participated in recent (( )) audits on 50.69 and 4b, with the primary goal of their participation to go to the site and engage the applicant in hope it may become clear what needs to be reviewed and assessed to produce an SE.

• Everyone should be considering risk and or risk insights in their decision making except where the Commission's rules and regulations have established what is required for adequate safety (see Enforcement Policy).

• Process. Currently there is a gap in including risk insights consistently for non RG 1.174 licensing actions. This includes guidance, procedures, and training, all of which are important to repeatedly performing a process. For RG 1.174, current practice is very inconsistent and not efficient, as shown by difficulty in risk analyst review results (differing conclusions, excessive review time, varied RAI focus, independent expert opinion.

• Update guidance, culture and training.

, Attachment 5 Page 7 of 9

• Lack of guidance on how to consider risk insights for non-RG 1.174 submittals; talking about risk in a qualitative sense given that probabilities and frequencies are numerically derived (2nd Q in the risk triplet); lack of guidance explaining the threshold between risk-based and risk-informed; management expectations do not align with guidance and/or staff understanding; management rushing this RIDM effort without researching the information first - task 2 should've been done first, followed by task 3 then task 1.

• Guidance documents I use (SRP, GALL-SLR) do not discuss risk clearly or explain how it should be used when determining acceptability of an application.

• For RG 1.174 type submittals, I think we have a process in place. However, for other uses of risk, we haven't developed guidance for staff to use in making decisions.

• Information and tools that can be used to obtain risk insights are not readily available to non-risk analyst. (2) Management has not addressed the apparent conflict between complying with requirements with little to no safety significance and reviewers having to make regulatory findings against those requirements.

• Hang-ups about compliance on issues with little risk significance.

• Complete understanding of PRA.

• A misperception that a PRA number is a precise answer - there can be tremendous uncertainty and input assumptions can strongly influence the outcome value.

• Further understanding to management and staff that Risk-Insights do not require a Risk-Informed regulatory review.

• Difficulty in weighing the relative importance of risk insights against other factors to be considered in the risk-informed decision-making process. The agency would benefit from making explicit its values, preferences, and tradeoffs among these sometimes competing factors.

• Use of risk has been extensively and justifiably incorporated in many regulatory activities.

Further expansion needs to be made only if use of risk can solve a problem.

• Risk = Probability

• Consequences... much of my work is in determining (1) the consequences, or (2) the probability. However, there are many managers that want to risk inform my work. How can you risk inform the very act of determining the parameters you need to define risk?

• Misunderstanding of risk, PRA, performance-based. And also lack of trust of licensees.

• Lack of understanding of the PRA model and inputs into PRA model to determine accuracy of risk models. Inability to fully understand what level of risk is acceptable to take place of deterministic aspects of components and functions.

• Additional training is needed.

• The lack of guidance for the technical branches presents a challenge for integrated reviews (deterministic and risk-insights together). The reviews that have been completed so far (4b) have taken too many hours that could have been avoided with proper guidance on the integration of risk-insight to technical (deterministic) reviews.

• Lack of shared vision among all staff and management on how risk insights should be used in carrying out our work.

, Attachment 5 Page 8 of 9

• A barrier in staff communications and work flow would result, if management/staff's expectation of its role/application in the review process is not consistent.

• Resistance to learn and use different tools.

• Zealous management support to approve a risk initiative when the actual licensee submittal lacks rigor and substance.

• Lack of experienced staff in risk-informed licensing and lack of training for staff outside of risk branches.

• Some staff have a deep distrust of PRA and actively discount its importance in the decision-making process.

• Deterministic mindset. Lack of true integrated decision-making.

• Primarily that our regulatory regime in my area (SRP Chapters 4 and 15) is based on meeting deterministic safety limits.

• PRA provides licensees almost unlimited opportunities to make slight adjustments in their calculations. This allows licensees to be able to start with the answer they want and create input to match it. I view the flexibility of the PRA to be a huge barrier because it allows the licensees to fix the paper and not the plant.

• Deterministic mind-set.

• Using realistic numbers for input into risk evaluations.

• Treatment of uncertainties when risk numbers are small. Separate issue is perception that you can make the risk numbers come out anyway you want (e.g., how long it took us to determine that the Davis Besse vessel head issue was a red finding).

• Risk insights cannot be applied to all daily work issues.

• Lack of common understanding and acceptance of the spectrum of potential application of risk insights, from use of engineering judgement to application of detailed risk analyses.

• Lack of understanding what risk or risk insights really means.

• Management pushing risk-based outcomes versus performance-based risk informed, prevents proper balance and requires the staff to adopt a risk adverse approach in order to balance the inequity.

• Many regulations and much regulatory guidance is deterministic/prescriptive using bounding scenarios to establish a design envelope. Under existing regulatory paradigm, incorporation of risk and risk insights cannot / should not occur to a great extent in certain areas. For example, long before PRA, events like double-ended guillotine LOCA have all along been considered improbable. Because a quantitative value calculated by a complex analysis can now be arrived at for this improbability, does this mean the long-standing conservative design philosophy for current nuclear power plants should be changed? If it does, will the regulations, GDCs, guidance, etc., be revised to support incorporating risk or risk insights in these reviews? If not, then in what way is it reasonable for staff in these areas to use risk / risk insights?

• There needs to be a more user-friendly way to handshake risk insights with deterministic safety considerations.

, Attachment 5 Page 9 of 9

• Its the way we've always done this.

• Force fitting instead of explaining.

• Lack of familiarity with using risk insights would be a barrier. Clear examples of where risk insights have been used before would help reduce this barrier.

• Cultural inertia, lack of risk understanding and experience.

• Clear guidance.

• A culture barrier. Need top agency direction to empower and expect the use of risk insights into our licensing. Need some guidance to give staff examples. Need to hold all accountable in using risk insights. It should not be an option--it should be part of the agency's fabric/culture.

• Lack of familiarity with PRAs & applications of risk insights.

• I believe question 6 asks about Probabilistic and deterministic methodologies coexisting but many deterministic methods were based on risk insights. Now we are being ask to re-evaluate while making safety decisions on the fly.

• Risk is a complex technical field with specific capabilities and limitations being misused by non-experts. People are essentially giving their gut feelings false legitimacy by calling them risk insights. This is similar to non-nuclear engineers with no understanding of neutronics trying to refuel a reactor using neutronic insights.

• Lack of consistent understanding of risk concepts and RGs.

• As a PM I don't think I am involved with risk-informed decision making or deterministic decision making....technical staff are.

• Barriers include: 1) there is no guidance (i.e., branch technical positions, office instructions, updates to SRP, MD, etc.) to NRC technical review staff to perform their jobs for risk reviews. This most basic necessity puts uneasiness and liability on the staff to perform these risk reviews at risk to themselves and the agency. There are very professional individuals in this agency who take pride in what they do and who have very strong values of ethics. To not provide individuals with the tools they need to perform their work is just not keeping with the NRC values and principles of good regulation. 2) Staff are not willing to perform risk reviews because they see it as further uncertainty and creates the perspective of less margin and defense in depth. 3) Staff is struggling with applying the defense in depth definition and philosophy to conduct their reviews. The definition calls out design and operations during accident conditions. The current reviews are omitting the accident portions. 4) There is a chilling environment today with those individuals performing these reviews and management. Management is directing staff to perform risk reviews consistent with a previously issued LAR. Because staff are not trained to perform risk reviews and there is no guidance available, it is unclear what the technical staff is to do and concerned to do a review without guidance to the staff but rather just a do what I tell you direction.

• Dealing with low risk/low safety significance compliance issues can be challenging. This is where the guidance is weakest.

• None right now.

ENCLOSURE 4 Findings and Recommendations Report Related to NRR RIDM Action Plan Task 4 Page 1 of 11 FINDINGS AND RECOMMENDATIONS REPORT RELATED TO NRR RIDM ACTION PLAN TASK 4

1.0 INTRODUCTION

By revision dated May 4, 2018 (Agencywide Documents Access and Management System (ADAMS) Accession No. ML18116A023), the Office of Nuclear Reactor Regulation (NRR)

Risk-Informed Decision-Making (RIDM) action plan defines the NRR RIDM tasks and projects process, including the use of a systematic or two-phased approach to successfully complete the project. Phase 1 is defined as the evaluation and analysis phase to support the completion of the working groups findings and recommendations reports. This enclosure documents the findings and recommendations associated with NRR RIDM Task 4, Review of NUREG-0800, Standard Review Plan for the Review of Safety Analysis Reports for Nuclear Power Plants (SRP), Branch Technical Position (BTP) 8-8, Onsite Emergency Diesel Generators and Offsite Power Sources AOT [Allowed Outage Times] Extensions (ADAMS Accession No. ML113640138).

In support of meeting these goals, the NRR RIDM Task 4 working groups data collection consisted of identifying and evaluating guidance related to onsite emergency diesel generators (EDGs) and offsite power sources AOT extensions to determine if clarification is needed (e.g.,

acceptability of long-duration AOTs for one-time extensions, and maximum AOTs (i.e., a firm completion time (CT) backstop), even when supported with risk information, to limit the permitted amount of time operation without single failure protection). The working group interviewed stakeholders on their interpretation of BTP 8-8 and related guidance documents. In addition, the working group evaluated the Region IV Palo Verde Nuclear Generating Station (Palo Verde) Differing Professional Opinion (DPO) package (ADAMS Accession No. ML17202G468) and a sample of reviews that were performed in accordance with BTP 8-8.

This report summarizes the insights from the above efforts and captures the working groups recommendations for consideration as the RIDM effort proceeds to Phase 2. The recommendations support Strategy 1 (evaluate and update guidance) from SECY-17-0112, Plans for Increasing Staff Capabilities to Use Risk Information in Decision-Making Activities, dated November 13, 2017 (ADAMS Accession No. ML17270A192).

2.0 BACKGROUND

Differing Professional Opinions (DPO-2017-001 and DPO-2017-002) were received on December 28, 2016, and January 9, 2017, respectively (ADAMS Accession No. ML17202G468). The concerns involved two U.S. Nuclear Regulatory Commission (NRC) staff approved emergency license amendments for Palo Verde, Unit 2 related to a failed EDG.

The DPO Panel was tasked on January 19, 2017, to conduct a review of the issues, maintain the scope within those identified by the original written DPOs, provide monthly status reports, and issue a report. The DPO Panel established the following concise statement of concerns:

1. The license amendment used a two-step process with a shorter duration deterministically-based initial license amendment request (LAR) used to provide time for the licensee to develop a risk-informed longer duration follow-up LAR. The initial LAR was approved by the NRC with full knowledge that the licensee would not complete the necessary repairs within 21 days.

Page 2 of 11

2. The use of the emergency provisions of Title 10 of the Code of Federal Regulations (10 CFR) 50.91(a)(5) for the license amendment review process bypassed public and Commission involvement even though some engagement with the Commission and public was possible.

3. The circumstances at Palo Verde did not meet emergency license amendment criteria of 10 CFR 50.91(a)(5) because the licensee could have avoided the emergency situation.

4. Safety margins were not maintained due to inappropriate compensatory measures being credited, routine maintenance and surveillances on other equipment inappropriately being extended, and operator actions that cannot be accomplished consistent with the accident analyses.

5. The license amendment was approved contrary to internal NRC guidance on the maximum AOT for EDGs. In particular, the LARs were approved contrary to staff guidance (SRP BTP 8-8) that would preclude extensions of EDG AOTs beyond 14 days. Additionally, the NRC has no guidelines for establishing maximum AOT limits.

6. Inappropriate assumptions were made with respect to the safety evaluation (SE) including the elimination of consideration of a single failure on the operable train despite the long allowed CT (62 days vice the initial technical specification (TS) time of 10 days) and exclusion of certain design basis events due to low likelihood (e.g., loss-of-coolant accident (LOCA), loss-of-offsite power (LOOP)).

Due to inconsistent interpretations related to the BTP 8-8, the DPO recommended review of BTP 8-8 to determine if clarification is needed for (a) the use of a 14-day backstop for deterministic evaluations, (b) applicability of the guidance to one-time and permanent TS CT extensions, and (c) defense-in-depth consideration; particularly with respect to mitigating the consequences of a LOOP/LOCA (with a single failure).

Similarly, Recommendation 8 calls for a review of the BTP 8-8 to determine if clarification is needed for (a) acceptability of long-duration CTs for one-time extensions and (b) maximum AOTs (i.e., a firm CT backstop), even when supported with risk information, to limit the amount of time operation without single failure protection is permitted.

3.0 DISCUSSION 3.1 Data Collection 3.1.1 Sample Reviews Performed in Accordance with BTP 8-8 The NRR RIDM Task 4 working group conducted a search for examples of the use of BTP 8-8 in extended CT LARs to establish a basis for additional guidance or revision of BTP 8-8.

Table 1 provides a list of BTP 8-8 licensing actions that the NRR RIDM Task 4 working group identified.

Page 3 of 11 Table 1: BTP 8-8 Licensing Actions ADAMS Plant Licensing Action System(s) Date Accession No.

Essential service water Braidwood, (ESW) pump CT Service Water ML16315A302 11/23/2016 Unit 2 extension for one-time repairs Service water AOTs and deletion of expired Surry Service Water ML17100A253 5/31/2017 temperature service water jumper requirements Shearon Harris, ESW pump replacement Service Water ML16253A059 9/16/2016 Unit 1 AOT ML063350069 12/5/2006 Palo Verde, EDG AOT extension EDG ML16358A676 12/23/2016 Units 1, 2, and 3 ML17004A020 1/4/2017 ML15149A412 Donald C. ML15150A035 EDG AOT extension EDG 6/9/2015 Cook, Unit 1 ML15154B045 ML15156A915 CT extension for ML15342A477 Watts Bar, inoperable diesel EDG ML16071A456 1/13/2017 Units 1 and 2 generator ML17006A271 Extend AOT for a single Fitzpatrick EDG ML010960359 7/30/1999 EDG South Texas Project, Units 1 EDG AOT extension EDG ML021300535 10/31/1996 and 2 3/25/2011 Hope Creek EDG AOT extension EDG ML110610501 8/1/1995 CT extension from 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> to 14 days will be Comanche used while completing the ML14192A338 Transformer 2/24/2015 Peak plant modification to install ML15008A133 alternate startup transformer XST1A ESW piping replacement ML082810643 10/31/2008 Callaway, Unit 1 AOT (extension of the TS ESW and EDG ML090360533 2/24/2009 ESW and EDG CTs)

Page 4 of 11 Table 1: BTP 8-8 Licensing Actions ADAMS Plant Licensing Action System(s) Date Accession No.

Temporarily modify TS to allow the Nuclear Service Water System headers for Catawba, ML030070375 1/7/2003, each unit to be taken out ESW and EDG Units 1 and 2 ML053250121 11/17/2005 of service for up to 14 days each for system upgrades Temporarily modify TS to allow the Nuclear Service Water System headers for McGuire, ML15191A025 each unit to be taken out NSWS 6/30/2015 Units 1 and 2 ML15306A141 of service for up to 14 days each for system upgrades South Texas B Train essential cooling Essential ML050100291 1/10/2005 Project, Unit 1 water pump repairs Cooling Water Temporary TS change to add a required action CT Oconee, Generator Field ML12181A312 6/27/2012 for one Keowee hydro unit Units 1, 2, and 3 Pole ML13357A674 1/8/2014 inoperable for generator field pole rewinds Request for a temporary TS change to add a Oconee, required action CT for one Generator ML16064A020 2/26/2016 Units 1, 2, and 3 Keowee hydro unit Stator inoperable for generator stator replacement Add one time footnote to TS related to diesel Diesel generator fuel oil storage Diablo Canyon, Generator Fuel and transfer system for ML022390574 1/3/1996 Units 1 and 2 Oil Storage and 60-day one-time Transfer System risk-informed EDG AOT extension Control room air Watts Bar, conditioning one-time AOT Main Control ML110190280 2/8/2011 Unit 1 from 30 to 60 days. Not Room Chiller risk-informed.

To extend the CTs for the required actions Diablo Canyon, associated with restoration EDG ML041120264 4/20/2004 Units 1 and 2 of an inoperable diesel generator Page 5 of 11 Table 1: BTP 8-8 Licensing Actions ADAMS Plant Licensing Action System(s) Date Accession No.

Extension of diesel Calvert Cliffs, generator required action EDG ML040980651 4/13/2004 Units 1 and 2 time Donald C.

AOT extension for the Cook, Units 1 EDG ML052720032 9/30/2005 EDGs and 2 Beaver Valley, Increase of EDG AOT EDG ML052720259 9/29/2005 Units 1 and 2 Seabrook, To extend the EDG AOT EDG ML042240471 9/21/2004 Unit 1 Solid State Protection Relax CTs for placing TSTF-418, Rev. System, inoperable instrument 2 for Reactor Trip channels into trip and NUREG-1431, System, ML030920633 4/2/2003 restoration of inoperable Rev. 2 - West. Emergency logic cabinet or master Plants Safety Feature and slave relays Actuation System Plant-specific Vogtle implementation of Any ML15127A669 8/8/2017 risk-informed CTs 3.1.2 Guidance Document Review In addition to reviewing previously issued licensing actions that were reviewed in accordance with BTP 8-8, the NRR RIDM Task 4 working group developed a list of guidance documents from various LARs. Table 2 provides the list of guidance documents reviewed by the NRR RIDM Task 4 working group.

The NRR RIDM Task 4 working group searched all documents for certain words/terms pertinent to extended CTs for uniformity of review. The working group also reviewed the details of pertinent documents to properly understand the terms in context or to establish if different terms were used that were intended to have the same meaning as the searched term. The following specific search words/terms were used: defense-in-depth, extension, risk, CTs, AOTs, backstop, and frontstop.

Table 2: Guidance Documents Reviewed NUREG-0800 SRP BTP 8 Onsite (Emergency Diesel Generators) and Offsite Power Sources AOT Extensions, Revision Initial, February 2012 NUREG-0800 SRP Section 16.1 - Risk-Informed Decision-Making: Technical Specifications, Revision 1, March 2007 NUREG-0800 SRP Chapter 19.0 - Probabilistic Risk Assessment and Severe Accident Evaluation for New Reactors, Revision 3, December 2015 Page 6 of 11 Table 2: Guidance Documents Reviewed NUREG-0800 SRP Section 9.2.1 - Station Service Water System, Revision 5, March 2007 NUREG-1431 - Standard Technical Specifications - Westinghouse Plants, Volume 1, Revision 4, April 2012 Regulatory Guide (RG) 1.9 - Application and Testing of Safety-Related Diesel Generators in Nuclear Power Plants, Revision 4, March 2007 RG 1.93 - Availability of Electric Power Sources, Revision 1, March 2012 RG 1.155 - Station Blackout, August 1988 RG 1.160 - Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, Revision 3, May 2012 RG 1.174 - An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plants Specific Changes to the Licensing Basis, Revision 3, January 2018 RG 1.177 - An Approach for Plant-Specific, Risk-Informed Decision-Making: Technical Specifications, Revision 1, May 2011 RG 1.182 - Assessing and Managing Risk Before Maintenance Activities at Nuclear Power Plants, May 2000 RG 1.187 - Guidance for Implementation of 10 CFR 50.59, Changes, Tests, and Experiments, November 2000 NUMARC 93 Industry Guideline for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, Revision 4A, April 2011 NUMARC 8700 - Guide lines and Technical Bases for NUMARC Initiatives Addressing Station Blackout at Light Water Reactors, November 1987 Nuclear Energy Institute (NEI) 96-07, Guidelines for 10 CFR 50.59 Evaluations, Revision 1, February 2000 NEI 06 Risk-Informed Technical Specifications Initiative 4b, Revision 0, November 2006 Technical Specifications Task Force (TSTF) Traveler TSTF-505 - Provide Risk-Informed Extended Completion Times - RITSTF Initiative 4b, Revision 1, June 2011 NUREG/CR-6141 - Handbook of Methods for Risk-Based Analyses of Technical Specifications, December 1994 Generic Letter 84 Proposed Staff Actions to Improve and Maintain Diesel Generator Reliability, July 2, 1984 3.1.3 Interview Questions The NRR RIDM Task 4 working group also sent a questionnaire to select staff members and management involved in the original development of BTP 8-8 and reviews of AOT/CT extension requests to further establish a need for additional guidance or revision to BTP 8-8. Table 3 provides a list of the 15 questions that were included in the questionnaire.

Table 3: Interview Questions 1 What do you believe is the current strict purpose of BTP 8-8 (deterministic? risk-informed, permanent changes and/or one-time changes)?

2 Is it sufficient for the complete spectrum of EDG and offsite power AOT extension LARs we are receiving?

Page 7 of 11 Table 3: Interview Questions 3 Does it [BTP 8-8] need to be revised?

a. Why?

b. Why not?

4 What revisions have been proposed in the past?

5 Were the proposed revisions successful?

a. If not, why not?

6 Should BTP 8-8 guidance be evaluated for clarification of the use of a 14 day backstop for deterministic evaluations?

7 Should BTP 8-8 guidance apply to both one-time and permanent CT extensions?

8 Is it acceptable to apply BTP 8-8 guidance for CTs longer than 14 days?

a. Acceptability of CTs longer than 14 days for one-time extensions?

b. Firm CT backstop even when supported with risk information?

c. Do we need a backstop?

9 Should one-time CT extensions have separate guidance?

10 Should any CT longer than 14 days be supported by risk insights or risk information?

11 Does NRC staffs understanding of a risk-informed (i.e. RG 1.200 compliant) submittal need to be changed?

12 How can we review AOT applications with risk insights that arent RG 1.200 compliant?

13 Is there any other guidance used for AOTs?

14 What do you think is the largest hurdle in the way of a successful revision of BTP 8-8?

15 Is there a difference between AOTs discussed in BTP 8-8 and other AOTs (e.g., with respect to use of risk insights, establishing backstops, handling one-time extensions)?

3.2 Data Analysis and Findings As stated above, the NRR RIDM Task 4 consisted of reviewing BTP 8-8. The main focus of the RIDM Task 4 working group is to address inconsistent interpretations of BTP 8-8. To that effect, the working group formulated a strategy that included two goals:

• Short-term: issue an expectation memorandum on how to use the range of applicable existing guidance for AOT extension requests. This memorandum was issued on May 31, 2018 (ADAMS Accession No. ML18052A573).

• Long-term: provide a redline strikeout version of recommended changes to BTP 8-8 that will eventually be incorporated into the guidance document.

Page 8 of 11 To accomplish the tasks goals, first, the working group reviewed the guidance in BTP 8-8 against current agency positions and guidance to determine if clarification is needed for:

a. Use of a 14-day backstop for deterministic evaluations

b. Applicability of the guidance to one-time and permanent CT extensions

c. Defense-in-depth consideration, particularly with respect to mitigating the consequences of a LOOP/LOCA with a single failure.

Second, the working group reviewed current agency positions and guidance to determine if clarifications are needed for:

a. Acceptability of long-duration CTs for one-time extensions

b. Maximum CTs (i.e., a firm CT backstop), even when supported with risk information, to limit the amount of time operation without single failure protection is permitted.

Overall the working group determined through staff interviews and based on analyses of past licensing actions issued for extending CTs that the staff had applied guidance consistent with the objectives of the Commissions policy for increasing the use of probabilistic risk assessment (PRA) technology, Use of Probabilistic Risk Assessment Methods in Nuclear Regulatory Activities: Final Policy Statement, which was published in the Federal Register on August 16, 1995 (60 FR 42622). However, the working group also determined that the guidance contained in BTP 8-8 is not adequate regarding the extent to which PRA could be used in granting extended CTs for DGs.

The working group interviewed several staff members who have experience in using BTP 8.8 or management who are familiar with BTP 8.8. The results of the interviews are summarized as follows:

• The BTP 8-8 guidance needs revision to clarify the use and application of risk information.

• The BTP 8-8 guidance provides an adequate backstop for deterministic reviews.

• It is acceptable to apply guidance for CTs longer than 14 days as long as it is adequately risk-informed.

• A maximum backstop is needed, even when supported with risk information, which would be a new agency position.

For BTP 8-8 specifically, the NRR RIDM Task 4 working group found that in general the NRC staff are applying the guidance consistently although the guidance in BTP 8-8 is somewhat inconsistent within itself. For example, regulatory guides for application of current guidance for use of PRA in decision-making and TS changes are included in the background of the document but are not applied in implementation within the document. The BTP 8-8 guidance specifically discusses the defense-in-depth aspects of offsite and onsite power sources from a deterministic perspective and states that risk evaluation is separately performed by the PRA licensing branch. Thus, there is no specific guidance for the appropriate integration of deterministic and PRA input for risk-informed CT extensions. In addition, the working group Page 9 of 11 determined from the interview responses that the 14-day backstop was determined to be needed for an adequate time to perform the worst-case maintenance on the largest diesel generators. Extension of diesel generator CTs for greater than 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> was recommended (Recommendation 7) in the attachment to Enclosure 3 of Generic Letter 84-15. The working group did not identify a strong technical basis for the use of a 14-day backstop for deterministic evaluations. The BTP 8-8 guidance takes the deterministic defense-in-depth approach that an alternate diesel generator be available as a backup to the DG that is out-of-service to allow a CT to be extended beyond 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. The working group determined that this may be inconsistent with current agency positions RG 1.174 and RG 1.177 when a review is risk-informed.

The Task 4 working group analyzed the data collected for inconsistencies between current agency positions, and the way the staff is interpreting or implementing the guidance. The working group found that, although current agency guidance does not provide clear positions on acceptability of long-duration CTs for one-time extensions or maximum CTs:

• 30 days is the longest CT used in the current Standard Technical Specifications in NUREG-1430, NUREG-1431, NUREG-1432, NUREG-1433, and NUREG-1434; and

• Changes made for a time longer than 90 days may be considered permanent changes.

4.0 RECOMMENDATIONS Recommendation 1 Provide training to staff on the expectation memorandum, Current Expectations for Using Existing Guidance for Reviewing License Amendment Requests for Diesel Generator Technical Specification Completion Time Extensions.

BASIS:

The training will realign NRR staff on the proper use of agency guidance.

Recommendation 2 Revise BTP 8-8, Onsite (Emergency Diesel Generators) and Offsite Power Sources AOT Extensions (see attachment for the NRR RIDM Task 4 redline strikeout of proposed changes).

BASIS:

To further clarify how to conduct integrated reviews by considering both deterministic and risk-informed insights, thereby assuring adherence to the Commission PRA policy statement.

Recommendation 3 Revise SRP Section 16.1, Risk-Informed Decision-making: Technical Specifications, to include key points of the expectation memorandum, and to reference the current version of RG 1.177, An Approach for Plant-Specific, Risk-Informed Decision-making: Technical Specifications.

Page 10 of 11 BASIS:

To ensure consistency with related guidance documents.

Recommendation 4 Revise RG 1.177, An Approach for Plant-Specific, Risk-Informed Decision-making: Technical Specifications.

BASIS:

The revisions will address the following:

a. Long-duration CTs for one-time extensions

b. Backstop CTs Recommendation 5 All technical reviewers should have risk-informed application training, training on RG 1.200, RG 1.174, and RG 1.177, and associated refresher training on integrated reviews every 24 months.

BASIS:

Staff will have a better understanding on risk-informed decision-making with respect to licensing actions.

Recommendation 6 All PRA licensing staff should have training on how to review/disposition LARs with risk insights.

BASIS:

The training will ensure consistency between reviews.

Page 11 of 11 The above recommendations will enhance risk in our decision-making activities as follows:

• Unifies NRR understanding of appropriate use of risk information in licensing applications

• Increase the staffs understanding of risk and risk tools

• Increase the staffs ability to understand and effectively apply risk

• Improves the technical basis for regulatory activities

• Increases efficiency, effectiveness, and consistency in licensing activities

Attachment:

Recommended Draft Changes -- Redline Strikeout of BTP 8-8 (ADAMS Accession No. ML18169A214)