ML19007A355

Enclosure 3 - Integrated Review Team Process

ML19007A355
Person / Time
Issue date:	01/30/2019
From:	Michael Orenak Plant Licensing Branch II
To:
Orenak, 415-3229
References
CAC A11008
Download: ML19007A355 (27)

Text

ENCLOSURE 3 Revised Integrated Review Team Process Based on Lessons Learned from Trial Period Page 1 of 23 Revised Integrated Review Team Process Based on Lessons Learned from Trial Period 1.0 INTEGRATED REVIEW TEAM PROCESS 1.1 Integrated Review Team Process Flow Chart The Integrated Review Team (IRT) process was discussed in the June 26, 2018, Risk-Informed Decision-Making (RIDM) Phase 1 report (Agencywide Document Access and Management System (ADAMS) Accession No. ML18169A205). The Team A, Phase 2, activities revised the IRT process flow chart and made minor changes to the IRT process to incorporate lessons learned from the trial period implementing the IRT.

Box 1: Upon receipt of a license amendment request (LAR), the Project Manager (PM) will review the request (i.e., an application or submittal) to identify the level of probabilistic risk assessment (PRA) related information in the submittal. The checklists discussed in Sections 5.0 through 8.0 of this document provide guidance to facilitate the implementation of the IRT process.

Box 2: Based on the level of PRA information, the PM will use the PM checklist (see Section 5.0 of this document) to categorize the submittal as being Type 1, 2, or 3.

Page 2 of 23 Box 3: There is little or no risk information and no quantitative risk (e.g., core damage frequency (CDF)) discussed in the licensee submittal. The technical reviewers (TRs) and PM determine if an IRT is needed through the use of the IRT Checklist (see Section 6.0 of this document). A broad understanding of the concepts of the Office of Nuclear Reactor Regulation (NRR) Division of Risk Assessment (DRA) Checklist may help in the determination (see Section 7.0 of this document).

Box 3a: The action in this box depends on the input and also how the IRT decides to use the input. This is described as technical reviewer led - since licensees have not provided information in accordance with Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, and therefore this will result in a deterministic regulatory decision, albeit possibly with some supporting risk-insight information.

Box 4: The submittal is risk-informed in accordance with RG 1.174. DRA leads the review.

Boxes 5 through 8 - Overview In order to integrate the reviews of submittals that have varying levels of risk information, SE template language is discussed below in association with Boxes 5 through 8 of the IRT Process Flowchart.

The staff should use consistent and commonly understood language when integrating risk information into regulatory decision-making as defined in NUREG-2122, Glossary of Risk-Related Terms in Support of Risk-Informed Decisionmaking, November 2013 (ADAMS Accession No. ML13311A353). To consistently communicate the use of risk in regulatory decisionmaking, three categories of SE language will be pre-described for use, as applicable to the determination for risk-insight language appropriate for the licensing action.

Primary consideration for properly describing the use of risk information in regulatory decisionmaking should be given to ensure that the term probabilistic risk assessment (PRA) is only used when there is a PRA calculation used as part of the basis or to support the decision.

Secondly, PRA can only be part of the basis for a decision when the PRA is acceptable for the application; that is, the PRA acceptability is addressed in accordance with RG 1.174. If PRA acceptability is not addressed in accordance with RG 1.174, then the PRA results or risk insights could be used only to verify the deterministic conclusion. Lastly, if a PRA or risk analysis was not used, even if there is quantitative information or calculations, the term probabilistic risk assessment or PRA should not be used in the staffs SE.

Box 5: This is a traditional engineering decision. If there is risk information in the licensees submittal, the SE needs to clearly state that the risk information was excluded in the staff review. An excerpt from an August 2018 Watts Bar Nuclear Plant SE is quoted below and provides a good example for this type of decision (ADAMS Accession No. ML18204A252):

The licensee stated that the proposed extension offered the lowest risk among the four options available. No numerical support for the conclusion was submitted. Because the LAR is not a risk-informed application, the staff did not review the method used by the licensee to derive risk insights. The evaluation performed was not used to support a risk-informed decision by the staff.

However, the staff considered the insights provided, judged that they were Page 3 of 23 reasonable, and determined that they did not challenge the conclusion that the proposed change maintains defense-in-depth.

Box 6: Qualitative information of risk significance or supporting information can be used to support regulatory decisions. Examples include probabilities of failures of equipment, the frequency of initiating events, or other pertinent information. This also includes consideration of supporting PRA information, such as CDF or large early release frequency (LERF) submitted not in accordance with RG 1.174. Care is needed to avoid calling this supporting information PRA. This information does not have to be used within a risk analysis to help support a deterministic evaluation. Perhaps a better name for these would be risk inputs, or the factors that could be used in a risk analysis.

The language that would be used for this supporting information could be called, likelihood information, probabilistic analyses, or other quasi-qualitative terms. However, this information should not be called PRA results or input. Care should be taken to ensure that users of this probabilistic information do not mistakenly assume probabilistic information is PRA results or insights just because such information has the potential to be used in PRA or risk evaluation that is PRA.

The DRA risk analysts may assist the staff in crafting an SE that uses probabilistic information in addition to the typical deterministic information to support a regulatory decision. When discussing probabilistic information or risk inputs, technical staff have significant flexibility within the templates provided in NRR Office Instructions such as LIC-101, License Amendment Review Procedures, to describe the technical basis for their regulatory decision. The limitation is that when describing probabilistic or risk input, the term PRA or use of quantitative CDF or LERF should not be used. Use of these terms would indicate PRA as a basis of the decision or used to support the decision; therefore, it would be in Box 8.

Box 7: PRA insights or results can help support or verify regulatory decisions made using traditional engineering evaluations is a primary outcome of the graded approach in RIDM. The output of PRAs often includes numbers, such as CDF or LERF. Probabilistic risk assessments are also used to gain insights about a facilitys response to initiating events and accident progression, including the expected interactions among facility systems, structures, and components (SSCs), and between the facility and its operating staff. A risk assessment is a systematic method for addressing these questions as they relate to understanding issues, such as important hazards and initiators, important accident sequences and their associated SSC failures and human errors, system interactions, vulnerable plant areas, likely outcomes, sensitivities, and areas of uncertainty. Risk insights can be obtained via both quantitative and qualitative investigations. Quantitative risk results from PRA calculations are typically the most useful and complete characterization of risk, but they are generally supplemented by qualitative risk insights and traditional engineering analysis. Qualitative risk insights include generic results; that is, results that have been learned from numerous PRAs that have been performed in the past, and from operational experience, and that are applicable to a group of similar plants.

These qualitative or quantitative risk insights also need not be necessarily developed by PRA models that meet the guidance of RG 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities, and may be developed using standardized plant analysis risk (SPAR) models or other PRA tools.

The language used in SEs would describe the information from a PRA but would be clear that the information was not used as the basis in U.S. Nuclear Regulatory Commissions (NRC)

RIDM framework. For example, verification might be described as, PRA information was Page 4 of 23 determined to be consistent with this conclusion, or Nothing in the PRA was inconsistent with this deterministic evaluation.

Using PRA tools such as SPAR models, human reliability analysis (HRA) worksheets, or Plant Risk Information e-Book (PRIB) handbooks, which do not generally meet the acceptability needed for RG 1.174 applications, can also be described as PRA insights or information used to verify a regulatory decision, as described above.

When describing the use of PRA to support regulatory decisions, language such as the following can be used as a template in the following manner:

In the subject application, the licensee stated that the basis for the proposed []

is based upon []. The licensee submitted a risk evaluation that was not consistent with risk-informed decision-making.

Because this is not a risk-informed application, the PRA models used to derive risk insights in [] were not reviewed by the staff to determine their technical acceptability as a basis to support this application. As a result, the staff did not rely on the numerical results provided by the licensee. However, the staff considered the licensee-provided risk insights to aid in the deterministic review of the proposed change. The staff also performed an independent assessment using [] to evaluate the risk contribution. The licensee-provided risk insights and the risk insights developed by staff both supported the engineering conclusions []. The currently available risk insights and results did not challenge the engineering conclusions that the proposed change maintains defense-in-depth.

Box 8: Using the results of a PRA as a part of the basis of a regulatory decision involves using the five principles of the RIDM framework from RG 1.174. These features are: (1) the change meets the regulations, (2) is consistent with defense-in-depth, (3) maintains safety margins, (4) results in small increase in risk using PRA information, and (5) is monitored. For PRA information to be used in this RIDM framework, the PRA should be acceptable to support the application as described in RG 1.174.

The language used to describe PRA results as of the elements of a regulatory decision are documented in RG 1.174 and other RIDM regulatory guidance. The language should focus on PRA. Language such as risk insights, or PRA insights, may be used, consistent with the guidance in RG 1.174 or RG 1.200. Therefore, existing precedents and guidance from RG 1.174 apply.

1.2 IRT Process for Licensing Actions The following are general licensing actions:

• routine plant-specific licensing actions that are requested via Type 1, 2, or 3 submittals

• emergent licensing actions (i.e., emergency and exigent amendments and verbal relief requests)

The IRT process does not constitute a review methodology for integrating risk and traditional insights to complete a technical evaluation; rather, it describes a framework for forming the teams that will need to integrate the review products from technical reviewers and risk analysts.

Page 5 of 23 The use of the terms technical branches (TBs), technical reviewers, or traditional engineering refers to staff that are not qualified risk analysts or reliability experts. The use of the terms risk analyst or PRA analysts refers to qualified PRA analysts or reliability experts, who would typically be assigned to PRA Licensing Branch A or B (APLA or APLB) in DRA of NRR, or their equivalent after the NRR and Office of New Reactors (NRO) merger.

Integrated review team (IRT) typically refers to a review team consisting of at least one PM, a technical reviewer, and a risk analyst, whereby the team members are responsible for developing requests for additional information (RAIs), audit plans, and the SE. However, an IRT does not have to include a risk analyst if the team finds the IRT concept is beneficial to enhance the review integration of various technical disciplines and the consolidation of SE and RAI input.

The IRT concept encourages the reviewers from many technical disciplines, including a risk analyst (if any), to work together as a team. The communication within a team and developing joint consolidated products enhance the efficiency of the licensing process. If a branch has a concurrence-only role, the inclusion of this branch in the IRT may not be necessary. An IRT can be created to review Type 1, 2, or 3 submittals. Ultimately, the IRT generates a single SE, with a primary technical reviewers branch responsible for the body of the draft SE, and other technical reviewers providing written additions to the draft SE (see Section 8.0 of this document).

Although the IRT process described below consists of specific team meetings, the IRT has the flexibility in determining whether certain meetings are needed on a case-by-case basis.

2.0 INTEGRATED REVIEW TEAM ROLES AND RESPONSIBILITIES The following describes roles, responsibilities, knowledge, skills, and abilities of the IRT team.

As a reminder, the staffs performance plans have been revised to incorporate RIDM and the IRT process provides guidance in implementing RIDM. The staff should charge time to the specific CAC/EPID of the license amendment when participating in an IRT for that review.

2.1 Division of Operating Reactor Licensing Project Manager The Division of Operating Reactor Licensing (DORL) PM will:

• Facilitate team formation, meetings, written products, and meeting milestones.

• Have a leadership role in consolidating products (e.g., RAIs and SEs) by ensuring that the products are consistent with office instructions and training courses involving the fundamentals of reactor licensing.

• Ensure that the consolidated SE conforms to the guidance in LIC-101, Appendix B, Attachment 2 or LIC-102, Relief Request Reviews.

• Ascertain the teams abilities with using available information technology (IT) tools, such as ADAMS, SharePoint, Office 365, Skype, and Go-To-Meetings to determine which platform would best suit the team for consolidated products and accommodating remote workers.

• Be familiar with various IT tools, such as ADAMS, SharePoint, Office 365, Skype, and Go-To-Meetings.

Page 6 of 23

• Be familiar with using risk insights in licensing reviews, various risk-informed licensing initiatives (e.g., Technical Specification Task Force (TSTF) Travelers and Title 10 of the Code of Federal Regulations (10 CFR) Section 50.69 LARs), and the NRC Regulatory Guides for risk-informed reviews (e.g., RG 1.174, RG 1.200, etc.) to better inform team formation and review needs. The PMs will facilitate issue resolution and escalation to management.

• Be familiar with the Type 1, 2, and 3 definitions.

2.2 Technical Reviewers Technical reviewers will:

• Be familiar with the Type 1, 2, and 3 definitions.

• Be familiar with the application of risk in licensing reviews to assist PMs with team formation, integrate risk and traditional engineering reviews, and enable early identification of the need to involve risk analysts with the reviews.

• Understand the various SE formats and guidance in LIC-101 and LIC-102.

• Serve a technical coordination role in the integration of risk and traditional engineering insights.

• Be familiar with various IT tools, such as ADAMS, SharePoint, Office 365, Skype, and Go-To-Meetings.

2.3 Risk Analysts Risk analysts will:

• Be familiar with the Type 1, 2, and 3 definitions.

• Assist PMs with team formation and the technical reviewers with integrating the risk and traditional engineering reviews.

• Understand the various SE formats and guidance in LIC-101 and LIC-102 to assist with the consolidated SE format.

• Serve a technical coordination role in the integration of PRA and traditional engineering insights.

• Be familiar with various IT tools, such as ADAMS, SharePoint, Office 365, Skype, and Go-to-Meetings.

• Be able to identify, with PMs assistance, which technical branch or discipline will be needed to review RG 1.174 applications.

Page 7 of 23 2.4 Management NRR management will:

• Maintain awareness of industry initiatives that will result in a large volume of similar risk-informed applications and allot the resources needed to efficiently address those applications.

• Ensure that the PMs, technical reviewers, and risk analysts are aware of such situations and will communicate with management peers through workload management meetings.

• Ensure the staff is sufficiently trained in the integrated review approach.

• Assist, as needed, in resolving differing views that may arise during the integration of risk and traditional engineering insights.

Branch Chiefs will:

• Review and concur on their staff input to the consolidated SE and RAIs to ensure the adequacy of the review methodology, approach and result documentation.

• Review and concur on the final consolidated SE and RAIs as requested by the PMs.

• Adjust resources or redirect their staff to support projects as needed.

3.0 ROUTINE PLANT-SPECIFIC LICENSING ACTIONS The following process description is for establishing IRTs for routine plant-specific licensing actions submitted as Type 1, 2, or 3 applications from licensees. The process description covers submittal type designation, staffing assignments, determination of whether to use an IRT, and consolidated RAI and SE development. The processes described in established and applicable NRR office instructions apply unless they are specifically addressed by the following description. The term routine means that the application is not an emergency or exigent (i.e.,

emergent) request or a verbal relief request.

3.1 Submittal Type Determination The following process for submittal type determination applies when the PM creates the project in RRPS 1 and makes initial branch assignments.

A. Within the typical timeliness for adding a project to RRPS when an application is received, the PM will review the application to determine whether it is a Type 1, 2, or 3 application based on the level of quantitative PRA information in the application (e.g.,

CDFs, LERFs, initiating event frequencies, references to PRAs, etc.). For large applications, doing a Find for PRA terms or asking the licensee during routine discussions can be helpful. If the PM is uncertain how to determine the type of application, the PM can consult with DRA. Until RRPS is modified to designate a Project as Type 1, 2 or 3, the PM will add Type 1, Type 2, or Type 3, as applicable, in 1

NRRs Reactor Program System - Licensing/Workload Management software.

Page 8 of 23 RRPS, in the Other Considerations field (which is within the Project Details tab and then the Project Attributes section) when opening a project. For Type 3 applications, the PM should consult with DRA, as needed, to determine which branches have reviewed similar applications in the past. For the purposes of implementing this step, Type 1, 2, and 3 applications are defined as follows:

Type 1: Applications contain little or no risk/PRA information.

Type 2: Applications contain PRA information, but are not submitted as risk-informed applications in accordance with RG 1.174.

Type 3: Applications submitted as risk-informed applications in accordance with RG 1.174.

B. The PM will assign branches in RRPS as follows:

• Routine Type 1: Assign technical branches, but not the DRA branches. The need for risk analysts will be determined after meeting with the technical reviewer(s) or after the PM determines that it needs assistance with the no significant hazards consideration (NSHC) determination regarding frequency of accidents. The PM may assign DRA at this stage if it is already known that the technical staff will want a consideration of risk insights (e.g., if this was discussed after a pre-submittal meeting).

• Routine Type 2 or Emergent: Assign both DRA and technical branches.

• Routine Type 3: Assign DRA and technical branches. If technical branch assistance is not needed, the reviewer can be removed later from the project at a later time.

Assigning the technical branches at the start of the project will provide the technical branches an early opportunity to raise concerns with risk-informed applications.

3.2 Integrated Review Team Determination The following process for team formation and project scoping is applicable from the time the initial set of reviewers are assigned in RRPS to when the acceptance review is issued. It includes explicit instructions to hold team meetings prior to issuance of the acceptance review to verify team composition, scopes of review, and any issues, regardless of the hours estimate, and the possibility that more than 25 days may be needed to do an acceptance review for an IRT approach. If more than 25 days are required for an acceptance review, request division management approval for a longer review period as discussed in LIC-109, Acceptance Review Procedures.

A. The PM will hold an initial staffing meeting with the assigned reviewers, regardless of the estimated hours, within 10 working days of the application being added to ADAMS to determine if additional branches should be added to or consulted about the review. The PM and assigned reviewers should consult the checklists in Sections 5.0 through 8.0 of this document. Assigned reviewers should use RRPS to identify other branches needed and start recommending scopes of review. However, the PM will confirm with the reviewers via a meeting to understand the bases for their recommendations. For Type 1 applications, the staffing meeting will be used to determine if risk analysts or any other technical branches should be assigned to the review using the checklists in Sections 6.0 and 7.0 of this document. If risk analysts will not be used (i.e., the review team agrees that it does not want risk insights from DRA), then staff should follow the acceptance Page 9 of 23 review processes established via the applicable NRR office instructions. For Type 2 applications, the staffing meeting will be used to confirm that assigned branches will remain on the review and whether to assign other branches using the checklists for assistance. For Type 3 applications, the staffing meeting will be used to confirm whether the technical branches will remain on the review and whether to assign other branches.

If technical staff will not be assigned to Type 3 reviews, then staff should follow the acceptance review processes in LIC-109.

B. After the staffing meeting, the PM will use RRPS to assign other branches to the review, as needed. It is not expected at this stage that the extent of each reviewers scope is known; therefore, it is possible that at a later time, some reviewers may still be added, denoted as having a concurrence-only scope, or no longer be needed. However, the meeting provides assurance that the reviewers have read the application and have considered whether risk insights will be used.

C. After the review team is staffed and assigned in RRPS, the PM will hold a scoping meeting, regardless of the estimated hours, within 15 working days of the application being added to ADAMS. The following will occur during the scoping meeting:

• The PM will discuss the application, including all of the proposed changes that need evaluation; plant-specific issues or background information (e.g., licensing and design basis, stakeholder interests, time-charging expectations, or generic RAI and SE expectations). The PM will also identify if there are other ongoing reviews for applications that may overlap with the current application (e.g., multiple risk-informed applications from the same plant that use the same PRA information).

• Team members will discuss their expertise as it pertains to the review.

• The PM will introduce any scoping recommendations with input from the technical reviewers and risk analysts. The PM should also identify whether it will need the teams help with the NSHC determination (e.g., whether the change could affect the frequency of accidents).

• For Type 1 and 2 applications, the team will use the checklists in Sections 6.0 and 7.0 of this document to decide if DRA should be on the review. The DRA division must be on the review if the staff intends to document risk insights in the SE. DRA has the responsibility to determine if risk insights are appropriate or beneficial to the SE. Consideration for cost/benefit from a staff resource, as well as licensee fee billing, will be the general responsibility of the PM (with DORL maintaining cost tracking for hours and contract expenses through the fee validation process, as well as informing the licensee the added risk review).

• For each change identified in the submittal, the review team will confirm each change proposed by the licensee, which team member is reviewing each of the change(s),

and the scope of each team members review. The PM will ensure that the entire application (i.e., every proposed change) has an assigned reviewer. The team members will identify what information, evaluations, and conclusions the reviewers will need from each other to help inform the hours estimates. This action may take more than one meeting because it requires the staff to understand how risk and traditional engineering insights can be integrated.

Page 10 of 23

• With assistance from the review team, the PM will document the proposed changes and reviewer assignments and note the scoping assignments in an RRPS comment.

At this time, the PM will denote in RRPS whether an IRT approach will be used.

Until RRPS is modified to designate that an IRT is being used, the PM will type Integrated Review Team if applicable, in RRPS, in the Other Considerations field (which is within the Project Details tab and then the Project Attributes section) after the Type designation (e.g., Type 2, Integrated Review Team). If it is determined that both risk analysts and technical reviewers are providing input to the review, or if the risk analysts are assisting with documenting qualitative risk insights, then an IRT approach will be used. If either the risk analysts or technical reviewers are on concurrence-only, then an IRT approach is not necessary; however, the staff can and should consult with each other throughout the review to ensure any issues are identified and resolved early in the process.

D. Within 20 working days2 after the application is added to ADAMS, the PM will hold an acceptance review meeting. This meeting will cover the following topics:

• Hours estimates and the need for a metric exclusion memorandum. With the IRT approach, the average hours per review may increase; therefore, the hours estimates per reviewer in the expectations memorandum (ADAMS Accession No. ML16202A029) may no longer be accurate, and the total hours estimate for the review may no longer be a sufficient gauge to determine whether the review may be complex.

• Acceptance review issues (e.g., information insufficiencies).

• The IT tools (e.g., e-mail, ADAMS, SharePoint, or Office 365) to be used to collectively develop the consolidated RAIs and SE. The team will also determine the need for Skype or Go-To-Meetings for teleworking staff.

E. Prior to issuing the acceptance review results to the licensee, the PM may hold additional team meetings or discussions to finalize any acceptance review issues, and to confirm hour estimates, review scopes, and milestone dates. Reviews with an IRT might exceed the 25-day metric for acceptance reviews because the review team will need to understand how it plans to use risk insights or how technical staff will contribute to Type 3 reviews. In these instances, the staff should follow the guidance in LIC-109 to request division management approval.

4.0 EMERGENT LICENSING ACTIONS The efficiency of IRT can be beneficial for emergent or emergency licensing actions, though the prescribed process here may be adjusted for the situation by the PM. The PM should consider early inclusion of DRA staff, especially if risk insights are being discussed by technical reviewers. The IRT should leverage existing completed PRA work and other risk insights to expedite the review.

2 The technical staffs acceptance review is due to the PM within 20 days based on guidance in LIC-109. The PM may hold the meeting prior to this date if efficiencies can be gained (e.g., if previous meetings or discussions can be held sooner).

Page 11 of 23 The NRR staff will maintain the current processes and practices for emergent actions but apply the IRT approach to the extent practicable. For an emergent licensing action, the PM should involve the technical and risk branches, branch chiefs, and management, as appropriate, in an initial (and subsequent) staffing and scoping meetings as soon as practical, to understand the methodology for the review and who will be reviewing which aspects of the proposed changes.

The review team should decide whether there is sufficient time to develop a consolidated SE at the start of the review and if individual branch SE inputs are needed or preferred. The PM or a technical coordinator will need to consolidate the SE inputs into a LIC-101, Appendix B, or LIC-102 SE format; therefore, the PM with assistance from the review team will begin developing the consolidated SE outline at the start of the review.

5.0 PROJECT MANAGER CHECKLIST There are several checklists to facilitate implementation of the IRT process and they are:

• Project Manager Checklist (Section 5.0 of this document)

• Integrated Review Team Checklist (Section 6.0 of this document)

• DRA Checklist (Section 7.0 of this document)

• Project Checklist for Consolidated SEs and RAIs (Section 8.0 of this document)

The PM uses the following checklist to determine where a license amendment is Type 1, 2, or 3.

PM Checklist License Amendment Type License Amendment Request Type 1 Applications contain little or no risk/PRA information Type 2 Applications contain PRA information, but are not submitted as risk-informed applications in accordance with RG 1.174 Type 3 Applications submitted as risk-informed applications in accordance with RG 1.174 6.0 INTEGRATED REVIEW TEAM CHECKLIST The PM will determine which branches to initially assign to the review based on the level of probabilistic risk assessment (PRA) information in a submittal. The team then meets to determine if additional reviewers are needed and if an IRT should be used. The IRT checklist will assist the teams with determining whether to apply an IRT. This checklist (i.e., Table 1 below) can be used for Type 1, 2, and 3 submittals, including license amendments, relief requests (or proposed alternatives), and exemption requests. The checklist correspond to the decision on whether to form an IRT based on considerations of the review or licensee submittals.

Table 1 lists decisions (i.e., Integration Recommended, Integration Considered, and Integration Not Necessary) and descriptions under Description of Review or Submittal Topic. If any of the descriptions in Table 1 applies, then the corresponding decision can be selected.

Page 12 of 23 Table 1, Integrated Review Team Checklist Decision Description of Review or Submittal Topic Integrated 1. Risk-informed reviews in accordance with Regulatory Guide Review Team (RG) 1.174 Recommended

2. Requests to adopt Risk-informed Technical Specifications Task Force (TSTF) travelers (e.g., Consolidated Line Item Improvement Process, or CLIIPs), unless risk considerations were taken into account during the TSTF approval process

3. Submittals that include risk information, but not in accordance with RG 1.174

4. Submittals that are complex and involve many technical disciplines, even though they do not benefit from including a risk analyst, should be reviewed by an IRT without a risk analyst to enhance review integration Integrated A. Reviews that technical divisions have historically found to be Review Team challenging, specifically where PRA insights may help in Considered completing the review more effectively B. Reviews where it is not feasible to complete a portion of the deterministic justification in an appropriate time frame C. Changes to Technical Specifications where single failure and other conservatisms in Updated Final Safety Analysis Report (UFSAR) Chapter 15 are no longer considered as part of the assumption D. Reviews that involve multiple system integration E. Reviews that are related to SSCs that are important to safety, but risk information is not provided F. Submittals that involve many technical disciplines, even though they do not benefit from including a risk analyst, should be considered for an IRT without a risk analyst to enhance review integration Page 13 of 23 Table 1, Integrated Review Team Checklist Decision Description of Review or Submittal Topic Integrated 1. Administrative changes Review Team Not Necessary 2. Reactor risk neutral activities, such as offsite sirens, domestic water, etc.

3. Other reviews where integrating the review with risk analysts will have no significant use, such as those where risk methods are not available, or reviews where including risk information is not needed to ensure an efficient and effective review

4. Submittals that involve a single technical discipline and do not benefit from including a risk analyst NOTE: If NRC SEs will include some discussion of risk insights, then the DRA should be involved.

6.1 IRT Recommended 6.1.1 Risk-informed reviews in accordance with RG 1.174 These reviews, which are Type 3 reviews, require the use of risk analysts from DRA. These reviews should be integrated as soon as practicable to ensure that the technical reviewers and risk analysts understand the scope of their respective portions of the review, the justification for the bases for acceptance in each of the technical and risk areas, and technique for formatting an integrated technical evaluation.

Technical reviewers should become familiar with risk-informed reviews in accordance with RG 1.174, with focus on the principles of RIDM. Also, see available references in Module 1 of iLearn Course_ID 322150, Inspector Training on Risk-Informed Completion Times. The Technical Specification Branchs branch-specific training, Section B (ADAMS Accession No. ML17086A415), contains a primer about risk. Technical reviewers may also reach out to risk analysts for information about the RIDM process or precedents involving risk reviews.

Risk analysts that are new to the topic area may find technical information in Technical Training Center series training, or reach out to the technical branches for references.

6.1.2 Risk-informed TSTF traveler CLIIPs The Consolidated Line Item Improvement Process (CLIIP) is intended to make the review of technical specification changes more efficient. Reviews that use the CLIIP are called CLIIPs.

Even though CLIIPs are highly streamlined, CLIIPs that rely on risk information typically involve the review by a risk analyst from DRA.

Revision 5 of LIC-101 states, in part, Only the NRR Division of Safety Systems (DSS)

Technical Specifications Branch (STSB) and the PM in DORL typically need to review a CLIIP license amendment request, unless it is a risk-informed CLIIP license amendment request, which would also need to be reviewed by DRA/APLA. In some cases, risk considerations are Page 14 of 23 taken into account during the initial CLIIP review process. In these cases, the risk review of plant-specific applications of the CLIIP may be unnecessary.

6.1.3 Submittals that include risk information, but not in accordance with RG 1.174 For these reviews, the licensee is presenting what it believes is risk information that supports its safety case. This information is not sufficient to form the bases of a potential NRC acceptance of the submittal because it does not meet RG 1.174. However, the proposed change may lend itself to risk evaluations, and DRA may be able to provide some technical, rather than quantitative, insights to the review. The DRA risk analysts should be engaged early in these reviews to ensure that the provided risk information is used appropriately and within the correct context.

6.1.4 Submittals that are complex and involve many technical disciplines For these submittals, many technical disciplines are involved in the review. Even though the reviewers and PM decide that the risk analyst would not benefit from the review, an IRT can still be formed without a risk analyst, as discussed in Section 1.2.

The IRT concept encourages the reviewers from many technical disciplines to work together as a team. The communication within a team and developing joint consolidated products enhance the efficiency of the licensing process. The IRT enhances the integration of staff review and preparation of consolidated SE and RAIs from the many technical disciplines.

6.2 Integrated Review Team Considered For these reviews, there may be usefulness in engage with DRA risk analysts to provide supporting information or other insights to the staff reviews. In general, these are listed below in the order of the most likely to involve risk analyst involvement to the least likely.

6.2.1 Reviews that technical divisions have historically found challenging and where PRA insights may help complete the review more effectively For complex reviews that are technically challenging but without a clear challenge to plant safety, the staff should consider integrating risk insights. Without a fully risk-informed submittal, risk insights may not be used as the bases for acceptance of a request; however, the risk insights would be expected to assist the technical reviewers by supporting their safety conclusions and enhancing the technical reviewers confidence in their technical evaluations.

6.2.2 Emergency and exigent reviews Emergency and exigent reviews typically involve a request to quickly change a deterministic requirement. It is prudent to involve DRA risk analysts in these reviews early. The role of the use of risk insights would be to assist the technical reviewers in supporting their safety conclusions and enhancing the technical reviewers confidence in their technical evaluations.

Page 15 of 23 6.2.3 Changes to technical specifications where single failure, and other conservatisms in UFSAR Chapter 15, are no longer considered as part of the assumptions Safety-related SSCs and analyses often have numerous layers of protection, such as assuming a single failure or worst case scenarios. Where these numerous layers of protection are in place there may not be any use in applying risk insights developed by risk tools because the risk tools do not make the same assumptions (for example, regarding single failure). In cases where a safety-related analysis no longer relies on these prescriptive assumptions it is both useful and prudent to look to risk analyses for insights. This is often the case when a licensee is already in a technical specification completion time because single failure does not apply in that case, or for a submittal where the licensee is requesting not to consider single failure for a period of time (for example, when maintenance is planned).

6.2.4 Reviews that involve multiple system integration Reviews of submittals that involve multiple system integration that may not have an obvious risk connection should be considered to be discussed with risk analysts.

6.2.5 Reviews that are related to SSCs that are important to safety, but risk information is not provided Structures, systems and components that are important to safety are typically modeled in the PRA. For SSCs that are important to safety, there are typically more tools (e.g., SPAR, notebooks, etc.) available to the risk analysts to provide insights to technical reviewers regarding the risk insights related to an SSC that is important to safety. Discussion with risk analysts may be prudent to ensure all the roles of that equipment are considered in the deterministic review. In this case, the risk analyst may not provide risk insights, but may provide connections to other review organizations that are not obviously related to the review topic.

6.2.6 Submittals that involve many technical disciplines For these submittals, even though they may not benefit from including a risk analyst in the staff review, an IRT without a risk analyst should be consider to enhance the integration of staff interactive review and preparation of consolidated SE and RAIs as discussed in Section 1.2 of this document.

The IRT concept encourages the reviewers from many technical disciplines to work together as a team vs. the traditional silo reviews that each reviewer works independently and transmit the responsible section to the SE to the project manager at the end for integration. The communication within a team and developing joint consolidated products enhance the efficiency of the licensing process. The IRT enhances the integration of staff review and preparation of consolidated SE and RAIs from the many technical disciplines.

6.3 Integrated Review Team Not Necessary For these reviews, there is currently not an identified need to engage with DRA risk analysts.

Either there is not a risk aspect to be considered, or the technical review process is mature enough such that adding risk resources would not improve the efficiency or effectiveness or safety decision of the review.

Page 16 of 23 6.3.1 Administrative changes If administrative changes do not impact the operation or maintenance of the plant, then there is no need to engage with DRA risk analysts for reviewing these changes. Requests for NRC review of operational, testing, or maintenance procedure related changes could potentially impact plant risk and should not be excluded from possible integration by categorizing these types of changes as administrative changes.

6.3.2 Reactor risk neutral activities This includes reviews of topics that are not related to components that could have an impact on the safety of the reactor, such as the use of offsite sirens or domestic water. If the technical reviewers for such changes are new to the reviews, discussions with risk analysts may be prudent.

6.3.3 Other reviews where integrating the review with risk analysts will have no significant benefit, as agreed on by DRA For a large number of reviews there currently is no integrated review and it may continue to be the most efficient and effective way to manage these reviews. For example, where single failure is required to be considered, there is little value in layering in a risk review that uses different criteria. Other large reviews, such as power uprates and license renewals, have well established processes - and there is no need for PMs and technical reviewers to ask if risk analyst involvement is necessary, unless a new or unique situation dictates that risk may play a larger factor.

6.3.4 Submittals that involve a single technical discipline For submittals that do not benefit from including a risk analyst in the staff review, an IRT is not necessary because the SE and RAIs input is from a single technical branch.

7.0 DRA CHECKLIST During the scoping meetings the PMs and reviewers will need to determine whether to use risk insights and therefore use an IRT approach. The DRA checklist below identifies risk tools to provide insights for reviews of applications that were not submitted in accordance with RG 1.174. These risk insights may be developed based on quantitative information but will be characterized as qualitative risk insights when included in SEs. The staff may use risk tools such as SPAR models, the risk triplet, or event frequencies to develop these qualitative risk insights. The RG 1.174 quantitative risk thresholds are not expected to be used as the basis for approving changes that were not submitted in accordance with RG 1.174.

The questions in the checklist below provide a framework for developing risk insights that may provide additional confidence in conclusions reached by technical reviewers using traditional defense-in-depth and deterministic evaluations. The DRA staff will be responsible for facilitating the use of this checklist with the review team when considering whether PRA risk insights should be included in licensing action decision-making. Quantitative or qualitative PRA risk insights are considered in this checklist. The DRA staff should assist the IRT in differentiating probabilistic information from PRA risk insights. The staff will use this checklist to determine whether and in what form PRA risk insights should be included as part of an SE. The DRA risk analysts should be considered as the risk insight specialists and will be responsible for Page 17 of 23 reviewing risk-related information contained in SEs. The DRA staff will use NRC SPAR models or publicly available information provided by a licensee as part of risk-informed reviews to develop these risk insights, when possible.

If there is significant risk discussion or quantitative risk information, (e.g., CDF), but the LAR is not in accordance with RG 1.174 principles, a risk analyst should be included. Significant risk discussion is not a casual mention of a submittal being low risk, but substantive discussion of risk insights. If risk discussion is substantive or quantitative risk is mentioned, then risk analyst should be included.

The review team, with DRAs lead, should compile answers for all questions in the checklist when considering whether the review may use PRA risk insights. If the risk analysts response to Questions A-E is No (N), the proposed change is not considered to be risk significant. If the answer to any of those questions is Yes (Y) or not available (NA), the risk analysis may perform additional analysis in response to Question F or develop risk insights using SPAR models or other tools.

The checklist helps focus discussion between DRA and technical reviewers.

Page 18 of 23 DRA Checklist for the Integrated Review Team Process Description of the Proposed Change:

Y N NA A. PREVENTION: Does the proposed change affect the likelihood of events that perturb operation of the plant and lead to an undesired plant condition?

1. Does the proposed change affect the likelihood of perturbations or add new perturbations from internal plant causes (e.g., hardware faults, floods, or fires)?

2. Does the proposed change affect the likelihood of perturbations or add new perturbations from external plant causes (e.g., earthquakes or high winds)?

Y N N/A B. PROTECTION: Does the proposed change adversely affect common cause failures (e.g., an operable structure, system, and/or component (SSC) may be subject to the same failure mode as an inoperable SSC)?

1. Does the proposed change increase the likelihood of a cause or event that could cause simultaneous multiple component failures?

2. What is the risk significance of common cause failures? Is a related SSC that would be subject to common cause failure highly relied upon by the proposed change?

Page 19 of 23 Y N N/A C. MITIGATION: Does the proposed change affect the likelihood of successful plant response to plant perturbations (events that would cause the plant to implement abnormal operating procedures (AOPs) and emergency operating procedures (EOPs))?

1. Is the likelihood of the affected SSCs to successfully perform its required function(s) (during a specified period) affected by the proposed change?

2. What is the likelihood of the affected SSCs not being capable to supporting their functions due to being unavailable for test or maintenance? Is this unavailability of an SSC due to test and maintenance (T&M) affected by the proposed change?

3. Does the proposed change affect the likelihood of restoring a function due to failure of affected SSCs?

i. Does the affected function rely on diverse/redundant SSCs?

What is the recovery likelihood of the affected SSCs or affected function?

ii. Do the proposed compensatory measures manage risk-significant configurations?

Y N N/A D. DEFENSE-IN-DEPTH CONSIDERATIONS

1. Does the proposed change significantly increase the likelihood of an event or introduce a new event that could simultaneously challenge multiple barriers (e.g., interfacing systems loss-of-coolant accident (ISLOCA) and steam generator tube rupture)?

2. Does the balance among the layers of defense remain appropriate?

Page 20 of 23 Y N N/A E. HUMAN ACTIONS: Does the proposed change increase failures or unavailability of SSCs (or function) caused by human inaction or inappropriate actions?

1. Does the proposed change create new human actions?

2. What is the likelihood of errors associated with new or affected human actions?

3. Are new human actions important to preserving layers of defense?

4. What are the absolute or relative contributions of new or affected human actions to overall risk?

F. RISK INFORMATION:

1. If risk information is known, what are some generic risk insights for the plant?

a. What is the overall risk of the plant (i.e., baseline risk)?

b. What are the drivers of risk (or change in risk) (e.g., at the initiating events, accident sequences, and cut sets levels)?

c. Has the NRC reviewed applications that proposed similar changes? What conclusions were made in those reviews by the NRC?

2. What are the absolute and relative contributions of the affected SSCs, collectively and individually, to overall risk?

a. What is the increase in risk if the affected SSC (or a collection of SSCs) was assumed to be failed or unavailable?

b. What is the relative contribution of the affected SSC (or a collection of SSCs) to the calculated risk?

c. What is the sensitivity of risk to the performance of the affected SSC?

Overall

Conclusions:

Page 21 of 23 8.0 CONSOLIDATED SE AND RAI INPUT There is efficiency when members of an IRT work together to produce consolidated SE and RAIs. This process should be considered when the staff review involves multiple technical disciplines, with or without a risk analyst. The following process steps are for the development of the consolidated SE and RAIs. The discussion below about a risk analyst does not apply if there is no risk analyst involved in the review.

Figure 1 shows the timelines for a traditional review and an IRT review from the time a licensee submittal is entered into ADAMS to when the SE is issued. The Project Checklist for Consolidated SEs and RAIs provides details of the IRT process. Existing staff guidance continues to apply such as branch chief review of staff input to RAIs and the SE, PM review of RAIs and SE, and PM preparation of the licensing package. However, the IRT approach requires a preparation of a consolidated SE outline early and more internal meetings for communication and coordination than the traditional approach. It is important to have an agreed-upon consolidated SE outline, so each member of the IRT knows where his/her input fits in the SE and the PM can confirm that there is no gap in the SE.

Figure 1. Review Timelines for Traditional and Integrated Review Teams Review Timelines 12 Months with 9 Months as Stretched Goal PMs SE consolidation &

AR Issued in DORL, OGC Traditional 25 working Branch-Specific Draft SEs Branch-Specific Concurrences &

days and RAIs to DORL RAI Responses SEs to DORL Issuance Review Kick-Off Team Meeting Timeline AR Issued in 25 working OGC, DORL BC Integrated days Concurrences &

Draft SE Tech, DORL LA Review Outline Draft SE & RAIs developed RAI Responses SE Concurrences Issuance Team Staffing, Timeline Scoping, &

AR Meetings ADAMS Add T0 1m 2m 3m 4m 5m 6m 7m 8m 9-12m Page 22 of 23 Project Checklist for Consolidated SEs and RAIs The following steps are in addition to those in other applicable office instructions.

Who? Task Adding Project to RRPS PM Add Project to RRPS & assign Type 1, 2, or 3 o Type 1 (no PRA) o Type 2 (PRA info, but not RG 1.174) o Type 3 (formally risk-informed; RG 1.174)

Key search words: risk, probabilistic, CDF, LERF, frequency PM Assign branches (or the appropriate DRA branch) based on Type number o Type 1 - technical branches (TBs) only o Type 2 & 3 - technical branches and DRAFO Staffing Meeting PM Schedule Staffing Meeting within 10 working days Team For Type 1 & 2, determine if other reviewers and DRA need to be added or remain.

For Type 3, determine if tech branches should remain or need other branches.

PM Assign any new branches.

Tech Remove branches not needed.

Scoping Meeting PM Schedule Scoping Meeting within 15 working days.

Team Identify all changes and who is reviewing each change.

Team Decide if IRT will be used.

Acceptance Review Meeting PM Schedule Acceptance Review meeting within 20 working days.

Team Discuss hours estimates, complex review determination.

Team Identify any acceptance review issues.

Team Discuss IT format for consolidated SE (email, ADAMS, SharePoint, etc).

Team Identify challenges to meeting 25-day metric -refer to LIC-109, Section 3, page 6 Consolidated SE Prep PM Schedule consolidated SE meeting in about 6 weeks from acceptance review

- schedule laptop/projector if needed

- establish SharePoint site location or ADAMS ML# if needed PM Develop LAR/relief request/exemption package with draft SE outline; For LARs, start filling out Sections 1, 2.1, 2.2, 2.3 if pre-GDC TB/DRA Technical reviewers & DRA nominate a technical coordinator TB/DRA Begin Section 2.3 and outlining Section 3 (for LARs)

TB/DRA Send PM draft outline within 4 weeks of acceptance review PM Combine technical branch/DRA SE outline with PMs outline; send to team within 5 weeks Consolidated SE Meeting (may need multiple meetings)

PM Discuss version control of SE (e.g., have technical branch/DRA only check out document when updating; not while developing)

Team Discuss and agree on an SE outline Team Ensure Section 2 is accurate Team Agree on review methodology PM Get branch chiefs informal approval on draft SE outline (after meeting)

SE and RAI Development Team Finish draft SE, with any holes prior to sending RAIs.

Team Develop and consolidate RAIs into one document (no branch-specific identifiers)

PM Get branch chiefs concurrences on draft consolidated RAIs.

Team Finalize consolidated SE after any RAI responses received Page 23 of 23 The process as described above contains specific timeframes. When necessary and except for timelines associated with metrics, the staff has flexibility to manage their review with the PM and propose timelines that suit the workload. In addition, it may not be necessary to have the formal meetings as described because the IRT has flexibility to decide when it is appropriate to hold meetings.