ML22090A108

Issuance of LIC-206, Revision 1 Rollout Working Group Final Report (Cac/Epid: A11008/L-2020-PPM-0010)

ML22090A108
Person / Time
Issue date:	03/30/2022
From:	Jeremy Bowen, Meena Khanna, Ross-Lee M, Steve Ruffin, Gregory Suber, Bernie Thomson NRC/NRR/DEX, NRC/NRR/DNRL, Division of Operating Reactor Licensing, NRC/NRR/DRA, NRC/NRR/DSS
To:	Andrea Kock Office of Nuclear Reactor Regulation
Rivera R
References
CAC A11008, EPID L-2020-PPM-0010
Download: ML22090A108 (52)

Text

March 30, 2022

MEMORANDUM TO: Andrea L. Kock, Deputy Director for Engineering Office of Nuclear Reactor Regulation

THRU: Jane E. Marshall, Director Division of Decommissioning, Uranium Recovery, and Waste Program Office of Nuclear Material Safety and Safeguards

FROM: Steve R. Ruffin, Acting Deputy Director /RA/

Division of Engineering and External Hazards Office of Nuclear Reactor Regulation

Mary Jane Ross-Lee, Deputy Director /RA/

Division of Safety Systems Office of Nuclear Reactor Regulation

Bernie I. Thomson, Deputy Director /RA/

Division of New and Renewed Licenses Office of Nuclear Reactor Regulation

Jeremy S. Bowen, Deputy Director /RA/

Division of Advanced Reactors and Non-Power Production and Utilization Facilities Office of Nuclear Reactor Regulation

Gregory F. Suber, Deputy Director /RA/

Division of Operating Reactor Licensing Office of Nuclear Reactor Regulation

Meena K. Khanna, Deputy Director /RA/

Division of Risk Assessment Office of Nuclear Reactor Regulation

SUBJECT:

ISSUANCE OF LIC-206, REVISION 1 ROLLOUT WORKING GROUP FINAL REPORT (CAC/EPID: A11008/L-2020-PPM -0010)

This memo summarizes the activities conducted by the working group established to drive the integration of Office Instruction (OI) LIC-206, Revision 1, Integrated Risk -Informed Decision-Making for Licensing Reviews, into the culture of the Office of Nuclear Reactor Regulation (NRR). The intent of LIC -206, Revision 1 was to establish a framework and practical guidance for staff to use in forming integrated review teams (IRTs) and applying risk-informed decision-making (RIDM) using a structured and consistent approach that could be easily used by all reviewers, including strategies and tools for incorporating risk insights. The LIC-206 process is a major cornerstone of the Be riskSMART framework and provides the details needed to implement its high-level goals. Consistent with the scoping of LIC -206, Revision 1, the working group focused its efforts on license amendment requests for operating reactors and worked with the respective branches responsible for those licensing actions. of this memo describes the format of the working group to support three critical needs in rolling out LIC-206, Revision 1 to the NRR staff for proactive adoption and consistent implementation. These needs were communications and training, observation and facilitation, and metrics and tracking. Enclosure 1 also highlights the accomplishments of the work ing group by summarizing the objectives, approaches, and outputs/outcomes for each of the teams activities. A selection of deliverables and in-depth supporting information from all three teams are provided in the remaining enclosures. L essons learned from working group activities are captured throughout this memo and inform the working groups recommendations, which are summarized in Section 7 of Enclosure 1. Notable recommendations include the continuation of routine training and communications, maintenance of dedicated resources that promote and support the use of LIC-206, Revision 1, appropriate engagement with external stakeholders, and ensuring that continued messaging on RIDM avoids common pitfalls that di scourage staff from adopting RIDM. Implementation of these recommendations facilitate the creation of a sustainable infrastructure whereby RIDM can be effectively and permanently ingr ained into NRR culture.

CONTACT: Richard Rivera, NRR /DANU/ UARL Ian Tseng, NRR/DEX/EMIB Dan Widrevitz, NRR/DNRL/NVIB Alissa Neuhausen, NRR/DRA/ APLC Page 1 of 21

ENCLOSURE 1

LIC-206, Revision 1 Rollout Team

Final Report Page 2 of 21

OFFICE OF NUCLEAR REACTOR REGULATION FINAL REPORT FOR ROLLOUT OF OFFICE INSTRUCTION LIC-206, REVISION 1, INTEGRATED RISK -INFORMED DECISION -MAKING FOR LICENSING REVIEWS

1.0 Executive Summary The U.S. Nuclear Regulatory Commission (NRC, the Commission) Office of Nuclear Reactor Regulation (NRR) developed Office Instruction (OI) LIC-206, Integrated Risk-Informed Decision-Making for Licensing Reviews, (ADAMS Acce ssion No. ML19263A645 ) as a culmination to the recommendations of the NRR Risk -Informed Decision-Making (RIDM) Action Plan (ADAMS Accession No. ML18169A205). LIC-206 provides staff guidance for RIDM in the review of routine and emergent plant-specific license amendment requests to enhance process efficiency and effectiveness and to advance the nuclear reactor safety program towards becoming a more effective and risk-informed regulator. The NRR staff developed and deployed Phase s 1 and 2 of the RIDM Action Plan from 2017 through 2019, to continue advancing, for the operating reactor program, NRCs longstanding commitment to increased use of RIDM. Two of the ticketed action items resulting from the RIDM Action Plan were the development of a new OI on RIDM, and the development of new staff guidance for RIDM for inclusion in the new RIDM OI. The first of these two actions led to the development of Revision 0 of LIC -206, and the second led to the development of Revision 1 of LIC-206, with special focus on authoring the new Appendix C, which provides technical reviewers in NRR the guidance and tools needed to systematically increase their use of RIDM in their licensing work.

The focus of this report is on the accomplishments of the working group established following the issuance of Revision 1 to LIC-206. To familiarize the staff with Revision 1 of LIC -206 and to lay the groundwork for its successful implementation, a three-pronged rollout team was assembled to:

1. Address communication and training needs.

2. Observe and facilitate Integrated Review Teams (IRTs).

3. Recommend and develop metrics and tracking to understand and motivate the use of RIDM.

The methods and results of the rollout teams efforts are detailed in this attachment. In summary, the rollout team was successful in its endeavors by:

1. Reaching out with targeted communications and providing training to more than 85%

of NRR staff.

2. Observing 13 and facilitating more than 5 IRT meetings.

3. Identifying key parameters to measure the use of LIC-206 and implementing workload management changes to track them.

By completing these tasks, the rollout team has increased NRRs use of RIDM and laid the groundwork for future steps in creating a sustainable infrastructure whereby RIDM can be effectively and permanently ingrained into NRR culture.

Section 2 of this report documents the background and objectives of this project. The rollout group structure is described in Section 3. Sections 4, 5, and 6 provide detailed task goals and results organized by the three teams conducting the rollout. Section 7 includes recommendations on how to proceed into the next phase of realizing NRRs goal in becoming a more modern risk-informed regulator. Conclusions are documented in Section 8. Page 3 of 21

2.0 Background and Objectives In response to Commission direction to increase the adoption of RIDM at the NRC, the Ac tion Plan described a systematic, two-phased approach for implementing the NRR leadership vision for RIDM. Phase 1 of the RIDM Action Plan consisted of the analyses, evaluations, findings, and recommendations for enhancing the integration of risk information into NRRs decision-making activities. Phase 1 concluded with the issuance of a report in June 2018 that provided recommendations for fostering a lasting cultural impact (ADAMS Accession No. ML 18169A205).

During Phase 2, the staff obtained feedback on the Phase 1 recommendations through trial period activities, interviews, and surveys to determine how to implement the recommendations (ADAMS Accession No. ML19007A339).

An outcome of the RIDM Action Plan was the two -phased creation of new staff guidance for RIDM. The first phase resulted in the creation of LIC-206, which formalized the Integrated Review Team (IRT) process, a licensing review approach that promotes early engagement and increased collaboration between risk analysts and traditional engineering technical reviewers with a systematic, graded approach for expanding the use of r isk insights within the existing NRC regulatory framework. The second phase resulted in Revision 1 of LIC-206 (ADAMS Accession No. ML19263A645), which built upon the initial revision to add Appendix C to LIC-206. The revision further enhanced and refined the IRT process based on implementation feedback, and provided technical reviewers with str uctured and actionable guidance and tools to implement and document the use of RIDM in their review s to:

1. Use risk information to tailor the focus, depth, and scope of reviews.

2. Consider licensee-submitted and staff-generated risk information.

3. Independently assess the adequacy of licensee approaches considering risk.

Following the publication of Revision 1, the LIC-206 team pivoted to focus on implementation of the OI across NRR and was rebranded as the rollout working group with updated membership.

The LIC-206 rollout working group was established and structured as three interconnected teams following the official release of LIC-206, Rev ision 1 in the summer of 2020. The selected membership of the three t eams included substantial overlap to ensure consistency and support for the holistic team objectives. Team A focused on communication and training to promote the use and success of RIDM and IRTs. Team B focused on gathering information on the IRT process in real time, including facilitating IRTs, and using this information to increase the use of RIDM. Team C focused on the evaluation and generation of metrics and tracking to capture the use, cost, and effectiveness of RIDM and IRTs.

3.0 LIC-206 Rollout Working Group Structure Early in the rollout for LIC -206, Revision 1, the working group identified three focus areas to support outreach efforts related to the use of risk information. A team was formed under the broader LIC-206 rollout working group for each focus area, namely Communications & Training (Team A), Observation of IRTs (Team B), and Metrics & Tracking (Team C), as addressed above. Each team included a dedicated Branch Chief champion and was composed of multi-disciplinary staff, to ensure the inclusion of diverse ideas and perspectives on the use of risk information.

The overall working group (Figure 1) wa s led by Jane Marshall, the LIC -206, Revision 1 rollout team SES champion, with input from the RIDM Counterpart Members, composed of the Deputy Directors in NRR. Day to day implementation of working group activities was shepherded by the Team Leaders, who leveraged the Branch Chief Champions, as needed, to effectively reach audiences at all levels of the NRR organization. The full composition of the Teams and all Page 4 of 21

counterparts are listed in Enclosure 2.

SES Champion

• RIDM Counterpart Members Jane Marshall

Angie Buford Stacey Angie Buford Branch Chief Champions Team A Rosenberg Team C Team B

Team Leaders and PM Alissa

• Richard Rivera, PM Ian Tseng Dan Widrevitz Neuhausen

Working Group Members, Multi-Division Multi-Division Multi-Division Contributors, and Liaisons & -Disciplinary & -Disciplinary & -Disciplinary

Figure 1. LIC-206 Rollout Working Group Structure

4.0 Team A - Communications & Training Actions and Results The primary objective of Team A wa s to promote awareness and understanding of the guidance in LIC-206, Revision 1 through presentations, promotional materials, workshops, and training such that technical staff gain a level of comfort using risk information to support and inform reviews at an increased capacity. Secondarily, the team had an intangible goal to win over hearts and minds and ultimately, to inspire a cultural shift in how the NRR staff perceives RIDM.

In inspiring a cultural shift, the team aimed to change the perception of RIDM from a foreign and external imposition to a useable methodology that the staff understands, identifies with, and can implement and control. The team identified early that most of the communications regarding the agencys use of RIDM ha d been at a high level and for a general audience. It was strategically decided for the rollout team to personalize the RIDM message as it relates to LIC-206, for technical reviewers, and to address smaller groups of NRR staff with communications and training targeted to their specific needs. To better understand the staffs needs, the team leaned heavily on focus group sessions and workshops hosted for groupings of staff who work closely together, generally organized by branch or discipline. By employing this labor-intensive personalized approach, the team was able to iterate and hone its message and overcome many of the hurdles that previously hindered widespread usage of RIDM to realize the desired cultural shift in NRR.

4.1 Centralized Communication

4.1.1 Objective The team identified early in the project that some staff were unsure who to contact with questions about how and when to apply RIDM in their work. This confusion was further exacerbated by the existence of a breadth of viewpoints across the office about what RIDM is and about how and Page 5 of 21

when it applies to our regulatory processes. To send a clear and consistent message to the NRR staff, Team A identified the need for a centralized point of communication for users of LIC -

206, Revision 1 for asking questions, get ting help, presenting suggestions, and providing comments and feedback.

Additionally, as the rollout of LIC-206 continued, it became clear that having an identified and centralized location where users can easily access the OI and the promotional and supporting materials, including the safety evaluation repository, would increase the use of these materials and improve adoption of the LIC -206 process. When this topic was discussed during the DORL Focus Group session, it was suggested that the staff knows how to find things on Nuclepedia, since one only needs to access Nuclepedia and then search for LIC -206 to find its corresponding page. The rollout team agreed that the best solution in providing long-term durable easy access to those reviews would be to create and maintain a Nuclepedia page for LIC -206 with hyperlinks to the OI and its promotional and supporting materials.

4.1.2 Approach The team created and monitors a RIDS mailbox, RIDM.Resource@nrc.gov to serve as the centralized point of communication for all LIC -206 users. The team provided numerous pointers to this mailbox in all rollout team presentations, promotional materials, and trainings. The team also directed management, project managers, and technical staff to use the mailbox to contact the team for support and advertised its use at management briefings and RIDM c ounterpart meetings for management to use for LIC-206 related requests. The mailbox was heavily advertised and included on all materials to increase familiarity with its availability to all staff and management, and to normalize its use for everything LIC -206.

The LIC-206 rollout team created a Nuclepedia page and consistently referred users to use this page to locate documents throughout the latter half of the rollout, with the goal of solidifying this as a durable location that users can refer to when they need to access the OI or its promotional and supporting materials.

4.1.3 Output/Outcome Throughout the rollout, the team identified that the RIDM.Resource@nrc.gov mailbox showed increasing levels of use, especially following various LIC -206 activities that involved staff participation. The mailbox was used as a vehicle to request specific feedback and survey responses, and to respond to individual staff questions. The mailbox also helped streamline communications within the team.Also, after steadily promoting Nuclepedia as a location for users to find the OI and its promotional and supporting materials, the team received anecdotal feedback from staff that they knew to go to Nuclepedia to locate the documents. Going forward, it is recommended that both the LIC -206 Nuclepedia page, and the RIDM.Resource@nrc.gov mailbox should be maintained as an effective centralized communication vehicle - see Recommendation 7.2.

4.2 Supporting and Promotional Materials

4.2.1 Objective The team recognized that as part of its communication strategy to encourage the use of LIC-206, targeted promotional materials needed to be developed for several purposes and audiences throughout NRR. The team sought to market LIC -206 using an approach that combined instruction, creativity, and brevity. These materials were shared via multiple communication modes to optimize availability and accessibility. Page 6 of 21

4.2.1 Approach Promotional materials were created to target specific audiences with tailored messaging. A sample of the promotional materials that were created include:

• Guide to LIC-206, Revision 1 Comic Strip

• Delta Document explaining differences between Rev ision 0 and Revision 1 of LIC-206

• Discipline-specific one-pagers created for distribution to t echnical reviewers, risk analysts, project managers, technical branch chiefs, and projects branch chiefs

A guide to LIC-206, Revision 1 Comic Strip was developed as an eye-catching and engaging way to personify the roles in the IRT process by casting them as highly relatable staff superheroe s, and to promote awareness of the LIC -206, Revision 1 process and contents.

Consistent use of this imagery was carried through to provide illustrative examples of the IRT process in Team As presentation materials. Multiple layouts and permutations of this comic strip were developed for a variety of targeted uses and its appearance was optimized for maximum impact for use in print media to be displayed in the office.

During its engagements with the staff, the team noted a lack of awareness that there were two revisions of LIC-206, and that this confusion was a barrier to successful messaging and consistent implementation of the LIC-206, Revision 1 process. A delta document highlighting and explaining the differences between Rev ision 0 and Revision 1 was created to head off and reduce this confusion.

Discipline-specific one-pagers were created to promote cohesion of understanding across disciplines office-wide about what RIDM is about and how and when it applies to our regulatory processes. Instead of taking a high-level and non-specific approach to communication regarding the agencys use of RIDM, the team identified the major NRR staff positions that have a role in the success of implementing RIDM in licensing reviews, ascertained the important differences each role has in the use of RIDM, and highlighted these discipline-specific elements in targeted position-specific one-pagers that staff can keep on hand as a quick reference to the LIC -206, Revision 1 process. The appearance of the one-pagers, as well as the language used, were designed to be punchy and interesting, with a distinctive appearance consistent with other promotional materials and presentations from the team.

A selection of these promotional materials is included in Enclosure 3.

4.2.3 Output/Outcome Promotional materials were distributed to the staff via e-mail, advertised during focus groups and training sessions, and marketed at ET Significant Topics and Division All Hands meetings. For posterity, a selection of promotional materials were made available o n the LIC-206 SharePoint site and the LIC-206 Nuclepedia page. It is recommended that these materials remain available to all NRR staff to assist and promote implementation of the LIC -206 process.

4.3 Targeted Presentations

4.3.1 Objective Team A gave internal presentations throughout NRR t o foster common understanding of LIC-206, Revision 1 to promote awareness, understanding, and use of the guidance, explain how this guidance supports NRRs transformation efforts to become a more modern risk-informed regulator, and to introduce the rollout team as a resource available to anyone looking to apply risk insights in their work. External presentations were delivered to share NRRs LIC-206 process for awareness such that licensees can appropriately communicate with the NRC for Page 7 of 21

greater efficiency when applying risk insights to inform license amendment requests and other licensing actions.

4.3.2 Approach The team pro actively approached various divisions and stakeholders to increase awareness and understanding of the LIC-206 framework and responded to requests made by various stakeholders to share the ongoing status of the LIC-206 rollout activities as part of the agencys transformation efforts. Selected team presentations are listed below. Sections 4.4 and 4.5 provide additional details on the focus groups and technical workshops.

• DNRL Division Meeting (June 2020)

• Risk-Informed Steering Committee (RISC) public m eeting (June 2020)

• RIDM Counterparts Meeting (June 2020)

• ASCC Newsletter Article (September 2020)

• DORL Division Meeting (September 2020)

• LT Brief (October 2020)

• DSS DeSSerts and DiscuSSions (November 2020)

• ET Significant Topics (December 2020)

• 6 DORL PM and Branch Chief Focus Group Sessions (February 2021)

• ET Brief (March 2021)

• RIDM Counterparts Meetings (June 2020, July 2020, November 2020, April 2021, August 2021)

• 16 Technical Branch Workshop Sessions across NRR (April/May 2021)

• NEI Risk Informed Forum 2021 (February 2021)

• PWROG Risk-Futures Initiative (June 2021)

• DEX Lunch and Learn (September 2021)

• Fall Integrated and Risk-Informed Decision-Making Forum (September 2021)

• Virtual NRC & BWROG Executive Oversight Committee (December 2021) 4.3.3 Output/Outcome The presentations, workshops, and articles resulted in greater awareness, understanding, and improved perception of LIC-206 across NRR and to external stakeholders and were broadly met with acclaim. Through increased awareness and understanding both internal and external to the organization, these presentations have enhanced NRRs stance and trajectory toward its goal of becoming a more modern risk-informed regulator.

4.4 DORL Focus Group Sessions

4.4.1 Objective Early in the process, consistent and proactive facilitation by DORL project managers was identified as critical to the successful adoption of the LIC -206, Revision 1 process. To better understand the situation, DORL held focus group sessions to assess the common understanding and practice of accepting risk insights among DORL staff, to share perspectives and experiences, and to ensure a consistent understanding across the DORL project managers of LIC-206, Revision 1 and their roles and responsibilities. These sessions were also used to collect feedback for consideration in the next revision of LIC-206 and to offer the rollout team as a facilitation resource available to all review teams and individuals.

4.4.2 Approach Each DORL focus group session was facilitated by two members of Team A, one to lead the Page 8 of 21

discussion, and one to act as scribe to ensure that a good record would be maintained from each session. A DORL PM also helped to facilitate discussions and helped to get the ball moving when needed.

Each session started with a short presentation introducing LIC-206, Revision 1 and how it fits within the NRCs licensing process. Each session administered a short survey using Microsoft Forms to gau ge current awareness and practices of the LIC-206 process and overall impression of the IRT process. Each focus group session then engaged in a facilitated two -way discussion to understand current practices regarding identification/classification of incoming submittals, current positions and comfort toward the use of risk, and suggestions to increase/enhance the use of RIDM and IRTs.These facilitated discussions were tr anscribed live by the facilitators directly on the PowerPoint slide deck to ensure the ac curacy of the feedback captured. The focus group sessions for DORL project managers and branch chiefs were intentionally kept separate to promote open discussion and to gauge differences in viewpoints between the two groups. The PowerPoint slide deck, including survey results and transcribed discussion, as well as supporting materials (including the project manager role -specific one-pager discussed earlier) were distributed to all participants shortly after each focus group session.

4.4.3 Output/Outcome Focus group sessions were held for project managers in each of the five plant licensing branches in DORL, plus one session for DORL branch chiefs. The focus group sessions resulted in improved awareness and perception of LIC -206, Revision 1 in DORL with several project managers engaging with the rollout team after their sessions for additional discussions, and to request facilitation for IRTs. It was also noted that the project managers who engaged with the rollout team in facilitation for IRTs were more likely to continue to request facilitation assistance, suggesting that the facilitations were beneficial.

The focus group sessions also resulted in several staff-proposed actions for the rollout team, including:

• Suggestions and feedback for a future revision of LIC-206 and for other associated guidance, which the rollout team captured,

• Suggestions to collect data to better understand resource implications,

• Suggestions for the office to explore budgeting overhead for staff performing risk assessments to support fee-billable work,

• Recommendations to hold similar discussions with technical staff, and

• Suggestions to create a Nuclepedia site for LIC-206 as a centralized location that users can find the OI and its prom otional and supporting materials.

The focus group sessions also yielded suggestions for the DORL SES team, from project managers and branch chiefs, t o:

• Further embrace proactive use of risk insights,

• Explain why risk should be used, and

• Improve early communication with licensees using risk insights.

About two months after the DORL focus group sessions, the team followed up with participants with the following list of actions that were taken as a result of their feedback:

• Revised our one pagers for PMs, technical staff, and branch chiefs (available in Nuclepedia) Page 9 of 21

• Scheduled similar discussions with technical staff in other divisions during upcoming months

• Identified and shared communication pitfalls with the L eadership Team

• Captured potential improvements for the next revision of LIC -206

• Updated the annotated PRIB desk reference to make it more user friendly

• Made more resources available in our Nuclepedia Page For more information regarding the DORL Focus Group Sessions, please see Enclosure 4.

4.5 Technical Workshops

4.5.1 Objective One of the key additions to Revision 1 of LIC -206 was Appendix C, which provides technical reviewers with the guidance and tools needed to leverage RIDM in their reviews. Consistent with the results from the RIDM Cafés conducted during the first two phases of the RI DM Action Plan, the LIC-206 rollout team observed early in the rollout that some technical reviewers would benefit from additional training activities to support positive change management and many technical reviewers did not feel that the guidance in LIC -206 applied to their review areas. Indeed, since all messaging for RIDM was historically targeted for broad audiences, none of the high-level messages could touch on the specifics of how any particular discipline could apply RIDM in their review work. The team decided that the best way to introduce technical reviewers to LIC-206 and realize the desired culture change would be in small branch-specific customized workshops where the team could ascertain the core functions and actions of each branch. These workshops also allowed the team to understand the branchs current views with respect to RIDM, give meaningful discipline-specific hands-on examples, and collaborate with the branch members to walk through actionable ways their work can be risk-informed. This approach would allow both the branch to gain a feeling of ownership toward future steps, and also provide the rollout team with valuable insights about how RIDM can apply to each discipline in future engagements and training, for future Standard Review Plan or Regulatory Guide level guidance updates, and as feedback to Team B to assist their facilitation of IRTs.

4.5.2 Approach Each technical workshop session was facilitated by at least t wo members of Team A; typically, one team member led the workshop and a risk analyst provided support. For some disciplines, an additional specialized contributor was identified ahead of the workshop to assist with tailoring the workshop content to the specific branch and help with facilitating the workshop session.

Each session began with an overview of the workshop, followed by an overview of the contents and purpose of LIC-206, Revision 1, including a walk -through of the IRT process, and an overview of the tools and guidance in Appendix C. Each workshop then presented a tailored hands-on risk -tool case study, which typically walked the participants through a relatable real-world example where they determine the submittal type, and explored parts of the checklist tool in Appendix C, which demonstrates how information gleaned from an electronic plant risk information book (e-PRIB) can be used to determine review scope or help support a safety finding. The case study ended with examples of how the actual review that served as the basis for the case study, was reflected in real approved safety evaluation language.

Each workshop then transitioned to a participant led discussion where the branch explains to the facilitators what their branch does, and how they do it. This discussion was captured live by the facilitators in a bulleted list directly on the PowerPoint slide, visible to the participant s to Page 10 of 21

encourage discussion and to ensure accurate transcription. After that, the facilitators led a brainstorming session with the branch on how each of the identified job functions listed previously could be enhanced with risk information. This discussion was further captured on a PowerPoint slide. Each session closed out with some time set aside for questions, comments, and general open discussion, and a plug for Team Bs facilitation services, and inviting participants to contact the LIC-206 rollout team through the RIDS mailbox for support.

Participants were e-mailed with the tailored PowerPoint slides containing their example problem and their captured feedback from their brainstorming discussions and their open discussion, as well as a copy of LIC-206, Revision 1 and the technical reviewer one-pager handout.

4.5.3 Output/Outcome Technical workshops were held with staff from branches in DEX, DSS, DNRL, and DRA that are most often responsible for the review of licensing actions. In total, workshops were held with 16 branches. These workshops enhanced technical reviewer understanding of how to interface with the PM and when to suggest using LIC-206 for a review, how to apply LIC-206 to their discipline-specific work, and shared available risk-informed resources to inform technical reviews.

Technical workshops with DRA were focused on how to interface with the PM and technical reviewers but were primarily focused on gathering additional feedback on improving the LIC -206 framework such that there is no conflict between it and existing Regulatory Guides and NUREGs for risk-informed applications.

Feedback from technical staff and branch chiefs indicate that these technical workshop sessions helped many technical reviewers gain an appreciation and better understanding of using the RIDM process in their work.

4.6 Team A Success Criteria Team A set out to promote awareness and understanding of the LIC-206 process through presentations, promotional materials, workshops, and training.To measure its success in making appreciable gains in awareness and fluency of RIDM and LIC -206, Revision 1 concepts in NRR, the team set a goal of engaging in communication with at least 85% of NRR staff responsible for leveraging the use of RIDM and applying the LIC -206 process.

Through the creation of a channel for centralized communication at RIDM.Resource@nrc.gov, establishing the LIC-206 Nuclepedia Page as a centralized repository for LIC-206 related materials, the team built a durable two-way communication path for users of the process and the rollout team. Through the development of supporting and promotional materials, the team provided engaging and accessible materials that promote and support the consistent implementation of the LIC-206 process. Through its presentations with stak eholders both internal and external to the agency, the team enhanced awareness and understanding of the LIC-206 process and its purpose, connecting it with the Be riskSMART framework, and with broader office and agency goals. Through its DORL focus group sessions, the team developed the understanding and provided the training needed to identify and break down barriers that were restricting the use of RIDM by project managers. Through its technical reviewer workshops, the team made substantial progress toward a broad embrace and ownership of RIDM concepts and the LIC-206 process by technical staff. Throughout its rollout process, Team A engaged with NRR staff responsible for a wide variety of tasks and functions. By its own estimates, Team A engaged with over 90% of the project management and technical staff in NRR who are most responsible for leveraging the use of RIDM and applying the LIC -206 process.

Based on the actions and results detailed above, Team A achieved succes s in its strategic implementation of communications and training to support the broader rollout of LIC -206, Page 11 of 21

Revision 1.

5.0 Team B - Observation and Facilitation Actions and Results The primary objective of Team B was to observ e IRTs and the implementation of the LIC-206 process. The secondary objective was to provide a facilitation resource for IRT meetings and its members. Observation was deemed a critical priority as creating a real-time feedback loop supporting Team A and identifying potential improv ements to the LIC-206 process required verification and evaluation of actual practice. To support these goals Team B observed and facilitated IRTs; and collected feedback, safety evaluation language, and documented lessons learned.

5.1 Interface with DORL

5.1.1 Objective As LIC-206 is a DORL led process, observations and facilitations depend ed on strengthening relationships between Team B and DORL peers. By establishing reliable contact points, Team B aimed to improve the identification of observation opportunities and increase the willingness of IRT members to request facilitation from the team.

5.1.2 Approach The team took a two-pronged approach. First, the team leveraged internal DORL tools and connections to identify IRT opportunities by sifting through data in the Reactor Program System (RPS ). Team C interactions concerning metrics and data collection, as discussed in Section 6 of this report, synergized with Team B goals to further enable these efforts to identify candidate IRTs for observation. Second, the team reached out to DORL counterparts and informal contacts to spread the word internally and to assist in identifying IRTs early in the review phase.

5.1.3 Output/Outcome Team B successfully built a process to identify IRTs and requested that team members are included in IRT meetings as observers or facilitators. This process was assisted by Team A and Team C efforts, which provided further contacts and communication venues to promote the use of our facilitation role.The use of automated systems, in particular RPS, enabled Team B to identify large numbers of projects and complete initial screening to identify observation candidates. Direct outreach to DORL project managers also identified several high-pertinence observation candidates. Ultimately, numerous IRTs were identified in a timely fashion, including several at the very beginning of their review process.

5.2 Observational and direct feedback on RIDM and IRTs

5.2.1 Objective Numerous efforts concerning RIDM and IRT observation have been, and continue to be, implemented at varying levels of the NRC. Team B intended to add value to existing efforts by establishing a durable feedback loop between the promotional efforts and actual staff comprehension and use of RIDM and IRTs. To support this effort, Team B observed and/or facilitated IRTs, evaluated safety evaluation language that discussed the use of risk insights in the staffs decision, provided real-time feedback to Team s A and C, and documented lessons learned. Page 12 of 21

5.2.2 Approach Team B conducted observations using a detailed template (See Enclosure 4). In addition, Team B employed Teams A and C activities, as well as connections with various counterparts, to identify further data and information, including additional safety evaluations for example language discussing the use of risk insights in the staffs decision. This effort also inc luded an exit survey sent to a broad audience of IRT participants (e.g., project managers, technical reviewers, and risk analysts). The purpose of this survey was to give participants a n opportunity to provide direct feedback to the LIC-206 working group on their experiences with RIDM and IRTs.

5.2.3 Output/Outcome Team B collected 1 7 observation reports covering 13 IRTs. This represented the majority of IRTs reported and identified between December 2020 and May 2021, with an emphasis on projects without a well-established precedent, typically Type 2 submittals as defined in LIC-206, Revision 1. For reference, Team B did not expend substantial effor t observing Type 3 submittals (i.e., TSTF-505, 10 CFR 50.59) because these types of review have a well-established precedent which include the use of IRTs, formal regulatory guidance, and dedicated risk analysts assigned to every review. This emphasis was informed by Team C s input. The exit survey provided important participant reported results confirming observations and improv ing the ability of Team B to reach conclusions. A sample of the exit survey results is included in Enclosure 6.

Team B additionally collected 3 safety evaluations that were deemed unique relative to the materials collected in support of the LIC-206, Rev ision 1 writing process. These safety evaluations were used to evaluate lessons learned and determine potential improvements to the LIC-206 process or communications effort.

5.3 Facilitation of IRTs

5.3.1 Objective The objective of facilitation was to assist all LIC-206 users with adjusting to the relatively new LIC-206 approach and general use of RIDM principles, and to provide an element of consistency to the approach. By providing a non-fee billable consulting resource (i.e., no charge to specific projects), Team B could be leveraged to enable more rapid uptake of IRTs and RIDM concepts and to relieve the project manager of perceived pressure to explain, advocate for, and coach the LIC-206 approach.

5.3.2 Approach Team B leveraged DORL contacts, Team A communications, and IRT observation s to spread awareness of available facilitation resources. Facilitation resources included assistance in conducting the LIC-206 process, which included interpreting how to classify an application using LIC-206, forming an IRT, using RIDM to support a primarily deterministic review, and documenting the use of RIDM in the safety evaluations.

5.3.3 Output/Outcome Team B was requested, by both formal and informal contacts, to assist in IRT conduct and RIDM application. The facilitation role overlapped in some respects with Team A communications and synergized to increase the comfort and awareness of NRR staff with IRTs and RIDM concepts.

Staff became more comfortable moving forward on their own initiative after receiving Team B confi rmation of their understanding. Further, Team B members from DRA provided specific PRA information to support numerous IRTs and in so doing, provided unique input to LIC-206 reviews. Page 13 of 21

5.4 Safety Evaluation L anguage Repository

5.4.1 Objective Team B planned to expand the database of information gathered during the LIC -206 revision process to identify safety evaluation language that could help support reviewers facing novel review challenges that also used RIDM.

5.4.2 Approach Input was solicited from LIC-2 06 Team members, NRR counterparts, and through information gleaned from IRT observations.

5.4.3 Output/Outcome A significant sample of safety evaluations were scanned and found consistent with or derived from a small number of precedents. A l imited amount of novel language was identified, and the language that that was identified was not sufficiently general to be extended for use beyond the subject of the documents. Team B concluded that the example language provided as a reference in LIC-206, Revision 1 is sufficiently broad to act as a template for most reviews and decided not to add new examples at this time. Additionally, the issuance of several relevant regulatory guides indicated that RIDM language is becoming more commonly understood amongst the staff.

It is clear, however, that unique and pertinent safety evaluation language continues to be generated and future efforts to make these examples available to staff will be an important component to NRRs progress towards becoming a more risk -informed regulator. The Team B members expect that safety evaluations stemming from the use of LIC -206 and other related efforts are likely to increase in number over time.

The team recommends that future phases of this project continue to evolve the safety evaluation language repository to suit the identified needs of technical reviewers (ADAMS Accession No. ML20108E881). This recommendation is further discus sed in Section 7.

5.5 Team B Success Criteria The Team B initial success criteria were to observe at least 10 IRTs, provide facilitation resources for at least 5 IRTs, and evaluate lessons learned. Building on these initial goals, the team planned to conduct exit interviews to glean observations directly from IRT reviewers and evaluate safety evaluation language to add value in the future. Finally, the team synthesized the results to support any proposed revisions to LIC -206 or related work items and provided final written insights to close the Team B efforts.

Team B was fully successful in achieving all these goals, exceeding the IRT observation and facilitation targets.

6.0 Team C - Tracking and Metrics Actions and Results Team C was tasked with exploring the use of metrics and tracking to support agency and office level initiatives for increasing the use of RIDM. Team C activities can be broadly broken down into two phases: 1) identify and evaluate existing data and metrics being used in NRR that support RIDM for applicability to LIC-206, and 2) evaluate, propose, and implement new metrics and tracking to support the increased use of LIC -206. When looking at possible metrics and tracking, Team C relied heavily on the Reactor Program System (RPS) and on the expertise in the DORL Workload Management Team (WMT). Page 14 of 21

Team C coordinated closely with the DORL WMT throughout the term of the rollout activities.

Initially, it was important for Team C to understand current office risk metrics and how they are tracked to prevent the duplication of efforts and to ensure that any proposals were in accordance with the expectations set by DORL and items needed for the Quarterly Performance Review.

After aligning with DORL, Team C formulated a list of input parameters and analyzed potential changes to office metrics and tracking to capture the use of risk information.

Once broad consensus was reached, the team formulated ideas on how to modify workload management tools (e.g. RPS) to capture data to analyze potential new metrics and tracking and continued further discussion with the WMT to submit proposed changes through the appropriate channels. Ultimately, consistent with the LIC-206, Revision 1 addition of Appendix C, which was geared at providing guidance to technical reviewers, Team C p roposed changes to RPS to better capture the use of risk insights by technical reviewers. Team C also consulted with experts in the WMT and dashboard developers to create a dashboard to help track the use of risk information in licensing reviews.

6.1 Identification and Evaluation of Existing Data and M etrics

6.1.1 Objective The primary objective for Team C was to identify and implement metrics and tracking to support the office goal of increasing the use RIDM. In determining the appropriate metrics and tracking for the use of LIC-206 and other RIDM initiatives, it was important to identify existing data, evaluate how the existing data could be used, and propose any new data points for collection. In tandem with looking at data, Team C focused on defining the goals of any proposed metric. A secondary objective for Team C was to provide feedback to Team B (Observation and Facilitation) to help identify good candidate IRTs.

6.1.2 Approach Team C looked at the existing data entered into RPS by project managers. The data initially identified as potentially relevant to metrics and tracking for LIC -206 is shown in Table 6.1. Of the three types of risk data already collected, risk type and IRT type are directly related to and defined in LIC-206. Risk type captures whether a review is Type 1, 2, or 3. IRT type captures when an IRT was used and if a DRA risk analyst was a part of the project team.

Table 6.1: Pre-existing data available in RPS Risk Complexity Sorting Risk Type Estimated Hours Licensing Action Type IRT Type Actual Hours Acceptance Date Risk Informed Precedence Divisions

1. of branches Branches Schedule Priority

Team C evaluated whether there was any correlation between the risk data collected (Table 6.1, column 1) and the complexity of the review (Table 6.1, column 2). Column 3 of Table 6.1 identifies how the data was sorted to look for possible correlation. s Some of the questions asked while looking at the data included:

• How many submittals were categorized as Type 1, 2, and 3 submittals in 2019 and 2020?

• What was the range of estimated and actual hours for reviews of Type 1, 2, and 3 submittals? Page 15 of 21

• How many reviews with 3 or more branches used an IRT a) with a risk analyst, b) without a risk analyst?

Team C considered three high level metric descriptions to evaluate whether any additional data collection was necessary.

1. Is the use of RIDM IRTs increasing?

2. Is the use of RIDM IRTs resulting in more effective reviews?

3. Is staff perception/adoption of RIDM IRTs (surrogate for culture change) improving?

6.1.3 Output/Outcome Team C identified several inputs to RPS related to risk that are entered by the project manager ;

and noted that these entries are all made at the start of the project. The milestone risk type and IRT type were identified as existing data that should be used to track the use of LIC-206.

Additional information that the team determined was necessary is discussed in Section 6.2 of this report. Team C did not identify any strong correlations between any data and ongoing IRTs stemming from the application of the LIC -206 guidance.

Team C further found that Type 3 reviews are well -defined and easily iden tified and recommended focusing observations and facilitations on submittals identified as Type 1 and Type 2. However, no strong correlations were found to help identify a likely Type 1 or Type 2 submittal as a good candidate for an IRT. A weak pattern was observed between the use of IRTs and the number of branches involved in a review. This is consistent with qualitative observations that IRTs may be most useful to larger reviews, but still provide value even for small review teams.

6.2 RPS Changes and Data Collection

6.2.1 Objective Team C proposed changes to RPS to support the collection of data related to the use of LIC -206 and RIDM for licensing actions. The objective of these changes is to have data available to support the high-level metric descriptions in Section 6.1.2.

6.2.2 Approach To propose new data for collection, Team C looked at the gaps between the existing data and the information necessary to support the high-level metric descriptions. Once the needed data was identified, Team C considered options for collecting this data, including RPS, eRAI, staff surveys, or the creation of a new system.

6.2.3 Output/Outcome The results of Team Cs approach identified that, pertaining to LIC-206 reviews and other RIDM, there is no formal data collection on the use of risk insights. The existing data on the use of risk is collected and entered by the project manager during the acceptance review process.

Consistent with LIC-206, Rev ision 1 Appendix C, which provides guidance on the use of risk information for technical reviewers, Team C determined that collecting information on the use of risk insights by the technical reviewers would better capture the use of LIC-206. Including this delineation in RPS for technical reviewers further reinforces the intention that using risk insights should be a consideration in the review.

The initial proposal, with some modifications has been adopted into RPS. For technical reviewers, the milestones identified as 1) RAI sent to PM ( Request for Additional Information sent to the project manager) and 2) SE sent to PM (Safety Evaluation sent to the project manager) Page 16 of 21

include options for the reviewer to denote if risk insights were used in their review process (See ). This change allows RPS to capture a project that uses only some elements of the LIC-206 OI, for instance, Appendix C, even if an IRT is not formed. Essentially, Appendices B and C are decoupled for reporting purposes. This change also allows data to be captured on all risk insights and is not restrictive to the use of risk insights in coordination with LIC -206 alone. A future RPS change has been accepted by the RPS Change Control Board to capture when specific techniques (e.g., LIC -206, Be riskSMART, the risk triplet) are used.

These changes will not feed directly into a metric because Team C did not identify an appropriate quantitative goal for LIC-206. However, these changes will provide the data necessary for tracking the usage of risk insights in and staff perception of RIDM IRTs and track the use of risk insights. These change s could also be used to ascertain a baseline on NRRs use of RIDM in licensing reviews. The main reason no quantitative metric was selected is that normalizing the data based on applicant submittals did not seem reasonable at this time. The formulation of a baseline use of RIDM may be able to support the creation of a quantitative metric in the future.

6.3 Data Analysis and Uses

6.3.1 Objective As stated above, Team C determined that, currently, the best use of data for LIC-206 and RIDM is to track and trend the use of risk insights for licensing actions in NRR. This information is collected on a dashboard for the LIC -206 Team, managers, and staff to better understand how risk information is being used in reviews of licensing actions. This data can be expanded beyond licensing actions and beyond NRR in the future, as needed.

6.3.2 Approach Team C worked with the DORL WMT and with the EMBARK MAP Dashboard Developers Team to design a dashboard that summarizes the total number of licensing actions submitted as Type 1, 2, and 3, and the total number of projects that relied on risk insights in developing RAIs or the safety evaluation. This dashboard incl udes a graphical representation of how IRTs and risk insights are used over time to facilitate the identification of trends in the use of LIC-206 and other RIDM techniques.

6.3.3 Output/Outcome A dashboard was developed and is available on the MAP Analytics web site. Potential future uses for the dashboard are discussed as recommended actions in Section 7.0.

6.4 Team C Success Criteria The Team C initial success criteria were to evaluate existing metrics, to develop new metrics or tracking on the use of LIC-206, and to provide the means to collect the necessary data to support the office goal to increase the use of RIDM. The team successfully implemented tracking of risk insights in licensing reviews through the implementation of one change to RPS. The team further proposed an additional change to RPS (currently pending review by the executive change control board) and developed a dashboard to track the use of risk insights in licensing reviews.

The Team was fully successful in achieving its goal to provide the means to support the office goal to increase the use of RIDM. Further activities are underway to bolster the existing tracking capabilities and to provide the data to management and staff to support ongoing and future projects. Page 17 of 21

7.0 Recommended Actions As discussed earlier, the LIC-206, Revision 1 rollout team made large and tangible strides in NRR toward overcoming cultural barriers limiting the use of RIDM, improved awareness and understanding of RIDM, and furthered the trend of increasing and diversifying the use of RIDM in NRRs products.The rollout team believes that without continued championing of the LIC -206 process, and RIDM more generally, NRR stands to lose the gains and inroads made to date in the expanded use of RIDM.

To maintain momentum and continue building on the gains achieved thus far, the rollout team recommends the following actions.

7.1 Future Training and C ommunications Development The LIC-206, Revision 1 rollout team provided general presentations and training to the broad NRR audience and developed targeted training in the form of focus group sessions for DORL project managers, and discipline specific workshops for technical reviewers. As discipline-specific use-cases evolve, and LIC-206, Revision 1 tools and guidance matures, updated training sessions should be developed and continued to facilitate positive change management. This training would help to publicize these tailored uses of the tools and guidance to subsets of NRR, and to ensure awareness and consistency of these evolving practices across the office.

Furthermore, the rollout team identified that increased dialogue among the staff is a key driver toward the increased use of RIDM. The team strongly feel s that continued training and communications are needed to maintain this dialogue, cement the cultural changes that have been realized, and attain the vision of becoming a more modern risk-informed regulator.

As described in Section 7.10, the rollout team recommends standing up a team that retains several key members from the existing working group and combines them with select new members for the next phase to continue with periodic training and communications, including communicating successes in the implementation of RIDM, thereby ensuring that LIC-206 becomes a permanent fac et of licensing reviews.

7.2 Maintenance of resources to support LIC-206 (i.e., facilitation, Nuclepedia, and RIDS Box)

The LIC-206, Revision 1 rollout team provided facilitation to IRTs, which helped garner support for the LIC-206 process and helped assuage misconceptions that using the LIC-206, Revision 1 process could increase project hours. Because the LIC -206 team was abl e to offer non -fee billable support, this was an incentive to using LIC -206. Since the team facilitated for a limited number of projects during the rollout phase, additional benefit will be gained from continued participation with project managers and technical reviewers that have had minimal exposure to IRTs or Appendix C to support positive change management.

Additionally, as LIC-206 is a living and operational process, the continued maintenance of its attendant Nuclepedia site and email resource requires allocation of continued resources.

7.3 Preservation of the Safety Evaluation Repository LIC-206, Rev ision 1 references a living document containing sample safety evaluation language. It is recommended that a RIDM expert be assigned the responsibility to maintain and expand this resource until such time as documentation of risk insights in safety evaluations is routine.Th e current template language represents a limited scope of RIDM application and should be extended to cover further review types, forms, and concerns.

The rollout team predicts that the safety evaluation repository will become increasingly useful Page 18 of 21

and critical in the future, both as inspiration for reviewers, and perhaps more importantly, to support and manage consistency, knowledge management, and precedent. The safety evaluation repository is expected to confer increasing benefits to staff as it becomes more comprehensive, further driving staff to iterate on the approved safety evaluation language that can be added to the repository. This iterative process of improvement underpins much of the strategy for the next phase, which focuses on maturing the LIC -206 processes s uch that its uses can be expanded to additional licensing and non-licensing activities.

7.4 Changes to RPS, Dashboard, and Metrics The first version of the LIC-206 dashboard is available for use (see Enclosure 8 for dashboard images). This dashboard provides an overview of licensing actions that are designated as Type 1, 2, and 3 submittals, and whether each review used risk insights for either RAIs or the SE. An update to the LIC-206 dashboard will incorporate the next set of RPS changes and will break down risk insights by the tools and techniques used by the NRC staff to incorporate risk insights in reviews.

In lieu of specific LIC-206 metrics, the dashboard will enable other entities and groups, such as DORL or the Objectives and Key Results (OKR) Team, to develop future measurements and to provide another method to track NRR performance towards becoming a more modern risk-informed regulator. The use of these dashboards should be monitored so that improvements can be made based on staff feedback, from the standpoint of both data entry and use.

7.5 RIDM Messaging General communication discipline continues to benefit uptake of RIDM concepts amongst staff.

Repetition and time-tested successful messaging should build upon the successful approaches achieved in rollout. Examples include technical w orkshops and avoiding RIDM messaging pitfalls to maximize credibility with audiences. Sustained vigilance is needed to continue these practices. Messaging should also include advertising our success stories in the implementation of RIDM, internally and externally to the organization.

Common RIDM messaging pitfalls identified by the rollout team:

• Implying RIDM is code for scheduling tightening.

• Presenting schedule/metric adherence as top priority or take-away.

• Perceived demotion of technical staffs ownership of safety.

• Equating old as bad and new as good.

• Implying that RIDM means doing less instead of doing right.

• PRAs are the only form of risk information.

Common successful RIDM messaging approaches:

• The application of RIDM supports our safety mission; in fact, it allows us to focus on the most risk and safety significant reviews.

• Present RIDM as an evolution in NRC practice based on current understanding of risk.

• Encourage audience to reframe RIDM principles within the language of their special ty.

• Emphasize ownership of RIDM approaches by specialties.

• Prioritize achieving technical and mission goals using RIDM, rather than RIDM being its own distinct goal.

7.6 Engagement with external stakeholders The staffs review approach is dependent on the type of submittals received from the industry (i.e., Type 1, 2, or 3 for LIC-206). Most current licensing submittals do not provide enough risk insights for the staff to consider beyond the highly qualitative procedural element of No Page 19 of 21

Significant Hazard Considerations. Thus, the next step is to encourage industry to include more risk insights in licensing action requests. To ensure consistency and alignment, staff and industry need to work together to develop a common understanding of the content of submittals containing risk insights and, if necessary, additional guidance for this purpose. The development of externally facing guidance to further support this need is discussed in Section 7.8.

The LIC-206 rollout team initiated a substantial grass-roots improvement to discussions surrounding RIDM within NRR. As the internal capacity to conduct RIDM reviews increases, it becomes increasingly important to communicate our processes and lessons learned to external audiences. The NRC Principles of Good Regulation suggest that the NRC has an obvious interest in clarifying its internal understanding of RIDM to best enable the public to understand our decisions and reliably interact with us.The LIC -206 rollout team is collaborating with the staff to host an external workshop on RIDM in the fall of 2022.

7.7 Expansion to other NRR products and business lines Currently, the content of LIC-206, Revision 1 is narrowly tailored to support the rev iew of routine and emergent plant-specific license amendment requests. However, the LIC -206 process contains broadly applicable elements that can be translated to other NRR products and business lines to expand the scope of use. Leveraging risk-insights from PRA early in reviews can help focus attention and resources on the most safety significant aspects of a review. For situations in which PRA may not yield fruitful risk insights, especially those not modeled in PRA, constructing a risk argument approach (as detailed in LIC -206, Revision 1, Appendix C Section 5.0) may help focus a review. In this manner, LIC-206 can provide implementation guidance for several activities consistent with the agencys Be riskSMART initiative.

While the rollout team acknowledges that the guidance in LIC-206, Revision 1 can be applied to many additional work activities, it also recognizes that its precise applicability to these activities could be enriched. Furthermore, LIC -206, Rev ision 1, Appendix C provides two high-level technical approaches to use risk insights but does not clearly articulate the limits of PRA models,

nor does it provide substantive guidance for conditions not modeled in PRA.Refined guidance could be considered for the following work products, which may include updating other OI s or guidance documents to explicitly consider risk insights:

• Other types of license amendments o Power uprates o Improved Standard Technical Specifications conversions o Security, emergency preparedness (EP), quality assurance (QA) plans o National Fire Protection Association (NFPA) Standard 805 Transitions

• Relief requests and proposed alternatives

• License exemptions

• New and Advanced Reactor Applications (see below)

• Environmental reviews (environmental assessments and impact statements)

• License renewals and associated commitment reviews

• Orders/backfits

• Technical Specification Task Force (TSTF) Travelers

• Topical reports

Additionally, the LIC-206 rollout team has engaged the Division of Advanced Reactors and Non-power Production and Utilization Facilities (DANU). While they already have several initiatives either completed or underway to risk -inform the reviews of non-light water reactor submittals (e.g., RG 1.233 provides risk-informed performance-based technology neutral guidance for non-light water reactors and draft Content of Application guidance scales required information to the Page 20 of 21

safety significance of the item), they are considering the need for addi tional guidance on the use of risk-informed decision-making in these reviews. Any such guidance will be coordinated with the RIDM team.

7.8 Development of Externally Facing Guidance (e.g., Regulatory Guides)

The staffs u se of risk insights, especially for Type 2 submittals, relies on the content of the licensees requests and quality of the provided risk insights. Licensees may be hesitant to include risk insights without additional guidance for this new approach. In general, reviews leveraging LIC-206 have received positive feedback and reaped benefits from the technical guidance in LIC-206, Appendix C. Additionally, e xternal audiences have proven to be receptive to NRC presentations of the general LIC-206 approach. Building on this momentum to further refine and expand the LIC-206 approach, both in terms of the IRT process and Appendix C guidance, would continue to increase the value derived from the current work.

To support the engagement with external stakeholder, as described in Section 7.6, clear externally facing guidance should be created as appropriate to convey what RIDM can and cannot achieve, and how best to inc orporate risk insights into applications. The desire for clear guidance has been communicated to the NRC. Another potential option would include industry groups, such as NEI, developing guidance for the staff to endorse.

7.9 Updates to LIC -206 and other policy and guidance documents Internally, LIC-206 could be updated to improve upon Revision 1. For instance, the LIC -206 process could be tiered to match the scale of a review or refined to encompass different types of reviews more naturally.

Regarding potential updates to LIC-206, Appendix C in particular, the expansion of technical reviewer guidance between Rev ision 0 and Revision 1 was a major improvement to the guidance focused on existing interfaces with risk insights derived from PRAs, as well as useful guidance to leverage the risk-triplet for review. Expanding Appendix C to encompass a more robust discussion of the five principles of RIDM discussed in Regulatory Guide 1.174, Revision 3 (ADAMS Accession No. ML17317A256), and how to balance them for a review would provide significant additional clarity and reliability. Explicitly elaborating on how to balance the five principles of RIDM would improve application of these principles in novel reviews and provide an improved framework to address future challenges.

The substantial benefit to expanding staff understanding of the five principles of RIDM is that different discipline areas at the NRC rely with different emphasis on each of the five principles.

Improving cross-pollination between disciplines concerning successful approaches would provide new avenues for innovation. This is particularly pertinent for topics where uncertainties are large.

An additional important update for a future revision of LIC-206 would be to describe its precise relationship to the Be riskSMART framework. Given the increased awareness and use of Be riskSMART internal and external to the NRC, it would be pertinent for a future revision of LIC-206 to recognize it as implementation guidance for the staff to evaluate risk insights consistent with the Be riskSMART framework.

Staff has already started looking at other OI s and guidance documents (e.g., SRP Modernization) to explicitly address the use of risk insights for various staff activities. This should continue and it is expected that LIC-206 and Be riskSMART will continue to be used, as applicable.

7.10 Recommended Next-Phase Team Structure The team recommends the next phase be structured with two teams. The first team would Page 21 of 21

maintain and carry forward the actions and achievements of this current rollout team, namely the recommended actions in Sections 7.1 through 7.5. The rollout team projects that these actions could require approximately a 1 FTE investment, consisting of 3 technical s taff and 1 project manager, preferably from existing LIC-206 rollout team members to maintain consistency and reduce the need for turnover, each spending approximately 1/4 of their time. The second team would be tasked with expanding the use of LIC -206 externally and to other NRR business lines,

namely the recommended actions in Sections 7.6 through 7.9. The rollout team projects that these actions could require approximately a 3 FTE investment, consisting of 12 highly motivated staff representing all divisions in NRR, preferably with a mix of existing rollout team m embership and new members, each spending approximately 1/4 of their time.

8.0 Conclusions The division of the rollout team into three focus areas w ith separate leads and BC champions,

each playing to their strengths, led to the success of LIC -206 by ensuring that dedicated individuals were responsible for those activities deemed necessary to attain the teams goals.

Team A successfully promoted awareness and understanding of the LIC-206 process and engaged with more than 85% of NRR. Team B observed more than 10 IRTs, provide facilitation resources for more than 5 IRTs, and staff evaluated lessons learned to improve the next iteration of the process. Team C built a mechanism to collect data on the use of risk insights and created a dashboard to visualize the new data inputs.

The accomplishments of the LIC-206 rollout team buil t upon the solid foundation laid by the NRR Action Plan, and established by the authors of LIC-206, to frame out and furnish a structure of knowledge and culture needed to formalize enhanced and expanded use of RIDM to the core of NRRs licensing actions. The recommendations provided in this report propose continuing to build upon the success achieved to date with a new pair of task-optimized teams, made up of a mix of legacy members and new members. These teams will be chartered to steadfastly continue to solidify current accomplishments with deeper roots of guidance and richer furnishings of practice, while introducing this enhanced and expanded use of RIDM, along with lessons learned, to other business lines and products in NRR. By leveraging the solid backbone of all that we have accomplished, we can work together to continue building toward NRRs goal of becoming a more modern risk-informed regulator.

9.0 References Office of Nuclear Reactor Regulation, Integrated Risk -Informed Decision-Making for Licensing Reviews, LIC-206, Revision 1, dated June 26, 2020 (ADAMS Accession No. ML19263A645).

Office of Nuclear Reactor Regulation, Integrated Risk -Informed Decision-Making for Licensing Reviews, LIC-206, Revision 0, dated June 6, 2019 (ADAMS Accession No. ML19031C861 ).

Memo from Jason C. Paige to Laura A. Dudes, Findings and Recommendations Report Regarding Office of Nuclear Reactor Regulation Risk -Informed Decision-Making Action Plan Tasks 1, 2, 3, and 4 (CAC No. A11008), dated June 29, 2017, (ADAMS Accession No. ML18169A205).

U.S. Nuclear Regulatory Commission, Regulatory Guide 1.174, Revision 3, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant -Specific Changes to the Licensing Basis, dated January 2018 (ADAMS Accession No. ML17317A256 ). Page 1 of 2

ENCLOSURE 2 LIC-206, Revision 1 Rollout Team

Working Group Composition and Counterparts Page 2 of 2

Working Group Oversight

RIDM Counterparts

• Jane Marshall, NMSS, Formerly DEX (SES Champion)

• Meena Khanna, DRA

• Brian Smith, DANU

• Bob Caldwell, DNRL (retired)

• MJ Ross-Lee, DSS

• Gregory Suber, DORL

DORL Coordination

• Gregory Suber and Caroline Carusone, DORL (SES Counterparts)

• Jim Hickey, DORL (SLS Advisor)

• Mike Markley and Ed Miller, DORL (BC Liaison and Sr. Project Manager, respectively )

• Kimyata Morgan-Butler, Dave Wrona, Undine Shoop, DORL (BC Counterparts )

Team A Working Group Members

• Angela Buford, DNRL (BC Champ ion)

• Ian Tseng, DEX (Tech Lead)

• Richard Rivera, DANU (PM)

• Alissa Neuhausen, DRA

• Shilp Vasavada, DRA

• Dan Widrevitz, DNRL

• Milton Valentin, DRA

• Khadijah West, DSS Team A Contributors

• Robert Kuntz, DORL

• Fred Forsaty, DSS

Team B Working Group Members

• Stacey Rosenberg, DRA (BC Champion)

• Dan Widrevitz, DNRL (Tech Lead)

• Richard Rivera, DANU ( PM)

• Ian Tseng, DEX

• Alissa Neuhausen - DRA

• Shilp Vasavada, DRA

• Milton Valentin, DRA

• Nicholas Hansing, NMSS, Formerly DEX

• Andy Hon, DORL

Team C Working Group Members

• Angela Buford, DNRL (BC Champ ion after January 2021)

• Jeanne Johnston, DEX (BC Champion until January 2021)

• Alissa Neuhausen, DRA ( Tech Lead)

• Richard Rivera, DANU (PM)

• Ian Tseng, DEX

• Milton Valentin, DRA

• Samson Lee, DORL

• Andrew Hon, DORL Page 1 of 9

ENCLOSURE 3 LIC-206, Revision 1 Rollout Team

Team A Promotional Materials Page 2 of 9

Page 3 of 9

Page 4 of 9

Page 5 of 9

Page 6 of 9

Page 7 of 9

Page 8 of 9

Page 9 of 9 Page 1 of 6

ENCLOSURE 4 LIC-206, Revision 1 Rollout Team

Team A DORL Focus Group Session Summary Report Page 2 of 6

Summary Report of Observations from the February 2021 DORL Focus Groups

1. Introduction

This report presents the results of a series of focus group discussions facilitated by the LIC -206 Rollout Team. After issuance of LIC-206, Revision 1, the team has explored ways to ensure that the guidance is understood and used consistently across NRR to support the agencys transformation initiatives in becoming a more modern risk -informed regulator. The focus group discussions were among these activities. Also, this is a subsequent activity from the Risk -

Informed Decision-making (RIDM) Project Phase Reports (ADAMS Accession Package ML19007A339).

2. Purpose of Focus Groups

The LIC-206 rollout team facilitated these two-way discussions to grasp the current understanding and practice of RIDM among DORL project managers. This understanding would help the rollout team identify areas where attention might be needed. Also, the r ollout team sought feedback to improve the guidance in Revision 1 of LIC -206 to help optimize our resources and to improve our communications.

In addition, the rollout team saw this as an opportunity to share perspectives and help project managers get up to speed with the latest updates in Revision 1 for common understanding.

Here are some of the perspectives that were shared from the focus groups:

• The agency is transforming in becoming a more modern risk-informed regulator. The world out there is changing rapidly and is using risk insights; we must keep up with it.

• Risk has been around for a long time and there is enough information and understanding to help us inform our licensing reviews consistently. LIC -206 was created for that same reason.

• Connecting the dots within risk: RIDM is a Commission philosophy behind the Be riskS MART framework transformational initiative we keep hearing about for agency transformation. At the office level, LIC -206 is one of many activities supporting transformation. The use of the Integrated Review Teams (or IRTs) is a process within LIC-206, where different disciplines work together to inform a safety finding.

3. Participating Branches

The branches who participated in the focus group discussions were LPL1, LPL2-1, LPL2-2, LPL3, LPL4, and LLPB. There was no focus group discussion with LPMB because the branch doesnt conduct licensing work. However, LPMB has interacted with the rollout team regarding upgrades to RPS. A total of 35 project managers participated in the focus group discussions.

The focus group discussions with the different branches did not include the branch chiefs. A separate focus group discussion was held just with the branch chiefs ; 5 branch chiefs joined the Page 3 of 6

discussion.

4. General Results of NRR Polling Questions

To obtain the most information possible and allow time to hear from the participants, the focus group discussions included poll (yes or no) questions and discussion questions. The foll owing bullets include the poll questions and responses from the project managers who participated in the focus group discussions (data per branch included in the enclosure):

PQ 1. Have you used an IRT to support a licensing review?

Yes: 19 No: 16

PQ 2. Are you aware of the latest revision of LIC -206?

Yes: 17 No: 18

PQ 3. Have you read the latest revision of LIC -206?

Yes: 9 No: 26

PQ 4. Have you seen a project positively impacted by IRTs?

Yes: 15 No: 19 (one participant chose not to respond)

PQ 5. Have you seen a project negatively impacted by IRTs?

Yes: 6 No: 28 (one participant chose not to respond)

In summary, out of the project managers who attended the focus groups:

• About half of them knew about LIC -206 Rev 1.

• About a third of them have seen (and read) Rev 1.

• Most participants havent seen an IRT negatively impact a project.

Here the poll questions and answers from branch chiefs:

PQ 1. Has your staff used an IRT to support a licensing review?

Yes: 3 No: 2

PQ 2. Are you aware of the latest revision of LIC -206?

Yes: 4 No: 1

PQ 3. Have you read the latest revision of LIC -206?

Yes: 2 No: 3

PQ 4. Have you seen a project positively impacted by IRTs?

Yes: 2 No: 3

PQ 5. Have you seen a project negatively impacted by IRTs? Page 4 of 6

Yes: 2 No: 3

5. Feedback Received from Discussion

This section summarizes the questions for discussion and feedback gathered from the project managers in common themes (supporting data provided in the enclosure):

DQ 1. What changes would you suggest to increase your use of RIDM (and IRTs)?

Common Themes Times mentioned Dont force IRTs for no reason 7 Confusion or lack of clarity 14 Risk means additional work 2 Other 6

DQ 2. Any suggestions for identifying risk-informed submittals in RPS?

Common Themes Times mentioned Dont make RPS more complicated 6 Define Types 1, 2, 3 risk submittals in RPS 3 Explain qualitative and low risk significance in RPS 3 Other 4

Feedback from the branch chiefs (all feedback included here):

DQ1. What changes would you suggest to improve and/or increase your staffs use of RIDM (and IRTs)?

• Consider incorporating this concept in the SRP modernization effort as staff guidance for reviews.

• Share information on case studies where risk -information and/or IRTs have been used.

• Show examples of reviews that have been more streamlined using risk-information. Hard to see the benefit in using risk -informed approaches without seeing examples of success cases.

• Staff continues to question why is there a push to use IRTs in cases when it is not needed or there is no apparent benefit for it. DORL metrics on the use of RIDM call for an increase in the use of these approaches.

DQ2. Any suggestions to identify and manage risk -informed submittals that your staff is working on? (Considering all risk types)

Page 5 of 6

• There is a mindset among some of the staff that believe that the staff should only consider the information included in the submittal with regards to risk insights. It is not clear how the staff should bring in risk insights to perform the review.

• Engineering judgement still needs to support what the licensee says in the submittal. How can the staff use risk information to support this, when not provided by the licensee? We need to make sure that decisions are not made based on risk alone (not enforceabl e).

• Sometimes we are trying to force the use of IRTs in risk informed reviews, when they are not entirely necessary.

• How does fee billing fit in this effort of considering risk-insights ? (The use of PRA / risk-insights is voluntary for the licensees).

• We need to acknowledge that there are some things that are not covered by risk.

• Action for LIC-206: How do we communicate the use of risk -insights ?

6. Other General Feedback/Observations

Provided below are salient feedback and observations not gr ouped in common themes:

• Expand LIC-206 to consider Topical Reports.

• Consider defining more clearly in RPS if risk analysts should review (or close) each milestone (e.g., RAIs, SE input).

• Consider efficiencies from IRTs to improve ILRTs.

• Create a Sharepoint for LIC -206 resources and guidance, add the link in DORLs Sharepoint.

• Need more examples and case studies to assess effectiveness.

7. Proposed Action Items

For LIC-206 Working Group:

• Provide examples/case studies to prove benefits and level of effort.

• Improve LIC-206 descriptions of roles and responsibilities, especially the involvement of DRA staff and the role of the PM in the technical review (Appendix C).

• Further clarify involvement and level of effort for each stakeholder and potential cost to the project.

• Provide examples of different risk informed submittals and respective types.

• Clarify applicability of LIC-206 and RIDM to topical reports.

• Work with LBMB to add definitions of Types 1, 2, 3 and qualitative/low risk in RPS.

• Facilitate similar discussions in other divisions to help improve the general understanding on LIC-206. Page 6 of 6

For DORL Management:

• Embrace a more proactive use of risk insights, the current practice seems more reactive than proactive (we are limited to what licensees submit).

• Explain why IRTs are being requested, better selling points are needed.

• Consider pursuing earlier communication with licensees to know their intent in using risk insights and to ensure the right information is provided.

For DRA Management:

• Consistent with DRAs plans to conduct a workshop/refresher training to staff to address some of the feedback contained in this report and any follow -up activities to promote RIDM.

Please send any questions about this report to RIDM.Resource@nrc.gov Page 1 of 3

ENCLOSURE 5 LIC-206, Revision 1 Rollout Team

Team B Observation Template Page 2 of 3

IRT Observation and/or FacilitationRecord

Table 1 - Observation Metrics Initial No. of Use IRT # of Risk Observation Submittal Type or Subsequent Meeting Checklist? Branches at Analyst? Date(s)

Meeting Meeting

[1, 2, 3] [I/S] [1, 2, etc.] [Y/N] [1, 2, etc.] [Y/N]

Meeting

Title:

Plant Name, Brief Identifier for Topic, EPID, identifier if subsequent observed meeting of team

Observers and/or Facilitators:

RIDM Members Observing and/or Facilitating

Meeting Basis and

Description:

Basis for and description of meeting (IRT Initiation, IRT Working Group, etc.)

Basis:

==

Description:==

Attendance:

Active parties in meeting (Primary reviewer branch/Division, DRA, etc.)

Risk Information and Insights Discussed at Meeting(s):

What risk information was discussed? If IRT Checklist is being used, how?

Interactions among Disciplines:

Observations on inter-specialty collaboration

Outcome of Meeting:

Outcomes of the interactions in meeting?

Page 3 of 3

If this is an initial IRT:

On what basis was IRT convened:

Describe what led to an IRT being forme d

Will IRT be continued:

Did group conclude IRT was warranted, and if so, (a) on what basis and (b) to accomplish what goals?

Basis (a):

Goals (b):

If this is a late/closing IRT:

What sources of risk information were relied on:

Applicant submitted, internal pre-generated, PRA results; qualitative, quantitative, etc.

What, if any, impact did IRT or RIDM have on review:

Scope of review adjustment, applicant adjustments, etc

Recommendations:

What can be improved for other IRTs and what should be sustained (i.e., went well)?

Page 1 of 3

ENCLOSURE 6 LIC-206, Revision 1 Rollout Team

Team B Exit Survey Results Page 2 of 3

Page 3 of 3

Additional comments provided by survey respondents:

The IRT process is a great way to improve efficiency and collaboration in the office. I would suggest that IRT meetings should be held more frequently to further avoid the silo effect. Additionally, the project manager should take a more active role in establishing roles and responsibilities for the project up front to ensure that all team members are able to reap the benefits of the process.

I was involved with a LAR categorized as Type 2. From my perspective, I thought the review effort did benefit from an integrated review (in my experience this is normal) but I am not sure it was due to the IRT process.

The IRT process is a great initiative. However, it could be enhanced through the use of additional IRT meetings (possibly routine). Additionally, while my exposure to the process has been limited, it is not apparent that the technical reviewers nor the risk analysts clearly understand their role and the potential benefits associated with the IRT (e.g., collaboration, refined review scopes). This could be remedied through additional discussions with Project Managers who should articulate roles and responsibilities early in the project.

Some of the responsibilities outlined in LIC-206 are too subjective, like: - being familiar with pertinent elements of risk-informed and performance -based regulation; -appropriately engaging a risk analyst early for risk insights in licensing activities. These descriptive words are fuzzy and subjective and don't add real value to the document. How familiar is familiar? Who defines what is appropriate? It would seem to make more sense to have clear actions laid out for individuals. Since LARs always (in my experience anyway) arrive through DORL, the PM should be taskedto inspect the LAR for 50.69 references and/or risk-informed information in the LAR. Of course,the tech reviewers should also perform the same review, as they might note something that the PM does not. If noted, either could start the LIC-206 process.

Some tech reviewers are not comfortable with or ignore the "consolidated" SE format - not sure if it's a discomfort with using IT or the idea of starting the SE early (or both). Agency needs to enforce early SE development.

As it was an expedited review, the integrated review team let the review clarify responsibilities and resolve interfaces in a timely manner. More resources were used since more coordination was necessary. Also,the IRT process has the PM reviewing input before the branch chief. This means the PM may be reviewing material the BC will remove or significantly change. Page 1 of 3

ENCLOSURE 7 LIC-206, Revision 1 Rollout Team

Team C RPS Changes Page 2 of 3

Implemented May 2021 RIDM enhancement to RPS Status (Technical Reviewer)

The RPS Request for Additional Information (RAI ) and Safety Evaluation (SE) milestones have new dropdown options as described in the one-pager below. The information collected by these options will allow the agency to understand the staffs use of risk information in licensing reviews on our path to becoming a risk informed regulator. These options are one way for staff to convey that risk is being used consistent with Commission direction.

Page 3 of 3

Implementation Date TBD For the Complete (Risk Applied) selection in the SE milestone in RPS, the technical reviewer will be asked what types of risk information or insights were considered during the review. More than one option may be selected. The information collected by these options will allow the agency to understand the staffs use of risk information in licensing reviews and its effect on workload management on our path to becoming a risk -informed regulator. This information will help staff to convey that risk is being used consistent with Commission direction.

Source of Risk Insights Explanation Be riskSMART The technical reviewer used the Be riskSMART framework.

NUREG/KM -0016 outlines how various guidance and processes incorporate elements of the six steps of the framework.

Risk Triplet The technical reviewer considered the risk triplet to inform the review. SECY-98-144 provides additional information on the risk triplet.

Risk Informed Process for The technical reviewer/project team used RIPE. Additional Evaluations (RIPE) information on the process is available at RIPE_EMBARK.

LIC-206 Appendix C The technical reviewer/integrated review team walked through Appendix C to understand how risk insights may support the review. See the LIC-206 OI.

SPAR Models Risk insights used in the review were developed using SPAR models. It is expected that if SPAR models are selected as a source of risk insights that DRA/Staff-provided will also be selected. SPAR models are run using SAPHIRE.

Plant Risk Information The technical reviewer used risk insights found in the PRIBs.

Book (PRIB) PRIBs and PRIB resources are available.

Probabilistic Risk Risk insights were derived from an acceptable PRA, as Assessment determined by DRA. If this option is selected, it is expected that Licensee-Provided will also be selected. For example, this option would be selected for a TSTF-505 application.

Other E.g., Operating Experience (OpE) Risk, Engineering Judgement.

Licensee-Provided The submittal included risk insights. The risk insights may or may not be based on an acceptable PRA.

Risk Analyst/Staff-The staff developed risk insights independent of the licensee.

Provided This may include a confirmation of licensee-provided risk insights. These insights may or may not be provided by DRA.

N/A The staff used risk insights, but none of the above are applicable.

Page 1 of 2

ENCLOSURE 8 LIC-206, Revision 1 Rollout Team

Team C Dashboard Page 2 of 2

The first version of the LIC-206 dashboard includes four views :

1. RAI Milestone - Risk Type

2. RAI Milestone - Risk Usage

3. SE Milestone - Risk Type

4. SE Milestone - Risk Usage

The first image below indicates how the data is presented for Risk Type and the second image below indicates how the data is presented for Risk Usage. Both views convey the exact same information, but allow the user to interpret the information more easily without having to do additional data manipulation.

For the Risk Type display, data is organized first by the total number of Type 1, 2, and 3 applications (top left) submitted over a specified time (bottom). The risk type is further broken down into deterministic and risk applied (top right).

For the Risk Usage display, data is organized first by the total number of projects with deterministic and risk applied selecte d for the milestone (top left) over time ( bottom). The risk usage is then further broken down into Type 1, 2, and 3 applications (top right).

Pkg MLXXX NRR-106 OFFICE NRR/DEX/DDD NRR/DSS/DDD NRR/DNRL/DDD NRR/DANU/DDD NAME SRuffin MRoss-Lee BThompson JBowen DATE 03/18/2022 03/29/2022 03/29/2022 03/29/2022 OFFICE NRR/DORL/DDD NRR/DRA/DDD NMSS/DUWP/DD NAME GSuber MKhanna JMarshall DATE 03/30/2022 03/29/2022 03/03/2022