ML22122A002

Regulatory Information Conference (RIC) 2022 - Technical Session - Be Risksmart - W16-Transcript

ML22122A002
Person / Time
Issue date:	03/09/2022
From:	NRC/OCFO
To:
References
Download: ML22122A002 (73)

Text

1 UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION

+ + + + +

34TH REGULATORY INFORMATION CONFERENCE (RIC)

+ + + + +

TECHNICAL SESSION - W16 Be riskSMART: REAL EXAMPLES AND REAL IMPACTS

+ + + + +

WEDNESDAY, MARCH 9, 2022

+ + + + +

The Technical Session met via Video-Teleconference, at 3:00 p.m. EST, Ben Ficks, Deputy Chief Financial Officer, Office of the Chief Financial Officer, Nuclear Regulatory Commission, presiding.

PRESENT:

BEN FICKS, Session Chair, Deputy Chief Financial Officer, Office of the Chief Financial Officer, Nuclear Regulatory Commission REED ANZALONE, Reactor Safety Program Specialist, Embark Venture Studio, Office of New Reactor Regulation, Nuclear Regulatory Commission BILLY DICKSON, Chief, Division of Reactor Projects NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

2 Branch 2, Region III, Nuclear Regulatory Commission STACY PRASAD, Security Risk Analyst, Security Oversight and Support Branch, Division of Security Operations, Nuclear Security and Incident Response, Nuclear Regulatory Commission MIRABELLE SHOEMAKER, International Safeguards Analyst, Material Control and Accounting Branch, Division of Fuel Management, Nuclear Material Safety and Safeguards, Nuclear Regulatory Commission NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

1 P R O C E E D I N G S (3:00 p.m.)

MR. FISK: As the Chief Financial Officer at the US NRC, I'm very happy to welcome you to this virtual RIC session on Be riskSMART. Real examples.

Real impact. I have the honor to serve as the Chair of this panel.

As one of the three original executive sponsors for the Be riskSMART Initiative, representing the corporate perspective, it's wonderful for me to see the significant amount of progress we've gained over the last two years. We developed the Be riskSMART framework and issued guidance last March that is publicly available.

We also developed Be riskSMART training for NRC staff, which over half of the NRC staff have completed. We also have numerous success stories across the agency of staff applying the Be riskSMART framework, and we're excited to share some of those with you today.

Please submit your questions as we go along, and we will answer as many as possible after our panelist presentations. We'll be asking you two poll questions, so as they pop up, please answer those and we can share the results as we have them with NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

2 you.

Please click on the word poll on the right side of the screen. We actually have our first question, which is, "Have you heard about the NRC's Be riskSMART framework decision-making?" We will give you a moment to answer that question, and then we'll share the results.

So I think it just takes a second here for it to come up. So I think you for your patience.

So it looks like 63 percent of you have heard about the NRC's Be riskSMART framework and 39 percent no.

All right. Well, that helps us as we go in for our panel, so I appreciate that. So Slide 2, just take a second for that to come up. Okay, so we have a wonderful panel assembled of NRC staff members.

Reed Anzalone will describe the Be riskSMART framework and share reactor examples. Mr.

Anzalone is a data scientist for the Embark Venture Studio in the NRC's Office of Nuclear Reactor Regulation. He's the Product's Manager for the NRC's Mission Analytics Portal, a suite of data analytics tools designed to facilitate access to data and enhance data-driven decision-making at the NRC.

He's also a member of the Be riskSMART core team. Mr. Anzalone holds a bachelor's degree NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

3 in mechanical engineering from the Virginia Tech University and also a master's degree in Mechanical Engineering from the University of Texas at Austin.

He has recently gotten into 3D printing and is known for having a dry sense of humor.

We also have Mirabelle Shoemaker, International Safeguards Analyst in our Office of Nuclear Material Safety and Safeguards, who will be sharing materials program examples. Ms. Shoemaker began supporting the agency's Be riskSMART team in 2020 due to her curiosity about how to apply the framework to her niche area, International Safeguards in Material Control and Accounting for special nuclear material.

She has become a Be riskSMART ambassador for NMSS, supporting multiple working groups as they apply the framework to activities across the NMSS business lines. She's a notorious house plant killer but is a proud parent to three thriving children ages six, three, and one. She has a general BA/BS from George Washington University in international affairs and political science, and she has an MA in international affairs from Catholic University.

We also have Billy Dickson, Branch Chief in Region III. He will be sharing inspection NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

4 examples. Mr. Dickson joined the NRC in 1996. He started his NRC career as a Reactor Engineer in Region III, and he has held positions as a Resident Inspector at Dresden, Senior Resident at both Clinton Power Station and Braidwood Nuclear Station, and Branch Chief at Health Physics and Incident Response Branch in the Division of Reactor Safety in Region III. Mr.

Dickson received a BS degree in nuclear engineering from Mississippi State University, and he loves fishing.

Rounding out our panel is Stacy Prasad, Security Risk Analyst, Nuclear Security and Incident Response, who will be sharing security examples. Ms.

Prasad was technically trained as a computer and electrical engineer at Drexel University before beginning her career at the NRC in 2003.

She's worked in the regional office supporting the resident inspector program and regional inspections before moving to NRC headquarters to support Electrical Inspections for vendors supplying components to operating reactors, aircraft impact inspections, and in other divisions of NRC's enforcement program. And at the insert (phonetic), she became one of the ambassadors for the Be riskSMART initiative. Stacy indicated that she's NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

5 using her three extra hours a day that she has saved by not commuting during the pandemic by picking up the violin again. Her husband is grateful to the Be riskSMART framework, as he was able to utilize the framework to promote a different, quieter alternative to practicing a musical instrument by determining the risks to his hearing was not justified.

At this point, I'll turn it over to Reed.

MR. ANZALONE: Thanks, Ben. So if we can go to the next slide. So I'll be walking you all through the Be riskSMART framework. And before I get started with that, I want to acknowledge that Be riskSMART is kind of a cutesy name, but you shouldn't be fooled by that. It's a robust framework for integrated risk-informed decision-making.

So next, please. The first step is to be clear about the, problem. This step frames the scope of what you're trying to address, which helps you a lot down the line as you begin to identify risks and how to manage them. And we actually found that without this step, it's really difficult to make sure you're appropriately constraining the scope of the decision that you're trying to make.

Next. Then you spot what can go wrong or right. What are the consequences and how likely NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

6 is it? And for those of you who might be familiar with probabilistic risk assessment, this is essentially just identifying elements of the risk triplet. Risk is likely to -- with just a couple of minor differences from how we normally do it.

The first is that we want to explicitly consider benefits in the decision, because those are an important factor. And as we work through the framework, we found often times that it was easier to consider consequences before probability, but that's not a hard and fast rule.

Next, please. Then you manage what you can. This step gives you a space to think about the risks and benefits that you can affect, and how you would manage them to attenuate the risk and extenuate the benefits.

Next. The next step is to act on the decision. Once you've got all the risks and management strategies figured out, you make a decision while considering your risk appetite for this particular decision, and that can vary depending on the kind of decision that you're trying to make.

If it's a decision that's just for you, it might be your own personal risk appetite. If it's a decision for your organization, you might need to factor in NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

7 your organizational risk appetite.

Next. The next step is to realize the result. And this seems to kind of trip some people up, but really, it's just the step where you take the decision that you've made, and you go forth and implement it.

And then while you do that, you manage what you need to make the risks accessible or the benefits more appealing, and you monitor your performance and progress as you go through.

Next. The next step is to teach others what you learned. And this step is really important for us at the NRC. They create a record of our decisions. But it's also really important for knowledge management and to help spread the word of how Be riskSMART helped in the decision-making process.

And finally, next. So at the bottom of our logo there is the little arrow underneath, and that emphasizes the iterative nature of the process.

As you make one decision, you may need to roll right into the next one. Or in some cases, if the conditions that run into your Be riskSMART evaluation change, you might want to take another look at it.

So that's the process. And now I'm going to go NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

8 through some examples from the reactor area. The next slide please.

So the first impact that I'm going to talk about for what that Be riskSMART assessment has in the reactor area is making decisions in the presence of uncertainty. So the Be riskSMART helps you identify and monitor conditions that are uncertain so you can manage them appropriately before they become an issue, particularly using the manage what you can and realize the result step of the framework.

And in general, having a structured framework like Be riskSMART, it helps to be confident that you're adequately considering what you need to, even when there's a lot of uncertainty about what could happen or how likely it is.

The first two things I'm going talk through on this slide is the local 10 CFR 52.103(g) findings. So those of you who might not be aware, Vogtle Units 3 and 4 are Westinghouse AP1000 reactors that are under construction in Georgia. And these plants can't officially operate until the NRC makes a finding under 10 CFR 52.103(g) that all the acceptance criteria have been met. And there are a lot of inspections, tests, analyses, and acceptance NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

9 criteria leading up to that finding. Excuse me.

It seems there are a wide variety of scenarios that might occur in the late stage of the construction process that could delay an otherwise timely regulatory findings if we're not prepared for them. The statutes of the Be riskSMART framework to identify all the possible risks and outcomes and put in place a plan to manage those risks where necessary.

This has included training, tabletop exercises, and new office instructions to provide support to the staff.

And one element that we put in place is shows on the screen, and that's a dashboard to help us track completion of activities related to Vogtle 3 and 4. And while we haven't made a finding under 52.103(g) yet, we are confident that we will be able to handle whatever comes our way, because we've thought it through.

Another example of how Be riskSMART helps us make decisions in the presence of uncertainty is related to CRDM funnel issue that happened at a pressurized water reactor. So during the refueling outage, there was a pressurized water reactor licensee that identified that a CRDM funnel had detached - control rod drive mechanism, I'm sorry.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

10 And the NRC had to figure out the safety significance of this issue and what level of oversight was appropriate as the licensee worked to repair the problem.

A lot could potentially go wrong if the control logs don't work correctly, but the NRC staff evaluated the issue using the Be riskSMART framework and found that the risks if the control logs weren't inserted correctly was managed appropriately during startup with multiple tests.

Risks of foreign material associated with the funnel repair were managed through the licensee foreign material exclusion program. The NRC inspectors at the site decided to focus on the foreign material exclusion program and only nominally inspect the other aspects of the event because they were routine at the site. There were no issues at startup, and we were able to minimize the use of NRC resources and focus only the most safety-significant issue.

And one final example for this slide is related to Regulatory Guide 1.99, which provides the model for radiation embrittlement of reactor vessels.

So there is some indication that this model may be non-conservative relative to actual test data, and NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

11 high neutron fluences that would occur at extended plant lifetime since the data is limited because we don't have a lot of operating experience in that regime.

A model that's been developed by the international community appears like it represents the high fluence data better than Reg Guide 1.99, but again, the data is limited. Staff considered all possible outcomes that could result from a non-conservative Reg Guide 1.99 model, and staff also did a probabilistic fracture mechanics evaluation but found the risk of brittle failure to be low, even if the model is non-conservative.

They factored this into a Be riskSMART evaluation that included all of these considerations and found that the model was adequate for now, but we're going to continue monitoring material samples collected under 10 CFR Appendix H and operating experience related to pressure vessel embrittlement to ensure that the model continues to be adequate.

All right. Next slide. So the second impact is making use of all of the available evidence to make a decision. And this has been alluded to a couple of times in different plenary sessions. I know the Commissioner -- or the Chairman is very keyed NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

12 into making sure that we are using data and evidence needed to make decisions. And Be riskSMART is one way that we can help bring all of the available information to bear, including risks, into our decision.

So the example I'm going to talk about on this is about operator licensing for Vogtle Unit 4.

And as I mentioned, that's one of the units that under construction down in Georgia. Typically, for licensed operators, comparable units are added to the operator's license using the waiver process under 10 CFR 55.47 to avoid the need for additional examination.

However, this waiver requires the operator to have extensive experience on the original unit. And for operators like with Vogtle 3, the waiver process for Vogtle 4 is complicated by the fact that Vogtle 3 is still under construction.

The staff evaluated this issue using Be riskSMART and found that there were both legal and enterprise risks with attempting to use the regular provision in sight of these challenges. However, the staff also didn't think it would be necessary or efficient to have a licensed operator take another exam for Vogtle Unit 4. So what to do?

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

13 Using the risks and benefits uncovered by Be riskSMART, staff decided that exemptions would be both more appropriate and more legally defensible than using waivers. Staff also decided that in the long term, rule-making was needed to address this issue going forward for other multiunit sites under construction.

Now I'm going to turn it over to Mirabelle to talk about some examples from the materials area.

MS. SHOEMAKER: Thanks, Reed. So something I've experienced firsthand when using the Be riskSMART framework is discovering access to the greater toolbox for risk assessment and risk management and decision-making. Being able to identify the entire applicable toolbox for risk-informed decision-making is especially important in an office like NMSS, which works across multiple business lines and disciplines. It's also incredibly valuable for decision-making that considers qualitative information.

Whether I worked in a matrix team or within my own working unit, Be riskSMART has enabled our team to identify guidance and tools that will aid in decision-making at all levels. So as the first NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

14 bullet states, when using Be riskSMART, staff doesn't need to reinvent the wheel. Instead, the framework provides a common approach to identify and assess risk across various disciplines.

A good example of this is when NMSS was faced with two activities, a rulemaking discontinuation and a pilot program for graded approach. Staff faced the question of whether they could discontinue the rulemaking before completing the pilot and determining it was successful. So this became the problem statement for both working groups and by using the framework in parallel, the staff identified that there could be risks related to public confidence and effective regulation if the activities were not sequenced appropriately.

So using the Be riskSMART framework, the staff spotted these risks and identified what could go right or wrong for both rulemaking and licensing activities. And in using this common approach to spot, manage, and then act on risk insights, staff determined that publishing the rulemaking discontinuation before completing the pilot would likely produce concerns of public perception that the pilot was not fully considered before the rulemaking discontinuation.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

15 So staff assessed that waiting to publish the rulemaking after the pilot would allow the pilot to reach a level of maturity that would give enough information to demonstrate a risk-informed approach and uphold principles of good regulation. So in this example, we sequence both packages for concurrence.

When the pilot on the grade approach was confirmed with no significant adverse comments received, the rulemaking discontinuation package could reference the completion of the pilot. So in this example, two groups responsible for different disciplines were able to identify and manage risks by coordinating their efforts through Be riskSMART.

Now there are time where disciplines can merge, and a matrix team is needed to support NMSS activities. Contracting renewals, for example, require coordination with corporate and technical staff before we can renew a contract. In these cases where matrix teams are assembled, team members come the table focused on their expertise areas equipped with the tools that allow them to identify and assess risk.

As the second bullet states, Be riskSMART is beneficial to matrix teams because when the team has access to the full toolbox for risk-informed NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

16 decision-making, they can have a coordinated and systematic approach when considering multiple risk factors with varying uncertainties.

So consider a licensing for a fuel facility. The team reviewing the application includes safety and security experts, environmental technical reviewers, legal staff, and of course management. And Be riskSMART framework supports a collaborative approach to identifying the problem statement and spotting risk and balancing those considerations in multiple risk areas.

During a recent license renewal for a fuel facility, staff conducted the environmental review in accordance with the NEPA requirements.

They published a draft environmental assessment and a finding of no significant impact for public review and comment.

And during the comment period, new information arose and produced public concern over groundwater contamination and other potential hazards in the area. So staff swiftly applied the framework to the problem statement, how does this new information affect the current environmental assessment and the finding of no significant impact?

And should the NRC change its review to an NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

17 environmental impact statement considering this new information? So by using the framework, the staff balanced risks related to safety and public perception.

In this example, they took a more conservative approach by proceeding with an environmental impact statement. This is the first time an environmental review for a fuel facility resulted not in an EA FONSI decision. So management was incorporated in the consideration well in advance through use of the Be riskSMART framework. It was critical for communicating the staff's risk assessment and management plan and to avoid significant delays to the project schedule by commencing the EIS as soon as possible.

So to summarize, the key messages on this slide, Be riskSMART framework serves like an umbrella. It houses all the available risk tools under it. One division of NMSS developed an infographic that I included on this slide that articulates the steps of decision-making, highlighting Be riskSMART at the beginning. The elements of Be riskSMART are conducive for early and comprehensive identification of risks and a coordinated application of quantitative or NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

18 qualitative risk assessment tools. And the goal is communicating a decision, achieving alignment, and acting on that decision.

Next slide. Now I alluded to a benefit of using Be riskSMART. I want to dive into that a little bit more with examples on this slide. So using Be riskSMART, staff is able to achieve early alignment on risks with decision-makers.

Specifically, early alignment on potential risks allow the NRC to develop a management plan in environmental reviews.

I mentioned a license renewal for a fuel facility a moment ago where alignment in action was needed before staff could proceed with an environmental impact statement. Early alignment enables earlier action, whether that's allocating resources or developing a communication plan. And earlier action can reduce impacts to the project timeline.

So let me highlight another example. The Be riskSMART approach has been used pretty faithfully in environmental reviews within NMSS, especially when issues involved public stakeholder engagement. So Be riskSMART allows staff to incorporate public perception risks alongside safety and security NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

19 issues. And a recent example is when Be riskSMART was applied when considering whether the environmental impact statement comment period should be extended based on public request.

So in accordance with NEPA, NMSS prepared an EIS and the typical comment period lasts about 45 to 60 days. Public stakeholders can submit requests to extend the public comment period and there's no limit on the number of requests that may be submitted or the length of time for the extension.

Be riskSMART has been helpful in guiding these environmental changes through consideration of these requests. The framework gives weight to the value of public participation in environmental reviews. And in the spotting and managing steps, teams will consider potential risks if public participation is not maximized, or doubt arises in public confidence. Using Be riskSMART, the staff has been able to efficiently disposition requests for public comment period extensions to minimize impacts to project timelines while supporting full public participation.

We also saw the benefits of early alignment very beneficial with the onset of the COVID-19 public health emergency. Identifying risks NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

20 early allowed the NRC to focus on risk significance during a time when we were in mandatory telework.

I'd like to highlight a specific example before turning it to my colleague, Billy, who will give another example of Be riskSMART during COVID-19.

So during the public health emergency, when we were in mandatory telework like the rest of the government, we were faced with a challenge in continuing onsite inspections at facilities.

Specifically, we had plans for an onsite inspection and independent spent fuel storage installation site.

And the purpose of the inspection was to observe the specs for loading campaign.

This is the inspections of this nature are conducted every two years, and this particular site was due for completion in 2021. So correspondingly, it would result in a missed opportunity to evaluate cask loading, which is the most risk-significant activity at an ISFSI.

Understandably, this problem was of interest across management and office levels. It was a prime opportunity to deploy Be riskSMART and gain alignment early, especially with the small window of opportunity to observe a cask loading.

So following the elements Be riskSMART, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

21 the staff identified and weighed the risks of what could go right or wrong in conducting this inspection. And then the likelihood and then the balancing of those considerations of safety in the traditional sense assist the operation as well as the public health and safety considerations in the pandemic.

The decision-makers agreed with the recommendation for remote inspection with support from the licensee cameras and remote monitoring to allow us to observe cast loading in real time. A direct line of communication was maintained with the site and remote administrative reviews facilitated observing the loading campaign. And the result that was realized was early alignment, enabled efficiency, and decision-making during a narrow window of opportunity.

It also produced a secondary benefit of developing innovated solutions to deploy remote monitoring, which resulted longer observations of multiple simultaneously occurring activities with reduced accumulated dosing inspectors.

My colleague in the region has personal experience using Be riskSMART during COVID-19. So thank you for your time, and Billy, I look forward to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

22 hearing your example next.

MR. DICKSON: Thank you. So use of the Be riskSMART process has allowed regional organizations to identify, evaluate, and treat risks to different degrees of formality. In some cases, it's helped to limit scope of problems to individual branches, divisions, facilities. In other, it's helped expand solutions to the enterprise level.

It's provided means for evaluating, assessing, diminishing risks whether it's regulatory risk, operational risks, or reputational risks in areas where traditional standards have been established but don't explicitly address these types of risks.

One of the more significant impacts of the use of the Be riskSMART process allow for adaptation in use of new and existing technology.

Specifically, during the onset and just prior to the declaration of a national health emergency, when there were several unknowns and uncertainties, it was essential that the inspection staff was protected against this threat while maintaining the agency's essential need to have situational awareness.

The Be riskSMART methodology was used to evaluate the feasibility of using licensee-provided NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

23 computer equipment offsite to substitute for the presence of the resident inspector staff for specific inspection activities. The Be riskSMART process was also used at the inspection program implementation level to find acceptable methods to performed key ROP inspections such as the problem identification and resolution inspection and the cybersecurity inspections remotely, or partly remotely.

This process has resulted in organizations being more strategic about to manage and build with the unexpected, resulting in a more agile organization.

Next slide please. So reasonable organizations utilize the Be riskSMART process to provide for greater flexibility and the ability to adapt to change in conditions when faced with certain decisions within well-defined processes. For example, in one of the regional offices, a decision had been made to commence a special inspection to assess a series of failures of a diesel generator at a site. During a preparation for the start of this special inspection, what was described as a complicated reactor trip occurred at that site.

When evaluated using the Management Directive 8.3 process, the event landed in the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

24 overlap region between the baseline inspection and conducted a special inspection. The Be riskSMART process was used to assess a range of options related to the region's follow-up response to this event.

Ultimately, through options derived during the use of adopting of the Be riskSMART methodology, the regions decided to provide both the diesel generator issue along with issues observed during the complicated reactor trip into a single special inspection.

The exercise of answering the questions of what could go right or wrong, or what are the consequences help inform decisions such as team composition, the communications with the licensing, and the public. It also helped develop guideposts and warning signs for regional management related to when that decision to combine the special inspection needed to be reassessed.

In another example, the Be riskSMART process was used to evaluate a suggestion from the Region III Risk Informed Decision-Making Team to assign a dedicated inspector strictly to evaluate the licensee's management of risk during outages. So recently, outages are a part of the baseline inspection program, and the inspection procedure NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

25 contained a variety of inspection requirements. But depending on what activities are planned during the refueling outage, not all of the inspection requirements are of equal risk importance, and some, such as the container walkdowns, require inspector attention regardless of other ongoing outage activities.

Over the years, outage durations have significantly decreased, making it challenging for the resident inspectors to complete the inspection requirement while focusing on the most safety significant issues. According to resident inspectors when they're asked, when the outage schedules change, the sequencing and risk assessment of activities and the related continuance can consume a great of time and inspector focus.

The regions have supplemented the resident inspectors with other inspectors doing refueling outages but the resources were not always specifically focused on assessment, the management of outer spreads that created elevated risk. The challenge is to complete the inspection, use risk-based insights to focus the additional inspectors assigned, and to remain within the inspection baseline procedure samples and estimated hours.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

26 Using the Be riskSMART process Region III implemented, the practice of supplementing the resident inspectors during carefully selected analysis, the staff is specifically focused on licensee assessment in management of outage space during periods of elevated risk.

By having the dedicated inspectors, those activities were able to be assessed at a level commensurate with risk whether they need availability of a resident inspector staff. The dedicated inspector position permitted the resident inspector staff to remain aware of all outage activities and the associated risks while the dedicated inspector was able to closely inspect specific activities and associated contingencies based on their detailed review of a licensee shut down plant.

So it's important to realize the teach phase of the Be riskSMART process the dedicated inspector provided the coordinator of this effort with notes regarding lessons learned associated with implementing these activities. The lessons learned were summarized and have been shared with the regional inspection staff during knowledge management sessions and the regional staff has provided feedback to the inspection program office at NMR regarding NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

27 ways to risk rank or prioritize this specific inspection procedure requirements within the outage inspection procedure. I believe that this effort will help improve the effectiveness of the baseline inspection process and program.

In summary, during my participation applying some of these examples discussed today in my setting of other examples from other regions, I have concluded that the Be riskSMART process has served as a source of improved risk communications, that each phase of the Be riskSMART methodology, the impact of uncertainties from other types of risks must be discussed, evaluated, and managed. And in my opinion, the Be riskSMART process has helped advance the idea that risk communications are not just a separate component of a specific process such as the ROP significance determination process. I've concluded the Be riskSMART process must be integrated in all aspect of decision-making, at all levels, both informally and formally.

So next Stacy will discuss other examples where the Be riskSMART methodology was used to further refine other processes. Stacy.

MS. PRASAD: Thanks, Billy. All of us make public safety and trust steps when we make safety NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

28 decisions as highlighted by the Chairmans speech that kicked off the RIC this year. So lucky you, you're going to get a couple more slides of how the framework was used in NRC processes.

In the first phase of the three phases on this slide, we use Be riskSMART to take a look at the process to issue National Special Security Event Security Advisories. That's a mouthful.

The purpose of these advisories is to communicate urgent, time-sensitive, operational information that directly relates to the security and common defense of national infrastructure. They are operational in nature and are issued in response to urgent situations or at least may identify vulnerabilities. Using the framework helped identified a way to remove the review timeline from previously approved lit for on one point (phonetic) reviews.

This basically means we're now able expedite issuance without compromising the review process. This example, you have to keep in mind that just because a process has worked in the past, it does not mean it can't be challenged and that the Be riskSMART tools cannot be employed to identify ways to further risk-informed processes.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

29 A second example focuses on an inspection procedure process. By revising the procedures staff uses to perform cyber-security inspections, staff utilized the Be riskSMART framework to determine whether performance metric submitted by licensees could be used to perform cyber-security inspections more efficiently.

In this case, there was a particular suggestion in mind from the staff from the headquarters office for how those should be updated.

But we'll continue the process and further communication with regional offices that help identify additional what can go right/what can go wrong criteria that were not initially identified.

This helped to further risk-informed updates and ultimately changed the procedure a little bit. In this example found that early engagement is beneficial for procedural frontline processes.

In the last example on this slide, it discusses the current process the NRC uses to provide alerts for warnings for geomagnetic disturbances.

Staff (phonetic) gives a framework to determine if alerts for warnings should also be provided for possible or future events, kind of like a pre-notification.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

30 So this was evaluated and it was determined that the process should remain as is, that there were no safety consequences related to making these pre-notifications.

This last example is particularly important to show that the framework is being used by staff to ask questions, to identify what can go right and wrong, assess the consequence, and make an informed decision that does not necessarily always result in a change.

Next slide. All right, switching gears a little, and going back to the public health emergency. As the COVID public health emergency was first emerging, the agency had to make some tough decisions on how to best implement our inspection program while considering both the safety of those being inspected and the staff going out to perform those inspections.

Federal security inspections we perform are the four sub-force (phonetic) inspections and they verify effective implementation of a licensees overall protection program.

These inspections involve a significant amount of people that can be in close proximity. It will be a properly control of them in performing NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

31 inspection. These are bare bones inspections compared to most other NRC inspections. Before deciding on the best way to conduct these inspections, the NRC has multiple options, assess the risks of each option using the Be riskSMART framework to make a final decision on how to best and most importantly safely perform the inspection.

In this example, we created a matrix where we weighed the regulation to inform our decision. We believe if we use this matrix to help us inform our decisions in the areas of efficiency, openness, clarity, reliability, and independence.

In this case, using the framework helped staff identify other options but while they're not best options during public health emergency can be used again to better inform the future of our security program.

Continuing with the theme of how the framework is being used in informed decisions during COVID, the NRC identified the risk of the impact of delayed billing for applicants and licensees during a public health emergency has potential impact as far as meeting guidance for fully functioning environments.

The public health emergency caused NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

32 financial impacts on NRC licensees, disrupting their operations and challenging their ability to pay NRC fees.

Using the framework, a decision was made to suspend billing of fees for a 90-day period in 2020. The deferral mitigated the risk of significant unpaid invoices. The actual activities to mitigate financial impact demonstrated that the NRC can be flexible while continuing to comply with statutory requirements. Okay, I'll hand it over to Ben to give you guys a little break from all the process talk.

MR. FISK: Thank you, Stacy. Please feel free to put in your questions at any time in a check box in the right-hand side. We're happy to answer your questions after you get through our presentation.

So I would like share a corporate example now related to advancing strategic workforce planning. I am choosing this example as I know we're not unique at NRC with this challenge and that many of you may be likely experiencing similar challenges with an uptick in retirements in your organizations.

Our office's Chief Human Capital Officer applied Be riskSMART framework to the NRC's summer hire program to mitigate the risk of missing the best NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

33 candidates by making selections too late in the year.

After analyzing using the Be riskSMART framework, they decided to shift their recruitment to start earlier and to use a tiger team approach of supervisors across NRC to speed the process and to foster greater collaboration.

Those changes have improved the timeliness, quality, and diversity of that important entry-level feeder pool for the agency. Our Office of Research was able to leverage that summer hire program along with other hiring innovations in FY2021 to fully utilize their staffing levels going into FY22.

Specifically, the applied the Be riskSMART framework to address this risk and implemented proactive strategies including aggressive use of summer hires and conversions of the summer hires to coop positions, external hiring at lower grades in critical skill areas including strategic overhires, and they leveraged the university grants list.

This chart shows this tapping utilization over three years. The agency is in orange and the research is in blue. You can see that the Office of Research achieved 100 percent utilization going into NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

34 FY22. The lessons they learned will be very important as the agency works to address this enterprise this risk this year across all offices consistent with the EDO's vision.

Next slide. I'm going to pause for a second and ask you to complete our next poll. So, again, on the right-hand side, please click poll and you should see a question, "Have you seen impacts from NRC using Be riskSMART?" So I will take a moment to let you fill that out and we will share the results.

Okay. So it looks like the data is starting to stabilize. Thank you very much for your feedback. We have about 43 percent of you guys saying that I have not seen any impacts. We've seen 34 percent where you've seen impacts and they specifically impacted you, and yes you've seen impacts 22 percent but they haven't specifically impacted you.

All right. So that's very helpful. We appreciate the feedback as we continue to improve the rollout of the Be riskSMART throughout the agency.

So going back to my slide, so just wanted to conclude by saying that these examples demonstrate that we shared with you the impact that we've had so far NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

35 applying Be riskSMART across all parts of NRC, programmatic, legal, and corporate.

And, as highlighted by the panel, the application of Be riskSMART has had multiple impacts including improving confidence and making decisions in the face of uncertainties, helping to achieve early identification and alignment on potential risks, enhancing decision-making and risk communications in the region, helping staff assess and select the best option, and also advancing strategic workforce planning.

And on this list, we have a series of resources for you to use, if you're interested, off the NRC homepage. You just click on transformation on the left-hand of the NRC's homepage and then click on risk and you'll the Be riskSMART guidance, training video on Be riskSMART, more case studies, and also a link to our transformation survey for NRC's external stakeholders that includes risk-informed decision-making questions.

I would like to acknowledge Mirela Gavrilas, PhD, and Director of the Office of Nuclear Security and Incident Response, for doing an amazing job leading the Be riskSMART team. And now we're going to turn it to questions. So to get the panel NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

36 started, we wanted to ask the question whether or not you've seen Be riskSMART make a difference. If you have any data to support whether or not you believe that it has. So Reed, I wanted to start off with you and then I can share some information as well.

MR. ANZALONE: Sure, so internally, we have data from some polling that we've done for the staff showing a noticeable positive shift in attitude towards risk-informed decision-making over the last two years. We've also conducted surveys as part of Be riskSMART to better understand who's driving risk-informed decision-making at the NRC. And most of the staff that we've polled think that management and individual contributors are about equally involved in driving forward with informed decision-making.

So we think that that's been at least partially a result of the implementation of Be riskSMART. And I would say, sort of more anecdotally, we're having more conversations about risks and how to manage risk and I would say also, in large part thanks to the pandemic, people are starting to have more conversations about, you know, what risks there are and how to manage them. And I think Be riskSMART has given staff a commonsense framework to think about risks as these conversations NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

37 are occurring. That's just for internal. So you wanted to tackle external, Ben?

MR. FISK: Yes, unless any other panel members wanted to add to what Reed had said before I do that. Okay, so we do have a goal of at least 75 percent of our external stakeholders that we're currently surveying this year agreeing to the fact that risk information and data has improved the timeliness and quality of agency decision-making.

And while we don't have a lot of people responding right yet, we're still collecting data.

The ones who responded so far, 50 percent of the respondents agreed that the use of risk information has improved the quality of decision-making over the past year and that 41 percent agree that the use of risk information and data has improved the timeliness of decision-making.

So I do think that we are making progress but there's more work to be done and I think that somewhat correlates with what we were seeing internally that you were sharing.

MR. ANZALONE: Okay. So let's see. I think we have another question here. This one is to Reed and to Billy. Is Be riskSMART or risk-informed decision-making at the point where we're able to say NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

38 PRAs are worth it?

MR. DICKSON: I can go. So, you know, there are a lot of licensees who have NRC-reviewed PRAs. I want to say it's probably around half of them. And Be riskSMART is one of the tools that we have that can incorporate that into our decision-making process. There are other ones, like the risk-informed process evaluation or the very low-safety significance issue resolution effort and lots of other risk-informed decision-making processes that are all, I would say more or less, consistent with Be riskSMART.

As to whether they're worth it or not, I think that is something that the individual will have to figure out for themselves. But you know, we are trying to do more to be able to take advantage of that, so the insights that we get from PRAs and other risk analysis tools and incorporate those better into our decision-making processes because we that that's important data that we can use to make better decisions and make sure that we're focusing on the most significant thing.

MR. DICKSON: Thank you, Reed. And from my perspective as a manager of the inspection group, I believe that the PRA is worth it. I mean, it allows NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

39 our inspectors to focus on the issues that matter.

When we're doing assessments of issues, PRA results help us key in on those risk-important issues that we need to key in on to be an effective regulator. So yes, I do believe the PRAs are worth it.

MR. FISK: Okay, there's another question. This is to Reed, and I think, Mirabelle, you said that you would be able to add in after he answers as well. What role does considering benefit play in your Be riskSMART decision-making?

MR. ANZALONE: So I think the benefits are there to make sure that when you are originally coming up with the framework, we wanted to make sure that you would acknowledge that there might be a benefit that you would miss out on if you chose a decision one way versus the other.

So for instance, let me think of a good example. If you decided to do tackle a problem one way, there might be a benefit that you would miss out on that could be gained from another way of approaching it. So that was one of the reasons we wanted to include benefits. Also we wanted to make sure that we were really factoring all of the available information. You know, benefits and risks kind of go hand-in-hand. I guess that's what I can NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

40 say to that.

MS. SHOEMAKER: Billy, I think you touched on what I wanted to highlight about benefits is that when you look at the risk triplet, there is the opportunity to consider what can go wrong and what can go right. And what can go right is an opportunity. There is a yielded benefit potentially from one decision-making to another.

I think I highlighted in the example of the COVID-19 public health emergency, when we were considering whether or not to conduct inspections remotely, we came up with our decision to do remote monitoring and there was the added benefit in another area to be able to use technology, observe the inspection in multiple ways, simultaneous activities that were occurring at the same time, and we actually ended up with, I would say, a greater risk-informed decision.

So benefit in the early stages is spotting what can go right, and identifying an opportunity allows you to make a comprehensive risk-informed decision.

MR. FISK: So another question, I think this is for Mirabelle and Stacy, is the agency thinking about extending the very low safety NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

41 significance issue resolution process to other business lines and activities?

MS. SHOEMAKER: I'll take the question first. So with respect to very low-safety significance issue resolution, NMSS has created a working group. One was set up in 2020 to consider whether we could apply NRR's B-lister concept in NMSS. And actually the working group created a digital exhibit that's available at this RIC.

In addition to this digital exhibit which provides additional information and resources you can look through online, the working group created screening guidance to be incorporated in the technical assistance review process to aid staff in identifying issues that might be candidates for the application of Be riskSMART.

The working group is mostly working with supporting staff that are in the process of updating inspection manual chapters to see how Be riskSMART principles can be consistently incorporated into these revisions of the chapters.

The working group is also supporting engagement with industry. Recently a topic on the issue of short-term operations at ISFSIs was identified by general license holders as maybe a NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

42 potentially appropriate topic.

So the list of principles have been pulled into those public meetings for consideration.

And something that's currently underway, so I can't speak to it too much, but we're soliciting from industry potential topics where Be riskSMART can be applied. The working group will be looking to create a public-facing webpage after the RIC to be able to provide updates on the application of Be riskSMART around fuel cycle decommissioning in ISFSIs and material users. So we welcome that kind of feedback after the RIC.

MS. PRASAD: I'll just add a little bit for security here. So Be riskSMART is part of the process we use when we determine how to process an issue of concern, and that includes security concerns. So if any region comes to us and they have a security issue, the security form or input is received from all staffs and regions, develop a little paragraph with our opinion, and make sure we assist with the application of regulations. And then also we include assessing the significance of security performance deficiencies. So we have been using it actively since 2019 and are beginning to see staff question it more and more as time goes on.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

43 MR. FISK: All right. So we have another question for the entire panel. How do you consider multiple risks while making a decision? Whoever wants to take that first.

MR. ANZALONE: I can take a first crack at it. So I'm speaking for myself personally, and I think we talk about this in the Be riskSMART guidance, especially because this is a tool that is commonly used in data finds in engineering. A heat map is a very useful tool for being able to compare multiple risks. So whether they're quantitative or qualitative, you know, if they're quantitative you can have likelihood and consequences on the X and Y-axes. If they're qualitative, you can have likelihood and consequences on the X and Y-axes but instead of numbers you've got low, medium, high.

And being able to put those together on a single chart and sort of consider everything together all at once is really helpful. And it also helps, and I kind of alluded to this when I was walking through the framework, it helps you understand, you know, maybe you draw a line where your risk appetite is in terms of consequences and probability. So maybe you really don't like high consequence, low probability events but you, you NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

44 know, decide outside the ends of that where that you're okay with the decision and you can sort of factor in multiple different risks all at the same time.

MS. SHOEMAKER: I'll go ahead and add to what Reed just mentioned. The issue of risk appetite is really important there. When we rolled the idea of Be riskSMART to NMSS, we started with discussion about risk appetite and what that looks like when you're weighing multiple risks and then a decision.

It gets the conversation going, especially for qualitative data, what methodology is going to be deployed to weigh risks and managing those risks.

So NMSS benefited from learning how NRR conducts the heat map and allowed us to take an approach to apply qualitative data in a heat map.

MR. FISK: Mr. Billy, did you want to add anything?

MS. PRASAD: If I can add something real quick. I agree the heat map is very useful for what we're talking about here. Sometimes personally when I use the Be riskSMART framework, I may get into an area that's kind of in-depth or very technical, and you'll find out, it's just going everywhere. There's so many things to assess.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

45 So what I've seen a couple times and there are certain days to look at the heat map, where we'll look at the appetite for it, but you might also determine that this is an opportunity to have a different Be riskSMART, ask a different question that could be more broadly applicable to your program.

And I think that has helped with answers in quite a few cases.

MR. DICKSON: And I actually believe that tracking and training of all of the risk factors, understanding the brackets in which you have risk tolerance, is all important to understanding all of the impacts of the different risks. And we try to do that implementing the Be riskSMART process itself.

MR. FISK: So this question is for Reed.

Sorry, Reed. It seems like a lot are for you. Can you explain what risk-informed process for evaluations (RIPE) is, and how it's been applied to date?

MR. ANZALONE: Sure. RIPE is the risk-informed process for evaluations like you said, and it's a process for evaluating low-safety significance issues that are known to be within the plant's licensing basis. And it's currently available to resolve those kinds of issues for plants that are NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

46 already known to have technically acceptable PRAs but ones that the NRC has kind of looked at typically through one of the tech spec standards, and who have also established an integrated decision-making panel and laid guidance out there for how to do that.

And so if a licensee has established a technically separate DRA and an IOED and the licensee can characterize whatever the safety issue is as low-safety significance, they can require some kind of licensing activity that can be reviewed by the NRC using a streamlined process.

But currently, it's not available to make changes to plant technical specifications, but there's a revision in the works that is -- were working on that, that we're hoping to roll out later this year. And we got our first application late last year or early this year and accepted it.

MR. FISK: So this question is for the entire panel. What is the difference between Be riskSMART and risk-informed?

MS. PRASAD: I think I can start off on this one. So Be riskSMART is a risk-informed approach. And it kind of lies between risk-based and deterministic approaches. So risk-informed really ensures that risk insights are considered together NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

47 with other factors to establish requirements commensurate with the report in total health and safety.

So what we're really saying here is we'll look at risk, but its being looked at in conjunction with safety margins, defense in depth, performance monitoring, compliance with regulations, stuff like that. And the framework has exclusive places where the excessive areas make the informed decisions. So I guess the bottom line to the question is, Be riskSMART is a risk-informed approach.

MR. FISK: Does anybody want to add anything? Okay. So the next question, and this is for the entire panel, whoever feels comfortable, what is the risk metrics used when risk informing the NRC decisions and processes? Is it something other than core damage frequency and large early frequency or is it more qualitative with respect to the risk metrics?

MR. ANZALONE: So I can, again I guess, take a stab at that. So fundamentally, I mean, for reactor safety question, yes, the metric is core damage frequency and early release frequency. But when you have multiple options for decision-making that all area at least assumed good enough, there are other metrics that can come into play. You start NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

48 talking about reputational risks. You start talking about which risks might incur the greater legal risk.

And those are obviously more qualitative. I hope that answers the question. If sort of a combination, you might have both quantitative and qualitative risks that you need to bring together and Be riskSMART is a great way to accomplish that.

MR. FISK: Does anybody want to add anything else?

MS. SHOEMAKER: Yes, I think Reed, what you were saying was, in reference to the heat map you talked about earlier, and Be riskSMART facilitating an early discussion about the methodology for identifying the risks and then giving weight to them, which is particularly helpful in a qualitative setting.

MR. FISK: Okay, question for Billy and the rest of the panel. Is Be riskSMART used to cut resources?

MR. DICKSON: Not in my experience. In fact, Be riskSMART has been used for, and the reasons are, to help gain efficiency with the current staff that we have. We've experienced some losses due to retirement and retention and we use Be riskSMART to deal with those resource issues. But in my NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

49 experience, to answer the question directly, no, I haven't seen Be riskSMART used to cut staff.

MR. ANZALONE: And I'd like to weigh into that a little bit. So you know, sort of from a broader standpoint, we tried to structure Be riskSMART to be neutral. It doesn't really say you should, you know, use less resources. It's supposed to tell you that you should be using the amount of effort that's commensurate with the level of risk associated with something, right? So it should be agnostic to whether something is going to consume more or less resources.

And you know, we do see examples and I think that several of those were highlighted here by the panel earlier today. We see examples where people have used more resources or at least not the least amount of resources and resulted in something that they evaluated with Be riskSMART. It helps us align the resources that we're using with the means and can support increasing the resources to areas that are more risk significant.

And sort of more broadly also, I think we need to -- if we want to make sure that we're not just using it to cut resources, we as an agency, and this is my opinion, need to be using Be riskSMART in NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

50 more decisions and be more disciplined in applying it not only in areas that we want to drive efficiency but we want to be able to say we're focusing the resources where they need to be. And so that means we need to be using risk in more of our decisions.

MR. FISK: Does anybody else want to add anything.

MS. SHOEMAKER: Yes. I was just having a conversation with my office director yesterday about this and it kind of aligns with, you know, are we seeing results from Be riskSMART in our area. And an example similar to what Billy mentioned was the Decommissioning Reactors Oversight Program used Be riskSMART when reviewing the program and it didn't yield a decrease or increase of resources. It simply reprioritized them based on the risk areas associated with the program and allowed the inspectors to, you know, sharpen their focus areas when they go in.

So we're seeing a benefit there and we're seeing Be riskSMART realized with positive results and not necessarily with our eye towards, you know, changing resources in the process.

MR. FISK: So just a reminder. Please continue to put in your questions into the chat and we will try to answer them, put them in the queue.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

51 All right. So I have another question. This one's for Billy. Is there alignment between Be riskSMART and the use of probabilistic risk assessment? Would you say that Be riskSMART positively or negatively impacted the NRC's use of PRA?

MR. DICKSON: So I am very confident that the Be riskSMART has not harmed the PRA process. We used insights being from the PRA process to apply to the Be riskSMART process. So we're using them in conjunction with each other.

So PRA is a very important vehicle to risk-informed decision-making during the development of the framework. The agency focused on making it consistent, not making it inconsistent with preexisting processes. And again, I'm talking about the Be riskSMART process. We wanted to make sure that they were not inconsistent with processes like the PRA.

MR. FISK: We have another question.

Does the risk-informed decision-making process allow us to state how safe is safe enough? Who would like to take that?

MR. ANZALONE: I can take a stab at it.

So, I mean, fundamentally the standards that we hold to for safe enough is reasonable assurance as NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

52 adequate protection. And that's assigned through, you know, compliance, the regulations, a bunch of different other things.

And going back to the answer that I gave before about, you know, what the risk metrics are, there are, you know, specific risk metrics for core damage frequency and large early release for reactors. But those don't really apply across the board. I wouldn't say that rhythm itself defines how safe is safe enough. There are lots of different factors that play into that. I don't know if anybody else wants to add to that.

MS. PRASAD: Yes, I agree with you said.

And I just wanted to add, it really is the risk appetite in the specific question that you're asking.

So sometimes you may be willing to take a little more risk in an area.

And maybe if there's a little less and things like that, because as we said, this is a risk-informed process, right? So we're looking at risk and also looking at other deterministic factors. So that's all going to apply to the decision you're going make and we look at both.

MR. FISK: So we have another question.

And I think Mirabelle, this might be one for you, but NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

53 other panel members are certainly welcome to add to it. How does Be riskSMART fit into the vision of NRC being a modern risk-informed regulator? What does that really mean?

MS. SHOEMAKER: Sure. So I think Be riskSMART in terms of a timeline was really born out of the idea or that, you know, buzz term of modern risk-informed regulator. I remember shortly after hearing management talk about modern risk-informed regulator, there was the push to create framework where we could apply a risk-informed approach to decision-making. So really it's chicken-egg sort of question.

For Be riskSMART then enables us to answer those questions about, you know, where we now live in a digital era where we didn't foresee sharing information through certain mediums or receiving it a certain way. We didn't have insights about what the nuclear landscape looked like 20 years ago when regulations were written.

And we, you know, advance reactors are accident tolerant fuel. There are developments that are, you know, we're living through them, and we need ways to assess them for licensing. And Be riskSMART allows us to do that because it allows new information NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

54 to be considered alongside the traditional risk that we've assessed and make our risk-informed decision to ensure that we're meeting our mission.

MR. FISK: Does anybody want to add anything? Okay. So we have another question, and this is a really important question. How would the NRC know that it's transformation efforts on the use of risks in the decision-making is reaching all levels of staff?

I know that many of you have served as ambassadors for the Be riskSMART and also part of the core team and, you know, we've gotten some survey results and, you know, the ultimate vision ideal state would be having everybody intercede factoring and risk information in a common way across the enterprise so that we could have meaningful discussions around risk and make the best decisions and the most timely decisions. So what would you say on that?

MR. ANZALONE: So sorry, were you about to go, Stacy?

MS. PRASAD: I was but I'm sure yours will be a great answer and I'll just sit here and listen.

MR. ANZALONE: Well, I was just going to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

55 say, you know, you take a look at the cross-section of the staff that we've got here today. We've got people from all different offices. We've got, you know, line supervisors, staff, senior managers all represented here. I think it is happening. We are reaching everybody at the NRC and across the business lines and then extending to, you know, not the technical areas as well, you know, into the corporate and legal areas too. So I think we're doing a great job of that. And yes, so I'll leave it there.

MS. PRASAD: I was just going to add so people are resistant to change and anything new that comes out. And I remember when this first rolled out people were like, oh, another risk-informed initiative, and it was kind of like that cooler talk.

where no one wanted to actually use the framework.

So that's how it started. That's how a lot of things started for us in the beginning. Now when I go to meetings or we have teams where we can chat privately with other employees, it's more open than that. But now people are even bringing up in those discussions, oh, I used the framework for this.

What do you think of this heat map? And it's becoming part of the normal conversation when a big decision is made.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

56 Management - our office is under Mirela, and she's a big support of the Be riskSMART framework.

So we use it all the time in management and insert it at all levels down. And I can't think of a meeting that I've been to, at least in the past couple months, where the Be riskSMART framework wasn't utilized.

MR. ANZALONE: May I tell you some -- my perspective, many of the examples that I've studied and even presented today were born out of the suggestion from the staff in doing the application of Be riskSMART in their presentation to myself and to management. So it makes me believe that the staff has embraced the process and they're using it for their benefit, you know.

Just last week, I listened to a discussion on Be riskSMART from the inspectors in our Health Physics Branch to talk about the Part 37, the documentation of Part 37. It really exemplified the staff's use and acceptance of Be riskSMART process.

So yes I've seen it, and I think it's taken hold at all levels of the staff.

MS. SHOEMAKER: So then I'm going to end out with this cynical voice that hopefully has a little positive ending at the end of it. As you mentioned in my bio, I came to the working group with NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

57 some skepticism about application of Be riskSMART, specifically to my working area. And that's something I had really heard throughout NMSS.

Without the use of a PRA, how are going to deploy this framework, you know, for corporate, for technical, for legal decisions? How do you consider risk in these different disciplines?

So that was a challenge for me coming into it. And I think it's through that practiced approach of Be riskSMART and teaching and talking to other staff that it's sort of become second nature for me. At the same time, I think there's still a lot that can be done, especially by the working group, to continue to reach staff at all levels.

I'll get a little bit into the weeds on what we do in NMSS and hopefully it sparks some feedback that we can consider. But in our office, we have a working group that is constantly talking to staff and collecting examples of Be riskSMART wins.

And we really are emphasizing the teaching element of Be riskSMART. So we're making sure it gets into Nuclopedia, which is a resource for teaching examples and lessons learned on various high-profile topics.

We created a SharePoint where we have been hosting popup seminars where staff talk about NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

58 their use of Be riskSMART in a scenario. We save the recording and make them available on SharePoints so staff can go and watch them and hopefully identify with the area that's being covered.

You know, we've covered corporate examples, things about contract renewal. We've talked about engagement with stakeholders. We' talked about safety analysis. We're trying to cover all areas because especially in NMSS, where there are multiple business lines and multiple disciplines, we want to make sure that we're reaching all those levels. So that's a work in progress right now for us.

MR. FISK: Reed, did you want to add something?

MR. ANZALONE: Yes, so when I joined the agency however many years ago it was, my background was in deterministic safety analysis work, and I wanted to kind of echo what Mirabelle said. So my cynicism and reluctance I would say predates even the entrance of Be riskSMART. And when I joined the agency, I was sort of like, oh, well, risk is the thing for those PRA people to do and I don't need worry about it. I'll focus on my deterministic safety analysis.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

59 And I think that working with those people who are in NRR's risk assessment and sort of opening my mind to the possibility of what you could consider with this and understanding how it's used in the inspections program and elsewhere in the agency, really helped me see its value and utility in making decisions.

And I think that a lot of people are starting to come around, even in the staunches parts of the agency where people would have that opinion of, well, that's something for somebody else to worry about. They are coming around that because they are actually seeing it make a difference.

MR. FISK: So I'll just add that from a corporate perspective, we have enterprise risk management, which is part of our internal controls and strategic plan and performance management system.

And the senior leaders quarterly talk about enterprise risks.

So we don't just find senior management driving this discussion. We want to have this discussion happen everywhere within the agency so that people who are positioned to identify risks are managing that risk and then engaging with the senior leadership as appropriate.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

60 But I can reflect on some of those discussion and we've managed so many risks I can't even mentioned how many. But we have a whole very systematic process where quarterly we look at whatever major risk before the agency, and I think anybody who's been sitting listening to the commissioners and the EDO and chairman has heard over and over this hiring risk.

Because I think when you look back and saw the data, we hadn't been hiring enough externally. We'd been moving people more internally, and that's why we're shifting that focus in this fiscal year 2022, where we're really trying to shift the agency's focus on bringing in more people externally while still allowing people to grow. We definitely want our internal staff to have opportunities to get promoted as well.

So that's just something that is an example of an enterprise risk, and a lot of that was driven by some corporate concerns. And now it's become like an agency focus. So it's actually really exciting to see.

We actually put together a steering group and a corporate work group to really focus on it.

And once we put the work group together, we identified NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

61 that there were multiple risks. There's the onboarding risk, right? We can't just hire people.

We have to onboard them. And so we really want to retain people. We want to import them. We just want the whole process to be addressed. So anyway, a lot of that was born out of thinking about risks.

So Reed, a question for you. Can you talk more specifically about the mission analytics portal? How is it helping to improve decisions?

MR. ANZALONE: I mean I'm happy to tackle that. I'm not sure it's a Be riskSMART question but if they --

MR. FISK: Well, it's the smart piece, so if you could connect your --

MR. ANZALONE: Right. I will. Use of data and decision-making and that's definitely part of this. So I think that one of the big things that map is doing, at least in NRR, is helping make information just readily accessible to make decisions. So we used to have a meeting where we go through all of the projects that we were working on and talk about which ones might be at risk of failing the metric and whether those needed more management attention on them to make sure that they were getting done and we were going to meet the metric.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

62 That meeting has gotten streamlined and condensed thanks to use of dashboards and data to where, like it basically doesn't even need to happen anymore. So that's a risk that were managing. So we're managing risks on each one of those individual projects by being able to see exactly where things are in the process at any given point in time. So that's one example. I'm sure there are others in other business lines.

I know that we recently just implemented a change or a new set of dashboards to help facilitate the end of cycle meetings in the regions for their inspection cycle. And I think they're starting to make a difference there.

MR. FIS: There are. So the next question's for the entire panel. Do you have any indication how many people are using this framework?

MS. SHOEMAKER: So I'll start off with, in terms of data and a specific number, I couldn't ballpark that for you. But what I can say is that, for NMSS to get an idea, we have one of our objectives is key result being use of the Be riskSMART framework.

And in -- with that risk, we wanted to, you know, encourage staff to use Be riskSMART in their decision and document it so that there is that NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

63 opportunity to teach down the line but also the ability to track that metric.

And how we're helping staff is identifying the types of meetings where they can expect that Be riskSMART will be asked and give them some tips for preparing for it.

So when we were talking about external stakeholder engagement, we can expect Be riskSMART will be inquired by the upper levels of management.

So we have staff thinking about that at an earlier point in time.

When we're talking about assignment and alignment meetings, that's a great opportunity to use Be riskSMART to ensure that we're aligning on the risk-significant issues that are under consideration.

So I don't have an exact number, but what I can say is that it's very -- it's being promoted within NMSS, which areas absolutely should be considering Be riskSMART and capturing documentation of that in our repository so that we can continue to teach.

I will say that, right now, our objective and key result is set around we want to see about 75 percent of decision-making not only is using Be riskSMART but it's also focusing on that data element NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

64

-- kind of Reed was alluding to -- and looking at more ways to incorporate the data aspect into the decision-making and capture it. So that's something we're working towards.

MR. ANZALONE: If I can add on to, a little bit, what Mirabelle was just saying, so we also, at NRR, have an OKR related to use of Be riskSMART in decisions.

But I would also say that, you know, not all decisions that people might use, concepts for Be riskSMART are ones that we would track or get a sense of if people are, you know, talking about very low-level things, decisions in their own everyday work that they're trying to decide.

They wouldn't document that and pass it along to somebody else or put it into a dashboard someplace. And I think that that's the kind of groundswell of interest in risk-informed decision-making because it -- we're seeing this kind of as that primal.

So I think it's hard to objectively measure exactly who's using Be riskSMART and in what sense. I mean, definitely, with what Mirabelle's talking about, we are doing that at the office while there's -- there are lots of lower-level decisions NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

65 that happen all the time that are a lot harder to track. And hopefully, we're making the impact there.

MS. SHOEMAKER: I think, Reed, you just reminded me, we are -- when we've had conversations with staff, we find that they are using Be riskSMART.

They just didn't know that that's the name of it.

So part of our effort to engage with staff on Be riskSMART is to help them realize that they're already using this umbrella framework. And it could applied at day-to-day, everyday activities and just making it feel more organic and part of our regular toolbox.

MR. FISK: All right, so here's a good question. How do we make sure that Be riskSMART does not become just another fad? I think, Stacy, you wanted to weigh in first and maybe others want to add after that.

MS. PRASAD: Thank you for all of this.

So the team that put together this framework, they did a commendable job, so we have a solid framework that really has all the elements that you need to succeed.

My take, it'd still be a fad? Maybe.

But to keep the framework relevant, we purposely added that little arrow at the end of the process NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

66 that allows you to reassess so we can be sure they're using that process to reassess as needed, so we're keeping apace with regulatory environment transformation.

I mean, it's all part of the transformation process, we had to fill it in. So far, it's looking good and we're hoping it continues down that path. But only time will tell.

MS. SHOEMAKER: Yeah, I'll add to what Stacy said. When I was talking to me office director about this the other day, he made a good point that, you know, Be riskSMART is at a certain level of maturity where we're going to start seeing the benefit of some actions as we go through that Realize step now.

So I don't think it's going to be a fad in the sense of disappearing, you know, in a year or two because there are projects that are underway that still need to go through the Realize element to understand, you know, where we need to correct course and, you know, go through the iterative process, like Stacy said.

There are ways that we are ensuring we're engraining the framework into current guidance. For example, I mentioned the inspection manual chapters NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

67 that are under review.

In many cases, the text of those documents are incorporating Be riskSMART language and terminology just to help make sure that we're ingraining it in the guidance that staff can put their hands on.

MR. FISK: So the one thing that I would just mention is this is very correlated with Enterprise Risk Management. The whole vision of Enterprise Risk Management is really to engage staff.

So I see them all very connected. You know, it's just going through systematic methodology where you're really thinking about these things. And it's common sense.

And it's giving us the language where corporate people like myself can talk to technical people like you as well as legal people.

And I've also, you know, found myself pushing back on Legal sometimes when they just say, here's one option. I'm like, well, that can't be the only option. We need to generate other options.

We need to look at the risks associated with each option. We need to look at the benefits and we also need to look at what problem we're trying to solve.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

68 So I'm like, constantly saying these things over and over and over. And I remember Mike Webber. I don't know if you guys remember Mike Webber, but he used to say, when you're exhausted from communicating, that's when you're finally reaching people.

So I don't see this as a fad because it's just common sense, you know. And I think that it's adding value. And I think that that's what will make it thrive and live. And -- anyway, I just -- I wanted to add that.

MR. ANZALONE: I wanted to chime in with one more thing. And I think this kind of goes back to what Mirabelle and I said on the last question.

I think where we get to not fad status is people are using it. They're comfortable with it.

It becomes part of their regular everyday work. And then it's not a fad because it's just something that you do. And it doesn't go away because it's part of the organizational culture, and it's something that, you know, you learn when you come to the NRC, is this is something that you do.

That's where I think real success is and, you know, not being a fad. And I would say we've done -- like I mentioned in my talk, we did some NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

69 polling. Internally, we talked about who's driving risk-informed decision-making.

And right now, we think it's ethereal.

You know, we want to see that continue over time to be sure that we're being successful.

MR. FISK: So I'll just give you an opportunity for any final thoughts before I turn to closing remarks. We're almost out of time. Billy, did you want to make a closing statement of any kind?

MR. DICKSON: No. As I stated -- well, I'll just mention, as I stated during the presentation, and we talked about this. We talked about this through this, throughout this Question-and-Answer session.

I believe that the staff is intimately engaged in the Be riskSMART process. You know, as it's managed daily, I constantly have staff presenting issues to me and they're presented on the Be riskSMART framework.

So I think, from that perspective, it's been a real success. I've seen decisions that have created effectiveness and efficiency that, I think, are -- have been invaluable over the last two years or so.

So, again, I believe that the Be NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

70 riskSMART process should -- is a process that should be incorporated at all levels of the Agency and in ways found to incorporated it in all of our decision-making processes. So that's it.

MR. FISK: Anyone else would like to make a final statement? Okay, well you guys are just an amazing panel.

That's pretty much all the time that we have. This has been a wonderful discussion. And I would like to call up this email on this slide.

You can send any remaining comments or questions you have that did not get answered for you.

And we can ensure follow-up.

I want to thank the panel members for their wonderful presentations. I also want to thank Katie McCurry, Elizabeth Bowlin and Mirela Gavrilas and other Be riskSMART team members for the critical roles that they played in making this panel a success.

I also want to thank the audience for your engagement and your fantastic questions. And I do want to remind you about the stakeholder survey on transformation. Please take that if you haven't already.

That concludes our Be riskSMART panel.

And thank you for attending.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com

71 (Whereupon, the above-entitled matter went off the record at 4:28 p.m.)

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1716 14th STREET, N.W., SUITE 200 (202) 234-4433 WASHINGTON, D.C. 20009-4309 www.nealrgross.com