ML21176A179

M210622B: Transcript - Transformation at the NRC - Midyear Review

ML21176A179
Person / Time
Issue date:	06/22/2021
From:	NRC/SECY
To:
Shared Package
ML21050A177	List: ML21133A042 ML21137A306 ML21167A042 ML21167A043 ML21167A062 ML21176A179
References
M210622B
Download: ML21176A179 (120)

Text

1 UNITED STATES NUCLEAR REGULATORY COMMISSION

+++++

MEETING ON TRANSFORMATION AT NRC:

MID-YEAR REVIEW

+++++

TUESDAY, JUNE 22, 2021

+++++

The Commission met via Video Teleconference, at 9:00 a.m. EDT, Christopher T. Hanson, Chairman, presiding.

COMMISSION MEMBERS:

CHRISTOPHER T. HANSON, Chairman JEFF BARAN, Commissioner ANNIE CAPUTO, Commissioner DAVID A. WRIGHT, Commissioner ALSO PRESENT:

ANNETTE VIETTI-COOK, Secretary of the Commission MARIAN ZOBLER, General Counsel

2 NRC STAFF:

MARGARET DOANE, Executive Director for Operations NICHOLAS BUGGS, OCIO JUNE CAI, ADM ERIN CARFANG, Region I CANDACE de MESSIERES, NRR STEPHANIE MORROW, RES TAMMIE WILLIAMS, NRR

3 1 PROCEEDINGS 2 9:00 a.m.

3 CHAIRMAN HANSON: Good morning, everyone. I 4 convene the Commission's public meeting on NRC transformation. This is 5 the mid-year review. Our last meeting on the subject was in September of 6 2020.

7 We will hear today from two panels. First we'll hear from 8 the NRC staff, and following that, after a short break, we'll hear from 9 participants in our external panel. With each panel, we'll hold questions until 10 the end, and then we'll hear questions from the Commissioners.

11 Before I start, I'll ask if my colleagues have any remarks to 12 make. No? Okay, so with that, we'll begin with the NRC staff panel. We'll 13 begin with our Executive Director of Operations, Margie Doane.

14 MS. DOANE: Good morning, Chairman and 15 Commissioners. We look forward to providing you an update on the NRC's 16 transformation to become a modern, risk-informed regulator. As you will hear 17 today, we continue to make progress in all transformation focus areas.

18 Our mission to protect public health and safety, security, and 19 the environment has not changed. What is changing is how we go about 20 meeting this mission. Transformation is a whole Agency endeavor, and 21 though today we have success stories showing change across the Agency in 22 technical, corporate, policy, and legal programs, we know that there is more 23 to be done as this a journey, not a one-time event.

24 For background, I'll briefly discuss the highlights of our

4 1 transformation journey and how we got here. Although I'm sure that this is 2 sounding familiar, it will be helpful to set the stage for today's briefing.

3 As we discussed in our last meeting in September 2020, the 4 NRC built upon its 2019 futures assessment to develop the dynamic futures 5 for NRC mission areas report, which describes four hypothetical future 6 scenarios in which the NRC might operate in the year 2030 and beyond.

7 Using the futures report and information we gathered from 8 various efforts like the NRC Futures Jam, we identified four focus areas for 9 transformation: our people, use of technology, Be RiskSMART, and 10 innovation. These form the basis for a transformation journey roadmap.

11 This roadmap first identified and prioritized eight initiatives, 12 all of which are complete, except for our ongoing efforts to achieve the 13 Agency's desired culture. That culture initiative is focused on ensuring that 14 behavioral norms and sustainable processes are incorporated into how we do 15 business. You'll hear more about this later in the presentation.

16 The goal for 2020 was largely about developing the tools 17 and resources that would fuel our transformation. Each of the eight initiative 18 teams worked throughout the first year to familiarize the staff with their topics 19 and work collectively to identify ways to modernize.

20 Each team developed tools or products in pursuit of their 21 desired outcomes or goals and supported individual offices and regions in 22 developing and implementing their own transformation efforts. This work has 23 now put in place new infrastructure and tools for the staff to use to continue 24 the change journey, such as the Be RiskSMART framework.

5 1 We also have more than 200 transformation champions and 2 ambassadors, many of whom were previously on the initiative teams. They 3 regularly meet to discuss the specific initiatives they support, challenges they 4 face, and share the lessons learned that can help make progress across the 5 entire Agency.

6 Having built the infrastructure, our efforts in 2021 are 7 focused on sustaining the change that we have seen in the first year by 8 seeking ways to encourage widespread application of these new tools, 9 approaches, and resources across the Agency.

10 Our goal of transformation has always been on enhancing 11 our ability to solve the challenges of today and tomorrow, including those that 12 may not even be on our radar yet. And assessing risk is at the center of this 13 effort.

14 As a safety regulator, NRC has had a longstanding risk-15 averse culture in our approaches to meeting our mission. While this has 16 served us well in many areas, it has also led to long stretches with little or no 17 innovation and approaches that did not take advantage of modern technology 18 or other resources.

19 Through our transformation efforts, we've been working to 20 have a broader perspective of risk and have more consistent application and 21 management of risk across disciplines. For example, we're trying to be more 22 proactive in identifying how certain well-managed risk can lead to better 23 decisions that consider the most current information. These efforts will be 24 highlighted in our presentation today.

6 1 The focus of this meeting is on these newly-developed risk 2 tools and other resources that were developed in 2020, and how the Agency 3 is making progress to sustain these efforts at both the Agency and staff levels.

4 Next slide, please.

5 Okay, so today we'll hear from Stephanie Morrow from the 6 Office of Nuclear Regulatory Research, who will provide an update on the 7 Agency's desired culture efforts. Candace de Messieres from the Office of 8 Nuclear Reactor Regulation, who will discuss the Be riskSMART framework 9 in decisionmaking.

10 And we will also two mini perspectives regarding the Be 11 riskSMART framework. First, Erin Carfang from Region I will discuss the 12 reactor materials regional perspective. And then Tammie Williams from the 13 Office of Administration will present on the Be riskSMART framework from the 14 corporate perspective.

15 Nicholas, or Nick, Buggs from the Office of the Chief 16 Information Officer, OCIO, will provide an overview of the NRC's enterprise IT 17 roadmap and data strategy. And finally, June Cai from the Office of Nuclear 18 Materials Safety and Safeguards will discuss the transformation of the 19 rulemaking process, including the application of agile project management 20 concepts.

21 But before I turn it over to my fellow presenters, I wanted to 22 highlight a few significant achievements since our last meeting with you on 23 transformation. Notably, the COVID-19 pandemic has underscored the 24 Agency's transformation efforts and galvanized the already increasing use of

7 1 technology in our day-to-day work efforts.

2 We transitioned our communication platform from Skype to 3 Teams, that we adopted midstream during the pandemic. We have 4 dashboards proliferating throughout the Agency showing data in a more 5 understandable and useful way. Our volume of easily accessible data has 6 increased by millions of documents that have been digitized from microfiche, 7 and we are actively breaking down silos of information to enhance 8 retrievability.

9 We are currently developing a technology road map that will 10 ensure access and use of data continues to be enhanced. We have worked 11 almost entirely paperless during the pandemic. These and other 12 transformation efforts enable the NRC to continue to meet its mission in this 13 very dynamic environment. We gained significant experience that will carry 14 forward in our lessons learned.

15 I cannot say enough positive things about the NRC staff's 16 efforts during the pandemic to meet our mission and to embrace positive 17 change in the ways that we work. Many of these were sparked by the 18 transformation effort. This has not been easy, and it underscores our staff's 19 dedication and commitment to making the NRC a modern regulator.

20 It bears repeating that we have more work to do as we 21 cannot say that we are transformed. So with this framework established, 22 we're focusing on what's next and how to keep this momentum going.

23 This is especially challenging considering that our focus 24 areas for the Agency are making sure that we continue to work well during the

8 1 pandemic and that our eventual full reentry into our building is supportive of 2 our staff who are key to meeting our mission efforts. We are keeping all of 3 this in mind as we continue to make transformation changes.

4 One thing that we are doing to keep our momentum for 5 change going is transitioning our transformation efforts from the Office of the 6 EDO to the offices themselves. These next steps are designed to help 7 empower not only the offices collectively, but each and every one of the NRC 8 staff to make innovations on their own and adopt innovations by others.

9 One example we mentioned during our last meeting got its 10 fueling innovation at the staff level is IdeaScale that was launched by Innovate 11 NRC 2.0 Initiative Team. It enables the NRC staff to share successful 12 innovations and crowd source solutions.

13 Last week, we celebrated the first year of the platform, which 14 has been visited by over 80% of our staff. Using IdeaScale, we've achieved 15 the completion of 17 challenge campaigns and captured over 480 success 16 stories.

17 To hold ourselves accountable for sustaining change 18 throughout the Agency, our offices have also developed objectives and key 19 results, or OKRs, for the 2021 calendar year.

20 One aspect to highlight is that this year, one of our key 21 results is to achieve positive feedback through a first-time survey we will be 22 asking of our external stakeholders about whether they perceive any changes 23 in our mission execution as a result of our transformation efforts.

24 This concludes my introduction. I will now turn the

9 1 presentation over to Stephanie for her update on the Agency's desired culture 2 effort. Next slide, please.

3 MS. MORROW: Thank you, Margie. Good morning, 4 Chairman and Commissioners. It's a pleasure to be speaking with you and 5 providing this update today. I'm Stephanie Morrow, a Human Factors 6 Engineer in the Office of Nuclear Regulatory Research and a member of the 7 Agency culture team.

8 Today I'd like to give you an update on our efforts to 9 intentionally manage our culture as part of focusing on our people. I'll 10 highlight some recent accomplishments, how we're measuring progress in 11 closing our culture gaps, things we've learned over the past year, and how we 12 plan to maintain our momentum. Next slide, please.

13 First, culture is important because it sets the tone for how 14 we interact with each other, how we make decisions, and how we innovate 15 and collaborate with one another. Our culture drives how we accomplish our 16 public health, safety, and security mission.

17 And even as we've experienced significant changes in our 18 work environment over the past year, our culture helped to ground us and 19 keep us focused on our mission. If anything, our recent experiences have 20 given us even more of an opportunity to reflect on what behaviors and norms 21 help us be successful. Next slide, please.

22 Over the past year, we've accomplished a lot, and not just 23 the Culture Team, but our entire agency as we've continued forward with our 24 transformation focus areas and found new ways to work together. This slide

10 1 includes just a few accomplishments that reflect the shifts we're making in our 2 culture.

3 We've engaged staff in culture change. One example is 4 growing our change agent network to over 200 people from across our 5 different offices and regions. We've increased recognition. We've done this 6 in informal ways, like taking time to acknowledge people in the moment, but 7 also more formal ways. The Innovate NRC Team incorporated a new 8 innovation awards category into our formal awards program.

9 We've enhanced communications. Last year our EDO 10 hosted a first-even virtual town hall meeting with all staff, and we committed 11 to continuing this practice on a quarterly basis as part of our culture 12 improvement plans. Over 1600 staff attended each of the two quarterly 13 meetings we've held so far this year.

14 And we shared good practices. We've used a variety of 15 forums, including our change agent network, meetings with individual offices 16 and regions, and Agency leadership meetings to intentionally create 17 opportunities for shared learning, exchange ideas on what's working, and 18 address common challenges.

19 We've also been documenting successes in our innovation 20 platform, IdeaScale. There are currently over 60 culture success stories in 21 IdeaScale, and that number continues to grow. Next slide, please.

22 As we make changes and implement new things, we're 23 using surveys and other assessment tools to measure whether we see 24 corresponding changes in what our staff experience. Last year, the culture

11 1 team took a holistic approach to assessing our culture, which included 2 integrating data from different surveys, interviews, and focus groups. That 3 was our 2020 baseline culture assessment.

4 This baseline allowed us to measure where we were starting 5 from. At that time, we also defined what our ideal culture would look like so 6 we would know where we wanted to go. We know that culture change takes 7 time, so we plan to do a full assessment again in 2022, two years after our 8 baseline.

9 For this year, we conducted an abbreviated survey as a 10 pulse check on our culture. We also have the results from the 2020 Federal 11 Employee Viewpoint Survey. The FEVS is one of the data sources we used 12 as part of our culture assessment process.

13 And both the recent Culture Pulse Survey and the 2020 14 FEVS provide indications of whether we are moving in the right direction with 15 our culture and climate. Next slide, please.

16 So I'll start with the Federal Employee Viewpoint Survey.

17 NRC staff took the 2020 FEVS in September, about one year after the launch 18 of our transformation initiative, and six months after our baseline culture 19 assessment. Our FEVS results show a four percent increase in the 20 employee engagement index, which is a key result we track as part of the 21 Agency objective and key results, or OKRs.

22 This is the highest our engagement index has been since 23 2011. Moreover, the FEVS questions that increased the most touch on areas 24 that have been a focus for our transformation efforts. Staff have seen

12 1 improvement in recognition, a 14% increase in the item, differences in 2 performance are recognized in a meaningful way.

3 And the next highest item, with a 12% increase, employees 4 are recognized for providing high quality products and services. Leadership, 5 an 11% increase in senior leaders generate high levels of motivation and 6 commitment.

7 Innovation, an eight percent increase in; I feel encouraged 8 to come up with new and better ways of doing things. And communication, a 9 seven percent increase in satisfaction with the information received from 10 management. Next slide, please.

11 The Culture Pulse Survey was open to employees in April 12 of this year, just over one year after our baseline assessment. The results 13 show slight increases in constructive behaviors. People feel like there's more 14 of an expectation to encourage others, help others to grow and develop, and 15 give positive rewards to others.

16 We also saw significant decreases in defensive behaviors.

17 People feel like there's less of an expectation to look for mistakes, remain 18 aloof, or do things for the approval of others, behaviors that can be detrimental 19 when practices in excess.

20 Finally, we saw improvements in measures of employee 21 involvement, communication, and perceptions of the agencys ability to adapt 22 to change, consistent with what we saw in the FEVS. Collectively, these 23 results tell us we are heading in the right direction toward closing our culture 24 gaps, but we still have room to grow and improve. Next slide, please.

13 1 Part of measuring our progress on a regular basis is learning 2 and adapting as we go. Weve learned that we need leadership at all levels.

3 The constructive behaviors we want to see in our ideal culture have direct ties 4 to our NRC Leadership Model, and they apply to everyone, not just those in 5 formal leadership roles - we can all encourage others and help others grow 6 and develop.

7 Our Culture Pulse Survey results show that managers and 8 senior executives experience our culture as much more constructive than 9 those at other levels. So s we move forward, we want to place even more 10 emphasis on role-modeling constructive behaviors so that our first-line 11 supervisors and non-supervisory staff also experience these behaviors as 12 valued and expected in our culture.

13 We've learned that we need to be very intentional with 14 connecting the dots. More communication does not have to be information 15 overload. And we need to be clear about how our activities fit together and 16 work to support accomplishing our mission so they don't feel like flavors of the 17 month or add to change fatigue.

18 We've also learned that it's important to actively create safe 19 spaces to bring our whole selves to work, fostering psychological safety.

20 Whether it's taking a moment in a team meeting to talk about what's going on 21 in our lives, or a larger forums to share perspectives on what's going on in the 22 world around us, like issues of racism and social justice.

23 Creating the psychological safety to speak up helps us to 24 make new connections with each other, more freely share information, and

14 1 ultimately makes us more effective at working together to accomplish our 2 mission. Next slide, please.

3 As we continue this effort, some of the ways we plan to 4 sustain our momentum and embed our desired culture into our processes and 5 practices include aligning our employee development programs and plans 6 with the NRC leadership model to address our culture gaps and develop 7 constructive behaviors, skills, and habits. Role model constructive behavior 8 with leaders and in teams to reinforce the behaviors we want to see in our 9 desired culture.

10 We're starting by facilitating conversations with senior 11 leadership teams in each office and region, and we plan to expand this 12 approach to first-line supervisors and non-supervisory staff.

13 We'll also include our NRC leadership model behaviors in 14 performance plans, starting with our Senior Executive Service and Senior 15 Level Service members, to make behavior expectations clear and ensure 16 accountability. Next slide, please.

17 Finally, I'd like to thank and acknowledge the rest of our 18 culture team. Patrice Reid, our team lead. Our full-time rotational staff, who 19 have taken on the heavy load of implementing our Agency culture plans. My 20 fellow part-time support who I've been working with on transformation since 21 the Futures Jam in 2019. And our senior executive sponsors, who continue 22 to provide their time and enthusiasm in championing this effort.

23 I also include our 200-plus member change agent network 24 as part of this culture team. Their grassroots involvement is critical to

15 1 reaching all parts of our agency.

2 Ultimately, our goal as the culture team is to facilitate 3 change, start conversations, and help apply what we learn so we can 4 consistently reinforce norms and expectations that will help us be a modern, 5 risk-informed regulator.

6 Thank you for your time. I will now turn it over to Candace 7 de Messieres to talk about Be RiskSMART.

8 MS. de MESSIERES: Thank you, Stephanie, and good 9 morning, Chairman and Commissioners. My name is Candace de 10 Messieres, a Technical Assistant for the Office of Nuclear Reactor Regulation.

11 It's an honor to be here today to share how the NRC staff can use the Be 12 RiskSMART framework in all decisionmaking.

13 As you know, the NRC has a long history and firm 14 foundation in risk-informed decisionmaking that includes many successful 15 applications of the risk triplet. That is, the set of three questions that NRC 16 uses to define risk: What can go wrong, how likely is it, and what are the 17 consequences.

18 While we have continuously improved how we use risk 19 insights to make better-informed decisions to meet our important safety and 20 security mission, risk-informed decisionmaking was not consistently applied 21 across the Agency or used for all types of decisions. And that's why I'm 22 thrilled to be here to discuss Be riskSMART. But before I do, I want to 23 highlight that while being risk-informed can provide for efficiencies and help 24 identify unnecessary conservatisms, it more importantly leads to a focusing of

16 1 attention on those issues of greatest importance.

2 This includes increased regulatory attention and oversight 3 on those items that matter for safety and security. We are making great 4 strides in fostering an Agency culture that recognizes the benefits that risk-5 informed decisionmaking can afford, as evidenced in the recent survey results 6 that Stephanie mentioned that indicate nearly 70% of staff thinks risk 7 information is important to accomplish our mission. Next slide, please.

8 The Be riskSMART risk-informed decisionmaking 9 framework was developed to provide staff confidence to consistently apply 10 and communicate risk insights for all NRC decisions without compromising 11 our mission. It is applicable in all NRC programmatic areas, including 12 corporate, technical, and legal support, and is written in plain language.

13 That is, the framework provides a common vocabulary that 14 can help users break down a problem and make sound decisions based on 15 the best available information, regardless of the application. The framework 16 helps remove the stigma from risk, the term risk, and replaces it with an 17 understanding of the value added by considering risk information, and how a 18 systematic approach can help ensure NRC is focused on items of greatest 19 importance.

20 Importantly, the framework does not revise or change 21 existing requirements or criteria, such as those supporting oversight or 22 licensing decisions. It also does not invent new concepts, but rather serves 23 as an umbrella to increase consistency, awareness, and usability of existing 24 risk-informed approaches.

17 1 Finally, the framework recognizes the multi-faceted nature 2 of risk and helps staff consider those multi-faceted risks systematically and in 3 a transparent manner, whether those risks are technical, programmatic, 4 reputational, information technology, human capital, or others. Next slide, 5 please.

6 This slide summarizes the key steps of the Be riskSMART 7 Framework. The first step is be clear about the problem. The problem could 8 range from a simple binary question that one faces on a daily basis, to a more 9 complex decision involving multiple individuals or entire organizations, such 10 as how to enhance a process to more fully realize the NRC's principles of good 11 regulation.

12 The next step is Spot. Spotting involves evaluation of the 13 risk triplet. But this step also explicitly includes consideration of both the 14 challenges and opportunities associated with a decision. That is, what can 15 go wrong or right. This step can involve both quantitative or qualitative 16 considerations.

17 Once spotting is completed, then you manage what you can.

18 That is, you consider strategies to reduce the likelihood or negative 19 consequences or enhance what can go right. These strategies could take 20 many forms, from increased communications or training, to additional 21 inspections or review resources.

22 The Act on the Decision step includes considering all 23 stakeholder perspectives and evaluating what you spotted and managed in 24 the context of the risk appetite for the decision.

18 1 In plain language, risk appetite or risk tolerance is simply the 2 amount of risk one is willing to accept. Depending on the decision, it can be 3 defined by an individual, a group of individuals, or an entire organization. And 4 it can be different for different aspects of the decision.

5 While simple decisions may not require extensive 6 documentation, the framework does provide a tool from enterprise risk 7 management called a heat map to help decisionmakers visualize the 8 challenges and opportunities, the effects of management techniques, and the 9 risk appetite associated with the decision. It is essential for all stakeholders 10 to be transparent about the risk appetite. A populated heat map can support 11 discussions about different perspectives and lead to an optimal decision.

12 The framework doesn't stop at the action. The Realize step 13 involves implementing the decision while managing what you can and 14 measuring your performance and progress. Realizing could involve 15 effectiveness reviews and other similar assessment tools to measure results 16 that can then be used as part of the continuous performance management 17 strategy to adjust and fine-tune.

18 The Teach step involves sharing knowledge to a broader 19 audience, enabling staff to apply best practices in new contexts to address 20 novel problems.

21 Finally, as a continual learning organization, the arrow 22 reflects the iterative nature of the framework, encouraging users to revisit any 23 of the steps at any time, or to even formulate a new problem. Next slide, 24 please.

19 1 Now I will shift to discuss how the Be riskSMART initiative 2 team is using a diagnostic tool to assess how the framework is inculcated in 3 our agency's culture. This slide summarizes the tool's features. The vertical 4 and horizontal axes of the graph depict the individual contributor and manager 5 roles in driving risk-informed decisionmaking.

6 In an ideal state, managers have established the 7 appropriate infrastructure and culture and play active and key roles.

8 However, risk informing starts earlier in the decision process and occurs 9 organically as part of everyday operations. The ideal state considers 10 everyone's opinion, individual contributors, supervisor, every layer of 11 management, and external stakeholders.

12 The importance of achieving the ideal state cannot be 13 overstated. This state fosters receptivity to new ideas and thinking, promotes 14 participative decisionmaking, transparency and trust, as well as encourages 15 diverse views.

16 Data collected by the Be riskSMART Initiative Team in 2020 17 reflected a majority of NRC staff see their organizational unit falling into the 18 center box, reflecting good progress. However, additional work is needed to 19 fully empower individuals.

20 Notably, the tool also includes specific strategies for an 21 organization to achieve the ideal state, such as enhanced communication, 22 training, and practices that engender trust.

23 I should also mention this tool is not the only way the Be 24 riskSMART team is tracking progress, as questions on the staff use and

20 1 attitudes towards risk-informed decisionmaking are included in several office 2 and Agency-wide level surveys, such as the surveys that Stephanie 3 mentioned.

4 While additional work is needed, the Be riskSMART initiative 5 is helping the Agency become a more modern risk-informed regulator, as 6 evidenced by the 2021 Culture Pulse Survey results I mentioned earlier, 7 showing that nearly 70% of staff thinks risk information is important to 8 accomplish our mission. Next slide, please.

9 I would now like to briefly reiterate that the framework 10 envelops the entire spectrum of NRC activities. Specifically, the team 11 deliberately evaluated the mapping of specific guidance or processes in all 12 areas, such as material safety, as highlighted on the top left of the slide, as 13 well as in corporate support, research, reactor safety, legal support, materials 14 safety, and security, emergency preparedness, and response.

15 As you can see, the framework is compatible with and can 16 enhance the many effective existing approaches and has broad applicability 17 across business lines. A few early examples are highlighted here for 18 illustration, and you will hear about a few more shortly from Erin and Tammie.

19 Next slide, please.

20 Being a risk-informed regulator is a journey. While it takes 21 a great deal of effort from all staff and unwavering support from senior 22 leadership, it is worth it. And Be riskSMART is there to meet staff wherever 23 they are.

24 A few items that demonstrate the team's progress are

21 1 highlighted here, including the issuance of a new knowledge management 2 NUREG, the launch of new NRC staff training, and the advancement of the 3 framework across the Agency through objectives and key results, or OKRs.

4 The NUREG provides a one-stop resource for using the 5 framework, including the case studies, mappings of discipline-specific 6 processes and guidance, and initial diagnostic tool and survey data. The new 7 training enables staff to explore how the framework can be used both 8 generally and to make discipline-specific decisions.

9 The course consists of three modules, including a general 10 introductory training video, a choose-your-own-adventure scenario section, 11 and a knowledge check.

12 Finally, aggressive goals have been established through the 13 OKRs, including that 85% of staff complete the training and that the framework 14 is then used in at least 50% of key decisions by the end of calendar year 2021.

15 In addition, OKRs encourage the featuring of the Be riskSMART application in 16 the staff IdeaScale success gallery platform.

17 I'm pleased to report that while we're still in the early stages, 18 offices are making great progress toward meeting OKRs, including that nearly 19 20% of staff have now completed the Be riskSMART training. Next slide, 20 please.

21 Lastly, I would like to thank the entire Be riskSMART 22 Initiative Team, ambassadors, and community of practice. It has been a 23 pleasure working over the past two years with such an amazing and diverse 24 team of talented and dedicated individuals empowered by our team lead,

22 1 Mirela Gavrilas, and our several executive sponsors.

2 Thank you for your attention, and with that, I'll turn it over to 3 Erin.

4 MS. CARFANG: Thank you, Candace. Good morning, 5 Chairman and Commissioners. My name is Erin Carfang, and I'm a Branch 6 Chief in Region I, with recent experience at both the reactors and materials 7 programs.

8 I have two examples where Region I had an opportunity to 9 demonstrate the concepts captured in the Be riskSMART framework Candace 10 presented. Next slide, please.

11 In the reactor program, there are many priorities inspectors 12 need to balance. One site in Region I recently experienced several safety-13 related breakers failing to actuate as expected over a four-month time period, 14 and licensee actions were slow to address the issue.

15 This resulted in the question of should additional resources 16 be assigned to reviewing the breaker failures, given the actions that licensees 17 had already taken to address the failures. Using this model, we spotted that 18 the licensee had followed the established maintenance practices and took 19 these issues into the corrective action program and identified the cause of the 20 failures.

21 The Resident Inspectors were concerned that the site's 22 planned actions were not timely and did not identify the correct extent of 23 condition. Additional electrical expertise was requested by the Resident, and 24 this request coincided with high demand for inspection resources.

23 1 The regional staff and management worked together to 2 align on the best path forward, reviewing inspection resources, adequacy of 3 licensee action, and concerns with equipment performance. We agreed that 4 the breaker inspection deserved priority.

5 And as a result of the extra inspection efforts, the licensee 6 agreed to pull forward the scheduled maintenance, schedule these tasks for 7 the high priority susceptible breakers this summer.

8 Additionally, a review of the other preventive maintenance 9 strategies and the procedures is being performed. Teaching is one of the 10 most important steps in this process. The Residents have presented to the 11 technical staff and management on the issue, ensuring that other inspectors 12 are aware of the potential failure when conducting inspections. Next slide, 13 please.

14 A gamma sterilization irradiator licensee requested 15 authorization to replaces sources in its panoramic irradiator. This activity is 16 typically performed by the source manufacturer and involves bringing in casks, 17 lowering the casks into the pool, and loading source racks with the source 18 pencil under water.

19 The irradiator serves a critical function because it used to 20 sterilize medical supplies. The licensee was initially prompted to submit a 21 license amendment request due to COVID-19 travel restrictions. However, 22 they now feel they need this authorization because they anticipate reduced 23 source replacement support from the source manufacturer in the future.

24 This brings us to the clear question should the region

24 1 approve the licensing action. So we evaluated the risk triplet and considered 2 the benefit of taking the action. The benefit of the action is improved reliability 3 for resourcing in the future for the licensee. While the challenge is potentially 4 improperly installed sources that could be brought to the surface.

5 The potential consequences include the benefit of ensuring 6 critical medical supplies remain available, while the challenge could result in 7 an extremely high dose rate to site personnel if the source is exposed on the 8 surface of the pool.

9 In managing the risk, the licensing staff considered the 10 likelihood of consequences is highly dependent on the training provided to the 11 licensee staff to conduct the activity safely, which is firmly within the licensee's 12 control.

13 The regional staff worked collaboratively with Headquarters 14 to ensure other views outside of the region were considered in the 15 decisionmaking. In consultation with NMSS and OGC, Region I acted by 16 issuing the license amendment in May of 2021, allowing the licensee to 17 perform the action.

18 The results of this licensing action include the inspection of 19 the source racks, given recent operating experience, as well as the inspection 20 being a high priority observation item for the regional staff while providing 21 operational stability for the licensee.

22 Knowledge transfer sessions are planned on this topic 23 within the region and the activity slated for inclusion in the Region I monthly 24 newsletter put out by our regional risk-informed decisionmaking working

25 1 group.

2 Thank you for your time. I will now turn it over to Tammie 3 Williams to talk about a corporate perspective using the Be riskSMART 4 framework.

5 MS. WILLIAMS: Thank you, Erin. Good morning, 6 Chairman and Commissioners, my name is Tammie Williams. I'm a Senior 7 Space Management Specialist in the Office of Administration, and I'm here 8 today to talk about how the Be riskSMART framework can be used broadly in 9 different ways beyond the Program Office to include a corporate support 10 perspective.

11 To help with that, I'll walk you through an overview of a 12 corporate example we have in our training materials on how Be riskSMART 13 can be used. We took the implementation of the Two White Flint restack 14 project, which is an example of how the decisionmaking process can benefit 15 from accepting well-managed risk, and assessed it using the new risk model.

16 Next slide, please.

17 NRC committed to releasing one floor of the Two White Flint 18 building to the General Services Administration, or GSA, by the end of the 19 fiscal year 2020. This was in support of federal-wide initiatives to reduce 20 office space across the country.

21 The restack process impacted several offices, Research, 22 OCFO, NSIR, Admin, OCIO, and multiple floors across the building. Early in 23 the restack process, the COVID-19 public health emergency occurred, and we 24 were challenged with finishing the project in a compressed timeframe and to

26 1 adapt many aspects of the project to a virtual environment for the first time.

2 The team needed to determine if there were any parts of the 3 process that could be expedited, delayed, or avoided without imposing 4 unacceptable risk. The team spotted where things could go right or wrong, 5 the consequences, and the likelihood of occurrence. For example, if 6 elements of the planning and office selection phases were missed, then 7 collective bargaining agreement, CBA, commitments with the union may be 8 impacted.

9 Experience showed a high likelihood for non-compliance if 10 these earlier phases were expedited, affecting the subsequent phases and 11 ultimately resulting in an overall schedule delay. Unbudgeted rents would 12 incur, and NRC would experience a reputational impact with the GSA.

13 The risks outweighed the potential benefit of saving time 14 during these early phases. Instead, the risks inherent in the physical aspects 15 of the move were managed by utilizing experienced movers and technicians 16 and relying on staff. The risk of conducting some activity virtually for the first 17 time was managed by increasing communication and coordination with the 18 office point of contact.

19 One of the actions taken to expedite the process was 20 conducting the move incrementally as the office selections were still being 21 completed. Additionally, the turnover process including a tour of the space 22 was conducted virtually with GSA.

23 The result what was realized was the majority of the moves 24 were completed by the end of the fiscal year. Decommissioning activities

27 1 took place in the October timeframe, and turnover of the space to GSA 2 occurred in November.

3 The timely completion of the project allowed the Agency to 4 release one floor to GSA, enhanced NRC's reputation for fulfilling 5 requirements on time, and reduced NRC's rent and real property-related 6 costs.

7 To teach what we've learned, this example is included as a 8 case study in the NUREG that Candace had mentioned earlier. We're also 9 implementing lessons learned in other projects. Next slide, please.

10 Looking back at the assessment exercise, it helped us to 11 realize that the Be riskSMART framework can be applied anywhere, and that 12 many of the existing processes fit within the framework or enhanced by it.

13 It showed us that staff at a working level in corporate support 14 are already organically assessing and managing risks in their daily activities, 15 with the most successful aspects of the project being those that were staff-16 driven, which is consistent with the diagnostic ideal state that Candace had 17 described earlier.

18 Additionally, we realized the importance and the key 19 interdependence on other NRC initiatives for the success of Be riskSMART, 20 including the Agency desired culture initiative and the associated tools, such 21 as speed of trust and the NRC leadership behaviors, that really helped move 22 us to the ideal state.

23 Thank you very much for your time, and I will now turn it over 24 to Nicholas Buggs to talk about the NRC enterprise IT roadmap and data

28 1 strategy.

2 MR. BUGGS: Why, thank you, Tammie. Good morning, 3 Chairman and Commissioners, my name is Nicholas Buggs and I'm a Senior 4 Enterprise Architect in the Office of the Chief Information Officer, or OCIO.

5 I would like to thank you for the opportunity to speak with 6 you today about the IT roadmap and the data strategy. Next slide, please.

7 As exemplified in the previous topics, the Agency continues 8 its transformation journey towards becoming a more modern risk-informed 9 regulator. Throughout this journey, the role of strategic investments in IT 10 modernization as mission enablers has become more apparent than ever.

11 Process changes, culture shifts, workforce enhancements, 12 and a greater reliance on collaborative tools in a mixed working environment 13 all require changes to the IT architecture to enable this transformative change.

14 Next slide, please.

15 The desire for change inspired by innovative thinking, a 16 culture of transformation, must be balanced by strategic thinking and 17 collaborative planning to ensure that the Agency is able to meet its goals in 18 the most efficient and effective way possible.

19 This balance can only be achieved when Agency leadership 20 is aligned on its priorities and promotes the implementation of new initiatives 21 that provide the greatest value to the Agency. Next slide, please.

22 To this end, the Office of the Chief Information Officer, in 23 consult with subject matter experts and leadership across the Agency, have 24 been working collaboratively to develop an IT roadmap. An IT roadmap is a

29 1 strategic planning tool that aids in the identification and selection of key 2 strategies and initiatives to support the continual operation and improvement 3 of the Agency's IT portfolio.

4 This approach enables the Agency to more effectively select 5 and implement the appropriate set of IT investments to maintain operational 6 readiness and to meet evolving Agency mission needs.

7 Initial development of the IT roadmap took a three-phased 8 approach. The first phase was designed to articulate the current state 9 architecture of the Agency. This involved outreach to subject matter experts 10 in every office and region to identify the key processes, stakeholders, and 11 systems involved in meeting functional and organizational mission objectives.

12 Through exploration of the current state, we developed a set 13 of observations that suggested opportunities for enterprise-level improvement.

14 The second phase was designed to articulate the future 15 state architecture of the Agency by validating the opportunities for 16 improvement and envisioning the desired capabilities to close operational 17 gaps and enhance operational effectiveness. With the current state 18 articulated where we are, the future state defines where we want to be.

19 The final phase includes the development of key strategies 20 and initiatives that are needed to achieve the desired future state over the next 21 three to five years. This phase of the effort is still ongoing as we seek to 22 consolidate strategic input and the set of projects needed to meet our strategic 23 objectives. Next slide, please.

24 Some of the key business drivers that have been identified

30 1 throughout this effort include optimized Agency business processes and 2 modernized workflow and workload management solutions to support greater 3 process efficiency, improved visibility into staff workloads, and more effective 4 resource allocation. Improved staff ability to access and analyze system 5 data, as well as documents within Agency repositories, as a means of 6 promoting the staff's ability to leverage the Agency's collective body of 7 knowledge.

8 Expanded use of modern IT solutions to collect and 9 communicate information from and to external stakeholders to enhance the 10 automated use of data for decisionmaking and downstream analysis.

11 And lastly, optimized workspace design to support a hybrid 12 working environment to accommodate full and part-time on premise and 13 remote work. Next slide, please.

14 Once complete, the IT roadmap will serve as a strategic 15 planning tool in the selection of strategic investments year after year. It will 16 help to ensure that fact-of-life changes, emergent requests, and risks and 17 issues are considered with respect to existing strategies, while providing 18 opportunities to consider strategic course changes when needed. As such, 19 the IT roadmap will enable agility while providing a common strategic thread 20 to support decisionmaking. Next slide, please.

21 One significant input into the continued evolution of the IT 22 roadmap among many is the Agency data strategy. The data strategy will 23 describe the Agency's overarching goals for the enterprise management and 24 use of data and the key strategies needed to achieve them.

31 1 These strategies are used as input to the ever-evolving IT 2 roadmap by articulating the strategic initiatives needed to maximize the NRC's 3 ability to use data as a strategic asset. The data strategy will consider our 4 workforce, our processes, our use of technology, and of course our 5 stakeholders.

6 It will articulate the Agency's approach to meeting its priority 7 objectives through initiatives designed to enhance our data culture, which 8 represents our collective desire to promote the use of data, inform the 9 evolution of our business processes, leverage modern capabilities, and 10 improve our ability to collect actionable information from external 11 stakeholders. Next slide, please.

12 The first step in the development of the data strategy was 13 the definition of the set of strategic objectives or goals that the Agency is 14 seeking to achieve. These goals include promote the secure management 15 and control of Agency data, promote the collection, transformation, and use of 16 data. Promote the continual data management and maturity of Agency staff 17 and processes, and promote the use of data-driven insights.

18 Pursuit of these goals will enable the Agency to continue to 19 advance in the areas of data architecture and governance, data management, 20 data maturity, and data analytics. Next slide, please.

21 With these goals in hand, the staff will work with 22 stakeholders across the Agency to gather information and to perform 23 assessments across a number of key areas. These areas included Agency 24 systems and tools, data assets, and data management maturity. These

32 1 inputs were used to identify key findings, observations, and recommendations 2 for improvement which served as the foundation for the development of 3 specific data strategies needed to achieve the data goals.

4 Examples of these recommendations include expanding the 5 integration of system data sources and the data warehouse, establishing 6 enterprise data standards for key cross-functional data sets, and filling data 7 literacy gaps through selective hiring, cross-training, and professional 8 development. Next slide, please.

9 These strategies not only identify new opportunities for 10 improvement, but they also encourage the continued exploitation of existing 11 data initiatives, such as expansion of the Agency's enterprise data warehouse 12 to the identification and ingestion of new data.

13 Expanded use of the Mission Analytics Program, or MAP, 14 through Agency-wide exploration of business analytics use cases. The 15 enhanced data stewardship of functional organizations through the localized 16 identification and curation of data for analytics. And promoting the expanded 17 use of data in the generation of evidence to support programmatic evaluations 18 to answer priority Agency questions. Next slide, please.

19 As you can see, the IT roadmap and the data strategy 20 provide the Agency with strategic planning tools to promote the identification, 21 selection, and implementation of key IT investments that will support the 22 NRC's transformation journey. These tools are meant to evolve in concert 23 with the business needs of the Agency's workforce, business processes, 24 technologies, and the stakeholders.

33 1 Again, thank you all for your time. I will now turn it over to 2 June Cai to talk about the transformation of the rulemaking process.

3 MS. CAI: Thank you, Nick. Good morning, Chairman and 4 Commissioners. I am June Cai, a Branch Chief from the Rulemaking Center 5 of Expertise in the Office of Nuclear Materials Safety and Safeguards. Today 6 I will be providing an overview of our recent rulemaking innovation effort and 7 showing you some of our completed products.

8 Before I begin, I would like to take the opportunity to 9 acknowledge a number of individuals who have contributed to the 10 presentation contents you will see and hear about today, including Caylee 11 Kenny, Dan Doyle, Mary Anderson, Alexa Sieracki, Cindy Bladey, Ilka Berrios, 12 and Tania Martinez Navedo. This has been truly a full team effort. Next 13 slide, please.

14 In 2019, the NRC's Rulemaking Center of Expertise self-15 initiated an effort to strengthen the Agency's rulemaking process to align with 16 our vision of being a modern risk-informed regulator. For this initiative, we 17 focused on identifying opportunities to enhance the quality and timeliness of 18 our rulemaking products and further strengthen staff and stakeholder 19 engagement.

20 We took a holistic, integrated review of all aspects of the 21 rulemaking process that were within the staff's control to look for ways to 22 innovate and improve. In carrying out our efforts, we leveraged the diversity 23 in the backgrounds and skillsets of our staff. We energized and empowered 24 our staff to take ownership so that innovation and continuous improvement

34 1 can become more fully integrated into our culture and routine operations.

2 Next slide, please.

3 We reviewed a variety of internal and external inputs and 4 resources. Based on our analysis, we identify opportunities for 5 enhancements at both strategic and tactical levels and we organize them to 6 five key enhancement areas. This simple graphic provides a summary of the 7 results. Next slide, please.

8 The first area, decision tools for developing a graded 9 approach to rulemaking, establishes a framework for facilitating a strategic 10 risk-informed decision path for rulemaking and expected to provide the most 11 impact. Next slide, please.

12 The other four areas help develop and implement 13 rulemaking activities in more efficient, effective, innovative ways. These 14 include strategic thinking and planning and prioritization; process 15 enhancements and efficiencies, such as application of agile project 16 management concepts; product development and concurrence, such as 17 modernizing review and concurrence practices; and organizational 18 effectiveness, such as serving as early adopters of Agency transformation 19 tools and processes.

20 The full set of initiatives are designed to operate together to 21 support the entire rulemaking process. We issued a report about our effort in 22 July 2020 and have been implementing and refining the associated actions 23 since that time. This is a living and continuous effort. We have completed 24 some products, others are in progress, and some new efforts have been

35 1 recently started.

2 Over the next several slides, I will be highlighting some 3 examples of our recently completed key products. Next slide, please.

4 We call this our super infographic. It serves as our 5 navigation tool and contains references to all the products we have developed 6 to support the full process, from initiating rulemaking to developing and 7 reviewing rulemaking products, to highlighting successful outcomes. We 8 plan to keep this graphic updated as we develop additional items. Next slide, 9 please.

10 To begin the rulemaking process, here are the two tools we 11 have developed to help guide the decisionmaking to identify the best path 12 forward. The first tool helps us determine if rulemaking is the most 13 appropriate path to meet a regulatory need.

14 And the second tool helps us to develop a tailored approach 15 to meet that need in a risk-informed manner based on consideration of factors, 16 such as the complexity of the issues involved and effective stakeholder 17 engagement. Next slide, please.

18 An important consideration on many rulemaking products is 19 appropriate engagement with Agreement States as our regulatory partners in 20 the materials safety area. This infographic serves as a helpful tool for both 21 internal Agency use and for communicating with our stakeholders to explain 22 why and how we engage Agreement States through the different stages of the 23 rulemaking process and the expected benefits.

24 Once we delve into the developmental process, the tools

36 1 described in the next three slides are helpful for supporting different project 2 stages. Next slide, please.

3 First, to facilitate overall management of rulemaking 4 products, these two infographics provide helpful insights on the potential 5 application of concepts from agile project management methods, such as the 6 general Scrum framework, which is traditionally used in the IT arena to 7 achieve improved project outcomes, such as increasing product quality and 8 minimizing project redirection. Next slide, please.

9 Once we move into development of our rulemaking 10 packages, our staff can use these two infographics, which provide best 11 practices and helpful tips for writing quality to develop well-written materials 12 and reduce the need for significant revisions. Next slide, please.

13 After we complete development of the rulemaking package, 14 we use many of the practices on this infographic to facilitate the review and 15 concurrence process in a more streamlined manner, leverage new technology 16 tools, and tailor approaches based on specific project needs.

17 In identifying and developing these practices, we 18 coordinated closely with the Agency's Process Simplification futures team, 19 and we designed these to complement their efforts. Next slide, please.

20 Finally, once our rulemaking projects are completed, we 21 have recently started publishing publicly available summaries called 22 rulemaking highlights to help illustrate how specific rulemaking activities have 23 contributed to the Agency's mission and how staff have applied specific 24 innovations and efficiency measures during the development process. Next

37 1 slide, please.

2 Next, I'd like to share some examples of the impacts of our 3 progress. To begin with, we have been able to develop tailored approaches 4 to meet different regulatory needs and schedule needs. I have three specific 5 examples to share here.

6 First, for the rulemaking on the development of a risk-7 informed, technology-inclusive regulatory framework for advanced reactors for 8 Part 53, we're performing significant public outreach, including releasing 9 preliminary proposed rule language for public discussion and holding monthly 10 public meetings and workshops.

11 Second, by applying process efficiencies, we have been 12 tightening timeframes needed for conducting rulemaking to address spent fuel 13 casks' certificate of compliance amendments, while at the same time being 14 able to accommodate an increase in volume of requests during Fiscal Year 15 2021.

16 Third, we have taken focused steps to provide additional 17 opportunities for Agreement State members to participate directly on working 18 groups on a number of different rulemaking projects in the materials area.

19 The next example of an area of impact is we have been able 20 to demonstrate the ability to quickly adapt to using new technologies, such as 21 new IT tools for conducting collaboration and parallel reviews. Also, we 22 recently used IdeaScale platform to seek Agency-wide input for the 23 categorical exclusions from environmental reviews rulemaking.

24 We have also been able to quickly incorporate new process

38 1 improvements, such as the new commission paper alignment process into 2 ongoing projects. Furthermore, we have been able to enhance how we 3 communicate with the stakeholders using more innovative and modern tools, 4 such as the case studies I just described, as well as through multiple social 5 media channels.

6 Finally, we have been able to strengthen our organizational 7 capacity. Our staff have been able to learn to use new IT tools. They've 8 been able to broaden their project management practices. And they've been 9 able to work together on cross-discipline topics.

10 In addition, all these efforts will help us facilitate knowledge 11 management and the growth of the next generation of rulemaking staff. Next 12 slide, please.

13 In closing, I'd like to share some reflections on what worked 14 well in this initiative from a change management perspective. First, we 15 recognize that there's always room for improvement. Even in an area like 16 rulemaking where there are many requirements we have to meet and 17 processes and procedures we have to follow, there are opportunities to 18 innovate and make improvements.

19 Second, this has truly been a grassroots effort, and we were 20 fortunate to also have strong management support, which helped amplify our 21 progress.

22 Third, it truly takes a village, or in this case, a center of 23 expertise. We have staff across our organization working and pulling 24 together to achieve these results.

39 1 Next, I want to highlight the importance of focusing on 2 progress, not perfection. Incremental improvements can truly add up over 3 time to make more significant and lasting impacts.

4 And finally, I want to emphasize the importance of creating 5 a safe environment for taking informed risk and applying creativity. In order 6 to innovate, our staff has to feel comfortable taking some calculated risks and 7 try out new things, even if they don't work out perfectly. The only way we can 8 grow and adapt is if we keep trying and learning from our experiences. Next 9 slide, please.

10 Our rulemaking innovation efforts are expected to mature, 11 grow, and adapt over time. NRC's Rulemaking Center of Expertise is 12 committed to continuously strengthening the Agency's rulemaking process 13 and serving as a model of federal rulemaking excellence.

14 I am honored to have been able to share about innovation 15 efforts with you today and for this chance to represent the committed and 16 talented staff from the Agency's Rulemaking Center of Expertise. I look 17 forward to answering any questions you may have. I will now turn it back to 18 our EDO for closing remarks.

19 Thank you.

20 MS. DOANE: Okay, thanks, June.

21 As you've heard today, we have realized so many 22 accomplishments on our journey, while also responding to the dynamic 23 environment in which we are currently operating. Our operating experience 24 has taught us that transformation is absolutely necessary and should be

40 1 maintained. We will continue to closely monitor our external environment in 2 order to appropriately and timely respond to outside factors.

3 We must do this so as to not find ourselves in a position 4 where we are at risk of not being able to keep pace with what is occurring 5 around us.

6 We will continue to encourage and support innovative 7 approaches to carrying out our important safety and security mission. And to 8 do this, we'll work on realizing our ideal culture, one that supports creativity, 9 diversity in all its forms, and appreciates the need for change.

10 We will continue to carry forth the valuable lessons we have 11 learned throughout our transformation journey to best position ourselves for a 12 successful future. As you've heard today, while there is always room for 13 innovation, there's always room to learn as well, especially in a safe 14 environment for taking informed risks while creating these innovative 15 approaches.

16 These past few years have been about developing a strong, 17 clear plan for transformation. Now that the framework and goals have been 18 established, we want to empower the staff to continue to evolve these efforts 19 and truly make them their own.

20 I wanted to say thanks to these individuals at the table with 21 me today and to the core team, especially Maria Arribas and Mary Frances 22 Woods, who is on a rotation to the OEDO's office from OGC, for their hard 23 work in putting this material together. I also wanted to acknowledge all those 24 that have supported in one way or another the initiative teams, and to the staff

41 1 at large for embracing and helping to achieve all of these successes.

2 As I have said many times and I feel like we should say it 3 again, our staff is what makes the NRC a great place to work.

4 This concludes our remarks, Chairman and Commissioners, 5 and we are happy to take your questions.

6 CHAIRMAN HANSON: Thank you very much, Margie, and 7 thanks to the staff for that presentation. I appreciate it very much. We're 8 going to start the questions this morning with Commissioner Caputo.

9 COMMISSIONER CAPUTO: Good morning, everyone.

10 Thank you very much for all of these presentations. It's clear that there is a 11 significant amount of staff engagement and creativity being applied here, and 12 I thank everyone for their presentations.

13 Margie, I'm going to start with you because as Executive 14 Director, the success of the Agency's transformation rests upon your 15 leadership.

16 This meeting reflects the Commission's recognition of the 17 need for transformation, and our focus on it over the past several years.

18 Transformation, innovation and agility have entered our lexicon and have 19 become frequent buzzwords.

20 I see innovation as an integral part of our value of 21 excellence, high quality, continuously improving and self-aware.

22 Transformation, however, is different and implies a much larger, wide ranging 23 change.

24 We spoke early on about the goal of our transformation

42 1 efforts and the result was to become a modern risk-informed regulator.

2 Margie, you stated that we are making progress in all 3 transformation focused areas. Other than culture surveys, what objective, 4 measurable metrics are you tracking to support that conclusion?

5 MS. DOANE: Okay. Thank you, Commissioner. So let 6 me focus on what the areas are, so people, technology, and Be riskSMART, 7 and I'll come up with the fourth. I don't know, for some reason I'm blanking.

8 But any focus areas, what we have done is a number of different things.

9 So let's focus on our people. From the perspective of our 10 people, the objective tracking measures that we've been using are two-fold.

11 First, we've been monitoring what's been going on with the 12 creativity in our journey, our staff journey, and also the other activities that the 13 initiative team focused on our people from the perspective of mid-career staff 14 and have been identified as what would be helpful to improving their journey 15 and also their experiences. And so, as you know, we have created a number 16 of different tools for them to use.

17 And so the way that we've objectively monitored whether 18 those tools have been helpful is by monitoring how many people touched the 19 Web site, how many people use it and also we get feedback regularly on that 20 Web site. So that's one thing that we're doing and focusing on our people.

21 Another area that we're looking at focusing on our people is 22 the culture. And as you heard we're doing Pulse Surveys, and both our FEVS 23 and our Pulse Surveys have improved in meeting our desired culture. So that 24 means --

43 1 (Simultaneous speaking.)

2 COMMISSIONER CAPUTO: Right. We've already heard 3 about those this morning. Can you give me a different --

4 MS. DOANE: Okay.

5 COMMISSIONER CAPUTO: -- set of metrics?

6 MS. DOANE: Okay. So just heading into these, okay?

7 And then IdeaScale, I went over a few of those metrics, who has touched the 8 site and where our success stories are and things like that. So those are the 9 metrics that we're using for innovation.

10 For technology, what we're using are various tools like how 11 to measure whether we're retrieving data better, whether we're using -- we 12 keep all kinds of metrics on whether we're using the technology.

13 So for example, I don't have the figures with me, but we 14 tracked how many people were using Teams. And our daily metrics showed, 15 like for example when we went into the pandemic that we went from almost 16 little to no usage to immediate usage, very effective usage. And then we also 17 tracked our transition to Teams to see that we had the same uses and effective 18 uses.

19 So we had various metrics, all kinds of metrics in that area.

20 COMMISSIONER CAPUTO: So you're tracking 21 implementation of a lot of tools and whether or not the tools are being used.

22 Do you have any results to suggest it's actually improving staff productivity or 23 increasing efficiency?

24 MS. DOANE: Yes. So let me start with -- and Annette's at

44 1 the table so she can help me, but let's talk about like productivity, for example, 2 on process simplification.

3 For process simplification, we focused on SECY papers, as 4 you know, and depending on how -- I've seen two different data measures for 5 this, but depending on how you are measuring whether you've seen efficiency 6 in this area, we've seen a 30 percent, at least a 30 percent reduction in the 7 time it took to put papers forward.

8 So that's an example of a very specific outcome and a 9 measurement from what we did before and what we did now and how that's 10 an improvement. Like I said --

11 (Simultaneous speaking.)

12 COMMISSIONER CAPUTO: So do you have any way of 13 comparing the time the staff has spent training on these tools, learning these 14 tools and making advances, transformation advances? Do you have any way 15 of comparing that to those productivity improvements to see if we're, you 16 know, seeing benefits from this activity?

17 MS. DOANE: I mean, I think we have a way of measuring 18 the benefits. The time it took to get there is not -- I don't think -- we do 19 measure it because they have to keep track of their time. But we haven't 20 done a comparison of, like, for example, the first -- so let's say we have on 21 these routine papers we've seen a 30 percent increase in the time it took to 22 get the papers to the Commission, right?

23 And we haven't done a comparison to well, did it take 30 24 percent of staff, more than 30 percent of staff time to get trained and to learn

45 1 these things? No.

2 And because that's like a one-time investment, you would 3 have to do that over a series of time. In other words, you would have to watch 4 these things over a series of time to know whether your initial investment.

5 But, no, we haven't done that function, that comparison.

6 COMMISSIONER CAPUTO: Okay. I think that would be 7 a worthwhile review just to understand how much benefit we're getting out of 8 the amount of staff time being invested.

9 I'm going to move on to a question about IT. So Nicholas 10 discussed a lot of important concepts and aspirational goals with regard to the 11 IT roadmap. But Margie, I'm going to address this question to you.

12 It would be more fairly addressed to the Chief Information 13 Officer and certainly unfair to ask Nicholas. But let me start by saying our 14 people are our most important asset. And we have strategic workforce 15 planning to ensure we have the right people for the right work at the right time.

16 They're our most important tool.

17 And our second biggest line item is IT. And while I 18 appreciate Nicholas' presentation and consider it a step forward, I'm not clear 19 how detailed the roadmap will be.

20 So my question, Margie, and my understanding is that it is 21 not ready yet and thus has not been available to inform budget development.

22 So my question to you, Margie, is how much longer are we 23 going to continue to spend in excess of $150 million on IT without having a 24 comprehensive plan to ensure we have the right tools at the right time to meet

46 1 the Agency's evolving needs and prevent last minute scrambling to address 2 easily foreseen issues such as software obsolescence and Wi-Fi installation?

3 MS. DOANE: Okay. Let me see if I can -- thank you, 4 Commissioner, for these questions. Let me see if I can answer them in the 5 way that you put them.

6 Okay. So how much longer until we have a comprehensive 7 plan? We are anticipating this for later during the summer that we'll have the 8 plan. So I appreciate that the budget formulation process that's going on now 9 hasn't been informed by that process.

10 But we certainly have had planning in place for many, many 11 years. It's how we were ready for the pandemic in going all virtual. What we 12 didn't have, and I will tell you this honestly, so we had, for example, we knew 13 we needed to move to laptops. And we knew needed to move to Wi-Fi. We 14 knew we needed to do a lot of things.

15 So we were bringing the Agency down. We were reducing 16 the number of staff and corporate investment. We were going down, not up.

17 So we were trying to do a number of different things in a sequenced event so 18 that we -- sequencing them so that we could meet this plan, like, for example, 19 get laptops, Wi-Fi and different things, but we were doing it over a number of 20 years because we never had the funds to do it in any one year. So we did 21 have planning.

22 We had a number of things that were in the queue. We had 23 laptops. We had Office 365 that was coming down. We had Skype at the 24 time, as you know, Commissioners. We've had this conversation before.

47 1 So we had a lot of things that were good things that were in 2 the planning and were being carried out as funding became available.

3 But there were pressures on this same corporate support 4 funding. And so we couldn't fund all of them. So it looks frenetic as you tried 5 to then fund them later or pick them up from resources that become available 6 but in fact it wasn't. We had these things listed, and we always knew that 7 these were where we wanted to go.

8 What the transformation did for us is it helped us focus.

9 And I give the OCIO credit for this, Dave Nelson. It helped us focus on why 10 we weren't putting these technology advances into our working environment.

11 And it was because we were not focused on technology adoption for our staff.

12 We didn't look at what was keeping them from doing it. And that's really been 13 the transition and that's made all the difference.

14 So I think we have had good planning. I think we've got a 15 lot of great ideas. We just haven't been able to implement them.

16 Why is this different? So then you might say well, then, 17 why do you need this IT roadmap and data strategy? Because information 18 and data are flowing in at an increasing rate, and technology is continuing to 19 evolve. And so the benefit that we have now that we're getting this -- the staff 20 is now accepting using this technology is now we have a transformation that 21 will enable us to use technology at a faster rate and to make better decisions.

22 We also have --

23 COMMISSIONER CAPUTO: All right. My time has 24 expired. So I'm going to cut you off right there. I think it's usually important

48 1 when managing expenditures of that size that there be a significantly detailed 2 plan to ensure we're spending the money prudently and meeting needs.

3 And I'm going to beg my colleague's indulgence. This is 4 my last meeting. There's one last comment I would like to make. I'm sorry 5 that we haven't had a speaker here today from the CFO's office. And I want 6 to take a moment to commend them for their work on budget execution 7 dashboards.

8 This has been a strong step forward toward the better use 9 of data in budget formulation and execution. And I think it clearly reflects a 10 fair amount of hard work, and I appreciate their efforts.

11 However, I would characterize this as an innovation, not a 12 transformation. Our oversight and licensing workload is down 45 percent 13 since 2016 and likely to decrease further given the potential for as many as 14 five reactors to close within the next year.

15 In 2021, this workload was the equivalent of 439 FTE. That 16 means for every FTE doing oversight and licensing work, there are 5-1/2 17 others doing other things.

18 In 2016, that ratio was 1 to 3-1/2. This is not a sustainable 19 trend. I personally believe this is an enterprise risk, and there is a strong 20 need for transformation in budget formulation execution to manage it. Thank 21 you for the time, Chairman.

22 CHAIRMAN HANSON: Commissioner Wright.

23 COMMISSIONER WRIGHT: Thank you, Mr. Chairman.

24 And thank you so much, everyone, for your presentations.

49 1 I continue to be impressed with your enthusiasm for 2 transformation and the impact that it's having on the way we do business and 3 even at last week's innovation celebration, staff called IdeaScale, I think, an 4 ecosystem of innovation, if I remember correctly. And then that connects and 5 inspires us. And I like that term, you know. And I think that it also applies to 6 the network of interconnected transformation projects.

7 We could also say that we have an ecosystem of 8 transformation that drives us toward our goal of becoming a modern risk-9 informed regulator. So with that in mind, I've got several questions I'd like to 10 get to. And Margie, I'm going to stay with you.

11 There have been several change initiatives undertaken in 12 the past decade and longer. But it seems transformation seems to be one of 13 the most or the more successful in terms of sustained engagement and 14 somewhat measurable results.

15 You know transformation, it's a broad term. I've really not 16 personally been a big fan of that word as a standalone for the Agency because 17 I kind of agree here with Commissioner Caputo here. I think it's more what 18 we're trying to do and what we're going through at the Agency, it's really big 19 on innovation. It's really big on adaptation of how to use data in better ways.

20 You can call it transformation if you want. That's fine. But 21 I do think it's probably more than just one word. I mean, we're morphing.

22 We're changing. We're constantly moving. And I guess that's what you 23 mean when you're using the term transformation.

24 But it doesn't work unless we have a proper vision of where

50 1 we want to go, right? What's our end game? Otherwise I think it's 2 problematic.

3 What do you think is different about this effort that is 4 contributing to the successes that you have seen so far that you point to?

5 MS. DOANE: So I think -- first I will tell you that past 6 transformations had their own successes like, I know, the Reactor Oversight 7 Program, the ROP. So there have been successful transformations. It's just 8 that we didn't sustain a culture of transformation and innovation.

9 So what I think is really different about what we're doing 10 today is we're really marrying these transformation efforts. We can call them 11 innovations or I'm open to any of it. But they're just good ways of enabling 12 staff to do their work better.

13 And so we're marrying that with really paying very close 14 attention to the health of the organization. And having a real effort on culture 15 and what a culture that embraces change looks like, why that's good for the 16 staff, making this like a really, making every day a great experience for the 17 staff as they go through doing different things and looking for ways to improve 18 how they do their work, all with an ending towards making better decisions.

19 That's in the end. That's our product, right? That's the 20 NRC's product. It's our decision-making. We're a regulator. We want to 21 make better decisions, more efficient, more timely, better, taking into 22 consideration more data.

23 So the vision is that we'll get to a point where our surveys 24 are showing that this culture has really moved. We've seen a real shift. And

51 1 so now we have the culture and how the organization is marrying up with these 2 different things that you're hearing about. You know, you can't help but be 3 excited about all the different things that staff are working on. But there 4 needs to be this marrying of the two, the health of the organization with all the 5 goals that we're setting.

6 COMMISSIONER WRIGHT: I --

7 MS. DOANE: And I think -- I'm sorry.

8 COMMISSIONER WRIGHT: No, go ahead, finish. Go 9 ahead.

10 MS. DOANE: So that's really the vision. The vision is, you 11 know, it morphs into -- what it morphs into is no longer this like push down 12 from the top saying, you know, we have to do things better. We have all these 13 areas of stagnation, different things, to the staff, telling us. And they're 14 already doing it. You're not going fast enough. You're not doing this. You 15 know, we could do more. And that's really the vision of a transformed agency 16 for the NRC.

17 COMMISSIONER WRIGHT: All right. So, I mean, I'm a 18 huge fan of the use of people like Embark and Innovate and IdeaScale and 19 those. I'm a big fan.

20 So this brings me to two questions. And it's two different 21 questions, but I'm asking them at the same time.

22 So how are we capturing those lessons learned for future 23 initiatives? That's the first one.

24 And my second question, how often are you recalibrating or

52 1 refocusing on the vision as a whole for the Agency? Because you mentioned 2 just a second ago that it's constantly morphing. So how often are you taking 3 a hard look and recalibrating?

4 MS. DOANE: Okay. So first how are we capturing the 5 lessons learned? So we are doing that in a number of different ways. So 6 mostly we use our Web site to capture where we have gone in these various 7 initiative areas and then we get feedback from the staff and then we're 8 continually trying to answer that feedback.

9 And so depending on how significant the issue is, we'll take 10 it on all the way up to the EDO's level because it's something that really needs 11 that kind of attention or it will be done at the grassroots level.

12 But we're trying to identify a number of different areas where 13 staff can give us feedback. And so that's how we're really learning the lesson 14 of are we moving in the direction that we want? And maybe that goes to the 15 second really about, you know, are we looking at the vision?

16 The other thing we're doing is we know that once we built 17 the infrastructure and thank you for mentioning Embark. It's very important.

18 They're our innovation spark plug if you ever -- you know, I know you've 19 worked with them and you've gotten briefings from them many times.

20 But what we really want to see is we want to see a place in 21 the Agency for these things long-term. And so what we're looking at now is 22 whether we have the right organizational structure. We want to make sure 23 that we can sustain these efforts.

24 We're trying to, like I said, survey and do other things. Look

53 1 at the products that are being produced and see whether we really do see 2 innovations or changes in small groups and also in large projects. There are 3 really important things that are going along to make sure we are seeing 4 progress in these areas.

5 So we're constantly looking at these issues. We have a 6 directorate that pays attention to the survey results, pays attention to the 7 issues that are going on. The assistant for operations, as you know in the 8 EDO's office, is really focused with a good portion of her time on 9 transformation looking at these various things.

10 And the way that we make sure that we are capturing the 11 lessons learned is every other Wednesday we bring a transformation example 12 to our executives, every Wednesday, every other Wednesday we do this, so 13 that they're hearing what's going on around the Agency and other parts of the 14 Agency. And they bring them into their organization.

15 And then finally IdeaScale is really good at capturing 16 successes. And what it's linked with is something called Nuclepedia, which 17 just like Wikipedia, it has pages of how we've learned different lessons from 18 going forward during transformation so that staff can either mimic these or 19 learn from these experiences or use in a training tool.

20 So there's about, you know, too many different things for me 21 to capture. But I hope that that's given kind of a glimpse of how we're 22 capturing the lessons learned and then relooking at the vision by we still have 23 a steering committee for transformation, and we have the AO looking at this.

24 We also have survey results, and these different issues that are brought to us.

54 1 And then we bring them to the executive core.

2 COMMISSIONER WRIGHT: Okay. Thank you. I've got 3 about a minute and a half left. I'm going to switch over to Candace for a 4 second. So, Candace, it's good to see you. And I'd like to again take this 5 opportunity to thank you for all you did as my session coordinator for my RIC 6 grid panel that I'm on already. So thank you. I look forward to working with 7 you in the future, too.

8 You mentioned the success of the Be riskSMART 9 framework for risk-informed decision-making. We've heard many examples 10 of its use today from Erin and from Tammie as well during the business line 11 briefings.

12 So Candace, but Erin and Tammie can jump in if they want 13 and they're welcome to add their thoughts, now that you've had some 14 experience using the framework in multiple parts of the Agency and you've 15 completed your first Pulse check, do you think any revisions are needed?

16 And if so, what might they look like?

17 MS. de MESSIERES: Thank you, Commissioner. And 18 we are still in the early phases of deployment. But I think some of the 19 foundational work that the initiative team completed has really paid dividends 20 in this regard.

21 For example, we built off of the multiple decades worth of 22 experience that pockets of the organization has in risk-informed decision-23 making. And we also exercised extensive pilots and case studies.

24 We sought staff feedback throughout the process of

55 1 development at multiple office and divisional level meetings and individual 2 targeted outreach across disciplines.

3 And one of our guiding principles was that we first do no 4 harm. We deliberately ensured that the framework could accommodate, be 5 consistent with, or enhance the many successful current NRC processes and 6 guidance. And therefore, it's broad, it's flexible, it's scalable.

7 And, you know, given our initial progress and successes, I 8 don't think that, you know, revisions are needed at this time. But as you've 9 heard throughout this whole meeting, you know, we are a continual learning 10 organization.

11 And I think that we are always open to feedback. I mean, 12 throughout the whole process, we've been open to feedback, and we've 13 adjusted as needed. The little arrow on the Be riskSMART logo there talks 14 about the iterative nature and that continual learning.

15 And so while I don't think we need revisions at this time, I 16 would say that I think a lot of additional work will be needed in the future to 17 continue the positive progress to increasing the use of risk-informed decision-18 making and also maintaining a supportive agency culture.

19 I think this is really continual work that needs to be done in 20 this area every single day at all levels of the organization.

21 COMMISSIONER WRIGHT: Thank you so much for that.

22 I just want to, again, focus back on the vision. I really appreciate the fact that 23 we are constantly looking to improve and stuff. And that does not happen if 24 you do it just to do it, you're not accomplishing anything. So the fact that you

56 1 are taking a good hard look at how you are doing things and looking how you 2 can improve on that I think is very important and just not checking a box. So 3 thank you so much. Mr. Chairman, I yield.

4 MS. de MESSIERES: Thank you.

5 CHAIRMAN HANSON: Thank you. Thanks again to the 6 staff for the updates this morning. You know, as I've said before, I think the 7 challenge for transformation going forward is kind of fourfold.

8 The first is to keep the safety mission front and center in all 9 of our efforts. The second is to sustain a culture of innovation throughout the 10 Agency.

11 The third is to realize the return on investment from past 12 initiatives to make sure that the learnings, as Commissioner Wright pointed 13 out, are getting captured and that we're institutionalizing some of the business 14 processes that we've been driven to change. And the fourth is then to expand 15 and apply transformation thinking to some new areas.

16 I was really encouraged at the level of participation 17 throughout the Agency in transformation efforts, right? There were, you 18 know, something on the order of 1,600 people that were engaged in IdeaScale 19 events. But I have also heard some concerns from staff that, you know, 20 transformation is really just, you know, a means to cutting people or 21 regulations or that it's change just for the sake of change.

22 And Margie, I guess I'd like to hear from you just a little bit 23 about how Agency leadership are responding to some of those concerns.

24 MS. DOANE: So, thank you, Chairman. And I think that

57 1 one of the main things that we try to do in all of our communications is to make 2 sure that this is, and you heard it with the speakers, but make sure that it's an 3 iterative process so that we can get our messages out.

4 And we always start with, you know, what's our mission?

5 It's what unites us. We explain to staff that our mission is something that 6 unifies the staff and the staff believes, every one of them believes in that 7 commitment. And we make sure that we focus ourselves in those meetings 8 of thinking about how did it enable us to meet our mission, to do better, not to 9 somehow impact it.

10 So I understand that concern. In fact, a lot of what we did 11 at the very beginning when we were setting up these initiatives is to ask 12 ourselves, we kind of kept this as a mantra, like, you know, how do we make 13 sure that whatever we do doesn't impact our ability to meet our mission but 14 instead enhances it?

15 Okay? So first of all, one thing I would tell you is the 16 concerns about staff, that this is just a means of cutting or doing less. We 17 counter that every day with both our communication from our executives but 18 also we have to be role models.

19 And so one of the things we've said as being a role model 20 to show that we are improving in this area is the way that we're looking at the 21 culture, to empower staff, to let them bring their perspectives forward, their 22 whole self forward, that is a culture. If we get to our desired result, we will 23 have many more perspectives to bring to the issues. And when staff feels 24 comfortable speaking up, right, then we have more perspectives so we can

58 1 make better decisions.

2 So we're countering this in a number of different ways.

3 Those are two examples.

4 And to make sure that it isn't change for change, we have 5 had to refocus this actually. Because we have had situations where we got 6 a little excited about, like, you know, doing things.

7 And we didn't put in any guardrails because we just wanted 8 to spark innovation. Then we just had innovation everywhere. It was very 9 frenetic. And some of the staff said, you know, it's too much.

10 So we're putting in guardrails to make sure that we are 11 identifying the problems, and we're doing this in all different areas. You heard 12 Candace talk about it, an IdeaScale. You have to identify this problem.

13 We were also looking at Embark, Innovation Accelerator 14 and making sure that we're really focusing on the important problems and that 15 we're prioritizing in a way that really moves the Agency forward with what's 16 most important.

17 So the first year was all about just running as fast as we 18 could to show innovation successes and things like that. But now we're really 19 trying to hone that in a bit, put guardrails in to make sure that we're focusing 20 on those that we're prioritizing and getting results.

21 CHAIRMAN HANSON: Thanks very much for that, Margie.

22 I appreciate your response on that.

23 June, I've got a question for you. I kind of want to follow 24 through on this theme just a little bit. I was really interested in your

59 1 presentation about kind of improving the rulemaking process, particularly 2 some of the things that we're doing internally.

3 And I think it's important for us to look at those ways in 4 making the rulemaking process more efficient. But I want to make sure that 5 efficient doesn't mean reducing opportunities for public involvement in the 6 process. And so can you talk a little bit about what staff is doing to strike the 7 right balance there?

8 MS. CAI: Thank you, Chairman, for the question. So 9 stakeholder engagement is definitely very important to us in rulemaking. And 10 it's one of the centerpieces of our decision-making tools that we built in.

11 I mentioned one of the tools talks about developing an 12 approach, address whatever the issue and the needs are. And within that 13 consideration is definitely how do we effectively involve stakeholders?

14 So we have different examples listed, different guidance, 15 you know, tools that staff can use, public meetings, putting documents out, 16 Federal Register notices, you know, social media communications.

17 And I also mentioned Agreement States involvement. And 18 we have our separate infographic on how to engage effectively with the 19 Agreement States.

20 So in addition to the focus on the process improvement 21 efficiencies, the effective stakeholder engagement involvement was also a 22 strong focus area for this whole innovation effort and rulemaking.

23 CHAIRMAN HANSON: Okay. Thank you very much.

24 Stephanie, I thought your discussion on culture was really important. And,

60 1 you know, I think as Commissioner Caputo has said on occasion you get what 2 you measure in a lot of these things.

3 And one of the ways we measure progress in culture and 4 encourage people to take calculated risks and other things is in part through 5 kind of the appraisal system. And are there changes kind of in the appraisal 6 system that allow for risk acceptance to innovate and change to send the right 7 signals to staff that they should be able to come up with new ideas without 8 fear of failing or being penalized in some way?

9 MS. MORROW: Well, I can't speak directly to the appraisal 10 system, but one thing that we've been working on with the culture team is 11 including more constructive behaviors as part of the appraisal system. That's 12 something we've been working with OCHCO on, focusing first on our senior 13 executive service.

14 There's also, I can't say that it is uniform, but there is 15 organizational effectiveness as an element and standard in our appraisal 16 system that I feel like has evolved over the years in terms of how that's 17 evaluated.

18 And there seems to be much more of a focus on improving 19 the organization and really contributing to the organization. And that is where 20 I would see, you know, the opportunity to embrace risk and make that -- you 21 know, accept that as part of our culture and incorporate that into our appraisal 22 system.

23 CHAIRMAN HANSON: Great. Thank you. So we know 24 that one of the ways that kind of culture changes and is transmitted is through

61 1 stories. So I was really interested in Erin and Tammie's kind of case studies 2 of the application of the Be riskSMART framework because I think sometimes 3 these things can be a little abstract. I admit I don't always understand what 4 we're talking about when we talk about these things.

5 So I was really kind of pleased to see this. And I know 6 Candace, you and others have briefed me on this before, but could you share 7 another example of a success of application of the Be riskSMART framework?

8 And particularly could you share an example which maybe the riskSMART 9 framework resulted in putting more resources towards a particular activity due 10 to its risk significance?

11 MS. de MESSIERES: Sure. Thank you, Chairman. Yes.

12 There's many, many examples. And maybe I'll briefly touch on a couple.

13 You know, first of all the framework can be used for not just 14 technical but process enhancement. So we do have examples where, you 15 know, in preparing for important decisions, like the 10 CFR 52.103(g) finding 16 for Vogtle Unit 3, the framework was used not for the actual decision but to 17 ensure the process enhancements were available there.

18 The early implementation of the principles was used in 19 FLEX decisions regarding the finding level. So, you know, should all FLEX 20 findings be green? And in that case, you know, no, that wasn't appropriate 21 and that used the principles from the framework.

22 We have examples, you know, from operator reactor 23 licensing where, you know, it may be that it's more important to use, you know, 24 rulemaking processes to address an issue or, you know, do formal exemptions

62 1 versus other, you know, processes.

2 You heard from Erin about her example from the region.

3 But, you know, there's also many examples of using the framework that 4 resulted in increased inspection resources, for example, evaluating safety-5 related breakers at Region I. So the list, you know, goes on.

6 I also just want to mention from a personal standpoint in my 7 use of it on a daily basis, and I recognized this right away during the pilot, one 8 of the pilots that's highlighted in the NUREG, that there is a -- some of the 9 benefit of the enhancement area is that the decision is based on a full suite of 10 possibilities, scenarios.

11 It really enables you to hear all of the perspectives, to talk 12 frankly about the risk appetite, get, you know, enhanced documentation.

13 There are things that I've come and just again daily use of the framework that 14 I just wouldn't have thought about managing if I hadn't really stepped through 15 that systematic approach.

16 And so I think it enhances personally my decision-making 17 every day. And I've witnessed with many examples. I think I'll --

18 CHAIRMAN HANSON: Okay.

19 MS. de MESSIERS: -- take any other questions.

20 CHAIRMAN HANSON: Thank you very much, Candace.

21 I appreciate it. Commissioner Baran.

22 COMMISSIONER BARAN: Well, thank you all for your 23 presentations and your work to encourage innovation at NRC.

24 I'd like to ask about the effort to improve the concurrence

63 1 process for sign off on rulemaking packages and policy papers to the 2 Commission.

3 I think it's fair to say that the Agency has struggled with a 4 concurrence process that often takes a long time to complete and leaves 5 people on the concurrence chain uncertain about what they're responsible for 6 reviewing.

7 June, you talked about this a bit for the rulemaking process.

8 Are the practices you discussed on Slide 46 the same ones used for policy 9 papers to the Commission or are those two different process improvement 10 initiatives?

11 MS. CAI: So I'll speak for rulemaking products and then 12 perhaps turn it to Margie on the overall paper process.

13 So what you see on Slide 46, this is really a very organic 14 collection of best practices that rulemaking staff developed over the last 15 couple of years to help us improve the concurrence process because we have 16 a lot of documents that we're working on, you know, at any given time and 17 they all have to go through concurrence.

18 And so we had our staff coming up with these great ideas.

19 And I want to give a shout out to Alexa Sieracki and Caylee Kenny because 20 Alexa worked to collect all these good practices you see on the slide from the 21 various staff. And then Caylee worked with her to put it all together.

22 Around the same time, the Agency process simplification 23 team was doing their work. And so Alexa actually served on that group and 24 shared many of these practices and some of the ideas that we've been trying

64 1 out in our experiences. So she sent our experiences directly to that group.

2 And so you'll see in the end many of these practices complement what the 3 Agency is doing under the process simplification team.

4 There is a lot of focus on making sure that we get early 5 alignment. There is an alignment meeting that's held under this new 6 Commission paper development process to make sure we have all the key 7 players at the table and we agree who should be on concurrence and what 8 their roles are. And so we're definitely implementing that. And we have 9 earlier versions of that practice as reflected in the infographic.

10 We've also been using parallel concurrence extensively.

11 We made that shift about a year, a year and a half ago when we started using 12 OneDrive and SharePoint more readily in the Agency. And we were really 13 able to leverage those IT tools to do parallel reviews. And now we'll be able 14 to quickly to the e-concurrence system in there and it's parallel review feature.

15 And so that's part of the, I think, process simplification team's focus as well.

16 So I will say, to summarize, it's been very complementary.

17 We were trying a lot of these things out and at the same time feeding the 18 inputs to that team. And in the end, I think, where we came out are very 19 closely aligned.

20 Margie, did you want to kind of talk about the overall?

21 MS. DOANE: Yes, if that's all right, Commissioner?

22 COMMISSIONER BARAN: Sure. Go ahead.

23 MS. DOANE: Okay. So, yes. So the answer to your 24 question is it's not just for rulemaking. It is being used for policy papers. And

65 1 is there significant enough -- the good thing that came out of an SRM, and 2 they're affecting a large program area. So there is a significant enough policy 3 question.

4 There's alignment all the way up to the EEO's office, but it 5 could be just within an office. So the alignment is helping because it clarifies 6 issues upfront. And so we're seeing that this is leading to more expedited 7 decision-making because we understand the problems earlier in the process 8 and also we can get alignment more quickly so that less issues come up at 9 the end. That's the first thing.

10 Also by just sort of documenting the concurrence process in 11 this matrix, everybody can understand their role and understand how much 12 time they're going to have.

13 Even they put a time limit on me. Sometimes there's, like, 14 you know, the EDO gets two days. And so I know this thing is coming, and 15 I'll try to get it early on in some other form because I try to keep to the staff 16 schedule.

17 And we also try to make sure that we expedite more quickly 18 things that already got alignment in the beginning. So it's a work in progress, 19 but we're seeing this used throughout -- for significant so definitely policy 20 papers and rulemaking. And then I would say probably the best example, 21 Commissioner --

22 COMMISSIONER BARAN: Mm-hmm.

23 MS. DOANE: -- the recent example is the gas pipeline 24 issue where they used -- that wasn't a rulemaking but where they used all the

66 1 different tools like parallel concurrence and e-concurrence. And they used 2 the SharePoint site to really move that report along quickly but to get all of the 3 right people involved in the decision-making.

4 COMMISSIONER BARAN: Great. Well, let me follow-up 5 on a few of those things. There are a lot of good ideas there.

6 In terms of the parallel concurrence, I can see how that 7 would save quite a bit of time if people were reviewing things simultaneously.

8 How is that practice working? Is it saving time? Is it creating any challenges 9 to take that approach?

10 And I guess I could, maybe I'll ask June if she wants to 11 weigh in on kind of the rulemaking piece, hone a little bit on the rulemaking 12 concurrence process just to keep it kind of clear. And, obviously, Margie, if 13 you prefer to weigh in but, June, otherwise.

14 MS. DOANE: No, no, I wanted to say Annette. I wanted 15 to turn you to Annette, who has been the head of this concurrence process if 16 you --

17 COMMISSIONER BARAN: Annette is sitting behind you, 18 which is really hard, but that's fine.

19 MS. DOANE: I'm sorry. I didn't understand the room.

20 COMMISSIONER BARAN: That's okay.

21 MS. DOANE: But June is fine, too. They both have been 22 putting a lot of effort into this.

23 COMMISSIONER BARAN: I turn it over to you guys.

24 Who wants to talk about this?

67 1 MS. CAI: I can jump in a little bit and then maybe Annette 2 could add on because I guess that we've been working closely together.

3 So for us, the parallel concurrence has, I would say, been 4 transformative. It's really made a huge impact.

5 And one thing that's really helped us is availability of these 6 IT tools. You know, I look back to even a year and a half ago, we were still 7 handwriting comments, scanning them, emailing them to each other, calling 8 each other. I can't read the handwriting. I mean, that's really, you know, 9 some of the true examples of what we were doing.

10 And then we started using these tools and we would put it 11 out there. And now we have all the branch chiefs reviewing together, all the 12 division directors reviewing together.

13 And it's also helping not just the time it takes, but also that 14 they can kind of see each other's comments, and we don't have to, we being 15 the rulemaking staff, have to work so hard to reconcile the comments.

16 Because I can, for example as a branch chief, I can see what the other branch 17 chiefs are commenting on. And I can respond or I agree with their comments.

18 And now that saves me time from having to write.

19 So those are some of the tangible improvements we've seen 20 with the parallel concurrence. And truly we were only able to do it because 21 of the availability of these IT tools.

22 And so, again, we were doing this, you know, pretty early 23 on, like a year, a year and a half ago. And then when the Agency transitioned 24 to e-concurrence and it has the parallel concurrence feature, we were able to

68 1 really make that shift quickly because we were already operating in that way.

2 COMMISSIONER BARAN: Great. And it looks like based 3 just on the slide, another practice is under the heading of need to concur.

4 I assume the idea here is to try to keep the concurrence 5 chain as short as practical and reduce the number of individuals who need to 6 review and sign off on a document.

7 How is the staff deciding who should be on the concurrence 8 chain? Is that tough to figure out? And how is this approach working? Is 9 the staff doing a good job identifying the right individuals? Are we seeing 10 cases where people don't end up weighing in, and it raises issues later?

11 MS. CAI: So I think this practice has been working pretty 12 well. And I think there's two main factors that are contributing to it.

13 First, we have been really making a concerted effort to make 14 sure we have early alignment on rulemaking projects before we, you know, go 15 down too far. Like I said, we have the decision tool so we're making sure the 16 key decision-makers, the key players, are coming in early and having open 17 discussions, making sure we identify the path forward.

18 So during that early alignment, we make sure, okay, who 19 are all the impacted parties who all have, you know, some input and make 20 sure we're engaging with them and then, of course, we would include them on 21 concurrence.

22 The other piece is with this new Commission paper 23 process enhancement, we do have to do the alignment, early alignment 24 meetings with Margie. And so as part of that process, she mentioned the

69 1 concurrence matrix.

2 We have to document clearly here are all the people who 3 are going to concur and here's what their role is. So it's not just that they're 4 concurring perhaps to the whole package, but they're looking at it, for 5 example, from a legal perspective or editorial, you know, style perspective or 6 only on this technical issue or only this other piece. So it's very clear what 7 their role is.

8 And so at that alignment that we have with Margie at the 9 outset of the project, we make sure we clearly document and include all the 10 various parties.

11 So I think it's been working very well. And so we're able to 12 hone in on exactly who needs to be on concurrence but in addition also specify 13 what their roles are. And I think between those two pieces, it really helps us 14 be more efficient.

15 I wanted to add one other piece. The alignment agreement 16 does also have a section about awareness.

17 So a lot of times we do have a wider distribution network for 18 our packages where we share documents with people for awareness or early 19 engagement but not actually putting them on formal concurrence but that they 20 do have, you know, an opportunity to take a look at it. And, of course, if they 21 have any concerns that they have a chance to weigh in and give us their input.

22 COMMISSIONER BARAN: Great.

23 MS. VIETTI-COOK: I just want to mention, too.

24 COMMISSIONER BARAN: Sure.

70 1 MS. VIETTI-COOK: What's interesting, what we're 2 observing -- even though we have limited data, what we're observing is that 3 we're actually, even though we were encouraging collaboration with as many 4 people as you need to and only put those people that you really feel need to 5 be on concurrence and be thoughtful about it.

6 And, of course, like she said, we also encouraged, you 7 know, courtesy reviews, provided for awareness to all those people that you've 8 collaborated with. And if they did have an issue, wanted to review the paper, 9 they could always go to the author. They could go to their branch chief. And 10 they can always go to the differing views programs if they feel like they're not 11 getting satisfaction.

12 But what we're observing is that we are seeing a big 13 reduction in the concurrence time, but that we're not really seeing a huge drop 14 in the number of people concurring.

15 So what we think is happening is number one, the alignment 16 is helping move, you know, early alignment of expectations of what the 17 stakeholders want to see in the product. And I think, you know, 18 documentation of that and then, you know, so that -- and then realigning as 19 necessary as new information becomes available but also the technology 20 improvements, like all of the collaboration tools, you know, for sharing 21 documents for, you know, collaborating over Teams.

22 We did templates so that they're all -- you know, these are 23 all the questions you should be considering on data, on, you know, 24 concurrence and all these various issues.

71 1 And, of course, we have templates for SECY papers and 2 then there's the e-concurrence system. So it's really tracking and people can 3 see how much time people are spending on concurrence. It's very visible to 4 everybody.

5 So we actually have seen a 45 percent drop in the 6 concurrence of the annual papers that we've been tracking. So we are 7 seeing progress on the time. And it was an interesting observation that we 8 really haven't seen the number of people getting reduced on the concurrence 9 process. So we think it's these other things that are moving it along.

10 COMMISSIONER BARAN: Great. Well, I think it's great 11 that the staff is trying some new tools and approaches in this area. It sounds 12 like it's going well.

13 I think it's the kind of process improvement that has the 14 potential to increase both the timeliness and the quality of the concurrence 15 reviews. And that, of course, can increase the timeliness and quality of the 16 rulemaking or the SECY papers which is what in the end has the real world 17 benefit. So that's great. Thanks so much for walking me through that.

18 CHAIRMAN HANSON: Thank you. And now we'll take a 19 five-minute break. We'll reconvene at 10:52 and, yes, thank you.

20 (Whereupon, the above-entitled matter went off the record 21 at 10:47 a.m. and resumed at 10:52 a.m.)

22 CHAIRMAN HANSON: All right. Thank you, everyone.

23 The meeting will recommence now with the external panel. Each panelist will 24 have about 8 minutes to present.

72 1 And we'll start with Reginald Mitchell, the Chief Financial 2 Officer and Enterprise Risk Management Secretariat for the United States 3 Agency for International Development.

4 Mr. Mitchell.

5 MR. MITCHELL: Yes. Good morning, Chairman and 6 Commissioners. Thank you for the invitation to participate in today's briefing.

7 It is indeed a pleasure to be back before the Commission with an opportunity 8 to share the U.S. Agency for International Development's journey in 9 developing our risk appetite statement.

10 As the CFO, I serve as agency Enterprise Risk Management 11 Secretariat, also referred to as the ERM Secretariat. And in this capacity, I 12 lead the agency's ERM governance program.

13 As some of you know, USAID is the federal government's 14 largest development agency working in partnership with international and 15 humanitarian organizations, and we operate in over 100 countries worldwide.

16 Our work requires our foreign service officers to work in remote and 17 sometimes dangerous environments, delivering assistance: everything from 18 food security programs, to fighting pandemics such as COVID-19.

19 As we develop our programming within a variety of country 20 contexts, it is imperative for us to actively identify and manage our enterprise 21 risk. Today I was asked to focus on the development, best practices, and 22 use of USAID's risk appetite statement in our agencys day-to-day decision-23 making.

24 But before I begin, I'd like to, I think it would be helpful for

73 1 me to share a few moments to go over our ERM overarching governance 2 framework which uses our risk appetite statement as a governing principle for 3 managing risk.

4 Next slide, please.

5 Key to the success of the ERM program is an effective risk 6 governance structure. With support and engagement from top leadership 7 that reaches stakeholders and every employee within the agency.

8 As you can see on the chart, our framework is comprised of 9 three levels, with the head of the agency at the top. What is important to note, 10 is the framework is designed so that information flows in both directions, up 11 and down the organization. I will start with the assessable unit, which we 12 define as the roles of an organizational unit of the agency, which includes our 13 missions overseas and independent offices, each with their own management 14 controls on risk and internal controls.

15 Next is the bureau level, with each bureau having its own 16 risk council, like the accessible units, and it's tasked with reviewing and 17 evaluating aggregated data collected by missions and offices. Each bureau 18 then determines which high-risk items will be forwarded to the next level in our 19 ERM governance framework.

20 Our three agency councils are:

21 First, we have the Risk Management Council which covers 22 programmatic risks identified by the bureau.

23 Second, we have the Senior Assessment Team, referred to 24 as SAT, where SAT is responsible for financial risk, and financial-related

74 1 issues identified through internal controls, financial compliance assessments, 2 and internal and external audits.

3 And, finally, we have the Executive Management Council for 4 Risk and Internal Controls. We call it the EMCRIC, which receives input and 5 recommendations from the Risk Management Council and the Senior 6 Assessment Team on risk identification and mitigation for leadership 7 consideration. The EMCRIC is chaired by our deputy administrator and the 8 council makes final recommendation on risk adoption and mitigation strategies 9 to the administrator.

10 Now let me go over our risk appetite statement.

11 Next slide, please.

12 As previously mentioned, USAID works in very challenging 13 environments and has vast experience managing risk. We are a highly 14 decentralized organization due to the geographical footprint of the agency and 15 the remote locations we operate within. We believe it was critically important 16 to document the agency leadership's expectations related to identifying and 17 managing enterprise risk.

18 The agency leadership provided broad-based guidance on 19 the amount and type of risk AID is willing to accept based on evaluations of 20 opportunities and threats at a high level and in key categories.

21 In addition, OMB Circular A-123 encourages the agency to 22 develop a risk appetite statement as a high-level document that employees 23 can make use of as they conduct their day-to-day activities. So, we began 24 with a need-based analysis. We thought it was essential to develop formal

75 1 agency guidance to help staff identify where it made sense to encourage risk 2 forward thinking, and to seize opportunities and where we needed a more risk-3 cautious approach.

4 We used a collaborative approach by establishing a core 5 cross-functional working group with expertise in program development, 6 operations, performance, and management. The group was overseen by 7 agency leadership. The core group was primarily responsible for developing 8 the seven categories we used to identify each type of risk.

9 We have in the agency our risk categories are:

10 programmatic, fiduciary, reputational, legal, security, human capital, and 11 information technology. Each category of risk was assigned an owner 12 responsible for evaluating their assigned risk category. The owners made 13 recommendations on the risk tolerance to the agency leadership for approval.

14 For example, I own the fiduciary risk category, and I 15 provided a recommendation that all financial activity has a low tolerance level.

16 However, I recommended a medium risk level be adopted when implementing 17 our funding through our overseas partners.

18 Once we had our statement developed, we began our road 19 show, and we briefed OMB and our Oversight Committee. Our statement 20 was well received by both OMB and the Oversight Committee. As a matter 21 of fact, as USAID was one of the early adopters of a risk appetite statement 22 in the federal government, several agencies have reached out to us and asked 23 that we share our development process.

24 As every organization is different and has a unique culture,

76 1 I encourage agencies to determine the culture that best fits their needs and 2 here are some best practices for your consideration.

3 In addition to the strategic revision of policy and procedures, 4 ERM requires significant change in agency culture and the fostering of internal 5 and external partnerships.

6 As an institution, avoid being too risk averse. At USAID the 7 risk appetite statement encourages us to be bolder in those areas where our 8 tolerance for risk is high, for example, when seeking out new implementing 9 development partnerships. But also remain vigilant in areas where risk 10 tolerance is low, such as cyber security, financial, sexual abuse, and 11 exploitation.

12 Next slide, please.

13 Our risk appetite statement is an integral component in our 14 risk posture. As mentioned earlier, the risk appetite statement includes those 15 categories, it also includes gradation of risk categories, that is high, medium, 16 and low, to explain how each risk may be used in specific situations.

17 I would now like to share what we believe our best practices 18 for socializing a risk appetite statement.

19 We operationalize the risk appetite statement by conducting 20 agency-wide training on conceptual components of the risk appetite and our 21 Enterprise Risk Management program. The administrator issues 22 communications to all employees reiterating their responsibility for using the 23 risk appetite statement when determining and managing risks in their daily 24 work, and to encourage staff at all levels to get comfortable with taking risks.

77 1 When developing a risk appetite statement, agencies may 2 want to assess what others have done when deciding on structure. For 3 example, we surveyed other federal government agencies, state and local 4 governments, and other international organizations. And we built on our 5 expert knowledge and consulted the General Accountability Office ERM 6 Playbook for applicability.

7 I would like to leave you with three thoughts on socializing a 8 risk appetite statement we found specific to our success of its adoption, and 9 that is:

10 Educate the staff on identifying managing risk; 11 Communicate to the staff a leadership commitment; 12 And encourage the staff to be smart risk-takers.

13 Next slide.

14 In closing, I believe our risk appetite statement equips our 15 staff with the capability of making better-informed decisions.

16 And, again, thank you for the opportunity to share USAID's 17 risk appetite journey. This concludes my statement to the Commission.

18 CHAIRMAN HANSON: Thank you, Mr. Mitchell. It's great 19 to have you back here at the NRC, even just for a little while this morning.

20 With that we'll move on to Ms. Teresa Gerton, President and 21 CEO of the National Academy of Public Administration.

22 Ms. Gerton.

23 MS. GERTON: Good morning, Commissioners, I'm Terry 24 Gerton, President of the National Academy of Public Administration. And I

78 1 am honored to represent the academy before you today.

2 Next slide, please.

3 Before I begin, I want to commend the breadth and progress 4 of the NRC's transformation efforts that I've heard about this morning. You 5 all are already adopting critical aspects of the Agile government principles and 6 demonstrated profound results.

7 In the interests of time, I will assume that you all are familiar 8 with the National Academy of Public Administration. It's one of only two 9 congressionally chartered national academies. And I'll jump right into the 10 discussion of Agile government that you all have requested.

11 Next slide, please.

12 In November of 2019, when the Academy announced our 13 12 grant challenges in public administration, we also established the Agile 14 Government Center as an enabler for the seminal concepts that would 15 underlie any success against the grand challenges agenda. We knew that 16 every major issue facing government would require an intergovernmental, 17 intersectoral, and interagency approach to development, and that would 18 require a new way of doing government business, would require more agility.

19 The Center itself is a partnership between the Academy and 20 the IBM Center for the Business of Government. Over the past 18 months it 21 has become a hub with the establishment of our Agile government network of 22 organizations and individuals who are exploring the implementation of Agile 23 principles in government programs around the world.

24 Next slide, please.

79 1 As you've already heard this morning, Agile is a new 2 management paradigm. Compared to the sequential waterfall project, Agile 3 is a very different way for projects to be accomplished. And if it's adapted for 4 broader use, it's a different way for governments to operate.

5 Building on and understanding of Agile principles for 6 software development and IT initiatives, and their incorporation by some 7 federal agencies into their IT projects 8 (connection difficulties) 9 .organizations, and federal agencies themselves are now 10 adopting and adapting the principles for their systems and applications.

11 This is driving changes in numerous federal IT software 12 areas, and it's now increasingly common for federal agencies to use Agile for 13 projects, especially for IT projects. However, it is much rarer for federal 14 agencies to use Agile as a way of managing their organizations.

15 Next slide, please.

16 Through the research in cases promulgated by our Agile 17 Government Center, the Academy has developed and shared 10 fundamental 18 principles of Agile government. I won't read these to you, but you will 19 recognize several, such as the empowering of staff members, or small teams 20 doing work in multiple short periods of time, a critical focus on iteration and 21 learning, and an early identification of risk is already operational within the 22 NRC.

23 Far more important is the mental picture these principles 24 create. Imagine a government organization at any level that practiced these

80 1 principles. It would be more focused on program outcomes and less on strict 2 compliance, with a deliberate approach to managing risk.

3 It would be open to engagement with partners across a 4 variety of boundaries, including with private and nonprofit sector partners, 5 without concern about who gets the credit.

6 It would be intent on learning quickly what works, measured 7 by both what its customers think and what the data says, and it would be open 8 to trying new solutions while accepting that failure often accompanies early 9 trials, and it would be an engaging place for its employees who are both sold 10 on the mission and empowered to seek creative solutions that encourage 11 mission accomplishment.

12 And with that compelling vision, we might ask ourselves why 13 don't we see more of this in the government?

14 Next slide, please.

15 You can already imagine some of the institutional stumbling 16 blocks to implement Agile approaches in government. For example, there 17 exists a general lack of knowledge and understanding about Agile. Many 18 people simply don't know what it is.

19 There's cultural and behavior barriers, like risk aversion, and 20 a perceived absence of permission to implement Agile principles.

21 And, third, there are real regulatory and procedural barriers 22 in contracting, hiring, oversight, and budgeting.

23 The Academy is currently developing a training module at 24 the Federal Executive Institute that will introduce federal leaders to the

81 1 concept of Agile government as a first step to addressing the knowledge and 2 culture challenges in implementing Agile. And we'll conduct our first class 3 next month.

4 Next slide, please.

5 The Academy's recent report on implementing Agile 6 principles in the federal government, which is linked on this slide, include 7 several recommendations to address these challenges more broadly.

8 First, we recommend that Agile should be a cornerstone of 9 the President's management agenda.

10 Second, it should be incorporated into existing cross-11 agency performance goals, especially the goal to improve customer 12 experience with federal services. And ensure that Agile management is used 13 whenever appropriate for organizations, programs, and projects.

14 Third, we recommend that the General Services 15 Administration's Office of Government-wide Policy should assign or establish 16 an organizational unit designed to assist other departments and agencies with 17 their Agile management journey.

18 And, fourth, we recommend that the President's 19 Management Council coordinate across its member agencies to develop and 20 facilitate the implementation of strategies for accomplishing Agile-related 21 goals.

22 Next slide.

23 Making these proposals for implementation of Agile across 24 the federal government does not preclude any agency from taking up the Agile

82 1 challenge itself. The Academy's report offers readiness factors, and 2 assessment tools for both organizations and individuals to determine how 3 ready they are to embark on an Agile government transformation. They also 4 document case studies of activities that have embraced agility in their projects, 5 programs and organizations.

6 The concept of Agile government is catching on around the 7 world, and we are excited to see the first embers of interest within the Biden 8 administration. And we look forward to continuing to support this effort.

9 I'll be happy to answer any questions that you have on this 10 topic. And I thank you for the opportunity to present it to you this morning.

11 CHAIRMAN HANSON: Thank you very much, Ms. Gerton.

12 Next we'll hear from Dr. Steven Simon, President and 13 Founder, Culture Change Consultants, Incorporated.

14 Dr. Simon.

15 I think you're on mute, Dr. Simon.

16 MR. SIMON: Thank you.

17 CHAIRMAN HANSON: There we go.

18 MR. SIMON: Even after a year of virtual communication. I 19 appreciate it.

20 Thank you, Commissioner and to the Commission for the 21 invitation to speak today.

22 So, some brief background so you know who I am and how 23 I came to this topic of safety culture -- of culture transformation, and why I'm 24 focusing specifically on safety culture.

83 1 I guess first is that I am proud to be a pioneer and, in fact, 2 the originator of the concept of safety culture. In 1983 I coined the term 3 "safety culture," and for the last 38 years have been working to develop 4 approaches that show culture is the most powerful force to drive safe behavior 5 and achieve breakthrough safe performance.

6 A second part of my background is that I have been a roll-7 up-your-sleeves culture change consultant, and have led transformation 8 efforts at organizations, large organizations such as General Motors, GE, and 9 numerous electric and gas utilities.

10 And I guess a third point of specific relevance is I am trained 11 as a psychologist, so I approach culture and safety from the people side, not 12 the technical.

13 So, one of the I think contributions I might be able to make 14 today is lessons from, you know, my career in working with organizations who 15 have attempted to change and in any case to successfully transform their 16 safety cultures.

17 One of the real critical success factors for any approach to 18 safety culture transformation -- this may seem obvious, but often it's not -- is 19 actually to focus on the culture. There is in the safety field, as we all know, a 20 default to go to programs. And I use a model to differentiate safety programs 21 from safety culture to keep leadership focused in on the target.

22 Here's how it works.

23 I'd like you to imagine a pot of stew on the stove. Think of 24 it as a safety stew. And the solid ingredients, the meat and potatoes, the

84 1 vegetables, those are the programs that we use and, frankly, whether it's in 2 safety or other areas. But if you're making a good stew, you get the best 3 ingredients you can and then you have to drop it in something. The solid 4 ingredients go in liquid, which is known as the broth.

5 And in this analogy, the broth is the culture and the 6 programs, the meat and vegetable, are the solid ingredients.

7 I think the point is that you can have really excellent safety 8 program ingredients, but if it's a rancid broth, it doesn't matter how good your 9 sirloin is. And this goes whether it's for risk smart or whether it's for incident 10 investigation, safety, job hazard analyses, and so forth.

11 In a broth or culture where there's good caring, good dignity, 12 psychological safety, freedom to speak up, trust, obviously this is a two-factor 13 theory, you get the most out of the program. The program and the culture 14 work together.

15 In those cultures where you've got mistrust, lack of concern, 16 focus on numbers over people, blame fixing, no management visibility and so 17 forth, it doesn't matter how good the sirloin is, your program is, if it's in a rancid 18 broth you don't get the benefits that you want.

19 So, I have found this model very useful in working with 20 operational leaders who typically think that safety is the job of the Safety 21 Department.

22 And in the compliance part, the development of programs 23 certainly belongs to the experts, but the culture, the broth obviously is set by 24 the marks of the leadership.

85 1 So, I want to share -- and if we can go to the Slide 4, the 2 safety culture maturity path -- I would like to, I would like to share a maturity 3 path that I think talks to -- and it's a few slides previous, if you don't mind.

4 One more. Thank you very much.

5 And this is a maturity path that I developed that identifies 6 different stages in the development of the safety culture. And it gives you a 7 bit of a metric for self-assessment on where any gaps might lie. As I look 8 back on the history of how the concept of safety culture has evolved in the last 9 almost four decades now, I do see five phases.

10 And I want to emphasize that this maturity model is not a 11 values model. Some models state your organization should proceed from a 12 reactive to a proactive culture, or beginning to a leading culture, or dependent 13 or independent, or an interdependent culture. And that's great. Those are 14 important values-based maturity paths.

15 But this much more, instead of providing a direction for what 16 kind of organization your -- sorry -- culture your organization should strive 17 towards has to do with how to get there. And the five levels each talk to what 18 I see as some of what might be missing or present in an organization's 19 development.

20 So, number one, back in the '80s, safety culture 1.0 was 21 establishing safety as a value. Obviously, NRC has had it from the very 22 beginning. Not every company, not every organization, many companies 23 created safety as a value for their annual reports. For many it was just lip 24 service. The key question for safety culture 1.0 and the values is, obviously,

86 1 whether the value is real. But it's also the safety vs. production value, which 2 I know you deal with all the time.

3 Safety culture 2.0 is about leadership. The shadow of the 4 leader impacts, and impacting behavior of employees.

5 In the '90s, just a bit of a historical maturity path, many 6 consulting groups started to focus on the shadow that leaders cast, training of 7 upper management, assessing development and so on. Only more recently 8 has leadership been defined not only as upper management, but leaders at 9 all levels which, of course, for a culture is critical.

10 Which leads to safety culture 3.0, which is grass roots 11 engagement.

12 So, I would offer an effective safety culture cannot be driven 13 just top down, but also needs to be driven from the bottom up as well. This 14 has been true, of course, for the compliance aspects of safety, but even more 15 true when it has to do with employees focusing on the cultural foundations of 16 injury prevention.

17 To kind of paraphrase a commonly expressed meme, I think 18 all culture is local and, yes, upper levels have to be -- of leadership have to be 19 committed to safety with an urgent intensity.

20 But within that overall organizational concept -- context, 21 what happens off shift, what happens when people are alone, I guess my 22 favorite definition of safety culture is what happens when no one's watching.

23 That's when the grass roots matter because they are the only ones that are 24 there. And if they don't have control of their culture then you get safety culture

87 1 4.0, which is work group norms and the inevitable drift that comes in the way 2 we do things around here.

3 We all have our policies and procedures. Safety culture 4.0 4 provides tools for the work group not only to look at its behavior, but also the 5 invisible norms of the -- you know, that influence that, that behavior.

6 I would offer that very few organizations have gone past 7 safety culture 1.0, 2.0, and 3.0. They've done the value. Some did 8 leadership. Even some prepared engagement. Very few actually looked at 9 the work group level when it comes to culture.

10 And that leads to safety culture 5.0, which I would, you 11 know, even a layperson through the decades looking at the NRC would say 12 you've been clear leaders in the term which is now called psychological safety.

13 But, obviously, you know, going back a long way has to do with the speak-up 14 culture, speaking up retaliation -- without retaliation. And that's something 15 that is starting to gain more encouragement in industry in general.

16 So, I think that the point of this maturity model is for 17 organizations to look at whether there are some areas that they are missing.

18 I think the traditional focus, as I said, is on leadership. It's not only leadership 19 but all levels.

20 Next slide, please.

21 I would also offer that engagement is not enough. And 22 we're all in favor of employee engagement. You know, we all love apple pie, 23 motherhood, the flag, and employee engagement. And employee 24 engagement is critical but it is not enough. There has to be also a focus on

88 1 the norms that drive employee behavior, and a focus on culture and not just 2 programs.

3 So, I think, finally -- next slide, please -- I would like to say 4 from my standpoint as someone who's not been involved with the theory but 5 with implementing change, you know, throughout my career, that the couple 6 of lessons I've learned the biggest are that if you have the luxury of choice, 7 implementing village by village is always better when it comes to culture, not 8 programs, than enterprise-wide or systemwide or, you know, I'll just say 9 systemwide, and in a single, even in a single plant, because each of the 10 subcultures is so different. And I guess my mantras for change with culture 11 work is it's different from program work. You go slow to go fast. You go 12 small to go big because you want it to stick in the end.

13 And I think with that my time is up. And I appreciate the 14 opportunity to share some of my lessons accumulated on safety culture 15 transformation. And from having watched a lot of Senate hearings, I yield my 16 time.

17 Thank you very much.

18 CHAIRMAN HANSON: Thank you, Dr. Simon.

19 Next we'll hear from Jeff Place, Executive Vice President for 20 Strategy at the Institute for Nuclear Power Operations.

21 Mr. Place.

22 MR. PLACE: Good morning, Chairman and Commissioners.

23 Thank you for the privilege and the opportunity to discuss how new technology 24 is leading to improvements in operational performance of our nuclear fleet.

89 1 I'm going to focus my remarks on two main topical areas.

2 First I'm going to provide a high level overview of some of the new 3 technologies that nuclear stations are implementing.

4 And then, second, I will discuss how INPO is using 5 technology to improve our support and oversight of the nuclear industry.

6 Next slide.

7 So, in the past few years we've seen an acceleration in the 8 use of new technology, driven primarily by the efforts to improve safety and 9 reliability, increase efficiency, and reduce costs. The list of improvements is 10 extensive. And I'm only going to be able to highlight just a few today. And 11 as such, the discussion that follows is not fully representative of all the efforts 12 and activities across the nuclear fleet.

13 Use of robotics has a long history in our industry, but the 14 advancement in their capabilities has led to many more applications. In 15 general, robots greatly improve industrial and radiological safety, decrease the 16 time to conduct the work, and increase the frequency of monitoring.

17 Some examples would include submersibles that can now 18 support underwater intake system inspections without putting humans in 19 harm's way. And the use of drones have drastically increased, both inside 20 and outside the plant, including inside systems and also inside of containment 21 while at power.

22 The use of digital sensors of all types is now widespread.

23 Routing this data through software platforms has given engineers, operators, 24 and managers key information to identify and sometimes predict equipment

90 1 degradation before failure occurs.

2 Artificial intelligence and machine learning are now 3 beginning to be used to automate some knowledge work and tasks, 4 specifically focused on asset management and equipment reliability. For 5 example, some utilities are now using software applications to automate 6 corrective action program and work order screening, to automate parts 7 purchasing, intra-equipment anomaly detection. In these cases, the software 8 is making initial classifications or decisions and the plant staff review and 9 approve the classifications prior to taking action.

10 Training is also being enhanced by using virtual reality. We 11 are just really beginning to tap into the use of this technology to enhance 12 industrial safety, operations, and technical skills training.

13 To touch on just a few other uses of new technology, we're 14 seeing the use of digital platforms to build electronic procedures and work 15 packages. Some of these have embedded video and human error prevention 16 tools. Digital 3-D printers are being used to build training mock-ups and 17 produce replacement parts.

18 Overall, we're seeing a rapid adoption and expansion of new 19 technologies. The industry is just beginning to learn from each other and 20 understand all the applications in which technology can improve safety, 21 reliability, and cost competitiveness.

22 Next slide.

23 Like the industry we support, INPO is using new technology 24 to improve our work. Some of this work has been evolving over years, and

91 1 some of it has been accelerated because of the pandemic. And INPO leads 2 the National Academy for Nuclear Training. And through the Academy, we 3 conduct leadership courses and seminars for nuclear professionals from the 4 first line supervisor up to the board of directors.

5 When the pandemic hit, we were initially unable to carry out 6 this training, but our staff quickly began developing a virtual training platform 7 we refer to as VOLT, or Virtual Online Leadership Training. We began with 8 a few specific training modules, but we eventually fully converted some of our 9 one-week seminars into the virtual setting. The industry response has been 10 outstanding, and we've actually been able to reach individuals we would have 11 not otherwise had we done it just in person in the building in Atlanta.

12 While we are planning to return to face-to-face classrooms 13 later this year, we will continue to offer some of our course content virtually.

14 A longstanding role of INPOs has been to facilitate the 15 sharing of operating experience across the industry. Recently we began 16 producing podcasts to share some of this information. Again, industry 17 feedback's been very positive. People seem to like the more personal 18 approach of hearing people that were involved in the operating experience 19 giving their firsthand accounts.

20 We are currently working on other delivery methods to make 21 the operating experience more accessible in different formats and on different 22 platforms.

23 I'm going to now shift and discuss some fundamental 24 changes in ways we are monitoring and shaping industry performance.

92 1 Next slide, please.

2 The industry and INPO have long used indicators to 3 compare performance, set targets, and drive performance improvements.

4 Today, INPO collects between 700 and 800 datapoints and operating 5 experience records every month. From this data, approximately 2,200 6 separate indicators are created, and about 200 of these are displayed on our 7 member website.

8 Also, over the past four decades INPO has put teams of 9 nuclear professionals onto the nuclear sites at about a two-year periodicity to 10 conduct performance evaluations and peer reviews. An assessment of 11 performance is provided after each evaluation.

12 While we have other operations and activities to monitor 13 performance, we weren't satisfied with our ability to understand performance 14 in between this two-year evaluation period, so we began developing a plant 15 performance indicator that would give us a more current view on at least a 16 quarterly basis.

17 To help us accomplish this view, we used neural network 18 modeling techniques to find patterns between the indicators that correlate to 19 performance assessment scores over time. Derived models are designed to 20 think like our staff as they go about implementing the evaluation process. We 21 are on our third iteration of this model.

22 The original model was a simple linear model. And the 23 current one that utilizes neural modeling techniques is 96 percent accurate, 24 with precision of plus or minus 3.5 points.

93 1 We further use these techniques to create neural models for 2 all 15 individual functional and cross-functional areas that INPO assesses as 3 part of our operations. The accuracy of these area models range anywhere 4 from 76 to 95 percent, with a precision of plus or minus 5 points.

5 None of our indicators or models fully substitute for onsite 6 evaluations, however, we do leverage the information from the models to 7 performance base our operations. For example, the composition of our 8 teams are adjusted in part based on this information. Our base team for an 9 INPO evaluation today is about six professionals. And then we augment that 10 team based on our review of performance through, in part, these models.

11 Our teams now typically range between 6 and 12 team 12 members, compared to a previous team size of 25 to 30. The result of our 13 evaluations have not suffered based on self-assessments that we have 14 conducted and feedback from the industry.

15 We are also using machine learning to advance new 16 models, some of which we are anticipating will be more predictive. A recent 17 area of focus for us and for the industry is understanding and measuring 18 sustainability of performance improvement. Currently, our staff draw 19 conclusions on sustainability through manual review of data, interviews, and 20 field observations.

21 As the staff collaborates and assigns numerical 22 assessments on their work platform, artificial intelligence and machine 23 learning software is running in the background, trying to replicate the results 24 by developing correlations between our data and indicators that will give

94 1 similar results. Our goal is to be able to be better at automating our view of 2 sustainability, minimizing subjectivity, and reducing errors from new staff.

3 Next slide, please.

4 In wrapping up, I would say that the tools we've developed 5 to date give us a more current, accurate, and complete picture of performance, 6 and at a frequency that we did not have in the past. Using these tools, we've 7 been able to redirect our resources where the industry needs them most. It's 8 also been a catalyst for greater industry comparison and supports the 9 industry's continuous learning culture.

10 However, we have more work to do to make better use of 11 data science and advanced analytical techniques in support of our, INPO's, 12 transition to a new operating model over the next year. We've seen a nearly 13 constant slope of improvement in the industry over the past 8 years. By 14 nearly all metrics, the industry is performing at near its highest levels ever.

15 While there are many factors for this improvement, the use of new technology 16 has certainly played an important part in increased operational performance 17 and efficiency gains.

18 The graph in front of you today is actually a simple roll-up of 19 the average of the neural plant performance indicator for the entire U.S. fleet.

20 The industry as a whole passed into our exemplary range in early 2017 and 21 has continued to improve.

22 Again, thank you for the opportunity to speak with you all 23 this morning. And I will be happy to answer any questions you may have.

24 CHAIRMAN HANSON: Thank you, Mr. Place.

95 1 Commissioner Caputo.

2 COMMISSIONER CAPUTO: Good morning.

3 Mr. Place, I will stay with you for my first question.

4 So, for those not familiar with INPO, I'm just going to recap 5 their mission: to promote the highest levels of safety and reliability, to promote 6 excellence in the operation of commercial nuclear power plants.

7 And I'd like to ask you a little bit about INPO's indicator 8 index, which I think you've also expanded on that by talking this morning about 9 the Neural Performance Indicator Index, and stressed how the industry's 10 safety performance has improved over time. And so, you know, among those 11 many indicators that INPO tracks includes reactor performance, safety system 12 performance, occupational safety, chemistry performance, and radiation 13 exposure.

14 At our Commission meeting a couple weeks ago we 15 discussed the declining trend of green findings and findings or are low safety 16 significance. And while the NRC staff acknowledged the industry's improved 17 safety performance, they attributed the declining trend in findings to staff's 18 increased focus on risk significance.

19 So, my question is this: if we are a modern, risk-informed 20 agency, shouldn't we expect to see a decline in findings if industry safety 21 performance has improved? After all, if actual safety performance has 22 improved but it's not reflected in the number of findings, that would suggest 23 we're not risk-informed.

24 So, wouldn't the decline in findings be a logical outcome of

96 1 the combination of the agency's risk-informing and the industry's improved 2 safety performance?

3 MR. PLACE: Yes, Commissioner, thank you for the 4 question.

5 I don't know if I can speak directly to, you know, the reasons 6 why the number of green findings would be reduced, but I can share with you 7 what we are seeing through the way we look at performance and how that 8 may correlate to issues that would be of regulatory scrutiny.

9 As you know, we collect these operating experiences, and 10 we call them events. And not all of them are of great significance, but we do 11 characterize them. And we have different categories:

12 We have significant events, consequential events.

13 We've seen significant events in the last several years 14 almost entirely eliminated. We just don't see them.

15 Consequential events have been reduced by 50 percent 16 from 2010 to 2020. Fuel reliability is amongst the best -- well, it is the best 17 ever. SCRAMs are not complicated anymore.

18 And through our own evaluation process we're not 19 necessarily seeing a reduction in areas for improvement, which is what we like 20 to help drive, identify and drive closure of gaps we see in the industry. But 21 what we are seeing is that they're written at a different threshold. We've 22 lowered the threshold. So we're seeing many more of our gaps or our areas 23 for improvement written at the behavior level before real consequence has 24 occurred.

97 1 And so I think, you know, you roll those types of things up 2 and I think that it's possible that that would also mean that there's probably 3 more things -- less things that are happening in the power plant that would be 4 viewed under regulatory scrutiny that may result in a green finding or other 5 findings.

6 COMMISSIONER CAPUTO: Okay. Thank you very much.

7 Ms. Gerton, I have a question for you.

8 Getting staff to set and track outcome-based metrics is a 9 challenge. There is a constant temptation to set the bar low, to set the bar at 10 something that you know will be accomplished, thereby guaranteeing success 11 when, in reality it's ensuring business as usual under the aura of a shiny new 12 program.

13 So, whether it's our current evidence-based policymaking 14 effort currently underway, or the Agile method that you discussed, the ultimate 15 question is how do we get employees to embrace and use these strategies to 16 drive performance improvement rather than simply going through the 17 motions?

18 MS. GERTON: Thank you, Commissioner, that's a great 19 question.

20 One of the key aspects for setting good outcome measures 21 is good data collection. And, frankly, you don't always get the outcome 22 measures set exactly right on the first time. But as you begin to collect data 23 about your processes and your outcomes, you begin to see the trends and 24 you begin to see the gaps where performance can be improved.

98 1 So, it's a leadership challenge as well as a data collection 2 effort. But as you begin to monitor, even if you set those first metrics and 3 they give you 100 percent mission accomplishment, the next thing you need 4 to think about is, well, how do we, how do we extend ourselves and how do 5 we take a bit more risk to get better performance? And as you begin to do 6 that and you begin to see performance improvements over time, you begin to 7 tighten those metrics so that they continue to push you to improve outcomes.

8 And as the staff sees that and as they begin to perceive the 9 improved performance, and likely what goes along with that is improved 10 customer satisfaction or improved mission delivery, they begin to buy into the 11 process.

12 So, if you're just new to setting performance metrics, it's not 13 necessarily a bad thing to set them and understand how your data tracks and 14 informs your performance and how you measure it. But over time, with the 15 staff and the leadership support you begin to collect better data, you have a 16 better understanding of your processes and how they impact your outcomes, 17 and you could begin to fine tune those data so that they continue to drive 18 increased performance.

19 COMMISSIONER CAPUTO: All right. Thank you very 20 much.

21 One last question for Mr. Mitchell.

22 You stress the need to manage risk holistically. And you 23 listed fiduciary risk as a category for enterprise risk management. Can I just 24 get you to clarify, is that strictly limited to, you know, internal controls and

99 1 auditing? Or do you also look at risk in terms of budget formulation and 2 execution?

3 MR. MITCHELL: Thank you, Commissioner, for the 4 question.

5 Yes, it's the latter. We do look at it holistically, not just the 6 execution side of the house, but budgeting and execution. And when we look 7 at our partnerships that we developed in implementing our programs, that's a 8 fiduciary risk as well because of the actors that we come across and oversee 9 your arena.

10 COMMISSIONER CAPUTO: So, I mentioned this a little bit 11 earlier, but in 2016 we collected 320 million in fees for licensing and oversight 12 work. In 2021 this workload is down 45 percent. That means our work went 13 from being 31 percent of our total budget to only being 21 percent of our total 14 budget.

15 How important is data-driven decision making to anticipating 16 and managing risk in budget formulation?

17 MR. MITCHELL: From my perspective it's very important.

18 When you look at you're setting a budget, you're setting a budget to complete 19 a certain program with certain objectives. And those objectives will always 20 be impacted by the funding that you are able to obtain to meet the stated 21 goals.

22 So, again, if you don't factor in the risk that you are taking 23 on from a holistic perspective of the environment that you operate within, 24 understand the external risk, meaning that stakeholders, which is both of ours,

100 1 Congress and OMB, then you do, you risk, not being able to get the funding 2 you need to carry out your program because you didn't take adequate notice 3 of what are the risks that you have to be able to navigate in order to get the 4 funding that you need to do your work.

5 COMMISSIONER CAPUTO: Thank you very much for 6 joining us today. That's it for my questions.

7 CHAIRMAN HANSON: Commissioner Wright.

8 COMMISSIONER WRIGHT: Thank you very much, Mr.

9 Chairman.

10 Reggie, I'm going to stay with you for a minute. Thank you 11 for your discussion and your comments here recently here with Commissioner 12 Caputo on best practices and, you know, when you're looking at risk appetites.

13 So, you mentioned the need to foster partnerships, both 14 externally and internally. And I'm trying to understand that a little bit more.

15 I am familiar with USAID, and have worked with them when 16 I was a member of the state economic regulators through NARUC, the 17 National Association of Regulatory Utility Commissioners. So I'm familiar 18 with what you do.

19 But can you tell me, talk to me a little bit more about some 20 of the external, I guess, partnerships that you fostered and how those 21 partnerships have impacted your practices in this area?

22 MR. MITCHELL: Certainly. Thank you for the question, 23 Commissioner.

24 A couple of the external partnerships that I'm alluding to is

101 1 like the U.N. We do a number of partnerships with the U.N. on a variety of 2 different programs that we have like interests in.

3 We also partner with our NGOs and other implementing 4 partners on getting perspectives from them on risk and what they're involved 5 with. As you probably know, Commissioner, most of our work is performed 6 by implementing partners, whether that's governmental, whether private 7 sector, other public entities such as the U.N.

8 And when we went through our process of developing our 9 risk profile we solicited their perspective from where they sit and tried to 10 incorporate that into our thinking on, you know, the risks in these different 11 perspective areas so that we could pass that on down to the staff level.

12 But as relates to the external stakeholders, that's what I was 13 alluding: primarily Congress, OMB, the U.N., and others.

14 COMMISSIONER WRIGHT: All right, thank you.

15 So, the NRC's reputation as the gold standard for nuclear 16 safety, that's at the heart of who we are and how we do our work. And our 17 reputation as a safety regulator is sometimes questioned by members of the 18 public and other stakeholders when we change our processes and procedures 19 and, you know, because there's a fear that we may be reducing safety or 20 something.

21 So, can you talk to me about how, if at all, reputational risk 22 plays into your risk appetite model, and what advice you'd give to the NRC on 23 this point?

24 MR. MITCHELL: Certainly. Again, I guess I spent about 26

102 1 years at the Nuclear Regulatory Commission, so I'm somewhat present, 2 although it's dated now. But I am familiar with the reputational challenges 3 that we had early on as well, and then how we got to where you're at today.

4 When it comes to USAID, we deal with a lot of actors.

5 Okay? And some of these actors are not good actors. Okay? And but we 6 may not know that until after the fact. So, we have to deal with sexual 7 exploitation. We have to deal with fraudulent claims. We have to deal with 8 strategies where some of our workers are targeted and sought after by these 9 bad actors.

10 So, when you're operating in the international development 11 area, there's a lot of money, as you well know, coming through USAID and 12 being put, placed into implementing partners and governments, the host 13 governments that we work with. And we constantly have to, you know, put 14 up the barriers to make sure that we have the internal controls and other 15 mitigating, you know, strategies in place so that we are able to catch any bad 16 actors, you know, early on.

17 And when we don't, it hits the Washington Post. And 18 constantly that's one of the things I always tell my administrator: my job is to 19 keep you out of the Washington Post on the financial side of the house.

20 (Laughter.)

21 COMMISSIONER WRIGHT: Gotcha. Thank you.

22 So, Ms. Gerton, I'm going to come to you. So, thank you 23 for your presentation and for your work in this area.

24 You know, on the first panel we heard about the application

103 1 of Agile to our rulemaking process. And on one of your slides you listed some 2 implementation issues and challenges with applying Agile. As we, as the 3 NRC starts to use this process more broadly, what do you see as key actions 4 we could take to avoid those challenges?

5 MS. GERTON: Well, thank you, Commissioner. And I was 6 really excited to hear about the work in Agile regulation. And the Academy's 7 getting ready to start a government-wide project this fall that will look at better 8 ways to apply Agile to the regulatory process because we know that it is very 9 challenging to keep up with changes in, especially in the scientific 10 environment.

11 So, as you're thinking about changing your internal process, 12 especially in the rulemaking, a couple of things you want to keep in mind.

13 Advertising this effort to a broader staff than just those who are engaged in 14 the rulemaking transformation is important to building agency-wide 15 consensus.

16 And I think it gets to your earlier point about reputation.

17 One of the key principles of Agile is customer experience.

18 And so, engaging your stakeholders to communicate how you're working this, 19 being very transparent in the results of the rulemaking process changes 20 themselves, taking small bites at it so you're demonstrating progress in a very 21 positive way, and then continuing to demonstrate through the outcome 22 measures that we talked about earlier that you are in fact improving safety, 23 and improving operations, rather than introducing risk into the system. All of 24 those key pieces of the Agile framework I think fit exactly what you're trying to

104 1 do here in the Agile regulatory transformation:

2 Engaging you staff so that they broadly understand what 3 you're taking on; engaging your customers or stakeholders so that they 4 understand you're taking small bites at the process so that the changes can 5 be easily understood, and that the implications of those changes can be 6 understood; and then being very transparent in the impact so that you continue 7 to buy into the process, demonstrate change and over time you build a pretty 8 comprehensive transformation into the process. It sounds like you're already 9 well on your way.

10 COMMISSIONER WRIGHT: Thank you so much for your 11 answer.

12 So Jeff, I'm going to come to you in some of the time I've got 13 left. And I want to just, I want to thank Commissioner Caputo for her 14 questions to you and comments.

15 Nobody wants an unsafe plant. And as licensees out there 16 and nuclear operators, you know, in the industry it's important for them to be 17 safe, and reliable, and up and running. And so the things that you're doing 18 and the way that INPO is peer reviewing, it's been, it's been interesting to 19 watch over the last, you know, 8 or 10 years just how things have moved up 20 the scale, as you showed.

21 And you're trying to do a lot of things, right? I mean, you're 22 trying to do a lot of new technology and things that maybe some of those 23 things are going to require some type of regulatory hurdle to be jumped.

24 So, in those -- I'm going to go to just kind of combine two

105 1 questions, if I could.

2 Are there any other applications of advanced analytics that 3 the industry is considering? And do you see any potential technological or 4 regulatory hurdles to more broadly adopt some of these technologies?

5 CHAIRMAN HANSON: Sorry, Mr. Place, I think you're on 6 mute.

7 MR. PLACE: I beg your pardon.

8 Yes, Commissioner, thank you for the question. I think as 9 an industry we really are at the beginning of just being able to use advanced 10 analytics. I know there's a number of projects right now that our utilities are 11 working with national labs on other new ways to digest the data.

12 I mean, as we have digital sensors that are beginning to be 13 put in the plant, there's discussion about making decisions and making 14 decision trees based off of the vibrations and temperatures and flows that 15 these sensors now, you know, you used to be able to take an individual, an 16 operator to go out into the plant and take these readings. And now even 17 analog gauges are being able to be digitized and then almost on real-time 18 basis understand what's going on in the plant.

19 So, there's discussion about robotics, to be able to even 20 maybe do operator rounds.

21 So, I definitely would envision that there's going to be many 22 discussions to have to have with regulators.

23 Do I know if there's any specific hurdles right now? I don't.

24 I think the regulatory framework is there for those discussions to be had. I

106 1 think they're just going to require very open, transparent, and good 2 conversations with the regulators to get to an efficient decision point so things, 3 so decisions can be made and improvements being put into the plants.

4 COMMISSIONER WRIGHT: Okay. Thank you so much, 5 Mr. Chairman. My time is up.

6 CHAIRMAN HANSON: Commissioner Baran.

7 Oh, sorry, it's my turn next. I was jumping ahead.

8 Skipped myself. Sorry.

9 Ms. Gerton, let me start with you.

10 I was introduced to this idea of agility in government a 11 couple of years ago in my interactions with the National Nuclear Security 12 Administration and their efforts to develop what they were calling at the time 13 a more agile and responsive stockpile management and development system 14 for our nuclear weapons complex.

15 And the way they wanted to do that was to create a program 16 in which they were exercising certain capabilities within the weapons complex 17 on a regular basis so that, if needed, that is, in order to respond to 18 technological, or geopolitical, or policy contingencies, that they could be able 19 to respond and implement changes in a relatively rapid fashion. Right? This 20 is another industry that's focused on safety, has long time horizons, extremely 21 expensive, all things that we could also say potentially about the nuclear 22 industry.

23 But what was striking to me at the time was that this what 24 they called the Stockpile Responsiveness Program was in addition to the other

107 1 things that they were doing. That is, it was an added cost.

2 And I'm wondering, in your case studies around Agile 3 government are you seeing this as a -- are you seeing Agile efforts, or efforts 4 to become more agile as it were, as adds? And how are agencies kind of 5 financing or budgeting for these things?

6 Are they doing it within existing capabilities? Or are they 7 special initiatives that require additional budgets, or, or what?

8 MS. GERTON: Thank you, Commissioner. That's a great 9 question, and you may know that the Academy just recently completed a five-10 year congressionally-directed assessment of NNSA and its various programs.

11 And so we're very familiar with the program that you mentioned.

12 But your broader question is does it cost more to do Agile, I 13 think.

14 CHAIRMAN HANSON: Yes.

15 MS. GERTON: And, in truth, what we see in the early 16 efforts is Agile is often a response to crisis, and you can see that in some of 17 the programs that the U.S. Digital Service has been called in to fix. And so 18 the question about cost is relative to what? Is it more expensive relative to a 19 system failure? Is it more expensive relative to a crisis in service delivery?

20 And so I would say, in those cases, it's really not. It's taking 21 the assets that you currently have and thinking about a new way to solve a 22 problem. So you're using people, you're using the tools that you have, and 23 you may be developing new tools. A lot of times we see the need to develop 24 a new IT system as the camel's nose under the tent in Agile implementation

108 1 because those new IT systems, which are typically focused on better service 2 delivery and better customer satisfaction or more security, drive with them 3 changes in process. And that begins to get people oriented to a more 4 iterative form of project improvement.

5 And so, over time, certainly the Agile processes come to 6 replace and be more efficient and productive than your old processes, but 7 there may be a period of time initially where you are incurring cost because 8 you are bringing in a new capability, often through IT systems, to the challenge 9 at hand or you're repairing or fixing a crisis where new tools and investment 10 are necessary. But as you begin to build those Agile intuitions and Agile 11 processes within your organization, you should, in fact, over time actually find 12 that you're reducing cost because you're delivering better service, your 13 customers are happier, and your people are more in a mode of constantly 14 improving using the resources that they have to deliver services better.

15 And so I guess I would say there may be a period of time 16 where you may see some increased costs in implementation, over time it 17 should streamline your operations and actually improve your cost delivery 18 model.

19 CHAIRMAN HANSON: Thank you. That's super 20 interesting. In fact, you kind of anticipated my next question, and you may 21 have answered it already but I'll ask it anyway. And that was kind of, and you 22 touched on this, and what's the role of IT or IT investments in the overall ability 23 of agencies to adopt an Agile framework? And are there elements of IT that 24 you think are most important? So, for instance, knowledge management

109 1 and/or data warehousing or mobile technology or collaboration tools or 2 advanced computing, et cetera. Kind of what are you seeing out there in 3 other agencies?

4 MS. GERTON: Another great question. I think one of the 5 places where we see IT driving Agile practices immediately, and you can see 6 it over the past 18 months in response to the pandemic, is as organizations 7 focus on their customer, we might think of the IRS and how they have 8 delivered a massive increase in stimulus checks to individuals and 9 corporations. You might think of how DMVs are saying, well, we can't do 10 driver testing and we can't renew driver's licenses in person, so what are we 11 going to do? As they focus on better customer service, whether that is an 12 improved website, whether that's improved data collection so that there's a 13 single-user sign-on. There are a variety of ways. But that kind of IT 14 implementation where it's customer facing has been the most powerful in 15 driving Agile behavior because it changes how organizations think and it 16 changes how quickly they feel like they need to respond to changes in 17 customer demand.

18 I think the second most important piece of IT revolves 19 around data collection, data warehousing, and data analytics. You can't do 20 enterprise risk management without a really solid data system and data 21 management process. You can't do it without good data analytics so that 22 you're collecting the right data and using it to track your trend.

23 So as organizations are -- sometimes, it's very different to 24 implement an Agile culture change from the front-end, right? Inviting people

110 1 through the door to Agile doesn't necessarily motivate them. But going at it 2 from the back-end, which is putting in IT systems that are addressing 3 operational concerns and improving mission accomplishments can often 4 begin to inspire that kind of Agile behavior where a little ways down the road 5 a leader can't say, well, you weren't really interested in Agile, you know, on 6 the surface, but look at what we've done already in how Agile is improving how 7 we deliver our services and meet our customers' needs. Now let's take it to 8 the next level and be very intentional about our next and broader 9 implementation of Agile.

10 So those would be the two places in IT that I would say are 11 really driving change: customer focus and data warehousing and analytics.

12 CHAIRMAN HANSON: Thank you very much. And I 13 think, once again, you've kind of anticipated my next question. But my 14 question, actually, for Mr. Place. And I was very interested in your 15 presentation about technology adoption. We've gone through this, I think, 16 during the public health emergency, as well in the NRC, where we had, you 17 know, made some investments, and I think the public health emergency in a 18 lot of ways accelerated the leveraging of those adoptions in a serious way, 19 everything from mobile computing and collaboration tools to, actually, like the 20 podcast that Alysia Bone does. You were talking about podcasts as a great 21 way to transmit culture and knowledge throughout an organization. I think 22 we've found the same thing here.

23 But it does, for me, kind of raise the question of do you have 24 suggestions for how an agency like NRC can systematically evaluate and

111 1 potentially invest in and/or potentially adopt and continue down the path of 2 innovation in this area without kind of the pandemic prompt, if you will.

3 You're on mute again, Mr. Place.

4 MR. PLACE: I apologize. Chairman, we started an 5 innovation group, actually, several years ago, and that group, which is made 6 up of individuals, we've made purposeful decisions to run every one of our 7 operations through that innovation group. So all of our major operations, we 8 put a strategy in place, and we don't see that ending. So we've undertaken 9 that. We're in the process of developing a new operating model beginning at 10 the end of this year. We're putting a new ten-year strategy in place.

11 All of those things are driving us to continue to go back to 12 this innovation group. We don't have to use all pieces and parts of the 13 innovation. That's just one of the things we've had to do. You know, Dr.

14 Simon talked about culture, and we've had to adjust our culture because, you 15 know, we're very conservative, like the nuclear industry is in general. And at 16 first, everybody thought, well, everything had to go through all five parts of the 17 innovation process. Well, no, it doesn't based on what it is you're looking to 18 change, you know, how far you want to make those changes.

19 So that's very much part of how we do business now. And 20 then we engage the industry, as you know. We are the self-regulation arm, I 21 guess, or group for the industry, so we engage them along every one of these 22 changes to make sure that we are not taking on more risk than they're willing 23 for us to take. Also, in some cases, they push us to take risks, and we may 24 have to go back and forth and collaborate, and we make a final decision that,

112 1 no, we're not going to make that change.

2 But it's really constantly, through our strategy, pushing 3 things back through that innovation process.

4 CHAIRMAN HANSON: Great. Thank you. Now 5 Commission Baran.

6 COMMISSIONER BARAN: Well, thank you all for joining 7 us and for sharing your thoughts. I think it's been a good discussion.

8 I have a couple of big-picture questions that I'd like to pose 9 to the whole panel and invite anyone who has thoughts to weigh in. The first 10 is about innovation. Obviously, innovation is a good thing and we want to 11 foster a culture of innovation at the agency so that we're looking for and open 12 to better ways of doing our work. Of course, some balance with stability is 13 necessary. I think Margie alluded to this earlier in her comments. If the staff 14 is constantly in a state of churn or continuous change, it can be hard for them 15 to keep their footing and remain focused. And if our approaches and 16 expectations are constantly in flux, there could be a lack of predictability and 17 stability for regulated entities and other interested stakeholders.

18 Does anyone on the panel have thoughts about how to 19 strike this balance between encouraging innovation while maintaining a 20 reasonable level of stability for the NRC staff and external stakeholders?

21 MS. GERTON: I'll take a first crack, if I could.

22 COMMISSIONER BARAN: Please.

23 MS. GERTON: One of the things that we strongly suggest 24 that agencies who are embarking on any kind of Agile project do as a first step

113 1 is engage their oversight community. You all are an oversight and regulatory 2 agency yourself, but you have inspectors general, you have auditors, you have 3 a variety of folks in the community who are interested in oversight of your 4 operation. And we all know that oversight agencies are heavily invested in 5 stability and status quo.

6 So one of the ways to strike this balance is to engage your 7 oversight community with your transformation team or your Agile project team 8 to make sure that, as you are moving toward Agile, the oversight community 9 is there to go along with you and say we can push that far, but we're not 10 comfortable going farther than that, that there are some boundaries that the 11 oversight community may not be ready to breach in a first or second or even 12 third round of Agile transformation.

13 When you do that, that also helps address the concern on 14 your staff that you might be pushing too much transformation too quickly. If 15 the oversight community is along for the ride with you, that can oftentimes 16 allay some fears amongst the community that would prefer more stability 17 while, at the same time, you're doing the iterative learning and transformation 18 projects as you go.

19 So that's one method that we have identified about how to 20 manage that tension between innovation and stability, both within your 21 workforce and in your impacted community.

22 COMMISSIONER BARAN: Thank you very much. Any 23 other thoughts on that? I know it's a big --

24 MR. PLACE: Commissioner Baran, I would just add on to

114 1 what Ms. Gerton said. I think the advisory board or advisory groups, we put 2 those in place and it's a mixture of internal and external people. I think that 3 helps.

4 One of the things that we've begun or we're beginning to do 5 in the industry, the licensee does quite a bit, especially when they're in high 6 levels of change, they use pulse surveys. They don't wait until an annual 7 survey to understand what's going on in the organization. And so much more 8 frequent pulse surveys, sometimes even weekly, dependent on the rate of 9 change and what's going on can help give you a picture if it's too much.

10 COMMISSIONER BARAN: Great.

11 MR. SIMON: And, Commissioner, if I may, I would jump in 12 also just adding on. It's axiomatic but it's worth restating anyway that this sort 13 of challenge for whether it's balancing out innovation and people's resistance 14 to change or any other part, any other focus of culture change, to the degree 15 that you can establish grassroots teams all across the system that are 16 engaged in driving it, it's obviously going to go better and, often, that's the first 17 thing that goes. Just wanted to mention that.

18 COMMISSIONER BARAN: Great. Thanks. Another 19 high-level question I'd be interested in your thoughts about is about how talk 20 about risk and risk appetite. Because NRC is a public health and safety 21 regulator, we talk about risk in the context of radiological risk every day. Our 22 core mission is to keep those risks very low.

23 The Be SMART risk initiative, on the other hand, is more 24 focused on how we consider all kinds of risks in our day-to-day decision-

115 1 making on corporate support projects or hiring or legal strategies or 2 rulemaking approaches. In that context, we want to be open to trying new 3 things and maybe taking a chance on a new process or approach that may be 4 an improvement over how we've traditionally done things.

5 Do any of you have any advice on how we can avoid 6 conflating these concepts so that we're clear that the agency is not interested 7 in accepting increased radiological risk on behalf of the American public.

8 Maybe Dr. Simon is the place to start on this question but, others, I'm 9 interested in your thoughts, too.

10 MR. SIMON: Let me go second on this, if I may.

11 COMMISSIONER BARAN: Fair enough.

12 (Laughter.)

13 COMMISSIONER BARAN: Of course, that requires 14 someone else to go first.

15 MR. MITCHELL: Let me just jump in, Commissioner.

16 Thank you for the question. Yes, I think it's one of, you know, laying down a 17 framework that clearly stipulates, you know, when you're looking at risk and 18 how you're looking at that risk. You're absolutely correct. I don't think 19 anyone on this panel or on this call wants the NRC to relax or take 20 unnecessary risks with nuclear power. That's a given. But when you look at 21 the whole aspect of the other part that makes up the entity NRC, then you 22 have to bring it in and conduct findings with respect to when you're looking at 23 the socialization and you've got four different generations working in the 24 workforce and you've got some generations that want to be out there and

116 1 doing new stuff and you got generations such as myself that, you know, kind 2 of slow on the uptake.

3 So my perspective is, you know, by clearly setting up a 4 framework to understand that, when you're talking about technical risk in and 5 of itself and the analysis and all the other aspects that you deal with in the risk 6 of a nuclear power plant, you know, that's one form of risk and that's a 7 technical risk that you're going to work and a health risk. But then when you 8 deal about the agency itself and is it open, does it allow collaboration, does it 9 include inclusion with staff up and down the organization.

10 COMMISSIONER BARAN: Thanks.

11 MR. SIMON: Yes. And the only thing I think I would add 12 to that is, look, I think most culture change efforts fail because they're too big, 13 they're too broad. And everybody wants to look at the culture and identify ten 14 things that need to change and, you know, go to work on it.

15 The focus, I often think putting something in the bullseye, 16 one part of the culture, and then working it up and down and across. And I 17 don't, frankly, think it makes too much difference if it's an agency change or if 18 it's a plant change. You know, the principles are the same. It just doesn't 19 have to do with the procedures and the tasks and the models and the 20 programs. It has to do only with the people, and it has to do with engaging 21 everybody, you know, from bottom-up to top-down.

22 But most important, it has to do with workgroup norms.

23 And this, to me, I work in a couple of nuclear plants, you know, safety in plants 24 is so much better than almost anywhere else in industry and the attention, the

117 1 focus, the understanding of culture, it's just terrific. But workgroups are 2 where it, you know, the battle is won or lost. You know, you go out on the 3 highway. It says 65 miles an hour. What do we drive? You know, we don't 4 follow the policy. You know, we drive 75 or 72. If you're driving 75 in a 65 5 mile-an-hour zone, are you following the law or are you breaking the law?

6 You are breaking the law, but I would offer your following the most important 7 laws that govern behavior which is the norms of the group. And don't take 8 this to driver's court. You know, I tried once, and the judge threw me out and 9 doubled my fine. But, you know, until you get the workgroup itself working on 10 that drift, you know, you can have all the procedures and training that you 11 want. They learn it in the training center, and then they go and they pick up 12 what's done in the workgroup.

13 So I think this is true of innovation, I think this is true with 14 risk, I think it's true with all programs. But I'm just, you know, kind of putting 15 that blanket around it, you know, hoping that gives a little bit of a framework 16 from the people's side of the transformation.

17 COMMISSIONER BARAN: Great. Thanks. Well, 18 maybe I'll just leave it there, Chairman. Thank you. Thanks so much, 19 everyone.

20 CHAIRMAN HANSON: Thanks very much to our external 21 panel. Mr. Mitchell and Ms. Gerton, Dr. Simon, Mr. Place, really appreciate 22 your contributions today. I thought it was a great way to kind of follow-up the 23 staff panel. Obviously, it touched on many of the themes. So thanks to 24 SECY for pulling all this together for us this morning.

118 1 As we wrap up, I want to take a moment to recognize 2 Commissioner Caputo. This is her final public meeting, as the end of her 3 current term is quickly approaching. Thank you, Commissioner, for your 4 years of dedicated service to our agency, not just over the last three years but 5 also working as congressional staff and in the industry. The NRC has greatly 6 benefitted from your extensive knowledge and experience in nuclear energy 7 policy, obviously even before you came here. And I have no doubt that you 8 will continue to be involved in our important work in one form or another going 9 forward.

10 And in that spirit, this isn't a goodbye but, rather, a 11 recognition of your work over the years and a sincere expression of thanks.

12 COMMISSIONER CAPUTO: Thank you very much, Mr.

13 Chairman.

14 COMMISSIONER BARAN: Well, let me just add all the 15 best of luck in all your future endeavors, whether at NRC or elsewhere. All 16 the best.

17 COMMISSIONER WRIGHT: Commissioner Caputo, or 18 Annie, as I've known you over the years, you know, I first met you sometime 19 back, I don't know, '05 - '06 time frame, I think, when I was an economic 20 regulator and you were working for your bosses on the Hill or the committees 21 at the time working the issues of the day. You've always been passionate 22 about everything that you've done, whether it be on the Hill or be it here at the 23 NRC.

24 And, you know, during the last three years, I've gotten the

119 1 opportunity to get to know you on a personal level, get to know your family, 2 and I've learned that you're more passionate about your family than you are 3 about anything that we're doing here. And the things that you have done that 4 you're also, you know, you love the outdoors, and I've discovered just how 5 talented you are at quilting. I never met someone who really knows that like 6 you do.

7 You know, I'm a very hopeful man, and so I'm hopeful that 8 our professional and personal relationship here will continue for many years 9 to come. So wish you the best.

10 COMMISSIONER CAPUTO: Thank you. Thanks to each 11 of you for your very kind remarks. During my time here, I have worked with 12 all of you and earned a huge measure of respect for each of my colleagues 13 and learned a lot from each of you in terms of how committed you are and 14 dedicated to your job and the talents and skills that you bring to it.

15 And I also, you know, I think we all get spoiled to a certain 16 extent. We have the most talented staff here at the NRC, and I really have 17 developed a huge respect for the staff in ways that I didn't have in my previous 18 capacities and have relished working with them during my time here.

19 So thanks again to each of my colleagues, and thanks again 20 to Chairman Hanson for his leadership over the last several months. I've 21 appreciated working with all of you, and I've definitely enjoyed my time here.

22 Thank you.

23 CHAIRMAN HANSON: Thank you. Thank you all.

24 We're adjourned.

120 1 (Whereupon, the above-entitled matter went off the record 2 at 12:15 p.m.)