ML21225A717

Transcript for W16

ML21225A717
Person / Time
Issue date:	03/10/2021
From:	Office of Nuclear Reactor Regulation
To:
References
NRC-1420
Download: ML21225A717 (53)

Text

Official Transcript of Proceedings NUCLEAR REGULATORY COMMISSION

Title:

33rd Regulatory Information Conference Technical Session - W16 Docket Number: (n/a)

Location: teleconference Date: Wednesday, March 10, 2021 Work Order No.: NRC-1420 Pages 1-60 NEAL R. GROSS AND CO., INC.

Court Reporters and Transcribers 1323 Rhode Island Avenue, N.W.

Washington, D.C. 20005 (202) 234-4433

1 UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION

+++++

33RD REGULATORY INFORMATION CONFERENCE (RIC)

+++++

TECHNICAL SESSION - W16 THE IDHEAS PROGRAM A NEW HUMAN RELIABILITY ANALYSIS METHODS FOR ALL NRC APPLICATIONS

+++++

WEDNESDAY, MARCH 10, 2021

+++++

The Commission met via Video Teleconference, at 10:45 a.m. EST, PRESENTERS:

SEAN PETERS, Chief, Human Factors and Reliability Branch, Division of RiskAnalysis, RES/NRC JING XING, Senior Human Performance Engineer, Human Factors and Reliability Branch, Division of Risk Analysis, RES/NRC MICHELLE KICHLINE, Senior Reliability and Risk Analyst, PRA Oversight Branch, Division of Risk Assessment, NRR/NRC NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

2 ANDREW ROSEBROOK, Senior Reactor Analyst, Division of Reactor Projects, RII/NRC ROY LINTHICUM, Chairman, PWROC Risk Management Committee, Exelon / PWROG NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

3 PROCEEDINGS (10:46 a.m.)

MR. PETERS: Welcome everybody, this is Sean Peters. I am the Branch Chief of our Human Factors and Reliability Branch and we're also in Nuclear Regulatory Research. And I'd like to welcome you to Session W16 of the NRC's 2021 Regulatory Information Conference.

In this session we're going to be talking about our Integrated Human Event Analysis System program in the Office of Research and we're going to be telling you about its capabilities and we're going to be telling you a user's experience from various offices, from NRR, from the Region and from industry.

So, if you could bring up my slides, I'd appreciate that.

So, the first speaker is going to be, I'll give you a quick introduction.

The first speaker after that will be Dr. Jing Xing, we'll have Michelle Kichline from NRR, we'll have Andy Rosebrook from the Division of Reactor Projects and we'll have Roy Linthicum from the PWR Owner's Group and Exelon.

So, can you go to the next slide please. Next slide.

DR. XING: This is the opening slides, maybe you go to the --

MR. PETERS: Yes, can you go to the Sean Peters first slides?

DR. XING: Oh, Peter's presentation, yes.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

4 MR. PETERS: And while we're pulling that up, I'd definitely like to thank everybody who's in attendance because we are going against, the most sought-after session is also being broadcast at the same time as this one.

That's the one on microreactors so, everybody who's joined on with us, I hope we can live up to, at least the experience that you would have gotten in the microreactor session.

Okay, we're still waiting for the first slide from Sean Peters. Roy, you actually see the slide?

MR. LINTHICUM: Yes, I do. I think maybe it just needs to be advanced one more time. It's a different slide than you started with.

DR. XING: Yes, this is a closing slide so, we're waiting for the slide's name Sean Peter's presentation zero.

MR. PETERS: Okay. So, I see in the chat that there's an issue. Okay. I was told I can share straight from my screen so, I'm going to share the slides directly or at least my slides directly from my screen, so slide show.

So, you know, thank you everybody. Thanks for working through that technical problem that we had. So again, I'm Sean Peters, I'm our Chief of our Human Factors and Reliability Branch and I'll just move along.

So, the reason why we are here is that we were told by the Commission to work with the ACRS to propose either a single NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

5 model of HRA methods to use for the Agency or develop guidance on which models should be used in specific circumstances.

So, this was a Commission directive back in 2006 under M061020. I'm going to go to my next slide. And so, looking through the timeline of development of HRA, when you look at the timeframe back in 2006, we were already doing some evaluation of HRAs versus the good practices.

So, we're looking at the most commonly used set of methods and we were evaluating those based upon what we were thought were the best ways to perform HRA. So, coming from that standpoint we, and coming from the SRM, we determined that we would work on an international and U.S. empirical studies.

What these studies were, were where we took teams of operators and we ran them through experimental scenarios at simulated facilities. And we took teams of HRA analysts and we used those analysts and with methods that they were utilizing to try to predict the operator performance.

And what that allowed us to do was gain insights into how people are using the HRA methods and what are the strengths and weaknesses from those various methods.

So, in that process to try and answer that SRM, what we found was that there really wasn't one HRA method that was optimal for use for all applications that we were using the HRA for in the industry and for the NRC.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

6 And so, the challenge that we had was that we also found other technical issues like HRA variability became a major complement for us so, what we did was we talked to our teams of international U.S. researchers and we determined that we would create kind of a new methodology that incorporated those best practices of the, of existing methods.

And so, we wanted to kind of take away their weaknesses and incorporate the strengths of the various methodologies. So, we began on this program to develop what we called the IDHEAS at-Power method. It's right here on my screen.

And looking at these dates are not necessarily the dates we started the work but the dates that we did the final publications of the work.

And so, when we got down that process and we were part of the way in that development of the IDHEAS at-Power methodology we realized, you know, Fukushima Daiichi hit and what that did to the Agency and to the industry was it got us to focus a little bit different on HRA applications and HRA methodologies.

The, in Fukushima Daiichi, there are a lot of operator actions that were environmentally influenced. Say debris, high winds, flooding, so these kinds of things aren't typically done prior to 2011 in HRA.

We typically had all our HRA work in control in that power applications and so, that got us to develop an entire new set of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

7 work saying that not only did we need to look at this from at-Power scenario but we need to look at it with all these other scenarios in mind.

And on top of that, it got us to open our minds a little bit and think, what are the future uses of HRA. And so, based on a human centered approached we were able to begin the development of the IDHEAS-G suite of methodologies.

IDHEAS-G means, is our general methodology and what that does is it allows us to build constant specific methods out of it, out of a human centered approach.

And because it's human centered it's not technology dependent and it's not application dependent. What that allows us to do is create things that can be used in any domain.

So, for us those domains in the future might include low power shutdown operations, it's currently including FLEX and could feasibly include things like spent-fuel storage transportation, medical and isotope use.

So, we have a suite of methodologies here that we think can be utilized for those applications. And so, what you're looking at here in the 2020 timeframe are the project we just developed and just gotten out the door.

So, Dr. Jing Xing is going to be our next speaker and she's going to talk more about those and I'll introduce her in a second.

The last two slides I have are just timeline references.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

8 And these will be in the slide deck that's available for the industry and for our participants here. And that's the end of my presentation, so what I'm going to do is I'm going to stop sharing my screen and we can go back and we can put Dr. Jing Xing's slides up and I will introduce her.

So, Dr. Jing Xing is a Senior Human Performance Engineer in my branch, Human Factors and Reliability Branch in the Office of Research.

She joined the NRC in 2008 and has been the technical lead for the development of the NRC's human reliability analysis methods since then. She holds a Bachelor's Degree in Electronics Engineering and a Master's Degree in Biophysics and Computer Vision and a Ph.D. in Neuroscience.

Prior to joining the NRC, she worked on human factors engineering at NASA and the Federal Aviation Administration.

So, welcome, Dr. Xing.

DR. XING: Thank you, Sean, for the introduction and thanks to everyone. So, I will introduce IDHEAS-ECA, the Integrated Human Event Analysis System for Event and Condition Assessment.

Next.

Before starting, I would like to share with you these acronyms we talk in our session. So, HRA, Human Reliability Analysis. HFE, Human Failure Event, HEP, Human Error Probability and IDHEAS, Integrated Human Event Analysis System, CFM, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

9 Cognitive Failure Mode, PIF or PIF, performance influencing factors.

Next.

So, I will first give an overview of IDHEAS-ECA. then showing some human error data supporting IDHEAS-ECA. Lastly summarizing the development , validation and improvement of method. Next.

So, what is IDHEAS-ECA? It is a new method for all NRC's Human Reliability Analysis applications. It has a qualitative guidance for analyzing what can go wrong with operator actions and what may cause human errors.

It has a quantification model and software for calculating human error probabilities. The HEP calculation is supported with many sources of human error data, including those from nuclear power plant operator simulator training. Next.

So, this diagram shows how IDHEAS models human failure event. A human failure event may consist of one or more critical tasks. A critical task is achieved through five macrocognitive functions.

Those are detecting information, understanding situation for diagnosis, making decision, executing the action and inter-team coordination.

The reliability of macrocognitive function are affected by performance influencing factors. For example, detecting alarms is affected by how similar the alarms are.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

10 Therefore, a human failure event is modeled by the failure of these macrocognitive functions, we call them cognitive failure modes. So, this method, as Sean said, is human centered.

It does not rely on the types of tasks. That makes the method capable of modeling any human tasks with any kind of operational context. Next.

So, talking context, context describe the conditions that challenge or facilitate human performance. IDHEAS uses this PIF structure to model context.

We classify context into four categories, each category is modeled with a set of PIFs. So, say the environmental context is modeled with workplace accessibility, visibility, noise, heat or coldness.

System context is modeled with transparency, human system interface and usability of the equipment and tools. Personnel context is modeled with PIFs such as staffing, procedure, the training and so on.

Task context has seven PIFs such as scenario familiarity and task complexity. Totally we have 20 PIFs.

Now you can imagine that the context of a main control room accident and ex-control room accident can be modeled differently in some of these PIFs.

And IDHEAS furthermore used PIF attributes to describe the ways that a PIF may cause human errors. So, at the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

11 bottom, what you see here at the bottom row shows some example attributes for the PIFs in red.

For example, picking procedure as an example.

Some attribute will be, procedure lacks detail or procedure doesn't match the scenario. So, the HRA analysts identify applicable PIF attribute in the context. Then the HEP is quantified, it's the impact of the attribute on the failure of macrocognitive functions. Next.

So, with that basis, IDHEAS-ECA has eight steps for a full HRA analysis. Each step produces the understanding of the human performance challenges from different perspectives, with different level of details.

Together the output of these steps answers: what can go wrong in the human performance, what cause human errors and what is error probability. Next.

So, your next question must be, how do we calculate HEP. IDHEAS calculates HEP based on large amount of human error data. So, here is an example of human error data from the NRC's SACADA database.

SACADA collects operator performance data in simulator training. So, unsatisfactory performance rates for training objective tasks were calculated from the SACADA data and used for IDHEAS.

Let's look at the data point in the first row of this table. So, training task was the diagnosis. In the database by 2019, this type of task was NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

12 performed 69 times in the abnormal scenarios, and the error rate was eight out of 69. So IDHEAS uses this as the error probability to inform an HEP for failure of understanding due to the PIF attribute unfamiliar scenario.

And so, another data point in the next row is a failure of decision-making, also for abnormal scenarios. The error rate was one out of 92. So far, IDHEAS used more than 1,000 such data points, not just from SACADA but also from a variety of sources to support HEP calculation. Next please.

Okay, so let me quickly summarize the method development and testing.

So, we developed IDHEAS-ECA from IDHEAS general methodology with the input from our 2018 FLEX HRA expert elicitationand the use a large amount of human error data for HEPs.

And we have the IDHEAS-ECA report and the software calculating HEPs. Those are publicly available.

Some testing activities included the NRC's 2019 FLEX HRA evaluation use IDHEAS-ECA and our ongoing IDHEAS dependency model testing and NRC's staff and industry use of the method.

We are also looking for public comments on the method. So, the feedback will be used to update the report and the software. Next please.

Okay. So, my final words goes to IDHEAS-ECA NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

13 validation and improvement. Validation needs operational data and it is a life-cycle process. So, we will continue to validate this method by using the method with broader risk applications.

And we need operator performance data from nuclear power plants. We look forward to your participation, for validation to make improvement. Thank you. That's all.

MR. PETERS: Thank you, Jing. So, I'd like to present our next presenter. This is Michelle Kichline. She is a Senior Reliability and Risk Analyst of the Division of Risk Assessment in the NRC's Office of Nuclear Reactor Regulation.

Michelle joined the Agency in 2010. Prior to joining the NRC, Michelle earned a Senior Reactor Operator's License from Oyster Creek Generating Station.

She holds a Bachelor's of Science Degree in Chemical Engineering from the Pennsylvania State University and a Bachelor's of Arts Degree in Physics from Lycoming College.

Welcome, Michelle.

MS. KICHLINE: Thank you, Sean. So, as Sean said, my name's Michelle Kichline. Today I'm going to be following on where Jing left off and I'm going to be talking about the work that was done to model FLEX human actions using IDHEAS-ECA. Next slide please.

So, first I'm going to start with a little bit of background. In 2019 the Office of Nuclear Regulatory Research NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

14 worked with the Electric Power Research Institute or EPRI as everyone knows them.

Using our Memorandum of Understanding, we assembled teams of FLEX and HRA experts from both industry and the NRC. those teams had two purposes. The first was to create a set of realistic scenarios and human failure event descriptions from using FLEX equipment.

The second was to quantify those FLEX HFEs using the IDHEAS-ECA method as Jing told you about. So, those teams attended site visits at both a boiling-water and pressurized water reactor.

So that they could gain a common understanding about how FLEX has been implemented, during the site visits, the teams spoke with site personnel, reviewed plan procedures for implementing FLEX and walked down the FLEX equipment storage, staging and connection locations.

Then the HRA experts met at the NRC headquarters for a workshop in 2019.

During that workshop the HRA experts used IDHEAS-ECA to evaluate the human error probabilities associated with some typical FLEX actions for both FLEX and non-FLEX scenarios. Next slide please.

So, there were a total of three different scenarios evaluated. The first scenario was a beyond design basis seismic NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

15 event that occurs in a BWR and results in a station blackout.

There were four HFEs evaluated for this scenario.

The first was failure to declare an extended loss of AC power (ELAP).

This was also evaluated for three different variations that I'll talk about on the next slide.

Next, we evaluated failure to perform the FLEX DC load shed, third was the HFE for failure to deploy the FLEX diesel generator. And then lastly, we had one HFE for failure to perform containment venting.

However, we ran out of time in the workshop and we didn't talk about that during the workshop very much. And so, only some participants ended up evaluating that in their post-workshop worksheets.

Then the two remaining scenarios that we evaluated were ones that use FLEX equipment but they don't occur during an external event so, these are the non-FLEX scenarios.

These two scenarios were first use of a FLEX pump for a loss of feedwater event where there's only one onsite water pump available and that pump runs for an hour at the beginning of the event.

The second was use of a pre-saved FLEX-plus diesel generator. When the installed emergency diesel generator is out of service for maintenance and an SBO occurs. Next slide please.

So, this slide presents our workshop results for the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

16 FLEX scenario in which a beyond design basis seismic event occurs and results in a station blackout. Due to the seismic event, we said that it's obvious that power cannot be restored quickly.

First it was the HFE to declare an ELAP and this was evaluated for three cases. In the first case, the procedure has very definitive wording for when to declare an ELAP. And it also has diagnosis of the ELAP, it's very obvious because there's extensive site damage.

In the second case, declaration of ELAP requires more judgement because the procedure has less definitive wording for when an ELAP is appropriate. But in that case, we still said that the diagnosis is obvious because of the extensive site damage.

Then in case three, this is the worst-case scenario we evaluated where declaration requires judgement like case two but in this case, there's less site damage and so, diagnosis of the ELAP is less obvious.

So, the results for failure to declare an ELAP show that the HEP estimates increase when more judgement is required to make the declaration. Variation in the estimates for each case was due to analyst variation in choosing the performance influencing factors as well as differences in how each analyst assessed the impact of the scenario on the HFE. . So, each individual analyst thought that the HEP estimate should increase as more judgment was required to make the declaration.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

17 But they didn't agree on the amount of the increase.

Some analysts thought that the less definitive wording wouldn't have much of an impact because the operators are well trained to declare an ELAP when it's needed.

Other analysts were very concerned that the operators might delay their decision to declare an ELAP if they were allowed more judgement to make that decision. That's all I have on ELAP.

The next one was failure to perform the DC load shed and you can see the results on the screen. This HFE was evaluated as a single critical task. And then the last one that we evaluated during the workshop was failure to deploy the diesel generator.

This HFE was split into two critical tasks. One was to transport the diesel and one was to connect and start it. And as you can see the task to transport the diesel had very little variation in the estimate from our analysts.

And it was the simpler task and that's one of the reasons why there was less variation. All right, next slide please.

All right. This slide highlights some of the insights that we gained from our workshop. The first insight was that analyst-to-analyst variability remained in how analysts translated their personal qualitative analysis into inputs into the method.

Even though, as I talked about in the beginning, all of the analysts attended site visits to ensure we had a common NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

18 understanding about the scenarios and the HFEs that we were evaluating.

Thankfully, the HEP estimates between the analysts were generally within an order of magnitude, so they weren't all that far off.

The next insight was that the procedural cues for using FLEX equipment in non-FLEX scenarios or defense in depth may not be as clear as it was for FLEX scenarios.

The participants thought that there could be improvements in the strengths of these procedural cues that could help maximize the usefulness of FLEX in the typical or the non-typical, non-FLEX scenarios.

And then lastly, there were insights that integration of FLEX into the plant accident response had improved substantially since FLEX was initially implemented. Next slide please.

All right. So, this slide talks about how the NRC has started using IDHEAS-ECA. First, informal training has been given to our Agency Risk Analysts at counterpart meetings.

In addition, IDHEAS-ECA is being used for comparison with our existing method of SPAR-H for quantifying the human error probabilities in detail risk evaluations that are performed as part our Significance Determination Process.

And then thirdly, IDHEAS-ECA is being used for HEP comparisons in the Accident Sequence Precursor program risk NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

19 evaluations as well.

So basically, when an SDP or an ASP evaluation comes up that has a human action that's important to the risk significance, the risk analysts have been directed to go ahead and use IDHEAS-ECA to calculate the HEPs and compare those results with what they had gotten from SPAR-H. Next slide please.

This slide talks a little bit about user feedback, which has been positive to date. So, all the users that have used our interface have found it easy to use and understand.

Both the industry and the NRC workshop participants thought that the method provided reasonable results. And the users to date thought that IDHEAS-ECA provided more accurate HEPs than SPAR-H because of the more detailed performance influencing factor options.

Also, our users appreciated that the human failure events could be evaluated at the same level that they are now in SPAR-H. Next slide please.

So, this slide discusses the progress on IDHEAS.

IDHEAS was presented to the ACRS PRA Subcommittee in September of last year. And then it was presented to the full ACRS Committee in February of 2021.

And recently, the full ACRS did vote to endorse the IDHEAS method for Agency use. In the coming years we plan to or actually later this year we plan to complete the IDHEAS dependency NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

20 and recovery models.

And then in the future, after that is completed, we do plan to transition the NRC PRA models to use IDHEAS-ECA instead of SPAR-H. But that's definitely going to take a lot of work on our part to make sure, to have that done. So, next slide.

This is my last slide and it just presents a list of acronyms in case you got lost anywhere in the presentation. That's all I have for today. Thank you for your time and let's go back to Sean.

MR. PETERS: Okay. Thank you, Michelle. And I'd like to present our next speaker, this will be Andrew Rosebrook. He's a Senior Reactor Analyst in NRC's Region II. Mr. Rosebrook has with the NRC since 2004 serving in several positions of increasing responsibility in NRC's Region I Division of Reactor Safety and Division of Reactor Projects in Region IV.

And prior to joining the NRC, Mr. Rosebrook was a Maintenance Manager at the Mandalay Generating Station in Oxnard, California. He also served in the U.S. Navy as a submarine warfare Officer, shift engineer, combat systems Officer, electronics technician and qualified reactor operator.

He's a graduate of the California Maritime Academy with a Bachelor of Science Degrees in Mechanical Engineering and Marine Engineering Technology and holds a Master's in Business Administration from Southwest University. Welcome, Andrew.

MR. ROSEBROOK: Good morning. My NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

21 presentation's going to focus on the use of IDHEAS-ECA for SDP and an example of how this product can be used for an actual event.

In this case, we used IDHEAS-ECA to help do the risk assessment for the Turkey Point low power trip during the Turkey Point Special Inspection that occurred this year. Next slide please.

A little bit of background. Turkey Point is a Westinghouse three-loop pressurized water reactor with large dry containment. The NRC conducted a Special Inspection of Turkey Point following three plant trips in a four-day period during the week of August 17th.

The second one of these three trips involved an automatic reactor scram on August 19th due to Source Range High Flux Reactor Power Trip Setpoint being reached due to a number of operator errors.

As a result, several human error probabilities in their SPAR model needed to be adjusted due to the nature of the performance deficiency in order to adequately characterize the risk from this trip.

The Senior Reactor Analyst, in this case, he used IDHEAS-ECA as one of the tools to do this during a detailed risk assessment. And if you'd like more information about the Special Inspection and the detailed risk assessment it can found in the Special Inspection Report 2020050 with the ML number provided. Next slide please.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

22 In the SDP it's important to, you're evaluating the condition as presented by the performance deficiency. So, the definition of the performance deficiency is key.

The performance deficiency was that the operating crew failed to adequately manage reactivity additions to the core and failed to adequately monitor key reactor plant parameters during these reactivity additions.

Specifically, the reactor operating crew did not identify that during the reactor startup procedure that Startup Rate limits were exceeded by a factor of three and this eventually did result in an automatic RPS actuation reactor trip. Next slide please.

As a background, these next couple slides take a look at the accident sequences and the fault trees that are being used.

The first slide showed the fault tree for an Anticipated Transient Without SCRAM going from a 100 percent power.

This current slide shows the modified event tree that was used to model this event, which is a Continuous Rod Withdrawal event from the source range and one of the key things from a risk perspective is that RPS fails and then a number of manual operator actions are relied upon. Next slide please.

Looking specifically at one of the key top events. For the reactor protection system to fail you have four different paths that you can follow.

And each of these four paths will have one NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

23 mechanical failure of RPS and then embedded in that top event is an operator action for the operator to manually insert a reactor scram from the control room, either with a RPS signal present or without a RPS signal present based on the nature of the proceeding mechanical failure. So, this is our first, you know, operator action in the sequence.

Next slide please.

The next top event involves, basically backup actions that the operators can do in the event of an Anticipated Transient Without Scram. All three of these actions are in the EOP for the, you know, for a reactor trip with failure of rods to fully insert.

However, one of these actions potentially can be done even if the operators do not diagnose that they are in an Anticipated Transient Without Scram. And that is to manually insert control rods for approximately 60 seconds.

This actually was the procedural step in the startup procedure that they were on, which was to raise power from criticality to channel the mine as they advance in the intermediate range and then to level power.

So, it is possible even if they don't diagnose the condition that they would still perform this action. So, this human error action to manually insert rods was also considered. Next slide.

The next significant operator action is in the emergency boration fault tree. Emergency boration is a time critical operator action that is a very heavy hitter on most plants in terms of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

24 operator actions.

And that's because the operator action to initiate emergency boration is your, basically your dominant tool on this tree and its E-02 factor. Next slide.

So, putting these together you'll end with a certain accident sequence where I have the initiating event, which is an operator error and then I proceeded down to the failure of RPS, which it contains an operator action, the secondary recovery action, which contains an operator action.

And then, over to emergency boration, which contains an operator action. So, the cut set for this sequence will contain one mechanical failure and three operator actions, all of which are potentially dependent upon the initial error. Next slide please.

So, typically when you need to adjust a HEP you would use the THERP methodology and the SPAR-H methodology and you would use a table such as this to determine what the level of dependency is and what the adjustment to that HEP would be.

You know, in this case, there are no intervening successes. It is going to be the same crew that's going to take all these actions and the big question comes into with cognitive, are there additional cues being available or not.

In the case where we expected the operator to manually trip the reactor with no RPS signal present, you know, that would be completely dependent.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

25 For cases where additional cues come in, you can use this bottom path and then it's a question of is it, you know, sequential or non-sequential, and is it in the same location or not.

And once again, you can get a variety of different dependencies from low dependency if you assume the best case once again up to, you know, a complete dependency or high dependency depending on the amount of time that you have.

So, the tool is very rough and it can give you some very non-conservative answers. Next slide please.

So, using IDHEAS-ECA we can take up the same scenario and break the performance indicating factors split up among different cognitive failure modes. So, we can take the factors that affect detection, we can take the factors that affect the understanding of the scenario and the decision making of the scenario and use those factors to manipulate the HEP so it's consistent with the actual performance deficiency in the present.

Next slide, please. All right. So, in this case, one of the most significant factors that's available is the concept of groupthink. In the understanding and decision-making cognitive failure modes, there's a performance indication factor of SF4, a bias or preference for wrong strategy exists for mismatched mental models.

In this case, you're not just looking at the failure of a single individual. You're looking at the failure of an entire crew to diagnose that they were in a situation that the key reactor plant NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

26 parameters weren't there.

Because they had overconfidence in the experience level of the other members of their team. So, as a result they, a lot of the members of the team defaulted to the judgement of the operator at the controls.

And because he was misdiagnosing the condition and not monitoring the key reactor plant parameters, that influenced everybody else's decision making that, you know, they thought what he was doing was correct and they didn't question his judgment. Next slide.

Additional factors that are allowed to be included using IDHEAS-ECA is the crew experience level. In this case a number of the, quote experienced members of the crew had not actually performed the evolution in the plant.

So, that could be taken into account. The just in time training didn't go over those particular steps because they felt that the folks had appropriate levels of experience.

Crew dynamics such as communication and control were a factor that influenced this event. In this case the operator at the controls did not communicate his intentions to the rest of the crew so they could provide meaningful backup.

And also, a contingency to misleading indications.

There was one instrument that was deviating from the other four channels and that could have caused some misleading indications.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

27 And also, IDHEAS provides a better indication of time factors and restraint. Next slide please.

So, the bottom line is that the IDHEAS tool supported deviation from the traditional SDP methodology, which was alive because we were using 0609 Appendix M.

And confirmed the application of engineering judgement assumptions, the use of alternative guidance, including the WCAP guidance that was used to develop the event tree that had been previously reviewed and approved by the NRC for low power at loss situations.

This ended up in us reaching a timely, reasonable and quite scrutable risk assessment without, you know, applying excessive uses of time and resources or giving a falsely, basically a falsely high result because of that.

So, in this case I believe the use of IDHEAS-ECA was very useful. And my last slide is, almost getting like Michelle, is my list of acronyms. So that completes my presentation and I'll hand it over to Roy.

MR. PETERS: Thank you, Andrew. This is Sean Peters again and I'm going to introduce Roy Linthicum. He's the Chairman of the Pressurized Water Reactor's Owners Group Risk Management Committee.

He has 38 years of experience in risk management NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

28 with positions at Exelon, Sargent and Lundy, PSE&G, Arizona Public Service, Northeast Utilities and Knolls Atomic Power Laboratory.

Mr. Linthicum has been an active member of the PWROG Risk Management Committee since 2001 and has served as Risk Management Committee Chairman since 2012.

During his time as RMSC Chairman, he has provided support for the Reactor Oversight Process Task Force, Risk-Informed Steering Committee, Risk-Informed Tech Specs Committee, I'm sorry, Risk-Informed Tech Speck Task Force and the ASME ANSI PRA Standards Committee.

Mr. Linthicum graduated from Rensselaer Polytechnic Institute in 1981 with a Bachelor's of Science Degree in Nuclear Engineering. Now welcome, Roy.

MR. LINTHICUM: Thank you, Sean. You can go to the next slide. So, I'm going to be discussing kind of an industry perspective on IDHEAS-ECA but first I wanted to give a little bit of background on why it's important.

And it's not so much why IDHEAS-ECA is important but really why realistic human reliability analysis is important to the industry. So, taking a look from a recent NEI publication, NEI 2004.

There are some charts on the bottom.

You can see over the past several decades, you know, the U.S. Nuclear Plant capacity factors significantly improved as well as the core damage frequency, the average per site has also NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

29 significantly improved.

And a lot of those benefits have been achieved by focusing on equipment performance. Eliminating failures based upon risk significance. We've seen a lot of improvement in that area.

But as you can see from the chart on the bottom right, we've kind of reached a plateau and in my opinion a lot of that is we're now really being driven in our models by two main issues. One is human reliability and analysis and the other being common cause failure.

And I'm going to talk about common cause failure but in order to make additional improvements to the plant, we really need to know what to focus on.

And having, you know, a realistic human reliability analysis in our models can help us identify what truly is risk significant from a human performance perspective so we can go after those elements and make those changes to the plant. Next slide please.

So, one of the things we did in the PWRO's group is we actually benchmarked IDHEAS-ECA against EPRI's HRA calculator. And as part of the work, we saw IDHEAS-ECA is a significant step forward in obtaining realistic human error probabilities.

I say realistic, I mean, it is, we do get significant differences between IDHEAS-ECA and ECA and they're not always, in what some people might call a favorable direction.

In some cases, we get higher values coming out of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

30 IDHEAS-ECA than the calculator but we feel those are actually in some cases a more realistic representation and same thing in the reverse.

And in some cases, you know, these differences are minor. In some cases, they can be orders of magnitude different in either direction. And what that shows me is there is still a lot of work that can be done in this area of Human Reliability Analysis to get to something we can all agree is a realistic representation.

I do think as we move forward, the initial industry use will be focused on using IDHEAS-ECA in the SDP evaluations rather than rolling that into our model and discuss that a little bit. Next slide please.

So, as I mentioned we do believe IDHEAS-ECA is a significant improvement over SPAR-H. Allows us a better comparison of the drivers between what the NRC and the industry results are and when you get into SDP process, identifying those difference is really the important part of the discussion.

And so, those differences and assumptions and boundaries and what really is important to the SDP results, what drives those results really helps ensure we have good communication so we can appropriately, you know, characterize the various SDPs as to their true level of significance.

It also allows us to do appropriate sensitivity studies so we can once again reach those realistic conclusions and identify NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

31 the bounds of the evaluations. Next slide please.

As far as using IDHEAS-ECA and moving it into our industry PRA models in general, you know, there has been some discussion about that but it does have significant hurdles as far as going forward with making those changes.

First of which is just the cost of changing models.

There are numerous human error probabilities in all of our models, so to use a new methodology and new tools requires significant amount of modelling changes.

And since this is a new methodology, there's as a minimum, would require an implementation peer review. It's also still uncertain at this point whether, if a utility were to make use of IDHEAS-ECA whether or not that would be considered a newly developed method that would possibly require newly developed method's peer review prior to use.

And those discussions are still being had with the staff on how to incorporate NRC developed methods into that newly developed method's peer review process.

And incorporation of dependency model I thought was discussed in the last slide. I think that's an important element when the models and versions we tested actually did not have the dependency models.

So, we thought that was a limitation. So, with those models being developed and the software in there, actually we'd need NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

32 to take a look at that and see how that interacts with our models and how, you know, we even could incorporate those results into our model.

And downstream implementation, whenever you go to a new method, you know, there's that cost of both developing the model but then with various applications of using our models too, you have those downstream implementations of cost and activities.

Probably the biggest ones nowadays would be 50.69 implementation, how does that change 50.69 results, risk informed completion times as well as making it shoulder the significance.

So, you know, making a change to a new methodology is a tough decision for any utility because it does involve a significant amount of cost and there really needs to be a benefit associated with that.

And the benefit really needs to be, are you gaining, you know, true additional risk insights from incorporating those new methods. Next slide please.

So, in the short term as far as the industry use, I do see there may be the opportunity for some key human failure events to be assessed and added into the model on a limited basis.

That would be a case where we feel, you know, the methodology and ideas of IDHEAS-ECA actually does provide some additional granularity in risk insights. Although I say that, I think it will be very limited.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

33 And there may be some application specific use of that, you know, primarily I would believe in the SDP process.

As a minimum, we know that as the NRC starts moving towards incorporating IDHEAS-ECA into their models, whenever we get into an SDP on the industry side, we like to run our models as well as the NRC models, so we understand, you know, what the difference is in the modeling per term.

So, we do expect to get some additional use from that perspective as well. I also see it, you know, from addressing uncertainties.

IDHEAS-ECA being a different method can help bound alternate results and we can actually run that in comparison and use that from a sensitivity perspective in various applications as well. Next slide please.

As far as long-term use, we do need to fully understand the methodology and, in addition to the methodology, the underlying data. There was a discussion about the SACADA program.

I think as more data becomes available, if more utilities were to support that effort and get more data that would be a benefit to both the staff as well as the industry.

We also need to resolve whether or not IDHEAS-ECA would be a consensus model, so we don't have to do a separate review approval of the method for us to use that from a licensee NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

34 perspective.

And I do think the most appropriate use would not be to use IDHEAS-ECA directly but to incorporate the results and the underlying methodology where appropriate into the EPRI HRA calculator.

That helps reduce implementation cost. It would still require us to use an implementation peer review because there's a new method.

But there's a lot of other, you know, links between our models and the HRA calculator so it would be much better if we can, like I say, just transition where it's appropriate into the industry tools, so that we're all using the same underlying methodology. I think that would help.

Remove some of the, I would say energetic conversations we sometimes have especially in the SDP process.

And with that, that ends my presentation. I'll turn it back to Sean.

MR. PETERS: Thank you, Roy. So, what I'll do now is we will start the question-and-answer process. And I'm going to take the very first question and this is not specific to the methodologies.

But the very first question were there's an issue with seeing some of the slides on the screen and they're trying to find out if any of these are available for downloading.

I learned that they are not available for downloading NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

35 but if you would like to look at the particular slides from the presentation, if you would, if you could email myself or Dr. Jing Xing, we should be able to provide you the slides from this session.

And you can see our emails are attached to that slide right there. So, I'm going to just go in the order that the questions came.

So, the first question is what is the NRC's vision on whether, when and how IDHEAS will be used in SPAR models and licensee PRA models? I think we answered that a little bit but, Andrew, you said you had a concept for how to answer that question?

MR. ROSEBROOK: Yes. I think for, in a lot of cases particularly when you are in the SDP, you know a lot of these factors in there, so you can apply it in a case specific manner.

And you're getting an accurate representation of what, you know, actually occurred during the event and it's a much finer tool than the SPAR-H tool or the THERP tool is.

So, that's certainly one case where you can use it and as Roy mentioned, and we've also used this in other cases where we're looking at sensitivities.

One of the key things when you're doing sensitivity analysis is usually it comes down with human error probabilities and using this as one of the tools to, you know, what's a reasonable range to variate and whether or not that's something that IDHEAS also sets up very well to do.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

36 Because it takes the case specific issues into consideration much better than a generic tool such as SPAR-H does.

So that's where I see the key implementation is, is in the SDP area where I have a whole lot more certainty of the information and a lot less, you know, generic type issues.

But, you know, that being said we're also working on, you know, how can you use this tool to look at dependency iterations and what not. Even in the base model, which was, you know, sort of in a cut set they would be pretty self-evident that X is going to influence Y.

I think IDHEAS is probably a better tool in a lot of cases than the one that we have in existence and as Roy said we do, we always, one of the first steps at any SDP is comparing the licensee's results to the NRC's results and trying to resolve any differences.

And if this is a known difference then I think that's something that we can typically live with in the process if, you know, we recognize that this is a potential difference and you can narrow down on that.

MR. PETERS: Great. Roy, you said you had some insights on that also? And with that, you might be able to answer the last question also.

The last question is associated with what are the specific first applications that the industry would see IDHEAS-ECA?

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

37 MR. LINTHICUM: Yes. I think as Andrew mentioned, I think the first is definitely the SDP process, understanding the differences. I do think, you know, other applications could, primarily what comes to my mind is some of the non-regulatory approved applications such as MOD ranking, AOE ranking.

Where, in a lot of cases some of the rankings may be driven by some conservative HEPs, so once again this can be another tool, we can use to challenge those assumptions to see if, you know, we can really get to the true significance of some of those complements.

Potentially also I think another area of, once again it really depends on if we can resolve this issue of whether or not IDHEAS is considered a consensus model by the NRC.

And I do know that's being worked on. Plan to maybe challenge my MSDI margin. Maybe another area where that's margin being driven by some underlying conservative human error probabilities. This may be a tool we can use to help address that in limited situations.

MR. PETERS: And so, I'll just take a stab at that question. I do have my own opinions on whether or not this is a consensus model but I won't weigh in until the final decision is made.

But we do have plans in the Office of Research to incorporate IDHEAS-ECA into our SPAR models. So, that's a project that's starting right now in the conceptual stage and will be NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

38 progressing henceforth.

But that part of the program is not controlled by my branch but we're going to be supporting the branch that controls the SPAR models in doing that.

So, the next question, this is directed to Michelle Kichline and I think Jing might be able to answer or help support the answer too, either way.

This is associated with the questions on the FLEX testing. In case three, to what extent was margin considered? How quickly FLEX equipment can be deployed compared to regulatory required time periods?

MS. KICHLINE: Thanks, Sean. I can start off with that. So, we did consider time margin in the three cases.

And one of the things that we specifically discussed that I remember was that sometimes, a lot of times in general the plant that has less definitive wording in when to declare an ELAP, if they allow more judgment, it's often because they actually have a longer battery life.

And therefore, they do have more margin to when they need to declare the ELAP whereas some of the plants that need to declare the ELAP very quickly so they make a definitive statement that if you don't have power back in an hour just declare it now.

Those plants might have a shorter battery time and so that could be part of the reason that there's differences in numbers.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

39 But like I talked about during the presentation, different analysts thought that was important, more important than other analysts did.

So, some people thought that that was important, that they were going to have more time and so, they delayed, so the delay would be okay because they would know when to declare an ELAP.

Whereas, other analyst thought that the delay, there was a chance for delay and so that delay might be longer and it would be more important the fact that they did delay.

MR. PETERS: All right. Thanks, Michelle. The next question, and does IDHEAS have a screening value determination for HFEs for plants in design that do not have procedures available yet?

DR. XING: I'll take that question.

MR. PETERS: Okay. Thanks, Jing.

DR. XING: Sorry. So, IDHEAS does not have a separate screening process. However, the software for calculating accuracy allows you just to put down the PIF attribute that you are concerned the most, say not procedures.

Then the software will instantly generate as an HEP for that situation. So, that isn't equivalent to the screening process.

MR. LINTHICUM: If I can ask a follow up question. I think that answer concerns me a little bit. I think the question's related to, you know, when the plant starts operating it will be at procedures but we don't have them yet.

So, if you select that you don't have procedures then NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

40 you're going to get a very adverse HEP that won't reflect what the plants going to be at.

So, I think the question is really after how do you estimate something without the procedures but knowing there's going to be some down the road. And typically, we do that on our side, the industry by using screening values.

DR. XING: Thanks, Roy. So, my basis for that is for modeling procedure, IDHEAS offer I think, a longer, like ten different attributes.

So, even if you don't have procedure you can model these other attributes. Not no procedures but something you think that's will be likely, which will give you a reasonable HEP for your situation.

MR. LINTHICUM: Thanks, Jing.

DR. XING: Thank you.

MR. PETERS: All right, thank you. I'm sorry, I'm multitasking here and I'm trying to find the next question. We have tons of them still flowing in. Okay, there we are.

So, this is for Michelle Kichline. IDHEAS has been --

and feasibly Jing -- IDHEAS has been well tested for a number of FLEX scenarios where no well-established stated practice exists. Has it received the same level of testing and comparison for classic PRA scenarios that actually drive the risk parameters and used in risk-informed decision making?

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

41 MS. KICHLINE: So, yes and no. We haven't formally tested I would say like we did with the FLEX scenarios where we had a group of diverse people who went through the procedure for everything.

We only did that for FLEX. However, in the IDHEAS-ECA report, there are examples for more traditional items. And that's going to be kind of focus moving forward as we look to transition away from SPAR-H.

We do need to go ahead and look at our SPAR models and see what's modeled and see what kind of a difference it's going to make for us to transition that stuff over to IDHEAS-ECA. I don't know, does Jing have any additional comment on that?

DR. XING: Just a minor observation. Thanks, Michelle. So, ongoing IDHEAS dependency workgroup, we have six HRA analysts and every analyst use, I would call those more traditional, PRA, PRA scenario to try that out with IDHEAS-ECA and also compare the method that had been used before such as SPAR-H.

MR. PETERS: So, I think here's an easy question for the group. If IDHEAS identified a weakness, could increased training and simulator time on those scenarios reduce that weakness?

MR. ROSEBROOK: I certainly think, yes, ideally it feels like IDHEAS, if identify the weakness and we can communicate that to the licensee, you know, if they can use that information to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

42 address that issue that's how corrective action program is supposed work so.

And that sometimes is the output of a lot of these SDP evaluations is to identify a weakness and then you develop corrective actions for it, so I certainly think that would, you know, this tool could help do that. Because it does give you a little bit more granularity of the weakness.

MR. PETERS: Jing, did you have anything to add to that?

DR. XING: Yes, I concur with what Andy said.

Basically, I think one power IDHEAS-ECA has is this large detail, the concrete set of PIF attribute.

So, it's like you go to doctor, you're not just state hey, what's wrong, exactly tell you what happened so you can therefore that give you the direction how you can improve that from crew training. So, it's not just say hey, your HRA is bad.

MR. PETERS: There's granularity in that training performance shaping factor that can account for that improved training. So, thanks, Jing. So, okay, here's a tough question from our friends from Norway.

IDHEAS-ECA uses five crew failure modes and 20 PIFs. In effect this makes it a kind of PIF, PSF based method.

You have experience as to the ease of use of this type of method compared to IDHEAS at-Power that has many CF NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

43 zooms and fewer PIFs unlike other cause-based decision tree methods?

DR. XING: Well, I guess that's a question for me. So, we didn't talk IDHEAS At-Power in this session. For those of you don't know, IDHEAS at-Power is another application with IDHEAS with.

It was specifically for modeling up actions in control room at the Power Station operation. And it had more CF at failure mode but the failure modes are subset with predictors at high level IDHEAS-ECA failure modes.

Say IDHEAS-ECA has a failure mode detection and IDHEAS at-Power detection to source of the information detect alarm to something very specific for control room design.

And it has, IDHEAS at-Power has less PIFs because we assume the other PIFs for control room situation are because the environment factors, mostly has no impact check is nominal.

So, these two are the same approach. They both have the principles in IDHEAS general methodology. That's all.

MS. KICHLINE: And, Sean, I can talk a little bit about the ease of use. One of the great things we found about IDHEAS-ECA is that it has been extremely easy for people to pick up.

I was really amazed in the dependency workshops that we're doing right now that we asked people, a couple of people who were not familiar with IDHEAS-ECA at all to basically learn the method before they could do their dependency scenarios.

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

44 And they had no issue with it. I mean, within, you know, I might have met with some of the people for an hour to talk with them through how to use the software and they picked it up very quickly.

So, it seems very natural to people to be able to pick up IDHEAS-ECA and they all gave us good feedback on their ability to choose what influencing factors they thought were important.

MR. LINTHICUM: Yes, Sean, this is Roy. I would concur on the ease of use, you know, when we did our comparison, we had no training from the NRC, the people using it and it was very easy to learn to use and to use it.

We didn't know, I guess one caution is, I mean, there is a lot of flexibility in the tool, which is good. But that also leads to, potentially you get a lot of subjectivity and variance between different users that could result in significantly different results.

So, it may be a benefit to have some additional guidance on how to make some of the more key decisions that you're using.

MR. PETERS: Yes, thanks, Roy. One thing I've always been warned, ease of use also means ease of misuse. So, there is, very crucial that people get training and understand all the requirements of a qualitative analysis, a good qualitative analysis to be able to use the methods. So, thanks.

So, here's a question, is there a way to determine an NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

45 HEP estimate using IDHEAS is realistic versus conservative? How do we judge what is realistic?

MR. LINTHICUM: I'll take a quick stab. I mean, that's the hard part. So, you know, and this goes back to the more you can benchmark it to real data and I think that was the discussion in one of the previous group sessions, the need to validate your results against your data helps.

But to me it's always worth benchmarking with, you know, a licensed operator at the plant to see whether or not it matches expectations. And just because it doesn't, doesn't mean it's wrong or unrealistic.

But at least you can drill down and try and do a more subjective evaluation. Once again, what's driving the results that appear to be or may appear to be unrealistic before you make that call and determine if those are correct or not.

MR. PETERS: Thanks, Roy.

MR. ROSEBROOK: Actually, I'd concur with Roy there. One of the things you get into with these types of, one of reasons that we were looking for a better tool is, you know, the existing tools that we have like THERP and SPAR-H, a lot of times they will over-penalize the success of the human action.

And make something be completely dependent whereas, you know, realistically I have a whole crew of people in a control room and it only takes one of them to break the quote, group NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

46 think.

My adjustment factor, my dependency factor needs to be realistic to take that into account. I think IDHEAS does a much better job of dealing with the crew dynamic as opposed to some of the other tools that we have like the THERP.

Because they treat the individual and the crew as one in the same and THERP and IDHEAS does allow you to break some of that out and give you a much more realistic, you know, probability to failure.

I mean, but when you're dealing with theoreticals and what do we think the operator will do given the scenario, there's always going to be some deal of uncertainty but I think this does a much better job of getting you at least in the ballpark.

DR. XING: Thanks. So we thought the IDHEAS HEP calculation was based on real human error data and we planned in the future we will attach the data sources to the current IDHEAS-ECA software.

So, that is, you look at data sources, that is providing your benchmark how close your scenario to that, the scenario in the data source. Like for example, the data point I showed in my presentation.

A crew's failed eight times out of 69 times in diagnosis means a very abnormal scenario that they didn't see before. So that give you a sense where's the likelihood, the chance of failure. That's NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

47 the realistic part, you can compare your number with those realistic data. Thanks.

MR. PETERS: Thanks, Jing. Next question. During any reactor evolution, normal ops of BDBEs, beyond design basis events, isn't it the license SRO that makes the ultimate decision on whether to take any action?

Is so, does IDHEAS account for information from the various on and offsite groups that are stood up during certain events?

Anybody like to take that?

MS. KICHLINE: So, I can start and we're almost out of time so, if I don't answer it correctly, we'll just be like oh, sorry times up.

I was just going to say that it's nice that IDHEAS has a teamwork causal failure mode and that's something we don't always have. It, well, like in SPAR-H.

So, there's specifically a failure mode for teamwork when teamwork is appropriate for the situation. And so, you can look at those factors as well if that's part of your event.

MR. LINTHICUM: And I'd point out, I think that's a potential weakness of any of the current HRA tools including the ones the industry uses as far as, you know, can you take credit for say the emergency operating facility and the technical support center.

Once again, those are all predicated, do you have the time. I mean, if it's a real short action, so it hassn't fully been staffed NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

48 yet. But, you know, that is an area I think is something we should potentially look at further research on. See if we can better credit those other facilities.

MR. PETERS: Okay. It claims we're out of time but I think we only have a couple more questions and I'm trying to check if this is a hard deadline. So, here's a question for the group.

Is there any future intent to IDHEAS-ECA in human factors reviews as safety significant I&C human system interface license amendment requests for important human actions in addition to event analysis?

And I'm not sure we have the right people on the call to make that call. We don't have any of the human factor's reviewers in the NRC on that call.

But it's, Jing, if you'd like to take a cut at how it's designed for human factors use, I would like to hear that.

DR. XING: Okay. So, IDHEAS-ECA provided this and it you look at the PIFs, those are very familiar to the topics we look at in the human factors work, how were the procedures, how were the staffing.

And IDHEAS-ECA gave you this list of attributes like when you look at procedure here are the ten things that can cause, lead to human error. So, that could be a good, very useful prioritization tool for human factors review to focus on important topics.

So, Brian Green from NRR and I several years ago, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

49 we explored this possibility. We should do more on that.

MR. PETERS: Yes, thanks. And one of the crucial aspects of IDHEAS program is that we linked it to the human factors science and we backed it up with data so, its utility is beyond, the IDHEAS-G program is beyond just HRA.

You can use it for multiple applications. I think we answered what are the specific first applications that the industry would see for IDHEAS-ECA. But, Roy, if we didn't, if you'd like to weigh in anymore, just let me know.

And let's see, we have a comment that more operating experience data input to make IDHEAS results more realistic. Is there a plan to expand participation of SACADA data collection to enhancement data for IDHEAS?

And that's kind of a programmatic question. Roy hinted at it from a group perspective that he thinks it would be very helpful to industry if industry participated in the SACADA program.

And got that data and shared the data. So, from a programmatic standpoint we are trying to work with both U.S. and international counterparts to try to get them to get on board and use the SACADA program.

SACADA, what it is, is we partnered with certain utilities and they are putting all their operational training data and some of their field operational data into our database.

And they're using that to help optimize their training NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

50 programs and what we're using it for is we're pulling the data out and giving us information about human performance rates.

And as we're getting thousands upon thousands of data points each year, it's helping us get to the numbers of data points that we need to make informed human error probability assessments.

So, the more we can do, this is a free program that we give to the utilities. We give them free training and in exchange we get free data. So, it's kind of win-win for everybody.

And I think what Roy was hinting at and I've talked to him in the past, that it may also be very helpful to some of the PRA departments in the industry also.

So, this is just a win for everybody and there's no type of regulatory action we take on it other than just refining our models.

And what we're finding through making our models more realistic is that it actually lowers a lot of human error probabilities through the more conservative models that we have currently. So, that's also a big benefit to the industry. And --

MR. LINTHICUM: Yes, Sean, I think you hit on a key element there. I do think it would be a benefit but, you know, the first question that would always come up is well, what's the regulatory risk associated with participating in the program.

I think you pretty much addressed that. And to me, the other element is, you know, yes, it's good to help the NRC develop their models but how do we then roll that information into our PRA NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

51 models.

So, we want to make sure there's at least an arrangement, you know, with EPRI who does the development for the industry on these rate calc, which I think almost everyone if not everyone in the industry is using now on the U.S. side.

So, we can, you know, get our models in sync using the same data. So those are the types of things we would probably want to make sure are in place to expand the program. But I do think it would be a big benefit.

MR. PETERS: Absolutely. So, thanks, I totally agree.

That was the last question that we have and I guess we'll go ahead and end the session here. I just wanted to thank all of our panelists.

I think you guys did a fabulous job and I think, and I'd like to thank the audience because those are some really fabulous questions for the panelists to consider.

And with that, again on the last slide you will see that we have contact information. So, if you have any follow up questions or thoughts, please feel free to email us.

You can talk to myself, Dr. Jing Xing or if you want to get the software you can contact James Chang. He also is the leader of our SACADA program, if you have questions on the SACADA program. So, thank you very much for your time everybody.

(Whereupon, the above-entitled matter went off the record at 12:06 p.m.)

NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433

52 NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.

(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433